Poetry with Git Submodules

25 Sep, 2023

Something I might like to do in future is to use git submodules for pulling in private dependant packages, rather than set up my own pypi or equivalent.

Here's how that would look with poetry.

Let's imagine a new blog engine called blog needs a package called serve.

Start with the serve package (make sure you aren't already in a poetry shell or things will get confused):

poetry new serve
cd serve
git init
git add *
git commit -m 'First commit'
cd ..

We end up with serve like this:

.
├── README.md
├── pyproject.toml
├── serve
│   └── __init__.py
└── tests
    └── __init__.py

and pyproject.toml looks like this (email redacted):

[tool.poetry]
name = "serve"
version = "0.1.0"
description = ""
authors = ["James Gardner <james@...>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Notice that the authors is automatically generated for you, perhaps from a git config?

Now for the blog package:

poetry new blog
cd blog
git init 
git add *
git commit -m 'First commit'

NOTE: For this tutorial we explicitly have to use the git -c protocol.file.allow=always flag in key commands because the file protocol for submodules isn't supported by default in git anymore for security reasons. When you do this with real remote repos you won't need that flag and should leave it out.

Add in the dependency on serve as a submodule:

git -c protocol.file.allow=always submodule add ../serve
git add .gitmodules
git add serve
poetry add -e ./serve
git add poetry.lock pyproject.toml
git commit -m 'Added serve dependency'
cd ..

This time we end up with:

.
├── README.md
├── blog
│   └── __init__.py
├── poetry.lock
├── pyproject.toml
├── serve
│   ├── README.md
│   ├── pyproject.toml
│   ├── serve
│   │   ├── __init__.py
│   │   └── __pycache__
│   │       └── __init__.cpython-311.pyc
│   └── tests
│       └── __init__.py
└── tests
    └── __init__.py

and pyproject.toml looks like this (email redacted):

[tool.poetry]
name = "blog"
version = "0.1.0"
description = ""
authors = ["James Gardner <james@...>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
serve = {path = "serve", develop = true}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

This time there is poetry.lock with exact versions:

# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.

[[package]]
name = "serve"
version = "0.1.0"
description = ""
optional = false
python-versions = "^3.11"
files = []
develop = true

[package.source]
type = "directory"
url = "serve"

[metadata]
lock-version = "2.0"
python-versions = "^3.11"
content-hash = "c007f69ca89c7dd63bedf1787a3e2aa4524e7d97db4956ac20bf481dff05191b"

Now we can add some code to the two packages:

cd serve
cat << EOF > serve/__init__.py
serve_msg = 'Hello from serve'
EOF
git add serve/__init__.py
git commit -m 'Added serve_msg'
cd ../

cd blog
cat << EOF > blog/__init__.py
import serve
print(serve.serve_msg)
EOF
git add blog/__init__.py
git commit -m 'Using serve_msg'

Let's enter a poetry shell:

poetry shell

Now if we try to run the blog code It won't work yet as we haven't updated the submodule within for serve within the blog project:

python3 blog/__init__.py

Traceback (most recent call last):
  File "/Users/james/Desktop/tmp/blog/blog/__init__.py", line 2, in <module>
    print(serve.serve_msg)
          ^^^^^^^^^^^^^^^
AttributeError: module 'serve' has no attribute 'serve_msg'

So let's update the submodule:

git -c protocol.file.allow=always submodule update --remote
git add serve
git commit -m 'Updated serve dependency'

And try again:

python3 blog/__init__.py

Hello from serve

There we have it, no need to release a version of serve to a package index, or even to bump its version number.

Re-creating a Poetry Shell

By the way, if you get in a pickle with your poetry virtual env, and assuming you are already in a poetry shell, you can always destroy it like this:

poetry env remove $(which python3)
exit

Then re-create it like this:

poetry shell

And install the dependencies again like this:

poetry install

You can't run all the commands by copying and pasting them all at once, because of the changes being made to the shell, but running them as above works fine

Comments

Add Comment

Copyright James Gardner 1996-2020 All Rights Reserved. Admin.