25 Sep, 2023
13 years ago I wrote about how to use setuptools to have a plugin ecosystem in Python Setuptools Egg Plugins. It is also an approach followed by CKAN 10 years ago at least and you can see 267 extensions at the moment.
Well, times move on so here is how to do the same thing with Poetry
and Python 3.10's import lib.metadata module.
Let's follow on from the Poetry Git Submodules example where you have a blog package that depends on a serve package.
The packages themselves can be installed from pypi, local git submodules or via any other technique, it doesn't matter.
Let's imagine that the blog package wants to be able to support lots of different server messages, not just the one from serve.
To support this it will define an entry point called blog.serve_message that any package that wants to provide a serve message will implement.
First we change the implementation from:
import serve
print(serve.serve_msg)
to:
import importlib.metadata
entry_points = importlib.metadata.entry_points()
for entry_point in entry_points.select(group='blog', name='serve_msg'):
serve_msg = entry_point.load()
print(serve_msg)
Now if you run:
python3 blog/__init__.py
you will see no output, because no packages yet implement this entry point.
Next edit the serve project's pypackage.toml file to add this:
[tool.poetry.plugins."blog"]
"serve_msg" = "serve:serve_msg"
Then commit it, pull the change into the blog project as a git submodule (in this example) and install it:
cd serve
git add pyproject.toml
git commit -m 'Added entry point'
cd ../blog
git -c protocol.file.allow=always submodule update --remote
pip uninstall --yes serve && poetry install
You may have noticed a sleight of hand with that last command. Poetry doesn't notice changes to entry points for packages in editable mode see issue 6639 so you have to uninstall it using pip before running the install as normal.
Now that is done, try again:
python3 blog/__init__.py
Hello from serve
And you can see the output you expect, coming from the serve module, discovered via an entry point.
It is worth noting that the implementation doesn't have to have the name of the entry point. For example if the string was defined in a variable named message_for_server then the entry would be:
[tool.poetry.plugins."blog"]
"serve_msg" = "serve:message_for_server"
Entry points can point to any Python object, not just strings, so can be used for classes, functions etc etc.
In real life, just because a package with a matching entry point is installed, doesn't mean we always want to use it and alternatively, a single package might want to define multiple implementations of an entry point.
Systems like CKAN require that you pass the names of packages for which you want plugins enabled as a config option, see Enabling the Plugin.
To solve both these problems and enable CKAN-style behaviour we can take another approach.
This time we'll put the name of the plugin into the entry point group, which frees up the entry point name to be a unique identifier for the specific plugin implementation. For example:
[tool.poetry.plugins."blog.serve_msg"]
"my_serve_msg_plugin" = "serve:serve_msg"
In this latter case you would use this code to only load the entry points if the named plugin was enabled as an argument on the command line:
import sys
import importlib.metadata
entry_points = importlib.metadata.entry_points()
for name in sys.argv[1:]:
found = False
for entry_point in entry_points.select(group='blog.serve_msg', name=name):
found = True
serve_msg = entry_point.load()
print(serve_msg)
assert found, f'Could not find a "blog.serve_msg" entry point called "{name}".'
If you make the above two changes, then uninstall and re-install serve:
cd serve
git add pyproject.toml
git commit -m 'Alternative entry point approach'
cd ../blog
git -c protocol.file.allow=always submodule update --remote
pip uninstall --yes serve && poetry install
You now have this behaviour:
python3 blog/__init__.py my_serve_msg_plugin
Hello from serve
If you run it without my_serve_msg_plugin you won't get any output. If you run it with a plugin name that doesn't exist you'll get an error:
python3 blog/__init__.py no_such_plugin
Traceback (most recent call last):
File ".../blog/blog/__init__.py", line 11, in <module>
assert found, f'Could not find a blog.serve_msg entry point called "{name}".'
AssertionError: Could not find a blog.serve_msg entry point called "no_such_plugin".
To prevent packages declaring names that look like they are for other modules, you could also enforce a convention that names start with the distribution package name and an _ character.
In our case both package name and the module name are serve so to implement this let's start by changing the entry point definition in serve to something like this:
[tool.poetry.plugins."blog.serve_msg"]
"serve_my_serve_message" = "serve:serve_msg"
NOTE: Not all packages are named the same as the module(s) they provide. For example the PyYAML distribution package (installed with poetry add PyYAML provides the yaml module) so following this convention, any entrypoint names it used would start PyYAML_.
Now change the blog code to this, which uses a slightly different approach, only checking each package's entry points rather searching all of them for a name match:
import sys
from importlib.metadata import distribution
for name in sys.argv[1:]:
assert '_' in name, f'Entry point names must start with the package name and an underscore but "{name}" does not contain an underscore.'
expected_package = name[:name.find('_')]
dist = distribution(expected_package)
found = False
for entry_point in dist.entry_points:
if entry_point.name == name:
assert not found, 'Already loaded an entry point from "{expected_package}" named "{name}"'
serve_msg = entry_point.load()
print(serve_msg)
found = True
assert found, f'Could not find a "{name}" entrypoint for "blog.serve_msg" in "{expected_package}". Perhaps it is not installed or the entry point name is incorrect.'
After re-installing:
cd serve
git add pyproject.toml
git commit -m 'Entrypoint names start with module name'
cd ../blog
git -c protocol.file.allow=always submodule update --remote
pip uninstall --yes serve && poetry install
You can now run:
python3 blog/__init__.py serve_my_serve_message
Hello from serve
I think this is how I would write a plugin system personally now.
Of course CKAN is a bit different in that it only defines one entry point called ckan.plugins that should point to a class. The class can implement different interfaces depending on what it should do. I think I would now choose a different entry point for each thing an application wants to allow to be plugged in.
Thinking about this a bit more, what happens when you have plugins that load other plugins that themselves take the same approach? You'd need to pass on the list of configured plugins to them too.
After a bit of thought I think the best option is to use environment variables rather than sys.argv for the list of plugin names. A convention could be followed that named the environment variable as the normalised distribution package name, uppercased and with - replaced with _ and then _PLUGINS appended.
The normalised names are always unique so this should be OK.
PYYAML_PLUGINS=some_plugin SERVE_PLUGINS=serve_my_serve_message python3 blog/__init__.py
See Core Metadata Name and Name Normalization as the two relevant bits of spec.
The other bit is how to initialise plugins that need different things to happen at different stages of the application lifecycle? The answer is that for this case, entry points should probably point to a class instance (or equivalent) with different methods that will be called at different stages of the lifecycle.
Also the code that consumes the entry point can define a Python Protocol class in its code that the plugin should conform to, and then static type checking would work across plugin boundaries without the plugin needing to import and base class from the consuming package.
Be the first to comment.
Copyright James Gardner 1996-2020 All Rights Reserved. Admin.