Repository files navigation

plux is the dynamic code loading framework used inLocalStack.

Plux builds a higher-level plugin mechanism aroundPython's entry point mechanism.It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in yoursetup.py).

• PluginSpec: describes aPlugin. Each plugin has a namespace, a unique name in that namespace, and aPluginFactory (something that createsPlugin the spec is describing.In the simplest case, that can just be the Plugin's class).
• Plugin: an object that exposes ashould_load andload method.Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)
• PluginFinder: finds plugins, either at build time (by scanning the modules usingpkgutil andsetuptools) or at run time (reading entrypoints of the distribution usingimportlib)
• PluginManager: manages the run time lifecycle of a Plugin, which has three states:
  • resolved: the entrypoint pointing to the PluginSpec was imported and thePluginSpec instance was created
  • init: thePluginFactory of thePluginSpec was successfully invoked
  • loaded: theload method of thePlugin was successfully invoked

At run time, aPluginManager uses aPluginFinder that in turn uses importlib to scan the available entrypoints for things that look like aPluginSpec.WithPluginManager.load(name: str) orPluginManager.load_all(), plugins within the namespace that are discoverable in entrypoints can be loaded.If an error occurs at any state of the lifecycle, thePluginManager informs thePluginLifecycleListener about it, but continues operating.

Plux supports two modes for building entry points:build-hooks mode (default) andmanual mode.

To build a source distribution and a wheel of your code with your plugins as entrypoints, simply runpython setup.py plugins sdist bdist_wheel.If you don't have asetup.py, you can use the plux build frontend and runpython -m plux entrypoints.

How it works:For discovering plugins at build time, plux provides a custom setuptools commandplugins, invoked viapython setup.py plugins.The command uses a specialPluginFinder that collects from the codebase anything that can be interpreted as aPluginSpec, and creates from it a plugin index fileplux.json, that is placed into the.egg-info distribution metadata directory.When a setuptools command is used to create the distribution (e.g.,python setup.py sdist/bdist_wheel/...), plux finds theplux.json plugin index and extends automatically the list of entry points (collected into.egg-info/entry_points.txt).Theplux.json file becomes a part of the distribution, s.t., the plugins do not have to be discovered every time your distribution is installed elsewhere.Discovering at build time also works when usingpython -m build, since it calls registered setuptools scripts.

Manual mode is useful for isolated build environments where dependencies cannot be installed, or when build hooks are not suitable for your build process.

To enable manual mode, add the following to yourpyproject.toml:

[tool.plux]entrypoint_build_mode ="manual"

In manual mode, plux does not use build hooks. Instead, you manually generate entry points by running:

python -m plux entrypoints

This creates aplux.ini file in your working directory with the discovered plugins. You can then include this file in your distribution by configuring yourpyproject.toml:

[project]dynamic = ["entry-points"][tool.setuptools.package-data]"*" = ["plux.ini"][tool.setuptools.dynamic]entry-points = {file = ["plux.ini"]}

You can also manually control the output format and location:

python -m plux discover --format ini --output plux.ini

To build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded.And then, at runtime, you need a component that uses thePluginManager to get those plugins.

This is the way we went withLocalstackCliPlugin. Every plugin class (e.g.,ProCliPlugin) is essentially a singleton.This is easy, as the classes are discoverable as plugins.Simply create a Plugin class with a name and namespace and it will be discovered by the build timePluginFinder.

frompluximportPlugin# abstract case (not discovered at build time, missing name)classCliPlugin(Plugin):namespace="my.plugins.cli"defload(self,cli):self.attach(cli)defattach(self,cli):raiseNotImplementedError# discovered at build time (has a namespace, name, and is a Plugin)classMyCliPlugin(CliPlugin):name="my"defattach(self,cli):# ... attach commands to cli object

now we need aPluginManager (which has a generic type) to load the plugins for us:

cli=# ... needs to come from somewheremanager:PluginManager[CliPlugin]=PluginManager("my.plugins.cli",load_args=(cli,))plugins:List[CliPlugin]=manager.load_all()# todo: do stuff with the plugins, if you want/need#  in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argument

When you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin classfor each plugin. Instead we want to use the samePlugin class to do the same thing, but use several instances of it.ThePluginFactory, and the fact thatPluginSpec instances defined at module level are discoverable (inpiredbypluggy), can be used to achieve that.

frompluximportPlugin,PluginFactory,PluginSpecimportimportlibclassServicePlugin(Plugin):def__init__(self,service_name):self.service_name=service_nameself.service=Nonedefshould_load(self):returnself.service_nameinconfig.SERVICESdefload(self):module=importlib.import_module("localstack.services.%s"%self.service_name)# suppose we define a convention that each service module has a Service class, like moto's `Backend`self.service=module.Service()defservice_plugin_factory(name)->PluginFactory:defcreate():returnServicePlugin(name)returncreate# discoverables3=PluginSpec("localstack.plugins.services","s3",service_plugin_factory("s3"))# discoverabledynamodb=PluginSpec("localstack.plugins.services","dynamodb",service_plugin_factory("dynamodb"))# ... could be simplified with convenience framework code, but the principle will stay the same

Then we could use thePluginManager to build a Supervisor

frompluximportPluginManagerclassSupervisor:manager:PluginManager[ServicePlugin]defstart(self,service_name):plugin=self.manager.load(service_name)service=plugin.serviceservice.start()

with the@plugin decorator, you can expose functions as plugins. They will be wrapped by the frameworkintoFunctionPlugin instances, which satisfy both the contract of a Plugin, and that of the function.

frompluximportplugin@plugin(namespace="localstack.configurators")defconfigure_logging(runtime):logging.basicConfig(level=runtime.config.loglevel)@plugin(namespace="localstack.configurators")defconfigure_somethingelse(runtime):# do other stuff with the runtime objectpass

With a PluginManager viaload_all, you receive theFunctionPlugin instances, that you can call like the functions

runtime=LocalstackRuntime()forconfiguratorinPluginManager("localstack.configurators").load_all():configurator(runtime)

If you are building a python distribution that exposes plugins discovered by plux, you need to configure your projects build system so other dependencies creates theentry_points.txt file when installing your distribution.

For apyproject.toml template this involves adding thebuild-system section:

[build-system]requires = ['setuptools','wheel','plux>=1.3.1']build-backend ="setuptools.build_meta"# ...

You can pass additional configuration to Plux, either via the command line or your projectpyproject.toml.

The following options can be configured in the[tool.plux] section of yourpyproject.toml:

[tool.plux]# The build mode for entry points: "build-hooks" (default) or "manual"entrypoint_build_mode ="manual"# The file path to scan for plugins (optional)path ="mysrc"# Python packages to exclude during discovery (optional)exclude = ["**/database/alembic*"]# Python packages to include during discovery (optional), setting this will ignore all other pathsinclude = ["**/database*"]

Controls how plux generates entry points:

• build-hooks (default): Plux automatically hooks into the build process to generate entry points
• manual: You manually control when and how entry points are generated (seeManual mode)

Specifies the file path to scan for plugins. By default, plux scans the entire project.

A list of paths to include during plugin discovery. If specified, only the named items will be included. If not specified, all found items in the path will be included. Theinclude parameter supports shell-style wildcard patterns.

Examples:

# Include multiple patternspython -m plux discover --include"myapp/plugins*,myapp/extensions*" --format ini

You can also specify these values in the[tool.plux] section of yourpyproject.toml as shown above.

Note: Wheninclude is specified, plux ignores all other paths that would otherwise be found.

Whendiscovering entrypoints, Plux will try importing your code to discover Plugins.Some parts of your codebase might have side effects, or raise errors when imported outside a specific context like some databasemigration scripts.

You can ignore those Python packages by specifying the--exclude flag to the entrypoints discovery commands:

# Exclude database migration scriptspython -m plux entrypoints --exclude"**/database/alembic*"# Exclude multiple patterns (comma-separated)python -m plux discover --exclude"tests*,docs*" --format ini

The option takes a list of comma-separated values that can be paths or package names with shell-style wildcards.'foo.*' will exclude all subpackages offoo (but notfoo itself).

You can also specify these values in the[tool.plux] section of yourpyproject.toml as shown above.

pip install plux

Create the virtual environment, install dependencies, and run tests

make venvmake test

Run the code formatter

make format

Upload the pypi package using twine

make upload