Saturday, December 20, 2008

Plugins - part 6: setuptools based approach

setuptools is collection of enhancements to the standard Python distutils module. While it is not included in the Python standard library, chances are that it is already installed on your system if you have installed some additional Python libraries since it is so widely used to install Python packages ("eggs") via easy_install.

setuptools is also used in many applications as a handler for plugins. Many such applications include tutorials for creating new plugins using setuptools. For a somewhat general introduction to using setuptools to create plugin based applications, I suggest you have a look at the two tutorials from which this one is inspired.

Our starting point for this tutorial is essentially the same as in the third one in this series. We start with the following files:

root_directory/
calculator.py
setup.py
plugins/
__init__.py
base.py
op_1.py
op_2.py

where we have one new file, setup.py. This file is a special file for setuptools. Its content is as follows:

''' run with python setup.py develop '''

from setuptools import setup, find_packages

setup(
name="Calculator_s_tools",
version="1.0",
packages=['plugins'],
entry_points="""
[plugin_tutorial.s_tools]
add = plugins.op_1:operator_add_token
sub = plugins.op_1:operator_sub_token
mul = plugins.op_1:operator_mul_token
div = plugins.op_1:operator_div_token
pow = plugins.op_2:operator_pow_token"""
)

The key concept for setuptools is that of "entry_points". We define some entry points, with a name "plugin_tutorial.s_tools" chosen as unique to our application. Within this entrypoint we indicate which classes should be imported. This method effectively replace our custom method for finding and loading plugins. However, if you remember from previous tutorials, the way the application was designed originally (all within a single file) resulted in a "wart", where we had to create a link to a function ("expression") in each plugin module. Since setuptools will import the classes for us, we have no way to tell it how to fix that "wart" - we have to find another way. The method we chose was to create a different Plugin base class, one that implements the Borg idiom, so that each instance shares common attributes. Here's the new "base.py":

import os
import sys
import pkg_resources # setuptools specific

OPERATORS = {}
ENTRYPOINT = 'plugin_tutorial.s_tools' # same name as in setup.py

class Plugin(object):
'''A Borg class.'''
_shared_states = {}
def __init__(self):
self.__dict__ = self._shared_states

def init_plugins(expression):
'''simple plugin initializer'''
Plugin().expression = expression # fixing the wart
load_plugins()

def load_plugins():
'''setuptools based plugin loader'''
for entrypoint in pkg_resources.iter_entry_points(ENTRYPOINT):
plugin_class = entrypoint.load()
OPERATORS[plugin_class.symbol] = plugin_class

The actual plugin files are slightly modified to derive from the new Plugin class; for example, op_2.py contains the following:

from plugins.base import Plugin

class operator_pow_token(Plugin):
symbol = '**'
lbp = 30
def led(self, left):
return left ** self.expression(30-1)

where "expression" is now a class variable obtained from Plugin.

The code involved to make the setuptools approach is approximately the same level of complexity as the class-based plugin system covered previously. The advantages of using the setuptools approach are as follows:
  1. Since it is a widely used tool, many people know how to use it properly.
  2. It is possible to package plugins as eggs to be uploaded to a repository.
  3. It is possible to keep track of dependencies in a fairly detailed way (e.g. module X version Y required).
By comparison, it suffers from the following disadvantages:
  1. Some information about plugin location (entry_points name) is duplicated, appearing in both setup.py and base.py (in our example).
  2. Automatic plugin discovery without editing of a file (setup.py) is not possible, unlike the cases we covered before. Because of this, dynamic loading of "external" plugins while the application is already running may be problematic to achieve. (I am not familiar enough with setuptools to determine if it is feasible or not.) See the first comment par Phillip J. Eby on how to achieve this.
  3. A preliminary step ("python setup.py develop") is required to generate entrypoints information.
  4. A number of additional files are created by the previous step, "cluttering" slightly the file structure by adding an extra directory with a few files.
That being said, the differences between the two approaches are relatively minor when everything is taken into account. Choosing one approach over the other is a matter of individual taste - at least for simple applications such as the one we considered.

3 comments:

PJE said...

You can do dynamic loading using the find_plugins() function, as well as any other pkg_resources API for finding and adding eggs to sys.path at runtime. Also, you can use the add_activation_listener() function to receive a callback whenever eggs are added to sys.path, and you can then check the received distribution object for entry points you care about.

Many packages and tools out there use entry points to implement plugins, by the way. For example, the CherryPy/Turbogears template engine standard is entry-point driven, and so are tools like Paste, zc.buildout, and of course setuptools itself.

Oh, and one more thing... as far as I can tell, you don't need the borg pattern here. Just do "plugin_class.expression = expression" after loading each class. Personally, though, I'd just pass the expression as an initialization parameter to Plugin.__init__. Having a global (even stored on the class(es)) means you can't use threads, among other Bad Things.

André Roberge said...

@PJE: thanks for your comments and suggestion. I had thought of doing the plugin_class.expression = expression (which is similar to what I did at the module level before) but I thought I'd try something different this time.

You are also right, of course, about your comment regarding the problem with a global expression. As I pointed out at the beginning of the series, this is a "wart" due to the fact that I chose an already existing application not intended to be broken up the way I did, and I just wanted to show what was the minimal work required to make use of plugins and, from there, look at various ways of implementing plugins.

Finally, and perhaps more importantly, thank you for creating setuptools, among all of your other Python contributions.

CrapEDM(Carpe Diem) said...

That Borg class bent my brain a little. You effectively made a Singleton where you can make multiple instances, but they're all the same... It kind of makes prototypal inheritance...