Plugins - part 6: setuptools based approach

setuptools is collection of enhancements to the standard Python distutils module. While it is not included in the Python standard library, chances are that it is already installed on your system if you have installed some additional Python libraries since it is so widely used to install Python packages ("eggs") via easy_install.

setuptools is also used in many applications as a handler for plugins. Many such applications include tutorials for creating new plugins using setuptools. For a somewhat general introduction to using setuptools to create plugin based applications, I suggest you have a look at the two tutorials from which this one is inspired.

Our starting point for this tutorial is essentially the same as in the third one in this series. We start with the following files:

root_directory/
    calculator.py
    setup.py
    plugins/
        __init__.py
        base.py
        op_1.py
        op_2.py

where we have one new file, setup.py. This file is a special file for setuptools. Its content is as follows:

''' run with python setup.py develop '''

from setuptools import setup, find_packages

setup(
    name="Calculator_s_tools",
    version="1.0",
    packages=['plugins'],
    entry_points="""
        [plugin_tutorial.s_tools]
        add = plugins.op_1:operator_add_token
        sub = plugins.op_1:operator_sub_token
        mul = plugins.op_1:operator_mul_token
        div = plugins.op_1:operator_div_token
        pow = plugins.op_2:operator_pow_token"""
       )

The key concept for setuptools is that of "entry_points". We define some entry points, with a name "plugin_tutorial.s_tools" chosen as unique to our application. Within this entrypoint we indicate which classes should be imported. This method effectively replace our custom method for finding and loading plugins. However, if you remember from previous tutorials, the way the application was designed originally (all within a single file) resulted in a "wart", where we had to create a link to a function ("expression") in each plugin module. Since setuptools will import the classes for us, we have no way to tell it how to fix that "wart" - we have to find another way. The method we chose was to create a different Plugin base class, one that implements the Borg idiom, so that each instance shares common attributes. Here's the new "base.py":

import os
import sys
import pkg_resources  # setuptools specific

OPERATORS = {}
ENTRYPOINT = 'plugin_tutorial.s_tools'  # same name as in setup.py

class Plugin(object):
    '''A Borg class.'''
    _shared_states = {}
    def __init__(self):
         self.__dict__ = self._shared_states

def init_plugins(expression):
    '''simple plugin initializer'''
    Plugin().expression = expression  # fixing the wart
    load_plugins()

def load_plugins():
    '''setuptools based plugin loader'''
    for entrypoint in pkg_resources.iter_entry_points(ENTRYPOINT):
        plugin_class = entrypoint.load()
        OPERATORS[plugin_class.symbol] = plugin_class

The actual plugin files are slightly modified to derive from the new Plugin class; for example, op_2.py contains the following:

from plugins.base import Plugin

class operator_pow_token(Plugin):
    symbol = '**'
    lbp = 30
    def led(self, left):
        return left ** self.expression(30-1)

where "expression" is now a class variable obtained from Plugin.

The code involved to make the setuptools approach is approximately the same level of complexity as the class-based plugin system covered previously. The advantages of using the setuptools approach are as follows:

1. Since it is a widely used tool, many people know how to use it properly.
2. It is possible to package plugins as eggs to be uploaded to a repository.
3. It is possible to keep track of dependencies in a fairly detailed way (e.g. module X version Y required).
   

By comparison, it suffers from the following disadvantages:

1. Some information about plugin location (entry_points name) is duplicated, appearing in both setup.py and base.py (in our example).
2. Automatic plugin discovery without editing of a file (setup.py) is not possible, unlike the cases we covered before. Because of this, dynamic loading of "external" plugins while the application is already running may be problematic to achieve. (I am not familiar enough with setuptools to determine if it is feasible or not.) See the first comment par Phillip J. Eby on how to achieve this.
   
3. A preliminary step ("python setup.py develop") is required to generate entrypoints information.
4. A number of additional files are created by the previous step, "cluttering" slightly the file structure by adding an extra directory with a few files.
   

That being said, the differences between the two approaches are relatively minor when everything is taken into account. Choosing one approach over the other is a matter of individual taste - at least for simple applications such as the one we considered.

Plugins - part 5: Activation and Deactivation

(Note: the code indentation may appear to be wrong due to some blogspot's quirks...)

While plugins are a great way to extend the functionality of an application, sometimes it makes sense to limit the number of available features, based on a user's preference. For example, gedit, the official text editor of the Gnome environment, offers the possibility to activate or deactivate a plugin.

In this post, using the class-based plugin approach, I will explain how to add the possibility to activate or deactivate a given plugin. Furthermore, I will show how to use this capability to dynamically load new plugins.

Starting from the beginning...

Our starting point will be the following modified core application (calculator.py):

import re

from plugins.base import OPERATORS, init_plugins, activate, deactivate

class literal_token(object):
 def __init__(self, value):
     self.value = value
 def nud(self):
     return self.value

class end_token(object):
lbp = 0

def tokenize(program):
 for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
    if number:
        yield literal_token(int(number))
    elif operator in OPERATORS:
        yield OPERATORS[operator]()
    else:
        raise SyntaxError("unknown operator: %r" % operator)
 yield end_token()

def expression(rbp=0):
 global token
 t = token
 token = next()
 left = t.nud()
 while rbp < token.lbp:
     t = token
     token = next()
     left = t.led(left)
 return left

def calculate(program):
 global token, next
 next = tokenize(program).next
 token = next()
 return expression()

if __name__ == "__main__":
init_plugins(expression)
assert calculate("+1") == 1
assert calculate("-1") == -1
assert calculate("10") == 10
assert calculate("1+2") == 3
assert calculate("1+2+3") == 6
assert calculate("1+2-3") == 0
assert calculate("1+2*3") == 7
assert calculate("1*2+3") == 5
assert calculate("6*2/3") == 4

    # "**" has not been activated at the start in base.py
    try:
        assert calculate("2**3") == 8
    except SyntaxError:
        print "Correcting error..."
        activate("**")
    assert calculate("2*2**3") == 16

    deactivate('+')
    try:
        assert calculate("1+2") == 3
    except SyntaxError:
        activate('+')
    assert calculate("1+2") == 3

print "Done!"

The new features are indicated by different colours. In blue, we have two new functions imported to either activate or deactivate a given plugin. When the application is started, exponentiation is disabled - this can only be seen by looking at the modified version of base.py. When a disabled plugin is called, a SyntaxError already present in the old version) is raised and we activate the plugin.

To make this possible, we need to modify base.py. Before showing the new version, here's the result of running the above code:

Activating +
Activating -
Activating *
Activating /
Correcting error...
Activating **
Deactivating +
Activating +
Done!

And here's the new version of base.py:

import os
import sys

OPERATORS = {}

# We simulate a configuration file that would be based on a user's preference
# as to which plugin should be activated by default
# We will leave one symbol "**" out of the list as a test.
preferences = ['+', '-', '*', '/']

# We also keep track of all available plugins, activated or not
all_plugins = {}

class Plugin(object):
'''base class for all plugins'''

    def activate(self):
        '''activate a given plugin'''
        if self.symbol not in OPERATORS:
            print "Activating %s" % self.symbol
            OPERATORS[self.symbol] = self.__class__
        if self.symbol not in all_plugins:
            all_plugins[self.symbol] = self.__class__

    def deactivate(self):
        '''deactivate a given plugin'''
        print "Deactivating %s" % self.symbol
        if self.symbol in OPERATORS:
            del OPERATORS[self.symbol]

def activate(symbol):
    '''activate a given plugin based on its symbol'''
    if symbol in OPERATORS:
        return
    all_plugins[symbol]().activate()

def deactivate(symbol):
    '''deactivate a given plugin, based on its symbol'''
    if symbol not in OPERATORS:
        return
    all_plugins[symbol]().deactivate()

def init_plugins(expression):
'''simple plugin initializer
'''
find_plugins(expression)
register_plugins()

def find_plugins(expression):
'''find all files in the plugin directory and imports them'''
plugin_dir = os.path.dirname(os.path.realpath(__file__))
plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
sys.path.insert(0, plugin_dir)
for plugin in plugin_files:
 mod = __import__(plugin)
 mod.expression = expression

def register_plugins():
'''Register all class based plugins.

Uses the fact that a class knows about all of its subclasses
to automatically initialize the relevant plugins
'''
for plugin in Plugin.__subclasses__():
        # only register plugins according to user's preferences
        if plugin.symbol in preferences:
            plugin().activate()
        else:   # record its existence
            all_plugins[plugin.symbol] = plugin

Changes from the old version are indicated in blue (with corresponding comments in green). Note that we did not change a single line of code for the actual plugins! We did use the same names (activate and deactivate) both for a function and a class method. This should probably be avoided in a larger application. In this example, the code is short enough that it should not create too much confusion. In a real application we would also give the possibility of changing the user's preferences, storing the information in some configuration file.

Dynamic activation

Now that we now how to activate and deactivate a plugin, it might be useful to consider dynamic activation of an external plugin, not located in the normal plugins directory. For example, consider the following plugin (located in op_3.py):

from plugins.base import Plugin

class operator_mod_token(Plugin):
symbol = '%'
lbp = 10
def nud(self):
   return expression(100)
def led(self, left):
   return left % expression(10)

This file is located in subdirectory "external" which is at the same level as "plugins" in our sample code. To invoke this plugin from our base application, we need to add the following code to calculator.py:

if __name__ == "__main__":
#...

    # Simulating dynamic external plugin initialization
external_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                          'external')
sys.path.insert(0, external_dir)
mod = __import__('op_3')
mod.expression = expression
    # register this plugin using our default method
register_plugins()
    # Since it is not activated by default, we need to do it explictly
activate('%')
assert calculate("7%2") == 1

print "Done!"

Note that we also need to import register_plugins() from base.py to make this work.

That's it! If you get the code from the py-fun repository, you can try it out yourself.

A small svg module

Update: by combining suggestions made in comments, one can probably do away with much of what I describe in this blog post. To wit:

>>> from xml.etree import ElementTree as etree
>>> from functools import partial
>>> Circle = partial(etree.Element, 'svg:circle')
>>> c = Circle(cx='100', cy='200', fill='red')
>>> etree.tostring(c)
'<svg:circle cx="100" cy="200" fill="red" />'

The only minor drawback is that attributes have to be strings, whereas the module described in this post could handle integer attributes. (Python 2.5+ required for ElementTree)

Original post below
(Time to take a break from the plugins blog series...)

Scalable Vector Graphics (SVG) are becoming more and more common on the web due to the increased support by decent browsers. SVG specifications include basic shapes such as circle, rectangles, etc., as well as supporting clipping, masking and composition, filter effects and much more. Attempting to write a python-ic module supporting all possible SVG primitives and options via code like

test_circle = Circle(x=10, y=10, r=5, color='red')

can be a daunting task. Furthermore, documenting such a module would result in a lot of duplication with the official specification document. Fortunately, there is a simpler way than simply attempting to write a complete SVG module using Class-based definitions such as the one written above. The idea is to use instead an API similar to that of ElementTree (see also) - albeit much simplified.

Suppose that we would want to be able to create SVG circles, such as

<circle cx="600" cy="200" r="100" fill="red" stroke="blue" width="10"/>

and rectangles, such as

<rect x="1" y="1" height="398" fill="none" stroke="blue" width="1198"/>

using Python code. A simple way to achieve this would be to define the following class:

class XmlElement(object):
    '''First prototype from which all the xml elements are derived.

       By design, this enables all elements to automatically give a
       text representation of themselves - it is not quite complete.'''

    def __init__(self, tag, **attributes):
        '''A basic definition that will be replaced by the specific
           one required by any element.'''
self.tag = tag
if attributes is not None:
   self.attributes = attributes
else:
   self.attributes = {}

    def __repr__(self):
        '''This normal python method used to give a string representation
        for an object is used to automatically create the appropriate
        syntax representing an xml object.'''
attrib = ["  <%s" % self.tag] # open tag
for att in self.attributes:
   attrib.append(' %s="%s"' % (att, self.attributes[att]))
attrib.append("/>\n")
return ''.join(attrib)

Using this class, we can create a circle instance corresponding to the definition written previously as

circle = XmlElement("circle", cx=600, cy=200, r=100, fill="red",
            stroke="blue", width=10)

This is not quite as simple as the very first Circle() class-based example we wrote but it has the advantage of supporting all possible SVG attributes.

While the above XmlElement class definition is adequate for most basic SVG elements, it does not support such features as 1. text, 2. namespace (e.g. svg: prefix) and 3. grouping and sub-elements. All three additional features can be taken care of by the following modified class definition:

class XmlElement(object):
    '''Prototype from which all the xml elements are derived.

       By design, this enables all elements to automatically give a
       text representation of themselves.'''

    def __init__(self, tag, **attributes):
        '''A basic definition that will be replaced by the specific
           one required by any element.'''
 self.tag = tag
 self.prefix = ""
 self.sub_elements = []
 if attributes is not None:
     self.attributes = attributes
 else:
     self.attributes = {}

    def __repr__(self):
        '''This normal python method used to give a string representation
        for an object is used to automatically create the appropriate
        syntax representing an xml object.'''
 attrib = ["  <%s%s"%(self.prefix, self.tag)] # open tag
 for att in self.attributes:
     if att != 'text':
         attrib.append(' %s="%s"' % (att, self.attributes[att]))
 if 'text' in self.attributes:
     attrib.append(">%s\n" % (self.attributes['text'],
                                             self.prefix, self.tag))
 elif self.sub_elements:
     attrib.append(">\n")
     for elem in self.sub_elements:
         attrib.append("  %s" % elem)
     attrib.append("\n" % (self.prefix, self.tag))
 else:
     attrib.append("/>\n")
 return ''.join(attrib)

    def append(self, other):
        '''append other to self to create list of lists of elements'''''
 self.sub_elements.append(other)

That's almost it! With the exception of comments and Document Type Definition (dtd), we can use the above to create simple xhtml document containing ANY svg graphics without having to worry about xhtml syntax, opening and closing brackets, etc. However, we can possibly do even a little better. Consider the following xhtml document with embedded svg graphics:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink">
<head>
<title>This is the title.</title>
</head>
<body>
<p>This is the body.</p>
<svg:svg width="0" height="0">
<svg:defs>
<svg:circle cy="0" cx="0" r="20" id="red_circle" fill="red"/>
</svg:defs>
</svg:svg>
<svg:svg width="200" height="200">
<svg:use xlink:href="#red_circle" transform="translate(100, 100)"/>
</svg:svg>
<!-- This is a comment. -->
</body>
</html>

With just a few additional definitions, we can create this document using only Python code as follows:

doc = XmlDocument()
doc.head.append(XmlElement("title", text="This is the title."))

# A good practice is to define svg objects, and insert them
# using the definition; this is overkill for this example, but it
# provides a test of the class.
test_def = SvgDefs()
test_def.append(SvgElement("circle", cx=0, cy=0, r=20, fill="red",
                  id="red_circle"))

doc.body.append(XmlElement("p", text="This is the body."))
doc.body.append(test_def)

# we now create an svg object, that will make use of the definition above.
svg_window = SvgElement("svg", width="200", height="200")
use_circle = SvgElement("use", transform="translate(100, 100)")

# xlink:href can't be used as an attribute name passed to __init__
# this is why we use this two-step process.
use_circle.attributes["xlink:href"] = "#red_circle"

svg_window.append(use_circle)
doc.body.append(svg_window)
doc.body.append(Comment("This is a comment.")) # just for fun.

print doc

The additional definitions are as follow:

class XmlDocument(XmlElement):
    def __init__(self):
   self._begin = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
    xmlns:svg="http://www.w3.org/2000/svg"
    xmlns:xlink="http://www.w3.org/1999/xlink">\n"""
   self._end = "</html>"
   self.head = XmlElement("head")
   self.body = XmlElement("body")

    def append(self):
        '''Directly appending is not allowed'''
   assert False, "Append to either head or body."

    def __repr__(self):
        '''Gives an appropriate representation of an xml document.'''
   return self._begin + str(self.head) + str(self.body) + self._end


class SvgElement(XmlElement):
    '''Prototype from which all the svg elements are derived.

       By design, this enables all elements to automatically give an
       appropriate text representation of themselves.'''
    def __init__(self, tag, **attributes):
   XmlElement.__init__(self, tag, **attributes)
   self.prefix = "svg:"

class SvgDefs(SvgElement):
    '''Short-cut to create svg defs.  A user creates an instance of this
    object and simply appends other svg Elements'''
    def __init__(self):
   self.defs = SvgElement("defs")
   self.root = SvgElement("svg", width=0, height=0)
   self.root.append(self.defs)

    def append(self, other):
        '''appends other to defs sub-element, instead of root element'''
   self.defs.append(other)

    def __repr__(self):
        '''gives a string representation of an object, appropriate for
           insertion in an html document'''
   return str(self.root)

class Comment(object):
    '''Comment that can be inserted in code xml documents'''
def __init__(self, text):
   self.text = text
def __repr__(self):
   return "<!-- " + self.text + " -->\n"

That's it for real this time! Fewer than 100 lines of code that you can use if you need to programmatically create (x)html documents containing svg images. There are a few limitations (elements containing text may not be chained...) but it works for me. If you want to try it yourself, you can find the module here.

Plugins - part 4: Crunchy-style plugin

In this 4th post in the plugins series, I will explain the approach we used in Crunchy. While explaining the main features, I will also compare with the simple class-base plugin framework introduced in the third post in this series.

Crunchy's approach does not require a plugin to be class-based. In fact, most plugins used in Crunchy only make use of simple functions inside modules. While the class-based framework introduced in the third post used the fact that Python allowed automatic discovery of subclasses, the approach used in Crunchy requires an explicit registration of plugins. Using the same example as before, this means that op_2.py would contain the following code:

def register(OPERATORS):
    OPERATORS['**'] = operator_pow_token

class operator_pow_token(object):
 lbp = 30
 def led(self, left):
     return left ** expression(30-1)

Note that the class (operator_pow_token) is unchanged from the original application.

The method used to find plugins is similar to that introduced previously. The entire code required is as follows:

def init_plugins(expression):
  plugin_dir = (os.path.dirname(os.path.realpath(__file__)))
  plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
  sys.path.insert(0, plugin_dir)
  for plugin in plugin_files:
      mod = __import__(plugin)
       if hasattr(mod, "register"):
          mod.expression = expression
           mod.register(OPERATORS)

By comparison, the code used in the class-based plugin could have been written as:

def init_plugins(expression):
   plugin_dir = os.path.dirname(os.path.realpath(__file__))
   plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
   sys.path.insert(0, plugin_dir)
   for plugin in plugin_files:
       mod = __import__(plugin)
       mod.expression = expression
    for plugin in Plugin.__subclasses__():
        OPERATORS[plugin.symbol] = plugin

So, in one case (Crunchy-style), we have an explicit registration process with no need to create a sample base class (and, in fact, no need to work with classes at all), while in the other we have an automatic registration based on identifying subclasses. This does not mean that the Crunchy-style is better - just different. Both are equally good for this type of simple application. While we have not found the approach used in Crunchy to be limiting us in any way when extending Crunchy, something must be said for the fact that all the other Python examples of plugin-based application I have found have been based on using classes.

I can now give another motivation for having chosen the small expression calculator as a candidate for a plugin-based application: since all mathematical operations were already implemented as classes, it was readily suitable for the class-based approach (and the Zope Component Architecture one, etc.) whereas all my existing code samples that used plugins (from Crunchy, docpicture, etc.) had mostly functions rather than classes in plugins.

Plugins - part 3: Simple class-based plugin

In the first post of this series, I introduced a simple application to be used as a demonstration of a plugin-based application. The chosen application was an expression calculator contained in a single file. In the second post, I modularized the original file so that the new file structure would become a good representative of a plugin based application. In this post, I will explain how to make use of a simple class-base plugin framework. The model I have chosen follows fairly closely the tutorial written by Armin Ronacher. Another tutorial demonstrating a simple class-based plugin framework has been written by Marty Alchin.

The first step is to define a base Plugin class. All we need is to include the following in base.py:

class Plugin(object):
   pass

Next, we ensure that classes used in plugins derive from this base class. We only give one explicit example, that of the class included in op_2.py since the 4 classes included in op_1.py would be treated in exactly the same way.

from plugins.base import Plugin

class operator_pow_token(Plugin):
   symbol = '**'
  lbp = 30
  def led(self, left):
      return left ** expression(30-1)

Note that we added one more line of code to the class definition. We are now ready to deal with the plugin discovery and registration.

Rather than hard-coding the information about which plugin files to import as we did when we simply modularize the application, we give a way for our program to automatically find plugins. With the file structure that we have created, this can be accomplished as follows:

def find_plugins(expression):
   '''find all files in the plugin directory and imports them'''
   plugin_dir = os.path.dirname(os.path.realpath(__file__))
   plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
   sys.path.insert(0, plugin_dir)
   for plugin in plugin_files:
       mod = __import__(plugin)
       mod.expression = expression

Note that the last line of code is included because of the "wart" mentioned in the previous post and would not usually be included. To be safe, we should probably have ensured that expression was not already defined in the modules to be imported since, in theory, Python files other than plugins (such as __init__.py) might be present in the plugin directory. In this tutorial series we will often ignore the need to insert try/except clauses to simplify the code.

While we have imported the modules containing the plugins, they are not yet known in a useful form by the main application. To do so is very simple in this class-based approach, thanks to Python's treatment of (sub-)classes. Here's the code to do this:

def register_plugins():
 '''Register all class based plugins.

    Uses the fact that a class knows about all of its subclasses
    to automatically initialize the relevant plugins
 '''
 for plugin in Plugin.__subclasses__():
     OPERATORS[plugin.symbol] = plugin

That's it! It is hard to imagine anything simpler. With this last definition, the entire base.py module can be written as:

import os
import sys

OPERATORS = {}

class Plugin(object):
  pass

def init_plugins(expression):
  '''simple plugin initializer
  '''
  find_plugins(expression)
  register_plugins()

def find_plugins(expression):
  '''find all files in the plugin directory and imports them'''
  plugin_dir = os.path.dirname(os.path.realpath(__file__))
  plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
  sys.path.insert(0, plugin_dir)
  for plugin in plugin_files:
      mod = __import__(plugin)
      mod.expression = expression

def register_plugins():
  '''Register all class based plugins.

     Uses the fact that a class knows about all of its subclasses
     to automatically initialize the relevant plugins
  '''
  for plugin in Plugin.__subclasses__():
      OPERATORS[plugin.symbol] = plugin

In the next post, I will show another simple alternative approach similar to the one used in Crunchy.

Plugins - part 2: modularization

In the first post on the Plugins series, I introduced the small application used to demonstrate how one could modularize applications using a plugin architecture. The digital ink was barely dry on that post that already two people rose to the challenge and presented their solution, one using the standard method with the Zope Component Architecture, the other a modified method using grok. I will comment on these two solutions later in this series.

With apologies to the more advanced users, I have decided to proceed fairly slowly and cover many simple concepts with this series of plugins. Thus, this second post will not yet discuss plugins, but simply lay the groundwork for future posts. By the way, for those interested, and as pointed out by Lennart Regebro in his post, all the code samples that I will use can be browsed at, or retrieved from, my py-fun google code repository.

As a first step before comparing different approaches to dealing with plugins, I will take the sample application introduced in the first post and modularize it.

The core application (calculator.py) is as follows:

import re

from plugins.base import OPERATORS, init_plugins

class literal_token(object):
  def __init__(self, value):
      self.value = value
  def nud(self):
      return self.value

class end_token(object):
  lbp = 0

def tokenize(program):
  for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
      if number:
          yield literal_token(int(number))
        elif operator in OPERATORS:
            yield OPERATORS[operator]()
      else:
          raise SyntaxError("unknown operator: %r" % operator)
  yield end_token()

def expression(rbp=0):
  global token
  t = token
  token = next()
  left = t.nud()
  while rbp < token.lbp:
      t = token
      token = next()
      left = t.led(left)
  return left

def calculate(program):
  global token, next
  next = tokenize(program).next
  token = next()
  return expression()

if __name__ == "__main__":
  init_plugins(expression)
  assert calculate("+1") == 1
  assert calculate("-1") == -1
  assert calculate("10") == 10
  assert calculate("1+2") == 3
  assert calculate("1+2+3") == 6
  assert calculate("1+2-3") == 0
  assert calculate("1+2*3") == 7
  assert calculate("1*2+3") == 5
  assert calculate("6*2/3") == 4
  assert calculate("2**3") == 8
  assert calculate("2*2**3") == 16
  print "Done!"

For the next few posts, when I demonstrate some very simple plugin approaches, this core application will remain untouched. This is one important characteristic of plugin-based application: in a well-designed application, plugin writers should not have to modify a single line of the core modules to ensure that their plugins can be used.

Communication between plugins and the core application is ensured via an Application Programming Interface (API) unique to that application. In our example, the API is a simple Python dict (OPERATORS) written in capital letters only to make it stand out.

In a sub-directory (plugins), in addition to an empty __init__.py file, we include the following three files:

1. base.py

OPERATORS = {}

def init_plugins(expression):
    '''simulated plugin initializer'''
    from plugins import op_1, op_2

    op_1.expression = expression
    op_2.expression = expression

    OPERATORS['+'] = op_1.operator_add_token
    OPERATORS['-'] = op_1.operator_sub_token
    OPERATORS['*'] = op_1.operator_mul_token
    OPERATORS['/'] = op_1.operator_div_token
    OPERATORS['**'] = op_2.operator_pow_token

2. op_1.py

class operator_add_token(object):
   lbp = 10
   def nud(self):
       return expression(100)
   def led(self, left):
       return left + expression(10)

class operator_sub_token(object):
   lbp = 10
   def nud(self):
       return -expression(100)
   def led(self, left):
       return left - expression(10)

class operator_mul_token(object):
   lbp = 20
   def led(self, left):
       return left * expression(20)

class operator_div_token(object):
   lbp = 20
   def led(self, left):
       return left / expression(20)

and 3. op_2.py

class operator_pow_token(object):
   lbp = 30
   def led(self, left):
       return left ** expression(30-1)

The last two files have been simply extracted with no modification from the original application. Instead of having 2 such files containing classes of the form operator_xxx_token, I could have included them all in one file, or split into 5 different files. The number of files is irrelevant here: they are only introduced to play the role of plugins in this application.

The file base.py plays the role here of a plugin initialization module: it ensures that plugins are properly registered and made available to the core program.

Since I wanted to change the original code as little as possible, a "wart" is present in the code as written since it was never intended to be a plugin-based application: the function expression() was accessible to all objects in the initial single-file application. It is now needed in a number of modules. The file base.py takes care of ensuring that "plugin" modules have access to that function in a transparent way. This will need to be changed when using some standard plugin frameworks, as was done in the zca example or the grok one.

In the next post, I will show how to take this now modularized application and transform it into a proper plugin-based one.

Plugins - part 1: the application

My interest in plugins started two years ago listening to Ivan Krstić talk about the OLPC. Following his talk, I wrote the following on edu-sig:

One open issue (as I understand it) is that of finding the "best practice" for plugins. The idea is that the core programs should be as small as possible but easy to extend via plugins. I thought that there already was a "well known and best way" to design plugins - and it was on my list of things to learn about (to eventually incorporate rur-ple within crunchy).

After discussing this off-list with Johannes Woolard, I concluded that we should try to redesign Crunchy to make use of plugins. While I was thinking about how we might proceed to do this, Johannes went ahead and implemented a simple plugin framework which we eventually adopted for Crunchy.

While there are a few agreed-upon "standards" when it comes to dealing with plugins in Python (such as setuptools and Zope Component Architecture), I tend to agree with Ivan Krstić's observation that there are no "best practice" for plugins - at least, none that I have seen documented. As what might be considered to be a first step in determining the "best practice" for writing plugin-based applications with Python, I will take a sample application, small enough so that it can be completely included and described in a blog post, and not written with plugins in mind. I thought it would be a more representative example to use an arbitrary sample application, rather than trying to come up with one specifically written for the purpose of this series of post.

The application I have chosen is a small modification of an expression calculator written and described by Fredrik Lundh, aka effbot, a truly outstanding pythonista. The entire code is as follows:

""" A simple expression calculator entirely contained in a single file.

See http://effbot.org/zone/simple-top-down-parsing.htm for detailed explanations
as to how it works.

This is the basic application used to demonstrate various plugin frameworks.
"""

import re

class literal_token(object):
   def __init__(self, value):
       self.value = value
   def nud(self):
       return self.value

class operator_add_token(object):
    lbp = 10
    def nud(self):
        return expression(100)
    def led(self, left):
        return left + expression(10)

class operator_sub_token(object):
    lbp = 10
    def nud(self):
        return -expression(100)
    def led(self, left):
        return left - expression(10)

class operator_mul_token(object):
    lbp = 20
    def led(self, left):
        return left * expression(20)

class operator_div_token(object):
    lbp = 20
    def led(self, left):
        return left / expression(20)

class operator_pow_token(object):
    lbp = 30
    def led(self, left):
        return left ** expression(30-1)

class end_token(object):
   lbp = 0

def tokenize(program):
   for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
       if number:
           yield literal_token(int(number))
        elif operator == "+":
            yield operator_add_token()
        elif operator == "-":
            yield operator_sub_token()
        elif operator == "*":
            yield operator_mul_token()
        elif operator == "/":
            yield operator_div_token()
        elif operator == "**":
            yield operator_pow_token()
        else:
           raise SyntaxError("unknown operator: %r" % operator)
   yield end_token()

def expression(rbp=0):  # note that expression is a global object in this module
   global token
   t = token
   token = next()
   left = t.nud()
   while rbp < token.lbp:
       t = token
       token = next()
       left = t.led(left)
   return left

def calculate(program):
   global token, next
   next = tokenize(program).next
   token = next()
   return expression()

if __name__ == "__main__":
   assert calculate("+1") == 1
   assert calculate("-1") == -1
   assert calculate("10") == 10
   assert calculate("1+2") == 3
   assert calculate("1+2+3") == 6
   assert calculate("1+2-3") == 0
   assert calculate("1+2*3") == 7
   assert calculate("1*2+3") == 5
   assert calculate("6*2/3") == 4
   assert calculate("2**3") == 8
   assert calculate("2*2**3") == 16
   print "Done!"

The latest version used can be found online.

In the above code, I have highlighted in red classes that will be transformed into plugins. I have also highlighted in green hard-coded if/elif choices that will become indirect references to the plugin components.

In the next post in this series, I will break up this single file in a set of different modules as a required preliminary step before transforming the whole applications into a plugin-based one, with a small core. In subsequent posts, I will keep the core constant and compare various approaches that one can use to link the plugins with the core.

Seeing double at Pycon 2009

Jesse Noller is going to give two talks at Pycon 2009. So is Tarek Ziadé. And Mike Fletcher is as well. And Brett Cannon has a talk and a panel. So far there I have not seen any post on Planet Python about someone giving just one talk.

I would hate to be the one breaking the streak. So, I might as well announce that I will be giving two talks as well. :-)

Not surprisingly, the first one is about Crunchy. The title of the talk is Learning and Teaching Python Programming: The Crunchy Way, and the abstract reads as follows:

Crunchy (http://code.google.com/p/crunchy) is a program that transforms a static Python tutorial into an interactive session within a browser. In this talk, I will present Crunchy, focusing on the features that are specifically designed to be helpful in a formal teaching setting.

Not exactly Earth-shattering but hopefully of interest to anyone that has to teach programming in a formal setting or who would just be interested in showing off Python to anyone. This Crunchy talk is, of course, not going to be your traditional slide-based talk but rather more like an interactive demo using Crunchy. I am hoping to have a few surprises by the time the conference occurs.

My other talk is going to be very different. I doubt very much that I will be using Crunchy for it. The title is Plugins and Monkeypatching: increasing flexibility, dealing with inflexibility, and the abstract reads as follows:

By using plugins, one can create software that is easily extensible by others, thereby promoting collaborative development. The flip side of extensible software occurs when dealing with some standard framework whose interface is closed but which does not do exactly what is desired. In this case, monkeypatching may be worth considering.

In this talk, I'll give concrete examples of both plugin design and using monkeypatching, using small code samples from existing projects, and discuss the advantages and the shortcomings of the methods used. I will also include the design of a tiny, but flexible module for generating svg code - and compare it with other existing approaches.

I can not pretend to even come close to being an expert about designing plugin based applications. Still, I felt that I have had some potentially useful experiences to share about these topics which motivated my talk proposal. Now that it has been accepted, I have started working on fleshing out the original outline.

In preparation for the actual talk, which will not go into much code details due to time constraints, I plan to start a short series of posts about plugins. In the first post I will give an overview of a simple application (a calculator) that is written as a single file. In the second post, I will reorganize the code so as to use multiple files, with a number of modules located in a "plugins" directory, laying out the groundwork for working with actual plugins. Subsequent posts will be used to demonstrate different approaches used to transform the application into a truly plugin-based one.

Of course, the plugin model used in Crunchy will be one approach showcased. A second one (which I have already implemented) is a simple class based one inspired by a tutorial written by Armin Romacher. I also plan to demonstrate how to use the Zope component architecture approach as well as the setuptools based method (and possibly others depending on suggestions I might receive).

Since I have never actually written any code using the Zope component architecture or the setuptools based approach, I thought it would be interesting to do this in a truly open-source spirit. Therefore, once I have written the first two or three posts in this series, I would like to invite anyone interested to contribute their own code demonstrating their favourite framework. This way, experts could make sure that their favourite framework is properly showcased, and not misrepresented by me. Interested parties can contribute either by sending me the code directly or by blogging about it. (If your blog appears on either planet.python.org or planetpython.org, I will most likely read it.)

Anyone who contributes in this way to my talk will be mentioned at Pycon AND receive half of the stipend I get as a presenter. ;-)