Plugins - part 1: the application

My interest in plugins started two years ago listening to Ivan Krstić talk about the OLPC. Following his talk, I wrote the following on edu-sig:

One open issue (as I understand it) is that of finding the "best practice" for plugins. The idea is that the core programs should be as small as possible but easy to extend via plugins. I thought that there already was a "well known and best way" to design plugins - and it was on my list of things to learn about (to eventually incorporate rur-ple within crunchy).

After discussing this off-list with Johannes Woolard, I concluded that we should try to redesign Crunchy to make use of plugins. While I was thinking about how we might proceed to do this, Johannes went ahead and implemented a simple plugin framework which we eventually adopted for Crunchy.

While there are a few agreed-upon "standards" when it comes to dealing with plugins in Python (such as setuptools and Zope Component Architecture), I tend to agree with Ivan Krstić's observation that there are no "best practice" for plugins - at least, none that I have seen documented. As what might be considered to be a first step in determining the "best practice" for writing plugin-based applications with Python, I will take a sample application, small enough so that it can be completely included and described in a blog post, and not written with plugins in mind. I thought it would be a more representative example to use an arbitrary sample application, rather than trying to come up with one specifically written for the purpose of this series of post.

The application I have chosen is a small modification of an expression calculator written and described by Fredrik Lundh, aka effbot, a truly outstanding pythonista. The entire code is as follows:

""" A simple expression calculator entirely contained in a single file.

See http://effbot.org/zone/simple-top-down-parsing.htm for detailed explanations
as to how it works.

This is the basic application used to demonstrate various plugin frameworks.
"""

import re

class literal_token(object):
   def __init__(self, value):
       self.value = value
   def nud(self):
       return self.value

class operator_add_token(object):
    lbp = 10
    def nud(self):
        return expression(100)
    def led(self, left):
        return left + expression(10)

class operator_sub_token(object):
    lbp = 10
    def nud(self):
        return -expression(100)
    def led(self, left):
        return left - expression(10)

class operator_mul_token(object):
    lbp = 20
    def led(self, left):
        return left * expression(20)

class operator_div_token(object):
    lbp = 20
    def led(self, left):
        return left / expression(20)

class operator_pow_token(object):
    lbp = 30
    def led(self, left):
        return left ** expression(30-1)

class end_token(object):
   lbp = 0

def tokenize(program):
   for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
       if number:
           yield literal_token(int(number))
        elif operator == "+":
            yield operator_add_token()
        elif operator == "-":
            yield operator_sub_token()
        elif operator == "*":
            yield operator_mul_token()
        elif operator == "/":
            yield operator_div_token()
        elif operator == "**":
            yield operator_pow_token()
        else:
           raise SyntaxError("unknown operator: %r" % operator)
   yield end_token()

def expression(rbp=0):  # note that expression is a global object in this module
   global token
   t = token
   token = next()
   left = t.nud()
   while rbp < token.lbp:
       t = token
       token = next()
       left = t.led(left)
   return left

def calculate(program):
   global token, next
   next = tokenize(program).next
   token = next()
   return expression()

if __name__ == "__main__":
   assert calculate("+1") == 1
   assert calculate("-1") == -1
   assert calculate("10") == 10
   assert calculate("1+2") == 3
   assert calculate("1+2+3") == 6
   assert calculate("1+2-3") == 0
   assert calculate("1+2*3") == 7
   assert calculate("1*2+3") == 5
   assert calculate("6*2/3") == 4
   assert calculate("2**3") == 8
   assert calculate("2*2**3") == 16
   print "Done!"

The latest version used can be found online.

In the above code, I have highlighted in red classes that will be transformed into plugins. I have also highlighted in green hard-coded if/elif choices that will become indirect references to the plugin components.

In the next post in this series, I will break up this single file in a set of different modules as a required preliminary step before transforming the whole applications into a plugin-based one, with a small core. In subsequent posts, I will keep the core constant and compare various approaches that one can use to link the plugins with the core.