Plugins - part 2: modularization

In the first post on the Plugins series, I introduced the small application used to demonstrate how one could modularize applications using a plugin architecture. The digital ink was barely dry on that post that already two people rose to the challenge and presented their solution, one using the standard method with the Zope Component Architecture, the other a modified method using grok. I will comment on these two solutions later in this series.

With apologies to the more advanced users, I have decided to proceed fairly slowly and cover many simple concepts with this series of plugins. Thus, this second post will not yet discuss plugins, but simply lay the groundwork for future posts. By the way, for those interested, and as pointed out by Lennart Regebro in his post, all the code samples that I will use can be browsed at, or retrieved from, my py-fun google code repository.

As a first step before comparing different approaches to dealing with plugins, I will take the sample application introduced in the first post and modularize it.

The core application (calculator.py) is as follows:

import re

from plugins.base import OPERATORS, init_plugins

class literal_token(object):
  def __init__(self, value):
      self.value = value
  def nud(self):
      return self.value

class end_token(object):
  lbp = 0

def tokenize(program):
  for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
      if number:
          yield literal_token(int(number))
        elif operator in OPERATORS:
            yield OPERATORS[operator]()
      else:
          raise SyntaxError("unknown operator: %r" % operator)
  yield end_token()

def expression(rbp=0):
  global token
  t = token
  token = next()
  left = t.nud()
  while rbp < token.lbp:
      t = token
      token = next()
      left = t.led(left)
  return left

def calculate(program):
  global token, next
  next = tokenize(program).next
  token = next()
  return expression()

if __name__ == "__main__":
  init_plugins(expression)
  assert calculate("+1") == 1
  assert calculate("-1") == -1
  assert calculate("10") == 10
  assert calculate("1+2") == 3
  assert calculate("1+2+3") == 6
  assert calculate("1+2-3") == 0
  assert calculate("1+2*3") == 7
  assert calculate("1*2+3") == 5
  assert calculate("6*2/3") == 4
  assert calculate("2**3") == 8
  assert calculate("2*2**3") == 16
  print "Done!"

For the next few posts, when I demonstrate some very simple plugin approaches, this core application will remain untouched. This is one important characteristic of plugin-based application: in a well-designed application, plugin writers should not have to modify a single line of the core modules to ensure that their plugins can be used.

Communication between plugins and the core application is ensured via an Application Programming Interface (API) unique to that application. In our example, the API is a simple Python dict (OPERATORS) written in capital letters only to make it stand out.

In a sub-directory (plugins), in addition to an empty __init__.py file, we include the following three files:

1. base.py

OPERATORS = {}

def init_plugins(expression):
    '''simulated plugin initializer'''
    from plugins import op_1, op_2

    op_1.expression = expression
    op_2.expression = expression

    OPERATORS['+'] = op_1.operator_add_token
    OPERATORS['-'] = op_1.operator_sub_token
    OPERATORS['*'] = op_1.operator_mul_token
    OPERATORS['/'] = op_1.operator_div_token
    OPERATORS['**'] = op_2.operator_pow_token

2. op_1.py

class operator_add_token(object):
   lbp = 10
   def nud(self):
       return expression(100)
   def led(self, left):
       return left + expression(10)

class operator_sub_token(object):
   lbp = 10
   def nud(self):
       return -expression(100)
   def led(self, left):
       return left - expression(10)

class operator_mul_token(object):
   lbp = 20
   def led(self, left):
       return left * expression(20)

class operator_div_token(object):
   lbp = 20
   def led(self, left):
       return left / expression(20)

and 3. op_2.py

class operator_pow_token(object):
   lbp = 30
   def led(self, left):
       return left ** expression(30-1)

The last two files have been simply extracted with no modification from the original application. Instead of having 2 such files containing classes of the form operator_xxx_token, I could have included them all in one file, or split into 5 different files. The number of files is irrelevant here: they are only introduced to play the role of plugins in this application.

The file base.py plays the role here of a plugin initialization module: it ensures that plugins are properly registered and made available to the core program.

Since I wanted to change the original code as little as possible, a "wart" is present in the code as written since it was never intended to be a plugin-based application: the function expression() was accessible to all objects in the initial single-file application. It is now needed in a number of modules. The file base.py takes care of ensuring that "plugin" modules have access to that function in a transparent way. This will need to be changed when using some standard plugin frameworks, as was done in the zca example or the grok one.

In the next post, I will show how to take this now modularized application and transform it into a proper plugin-based one.