New plugin for Crunchy ... and bug fixes

Crunchy has a new plugin: getsource. What it does is enable a tutorial writer to embed a "link" to a python module inside an html file, or a class within that module, or a function or method, and have the source code being extracted by the inspect module and inserted within the html page.

Crunchy being Crunchy, it can also embed an interpreter or an editor right below the code source so that a user can interact with it.

And since not everyone likes to write documentation using straight html, a custom docutils directive is supported so that it works from reStructuredText files too.

The docutils directive looks as follows:

..getsource:: relative/path/to/module[.function] [linenumber] [editor or interpreter]

with a similar syntax for html files.

This plugin (which still has some minor bugs) is included in release 1.0.1 of Crunchy, which contains other minor bug fixes (as compared with release 1.0).

Crunchy 1.0 released!!

Crunchy 1.0 has finally been released. :-)

Crunchy 1.0 is compatible with Python 2.4, 2.5, 2.6 ... and 3.1. It is also compatible with Jython 2.5 modulo some bugs when trying to work with examples containing unicode strings.

Crunchy, for those that are not familiar with it, is a one-of-a-kind application. It is an application designed to transforms otherwise static html Python tutorials into interactive session viewed within a browser. Currently Crunchy has only been fully tested with Firefox. Crunchy should work with all operating systems - it has been tested fairly extensively on Linux, Windows and Mac OS.

In more details, here's what Crunchy does:

1. It loads an existing html file (or reStructuredText file) containing some Python code; this file can reside locally, or anywhere on the web.
2. It removes any existing javascript code & embedded interactive elements (applets, flash apps, etc.).
3. It further transforms the file by inserting some extra elements, including some custom javascript code.
4. It sends the transformed file to your browser of choice (read: Firefox).
5. It establishes a communication between the browser and the Crunchy back end and wait for user interaction.

Crunchy can embed Python interpreters and code editors (and more!) within an html page, enabling a user to enter (or edit) some Python code inside a browser window, send it to the Crunchy back end for execution, and observe the result back in the browser window.

In a sense, Crunchy is a Python programming environment embedded within a browser. However, it does not require a custom format for interactivity: for html files, as long as the Python code is included in a pre-formatted element (<pre>), Crunchy will recognize it, style it, and include the appropriate interactive element.

Crunchy comes with a fairly complete tutorial and supporting documentation. It is highly configurable.

Release 1.0 is NOT the end of the road for Crunchy. Future plans include support for interactivity with languages other than Python.

Python textbooks wanted

Python textbooks wanted(1). Actually, what I'm mostly interested in are links to textbooks and other resources that would be useful to educators either teaching Python as a programming language or other courses where Python feature prominently. The reason behind this request is that I volunteered to be responsible to take care of the edu-sig page on the Python site. Through no one's fault in particular, the old page (2) had gone stale and only one textbook was listed. We're now up to six and I am sure I have not included them all. By ensuring that up to date and complete information is available on that page, we can facilitate the adoption of Python as a language taught in High Schools, Colleges and Universities.

So, if you know of any useful information that should be added to the edu-sig page, please let me know.

(1) Actually, this is not exactly correct ... but if you want to send me some for reviews, I won't say no. ;-)

(2) The page linked here will eventually disappear...

Learning paths at ShowMeDo

ShowMeDo is a great site to learn about Python (and a few other subjects) by watching screncasts. One of my favourite videos is Learn Django: Create a Wiki in 20 minutes, which taught me the basic concepts so that I was later able to adapt and play around with Google App Engine. Many authors on ShowMeDo have multiple screencasts (I made a few on RUR-PLE and Crunchy a while ago), some of which (not mine) are very professional looking. Such series of videos give the viewpoint of a given author, introducing some concepts in a logical sequence.

The newest addition to the ShowMeDo site is called Learning Paths. The idea of Learning Paths is to improve upon the existing series by including videos from multiple authors into a coherent sequence. While this concept is very much in its infancy, it promises to become a great addition to what is already a fantastic site.

In case anyone were wondering: I have no financial interest whatsover in ShowMeDo; I am just a satisfied user who considers this site a very good resource, well worth exploring. If only I had more time...

Testing code highlighting

Update:

In order to have the styling work, I had to include by hand the styling information (keyword and string) from this style sheet - and adapt as desired.

- - - - - - -
This is a test of Python code highlighting on Blogger following these instructions.

print "Hello World!"

# And this is a comment

End of test.

Plugins - part 6: setuptools based approach

setuptools is collection of enhancements to the standard Python distutils module. While it is not included in the Python standard library, chances are that it is already installed on your system if you have installed some additional Python libraries since it is so widely used to install Python packages ("eggs") via easy_install.

setuptools is also used in many applications as a handler for plugins. Many such applications include tutorials for creating new plugins using setuptools. For a somewhat general introduction to using setuptools to create plugin based applications, I suggest you have a look at the two tutorials from which this one is inspired.

Our starting point for this tutorial is essentially the same as in the third one in this series. We start with the following files:

root_directory/
    calculator.py
    setup.py
    plugins/
        __init__.py
        base.py
        op_1.py
        op_2.py

where we have one new file, setup.py. This file is a special file for setuptools. Its content is as follows:

''' run with python setup.py develop '''

from setuptools import setup, find_packages

setup(
    name="Calculator_s_tools",
    version="1.0",
    packages=['plugins'],
    entry_points="""
        [plugin_tutorial.s_tools]
        add = plugins.op_1:operator_add_token
        sub = plugins.op_1:operator_sub_token
        mul = plugins.op_1:operator_mul_token
        div = plugins.op_1:operator_div_token
        pow = plugins.op_2:operator_pow_token"""
       )

The key concept for setuptools is that of "entry_points". We define some entry points, with a name "plugin_tutorial.s_tools" chosen as unique to our application. Within this entrypoint we indicate which classes should be imported. This method effectively replace our custom method for finding and loading plugins. However, if you remember from previous tutorials, the way the application was designed originally (all within a single file) resulted in a "wart", where we had to create a link to a function ("expression") in each plugin module. Since setuptools will import the classes for us, we have no way to tell it how to fix that "wart" - we have to find another way. The method we chose was to create a different Plugin base class, one that implements the Borg idiom, so that each instance shares common attributes. Here's the new "base.py":

import os
import sys
import pkg_resources  # setuptools specific

OPERATORS = {}
ENTRYPOINT = 'plugin_tutorial.s_tools'  # same name as in setup.py

class Plugin(object):
    '''A Borg class.'''
    _shared_states = {}
    def __init__(self):
         self.__dict__ = self._shared_states

def init_plugins(expression):
    '''simple plugin initializer'''
    Plugin().expression = expression  # fixing the wart
    load_plugins()

def load_plugins():
    '''setuptools based plugin loader'''
    for entrypoint in pkg_resources.iter_entry_points(ENTRYPOINT):
        plugin_class = entrypoint.load()
        OPERATORS[plugin_class.symbol] = plugin_class

The actual plugin files are slightly modified to derive from the new Plugin class; for example, op_2.py contains the following:

from plugins.base import Plugin

class operator_pow_token(Plugin):
    symbol = '**'
    lbp = 30
    def led(self, left):
        return left ** self.expression(30-1)

where "expression" is now a class variable obtained from Plugin.

The code involved to make the setuptools approach is approximately the same level of complexity as the class-based plugin system covered previously. The advantages of using the setuptools approach are as follows:

1. Since it is a widely used tool, many people know how to use it properly.
2. It is possible to package plugins as eggs to be uploaded to a repository.
3. It is possible to keep track of dependencies in a fairly detailed way (e.g. module X version Y required).
   

By comparison, it suffers from the following disadvantages:

1. Some information about plugin location (entry_points name) is duplicated, appearing in both setup.py and base.py (in our example).
2. Automatic plugin discovery without editing of a file (setup.py) is not possible, unlike the cases we covered before. Because of this, dynamic loading of "external" plugins while the application is already running may be problematic to achieve. (I am not familiar enough with setuptools to determine if it is feasible or not.) See the first comment par Phillip J. Eby on how to achieve this.
   
3. A preliminary step ("python setup.py develop") is required to generate entrypoints information.
4. A number of additional files are created by the previous step, "cluttering" slightly the file structure by adding an extra directory with a few files.
   

That being said, the differences between the two approaches are relatively minor when everything is taken into account. Choosing one approach over the other is a matter of individual taste - at least for simple applications such as the one we considered.

Plugins - part 5: Activation and Deactivation

(Note: the code indentation may appear to be wrong due to some blogspot's quirks...)

While plugins are a great way to extend the functionality of an application, sometimes it makes sense to limit the number of available features, based on a user's preference. For example, gedit, the official text editor of the Gnome environment, offers the possibility to activate or deactivate a plugin.

In this post, using the class-based plugin approach, I will explain how to add the possibility to activate or deactivate a given plugin. Furthermore, I will show how to use this capability to dynamically load new plugins.

Starting from the beginning...

Our starting point will be the following modified core application (calculator.py):

import re

from plugins.base import OPERATORS, init_plugins, activate, deactivate

class literal_token(object):
 def __init__(self, value):
     self.value = value
 def nud(self):
     return self.value

class end_token(object):
lbp = 0

def tokenize(program):
 for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
    if number:
        yield literal_token(int(number))
    elif operator in OPERATORS:
        yield OPERATORS[operator]()
    else:
        raise SyntaxError("unknown operator: %r" % operator)
 yield end_token()

def expression(rbp=0):
 global token
 t = token
 token = next()
 left = t.nud()
 while rbp < token.lbp:
     t = token
     token = next()
     left = t.led(left)
 return left

def calculate(program):
 global token, next
 next = tokenize(program).next
 token = next()
 return expression()

if __name__ == "__main__":
init_plugins(expression)
assert calculate("+1") == 1
assert calculate("-1") == -1
assert calculate("10") == 10
assert calculate("1+2") == 3
assert calculate("1+2+3") == 6
assert calculate("1+2-3") == 0
assert calculate("1+2*3") == 7
assert calculate("1*2+3") == 5
assert calculate("6*2/3") == 4

    # "**" has not been activated at the start in base.py
    try:
        assert calculate("2**3") == 8
    except SyntaxError:
        print "Correcting error..."
        activate("**")
    assert calculate("2*2**3") == 16

    deactivate('+')
    try:
        assert calculate("1+2") == 3
    except SyntaxError:
        activate('+')
    assert calculate("1+2") == 3

print "Done!"

The new features are indicated by different colours. In blue, we have two new functions imported to either activate or deactivate a given plugin. When the application is started, exponentiation is disabled - this can only be seen by looking at the modified version of base.py. When a disabled plugin is called, a SyntaxError already present in the old version) is raised and we activate the plugin.

To make this possible, we need to modify base.py. Before showing the new version, here's the result of running the above code:

Activating +
Activating -
Activating *
Activating /
Correcting error...
Activating **
Deactivating +
Activating +
Done!

And here's the new version of base.py:

import os
import sys

OPERATORS = {}

# We simulate a configuration file that would be based on a user's preference
# as to which plugin should be activated by default
# We will leave one symbol "**" out of the list as a test.
preferences = ['+', '-', '*', '/']

# We also keep track of all available plugins, activated or not
all_plugins = {}

class Plugin(object):
'''base class for all plugins'''

    def activate(self):
        '''activate a given plugin'''
        if self.symbol not in OPERATORS:
            print "Activating %s" % self.symbol
            OPERATORS[self.symbol] = self.__class__
        if self.symbol not in all_plugins:
            all_plugins[self.symbol] = self.__class__

    def deactivate(self):
        '''deactivate a given plugin'''
        print "Deactivating %s" % self.symbol
        if self.symbol in OPERATORS:
            del OPERATORS[self.symbol]

def activate(symbol):
    '''activate a given plugin based on its symbol'''
    if symbol in OPERATORS:
        return
    all_plugins[symbol]().activate()

def deactivate(symbol):
    '''deactivate a given plugin, based on its symbol'''
    if symbol not in OPERATORS:
        return
    all_plugins[symbol]().deactivate()

def init_plugins(expression):
'''simple plugin initializer
'''
find_plugins(expression)
register_plugins()

def find_plugins(expression):
'''find all files in the plugin directory and imports them'''
plugin_dir = os.path.dirname(os.path.realpath(__file__))
plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
sys.path.insert(0, plugin_dir)
for plugin in plugin_files:
 mod = __import__(plugin)
 mod.expression = expression

def register_plugins():
'''Register all class based plugins.

Uses the fact that a class knows about all of its subclasses
to automatically initialize the relevant plugins
'''
for plugin in Plugin.__subclasses__():
        # only register plugins according to user's preferences
        if plugin.symbol in preferences:
            plugin().activate()
        else:   # record its existence
            all_plugins[plugin.symbol] = plugin

Changes from the old version are indicated in blue (with corresponding comments in green). Note that we did not change a single line of code for the actual plugins! We did use the same names (activate and deactivate) both for a function and a class method. This should probably be avoided in a larger application. In this example, the code is short enough that it should not create too much confusion. In a real application we would also give the possibility of changing the user's preferences, storing the information in some configuration file.

Dynamic activation

Now that we now how to activate and deactivate a plugin, it might be useful to consider dynamic activation of an external plugin, not located in the normal plugins directory. For example, consider the following plugin (located in op_3.py):

from plugins.base import Plugin

class operator_mod_token(Plugin):
symbol = '%'
lbp = 10
def nud(self):
   return expression(100)
def led(self, left):
   return left % expression(10)

This file is located in subdirectory "external" which is at the same level as "plugins" in our sample code. To invoke this plugin from our base application, we need to add the following code to calculator.py:

if __name__ == "__main__":
#...

    # Simulating dynamic external plugin initialization
external_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                          'external')
sys.path.insert(0, external_dir)
mod = __import__('op_3')
mod.expression = expression
    # register this plugin using our default method
register_plugins()
    # Since it is not activated by default, we need to do it explictly
activate('%')
assert calculate("7%2") == 1

print "Done!"

Note that we also need to import register_plugins() from base.py to make this work.

That's it! If you get the code from the py-fun repository, you can try it out yourself.