Crunchy 1.0 released!!

Crunchy 1.0 has finally been released. :-)

Crunchy 1.0 is compatible with Python 2.4, 2.5, 2.6 ... and 3.1. It is also compatible with Jython 2.5 modulo some bugs when trying to work with examples containing unicode strings.

Crunchy, for those that are not familiar with it, is a one-of-a-kind application. It is an application designed to transforms otherwise static html Python tutorials into interactive session viewed within a browser. Currently Crunchy has only been fully tested with Firefox. Crunchy should work with all operating systems - it has been tested fairly extensively on Linux, Windows and Mac OS.

In more details, here's what Crunchy does:

1. It loads an existing html file (or reStructuredText file) containing some Python code; this file can reside locally, or anywhere on the web.
2. It removes any existing javascript code & embedded interactive elements (applets, flash apps, etc.).
3. It further transforms the file by inserting some extra elements, including some custom javascript code.
4. It sends the transformed file to your browser of choice (read: Firefox).
5. It establishes a communication between the browser and the Crunchy back end and wait for user interaction.

Crunchy can embed Python interpreters and code editors (and more!) within an html page, enabling a user to enter (or edit) some Python code inside a browser window, send it to the Crunchy back end for execution, and observe the result back in the browser window.

In a sense, Crunchy is a Python programming environment embedded within a browser. However, it does not require a custom format for interactivity: for html files, as long as the Python code is included in a pre-formatted element (<pre>), Crunchy will recognize it, style it, and include the appropriate interactive element.

Crunchy comes with a fairly complete tutorial and supporting documentation. It is highly configurable.

Release 1.0 is NOT the end of the road for Crunchy. Future plans include support for interactivity with languages other than Python.

Python textbooks wanted

Python textbooks wanted(1). Actually, what I'm mostly interested in are links to textbooks and other resources that would be useful to educators either teaching Python as a programming language or other courses where Python feature prominently. The reason behind this request is that I volunteered to be responsible to take care of the edu-sig page on the Python site. Through no one's fault in particular, the old page (2) had gone stale and only one textbook was listed. We're now up to six and I am sure I have not included them all. By ensuring that up to date and complete information is available on that page, we can facilitate the adoption of Python as a language taught in High Schools, Colleges and Universities.

So, if you know of any useful information that should be added to the edu-sig page, please let me know.

(1) Actually, this is not exactly correct ... but if you want to send me some for reviews, I won't say no. ;-)

(2) The page linked here will eventually disappear...

Learning paths at ShowMeDo

ShowMeDo is a great site to learn about Python (and a few other subjects) by watching screncasts. One of my favourite videos is Learn Django: Create a Wiki in 20 minutes, which taught me the basic concepts so that I was later able to adapt and play around with Google App Engine. Many authors on ShowMeDo have multiple screencasts (I made a few on RUR-PLE and Crunchy a while ago), some of which (not mine) are very professional looking. Such series of videos give the viewpoint of a given author, introducing some concepts in a logical sequence.

The newest addition to the ShowMeDo site is called Learning Paths. The idea of Learning Paths is to improve upon the existing series by including videos from multiple authors into a coherent sequence. While this concept is very much in its infancy, it promises to become a great addition to what is already a fantastic site.

In case anyone were wondering: I have no financial interest whatsover in ShowMeDo; I am just a satisfied user who considers this site a very good resource, well worth exploring. If only I had more time...

Testing code highlighting

Update:

In order to have the styling work, I had to include by hand the styling information (keyword and string) from this style sheet - and adapt as desired.

- - - - - - -
This is a test of Python code highlighting on Blogger following these instructions.

print "Hello World!"

# And this is a comment

End of test.

Plugins - part 6: setuptools based approach

setuptools is collection of enhancements to the standard Python distutils module. While it is not included in the Python standard library, chances are that it is already installed on your system if you have installed some additional Python libraries since it is so widely used to install Python packages ("eggs") via easy_install.

setuptools is also used in many applications as a handler for plugins. Many such applications include tutorials for creating new plugins using setuptools. For a somewhat general introduction to using setuptools to create plugin based applications, I suggest you have a look at the two tutorials from which this one is inspired.

Our starting point for this tutorial is essentially the same as in the third one in this series. We start with the following files:

root_directory/
    calculator.py
    setup.py
    plugins/
        __init__.py
        base.py
        op_1.py
        op_2.py

where we have one new file, setup.py. This file is a special file for setuptools. Its content is as follows:

''' run with python setup.py develop '''

from setuptools import setup, find_packages

setup(
    name="Calculator_s_tools",
    version="1.0",
    packages=['plugins'],
    entry_points="""
        [plugin_tutorial.s_tools]
        add = plugins.op_1:operator_add_token
        sub = plugins.op_1:operator_sub_token
        mul = plugins.op_1:operator_mul_token
        div = plugins.op_1:operator_div_token
        pow = plugins.op_2:operator_pow_token"""
       )

The key concept for setuptools is that of "entry_points". We define some entry points, with a name "plugin_tutorial.s_tools" chosen as unique to our application. Within this entrypoint we indicate which classes should be imported. This method effectively replace our custom method for finding and loading plugins. However, if you remember from previous tutorials, the way the application was designed originally (all within a single file) resulted in a "wart", where we had to create a link to a function ("expression") in each plugin module. Since setuptools will import the classes for us, we have no way to tell it how to fix that "wart" - we have to find another way. The method we chose was to create a different Plugin base class, one that implements the Borg idiom, so that each instance shares common attributes. Here's the new "base.py":

import os
import sys
import pkg_resources  # setuptools specific

OPERATORS = {}
ENTRYPOINT = 'plugin_tutorial.s_tools'  # same name as in setup.py

class Plugin(object):
    '''A Borg class.'''
    _shared_states = {}
    def __init__(self):
         self.__dict__ = self._shared_states

def init_plugins(expression):
    '''simple plugin initializer'''
    Plugin().expression = expression  # fixing the wart
    load_plugins()

def load_plugins():
    '''setuptools based plugin loader'''
    for entrypoint in pkg_resources.iter_entry_points(ENTRYPOINT):
        plugin_class = entrypoint.load()
        OPERATORS[plugin_class.symbol] = plugin_class

The actual plugin files are slightly modified to derive from the new Plugin class; for example, op_2.py contains the following:

from plugins.base import Plugin

class operator_pow_token(Plugin):
    symbol = '**'
    lbp = 30
    def led(self, left):
        return left ** self.expression(30-1)

where "expression" is now a class variable obtained from Plugin.

The code involved to make the setuptools approach is approximately the same level of complexity as the class-based plugin system covered previously. The advantages of using the setuptools approach are as follows:

1. Since it is a widely used tool, many people know how to use it properly.
2. It is possible to package plugins as eggs to be uploaded to a repository.
3. It is possible to keep track of dependencies in a fairly detailed way (e.g. module X version Y required).
   

By comparison, it suffers from the following disadvantages:

1. Some information about plugin location (entry_points name) is duplicated, appearing in both setup.py and base.py (in our example).
2. Automatic plugin discovery without editing of a file (setup.py) is not possible, unlike the cases we covered before. Because of this, dynamic loading of "external" plugins while the application is already running may be problematic to achieve. (I am not familiar enough with setuptools to determine if it is feasible or not.) See the first comment par Phillip J. Eby on how to achieve this.
   
3. A preliminary step ("python setup.py develop") is required to generate entrypoints information.
4. A number of additional files are created by the previous step, "cluttering" slightly the file structure by adding an extra directory with a few files.
   

That being said, the differences between the two approaches are relatively minor when everything is taken into account. Choosing one approach over the other is a matter of individual taste - at least for simple applications such as the one we considered.

Plugins - part 5: Activation and Deactivation

(Note: the code indentation may appear to be wrong due to some blogspot's quirks...)

While plugins are a great way to extend the functionality of an application, sometimes it makes sense to limit the number of available features, based on a user's preference. For example, gedit, the official text editor of the Gnome environment, offers the possibility to activate or deactivate a plugin.

In this post, using the class-based plugin approach, I will explain how to add the possibility to activate or deactivate a given plugin. Furthermore, I will show how to use this capability to dynamically load new plugins.

Starting from the beginning...

Our starting point will be the following modified core application (calculator.py):

import re

from plugins.base import OPERATORS, init_plugins, activate, deactivate

class literal_token(object):
 def __init__(self, value):
     self.value = value
 def nud(self):
     return self.value

class end_token(object):
lbp = 0

def tokenize(program):
 for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
    if number:
        yield literal_token(int(number))
    elif operator in OPERATORS:
        yield OPERATORS[operator]()
    else:
        raise SyntaxError("unknown operator: %r" % operator)
 yield end_token()

def expression(rbp=0):
 global token
 t = token
 token = next()
 left = t.nud()
 while rbp < token.lbp:
     t = token
     token = next()
     left = t.led(left)
 return left

def calculate(program):
 global token, next
 next = tokenize(program).next
 token = next()
 return expression()

if __name__ == "__main__":
init_plugins(expression)
assert calculate("+1") == 1
assert calculate("-1") == -1
assert calculate("10") == 10
assert calculate("1+2") == 3
assert calculate("1+2+3") == 6
assert calculate("1+2-3") == 0
assert calculate("1+2*3") == 7
assert calculate("1*2+3") == 5
assert calculate("6*2/3") == 4

    # "**" has not been activated at the start in base.py
    try:
        assert calculate("2**3") == 8
    except SyntaxError:
        print "Correcting error..."
        activate("**")
    assert calculate("2*2**3") == 16

    deactivate('+')
    try:
        assert calculate("1+2") == 3
    except SyntaxError:
        activate('+')
    assert calculate("1+2") == 3

print "Done!"

The new features are indicated by different colours. In blue, we have two new functions imported to either activate or deactivate a given plugin. When the application is started, exponentiation is disabled - this can only be seen by looking at the modified version of base.py. When a disabled plugin is called, a SyntaxError already present in the old version) is raised and we activate the plugin.

To make this possible, we need to modify base.py. Before showing the new version, here's the result of running the above code:

Activating +
Activating -
Activating *
Activating /
Correcting error...
Activating **
Deactivating +
Activating +
Done!

And here's the new version of base.py:

import os
import sys

OPERATORS = {}

# We simulate a configuration file that would be based on a user's preference
# as to which plugin should be activated by default
# We will leave one symbol "**" out of the list as a test.
preferences = ['+', '-', '*', '/']

# We also keep track of all available plugins, activated or not
all_plugins = {}

class Plugin(object):
'''base class for all plugins'''

    def activate(self):
        '''activate a given plugin'''
        if self.symbol not in OPERATORS:
            print "Activating %s" % self.symbol
            OPERATORS[self.symbol] = self.__class__
        if self.symbol not in all_plugins:
            all_plugins[self.symbol] = self.__class__

    def deactivate(self):
        '''deactivate a given plugin'''
        print "Deactivating %s" % self.symbol
        if self.symbol in OPERATORS:
            del OPERATORS[self.symbol]

def activate(symbol):
    '''activate a given plugin based on its symbol'''
    if symbol in OPERATORS:
        return
    all_plugins[symbol]().activate()

def deactivate(symbol):
    '''deactivate a given plugin, based on its symbol'''
    if symbol not in OPERATORS:
        return
    all_plugins[symbol]().deactivate()

def init_plugins(expression):
'''simple plugin initializer
'''
find_plugins(expression)
register_plugins()

def find_plugins(expression):
'''find all files in the plugin directory and imports them'''
plugin_dir = os.path.dirname(os.path.realpath(__file__))
plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
sys.path.insert(0, plugin_dir)
for plugin in plugin_files:
 mod = __import__(plugin)
 mod.expression = expression

def register_plugins():
'''Register all class based plugins.

Uses the fact that a class knows about all of its subclasses
to automatically initialize the relevant plugins
'''
for plugin in Plugin.__subclasses__():
        # only register plugins according to user's preferences
        if plugin.symbol in preferences:
            plugin().activate()
        else:   # record its existence
            all_plugins[plugin.symbol] = plugin

Changes from the old version are indicated in blue (with corresponding comments in green). Note that we did not change a single line of code for the actual plugins! We did use the same names (activate and deactivate) both for a function and a class method. This should probably be avoided in a larger application. In this example, the code is short enough that it should not create too much confusion. In a real application we would also give the possibility of changing the user's preferences, storing the information in some configuration file.

Dynamic activation

Now that we now how to activate and deactivate a plugin, it might be useful to consider dynamic activation of an external plugin, not located in the normal plugins directory. For example, consider the following plugin (located in op_3.py):

from plugins.base import Plugin

class operator_mod_token(Plugin):
symbol = '%'
lbp = 10
def nud(self):
   return expression(100)
def led(self, left):
   return left % expression(10)

This file is located in subdirectory "external" which is at the same level as "plugins" in our sample code. To invoke this plugin from our base application, we need to add the following code to calculator.py:

if __name__ == "__main__":
#...

    # Simulating dynamic external plugin initialization
external_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                          'external')
sys.path.insert(0, external_dir)
mod = __import__('op_3')
mod.expression = expression
    # register this plugin using our default method
register_plugins()
    # Since it is not activated by default, we need to do it explictly
activate('%')
assert calculate("7%2") == 1

print "Done!"

Note that we also need to import register_plugins() from base.py to make this work.

That's it! If you get the code from the py-fun repository, you can try it out yourself.

A small svg module

Update: by combining suggestions made in comments, one can probably do away with much of what I describe in this blog post. To wit:

>>> from xml.etree import ElementTree as etree
>>> from functools import partial
>>> Circle = partial(etree.Element, 'svg:circle')
>>> c = Circle(cx='100', cy='200', fill='red')
>>> etree.tostring(c)
'<svg:circle cx="100" cy="200" fill="red" />'

The only minor drawback is that attributes have to be strings, whereas the module described in this post could handle integer attributes. (Python 2.5+ required for ElementTree)

Original post below
(Time to take a break from the plugins blog series...)

Scalable Vector Graphics (SVG) are becoming more and more common on the web due to the increased support by decent browsers. SVG specifications include basic shapes such as circle, rectangles, etc., as well as supporting clipping, masking and composition, filter effects and much more. Attempting to write a python-ic module supporting all possible SVG primitives and options via code like

test_circle = Circle(x=10, y=10, r=5, color='red')

can be a daunting task. Furthermore, documenting such a module would result in a lot of duplication with the official specification document. Fortunately, there is a simpler way than simply attempting to write a complete SVG module using Class-based definitions such as the one written above. The idea is to use instead an API similar to that of ElementTree (see also) - albeit much simplified.

Suppose that we would want to be able to create SVG circles, such as

<circle cx="600" cy="200" r="100" fill="red" stroke="blue" width="10"/>

and rectangles, such as

<rect x="1" y="1" height="398" fill="none" stroke="blue" width="1198"/>

using Python code. A simple way to achieve this would be to define the following class:

class XmlElement(object):
    '''First prototype from which all the xml elements are derived.

       By design, this enables all elements to automatically give a
       text representation of themselves - it is not quite complete.'''

    def __init__(self, tag, **attributes):
        '''A basic definition that will be replaced by the specific
           one required by any element.'''
self.tag = tag
if attributes is not None:
   self.attributes = attributes
else:
   self.attributes = {}

    def __repr__(self):
        '''This normal python method used to give a string representation
        for an object is used to automatically create the appropriate
        syntax representing an xml object.'''
attrib = ["  <%s" % self.tag] # open tag
for att in self.attributes:
   attrib.append(' %s="%s"' % (att, self.attributes[att]))
attrib.append("/>\n")
return ''.join(attrib)

Using this class, we can create a circle instance corresponding to the definition written previously as

circle = XmlElement("circle", cx=600, cy=200, r=100, fill="red",
            stroke="blue", width=10)

This is not quite as simple as the very first Circle() class-based example we wrote but it has the advantage of supporting all possible SVG attributes.

While the above XmlElement class definition is adequate for most basic SVG elements, it does not support such features as 1. text, 2. namespace (e.g. svg: prefix) and 3. grouping and sub-elements. All three additional features can be taken care of by the following modified class definition:

class XmlElement(object):
    '''Prototype from which all the xml elements are derived.

       By design, this enables all elements to automatically give a
       text representation of themselves.'''

    def __init__(self, tag, **attributes):
        '''A basic definition that will be replaced by the specific
           one required by any element.'''
 self.tag = tag
 self.prefix = ""
 self.sub_elements = []
 if attributes is not None:
     self.attributes = attributes
 else:
     self.attributes = {}

    def __repr__(self):
        '''This normal python method used to give a string representation
        for an object is used to automatically create the appropriate
        syntax representing an xml object.'''
 attrib = ["  <%s%s"%(self.prefix, self.tag)] # open tag
 for att in self.attributes:
     if att != 'text':
         attrib.append(' %s="%s"' % (att, self.attributes[att]))
 if 'text' in self.attributes:
     attrib.append(">%s\n" % (self.attributes['text'],
                                             self.prefix, self.tag))
 elif self.sub_elements:
     attrib.append(">\n")
     for elem in self.sub_elements:
         attrib.append("  %s" % elem)
     attrib.append("\n" % (self.prefix, self.tag))
 else:
     attrib.append("/>\n")
 return ''.join(attrib)

    def append(self, other):
        '''append other to self to create list of lists of elements'''''
 self.sub_elements.append(other)

That's almost it! With the exception of comments and Document Type Definition (dtd), we can use the above to create simple xhtml document containing ANY svg graphics without having to worry about xhtml syntax, opening and closing brackets, etc. However, we can possibly do even a little better. Consider the following xhtml document with embedded svg graphics:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink">
<head>
<title>This is the title.</title>
</head>
<body>
<p>This is the body.</p>
<svg:svg width="0" height="0">
<svg:defs>
<svg:circle cy="0" cx="0" r="20" id="red_circle" fill="red"/>
</svg:defs>
</svg:svg>
<svg:svg width="200" height="200">
<svg:use xlink:href="#red_circle" transform="translate(100, 100)"/>
</svg:svg>
<!-- This is a comment. -->
</body>
</html>

With just a few additional definitions, we can create this document using only Python code as follows:

doc = XmlDocument()
doc.head.append(XmlElement("title", text="This is the title."))

# A good practice is to define svg objects, and insert them
# using the definition; this is overkill for this example, but it
# provides a test of the class.
test_def = SvgDefs()
test_def.append(SvgElement("circle", cx=0, cy=0, r=20, fill="red",
                  id="red_circle"))

doc.body.append(XmlElement("p", text="This is the body."))
doc.body.append(test_def)

# we now create an svg object, that will make use of the definition above.
svg_window = SvgElement("svg", width="200", height="200")
use_circle = SvgElement("use", transform="translate(100, 100)")

# xlink:href can't be used as an attribute name passed to __init__
# this is why we use this two-step process.
use_circle.attributes["xlink:href"] = "#red_circle"

svg_window.append(use_circle)
doc.body.append(svg_window)
doc.body.append(Comment("This is a comment.")) # just for fun.

print doc

The additional definitions are as follow:

class XmlDocument(XmlElement):
    def __init__(self):
   self._begin = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
    xmlns:svg="http://www.w3.org/2000/svg"
    xmlns:xlink="http://www.w3.org/1999/xlink">\n"""
   self._end = "</html>"
   self.head = XmlElement("head")
   self.body = XmlElement("body")

    def append(self):
        '''Directly appending is not allowed'''
   assert False, "Append to either head or body."

    def __repr__(self):
        '''Gives an appropriate representation of an xml document.'''
   return self._begin + str(self.head) + str(self.body) + self._end


class SvgElement(XmlElement):
    '''Prototype from which all the svg elements are derived.

       By design, this enables all elements to automatically give an
       appropriate text representation of themselves.'''
    def __init__(self, tag, **attributes):
   XmlElement.__init__(self, tag, **attributes)
   self.prefix = "svg:"

class SvgDefs(SvgElement):
    '''Short-cut to create svg defs.  A user creates an instance of this
    object and simply appends other svg Elements'''
    def __init__(self):
   self.defs = SvgElement("defs")
   self.root = SvgElement("svg", width=0, height=0)
   self.root.append(self.defs)

    def append(self, other):
        '''appends other to defs sub-element, instead of root element'''
   self.defs.append(other)

    def __repr__(self):
        '''gives a string representation of an object, appropriate for
           insertion in an html document'''
   return str(self.root)

class Comment(object):
    '''Comment that can be inserted in code xml documents'''
def __init__(self, text):
   self.text = text
def __repr__(self):
   return "<!-- " + self.text + " -->\n"

That's it for real this time! Fewer than 100 lines of code that you can use if you need to programmatically create (x)html documents containing svg images. There are a few limitations (elements containing text may not be chained...) but it works for me. If you want to try it yourself, you can find the module here.