Only Python

Porting to Python 3: What are you waiting for?

2011-12-09T22:30:00.001-04:00

Lately, there seems to be a lot of posts trying to encourage people to port their apps/libraries/modules to Python 3. I'd like to add my voice and perhaps, use as a reminder this old post: more than 2 years ago, Crunchy was made compatible with Python 2.4, 2.5, 2.6 ... and 3.1. At the time, we made the decision to not follow what seem to be the official recommendation, namely to have one code base based on 2.x and use the 2to3 tool to do the automatic translation. Instead, we decided to have a single code base, which seems the way people are doing it these days. There were of course some challenges, especially as we kept compatibility all the way back to Python 2.4: since Crunchy puts a Python interpreter inside your browser and manipulates what's there, regardless of the encoding, it has to be able to do a lot of string (and bytes for 3.x) manipulations, as well as dealing with incompatible syntax for handling exceptions, etc. when running the code that is fed to it. This was done when there was very little known about best practices for porting from 2.x to 3. Partly as a result, Crunchy's code is not the best example to look at when it comes to doing such a port ... but it certainly demonstrated that it was possible to port a non-trivial program with a tiny team. Now, the situation is quite different, and many more projects have been ported, and you can benefit from their experience.

So, what are you waiting for?

Errors should never pass silently

2011-09-22T00:38:00.000-03:00

- The Zen of Python

Lately, I have been programming a lot in Javascript, and have come to appreciate even more what Tim Peters expressed so well in writing what is now known as The Zen of Python. In particular, I appreciate the philosophy that "Errors should never pass silently."

Those that don't care about Javascript can stop reading now. Others may benefit, and perhaps even enlighten me even further, about a "feature" I came across.

I'm working on a new website which will include only interactive tutorials. [No, not Crunchy-based ones, but honest-to-goodness interactive tutorials that require nothing but a browser.] To make my life easier, I use jQuery as well as CodeMirror. A user can edit the code in the CodeMirror editor and view the result "live" in a separate window (html iframe). Every time the content of the editor is changed, the iframe gets updated, as illustrated here.

I started by using the standard $ jQuery notation inside the html iframe without including a link to jQuery in the iframe, since I already included it in the parent page, and got an error in the console to the effect that $ was not defined. Fair enough. I changed the source to also include a link to jQuery inside the content of the editor (to be copied in the iframe) and the error vanished. However, no javascript code was run from the iframe, not even a simple test alert. Meanwhile, the javascript console remained silent.

Errors should never pass silently.

Eventually, by searching on the web, I was able to find a solution. However, before I found a solution, I encountered a truly puzzling "feature".

After including a link to jQuery in the editor (which then was copied into the iframe) and reloading the page, if I then proceeded to remove that link from the editor, suddenly all javascript code embedded in the iframe, including all jQuery calls, would work.

I can just imagine giving these instructions to beginners:

We are going to use jQuery, as you can see from the code in the editor. Now, to begin, you have to select the line where jQuery is loaded and delete it so that the browser (iframe) can use it.

I still have not figured out how that phantom jQuery worked... and strongly believe that some error message or warning should have appeared in the javascript console (for Chrome, or Firebug for Firefox).

In any event, to solve my problem, I had to do 3 things:

Only load jQuery once, in the main window.
Write $ = parent.$; inside the iframe before using the $ notation.
As I wanted to draw things on a canvas, replace the usual $("#canvas") jQuery idiom by $("#canvas", document) everywhere.

Now everything appears to work just as expected.

Book review: Pro Python System Administration

2011-01-16T21:00:00.001-04:00

Summary: Pro Python System Administration is a comprehensive book showing how Python can be used effectively to perform a variety of system administration tasks. I would recommend it highly to anyone having to do system administration work. For more information, please consult the author's web site.
===

There is a saying that "no good deed goes unpunished". I feel that a counterpart should be "no bad talk goes unrewarded". At Pycon 2009, I gave a talk on plugins that has to be amongst the worst presentations I ever gave. Yet, as an unexpected result from that talk, I received a free copy of Pro Python System Administration written by Rytis Sileika. This blog entry is a review of that book.

This book is written for system administrators, something in which I have no experience; therefore, this review will definitely not have the depth that an expert may have given it.

Four general areas of system administrations are covered: network management, web server and web application management, database system management, and system monitoring. Examples are given on a Linux system with no indication as to whether or not a given example is applicable to other operating systems. Given that Python works with all major operating systems, and that the book focuses on using Python packages, I suspect that the content would be easily adaptable to other environments.

While the book is classified, on the cover, as being addressed to advanced Python programmers, the author in the book introduction indirectly suggests that this book would be appropriate for people that have some minimal experience with Python. I suspect that the classification on the book cover was done (wrongly) by the editor as I found the examples very readable and I would not claim to be an advanced Python programmer.

The book is divided into 13 chapters, each focused on one or a few well-defined tasks. While the tasks in a given chapter are relatively independent of those of other chapters, there is a natural progression in terms of topics introduced and it is probably better to read the book in the natural sequence rather than reading chapters randomly - in other words, this book is not simply a random collection of recipes for a series of tasks.

    1. Reading and Collecting Performance Data Using SNMP

In this chapter, Sileika introduces the Simple Network Management Protocol, or SNMP after which he shows how one can query SNMP devices using Python and the PySNMP library. In the second part of that chapter, he introduces RRDTool, an application for graphing monitoring data, and shows how to interact with it using the rrdtool module. In the last section of this first chapter, he shows how to create web pages (with the eventual goal of displaying on the web monitoring data) using the Jinja2 templating system.

    2. Managing Devices Using the SOAP API

In this chapter, Sileika introduces the Simple Object Access Protocol or SOAP and gives examples based on using the Zolera SOAP Infrastructure (ZSI) package. The bulk of the chapter focuses on explaining how to manage and monitor Citrix Netscaler load balancers. The Python logging module is introduced and used.

    3. Creating a Web Application for IP Address Accountancy

In this chapter, the Django framework is introduced and used to build a web application that maintains IP addresses allocation on an internal network. Rather than using the web server included with Django, Sileika shows how to use Django with the Apache web server.

    4. Integrating the IP Address Application with DHCP

This chapter is a continuation of the previous one, where the application previously developed is enhanced with the addition of Dynamic Host Configuration Protocol (DHCP) services as well as a few others. More advanced Django methods are included as well as some AJAX calls.

    5. Maintaining a List of Virtual Hosts in an Apache Configuration File

Another Django based application is introduced, this time with a focus on the administration side.

    6. Gathering and Presenting Statistical Data from Apache Log Files

This chapter focuses on building a plugin-based modular framework to analyze log files. The content of this chapter is the reason why I received a free copy of the book in the first place: Sileika mentioned to me in an email that the architecture described was mostly inspired by the presentation I gave at PyCon with a few modifications that allow for information exchange between the plug-in modules. When I got the original email, I was really surprised given that I had tried to forget about the talk I had given on plugins.

In my opinion, Sileika does an excellent job in explaining how plugins can be easily created with Python and used effectively to design modular applications.

Dynamic discovery and loading of plugins is illustrated, and the GeoIP Python library is used to find the physical location corresponding to a given IP address.

    7. Performing Complex Searches and Reporting on Application Log Files

This chapter shows how to use Exctractor, an open source log file parser tool, to parse more complex log files than those generated by an Apache server as illustrated in the previous chapter. Of note is the introduction and usage of Python generators as well as an introduction to parsing XML files with Python.

    8. A Web Site Availability Check Script for Nagios

This chapter shows how to use Python with Nagios, a network monitoring system. Python packages/modules illustrated include BeautifulSoup, urllib and urllib2. The monitoring scripts developed do more than simply checking for site availability and actually test the web application logic.

    9. Management and Monitoring Subsystem
    10. Remote Monitoring Agents
    11. Statistics Gathering and Reporting

Chapters 9, 10 and 11 explain how to build a "simple" distributed monitoring system. A number of Python module & packages are used including xmlrpclib, SimpleXMLRPCServer, CherryPy, sqlite3, multiprocessing, ConfigParser, subprocess, Numpy, matplotlib and others. The application developed is a good example of using Python as a "glue language", to use third-party modules & packages. One weakness is that the introduction to statistics included is rather elementary and, I believe, could have been shortened considerably given the intended audience.

    12. Automatic MySQL Database Performance Tuning

This chapter revisits the use of plugins, with a slightly more advanced application, where information can be exchanged between the different plugins.

    13. Using Amazon EC2/S3 as a Data Warehouse Solution

This last chapter gives a crash course on Amazon's "cloud computing" offerings. It is a good final note to the book, and a good starting point for future explorations.

Overall, I found the book quite readable even though it is outside of my area of expertise. Occasionally, I had the feeling that there were a few "fillers" (e.g. overlong log listings, etc.) that could have been shortened without losing anything of real value. This is very much an applied book, with real life examples that could be either used "as-is" or used as starting points for more extended applications.

I would recommend this book highly to anyone who has to perform any one of the system administration tasks mentioned above. It is also a good source of non-trivial examples demonstrating the use of a number of Python modules and packages. The code that is given would likely save many hours of development.   As is becoming the norm, the source code included in the book is available from the publisher. Also, many of the prototypes covered in the book are available as open source projects at the author's web site http://www.sysadminpy.com, where a discussion forum is also available.

My favourite Python one-liner

2010-08-01T13:01:00.000-03:00

I am slowly working on two new projects involving web-based interactive Python tutorials and had to set up a local web server for handling ajax request. After browsing the web, I stumbled upon the following one-liner in the Python documentation that I had completely forgotten about:

python -m SimpleHTTPServer 8000

Then, simply pointing the browser at "http://localhost:8000" and all is set to go. Granted, it is not what one might consider a traditional "one-liner" but it works for me. Python's batteries included is certainly a nice feature!

Profiling adventures and cython - Faster drawing and conclusion

2010-01-17T16:14:00.000-04:00

In the last blog post, I made use of cython to speed up some calculations involving the iterative equation defining the Mandelbrot set. From the initial profiling run with 1000 iterations in the second post to the last run in the previous post, the profiling time went from 575 seconds down to 21 seconds, which is a reduction by a factor of 27. This is nothing to sneer at. Yet, I can do better.
Let's start by having a sample profile run with 1000 iterations with the program as it was last time, but keeping track of more function calls in the profile.

      5441165 function calls in 20.484 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
494604    8.461    0.000   14.218    0.000 Tkinter.py:2135(_create)
    11    3.839    0.349   20.315    1.847 mandel2g_cy.pyx:27(create_fractal)
494632    2.053    0.000    5.309    0.000 Tkinter.py:1046(_options)
494657    1.809    0.000    2.811    0.000 Tkinter.py:77(_cnfmerge)
494593    1.522    0.000   16.475    0.000 mandel2g_cy.pyx:21(draw_pixel)
989240    0.752    0.000    0.752    0.000 {method 'update' of 'dict' objects}
494593    0.736    0.000   14.953    0.000 Tkinter.py:2155(create_line)
989257    0.698    0.000    0.698    0.000 {_tkinter._flatten}
494632    0.315    0.000    0.315    0.000 {method 'items' of 'dict' objects}
     1    0.158    0.158    0.158    0.158 {_tkinter.create}
494641    0.131    0.000    0.131    0.000 {callable}
    37    0.009    0.000    0.009    0.000 {built-in method call}
    11    0.001    0.000   20.317    1.847 viewer2b.py:20(draw_fractal)
    12    0.000    0.000    0.000    0.000 viewer.py:60(info)

I notice that there are many function calls: over 5 millions of them. While most of them appear to take very little time, they do add up in the end. It is time to adopt a smarter drawing strategy.
Currently, a "line" is potentially drawn for each pixel. If I look at a given fractal drawing, I can see that it could be drawn using "longer lines", when consecutive pixels are to be drawn with the same colour. I can easily implement this as follows.

def create_fractal(int canvas_width, int canvas_height,
                       double min_x, double min_y, double pixel_size,
                       int nb_iterations, canvas):
    cdef int x, y, start_y, end_y
    cdef double real, imag
    cdef bint start_line

    for x in range(0, canvas_width):
        real = min_x + x*pixel_size
        start_line = False
        for y in range(0, canvas_height):
            imag = min_y + y*pixel_size
            if mandel(real, imag, nb_iterations):
                if not start_line:
                    start_line = True
                    start_y = canvas_height - y
            else:
                if start_line:
                    start_line = False
                    end_y = canvas_height - y
                    canvas.create_line(x, start_y, x, end_y, fill="black")
        if start_line:
            end_y = canvas_height - y
            canvas.create_line(x, start_y, x, end_y, fill="black")

Note that I no longer need the function draw_pixel(). The result is a reduction from 20 seconds down to 4 seconds:

      79092 function calls in 3.959 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    11    3.552    0.323    3.815    0.347 mandel2h_cy.pyx:21(create_fractal)
  7856    0.150    0.000    0.250    0.000 Tkinter.py:2135(_create)
     1    0.131    0.131    0.131    0.131 {_tkinter.create}
  7884    0.037    0.000    0.091    0.000 Tkinter.py:1046(_options)
  7909    0.030    0.000    0.047    0.000 Tkinter.py:77(_cnfmerge)
  7845    0.014    0.000    0.263    0.000 Tkinter.py:2155(create_line)
 15761    0.014    0.000    0.014    0.000 {_tkinter._flatten}
 15744    0.013    0.000    0.013    0.000 {method 'update' of 'dict' objects}
    37    0.009    0.000    0.009    0.000 {built-in method call}
  7884    0.005    0.000    0.005    0.000 {method 'items' of 'dict' objects}
  7893    0.002    0.000    0.002    0.000 {callable}
    11    0.001    0.000    3.818    0.347 viewer2b.py:20(draw_fractal)
    12    0.000    0.000    0.000    0.000 viewer.py:60(info)
    22    0.000    0.000    0.001    0.000 Tkinter.py:1172(_configure)

And it is now again my own code in create_fractal() that appears to be the limiting factor. Thinking back of when I increased the number of iterations from 100 to 1000, thus only affecting the execution time of mandel(), it seemed like this might be a good place to look at for possible time improvements. Let's recall what the code looks like.

cdef inline bint mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i

    for i in range(0, max_iterations):
        z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
                           2*z_real*z_imag + imag )
        if (z_real*z_real + z_imag*z_imag) >= 4:
            return False
    return (z_real*z_real + z_imag*z_imag) < 4

I used a Pythonic tuple assignement to avoid the use of temporary variables. However, in a typical iteration, there will be 4 multiplications for the tuple re-assigment and two more for the "if" statement, for a total of 6. It is certainly possible to reduce the number of multiplications by using temporary variables, as follows:

cdef inline bint mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i
    cdef double zr_sq, zi_sq, z_cross

    for i in range(0, max_iterations):
        zr_sq = z_real*z_real
        zi_sq = z_imag*z_imag
        z_cross = 2*z_real*z_imag

        z_real = zr_sq - zi_sq + real
        z_imag = z_cross + imag
        if (zr_sq + zi_sq) >= 4:
            return False
    return (zr_sq + zi_sq) < 4

So, there are now fewer multiplications to compute. Surely, this will speed up the code:

      78982 function calls in 4.888 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    11    4.478    0.407    4.748    0.432 mandel2i_cy.pyx:26(create_fractal)
  7845    0.153    0.000    0.256    0.000 Tkinter.py:2135(_create)
     1    0.128    0.128    0.128    0.128 {_tkinter.create}
  7873    0.040    0.000    0.095    0.000 Tkinter.py:1046(_options)
  7898    0.031    0.000    0.048    0.000 Tkinter.py:77(_cnfmerge)
  7834    0.014    0.000    0.270    0.000 Tkinter.py:2155(create_line)
 15739    0.013    0.000    0.013    0.000 {_tkinter._flatten}
 15722    0.013    0.000    0.013    0.000 {method 'update' of 'dict' objects}
    37    0.009    0.000    0.009    0.000 {built-in method call}
  7873    0.005    0.000    0.005    0.000 {method 'items' of 'dict' objects}
  7882    0.002    0.000    0.002    0.000 {callable}
    11    0.001    0.000    4.750    0.432 viewer2b.py:20(draw_fractal)
    12    0.000    0.000    0.000    0.000 viewer.py:60(info)
     4    0.000    0.000    0.000    0.000 {posix.stat}

Alas, that is not the case, as the previous profiling run was slightly below 4 seconds. [Note that I did run each profiling test at least three times to prevent any anomalous result.] Apparently my intuition is not a very good guide when it comes to predicting how cython will be able to optimize a given function.
So far, the pictures the program has been able to produce have only been in black and white. It is time to spruce things up and add colour. To do this, we will need to make three general changes:

We will modify mandel() so that it returns the number of iterations required to evaluate that a given point does not belong to the set; if it does belong, we will return -1.
We will create a colour palette as a Python list. For a given number of iterations required by mandel(), we will pick a given colour, cycling through the colours from the palette.
We will need to change our line drawing method so that we keep track of the colour (number of iteration) rather than simply whether or not the point is in the set ("black") or not.

The code is a bit trickier to set up than the previous version, but it uses a similar logic. To determine what colour to draw, we first calculate the colour of the first pixel, and then loop through all the others along a same line. Each time we see a colour change, it signals that we need to draw a line up to that point in the colour used up to that point ("current_colour") and assign the new colour as the new "current" one. When we reach the end of a line (column in the code...), we need to ensure that we draw the last line segment. Without further ado, here is the complete code of the revised cython module.

# mandel3cy.pyx
# cython: profile=True

import cython

def make_palette():
    '''sample coloring scheme for the fractal - feel free to experiment'''
    colours = []

    for i in range(0, 25):
        colours.append('#%02x%02x%02x' % (i*10, i*8, 50 + i*8))
    for i in range(25, 5, -1):
        colours.append('#%02x%02x%02x' % (50 + i*8, 150+i*2,  i*10))
    for i in range(10, 2, -1):
        colours.append('#00%02x30' % (i*15))
    return colours

colours = make_palette()
cdef int nb_colours = len(colours)

@cython.profile(False)
cdef inline int mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i

    for i in range(0, max_iterations):
        z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
                           2*z_real*z_imag + imag )
        if (z_real*z_real + z_imag*z_imag) >= 4:
            return i
    return -1

def create_fractal(int canvas_width, int canvas_height,
                       double min_x, double min_y, double pixel_size,
                       int nb_iterations, canvas):
    global colours, nb_colours
    cdef int x, y, start_y, end_y, current_colour, new_colour
    cdef double real, imag

    for x in range(0, canvas_width):
        real = min_x + x*pixel_size
        start_y = canvas_height
        current_colour = mandel(real, min_y, nb_iterations)
        for y in range(1, canvas_height):
            imag = min_y + y*pixel_size
            new_colour = mandel(real, imag, nb_iterations)

            if new_colour != current_colour:
                if current_colour == -1:
                    canvas.create_line(x, start_y, x, canvas_height-y,
                                        fill="black")
                else:
                    canvas.create_line(x, start_y, x, canvas_height-y,
                                        fill=colours[current_colour%nb_colours])
                current_colour = new_colour
                start_y = canvas_height - y

        if current_colour == -1:
            canvas.create_line(x, start_y, x, 0, fill="black")
        else:
            canvas.create_line(x, start_y, x, 0,
                                        fill=colours[current_colour%nb_colours])

If we profile this code, we find out that it takes about three times as long to generate a colour picture than it did to generate a black and white one - at least, for the starting configuration...

      2370682 function calls in 12.638 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    11    4.682    0.426   12.184    1.108 mandel3_cy.pyx:36(create_fractal)
237015    4.310    0.000    7.084    0.000 Tkinter.py:2135(_create)
237043    1.019    0.000    2.575    0.000 Tkinter.py:1046(_options)
237068    0.877    0.000    1.353    0.000 Tkinter.py:77(_cnfmerge)
     1    0.443    0.443    0.443    0.443 {_tkinter.create}
237004    0.418    0.000    7.502    0.000 Tkinter.py:2155(create_line)
474062    0.361    0.000    0.361    0.000 {method 'update' of 'dict' objects}
474079    0.313    0.000    0.313    0.000 {_tkinter._flatten}
237043    0.143    0.000    0.143    0.000 {method 'items' of 'dict' objects}
237052    0.061    0.000    0.061    0.000 {callable}
    37    0.009    0.000    0.009    0.000 {built-in method call}
    11    0.000    0.000   12.186    1.108 viewer3.py:20(draw_fractal)
    12    0.000    0.000    0.000    0.000 viewer.py:60(info)
     4    0.000    0.000    0.000    0.000 {posix.stat}

We also generate some nice pictures! First, using only 100 iterations.

Notice how the Mandelbrot set boundary seems rather smooth ... this is clearly a sign that we might be missing something. Increasing the number of iterations to 1000 reveals a different picture.

We see much more details and many points that were wrongly identified as being part of the Mandelbrot sets are correctly excluded (and colored!). What happens if we look at a region that contains more points inside the Mandelbrot set (thus requiring more iterations) and increase the number of iterations to 1500 (as I found that 1000 was not enough in this region).

Unfortunately, the profiling and the timing information displayed does not tell the entire story. In practice, I found that it would take many more seconds (sometimes more than 10) for the canvas to be updated than the timing information given for each of the three pictures displayed above. Something is happening behind the scenewhen the picture is updated on the screen and which is not being recorded by my simple timing method.

For comparison, I had a look at Aptus, a program that uses a hand-coded C extension for speed. Actually, I had tried Aptus a few years ago, when Ned Batchelder first mentioned it, and tried it again for fun as I was working on my own version. Aptus can produce really nice pictures, really fast. Here's an example that was produced, according to the timing given by Aptus itself in 0.25 seconds.

Note that the timing given by Aptus seems to be truly representative, unlike the timing for my program. I should mention that, in addition to using a hand-crafted C extension, Aptus uses wxPython instead of Tkinter, and it also uses PIL and numpy, both of which are known for their speed. It might be possible that using PIL and numpy with my program would improve the speed significantly. However, all three libraries are not part of the standard library and so do not meet the constraints I had decided upon at the beginning.

This concludes this profiling experiment ... at least for now. I should emphasize that the goal of these posts was to record a profiling experiment using cython. I do not pretend that this code was especially hand-crafted for speed ... even though I did try some simple changes to improve its speed. I may revisit this application at a later time, especially if someone more experienced can point out ways to increase the speed significantly, preferably while staying within the constraints I set up at the beginning: other than cython, use only modules from the standard library. However, I would be interested if anyone adapts this code to use PIL and/or numpy in a straightforward way and, in doing so, increases the speed significantly.

Profiling adventures and cython - Introducing cython

2010-01-17T13:23:00.004-04:00

In the previous blog post, I made some attempts at speeding up the function mandel() by making changes in the Python code. While I had some success in doing so, it was clearly not enough for my purpose. As a result, I will now try to use cython. Before I do this, I note again the result from the last profiling run, limiting the information to the 5 longest-running functions or methods.

       3673150 function calls in 84.807 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
3168000   73.210    0.000   73.210    0.000 mandel1c.py:7(mandel)
     11   10.855    0.987   84.204    7.655 viewer1.py:23(draw_fractal)
      1    0.593    0.593    0.593    0.593 {_tkinter.create}
 504530    0.137    0.000    0.137    0.000 viewer1.py:17(draw_pixel)
     37    0.009    0.000    0.009    0.000 {built-in method call}

The goal of cython could be described as providing an easy way to convert a Python module into a C extension. This is what I will do. [There are other ways to work with cython extensions than what I use here; for more information, please consult the cython web site.] Note that I am NOT a cython expert; this is only the first project for which I use cython. While I am not interested in creating an application for distribution, and hence do not use the setup method for cython, it is quite possible that there are better ways to use cython than what I explore here.
I first start by taking my existing module and copying it into a new file, with a ".pyx" extension instead of the traditional ".py".

# mandel2cy.pyx
# cython: profile=True

def mandel(c, max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    z = 0
    for i in range(0, max_iterations):
        z = z**2 + c
        if abs(z) >= 4:
            return False
    return abs(z) < 2

Note that I have removed the equivalence between range and xrange. The reason I have done this is because with xrange present like this in the file results in a compilation error when running cython with Python 3.1. Furthermore, as will be seen later, it is not really needed even for Python 2.x when using cython properly.
I have also included a commented line stating that 'profile' was equal to True; this is a cython directive that will enable the Python profiler to also include cython functions in its tally.

In order to import this module, I also need to modify the viewer to import the cython module. Here is the new version.

# viewer2.py

import pyximport
pyximport.install()

from mandel2_cy import mandel
from viewer import Viewer
import time

import sys
if sys.version_info < (3,):
    import Tkinter as tk
    range = xrange
else:
    import tkinter as tk

class FancyViewer(Viewer):
    '''Application to display fractals'''

    def draw_pixel(self, x, y):
        '''Simulates drawing a given pixel in black by drawing a black line
           of length equal to one pixel.'''
        return
        #self.canvas.create_line(x, y, x+1, y, fill="black")

    def draw_fractal(self):
        '''draws a fractal on the canvas'''
        self.calculating = True
        begin = time.time()
        # clear the canvas
        self.canvas.create_rectangle(0, 0, self.canvas_width,
                                    self.canvas_height, fill="white")
        for x in range(0, self.canvas_width):
            real = self.min_x + x*self.pixel_size
            for y in range(0, self.canvas_height):
                imag = self.min_y + y*self.pixel_size
                c = complex(real, imag)
                if mandel(c, self.nb_iterations):
                    self.draw_pixel(x, self.canvas_height - y)
        self.status.config(text="Time required = %.2f s  [%s iterations]  %s" %(
                                (time.time() - begin), self.nb_iterations,
                                                                self.zoom_info))
        self.status2.config(text=self.info())
        self.calculating = False

if __name__ == "__main__":
    root = tk.Tk()
    app = FancyViewer(root)
    root.mainloop()

Other than the top few lines, nothing has changed. Time to run the profiler with this new version.

       6841793 function calls in 50.145 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
3168000   35.913    0.000   35.913    0.000 mandel2_cy.pyx:4(mandel)
     11   10.670    0.970   48.754    4.432 viewer2.py:26(draw_fractal)
3168000    2.001    0.000   37.914    0.000 {mandel2_cy.mandel}
      1    1.356    1.356    1.356    1.356 {_tkinter.create}
 505173    0.167    0.000    0.167    0.000 viewer2.py:20(draw_pixel)

A reduction from 85 to 50 seconds; cython must be doing something right! Note that the calls to abs() have been eliminated by using cython. All I did is import the module via Python without making any other change to the code.
Note also that mandel appears twice: once (the longest running) as the function defined on line 8 of mandel2_cy.pyx, and once as a object belonging to the module mandel2_cy. I will come back to this later but, for now, I will do some changes to help cython do even better.
As mentioned before, cython is a tool to help create C extensions. One of the differences between C and Python is that variables have a declared type in C. If one tells cython about what type a given variable is, cython can often use that information to make the code run faster. As an example, I know that two of the variables are of type integers which is a native C type; I can add this information as follows.

# mandel2a_cy.pyx
# cython: profile=True

def mandel(c, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef int i
    z = 0
    for i in range(0, max_iterations):
        z = z**2 + c
        if abs(z) >= 2:
            return False
    return abs(z) < 2

Running the profiler with this change yields the following:

       6841793 function calls in 39.860 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
3168000   27.431    0.000   27.431    0.000 mandel2a_cy.pyx:4(mandel)
     11    9.869    0.897   39.339    3.576 viewer2.py:26(draw_fractal)
3168000    1.906    0.000   29.337    0.000 {mandel2a_cy.mandel}
      1    0.511    0.511    0.511    0.511 {_tkinter.create}
 505173    0.131    0.000    0.131    0.000 viewer2.py:20(draw_pixel)

Another significant time reduction, this time of the order of 20%. And we didn't tell cython that "z" and "c" are complex yet.

Actually, C does not have a complex data type. So, I can choose one of two strategies:

I can change the code so that I deal only with real numbers, by working myself how to multiply and add complex numbers.
I can use some special cython technique to extract all the relevant information about the Python built-in complex data type without changing the code inside the function (other than adding some type declaration).

I will choose the second of these methods and see what it gives. The required changes are as follows:

# mandel2b_cy.pyx
# cython: profile=True

cdef extern from "complexobject.h":

    struct Py_complex:
        double real
        double imag

    ctypedef class __builtin__.complex [object PyComplexObject]:
        cdef Py_complex cval


def mandel(complex c, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef int i
    cdef complex z

    z = 0. + 0.j

    for i in range(0, max_iterations):
        z = z**2 + c
        if abs(z) >= 2:
            return False
    return abs(z) < 2

The timing results are the following:

       6841793 function calls in 38.424 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
3168000   26.771    0.000   26.771    0.000 mandel2b_cy.pyx:14(mandel)
     11    9.435    0.858   38.209    3.474 viewer2.py:26(draw_fractal)
3168000    1.865    0.000   28.636    0.000 {mandel2b_cy.mandel}
      1    0.205    0.205    0.205    0.205 {_tkinter.create}
 505173    0.136    0.000    0.136    0.000 viewer2.py:20(draw_pixel)

The time difference between this run and the previous one is within the variation I observe from one profiling run to the next (using exactly the same program). Therefore, I conclude that this latest attempt didn't speed up the code. It is possible that I have overlooked something to ensure that cython could make use of the information about the complex datatype more efficiently ... It seems like I need a different strategy. I will resort to doing the complex algebra myself, and work only with real numbers. Here's the modified code for the mandel module.

# mandel2c_cy.pyx
# cython: profile=True

def mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i

    for i in range(0, max_iterations):
        z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
                     2*z_real*z_imag + imag )
        if (z_real*z_real + z_imag*z_imag) >= 4:
            return False
    return (z_real*z_real + z_imag*z_imag) < 4

I also change the call within draw_fractal() so that I don't use complex variables. The result is extremely encouraging:

       6841793 function calls in 7.205 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     11    4.379    0.398    7.066    0.642 viewer2a.py:26(draw_fractal)
3168000    1.557    0.000    2.570    0.000 {mandel2c_cy.mandel}
3168000    1.013    0.000    1.013    0.000 mandel2c_cy.pyx:4(mandel)
      1    0.130    0.130    0.130    0.130 {_tkinter.create}
 505173    0.114    0.000    0.114    0.000 viewer2a.py:20(draw_pixel)

This total execution time has been reduced from 38 to 7 seconds. mandel() is no longer the largest contributor to the overall execution time; draw_fractal() is. However, the program is still a bit too slow: without actually doing any drawing, it takes approximately 0.6 seconds to generate one fractal image. However, I can do better. Looking at the code, I notice that draw_fractal() contains two embedded for loops, resulting to all those calls to mandel(). Remember how telling cython about integer types used in loops sped up the code? This suggest that perhaps I should do something similar and move some of the code of draw_fractal() to the cython module. Here's a modified viewer module.

# viewer2b.py

import pyximport
pyximport.install()

from mandel2d_cy import create_fractal
from viewer import Viewer
import time

import sys
if sys.version_info < (3,):
    import Tkinter as tk
    range = xrange
else:
    import tkinter as tk

class FancyViewer(Viewer):
    '''Application to display fractals'''

    def draw_fractal(self):
        '''draws a fractal on the canvas'''
        self.calculating = True
        begin = time.time()
        # clear the canvas
        self.canvas.create_rectangle(0, 0, self.canvas_width,
                                    self.canvas_height, fill="white")
        create_fractal(self.canvas_width, self.canvas_height,
                       self.min_x, self.min_y, self.pixel_size,
                       self.nb_iterations, self.canvas)
        self.status.config(text="Time required = %.2f s  [%s iterations]  %s" %(
                                (time.time() - begin), self.nb_iterations,
                                                                self.zoom_info))
        self.status2.config(text=self.info())
        self.calculating = False

if __name__ == "__main__":
    root = tk.Tk()
    app = FancyViewer(root)
    root.mainloop()

And here is the new cython module, without any additional type declaration.

# mandel2d_cy.pyx
# cython: profile=True

def mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i

    for i in range(0, max_iterations):
        z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
                     2*z_real*z_imag + imag )
        if (z_real*z_real + z_imag*z_imag) >= 4:
            return False
    return (z_real*z_real + z_imag*z_imag) < 4

def draw_pixel(x, y, canvas):
    '''Simulates drawing a given pixel in black by drawing a black line
       of length equal to one pixel.'''
    return
    #canvas.create_line(x, y, x+1, y, fill="black")

def create_fractal(canvas_width, canvas_height,
                       min_x, min_y, pixel_size,
                       nb_iterations, canvas):
    for x in range(0, canvas_width):
        real = min_x + x*pixel_size
        for y in range(0, canvas_height):
            imag = min_y + y*pixel_size
            if mandel(real, imag, nb_iterations):
                draw_pixel(x, canvas_height - y, canvas)

The profiling result is as follows:

       3673815 function calls in 3.873 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     11    2.632    0.239    3.706    0.337 mandel2d_cy.pyx:24(create_fractal)
3168000    1.002    0.000    1.002    0.000 mandel2d_cy.pyx:4(mandel)
      1    0.155    0.155    0.155    0.155 {_tkinter.create}
 505173    0.072    0.000    0.072    0.000 mandel2d_cy.pyx:18(draw_pixel)
     37    0.009    0.000    0.009    0.000 {built-in method call}

Simply by moving over some of the code to the cython module, I have reduced the profiling time to almost half of it previous value. Looking more closely at the profiling results, I also notice that calls to mandel() now only appear once; some overhead in calling cython functions from python modules has disappeared. Let's see what happens if I now add some type information.

def create_fractal(int canvas_width, int canvas_height,
                       double min_x, double min_y, double pixel_size,
                       int nb_iterations, canvas):
    cdef int x, y
    cdef double real, imag

    for x in range(0, canvas_width):
        real = min_x + x*pixel_size
        for y in range(0, canvas_height):
            imag = min_y + y*pixel_size
            if mandel(real, imag, nb_iterations):
                draw_pixel(x, canvas_height - y, canvas)

The result is only slightly better:

       3673815 function calls in 3.475 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     11    2.189    0.199    3.308    0.301 mandel2e_cy.pyx:24(create_fractal)
3168000    1.046    0.000    1.046    0.000 mandel2e_cy.pyx:4(mandel)
      1    0.135    0.135    0.135    0.135 {_tkinter.create}
 505173    0.074    0.000    0.074    0.000 mandel2e_cy.pyx:18(draw_pixel)
     37    0.028    0.001    0.028    0.001 {built-in method call}

However, one thing I remember from the little I know about C it that, not only do variables have to be declared to be of a certain type, but the same has to be done to functions as well. Here, mandel() has not been declared to be of a specific type, so cython assumes it to be a generic Python object. After reading the cython documentation, and noticing that mandel() is only called from within the cython module, I conclude that not only should I specify the type for mandel() but that it probably makes sense to specify that it can be "inlined"; I also do the same for draw_pixel().

# mandel2f_cy.pyx
# cython: profile=True

cdef inline bint mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i

    for i in range(0, max_iterations):
        z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
                     2*z_real*z_imag + imag )
        if (z_real*z_real + z_imag*z_imag) >= 4:
            return False
    return (z_real*z_real + z_imag*z_imag) < 4

cdef inline void draw_pixel(x, y, canvas):
    '''Simulates drawing a given pixel in black by drawing a black line
       of length equal to one pixel.'''
    return
    #canvas.create_line(x, y, x+1, y, fill="black")

This yields a nice improvement.

       3673815 function calls in 2.333 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     11    1.190    0.108    2.194    0.199 mandel2f_cy.pyx:24(create_fractal)
3168000    0.930    0.000    0.930    0.000 mandel2f_cy.pyx:4(mandel)
      1    0.127    0.127    0.127    0.127 {_tkinter.create}
 505173    0.074    0.000    0.074    0.000 mandel2f_cy.pyx:18(draw_pixel)
     37    0.009    0.000    0.009    0.000 {built-in method call}

However... I asked cython to "inline" mandel, thus treating them as a pure C function. Yet, they both appear in the Python profiling information, which was not the case for abs() once I used cython for the first time. The reason it appears is that cython has been instructed to profile all functions in the module, via the directive at the top. I can selectively turn off the profiling for an individual function by importing the "cython module" and using a special purpose decorator as follows.

# mandel2g_cy.pyx
# cython: profile=True

import cython

@cython.profile(False)
cdef inline bint mandel(double real, double imag, int max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    cdef double z_real = 0., z_imag = 0.
    cdef int i

    for i in range(0, max_iterations):
        z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
                     2*z_real*z_imag + imag )
        if (z_real*z_real + z_imag*z_imag) >= 4:
            return False
    return (z_real*z_real + z_imag*z_imag) < 4

cdef inline void draw_pixel(x, y, canvas):
    '''Simulates drawing a given pixel in black by drawing a black line
       of length equal to one pixel.'''
    return
    #canvas.create_line(x, y, x+1, y, fill="black")

The result is even better than I would have expected!

      505519 function calls in 0.817 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    11    0.605    0.055    0.676    0.061 mandel2g_cy.pyx:27(create_fractal)
     1    0.128    0.128    0.128    0.128 {_tkinter.create}
504877    0.070    0.000    0.070    0.000 mandel2g_cy.pyx:21(draw_pixel)
    37    0.010    0.000    0.010    0.000 {built-in method call}
    11    0.001    0.000    0.678    0.062 viewer2b.py:20(draw_fractal)

From 85 seconds (at the beginning of this post) down to 0.8 seconds: a reduction by a factor of 100 ...thank you cython! :-)

However, increasing the number of iterations to 1000 (from the current value of 100 used for testing) does increase the time significantly.

      495235 function calls in 3.872 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    11    3.653    0.332    3.723    0.338 mandel2g_cy.pyx:27(create_fractal)
     1    0.136    0.136    0.136    0.136 {_tkinter.create}
494593    0.071    0.000    0.071    0.000 mandel2g_cy.pyx:21(draw_pixel)
    37    0.009    0.000    0.009    0.000 {built-in method call}
    11    0.001    0.000    3.726    0.339 viewer2b.py:20(draw_fractal)

It is probably a good time to put back the drawing to see what the overall time profile looks like in a more realistic situation.

      5441165 function calls in 20.747 CPU seconds

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
494604    8.682    0.000   14.427    0.000 Tkinter.py:2135(_create)
    11    3.863    0.351   20.572    1.870 mandel2g_cy.pyx:27(create_fractal)
494632    2.043    0.000    5.326    0.000 Tkinter.py:1046(_options)
494657    1.845    0.000    2.861    0.000 Tkinter.py:77(_cnfmerge)
494593    1.548    0.000   16.709    0.000 mandel2g_cy.pyx:21(draw_pixel)

Clearly, the limiting time factor is now the Tkinter based drawing, and not the other code. It is time to think of a better drawing strategy. However, this will have to wait until next post.

Profiling adventures [and cython]: basic profiling

2010-01-16T20:48:00.006-04:00

In the previous blog post, I introduced a simple Tkinter-based viewer for the Mandelbrot set. As I mentioned at that time, the viewer was really too slow to be usable. In this post, I will start do some basic profiling and start looking for some strategies designed to make it faster.

The first rule for making an application faster is to do a proper profile rather than guessing. I make use of the profiler module, focusing on the main method (draw_fractal()) which I wish to make faster, and paying a closer look only at the most time-consuming functions/methods.

# profile1.py

import pstats
import cProfile

from viewer1 import tk, FancyViewer

def main():
    root = tk.Tk()
    app = FancyViewer(root)
    app.nb_iterations = 100
    for i in range(10):
        app.draw_fractal()

if __name__ == "__main__":
    cProfile.run("main()", "Profile.prof")
    s = pstats.Stats("Profile.prof")
    s.strip_dirs().sort_stats("time").print_stats(10)

The profile run will call draw_fractal() once, when app is created, with the number of iterations for mendel() set at 20 (the default) and then call it again 10 times with a larger number of iterations. Running the profiler adds some overhead. Based on the previous run with no profiles, I would have expected a profiler run to take approximately 75 seconds: a little over 4 seconds for the initial set up and slightly more than 7 seconds for each of the subsequent runs. Instead, what I observe is that

69802205 function calls in 115.015 CPU seconds

  Ordered by: internal time
  List reduced from 60 to 10 due to restriction <10>

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 3168000   59.908    0.000   81.192    0.000 mandel1a.py:3(mandel)
57908682   14.005    0.000   14.005    0.000 {abs}
      11   13.387    1.217  114.484   10.408 viewer1.py:23(draw_fractal)
  505184   10.950    0.000   17.672    0.000 Tkinter.py:2135(_create)
 3168001    7.279    0.000    7.279    0.000 {range}
  505212    2.359    0.000    6.220    0.000 Tkinter.py:1046(_options)
  505237    2.169    0.000    3.389    0.000 Tkinter.py:77(_cnfmerge)
  505173    1.457    0.000   19.902    0.000 viewer1.py:17(draw_pixel)
 1010400    0.906    0.000    0.906    0.000 {method 'update' of 'dict' objects}
 1010417    0.817    0.000    0.817    0.000 {_tkinter._flatten}

Clearly, running the profiler adds some overhead. I should also add that there are variations from run to run done with the profiler, caused by background activities. As a consequence, I normally run the profiler 3 times and focus on the fastest of the three runs; however I will not bother to do this here: I simply want to start by establishing some rough baseline to identify the main contributors to the relative lack of speed of this program.

It appears clear that the largest contributor to the overall execution time is mandel(). Going down the lists of functions that contribute significantly to the overall time, I notice quite a few calls to Tkinter function/methods. So as to reduce the time to take a given profile, and to focus on mandel(), I will temporarily eliminate some Tkinter calls by changing draw_pixel() as follows.

def draw_pixel(self, x, y):
    '''Simulates drawing a given pixel in black by drawing a black line
       of length equal to one pixel.'''
    return
    #self.canvas.create_line(x, y, x+1, y, fill="black")

Also, since I want to establish a rough baseline, I should probably see what happens when I increase the number of iterations from 100 to 1000 for mandel(), which is what I expect to have to use in many cases to get accurate pictures. I do this first using Python 2.5

465659765 function calls in 574.947 CPU seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  3168000  419.296    0.000  561.467    0.000 mandel1a.py:3(mandel)
458828552  100.464    0.000  100.464    0.000 {abs}
  3168001   41.707    0.000   41.707    0.000 {range}
       11   12.731    1.157  574.341   52.213 viewer1.py:23(draw_fractal)
        1    0.596    0.596    0.596    0.596 {_tkinter.create}
   494593    0.140    0.000    0.140    0.000 viewer1.py:17(draw_pixel)
       37    0.010    0.000    0.010    0.000 {built-in method call}
       11    0.000    0.000    0.001    0.000 Tkinter.py:2135(_create)
       54    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
       39    0.000    0.000    0.000    0.000 Tkinter.py:1046(_options)

Ouch! Close to 10 minutes of running time. However, it is clear that I have accomplished my goal of reducing the importance of Tkinter calls so that I can focus on my own code. Let's repeat this profiling test using Python 3.1.

462491974 function calls (462491973 primitive calls) in 506.148 CPU seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  3168000  386.169    0.000  493.823    0.000 mandel1a.py:3(mandel)
458828552  107.654    0.000  107.654    0.000 {built-in method abs}
       11   11.474    1.043  505.439   45.949 viewer1.py:23(draw_fractal)
        1    0.694    0.694    0.694    0.694 {built-in method create}
   494593    0.140    0.000    0.140    0.000 viewer1.py:17(draw_pixel)
       48    0.014    0.000    0.014    0.000 {method 'call' of 'tkapp' objects}
       39    0.000    0.000    0.001    0.000 __init__.py:1032(_options)
       64    0.000    0.000    0.001    0.000 __init__.py:66(_cnfmerge)
      206    0.000    0.000    0.000    0.000 {built-in method isinstance}
      2/1    0.000    0.000  506.148  506.148 {built-in method exec}

We note that the total time taken is significantly less. Doing a comparison function by function, two significant differences appear: The built-in function abs is 7% slower with Python 3.1, which is a bit disappointing. On the other hand, range no longer appears as a function in Python 3.1; this appears to be the main contributor to the significant decrease in time when using Python 3.1 as compared with Python 2.5. This is easily understood: range in Python 3.1 does not create a list like it did in Python 2.x; it is rather like the old xrange. This suggest that I modify mandel1a.py to be as follows:

# mandel1b.py

import sys
if sys.version_info < (3,):
    range = xrange

def mandel(c):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after 20 iterations, the absolute value of the resulting number is
       greater or equal to 2.'''
    z = 0
    for iter in range(0, 20):
        z = z**2 + c
        if abs(z) >= 2:
            return False
    return abs(z) < 2

I also do a similar change to viewer1.py. From now on, except where otherwise noted, I will focus on using only Python 2.5. So, after doing this change, we can run the profiler one more time with the same number of iterations. The result is as follows:

462491765 function calls in 503.926 CPU seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  3168000  391.297    0.000  491.642    0.000 mandel1b.py:7(mandel)
458828552  100.345    0.000  100.345    0.000 {abs}
       11   11.409    1.037  503.189   45.744 viewer1.py:23(draw_fractal)
        1    0.726    0.726    0.726    0.726 {_tkinter.create}
   494593    0.136    0.000    0.136    0.000 viewer1.py:17(draw_pixel)
       37    0.009    0.000    0.009    0.000 {built-in method call}
       11    0.000    0.000    0.001    0.000 Tkinter.py:2135(_create)
       54    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
       39    0.000    0.000    0.000    0.000 Tkinter.py:1046(_options)
       64    0.000    0.000    0.001    0.000 Tkinter.py:77(_cnfmerge)

This is now approximately the same as what we had for Python 3.1 as expected. Moving down on the list of time-consuming functions, we note that abs appears to be another function we should look at. Let's first reduce the number of iterations inside mandel to 100, so that a profiling run does not take as long but that proper attention is still focuses on mandel as well as abs. Here's the result from a typical run to use as a new baseline:

61582475 function calls in 81.998 CPU seconds

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 3168000   55.347    0.000   69.054    0.000 mandel1b.py:7(mandel)
57908682   13.706    0.000   13.706    0.000 {abs}
      11   11.591    1.054   80.773    7.343 viewer1.py:23(draw_fractal)
       1    1.213    1.213    1.213    1.213 {_tkinter.create}
  505173    0.125    0.000    0.125    0.000 viewer1.py:17(draw_pixel)
      37    0.011    0.000    0.011    0.000 {built-in method call}
      11    0.000    0.000    0.001    0.000 Tkinter.py:2135(_create)
       4    0.000    0.000    0.000    0.000 {posix.stat}
       3    0.000    0.000    0.000    0.000 Tkinter.py:1892(_setup)
      39    0.000    0.000    0.000    0.000 Tkinter.py:1046(_options)

Since a fair bit of time is spent inside abs(), perhIaps could speed things up by using another method. The way that we approximate the Mandlebrot set is by iterating over a number of time and checking if the absolute value of the complex number is greater than 2; if it is, then it can be proven that subsequent iterations will yield larger and larger values which means that the number we are considering is not in the Mandelbrot set. Since taking an absolute value of a complex number involves taking a square root, perhaps we can speed things up by not taking the square root. Let's implement this and try it out.

# mandel1c.py

import sys
if sys.version_info < (3,):
    range = xrange

def mandel(c, max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    z = 0
    for iter in range(0, max_iterations):
        z = z**2 + c
        z_sq = z.real**2 + z.imag**2
        if z_sq >= 4:
            return False
    return z_sq < 4

3673150 function calls in 84.807 CPU seconds

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
3168000   73.210    0.000   73.210    0.000 mandel1c.py:7(mandel)
     11   10.855    0.987   84.204    7.655 viewer1.py:23(draw_fractal)
      1    0.593    0.593    0.593    0.593 {_tkinter.create}
 504530    0.137    0.000    0.137    0.000 viewer1.py:17(draw_pixel)
     37    0.009    0.000    0.009    0.000 {built-in method call}
     11    0.000    0.000    0.001    0.000 Tkinter.py:2135(_create)
     54    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
     39    0.000    0.000    0.000    0.000 Tkinter.py:1046(_options)
     64    0.000    0.000    0.001    0.000 Tkinter.py:77(_cnfmerge)
     22    0.000    0.000    0.001    0.000 Tkinter.py:1172(_configure)

The result is worse than before, even though the total number of function calls has almost been cut in half! Actually, this should not come as a total surprise: abs is a Python built-in function, which has been already optimized in C. Extracting the real and imaginary parts explictly like we have done is bound to be a time-consuming operation when performed in pure Python as opposed to C. At this point, we might be tempted to convert complex numbers everywhere into pairs of real numbers so as to reduce the overhead of dealing with complex numbers ... but this would not have any significant effect on the overall time. [Those curious may want to try ... I've done it and it's not worth reporting in details.]

Clearly, I need a different strategy if we are to reduce significantly the execution time. It is time to introduce cython. However, this will have to wait until the next blog post!

Profiling adventures and cython - setting the stage

2010-01-15T22:17:00.003-04:00

Summary This post is the first in a series dedicated to
examining the use of profiling and, eventually, using cython, as a
means to improve greatly the speed of an application. The intended
audience is for programmers who have never done any profiling and/or
never used cython before. Note that we will not make use of cython
until the third post in this series.

Preamble

Python is a great multi-purpose language which is really fun to use. However, it is sometimes too slow for some applications. Since I only program for fun, I had never really faced a situation where I found Python's speed to be truly a limiting factor - at least, not until a few weeks ago when I did some exploration of a four-colouring grid problem I talked about. I started exploring ways to speed things up using only Python and trying to come up with different algorithms, but every one I tried was just too slow. So, I decided it was time to take the plunge and do something different. After considering various alternatives, like using shedskin or
attempting to write a C extension (which I found too daunting since I don't know C), I decided to try to use cython.

cython, for those that don't know it, is a Python look-alike language that claims

to make writing C extensions for the Python language
as easy as Python itself.

After looking at a few examples on the web, I concluded that such a rather bold statement might very well be true and that it was worthwhile trying it out on a more complex example. Furthermore, I thought it might be of interest to record what I've done in a series of blog posts, as a more detailed example than what I had found so far on the web. As I was wondering if an esoteric problem like the four-colouring grid challenge mentioned previously was a good candidate to use as an example, by sheer serendipity, I came accross a link on reddit by a new programmer about his simple Mandelbrot viewer.

Who does not like fractals? ... Furthermore, I have never written a fractal viewer. This seemed like a good time to write one. So, my goal at the end of this series of posts, is to have a "nice" (for some definition of "nice") fractal viewer that is fast enough for explorations of the Mandelbrot set. In addition, in order to make it easy for anyone having Python on their system to follow along and try their own variation, I decided to stick by the following constraints:

With the exception of cython, I will only use modules found in the
standard library. This means using Tkinter for the GUI.

The program should work using Python 2.5+ (including Python 3).

So, without further ado, and based on the example found on the reddit link I mentioned, here's a very basic fractal viewer that can be used as a starting point.

''' mandel1.py

Mandelbrot set drawn in black and white.'''

import time

import sys
if sys.version_info < (3,):
    import Tkinter as tk
else:
    import tkinter as tk

def mandel(c):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after 20 iterations, the absolute value of the resulting number is
       greater or equal to 2.'''
    z = 0
    for iter in range(0, 20):
        z = z**2 + c
        if abs(z) >= 2:
            return False
    return abs(z) < 2

class Viewer(object):
    def __init__(self, parent, width=500, height=500):
        self.canvas = tk.Canvas(parent, width=width, height=height)
        self.width = width
        self.height = height

        # the following "shift" variables are used to center the drawing
        self.shift_x = 0.5*self.width
        self.shift_y = 0.5*self.height
        self.scale = 0.01

        self.canvas.pack()
        self.draw_fractal()

    def draw_pixel(self, x, y):
        '''Simulates drawing a given pixel in black by drawing a black line
           of length equal to one pixel.'''
        self.canvas.create_line(x, y, x+1, y, fill="black")

    def draw_fractal(self):
        '''draws a fractal picture'''
        begin = time.time()
        print("Inside draw_fractal.")
        for x in range(0, self.width):
            real = (x - self.shift_x)*self.scale
            for y in range(0, self.height):
                imag = (y - self.shift_y)*self.scale
                c = complex(real, imag)
                if mandel(c):
                    self.draw_pixel(x, self.height - y)
        print("Time taken for calculating and drawing = %s" %
                                                (time.time() - begin))

if __name__ == "__main__":
    print("Starting...")
    root = tk.Tk()
    app = Viewer(root)
    root.mainloop()

At this point, perhaps a few comments about the program might be useful

I have tried to write the code in the most straightforward and pythonic way, with no thought given to making calculations fast. It should be remembered that this is just a starting point: first we make it work, then, if needed, we make it fast.
The function mandel() is the simplest translation of the Mandelbrot fractal iteration into Python code that I could come up with. The fact that Python has a built-in complex type makes it very easy to implement the standard Mandelbrot set algorithm.
I have taken the maximum number of iterations inside mandel() to be 20, the same value used in the post I mentioned before. According to the very simple method used to time the application, it takes about 2 seconds on my computer to draw a simple picture. This is annoying slow. Furthermore, by looking at the resulting picture, and trying out with different number of iterations in mandel(), it is clear that 20 iterations is not sufficient to adaquately represent the Mandelbrot set; this is especially noticeable when exploring smaller regions of the complex plane. A more realistic value might be to take 100 if not 1000 iterations which takes too long to be practical.
Tkinter's canvas does not have a method to set the colour of individual pixels. We can simulate such a method by drawing a line (for which there is a primitive method) of length 1.
The screen vertical coordinates ("y") increase in values from the top towards the bottom, in opposite direction to the usual vertical coordinates in the complex plane. While the picture produced is vertically symmetric about the x-axis, I nonetheless wrote the code so that the inversion of direction was properly handled.

This basic application is not really useful as a tool for exploring the Mandelbrot set, as the region of the complex plane it displays is fixed. However, it is useful to start with something simple like this as a first prototype. Once we know it is working we can move on to a better second version. So, let's write a fancier fractal viewer following the outline below:

class Viewer(object):
    '''Base class viewer to display fractals'''

        # The viewer should be able to enlarge ("zoom in") various regions
        # of the complex plane.  I will implement this
        # using keyboard shortcuts.
        #
        self.parent.bind("+", self.zoom_in)
        self.parent.bind("-", self.zoom_out)
    def zoom_in(self, event):
    def zoom_out(self, event):
    def change_scale(self, scale):

        #  Since one might want to "zoom in" quickly in some regions,
        # and then be able to do finer scale adjustments,
        # I will use keyboard shortcuts to enable switching back
        # and forth between two possible zooming mode.
        # A better application might give the user more control
        # over the zooming scale.
        self.parent.bind("n", self.normal_zoom)
        self.parent.bind("b", self.bigger_zoom)
    def normal_zoom(self, event, scale=1):
    def bigger_zoom(self, event):

        # Set the maximum number of iterations via a keyboard-triggered event
        self.parent.bind("i", self.set_max_iter)
    def set_max_iter(self, event):

        # Like what is done with google maps and other
        # similar applications, we should be able to move the image
        # to look at various regions of interest in the complex plane.
        # I will implement this using mouse controls.
        self.parent.bind("<button-1>", self.mouse_down)
        self.parent.bind("<button1-motion>", self.mouse_motion)
        self.parent.bind("<button1-buttonrelease>", self.mouse_up)
    def mouse_down(self, event):
    def mouse_motion(self, event):
    def mouse_up(self, event):

        # Presuming that "nice pictures" will be eventually produced,
        # and that it might be desired to reproduce them,
        # I will include some information about the region of the
        # complex plane currently displayed.
    def info(self):
        '''information about fractal location'''

Furthermore, while I plan to use proper profiling tools, I will nonetheless display some basic timing information as part of the GUI as a quick evaluation
of the speed of the application.
Finally, since I expect that both the function mandel() and the drawing method draw_fractal to be the speed-limiting factors, I will leave them out of the fractal viewer and work on them separately. If it turns out that the profiling information obtained indicates otherwise, I will revisit this hypothesis.

Here is a second prototype for my fractal viewer, having the features described above.

''' viewer.py

Base class viewer for fractals.'''

import sys
if sys.version_info < (3,):
    import Tkinter as tk
    import tkSimpleDialog as tk_dialog
else:
    import tkinter as tk
    from tkinter import simpledialog as tk_dialog

class Viewer(object):
    '''Base class viewer to display fractals'''

    def __init__(self, parent, width=600, height=480,
                 min_x=-2.5, min_y=-1.5, max_x=1.):

        self.parent = parent
        self.canvas_width = width
        self.canvas_height = height

        # The following are drawing boundaries in the complex plane
        self.min_x = min_x
        self.min_y = min_y
        self.max_x = max_x
        self.calculate_pixel_size()
        self.max_y = self.min_y + self.canvas_height*self.pixel_size

        self.calculating = False
        self.nb_iterations = 20
        self.normal_zoom(None)

        self.canvas = tk.Canvas(parent, width=width, height=height)
        self.canvas.pack()
        self.status = tk.Label(self.parent, text="", bd=1, relief=tk.SUNKEN,
                               anchor=tk.W)
        self.status.pack(side=tk.BOTTOM, fill=tk.X)
        self.status2 = tk.Label(self.parent, text=self.info(), bd=1,
                                relief=tk.SUNKEN, anchor=tk.W)
        self.status2.pack(side=tk.BOTTOM, fill=tk.X)

        # We change the size of the image using the keyboard.
        self.parent.bind("+", self.zoom_in)
        self.parent.bind("-", self.zoom_out)
        self.parent.bind("n", self.normal_zoom)
        self.parent.bind("b", self.bigger_zoom)

        # Set the maximum number of iterations via a keyboard-triggered event
        self.parent.bind("i", self.set_max_iter)

        # We move the canvas using the mouse.
        self.translation_line = None
        self.parent.bind("<button-1>", self.mouse_down)
        self.parent.bind("<button1-motion>", self.mouse_motion)
        self.parent.bind("<button1-buttonrelease>", self.mouse_up)

        self.draw_fractal()  # Needs to be implemented by subclass

    def info(self):
        '''information about fractal location'''
        return "Location: (%f, %f) to (%f, %f)" %(self.min_x, self.min_y,
                                                  self.max_x, self.max_y)

    def calculate_pixel_size(self):
        '''Calculates the size of a (square) pixel in complex plane
        coordinates based on the canvas_width.'''
        self.pixel_size = 1.*(self.max_x - self.min_x)/self.canvas_width
        return

    def mouse_down(self, event):
        '''records the x and y positions of the mouse when the left button
           is clicked.'''
        self.start_x = self.canvas.canvasx(event.x)
        self.start_y = self.canvas.canvasy(event.y)

    def mouse_motion(self, event):
        '''keep track of the mouse motion by drawing a line from its
           starting point to the current point.'''
        x = self.canvas.canvasx(event.x)
        y = self.canvas.canvasy(event.y)

        if (self.start_x != event.x)  and (self.start_y != event.y) :
            self.canvas.delete(self.translation_line)
            self.translation_line = self.canvas.create_line(
                                self.start_x, self.start_y, x, y, fill="orange")
            self.canvas.update_idletasks()

    def mouse_up(self, event):
        '''Moves the canvas based on the mouse motion'''
        dx = (self.start_x - event.x)*self.pixel_size
        dy = (self.start_y - event.y)*self.pixel_size
        self.min_x += dx
        self.max_x += dx
        # y-coordinate in complex plane run in opposite direction from
        # screen coordinates
        self.min_y -= dy
        self.max_y -= dy
        self.canvas.delete(self.translation_line)
        self.status.config(text="Moving the fractal.  Please wait.")
        self.status.update_idletasks()
        self.status2.config(text=self.info())
        self.status2.update_idletasks()
        self.draw_fractal()

    def normal_zoom(self, event, scale=1):
        '''Sets the zooming in/out scale to its normal value'''
        if scale==1:
            self.zoom_info = "[normal zoom]"
        else:
            self.zoom_info = "[faster zoom]"
        if event is not None:
            self.status.config(text=self.zoom_info)
            self.status.update_idletasks()
        self.zoom_in_scale = 0.1
        self.zoom_out_scale = -0.125

    def bigger_zoom(self, event):
        '''Increases the zooming in/out scale from its normal value'''
        self.normal_zoom(event, scale=3)
        self.zoom_in_scale = 0.3
        self.zoom_out_scale = -0.4

    def zoom_in(self, event):
        '''decreases the size of the region of the complex plane displayed'''
        if self.calculating:
            return
        self.status.config(text="Zooming in.  Please wait.")
        self.status.update_idletasks()
        self.change_scale(self.zoom_in_scale)

    def zoom_out(self, event):
        '''increases the size of the region of the complex plane displayed'''
        if self.calculating:
            return
        self.status.config(text="Zooming out.  Please wait.")
        self.status.update_idletasks()
        self.change_scale(self.zoom_out_scale)

    def change_scale(self, scale):
        '''changes the size of the region of the complex plane displayed and
           redraws'''
        if self.calculating:
            return
        dx = (self.max_x - self.min_x)*scale
        dy = (self.max_y - self.min_y)*scale
        self.min_x += dx
        self.max_x -= dx
        self.min_y += dy
        self.max_y -= dy
        self.calculate_pixel_size()
        self.draw_fractal()

    def set_max_iter(self, event):
        '''set maximum number of iterations'''
        i = tk_dialog.askinteger('title', 'prompt')
        if i is not None:
            self.nb_iterations = i
            self.status.config(text="Redrawing.  Please wait.")
            self.status.update_idletasks()
            self.draw_fractal()

    def draw_fractal(self):
        '''draws a fractal on the canvas'''
        raise NotImplementedError

I move the Mandelbrot set calculation in a separate file.

# mandel1a.py

def mandel(c, max_iterations=20):
    '''determines if a point is in the Mandelbrot set based on deciding if,
       after a maximum allowed number of iterations, the absolute value of
       the resulting number is greater or equal to 2.'''
    z = 0
    for iter in range(0, max_iterations):
        z = z**2 + c
        if abs(z) >= 2:
            return False
    return abs(z) < 2

And, finally, I implement the missing functions for the viewer in a new main application.

# viewer1.py

from mandel1a import mandel
from viewer import Viewer
import time

import sys
if sys.version_info &lt; (3,):
    import Tkinter as tk
    range = xrange
else:
    import tkinter as tk

class FancyViewer(Viewer):
    '''Application to display fractals'''

    def draw_pixel(self, x, y):
        '''Simulates drawing a given pixel in black by drawing a black line
           of length equal to one pixel.'''
        self.canvas.create_line(x, y, x+1, y, fill="black")

    def draw_fractal(self):
        '''draws a fractal on the canvas'''
        self.calculating = True
        begin = time.time()
        # clear the canvas
        self.canvas.create_rectangle(0, 0, self.canvas_width,
                                    self.canvas_height, fill="white")
        for x in range(0, self.canvas_width):
            real = self.min_x + x*self.pixel_size
            for y in range(0, self.canvas_height):
                imag = self.min_y + y*self.pixel_size
                c = complex(real, imag)
                if mandel(c, self.nb_iterations):
                    self.draw_pixel(x, self.canvas_height - y)
        self.status.config(text="Time required = %.2f s  [%s iterations]  %s" %(
                                (time.time() - begin), self.nb_iterations,
                                                                self.zoom_info))
        self.status2.config(text=self.info())
        self.status2.update_idletasks()
        self.calculating = False

if __name__ == "__main__":
    root = tk.Tk()
    app = FancyViewer(root)
    root.mainloop()

Let me conclude with few black and white pictures obtained using this program, which, if you look at the time, highlight the need for something faster. First for 20 iterations, drawn in 4 seconds.

Then, for 100 interations - better image, but 7 seconds to draw...

Next post, I'll start profiling the application and make it faster.

Python + cython: faster than C?

2010-01-10T17:39:00.000-04:00

Now that I have your attention... ;-)

I've been playing around with cython and will write more about it soon. But I could not resist doing a quick test when I read this post, comparing C and Java on some toy micro benchmark.

First, the result:

andre$ gcc -o fib -O3 fib.c
andre$ time ./fib
433494437

real    0m3.765s
user    0m3.463s
sys     0m0.028s

andre$ time python fib_test.py
433494437

real    0m2.953s
user    0m2.452s
sys     0m0.380s

Yes, Python+cython is faster than C! :-)

Now, the code. First, the C program taken straight from this post, except for a different, more meaningful value for "num" ;-)

#include 

double fib(int num)
{
   if (num <= 1)
       return 1;
   else
       return fib(num-1) + fib(num-2);
 }
 int main(void)
{
     printf("%.0f\n", fib(42));
    return 0;
 }

Next, the cython code ...

# fib.pyx

cdef inline int fib(int num):
    if (num <= 1):
        return 1
    else:
        return fib(num-1) + fib(num-2)

def fib_import (int num):
    return fib(num)

... and the Python file that calls it

# fib_test.py

import pyximport
pyximport.install()

from fib import fib_import

print fib_import(42)

I know, I declared fib() to be of type "double" in the C code (like what was done in the original post) and "int" in the cython code; however, if I declare it to be of type "int" in the C code, I get -0 as the answer instead of 433494437. I could declare it to be of type "unsigned int" ... but even when I did that, the Python code was marginally faster.
If I declared fib to be of type "double" in the cython code, it was slightly slower than the corresponding C code. However, what Python user would ever think of an integer variable as a floating point number! ;-)
What is most impressive is that the cython code is NOT pre-compiled, unlike the C code. However, to be fair ... I did run it a few times and took the best result, and it was always slower the first time.
Yes, it is silly to compare such micro-benchmarks ... but it can be fun, no? ;-)

More seriously: using cython is a relatively painless way to significantly increase the speed of Python applications ... without having to learn a totally different language.

Rur-ple is alive again!

2010-01-02T21:24:00.000-04:00

Thanks to some wonderful work by some developers who joined the project, rur-ple is moving forward again. :-) Its new home also features a brand new logo for it displayed above.

Thanks in particular go to Frederic Muller who is moving things along rather nicely.

The 17x17 challenge: setting up the grid

2009-12-25T10:37:00.000-04:00

In the previous post, I talked about the 17x17 challenge, but did not provide any code. Today, I will describe the code I use to represent a given grid. Consider the following grid where A and b are used to represent two different colours:

AbAb
bAbA
bbAA

bAAA

This grid has two rectangles, one of which is emphasized below

....
.A.A
....
.A.A

Finding such a rectangle in a small grid is fairly easy to do. A naive, but very inefficient way to do this is to have a function like the following:

def find_rect_candidates(row_1, row_2):
   points_in_common = [0, 0, ...] # one per colour
    for i, point in enumerate(row_1):
       if row_1[i] == row_2[i]:
          points_in_common[row_1[i]] += 1
   return points_in_common

where row_x[i] is set to a given colour, represented by an integer (0, 1, 2, ...). For an NxN grid, the first loop is of order N. Note that this does not tell us (yet) if we have found a rectangle; we still have to loop over points_in_common and see if any entry is greater than 1.

A better way to do the comparison, which does not grow with N, is mentioned in this post and is based on the following observation: for a given row, at a given point, a given colour is either present (True or 1) or not (False or 0). Thus, a given colour distribution on a single row can be represented as a string of 0s and 1s ... which can be thought of as the binary representation of an integer. For example, the row containing the colour "A" in the pattern ".....A..A" can have this pattern represented by the number 9="1001". Consider another row represented by the number 6="110". These two rows have no points in common (for this colour) and hence do not form a rectangle. If we do a bitwise "and" for these two rows, i.e. 9&6 we will get zero. This is achieved by a single operation instead of a series of comparisons.

What happens if two rows have a single point in common (for a given colour) so that no rectangle is present? For example, consider 9="1001" and 15="1110". If we do a bitwise "and" we have

answer = 9 & 15 = 8 = "1000"

Taking the bitwise "and" of answer and answer-1, we get

answer & (answer-1) = 8 & 7 = "1000" & "111" = 0.

If we have two points (1 bit) in common, it is easy to see that the bitwise comparison of answer & (answer-1) will not give zero.

Going back to the function above, we could rewrite it instead as follows:

def find_rect_candidates(row_1, row_2, colours):
   points_in_common = [0, 0, ...] # one per colour
      for c in colours:
         points_in_common[c] = row_1[c] & row_2[c]
   return points_in_common

where we still have to do the same (N-independent) processing of the return value as with the function above to determine if we do have rectangles.

Now, without further ado, here is the basic code that I use to represent a grid, together with two utility functions.

def rectangles_for_two_rows(row_1, row_2):
    '''returns 0 if two rows (given colour, encoded as a bit string)
        do not form a rectangle -
        otherwise returns the points in common encoded as a bit string'''
    intersect = row_1 & row_2
    if not intersect:   # no point in common
        return 0
    # perhaps there is only one point in common of the form 00010000...
    if not(intersect & (intersect-1)):
        return 0
    else:
        return intersect

def count_bits(x):
    '''counts the number of bits
    see http://stackoverflow.com/questions/407587/python-set-bits-count-popcount
    for reference and other alternative'''
    return bin(x).count('1')

class AbstractGrid(object):
    def __init__(self, nb_colours, max_rows, max_columns):
        self.nb_colours = nb_colours
        self.colours = list(range(nb_colours))
        self.max_rows = max_rows
        self.max_columns = max_columns
        self.powers_of_two = [2**i for i in range(max_columns)]
        self.grid = {}

    def initialise(self):
        '''initialise a grid according to some strategy'''
        raise NotImplemented

    def print_grid(self):
        '''prints a representation of the grid.
        Used for diagnostic only - no need to optimize further.'''
        for row in self.grid:
            row_rep = []
            for column in range(self.max_columns-1, -1, -1):
                for colour_ in self.colours:
                    if self.powers_of_two[column] & self.grid[row][colour_]:
                        row_rep.append(str(colour_))
            print("".join(row_rep))

    def identify_intersect_points(self):
        '''identifies the dots that contribute to forming rectangles'''
        # possibly could cut down the calculation time by only computing
        # for colours that have changed...
        nb_intersect = [0 for colour in self.colours]
        intersect_info = []
        for colour in self.colours:
            for row in self.grid:
                for other_row in range(row+1, self.max_rows):
                    intersect = rectangles_for_two_rows(
                                            self.grid[row][colour],
                                            self.grid[other_row][colour])
                    if intersect != 0:
                        nb_intersect[colour] += count_bits(intersect)
                        intersect_info.append([colour, row, other_row, intersect])
        return nb_intersect, intersect_info

The actual code I use has a few additional methods introduced for convenience. If anyone can find a better method to identify intersection points between rows (from which rectangles can be formed), I would be very interested to hear about it.

The 17x17 challenge

2009-12-25T00:55:00.002-04:00

In a recent blog post, William Gasarch issued a challenge: find a 17x17 four-colour grid for which there are no rectangles with 4 corners of the same colour. If you can do this, Gasarch will give you $289. For a more detailed explanation, you can either read Gasarch's post, or this post which contains a slightly friendlier explanation of the problem.

This problem, which is fairly easy to state, is extremely hard to solve as the number of possible grid configurations is 4²⁸⁹ which is a number much too large to solve by random searches or by naive methods. As can be seen from the comments of the posts mentioned above, many people have devoted a lot of cpu cycles in an attempt to find a solution, but with no success. Like others, I have tried to write programs to solve it ... but with no luck. In a future post, I may write about the various strategies I have implemented in Python in an attempt to solve this problem.

Gasarch mentions that it is not known if a solution can be found for a 17x18 four-colour grid or even an 18x18 grid or a 17x19 one. (Note that because of symmetry, an MxN solution can be rotated to give an NxM solution.) ~~While the number of configurations is large, the larger the grid is, I believe I have found a proof demonstrating that no solution can be found for a 17x18 grid.~~ This ~~proof~~ builds on the work of Elizabeth Kupin who has found the largest "colour class" i.e. the number of single-colour points on a 17x17 grid which is rectangle-free. A generic solution found by Kupin is reproduced below:

Consider a 17x18 grid (17 rows and 18 columns). Such a grid has 306 points. Using the pigeonhole principle, one can easily show that, for any colouring, at least one colour must cover 77 points. (Indeed, the most symmetric colouring would be 77, 77, 76, 76.) Also, if a 17x18 rectangle-free solution exist, it must contain a 17x17 rectangle-free subset, which we take to be the above solution (other solutions for the 17x17 grid can be derived from this one using interchanges of rows and/or columns).

Let us attempt to add points in an additional column. First, we can try to add points in a row with 4 elements. Without loss of generality, we can take this row to be the first one (row A). Once we do this, we can not add any more points to row 3-17 without creating a rectangle. The only row to which we can add a point is the second (row B) bringing the total number of points to 76 - one short of what we need.

Perhaps we can first remove a point from the 17x17 solution and then add a new column. There are three cases we must consider: 1) that of a point belonging to a row with 5 points and a column with 5 points; 2) that of a point belonging to a row with 5 points and a column with 4 points; 3) that of a point belonging to a row with 4 points and a column with 5 points.

Case 1) Without loss of generality, let us move the point on row F in the first column to a new additional column, keeping the number of points at 74. It is easy to show that the only rows to which we can then add an additional point without creating a rectangle are rows A, C, D, E. Once we add such a point (say on row A), we can no longer add a point on any of the remaining rows (C, D, E) without creating a rectangle.

Case 2) Without loss of generality, let us move the left-most point on row Q to a new column. The only row to which we can then add a point in this new column is row A, bringing the total to 75. We can't add another point without creating a rectangle.

Case 3) Again, without loss of generality, let us remove the top-left element (A) and move it to a new column added to the above solution. We can add one more point, bringing the total to 75, to that new column (in row B) without creating a rectangle; any other addition of a point on that new column will result in a rectangle.

~~This concludes the sketch of the proof.~~

UPDATE: Instead of adding a column, we can add a row (R) with 3 points located in the 9th, 12th and 17th column, bringing the total to 77 points and keeping the grid rectangle-free. So the 17x18 case is still open... but I can't see how one could add a row with an extra 4 points to build a "single-colour" rectangle-free 17x19 grid. (However, I won't venture again to say it is a proof until I have looked at all cases exhaustively.)

Note that construction of solutions for a 17x17 grid such as those found by Kupin, even for a single colours, can possibly be used to restrict the search space for the more general problem and help find a solution. Unfortunately, no one has been able to do this (yet).

Google Wave Fun

2009-11-15T15:45:00.000-04:00

After a long time waiting, I finally got an Google Wave invitation this week - just as I was starting to look forward to a work slowdown so I could start programming again in the evenings. The call of playing with a new toy was enough motivation to overcome inertia and spend a few hours, here and there in the evening, in order to create a new "gadget". Google Wave Gadgets are actually fairly easy to create. The only downside (from my point of view) is that they have to be written using javascript instead of Python. Since this blog is called "Only Python", you may wonder how a post about a Google Wave Gadget could possibly belong here. Believe me, it does.

A few years ago, in order to learn programming in Python, I created a "Karel the Robot" application in Python, to teach/learn Python, called rur-ple. rur-ple was actually inspired by Guido van Robot (or GvR) which uses a Python like language.

I thought it would be a good idea to have something like rur-ple as a wave gadget. Rather than re-inventing the wheel, I contacted Paul Carduner, who wrote gvr-online, and suggested that we could collaborate on a gadget - and he agreed. The end goal is to have something like gvr-online as a gadget, but one which would recognize both the traditional GvR syntax OR the Python syntax used with rur-ple, as a choice for the end user.

The idea of having something like gvr-online as a wave gadget is the following: a teacher can embed such a gadget in a wave, with a predefined world, and assign it as a problem to either one, or more students - who could collaborate in that wave. All wave participants, including the teacher, can thus contribute and help each other.

Having had the idea, I wrote what was basically an empty shell of a gadget which Paul quickly connected to the existing code for gvr-online. Some minor work remained for me to do to enable shared states and, apart from a minor UI bug, the prototype is now working. What's left to do includes incorporation the Python syntax as an alternative to the traditional GvR syntax.

For those that might be interested in such a gadget, I suggest you first check gvr-online itself to get a better idea of what it does. Then, if you have a Google Wave account, you can embed the gadget in a wave using this link. Note that this is our "work in progress" link, that should not be expected to be a permanent one.

Live preview of reStructuredText to HTML conversion in your browser

2009-09-13T17:21:00.000-03:00

I've implemented a new plugin for Crunchy which may be of interest to some: a reStructuredText editor with live preview inside the browser window. Each time a key is pressed in the editor (html textarea), an ajax request is sent to the Python backend with the entire content of the editor. This is processed by docutils with the result sent back to the browser as a full html page displayed inside the browser as illustrated below.

I expected this to be slow (make that very slow) and for the first prototype I wrote, I only sent the ajax request when "enter" was pressed. After playing with it for a while, I realized that there was no need for this.

If you type fast enough, the previewer does not keep up for each key press, but this does not hinder in any way the speed at which you can enter text in the editor. By the way, the editor (html textarea) can be converted into a true embedded editor which can save and load files, etc. - otherwise, it would not be a very useful tool for working with reStructuredText documents. When doing so, the live preview is turned off. I've also included the possibility to save the html file produced so that one does not have to invoke docutils separately. And this also incorporates the two custom directives (crunchy and getpythonsource) used by Crunchy and crst2s5.

This is included in the newest release (1.1.2) of Crunchy.

Something I am working on...

2009-09-03T14:08:00.000-03:00

Coming soon ... with "live" preview ...

Upcoming Pycon and crst2s5

2009-09-02T20:15:00.000-03:00

In a few months, many pythonistas will be converging to Atlanta to take part in the next Pycon. I'm hoping to be able to attend it as I have been really inspired by the many people I met in two previous conferences. While I most likely won't be presenting anything this time, I thought I would try to contribute to some presentations in an indirect way.

A while ago, Ned Batchelder wrote about presentation tools - more specifically about how he found all of them lacking. In the comments section, many people made various suggestions - including one suggestion by Steve Holden which read as follows:

Crunchy? http://code.google.com/p/crunchy/

HTML plus live Python - what could be better?

While I, of course, approve of any positive support of Crunchy, I had to admit to myself (and mention in a subsequent comment) that Crunchy was really not up to the task - at least not yet.

In his post, Ned addresses some of the weaknesses of the tools he looked at. In particular, he mentioned S5 about which he wrote:

S5 (A Simple Standards-Based Slide Show System) is all HTML, CSS, and Javascript, runs in the browser, and was created by Eric Meyer, a very nice pedigree. Remarkably, the thing I like least about it is that when I display the slides in a full-screen Firefox, they look horrible. The text is either too small or too large, and the line-spacing too tight, while the slide title overlaps the text in the wrong way, exposing some of the structure of the divs.
I would have hoped that a CSS-based slideshow by the king of CSS would be a shining example of how information could be cleanly authored and then sparklingly displayed. S5 seems to miss this mark, especially since there don't seem to be many themes available for it, another surprise given how CSS should have made it accessible to lots of designers. Also, although (or perhaps because) the format is native to the web, it's not possible to get the slides as illustrations.

Still, S5 is used for quite a few presentations by pythonistas thank to rst2s5, the docutils script that transform reStructuredText files into S5 presentations. If I can summarize, rst2s5 strengths are as follow:

uses reStructuredText as source - easier to write than html

produce html files - thus easy to post and share -see link above.

use well known S5 slides (version 1.1 designed by Eric Meyer) to switch between "full paper"/slides. It is possible to include extra notes in the "full paper".

Some of the weaknesses are the following:

From rst2s5's own documentation, it is impossible to have:

     - speaker's notes on the laptop screen
     - slides on the projector
     - two views stay in sync
     - presentation controlled from either display

like most (all?) slide-based presentation, content on one slide is limited to a given number of lines ... which makes it ackward to show long-ish code samples.
as Ned Batchelder alludes to, the automatic scaling done by S5 often yields unsatisfactory results in terms of line spacing, etc.
Finally, no interactive Python code in presentation ;-)

rst2s5 is based on S5 version 1.1. A new version (1.2 alpha) is available which addresses some of the shortcomings of version 1.1 with the addition of speaker's notes which are in a separate window (e.g. on the laptop screen) while the slides themselves are in a different window (which can be on the projector) with the two synchronized.

I have decided to hack at the source (I don't like javascript...) and adapt S5 1.2 in ways which address some of Ned Batchelder's criticisms about the line spacing, etc, by disabling the automatic font scaling. I incorporated it in a new program called crst2s5 which can be thought of as a Crunchy-extension to rst2s5. crst2s5 on its own can produce standalone presentations (like rst2s5). I consider it to be an improvement to rst2s5 ... time will tell if others share my opinion.

However, where crst2s5 can yield really interesting results is when you view the html output with Crunchy. For now (version 1.1), when you do so, the automatic synchronization of speaker notes and slides is disabled - this will be addressed hopefully in the near future. You can either try it by downloading the code or have a look at this sneak preview which I put together in one quick take as an experiment:

Get the Flash Player to see this movie.

Original location: 'Crunchy'
at ShowMeDo from the Python category.

Now, if only I had better graphical skills to design good looking slides...

New plugin for Crunchy ... and bug fixes

2009-08-23T21:56:00.003-03:00

Crunchy has a new plugin: getsource. What it does is enable a tutorial writer to embed a "link" to a python module inside an html file, or a class within that module, or a function or method, and have the source code being extracted by the inspect module and inserted within the html page.

Crunchy being Crunchy, it can also embed an interpreter or an editor right below the code source so that a user can interact with it.

And since not everyone likes to write documentation using straight html, a custom docutils directive is supported so that it works from reStructuredText files too.

The docutils directive looks as follows:

..getsource:: relative/path/to/module[.function] [linenumber] [editor or interpreter]

with a similar syntax for html files.

This plugin (which still has some minor bugs) is included in release 1.0.1 of Crunchy, which contains other minor bug fixes (as compared with release 1.0).

Crunchy 1.0 released!!

2009-08-23T01:24:00.002-03:00

Crunchy 1.0 has finally been released. :-)

Crunchy 1.0 is compatible with Python 2.4, 2.5, 2.6 ... and 3.1. It is also compatible with Jython 2.5 modulo some bugs when trying to work with examples containing unicode strings.

Crunchy, for those that are not familiar with it, is a one-of-a-kind application. It is an application designed to transforms otherwise static html Python tutorials into interactive session viewed within a browser. Currently Crunchy has only been fully tested with Firefox. Crunchy should work with all operating systems - it has been tested fairly extensively on Linux, Windows and Mac OS.

In more details, here's what Crunchy does:

1. It loads an existing html file (or reStructuredText file) containing some Python code; this file can reside locally, or anywhere on the web.
2. It removes any existing javascript code & embedded interactive elements (applets, flash apps, etc.).
3. It further transforms the file by inserting some extra elements, including some custom javascript code.
4. It sends the transformed file to your browser of choice (read: Firefox).
5. It establishes a communication between the browser and the Crunchy back end and wait for user interaction.

Crunchy can embed Python interpreters and code editors (and more!) within an html page, enabling a user to enter (or edit) some Python code inside a browser window, send it to the Crunchy back end for execution, and observe the result back in the browser window.

In a sense, Crunchy is a Python programming environment embedded within a browser. However, it does not require a custom format for interactivity: for html files, as long as the Python code is included in a pre-formatted element (<pre>), Crunchy will recognize it, style it, and include the appropriate interactive element.

Crunchy comes with a fairly complete tutorial and supporting documentation. It is highly configurable.

Release 1.0 is NOT the end of the road for Crunchy. Future plans include support for interactivity with languages other than Python.

Python textbooks wanted

2009-04-16T17:41:00.002-03:00

Python textbooks wanted(1). Actually, what I'm mostly interested in are links to textbooks and other resources that would be useful to educators either teaching Python as a programming language or other courses where Python feature prominently. The reason behind this request is that I volunteered to be responsible to take care of the edu-sig page on the Python site. Through no one's fault in particular, the old page (2) had gone stale and only one textbook was listed. We're now up to six and I am sure I have not included them all. By ensuring that up to date and complete information is available on that page, we can facilitate the adoption of Python as a language taught in High Schools, Colleges and Universities.

So, if you know of any useful information that should be added to the edu-sig page, please let me know.

(1) Actually, this is not exactly correct ... but if you want to send me some for reviews, I won't say no. ;-)

(2) The page linked here will eventually disappear...

Learning paths at ShowMeDo

2009-04-10T14:47:00.003-03:00

ShowMeDo is a great site to learn about Python (and a few other subjects) by watching screncasts. One of my favourite videos is Learn Django: Create a Wiki in 20 minutes, which taught me the basic concepts so that I was later able to adapt and play around with Google App Engine. Many authors on ShowMeDo have multiple screencasts (I made a few on RUR-PLE and Crunchy a while ago), some of which (not mine) are very professional looking. Such series of videos give the viewpoint of a given author, introducing some concepts in a logical sequence.

The newest addition to the ShowMeDo site is called Learning Paths. The idea of Learning Paths is to improve upon the existing series by including videos from multiple authors into a coherent sequence. While this concept is very much in its infancy, it promises to become a great addition to what is already a fantastic site.

In case anyone were wondering: I have no financial interest whatsover in ShowMeDo; I am just a satisfied user who considers this site a very good resource, well worth exploring. If only I had more time...

Testing code highlighting

2009-01-08T22:32:00.004-04:00

Update:

In order to have the styling work, I had to include by hand the styling information (keyword and string) from this style sheet - and adapt as desired.

- - - - - - -
This is a test of Python code highlighting on Blogger following these instructions.

print "Hello World!"

# And this is a comment

End of test.

Plugins - part 6: setuptools based approach

2008-12-20T23:44:00.007-04:00

setuptools is collection of enhancements to the standard Python distutils module. While it is not included in the Python standard library, chances are that it is already installed on your system if you have installed some additional Python libraries since it is so widely used to install Python packages ("eggs") via easy_install.

setuptools is also used in many applications as a handler for plugins. Many such applications include tutorials for creating new plugins using setuptools. For a somewhat general introduction to using setuptools to create plugin based applications, I suggest you have a look at the two tutorials from which this one is inspired.

Our starting point for this tutorial is essentially the same as in the third one in this series. We start with the following files:


root_directory/
    calculator.py
    setup.py
    plugins/
        __init__.py
        base.py
        op_1.py
        op_2.py

where we have one new file, setup.py. This file is a special file for setuptools. Its content is as follows:


''' run with python setup.py develop '''

from setuptools import setup, find_packages

setup(
    name="Calculator_s_tools",
    version="1.0",
    packages=['plugins'],
    entry_points="""
        [plugin_tutorial.s_tools]
        add = plugins.op_1:operator_add_token
        sub = plugins.op_1:operator_sub_token
        mul = plugins.op_1:operator_mul_token
        div = plugins.op_1:operator_div_token
        pow = plugins.op_2:operator_pow_token"""
       )

The key concept for setuptools is that of "entry_points". We define some entry points, with a name "plugin_tutorial.s_tools" chosen as unique to our application. Within this entrypoint we indicate which classes should be imported. This method effectively replace our custom method for finding and loading plugins. However, if you remember from previous tutorials, the way the application was designed originally (all within a single file) resulted in a "wart", where we had to create a link to a function ("expression") in each plugin module. Since setuptools will import the classes for us, we have no way to tell it how to fix that "wart" - we have to find another way. The method we chose was to create a different Plugin base class, one that implements the Borg idiom, so that each instance shares common attributes. Here's the new "base.py":


import os
import sys
import pkg_resources  # setuptools specific

OPERATORS = {}
ENTRYPOINT = 'plugin_tutorial.s_tools'  # same name as in setup.py

class Plugin(object):
    '''A Borg class.'''
    _shared_states = {}
    def __init__(self):
         self.__dict__ = self._shared_states

def init_plugins(expression):
    '''simple plugin initializer'''
    Plugin().expression = expression  # fixing the wart
    load_plugins()

def load_plugins():
    '''setuptools based plugin loader'''
    for entrypoint in pkg_resources.iter_entry_points(ENTRYPOINT):
        plugin_class = entrypoint.load()
        OPERATORS[plugin_class.symbol] = plugin_class

The actual plugin files are slightly modified to derive from the new Plugin class; for example, op_2.py contains the following:


from plugins.base import Plugin

class operator_pow_token(Plugin):
    symbol = '**'
    lbp = 30
    def led(self, left):
        return left ** self.expression(30-1)

where "expression" is now a class variable obtained from Plugin.

The code involved to make the setuptools approach is approximately the same level of complexity as the class-based plugin system covered previously. The advantages of using the setuptools approach are as follows:

Since it is a widely used tool, many people know how to use it properly.
It is possible to package plugins as eggs to be uploaded to a repository.
It is possible to keep track of dependencies in a fairly detailed way (e.g. module X version Y required).

By comparison, it suffers from the following disadvantages:

Some information about plugin location (entry_points name) is duplicated, appearing in both setup.py and base.py (in our example).
Automatic plugin discovery without editing of a file (setup.py) is not possible, unlike the cases we covered before. Because of this, dynamic loading of "external" plugins while the application is already running may be problematic to achieve. (I am not familiar enough with setuptools to determine if it is feasible or not.) See the first comment par Phillip J. Eby on how to achieve this.
A preliminary step ("python setup.py develop") is required to generate entrypoints information.
A number of additional files are created by the previous step, "cluttering" slightly the file structure by adding an extra directory with a few files.

That being said, the differences between the two approaches are relatively minor when everything is taken into account. Choosing one approach over the other is a matter of individual taste - at least for simple applications such as the one we considered.

Plugins - part 5: Activation and Deactivation

2008-12-20T15:18:00.005-04:00

(Note: the code indentation may appear to be wrong due to some blogspot's quirks...)

While plugins are a great way to extend the functionality of an application, sometimes it makes sense to limit the number of available features, based on a user's preference. For example, gedit, the official text editor of the Gnome environment, offers the possibility to activate or deactivate a plugin.

In this post, using the class-based plugin approach, I will explain how to add the possibility to activate or deactivate a given plugin. Furthermore, I will show how to use this capability to dynamically load new plugins.

Starting from the beginning...

Our starting point will be the following modified core application (calculator.py):

import re

from plugins.base import OPERATORS, init_plugins, activate, deactivate

class literal_token(object):
 def __init__(self, value):
     self.value = value
 def nud(self):
     return self.value

class end_token(object):
lbp = 0

def tokenize(program):
 for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
    if number:
        yield literal_token(int(number))
    elif operator in OPERATORS:
        yield OPERATORS[operator]()
    else:
        raise SyntaxError("unknown operator: %r" % operator)
 yield end_token()

def expression(rbp=0):
 global token
 t = token
 token = next()
 left = t.nud()
 while rbp < token.lbp:
     t = token
     token = next()
     left = t.led(left)
 return left

def calculate(program):
 global token, next
 next = tokenize(program).next
 token = next()
 return expression()

if __name__ == "__main__":
init_plugins(expression)
assert calculate("+1") == 1
assert calculate("-1") == -1
assert calculate("10") == 10
assert calculate("1+2") == 3
assert calculate("1+2+3") == 6
assert calculate("1+2-3") == 0
assert calculate("1+2*3") == 7
assert calculate("1*2+3") == 5
assert calculate("6*2/3") == 4

    # "**" has not been activated at the start in base.py
    try:
        assert calculate("2**3") == 8
    except SyntaxError:
        print "Correcting error..."
        activate("**")
    assert calculate("2*2**3") == 16

    deactivate('+')
    try:
        assert calculate("1+2") == 3
    except SyntaxError:
        activate('+')
    assert calculate("1+2") == 3

print "Done!"

The new features are indicated by different colours. In blue, we have two new functions imported to either activate or deactivate a given plugin. When the application is started, exponentiation is disabled - this can only be seen by looking at the modified version of base.py. When a disabled plugin is called, a SyntaxError already present in the old version) is raised and we activate the plugin.

To make this possible, we need to modify base.py. Before showing the new version, here's the result of running the above code:

Activating +
Activating -
Activating *
Activating /
Correcting error...
Activating **
Deactivating +
Activating +
Done!

And here's the new version of base.py:


import os
import sys

OPERATORS = {}

# We simulate a configuration file that would be based on a user's preference
# as to which plugin should be activated by default
# We will leave one symbol "**" out of the list as a test.
preferences = ['+', '-', '*', '/']

# We also keep track of all available plugins, activated or not
all_plugins = {}

class Plugin(object):
'''base class for all plugins'''

    def activate(self):
        '''activate a given plugin'''
        if self.symbol not in OPERATORS:
            print "Activating %s" % self.symbol
            OPERATORS[self.symbol] = self.__class__
        if self.symbol not in all_plugins:
            all_plugins[self.symbol] = self.__class__

    def deactivate(self):
        '''deactivate a given plugin'''
        print "Deactivating %s" % self.symbol
        if self.symbol in OPERATORS:
            del OPERATORS[self.symbol]

def activate(symbol):
    '''activate a given plugin based on its symbol'''
    if symbol in OPERATORS:
        return
    all_plugins[symbol]().activate()

def deactivate(symbol):
    '''deactivate a given plugin, based on its symbol'''
    if symbol not in OPERATORS:
        return
    all_plugins[symbol]().deactivate()

def init_plugins(expression):
'''simple plugin initializer
'''
find_plugins(expression)
register_plugins()

def find_plugins(expression):
'''find all files in the plugin directory and imports them'''
plugin_dir = os.path.dirname(os.path.realpath(__file__))
plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
sys.path.insert(0, plugin_dir)
for plugin in plugin_files:
 mod = __import__(plugin)
 mod.expression = expression

def register_plugins():
'''Register all class based plugins.

Uses the fact that a class knows about all of its subclasses
to automatically initialize the relevant plugins
'''
for plugin in Plugin.__subclasses__():
        # only register plugins according to user's preferences
        if plugin.symbol in preferences:
            plugin().activate()
        else:   # record its existence
            all_plugins[plugin.symbol] = plugin

Changes from the old version are indicated in blue (with corresponding comments in green). Note that we did not change a single line of code for the actual plugins! We did use the same names (activate and deactivate) both for a function and a class method. This should probably be avoided in a larger application. In this example, the code is short enough that it should not create too much confusion. In a real application we would also give the possibility of changing the user's preferences, storing the information in some configuration file.

Dynamic activation

Now that we now how to activate and deactivate a plugin, it might be useful to consider dynamic activation of an external plugin, not located in the normal plugins directory. For example, consider the following plugin (located in op_3.py):


from plugins.base import Plugin

class operator_mod_token(Plugin):
symbol = '%'
lbp = 10
def nud(self):
   return expression(100)
def led(self, left):
   return left % expression(10)

This file is located in subdirectory "external" which is at the same level as "plugins" in our sample code. To invoke this plugin from our base application, we need to add the following code to calculator.py:


if __name__ == "__main__":
#...

    # Simulating dynamic external plugin initialization
external_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)),
                          'external')
sys.path.insert(0, external_dir)
mod = __import__('op_3')
mod.expression = expression
    # register this plugin using our default method
register_plugins()
    # Since it is not activated by default, we need to do it explictly
activate('%')
assert calculate("7%2") == 1

print "Done!"

Note that we also need to import register_plugins() from base.py to make this work.

That's it! If you get the code from the py-fun repository, you can try it out yourself.

A small svg module

2008-12-19T23:12:00.004-04:00

Update: by combining suggestions made in comments, one can probably do away with much of what I describe in this blog post. To wit:


>>> from xml.etree import ElementTree as etree
>>> from functools import partial
>>> Circle = partial(etree.Element, 'svg:circle')
>>> c = Circle(cx='100', cy='200', fill='red')
>>> etree.tostring(c)
'<svg:circle cx="100" cy="200" fill="red" />'

The only minor drawback is that attributes have to be strings, whereas the module described in this post could handle integer attributes. (Python 2.5+ required for ElementTree)

Original post below
(Time to take a break from the plugins blog series...)

Scalable Vector Graphics (SVG) are becoming more and more common on the web due to the increased support by decent browsers. SVG specifications include basic shapes such as circle, rectangles, etc., as well as supporting clipping, masking and composition, filter effects and much more. Attempting to write a python-ic module supporting all possible SVG primitives and options via code like

test_circle = Circle(x=10, y=10, r=5, color='red')

can be a daunting task. Furthermore, documenting such a module would result in a lot of duplication with the official specification document. Fortunately, there is a simpler way than simply attempting to write a complete SVG module using Class-based definitions such as the one written above. The idea is to use instead an API similar to that of ElementTree (see also) - albeit much simplified.

Suppose that we would want to be able to create SVG circles, such as

<circle cx="600" cy="200" r="100" fill="red" stroke="blue" width="10"/>

and rectangles, such as

<rect x="1" y="1" height="398" fill="none" stroke="blue" width="1198"/>

using Python code. A simple way to achieve this would be to define the following class:


class XmlElement(object):
    '''First prototype from which all the xml elements are derived.

       By design, this enables all elements to automatically give a
       text representation of themselves - it is not quite complete.'''

    def __init__(self, tag, **attributes):
        '''A basic definition that will be replaced by the specific
           one required by any element.'''
self.tag = tag
if attributes is not None:
   self.attributes = attributes
else:
   self.attributes = {}

    def __repr__(self):
        '''This normal python method used to give a string representation
        for an object is used to automatically create the appropriate
        syntax representing an xml object.'''
attrib = ["  <%s" % self.tag] # open tag
for att in self.attributes:
   attrib.append(' %s="%s"' % (att, self.attributes[att]))
attrib.append("/>\n")
return ''.join(attrib)

Using this class, we can create a circle instance corresponding to the definition written previously as


circle = XmlElement("circle", cx=600, cy=200, r=100, fill="red",
            stroke="blue", width=10)

This is not quite as simple as the very first Circle() class-based example we wrote but it has the advantage of supporting all possible SVG attributes.

While the above XmlElement class definition is adequate for most basic SVG elements, it does not support such features as 1. text, 2. namespace (e.g. svg: prefix) and 3. grouping and sub-elements. All three additional features can be taken care of by the following modified class definition:


class XmlElement(object):
    '''Prototype from which all the xml elements are derived.

       By design, this enables all elements to automatically give a
       text representation of themselves.'''

    def __init__(self, tag, **attributes):
        '''A basic definition that will be replaced by the specific
           one required by any element.'''
 self.tag = tag
 self.prefix = ""
 self.sub_elements = []
 if attributes is not None:
     self.attributes = attributes
 else:
     self.attributes = {}

    def __repr__(self):
        '''This normal python method used to give a string representation
        for an object is used to automatically create the appropriate
        syntax representing an xml object.'''
 attrib = ["  <%s%s"%(self.prefix, self.tag)] # open tag
 for att in self.attributes:
     if att != 'text':
         attrib.append(' %s="%s"' % (att, self.attributes[att]))
 if 'text' in self.attributes:
     attrib.append(">%s\n" % (self.attributes['text'],
                                             self.prefix, self.tag))
 elif self.sub_elements:
     attrib.append(">\n")
     for elem in self.sub_elements:
         attrib.append("  %s" % elem)
     attrib.append("\n" % (self.prefix, self.tag))
 else:
     attrib.append("/>\n")
 return ''.join(attrib)

    def append(self, other):
        '''append other to self to create list of lists of elements'''''
 self.sub_elements.append(other)

That's almost it! With the exception of comments and Document Type Definition (dtd), we can use the above to create simple xhtml document containing ANY svg graphics without having to worry about xhtml syntax, opening and closing brackets, etc. However, we can possibly do even a little better. Consider the following xhtml document with embedded svg graphics:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink">
<head>
<title>This is the title.</title>
</head>
<body>
<p>This is the body.</p>
<svg:svg width="0" height="0">
<svg:defs>
<svg:circle cy="0" cx="0" r="20" id="red_circle" fill="red"/>
</svg:defs>
</svg:svg>
<svg:svg width="200" height="200">
<svg:use xlink:href="#red_circle" transform="translate(100, 100)"/>
</svg:svg>
<!-- This is a comment. -->
</body>
</html>

With just a few additional definitions, we can create this document using only Python code as follows:

doc = XmlDocument()
doc.head.append(XmlElement("title", text="This is the title."))

# A good practice is to define svg objects, and insert them
# using the definition; this is overkill for this example, but it
# provides a test of the class.
test_def = SvgDefs()
test_def.append(SvgElement("circle", cx=0, cy=0, r=20, fill="red",
                  id="red_circle"))

doc.body.append(XmlElement("p", text="This is the body."))
doc.body.append(test_def)

# we now create an svg object, that will make use of the definition above.
svg_window = SvgElement("svg", width="200", height="200")
use_circle = SvgElement("use", transform="translate(100, 100)")

# xlink:href can't be used as an attribute name passed to __init__
# this is why we use this two-step process.
use_circle.attributes["xlink:href"] = "#red_circle"

svg_window.append(use_circle)
doc.body.append(svg_window)
doc.body.append(Comment("This is a comment.")) # just for fun.

print doc

The additional definitions are as follow:


class XmlDocument(XmlElement):
    def __init__(self):
   self._begin = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
    xmlns:svg="http://www.w3.org/2000/svg"
    xmlns:xlink="http://www.w3.org/1999/xlink">\n"""
   self._end = "</html>"
   self.head = XmlElement("head")
   self.body = XmlElement("body")

    def append(self):
        '''Directly appending is not allowed'''
   assert False, "Append to either head or body."

    def __repr__(self):
        '''Gives an appropriate representation of an xml document.'''
   return self._begin + str(self.head) + str(self.body) + self._end


class SvgElement(XmlElement):
    '''Prototype from which all the svg elements are derived.

       By design, this enables all elements to automatically give an
       appropriate text representation of themselves.'''
    def __init__(self, tag, **attributes):
   XmlElement.__init__(self, tag, **attributes)
   self.prefix = "svg:"

class SvgDefs(SvgElement):
    '''Short-cut to create svg defs.  A user creates an instance of this
    object and simply appends other svg Elements'''
    def __init__(self):
   self.defs = SvgElement("defs")
   self.root = SvgElement("svg", width=0, height=0)
   self.root.append(self.defs)

    def append(self, other):
        '''appends other to defs sub-element, instead of root element'''
   self.defs.append(other)

    def __repr__(self):
        '''gives a string representation of an object, appropriate for
           insertion in an html document'''
   return str(self.root)

class Comment(object):
    '''Comment that can be inserted in code xml documents'''
def __init__(self, text):
   self.text = text
def __repr__(self):
   return "<!-- " + self.text + " -->\n"

That's it for real this time! Fewer than 100 lines of code that you can use if you need to programmatically create (x)html documents containing svg images. There are a few limitations (elements containing text may not be chained...) but it works for me. If you want to try it yourself, you can find the module here.

Plugins - part 4: Crunchy-style plugin

2008-12-19T16:49:00.002-04:00

In this 4th post in the plugins series, I will explain the approach we used in Crunchy. While explaining the main features, I will also compare with the simple class-base plugin framework introduced in the third post in this series.

Crunchy's approach does not require a plugin to be class-based. In fact, most plugins used in Crunchy only make use of simple functions inside modules. While the class-based framework introduced in the third post used the fact that Python allowed automatic discovery of subclasses, the approach used in Crunchy requires an explicit registration of plugins. Using the same example as before, this means that op_2.py would contain the following code:


def register(OPERATORS):
    OPERATORS['**'] = operator_pow_token

class operator_pow_token(object):
 lbp = 30
 def led(self, left):
     return left ** expression(30-1)

Note that the class (operator_pow_token) is unchanged from the original application.

The method used to find plugins is similar to that introduced previously. The entire code required is as follows:


def init_plugins(expression):
  plugin_dir = (os.path.dirname(os.path.realpath(__file__)))
  plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
  sys.path.insert(0, plugin_dir)
  for plugin in plugin_files:
      mod = __import__(plugin)
       if hasattr(mod, "register"):
          mod.expression = expression
           mod.register(OPERATORS)

By comparison, the code used in the class-based plugin could have been written as:


def init_plugins(expression):
   plugin_dir = os.path.dirname(os.path.realpath(__file__))
   plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
   sys.path.insert(0, plugin_dir)
   for plugin in plugin_files:
       mod = __import__(plugin)
       mod.expression = expression
    for plugin in Plugin.__subclasses__():
        OPERATORS[plugin.symbol] = plugin

So, in one case (Crunchy-style), we have an explicit registration process with no need to create a sample base class (and, in fact, no need to work with classes at all), while in the other we have an automatic registration based on identifying subclasses. This does not mean that the Crunchy-style is better - just different. Both are equally good for this type of simple application. While we have not found the approach used in Crunchy to be limiting us in any way when extending Crunchy, something must be said for the fact that all the other Python examples of plugin-based application I have found have been based on using classes.

I can now give another motivation for having chosen the small expression calculator as a candidate for a plugin-based application: since all mathematical operations were already implemented as classes, it was readily suitable for the class-based approach (and the Zope Component Architecture one, etc.) whereas all my existing code samples that used plugins (from Crunchy, docpicture, etc.) had mostly functions rather than classes in plugins.

Plugins - part 3: Simple class-based plugin

2008-12-19T15:26:00.002-04:00

In the first post of this series, I introduced a simple application to be used as a demonstration of a plugin-based application. The chosen application was an expression calculator contained in a single file. In the second post, I modularized the original file so that the new file structure would become a good representative of a plugin based application. In this post, I will explain how to make use of a simple class-base plugin framework. The model I have chosen follows fairly closely the tutorial written by Armin Ronacher. Another tutorial demonstrating a simple class-based plugin framework has been written by Marty Alchin.

The first step is to define a base Plugin class. All we need is to include the following in base.py:


class Plugin(object):
   pass

Next, we ensure that classes used in plugins derive from this base class. We only give one explicit example, that of the class included in op_2.py since the 4 classes included in op_1.py would be treated in exactly the same way.


from plugins.base import Plugin

class operator_pow_token(Plugin):
   symbol = '**'
  lbp = 30
  def led(self, left):
      return left ** expression(30-1)

Note that we added one more line of code to the class definition. We are now ready to deal with the plugin discovery and registration.

Rather than hard-coding the information about which plugin files to import as we did when we simply modularize the application, we give a way for our program to automatically find plugins. With the file structure that we have created, this can be accomplished as follows:


def find_plugins(expression):
   '''find all files in the plugin directory and imports them'''
   plugin_dir = os.path.dirname(os.path.realpath(__file__))
   plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
   sys.path.insert(0, plugin_dir)
   for plugin in plugin_files:
       mod = __import__(plugin)
       mod.expression = expression

Note that the last line of code is included because of the "wart" mentioned in the previous post and would not usually be included. To be safe, we should probably have ensured that expression was not already defined in the modules to be imported since, in theory, Python files other than plugins (such as __init__.py) might be present in the plugin directory. In this tutorial series we will often ignore the need to insert try/except clauses to simplify the code.

While we have imported the modules containing the plugins, they are not yet known in a useful form by the main application. To do so is very simple in this class-based approach, thanks to Python's treatment of (sub-)classes. Here's the code to do this:


def register_plugins():
 '''Register all class based plugins.

    Uses the fact that a class knows about all of its subclasses
    to automatically initialize the relevant plugins
 '''
 for plugin in Plugin.__subclasses__():
     OPERATORS[plugin.symbol] = plugin

That's it! It is hard to imagine anything simpler. With this last definition, the entire base.py module can be written as:


import os
import sys

OPERATORS = {}

class Plugin(object):
  pass

def init_plugins(expression):
  '''simple plugin initializer
  '''
  find_plugins(expression)
  register_plugins()

def find_plugins(expression):
  '''find all files in the plugin directory and imports them'''
  plugin_dir = os.path.dirname(os.path.realpath(__file__))
  plugin_files = [x[:-3] for x in os.listdir(plugin_dir) if x.endswith(".py")]
  sys.path.insert(0, plugin_dir)
  for plugin in plugin_files:
      mod = __import__(plugin)
      mod.expression = expression

def register_plugins():
  '''Register all class based plugins.

     Uses the fact that a class knows about all of its subclasses
     to automatically initialize the relevant plugins
  '''
  for plugin in Plugin.__subclasses__():
      OPERATORS[plugin.symbol] = plugin

In the next post, I will show another simple alternative approach similar to the one used in Crunchy.

Plugins - part 2: modularization

2008-12-19T12:24:00.002-04:00

In the first post on the Plugins series, I introduced the small application used to demonstrate how one could modularize applications using a plugin architecture. The digital ink was barely dry on that post that already two people rose to the challenge and presented their solution, one using the standard method with the Zope Component Architecture, the other a modified method using grok. I will comment on these two solutions later in this series.

With apologies to the more advanced users, I have decided to proceed fairly slowly and cover many simple concepts with this series of plugins. Thus, this second post will not yet discuss plugins, but simply lay the groundwork for future posts. By the way, for those interested, and as pointed out by Lennart Regebro in his post, all the code samples that I will use can be browsed at, or retrieved from, my py-fun google code repository.

As a first step before comparing different approaches to dealing with plugins, I will take the sample application introduced in the first post and modularize it.

The core application (calculator.py) is as follows:


import re

from plugins.base import OPERATORS, init_plugins

class literal_token(object):
  def __init__(self, value):
      self.value = value
  def nud(self):
      return self.value

class end_token(object):
  lbp = 0

def tokenize(program):
  for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
      if number:
          yield literal_token(int(number))
        elif operator in OPERATORS:
            yield OPERATORS[operator]()
      else:
          raise SyntaxError("unknown operator: %r" % operator)
  yield end_token()

def expression(rbp=0):
  global token
  t = token
  token = next()
  left = t.nud()
  while rbp < token.lbp:
      t = token
      token = next()
      left = t.led(left)
  return left

def calculate(program):
  global token, next
  next = tokenize(program).next
  token = next()
  return expression()

if __name__ == "__main__":
  init_plugins(expression)
  assert calculate("+1") == 1
  assert calculate("-1") == -1
  assert calculate("10") == 10
  assert calculate("1+2") == 3
  assert calculate("1+2+3") == 6
  assert calculate("1+2-3") == 0
  assert calculate("1+2*3") == 7
  assert calculate("1*2+3") == 5
  assert calculate("6*2/3") == 4
  assert calculate("2**3") == 8
  assert calculate("2*2**3") == 16
  print "Done!"

For the next few posts, when I demonstrate some very simple plugin approaches, this core application will remain untouched. This is one important characteristic of plugin-based application: in a well-designed application, plugin writers should not have to modify a single line of the core modules to ensure that their plugins can be used.

Communication between plugins and the core application is ensured via an Application Programming Interface (API) unique to that application. In our example, the API is a simple Python dict (OPERATORS) written in capital letters only to make it stand out.

In a sub-directory (plugins), in addition to an empty __init__.py file, we include the following three files:

1. base.py


OPERATORS = {}

def init_plugins(expression):
    '''simulated plugin initializer'''
    from plugins import op_1, op_2

    op_1.expression = expression
    op_2.expression = expression

    OPERATORS['+'] = op_1.operator_add_token
    OPERATORS['-'] = op_1.operator_sub_token
    OPERATORS['*'] = op_1.operator_mul_token
    OPERATORS['/'] = op_1.operator_div_token
    OPERATORS['**'] = op_2.operator_pow_token

2. op_1.py


class operator_add_token(object):
   lbp = 10
   def nud(self):
       return expression(100)
   def led(self, left):
       return left + expression(10)

class operator_sub_token(object):
   lbp = 10
   def nud(self):
       return -expression(100)
   def led(self, left):
       return left - expression(10)

class operator_mul_token(object):
   lbp = 20
   def led(self, left):
       return left * expression(20)

class operator_div_token(object):
   lbp = 20
   def led(self, left):
       return left / expression(20)

and 3. op_2.py


class operator_pow_token(object):
   lbp = 30
   def led(self, left):
       return left ** expression(30-1)

The last two files have been simply extracted with no modification from the original application. Instead of having 2 such files containing classes of the form operator_xxx_token, I could have included them all in one file, or split into 5 different files. The number of files is irrelevant here: they are only introduced to play the role of plugins in this application.

The file base.py plays the role here of a plugin initialization module: it ensures that plugins are properly registered and made available to the core program.

Since I wanted to change the original code as little as possible, a "wart" is present in the code as written since it was never intended to be a plugin-based application: the function expression() was accessible to all objects in the initial single-file application. It is now needed in a number of modules. The file base.py takes care of ensuring that "plugin" modules have access to that function in a transparent way. This will need to be changed when using some standard plugin frameworks, as was done in the zca example or the grok one.

In the next post, I will show how to take this now modularized application and transform it into a proper plugin-based one.

Plugins - part 1: the application

2008-12-18T20:41:00.003-04:00

My interest in plugins started two years ago listening to Ivan Krstić talk about the OLPC. Following his talk, I wrote the following on edu-sig:

One open issue (as I understand it) is that of finding the "best practice" for plugins. The idea is that the core programs should be as small as possible but easy to extend via plugins. I thought that there already was a "well known and best way" to design plugins - and it was on my list of things to learn about (to eventually incorporate rur-ple within crunchy).

After discussing this off-list with Johannes Woolard, I concluded that we should try to redesign Crunchy to make use of plugins. While I was thinking about how we might proceed to do this, Johannes went ahead and implemented a simple plugin framework which we eventually adopted for Crunchy.

While there are a few agreed-upon "standards" when it comes to dealing with plugins in Python (such as setuptools and Zope Component Architecture), I tend to agree with Ivan Krstić's observation that there are no "best practice" for plugins - at least, none that I have seen documented. As what might be considered to be a first step in determining the "best practice" for writing plugin-based applications with Python, I will take a sample application, small enough so that it can be completely included and described in a blog post, and not written with plugins in mind. I thought it would be a more representative example to use an arbitrary sample application, rather than trying to come up with one specifically written for the purpose of this series of post.

The application I have chosen is a small modification of an expression calculator written and described by Fredrik Lundh, aka effbot, a truly outstanding pythonista. The entire code is as follows:


""" A simple expression calculator entirely contained in a single file.

See http://effbot.org/zone/simple-top-down-parsing.htm for detailed explanations
as to how it works.

This is the basic application used to demonstrate various plugin frameworks.
"""

import re

class literal_token(object):
   def __init__(self, value):
       self.value = value
   def nud(self):
       return self.value

class operator_add_token(object):
    lbp = 10
    def nud(self):
        return expression(100)
    def led(self, left):
        return left + expression(10)

class operator_sub_token(object):
    lbp = 10
    def nud(self):
        return -expression(100)
    def led(self, left):
        return left - expression(10)

class operator_mul_token(object):
    lbp = 20
    def led(self, left):
        return left * expression(20)

class operator_div_token(object):
    lbp = 20
    def led(self, left):
        return left / expression(20)

class operator_pow_token(object):
    lbp = 30
    def led(self, left):
        return left ** expression(30-1)

class end_token(object):
   lbp = 0

def tokenize(program):
   for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
       if number:
           yield literal_token(int(number))
        elif operator == "+":
            yield operator_add_token()
        elif operator == "-":
            yield operator_sub_token()
        elif operator == "*":
            yield operator_mul_token()
        elif operator == "/":
            yield operator_div_token()
        elif operator == "**":
            yield operator_pow_token()
        else:
           raise SyntaxError("unknown operator: %r" % operator)
   yield end_token()

def expression(rbp=0):  # note that expression is a global object in this module
   global token
   t = token
   token = next()
   left = t.nud()
   while rbp < token.lbp:
       t = token
       token = next()
       left = t.led(left)
   return left

def calculate(program):
   global token, next
   next = tokenize(program).next
   token = next()
   return expression()

if __name__ == "__main__":
   assert calculate("+1") == 1
   assert calculate("-1") == -1
   assert calculate("10") == 10
   assert calculate("1+2") == 3
   assert calculate("1+2+3") == 6
   assert calculate("1+2-3") == 0
   assert calculate("1+2*3") == 7
   assert calculate("1*2+3") == 5
   assert calculate("6*2/3") == 4
   assert calculate("2**3") == 8
   assert calculate("2*2**3") == 16
   print "Done!"

The latest version used can be found online.

In the above code, I have highlighted in red classes that will be transformed into plugins. I have also highlighted in green hard-coded if/elif choices that will become indirect references to the plugin components.

In the next post in this series, I will break up this single file in a set of different modules as a required preliminary step before transforming the whole applications into a plugin-based one, with a small core. In subsequent posts, I will keep the core constant and compare various approaches that one can use to link the plugins with the core.

Seeing double at Pycon 2009

2008-12-17T21:04:00.002-04:00

Jesse Noller is going to give two talks at Pycon 2009. So is Tarek Ziadé. And Mike Fletcher is as well. And Brett Cannon has a talk and a panel. So far there I have not seen any post on Planet Python about someone giving just one talk.

I would hate to be the one breaking the streak. So, I might as well announce that I will be giving two talks as well. :-)

Not surprisingly, the first one is about Crunchy. The title of the talk is Learning and Teaching Python Programming: The Crunchy Way, and the abstract reads as follows:

Crunchy (http://code.google.com/p/crunchy) is a program that transforms a static Python tutorial into an interactive session within a browser. In this talk, I will present Crunchy, focusing on the features that are specifically designed to be helpful in a formal teaching setting.

Not exactly Earth-shattering but hopefully of interest to anyone that has to teach programming in a formal setting or who would just be interested in showing off Python to anyone. This Crunchy talk is, of course, not going to be your traditional slide-based talk but rather more like an interactive demo using Crunchy. I am hoping to have a few surprises by the time the conference occurs.

My other talk is going to be very different. I doubt very much that I will be using Crunchy for it. The title is Plugins and Monkeypatching: increasing flexibility, dealing with inflexibility, and the abstract reads as follows:

By using plugins, one can create software that is easily extensible by others, thereby promoting collaborative development. The flip side of extensible software occurs when dealing with some standard framework whose interface is closed but which does not do exactly what is desired. In this case, monkeypatching may be worth considering.

In this talk, I'll give concrete examples of both plugin design and using monkeypatching, using small code samples from existing projects, and discuss the advantages and the shortcomings of the methods used. I will also include the design of a tiny, but flexible module for generating svg code - and compare it with other existing approaches.

I can not pretend to even come close to being an expert about designing plugin based applications. Still, I felt that I have had some potentially useful experiences to share about these topics which motivated my talk proposal. Now that it has been accepted, I have started working on fleshing out the original outline.

In preparation for the actual talk, which will not go into much code details due to time constraints, I plan to start a short series of posts about plugins. In the first post I will give an overview of a simple application (a calculator) that is written as a single file. In the second post, I will reorganize the code so as to use multiple files, with a number of modules located in a "plugins" directory, laying out the groundwork for working with actual plugins. Subsequent posts will be used to demonstrate different approaches used to transform the application into a truly plugin-based one.

Of course, the plugin model used in Crunchy will be one approach showcased. A second one (which I have already implemented) is a simple class based one inspired by a tutorial written by Armin Romacher. I also plan to demonstrate how to use the Zope component architecture approach as well as the setuptools based method (and possibly others depending on suggestions I might receive).

Since I have never actually written any code using the Zope component architecture or the setuptools based approach, I thought it would be interesting to do this in a truly open-source spirit. Therefore, once I have written the first two or three posts in this series, I would like to invite anyone interested to contribute their own code demonstrating their favourite framework. This way, experts could make sure that their favourite framework is properly showcased, and not misrepresented by me. Interested parties can contribute either by sending me the code directly or by blogging about it. (If your blog appears on either planet.python.org or planetpython.org, I will most likely read it.)

Anyone who contributes in this way to my talk will be mentioned at Pycon AND receive half of the stipend I get as a presenter. ;-)

Thwarted by lack of speed

2008-11-28T22:46:00.002-04:00

I was hoping to make an announcement of a new cool app based on Google's App Engine but unfortunately I have been thwarted by Python's relative lack of speed.

I have started working on a new version of Crunchy that would run as a web app on Google's servers. While the current version of Crunchy fetches existing html pages, processes them and displays them in the browser, this new version would retrieve html page content (in reStructuredText format) from Google's datastore, transform it into html, process it to add interactive elements, and then displays them.

This new app was going to be usable as a wiki to create new material. This was my starting point, greatly helped by an already existing wiki example that I adapted to use reStructuredText. When requesting a page, the following was supposed to happen:

1. reStructuredText content (for the body of the html page) is fetched from the datastore.
2. said content is transformed (by docutils) into html
3. html content is further processed by modified "crunchy engine" to add interactive elements.
4. modified html content is inserted in page template and made available.

The user would then be able to enter some Python code which could be send back to the App Engine using Ajax for processing and updating the page display.

A normal user would only be able to interact with already existing pages. Special users ("editors") only would have been able to add pages. I was hoping that people teaching Python would be interested in writing doctest-based exercises and that a useful collection could be implemented over time.

Unfortunately, this approach can not work, at least not using Google's App Engine on Google's own servers. :-(

Just playing with small pages, steps 1 and 2 are long enough that I get warnings logged mentioning that requests are taking too long. I know from experience that step 3 (which I have not started to implement/port from the standard Crunchy) can take even longer for reasonably size pages. So, this does not appear to be feasible ... which is unfortunate.

I think I will continue to develop this app to be used as a local one and perhaps write a second wiki-based app that would take html code with no further processing. I could use the first one to create a page, have it processed and use the "view source" feature of Firefox to cut and paste the content into the online app. This would remove the need for any processing of pages on Google's servers - only Python code execution would need to be taken care of. (Of course, a user could enter some code sample that would take too long to execute and hit Google's time limit ...)

If anyone has a better idea, feel free to leave it as a comment.

docpicture progress

2008-11-01T00:59:00.003-03:00

For those interested, docpicture can now display images from the web. There's also a somewhat silly example where I embedded the code for a matplotlib example inside a docstring and have it displayed as a plot when viewing the docstring via docpicture inside a web browser. In order to do so I had to exec the code which is not exactly good practice ... but it serves to highlight the need to either only allow "parsers" from the standard distribution or require the user to give permission to a parser to be able to register itself with docpicture while it is running. I chose this second approach, although if you run the demo, you will not be given the opportunity to approve or not the parser - it will be done for you. This may need to be revisited...

I just announced a new release on the Python list. You can get docpicture 0.2 from here.

svg mathematical equation

2008-10-29T22:28:00.002-03:00

Ok, it's done: mathematical equations generated dynamically and displayed as svg graphics. Only using the standard Python library ... and one "tiny" additional download: matplotlib. Here's the first result (saved as a "hard-copy"; you may have to download the page and reopen it locally using Firefox.)

Note: do not bother looking for the files in the "py-fun" repository where I had the first release of docpicture. I will clean up things a bit and do a new release from a different place.

As usual, comments & suggestions are welcome.

docpicture and uml sequence diagrams

2008-10-28T17:26:00.002-03:00

In a previous post about docpicture, I gave an example of a graphics generated from this site as something that would be desirable to do. (You can find more examples here.) Well, it turned out to be easy to do ... at the cost of a server connection. I used the example given to embed a graphics inside a page and ... voilà, it is done. As long as one has a live internet connection (and assuming the websequencediagram server is not down), a graphics is generated as requested.

Eventually, I still would like to implement my own parser to create svg code for uml sequence diagrams rather than relying on an external service.

docpicture: initial release

2008-10-27T23:47:00.001-03:00

The subject line says it all. It's a small download: less than 22 kB, available from here. Feedback and suggestions are definitely welcome.

docpicture: working ... and a query.

2008-10-26T23:53:00.004-03:00

docpicture (see previous posts) is now working as a full prototype. By this, I mean that instead of doing


>>> help(some_object)

at the Python prompt, one can do


>>> from docpicture import view
>>> view(some_object)

and some_object's docstring will be displayed in your webbrowser, with any docpicture directive being translated so as to embed a nice picture. Well, by "any", I mean any turtle directive conforming to the limited syntax I have included.

When I compare the output of help() with that of docpicture.view(), I am struck at how much more information than simply the object's docstring is included. I have tried (briefly) to play with the pydoc module to see if I could redirect the output of help() to a string that I could process with docpicture.view() ... but to no avail.

If anyone knows how I could do this simply, I would be very grateful.

docpicture is going to be released (version 0.1) as soon as I complete a decent "readme" file.

--UPDATE-- Ok, after playing some more with pydoc, I found out how to do this.

In my module, I do the following:


import pydoc
from StringIO import StringIO
my_stdin = StringIO()
def my_pager(text):
   my_stdin.write(pydoc.plain(text))
   return
pydoc.pager = my_pager

pydoc.help(obj)
retrieved = my_stdin.getvalue()
my_stdin.close()

and use the retrieved text as I wish.

docpicture: getting closer

2008-10-24T00:28:00.003-03:00

This is just a progress report for the curious among you: the previous two images were generated automatically from the docpicture code written above them. If everything goes well, by the end of the weekend I'll be ready to give a sneak preview of the code to anyone interested. Feel free to contact me.

docpicture + svg generation: first prototype working

2008-10-19T19:24:00.005-03:00

As outlined in a previous post, I have decided to use svg to embed pictures in html pages generated from docstrings. Of course, this could be generalized to other cases than docstrings; for example, this could be implemented as a reStructuredText directive. In the course of playing with generating such pages with inline svg code, I observed the following:

If a file is saved locally and loaded within Firefox, it should be saved with a ".xml" (or possibly ".xhtml") extension.
If a file is served dynamically from a server, all that is needed is that its content be identified as "application/xhtml+xml" [as I had mentioned previously].
If the file is put on a "generic webserver" that can't be configured by the user, Firefox will ignore the svg code if the extension of the file is ".xml" or ".html". However, I did find a workaround: use a ".xhtml" extension and, when prompted by Firefox as to what application to use to open such file, select Firefox itself. The file will be downloaded locally and displayed correctly. At least, this is what happens on a Mac with Firefox 3.

There might be another way to do this; if so, I would be interested in knowing how. In the meantime, for those interested, here is the output of a first working test case.

Update: The test case has been improved with styling.

Update 2: A new picture perhaps gives a better idea of a more realistic use case.

Seeking advice on parsing

2008-10-19T11:46:00.002-03:00

Dear Lazyweb,

As a follow-up to my previous post, I have a question...

Suppose you wanted to design an application that used parsing as a core element and you wanted this application to be easily extended by users. Furthermore, you were hoping that the users would contribute back some parsers that could be included in future versions. Would you:

1a) Use pyparsing and require all potential users of your application to download it separately.
1b) Use pyparsing but include it bundled with your application.
2) Use regular expressions (re module in standard library) and expect everyone to do the same.
3) Use some other module in the standard library.
4) Use some other 3rd party parsing package.

More on docpictures and (almost) minimal example of web server with inline svg

2008-10-18T14:24:00.005-03:00

After playing some more with the idea of embedding pictures in docstrings, I've settled on using dynamically created svg images instead of png images. The basic idea (which I'll write about in more details later) is to have something like


.. docpictures:: some_type
highly readable description

and have the docpicture module parse the "highly readable description" based on the syntax defined in some_type. Note that I chose this notation to be compatible with reStructuredText directives. The "highly readable description" will depend on the context. For example, the mathematically inclined will be able to read this:


.. docpictures:: equation
e^{i\pi} = -1

while the following might be easy to understand by programmers:


..docpicture:: uml_sequence
User --> Browser: clicks on a link
Browser -> Crunchy: requests file
Crunchy --> Browser: Authentication Request
Browser --> Crunchy: Authentication Response
note over Crunchy: Retrieves and processes file
Crunchy -> Browser: Sends processed file
Browser --> User: Displays file

In this last example using the syntax of this site which generates the following picture:

(Btw, the picture above is generated automatically each time this page is loaded - so it depends on the availability of the server. I have used a static version of this picture in the documentation for Crunchy.)

Using svg to do graphics is fairly easy. Using svg as embedded objects in an html document requires a bit of searching on the internet. Creating such documents and displaying dynamically requires even more searching (or perhaps more careful reading...). The thing important to remember is to serve the document as "application/xhtml+xml" instead of the usual "text/html". I thought it would be useful to share an almost minimal working example (tested only on Firefox) and perhaps save some time for others that would like to do the same. Feel free to adapt it as you like.

svg_test = """<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:svg="http://www.w3.org/2000/svg">
<head></head>
<body>
<h1>SVG embedded inline in XHTML</h1>
<svg:svg width="300px" height="200px">
<svg:circle cx="150px" cy="100px" r="50px" fill="%s"
stroke="#000000" stroke-width="5px"/>
</svg:svg>
</body>
</html>
"""


import BaseHTTPServer
import webbrowser

colors = ["#330000", "#660000", "#990000",  "#cc0000", "#ff0000"]
index = 0
class WebRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def do_GET(self):
 global index
 self.send_response(200)
 self.send_header('Content-type', 'application/xhtml+xml')
 self.end_headers()
 self.wfile.write(svg_test% colors[index % 5])
 index += 1

port = 8000
server = BaseHTTPServer.HTTPServer(('',port), WebRequestHandler)
webbrowser.open("http://127.0.0.1:%s"%port)
server.serve_forever()

And, if you know how to suppress the output of the webserver, feel free to leave a comment.

Viewing embedded pictures within docstrings

2008-10-14T22:43:00.002-03:00

In a recent post on the Python mailing list, it was suggested that it would be useful if "small" pictures could be embedded within docstrings as additional information. As is often the case, many words were written ... but little code was produced. As I have been guilty of this myself in the past, I decided that it was time to do things differently. After a quick prototype sent to the Python list, I wrote this recipe which gives a simple way to embed and display images inside a docstring in a totally transparent way.

Is this something that people would find useful? Any suggestions for improvements? Should I implement this within Crunchy? (Btw, Crunchy 1.0 alpha 1 has been released just last week).

Monkeypatching doctest

2008-06-22T21:06:00.004-03:00

The Python doctest module rocks. Lately, I have been using it to write unit tests for Crunchy: for each module, I write a reStructuredText file which contains sample tests written as simulated interpreter sessions, using doctest.testfile(). This has worked really well in general ... however, I have encountered one small annoyance, which I managed to get rid of in an "elegant" way using Monkeypatching.

doctests allow the use of directives. One "powerful" directive is the ELLIPSIS directive. Quoting from the documentation:

When specified, an ellipsis marker (...) in the expected output can match any substring in the actual output. This includes substrings that span line boundaries, and empty substrings, so it's best to keep usage of this simple. Complicated uses can lead to the same kinds of "oops, it matched too much!" surprises that .* is prone to in regular expressions.

Unfortunately, I encountered a case where the ellipsis marker did not allow enough matching! Consider the following situation: I have a program (Crunchy!) that saves the user's preferences (including the language) in a configuration file each time its value is changed. It also gives some feedback to the user whenever this happens.


>>> original_value = crunchy.language
>>> set_language('en')  # setting this value for some standardized tests
Language has been set to English

At the end of the test, I want to restore the original value.

>>> set_language(original_value) #doctest: +ELLIPSIS
...

Here I want the ellipsis (...) to match the string that is going to be printed out in the original language as I have no idea what this string will look like. The problem is that the ellipsis in this case is thought to be a Python (continuation) prompt and not a string that is "matched". One workaround that I had been using was to modify set_language to add a parameter ("verbose") that was set to True by default but that I could turn off when running tests. While this is simple enough that it surely would never (!) introduce spurious bugs, it does not feel right; one should not modify functions only for the purpose of making them satisfy unit tests.

According to the documentation,

register_optionflag(name)

Create a new option flag with a given name, and return the new flag's integer value. register_optionflag() can be used when subclassing OutputChecker or DocTestRunner to create new options that are supported by your subclasses. register_optionflag should always be called using the following idiom:

  MY_FLAG = register_optionflag('MY_FLAG')

This is great ... except that I want to used doctest.testfile() which does not allow me to specify a subclass of OutputChecker to use instead of the default. Also, I wanted to use as much of possible of the existing doctest module, with as little new code as possible.

This is where monkeypatching comes in.

After a bit of work, I came up with the following solution:


from doctest import OutputChecker
original_check_output = OutputChecker.check_output
import doctest

IGNORE_ERROR = doctest.register_optionflag("IGNORE_ERROR")

class MyOutputChecker(doctest.OutputChecker):
    def check_output(self, want, got, optionflags):
       if optionflags & IGNORE_ERROR:
           return True
       return original_check_output(self, want, got, optionflags)

doctest.OutputChecker = MyOutputChecker

failure, nb_tests = doctest.testfile("test_doctest.rst")
print "%d failures in %d tests" % (failure, nb_tests)

And here's the content of test_doctest.rst


Test of the new flag:

>>> print 42
42
>>> print 2 # doctest: +IGNORE_ERROR
SPAM!

This yields a test with no failures. There might be a more elegant way of doing this; if so, I would be very interested in hearing about it.

Thoughts on Google Summer of Code 2008 - part 1

2008-04-18T21:29:00.001-03:00

In just a few days, Google will make some announcements that will please many hundreds of students and disappoint even more. I think we should all focus on the positive side when the announcements are made. It is, after all, a fantastic thing that a company is spending millions of dollars so that some students get the chance to program on Open Source software as a summer job.

Just think of it. Who could have predicted, just five years ago, that a company would spend that kind of money on students who would work on someone else's project?

This is amazing - and many are now taking it for granted.

I find it great that the Python Software Foundation is an organization that can mentor SoC students. With the excellent supporting work of James Tauber as coordinator, many promising students are going to be paired with a mentor, hopefully leading to great projects to be completed this summer.

I have seen some grumblings on some SoC related lists that have made me thought about some of the "problems" I have seen. Note that these are very minor compared with the strong positive points. I will be discussing those in part 2, after the official announcements are made.

Because the PSF is an umbrella organization, most students work on different projects, unrelated with each other. As a result, they tend to have limited interactions with the greater Python community. I think there should be a "meeting place" where all the students would meet - perhaps a mailing list to which they have all to contribute once a week, sharing their progress, etc.
Not enough positive publicity is given to "successful students", i.e. those that continue to contribute to the Python community after the summer is over. For Crunchy, it has been one success (Johannes Woolard) out of a total of 3 students over the past 2 years. I don't know of many other success stories from other projects ... Alex Holkner comes to mind ... but I feel I should know more names of successful students. (I know there's another Alex or Alessandro who has contributed to the Python core and was involved with GHOP....)
With the exception of a few people like James Tauber and Titus Brown with whom I have had a few email exchanges, I do not feel that as a past/potential mentor I am as much part of a community as I feel should be the case. There is a mentor discussion list, but it does not seem to be the kind of place to generate community building discussions.

In terms of projects submitted, I would describe them to belong in the following categories:

1. Contributions to the "standard" core (cpython code, or standard library)

2 a. Contributions to "non-standard core" (e.g. Jython, PyPy, TinyPy?)
2 b. Contributions to 3rd party libraries (e.g. Numpy, Pygame)
2 c. Contributions to major projects whose end users have to use Python (e.g. SAGE)
2 d. Contributions to projects that can be used to teach Python [Crunchy, of course ;-), but there are others ... that will be for part 2]

3 a. Contributions that propose some new "standards" for Python programmers, never discussed before in the Python community.
3 b. Projects that happen to be written in Python, but whose end users are not exposed (or minimally exposed) to Python.
3 c. Projects that are not written in Python, that may or may not be usable in all OS, and that aren't more useful to Python programmers than they would be to people using other languages.
3 d. et cetera

Assuming that all projects are well-thought of (which is not always the case), I feel that:

Projects in category 1 deserve to be fully supported. The Python community need more capable people contributing to the core to prevent burnout for the current contributors. Perhaps, in a few years, after working some more on Crunchy, I'll feel capable of joining that group and contributing effectively (and have the time to do so).
Projects in categories 2 a-d are worthy of support. There are of course more such projects submitted than can possibly be supported, so some difficult choices had to be made. (Kudos to James for guiding this process.) Many people are going to be disappointed, but this was unavoidable.
Projects in categories 3 a-d are a puzzle to me. I don't understand their appeal for the PSF (and I know I am not the only one), but it seems that very few people are willing to take a public stance on this and debate the issue. Note that this comment is made as an observation on the discussions that took place so far and does not necessarily reflect on any decision that has been made.

This is it for the negative comments. I can't wait for the announcements from Google to focus entirely on the more positive side.

Firefox 3b5: the pain of using the bleeding edge

2008-04-13T17:59:00.005-03:00

After seeing so many positive reviews of the upcoming Firefox 3, I decided to try the latest beta (5) version. It seems indeed to be fast when dealing with complex javascript. While there are a few features I am not too keen about [1], I liked the extra speed (and the reduced RAM usage) so much that I have been using it almost exclusively. That is until now, since I can't rely on it to test Crunchy. Update: this is no longer true, thanks to a reader's comment. The fix was to move the onblur event to the file input, indicated by HERE.

To load a local html file [2] into Crunchy, a two-step process has to be used due to normal javascript security:


<form name="browser_local"
onblur="document.submit_local.url.value =
    document.browser_local.filename.value">
<input name="filename"  type="file" HERE >
</form>

<form action="/local" method="get" name="submit_local">
 <input name="url" type="hidden">
 <input class="crunchy" type="submit">
        value="Load local html tutorial" />
</form>

The first form allows to browse the local drive for a particular file. The second one sends the chosen file's path to the browser as an argument to the "/local" action, something like /local?url=file_path. Unfortunately, when using Firefox 3 beta 5, no argument is passed and we get /local?url= instead. And of course no file can be loaded.

This file browser feature is not something I test regularly when working on Crunchy, nor is it something that can be tested via standard Python unit tests. [3] When I noticed the new bug, it never crossed my mind that this could be a "new Firefox feature" and thought it was something I had broken in Crunchy's code. [4] It was only after I tried a few old releases of Crunchy (to figure out when "I" broke the code) that I figured out that the problem was not due to anything I wrote.

I have not been able to find any note about this new behavior of Firefox. Since this is still a beta, I guess I'll have to wait until the final Firefox 3 release to figure out if I need to change the way I load files. [5]

====
[1] One change I don't like is the rather gaudy auto-suggest list when typing a url.

[2] The same method is used to load reStructuredText files and others.

[3] I really need to investigate twill for this.

[4] One more reason to have a complete unit test coverage. Since I don't, I automatically assumed it was something I had done.

[5] If anyone has any lead as to how to do so reliably in Firefox 3b5 as well as with other browsers, I'd be keen to hear about it.

Shell meme

2008-04-11T18:38:00.002-03:00

I'm responding to peer pressure. I pretty much only use the shell for one thing and rarely restart it...


andre$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
322 python
76 cd
26 ls
11 grep
8 pwd
8 find
5 sudo
3 rm
3 def
2 svn

Using Clone Digger

2008-03-22T19:54:00.004-03:00

There's a new tool available for Python programmers: Clone Digger. While it has not been officially released, it is available from the svn repository. Clone Digger finds code duplications in a given project, and creates a fairly comprehensive report (html file). Seeing the duplications on a screen is a powerful motivation for refactoring.

Check it out!

Update:

Just to make it clear: I had nothing to do with this project; I just found out about it via the gsoc-python-mentors list.

Inspiration and persistence

2008-03-22T11:43:00.003-03:00

While ~~mindlessly wasting time instead of programming~~ selectively reading the internet, I came across this gem by Seth Godin which I reproduce in its entirety:

Persistence

Persistence isn't using the same tactics over and over. That's just annoying.

Persistence is having the same goal over and over.

That's it.

A wiser person would most likely leave it at that. However, this lead me thinking of my goals when it comes to programming which I thought I should write down if only to help me reflect upon them again at a later time. I can sum them up as follows:

Do something that is fun but that gives me some sense of accomplishment rather than just wasting time.
Find ways to make it easier for others to learn programming (in Python).

In doing so, I have found myself oscillating between two extremes:

Trying to follow the "release early, release often" philosophy.
Trying to get everything "just perfect" before releasing anything.

Trying to get things "just perfect" is something that can lead to procrastination and delays. As an example, rur-ple's version 1.0 release candidate 3 has not been updated since July 2007. The next version should be the final 1.0 ... but somehow, I am not happy with many details and I'd like to get everything right for 1.0. Too often I read about (usually commercial) software which is officially released and is considered by its users to be a Beta version. All open source programmers I have met have a sense of pride in their work that I share. So I postpone the final release and end up working on something else...

I went the other way with a little utility called lightning compiler (now at version 2.1), whose version 1.0 was released as a recipe in the online Python cookbook. Much of the rapid evolution of lightning compiler came from user feedback, as expected from the "release early, release often" philosophy. Yet, following the same philosophy generated relatively little feedback for rur-ple of for crunchy to date. I did get some feedback for rur-ple which has been used at an elementary school in Austria, at a high school and a university in the U.S., among others, but it has often been very indirect.

Still, I am persistent. Following Seth Godin's definition of persistence, my second goal written above can be described as finding a solution as to Why Johnny can't code. Or, as I have written elsewhere

My goal is to provide an introduction to programming which is as "smooth" as possible. We sometime hear the phrase "steep learning curve" to characterize some difficult to grasp concept. I think it is important to have as few "steep learning curves" as possible in the learning process. GvR [Guido van Robot] uses a slightly easier syntax than Python ... but at the expense of having a "step-like learning curve" when one wants to go from GvR's world to Python programming. Since Rur-ple uses Python, there is no transition to speak of.

Both rur-ple and Crunchy, and to a lesser extent lightning compiler (which has been incorporated within rur-ple) have been inspired by that goal.

However, sometimes I stray from that goal. For example, inspired by an earlier post on Georg Brand's remarkable Sphinx, Crunchy now includes a prototype for an automated documentation testing framework along the lines of sphinx.ext.doctest which was released yesterday. My intention is to update Crunchy's implementation so that it can be totally compatible with Sphinx's. And while I believe that this is a neat (and fun!) thing to include in Crunchy, it only very indirectly contribute to my overall goal and ends up delaying the 1.0 release for Crunchy.

Blogging too can be a distraction. However, it is my hope that it may generate a few comments that will contribute to inspire me to make Crunchy even more useful.

Success is the result of inspiration and persistence.

Crunchy: Pycon 2008 release

2008-03-07T21:47:00.002-04:00

Crunchy is getting really close to a 1.0 version. To mark the Pycon 2008 event (that I won't be able to attend), I just did a new release (0.9.9). It has a few new goodies that I won't list here, leaving Johannes do the demonstration. As for me, I am heading down South for a vacation with my kids.

Note: the opening Crunchy page indicates that this is version 0.9.8.6 - which is incorrect.

What is left to be done for version 1.0 is cleaning up the existing documentation (proofreading, proofreading, proofreading) and adding a few more pages to it. New features will have to wait until after 1.0.... unless we get feedback from Pycon attendees for "must have" features that we could implement quickly.

As far as I know, there are no bugs (famous last words). If you find any, please let us know.

Pycon and Crunchy

2008-02-29T23:09:00.002-04:00

This year's Pycon program looks very interesting. I wish I could be there but, alas, the timing was just wrong for me this year. This is doubly disappointing as I would have been able to meet with Johannes Woolard in the flesh. Yes, forget Guido van Rossum, Alex Martelli and other famous names: the one person I wanted to meet is Johannes. For more than a year an a half, I have had the pleasure of collaborating with Johannes on Crunchy, without ever meeting him. This year, Johannes will be the one showing Crunchy off. I'm sure he'll do a great job.

And, if anyone is looking to hire a bright, young, hard-working programmer, Johannes will graduate from Oxford this year.

99 problems: looking for volunteers

2008-02-22T20:26:00.002-04:00

Some time ago, Dr. Werner Hett created a list of 99 Prolog problems that could be used to skills in logic programming. More recently, a Ruby learner posted a Ruby version of the first 10 problems, and his solutions. This seemed to be a good idea, especially if one makes use of doctests ... and Crunchy :-). So, I've started my own version of these which you can get as a zip file (containing 6 problems and their solutions) from the Crunchy main page. If you have never done so before, to load a local html file within Crunchy, you simply click on the "Browsing" menu on the left hand side and scroll down until you reach the "Closer to home" section and follow the instructions.

Note that with the next version of Crunchy (the current one is 0.9.8.6) you will be able to start Crunchy with an arbitrary file using something like

python crunchy.py --url=full_local_path_or_url

It would be nice if there could be a complete Python version of the 99 Prolog problems. If anyone is interested in helping, please do not hesitate to contact me.

Automated documentation code testing - part 2

2008-02-01T19:43:00.000-04:00

Thanks to Crunchy's simple plugin architecture, after only a few hours of coding the automated documentation testing (for html files) described here has been implemented (the "first approach" described, that is.) It will be part of the next Crunchy release. In theory, code samples for a complete book could be all tested at the click of a button, provided that the book is available as an html document. The next step will be to define a few new directives so that reStructuredText documents can be used as well.

Now, while I have a few sample test files, it would be nice is to find someone who has a real life document with embedded Python code samples as a test user...

Automated documentation code testing

2008-01-29T18:29:00.000-04:00

During the first year or so of development work on Crunchy, I probably got a nickname of "Dr. NO!" by early Crunchy adopters as I often resisted suggestions for adding new capabilities. At the time, Crunchy required that html pages have additional markup added (vlam = very little additional markup) so that Crunchy could process them properly. I wanted Crunchy-enabled tutorials to be very easily created, without much additional work from tutorial writers, so that Crunchy would be adopted by many people. Most of the suggestions that were made, including some by Johannes, both while he was sponsored by Google as a Summer of Code student and afterwards when he became a co-developer, were rejected by me for that reason. Since then, the situation has changed, mainly for two reasons:

Johannes created a new basic infrastructure for Crunchy where we can introduce new capabilities via plugins, without needing to change a single line of the core in most instances.
Based on the new architecture, I came up with a new way to process pages so that no additional markup was needed for Crunchy to do its magic. This is what makes it possible, for example, to interact with the official Python tutorial on the python.org site.

Now, that it is so easy to implement new capabilities, I am revisiting some ideas I had rejected or ignored before. The struggle I have is to decide when enough is enough before finally having a version 1.0 officially released.

In any event, after reading some comments on this post by Georg Brandl, I started thinking about adding a new option to test code embedded in documentation. To quote from the comments on that post:

One thing that is seriously needed is the ability to run and test code snippets in some fashion. It's just too easy for documentation to get out of date relative to the code, and if you can effectively "unit test" your docs, you're in much better shape.

And I don't mean like doctests, because not everything lends it self well to that style of testing. If it's possible to mark up some code as being for test fixture and some code as being what belongs in the doc, that would be good.

Alternatively, from another reader:

For me a key is being able to test code in the docs, and think the key is being able to "annotate" a code snipit with information about the context in which it should run, and the output it should give.

I think that Crunchy is a very good platform to implement this. There are currently three complementary options I am considering, one of which I have started to implement.

The first option is to have something like the following [note that while I use html notation, Crunchy is now capable of handling reStructuredText, including having the possibility of dealing with additional directives]:

Some normally hidden code, used for setup:
<pre title="setup_code name=first">
a=42
</pre>

Followed by the code sample to be tested:
<pre title="check_code name=first">
print a
</pre>

And the expected output:
<pre title="code_output name=first">
42
</pre>

Upon importing a document containing such examples, Crunchy would insert a button for each code sample allowing the user to test the code by clicking on the button, invoking the appropriate setup, and comparing with the expected output. Alternatively, all such code samples in a document could be run by a single click on a button inserted at the top of a page. A javascript alert could be used to inform the user that all tests passed - otherwise error messages could be inserted in the page indicating which tests failed or passed.

This type of approach could, in theory, be used for other languages than Python; code could be executed by passing information to a separate process launched in a terminal window, with the result fed back into Crunchy as described above.

A second approach is to use the same method used by doctest to combine code sample and expected output; the setup code could still be used as described above.

A third approach, this one completely different, could be used for more general situation than simply for documentation code testing.

Currently, the Python code needs to be embedded inside an html (or rst) document. However, one could create links to code that lives inside separate Python files. For example, one could have the following:


<pre title="python_file">
<span title="python_file_name"> file_path </span>
<span title="python_file_linenumbers"> some_range </span>
</pre>

When viewing the above using a normal browser, one would see something like (using a fictitious example)


../crunchy_dir/crunchy.py
[1-3, 5, 7, 10-15]

However, when viewing the same page with Crunchy, the appropriate lines would be extracted from the file and displayed in the browser. Alternatively, instead of specifying the line numbers, one could have a directive to extract a specific function/method/class as in

<span title="python_file_function"> function_name </span>

which would instruct Crunchy to extract all the code for the function definition, and inserting it in the document. By using such links, the code in the documentation would always (by definition) be kept in sync with the real code. I realize that this is not exactly a novel idea but one whose potential could be extended by using Crunchy in ways never seen before. However, this last approach will have to wait until after Crunchy version 1.0 has been released.

What do you think of these ideas?

Kudos to ActiveState

2008-01-28T20:23:00.001-04:00

ActiveState, the company behind the Python cookbook and many other useful and free resources, is a great supporter of free software. Its free Komodo Edit is a nice piece of software, well worth trying.

I started using Komodo Edit a few months after I switched from using Windows to using a Mac. On Windows, my Python editor of choice was SPE. However, I found that SPE, not being a native Mac application, had some small quirks that I found annoying. After trying Textmate, praised by many Mac users, and Wing among others, I settled on the free Komodo Edit. While I missed the class browser I had gotten used to with SPE, I found that Komodo was enough for my basic needs.

After reading this post by Jesse Noller, I started using pylint within Komodo and while I did not find any bugs (so far!) in my code, it did encourage me to improve the existing code. The possibility of easily adding new tools to Komodo Edit lead me to try its more powerful sibling, Komodo IDE. Komodo IDE has an integrated debugger (something I had *never* used before for code development but that I will likely use more and more in the future) and a code browser side bar which is even better than the one included with SPE. After using it for about a week, I decided to treat myself and purchase a license for it before the trial license ended. However, since I found the price a bit steep for something to use just for fun, I inquired about available discounts. I was told that, even though I did not use it for my work, I was eligible for an educational discount given that I work at a University.

However, there was more to come...

When I indicated that I intended to buy it online, I got an email telling me that I was actually eligible for a deeper discount since I had a license for an earlier version of Komodo Personal edition. This was a total surprise for me. Here's what happened: more than two and a half year ago, ActiveState had a special promotion for open source developers to get a free license for Komodo Personal edition. I had taken advantage of this offer at the time and installed Komodo on my Windows computer. However, I found it was comparable in functionality to SPE for which I had a slight preference. As a result, I gave up on Komodo after trying it for about a week.

Now, more than 2.5 years later, the friendly people at ActiveState reminded me that I had a valid (but free!) license and told me I could simply pay for an upgrade to what is in my opinion a much superior programming environment than the version I had a license for.

Talk about friendly customer service! Thank you ActiveState!

GHOP Python Related Success Story

2008-01-23T22:58:00.000-04:00

Early on during Google's HOP contest, I joined the Python mentors group following a call for task suggestions, and submitted many Crunchy-related tasks. I was amazed by the quality of some contributions. As more potential mentors involved with other Python related projects joined in, I decided to quietly refrain from submitting more Crunchy related tasks. Still, Crunchy got more than its share of contributions from students from all over the world.

Today, one student posted a blog entry about a Crunchy presentation he made to his class. He describes it as a success - and I would agree. However, it is clear to me that the success of his presentation is due by far more to Python's strength than to Crunchy itself. I thought it was a very good example to use when advocating for the use of Python - and therefore, worth linking to.

More power and removing an option

2008-01-19T23:24:00.000-04:00

One of the neat features of Crunchy, suggested to me by Andrew Dalke at Pycon 2007, is its ability to dynamically display images that are generated by some Python code. The example given in the Crunchy distribution uses matplotlib. However, the way this is done is slightly cumbersome. First, some pre-existing vlam (very little added markup) must already have been added to an html page, specifying the name of the image (e.g. foo.png) to be displayed. This special vlam will result in an editor appearing on the page together with two buttons (one to execute the code, the other to load the image). Second, the Python code must be such that an image will be generated with that name. By default, the image is saved in and retrieved from a special Crunchy temp directory.

While the approach used works, it does also mean that images can't be generated and displayed using a simple Python interpreter, nor can they be displayed from an arbitrary location. At least that was the case until now.

Prompted by a suggestion from Johannes, I wrote a very simple module whose core consists of only 7 Python statements, and which does away entirely with the cumbersome vlam image file option. Images can now be loaded and displayed from anywhere using two lines of code:

import image_display
image_display.show(path, [width=400, height=400])

And by anywhere, I mean from any interactive Python element (interpreter, editor),
and using any image source (local images and images on the web), like

image_display.show('http://imgs.xkcd.com/comics/python.png')

Furthermore, one can use this approach to create a "slide show" by alternating image_display.show() and time.sleep().

Since there is no more need to use the old image_file option, it will be removed in the next Crunchy release. I may have to give some more thoughts to the API for this new option (e.g. is there a better name than image_display? Should I add a function for a slide show, giving a list of images and a time delay? etc.); suggestions are welcome.

100 posts, olpc, rst and Crunchy

2008-01-17T21:34:00.000-04:00

This is the 100th post on this blog which I started to write a little over 3 years ago - shortly after I started my programming hobby. And, as it so happens, I received two gifts in the past 24 hours:

I received my give-one-get-one olpc in the mail today.
A student wrote a simple programs to add Crunchy-specific "directives" for docutils so that rst can be used to create Crunchy ready tutorials. More on this below.

I had seen the olpc at Pycon 2007 but, not having tried it back then, I did not realise how small the keyboard really is. I have fairly small hands but they are much too big to use the keyboard comfortably. I thought of using it to write this blog entry but quickly gave up on that idea.

It will take a while to fully explore the olpc. I am extremely impressed by the quality of the screen. However, it is a slow computer which apparently can only handle a "small" number of applications running concurrently - I managed to freeze it with about 8 applications running. I seemed to remember from the Pycon presentation that the track pad had 2 (or 3, depending on how you count them) active region, but only the central one seems to be active. Members of my family were surprise at the "plastic toy" appearance of the olpc but they all seem to be fairly impressed once they saw it running.

Now, all I have to do is to figure out how to make Crunchy run on it. :-)

Speaking of Crunchy, as part of Google's HOP contest, I had set up two tasks related to reStructured Text. One of them was to write a plugin so that a rst file could be loaded, transformed into a html file (by docutils) and displayed by Crunchy. This was done a while ago as I reported here. However, when this is done, Crunchy treats all the code elements on a page (inside "pre" tags) the same way, which is specified via the variable crunchy.no_markup. This does not allow for the same fine-tuning that is possible with the addition of vlam (very little added markup) to a normal html page.

The second rst task was to write "directives" so that vlam could be inserted inside rst pages. I had read quickly about rst directives and figured this would be fairly complicated. From my superficial reading, I *thought* it would require a modification to docutils that would have to be incorporated into the core (or make a special version of docutils that incorporated those directives). Of course this would make little sense. In any event, a student wrote a 200 lines program that defined most required "rst directives" for Crunchy, allowing to take a rst file as input and output an html file. All I had to do was to cut-and-paste the student's program into the existing rst plugin, change 2 lines of code, and I now have a fully working vlam-compatible rst loader for Crunchy.

This will be part of the next Crunchy release.

Small tip for porting unit tests to Python 3.0

2008-01-10T22:02:00.000-04:00

While working on increasing test coverage of Crunchy, I encountered a puzzling change when running doctest based unit tests under Python 3.0a and Python 2.5. One test (of a function that was definitely too long) was failing under Python 3.0a1/2 with a single line containing the number 20 appearing in the output - in addition to all the other expected ones. To make a long story short, the problem was linked to PEP 3116. Using Python 3.0a1/2, when a file is written using ".write()", there is a return value corresponding to the number of bytes written; the "old" behavior was to return None. In order to have doctest-based tests running successfully under both 2.x and 3.x, we need to replace things like

f.write()

by

dummy = f.write()

I understand that the plan is to backport many of the Python 3.0 features to Python 2.6 so as to ease the transition, and use an automated tool for the conversion. I don't think that features from PEP 3116 will be implemented in Python 2.6, since they are very different from 2.5. And I am doubtful that the conversion tool will take care of including a dummy assignment for functions that will now return a value different than None. I am hoping that this post might end up saving a little bit of time for any reader that tries to migrate their code from 2.x to 3.0.

On a different topic altogether, I have fixed the bug which resulted in removing the styles from remote web pages when using Python 3.0 in conjunction with Crunchy. This will make Crunchy a more attractive tool to use in going through the official Python tutorial for Python 3.0. For the moment, the Crunchy embedded interpreter still only works with Python 3.0a1 and not 3.0a2. Hopefully I'll have this resolved in new public release by Pycon 2008 so that Johannes can demonstrate it in his talk.

More encoding pains...

2008-01-07T20:52:00.000-04:00

It seems like every early January brings some new encoding pains...

Eons ago (less than a year) I used to program on a Windows based computer. My Windows user name was simply André, which is not surprising since it's my first name. However, the observant reader may have noticed that my name requires one non-ASCII character. Such a small detail... that can cause so much annoyance.

Two years ago, on January 6th, I wrote about using site customization so that my favourite Python editor at the time (SPE) could deal with my user name.

Last year, on January 3rd, I wrote about how Crunchy dealt with encoding issues in a way that was independent of any site customizations. At the time, an astute reader made the following comment (which I had forgotten until today, when I decided to write this blog entry):

Without having looked at the rest of your code, so I might be completely off here, this somehow looks wrong:

result = result.decode(sys.getdefaultencoding()).encode('utf-8')

The reason I say this is that you're decoding and encoding in the same place. Since Python unicode support is so good, it's generally a good idea to decode to unicode any use input you get as early as possible, and to encode only as late as possible when outputting strings. Since you're doing complicated web ui stuff here, so it may be that you're not doing anything with 'result' between input and returning it to the browser, but if you are, the string should have already been decoded by the time it gets to this line. Otherwise this will bite you anytime you try to do anything with the string like simple concatenation. [emphasis added]

Of course, since Crunchy was working properly, I quickly dismissed that comment. With the way everything was implemented, Crunchy was working just fine... In fact, to this day, if you download the latest public release (0.9.8.6), everything works just fine - even if your user name includes non-ASCII characters.

However ... in adapting Crunchy to works with Python 3.0, I do things with the various strings like simple concatenation ... and, of course, this cause some problems as I found out when I "borrowed" the old Windows computer from my daughter to try the latest changes I had made.

It. Did. Not. Work.

Ok, so I have two possible solutions:

Trade with my daughter for a while, letting her use my MacBook (which she loves!) while I use the "old" Windows desktop.
Create an account with an accented name on my MacBook.

Well, as much as I love my daughter, I could not face the pain of going back to using the Windows desktop as my main computer. So, solution 2 was an easy choice.

Except that it wasn't....

My account name on the mac is my full name (André Roberge). Hmmm... this has already a non-ASCII character. But my home directory is "andre" - no accent. Under Mac OS, a user has a full name that appears in the login window, and a short name used for the root directory (/Users/andre in my case). When I tried to create a new account with a non-ASCII character in the short name, it just beeped and refused to enter it.

In search for an answer, I posted on three Mac related forums, got either no answer or some unhelpful and wrong answer on two of them ... I was considering posting on the Python list, but, fortunately, I eventually got a useful answer.

So, if you are thinking of writing i18n applications that either can run from any directory or that save information in the user's default directory, or both, and you want to make sure that it will work on a Windows computer, here's how you can do it on a Mac (under OS X 10.5):

Create a test account user the account manager under any name of your choosing; however, the "short name" will have to be ASCII only (at this stage).
From the account manager, ctrl-click on the account after it has been created; this will bring an advanced dialog.
Edit the advanced dialog to change the short name, and the home directory, to the desired value. I chose the name "accentué" (self-referencing name, if you know French). Note that doing so does not change the actual name of the directory.
Go to a terminal window and do "sudo mv old_name new_name" to change the name of the home directory that was created at step 1.

After I did all this, the development version of Crunchy did not work from the new account. This pleased me very much: it likely will meant that I do not have to trade computers with my daughter to continue working on Crunchy. ;-)

More about Crunchy running under Python3.0a1

2007-12-30T22:21:00.001-04:00

Just in time for the new year, and after a major reorganization of Crunchy's code to make it easier to write more unit tests for it, I did manage to get all of Crunchy's core functionality working under Python 3.0a1. Crunchy can now import html files that are not conforming to strict xml convention (i.e. not having closing tags for some elements) and thus create problems for ElementTree. The way it does that is to launch an external process under Python 2.5 (or whatever default version that is invoked via typing "python" at a terminal - but it has to be a 2.x version for it to work). This process imports the file, "cleans" it (using a combination of BeautifulSoup, ElementTree and Crunchy's security module), and saves it locally. After a delay, Crunchy loads up the cleaned up file and display it. In this process, most of the styling is unfortunately removed. :-(

However, the good news is that I was able to load up the official 3.0 Python tutorial (work in progress) and try it out using Crunchy. I did find out one limitation of using Crunchy to do so. Crunchy encodes the Python output using utf-8 before forwarding to the browser. So, instead of having things like b'Andr\xc3\xa9' appearing on the screen, it would be converted to André. Thus, Crunchy is not a very good platform to teach about encoding/decoding of strings. For other aspects though, it is an ideal tool (if I may say so) for going through the Python tutorial: there is no need to switch back and forth between the browser and a separate Python environment to try things out. I still hope to have the time and energy to go through the entire 3.0 tutorial (something I have never done before, for 2.x) and see if I can find any bugs or come up with useful suggestions.

All I have left to do for the next release is to write up some documentation/tutorial on the new Turtle module and on launching Crunchy under Python3.0a1. With a bit of luck, this will all be finished before the end of the year.

In addition to the code reorganization mentioned above, I did fix a few bugs and made an improvement on Crunchy's Borg interpreters. For those that aren't familiar with it, Crunchy allows to embed a number of interpreters (html input box communicating with the Python backend) within a single page. These interpreters can either be isolated one from another (meaning that a variable defined in one interpreter is only known by that interpreter) or can share a common environment (aka Borg interpreters). Normally, in a single user mode, using a single open tab in Firefox, every time a new page is displayed, the Borg interpreters are effectively reset (the old ones are garbage collected and the new ones are created from an empty slate).

Previously, if one were to have multiple users (or multiple tabs open from the same user) on the same Crunchy server, all Borg interpreters ended up sharing the exact same state. This is not very convenient if two users are trying to use the same variable names! I had planned to address this "feature" at some point after the 1.0 release but was forced into it earlier due to the 3.0a1 work. The reason is the following.

To create "Borg interpreters", I was using the Borg idiom invented by Alex Martelli. It goes as follows:


class Borg(object):
    '''Borg Idiom, from the Python Cookbook, 2nd Edition, p:273

    Derive a class form this; all instances of that class will share the
    same state, provided that they don't override __new__; otherwise,
    remember to use Borg.__new__ within the overriden class.
    '''
    _shared_state = {}
    def __new__(cls, *a, **k):
        obj = object.__new__(cls, *a, **k)
        obj.__dict__ = cls._shared_state
        return obj

When using this idiom in a standard program under 3.0a1/2, a deprecation warning is raised about object.__new__() not taking any argument. When I was trying to make use of this idiom in Crunchy running under 3.0a1/2, the deprecation warning was actually replaced by an exception. Rather than trying to silence this exception, I decided to take a different approach and used instead the following:


class BorgGroups(object):
    '''Inspired by the Borg Idiom, from the Python Cookbook, 2nd Edition, p:273
    to deal with multiple Borg groups (one per crunchy page)
    while being compatible with Python 3.0a1/2.
    Derived class must use a super() call to work with this properly.
    '''
    _shared_states = {}
    def __init__(self, group="Borg"):
        if group not in self._shared_states:
            self._shared_states[group] = {}
        self.__dict__ = self._shared_states[group]

# The following BorgConsole class is defined such that all instances
# of an interpreter on a same html page share the same environment.

class BorgConsole(BorgGroups, SingleConsole):
    '''Every BorgConsole share a common state'''
    def __init__(self, locals={}, filename="Crunchy console", group="Borg"):
        super(BorgConsole, self).__init__(group=group)
        SingleConsole.__init__(self, locals, filename=filename)

The "group" variable is taken to be a unique id generated by Crunchy each time it processes a given html page. Thus, each page loaded by a different user (or the same user, at a different time) from the same Crunchy server will result in a unique set of Borg interpreters.

To be fair, I must admit that I did not come up with this solution totally on my own. A while ago, I asked for something like this on comp.lang.python. (Those interested in the details should search for "Borg rebellion".) I just derived the above solution from some suggestions made at that time.

Finally, in addition to all this, I found out a bug in code.py for Python 3.0a1/2. I tried to send an email to the python-3000 mailing list about it, but it was held up, waiting for a moderator approval for a few days. So, I canceled it and filed a bug report instead (which I should have done in the first place) on the bug tracker. I still haven't seen any follow up - perhaps due to the title I gave it. The bug is actually very easy to fix - three lines of code need to be replaced by a single one. The solution is related to my only other "official" contribution to Python to date. Hopefully, by this time next year, I'll have learned enough to contribute more to Python.

Crunchy and Python 3.0a2

2007-12-27T22:30:00.001-04:00

Continuing with my experiment of adapting Crunchy to Python 3.0, I managed to get Crunchy to start with Python 3.0a2 and get some code running from the editor - but not from the interpreter, nor the doctest. Most of the problems I have are dealing with bytes-to-string conversion and string-to-bytes. As mentioned by Guido van Rossum last June

We're switching to a model known from Java: (immutable) text strings are Unicode, and binary data is represented by a separate mutable "bytes" data type. In addition, the parser will be more Unicode-friendly: the default source encoding will be UTF-8, and non-ASCII letters can be used in identifiers

Later on, in a comment from that post, we find:

> In your presentation last night you had one slide which
> talked about the "str" vs "bytes" types in Python 3000. On
> the bottom of that slide was something like:
>
> str(b"asdf") == "b'asdf'"
>
> However, in discussing this slide (very briefly) you said
> that a type constructors like "str" could be used to do
> conversion. It seems like "str" is behaving more like
> "repr" in this case, which seems unusual and less useful
> to me. Was this a typo, or is this actually the way it's
> supposed to work? What's the rationale?

To be honest, this is an open issue. The slide was wrong compared to the current implementation; but the implementation currently defaults to utf8 (so str(b'a') == 'a'), which is not right either. The problem is that there are conflicting requirements: str() of any object should ideally always return something, but we don't want str() to assume a specific default encoding.

To be continued...

This change seems innocuous enough...

As a web server, Crunchy sends to and receives information from the browser as "binary data" or "bytes". As a generalized Python interpreter, Crunchy manipulates the information as "strings". It appears that the "bytes" implementation is done much more completely in Python 3.0a2 than it was in Python 3.0a1. And this is the source of many problems.

For example, Crunchy sends from the browser some information about the path to which a Python file should be saved and its content as follows:

'/Users/andre/.crunchy/temp.py_::EOF::_from Tkinter import *\nroot = Tk()\nw = Label(root, text="Crunchy!")\nw.pack()\nroot.mainloop()'

This is sent as a binary stream which needs to be converted to the string written above. This conversion is done via str(...). Using Python 3.0a1 (and 2.4 and 2.5), the result was as above; splitting the string gave the following:

['/Users/andre/.crunchy/temp.py', 'from Tkinter import *\nroot = Tk()\nw = Label(root, text="Crunchy!")\nw.pack()\nroot.mainloop()']

Now, with Python 3.0a2, it gets slightly more complicated. The first string acquires a "b" prefix upon conversion (as mentioned in the comment from Guido's blog mentioned before). After splitting, the result is

["b'/Users/andre/.crunchy/temp.py", 'from Tkinter import *\\nroot = Tk()\\nw = Label(root, text="Crunchy!")\\nw.pack()\\nroot.mainloop()\'']

So, we now have a first string with a "b'" prefix embedded in it, and a second one without. It seems that each case will have to be handled carefully on its own. And I suspect more problems will show up as we get closer to the final 3.0 release.

I know, I know, I'm really not following the "recommended" practice, as quoted on Guido's blog. I should probably wait first for Python 2.6 to come out. Then, I should have a complete unit test coverage and use the conversion tool to create a Python 3.0 version .... However, I am not convinced that the conversion tool will be smart enough to know when a function (that I write) expect a "str" object and when it expect a "byte" one. Furthermore, the few unit tests I had worked fine under both Python 2.5 and 3.0 ... but some functions that I had written with the expectation that they would receive some string arguments did not work in "production code", as they were getting some bytes arguments. And this failed completely silently...

If I had to give some advice to someone about creating Python programs that can work with both Python 2.x and Python 3.x, I would say like Guido: don't. :-) Unless of course you are like me and are doing this for fun and to get to learn about the differences between Python 2.x and 3.x along the way. But then, "be prepared for the unexpected" like the following: turning on a few print statements (via a "debug flag") can result in breaking some code; turn them off and the code works again... Yes, it did happen to me - I still have to figure out how...

Crunchy and Python 3.0a1

2007-12-27T16:27:00.001-04:00

It is often said that a picture is worth a thousand words...

I have managed to make Crunchy run under Python 3.0a1. Some of the features are not working but the interpreter and the editor work. The turtle module I have been working on also works "nicely" (read: as slow as before) with this new Python version. Unfortunately, when it is run under Python 3.0a1, Crunchy can not load most pages - including those of the official Python 3.0 tutorial. The reason is that is uses ElementTree to parse pages and it is unforgiving when it comes to having unclosed tags (as in <link> and <meta...> for example); it also seems to not be able to handle the <script>s that are included on the page. I have not yet found a way to reliably "clean" the pages before parsing them with ElementTree. While I believe that I should be able to do so with a bit more work, there is a bigger problem...

Unfortunately, Crunchy does not run under Python 3.oa2, and the error messages I get have not been too helpful in figuring out the error. However, perhaps this is due to a faulty installation. What makes me think so is that when I start a 3.0a2 session at a terminal, I get an error message when I use exit(). This is most unexpected.

In any even, the next release should include the new crunchy turtle module and be usable with 3.0a1. Perhaps Johannes, or some curious user, will be able to figure out how to make it run under 3.0a2 as well.

Slow turtle ... in time for Xmas

2007-12-25T16:18:00.001-04:00

One of the task assigned in Google's HOP contest was to design a simple turtle graphics module for Crunchy. This was done successfully by a student as a prototype. This prototype had some unfortunate limitations in terms of number of turtles and simultaneous graphics canvas existing on the same page, but it did give me the impetus to use the student code as a proof-of-concept and implement a more complete turtle module for Crunchy.

Playing with turtles, and trying to draw fairly complex shapes, made me realize that the combination of using an html canvas and the Crunchy comet communication makes for an extremely slow turtle. It would be really nice to find a better (faster) way.

The next Crunchy release should include that turtle module ... and an additional bonus: Crunchy can now be launched successfully using either Python 2.5 (or 2.4) and Python 3.0a1. And the turtle module works with both.

At the moment, not all of Crunchy's features are supported when using Python 3.0. However, this should no longer be the case by the time version 1.0 comes out.

And, for those that might be tempted to point out Guido's blog entry about not making programs compatible with both Python 2.x and 3.x, please don't bother. I realize that it is not wise in general to try to do so. However, given Crunchy's design philosophy to make it as easy for students/teachers/tutorial writers to use - it just does make sense: download, unzip, double-click; nothing else should be needed to start having fun with Python - no matter what new Python version gets installed.

(NOT) Bitten by PEP 3113

2007-12-18T23:22:00.000-04:00

UPDATE: The comments left on this post (1 and 3) in particular corrected my misreading of PEP 3113. There is no such wart as I describe in Python 3.0. I should have known better than to question GvR and friends. :-) I'm leaving this post as a reference.

In trying to make Crunchy useful & interesting for beginning programmers to learn Python, I designed a small graphics library following some "natural" notation. As an aside, Johannes Woolard is the one who made sure that this library could be easily used interactively within Crunchy. I mention his name since too many people seem to assume that I am the only one involved in Crunchy's design. Anyway, back to the library...

In that library, the function used to draw a line between two points uses the syntax

line((x1, y1), (x2, y2))

for example: line((100, 100), (200, 200))

which should be familiar to everyone. Unfortunately, following the implementation of PEP 3113 in Python 3.0, this syntax is no longer allowed. This is ... annoying! There are two alternatives I can use:

line(x1, y1, x2, y2)

for example: line(100, 100, 200, 200)

or

line(point_1, point_2)

where point_a = (x_a, y_a). Update: with this second definition, it will be possible to invoke the function as
line((100, 100), (200, 200))

Of course, either of these two option is easy to implement (and is going to be backward compatible with Python 2k). However, I don't find either one of them particularly clear for beginners (who might be familiar with the normal mathematical notation) and do not consider this a (small) wart of Python 3k.

reStructuredText files and Crunchy

2007-12-18T20:00:00.001-04:00

Crunchy can now handle reStructuredText (.rst) files in the same way it can process plain html ones! This requires the user to have docutils installed - which is normally the case for anyone that writes .rst files.

The test coverage for Crunchy is slowly improving. Currently, 10 modules are mostly covered by doctest-based unit tests, out of approximately 40. Since I make use of .rst files to keep the unit tests, these can now be browsed "pleasantly" using Crunchy itself.

Furthermore ... all the unit tests written so far work under Python 2.4, Python 2.5, and ... Python 3.0a1! This required some tedious rewriting of some parts of the code but the end result is well worth it - if only to really learn about differences between Python 2.5 and Python 3.0.

One thing that I found, which will be no surprise to TDD aficionados, is that code written without testing in mind can be quite tricky to write comprehensive tests for. Add to this the extra complication of making that code run under two incompatible Python versions, and you are on your way to major headaches. It's a good thing I am doing this only for fun!

Launching Python 3.0 program from Crunchy running under Python 2.5

2007-12-08T00:04:00.000-04:00

As part of Google's Highly Open Participation contest, Michele Mazzoni completed the task of creating a new option for Crunchy: one can now launch (starting with the next release of Crunchy - 0.9.8.5) a program using a different version of Python than the one used by Crunchy itself. While I had suggested that the alternate Python version could be set via the configuration options for Crunchy (usually accessible from a Python interpreter), Michele had the brilliant idea to add a simple input box where one can specify the path (or 'alias') of the Python version used right on the page where the program is launched from. This makes it extremely easy to change the interpreter version used to launch a user written program.

Michele has prepared a screencast demonstrating this, which should appear on ShowMeDo hopefully soon.

Thank you Michele - and thank you Google!

More results from GHOP

2007-12-04T07:50:00.000-04:00

Google's Highly Open Participation (GHOP) contest is attracting a lot of attention from the right people: pre-university students. The PSF is one of ten organizations mentoring students working on Python-related projects. Since I submitted tasks suggestions early on and volunteered to help following a call for volunteers from Titus Brown, Crunchy has benefited from many students contributions. Crunchy's messages have been translated in Estonian, Macedonian, Polish and Italian with, hopefully, more translations to come. Some new unit tests have been added with more to come. There may be a couple of nice surprises coming out soon too :-)

While other projects have also benefited from GHOP's students contributions, there could be more. If you have some good ideas for mini-projects (doable in 3-5 days, at a couple of hours per day with perhaps one full day), your suggestions would most likely be most welcome. Just check out the GHOP Python discussion group. And, if you would like to join the (too small) ranks of Python mentors, please do; we need all the help we can get.

First Crunchy screencast

2007-11-30T22:09:00.000-04:00

I finally got around to produce a first Crunchy screencast. Since Crunchy is an interactive program, I thought I should do screencasts that reproduced the interactive feel: that is to say, I would record a live, more or less improvised session. This means that the screen cast is not as polished as it could be - an even has a few minor mistakes in it. Still, I think it gives a good (superficial) overview of what Crunchy is capable of.

What do you think?

First result from GHOP: Crunchy Estonian

2007-11-28T21:16:00.000-04:00

By now everyone has probably heard of Google Highly Open Participation contest, and the participation of the PSF. And, initially through serendipity but later with more active involvement, a few Crunchy-related tasks were created including some involving translations. And, just a day after the contest started, a first translation (in Estonian!) has been made for Crunchy by a student named Tanel.

The tasks assigned are usually not very big but they provide nice stepping stones for getting people involved in open source software. Thank you Google for this wonderful idea.

A failed experiment

2007-11-27T20:04:00.000-04:00

On a whim, I decided to try a fun experiment: have Crunchy run correctly when invoked either from Python 2.4 or Python 2.5 ... or from Python 3.0! I thought that if I could make it to work, it would be a great tool to go through the 3.0 docs and tutorial, and finding out if there were any mistakes. (I was planning to do this, hoping to help the developing team in my own limited way.)

In the end, I had to give up, as Crunchy uses third-party modules (ElementTree, HTMLTreeBuilder, ElementSoup, BeautifulSoup, etc.) that I could not make compatible with both Python 2.x and 3.0 without essentially rewriting them in two separate modules each. Actually, this was already done for ElementTree (included in the stdlib under a different name) but not for the others.

The main stumbling block was string handling/encoding... This should not be a surprise to anyone who has followed the Py3k development.

Still, I've learned a fair bit from that experiment. One thing that I sort of knew already related to using print statements for debugging purpose. I often sprinkle my code with "if DEBUG: print ...", having multiple print statements appearing. Often I find that I want to have a few debug flags in the same file and think to myself

Shouldn't I replace these print statements by a debug() function with a variable setting the debug level ...

If I would have done something like this, I could have had just a few print statements located in a debug.py module. As it was, with the change in Py3k for print becoming a function, I had to do a lot of changes by hand. Yes, it might have been possible to use the automated 2to3 tool ... but I wanted to maintain compatibility with Python 2.x and, more importantly, get a feel for what was involved in making the code Py3k compatible.

If or when BeautifulSoup, ElementSoup and HTMLBuilder become Py3k compatible, I'll have to give it a try again...

Choosing a CMS ... Will Python make it

2007-10-16T14:38:00.000-03:00

This post is not related to programming per se; feel free to skip it ...

"We" are in the process of revamping our web site, and use a CMS. We have just signed off on the first part of the project (new look and wireframe) and are looking at the second (and "final") stage. As we are too small to afford a commercial solution, we are looking at open source CMS. So far, the preferred choice of our director of technology is Alfresco. The choice recommended by our usual technology provider is Joomla! My favourite would likely be Plone - given it is written in Python ;-)

Normally, we would proceed with a public tender - but given that there are few local providers, we may just proceed with the same firm that provided services in phase one. However ... If some of you reading this have work experience in this area, and would be interested in such a project, have a look at our current web site, and give me an informal time/cost estimate of migrating this website to use a [Python based? ;-)] CMS. If we do proceed to tender, you would most likely be included in our list of firms contacted to submit an offer.

pyglet and pygame: which one?

2007-08-31T21:28:00.000-03:00

Pygame is great. Pete Shinners has done some fantastic work in creating it. I have played a bit with it in the past and was planning to learn to use it better, and probably write some Crunchy-based tutorials for it. Unfortunately, this will not happen. Since I moved from a PC to a Mac, and installed Python 2.5, I can not use Pygame, as there is no ready-made version for my setup. Yes, I imagine I could find out how to compile it from the source ... but there is an easier alternative: pyglet. pyglet is a pure python library, with no external dependency. I downloaded it, tried it ... and it just worked. Not only that, but when Python 2.6 will come out, I know it will still work.

I do realise that pyglet is just in alpha stage ... but it's a very impressive alpha. It is not as fully featured as Pygame is ... but it works for me, thanks to Alex Holkner.

If you're currently using pygame, there's probably not any reason to switch. But if, like me, you find that you can not use pygame with your current setup, check out pyglet. You might be pleasantly surprised.

Python 3.0a1 has been released!

2007-08-31T15:03:00.001-03:00

Congratulations to all Python developers on the release of Python 3.0a1!

My favourite language is on its way to get even better!

Crunchy-like sites for javascript and ruby

2007-08-07T18:10:00.000-03:00

While I have been aware of the Try Ruby site for quite a while, recently I came across this interactive javascript tutorial. While none of these are nearly as flexible as Crunchy, they are nice resources that are currently available for anyone to try without needing to install anything. It would be nice if I could have a dedicated server running Crunchy in a sandbox so that people could try it out without having to download it.

flat is better than nested?

2007-08-07T18:00:00.000-03:00

In the process of making Crunchy more secure, I had extra attention to css files that contained import statements, since these can be used to insert javascript code. I was rather surprised when I looked at www.python.org. First, I noticed the line

  <link rel="stylesheet" type="text/css" media="screen"
  id="screen-switcher-stylesheet" href="/styles/screen-switcher-default.css" />

which, in itself, appears rather ordinary. Upon following the link, I found that the content of the css file consists of the single line:

@import url(../styles/styles.css);

In its default security mode, Crunchy just ignores any style information that contains "url(" in it. As a result, when viewed using the default security mode for Crunchy, the python.org site's formatting is lost.

I wonder why the link redirection is used...

Crunchy Security Advisory

2007-07-28T20:40:00.000-03:00

A security hole has been uncovered in Crunchy (version 0.9.1.1 and earlier).

Anyone using Crunchy to browse web tutorials should only visit sites that are trustworthy.

We are working hard at fixing the hole; a new release addressing the problems that have been found should be forthcoming shortly.

-----

The security problem is as follows:

In theory, a web page could contain some javascript code (or link to such code) that would bypass Crunchy's filter to be executed by the browser. If that is the case, the javascript code could be designed to send some Python code directly to the Python backend (i.e. without the Crunchy user pressing a button, or having the chance to view the code to be executed) so that it is executed. Such code could result in deleting the entire files or installing some virus on the user's machine.

At the moment, the risk is pretty low. Crunchy already removes all obvious (and most non-obvious) javascript code, links to such code, etc. The holes found require the use of some uncommon combination of html and css code, with a particular knowledge of Firefox.

(Note that browsers other than Firefox are likely to be even more vulnerable).

Furthermore, Crunchy is not that well known that it is likely to be a target by a cracker that would 1) write a "tutorial" interesting enough to lure current Crunchy users (who, at this point, are likely to include only advanced Python users) and 2) write some fairly involved javascript code to bypass the second security layer (where the commands enabling communication between the browser and crunchy are made up of random string generated uniquely at each new Crunchy session).

If anyone is interested in security issues related to Crunchy, feel free to contact me directly.

Rur-ple 1.0rc2 : bug fix for wxPython 2.8

2007-07-10T21:08:00.000-03:00

Three changes; a grand total of 3 lines of code. That's all I needed to change to make rur-ple work with wxPython 2.8 (and probably 2.7). It only took me 8 months to get around to fix it due to a combination of lack of time and, when I had time, my obsession with Crunchy.

Speaking of Crunchy, I found a bug today: any Python output that has something like

< ..... >

in it will be such that the stuff between angle brackets will not be shown. This is because the browser thinks it is a valid tag. The first time I noticed it was when I tried to do help(cmp). Then, I was going through "how to think like a computer scientist" (before I point my son to it) and noticed that the exercise with type(...) did not work. So, it is time to do a bug fix release - but no official announcement of this minor release. I'm curious to see if I'll hear from someone about this bug prior to the next "major" release.

New Crunchy (0.9) is out

2007-07-09T20:52:00.000-03:00

By now, the official announcement should have made it to many lists. After a mad rush of coding and writing documentation over the past week or so, and implementing lots of new cool features, the new Crunchy is officially taking over from the old one.

I believe that this new version can do much to promote Python usage. If you do agree, please mention it!

Here's a copy of the announcement I sent to various lists:

Crunchy 0.9 has been released. It is available at http://code.google.com/p/crunchy

What is Crunchy?

Crunchy is a an application that transforms html Python tutorials into interactive session viewed within a browser. We are not aware of any other application (in any language) similar to Crunchy. Currently Crunchy has only been fully tested with Firefox; we know that some browsers simply don't work with it. Crunchy should work with all operating systems - it has been tested fairly extensively on Linux, Windows and Mac OS.

What is new in this release?

Crunchy has been rewritten from scratch from the previous version (0.8.2), to use a custom plugin architecture. This makes easier to extend and add new functionality. Rather than list the differences with the old release, it is easier to list the essential features of this new version.

1. Crunchy can work best with specially marked-up html tutorials. However, it can now work with any html tutorials - including the official Python tutorial on the python.org site. Html pages can be loaded locally or remotely from anywhere on the Internet. Crunchy uses a combination of Elementtree and BeautifulSoup to process html pages. Non W3C-compliant pages can be handled, but the visual appearance is not guaranteed to reproduce that normally seen using a browser.

2. Crunchy can insert a number of Python interpreters inside a web page. In the default mode, it does that whenever it encounters an html [pre] element which is assumed to contain some Python code. These interpreters can either share a common environment (e.g. modules imported in one of them are known in the other) or be isolated one from another.

3. Crunchy adds automatic css styling to the Python code - you can look at the official Python tutorial using your browser (all Python code in blue) and compare with what Crunchy displays to give you a good idea.

4. Instead of inserting an interpreter, Crunchy can insert a code editor that can be used to modify the Python code on the page and execute it. The editor can be toggled to become a fairly decent syntax aware editor that can save and load files.

5. Crunchy has a "doctest" feature where the code inside the [pre] is taken to be the result of an interpreter session and the user has to write the code so as to make the interpreter session valid; this is useful in a teaching environment. Messages from the Crunchy's doctest are "friendlier" for Python beginners than the usual tracebacks.

6. Crunchy has a small graphics library that can be imported, either inside an editor or an interpreter, to produce simple graphics (even animations!) inside the browser.

7. For the user that needs better quality graphics, Crunchy supports programs (such as matplotlib) that can create image files; by executing the code, the image produced is loaded inside the browser window. In this capacity, Crunchy could be used as a front end for libraries such as matplotlib.

8. Crunchy supports code execution of files as separate processes, making it suitable to launch gui based application from the browser window.

9. Crunchy's interpreter has an interactive "help" feature like many python-aware IDEs.

10. Crunchy includes a fairly comprehensive tutorial on its own use, as well as a reference for tutorial writers that want to make their tutorials "crunchy-friendlier".

11. As a security feature, crunchy strips all pre-existing javascript code from an html page before displaying it inside the browser window.

Bug reports, comments and suggestions are always welcome.

André Roberge, for the Crunchy team.

Fun with the new Crunchy

2007-06-24T22:54:00.000-03:00

Since the new Crunchy can work with files that have no Crunchy-specific additional markup, some neat experiments are possible, in addition to playing with the official Python tutorial with a Python interpreter embedded on the web page that I mentioned in my last post. For the curious among you, here's something else to try.

Assuming you have the latest (version 0.2) alpha release of the new Crunchy, edit the file vlam.py, replacing "interpreter" by "editor" at lines 108 and 109. Then, launch Crunchy and click on the "tests" link. On the following page, click on the "Loading arbitrary tutorials" link. Then, in the box for loading remote tutorials, enter the address of the Python cookbook (http://aspn.activestate.com/ASPN/Python/Cookbook/).
The formatting will be off, but you can select your favorite recipe. When you do this, you can edit the code and execute it right on the page. Often, all you need to do is to replace __name__ == "__main__" by True and you are ready to try the examples, modifying them at will.

In the near future, Crunchy will provide an easy way to select which interactive element (Python interpreter or editor) is to be inserted by default, without having to edit the code by hand.

From vlam to nam: Using Crunchy to interact with the official Python tutorial

2007-06-24T16:21:00.001-03:00

Breaking news: you can now use Crunchy to browse and interact with the official Python tutorial. More below...

Since my last post that described briefly some features of the new Crunchy engine, I have not had much time to work on Crunchy. Johannes too has been fairly busy so that Crunchy development pretty much stopped, until the beginning of Google Summer of Code. Two students joined the Crunchy team: Edin Salković and Bryan Psimas. Unfortunately, due to other commitments, Edin was not able to continue with his SoC project - but he did manage to write the prototype of a new plugin for Crunchy that I will describe below.

The new Crunchy engine now works extremely well. As I wrote in my previous post, Crunchy is now more responsive; it supports simultaneous execution of multiple code sample, proper input (stdin) handling from the webpage browser and the api (for writing tutorials) has been significantly simplified as there is no longer be any need to have an embedded "canvas" call in a tutorial: the user is now able to load a graphics dynamically anywhere on a page. These graphics can even include simple animations using pure Python code (for the end user - javascript behind the scene for us, unfortunately).

There is more. For example:

it is now possible to style <code> elements;
Python code sample (either inside <pre> or <code>) that contain pre-existing html markup can now be handled by Crunchy without generating an error (blank page!) as before;
it is now possible to specify a starting number different from 1 when requesting that line numbers be added to a code sample;
when it styles the code, Crunchy now automatically detects if a given code sample represents an interpreter session or a "normal" code sample;
it is now possible to use input() and raw_input() with an interpreter, and the result appears inside the page displayed by the browser.

In addition, thanks to Edin's work, a new type of interaction with Python code has been added: it is possible to save the result of a computation in a graphics file and display the result in the browser. This has been used successfully with matplotlib. There is a "small" bug in that the Python code needs to be executed twice due to synchronization issues with the browser loading the file and Python saving it.

Quite a few features from the "old Crunchy" need to be implemented (e.g. menus including a way to properly quit the application, translations, update:~~the ability to load an arbitrary tutorials given a URL~~, etc.) in the new version but, overall, it is working very well.

One of the "problems" with the old Crunchy is that it did require two things from a tutorial writer:

That the html code written be W3C compliant (with no warning, no unclosed tags, etc.).
That Crunchy specific markup (vlam = very little additional markup) be added to instruct Crunchy to add interactive elements.

Unfortunately, the official Python tutorial fails on both accounts. However, with the use of a version of BeautifulSoup adapted by effbot (thanks!) to be used with ElementTree, the new Crunchy can now deal with non W3C compliant code.

And ... Crunchy has now been written so that it automatically inserts a Python interpreter whenever it sees a bare <pre> (i.e. with nam = no additional markup). As a result, one can now browse the official Python tutorial with Crunchy and interact with it.

This automatic insertion of an interpreter sometimes yields too many interpreter inserted on a page than really needed; it is better to hand craft a tutorial. However, it should make Crunchy a lot more useful to many more people.

For those interested in trying out the "new Crunchy", an alpha release is available. If you have never used Crunchy before, you should try version 0.8.2 first and go through the tutorial from the menu.

New Crunchy engine

2007-03-06T20:21:00.000-04:00

At the end of my talk at Pycon 2007, I gave an extremely brief demo of the new (not in production yet) upcoming version of Crunchy. Thanks to the work of Johannes Woolard, Crunchy's javascript core engine has been completely rewritten. As a result, Crunchy will be more responsive; it will also support simultaneous execution of multiple code sample, proper input (stdin) handling from the webpage browser and the api (for writing tutorials) will be greatly simplified as there will no longer be any need to have an embedded "canvas" call in a tutorial: the user will be able to load a graphics dynamically anywhere on a page.

But that is not all...

Inspired by a side remark by Ivan Krstić (of OLPC fame) about finding the best way to design plugins in Python, I contacted Johannes to discuss the idea of using plugins to extend Crunchy. After a few back and forth emails, while I was on the road on business trips, Johannes just went ahead and implemented a first (then a second...) way of using plugins to extend Crunchy. In the end, it looks like we'll have to rewrite almost completely (like we did during the Summer of Code 2006) Crunchy's code as we move from one minor version (0.8) to the next (0.9). However, the result will be well worth it.

The unstated goal: to make it (almost) as easy for developpers to add new capabilities to Crunchy as it is for tutorial writer to use Crunchy to create interactive tutorials. Ok, this might be a slight exaggeration ... but not much of one ;-) And most of this will be the result of Johannes' great work. So, for all of those employers (and there were many at Pycon) looking for a brilliant Python programmer to hire over the summer, I know of an Oxford student who certainly fits the bill.

Pycon experience

2007-02-26T19:49:00.000-04:00

Pycon 2007 is now officially over. It was very nice to be able to associate faces to so many familiar names from the Python community and get the opportunity to talk with a few of them.

My talk didn't go as smoothly as I had hoped. When I practiced it, it seemed to always take too long. So, when I had to do it in front of a fairly large crowd, I rushed through it a bit too quickly. Still, Crunchy did seem to generate a fair bit of interest and, in addition to the demo in my talk, I got the opportunity to demonstrate it "one-on-one" to about a dozen people who all seemed to be very keen.

Also, the idea of creating a special Python logo ("web badge") for identifying Crunchy-ready tutorials, which I had alluded to in a previous post, will likely be given the ok by the PSF. As soon as it is confirmed, I'll make an announcement here.

Since my last post, the PEP I wrote for keeping raw_input()'s functionality in Python 3000 [but renaming it "input()"] had been accepted. On Monday, I took part briefly in a Python 3000 sprint where I got to implement the required "raw_input(...) -> input(...)" and "input(...) -> eval(input(...))" conversion tool for the general tool that will be available to convert Python 2.x programs so that they work with Python 3k. I also implemented the corresponding simple unit tests as well a few missing tests for the "xrange() -> range" conversion. I would not have been able to do this if Guido van Rossum had not taken the time to explain a few things to me about the conversion framework. In the end, Guido checked in my changes himself. I get the feeling that he could have done the entire thing in about the same time it took him to explain to me the tricky bits. Nonetheless, I do get the satisfaction of having, with the PEP and the conversion tool, given back a bit to the Python community. Hopefully, with Crunchy, I (and Johannes, of course) will give back even more.

Switched to Mac OS

2007-02-12T19:58:00.000-04:00

I'm a statistics... I just got a Macbook and will likely use it as my main development platform soon. So far, moving from Windows XP to Mac OS has been fairly painless. I have not done any Python programming but did try out Crunchy and Rur-ple. I found out that Crunchy requires Python 2.4+ (I had 2.3.5 installed on the Mac), that it does not work with Safari, and that there is a "bug" so that the "tooltip" does not work with Firefox on the Mac. I was planning to bring my Macbook to Pycon but will have to reevaluate as I'd like the Drunchy demo to go as smoothly as possible. There also does not appear to be a sound module on the Mac similar to winsound or ossaudiodev... I also found a new bug with rur-ple, but have no time to look into it before Pycon.

Not having taken side in the emacs/vim religious war, I have to find a decent programming environment. On my PC, I used SPE which I really liked but I have read a lot of good things about textmate and nothing negative (other than the fact that it is not free). Any advice from Mac/Python users?

Rur-ple segmentation fault on Ubuntu: solved?

2007-02-05T23:46:00.000-04:00

Some kind readers have left comments on this blog regarding segmentation faults occurring when trying to run Rur-ple using Ubuntu linux. I finally got around to installing a vmware version of ubuntu so that I could check for myself and believe I found the source of the bug. I would appreciate if someone actually using Ubuntu could verify.

Near the bottom of rur_start.py, a splash screen is started with 100 (milliseconds) as a default. Changing that number to 1000 removed the segmentation fault when running rur-ple in a virtual ubuntu environment.

There is at least one other bug in rur-ple when using a version of wxPython greater than 2.6; it is in lightning.py. Changing "event.KeyCode()" to "event.GetKeyCode()" apparently solves the problem.

Of course, I would be interested in hearing about any other bugs that have been encountered so that I could do a final cleanup before finally releasing the 1.0 version.

Lightning talk?

2007-01-31T16:55:00.000-04:00

Living in a small village in Nova Scotia, 300 km from the nearest city (Halifax, pop.: 380,000), the opportunities to discuss face to face with other pythonistas are non-existent. This is one reason I've been really looking forward to going to my first Pycon. In addition to my scheduled talk, I have been thinking about giving a lightning talk of a different kind, but still touching upon a subject dear to me, that is the promotion of Python and its use as a teaching language. While lightning talks are so short that they normally touch upon only one topic, I was thinking of actually covering three topics, under the common theme of "interactive tutorials". These three topics are, from the more serious to the semi-frivolous:

1. The fate of input() and raw_input() in Python.
2. Using a distinctive logo for promoting interactive tutorials using Python.
3. Creating an interactive tutorial repository.

According to this, both input() and raw_input() are supposed to disappear in Python 3.0. After thinking about it for quite some time, I posted a message on edu-sig about the proposed change last September, expressing my "opposition" to the proposed change, and seeking the opinion of others on this topic. After a while, a near-consensus emerged, supporting the idea of keeping at least something like raw_input() in Python 3.0. I then wrote a pre-PEP, which generated more positive comments and encouragements but otherwise no further action from other edu-sig subscribers more involved in the Python community than I am. Unfortunately, due to my work, I had to drop all my Python-related activities for a few months and was not able to follow up until late December. With the end of the year approaching, I found the time to incorporate the comments received on edu-sig and offered a revised pre-PEP to the Python 3000 list on December 22nd. This was a single posting from me as I don't subscribe to that list. A bit of discussion ensued after which Guido stated that he like the proposal better than any of the other alternatives mentioned and, later, suggested that someone should "clean up and check in the proto-PEP and start working on an implementation or patch."

Unfortunately, and perhaps because this happened just before the holidays, nothing further was done. So, I'm thinking about raising the issue again at Pycon. Having something like raw_input() is, I believe, nearly essential for using interactive tutorials of the kinds produced by crunchy in a teaching environment aimed at beginners.

Further on the topic of interactive tutorials, I have been thinking of an easy means to identify them. For those that have tried crunchy, you will have noticed that an interactive tutorial viewed through a regular browser without using crunchy appears to be nothing else than your average (W3C compliant) html document. Since such tutorials could be included on any web site, I was thinking it might be useful to give a visual clue to crunchy users. There already exist a Python logo to identify applications, namely:
I've been thinking of asking the Python software foundation for the permission to modify this logo, replacing the word "powered" by "interactive", and adding a small change to crunchy so that every tutorial created with it would automatically have the logo added at the top. I'm debating whether I should do this prior to Pycon, or wait until I get the chance to lobby for it in person.

Finally, if enough people start writing interactive tutorials using crunchy, I thought it might be useful to have a central repository somewhere. Moving away from the Monty Python references, I thought that the Snake PIT (for Python Interactive Tutorials) might be a catchy name.

So, for you, patient reader that has made it this far, does it sound like a worthwhile for a Lightning talk, especially if it is supplemented by a one-minute demo of an interactive tutorial? I'm pretty sure I could cover all three topics above in less than the scheduled 5 minutes for a lightning talk, as I would not have to go into much detail about the history; just the facts...

Crunchy 0.8 is out

2007-01-27T18:16:00.000-04:00

Version 0.8 of Crunchy has been released. It is available at its new home on code.google.com.

Crunchy, the Interactive Python Tutorial Maker, is an application that
transforms an ordinary html-based Python tutorial into an interactive
session within a web browser. Currently, only Firefox is supported.
Crunchy is developed and tested on Windows XP and Ubuntu Dapper Drake,
but should work on any suitable windows or UNIX system.

Three major improvements have been made since version 0.7 had been released.

1. New editor

Instead of a simple html textarea, Crunchy now gives the option of
using a "real" editor, namely EditArea. EditArea support syntax coloring
and allows loading and saving local Python files among other features.
Within Crunchy, it is set up so that the tab key is translated into 4
spaces.

2. Language support

Crunchy now supports English and French, through the use of ".po"
files. When running Python code, some error messages have been
adapted/translated. EditArea itself support more languages
(currently: Danish, Dutch, English, French, German, Italian, Japanese,
Polish, Portuguese).

3. Graphical tutorial converter.

Crunchy uses some supplementary markup to transform html files into
interactive tutorials. Whereas previous versions required a tutorial
maker to edit an html file "by hand", version 0.8 includes a tutorial
editor: with a few clicks, you can easily add to an html file the
chosen interactive elements and options for Crunchy.

In addition to the above major improvements, the code has been
refactored significantly and a number of small bug fixes have been
made. Crunchy will be demonstrated at the upcoming Pycon 2007.

The next release will likely have a new, simplified API for tutorial writers, but with more powerful features, thanks to the work of Johannes Woolard. Unfortunately, it is unlikely to be ready in time for Pycon. Anyone planning to go to Pycon, and who is interested in Crunchy should feel free to contact me with any questions/suggestions they may want to have me address during my presentation.

Pycon 2007: it's Crunch time - take 2

2007-01-10T17:35:00.000-04:00

Well, it's done: I've registered for Pycon, booked a hotel room and bought plane tickets. Since November 1, when I first heard that my talk had been accepted, crunchy has changed quite a bit and my planned presentation will have to change. One demo I had planned to do, using codetch, and which I thought would take about 5 minutes will now be doable in less than 30 seconds without having to use anything else than the "new and improved" crunchy. Unfortunately, there has been so much changed since the last release that the documentation has to be re-organized significantly - this means that I can't really do a new release right now as it would likely be too confusing. So, I'm racing against the clock to put everything together to do a new release before Pycon. However, if some of you are interested in having a preview, drop me an email and I'll let you know where you can get the code from and give a quick description of how to use the new features.
Looking at the official statistics, I find the relatively small numbers of attendees compared with last year somewhat disappointing. Given that more talks are going to be presented this year, with a much larger rejection rate, I find this rather puzzling. I imagine the organizers are rather disappointed.

Unicode headaches ... and a solution

2007-01-03T23:37:00.000-04:00

Work on Crunchy has restarted by both Johannes and I over the holidays, after a few months long hiatus. Hopefully, we'll be in a position to do a new release soon (just a few more features...). Among the changes, Crunchy now has a proper embedded editor, EditArea. See this example to get an idea of what EditArea can do. Actually, the Python support has since been improved by Christophe Dolivet, the creator of EditArea, prompted by a few suggestions of my own, but the new version has not been made publicly available yet. It is however already used in the development version of Crunchy, and, with a few additions of my own, will definitively be showcased during my Pycon presentation.

Other Crunchy changes include a proper handling of English and French translations. Now, since EditArea's tooltips are available in many more language, I thought I should use a more complete encoding (like utf-8) rather than the one I normally use (latin-1). After all, if rur-ple has been found useful enough to be adapted to 6 languages (with Italian and Chinese in the pipeline), I figured that Crunchy is likely to eventually see the same kind of adaptation.

However, I soon encountered a most puzzling bug. All strings translated were properly rendered by Firefox except for those coming out of an interpretor session (some tracebacks have been customized and translated in French as well as being available in English). When French was selected as the default language, whenever an accented letter, like à, was supposed to appear as a result of a Python output, a ? instead was displayed.

After trying all kind of encoding/decoding tricks over the course of a few hours, I remembered that I had set the default sytem encoding on my computer to latin-1. I decided to change it to utf-8 and, sure enough, everything was working as expected. Success at last!

However, this was only the beginning of my problems. My favourite editor, SPE, stopped working. I also tried to run rur-ple which failed miserably. [Idle, on the other hand, which I use very rarely, was still working perfectly.] Clearly, changing the site default encoding was not an appropriate solution: I could certainly not depend on having a crunchy user set his or her site customization to utf-8. A different approach was needed.

After reverting back to latin-1 as my default python site encoding (so that paths that included my name, André, were properly read by SPE) and poring over the code, I finally figured out a more general solution.

Whenever Crunchy executes some Python code written by a user (or provided by a tutorial writer), it starts a Python interpreter in a separate thread. This Python interpreter uses the default system encoding for all its computation. When the result needs to be sent back by the Crunchy server to Firefox, it needs to have its encoding changed as


result = result.decode(sys.getdefaultencoding()).encode('utf-8')

Of course, in retrospect, it all makes sense but it did stump me for quite a few hours; perhaps the information included here will save a few minutes or hours to someone else.

Rur-ple 1.0rc

2006-11-30T19:23:00.000-04:00

Rur-ple 1.0 is finally out ... sort of. Actually, it's a Release Candidate version. For the next little while, any free time I'll have is going to be spent on preparing my Pycon presentation (and working on Crunchy as needed to improve it before Pycon 2007). Since I had three more localizations available (German, Turkish and Welsh) than there was on sourceforge, I decided that it was better to release early. In the meantime, someone is working on an Italian localization (and possible lessons translation) which should be included in the final 1.0 version.

Pycon 2007: it's Crunch time

2006-11-30T18:13:00.000-04:00

Yesterday I got some good news from the Pycon organizers: my proposed talk entitled Easy creation of interactive tutorials has been accepted. In this talk, I'll demonstrate how Crunchy can be used to create interactive Python tutorials. I've been thinking about how to organize my talk for about two months now but have not written a line yet. One thing I can say for sure at this time: this is not going to be your typical PowerPoint-type presentation. My current plan is to use Firefox for the presentation: one tab will display a fake html-based Python tutorial, the second tab will make use of codetch so that I can edit the same tutorial "live", and the third tab will display the resulting interactive tutorial as processed by Crunchy.

Rur-ple: double the number of languages

2006-11-13T18:15:00.000-04:00

While I haven't had time to do programming in the past few months, some Rur-ple users have been busy and submitted new material. The upcoming 1.0 release of Rur-ple will include 3 new languages [Turkish (including a translation of most lessons), German and Welsh] as well as previously included languages [English (with all lessons), French and Spanish].

Inspired by the user contributions, I spent a few hours today re-writing the localization code. To add a new language now only requires the following:

Adding one line in a Python file (translation.py), that specify the language and language code, as well as the name of the .po file.
Creating a .po file containing the string translations.
Creating an html file (rur.htm) used as a Welcome Page in that language.

The new version will also save the last language selected and will use it upon re-starting Rur-ple.

Securing Crunchy

2006-08-21T22:37:00.000-03:00

Following my last post, some alert readers pointed out that Crunchy could be a security risk for its users. Crunchy acts as a web server, fetching html-based Python tutorials and displaying them in your browser with controls allowing to execute the Python code. This is meant as a good thing... but it might not be if the code is allowed to run automatically when the page is loaded. In fact, this might have occurred with previous (<0.7) Johannes and I changed the way that Crunchy works so as to remove any security worries - at least, we hope so. Anyone interested can obviously look at the code; for others, here's what we do:

Each time Crunchy starts, it generates a random session number (10 digit long).
The javascript code used to communicate between Crunchy and the browser is fetched from a static file and modified to include the unique session number as part of each command; a new javascript file, with a filename that incorporates the session number is generated.
When Crunchy fetches an html page, it processes it to remove unwanted code. This means removing all (java-)scripts and only allowing html tags that are on a "whitelist". This whitelist excludes any of the usual "onload", "oninit", etc., automatic execution.
Next, Crunchy interactive elements (Python interpreter, code editor, button for Python code execution, etc.) are inserted with the javascript calls, each call given a unique identifier for that session.
Finally, the processed page is displayed in the browser, with the server waiting for user-generated interactions.
This procedure is repeated for any page that is loaded during that session.
At the end of a session, when Crunchy is terminated, the javascript file that was generated and used in that session is deleted.

The latest version (0.7) includes this security features and many new additions [a complete list of the changes from version 0.6 is appended at the end of this message]. Currently, Johannes is working on adapting How to think like a computer scientist to use with Crunchy, as well adding new interactive features. For my part, I have been working on internationalisation issues (adding a French interface) and other configuration choices. Due to end-of-summer holidays, work on Crunchy will likely slow down for a few weeks at least but we should be able to reach version 1.0 early in the Fall.

After that, I will be ready to start implementing rur-ple inside the Crunchy environment; if there is enough interest, I might implement a turtle graphics module as well.

===== Changes from 0.6 to 0.7 ================
0. Name change
To prevent confusion with an existing program named CrunchyFrog, "Crunchy Frog" has been renamed as "Crunchy", short for "Crunchy, the Interactive Python Tutorial Maker".

1. *Security fix*
The previous versions of Crunchy allowed tutorials containing arbitrary (and hidden) javascript code to be loaded within a browser window. The new version removes any existing javascript code prior to processing for display within a browser.

Feel free to try and break this - and please report any findings back to us.

2. New functionality.

It is possible to run external programs from within the browser; sample demos include GUI programs with 1. Tkinter, 2. pyGtk, 3. wxPython, 4. Pygame
Automatic syntax colouring of (static) Python code, including (as an option) line numbering.
New plotting canvas with simple to use api to draw mathematical functions
Drawing & plotting canvas can be set to arbitrary size.
Multiple canvas can now appear on same page.
New addition and simplification to the sound api.
Simplification of error message (tracebacks) more suitable for beginners.

New plotting canvas with simple to use api to draw mathematical functions
Drawing & plotting canvas can be set to arbitrary size.
Multiple canvas can now appear on same page.
New addition and simplification to the sound api.
Simplification of error message (tracebacks) more suitable for beginners.

3. New visual design:

Integrated menu which can be styled through custom css
Three sample css styles now included (selectable via the browser menu)

4. New content:

First draft sound tutorial exploring concepts of frequency, beats, harmonics (Fourier series), etc.
New addition to the basic "Crunchy user" tutorial
Additions to all reference documentation (sound & graphics api, vlam options, etc.)

Exciting Crunchy News

2006-08-04T01:23:00.000-03:00

While I knew all along it could be done, I've had a lot of fun during the last two days playing with the newest, and possibly most exciting feature of "The application formerly known as Crunchy Frog", or simply "Crunchy". It is now possible, from within a Firefox windows, to click a button and launch an external Python script. (I should mention that the bulk of this latest feature was implemented by Johannes Woolard - I just had to make a few minor changes to get it working on Windows.)

I'll provide more details later, but here's the short story. You write an html document in which you insert the Python script you want to execute inside a <pre> element with some minor markup (<pre title="external copycode">); actually, you can have as many scripts you want on any given page. Crunchy processes the page, displays the Python scripts in colour (!) and makes a copy of each available for further editing inside a <textarea>. You can then just click on a button and the script is launched.

This means that Crunchy can really transform *any* Python tutorial into an interactive experience. Crunchy can load html pages from the web (not only local ones) and perform its magic ;-)

So far, I've launched Tkinter windows, wxPython apps, Pygame programs, a terminal with a Python interpreter and, the latest, a gnuplot window that ran the gnuplot demo. You can actually have a number of independent external apps going at the same time, allowing for some interesting experiments (edit the code in the <textarea> and launch the new version).

Stay tuned for more news soon, and the announcement of the next release.

Crunchy news

2006-07-22T18:52:00.000-03:00

It's now been a month since my last blog post and a lot has happened in the meantime. We (Johannes and I) have just released a new version of Crunchy Frog. We were planning to do a release (version 0.5) a bit earlier but we kept "fixing this", "adding that little bit", "changing this", etc., until I felt it was well beyond what had been planned for the next release. So, after discussing briefly about it (at Johannes's request), we set up a roadmap for Crunchy Frog as a whole (as oppposed to just for Johannes' project) with a date set for the 1.0 release that would coincide with the end of this Summer of Code project for Johannes.

The new public release (0.6) looks very different from the previous public one (0.4). Crunchy now has its own site. Previously, Johannes had set up an svn repository which has come extremely handy. This summer of code project is supposed to be a learning experience from the student (Johannes) under the direction of a more experienced mentor (that would be me). However, I can say that it has been a huge learning experience for me. So far, I had only worked on my own project, and only very recently had learned to use an svn repository on my own computer. Working as part of a team (supposedly as the team leader [I have the final say, if need be]; in reality, it's pretty much a team of equals, each of us having different strengths) has been an experience in itself.

While we have settled on a roadmap, I can already see some additional features, not mentioned so far, that I would like to implement before the end of the summer. For those interesed in Crunchy Frog: stay tuned, there could be some very nice new stuff added soon.

The 0.6 release got some attention outside of "regular channels" and I just found out that there exists another Python project named CrunchyFrog (no space between the two words). This, I think, is most unfortunate and I am in a bit of a quandry as to what to do about the name. Note that the developper behind CrunchyFrog, who brought its existence to my attention, not only has not complained about me choosing this name, but has actually set up a link to Crunchy Frog on his own website. [This is something we'll definitely have to reciprocate; Johannes is responsible for the website and is taking a well deserved break after the last minute rush to get everything organised for the 0.6 release.] Nonetheless, the possibility of changing the name of the project has to be considered ... even though Johannes has designed a nice graphical interface based on a Frog theme.

With all this attention devoted to Crunchy Frog, Rur-ple's development has been suspended. However, just a few days ago, Michel Weinachter sent me

a "patch" so that the user-chosen language can be remembered from one session to the next;
an exe version, produced by py2exe, so that Windows users don't need to install Python nor wxPython to use Rur-ple;
some simple code to make use of a clipboard within rur-ple; I have not had time to see how to make this work though;
just today, a link to a Firefox extension useful to users of Crunchy Frog. This extension enables the "normal" use of a tab key within an html <textarea>, which comes in handy when typing Python code.

Merci Michel!

So, all in all, a busy month with lots of long nights of coding after work. I have a great job, which does not and never will require me to do any programming [sometimes, much to my chagrin.] However, as a hobby, programming in Python has really been a fantastic one. It is my hope that the programs I create will be found useful by others.

Almost there: version 0.9.9 of RUR-PLE has been released

2006-06-22T23:48:00.000-03:00

The title of this post pretty much says it all. I just need to write a few more lessons, read over once more all the existing lessons, do once more all the suggested exercises ... and, barring any surprises, I will be ready to release version 1.0 of RUR-PLE. Thus will end an adventure that started a little less than 2 years ago, as I decided to start a new hobby and learned about Python.

Chasing an unseen bug

2006-06-18T16:33:00.000-03:00

A few days ago I was contacted by M.H., a professor in the U.S. who teaches Introduction to Computer Programming every year and was thinking of using RUR-PLE. [Great, I thought, more user feedback to be expected.] Unfortunately, M.H. had tried the latest version and found that the code highlighting feature [described in this post] did not work.

How could I possibly find a bug that I can not reproduce? I thought... Thus began an exchange of emails that led me to find a way to reproduce the bug and, eventually, to solve it. The final solution was somewhat typical of my experience with Python: try the obvious!

These last three words are probably the best words of advice I could give to anyone who writes computer programs using Python. If you are reading this blog as a potential source of useful tips [you fool!], you should probably stop now as nothing else I will write is likely to come even close to being as useful to you.

However, if you want to get a good laugh at my expense, you are invited to continue reading.

I first started by asking M.H. what everyone always asks me when I claim to find a bug or when I don't understand the behaviour of some module and post some question on various lists.

What OS and what version of wxPython are you using?

I expected that M.H. was using Mac OS or Linux, in which case I was getting ready to diplomatically tell him that he was likely out of luck as I'm [shame on me], a Windows user. However, M.H. told me that he was using Win XP (same here), Python 2.4.3 (I'm using 2.4.2) and wxPython 2.6 (same here).

Ok. I was more stumped then ever. This was bugging me... These days, I'm working on Crunchy Frog which will, eventually, include and supersede RUR-PLE. As it is, I have a lot more ideas than time at my disposal to work on Crunchy Frog, and this "distraction" was not welcome. However, I could not ignore it.

Since I couldn't reproduce the bug, I asked M.H. if he could do me a favour and insert a print statement in the method where the line number was set for the purpose of highlighting. As I wrote the email, I had a "clever" thought. M.H. is in the U.S. He's probably using the ansi version of wxPython whereas I'm using the unicode version. I bet you something's wrong with that version... Sure enough, my first hunch was right: M.H. was using the ansi version. And the print statement was outputting the same result on his computer as it was on mine. So, if one hunch was right... I asked M.H. to install the wxPython unicode version, which he kindly agreed to.

Same result... Of course, you are no doubt thinking. However, you are not trying to quash a bug you can't even reproduce. [That's my excuse, anyway.]

At that time I was visiting my partner who, unfortunately, lives 300 km (200 mi) from where I live. Fortunately, she is extremely patient and puts up with me when I am distracted by a programming problem. I decided to install wxPython and the latest version of RUR-PLE on her computer to investigate some more.

Ooops... wxPython is now at version 2.6.3.2. I am sure the version installed on my computer is 2.6.1.0. I read over the "changes" document to see if I can find anything that I can use. There does not appear to be anything relevant. I install the new version ... sure enough, I can "see" the bug (or, should I say, I can't see anything as no highlighting is taking place).

I read the "changes" document a second (and third) time, just to be sure. No luck.

Time to dig in my code. [... Where is this object initialised?... This could have been written so much better... What was I thinking then (about a year ago)... ] After staring at the code for a while, being convinced that the highlighting information is properly set (it was working before, after all) I am convinced it is a wxPython bug that was introduced. I file a bug report and expect to leave it at that.

However, my conscience is nagging me. I decide to look at the code again, staring at the method where the highlighting updates should take place. Here it is, with a few added notes.

def update_refresh(self, robot, name):
  if 'robot' in self.robot_dict:  # (1)
      arg = self.parent.status_bar.beeper_field, self.robot_dict['robot']._beeper_bag
      event_manager.SendCustomEvent(self.parent, arg)
  time.sleep(robot.delay)  # (2)
  wx.Yield()   # (3)
  self.world.DoDrawing()  #(4)
  self.WorldDisplay.drawImage()  #(5)
  if name:
      self.WorldDisplay.scrollWorld(name)  # (6)
  self.WorldDisplay.Refresh()  # (7)
  wx.Yield()

The information in the status bar (number of beepers carried by robot, etc.) is updated.
The animation is paused at each step.
Making sure other controls of the application are responsive so that the program can be paused or stopped altogether by pressing the appropriate button, etc.
Update the (in-memory) image based on the latest robot action.
Update the screen image.
Prevent the robot from going off the screen.
Refresh the screen [Refresh() is a wxPython method.]

That's it. Not much there. I refresh the screen so as to ensure that the robot world, right next to the code "window", gets updated on the screen when a new line of code is executed, and should be highlighted in the code window.

Can you see the bug?

I didn't refresh the code window. It worked before though.

How do I do this... the robot world is a ScrolledWindow whereas the code window is a StyledTextControl... Where do I find the relevant documentation...

That's when I thought of the three important words I mentioned before: try the obvious. I typed in


  self.ProgramEditor.Refresh()

and launched RUR-PLE. Problem solved. I guess it was just dumb luck that this refresh statement was not required before. All I have to do now is fix the bug and update the version on sourceforge ... before I go back and rewrite the parts of the code that I didn't find easy to read just a few days ago.

Praise for Python

2006-06-05T19:42:00.000-03:00

Sometimes, I come accross something which I really wish I could have written. Usually, I just smile, try to commit it to memory, and move on. This time, I have to quote what Jim Roskind wrote on Guido's blog:

I was initially a bit put off by the Python indentation approach (I was a grammar guy… and got used to parsing braces and parens, and didn’t like the idea of white space having so much meaning), but I soon came to love it. Eventually I came to apply one of my favorite computer science sayings to Python as an endorsement to their indentation blocking approach: “The fundamental evil in computer science is the replication of code or data.” In almost all languages I had previously worked, the indentation of code had always been a critical PART of program nesting structure (at least for the reader). As I read and wrote more Python, its use of indentation as the ONLY way to specify blocks began to look better and better. All the silly bugs related to indentation errors (misleading the human reader) were gone in Python. The redundant use in other languages of braces AS WELL AS indentation (the former to help the parser, and the latter to help the human reader) was an effective duplication of the author’s intent. That duplication in other languages, a fundamental original sin, was missing in Python. It was cool! ;-)

Yes, indeed. Python is cool. :-)

RUR-PLE video

2006-06-04T18:28:00.000-03:00

The kind folks from ShowMeDo had contacted me and suggested I make a demonstration video for RUR-PLE. I'm happy to announce that this has been done, and that the video now appears on the ShowMeDo website. The link to this demonstration video can be found on the right hand side side-bar for this blog. The sound quality is not all that good, but the video should give a good idea of what RUR-PLE is all about. (I will definitely need to get a better microphone for future videos, and possibly for remaking this one.)

rur-ple bug fix for Mac and Linux users

2006-05-25T11:25:00.000-03:00

Brad Miller uncovered a bug (hard coded windows specific path separators) in version 0.9.8.2 of rur-ple. Version 0.9.8.3 has been uploaded with required changes. Apologies to Linux and Mac users.