From vlam to nam: Using Crunchy to interact with the official Python tutorial

Breaking news: you can now use Crunchy to browse and interact with the official Python tutorial. More below...

Since my last post that described briefly some features of the new Crunchy engine, I have not had much time to work on Crunchy. Johannes too has been fairly busy so that Crunchy development pretty much stopped, until the beginning of Google Summer of Code. Two students joined the Crunchy team: Edin Salković and Bryan Psimas. Unfortunately, due to other commitments, Edin was not able to continue with his SoC project - but he did manage to write the prototype of a new plugin for Crunchy that I will describe below.

The new Crunchy engine now works extremely well. As I wrote in my previous post, Crunchy is now more responsive; it supports simultaneous execution of multiple code sample, proper input (stdin) handling from the webpage browser and the api (for writing tutorials) has been significantly simplified as there is no longer be any need to have an embedded "canvas" call in a tutorial: the user is now able to load a graphics dynamically anywhere on a page. These graphics can even include simple animations using pure Python code (for the end user - javascript behind the scene for us, unfortunately).

There is more. For example:

• it is now possible to style <code> elements;
• Python code sample (either inside <pre> or <code>) that contain pre-existing html markup can now be handled by Crunchy without generating an error (blank page!) as before;
• it is now possible to specify a starting number different from 1 when requesting that line numbers be added to a code sample;
• when it styles the code, Crunchy now automatically detects if a given code sample represents an interpreter session or a "normal" code sample;
• it is now possible to use input() and raw_input() with an interpreter, and the result appears inside the page displayed by the browser.

In addition, thanks to Edin's work, a new type of interaction with Python code has been added: it is possible to save the result of a computation in a graphics file and display the result in the browser. This has been used successfully with matplotlib. There is a "small" bug in that the Python code needs to be executed twice due to synchronization issues with the browser loading the file and Python saving it.

Quite a few features from the "old Crunchy" need to be implemented (e.g. menus including a way to properly quit the application, translations, update:the ability to load an arbitrary tutorials given a URL, etc.) in the new version but, overall, it is working very well.

One of the "problems" with the old Crunchy is that it did require two things from a tutorial writer:

1. That the html code written be W3C compliant (with no warning, no unclosed tags, etc.).
2. That Crunchy specific markup (vlam = very little additional markup) be added to instruct Crunchy to add interactive elements.

Unfortunately, the official Python tutorial fails on both accounts. However, with the use of a version of BeautifulSoup adapted by effbot (thanks!) to be used with ElementTree, the new Crunchy can now deal with non W3C compliant code.

And ... Crunchy has now been written so that it automatically inserts a Python interpreter whenever it sees a bare <pre> (i.e. with nam = no additional markup). As a result, one can now browse the official Python tutorial with Crunchy and interact with it.

This automatic insertion of an interpreter sometimes yields too many interpreter inserted on a page than really needed; it is better to hand craft a tutorial. However, it should make Crunchy a lot more useful to many more people.

For those interested in trying out the "new Crunchy", an alpha release is available. If you have never used Crunchy before, you should try version 0.8.2 first and go through the tutorial from the menu.

New Crunchy engine

At the end of my talk at Pycon 2007, I gave an extremely brief demo of the new (not in production yet) upcoming version of Crunchy. Thanks to the work of Johannes Woolard, Crunchy's javascript core engine has been completely rewritten. As a result, Crunchy will be more responsive; it will also support simultaneous execution of multiple code sample, proper input (stdin) handling from the webpage browser and the api (for writing tutorials) will be greatly simplified as there will no longer be any need to have an embedded "canvas" call in a tutorial: the user will be able to load a graphics dynamically anywhere on a page.

But that is not all...

Inspired by a side remark by Ivan Krstić (of OLPC fame) about finding the best way to design plugins in Python, I contacted Johannes to discuss the idea of using plugins to extend Crunchy. After a few back and forth emails, while I was on the road on business trips, Johannes just went ahead and implemented a first (then a second...) way of using plugins to extend Crunchy. In the end, it looks like we'll have to rewrite almost completely (like we did during the Summer of Code 2006) Crunchy's code as we move from one minor version (0.8) to the next (0.9). However, the result will be well worth it.

The unstated goal: to make it (almost) as easy for developpers to add new capabilities to Crunchy as it is for tutorial writer to use Crunchy to create interactive tutorials. Ok, this might be a slight exaggeration ... but not much of one ;-) And most of this will be the result of Johannes' great work. So, for all of those employers (and there were many at Pycon) looking for a brilliant Python programmer to hire over the summer, I know of an Oxford student who certainly fits the bill.

Pycon experience

Pycon 2007 is now officially over. It was very nice to be able to associate faces to so many familiar names from the Python community and get the opportunity to talk with a few of them.

My talk didn't go as smoothly as I had hoped. When I practiced it, it seemed to always take too long. So, when I had to do it in front of a fairly large crowd, I rushed through it a bit too quickly. Still, Crunchy did seem to generate a fair bit of interest and, in addition to the demo in my talk, I got the opportunity to demonstrate it "one-on-one" to about a dozen people who all seemed to be very keen.

Also, the idea of creating a special Python logo ("web badge") for identifying Crunchy-ready tutorials, which I had alluded to in a previous post, will likely be given the ok by the PSF. As soon as it is confirmed, I'll make an announcement here.

Since my last post, the PEP I wrote for keeping raw_input()'s functionality in Python 3000 [but renaming it "input()"] had been accepted. On Monday, I took part briefly in a Python 3000 sprint where I got to implement the required "raw_input(...) -> input(...)" and "input(...) -> eval(input(...))" conversion tool for the general tool that will be available to convert Python 2.x programs so that they work with Python 3k. I also implemented the corresponding simple unit tests as well a few missing tests for the "xrange() -> range" conversion. I would not have been able to do this if Guido van Rossum had not taken the time to explain a few things to me about the conversion framework. In the end, Guido checked in my changes himself. I get the feeling that he could have done the entire thing in about the same time it took him to explain to me the tricky bits. Nonetheless, I do get the satisfaction of having, with the PEP and the conversion tool, given back a bit to the Python community. Hopefully, with Crunchy, I (and Johannes, of course) will give back even more.

Switched to Mac OS

I'm a statistics... I just got a Macbook and will likely use it as my main development platform soon. So far, moving from Windows XP to Mac OS has been fairly painless. I have not done any Python programming but did try out Crunchy and Rur-ple. I found out that Crunchy requires Python 2.4+ (I had 2.3.5 installed on the Mac), that it does not work with Safari, and that there is a "bug" so that the "tooltip" does not work with Firefox on the Mac. I was planning to bring my Macbook to Pycon but will have to reevaluate as I'd like the Drunchy demo to go as smoothly as possible. There also does not appear to be a sound module on the Mac similar to winsound or ossaudiodev... I also found a new bug with rur-ple, but have no time to look into it before Pycon.

Not having taken side in the emacs/vim religious war, I have to find a decent programming environment. On my PC, I used SPE which I really liked but I have read a lot of good things about textmate and nothing negative (other than the fact that it is not free). Any advice from Mac/Python users?

Rur-ple segmentation fault on Ubuntu: solved?

Some kind readers have left comments on this blog regarding segmentation faults occurring when trying to run Rur-ple using Ubuntu linux. I finally got around to installing a vmware version of ubuntu so that I could check for myself and believe I found the source of the bug. I would appreciate if someone actually using Ubuntu could verify.

Near the bottom of rur_start.py, a splash screen is started with 100 (milliseconds) as a default. Changing that number to 1000 removed the segmentation fault when running rur-ple in a virtual ubuntu environment.

There is at least one other bug in rur-ple when using a version of wxPython greater than 2.6; it is in lightning.py. Changing "event.KeyCode()" to "event.GetKeyCode()" apparently solves the problem.

Of course, I would be interested in hearing about any other bugs that have been encountered so that I could do a final cleanup before finally releasing the 1.0 version.

Lightning talk?

Living in a small village in Nova Scotia, 300 km from the nearest city (Halifax, pop.: 380,000), the opportunities to discuss face to face with other pythonistas are non-existent. This is one reason I've been really looking forward to going to my first Pycon. In addition to my scheduled talk, I have been thinking about giving a lightning talk of a different kind, but still touching upon a subject dear to me, that is the promotion of Python and its use as a teaching language. While lightning talks are so short that they normally touch upon only one topic, I was thinking of actually covering three topics, under the common theme of "interactive tutorials". These three topics are, from the more serious to the semi-frivolous:

1. The fate of input() and raw_input() in Python.
2. Using a distinctive logo for promoting interactive tutorials using Python.
3. Creating an interactive tutorial repository.

According to this, both input() and raw_input() are supposed to disappear in Python 3.0. After thinking about it for quite some time, I posted a message on edu-sig about the proposed change last September, expressing my "opposition" to the proposed change, and seeking the opinion of others on this topic. After a while, a near-consensus emerged, supporting the idea of keeping at least something like raw_input() in Python 3.0. I then wrote a pre-PEP, which generated more positive comments and encouragements but otherwise no further action from other edu-sig subscribers more involved in the Python community than I am. Unfortunately, due to my work, I had to drop all my Python-related activities for a few months and was not able to follow up until late December. With the end of the year approaching, I found the time to incorporate the comments received on edu-sig and offered a revised pre-PEP to the Python 3000 list on December 22nd. This was a single posting from me as I don't subscribe to that list. A bit of discussion ensued after which Guido stated that he like the proposal better than any of the other alternatives mentioned and, later, suggested that someone should "clean up and check in the proto-PEP and start working on an implementation or patch."

Unfortunately, and perhaps because this happened just before the holidays, nothing further was done. So, I'm thinking about raising the issue again at Pycon. Having something like raw_input() is, I believe, nearly essential for using interactive tutorials of the kinds produced by crunchy in a teaching environment aimed at beginners.

Further on the topic of interactive tutorials, I have been thinking of an easy means to identify them. For those that have tried crunchy, you will have noticed that an interactive tutorial viewed through a regular browser without using crunchy appears to be nothing else than your average (W3C compliant) html document. Since such tutorials could be included on any web site, I was thinking it might be useful to give a visual clue to crunchy users. There already exist a Python logo to identify applications, namely:
I've been thinking of asking the Python software foundation for the permission to modify this logo, replacing the word "powered" by "interactive", and adding a small change to crunchy so that every tutorial created with it would automatically have the logo added at the top. I'm debating whether I should do this prior to Pycon, or wait until I get the chance to lobby for it in person.

Finally, if enough people start writing interactive tutorials using crunchy, I thought it might be useful to have a central repository somewhere. Moving away from the Monty Python references, I thought that the Snake PIT (for Python Interactive Tutorials) might be a catchy name.

So, for you, patient reader that has made it this far, does it sound like a worthwhile for a Lightning talk, especially if it is supplemented by a one-minute demo of an interactive tutorial? I'm pretty sure I could cover all three topics above in less than the scheduled 5 minutes for a lightning talk, as I would not have to go into much detail about the history; just the facts...

Crunchy 0.8 is out

Version 0.8 of Crunchy has been released. It is available at its new home on code.google.com.

Crunchy, the Interactive Python Tutorial Maker, is an application that
transforms an ordinary html-based Python tutorial into an interactive
session within a web browser. Currently, only Firefox is supported.
Crunchy is developed and tested on Windows XP and Ubuntu Dapper Drake,
but should work on any suitable windows or UNIX system.

Three major improvements have been made since version 0.7 had been released.

1. New editor

Instead of a simple html textarea, Crunchy now gives the option of
using a "real" editor, namely EditArea. EditArea support syntax coloring
and allows loading and saving local Python files among other features.
Within Crunchy, it is set up so that the tab key is translated into 4
spaces.

2. Language support

Crunchy now supports English and French, through the use of ".po"
files. When running Python code, some error messages have been
adapted/translated. EditArea itself support more languages
(currently: Danish, Dutch, English, French, German, Italian, Japanese,
Polish, Portuguese).

3. Graphical tutorial converter.

Crunchy uses some supplementary markup to transform html files into
interactive tutorials. Whereas previous versions required a tutorial
maker to edit an html file "by hand", version 0.8 includes a tutorial
editor: with a few clicks, you can easily add to an html file the
chosen interactive elements and options for Crunchy.

In addition to the above major improvements, the code has been
refactored significantly and a number of small bug fixes have been
made. Crunchy will be demonstrated at the upcoming Pycon 2007.

The next release will likely have a new, simplified API for tutorial writers, but with more powerful features, thanks to the work of Johannes Woolard. Unfortunately, it is unlikely to be ready in time for Pycon. Anyone planning to go to Pycon, and who is interested in Crunchy should feel free to contact me with any questions/suggestions they may want to have me address during my presentation.