Crunchy 0.8 is out

Version 0.8 of Crunchy has been released. It is available at its new home on code.google.com.

Crunchy, the Interactive Python Tutorial Maker, is an application that
transforms an ordinary html-based Python tutorial into an interactive
session within a web browser. Currently, only Firefox is supported.
Crunchy is developed and tested on Windows XP and Ubuntu Dapper Drake,
but should work on any suitable windows or UNIX system.

Three major improvements have been made since version 0.7 had been released.

1. New editor

Instead of a simple html textarea, Crunchy now gives the option of
using a "real" editor, namely EditArea. EditArea support syntax coloring
and allows loading and saving local Python files among other features.
Within Crunchy, it is set up so that the tab key is translated into 4
spaces.

2. Language support

Crunchy now supports English and French, through the use of ".po"
files. When running Python code, some error messages have been
adapted/translated. EditArea itself support more languages
(currently: Danish, Dutch, English, French, German, Italian, Japanese,
Polish, Portuguese).

3. Graphical tutorial converter.

Crunchy uses some supplementary markup to transform html files into
interactive tutorials. Whereas previous versions required a tutorial
maker to edit an html file "by hand", version 0.8 includes a tutorial
editor: with a few clicks, you can easily add to an html file the
chosen interactive elements and options for Crunchy.

In addition to the above major improvements, the code has been
refactored significantly and a number of small bug fixes have been
made. Crunchy will be demonstrated at the upcoming Pycon 2007.

The next release will likely have a new, simplified API for tutorial writers, but with more powerful features, thanks to the work of Johannes Woolard. Unfortunately, it is unlikely to be ready in time for Pycon. Anyone planning to go to Pycon, and who is interested in Crunchy should feel free to contact me with any questions/suggestions they may want to have me address during my presentation.

Pycon 2007: it's Crunch time - take 2

Well, it's done: I've registered for Pycon, booked a hotel room and bought plane tickets. Since November 1, when I first heard that my talk had been accepted, crunchy has changed quite a bit and my planned presentation will have to change. One demo I had planned to do, using codetch, and which I thought would take about 5 minutes will now be doable in less than 30 seconds without having to use anything else than the "new and improved" crunchy. Unfortunately, there has been so much changed since the last release that the documentation has to be re-organized significantly - this means that I can't really do a new release right now as it would likely be too confusing. So, I'm racing against the clock to put everything together to do a new release before Pycon. However, if some of you are interested in having a preview, drop me an email and I'll let you know where you can get the code from and give a quick description of how to use the new features.
Looking at the official statistics, I find the relatively small numbers of attendees compared with last year somewhat disappointing. Given that more talks are going to be presented this year, with a much larger rejection rate, I find this rather puzzling. I imagine the organizers are rather disappointed.

Unicode headaches ... and a solution

Work on Crunchy has restarted by both Johannes and I over the holidays, after a few months long hiatus. Hopefully, we'll be in a position to do a new release soon (just a few more features...). Among the changes, Crunchy now has a proper embedded editor, EditArea. See this example to get an idea of what EditArea can do. Actually, the Python support has since been improved by Christophe Dolivet, the creator of EditArea, prompted by a few suggestions of my own, but the new version has not been made publicly available yet. It is however already used in the development version of Crunchy, and, with a few additions of my own, will definitively be showcased during my Pycon presentation.

Other Crunchy changes include a proper handling of English and French translations. Now, since EditArea's tooltips are available in many more language, I thought I should use a more complete encoding (like utf-8) rather than the one I normally use (latin-1). After all, if rur-ple has been found useful enough to be adapted to 6 languages (with Italian and Chinese in the pipeline), I figured that Crunchy is likely to eventually see the same kind of adaptation.

However, I soon encountered a most puzzling bug. All strings translated were properly rendered by Firefox except for those coming out of an interpretor session (some tracebacks have been customized and translated in French as well as being available in English). When French was selected as the default language, whenever an accented letter, like à, was supposed to appear as a result of a Python output, a ? instead was displayed.

After trying all kind of encoding/decoding tricks over the course of a few hours, I remembered that I had set the default sytem encoding on my computer to latin-1. I decided to change it to utf-8 and, sure enough, everything was working as expected. Success at last!

However, this was only the beginning of my problems. My favourite editor, SPE, stopped working. I also tried to run rur-ple which failed miserably. [Idle, on the other hand, which I use very rarely, was still working perfectly.] Clearly, changing the site default encoding was not an appropriate solution: I could certainly not depend on having a crunchy user set his or her site customization to utf-8. A different approach was needed.

After reverting back to latin-1 as my default python site encoding (so that paths that included my name, André, were properly read by SPE) and poring over the code, I finally figured out a more general solution.

Whenever Crunchy executes some Python code written by a user (or provided by a tutorial writer), it starts a Python interpreter in a separate thread. This Python interpreter uses the default system encoding for all its computation. When the result needs to be sent back by the Crunchy server to Firefox, it needs to have its encoding changed as

result = result.decode(sys.getdefaultencoding()).encode('utf-8')

Of course, in retrospect, it all makes sense but it did stump me for quite a few hours; perhaps the information included here will save a few minutes or hours to someone else.

Rur-ple 1.0rc

Rur-ple 1.0 is finally out ... sort of. Actually, it's a Release Candidate version. For the next little while, any free time I'll have is going to be spent on preparing my Pycon presentation (and working on Crunchy as needed to improve it before Pycon 2007). Since I had three more localizations available (German, Turkish and Welsh) than there was on sourceforge, I decided that it was better to release early. In the meantime, someone is working on an Italian localization (and possible lessons translation) which should be included in the final 1.0 version.

Pycon 2007: it's Crunch time

Yesterday I got some good news from the Pycon organizers: my proposed talk entitled Easy creation of interactive tutorials has been accepted. In this talk, I'll demonstrate how Crunchy can be used to create interactive Python tutorials. I've been thinking about how to organize my talk for about two months now but have not written a line yet. One thing I can say for sure at this time: this is not going to be your typical PowerPoint-type presentation. My current plan is to use Firefox for the presentation: one tab will display a fake html-based Python tutorial, the second tab will make use of codetch so that I can edit the same tutorial "live", and the third tab will display the resulting interactive tutorial as processed by Crunchy.

Rur-ple: double the number of languages

While I haven't had time to do programming in the past few months, some Rur-ple users have been busy and submitted new material. The upcoming 1.0 release of Rur-ple will include 3 new languages [Turkish (including a translation of most lessons), German and Welsh] as well as previously included languages [English (with all lessons), French and Spanish].

Inspired by the user contributions, I spent a few hours today re-writing the localization code. To add a new language now only requires the following:

1. Adding one line in a Python file (translation.py), that specify the language and language code, as well as the name of the .po file.
   
2. Creating a .po file containing the string translations.
3. Creating an html file (rur.htm) used as a Welcome Page in that language.

The new version will also save the last language selected and will use it upon re-starting Rur-ple.

Securing Crunchy

Following my last post, some alert readers pointed out that Crunchy could be a security risk for its users. Crunchy acts as a web server, fetching html-based Python tutorials and displaying them in your browser with controls allowing to execute the Python code. This is meant as a good thing... but it might not be if the code is allowed to run automatically when the page is loaded. In fact, this might have occurred with previous (<0.7) Johannes and I changed the way that Crunchy works so as to remove any security worries - at least, we hope so. Anyone interested can obviously look at the code; for others, here's what we do:

• Each time Crunchy starts, it generates a random session number (10 digit long).
• The javascript code used to communicate between Crunchy and the browser is fetched from a static file and modified to include the unique session number as part of each command; a new javascript file, with a filename that incorporates the session number is generated.
  
• When Crunchy fetches an html page, it processes it to remove unwanted code. This means removing all (java-)scripts and only allowing html tags that are on a "whitelist". This whitelist excludes any of the usual "onload", "oninit", etc., automatic execution.
  
• Next, Crunchy interactive elements (Python interpreter, code editor, button for Python code execution, etc.) are inserted with the javascript calls, each call given a unique identifier for that session.
• Finally, the processed page is displayed in the browser, with the server waiting for user-generated interactions.
• This procedure is repeated for any page that is loaded during that session.
  
• At the end of a session, when Crunchy is terminated, the javascript file that was generated and used in that session is deleted.

The latest version (0.7) includes this security features and many new additions [a complete list of the changes from version 0.6 is appended at the end of this message]. Currently, Johannes is working on adapting How to think like a computer scientist to use with Crunchy, as well adding new interactive features. For my part, I have been working on internationalisation issues (adding a French interface) and other configuration choices. Due to end-of-summer holidays, work on Crunchy will likely slow down for a few weeks at least but we should be able to reach version 1.0 early in the Fall.

After that, I will be ready to start implementing rur-ple inside the Crunchy environment; if there is enough interest, I might implement a turtle graphics module as well.

===== Changes from 0.6 to 0.7 ================
0. Name change
To prevent confusion with an existing program named CrunchyFrog, "Crunchy Frog" has been renamed as "Crunchy", short for "Crunchy, the Interactive Python Tutorial Maker".

1. *Security fix*
The previous versions of Crunchy allowed tutorials containing arbitrary (and hidden) javascript code to be loaded within a browser window. The new version removes any existing javascript code prior to processing for display within a browser.

Feel free to try and break this - and please report any findings back to us.

2. New functionality.

• It is possible to run external programs from within the browser; sample demos include GUI programs with 1. Tkinter, 2. pyGtk, 3. wxPython, 4. Pygame
• Automatic syntax colouring of (static) Python code, including (as an option) line numbering.
• New plotting canvas with simple to use api to draw mathematical functions
• Drawing & plotting canvas can be set to arbitrary size.
• Multiple canvas can now appear on same page.
• New addition and simplification to the sound api.
• Simplification of error message (tracebacks) more suitable for beginners.

• New plotting canvas with simple to use api to draw mathematical functions
• Drawing & plotting canvas can be set to arbitrary size.
• Multiple canvas can now appear on same page.
• New addition and simplification to the sound api.
• Simplification of error message (tracebacks) more suitable for beginners.

3. New visual design:

• Integrated menu which can be styled through custom css
• Three sample css styles now included (selectable via the browser menu)

4. New content:

• First draft sound tutorial exploring concepts of frequency, beats, harmonics (Fourier series), etc.
• New addition to the basic "Crunchy user" tutorial
• Additions to all reference documentation (sound & graphics api, vlam options, etc.)