Pycon and Crunchy

This year's Pycon program looks very interesting. I wish I could be there but, alas, the timing was just wrong for me this year. This is doubly disappointing as I would have been able to meet with Johannes Woolard in the flesh. Yes, forget Guido van Rossum, Alex Martelli and other famous names: the one person I wanted to meet is Johannes. For more than a year an a half, I have had the pleasure of collaborating with Johannes on Crunchy, without ever meeting him. This year, Johannes will be the one showing Crunchy off. I'm sure he'll do a great job.

And, if anyone is looking to hire a bright, young, hard-working programmer, Johannes will graduate from Oxford this year.

99 problems: looking for volunteers

Some time ago, Dr. Werner Hett created a list of 99 Prolog problems that could be used to skills in logic programming. More recently, a Ruby learner posted a Ruby version of the first 10 problems, and his solutions. This seemed to be a good idea, especially if one makes use of doctests ... and Crunchy :-). So, I've started my own version of these which you can get as a zip file (containing 6 problems and their solutions) from the Crunchy main page. If you have never done so before, to load a local html file within Crunchy, you simply click on the "Browsing" menu on the left hand side and scroll down until you reach the "Closer to home" section and follow the instructions.

Note that with the next version of Crunchy (the current one is 0.9.8.6) you will be able to start Crunchy with an arbitrary file using something like

python crunchy.py --url=full_local_path_or_url

It would be nice if there could be a complete Python version of the 99 Prolog problems. If anyone is interested in helping, please do not hesitate to contact me.

Automated documentation code testing - part 2

Thanks to Crunchy's simple plugin architecture, after only a few hours of coding the automated documentation testing (for html files) described here has been implemented (the "first approach" described, that is.) It will be part of the next Crunchy release. In theory, code samples for a complete book could be all tested at the click of a button, provided that the book is available as an html document. The next step will be to define a few new directives so that reStructuredText documents can be used as well.

Now, while I have a few sample test files, it would be nice is to find someone who has a real life document with embedded Python code samples as a test user...

Automated documentation code testing

During the first year or so of development work on Crunchy, I probably got a nickname of "Dr. NO!" by early Crunchy adopters as I often resisted suggestions for adding new capabilities. At the time, Crunchy required that html pages have additional markup added (vlam = very little additional markup) so that Crunchy could process them properly. I wanted Crunchy-enabled tutorials to be very easily created, without much additional work from tutorial writers, so that Crunchy would be adopted by many people. Most of the suggestions that were made, including some by Johannes, both while he was sponsored by Google as a Summer of Code student and afterwards when he became a co-developer, were rejected by me for that reason. Since then, the situation has changed, mainly for two reasons:

1. Johannes created a new basic infrastructure for Crunchy where we can introduce new capabilities via plugins, without needing to change a single line of the core in most instances.
2. Based on the new architecture, I came up with a new way to process pages so that no additional markup was needed for Crunchy to do its magic. This is what makes it possible, for example, to interact with the official Python tutorial on the python.org site.
   

Now, that it is so easy to implement new capabilities, I am revisiting some ideas I had rejected or ignored before. The struggle I have is to decide when enough is enough before finally having a version 1.0 officially released.

In any event, after reading some comments on this post by Georg Brandl, I started thinking about adding a new option to test code embedded in documentation. To quote from the comments on that post:

One thing that is seriously needed is the ability to run and test code snippets in some fashion. It's just too easy for documentation to get out of date relative to the code, and if you can effectively "unit test" your docs, you're in much better shape.

And I don't mean like doctests, because not everything lends it self well to that style of testing. If it's possible to mark up some code as being for test fixture and some code as being what belongs in the doc, that would be good.

Alternatively, from another reader:

For me a key is being able to test code in the docs, and think the key is being able to "annotate" a code snipit with information about the context in which it should run, and the output it should give.

I think that Crunchy is a very good platform to implement this. There are currently three complementary options I am considering, one of which I have started to implement.


The first option is to have something like the following [note that while I use html notation, Crunchy is now capable of handling reStructuredText, including having the possibility of dealing with additional directives]:

Some normally hidden code, used for setup:
<pre title="setup_code name=first">
a=42
</pre>

Followed by the code sample to be tested:
<pre title="check_code name=first">
print a
</pre>

And the expected output:
<pre title="code_output name=first">
42
</pre>

Upon importing a document containing such examples, Crunchy would insert a button for each code sample allowing the user to test the code by clicking on the button, invoking the appropriate setup, and comparing with the expected output. Alternatively, all such code samples in a document could be run by a single click on a button inserted at the top of a page. A javascript alert could be used to inform the user that all tests passed - otherwise error messages could be inserted in the page indicating which tests failed or passed.

This type of approach could, in theory, be used for other languages than Python; code could be executed by passing information to a separate process launched in a terminal window, with the result fed back into Crunchy as described above.

A second approach is to use the same method used by doctest to combine code sample and expected output; the setup code could still be used as described above.

A third approach, this one completely different, could be used for more general situation than simply for documentation code testing.

Currently, the Python code needs to be embedded inside an html (or rst) document. However, one could create links to code that lives inside separate Python files. For example, one could have the following:

<pre title="python_file">
<span title="python_file_name"> file_path </span>
<span title="python_file_linenumbers"> some_range </span>
</pre>

When viewing the above using a normal browser, one would see something like (using a fictitious example)

../crunchy_dir/crunchy.py
[1-3, 5, 7, 10-15]

However, when viewing the same page with Crunchy, the appropriate lines would be extracted from the file and displayed in the browser. Alternatively, instead of specifying the line numbers, one could have a directive to extract a specific function/method/class as in

<span title="python_file_function"> function_name </span>

which would instruct Crunchy to extract all the code for the function definition, and inserting it in the document. By using such links, the code in the documentation would always (by definition) be kept in sync with the real code. I realize that this is not exactly a novel idea but one whose potential could be extended by using Crunchy in ways never seen before. However, this last approach will have to wait until after Crunchy version 1.0 has been released.

What do you think of these ideas?

Kudos to ActiveState

ActiveState, the company behind the Python cookbook and many other useful and free resources, is a great supporter of free software. Its free Komodo Edit is a nice piece of software, well worth trying.

I started using Komodo Edit a few months after I switched from using Windows to using a Mac. On Windows, my Python editor of choice was SPE. However, I found that SPE, not being a native Mac application, had some small quirks that I found annoying. After trying Textmate, praised by many Mac users, and Wing among others, I settled on the free Komodo Edit. While I missed the class browser I had gotten used to with SPE, I found that Komodo was enough for my basic needs.

After reading this post by Jesse Noller, I started using pylint within Komodo and while I did not find any bugs (so far!) in my code, it did encourage me to improve the existing code. The possibility of easily adding new tools to Komodo Edit lead me to try its more powerful sibling, Komodo IDE. Komodo IDE has an integrated debugger (something I had *never* used before for code development but that I will likely use more and more in the future) and a code browser side bar which is even better than the one included with SPE. After using it for about a week, I decided to treat myself and purchase a license for it before the trial license ended. However, since I found the price a bit steep for something to use just for fun, I inquired about available discounts. I was told that, even though I did not use it for my work, I was eligible for an educational discount given that I work at a University.

However, there was more to come...

When I indicated that I intended to buy it online, I got an email telling me that I was actually eligible for a deeper discount since I had a license for an earlier version of Komodo Personal edition. This was a total surprise for me. Here's what happened: more than two and a half year ago, ActiveState had a special promotion for open source developers to get a free license for Komodo Personal edition. I had taken advantage of this offer at the time and installed Komodo on my Windows computer. However, I found it was comparable in functionality to SPE for which I had a slight preference. As a result, I gave up on Komodo after trying it for about a week.

Now, more than 2.5 years later, the friendly people at ActiveState reminded me that I had a valid (but free!) license and told me I could simply pay for an upgrade to what is in my opinion a much superior programming environment than the version I had a license for.

Talk about friendly customer service! Thank you ActiveState!

GHOP Python Related Success Story

Early on during Google's HOP contest, I joined the Python mentors group following a call for task suggestions, and submitted many Crunchy-related tasks. I was amazed by the quality of some contributions. As more potential mentors involved with other Python related projects joined in, I decided to quietly refrain from submitting more Crunchy related tasks. Still, Crunchy got more than its share of contributions from students from all over the world.

Today, one student posted a blog entry about a Crunchy presentation he made to his class. He describes it as a success - and I would agree. However, it is clear to me that the success of his presentation is due by far more to Python's strength than to Crunchy itself. I thought it was a very good example to use when advocating for the use of Python - and therefore, worth linking to.

More power and removing an option

One of the neat features of Crunchy, suggested to me by Andrew Dalke at Pycon 2007, is its ability to dynamically display images that are generated by some Python code. The example given in the Crunchy distribution uses matplotlib. However, the way this is done is slightly cumbersome. First, some pre-existing vlam (very little added markup) must already have been added to an html page, specifying the name of the image (e.g. foo.png) to be displayed. This special vlam will result in an editor appearing on the page together with two buttons (one to execute the code, the other to load the image). Second, the Python code must be such that an image will be generated with that name. By default, the image is saved in and retrieved from a special Crunchy temp directory.

While the approach used works, it does also mean that images can't be generated and displayed using a simple Python interpreter, nor can they be displayed from an arbitrary location. At least that was the case until now.

Prompted by a suggestion from Johannes, I wrote a very simple module whose core consists of only 7 Python statements, and which does away entirely with the cumbersome vlam image file option. Images can now be loaded and displayed from anywhere using two lines of code:

import image_display
image_display.show(path, [width=400, height=400])

And by anywhere, I mean from any interactive Python element (interpreter, editor),
and using any image source (local images and images on the web), like

image_display.show('http://imgs.xkcd.com/comics/python.png')

Furthermore, one can use this approach to create a "slide show" by alternating image_display.show() and time.sleep().

Since there is no more need to use the old image_file option, it will be removed in the next Crunchy release. I may have to give some more thoughts to the API for this new option (e.g. is there a better name than image_display? Should I add a function for a slide show, giving a list of images and a time delay? etc.); suggestions are welcome.

100 posts, olpc, rst and Crunchy

This is the 100th post on this blog which I started to write a little over 3 years ago - shortly after I started my programming hobby. And, as it so happens, I received two gifts in the past 24 hours:

• I received my give-one-get-one olpc in the mail today.
• A student wrote a simple programs to add Crunchy-specific "directives" for docutils so that rst can be used to create Crunchy ready tutorials. More on this below.

I had seen the olpc at Pycon 2007 but, not having tried it back then, I did not realise how small the keyboard really is. I have fairly small hands but they are much too big to use the keyboard comfortably. I thought of using it to write this blog entry but quickly gave up on that idea.

It will take a while to fully explore the olpc. I am extremely impressed by the quality of the screen. However, it is a slow computer which apparently can only handle a "small" number of applications running concurrently - I managed to freeze it with about 8 applications running. I seemed to remember from the Pycon presentation that the track pad had 2 (or 3, depending on how you count them) active region, but only the central one seems to be active. Members of my family were surprise at the "plastic toy" appearance of the olpc but they all seem to be fairly impressed once they saw it running.

Now, all I have to do is to figure out how to make Crunchy run on it. :-)

Speaking of Crunchy, as part of Google's HOP contest, I had set up two tasks related to reStructured Text. One of them was to write a plugin so that a rst file could be loaded, transformed into a html file (by docutils) and displayed by Crunchy. This was done a while ago as I reported here. However, when this is done, Crunchy treats all the code elements on a page (inside "pre" tags) the same way, which is specified via the variable crunchy.no_markup. This does not allow for the same fine-tuning that is possible with the addition of vlam (very little added markup) to a normal html page.

The second rst task was to write "directives" so that vlam could be inserted inside rst pages. I had read quickly about rst directives and figured this would be fairly complicated. From my superficial reading, I *thought* it would require a modification to docutils that would have to be incorporated into the core (or make a special version of docutils that incorporated those directives). Of course this would make little sense. In any event, a student wrote a 200 lines program that defined most required "rst directives" for Crunchy, allowing to take a rst file as input and output an html file. All I had to do was to cut-and-paste the student's program into the existing rst plugin, change 2 lines of code, and I now have a fully working vlam-compatible rst loader for Crunchy.

This will be part of the next Crunchy release.

Small tip for porting unit tests to Python 3.0

While working on increasing test coverage of Crunchy, I encountered a puzzling change when running doctest based unit tests under Python 3.0a and Python 2.5. One test (of a function that was definitely too long) was failing under Python 3.0a1/2 with a single line containing the number 20 appearing in the output - in addition to all the other expected ones. To make a long story short, the problem was linked to PEP 3116. Using Python 3.0a1/2, when a file is written using ".write()", there is a return value corresponding to the number of bytes written; the "old" behavior was to return None. In order to have doctest-based tests running successfully under both 2.x and 3.x, we need to replace things like

f.write()

by

dummy = f.write()

I understand that the plan is to backport many of the Python 3.0 features to Python 2.6 so as to ease the transition, and use an automated tool for the conversion. I don't think that features from PEP 3116 will be implemented in Python 2.6, since they are very different from 2.5. And I am doubtful that the conversion tool will take care of including a dummy assignment for functions that will now return a value different than None. I am hoping that this post might end up saving a little bit of time for any reader that tries to migrate their code from 2.x to 3.0.

On a different topic altogether, I have fixed the bug which resulted in removing the styles from remote web pages when using Python 3.0 in conjunction with Crunchy. This will make Crunchy a more attractive tool to use in going through the official Python tutorial for Python 3.0. For the moment, the Crunchy embedded interpreter still only works with Python 3.0a1 and not 3.0a2. Hopefully I'll have this resolved in new public release by Pycon 2008 so that Johannes can demonstrate it in his talk.

More encoding pains...

It seems like every early January brings some new encoding pains...

Eons ago (less than a year) I used to program on a Windows based computer. My Windows user name was simply André, which is not surprising since it's my first name. However, the observant reader may have noticed that my name requires one non-ASCII character. Such a small detail... that can cause so much annoyance.

Two years ago, on January 6th, I wrote about using site customization so that my favourite Python editor at the time (SPE) could deal with my user name.

Last year, on January 3rd, I wrote about how Crunchy dealt with encoding issues in a way that was independent of any site customizations. At the time, an astute reader made the following comment (which I had forgotten until today, when I decided to write this blog entry):

Without having looked at the rest of your code, so I might be completely off here, this somehow looks wrong:

result = result.decode(sys.getdefaultencoding()).encode('utf-8')

The reason I say this is that you're decoding and encoding in the same place. Since Python unicode support is so good, it's generally a good idea to decode to unicode any use input you get as early as possible, and to encode only as late as possible when outputting strings. Since you're doing complicated web ui stuff here, so it may be that you're not doing anything with 'result' between input and returning it to the browser, but if you are, the string should have already been decoded by the time it gets to this line. Otherwise this will bite you anytime you try to do anything with the string like simple concatenation. [emphasis added]

Of course, since Crunchy was working properly, I quickly dismissed that comment. With the way everything was implemented, Crunchy was working just fine... In fact, to this day, if you download the latest public release (0.9.8.6), everything works just fine - even if your user name includes non-ASCII characters.

However ... in adapting Crunchy to works with Python 3.0, I do things with the various strings like simple concatenation ... and, of course, this cause some problems as I found out when I "borrowed" the old Windows computer from my daughter to try the latest changes I had made.

It. Did. Not. Work.

Ok, so I have two possible solutions:

1. Trade with my daughter for a while, letting her use my MacBook (which she loves!) while I use the "old" Windows desktop.
   
2. Create an account with an accented name on my MacBook.

Well, as much as I love my daughter, I could not face the pain of going back to using the Windows desktop as my main computer. So, solution 2 was an easy choice.

Except that it wasn't....

My account name on the mac is my full name (André Roberge). Hmmm... this has already a non-ASCII character. But my home directory is "andre" - no accent. Under Mac OS, a user has a full name that appears in the login window, and a short name used for the root directory (/Users/andre in my case). When I tried to create a new account with a non-ASCII character in the short name, it just beeped and refused to enter it.

In search for an answer, I posted on three Mac related forums, got either no answer or some unhelpful and wrong answer on two of them ... I was considering posting on the Python list, but, fortunately, I eventually got a useful answer.

So, if you are thinking of writing i18n applications that either can run from any directory or that save information in the user's default directory, or both, and you want to make sure that it will work on a Windows computer, here's how you can do it on a Mac (under OS X 10.5):

1. Create a test account user the account manager under any name of your choosing; however, the "short name" will have to be ASCII only (at this stage).
2. From the account manager, ctrl-click on the account after it has been created; this will bring an advanced dialog.
3. Edit the advanced dialog to change the short name, and the home directory, to the desired value. I chose the name "accentué" (self-referencing name, if you know French). Note that doing so does not change the actual name of the directory.
   
4. Go to a terminal window and do "sudo mv old_name new_name" to change the name of the home directory that was created at step 1.

After I did all this, the development version of Crunchy did not work from the new account. This pleased me very much: it likely will meant that I do not have to trade computers with my daughter to continue working on Crunchy. ;-)

More about Crunchy running under Python3.0a1

Just in time for the new year, and after a major reorganization of Crunchy's code to make it easier to write more unit tests for it, I did manage to get all of Crunchy's core functionality working under Python 3.0a1. Crunchy can now import html files that are not conforming to strict xml convention (i.e. not having closing tags for some elements) and thus create problems for ElementTree. The way it does that is to launch an external process under Python 2.5 (or whatever default version that is invoked via typing "python" at a terminal - but it has to be a 2.x version for it to work). This process imports the file, "cleans" it (using a combination of BeautifulSoup, ElementTree and Crunchy's security module), and saves it locally. After a delay, Crunchy loads up the cleaned up file and display it. In this process, most of the styling is unfortunately removed. :-(

However, the good news is that I was able to load up the official 3.0 Python tutorial (work in progress) and try it out using Crunchy. I did find out one limitation of using Crunchy to do so. Crunchy encodes the Python output using utf-8 before forwarding to the browser. So, instead of having things like b'Andr\xc3\xa9' appearing on the screen, it would be converted to André. Thus, Crunchy is not a very good platform to teach about encoding/decoding of strings. For other aspects though, it is an ideal tool (if I may say so) for going through the Python tutorial: there is no need to switch back and forth between the browser and a separate Python environment to try things out. I still hope to have the time and energy to go through the entire 3.0 tutorial (something I have never done before, for 2.x) and see if I can find any bugs or come up with useful suggestions.

All I have left to do for the next release is to write up some documentation/tutorial on the new Turtle module and on launching Crunchy under Python3.0a1. With a bit of luck, this will all be finished before the end of the year.

In addition to the code reorganization mentioned above, I did fix a few bugs and made an improvement on Crunchy's Borg interpreters. For those that aren't familiar with it, Crunchy allows to embed a number of interpreters (html input box communicating with the Python backend) within a single page. These interpreters can either be isolated one from another (meaning that a variable defined in one interpreter is only known by that interpreter) or can share a common environment (aka Borg interpreters). Normally, in a single user mode, using a single open tab in Firefox, every time a new page is displayed, the Borg interpreters are effectively reset (the old ones are garbage collected and the new ones are created from an empty slate).

Previously, if one were to have multiple users (or multiple tabs open from the same user) on the same Crunchy server, all Borg interpreters ended up sharing the exact same state. This is not very convenient if two users are trying to use the same variable names! I had planned to address this "feature" at some point after the 1.0 release but was forced into it earlier due to the 3.0a1 work. The reason is the following.

To create "Borg interpreters", I was using the Borg idiom invented by Alex Martelli. It goes as follows:

class Borg(object):
    '''Borg Idiom, from the Python Cookbook, 2nd Edition, p:273

    Derive a class form this; all instances of that class will share the
    same state, provided that they don't override __new__; otherwise,
    remember to use Borg.__new__ within the overriden class.
    '''
    _shared_state = {}
    def __new__(cls, *a, **k):
        obj = object.__new__(cls, *a, **k)
        obj.__dict__ = cls._shared_state
        return obj

When using this idiom in a standard program under 3.0a1/2, a deprecation warning is raised about object.__new__() not taking any argument. When I was trying to make use of this idiom in Crunchy running under 3.0a1/2, the deprecation warning was actually replaced by an exception. Rather than trying to silence this exception, I decided to take a different approach and used instead the following:

class BorgGroups(object):
    '''Inspired by the Borg Idiom, from the Python Cookbook, 2nd Edition, p:273
    to deal with multiple Borg groups (one per crunchy page)
    while being compatible with Python 3.0a1/2.
    Derived class must use a super() call to work with this properly.
    '''
    _shared_states = {}
    def __init__(self, group="Borg"):
        if group not in self._shared_states:
            self._shared_states[group] = {}
        self.__dict__ = self._shared_states[group]

# The following BorgConsole class is defined such that all instances
# of an interpreter on a same html page share the same environment.

class BorgConsole(BorgGroups, SingleConsole):
    '''Every BorgConsole share a common state'''
    def __init__(self, locals={}, filename="Crunchy console", group="Borg"):
        super(BorgConsole, self).__init__(group=group)
        SingleConsole.__init__(self, locals, filename=filename)

The "group" variable is taken to be a unique id generated by Crunchy each time it processes a given html page. Thus, each page loaded by a different user (or the same user, at a different time) from the same Crunchy server will result in a unique set of Borg interpreters.

To be fair, I must admit that I did not come up with this solution totally on my own. A while ago, I asked for something like this on comp.lang.python. (Those interested in the details should search for "Borg rebellion".) I just derived the above solution from some suggestions made at that time.

Finally, in addition to all this, I found out a bug in code.py for Python 3.0a1/2. I tried to send an email to the python-3000 mailing list about it, but it was held up, waiting for a moderator approval for a few days. So, I canceled it and filed a bug report instead (which I should have done in the first place) on the bug tracker. I still haven't seen any follow up - perhaps due to the title I gave it. The bug is actually very easy to fix - three lines of code need to be replaced by a single one. The solution is related to my only other "official" contribution to Python to date. Hopefully, by this time next year, I'll have learned enough to contribute more to Python.

Crunchy and Python 3.0a2

Continuing with my experiment of adapting Crunchy to Python 3.0, I managed to get Crunchy to start with Python 3.0a2 and get some code running from the editor - but not from the interpreter, nor the doctest. Most of the problems I have are dealing with bytes-to-string conversion and string-to-bytes. As mentioned by Guido van Rossum last June

• We're switching to a model known from Java: (immutable) text strings are Unicode, and binary data is represented by a separate mutable "bytes" data type. In addition, the parser will be more Unicode-friendly: the default source encoding will be UTF-8, and non-ASCII letters can be used in identifiers

Later on, in a comment from that post, we find:

• > In your presentation last night you had one slide which
  > talked about the "str" vs "bytes" types in Python 3000. On
  > the bottom of that slide was something like:
  >
  > str(b"asdf") == "b'asdf'"
  >
  > However, in discussing this slide (very briefly) you said
  > that a type constructors like "str" could be used to do
  > conversion. It seems like "str" is behaving more like
  > "repr" in this case, which seems unusual and less useful
  > to me. Was this a typo, or is this actually the way it's
  > supposed to work? What's the rationale?
  
  To be honest, this is an open issue. The slide was wrong compared to the current implementation; but the implementation currently defaults to utf8 (so str(b'a') == 'a'), which is not right either. The problem is that there are conflicting requirements: str() of any object should ideally always return something, but we don't want str() to assume a specific default encoding.
  
  To be continued...

This change seems innocuous enough...

As a web server, Crunchy sends to and receives information from the browser as "binary data" or "bytes". As a generalized Python interpreter, Crunchy manipulates the information as "strings". It appears that the "bytes" implementation is done much more completely in Python 3.0a2 than it was in Python 3.0a1. And this is the source of many problems.

For example, Crunchy sends from the browser some information about the path to which a Python file should be saved and its content as follows:

'/Users/andre/.crunchy/temp.py_::EOF::_from Tkinter import *\nroot = Tk()\nw = Label(root, text="Crunchy!")\nw.pack()\nroot.mainloop()'

This is sent as a binary stream which needs to be converted to the string written above. This conversion is done via str(...). Using Python 3.0a1 (and 2.4 and 2.5), the result was as above; splitting the string gave the following:

['/Users/andre/.crunchy/temp.py', 'from Tkinter import *\nroot = Tk()\nw = Label(root, text="Crunchy!")\nw.pack()\nroot.mainloop()']

Now, with Python 3.0a2, it gets slightly more complicated. The first string acquires a "b" prefix upon conversion (as mentioned in the comment from Guido's blog mentioned before). After splitting, the result is

["b'/Users/andre/.crunchy/temp.py", 'from Tkinter import *\\nroot = Tk()\\nw = Label(root, text="Crunchy!")\\nw.pack()\\nroot.mainloop()\'']

So, we now have a first string with a "b'" prefix embedded in it, and a second one without. It seems that each case will have to be handled carefully on its own. And I suspect more problems will show up as we get closer to the final 3.0 release.

I know, I know, I'm really not following the "recommended" practice, as quoted on Guido's blog. I should probably wait first for Python 2.6 to come out. Then, I should have a complete unit test coverage and use the conversion tool to create a Python 3.0 version .... However, I am not convinced that the conversion tool will be smart enough to know when a function (that I write) expect a "str" object and when it expect a "byte" one. Furthermore, the few unit tests I had worked fine under both Python 2.5 and 3.0 ... but some functions that I had written with the expectation that they would receive some string arguments did not work in "production code", as they were getting some bytes arguments. And this failed completely silently...

If I had to give some advice to someone about creating Python programs that can work with both Python 2.x and Python 3.x, I would say like Guido: don't. :-) Unless of course you are like me and are doing this for fun and to get to learn about the differences between Python 2.x and 3.x along the way. But then, "be prepared for the unexpected" like the following: turning on a few print statements (via a "debug flag") can result in breaking some code; turn them off and the code works again... Yes, it did happen to me - I still have to figure out how...

Crunchy and Python 3.0a1

It is often said that a picture is worth a thousand words...

I have managed to make Crunchy run under Python 3.0a1. Some of the features are not working but the interpreter and the editor work. The turtle module I have been working on also works "nicely" (read: as slow as before) with this new Python version. Unfortunately, when it is run under Python 3.0a1, Crunchy can not load most pages - including those of the official Python 3.0 tutorial. The reason is that is uses ElementTree to parse pages and it is unforgiving when it comes to having unclosed tags (as in <link> and <meta...> for example); it also seems to not be able to handle the <script>s that are included on the page. I have not yet found a way to reliably "clean" the pages before parsing them with ElementTree. While I believe that I should be able to do so with a bit more work, there is a bigger problem...

Unfortunately, Crunchy does not run under Python 3.oa2, and the error messages I get have not been too helpful in figuring out the error. However, perhaps this is due to a faulty installation. What makes me think so is that when I start a 3.0a2 session at a terminal, I get an error message when I use exit(). This is most unexpected.

In any even, the next release should include the new crunchy turtle module and be usable with 3.0a1. Perhaps Johannes, or some curious user, will be able to figure out how to make it run under 3.0a2 as well.

Slow turtle ... in time for Xmas

One of the task assigned in Google's HOP contest was to design a simple turtle graphics module for Crunchy.  This was done successfully by a student as a prototype.  This prototype had some unfortunate limitations in terms of number of turtles and simultaneous graphics canvas existing on the same page, but it did give me the impetus to use the student code as a proof-of-concept and implement a more complete turtle module for Crunchy.

Playing with turtles, and trying to draw fairly complex shapes, made me realize that the combination of using an html canvas and the Crunchy comet communication makes for an extremely slow turtle. It would be really nice to  find a better (faster) way.

The next Crunchy release should include that turtle module ... and an additional bonus: Crunchy can now be launched successfully using either Python 2.5 (or 2.4) and Python 3.0a1.  And the turtle module works with both.

At the moment, not all of Crunchy's features are supported when using Python 3.0.  However, this should no longer be the case by the time version 1.0 comes out.

And, for those that might be tempted to point out Guido's blog entry about not making programs compatible with both Python 2.x and 3.x, please don't bother.  I realize that it is not wise in general to try to do so.  However, given Crunchy's design philosophy to make it as easy for students/teachers/tutorial writers to use - it just does make sense: download, unzip, double-click; nothing else should be needed to start having fun with Python - no matter what new Python version gets installed.

(NOT) Bitten by PEP 3113

UPDATE: The comments left on this post (1 and 3) in particular corrected my misreading of PEP 3113. There is no such wart as I describe in Python 3.0. I should have known better than to question GvR and friends. :-) I'm leaving this post as a reference.

In trying to make Crunchy useful & interesting for beginning programmers to learn Python, I designed a small graphics library following some "natural" notation. As an aside, Johannes Woolard is the one who made sure that this library could be easily used interactively within Crunchy. I mention his name since too many people seem to assume that I am the only one involved in Crunchy's design. Anyway, back to the library...

In that library, the function used to draw a line between two points uses the syntax

line((x1, y1), (x2, y2))

for example: line((100, 100), (200, 200))

which should be familiar to everyone. Unfortunately, following the implementation of PEP 3113 in Python 3.0, this syntax is no longer allowed. This is ... annoying! There are two alternatives I can use:

line(x1, y1, x2, y2)

for example: line(100, 100, 200, 200)

or

line(point_1, point_2)

where point_a = (x_a, y_a). Update: with this second definition, it will be possible to invoke the function as
line((100, 100), (200, 200))

Of course, either of these two option is easy to implement (and is going to be backward compatible with Python 2k). However, I don't find either one of them particularly clear for beginners (who might be familiar with the normal mathematical notation) and do not consider this a (small) wart of Python 3k.

reStructuredText files and Crunchy

Crunchy can now handle reStructuredText (.rst) files in the same way it can process plain html ones! This requires the user to have docutils installed - which is normally the case for anyone that writes .rst files.

The test coverage for Crunchy is slowly improving. Currently, 10 modules are mostly covered by doctest-based unit tests, out of approximately 40. Since I make use of .rst files to keep the unit tests, these can now be browsed "pleasantly" using Crunchy itself.

Furthermore ... all the unit tests written so far work under Python 2.4, Python 2.5, and ... Python 3.0a1! This required some tedious rewriting of some parts of the code but the end result is well worth it - if only to really learn about differences between Python 2.5 and Python 3.0.

One thing that I found, which will be no surprise to TDD aficionados, is that code written without testing in mind can be quite tricky to write comprehensive tests for. Add to this the extra complication of making that code run under two incompatible Python versions, and you are on your way to major headaches. It's a good thing I am doing this only for fun!

Launching Python 3.0 program from Crunchy running under Python 2.5

As part of Google's Highly Open Participation contest, Michele Mazzoni completed the task of creating a new option for Crunchy: one can now launch (starting with the next release of Crunchy - 0.9.8.5) a program using a different version of Python than the one used by Crunchy itself. While I had suggested that the alternate Python version could be set via the configuration options for Crunchy (usually accessible from a Python interpreter), Michele had the brilliant idea to add a simple input box where one can specify the path (or 'alias') of the Python version used right on the page where the program is launched from. This makes it extremely easy to change the interpreter version used to launch a user written program.

Michele has prepared a screencast demonstrating this, which should appear on ShowMeDo hopefully soon.

Thank you Michele - and thank you Google!

More results from GHOP

Google's Highly Open Participation (GHOP) contest is attracting a lot of attention from the right people: pre-university students. The PSF is one of ten organizations mentoring students working on Python-related projects. Since I submitted tasks suggestions early on and volunteered to help following a call for volunteers from Titus Brown, Crunchy has benefited from many students contributions. Crunchy's messages have been translated in Estonian, Macedonian, Polish and Italian with, hopefully, more translations to come. Some new unit tests have been added with more to come. There may be a couple of nice surprises coming out soon too :-)

While other projects have also benefited from GHOP's students contributions, there could be more. If you have some good ideas for mini-projects (doable in 3-5 days, at a couple of hours per day with perhaps one full day), your suggestions would most likely be most welcome. Just check out the GHOP Python discussion group. And, if you would like to join the (too small) ranks of Python mentors, please do; we need all the help we can get.

First Crunchy screencast

I finally got around to produce a first Crunchy screencast. Since Crunchy is an interactive program, I thought I should do screencasts that reproduced the interactive feel: that is to say, I would record a live, more or less improvised session. This means that the screen cast is not as polished as it could be - an even has a few minor mistakes in it. Still, I think it gives a good (superficial) overview of what Crunchy is capable of.

What do you think?

First result from GHOP: Crunchy Estonian

By now everyone has probably heard of Google Highly Open Participation contest, and the participation of the PSF. And, initially through serendipity but later with more active involvement, a few Crunchy-related tasks were created including some involving translations. And, just a day after the contest started, a first translation (in Estonian!) has been made for Crunchy by a student named Tanel.

The tasks assigned are usually not very big but they provide nice stepping stones for getting people involved in open source software. Thank you Google for this wonderful idea.