Thwarted by lack of speed

I was hoping to make an announcement of a new cool app based on Google's App Engine but unfortunately I have been thwarted by Python's relative lack of speed.

I have started working on a new version of Crunchy that would run as a web app on Google's servers. While the current version of Crunchy fetches existing html pages, processes them and displays them in the browser, this new version would retrieve html page content (in reStructuredText format) from Google's datastore, transform it into html, process it to add interactive elements, and then displays them.

This new app was going to be usable as a wiki to create new material. This was my starting point, greatly helped by an already existing wiki example that I adapted to use reStructuredText. When requesting a page, the following was supposed to happen:

1. reStructuredText content (for the body of the html page) is fetched from the datastore.
2. said content is transformed (by docutils) into html
3. html content is further processed by modified "crunchy engine" to add interactive elements.
4. modified html content is inserted in page template and made available.

The user would then be able to enter some Python code which could be send back to the App Engine using Ajax for processing and updating the page display.

A normal user would only be able to interact with already existing pages. Special users ("editors") only would have been able to add pages. I was hoping that people teaching Python would be interested in writing doctest-based exercises and that a useful collection could be implemented over time.

Unfortunately, this approach can not work, at least not using Google's App Engine on Google's own servers. :-(

Just playing with small pages, steps 1 and 2 are long enough that I get warnings logged mentioning that requests are taking too long. I know from experience that step 3 (which I have not started to implement/port from the standard Crunchy) can take even longer for reasonably size pages. So, this does not appear to be feasible ... which is unfortunate.

I think I will continue to develop this app to be used as a local one and perhaps write a second wiki-based app that would take html code with no further processing. I could use the first one to create a page, have it processed and use the "view source" feature of Firefox to cut and paste the content into the online app. This would remove the need for any processing of pages on Google's servers - only Python code execution would need to be taken care of. (Of course, a user could enter some code sample that would take too long to execute and hit Google's time limit ...)

If anyone has a better idea, feel free to leave it as a comment.

docpicture progress

For those interested, docpicture can now display images from the web. There's also a somewhat silly example where I embedded the code for a matplotlib example inside a docstring and have it displayed as a plot when viewing the docstring via docpicture inside a web browser. In order to do so I had to exec the code which is not exactly good practice ... but it serves to highlight the need to either only allow "parsers" from the standard distribution or require the user to give permission to a parser to be able to register itself with docpicture while it is running. I chose this second approach, although if you run the demo, you will not be given the opportunity to approve or not the parser - it will be done for you. This may need to be revisited...

I just announced a new release on the Python list. You can get docpicture 0.2 from here.