Thwarted by lack of speed

I was hoping to make an announcement of a new cool app based on Google's App Engine but unfortunately I have been thwarted by Python's relative lack of speed.

I have started working on a new version of Crunchy that would run as a web app on Google's servers. While the current version of Crunchy fetches existing html pages, processes them and displays them in the browser, this new version would retrieve html page content (in reStructuredText format) from Google's datastore, transform it into html, process it to add interactive elements, and then displays them.

This new app was going to be usable as a wiki to create new material. This was my starting point, greatly helped by an already existing wiki example that I adapted to use reStructuredText. When requesting a page, the following was supposed to happen:

1. reStructuredText content (for the body of the html page) is fetched from the datastore.
2. said content is transformed (by docutils) into html
3. html content is further processed by modified "crunchy engine" to add interactive elements.
4. modified html content is inserted in page template and made available.

The user would then be able to enter some Python code which could be send back to the App Engine using Ajax for processing and updating the page display.

A normal user would only be able to interact with already existing pages. Special users ("editors") only would have been able to add pages. I was hoping that people teaching Python would be interested in writing doctest-based exercises and that a useful collection could be implemented over time.

Unfortunately, this approach can not work, at least not using Google's App Engine on Google's own servers. :-(

Just playing with small pages, steps 1 and 2 are long enough that I get warnings logged mentioning that requests are taking too long. I know from experience that step 3 (which I have not started to implement/port from the standard Crunchy) can take even longer for reasonably size pages. So, this does not appear to be feasible ... which is unfortunate.

I think I will continue to develop this app to be used as a local one and perhaps write a second wiki-based app that would take html code with no further processing. I could use the first one to create a page, have it processed and use the "view source" feature of Firefox to cut and paste the content into the online app. This would remove the need for any processing of pages on Google's servers - only Python code execution would need to be taken care of. (Of course, a user could enter some code sample that would take too long to execute and hit Google's time limit ...)

If anyone has a better idea, feel free to leave it as a comment.

docpicture progress

For those interested, docpicture can now display images from the web. There's also a somewhat silly example where I embedded the code for a matplotlib example inside a docstring and have it displayed as a plot when viewing the docstring via docpicture inside a web browser. In order to do so I had to exec the code which is not exactly good practice ... but it serves to highlight the need to either only allow "parsers" from the standard distribution or require the user to give permission to a parser to be able to register itself with docpicture while it is running. I chose this second approach, although if you run the demo, you will not be given the opportunity to approve or not the parser - it will be done for you. This may need to be revisited...

I just announced a new release on the Python list. You can get docpicture 0.2 from here.

svg mathematical equation

Ok, it's done: mathematical equations generated dynamically and displayed as svg graphics. Only using the standard Python library ... and one "tiny" additional download: matplotlib. Here's the first result (saved as a "hard-copy"; you may have to download the page and reopen it locally using Firefox.)

Note: do not bother looking for the files in the "py-fun" repository where I had the first release of docpicture. I will clean up things a bit and do a new release from a different place.

As usual, comments & suggestions are welcome.

docpicture and uml sequence diagrams

In a previous post about docpicture, I gave an example of a graphics generated from this site as something that would be desirable to do. (You can find more examples here.) Well, it turned out to be easy to do ... at the cost of a server connection. I used the example given to embed a graphics inside a page and ... voilà, it is done. As long as one has a live internet connection (and assuming the websequencediagram server is not down), a graphics is generated as requested.

Eventually, I still would like to implement my own parser to create svg code for uml sequence diagrams rather than relying on an external service.

docpicture: initial release

The subject line says it all. It's a small download: less than 22 kB, available from here. Feedback and suggestions are definitely welcome.

docpicture: working ... and a query.

docpicture (see previous posts) is now working as a full prototype. By this, I mean that instead of doing

>>> help(some_object)

at the Python prompt, one can do

>>> from docpicture import view
>>> view(some_object)

and some_object's docstring will be displayed in your webbrowser, with any docpicture directive being translated so as to embed a nice picture. Well, by "any", I mean any turtle directive conforming to the limited syntax I have included.

When I compare the output of help() with that of docpicture.view(), I am struck at how much more information than simply the object's docstring is included. I have tried (briefly) to play with the pydoc module to see if I could redirect the output of help() to a string that I could process with docpicture.view() ... but to no avail.

If anyone knows how I could do this simply, I would be very grateful.

docpicture is going to be released (version 0.1) as soon as I complete a decent "readme" file.

--UPDATE-- Ok, after playing some more with pydoc, I found out how to do this.

In my module, I do the following:

import pydoc
from StringIO import StringIO
my_stdin = StringIO()
def my_pager(text):
   my_stdin.write(pydoc.plain(text))
   return
pydoc.pager = my_pager

pydoc.help(obj)
retrieved = my_stdin.getvalue()
my_stdin.close()

and use the retrieved text as I wish.

docpicture: getting closer

This is just a progress report for the curious among you: the previous two images were generated automatically from the docpicture code written above them. If everything goes well, by the end of the weekend I'll be ready to give a sneak preview of the code to anyone interested. Feel free to contact me.