Scratching an itch: improving pydoc

Pydoc is great - no make that fantastic.  However, I was never fond of the look of help pages displayed when pydoc is used in webserver mode.    Pydoc uses hard-coded styling options (including font and colors) based on deprecated html 4 syntax.

I decided to see if I could make it more customizable and, after a half-day coding sprint, I offer you mod_pydoc, which is somewhat more easily customizable using css.  The new version has a simplified styling as default.

mod_pydoc is only available from github (for now); I'm hoping that people with better designing skills will contribute some code to make the output look better than my feeble attempt.  Eventually, my intention would be to file a feature request to python.org and submit it as a possible improvement on the current module.

Disgusted by LavaSoft AdAware antivirus

A short while ago, I installed LavaSoft AdAware free anti-virus after reading a review praising it.  Many years (and computers) ago, before it had an anti-virus, I found LavaSoft AdAware very useful in cleaning up my PC and keeping it that way, free of various malware.

To make a long story short, I found that it caused the following problems:

1. I could no longer update Bracket's extensions.

2. I could not run IPython notebooks, getting the error message:

WebSocket connection failedA WebSocket connection could not be established. You will NOT be able to run code. Check your network connection or notebook server configuration.

It took me many hours, deleting my IPython profile, uninstalling and reinstalling IPython, doing the same with Tornado, trying in various browsers, searching for posts about people having had similar problems, etc., etc., etc. until I found this stackoverflow post which had a different problem but made me think that it was perhaps related to an anti-virus program issue.

I stopped MadAware (and restarted the default Windows Defender) ... no change.  On a hunch, I uninstalled it and rebooted (for the Nth time) my computer.

Sure enough, that was the problem.

After deleting MadAware, my problems went away.

I had to get this off my chest before I could resume working on tomorrow's lesson where I wanted to show off the IPython notebook.  

Where are the edu-sig folks at the Python Education Summit?

After many years of being unable to attend Pycon, I look forward to this year's Pycon to be held (for the second time) in my favourite city, Montreal.  In the past, one of the highlights of that conference was meeting face to face other educators, many of whom I knew only through emails sent to edu-sig. At that time, there was no such thing as an official education summit.  Now there is.

According to the official description of the Education Summit,   We are taking proposals for talks from educators from all venues: authors; schools, colleges, universities; community-based workshops; online programs and government.

When I look at the current list of talk proposals, I see a list where very none of the proposals (other than my own) are from people who have contributee to edu-sig.  My own talk proposal (whose description is a little terse, I'll admit) has the dubious distinction of having been downvoted the most at this time! Unlike a few years ago, when I attended Pycon, there seems to be a huge disconnect between the people that contribute to edu-sig and who write books/tutorials/tools widely used in teaching Python, and the people currently interested in and submitting proposal to the Education Summit at Pycon.

(I would love to attend talks by people who subscribe to edu-sig and who have written books and/or tutorials about Python or created useful tools, like Naomi Ceder, Andrew Harrington, John Zelle or, probably the one I'd like to see the most, Philip Guo, who created the fantastic online python tutor.  One non-edu-sig contributor (as far as I know) who would be most interesting to hear from is Albert Sweigart who, in addition to writing many interesting books on Python is also thinking about making IDLE a better tool.)

There is one other talk that focus on a specific "tool" created to teach Python ("Python birds") in a postsecondary setting which has (and rightly so) seen an upsurge in votes in the past day or so.   While I do find some talks about teaching Python in non-traditional environments potentially interesting (and I have upvoted quite a few of the proposals for such talks), I don't think I would consider attending the Education Summit if that's all that there was.  However, the "Python birds" talk, if it gets accepted, would likely tip the balance for me and have me wish that I could be invited to attend, since it seems that participation is by invitation only.

EasyGUI_Qt progressing well.

A short while ago, I started working on an adaptation of EasyGUI for PyQt.  This new project is called EasyGUI_Qt.  For those that are not familiar with EasyGUI, the idea is to provide a series of graphical "widgets" that can be used in procedural programs.  For example, suppose you have a program where you query a user with Python's input:

    >>> input("Enter your reponse " )

Normally, you'd see a copy of the prompt appear, waiting for the user's response.  Using EasyGUI_Qt's  analogue of input(), this is what would show instead of a prompt:

There is also a demo launcher which can be used to see all of the available widgets:

A complete description of the existing widgets is available.  EasyGUI_Qt will not have all the same widgets available as the original EasyGUI.  While the current version number is 0.4, it is, in fact, nearly complete.  I have written a roadmap indicating what's left to be done until the 1.0 release. Anyone should feel free to make comments about the proposed roadmap including making suggestions for additional widgets (with justification provided).

EasyGUI_Qt targets Python 3.3 (and PyQt4) but should support (except for some pesky unicode issues) Python 2.7 - following some requests by users.

Review of IPython Notebook Essentials

Disclaimer: following a post on Google+ by a Packt representative, I volunteered to do a review of IPython Notebook Essentials and got a free copy as an ebook (I used the pdf format).

The verdict: Not recommended in its current state.

IPython Notebook Essentials starts very well.  It suggests to use the Anaconda 3.4 distribution (which I have on my computer - more on that later) or connecting online using Wakari.  Both methods worked fine for simple examples.

In chapter 1, we are given a quick tour of what the IPython notebook can do using the modeling of a cup of coffee as an example. The idea, on its own works well.  [As a scientist, I found the use of degree Farenheit a bit odd and reminded me of books I read that were written 50 or more years ago.]  However, while the authors used variables that follow the normal Python convention, separating names by underscore as in temp_cream, the code formatting is at times atrocious as variable names are sometimes split up after an underscore character as in the following two lines of code:

initial_temp_mix_at_shop = temp_mixture(temp_coffee, vol_coffee, temp_
cream, vol_cream)

which are, ironically enough, in bold in the text as the author wants us to focus our attention on their meaning (but no mention of the error in formatting).

While I usually prefer holding a paper book over reading an ebook on my screen, I must admit that being able to copy-paste code samples beats having to retype everything.  So, it was possible to quickly reproduce the examples by copy-pasting and fixing the typos to run the examples.

However, a much better way exists which is often used by Packt books: making code samples available for download.  The IPython notebooks have been designed for easy sharing.  Here, the author has chosen not to make use of this.  This, in my opinion, is something that is a major flaw in this case.


Chapter 2 covers in more details the notebook interface. This is undoubtably the most important chapter of the book for anyone who wishes to learn about the IPython notebook.  It covers a lot of grounds.

However, I encountered many problems, some more serious than others. The first, which is perhaps more an annoyance than a real problem, is that one example intended to show code completion using the tab key is given as follows:

print am

Python programmers will immediately recognize this as being a Python 2 example.  However, as far as I could tell (using ctrl-f "python 2") there isn't a single mention anywhere that Python 2 is used.
I happened to have the Anaconda 3.4 distribution installed (as recommended) but with Python 3.4 and not Python 2.  Python 3 is now 6 years old and there is no excuse to focus on an old, and soon to be obsolete, version of Python without at least mentioning why that choice was made. Minor syntax difference, like adding parentheses for print statements, are easily fixed; some more subtle ones are not.   Furthermore, while I had the Anaconda distribution installed, I was still using the online Wakari site to work through the examples, so that was not a major problem at that point.

While still in chapter 2,  we are invited to replace "%pylab inline" by "%pylab" and run an example again to see a plot appear in a window (I first assumed a separate browser window) instead of being shown in the document.    This does not work using the online Wakari site: the window is actually a Qt window, not something that works in a browser.  So, to try this example, I had to start my local version and recopy the code, which worked correctly.

Shortly thereafter, we are introduced to more "magic" methods and invited to try running a cython example, loading from a local file. This did not work.  The recommended "%%cython" magic method is no longer working in the latest IPython version included with the Python 3.4 Anaconda 3.4 distribution.  After a while, I found the proper way to run cython code with the latest version BUT the example provided raised a (numpy-related?) syntax error.  I copy-pasted the code from my browser to the Wakari online version and it worked correctly, confirming that I had not made an error in reproducing the code given by the author. However, I was not able to figure out the source of the error using the local version.

After finishing Chapter 2, I stopped trying to run every single examples and simply skimmed the book.

Chapter 3 focuses on producing plots with matplotlib, including animations. While not specific to the IPython notebook, this topic felt like an appropriate one to cover.

In Chapter 4, we learn about the pandas library which has very little to do with the IPython notebook.  The situation is similar with Chapter 5 which covers SciPy, Numba and NumbaPro, again three libraries that have very little to do with the notebook as such.  The choice of NumbaPro struck me as a bit odd since it is a commercial product.  True enough, it can be tried for free - however, it is not something that I would consider to be an "essential" for the IPython notebook.

I know very little more about the IPython notebook than what I have learned from this book. However, I do know that it is possible to write extensions for the IPython notebook which is something that I would have thought should be included in any book titled  "IPython Notebook Essentials", well before discussing specialized libraries such as Pandas, SciPy, Numba and NumbaPro.

There might very well be other topics more notebook specific that should be included, but I have no way to know from this book.

The book includes three appendices: a brief IPython notebook reference card, a brief review of Python, and an appendix on Numpy arrays.   Both the Reference card and the Numpy arrays appendices seem like worthwhile additions.  However, the brief review of Python seems a bit out of place.  By including code like:

def make_lorenz(sigma, r, b):
   def func(statevec, t):
      x, y, z = statevec
      return [ sigma * (y - x),
              r * x - y - x * z,
              x * y - b * z ]
   return func


in Chapter 2, the author seems to assume, and rightly so in my opinion, that the reader will be familiar with Python.  However, the appendix only covers the standard Python construct that one may find in a beginner's book intended for programmers that are familiar with other languages.  As such, the Python review appendix seems just like a filler, increasing the page count while adding nothing of value to the book. Thankfully, it is relegated to an appendix instead of being inserted in an early chapter.

In summary, about half of the book contains information of value for someone who wants to learn about the IPython notebook itself; the choice of Python 2 over Python 3 is odd, and almost inexcusable given that it is not even mentioned anywhere; the lack of downloable code samples" (mostly IPython notebooks in this case) greatly reduces the value of this book and is something that could be remedied by the author.  In fact, given the typos mentioned (where variable names are split over two lines), downloadable copies of notebooks should be made available.

As I write this review, Packt is having a sale during which ebooks are available for $5.  At that price, I would say that IPython Notebook Essentials is worth it if one wants to learn about the IPython Notebook; however, based on a quick scan of other Packt books covering the IPython notebook, I  believe that better choices exist from the same editor.

Reloadable user-defined library with Brython/Python

[Note: this post is a more detailed explanation of something that is briefly described in a previous post on supporting multiple human languages for Reeborg's World.]

In Reeborg's World, I want to have programmers (read: Python beginners who have never programmed before) to be able to learn about using libraries in Python.  As usual, I was looking at the simplest way to introduce the idea of libraries.  Since almost all of the programs that programmers write make use of their own definition for turning right:

def turn_right():
    turn_left()
    turn_left()
    turn_left()

it makes sense to have this "reusable" function be put in a library. So, instead of a single code editor, the code editor has two tabs: one for the basic program and one for the library. Initially, I only had a Javascript version of Reeborg's World, but I still wanted to introduce the concept of using libraries. So, when I first introduced support for using a library with Javascript, I cheated.  And I continued to cheat when I added Python support.  If the programmer wanted to use the functions (or anything else) defined in their library, I required them to insert the following statement in their program.

from my_lib import *

Before running the program using Brython's exec(), I would scan the code in the editor's program tab. If I found this general import statement, I would replace the line by the entire content of the editor's library tab and execute this modified source instead.   [Note: I have since removed the idea of using a library in the Javascript version since there is no natural syntax for importing a library using Javascript.]

However, this approach had two problems:

1. It did not support the other ways to use an import statement in Python (see below for an example).
2. It encouraged a bad practice of including everything, polluting the program's namespace.  I already had shown the idiom "from some_lib import *" when explaining how to have Reeborg understand instruction using other human languages (such as from reeborg_fr import * for the French version, or from reeborg_es import * for the Spanish version; other translations welcome! ;-)

I wanted to encourage good programming practice, such as using

from my_lib import turn_right

One problem I had is that Brython's import mechanism is based on retrieving files on the server using ajax calls.  This by itself might not be a problem ... except that I do not want to store anything created by users on the server: Reeborg's World is meant to be run entirely inside the browser, with no login required. (The content of the editor and library tabs are saved in the browser local storage so that they are available when the user comes back to the site using the same browser with local storage enabled.)

Another problem I had is that, once a module is imported, future import statements for that module make use of the cached version.  If the programmer modifies the code in their library (tab), the corresponding module needs to be reloaded.  I need for this to be done automatically, without the programmer having to do anything special.

One solution to these problems might have been to create a special importer class that could import code directly from the library tab and add it to sys.meta_path.  Then, after a program has been run, remove all traces of the imported module (user's library) so that the next time it is executed, the import takes place all over again.

I decided instead on a different approach.  I created a simple module, called my_lib.py  (and another one, biblio.py, for the French version) and put it in Brython's site package directory.  The content of that module is simply:

from reeborg_en import *

which ensures that all normal robot commands can be used in that module.  When Reeborg's World is first loaded, I import that module so that it is cached.  Then, whenever the programmer's code needs to be executed, instead of simply having exec(src) called, the following is called instead:

def generic_translate_python(src, lib, lang_import, highlight):
    ''' Translate Python code into Javascript and execute

        src: source code in editor
        lib: language specific lib (e.g. my_lib in English, biblio in French)
             already imported in html file
        lang_import: something like "from reeborg_en import *"
    '''
    # save initial state of lib
    initial_lib_dict = {}
    for key in lib.__dict__:
        initial_lib_dict[key] = lib.__dict__[key]

    exec(library.getValue(), lib.__dict__)
    exec(lang_import)
    if highlight:
        src = insert_highlight_info(src)
    exec(src)

    # remove added definitions
    new_keys = []
    for key in lib.__dict__:
        if key not in initial_lib_dict:
            new_keys.append(key)
        else:
            lib.__dict__[key] = initial_lib_dict[key]

    for key in new_keys:
        del lib.__dict__[key]

In the above, highlight refers to some pre-processing of the code which allow to show which line of the code is executed as illustrated in two previous blog posts.  library.getValue() is a method that returns the content of the library tab.

Still baffled by the Python 2/3 discussions

I'm ... baffled...

For the past few years, I've been focused mostly on doing my own things, and not really following what was happening in the "core" Python community.   Reading this post today by Brett Cannon about the "consensus" that has apparently emerged  by the language summit at PyCon 2014 about writing code compatible for both Python 2 and 3, I was reminded about the release of version 1.0 of Crunchy 

Crunchy 1.0 is compatible with Python 2.4, 2.5, 2.6 ... and 3.1. It is also compatible with Jython 2.5 modulo some bugs when trying to work with examples containing unicode strings.

That was in 2009.   At 2.1 MB (zipped), Crunchy was not exactly a small script...

Why has it taken so long for this to become the norm?....

Due to a lack of interest in Crunchy, I have essentially not developed it much further past that point, and it is almost certainly not compatible with newer versions of Python...