Reloadable user-defined library with Brython/Python

[Note: this post is a more detailed explanation of something that is briefly described in a previous post on supporting multiple human languages for Reeborg's World.]

In Reeborg's World, I want to have programmers (read: Python beginners who have never programmed before) to be able to learn about using libraries in Python.  As usual, I was looking at the simplest way to introduce the idea of libraries.  Since almost all of the programs that programmers write make use of their own definition for turning right:

def turn_right():
    turn_left()
    turn_left()
    turn_left()

it makes sense to have this "reusable" function be put in a library. So, instead of a single code editor, the code editor has two tabs: one for the basic program and one for the library. Initially, I only had a Javascript version of Reeborg's World, but I still wanted to introduce the concept of using libraries. So, when I first introduced support for using a library with Javascript, I cheated.  And I continued to cheat when I added Python support.  If the programmer wanted to use the functions (or anything else) defined in their library, I required them to insert the following statement in their program.

from my_lib import *

Before running the program using Brython's exec(), I would scan the code in the editor's program tab. If I found this general import statement, I would replace the line by the entire content of the editor's library tab and execute this modified source instead.   [Note: I have since removed the idea of using a library in the Javascript version since there is no natural syntax for importing a library using Javascript.]

However, this approach had two problems:

1. It did not support the other ways to use an import statement in Python (see below for an example).
2. It encouraged a bad practice of including everything, polluting the program's namespace.  I already had shown the idiom "from some_lib import *" when explaining how to have Reeborg understand instruction using other human languages (such as from reeborg_fr import * for the French version, or from reeborg_es import * for the Spanish version; other translations welcome! ;-)

I wanted to encourage good programming practice, such as using

from my_lib import turn_right

One problem I had is that Brython's import mechanism is based on retrieving files on the server using ajax calls.  This by itself might not be a problem ... except that I do not want to store anything created by users on the server: Reeborg's World is meant to be run entirely inside the browser, with no login required. (The content of the editor and library tabs are saved in the browser local storage so that they are available when the user comes back to the site using the same browser with local storage enabled.)

Another problem I had is that, once a module is imported, future import statements for that module make use of the cached version.  If the programmer modifies the code in their library (tab), the corresponding module needs to be reloaded.  I need for this to be done automatically, without the programmer having to do anything special.

One solution to these problems might have been to create a special importer class that could import code directly from the library tab and add it to sys.meta_path.  Then, after a program has been run, remove all traces of the imported module (user's library) so that the next time it is executed, the import takes place all over again.

I decided instead on a different approach.  I created a simple module, called my_lib.py  (and another one, biblio.py, for the French version) and put it in Brython's site package directory.  The content of that module is simply:

from reeborg_en import *

which ensures that all normal robot commands can be used in that module.  When Reeborg's World is first loaded, I import that module so that it is cached.  Then, whenever the programmer's code needs to be executed, instead of simply having exec(src) called, the following is called instead:

def generic_translate_python(src, lib, lang_import, highlight):
    ''' Translate Python code into Javascript and execute

        src: source code in editor
        lib: language specific lib (e.g. my_lib in English, biblio in French)
             already imported in html file
        lang_import: something like "from reeborg_en import *"
    '''
    # save initial state of lib
    initial_lib_dict = {}
    for key in lib.__dict__:
        initial_lib_dict[key] = lib.__dict__[key]

    exec(library.getValue(), lib.__dict__)
    exec(lang_import)
    if highlight:
        src = insert_highlight_info(src)
    exec(src)

    # remove added definitions
    new_keys = []
    for key in lib.__dict__:
        if key not in initial_lib_dict:
            new_keys.append(key)
        else:
            lib.__dict__[key] = initial_lib_dict[key]

    for key in new_keys:
        del lib.__dict__[key]

In the above, highlight refers to some pre-processing of the code which allow to show which line of the code is executed as illustrated in two previous blog posts.  library.getValue() is a method that returns the content of the library tab.