from __experimental__ import something_new

Python programmers are used to the notation:

from __future__ import new_functionality

which allows to experiment with changes that are to become part of a future Python version.  These are hard-coded in the Python interpreter and cannot be created at will by an average Python programmer.   Wouldn't it be nice if there was a simple way to try out possible new syntax by an average Python programmer?   This blog post describes a way to do this, with  a link to working code.  The code is not very sophisticated: firstly, I am not a professional programmer and have no training in computer science; as a result, all I do is simple stuff.  Secondly, I wrote the bulk of the code in a single day (ending with this blog post); I am sure it could be greatly improved upon.

However, like I say to beginners on various forums: the important thing is to first make your program do what you want it to.

Before I discuss the details of the code, I am going to make a rather long digression to explain my original motivation and what I found along the way.

Something special about Pattis's Karel the Robot

Anyone that has read more than a few posts on this blog is likely familiar with my Python implementations of Karel the Robot:  a desktop version, named RUR-PLE, and a more modern and sophisticated web version, Reeborg's World.  In what follows, I will give examples using code that could be readily executed in Reeborg's World.  These are extremely simple ... but I give them to illustrate a very important point about what motivated me.

Reeborg is a faulty robot.  It can move forward and turn left (by 90 degrees) as follows:

move()
turn_left()

However, it cannot turn right directly.  Nonetheless, one can define a new function to make it turn right by doing three consecutive left turns:

def turn_right():
    turn_left()
    turn_left()
    turn_left()

Reeborg (like the original Karel) can make single decisions based on what it senses about its environment

if wall_in_front():
    turn_left()
else:
     move()

or it can make repeated decisions in a similar way:

while not wall_in_front():
    move()

Using simple commands and conditions, beginners can learn the basics of programming using such an environment.   Notice something important in the above code samples: no variables are used and there are no function arguments.

What if we wanted Reeborg to draw a square?   If I were to use Guido van Robot,  Python-like implementation, I would write the following

do 4:
    move
    turnleft

Once again, no variable nor any function arguments.    However, if I wanted to do the same thing in Reeborg's World, at least up until a few days ago days ago, I would have needed to write:

for var in range(4):
    move()
    turn_left()

So much for the idea of not having variables nor function arguments ....    I've always hated the "don't worry about it now" kind of statements made to students.  However, the alternative is to explain a rather complicated expression (for beginners at this stage, especially young children who have never seen algebra and the idea of a variable) ... Wouldn't it be nice if
one could write instead:

repeat 4:
    move()
    turn_left()

or, if one does not want to introduce a new keyword, write

for 4:
    move()
    turn_left()

and have Python recognize it as having the same meaning?...  So, I subscribed to the python-ideas list and made this suggestion.   The result was more or less what I was expected...  And, to be honest, I completely understand the reluctance to introduce such a change.  While I expected a lack of support for this suggestion, (and, to be honest, I wasn't myself convinced that it fitted entirely Python's design philosophy) I wasn't expecting the amount of heat it would generate ... However, a couple of people with experience teaching to young children were sympathetic, including Luciano Ramalho.  Terry Jan Reedy, who thought I was using IDLE, suggested that I just patch it and transform the code prior to execution. Since I was already transforming the code prior to execution to support line highlighting in Reeborg's World, it was rather trivial to add yet one more code transformation to support the "repeat" keyword as illustrated above.

So, I have a way to support a "repeat" keyword in Reeborg's World ... but it left me somewhat unsatisfied to have something like this not easily available elsewhere in the Python world.

Additional motivation

In one of his posts to python-ideas, Terry Jan Reddy mentioned a discussion on the idle-dev list about making idle friendlier to beginners. In one of his post, he mentioned the idea of having non-English keywords. This idea is not new. There already exists an unmaintained version with Chinese Keywords as well as a Lithuanian and Russion version. Maintaining a version based on a different language for keywords is surely not something simple ... nor I think it would be desirable. However, it might be possible to essentially achieve the same goal by using an approach I describe in the next section.

Even just adding new keywords can be quite difficult. For example, in this post, Eli Bendersky explains how one can add a new keyword to Python. "All" you ned to do is

1. Modify the grammar to add the new keyword
2. Modify the AST generation code; this requires a knowledge of C
3. Compile the AST into bytecode
4. Recompile the modified Python interpreter

Not exactly for the faint of heart...

A simpler way...

Using the unmodified standard Python interpreter, I have written some proof-of-concept code which works when importing modules that satisfy certain conditions. For example, if an imported module

contains as its first line

for __experimental__ import repeat_keyword

it will support constructs like

repeat 4:
    move()
    turn_left()

If instead, it has

from __experimental__ import function_keyword

then one will be able to write "function" instead of "lambda" (which has been identified as one of Python's warts by a few people including Raymond Hettinger.)
One can combine two transformations as follows:

from __experimental__ import repeat_keyword, function_keyword


def experimental_syntax():
    res = []
    g = function x: x**2
    repeat 3:
        res.append(g(2))
    return res


def normal_syntax():
    res = []
    g = lambda x: x**2
    for i in range(3):
        res.append(g(2))
    return res

and both functions will have exactly the same meaning.   One caveat: this works only when a module is imported.

How does it work?

Each code transformer, such as repeat_keyword.py and function_keyword.py, contains a function named "transform_source_code"; this function takes an existing source code as input and returns a modified version of that code.  These transformations can thus be chained easily, where the
output from one transformation is taken as the input from another.

The transformation happens when a module is imported using an import statement; this is an unfortunate limitation as one can not execute

python non_standard_module.py

from the command line and expect "non_standard_module.py" to be converted.

The magic occurs via an import hook.  The code I have written uses the deprecated imp module.  I have tried to figure out how to use the importlib module to accomplish the same thing ... but I failed. I would be greatful to anyone who could provide help with this.

Other potential uses

In addition to changing the syntax slightly to make it easier to (young) beginners, or to make it easier to understand to non-English speaker especially if they use a different character set, more experienced programmers might find this type of code transformation potentially useful.

When PEPs are written, they often contain small sample codes.  When I read PEPs which are under discussion, I often wish I could try out to write and modify such code samples to see how they work. While not all proposed code could be made to work using the approach I have described, it might be possible in many cases.  As an example, PEP 0465 -- A dedicated infix operator for matrix multiplication could almost certainly have been implemented as a proof-of-concept using the approach I used.  I suspect that the existence of the appropriate code converter would often enrich the discussions about proposed changes to Python, and allow for a better evaluation of different alternatives.

A suggestion to come?...

I'm not ready yet to bring another suggestion to the python-ideas list ... However ...

Python support the special "from __future__ import ..." construct to determine how it will interpret the rest of the code in that file.  I think it would be useful if a similar kind of statement "from __experimental import ..." would also benefit from the same kind of special treatment so that it
would work in all instances, and not only when a module is imported.   People could then share (and install via pip) special code importers and know that they would work in all situations, and not limited to special environments like Reeborg's World, or IDLE as suggested by T.J. Reddy and likely others.

However, a concrete suggestions along these lines will have to wait for another day...