Wednesday, May 17, 2017

What if range did not exist?

Over the years, various proposals for new syntactic constructs have been put forward to supplement or replace the range() function for looping over integers. Some of them have been documented in PEPs, whereas various others arose during discussions on the python-dev, comp.lang.python, and python-ideas lists.

This lead Guido van Rossum to write in 2005:

I don’t think that anyone can reasonably disagree with this assessment, even if I must admit to have suggested something somewhat related not too long ago on the python-ideas list.

Nonetheless, can we imagine what would have happened if range() had not been invented so early in Python’s history?

Consider a situation, in an alternative universe, where Monty, initially known as the Monty Python Programming Language, has evolved without guidance from a BDFL. Instead, through the wisdom of the crowds, and by sheer serendipity, it has grown into something that strongly resembles Python, except that there is no range() function - or, at least, not yet.  Please note that any reference to actual Python related discussions are made simply as a starting point for people interested in exploring what really happened, as opposed to what we invented below. Also, the fictitious names chosen below have been added in after I wrote the invented dialogues, with no connection whatsoever to any living person other than me - save for one obvious person who never actually wrote the 3 words I attributed to him.

Excerpts from the Monty-Musings IRC channel logs

# The evolution of the Monty Python Programming Language was known to be greatly influenced by discussions taking place on the Monty-Musings IRC channel.  The logs of these discussions are scattered over the Internet and it can be difficult to piece them together. As a self-appointed historian, I have assembled some relevant contributions found in the IRC logs; these discussions have taken place over many years. You will find these below, together with some added comments which will hopefully be useful to understand the historical context.

The_OP: I find myself often iterating over the integers.  Each time I end up writing an iterator to do so.  I think it would be useful if such an iterator could be included as a builtin in Monty, so that we could write something like:
   seq = [x for x in integers(n)]
and this would give us a list with integers from 0 to n-1.

The_Self_Appointed_Gate_Keeper: New builtins should only be added when there is a strong case to be made for it. Given the very weak motivation for this addition, there really is no point in continuing this discussion.

# As usual, everyone ignored comments from The_Self_Appointed_Gate_Keeper and the discussion continued. Nonetheless, since The_Self_Appointed_Gate_Keeper maintained such a long and consistent presence on the Monty-Musings IRC channel, we felt it necessary to mention this contribution as a typical example.

The_Mathematician: what you refer to as “integers” should really be named “Natural numbers”, although this would leave some ambiguity as to whether zero is included or not. Furthermore, integers are an infinite set; what you propose to have here is a half-closed interval [0, n), which should not be confused with a half-closed interval (0, n].
You should really be more careful when you use mathematical terms so as to avoid confusion.

The_Minimalist: Why not simply turn integers into iterables so that we could write, for example:

seq = [x for x in 5]   # seq = [0, 1, 2, 3, 4]

# Reference SPAM 276.   For those unaware of it, the normal process by which changes are made to Monty involves the writing of a Standard Proposal for an Addition to Monty, usually simply referred to as a SPAM.

The_Proofreader:  Turning integers into iterators would likely lead to potential problems since something like

x, = 1

would become syntactically valid.I really think we should avoid introducing syntax that is valid and can lead to very different results based to the addition of a possibly single misplaced comma.

# This observation likely contributed to the rejection of SPAM 276.

The_Zen_Fence_Sitter: The proposal needs to follow the “Explicit is better than implicit” principle, and The_Mathematician makes a good point.  I suggest instead to name this function sequence_of_integers_on_a_semi_closed_interval(n).  However, this would still leaves some ambiguity as to which end is closed and which end is open.   And furthermore, following “Special cases aren't special enough to break the rules”, I am not sure if the use case warrants the introduction of a new builtin.  Then again, given that “Although practicality beats purity”, perhaps there is a good case to be made for this addition.

The_Practical_One:  Given that slices already are half-closed, half-open, following Monty’s convention, it should be clear that it is the upper-half that is open.  However, I would make the suggestion that it should be possible to start at something else than zero. This could be written as semi_open_interval(lower, upper) when the lower argument is non-zero -- although I am not too keen on such a long name.

The_Proofreader: The comparison to slices leaves out the fact that a “slice” notation with a single index, like [n], refers to a single item.  Thus, with a single argument, the meaning of semi_open_interval(n) cannot be obtained by using a parallel with the slice notation. In fact, it would lead to confusion given that when only one argument is used, it is meant to represent the _second_ argument of two, with the first one having the value 0 by default, which would be very unusual for a Monty builtin.

The_Unusual_Syntax_Enthusiast: What about using some standard mathematical notation like
 for x in [lower, upper)

# At this point, both The_Practical_One and The_Proofreader jumped in and pointed out the risk of confusion with the notation already existing for lists and for tuples, and the problems associated with parsing more complex expressions with parentheses. This lead to a rather long discussion between the three of them, with a few others occasionally  joining in, unable to resist adding comments to something clearly irrelevant.

The_Mathematician: I think that The_Unusual_Syntax_Enthusiast has a point. In the general situation, the solution is simple.  There are only 4 cases worth considering and, as Knuth has shown many years ago, they can be written as follows:

for x in $[s,e]$:
for x in $(s,e]$:
for x in $[s,e)$:
for x in $(s,e}$:

This does not require any new keyword or builtin function, just a small change to Monty’s syntax to add $ as a useful symbol. Anyone with even the most basic knowledge of \TeX or \LaTeX would be able to quickly grasp the meaning of each of these 4 cases; it’s hard to think of anything that could possibly be more readable, especially if you keep your variable names short like we do in Mathematics.

By the way, it would be so much better if keywords would be prefixed by a backslash, like \for and \in as it would remove any limitation when choosing variable names as it is, most unfortunately, sometimes completely unavoidable to use multi-letter variable names. Furthermore, it would ensure total backwards compatibility when introducing new keywords. I cannot believe I would be the first one to suggest this obvious improvement to Monty.

# The_Mathematician, whose research on the Riemann Hypothesis had been at a standstill for years and had joined the Monty-Musings channel to maintain the delusion that she could contribute something useful to an important project, left the discussion at this point and, if rumours are to be trusted, went off to work on an alternative implementation of Monty done entirely using TeX macros.

The_Unusual_Syntax_Enthusiast: What about using a simple new notation like  
for x in [lower .. upper]

# A similar syntax had been mentioned by someone else before. Again, as was often the case following a suggestion from The_Unusual_Syntax_Enthusiast, a pointless and long discussion followed. Reporting on this discussion would not contribute to the understanding of what lead to the final result.

The_Teacher: I do like this basic idea of having such a built-in iterator.  When teaching Monty to beginners, I often use something like
for number in 0, 1, 2, 3, 4:

This is flexible enough and works for short sequences of numbers, but is not very useful for longer sequences.  However, instead of a builtin with a meaning that is not immediately obvious, what about having something like

for 0 <= number < 4:
  print number

It is both explicit and highly adaptable to other types of sequences. Its meaning would also be clear to my students.

# Historical note 1: A similar proposal gave rise to SPAM 284

# Historical note 2: the use of print both as a keyword and as a function in this contribution by The_Teacher helps us to identify the period at which it was made.  Monty went through what is now referred to as The Great Schism where the language originally known as the Monty Python Programming Language, and later referred to simply as Monty, introduced some backward incompatible changes, the most visible of which was removing print from the keyword list and making it a function. For a while, people referred to the two incompatible versions as Monty (the original version) and New Monty; this eventually changed to Monty Classic (the original version) and New Monty became referred to simply as Monty.  
During the period of upheaval, some rather heated discussions took place which lead some people to describe the Monty-Musing IRC channel as Monty Python’s F-ing Circus. A small troupe of British comedians adopted a sanitized version of this name, and based their comedy act on short plays with nonsensical dialogues, no doubt inspired by the Monty-Musings IRC channel discussions. To this day, the Monty Python’s Flying Circus has essentially disappeared save for a tiny cult following, unlike the ubiquitous Monty programming language.

# Historical note 3. Monty programs are easily recognized by their “.my” extension. Monty enthusiasts, aka Montinistas, have been known to name packages meant to be shared widely with a name also starting with “my”, enjoying the apparent contradiction of having something named “My…” and meant to be shared with all.  For example, one of the best known of such packages is Mygame, the famous gaming framework; another one is Mycroft (, an open source artificial intelligence package.  It has been rumoured that Mynecraft’s name has been inspired by Monty’s tradition. Thus, it is understandable that Montinistas, from both sides of the Classic vs New debate, jointly became upset when Mypy ( became public as it was seen clearly to be an attempt to do something perceived as going against the spirit of Monty and with a clear indication that it was not meant to be used except by a small elite group -- this, even though its name started with “My”. Eventually, a united front against Mypy, seen as a common enemy, contributed to the now near-unanimous adoption of New Monty as the true successor to the original Monty Python Programming Language.

The_Unusual_Syntax_Enthusiast: What about using an even simpler new notation and drop the square brackets  
for x in lower .. upper

# Since this was clearly never going to be considered seriously, another very long and pointless side discussion ensued.

The_Grammarian: While the latest suggestion by The_Teacher is interesting, it does not conform to the established pattern which, in its simplest form starts with  

for_stmt: 'for' looping_variable ...

whereas what is proposed does not follow. This could lead to some confusion and would require some non-trivial changes to the parser.

The_Unknown_Coder: What about something like this?

# According to the people we’ve been able to interview, The_Unknown_Coder had the unique habit of including screenshots of the code he ran, instead of using text like all other Montinistas. Since the Monty-Musing IRC Channels logs only preserved the text, we are unable to confirm that what we are presenting is faithfully accurate. We can only guess as to what code was submitted based from comments of other contributors and some minimal recollections from people we interviewed.  There is no evidence that The_Unknown_Coder made any positive contributions to Monty. However, during the discussion that we are documenting, The_Unknown_Coder did make several interventions which seemed to garner some interest from at least one respectable Montinista, even though there is little evidence that those musings had any influence on the final outcome.

The_Grammarian: What you are proposing would still require some difficult adjustments to the parser. It would be much better to use a new keyword instead of “in”, perhaps something like “between” or “over”.  

The_Self_Appointed_Gate_Keeper: New keywords should only be added when there is a strong case to be made for it. Given the very weak motivation for this addition, there really is no point in continuing this discussion.

The_Minimalist: This suggestion, which requires one to write the looping variable twice seems wasteful compare to the suggestion from The_Teacher.  

What about using a bare slice notation?

# Historical note: this last suggestion from The_Minimalist, and the discussion that followed, gave rise to SPAM 204. The fact that a suggestion by The_Minimalist gave rise to a SPAM was very unusual and is worth noting.

The_Unknown_Coder: (# Apparently in reply to The_Grammarian, although we cannot confirm that the following image reflects the code shown in the IRC channel.)
What about something like this instead, where the keyword is meant to be a short form of in_sequence?

The_Grammarian: That would work.  And I can see how your suggestion would naturally include both types of half-closed intervals, as well as open or closed intervals.

The_NASA_Engineer: In our script, we have the need for a sequence in reverse order. Could the proposal be made to accommodate this?

The_Old_Timer: We introduced some syntax to handle this case many years ago; it would be

for i in reversed(half_close_interval(n)):

The_OP: I would prefer a single iterator to handle all cases. Perhaps, like we do for slices, we can use -1 as an optional third argument, to indicate that we are using a reverse order.

The_Dutch_Guy: Call it range.

# Note: we dug through all the available records and, as far as we could tell, this terse statement, seemingly coming out of nowhere, was the only contribution from The_Dutch_Guy to this discussion.  It was also the only time that the word “range” was mentioned.

The_Unknown_Coder: My code can handle this case quite naturally.

The_Practical_One: Actually, using a third argument could allow one have integer sequences with steps greater than 1:

for i in half_close_sequence(0, 5, 2):
Which would print the first 3 even integers.

The_Unknown_Coder: My code can handle such cases naturally as well:

The_English_Litt_Graduate: I don’t like it. There are too many symbols and not enough words.

# The discussion continued for many more months but, as far as we could tell, nothing of any significance was added until a suggestion to write a SPAM was made.

The_Friendly_Dev: Dear OP, I think this discussion has been fruitful and deserves to be recorded.  Please, write your idea following the template for a Standard Proposal for an Addition to Monty. When you have finished, upload your SPAM to the DorkCentre repository and post an announcement on 4chan asking people to vote. A month later, we’ll do a vote count, with a weight given to the various 4chan boards following the historical precedent established before most people were able to access 4chan; some people refers to this as Monty’s Electoral College. This vote will decide if your SPAM is accepted or rejected.

Oh, and make sure to submit in triplicate a signed Hacking Agreement for Monty, which we require for anything that someone contribute which may be incorporated in Monty.. Submitting a HAM is required before a SPAM can be made public and a request for votes is posted.

The_OP: Is it wise to ask known trolls to vote on a SPAM like this?

The_Old_Timer: Look, this is how we’ve always proceeded so far, and it has worked out well. You don’t want to have a Committee decide on what to add to the language, otherwise you’ll end up with things like SimpleBeanFactoryAwareAspectInstanceFactory as typical names in the standard library. At the other extreme, you don’t want to have a single person decide or you might end up with something like the Oystr language.  No, democracy using the 4chan weighted Board Electoral System with Trolls  is clearly and tautologically the BEST  way to decide on improvements to Monty. Just trust the wisdom of the crowds. In fact, some countries have adopted a similar practice, also calling it an Electoral College.

The_Friendly_Dev: One more thing, The_OP.  Make sure to include any contribution made by The_Dutch_Guy in your SPAM, however insignificant that contribution seems to you. For some reason, whenever The_Dutch_Guy writes anything, the folks on 4chan get really interested in the outcome and vote in large numbers, all converging towards a sensible solution. Nobody knows why, but it is so striking that it has become immortalized in the Zen of Monty.

# After a hugely popular and positive vote on 4chan, range() became a Monty builtin.

Friday, May 05, 2017

What's in a name?

tl;dr: I'm changing the name of "__nonstandard__" back to "__experimental__".

Struggling with naming things

Since I discovered Python and started programming as a hobby in my early forties, I've often found it difficult to come up with "the right name". My first real program followed in the footsteps of Guido van Robot (aka GvR, one of the best ever named Python computer programs), and was uninspiringly named RUR-PLE, meant as an acronym.  It spite of this poorly constructed name, RUR-PLE/rur-ple  became somewhat popular, having been downloaded well over 50000 times - but I always wondered if it would have been more easily found and popular if its name had been a short phrase with either the word "robot" of "karel" in it.

I remember reading comments about Crunchy (originally known as Crunchy Frog, after a famous Monty Python sketch, and later shortened since another open source program was named Crunchy Frog) indicating that such a silly name should not be taken seriously.

My best-named experimental program was probably docpicture, which was meant to easily enhance docstrings so that, using an alternative to Python's help() function, one could see docstrings with relevant embedded images. As an example, here was an example for a turtle module, and another one for sequence diagrams.  When using docpicture, a browser would be launched to view the resulting enhanced docstring. This felt somewhat unnatural when dealing with Python code and contributed to my abandoning this idea.  Nowadays, with the Jupyter notebook available, such an enhancement to traditional doctrings might be worth revisiting.

However, docpicture was an exception: I do not find the names of other programs I have written inspiring in any way.

This brings me to "experimental/nonstandard".  When I first started looking at ways to easily explore modifications/additions to Python syntax,  having a construct like

from __experimental__ import some_syntax

seemed a logical choice.  When I found out that this construct had been suggested before (in a different context), and even though it had not been adopted, I thought I should come up with an alternative.  Thus __experimental__ became __nonstandard__.

However, I believe that this new name gives the wrong impression. If I come accross some code in a repository and I see something like

from __nonstandard__ import something

I would likely see the word "nonstandard" as a warning and interpret this as "the authors are doing their own thing, straying away from the standard Python code; it's probably better to stay away."

On the other hand, if I read

from __experimental__ import something

I would likely interpret this as "the authors are trying out something different; it might be worth having a look as I may learn something new".

This is what I would like experimental to be: a Python package that lets people try out new syntactic constructs easily, coming up with new ideas or testing ideas proposed by others (in PEPs, for instance); not something to be used in production code, but something to experiment with: hence why I am reverting back to using the first name I had chosen.

Saturday, April 29, 2017

nonstandard module and PEP 542

PEP 542 proposes a shortcut notation when defining functions that would be added to a class. A discussion of this idea can be found here.   Personally, I am not too fond of the idea ... but it's a good test case to see if I my nonstandard module, which I have mentioned before on this blog, could allow the proposed syntax to be used right now.

$$ type
from __nonstandard__ import pep542

def test_pep542():
    class MyClass:

    def MyClass.square(self, x):
        return x**2

    a = MyClass()

    def a.out():
        return 42

    assert a.out() == 42
    assert a.square(3) == 9
$$ python
Successfully tested function test_pep542

The answer seems to be yes ...

Monday, April 24, 2017

Easily modifiable Python

You want to try out some new syntactic construction in Python but do NOT want to have to go through the trouble of

1. modifying Python's grammar file
2. modifying Python's lexer
3. modifying Python's parser
4. modifying Python's compiler
5. re-compile all the sources  so as to create a new Python interpreter?

(These steps are described in these two blog  posts.)

It can be done much more easily using any existing Python interpreter! ;-)

More details soon...

Monday, December 12, 2016

Reeborg's answer to pip

If you are not familiar with Reeborg's World, then this post is likely not of interest to you and I apologize for the title.

As Python programmers know, pip is extremely convenient to install Python modules.  However, when creating a website like Reeborg's World which does not store any user code on the server and run everything in the client (browser), the idea of installing a Python module may seem to be totally nonsensical: how can one install a Python module in a web browser?

Yet, since Reeborg's World aims to teach not only basic Python programming but also Python's concepts like creating and using a library and third-party modules, it only made to me that I should attempt to reproduce the basic idea of installing third-party modules.  After all, it already allow users to write their own functions in a library and import them; so why not finding a way to "install" third party modules from the web.

Before I explain how to do so, let me recap what can be done with Reeborg's World (just in case you are not familiar with it and did not get put off by the opening sentence of this blog post).

The user (student) enters Python code [1] in the editor and runs it, making Reeborg accomplish various tasks.

The student can also enter functions in the library and import them as usual. Reeborg's World uses Brython as its Python interpreter.

World creators (educators) can create their own worlds, making use potentially of six different editors:

  • The Python code editor: one can save some program (e.g. a stub) together with a world as a starting point for the students.
  • The library - which can be prepopulated by the educator
  • The pre- and post- editors. When a program is run, the code found in the "pre-" editor is concatenated with the code found in the main editor followed by the code found in the "post-" editor.  This allow the world creator to set up some particular condition, going beyond what is possible to do with the graphical world editor.
  • The "description" editor. The world creator can add information about the world using html; this can include images.  Some information (position of robot, final position to reach, etc., is extracted automatically from the world file content and added to the information.
  • The "onload" editor: this allow the world creator to add some (Javascript) code that will be automatically executed once when the world is loaded. One can use this to load new images to use for the robot, define new objects types (with pictures) that can be used, etc.  The images can be loaded from anywhere on the web.
In theory, an educator could design worlds that would include either in the "post-" editor or in the library some code to be used by the students.  However, this would have to be done every time a world is created, which can be quite tedious.  With the new functionality, one can load a script from anywhere on the web and have it made available as the content of a module named "extra".

Instead of calling the new function "install_extra()", I was toying with the idea of calling it "pip_install()" ... but I thought this might be 1) too cheeky and 2) potentially confusing for beginners when they moved to a "real" Python environment and had to use pip from the command line, and not from within an editor.  However, I am  not totally convinced that "extra" is the best name for this library nor that "install_extra()" is the best name for the new function.  If you have any suggestion or comments, feel free to do so.

For those that are curious and would like to try it, click on this link and then make sure to click on the button to have the content of the editor replace by that included with this sample world.
Note that this link is for a development version and may not work in the future.

This feature has just been implemented and may very well be buggy. To load modules, I make use of cross-domain http requests and a third-party server ( When I first attempted to write code on pastebin, and then edit it, I found that the original version was cached ... which was not very helpful as I did not realize it at first and was wondering how come the changes I made to the code were not reflected in what I saw.  Coders beware...

Suggestions and comments, as usual, are most welcome.

[1] In addition to Python, Javascript is also supported. Instead of a traditional editor, one can also use Blockly or a Python REPL to program the robot.

Monday, September 19, 2016

Backward incompatible change in handling permalinks with Reeborg coming soon

About two years ago, I implemented a permalink scheme which was intended to facilitate sharing various programming tasks in Reeborg's World. As I added new capabilities, the number of possible items to include grew tremendously. In fact, for rich enough worlds, the permalink can be too long for the browser to handle. To deal with such situations, I had to implement a clumsy way to import and interpret those permalinks after the page had been loaded. To give an idea of the potential complexity, here is a permalink (used by a teacher in Lithuania) that is still short enough to be handled by most browsers:*%27%20in%20Python%20Code%20is%20required%20to%20use%0A%23%20the%20code%20in%20this%20library.%20%0A%0A

The above permalink appears to work ... but, due to a small addition to the old version of the site that handles these permalinks, the result is not quite what is desired: one must change "by hand" the programming mode (at the top) from Blockly to Editor.

The new permalinks will be much shorter and should be future proof; they can include, in any order:

  1. the programming mode/language
  2. the human language for the interface
  3. a URL pointing to a file containing the world definition
  4. an optional name to display in the world select menu instead of the URL.

The key here is the 3rd item. Instead of having to potentially deal with changing world format (to add new capabilities) AND permalinks to encode the new items, I only have to ensure that the world format remains backward compatible. Normal URL where a world can be found are usually quite short, and there is no need to worry about hitting some browser limit in terms of the number of characters.

For example, here is a URL that works on a temporary testing site, which loads a world definition I put on Pastebin (which anyone can do):

If you use Reeborg's World to teach and wish to be kept abreast of any major changes, please email me and I will add you to a mailing list that I just set up for this purpose.