HackInScience: friendly Python learning

A short while ago I discovered HackInScience, a fantastic site for learning Python by doing exercises. It currently includes 68 programming exercises, with increasing level of difficulty.
I learned about it via an issue filed for Friendly-traceback: yes, HackInScience does use Friendly-traceback to provide feedback to users when their code raises Python exceptions.  These real-life experiences have resulted in additional cases being covered by Friendly-traceback: there are now 128 different test cases, each providing more helpful explanation as to what went wrong than that offered by Python. Python versions 3.6 to 3.9 inclusively are supported.

Previously, I thought I would get feedback about missing cases from teachers or beginners using either Mu or Thonny - both of which can make use of Friendly-traceback. However, this has not been the case yet, and this makes me extremely grateful for the feedback received from HackInScience.

While Friendly-traceback can provide feedback in either English or French [1], HackInScience only uses the English version - this, in spite of the fact that it was created by four French programmers.  I suspect that it is only a matter of time until they make a French version of their site.

One excellent additional feature provided by HackInScience is the addition of formatting (including some colour) in the output provided by Friendly-traceback.

The additional cases provided by Julien Palard from HackInScience have motivated me to clear out the accumulated backlog of test cases I had identified on my own. Now, there is only one (new) issue: enabling coloured output from Friendly-traceback's console.

Please, feel free to interrupt my work on this new issue by submitting new cases that are not covered by Friendly-traceback! ;-)

[1] Anyone interested in providing translations in other languages is definitely welcome!

True constants in Python - part 2, and a challenge

Like the title says, this is the second post on this topic. If you have not done so, you should really read the first one, which is much shorter, before continuing to read here.

The first clue

The first clue is that, rather than executing test.py as the main module, I imported it. Can you think of a situation where this would make a difference?

I'll leave a bit of space below to give you the opportunity to possibly go back to part 1, without reading about the solution below.

PEP 302

Back in 2002, with the adoption of PEP 302, Python enabled programmers to modify what happens when a module is imported; this is known as an import hook.  For example, it is possible to modify the source code in a module prior to it being executed by Python. This is NOT what I have done here - but I have done this for other examples that I will refer to below.

If the only thing required would be to modify the source, one could use what is described in PEP 263 and define a custom encoding that would transform the source. In this case, by adding an encoding declaration, it would have been possible to run test.py directly rather than executing it.  I thought of it a while ago but, in order to cover all possible cases, one would pretty much have to write a complete parser for Python that could be used to identify and replace the various assignments statement done by print statements so as to show what you saw in part 1.  However, this still would not be enough to protect against the reassignement done externally, like I did with

test.UPPERCASE = 1

The actual solution I used required three separate steps. The challenge I will mention at the end is to reduce this to two steps - something that I think is quite possible but that I have not been able to do yet - and to remove one left-over "cheat" which would allow one to redefine a constant by monkeypatching.  I think that this is possible but I have not actually sat down to actually do it. I thought of waiting for a few days to give an added incentive for anyone who would like to try and get the bragging rights of having it done first! ;-)

Step 1

Step 1 and 2 involve an import hook. They are independent one of another and can be done in any order.

When importing a module, Python roughly does the following:

1. Find the source code
2. Create a module object
3. Execute the source code in the module object's dict.

The module object created by Python comes with a dict that is, in some sense, "read-only": you cannot write code to modify its behaviour, nor replace it by a custom dict. (However, see the challenge.)  However, one can define a custom dict, which is designed so that its various methods (__setitem__, __delitem__, etc.) prevent the reassignment of variables we intend to be constants. In the example I have chosen, these are variables whose names are in UPPERCASE.  (Not shown in the example of part 1: I have also added a scan of the code to identify any variable that used the type hint Final and add them automatically to the list of variables intended to be constants.)

Instead of executing the code in the module object's dict, it is executed in this special dict. The content of that dict is then copied into the module object's dict.

Doing this ensures that code run directly in the module is guaranteed to prevent variable reassignement. At least, I have not found a way to cheat from within a module and change the value of variables intended to be a constant.

Step 2

Step 2 is to define a custom class that prevent changes of attributes. This custom class is used to replace the module's own class, something that can be done.

Step 3

Step 3 is to make Python use our import hook. To do so, we must have some code being executed earlier than what is shown. There are a couple of ways to do this as describe in the Site-specific configuration hook section of the Python documentation. The method I have chosen is one that is easily done on an ad-hoc basis.  I created a file named usercustomize.py whose content is the following:

from ideas.examples import constants

constants.add_hook()

This calls my code that sets up an import hook as described above. To have Python execute this code, I set the environment variable PYTHONPATH to be equal to the directory where usercustomize is located. On Windows (which is what I use), this is most easily achieve by navigating to that directory in the terminal and entering the following:

set PYTHONPATH=%CD%

Doing so will ensure that the code in usercustomize.py is executed before any user code.

The challenge

As mentioned in part 1, attempting to modify the value of a constant from outside, as in:

This leaves one possible cheat. From an external module, instead of writing

import test

test.UPPERCASE = "new value"

which is prevented, one can use the following cheat

import test
test.__dict__["UPPERCASE"] = "new value"

This is because the module's __dict__ is a "normal" Python dict. 

However, instead of using a module object created by Python, it should be possible to create a custom module object that uses something like the special dict mentioned before. Thus one would not need to change the way that Python execute code in the module's dict.

The challenge is to write code that creates such a module object.   I would not be surprised if there remained some other ways to cheat after doing so, but hopefully none as obvious as the one shown above.

Resources

The code I have written is part of my project named ideas.  The actual code for the constants example is given by this link.  See also the documentation for the project.  Note that, token_utils mentioned in the documentation has been put in a separate project; I need to update the documentation.

Both ideas and token-utils can be installed from Pypi.org as usual.

True constants in Python - part 1

tl;dr: I'm always trying to see if what everyone "knows to be true" is really true...

In many programming languages, you can define constants via some special declaration. For example, in Java you can apparently write something like:

public static final String CONST_NAME = "Name";

and this will result in a value that cannot be changed.  I wrote "apparently" since I do not program in Java and rely on what other people write.

Everyone "knows" that you cannot do the same in Python.  If you want to define constants, according to Python's PEP 8, what you should do is the following

Constants are usually defined on a module level and written in all capital letters with underscores separating words. Examples include MAX_OVERFLOW and TOTAL.

and rely on the fact that everyone will respect this convention.  However, nothing prevents you from redefining the value of these variables later in the same module, or from outside (monkeypatching) when importing the module.

Thus, if I write in a module

TOTAL = 1
# some code
TOTAL = 2

the value of the variable will have changed.

If you are willing to use optional type declaration and either use Python 3.8 with the typing module, or some earlier version of Python but using also the third-party typing-extension package, you can use something like the following:

from typing import Final

TOTAL: Final = 0 

and use a tool like mypy that will check to see if the value of is changed anywhere, reporting if it does so.  However, if you do run such an incorrect program (according to mypy), it will still execute properly, and the value of the "constant" will indeed change.

For people that want something a bit more robust, it is often recommended to use some special object (that could live in a separate module) whose attributes cannot change once assigned. However, this does not prevent one from deleting the value of the "constant object" (either by mistake within the module, or by monkeypatching) and reassign it.

Every Python programmer knows that the situation as described above is the final word on the possibility of creating constants in Python ... or is it?

For example, here's a screen capture of an actual module (called test.py)

Notice how the linter in my editor has flagged an apparent error (using UPPERCASE after deleting it.)  And here's the result of importing this module, and then attempting to change the value of the constant.

Can you think of how I might have done this?  (No photoshop, only the normal Python interpreter used.)

In part 2  , I explain how I have done this and will leave you with a (small) challenge.

Implicit multiplication in Python - part 1

Quoting from a post from Guido van Rossum

    ... The power of visual processing really becomes apparent when you combine
    multiple operators. For example, consider the distributive law
        mul(n, add(x, y)) == add(mul(n, x), mul(n, y))  (5)
    That was painful to write, and I believe that at first you won't see the
    pattern (or at least you wouldn't have immediately seen it if I hadn't
    mentioned this was the distributive law).
    Compare to
        n * (x + y) == n * x + n * y    (5a)
    Notice how this also uses relative operator priorities. Often
    mathematicians write this even more compact
        n(x+y) == nx + ny    (5b)
    but alas, that currently goes beyond the capacities of Python's parser.
    ...
    Now, programming isn't exactly the same activity as math, but we all know
    that Readability Counts, and this is where operator overloading in Python
    comes in.  ...

What if we could do something half-way between what Python currently allow
and what mathematicians would write by transforming something that is currently a SyntaxError into valid Python code?


    >>> from ideas.examples import implicit_multiplication as mul
    >>> hook = mul.add_hook()
    >>> from ideas import console
    >>> console.start()
    Configuration values for the console:
        callback_params: {'show_original': False, 
                          'show_transformed': False}
        transform_source from ideas.examples.implicit_multiplication
    --------------------------------------------------
    Ideas Console version 0.0.7a. [Python version: 3.7.3]

    ~>> 2(3 + 4)
    14
    ~>> a = 3
    ~>> b = 4
    ~>> 2a
    6
    ~>> a b
    12

All that is needed is to change the way the code is tokenized before the code is parsed by Python.

From a rejected Pycon talk to a new project.

Like many others, my talk proposal (early draft here) for Pycon US was rejected. So, I decided to spend some time putting everything in a new project instead. (Documentation here.)  It is still a rough draft, but usable ... and since I've mentioned it in a few other places, I thought I should mention it here as well.

Xmas present from Thonny

Today, a new version (3.2.5) of Thonny has been released. It incorporates support for Friendly-traceback (which needs to be installed separately). Currently, the download link on Thonny's homepage still links to version 3.2.4. The latest version can be found on Github.

Thonny is a fantastic IDE for beginners, especially those learning in a classroom environment, as it offers many useful tools that can be used effectively by teachers to demonstrate some programming concepts.  Thonny is the work of Aivar Annamaa, who is apparently recognized as an excellent lecturer -- which does not suprise me given the thoughtful design of Thonny. He has been interviewed about Thonny on PythonPodcast.

Real Python has a fairly comprehensive review here.

Friendly Mu

Link --> See the documentation for details.

A Tiny Python Exception Oddity

Today, while working on Friendly-traceback (improved documentation !) as I have been doing a lot recently, I came into an odd SyntaxError case:

• The inconsistent behaviour is so tiny, that I doubt most people would notice - including myself before working on Friendly-traceback.
• This is SyntaxError that is not picked up by flake8; however, pylint does pick it up.
• By Python, I mean CPython.  After trying to figure out why this case was different, I downloaded Pypy and saw that Pypy did not show the odd behaviour.
• To understand the origin of this different behaviour, one needs to look at some obscure inner parts of the CPython interpreter.
• This would likely going to be found totally irrelevant by 99.999% of Python programmers. If you are not the type of person who is annoyed by tiny oddities, you probably do not want to read any further.

You have been warned.

Normal behaviour

When Python finds a SyntaxError, it flags its location.  Let's have a look at a simple case, using CPython 3.7.

Notice how it indicates where it found the error, as shown by the red arrow: this happened when it reached a token that was inconsistent with the code entered so far. According to my experience until today, this seemed to be always the case.  Note that using CPython 3.6 yields exactly the same behaviour, and unhelpful error message.

Before discussing the case with a different behaviour, let's make a detour and look at Pypy's handling of the same case.

Same location indicated, but a much more helpful error message, even though this is version 3.6.  This improved error message was discussed in this Pypy blog post.  I strongly suspect that this is what lead to this improved error message in CPython 3.8.

Same error message as Pypy ... but the exact location of the error, previously indicated by ^, no longer appears - which could be unfortunate when nested parenthesis (including square and curly brackets) are present.

What about Friendly-traceback you ask? I thought you never would! ;-)  

Well, here's the information when using CPython 3.7.

The line about not having enough information from Python refers to the unhelpful message ("invalid syntax"). Hopefully you will agree that the information given by Friendly-traceback would be generally more useful, and especially more so for beginners.   

But enough about this case. It is time to look at the odd behaviour one.

Odd case

Consider the following:

Having a variable declared both as a global and nonlocal variable is not allowed.  Let see what happens when this is executed by Pypy.

So, pypy processed the file passed the nonlocal statement and flagged the location where it encountered a statement which was inconsistent with everything that had been read so far: it thus flagged that as the location of the error.

Now, what happens with CPython:

The location flagged is one line earlier. The nonlocal statement is flagged as problematic but, reading the code up to that point, there is no indication that a global statement was encountered before.

Note that, changing the order of the two statements does not change the result: pypy shows the beginning of the second statement (line 6) as the problem, whereas CPython always shows the line before.

Why does it matter to me?

If you go back to the first case I discussed, with the unmatched parenthesis, in Friendly-traceback, I rely on the location of the error shown by Python to indicate where the problem arose and, when appropriate, I look *back* to also show where the potential problem started.  Unfortunately, I cannot do that in this case with CPython.

Why is this case handled differently by CPython?

While I have some general idea of how the CPython interpreter works, I absolutely do not understand well enough to claim with absolute certainty how this situation arise.  Please, feel free to leave a comment to correct the description below if it is incorrect.

 My understanding is the following:

After breaking down a file into tokens, parsing it according to the rules of the Python grammar, an abstract syntax tree (AST) is constructed if no syntax error is found.  The nonlocal/global problem noted is not picked up by CPython up to that point - which also explains why flake8 would not find it as it relies on the AST, and does not actually executes the code.  (I'm a bit curious as to how Pylint does ... I'll probably have to look into it when I have more time).

Using the AST, a control flow graph is created and various "frames" are created with links (GOTOs, under a different name...) joining different parts.  It is at that point that relationships between variables in different frames is examined in details.  Pictorially, this can be represented as follows:

(This image was taken from this blog post by Eli Bendersky)  In terms of the actual code, it is in the CPython symtable.c file. At that point, errors are not found by scanning lines of code linearly, but rather by visiting nodes in the AST in some deterministic fashion ... which leads to the oddity mentioned previously: CPython consistently shows the first of two statements as the source of the problem, whereas Pypy (which relies on some other method) shows the second, which is consistent with the way it shows the location of all SyntaxError messages.

Conclusion

For Friendly-traceback, this likely means that for such cases, and unlike the mismatched parenthesis case, I will not attempt to figure out which two lines are problematic, and will simply expand slightly on the terse one liner given by Python (and in a way that can be translated into languages other than English).

pydeps: a very useful program

A few weeks ago, I was doing some refactoring of Friendly-traceback and had some minor difficulty in avoiding the creation of circular imports.  For some reason (age perhaps), I could not visualize the file structure properly.  Enter pydeps.  After I used it to generate a graph for all the files internal to Friendly-traceback, I was able to use that graph to figure out a better way to structure my program.

Today, as I stared at that graph, after including it in the newly styled documentation, I noticed that the "version" file I had created early on, was really redundant since its content (a single variable) could easily be incorporated in the Public API file.

So, one less file to deal with!

I think I am going to use pydeps a lot more from now on when I want to try to understand the how projects are structured, as I do find this type of graph very useful.

Significant changes for some error messages in Python 3.8

As I work on including more exceptions in Friendly-traceback, I am mostly pleasantly surprised by generally more precise error messages.  For example, in Python 3.7, the following

__debug__ = 1

would yield "SyntaxError: assignment to keyword" which likely would baffle almost everyone looking up the list of Python keywords.   In Python 3.8, that message has been replaced by the more precise: "SyntaxError: cannot assign to __debug__". Much better, in my opinion, even though one may be surprised to learn about this constant.

However, today as I was working on adding another case, I came accross the following:

This change is ... unexpected.  And the "helpful hint", is not so helpful in this case.  However, I can guess as to how it came about.  It will be a challenge to provide a "friendly" explanation that does not lead the users looking for an incorrect solution to their problem.

Edit: Based on my own (limited) experience working on Friendly-traceback, I do realize that trying to provide helpful hints to assist programmers in fixing code that raises exceptions is a very difficult problem.  My task, with Friendly-traceback, is generally made much easier thanks to accurate and usually helpful error messages provided by Python. So, please do not read this post and conclude that I am dismissive of the efforts of the Python core developers in this area. 

Friendly-traceback, Real Python, Pycon, and more

After an interruption that lasted a few months, I've finally been able to return to programming, more specifically working mostly on Friendly-traceback. For those that do not know Friendly-traceback: it aims to replace the sometimes obscure traceback generated by Python with something easier to understand. Furthermore, Friendly-traceback is designed with support for languages other than English so that, in theory, beginners (who are the main target audience for Friendly-traceback) could benefit no matter what their native language is ... provided someone would have done the translation into that language, of course.

As of now, 75 different cases have been tested; you can find them in the documentation.  [If you have suggestions for improvements, please do not hesitate to let me know.]

Recently, a post by Real Python on SyntaxError has given me added impetus to work on Friendly-traceback. I'm happy to report that, other than the cases mentioned dealing with misspelled or missing keywords, all of the other examples mentioned in that post can be analyzed by Friendly-traceback with an appropriate explanation provided. Note that these are not hard-coded examples from that post, so that any similar cases should be correctly identified.

Friendly-traceback works with Python 3.6, 3.7 and 3.8.  As I included support for 3.8, I found that some error messages given by Python changed in this newer version, and were generally improved. However, this meant that I had to change a few things to support all three versions.

Working on Friendly-traceback, and on AvantPy, has been so far a fun learning experience for me. I was hoping and looking forward to submit a talk proposal dealing with both these project to the Pycon Education Summit, as I thought that both projects would be of interest to Python educators. However, the call for proposals is focused on people's experience with actual teaching case studies about how teachers and Python programmers have implemented Python instruction in their schools, communities, and other places of learning ... So, definitely no interest in talks about tools like those I create. I certainly do understand the reason for this choice, but I cannot help but feeling disappointed as I was definitely hoping to get an opportunity to give a talk on these projects, and exchange ideas with interested people afterwards.

I did submit a proposal for a reasonably advanced and more technical talk dealing with import hooks and exception hooks, to share what I have learned (while working on Friendly-traceback and AvantPy) with the Pycon crowd. The last time I gave a talk at Pycon was in 2009 and the "competition" to have a talk accepted was much less than what it is now.  Giving a talk is the only way that I can justify taking a leave from my day job to attend Pycon, something I really miss.

Back to Real Python ... I remember purchasing some books from them some time in 2014, and, more recently, I did the same for the "course" on virtual environments. I had never bothered with virtual environments until recently and thought that, if I actually paid to get some proper tutorial, I would have no excuse not to start using virtual environments properly.  The "course" that I bought was well put together.  Compared to standard books, I find it a bit overpriced for the amount of material included. 

As a pure Python hobbyist, I appreciate the material Real Python make freely available, but do find their membership price rather steep.  However, I did note that their tutorial writers could get free access to their entire collection ... 

;-) Perhaps I should offer to write tutorials on 1) using import hooks; 2) using exception hooks; 3) designing libraries with support for translations in a way that they "play well together" -- all topics I had to figure out on my own.  While there are tutorials about translation support, I found that all of them give the same gettext-based approach of defining a global function named "_" which works very well for isolated packages, but can fail spectacularly in some corner cases as I found out while developing Friendly-traceback and AvantPy. 

However, writing clear tutorials takes a lot of time and effort, and is not as fun to me as writing code. So, I think that, for now, I'll just go back to add support for more Python exceptions in Friendly-traceback - and hope that I will have soon to focus my entire free time in putting together material for a Pycon talk.

Abolishing SyntaxError: invalid syntax ...

... and other cryptic messages.

Do you remember when you first started programming (possibly with Python) and encountered an error message that completely baffled you? For some reason, perhaps because you were required to complete a formal course or because you were naturally persistent, you didn't let such messages discourage you entirely and you persevered. And now, whenever you see such cryptic error messages, you can almost immediately decipher them and figure out what causes them and fix the problem.

Congratulations, you are part of an elite group! Even a large number of people who claim that they can program are almost certainly less capable than you are.

Given your good fortune, would you mind donating 5 to 10 minutes of your time to help countless beginners that are struggling in trying to understand Python error messages?  All you need to do is:

1. Glance through of exceptions on this page and use your experience to find a case not covered. Note that this excludes SyntaxError cases, some of which are included here, but would require more of your time.
2. Fire up your favourite Python REPL and write some simple code that generates an exception not already covered.  Perhaps, something like ValueError: could not convert string to float: 'a'
3. Create a new issue with the error message as the title, including the code that generated the exception in the description of the issue together with a simple explanation (in a couple of sentences) of what the error message means.  Imagine that you are writing an explanation for the twelve year old child of your best friend who has expressed some interest in learning how to program.  This simple explanation is the most important part ... however, do not worry about getting it absolutely perfect as it will likely be improved upon based on feedback from future "real beginners".
4. Go back to whatever you were doing before, knowing that the few minutes you have invested will cumulatively save many hours to future generation of programmers that encounter the exception you wrote about.

It should go without saying that contributions that require more time and effort that what is described above are also very welcome!  If you feel particularly ambitious, you can certainly improve the existing code that currently analyses cases of SyntaxError: invalid syntax, which currently handles only a few cases, and should be seen more as a prototype/proof-of-concept.

Future plans for friendly-traceback

Friendly-traceback is being written so that it could be easily incorporated into editors or IDEs that are designed for beginners. I intend to ensure that it can be easily added to Python's IDLE, as well as Mu and Thonny. I hasten to add that Thonny already includes an excellent tool (its "Assistant") which provides amazing feedback to beginners in some cases. Thonny's assistant uses a complementary approach to that of Friendly-traceback and it is quite likely that a future version of Friendly-traceback will include and expand upon the type of analysis performed by Thonny's assistant to help beginners. However, for the moment, the current development of Friendly-traceback is focused on breadth of coverage (i.e. increasing the number of exceptions included), providing a single most-likely explanation for each exception, rather than looking at multiple possible causes for a given exception as is done by Thonny's Assistant.

Avant-IDLE: an experiment

[Edit: this post has an embedded video, which is not visible for PlanetPython readers.]



This is a follow-up from a previous post where I showed just a screenshot made after one hacking session.  A week later, much has changed.

As noted at the end of the video, the code is not (yet) available in a public repository.  Among other things, I need to figure out what license I can use and if I can reuse all of the content from Python's version, or if I need to remove certain assets, etc.

Avant-IDLE makes use of two projects that I started recently and mentioned before on this blog:

Friendly-traceback (code, documentation) and AvantPy (code, documentation). AvantPy itself depends on Friendly-traceback.

Both Friendly-traceback and AvantPy welcome contributions.  Publishing this video about AvantPy is a way to bring more attention to these other two projects.

Going forward with Avant-IDLE

As for Avant-IDLE, if I can go forward with it:

• I would like to translate its menus.
• I would like to add syntax coloring for each individual dialect: this would likely mean some significant change from the way that syntax coloring is done in IDLE.
• I would like to have friendly tracebacks make use of color, like iPython does.
• I would like to add line numbers to the file editor.
• etc.

I might have to remove the debugger, as it makes little sense to have someone entering code in their native language and suddenly have to deal with standard Python since that is the code that is truly executed.  I think that, by the time someone is ready to use a debugger, they are definitely ready to use Python itself.

In an ideal world, save for the "dialect converter" demonstrated in the video, I would like for the rest of Avant-IDLE to be in a single window, with files in tabs and the interpreter below, like Al Sweigart described:

However, that last idea might be something simply too ambitious. After all, Al Sweigart, who is far more competent and productive than I am, never managed to get that project off the ground. The more the code base from Avant-IDLE diverges from that of IDLE itself, the more difficult it will become to incorporate code changes made by Python's core developers to improve IDLE.

Making the code public

Even if I figure out all I need to do to make Avant-IDLE public, I am somewhat hesitant in doing so.

There is no doubt in my mind that I can continue working and improving both Friendly-traceback and AvantPy.  I truly believe that Friendly-traceback could be very helpful for Python beginners.  It could be integrated in other beginner-friendly editors, like Mu or Thonny - two fantastic projects.

I think that AvantPy could be useful for beginners as well ... but I do admit that it is a bit quirky.

When it comes to Avant-IDLE itself, as the saying goes, I don't want to bite more than I can chew, and find out that I created something which I cannot maintain. I do realize my limitations: I am just a hobbyist with no formal training in programming and who likes to do quirky experiments in his spare time. Publishing code on Github and/or making it available at Pypi automatically raises people's expectations, and demands on one's time.

Still, if this video can inspire you to create something useful for beginners, it will have been worthwhile.  Even more so if it inspires you to contribute to either Friendly-traceback or AvantPy, ;-)

Comments?

If you have any constructive criticism, or suggestions to offer, please feel free to do so, either on this blog, or by email.

Test - please ignore

Simple test to embed a gist.


That's all.

Idle hacking - part 1

1. File test_py.pyfr.  The pyfr extension indicates that this is written in the "French Python" dialect. Since this is the main file we'll run, it will also automatically set the language to French. This is all done by AvantPy.
2. "afficher" is the French version of "print"
3. "import" is the normal Python keyword; we can mix and match keywords from a given dialect with the normal Python ones.  Here, we are importing a module whose base name is "upper".
4. The constant "a" is defined.
5. File "upper.pyupper"; the only file whose base name is "upper" - thus, the one that will be imported. The "pyupper" extension indicates that the dialect is an "upper case" version of Python - designed for testing.
6. The code that will be run when the file is imported.
7. Output from both files.
8. "a" is defined.
9. We made a syntax mistake. However, Friendly-traceback is there to help us understand what went wrong ... picking up the default language to use (French) from that first file that was run.

All this was run with a crudely hacked version of Idle. I'm still confused with the communications done with the RPCServer and the TPCServer, having never looked at similar code before.  So, I haven't been able to make it do all that I wanted in this first hacking session. Hopefully, there will be more to come ...

Friendlier tracebacks

When beginners run programs that generate Python tracebacks, they are almost always confused by what the information shown and have no clue as to what this all means. More experienced programmers can sometimes extract enough information directly from tracebacks to figure out what what wrong, but they will often have to resort to inserting a few print calls and running their program again to truly figure out what went wrong and how to fix it. (A minority of programmers might eschew using print calls and use a debugger instead.)

In order to make tracebacks more useful for them, some advanced programmers have designed tools to add more information so that simply looking at the enhanced traceback might be sufficient to diagnose properly the problem.  These tools include better_exchook, infi.traceback, rich-traceback, stackprinter, as well as the beautiful better-exceptions, and many others including Python's own cgitb module.  While there is no doubt that the additional information provided by these tools is useful for advanced programmers, it would likely only add to their confusion if it were used by beginners.

Friendly-traceback, a project I have been working on, and mentioned briefly in an addendum of my last post, aims at improving the situation for beginners.

A quick example

Have a look at the following Python traceback that might be generated by code written by a beginner.

Compare this with the following when using Friendly-traceback's own REPL

The only thing that is shown in exactly the same way is the line showing the exception name an a message. Everything else can be made available in other languages as shown below with a French translation.

Currently, Friendly-traceback can offer a more helpful message than a normal Python traceback in approximately 50 different cases, most of which are cases of either TypeError or SyntaxError. My goal is to eventually include explanations for all standard Python exceptions, and include as many sub-cases as possible.

As shown above, it is possible to translate the information in any language. Currently, only English and French are included; inclusion of other languages will require the help of volunteers.

If you write programs with your own custom exceptions, it is possible to write them so that they could be interpreted correctly by Friendly-traceback.  I have done so in another project, AvantPy, which I mentioned previously here. AvantPy includes 10 custom exceptions.

Both AvantPy and Friendly-traceback are available from Pypi. Contributors are most definitely welcome.

AvantPy needs you

Update: Since this blog post was published, and before Pycoder's weekly mentioned it bringing in quite a few visitors, I had decided to carve out the friendlier tracebacks into a separate project. If you are interested in this idea, you can have a look at the design document and feel free to comment.  Work on AvantPy itself has been paused until friendly-traceback's API is completely implemented.  The API itself is easy to do - the tedious work is adding in an explanation for every possible Python exception, and translating each of them ...

Imagine that you are either learning Python or helping someone that is learning Python. It is almost a certainty that you will see some Python tracebacks, often much more complicated to decipher than

Imagine if you could see something like the following instead:

Or, if you speak French:

Friendlier tracebacks, translated into various languages, is only one of many things that AvantPy can do to help beginners learn programming.

AvantPy is very much in its infancy. You can think of it as a proof-of-concept that can be built upon.

PSF's request for proposal

Yesterday, on the PSF's blog, some information about a request for proposal has been posted.
I have jotted down some thoughts about AvantPy in this context. If you wish to add your own thoughts on this, you can do so here. I do not wish to get any direct funding from the PSF for this project. However, some potential collaborators might need some financial support depending on their goals.

What is AvantPy

Here are some useful links if you want to find out more about AvantPy.

• AvantPy documentation; this should probably be your starting point
• Code repository
• AvantPy on pypi.org (not currently tested as I work from my local version.)

Contributors to the AvantPy project are very much welcome.

User feedback is essential for improvements

A while ago, in Reeborg's World, I implemented a way to require Reeborg to follow a specific path in order to complete a task. For a world creator, this meant to include a path as a list of successive grid location, something like

path = [[1,1], [2, 1], [2,2], ...]

With the appropriate information given, the task shown to the user includes a visual view of the path to follow.  This is what it looked like:

This works well. However, if Reeborg has to retrace some steps, to accomplish the task, the two arrow heads visually combine and appear to form an X which could be interpreted to mean that a given path segment should not be included.

(In addition to the arrow heads combining to look like an X, the dashes do not overlap and instead combine to form a solid line.) Most users of Reeborg's World are students learning in a formal setting. I surmise that those teachers quickly figured out what the correct information was and never reported it.  As I created this visual information, I knew its meaning and was simply blind to the other possibility.

A while ago, a user learning on their own asked me why their program was not working. After a few email exchanges, I finally understood the source of the confusion. I took note of it. I had a quick stab at finding a better way but it didn't work.

A couple of days ago, a second user learning on their own contacted me with the same problem. Clearly, something had to be done ...

This is what it now looks like

In addition to clearing the confusion (or, I hope it does), I actually think it looks much nicer. This improvement would not have been possible if I didn't get some user feedback. This is why I am always thankful when someone contact me to suggest some improvements -- even though I may not always be in a position to implement the required changes quickly.

Pythonic switch statement

Playing with experimental and an old recipe created by Brian Beck.

The content of a test file:

from __experimental__ import switch_statement

def example(n):
    result = ''
    switch n:
        case 2:
            result += '2 is even and '
        case 3, 5, 7:
            result += f'{n} is prime'
            break
        case 0: pass
        case 1:
            pass
        case 4, 6, 8, 9:
            result = f'{n} is not prime'
            break
        default:
            result = f'{n} is not a single digit integer'
    return result

for i in range(11):
    print(example(i))

Trying it out

$ python -m experimental test_switch
0 is not prime
1 is not prime
2 is even and 2 is prime
3 is prime
4 is not prime
5 is prime
6 is not prime
7 is prime
8 is not prime
9 is not prime
10 is not a single digit integer

Just having fun ... Please, do not even think of using this for serious work.

Javascript tools for Python hobbyists

I am just a hobbyist, Python enthusiast who has been, over the course of many years, writing what is now a relatively big Javascript program (close to 20,000 lines of code so far). If you like Python and the Pythonic way of programming, and find yourself writing more JavaScript code than you'd like for a fun side-project meant as a hobby, you may find some merit in the approach I use.  I wish I could have read something like this blog post when I started my project or even just a few years ago, when I did a major rewrite, and started using some of the tools described in this post.

If you are a professional programmer, you can just stop reading as you know much more than I do, and you surely have a better, more efficient and cutting edge way of doing things right now - and you will likely use yet a different way next year, if not next month. So, you would likely find my advice to look the same as my site: dated and not using the latest and coolest techniques - in short, for you, not worth looking at. ;-)

Summary:

• If you must, choose a well-supported Javascript library and stick with it.
• Use npm for installing Javascript tools
• Avoid depending on third-party packages whenever possible
• Use npm to manage your workflow
• Supplement with your own Python or shell/batch script when needed
• Use browserify to concatenate all your Javascript files
• Use tape for unit testing
• Use faucet for formatting of unit test results
• Use QUnit for integration testing
• Optional: use Madge for identifying circular dependencies
• Optional: use Dependo to identify any overlooked module
• Optional: use JSDoc for creating an API
• Use jshint instead of jslint

Warning

This blog post is long. I've attempted to provide enough details for you to determine in each case if my use-case corresponds to yours and thus if and when my recommendation might make sense for you and your project.

The context

I started working on Reeborg's World many years ago. The first version was created as a desktop program (rur-ple) in 2004. My first primitive attempt at a web version was done around 2007.  During the years I have worked on it, tools and libraries have come, evolved, and gone, to be replaced by better ones.  As programming is only a hobby for me which I work on when I have some free time, I cannot afford to change the set of tools I use every year to follow the latest trend.

I've started working on the current version when using color gradients for buttons and menu bars was the latest and coolest thing - well before the current flat UI became the norm.

Admittedly, my site looks dated - but since I do not have enough time to add all the new ideas for functional improvements I want to make, investing time to modernize the look is not a priority.

The Javascript code I wrote is split over many files and has become a tangled mess - in spite of some occasional attempts at reorganizing the code, including a near-complete rewrite a few years ago.

Some of the complexity is required as I want to make it easier for would-be collaborator to add new programming language or paradigms for learners [1] or additional human language support [2]. However, it is likely that some of this tangled mess could be simplified with a significant effort.

In addition, there is more to Reeborg's World than a single site; there is also a basic programming tutorial available in three languages [3] with additional languages in the works, a Teacher's Guide [4], an API documentation for advanced features [5], and more [6].  Each of these act almost like an independent project pulling me in different directions.

In order to preserve my sanity, as my project slowly evolves I need some constancy and simplicity in the tools I use.

Using a well-supported library

Unlike the situation with Python, which comes "batteries included", there is no standard library for Javascript. Using a library means choosing between various alternatives, and communities.

When I started this project, the main problem facing people writing Javascript code was browser incompatibilities.  There was one obvious solution: use jQuery.  Nowadays, it is most likely no longer needed for that purpose, but that was not the case back then.

I also knew that I wanted the ability to have floating windows for additional menus and dialogs. After examining a few choices, I settled on jQuery UI, since there was good documentation for it and an active community ... and I was already using jQuery which meant a smaller footprint than some other alternatives.

Libraries like jQuery and jQuery UI can be included with a link to a CDN (Content delivery network) which can reduce the load on the server where my project lives. I can also link to a specific version of these libraries, which means that I do not have to update code that depend on them (except if security issues are discovered).

10 years later, both libraries are still alive and well and I haven't needed to make any significant changes to any code that uses them.

Use npm for installing Javascript tools

npm is described as both a package manager for Javascript and as the world's largest software respository. I use it to install various Javascript tools I use (like tape, browserify, jsdoc, etc. which I describe below). 

I do not use it to install javascript libraries (big or small) called by my own code. From what I can tell, the "best/most common" practice in the Javascript world is to make use of tons of modules found on the npm repository, some of which are simply one line of code.  Requiring a single module can mean in reality that the project can depend on dozens of other modules, none of them being vetted - unlike the Python standard library. Upgrade to a single modules can result in a bug affecting hundreds of other modules ...  For example, one developer broke Node, Babel and thousands of projects in 11 lines of JavaScript.

When I resume working on my project after months of inactivity, I never have to worry about how any change to any such third party module could require updating my code.  (Yes, there are most likely ways to mitigate such problems, but I prefer to avoid them in the first place.)

There are alternatives to npm (such as yarn, and others), but, from what I can tell, they do not offer any advantages when it comes to installing Javascript tools - a task that is performed very rarely for a given project.

Use npm to manage your workflow

When reading about Javascript, I most often saw either gulp or grunt mentioned mentioned as tools to automate tasks. From what I read, it seems that were essential to do any serious Javascript development.  Each of them came had its own way to do things ... and it was not easy for me to see which would be the best fit.  In the various posts I read about gulp vs grunt, npm was never mentioned as an alternative.

However, as I learned more about npm, I found that, together with a very simple batch file it could do all the automation that I needed in a very, very simple way, by defining "scripts" in a file named package.json.  Chaining tasks with npm scripts is a simple matter of "piping" them (with the | character).  Since I had already installed npm, it became an easy choice.

Use browserify to concatenate all your Javascript files

Once my Javascript code became much too long to fit into a single file, I broke it up into various files. With Python, I would have use an import statement in individual files to take care of dependencies. With Javascript, the only method that I knew of at the beginning of my project (10 years ago) was to add individual links in my html file. As the number of Javascript files increased, it became difficult to ensure that files were inserted in the proper order to ensure that dependencies were taken care of ... In fact, it soon became almost impossible. 

This required a major rewrite.  Fortunately, when I had to do this, some standardized way of ensuring dependencies had emerged.  The simplest was to use something like

 require("module_a.js");

at the top of, say module_b.js, and use some tools to concatenate the javascript files, ensuring that proper dependencies were taken care of.  The simplest tool I found for this purpose is browserify, originally created, as far as I can tell, by James Halliday.

browserify can be installed using npm.

Use tape for unit testing

Sigh ... I find testing boring ... But, as my project grew larger, it became necessary to write some tests.

When I did a search on testing tools/framework for Javascript, I most often saw mentions of Chai, Jasmine, Mocha QUnit and Sinon.  A recent search yields a few more potential candidates like Cucumber, Karma, etc.    

The Javascript world seems to really, really like so-called Behaviour Driven Development, where writing tests can mean writing code like:

tea.should.have.property('flavors').with.lengthOf(3);

If.I.wanted.to.write.code.that.read.like.English.I.would.likely.use.Cobol.

It is only by accident that I came accross tape as a testing framework that felt "right" to me. I like my tests to look like my code. With Python, I would use assert statements to ensure that the a function produces the correct result.  My favourite unit testing framework for Python is, not surprisingly, pytest.

From what I have seen, tape is the closest Javascript testing framework to pytest.  Here's an actual example where I test some code which is expected to raise/throw an exception/error:

test('add_wall: invalid orientation', function (assert) {

    assert.plan(2);

    try {

        RUR.add_wall("n", 1, 2, true);

    } catch (e) {

        assert.ok(e.reeborg_shouts, "reeborg_shouts");

        assert.equal(e.name, "ReeborgError", "error name ok");

    }

    assert.end();

});

I make use of "assert.plan()" to ensure that the number of assertions tested matches my expectations.

It was only after I had used tape for a while that I found out that it was also written by James Halliday.

tape can be installed using npm.

Use faucet for formatting of unit test results

Tape's output is in the TAP format (Test Anything Protocol) which, by default, is extremely verbose. Most often, it is recommended to pipe the results into formatters which produce more readable results. 

Depending on what I am doing, I use different formatters, some more verbose than others. After trying out about a dozen formatters, I now use faucet by default.  faucet can be installed using npm and has been written by, ... you guessed it, James Halliday.

Use QUnit for integration testing

Unit tests are fine, but they miss problems arising from putting all the code together.  I used different strategies to do integration testing, all of which seem to create almost more problems than they solved, until I stumbled upon a very easy way that just works for me.  Using a Python script, I take the single html file for my site, put all the code inside an html div with display set to none, insert some qunit code and my own tests, and let everything run.

Optional: use Madge for identifying circular dependencies

To help identify potential problems with circular dependencies, I use madge, which can be installed with npm.

There is one remaining dependency in my code, which I silence by not inserting a require() call in one of my modules: when the site is initialized, I want to draw a default version of the world which I by calling functions in the drawing module when loading some images. Later, when calling the drawing module, I do need the definitions found in the module where I load the images.  I could get rid of the dependencies at the cost of duplicating some code ... but since the initializing of the site and the execution of user-entered code are done in separate phases, the circular dependency does not cause any problems.

Optional: use Dependo to identify any overlooked module

The image of the tangled mess of modules shown above was created using dependo. As I was refactoring code and adding various require() statement, dependo was helpful in identifying any module not included, either because they had been accidently forgotten or because they had become irrelevant.  dependo can also be installed using npm.

Optional: use JSDoc for creating an API

While I do not particularly like it, as I cannot figure out how to extend it to address my particular needs, I found that jsdoc useful to produce an API for people wanting to use advanced features in creating unusual programming tasks (aka "worlds").  When I started using it, there did not seem to be any easy way to use Sphinx to create such API. I gather that this might no longer be the case ... but it would likely require too much effort to make the change at this point.

jsdoc can also be installed using npm.

Use jshint instead of jslint

A linter can often be useful in identifying potential or real problems with some code. When I started working on this project, the only linter I knew was jslint. jshint is friendlier and more configurable to use, and is my preferred choice. And, you guessed it, jshint can be installed using npm.

Last thoughts

There might very well be other tools that would be better for your own projects but, if you love Python and find yourself not overly enthusiastic at the thought of adopting the Javascript way when working on a project that requires Javascript, you might find that the tools I use match more closely the way you do things with Python.  Or not.



[1] Currently, programs can be written in Python, Javascript, using blockly, or in Python using a REPL.

[2] Language support can mean one of two things: either the programming library for users (like using "avance()" in French as equivalent to "move()" in English, or for the UI, or both. Currently, French and English are implemented for both, while Korean and Polish are only available for UI. Work is underway to provide Chinese support for both.

[3] The tutorial can be found here; you can change the default language using the side-bar on the right. The repository is at https://github.com/aroberge/reeborg-docs. The tutorial is currently available in French, English and Korean, with additional languages in the works.

[4] https://github.com/aroberge/reeborg-howto is a site aimed at creators of advanced tasks for Reeborg's World. It has very little content currently but will have more to be migrated from https://github.com/aroberge/reeborg-world-creation which was written as an online book (a format which I found to be unsatisfactory.)

[5] https://github.com/aroberge/reeborg-api is a documentation site for the API that creators of advanced tasks can use. 

[6] 

• https://github.com/aroberge/reeborg-staging is a staging version used for testing or for those that want absolute privacy as it is available as a secure site https://aroberge.github.io/reeborg-staging/
• https://github.com/aroberge/reeborg-dist is for those interested in installing their own version of Reeborg's World by simply unzipping a file, instead of cloning a repository with unnecessary files.