Xmas present from Thonny

Today, a new version (3.2.5) of Thonny has been released. It incorporates support for Friendly-traceback (which needs to be installed separately). Currently, the download link on Thonny's homepage still links to version 3.2.4. The latest version can be found on Github.

Thonny is a fantastic IDE for beginners, especially those learning in a classroom environment, as it offers many useful tools that can be used effectively by teachers to demonstrate some programming concepts.  Thonny is the work of Aivar Annamaa, who is apparently recognized as an excellent lecturer -- which does not suprise me given the thoughtful design of Thonny. He has been interviewed about Thonny on PythonPodcast.

Real Python has a fairly comprehensive review here.

Friendly Mu

Link --> See the documentation for details.

A Tiny Python Exception Oddity

Today, while working on Friendly-traceback (improved documentation !) as I have been doing a lot recently, I came into an odd SyntaxError case:

• The inconsistent behaviour is so tiny, that I doubt most people would notice - including myself before working on Friendly-traceback.
• This is SyntaxError that is not picked up by flake8; however, pylint does pick it up.
• By Python, I mean CPython.  After trying to figure out why this case was different, I downloaded Pypy and saw that Pypy did not show the odd behaviour.
• To understand the origin of this different behaviour, one needs to look at some obscure inner parts of the CPython interpreter.
• This would likely going to be found totally irrelevant by 99.999% of Python programmers. If you are not the type of person who is annoyed by tiny oddities, you probably do not want to read any further.

You have been warned.

Normal behaviour

When Python finds a SyntaxError, it flags its location.  Let's have a look at a simple case, using CPython 3.7.

Notice how it indicates where it found the error, as shown by the red arrow: this happened when it reached a token that was inconsistent with the code entered so far. According to my experience until today, this seemed to be always the case.  Note that using CPython 3.6 yields exactly the same behaviour, and unhelpful error message.

Before discussing the case with a different behaviour, let's make a detour and look at Pypy's handling of the same case.

Same location indicated, but a much more helpful error message, even though this is version 3.6.  This improved error message was discussed in this Pypy blog post.  I strongly suspect that this is what lead to this improved error message in CPython 3.8.

Same error message as Pypy ... but the exact location of the error, previously indicated by ^, no longer appears - which could be unfortunate when nested parenthesis (including square and curly brackets) are present.

What about Friendly-traceback you ask? I thought you never would! ;-)  

Well, here's the information when using CPython 3.7.

The line about not having enough information from Python refers to the unhelpful message ("invalid syntax"). Hopefully you will agree that the information given by Friendly-traceback would be generally more useful, and especially more so for beginners.   

But enough about this case. It is time to look at the odd behaviour one.

Odd case

Consider the following:

Having a variable declared both as a global and nonlocal variable is not allowed.  Let see what happens when this is executed by Pypy.

So, pypy processed the file passed the nonlocal statement and flagged the location where it encountered a statement which was inconsistent with everything that had been read so far: it thus flagged that as the location of the error.

Now, what happens with CPython:

The location flagged is one line earlier. The nonlocal statement is flagged as problematic but, reading the code up to that point, there is no indication that a global statement was encountered before.

Note that, changing the order of the two statements does not change the result: pypy shows the beginning of the second statement (line 6) as the problem, whereas CPython always shows the line before.

Why does it matter to me?

If you go back to the first case I discussed, with the unmatched parenthesis, in Friendly-traceback, I rely on the location of the error shown by Python to indicate where the problem arose and, when appropriate, I look *back* to also show where the potential problem started.  Unfortunately, I cannot do that in this case with CPython.

Why is this case handled differently by CPython?

While I have some general idea of how the CPython interpreter works, I absolutely do not understand well enough to claim with absolute certainty how this situation arise.  Please, feel free to leave a comment to correct the description below if it is incorrect.

 My understanding is the following:

After breaking down a file into tokens, parsing it according to the rules of the Python grammar, an abstract syntax tree (AST) is constructed if no syntax error is found.  The nonlocal/global problem noted is not picked up by CPython up to that point - which also explains why flake8 would not find it as it relies on the AST, and does not actually executes the code.  (I'm a bit curious as to how Pylint does ... I'll probably have to look into it when I have more time).

Using the AST, a control flow graph is created and various "frames" are created with links (GOTOs, under a different name...) joining different parts.  It is at that point that relationships between variables in different frames is examined in details.  Pictorially, this can be represented as follows:

(This image was taken from this blog post by Eli Bendersky)  In terms of the actual code, it is in the CPython symtable.c file. At that point, errors are not found by scanning lines of code linearly, but rather by visiting nodes in the AST in some deterministic fashion ... which leads to the oddity mentioned previously: CPython consistently shows the first of two statements as the source of the problem, whereas Pypy (which relies on some other method) shows the second, which is consistent with the way it shows the location of all SyntaxError messages.

Conclusion

For Friendly-traceback, this likely means that for such cases, and unlike the mismatched parenthesis case, I will not attempt to figure out which two lines are problematic, and will simply expand slightly on the terse one liner given by Python (and in a way that can be translated into languages other than English).

pydeps: a very useful program

A few weeks ago, I was doing some refactoring of Friendly-traceback and had some minor difficulty in avoiding the creation of circular imports.  For some reason (age perhaps), I could not visualize the file structure properly.  Enter pydeps.  After I used it to generate a graph for all the files internal to Friendly-traceback, I was able to use that graph to figure out a better way to structure my program.

Today, as I stared at that graph, after including it in the newly styled documentation, I noticed that the "version" file I had created early on, was really redundant since its content (a single variable) could easily be incorporated in the Public API file.

So, one less file to deal with!

I think I am going to use pydeps a lot more from now on when I want to try to understand the how projects are structured, as I do find this type of graph very useful.

Significant changes for some error messages in Python 3.8

As I work on including more exceptions in Friendly-traceback, I am mostly pleasantly surprised by generally more precise error messages.  For example, in Python 3.7, the following

__debug__ = 1

would yield "SyntaxError: assignment to keyword" which likely would baffle almost everyone looking up the list of Python keywords.   In Python 3.8, that message has been replaced by the more precise: "SyntaxError: cannot assign to __debug__". Much better, in my opinion, even though one may be surprised to learn about this constant.

However, today as I was working on adding another case, I came accross the following:

This change is ... unexpected.  And the "helpful hint", is not so helpful in this case.  However, I can guess as to how it came about.  It will be a challenge to provide a "friendly" explanation that does not lead the users looking for an incorrect solution to their problem.

Edit: Based on my own (limited) experience working on Friendly-traceback, I do realize that trying to provide helpful hints to assist programmers in fixing code that raises exceptions is a very difficult problem.  My task, with Friendly-traceback, is generally made much easier thanks to accurate and usually helpful error messages provided by Python. So, please do not read this post and conclude that I am dismissive of the efforts of the Python core developers in this area. 

Friendly-traceback, Real Python, Pycon, and more

After an interruption that lasted a few months, I've finally been able to return to programming, more specifically working mostly on Friendly-traceback. For those that do not know Friendly-traceback: it aims to replace the sometimes obscure traceback generated by Python with something easier to understand. Furthermore, Friendly-traceback is designed with support for languages other than English so that, in theory, beginners (who are the main target audience for Friendly-traceback) could benefit no matter what their native language is ... provided someone would have done the translation into that language, of course.

As of now, 75 different cases have been tested; you can find them in the documentation.  [If you have suggestions for improvements, please do not hesitate to let me know.]

Recently, a post by Real Python on SyntaxError has given me added impetus to work on Friendly-traceback. I'm happy to report that, other than the cases mentioned dealing with misspelled or missing keywords, all of the other examples mentioned in that post can be analyzed by Friendly-traceback with an appropriate explanation provided. Note that these are not hard-coded examples from that post, so that any similar cases should be correctly identified.

Friendly-traceback works with Python 3.6, 3.7 and 3.8.  As I included support for 3.8, I found that some error messages given by Python changed in this newer version, and were generally improved. However, this meant that I had to change a few things to support all three versions.

Working on Friendly-traceback, and on AvantPy, has been so far a fun learning experience for me. I was hoping and looking forward to submit a talk proposal dealing with both these project to the Pycon Education Summit, as I thought that both projects would be of interest to Python educators. However, the call for proposals is focused on people's experience with actual teaching case studies about how teachers and Python programmers have implemented Python instruction in their schools, communities, and other places of learning ... So, definitely no interest in talks about tools like those I create. I certainly do understand the reason for this choice, but I cannot help but feeling disappointed as I was definitely hoping to get an opportunity to give a talk on these projects, and exchange ideas with interested people afterwards.

I did submit a proposal for a reasonably advanced and more technical talk dealing with import hooks and exception hooks, to share what I have learned (while working on Friendly-traceback and AvantPy) with the Pycon crowd. The last time I gave a talk at Pycon was in 2009 and the "competition" to have a talk accepted was much less than what it is now.  Giving a talk is the only way that I can justify taking a leave from my day job to attend Pycon, something I really miss.

Back to Real Python ... I remember purchasing some books from them some time in 2014, and, more recently, I did the same for the "course" on virtual environments. I had never bothered with virtual environments until recently and thought that, if I actually paid to get some proper tutorial, I would have no excuse not to start using virtual environments properly.  The "course" that I bought was well put together.  Compared to standard books, I find it a bit overpriced for the amount of material included. 

As a pure Python hobbyist, I appreciate the material Real Python make freely available, but do find their membership price rather steep.  However, I did note that their tutorial writers could get free access to their entire collection ... 

;-) Perhaps I should offer to write tutorials on 1) using import hooks; 2) using exception hooks; 3) designing libraries with support for translations in a way that they "play well together" -- all topics I had to figure out on my own.  While there are tutorials about translation support, I found that all of them give the same gettext-based approach of defining a global function named "_" which works very well for isolated packages, but can fail spectacularly in some corner cases as I found out while developing Friendly-traceback and AvantPy. 

However, writing clear tutorials takes a lot of time and effort, and is not as fun to me as writing code. So, I think that, for now, I'll just go back to add support for more Python exceptions in Friendly-traceback - and hope that I will have soon to focus my entire free time in putting together material for a Pycon talk.