Monday, March 02, 2020

True constants in Python - part 2, and a challenge

Like the title says, this is the second post on this topic. If you have not done so, you should really read the first one, which is much shorter, before continuing to read here.

The first clue


The first clue is that, rather than executing test.py as the main module, I imported it. Can you think of a situation where this would make a difference?

I'll leave a bit of space below to give you the opportunity to possibly go back to part 1, without reading about the solution below.















PEP 302


Back in 2002, with the adoption of PEP 302, Python enabled programmers to modify what happens when a module is imported; this is known as an import hook.  For example, it is possible to modify the source code in a module prior to it being executed by Python. This is NOT what I have done here - but I have done this for other examples that I will refer to below.

If the only thing required would be to modify the source, one could use what is described in PEP 263 and define a custom encoding that would transform the source. In this case, by adding an encoding declaration, it would have been possible to run test.py directly rather than executing it.  I thought of it a while ago but, in order to cover all possible cases, one would pretty much have to write a complete parser for Python that could be used to identify and replace the various assignments statement done by print statements so as to show what you saw in part 1.  However, this still would not be enough to protect against the reassignement done externally, like I did with

test.UPPERCASE = 1

The actual solution I used required three separate steps. The challenge I will mention at the end is to reduce this to two steps - something that I think is quite possible but that I have not been able to do yet - and to remove one left-over "cheat" which would allow one to redefine a constant by monkeypatching.  I think that this is possible but I have not actually sat down to actually do it. I thought of waiting for a few days to give an added incentive for anyone who would like to try and get the bragging rights of having it done first! ;-)

Step 1

Step 1 and 2 involve an import hook. They are independent one of another and can be done in any order.

When importing a module, Python roughly does the following:


  1. Find the source code
  2. Create a module object
  3. Execute the source code in the module object's dict.
The module object created by Python comes with a dict that is, in some sense, "read-only": you cannot write code to modify its behaviour, nor replace it by a custom dict. (However, see the challenge.)  However, one can define a custom dict, which is designed so that its various methods (__setitem__, __delitem__, etc.) prevent the reassignment of variables we intend to be constants. In the example I have chosen, these are variables whose names are in UPPERCASE.  (Not shown in the example of part 1: I have also added a scan of the code to identify any variable that used the type hint Final and add them automatically to the list of variables intended to be constants.)

Instead of executing the code in the module object's dict, it is executed in this special dict. The content of that dict is then copied into the module object's dict.

Doing this ensures that code run directly in the module is guaranteed to prevent variable reassignement. At least, I have not found a way to cheat from within a module and change the value of variables intended to be a constant.

Step 2

Step 2 is to define a custom class that prevent changes of attributes. This custom class is used to replace the module's own class, something that can be done.

Step 3

Step 3 is to make Python use our import hook. To do so, we must have some code being executed earlier than what is shown. There are a couple of ways to do this as describe in the Site-specific configuration hook section of the Python documentation. The method I have chosen is one that is easily done on an ad-hoc basis.  I created a file named usercustomize.py whose content is the following:

from ideas.examples import constants
constants.add_hook()

This calls my code that sets up an import hook as described above. To have Python execute this code, I set the environment variable PYTHONPATH to be equal to the directory where usercustomize is located. On Windows (which is what I use), this is most easily achieve by navigating to that directory in the terminal and entering the following:

set PYTHONPATH=%CD%

Doing so will ensure that the code in usercustomize.py is executed before any user code.

The challenge


As mentioned in part 1, attempting to modify the value of a constant from outside, as in:

This leaves one possible cheat. From an external module, instead of writing

import test
test.UPPERCASE = "new value"

which is prevented, one can use the following cheat

import test
test.__dict__["UPPERCASE"] = "new value"

This is because the module's __dict__ is a "normal" Python dict. 

However, instead of using a module object created by Python, it should be possible to create a custom module object that uses something like the special dict mentioned before. Thus one would not need to change the way that Python execute code in the module's dict.

The challenge is to write code that creates such a module object.   I would not be surprised if there remained some other ways to cheat after doing so, but hopefully none as obvious as the one shown above.

Resources


The code I have written is part of my project named ideas.  The actual code for the constants example is given by this link.  See also the documentation for the project.  Note that, token_utils mentioned in the documentation has been put in a separate project; I need to update the documentation.

Both ideas and token-utils can be installed from Pypi.org as usual.




True constants in Python - part 1

tl;dr: I'm always trying to see if what everyone "knows to be true" is really true...

In many programming languages, you can define constants via some special declaration. For example, in Java you can apparently write something like:

public static final String CONST_NAME = "Name";

and this will result in a value that cannot be changed.  I wrote "apparently" since I do not program in Java and rely on what other people write.

Everyone "knows" that you cannot do the same in Python.  If you want to define constants, according to Python's PEP 8, what you should do is the following
Constants are usually defined on a module level and written in all capital letters with underscores separating words. Examples include MAX_OVERFLOW and TOTAL.
and rely on the fact that everyone will respect this convention.  However, nothing prevents you from redefining the value of these variables later in the same module, or from outside (monkeypatching) when importing the module.

Thus, if I write in a module

TOTAL = 1
# some code
TOTAL = 2

the value of the variable will have changed.

If you are willing to use optional type declaration and either use Python 3.8 with the typing module, or some earlier version of Python but using also the third-party typing-extension package, you can use something like the following:

from typing import Final
TOTAL: Final = 0 

and use a tool like mypy that will check to see if the value of is changed anywhere, reporting if it does so.  However, if you do run such an incorrect program (according to mypy), it will still execute properly, and the value of the "constant" will indeed change.

For people that want something a bit more robust, it is often recommended to use some special object (that could live in a separate module) whose attributes cannot change once assigned. However, this does not prevent one from deleting the value of the "constant object" (either by mistake within the module, or by monkeypatching) and reassign it.

Every Python programmer knows that the situation as described above is the final word on the possibility of creating constants in Python ... or is it?

For example, here's a screen capture of an actual module (called test.py)


Notice how the linter in my editor has flagged an apparent error (using UPPERCASE after deleting it.)  And here's the result of importing this module, and then attempting to change the value of the constant.


Can you think of how I might have done this?  (No photoshop, only the normal Python interpreter used.)

In part 2  , I explain how I have done this and will leave you with a (small) challenge.

Friday, February 28, 2020

Implicit multiplication in Python - part 1

Quoting from a post from Guido van Rossum

    ... The power of visual processing really becomes apparent when you combine
    multiple operators. For example, consider the distributive law
       
mul(n, add(x, y)) == add(mul(n, x), mul(n, y))  (5)
    That was painful to write, and I believe that at first you won't see the
    pattern (or at least you wouldn't have immediately seen it if I hadn't
    mentioned this was the distributive law).
    Compare to
        n * (x + y) == n * x + n * y    (5a)
    Notice how this also uses relative operator priorities. Often
    mathematicians write this even more compact
        n(x+y) == nx + ny    (5b)
    but alas, that currently goes beyond the capacities of Python's parser.
    ...
    Now, programming isn't exactly the same activity as math, but we all know
    that Readability Counts, and this is where operator overloading in Python
    comes in.  ...
What if we could do something half-way between what Python currently allow
and what mathematicians would write by transforming something that is currently a SyntaxError into valid Python code?


    >>> from ideas.examples import implicit_multiplication as mul
    >>> hook = mul.add_hook()
    >>> from ideas import console
    >>> console.start()
    Configuration values for the console:
        callback_params: {'show_original': False, 
                          'show_transformed': False}
        transform_source from ideas.examples.implicit_multiplication
    --------------------------------------------------
    Ideas Console version 0.0.7a. [Python version: 3.7.3]

    ~>> 2(3 + 4)
    14
    ~>> a = 3
    ~>> b = 4
    ~>> 2a
    6
    ~>> a b
    12

All that is needed is to change the way the code is tokenized before the code is parsed by Python.

Monday, February 24, 2020

From a rejected Pycon talk to a new project.

Like many others, my talk proposal (early draft here) for Pycon US was rejected. So, I decided to spend some time putting everything in a new project instead. (Documentation here.)  It is still a rough draft, but usable ... and since I've mentioned it in a few other places, I thought I should mention it here as well.




Wednesday, December 25, 2019

Xmas present from Thonny

Today, a new version (3.2.5) of Thonny has been released. It incorporates support for Friendly-traceback (which needs to be installed separately). Currently, the download link on Thonny's homepage still links to version 3.2.4. The latest version can be found on Github.

Thonny is a fantastic IDE for beginners, especially those learning in a classroom environment, as it offers many useful tools that can be used effectively by teachers to demonstrate some programming concepts.  Thonny is the work of Aivar Annamaa, who is apparently recognized as an excellent lecturer -- which does not suprise me given the thoughtful design of Thonny. He has been interviewed about Thonny on PythonPodcast.

Real Python has a fairly comprehensive review here.

Saturday, December 14, 2019

A Tiny Python Exception Oddity

Today, while working on Friendly-traceback (improved documentation !) as I have been doing a lot recently, I came into an odd SyntaxError case:

  • The inconsistent behaviour is so tiny, that I doubt most people would notice - including myself before working on Friendly-traceback.
  • This is SyntaxError that is not picked up by flake8; however, pylint does pick it up.
  • By Python, I mean CPython.  After trying to figure out why this case was different, I downloaded Pypy and saw that Pypy did not show the odd behaviour.
  • To understand the origin of this different behaviour, one needs to look at some obscure inner parts of the CPython interpreter.
  • This would likely going to be found totally irrelevant by 99.999% of Python programmers. If you are not the type of person who is annoyed by tiny oddities, you probably do not want to read any further.
You have been warned.

Normal behaviour


When Python finds a SyntaxError, it flags its location.  Let's have a look at a simple case, using CPython 3.7.

Notice how it indicates where it found the error, as shown by the red arrow: this happened when it reached a token that was inconsistent with the code entered so far. According to my experience until today, this seemed to be always the case.  Note that using CPython 3.6 yields exactly the same behaviour, and unhelpful error message.

Before discussing the case with a different behaviour, let's make a detour and look at Pypy's handling of the same case.

Same location indicated, but a much more helpful error message, even though this is version 3.6.  This improved error message was discussed in this Pypy blog post.  I strongly suspect that this is what lead to this improved error message in CPython 3.8.

Same error message as Pypy ... but the exact location of the error, previously indicated by ^, no longer appears - which could be unfortunate when nested parenthesis (including square and curly brackets) are present.

What about Friendly-traceback you ask? I thought you never would! ;-)  

Well, here's the information when using CPython 3.7.


The line about not having enough information from Python refers to the unhelpful message ("invalid syntax"). Hopefully you will agree that the information given by Friendly-traceback would be generally more useful, and especially more so for beginners.   

But enough about this case. It is time to look at the odd behaviour one.

Odd case


Consider the following:

Having a variable declared both as a global and nonlocal variable is not allowed.  Let see what happens when this is executed by Pypy.


So, pypy processed the file passed the nonlocal statement and flagged the location where it encountered a statement which was inconsistent with everything that had been read so far: it thus flagged that as the location of the error.

Now, what happens with CPython:


The location flagged is one line earlier. The nonlocal statement is flagged as problematic but, reading the code up to that point, there is no indication that a global statement was encountered before.

Note that, changing the order of the two statements does not change the result: pypy shows the beginning of the second statement (line 6) as the problem, whereas CPython always shows the line before.

Why does it matter to me?

If you go back to the first case I discussed, with the unmatched parenthesis, in Friendly-traceback, I rely on the location of the error shown by Python to indicate where the problem arose and, when appropriate, I look *back* to also show where the potential problem started.  Unfortunately, I cannot do that in this case with CPython.

Why is this case handled differently by CPython?

While I have some general idea of how the CPython interpreter works, I absolutely do not understand well enough to claim with absolute certainty how this situation arise.  Please, feel free to leave a comment to correct the description below if it is incorrect.

 My understanding is the following:

After breaking down a file into tokens, parsing it according to the rules of the Python grammar, an abstract syntax tree (AST) is constructed if no syntax error is found.  The nonlocal/global problem noted is not picked up by CPython up to that point - which also explains why flake8 would not find it as it relies on the AST, and does not actually executes the code.  (I'm a bit curious as to how Pylint does ... I'll probably have to look into it when I have more time).

Using the AST, a control flow graph is created and various "frames" are created with links (GOTOs, under a different name...) joining different parts.  It is at that point that relationships between variables in different frames is examined in details.  Pictorially, this can be represented as follows:


(This image was taken from this blog post by Eli Bendersky)  In terms of the actual code, it is in the CPython symtable.c file. At that point, errors are not found by scanning lines of code linearly, but rather by visiting nodes in the AST in some deterministic fashion ... which leads to the oddity mentioned previously: CPython consistently shows the first of two statements as the source of the problem, whereas Pypy (which relies on some other method) shows the second, which is consistent with the way it shows the location of all SyntaxError messages.

Conclusion

For Friendly-traceback, this likely means that for such cases, and unlike the mismatched parenthesis case, I will not attempt to figure out which two lines are problematic, and will simply expand slightly on the terse one liner given by Python (and in a way that can be translated into languages other than English).