New milestone for friendly: version 0.5

 Friendly (previously at 0.4.41) and friendly-traceback (previously at 0.4.111) are now at version 0.5. The joint documentation for both projects has not yet been updated.  In addition to the many new cases added for which friendly/friendly-traceback can help with, which includes close to 400 test cases, I am very excited to report to three new important features

1. Getting help when a traceback is generated before friendly is imported
2. Not having to set non-default configurations each time friendly is used
3. The addition of two new languages.

1. Getting help after the fact

Let's start with the first.  Previously, if one wanted help from friendly/friendly-traceback, it had either to be used to run a program, via something like "python -m friendly user_program.py", or it had to be imported and installed (either implicitly or explicitly) before any other code was executed. This still works as before and is the best way to use friendly.

Now, it can be imported *after* a traceback has been generated, and can provide its usual help when using:

• IPython in a terminal
• Jupyter notebooks, and Jupyter lab
• Mu
• Programs run with cPython using "python -i user_program.py"
• Code entered in a cPython terminal, with the caveat that it only works in a limited fashion for some SyntaxErrors but almost never for run time errors.
• The same when using pypy with the exception that using languages other than English may yield some undesirable results.
• Code saved in files and run from IDLE  (Python 3.10 and possibly later versions of Python 3.9) -- but excluding SyntaxErrors
• Code entered in IDLE's shell - but excluding SyntaxErrors.

Before explaining the origin of the (different) limitations when using cPython's interactive interpreter or IDLE, let me show the results using IPython, both for SyntaxErrors and run time errors starting with a very unlikely example of SyntaxError

Of course, we can ask for more details

Instead of a SyntaxError, let's see an example of a run time error.

Again, it just works. :-)

Moving on to SyntaxErrors with the cPython interpreter. Let's use the same example as above, with Python 3.10.1:

This works. However, let's have a more detailed look at the information available:

Python does not store the content of the code entered in the interpreter; for SyntaxErrors, it does include the very last line of code where the error was located. This will not work in other situations where a statement spans multiple lines; in some cases, if the error message is precise enough, friendly might still be able to guess the cause of the error.

By contrast, friendly does store the entire code entered in its interpreter.

Let's have a look at a run time error with cPython.

Notice how the traceback contains no information about the code in the file(s) named "<stdin>".

Let's see what information we can get from friendly.

If you use friendly, you would never see the log message (1) as it is something that is enabled by default on my computer. Note that, in spite of not having access to the exact code that produced the exception, in this case friendly is still able to provide some help. This information is similar to what is available with Python 3.10+; however, you can use friendly with Python 3.6 and still get the same information!

Of course, it is better still if you use friendly from the start:

Let's now have a look at IDLE. Recently, IDLE has added support for custom exception hook. Instead of requiring the use of its own console when using IDLE, friendly can make use of this new capability of IDLE to provide help with run time errors - but not SyntaxErrors as those are handled in a peculiar way by IDLE.

For this type of error, trying to use friendly after the fact yields very little useful information.

For SyntaxErrors, the situation is even worse: IDLE does not make any information available.

Note that this is a bug which I did report.

1.a) A possible improvement to fix the problem with cPython

If you look at the tracebacks from IDLE for runtime errors, you will see "files" with names like 

"<pyshell#4>": each code block entered by the user is saved in such a "file", each file having a different name.  IDLE works around a limitation of Python's linecache module to store the content of these files so that they can be retrieved and analyzed.  By contrast, cPython shows the code entered by a user as belonging to files with identical names "<stdin>" whose content can never be retrieved.

For code executed using exec("code"), the content is shown to belong to a file named "<string>" whose content is also not available.  If cPython were to store the code in files whose named included a different integer each time, like IDLE does, then it could be retrieved by programs like friendly and provide additional help.  This was suggested on Python-ideas for code run using exec, but got not traction, even though related questions are often asked on StackOverflow.

Moving on ...

2. Friendly saves settings

Previously, each time that friendly was used, it started with default values for the preferred language (French and English only in previous versions), color scheme (light or dark, depending on the background of the terminal or other application), formatter type, etc.

Now, friendly saves the values specified which are then used by default when a new session starts. For the language choice, this is a global settings, that carries in all environments.  For other settings, friendly (at least on Windows) can determine if it is running in a PowerShell terminal or an old-fashion cmd, in a Visual Studio Code terminal, in a PyCharm terminal, if it is run with IPython, or in a Jupyter notebook, etc.   Here's an example of adjusting the background color so that the information provided by friendly blends in better.

Friendly only includes two different color scheme: one that is designed to work with a white (or similar) background and another with a black (or similar) background.  Anyone working with terminals (or notebooks) with background colors that do not work well with either of the two existing color schemes is welcome to provide different color schemes to be added to friendly.

So far, I have only tested this with Windows. Mac and Linux users are encouraged to try it out and see if their different environments can be detected correctly so that friendly can work well in all the environment they use it.

3. New languages

In addition to English and French, friendly is available in Spanish (approximately 99% of the translation is done, as I keep adding new information) and about 10% has been translated into Italian.

Conclusion

There is much more I could write about new but smaller additions to friendly since version 0.4.  However, this blog post is already too long and this will have to wait until later - perhaps after I update the existing documentation.

Friendly-traceback en español

 Friendly and Friendly-traceback are now partially available in Spanish thanks to the work of Mrtín René (https://github.com/martinvilu).

You can have a look at the Spanish translations in context for SyntaxErrors and for other exceptions.

If you are interested in contributing to translations, please join this discussion and have a look at this online collaborative site.

Update: Someone just volunteered to help with the Italian translation. Note that there are more than 600 pieces of text to translate and that more volunteers can help!

Forgot to set up a custom exception hook? Perhaps it is not too late.

There are many custom exception hook that seek to improve upon the standard one from Python. The very first one is Python's cgitb module, but there are many others including, in no particular order, better_exchook, infi.traceback, the traceback submodule of Rich, the different but similarly named rich-traceback, IPython's ultratb module, better-exceptions, stackprinter, pretty-errors, tbvaccine, stack_data, and likely many more others, including my own friendly/friendly-traceback.

To use any of them, they have to be either installed as a custom sys.excepthook before an exception occurred or explicitly written in an except block to process the information.  They normally get the relevant information from sys.exc_info().

What happens if we try to access this information after a normal traceback has be printed?

Too late: no information is available.

At least, that's what I thought until very recently.  Let's see what we can do using the same problematic code in IPython (just to be different) and using friendly after the fact.

As it turns out, the individual content of the tuple obtained from sys.exc_info() before a traceback is printed are available as individual items: sys.last_type, sys.last_value, and sys.last_traceback. As explaine in sys:

These three variables are not always defined; they are set when an exception is not handled and the interpreter prints an error message and a stack traceback. 

...

The meaning of the variables is the same as that of the return values from exc_info() above.

Friendly now uses this knowledge: when friendly is imported, it now checks to see if an exception has occurred previously resulting in a printed traceback. If so, it attempts to make use of the information available to help users. This has now been tested with IPython, Jupyter, Colab, Mu, etc. It also works with recent versions of IDLE [1] and the normal CPython interpreter [2].

I have not checked ... but I'd be curious to hear if any other "traceback enhancer" can be used in this way. If you know of any, please let me know.

[1] Excluding SyntaxErrors for now; see this bug report.

[2] With limited success since the information about the code entered is temporarily saved in a file named "<stdin>" whose content can never be retrieved.

New interactive friendly tracebacks for Jupyter

 Summary: I have a draft version of a new and arguably better way to display information from friendly for Jupyter labs/notebooks. For now, it only works with the default light theme.

The following screen captures illustrate a new way to display the information from friendly inside Jupyter lab/notebooks.  Instead of having to type some commands, the user can click on buttons.  Initially only the error message is shown.

Clicking on "More ..." reveals the friendly traceback (notice how the traditional file names are replaced by references to code blocks) and a few more buttons.

One can click on individual buttons to show or hide the desired information.

Using the default formatter, instead of this new one, works somewhat acceptably with the dark theme.

Most unfortunately, the new interactive display just does not pick up the dark theme appropriately; I have not been able to figure out why.

Ideally, I would like to be able to use the custom dark theme that I have created to work with Rich, but have not figured out how to do it yet.

Anyone having some insights as to how to fix these problems should not hesitate to comment here or on the Github issue.

Friendly-traceback 0.4 and PyConAu

 Just a quick update ...

After over 150 commits, most of which included new traceback cases explained by Friendly-traceback, it is time to go from version 0.3 to 0.4.  There are too many changes to mention in this blog post and the documentation needs to be updated.  

The version change is also to allow me to switch gear and focus on preparing a talk for the next PyConAu. I'm hoping that this can be done reasonably quickly (say, within a week) so that I can have the time to write a proper blog post about all the new features of Friendly-traceback.

Friendlier tracebacks in REPLs (including Jupyter)

Traceback:  Determination of origin; the process of tracing something back to its source.

I have been working towards release 0.4 of friendly/friendly_traceback as well as preparing for a talk at the upcoming PyconAU. Those familiar with PyconAU might be interested to note that I plan to include an example related to the flipfloperator.  :-)

In this preparatory work, I have been revisiting almost all aspects of friendly/friendly-traceback, making various improvements. Unfortunately, this means that many of the screenshots included in the documentation will have to be updated.

In this blog post, I want to illustrate some of the changes that I have finished implementing with regards to the traceback themselves. By this, I mean the sequence of calls that were done and not the message, such as "IndexError: list index out of range".  I will look at the same simple example using a variety of interpreters. Since this is a simple example, the traceback is not going to be very long ... but I want you to imagine situations where it could be much longer.

First, let me introduce this example using the CPython REPL:

We see a reference to a "file" named "<stdin>" which appears many times. While we have some line numbers indicated, CPython does not tell us anything about the content of these lines.  Thus, it could be difficult to trace back the source of the error.

The situation is slightly better with IDLE, as it keeps track of different "<pyshell> files" and show us the relevant lines of code.

Up until recently, this was the strategy I was using by default with the friendly console. Here we see it running within IDLE.

For longer traceback, the user might have difficulty identifying the block of code corresponding to each "<friendly-console> file".  Fortunately, friendly includes the function where() which can provide some help in this situation.

For perhaps a slightly better approach, we can look at what IPython does.

Each code block is identified by a number in brackets, [1], [2], etc..  The traceback includes a weird name of the form "<ipython-input-N-...>".  By looking closely at enough examples, we can conclude that the "N" correspond to the digit between the square brackets in a code block.

Moving on to Jupyter (lab). Up until fairly recently, Jupyter lab (and notebooks), which are based on IPython, were using "filenames" of the form "<ipython-input-N-...>".  However, this seems to have been changed recently, and we now have "real" filenames ... with names that do not contain any clue about their origin. [I filed an issue suggesting an improvement.]

However, it is possible to do better.  Here's how friendly now shows this information within Jupyter lab.

Instead of a "File" we see mention of "Code block" which might be less confusing, especially to beginners that are starting with Jupyter and never dealt with Python code in separate files before.

Of course, this being friendly, we can show much more than simply the traceback.  Here's the complete information available:

To properly format tracebacks with "Code block" instead of "File", I had to monkey patch pygment's lexer.

Since I thought that such numbered code blocks were more informative than fake filenames like "<friendly-console>", I decided to adopt them by default in friendly's REPL:

This new REPL style is also available in friendly.idle (not shown in this post).

Finally, here's an example showing a mixture of code from files and code blocks.

If you have any suggestion for possible improvements to friendly/friendly-traceback, please do not hesitate to let me know by filing an issue.  I am especially interested to hearing from teachers/mentors that work with beginners.

.  

friendly-traceback is back!

 A few months ago, I mentioned that friendly-traceback had been renamed friendly. As friendly/friendly-traceback evolved, the number of dependencies increased. For some third-party projects that use friendly/friendly-traceback, such as futurecoder and HackInScience, many of these dependencies are simply redundant and represent a performance hit.  For this reason, I have split friendly/friendly-traceback into two separate projects, and created a new "organization" on Github [1] with separate repositories:  one for friendly-traceback, and a separate one for friendly (which has friendly-traceback as a dependency). I even temporarily added a third project (friendly_idle) but folded it back into friendly as the extra burden of maintaining a relatively small project with lots of duplication was too much work for very little benefit.

Much progress has been done with these projects since the last minor release (0.3) and I am in the process of doing a major update to the documentation prior to the 0.4 release; friendly-traceback is currently at version 0.3.154.

If you are a user of friendly/friendly-traceback, you likely need to do "pip install friendly".  If you want to use the information provided by friendly/friendly-traceback into your own project, you likely only need to do "pip install friendly-back".

[1] This change meant going back to zero stars on Github! 

Friendly version 0.3 has been released

Friendly version 0.3 has been released. This version also marks the official name change from the former friendly-traceback.

Before I started working on Friendly, I assumed that to do custom exception handling, one simply had to redefine sys.excepthook. However, I since found out that this does not work in many environments, including all those based on IPython (which include Jupyter notebooks). Even Python's IDLE, at least up until Python version 3.10.0a5, does not work with a simple replacement of sys.excepthook.

Additionally, creating coloured output varies depending on the programming environment. Thankfully, this can be done in most environments simply by using Rich -- sometimes supplemented by a bit of additional code.

As a result, friendly includes various custom modules so that it can work with the following:

• IDLE (see this and the following 2 pages)
• Mu (see four pages starting here)
• Jupyter notebooks and Jupyter Lab (which also work with Google colab)
• IPython console
• VS Code (either running from the terminal or with embedded notebooks)

Friendly has now over 200 test cases. Most of these are not unit tests, but instead are integration tests: given code X that raises exception Y (including message Z), ensure that the output is as expected.  For example, suppose that after module M is imported, we try to use attribute A containing a typo. Such a base scenario might need 2 test cases to over all possibilities: whether we can find an attribute that is a similar word, likely indicating a typo, or not.  Here's an example of what one can get in the first scenario:

>>> import math
>>> a = math.sine(1)

Traceback (most recent call last):
File "<friendly-console:2>", line 1, in <module>
a = math.sine(1)
AttributeError: module 'math' has no attribute 'sine'

Did you mean one of the following: `sin, sinh, asin`?

Friendly included a "hint" (Did you mean one of the...). Further information can be obtained by using what(), why(), where(), etc., as described in previous posts.

[Note to self: for the above example, the hint should only include the most likely candidate, namely 'sin', leaving the other choices for the longer explanation done with "why()", as was done for other type of exceptions.]

Currently, about half of the test cases are examples of SyntaxError. I suspect that this will continue to be the case as work on friendly continues and the total number of test cases continues to grow. I would not be surprised if the total number of test cases were to double by the time version 1.0 is ready.  My goal for version 1.0 is to try to cover all possible exception cases that can occur when making mistakes while trying to use code from Python's standard library.

Going back in history

Imagine that you wish to run a program that takes a long time to run. Just in case somethings goes wrong, you decide to use friendly-traceback (soon to be renamed...) in interactive mode to run it.  This turns out to be a good decision:

Time to explore what might be the problem, and where exactly things might have gone wrong.

Ooops ... a silly typo. Easy enough to correct:

Unfortunately, that did not work: Friendly-traceback, like all Python programs designed to handle exceptions, only capture the last one.

This has happened to me so many times; granted, it was always with short programs so that I could easily recreate the original exception. However, I can only imagine how frustrating it might be for beginners encountering this situation.

A solution

Fortunately, you are using the latest version of Friendly-traceback, the one that records exceptions that were captured, and allows you to discard the last one recorded (rinse, and repeat as often as needed), thus going back in history.

Now that we are set, we can explore further to determine what might have gone wrong.

Friendly-traceback will have a new name

tl; dr: I plan to change the name from friendly_traceback to friendly.

Friendly-traceback: testing with Real Python

Real Python is an excellent learning resource for beginning and intermediate Python programmers that want to learn more about various Python related topics. Most of the resources of RealPython are behind a paywall, but there are many articles available for free. One of the free articles, Invalid Syntax in Python: Common Reasons for SyntaxError, is a good overview of possible causes of syntax errors when using Python. The Real Python article shows code raising exceptions due to syntax errors and provides some explanation for each case.

In this blog post, I reproduce the cases covered in the Real Python article and show the information provided by Friendly-traceback. Ideally, you should read this blog post side by side with the Real Python article, as I mostly focus on showing screen captures, with very little added explanation or background.

If you want to follow along using Friendly-traceback, make sure that you use version 0.2.34 or newer.

Missing comma: first example from the article

The article starts off showing some code leading to this rather terse and uninformative traceback.

Since the code is found in a file, we use python -m friendly_traceback theofficefacts.py to run it and obtain the following.

Misusing the Assignment Operator (=)

We only show one example here, as the others mentioned in the article would be redundant. We remind you for one last time that, if you are not doing so, you should really look at the Real Python article at the same time as you go through this rather terse blog post.

Friendly traceback provides a "hint" right after the traceback. We can get more information by asking why().

Misspelling, Missing, or Misusing Python Keywords

Identifying misspelled keywords was actually inspired by that article from Real Python.

Note that Friendly-traceback identifies "for" as being the most likely misspelled keyword, but gives other possible valid choices.

Friendly-traceback can also identify when break (and return) are used outside a loop.

To the English reader, Friendly-traceback might seem to add very little useful information. However, keep in mind that all this additional information can be translated. If you read the following and do not understand what "boucle" means, then you might get an idea of the some of the challenges faced by non-English speakers when using Python.

In some other cases, like the example given in the Real Python article, Friendly-traceback can identify a missing keyword.

As long as there is only one instance of "in" missing, Friendly-traceback can identify it properly.

Finally, two more cases where a Python keyword is not used properly.

Missing Parentheses, Brackets, and Quotes

Five examples taken from the Real Python article offered without additional comments.

Mistaking Dictionary Syntax

Using the Wrong Indentation

Real Python gives many examples. They would all be handled correctly by Friendly-traceback in a similar way as the single example we decided to use for this post.

Defining and Calling Functions

Changing Python Versions

Friendly-traceback requires Python version 3.6 or newer. Not shown here is that it can recognize that the walrus operator, :=, is not valid before Python version 3.8 and give an appropriate message.

Last example: TypeError result of a syntax error.

Let's look at the last example in the Real Python article.

The explanation given by Friendly-traceback might seem weird "the object (1, 2) was meant to be a function ...".  Often one might have assigned a name to that object, which leads to an explanation that should be seen as more reasonable.

The explanation of looking for a "missing comma" when this TypeError is raised was actually added following a suggestion by S. de Menten in the recent contest I held for Friendly-traceback.

There is more ...

Friendly-traceback includes many more cases that those shown above and mentioned in the Real Python article. However, it is limited in that it can only identify the cause of syntax errors there is a single word or symbol used incorrectly or if the error message provided by Python is more informative than the dreaded SyntaxError: invalid syntax.