After a long hiatus, I have gone back to work on a new version of my very first project - essentially a clone of Karel the Robot - this time delivered entirely via the web (not yet available though). Like rur-ple, and Guido van Robot, and many others I am sure, the user will have the option of executing all the code without interruptions, or stepping through, one instruction at a time. Since Javascript does not have a "sleep()" function, the way I implemented the program was to create a series of frames (like a movie) which can be then played back at a given frame rate, or stepped through one frame at a time, etc. This makes is easy to include the possibility of stepping back from a given frame instead of being restricted to only moving forward. However, since I use the browser to evaluate the code (rather than creating my own interpreter), I do not have the information regarding the code line whose execution corresponds to a given frame.
I know that Greg Wilson, Brad Miller, and many others I am sure, have expressed opinions about this being a desirable feature in a teaching environment. However, I don't want to implement things that are not needed (and will needlessly complicate the user interface) and, given the non-availability of the line of code which corresponds to a given frame, I am not sure that a really good case can be made for the possibility of going backwards. (Note that I have implemented a "pause" instruction which would enable a user to effectively set a breakpoint and, knowing where they inserted this instruction in the code, be able to follow step-by-step
the result.) The fact that *I* do not have a good use case for it right now does not mean that such a good use case exist. So, dear reader, can you think of a situation where such a feature would be useful? (in this type of programming environment.)
Friday, February 08, 2013
Thursday, March 15, 2012
Python for iOS: first look
Jonathan Hosmer has put together a nice app Python for iOS, bundling a Python interpreter, a basic editor with output window, the documentation for Python 2.7 and links to a discussion forum. While it is a bit limited, due to some Apple policies regarding importing/exporting files, it is a nice tool to have on an iPad when a laptop is not available and one wants to play with Python - even in the absence of connectivity. At $3, it is quite affordable. He has even bundled sympy with it and, I believe, plan on including numpy as well. If you have an iPad, and would like to be able to fire up a Python interpreter anywhere, I recommend it highly.
Friday, December 09, 2011
Porting to Python 3: What are you waiting for?
Lately, there seems to be a lot of posts trying to encourage people to port their apps/libraries/modules to Python 3. I'd like to add my voice and perhaps, use as a reminder this old post: more than 2 years ago, Crunchy was made compatible with Python 2.4, 2.5, 2.6 ... and 3.1. At the time, we made the decision to not follow what seem to be the official recommendation, namely to have one code base based on 2.x and use the 2to3 tool to do the automatic translation. Instead, we decided to have a single code base, which seems the way people are doing it these days. There were of course some challenges, especially as we kept compatibility all the way back to Python 2.4: since Crunchy puts a Python interpreter inside your browser and manipulates what's there, regardless of the encoding, it has to be able to do a lot of string (and bytes for 3.x) manipulations, as well as dealing with incompatible syntax for handling exceptions, etc. when running the code that is fed to it. This was done when there was very little known about best practices for porting from 2.x to 3. Partly as a result, Crunchy's code is not the best example to look at when it comes to doing such a port ... but it certainly demonstrated that it was possible to port a non-trivial program with a tiny team. Now, the situation is quite different, and many more projects have been ported, and you can benefit from their experience.
So, what are you waiting for?
So, what are you waiting for?
Thursday, September 22, 2011
Errors should never pass silently
- The Zen of Python
Lately, I have been programming a lot in Javascript, and have come to appreciate even more what Tim Peters expressed so well in writing what is now known as The Zen of Python. In particular, I appreciate the philosophy that "Errors should never pass silently."
Those that don't care about Javascript can stop reading now. Others may benefit, and perhaps even enlighten me even further, about a "feature" I came across.
I'm working on a new website which will include only interactive tutorials. [No, not Crunchy-based ones, but honest-to-goodness interactive tutorials that require nothing but a browser.] To make my life easier, I use jQuery as well as CodeMirror. A user can edit the code in the CodeMirror editor and view the result "live" in a separate window (html iframe). Every time the content of the editor is changed, the iframe gets updated, as illustrated here.
I started by using the standard $ jQuery notation inside the html iframe without including a link to jQuery in the iframe, since I already included it in the parent page, and got an error in the console to the effect that $ was not defined. Fair enough. I changed the source to also include a link to jQuery inside the content of the editor (to be copied in the iframe) and the error vanished. However, no javascript code was run from the iframe, not even a simple test alert. Meanwhile, the javascript console remained silent.
Errors should never pass silently.
Eventually, by searching on the web, I was able to find a solution. However, before I found a solution, I encountered a truly puzzling "feature".
After including a link to jQuery in the editor (which then was copied into the iframe) and reloading the page, if I then proceeded to remove that link from the editor, suddenly all javascript code embedded in the iframe, including all jQuery calls, would work.
I can just imagine giving these instructions to beginners:
In any event, to solve my problem, I had to do 3 things:
Now everything appears to work just as expected.
Lately, I have been programming a lot in Javascript, and have come to appreciate even more what Tim Peters expressed so well in writing what is now known as The Zen of Python. In particular, I appreciate the philosophy that "Errors should never pass silently."
Those that don't care about Javascript can stop reading now. Others may benefit, and perhaps even enlighten me even further, about a "feature" I came across.
I'm working on a new website which will include only interactive tutorials. [No, not Crunchy-based ones, but honest-to-goodness interactive tutorials that require nothing but a browser.] To make my life easier, I use jQuery as well as CodeMirror. A user can edit the code in the CodeMirror editor and view the result "live" in a separate window (html iframe). Every time the content of the editor is changed, the iframe gets updated, as illustrated here.
I started by using the standard $ jQuery notation inside the html iframe without including a link to jQuery in the iframe, since I already included it in the parent page, and got an error in the console to the effect that $ was not defined. Fair enough. I changed the source to also include a link to jQuery inside the content of the editor (to be copied in the iframe) and the error vanished. However, no javascript code was run from the iframe, not even a simple test alert. Meanwhile, the javascript console remained silent.
Errors should never pass silently.
Eventually, by searching on the web, I was able to find a solution. However, before I found a solution, I encountered a truly puzzling "feature".
After including a link to jQuery in the editor (which then was copied into the iframe) and reloading the page, if I then proceeded to remove that link from the editor, suddenly all javascript code embedded in the iframe, including all jQuery calls, would work.
I can just imagine giving these instructions to beginners:
We are going to use jQuery, as you can see from the code in the editor. Now, to begin, you have to select the line where jQuery is loaded and delete it so that the browser (iframe) can use it.I still have not figured out how that phantom jQuery worked... and strongly believe that some error message or warning should have appeared in the javascript console (for Chrome, or Firebug for Firefox).
In any event, to solve my problem, I had to do 3 things:
- Only load jQuery once, in the main window.
- Write $ = parent.$; inside the iframe before using the $ notation.
- As I wanted to draw things on a canvas, replace the usual $("#canvas") jQuery idiom by $("#canvas", document) everywhere.
Now everything appears to work just as expected.
Sunday, January 16, 2011
Book review: Pro Python System Administration
Summary: Pro Python System Administration is a comprehensive book showing how Python can be used effectively to perform a variety of system administration tasks. I would recommend it highly to anyone having to do system administration work. For more information, please consult the author's web site.
===
There is a saying that "no good deed goes unpunished". I feel that a counterpart should be "no bad talk goes unrewarded". At Pycon 2009, I gave a talk on plugins that has to be amongst the worst presentations I ever gave. Yet, as an unexpected result from that talk, I received a free copy of Pro Python System Administration written by Rytis Sileika. This blog entry is a review of that book.
This book is written for system administrators, something in which I have no experience; therefore, this review will definitely not have the depth that an expert may have given it.
Four general areas of system administrations are covered: network management, web server and web application management, database system management, and system monitoring. Examples are given on a Linux system with no indication as to whether or not a given example is applicable to other operating systems. Given that Python works with all major operating systems, and that the book focuses on using Python packages, I suspect that the content would be easily adaptable to other environments.
While the book is classified, on the cover, as being addressed to advanced Python programmers, the author in the book introduction indirectly suggests that this book would be appropriate for people that have some minimal experience with Python. I suspect that the classification on the book cover was done (wrongly) by the editor as I found the examples very readable and I would not claim to be an advanced Python programmer.
The book is divided into 13 chapters, each focused on one or a few well-defined tasks. While the tasks in a given chapter are relatively independent of those of other chapters, there is a natural progression in terms of topics introduced and it is probably better to read the book in the natural sequence rather than reading chapters randomly - in other words, this book is not simply a random collection of recipes for a series of tasks.
1. Reading and Collecting Performance Data Using SNMP
In this chapter, Sileika introduces the Simple Network Management Protocol, or SNMP after which he shows how one can query SNMP devices using Python and the PySNMP library. In the second part of that chapter, he introduces RRDTool, an application for graphing monitoring data, and shows how to interact with it using the rrdtool module. In the last section of this first chapter, he shows how to create web pages (with the eventual goal of displaying on the web monitoring data) using the Jinja2 templating system.
2. Managing Devices Using the SOAP API
In this chapter, Sileika introduces the Simple Object Access Protocol or SOAP and gives examples based on using the Zolera SOAP Infrastructure (ZSI) package. The bulk of the chapter focuses on explaining how to manage and monitor Citrix Netscaler load balancers. The Python logging module is introduced and used.
3. Creating a Web Application for IP Address Accountancy
In this chapter, the Django framework is introduced and used to build a web application that maintains IP addresses allocation on an internal network. Rather than using the web server included with Django, Sileika shows how to use Django with the Apache web server.
4. Integrating the IP Address Application with DHCP
This chapter is a continuation of the previous one, where the application previously developed is enhanced with the addition of Dynamic Host Configuration Protocol (DHCP) services as well as a few others. More advanced Django methods are included as well as some AJAX calls.
5. Maintaining a List of Virtual Hosts in an Apache Configuration File
Another Django based application is introduced, this time with a focus on the administration side.
6. Gathering and Presenting Statistical Data from Apache Log Files
This chapter focuses on building a plugin-based modular framework to analyze log files. The content of this chapter is the reason why I received a free copy of the book in the first place: Sileika mentioned to me in an email that the architecture described was mostly inspired by the presentation I gave at PyCon with a few modifications that allow for information exchange between the plug-in modules. When I got the original email, I was really surprised given that I had tried to forget about the talk I had given on plugins.
In my opinion, Sileika does an excellent job in explaining how plugins can be easily created with Python and used effectively to design modular applications.
Dynamic discovery and loading of plugins is illustrated, and the GeoIP Python library is used to find the physical location corresponding to a given IP address.
7. Performing Complex Searches and Reporting on Application Log Files
This chapter shows how to use Exctractor, an open source log file parser tool, to parse more complex log files than those generated by an Apache server as illustrated in the previous chapter. Of note is the introduction and usage of Python generators as well as an introduction to parsing XML files with Python.
8. A Web Site Availability Check Script for Nagios
This chapter shows how to use Python with Nagios, a network monitoring system. Python packages/modules illustrated include BeautifulSoup, urllib and urllib2. The monitoring scripts developed do more than simply checking for site availability and actually test the web application logic.
9. Management and Monitoring Subsystem
10. Remote Monitoring Agents
11. Statistics Gathering and Reporting
Chapters 9, 10 and 11 explain how to build a "simple" distributed monitoring system. A number of Python module & packages are used including xmlrpclib, SimpleXMLRPCServer, CherryPy, sqlite3, multiprocessing, ConfigParser, subprocess, Numpy, matplotlib and others. The application developed is a good example of using Python as a "glue language", to use third-party modules & packages. One weakness is that the introduction to statistics included is rather elementary and, I believe, could have been shortened considerably given the intended audience.
12. Automatic MySQL Database Performance Tuning
This chapter revisits the use of plugins, with a slightly more advanced application, where information can be exchanged between the different plugins.
13. Using Amazon EC2/S3 as a Data Warehouse Solution
This last chapter gives a crash course on Amazon's "cloud computing" offerings. It is a good final note to the book, and a good starting point for future explorations.
Overall, I found the book quite readable even though it is outside of my area of expertise. Occasionally, I had the feeling that there were a few "fillers" (e.g. overlong log listings, etc.) that could have been shortened without losing anything of real value. This is very much an applied book, with real life examples that could be either used "as-is" or used as starting points for more extended applications.
I would recommend this book highly to anyone who has to perform any one of the system administration tasks mentioned above. It is also a good source of non-trivial examples demonstrating the use of a number of Python modules and packages. The code that is given would likely save many hours of development. As is becoming the norm, the source code included in the book is available from the publisher. Also, many of the prototypes covered in the book are available as open source projects at the author's web site http://www.sysadminpy.com, where a discussion forum is also available.
===
There is a saying that "no good deed goes unpunished". I feel that a counterpart should be "no bad talk goes unrewarded". At Pycon 2009, I gave a talk on plugins that has to be amongst the worst presentations I ever gave. Yet, as an unexpected result from that talk, I received a free copy of Pro Python System Administration written by Rytis Sileika. This blog entry is a review of that book.
This book is written for system administrators, something in which I have no experience; therefore, this review will definitely not have the depth that an expert may have given it.
Four general areas of system administrations are covered: network management, web server and web application management, database system management, and system monitoring. Examples are given on a Linux system with no indication as to whether or not a given example is applicable to other operating systems. Given that Python works with all major operating systems, and that the book focuses on using Python packages, I suspect that the content would be easily adaptable to other environments.
While the book is classified, on the cover, as being addressed to advanced Python programmers, the author in the book introduction indirectly suggests that this book would be appropriate for people that have some minimal experience with Python. I suspect that the classification on the book cover was done (wrongly) by the editor as I found the examples very readable and I would not claim to be an advanced Python programmer.
The book is divided into 13 chapters, each focused on one or a few well-defined tasks. While the tasks in a given chapter are relatively independent of those of other chapters, there is a natural progression in terms of topics introduced and it is probably better to read the book in the natural sequence rather than reading chapters randomly - in other words, this book is not simply a random collection of recipes for a series of tasks.
1. Reading and Collecting Performance Data Using SNMP
In this chapter, Sileika introduces the Simple Network Management Protocol, or SNMP after which he shows how one can query SNMP devices using Python and the PySNMP library. In the second part of that chapter, he introduces RRDTool, an application for graphing monitoring data, and shows how to interact with it using the rrdtool module. In the last section of this first chapter, he shows how to create web pages (with the eventual goal of displaying on the web monitoring data) using the Jinja2 templating system.
2. Managing Devices Using the SOAP API
In this chapter, Sileika introduces the Simple Object Access Protocol or SOAP and gives examples based on using the Zolera SOAP Infrastructure (ZSI) package. The bulk of the chapter focuses on explaining how to manage and monitor Citrix Netscaler load balancers. The Python logging module is introduced and used.
3. Creating a Web Application for IP Address Accountancy
In this chapter, the Django framework is introduced and used to build a web application that maintains IP addresses allocation on an internal network. Rather than using the web server included with Django, Sileika shows how to use Django with the Apache web server.
4. Integrating the IP Address Application with DHCP
This chapter is a continuation of the previous one, where the application previously developed is enhanced with the addition of Dynamic Host Configuration Protocol (DHCP) services as well as a few others. More advanced Django methods are included as well as some AJAX calls.
5. Maintaining a List of Virtual Hosts in an Apache Configuration File
Another Django based application is introduced, this time with a focus on the administration side.
6. Gathering and Presenting Statistical Data from Apache Log Files
This chapter focuses on building a plugin-based modular framework to analyze log files. The content of this chapter is the reason why I received a free copy of the book in the first place: Sileika mentioned to me in an email that the architecture described was mostly inspired by the presentation I gave at PyCon with a few modifications that allow for information exchange between the plug-in modules. When I got the original email, I was really surprised given that I had tried to forget about the talk I had given on plugins.
In my opinion, Sileika does an excellent job in explaining how plugins can be easily created with Python and used effectively to design modular applications.
Dynamic discovery and loading of plugins is illustrated, and the GeoIP Python library is used to find the physical location corresponding to a given IP address.
7. Performing Complex Searches and Reporting on Application Log Files
This chapter shows how to use Exctractor, an open source log file parser tool, to parse more complex log files than those generated by an Apache server as illustrated in the previous chapter. Of note is the introduction and usage of Python generators as well as an introduction to parsing XML files with Python.
8. A Web Site Availability Check Script for Nagios
This chapter shows how to use Python with Nagios, a network monitoring system. Python packages/modules illustrated include BeautifulSoup, urllib and urllib2. The monitoring scripts developed do more than simply checking for site availability and actually test the web application logic.
9. Management and Monitoring Subsystem
10. Remote Monitoring Agents
11. Statistics Gathering and Reporting
Chapters 9, 10 and 11 explain how to build a "simple" distributed monitoring system. A number of Python module & packages are used including xmlrpclib, SimpleXMLRPCServer, CherryPy, sqlite3, multiprocessing, ConfigParser, subprocess, Numpy, matplotlib and others. The application developed is a good example of using Python as a "glue language", to use third-party modules & packages. One weakness is that the introduction to statistics included is rather elementary and, I believe, could have been shortened considerably given the intended audience.
12. Automatic MySQL Database Performance Tuning
This chapter revisits the use of plugins, with a slightly more advanced application, where information can be exchanged between the different plugins.
13. Using Amazon EC2/S3 as a Data Warehouse Solution
This last chapter gives a crash course on Amazon's "cloud computing" offerings. It is a good final note to the book, and a good starting point for future explorations.
Overall, I found the book quite readable even though it is outside of my area of expertise. Occasionally, I had the feeling that there were a few "fillers" (e.g. overlong log listings, etc.) that could have been shortened without losing anything of real value. This is very much an applied book, with real life examples that could be either used "as-is" or used as starting points for more extended applications.
I would recommend this book highly to anyone who has to perform any one of the system administration tasks mentioned above. It is also a good source of non-trivial examples demonstrating the use of a number of Python modules and packages. The code that is given would likely save many hours of development. As is becoming the norm, the source code included in the book is available from the publisher. Also, many of the prototypes covered in the book are available as open source projects at the author's web site http://www.sysadminpy.com, where a discussion forum is also available.
Sunday, August 01, 2010
My favourite Python one-liner
I am slowly working on two new projects involving web-based interactive Python tutorials and had to set up a local web server for handling ajax request. After browsing the web, I stumbled upon the following one-liner in the Python documentation that I had completely forgotten about:
python -m SimpleHTTPServer 8000
Then, simply pointing the browser at "http://localhost:8000" and all is set to go. Granted, it is not what one might consider a traditional "one-liner" but it works for me. Python's batteries included is certainly a nice feature!
python -m SimpleHTTPServer 8000
Then, simply pointing the browser at "http://localhost:8000" and all is set to go. Granted, it is not what one might consider a traditional "one-liner" but it works for me. Python's batteries included is certainly a nice feature!
Sunday, January 17, 2010
Profiling adventures and cython - Faster drawing and conclusion
In the last blog post, I made use of cython to speed up some calculations
involving the iterative equation defining the Mandelbrot set. From
the initial profiling run with 1000 iterations in the second post
to the last run in the previous post, the profiling time went from
575 seconds down to 21 seconds, which is a reduction by a factor of
27. This is nothing to sneer at. Yet, I can do better.
Let's start by having a sample profile run with 1000 iterations with the program as it was last time, but keeping track of more function calls in the profile.
Currently, a "line" is potentially drawn for each pixel. If I look at a given fractal drawing, I can see that it could be drawn using "longer lines", when consecutive pixels are to be drawn with the same colour. I can easily implement this as follows.
So far, the pictures the program has been able to produce have only been in black and white. It is time to spruce things up and add colour. To do this, we will need to make three general changes:
Notice how the Mandelbrot set boundary seems rather smooth ... this is clearly a sign that we might be missing something. Increasing the number of iterations to 1000 reveals a different picture.
We see much more details and many points that were wrongly identified as being part of the Mandelbrot sets are correctly excluded (and colored!). What happens if we look at a region that contains more points inside the Mandelbrot set (thus requiring more iterations) and increase the number of iterations to 1500 (as I found that 1000 was not enough in this region).
Unfortunately, the profiling and the timing information displayed does not tell the entire story. In practice, I found that it would take many more seconds (sometimes more than 10) for the canvas to be updated than the timing information given for each of the three pictures displayed above. Something is happening behind the scene when the picture is updated on the screen and which is not being recorded by my simple timing method.
For comparison, I had a look at Aptus, a program that uses a hand-coded C extension for speed. Actually, I had tried Aptus a few years ago, when Ned Batchelder first mentioned it, and tried it again for fun as I was working on my own version. Aptus can produce really nice pictures, really fast. Here's an example that was produced, according to the timing given by Aptus itself in 0.25 seconds.
Note that the timing given by Aptus seems to be truly representative, unlike the timing for my program. I should mention that, in addition to using a hand-crafted C extension, Aptus uses wxPython instead of Tkinter, and it also uses PIL and numpy, both of which are known for their speed. It might be possible that using PIL and numpy with my program would improve the speed significantly. However, all three libraries are not part of the standard library and so do not meet the constraints I had decided upon at the beginning.
This concludes this profiling experiment ... at least for now. I should emphasize that the goal of these posts was to record a profiling experiment using cython. I do not pretend that this code was especially hand-crafted for speed ... even though I did try some simple changes to improve its speed. I may revisit this application at a later time, especially if someone more experienced can point out ways to increase the speed significantly, preferably while staying within the constraints I set up at the beginning: other than cython, use only modules from the standard library. However, I would be interested if anyone adapts this code to use PIL and/or numpy in a straightforward way and, in doing so, increases the speed significantly.
Let's start by having a sample profile run with 1000 iterations with the program as it was last time, but keeping track of more function calls in the profile.
5441165 function calls in 20.484 CPU seconds
ncalls tottime percall cumtime percall filename:lineno(function)
494604 8.461 0.000 14.218 0.000 Tkinter.py:2135(_create)
11 3.839 0.349 20.315 1.847 mandel2g_cy.pyx:27(create_fractal)
494632 2.053 0.000 5.309 0.000 Tkinter.py:1046(_options)
494657 1.809 0.000 2.811 0.000 Tkinter.py:77(_cnfmerge)
494593 1.522 0.000 16.475 0.000 mandel2g_cy.pyx:21(draw_pixel)
989240 0.752 0.000 0.752 0.000 {method 'update' of 'dict' objects}
494593 0.736 0.000 14.953 0.000 Tkinter.py:2155(create_line)
989257 0.698 0.000 0.698 0.000 {_tkinter._flatten}
494632 0.315 0.000 0.315 0.000 {method 'items' of 'dict' objects}
1 0.158 0.158 0.158 0.158 {_tkinter.create}
494641 0.131 0.000 0.131 0.000 {callable}
37 0.009 0.000 0.009 0.000 {built-in method call}
11 0.001 0.000 20.317 1.847 viewer2b.py:20(draw_fractal)
12 0.000 0.000 0.000 0.000 viewer.py:60(info)
I notice that there are many function calls: over 5 millions of them.
While most of them appear to take very little time, they do add up
in the end. It is time to adopt a smarter drawing strategy.
Currently, a "line" is potentially drawn for each pixel. If I look at a given fractal drawing, I can see that it could be drawn using "longer lines", when consecutive pixels are to be drawn with the same colour. I can easily implement this as follows.
def create_fractal(int canvas_width, int canvas_height,
double min_x, double min_y, double pixel_size,
int nb_iterations, canvas):
cdef int x, y, start_y, end_y
cdef double real, imag
cdef bint start_line
for x in range(0, canvas_width):
real = min_x + x*pixel_size
start_line = False
for y in range(0, canvas_height):
imag = min_y + y*pixel_size
if mandel(real, imag, nb_iterations):
if not start_line:
start_line = True
start_y = canvas_height - y
else:
if start_line:
start_line = False
end_y = canvas_height - y
canvas.create_line(x, start_y, x, end_y, fill="black")
if start_line:
end_y = canvas_height - y
canvas.create_line(x, start_y, x, end_y, fill="black")
Note that I no longer need the function draw_pixel().
The result is a reduction from 20 seconds down to 4 seconds:
79092 function calls in 3.959 CPU seconds
ncalls tottime percall cumtime percall filename:lineno(function)
11 3.552 0.323 3.815 0.347 mandel2h_cy.pyx:21(create_fractal)
7856 0.150 0.000 0.250 0.000 Tkinter.py:2135(_create)
1 0.131 0.131 0.131 0.131 {_tkinter.create}
7884 0.037 0.000 0.091 0.000 Tkinter.py:1046(_options)
7909 0.030 0.000 0.047 0.000 Tkinter.py:77(_cnfmerge)
7845 0.014 0.000 0.263 0.000 Tkinter.py:2155(create_line)
15761 0.014 0.000 0.014 0.000 {_tkinter._flatten}
15744 0.013 0.000 0.013 0.000 {method 'update' of 'dict' objects}
37 0.009 0.000 0.009 0.000 {built-in method call}
7884 0.005 0.000 0.005 0.000 {method 'items' of 'dict' objects}
7893 0.002 0.000 0.002 0.000 {callable}
11 0.001 0.000 3.818 0.347 viewer2b.py:20(draw_fractal)
12 0.000 0.000 0.000 0.000 viewer.py:60(info)
22 0.000 0.000 0.001 0.000 Tkinter.py:1172(_configure)
And it is now again my own code in create_fractal() that appears
to be the limiting factor. Thinking back of when I increased the number
of iterations from 100 to 1000, thus only affecting the execution time
of mandel(), it seemed like this might be a good place
to look at for possible time improvements. Let's recall what the code
looks like.
cdef inline bint mandel(double real, double imag, int max_iterations=20):
'''determines if a point is in the Mandelbrot set based on deciding if,
after a maximum allowed number of iterations, the absolute value of
the resulting number is greater or equal to 2.'''
cdef double z_real = 0., z_imag = 0.
cdef int i
for i in range(0, max_iterations):
z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
2*z_real*z_imag + imag )
if (z_real*z_real + z_imag*z_imag) >= 4:
return False
return (z_real*z_real + z_imag*z_imag) < 4
I used a Pythonic tuple assignement to avoid the use of temporary
variables. However, in a typical iteration, there will be 4 multiplications
for the tuple re-assigment and two more for the "if" statement, for a total
of 6. It is certainly possible to reduce the number of multiplications
by using temporary variables, as follows:
cdef inline bint mandel(double real, double imag, int max_iterations=20):
'''determines if a point is in the Mandelbrot set based on deciding if,
after a maximum allowed number of iterations, the absolute value of
the resulting number is greater or equal to 2.'''
cdef double z_real = 0., z_imag = 0.
cdef int i
cdef double zr_sq, zi_sq, z_cross
for i in range(0, max_iterations):
zr_sq = z_real*z_real
zi_sq = z_imag*z_imag
z_cross = 2*z_real*z_imag
z_real = zr_sq - zi_sq + real
z_imag = z_cross + imag
if (zr_sq + zi_sq) >= 4:
return False
return (zr_sq + zi_sq) < 4
So, there are now fewer multiplications to compute. Surely, this will
speed up the code:
78982 function calls in 4.888 CPU seconds
ncalls tottime percall cumtime percall filename:lineno(function)
11 4.478 0.407 4.748 0.432 mandel2i_cy.pyx:26(create_fractal)
7845 0.153 0.000 0.256 0.000 Tkinter.py:2135(_create)
1 0.128 0.128 0.128 0.128 {_tkinter.create}
7873 0.040 0.000 0.095 0.000 Tkinter.py:1046(_options)
7898 0.031 0.000 0.048 0.000 Tkinter.py:77(_cnfmerge)
7834 0.014 0.000 0.270 0.000 Tkinter.py:2155(create_line)
15739 0.013 0.000 0.013 0.000 {_tkinter._flatten}
15722 0.013 0.000 0.013 0.000 {method 'update' of 'dict' objects}
37 0.009 0.000 0.009 0.000 {built-in method call}
7873 0.005 0.000 0.005 0.000 {method 'items' of 'dict' objects}
7882 0.002 0.000 0.002 0.000 {callable}
11 0.001 0.000 4.750 0.432 viewer2b.py:20(draw_fractal)
12 0.000 0.000 0.000 0.000 viewer.py:60(info)
4 0.000 0.000 0.000 0.000 {posix.stat}
Alas, that is not the case, as the previous profiling run was slightly
below 4 seconds. [Note that I did run each profiling test at least three
times to prevent any anomalous result.] Apparently my intuition is not a very good
guide when it comes to predicting how cython will be able to optimize
a given function.
So far, the pictures the program has been able to produce have only been in black and white. It is time to spruce things up and add colour. To do this, we will need to make three general changes:
-
We will modify
mandel()so that it returns the number of iterations required to evaluate that a given point does not belong to the set; if it does belong, we will return -1. -
We will create a colour palette as a Python list. For a given number
of iterations required by
mandel(), we will pick a given colour, cycling through the colours from the palette. - We will need to change our line drawing method so that we keep track of the colour (number of iteration) rather than simply whether or not the point is in the set ("black") or not.
# mandel3cy.pyx
# cython: profile=True
import cython
def make_palette():
'''sample coloring scheme for the fractal - feel free to experiment'''
colours = []
for i in range(0, 25):
colours.append('#%02x%02x%02x' % (i*10, i*8, 50 + i*8))
for i in range(25, 5, -1):
colours.append('#%02x%02x%02x' % (50 + i*8, 150+i*2, i*10))
for i in range(10, 2, -1):
colours.append('#00%02x30' % (i*15))
return colours
colours = make_palette()
cdef int nb_colours = len(colours)
@cython.profile(False)
cdef inline int mandel(double real, double imag, int max_iterations=20):
'''determines if a point is in the Mandelbrot set based on deciding if,
after a maximum allowed number of iterations, the absolute value of
the resulting number is greater or equal to 2.'''
cdef double z_real = 0., z_imag = 0.
cdef int i
for i in range(0, max_iterations):
z_real, z_imag = ( z_real*z_real - z_imag*z_imag + real,
2*z_real*z_imag + imag )
if (z_real*z_real + z_imag*z_imag) >= 4:
return i
return -1
def create_fractal(int canvas_width, int canvas_height,
double min_x, double min_y, double pixel_size,
int nb_iterations, canvas):
global colours, nb_colours
cdef int x, y, start_y, end_y, current_colour, new_colour
cdef double real, imag
for x in range(0, canvas_width):
real = min_x + x*pixel_size
start_y = canvas_height
current_colour = mandel(real, min_y, nb_iterations)
for y in range(1, canvas_height):
imag = min_y + y*pixel_size
new_colour = mandel(real, imag, nb_iterations)
if new_colour != current_colour:
if current_colour == -1:
canvas.create_line(x, start_y, x, canvas_height-y,
fill="black")
else:
canvas.create_line(x, start_y, x, canvas_height-y,
fill=colours[current_colour%nb_colours])
current_colour = new_colour
start_y = canvas_height - y
if current_colour == -1:
canvas.create_line(x, start_y, x, 0, fill="black")
else:
canvas.create_line(x, start_y, x, 0,
fill=colours[current_colour%nb_colours])
If we profile this code, we find out that it takes about three times as
long to generate a colour picture than it did to generate a black
and white one - at least, for the starting configuration...
2370682 function calls in 12.638 CPU seconds
ncalls tottime percall cumtime percall filename:lineno(function)
11 4.682 0.426 12.184 1.108 mandel3_cy.pyx:36(create_fractal)
237015 4.310 0.000 7.084 0.000 Tkinter.py:2135(_create)
237043 1.019 0.000 2.575 0.000 Tkinter.py:1046(_options)
237068 0.877 0.000 1.353 0.000 Tkinter.py:77(_cnfmerge)
1 0.443 0.443 0.443 0.443 {_tkinter.create}
237004 0.418 0.000 7.502 0.000 Tkinter.py:2155(create_line)
474062 0.361 0.000 0.361 0.000 {method 'update' of 'dict' objects}
474079 0.313 0.000 0.313 0.000 {_tkinter._flatten}
237043 0.143 0.000 0.143 0.000 {method 'items' of 'dict' objects}
237052 0.061 0.000 0.061 0.000 {callable}
37 0.009 0.000 0.009 0.000 {built-in method call}
11 0.000 0.000 12.186 1.108 viewer3.py:20(draw_fractal)
12 0.000 0.000 0.000 0.000 viewer.py:60(info)
4 0.000 0.000 0.000 0.000 {posix.stat}
We also generate some nice pictures! First, using only 100 iterations.
We see much more details and many points that were wrongly identified as being part of the Mandelbrot sets are correctly excluded (and colored!). What happens if we look at a region that contains more points inside the Mandelbrot set (thus requiring more iterations) and increase the number of iterations to 1500 (as I found that 1000 was not enough in this region).
Unfortunately, the profiling and the timing information displayed does not tell the entire story. In practice, I found that it would take many more seconds (sometimes more than 10) for the canvas to be updated than the timing information given for each of the three pictures displayed above. Something is happening behind the scene when the picture is updated on the screen and which is not being recorded by my simple timing method.
For comparison, I had a look at Aptus, a program that uses a hand-coded C extension for speed. Actually, I had tried Aptus a few years ago, when Ned Batchelder first mentioned it, and tried it again for fun as I was working on my own version. Aptus can produce really nice pictures, really fast. Here's an example that was produced, according to the timing given by Aptus itself in 0.25 seconds.
Note that the timing given by Aptus seems to be truly representative, unlike the timing for my program. I should mention that, in addition to using a hand-crafted C extension, Aptus uses wxPython instead of Tkinter, and it also uses PIL and numpy, both of which are known for their speed. It might be possible that using PIL and numpy with my program would improve the speed significantly. However, all three libraries are not part of the standard library and so do not meet the constraints I had decided upon at the beginning.
This concludes this profiling experiment ... at least for now. I should emphasize that the goal of these posts was to record a profiling experiment using cython. I do not pretend that this code was especially hand-crafted for speed ... even though I did try some simple changes to improve its speed. I may revisit this application at a later time, especially if someone more experienced can point out ways to increase the speed significantly, preferably while staying within the constraints I set up at the beginning: other than cython, use only modules from the standard library. However, I would be interested if anyone adapts this code to use PIL and/or numpy in a straightforward way and, in doing so, increases the speed significantly.
Subscribe to:
Posts (Atom)