Ajax stands for "asynchronous JavaScript and XML," but these days json is often used as a lighter-weight alternative to XML. In this tutorial, I'm therefore going to show how to use the python simplejson library to enable communication between python and JavaScript using json.

What you will need for this tutorial:

• cherrpy
• simplejson
• jQuery
• The sample project

You can get cherrypy and simplejson from pypi via easy_install; there's a copy of jQuery in my sample project. The sample project consists of three files, in the following structure:

ajax_app.py
media/
  + jquery-1.4.2.min.js
  | index.html

First, I'll show the HTML file:

Let's see what this is doing. In the body, you've got a title tag, and bog standard HTML form, which asks for a name.

In the head, you've got an include to the jQuery library, and the following JavaScript code:

The initial $(function() statement is a jQuery construct that says, "When the page loads, execute this anonymous function." That function then creates a callback for the submit event of the element with id "testform" (the form). When the form is submitted, the jQuery post function is executed, which is where the magic happens.

jQuery.post is an Ajax method. The first argument is the URL to post to ("/submit"); the second argument is the data to send in the post (the "name" value from the form); and the third argument is the callback to call with the data we receive from the post.

The function then takes this data, and uses its "title" element to update the title of the page.

Next, here's the ajax_app.py file:

First, the AjaxApp class is our application. It's pretty simple, having only two methods: index and submit.

When the index method is called, the app opens the index file and returns it.

The submit method is our Ajax method; when it is called with the form data, we use the information to create a new title, and pass that back as json data using simplejson.dumps. The JavaScript function that called this method can then use the data to update the page content.

This little piece of code is a convenience for development; when you run the cherrpy app, it loads the index page in your web browser.

Finally, the last two lines mount the app, and start up the server. The "config" dict tells cherrpy to serve up static files from the "media" dir in the script's home directory.

If you download the sample project, unzip it, and run ajax_app.py, you should get the following screen (shown here with the name filled in):

Ajax app -- step 1

Here it is after filling in the name and clicking Set:

Ajax app -- step 2

When to use Ajax

Ajax is useful when you want to update the page content without reloading the entire page. This is useful for CRUD apps when you want to do things like edit items in place, delete items, or add items.

When not to use Ajax

Since Ajax actions don't correspond to URIs, you can't bookmark Ajax actions, send links to friends, or use the back and forward buttons to navigate like in a normal website.

For this reason, avoid Ajax if you want certain application states to be navigable as URIs. For example, if you have a calendar application, and use Ajax to show all the data, the user won't have any way to bookmark an event on December 12th, 2011. Likewise, if you use an Ajax data grid to show tables of data, you won't be able to email your data views to your colleagues.

One middle way is to offer a "permalink" for each Ajax view, which allows for bookmarking, sending links, etc. This still breaks the back-button functionality, but may be worth the trade-off.

In this post, I want to share a small helper module called sftp (zip file) (code in post below) that wraps paramiko.SFTPClient and makes uploading/downloading via SFTP even simpler.

First, some usage

Upload or download a file

with statement also supported:

Finally, a demo recipe for uploading all the png files from a specified local directory to a specified directory on the server:

Now, here's the code for my sftp module:

I've been reading Richard Feynmann's fascinating reminiscences in Surely You're Joking, Mr. Feynmann. The entire text is available online.

One passage that really struck me was where he described the cyclotrons at Princeton, MIT, and Cornell:

It reminded me of my lab at home. Nothing at MIT had ever reminded me of my lab at home. I suddenly realized why Princeton was getting results. They were working with the instrument. They built the instrument; they knew where everything was, they knew how everything worked, there was no engineer involved, except maybe he was working there too. It was much smaller than the cyclotron at MIT, and "gold-plated"? — it was the exact opposite. When they wanted to fix a vacuum, they'd drip glyptal on it, so there were drops of glyptal on the floor. It was wonderful! Because they worked with it. They didn't have to sit in another room and push buttons! (Incidentally, they had a fire in that room, because of all the chaotic mess that they had — too many wires — and it destroyed the cyclotron. But I'd better not tell about that!)

(When I got to Cornell I went to look at the cyclotron there. This
cyclotron hardly required a room: It was about a yard across — the diameter of the whole thing. It was the world's smallest cyclotron, but they had got fantastic results. They had all kinds of special techniques and tricks. If they wanted to change something in the "D's" — the D-shaped half circles that the particles go around — they'd take a screwdriver, and remove the D's by hand, fix them, and put them back. At Princeton it was a lot harder, and at MIT you had to take a crane that came rolling across the ceiling, lower the hooks, and it was a hellllll of a job.)

Princeton and Cornell had inferior equipment, but they got superior results partly because of the quick iteration loop, eliminating the step of waiting on the engineers to make changes. Scientists at these schools could very quickly try something, make adjustments, and try again, where at MIT the same adjustments took a lot of time.

Shuji Nakamura, inventor of the blue LED, said something similar about his work:

I had a very, very small budget and had to make everything I needed myself. I even made my own reactors—the furnaces needed to do the crystal work.

Nakamura succeeded in creating a blue LED, working by himself, when huge multinationals were pouring hundreds of millions of dollars into R&D without results. Elsewhere, he commented that the ability to make adjustments to his equipment quickly, without getting engineers involved, was one of the secrets to his success.

It struck me that this is one of the reasons why I get better results with Python than with compiled languages like C and C++. With Python, there's no compiler in the loop, so the iteration cycle is very fast — rarely more than 10 seconds to run all the unit tests.

This affects every part of the development cycle. Development itself is a lot faster, so you get more done. With a faster cycle, refactoring code in small steps is also quick and painless, so it's easier to evolve code. The cost of trying out new ideas is low, so you're encouraged to prototype lots of designs to get the best ones.

I think that in just about any kind of creative endeavor, the ability to rapidly iterate between creation and feedback gives a tremendous boost to productivity. This is one of the reasons why I'm so much more productive with Python.

• Project homepage
• pypi page

The result is my kanjinums module, with a function kanji2num that will convert a string containing a kanji num to a Python integer.

Download the source distribution (kanjinums-0.1.zip)
Download the Windows installer (kanjinums-0.1.win32.exe)
Download the unit tests (test_kanjinums.zip)

Examples:

Here's the full code:

Improvements in this version:

• Default arguments in Message.__init__() method
• Support for non-ascii charsets (in body and subject)
• Support for Python 2.4

With the support for non-ascii encodings, you can now do this:

And the Message instance will properly encode the subject line and message body. Note that there's currently no support for Unicode strings; you've got to pass in encoded strings.

Thanks to everyone who provided feedback on this module.

The mailer module also has a home page.

The full code of the new module is below.

2D Land: A Demonstration of the Adapter Pattern

Let's motivate that with a little example. We're going to start with a simple system, then add components to it with an interface that's different from the one our system expects. We'll use the Adapter pattern to wrap the new components, so we can use them without changing the code in our existing system.

So let's get started. We've been tasked with writing the back-end code for 2D Land, a cutting-edge world simulation program involving poorly drawn cartoon figures. Our initial spec is pretty simple: our world consists of people, and when a person is clicked, their name and a speech bubble is printed. We need to code a Person class, and a click_creature() function that will return a person's name and speech bubble to the GUI team.

So let's get coding. One way to code this would be as shown below. We have a Person class, with a name property and a make_noise() method. These can be accessed to retrieve the Person's name and speech bubble.

The Person class takes a name argument, which it stores in its name attribute. The click_creature() function retrieves the person's name and speech-bubble text. We test it, and everything works as expected. Looks like it's time for a launch party!

More Features in 2D Land

After we get back from our team trip to Cancun, we find that our users are so happy with 2D Land that management has decided to build 2D Land version 2.0! The killer new feature of version 2.0 is that in addition to people, 2D Land will now also have dogs.

Luckily, the programmers over in the Pet Land team are going to let us use their Dog class. The Dog class has a name property, and a bark() method that returns the dog's bark sound.

As it stands, the Dog class doesn't have the interface that our clients expect. It has a name property, but instead of a make_noise() method, it's got a bark() method. But we can use the Adapter pattern to wrap the Dog class and give it the expected interface.

Let's start by solving this the classic GoF way. There are actually two ways to implement Adapter in GoF: the Class Adapter and the Object Adapter. The Class Adapter uses multiple inheritance, while the Object Adapter uses encapsulation. For the sake of completeness I'm going to show both of them, and then explain why the Object Adapter is almost always the best choice for Python.

According to the classic implementation of the Adapter pattern, we need a common class that Person and a dog adapter can inherit from; let's call it a Creature class. The DogClassAdapter and DogObjectAdapter classes will each inherit from Creature; DogClassAdapter will then inherit from Dog, and DogObjectAdapter will wrap Dog.

According to the pattern, Creature is supposed to be an abstract base class (ABC); that is, it's not supposed to be instantiated directly. Python 2.X doesn't have ABCs as a language feature, but we can fake it here by having calls to make_noise() raise a NotImplementedError. This means that we need classes to inherit from Creature and implement make_noise() for us.

DogClassAdapter inherits from Creature and Dog, getting the functionality of both. It takes a name parameter in its __init__() method, which it passes on to Dog. It then implements a make_noise() method, returning the results of the bark() method it inherited from Dog.

DogObjectAdapter only inherits from Creature. It takes a Dog instance as a parameter to its __init__() method, and stores that instance as self.canine. It implements a make_noise() method, which returns the result of calling the bark() method of its wrapped Dog object. All other calls on the class are passed to its canine instance via the __getattr__() method.

In both cases, we only need to implement the interface we're adapting. DogClassAdapter inherits the rest of the Dog interface, while DogObjectAdapter uses __getattr__() to provide the same interface as the Dog class, with the exception of the methods it implements itself.

And here's why we don't need the Class Adapter. Class Adapter is basically a shortcut to avoid coding stubs. In languages like C++, Object Adapter would force us to implement stubs for every method on the Dog class, just so we could pass them on to the wrapped Dog instance. With Class Adapter, however, we'd already have the behaviour implemented, so wouldn't need to write stubs. This doesn't make any difference in a simple class like Dog, but consider a huge class like StringIO.StringIO from the standard library; writing stubs for every method in that class just to adapt the interface slightly would be a real pain. But in Python, we don't have to write stubs for the behaviour we don't wrap; we can take advantage of dynamic dispatching instead, by implementing the __getattr__() method.

__getattr__() is a "magic" method, as denoted by the double underscores. The Python interpreter calls it when it does an attribute lookup on the object, and can't find a matching attribute. By intercepting this call and passing on the attribute lookup to the wrapped object, DogObjectAdapter only needs to implement the make_noise() method; everything else gets passed on to its wrapped object.

Flattening the Object Hierarchy

The code above works; we can created adapted dogs, and feed them into our click_creature() function. But is inheriting from a Creature base class the best way? With statically typed languages, inheritance is used for two purposes: to coerce types, and to inherit behaviour. By coercing types, I mean using a common base class so that we can feed a common type to the click_creature() function. These are the kinds of hoops you need to jump through to make the compiler happy in statically typed languages. In Python, we don't need to use inheritance to coerce the type system, because we use duck typing; we only need to use inheritance for behaviour.

Duck typing: ask what an object can do, rather than ask what type an object is (i.e. don't ask if it's a duck; ask if it quacks like a duck).

So we don't need to inherit from a Creature base class in order to use our various objects in the click_creature() function. And in fact, there's a big problem with this inheritance approach: it would force us to change our existing codebase, by modifying the Person class to inherit from Creature. So let's flatten the hierarchy out:

Here's our code without the Creature base class, and getting rid of the Class Adapter:

This is a lot simpler. We've captured the intent of the Adapter pattern, without bothering with the trappings of less dynamic languages.

Just because the GoF implement a pattern in a certain way doesn't mean it's the only or even best way to do it in Python. Use the essence of the pattern, and leave out the accidents of implementation.

Testing out the System

Let's test out our DogAdapter class with a function that simulates clicks on each type of creature in 2D Land.

animate_objects() creates instances of classes Person and DogAdapter; it then prints the name and speech bubbles of each object. The output is shown below.

Output:

Perfect! Looks like we're ready to launch version 2.0.

Extending the Adapter Concept

We've got our Adapter working pretty well now, but we can still make it better. Think about how we added a Dog class to 2D Land. What if we later want to add a Cat class, and Bird class, and so on? Here's a Cat class, that has a name attribute, and a meow() method that returns the cat's meow sound.

The problem with the current implementation is that every time we add a new class to our system, we're potentially going to have to write a new adapter class for it.

We can take advantage of Python's dynamic nature to write a single, generic adapter class for all the new classes we add. Our generic adapter will take two arguments in its __init__() method: the object it's going to wrap, and an implementation of make_noise().

Note that the code no longer has to know anything about the Dog class or other creatures in 2D land. All we need to do is supply an object to adapt and a method to use for make_noise(), and we're done.

So let's test out the new system, and see how we'd specify the make_noise() method.

We have to specify which method to use as the make_noise() method on instance creation, but in return we no longer have to create an Adapter for each new class in the system. I'd say that's a big win.

Output:

One final note about this system: you may have been wondering what happens if the class we're adapting doesn't have a name attribute (for example, if the attribute is called "nickname"). In that case, we could simply create new class that inherits from the creature class, implementing the name attribute via a property, and pass an instance of that to our CreatureAdapter class.

We could then do something like this:

Note that this presupposes that the adapted class has some functionality that serves as the name attribute; the Adapter class is all about changing the interface of an object, not adding to its functionality.

Using Adapter with Objects Other Than Classes

The Adapter pattern as classically implemented works with classes, but that's just an accident of the languages that the pattern was initially made for. Essentially any object in Python is ripe for adapting. For example, you could adapt a function so that its signature was something that the client expected.

As an example, below we have a countdown() function. It takes as its arguments a number to count down from, and a callback function to call when it's done (callbacks are a common programming technique in things like multithreaded and asynchronous programs). We want to use the completed() function as its callback, but completed() takes an argument (the source), while countdown() calls its callback without any arguments. We can use the adapter pattern to change the signature of completed() to what countdown() expects.

Above, we adapted the completed() function using the nullarg_callback() function.

Output:

The Adapter pattern changes the interface of an object in order to match the interface that's expected. In the 2D Land example, we adapted the Dog class to provide make_noise() method that our system expects. The Adapter pattern is a good choice any time you're integrating some code into an existing system, and the code has the needed functionality, but not the needed interface.

Here's the code for this post, together with unit tests.

The email and smtplib modules in Python are very powerful, but they're also a bit complex when you just want to send an email.

I wrote the mailer module as a front end to these two modules, in order to make the task of sending email in Python using SMTP as simple as possible.

Below are some examples.

Send a simple plain text email

Send an email with an attachment

Send an HTML email

Download the mailer module (zip file).

Edit: Here's the source code of the mailer module.

The problem

David Beazley has a great article about generator pipelining using Python. This is a technique for handling (potentially very large) streams of data in a flexible yet efficient way. As an example, here's a code snippet he gives for summing the total bytes from a log file:

The code above first opens a log file, then gets the byte column for each entry. The byte value (if any) is then calculated for each row. Finally, the generator is consumed (or "pumped"), yielding the sum.

Since the entire file is never loaded into active memory, you could run this on quite huge log files, or even add a few steps and run it on collections of log files, without blowing up your memory. Another feature of this technique is that it's very flexible: you can add steps, combine steps into atomic actions, rearrange them, and so on.

This works great, as long as your pipe doesn't branch. If you want to split your pipe — say, dividing a stream of integers into one stream of even numbers and another of odds, things get a little complicated. One really elegant way to handle this situation is with the itertools.tee function. tee takes an iterable sequence, and returns n "copies" of that sequence that can be iterated independently.

Using itertools.tee

The code first opens a file containing a bunch of random integers, and creates a generator that's a stream of integers. It then uses itertools.tee to make two copies of that generator (first and second), and applies generator expressions to create two streams: one of even numbers, and one of odd numbers. The built-in sum function is then used to consume each tee.

Here's the number file that I used for this code. It's a list of 1,000 random integers between 1 and 1,000,000.

That's fine in this case, where the filter expression is relatively inexpensive. But what if we have an expensive filter, like testing whether the number is prime, or making a database query? It could really hurt our performance if we have to perform the same test twice. Ideally, we'd just like to perform the test once for each element in our pipeline.

There are lots of ways to handle that situation. One common way is the "continuation-passing" style, where you pass some data, a filter condition, and one or more functions to perform depending on the results of the test.

This works, but it disrupts the pipeline. That costs us the flexibility and dynamic nature of the generator paradigm.

I wrote the conditional tee (ctee) module for cases when you want to use a generator pipeline, but you need to split the sequence into two generators, and the filter condition is expensive. It creates a pair of instances of the ConditionalTee class, which are linked to each other.

Using conditional tee

Here's the meat of the code. The module can be downloaded here (ctee.zip).

The ConditionalTee class takes a sequence and a filter condition as arguments to its __init__ method. The __init__ method also creates an empty queue member, and an othertee member that's initialized to None.

When the next method of a ConditionalTee instance is called, it first looks for any items in its queue. If there is an item on the queue, it returns the first one. Otherwise, it iterates through its sequence; it keeps adding any items that don't match to the queue of its othertee member, until it either finds an item that matches or raises a StopIteration exception.

The ctee function also takes a sequence and a filter condition as arguments. It creates two ConditionalTee instances, and sets their othertee members to each other, then returns the two instances as a pair.

Here's some sample code using ctee:

There's still a problem if you "pump" each of these generators in succession, though: if the amount of data is large, the other generator class is going to accumulate a huge queue of data. It would be better to pump each generator expression alternately, taking an item and processing it from each generator in turn, in order to avoid building up a big queue.

Here's a function that'll do that:

Pumping generators alternately instead of consecutively

This takes a sequence of (sequence, action) pairs. It iterates through each pair, taking the next item in the sequence and applying the action to it. If the sequence raises StopIteration, it's removed from the list of sequences. The list comprehension at the start of the function is to make sequence test False when empty, and to ensure each sequence in it is an iterable (i.e. supporting next).

Here's an example of using this function:

This code will write all the even numbers to "evens.txt", and all the odd numbers to "odds.txt".

You might ask, how is this different from the continuation passing style? And you'd have a point; this is essentially continuation passing.

The thing is that here, the pumping only happens at the end. You can still go on wrapping all sorts of other filtering and transforming generators around your two conditional tees; the sequence won't actually be processed until you start pumping the generators at the end, so you won't build up enormous queues of data.

If you only knew a statically typed and compiled language like C++ or Java before, and learning Python hasn't changed the way you think about programming, then check your programs for lots of type-checking code, inheritance, array indexing, and mutable variables. Look at some excellent Python code, and identify where your code diverges from it.