• Project homepage
• pypi page

The result is my kanjinums module, with a function kanji2num that will convert a string containing a kanji num to a Python integer.

Download the source distribution (kanjinums-0.1.zip)
Download the Windows installer (kanjinums-0.1.win32.exe)
Download the unit tests (test_kanjinums.zip)

Examples:

Here's the full code:

The problem

David Beazley has a great article about generator pipelining using Python. This is a technique for handling (potentially very large) streams of data in a flexible yet efficient way. As an example, here's a code snippet he gives for summing the total bytes from a log file:

The code above first opens a log file, then gets the byte column for each entry. The byte value (if any) is then calculated for each row. Finally, the generator is consumed (or "pumped"), yielding the sum.

Since the entire file is never loaded into active memory, you could run this on quite huge log files, or even add a few steps and run it on collections of log files, without blowing up your memory. Another feature of this technique is that it's very flexible: you can add steps, combine steps into atomic actions, rearrange them, and so on.

This works great, as long as your pipe doesn't branch. If you want to split your pipe — say, dividing a stream of integers into one stream of even numbers and another of odds, things get a little complicated. One really elegant way to handle this situation is with the itertools.tee function. tee takes an iterable sequence, and returns n "copies" of that sequence that can be iterated independently.

Using itertools.tee

The code first opens a file containing a bunch of random integers, and creates a generator that's a stream of integers. It then uses itertools.tee to make two copies of that generator (first and second), and applies generator expressions to create two streams: one of even numbers, and one of odd numbers. The built-in sum function is then used to consume each tee.

Here's the number file that I used for this code. It's a list of 1,000 random integers between 1 and 1,000,000.

That's fine in this case, where the filter expression is relatively inexpensive. But what if we have an expensive filter, like testing whether the number is prime, or making a database query? It could really hurt our performance if we have to perform the same test twice. Ideally, we'd just like to perform the test once for each element in our pipeline.

There are lots of ways to handle that situation. One common way is the "continuation-passing" style, where you pass some data, a filter condition, and one or more functions to perform depending on the results of the test.

This works, but it disrupts the pipeline. That costs us the flexibility and dynamic nature of the generator paradigm.

I wrote the conditional tee (ctee) module for cases when you want to use a generator pipeline, but you need to split the sequence into two generators, and the filter condition is expensive. It creates a pair of instances of the ConditionalTee class, which are linked to each other.

Using conditional tee

Here's the meat of the code. The module can be downloaded here (ctee.zip).

The ConditionalTee class takes a sequence and a filter condition as arguments to its __init__ method. The __init__ method also creates an empty queue member, and an othertee member that's initialized to None.

When the next method of a ConditionalTee instance is called, it first looks for any items in its queue. If there is an item on the queue, it returns the first one. Otherwise, it iterates through its sequence; it keeps adding any items that don't match to the queue of its othertee member, until it either finds an item that matches or raises a StopIteration exception.

The ctee function also takes a sequence and a filter condition as arguments. It creates two ConditionalTee instances, and sets their othertee members to each other, then returns the two instances as a pair.

Here's some sample code using ctee:

There's still a problem if you "pump" each of these generators in succession, though: if the amount of data is large, the other generator class is going to accumulate a huge queue of data. It would be better to pump each generator expression alternately, taking an item and processing it from each generator in turn, in order to avoid building up a big queue.

Here's a function that'll do that:

Pumping generators alternately instead of consecutively

This takes a sequence of (sequence, action) pairs. It iterates through each pair, taking the next item in the sequence and applying the action to it. If the sequence raises StopIteration, it's removed from the list of sequences. The list comprehension at the start of the function is to make sequence test False when empty, and to ensure each sequence in it is an iterable (i.e. supporting next).

Here's an example of using this function:

This code will write all the even numbers to "evens.txt", and all the odd numbers to "odds.txt".

You might ask, how is this different from the continuation passing style? And you'd have a point; this is essentially continuation passing.

The thing is that here, the pumping only happens at the end. You can still go on wrapping all sorts of other filtering and transforming generators around your two conditional tees; the sequence won't actually be processed until you start pumping the generators at the end, so you won't build up enormous queues of data.

If you only knew a statically typed and compiled language like C++ or Java before, and learning Python hasn't changed the way you think about programming, then check your programs for lots of type-checking code, inheritance, array indexing, and mutable variables. Look at some excellent Python code, and identify where your code diverges from it.

Output:

I wanted a similarly simple way to write XML, so I wrote a little module named xmlwriter (zip file) to do it.

Usage:

Output:

It's still pretty basic. For example, you can't have sibling nodes with the same tag name. But it's a very simple way to do some down and dirty XML writing.

Here's the xmlwriter module code.

My first stab at a solution was pretty straightforward:

This works for the intended purpose, although the C-style looping doesn't sit too well with me. But this code has a major flaw: it doesn't work for generators.

After a little more thought, I decided that what I wanted was to iterate over a window of two items in the sequence, and if the first item matched, return the second one.

To illustrate how this works:

The new item_after function:

In the new item_after function, I peel off the first pair so I can cache the first element, then glob it back on the front of the sequence using itertools.chain.

After a little more thought, I realized that the by_pairs function can be made generic to handle a window of any size.

The item_after function now looks like this:

And with a little data: one of my pets gets a treat every day. I know who got the treat yesterday; who gets it next?

The output:

I can think of a lot of potential uses for this window iterator. One is in natural language processing, when calculating the "stickiness" of words in a corpus. You could iterate over a window of three words, calculating the collocations of words before and after the middle word.

Here's a rather contrived example: calculating the average temperature on days when the previous two days were 80 degrees or higher.

Output:

It strikes me that this would be great for those baseball statistics guys:

This is just the third time since 1963 that a left-handed pinch hitter has broken a bat on three consecutive Wednesdays.

I'm waiting for a call from The Show any day now.

I'm currently reading Expert Python Programming by Tarek Ziadé. I plan on writing a review once I'm done, but in the meantime the publishers over at Packt have kindly sent me an excerpt from the book to publish here.

Download "Writing a Package in Python" (PDF)

To avoid going over old ground, for a primer please see this excellent article on using Unicode with Python. Here, I want to talk about some of the corner cases remaining after you've absorbed the great advice in that article.

This is not, of course, to say that Unicode support in Python is in any way buggy. Nay, Python's Unicode support is a unique snowflake, perfect in its own special way. It's just us flawed humans who have trouble appreciating fully its snowy beauty, especially if we're not Dutch.

And of course, all strings are Unicode in Python 3.0. That and the new syntax for extended iterable unpacking are the two main reasons I'm looking forward to Python 3.0. But alas, we'll have to enjoy the unique aspects of Unicode in Python for a bit more, now.

Input

I like to keep my programs as bastions of sanity, where all text is handled as Unicode. I thus try to put gatekeepers on all code accepting input, passing it on to the rest of the program logic as Unicode.

Programs that fail to do this often break when dealing with text input that they were sure would be fine as "ascii." One example of this is file paths. Programmers generally expect paths to be in nice, ASCII characters, and that's why their scripts often break when I run them on my Japanese system. For example, on my system the Desktop folder contains Japanese characters:

When a random python script breaks when run from my Desktop folder, I peek inside, and it's invariably because the programmer never expected the path to contain characters that couldn't be expressed as ASCII.

Put it into Unicode as soon as you get it.

As mentioned in the article above, the codecs module makes reading text files as Unicode very simple:

There are just a couple of twists to watch out for when using the codecs module.

1. It obviously can't guess the encoding; you've got to figure this out yourself.
2. open() converts the UTF-8 byte-order mark (BOM) ('\xef\xbb\xbf') into the UTF-16 BOM character ('\ufeff'), while removing the UTF-16 and UTF-16BE BOMs. This might not be what you expected.

Because of these shortcomings unique aspects of the codecs module, I normally use the chardet module in a custom function to get a random (i.e. user-supplied) text file as Unicode:

Output

As I mentioned, I like to get my text into Unicode as early as possible, and keep it as Unicode as late as possible. Ideally, I'd like to just output my text as Unicode, and let the output stream take care of the encoding (if any).

That's why when I need to output Unicode as a stream of bytes, I use the codecs module for files, and wrap the output stream otherwise. This is needed, for example, when using cStringIO, which chokes on Unicode.

Error message:

I can fix this by wrapping out with a class that intercepts the write() method, and converts Unicode strings to the specified encoding just before writing.

Now the example above works:

IDLE

IDLE has its own peculiarities regarding Unicode. It actually handles Unicode like a champ, but it assumes that everything you type at the command prompt is in the file-system encoding. Since I'm on a Japanese system, this is "mbcs." You can thus get into some odd states:

The general way to avoid these problems in IDLE is using sys.getfilesystemencoding().

Doctests

doctest is so full of snow-flaky uniqueness, I could put cherry syrup on it and call it a snow cone. Note in the example below that my "is_asian" function's doctests contain a Japanese character (日).

Running doctest on this gives a rather cryptic error:

It turns out that doctests can't handle Unicode characters. It's making the same "string of utf-8 bytes as Unicode characters" error as IDLE, and thus interpreting one character ("日") as three.

So we have to trick doctest by taking the repr value of the Unicode text (I usually stick the actual characters in a comment above it). Here's a repaired version, which runs without errors:

To see the silver lining in this, at least it encourages you to keep your complicated tests in unit tests, and save doctests for simple, illustrative purposes.

Conclusion

Unicode support in Python is actually quite good — much better than most languages. And it will get even better with Python 3.0. In the meantime, however, there are a few gotchas to look out for when using Unicode in Python.

This adapter can be used to adapt many objects with different interfaces into a single, unified interface.

Output:

In my opinion, this article is an egregious example of overuse of classes. I don't want to pick on the author in particular, but it illustrates my point so well that I want to examine the article's code here.

The article was about using Python for exploratory programming, but I think that the class-heavy style makes things more complicated than they need to be. The classes in the code essentially have no state. The one exception is TopRowsWBZipContent, where state is passed into the __init__ method, but is only used in one method and could just as easily have been passed in there. The author also uses extensive inheritance to get the various methods onto the class instances, where if vanilla functions were used, that could all be eliminated.

Here, I want to post the code from the article, and below that my rewrite using plain functions.

First, the article's code (I've made some of the interspersed text into comments):

Here's my rewrite using vanilla functions. The code is now a lot shorter, and I think easier to understand. (It's also easier to test. Yes, I believe in unit testing exploratory code, at least once it settles down a bit.) I've been a bit snooty and used more Pythonic coding conventions while I was at it.

Note that this code is even more suited to exploratory programming than the class-based code, because we don't have to write all the class machinery, and we can mix around functions without the need for inheritance or other abuses.