The GITS Blog

Counting words (etc.) in an HTML file with Python

Ryan Ginstrom — Sat, 17 May 2008 00:50:38 +0000

In a previous post, I wrote about how to count words, characters, and Asian characters using python.

In this post I want to pull that together with code to get a word count from an HTML file.

What needs counting

What needs counting depends to some extent on what you need the word count for, but here I'm going to be assuming that the word count is going to be used to count billable/localizable content.

In that scenario, you've got to count the text in the title tag, as well as the visible text in the body, and certain other localizable content: img alt attributes, a title attributes, and input value attributes (am I missing any?).

The Code

The code for counting the actual text is in the above link. Here we need code to extract the text from the HTML file, and to accumulate the counts for all the chunks we've extracted.

Here's the Segment class for accumulating counts:

class Segment(object):
"""Represents a text segment.
(For bookkeeping)
"""

def __init__(self, text=""):
""" text is the segment of text we will calculate.
Leave it empty if this will be a master count for a document

@param text: The text of the segment
"""

self.characters = len(text)

num_spaces = len([x for x in text if x.isspace()])
self.chars_no_spaces = self.characters - num_spaces

self.asian_chars = len([x for x in text if is_asian(x)])

self.non_asian_words = non_j_len(text)

self.words = self.non_asian_words + self.asian_chars

def accumulate(self, seg):
"""Add the stats from to this one.
Use this to keep a count for the entire document;
use another for the whole batch of documents

@param seg: The segment to accumulate

>>> seg = Segment(u"")
>>> seg2 = Segment(u"abc")
>>> seg.accumulate(seg2)
>>> seg.words
1
>>> seg.characters
3
"""

self.words += seg.words
self.characters += seg.characters
self.chars_no_spaces += seg.chars_no_spaces
self.asian_chars += seg.asian_chars
self.non_asian_words += seg.non_asian_words

Next, the code for extracting (segmenting) the text from an HTML file. For this, you'll need the excellent Beautiful Soup module.

#coding: UTF8
"""Html segmenter"""

from BeautifulSoup import BeautifulSoup as bsoup
from BeautifulSoup import BeautifulStoneSoup
import re

def normalize(text):
"""Normalize whitepace in C{text}.

>>> normalize(u" spam\\n\\tspam SPAM")
u'spam spam SPAM'
"""

return u' '.join(text.split())

class Segmenter(object):
"""Html segmenter
Retrieves the editable/translatable text from an HTML document.
"""

def __init__(self):
"""Set up various regular expressions for splitting the text"""

self.pre_parse_stripper = re.compile(u"|".join([u"|",
u"\s\S]*?>|",
u"\s\S]*?>|",
u"\s\S]*?>|",
u"[\s\S]*?",
u"\s\S]*?>|"]),
re.I | re.M)
"""Strip out unsightly tags before heading to the splitter"""

self.splitter = re.compile(u'|'.join([u"|

",
u"|

",
u"|",
u"|",
u"\d*?>|\d>",
u"|",
u"|",
u""]),
re.I | re.M)
"""Split segments by certain tags (removing tags in bargain)
These tags indicate a segment boundary"""

self.charset_finder = re.compile(u'[\s\S]*\s\S]*?charset\s*=\s*([\S]+)"[\s\S]*?>[\s\S]*', re.I)
"""Find the charset if necessary"""

self.soup = None

def __str__(self):
"""So we can tell which segger we have (assuming multiple segmenter classes)"""
return "HTML"

def get_chunks(self, html_text):
"""Extract the text from the HTML file"""

self.soup = bsoup(html_text, fromEncoding=self.getEncoding(html_text))

# document title
if self.soup.head:
title = self.soup.head.title
if title:
yield title.string

# image alt attributes, anchor title attributes, input value attributes
for tag, attr in ((u"img", u"alt"),
(u"a", u"title"),
(u"input", u"value")):
for chunk in self.getAttributes(tag, attr):
if chunk:
yield chunk

# Parse the body text
if self.soup.body:
text = self.pre_parse_stripper.sub(u"", unicode(self.soup.body))
for chunk in self.splitter.split(text):
normal = normalize(html2plain(chunk))
if normal:
yield normal

def getAttributes(self, tagName, attrName):
"""Get all attrName values for tagName tags"""

attrs = []

tags = self.soup.findAll(tagName)

for tag in tags:
try:
attr = tag[attrName]
if attr:
attrs.append(attr)
except KeyError, e:
#print "Tag %s does not have attribute %s" % (tagName, attrName)
pass

return attrs

def getEncoding(self, text):
"""Retrieve the encoding META tag, if present"""

m = self.charset_finder.match(text)
if m:
return m.groups(0)[0]
return None

TAG_STRIPPER = re.compile(u"<[!\w/][\s\S]*?>", re.I | re.M)

def strip_tags(line):
"""strip the HTML tags from the line

>>> strip_tags(u"spam")
u'spam'

"""

return TAG_STRIPPER.sub(u"", line)

def html2plain(text):
"""Strips out tags from HTML text

>>> html2plain('spam eggs')
u'spam\\xa0eggs'
>>> html2plain('–>')
u'–>'
"""

entities = BeautifulStoneSoup.HTML_ENTITIES
text = unicode(BeautifulStoneSoup(strip_tags(text),
convertEntities=entities))
return text.replace(u">", ">").replace(u"<", "<")

And here's some code to get the actual wordcount:

wordcount = docstats.Segment()
segger = htmlseg.Segmenter()

for chunk in segger.get_chunks(open("thefile.html").read()):
wordcount.accumulate(docstats.Segment(chunk))

Here are the docstats and htmlseg modules, and here is an online tool using the code for the HTML word counts.

The invisible translator

Ryan Ginstrom — Thu, 15 May 2008 03:29:03 +0000

By the nature of our profession, translators are generally invisible when they're doing their jobs right.

I say "generally" because this isn't quite a universal truth. For example, unlike in the United States, Japan is a country where a movie subtitle translator (and arguably not even a stellar one) can become a television celebrity. But that's a post for a different time (and a different blogger!).

So let's assume that a translator who has done her job is effectively invisible — there is no awareness of her presence between the author and reader. The implication is that despite the ubiquity of translation, the general public has a poor awareness of it.

This makes customer education pretty difficult, to say the least. Yes, we can sometimes find savvy consumers of translation, but most of the time they're in turn beholden to an ignorant consumer of translation down the line.

It also makes marketing translation as a profession a losing proposition.

Not be be a pessimist, but I don't see the market changing any time in the near future. Translation has been around a long time, and attitudes haven't really changed. The best we can do as translators is insulate ourselves from the ignorant consumers as best we can by surrounding ourselves with the savvy ones, and slowly, slowly educating our existing customers and culling out the bad ones.

Speeding up search on Honyaku archive site

Ryan Ginstrom — Tue, 29 Apr 2008 11:28:48 +0000

Last summer, I launched a new archive site for the Honyaku mailing list.

The site is written in Python using the django framework, with MySQL as the database. I chose MySQL because my tests showed that it was much faster than PostgreSQL at text searching.

Lately, however, the searches have been taking a huge amount of time. Sometimes they would even time out. It makes sense, since I've got more than 216,000 emails in there now, and body__icontains isn't exactly a speed demon.

But it was also taking forever just to get the posts for a given day. That was pretty easy to solve, though: duh, create an index on the date_sent field. So simple I never thought about it until the system was bogging down like a Golden Week traffic jam.

That solved the date problem, but my text search problem remained. In the end, I had to create a full-text index for the simple search. This solved the speed problem — queries take a second or two now — but the problem with MySQL's full-text index is that it has lousy support for Japanese text (which isn't delimited by spaces). For that reason, I kept the old, slow search method for the advanced search. If you use that, I recommend narrowing the search rather than just entering some body text.

In the end, I'm going to have to bite the bullet and install some kind of n-gram indexing scheme that will support Japanese. Right now, though, I simply don't have the time.

As a stopgap measure, I added a Google search for the Honyaku archive. Google doesn't seem to have indexed the site yet (I just took the main archive out of the robots.txt file), but when it does it'll be a quick way to search with good Japanese support. They even have a gadget that I can put on the Honyaku archive site, but I can't get it to keep the height I set for it.

Do the math

Ryan Ginstrom — Sat, 22 Mar 2008 15:01:54 +0000

I don't like doing so-called "native checks," or proofing other translators' work in general. It just turns into a bad experience way too often.

I'm candid about this with clients. I tell them I prefer not to do that sort of work. Sometimes they ask anyway, and if they're good clients (i.e. they send me lots of the kind of work that I like), then sometimes I'm not too busy to do it.

The other day, a client I occasionally work for (the one that was having quality issues with another translator) called me and asked me to proof their new translator's work.

Feeling vaguely sorry for them because of their lousy past translations, I said I'd do it. They asked me to quote a rate for doing this proofing work, and I quoted roughly one third my translation rate, softy that I am (a lot of people charge half).

Now, this client obviously knows what I charge them for translation. From my turnaround, they should also have a decent idea of what I earn per day. And they know that I don't like proofing. So when I gave them my rate, did they thank me for the great deal I was cutting them? No. Instead I get the sucking of teeth, and asking me to work for an hourly rate instead of the per-word rate I quoted, for what amounts to less than a third of what I can make translating.

Let's see: translate for some other client doing work I enjoy, or work for less than a third of the money doing something I hate. Tough call!

Come to think of it, the reason for this company's quality problems have now become a bit clearer.

What price elegance?

Ryan Ginstrom — Fri, 21 Mar 2008 04:03:03 +0000

In a recent post, I gave some code for counting the top n most frequent words in an arbitrary text file using itertools.groupby.

The code is written in a somewhat functional style. It's short and, dare I say, kind of elegant. But it turns out that this code is quite a bit slower than an imperative style using collections.defaultdict.

Here are the two functions:

from itertools import groupby
from collections import defaultdict

def get_top_freqs_gb(filename, num):
"""Get the top num words from filename as a list
of (word, freq) tuples, using itertools.groupby
"""
freqs = [(len(list(g)), k)
for k, g in groupby(sorted(get_words(filename)))]
return get_top(freqs, num)

def get_top_freqs_dd(filename, num):
"""Get the top num words from filename as a list
of (word, freq) tuples, using collections.defaultdict
"""
freq_dict = defaultdict(int)
for word in get_words(filename):
freq_dict[word] += 1
freqs =[(v, k)
for k, v in freq_dict.iteritems()]
return get_top(freqs, num)

Here are the helper functions:

import re

def get_words(filename):
"""Get the words from filename"""
split = re.compile(r"\b\w+\b").findall
return [word
for line in open(filename)
for word in split(line.lower())]

def get_top(freqs, num):
return [(b, a) for a, b in reversed(sorted(freqs)[num*-1:])]

The groupby version is shorter than the defaultdict version, and I'd say that it's simpler and more readable as well. Because it's shorter, the groupby version is less likely to contain bugs. In particular, the defaultdict version has a mutable local variable (used as an accumulator in the for loop), which is a classic source of bugs. The groupby version is also likely to be easier to maintain because it's shorter and simpler.

But the defaultdict version of the function winds up being considerably faster.

The times it took to run these functions 10 times on my computer, retrieving the top 50 most frequent words for "/python25/readme.txt", are as follows (seconds rounded to 4 decimal places).

	Without psyco	With psyco
groupby version	0.3133 s	0.2193 s
defaultdict version	0.2852 s	0.1818 s
groupby / defaultdict	1.41	1.58

The defaultdict version is 1.4x faster than the groupby version. This gap grows even further when psyco is used, making the defaultdict version nearly 1.6x as fast. I'd say that most of the reason for the slowness is that the groupby version of the function performs two sorts, compared to one sort in the defaultdict version.

(The psyco speedup for the defaultdict version comes from the for loop; changing get_words to return a generator expression eliminates the speedup. The speedup for the groupby version comes from the freq list comprehension; changing this to a generator expression eliminates its speedup.)

So which one should I use?

It's pretty common for Python code written in a functional style to be slower than equivalent code written in an imperative style. Nevertheless, I tend to prefer the more functional style of programming, switching to a more imperative style (or other forms of optimization) if performance isn't satisfactory.

It is easier to optimize correct code, than correct optimized code.

–Yves Deville

A big question here is how to tell if the functional version is fast enough. My general rule of thumb is that the user would be prepared to wait up to two seconds for a typical "grovel through these files and tell me something interesting" command that's performed infrequently (how frequently do you need to get word frequencies from files?). For a more common action, the wait time should be under a second, with < .5 seconds being optimal (this includes GUI responsiveness but not Web page loading).

Given the times above, and assuming that the user will search no more than 50 files of sizes comparable to Python's README file, then either version of the function is sufficient. If we assume that the user will search up to 100 files, or files substantially larger than the README file, then only the imperative version is acceptable (and we may need to optimize this further if our demands are higher than this).

That's why it's so important to profile and test Python programs from the very beginning. I keep a suite of test cases that I profile with every build (performed at least daily), noting trends in performance and optimizing when the code has gelled and bottlenecks remain.

Here is the test code:

from time import clock

def time_func(func, iterations, *args, **kwargs):
"""Return the time it takes to execute func
itertations times."""
start = clock()
for x in xrange(iterations):
func(*args, **kwargs)
return clock() - start

def main():
filename = "/python25/readme.txt"
top_gb = get_top_freqs_gb(filename, 100)
top_dd = get_top_freqs_dd(filename, 100)
assert top_gb == top_dd

for func in [get_top_freqs_gb, get_top_freqs_dd]:
name = func.__name__
seconds = time_func(func, 10, filename, 50)
print "%s: %s" % (name, seconds)

print "With psyco"

import psyco
psyco.full()

for func in [get_top_freqs_gb, get_top_freqs_dd]:
name = func.__name__
seconds = time_func(func, 10, filename, 50)
print "%s: %s" % (name, seconds)

if __name__ == "__main__":
main()

The whole shebang:

#coding: UTF8
"""
Testing functional programming stuff
"""

from itertools import groupby
from collections import defaultdict
import re
from time import clock

def get_words(filename):
"""Get the words from filename"""
split = re.compile(r"\b\w+\b").findall
return [word
for line in open(filename)
for word in split(line.lower())]

def get_top(freqs, num):
return [(b, a) for a, b in reversed(sorted(freqs)[num*-1:])]

for func in [get_top_freqs_gb, get_top_freqs_dd]:
name = func.__name__
seconds = time_func(func, 10, filename, 50)
print "%s: %s" % (name, seconds)

if __name__ == "__main__":
main()

Counting occurrences in a sequence with itertools.groupby

Ryan Ginstrom — Thu, 13 Mar 2008 05:29:38 +0000

itertools.groupby is a great tool for counting the numbers of occurrences in a sequence.

Here are some examples from the interactive interpreter.

A list of numbers

>>> # Create a random list of numbers
>>> from random import random
>>> numbers = [int(random() * 10) for x in range(20)]
>>> numbers
[8, 0, 3, 2, 3, 9, 8, 2, 8, 3, 0, 2, 3, 8, 6, 5, 3, 6, 1, 8]
>>> # Now create a dictionary of numbers and numbers
>>> # of occurrences. Feed generator expression of
>>> # (number, frequency) pairs to dict().
>>> from itertools import groupby
>>> valdict = dict((k, len(list(g)))
for k, g in groupby(sorted(numbers)))
>>> for key, val in valdict.items():
print key, ":", val

0 : 2
1 : 1
2 : 3
3 : 5
5 : 1
6 : 2
8 : 5
9 : 1

And a function that does this for any iterable:

from itertools import groupby

def count_occurrences(iterable):
"""return a dictionary with items and numbers of occurrences
in iterable"""

return dict((item, len(list(group)))
for item, group
in groupby(sorted(iterable)))

Top 20 most frequent words in a file

>>> # get a wordlist from the Python README
>>> text = open("/python25/readme.txt").read()
>>> words = text.lower().split()
>>> words[:5]
['this', 'is', 'python', 'version', '2.5.2′]
>>> # get the frequency list, using DSU to sort top words
>>> freqs = [(len(list(g)), k)
for k, g in groupby((sorted(words)))]
>>> # sort the freqs, get last 20, and reverse
>>> # to put most frequent first
>>> for a, b in reversed(sorted(freqs)[-20:]):
print "%s %s" % (b.ljust(7), str(a).rjust(3))

the 442
to 227
is 127
and 127
you 118
a 117
of 110
in 107
for 94
python 81
on 79
if 77
this 72
or 62
be 58
with 56
it 53
are 53
that 52
as 47

Here's a function that will do this.

from itertools import groupby

def get_top_freqs(filename, num=20):
"""Get the top num words from filename as a list
of (word, freq) tuples
"""

text = open(filename).read()
words = text.lower().split()

freqs = ((len(list(g)), k) for k, g in groupby(sorted(words)))

return [(b, a) for a, b in reversed(sorted(freqs)[num*-1:])]

Making the robot dance

Ryan Ginstrom — Wed, 12 Mar 2008 14:33:39 +0000

Some time around 1980, my elementary school classroom got a computer. While most of the other kids fooled around playing Hunt the Wumpus, my friend and I found the BASIC manual that came with the computer. We laboriously copied in the code to make a "robot" appear on the screen. After a lot of typos, we finally got the program working: a crudely drawn robot appeared on the side of the screen, and moved to the center.

Then my friend found the section on modifying the code to make the robot "dance" (basically move its arms up and down a few times after it got to the middle of the screen). I thought that was just about the coolest thing I'd ever seen. I started tweaking the program in various ways, turning the program into a mini disco, with flashing colors and sounds.

I was hooked. I started staying in at lunch so I could have time with the computer. I also pestered my mom to buy me a computer. And she did: a Tandy PC with 3KB of memory for programs, and a tape cassette deck for memory storage. It was great. One of my more ambitious projects was tweaking the horse race game that came with the computer to have odds and varying payoffs like at the tracks. I also made a "lemonade stand" clone. I think the only non-game I made was a "conversation" program that would chat with you and say different things depending on what you typed. It was very simple, of course: like if you typed "HELLO," it would respond, "HELLO, HOW ARE YOU?" (For some reason, I think the BASIC environment only had capital letters.)

In college, I would go on to learn more advanced programming languages, and fancy concepts like finite state theory and formal logic. But that same joy was always there: the joy of getting the computer to do my bidding. At one point I wrote a program to parse English text, and just like when I was 11 and wrote that "conversation" program, it was just so neat to have the computer respond to my input.

Now, I'm a technical translator, but I also program. Although I do program professionally to some extent, most of my programming is just for the pure enjoyment of it. I actually think that's better than being paid to program, because I have a lot more freedom about what I do.

I've often tried to pin down just what is so interesting about controlling a computer. My gut feeling is that it's just intrinsically fun — who wouldn't want to program computers? But of course that's not true: most people think that programming computers is really boring. What makes programming interesting to some and boring to others? Maybe it's a certain way of thinking. Or maybe it's ability: some people "get" programming, and since we tend to like what we're good at, we just gravitate to it.

Of course, there was always the tedious side of programming. Back when I was 11, it was copying programs out of books using hunt and peck, and loading in my programs from cassette tapes. Now, it's fiddling with user interfaces and installers and interoperability. But it's all worth it when I run the program and get to see the robot dance.

Using chardet to convert arbitrary byte strings to Unicode

Ryan Ginstrom — Sat, 08 Mar 2008 02:24:36 +0000

chardet is a fantastic module for finding the encoding of arbitrary byte strings. You can combine this with a check for a BOM to pretty reliably turn them into Unicode.

Edit: Thanks to Kirit's comment below, I added code to check for UTF-32.

import chardet

def bytes2unicode(bytes, errors='replace'):
"""Convert a byte string into Unicode.
First checks for a BOM, and if one is found returns
the Unicode text minus the BOM. If there is no BOM,
falls back to chardet."""

encoding_map = ('\xef\xbb\xbf', 'utf-8′),
　　　　('\xff\xfe\0\0', 'utf-32′),
　　　　('\0\0\xfe\xff', 'UTF-32BE'),
　　　　('\xff\xfe', 'utf-16′),
　　　　('\xfe\xff', 'UTF-16BE'))

for bom, encoding in encoding_map:
if bytes.startswith(bom):
return unicode(bytes[len(bom):],
encoding,
errors=errors)

# No BOM found, so use chardet
detection = chardet.detect(bytes)
encoding = detection.get('encoding') or 'utf-16′
return unicode(bytes, encoding, errors=errors)

Usage:

text = bytes2unicode(open(filename).read(), 'replace')

Discussion: Why check for a BOM?

You might ask, why check for a BOM if chardet already does this? This is because although chardet will correctly detect the BOM, it won't tell you that it found it, so you won't know to chop it off before processing the text. Which means that you'd have to check for a BOM anyway in most cases.

Delivering the bad news

Ryan Ginstrom — Sat, 01 Mar 2008 03:01:22 +0000

A few weeks ago, a translation agency I work for occasionally called me in a panic. It seems that a major client had rejected one of their Japanese-to-English translations, calling it "unreadable," and providing another translation as a sample of the quality they were after.

The agency wanted to pay me to review their translation, and the sample provided by the client, and point out specifically what the quality problems were. They wanted to feed this back to their translator, who had been with them for several years and never had any quality complaints.

Being kind of a sucker, I agreed.

The original translation was indeed a big steaming pile. It looked like the (native Japanese-speaker) translator had spent a lot of time translating Japanese into English, but had never actually seen an English document.

The first thing that jumped out was that the file was full of double-byte characters. There were double-byte spaces mixed in, and instead of using a space and parenthesis ("hello (world)"), the translator had used double-byte parentheses ("hello（world）").

It was like a demonstration of all the things I have said to avoid in J-E translation. The English text itself was, as the client complained, unreadable. It was as if the translator had semi-randomly chosen an English translation for each Japanese word out of a dictionary, and mashed them together into semi-grammatical sentences. If it had ever gone through a "native checker," that person had done a lousy job, because the translation was still rife with basic errors like subject-verb agreement. Completely worthless as a translation.

I've seen a lot in this biz, but I was shocked by this. How could such an obviously unqualified translator have made it so long without complaint? My first thought was that the translator had been using a very good checker/rewriter, and for some reason, the translator had been unable to get the checker's services and had turned in the translation as-is. But then how had the agency missed it?

Then I saw what "Mika Jz" said in this post to the Honyaku mailing list regarding strange English in an email received from a client:

英語の読解ができる日本人でも、
それを見て「よく書けている、いい英文だ。」
と感じる人は多いのではないでしょうか。

それのどこがおかしいのか、については
おそらく、全く通じていないと思います。

And another possibility occurred to me: maybe the agency and its clients had been using "native Japanese speakers with good English skills" to proof this translator's translations. And since every Japanese word was translated, and every translation could be found in a bilingual dictionary, they must have thought that they were fine.

So I had a couple of possible explanations, but the question was how to deliver the bad news. If someone had asked me to simply evaluate the translation, I'd have said it was unreadable and useless, end of story. But the agency wanted feedback, presumably so the translator could improve. What to do? I personally don't think that this translator will be up to producing professional-level English text for many years (if ever), but I had to put it in a somewhat more diplomatic way.

So I wrote up a report, comparing several passages of the translation with the client's sample (which was actually quite good), pointing out errors or poor style (e.g. write "the team investigated the issue," not "the investigation of the issue was conducted by the issue-investigation team"), and finally stating that writing natural English requires an extremely advanced grasp of English that takes many, many years to acquire. Hopefully they'll get the hint.

Or at least make sure to get a "native checker" to rewrite all future translations by this translator.