might as well hack it

Oblivious random sampling

2012-11-12T00:00:00-08:00

Here’s another neat trick I picked up from Programming Pearls:

How can you select one of n objects at random, when you see the objects sequentially but you don’t know the value of n beforehand?

This is Problem 10 of Column 12 of the book. This problem could arise, for example, when reading a file line by line to select one line randomly without storing the entire contents of the (potentially big) file in memory.

The solution turns out to be quite an elegant loop: simply keep track of the currently selected object, and replace it with the i-th object with probability 1/i. That is, always select the first object, then replace it with the second with probability 1/2, and replace the then-selected object with the third with probability 1/3, and so on. It is not very hard to see that the probability for a given object to remain selected until the end of this loop is 1/n, which is exaclty what we wanted.

A few days ago, I caught myself in need to select k different elements randomly from a set of n elements. I immediately considered trying to generalize the algorithm above, by iterating over the set of n elements keeping track of the k elements picked so far, and replacing the (i % k)-th element that is currently selected with the i-th element traversed with probability k/i. As expected, this reduces to the algoritm above when k is 1, and unless element clashes in the same (i % k) slot cause problems (which I suspect they will, but have not proven ;) ), every element has k/n probability of being picked.

After doing some research, I found a slightly simpler and (more importantly) tried-and-true solution: Reservoir Sampling. The main difference here is that the slot that a newly-selected element is also random. This solves any problem my algorithm sketch could have with slot clashes, and is the solution I ended up using. Interestingly, for my application, I actually have all elements for random access, but ended up using a very elegant algorithm that generalizes one for the restricted scenario in which only sequential access is possible.

...keep reading at Oblivious random sampling

Three guys walk into a bar...

2012-05-03T00:00:00-07:00

Three guys walk into a bar: the algorithmist, the coder and the mathematician.

As they approach the counter, a short, grouchy employee with greasy hair cleans up their spots with a dirty piece of dustcloth. They sit on three adjacent stools and are greeted by their old acquaintance, the aging owner of the bar.

“Howdy! What can I get you tonight?”

“Hello there! New guy around here?”, says the coder, discretely pointing at the short employee.

“Very observant. He is, indeed, new. And oh, so dull! He never pays any attention to what he’s doing and keeps breaking glasses!”

The owner of the bar then turns around and points to a shelf, where lies a row of n identical glasses. In each glass, a different number is painted with red marker, from 1 to n. He continues:

“And that isn’t even the worst part. After breaking a glass, he doesn’t bother to mark the replacement with the same number as the one he broke. In fact, he never uses the same number, and just copies the replacement’s number from another glass in the shelf.”

“Why is that?”, asks the intrigued algorithmist.

“No idea! And I always need to keep an eye on the shelf, looking for missing and repeated numbers in the glasses, so I can properly replace them! As if I had nothing better to do than to run an O(n²) algorithm to check the numbers every single day!”

At this point the mathematician picks up a napkin from the counter and a pen from his pocket and scribbles something. He then glances at his two friends and at the other man and proposes:

“I’m pretty sure we can help you come up with a better way of doing that. In fact, I think this can be done in linear time with constant space.”

He then shows the napkin to the algorithmist and the coder. The napkin reads:

There is an array containing all the integers from 1 to n in some order, except that one integer is missing and another is duplicated. Suggest an efficient algorithm for finding both numbers.

After pondering for nearly an hour, the algorithmist snatches the napkin from the mathematician’s hands and writes on it for a few minutes. He then grins with satisfaction as he begins to explain his solution.

The Algorithmist’s Solution

Let’s start from two simpler derivatives of the original problem and work our way up to this solution.

First, suppose you have an array containing all integers from 1 to n in some order, except one of them is missing. This is the same as the original problem, without the duplicated number. How can you find the missing number?

That’s much easier: just take the sum S_a of all numbers in the array and compare it to S_n, the sum of the first n natural numbers. Since the first n natural numbers constitute an arithmetic progression, it follows that S_n = n(n+1)/2. The missing number is therefore just S_n – S_a.

Let’s now examine the converse problem: instead of removing a number from the array, we add one. This is the same as the original problem above, except there’s only a duplicated number, and none is missing.

The solution is straightforward: once again we compare S_a with S_n, and this time the duplicated number is S_a – S_n.

Notice how, since adding all numbers in the array takes linear time, both solutions above also take linear time and constant space.

The algorithmist’s solution uses a composition of the two simplified algorithms above to build a divide-and-conquer algorithm for the original, more complicated problem. Imagine the input array in the original problem can be decomposed in two halves: one containing all numbers from 1 to n/2, except one of the numbers is missing, and another containing all numbers from n/2 to n, with one duplicated. Then, finding the missing and extra numbers in the array would only be a matter of applying the simplified algorithms above to the halves of the array.

But how can we do that decomposition? And how do we know which half has a missing and which has an extra number?

For the splitting, we can just use a derivative of the pivoting subroutine from quicksort. Given an array of n integers from 1 to n, we can pivot around the middle value (n + 1)/2 even if such value is not in the array and expect about half the integers to fall to the left of the pivot, and the other half to fall to its right. If that property doesn’t hold, that is, if one side of the array is larger than the other, then the extra element must be in the larger side and the missing element should have been in the smaller side.

4, 1, 3, 5, 9, 10, 8, 7, 6, 7

— Numbers from 1 to 10 with 2 missing and an extra 7, pivoted around 5.5

Notice how the left side (4, 1, 3, 5) is smaller and adds up to 13, while we would expect 15. Likewise, the right side is larger and adds up to 47, while we expected 40.

We may be unlucky and find out that both sides have the same size, which means both the missing and the extra integers belong in the same side. However, this also means that the other side is untainted — it contains exactly half of the numbers in the range from 1 to n, so we can identify it by taking the sum of its elements and comparing it with what we expected it to be: the sum of either the upper or lower half of that range. We have thus split our array in two halves, one of which is a subproblem, in that it contains half of the numbers in the range from 1 to n, except with one missing and one duplicated, and another which can be easily identified once we have the sums of the numbers in each half. All we need to do now is recurse into the subproblem, reducing the size of our problem in half.

The running time of this algorithm is described by the recurrence T(n) = T(n/2) + O(n), which can be proved to be O(n), and it uses constant space. I was actually quite sloppy in the explanation above with regard to the sizes of each “half” of the array, or the exact behavior of the algorithm when n is even or odd, but these issues are of minor concern and are illustrated in the implementation below, in Python:

import math

def sum_arith_progression(n, m):
    """ Returns sum(n, n+1, n+2, ..., m) """

    return (m**2 - n**2 + n + m) / 2

def partition(l, p):
    """ Partition a list with p as pivot value.
    Returns left_sum, right_sum, idx such that sum(l[:idx+1]) == left_sum,
    sum(l[idx+1:]) == right_sum and all elements in l[:idx+1] are smaller
    than or equal to p, and all elements in l[idx+2:] are greater than p.
    """

    i = 0
    j = len(l)-1

    left_sum = 0
    right_sum = 0

    while True:
        while i <= len(l)-1 and l[i] <= p:
            left_sum += l[i]
            i += 1
        while j >= 0 and l[j] > p:
            right_sum += l[j]
            j -= 1

        if i > j:
            break

        l[i], l[j] = l[j], l[i]

    return left_sum, right_sum, i-1

def find_extra_and_missing(l, s, t):
    """ Given a list of integers in the range [s, t] in which a single integer
    has been removed and another one has been added, find the missing and the
    extra integer.
    Returns a tuple (missing, extra).
    """

    while l:
        mid = (s + t) / 2

        left_sum, right_sum, p = partition(l, mid)
        expected_left = sum_arith_progression(s, mid)
        expected_right = sum_arith_progression(mid+1, t)
        expected_p = math.ceil(len(l)/2.0) - 1

        if p == expected_p: # missing and extra are on the same side
            if left_sum == expected_left:
                l = l[p+1:]
                s = mid + 1
            else:
                l = l[0:p+1]
                t = mid
        else:
            if p > expected_p: # extra on the left, missing on the right
                missing = expected_right - right_sum
                extra = left_sum - expected_left
            else: # extra on the right, missing on the left
                missing = expected_left - left_sum
                extra = right_sum - expected_right

            return missing, extra

Notice how the code implements two simple tweaks to the algorithm: we calculate the sums of the elements in each half of the array while pivoting, and the tail recursion has been converted into a loop.

“That’s… interesting!”, says the mathematician as the enthusiastic algorithmist finishes his explanation.

“Well, yes, but a bit too convoluted for me. I have a simpler way, let me show you.”, says the coder.

The Coder’s Solution

Take Y, the XOR of all numbers from 1 to n, XOR’ed with the XOR of all numbers in the array: Y = 1 ^ 2 ^ 3 … ^ n ^ A₁ ^ A₂ … ^ A_n.

Every bit in every number that is both in the array and the range exactly once will be cleared; the result will be the XOR of the missing and extra numbers: Y = missing ^ extra.

This happens because all other numbers end up being XOR’ed with themselves once, thus clearing their bits in the result. The missing number, however, only gets XOR’ed in once, and the extra number gets XOR’ed three times — recall that a ^ a ^ a = 0 ^ a = a for any a, and thus the bits of the extra number are never cleared.

Now, since the missing and extra numbers are different, they must differ at some bit in their binary representation. Thus Y has at least one bit set. Let the i-th bit be one of those bits. Then, all numbers in the range of 1 to n whose i-th bit is set are candidates for being the missing or extra number.

We take the sum of all numbers with the i-th bit set in the range from 1 to n, and compare it with the sum of the numbers with the same property in our array. If the former is bigger, their difference is the missing number. Else, the difference (in absolute value, of course) is the extra number.

Having either number, the other can be found with a mere XOR with Y. For example, Y ^ missing = (missing ^ extra) ^ missing = extra ^ (missing ^ missing) = extra ^ 0 = extra. This algorithm might make a few passes over the input array and its range of values, but it is definitely linear and uses constant space as well.

“And voilà!”, exclaims the coder.

“Impressive, though I still like how elegant my algorithm turned out to be…”, adds the algorithmist.

“So what about you?”, asks the coder to the mathematician.

The Mathematician’s Solution

The sum of all numbers in our array minus the sum of all numbers in the range from 1 to n is of course extra – missing. Now, what is the sum of the squares of all numbers in our array, minus the sum of all squares of the numbers in the range from 1 to n? It is extra² – missing² = (extra + missing)(extra – missing).

So now we have extra – missing, and (extra + missing)(extra – missing). As you can see, we are a simple substitution away from finding both numbers.

“…and that’s about it.”, concludes the mathematician.

Beers are handed to each of the daring problem-solvers in glasses, all with a red 8 marked on them, and the night proceeds with lighter conversations.

Credits

I found the original problem at Tanya Khovanova’s blog through Hacker News.

The coder’s and the mathematician’s solutions were taken directly from the comments in that blog’s post. The algorithmist’s solution, however, is my own beast :)

...keep reading at Three guys walk into a bar...

Sorting networks revisited

2012-03-27T00:00:00-07:00

It’s been a few months since I made a brief and practical incursion into sorting networks. Little did I know at the time how rich and interesting is the theory behind those clever constructs.

Fast-forward to present date and I just came across two rich resources for learning the theory that drives sorting networks: Robert Sedgewick’s Algorithms in C and (wait for it) Don Knuth’s The Art of Computer Programming. Upon revisit, the Wikipedia article seems to have evolved quite a bit as well.

So it is time to stand on the shoulders of giants and take another look at sorting networks, this time from a more theoretical standpoint. This post is by no means intended as a complete resource for this theory — rather, I’m just pointing out some interesting facts about sorting networks that either helped me understand their working or just made me mumble “cool!” as I learned them.

Let’s start with a quick recap: sorting networks are representations of sorting algorithms using a network of interconnected compare-and-swap gates.

For example, consider the task of sorting an array of three integers, A[1..3]. One possible way of sorting this array would be to follow the procedure below:

if A[2] > A[3]:
    swap A[2] and A[2]
if A[1] > A[3]:
    swap A[1] and A[3]
if A[1] > A[2]:
    swap A[1] and A[2]

This algorithm can be represented (and implemented in hardware) as a network of compare-and-swap gates like the one below:

— A sorting network for three elements

In the image, the horizontal lines represent wires that serve as input and output to compare-and-swap gates, the vertical lines. The upper output of a compare-and-swap gate is the smallest of its inputs, and the bottom output wire receives the largest of its inputs. The network is read left to right, and by the end the elements in A are sorted in nondecreasing order.

Notice how the comparisons made by the algorithm above are the same regardless of what the input sequence looks like, that is, the sequence of comparisons is “oblivious”, as Knuth puts it. This algorithm is thus what Sedgewick calls a “nonadaptive algorithm” in the definition I quote below:

A nonadaptive sorting algorithm is one where the sequence of operations performed depends only on the number of inputs, rather than on the values of the keys.

Sorting networks have the interesting property of low-level parallelism: gates that don’t operate on the same inputs can often work in parallel. The efficiency of a sorting network can then be defined either in terms of its depth, the number of comparators in the longest path from an input to an output, or in terms of the number of comparator gates used to build it.

But how can we make a sorting network?

Other than devising them through logic (which doesn’t really gets me further than very small n), there are more systematic methods and heuristics. Knuth even shows in his book a sorting network for 13 elements obtained from a genetic algorithm which “bears no rhyme or reason, but it works — and it’s shorter than any other construction devised so far by human ratiocination”.

Interestingly, we can often adapt a known algorithm to use only cmpexch(x, y) instructions, which are functionally equivalent to the gates in the network above, and then translate it into a sorting network.

For a recursive example, let’s assume we already have a sorting network for n elements, and we want to build a sorting network for n+1 elements. One valid strategy would be to sort the n first elements with the network we already have, then insert the last element into its place, like so:

— Insertion sorting network

This strategy, as you probably have noticed, is analogous to that of insertion sort. The same transformation can be made with bubble sort, and perhaps surprisingly, the parallel versions of the networks for both algorithms are exactly the same!

But the real breakthrough is due to Ken Batcher, who devised a recursive merge routine using cmpexch instructions that made it possible to express merge sort as a sorting network. The result is far from trivial and best explained by Knuth. The Wikipedia article Batcher’s odd-even mergesort contains an implementation in Python. This algorithm uses O(n log²n) comparators for n elements, which is not optimal, but generally good enough, especially since faster alternatives hide large constants behind the asymptotic notation and are mainly of academic interest.

Finally, given a sorting network, how can we prove it is correct? Recall that the comparisons made by a sorting network are independent of the nature of the input, which may make it hard for us to argue it works. It would appear that we would need to test the network against all n! possible permutations of n input elements.

However, there is a better way. The simpler method uses the zero-one principle, quoted below from Sedgewick:

Zero-One Principle: If a nonadaptive algorithm produces sorted output when the inputs are all either 0 or 1, then it does so when the inputs are arbitrary keys.

The proof for the theorem above is quite simple. A proof by contradiction can be found here, and Knuth gives an analogous proof by contrapositive.

So we only really need to test the network using the 2 ⁿ possible strings of n bits, which is a large number, but nevertheless smaller than n!.

...keep reading at Sorting networks revisited

Highlights from PyCon US 2012 lecture videos (part 2)

2012-03-20T00:00:00-07:00

Time for the final round of talks from PyCon 2012!

In case you missed it, in the first of this series of posts I talked about an introductory lecture on PDB, the Python debugger, a cloud hosting 101 from a noob to noobs, and a two and a half-hour demystification of some often misunderstood behavior of the Python programming language.

Graph processing in Python

Van Lindberg presents a lecture on applied graph theory disguised as a Python lecture. But don’t think that’s bad at all!

The lecture consists of three studies on how graphs can model and help analyze the Python community and development process, with interesting results.

First, the python-dev mailing list is modeled using a directed graph in which there’s an edge between two people if one replied to a topic by the other. Using the concept of vertex centrality to model activity, the top three contributors to the mailing list are elicited. Interestingly, although the graph clearly shows a central core of active contributors, it contains no cliques, that is, no group of people always interact with each other.

Then, a graph is used to correlate people in the mailing list with their topics of interest. Finally, another graph tracks down the different licenses used in the Python code base.

I may be biased due to the fact that I happen to like graph theory, but I definitely enjoyed watching this. On the Python front, I learned a bit about NetworkX, an impressive package for graph manipulation and analysis.

Stepping through CPython

Larry Hastings gives an “architectural sightseeing” of the internals of CPython, the reference implementation of Python, which promises to get the audience started on the path to becoming a core developer.

As the presenter himself puts it, this talk is more about “running” than “stepping” through CPython due to lack of time. It glosses over several architectural aspects of CPython, from the directory structure in the source code to the bytecode interpreter, through the custom memory allocator and internal object protocols, though never with quite enough information to make them digestible for those with no previous contact with the implementation.

Even so, I’d recommend following this lecture closely while browsing the CPython source code for those interested in what happens under the hood. Just make sure you use ctags or similar to keep up with the fast pace in this lecture.

As a curiosity, did you know CPython uses gcc’s computed goto extension where available when interpreting bytecodes?

Fake it til you make it: unit testing patterns with mocks and fakes

An overview of several techinques and tools for unit testing Python code by Brian K. Jones. While aiming for an audience with intermediate familiarity with the programming language, this talk assumes little prior experience with testing techniques. Which is great — testing and validation are areas I (still) know little about.

This turned out to be a very enlightening talk in which you may learn a few general guidelines for writing testable code, such as creating wrapper classes around external resources and libraries so they can be easily mocked during testing, and also see a couple of real life applications of that knowledge.

An interesting Python-specific technique I gathered from this talk is resorting to monkey patching, that is, run-time modification of live objects and classes, which is made very easy by the language, to mock specific subsystems while testing. This, if used sparingly, turns out to be a Pythonic replacement for dependency injection.

The talk also promotes the use of some very interesting tools, such as coverage for measuring code coverage, and mock, which is even making its way into the standard library in the Python 3 series as I write this.

And that’s it for now! There are, of course, several other lectures available (check pyvideo.org for that), and I’ll probably watch a few more some time in the future. Let me know if I missed an important one!

...keep reading at Highlights from PyCon US 2012 lecture videos (part 2)

Highlights from PyCon US 2012 lecture videos (part 1)

2012-03-11T00:00:00-08:00

This year’s PyCon US, the “largest annual gathering for the community using and developing the open-source Python programming language”, as the event’s website puts it, featured a wide range of interesting presentations directed to people with varying degrees of familiarity with language and associated technologies.

The videos for the lectures were released online a couple of days ago, so having not been able to witness the presentations first-hand, I decided to watch and comment some of those that caught my attention. And it was definitely worthwile!

I’ll split this into a series of posts, as there is quite a bit of material to be covered.

Let’s start with a couple of introductory lectures and an intermediate one. The lecture titles are links to the actual videos, be sure to check them out.

Introduction to PDB

Chris McDonough gives an introductory talk on PDB, the Python Debugger.

Lots of people aren’t really used to the idea of a specialized tool for debugging Python code: it sometimes just sounds a lot easier to sprinkle log or print statements all around the code to track its execution and state.

While this approach can be perfectly valid in some scenarios, a debugger generally provides much more powerful tools that can make a difference when tackling a really elusive bug.

This lecture teaches the basics of PDB, a resourceful Python debugger that shares many features with good old GDB, the GNU Debugger. It would be very nice for every Python programmer to at least learn the basics of PDB, if only to consider sprinkling the following snippet instead of print statements in their code:

# some safe code here

# load PDB prompt
# let's proceed carefully
import pdb
pdb.set_trace()

# some dangerous code here

A noob speaks to noobs: your first site in the cloud

Katie Cuningham, a Python and Django developer with zero sysadmin background, tells the tale of her first incursions into hosting a few personal sites in the cloud.

An entertaining talk by a charismatic speaker, and although I didn’t find her advice to be really essential, it was nice to learn about her use of Fabric, a promising project I hadn’t known before.

Python epiphanies

Stuart Williams gives a lecture targeted at software developers with a background in statically typed languages and some experience with Python aiming to clarify some common pitfalls and seemingly counterintuitive behavior in the language.

Even though it is targeted at intermediate level developers, beginners might benefit from the fact that the lecture starts slowly and builds on basic concepts like the oppositions between variables (as in little boxes that contain values) versus namespaces (which map names to objects), global versus local scope and mutable versus immutable objects.

The second half of the lecture picks up speed and briefly visits decorators before plunging into (meta) classes. The distinctions between classes and objects and bound and unbound methods are carefully explained with plenty of examples. Definitely worth watching if you’re not entirely sure why the following snippet prints 4 and 6 before crashing with a TypeError:

class Num(object):
    def __init__(self, amount):
        self.amount = amount

    def double(self):
        return self.amount * 2

def triplicate(self):
    return self.amount * 3

num2 = Num(2)
print num2.double()

Num.double = triplicate
print num2.double()

num2.double = triplicate
print num2.double()

Finally, the talk discusses iterators, generators and the mechanics behind for loops (consider watching if the name “StopIteration” doesn’t ring a bell) and closes with two cute examples of how dicts may be used to blur the line between code and data.

And that’s it for the first part. I still have a few lectures in the watch queue, which is certainly enough for a couple more installments of this series, so check back often!

...keep reading at Highlights from PyCon US 2012 lecture videos (part 1)

Weighty Hamming weights

2012-02-05T00:00:00-08:00

I recently got myself a copy of Jon Bentley’s Programming Pearls. It’s been quite an instructive reading, and I’ve been trying hard to pace myself, understanding the examples and working out some of the problems at the end of each Column without rushing through the pages.

Having just finished Part II of the book, which overviews several interesting principles for improving the performance of a program, from choice of algorithms and data structures to low-level code tuning, I thought I’d try to apply some of these concepts in a thorough resolution of a proposed problem.

The problem in question is the 7th of Column 9, which asks:

Given a very long sequence (say, billions or trillions) of bytes, how would you efficiently count the total number of one bits? (That is, how many bits are turned on in the entire sequence?)

The count of one bits in a binary string is known as its Hamming weight, so the problem could also be stated as requiring the Hamming weight of a big string of bits.

The Hamming weight of a binary string may play an important role in applications in which each bit encodes information about a distinct entity, and one wishes to count entities with a given property. For instance, given a filesystem bitmap which encodes used and free blocks as zero and one bits, respectively, the Hamming weight can be used to calculate the amount of free disk space.

There are known and obscure techniques for making such calculation. In particular, the thread Best algorithm to count the number of set bits in a 32-bit integer on Stack Overflow abounds with solutions for strings of 32 bits.

I approached the problem by first creating a 2.8Gib file of random bits to serve as input for my algorithms. Then, I devised a solution using a lookup table of bit counts so as to avoid having to iterate through each of the input bits and maybe take advantage of caching for the lookup table. This is the resulting C code:

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>

// A lookup table, maps char values to the number of one bits in them
const unsigned char v2b[256] = {
    0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4,
    1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
    1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
    1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
    3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
    1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5,
    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
    3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
    2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6,
    3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
    3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7,
    4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8,
};

int main(int argc, char **argv) {
    int fd;
    register int i;
    unsigned long rd, sum = 0;
    unsigned char buf[BUFSIZE];

    fd = open(argv[1], O_RDONLY);
    while ((rd = read(fd, buf, BUFSIZE)) > 0) {
        for (i = 0; i < rd; i++) sum += v2b[buf[i]];
    }

    close(fd);
    printf("total bits set: %lu
        
            
            

            ", sum);
    return 0;
}

Basically, lines 7 to 24 define a precomputed table mapping each possible value of a byte to its Hamming weight. I then read the input file in chunks into a buffer, and for each byte in each chunk, look up its Hamming weight in the table and accumulate that value into a variable. Notice how the chunk size (BUFSIZE) is intentionally left unefined in the code — I’ll plug the value in at compilation time.

For comparison purposes, I also implemented Brian Kernighan’s very elegant bit-clearing method for achieving the same purpose in the inner loop, resulting in the following code:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>

int main(int argc, char **argv) {
    int fd;
    register int i;
    unsigned long rd, sum = 0;
    unsigned char buf[BUFSIZ];

    fd = open(argv[1], O_RDONLY);
    while ((rd = read(fd, &buf, BUFSIZ)) > 0) {
        for (i = 0; i < rd; i++) {
            for (; buf[i]; sum++)
                buf[i] &= (buf[i]-1);
        }
    }

    close(fd);
    printf("total bits set: %lu
        
            
            

            ", sum);
    return 0;
}

This method iterates through each byte of the input, and clears the less significant bit in it (with the clever expression b &= b – 1) as long as the byte’s value is not zero. The count of how many bits were cleared in that byte is of course its Hamming weight.

Needless to say, both programs compute the exact same amount of one bits in my data sample. But let’s have a look at their performances.

One important lesson from Column 6 of the book is that performance can often be achieved by tuning the program at several different design levels, from high level algorithmic choices down to system-dependent code tweaks and hardware upgrades.

For this problem, both programs above are in and of themselves algorithmic improvements over the naive approach of looping through all input bits. So in order to open a new front of attack, I chose to experiment with two lower-level tuning parameters: loop unrolling for the inner bit-counting loop, and input chunk size.

I compiled and ran the first program above using chunk sizes of 512, 1024, 4096 (my filesystem’s block size), 8192 (my system’s value for BUFSIZ), 12288 and 16384 bytes, with and without loop unrolling. For each test, the running time was measured three times with the shell built-in time command, the first of which on cold cache, and the other two on warm cache.

The tests were conducted on a Core I3 380M notebook with 4GB RAM and using gcc 4.5.3. The compilations were made using -O2 and the loop unrolling was made with the -funroll-loops compilation parameter. I didn’t tweak the chunk size of the second program, but I did measure its running time with and without loop unrolling.

Let’s see the results for the lookup table method:

Chunk size (bytes)

Loop unrolling

Times (seconds, real/user/sys)

Yes

2m52.240s / 0m11.791s / 2m38.049s

2m47.847s / 0m11.893s / 2m35.125s

2m47.048s / 0m11.934s / 2m34.432s

3m30.076s / 0m12.174s / 3m15.364s

3m19.436s / 0m12.514s / 3m5.905s

3m18.667s / 0m12.576s / 3m5.227s

512

Yes

0m29.340s / 0m7.138s / 0m9.451s

0m8.360s / 0m2.750s / 0m3.226s

0m6.285s / 0m2.631s / 0m3.015s

0m29.654s / 0m7.075s / 0m10.383s

0m5.697s / 0m2.502s / 0m3.184s

0m5.636s / 0m2.559s / 0m3.067s

1024

Yes

0m37.643s / 0m5.719s / 0m4.975s

0m4.230s / 0m2.517s / 0m1.706s

0m4.162s / 0m2.475s / 0m1.680s

0m31.443s / 0m6.713s / 0m6.254s

0m4.304s / 0m2.444s / 0m1.853s

0m4.239s / 0m2.512s / 0m1.719s

4096

Yes

0m30.454s / 0m6.560s / 0m3.302s

0m3.323s / 0m2.420s / 0m0.897s

0m3.271s / 0m2.415s / 0m0.849s

0m30.030s / 0m6.828s / 0m3.726s

0m3.361s / 0m2.388s / 0m0.966s

0m3.302s / 0m2.398s / 0m0.899s

8192

Yes

0m29.580s / 0m6.805s / 0m2.963s

0m3.193s / 0m2.364s / 0m0.822s

0m3.124s / 0m2.389s / 0m0.729s

0m29.372s / 0m6.784s / 0m3.044s

0m3.212s / 0m2.430s / 0m0.775s

0m3.144s / 0m2.410s / 0m0.727s

12288

Yes

0m30.662s / 0m6.507s / 0m2.830s

0m3.168s / 0m2.412s / 0m0.750s

0m3.096s / 0m2.406s / 0m0.684s

0m30.096s / 0m6.564s / 0m2.961s

0m3.149s / 0m2.430s / 0m0.713s

0m3.091s / 0m2.422s / 0m0.662s

16384

Yes

0m29.446s / 0m6.863s / 0m2.806s

0m3.145s / 0m2.380s / 0m0.758s

0m3.076s / 0m2.428s / 0m0.642s

0m30.313s / 0m6.611s / 0m2.930s

0m3.136s / 0m2.409s / 0m0.721s

0m3.085s / 0m2.464s / 0m0.614s

For every chunk size and loop unrolling setting, the table above contains three timing values, the top one being on cold cache and the other two on warm cache.

The running times were mostly the same for all cold cache runs regardless of chunk size, with the real (wall clock) time being much greater than the sum of user and system time. This indicates that the program was largely I/O bound in these runs, which is entirely expected and rules out the effects of the other optimizations.

The other runs show that the timings decreases slowly but surely as the chunk size increases. While I did expect to reach optimal performance around 4096 or maybe 8192 bytes, the fact that the run times continued to decrease after that surprised me a bit. However, the timings for large chunk sizes are pretty close to one another, so the apparent decrease might be just due to system noise.

Loop unrolling did not seem to affect the running times consistently. Although it was enabled on the best run overall (3.076s with chunk size of 16384 bytes), the runner-up (3.085s with the same chunk size) is only 0.009s behind.

Finally, for comparison purposes, I timed the second program above, which employs the bit-clearing method and uses chunks of size BUFSIZ (8192 for me), compiled with loop unrolling, and got the following results:

Chunk size (bytes)

Loop unrolling

Times (seconds, real/user/sys)

8192

Yes

0m47.605s / 0m44.523s / 0m1.616s

0m45.788s / 0m44.079s / 0m1.070s

0m45.554s / 0m44.077s / 0m0.948s

Wow! This method is almost 15 times slower, and spends most of its time in user space — it’s CPU-bound even in cold cache!

The lookup table method is a lot faster, likely for making good use of the system’s caching mechanism and relying on fast array indexing for computing the Hamming weights of individual bytes.

Update (09/02/2012): Experimenting with different data types for the bit buffer also gives interesting results. Replacing the array of unsigned chars with an array of unsigned longs (whose size matches the word size for my machine) reduces the timings for the bit-counting method to 14.425s on warm cache, of which 13.706s are spent in user space.

...keep reading at Weighty Hamming weights

JSON parsing with Python coroutines

2011-12-29T00:00:00-08:00

A while ago, I read a blog post by Python developer Eli Bendersky on Co-routines as an alternative to state machines, in which he presents evidence to support a very interesting observation quoted below:

Co-routines are to state machines what recursion is to stacks.

which is to say, state machines can be expressed as concisely and elegantly through the use of coroutines as recursive algorithms through recursive functions (as opposed to using an explicit stack).

Now, coroutines you say?

To quote from the Wikipedia entry:

Coroutines are computer program components that generalize subroutines to allow multiple entry points for suspending and resuming execution at certain locations.

Which, in simpler terms, means a coroutine is like a function from which you may yield and resume control at arbitrary instructions — there is no longer a single entry point to the procedure expressed as a coroutine.

The emphasized statement above is probably the main argument in favor of using coroutines to implement state machines, as it makes restoring the coroutine’s execution context (that is, the state in a state machine) when new input arrives much simpler, without requiring boilerplate control structures such as if in state 1: … elif in state 2 …

While coroutines don’t seem to be a recent addition to Python (support for them has apparently been under discussion since 2005, when PEP 342, which describes the implementation, was written), looks like they haven’t achieved widespread usage, which is my way of kindly stating I had never heard of them until very recently.

I won’t really go into details with regard to the syntax for coroutines, as their use is quite well described in Eli Bendersky’s post and references, and their abuse is cleverly documented by David Beazley’s presentation Generator Tricks for Systems Programmers.

So over the last couple of days I’ve been experimenting with the ideas above in a pet project: a simple JSON parser using state machines implemented as coroutines. I must remark that this parser has little to no actual utility, as it may still be buggy and is much, much less robust and efficient than the json module in Python’s standard library. Even so, I hope it might be useful for illustration and educational purposes.

Basically I converted the railroad diagrams in the JSON website into state machines that operate on one character at a time from the input, and made a familiar loads() function on top of those to simplify parsing from a string.

— Railroad diagram for JSON strings

As an appetite whetter, let’s have a look at the string parser corresponding to the diagram above:

@coroutine
def string_fsm():
    '''
    A coroutine implementing a finite state machine for parsing JSON strings.
    Accepts the string one character at a time, and yields NOT_PARSED_YET until
    the string has been successfully parsed.
    Once the JSON string has been parsed, yields the corresponding Python
    string. The coroutine can't be used after that.
    May raise JSONParseError on malformed strings.
    Expects data with no leading or trailing whitespace (i.e., the first and
    last input characters should be ").
    '''

    value = []

    c = (yield NOT_PARSED_YET)
    if c != '"':
        raise JSONParseError("JSON strings must start with a quote")

    while True:
        c = (yield NOT_PARSED_YET)
        if c == '"':
            # end of string
            break
        elif c == '\\':
            c = (yield NOT_PARSED_YET)
            if c == 'u':
                # unicode 4-digit hexadecimal
                hexval = ""
                for i in range(4):
                    hexval += (yield NOT_PARSED_YET)

                value.append(unichr(int(hexval, 16)))
            elif c == 'b': value.append('\b')
            elif c == 'f': value.append('\f')
            elif c == 'n': value.append('
        
            
            

            ')
            elif c == 'r': value.append('\r')
            elif c == 't': value.append('\t')
            elif c in ('"', '\\', '/'): value.append(c)
            else: raise JSONParseError("Invalid escape character")
        else:
            value.append(c)

    yield ''.join(value)

As you can see, there is a straightforward correspondence between the diagram and the code, with the yield expressions bringing in the characters that govern state transitions.

Writing this parser was a very pleasant experience that helped get my head around a convoluted construct. While I must concede that the parsing state machines do look pleasingly concise, I also felt that the use of coroutines also contributed to the complexity of the code in some aspects. For one, debugging resumable functions with multiple entry points certainly takes some time to get used to.

In case anyone wants to try it, the code is in my Github page, under the name jsonfsm. The repository contains the full code for the module and a few tests, some of which were borrowed from xmms2 (thanks!). Check it out!

...keep reading at JSON parsing with Python coroutines

Fun with sorting networks

2011-09-02T00:00:00-07:00

A pair of interesting threads about sorting algorithms recently surfaced on Stack Overflow, bringing to discussion an exciting concept (which was entirely unknown by me): sorting networks.

The first thread poses a straightforward question: what is the fastest way to sort 6 integers? The question calls for optimized code to sort an array of integers whose size is fixed and known in advance to be 6.

Experience shows that we can often sort elements a lot faster if we have some extra information about them, such as their distribution or range of values. So intuitively, one might expect better (albeit maybe not asymptotically better) performance from a sorting algorithm tailored for fixed-size arrays when compared to a generic version, fit for any input size.

And indeed some very interesting techniques were elicited in the replies to that thread. However, one of them struck me for its simplicity and sheer speed: using a sorting network. This turned out to be faster than the generic solutions (such as insertion sort) that were posted as replies, according to the original poster’s tests.

A brief explanation on that solution: sorting networks are abstract representations of the comparisons and swaps used to sort a fixed number of integers. They are often schematized as circuits in which input wires carry the values through pairwise compare-and-swap gates.

— A simple sorting network for four elements

Sorting networks leverage the fact that, given a fixed number of integers, the comparisons needed to sort them are also fixed. That is, the same circuit for n integers can sort all inputs of that size just by comparing and swapping elements in a fixed order.

I must not have been the only one puzzled by these clever constructs, since a second thread on Stack Overflow soon wondered, why are sorting networks so fast? The answer appears to be a combination of two factors: one, sorting networks tend to make less comparisons than generic sorting algorithms, and two, the comparisons made by sorting networks are more easily parallelizable at CPU level.

So, in order to gain perspective on the power of that new sorting technique, I decided to run a few practical tests of my own.

The methodology is simple. I implemented the quicksort algorithm in three forms. The first is a plain and simple version, which can be found in most textbooks. The second uses the common optimization of resorting to insertion sort for small arrays, of sizes up to eight. And the third used sorting networks for small arrays, again for sizes eight and smaller.

I then ran tests for input arrays of ascending (that is, already sorted), descending (sorted in reverse), all-zeroes and random integers, of sizes doubling from 2^16 (65536) to 2^22 (4194304), and compared the average user CPU times of five runs as measured with time.

The codes were compiled with clang 2.9, using -O1 and -O2, and ran on a 64-bit Linux system, with a Core I3 380M processor.

The code I used for testing is available in my Github page, in the sorting-networks-test repository. Anyone is welcome to review and try it. I also included all raw test results in that repository.

For a short summary of my results, I did not notice much difference between the insertion sort and the sorting networks versions. In fact, the former slightly outperformed the latter in almost all tests. On the bright side, both were consistently (and considerably) faster than the textbook algorithm.

It may be the case that the speed-up provided by sorting networks is highly compiler/CPU dependent, so I’d like to see more test results on the matter before drawing a definitive conclusion. But the fact that this new technique was a match for the old and well-known quicksort + insertion sort duo in terms of performance makes it a nice addition to my toolbox anyway.

...keep reading at Fun with sorting networks

Further steps towards RPC

2011-08-17T00:00:00-07:00

In the last progress update, I talked about how we were moving to the client side of the Service Clients implementation.

Things have progressed quite a bit since then, but unfortunately that part of the project is nowhere near ready (nor will it be by the end of GSoC, which is right around the corner). Anyway, I thought I’d share a bit of what I’ve been working on during the past weeks of coding.

The lower level of what we have in mind for the client side is complete and somewhat solid. This means it is possible for clients to pass messages and replies around reliably, using the server as a bridge. The plan now is to use that capability to model the Service Client functionality.

As a demonstration/proof-of-concept, we decided to brush up the very first two service clients that were conceived, sumclient and sumservice, and add them to the xmms2-tutorials submodule as tut9 and tut10.

I’m currently working on a higher level layer on top of that, which is the actual modelling of remote procedure calls and introspection. As a general concept, we agreed on allowing service clients to define and expose three types of basic entities to other clients: namespaces, methods and broadcasts. This means offering a service will be a matter of defining one (or more (or nested)) namespaces containing methods and broadcasts which can be accessed by regular clients by sending a specially-formatted message to the service client.

My work on this last week of GSoC has been on the code that defines these entities, which (I think) started to take shape around this commit. That code hasn’t yet been reviewed and may soon suffer alterations.

With that out of the way, the remaining work roughly boils down to implementing the code that articulates these data structures to offer the Service Client functionality. Basically that code should be a callback to the MESSAGE broadcast (which carries client-to-client messages routed through the server) that reacts to predefined types of messages, thus being able to dispatch method calls, reply to introspection calls, record a subscription to a broadcast and so on.

...keep reading at Further steps towards RPC

Cyclical iterators in Python

2011-08-12T00:00:00-07:00

I recently came across an amazing technique for manipulating Python iterators in a recipe by Raymond Hettinger.

The technique consists in defining an iterator as a function of itself, thus making a cyclical definition. The recipe hinges on using itertools.tee to split the to-be iterator, which is forward-referenced in a function before it is built, and feeding one (or more) of the tee’d iterators into the other.

— The Ouroboros comes to mind

Bear with me, I’ll explain in a moment.

Cyclical iterators can be quite useful when defining infinite iterators for recursive sequences. The problem solved by the recipe (generating 5-smooth numbers) has this very recursive notion, in the sense that, given a set of these numbers, mutiplying each by 2, 3 and 5 yields a larger set of such numbers.

Or, to explore the inductive side of the problem, given an iterator for the first n 5-smooth numbers, you can define a larger iterator for 5-smooth numbers by exhausting the first one and multiplying each number it outputs by 2, 3 and 5 (and eliminating duplicates).

And that is exactly what the recipe does: starting with a seed value of 1 (the first 5-smooth number), it builds an iterator that swallows itself, multiplying each already-generated number by 2, 3 and 5, sorting increasingly and eliminating duplicates.

I came up with this different (simpler) example while getting my head around the technique, which I’ll explain in greater detail:

import operator
import itertools

def factorials():
    def delayed_output():
        for i in output:
            yield i

    result, feedback = itertools.tee(delayed_output())
    seeded = itertools.chain([1], feedback)
    output = itertools.imap(operator.mul,
                            seeded,
                            itertools.count(1))

    return result

if __name__ == "__main__":
    print list(itertools.islice(factorials(), 10))

This returns a factorial iterator, which outputs sequentially 1!, 2!, 3! and so on. The inductive nature of the problem presents itself more clearly this time: given an iterator for the first n factorials, we can exhaust it and multiply the last value it generated by n+1 to obtain an iterator for the first n+1 factorials.

Let’s walk by it one line at a time.

Lines 1 to 3 aren’t of much interest. They only make a few imports we’ll need and define the function factorials(), which returns the factorial iterator.

Things start to get interesting in lines 04 to 07 as we define a generator inside factorials() which simply iterates through some output variable and yields every item in it. This is a delayed reference to output: we don’t want to evaluate it right now because it points to nowhere (that is, it’s not defined), but we want to use it anyway. Hence, we wrap output in a generator, so that it only gets evaluated when the generator is being traversed.

Then on line 9 we split the iterator returned by delayed_output() in two: one result, which we will return from factorials(), and a feedback iterator, which we will feed into itself.

This is the picture we have so far:

— Diagram for line 9

Note that output still points to nowhere, and if we get it to point to feedback somehow, then we will be making feedback indirectly point to itself.

Also note that I represented output twice. This emphasizes that, once we tee’d the delayed reference to it, we can have result and feedback pointing to different parts of the iteration. More on that later.

In line 10 we seed the cyclical iterator with the first value, 1!, which is just 1. We do that by defining another iterator, seeded, which will first yield the seed value, then proceed with the values it finds in feedback. Here’s the diagram:

— Diagram for line 10

And finally, on line 11, we define yet another iterator, which grabs one value from seeded, our inductive factorial generator, and one value from the infinite sequence 1,2,3,4…, multiplies them and yields the result. This is where the iterator for n factorials becomes an iterator for n+1 factorials.

But wait, what is that iterator called? output! The Python bites its own tail as we close the cycle and make feedback point to itself indirectly. Check out the diagram:

— Diagram for line 11

And then we return result, which remained unchanged since line 09. It helps to think that result always points to one value before seeded. That is, every time a new factorial is requested from result, seeded moves one position ahead, on to the next factorial.

...keep reading at Cyclical iterators in Python

ec09bot, the time-waster Python IRC bot (and what it taught me)

2011-07-29T00:00:00-07:00

One of the things I like the most about vacation is having the time and disposition to code weird/silly/funny ideas, which I would never bother with if it weren’t for the fact that I’ve been having 16 free hours every day.

It is with that spirit that, once I learned that a few friends of mine (the geekiest, of course) had created an IRC channel, I came up with the idea of coding a bot.

Not a useful bot. Not even a smart bot. Just a funny bot.

To me, creating an IRC bot sounds like one of these things every hacker must absolutely do once, so the task was somewhat charmful. And as with almost every other pet project/prototype of mine, I decided to use Python for extra coolness.

After a little research, I came across python-irclib, which provides a framework for developing IRC bots. Yay, just what I needed, specially since I had spent most of the afternoon of the previous day just to get to know the IRC protocol and hand-code a vegetative-state bot that was pretty much only capable of keeping itself alive by replying to PINGs from the server. Lesson learned for code reuse.

The first command to be implemented was !batima, which makes the bot reply with a quote from Feira da Fruta, a classic trashy humor redub of an episode of the Batman series of the 1960s.

The implementation sounded pretty straightforward: given a list of such quotes, pick one at random and send it to the channel. The problem lied exactly in finding the list of quotes, which neither I nor any of my friends wanted to maintain.

Luckily over the years that redub has flocked a cult of fans, and there’s even a Twitter account dedicated to posting almost daily quotes from the video, all-caps for extra trashiness I presume. With some help from python-twitter, the rest was quite easy.

~~And along came !chuck, which yields a Chuck Norris Fact taken from another twitter account.~~ Update: The account in question seems to have been disabled. I’ll probably re-implement the command as a fortune command.

Then came the fortune-based commands, suggested by Ivan. What better repository of funny/interesting/nonsensic quotes than the one used by the fortune Unix command? By implementing the generic functionality of grabbing a fortune cookie and sending it to the channel, we soon had Calvin & Hobbes, Futurama and even BOFH quotes at our disposal.

And then there’s the cool hack used for !wikipedia, which accesses Wikipedia Over DNS using dnspython to grab article summaries from our favorite encyclopedia.

I tried to keep the general structure of the code very modular with regard to adding/removing commands. Basically, each command is implemented as a separate module, and registered to a dispatcher class which resides in the main file, ec09bot.py. The command registration code is somewhat facilitated by not having the main file import all command modules; instead, they are grouped in a single Python package. This design was somewhat inspired by the http package on newer Python versions.

All in all, between new Python modules and a few design techniques, I have learned quite a bit in doing this. It has been quite fun as well. The code of the bot is on Github under the MIT license, anyone is welcome to try it.

...keep reading at ec09bot, the time-waster Python IRC bot (and what it taught me)

On to the client side...

2011-06-26T00:00:00-07:00

Now that the server side of the Service Clients implementation has taken shape, it’s time to turn our attention to the design of the client side implementation, which is still somewhat immature.

The basic idea is that, now that the server is capable of dumbly passing messages and replies among clients, we can use that capability to model the actual Service Client functionality on the client-side. In other words, we want the clients to establish a protocol for remote procedure calls and introspection with the server, unaware of what is really going on, simply forwarding messages and replies back and forth between clients.

Our plan for the actual implementation will require, as expected, changes to libxmmsclient, but also the development of a new library, so far called libxmmsclient-sc. The new functionality added to libxmmsclient will merely facilitate message exchanging between clients, while the actual Service Client sugar will be laid on top of that in libxmmsclient-sc.

We already had some sketches for the client side of the implementation, most notably this one by nesciens, which gives a pretty good idea of where we are headed. Last week I published a sketch of my own, with an updated and slightly more concrete view of the implementation details now that we know what the server is capable of. It also contained a few more ideas.

While we have not yet throughly discussed these sketches and turned them into a more solid idea, they do provide a good basis we can work on refining over the next weeks.

One aspect of these sketches we had quite a lengthy discussion about was the alterations needed to accommodate the new message passing functionality in the asynchronous result mechanism in libxmmsclient. With valuable input from vdust, we reached a nice idea that I began implementing with this commit. It still needs some testing and polishing, which I will be doing in the next days.

...keep reading at On to the client side...

PyExt2lib

2011-06-10T00:00:00-07:00

I pushed a new project to Github about a week ago: PyExt2lib, Python bindings around the ext2fslib that comes with e2fsprogs.

The motivation for this project and the story behind it are quite interesting, so I though I would share them.

It all started with an Operating Systems class assignment in which the students (divided into pairs) were supposed to develop a tool to extract some information from filesystems, such as sizes of segments of contiguous free blocks and sizes of blocks allocated per inode, and display some of that information as histograms.

No one in the class had ever come close to doing anything related, so we all started looking into how to implement that. Some attempted to use the Virtual File System layer in the kernel to no avail, while others looked up how other tools, such as dumpe2fs, did it, and came up with the idea of using ext2lib.

We thus faced the challenge of learning the poorly documented API of a complex library we didn’t even know existed before, as a prerequisite to actually starting the assignment.

PyExt2lib, albeit not crucial to overcoming this problem (which required countless hours reading the e2fsprogs source code), provided a nice abstraction over the underlying library once we got the hang of it. That is, my partner and I could rapidly encapsulate chains of undocumented ext2lib API calls into Python code implementing the functionalities we actually needed, such as iterating over all free blocks.

Using Python also helped us with post-processing the information, a task that was made quite simple using the built-in types as data structures and matplotlib for the histograms.

I had already used Cython before in a patch for xmms2, liked it and wanted to learn more about it. That and the fact that GSoC may require me to use it again made it the tool of choice to develop PyExt2lib.

Even though PyExt2lib is incomplete and might never get finished, and contains some plain ugly hacks, I like to think it served its purpose and taught a few things in the process.

We did, after all, deliver the assignment in time, and aside from the bindings themselves, it boiled down to about 400 lines of Python that came together in a few days.

...keep reading at PyExt2lib

It's ALIVE!

2011-06-08T00:00:00-07:00

It’s been two weeks since I began coding for GSoC, and it’s time to share some of the progress we’ve made. For the anxious: the courier object, as described in the last post, is already crawling on its own. I have implemented and briefly tested both methods described in the wiki, as well as a few other features to ipc.c and GenIPC.

My first attempt at implementing the courier object was to expose some data structures from ipc.c to courier.c and let the latter manipulate them at will. After discussing this design with the community, we settled for hiding these data structures and only exposing through new functions the bare minimum necessary for the courier to operate.

That is how this commit came to be. Since some of the new functions are mere copies of code that already existed elsewhere in the same file, they could be used instead of that code. I should add “cleaning up ipc.c” to my TODO list.

At that point I was planning on not making the courier a xmms_object_t, and have it special-cased for commands and replies in ipc.c (just like the SIGNAL object). This works for the two methods described in the wiki, but would complicate things for the other methods we have in mind. So after discussing the issue with nesciens, we decided to adapt GenIPC to our needs.

This means making it possible to specify that the server should not automatically send replies to clients that call a server object’s method, and that it instead should leave that responsibility to the method itself.

This set of changes was made in a separate branch of my repository, which has now been merged into master. This is the most relevant commit, describing the changes. The other commits are fixes and cleanups both to new and old code.

With that issue out of the way, I integrated the courier object to GenIPC with these two commits. It should now be fairly easy to implement the remaining methods we have in mind. So far, we are planning on implementing a method that returns a list of ids of connected clients, and a method for clients to consult their own id.

Today I made a small breakthrough by writing a very silly but functional service client. In order to do that, I had to hack support for the courier object in libxmmsclient and main.c. I didn’t commit such ugly code, but it helped me test what I already had and, after a few hours of debugging, I had sumservice, the client that accepts messages containing a list of two integers from other clients, sums them and returns the result, and sumclient, the client that takes two integers from the command line and sends them for summation :)

Next steps: Aside from the remaining methods for the courier, I’m still unsure how to integrate this new object in main.c (i.e., its initialization and destruction). Also, the code written so far still needs revision and testing, as I may (as in, probably) have screwed up refcounting and other tricky parts.

See you soon!

...keep reading at It's ALIVE!

The coding begins!

2011-05-24T00:00:00-07:00

Yesterday was the official start of the GSoC program!

We had some nice ideas about my proposal during the Community Bonding Period and documented them in a dedicated wiki page. And now it’s time to put them to practice.

I started coding the server object described in the last section of the wiki page yesterday, and also made some progress today. I’ll have some code to show in the next few days.

And before I forget: last week I copied over the official xmms2 development repository to a repository of my own in Github, which is where my changes will end up when they are a little more mature.

I’ll also update the wiki page with the issues I encounter and necessary clarifications as I go.

...keep reading at The coding begins!

XChat notifications on GNOME 3

2011-05-07T00:00:00-07:00

Yesterday I decided to upgrade my Gentoo system to GNOME 3. It was not a painless process, although I must admit I expected worse.

One of the small annoyances that issued from the migration was that I could no longer rely on a blinking XChat tray icon to warn me of new IRC messages since, well, there’s no immediately visible tray. Messing with the alarm settings in XChat, I came across an option to display Notification Baloons with libnotify. Thinking it might be nice to have an occasional balloon popping up with the new message text, I enabled it.

It did work, but:

— So many messages!

For every message sent, not only a balloon would pop up and vanish after a few seconds, but also an icon would persist in the notification panel. The notification panel would shortly be cluttered by lots of useless icons. (Ok, so this screenshot is actually me reproducing the problem manually, and not XChat causing it, but you get the idea).

Looking up the problem a bit, I found out that it had happened before, and could be solved by simply spawning the notification balloon with the appropriate parameters (namely, with a new transient hint set).

So, by grepping the XChat 2.8.8 source code for notify-send, I eventually reached the file where the magic happens, which looked somewhat like this.

After tinkering a bit, I found out that XChat 2.8.8 first attempts to load libnotify dynamically in order to use notification balloons, and if that fails, it falls back to calling the notify-send utility. So adding the transient hint would require changes in two sections of the code.

And here’s the diff of my modifications. Basically I used notify_notification_set_hint in the code section that attempts to use libnotify and added the command line parameter described in the bug above to notify-send.

After applying this and compiling, notifications no longer leave icons in the panel, which is exactly what I wanted.

I eventually found out that due to some bugs present in my version of XChat, the libnotify code (but not the notify-send code) always fails even before my changes have a chance to run. Well, at least I didn’t break it any further.

Come to think of it, I should probably have tried the latest SVN version first, since it fixes several issues with that specific file. But I had my fun anyway, and will gladly upgrade when the next XChat version comes out.

...keep reading at XChat notifications on GNOME 3

5-minute battery monitor

2011-04-29T00:00:00-07:00

I was left with no AC adapter for my notebook for a few hours on Tuesday.

Having no battery monitor installed and being unable to install one, I was clueless about how much battery life I still had.

So I did the best I could at the time: ~~shut down the notebook~~ hacked together a very simple battery monitor by parsing the contents of /proc/acpi/battery/BAT0/state and /proc/acpi/battery/BAT0/info.

And here it is, too pointless for Github, and too useful for oblivion:

#!/bin/bash

while :; do
    now=$(awk '(NR == 5) {print $3}' /proc/acpi/battery/BAT0/state)
    total=$(awk '(NR == 3) {print $4}' /proc/acpi/battery/BAT0/info)

    echo $now $total | \
        awk '{ printf("\r%d/%d mWh (%.1f%%)", $1, $2, 100*$1/$2) }'

    sleep 2
done

A fun time-waster would be to try to convert it into a pure awk script, but that’s going to have to wait.

...keep reading at 5-minute battery monitor

So it begins...

2011-04-27T00:00:00-07:00

I have been accepted by xmms2 as a student for GSoC this year!

My proposal is to implement Service Clients: basically, clients that can offer functionalities to other clients by exposing methods for remote calls. This is actually a long-standing idea, and there’s even an incomplete implementation already.

It will be very interesting to see all sorts of crazy services popping up after I’m done with my project…

Stay tuned!

...keep reading at So it begins...