• Honey is made from pollen.

• Only honey bees spread pollen.

• If plants remain unpollenated for more than a week, they will shrivel and die.

• Fortunately, if you sprinkle pollen on the shriveled plants, they will immediately be restored to health and begin flowering.

• It's okay to exploit animals as long as they get a cut of the profits.

Isn't learning fun!

(Part 1 in a continuing series.)

"Leading isn't about being aggressive, it's about being assertive."

One of the new features in µC++ 5.5.0 that I'm most proud of has to do with our new support for asynchronous communication using futures. For those unfamiliar with futures, the basic idea is that a future is a value that “stands in” for a value that is being computed in some other thread. A thread (I'll call it the client) obtains a future from another thread (the server) in lieu of the actual value. The server computes the actual result while the client proceeds concurrently. When the client needs the real value, it accesses that value through the future.

Future_ISM<int> f = server.method();
// client does other work while value is computed
std::cout << f();   // use value, block until available

If the client doesn't need the actual value until after the server has finished computing it, then the client can grab the value and proceed immediately. Otherwise, the client blocks until the value is available.

Futures are neat and potentially quite useful, but that's not what I want to focus on right now. What's important about a future is that it's something that a thread might have to wait for — a blocking operation. And the interesting question is: what can you do if you have more than one potential blocking operation?

For instance, suppose a client thread has two futures, from two different servers. At some point the client can't proceed without one value or the other. But which value should it wait for? It might be able to do more work if it had either value, but it likely doesn't know which value will become available first. If it chooses to wait for value A, but value B comes ready first, then it's wasting time waiting for value A when it could be working with value B. What is needed is some way to wait for a variety of futures, continuing whenever one of them comes ready, and possibly doing different things depending on which future it is that came ready.

I'm not the first one to tackle this problem, but I've come up with a nice way of representing this sort of selection in C++. If you have two futures, f1 and f2, then in µC++ it's now possible to write:

orselect(f1)(f2);

which then will wait until either f1 or f2 is available before proceeding. To perform different actions in each case, attach functions or function-objects (functors) to each orselect clause:

orselect(f1, action1)(f2, action2);

To wait for both futures, there's andselect:

andselect(f1, action1)(f2, action2);

The advantage of andselect over waiting for each future in sequence is that the action corresponding to a future will be performed as the future comes ready, but the statement following the andselect won't be executed until all futures are available.

orselect and andselect are generalized in the obvious way for more than two futures, but they are also general in other respects. For instance, it's possible to build up ands and ors into arbitrary expressions:

orselect(andselect(f1)(f2))(f3);

This expression waits until either f3 is available, or until both f1 and f2 are available. Actions can be attached to any sub-expression.

Perhaps more fundamentally, this mechanism is general because it's generic: the arguments don't necessarily have to be futures. They could in principle be any sort of blocking construct: a message queue, or some sort of I/O completion token, or even a lock or condition variable. The target object must implement a three-method interface, since there needs to be cooperation between the object and the selectors to avoid race conditions. Assuming that the interface is implemented by all participating objects, the mechanism allows selection among utterly heterogeneous types:

orselect(future1)(msgqueue1)(andselect(future2)(msgqueue2));

In implementing this I decided at the outset that I wanted it to be maximally general but also perfectly type-safe, while also avoiding any dynamic memory allocation. As a result, the implementation makes heavy use of templates and isn't exactly easy on the eyes. For the strong-stomached, I present the source: uBaseSelector.h and uSelector.h. uBaseSelector.h contains the basic expression-tree class, while uSelector.h contains the actual orselect and andselect functions that build the expression trees.

The ten-cent tour goes like this: orselect (and andselect as well) creates a single-node expression tree from its argument. This expression tree is contained in a wrapper class that has an operator(), which creates a single-node expression tree from its argument as well, and then builds a new two node expression tree linking the two single nodes. In this way, a binary tree is built as a temporary object through the evaluation of the chain of operator()'s. To ensure maximum genericity, each node of the expression tree is templated on its argument type, which means that the type of the tree has a structure that reflects the structure of the tree itself. At the end of the expression the temporary object is destroyed, but its destructor evaluates the expression tree and will block until the condition it expresses is satisfied.

There's a lot more that I could say about the mechanism and its implementation, but I think I've said enough for now. We also implemented a similar selection mechanism that actually extends the language:

_Select(f1) {
    // action1
} or _Select(f2) {
    // action2
}

The advantage here is that you avoid the verbosity of C++ function objects by writing the code in-line. For more information about µC++ futures and _Select, see Chapter 3 of the µC++ Annotated Reference Manual. What makes andselect/orselect interesting, however, is that they can be implemented in standard C++ augmented with a minor amount of concurrency support (all it takes is a semaphore). Although their current implementation is tied to µC++ concurrency facilities, they could also be implemented using POSIX semaphores or using the proposed standard threading primitives.

Geoff Pullum on what not to correct.

A remarkable thing is happening in the Province of Ontario: the government has thrown the basic mechanism of our electoral system wide open to change. A group of citizens has been randomly selected, two from each riding, and they are traveling the province holding consultations in order to craft a proposal for a new electoral system. The proposal will be voted on in a referendum set to coincide with the provincial election next October 4.

They're going to be holding a public consultation in Waterloo, on 10 January 2007 from 7pm – 10pm, at the Ontario Early Years Centre Training Room, 425 University Ave. East. I'm told this is in the same plaza as Price Chopper. It's charming that they picked a place so hard to get to by public transportation.

The Citizens’ Assembly hasn't been getting a lot of press since they were commissioned, nor has there yet been much of an informed discussion of the different electoral systems that they have to choose from. On one level this is to be expected (it's even arguably rational): it's hard enough to get people to care about an ordinary election, with real people and policies at stake; how hard will it be to inspire interest in a meta-election, where the alternatives (and their implications and consequences) are only really familiar to a self-selected population of election-system geeks and small-party advocates? On the other hand, the choice of system to be proposed by the Assembly, and the result of the referendum, will have a much larger effect on government than any single election, since they will affect every election henceforth, influencing the composition of the house and the nature of government.

The traditional justification for moving from the current system is that it discriminates against smaller parties and the people who vote for them (read Coyne for a nice summary). Of course, increasing the representation of smaller parties decreases the representation of larger parties, leading to more coalition and minority governments. As a result, the net effect of a more proportional and “fair” voting system is that government passes less legislation, and that the legislation that does get passed is the product of negotiation and compromise among multiple parties. Therefore, you ought to support a more proportional voting system if you don't feel that one of the large parties is adequately representing your interests, and you feel that our governments in general have too much freedom to pass legislation.

(I think there's a reasonably good argument to be made that libertarians ought to support proportional representation as a consequence of that last point, under the theory that the chance of government deciding to make itself smaller under any system is small, but at least minority governements are less capable of expanding government.)

I'm not sure which of the many alternative models to support, but I do believe that it's possible to do much better (both in terms of representation and in terms of legislative outcomes) than the current system. If you support change, it's important to participate in the process to increase its legitimacy. Furthermore, whether you support change or not, it's important to be informed about the alternatives and aware of the activities and progress of the Citizens' Assembly.

Fair Vote Canada, an organization dedicated to electoral reform, provides a good set of links.

Ken Jennings:

Which sort of reminds me of why I'm relieved not to be a computer programmer anymore. You quickly get tired of, "Oh, you're a computer programmer? Perfect -- you can tell me how to fix my modem/get rid of these pop-ups/get Windows to recognize my scanner/et cetera." Um, no. No, I probably can't. This is worse than asking a doctor at a party why your feet hurt. This is like asking a dentist at a party why your feet hurt.

Boy, does that ring a bell.

I don't feel the same sort of outrage as Ken does, because I do have a fair bit of experience with the trials of tribulations of hardware and software configuration, and I don't usually mind helping people out if I can. But that's not my job, that's just something I've picked up over the years, and there are plenty in this field who have never bothered. I've only rarely been able to explain that to people though, partly because programming is a sufficiently strange and abstract discipline that those who don't do it often have a difficult time mentally latching on to any part of my explanation. Hardware and software maintenance is something that they can latch on to because it's something that touches their lives.

According to cat-world.com.au:

Around 2-15% of the worlds population is allergic to cats and 1/3 of these have a cat in their home.

I wonder how that 1/3 compares to the proportion of the entire population that owns cats.

As a cat-owning allergy-sufferer myself, I can understand some of the motives that inspire people to keep cats in spite of their suffering. Yet I'm surprised that there are so many people who are undeterred by the obvious negative consequences.

(In the spirit of Friday I would post a picture of Tony and Cleo, but I don't have any pictures on my work machine. If I remember tonight I'll upload one.)

The popular-science book is an important but dangerous form. At its best, it translates important scientific theories into a form that is understandable and relevant to non-scientists. At its worst, it misrepresents the science in order to further hidden aims of the authors, whether for the sake of fame or to influence public policy.

For non-scientists, it is very difficult to know whether a particular book is of the first kind or the second. While any good popular-science book will contain many references and footnotes connecting the claims made in the book with peer-reviewed supporting research, it's usually so much effort for non-scientists to verify these references that the claims are often taken at face value, assuming good faith on the part of the author. But references can't be judged on their existence or quantity alone: it's also necessary to ensure that they are relevant to the claim being made, that in fact they support the claim being made, and that limitations of the research (e.g., small sample sizes) are acknowledged. However, to do this properly is a time-consuming process that requires a certain understanding of the particular branch of science involved, as well as a good understanding of the methods of science in general. As a result, deep analyses of popular science works are rare.

This is why I want to call attention here to the work of University of Pennsylvania linguist Mark Liberman, founder of the fascinating blog Language Log, in analyzing the claims made by Louann Brizendine, in her book The Female Brain, and by Leonard Sax, in his book Why Gender Matters: What parents and teachers need to know about the emerging science of sex differences. A good starting point is one of his recent pieces, "The superior cunning of women", which serves as an index for his previous writing on the subject. Liberman considers many of the claims put forward by Brizendine and Sax, particularly but not exclusively related to linguistics, and finds many, if not most of the references failing to support the respective authors' conclusions. This is key, because the underlying science represents the foundation for an expansive series of arguments that attempts to influence the treatment of children in profound ways. If the foundation is rotten, the arguments carry no weight, yet I have seen a number of credulous reviews and other mentions of these books in the press.

I haven't read the books, and I don't have any particular position to support on the underlying question of the biological nature of sex differences, but Liberman's articles have caused me serious concern, especially considering the warm reception these books have received. Consider this article an attempt to do my part to ensure that these books get the scrutiny that they deserve.

I figured we could use a splash of color around here. This is from my very own garden, a nice present from the woman who used to live here. "Perhaps not the world’s most refined flower" suggests Tim Bray of the California variety, and it's true more so of this breed, but I don't care. Click the image for the big version.

I must Create a System, or be enslav'd by another Mans
I will not Reason & Compare: my business is to Create

Wm. Blake, JERUSALEM: The Emanation of The Giant Albion (1804)

I've just had the same problem that I originally had with KB898461, but now with the updates KB911564 and KB911565. The same fix (moving aside C:\WINDOWS\system32\spupdsvc.exe) works here too, although you have to move the file, install one of the updates, move the file again, install the other update. Perhaps its time to seriously consider reinstalling.

Malcom Gladwell: "On a list of two hundred and forty cities in the United States with a population of a hundred thousand or more, New York City now ranks two hundred-and-twenty-second in crime, down near the bottom with Fontana, California, and Port St. Lucie, Florida."

I assume he means "incidents of crime per capita." I wonder if NYC will ever overcome its violent and dangerous image. Certainly American television tends to dwell on its sordid side.

Gladwell's article is really good, and it's not (really) about NYC.

To Kill a Mockingbird relates the Cult of Childhood to the Negro Problem with disastrous results. Before the intellectual confusion of the project is considered, it should be noted that this is not much of a movie even by purely formal standards.

That's Andrew Sarris of the Village Voice, 7 March 1963 (source).

Eric Sink has, as usual, some insightful observations about the nature of software in his latest article, Yours, Mine and Ours. His article contrasts MeWare (software that you write only for yourself), ThemWare (software you write for others), and UsWare (software that you write for you and for others). It's an important distinction, and a good article. But when he says:

So let's face it -- MeWare isn't actually very interesting. Aside from the occasional outlier like myself, nobody is developing MeWare. All of us are developing software on behalf of other people. Whether your software is commercial or open source, it's all about the users.

I think he underestimates the colossal self-absorption of hobbyist programmers. (Insert smiley face here.) After all, Eric Raymond's first lesson is: "Every good work of software starts by scratching a developer's personal itch." Of course, I suspect Messrs. Sink and Raymond would disagree on what makes a "good work of software"; I suspect that Sink would start by identifying a niche market. Sink doesn't seem to consider his own personal MeWare to be good, merely useful. This, it's important to realize, is a subjective distinction.

A quick scan of freshmeat.net shows a wide array of MeWare. Of course, the authors don't intend it to be MeWare (or presumably they wouldn't be advertising it in freshmeat), but it's effectively MeWare, since they haven't made a serious attempt to cross the canyon that separates them from other potential users. My feeling is that this just scratches the surface of MeWare. I think Raymond goes too far to claim that every good work of software has such a source, but I do suspect that most of the software that doesn't come from big corporations or research institutions descends ultimately from itch-scratching.

For me, the MeWare lifecycle runs something like this:

1. Identify a problem in my life that a computer could help with. Probably I'm already using the computer in some way to deal with this problem, but not very effectively.

2. Dream up the ideal solution. Usually I'm already thinking in pseudocode at this point, although I try to repress this.

3. Explore the existing solutions. Typically this involves scrutinizing mature programs that I've heard about through some other source, although I usually check freshmeat for the sake of completeness.

4. Reject the existing solutions. Usually I rule out anything I would have to pay for early on, since by this point I've already convinced myself that I could code it myself if I had to (and because I tend to under-value my time relative to my money). The rest is usually rejected because it doesn't properly support the sorts of interactions that I see myself doing (this is why Eric Sink abandoned CityDesk). In the worst case, I've already thought up some elegant design in step 2, and I'm sufficiently worked up about it that my exploration in step 3 is hardly even a good-faith exercise.

5. Code away!

I sometimes feel a little silly about this. Most people don't write their own software. The basic tasks that I need to solve (e.g., calendar, address book, money management) have been solved well, and those solutions have been used by many different people. Why can't I be satisfied with what came before?

Well, to use Raymond's word, because it itches. This is another reason why MeWare exists primarily in the world of software developers: non-developers are conditioned from the beginning to accept software quirks. And "quirks" is really too friendly a term: I mean not only outright bugs, but also cases where the software doesn't support the user's approach to doing work. Very quickly users train themselves to adapt to the computer, because it's so much easier than adapting the computer to their work. But I refuse to accept this, because I'm empowered to change the computer.

Brian Kernighan is fond of saying: "There are really only two problems in computing: machines are too hard to use, and they are too hard to program." Whenever I think about that, I wonder how far apart those two problems really are. Right now, the software development model fundamentally involves a "priest class" of software designers and developers who are the only ones capable of channeling the raw power of the computer into a form that's suitable for the masses. However, this inevitably puts a lot of stress on the user/developer interface. The user is squeezed into one-size-fits-all interactions, or given rigid axes of customization. Do I tell you how to organize your filing cabinet, or what order to do your chores in? No, but if I'm a ThemWare or UsWare developer I'm forced to make similar choices for my users all the time.

I say "forced" because my point is not to lay this all at the feet of developers; most of us are making the best of a bad situation. My point is that MeWare, far from being irrelevant, is the future. Probably the distant future: MeWare implies absolute power and flexibility, at a cost of what, for most people, is brain-crushing complexity. But the tools are getting better: as Sink himself points out, better tools means more MeWare. I think we're going to see a long, slow convergence of the roles of user and programmer. It may take decades, or even longer, but it will be interesting to watch.

Ok, the CBC lockout has officially ceased to amuse me. The morning hosts are terrible, the news is hardly worth listening to, and the music, while often good, is not usually identified in any way. (Can someone please tell me what song Radio One was playing at 22:30 on 25 August?) The constant stream of repeats is becoming irritating as well: yes, the Massey lectures are interesting; no, thank you, I've heard them twice already.

So I hereby revoke my previous approval of the situation: it's time to get this resolved. On the other hand, I hope the CBC producers take some inspiration from the surprising successes of "Management Radio" — sometimes less is more.

Apply this script to the output of gcc -E. The output is in the form of grep -n, but with the lines properly attributed to the original file. For example:

plg2=; g++ -E notify.cc | pregrep strcpy
/usr/include/iso/string_iso.h:66:extern char *strcpy(char *, const char *);
/fsys1/.software/arch/gcc-3.3/include/c++/3.3.4/cstring:81:  using ::strcpy;
/fsys1/.software/arch/gcc-3.3/include/c++/3.3.4/sparc-sun-solaris2.8/bits/c++locale.h:63:        strcpy(__sav, __old);
/fsys1/.software/arch/gcc-3.3/include/c++/3.3.4/bits/locale_classes.h:314:              strcpy(_M_tmp_names[__i], "*");
/fsys1/.software/arch/gcc-3.3/include/c++/3.3.4/sparc-sun-solaris2.8/bits/time_members.h:51:      strcpy(_M_name_timepunct, __s);

If you're looking for a #define, use the -dD option to gcc, which leaves the #define directives in the preprocessor output:

plg2=; g++ -E -dD notify.cc | pregrep assert
/usr/include/assert.h:11:#pragma ident "@(#)assert.h    1.9     92/07/14 SMI" 
/usr/include/assert.h:18:extern void __assert(const char *, const char *, int);
/usr/include/assert.h:45:#define assert(EX) (void)((EX) || (__assert(#EX, __FILE__, __LINE__), 0))

Here's the script. I wrote it on Solaris, so it uses nawk; substitute gawk or mawk or any awk that has gsub.

#!/bin/sh

cmd=`basename $0`

if test "$1" = ""
then
    echo "usage: $cmd pattern [files...]" 1>&2
    exit 1
fi

pat=$1
shift

nawk '

/^# / {
    file=$3;
    gsub( "\"", "", file );
    NR = $2 - 1;
    next;
}

$0 ~ pat {
    print file ":" NR ":" $0;
}

' pat="$pat" "$@"

[Updated 30 August 2005 to add some useful quoting.]

I'm really enjoying the CBC lockout programming. They're playing a lot of really good music that I wouldn't otherwise hear. Why can't I have a radio station like that all the time? (Tyler Cowen explains.) Really what I need to do is to put Internet radio, or the cable digital music, into more rooms of my house. But good old FM is so cheap and easy.

It looks like Doc Palmer agrees.

So here's an attempt to provide a source for people who, like me, have installation issues with KB898461. In particular, Automatic Updates wouldn't install this update; downloading it directly from the KB site was no help either — after accepting the license, and a small amount of grinding, it would display a message box saying "KB898461 Setup canceled". Not that I canceled it; it was clearly some internal error bubbling up into an erroneous and unhelpful message.

Internet searches over the past month (the update was released at the end of June) have turned up very little: most people don't have the problem, and those who do generally don't have a clue what to do about it. I found one report today from someone who used system restore to go back to before he tried installing it the first time, which solved things for him. But I don't have restore points that far back.

What I did discover was the log file C:\Windows\KB898461.log. It contained a section for each time I had tried to install the update; most of them ended with the lines:

18.656: Failed To Move File C:\WINDOWS\system32\spupdsvc.exe error = 0x11
18.656: Failed to copy spupdsvc.exe to system32
18.887: DeRegistering the Uninstall Program -> KB898461, 0
18.887: KB898461 Setup canceled.
242.268: Message displayed to the user: KB898461 Setup canceled.
242.268: User Input: OK
242.278: Update.exe extended error code = 0xf00d
242.278: Update.exe return code was masked to 0x643 for MSI custom action compliance.

On a whim, I tried moving aside (i.e., renaming) C:\WINDOWS\system32\spupdsvc.exe, and installing the update again. This worked, and installed a new version of spupdsvc.exe. I don't have any explanation as to why this works, or what the problem really is.

Every once in a while I (or more likely, my boss Peter) will get an e-mail that says "Cforall looks really neat! Where's the compiler?" The answer is that it languishes in an unreleaseable state. This fact, and the reasons behind it, are a real disappointment to me.

When I wrote my Master's thesis, it was based on a working implementation of the algorithms described therein, which I had been working on (along with Peter and Rodolfo) for a year-and-a-half. This was in the form of a translator, a program that takes as input Cforall code and outputs plain C code to be compiled by an ordinary C compiler. While it pleased me to be able to submit real code samples to my translator and have it do reasonably intelligent things with them, the translator has two unfortunate properties: it is unusably slow, and certain parts of its source code (specifically the interesting parts, the implementation of the algorithms described in the thesis) are inscrutable and nearly unmaintainable. As the person who really drove the design of the translator, I bear responsibility for these failures, as they are truly failures of design. Let me enumerate some of my major mistakes, as a penance and as a reminder for the future:

• I threw away all of the old code. There was a translator infrastructure already available to me, which I considered (mostly at Peter's insistence) and then rejected. This was not necessarily a mistake, as the old code had little to recommend it and might indeed have been more of a burden than a help, but I can't say that the new code is superior in any essential way to the old. I fear that I may have been motivated too much by the fact that reading code is harder than writing it.

• I based my high-level design for the translator off of some idea that I read in a book. I'm not going to mention the book, because it's a good book and the idea is a good idea. But the idea was fundamentally unsuitable as an overarching design for a translator. It's necessary to try new things, but doing so is a gamble.

• I assumed away problems of efficiency. While the prevailing wisdom in computing recommends deferring hand-optimization as long as possible (and there are good reasons for doing so), it's also true that efficiency needs to be a design criterion. If your design demands significant redundant computation, you need a new design.

The sad part is that people are becoming aware of the usefulness of constrained polymorphism, but it's not through Cforall or through my work, because there's no usable manifestation of either. (Props to Jeremy Siek and his various collaborators, who are working on a language similar to Cforall, and who have graciously acknowledged Cforall's precedence in their publications.) Enough self-flagellation, though. The point is that I need to make the system work, even if it's only to prove (to myself) that I can do it.

I'm trying out Markdown as a way of formatting my blog entries. It seems like a generally good idea. I've already been having fun playing with Kwiki, and the formatting rules seem very similar, which is good. It would be nice if they were the same. I'm sure it's only a matter of time before someone makes that happen.

Oh look, somebody did.

Aside from that there's really nothing to say. This is mostly a test post.

Since last May, I've been working to get μC++ a home in Debian GNU/Linux. Having already discussed my impressions of the technical details and the community of developers, herewith some thoughts on the political aspects of the process:

Debian stands in contrast to most other distributions in that it's inherently political. It, and its parent foundation Software in the Public Interest work to change public policy and individual perceptions by encouraging the development and adoption of free (as-in-freedom) software. A program has to be very free to make it into the "main" archive — free to use for any purpose, free to modify, and free to distribute (plus other related restrictions as outlined in the Debian Free Software Guidelines). And not only programs, but documentation, images, and anything else that forms part of a software package. Sometimes their high standards appear almost comical, as with last year's decision that the Free Software Foundation doesn't give people enough freedom.

Whenever politics comes into contact with software, it almost immediately polarizes the debate, between those who insist on freedom and those who want something that just works, and want it now. As with most things there is inevitably some compromise involved: even the people who demand freedom above all else need a system that works. However, it's important to remember that the "freedom" argument has a practical foundation behind its principle — i.e., you need those rights to protect you from being screwed over by a vendor. This tension has characterized the past year of the sarge release project, and is one reason why the release has been delayed. On the other hand, the GFDL issue illustrates the practical compromises involved: the offending documentation will be removed or rewritten, but sarge can ship without those changes.

There has been a significant amount of (sometimes justifiable) criticism that Debian is too focussed on its principles and not sufficiently concerned with the technology. In the big picture, though, it's amazing to look at the increasing size of the "main" archive (and the dwindling importance of the "non-free" archive). It really is an entirely usable and useful computer system that is unencumbered by proprietary restrictions, and the Debian project, through its combination of good technology, community-based development, and strong principles, has been a driving force in making such a thing possible.

I know I'm a week late with this, but I have to reference Bryan Cantrill's post Solaris 10 Revealed. Unlike many (most?) people, when Cantrill says "this is an interesting piece of code" it truly is interesting. Well, to me at least, which probably indicates some similarities in the sorts of problems that he and I are often called on to solve. His dtrace_fish example really touched my heart — I've been trying to grab information from lower stack frames for quite a while for the benefit of the μProfiler, and up until now my approaches have been slow and clunky. He's made more work for me now: now that I know it can be done right, I have no excuse for not doing it.

He also uses an obscure SPARC technique that he calls "instruction picking", which is too cool for words. He implies that it's a well-known idiom, but as far as I can tell it must be well-known only around Sun itself, because a Google search for "instruction picking" gets mostly banjo lessons. This is an example of why it's so difficult to become a really good assembly-language programmer in this day and age — the supply of really good assembly-language programmers is so small that gems like this stay within the oral traditions of select groups of people and never migrate into the web of the wider world. And the supply of prospective assembly programmers is so small that no one will ever write a book to collect these idioms. So, a big thank you to Bryan Cantrill for telling me something I didn't know.

This is just another illustration of why open source is good: despite the fact that reading code is harder than writing it, sometimes there's no better way to understand what a program/library/operating system is doing than to crack it open and see for yourself. I've wanted to do that to Solaris a number of times since I started working on μC++, and it looks like I may soon have my chance. My advice to all programmers is to simply face the pain and read a lot of code; it gets easier the more you do it, and you learn much more than you ever could just by writing code from scratch, or reading programming books. Joel Spolsky's article has some good tips for reading code.

Since last May, I've been working to get μC++ a home in Debian GNU/Linux. Having already discussed my impressions of the technical details, herewith some thoughts on the community aspects of the process:

Debian must have one of the largest development teams of any piece of software. This is notable, because its sheer size challenges one of the fundamental assumptions of modern software project management:

Brooks' Law states that programming work performed increases with direct proportion to the number of programmers (N), but the complexity of a project increases by the square of the number of programmers (N²). Therefore, it should follow that thousands of programmers working on a single project should become mired in a nightmare of human communication and version control.

Debian literally has more than a thousand developers, making it a case in point here. It does work, certainly to the extent of producing a usable result, without slipping into the swamp that Brooks warns about. Part of that success is due to well-defined procedures that formalize both communication and version control. But I think that the even larger reason is that the project naturally decomposes into independent parts (the packages) which interact using very simple interfaces. Where the interfaces become less independent, the development becomes more complicated, but even then the complexity can be restricted to the packages that specifically interact, rather than all packages and all developers. It's rare to have an issue that requires the collaboration of all developers.

It's not an unqualified success, however. It has been 30 months now since the last "stable" release, and most of the software therein is no longer considered even modern, much less state-of-the-art. It's usual now to use the "testing" release, even though it doesn't guarantee any security or stability. While politics has and continues to play a large role in the delay, technical considerations seem to be an even larger cause. While the biggest political issue has been shelved for the moment, technical issues, from the installer to the build system to plain ordinary bugs, continues to hold back a release. Not coincidentally, the central problem in most technical issues is coordination and communication among the developer population. Even the act of releasing anything becomes a problem of mass coordination, since it requires every developer to stop hacking and start fixing (or at least to let things alone for a while). The irony is that Debian's slow release cycle leads package maintainers to want to ensure that the released version of their package has the most recent code possible; but the delays involved in packaging more recent code (including fixing bugs and integration with the rest of the system) postpone the release, which only serves to aggravate the problem.

It's not obvious to me that there's an easy fix for the release problem. It may in fact be a fundamental property of Debian that releases in the traditional sense are not possible, given the number of independent developers involved. The good news is that smart people are gathering ideas. The even better news is that there are lots of smart people, many of whom are very nice. I was a little put off initially by the tone of some of the mailing list discussions, but my experience in package maintenance has put me in touch with people who can be pleasant, even when you disagree with their opinions. Not to mention people who mail me out of the blue with congratulations. And there's a certain excitement to it all, in the knowledge that none of this has been tried before, and even if it doesn't succeed completely it has still succeeded beyond the wildest expectations of its founders, changing the way software is made along the way.

In the last installment, I'll say a few words about the politics of Debian, undoubtedly its most controversial aspect. Incidentally, I should mention that my u++ package has actually made it as far as testing, so that it will be a part of the next stable release, when (and if) it happens.

It's been over two months since I taunted the gods of spam by posting my e-mail address on this site in an obviously machine-readable form. Interestingly enough, this hasn't seemed to cause any significant increase in spam. I have had an increase in spam in recent weeks, but I attribute that to the appearance of my e-mail address on the Debian web pages. Even then, I don't get a notably large amount of spam anyway — typically fewer than 100 per week, although in recent weeks that has climbed over 150. And CRM114 is keeping up, even though I'm using a positively ancient version.

I also have a shiny new Gmail account that I'm pondering. Sometimes it would be nice to have a separate account for personal e-mail, but sometimes it's nice to keep it all together.

Since last May, I've been working to get μC++ a home in Debian GNU/Linux. Herewith some thoughts on the technological aspects of the process:

Debian has developed significant geek cred through having had, for years, a slick, no-nonsense package management infrastructure. (This, some might say, is because geeks use "apt-get" and not "dselect".) There is a universe of useful packages, and installing a new package from the stable distribution virtually always "just works". Historically, this was in stark contrast to the behavior of other distributions. Presently, other distributions are catching on, but Debian still has a lot of technological momentum, not to mention an impressive array of packages.

Of course, there's no free lunch, and in this case the tab is picked up by the package maintainers. In order to create a Debian package, you have to read and understand the entirety of the Policy Manual which stipulates the structure and behavior of packages, and the ways in which they must interact with other packages. This is a big book, but it's not sufficient: creating a package from scratch based only on the policy manual specifications would be a huge task, and the resulting package would likely violate some unspecified assumption that current packages take for granted (the archives of the debian-devel list show that "policy-as-written" and "policy-as-implemented" are openly acknowledged to be two different things). And you have to accept that there is no generally-accepted comprehensive set of best practices, even for simple packages. So you have to do more reading, and you have to learn the motley collection of tools that are available to help. And even then, there's no way that you'll get it right on your fourth attempt. So there's a lot of effort involved.

The obvious question: is there a way to make this easier? The obvious benefit: more software can be packaged in less time, and existing packages are easier to maintain. Clearly, the large number of existing packages makes any re-engineering difficult. A more interesting question to me, however, is: is it possible to make it easier in a fundamental way, even when designing such a system from scratch? The Debian package infrastructure is a system that has evolved greatly over the years, and has become ever more complex through time. It's possible to argue that the complexity is a historical accident, but it seems more likely to me that the complexity is inherent to the problem. A package is a useful abstraction, but it's not possible to abstract away an entire computer system into a collection of independent packages.

I plan to write more on package management in the future, talking about my experiences in trying to cobble together my own system for managing software. In the meantime, stay tuned for "The Debian Experience, part 2: The Community".

We've finally posted pictures of our big fix-up blitz on the new house.

An article such as this one upsets me. While governments have come to rely heavily on gambling as a source of revenue that remains much more popular with voters than taxation, the recent study that showed that 4.8% of gamblers in Ontario account for 35% of gambling revenues indicates what a socially regressive policy this is. Furthermore, it puts government in an unavoidable conflict of interest: government, who should be helping problem gamblers, is at the same time dependent on exploiting them. Particularly galling was this comment:

Asked if the province's casinos are enabling problem gamblers, Health Minister George Smitherman said: "I don't think anyone would characterize us as that."

Well, minister, let me characterize you as that. Right here, right now, for the record. It's not the intention of course, but it's an inevitable consequence of making casinos accessible and inviting for everyone. To pretend that you're not enabling the behavior — or worse, to actually believe that — can only lead to tragedy.

I believe that there's a place for a little vice in everyone's life. But I think that the need for government to help people overcome their worst impulses outweighs the benefits of profiting from them. And I see those two objectives as inevitably incompatible when it comes to policy. For the same reason, I believe that the LCBO ought to be privatized. The fact that these enterprises turn hefty profits is just another reason why they ought to be operated as regulated private enterprises rather than legislated monopolies.

I guess it's time to write my M.P.P.

Well, actually, Technorati did, so let me just say, as someone who has absolutely zero control over the outcome, that I can't bring myself to truly support anyone in the U.S. election. Kerry doesn't give me good feelings — he's spent approximately 100% of the campaign bashing Bush and approximately 0% articulating reasonable and realistic policies — but I oppose Bush. Why? Tim Bray and Andy Tanenbaum offer a couple of different reasons, both of which I agree with.

Our friend Shawn came up with a couple of extra tickets to see Theatre Newfoundland Labrador's production of Tempting Providence at the Centre in the Square last night, so Jean and I went. Not without some trepidation, mind you, since the "coming of age in Newfoundland" play is, shall we say, thematically cliché; if the tickets hadn't been free we certainly wouldn't have gone. It turns out that the story is more interesting than we had feared (and based on the truth, which usually helps), but what really made it good was the tight direction and staging. I always admire a play that uses a minimal number of props to get its point across, and the pacing of the action felt just right. The text was good too, although it left us hanging on a few points, but it managed to be affecting in a subtle way, without melodrama, which is harder than it sounds to achieve. Plus Jean's biggest fear, that the cast would affect phony Newfoundland accents, was alleviated when we discovered that they were all genuine Newfies.

All in all, I'd recommend it, but I have no idea where it plays next — it was only in Kitchener for the one show.

I'm running a bit of an experiment here: the E-mail link at the right actually contains my true, unmangled internet e-mail address. This has become an increasing rarity, it seems, due to the proliferation of address-harvesting web-crawlers and spammers who are willing to use them. I can't fault people for trying to hide from these low-lifes, but at the same time, it feels like something is lost. In particular, such methods are of relatively little concern to spammers (who, lacking any interest in me personally, can move on to other people's addresses and other methods of obtaining addresses), but cause the most hassle to exactly the people I want to hear from: people who come upon my site and want to correspond with me. So my e-mail is here unmangled as much for the sake of principle as for its practical benefit. I don't want spammers to restrict how I communicate on the Internet, or how other people communicate with me. And given that I feel that way, I feel obliged to act accordingly.

Having said that, I'm also a practical person. I only feel safe doing this backed up by a more-than-decent spam filter. I've been playing with CRM114 for about 4 months now, and it really seems to hit the spot (although it took quite a bit of training to get it there). As of yet, I haven't noticed a spike in the volume of spam I receive, although it is up slightly. And it's all going into the spam box automatically.

Interestingly, around the time I started with CRM114, Gord Cormack and Tom Lynam trashed it (well, at least as much as a prospective journal article supported by a scientific method and statistical analysis can "trash" something) in a well publicized preprint that was picked up on Slashdot. Gord and Tom are definitely smart people, and I know that they feel strongly about the spam problem and that they're employing legitimate scientific methods to study the problem, but I can also understand the indignation of those who disagree with their conclusions. For me, SpamAssassin never worked well, while CRM114 is akin to magic. I think that the composition of an individual's mail stream must have something to do with it.

As part of changing the look of my site I managed to make almost everything inaccessible over the long weekend. Hopefully it's fixed now, and the crawlers will pick it up soon.

Incidentally, I noticed this by reading through web server access logs. It's interesting that a number of people (university CS students, probably) from all over come here for my UNIX tutorial notes, even though they've never been officially advertised. Such is the power of Google, I guess. I hope those who find them find satisfaction within them.

I attended a memorial service for a friend on Sunday. There was a lot of pain for all involved. The minister (apparently Anglican) was terrible — he was nauseatingly politically correct. At one point he went so far as to refer to God as "supposedly all-powerful". Come on, sir: either you're drawing on your faith or you're not; don't hedge your bets. It didn't make his sermon any more inclusive anyway — the committed atheists in the crowd were still complaining about the quantity of God references.

There was a well produced video slide-show of pictures, accompanied by such songs as "Walk On" and "Do You Realize?" The overall effect was to reduce the collection of mourners to a soggy mess. Even Jean, who only met Drew once, was in tears. I left feeling manipulated — it was the sort of presentation that makes you cry on general principle, regardless of how you feel about the deceased. On further reflection, though, maybe it's necessary. Perhaps the point of a memorial service is to bring everything to the surface and force you to deal. If so, this one succeeded. And I'm not sorry I went.

Drew was a great guy, one of the deepest thinkers I've known. He provided me with a crucial push when I stood wavering at an important crossroads in my life. He had great expectations — expectations in the Dickensian sense, but also expectations of himself; this last may have been his undoing, as his sister suggested in a courageous and heartbreaking eulogy. Whatever the reason, I can only trust that the pain he felt in life was worse than the pain his death has inflicted upon those who cared the most.