Out, damn’d open source! out, I say!

I have had an usual number of requests for files this last week. These are files that I’ve either accidentally removed during moves between tools; or deliberately removed because I need the space for new development.

Among files accidentally lost were ones for my post Stepping Stones to a Safer Blog, about how to make Movable Type safer against comment spam. I was surprised, because the technologies featured are older versions of what exists now, and wouldn’t be compatible — but people are still using older versions of the applications in question.

I looked for the files on my local machines, but unfortunately couldn’t find them. And chances are they were lost so long ago that my ISP wouldn’t have them. To be honest, I’m also a bit concerned about putting out such old versions, in case people mix old and new and completely break their sites.

I’ve also noticed in this time that I’m getting an enormous number of hits for all of my old comment spam postings. I found out why when Al Sessions sent sent a link to this post at Photo Dude’s and found out that the situation with MT comment spam is getting serious enough for ISPs to consider shutting down MT comments permanently for all hosted sites — or even forbidding Movable Type altogether.

So when Marius, responding to the recent problems with WordPress vulnerabilities, wrote the following:

The WordPress open source developers nailed their colors to the mast the other day. They think it’s a minor inconvenience that if you enter an erroneous URL, you make a site inaccessible. And anyone asked for urgent action to fix that was “freaking out”. [Big Pink Cookie][Shelley]

Dana Blankenhorn just dared identify the lack of documentation of open source projects as their Achilles heel. Just watch the apologists come out of the woodwork in the comments to that post.

Jonathon, can we move this Weblog to Movable Type (not to MSN Spaces), please.

(emph. mine)

…my first reaction was to write in comments, ‘Eh, you may want to hold on that at the moment.’

I think both Marius and Dana are providing a service to the open source community with their criticism. It has become the darling child that can do no wrong, and this adulation can lead to arrogance. However, in both cases the writers are focusing on one specific open source project and using this as a brush to tar most open source applications, and that’s not particularly fair. About as unfair as to paint all commercial proprietary source application development as greedy and insecure, based on past actions of one or two companies.

The issue is really less to do with the openness of the code than it has to do with the mindsets of the people involved — both user and developer.

For instance, let’s look at two popular weblogging tools: WordPress and Movable Type. WordPress is an example of a non-commercial open source weblogging application; while Movable Type represents the (mainly) commercial, proprietary side of things. Yet the developers from both applications have demonstrated the exact same difficulty in recent times, and that’s an unwillingness to adapt.

In WordPress,when faced with a vulnerability in the code in both WordPress 1.2.1 and 1.3a, the developers edited the lines that caused the initial more severe vulnerability. However, when it was demonstrated that even the edited version could lead to problems, they seem to be unwilling to adjust the code–or even respond to the concerns. Worse, they’ve downplayed the problem in addition to those who reported on it, and have refused to publicly document that the problem exists, and how people can fix it.

Sure enough, it was only a few days later that this ended up re-appearing in the support forum.

However, the Movable Type people have also demonstrated this same inability to adapt to problems, as they arise. When faced with the fact that TextDrive is considering dropping Movable Type from its servers because of problems with MT comments, Anil Dash from Six Apart responds in a comment about a new version of MT-Blacklist coming out, and how TypeKey prevents this problem.

Yet if he were to read what Jason at TextDrive was saying, he would realize that rather than helping, both of these solutions could add to the problem; because what’s happening is that the access to the file, mt-comment.cgi (or its renamed version), and the underlying construction of this file is what’s causing the problem — not the spam appearing in people’s comments.

Comment spammers are now hitting mt-comments.cgi several thousand times in a short period of time, using a variety of IP addresses in each. The mt-comments.cgi application file, modified by MT-Blacklist or not, is not handling this load effectively, and is leaving unterminated threads of excution (if I’m reading what Jason wrote correctly, and for want of a more non-tech friendly term): i.e. too many instances of mt-comments.cgi are left running.

Too many of these unterminated threads of execution, and you basically have no site left — these threads are the rarest resource on any given server.

This isn’t an issue of bandwidth, or IO access, or comments appearing in posts or the sanctity of people’s space being violated. Nor does this have to do with Google’s underlying Page Rank, and people having to delete too many viagra comments, using tools that don’t make this easy. This has to do with the server’s resources being overrun: all before TypeKey or mt-blacklist come into play. In fact, adding yet more fancy mechanisations to mt-blacklist, or more dependence on a central authority system is likely to increase the problems experienced by ISPs when it comes to MT systems; not decrease it.

In other words, the developers are not adapting.

(Of course, in the same thread, I found it ironic to read the representations of the WordPress legions, pointing out how a complex multi-SQL access spam terminator in WordPress would solve all a person’s problems; not realizing that if WordPress ever gets hit the way Movable Type is hit now, those fancy solutions are going to become strands of pure, naked, steel twisted about the weblog user’s throat. I agree with Jason et al — comment spam is going to require a system solution, not an application fix. )

So the problem isn’t open source or proprietary, as much as it is the developers keeping their minds open and not becoming so enamored over their own cleverness, that they literally cut off their nose to spite their own faces.

A good case in point of tools, open and proprietary source, demonstrating a willingness to adapt is can be seen with both Microsoft and Firefox. Yes, Microsoft and Firefox, two tools at the extreme opposite when it comes to the open nature of the code and the development process.

I wrote recently about problems I had with having to re-install Windows 2000 on a machine, using a phone modem, and how something crawled in while I was downloading the 100M or so of fixes. I grumbled and bitched about Microsoft and its damn security problems. I had planned on continuing this discussion the next day, but got sidetracked when the Kitchen weblog was seemingly hacked.

What I wanted to make note of in the second posting was the fact that I was installing an operating system that was first developed close to eight years ago, using a four year old CD to do so. I wanted to point out how Microsoft is still supporting this operating system, after all this time–and how all my newer purchased goodies worked in this old OS.

Windows 2000 has been hit with a string of security problems, and the company has suffered from a general lack of confidence because of it. Yet rather than downplay this as a problem, the company made it easier to update a machine to meet each new break. I could have bought a CD with all the recent service pack and security updates for the cost of material and shipping. If I had broadband, it wouldn’t have taken more than an hour to download and install all the updates — a process made easier by the company providing that lovely little update program, which determines what fixes your site needs, and then lists them out for your single or group download.

Now I have a Windows 2000 machine, which sits with a little upgrade wizard in it that checks periodically at Microsoft and lets me know that there is a new fix I should think about installing. My machine works beautifully, even being an old machine using an old operating system. All because Microsoft has learned how to adapt.

It isn’t a case that the company makes perfect software, and always makes the correct decision when it comes to technical directions; it’s dropping ego when push comes to shove and making the best of a bad situation.

Now, let’s look at Firefox, the exact opposite of Windows 2000 in the proprietary/open source scale.

Firefox is not perfect software and I’ve experienced some glitches a time or two with it. It also seems to meet Dana’s concern about lack of documentation, though I think this will soon change (note to self — write more Firefox how-tos). But the tool is also one that has adapted to user’s needs, even when these needs seem to be small, and trivial.

An example is the fact that with the earlier Mozilla, extensions would require people placing code in specific directories. This doesn’t seem that complicated — download the code, unzip it, and move it to a specific location. However, it was found over time and painful experience that this was confusing for many users.

Rather than curse the users for being idiots, or find a fancier way for people to do this same task, the software developers adapted the tool to meet the user’s needs. They most likely did so at the cost of a slower release, and perhaps even leaving out some new features, which would have been ‘more fun’ to work on.

Like Microsoft’s software update, the developers created a ‘one-click’ method of installing extensions. Now if you want to use an extension, just click a link, do the install, restart your browser and it’s available. Not only that, but there’s a link that will take you to a place where you can look for more extensions; and another link that will check to see if there’s updated versions of the extensions you do have installed.

Both organizations have demonstrated the most important aspect of product development: there is more to a software application than the code.

There is no room in software development for either arrogance or an inability to adapt; with holding on to a way of doing things from pure stubbornness and pride. You shouldn’t even start a project if you can’t face the inevitable: that you will make mistakes. Developers, yes even ‘free’ source developers, owe their users respect, because users have placed in them their ultimate compliment: their trust. This respect then means the developers have to be able to admit when they’ve made a mistake, or pursued a wrong course.

However, users also have a responsibility in the development process, and should not see themselves as passive consumers of functionality; they should be encouraged, nay, expected, to participate in the development process. At a minimum, ‘bad things’ should not be kept from them, as if they are children who can’t deal with difficulty. And if the developers do publicly say, “We screwed up and here’s a solution”, the users have an obligation to meet the situation with grace rather than petty ire:

“These things happen, and thanks for letting us know and giving us a solution to fix the problem”, rather than, “You moronic developers and your buggy code!”

Documentation should be written at the same time as code, and the documenters given the same prominance as the developers. For instance, there is documentation of WordPress 1.3 available, but there’s no link to it at the WordPress site. The documentation text should be given as much respect as the application code, though this rarely happens. And when you send a note of thanks to the developers of a product, you might think about sending one to the documenter, as the only time documentation is mentioned with applications is when it’s missing, as Dana demonstrated.

In June I wrote:

Bluntly, the WordPress development crew is not happy with me because I’ve been pushing them pretty hard for the last month. What I’ve been saying is that software is only 50% code – the rest is documentation and infrastructure, quality testing, and communication. Particularly communication.

Oh, you don’t need these things if your code is used by hackers or a small group of friends. But if you want your application to be used by strangers who don’t code – you can’t force them into learning code to communicate, or having to beg pretty please in order not to piss off the development people.

I’ve gotten a lot of flack for my criticisms of past weblogging tools. I stand by these criticisms, every single one of them. I’m not, now, going to play ‘touch not the programmer’ just because the source code I’m now using is open source. If anything, I want the open source solution to work, so will be harder, not easier, on the team behind the product. Is this unfair? What’s fair? Not being critical because this just isn’t done in weblogging?

I’ve been told, “easy to criticize when you’re not the one behind the code”. So I’ve since forked WordPress and will put myself into that position, though I have been there with numerous jobs in the past. I guess we’ll see over time if I practice what I preach.

At the least, I can write code and take care of my own problems. What about those who can’t?

Out, damned spot! out, I say!–One: two: why,
then, ’tis time to do’t.–Hell is murky!–Fie, my
lord, fie! a soldier, and afeard? What need we
fear who knows it, when none can call our power to
account?–Yet who would have thought the old man
to have had so much blood in him.

Shakespeare’s Macbeth