Once in awhile, for entertainment, I visit online freelance software development exchanges. It’s a fascinating study of the endless human capacity for witless self-deception. I actually saw a posting today from a guy who wants someone to clone an entire accounting system. Other than the name of the system to be cloned and an unexplained desire to have the work done in PHP, that’s the entire specification. And the budget? Two hundred dollars. I kid you not.
Apparently this guy thinks there are people who can knock out an entire accounting system in two hours. Okay, maybe he’s thinking twenty hours with third world labor. But still.
Apparently his reasoning process goes something like this: (1) I can buy Quicken for $79, therefore (2) it will cost a LOT more than $79 to write my own clone of Quicken — maybe as much as three times as much, but (3) I’ll cut it down to an even $200 as a starting point for negotiations.
I recall a couple of years ago getting a contract pitched to me where the objective was essentially to clone the whole Google search engine. The guy seriously believed that I could lead a team of a half dozen others and pull off in six months what Google, a company with thousands of employees, has spent a decade building. We would all work for fifty grand a year but in six months we’d go public and we’d all be millionaires. Hooray for us!
This guy got my award for Most Improbable Personal Delusion of the Year, until a scant week later I saw a very similar posting to a freelance site with a budget of $500!
There are literally hundreds of such postings floating around the Internet at any moment in time. It makes me wonder if these boards exist for no purpose other than to fleece the simple minded. What kind of contractor would respond to such a post? It must be the sort who will take a one-third deposit up front and then disappear.
What disturbs me most about all this is that while it represents an extreme, it seems to reflect that the commoditization of the craft of software development has reached some kind of fever pitch. I turn down many projects that come my way, for the simple reason that the customer has totally unrealistic expectations. Thankfully no one has yet asked me to write an accounting system or a major public web site or a spacecraft control system in one afternoon for under a hundred dollars — yet — but sometimes it seems like things are heading that way.
What this tells me is that the average person no longer values software. We are used to freeware, shareware, open source, and sub-$100 list prices for commercial software. The gargantuan size of the software market hides the gargantuan effort and expense that went into developing all those general-purpose software products.
Additionally, software development remains a painful and challenging process with many pitfalls for the unwary, and it just doesn’t deliver the effortless and instantaneous results we’ve come to demand of every aspect of life. People will jump through rings of fire and eat little pieces of glass before they will make allowances in their business plans for the fact that the mission-critical software they need will take six months and a couple hundred thousand to put together. “Screw it … we’ll use Excel!” they say.
So … what’s the deal? Is software a commodity now? Can we really put together complex systems in minutes from off-the-shelf components with no planning or testing?
According to a recent profile in Technology Review, no less a person than uber-programmer Charles Simonyi (the guy who was once the driving force behind Microsoft Word) plans to give the public what they are clamoring for. He is on a multi-year quest to create something he calls “intentional software”.
The over-simplified summary of what Simonyi wants to create is: “an expert system that will allow non-programmers to put together software specifications.” Then, by pushing a button, all the code will be generated to produce complex applications that fulfill those specifications.
I’ll concede that I probably don’t appear to be the best candidate to be impartial about the feasibility of Charles’ dream. Still, I doff me hat to ‘im, and wish ‘im well, I do. Because if he actually pulls it off, we will be living in a world of luxury and abundance and riding the wave of another “dot boom” and I will find a way to prosper from it, as will you, gentle reader.
However, my personal prediction is that what will come out of it, is something akin to Charles’ original brainchild, the graphical / WYSIWYG word processor. By which I mean that it will be something we will all take for granted and wonder how we could possibly live without, but it will also fall short in ways that will annoy the living bejesus out of us. (In the above referenced article there is a priceless vignette where Charles’ PowerPoint presentation hangs and the much-hated Clippy the paper clip pops up to offer useless advice. Charles confesses ruefully to the interviewer that even he has no idea how to disable the blasted thing).
Why do I think Charles will fall short of his lofty goal?
One of the reasons is well presented in a sidebar to that article in Technology Review, and that is generically known as the “leaky abstraction problem”.
At a more fundamental level, Charles’ earnest vision depends on reductionism, the idea that every problem can be solved if you can break it down far enough. This is an understandable stance for a software developer, as much of what we do involves breaking large problems down into smaller sub-problems and then putting it all back together. But it has its limitations. When developing software, you are ultimately solving problems that involve “wetware” (people) and that is inherently subjective and messy and chaotic. At some point that interferes with doing science and you have to make very subjective judgments to find a way forward. No development methodology or tool will ever fully automate those judgments.
Now I’m going to say something provocative and easily misunderstood: most of the world’s business is conducted by small to medium-sized companies. Most custom development needed by such companies are relatively modest compared to an enterprise-scale “Manhattan project”. In almost every case, by the time you’ve adequately spec’d most of these projects, you have finished implementing them. It is literally a case of “we’ll know when we are done”. I know this will make the Agile folks, the bean counters, and others apoplectic, but it’s the truth. Anyone who tries to paper it over is just patronizing you.
Software development is an iterative process of discovery. It’s done when it’s done. If you have a big enough project you can do proof-of concept “mini” implementations of various things, run simulations, conduct focus groups, have committee meetings and fill out ISO forms to your heart’s content. And then maybe, assuming all the stakeholders have been honest and diligent (a huge assumption), you will be able to come up with a reasonably close cost estimate / budget.
But in a smaller setting the grim reality is, you’ve got a champion with a fairly vague idea of what they want — more of a vision than a spec — which is communicated in glowing terms to someone like me, who does their best to ask pertinent questions and classify the project as a one, three, six or twelve month project times X number of developers. As the project rolls on the champion has a thousand suggestions for changes, the actual users inform you that for various reasons certain aspects of the vision are completely unrealistic, certain others decide they want the project to fail and employ passive-aggressive tactics to bring that about … and in the end, you have something that is not really what anyone expected up front, in terms of its final form or its cost, other than in very general terms, such as, for instance, “an online credit bureau for trucking companies”. Which comes to think of it, sounds suspiciously like the tiny postings on those freelance boards.
What the online freelance exchanges represent is the earnest desire of many to be able to express a complex system in one sentence, pay a tiny, nay, microscopic fixed amount of money and have it appear magically in a few days or even hours (another common subtext in these postings is “I’ve known about this for two months but just remembered that the deadline is tomorrow”).
Listen to me carefully: IT ISN’T. GOING. TO HAPPEN.
What Charles Simonyi’s more refined vision represents is the understanding that you can’t express complex systems in one sentence. But he still labors under the belief that with the right tools you could express it in terms that people who know nothing about software architecture could comprehend, in ways that will reproducibly result in relatively fast, easy implementations that are also accurate. This, I don’t think is going to happen either, at least not in my lifetime. I strongly suspect it’s a reductionist fantasy.
Check back in fifteen or twenty years and we’ll see!
Update: For more on this popular topic, please refer to the follow up post.