I am hard pressed to name an experience in my life that doesn

I am hard pressed to name an experience in my life that doesn’t relate in some way to the fascinating profession in which I have engaged. Musical performance, sports, marriage, having children, gambling and writing fiction are all just as relevant as the Discrete Data Structures class I completed in my junior year in college. Art, literature, mathematics, philosophy, history, political science, economics, physics, biology, psychology, sociology, anthropology, and linguistics all contain lessons for those who would undertake to learn them.

Computer code is much like other stuff in some ways and vastly different in others. Human intellect has a marvelous capacity to generalize. We naturally lump things into known classes and infer their character based on what is known about the classes. When we encounter a furry quadruped we figure it’s a mammal. We infer it’s a warm-blooded critter that gives birth to live young that will be fed mothers milk. We all know that what quacks is beyond reasonable doubt a duck, which will reliably exhibit the well-known characteristics of duck-ness. Software is a platypus.

Building software is often compared to putting up buildings. In some ways, it is similar. Yet it is the dimensions in which it is different that bite the asses those that are overly attached to this metaphor. Both begin with functional goals, which are the basis for an architect’s specification. Next, engineers fill in the details, using their experience in specialized areas: user interfaces, databases, HVAC or plumbing. Construction crews pour concrete, lay down database schemas, and assemble joists, studs, interactive dialogs, data validation programs, pipes, clients and servers.

But there are deep and fundamental differences between software and buildings. Building materials present themselves in a very direct way. What appears to be a pile of bricks is almost always a pile of bricks. Imagine putting up a building if bricks were only visible to bricklayers, boards only to carpenters and wiring only to electricians. The construction foreman would have to trust each contractor to report his status accurately. Only when the entire building was complete would it appear to outside observers.

A lot has been said about the methods that are applied to the creation of software. Advocates of many diverse techniques tout benefits that include cost savings, risk reduction and improved quality. I once asked the headmaster of a small and highly respected private grammar school what characteristics he looked for in the teachers he hired. He replied that there was no single dominating factor except that he categorically rejected any applicant who was strongly committed to any single teaching methodology. Having struggled with the pros and cons of various methodologies in my field, I found this fascinating and asked him why. His reply was simple and compelling: All the children are different!

A very successful CEO in the software industry once said to me, “Good marketing is like good pornography. I can’t define it, but I know it when I see it!” Can we define “good software”? Do we know it when we see it? In its platypusimal way, good software is both definable to a certain extent, yet subjective and elusive in others. Good software generally exhibits well-defined, clear and orderly partitioning of function. It exhibits a sufficient level of generality so as to be easily adapted to minor changes in requirements or business rules. It is robust, not easily broken by bad data or bizarre user behavior. But these characteristics fall far short of being sufficient, and in truth they are not even necessary. Consequently we hear terms like “stickiness” used to describe software that is attractive to users. Must we resort to the squishy lexicon of wine tasters, audiophiles and music critics to describe our product?

In the beginning, computers were programmed by engineers and scientists for their own use. The best thing about this was that they couldn’t bitch about the programming department. Later, in the golden age of mainframes, most software was built for business users by internal systems departments. Scarce IT resources were allocated through a centralized corporate planning process that was just about as effective as the Soviet economy. The value system by which software was judged grew out of the politics of corporate IT departments. Everybody talked the cost / benefit talk, but the walk was pure politic. The first real challenge to big IT came from business application package vendors. The departmental mini-computer put of few more cracks in the wall. Personal computers, LANs, client / server computing and finally the Internet reduced it to rubble.

Over the years, as the dictatorship of IT has given way to the Darwinian chaos of the Internet revolution, we have learned a few things about the goodness of software. First of all, we know that the best technology doesn’t necessarily win. Rather, successful software is that which establishes itself in an evolutionary niche. Oracle was fabulously successful because it was relational, portable, had in integrated fourth generation language and a cheap PC version that programmers could afford for personal use at exactly the moment in time when these needs converged. It had the right “shape”. A lot of people liked Sybase better, but Oracle had a version for the mainframe. Hardly anyone ever used it, but it met an important scalability objection. Some of the factors that enabled Oracle to establish itself with early adopters didn’t turn out to be all that important in the end, but by that time its competitors were all but extinct.

Software is a fashion industry. I say this without a trace of the cynicism I held the first time I said this, years ago. The strict business utilitarian view of twenty-five years ago produced software that was analogous to the Model T, available in any color you wanted as long as it was black. In the modern, competitive world of cars and software, where there are choices, buyers make their decisions based on factors other than the strictly utilitarian. Competition makes software development harder, but it’s a lot better for consumers. Basic utility is necessary but insufficient. Variation allows users to choose the solution that meets their needs and desires, no matter how fickle. For any particular producer, this is a royal pain in the butt. In the aggregate, various customers influence various vendors to produce various features. Users get to try different approaches. Yet another vendor synthesizes the best of the best.

How ultimately shall we judge the software that we build? In the end we must judge it based on its use, the final measure of its success. I’m enough of a bit-head to have feelings about the elegance, coolness or beauty of a software solution. I thought Multics was really cool. Digital’s OpenVMS is a lot more elegant than Microsoft Windows or NT, but let’s not be stupid here. Windows is magnificently successful.

Building software involves a lot of people. Scott McNealy once said, “People aren’t all bad, but most of them are.” His point was that more people don’t produce more results, but rather, work grows to meet available resources. I’ve heard this said another way. Work is a gas, it expands to fill its container. If this is the first law of software project staffing thermodynamics, then the second is that you can’t get blood from a turnip. I thoroughly believe that there is a right size for any given project team. What is amazing to me is the distribution of staffing error. One would expect that most projects (say eighty percent for the sake of argument) would be staffed about right and the other twenty percent would be either over or under by varying degrees. I think that reality is the mirror image of this. Most projects are either grossly over or under staffed. Getting it right is the exception.

Software project management is a joke. Nothing is ever completed on time. Plans are obsolete before they’re published. Progress reports are works of fiction. And everybody knows it! We’ve tried everything. We have methodologies, capability maturity models endless project management tools and lots of opinions … but no real solutions. I think it’s time we question the validity of our assumption structure. Wrong assumption number one is that we know what we’re going to do before we do it. It’s not for lack of trying, but we’re trying to defy the laws of human nature. People understand things primarily through direct personal experience, not by talking about them. This is why children make the same mistakes as their parents. The second wrong assumption is that all other things are equal. The cavemen are cold at night. Give them butane lighters. All other things being equal, they can build fires and stay warm. But of course, they’re not just going to build fires to keep warm. They’re going to burn down the forest and each other’s villages. Never underestimate unintended consequences. If you want a more current, technological example, ask yourself where the Internet came from.

In a few rare cases, it has been possible to directly measure the economic benefit of software systems. In the early days, when data processing techniques were used to eliminate massive manual labor, we counted clerical headcount reductions. More recently, we’ve seen real reductions in inventory levels as MRP and Internet-based b2b tools have made supply chains more efficient. The scenario we more often encounter is one in which cost justification is required to satisfy IT bureaucrats but not the driving force behind the development or purchase of software systems. I believe that the basic principals of economic theory readily explain this phenomenon. Supply is not a function of demand, nor is demand a function of supply. Rather, guided by Adam Smith’s invisible hand (his metaphor, not his hand), the two find equilibrium. Since the marginal cost to produce the next unit of software is near zero, supply rises to meet the weakest demand once the right set of conditions are met … like everybody having a computer on their desk. Software reproduces like Tribbles. Can we imagine another resource that is so enthusiastically propagated by it’s corporate users, at an original cost so low as to be negligible, that is so prolifically adopted that policies are created to stop it because of the nuisance cost of supporting it?

Almost every software engineer I ever hired told me at their job interview that they hated politics. My answer to this is that politics begin whenever there is more than one person involved. Don’t kid yourself, programmers love politics. They form factions, double deal, backbite and defect, and they are exceedingly clever at cloaking self-interest in terms of the greater good. In business school, I took a course in the management of multi-national corporations. The professor drilled into us the need to understand cultural differences. Later, working in Japan, I gained first hand experience in just how polite and deferent someone could be in the process of drilling you a new orifice. Later I learned that you don’t have to travel far to experience vast cultural differences. Just take a walk from the programming department to network operations. It’s like going from France to Germany. Visit Marketing and Finance. Think about the difference between Bangkok and West Texas. If you want to manage the interactions between diverse organizational groups you need to understand the norms that characterize them as well as the factions within them.

Everyone who has participated in a large software project knows that there are inevitable tugs-of-war between competing parties: users who are advocates for their pet features, factions within development over who gets to build what and endless battles over features versus schedule and budget. Funded projects are the food and water of software development organizations. Big projects often result in feeding frenzies. As in government, debate serves its purpose. All points of view are aired. Compromises and coalitions are created, but not everyone will be satisfied. Where there are competing interests, there are winners and losers. Some of the conflicts will not yield to compromise. It is at this stage of the process that projects get completely FUBAR in the absense of smart, strong and savvy leadership. Smart, meaning able to sort out it all out and make the right decisions. Strong, meaning with sufficient authority to impose them on dissenters and savvy meaning knowing how to pick the fights that are worth fighting.

For those who haven’t cashed out their options and retired to Maui, the Next Great Thing is always just over the horizon. Great Things I can recall (I reveal my age here) include COBOL, virtual memory, multi-tasking, multi-processing, solid-state-memory, time-sharing, 3270 terminals, PL/I, ADA, Top Down Design and Development, Prototyping, Databases, 4GLs, SQL, C, C++, Smalltalk, LISP, mini-computers, microcomputers, word-processing, personal computers, GUIs, Virtual Reality, Artificial Intelligence, Expert Systems, Neural Networks, Fuzzy Logic, CASE, Repositories, LANs, WANs, Object Oriented Programming, distributed computing, Data Warehousing, the Internet, Java, eCommerce, wireless and web services. Some have been greater than others. For others, history will be the judge. The one thing that’s been consistent is our inability to pick the winners until after the fact. For some odd reason, we just don’t seem to know what’s going to catch on until we try it. The revolutions seem to come when we least expect them. They also take about 10 years longer to really catch on than we expect.

Everyone in the software business who has ever read Scott Adam’s Dilbert comics recognizes the characters and their laughable stupidity. It’s funny but eerily real. I once heard an old executive advising a room full of managers in a company that was growing by two hundred percent per year more or less the following: On a one to ten scale, the people who start and run successful companies are nines and tens. The people they are hire are sevens and eights. They people they hire are fives and sixes. The people they hire are Dilbert and Wally. The conclusion is obvious. When you get big, you get average. If you’re the phone company, or you’ve got really big oil reserves or a near-monopoly on laser printers or PC Operating systems, this is non-fatal. I got really depressed over this idea. Was this the best we could do? Does success ultimately lead to being big and stupid? I believe that it’s not necessarily so, but it’s a battle that is not easily fought and never conclusively won. The fight against mediocrity must be fought until hell freezes over and then fought on the ice.