One Counter-Example Is Enough

There is quite a debate going on about the evidence for why requirements are needed. The Standish ‘Chaos’ Report has come under fire and its owners have so far kept an eerie silence. What the Standish Corp. and others have tried to do is to gather statistical evidence on how many projects fail, and for what reasons. This is pretty hard to do, as failing projects may well want to keep quiet, and companies do not want to give their competitors ammunition in the struggle to win new work.

However, the Chaos report did not assert that, for instance, bad requirements work caused x% of all projects to fail. Instead, it asked large corporations how many of their projects failed, and then which factors it considered important in causing failure.

There are plenty of things you could say about this as an inquiry method – for instance, suppose that 3 out of 4 corporations just threw the survey form in the bin, and that of the rest, most of the forms were filled in by the process improvement department. The 3 out of 4 might have been those who didn’t acknowledge that requirements were even an issue, and who consequently had the worst projects: we don’t know, and nor presumably did Standish.

Process improvement departments might deliberately colour their responses to show how vital their work was, and therefore how problem-ridden the unimproved projects were. Shock! Horror! There are heaps of problems! Quick, spend money on process improvement!

Furthermore, a list of factors that someone thinks might be causes of failure is not the same as an analysis of what actually went wrong on any given project, nor as it might seem to be, an analysis of the relative impact of the different causes of failure of a set of projects: it’s a generalisation at arm’s length, a piece of ‘tender-minded’ thinking (see the book review below).

Quite different statistical evidence could be collected by experiments on students, but the applicability of such results is also a moot point.

So, if statistics cannot deliver, is there any way to gather evidence that bad requirements work is a problem?

Yes there is.

Remember that all you need to disprove someone’s claim that X does not exist is one instance of X, or two or three if you are cautious. You don’t have to do statistics to try to show that lack of requirements is always bad; you just have to find one case where it was a serious problem, and the claim that requirements work doesn’t matter is disproved. In other words, showing that

Exists X: problematic(X)

is sufficient, and a great deal easier than showing that

Forall X: problematic(X).

Let’s be concrete about this, then.

·         Do I know of an actual project where lack of requirements work caused disaster?

·         Yes.

A scientific analysis-and-modelling package project was knocked together based on a chat in a pub, as a ‘good idea’ that was ‘obviously useful’ and which ‘could easily be developed by the graduates’. No-one was given responsibility for the overall specifications, and the system lacked a coherent architecture and interface specifications. In fact there were scarcely any written requirements. Nobody looked at the user interface either. The new programmers did their best, and the individual components, some of them doing very complicated three-dimensional geometry and interpolation, certainly worked. But the interfaces were a continuing and very costly nightmare, and when the product was finally shown to clients, it remained very buggy.

At least as importantly, it was very difficult to use: nobody had thought how it would be applied to problems, or indeed what problems it would actually be used to solve. The system was ‘inside-out’, making sense to the designers of its component parts, but almost no sense to its users. A whole lot of the complicated work had to be done in the user’s head. Not surprisingly, it didn’t exactly sell like hot cakes.

Let’s go further.

·         Do I know of an actual project where a single bad requirement was enough to cause serious trouble?

·         Yes.

An image processing and distribution service in the days of slow modems and plain-old-telephone-system phone lines allowed users to view a catalogue with thumbnail images, and to place orders for high-resolution images. This meant that users had to be allowed to ask to see what they were about to buy before committing themselves.

The requirement should have said something like

“Users shall be able to view thumbnail images.”

 Unfortunately it said

“…to view images.”

Yes it did. At the acceptance tests, the customer asked to be shown how to download an image. The head programmer showed him how to get a thumbnail on screen in a few moments.

“Yes, that’s fine, but I want to see a full image please.”

There was some umming and erring.

“Here it is in the requirements specification, ‘view images’.”

“Yes, but that will take 24 hours or so over the slow modem link; the phone line will go down every few hours, so you’ll never be able to transfer a whole image.”

“Never mind the ‘yes, buts’, it says I can have it and I want it.”

Really. So the project went over time and over budget to do its best to make it possible for the customer to do something that was never going to work properly given the available technology.

The developers of the image distribution service got paid eventually, which is more than happened in another company which foolishly included the requirement

“Each search shall be completed within 2.5 seconds”

in their telephone system specifications. At the acceptance tests, the developers showed off the pre-arranged searches which all easily passed. Unluckily,

“each search”

could include the surname

“Chang”

which is very common in countries with a large Chinese population. The customer asked for a demonstration, which, as he presumably anticipated, failed by a wide margin.

The search was painstakingly optimised, but the time never did come down below 3 seconds, and the customer got a free system. He’d found an existence proof of a broken requirement, and didn’t need statistical evidence to see him through a lawsuit: he had a watertight case, and he knew it.

Could you give similar examples? Yes, I thought so. Let’s not be shy: we know bad requirements can be disastrous.

© Ian Alexander 2005

Home Page