It's always a pleasure to find two good books on a subject, especially when, as here, they are completely different.

Mike Kuniavsky's is a big, practical, lively, industrial book full of common sense and earthy recommendations.

"A one-on-one usability test can quickly reveal an immense amount of information about how people use a prototype, whether functional, mock-up, or just paper." (p259)

Andy Crabtree's is a slim, quiet, slightly academic book that somehow manages to bring the airy theoretical concepts to life, and to recommend simple and practical ways of using the ideas to get better requirements:

"Quick and dirty Ethnography
Quick and dirty studies are designed to support scoping activities in design, particularly in large-scale settings..." (p89)

In other words, Kuniavsky and Crabtree are approaching the subject of observing users from opposite ends of the telescope - or to be precise, the software house and the research lab. What they share is a conviction that observation is a valuable (if not essential) tool in system development.

Observing the User Experience begins with a true story, in which the author recounts how he 'watched Typhoon's creation, was the person who ran those tests, and watched it disintegrate. Years later, some of the people involved still have feelings of bitterness that so much effort, so many ideas, and so much innovation was abandoned.' Chapter 2 dives right in to the answer: it's called 'Do a Usability Test Now!' (original italics) and hardly needs explaining. Chapters 5 through 16 - most of the book - are about how to conduct large-scale, repeatable usability tests. On the way, major techniques like interviewing ('a universal tool'), contextual inquiry (aka Beyer & Holtzblatt's contextual design), task analysis, card sorting, focus groups and surveys are discussed. 'Usability Test' is thus used in both the narrow sense of carefully-monitored structured interviews involving a prototype of a system's user interface, and more broadly to mean anything that helps to determine whether you have a usable system, and if not how to make it into one.

His advice is direct, and always well illustrated - where necessary by photographs and screenshots, otherwise by tables and examples. For instance, he tells you not to make people predict their behaviour. Don't ask them 'Will you use an online scheduling system?' (answer, umm…) but 'Have you ever used or ever wanted to use an online calendar system such as Yahoo! Calendar or eCalendar?' (answer, yes or no). If more survey designers read Kuniavsky there'd be much better questionnaires and interview reports in the world.

In a way it is a pity that Kuniavsky limits himself to software 'design', as most if not all the techniques he describes apply as much or more to systems of all kinds. Indeed the father of industrial design, Henry Dreyfuss (he of the standard bell telephone and the Hoover vacuum cleaner) applied the same principles in his pioneering work.

Designing Collaborative Systems begins by looking at what Crabtree calls 'The Requirements Problem' - how do you know what you are supposed to build? To a requirements person, this makes the 'Designing' of the title seem a bit strange, but it is only a matter of perspective. People initially tried to apply ethnography directly to system development, hoping that design solutions would emerge straight from descriptions of work. This led to the Human-Computer Interaction (HCI) approach, in which both the world of work and the design of the system were viewed through a user interface design. Crabtree argues that this is doomed to fail, as users do not see the computer solely through the user interface with keyboard, mouse, visual display etc, but through field support, system administration, documentation, training, management and colleagues as well - in fact, these things are enormously important. Kuniavsky would probably agree - certainly, his book does not fall into the trap of only measuring what people think of an interface design, though inevitably that's his focus.

Instead, Crabtree argues that what is needed is a practical, non-ideological approach that can look at 'the real world, real time character of work' (p46). Ethnography emerged from Anthropology, throwing away its grass skirts in the 1920s as people like Malinowski applied it to ordinary social inquiry. Robert Park memorably told his students to 'go get your hands and the seat of your pants dirty' - "i.e. to conduct research through first-hand observation". There are no fixed procedures for this, but you need exploration, inspection, data or material collection, and analysis.

The big problem is that it's all too easy to bring along a pile of theories which only contaminate the analysis, and Crabtree is delightfully rude about 'the generic accounts of the social sciences'. The orthodox approaches take a heap of individual instances observed during fieldwork, and merge them into composite, generic descriptions of what normally happens. The trouble with this is that it destroys the original observations! (Kuniavsky goes to the other extreme, recommending that you videotape every interview.) Of course this is the reason why scientists and practical engineers are so skeptical about sociology - it seems to be all words and theory and no facts. Crabtree proposes something quite radical - looking at work without trying to generalize from it or applying clever theories: "the ethnographer will have to rid himself of disciplinary baggage". Apart from Quick and Dirty Ethnography, he suggests Concurrent Ethnography (work study and system design proceed in parallel); Evaluative Ethnography (doing a short bit of ethnographic study to provide a sanity check of a proposed design); and re-examination of earlier studies. With these four approaches you still have the challenge of how to apply the results to 'give form to design'. Here the problem is communication, and Crabtree becomes quite concrete at this point, discussing how tools like the Designer's Note Pad allow you to model the physical layout of the workplace, the workflow (sequence of activities that link people into a process) and the views of work (co-ordination, awareness of current work status, and plans and procedures) as related Kotonya & Sommerville-style Viewpoints. The issues of supporting business processes, and of computer-supported co-operative work (the CSCW of the book series) are very much open today.

Kuniavsky says in his bibliography 'This book isn't a scientific text, it's a how-to manual'. He sensibly lists just a few good books under each of the headings Qualitative Research, Quantitative Research, Design Philosophy, The Business of Usability, and Software Project Management. His book succeeds admirably in its objective of providing plain and helpful guidance for practitioners who want to know what their users really need.

Crabtree's bibliography runs to more than 7 pages of fine print listing papers, PhD theses and books on everything from the methodological position of symbolic interactionism to Wittgenstein's Philosophical Investigations, so it is a small miracle that his book is as readable as it is. He says in his summary that 'The purpose of this book has been to sensitize the reader to a discrete ensemble of practical strategies and methods for the study of work and the use of .. ethnography in the creative process of design.'

Both books cover topics like Participatory Design (in that instance, in quite similar ways too). Kuniavsky is as you have probably gathered far more concrete in his advice; Crabtree essentially just sketches in the sorts of things you would have to do. While he does discuss what each technique involves and gives some illustrations, it's doubtful whether a reader could then go away and try the techniques out. So although the cover blurb does claim that Crabtree 'provides the practitioner with an invaluable introduction to this approach, and presents a unique set of practical strategies for incorporating it into the design process', it will probably be only the most researchy of designers who will find the time to study this book for hints on how to put ethnography into practice. What he does do - and it seems it's the first time anybody has tried - is to unpack the dense verbiage of ethnomethodologically-informed ethnography (gulp!) into something close to a practical method for system developers.

To sum up, Crabtree has produced a fine pioneering essay which sketches, on the basis of a mountainous literature, how in principle ethnography could migrate from the research lab of a university to the research and development lab of a large software or systems corporation. In contrast, Kuniavsky has written a splendidly practical how-to guide for software developers who need to ensure that their products find a genuine market. Both men are wonderfully well-informed about their fields, and it will be interesting to see how the area of overlap - the use of ethnographic observation and fieldwork involving real users in system design - grows in the coming years.

© Ian Alexander 2003