Observation and contextual inquiry: seeing the work as it is done

Every method so far has asked the participant to produce an account. Observation, the fifth of the ten units, asks for none of it. Its premise is that the consequential features of work are the ones nobody would think to mention, such as the paper list beside the screen. This unit argues that watching work in its own setting reaches routines and workarounds that participants no longer notice and cannot report. The same act carries two measured costs. Watching changes what is watched, and one site is not a population.

A workaround is invisible to the person doing it, because it stopped being a decision long ago. That is the argument for going to look, and the reason the participant cannot simply be asked.

Work as done is not work as described

The premise is older than the method. Suchman (1987) recorded pairs of users working a Xerox photocopier with an expert help system. She found the difficulties arose from the machine's model of the user rather than from the user's competence. Its help system could see only a small part of what its users did. The system assumed action follows a plan. The people it was meant to help improvised, adjusting as the situation unfolded. Her conclusion is that plans are resources used while acting rather than blueprints that determine action. Procedure and practice are therefore two different things.

Crabtree, Nichols, O'Brien, Rouncefield and Twidale (2000) then demonstrated the gap in design work. They report an ethnomethodologically informed study in a university library, in which close analysis of work practice informed design. They set out the difficulty of coordinating that analysis with what system designers need. Their point is that an interview cannot reach what the participant has stopped seeing as an object. Beyer and Holtzblatt (1998) give the practitioner's version of the premise. The field visit opens a design argument rather than testing a design.

A visit is only as good as the record it leaves

Contextual inquiry is the method that follows. Beyer and Holtzblatt (1998) define it as going to the workplace, watching real work and asking questions in the moment rather than afterwards. The participant is treated as the expert on their own work. Its output is a work model. The model represents how the work is organised, including its informal practices, its exceptions and its dependence on other people. The model, not the transcript, is the unit of analysis.

Shadowing is the austere version. McDonald (2005) sets it out as following one person through a working day, and she is candid about the costs. The technique is intrusive, recording is difficult, and access is the binding constraint. She built a threefold classification of modes of shadowing from every study that had used the technique. A study that follows a person for one day samples one day.

The finding the method exists to produce is the workaround. A receptionist keeps a paper list beside the screen because the search is slower than her memory of it, and the list is what she uses when the connection drops. She will not offer it as a finding, because she no longer sees it as a decision. That example is constructed for this unit. The pattern is the one Crabtree and colleagues (2000) report from a real site.

Heath, Hindmarsh and Luff (2010) argue that video preserves embodied and interactional detail that written notes flatten. One example is the timing of a glance against the spoken turn it answers. Their guide covers access and consent, camera placement and the sequential analysis of the recording. The choice between a notebook and a camera is a choice about which evidence can be recovered later.

The watcher is in the scene

The first limit is reactivity, and the literature that names it is in poor shape. Adair (1984) reviewed the Hawthorne literature. He found that the original studies do not reveal the effect they are famous for, because too many variables went uncontrolled. The effect is described inconsistently across textbooks. Of thirteen studies designed to produce one, only four succeeded, and all four used adult participants. His proposal is that people define an identified situation for themselves and act on that interpretation.

McCambridge, Witton and Elbourne (2014) reviewed the field systematically and reached a similarly destabilising conclusion. Across nineteen purposively designed studies, twelve found some statistically significant evidence of an effect. Methods, contexts and findings were highly heterogeneous. The authors conclude that there is no single Hawthorne effect, and that little can be securely known about its conditions, mechanisms or magnitude. Reactivity therefore cannot be corrected by a formula. It can only be recorded.

Levitt and List (2011) re-examined the most famous instance at source. They recovered the original illumination-experiment data, long thought destroyed. They found that the descriptions everyone repeats are fictional: productivity did not rise every time the light changed. They report only subtle and ambiguous hints of an effect, and they propose a test based on excess responsiveness to experimenter-induced variation. The canonical example of observer effects is itself contested.

One site is one site

The second limit is scope. Rogers and Marshall (2017) set out what studies conducted outside the laboratory trade. Control is lost, ecological validity is gained, and the trade is a design decision to be stated rather than a weakness to conceal. A field study is not a test of a system. It is evidence about how a practice works, in a setting, with the conditions that setting supplies. Rogers (2011) makes the positive case. The field should strive for theory adequate to what is observed in real settings, rather than applying laboratory theory to them. Crabtree and colleagues' study was one university library. What it established is how work is organised there, not a rate that transfers to libraries in general. A single site is then not only a limitation to be confessed. It is the setting in which the phenomenon exists at all. The sites that agree to be watched tend to be the well-resourced ones, so the same system may run on a shared phone where the study never reached.

Contested question: is a field study's job to yield implications for design?

Everyone here agrees that fieldwork can be done well and disagrees about what it is for. The first position holds that relevance is the test. Crabtree and colleagues (2000) are the demonstration, reporting a library study whose analysis produced requirements interviews had missed. Randall, Harper and Rouncefield (2007) put the analytic work between the field notes and the design decision at the centre of the method. That move decides whether the study matters.

The second position holds that this test misdescribes the work. Dourish (2006) argues that HCI has fallen into judging ethnographic studies by the design implications they yield. That criterion misconstrues ethnographic inquiry and, ironically, misses where it does provide insight. Crabtree, Rodden, Tolmie and Button (2009) push further. Design's recruitment of ethnography as a problem-finding technique is harmful, and the value lies in methodical analytic treatment of the setting.

A third position names why the argument resists resolution. Button and Dourish (1996) examine the paradox of using ethnomethodology in systems design. An approach that refuses to treat social order as an engineerable system is asked to inform engineering. The tension is structural, so the demand for implications is a category error rather than a weak criterion.

What would settle it is a comparison, and this unit's literature contains none. Two teams with comparable questions would take part. One would receive an analytic account and one a bulleted list of implications. They would be measured on the same downstream outcome, such as requirement coverage or design quality. If reports without implications produced better designs, the strict position would be weakened. If not, the criterion would be vindicated empirically.

The disagreement is about purpose rather than method, and the evidence is argumentative on both sides. Crabtree and colleagues argue from one applied case that produced requirements an interview had missed. Dourish and Crabtree and colleagues argue from the theory of what ethnographic inquiry is for. Button and Dourish argue that the tension is structural. While the dispute is open, read a field study for its account of the setting, and supply the implications by argument rather than expecting them on a list. The strongest objection is that a participant can describe their own work well when the question arrives in the setting where the work happens.

Where the argument is heading

The direction is towards stating the trade explicitly and treating the field as a place where theory is made rather than applied. Heath and colleagues (2010) have moved the recording question from technique to argument, showing what notes cannot recover and what video can. Rogers (2011) supplies the epistemic case, and Rogers and Marshall (2017) turn the control-validity trade into something a methods section is expected to declare. The reactivity literature points the same way. McCambridge and colleagues (2014) have shown that there is no single participation effect. A study must therefore describe what happened at the site instead of naming a bias. A methods section should now say what was observed and how the observer was received. It should say what the site cannot support a claim about, and which of the two contested purposes the study serves.

The sources this unit rests on

• Beyer, H. and Holtzblatt, K. (1998). Contextual Design: Defining Customer-Centred Systems. Morgan Kaufmann. url:https://archive.org/details/contextualdesign00beye Defines contextual design and with it contextual inquiry, in which the researcher goes to the workplace, watches real work and asks questions in the moment rather than testing a prototype. Its device is the work model, built from how work is actually organised rather than from what participants say about it.
• Suchman, L. A. (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press. url:https://archive.org/details/planssituatedact0000such Argues from studies of people using a photocopier and its expert help system that plans do not determine action but are resources used in a situation. It is the theoretical basis for the gap between procedure and what is actually done.
• Crabtree, A., Nichols, D. M., O'Brien, J., Rouncefield, M. and Twidale, M. B. (2000). Ethnomethodologically informed ethnography and information system design. Journal of the American Society for Information Science. doi:10.1002/(sici)1097-4571(2000)51:7<666::aid-asi8>3.0.co;2-5 Reports an ethnomethodologically informed study in a university library and shows how close analysis of work practice informed design, while setting out the problems of coordinating that analysis with what system designers need.
• Dourish, P. (2006). Implications for design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1124772.1124855 Argues that judging ethnographic and other field studies by the design implications they yield misconstrues what such inquiry produces and undervalues the models of social settings it provides. It is the primary statement that describing a context is a contribution in its own right.
• Crabtree, A., Rodden, T., Tolmie, P. and Button, G. (2009). Ethnography considered harmful. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1518701.1518835 Assesses the place of ethnography in systems design and argues against the way design has recruited new ethnographies as a problem-finding technique, locating the value in methodical analytic treatment of the setting.
• Button, G. and Dourish, P. (1996). Technomethodology. Proceedings of the SIGCHI conference on Human factors in computing systems. doi:10.1145/238386.238394 Examines the paradox of applying ethnomethodology within systems design, where an approach that refuses to treat social order as an engineerable system is asked to inform engineering. It names why observational findings resist conversion into requirements.
• Randall, D., Harper, R. and Rouncefield, M. (2007). Fieldwork for Design. Computer Supported Cooperative Work. doi:10.1007/978-1-84628-768-8 A book-length treatment of fieldwork for design covering the problem of moving from what was observed to an implication for the system. Its argument is that the analytic work between the notes and the design decision decides whether the study matters.
• McCambridge, J., Witton, J. and Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of Clinical Epidemiology. doi:10.1016/j.jclinepi.2013.08.015 Reviews nineteen purposively designed studies of research participation effects and finds that twelve showed some statistically significant effect, but that methods, contexts and findings were highly heterogeneous. The authors conclude that there is no single Hawthorne effect and that little can be securely known about its magnitude.
• Adair, J. G. (1984). The Hawthorne effect: A reconsideration of the methodological artefact. Journal of Applied Psychology. doi:10.1037/0021-9010.69.2.334 Reviews the Hawthorne literature and finds that the original studies do not reveal the effect they are famous for, that textbooks describe it inconsistently, and that of thirteen studies designed to produce one, only four, all using adult participants, succeeded.
• Levitt, S. D. and List, J. A. (2011). Was there really a Hawthorne effect at the Hawthorne plant? An analysis of the original illumination experiments. American Economic Journal: Applied Economics 3(1), 224-238. doi:10.3386/w15016 Recovered the original illumination-experiment data, thought destroyed, and found that the widely repeated descriptions of productivity rising with every change of light are fictional, with only subtle and ambiguous hints of an effect. It is the strongest evidence that the canonical example of reactivity is a misreading.
• McDonald, S. (2005). Studying actions in context: a qualitative shadowing method for organisational research. Qualitative Research. doi:10.1177/1468794105056923 Sets out shadowing as following one person through a working day, and discusses its practical problems: intrusiveness, the difficulty of recording, and access.
• Heath, C., Hindmarsh, J. and Luff, P. (2010). Video in Qualitative Research: Analysing Social Interaction in Everyday Life. SAGE Publications. doi:10.4135/9781526435385 A guide to video-based research covering access and consent, camera placement and recording, and the sequential analysis of interaction. Its argument is that video preserves embodied and interactional detail that field notes flatten.
• Rogers, Y. and Marshall, P. (2017). Research in the Wild. Synthesis Lectures on Human-Centred Informatics. doi:10.1007/978-3-031-02220-3 Synthesises what studies conducted outside the laboratory can and cannot establish, stressing the loss of experimental control and the gain in ecological validity. It argues that the trade is a design decision to be stated rather than a weakness to be hidden.
• Rogers, Y. (2011). Interaction design gone wild. Interactions. doi:10.1145/1978822.1978834 A position paper arguing that the field should strive for theory adequate to what is observed in real settings rather than applying laboratory theory to them, and that in-the-wild studies are where such theory is built.