Mental models and the two gulfs: engineering the mind into the interface

A person forms a mental model of a system and acts on it. Interaction fails at one of two distances (Norman, 1986). The gulf of execution runs between what a person intends and what the interface allows. The gulf of evaluation runs between what the system did and what the person can tell it did.

This unit argues that both are engineering instruments rather than descriptions of mental contents. They predict performance only when the task and the interface are specified precisely enough to be modelled. Where the specification has been done, they yield checkable numbers, and where it has not, a vocabulary that organises design discussion and cannot be falsified.

The same words carry four objects: a designer's simplification, the image the system presents, the theory a user holds, and a researcher's reconstruction of it (Norman, 1983; Norman, 1986). The next unit asks whether the user's action can be modelled that way at all.

Two gulfs, and the stages that carry them

Norman (1986) sets out the account as a mismatch of description. The user has goals in psychological terms. The system presents its state in physical terms, and the difference creates two gulfs. Either side can bridge them: the designer moves the system towards the person, or the user moves the goals towards it.

The bridging is described as seven stages, formed by splitting Norman's original four so that evaluation mirrors execution. He then qualifies the device in a way that retellings drop.

The notion of stages is a simplification of the underlying theory: I do not believe that there really are clean, separable stages. However, for practical application, approximating the activity into stages seems reasonable and useful. (Norman, 1986)

A stage is not observed. It is drawn, and judged by what it lets a designer predict, the first instance of the thesis.

Norman (2013) gives the operative definition of the object being designed. A conceptual model is an explanation, usually highly simplified, of how something works. It need not be complete or accurate, as long as it is useful. Usefulness is a claim about performance on a specified task, so the definition fixes the test.

The observations that make the instrument plausible come from Norman (1983). His studies of calculators, text editors, watches and aircraft found models that were incomplete, unstable, unbounded and unscientific. The inventory behind them matters as much as the tendencies. Some devices were studied extensively, the text editor among them, while others he observed only informally. A tendency established on watches and calculators is a claim about devices of that kind. The chapter does not test whether it holds for a screen. These tendencies are also unattached to a condition or a measure, so they support an expectation, not a prediction.

Three objects share the name of model, and the user meets only the image

Norman (1986) names the three objects design work turns on. The design model is the designer's conceptualisation. The system image is what the user meets, including controls, displays, documentation and messages. The user's model is what the user builds. The decisive claim is that the user's model is formed not from the design model but from the user's interpretation of the system image. A designer therefore cannot transfer a model, only an interface. Where the image arrives through a screen reader, part of it is text the designer wrote, so the model is built from a description of the arrangement rather than from the arrangement.

In a footnote Norman concedes that the term slides between an individual's model and the typical user a designer has in mind.

Wilson and Rutherford (1989) reported the consequence: the term had been applied to several different things, including the model the designer builds and the knowledge the user holds, without the two being separated.

Gentner and Stevens (1983) edited the collection in which much of this vocabulary was fixed. Their introduction characterises the research by domain, approach and methodology. The domains were simple devices, the formalisms came from artificial intelligence, and the methods ranged from protocol analysis to simulation and field observation. Studies differing on all three dimensions do not measure one object.

Where the term came from, and what a designer inherited with it

The term arrived in design from psychology, with a claim attached: Johnson-Laird (1983) argued that people apprehend the world by building inner models of the relations among the objects and events that concern them.

Newell and Simon (1972) supplied the working unit for the programme that cognitive engineering inherited. Their theory separates the task environment, which the experimenter defines objectively, from the problem space, the way a solver represents the task to work on it. The space is a construction by the solver, not a given of the task, so different solvers can build different spaces. An analysis must therefore specify which representation the user will build. That specification is an empirical claim about users rather than an inspection of the interface.

Young (1983) turned the choice into a design-level distinction. A surrogate model is a mechanistic account of how a device works, offered in place of the device for answering questions about its behaviour. A task and action mapping model relates the user's task to the actions the device offers. The volume's editors report Young's argument that the two kinds predict different errors and orders of acquisition. He supported it with protocol analysis and with observations of people using hand calculators, so the instrument is working at the level of a class of devices.

When the specification is done, the instrument predicts

Card, Moran and Newell (1980) are the clearest case of the instrument working. They modelled expert manuscript editing as goals, operators, methods and selection rules, and assessed the model against timed protocols from several users. Two results carry the argument. A few simple rules predicted method choices about 80 per cent of the time. Accuracy was little affected by the grain size of the analysis.

Kempton (1986) shows the other half of the case on a device that is not a computer. Interviewing households about home heating, he identified two working theories. In a feedback theory, the thermostat senses temperature and switches the heat. In a valve theory, the dial controls a flow of heat, so turning it up heats the house faster. The valve theory is contrary to engineering knowledge. Kempton estimates that 25 to 50 per cent of Americans hold it, and he reports that it may be highly functional in day-to-day use.

Kieras and Polson (1985) pushed the specification further. They proposed a method for analysing the knowledge a user must acquire to operate a device, expressed as production rules. Counted in that formalism, the number of productions a task requires is the quantity. From it the method derives predictions of learning time, transfer between systems and the delay between one action and the next. The prediction repeats because the notation is written down. The cost is that the analyst's choices about what counts as one production carry it.

When the specification is not done, nothing can be checked

Staggers and Norcio (1993) reviewed the HCI literature this unit draws on and reported what it does not do. Several terms cover the same territory, among them mental models, conceptual models, cognitive models and causal models. Most research was qualitative, using protocol analysis, and little of the method was reported. Above all the unit of analysis is unsettled. An error might be a property of the user, of the design, or of the interaction. The literature generally treated it as a user defect.

Payne (1991) supplies the empirical challenge. Interviewing users about how cash machines work, he found that some had formed an explanation of the mechanism without prompting. Accounts varied between people, preferring metaphor and their own direct experience of the machine. A specification that speaks of the user's model then describes an average that no user holds.

Rogers (2012) puts the general case: the classical phase looked for laws of interaction while later phases looked for accounts of meaning and practice. She concludes that theory in HCI is often imported and rarely tested.

Contested question: is a mental model a measurable object or a design vocabulary?

The disagreement is not about whether users understand systems. It is about what the word model picks out. Card, Moran and Newell (1980), with Kieras and Polson (1985), stand for the measurable object. With the task, device and operator set fixed, a model of what the user knows yields predictions that can fail. One was confirmed at about 80 per cent. The evidence on the two sides differs in kind. Card, Moran and Newell's 80 per cent is one prediction confirmed on experts using one editor. The opposing case is a set of reviews and interview studies. A single confirmed prediction is not a general law, and a confused terminology is not a measurement.

Against that stand Payne (1991), Wilson and Rutherford (1989) and Staggers and Norcio (1993). Payne found models varying between users, Wilson and Rutherford found the term covering a design model and a user's knowledge without separation, and Staggers and Norcio found the overlap across several names with methods largely unreported. On that reading the instrument makes no testable claim until the terminology is fixed.

A third position dissolves the question. Kempton's (1986) valve theory may work well in ordinary use, so a wrong model can support competent performance, and the presence of a model is not what makes a user effective.

Three pieces of evidence would settle it. First, a model fixed before data collection, with the operator set, the counting rule and the assumed task published. Second, elicitation studies that report method, coding scheme and agreement between coders. They choose one unit of analysis in advance rather than attributing failure to the user by default. Third, predictions stated before the test. Where the literature has done the first, in Card, Moran and Newell (1980), it has a number. Where it has done the second, in Payne (1991), it has variety.

While it is open, treat a mental model as a hypothesis to be written down rather than a description of the user. Fix the operator set and the counting rule before collecting data. Where you cannot, ask which error a user is likely to make rather than assert what the user knows. The strongest counter to this unit's view is that prediction is not the only test, since Kempton's users held a wrong model and worked the thermostat competently.

Where the argument is heading

The direction is from the model in the head to the things that can be counted. Staggers and Norcio (1993) set a measurement agenda in the same place. Report the method, fix the unit of analysis, and keep the user's model distinct from the researcher's reconstruction of it. Norman's own instruments have moved the same way. He narrows the designer's vocabulary to signifiers, which are perceptible and therefore countable (Norman, 2013).

The sources this unit rests on

• Norman, D. A. (1983). Some observations on mental models. Mental Models, 7-14. doi:10.4324/9781315802725-5 Reports that the models people hold are incomplete, unstable, unbounded and unscientific.
• Norman, D. A. (1986). Cognitive engineering. User Centred System Design, 31-62. https://cel.cs.brown.edu/static/pdf/norman-cognitive-engineering.pdf Names the gulfs of execution and evaluation and separates the design model, the system image and the user's model, which is built from the image.
• Norman, D. A. (2013). The Design of Everyday Things: Revised and Expanded Edition. Basic Books. https://www.basicbooks.com/titles/don-norman/the-design-of-everyday-things/9780465050659/ Defines a conceptual model as a simplified explanation that need not be accurate provided it is useful, and introduces the signifier for the perceptible cue.
• Johnson-Laird, P. N. (1983). Mental Models. Harvard University Press. https://archive.org/details/mentalmodelstowa0000john Argues that people understand the world by building inner models. The psychological source of the term.
• Gentner, D. and Stevens, A. L. (eds.) (1983). Mental Models. Lawrence Erlbaum. doi:10.4324/9781315802725 Its introduction characterises the research by domain, approach and methodology, from protocol analysis to simulation and field observation.
• Kempton, W. (1986). Two theories of home heat control. Cognitive Science 10(1), 75-90. doi:10.1207/s15516709cog1001_3 Found two thermostat theories, feedback and valve, the second contrary to engineering knowledge, yet estimated to be held by 25% to 50% of Americans.
• Young, R. M. (1983). Surrogates and mappings: two kinds of conceptual models for interactive devices. Mental Models, 35-52. https://www.taylorfrancis.com/chapters/edit/10.4324/9781315802725-3/surrogates-mappings-two-kinds-conceptual-models-interactive-devices-young Distinguishes surrogate models, mechanistic accounts of how a device works, from task and action mapping models relating the user's task to the actions available.
• Staggers, N. and Norcio, A. F. (1993). Mental models: concepts for human-computer interaction research. Int. J. Man-Machine Studies 38(4), 587-605. doi:10.1006/imms.1993.1028 Reviews the HCI literature: several terms cover the same territory, most studies used protocol analysis with little method reported, and the unit of analysis was left open.
• Payne, S. J. (1991). A descriptive study of mental models. Behaviour and Information Technology 10(1), 3-21. doi:10.1080/01449299108924268 Interviewed users about how cash machines work: some formed explanatory models unprompted, models varied between people, and users preferred explanations from metaphor and direct experience.
• Wilson, J. R. and Rutherford, A. (1989). Mental models: theory and application in human factors. Human Factors 31(6), 617-634. doi:10.1177/001872088903100601 Reports confusion over the concept and its use, arguing that the term has covered both the designer's model and the user's knowledge without separation.
• Kieras, D. and Polson, P. G. (1985). An approach to the formal analysis of user complexity. Int. J. Man-Machine Studies 22(4), 365-394. doi:10.1016/S0020-7373(85)80045-6 Proposes a method for specifying the knowledge a user must acquire, expressed as production rules, from which predictions of learning time and transfer follow.
• Card, S. K., Moran, T. P. and Newell, A. (1980). Computer text-editing: an information-processing analysis of a routine cognitive skill. Cognitive Psychology 12(1), 32-74. doi:10.1016/0010-0285(80)90003-1 Modelled expert editing as goals, operators, methods and selection rules, with method choices predicted about 80% of the time and accuracy little affected by grain size.
• Newell, A. and Simon, H. A. (1972). Human Problem Solving. Prentice-Hall. https://archive.org/details/humanproblemsolv0000newe Sets out the programme in which a solver represents a task as a problem space, holding that the space is constructed by the solver rather than given by the task.
• Rogers, Y. (2012). HCI Theory: Classical, Modern, and Contemporary. Synthesis Lectures on Human-Centred Informatics 5(2), 1-129. doi:10.2200/S00418ED1V01Y201205HCI014 Reviews three generations of HCI theory: the classical phase sought laws of interaction, later phases accounts of meaning and practice.