GOMS predicts the expert doing a routine task error-free, which is the case most systems are never tested on

A proposal arrives as a slide. It shows two versions of a task and a number: the revision saves about a second per use, and the presenter calls it a GOMS analysis. This seventh of ten units turns from measuring an interaction to predicting one without a user, and asks what such a number is worth. GOMS predicts the execution time of an expert performing a routine task, from a task description, before any prototype exists (Card, Moran and Newell, 1980). It can do that only by refusing the cases a usability test exists to catch. Those cases are expertise, a routine task, a fully specified method and error-free performance. Its cheapness and its boundary are the same fact: the validated successes and the documented limits together tell you when the model can stand in for a participant.

The model is a claim about a task and a user, and it says so itself

Card, Moran and Newell (1980) write execution as the sum of a small operator set: keystroking (K), pointing (P), homing (H), drawing (D), mental preparation (M) and system response (R). The mental operator makes the model more than a keystroke counter, and it was fitted at 1.35 seconds with an R-squared of .84. Pointing averages 1.1 seconds, ranging from 0.8 to 1.5 with distance and target size. Against 32 tasks on 10 systems the prediction error was 21 per cent. The same paper states its restriction without hedging: expert user, routine unit task, method specified in detail, error-free performance. It predicts execution time only. It does not predict the acquisition of a unit task, which may cost 0.5 seconds from memory or about 4 seconds read and located on a display, a reading time fitted to an average reader rather than to one working in a second language.

The validation record is real, and its shape says what it measures

Project Ernestine (Gray, John and Atwood, 1993) is the strongest field evidence: a four-month trial over 72,450 telephone calls compared two operator workstations, and the expectation was that the new one would be faster. GOMS predicted it would be slower, and the data confirmed the prediction, with the difference varying by call category and performance stabilising after the first month. The predicted slowdown was worth almost two million dollars a year.

Gong and Kieras (1994) supply the industrial replication and the qualification that matters: they modelled a commercial application and predicted a 40 per cent execution-time improvement and a 46 per cent learning-time reduction. The evaluation observed 39 per cent and 46 per cent. The relative predictions were excellent. The absolute ones were not. Execution times were overpredicted by more than a factor of two until the estimation rules were corrected, after which the error fell within 9 per cent. A model that ranks two designs is not the same instrument as one that forecasts a duration.

John (1995) reports what the method costs: 12 per cent of project time and about a fifth of interface coding time. A map-digitising case (Haunold and Kuhn, 1994) predicted a 73 per cent task-time reduction worth 730,000 dollars. She also warns that designers without formal psychological training should not use GOMS to predict errors until that validation matures.

What the model cannot see is structural, not incidental

John and Kieras (1996) put the restriction at the centre of the method: GOMS applies only to the routine end of the skill dimension, and the analyst must bring the user's goals from outside the model. The model cannot generate them, and it cannot correct a wrong list. Problem solving is outside it, and so are errors. The authors report that no methodology for predicting what errors users will make has yet been validated as an engineering tool. The restriction is what makes a priori prediction possible.

The field's retrospective makes the same point. Olson and Olson (1990) reviewed cognitive modelling since GOMS. John and Kieras (1996) read them as identifying gaps in cognitive theory that keep modelling from addressing aspects of HCI such as fatigue. They also read them as arguing that such models are the wrong granularity for questions of acceptance and organisational fit. Two decades on, Kotseruba and Tsotsos (2018) survey forty years of cognitive architectures and still catalogue cognition's unmodelled parts.

GOMS is a family, and its tooling bounds what a model can express

GOMS is not one method. John and Kieras (1996) compare the four variants mature enough for design, the keystroke-level model, the original GOMS formulation, NGOMSL and CPM-GOMS. They differ in appearance, architectural assumptions and predictive power, and the companion paper (John and Kieras, 1996) says which fits which design question. Kieras (1988) supplies the methodology that assigns operator times, and Kieras (1997) turns it into the NGOMSL guide a designer follows. Choosing a variant is choosing a question.

The tooling bounds the reach further. Baumeister, John and Byrne (2000) compared three model-building tools, QGOMS, CATHCI and GLEAN3. Each has strengths, but all fall short as a practical design tool. The comparison covered the constructs each tool offers, the design information it returns, its support for building cognitively plausible models, and how easily a model can be modified.

Modelling can also explain, not only predict. Hornof and Kieras (1997) modelled search for a known target in an unordered menu and found it both random and systematic, separating strategies an aggregate time conceals. Byrne (2001) modelled the same menu-selection data with ACT-R/PM, a general architecture with perceptual and motor modules. He reported a zero-parameter fit that was excellent but not perfect, the trade of a general theory for a fitted model.

Contested question: is GOMS a validated instrument or a research programme?

John (1995) holds the strong position: the validity of GOMS for cognitive skill is settled. John and Kieras (1996) support it with case histories across tutoring, clerical, process-control, consumer-electronics and domain-expert systems. They report usable design guidance in real development.

A second position treats the same evidence as establishing a comparison instrument rather than a forecasting one. Gong and Kieras (1994) show relative predictions within one percentage point, while absolute times were overpredicted more than twofold. John and Kieras (1996) draw the implication with an analogy. Execution-time predictions give a best-case basis for comparing designs, as a published fuel-economy rating lets a buyer compare cars even though it is not the mileage they will get. The number answers which design is faster, not how long the task will take.

A third position denies that the restriction is narrow. John and Kieras (1996) argue that most computer-based tasks, even open-ended ones, contain substantial routine components. They add that many tasks evolve from problem solving into routine skill with use. If that is right, the expert-and-routine case is most of what a deployed system is used for.

What would settle the dispute is a design none of these papers runs. Independent analysts would model the same interface blind to each other and to the outcome. An empirical study would run afterwards, so the prediction is not recalibrated against its own data. Independent judges would score both relative and absolute predictions against the same criterion. Modellers without the authors' training would run the exercise again. The validations are largely by the model's own authors, and the sharpest statements of the limits come from the same papers that defend the method.

The evidence is strong and largely self-reported. John (1995) rests on case histories and a cost figure, Gray, John and Atwood (1993) on a four-month field trial over 72,450 calls, and Gong and Kieras (1994) on an industrial application whose relative predictions landed within a percentage point of the observation. The strongest objection is that the unit's thesis can be turned against it, since if most deployed use is routine and expert then the restriction it treats as a limit is the ordinary case, and GOMS stands in for a participant more often than the unit allows. A designer should use GOMS to rank designs and not to forecast a duration while the dispute is open.

Where the argument is heading

The direction is from a hand-built, task-specific model towards an architectural simulation, and from predicting a time towards explaining a strategy. Byrne (2001) shows a general architecture reaching a zero-parameter fit on menu selection. Hornof and Kieras (1997) show modelling surfacing the search strategies behind an aggregate time. Kotseruba and Tsotsos (2018) document how many architectures exist and how many are still developed, a warning as well as a resource. The practical constraint is unchanged since Baumeister and colleagues (2000) found the tools wanting.

The sources this unit rests on

• Baumeister, L. K., John, B. E. and Byrne, M. D. (2000). A comparison of tools for building GOMS models. Proceedings of CHI 2000. doi:10.1145/332040.332485 Compared three tools for creating GOMS models, QGOMS, CATHCI and GLEAN3, on representation, on the design information they provide and on the pragmatics of use. Each has strengths, but all leave something to be desired as a practical design tool.
• Byrne, M. D. (2001). ACT-R/PM and menu selection: applying a cognitive architecture to HCI. International Journal of Human-Computer Studies. doi:10.1006/ijhc.2001.0469 Applies ACT-R/PM, a production-system architecture augmented with perceptual and motor modules, to Nilsen's random menu-selection task, drawing on the original study, two earlier models and eye-tracking data. The final model gives a zero-parameter fit that captures the data excellently but not perfectly.
• Card, S. K., Moran, T. P. and Newell, A. (1980). The keystroke-level model for user performance time with interactive systems. Communications of the ACM. doi:10.1145/358886.358895 Writes expert execution of a routine task as a method over the operators K, P, H, D, M and R whose times are summed, with heuristic rules for placing the mental operator. Against 32 tasks on 10 systems the prediction error was 21 per cent, under restrictions the paper states plainly: expert user, routine unit task, method specified, error-free performance.
• Gong, R. and Kieras, D. (1994). A validation of the GOMS model methodology in the development of a specialized, commercial software application. Proceedings of CHI 1994. doi:10.1145/191666.191782 Applies a formal GOMS approach to a commercial application. Predicted improvements of 40 per cent in execution time and 46 per cent in learning time were observed at 39 per cent and 46 per cent, but absolute execution times were overpredicted by more than a factor of two until the estimation rules were corrected to within 9 per cent.
• Haunold, P. and Kuhn, W. (1994). A keystroke level analysis of a graphics application: manual map digitizing. Proceedings of CHI 1994. doi:10.1145/191666.191779 Applies the keystroke-level model to manual map digitising at a national mapping agency, defining two new operators and testing the model under production conditions. It is the primary study behind the unit's cost figure, which John (1995) reports at second hand.
• Gray, W. D., John, B. E. and Atwood, M. (1993). Project Ernestine: validating a GOMS analysis for predicting and explaining real-world task performance. Human-Computer Interaction. doi:10.1207/s15327051hci0803_3 Compares telephone operators' worktimes on two workstations in a four-month trial over 72,450 calls. GOMS predicted, against expectation, that the proposed workstation would be slower, and the data confirmed it; the difference varied with call category, and the slowdown was worth almost two million dollars a year.
• Hornof, A. J. and Kieras, D. E. (1997). Cognitive modelling reveals menu search in both random and systematic. Proceedings of CHI 1997. doi:10.1145/258549.258621 Models how people search for a known target in an unordered pull-down menu, varying serial and parallel processing, random and systematic search and the number of items within the fovea. It separates two strategies that aggregate response times conceal, showing that a model can explain a strategy as well as predict a time.
• John, B. E. (1995). Why GOMS? Interactions. doi:10.1145/225362.225374 Argues for analytic evaluation as a design instrument, reporting that a GOMS analysis took 12 per cent of project time and about a fifth of interface coding time, and that a map-digitising case predicted a 73 per cent task-time reduction worth 730,000 dollars. It states that for cognitive-skill aspects the validity of GOMS is settled, and warns against using it to predict errors until that validation matures.
• John, B. E. and Kieras, D. E. (1996). The GOMS family of user interface analysis techniques: comparison and contrast. ACM Transactions on Computer-Human Interaction. doi:10.1145/235833.236054 Compares the four GOMS variants, the keystroke-level model, the original GOMS formulation, NGOMSL and CPM-GOMS, by applying all four to a single task example. They differ in appearance, architectural assumptions and predictive power, so the family is not one measure.
• John, B. E. and Kieras, D. E. (1996). Using GOMS for user interface design and evaluation: which technique? ACM Transactions on Computer-Human Interaction. doi:10.1145/235833.236050 Synthesises GOMS for practice, with guidance on which variant fits which design situation and case histories from development. It states the limits directly: routine cognitive skill only, the analyst supplies the goals, error prediction is not yet an engineering tool, and predictions read as a best-case comparison.
• Kieras, D. E. (1988). Towards a practical GOMS model methodology for user interface design. In Handbook of Human-Computer Interaction. doi:10.1016/b978-0-444-70536-5.50012-9 Sets out the practical methodology that gives the models their operator assignments and construction procedures, and that later work cites for how an analysis is carried out.
• Kieras, D. E. (1997). A guide to GOMS model usability evaluation using NGOMSL. In Handbook of Human-Computer Interaction. doi:10.1016/b978-044481862-1.50097-2 Provides the standard guide to running an analytic evaluation with NGOMSL, describing how to build the models and use their predictions in interface design.
• Kotseruba, I. and Tsotsos, J. K. (2018). 40 years of cognitive architectures: core cognitive abilities and practical applications. Artificial Intelligence Review. doi:10.1007/s10462-018-9646-y Surveys forty years of cognitive architectures, settling on a final set of 84 of which 49 are still in active development, plus over 900 practical projects, and notes which aspects of cognition still need research.
• Olson, J. R. and Olson, G. M. (1990). The growth of cognitive modelling in human-computer interaction since GOMS. Human-Computer Interaction. doi:10.1207/s15327051hci0502&3_4 Reviews the growth of cognitive modelling in the GOMS tradition. As read by John and Kieras (1996), it identifies gaps in cognitive theory that keep modelling from addressing aspects of HCI such as fatigue, and argues that such models are the wrong granularity for questions of acceptance and organisational fit.