Inspection methods are cheap, unreliable, and still worth using when the variance is treated as part of the result

Year 2 / Module 15 Interaction techniques and usability evaluation / Unit 8 / Inspection methods are cheap, unreliable, and still worth using when the variance is treated as part of the result

Inspection methods are cheap, unreliable, and still worth using when the variance is treated as part of the result

HCI2107 · Unit 8 of 10

Heuristic evaluation and cognitive walkthrough find real problems at low cost, but the evaluator effect shows that one evaluator's report is not a stable measurement of an interface, so the method is defensible only when its variance is reported and managed by several evaluators and a merge rather than by one expert's severity ranking.

Inspection methods are cheap, unreliable, and still worth using when the variance is treated as part of the result

A consultant delivers a heuristic evaluation: forty ranked problems, and the client acts on the top five. The document reads as a measurement of the interface, and this eighth of ten units argues that it is not one. Heuristic evaluation and cognitive walkthrough surface real problems cheaply, but the evaluator is a large and quantifiable source of variance. The repair is arithmetic and reporting rather than a better expert: aggregate several evaluators, report how far they agreed, and treat a merged severity list as a weak claim.

The method finds real problems at a price a project can pay

Molich and Nielsen (1990) surveyed seventy-seven designers and programmers. Identifying specific, potential problems in a dialogue design was difficult, and evaluators missed many. Nielsen and Molich (1990) ran four experiments in which evaluators applied a small set of principles to an interface. Individual evaluators were mostly quite bad, finding between 20 and 51 per cent of the problems. Aggregating three to five did rather well.

Cheapness is why the method travels. Desurvire, Kondziela and Atwood (1992) surveyed the alternatives to laboratory testing for teams without funds or human factors expertise, and reported that heuristic results were predictive of laboratory performance while laboratory testing identified four to five times as many problems.

Jeffries, Miller, Wharton and Uyeda (1991) had four groups evaluate the same pre-release interface with heuristic evaluation, software guidelines, cognitive walkthroughs and usability testing. Heuristic evaluation by several specialists found the most serious problems with the least effort, but also many low-priority problems.

The method is a family, and its principles are an empirical construct

Lewis, Polson, Wharton and Rieman (1990) derived the cognitive walkthrough. They had four people apply it independently to four alternative interfaces for which empirical usability data existed. The walkthrough asks, at each step of a task, whether a user could tell what to do next, a user the evaluator supplies, and one who cannot see the screen supplies a different step sequence.

The extensions show what the field believes is missing. Blackmon, Polson, Kitajima and Lewis (2002) turned the walkthrough into the Cognitive Walkthrough for the Web, estimating the information scent between a goal statement and a page's headings and links with latent semantic analysis, and testing its predictions of users' success rates in three experiments. Bligard and Osvalder (2013) built an Enhanced Cognitive Walkthrough after naming three weaknesses in the original, and applied it to medical equipment including ventilators, infusion pumps and dialysis machines.

The principles are not a fixed standard either. Nielsen (1994) compared published heuristic sets against a database of real problems and derived a revised set of nine by factor analysis.

One evaluator's report is not a measurement of an interface

The reliability evidence is the empirical core of this unit. Hertzum and Jacobsen (2003) reviewed eleven studies of cognitive walkthrough, heuristic evaluation and thinking-aloud studies, and measured how often two evaluators using the same method agree on a problem. Average agreement ranged from 5 to 65 per cent. The effect held for novice and experienced evaluators, for cosmetic and severe problems, and for detection and severity assessment. No method was consistently better.

The effect is not confined to inspection. Jacobsen, Hertzum and John (1998) had four evaluators independently analyse four videotaped usability test sessions. Of the ninety-three problems detected, only 20 per cent were found by all four and 46 per cent by a single evaluator. None of the problems each named as most severe appeared on all four lists. Usability testing is the yardstick against which inspection is judged, and the yardstick carries the same variance.

Severity, the number the client actually spends, carries the same variance. Nielsen (1992) found single-evaluator ratings very unreliable, but the mean from four evaluators came within half a rating point of the true severity 95 per cent of the time.

Duan, Warner, Li and Hartmann (2024) added a machine to the family. Their Figma plugin applies GPT-4 to a user interface and a written heuristic set. On 51 interfaces against three guideline sets its feedback caught subtle errors and semantic problems that human experts also raised, but its usefulness fell over repeated iterations, and twelve expert designers treated it as a first pass.

Inspection predicts problems; a test confirms them

John and Mashyna (1997) followed a computer scientist with little psychology or human-computer interaction training learning the cognitive walkthrough on a multimedia authoring tool. They compared its predictions with the problems tests found. The method was learnable and flexible, but disappointingly ineffective at predicting actual user problems. Inspection predicts, and a test confirms.

Cockton and Woolrych (2001) tested the split at scale. Ninety-nine analysts in groups applied heuristic evaluation to an office application's drawing editor, and a comparison of their predictions with problems from user tests produced a model separating the discovery of candidate problems from their later confirmation. Heuristics did not support discovery but were applied to confirm problems of low impact or frequency, and they eliminated improbable problems badly: 65 per cent of all predictions were false.

Contested question: is the evaluator effect fatal or manageable?

The first position is Hertzum and Jacobsen's, and it is the most sceptical. An agreement range of 5 to 65 per cent across expertise, severity and method means a single evaluator's report is not a measurement, and presenting it as one makes a claim the evidence cannot support.

The second accepts the effect as a variance to be averaged away. Nielsen and Molich (1990) show that three to five evaluators aggregated find substantially more than one. Aggregation is not a repair of the method but the method, and the dispute is whether that arithmetic is a fix or an admission.

A third position holds that both sides trade on evidence that supports neither conclusion. Gray and Salzman (1998) examined five influential experiments comparing evaluation methods and found small problems in how they were designed and run, which call into question what the field thought it knew, though those outcomes had been used to justify advice to practitioners.

The fourth position is that practice did not follow the alarm. Hollingsed and Novick (2007) found heuristic evaluation and the cognitive walkthrough the most actively used and researched techniques after fifteen years, with formal inspections abandoned or absorbed and the pluralistic walkthrough little studied afterwards.

What would settle the dispute is a design none of these papers runs. The study would measure agreement against a common problem inventory rather than each evaluator's own list, with independent evaluators blind to one another and to the outcome, and would report the proportion of problems that only one evaluator found, with the definition of severity fixed before the study.

The evidence on each side differs in weight. Hertzum and Jacobsen (2003) rest on a review of eleven studies with a 5 to 65 per cent agreement range, while Nielsen and Molich (1990) rest on four experiments and Nielsen (1992) on one severity study, and Gray and Salzman (1998) fault the design of five comparative experiments. The strongest objection is that the unit's compromise has no evidential floor, because agreement as low as 5 per cent leaves a merged list unreliable. A designer should report agreement beside the list while the dispute is open.

Where the argument is heading

The direction of travel is from a single report towards an aggregation whose variance is reported, and towards inspection methods that state a testable prediction. Blackmon and colleagues (2002) show the first, an information-scent estimate compared with measured success rates, and Bligard and Osvalder (2013) the second. Hollingsed and Novick (2007) supply the constraint: methods whose comparative evidence was never settled survived on economics and flexibility, not on superiority. The claim to make of an inspection is narrow: here is a candidate list, here is how far evaluators agreed on it, and here is what only a test can confirm.

The sources this unit rests on

Cockton, G. and Woolrych, A. (2001). Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation. People and Computers XV. doi:10.1007/978-1-4471-0353-0_11 Had 99 analysts in groups apply heuristic evaluation to an office application's drawing editor and compare their predictions with problems from user tests. A model separates the discovery of candidate problems from their confirmation, and heuristics did not support discovery: 65 per cent of all predictions were false.
Blackmon, M. H., Polson, P. G., Kitajima, M. and Lewis, C. (2002). Cognitive walkthrough for the web. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/503376.503459 Transforms the walkthrough into the Cognitive Walkthrough for the Web, estimating information scent with latent semantic analysis. Three experiments test its predictions of users' success rates, so the method states a measurable prediction.
Bligard, L. O. and Osvalder, A. L. (2013). Enhanced Cognitive Walkthrough: development of the cognitive walkthrough method to better predict, identify, and present usability problems. Advances in Human-Computer Interaction. doi:10.1155/2013/931698 Develops an enhanced walkthrough after naming three weaknesses in the original, and applies it to medical equipment such as ventilators and infusion pumps. The method is strengthened where a missed problem costs most.
Desurvire, H., Kondziela, J. and Atwood, M. E. (1992). What is gained and lost when using methods other than empirical testing. Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems. doi:10.1145/1125021.1125115 Surveys the cheap alternatives to laboratory testing. Heuristic results were predictive of laboratory performance while laboratory testing found four to five times as many problems: the cost-and-coverage trade the unit relies on.
Duan, P., Warner, J., Li, Y. and Hartmann, B. (2024). Generating Automatic Feedback on UI Mockups with Large Language Models. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3613904.3642782 Applied GPT-4 to heuristic evaluation on 51 interfaces against three guideline sets, compared its suggestions with human experts' and ran a study with twelve expert designers. Its feedback caught subtle errors but fell in utility over repeated iterations, so an automated evaluator carries its own variance.
Gray, W. D. and Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction. doi:10.1207/s15327051hci1303_2 Reviews the design of five influential experiments comparing evaluation methods. Small faults in how they were designed and run call into serious question what the field thought it knew.
Hertzum, M. and Jacobsen, N. E. (2003). The evaluator effect: a chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction. doi:10.1207/s15327590ijhc1501_14 Reviews eleven studies of cognitive walkthrough, heuristic evaluation and thinking-aloud studies. Agreement between two evaluators using the same method ranged from 5 to 65 per cent across expertise, severity and method, so one report is highly questionable.
Hollingsed, T. and Novick, D. G. (2007). Usability inspection methods after 15 years of research and practice. Proceedings of the 25th annual ACM international conference on Design of communication. doi:10.1145/1297144.1297200 Traces which inspection methods led to use and research, and which became dead ends. Heuristic evaluation and the cognitive walkthrough are the most used and researched; formal inspections were absorbed or abandoned.
Jacobsen, N. E., Hertzum, M. and John, B. E. (1998). The evaluator effect in usability studies: problem detection and severity judgments. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. doi:10.1177/154193129804201902 Has four evaluators independently analyse four videotaped usability test sessions. Of ninety-three problems, only 20 per cent were found by all four and 46 per cent by one, and no top-severity problem appeared on all four lists, extending the effect to testing.
Jeffries, R., Miller, J. R., Wharton, C. and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/108844.108862 Has four groups evaluate the same pre-release interface using heuristic evaluation, software guidelines, cognitive walkthroughs and usability testing. Heuristic evaluation by several specialists found the most serious problems with the least effort but also many low-priority ones.
John, B. E. and Mashyna, M. M. (1997). Evaluating a multimedia authoring tool. Journal of the American Society for Information Science. doi:10.1002/(sici)1097-4571(199711)48:11%3C1004::aid-asi4%3E3.3.co;2-t Follows a computer scientist with little training in psychology or human-computer interaction learning the cognitive walkthrough on a multimedia authoring tool, then compares its predictions with the problems empirical tests found. Learnable and flexible, but disappointingly ineffective at predicting real user problems.
Lewis, C., Polson, P. G., Wharton, C. and Rieman, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90. doi:10.1145/97243.97279 Derives the cognitive walkthrough from a theory of exploratory learning and has four people apply it independently to four alternative interfaces with empirical usability data. The origin of the more structured inspection method.
Molich, R. and Nielsen, J. (1990). Improving a human-computer dialogue. Communications of the ACM. doi:10.1145/77481.77486 Surveys seventy-seven highly motivated industrial designers and programmers and finds that identifying specific, potential problems in a dialogue design is difficult and that their evaluators missed many. The evidence that unstructured expert judgement is a weak instrument.
Nielsen, J. (1992). Reliability of severity estimates for usability problems found by heuristic evaluation. Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems. doi:10.1145/1125021.1125117 Has specialists judge the severity of problems found by heuristic evaluation. Single-evaluator ratings were very unreliable, but the mean from four evaluators came within half a rating point of the true severity 95 per cent of the time.
Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Conference companion on Human factors in computing systems - CHI '94. doi:10.1145/259963.260333 Compares published sets of usability heuristics against a database of real problems and uses factor analysis to derive a revised set of nine. The principles are an empirical construct fitted to a problem corpus, not a fixed standard.
Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90. doi:10.1145/97243.97281 Runs four experiments in which evaluators apply a small set of principles to an interface. Individual evaluators were mostly quite bad, finding between 20 and 51 per cent of the problems, but aggregating three to five did rather well.

Check your understanding & apply it · 4 questions, 1 essay

1A consultant delivers forty problems from one evaluator, each ranked by severity, and the client plans to act on the top five and treat the ranking as a measurement. Which diagnosis fits the unit?

2A team replaces one evaluator with four and averages their severity ratings. What does the unit predict for the severity estimate and for the problem list?

3A team has an inspection list and must decide whether to act on it or wait for a test. Which judgement fits the unit?

4A team argues that a single expert's report is adequate because that expert identified the most severe problem in an earlier project. Which diagnosis fits the unit's evidence?

5Essay A product team asks you to run a heuristic evaluation on a public service website and to deliver a ranked list of defects the team can fix before launch. They have budget for exactly one evaluator and want the ranking to carry the authority of a number. Write an evaluation design and a statement of what the evaluation can and cannot support. Use Molich and Nielsen (1990) and Nielsen and Molich (1990) for why review was made into a method and for the coverage of a single evaluator compared with an aggregate. Use Hertzum and Jacobsen (2003) and Jacobsen, Hertzum and John (1998) to say what the evaluator effect is, how large it is and why it reaches beyond inspection. Use Nielsen (1992) to state what can and cannot be recovered by aggregating severity ratings. Use John and Mashyna (1997), Jeffries, Miller, Wharton and Uyeda (1991) and Desurvire, Kondziela and Atwood (1992) to say why an inspection list is a set of hypotheses rather than a defect list, and what a test adds. Use Gray and Salzman (1998) to say how much weight the comparative literature can carry. Then state what you would report to the team, how you would present severity, and what you would refuse to claim.

The team's request contains a reasonable premise and an unreasonable one. The reasonable premise is that heuristic evaluation is worth running: Molich and Nielsen (1990) surveyed seventy-seven designers and programmers and found that identifying specific, potential problems in a dialogue design is difficult, and Nielsen and Molich (1990) showed that review returns real problems cheaply. The unreasonable premise is that one evaluator's ranked list can carry the authority of a number. In the founding study an individual evaluator found between 20 and 51 per cent of the problems present, and it was aggregating three to five evaluators that did rather well, so the unit of inspection is a panel rather than one expert. With budget for one evaluator I would either buy three to five cheap evaluators or relabel the output. The reliability evidence fixes the limit. Hertzum and Jacobsen (2003) reviewed eleven studies and found that two evaluators using the same method on the same system agreed on 5 to 65 per cent of the problems they reported, with the effect holding for novice and experienced evaluators, for cosmetic and severe problems, and for detection and severity alike. Jacobsen, Hertzum and John (1998) had four evaluators analyse four videotaped test sessions and found that only 20 per cent of ninety-three problems were detected by all four and 46 per cent by a single evaluator, so the variance is not confined to inspection. That is why I would report agreement rather than a ranking. Severity is the part the team most wants and the part least able to bear weight. Nielsen (1992) found single-evaluator severity ratings very unreliable, but the mean of four evaluators came within half a rating point of the true severity 95 per cent of the time. So severity is usable only as an aggregate, and only from several evaluators. The list is a set of hypotheses about the interface. John and Mashyna (1997) had an analyst learn and apply the cognitive walkthrough to a multimedia authoring tool and found it learnable and flexible but disappointingly ineffective at predicting the actual user problems. Jeffries, Miller, Wharton and Uyeda (1991) found that four techniques overlapped only partly, and Desurvire, Kondziela and Atwood (1992) report that laboratory testing identified four to five times as many problems as heuristic evaluation. Gray and Salzman (1998) examined five influential comparisons and found design faults that call into question what the field thought it knew, so I would not claim that inspection outperforms testing or the reverse. To the team I would deliver a candidate list, a statement of how many evaluators produced it and how far they agreed, and a separate short list of the hypotheses that a test with real users should confirm. I would present severity as an interval from an aggregate rather than as a rank, I would mark each item as observed or inferred, and I would refuse to claim that the list is complete, that the ranking is a measurement, or that fixing every item will make the site usable.