Judging work that is not yours: heuristics, evidence and design rationale

The studio's last act is judgement. A critic looks at work that is not theirs, and a team reports what a test found, usually defended with a number that is less secure than it looks. Inspection and testing find partly different sets of problems, and no method's findings reproduce across evaluators, so a count measures the evaluator as much as the design.

A defensible judgement is a written rationale that keeps evidence apart from preference. This is the tenth of ten units; it asks who judges the work.

Inspection is cheap, partial, and publishes its own limits

Nielsen and Molich (1990) introduced heuristic evaluation, reporting four experiments in which individual evaluators found between 20 and 51 per cent of the problems they reviewed, while aggregating three to five evaluators did rather well.

Nielsen and Landauer (1993) made that quantitative, fitting problem detection to a Poisson process across 11 studies. They estimated that on a medium-sized project 16 evaluations would be worth their cost, with the best ratio of benefit to cost at four.

Nielsen (1994) then compared several published heuristic sets against a database of usability problems and derived by factor analysis the set with the widest explanatory coverage, an instrument calibrated against findings rather than first principles.

Testing and inspection do not find the same problems

Jeffries, Miller, Wharton and Uyeda (1991) had four groups evaluate the same interface before release, each using a different technique: heuristic evaluation, software guidelines, cognitive walkthrough and usability testing. Heuristic evaluation by several specialists found the most serious problems for the least effort, but also many low-priority ones, so the techniques overlapped without coinciding.

Karat, Campbell and Fiegel (1992) compared empirical testing with individual and team walkthroughs on two office systems. Testing identified the largest number of problems, and a significant number of severe problems the walkthroughs missed. About a third of the significant problems were found by every method. Each is a different net.

The comparison base is itself damaged

Gray and Salzman (1998) reviewed five influential experiments comparing usability evaluation methods and concluded that small flaws called into serious question the field's knowledge of which method was better.

Cockton and Woolrych (2001) found a specific failure inside the method: ninety-nine analysts working in groups applied heuristic evaluation to the drawing editor of an office application, and their predictions were matched against problems extracted from user tests. Heuristics did not help them discover possible problems, and were used most appropriately to confirm problems of low impact or frequency, while 65 per cent of all predictions were false. Their DR-AR model separates discovery from confirmation, which is why a raw count of predictions is not a measure of value.

No two evaluators agree on what they found

Hertzum and Jacobsen (2001) reviewed 11 studies of cognitive walkthrough, heuristic evaluation and thinking-aloud studies, and named what they found the evaluator effect: evaluators applying the same method to the same interface detected markedly different sets of problems, for novices and experts alike, with average agreement between any two evaluators ranging from 5 to 65 per cent and no one method consistently better.

A non-human evaluator has now been put through the same comparison: Guerino, Rodrigues, Capeleti, Mello, Freire and Zaina (2025) ran GPT-4o and human experts over the same web system with Nielsen's heuristics, and the model reported 21.2 per cent of the issues the experts found and produced false positives from hallucination.

Molich, Ede, Kaasgaard and Karyukin (2004) had nine independent organisations evaluate the same website, Microsoft Hotmail, which reported the following.

• problems reported in total: 310.
• problems reported by six or more organisations: two.
• problems reported by no other organisation: 232, or 75 per cent.

Molich and Dumas (2008) repeated the exercise in CUE-4, where seventeen professional teams evaluated a hotel website, nine by usability testing and eight by expert review. The teams reported 340 issues; only nine were reported by more than half the teams, and of the rest 205, or 60 per cent, were unique to a single team, 61 of those assessed as serious or critical. The assumption that two teams running a usability test get the same results is plainly wrong.

Contested question: do the two families find the same problems, or different ones?

One position holds that the methods find different problems, and that testing finds more of the severe ones. Jeffries and colleagues (1991) found heuristic evaluation best for severe problems at low cost. Karat and colleagues (1992) found testing ahead on count and severity.

A second position holds that the family of method makes no practical difference for the problems identified. Molich and Dumas (2008) found none between the issues found by usability testing and by expert review. They could not demonstrate missed problems or false alarms in the expert reviews.

A third position holds that the comparisons cannot settle it. Gray and Salzman (1998) judged the designs of the influential comparisons too flawed to carry the conclusions drawn from them.

A fourth holds that the dispute aims at the wrong variable. Hertzum and Jacobsen (2001) found that evaluators using one method disagreed with one another far more than the methods differed from one another, so asking which method is better without fixing who evaluates is malformed. A study crossing method with evaluator would settle it, using one interface and a problem list fixed in advance by an independent panel, assigning evaluators randomly and running several teams per method, so that evaluator variance is separated from method variance, and reporting detection and false-alarm rates together.

The four positions rest on evidence of unequal strength. Jeffries and colleagues and Karat and colleagues each ran one comparison and found the methods complementary, while Gray and Salzman found five influential comparisons confounded, so the positive claims are weaker. Molich and Dumas ran 17 professional teams, the firmer result, and Hertzum and Jacobsen showed across 11 studies that evaluator variance swamps method variance, which makes any ranking unstable. The strongest objection is that the unit's remedy is untested: nothing here shows that writing a rationale improves a judgement. While the dispute is open, a designer fixes the evaluator, reports agreement between evaluators, and separates a written preference from a finding.

A judgement has to be written as a rationale

MacLean, Young, Bellotti and Moran (1991) set out design space analysis, in which the space around an artefact is represented as Questions, the Options that answer each one and the Criteria for comparing them. Their QOC notation is a coproduct of designing rather than a record of it, produced alongside the artefact and open to inspection by anyone who disagrees.

Conklin and Begeman (1988) built the same assumption into a tool: gIBIS captures early design deliberations as typed networks of issues, positions and arguments, so someone who was not in the room can navigate the reasoning, though they reported the method behind the tool was still incomplete. The artefact is the point: reasoning a team can keep.

Dourish (2006) supplies the warning that keeps such a rationale honest: ethnographic findings are routinely converted into implications for design that the study cannot support, and his own worked example, Miller and Slater's account of internet use in Trinidad, is valued for how it describes the natural and the virtual being separated there rather than for a list of features the software should have. A rationale that claims consequences the evidence does not license repeats the error. A rationale also has to name whose use it judged: a method assuming a sighted, fluent user inherits that standard.

A design judgement can claim provisional theory, not a score

Gaver (2012) argues that research through design is likely to produce theory that is provisional, contingent and aspirational. The honest form for it, he argues, is theory as annotation of realised design examples, particularly a portfolio of related pieces. He warns the field against impulses towards convergence and standardisation. Read beside Dourish (2006), that is the rule for a studio portfolio. The artefact is the evidence, and the writing is the argument attached to it, not a score attached to it.

Where the argument is heading

The first is the portfolio as the unit of argument, where Gaver (2012) proposes annotation of realised design examples and MacLean and colleagues (1991) supplied the notation it descends from. The second is reproducibility treated as a measured property: the comparative usability evaluation series (Molich et al., 2004; Molich and Dumas, 2008) turned team disagreement into a number the field can track, and most problems are reported once, the baseline any evaluation claim has to beat. Guerino and colleagues (2025) show it now applies to a model too.

The gap is worth naming: the authors make the case for QOC and for gIBIS from experience.

The sources this unit rests on

• Cockton, G. and Woolrych, A. (2001). Understanding inspection methods: lessons from an assessment of heuristic evaluation. People and Computers XV. doi:10.1007/978-1-4471-0353-0_11 Ninety-nine analysts applied heuristic evaluation to a drawing editor, and their predictions were checked against problems extracted from user tests. Heuristics did not help them discover possible problems, and 65 per cent of all predictions were false.
• Conklin, J. and Begeman, M. L. (1988). gIBIS: a hypertext tool for exploratory policy discussion. ACM Transactions on Information Systems. doi:10.1145/58566.59297 They built a hypertext tool that records design deliberations as typed networks of issues, positions and arguments, so the reasoning is navigable by someone who was not present. It is the origin of rationale as a kept artefact.
• Dourish, P. (2006). Implications for design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/1124772.1124855 He argues that ethnographic findings are routinely converted into design implications the study cannot support, and that implications for design may itself be the wrong criterion. It is the unit's rule against claiming more for evidence than it carries.
• Gaver, W. (2012). What should we expect from research through design? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/2207676.2208538 He argues that research through design yields theory that is provisional, contingent and aspirational, and proposes theory as annotation of realised design examples, especially portfolios. It gives the unit its honest form for a design claim.
• Gray, W. D. and Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction. doi:10.1207/s15327051hci1303_2 They examined the design of five influential experiments comparing usability evaluation methods and found small flaws that left the conclusions in serious question. It is why this unit refuses to rank methods.
• Guerino, G., Rodrigues, L., Capeleti, B., Mello, R. F., Freire, A. and Zaina, L. (2025). Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation. Human-Computer Interaction: INTERACT 2025. doi:10.1007/978-3-032-05005-2_20 The same web system was evaluated with Nielsen's heuristics by GPT-4o and by human experts. The model found 21.2 per cent of the experts' issues and produced false positives from hallucination.
• Hertzum, M. and Jacobsen, N. E. (2001). The evaluator effect: a chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction. doi:10.1207/S15327590IJHC1304_05 Their review of 11 studies found that evaluators using the same method on the same interface detect markedly different problem sets, for novices and experts alike. Average agreement between two evaluators ranged from 5 to 65 per cent, and no method was consistently better.
• Jeffries, R., Miller, J. R., Wharton, C. and Uyeda, K. M. (1991). User interface evaluation in the real world: a comparison of four techniques. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/108844.108862 Four groups evaluated one interface before release using heuristic evaluation, software guidelines, cognitive walkthrough and usability testing. Heuristic evaluation found the most serious problems for the least effort, but also many low-priority ones.
• Karat, C. M., Campbell, R. and Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/142750.142873 Testing was compared with individual and team walkthroughs on two office systems. Testing found the most problems and several severe ones the walkthroughs missed, while about a third of significant problems were common to every method.
• MacLean, A., Young, R. M., Bellotti, V. M. E. and Moran, T. P. (1991). Questions, options, and criteria: elements of design space analysis. Human-Computer Interaction. doi:10.1207/s15327051hci0603&4_2 They set out QOC, a notation representing a design space as questions, the options answering them and the criteria for comparison. They treat the analysis as a coproduct of designing rather than a record of it.
• Molich, R., Ede, M. R., Kaasgaard, K. and Karyukin, B. (2004). Comparative usability evaluation. Behaviour and Information Technology. doi:10.1080/0144929032000173951 Nine independent organisations evaluated the same website, Microsoft Hotmail. They reported 310 problems, only two of which six or more organisations found, while 232, or 75 per cent, were unique to one organisation.
• Molich, R. and Dumas, J. S. (2008). Comparative usability evaluation (CUE-4). Behaviour and Information Technology. doi:10.1080/01449290600959062 Seventeen professional teams, nine using usability testing and eight expert review, evaluated a hotel website. They reported 340 issues; only nine were reported by more than half the teams, 205, or 60 per cent, were unique to a single team, and 61 of those were serious or critical.
• Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/191666.191729 He compared published sets of usability heuristics against a database of existing usability problems and derived by factor analysis the set with the widest explanatory coverage. It is the revision that produced the list the studio teaches.
• Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of the INTERACT and CHI Conference (INTERCHI). doi:10.1145/169059.169166 Fitting detection to a Poisson process across 11 studies, they estimated that 16 evaluations would be worth their cost on a medium-sized project, with the best ratio of benefit to cost at four.
• Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/97243.97281 In four experiments, individual evaluators found between 20 and 51 per cent of the problems they reviewed, while aggregating three to five evaluators did rather well. It is the origin of the method and of the unit's claim that inspection is cheap but partial.