A usability score is a model output, and both the instrument and the sample size carry the assumptions

A team reports that its product scores 74 on the System Usability Scale and that five users found the problems worth fixing. Both arrive as findings about usability. Neither is one. This last of ten units turns from the methods to the numbers they produce. The 74 is the output of a scoring model applied to ten answers. Its meaning depends on the construct the questionnaire was validated against, and on the group it is compared with. The problem count is the output of a statistical model of how problems are met, and it rests on an assumption about how evenly problems are spread across users, and on who the five were: a panel from one office leaves the older user and the screen reader user outside the count.

The consequence is not that scores should be abandoned: a score belongs beside the model beneath it, which names the construct, the comparison group and the sample rule. One literature built instruments for perceived usability and workload, the other a formula for how many participants an evaluation needs.

The instrument measures the construct it was validated against

Brooke (1996) introduced the System Usability Scale. His chapter argues that usability is not a quality that exists in any real or absolute sense, but a general quality of appropriateness to a purpose. That quality cannot be specified without first naming the intended users and their tasks. The instrument that became the field's default is a ten-item questionnaire yielding a single value (Lewis and Sauro, 2009; Lewis, 2018).

Bangor, Kortum and Miller (2008) assembled nearly ten years of SUS data from numerous products across the development lifecycle, and found the scale highly robust and versatile, the empirical case that made it a standard. What the scale collects is a user's subjective rating of a product's usability, so the score supports a claim about judgement rather than about the interface.

Lewis and Sauro (2009) then showed that the single score conceals structure. Factor analysis of two independent data sets produced two factors, eight usable items and the two learnability items, with reliabilities of .91 and .70 and correlations with the overall score of .985 and .784. The authors allow practitioners to keep the score as it stands, but it is a summary that discards a recoverable distinction.

A score needs a comparison group, and the measures do not move together

A raw score means nothing alone, so the instruments arrive with norms. The usual SUS benchmark is a mean of 68 with a standard deviation of 12.5, and whether it transfers is itself a result. Hyzy and colleagues (2022) meta-analysed 117 SUS scores from 114 digital health apps and found the benchmark largely held, the mean 68.05 once physical activity apps were set aside. Nicora and colleagues (2025) meta-analysed 67 studies of robotic rehabilitation devices and found it did not transfer: an average of 74, a statistically significant deviation from the benchmark in most device categories, from studies averaging only 13 participants.

Workload uses a different model. Hart and Staveland (1988) built the NASA Task Load Index from a multi-year programme of subjective evaluations of ten workload-related factors across sixteen experiments. It reduces to a rating scale that combines the magnitude and sources of six factors into one estimate. Grier (2015) later gave that weighting decision a norm by meta-analysing more than a thousand global Task Load Index scores from more than two hundred publications.

The strongest evidence that a usability score is not usability is that the field's own measures are only weakly related. Hornbaek and Law (2007) meta-analysed correlations among usability measures computed from the raw data of 73 studies. Effectiveness and efficiency correlated at .247, efficiency and satisfaction at .196, and effectiveness and satisfaction at .164, each with a wide confidence interval. The three are largely independent, so no single score stands for the construct as a whole.

The problem-discovery formula is a model with an assumption inside it

The other half of the argument is the sample. Nielsen and Landauer (1993) fitted problem detection, as a function of the number of users tested or evaluators employed, across eleven studies. They found the process well modelled as a Poisson process. The model estimates how many problems remain and where a further session stops paying for itself. In their worked case sixteen evaluations were worth their cost, with the best benefit-to-cost ratio at four.

Virzi (1992) supplied the empirical anchor, reporting three experiments in which four or five participants detected around 80 per cent of the problems and severe problems tended to appear early. That is a result about three studies. The five-user rule is the generalisation of it into a budget assumption.

The weak part is the assumption, not the arithmetic

The model assumes that every user meets every problem with the same probability and independently of the others. Lewis (1994) tested Virzi's claims against an independent study and found the second supported, the first only partly, and the third unsupported. The 80 per cent figure survives only while the average likelihood of detecting a problem lies between .32 and .42, as in Virzi's data. The arithmetic is right. The single average rate fails, because problems range from those almost everyone meets to those one person in twenty meets.

Faulkner (2003) made the consequence vivid: she tested sixty users and sampled random sets of five or more. Some sets of five found 99 per cent of the problems, others only 55 per cent. With ten users the worst set found 80 per cent, and with twenty the worst found 95 per cent. The rule that promises coverage most of the time does not say which case is in hand.

Schmettow (2012) states the conclusion directly: magic numbers are strictly hocus-pocus, and usability studies must test many more subjects than is usually assumed. Borsci and colleagues (2013) reviewed the debate. They argue that the five-user assumption is often adopted with little understanding of its basis, and they offer a grounded procedure for deciding the sample instead. The rule is not false. It is a point estimate whose parameter varies from product to product.

Contested question: is the five-user rule a description or a decision?

The first position, held by Nielsen and Landauer (1993) and Virzi (1992), is that the model describes problem discovery well enough to plan an evaluation and that diminishing returns are robust. On this reading five users are a reasonable default and the formula says when to stop.

The second position, held by Lewis (1994), Faulkner (2003) and Schmettow (2012), is that the assumption of one homogeneous detection rate is the whole argument, and that it fails. The same five users find 55 per cent of the problems in one sample and 99 per cent in another, and the required sample rises sharply as the average detection rate falls.

The third position, held by Borsci and colleagues (2013), says the dispute is about application rather than mathematics. The formula needs an estimate of the detection rate, and the field's error was to detach the number five from the estimation procedure that gave it meaning.

The positions are closer than they look. The model is not contradicted by the variability, since variance around a Poisson rate is what a Poisson model predicts. The live question is whether a single default can stand in for an estimate made per product. What would settle it is a study none of these papers runs. It would measure the distribution of per-problem detection rates across real products, report its spread rather than its average, and show how much the required sample changes across it.

The sides rest on different evidence. Nielsen and Landauer (1993) fitted a Poisson model across eleven studies and Virzi (1992) tested three experiments, while the critique rests on Lewis's (1994) reanalysis, Faulkner's (2003) resampling and Schmettow's (2012) statistical argument. The strongest objection is that the instrument half is better behaved than the unit implies, since Bangor, Kortum and Miller (2008) found the scale robust across products and Hyzy and colleagues (2022) found its benchmark transferred to digital health apps. A designer should report the construct and the sample rule beside the score while the dispute is open.

Where the argument is heading

The direction is from a fixed number to a stated procedure. Borsci and colleagues (2013) replace the five-user default with a grounded procedure. Schmettow (2012) ties the sample to a coverage target rather than a habit. Hyzy and colleagues (2022) and Nicora and colleagues (2025) test the benchmark in a new domain and reach opposite conclusions. The claim to make stays narrow: the number, the construct the instrument measures, the comparison group, and the sample rule with its assumption.

The sources this unit rests on

• Bangor, A., Kortum, P. T. and Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. doi:10.1080/10447310802205776 An analysis of nearly ten years of SUS data from numerous products across the development lifecycle found the scale highly robust and versatile. The reliability case that made it a default.
• Borsci, S., Macredie, R. D., Barnett, J., Martin, J., Kuljis, J. and Young, T. (2013). Reviewing and extending the five-user assumption. ACM Transactions on Computer-Human Interaction. doi:10.1145/2506210 Reviewed how many participants interaction testing needs and how the original research was applied. Argues the five-user assumption is often adopted with little understanding of its basis, and proposes a grounded procedure instead of a fixed number.
• Brooke, J. (1996). SUS: a quick and dirty usability scale. In Usability Evaluation in Industry. Taylor and Francis. doi:10.1201/9781498710411-35 Introduced the System Usability Scale as a quick and reliable measure of perceived usability. The chapter argues that usability does not exist in any real or absolute sense and cannot be specified without naming users and tasks.
• Faulkner, L. (2003). Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav. Res. Methods Instrum. Comput. doi:10.3758/bf03195514 Tested sixty users and sampled random sets of five or more from the whole. Some sets of five found 99 per cent of the problems and others only 55 per cent, and the worst set found 80 per cent at ten users and 95 per cent at twenty.
• Grier, R. A. (2015). How high is high? A meta-analysis of NASA-TLX global workload scores. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. doi:10.1177/1541931215591373 Meta-analysed more than a thousand global Task Load Index scores from more than two hundred publications, reporting distributions and central tendencies to help practitioners interpret a score. A workload number becomes interpretable only against a norm.
• Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In Human Mental Workload. North-Holland. doi:10.1016/s0166-4115(08)62386-9 Reports a multi-year programme of subjective evaluations of ten workload-related factors across sixteen experiments, reduced to one instrument. The rating scale combines the magnitude and sources of six factors into a single estimate, a weighting decision a lone score conceals.
• Hornbaek, K. and Law, E. L. C. (2007). Meta-analysis of correlations among usability measures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1240624.1240722 Meta-analysed correlations among usability measures from the raw data of 73 studies. Effectiveness and efficiency correlated at .247, efficiency and satisfaction at .196, and effectiveness and satisfaction at .164, so the aspects are largely independent.
• Hyzy, M., Bond, R., Mulvenna, M., Bai, L., Dix, A., Leigh, S. and Hunt, S. (2022). System usability scale benchmarking for digital health apps: meta-analysis. JMIR mHealth and uHealth. doi:10.2196/37290 Meta-analysed 117 SUS scores from 114 digital health apps published between 2011 and 2021 to test whether the usual benchmark of 68 applies. The distribution was skewed by physical activity apps, but excluding those the mean was 68.05, so the benchmark held for this domain.
• Lewis, J. R. (1994). Sample sizes for usability studies: additional considerations. Human Factors. doi:10.1177/001872089403600215 Re-examined Virzi's three claims against an independent usability study, finding diminishing returns supported, the eighty per cent claim only partly supported, and the early-severe-problems claim unsupported. The eighty per cent figure holds only while average detection likelihood lies between .32 and .42.
• Lewis, J. R. (2018). The System Usability Scale: past, present, and future. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2018.1455307 Reviews the scale from its inception to recent research and records that it is the most widely used standardised questionnaire for perceived usability. It is the source for the claim that the SUS became the field's default.
• Lewis, J. R. and Sauro, J. (2009). The factor structure of the System Usability Scale. Lecture Notes in Computer Science. doi:10.1007/978-3-642-02806-9_12 Factor-analysed two independent SUS data sets against the assumption that the ten-item scale is unidimensional, finding two factors, eight usable items and two learnability items, with reliabilities of .91 and .70 and correlations with the overall score of .985 and .784. The single score discards a recoverable distinction.
• Nicora, G., Santangelo, G., Marzero, G., Dagliati, A., Aprile, I. and Quaglini, S. (2025). Exploring Usability Standards in Robotic Rehabilitation: A Meta-Analysis of System Usability Scale Scores. 2025 International Conference on Rehabilitation Robotics (ICORR), 476-482. doi:10.1109/icorr66766.2025.11063026 Meta-analysed 67 studies of robotic rehabilitation devices and found an average of 74, a significant deviation from the benchmark of 68 in most device categories, so the benchmark does not transfer unchanged.
• Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/169059.169166 Fitted problem detection as a function of the number of users or evaluators across eleven studies and found it well modelled as a Poisson process. In the worked case sixteen evaluations were worth their cost, with the best benefit-to-cost ratio at four.
• Schmettow, M. (2012). Sample size in usability studies. Communications of the ACM. doi:10.1145/2133806.2133824 Argues that magic numbers in usability testing are unfounded and that studies must test many more participants than is usually assumed. Draws on the statistical properties of problem discovery rather than on a single default.
• Virzi, R. A. (1992). Refining the test phase of usability evaluation: how many subjects is enough? Human Factors. doi:10.1177/001872089203400407 Reports three experiments relating the proportion of problems identified to the number of participants, finding about eighty per cent of problems detected with four or five subjects and diminishing returns beyond that.