How many participants: power, saturation and the five-user rule

The question "how many participants do I need" sounds like a request for a number. It is really a request for a reason, because a size offered without one cannot be checked or defended. This third of the ten units argues that every defensible sample size is the output of a stated rule. A power calculation settles a quantitative claim, and a saturation argument settles a qualitative one. The constant of five participants is neither. It is a property of one model fitted to one family of studies. The number survived long after the reasoning behind it was shown to be conditional.

A number is a conclusion drawn from a rule. Where the rule is unstated, the number cannot be examined, and a sample size nobody can examine is a habit dressed as a finding.

The claim that five users are enough came from arithmetic, not folklore. Nielsen and Landauer (1993) fitted a Poisson process to eleven studies of usability testing and heuristic evaluation. They assumed that each evaluator has an independent chance of finding each problem, and found the proportion of problems detected rising steeply and then flattening. Virzi (1992) reached the same conclusion empirically. He counted how many problems each additional participant revealed, and reported most appear early with diminishing returns. Neither presents five as a threshold a study has met or failed.

The five-user rule came from a model, and its own arithmetic undermined it

Lewis (1994) tested Virzi's three claims against an independent usability study, finding diminishing returns supported but the eighty per cent claim only partly: four or five participants uncover about eighty per cent of problems only where the average chance of detecting a problem sits between 0.32 and 0.42, as in Virzi. Below that, more participants are needed. Faulkner (2003) turned the dispute into a direct test. She ran sixty participants and drew random sets of five. Some sets uncovered 99 per cent of the problems and others only 55 per cent. At ten participants the worst set found 80 per cent, and at twenty, 95 per cent. The finding is the spread, not the average: five users is one draw from a distribution that sometimes lands far below the mean. The distribution is fitted to one group of users, so it does not cover a population that contains older participants, other devices and other access needs.

The recommended number moves with the assumptions

The recommendation shifts when the model does. Hwang and Salvendy (2010) reviewed the problem-discovery literature and derived a different figure from the same family of models. They concluded that roughly ten evaluators are needed (their 10 plus or minus 2 rule) to reach an 80 per cent problem-discovery rate. Schmettow (2012) changed another assumption. He allowed problems to differ in how easy they are to find, and found the number needed rising well beyond the customary figures. He argues that the magic numbers are not defensible. Borsci and colleagues (2013) accept a small number, but only within a stated model and with the uncertainty reported. They estimate that curve for the system in hand and report the interval around it.

A quantitative claim is settled by a power calculation

When the claim is a comparison, the rule is a power calculation, and it can be stated before anyone is recruited. Cohen (1992) set out the arithmetic in three tables. The tables give the sample size needed for small, medium and large effects at 80 per cent power and conventional significance across eight standard statistical tests, the conventions most later power calculations cite. Button and colleagues (2013) estimated the median statistical power of studies in the neurosciences. They worked from published meta-analyses and found it between roughly 8 and 31 per cent. Low power reduces the chance of detecting a true effect. It also makes a significant result less likely to be true, so the consequences are overestimated effects, poor reproducibility and wasted research.

A qualitative claim is settled by saturation, when it can be estimated

For a study that builds an account from interviews, the rule is saturation: the point at which further sessions stop producing new themes. Guest, Bunce and Johnson (2006) ran sixty interviews with women in two West African countries and tracked when new themes stopped appearing. They found most themes present within the first twelve. The basic elements of metathemes were visible as early as six, and saturation arrived well before the full set. Hennink and Kaiser (2022) tested that range in a systematic review of 23 saturation studies. They found saturation commonly reached at nine to seventeen interviews and at four to eight focus groups. The range depends on how homogeneous the sample is and how specific the question. The two give a stated rule and a measured range, but the range is wide and its conditions are part of the finding. Wutich, Beresford and Bernard (2024) integrated the guidance for ten kinds of analysis and found that one range conceals four quantities. Theme saturation arrived near nine interviews, meaning saturation near twenty-four, theoretical saturation at twenty or more, and metatheme saturation at twenty to forty per site. Reflexive thematic analysis and ethnography carry little agreed guidance. A target defended with one range is defended with a number that answers a different question.

Practice reports a number no single rule produces

Caine (2016) supplies the evidence on practice. She reviews the approaches available, among them power analysis, saturation and cost-return reasoning. She then analysed every paper published at CHI in 2014. Sample sizes ran from one to nine hundred and sixteen thousand, and the most common value was twelve. Her recommendation is that the number should match the method to the analysis, and that the paper should report how it was chosen. A range spanning six orders of magnitude cannot be the output of one calculation.

Contested question: is the rule stated before the study or invented after it?

The deepest disagreement is not about which number but about whether a number can honestly be fixed in advance. The discovery-curve tradition holds that it can. Nielsen and Landauer (1993) and Hwang and Salvendy (2010) treat the number as the output of assumptions set before data collection. Hennink and Kaiser (2022) give measured saturation points that imply a range a planner can use. Braun and Clarke (2020) hold that it cannot. They argue that data saturation is conceptually incompatible with reflexive thematic analysis. Meaning is produced by the analyst's interpretation rather than lying in the data waiting to be exhausted. They propose information power instead, the rationale Malterud, Siersma and Guassora (2016) derive from what a sample holds for the study. Varpio and colleagues (2017) strengthen the sceptical case. They examine four markers that qualitative researchers use, thematic emergence, triangulation, saturation and member checking. They argue each is often invoked as a ritual rather than performed as claimed.

Argument will not settle this, because the sides use the same word for different objects. What could settle it is a study that fixes a stopping rule in advance. It would record where it would have stopped, and report what a different rule would have missed on the same data. Their empirical tests operationalise saturation differently, which is why Braun and Clarke can deny that the concept travels.

The evidence is not symmetric. Nielsen and Landauer's number comes from a model fitted to eleven studies. Hwang and Salvendy's comes from a review that assumes the same independence. Hennink and Kaiser offer the strongest empirical footing here, because they compile twenty-three studies that tested saturation directly. Braun and Clarke argue from the logic of reflexive analysis rather than from data. State the rule before collecting, record where it would have stopped, and report what a different rule would have found on the same data. The strongest objection is that no quantity can be fixed in advance for reflexive work, because the analyst's reading is the result.

Where the argument is heading

The direction is towards stating the basis of the number and its uncertainty rather than asserting the number. Borsci and colleagues (2013) ask for the discovery curve of the specific system to be estimated, with its uncertainty reported. Hennink and Kaiser (2022) replace a single convention with a range that carries its conditions. Button and colleagues (2013) show that an underpowered study distorts the effect rather than merely adding noise. Braun and Clarke (2020) push the other way, towards a rationale that refuses a quantity at all. A reader may soon expect to see which rule produced a number and what assumptions it rested on. That is not a stricter five-user rule but the abandonment of the rule for the argument it replaced.

The sources this unit rests on

• Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. CHI '93. doi:10.1145/169059.169166 Fitted a Poisson process to eleven studies of usability testing and heuristic evaluation, assuming each evaluator finds each problem independently, and found detection rising steeply then flattening. The arithmetic behind the claim that five users are enough.
• Virzi, R. A. (1992). Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? Human Factors. doi:10.1177/001872089203400407 Recorded how many problems each additional participant revealed, finding most appear early with diminishing returns. The first empirical statement of the five-participant convention and the study later work attacks.
• Lewis, J. R. (1994). Sample Sizes for Usability Studies: Additional Considerations. Human Factors. doi:10.1177/001872089403600215 Tested Virzi's three claims against an independent usability study: diminishing returns held, the eighty per cent claim only where the average chance of detecting a problem ranged from 0.32 to 0.42, and the severe-problems claim failed. The earliest published challenge to the five-user assumption.
• Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. doi:10.3758/bf03195514 Ran sixty participants and drew random sets of five from the whole. Some sets uncovered 99 per cent of the problems and others only 55 per cent, and the worst set of twenty reached 95 per cent. The variability, not the average, is the finding.
• Hwang, W. and Salvendy, G. (2010). Number of people required for usability evaluation. Communications of the ACM. doi:10.1145/1735223.1735255 Reviews the problem-discovery literature and derives a different recommendation from the same family of models: roughly ten evaluators (their 10 plus or minus 2 rule) for an 80 per cent discovery rate. The number moves with the model assumed.
• Schmettow, M. (2012). Sample size in usability studies. Communications of the ACM. doi:10.1145/2133806.2133824 Reanalyses usability datasets with a model that lets problems differ in how easy they are to find rather than assuming every evaluator is equally likely to find each problem. The number needed rises well beyond the customary figures.
• Borsci, S., Macredie, R. D., Barnett, J., Martin, J., Kuljis, J. and Young, T. (2013). Reviewing and Extending the Five-User Assumption. ACM Transactions on Computer-Human Interaction. doi:10.1145/2506210 Reports a procedure in which the number of evaluators is justified from an estimate of the problem-discovery curve for the system under test, so a small number is adequate only within a stated model, with the uncertainty reported.
• Caine, K. (2016). Local Standards for Sample Size at CHI. CHI '16. doi:10.1145/2858036.2858498 Reviews the approaches to sample size, then analyses every paper published at CHI in 2014. Sizes ran from one to nine hundred and sixteen thousand with twelve the most common, and the paper recommends reporting how the number was chosen.
• Guest, G., Bunce, A. and Johnson, L. (2006). How Many Interviews Are Enough? Field Methods. doi:10.1177/1525822x05279903 Ran sixty interviews in two West African countries and tracked when new themes stopped appearing, finding most present within the first twelve. The empirical basis for the common range of twelve to twenty interviews.
• Hennink, M. and Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science and Medicine. doi:10.1016/j.socscimed.2021.114523 A systematic review of empirical studies that tested for saturation, finding it commonly reached at nine to seventeen interviews, depending on sample homogeneity and question specificity.
• Wutich, A., Beresford, M. and Bernard, H. R. (2024). Sample sizes for 10 types of qualitative data analysis: an integrative review, empirical guidance, and next steps. International Journal of Qualitative Methods 23. doi:10.1177/16094069241296206 Integrates the sample-size literature for ten kinds of analysis and finds the guidance splits by construct: theme saturation near nine interviews, meaning saturation near twenty-four, theoretical saturation at twenty or more, and metatheme saturation at twenty to forty per site. Three methods carry little agreed guidance.
• Malterud, K., Siersma, V. D. and Guassora, A. D. (2016). Sample size in qualitative interview studies: guided by information power. Qualitative Health Research 26(13), 1753-1760. doi:10.1177/1049732315617444 Proposes information power as the sample-size rationale: the more information a sample holds for the study, the fewer participants are needed. A judgement framework rather than a computed quantity.
• Braun, V. and Clarke, V. (2020). To saturate or not to saturate? Questioning data saturation as a useful concept for thematic analysis and sample-size rationales. Qualitative Research in Sport, Exercise and Health 13(2), 201-216. doi:10.1080/2159676x.2019.1704846 Argues that data saturation is conceptually incompatible with reflexive thematic analysis, because meaning is produced by the analyst's interpretation rather than lying in the data waiting to be exhausted. It rejects saturation as a sample-size rationale.
• Varpio, L., Ajjawi, R., Monrouxe, L. V., O'Brien, B. C. and Rees, C. E. (2017). Shedding the cobra effect: problematising thematic emergence, triangulation, saturation and member checking. Medical Education 51(1), 40-50. doi:10.1111/medu.13124 Examines four quality markers, thematic emergence, triangulation, saturation and member checking, and argues each is often invoked as a ritual rather than performed, so a stated safeguard can disguise an absent practice.
• Cohen, J. (1992). A power primer. Psychological Bulletin. doi:10.1037/0033-2909.112.1.155 A short tutorial setting out power analysis in three tables giving the sample size needed for small, medium and large effects at conventional power and significance. The source of the effect-size conventions later calculations cite.
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J. and Munafo, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. doi:10.1038/nrn3475 Estimates the median statistical power of neuroscience studies from published meta-analyses and finds it between roughly 8 and 31 per cent, so low power makes a significant result less likely to be true.