Sampling bounds the claim: the central limit theorem, small samples and the replication problem

A design review reports that a new toolbar cut task time by 12 per cent, measured on eight participants. The number is not false, but it is unsupported. This unit argues that a design effect measured on a convenience sample usually carries an interval wider than the effect itself, and that this is structural rather than a slip in arithmetic. Eight participants put an error bar about that 12 per cent which is wider than 12 per cent, so a reader cannot tell an improvement from the spread of a small sample. Sampling mathematics states how much a small study can show, and what decides whether a comparison means anything is the interval about the estimate rather than the significance decision. In human-computer interaction the samples are small, the measures correlate weakly and the replication record is thin. Unit 10 turns from the size of the claim to the floor the system puts under the response.

The error bar is the claim, not the p-value

Statistics enters a design report at one point: the estimate needs an interval. Grinstead and Snell (2003) develop the two results that fix it. The central limit theorem states that the mean of many independent observations is approximately normal, whatever the shape of the individual distribution, and the standard error, the spread of that mean, is the standard deviation of the observations divided by the square root of their number, so the interval narrows only as the square root of the sample grows.

Cumming (2014) turns those results into a programme. He argues that reliance on null hypothesis significance testing has left the published literature incomplete and untrustworthy, and the remedy is estimation: report effect sizes and confidence intervals, prespecify the study where possible, encourage replication, and give no place to a dichotomous decision. The interval does not decorate the estimate; it is the claim. An interval belongs to a population. The sample fixes who the estimate is about, so an estimate from a convenience sample of students is an estimate for students, and no arithmetic carries it to users who were not in the room (Henrich, Heine and Norenzayan, 2010).

A published number makes the interval concrete. Hornbaek and Law's pooled correlation between effectiveness and efficiency is .247, with a confidence interval of about .06, so the interval runs from roughly .19 to .31 and excludes zero. The finding is real and it is weak, and the interval says both at once; a significance test for the same correlation would report only that it differs from zero. The width of an interval about a correlation falls roughly with the square root of the sample, so a correlation estimated from eight participants carries one several times wider than the same correlation pooled here from 73 studies.

A small sample cannot support a small difference

A constructed case shows the arithmetic rather than reporting a result. Take a task time with a mean of 400 milliseconds and a standard deviation of 100 milliseconds. With eight participants the standard error is 100 divided by the square root of eight, or 35 milliseconds, and a 95 per cent interval runs from about 331 to 469 milliseconds. A claimed improvement of 12 per cent is 48 milliseconds, which sits inside that interval. Halving the standard error needs 32 participants, four times the data (Grinstead and Snell, 2003).

Cohen (1992) makes the required sample size a quantity a reader can look up, tabling the sample size needed for 0.80 power to detect conventional small, medium and large effects across eight standard statistical tests, and arguing that the neglect of power analysis is a problem of access to the reference material rather than a disputed case. The direction of the calculation is the useful part: the effect size is fixed by the design question, the power by the risk the team will accept, and the sample size falls out of the two.

The intervals the field reports are usually wide

Two studies measure how wide the intervals in this field are. The Open Science Collaboration (2015) replicated 100 experimental and correlational studies from three psychology journals with high-powered designs, finding that the mean effect size of the replications was half the mean of the originals, with 36 per cent of replications reaching significance against 97 per cent of the originals. A small effect measured once shrinks when it is measured again, and the shrinkage is the size of the interval the first study did not report.

Hornbaek and Law (2007) pooled the raw data of 73 usability studies and found the field's own measures correlate weakly. Effectiveness with efficiency is .247, efficiency with satisfaction .196, and effectiveness with satisfaction .164, each with a confidence interval of about .06. If a usability claim measured one way does not transfer to another measure of the same system, an interval about a single measure bounds only that measure. Dragicevic (2016) makes the same case from inside the discipline.

The field has not changed its statistics

The critique is old, the response is known, and neither has moved the practice. Besancon and Dragicevic (2019) analysed CHI proceedings over nine years and found that dichotomous inference, the classification of evidence as sufficient or insufficient through null hypothesis significance testing, remained dominant despite sustained methodological criticism; they concluded that the influence of the critics on the field's practice has been negligible. The reporting convention that would carry the interval is not unavailable, only unused.

A small sample also flatters the study

A second and quieter bias runs through the field's own sample-size folklore. Nielsen and Landauer (1993) fitted the detection of usability problems as a function of the number of users tested, found it well modelled as a Poisson process across eleven studies, and estimated that 16 evaluations would be worth their cost for a medium example. The model is a planning tool, and it was read as a rule.

Lewis (1994) tested the three claims drawn from that work and found the record uneven: diminishing returns was supported, the claim that four or five participants find 80 per cent of problems was only partly supported, since it holds when the average likelihood of detecting a problem lies between 0.32 and 0.42, and the claim that severe problems are found earlier was not supported. Lewis (2001) then named the mechanism by which the folklore survives: small-sample estimates of the problem-discovery rate are biased upward, which underestimates the sample size a study needs and overestimates the proportion of problems found.

Contested question: an arithmetic failure or a structural one

The literature does not dispute the arithmetic; it disagrees about what follows. One position holds that the problem is the reporting convention: Cumming (2014) sets out estimation, confidence intervals and prespecification, and Dragicevic (2016) gives the human-computer interaction version, arguing that the fault lies with the tools the field adopted rather than with its researchers.

A second position holds that the problem is structural. Ioannidis (2005) models the probability that a reported finding is true as a function of study power, bias, the number of studies and the ratio of true to tested relationships, and shows it falls when studies are small, effects are small and design flexibility is high. On this account a better test does not help, because the field tests many relationships and reports the ones that reach a threshold. Cockburn, Gutwin and Dix (2018) propose the structural remedy, preregistration, since registering experimental intentions in advance lets a reader separate a predicted finding from one hypothesised after the results were known.

Hornbaek (2013) presses the strongest counter-argument: experiments remain indispensable for advancing the field, he argues, and the remedy for weak evidence is design quality: build on existing work when forming hypotheses, make challenging comparisons rather than biased win-lose setups, and rule out alternative explanations. The counter has teeth because the evidence for a replication failure inside human-computer interaction is inferential: the measured collapse is psychology's.

Nosek and colleagues (2022) supply the neighbouring field's evidence, describing the 2010s in psychology as a decade of confrontation in which systematic replications failed to replicate many published findings. The evidence behind the structural reading is largely imported: the collapse of effect sizes on replication was measured in psychology, and the census of interval reporting inside human-computer interaction has not been taken, with Besancon and Dragicevic's CHI analysis the closest the field has to its own measurement. What would settle the disagreement is two measurements the field does not have: a census of interval reporting in its own venues, and preregistered replications of published interface findings on the same measures and tasks. A programme that reproduced the effect sizes would refute the structural reading. One that found them shrinking to half, as the Open Science Collaboration (2015) found in psychology, would confirm it. Meanwhile, report the interval, fix the sample size from the effect you claim, and record the analysis plan before the data. A small convenience sample can be used to find problems, not to claim an effect.

Where the argument is heading

The direction of travel makes the interval part of the report and the replication part of the record. Cockburn, Gutwin and Dix (2018) set out preregistration for human-computer interaction experiments, Nosek and colleagues (2022) document the neighbouring field's infrastructure of multi-site replication, and Cumming (2014) and Dragicevic (2016) supply the estimation vocabulary. Whether the next decade of interface research reports intervals is a question about practice, and the arithmetic will not answer it.

The sources this unit rests on

• Grinstead, C. M. and Snell, J. L. (2003). Introduction to Probability. American Mathematical Society. https://math.dartmouth.edu/~prob/prob/prob.pdf Develops the central limit theorem: the mean of many independent observations is approximately normal whatever the shape of the individual distribution, and the standard error of a mean falls with the square root of the number of observations.
• Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155 Tables the sample size needed for 0.80 power to detect conventional small, medium and large effects, for eight standard statistical tests. The neglect of power analysis, Cohen argues, is a problem of access to the reference material rather than a disputed case.
• Cumming, G. (2014). The new statistics. Psychological Science, 25(1), 7-29. doi:10.1177/0956797613504966 Argues for a shift from null hypothesis significance testing to estimation based on effect sizes, confidence intervals and meta-analysis, with prespecification of studies and encouragement of replication.
• Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124 Models the probability that a research claim is true as a function of study power, bias, the number of studies on the question and the ratio of true to tested relationships. The probability falls when studies are small, effects are small and design flexibility is high.
• Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi:10.1126/science.aac4716 Replicated 100 experimental and correlational studies from three psychology journals. Replication effects averaged half the magnitude of the originals, 36 per cent of replications reached significance against 97 per cent of originals, and 47 per cent of original effect sizes fell inside the replication interval.
• Dragicevic, P. (2016). Fair statistical communication in HCI. Modern Statistical Methods for HCI. Springer. doi:10.1007/978-3-319-26633-6_13 Argues that human-computer interaction has adopted the p-value and dichotomous testing, tools that communicate findings poorly, and that the fault lies with the tools rather than their users. It sets out estimation reporting effect sizes and interval estimates.
• Besancon, L. and Dragicevic, P. (2019). The continued prevalence of dichotomous inferences at CHI. CHI Extended Abstracts. doi:10.1145/3290607.3310432 Analysed CHI proceedings over nine years and found dichotomous inference remained dominant despite sustained methodological criticism. The influence of the critics on the field's practice has been negligible.
• Hornbaek, K. and Law, E. L.-C. (2007). Meta-analysis of correlations among usability measures. Proceedings of CHI. doi:10.1145/1240624.1240722 Pooled the raw data of 73 usability studies and found the field's measures correlate weakly: effectiveness with efficiency .247, efficiency with satisfaction .196 and effectiveness with satisfaction .164, each with a confidence interval of about .06.
• Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of CHI. doi:10.1145/169059.169166 Across eleven studies the detection of usability problems as a function of the number of users tested was well modelled as a Poisson process. For a medium example the authors estimate that 16 evaluations would be worth their cost, with the maximum benefit-to-cost ratio at four.
• Lewis, J. R. (1994). Sample sizes for usability studies: additional considerations. Human Factors, 36(2), 368-378. doi:10.1177/001872089403600215 Tested three claims about sample size: diminishing returns was supported, the claim that four or five participants find 80 per cent of problems held only when the detection likelihood is between 0.32 and 0.42, and the claim that severe problems are found earlier was not supported.
• Lewis, J. R. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13(4), 445-479. doi:10.1207/S15327590IJHC1304_06 Shows that small-sample estimates of the problem-discovery rate suffer a serious overestimation bias, which underestimates required sample sizes and overestimates the proportion of problems found; averaging a normalisation procedure with Good-Turing discounting corrects it.
• Cockburn, A., Gutwin, C. and Dix, A. (2018). HARK no more: on the preregistration of CHI experiments. Proceedings of CHI. doi:10.1145/3173574.3173715 Reviews the motivation and outcomes of experimental preregistration in other disciplines and discusses how to adapt it to human-computer interaction. Preregistration lets a reader distinguish a predicted finding from one hypothesised after the results were known.
• Hornbaek, K. (2013). Some whys and hows of experiments in human-computer interaction. Foundations and Trends in Human-Computer Interaction, 5(4), 299-373. doi:10.1561/1100000043 Argues that experiments remain indispensable for advancing human-computer interaction beyond technical innovation, and sets out heuristics for doing them well, including building on existing work when forming hypotheses and ruling out alternative explanations.
• Nosek, B. A., Hardwicke, T. E., Moshontz, H. and others (2022). Replicability, robustness and reproducibility in psychological science. Annual Review of Psychology, 73(1), 719-748. doi:10.1146/annurev-psych-020821-114157 Surveys replication in psychological science, describing the 2010s as a decade of active confrontation in which systematic and multi-site replication projects observed surprising failures to replicate many published findings.
• Henrich, J., Heine, S. J. and Norenzayan, A. (2010). The weirdest people in the world? Behavioural and Brain Sciences, 33(2-3), 61-83. doi:10.1017/S0140525X0999152X Reviews behavioural-science sampling and finds it drawn overwhelmingly from Western, educated, industrialised populations.