Power is a decision about the smallest effect worth detecting

A power calculation looks like arithmetic, and it is really a decision. Sample size, the significance criterion, the effect size and power are bound together: any one follows once the other three are fixed (Cohen, 1992). Three are usually settled by convention, a criterion of .05, power of .80, and an effect size from Cohen's small, medium and large values. This fourth of ten units moves from the arrangement to the sample, and argues that sample size should follow from the smallest effect that would change a decision rather than from a default, and that a literature built on underpowered studies reports inflated effects. For that reason, a null result from a small study is usually no evidence, and a significant one is usually an overestimate.

The figure that is really a decision

Cohen tabled the sample sizes needed for 80 per cent power across eight standard tests, setting .20, .50 and .80 as small, medium and large for the standardised difference between two means, and .10, .30 and .50 for a correlation (Cohen, 1992). He offered the values as conventions rather than measurements.

My intent was that medium ES represent an effect likely to be visible to the naked eye of a careful observer (Cohen, 1992).

One measurement of a protein detects a 10 per cent increase at the .05 criterion only 64 per cent of the time, and averaging seven raises that to 84 per cent (Krzywinski and Altman, 2013). The example fixes a 10 per cent difference, and an HCI team must still say what change in completion or error rate matters. Suppose a change must deliver five percentage points of task completion, or must let an older user finish without an error, to be worth shipping; the effect that matters is then that figure, not .50. Lakens (2022) sets out six defensible ways to justify a sample size, from collecting almost the whole population to admitting there is none. Brysbaert (2019) supplies an anchor: taking d = .4 as a good first estimate of the smallest effect of interest, two within-participant conditions need more than 50 participants for 80 per cent power, and a between-groups variable or an interaction pushes the number to 100, 200 or more.

What low power does to the answer, not only to the chance of getting one

Button and colleagues (2013) estimated the median statistical power of neuroscience studies at between about 8 and 31 per cent: low power also lowers the probability that a significant result reflects a true one, so the results that appear are drawn from the tail of overestimates. Bakker, van Dijk and Wicherts (2012) show the same at the level of a field, where around 96 per cent of papers using significance testing report significant main results while the typical study is too small to be powerful. Across 13 meta-analyses covering 281 primary studies they found bias or an excess of significance in seven, and several small samples can reach a result below .05 more efficiently than one larger study.

Gelman and Carlin (2014) name the two errors a power figure conceals: type S error, the probability of a wrong sign, and type M error, the factor by which magnitude is exaggerated. In small samples a significant result can be both wrongly signed and several times too large. Anderson, Kelley and Maxwell (2017) show that planning from a pilot or a published effect size often leaves the study underpowered, because both carry publication bias and sampling uncertainty.

The literature, not the study, is the unit of the problem

Maxwell (2004) puts the curse of multiplicities at the centre: a study that tests several hypotheses or measures many outcomes has a power worse than any single calculation suggests. Vankov, Bowers and Munafo (2014) observe that calls for more power have failed since Cohen's survey, that a later review found the situation worse with average power near 20 per cent, and that significance testing rewards the significant result over the precise one.

Schonbrodt and Perugini (2013) find by simulation that a correlation should approach a sample of 250 before its magnitude is stable, the size depending on the effect, the tolerated corridor and the confidence demanded. Brysbaert and Stevens (2018) put a repeated-measures reaction time experiment at 1,600 word observations per condition, 40 participants each seeing 40 stimuli, and note replication rates below 40 per cent where a powered literature would predict about 80. A study of 20 participants is not a smaller version of an adequate one.

HCI's own numbers describe practice rather than a standard

Caine (2016) analysed every manuscript published at CHI 2014. She found sample sizes from 1 to 916,000, with the most common size 12, varying by study setting and methodology. Schweizer and Furley (2016) analysed the sample sizes of quantitative studies in four leading sport and exercise psychology journals from 2009 to 2013.

Caine's finding also warns against over-claiming: a modal sample of 12 includes interview and observational studies for which power is not the relevant criterion, and a small sample cannot support a statistical claim about an effect however useful it is for its own question, nor speak for a group it is unlikely to contain, since a sample of 12 rarely includes a switch user.

Contested question: convention, smallest effect of interest, or the error you care about?

The literature agrees that an underpowered claim is weak and disagrees about what should replace the convention. One position keeps the convention as a usable default: Cohen (1992) proposed the values so that a researcher with no prior estimate could plan, and Caine's (2016) local standards and Brysbaert's (2019) tables are the same move at the level of a field.

The opposing position holds that the benchmark is the problem. Lakens (2022) argues that the sample size must follow from the smallest effect the study is designed to detect, derived from the decision the result will inform rather than read from a table. Brysbaert's anchor of d = .4 sits between the two positions: he defends it empirically as the effect size typical of the field, a convention recovered by measurement.

A third position moves the target. Gelman and Carlin (2014) argue that power calculation itself misleads, and that the design should be judged by the type S and type M errors it tolerates. A study can be powered for the effect it assumed and still mislead when the true effect is smaller. A fourth position questions the field diagnosis: Schweizer and Furley (2016) find it concentrated in experiments, and Caine (2016) finds it varying with methodology. A fifth rejects both a field-wide effect size and an interest-based minimum: for a novel question, Giner-Sorolla and colleagues would set the sample from the effect that others could cost-effectively implement or study, and place precision analysis and sequential analysis beside power analysis (Giner-Sorolla et al., 2024). What would settle the disagreement is a study that fixes a decision threshold, designs one set of studies against it and another against a field convention, and reports how well the estimates are calibrated, with the two error types for each. No source here runs that comparison for HCI.

The positions rest on evidence of different kinds: a table, an argument plus one recovered estimate, a worked demonstration, and an argument from what other people would act on. None of them has a trial.

The strongest objection is that the rule needs a figure the study does not yet have: Anderson, Kelley and Maxwell (2017) show a pilot or a published effect size to be a biased guide, so the smallest effect worth detecting is often a guess, and a registered guess is still a guess.

What to do while the dispute is open

While the dispute is open, fix the smallest effect that would change the decision, say where the figure came from and whether the team can afford the sample it implies. Report the interval beside it, and treat a null from a small study as no evidence either way.

Where the argument is heading

Brysbaert and Stevens (2018) extend power analysis to mixed-effects models, where stimuli as well as participants contribute variance, and Lakens and Caldwell (2021) provide simulation-based power analysis for factorial designs, which reaches within-subject factors and interactions that closed-form procedures cannot handle. Caine (2016) points to a benchmark the field could police itself against, and Gelman and Carlin (2014) would replace the power number with the errors a designer cares about.

The sources this unit rests on

• Anderson, S. F., Kelley, K. and Maxwell, S. E. (2017). Sample-Size Planning for More Accurate Statistical Power. Psychological Science 28(11). doi:10.1177/0956797617723724 They show that planning a sample from a prior study's observed effect, which carries publication bias and sampling uncertainty, often leaves the study underpowered, and propose an adjustment with an open-source tool.
• Bakker, M., van Dijk, A. and Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science 7(6). doi:10.1177/1745691612459060 They report that around 96 per cent of papers using significance testing present significant main results while the typical study is too small to be powerful. Across 13 meta-analyses covering 281 studies they found bias or excess significance in seven.
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14. doi:10.1038/nrn3475 They estimate the median statistical power of neuroscience studies at between about 8 and 31 per cent, and argue that low power lowers the chance that a significant result is true and inflates the effects found.
• Brysbaert, M. (2019). How Many Participants Do We Have to Include in Properly Powered Experiments? Journal of Cognition 2(1). doi:10.5334/joc.72 Treating d = .4 as the smallest effect of interest, he gives reference numbers for common designs: over 50 participants for a two-condition within-participant comparison, and 100 or more once a between-groups variable or an interaction is involved.
• Brysbaert, M. and Stevens, M. (2018). Power Analysis and Effect Size in Mixed Effects Models. Journal of Cognition 1(1). doi:10.5334/joc.10 They review power for mixed-effects models, where stimuli as well as participants contribute variance, and recommend at least 1,600 word observations per condition for a repeated-measures reaction time experiment, noting replication rates below 40 per cent where a powered literature would predict about 80.
• Caine, K. (2016). Local Standards for Sample Size at CHI. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. doi:10.1145/2858036.2858498 She analysed every manuscript published at CHI 2014 and found sample sizes from 1 to 916,000 with a most common size of 12, and proposes reporting local standards, the typical sample sizes at the venue, as a benchmark.
• Cohen, J. (1992). A power primer. Psychological Bulletin 112(1). doi:10.1037/0033-2909.112.1.155 He tables the sample sizes needed for 80 per cent power across eight standard tests, with conventions of .20, .50 and .80 for the standardised mean difference and .10, .30 and .50 for a correlation, and relays a review of 54 articles in which only two mentioned power and the median power was .25.
• Gelman, A. and Carlin, J. (2014). Beyond Power Calculations. Perspectives on Psychological Science 9(6). doi:10.1177/1745691614551642 They argue that power analysis is flawed because it concentrates on significance, and recommend design calculations that estimate type S error, a wrong sign, and type M error, an exaggerated magnitude; a significant result can be both.
• Giner-Sorolla, R., Montoya, A. K., Reifman, A., et al. (2024). Power to Detect What? Considerations for Planning and Evaluating Sample Size. Personality and Social Psychology Review. doi:10.1177/10888683241228328 They argue that for a novel question the sample should follow the effect that others could cost-effectively implement or study, not an interest-based minimum or a field-wide effect size, and they set precision and sequential analysis beside power analysis. Limits: an argument for one subfield, not a trial.
• Lakens, D. (2022). Sample Size Justification. Collabra: Psychology 8(1). doi:10.1525/collabra.33267 He sets out six defensible ways to justify a sample size, from collecting almost the whole population to admitting none exists, and argues the researcher must state which is being used and which effects matter.
• Lakens, D. and Caldwell, A. R. (2021). Simulation-Based Power Analysis for Factorial Analysis of Variance Designs. Advances in Methods and Practices in Psychological Science 4(2). doi:10.1177/2515245920951503 They present the R package Superpower and companion apps for simulation-based power analysis of factorial designs with up to three within- or between-subject factors, including main effects and interactions. Partial eta-squared does not generalise across designs, so the researcher specifies means, standard deviations and correlations.
• Maxwell, S. E. (2004). The Persistence of Underpowered Studies in Psychological Research. Psychological Methods 9(2). doi:10.1037/1082-989x.9.2.147 He examines why underpowered studies persist and places the curse of multiplicities at the centre: a study testing several hypotheses or measuring many outcomes has power worse than a single calculation suggests. Effect sizes and intervals help but do not remove the problem.
• Schonbrodt, F. D. and Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality 47(5). doi:10.1016/j.jrp.2013.05.009 Using Monte-Carlo simulation they find that in typical scenarios a correlation should approach a sample of 250 before the estimate is stable, the size depending on the effect and the corridor tolerated. A 2018 corrigendum corrected a coding error; the main conclusion stands.
• Schweizer, G. and Furley, P. (2016). Reproducible research in sport and exercise psychology: the role of sample sizes. Psychology of Sport and Exercise 23. doi:10.1016/j.psychsport.2015.11.005 They analysed the sample sizes of quantitative studies published from 2009 to 2013 in four leading sport and exercise psychology journals. Experiments were often underpowered for small-to-medium effects while correlational studies were often adequate.
• Vankov, I., Bowers, J. and Munafo, M. R. (2014). On the Persistence of Low Power in Psychological Science. Quarterly Journal of Experimental Psychology 67(11). doi:10.1080/17470218.2014.885986 They review power in psychology since Cohen's survey, note that a later review found the situation worse with average power near 20 per cent, and ask why repeated calls have failed, pointing to how significance testing rewards the significant result.
• Krzywinski, M. I. and Altman, N. (2013). Power and sample size. Nature Methods 10, 1139 to 1140. doi:10.1038/nmeth.2738 They explain how power, sample size, effect size and the significance threshold follow one another, working a protein example in which one measurement detects a 10 per cent increase 64 per cent of the time.