Replication, researcher degrees of freedom and the credibility of HCI findings

A team tests a redesigned menu on twenty-four participants, all of working age, finds task completion significantly higher, and publishes the redesign as a recommendation. The evidence of the last fifteen years says the reported number is probably too large. This eighth of ten units asks how far those published numbers can be trusted, and argues that the replication crisis reached human-computer interaction as an argument rather than a measurement: HCI's own contribution is one count of how rarely it replicates anything, while the size of the problem was measured elsewhere. It matters because HCI turns a single study's effect size into a design rule, and attenuation is what makes such a rule unsafe.

A literature's reproducibility is a quantity, not an opinion

The Open Science Collaboration replicated 100 studies from three psychology journals with high-powered designs and original materials where available. Replication effects were half the magnitude of the originals, and success was better predicted by the original evidence than by either team (Open Science Collaboration, 2015). Camerer and colleagues replicated 21 studies from Nature and Science with preregistered plans and samples five times larger, finding a same-direction effect in 13 and replication effect sizes half the originals (Camerer et al., 2018). Many Labs 2 preregistered 28 findings across 125 samples and 15,305 participants, a scale no single laboratory can afford on its own: 15 findings, 54 per cent, kept a significant effect in the original direction and the median effect size fell from 0.60 to 0.15 (Klein et al., 2018). No replication programme here sampled an assistive-technology population, so whether attenuation holds there is untested.

The mechanism is flexibility, not misconduct

None of this requires fraud. Simmons, Nelson and Simonsohn identified four ordinary choices, which dependent variable to report, how large a sample to collect, which covariate to include and which conditions to compare, and showed by simulation and two real experiments that combinations of them make it easy to publish significant evidence for a false hypothesis. In the second experiment, participants were nearly a year and a half younger by their reported birth dates, p = .040 (Simmons, Nelson and Simonsohn, 2011).

John, Loewenstein and Prelec surveyed more than 2,000 psychologists anonymously and found self-admissions high, and higher for the practices respondents judged least defensible (John, Loewenstein and Prelec, 2012). Wicherts and colleagues catalogued the options themselves, 34 researcher degrees of freedom in designing, running, analysing and reporting a study (Wicherts et al., 2016). The probability that a published claim is true falls when studies and effects are small and flexibility is great, so for most designs a false claim is more likely than a true one (Ioannidis, 2005).

The pattern is not a property of psychology

HCI might argue the problem belongs to psychology. Audits outside psychology closed that exit. The Reproducibility Project: Cancer Biology repeated 50 experiments from 23 high-impact papers under a published protocol: for positive effects the median replication effect size was 85 per cent smaller and the success rate 46 per cent (Errington et al., 2021). Prinz, Schlange and Asadullah found published data in line with Bayer's internal findings in only about twenty to twenty-five per cent of sixty-seven in-house projects, and Begley and Ellis reported that Amgen's scientists could reproduce six of fifty-three landmark preclinical oncology studies. Neither is a formal replication programme, since the criterion is agreement with an in-house result rather than a replication rate, but they show the pattern reaching industry laboratories.

The only number HCI has measured about itself

HCI's own evidence is thin, and saying so is part of the argument: every attenuation figure in this unit was measured elsewhere, so the transfer is an argument by analogy. Hornbaek and colleagues examined 891 papers from four HCI outlets and found about 3 per cent attempting to replicate an earlier result, with those attempts treated as confirm or not-confirm decisions (Hornbaek et al., 2014). The figure counts attempts rather than failures, so it measures how rarely HCI looks rather than how often its findings hold, and a field whose idea of corroboration is a binary test lets one confirming study settle a question.

Attenuation is the quantity a design decision needs

Attenuation matters to HCI because HCI rarely tests a theory; it chooses between designs. Many Labs 2 measured a median effect size of 0.60 in the originals and 0.15 in the replications, and the Open Science Collaboration found replication effects half the magnitude of the originals (Klein et al., 2018; Open Science Collaboration, 2015). An effect that shrinks by half moves a difference that justified a redesign into the range where it is indistinguishable from noise at the sample sizes interaction studies use.

The record is fixed before the data arrive

The remedies are procedural, and all of them move choices into the record before the data arrive: Wicherts and colleagues' 34 degrees of freedom become a checklist fixed at design time (Wicherts et al., 2016). Simmons, Nelson and Simonsohn propose six disclosure requirements for authors and four reviewer guidelines that do not depend on changing anyone's character (Simmons, Nelson and Simonsohn, 2011). Munafo and colleagues set the same measures inside a wider programme, and are candid that the evidence for their effectiveness is partial (Munafo et al., 2017). Lakens argues that a principled use of preregistration does not treat every deviation as a problem, because a deviation can lower or raise the severity of a test, and he sets out how to report each kind of deviation (Lakens, 2024). Register the analysis, state the smallest effect worth acting on, report the interval, and treat one study as one estimate.

Contested question: false findings, or true findings with inflated sizes?

The literature agrees on attenuation and disagrees about its cause. One position holds that the problem is largely false positives: the originals that failed should never have been significant, ordinary flexibility manufactured them, and preregistration will stop manufacturing them. Ioannidis's framework is the analytical form of that claim (Ioannidis, 2005) and Simmons, Nelson and Simonsohn's demonstration the experimental form (Simmons, Nelson and Simonsohn, 2011).

A second position holds that the originals were mostly real and mostly too large. Camerer and colleagues' Bayesian analysis estimated 67 per cent true positives, at 71 per cent of their original size, so false positives and the inflation of true effects both contribute to imperfect reproducibility (Camerer et al., 2018). Many Labs 2 supports the reading: more than half its replications kept a significant effect in the original direction while the median effect size fell to a quarter (Klein et al., 2018). Reform must then change practice even where the effect is genuine.

A third strand questions the measurement. Errington and colleagues reported several indicators, and the answer depended on which was used: success on binary criteria was 40 per cent against an 85 per cent reduction in median effect size for the same effects. A successful replication does not confirm an original outright, and a failure does not disconfirm it (Errington et al., 2021).

What would settle the dispute is originals whose expected effect size was registered in advance, so that inflation can be estimated rather than inferred backwards, with the replications reporting a decision and an effect size together. The preregistered protocols of Camerer and colleagues and Many Labs 2 are that design (Camerer et al., 2018; Klein et al., 2018). Until such studies are common, HCI's replication rate is unmeasured.

The evidence is strong and mostly imported: two preregistered replication programmes with large samples agree on attenuation, and a third outside psychology agrees on the direction, while the HCI figure counts attempts, not outcomes.

The strongest objection is that the transfer rests on other fields: HCI may attenuate less, because its effects are often large and measured directly, and its only figure is a count of how rarely the field looks.

What to do while the dispute is open

While the dispute is open, plan against the attenuated estimate rather than the published one, and say which was used.

Where the argument is heading

Hardwicke and colleagues review the evidence supporting each stage of their translational framework (Hardwicke et al., 2019). Nosek and colleagues describe the 2010s as a decade of active confrontation, name the disincentives to replication, and note the tendency to read one as a personal attack (Nosek et al., 2021). Errington and colleagues show what evaluation costs and what it buys: a multi-year, multi-indicator programme whose conclusion is that a single replication settles less than its headline number suggests (Errington et al., 2021).

The sources this unit rests on

• Camerer, C. F., Dreber, A., Holzmeister, F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour. doi:10.1038/s41562-018-0399-z They replicated 21 experiments with preregistered plans and samples about five times larger, finding a same-direction effect in 13 and replication effect sizes about half the originals; a Bayesian analysis put true positives at 67 per cent, at 71 per cent of their original size. It supplies the second position in the contested question.
• Errington, T. M., Mathur, M. B., Soderberg, C. K., et al. (2021). Investigating the replicability of preclinical cancer biology. eLife. doi:10.7554/eLife.71601 The Reproducibility Project: Cancer Biology repeated 50 experiments from 23 papers. Median replication effect sizes for positive effects were 85 per cent smaller, 40 per cent succeeded on three or more of five binary criteria, and the success rate was 46 per cent. It shows the answer depends on the indicator chosen.
• Hardwicke, T. E., Serghiou, S., Janiaud, P., et al. (2019). Calibrating the Scientific Ecosystem Through Meta-Research. Annual Review of Statistics and Its Application. doi:10.1146/annurev-statistics-031219-041104 They set out a translational framework for research on research: identifying problems, investigating them, developing solutions and evaluating them, and reviews the evidence at each stage. It anchors the claim that the field has moved from documenting the crisis to testing remedies.
• Hornbaek, K., Sander, S., Bargas-Avila, J. A., et al. (2014). Is once enough?. CHI Conference on Human Factors in Computing Systems. doi:10.1145/2556288.2557004 They examined 891 papers from four HCI outlets and found about 3 per cent attempting replication, with those attempts treated as confirm or not-confirm decisions that rarely analysed effect sizes. It is the only measurement of the problem inside HCI.
• Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine. doi:10.1371/journal.pmed.0020124 He argues that the probability a published claim is true falls when studies and effects are small and flexibility great, and his simulations make a false claim more likely than a true one for most designs. It is the framework joining flexibility to a literature's reliability.
• John, L. K., Loewenstein, G. and Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science. doi:10.1177/0956797611430953 They surveyed more than 2,000 psychologists anonymously with incentives for honest reporting. Self-admissions were high and greater for the least defensible practices, which they read as evidence that some had become the prevailing norm. It shows the flexibility is used, not merely available.
• Klein, R. A., Vianello, M., Hasselman, F., et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245918810225 They preregistered replications of 28 findings across 125 samples and 15,305 participants. Fifteen findings, 54 per cent, were significant in the original direction, the median effect size fell from 0.60 to 0.15, and nine came out in the opposite direction. It is the closest thing to a definitive attenuation estimate.
• Lakens, D. (2024). When and How to Deviate From a Preregistration. Collabra: Psychology. doi:10.1525/collabra.117094 He argues that a principled use of preregistration does not treat every deviation as problematic, since a deviation can lower or raise the severity of a test, and sets out how to report each kind of deviation. Limits: an argument, not an empirical study.
• Munafo, M. R., Nosek, B. A., Bishop, D. V. M., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour. doi:10.1038/s41562-016-0021 They propose measures across methods, reporting, reproducibility, evaluation and incentives, and state that simulations and empirical studies give some support while wide adoption will require iterative evaluation. It turns the unit's remedies into an institutional programme.
• Nosek, B. A., Hardwicke, T. E., Moshontz, H., et al. (2021). Replicability, Robustness, and Reproducibility in Psychological Science. Annual Review of Psychology. doi:10.1146/annurev-psych-020821-114157 They review the evidence on replicability, robustness and reproducibility as distinct properties of a literature, name the disincentives to running replications, and argue that replication drives self-correction. It states where the field's own understanding stands when the unit closes.
• Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. doi:10.1126/science.aac4716 They replicated 100 experimental and correlational studies from three psychology journals with high-powered designs and original materials where available. Replication effects were half the magnitude of the originals, 97 per cent of originals and 36 per cent of replications were significant, and success was better predicted by the original evidence than by either team.
• Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2011). False-Positive Psychology. Psychological Science. doi:10.1177/0956797611417632 They show in simulation and in two real experiments that four ordinary analytic choices, which dependent variable, sample size, covariate and set of conditions to report, make it easy to obtain significant evidence for a false hypothesis; the second experiment makes participants nearly a year and a half younger by their reported birth dates, p = .040. It names the mechanism without requiring misconduct.
• Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., et al. (2016). Degrees of Freedom in Planning, Running, Analysing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking. Frontiers in Psychology. doi:10.3389/fpsyg.2016.01832 They catalogue 34 researcher degrees of freedom across designing, running, analysing and reporting a study, and offer the list as a checklist for preregistrations. It gives the remedy its concrete form.
• Prinz, F., Schlange, T. and Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery. doi:10.1038/nrd3439-c1 A Bayer survey of sixty-seven projects found published data in line with internal findings in only about twenty to twenty-five per cent of them.
• Begley, C. G. and Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature. doi:10.1038/483531a An Amgen team could reproduce only six of fifty-three landmark preclinical oncology studies.