Most findings rest on one study, and one study is weak evidence

The previous unit asked who judges a claim; this one asks what a single study can carry. This unit argues that credibility should track how many independent lines of evidence support a claim, not how confidently it is stated. A finding confirmed once is a hypothesis with a result attached: one study records one observation under one set of conditions, and a replication is what tests the step to a general claim. HCI rarely takes that step: Hornbaek and colleagues examined 891 papers across four publication outlets and found that 3 per cent attempted to replicate an earlier result (Hornbaek et al., 2014). Audits of the neighbouring sciences find more replication, and still find that effects shrink when estimated again, though they disagree about the size of the problem. That disagreement does not release a reader from the conclusion. A single study also carries a narrow sample: the users it stands in for are assumed to be able-bodied and of working age, and the older reader and the disabled user are not in the warrant.

One study records one observation under one set of conditions

Goodman, Fanelli and Ioannidis (2016) separate three things routinely conflated. Methods reproducibility runs the same procedures on the same data to get the same numbers. Results reproducibility gets a corroborating result in a new study using the same methods. Inferential reproducibility draws the same conclusion from a replication or a reanalysis. Nosek and Errington (2020) define replication from the other direction, as a study in which any outcome would count as diagnostic evidence about the original claim. A failed replication therefore shows something only when the claim it targets is stated precisely enough to be confronted.

HCI replicates three per cent of the time, and rarely to the number

Hornbaek and colleagues' second finding carries more of the argument. Those replications rarely compared effect sizes with the original, and many papers not framed as replications could have corroborated earlier work. The gap is not only that replication is rare: the field's studies are not positioned to add evidence to an existing claim even when their data could carry it. Baker (2016) reports a Nature survey of 1,576 researchers. More than 70 per cent said they had failed to reproduce another scientist's experiment, and more than half their own, while 52 per cent called the situation a significant crisis. The survey records what researchers believe about reproducibility rather than what a re-run found: the problem is recognised as well as measured, and the belief is not itself a replication rate.

Stower and colleagues reviewed five years of child-robot interaction research, coding 195 papers for reporting transparency. They found deficiencies in participant demographics, study design and open science practices (Stower et al., 2026). Cockburn and colleagues argue that small samples, flexible analyses and a system rewarding novelty are present in computing too (Cockburn et al., 2020).

The neighbouring sciences replicate more, and their effects still shrink

The largest single audit replicated 100 psychology studies published in 2008 (Open Science Collaboration, 2015). Ninety-seven per cent of the originals had statistically significant results, against 36 per cent of the replications. Replication effects were half the magnitude of the originals.

Other fields differ. Camerer and colleagues replicated 21 social science experiments published in Nature and Science, with pre-registered protocols and larger samples. They found a significant same-direction effect for 13 studies, or 62 per cent, with effect sizes about 50 per cent of the original (Camerer et al., 2018). Two multi-site projects show the same variation within a field. One re-tested 13 effects across 36 samples and 6,344 participants, of which 10 replicated consistently (Klein et al., 2014). The other re-tested 28 findings across 125 samples and 15,305 participants, 15 of which, or 54 per cent, replicated (Klein et al., 2018).

Beyond the social sciences, Errington and colleagues repeated 50 experiments from 23 high-impact cancer biology papers, generating data on 158 effects, 136 previously reported as positive. Effect sizes were frequently smaller and protocols often insufficient (Errington et al., 2021).

What the replication numbers do not settle

Gilbert and colleagues argued that the Open Science Collaboration's article contained three statistical errors and no support for its conclusion. Their reanalysis, they said, made the data consistent with the opposite conclusion (Gilbert et al., 2016). The replication team replied that the optimistic assessment rested on statistical misconceptions and on causal inference from selectively interpreted correlational data. From the same data, they added, both optimistic and pessimistic conclusions are possible (Anderson et al., 2016).

A newer case reaches the meta-scientific literature itself. Protzko and colleagues reported in 2023 that sixteen findings, tested under preregistration and large samples, replicated in 86 per cent of attempts; the journal retracted the paper in 2024 after post-publication review found the preregistration misstated (Protzko et al., 2024). A headline figure for high replicability, still quoted, no longer has a source.

Contested question: does a replication rate measure a crisis?

The literature disagrees about whether a low replication rate is a crisis or an artefact of how the rate is computed. One position holds that the concern is justified. The audits report underperformance across psychology, social science, cancer biology and computing (Camerer et al., 2018; Klein et al., 2018; Errington et al., 2021; Cockburn et al., 2020). Pashler and Harris (2012) meet the three strongest objections directly. The first is that a 5 per cent significance level bounds the error rate, which they call a statistical misunderstanding. The second is that conceptual replication is a better test than direct replication, which they argue interacts with publication bias. The third, that errors will be pruned with patience, finds no scenario.

The opposing position is that the audits have not produced a reliable measure, of which Gilbert and colleagues' comment is the sharpest form (Anderson et al., 2016). What would settle the dispute is pre-registered, multi-site protocols, with analysis plans agreed in advance and effect sizes compared rather than a verdict recorded. Camerer and colleagues add evidence that the question is answerable, since the community's beliefs about which results would replicate were strongly related to which ones did (Camerer et al., 2018).

The two sides do not rest on the same kind of evidence. The crisis position rests on four large multi-site audits, three of them outside HCI, and on the field's own count of 3 per cent of 891 papers. The sceptical position rests on one reanalysis of a single audit and a conceptual examination of three arguments. The audits supply the larger body of measurement; the reanalysis supplies the sharper point about what it means. While the dispute is open, a designer should count the independent lines of evidence behind a claim rather than weigh the confidence with which it is stated.

The strongest counter-argument: the thesis imports an epistemology

The strongest objection is that the thesis imports an epistemology the field does not have: HCI is partly a design discipline, and many of its claims are about what can be built rather than about a general law holding across populations, so a low direct-replication rate would be a category error. The audits' spread runs from 36 per cent through 54 per cent to 62 per cent. Camerer and colleagues sharpen the case: of the 21 effects they could test, 13 replicated in direction, and the effect sizes of true positives were 71 per cent of the original, so the audits describe modest inflation rather than a collapse (Camerer et al., 2018).

The reply is that the objection changes the scope of the claim rather than defeating it. Nosek and Errington's (2020) definition accommodates it, since a claim about what is possible faces a different test from a claim about a population. Hornbaek and colleagues' second finding locates the problem in the field's habit of not framing its studies as additions to an existing claim (Hornbaek et al., 2014).

Why credibility should be a count, not a confidence

Two conclusions survive the disagreement. The first is that a single study is a weak basis for a general claim. One observation cannot separate a real effect from a feature of the conditions under which it was made. The second is that credibility is better treated as a count than as a degree of confidence. A claim resting on several independent studies, with different samples and designs, is a different kind of object from one resting on a single study in a single setting. The audits support the count view even where they dispute the rate, because they agree on the direction that effects shrink when estimated again (Klein et al., 2018; Errington et al., 2021). Nosek and colleagues (2022) report that the replication projects of the 2010s contributed to self-correction, while incentives still work against doing it.

Where the argument is heading

The many-labs design, one protocol run across many samples and settings, replaced the single-laboratory replication as the field's instrument (Klein et al., 2014; Klein et al., 2018). Camerer and colleagues used preregistered protocols and prediction markets to estimate in advance which results would hold (Camerer et al., 2018). The reporting instruments are becoming field-specific: Stower and colleagues' checklist for child-robot interaction (Stower et al., 2026) and Cockburn and colleagues' proposals for computing (Cockburn et al., 2020).

The sources this unit rests on

Each note states what the cited document did and found, not a quotation, and every record was checked against the publisher's registered metadata and abstract on 1 November 2026.

• Hornbaek, K., Sander, S. S., Bargas-Avila, J. A., et al. (2014). Is once enough? On the extent and content of replications in human-computer interaction. doi:10.1145/2556288.2557004 Examined 891 papers across four HCI outlets and found that 3 per cent attempted replication, rarely comparing effect sizes with the original. It is the field's own audit, and the number the thesis rests on.
• Stower, R., Spitale, M., Yadollahi, E., et al. (2026). Children, robots, and reproducibility: a review of recent reproducibility challenges in child-robot interaction. doi:10.1145/3848624 Reviewed five years of child-robot interaction research, 2020 to 2024, coding 195 papers for transparency and finding reporting deficiencies in demographics, study design and open science practices. Shows the gap persisting in a growing subfield.
• Cockburn, A., Dragicevic, P., Besancon, L., et al. (2020). Threats of a replication crisis in empirical computer science. doi:10.1145/3360311 Argues that small samples, flexible analyses and novelty incentives are present in empirical computer science, and sets out threats and changes to reviewing. Places the argument inside the field the reader will publish in.
• Open Science Collaboration (2015). Estimating the reproducibility of psychological science. doi:10.1126/science.aac4716 Replicated 100 psychology studies, finding 97 per cent of originals significant against 36 per cent of replications, with replication effects half the magnitude. The benchmark the rest of the literature argues from.
• Camerer, C. F., Dreber, A., Holzmeister, F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. doi:10.1038/s41562-018-0399-z Replicated 21 social science experiments with pre-registered protocols, finding a significant same-direction effect for 13 studies, or 62 per cent, with effect sizes about half the original. Shows the rate is field-dependent.
• Klein, R. A., Ratliff, K. A., Vianello, M., et al. (2014). Investigating variation in replicability: a many labs project. doi:10.1027/1864-9335/a000178 Re-tested 13 effects across 36 samples and 6,344 participants, of which 10 replicated consistently and two did not. Introduces the many-labs design.
• Klein, R. A., Vianello, M., Hasselman, F., et al. (2018). Many Labs 2: investigating variation in replicability across samples and settings. doi:10.1177/2515245918810225 Ran pre-registered replications of 28 findings across about 125 samples and 15,305 participants; 15, or 54 per cent, replicated and 21 produced smaller effects. Shows replication success has a distribution.
• Errington, T. M., Mathur, M., Soderberg, C. K., et al. (2021). Investigating the replicability of preclinical cancer biology. doi:10.7554/elife.71601 Repeated 50 experiments from 23 cancer biology papers, generating data on 158 effects; many reproduced in part, but effect sizes were often smaller and protocols insufficient. Extends the audit beyond the social sciences.
• Gilbert, D. T., King, G., Pettigrew, S., et al. (2016). Comment on Estimating the reproducibility of psychological science. doi:10.1126/science.aad7243 Argues that the replication project contained three statistical errors and no support for its conclusion, and that the reanalysis supported quite high reproducibility. Shows a headline rate is a measurement with assumptions in it.
• Anderson, C. J., Bahnik, S., Barnett-Cowan, M., et al. (2016). Response to Comment on Estimating the reproducibility of psychological science. doi:10.1126/science.aad9163 Replies that the optimistic assessment rests on statistical misconceptions, and that both optimistic and pessimistic conclusions are possible from the same data. Shows what the audit did not settle.
• Pashler, H. and Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. doi:10.1177/1745691612463401 Examines three arguments that the concern is overstated, including the alpha-level argument and the conceptual-replication argument, and rejects each. The strongest counter-argument to the thesis, answered within the literature.
• Nosek, B. A. and Errington, T. M. (2020). What is replication? doi:10.1371/journal.pbio.3000691 Argues that replication should be defined by the question a study answers, so a failure is informative only when the inference target is stated. Supplies the definition the unit rests on.
• Goodman, S. N., Fanelli, D. and Ioannidis, J. P. A. (2016). What does research reproducibility mean? doi:10.1126/scitranslmed.aaf5027 Distinguishes methods reproducibility, results reproducibility and inferential reproducibility, arguing the three are routinely conflated. Provides the vocabulary the unit needs to keep its claim precise.
• Nosek, B. A., Hardwicke, T. E., Moshontz, H., et al. (2022). Replicability, robustness, and reproducibility in psychological science. doi:10.1146/annurev-psych-020821-114157 Reviews replicability in psychology and the multi-site projects of the 2010s, arguing replication aided self-correction but that incentives still work against it. The synthesis that says where the programme has reached.
• Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature. doi:10.1038/533452a Reports a Nature survey of 1,576 researchers: more than 70 per cent had failed to reproduce another scientist's experiment and more than half their own, while 52 per cent called the situation a significant crisis. It records researchers' beliefs about reproducibility rather than a measured rate.
• Protzko, J., Krosnick, J., Nelson, L., et al. (2024). Retraction Note: High replicability of newly discovered social-behavioural findings is achievable. doi:10.1038/s41562-024-01997-3 Withdraws a 2023 report that sixteen social-behavioural findings replicated in 86 per cent of attempts, after post-publication review found the meta-study's preregistration misstated. A headline replicability figure can itself fail an audit.