The replication crisis and the credibility of a single study

Most results a project reports come from one study, run once, in one laboratory, on one convenience sample of able-bodied participants, by one person.

The fourth of ten units argues that such a result is weak evidence on its own, and the fifth asks what that leaves a project able to claim. That is not because the student was careless: the incentives and practices that decide what becomes publishable also raise the share of published findings that are false. Write a single study so that no reader can mistake it for the only test of its claim.

Why a false finding can be the likely one

Ioannidis (2005) models the probability that a published claim is true. He shows it depends on the power of the study, the bias in its design, the number of teams working on the question, and the ratio of true to false relationships tested in the field. His simulations find that for most designs and settings a claim is more likely to be false than true. A finding is less likely true when studies are small, effects are small, researchers test many relationships with little preselection, and designs, definitions and analyses are flexible.

Smaldino and McElreath (2016) supply the mechanism that keeps that arithmetic unchanged. Their meta-analysis of statistical power in the behavioural sciences finds power has not improved over sixty years. Their model of competing laboratories shows that selection for high output produces poorer methods and higher false discovery rates without anyone intending it, so the drift is mechanical rather than moral. Replication slows the deterioration but does not stop it, and they conclude that quality requires institutional change.

The practices are ordinary rather than fraudulent

Fanelli (2009) separates misconduct from flexibility by meta-analysing survey data. Across eighteen surveys he estimates that 1.97 per cent of scientists admitted fabricating, falsifying or modifying data at least once, with a confidence interval from 0.86 to 4.45 per cent. Up to 33.7 per cent admitted other questionable research practices, so the false-positive rate is driven mainly by practices that are not fraud.

John, Loewenstein and Prelec (2012) surveyed more than two thousand psychologists with anonymity and explicit incentives for honest reporting, and found the share admitting questionable practices surprisingly high. They conclude that some may amount to the prevailing research norm. Baker (2016) finds the same failure reported in others' work. Of 1,576 respondents to Nature's survey, more than 70 per cent had tried and failed to reproduce another scientist's experiments, and more than half had failed to reproduce their own.

Two replication projects find half the original effect

The Open Science Collaboration (2015) ran the largest test, replicating 100 experimental and correlational studies from three psychology journals with high-powered designs. Ninety-seven per cent of the originals had significant results and 36 per cent of the replications did. Replication effects were about half the magnitude of the originals, and reviewers judged only 39 per cent of the effects to have replicated. A published effect size, on this evidence, is systematically larger than a new study will find.

Camerer and colleagues (2018) found the same pattern outside psychology's own journals. They replicated 21 social science experiments from Nature and Science published between 2010 and 2015, with samples about five times the originals. Thirteen of the twenty-one, 62 per cent, showed a significant effect in the same direction, and replication effects were about half the originals. Their Bayesian analysis put the true-positive rate at 67 per cent.

In cancer biology the missing materials block the test

Errington and colleagues (2021) report the Reproducibility Project: Cancer Biology, which aimed to repeat 193 experiments from 53 high-impact papers and managed 50 from 23. The barriers were practical before they were statistical. Data for computing effect sizes were publicly accessible for four of the 193 experiments and unobtainable for 68 per cent. No experiment carried a description detailed enough to design a protocol unaided. A result that cannot be reconstructed cannot be checked, whatever its statistics.

The published effect is inflated before anyone tries to replicate it

Loken and Gelman (2017) show that with noisy measurement and small samples, selecting on statistical significance raises the reported effect. A significant estimate must sit far enough from zero to pass the threshold, so in a small noisy study it tends to be large even when the underlying effect is small. They warn that reported estimates can be an order of magnitude larger than any plausible underlying effect. They also warn that the common assumption that measurement error always shrinks an effect is false. Scheel, Schijen and Lakens (2021) show the distortion at the level of the literature. Comparing 71 published Registered Reports with a random sample of 152 standard hypothesis-testing papers, they find 96 per cent positive results in the standard literature against 44 per cent in Registered Reports.

The record does not correct itself

Serra-Garcia and Gneezy (2021) test whether news of a failure travels. Across psychology, economics and general-interest journals they find that researchers cite papers which fail to replicate more than comparable papers that replicate. They find that the difference does not change after the failure appears in print. Only 12 per cent of citations made after a failed replication acknowledge it. The incentive that rewarded the original survives the discovery.

The contested question: does a failed replication discredit the original?

One position, taken by the replication projects and by Ioannidis, treats the published record as biased and a single published effect as discounted evidence. The opposing position, stated most directly by Maxwell, Lau and Howard (2015), denies that a failure to replicate is automatically a disconfirmation. A single replication is often underpowered, a true original can fail to reappear in one attempt, and firm evidence usually needs several replications, which is why they recommend meta-analysis and Bayesian methods. Klein and colleagues (2014) add that effects differ in how they behave. In the Many Labs project, the team tested thirteen effects across 36 independent samples totalling 6,344 participants. Ten replicated consistently, two did not, and one was weak, with the effect itself driving the pattern more than the laboratory or sample.

Buzbas and Devezer (2024) press the disagreement further by questioning the assumption that reproducibility is itself a virtue. They argue the useful question is not whether a study reproduced but how far a replication lies from its original. They propose a measure of replication distance and reproducibility rate, illustrated by simulation, together with purposefully planned modifications rather than exact repeats. What would settle the dispute is pre-registered multi-site replications scored on a stated replication-distance metric, read alongside a citation audit.

Read by kind, the replication side rests on multi-site projects, one of a hundred studies and one of twenty-one, and on a model of how often a published claim is true. The dissent rests on an argument about the power of a single repeat and on a many-labs test in which ten of thirteen effects held. While the dispute is open, a project should report its estimate with an interval, state the effect its design could detect, say which population its sample can stand for, and name the repeat that would test the claim without treating one failure as a refutation. The strongest objection to this unit's thesis is that one study built for its own question is good evidence for that question, so discounting it may push a student to underclaim.

What this means for a project

Take a project whose evidence is one laboratory study of twenty-four participants drawn from one population, comparing two interfaces. It may report a difference in completion time under those conditions, with an interval. It may state the smallest effect the design could detect. It may not present that difference as the answer to the question, because the design cannot rule out the inflation Loken and Gelman describe and no second study tests the claim. The honest closing paragraph names the study that would: a pre-registered repeat at larger scale with a stated replication distance. Naming the missing test makes the claim one estimate among many rather than a settled fact.

Where this is heading

Buzbas and Devezer (2024) argue for replication distance as the quantity a field should track. The citation evidence of Serra-Garcia and Gneezy (2021) points to the harder problem: a literature will not correct itself while the uncorrected finding remains the better citation. Korbmacher and colleagues (2023) read the same decade instead as a credibility revolution, and catalogue the mandates, tools and communities that have already changed the machinery around the record. The question is whether the metric or the incentive moves first.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked against Crossref, Europe PMC and the publisher record on 31 October 2026.

• Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. doi:10.1371/journal.pmed.0020124 Models how power, design bias and the number of tested relationships decide whether a published claim is true. Finds a claim is more likely false than true for most designs, so single-study credibility is structural.
• Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), Article e5738. doi:10.1371/journal.pone.0005738 Meta-analysed 21 surveys of research behaviour, 18 in the pooled estimate. Estimates 1.97 per cent admitted fabricating or falsifying data at least once against up to 33.7 per cent admitting other questionable practices, locating the problem in ordinary practice rather than fraud.
• John, L. K., Loewenstein, G. and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532. doi:10.1177/0956797611430953 Surveyed more than two thousand psychologists with anonymity and incentives for honest reporting. Finds the share admitting questionable practices is surprisingly high and argues some may have become the prevailing research norm.
• Klein, R. A. and colleagues (2014). Investigating variation in replicability: A many labs replication project. Social Psychology, 45(3), 142-152. doi:10.1027/1864-9335/a000178 Tested 13 effects across 36 independent samples totalling 6,344 participants. Ten replicated consistently, two did not and one was weak, with the laboratory and sample predicting little, so a single result is not simply true or false.
• Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. doi:10.1126/science.aac4716 Replicated 100 studies from three psychology journals with high-powered designs. Found 97 per cent of originals significant against 36 per cent of replications, effects about half the original magnitude, and 39 per cent judged to have replicated.
• Maxwell, S. E., Lau, M. Y. and Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does failure to replicate really mean? American Psychologist, 70(6), 487-498. doi:10.1037/a0039400 Argue that many failures to replicate reflect the low power of single replication studies rather than false originals, and recommend meta-analysis and Bayesian methods. Carries the dissent that a non-replication is not automatically a disconfirmation.
• Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452-454. doi:10.1038/533452a Reports Nature's survey of 1,576 researchers. More than 70 per cent had tried and failed to reproduce another scientist's experiments and more than half their own, with 52 per cent describing a significant crisis.
• Smaldino, P. E. and McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), Article 160384. doi:10.1098/rsos.160384 Combine a sixty-year meta-analysis of statistical power with a model of competing laboratories. Power has not improved, and selection for high output spreads poorer methods and raises false discovery rates without deliberate cheating; replication slows but does not stop it.
• Loken, E. and Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584-585. doi:10.1126/science.aal3618 Show that with noisy measurement and small samples, selecting on statistical significance inflates the reported effect, which can be an order of magnitude larger than any plausible underlying effect. Supplies the mechanism behind shrinking replication effects.
• Camerer, C. F. and colleagues (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644. doi:10.1038/s41562-018-0399-z Replicated 21 social science experiments on pre-registered plans with samples about five times the originals. Thirteen replicated in the same direction, effects were about half the originals, and the true-positive rate was estimated at 67 per cent.
• Scheel, A. M., Schijen, M. R. M. J. and Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). doi:10.1177/25152459211007467 Compared 71 published Registered Reports with a random sample of 152 standard hypothesis-testing papers. Finds 96 per cent positive results in the standard literature against 44 per cent in Registered Reports, so the record overstates support for its hypotheses.
• Errington, T. M. and colleagues (2021). Challenges for assessing replicability in preclinical cancer biology. eLife, 10, Article e67995. doi:10.7554/eLife.67995 Report the Reproducibility Project: Cancer Biology, which aimed to repeat 193 experiments from 53 papers and completed 50 from 23. Data for effect sizes were public for 4 of 193 experiments and unobtainable for 68 per cent, so missing materials block replication before statistics.
• Serra-Garcia, M. and Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21), Article eabd1705. doi:10.1126/sciadv.abd1705 Used public citation data across psychology, economics and general-interest journals. Finds papers that fail to replicate are cited more than comparable replicable papers, the difference persists after the failure is published, and only 12 per cent of later citations acknowledge it.
• Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., et al. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1, Article 3. doi:10.1038/s44271-023-00003-2 Reframes the crisis as a credibility revolution and catalogues structural, procedural and community change, including mandates, multiverse analysis, statistical screening tools and open-scholarship networks. It reads the change as durable, and documents it rather than measuring its effect.
• Buzbas, E. O. and Devezer, B. (2024). Statistics in service of metascience: Measuring replication distance with reproducibility rate. Entropy, 26(10), Article 842. doi:10.3390/e26100842 Argue against treating reproducibility as inherently desirable and propose a measure of replication distance and reproducibility rate, illustrated by simulation. Gives the unit's direction: the question becomes how far a result is from its original, not whether it reproduced.