Researcher degrees of freedom turn noise into findings

The previous unit left one study as weak evidence; this one says why. This unit argues that significance is partly a property of the researcher's choices rather than of the world, so an absent effect can be presented as a present one. A significance test is usually read as a property of the world: it reports how often a result as extreme as the one observed would appear if nothing were going on. That reading is wrong. The number comes from a chain of decisions that belongs to the researcher: which outcome to measure, which participants and trials to keep, which comparisons to run, when to stop collecting data. The defence is not a better statistic.

A significance test is a summary of a chain of choices

Wicherts and colleagues count the decisions open to a researcher and reach thirty-four (Wicherts et al., 2016). Used opportunistically, they argue, these raise the chance of a false positive and inflate effect-size estimates, and the count becomes a checklist to complete before data collection. A study that records several outcomes can report the one that moved, so the measure matters. Participants or trials can be excluded on grounds that look principled and were chosen once the pattern was visible, so the subset matters. Collection can end when the result looks good, so the stopping point matters.

Reporting a decision as though it had been made in advance has a name. Kerr called it HARKing, hypothesising after the results are known, and described presenting a post hoc hypothesis as if it were a priori (Kerr, 1998). Gelman and Loken remove the need for bad intent. An analysis whose details are contingent on the data invalidates the published p value even when the researcher ran one analysis and did no conscious fishing. Gelman and Loken name the situation the garden of forking paths.

One demonstration manufactured a false result

Simmons, Nelson and Simonsohn gave the problem its central demonstration. Using simulations and a pair of experiments, they showed that flexibility in data collection, analysis and reporting dramatically increases the real false-positive rate. A researcher is then in many cases more likely to find evidence for an effect that is not there than for one that is (Simmons et al., 2011). The experiments are the part that travels. They accumulated significant evidence for a hypothesis they knew to be false, using only decisions a reviewer would have accepted.

Researchers admit the choices under survey

The practice sits in ordinary research rather than in fraud. John, Loewenstein and Prelec surveyed 2,155 psychologists with incentives for honest reporting. Self-admission rates were 63.4 per cent for failing to report all dependent measures, and 55.9 per cent for deciding whether to collect more data after checking significance. They conclude that some questionable practices may be the prevailing norm (John et al., 2012). Fanelli's meta-analysis found 1.97 per cent admitting fabricating, falsifying or modifying data at least once, while up to 33.7 per cent admitted other questionable practices (Fanelli, 2009). Across over 4,600 papers published between 1990 and 2007, the frequency of positive support for a tested hypothesis grew by over 22 per cent. The growth was steeper in the social and some biomedical disciplines (Fanelli, 2012).

The fingerprint the practice leaves

Rosenthal (1979) set out the file drawer problem. In its extreme form, the journals publish the 5 per cent of studies that show a Type I error. The drawers of the laboratory hold the 95 per cent that found nothing significant. He then made the worry computable, with a procedure for how many filed null results would be needed to overturn a published conclusion. Head and colleagues used text-mining to find the shape a p-value distribution takes when non-significant results are pushed over the line. They found an excess of values just below 0.05 with a deficit just above (Head et al., 2015). Simonsohn, Nelson and Simmons supply the diagnostic that reads that shape. P-curve is the distribution of significant p values across a set of studies. Only true effects should produce a right-skewed curve, so it tells a reader whether findings carry evidential value or are explained by selective reporting (Simonsohn et al., 2014). Among almost 250 funded social-science experiments, Franco, Malhotra and Simonovits found that only 10 of 48 null results were published, against 56 of 91 with strongly significant results (Franco et al., 2014).

Contested question: does p-hacking distort the literature, or is its effect weak?

The literature agrees that flexibility inflates individual findings and disagrees about how far the damage reaches. One position holds that the practice distorts the published record at scale. The evidence is how common the choices are (John et al., 2012; Wicherts et al., 2016) and the raised false-positive rate in simulation (Simmons et al., 2011). The full file drawer (Franco et al., 2014) and the detectable fingerprint (Head et al., 2015) point the same way. The opposing position holds that the aggregate effect is weaker than the rhetoric suggests. Its strongest statement is internal to the same evidence. Head and colleagues found p-hacking widespread, yet concluded that its effect seems weak relative to the real effect sizes measured (Head et al., 2015). Kerr had already conceded that the case against HARKing was not settled empirically (Kerr, 1998).

What would settle the disagreement is a change in reporting rather than another argument about the same papers. If authors published the whole set of defensible analyses, the spread of conclusions a dataset can support would become visible instead of inferred. That is what Dragicevic and colleagues' multiverse reports invite (Dragicevic et al., 2019).

The two sides draw on evidence of different kinds. The distortion position rests on two self-report surveys, on a simulation, on a text-mining study of p-value distributions and on a count of published against funded results. The weak-effect position rests on one sentence of the same text-mining study and on a concession in the founding paper on HARKing. One side measures what researchers report and what the record looks like. The other infers the size of the effect from the same data. While the dispute is open, a designer should fix and record the outcome, the exclusions and the stopping rule before the data are seen.

Why the defence is not a better statistic

The obvious response is to replace the p value with something better behaved. Bishop's diagnosis explains why that is not enough. She argues that irreproducibility has four sources, publication bias, low statistical power, p-value hacking and HARKing, each needing its own remedy rather than a single statistical fix (Bishop, 2019). A new test changes how a result is summarised. It does not stop a researcher choosing the measure, the subset or the stopping point after seeing the data. The remedy that meets the choices directly is to fix and record them before the data are seen. Wicherts and colleagues' checklist is built for that (Wicherts et al., 2016). The checklist has a cost: a researcher with no time to spare, or a team with no second coder, carries more of the risk than one that can complete it.

Interaction research adds a choice of outcome measure

Interaction research has its own version of the problem, because its outcomes are often chosen rather than fixed by a standard measure. Dragicevic and colleagues present explorable multiverse analysis reports, in which a reader can re-run a study's analysis under alternative pipelines and watch the conclusion move. They developed the format from five examples and a design-space analysis (Dragicevic et al., 2019). The point is not that the original analysis was wrong but that a reader can see how much of a conclusion survives a reasonable change of decision.

The strongest counter-argument: the measured effect is weak

The strongest objection is that the thesis treats the freedom to analyse as the freedom to deceive. Its firmest source is Head and colleagues' finding that p-hacking's effect appears weak relative to the real effects measured (Head et al., 2015). Kerr's concession points the same way (Kerr, 1998). If they are right, a significant result is only mildly inflated by the choices behind it. The reply is that the thesis concerns the warrant of a single reported number rather than the health of the aggregate literature. Head and colleagues still found the practice widespread and Franco and colleagues still found the file drawer full. A reader who cannot tell which decisions were made in advance cannot judge the warrant of the number at all.

Where the argument is heading

Dragicevic and colleagues make that space interactive in the paper (Dragicevic et al., 2019). Wicherts and colleagues turn it into a checklist completed before collection (Wicherts et al., 2016). Simmons, Nelson and Simonsohn now treat preregistration of analyses as the central requirement (Simmons et al., 2018). Bishop's four horsemen keeps the reforms from collapsing into one, since each source needs its own remedy (Bishop, 2019). Whether the reform reaches interaction research, where the outcome measure is often the researcher's own construction, remains open.

The sources this unit rests on

Each note states what the cited document did and found rather than quoting it, and every record was checked against the publisher's registered metadata and abstract on 1 November 2026.

• Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., et al. (2016). Degrees of freedom in planning, running, analysing, and reporting psychological studies: a checklist to avoid p-hacking. doi:10.3389/fpsyg.2016.01832 Catalogues 34 degrees of freedom open to a researcher, arguing that using them opportunistically inflates false positives and effect sizes, and turns the list into a checklist to complete before data collection. It is the unit's inventory of decisions and its constructive remedy.
• Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2018). False-positive citations. doi:10.1177/1745691617698146 Examines how the authors' own false-positive demonstration has been cited as though it supported the findings it was written to undermine, and argues that integrity depends on full disclosure and preregistration of analyses. It shows the problem spreading through citation as well as analysis.
• Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. doi:10.1177/0956797611417632 Shows by simulation and experiment that flexibility in collection, analysis and reporting raises the real false-positive rate far above the nominal threshold, accumulating significant evidence for a hypothesis known to be false. It is the founding demonstration that analyst choices produce findings.
• Kerr, N. L. (1998). HARKing: hypothesizing after the results are known. doi:10.1207/s15327957pspr0203_4 Names HARKing, presenting a post hoc hypothesis as though stated in advance, reports survey data suggesting some forms are widely practised, and concedes that whether the costs exceed the benefits is a question for research. It is the origin of the planned and exploratory distinction the unit recommends.
• Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. doi:10.1371/journal.pone.0005738 Meta-analysed 21 misconduct surveys: 1.97 per cent admitted fabricating or falsifying data at least once, while up to 33.7 per cent admitted other questionable practices, placing the milder practices on a continuum with misconduct.
• Dragicevic, P., Jansen, Y., Sarma, A., et al. (2019). Increasing the transparency of research papers with explorable multiverse analyses. doi:10.1145/3290605.3300295 Presents reports in which a reader can re-run a paper's analysis under alternative pipelines and see the conclusion move, developed from five examples and a design-space analysis. It is the unit's HCI-specific remedy, restoring the reader's ability to judge robustness.
• John, L. K., Loewenstein, G. and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. doi:10.1177/0956797611430953 Surveyed 2,155 psychologists with incentives for honest reporting: 63.4 per cent admitted failing to report all dependent measures and 55.9 per cent admitted deciding whether to collect more data after checking significance. It is the self-report evidence that the behaviour is common.
• Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. doi:10.1007/s11192-011-0494-7 Analysed over 4,600 papers published between 1990 and 2007 and found positive support for a tested hypothesis growing by over 22 per cent, more steeply in the social and some biomedical fields. It is the field-level evidence that the incentive structure drives the pattern.
• Simonsohn, U., Nelson, L. D. and Simmons, J. P. (2014). P-curve: a key to the file-drawer. doi:10.1037/a0033242 Introduces p-curve, the distribution of significant p values across a set of studies: only true effects should produce a right-skewed curve, so a flat one indicates selective reporting. It is the diagnostic showing the practice leaves a measurable trace.
• Franco, A., Malhotra, N. and Simonovits, G. (2014). Publication bias in the social sciences. doi:10.1126/science.1255484 Compared almost 250 funded social-science experiments with what was published: only 10 of 48 null results appeared, against 56 of 91 studies with strongly significant results. It is the direct measure of the publication side of the problem.
• Bishop, D. (2019). Rein in the four horsemen of irreproducibility. doi:10.1038/d41586-019-01307-2 Argues that irreproducibility has four sources, publication bias, low statistical power, p-value hacking and HARKing, each needing its own remedy rather than a single reform. A commentary rather than a study, it supplies the remedy map the unit uses.
• Head, M. L., Holman, L., Lanfear, R., et al. (2015). The extent and consequences of p-hacking in science. doi:10.1371/journal.pbio.1002106 Used text-mining to test for an excess of p values just below 0.05 and found it widespread across science, but concluded that p-hacking's effect seems weak relative to the real effect sizes measured. It is the sharpest limit on the thesis.
• Gelman, A. and Loken, E. (2014). The statistical crisis in science. doi:10.1511/2014.111.460 Argues that an analysis whose details are contingent on the data invalidates the published p value even without conscious fishing, calling it the garden of forking paths, and recommends analysing all the data. It explains how ordinary analysis produces the problem.
• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin. doi:10.1037/0033-2909.86.3.638 Names the file drawer problem, in which journals are filled with the 5 per cent of studies showing a Type I error while the drawers hold the 95 per cent that found nothing significant, and gives a procedure for how many filed null results would overturn a published conclusion.