Preregistration, HARKing and the planned/emerged boundary

A project that ran its study in March and analysed it in June cannot remember, in July, what it expected in March.

The third of ten units argues that separating what was planned from what emerged is a reporting obligation rather than a design choice, while the fourth asks what one study can carry. A report must say which analysis the team committed to in advance, which decision changed after the data arrived, and which finding the study discovered rather than predicted.

HARKing is a writing practice, not a data practice

Kerr (1998) named the failure. HARKing, or hypothesising after the results are known, is presenting a post hoc hypothesis, one informed by the results, as if it were an a priori hypothesis. He reports survey evidence that researchers practise some of these forms widely while judging them inappropriate. He argues that the practice damages cumulative knowledge.

The writer commits the offence in the introduction, by rewriting what the study was for. Treating post hoc reasoning as fraud overshoots his evidence. The objection is to the pretence of prediction, not to the reasoning.

The flexibility sits in the whole study, not only in the test

Simmons, Nelson and Simonsohn (2011) turn the problem into a measurement and a remedy. Using simulations and two experiments, they show that flexibility in data collection, analysis and reporting raises false-positive rates. A researcher is then often more likely to find false evidence that an effect exists than to find correctly that it does not. Their answer is disclosure: six requirements for authors and four guidelines for reviewers. They collected data from twenty students who listened to either the Beatles song When I'm Sixty-Four or Kalimba, and the students then reported their birth date and their father's age. One defensible analysis controlled for the father's age and found the listeners nearly one and a half years younger, F(1, 17) = 4.92, p = 0.040. The analysis without the covariate lost the finding. Nothing in the data changed between the two reports, only the analysis, and neither version was improper in itself.

Wicherts and colleagues (2016) widen the scope. They list thirty-four researcher degrees of freedom across a study's planning, running, analysis and reporting. They offer the list as a checklist for judging a preregistration or the bias risk of an unregistered study. A comparison of twenty-four participants with several defensible exclusions and two outcome measures contains more than one publishable analysis, and the exclusions decide whose data enter it.

A registry marks prediction from postdiction

Nosek and colleagues (2018) give preregistration its rationale in one distinction. Science generates hypotheses with existing observations and tests them with new ones, and ordinary hindsight bias makes the two hard to keep apart. Preregistration marks which analyses follow from a prediction and which from a postdiction.

Cockburn, Gutwin and Dix (2018) ask what the instrument means in this field. They review preregistration's motivation and outcomes across disciplines, and they ask how it could be adapted to HCI empirical research. The benefit is a reader's confidence that the evidence for a hypothesis is not the product of HARKing.

The adherence problem: a registration nobody reads proves nothing

Claesen and colleagues (2021) compared every Preregistered-badge article in Psychological Science from February 2015 to November 2017 against its registration. Two of twenty-seven contained no deviations. Of the rest, one disclosed all of its deviations and nine disclosed none. The deviations clustered in sample size, exclusion criteria and statistical analysis.

Syed (2025) asked whether the deposit is ever opened, and found that across 201 preregistered PLOS articles only 14 per cent of review records recorded the registration being accessed.

Bakker and colleagues (2020) approach the weakness from the design side. They compare an unstructured Open Science Framework format with a structured workflow and independent review, on fifty-three registrations each. The structured format restricted the opportunistic use of researcher degrees of freedom better, with a Cliff's delta of 0.49, but neither eliminated all researcher degrees of freedom. Yamada (2018) names the ways a researcher can work around the format, including pre-registering after the results are known, which he calls PARKing:

There is currently no method for journals or reviewers to detect PARKing.

A boundary can be labelled without a registry

If the obligation is the labelled boundary, a registry is one way of meeting it, not the only one. The checklist reviewed by Wicherts and colleagues (2016) needs no registration. A team can publish it against a finished study to show where the arbitrary decisions lay. Simonsohn, Simmons and Nelson (2020) offer a stronger alternative. Their specification curve analysis identifies every theoretically justified and non-redundant specification of a hypothesis, displays the results so consequential decisions are visible, and conducts joint inference across them. It answers the flexibility objection by showing its effect, rather than by pre-committing to one path. Across the specifications they examine, one of their three findings holds, one is weak and one does not hold at all.

The contested question: does the format work, and who pays for it?

Chambers (2013) introduced the format at Cortex, where a proposal is peer reviewed and pre-accepted before the research is undertaken, so that publication turns on the methods rather than the results. Chambers and Tzavella (2022) review its history and prospects and report early evidence that the format works as intended, while conceding that it is not a universal solution. Lakens and colleagues (2024) confirm from metascientific work that the formats meet their aims and improve study quality. They find little empirical support for the criticisms, and they concede one: a reader may use a preregistration as a proxy for quality without reading it.

The opposing position denies not that the instruments are useful but that they are the discipline. The registry is a deposit, and the comparison sits with a reader who mostly does not make it. Claesen and colleagues (2021) and Yamada (2018) supply the failure. Hardwicke and colleagues (2020) supply the frame, arguing that reform should be calibrated by meta-research that evaluates its own solutions.

Haven and Van Grootel (2019) answer the objection that preregistration is incompatible with inductive work. Making hypotheses explicit is only one feature of the format, flexibility can be tracked rather than forbidden, and a registration checks a subjectivity the design does not otherwise constrain. Allen and Mehler (2019) set three benefits of open practices against three challenges, naming reduced flexibility, extra time and an unrewarding incentive structure as costs falling most acutely on early-career researchers. They report that null findings are substantially more likely to be published through Registered Reports.

What would settle this is a measurement, and the first now exists. Syed (2025) reads the review records rather than the papers, and finds the deposit is often not opened. That is enough to show a registry is not self-enforcing. No study yet pairs a labelled deviation with the reviewer who saw it.

Read by kind, the evidence is uneven. The case for the formats rests on two reviews, both by the formats' own advocates. The case against rests on one audit of twenty-seven badged articles and one list of ways around a registration. The direct measurements are small, and all cut against the formats. While the dispute is open, a project should register what it can, date every deviation, and treat a registration as a record, not a certificate. The strongest objection to this unit's thesis is that a decision log kept honestly meets the same obligation, so a registration adds a deposit a reader may never open.

What this means for the project

Consider a project that preregistered one contrast between two conditions of a text-entry study, then found the difference depends on the input method, including participants who used switch or on-screen keyboards. The contrast was planned and reported as planned. The interaction was not planned, and the paper reports it as exploratory. The team also reworded the prompt after six participants, and that version boundary belongs in the record. A registration silent about the other two is less honest than none.

Where this is heading

Chambers and Tzavella (2022) conclude that Registered Reports are promoting reproducibility, transparency and self-correction, while setting out the policy changes they judge necessary. Lakens and colleagues (2024) close by recommending that the quality, not the quantity, of preregistration be raised. The structured format tested by Bakker and colleagues is the clearest instance so far. It adds a template and a review step, which constrain what a writer may change after the data arrive.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked on 31 October 2026.

• Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217. doi:10.1207/s15327957pspr0203_4 Names the practice of presenting a post hoc hypothesis as though it were a priori, and reports survey evidence that some forms are widely practised and widely seen as inappropriate. It locates the offence in the writing, not the data.
• Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. doi:10.1177/0956797611417632 Shows through simulations and experiments that ordinary flexibility raises false-positive rates so far that an analyst is often more likely to find a false effect than a true one. Its remedy is disclosure.
• Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., et al. (2016). Frontiers in Psychology, 7, Article 1832, a review of researcher degrees of freedom. doi:10.3389/fpsyg.2016.01832 Lists thirty-four researcher degrees of freedom across hypotheses, design, running, analysis and reporting, as a checklist for judging a preregistration or the bias risk of an unregistered study. The boundary spans the whole study.
• Cockburn, A., Gutwin, C. and Dix, A. (2018). HARK no more: On the preregistration of CHI experiments. CHI '18, Paper 141, 1-12. doi:10.1145/3173574.3173715 Reviews the motivation and outcomes of experimental preregistration across disciplines, then asks how it could be adapted to HCI research. It shows the instrument argued for in this field on its own terms.
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C. and Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606. doi:10.1073/pnas.1708274114 Argues that hindsight bias makes prediction and postdiction hard to keep apart, and that preregistration separates them, improving credibility whatever the result.
• Claesen, A., Gomes, S., Tuerlinckx, F. and Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), Article 211037. doi:10.1098/rsos.211037 Compared every Preregistered-badge article in Psychological Science from 2015 to 2017 against its registration: only two of twenty-seven contained no deviations, and nine disclosed none.
• Syed, M. (2025). Some data indicating that editors and reviewers do not check preregistrations during the review process. Journal of Trial and Error, 6(1), 8-19. doi:10.36850/e5ce-4cc5 Coded the published review histories of 201 preregistered PLOS articles for mention, access and comparison of the registration. At the article level the rates were 43, 14 and 10 per cent, falling to 18, 5 and 3 per cent by person. It shows a registration is often never opened.
• Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., et al. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), Article e3000937. doi:10.1371/journal.pbio.3000937 Compared a structured preregistration workflow with independent review against an unstructured standard, fifty-three registrations each, and found the structured format restricted opportunistic flexibility better (Cliff's delta 0.49), though neither eliminated all degrees of freedom.
• Yamada, Y. (2018). How to crack pre-registration: Toward transparent and open science. Frontiers in Psychology, 9, Article 1831. doi:10.3389/fpsyg.2018.01831 Examines how preregistration can be worked around, naming selective reporting, repeated experimenting, overissuing near-identical registered studies, and pre-registering after the results are known, which he calls PARKing, and concludes that no method currently lets reviewers or journals detect it.
• Haven, T. L. and Van Grootel, D. L. (2019). Preregistering qualitative research. Accountability in Research, 26(3), 229-244. doi:10.1080/08989621.2019.1580147 Argues that qualitative research can be preregistered, treating it as a real case of postdiction research, and answers the objections that it tests no hypotheses and that its design is too flexible to fix.
• Simonsohn, U., Simmons, J. P. and Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4(11), 1208-1214. doi:10.1038/s41562-020-0912-z Introduces specification curve analysis, which identifies every theoretically justified and non-redundant specification, displays the results so consequential decisions are visible, and conducts joint inference across them.
• Hardwicke, T. E., Serghiou, S., Janiaud, P., et al. (2020). Calibrating the scientific ecosystem through meta-research. Annual Review of Statistics and Its Application, 7, 11-37. doi:10.1146/annurev-statistics-031219-041104 Sets out a translational framework for research on research: identify a problem, investigate it, develop a solution and evaluate it. Its premise that the ecosystem keeps evolving rules out assuming a new format has improved anything.
• Allen, C. and Mehler, D. M. A. (2019). Open science challenges, benefits and tips in early career and beyond. PLOS Biology, 17(5), Article e3000246. doi:10.1371/journal.pbio.3000246 Sets three benefits of open practices against three challenges, naming reduced flexibility, extra time and an unrewarding incentive structure as costs falling most acutely on early-career researchers.
• Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609-610. doi:10.1016/j.cortex.2012.12.016 Announces the format reviewed before data collection and accepted in principle, and holds that a post hoc analysis stays legitimate once it is labelled as such. It is the primary description of the format behind the later review.
• Chambers, C. D. and Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6(1), 29-42. doi:10.1038/s41562-021-01193-7 Reviews the history, progress and prospects of Registered Reports, in which a proposal is peer reviewed and pre-accepted before the research is done, and reports early evidence that the format works as intended while denying that it is universal.
• Lakens, D., Mesquida, C., Rasti, S. and Ditroilo, M. (2024). The benefits of preregistration and Registered Reports. Evidence-Based Toxicology, 2(1), Article 2376046. doi:10.1080/2833373x.2024.2376046 Reviews metascientific research showing that preregistration and Registered Reports meet their aims and improve study quality, examines the criticisms, and finds little empirical support for them. It concedes that a preregistration can act as a proxy for quality on its own.