Open science answers the failures, at a price

The earlier units ended in a set of failures: significance shaped by the analyst's choices, findings resting on a single study, low replication rates in HCI and in the sciences next door. Open science is the field's answer. Its main measures, preregistration, the registered report and open data, move decisions ahead of the data, so a reader can tell what was planned from what was found. This unit argues that the evidence for these measures is real but partial. They change behaviour and improve some things without removing the failures.

The remedy set, and what each measure claims to fix

Munafo and colleagues set the case out as a single programme, covering methods, reporting, reproducibility, evaluation and incentives. They name the failure each element answers (Munafo et al., 2017). They state that the evidence is partial and will need iterative evaluation. The measure with the widest reach is preregistration, and Nosek and colleagues give it its clearest statement. It separates analyses and outcomes that result from predictions from those that result from postdictions. Hindsight bias makes the two feel alike afterwards, so the only reliable separation is made before the data are seen (Nosek et al., 2018).

In their words:

Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. (Nosek et al., 2019)

Registered reports are the strongest remedy the field has built

Chambers announced the format at Cortex in 2013. A journal reviews and can accept a study's method before the data are collected, so the decision to publish no longer depends on the result (Chambers, 2013). A decade later, Chambers and Tzavella report early evidence that registered reports work as intended. They state that they are not a universal solution for irreproducibility (Chambers and Tzavella, 2022). Scheel, Schijen and Lakens (2021) tested that claim directly: of 71 published registered reports, 44 per cent reported a positive result, against 96 per cent of a random sample of 152 standard hypothesis-testing studies. By removing the result from the publication decision, the format removes the incentive that turned noise into findings.

Where a policy is enforced, behaviour and sharing both move

The strongest quasi-experimental evidence that an incentive changes practice comes from badges. Kidwell and colleagues compared Psychological Science, which offered badges for open data and open materials from January 2014, with four comparison journals. Before badges fewer than 3 per cent of articles reported open data. After badges 23 per cent did, rising to 39 per cent in the first half of 2015, while comparison journals did not change (Kidwell et al., 2016). Data were also more likely to be genuinely available when a badge had been earned than when it had not.

Adherence, not adoption, is where the remedies narrow

Claesen and colleagues compared the first generation of preregistered studies, published with a badge in Psychological Science, against their own plans. Only two of 27 studies contained no deviations, and nine disclosed none of theirs. The deviations clustered on sample size, exclusion criteria and statistical analysis (Claesen et al., 2021). A preregistration that is not followed and whose departures are not disclosed gives the reader the false assurance the remedy was meant to remove.

Hardwicke and colleagues evaluated a mandatory open data policy at the journal Cognition. Availability statements rose from 25 per cent of articles before the policy to 78 per cent after it, and reusable data from 22 per cent to 62 per cent (Hardwicke et al., 2018). Of 1,324 reported values re-checked from those articles, 64 could not be reproduced within a ten per cent margin. For 13 articles at least one value failed even with help from the authors, though conclusions were not seriously affected. Availability is not reproducibility. Sharing is also uneven. Wicherts, Bakker and Molenaar (2011) asked authors for the data behind 1,148 statistically significant results reported in 49 papers in two psychology journals. Reluctance to share was greatest where the evidence was weakest and where the article carried more errors in its statistics.

The remedies are asked to do more than they were built to do

The case against open science is not that the remedies fail but that they are asked to do more than they can. Szollosi and colleagues argue that preregistration is defended for improving the diagnosticity of statistical tests, and that the defence fails. Whether a test is diagnostic depends on how well the statistical model maps onto the theory. Registration does not improve that mapping (Szollosi et al., 2020). The cost also falls unevenly. Allen and Mehler describe three benefits and three challenges of open science for early-career researchers. They find that null findings are substantially more likely to be published as registered reports, while the practices cost flexibility and time (Allen and Mehler, 2019). The cost falls on the connection as well: an author with a metered connection or no institutional repository can complete the plan and still not deposit the data. Chakravorti, Gautam and Rajtmajer interviewed 18 early-career HCI researchers with mixed-methods or qualitative backgrounds. They report barriers of missing incentives, cultural resistance and intellectual-property concern (Chakravorti et al., 2025).

Contested question: does preregistration earn its price?

The literature agrees that moving decisions ahead of the data is the right direction and disagrees about what preregistration is for. Nosek and colleagues hold that the practice replaces after-the-fact reporting with planned prediction. It clarifies planned and unplanned research by reducing unnoticed flexibility, which is worth the difficulty it imposes (Nosek et al., 2018; Nosek et al., 2019). Szollosi and colleagues hold that this defence misidentifies the source of weak inference. If the statistical model maps poorly onto the theory, then knowing in advance which analysis was planned improves nothing that matters (Szollosi et al., 2020). The costs fall where the critics place them, on the flexibility and time Allen and Mehler describe and on the early-career researchers Chakravorti and colleagues interviewed. What would settle the disagreement is not another editorial but evidence on two questions. Are preregistered studies better adhered to? Claesen and colleagues begin to answer with a negative finding for the first generation (Claesen et al., 2021). Does the format change the conclusions the field reaches? Chambers and Tzavella report only early evidence (Chambers and Tzavella, 2022).

The two positions are not symmetric in evidence. The case for preregistration rests on a quasi-experimental comparison of one journal that offered badges against four that did not, and on the adoption record of registered reports. The case against rests on a conceptual argument about how a statistical model maps onto a theory, and no study tests it at scale. The strongest objection is that the price is measured and the benefit is not, since no study yet compares the conclusions a field reaches with and without preregistration. While the dispute is open, a designer should label planned and exploratory work separately, and treat a registered result as evidence about the planning rather than about the theory.

Open data is a promise with an expiry date

Open data carries a second limit that no policy addresses: the object it preserves does not last by itself. Vines and colleagues requested datasets from 516 articles published between two and twenty-two years earlier. Where authors reported the status of their data, the odds that a dataset was still extant fell by about 17 per cent per year. The odds of a working email address for a first, last or corresponding author fell by about 7 per cent per year (Vines et al., 2014). Data cannot be reliably preserved by individual researchers, which is why the remedies that work direct deposit to independent archives.

What no single remedy can carry

The bounding claim comes from Munafo and Davey Smith. They argue that robustness is not a property of any single study or remedy, and that knowledge becomes dependable only when many independent lines of evidence converge (Munafo and Davey Smith, 2018). Preregistration can raise the credibility of one claim. A registered report can remove the result from the publication decision. Open data can let a reader re-run an analysis. None supplies the independent line of evidence a finding needs.

The honest position: label planned and exploratory work separately

If the remedies are partial, the position the evidence supports is not to abandon them but to stop them doing work they cannot do. The distinction Nosek and colleagues draw between prediction and postdiction is the one to keep. The same authors restate it as respecting both what was planned and what happened when claims are written (Nosek et al., 2018; Nosek et al., 2019). Szollosi and colleagues do not ask for the practice to be dropped, and their argument is compatible with it. The claim attached to a preregistered result must be about the planning rather than the theory. Exploratory analysis is not the enemy; unlabelled exploratory analysis presented as confirmation is. The remedy buys credibility with freedom: the researcher gives up the freedom to present a post hoc finding as a prediction and keeps the freedom to find it.

Where the argument is heading

The direction of travel is from a policy attached to the paper to a property of the workflow. Chambers and Tzavella report the registered-report format spreading across disciplines, while acknowledging that it is not universal. They describe the policies and practices that must still change to address its limitations (Chambers and Tzavella, 2022). In interaction research the frontier is narrower and more concrete. Chakravorti and colleagues argue that relatively small changes at CHI and CSCW could shift community norms. They add that policy changes do not by themselves change what an early-career researcher can afford to do (Chakravorti et al., 2025). For a field whose evidence is so often qualitative and whose datasets are recordings of people, the open-data promise is the hardest to keep.

The sources this unit rests on

Each note states what the cited document did and found rather than quoting it, and every record was checked against the publisher's registered metadata and abstract on 1 November 2026.

• Munafo, M. R., Nosek, B. A., Bishop, D. V. M., et al. (2017). A manifesto for reproducible science. doi:10.1038/s41562-016-0021 Sets out measures across methods, reporting, reproducibility, evaluation and incentives, and states that the evidence for them is partial.
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C., et al. (2018). The preregistration revolution. doi:10.1073/pnas.1708274114 Argues that progress depends on separating prediction from postdiction, and that preregistration makes the separation credible.
• Chambers, C. D. (2013). Registered reports: a new publishing initiative at Cortex. doi:10.1016/j.cortex.2012.12.016 Announces the registered-report format, in which a journal can accept a study's method before data collection, so publication no longer depends on the result.
• Chambers, C. D. and Tzavella, L. (2022). The past, present and future of registered reports. doi:10.1038/s41562-021-01193-7 Reviews the first years of registered reports, reports early evidence that they work as intended, and states that they are not universal.
• Kidwell, M. C., Lazarevic, L. B., Baranski, E., et al. (2016). Badges to acknowledge open practices. doi:10.1371/journal.pbio.1002456 Beginning January 2014, compared Psychological Science, which offered open-data badges, with four journals without them: reported open data rose from under 3 per cent to 23 per cent, and to 39 per cent in the first half of 2015.
• Claesen, A., Gomes, S., Tuerlinckx, F., et al. (2021). Comparing dream to reality: adherence of the first generation of preregistered studies. doi:10.1098/rsos.211037 Compared the plans of 27 badge-holding studies against the published papers: only two contained no deviations and nine disclosed none of theirs.
• Hardwicke, T. E., Mathur, M. B., MacDonald, K., et al. (2018). Data availability, reusability, and analytic reproducibility at Cognition. doi:10.1098/rsos.180448 Evaluated a mandatory open data policy: availability statements rose from 25 to 78 per cent and reusable data from 22 to 62 per cent, but 64 of 1,324 re-checked values could not be reproduced. Availability is not reproducibility.
• Szollosi, A., Kellen, D., Navarro, D. J., et al. (2020). Is preregistration worthwhile? doi:10.1016/j.tics.2019.11.009 Argues that the diagnosticity of a test depends on how well the model maps onto the theory, which registration does not improve, so the claimed benefit fails.
• Allen, C. and Mehler, D. M. A. (2019). Open science challenges, benefits and tips in early career and beyond. doi:10.1371/journal.pbio.3000246 Sets out three benefits and three challenges for early-career researchers, finding that null findings are more likely to be published as registered reports while the practices cost flexibility and time.
• Chakravorti, T., Gautam, S. and Rajtmajer, S. (2025). Open science practices among early-career human-computer interaction researchers in the US. doi:10.1371/journal.pone.0334692 Interviewed 18 early-career HCI researchers with mixed-methods or qualitative backgrounds: adoption was shaped by missing incentives, cultural resistance and intellectual-property concern.
• Nosek, B. A., Beck, E. D., Campbell, L., et al. (2019). Preregistration is hard, and worthwhile. doi:10.1016/j.tics.2019.07.009 Argues that preregistration clarifies planned and unplanned research by reducing unnoticed flexibility, and that doing it well takes practice.
• Vines, T. H., Albert, A. Y. K., Andrew, R. L., et al. (2014). The availability of research data declines rapidly with article age. doi:10.1016/j.cub.2013.11.014 Requested datasets from 516 articles two to twenty-two years old and found the odds of a dataset still existing fell by about 17 per cent per year and of a working author email by about 7 per cent.
• Munafo, M. R. and Davey Smith, G. (2018). Robust research needs many lines of evidence. doi:10.1038/d41586-018-01023-3 Argues that robustness is not a property of any single study or remedy and that knowledge becomes dependable only when many independent lines of evidence converge.
• Wicherts, J. M., Bakker, M. and Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE. doi:10.1371/journal.pone.0026828 Asked authors for the data behind 1,148 significant results in 49 psychology papers and found reluctance to share greatest where the evidence was weakest and the reporting carried more errors.
• Scheel, A. M., Schijen, M. R. M. J. and Lakens, D. (2021). An excess of positive results: comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science. doi:10.1177/25152459211007467 Compared the first hypothesis of 71 published registered reports with a random sample of 152 standard studies and found 44 per cent positive results in the registered reports against 96 per cent in the standard literature.