Reproducibility packages, artefacts and open materials

A project leaves more than a paper. It leaves the analysis scripts, the instrument, the codebook and the raw logs, and those are the part of the work a reader can use.

The ninth of ten units calls that set the reproducibility package, a digital object created for the study or generated by it (ACM, 2020), and argues that the package is the part of the argument a reader can check. The tenth asks what a defence adds to it. What sets its worth is what actually runs rather than what is nominally shared. A paper can report that data and code are available and still leave a reader unable to regenerate a number. The package is therefore a testable claim: built and executed before submission, with what cannot be shared named.

The three badges measure three different things

The FAIR principles ask that data be findable, accessible, interoperable and reusable, putting the weight on machine-actionable reuse, reached through whatever repository and bandwidth a reader has (Wilkinson et al., 2016). The ACM badging policy separates three measurements. Repeatability is the same team repeating its own computation. Reproducibility is a different team obtaining the same result from the author's artefacts. Replicability is a different team doing so from independently built artefacts. The policy attaches three badges, Artifacts Available, Artifacts Evaluated and Results Validated (ACM, 2020).

Availability is a property of the deposit. The evaluated badge adds an audit, and validated results belong to a later study. A project reaches the first cheaply and the third rarely. The work it controls is to exercise the package and verify that its reported numbers can be regenerated from it. The middle badge is the one with a procedure behind it. A committee downloads the package, builds it and exercises it, and awards the badge only when the material is documented, consistent, complete and exercisable from a modest machine on an ordinary connection.

What is shared is not what runs

Collberg and Proebsting (2016) started from 601 papers in eight ACM conferences and five journals, identified the 402 whose results were backed by code, and tried to build it. They built 32.3 per cent within thirty minutes and 48.3 per cent with extra effort, and they either built 54.0 per cent or confirmed with the authors. A claim of availability is therefore a claim about a deposit, not about a working build.

Stodden, Seiler and Ma (2018) tested a policy directly. They requested the materials for 204 computational articles published in Science under its data and code policy, then attempted replications. They obtained artefacts for 44 per cent and reproduced the findings for 26 per cent. They conclude that remission on request beats no policy and remains insufficient.

Ioannidis and colleagues (2009) reached the same place from another field. Two teams each reproduced one table or figure from eighteen microarray analyses. They managed two in principle, six partly and ten not at all, with data unavailability the main reason. Trisovic and colleagues (2022) give the largest recent version. They executed over 9,000 R files from more than 2,000 Harvard Dataverse replication datasets. They found 74 per cent failed to complete without error, and 56 per cent still failed after automatic cleaning. They locate many failures in missing dependencies rather than untidy logic. Their framing is the one this unit adopts: re-execution is necessary but not sufficient for reproducibility.

An incentive changes what is reported, and not always what is usable

Kidwell and colleagues (2016) studied the badges introduced by Psychological Science in 2014. Reported open data rose from under 3 per cent before the badges to 23 per cent after, and 39 per cent in early 2015, while comparison journals changed little. Their qualification is the important one: reporting openness does not guarantee openness, though badged data were likelier to be actually available and complete. A policy is not a package and is not nothing. Trisovic and colleagues (2022) find policy strictness associated with the re-execution rate of deposited code.

Reproducibility names several different standards

Gentleman and Temple Lang (2007) set the primary standard, an analysis integrated with the document that describes it so that a reader can regenerate its results. Peng (2011) argues that computational results should be published with the code and data that produce them, and he makes reproducibility a minimum standard, distinct from repeatability. Vandewalle, Kovacevic and Vetterli (2009) show a field that built the standard into its publishing. There, reproducible research means an independent reader can reproduce the results from the released text, data and code. Eglen and colleagues (2017) supply the practical level: licensing, documentation and archiving are what make a package usable.

Some material cannot be shared, and the package must say so

Fecher and Friesike (2014) sort the literature into five schools of thought, the infrastructure, public, measurement, democratic and pragmatic schools. A claim to openness should say which programme it serves. The hard limit is material that ethics will not let travel. Bietz and colleagues (2015) report the consent, privacy and infrastructure obstacles to sharing personal health data. They argue that some data cannot be shared openly and must be planned for. The honest package for a study of people therefore holds the instrument, the codebook, the analysis script and the derived data. The paper states what is withheld, and a participant's terms of consent set that boundary.

The contested question: a universal standard or a field-specific practice?

The universalist position treats reproducibility as an obligation that holds wherever empirical claims are made. Peng (2011) calls it a minimum standard, and Vandewalle and colleagues (2009) want the field to adopt it as its practice. Wilkinson and colleagues (2016) propose principles meant to apply to any shared object. The ACM (2020) applies one set of definitions across its venues, conceding that review remains optional.

The field-specific position holds that sharing practice is coupled to how a community produces knowledge, so one standard fits some fields better than others. Velden and Xharavina (2026) draw on a large survey of researchers at German universities. They examine nineteen subfields and find that public sharing and reuse track field-specific epistemic practices, especially the accessibility of the empirical object. They recommend that policy account for these differences at a granular level. Fecher and Friesike (2014) support this conceptually, since openness names five programmes with different aims. Bietz and colleagues (2015) give the limiting case of material that cannot be shared at all.

What would settle the dispute is evidence neither side has. One study would hold the standard fixed and vary the field, and show that a single standard enables independent reproduction across fields. Another would show that a field-adapted standard produces more reproduction than a general standard. The measurements here cannot adjudicate, because each is confined to a single field. A related disagreement is whether reporting equals sharing, since Kidwell and colleagues (2016) show an incentive raises reported sharing while still not guaranteeing openness.

Read by kind, the universalist side rests on principles, on a set of definitions and on a stated minimum standard, while the field-specific side rests on a survey across nineteen subfields. Every measurement that bears on the question is confined to one field, whether six hundred papers in computer systems, eighteen microarray analyses or nine thousand R files. While the dispute is open, a project should build its package on a clean machine, regenerate every reported number from it, and state what is withheld and why. The strongest objection to this unit's thesis is that a package tests only whether a number can be produced again, so a study may be fully reproducible and still have measured the wrong thing.

What a project can do before the deadline

Three checks follow. Build the package on a clean machine, as Collberg and Proebsting (2016) did. Regenerate every reported number from the deposited materials, the line Stodden and colleagues (2018) measured at 44 and 26 per cent. State what is withheld, since Bietz and colleagues (2015) show that some material cannot travel.

Where this is heading

The direction of travel is away from the binary of shared or not shared and towards graded, tested packages. Trisovic and colleagues (2022) end with recommendations for researchers, journals and repositories together, a problem now understood as infrastructural. Velden and Xharavina (2026) would design open science policy at the granularity of the field. Cordeiro and Oliveira Jr (2025) review 25 primary studies in software engineering. They find reproducibility defined variously as a method, a repeatability, a probability and an ability, a still-consolidating practice. The ACM (2020) already allows the availability and validation badges to be awarded after publication, turning the package into a living object. Whether graded review makes packages more usable or merely more numerous remains open.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers were checked against Crossref or the archived copy on 31 October 2026.

• Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M. et al. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. doi:10.1038/sdata.2016.18 Sets out the FAIR principles, that data be findable, accessible, interoperable and reusable, with the emphasis on machine-actionable reuse. Defines the reusable shared object the unit asks a project to build.
• ACM (2020). Artifact Review and Badging, Version 1.1. Association for Computing Machinery. https://www.acm.org/publications/policies/artifact-review-and-badging-current Defines repeatability, reproducibility and replicability as distinct measurements and proposes three badges, Artifacts Available, Artifacts Evaluated and Results Validated, the evaluated badge requiring a documented, consistent, complete and exercisable package. Gives the unit its vocabulary.
• Collberg, C. and Proebsting, T. A. (2016). Repeatability in computer systems research. Communications of the ACM, 59(3), 62-69. doi:10.1145/2812803 Identified the 402 of 601 sampled papers whose results were backed by code and tried to build it, succeeding within thirty minutes for 32.3 per cent and with extra effort for 48.3 per cent. Availability is not buildability.
• Stodden, V., Seiler, J. and Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584-2589. doi:10.1073/pnas.1708290115 Requested materials for 204 Science articles published under a sharing policy and attempted replications, obtaining artefacts for 44 per cent and reproducing findings for 26 per cent. Concludes that remission on request is insufficient.
• Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S. et al. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14(5), Article e1002456. doi:10.1371/journal.pbio.1002456 Found reported open data in Psychological Science rising from under 3 per cent before its 2014 badges to 23 per cent after, while comparison journals changed little. Warns that reporting openness does not guarantee it.
• Trisovic, A., Lau, M. K., Pasquier, T. and Crosas, M. (2022). A large-scale study on research code quality and execution. Scientific Data, 9, Article 60. doi:10.1038/s41597-022-01143-6 Executed over 9,000 R files from more than 2,000 Dataverse replication datasets, finding 74 per cent failed to complete without error and 56 per cent still failed after cleaning. Re-execution is necessary but not sufficient.
• Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X. et al. (2009). Repeatability of published microarray gene expression analyses. Nature Genetics, 41(2), 149-155. doi:10.1038/ng.295 Had two teams each reproduce one table or figure from eighteen microarray analyses, reproducing two, six partly and ten not at all, with unavailable data the main reason. Shows reproduction failing on the materials.
• Gentleman, R. and Temple Lang, D. (2007). Statistical analyses and reproducible research. Journal of Computational and Graphical Statistics, 16(1), 1-23. doi:10.1198/106186007X178663 Sets out the reproducible-research framework: text, code and data integrated into a dynamic document that regenerates its results. It is the primary account of a reproducible analysis artefact.
• Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226-1227. doi:10.1126/science.1213847 Argues that computational results should be published with the code and data that produce them and distinguishes reproducibility from repeatability. The unit's statement of reproducibility as a minimum standard.
• Vandewalle, P., Kovacevic, J. and Vetterli, M. (2009). Reproducible research in signal processing. IEEE Signal Processing Magazine, 26(3), 37-47. doi:10.1109/msp.2009.932122 Introduces reproducible research in signal processing, where an independent researcher can reproduce the results from the released text, data and code, and describes the tools that make it possible. Presents it as a practice the community should adopt rather than an optional extra.
• Eglen, S. J., Marwick, B., Halchenko, Y. O., Hanke, M., Sufi, S. et al. (2017). Toward standard practices for sharing computer code and programs in neuroscience. Nature Neuroscience, 20(6), 770-773. doi:10.1038/nn.4550 Sets out practical standards for sharing research code, including licensing, documentation and archiving, and explains how each makes a package usable by another person. Sharing without them does not achieve reproducibility.
• Fecher, B. and Friesike, S. (2014). Open science: One term, five schools of thought. In S. Bartling and S. Friesike (eds), Opening Science, 17-47. doi:10.1007/978-3-319-00026-8_2 Sorts the open science literature into five schools of thought, infrastructure, public, measurement, democratic and pragmatic, each with different aims and assumptions. A claim to openness should say which programme it serves.
• Bietz, M. J., Bloss, C. S., Calvert, S., Godino, J. G., Gregory, J. et al. (2015). Opportunities and challenges in the use of personal health data for health research. Journal of the American Medical Informatics Association, 23(e1), e42-e48. doi:10.1093/jamia/ocv118 Reports the practical and ethical obstacles to sharing personal health data, including consent, privacy and infrastructure, and argues that some data cannot be shared openly. Names what a package must leave out.
• Velden, T. and Xharavina, V. (2026). The field-specificity of data practices and the public sharing of research data (Part 2). Quantitative Science Studies, 7(1), 1416-1447. doi:10.1162/qss.a.507 Examines nineteen subfields from a large survey of researchers at German universities and finds that public sharing and reuse track field-specific epistemic practices, especially the accessibility of the empirical object. Policy should account for these differences.
• Cordeiro, A. F. R. and Oliveira Jr, E. (2025). Reviewing reproducibility in software engineering research. Proceedings of the 27th International Conference on Enterprise Information Systems (ICEIS 2025), 364-371. doi:10.5220/0013456000003929 A systematic mapping study of 25 primary studies on reproducibility in software engineering, grouping definitions into method, repeatability, probability and ability. Finds the practice still consolidating.