Reporting and replication: a deployment nobody can rebuild

Year 4 / Module 29 In-the-wild and longitudinal deployment studies / Unit 6 / Reporting and replication: a deployment nobody can rebuild

Reporting and replication: a deployment nobody can rebuild

HCI4105 · Unit 6 of 10

Deployments have no reporting standard equivalent to the trial's, and the replication record shows what is lost when a study cannot be rebuilt from its report.

Reporting and replication: a deployment nobody can rebuild

A deployment is expensive, situated and singular, and its report is the only version of it that survives. This unit argues that deployments have no reporting standard equivalent to the trial's. It also argues that the replication record shows what is lost when a study cannot be rebuilt from it. A report that omits the device or the deviation from plan is unusable to anyone but its authors. This sixth of ten units takes the demand the fifth made, that an evaluation state its obligation, and applies it to the document the evaluation leaves behind.

Two literatures meet here. One built a disciplined apparatus for writing a study down. The other measured what happens when it is absent. That apparatus was never written for a setting whose variables are people in their own homes. Neither the checklist nor the replication rate can be borrowed whole.

The trial built an apparatus for writing a study down

The trial standard is a pair of documents, one for the plan and one for the write-up, and both were replaced in 2025. Hopewell, Chan and Collins (2025) set out CONSORT 2025, a 30-item checklist for a parallel-group randomised trial report, adding seven items, revising three and opening a section on open science. It supersedes the 2010 statement, which should no longer be used. Chan, Boutron and Hopewell (2025) supply the prospective half, SPIRIT 2025, a 34-item checklist for the trial protocol, built through a scoping review and a three-round Delphi survey with 317 participants.

The defects the protocol standard was written against are the ones a deployment commits routinely. The SPIRIT 2025 authors cite meta-research finding that many protocols fail to describe the primary outcomes, the allocation method, the measurement of adverse events, and the roles of sponsors and investigators (Chan et al., 2025). A deployment that never states who ran the study, or who could change it, reproduces that failure with no registry to correct the record.

The apparatus exists because a report cannot rebuild a treatment

Glasziou, Meats, Heneghan and Shepperd (2008) assessed 80 consecutive studies of treatment, 55 randomised trials and 25 systematic reviews. Elements of the intervention were missing in 41 of the 80 descriptions. Asking authors raised completeness from 49 to 76 per cent, yet elements remained missing in 19 studies.

Hoffmann and colleagues (2014) turned that measurement into a template. TIDieR is a 12-item extension of CONSORT item 5 and SPIRIT item 11. It asks for the brief name, rationale, materials, procedure, provider, mode, location, timing and dose, tailoring and modification. A report that omits the metered connection, the assistive output or the language participants must read cannot be rebuilt. Drug interventions were described adequately in 67 per cent of cases, against 29 per cent of non-pharmacological ones. Only 39 per cent of 137 interventions were adequate, rising to 59 per cent when authors were asked (Hoffmann et al., 2014).

Without a complete published description of interventions ... other researchers cannot replicate or build on research findings (Hoffmann et al., 2014).

The gap is widest for complex, technology-mediated interventions

McGrath, McHale, Hanson, McLelland and Hamilton (2023) applied TIDieR to technology-enabled cardiac rehabilitation, screening 162 articles to 12 randomised trials covering 1,588 participants. No trial fully reported its intervention and the median score was eight of the 12 items.

Signal and colleagues (2024) tested the generic checklist against a domain directly, through a modified Delphi process with 35 rehabilitation experts. TIDieR-Rehab keeps seven original items, adapts three and adds 12, covering population, dosage, personalisation and harms. A generic checklist shows that reporting is thin; a domain checklist shows which omissions matter, and no equivalent set exists for in-the-wild studies.

What is lost when a study cannot be rebuilt

The Open Science Collaboration (2015) replicated 100 studies from three psychology journals. Of the originals, 97 per cent had significant results. Only 36 per cent of the replications did. The mean replication effect was half the original. Camerer and colleagues (2018) replicated 21 experiments, finding a significant effect in the same direction for 13, or 62 per cent, with effects about half the original.

Prinz, Schlange and Asadullah (2011) found the same outside the social sciences. Across 67 drug target validation projects, published data agreed with their own findings in only 20 to 25 per cent.

One result complicates the story for deployments. Klein and colleagues (2014) tested 13 effects across 36 samples and 6,344 participants, and found ten replicated. Variation in effect size was mostly attributable to the effect rather than the sample or setting. Context is a question a single repeat does not answer, and one a deployment raises but rarely can test.

The same record inside HCI, and why a correct report is not enough

HCI does not escape the pattern, and its measurements matter most here because they are made on the kind of paper a deployment is. Hornbaek, Sander, Bargas-Avila and Grue Simonsen (2014) examined 891 papers across four outlets and found 3 per cent attempting replication. Most rarely analysed effect sizes or compared with the replicated paper. Iarygina, Hornbaek and Mottelson (2026) found that roughly one CHI paper in a hundred shares data and code, and reproduced 49 per cent of those fully.

Better writing cannot repair a weak claim. Ioannidis (2005) argued from statistical first principles that small studies, small effect sizes and flexibility in designs, definitions and outcomes leave a majority of published findings false. A flawlessly reported underpowered study remains underpowered.

Contested question: does a deployment need its own checklist?

Three positions are on the table. The first holds that the deployment needs a standard modelled on the trial's and extended for complexity. A consensus checklist can be built, and description improves once it exists (Hoffmann et al., 2014). A generic checklist under-serves complex interventions, while a domain checklist can be derived (Signal et al., 2024; McGrath et al., 2023).

Psychological Science began awarding badges for open data in January 2014. The share of its articles reporting open data rose from under 3 per cent to 23 per cent. That share reached 39 per cent in the first half of 2015, while comparison journals did not move (Kidwell et al., 2016). Badge-earning data was also more often actually available, correct and complete. What the case does not show is the deployment's problem. A badge raises sharing, and says nothing about whether a shared file lets a reader rebuild a study.

The second holds that the trial's apparatus cannot be borrowed, because what fails to reproduce in a deployment is the setting rather than the write-up. Its evidence is Vanderdonckt and Vatavu (2025), who argue that general frameworks miss the context of use and separate replication across cohorts, platforms, environments and contexts. A checklist should then ask for the setting and the participants' circumstances, not only the device.

The third holds that reporting is necessary and insufficient. Prinz and colleagues (2011), the Open Science Collaboration (2015) and Camerer and colleagues (2018) show large shares of studies failing to return, even when description and materials are available. Ioannidis (2005) explains why.

The first rests on a consensus checklist, domain trials of it and a badge field experiment (Hoffmann et al., 2014; Signal et al., 2024; Kidwell et al., 2016). The second rests on a conceptual framework for context (Vanderdonckt and Vatavu, 2025). The third rests on replication projects in which most results did not return (Open Science Collaboration, 2015). What would settle it is a trial of the checklist: prespecify one, publish deployments under it, and report whether it predicted which studies independent teams could rebuild (McGrath et al., 2023; Iarygina et al., 2026). While the dispute is open, report the setting and the device, and treat no shared file as proof that a study holds. The strongest objection is that a longer report is not a rebuildable one, so a checklist may raise compliance without raising replication.

Where the area is heading

Vanderdonckt and Vatavu (2025) replace a single notion of replication with replication across cohorts, platforms, environments and contexts. Signal and colleagues (2024) show the same movement in health research, where a generic checklist needed 12 new items. Iarygina and colleagues (2026) supply a floor rather than a ceiling, arguing that computational reproducibility is the modest target HCI can reach. Whether context-aware reporting cures the problem or relocates it is open.

The sources this unit rests on

Each note states what the cited document did and found. Titles are in British spelling, and identifiers were checked against registered metadata on 2 November 2026.

Glasziou, P., Meats, E., Heneghan, C. and Shepperd, S. (2008). What is missing from descriptions of treatment in trials and reviews? doi:10.1136/bmj.39590.732037.47 Assessed 80 consecutive studies of treatment, 55 trials and 25 reviews, and found elements of the intervention missing in 41 of the published descriptions. Asking authors raised completeness from 49 to 76 per cent, yet elements remained missing in 19 studies.
Hoffmann, T. C. et al. (2014). Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. doi:10.1136/bmj.g1687 Develops the 12-item TIDieR checklist as an extension of CONSORT item 5 and SPIRIT item 11, and documents that non-pharmacological interventions are described far less adequately than drug interventions. Only 39 per cent of 137 interventions were adequately described, rising to 59 per cent when authors were asked.
Hopewell, S., Chan, A. W. and Collins, G. S. (2025). CONSORT 2025 statement: updated guideline for reporting randomised trials. doi:10.1136/bmj-2024-081123 Sets out the 30-item CONSORT 2025 checklist for parallel-group randomised trial reports, adding seven items, revising three and opening a section on open science. It supersedes CONSORT 2010, which the authors say should no longer be used, and is the trial standard against which the unit measures deployment reporting.
Chan, A. W., Boutron, I. and Hopewell, S. (2025). SPIRIT 2025 statement: updated guideline for protocols of randomised trials. doi:10.1136/bmj-2024-081477 Defines the 34-item SPIRIT 2025 checklist for trial protocols, revised through a scoping review and a three-round Delphi survey with 317 participants. It supersedes SPIRIT 2013 and cites meta-research that protocols often omit the primary outcomes and the roles of sponsors and investigators, giving the unit the prospective half of good reporting.
McGrath, A. et al. (2023). Completeness of intervention reporting in randomised trials of technology-enabled remote or hybrid exercise-based cardiac rehabilitation. doi:10.1080/09638288.2023.2274887 Screened 162 articles to 12 randomised trials covering 1,588 participants, scoring each against the 12-item TIDieR checklist. No trial fully reported its intervention and the median score was eight of 12, so the gap persists for complex, technology-mediated interventions.
Signal, N. et al. (2024). Developing the TIDieR-Rehab checklist: a modified Delphi process to extend TIDieR for rehabilitation intervention reporting. doi:10.1136/bmjopen-2024-084319 A modified Delphi study with 35 rehabilitation experts that extended TIDieR over two rounds, producing TIDieR-Rehab from seven original, three adapted and 12 new items covering population, dosage, personalisation and harms. It shows the unit how a domain reporting standard is derived.
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. doi:10.1126/science.aac4716 Replicated 100 studies from three psychology journals using high-powered designs and the original materials where available. Ninety-seven per cent of the originals had significant results against 36 per cent of the replications, with replication effects half the original, which is the unit's flagship result and its method.
Camerer, C. F. et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. doi:10.1038/s41562-018-0399-z Replicated 21 experiments from two journals with preregistered plans and sample sizes about five times the originals, and found a significant effect in the same direction for 13, or 62 per cent, with effects about half the original. It extends the unit's replication evidence beyond psychology.
Prinz, F., Schlange, T. and Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? doi:10.1038/nrd3439-c1 Compared in-house findings against the published literature across 67 drug target validation projects, finding published data in agreement in only 20 to 25 per cent. About two thirds were affected by inconsistencies, so the problem is not confined to the social sciences.
Klein, R. A. et al. (2014). Investigating variation in replicability: a many labs replication project. doi:10.1027/1864-9335/a000178 Tested 13 effects across 36 independent samples and 6,344 participants and found ten replicated, with variation in effect size mostly attributable to the effect rather than the sample or setting. It carries the unit's point that context is a separate question a single repeat does not settle.
Ioannidis, J. P. A. (2005). Why most published research findings are false. doi:10.1371/journal.pmed.0020124 Argues from statistical first principles that small studies, small effect sizes and flexibility in designs, definitions and outcomes leave a majority of published findings false. It is the unit's reason that better reporting cannot repair an unsupported result.
Hornbaek, K., Sander, S. S., Bargas-Avila, J. A. and Grue Simonsen, J. (2014). Is once enough? On the extent and content of replications in human-computer interaction. doi:10.1145/2556288.2557004 Examined 891 papers across four publication outlets and found 3 per cent attempting replication, mostly treated as a confirm-or-not decision that rarely analysed effect sizes. It shows the reporting defect repeated one level up.
Iarygina, O., Hornbaek, K. and Mottelson, A. (2026). On the computational reproducibility of human-computer interaction. doi:10.1145/3772318.3791129 Identified the CHI papers that shared study data and analysis code over 17 years and attempted to reproduce them. About one in a hundred shares materials permitting a reproduction attempt and 49 per cent of those reproduced fully, giving the unit its HCI-specific reproducibility floor.
Vanderdonckt, J. and Vatavu, R. D. (2025). Context is key for reproducibility of empirical studies in human-computer interaction. doi:10.1145/3736731.3746142 Proposes Ampliatum, a framework separating replication across cohorts, platforms, environments and contexts, demonstrated on gesture-based input. It argues the context of use is rarely stated clearly enough for results to be reproducible, and is the unit's proposal for a context-aware deployment report.
Kidwell, M. C. et al. (2016). Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. doi:10.1371/journal.pbio.1002456 Reports that, when Psychological Science awarded badges for open data from January 2014, the share of its articles reporting open data rose from under 3 per cent to 23 per cent, and to 39 per cent in the first half of 2015, with no change in comparison journals. Badge-earning data was more often actually available and complete.

Check your understanding & apply it · 4 questions, 1 essay

1A deployment paper reports the app version, the outcome and the result, but never says who ran the study and who could change it. Which diagnosis fits this unit?

2A journal introduces a badge for papers that publish their data. On this unit's evidence, what is the likeliest effect?

3Two positions reach a review. One says a deployment needs a checklist modelled on the trial's and extended for complexity. The other says context-aware reporting is what is missing. Which does this unit support?

4A team wants its deployment report to let an independent group attempt a rebuild. Which plan fits this unit's terms?

5Essay A research group has finished an eight-month deployment of a home energy-monitoring system across 24 households and wants its conference paper to let another team rebuild the study and test the result. Using this unit's sources, set out the reporting plan you would require of the group, say where you would draw each item from, and state precisely what the completed report can and cannot guarantee about whether the study will be rebuildable. Then design the evidence that would show whether your reporting plan worked, and identify the one objection to reporting-led reform that your plan cannot answer.

The plan should begin with the two documents the trial standard treats as a pair. For the write-up I would require the 25-item CONSORT 2010 checklist as the structural spine (Schulz, Altman and Moher, 2010), and for the record of what was planned before the deployment I would note that SPIRIT 2013 supplies a 33-item protocol checklist that asks authors to state in advance what they intend, including roles and data access (Chan et al., 2013), which addresses the SPIRIT finding that sponsor and investigator roles were inadequately described in 89 to 100 per cent of protocols. For the intervention itself I would require the 12 TIDieR items (Hoffmann et al., 2014): the brief name, rationale, materials, procedure, provider, mode, location, timing and dose, tailoring, modification, and fidelity against plan. For a home deployment I would add the domain items that the trial checklist omits. Signal and colleagues (2024) show how such additions are derived, since TIDieR-Rehab needed 12 new items to fit rehabilitation, covering the population, dosage and difficulty, personalisation and harms. The deployment equivalent should require the setting in detail, the recruitment route, the version of the device and its software, and every deviation from plan. This is also where Vanderdonckt and Vatavu (2025) bite, because they argue that the context of use is rarely stated clearly enough for reproduction and separate replication across cohorts, platforms, environments and contexts. McGrath and colleagues (2023) give the auditing method, since they scored 12 trials against the 12-item checklist and found a median of eight of 12 with none complete. The report cannot guarantee rebuildability. It can guarantee that the description is complete enough for another team to attempt the deployment, but it cannot make the claim true. Ioannidis (2005) explains why: small samples, small effect sizes and flexibility in analysis leave many published findings false whatever the writing, and a deployment with 24 households and one setting is exposed to all three. Nor can the report guarantee that the result will return, given that the Open Science Collaboration (2015) replicated 100 studies and found 36 per cent significant against 97 per cent originally, that Camerer and colleagues (2018) found 13 of 21 reproducing with effects about half the original, and that Prinz, Schlange and Asadullah (2011) found agreement in only 20 to 25 per cent of 67 drug target projects. The evidence that would show whether the plan worked is a trial of the checklist itself, joining the two measurements the unit already has: score the reports against the checklist as McGrath and colleagues did, then hand the reports and materials to independent teams and ask whether they can rebuild the deployment and reproduce the numbers, as Iarygina and colleagues (2026) did when they found they could fully reproduce 49 per cent of the CHI papers that shared materials. Report whether checklist scores predicted rebuildability. The objection the plan cannot answer is that replication may not transfer because the setting is a variable rather than a nuisance: Klein and colleagues (2014) found that variation in effect size across 36 samples was mostly attributable to the effect under study, and Hornbaek and colleagues (2014) found that HCI replications rarely analysed effect sizes at all, so a rebuildable report may still yield a result that differs for reasons the report cannot capture.