What an evaluation owes: the trial standard and its critics

Every deployment asks to be judged, and the field has no agreed standard for the judgement. The inherited answer comes from clinical research: show that an intervention can work under controlled conditions, then that it works in ordinary ones, then disseminate it.

This unit argues that the sequence does not transfer to deployments. This fifth of ten units asks what such a study is owed: the four before it asked what a deployment, a probe, a record of use and a departure can show. It has not settled whether an evaluation owes a measured change in behaviour or a deepened account of use, and the two answers imply different instruments and conclusions.

The sequence was borrowed, and its authors doubted it

The staged logic is not a natural order of knowledge but a claim about how a result travels, and the researchers who built the vocabulary doubted it early. Glasgow, Vogt and Boles (1999) proposed RE-AIM, five dimensions on which a public health intervention should be judged: reach, efficacy, adoption, implementation and maintenance. These hold at the individual, clinic and community levels and combine to determine population impact.

Glasgow, Lichtenstein and Marcus (2003) named the assumption directly. They argue that the gap between research and practice rests on a single assumption: that effectiveness follows naturally and logically from successful efficacy research, because the two traditions evolved different methods and values. A successful efficacy intervention differs inherently from a successful effectiveness one, and the factors that limit robustness across settings, populations and staff belong in both. They ask for reach, adoption, implementation and maintenance to be documented.

A deployment is a setting where the factors that decide whether a system works are the factors a controlled study holds constant. Who adopts the system, and whether anyone maintains it, are examples.

Two traditions do not share a standard, and the pipeline between them runs one way

If efficacy and effectiveness differ in values, the argument about what an evaluation owes is not one about rigour. Blandford and colleagues (2018) draw seven lessons from interdisciplinary digital health research and put the friction at the nature of accepted evidence and the culture of reporting: the contrast they discuss most is between formative evaluation, emphasised in HCI, and summative evaluation, emphasised in health research, but they say the differences run well beyond it. Blandford (2019) frames the same problem as paradoxes, setting substantial investment in innovative health technologies beside deployed systems that are difficult to use, and naming threats to privacy, patient trust and experience.

Green (2008) gives the sharpest account of what the trial removes. Looking upstream from the research-practice gap, he identifies two fallacies: the fallacy of the pipeline, implicit in one-way models of translation, and the fallacy of the empty vessel, implicit in treating the practitioner as a recipient of guidelines. Vetting research for reviews can render it irrelevant to practice, and his remedies favour practice-based production of research and more attention to external validity.

If an evaluation owes a measured change: designs that survive a moving system

The first answer is that an evaluation owes a measured effect, and its defenders concede that the standard trial cannot gauge a system that changes. Mohr and colleagues (2013) make the timing argument: trials and implementation studies can require years to validate an intervention, a timeline they call fundamentally incompatible with technologies that change quickly. Their proposal, continuous evaluation of evolving behavioural intervention technologies, judges several versions at once, drops the poorer performers and admits new ones at any time. Mohr and colleagues (2015) make the object argument: locking an intervention down during evaluation locks in its defects and validates tools that are outdated when the results appear, so a trial should evaluate the intervention principle.

Collins, Nahum-Shani and Almirall (2014) supply the design for interventions that are sequences rather than doses. Within the multiphase optimisation strategy, the sequential, multiple assignment, randomised trial gathers the information needed to optimise a dynamic treatment regimen. In such a regimen the type or intensity of treatment varies with the participant's needs and progress. Hekler and colleagues (2016) turn the process into a knowledge base. Agile science targets the smallest meaningful behaviour change modules, computational models of the interaction between modules, individuals and context, and personalisation algorithms as decision rules for adaptation.

The condition for a trial still holds, and the first pooled estimate is small

Murray and colleagues (2016) set the condition under which the behaviour-change standard applies. A randomised trial is best undertaken only when the intervention and its delivery package are stable. Staff can deliver the package with high fidelity. The benefits must be likely to be clinically meaningful. Read as a rule, a deployment in flux should not be judged by a frozen comparison, and a stable one has no excuse for avoiding one.

A first pooled estimate of those designs now exists, and it is small. Von Lutzow, Neuendorf and Scherr meta-analysed 23 studies of just-in-time adaptive and ecological momentary interventions, with 2563 participants. The between-group effect was g = 0.15, significant only against alternative interventions, and shorter interventions held their effect longer (von Lutzow et al., 2025).

If an evaluation owes an account of use

The second answer refuses the obvious test of whether a technology brought about the intended change in behaviour. Klasnja, Consolvo and Pratt (2011) argue that this answer is too limited: demonstrating behaviour change is often infeasible, and it is unnecessary for a meaningful contribution to HCI. They propose efficacy evaluations matched to the behaviour-change strategies inside the system, such as self-monitoring, alongside studies of experience, and one such measure assumes a participant who needs no screen reader. Blandford and colleagues (2018) show why the two answers are hard to reconcile: the traditions differ over whether an account of use counts as a result at all.

Contested question: what does an evaluation owe?

Three positions are on the table. The first holds that an evaluation owes a measured change in behaviour, with the design adapted to a moving system. Its instruments are RE-AIM and its descendants (Glasgow, Vogt and Boles, 1999; Glasgow, Lichtenstein and Marcus, 2003), continuous and adaptive designs (Mohr et al., 2013, 2015; Collins et al., 2014) and a conditional rule for when a trial is warranted (Murray et al., 2016). The second holds that an evaluation owes a deepened account of use, matched to the intervention strategy inside the system, in which behaviour change is often infeasible and unnecessary (Klasnja, Consolvo and Pratt, 2011), and which sits against two traditions that disagree about what evidence is (Blandford et al., 2018; Blandford, 2019).

The third holds that the argument about what an evaluation owes is itself the symptom. Green (2008) locates the problem in a one-way pipeline that renders research irrelevant before it reaches practice. Hall, Gomez Bergin and Rennick-Egglestone (2024) draw eight critical challenges with ethical or legal consequences from their experience of evaluating digital mental health interventions: harm caused by online recruitment, monitoring of intervention safety, exclusion of specific demographic or clinical groups, structural barriers to implementation, and regulatory ambiguity about interventions that are medical devices. They report at least one research programme delayed by more than a year when its intervention was unexpectedly classified as a medical device, though only 2 of 23 interventions in their review of trials carried that classification, and the delay is reported from their own experience rather than as a measured rate. Several of those obligations hold whichever position on measurement an evaluation takes.

The measurement position rests on a framework and its critics rather than a head-to-head comparison (Glasgow, Vogt and Boles, 1999; Green, 2008); the account-of-use position is an argument from theory rather than measurement (Klasnja, Consolvo and Pratt, 2011); the third rests on one programme's experience (Hall et al., 2024). What would settle it is a comparison, not another argument: run both kinds of evaluation on one deployment, and report whether the account of use explains a null result. While the dispute is open, state which obligation the evaluation is meeting and hold it to its own standard. The strongest objection is that the two standards cannot run on one deployment, because measuring a change fixes the conditions the account of use describes.

Where the area is heading

The direction is towards designs that merge evaluation into the deployment. Adaptive trial designs (Collins et al., 2014) and continuous evaluation (Mohr et al., 2013) treat the running system as the instrument. Trials of intervention principles move the object of evaluation from a frozen product to a principle that survives change (Mohr et al., 2015). Agile science makes the deployment itself the knowledge-producing process (Hekler et al., 2016). Under each design the system still changes while it is judged. What is held fixed therefore moves from the technology to the principle. The account of use that Klasnja, Consolvo and Pratt (2011) asked for is still not measured. The obligation Hall and colleagues (2024) identify, to face the ethical and legal consequences of how deployments are run, is the part no design choice removes.

The sources this unit rests on

Each note states what the cited document did and found. Titles are in British spelling, and identifiers were checked against registered metadata on 2 November 2026.

• Glasgow, R. E., Vogt, T. M. and Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. doi:10.2105/AJPH.89.9.1322 Proposes RE-AIM, assessing reach, efficacy, adoption, implementation and maintenance, and argues that the five interact to determine a programme's population impact. Failing to measure all five, the authors say, wastes resources and creates discontinuities between stages of research.
• Glasgow, R. E., Lichtenstein, E. and Marcus, A. C. (2003). Why don't we see more translation of health promotion research to practice? doi:10.2105/AJPH.93.8.1261 Challenges the assumption that effectiveness research naturally follows from successful efficacy research, because the two traditions evolved different methods and values. It asks for reach, adoption, implementation and maintenance to be documented rather than assuming that one stage follows from the other.
• Klasnja, P., Consolvo, S. and Pratt, W. (2011). How to evaluate technologies for health behaviour change in HCI research. doi:10.1145/1978942.1979396 Argues that the obvious standard, whether the technology changed behaviour, is too limited, because demonstrating behaviour change is often infeasible and unnecessary for a meaningful HCI contribution. It proposes efficacy evaluations matched to the behaviour-change strategies inside the system, alongside studies of experience.
• Blandford, A. et al. (2018). Seven lessons for interdisciplinary research on interactive digital health interventions. doi:10.1177/2055207618770325 Distils seven lessons from interdisciplinary digital health research, locating the friction between HCI and health research in the nature of accepted evidence and the culture of reporting. Formative and summative evaluation are the best-known contrast, but the differences extend well beyond it.
• Blandford, A. (2019). HCI for health and wellbeing: challenges and opportunities. doi:10.1016/j.ijhcs.2019.06.007 Frames health technology as paradoxes: substantial investment in innovative technologies alongside deployed systems that are difficult to use and few that achieve significant market penetration. It presents a space of interactive health technologies, users and uses, and names threats to privacy, patient trust and experience.
• Green, L. W. (2008). Making research relevant: if it is an evidence-based practice, where's the practice-based evidence? doi:10.1093/fampra/cmn055 Identifies a fallacy of the pipeline, implicit in one-way models of translation, and a fallacy of the empty vessel, implicit in treating the practitioner as a recipient of guidelines. It argues that vetting research for reviews and guidelines can make it irrelevant to practice, and favours practice-based production of research.
• Mohr, D. C. et al. (2013). Continuous evaluation of evolving behavioural intervention technologies. doi:10.1016/j.amepre.2013.06.006 Argues that randomised trials can take years to validate an intervention, a timeline it calls fundamentally incompatible with technologies that change quickly. It proposes continuous evaluation of evolving versions, which judges several at once, removes poorer performers and admits new ones at any time.
• Mohr, D. C. et al. (2015). Trials of intervention principles: evaluation methods for evolving behavioural intervention technologies. doi:10.2196/jmir.4391 Argues that locking an intervention down during evaluation locks in its defects and validates tools that are outdated when the results appear. It proposes trials of intervention principles, in which the principle stays fixed while the technology is improved, moving the object of evaluation from a product to a principle.
• Collins, L. M., Nahum-Shani, I. and Almirall, D. (2014). Optimisation of behavioural dynamic treatment regimens based on the sequential, multiple assignment, randomised trial (SMART). doi:10.1177/1740774514536795 Sets out how the multiphase optimisation strategy develops a dynamic treatment regimen, in which the type or intensity of treatment varies with the participant's progress. The sequential multiple assignment randomised trial gathers the information needed to optimise such a regimen.
• Hekler, E. B. et al. (2016). Agile science: creating useful products for behaviour change in the real world. doi:10.1007/s13142-016-0395-7 Proposes agile science, with three products: the smallest meaningful behaviour change modules, computational models of the interaction between modules, individuals and context, and personalisation algorithms as decision rules for adaptation. Development moves through a generation phase and an evaluation phase, making the deployment the process that builds knowledge.
• Murray, E. et al. (2016). Evaluating digital health interventions: key questions and approaches. doi:10.1016/j.amepre.2016.06.008 Sets out the questions an evaluation should answer and the methods that fit each, arguing that methods from biomedical, behavioural and computing research are all required. It holds that a randomised trial is best undertaken only when the intervention is stable, implementable with high fidelity and likely to deliver meaningful benefit.
• Von Lutzow, U., Neuendorf, N. L. and Scherr, S. (2025). Effectiveness of just-in-time adaptive interventions for improving mental health and psychological well-being: a systematic review and meta-analysis. BMJ Mental Health. doi:10.1136/bmjment-2025-301641 Meta-analysed 23 studies of just-in-time adaptive and ecological momentary interventions, 2563 participants. It reports a small between-group effect of g = 0.15, significant only against alternative interventions, with effects lasting longer under six weeks.
• Hall, C. L., Gomez Bergin, A. D. and Rennick-Egglestone, S. (2024). Research into digital health intervention for mental health: 25-year retrospective on the ethical and legal challenges. doi:10.2196/58939 Identifies eight critical challenges with ethical or legal consequences from the authors' own experience of designing and evaluating digital mental health interventions. These include harm from online recruitment, monitoring of safety, exclusion of demographic or clinical groups, structural barriers to implementation, and regulatory ambiguity about medical devices.