Digital health interventions and the evaluation gap: an evidence base that outruns its methods

Digital health is this module's outlier: the interventions ship faster than the studies that would judge them. The claim here is narrower than the claim that digital health does not work. It is that a study of a digital health intervention can support only the claim its design was built to test. Formative usability evidence and summative trial evidence therefore answer different questions.

This tenth of ten units turns to the newest interventions, and asks whether the evidence base can keep pace with them.

Two kinds of evidence, and only one of them is a trial

Blandford and colleagues set the two traditions side by side (Blandford et al., 2018).

Murray and colleagues list the questions an evaluation must answer, including benefit, reach and uptake, the causal model, the components, and effectiveness, cost effectiveness and harms (Murray et al., 2016). Trials remain important, they argue, but are best run only when the intervention and its delivery package are stable, fidelity is high, and benefit is plausible. A usability study on a prototype is not a weak trial but a different measurement.

Writing on 26 March 2020, Torous and colleagues record that the United States government waived a set of telehealth rules on 17 March 2020 (Torous et al., 2020). They describe the change as unprecedented and as possible because of a pre-existing evidence base for telehealth. The waiver changed what could be delivered within weeks, and it carried nothing about the services being stood up. The authors name workforce training, high-quality evidence and digital equity as the outstanding needs. The piece is an editorial that the journal flags as not peer reviewed, and it reports no outcome, so it dates a deployment rather than judging it.

An intervention literature that grew faster than its methods

Moller and colleagues reviewed active video game interventions requiring gross movement and at least two conditions with ten or more participants (Moller et al., 2023). They found 232 studies published between 1996 and 2020 enrolling 14,849 participants. 69.8 per cent targeted physical fitness, and 72.0 per cent of the games were built by industry for the commercial market, against 13.3 per cent funded by grant.

Guo and colleagues describe the mismatch (Guo et al., 2020): methodologies for timely, cost effective and robust evaluation have not kept pace with the number of applications, and conventional methodologies such as clinical trials have seldom been applied.

Engagement is the hinge, and real-world engagement is small

If an intervention works only when it is used, use is not a secondary outcome. Perski and colleagues synthesised 117 articles and found engagement conceptualised in both experiential and behavioural terms (Perski et al., 2016). Yardley and colleagues argue that establishing effective engagement, engagement with the behaviour targeted, matters more than more engagement (Yardley et al., 2016).

Baumel and colleagues analysed independent panel data for 93 Android mental health apps with ten thousand or more installs (Baumel et al., 2019). They found a median daily active user rate of 4.0 per cent, and median retention of 3.9 per cent at 15 days and 3.3 per cent at 30 days. Fleming and colleagues found completion or sustained use ranging from 0.5 per cent to 28.6 per cent across seven publicly available self-help interventions (Fleming et al., 2018). A trial that recruits willing participants measures an effect the install base does not reproduce. The install base is also uneven, since an app needs a data plan and a charged phone. A blind patient or a reader with low literacy is not the user it was built for.

What the evaluations actually measure

Enam and colleagues asked how eHealth interventions are evaluated in practice (Enam et al., 2018). Evaluation is performed at the end of the intervention, and no case study conducted it throughout. Conceptual papers list clinical, human and social, organisational, technological, cost and other aspects, while the case studies assessed only the clinical and the human and social ones. The frameworks describe more than the studies measure.

Mohr and colleagues accept the trial record and then limit it (Mohr et al., 2017). Randomised controlled trials have demonstrated efficacy. Yet there is little evidence that these tools can be implemented in clinical settings, a state of affairs the authors attribute to three widely held misconceptions. Showing that an intervention can work is not showing that a service can use it.

From a working tool to a working service

Greenhalgh and colleagues built the NASSS framework from a hermeneutic systematic review and six case studies (Greenhalgh et al., 2017). Its seven domains classify challenges as simple, complicated or complex, and complexity in multiple domains rarely led to mainstream adoption. The patient handed the app at discharge is often an older person with no help setting it up. Marwaha and colleagues reach the same place from the buyer's side, proposing nine dimensions to examine before adoption (Marwaha et al., 2022). They note that few strategies help tools cross the chasm from clinical validation to integration in large systems' workflows. A validated tool therefore has a second evaluation problem, and it is a systems question rather than a usability question.

The regulator's answer: evidence by tier

The National Institute for Health and Care Excellence has converted the question into a tiered obligation: its evidence standards framework assigns a technology to a tier and scales the evidence requirement to that tier. The 2022 update added artificial intelligence and data-driven technologies with adaptive algorithms, and NICE states that meeting the standard means neither endorsement nor regulatory approval. The framework replaces the general question with a conditional one: which tier, and therefore which evidence, for this product.

The contested question: a shortage of trials or a mismatch of methods?

The first position holds that the trials exist and are misapplied. Murray and colleagues give the conditions a trial needs. That implies the problem is trials run on unstable products or with insufficient fidelity, not that trials are the wrong instrument (Murray et al., 2016). Mohr and colleagues locate the obstacle in beliefs about what a trial should test (Mohr et al., 2017).

The second position holds that the method itself is behind. Guo and colleagues argue that conventional methodologies have seldom been applied and that more pragmatic evidence generation is needed. On that reading the deficit is in the repertoire rather than in its use (Guo et al., 2020). The real-world engagement data support this. Sustained use, the outcome a service depends on, is not what a conventional efficacy trial estimates (Baumel et al., 2019; Fleming et al., 2018).

The third position is the strongest counter-argument to this unit's thesis. Blandford and colleagues claim no absence of evidence. They claim two mature research cultures with different notions of acceptable evidence and different reporting habits, and they offer seven lessons for making the collaboration work (Blandford et al., 2018). If that is right, the thesis overstates the case. What is missing is a shared standard for what counts and how it is reported, so the gap is a coordination failure rather than an evidential one.

What would settle it is a prospective study of one product across its lifecycle. It would pre-specify objective engagement outcomes of the kind Baumel and colleagues measured, and include a summative trial meeting the conditions Murray and colleagues set. It would state the product's NICE tier in advance, and report the formative work beside the result. No paper in this unit reports such a study, and Enam and colleagues found evaluation is conducted at the end rather than throughout (Enam et al., 2018).

What to do while the dispute is open

The evidence on the two sides is unequal. That the method lags rests on a call for new approaches and on real-world engagement data. That the trials exist and are misapplied rests on a methodological argument. The strongest objection to this unit's thesis is that two mature research cultures already produce different evidence, so the gap may be one of reporting and standard rather than of method. While the dispute is open, a designer should state the product's tier and engagement outcome in advance, report the formative work beside the summative result, and treat sustained use as the outcome that matters.

Where this is heading

Michie and colleagues describe a Knowledge System that uses machine learning to extract, synthesise and interpret behaviour change findings, so that evidence is timelier and prediction of what works for whom improves (Michie et al., 2017). Greenhalgh and colleagues offer NASSS for prospective use, to identify technologies with, despite enthusiasm, a limited chance of large-scale adoption (Greenhalgh et al., 2017). The NICE tiers push the evidence obligation upstream by risk (NICE, 2022).

Sources

• Baumel, A. et al. (2019). Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. Journal of Medical Internet Research. doi:10.2196/14567 Used independent panel data, not developer reports, for 93 Android mental health apps with ten thousand or more installs. Median daily active users were 4.0 per cent and median retention was 3.9 per cent at 15 days and 3.3 per cent at 30 days.
• Blandford, A. et al. (2018). Seven lessons for interdisciplinary research on interactive digital health interventions. DIGITAL HEALTH. doi:10.1177/2055207618770325 Sets out seven lessons from the research lifecycle for interdisciplinary work on these interventions. Its central contrast is between formative evaluation, emphasised in Human-Computer Interaction, and summative evaluation, emphasised in health research, with the differences extending to accepted evidence and reporting.
• Enam, A., Torres-Bonilla, J. and Eriksson, H. (2018). Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review. Journal of Medical Internet Research. doi:10.2196/10971 Reviewed how eHealth interventions are actually evaluated, covering conceptual papers and case studies. Evaluation was performed at the end of the intervention and no case study did it throughout, and case studies assessed only the clinical and the human and social aspects while conceptual papers list more.
• Fleming, T. et al. (2018). Beyond the Trial: Systematic Review of Real-World Uptake and Engagement With Digital Self-Help Interventions. Journal of Medical Internet Research. doi:10.2196/jmir.9275 Reviewed implemented, unguided digital self-help interventions, covering seven publicly available ones. Completion or sustained use ranged from 0.5 per cent to 28.6 per cent, and the measures reported were inconsistent.
• Greenhalgh, T. et al. (2017). Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. Journal of Medical Internet Research. doi:10.2196/jmir.8775 Built the NASSS framework from a hermeneutic systematic review and six case studies studied for up to three years across more than 20 organisations. Complexity in multiple domains rarely led to mainstream adoption.
• Guo, C. et al. (2020). Challenges for the evaluation of digital health solutions: a call for innovative evidence generation approaches. npj Digital Medicine. doi:10.1038/s41746-020-00314-2 Argues that methodologies for timely, cost effective and robust evaluation have not kept pace with the number of applications. It reports that clinical trials have seldom been applied and calls for pragmatic approaches including a clinical simulation test bed.
• Marwaha, J. S. et al. (2022). Deploying digital health tools within large, complex health systems. npj Digital Medicine. doi:10.1038/s41746-022-00557-1 Proposes nine dimensions for a health system to examine before adopting a clinically validated tool. It notes that few strategies exist to help tools cross the chasm from clinical validation to integration in a large system's workflows.
• Michie, S. et al. (2017). The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implementation Science. doi:10.1186/s13012-017-0641-5 Describes a project to build a Knowledge System that uses machine learning to extract, synthesise and interpret behaviour change findings. Its purpose is to make evidence timelier and improve prediction of what works for whom.
• Mohr, D. C. et al. (2017). Three Problems With Current Digital Mental Health Research. Psychiatric Services. doi:10.1176/appi.ps.201600541 Accepts that a large body of randomised trials has demonstrated efficacy for mental health technologies, and argues there is little evidence the tools can be implemented in clinical settings. It names three widely held misconceptions it holds responsible.
• Moller, A. C. et al. (2023). A comprehensive systematic review and content analysis of active video game intervention research. DIGITAL HEALTH. doi:10.1177/20552076231171232 Reviewed active video game interventions meeting strict inclusion criteria, identifying 232 studies published between 1996 and 2020 enrolling 14,849 participants. It found 69.8 per cent targeted physical fitness and 72.0 per cent of the games were developed by industry for the commercial market, against 13.3 per cent grant funded.
• Murray, E. et al. (2016). Evaluating Digital Health Interventions: Key Questions and Approaches. American Journal of Preventive Medicine. doi:10.1016/j.amepre.2016.06.008 Sets out the questions an evaluation must answer, including benefit, reach and uptake, the causal model, the components, and effectiveness, cost effectiveness and harms. It argues trials are best undertaken only when the intervention and its delivery package are stable, fidelity is high, and benefit is plausible.
• National Institute for Health and Care Excellence (2022). Evidence standards framework for digital health technologies. NICE. https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies Assigns a digital health technology to a tier and scales the evidence requirement to that tier. The 2022 update added artificial intelligence and data-driven technologies with adaptive algorithms, and NICE states that meeting the standard is neither endorsement nor regulatory approval.
• Perski, O. et al. (2016). Conceptualising engagement with digital behaviour change interventions: a systematic review. Translational Behavioural Medicine. doi:10.1007/s13142-016-0453-1 A synthesis of 117 articles which finds engagement conceptualised in both experiential and behavioural terms. It proposes a framework in which engagement is influenced by the intervention, the context and population, and the behaviour targeted.
• Yardley, L. et al. (2016). Understanding and promoting effective engagement with digital behaviour change interventions. American Journal of Preventive Medicine. doi:10.1016/j.amepre.2016.06.015 An expert consensus paper on how engagement is understood and promoted. Its conclusion is that it is more valuable to establish effective engagement, engagement with the behaviour targeted, than simply more engagement.
• Torous, J., Myrick, K. J., Rauseo-Ricupero, N. and Firth, J. (2020). Digital Mental Health and COVID-19: Using Technology Today to Accelerate the Curve on Access and Quality Tomorrow. JMIR Mental Health. doi:10.2196/18848 An editorial recording the waiving of United States telehealth rules on 17 March 2020, attributed to a pre-existing evidence base rather than to evidence about the services deployed. It names the outstanding needs as workforce training, high-quality evidence and digital equity.