Engagement: the distance between using a system and benefiting from it

A deployment leaves a record: logins, sessions, screens opened, prompts answered and the day each user stopped. It can be counted without asking anybody anything.

This unit argues that the quantity and the benefit are different things. This third of ten units brings the question to the record a deployment leaves: the first two fixed what a deployment can claim and what a probe can show. Sustained engagement is not effective engagement, so the field has moved from counting use to modelling the relationship between use and an outcome.

Two things are called engagement, and only one is in the log

O'Brien and Toms analysed the literature and interviewed users of four applications. They concluded that engagement is a process rather than an amount (O'Brien and Toms, 2008). They describe a point of engagement, sustained engagement, disengagement and reengagement. They define engagement as a quality of experience. A session count indexes part of the second stage and records disengagement as an absence.

Perski, Blandford, West and Michie sharpened the split. They reviewed 117 articles on engagement with digital behaviour change interventions. Engagement carried two ways at once. The experiential one covers attention and affect. The behavioural one covers what people do with the system (Perski et al., 2017). Their framework places engagement between the intervention, its context and its mechanisms of action. Two registers share one word, and only one lands in the log. A screen-reader user never appears in it, and neither does the visitor on a slow connection.

The word is overloaded across the field

This is not a health research quirk. Doherty and Doherty reviewed engagement across human-computer interaction and computer science. Their corpus of 351 articles carried 102 definitions, and they found its conception and measurement still unsettled (Doherty and Doherty, 2018). A study that reports engagement without saying which definition it means has reported nothing comparable.

Short and colleagues surveyed the measures available to evaluations. The list runs across qualitative work, questionnaires, ecological momentary assessment, system usage data, sensor data, social media data and psychophysiological measures (Short et al., 2018). Each answers a different question about a different construct, and the limitation they name is heavy reliance on system usage data as the sole assessment. A count of use is behavioural and describes the system as much as the person. A questionnaire score is experiential and describes the person at the moment of asking.

A validated instrument measures experience, not benefit

The experiential register has instruments. O'Brien, Cairns and Hall re-analysed the original user engagement scale data, with 427 and 779 valid responses. They added 344 responses from a three-year digital library project. The analysis verified a four-factor structure, and the authors published a short form (O'Brien et al., 2018). Such a scale can be repeated across a long deployment without exhausting the participant. But no item in it measures the behaviour the intervention was built to change, so the scale validates one side of the gap.

The original user engagement scale carried 31 items and claimed six dimensions. They are aesthetic appeal, focused attention, novelty, perceived usability, felt involvement and endurability (O'Brien and Toms, 2010). The re-analysis cut that to four factors and a short form, so the construct itself moved while the instrument was being validated. A four-factor score is a claim about how a participant experienced a session, and no factor is an outcome measure.

Effective engagement is a redefinition, not a refinement

Yardley and colleagues move the standard rather than the instrument. Their consensus paper argues it may be more valuable to establish and promote effective engagement than more engagement. It defines effective engagement as sufficient engagement with the intervention to achieve intended outcomes. It holds that the relationship between engagement and the desired behaviour change must be understood rather than assumed (Yardley et al., 2016). Two consequences follow. The outcome must be named before engagement can be called effective, since sufficient is empty without a target behaviour and a period. The amount of use is not the finding. The finding is the shape of the relation between use and outcome, and that shape may be flat, may saturate, or may not be monotonic.

Neither the intended dose nor the features meant to raise use are evidenced

Sieverink, Kelders and van Gemert-Pijnen reviewed 62 eHealth evaluations to ask when usage becomes adherence. In 34 studies, adherence was operationalised as the more use the better. Another 28 set a threshold for intended use, and of those 28, only six reported a justification (Sieverink et al., 2017). Their conclusion: a use measure with no justified intended dose should be reported as actual usage.

Justifying sufficiency means saying what the system does. Michie and colleagues give that vocabulary in a consensus taxonomy of 93 behaviour change techniques in 16 groups. The taxonomy was built from 124 candidates rated by 14 experts (Michie et al., 2013). With the active ingredient named, a claim about dose becomes arguable: this technique, at this rate, is expected to move this behaviour. Without it, a threshold is a convention.

Brouwer and colleagues reviewed 64 internet-delivered lifestyle interventions to ask which characteristics relate to greater exposure. Feedback, interactive elements and email or phone contact were used most often. Yet only peer and counsellor support, email or phone contact and website updates related to better exposure. Inconsistent measurement prevented firmer conclusions (Brouwer et al., 2011). The features a designer believes will buy use are, on this evidence, mostly unaccounted for.

The users who benefited most were not the ones who used it most

Hoffman and colleagues clustered 202 users of an eight-week digital mental health intervention guided by a relational agent. They used behavioural, cognitive and affective indicators together with symptom change (Hoffman et al., 2023). Three clusters appeared. Typical users were 81 people at 40 per cent, with intermediate behavioural engagement. Early users were 58 at 29 per cent, with the nominally highest behavioural engagement in week one. Efficient engagers were 63 at 31 per cent. Their behavioural engagement was the lowest of the three, but their affective and cognitive engagement was significantly higher. Efficient engagers also had greater declines in depressive symptoms and stress than typical users.

That is what an outcome-relevant model looks like, and what a count would have missed. The lightest users were among those whose symptoms fell most, and a ranking by sessions would have placed them last. The analysis needed both registers and an outcome measured separately from use.

Use is a property of the setting as much as of the person

Borgnolo and colleagues studied a measurement-based digital mental health platform across 12 Australian services, with 2682 help-seeking clients. Of them, 75.43 per cent completed the initial assessment only, and 13.01 per cent used the summary questionnaire twice or more. The service centre was the strongest predictor of engagement, with one private practice showing more than eight times the engagement of the other centres (Borgnolo et al., 2025).

Wallace and colleagues make the point where no clinician is involved. They studied Drafty, a public tabular dataset of computer science faculty profiles, and released four sources of visitors in March 2022 (Wallace et al., 2026). Visits from automatic insights produced three times more edits than visits from everyday organic visitors. The low-effort dynamic source and the high-effort asking source were similarly accurate. The effective intervention was the design of the entry points, not a message to contributors.

Contested question: an empirical quantity or a per-study judgement?

Four positions are visible. Yardley and colleagues hold that the relation can be estimated, with effective engagement defined against a stated outcome (Yardley et al., 2016). O'Brien and Toms and Doherty and Doherty hold that behavioural counts cannot carry the relation. That is because a count measures a different construct from the one the theory names (O'Brien and Toms, 2008; Doherty and Doherty, 2018). Sieverink and colleagues and Michie and colleagues hold that the threshold inside sufficient is a design decision. It stays a design decision until a mechanism and an active ingredient are named (Sieverink et al., 2017; Michie et al., 2013). Borgnolo and colleagues hold that the relation is not a property of the person at all. The service outweighed every individual clinical variable (Borgnolo et al., 2025).

Two argue from consensus processes rather than data (Yardley et al., 2016; Michie et al., 2013). O'Brien and Toms and Doherty and Doherty argue from a conceptual framework and a systematic review (O'Brien and Toms, 2008; Doherty and Doherty, 2018). Hoffman and colleagues' clusters are the closest empirical test, but single-site, exploratory and secondary to a trial (Hoffman et al., 2023). What would settle it is a multicentre deployment that fixes the outcome and the mechanism in advance, justifies its intended dose, measures both registers with validated instruments (O'Brien et al., 2018) and puts the setting in the model (Borgnolo et al., 2025). While the dispute is open, name the outcome first, then say what use was for. The strongest objection is that the setting-level result of Borgnolo and colleagues shows use without showing benefit.

Where the area is heading

The direction of travel is from a count to a model. Borgnolo and colleagues put service-level factors into the analysis (Borgnolo et al., 2025). Hoffman and colleagues close by asking for recommendations about optimal engagement and the sequencing of components. That is a call to vary dose deliberately (Hoffman et al., 2023). Wallace and colleagues sustain contribution by designing how visitors arrive (Wallace et al., 2026). O'Brien, Cairns and Hall supply short instruments that can be repeated across a long deployment (O'Brien et al., 2018). The open question is whether effective engagement becomes an estimable quantity, or stays a per-study judgement. This unit argues that the second is not a failure, provided the judgement is stated and both registers are reported.

The sources this unit rests on

Each note states what the cited document did and found. Titles are in British spelling, and identifiers were checked against registered metadata on 2 November 2026.

• O'Brien, H. L. and Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. doi:10.1002/asi.20801 Analyses the engagement literature and interviews users of four applications. Engagement emerges as a quality of experience with four stages, from a point of engagement to disengagement and reengagement. It supplies the definition a count of sessions cannot reach.
• Perski, O., Blandford, A., West, R. and Michie, S. (2017). Conceptualising engagement with digital behaviour change interventions. doi:10.1007/s13142-016-0453-1 A systematic review of 117 articles that finds engagement conceptualised as experiential and behavioural. It places engagement between the intervention, its context and its mechanisms of action, and supplies this unit's central distinction.
• Doherty, K. and Doherty, G. (2018). Engagement in HCI: conception, theory and measurement. doi:10.1145/3234149 A systematic review of 351 articles carrying 102 definitions, mapping the varied interpretations, theories and measures of engagement across HCI. It shows the definitional problem is field-wide, not confined to health research.
• Short, C. E. et al. (2018). Measuring engagement in eHealth and mHealth behaviour change interventions. doi:10.2196/jmir.9397 A methodological viewpoint appraising questionnaires, ecological momentary assessment, system usage data, sensor data and psychophysiological measures. It names sole reliance on system usage data as the limitation of current practice.
• O'Brien, H. L. and Toms, E. G. (2010). The development and evaluation of a survey to measure user engagement. doi:10.1002/asi.21229 Develops a multidimensional scale from two studies of 440 and 802 online shoppers and identifies six attributes, including focused attention, novelty and endurability. It is the primary for the original instrument's six dimensions.
• O'Brien, H. L., Cairns, P. and Hall, M. (2018). A practical approach to measuring user engagement with the refined user engagement scale. doi:10.1016/j.ijhcs.2018.01.004 Re-analyses the original scale data, 427 and 779 valid responses, and adds 344 from a three-year digital library project. It verifies a four-factor structure and publishes a short form, making the experience of use cheaper to measure.
• Yardley, L. et al. (2016). Understanding and promoting effective engagement with digital behaviour change interventions. doi:10.1016/j.amepre.2016.06.015 A consensus paper defining effective engagement empirically as sufficient engagement with the intervention to achieve intended outcomes. It argues the use-outcome relationship must be understood rather than assumed.
• Sieverink, F., Kelders, S. M. and van Gemert-Pijnen, J. E. (2017). Clarifying the concept of adherence to eHealth technology. doi:10.2196/jmir.8578 A systematic review of 62 eHealth evaluations. In 34, adherence was the more use the better, and of the 28 that also set a threshold for intended use only six justified it. It shows sufficiency is asserted rather than argued.
• Michie, S. et al. (2013). Behaviour change technique taxonomy (v1) of 93 hierarchically clustered techniques. doi:10.1007/s12160-013-9486-6 A consensus taxonomy of 93 techniques in 16 groups, from 124 candidates rated by 14 experts and coded on 85 intervention descriptions. It names the active ingredient a justification of dose requires.
• Brouwer, W. et al. (2011). Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? doi:10.2196/jmir.1639 A systematic review of 64 studies reporting objective exposure measures. Only peer and counsellor support, email or phone contact and website updates related to better exposure, so the levers assumed to raise use are weakly evidenced.
• Hoffman, V. et al. (2023). User engagement clusters of an 8-week digital mental health intervention guided by a relational agent. doi:10.2196/47198 Clusters 202 users of an eight-week relational-agent intervention. The 63 efficient engagers had the lowest behavioural engagement but the highest affective and cognitive engagement and greater symptom declines than typical users. It is the unit's evidence that more use is not more benefit.
• Borgnolo, L. J. et al. (2025). Understanding engagement with digital mental health technology in mental health services. doi:10.2196/67597 A multicentre observational study of 2682 clients across 12 Australian mental health services. The service centre was the strongest predictor of engagement, and 75.43 per cent completed the initial assessment only. It shows use is partly a property of the arrangement.
• Wallace, S. et al. (2026). Attract and engage visitors for tabular data maintenance: a longitudinal naturalistic field study. doi:10.1145/3816901 A longitudinal field study of Drafty, a public dataset of computer science faculty profiles, with four visitor sources. Automatic insights produced three times more edits than everyday organic visitors. It extends the unit's question to a design that produced its own engagement.