In-the-wild and longitudinal deployment studies

The laboratory buys control. Control makes a result interpretable, and it also makes the result fragile. A finding obtained with rested volunteers, a task set by the researcher and a device that behaves holds only while those conditions hold. Move the same system into a household and every condition moves with it.

The contested question here is what a study outside the laboratory can claim. One position treats a deployment as an early trial that ought to end in a measured change in behaviour. The other treats it as an instrument for understanding experience, where a behaviour change is often infeasible to demonstrate and sometimes beside the point. The position you take decides who you recruit, what you log, and what you are entitled to conclude.

A laboratory result is a result about the laboratory

The sharpest evidence is that a field trial partly produces the behaviour it reports. Brown, Reeves and Sherwood studied a photo-sharing deployment as a method rather than as a product, in what they called a trial of trials (Brown, Reeves and Sherwood, 2011). Participants told the researchers they had tried to use the system so as to give them the results they seemed to want.

Brown and colleagues name three effects of this kind. Demand characteristics are the adjustments people make to fit what they believe the researchers expect. Lead participants are the few users who carry a deployment: here one participant took it on himself to organise and encourage use by others, and his insight came to stand for the group. Interdependence of methods and results is the way the questions asked narrow the behaviour that can appear at all (Brown, Reeves and Sherwood, 2011).

The conclusion is not that field work is contaminated and laboratory work is clean. It is that a deployment result describes a system, a setting and a method together, and the method belongs in the finding rather than in a footnote.

Technology probes were built to inspire, not to answer

A technology probe is a deliberately simple tool placed in a real setting so that its use can teach the researcher something. Hutchinson and colleagues define probes as simple, flexible and adaptable technologies with three goals at once: understanding what people want in a real setting, field-testing the technology, and inspiring users and researchers with new ideas (Hutchinson et al., 2003).

The probe is defined against the prototype. Probes are not primarily about usability in the ordinary sense, they are not revised during the study, and a function may be left out on purpose in order to provoke a reaction (Hutchinson et al., 2003). Their project put a messageProbe and a videoProbe into family homes in France, Sweden and the United States for a month or longer.

Getting them there was ordinary engineering. The team met administrative problems installing high-speed internet in some homes, and their own hardware and software broke in ways that required extra visits to families (Hutchinson et al., 2003). Judging a probe by task success is therefore a category error. Its output is inspiration and situated understanding, not an effect size.

Research in the wild is a position, and it makes demands

The phrase marks where a study begins and ends. Ethnography and user-centred design usually start by observing existing practice and then propose requirements or design implications. Research in the wild creates and evaluates a new technology in situ, often to augment a place or a setting rather than to answer a stated user need (Rogers and Marshall, 2017).

Rogers and Marshall set out the emergence of this style, its approaches and methods, its case studies and its outcomes, and they give a chapter to practical and ethical issues (Rogers and Marshall, 2017). The demands follow from the setting. A deployment needs a device that survives without you, a way to explain the study to someone you are not standing beside, and a plan for the day it fails.

The position is contested. A deployment can support local, transferable insight about how a system is taken up. It seldom licenses the general claim that a controlled comparison supports, and the field has no settled test for which insights travel.

What a probe means is contested, and that is instructive

Boehner, Vertesi, Sengers and Dourish traced how cultural probes were adopted and adapted, and found a proliferation of divergent uses and derivatives (Boehner et al., 2007). Their argument is that the resulting disagreements are epistemological rather than technical. They concern what counts as valid interpretation, and what relationship a method has to the methodology behind it.

That reframing is the most useful thing in the paper for a reader of deployment studies, because it supplies a question to ask of any study: what does this work assume that knowledge is? A technique borrowed from an interpretive tradition and judged against a positivist standard will look slack to one reader and unambitious to another, and neither reading is the author's.

The case is the probe itself. Within a few years the same term covered practices built on incompatible assumptions, so that two studies both described as probe studies could disagree about whether a finding was supposed to be reproducible (Boehner et al., 2007).

Recruitment, consent and support are the real cost

A deployment is mostly logistics, and the logistics decide the science. Recruitment in context is slow, because participants are found where the problem is and not where the students are. Consent has to work at a distance, through a document read at home, and the person who signs is often not the only person the device measures.

Support is the cost teams underestimate. Hutchinson and colleagues met internet installation problems and equipment failures that each needed a visit (Hutchinson et al., 2003). A participant who cannot get the system working is not a missing data point. They are a cost centre, and a design that assumes a researcher on call will not scale.

Where the system records, the ethics are settled before deployment rather than after. Decide what may be logged, how long it is kept, and how a person withdraws data about themselves that also describes someone else. Rogers and Marshall give practical and ethical issues their own chapter, and Brown and colleagues show that participants know perfectly well that they are being studied (Rogers and Marshall, 2017; Brown, Reeves and Sherwood, 2011).

Engagement is not one number, and the curve carries the finding

Perski, Blandford, West and Michie reviewed 117 articles on engagement with digital behaviour change interventions and found it conceptualised in two ways (Perski et al., 2017). The experiential account covers attention, interest and affect. The behavioural account covers what people do with the system.

Those measures do not measure the same thing. Self-report questionnaires and interviews reach the experience, automatic records of logins and page views reach the behaviour, and physiological measures have shown only weak associations with self-report (Perski et al., 2017). A login count is a proxy for use, not a measure of benefit, and treating the two as equivalent is how a deployment reports a success that nobody felt.

Yardley and colleagues draw the consequence. What matters is effective engagement, defined empirically as sufficient engagement to reach the intended outcome, and sustained engagement is not always required (Yardley et al., 2016). More use is not automatically better use, so the relationship between use and outcome has to be established rather than assumed.

A rise in the first fortnight is also the least trustworthy part of any curve. Attention from being studied inflates early use, and Brown and colleagues recorded participants working to meet what they took to be the researchers' expectations (Brown, Reeves and Sherwood, 2011). Plot the curve, and read the first weeks as suspicion rather than as a baseline.

Attrition is a finding, not a failure

Eysenbach named the pattern the law of attrition: in an eHealth study a substantial share of users stop using the system or drop out, and this is a characteristic of the medium rather than a defect of one study (Eysenbach, 2005). The numbers he brought together are blunt. In one online panic programme 12 of 1161 participants completed the twelve-week course. In an open depression service 97 of 19607 users completed all five modules, against 41 of 182 in a trial setting (Eysenbach, 2005).

The setting itself changed the completion rate by a margin that no interface explains. Eysenbach's response is to make attrition a result. Plot the non-usage attrition curve, model discontinuation with survival methods such as Kaplan-Meier estimation, report a usage half-life, and describe who stays (Eysenbach, 2005).

Two further habits follow. Report intention-to-treat results alongside the results for people who kept using the system, because one analysis over heavy dropout can understate the effect on those who stayed (Eysenbach, 2005). Then treat missing data as a topic with its own paragraph: how much is missing, why, and what the analysis assumed about it.

Efficacy in a trial is not effectiveness in a service

Glasgow, Lichtenstein and Marcus attack the assumption that effectiveness research follows naturally from a successful efficacy study. The two traditions have evolved different methods and values, so an intervention that succeeds under trial conditions need not survive a service (Glasgow, Lichtenstein and Marcus, 2003). The familiar staged logic, efficacy then effectiveness then dissemination, is what they call trickle-down, and they judge it flawed or at least incomplete.

What they ask for instead is documentation of reach, adoption, implementation and maintenance, the properties that decide whether a working intervention is used at all (Glasgow, Lichtenstein and Marcus, 2003).

From the HCI side, Klasnja, Consolvo and Pratt argue that demonstrating a change in behaviour is often infeasible and can be unnecessary for a meaningful contribution, especially early in design. They propose efficacy evaluations matched to the strategy inside the system, such as self-monitoring, alongside studies that deepen the account of experience (Klasnja, Consolvo and Pratt, 2011). The two papers disagree about what an evaluation owes, and that disagreement is the live one in this area.

Living labs, and reporting a deployment a reader can trust

A living lab manages some of the uncertainty of a real setting while keeping the setting recognisable. Brown and colleagues date the rise of spaces such as MIT's PlaceLab, built so that studies could retain features of a field trial under more control (Brown, Reeves and Sherwood, 2011). The counterweight to control is partnership, and Hutchinson and colleagues treat families as partners in design rather than as subjects (Hutchinson et al., 2003).

Reporting is the weakest joint. Deployment studies have no standard equivalent to the one used for trials, and the nearest discipline comes from intervention reporting. TIDieR is a twelve-item checklist, an extension of an item in the CONSORT statement, asking for the intervention's name, rationale, materials, procedure, provider, mode, location, timing and dose, any tailoring, any modification, and how well it was delivered against plan (Hoffmann et al., 2014).

It exists because reporting was poor. In one comparison 67 per cent of drug interventions were described adequately against 29 per cent of non-pharmacological ones, and in another only 39 per cent of 137 non-drug interventions were described adequately, rising to 59 per cent when the authors were asked directly (Hoffmann et al., 2014).

Write a deployment so that another researcher could rebuild it. State the device and its version, who was recruited and who left, what was recorded and what was not, and every deviation from the plan. The same discipline transfers when the unit of study stops being one person with one device and becomes a group whose members are visible to one another, which is where the study of systems that mediate other people begins.