A laboratory result is not a deployed product, and where the setting is safety critical the gap is measured in harm

Every unit in this module has ended at the same border, and this tenth of ten units names it. A sensor works in a laboratory because somebody chose the room, the task and the person. A product meets a room, a task and a person nobody chose. A controlled study and a field deployment are not the same evidence at two levels of quality. Where the setting is safety critical, those findings become obligations whose failure costs harm.

The field's default was the laboratory, and that default was a choice

Kjeldskov and Graham (2003) classified 102 mobile human-computer interaction papers published between 2000 and 2002. Laboratory experiments accounted for 31 per cent of them against 20 per cent reporting field studies. Of the 41 per cent that evaluated anything at all, 71 per cent used laboratory experiments and 19 per cent field experiments.

Eleven years later Kjeldskov and Skov (2014) read the 142 papers citing the 2004 study. That study had asked whether fieldwork repaid its cost. They found 62 studies conducted in a laboratory or in the field, 23 and 39 respectively, and only 16 empirical comparisons. No definite answer had emerged, and the useful question had shifted from whether to go into the field to when and how.

A direct comparison finds different problems, not the same problems more clearly

Direct comparison of the two settings is rare. Duh, Tan and Chen (2006) evaluated one mobile phone twice, in a laboratory and in real-life situations. They reported significant differences in the frequency and severity of the problems found, in participants' behaviour and in their subjective responses. A laboratory test is not a cheaper copy of a field test with a smaller sample. It observes a different distribution of problems, which is why the criterion applied to field work matters as much as the setting.

Deployment changes the question and the sample

Consolvo and colleagues (2008) built UbiFit Garden, which infers physical activity from on-body sensing and shows it on a glanceable mobile display. They ran a three-week field trial with twelve participants in their own lives. Their findings concentrate on the participants' experience of the sensing, which sustained use shows and a session in a room cannot.

Henze and colleagues (2011) reported five experiments published through the Android Market, whose applications were installed more than 30,000 times. Store deployment buys reach at the cost of control, and applications are often used briefly, so data must be collected early.

In safety-critical work a design decision becomes an obligation

Cvach (2012) reviewed 72 articles on monitor alarm fatigue. She described it as the number one medical device technology hazard of 2012. She tied it to a high false alarm rate, poor positive predictive value, missing standardisation and too many alarming devices.

Edworthy and Hellier (2006) reviewed the human factors of medical alarms. They covered alarm numbers, false alarm rates, auditory design, intelligent systems and a worldwide alarms standard. They reported that alarms are frequently too loud, irritating, confusing and too numerous, so they are often turned off, hindering rather than enhancing task performance. An alarm the monitor plays rather than displays does not reach a patient who is hard of hearing or an older patient. A hospital buying a monitor should expect a tested interface, because the alternative is a device whose alarms its users silence.

The obligation is clearest where the interface, not the operator, failed

Fairbanks and Caplan (2004) compiled a scenario from several real incidents, describing seven medical errors that look like staff mistakes. In the first, a paramedic tries to defibrillate a 67-year-old man in ventricular tachycardia and nothing happens. The device displayed that it was in synchronised mode, but gave no signal that it was withholding the shock (Fairbanks and Caplan, 2004). The heart rhythm was too low to fire on. The scenario is constructed, so it is not evidence about how often this occurs, and the authors use it to argue for testing before purchase. The mechanism is the point, a display that reports its state without reporting its readiness.

An unreliable signal trains its own users

A laboratory study demonstrated the mechanism. Bliss, Gilson and Deaton (1995) gave 138 undergraduates a demanding psychomotor task. They presented alarms of 25, 50 and 75 per cent reliability. About 90 per cent did not answer every alarm; they matched their response rate to the expected probability that an alarm was true.

Lewandowska and colleagues (2026) applied alarm management in one anaesthesiology and intensive care unit around staff education and technical adjustment. They recorded 119,158 alarms before and 151,840 after, a rise of 27.4 per cent. Red technical alarms fell by 61.1 per cent. The paper records that staff reported high noise levels, selective responsiveness to alarms and varied proficiency in operating the monitors.

In safety-critical settings the outcome is measured in events

Klauer and colleagues (2014) installed sensors and cameras in the vehicles of 42 newly licensed drivers and 109 experienced adults. The novices ranged in age from 16.3 to 17.0 years. They analysed 167 crashes and near-crashes among the novices and 518 among the experienced group. Dialling a phone raised the risk of a crash or near-crash among novices, with an odds ratio of 8.32. The 95 per cent confidence interval ran from 2.83 to 24.42. Among experienced drivers the odds ratio was 2.49 (1.38 to 4.54). The same interaction carries different risk in different populations, which a single usability score cannot express.

Dingus and colleagues (2016) analysed 905 injurious and property-damage crash events from naturalistic driving data. They found driver-related factors, meaning error, impairment, fatigue and distraction, present in almost 90 per cent of crashes.

The contested question: does the field repay its cost, and does the cry-wolf effect survive deployment?

The first disagreement is about value for effort. Duh and colleagues (2006) found the field surfacing problems different in frequency and severity. The 2014 review records a study rating overall usability more highly in the field. Against that, the comparison that started the argument, reported in the same review, found 36 usability problems in the laboratory against 23 in the field. Of the 37 problems, 14 were unique to the laboratory and only one to the field. The laboratory took 34 person-hours against 65. Kjeldskov and Skov (2014) also record comparative studies finding no difference, so the field camp cannot claim a settled result.

A third position disputes the question. Kjeldskov and Skov (2014) observe that the 16 comparative studies differed so much in what counted as a laboratory and as a field. Real replication has not happened, so they conclude the field should ask when and how rather than whether. Dourish (2006) holds that the field judges field work by the wrong criterion. What would settle it is a set of studies holding the system, the tasks and the participants fixed and varying only the setting. They would report the problems found, their severity and the person-hours, and state each definition.

The second disagreement concerns whether a laboratory mechanism travels. Bliss and colleagues (1995) produced probability matching under controlled conditions. Wickens and colleagues (2009) examined the same claim where it matters most. They analysed data around 495 conflict alerts in air traffic control, 45 per cent of them false. Centres with more false alerts produced more non-responses. The study found no evidence that controllers failed to respond to true alerts or responded more slowly. They issued trajectory changes before the alert arrived. That failure to reproduce the mechanism outside the laboratory is the strongest counter-evidence in this unit. Linking false-alarm rate to responses to true events across many sites would decide it. The clinical data Lewandowska and colleagues (2026) report show the question is still open.

Rated by kind, the field-value debate rests on one direct comparison (Duh et al., 2006) and on a review that aggregates studies with no shared definition of setting (Kjeldskov and Skov, 2014), so neither camp holds controlled evidence. The cry-wolf debate rests on a controlled experiment (Bliss et al., 1995) against an analysis of real alerts (Wickens et al., 2009), which is the stronger evidence because the harm occurs there. While the dispute is open, a designer tests the interface in the setting that carries the harm and reports events rather than scores. The strongest objection is that a laboratory mechanism may not reproduce in the field at all.

Where this is heading

The direction of travel is towards deployment as the unit of analysis. Kjeldskov and Skov (2014) ask for field studies that are truly in the wild and longitudinal rather than brief tests outside the building. Henze and colleagues (2011) show that a product release is an instrument with a known sampling bias. Weichbroth (2024) specifies what a mobile evaluation must record for laboratory and field results to be read together. The market context is stark: fewer than five per cent of applications released through 2023 succeeded. In the safety-critical end the pressure is towards standards and measurement. Edworthy and Hellier (2006) find the research still incompletely integrated into alarm design. Lewandowska and colleagues (2026) call for coordinated thresholds and work across nursing, medicine, clinical engineering and manufacturers. Klauer and colleagues (2014) and Dingus and colleagues (2016) price the gap in crashes, and Cvach (2012) prices it in alarms that go unanswered.

Sources

• Bliss, J. P., Gilson, R. D. and Deaton, J. E. (1995). Human probability matching behaviour in response to alarms of varying reliability. Ergonomics. doi:10.1080/00140139508925269 A dual-task study of 138 undergraduates with alarms at 25, 50 and 75 per cent reliability; about 90 per cent matched their response rate to the expected probability of a true alarm. The mechanism behind alarm fatigue.
• Consolvo, S., McDonald, D. W., Toscos, T. et al. (2008). Activity sensing in the wild: UbiFit Garden. CHI. doi:10.1145/1357054.1357335 A three-week field trial with twelve participants using on-body sensing, activity inference and a glanceable display, reporting how people experience being inferred.
• Cvach, M. (2012). Monitor alarm fatigue. Biomedical Instrumentation and Technology. doi:10.2345/0899-8205-46.4.268 A review of 72 articles describing alarm fatigue as the number one medical device technology hazard of 2012, tied to a high false alarm rate and poor positive predictive value. An attention finding becomes a hazard category.
• Dingus, T. A., Guo, F., Lee, S. et al. (2016). Driver crash risk factors using naturalistic driving data. PNAS. doi:10.1073/pnas.1513271113 An analysis of 905 injurious and property-damage crash events, with driver-related factors present in almost 90 per cent of crashes and handheld devices showing high use rates and risk. Interface consequences are measured as event rates.
• Dourish, P. (2006). Implications for design. CHI. doi:10.1145/1124772.1124855 An argument that the customary evaluative metric for field studies, implications for design, may fail to capture the value of the investigation. Grading field evidence on laboratory terms is the error it names.
• Duh, H. B. L., Tan, G. C. B. and Chen, V. H. H. (2006). Usability evaluation for mobile device: laboratory and field tests. Mobile HCI. doi:10.1145/1152215.1152254 One mobile device evaluated in both settings, with significant differences in the frequency and severity of problems, in behaviour and in subjective responses. The clearest evidence here that the setting changes the finding.
• Edworthy, J. and Hellier, E. (2006). Alarms and human behaviour: medical alarms. British Journal of Anaesthesia. doi:10.1093/bja/ael114 A review of five areas, from alarm numbers and false alarm rates to auditory design and the alarms standard, reporting that alarms are too loud, irritating and too numerous, and are therefore turned off. Disabled alarms are the obligation the setting imposes.
• Henze, N., Pielot, M., Poppinga, B. et al. (2011). My app is an experiment. International Journal of Mobile Human Computer Interaction. doi:10.4018/jmhci.2011100105 Five experiments published through the Android Market, with more than 30,000 installations, showing that deployment buys scale at the cost of control and that users are not a representative sample.
• Kjeldskov, J. and Graham, C. (2003). A review of mobile HCI research methods. Mobile HCI. doi:10.1007/978-3-540-45233-1_23 A classification of 102 mobile HCI papers from 2000 to 2002, finding 31 per cent laboratory experiments against 20 per cent field studies, and 71 per cent laboratory experiments among papers that evaluated anything.
• Kjeldskov, J. and Skov, M. (2014). Was it worth the hassle? Mobile HCI. doi:10.1145/2628363.2628398 A review of the 142 papers citing the 2004 study, finding 62 laboratory or field studies and only 16 empirical comparisons, and reporting 36 laboratory problems against 23 field problems, with 14 unique to the laboratory and one to the field.
• Klauer, S. G., Guo, F., Simons-Morton, B. G. et al. (2014). Distracted driving and crash risk among novice and experienced drivers. New England Journal of Medicine. doi:10.1056/nejmsa1204142 Naturalistic driving data for 42 novice and 109 experienced drivers, with 167 and 518 crash or near-crash events, giving a dialling odds ratio of 8.32 for novices and 2.49 for experienced drivers.
• Lewandowska, K., Medrzycka-Dabrowska, W., Miklas, M. and Wujtewicz, M. (2026). Improving alarm management in critical care. Intensive and Critical Care Nursing. doi:10.1016/j.iccn.2025.104254 An action research study in one intensive care unit, recording 119,158 alarms before and 151,840 after a technical and educational intervention, while red technical alarms fell by 61.1 per cent. Volume and quality move in opposite directions.
• Weichbroth, P. (2024). Usability testing of mobile applications. Applied Sciences. doi:10.3390/app14051792 A literature-based framework specifying settings, techniques and designs for mobile usability testing, including laboratory and field settings, in a market where fewer than five per cent of applications released through 2023 succeeded.
• Wickens, C. D., Rice, S., Keller, D. et al. (2009). False alerts in air traffic control conflict alerting. Human Factors. doi:10.1177/0018720809344720 An analysis of 495 conflict alerts, 45 per cent of them false, finding more non-responses where false alerts were more frequent but no failure or delay in responding to true alerts, because controllers anticipated them.
• Fairbanks, R. and Caplan, S. H. (2004). Poor interface design and lack of usability testing facilitate medical error. The Joint Commission Journal on Quality and Safety. doi:10.1016/s1549-3741(04)30068-7 A scenario compiled from real events, describing seven medical errors, including a paramedic whose defibrillator withheld a shock while displaying that it was in synchronised mode.