Whose failure counts: safety, exclusion and the ethics of interface quality

Year 1 / Module 1 Foundations of human-computer interaction / Unit 9 / Whose failure counts: safety, exclusion and the ethics of interface quality

Whose failure counts: safety, exclusion and the ethics of interface quality

HCI1101 · Unit 9 of 10

Where a design failure can injure or exclude someone, usability stops being a preference and becomes a regulated risk process, and the field's own methods for finding that failure have been shown to be weakest for exactly those users.

Whose failure counts: safety, exclusion and the ethics of interface quality

Most interface failures cost a minute. Some cost money, health, or a place in a queue for a benefit. The difference is not the size of the error but who carries it. This unit argues that once a failure can injure a person or shut them out of a service, usability stops being a preference and becomes a regulated risk process. It also argues that the field's methods for finding those failures are weakest for the people most exposed.

A risk process demands evidence made with people. If those people are not the people who will be hurt by the design, the process produces a document rather than a safeguard. The next unit turns to the newest systems, where the evidence problem is wider.

Where failure can injure, the vocabulary changes

IEC 62366-1 governs usability for medical devices, specifying a process by which a manufacturer analyses, specifies, develops and evaluates usability as it relates to safety, so that the risks of correct use and of use error can be assessed and mitigated (International Electrotechnical Commission, 2020). It addresses normal use only, and can identify but does not assess or mitigate the risks of abnormal use. Interface quality there is a residue of risk.

The Food and Drug Administration (2016) asks for the same thing differently. Its guidance recommends human factors and usability engineering so that a device is safe and effective for its intended users, uses and use environments, and is written to reduce use errors that could cause harm. Between a standard that fixes a process and a regulator that asks for its output, interface quality acquires legal weight, and that weight now reaches the general public service: in 2024 the United States Department of Justice adopted the Web Content Accessibility Guidelines version 2.1, level AA, as the technical standard for state and local government web content under the Americans with Disabilities Act, with compliance dates later extended to 2027 and 2028 (Department of Justice, 2024).

The error sits in the system before it sits in the person

Risk language invites a temptation: treat the use error as the user's failure, and the fix as more training. Reason (1990) is the standing objection: accidents in complex systems arise from latent conditions built into the organisation as much as from the acts of front-line operators, and unsafe acts separate into slips, lapses and mistakes, each pointing backwards to the design that shaped it.

The Therac-25 shows the point at the interface. Between June 1985 and January 1987 it was involved in six accidents with massive overdoses, several fatal. In one recorded in detail, the dosimetry display reported no dose and a treatment pause, and the operator pressed proceed and tried again, four times, because that was the standard procedure and the machine's frequent malfunctions had usually done no harm. The patient was receiving a massive overdose while the display said that none had been delivered. The investigation found software faults and an interface that reported the machine's internal state rather than the patient's danger (Leveson and Turner, 1993). A run of harmless malfunctions had trained the operator to override the one signal that mattered.

Karsh, Weinger, Abbott and Wears (2010) push the objection into health technology policy, setting out twelve fallacies, including that the technology improves safety by itself and that users can be trained around a poor design. On their account the technology shifts work rather than removing it, and can create hazards that did not exist before.

The unit of analysis is the whole system, not the screen

Sittig and Singh (2010) propose an eight-dimensional sociotechnical model: hardware and software, clinical content, the human-computer interface, people, workflow and communication, internal policy, external rules, and measurement and monitoring. The dimensions are interdependent, and failing to consider all of them is why projects fail; the interface is one of eight, so a fix aimed at the screen alone is a partial repair.

Rajkomar and Blandford (2012) supply the case. They applied a distributed cognition method to infusion pump use in an intensive care unit, modelling information flow, physical layout, social structure and artefacts, and found cognition distributed across the nurses, the arrangement of the unit and the devices. A nurse who reads the pump layout as an external memory of the task relies on part of the design that no device-level test records, so the risk process cuts the system at the wrong joint.

Exclusion is a design outcome, not an accident

Wobbrock, Kane, Gajos, Harada and Froehlich (2011) propose ability-based design. It begins from the abilities a user has and treats the system as the thing to be adapted rather than a disability to be accommodated afterwards, and sets out seven principles including adaptability and transparency. If exclusion is produced by a design, it can be produced differently.

The measurement of that exclusion is where the evidence is thinnest. Power, Freire, Petrie and Swallow (2012) ran task-based evaluations with 32 blind users across 16 websites and recorded 1,383 problems. Just over half, 50.4 per cent, were covered by a success criterion in the Web Content Accessibility Guidelines; where a guideline was implemented, 16.7 per cent of the websites used the recommended technique while the users still met the problem: a checklist pass and an accessible experience are different claims.

The social layer is harder to design away. Shinohara and Wobbrock (2011) interviewed twenty people who used assistive technology and found that some devices marked their users as disabled in the eyes of others, with two misperceptions running through the accounts: that a device removes a disability, and that its user is helpless without it. Bennett, Brady and Branham (2018) expose a related assumption: assistive technology research imagines an independent individual user, and that picture hides the family, paid support and infrastructure that make a device work. A stated model of the user decides whose effort is counted.

Contested question: is exclusion repaired by better method, or by moving who decides?

The literature agrees that exclusion and injury are design outcomes and disagrees about the remedy. Wobbrock and colleagues (2011, 2018) hold that the repair is a change in design practice: focusing on ability and adapting the system shrinks the mismatch. Bennett and Rosner (2019) attack that class of remedy: empathy activities often substituted a designer's imagined experience for the account of the people concerned, moving authority rather than improving a method.

A second disagreement is about whether the regulated process closes the gap at all. The standards position, held by the IEC and the FDA, is that a specified process with evidence of validation reduces harm. Karsh and colleagues (2010) hold the opposite about the same technology: its adoption has not delivered the promised gains and it can add hazards, so a process built on the assumption of benefit may certify risk it cannot see. Frauenberger, Good, Fitzpatrick and Iversen (2015) supply the criterion: in participatory design, rigour and accountability come through debate, critique and reflection, which need a shared language so that judgements are inspectable. The evidence is unevenly strong: the two standards are process documents reporting no outcome data, Reason (1990) is an argument supported by accident analyses, Leveson and Turner (1993) an investigation of six events, and the measured case Power and colleagues (2012), with 32 users and 1,383 problems.

What would settle the first dispute is a comparison the literature does not contain: two design efforts for the same excluded group, one led by designers applying ability-based design and one led by the users whose access is at stake, measured on the users' own terms. What would settle the second is post-market evidence, use-error and harm rates for devices with and without a documented human factors validation, recorded after release rather than at submission. Neither is present here, and the strongest counter to this unit's claim is that regulation is not the same as safety: the standard covers normal use only, and the evidence it demands is produced at submission rather than after release.

While it is open, study the work as done rather than as described, and let the people exposed to the failure decide what counts as one. Validate with participants who represent the intended users, including the people in later life who will use it at home, and record the design's assumption about who the user is.

Where the argument is heading

The constructive line is moving into the mainstream. Wobbrock, Gajos, Kane and Vanderheiden (2018) restate ability-based design for a general computing readership, arguing that designing for abilities rather than assigned disabilities produces systems better matched to their users. The political line moves from the project to the system: Bodker and Kyng (2018) ask participatory design to take on technology's pressure on work and lives, which means arguing about deployment, scale and control.

The sources this unit rests on

International Electrotechnical Commission. (2020). IEC 62366-1:2015+AMD1:2020. Medical devices. Part 1: Application of usability engineering to medical devices. IEC, Geneva. https://webstore.iec.ch/en/publication/67220 Specifies a process to analyse, develop and evaluate device usability as it relates to safety, so that the risks of correct use and use error can be assessed and mitigated. It addresses normal use only.
U.S. Food and Drug Administration. (2016). Applying human factors and usability engineering to medical devices: guidance for industry and Food and Drug Administration staff. FDA. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/applying-human-factors-and-usability-engineering-medical-devices Recommends human factors and usability engineering so that a device is safe and effective for its intended users, uses and use environments, and aims to reduce use errors that could cause harm.
Reason, J. (1990). Human Error. Cambridge University Press. doi:10.1017/CBO9781139062367 Argues that accidents in complex systems arise from latent conditions in the organisation as much as from front-line acts, and classifies unsafe acts into slips, lapses and mistakes.
Karsh, B. T., Weinger, M. B., Abbott, P. A. and Wears, R. L. (2010). Health information technology: fallacies and sober realities. Journal of the American Medical Informatics Association 17(6), 617-623. doi:10.1136/jamia.2010.005637 Sets out twelve fallacies behind health technology policy, including that the technology improves safety by itself and that users can be trained around a poor design, and argues that it shifts work and creates hazards.
Sittig, D. F. and Singh, H. (2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. BMJ Quality and Safety 19(Suppl 3), i68-i74. doi:10.1136/qshc.2010.042085 Proposes an eight-dimensional sociotechnical model covering hardware and software, clinical content, the human-computer interface, people, workflow and communication, internal policy, external rules, and measurement and monitoring.
Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S. and Froehlich, J. (2011). Ability-based design: concept, principles and examples. ACM Transactions on Accessible Computing 3(3), 1-27. doi:10.1145/1952383.1952384 Proposes ability-based design, which begins from the abilities a user has and treats the system as the thing to be adapted rather than centring disability, setting out seven principles including adaptability and transparency.
Bennett, C. L. and Rosner, D. K. (2019). The promise of empathy: design, disability, and knowing the 'other'. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19), 1-13. doi:10.1145/3290605.3300528 Analyses empathy activities aimed at understanding disabled people and finds that building empathy often substituted a designer's imagined experience for the account of those concerned.
Frauenberger, C., Good, J., Fitzpatrick, G. and Iversen, O. S. (2015). In pursuit of rigour and accountability in participatory design. International Journal of Human-Computer Interaction 74, 93-106. doi:10.1016/j.ijhcs.2014.09.004 Argues that rigour and accountability in participatory design are delivered through debate, critique and reflection rather than the positivist framework.
Bodker, S. and Kyng, M. (2018). Participatory design that matters: facing the big issues. ACM Transactions on Computer-Human Interaction 25(1), 1-31. doi:10.1145/3152421 Argues that participatory design should aim at the big issues, including technology's pressure on work and lives, rather than at consensus around a product, and calls for a changed partnership with users and democratic control.
Rajkomar, A. and Blandford, A. (2012). Understanding infusion administration in the ICU through distributed cognition. Journal of Biomedical Informatics 45(3), 580-590. doi:10.1016/j.jbi.2012.02.003 Applies a distributed cognition method to infusion pump use in an intensive care unit, modelling information flow, layout, social structure and artefacts.
Wobbrock, J. O., Gajos, K. Z., Kane, S. K. and Vanderheiden, G. C. (2018). Ability-based design. Communications of the ACM 61(6), 62-71. doi:10.1145/3148051 Restates ability-based design for a computing readership: designers who focus on the abilities a user has, rather than the disability a user is assigned, build systems better matched to those abilities.
Shinohara, K. and Wobbrock, J. O. (2011). In the shadow of misperception: assistive technology use and social interactions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 705-714. doi:10.1145/1978942.1979044 Interviews twenty people who used assistive technology and finds that some devices marked their users as disabled, with two misperceptions running through the accounts.
Bennett, C. L., Brady, E. and Branham, S. M. (2018). Interdependence as a frame for assistive technology research and design. Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18), 161-173. doi:10.1145/3234695.3236348 Argues that assistive technology research frames disabled people as independent individual users, hiding the family, paid support and infrastructure that make devices work, and proposes interdependence as the frame.
Leveson, N. G. and Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer 26(7), 18-41. doi:10.1109/MC.1993.274940 The investigation of six massive overdoses between June 1985 and January 1987: the operator interface reported no dose and a treatment pause while the patient was overdosed, and the accidents are read across development, operation and regulation.
Power, C., Freire, A. P., Petrie, H. and Swallow, D. (2012). Guidelines are only half of the story: accessibility problems encountered by blind users on the web. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12), 433-442. doi:10.1145/2207676.2207736 Task-based evaluations with 32 blind users on 16 websites yielded 1,383 problems, 50.4 per cent covered by a success criterion, while websites using recommended techniques did not always remove the problem.
U.S. Department of Justice. (2024). Nondiscrimination on the Basis of Disability; Accessibility of Web Information and Services of State and Local Government Entities. Final rule, 89 FR 31320. https://www.ada.gov/resources/2024-03-08-web-rule/ Adopts the Web Content Accessibility Guidelines version 2.1, level AA, as the technical standard for state and local government web content under the Americans with Disabilities Act.

Check your understanding & apply it · 4 questions, 1 essay

1A treatment machine's display reported that no dose had been delivered and that treatment was paused, while the patient was being overdosed. The operator pressed the proceed key and tried again four times after the first attempt, following the procedure that the machine's frequent harmless malfunctions had made routine. Which reading does this unit support?

2An evaluation reports 1,383 problems raised by 32 blind users across 16 websites, of which 50.4 per cent were covered by a success criterion, and notes that where a recommended technique was implemented some users still met the problem. What does this unit draw, and which claim is left unsupported?

3A team runs a workshop in which designers simulate using a service with a screen reader for an hour, then redesigns the form. Applying Wobbrock and colleagues (2011, 2018) and Bennett and Rosner (2019) as this unit does, what does the method risk, and what does ability-based design change?

4A manufacturer submits a validation report for a home-use device: fifteen participants drawn from its own staff, no use errors recorded, and a statement that the device was validated in accordance with the process standard. Which reading of the unit's standards material is supported?

5Essay A health service introduces a self-service online form for a disability benefit. The service's usability report records twelve sessions with participants recruited from a general user panel, a task-completion rate above ninety per cent, and no injuries, and the agency accepts it as evidence of safe and usable design. Disability organisations object that the people who need the benefit most, including claimants with low literacy, claimants using a shared metered phone, and claimants who read in a second language, have not been represented. The agency replies that the design followed the standard's process, applied good practice, and reduced errors, and that the remaining problems are matters of individual support rather than of design. Using IEC 62366-1 and the FDA guidance on usability as a safety-related process with specified users and an evidenced process, Reason (1990) and Karsh and colleagues (2010) on locating the cause of error in the system rather than in the person, Sittig and Singh (2010) on the sociotechnical system and the work system, Rajkomar and Blandford (2012) on distributed cognition in a care setting, Wobbrock and colleagues (2011, 2018) on ability-based design, Shinohara and Wobbrock (2011) and Bennett, Brady and Branham (2018) on the social production of exclusion and on interdependence, Bennett and Rosner (2019) on empathy as a substitute for the user's account, and Frauenberger and colleagues (2015) and Bodker and Kyng (2018) on accountability and power in participatory design, write an assessment that (a) states what the reported evidence does and does not establish about who may be excluded, (b) identifies the design and process decisions that produced the exclusion, and (c) states what evidence the agency should gather, how it should be gathered, and what result would count against its case.

The report answers the wrong question. It shows that twelve users drawn from a general panel could complete the form, and a completion rate of above ninety per cent establishes that the form is usable by people like those participants; it does not establish that the form is usable by the specified users the benefit is for. The standard's own logic supplies the objection, because IEC 62366-1 frames usability as a process in relation to safety and the FDA guidance asks for evidence about the intended users, uses and use environments, so a sample that omits low-literacy claimants, claimants on a shared metered phone and claimants reading in a second language leaves the intended-use claim untested. Reading the omission as an individual support matter repeats the fallacy Karsh and colleagues identify: the technology does not improve access by itself, it shifts the work, and here the work moves to the claimant in the form of interpretation, connectivity and a device they may not control. Reason's model locates the cause in latent conditions and in the design that shaped the act, so a claimant who abandons the form has not failed; the form failed a person it was built for. The agency has also mis-specified the system. Sittig and Singh place the human-computer interface in an eight-dimensional model whose dimensions interact, so a form redesigned while the literacy demand, the device assumption and the support route stay as they are is a partial repair. Rajkomar and Blandford show why this matters in a care setting, since cognition and safety were distributed across people, layout and artefacts and a device-level test recorded none of it. The exclusion the disability organisations name is also social, not only technical: Shinohara and Wobbrock found that devices mark their users and that a functional need outweighs the embarrassment of that mark, while Bennett, Brady and Branham show that the independent individual user assumed by the design hides the support networks that make access work. What the agency should do follows from these findings. It should recruit participants who meet the specified-user description, including the three groups named, and it should observe the form where those claimants actually use it, on their own devices and connections, rather than in a lab. It should record abandonment and recovery, not only completion, because the harm of exclusion appears as a person who leaves. It should test the redesigned form with the users as decision-makers rather than as test subjects, since Bennett and Rosner show that a designer's empathy is not the same as the user's account, and Frauenberger and colleagues give the criterion for judging the process: the judgements must be inspectable and the account owed to the participants, not only to the agency. A result that would count against the service is a redesigned form whose specified-user participants complete it at a rate comparable to the panel and abandon it at a comparable rate, with the claimants themselves recording that they could complete it without help. If, after that, claimants still needed assistance the form was supposed to remove, the agency's claim would not survive.