Automation and the allocation of function: who holds control

Every automated system contains a division of labour. The machine takes the fast, repeatable work, and the person keeps the judgement and the unexpected. The list is old, and this unit argues that it misstates the problem.

Automation rarely removes the human. It relocates the work to monitoring and recovery, and that work is harder than what it replaced: the operator watches a process that seldom needs them, then intervenes in the rare case that is by definition the most demanding. The cost depends on the level of automation, the interface and the machine's reliability. The next unit asks who carries the failure when that cost lands.

The capability list became a design rule, and the rule cannot work

Fitts (1951) edited the report that gave designers a two-column comparison of human and machine capability: people were credited with perceiving patterns, improvising and handling the unexpected, the machines of the period with speed, repetition and precision. Later readers turned it into a rule and assigned each function to the stronger side.

Dekker and Woods (2002) argue the rule cannot work: comparing two components cannot decide how they are arranged together, because the outcome depends on how the two coordinate once both are in the system. The comparison had justified automation without examining its consequences, and they propose the coupling, not the parts, as the design object.

The work the comparison leaves out

Bainbridge (1983) stated the consequence first, writing about the industrial process operator:

The designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate.

Bainbridge reports that vigilance research found attention to a source on which little happens cannot be sustained for more than about half an hour, and that an operator kept out of the loop loses the skills needed to take over, while the situations requiring take-over exceed the routine.

Parasuraman and Manzey (2010) modernise the claim: complacency and bias follow from how attention is allocated, not from a failure of vigilance. Complacency appears under multiple-task load, when manual work competes with the automated task for attention. Bias with an imperfect aid produces omission and commission errors.

The binary gave way to ten levels and four classes of function

The binary of the original list gave way to a graded scale. Sheridan and Verplank (1978) studied undersea teleoperation and set out ten levels of automation in man-computer decision-making, from the human doing everything to the computer doing everything, even deciding whether to inform the human, with intermediate levels at which the computer suggests an option the human need not follow. The report notes that awkward communication makes supervisory control inferior to manual control.

Parasuraman, Sheridan and Wickens (2000) turned the scale into a description of types. Automation applies to four classes of function, information acquisition, information analysis, decision and action selection and action implementation, each able to sit at a different level. The authors note that automation changes human activity and imposes new coordination demands.

Monitoring is a task, and people are worse at it

Endsley and Kiris (1995) tested the claim. Participants either monitored an automated navigation system or performed the same navigation manually, and the monitors were slower and worse at taking over after a failure, with the difference traced to a loss of situation awareness built during monitoring and moderated by the control the operator retained.

A recent case shows the same structure with the authority reversed. On the Boeing 737 MAX, the manoeuvring characteristics augmentation system took stabiliser authority automatically to push the nose down, drawing on a single angle-of-attack sensor while one flight control computer was active. The Lion Air accident report of October 2018 records it activating 24 times after faulty data. A single erroneously high sensor input could produce repeated nose-down trim and, with other flight deck effects, prevent the crew from maintaining safe flight; the corrective software required inputs from both sensors (U.S. Department of Transportation, 2021; Federal Aviation Administration, 2020). The automation was reliable enough to be left running and wrong often enough to require a take-over.

Two results qualify the thesis. Kaber and Endsley (2004) varied the level of automation and its adaptivity in a dual-task experiment: low-level automation produced the best primary-task performance and intermediate levels the highest situation awareness, and neither came with reduced workload. Wickens and Dixon (2007) pooled 20 studies into 35 comparisons of aided against unaided performance, finding benefits rising with reliability and a crossover at 0.70 below which automation was worse than none. The thesis therefore holds conditionally: level, allocation and reliability decide the outcome.

Trust is a calibration problem, not a trust problem

If the operator decides when to rely on the machine, the governing state is trust, and the target is not more trust. Lee and See (2004) define trust as the attitude that an agent will help achieve an individual's goals in a situation characterised by uncertainty and vulnerability, and argue the aim is appropriate reliance: over-trust produces misuse and under-trust disuse. The problem is not that operators trust machines, but that they trust them uniformly when reliability is not uniform.

Hoff and Bashir (2015) set the boundary: their systematic review covered January 2002 to June 2013, 101 papers containing 127 eligible studies, and organised the influences on trust into dispositional, situational and learned layers. An interface change therefore acts on one layer of three.

Contested question: does the level of automation decide the outcome, or the coupling?

The literature disagrees about what determines performance. The allocation tradition holds that performance is set by how much is automated: automate the stronger side, and the human's share shrinks. Dekker and Woods (2002) hold the opposite: the outcome is set by the coordination between the two, so a level chosen well in the abstract can fail in a particular setting, and the same level can help or harm. Bainbridge (1983) and Endsley and Kiris (1995) show high automation lowering awareness and take-over ability, while Kaber and Endsley (2004) and Wickens and Dixon (2007) show intermediate levels and reliable aids helping. That is the strongest counter to this unit's claim: the relocation to monitoring is a loss only under conditions a design can change. The evidence is uneven: Fitts (1951) is a capability list, Bainbridge (1983) argues from experience, and the controlled experiments are few.

Endsley is an author of both the conundrum and an experiment that qualifies it, so the field's leading proponent treats the effect as conditional rather than fixed. A second disagreement concerns trust: Lee and See (2004) treat it as governing reliance and so as a design target, while Hoff and Bashir (2015) show it is layered across disposition, situation and learning, so an interface change moves one layer of three. Whether trust causes reliance or summarises it remains open.

What would settle the first question is an experiment varying the level of automation and the coordination independently, measuring monitoring and take-over time separately, and reporting the aid's reliability and error structure. What would settle the second is a field study measuring trust at more than one point, with reliance recorded independently of the self-report.

The design has to make the rare intervention recoverable

If intervention is rare, the intervention is the design problem. Norman (1983) separates a slip, a correct intention executed wrongly and caught by feedback, from a mistake, a wrong intention corrected only by changing the operator's model. Both are likely at take-over: the operation is unpractised, and the operator's account of the system comes from a console, not from the task. Reversibility and few modes reduce the first. A display of the machine's state is the usual answer to the second, but Romeo and Conti (2025), reviewing 35 studies of automation bias, report that the explanation a display supplies often leaves over-reliance intact and can strengthen misplaced trust among less experienced users; professional expertise protects against it.

While it is open, decide the level task by task rather than for the whole system, report the aid's reliability and which way it errs, and keep the human able to cross-check the raw data, which presumes a reader of the console's language at the console's reading level. A document that says only that a system is automated has decided nothing.

Where the argument is heading

The argument has moved from allocating functions to designing the collaboration. Sheridan (2016) reviewed human-robot interaction across four application areas and reported that evidence of success was largely live demonstrations rather than validated findings about long-term use, with self-driving cars and humanoids at initial stages. Endsley (2017) named the automation conundrum: as autonomy grows and the machine becomes more reliable, the operator's situation awareness and the chance of a competent take-over fall, and her response is an oversight model intervening at the level of automation and its granularity of control. The field can state the problem and cannot yet say which collaboration to build.

The sources this unit rests on

• Fitts, P. M. (ed.). (1951). Human engineering for an effective air-navigation and traffic-control system. National Research Council, Washington DC. https://archive.org/details/perma_cc_W3UD-MGPW Compares human and machine capability in two columns, giving people perceiving patterns, improvising and handling the unexpected, and the machines speed and precision.
• Dekker, S. W. A. and Woods, D. D. (2002). MABA-MABA or abracadabra? Progress on human-automation co-ordination. Cognition, Technology and Work 4(4), 240-244. doi:10.1007/s101110200022 Argues that comparing human and machine capabilities cannot decide the allocation, because the question is how the two coordinate in a setting.
• Bainbridge, L. (1983). Ironies of automation. Automatica 19(6), 775-779. doi:10.1016/0005-1098(83)90046-8 Argues that automating the parts a designer can automate leaves the operator the parts a designer cannot. It reports that attention to a source on which little happens cannot be sustained for more than about half an hour, and that an operator out of the loop loses the skills for take-over.
• Sheridan, T. B. and Verplank, W. L. (1978). Human and computer control of undersea teleoperators. Man-Machine Systems Laboratory, Massachusetts Institute of Technology. https://apps.dtic.mil/dtic/tr/fulltext/u2/a057655.pdf Sets out ten levels of automation in man-computer decision-making, with intermediate levels such as the computer suggesting an option the human need not follow.
• Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man and Cybernetics, Part A 30(3), 286-297. doi:10.1109/3468.844354 Proposes that automation be described by four classes of function, information acquisition, information analysis, decision and action selection, and action implementation, and argues that automation changes human activity.
• Parasuraman, R. and Manzey, D. H. (2010). Complacency and bias in human use of automation: an attentional integration. Human Factors 52(3), 381-410. doi:10.1177/0018720810376055 Reviews empirical studies of complacency and bias and proposes an attentional account of both.
• Endsley, M. R. and Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors 37(2), 381-394. doi:10.1518/001872095779064555 Compares participants monitoring an automated system with participants performing the task manually, and finds low situation awareness corresponded with out-of-the-loop performance decrements in decision time after a failure.
• Kaber, D. B. and Endsley, M. R. (2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science 5(2), 113-153. doi:10.1080/1463922021000054335 Varies the level of automation and the timing of dynamic function allocation in a dual-task experiment: low-level automation gave the best primary-task performance and intermediate levels the highest situation awareness, but neither reduced workload.
• Lee, J. D. and See, K. A. (2004). Trust in automation: designing for appropriate reliance. Human Factors 46(1), 50-80. doi:10.1518/hfes.46.1.50.30392 Defines trust as the attitude that an agent will help achieve an individual's goals in a situation of uncertainty and vulnerability, and argues the design aim is appropriate reliance rather than maximum trust.
• Hoff, K. A. and Bashir, M. (2015). Trust in automation: integrating empirical evidence on factors that influence trust. Human Factors 57(3), 407-434. doi:10.1177/0018720814547570 Systematically reviews experiments published from January 2002 to June 2013, 101 papers containing 127 eligible studies, and organises the influences on trust into dispositional, situational and learned layers.
• Wickens, C. D. and Dixon, S. R. (2007). The benefits of imperfect diagnostic automation: a synthesis of the literature. Theoretical Issues in Ergonomics Science 8(3), 201-212. doi:10.1080/14639220500370105 Pools 20 studies of imperfect diagnostic automation into 35 comparisons of aided against unaided performance. Benefits rose with reliability, with a crossover at 0.70 below which the automation was worse than no automation.
• Norman, D. A. (1983). Design rules based on analyses of human error. Communications of the ACM 26(4), 254-258. doi:10.1145/2163.358092 Classifies human error into slips of action and mistakes of intention, and derives design rules including making actions reversible and reducing the number of modes.
• Sheridan, T. B. (2016). Human-robot interaction: status and challenges. Human Factors 58(4), 525-532. doi:10.1177/0018720816644364 Reviews human-robot interaction across four application areas and sets out the challenges, including authority allocation and transparency. Evidence of success was largely live demonstrations, with self-driving cars and humanoids at initial stages.
• Endsley, M. R. (2017). From here to autonomy: lessons learned from human-automation research. Human Factors 59(1), 5-27. doi:10.1177/0018720816681350 Reviews decades of human-automation research and names the automation conundrum, in which adding autonomy and improving reliability lowers the operator's situation awareness and the chance of a successful take-over. It proposes interventions at the level of automation and its granularity of control.
• Federal Aviation Administration. (2020). Airworthiness Directives; The Boeing Company Airplanes. 14 CFR Part 39, Docket No. FAA-2020-0686, Amendment 39-21332. https://www.faa.gov/sites/faa.gov/files/2022-08/737_AD_2019-NM-035fr.pdf States the unsafe condition as a single erroneously high angle-of-attack sensor input producing repeated nose-down trim, and requires software comparing both sensor inputs before the system can activate.
• U.S. Department of Transportation, Office of Inspector General. (2021). FAA oversight of Boeing 737 MAX certification timeline. Report AV2021021. https://www.oig.dot.gov/sites/default/files/FAA%20Oversight%20of%20Boeing%20737%20MAX%20Certification%20Timeline%20Final%20Report.pdf Records that the augmentation system relied on a single angle-of-attack sensor, that the Lion Air accident report has it activating 24 times after faulty data, and that the failure scenario was rated Major, so no redundancy was required.
• Romeo, G. and Conti, D. (2025). Exploring automation bias in human-AI collaboration: a review and implications for explainable AI. AI & Society 41(1), 259-278. doi:10.1007/s00146-025-02422-7 Reviewed 35 studies of automation bias under PRISMA 2020: the explanations offered as the remedy often left over-reliance intact and could strengthen misplaced trust, while expertise was more protective.