Trust is not the target: overreliance, automation bias and skill decay

A model produces a score, and a product turns it into advice. The question is not whether people trust that advice but whether they rely on it in proportion to how often it is right. This fourth of the ten units takes the person's reliance. Trust is the wrong design target. A highly reliable aid can make a person's monitoring worse. The interventions that raise trust and those that raise appropriate reliance are different, and they can work in opposite directions.

Reliance fails in two opposite directions

Lee and See (2004) reviewed trust in automation across organisational, sociological, interpersonal, psychological and neurological perspectives. People fail to rely on automation appropriately, and trust guides reliance precisely where complexity makes complete understanding impractical. The goal is therefore the appropriateness of trust rather than its quantity.

Parasuraman and Riley (1997) name the two directions this goes wrong in. Misuse is over-reliance that degrades monitoring or biases decisions. Disuse is the neglect of automation. Abuse, their fourth category, is the automation of functions without regard for the operator's performance. Gaube and colleagues (2021) showed physicians chest radiographs with diagnostic advice. Some of the advice was wrong. Human experts had written all of it, and the study presented it either as coming from an AI system or from a colleague. Radiologists rated the AI-labelled advice as lower quality, while less expert physicians did not. Inaccurate advice degraded accuracy whatever source it was said to come from, so discounting the label did not protect against the content.

The aid changes which errors a person makes

Skitka, Mosier and Burdick (1999) ran a simulated flight task in which participants monitored system states with and without a computer that made recommendations from the same states. Participants without the aid outperformed those with a very but not perfectly reliable aid on the monitoring task. The aided made errors of omission, missing events the aid did not prompt. They also made errors of commission, following the aid against their training and other valid indicators.

Parasuraman and Manzey (2010) integrated complacency and automation bias into one attentional account. Complacency arises under multiple-task load. Automation bias is a decision-level effect producing omission and commission errors when aids are imperfect. Both appear in naive and expert participants, and neither is removed by training or instructions, so better training is an incomplete remedy.

Automation leaves the operator its hardest residual tasks

Bainbridge (1983) set out the argument that survives the change of technology. Automating a process may expand rather than eliminate problems for the operator. The operator is left the tasks the designer cannot think how to automate, and responsibility for abnormal conditions. Two of her claims matter here. Attention to a source on which very little happens cannot be sustained for more than about half an hour. Monitoring for rare failures is therefore close to impossible for a person. Physical skills deteriorate when they are not used, so a long-serving monitor may take over as an inexperienced one.

Trust and accuracy come apart, so reliance must be counted twice

Dzindolet and colleagues (2003) ran three studies in which students searched terrain images for a camouflaged soldier with an automated aid. Participants first found the aid trustworthy. They then distrusted even reliable aids after seeing errors, unless they were told why the aid might err. That explanation increased reliance even where the trust was unwarranted.

Zhang, Liao and Bellamy (2020) found that showing a model's confidence score helped participants calibrate their trust. Calibration alone did not improve joint accuracy, which also depended on whether people could bring knowledge that covered the model's errors.

In one study a display raised trust and reliance where the aid had not earned it. In the other it improved calibration without improving the decision.

Interventions that work are not trust interventions

Bucinca, Malaya and Gajos (2021) designed three cognitive forcing interventions, informed by dual-process theory, that require a person to engage with a recommendation before acting. In an experiment with 199 participants these designs reduced over-reliance compared with explanation-based approaches and a no-AI baseline. The lever was the demand the interface placed on the person, and it was paid for in satisfaction.

Vasconcelos and colleagues (2023) formalise the choice to engage with an explanation as a comparison of costs and benefits. Across five studies with 731 participants, task difficulty, explanation difficulty and monetary compensation all moved over-reliance. A null result for an explanation may therefore reflect the effort it demands rather than a fixed human bias.

Schemmer and colleagues (2023) propose appropriateness of reliance as a two-dimensional construct. They measure it as following correct advice and as correctly overriding incorrect advice, and they test explanations with 200 participants. Explanations raised the first and left the second unchanged, and they did not raise trust, so a single reliance score would have hidden the effect.

Contested question: mitigate the bias, or change what is being compared?

The literature disagrees about where the defect lies, and therefore about what a designer should do.

The first position holds that over-reliance is a robust property of human decision making under time and effort costs. The remedy it proposes is an interface that forces engagement. Its evidence is the laboratory record. Cognitive forcing reduced over-reliance in 199 participants (Bucinca, Malaya and Gajos, 2021). Effort and benefit moved it in 731 (Vasconcelos et al., 2023).

The second position holds that over-reliance is not the main effect to design around. The quality of the support and the expertise of the user decide whether a person ends up better off. Tschandl and colleagues (2020) varied how AI-based support was represented across levels of clinical expertise and several workflows. Good quality support improved diagnostic accuracy over either the AI or the physicians alone, with the least experienced clinicians gaining most. Faulty AI misled experts as well as non-experts. A mitigation aimed at the average user therefore misses the population that gains most.

The third position rejects the frame. Bansal and colleagues (2021) argue that the most accurate model is not necessarily the best teammate. A person who oversees a recommendation chooses between accepting it and solving the task themselves. Training the model for team utility rather than standalone accuracy improved expected team utility across real-world datasets. They argue that predictable performance may be worth a slight sacrifice in accuracy. If team outcome is the objective, the model's training objective rather than the interface is where change belongs.

What would settle the disagreement is a comparison that none of the three runs. The model stays fixed while the interface varies, and joint decision quality rather than trust is the outcome. The study reports both dimensions of reliance separately (Schemmer et al., 2023). It would run in the population the deployment will serve, because Tschandl and colleagues (2020) show the gain is concentrated in the least expert. It would also train the model for the team objective (Bansal et al., 2021).

What to do while the dispute is open

The three positions carry different evidence. The claim that over-reliance is a robust bias rests on laboratory experiments with 199 and 731 participants. The claim that support quality and expertise decide the outcome rests on one clinical study run across levels of expertise. The claim that the team objective belongs in the model rests on experiments over datasets rather than over people. The first two are experiments with participants, and the third is a modelling proposal. The strongest objection to this unit's thesis is that it sets aside the one construct with a developed theory and a validated account of calibration. Lee and See (2004) make trust measurable in a way that a two-dimensional reliance score has only recently become. While the dispute is open, a designer should price the intervention, because the designs that reduced over-reliance most also drew the least favourable subjective ratings. It should also check that a user reading in a second language, or an older user, can meet the demand it adds.

Where the argument is heading

The direction of travel is away from trust ratings and towards measured reliance and joint performance. Schemmer and colleagues (2023) make the outcome two-dimensional. Bucinca, Malaya and Gajos (2021) and Vasconcelos and colleagues (2023) shift the question from whether explanation works to what engagement costs. Bansal and colleagues (2021) move the optimisation into the model, and Tschandl and colleagues (2020) require effects to be reported per level of expertise.

The sources this unit rests on

• Bainbridge, L. (1983). Ironies of automation. Automatica 19(6), 775-779. doi:10.1016/0005-1098(83)90046-8 The paper argues that automating a process may expand rather than eliminate problems for the operator, who is left an arbitrary collection of residual tasks. It shows that vigilance to a source on which little happens cannot be sustained for more than about half an hour, and that unused skills deteriorate, so a long-serving monitor may take over as an inexperienced operator.
• Bansal, G., Nushi, B., Kamar, E., et al. (2021). Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork. Proceedings of the AAAI Conference on Artificial Intelligence 35(13), 11405-11414. doi:10.1609/aaai.v35i13.17359 The authors train models for the expected utility of the human-AI team rather than for standalone accuracy, in a setting where a person chooses between accepting a recommendation and solving the task. The most accurate model was not the best teammate, and training for team utility improved expected team utility across real-world datasets.
• Bucinca, Z., Malaya, M. B. and Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proceedings of the ACM on Human-Computer Interaction 5(CSCW1), 1-21. doi:10.1145/3449287 In an experiment with 199 participants, three designs that required engagement before acting reduced over-reliance compared with explanation-based designs. The designs that reduced it most drew the least favourable subjective ratings, and the benefit was uneven across participants' need for cognition.
• Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., et al. (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies 58(6), 697-718. doi:10.1016/s1071-5819(03)00038-7 Three studies had students search terrain images for a camouflaged soldier with an automated aid. Participants distrusted even reliable aids after seeing errors, and an explanation of why the aid might err restored reliance even where that reliance was unwarranted.
• Gaube, S., Suresh, H., Raue, M., et al. (2021). Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digital Medicine. doi:10.1038/s41746-021-00385-9 Physicians received chest radiographs with diagnostic advice, some of it wrong, all written by human experts but presented as coming from an AI system or a colleague. Radiologists rated AI-labelled advice lower, and inaccurate advice degraded accuracy whatever its stated source.
• Lee, J. D. and See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors 46(1), 50-80. doi:10.1518/hfes.46.1.50.30392 This review draws on several disciplinary traditions and concludes that the design goal is appropriate reliance, because trust guides reliance exactly where complexity makes complete understanding impractical. Its model ties trust to the context of use and to display characteristics, which is the unit's reason to treat calibration as a design output.
• Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3), 381-410. doi:10.1177/0018720810376055 The review integrates complacency and automation bias into one attentional model: complacency occurs under multiple-task load when manual tasks compete with the automated one for attention, and automation bias produces omission and commission errors when aids are imperfect. Both appear in naive and expert participants, and neither is removed by training or instructions.
• Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors 39(2), 230-253. doi:10.1518/001872097778543886 The taxonomy separates two opposite failures: misuse is over-reliance that degrades monitoring or biases decisions, and disuse is the neglect of automation, commonly caused by alarms that fire falsely because the base rate was ignored. Use depends on trust, workload and risk with large individual differences, and abuse is the automation of functions without regard for the operator.
• Schemmer, M., Kuehl, N., Benz, C., et al. (2023). Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. Proceedings of the 28th International Conference on Intelligent User Interfaces, 410-422. doi:10.1145/3581641.3584066 The authors define appropriateness of reliance as a two-dimensional construct whose components are following correct advice and correctly overriding incorrect advice, and test explanations with 200 participants. Explanations increased the first and left the second unchanged, and they did not increase trust, which a single reliance score would conceal.
• Skitka, L. J., Mosier, K. L. and Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies 51(5), 991-1006. doi:10.1006/ijhc.1999.0252 In a simulated flight task with a very but not perfectly reliable aid, participants with it performed worse on the monitoring task than those without. They made errors of omission, missing events the aid did not flag, and errors of commission, following the aid against their training and other valid indicators.
• Tschandl, P., Rinner, C., Apalla, Z., et al. (2020). Human-computer collaboration for skin cancer recognition. Nature Medicine 26(8), 1229-1234. doi:10.1038/s41591-020-0942-0 The study varied how AI-based support was represented across levels of clinical expertise and several diagnostic workflows. Good quality support improved diagnostic accuracy over either the AI or the physicians alone, with the least experienced clinicians gaining most, and faulty AI misled experts as well as non-experts.
• Vasconcelos, H., Jorke, M., Grunde-McLaughlin, M., et al. (2023). Explanations Can Reduce Overreliance on AI Systems During Decision-Making. Proceedings of the ACM on Human-Computer Interaction 7(CSCW1), 1-38. doi:10.1145/3579605 The authors model the choice to engage with an explanation as a comparison of costs and benefits, and manipulate both in a maze task across five studies with 731 participants. Over-reliance moved with task difficulty, explanation difficulty and monetary compensation, so a null result for an explanation may reflect the effort it demands rather than a fixed human bias.
• Zhang, Y., Liao, Q. V. and Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 295-305. doi:10.1145/3351095.3372852 In two experiments, showing the model's confidence score helped participants calibrate their trust, but calibration alone did not improve joint accuracy. Improvement also depended on whether people could bring knowledge covering the model's errors, which makes joint performance the outcome to measure.