Conversational agents fail at repair, not at language

Dialogue is the only interface that can ask what it does not know. A conversation can turn a missing inference into a question. The deployed record does not show conversational agents using it. The mismatch between what people expect of a conversation and what the system can do ends interactions more often than recognition errors. The test that matters is whether the task survives a misunderstanding (Luger and Sellen, 2016; Clark et al., 2019).

A reader must hold two claims apart: whether the system heard and understood the request, and whether, given an imperfect hearing, the two can agree on what went wrong. The second is where the failures concentrate, so the design question is whether the interface lets the person see and correct the system's understanding. The ninth of ten units, this one locates the failure after the misunderstanding. The tenth unit asks which adaptation the field has actually measured.

The social script arrives before any competence is tested

Nass and Moon (2000) review experiments in which people mindlessly applied social rules and expectations to computers, showing politeness and reciprocity even knowing the machine was not a person. The script arrives with anything that speaks, so a conversational interface meets a contract it cannot honour.

One experiment is easy to follow. Participants worked with a computer and were then asked to evaluate it, either by that same computer, by a different one, or on paper, and evaluations were most positive when the computer asked about its own performance, the politeness people show to a person who asks about themselves. The result held although the participants later said a computer has no feelings and needs no courtesy. The case shows the social script is real and automatic. The study measured it with text-based machines, and it says nothing about whether an agent can repair a misunderstanding.

Developers built hundreds of thousands of task-focused chatbots after 2016, yet creating a successful non-trivial one proved harder than anticipated. Some developers now design for human-bot teams in which people handle the difficult queries, an admission that repair is not something the agent completes alone.

Cowan and colleagues (2017) studied people who tried personal assistants and stopped. Their concerns centred on social embarrassment, privacy, monetisation, permanency and transparency, the human metaphor inviting a comparison the systems lost.

The interface gives no way to tell what the system understood

Luger and Sellen (2016) interviewed fourteen users of conversational agents and found expectations dramatically out of step with the systems' intelligence, capability and goals. The failure they name is a gulf of evaluation: users could not judge how much the agent had understood, revised their model of it only through failure, and relied on visual confirmation for all but the simplest tasks.

Clark and colleagues (2019) put the deficit in the user's own words. Interviewing people about what a good conversation with an agent would require, they found participants described it in almost purely transactional terms. They treated common ground as a one-way product of personalisation rather than something built, checked and repaired together. The collaborative negotiation of meaning that human talk uses to fix a misunderstanding is neither expected of the agent nor supplied by it.

Repair is the mechanism, and the interface does not have it

Myers and colleagues (2018) observed twelve participants using a voice calendar over three sessions, coding obstacles and tactics: natural-language errors were the most frequent obstacle, other obstacles more likely to frustrate or confuse, and participants guessed rather than consult a visual aid, the signature of a missing repair mechanism.

Ashktorab and colleagues (2019) treat repair as a design object, comparing eight strategies with 203 participants. People most favoured offering options and explanations of what the system had understood. Deferring to a person suited failed repairs.

Moore and Arar (2019) turn that into a practitioner method. Their Natural Conversation Framework, adapted from conversation analysis, names repair sequences as patterns: agent repair, user repair and sequence abort are design objects.

The repair work does not disappear, it moves to the people nearby

Beneteau and colleagues (2019) placed an Echo Dot with ten families for four weeks and analysed 59 breakdown interactions. Every family hit breakdowns, and families collaborated to repair them through discourse scaffolding and speech adjustments. Porcheron and colleagues (2018) reached the same place from a month-long study of Echo deployments in five households: requests sit inside ongoing social activity, and people rephrased, repeated and altered prosody, reading a response or its silence as an account of system state that it rarely gives. When repair moves outside the system, an interaction can look successful in aggregate while the effort has moved onto the person. A person with a difficulty in hearing, though, has no equivalent route into the system.

Gao, Galley and Li (2019) survey neural approaches across question answering, task-oriented dialogue and open chat. Measured progress concentrates in tasks with a checkable answer, where success is scored against a target rather than against whether the exchange read well, and their review records automated overlap scores correlating weakly with human judgement on single responses, so a system can score well on the metric and still fail the user.

The record measures preference and fluency, not repair

Laranjo and colleagues (2018) systematically reviewed conversational agents in healthcare: seventeen articles covering fourteen agents met their criteria, most studies quasi-experimental and only two randomised controlled trials. The single efficacy trial, a two-week randomised comparison of 70 students against an information-only control, reported a significant reduction in depression symptoms (Fitzpatrick, Darcy and Vierhile, 2017); patient safety was rarely evaluated. Chaves and Gerosa (2020) surveyed 56 papers on chatbot social characteristics and found most results exploratory, with few testing how far a design strategy changes behaviour.

The contested question: repair deficit, expectation gap, or capability gap?

Three positions are available, each implying a different fix. The first holds that the constraint is the interaction mechanism: Luger and Sellen (2016), Myers and colleagues (2018) and Ashktorab and colleagues (2019) locate the failure in one place, the absence of any way to establish and correct what was understood, and Moore and Arar (2019) supply the method.

The second holds that the constraint is expectation. People bring a human conversational contract to a tool (Nass and Moon, 2000), and the metaphor of the assistant invites a comparison the system cannot win (Cowan et al., 2017; Grudin and Jacques, 2019), so the remedy is to change what the interface claims to be. Clark and colleagues (2019) support that diagnosis.

The third holds that the constraint is plain capability: Gao and colleagues (2019) document real, measured progress in the parts of dialogue with a checkable answer, so the repair deficit is a symptom of a system not yet able to understand enough to repair.

The disagreement is partly methodological, because the repair evidence is largely observational or preference-based and cannot separate mechanism from capability. What would settle it is a design that holds recognition accuracy fixed, varies only the repair support after a misunderstanding, and measures task completion rather than satisfaction in a deployment: if completion rises while accuracy stays unchanged, the mechanism position holds; if only accuracy predicts completion, the capability position does.

The mechanism position rests on interview studies (Luger and Sellen, 2016; Clark et al., 2019), an observational study of twelve participants (Myers et al., 2018) and deployments in real homes (Beneteau et al., 2019; Porcheron et al., 2018); the expectation position on laboratory experiments about social responses (Nass and Moon, 2000) and a qualitative study of non-use (Cowan et al., 2017); the capability position on a survey of measured progress on tasks with a checkable answer (Gao, Galley and Li, 2019). While the dispute is open, build a repair sequence and measure completion of a task. The strongest objection is that none of the repair evidence holds recognition accuracy fixed.

Where this is heading

The direction is towards making the misunderstanding visible and the recovery shared. Moore and Arar (2019) give the pattern language, and Ashktorab and colleagues (2019) show that users want the system to say what it understood and to offer options. Grudin and Jacques (2019) record an industry already designing for human-bot teams. Two questions are open. Do models now writing fluent text change the repair picture, or only its fluency side (Gao, Galley and Li, 2019)? One recent study suggests they change part of it: Mahmood and colleagues (2025) found that an LLM-powered voice assistant absorbed the majority of intent-recognition failures, addressing breakdowns earlier assistants could not, though the exploratory study of twenty participants measures no repair sequence. Do the repair patterns hold under deployment rather than scenario studies (Ashktorab et al., 2019; Chaves and Gerosa, 2020)?

Sources

• Nass, C. and Moon, Y. (2000). Machines and mindlessness: social responses to computers. Journal of Social Issues. doi:10.1111/0022-4537.00153 Reviews experiments in which people applied social rules, gender stereotypes, politeness and reciprocity to computers. Cited to show that the conversational contract arrives with any speaking system, before competence is tested.
• Luger, E. and Sellen, A. (2016). "Like having a really bad PA": the gulf between user expectation and experience of conversational agents. Proceedings of CHI 2016. doi:10.1145/2858036.2858288 Interviewed fourteen users of conversational agents. Expectations were dramatically out of step with the systems' capability, users could not judge what the agent had understood, and visual confirmation was needed on all but the simplest tasks.
• Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J. et al. (2019). What makes a good conversation? Challenges in designing truly conversational agents. Proceedings of CHI 2019. doi:10.1145/3290605.3300705 Ran semi-structured interviews on what people value in conversation and how it should appear in an agent. Participants described agent conversation in almost purely transactional terms and treated common ground as a one-way product of personalisation, not something negotiated.
• Gao, J., Galley, M. and Li, L. (2019). Neural approaches to conversational AI. Foundations and Trends in Information Retrieval. doi:10.1561/1500000074 Surveys neural approaches across question answering, task-oriented dialogue and social chat, with the evaluation practice for each. Progress concentrates in tasks with a checkable answer, and automated overlap metrics correlate weakly with human judgement, the counterweight against fluency as a measure.
• Ashktorab, Z., Jain, M., Liao, Q. V. and Weisz, J. D. (2019). Resilient chatbots: repair strategy preferences for conversational breakdowns. Proceedings of CHI 2019. doi:10.1145/3290605.3300484 Compared eight repair strategies with 203 participants. Offering options was the most favoured, explanations of the system's understanding were generally favoured, and deferring to a person suited failed repairs, which makes repair a design object with measurable preferences.
• Moore, R. J. and Arar, R. (2019). Conversational UX design: a practitioner's guide to the natural conversation framework. New York: Association for Computing Machinery. doi:10.1145/3304087 Presents the Natural Conversation Framework, adapting conversation analysis to interface design through an interaction model, a pattern language and named repair sequences. Supplies the constructive design method the argument implies.
• Myers, C., Furqan, A., Nebolsky, J., Caro, K. and Zhu, J. (2018). Patterns for how users overcome obstacles in voice user interfaces. Proceedings of CHI 2018. doi:10.1145/3173574.3173580 Analysed twelve participants using a voice calendar over three sessions, coding obstacles and the tactics used to overcome them. Natural-language errors were the most frequent obstacle but others were more likely to frustrate, and participants guessed rather than consult visual aids.
• Beneteau, E., Richards, O. K., Zhang, M., Kientz, J. A., Yip, J. and Hiniker, A. (2019). Communication breakdowns between families and Alexa. Proceedings of CHI 2019. doi:10.1145/3290605.3300473 Deployed an Echo Dot with ten families for four weeks and analysed 59 breakdowns. All ten families hit breakdowns, and the burden of repair rested with family members who scaffolded each other's attempts, showing the deficit in the home rather than the laboratory.
• Porcheron, M., Fischer, J. E., Reeves, S. and Sharples, S. (2018). Voice interfaces in everyday life. Proceedings of CHI 2018. doi:10.1145/3173574.3174214 Studied month-long Amazon Echo deployments in five households using conversation analysis. Requests are embedded in ongoing social activity, and people repaired troubles by rephrasing, repeating and altering prosody, reading the response or its absence as an account of system state.
• Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R. et al. (2018). Conversational agents in healthcare: a systematic review. Journal of the American Medical Informatics Association. doi:10.1093/jamia/ocy072 Screened 1513 citations and included seventeen articles covering fourteen agents. Most studies were quasi-experimental, only two were randomised controlled trials, the single efficacy trial reported an effect size of 0.44, and safety was rarely evaluated.
• Chaves, A. P. and Gerosa, M. A. (2020). How should my chatbot interact? A survey on social characteristics in human-chatbot interaction design. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2020.1841438 Surveyed 56 papers and derived eleven social characteristics of chatbots in three groups. Most results are exploratory, and few studies test how far a design strategy changes behaviour.
• Fitzpatrick, K. K., Darcy, A. and Vierhile, M. (2017). Delivering cognitive behaviour therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomised controlled trial. JMIR Mental Health. doi:10.2196/mental.7785 Randomised 70 students reporting symptoms of anxiety or depression to a two-week conversational self-help programme or an information-only control. It is the primary trial behind the effect the healthcare review reports.
• Cowan, B. R., Pantidi, N., Coyle, D., Morrissey, K., Clarke, P., Al-Shehri, S. et al. (2017). "What can I help you with?": infrequent users' experiences of intelligent personal assistants. Proceedings of MobileHCI 2017. doi:10.1145/3098279.3098539 Ran a qualitative study of people who had tried intelligent personal assistants and stopped. Non-use turned on social embarrassment, privacy, monetisation and transparency, and the human metaphor invited comparisons the systems lost.
• Grudin, J. and Jacques, R. (2019). Chatbots, humbots, and the quest for artificial general intelligence. Proceedings of CHI 2019. doi:10.1145/3290605.3300439 Analyses the conversational agent space after the 2016 chatbot boom. Hundreds of thousands of task-focused chatbots were built, non-trivial ones proved harder than anticipated, and some developers now design for human-bot teams with people handling difficult queries.
• Mahmood, A., Wang, J., Yao, B., Wang, D. and Huang, C.-M. (2025). User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants. International Journal of Human-Computer Studies. doi:10.1016/j.ijhcs.2024.103406 Had twenty participants use a ChatGPT-powered voice assistant across three scenarios and found that the language model absorbed the majority of intent-recognition failures, addressing some breakdowns earlier systems could not. An exploratory study that measures no repair sequence.