Generative models as interaction partners: the frontier and its evidence problem

The previous unit asked what makes a review last; this one takes one frontier apart. The claim is that generative models have become interaction partners: systems a person prompts and corrects rather than a tool a person operates. This unit argues that the frontier has changed what interaction researchers build faster than it has produced evidence about whether the results help; the field's guidelines and design principles are syntheses of practice, not tests of outcomes. As Weisz and colleagues describe it, the user specifies the outcome and the model decides how to produce it, moving control over computation from the user to the model (Weisz et al., 2024).

What the field wrote down: guidance as a synthesis of practice

Amershi and colleagues set out eighteen guidelines for human-AI interaction validated by 49 design practitioners against 20 popular AI-infused products (Amershi et al., 2019). Weisz and colleagues then built six principles for generative applications, three reinterpreting known issues and three naming considerations new to generative systems. Their sources were a literature review, practitioner feedback and validation against existing applications (Weisz et al., 2024).

Both are distillations: the evidence for a guideline is that practitioners could apply it, which shows usefulness as guidance, not that an application built to it helps its user. Neither study compared applications that followed the guidance with applications that did not, on a measured outcome. Weisz and colleagues offer their principles to inform design rather than as tested claims, and that honesty is the hinge of the argument.

The frontier inherited a problem of initiative

The design problem is older than the generative model. Horvitz set out principles for mixed-initiative interfaces, in which both user and system may take initiative, using a decision-theoretic model of the cost of interrupting the user to decide when to act (Horvitz, 1999). Luger and Sellen (2016) interviewed fourteen users of the everyday conversational agents then shipping on phones. Their expectation of a competent assistant was out of step with systems that lost the thread and demanded constant repair. Read through the gulfs of execution and evaluation that Norman (1986) sets out, much of the disappointment sits in the interaction design rather than the language technology.

Miller makes the point the generative literature had to relearn: reviewing how people explain in the social sciences, he argued that explainable AI had largely ignored it (Miller, 2019).

Where the measured effects are, and why they do not settle the question

Bansal and colleagues asked whether explanations let a person and a model reach a combined accuracy above either alone. Across three datasets, augmentation raised complementary performance but explanations did not. Explanations raised instead the chance the person would accept the recommendation whether or not it was correct (Bansal et al., 2021).

Bucinca, Malaya and Gajos tested the intervention that follows. They built cognitive forcing functions, which require a person to think before accepting a recommendation. They compared them with two explainable-AI approaches and a no-AI baseline in an experiment with 199 participants. Forcing functions significantly reduced overreliance, but at a cost. The designs that reduced it most drew the least favourable ratings, and the benefit fell more on participants higher in need for cognition (Bucinca et al., 2021).

Dell'Acqua and colleagues ran a preregistered field experiment with 758 knowledge workers on 18 consulting tasks. Inside the model's capabilities, workers with access completed 12.2 per cent more tasks and 25.1 per cent faster. On one task outside them, they were 19 percentage points less likely to be correct (Dell'Acqua et al., 2023). They named the boundary a jagged frontier. That shape, not the average gain, is the finding to carry into design: assistance helps where the model is strong and harms where it is weak. Brynjolfsson, Li and Raymond followed the staged rollout of a conversational assistant to 5,179 customer-support agents: access raised issues resolved per hour by about 14 per cent, with the gain concentrated among novice and lower-skilled agents, at about 34 per cent, and little change for the experienced (Brynjolfsson et al., 2025). Assistance substitutes for experience where experience is thinnest.

The frontier's instruments are themselves untested

The instruments are new, which is part of why the outcome evidence is thin. Lee and colleagues surveyed 319 knowledge workers and collected 936 first-hand examples of using generative AI at work. They found that higher confidence in the AI was associated with less critical thinking, while higher self-confidence was associated with more (Lee et al., 2025). The study is self-reported and cross-sectional, so it shows an association and not a causal direction.

Zamfirescu-Pereira and colleagues studied how ten non-experts designed prompts. The participants treated prompts as human-to-human instructions and repeatedly failed to anticipate the model's behaviour (Zamfirescu-Pereira et al., 2023). Tabone and de Winter turned the new instrument on the researcher, applying ChatGPT to questionnaire responses, interviews and think-aloud transcripts. They found it usable for some text-analysis tasks and unsuitable for others. Its validity as a research instrument must be established task by task (Tabone and de Winter, 2023).

Two critiques mark where the scale becomes a risk

Two critiques mark the boundary of enthusiasm. Bender and colleagues argued that large language models carry the risks of their size, including environmental cost, uncurated training data and the encoding of hegemonic views. They asked whether the models should be built at that scale at all (Bender et al., 2021). Weidinger and colleagues supplied a taxonomy of twenty-one risks in six areas, separating those observed from those anticipated (Weidinger et al., 2022). The training text is also English-dominant, so a user of a lower-resourced language, and a user who needs the output read aloud, are outside the evaluations the unit reports.

Contested question: is a design principle a form of evidence?

The literature disagrees about whether a design principle is evidence. The synthesis camp holds that it is. Amershi and colleagues, Weisz and colleagues and Shneiderman treat guidance built from literature, practice and validation as the right output for a field whose technology outruns its studies. They argue a principle is meant to guide design rather than to stand as a tested law (Amershi et al., 2019; Weisz et al., 2024; Shneiderman, 2022).

The evidence camp holds that a principle is a hypothesis about a benefit until a comparison shows the benefit. Explanations raised acceptance whether or not the model was right (Bansal et al., 2021). The designs that best reduced overreliance were the least liked (Bucinca et al., 2021). Assistance lowered accuracy in a measured range (Dell'Acqua et al., 2023).

What would settle it is a study the field has not run: compare applications that follow one principle with applications that do not, on a stated outcome for a stated user, fixed in advance. Until then, the field's most cited design documents are claims about practice with no outcome attached.

The strongest counter-argument comes from inside the evidence camp. Bansal and colleagues and Bucinca and colleagues each found a design that improved one measure while worsening another, so an intervention that harms on an unmeasured axis can pass a single outcome test (Bansal et al., 2021; Bucinca et al., 2021). If that is right, no single comparison settles a principle, and the synthesis camp gains ground. A principle may be the right unit, because it summarises many outcomes rather than predicting one. The reply is that a principle must still name the outcome it claims to improve and the cost it accepts: a synthesis that names neither cannot be contradicted, and a claim that cannot be contradicted cannot be learned from.

The two camps differ in what they measure. The synthesis camp rests on the recognition and use of guidance by practitioners, and one document was checked against twenty products. The evidence camp rests on three outcome studies, the largest a preregistered field experiment with 758 knowledge workers. Both are empirical; only the second measures whether a person was helped. While the dispute is open, a designer should choose the principle whose outcome and accepted cost are both named, and test the benefit where the model is weakest.

Where the argument is heading

Lee and colleagues show the field asking what the tools do to the user's thinking (Lee et al., 2025); Dell'Acqua and colleagues show a field experiment whose headline is a boundary rather than an average (Dell'Acqua et al., 2023); Bucinca and colleagues show a design intervention tested against a no-AI baseline (Bucinca et al., 2021). The evidence problem will be answered the way the field's others were, with a named outcome, a comparison group and a result that can disappoint the authors.

The sources this unit rests on

Each note states what the cited document did and found. Every DOI was resolved and its record read on 1 November 2026.

• Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/302979.303030 Sets out principles for interfaces in which both user and system may take initiative, using a decision-theoretic model of the cost of interrupting the user to decide when to act.
• Miller, T. (2019). Explanation in artificial intelligence: insights from the social sciences. Artificial Intelligence. doi:10.1016/j.artint.2018.07.007 Surveys how people explain in the social sciences, argues that explainable AI has largely ignored those findings, and draws out what they imply for building usable explanations.
• Amershi, S., Weld, D., Vorvoreanu, M., et al. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3290605.3300233 Present eighteen guidelines for human-AI interaction, validated by 49 design practitioners against 20 popular AI-infused products, but against no comparison application and on no measured outcome.
• Weisz, J. D., He, J., Muller, M., et al. (2024). Design principles for generative AI applications. Proceedings of the CHI Conference on Human Factors in Computing Systems. doi:10.1145/3613904.3642466 Derive six principles for generative AI applications from a literature review, practitioner feedback, validation against existing applications and use in two applications. They test no outcomes.
• Shneiderman, B. (2022). Human-centred AI. Oxford University Press. doi:10.1093/oso/9780192845290.001.0001 Argues for human-centred AI organised around high-level human control and systems designed to be reliable, safe and trustworthy. A position rather than a study.
• Bansal, G., Wu, T., Zhou, J., et al. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3411764.3445717 Ran mixed-method studies on three datasets: augmentation raised complementary performance but explanations did not, raising instead the chance a person would accept a recommendation whether or not it was correct.
• Bucinca, Z., Malaya, M. B. and Gajos, K. Z. (2021). To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction. doi:10.1145/3449287 Compared three cognitive forcing designs with two explainable-AI approaches and a no-AI baseline in an experiment with 199 participants; forcing significantly reduced overreliance, while the designs that reduced it most drew the least favourable subjective ratings.
• Dell'Acqua, F., McFowland III, E., Mollick, E. R., et al. (2023). Navigating the jagged technological frontier: field experimental evidence of AI effects on knowledge worker productivity and quality. Harvard Business School Working Paper. doi:10.2139/ssrn.4573321 Ran a preregistered field experiment with 758 knowledge workers on 18 consulting tasks; inside the model's capabilities, workers with access completed 12.2 per cent more tasks and 25.1 per cent faster, but on one task outside were 19 percentage points less likely to be correct.
• Lee, H. P., Sarkar, A., Tankelevitch, L., et al. (2025). The impact of generative AI on critical thinking: self-reported reductions in cognitive effort and confidence effects. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3706598.3713778 Surveyed 319 knowledge workers on 936 examples of generative AI use at work: higher confidence in the AI was associated with less critical thinking, higher self-confidence with more.
• Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., et al. (2023). Why Johnny can't prompt: how non-AI experts try (and fail) to design LLM prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3544548.3581388 Studied how ten non-experts tried to design prompts and found that they treated prompts as human-to-human instructions and repeatedly failed to anticipate the model's behaviour.
• Tabone, W. and de Winter, J. (2023). Using ChatGPT for human-computer interaction research: a primer. Royal Society Open Science. doi:10.1098/rsos.231053 Applied ChatGPT to questionnaire responses, interview data and think-aloud transcripts, comparing its output with human coding: usable for some text-analysis tasks, unsuitable for others.
• Bender, E. M., Gebru, T., McMillan-Major, A., et al. (2021). On the dangers of stochastic parrots: can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3442188.3445922 Argue that large language models carry the risks of their size, including environmental cost, uncurated training data and encoded hegemonic views, and question whether they should be built at that scale.
• Weidinger, L., Uesato, J., Rauh, M., et al. (2022). Taxonomy of risks posed by language models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3531146.3533088 Build a taxonomy of twenty-one risks in six areas, from discrimination and information hazards to human-computer interaction harms, separating risks already observed from risks anticipated.
• Luger, E. and Sellen, A. (2016). Like having a really bad PA: the gulf between user expectation and experience of conversational agents. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. doi:10.1145/2858036.2858288 Interviewed fourteen users of everyday conversational agents and found their expectation of a competent assistant out of step with systems that lost the thread and needed constant repair.
• Brynjolfsson, E., Li, D. and Raymond, L. R. (2025). Generative AI at work. The Quarterly Journal of Economics. doi:10.1093/qje/qjae044 Followed the staged rollout of a conversational assistant to 5,179 customer-support agents: access raised issues resolved per hour by about 14 per cent, with about 34 per cent gains for novice and lower-skilled agents and little change for the experienced. A production deployment for the unit's evidence.
• Norman, D. A. (1986). Cognitive engineering. In D. A. Norman and S. W. Draper (Eds.), User Centred System Design: New Perspectives on Human-Computer Interaction (pp. 31-62). doi:10.1201/9780367807320-3 Sets out cognitive engineering as applied cognitive science and introduces the gulfs of execution and evaluation. The primary source for the frame this unit reads the conversational-agent failures through.