The generative turn and the field's evidence problem

Generative systems changed what a person does at an interface. Instead of naming each step, the user states the outcome wanted and the machine produces it (Weisz and colleagues, 2024). This unit argues that the field adopted the change as a paradigm before it had the apparatus to test what the change does. It also argues that the field's own audit of its literature makes the gap visible.

The paradigm arrived as a proposal, not as a measured effect

Weisz, He, Muller, Hoefer, Miles and Geyer (2024) set out the case for a new paradigm and supplied its design guidance. They argued that generative systems shift control from naming each step to stating the outcome wanted. They proposed six principles, three new for generative applications and three reinterpretations of known artificial intelligence design problems. Those principles came from a literature review, practitioner feedback and two projects. The authors call the result a proposal rather than a measured effect.

That guidance had a predecessor. Amershi and colleagues (2019) derived eighteen guidelines for human-artificial intelligence interaction from practitioner guidance and validated them with designers who applied them to example interfaces. Guidance for generative interaction therefore predates the generative turn. That dates the paradigm by what systems do rather than by principles.

The measured effects are real and narrow

Noy and Zhang (2023) ran a preregistered experiment in which 453 college-educated professionals were given occupation-specific, incentivised writing tasks, and half were randomly given an assistant. Average time taken fell by about forty per cent, and rated output quality rose by about eighteen per cent. The spread between workers narrowed, and exposure roughly doubled reported use of such a tool in the real job two weeks later.

Dell'Acqua and colleagues (2026) found the boundary of that effect. In a preregistered field experiment with 758 consultants, access to a generative model produced more tasks completed and higher-rated work inside the model's capability. On a task just outside it, the same access produced worse performance than controls. The authors attributed the loss to accepting plausible but wrong output. The two studies support a claim about a class of writing and reasoning tasks, not about generative tools.

The failure at that boundary has been recorded outside the laboratory. In 2023 two lawyers filed a brief in a New York personal injury case citing six judicial opinions that a generative assistant had produced. The opinions did not exist, and the lawyers stood by them after the court called their existence into question. The judge sanctioned the two lawyers and their firm (Mata v. Avianca, 2023). Fluent, well formed and internally cited output was taken as evidence that a source existed.

What the output measures leave out

Wadinambiarachchi, Kelly, Pareek, Zhou and Velloso (2024) ran a between-participants experiment with sixty participants on a visual ideation task. Those given an artificial intelligence image generator showed higher fixation on an initial example and produced fewer ideas, with less variety and lower originality, than a baseline group.

Barke, James and Polikarpova (2023) supply the mechanism in skilled work. Studying programmers with a code-generating assistant, they described acceleration when the intent was known and exploration when it was not. They found that users spent much of their time verifying and redirecting proposals, so verification became part of the work. A stopwatch cannot show whether removing typing outweighed adding verification.

Lee and colleagues (2025) surveyed three hundred and nineteen knowledge workers describing nine hundred and thirty-six first-hand uses. They found that confidence in the tool predicted less critical thinking, while confidence in the worker's own skill predicted more. Kosmyna and colleagues (2025) recorded electroencephalography as fifty-four participants wrote essays with a language model, a search engine or no tool. They reported the strongest connectivity with no tool and the weakest with the language model, which they read as under-engagement and call cognitive debt.

The audit that makes the gap visible

Cockburn, Dragicevic, Besancon and Gutwin (2020) examined publication and statistical practice in empirical computer science. They found low statistical power, selective reporting and a strong bias towards positive results, which together make many published effects unreliable. Wobbrock and Kientz (2016) set out seven types of contribution. They argued that a paper should state which type it makes, so it is judged by the fitting standard.

The principles of Weisz and colleagues (2024) are a design proposal, judged by whether they help a designer. The electroencephalography study of Kosmyna and colleagues (2025) is an empirical claim, and the audit is the standard for a fifty-four-participant effect.

Noy and Zhang (2023) supply power, randomisation and a task with stakes. Nothing in the unit's generative-interaction literature supplies the equivalent for its claims about the design process, reasoning or attention.

The critique the design guidance does not reach

Bender, Gebru, McMillan-Major and Shmitchell (2021) argued that large language models concentrate environmental and financial costs. The models also encode hegemonic views through their training data and produce fluent text without meaning, and documentation alone cannot address these harms. None of this is a property of an interaction. A design-led paradigm can shape what a user does with the output. It cannot decide who pays for it, whose language it repeats, or whether the language a user writes in was in the training data at all.

The concentration is measurable. Strubell, Ganesh and McCallum (2019) estimated the energy and emissions of training several language models and set them beside familiar quantities. A car's lifetime emissions are about 126,000 pounds of carbon dioxide equivalent, and the training of one model with architecture search and tuning cost about 626,155 pounds. The number belongs to whichever laboratory can pay it, to the grid that supplied the electricity, and to every user who then pays per query on a metered connection.

Shneiderman (2022) is the constructive counter-position. He sets human control and automation both high rather than trading one against the other, and he aims at supertools and telepresence rather than autonomous agents that remove the person.

Contested question: is the gap in method or in category?

The literature disagrees about what kind of deficit the generative turn has. One position is methodological, held by Cockburn and colleagues (2020): the field's empirical practice is underpowered, selectively reported and biased towards positive results, and preregistration, adequate samples and replication would repair it. Noy and Zhang (2023) and Dell'Acqua and colleagues (2026) show the apparatus working when it is applied.

The other position is categorical. Weisz and colleagues (2024) defend their principles through practitioner use rather than measurement, and Wobbrock and Kientz (2016) argue that an artefactual contribution should be judged by its own standard. On this reading, demanding measurement of the paradigm misclassifies its claims. The evidence for each reading is uneven. Noy and Zhang (2023) supply a preregistered experiment with 453 participants and a two-week follow-up, the strongest measurement here, and Dell'Acqua and colleagues (2026) a field experiment with a boundary. Weisz and colleagues (2024), on the other side, rest on practitioner use and two projects, and Kosmyna and colleagues (2025) is a preprint with small late-session groups.

What would settle it is work neither side has produced. State a generative-interaction claim as a falsifiable prediction: name the population and the task in advance, register the analysis before data are collected, and replicate with enough power to detect the asserted effect. If such predictions hold, the methodological reading is correct; if the claims survive only as guidance and never as predictions, the categorical reading is correct. The strongest counter to this unit's thesis is that the apparatus was not absent: Noy and Zhang (2023) ran the test the thesis asks for.

A second disagreement runs beside it. Shneiderman (2022) holds that the harms are addressable through design that keeps the person in control. Bender and colleagues (2021) hold that they are structural, located in cost, training data and fluent text without meaning. That dispute would be settled by outcome evidence on the groups the structural critique names.

While it is open, state the generative claim as a prediction the product can fail, and instrument the work so it can be checked. Log the verification the output triggers and the errors that survive it, since a time saving counts what left the task and not what replaced it.

Where the argument is heading

The next round is shaped by the papers that asked what the tool does to the person. Noy and Zhang (2023) supply a template a replication could carry outside writing. Lee and colleagues (2025) and Kosmyna and colleagues (2025) move measurement from throughput to reasoning and attention. Shneiderman (2022) supplies the constructive programme and Weisz and colleagues (2024) its first principles.

The sources this unit rests on

• Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R. and Geyer, W. (2024). Design principles for generative AI applications. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24), 1-22. doi:10.1145/3613904.3642466 Argues that generative systems shift control from naming each step to stating the outcome wanted, and proposes six design principles. They are presented as a proposal, not a measured effect.
• Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J. and Iqbal, S. (2019). Guidelines for human-AI interaction. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19), 1-13. doi:10.1145/3290605.3300233 Derived eighteen guidelines for human-artificial intelligence interaction from practitioner guidance and validated them with designers who applied them to example interfaces, reporting which were already followed.
• Shneiderman, B. (2022). Human-centred AI. Oxford University Press. doi:10.1093/oso/9780192845290.001.0001 Argues for setting human control and automation both high rather than trading one against the other, and proposes supertools and telepresence rather than autonomous agents that remove the person.
• Noy, S. and Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science 381(6654), 187-192. doi:10.1126/science.adh2586 Assigned incentivised writing tasks to four hundred and fifty-three college-educated professionals and randomly exposed half to a generative assistant: average time fell by about forty per cent, rated quality rose by about eighteen per cent, and the spread narrowed.
• Dell'Acqua, F., McFowland, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F. and Lakhani, K. R. (2026). Navigating the jagged technological frontier: field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality. Organisation Science (the journal's title uses the American spelling) 37(2), 403-423. doi:10.1287/orsc.2025.21838 A preregistered field experiment with 758 consultants in which access to a generative model produced more completed tasks and higher-rated work inside the model's capability but worse performance than controls just outside it. First circulated as a 2023 working paper.
• Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S. (2021). On the dangers of stochastic parrots: can language models be too big? Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT '21), 610-623. doi:10.1145/3442188.3445922 Argues that large language models concentrate environmental and financial costs, encode hegemonic views through their training data and produce fluent text without meaning, and that documentation alone cannot address these harms.
• Wadinambiarachchi, S., Kelly, R. M., Pareek, S., Zhou, Q. and Velloso, E. (2024). The effects of generative AI on design fixation and divergent thinking. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24), 1-18. doi:10.1145/3613904.3642919 A between-participants experiment with sixty participants on visual ideation in which access to an artificial intelligence image generator increased fixation on an initial example and produced fewer ideas, with less variety and lower originality, than a baseline.
• Barke, S., James, M. B. and Polikarpova, N. (2023). Grounded Copilot: how programmers interact with code-generating models. Proceedings of the ACM on Programming Languages 7(OOPSLA1), 85-111. doi:10.1145/3586030 Studied programmers with a code-generating assistant and described two modes of use, acceleration when the intent was known and exploration when it was not, finding that users spent much of their time verifying and redirecting proposals.
• Cockburn, A., Dragicevic, P., Besancon, L. and Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM 63(8), 70-79. doi:10.1145/3360311 Examined publication and statistical practice in empirical computer science and found low statistical power, selective reporting and a strong bias towards positive results, which make many published effects unreliable.
• Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I. and Maes, P. (2025). Your Brain on ChatGPT: accumulation of cognitive debt when using an AI assistant for essay writing task. arXiv preprint arXiv:2506.08872. https://arxiv.org/abs/2506.08872 Recorded electroencephalography as fifty-four participants wrote essays with a language model, a search engine or no tool, and reported the strongest connectivity with no tool and the weakest with the language model. It is a preprint with small late-session groups.
• Wobbrock, J. O. and Kientz, J. A. (2016). Research contributions in human-computer interaction. Interactions 23(3), 38-44. doi:10.1145/2907069 Sets out seven types of contribution that human-computer interaction work can make, from the empirical to the artefactual, and argues that a paper should state which type it makes so that it is judged by the appropriate standard.
• Lee, H. P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R. and Wilson, N. (2025). The impact of generative AI on critical thinking: self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '25), 1-22. doi:10.1145/3706598.3713778 Surveyed three hundred and nineteen knowledge workers describing nine hundred and thirty-six first-hand uses of a generative assistant and found that confidence in the tool predicted less critical thinking while confidence in the worker's own skill predicted more. It is self-report evidence about reasoning, not a measure of it.
• Strubell, E., Ganesh, A. and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645-3650. doi:10.18653/v1/P19-1355 Quantified the financial and environmental cost of training language models: a car's lifetime emissions at about 126,000 pounds of carbon dioxide equivalent, and one model with architecture search and tuning at about 626,155 pounds.
• Mata v. Avianca, Inc. (2023). Opinion and order on sanctions, 678 F. Supp. 3d 443 (S.D.N.Y. 22 June 2023), Castel J. https://national.clla.org/wp-content/uploads/2023/Eastern2023/PresentationTwo/02-Mata-v.-Avianca-Opinion-and-Order-on-Sanctions.pdf The court found that two lawyers filed six non-existent judicial opinions produced by a generative assistant with fabricated quotations and citations, stood by them after being questioned, and sanctioned the lawyers and their firm.