The user model is a defeasible inference, and it fails first

An adaptive system changes what it shows or does in response to the person using it. Every such change is a claim about that person. The system infers the claim from evidence it cannot check. The design question is not whether to model the user, but whether the person can inspect, correct and refuse the model that is inferred. A system that hides its model turns an ordinary inference error into a failure the person experiences as their own.

The first of ten units, this one sets the check the rest run. The second unit asks why the field that built the design taxonomy then stalled.

An inferred model is the price of adaptation

Fischer (2001) makes the necessity case directly. A system cannot tailor its behaviour to a person without a representation of that person, assembled from partial evidence. He reviews the kinds of user model human-computer interaction needs, holds that a system should support people in a task rather than do it for them, and treats the extent of exposure as a design problem rather than an implementation detail.

Rich (1979) had already shown what such a model costs. Grundy inferred a few reader stereotypes from a short interview and used them to suggest fiction, which showed that coarse group-level models acquired cheaply could support useful adaptation. Conflicts arose along those inferences, and the model changed as evidence arrived, the first statement of the defeasibility the argument rests on.

Brusilovsky and Millan (2007) supply the taxonomy of what an adaptation can be a claim about. They separate knowledge, goals, interests, preferences and traits, and name a technique for each. The taxonomy is not neutral here, because these claim types are not equally checkable. A person can usually report a goal, and can test a knowledge claim by attempting a task, but can rarely check an imputed trait at all.

The inference is only as good as the data behind it

Zukerman and Albrecht (2001) review statistical and machine-learning approaches to predictive user modelling and make a demand the field has to meet. They observe that the field judges predictive models by measures inherited from machine learning, such as recall, precision and accuracy. A system that acts on a user model has no accepted evaluation methodology of its own, so a good score on those measures does not settle whether the model is valid.

Webb, Pazzani and Billsus (2001) survey the algorithm families that acquire and update a model from interaction data. Performance turns on the quantity and quality of the observations available, and the obstacles that follow are the need for large data sets and for labelled examples, concept drift, and computational cost.

Carmichael, Kay and Kummerfeld (2005) give that problem a working form. Their approach keeps one user model across the devices and sensors of a ubiquitous environment, holding evidence by accretion and resolution so that conflicting observations of a person's location are reconciled rather than discarded. The person can inspect the model and how it is set. The case shows a wrong model is findable; it does not show the repaired one is right.

Scrutability answers a cost the field had already priced

Jameson (2007) reviews adaptive interfaces and agents and separates the layers at which adaptation can occur. He argues that whether adaptivity pays depends on the task and on the cost to the user of attending to the change: the attention needed to notice an adaptation is a second cost the design can raise or lower.

Kay (2006) states the position the argument needs. Adaptive systems should be scrutable rather than merely accurate, so that a person can inspect the model and the adaptation and correct them where they disagree. He frames this as a design principle rather than a result, holding that a system's inferences should be accountable to the person they are about.

Bull and Kay (2010) review open learner models, which display the learner's model to the learner. They set out what opening a model can achieve: reflection, self-monitoring and the negotiation of what the learner knows. They catalogue the design problems: what to show, how to present uncertainty, and what happens when the learner disagrees.

Amershi and colleagues (2019) turn the principle into a codified requirement. They propose 18 generally applicable guidelines for human-AI interaction, validated with 49 design practitioners who tested them against 20 popular AI-infused products. The guidelines that bear here ask a system to make clear what it can do and how well, and to provide a global control. The validation establishes relevance, not an improvement in any user outcome.

Shared control wins where it is built, and field interfaces rarely build it

Bunt, Conati and McGrenere (2007) supply the strongest experimental result here. They built a mixed-initiative framework that uses an online GOMS analysis to offer tailoring suggestions non-intrusively, leaving the user in control.

Kay and Kummerfeld (2012) report the drivers, principles and deployment experience behind personalisation people can scrutinise and control, drawing on the Personis and PebbleBed systems. They state which principles survived contact with users and which did not, and argue that scrutability is an engineering property of the architecture rather than a display decision.

Jannach, Naveed and Jugovac (2017) review user control in recommender systems and distinguish control over the profile, over the recommendation and over the interaction. They report that interfaces rarely implement the control designers say users should have. The exposure is often cosmetic, and an inspection view is itself an interface, so an exposed model still has to reach a person using a screen reader.

The contested question: is scrutability an obligation or a feature?

Two positions exist, and they disagree about what would count as a reason to build the control.

The first holds that inspection and correction are a design obligation owed to the person the model describes. Kay (2006) states it as a principle. Bull and Kay (2010) show what opening a model can do, and name where it is hard. Amershi and colleagues (2019) codify visible capability and a global control.

The second holds that scrutability is a feature whose cost a team must justify case by case, and that the evidence does not yet make it a requirement. Jameson (2007) shows that attending to an adaptation is a real cost, so an exposed model is not free. Jannach, Naveed and Jugovac (2017) show that field interfaces rarely implement the control in a form users can act on. Kay and Kummerfeld (2012) themselves report principles that did not survive contact with users.

What would settle it is a pair of studies. The first manipulates model visibility and measures recovery from a model error, with the model's state measured before and after correction. The second is deployment evidence that a correctable model produces fewer uncorrected errors than a hidden one. The sources here contain the design argument and one preference-and-performance experiment, but no study that measures the self-blame the thesis claims.

The two positions rest on different kinds of evidence. The obligation position rests on a design principle (Kay, 2006) and a review of the design space (Bull and Kay, 2010), whose one controlled experiment measured preference and performance rather than recovery from an error (Bunt, Conati and McGrenere, 2007); the feature position rests on a cost framework (Jameson, 2007) and a review of field interfaces. While the dispute is open, a designer should expose the model, offer correction and log the overrides. The strongest objection is that the thesis names a self-blame that no study here measured.

Where this is heading

The direction of travel is from displaying the model towards making it contestable in the architecture. The named work is Kay and Kummerfeld (2012) for the engineering form and Amershi and colleagues (2019) for the codified requirement. The open problem is the control gap that Jannach, Naveed and Jugovac (2017) document: the obligation is well argued and the interfaces do not build it. The newest form of the principle moves the model into language: Ramos and colleagues (2024) recommend from a natural language profile of a person's preferences that the person can read and edit, matching established recommenders in the warm start, though the profile was edited in code rather than by a user repairing the model.

Sources

• Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P. et al. (2019). Guidelines for Human-AI Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3290605.3300233 Proposed 18 design guidelines for human-AI interaction, validated with 49 practitioners who tested them against 20 AI-infused products, including that a system make clear what it can do and how well and offer a global control. The validation shows relevance rather than an effect on users.
• Brusilovsky, P. and Millan, E. (2007). User models for adaptive hypermedia and adaptive educational systems. The Adaptive Web: Adaptation Methods and Strategies, Lecture Notes in Computer Science. doi:10.1007/978-3-540-72079-9_1 Classified the models of adaptive hypermedia and adaptive educational systems, separating knowledge, goals, interests, preferences and traits with the techniques used for each. It is the standard account of what an adaptation can be a claim about, and its claim types are not equally checkable.
• Bull, S. and Kay, J. (2010). Open Learner Models. Studies in Computational Intelligence. doi:10.1007/978-3-642-14363-2_15 Reviewed open learner models, which display the learner's model to the learner, and set out what opening a model achieves: reflection, self-monitoring and the negotiation of what the learner knows. They catalogue the design problems of what to show, how to present uncertainty, and what happens when the learner disagrees.
• Bunt, A., Conati, C. and McGrenere, J. (2007). Supporting interface customization using a mixed-initiative approach. Proceedings of the 12th international conference on Intelligent User Interfaces. doi:10.1145/1216295.1216317 Built a mixed-initiative framework that uses an online GOMS analysis to offer tailoring suggestions non-intrusively, leaving the user in control. A user evaluation found the approach was preferred to a purely adaptable alternative and that its suggestions helped task performance. It is the unit's controlled evidence for shared control.
• Carmichael, D. J., Kay, J. and Kummerfeld, B. (2005). Consistent Modelling of Users, Devices and Sensors in a Ubiquitous Computing Environment. User Modelling and User-Adapted Interaction. doi:10.1007/s11257-005-0001-z Kept one user model across devices and sensors by accreting evidence and resolving conflicts, and made the model available for scrutiny.
• Fischer, G. (2001). User Modelling in Human-Computer Interaction. User Modelling and User-Adapted Interaction. doi:10.1023/A:1011145532042 Argued that a system cannot tailor its behaviour without a representation of the user, reviewed the kinds of model human-computer interaction needs, and set the requirement that a system support people in doing a task rather than doing it for them. The essay states the necessity and the fallibility of the model in one argument.
• Jameson, A. (2007). Adaptive Interfaces and Agents. The Human-Computer Interaction Handbook. doi:10.1201/9781410615862.ch22 Reviewed adaptive interfaces and agents, separated the layers at which adaptation can occur, and argued that whether adaptivity pays depends on the task and on the cost to the user of attending to the change. The chapter is a cost accounting, and it supplies the framework in which attention to an adaptation is itself a cost.
• Jannach, D., Naveed, S. and Jugovac, M. (2017). User Control in Recommender Systems: Overview and Interaction Challenges. Lecture Notes in Business Information Processing. doi:10.1007/978-3-319-53676-7_2 Reviewed user control in recommender systems, distinguishing control over the profile, the recommendation and the interaction, and reported that interfaces rarely implement the control designers say users should have. Control is easy to promise and hard to deliver.
• Kay, J. (2006). Scrutable Adaptation: Because We Can and Must. Lecture Notes in Computer Science. doi:10.1007/11768012_2 Argued that adaptive systems should be scrutable rather than merely accurate, so that a person can inspect the model and the adaptation and correct them where they disagree. The argument is a design principle rather than a result, holding that a system's inferences should be accountable to the person they are about.
• Kay, J. and Kummerfeld, B. (2012). Creating personalized systems that people can scrutinize and control. ACM Transactions on Interactive Intelligent Systems. doi:10.1145/2395123.2395129 Reported the drivers, principles and deployment experience behind personalisation people can scrutinise and control, drawing on the Personis and PebbleBed systems. They state which principles survived contact with users and which did not, and argue that scrutability is an engineering property of the architecture rather than a display decision.
• Ramos, J., Rahmani, H. A., Wang, X., Fu, X. and Lipani, A. (2024). Transparent and Scrutable Recommendations Using Natural Language User Profiles. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2024.acl-long.753 Had a language model write a natural language profile of a person's preferences and recommend from it, matching established recommenders in the warm start, with the recommendations changing after the profile was edited. The scrutability it demonstrates is tested by programmatic edits rather than by a user repairing the model.
• Rich, E. (1979). User modelling via stereotypes. Cognitive Science. doi:10.1016/S0364-0213(79)80012-9 Built Grundy, which inferred a small number of reader stereotypes from a short interview and used them to suggest fiction, showing that coarse group-level models acquired cheaply could support useful adaptation. It records that uncertain knowledge had to enter the model for the inference to be fast, and that the model was revised as evidence arrived.
• Webb, G. I., Pazzani, M. J. and Billsus, D. (2001). Machine learning for user modelling. User Modelling and User-Adapted Interaction. doi:10.1023/A:1011117102175 Surveyed machine learning for user modelling, describing the main algorithm families and how they acquire and update a model from interaction data. They emphasised that performance depends on the quantity and quality of the observations available and named the obstacles that follow: the need for large data sets and labelled examples, concept drift, and computational cost.
• Zukerman, I. and Albrecht, D. W. (2001). Predictive Statistical Models for User Modelling. User Modelling and User-Adapted Interaction. doi:10.1023/A:1011175525451 Reviewed statistical and machine-learning approaches to predictive user modelling and observed that predictive user models are judged by measures inherited from machine learning, such as recall, precision and accuracy, and that a system which acts on a user model has no accepted evaluation methodology of its own. A good predictive score therefore does not settle whether the model is valid.