Designing and evaluating adaptive and intelligent systems

A system that adapts changes what it shows or does in response to the person using it. That change is a claim about the person, inferred from data, and the system cannot check whether the claim is right. Design and evaluation therefore arrive together. You cannot judge an adaptation without the signal behind it. You cannot judge that signal without watching what the interface does to the data.

The contested question is whether adaptivity earns its cost. One position holds that personalisation reduces overload by removing irrelevant options. The other holds that it removes the information a person needs to notice the system is wrong. The evidence supports something narrower: adaptation changes behaviour, and how much depends on the task and on whether the person can tell what was adapted.

Adaptation is a claim about the user, and the claim can be wrong

Every adaptive system keeps a model of the person, and that model fails first. Preferences can be acquired implicitly, by monitoring what someone views or buys, or asked for explicitly (Jannach et al., 2010). Implicit signals are cheap, continuous and ambiguous. A click is not an endorsement. Opening an article to check claims teaches the system that you like them.

Two failures follow. The cold start: a new user has no history, so the system falls back on population defaults, which personalisation was meant to replace. The feedback loop: what the system shows shapes the behaviour it later learns from. A recommender that offers one kind of item collects the evidence that it was right.

Jannach and colleagues open with a bookstore. A regular visitor there sees a personalised list, because the system remembers their history (Jannach et al., 2010). The list gives no account of itself. Amershi and colleagues codified over 150 design recommendations into 18 guidelines, validated by 49 practitioners against 20 AI-infused products (Amershi et al., 2019). The relevant ones ask a system to make clear what it can do, and how well. They also ask it to learn from behaviour, and to offer a global control.

Two ways to recommend, and the interface each one forces

Collaborative filtering bets on resemblance. If two people agreed in the past, they will probably agree in future. An item one of them has since chosen can then be offered to the other (Jannach et al., 2010). It works where many users rate many of the same things. Its interface consequence is that other people justify the recommendation: the standard label reports that customers who bought this also bought it. That says others like you chose it, not what it is like.

A content-based recommender describes each item by its attributes and matches them to a profile. The interface can then give a reason of the form because you read this (Jannach et al., 2010). That reason is only as good as the metadata, which says what an item is, not whether it is good. Knowledge-based and hybrid methods cover the case where behaviour is too thin to learn from.

The measurement problem sits underneath both: an offline score is computed against choices people made under an earlier version of the system. Jannach and colleagues give a whole chapter to evaluating recommender quality (Jannach et al., 2010, chapter 7).

The filter bubble claim, and what the evidence shows

The strong filter bubble claim is that personalisation narrows exposure and produces an echo chamber. Flaxman, Goel and Rao tested it on browsing histories for 50,000 United States users who regularly read online news (Flaxman, Goel and Rao, 2016). Their findings cut both ways. Social networks and search engines were associated with a rise in the mean ideological distance between individuals. The same channels were also associated with a rise in a person's exposure to the side of the spectrum that they preferred less. Most news consumption was direct visits to favourite mainstream home pages, and the authors call the magnitudes modest.

What the study cannot establish matters as much. It is an association from browsing logs in one country, so it cannot separate what an interface offered from what the person opened. The outcome is the mix of articles reached, not what anyone believed. The firm reading is narrower: the interface controls exposure, and exposure is measurable.

Mixed initiative and the timing of action

Horvitz's 1999 paper founded the idea of shared control. It argues for coupling automation and direct manipulation rather than opposing them (Horvitz, 1999). Its decision-theoretic core is the part to keep: acting under uncertainty compares expected costs and benefits, not model accuracy. When the expected value of acting exceeds the value of waiting, autonomy is warranted. The principles that follow ask a system to reason about the user's goals. They ask it to time interventions against attention, to resolve key uncertainties by dialogue, and to keep a wrong guess cheap.

The worked case is Lookout, a scheduling and meeting assistant. It reads the message on screen and infers whether a meeting is proposed. It then decides whether to offer to add it to the calendar (Horvitz, 1999). The design assumes wrong guesses, so a bad inference still leaves the person near the right point. When confidence falls between the thresholds for dialogue and for action, the system can ask rather than act.

The limit is that this is a position statement built on one prototype with no controlled evaluation. Cite it for the framing, not for an effect size.

Conversational agents fail at repair, not at speech

Luger and Sellen interviewed 14 users of conversational agents about everyday use (Luger and Sellen, 2016). Expectations were dramatically out of step with how the systems worked, particularly about the machine's intelligence, capability and goals. People brought a model of human conversation to the interaction and revised it only through failure, because nothing told them what the system can do. The result is a deep gulf of evaluation. Users were unable to judge how much the system understood. They confirmed visually on all but the simplest tasks and avoided it for complex or sensitive ones.

Reveal capability before failure, not inside a failure message. A person who cannot judge capability blames themselves and drops the task. The paper records where dialogue pays. Its list includes filtering a large set and following a path, plus queries built over several turns. Against that promise it reports recognition errors, unintelligible responses and hard to navigate dialogues. Its sample used the agents hands free, with attention elsewhere.

The test for voice is whether the task survives the failure, not whether the system sounds natural. Voice earns its place in short repeated commands with a small vocabulary, used while the hands are busy. Browsing a list and comparing two options cost the most. So does repairing a mistake. The study is small, mostly male, United Kingdom based and from 2016.

Systems that write: what fluency hides

Prompt text is the interaction surface, and what an assistant remembers of earlier turns is context that shapes the next answer. Bender and colleagues ask whether language models can be too big (Bender et al., 2021). Two points carry into design. These models are not performing natural language understanding and succeed on tasks handled by manipulating linguistic form. And building datasets larger than a team can document creates documentation debt. Nobody can then say what it was trained on, or what it excluded.

Hallucination follows from the first point rather than from a bug. A generator optimised for fluency produces a sentence with no source behind it that reads exactly like a grounded one. People impute meaning where there is none (Bender et al., 2021). The interface duty is separation, so that retrieved text looks different from generated text and a claim carries its source.

Sycophancy arrives through the training signal. Sharma and colleagues found five state-of-the-art assistants showing sycophancy across four free-form generation tasks. They traced it to preference data: answers agreeing with a user's view were more likely to be preferred (Sharma et al., 2023). Human raters and preference models also preferred a convincing sycophantic answer over a correct one a non-negligible fraction of the time. A system tuned on approval is a poor source of disagreement, so the interface needs a route to a second opinion.

The retrieved page is part of the attack surface

Prompt injection is usually described as a text problem, but Greshake and colleagues show it is a boundary problem. Applications built on language models blur the line between data and instruction. An attacker can therefore place instructions in content the system is likely to retrieve, without touching the interface (Greshake et al., 2023). They demonstrated the attacks against deployed systems, including Bing's GPT-4 powered chat, and report that processing retrieved text can amount to arbitrary code execution. They also state that effective mitigations were lacking at the time.

Guardrails follow, and they are design decisions. Treat retrieved content as data, in a channel the model cannot read as instruction. Put a confirmation step before any irreversible action, and scope permissions to the task. Amershi and colleagues reach the same place, with guidelines on scoping a service and on correction (Amershi et al., 2019). These patterns reduce the attack surface, and no source read here claims they close it.

Evaluating a system with no single right answer

Benchmarks are built for tasks with one right answer. Zheng and colleagues describe the gap that they filled. Existing benchmarks were inadequate for measuring human preferences about chat assistants (Zheng et al., 2023). They built MT-bench, a multi-turn question set, and Chatbot Arena, a crowdsourced comparison platform, and studied a strong model as judge. A strong judge matched controlled and crowdsourced human preferences with over 80 per cent agreement. The authors call that agreement human-level.

The judge also carries documented biases, including position, verbosity and self-enhancement, and limited reasoning (Zheng et al., 2023). The circularity matters most. A judge from the same family as the system under test is not an independent rater. Sharma and colleagues supply the case. A preference model that favours a convincing sycophantic answer rewards the behaviour the team is trying to reduce (Sharma et al., 2023).

Where benchmarks reach nothing, red teaming is the fallback. Ganguli and colleagues red teamed four model types at three sizes and released the 38,961 attacks they collected. They found that models trained with human feedback grew harder to red team with scale. The other types stayed flat (Ganguli et al., 2022). They publish their methods and are candid about their uncertainty. Two limits follow. Red teaming samples attacks rather than estimating how often harm occurs in use. It also finds what the team thought to look for.

Reporting practice is what lets a reader judge. Name the model version and date, the raters and how they were recruited, the agreement between them, and the failure modes that it cannot see.

Showing what the system does not know, and shipping it anyway

Uncertainty has to be shown at the moment of decision. Horvitz asks a system to reduce the precision of its service as uncertainty rises. His argument is that doing less correctly beats doing more wrongly (Horvitz, 1999). Amershi and colleagues restate it. Make clear how well the system can do what it can do. When in doubt, scope the service (Amershi et al., 2019). A confidence percentage rarely helps: it moves the problem onto the user, who cannot calibrate it. The useful display is a boundary: the cases where the system will not act, and what it does instead.

This assumes that the model stays still, and it does not. A feature is tested against one version and runs against the next, so the evaluation describes a system that no longer exists. Bender and colleagues name documentation debt. They argue for documenting the data behind a model rather than ingesting everything available (Bender et al., 2021). Amershi and colleagues give the interface half: update and adapt cautiously, and notify users about changes (Amershi et al., 2019). One participant in that study reported recommendations that shifted every week with no explanation of what had changed.

The next problem is the display of evidence around the model. Show a person the distribution, the gap and the source instead of one confident answer. Data visualisation and visual analytics continue from there.