Human-AI interaction and machine learning interfaces

A deterministic system does what its specification says. A machine learning system does what its training data suggests, and it is usually a little wrong. That difference reorganises the design work. You no longer specify every transition between states. You set what the system may do, how it signals what it knows, when a person can overrule it, and what happens when it fails.

A button means the same thing every time it is pressed. A suggestion, a score or a generated paragraph does not. The same input can produce a different output tomorrow, and for someone else. Thinking in distributions rather than rules is the skill.

A model is a guess fitted to data, and confidence is not certainty

A model is a function whose parameters are estimated from examples rather than written by hand. The examples are the training data, and the features are the properties the model uses, whether chosen by an engineer or learned.

Two failures follow. A model that memorises its training set scores well there and poorly on anything new. That is overfitting. A model that generalises has learned something that survives a change of sample. The second failure is quieter and worse. Training data came from somewhere, and deployment is not that somewhere, and the gap is called distribution shift.

Take a constructed case, not a reported result. A support team trains a model to mark tickets urgent, using labels from a queue an experienced agent had already triaged. The model learns that tickets naming an enterprise customer are urgent, because that is what the agent did. It has learned the agent's habit, not urgency. When the relationship changes, urgent tickets naming no customer are missed. No code is wrong. The label was a proxy, and the proxy drifted.

A designer cannot read a model's competence from a screenshot. It lives in the data, which is a design material with a history.

A model rarely outputs a decision. It outputs a score, and something in the product turns that score into an action by comparing it with a threshold. A score of 0.9 says that many training cases like this one carried the label. It does not say the model is right nine times in ten about this case.

The threshold sets the trade between two errors. A spam filter acting at 0.5 sends some real mail to the junk folder. A clinical alert at 0.5 produces false alarms, and false alarms are expensive and easy to underestimate. Parasuraman and Riley (1997) traced disuse, the neglect of a warning system, to alarms that fire falsely. Designers set the false alarm rate without considering the base rate of the condition, so for a rare condition almost every alarm is wrong and the operator learns to ignore them all. The threshold decides which error the product would rather make, and that belongs to the people who bear the errors.

From suggestion to automation: mixed initiative

An AI feature sits between a hint the user can ignore and an action the system takes alone. Horvitz (1999) wrote the founding statement of the middle ground. His principles for mixed-initiative interfaces are the source of much of what the field now does: automate only where it adds genuine value over direct manipulation; reason about uncertainty in the user's goals; weigh expected costs and benefits before acting; resolve uncertainties through dialogue; give efficient means to invoke and terminate the service; and make a poor guess cheap.

The Lookout system Horvitz built to demonstrate the principles scheduled meetings by reading the message in focus, and it was sometimes wrong. The design therefore had to make being wrong cheap: an easy refusal, a reversible action, a visible reason. The method matters more than the demonstration: the decision to act is a calculation over expected costs, not a property of the model.

The contested question is how much initiative a system should take. More automation removes routine work and quietly removes the operator's feel for the task. Less automation keeps judgement sharp and hands back the drudgery the feature was meant to absorb. Horvitz's answer is not a level of autonomy but a comparison: act when the expected benefit exceeds the expected cost of being wrong, including the cost of interrupting. That comparison forces the designer to state the costs, which is where the argument happens.

The eighteen guidelines, and the evidence behind them

Amershi and colleagues (2019) gathered more than 150 design recommendations from two decades of academic and industry work and reduced them to 18 guidelines for human-AI interaction. The set is organised by when it applies: before use, during interaction, when it is wrong, and over time. The first pair asks the designer to make clear what the system can do and how well it does it. The last asks for global controls over what it monitors and notification when its capabilities change.

The provenance matters as much as the contents. These are not experimental findings. They are a codification. The authors validated the set in a study with 49 design practitioners who applied it to 20 popular AI-infused products. That study checked relevance, not effect: it did not show that following them improves any measured outcome, and the paper does not claim otherwise.

Worked case: a suggested-reply feature in an email client. The guidelines on making clear what the system can do, and how well it does it, mean the feature should not present itself as a full writing assistant and should favour hedged language over a confident assertion. A global control means the user can switch suggestion off without abandoning the client. The honest limit is that a checklist produces consistency, which is not the same as a good decision.

Explanation is a social act, and a feature attribution is rarely one

Miller (2019) reviewed the philosophy, cognitive science and social psychology of explanation and found that explainable AI research had mostly used the researchers' own intuition of a good explanation. The social-science literature disagrees in three ways. People want contrastive explanations: not why this happened, but why it happened rather than the alternative they expected. People want a small number of causes selected for relevance, not a complete account. And people want causes they can act on.

Worked case: a loan application refused by a model. The interface shows a feature attribution, stating that the score fell mainly because of the applicant's postcode and account age. That is accurate about the model's weights and answers a question nobody asked. The applicant asks why they were refused and the neighbour approved, which calls for a counterfactual and a route to action.

Miller's point is not that attribution is worthless for an engineer debugging a model. It is that attribution is not explanation, and the two serve different people. The evidence needs a caveat. Much of the work he reviews concerns how people explain to each other in conversation, not how they read a machine's reasons on a screen. The move to an interface is a reasonable inference rather than a tested design claim.

Automation bias, complacency and skill decay

Two papers from outside HCI explain most of what goes wrong when a working AI feature meets an operator. Parasuraman and Riley (1997) set out a taxonomy of use, misuse, disuse and abuse. Misuse is overreliance, where the operator stops monitoring and accepts the automation's answer. Disuse is the opposite failure, where a system that alarms too often is ignored. Abuse is what designers do when they automate a function without regard for its consequences for human performance.

Bainbridge (1983) described the ironies automation creates. Automating the routine parts of a task leaves the human the difficult abnormal cases, which are exactly the cases the human has least practice with. She also observed that a person who monitors an automated process for long periods becomes a poor monitor.

Worked case: a tool that flags chest radiographs for a radiologist. If it rarely misses, and the interface presents its negative output as settled, the clinician's checking behaviour erodes. When it fails, it fails on the unusual case, which the clinician is least practised at spotting. The remedy is not more trust or less but calibration, where the operator's confidence matches the system's actual reliability in this task, in this setting. A design that pursues maximum trust is aiming at the wrong target, because trusting a system more than it deserves is the definition of misuse.

Evaluating an AI feature when ground truth is expensive

Offline evaluation compares model outputs with a labelled set. It is cheap and repeatable, and often misleading: the labelled set is a sample from somewhere, and deployment is somewhere else. Where the real target is human judgement, the target is itself noisy, since two experts disagree and agreement is not truth.

Worked case: an assistant that summarises meeting transcripts. An overlap metric compares the generated summary with a reference, so it rewards copying sentences from the transcript and penalises a faithful paraphrase. It cannot see the failure that matters, which is a fluent sentence that was never said. Ji and colleagues (2022) survey hallucination across generation tasks and catalogue the metrics and the mitigations. The impression their survey leaves is that the problem is measured better than it is solved.

Red teaming adds a different kind of evidence: people try to make the system fail on purpose. Greshake and colleagues (2023) used that approach to demonstrate prompt injection against deployed systems, hiding instructions inside content that a model retrieves. Red teaming produces existence proofs rather than coverage estimates. Finding one attack does not tell you how many remain, and a clean report is not evidence of safety.

Documentation and the fairness question a product must answer out loud

Mitchell and colleagues (2019) proposed the model card, a short document released with a trained model. It records the intended use, the evaluation conditions, performance broken down by demographic and intersectional groups, and the known limitations. They demonstrate the format with two example cards. Gebru and colleagues (2021) proposed the companion practice for data: the datasheet, documenting a dataset's motivation, composition, collection process and recommended uses.

Fairness is where the documentation has to say something difficult. Several fairness definitions are individually reasonable and jointly unsatisfiable. Kleinberg, Mullainathan and Raghavan (2017) proved that three common conditions on a probabilistic risk score cannot hold at once except in constrained special cases. The mathematics does not matter at the interface. The consequence does. A team cannot be fair in the abstract. It must decide which definition it is using, state that decision in the product documentation, and accept what the choice gives up. A model can be calibrated within each group or can equalise error rates across groups, and the choice changes who is helped and who is harmed.

Two limits apply. A model card can be accurate and still not prevent harm. A datasheet can be complete and still describe a dataset collected without proper consent. Documentation is a discipline of honesty, not a guarantee of safety.

Generative interfaces: hallucination, sycophancy and prompt injection

Large language model interfaces add a failure earlier feature patterns did not have. The system produces fluent, confident text that can be false. Ji and colleagues (2022) name this hallucination and show that it appears across summarisation, dialogue, translation and question answering. A second failure is sycophancy, where the assistant agrees with the user's stated position rather than with the evidence. The work around it is newer and less settled than the work on hallucination. A third is prompt injection. Greshake and colleagues (2023) showed that instructions hidden in content a model retrieves can override the developer's instructions. That effective mitigations were lacking is why this is now an interface problem as well as a security one.

Treat retrieved text as data and never as instructions, and never let it trigger an action on its own. Ground answers in sources the interface shows, so a reader can check rather than trust. Put a confirmation step in front of any irreversible action, which is Horvitz's principle about the cost of a poor guess applied to a language model. Keep automatic action small enough that the person keeps the judgement the feature was meant to support.

These problems reappear when the output is spatial rather than verbal, and the next stage of the work takes up interfaces that place such systems in the room with the user, where failures are physical and the evidence for benefit is thinner than the demonstrations suggest.