Explanation: what interpretability tools offer and what people ask for

A model outputs a score, which a product turns into a decision. The score answers what and leaves why open. The field that grew around that gap answered mostly with one artefact: a per-prediction attribution ranking input features by their influence. This third of the ten units takes what a person is shown. Attribution answers an engineer's question about a model's internals. The person facing the interface asks a different question. The experimental record does not show that adding an explanation improves the decision.

The two methods that became the field's answer

Ribeiro, Singh and Guestrin (2016) introduced LIME, which explains any classifier's prediction by fitting an interpretable model in the neighbourhood of the instance. Lundberg and Lee (2017) presented SHAP. It recovers six existing attribution methods as special cases of additive feature attribution. It also proves a unique solution in that class under stated properties. Together they fix the field's answer: a local, additive, feature-level score, general enough to compute for any model.

The most visible explanations can be silent about the model

Adebayo and colleagues (2018) proposed sanity checks: randomise the model's parameters or its training labels, recompute the explanation, and see whether it changes. Some widely used saliency methods produced essentially unchanged maps when the model or the data generating process was destroyed. Visual appeal is not evidence of explanation. Ghorbani, Abid and Zou (2019) generated perceptually indistinguishable inputs with the same predicted label and dramatically different feature importance maps. Explanation stability is therefore separable from prediction stability.

Nothing is agreed on what explanation is for

Lipton (2018) argues that interpretation is underspecified and that the motivations behind it are diverse and occasionally discordant. Transparency to a human and post-hoc explanation are, on his account, competing notions. Doshi-Velez and Kim (2017) define interpretability and describe when it is needed. They report no consensus on what interpretable machine learning is, or how to measure it. A tool cannot be justified by the label interpretable, only by the question it answers.

The tools answer a question people do not ask

Miller (2019) reviewed explanation as studied in philosophy, cognitive science and social psychology. He concluded that explainable AI research had mostly relied on the researcher's intuition of a good explanation. The reviewed literature points away from complete accounts of causes. People want contrastive explanations, a few causes selected for relevance, and causes they can act on. Liao, Gruen and Miller (2020) interviewed 20 UX and design practitioners working on AI products and built a question bank of the explanations users ask for. Their needs centred on how the system behaves in general and what to do when it fails, which the algorithmic literature addresses poorly. The tool answers which features moved the score. The person asks why this outcome rather than the expected one, and what would change it.

Deployed explanations serve the engineer

Bhatt and colleagues (2020) studied how organisations use explainability for the stakeholders around a model. They found that most deployments were for machine learning engineers debugging the model, not for the people affected. The stated goal of transparency and the practice of explanation had therefore come apart. What that use looks like in practice is measurable. Kaur and colleagues (2020) observed eleven data scientists using the InterpretML implementation of GAMs and the SHAP package, and surveyed 197 more. Participants over-trusted and misused the tools, and few could describe the visualisations accurately. The case covers one population of practitioners reading two tools.

The experimental record does not show that explanation helps

Poursabzi-Sangdeh and colleagues (2021) ran pre-registered experiments with 3,800 participants, varying only the number of features and whether the internals were visible. Participants could better simulate a clear, compact model. They did not follow those predictions more closely. The same presentation also reduced their ability to detect and correct the model's large mistakes. Wang and Yin (2021) compared four model-agnostic explanation methods against three properties an explanation is often expected to have: improving understanding, recognising uncertainty, and supporting calibrated trust. Many satisfied none of the properties where participants had little domain expertise.

The relevant question is therefore not whether an explanation is faithful to the model, but whether the interface containing it changes what the person does. The measured answer is often no, or worse than no.

Contrastive explanations show what a defensible claim looks like

If a tool answers the wrong question, build the answer to the right one and measure it. Bove and colleagues (2023) built an interface presenting plural counterfactual examples with comparative features for a loan-application scenario. Their lab study with 112 participants found that plural examples improved objective understanding and satisfaction over a single counterfactual. VanNostrand and colleagues (2024) compared counterfactual types and presentations in a crowd-sourced between-subjects study with 252 participants. They found that region-based counterfactuals increased objective understanding, subjective understanding and response confidence over point-based ones. Both work with lay decision subjects, and both show what a defensible claim looks like. It is a comparison against an alternative, on a stated outcome, in the population the interface will serve.

Contested question: explain the black box, or stop building one?

The sharpest disagreement is not about which explanation method works but about whether the explanation enterprise should exist in its present form.

Rudin (2019) holds that explaining black-box models for high-stakes decisions perpetuates bad practice and can cause harm. He argues that interpretable models should be designed instead, citing cases where they achieve comparable accuracy. On this view the unit's subject is a wrong turn.

The second position is that post-hoc explanation is worth having and worth improving. Ribeiro and colleagues (2016) and Lundberg and Lee (2017) built the tools, and Bhatt and colleagues (2020) document their use for debugging. Bove and colleagues (2023) and VanNostrand and colleagues (2024) show that a contrastive explanation can improve understanding for the people a decision affects. On this view the tools are young rather than wrong.

The third position is that argument cannot settle the dispute, because the two sides measure different things. One measures which model to build. The other measures what to show once one exists. Doshi-Velez and Kim (2017) name the missing evidence, evaluation at the application level with human-grounded studies in between.

What would settle it is a comparison that is rarely run. The same task goes to comparable participants twice: once with an inherently interpretable model, and once with an opaque model plus its explanation. The comparison measures decision quality, error detection and calibrated reliance. Poursabzi-Sangdeh and colleagues (2021) come closest, and their result runs against the assumption that legibility helps. Wang and Yin (2021) show why one measure cannot settle it: the properties they test dissociate.

What to do while the dispute is open

The two sides rest on different kinds of evidence. Rudin (2019) argues from a principle and from selected cases, so the claim that an interpretable model can match a black box is asserted rather than compared. The defence of post-hoc explanation rests on two method papers and two laboratory studies of 112 and 252 participants, and those studies measure understanding, which is not the decision. The strongest objection to this unit's thesis is that its constructive turn may be the exception rather than the direction. Poursabzi-Sangdeh and colleagues (2021) found that a legible model reduced people's ability to notice its large mistakes, so the difficulty may lie in the person and not only in which question the tool answers. While the dispute is open, a designer should choose the simplest model that meets the accuracy bar, keep an explanation only where a study shows it changes the decision, and state the question it answers. An explanation that is a chart also needs a textual form, because a screen reader cannot read it.

Where the argument is heading

The direction of travel is from method papers towards interfaces and their measurement. Attribution methods came first (Ribeiro et al., 2016; Lundberg and Lee, 2017), then checks on whether they carry information at all (Adebayo et al., 2018; Ghorbani, Abid and Zou, 2019). Then came experiments on what happens to a person who sees one (Poursabzi-Sangdeh et al., 2021; Wang and Yin, 2021). Finally came contrastive explanations evaluated with the people they are meant to serve (Bove et al., 2023; VanNostrand et al., 2024).

The sources this unit rests on

• Adebayo, J., Gilmer, J., Muelly, M., et al. (2018). Sanity Checks for Saliency Maps. arXiv preprint arXiv:1810.03292. doi:10.48550/arXiv.1810.03292 The authors randomise a model's parameters or training labels and recompute the explanation. Some widely used saliency methods produced essentially unchanged maps, so their output is independent of the model and of the data, which sets the unit's precondition for an explanation claim.
• Bhatt, U., Xiang, A., Sharma, S., et al. (2020). Explainable machine learning in deployment. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. doi:10.1145/3351095.3375624 The study asked how organisations use explainability for the stakeholders around a deployed model. Most deployments used explanations for machine learning engineers debugging the model, not for the people affected, which is the unit's deployment-side evidence.
• Bove, C., Lesot, M.-J., Tijus, C. A., et al. (2023). Investigating the Intelligibility of Plural Counterfactual Examples for Non-Expert Users: an Explanation User Interface Proposition and User Study. Proceedings of the 28th International Conference on Intelligent User Interfaces, 188-203. doi:10.1145/3581641.3584082 The authors built an interface presenting plural counterfactual examples with comparative features for a loan-application scenario and tested it with 112 participants. Plural examples raised objective understanding and satisfaction over a single counterfactual, the unit's constructive case.
• Doshi-Velez, F. and Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv preprint arXiv:1702.08608. doi:10.48550/arXiv.1702.08608 This position paper defines interpretability, describes when it is needed, and proposes a taxonomy of evaluation from application-level assessment through human-grounded studies to functionally-grounded proxies. It reports no consensus on what interpretable machine learning is, the evidence gap the unit turns on.
• Ghorbani, A., Abid, A. and Zou, J. (2019). Interpretation of Neural Networks Is Fragile. Proceedings of the AAAI Conference on Artificial Intelligence 33(1), 3681-3688. doi:10.1609/aaai.v33i01.33013681 The authors generated perceptually indistinguishable inputs with the same predicted label and measured how far explanations moved. Feature importance maps changed dramatically across several methods, and exemplar-based explanations were also vulnerable.
• Liao, Q. V., Gruen, D. and Miller, S. (2020). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3313831.3376590 The authors interviewed 20 UX and design practitioners working on AI products and built a question bank of the explanations users ask for. Their needs centred on how the system behaves in general and what to do when it fails, which the algorithmic literature addresses poorly.
• Kaur, H., Nori, H., Jenkins, S., et al. (2020). Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3313831.3376219 A contextual inquiry with eleven data scientists and a survey of 197 observed how practitioners use the InterpretML implementation of GAMs and the SHAP package. Participants over-trusted and misused the tools, and few could describe the visualisations, so the difficulty sits in reading the output.
• Lipton, Z. C. (2018). The Mythos of Model Interpretability. Queue. doi:10.1145/3236386.3241340 Lipton argues that interpretation is underspecified, that the motivations behind it are diverse and occasionally discordant, and that transparency to humans and post-hoc explanation are competing notions. He questions the assertions that linear models are interpretable and deep networks are not.
• Lundberg, S. and Lee, S. (2017). A Unified Approach to Interpreting Model Predictions. arXiv preprint arXiv:1705.07874. doi:10.48550/arXiv.1705.07874 The authors present SHAP, which recovers six existing attribution methods as special cases of additive feature attribution and proves a unique solution in that class under stated properties. It is the second half of the field's canonical answer.
• Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence. doi:10.1016/j.artint.2018.07.007 This review draws on philosophy, cognitive psychology and social psychology to argue that explainable AI research has mostly used the researcher's own intuition of a good explanation. The reviewed literature holds that people want contrastive, selective and actionable explanations.
• Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., et al. (2021). Manipulating and Measuring Model Interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3411764.3445315 In pre-registered experiments with 3,800 participants, the authors varied only the number of features and the visibility of a functionally identical model's internals. Participants could better simulate a clear, compact model but did not follow it more closely, and were less able to detect its large mistakes.
• Ribeiro, M., Singh, S. and Guestrin, C. (2016). "Why Should I Trust You?": Explaining the Predictions of Any Classifier. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, 97-101. doi:10.18653/v1/n16-3020 The authors introduced LIME, which explains any classifier's prediction by fitting an interpretable model locally around the instance, and reported simulated and human-subject experiments on tasks that require trust. It is the canonical attribution method the unit contrasts with the user's question.
• Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 206-215. doi:10.1038/s42256-019-0048-x Rudin argues that explaining black-box models for high-stakes decisions perpetuates bad practice and can cause harm, and that interpretable models should be designed instead, citing cases where they achieve comparable accuracy. It is the strongest critique of the unit's subject.
• VanNostrand, P. M., Hofmann, D. M., Ma, L., et al. (2024). Actionable Recourse for Automated Decisions: Examining the Effects of Counterfactual Explanation Type and Presentation on Lay User Understanding. The 2024 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3630106.3658997 In a crowd-sourced between-subjects study with 252 participants, the authors compared counterfactual explanation types and presentations for their effect on lay decision subjects' understanding. Region-based counterfactuals raised objective, subjective and confidence measures over point-based ones.
• Wang, X. and Yin, M. (2021). Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making. 26th International Conference on Intelligent User Interfaces. doi:10.1145/3397481.3450650 The authors compared four model-agnostic explanation methods in randomised experiments against three properties: improving understanding, helping people recognise uncertainty, and supporting calibrated trust. The methods had different profiles and the counterfactual explanation did not improve calibrated trust, so the properties are separable.