Evaluating a visualisation: comprehension, trust, decisions, and the ethics of persuasion

Year 3 / Module 33 Data visualisation in practice / Unit 10 / Evaluating a visualisation: comprehension, trust, decisions, and the ethics of persuasion

Evaluating a visualisation: comprehension, trust, decisions, and the ethics of persuasion

HCI3109 · Unit 10 of 10

Evaluating a visualisation means choosing what counts, comprehension, trust or decision quality, and living with the limits of that choice, while the ethics of persuasion begins where an accurate display still leads a reader to the author's conclusion.

Evaluating a visualisation: comprehension, trust, decisions, and the ethics of persuasion

Every unit before this one ended with a reader and a chart. This tenth of ten units asks how you would know whether the chart worked. Its answer is uncomfortable: there is no single method, only a target you choose and then have to live with. Comprehension, insight, confidence and decision quality are different things, and a study of one says little about the others. An evaluation is only as good as the target it names and the limits it admits.

The line between informing and persuading is drawn by the evaluation that screened the design, not by the data. The ethical question begins where the measurement stops.

Evaluation begins by choosing a target

North (2006) argued that the purpose of visualisation is insight, and that evaluation should determine how well a visualisation produces it. He argued that the field's default method, the controlled experiment with benchmark tasks, does not effectively measure it. He named characteristics of insight, among them complexity, depth, unexpectedness and relevance, and proposed a reworked experiment that could capture them.

Saraiya, North and Duca (2005) turned the argument into an instrument. They developed a method that recognises and quantifies insight in open-ended user tests. They applied it to five microarray visualisation tools, measuring how much insight each produced, of what type, and how long it took. No benchmark task reports that comparison, since a benchmark fixes the question in advance (North, 2006).

The five tools were named and the ranking was not uniform. The authors tested Cluster/Treeview, TimeSearcher, the Hierarchical Clustering Explorer, Spotfire and GeneSpring on time-series, viral and lupus microarray data. The best tool changed with the dataset. The Hierarchical Clustering Explorer produced the most insight on the viral data, while Spotfire and TimeSearcher led on the time series. Across all three, Spotfire had the highest insight count and value, significantly more value than GeneSpring. Cluster/Treeview reached a first insight fastest, where the Hierarchical Clustering Explorer and GeneSpring took about twice as long on average. What it does not show is whether any of those insights changed a decision.

Insight, value and adoption are three different measures

Stasko (2014) renamed the target and widened it. Against low-level benchmark questions he set a value-driven evaluation, in which a system's worth is shown through four capabilities. They are minimising the time to answer diverse questions, spurring insights and insightful questions, conveying the essence of the data, and generating confidence and knowledge about its domain. The fourth is the one a comprehension test cannot reach, because confidence is a state of the reader.

Plaisant (2004) had already argued that the field should look past its own instruments. Usability studies and controlled experiments, she wrote, are useful for a tool's potential and limits. They cannot account for the long exploratory nature of real tasks, the value of a potential discovery, or the benefit of general awareness.

Cheap methods catch what they are aimed at, and miss the rest

Zuk, Schlesier, Neumann, Hancock and Carpendale (2006) ran a meta-analysis. Three published sets of heuristics were used to assess one visual decision support system for simulation data. The results depended heavily on which heuristics and which types of evaluator were chosen, and the sets carried problems of interpretation, redundancy and conflict. Their constructive finding was that visualisation-specific heuristics caught faults usability heuristics would not. Heuristic evaluation is cheap, but its output is a function of the instrument, so two teams can report different faults on the same system.

The one target with a written error catalogue is uncertainty

Where the rest of the field argues about targets in general terms, the uncertainty literature does the accounting. Hullman, Qiao, Correll, Kale and Kay (2019) built a taxonomy of the six levels of decision in an evaluation of an uncertainty display. The levels run from the behavioural target down to the measures, elicitation techniques and analysis. They applied the taxonomy to eighty-six user studies. They found practice concentrated on performance and satisfaction measures that assume a tidier and more predictable judgement than research on human judgement and decision making supports. The studies that look most objective may be measuring a reader who does not exist. A reader who cannot see the marks, or who reads in a second language, is measured on the instrument rather than on the chart.

Even the vocabulary is contested

The disagreement runs deeper than the choice of measure. Dimara and Perin (2020) asked what interaction means for data visualisation and found the term ambiguous, the confusion traced to the absence of a consensual definition. They synthesised views across visualisation fields, compared them with human-computer interaction definitions, and proposed one as a thinking tool. When the object being evaluated is named differently from study to study, comparisons measure the vocabulary as much as the design. One evaluation target, trust, now has a standard instrument. Wang and colleagues (2025) derived an operational definition of trust by factor analysis and validated an eight-item inventory in two trust games with real stakes.

The reader is not a neutral instrument

Two reviews establish that the person reading is a source of error in their own right. Dimara, Franconeri, Plaisant, Bezerianos and Dragicevic (2020) surveyed the literature and proposed a taxonomy of 154 cognitive biases in seven main categories. It is organised by visualisation task rather than by cognitive theory, so a designer can associate a design with the biases it may trigger. Bresciani and Eppler (2015) reviewed the pitfalls of visual representation and classified its cognitive, emotional and social risks. The errors are therefore systematic and task-linked, and some can be worked on deliberately by someone who wants a particular reading.

The contested question: does evaluation measure the person or the work?

One side holds that evaluation measures the person, and that the target with a defensible claim on the name is insight. North (2006) is its clearest statement. If the purpose of visualisation is insight, a method that ignores insight is not measuring what matters. The controlled experiment should be rebuilt rather than abandoned. Saraiya and colleagues (2005) supply the worked method.

The other side holds that evaluation measures the work, the difference a tool made to people doing real tasks. Plaisant (2004) asks for accounts of long exploratory use, of potential discoveries and of general awareness, and for reports of adoption. Stasko (2014) asks for the value a system delivers, including confidence and domain knowledge. On this account a laboratory comprehension score is a poor proxy, and the field should be judged by the tools that reached users.

A third reading, the one this unit leans on, is that the two sides are rival measurements of the same event rather than rival answers. Hullman and colleagues (2019) show the default measures assume a predictable reader. Zuk and colleagues (2006) show even a cheap method's output is a function of the instrument.

What would settle it is a single study that takes one visualisation, one task and one sample of readers, and measures all three targets at once. It would run an objective comprehension test on the taught search, an insight count under an open-ended protocol, and a decision outcome scored independently. If the measures correlate, the disagreement is about taste. If they diverge, it is about substance, and the divergence would tell a designer which target their claim rests on. A second study would strengthen it: a pre-registered list of a system's known defects, scored blind against a heuristic evaluation and an insight study of the same system. Until then, it is honest to name the target you chose and the target you gave up.

The two sides rest on different kinds of evidence. The person side rests on two argued positions (North, 2006; Plaisant, 2004) and on one insight instrument applied to five microarray tools (Saraiya, North and Duca, 2005). The work side rests on argued positions about value and adoption (Stasko, 2014) rather than on measured studies. While the dispute is open, a designer should name the target that was measured and the target that was given up, and run at least two instruments on the same system. The strongest objection is that the unit's own third reading rests on a survey of 86 studies and a meta-analysis of one system, so no study here measures comprehension, insight and decision together.

A chart can tell the truth and still steer

The argument closes where the module opened. Correll (2019), drawing on historical and contemporary examples, set out the moral components of the design and use of visualisations. He identified areas of research that carry ethical dilemmas, and proposed additional moral obligations for those who design, build and study them. His point is not that a chart can lie. The steering is a design decision with an author. Bresciani and Eppler (2015) classify cognitive, emotional and social risks, so some of the harm is produced in the reader rather than the data. The ethical requirement is to state the reading you built in and the readings you did not measure.

Where this is heading

The direction of travel is away from a single default method and towards declared, plural, named targets. Hullman and colleagues (2019) offer their taxonomy as a guide for evaluators navigating uncertainty. Dimara and Perin (2020) offer a definition of interaction as a thinking tool. Correll (2019) proposes obligations that would make the ethical choice part of the design record. Stasko (2014) offers a value-driven frame that reports delivered benefit rather than a benchmark score. The field is learning to say what it measured and on whom, the discipline this module asks of every chart.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked on 2 November 2026.

Bresciani, S. and Eppler, M. J. (2015). The Pitfalls of Visual Representations. SAGE Open, 5(4). doi:10.1177/2158244015611451 Reviewed the multidisciplinary literature on visual representation, which had covered benefits but not risks, and classified its cognitive, emotional and social risks, populated with documented pitfalls. It shows the reader, not only the data, can be the site of failure.
Correll, M. (2019). Ethical Dimensions of Visualization Research. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1-13. doi:10.1145/3290605.3300418 Examined the moral components of designing and using visualisations, identified areas of research that carry ethical dilemmas, and proposed additional moral obligations for visualisation designers, builders and researchers. It grounds the claim that an accurate display can still steer its reader.
Dimara, E., Franconeri, S., Plaisant, C., Bezerianos, A. and Dragicevic, P. (2020). A Task-Based Taxonomy of Cognitive Biases for Information Visualization. IEEE Transactions on Visualization and Computer Graphics, 26(2), 1413-1432. doi:10.1109/tvcg.2018.2872577 Surveyed the literature and proposed a task-based taxonomy of one hundred and fifty-four cognitive biases in seven main categories, organised by visualisation task rather than cognitive theory because existing taxonomies were hard to tie to tasks. It makes the reader's errors systematic and task-linked, not random.
Dimara, E. and Perin, C. (2020). What is Interaction for Data Visualization? IEEE Transactions on Visualization and Computer Graphics, 26(1), 119-129. doi:10.1109/tvcg.2019.2934283 Found that interaction, though fundamental to visualisation, was ambiguous and confusing, and traced the confusion to the absence of a consensual definition. They synthesised views from across visualisation and compared them with human-computer interaction before proposing a definition as a thinking tool. It shows the units of an evaluation are themselves contested.
Hullman, J., Qiao, X., Correll, M., Kale, A. and Kay, M. (2019). In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation. IEEE Transactions on Visualization and Computer Graphics, 25(1), 903-913. doi:10.1109/tvcg.2018.2864889 Built a taxonomy separating the six levels of decision in an uncertainty evaluation, from behavioural target to analysis, and applied it to eighty-six user studies. They found practice concentrated on performance and satisfaction measures that assume a more predictable judgement than research on human judgement supports.
North, C. (2006). Toward measuring visualization insight. IEEE Computer Graphics and Applications, 26(3), 6-9. doi:10.1109/mcg.2006.70 Argued that the purpose of visualisation is insight, named characteristics of insight including complexity, depth, unexpectedness and relevance, and argued that controlled experiments with benchmark tasks do not effectively measure insight. It is the unit's clearest statement that the default method tests less than the stated purpose.
Plaisant, C. (2004). The challenge of information visualization evaluation. Proceedings of the Working Conference on Advanced Visual Interfaces, 109-116. doi:10.1145/989863.989880 Argued that as information visualisation matures, usability studies and controlled experiments cannot account for the long exploratory nature of real tasks, the value of a possible discovery or the benefit of general awareness. She called for better metrics, benchmark repositories and reports of adoption.
Saraiya, P., North, C. and Duca, K. (2005). An Insight-Based Methodology for Evaluating Bioinformatics Visualizations. IEEE Transactions on Visualization and Computer Graphics, 11(4), 443-456. doi:10.1109/tvcg.2005.53 Developed a method that recognises and quantifies insight in open-ended user tests and applied it to five microarray visualisation tools, measuring the amount and type of insight and the time taken. They found that interaction techniques played a key role in producing insight.
Stasko, J. (2014). Value-driven evaluation of visualizations. Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, 46-53. doi:10.1145/2669557.2669579 Argued that low-level benchmark questions determine usability but overlook what visualisation uniquely provides, and proposed a value-driven evaluation built on four capabilities: answering diverse questions quickly, spurring insights and insightful questions, conveying the essence of the data, and generating confidence and domain knowledge. It widens the target to include the reader's confidence.
Wang, H. W. et al. (2025). Do You Trust This Visualization? An Inventory to Measure Trust in Visualizations. IEEE Transactions on Visualization and Computer Graphics. doi:10.1109/tvcg.2025.3646847 Derived an operational definition of trust by factor analysis, then validated an eight-item inventory in two trust games with real stakes. Gives one contested evaluation construct a comparable measure.
Zuk, T., Schlesier, L., Neumann, P., Hancock, M. S. and Carpendale, S. (2006). Heuristics for information visualization evaluation. Proceedings of the 2006 AVI Workshop on BEyond time and errors, 1-6. doi:10.1145/1168149.1168162 Ran a meta-analysis in which three published sets of heuristics assessed one visual decision support system for simulation data. Results depended heavily on the heuristics and evaluator types chosen; the sets carried interpretation, redundancy and conflict problems; and visualisation-specific heuristics caught faults usability heuristics would not.

Check your understanding & apply it · 4 questions, 1 essay

1A data team plans to test a published piece with a single multiple-choice comprehension quiz on the readings the piece explains, and to report that the piece 'works' if the scores are high. Using the unit's evidence, which judgement is defensible?

2Two teams run heuristic evaluation on the same visualisation and report substantially different lists of faults. Which diagnosis does the unit's evidence support?

3A designer assumes that any reader error on her chart is random and will average out across a sample. Using the unit's evidence, which prediction is best supported?

4A methodologist argues: 'An evaluation either measures the reader or it measures the work, and the field must decide which one is right.' Which judgement does the unit's reading of the evidence support?

5Essay A city data team has built an interactive map and chart showing monthly air-quality readings by neighbourhood, and plans to publish it with a short article. The team lead says the piece "does its job if people understand the readings" and wants a single multiple-choice comprehension quiz run with twenty colleagues before publication. Using this unit's sources, describe the evaluation you would design, naming the target you choose and at least one target you are knowingly giving up, and justify each choice from the literature. Say which systematic bias you would predict in the readers of the published piece, name one method you would add to the comprehension quiz and what it buys you, then state the strongest counter-argument to your approach and answer it.

I would begin by refusing the premise that there is one job the piece does, because the unit's whole point is that comprehension, insight, confidence and decision quality are different targets and a study of one says little about the others. The lead has named comprehension, and North (2006) is the reason I would not simply accept it: if the purpose of visualisation is insight, then a multiple-choice quiz on readings that were already explained measures the search we taught, not whether a resident learned anything about their own neighbourhood. So I would keep the comprehension quiz, because it is repeatable, but I would add an open-ended insight protocol of the kind Saraiya, North and Duca (2005) built, in which readers talk through the map and I count and type the observations they reach, accepting that it is labour-intensive and hard to standardise. That addition buys me the discovery-style finding a quiz cannot record, at a named cost. Knowingly, I would give up decision quality: twenty colleagues cannot supply a decision outcome, and Plaisant (2004) and Stasko (2014) ask for the value a tool delivers in real use, including confidence and adoption, which a pre-publication session cannot show. I would say so in the write-up rather than imply the quiz covered it. The bias I would predict is an order or framing effect from the map's default view, because Dimara, Franconeri, Plaisant, Bezerianos and Dragicevic (2020) show that biases in visualisation are systematic and task-linked, and a seasonal reading invites a rank that depends on the month shown. One method I would add beyond the quiz is an annotated pre-registered list of the map's known weaknesses, scored blind against a heuristic evaluation, following Zuk and colleagues (2006), who found that heuristic results depend on the heuristics and the evaluators chosen and that visualisation-specific heuristics catch problems usability heuristics miss. The strongest counter-argument is Plaisant's and Stasko's: that a comprehension and insight study on colleagues tells us nothing about whether the published piece changes what residents do, and the effort would be better spent tracking adoption. I would answer it by agreeing on the target and disagreeing on the timing: adoption data arrive only after publication, so the pre-publication study should be scoped to what it can honestly claim, and its limits stated.