Evaluating visualisation: insight, metrics and the credibility of the evidence

How the field evaluates a visualisation has changed, and the change has made its evidence look weaker than its claims. Evaluation began with the controlled comparison: two displays, one task, accuracy and time. It moved towards insight, decision quality and the analysis people actually do. This ninth of ten units, after the eighth built the dashboard, argues that the shift is right and incomplete. The outcome measures are young, the comparisons are seldom repeated, and the statistics overstate what one study shows. Put one question to any visualisation claim: what was measured, on whose task, by what method, and has anyone run it twice.

Evaluation is a design decision, not a default

Plaisant (2004) reviewed visualisation evaluation and named why the default instrument under-serves the field: tasks are hard to define, the tools are hard to compare, and real use takes a long time. Carpendale (2008) set out the space of methods, from laboratory experiments and judgement studies to case studies and field work, and the trade-off between generalisability, precision and realism. Lam and colleagues (2012) derived seven scenarios from a review of over eight hundred publications, from evaluating visual data analysis and reasoning to collaborative analysis. They argued that the scenario should determine the method and the measures.

Insight was proposed as the goal before it could be measured

North (2006) proposed that the purpose of visualisation is insight, so evaluation should measure how well a display generates it, and asked whether the controlled experiment can do so at all. He named the costs, including insight's subjectivity and the expertise its coding requires. Saraiya, North and Duca (2005) tested the position. They defined an insight as an individual observation a think-aloud protocol can record, and ran thirty participants, each on one of five microarray tools and one of three datasets, without benchmark tasks. Insight counts and domain value did not depend on the participant's background; believed learning did, with novices overconfident and software developers sceptical. The sample behind every count here is assumed to be of working age, at ease with a screen and able to give up an hour to a session.

What a controlled comparison can and cannot settle

Stasko and colleagues (2000) ran the missing controlled comparison. They set two space-filling tools for file hierarchies, the rectangular Treemap and the circular Sunburst, against each other on search and analysis tasks, first on small hierarchies and then on larger ones. Performance trends generally favoured Sunburst, particularly on first use and on the larger hierarchies, and each tool afforded different search strategies. Performance with Treemap improved with practice, suggesting a greater learning cost. Participants strongly preferred Sunburst for conveying structure. The study ranks two tools on a task, and it does not show that either improved anyone's work.

Grounded evaluation and the laboratory's blind spot

Isenberg and colleagues (2008) proposed grounded evaluation, situating a study in the context of a tool's intended use and in the questions of the people who will use it, and arguing for qualitative inquiry early in the development life cycle. Their three case studies show observation informing understanding in group visual analytics, medical diagnostic reasoning and visualisation use among computational linguists. Greenberg and Buxton (2008) went further: evaluation done by rule rather than by thought can be ineffective and even harmful, killing ideas during brainstorming and a vision when run prematurely.

The measurement apparatus is extensive and loosely tied to outcomes

Behrisch and colleagues (2018) surveyed quality metrics for visualisation, organising a large corpus by data type and formalising what a metric consists of. Micallef and colleagues (2017) show what a validated metric can do: a perceptual cost function for scatterplot design, pre-calibrated for correlation estimation, class separation and outlier detection, produced designs matching hand-made presets such as those in R or MATLAB. That is the strongest reply to the thesis, within three calibrated tasks.

Bertini, Tatu and Keim (2011) systematised the techniques that use a quality metric to steer exploration of high-dimensional data. They sorted them by the pipeline stage they act on, data transformation, view transformation and visual mapping. The metric exists to steer a search too large to run by eye, narrower than judging whether a view answers the reader's question.

Exploration can manufacture its own findings

Zgraggen and colleagues (2018) measured the accuracy of insights users reported on synthetic datasets with known ground truth, and over sixty per cent were false, because a reader who examines many views makes many comparisons and the chance of a spurious pattern rises with the number. Hullman and Gelman (2021) give the theoretical reading: interactive exploratory analysis is a form of statistical inference, and design without a theory of human inference risks confusing a need for flexibility with the conclusion that exploration can be model free. Their Bayesian model check unites exploration and confirmation.

Replications are rare, which is not the same as robust

Kosara and Haroz (2018) surveyed replication in visualisation research and examined six threats to study validity, among them statistical flukes, questionable research practices and overgeneralised conclusions. They state that replications are rare, and that if they were common, a rate of unreproducible results similar to that in psychology and the social sciences would be unsurprising. Dragicevic (2016) supplies the reporting standard, arguing for effect sizes with interval estimates rather than p-values, and for never wording a null result as evidence of no effect.

The infrastructure for running the missing comparisons has also arrived. reVISit (Ding et al., 2023) is an open-source toolkit that compiles a study description and stimuli into a deployable web experiment with behaviour provenance and monitoring, demonstrated by re-implementing a graphical perception task and an interactive study.

Contested question: is the evidence base thin, or is the standard wrong?

The first position is that the evidence base is thin. Kosara and Haroz (2018) hold that replications are rare and that many studies have underlying problems. Zgraggen and colleagues (2018) measured over sixty per cent of reported insights as false. Dragicevic (2016) argues that the field's reporting overstates its results. The fix, on this reading, is more and better studies.

The second position is that the standard is applied to the wrong kind of work. Greenberg and Buxton (2008) and Isenberg and colleagues (2008) locate the problem in the over-use of one method rather than in the number of studies. Carpendale (2008) and Lam and colleagues (2012) hold that the method should fit the question, so a design study's claim about building a system for a context is not the kind a replication settles. The fix is methodological variety, and the thesis mistakes a difference of kind for a deficit.

What would settle it is different evidence on each side. Test the thin-evidence position with an audit coding published visualisation comparisons for effect sizes and intervals, then run preregistered replications of canonical comparisons, such as the perceptual ranking of encodings. Test the wrong-standard position by showing that studies from different scenarios answer questions no single outcome measure could span. Lam and colleagues (2012) assert this, and nobody has demonstrated it.

The two positions rest on different evidence. The thin-evidence position rests on a survey of replication (Kosara and Haroz, 2018), a measured multiple-comparisons study (Zgraggen et al., 2018) and a reporting standard (Dragicevic, 2016). The wrong-standard position rests on two methodological arguments (Greenberg and Buxton, 2008; Isenberg et al., 2008) and two reviews of evaluation scenarios (Carpendale, 2008; Lam et al., 2012). While the dispute is open, match the method to the question, report an interval around the effect, and treat a single study as one estimate rather than a finding.

The strongest counter-argument

The strongest objection is that the thesis applies an experimenter's yardstick to a design field. Carpendale (2008) and Lam and colleagues (2012) hold that a study's value is relative to its question, and Isenberg and colleagues (2008) that grounded evaluation is complementary rather than inferior. The reply narrows the thesis: the credibility problem belongs to claims presented as general findings about readers, tasks or decisions, and the design contributions of the same literature are not implicated.

Where the area is heading

The direction is from reporting an outcome to stating and testing the inference behind it. Hullman and Gelman (2021) name the missing theory and the experiment that would develop it, Zgraggen and colleagues (2018) supply the tool-side version, and Kosara and Haroz (2018) name the structural fix, publication models that let a replication be published. The study the field is short of runs a canonical comparison a second time, with an interval around the result.

The sources this unit rests on

Each note states what the cited document did and found, from the resolved record and abstract.

• Saraiya, P., North, C. and Duca, K. (2005). An Insight-Based Methodology for Evaluating Bioinformatics Visualizations. doi:10.1109/tvcg.2005.53 Counted insights in think-aloud sessions with thirty participants, five microarray tools and three datasets, without benchmark tasks; measured insight did not depend on background, believed learning did. A new outcome measure carries its own divergence and cost.
• North, C. (2006). Toward measuring visualization insight. doi:10.1109/mcg.2006.70 Argued that the purpose of visualisation is insight and that evaluation should measure how well a display generates it, and asked whether the controlled experiment can. It is the manifesto for the shift, and names insight's subjectivity and coding cost.
• Plaisant, C. (2004). The challenge of information visualization evaluation. doi:10.1145/989863.989880 Reviewed the state of evaluation and named the obstacles: hard-to-define tasks, tools that are hard to compare, and the long time real use requires. It dates the field's awareness of the gap and calls for benchmark repositories and adoption reports.
• Lam, H., Bertini, E., Isenberg, P., Plaisant, C. and Carpendale, S. (2012). Empirical Studies in Information Visualization: Seven Scenarios. doi:10.1109/tvcg.2011.279 Distinguished seven scenarios for empirical studies of visualisation, after a review of over eight hundred publications, from evaluating visual data analysis and reasoning to collaborative analysis. The scenario should determine the method and the measures.
• Isenberg, P., Zuk, T., Collins, C. and Carpendale, S. (2008). Grounded evaluation of information visualizations. doi:10.1145/1377966.1377974 Proposed grounded evaluation, situating a study in a tool's intended context of use and in its users' questions, and argued for qualitative inquiry early in development; three case studies cover group analytics, medical diagnosis and computational linguistics. It complements rather than replaces the controlled study.
• Carpendale, S. (2008). Evaluating Information Visualizations. doi:10.1007/978-3-540-70956-5_2 Set out the space of evaluation methods, from laboratory experiments to case studies and field work, and the trade-off between generalisability, precision and realism. The methodology should fit the research question, so the measures must match the claim.
• Stasko, J., Catrambone, R., Guzdial, M. and McDonald, K. (2000). An evaluation of space-filling information visualizations for depicting hierarchical structures. doi:10.1006/ijhc.2000.0420 Compared two space-filling tools for file hierarchies, the rectangular Treemap and the circular Sunburst, in two studies on search and analysis tasks. Performance trends favoured Sunburst, particularly on first use, while performance with Treemap improved with practice and each tool afforded different search strategies; participants preferred Sunburst for conveying structure. It shows what a comparison ranks and what it leaves open.
• Greenberg, S. and Buxton, B. (2008). Usability evaluation considered harmful (some of the time). doi:10.1145/1357054.1357074 Argued that usability evaluation applied by rule rather than by thought can be ineffective and even harmful, killing ideas in brainstorming and suggesting a prototype's worthiness rather than critiquing its adoption. It should not validate early design stages or culturally sensitive systems.
• Behrisch, M., Blumenschein, M., Kim, N. W., Shao, L., El-Assady, M., Fuchs, J. and others (2018). Quality Metrics for Information Visualization. doi:10.1111/cgf.13446 Surveyed quality metrics for visualisation, organised by data type from multi-dimensional to text data, and formalised the parts of a metric. The community holds many intuitions about the quality of a view and needs a common vocabulary.
• Bertini, E., Tatu, A. and Keim, D. (2011). Quality Metrics in High-Dimensional Data Visualization: An Overview and Systematization. doi:10.1109/tvcg.2011.229 Systematised the techniques that use a quality metric to steer exploration of high-dimensional data, organised by the pipeline stage they act on and by the search through alternative views they automate. It maps what the metrics compute, and tests no reader.
• Micallef, L., Palmas, G., Oulasvirta, A. and Weinkauf, T. (2017). Towards Perceptual Optimization of the Visual Design of Scatterplots. doi:10.1109/tvcg.2017.2674978 Built a perceptual cost function for scatterplot design and used it in an optimiser, pre-calibrated for correlation estimation, class separation and outlier detection. Optimised designs matched human presets such as those in R or MATLAB, within the tasks calibrated.
• Zgraggen, E., Zhao, Z., Zeleznik, R. and Kraska, T. (2018). Investigating the Effect of the Multiple Comparisons Problem in Visual Analysis. doi:10.1145/3173574.3174053 Measured the accuracy of insights users reported on synthetic datasets with known ground truth: over sixty per cent were false, because more views mean more comparisons and more spurious patterns. A confirmatory approach accounting for all comparisons approaches the result of one with a validation dataset.
• Hullman, J. and Gelman, A. (2021). Designing for Interactive Exploratory Data Analysis Requires Theories of Graphical Inference. doi:10.1162/99608f92.3ab8a587 Argued that interactive exploratory analysis is a form of statistical inference whose design should be grounded in theories of how readers draw inferences, since research without such grounding yields objectives that discourage valid inferences. The Bayesian model check unites exploration and confirmation.
• Kosara, R. and Haroz, S. (2018). Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them. doi:10.1109/beliv.2018.8634392 Surveyed replication in visualisation research and examined six threats to study validity, including statistical flukes and overgeneralised conclusions. Replications are rare, and a rate of unreproducible results similar to that in psychology and the social sciences would be unsurprising.
• Dragicevic, P. (2016). Fair Statistical Communication in HCI. doi:10.1007/978-3-319-26633-6_13 Set out how HCI studies should report statistics, arguing for an estimation approach with effect sizes and interval estimates rather than p-values, and recommending that a null result never be worded as evidence of no effect.
• Ding, Y., Wilburn, J., Shrestha, H., Ndlovu, A., Gadhave, K., Nobre, C., Lex, A. and Harrison, L. (2023). reVISit: Supporting Scalable Evaluation of Interactive Visualizations. doi:10.1109/vis54172.2023.00015 Built an open-source toolkit that compiles a study description and stimuli into a deployable web experiment with behaviour provenance and monitoring, demonstrated by re-implementing a graphical perception task and an interactive study. It lowers the cost of the repeated comparisons the unit says are rare.