Uncertainty: displaying the error and evaluating the decision

A point estimate tells a reader that the number is the answer; an interval or an ensemble tells the reader that the number is one draw among many. Encodings that beat the error bar on reading tasks now exist, but the evidence on decisions is weaker. This fourth of ten units, after the third mapped the colour, argues that the better display has changed decisions in some studies and worsened them in others. It also argues that the field's evaluations have largely measured reading accuracy rather than decision quality. Whether a better display produces a better decision is therefore not yet settled by the literature that would have to test it.

The choice is a trade, not a fix

Correll and Gleicher (2014) ran crowd-sourced experiments on how readers decide from mean and error data, and found that the encoding significantly changes those decisions. They did not recommend one replacement everywhere, proposing gradient plots, which use transparency, and violin plots, which use width, as better alternatives for inferential tasks than bar charts with error bars, and treating the differences as trade-offs.

Showing the distribution beats summarising it, on some tasks

The strongest positive results come from showing a sample of the distribution rather than a summary. Hullman, Resnick and Adar (2015) tested hypothetical outcome plots against error bars and violin plots. The plots animate a finite set of individual draws, and an animated plot needs a metered connection and a low-end device, so the same draws should also be readable as a static image. Readers were much more accurate on plots of two and three quantities, but no better on most single-quantity questions. Kale, Nguyen, Kay and Hullman (2019) found that readers asked which of two trends produced a noisy series recovered the correct trend at a lower level of evidence with such plots than with static displays. Animation helps where the reader must compare quantities, not as a general improvement to reading.

Displays built for a decision can improve one

Three studies show a display matched to a decision improving that decision. Kay, Kola, Hullman and Munson (2016) found quantile dotplots reduced the variance of readers' probability estimates by about 1.15 times compared with density plots on a bus arrival interface. Greis and colleagues (2018) found uncertainty information improved readers' estimate of the true value from two conflicting sensor readings, most where the readings differed most in variability. Joslyn and LeClerc (2012) give the strongest case: participants deciding whether to salt roads against a penalty for ice made better decisions with uncertainty forecasts, and were less deterred when forecast error increased.

The same class of display can mislead on another task

Padilla, Ruginski and Creem-Regehr (2017) compared summary and ensemble displays of hurricane track forecasts. Readers misread salient features of the summary displays as size information, which those displays do not encode. Ensemble displays evoked readings closer to the underlying probability distribution, though point judgements from an ensemble overweighted individual members. Hofman, Goldstein and Hullman (2020) found the sharper result. They compared 95 per cent confidence intervals, carrying inferential uncertainty, with 95 per cent prediction intervals, carrying outcome uncertainty. They found readers shown confidence intervals overestimated the treatment effect and underestimated the variability of individual outcomes. Showing uncertainty is not one intervention with one sign. Which uncertainty is shown decided whether the display helped or harmed the inference.

Matzen and colleagues (2024) compared geospatial visualisations of wildfire risk with verbal descriptions of the same risk, and found that different representations of the same information produced different patterns of decision.

The readers include experts who also misread

Belia, Fidler, Williams and Cumming (2005) asked authors of journal articles in psychology, behavioural neuroscience and medicine to adjust a figure until two means with error bars were just significantly different. Of 473 respondents, many leading researchers held severe misconceptions about how error bars relate to significance, did not distinguish confidence intervals from standard error bars, and ignored whether the means were independent.

The field measures reading, not deciding

Hullman, Qiao, Correll, Kale and Kay (2019) set out a taxonomy of six levels of decisions an uncertainty evaluation must make and applied it to 86 studies, finding practice concentrated on performance and satisfaction measures that assume more predictable judgement than the research supports. Kinkeldey, MacEachren and Schiewe (2014) found from geospatial displays that most research develops new depictions of uncertainty and only a small part evaluates them empirically.

Practice has its own account of the gap. Boukhelifa and Duke (2009) reflected on an industrial case study of modelling and mapping underground assets, where uncertainty mattered and was still not shown, and named three limits on the value of the depiction: the quality and scope of the uncertainty data, a low confidence in the data itself, and the perceptual confusion a depiction can cause. Their argument is that the gap is not only technical.

Accuracy and decision quality come apart, and the newest review inherits the gap

Kale, Kay and Hullman (2021) ran a mixed-design experiment on eight uncertainty designs, each with and without a mean, and measured both estimation and decisions. Adding a mean had small biasing effects on both. The decisive result is that the designs supporting the least biased effect size estimation were not those supporting the best decision making, so an evaluation that measures only accuracy is not a proxy for an evaluation of decisions.

Heltne and colleagues (2023) systematically reviewed 29 studies of formats for communicating measurement uncertainty in standardised clinical assessments. Distribution-shape formats promoted more accurate probability estimation. Participants did not seem to benefit from the added information during decision-making tasks. No reviewed study used a clinician sample with a clinical decision scenario.

Contested question: does showing uncertainty improve decisions, or only reading?

The disagreement is about what those positive results license. One position holds that the improvement is real but domain-bound, and that its repetition makes the general claim defensible. Joslyn and LeClerc (2012) improved decisions where decision, cost and outcome were defined. Greis and colleagues (2018) improved the aggregation of conflicting readings. Kale, Kay and Hullman (2021) found display choices moved decisions.

The other holds that the evidence does not yet support the general claim, and that a display can make a decision worse. Hofman, Goldstein and Hullman (2020) found confidence intervals led readers to overestimate treatment effects. Heltne and colleagues (2023) found no decision-making benefit. Hullman and colleagues (2019) and Kinkeldey, MacEachren and Schiewe (2014) found evaluation has measured reading rather than deciding.

What would settle it is two studies. Heltne and colleagues (2023) name one as missing: a decision-level trial in the population the display serves. The other is an experiment measuring judgement and decision in one design, as Kale, Kay and Hullman (2021) did. Until both exist, a claim that a display improves decisions is a hypothesis with a good example behind it.

The two positions rest on different evidence. The positive side rests on decision experiments with defined tasks (Joslyn and LeClerc, 2012; Greis et al., 2018; Kale, Kay and Hullman, 2021). The negative side rests on controlled experiments on how readers read the display (Padilla, Ruginski and Creem-Regehr, 2017; Hofman, Goldstein and Hullman, 2020). It also rests on a systematic review of twenty-nine studies (Heltne et al., 2023). While the dispute is open, show the uncertainty the decision turns on and name its kind, then check the reading on the task rather than assume the display helped.

The strongest counter-argument

The strongest objection is that the field can already say what the thesis claims it cannot. Joslyn and LeClerc (2012) measured decision quality directly and found improvement, and Kale, Kay and Hullman (2021) measured decisions and found them moved. The reply is that the demonstrations are real but not general. Each holds for a task, domain and measure its authors chose, and none recruits the population the display is meant to serve. That is the study Heltne and colleagues (2023) found missing.

Where the area is heading

The direction is towards decision-level evaluation and away from the chart as the unit of analysis. Heltne and colleagues (2023) close with recommendations for the remaining questions, and Kale, Kay and Hullman (2021) propose modelling effectiveness through the strategies users actually employ. Spiegelhalter, Pearson and Short (2011) noted that evidence on how these displays are processed was limited. The open work is not a better palette but a better experiment.

The sources this unit rests on

Each note states what the cited document did and found, from the resolved record and abstract.

• Boukhelifa, N. and Duke, D. J. (2009). Uncertainty visualization: why might it fail? doi:10.1145/1520340.1520616 Reflected on an industrial case study of modelling and mapping underground assets, where uncertainty mattered but was rarely shown, and named three limits on its depiction: the quality and scope of the uncertainty data, a low confidence in the data itself, and the confusion a depiction can cause. It reports practice rather than an experiment.
• Belia, S., Fidler, F., Williams, J. and Cumming, G. (2005). Researchers Misunderstand Confidence Intervals and Standard Error Bars. doi:10.1037/1082-989x.10.4.389 Asked journal authors to adjust a figure until two means with error bars were just significantly different; of 473 respondents, many leading researchers misread how error bars relate to significance. Experts misread these displays too.
• Correll, M. and Gleicher, M. (2014). Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error. doi:10.1109/tvcg.2014.2346298 Ran crowd-sourced experiments and found the encoding of mean and error significantly changes readers' decisions, recommending gradient and violin plots as better alternatives for inferential tasks than bar charts with error bars.
• Greis, M., Joshi, A., Singer, K., Schmidt, A. and Machulla, T. (2018). Uncertainty Visualization Influences how Humans Aggregate Discrepant Information. doi:10.1145/3173574.3174079 Gave readers two conflicting sensor readings and tested four uncertainty representations; uncertainty information improved their estimates of the true value, most where the readings differed most in variability. The error improved a judgement combining uncertain sources.
• Heltne, A., Frans, N., Hummelen, B., Falkum, E., Germans Selvik, S. and Paap, M. C. S. (2023). A systematic review of measurement uncertainty visualizations in the context of standardised assessments. doi:10.1111/sjop.12918 Reviewed 29 studies of clinical measurement-uncertainty formats: distribution-shape displays improved probability estimation, but the added information did not seem to help decision tasks, and no reviewed study used a clinician sample with a decision scenario.
• Hofman, J. M., Goldstein, D. G. and Hullman, J. (2020). How Visualizing Inferential Uncertainty Can Mislead Readers About Treatment Effects in Scientific Results. doi:10.1145/3313831.3376454 Compared 95 per cent confidence intervals with prediction intervals across two experiments: readers shown confidence intervals overestimated the treatment effect and underestimated the variability of individual outcomes. Which uncertainty is displayed can make a decision worse.
• Hullman, J., Resnick, P. and Adar, E. (2015). Hypothetical Outcome Plots Outperform Error Bars and Violin Plots for Inferences about Reliability of Variable Ordering. doi:10.1371/journal.pone.0142444 Tested animated hypothetical outcome plots against error bars and violin plots: readers were much more accurate on plots of two and three quantities, but no better on most single-quantity questions. It is the primary positive result for animated uncertainty, bounded to comparison tasks.
• Hullman, J., Qiao, X., Correll, M., Kale, A. and Kay, M. (2019). In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation. doi:10.1109/tvcg.2018.2864889 Set out a taxonomy of six levels of evaluation decisions and applied it to 86 studies, finding practice concentrated on performance and satisfaction measures with a bias towards accuracy rather than decision quality.
• Joslyn, S. L. and LeClerc, J. E. (2012). Uncertainty forecasts improve weather-related decisions and attenuate the effects of forecast error. doi:10.1037/a0025185 Had participants decide whether to salt roads against a penalty for ice across three experiments; uncertainty forecasts improved decision quality and lessened the reluctance to act when forecast error increased, while categorical advice alone did not help. It is the clearest decision-level benefit.
• Kale, A., Nguyen, F., Kay, M. and Hullman, J. (2019). Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data. doi:10.1109/tvcg.2018.2864909 Ran two crowdsourced experiments in which readers inferred which of two trends produced a noisy series; they recovered the correct trend at a lower level of evidence with hypothetical outcome plots than with static displays. It extends the animation result to a trend judgement.
• Kale, A., Kay, M. and Hullman, J. (2021). Visual Reasoning Strategies for Effect Size Judgments and Decisions. doi:10.1109/tvcg.2020.3030335 Ran a mixed-design experiment on eight uncertainty designs, each with and without a mean, measuring estimation and decisions: the designs supporting the least biased effect size estimation were not those supporting the best decisions. Accuracy and decision quality can diverge in one experiment.
• Kay, M., Kola, T., Hullman, J. R. and Munson, S. A. (2016). When (ish) is My Bus?: User-centred Visualizations of Uncertainty in Everyday, Mobile Predictive Systems. doi:10.1145/2858036.2858558 Designed uncertainty displays for mobile transit predictions, including quantile dotplots, and found in a controlled experiment that they reduced the variance of readers' probability estimates by about 1.15 times compared with density plots.
• Kinkeldey, C., MacEachren, A. M. and Schiewe, J. (2014). How to Assess Visual Communication of Uncertainty? A Systematic Review of Geospatial Uncertainty Visualisation User Studies. doi:10.1179/1743277414y.0000000099 Systematically reviewed user studies of geospatial uncertainty visualisation and found that most research develops new depictions while only a small part evaluates them empirically. The measurement problem is field-wide, not local to one group.
• Matzen, L. E., Howell, B. C., Tuft, M. and Divis, K. (2024). Transparent Risks: The Impact of the Specificity and Visual Encoding of Uncertainty on Decision Making. doi:10.1111/cgf.15094 Compared geospatial visualisations of wildfire risk with verbal descriptions of the same risk across three experiments and found that different representations of the same information produced different patterns of decision. It adds a current decision-level comparison to a unit whose evidence on decisions is thin.
• Padilla, L. M., Ruginski, I. T. and Creem-Regehr, S. H. (2017). Effects of ensemble and summary displays on interpretations of geospatial uncertainty data. doi:10.1186/s41235-017-0076-1 Compared summary and ensemble displays of hurricane track uncertainty; summary features were misread as size information, ensemble displays evoked readings closer to the underlying distribution, and point judgements overweighted individual members.
• Spiegelhalter, D., Pearson, M. and Short, I. (2011). Visualizing Uncertainty About the Future. doi:10.1126/science.1191181 Reviewed how uncertainty about the future is communicated visually, finding limited experimental evidence on how these displays are processed, an effect of audience numeracy, and increasingly easy interactive presentation. It dates this unit's limited evidence base.