Dashboards, design studies and analytics in organisations

A dashboard is one of the most common uses of data visualisation, and one of the least examined relative to its reach. It promises a decision from a glance at a screen, and it is built out of definitions: of the measure, of the period, of what counts as an exception. The literature supports a narrower claim. Dashboards succeed or fail on the decision they serve and on the definitions behind each measure. The research base is thin relative to their reach, so most dashboard guidance remains craft rather than evidence. This eighth of ten units, after the seventh named the deceptive chart, separates three things the field runs together: a design discipline, an evaluation method, and a claim about decisions.

A genre with no definition, whose claims outran the evidence

Sarikaya and colleagues (2019) began from an observation about dashboards. A dashboard is one of the most common uses of data visualisation, and its design and contexts of use differ from exploratory tools. Few (2006) gave the practitioner's account a decade earlier: a dashboard is read at a glance, so it should be organised around the reader's decisions rather than the available data.

Grammel, Tory and Storey (2010) had novices explore a fictitious sales dataset by describing the views they wanted to a mediator who built them in commercial software. The barriers were turning a question into attributes, designing the mapping and interpreting the outcome. Partial specifications were common and participants stayed with familiar forms. The bottleneck is the step before the chart, and the study measures a facilitated process, not a dashboard in use.

Yigitbasioglu and Velcu (2012) read across disciplines. They reported that, despite the popularity of dashboards, little was known about the extent of their effectiveness, that is, which dashboards work for which users or tasks. They closed with a research agenda. Their own recommendations, a high data-ink ratio, drill-down, flexibility and theory-driven warnings, are transferred from general visualisation principles rather than demonstrated on dashboards.

Why a laboratory task cannot see a dashboard

Controlled studies observe short-term use of a tool on preselected data and benchmark tasks. Shneiderman and Plaisant (2006) argued that this misses what an analytics tool is for. Their alternative documents usage by observation, interview, survey and logging, and asks whether expert users reach their own goals. Saraiya and colleagues (2006) ran the study. They followed a bioinformatics analysis from raw data to the insights sought, and reported that short studies miss long-term use, and that representations, interaction and insight generation changed across the work. The outcome of a dashboard is a goal reached, not accuracy on a task the experimenter chose.

The design study is how the field claims organisational knowledge

Sedlmair, Meyer and Munzner (2012) reflected on twenty-one design studies and defined the form. A researcher analyses a real problem faced by domain experts, designs a system that supports solving it, validates the design and reflects on lessons that refine guidelines. The methodology has nine stages and two axes, running from fuzzy to crisp tasks and from the expert's head to the computer. This is the field's main route to knowledge about analytics in organisations, and it stops short of the decision. A design study validates a design and produces a lesson. It does not measure whether the deployed system changed what anyone did.

The dashboard is the end of a pipeline, not the start

Every number on a dashboard has been defined upstream. Kandel and colleagues (2012) built a tool to find quality problems in tabular data. They argued that automated methods can flag anomalies, but that determining what counts as an error is context-dependent and requires human judgement. Battle and Heer (2019) reviewed how exploratory analysis is reported, and then logged twenty-seven Tableau users on three data sets. They found over eighty per cent accuracy on focused tasks, but a cadence far slower than the literature had assumed. A dashboard inherits both layers: a human definition of the measure, and an analysis process nobody has recorded.

The decision the dashboard claims to serve

Dimara and colleagues (2022) surveyed one hundred and twenty-two decision makers and forty-two analysts, and interviewed decision makers. They found them weighing uncertain or incomplete data against non-formalised knowledge in a multi-objective space, and distrusting analyst recommendations that arrive without context. What they need goes beyond displayed results: trade-off overviews, scenario-based analysis, interrogation tools, flexible data input and collaboration. Oral and colleagues (2023) then coded eighty-eight decision-focused tools against the intelligence, design and choice stages of decision making. They found most offering visibility for criteria and alternatives but limited flexibility, and little support for decision preferences. A dashboard read on a shared phone on a factory floor is a design condition too, and the measure definitions have to survive it.

Automation moves the definition into the software

Two design responses avoid asking the user to specify anything. SeeDB explores the space of visualisations for a data subset and recommends those it judges useful or interesting, using deviation from a reference as the utility metric. The paper reports orders-of-magnitude speedups and a user study (Vartak et al., 2015). Keshif aggregates records within attribute summaries and makes linked highlighting, filtering and comparison the basic moves, so a novice can build a dashboard without instruction in chart design (Yalcin, Elmqvist and Bederson, 2018). The definition of interesting, and the choice of what deserves a panel, sits in a rule inside the recommender.

Contested question: a design problem or a decision problem?

The first position is that dashboards fail because they are badly designed. Few (2006) named the failure modes. Yigitbasioglu and Velcu (2012) prescribed flexibility and drill-down, and Yalcin and colleagues (2018) automated good design. Bach and colleagues (2022) derived a pattern language from a hundred and forty-four dashboards.

The second position is that a well-designed display cannot fix a missing decision. Oral and colleagues (2023) locate the gap in the design and choice stages and in decision preferences, and Dimara and colleagues (2022) locate it in trade-offs, scenarios and context. A third position explains the dispute: the field's instruments cannot see the decision, as Shneiderman and Plaisant (2006) argued and Saraiya and colleagues (2006) demonstrated.

What would settle it is a study nobody has run. Rate a corpus of deployed dashboards for design quality, classify each by the decision it claims to serve, and measure the decision actually taken over time inside the organisation.

The two positions rest on different evidence. The design position rests on practitioner guidance (Few, 2006), a review that prescribes (Yigitbasioglu and Velcu, 2012) and a pattern language from a hundred and forty-four dashboards (Bach et al., 2022). The decision position rests on a survey of decision makers and analysts (Dimara et al., 2022) and a coding of eighty-eight tools (Oral et al., 2023). The instruments argument rests on two methodological papers (Shneiderman and Plaisant, 2006; Saraiya et al., 2006). While the dispute is open, define the decision and the measures with the people who own them, and treat a dashboard as a claim about a decision rather than a screen.

The strongest counter-argument

The strongest objection is that this thesis applies an experimenter's standard to a design field. Sedlmair and colleagues (2012) hold that a design study is a legitimate contribution in itself. Such a study produces knowledge about how to build systems for domain experts, and demanding a measured decision outcome misreads the method. Bach and colleagues (2022) strengthen it, since a hundred and forty-four dashboards analysed into a pattern language is empirical work at scale. The reply narrows the claim: the field has evidence about design and about analysis, and the decision remains the part with a research agenda attached.

Where the area is heading

The direction of travel is from describing dashboards to testing them against decisions. Bach and colleagues (2022) give the design vocabulary and a workshop method for using it. Oral and colleagues (2023) name the requirements on the decision side, preference elicitation and flexibility over alternatives and criteria. Dimara and colleagues (2022) supply the decision maker's own list of needs.

The sources this unit rests on

Each note states what the cited document did and found, from the resolved record and abstract.

• Few, S. (2006). Information Dashboard Design: The Effective Visual Communication of Data. URL: https://archive.org/details/informationdashb0000fews Argued that a dashboard is read at a glance, so its layout should follow the reader's decisions rather than the data available, and named the failures that follow from ignoring them. It argues by example and reports no study of readers or decisions, which is the craft the thesis contrasts with evidence.
• Grammel, L., Tory, M. and Storey, M. (2010). How Information Visualization Novices Construct Visualizations. doi:10.1109/tvcg.2010.164 Had novices explore a fictitious sales dataset by describing the views they wanted to a mediator who built them in commercial software. Three activities were central, attribute selection, template selection and mapping, and the barriers were turning questions into attributes, designing mappings and interpreting the result.
• Saraiya, P., North, C., Lam, V. and Duca, K. A. (2006). An Insight-Based Longitudinal Study of Visual Analytics. doi:10.1109/tvcg.2006.85 Followed a bioinformatics analysis from raw data to the insights the analyst sought, recording how representations, interaction and insight generation changed across the work. They report that short controlled studies miss the long-term use of a tool. The right outcome for analytics in organisations is a professional goal, not a task answer.
• Shneiderman, B. and Plaisant, C. (2006). Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. doi:10.1145/1168149.1168158 Proposed multi-dimensional in-depth long-term case studies, arguing that the value of a tool used by experts on their own data cannot be seen in a controlled laboratory task. It documents usage by observation, interview, survey and logging and asks whether expert users reach their professional goals. The unit's evidence is case-based for this reason.
• Kandel, S., Parikh, R., Paepcke, A., Hellerstein, J. M. and Heer, J. (2012). Profiler: integrated statistical analysis and visualization for data quality assessment. doi:10.1145/2254556.2254659 Built a tool that flags problematic values in tabular data and suggests coordinated summary views for assessing them in context. They report that quality problems undermine analysis and are costly to fix, and that what constitutes an error is context-dependent and needs judgement. The definition of a measure sits upstream of the dashboard.
• Sedlmair, M., Meyer, M. and Munzner, T. (2012). Design Study Methodology: Reflections from the Trenches and the Stacks. doi:10.1109/tvcg.2012.213 Reflected on twenty-one design studies to define the form and set out a nine-stage methodology with two axes, from fuzzy to crisp tasks and from the expert's head to the computer. It makes explicit how the field claims knowledge by building a system for real domain experts; the claim it validates is a system and a lesson, not a decision outcome.
• Yigitbasioglu, O. M. and Velcu, O. (2012). A review of dashboards in performance management: Implications for design and research. doi:10.1016/j.accinf.2011.08.002 Reviewed the multidisciplinary dashboard literature and reported that little was known about the extent of dashboards' effectiveness, that is, which dashboards work for which users or tasks, and closed with a research agenda. Its own advice, a high data-ink ratio and drill-down, is carried over from general visualisation principles. It shows both thin evidence and craft prescription.
• Vartak, M., Rahman, S., Madden, S., Parameswaran, A. and Polyzotis, N. (2015). SeeDB: efficient data-driven visualization recommendations to support visual analytics. doi:10.14778/2831360.2831371 Built an engine that explores the space of visualisations and recommends those it judges useful or interesting, using deviation from a reference as the utility metric, with orders-of-magnitude speedups and a user study. The definition of interestingness is the designer's, so the recommender moves a definition into software without testing it.
• Yalcin, M. A., Elmqvist, N. and Bederson, B. B. (2018). Keshif: Rapid and Expressive Tabular Data Exploration for Novices. doi:10.1109/tvcg.2017.2723393 Designed a system that aggregates records within attribute summaries and uses a consistent design based on data types, with linked highlighting, filtering and comparison replacing manual specification, so a novice can build a dashboard without chart-design instruction. The remedy arrives as automation, not as evidence about decisions.
• Battle, L. and Heer, J. (2019). Characterizing Exploratory Visual Analysis: A Literature Review and Evaluation of Analytic Provenance in Tableau. doi:10.1111/cgf.13678 Reviewed how exploratory analysis is reported and analysed the logged behaviour of twenty-seven Tableau users on three data sets, finding over eighty per cent accuracy on focused tasks but an analysis cadence far slower than published assumptions had suggested. The published record of analysis is thinner than the practice.
• Sarikaya, A., Correll, M., Bartram, L., Tory, M. and Fisher, D. (2019). What Do We Talk About When We Talk About Dashboards? doi:10.1109/tvcg.2018.2864903 Reviewed how dashboards are described in research and practice, found the term applied to several distinct artefacts from monitoring displays to analytic workspaces, and built a design space and major dashboard types. An object with several purposes needs a definition before evidence can accumulate, and this supplies it.
• Bach, B., Freeman, E., Abdul-Rahman, A., Turkay, C., Khan, S., Fan, Y. and Chen, M. (2022). Dashboard Design Patterns. doi:10.1109/tvcg.2022.3209448 Reviewed a hundred and forty-four dashboards and derived eight groups of design patterns, described their combination in genres such as narrative, analytical and embedded dashboards, and ran a two-week workshop with twenty-three participants. The authors report surprisingly little design guidance for dashboards. It is evidence about design at scale, not about decisions.
• Dimara, E., Zhang, H., Tory, M. and Franconeri, S. (2022). The Unmet Data Visualization Needs of Decision Makers Within Organisations. doi:10.1109/tvcg.2021.3074023 Surveyed one hundred and twenty-two decision makers and forty-two analysts, interviewed decision makers, and found them weighing uncertain or incomplete data against non-formalised knowledge in a multi-objective space and distrusting recommendations that arrive without context. They need trade-off overviews, scenarios, interrogation tools, flexible data input and collaboration, beyond displayed results.
• Oral, E., Chawla, R., Wijkstra, M., Mahyar, N. and Dimara, E. (2023). From Information to Choice: A Critical Inquiry Into Visualization Tools for Decision Making. doi:10.1109/tvcg.2023.3326593 Reviewed eighty-eight decision-focused visualisation tools, coding each for the visibility of alternatives, criteria and preferences and the flexibility to manipulate them across the intelligence, design and choice stages. Most offered visibility but limited flexibility, with a scarcity of tools supporting all three stages and little support for decision preferences.