Interaction, coordinated views and visual analytics

Visual analytics rests on a claim that interaction is what makes scale manageable. A data set too large to draw at once becomes tractable when the reader can filter, zoom and link views. This fifth of ten units, after the fourth displayed the uncertainty, finds the support for that claim thin in one specific way. The field built a vocabulary of interaction, guidelines for coordinated views, and systems that worked in use. It did not, for the most part, build a controlled comparison of interaction techniques against a defined measure of analytic outcome. The claim is therefore a design argument supported by case studies and one strong early result, not a mature experimental literature.

Scale was the problem, and interaction was named as the answer

Shneiderman (1996) proposed the visual information seeking mantra: overview first, zoom and filter, then details on demand. The same paper offered a taxonomy of seven data types, from one, two and three dimensional data through temporal and multi dimensional data to tree and network data. It also listed seven tasks, among them overview, zoom, filter, relate, history and extract. Shneiderman calls the mantra only a starting point. The claim that interaction manages scale therefore arrived as an organising heuristic rather than a finding.

Thomas and Cook (2005) defined visual analytics as the science of analytical reasoning facilitated by interactive visual interfaces. Keim and colleagues (2008) gave the definition now repeated. Visual analytics combines automated analysis techniques with interactive visualisations, for understanding, reasoning and decision making on very large and complex data sets, with a process from data through models to knowledge. Both place interaction at the centre of the answer to scale.

A vocabulary arrived before an evidence base

Interaction needed names before it could be compared, and naming was the field's first contribution. Heer and Shneiderman (2012) set out twelve interactive task types in three groups. The groups are data and view specification (visualise, filter, sort, derive), view manipulation (select, navigate, coordinate, organise) and process and provenance (record, annotate, share, guide). Yi and colleagues (2007) reviewed interactive capability across systems. They proposed seven categories organised around the user's intent: select, explore, reconfigure, encode, abstract or elaborate, filter, and connect.

Yi and colleagues made the gap explicit. Interaction had attracted relatively little attention from the community, and the frameworks that existed covered low level operations rather than the benefits interaction provides. A taxonomy is not evidence that interaction works, only the precondition for testing it, and neither list names the input device, so a reader who reaches a linked view with a keyboard has no place in the vocabulary yet.

Coordinated views are a design answer that carries its own caution

Wang Baldonado, Woodruff and Kuchinsky (2000) gave eight guidelines for multiple view systems. Four decide whether multiple views are warranted (diversity, complementarity, decomposition, parsimony), and four present them (space and time resource optimisation, self evidence, consistency, attention management). The rule of parsimony states the cost: several views impose context switching and attention, so an extra view must be justified.

North and Shneiderman (2000) built a system in which users assemble coordinated views without programming and ran two studies. In the first, six users, three of them analysts or statisticians at the census bureau and three programmers, built coordinated interfaces of their own. The second put three interfaces in front of eighteen participants on nine browsing tasks over census data: a scrolling detail view, the same view with an uncoordinated overview added, and the two views coordinated. The coordinated pair was about 80 per cent faster than the detail view on the easy tasks and 30 to 50 per cent faster on the difficult ones, both significant. The uncoordinated overview matched it only where the overview alone carried the answer. It fell to the detail-only time where the detail was needed. That is the mechanism: coordination pays when a task has to cross between the two views. It remains one coordination type, in one system, on constructed tasks, measured by speed rather than analytic quality.

The systems worked in use, and use is not comparison

The field's strongest cases are systems that plainly helped someone. Stasko, Gorg and Liu (2008) built Jigsaw, a visual analytics system with multiple coordinated views of documents and their entities. They entered the 2007 VAST analytical contest using it, and won the university division. The paper reports a professional analyst working with it in a live session. The evidence it offers is contest performance and expert reaction rather than comparison. Gotz and Zhou (2008) logged how users acted in a visual analytics tool and characterised analytic activity at several levels of granularity. They identified a level they called Actions, which can serve as building blocks for a record of how an insight was derived.

Pirolli and Card (2005) modelled intelligence analysis from cognitive task analysis and think aloud protocols. The model has a foraging loop of searching, filtering and reading, and a sensemaking loop of building and revising a representation that fits the evidence. Sacha and colleagues (2014) tied such accounts into a knowledge generation model, with a computer side and a human side that runs through exploration, verification and knowledge generation.

Contested question: has interaction been shown to make scale manageable?

The disagreement is about what the field's cases license. One position holds that interaction is demonstrably the mechanism. North and Shneiderman (2000) measured a large performance gain from one coordination. Yi and colleagues (2007) named the benefits interaction provides, and Stasko, Gorg and Liu (2008) showed a coordinated system performing in a competitive task. The claim is then established at the level of instances.

The other holds that the evidence is thinner than the claim. Wang Baldonado, Woodruff and Kuchinsky (2000) drew guidelines from a workshop and design experience. Heer and Shneiderman (2012) offered a taxonomy illustrated with systems. Roberts (2007) reviewed fifteen years of coordinated and multiple views. He found early work on models and techniques and later work on domain application, and concluded that much research remained. Dimara and Perin (2020) found that interaction in visualisation is ambiguous, because the field lacks a consensual definition. Two techniques cannot be compared if the term does not name one thing. Dimara and colleagues (2020) added that empirical research on how cognitive biases affect visual data analysis is limited. Whether interaction mitigates or introduces error is therefore largely unmeasured.

What would settle it is two kinds of study. One is a controlled comparison of named interaction techniques against a defined analytic outcome rather than task time. The other is a study of interaction in use over time with the process recorded. Provenance logging (Gotz and Zhou, 2008) and the process models (Pirolli and Card, 2005; Sacha et al., 2014) make that possible.

The two positions rest on different evidence. The affirmative position rests on one controlled comparison with a significant speed gain (North and Shneiderman, 2000) and on systems reported from contest use (Stasko, Gorg and Liu, 2008). The sceptical position rests on guidelines from workshop experience (Wang Baldonado, Woodruff and Kuchinsky, 2000), a state-of-the-art review (Roberts, 2007) and a conceptual review of interaction (Dimara and Perin, 2020). While the dispute is open, treat coordination as a hypothesis to test on the task, and record the process rather than the time alone.

The strongest counter-argument

The strongest objection is that the thesis denies evidence that exists. North and Shneiderman (2000) did run controlled user studies, comparing a coordinated interface against detail only and uncoordinated baselines. They reported improvements of 30 to 80 per cent on most tasks, so the field may have one strong early result that it has not generalised. The reply concedes the result and holds the boundary. It is one coordination type, in one system, with data savvy users on constructed tasks, measured by speed. The fifteen years Roberts (2007) reviews produced no comparative programme around it. The thesis is therefore bounded: interaction has been shown to help in a specific configuration, and not to be the general mechanism that makes scale manageable.

Where the area is heading

The direction is away from the mantra and towards the process. Dimara and Perin (2020) propose a definition of interaction meant as a thinking tool for bolder design. Dimara and colleagues (2020) organise 154 cognitive biases into seven task categories, so that a bias can be attached to a stage of a visualisation task rather than to a cognitive theory. Sacha and colleagues (2014) offer their knowledge generation model as a common language across analysis systems, and Gotz and Zhou (2008) treat logged action as the raw material for insight provenance. The work these papers name as open is a measured account of what interaction does between the data and the knowledge.

The sources this unit rests on

Each note states what the cited document did and found, from the resolved record and abstract or full text.

• Dimara, E. and Perin, C. (2020). What is Interaction for Data Visualization? doi:10.1109/tvcg.2019.2934283 Reviewed definitions of interaction across visualisation, visual analytics, scientific visualisation and human-computer interaction, and argued that the field's confusion follows from the absence of a consensual definition. That missing definition is the obstacle to comparing techniques.
• Dimara, E., Franconeri, S., Plaisant, C., Bezerianos, A. and Dragicevic, P. (2020). A Task-Based Taxonomy of Cognitive Biases for Information Visualization. doi:10.1109/tvcg.2018.2872577 Proposed a taxonomy of 154 cognitive biases in seven main categories, organised by the stage of the visualisation task rather than by cognitive theory. They report that empirical research on how these biases affect visual data analysis is limited.
• Gotz, D. and Zhou, M. X. (2008). Characterizing users' visual analytic activity for insight provenance. doi:10.1109/vast.2008.4677365 Logged how users acted in a visual analytics tool and characterised analytic activity at several levels of granularity, identifying a level they called Actions as building blocks for a record of how an insight was derived. The analytic process is therefore recordable.
• Heer, J. and Shneiderman, B. (2012). Interactive dynamics for visual analysis. doi:10.1145/2133806.2133821 Presented a taxonomy of twelve interactive task types in three groups, data and view specification, view manipulation, and process and provenance, illustrated with real systems rather than tested. The unit uses it as the vocabulary interaction now has.
• Keim, D., Andrienko, G., Fekete, J.-D., Gorg, C., Kohlhammer, J. and Melancon, G. (2008). Visual Analytics: Definition, Process, and Challenges. doi:10.1007/978-3-540-70956-5_7 Defined visual analytics as the combination of automated analysis techniques with interactive visualisations for understanding, reasoning and decision making on very large and complex data sets, with a process from data through models to knowledge. That is the claim the unit examines.
• North, C. and Shneiderman, B. (2000). Snap-together visualization: can users construct and operate coordinated visualizations? doi:10.1006/ijhc.2000.0418 Built a system in which users assemble coordinated views without programming, and ran two studies. Six data savvy users, census statisticians and programmers, built coordinated interfaces of their own. In the second, eighteen participants used three interfaces on nine census browsing tasks, and a coordinated overview and detail pair was about 80 per cent faster than the detail view on the easy tasks and 30 to 50 per cent faster on the difficult ones, both significant, while an uncoordinated overview helped only where the overview alone carried the answer.
• Pirolli, P. and Card, S. (2005). The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. URL: https://andymatuschak.org/files/papers/Pirolli,%20Card%20-%202005%20-%20The%20sensemaking%20process%20and%20leverage%20points%20for%20analyst%20technology%20as.pdf Derived a notional model of intelligence analysis from cognitive task analysis and think aloud protocols, organised into a foraging loop of searching, filtering and reading and a sensemaking loop of building and revising a representation that fits the evidence. The loops name the leverage points where technology can intervene.
• Roberts, J. C. (2007). State of the Art: Coordinated and Multiple Views in Exploratory Visualization. doi:10.1109/cmv.2007.20 Reviewed fifteen years of coordinated and multiple views, observing that early work concerned models and techniques while later work applied the ideas to domains, and argued that much research remained. The unit uses it as a state of the art that reads as a research agenda.
• Sacha, D., Stoffel, A., Stoffel, F., Kwon, B. C., Ellis, G. and Keim, D. A. (2014). Knowledge generation model for visual analytics. doi:10.1109/tvcg.2014.2346481 Proposed a knowledge generation model tying earlier frameworks together, with a computer side of data, visualisation and model and a human side running through exploration, verification and knowledge generation. The unit uses it as the process account a study of interaction would need.
• Shneiderman, B. (1996). The eyes have it: a task by data type taxonomy for information visualizations. doi:10.1109/vl.1996.545307 Proposed the visual information seeking mantra, overview first, zoom and filter, then details on demand, with a taxonomy of seven data types and seven tasks. The paper presents the mantra as a starting point rather than a finding.
• Stasko, J., Gorg, C. and Liu, Z. (2008). Jigsaw: Supporting Investigative Analysis through Interactive Visualization. doi:10.1057/palgrave.ivs.9500180 Built a visual analytics system with multiple coordinated views of documents and their entities, used it in the 2007 VAST analytical contest, won the university division, and reports a professional analyst using it in a live session. The evidence is contest performance and expert reaction rather than comparison.
• Thomas, J. J. and Cook, K. A. (2005). Illuminating the Path: The Research and Development Agenda for Visual Analytics. URL: https://www.pnnl.gov/publications/illuminating-path-research-and-development-agenda-visual-analytics Defined visual analytics as the science of analytical reasoning facilitated by interactive visual interfaces, and set out the field's research and development priorities. The unit uses it as the founding claim about interaction.
• Wang Baldonado, M. Q., Woodruff, A. and Kuchinsky, A. (2000). Guidelines for using multiple views in information visualization. doi:10.1145/345513.345271 Presented eight guidelines for multiple view systems, four for deciding whether multiple views are warranted (diversity, complementarity, decomposition, parsimony) and four for presenting them (space and time optimisation, self evidence, consistency, attention management). Their basis was workshop discussion and design experience, and the rule of parsimony warns that extra views cost attention.
• Yi, J. S., Kang, Y., Stasko, J. and Jacko, J. A. (2007). Toward a Deeper Understanding of the Role of Interaction in Information Visualization. doi:10.1109/tvcg.2007.70515 Reviewed interactive capability across information visualisation systems and proposed seven categories organised around the user's intent: select, explore, reconfigure, encode, abstract or elaborate, filter and connect. They report that interaction had received relatively little attention and that existing frameworks covered low level operations.