Annotation and sequence: making a static figure carry an argument

Year 3 / Module 33 Data visualisation in practice / Unit 5 / Annotation and sequence: making a static figure carry an argument

Annotation and sequence: making a static figure carry an argument

HCI3109 · Unit 5 of 10

A static figure carries an argument through what the author selects, annotates and orders, and the narrative-visualisation literature shows that this framing is rhetoric, so the author's reading enters the chart and has to be disclosed.

Annotation and sequence: making a static figure carry an argument

A chart that shows only data leaves the reading to the reader. A reader with ten seconds will supply one. This fifth of ten units argues that a figure carries an argument through choices made before any mark is drawn. Those choices are which cases are included, which points are called out, and in what order the frames arrive. They are rhetoric, because they change what a reader takes from identical data. The author's reading has therefore already entered the figure, and it ought to be disclosed.

Selection is what a figure contains and omits. Annotation is what the author adds to the marks: labels, callouts, reference lines and a title that states the message. Sequence is the order in which the reader meets the frames.

Presentation is a design problem, not a finish applied afterwards

Kosara and Mackinlay (2013) argued that presentation, and the use of storytelling elements, is the next logical step for visualisation research, with weight equal to exploration and analysis.

Segel and Heer (2010) gave that agenda an empirical base. They analysed 58 published narrative visualisations, mainly from online journalism, and derived seven genres, among them the magazine style and the annotated chart. Their organising distinction is the balance between the narrative flow the author intends and the story discovery left to the reader. They describe hybrid schemas such as the martini glass, in which an author-led path opens into exploration.

Identical data supports different readings, which makes framing a design choice

Hullman and Diakopoulos (2011) proposed visualisation rhetoric as a framework for how design techniques prioritise particular interpretations. They drew a parallel with framing research in political messaging. They classified rhetorical techniques by the level at which information is added or omitted: the data, the visual representation, the textual annotation and the interactivity. A figure that tells a story makes the same move a frame makes in a news report.

Pandey, Manivannan, Nov, Satterthwaite and Bertini (2014) compared the same data presented as bar and line charts against the same data in tables. Charts led to higher persuasion likelihood and greater attitude change among participants without a strong prior view. Tables seemed to outperform charts among those opposed.

Annotation decides what survives the ten seconds

Borkin, Bylinskii, Kim, Bainbridge, Yeh, Borkin, Pfister and Oliva (2016) labelled 393 visualisations, tracked 33 participants' eye movements and collected thousands of written descriptions. They found that titles and supporting text should convey the message. A reader in a second language, or one who meets the marks only through a screen reader, depends on the text. Pictograms used appropriately can improve recognition, and figures memorable at a glance were effective at conveying it.

Shah and Hoeffner (2002) explain why. Their review of graph comprehension identified three influences: the figure's visual characteristics, the reader's knowledge about graphs, and their expectations about the content. An annotation acts on the third, and a reader without a prior view depends most on it.

Sequence: order is a variable, not a container

Hullman, Drucker, Riche, Lee, Fisher and Adar (2013) analysed 42 professional narrative visualisations. They built a method that prioritises transitions between views by their cost to the audience, and they validated it in two studies. Their results include a ranking of transition types by audience cost. A sequence that uses parallelism as a structural device gained memory and rating benefits.

Cohn (2012) supplies the theory. Working from sequential images rather than data figures, he set out a narrative grammar of categories arranged along a canonical arc. He then argued that the arc is a generative schema, in which any category can expand into a tree node. Pacing is therefore a pattern of embedding rather than a property of a single image.

Motion carries the same rhetoric with less analytic return

Heer and Robertson (2007) ran two controlled experiments on animated transitions between bar charts, pie charts and scatter plots. They found that animated transitions can significantly improve graphical perception through object constancy, as the reader tracks marks across views.

The rest of the evidence is less kind. Chevalier, Dragicevic and Franconeri (2014) tested staggering, the incremental delay in start times widely believed to reduce occlusion and crowding, in tracking tasks chosen to favour it. Staggering had a negligible or negative effect, and it cost readers common-motion grouping. Heer and Robertson (2007) had reported the opposite for staged movement, on evidence they describe as resting mainly on preference. Robertson, Fernandez, Fisher, Lee and Stasko (2008) compared a Gapminder-style animated trend display against overlaid traces and small multiples. Animation was the least effective form for analysis. Both static forms were faster, and small multiples were more accurate. In presentation, animation was fastest and most enjoyed, but it produced many errors. Motion persuades; it does not reliably inform.

The argument becomes the product: data videos and graphical abstracts

Amini, Henry Riche, Lee, Hurter and Irani (2015) analysed 50 professionally designed data videos and ran storyboarding workshops with storytellers from cinematography, graphic design and screenwriting. They concluded that authoring tools should support a non-linear process. Hullman and Bach (2018) analysed 54 graphical abstracts and described the design dimensions and templates by which they communicate, organised around layout, the depiction of time, the use of text and representational genre. They named common defects: inconsistent visual style, unclear relations and missing annotation.

The graphical abstract is the extreme case: one image in which pictorial choices carry the framing of a whole research contribution.

The contested question: is framing the same as distortion?

One position holds that rhetoric is a legitimate part of communication and should be used well. Kosara and Mackinlay (2013) treat storytelling as the next step for the field. Segel and Heer (2010) describe the strongly author-driven genres without treating them as defective. Hullman and Diakopoulos (2011) argue that annotation and sequence make a complex finding legible to a non-expert.

The second holds that the same techniques are where visualisation misleads. Pandey and colleagues (2014) found that charts moved the attitudes of readers without strong prior views, which is persuasion rather than communication. The size of the effect depended on the reader's prior position rather than on the data. Chevalier and colleagues (2014) tested a widely believed convention and found it did not help. Robertson and colleagues (2008) found that the presentation format participants enjoyed most produced the most errors.

What would settle the question is a test most of this literature has not run. The camps disagree partly because they measure different things. Segel and Heer (2010) and Hullman and colleagues (2013) describe designs, while Pandey and colleagues (2014) measure attitude change. Attitude change is not comprehension. The deciding experiment would hold the data constant and vary only annotation and order. It would measure two things: whether the reader's conclusion agrees with the author's reading, and whether the reader can state the case against it. Shah and Hoeffner's (2002) three-factor account gives the design, because it separates what the figure supplies from what the reader brings. Part of that experiment has now been run. Shao and colleagues (2024) held the data and the chart type constant and varied only the storytelling elements; with 103 participants, comprehension accuracy improved, and the gain did not depend on visualisation literacy.

The two positions rest on different evidence. The first rests on corpus analyses of 58 and 42 narrative visualisations and on one controlled experiment (Segel and Heer, 2010; Hullman and colleagues, 2013; Shao and colleagues, 2024). The second rests on controlled experiments measuring attitude change and task performance (Pandey and colleagues, 2014; Chevalier and colleagues, 2014; Robertson and colleagues, 2008). While the dispute is open, a designer should disclose the reading built into the figure, keep the underlying data recoverable, and test both agreement with the author and the reader's ability to state the case against it. The strongest objection is that the one controlled test measures comprehension rather than the reader's agreement with the author, so the claim that framing is rhetoric is only partly measured.

Where this is heading

The first direction is tooling for an authored argument. Amini and colleagues (2015) call for authoring environments that support non-linear creation, and Hullman and Bach (2018) set out templates for graphical abstracts. Together they point to a future in which the annotated, sequenced figure is generated from a specification. The second is measurement of the reading experience. McKenna, Henry Riche, Lee, Boy and Meyer (2017) analysed 80 published stories, derived seven flow factors, and tested their effect on engagement with 240 participants. Visuals and navigation feedback affected engagement, while the level of control did not. The open question is whether a figure's framing can be made auditable.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked on 2 November 2026.

Amini, F., Henry Riche, N., Lee, B., Hurter, C. and Irani, P. (2015). Understanding Data Videos. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 1459-1468. doi:10.1145/2702123.2702431 Analysed 50 professionally designed data videos and ran storyboarding workshops with storytellers from cinematography, graphic design and screenwriting, concluding that authoring tools should support a non-linear process.
Borkin, M. A., Bylinskii, Z., Kim, N. W., Bainbridge, C. M., Yeh, C. S., Borkin, D., Pfister, H. and Oliva, A. (2016). Beyond Memorability: Visualization Recognition and Recall. IEEE Transactions on Visualization and Computer Graphics, 22(1), 519-528. doi:10.1109/tvcg.2015.2467732 Labelled 393 visualisations and tracked 33 participants' eye movements alongside thousands of written descriptions. Found that titles and supporting text should convey the message, and that memorable figures were also effective at conveying it.
Chevalier, F., Dragicevic, P. and Franconeri, S. (2014). The Not-so-Staggering Effect of Staggered Animated Transitions on Visual Tracking. IEEE Transactions on Visualization and Computer Graphics, 20(12), 2241-2250. doi:10.1109/tvcg.2014.2346424 Tested staggering in animated transitions with simulations and tracking tasks chosen to favour it. Staggering had a negligible or negative effect, against its reputation for reducing crowding, and cost readers common-motion grouping.
Cohn, N. (2012). Visual Narrative Structure. Cognitive Science, 37(3), 413-452. doi:10.1111/cogs.12016 Set out a narrative grammar for sequential images, with categories arranged along a canonical arc and a generative tree structure in which any category can expand into a node. Pacing is thus a pattern of embedding, so a sequence has a syntax.
Heer, J. and Robertson, G. (2007). Animated Transitions in Statistical Data Graphics. IEEE Transactions on Visualization and Computer Graphics, 13(6), 1240-1247. doi:10.1109/tvcg.2007.70539 Ran two controlled experiments on animated transitions between common statistical graphics, with a taxonomy of transition types and design principles. Animated transitions can significantly improve graphical perception through object constancy.
Hullman, J. and Diakopoulos, N. (2011). Visualization Rhetoric: Framing Effects in Narrative Visualization. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2231-2240. doi:10.1109/tvcg.2011.255 Proposed visualisation rhetoric as a framework for how design prioritises particular interpretations, drawing a parallel with framing research in political messaging and decision making. Classified rhetorical techniques by the level at which information is added or omitted.
Hullman, J., Drucker, S., Riche, N. H., Lee, B., Fisher, D. and Adar, E. (2013). A Deeper Understanding of Sequence in Narrative Visualization. IEEE Transactions on Visualization and Computer Graphics, 19(12), 2406-2415. doi:10.1109/tvcg.2013.119 Analysed 42 professional narrative visualisations and built a graph-driven method for sequencing views by their transition cost to the audience, validated in two studies. Found memory and rating benefits where a sequence uses parallelism as a structural device.
Hullman, J. and Bach, B. (2018). Picturing Science: Design Patterns in Graphical Abstracts. Diagrammatic Representation and Inference (Diagrams 2018), Lecture Notes in Computer Science, 183-200. doi:10.1007/978-3-319-91376-6_19 Analysed 54 graphical abstracts alongside publisher guidance, describing design dimensions organised around layout, the depiction of time, the use of text and representational genre. Named common defects including inconsistent visual style and missing annotation.
Kosara, R. and Mackinlay, J. (2013). Storytelling: The Next Step for Visualization. Computer, 46(5), 44-50. doi:10.1109/mc.2013.36 Argued that presentation, and the use of storytelling elements, is the next logical step for visualisation research and deserves weight equal to exploration and analysis.
McKenna, S., Henry Riche, N., Lee, B., Boy, J. and Meyer, M. (2017). Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences. Computer Graphics Forum, 36(3), 377-387. doi:10.1111/cgf.13195 Analysed 80 published data stories and derived seven flow factors, then tested their effect on engagement with 240 participants. Visuals and navigation feedback affected engagement while the level of control did not.
Pandey, A. V., Manivannan, A., Nov, O., Satterthwaite, M. and Bertini, E. (2014). The Persuasive Power of Data Visualization. IEEE Transactions on Visualization and Computer Graphics, 20(12), 2211-2220. doi:10.1109/tvcg.2014.2346419 Ran an attitude change experiment comparing the same data as bar and line charts against tables. Charts produced higher persuasion likelihood and greater attitude change among participants without a strong prior view; among those opposed the authors report the reverse as a trend rather than a tested effect.
Robertson, G., Fernandez, R., Fisher, D., Lee, B. and Stasko, J. (2008). Effectiveness of Animation in Trend Visualization. IEEE Transactions on Visualization and Computer Graphics, 14(6), 1325-1332. doi:10.1109/tvcg.2008.125 Evaluated an animated trend display against overlaid traces and small multiples. Animation was least effective for analysis, both static forms were faster and small multiples more accurate, while in presentation animation was fastest and most enjoyed but produced many errors.
Segel, E. and Heer, J. (2010). Narrative Visualization: Telling Stories with Data. IEEE Transactions on Visualization and Computer Graphics, 16(6), 1139-1148. doi:10.1109/tvcg.2010.179 Analysed 58 published narrative visualisations and derived seven genres, from the annotated chart to film and video, on a spectrum from author-driven flow to reader-driven discovery. Shows that the strongly author-driven genres are conventions rather than neutral defaults.
Shah, P. and Hoeffner, J. (2002). Review of Graph Comprehension Research: Implications for Instruction. Educational Psychology Review, 14(1), 47-69. doi:10.1023/a:1013180410169 Reviewed the cognitive literature on how readers comprehend graphs and identified three classes of influence: the visual characteristics of the figure, the reader's knowledge about graphs, and the reader's knowledge and expectations about the content.
Shao, H., Martinez-Maldonado, R., Echeverria, V., Yan, L. and Gasevic, D. (2024). Data Storytelling in Data Visualisation: Does it Enhance the Efficiency and Effectiveness of Information Retrieval and Insights Comprehension? Proceedings of the CHI Conference on Human Factors in Computing Systems, 1-21. doi:10.1145/3613904.3643022 Held data and chart type constant while varying only the storytelling elements, with 103 participants. Comprehension accuracy improved; information retrieval did not.

Check your understanding & apply it · 4 questions, 1 essay

1A designer says a figure is neutral because it shows all the data and every number on it is correct. Using the unit's sources, which judgement is defensible?

2A campaign finds that a chart-led figure shifted the views of readers who held no strong prior view, but had little effect on readers who began opposed. Which diagnosis does the unit's evidence support?

3A team proposes an animated trend display for an analyst who must answer questions afterwards, because 'motion keeps people engaged'. Using the unit's evidence, which prediction is best supported?

4A writer argues: 'Framing is the same as distortion, so any figure that has been annotated or ordered to make a point is misleading.' Which judgement does the unit's reading of the evidence support?

5Essay A national charity must publish a one-page static figure for a press release. The figure will show that reported waiting times for a service rose sharply in three of twelve regions last year, and the board wants readers to conclude that the service is failing and needs more funding. The data are sound and the three increases are real, but those three regions are also the smallest by population, and the previous year was unusually low in two of them. Using this unit's sources, say what you would put in the figure and what you would leave out, what you would annotate and in what order the elements would appear, predict before releasing it what readers are most likely to take from it wrongly, name the one check you would run, and then give the strongest counter-argument to your approach and answer it.

The board's wanted conclusion is a reading, not a fact, so the first decision is to disclose it rather than let the figure imply it. I would include all twelve regions, the population or case count behind each rate, and the two-year series for the three named regions, because Pandey, Manivannan, Nov, Satterthwaite and Bertini (2014) found that a figure moves the attitudes of readers who hold no strong prior view, and a press release is aimed exactly at those readers. Following the visualisation rhetoric framework of Hullman and Diakopoulos (2011), I would treat the annotation, the selection and the order as the places where framing enters, and I would state in the figure's own text the reading it supports and what would count against it. I would annotate the three regions with their population and the prior-year base, and I would mark the small-numbers caveat inside the figure rather than in a note, because Shah and Hoeffner (2002) identify the reader's expectations about the content as one of the three influences on comprehension, and the annotation is what supplies that expectation. In sequence I would open with the twelve-region overview, then the three changes with their denominators, and I would use parallel structure for the three panels, because Hullman, Drucker, Riche, Lee, Fisher and Adar (2013) found memory and rating benefits when a sequence uses parallelism as a structural device. I would avoid an animated transition entirely and would not stagger any motion, since Robertson, Fernandez, Fisher, Lee and Stasko (2008) found animation least effective for analysis and most error-prone in presentation, and Chevalier, Dragicevic and Franconeri (2014) found the popular staggering convention did not help tracking. Before release I would predict that readers take the three increases as evidence about the whole service and ignore the small denominators, which is the pattern Pandey and colleagues measured in attitude change. The one check I would run is a short reading task with readers of the target audience asking two things: what they now believe about the service nationally, and what they think could count against that belief. The strongest counter-argument is that annotating the denominators and the counter-case weakens the charity's case, and that rhetorical restraint is a luxury an underfunded service cannot afford, which is the position Kosara and Mackinlay (2013) imply when they treat storytelling as central to communication. I would answer it by noting that the funding argument depends on a lasting change in attitude rather than an immediate one, that Pandey and colleagues found readers who start opposed are moved by tables rather than charts, and that a figure which raises agreement while erasing the counter-case is the distortion the unit defines, not the communication.