Grammar, abstraction and automated encoding recommendation

A grammar of graphics holds that a chart is not a named object but the output of a specification: data, mappings from variables to visual properties, transformations, scales, and a composition of marks. Say that, and a machine can write the specification, which is what a recommender does when it ranks candidate encodings. This second of ten units, after the first ranked the encodings, argues that formal grammar and automated recommendation are one story, and that the second inherits the weakness of the first. A recommender is a grammar plus a scoring rule, and in every system examined here the rule is the perceptual ranking of the first unit or a statistical proxy for it, so a system that treats the ranking as a law rather than an ordinal result about isolated channels reproduces its blind spots at scale. The systems that take a request in words take it in English, so an analyst working in a second language is outside the demonstration.

The grammar makes a chart a specification

Wilkinson (1999) supplied the founding claim: a statistical graphic is generated from data by a sequence of operations, with variables mapped to aesthetic attributes and composition building the display, so chart construction becomes explicit and programmable rather than a matter of templates.

Wickham (2010) reworked the grammar into layers, declaring mappings, statistical transformations, geometric objects, scales and coordinate systems separately, which reaches the deep structure that named graphics such as the scatterplot obscure. Polaris (Stolte, Tang and Hanrahan, 2002) shows the idea as an interface: analysts build a visual specification on shelves, and the system generates the displays and the relational queries behind them.

Protovis (Bostock and Heer, 2009) defines a visualisation as a hierarchy of marks whose properties are functions of data. Vega-Lite (Satyanarayan et al., 2017) composes a grammar of graphics with a grammar of interaction: a selection abstracts over input events, and the compiler synthesises the data flow. Interaction is therefore part of what a specification can say.

Abstraction decides what a machine can be told

Munzner (2014) orders design as three questions: what data the reader must see, why they must see it, and how marks and channels can carry it. The middle question does not reduce to data properties, because it concerns the reader's task rather than the table.

Amar, Eagan and Stasko (2005) supply the working vocabulary, listing ten low-level analytic tasks, including retrieve value, filter and characterise distribution. A recommender that reasons about tasks reasons about a closed set of ten, while the mapping from an analyst's goal onto one of them stays outside the system.

Text-to-Viz (Cui et al., 2020) takes a natural-language statement about a simple proportion, parses the relation out of the sentence, and returns an infographic. The design was demonstrated through sample results, exhibits and expert reviews, and nothing in it says whether the chart fits the purpose the statement was made for.

ChartGPT (Tian et al., 2024) generates charts from abstract natural language through a step-by-step reasoning pipeline, returning intermediate steps for the user to check. VisEval (Chen et al., 2024) benchmarks the same task on 2,524 queries over 146 databases, scoring validity, legality and readability rather than agreement with past charts.

Recommendation in a product reshapes attention rather than checking the reading

Show Me (Mackinlay, Hanrahan and Stolte, 2007) is where automatic presentation entered commercial practice, selecting mark types automatically inside Tableau through VizQL and extending the same machinery to small multiples. Its authors state that the defaults are optional and treat interface logs as evidence that commercial users rely on them.

Voyager (Wongsuphasawat et al., 2016) makes the machinery mixed-initiative, generating a gallery of charts chosen by statistical and perceptual measures for the analyst to browse along data facets. Against a manual specification tool it facilitated exploration of previously unseen data and increased coverage of the variables considered, and the authors conclude that a tool must balance rapid exploration against targeted answering. Recommendation changes what an analyst looks at, and coverage is breadth, not quality of reading.

Constraints make the rules explicit

Draco (Moritz et al., 2019) writes design knowledge as weighted constraints and solves them with answer set programming, so a specification can be evaluated and completed automatically. The weights matter, and the paper learns them for soft constraints from experimental data, demonstrating systems whose weights come directly from graphical perception experiments. Isolated judgements become coefficients inside a solver recommending encodings for displays nobody tested.

Kindlmann and Scheidegger (2014) approach the same problem as an algebra, with three components: the mathematical structure of the data, its concrete representation, and a mathematical description of how humans perceive the visualisation.

Learning the rules from the charts that already exist

The recommender has lately begun to be learned rather than written. VizML (Hu et al., 2019) trained models on one million dataset-visualisation pairs from an online platform to predict five analyst design choices, reporting consensus chart-type predictions comparable to human performance.

Data2Vis (Dibia and Demiralp, 2019) treats generation as translation, mapping a data description to a Vega-Lite specification with a sequence-to-sequence model and reporting that it learns valid vocabulary, syntax and common transformations. Its metrics are language syntax validity and visualisation grammar validity, so a well-formed specification is scored and a fitting display is not.

Contested question: can a recommender be told the task, or only the data?

The disagreement is about what the machine is given. One position holds that a grammar and a constraint system can carry the task as well as the data, so a recommendation can be right for the question asked. Munzner (2014) makes task abstraction the step preceding form, Amar, Eagan and Stasko (2005) supply a task vocabulary, and Draco (Moritz et al., 2019) encodes that vocabulary and solves it.

The other position holds that recommendation works from the data and that the task stays with the analyst. Show Me selects marks from the fields placed in a view (Mackinlay, Hanrahan and Stolte, 2007), Voyager recommends by statistical and perceptual measures and leaves the question to browsing (Wongsuphasawat et al., 2016), and VizML predicts past analyst choices rather than the choice a question requires (Hu et al., 2019).

What would settle it is two studies, and neither is in this unit's list.

1. A study that holds the data fixed and varies the analytic question, asking whether the recommendation changes as the task requires.
2. A study that evaluates a learned recommender on task-relevant quality rather than agreement with existing charts, since agreement cannot distinguish a good recommendation from a conventional one.

The two positions rest on different evidence. The task position rests on a framework argument, a task vocabulary and a constraint solver whose weights come from perception experiments. The data position rests on deployed systems and learned models, reported by their authors. While the dispute is open, treat a recommendation as a candidate to check, and name the question the view is for before accepting a default.

The strongest counter-argument

The strongest objection is that no blind spot is inherited. The recommender never decides: it orders candidates, a person chooses, and Show Me's own authors state that the defaults are optional, so the ranking is an ordering offered to an expert who supplies the task.

The reply is that this division of labour is not what those papers report. The defaults are used, which is why the Show Me authors treat interface logs as evidence; Draco's weights are learned from the perception experiments, so the ranking sits inside the scorer. Recommendation reliably shapes attention and does not reliably account for the task.

Where the area is heading

The direction is towards systems that carry more of the argument inside the specification. Draco (Moritz et al., 2019) is built to be extended and reweighted as new experiments arrive, which is the workable answer to inherited rules, and Vega-Lite (Satyanarayan et al., 2017) puts interaction into the specification. What remains missing is the evaluation this unit has not found: no source here measures whether a recommended chart improves the decision reached.

The sources this unit rests on

Each note states what the cited document did and found, from the resolved record and abstract in the module plan.

• Amar, R., Eagan, J. and Stasko, J. (2005). Low-level components of analytic activity in information visualization. doi:10.1109/infvis.2005.1532136 Collected nearly 200 sample analytic questions from students about five datasets and derived ten low-level analytic tasks, including retrieve value, filter and characterise distribution. It is the vocabulary a task-aware recommender must encode, and ten tasks is a coarse one.
• Bostock, M. and Heer, J. (2009). Protovis: A Graphical Toolkit for Visualization. doi:10.1109/tvcg.2009.174 Built a toolkit in which a visualisation is a hierarchy of marks whose visual properties are functions of data, claiming the expressiveness of low-level graphics with less specification effort, supported by examples rather than a user study. It is the lineage that makes a chart a program, and so fit for automation.
• Chen, N., Zhang, Y., Xu, J., Ren, K. and Yang, Y. (2024). VisEval: A Benchmark for Data Visualization in the Era of Large Language Models. doi:10.1109/tvcg.2024.3456320 Built a benchmark of 2,524 natural-language queries over 146 databases with a multi-dimensional check of validity, legality and readability. It shows what the evaluation of generated charts now measures, and it still stops short of whether a chart serves the question.
• Cui, W., Zhang, X., Wang, Y. et al. (2020). Text-to-Viz: Automatic Generation of Infographics from Proportion-Related Natural Language Statements. doi:10.1109/tvcg.2019.2934785 Built a proof-of-concept system that converts statements about simple proportion-related statistics into infographics in pre-designed styles, after a preliminary study of the design space, and demonstrated it through sample results, exhibits and expert reviews rather than a controlled study.
• Dibia, V. and Demiralp, C. (2019). Data2Vis: Automatic Generation of Data Visualizations Using Sequence-to-Sequence Recurrent Neural Networks. doi:10.1109/mcg.2019.2924636 Formulated chart generation as machine translation, training a sequence-to-sequence model to map a data specification to a Vega-Lite specification, and reported that it learns valid vocabulary and syntax. Evaluation used syntax validity metrics, so the model is scored on well-formed specifications rather than on fitting displays.
• Hu, K., Bakker, M. A., Li, S., et al. (2019). VizML: A Machine Learning Approach to Visualization Recommendation. doi:10.1145/3290605.3300358 Trained models to predict five analyst design choices from one million dataset-visualisation pairs from an online platform, reporting consensus chart-type predictions comparable to human performance. The target is the choices analysts made in charts that exist, so the model learns past decisions rather than the choice a question requires.
• Kindlmann, G. and Scheidegger, C. (2014). An Algebraic Process for Visualization Design. doi:10.1109/tvcg.2014.2346325 Proposed a model of design with three components, the mathematical structure of the data, its concrete representation, and a mathematical description of how humans perceive the display, to be used alongside user studies rather than in place of them. The perceptual component is qualified as possible only to an extent, so the formalisation admits its weakest joint.
• Mackinlay, J., Hanrahan, P. and Stolte, C. (2007). Show Me: Automatic Presentation for Visual Analysis. doi:10.1109/tvcg.2007.70594 Described automatic presentation inside a commercial visual analysis system, using VizQL and automating the selection of mark types and of small multiples. The authors state that the defaults are optional and that interface logs show commercial users rely on them, so the ranking enters the product as a default.
• Moritz, D., Wang, C., Nelson, G. L., et al. (2019). Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. doi:10.1109/tvcg.2018.2865240 Modelled design knowledge as constraints solved with answer set programming, learning weights for soft constraints from experimental data and demonstrating systems whose weights come directly from graphical perception experiments. It is the clearest case of automation inheriting the ranking, whose results are the recommender's coefficients.
• Munzner, T. (2014). Visualization Analysis and Design. doi:10.1201/b17511 Set out a framework ordering design as decisions about what data the reader must see, why they must see it, and how marks and channels can carry it, with task abstraction preceding form. It supplies the why a recommender must model, which does not reduce to data properties.
• Satyanarayan, A., Moritz, D., Wongsuphasawat, K. and Heer, J. (2017). Vega-Lite: A Grammar of Interactive Graphics. doi:10.1109/tvcg.2016.2599030 Defined a grammar combining visual encoding rules with a composition algebra and a grammar of interaction, in which a selection abstracts over input events and an inclusion test, and a compiler synthesises the data flow. Evaluation is by example, covering panning, zooming and linked selection, so interaction sits in the same specification as the encodings.
• Stolte, C., Tang, D. and Hanrahan, P. (2002). Polaris: a system for query, analysis, and visualization of multidimensional relational databases. doi:10.1109/2945.981851 Built an interface in which analysts construct visual specifications of table-based displays and the system generates the displays and the relational queries behind them. It is the point at which a view became something a program builds and can search.
• Tian, Y., Cui, W., Deng, D., Yi, X., Yang, Y., Zhang, H. and Wu, Y. (2024). ChartGPT: Leveraging LLMs to Generate Charts From Abstract Natural Language. doi:10.1109/tvcg.2024.3368621 Built a chart generator that takes abstract natural language, decomposes generation into a step-by-step reasoning pipeline and fine-tunes a language model on utterance-chart pairs, with an interface for checking intermediate outputs. It is the point at which the recommender moved from charts to language.
• Wickham, H. (2010). A Layered Grammar of Graphics. doi:10.1198/jcgs.2009.07098 Reworked the grammar of graphics into layers, declaring data, aesthetic mappings, statistical transformations, geometric objects, scales and coordinate systems separately, so that the deep structure behind a named graphic such as the scatterplot is described. It is the grammar in the form that reached analysts.
• Wilkinson, L. (1999). The Grammar of Graphics. doi:10.1007/978-1-4757-3100-2 Set out a formal grammar in which a statistical graphic is generated from data by a sequence of operations, with variables mapped to aesthetic attributes and the display built by composition, making construction programmable rather than a matter of templates. Only a specified view can be ranked or generated.
• Wongsuphasawat, K., Moritz, D., Anand, A., et al. (2016). Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations. doi:10.1109/tvcg.2015.2467191 Built a mixed-initiative system recommending charts chosen by statistical and perceptual measures, with faceted browsing of the recommendations, and compared it against manual specification. It facilitated exploration of previously unseen data and increased coverage of the data variables considered; the authors conclude that tools must balance rapid exploration against targeted question answering.