Whose question is this chart answering? Commissioning and framing visual evidence

A brief arrives as a deliverable. It may be a dashboard, a map, or the figure the board saw last quarter.

This first of ten units argues that the first design decision is not the object. It is the question the chart must answer, and the reader who holds it. Encoding is a later level, whose failure is recoverable and whose success is not sufficient. A chart that answers a question nobody is asking is a different artefact.

A presentation tool still has to be told what to present

Mackinlay (1986) set out to build a tool that would design graphical presentations automatically, with no designer in the loop. The criteria it needed split in two. Expressiveness criteria determine whether a graphical language can express the desired information at all. Effectiveness criteria determine whether a presentation built in that language is easily understood. The first criterion is a relation between the language and the information. Even an automatic designer cannot begin until someone tells it what to express. A designer handed a request for a dashboard is not exempt.

The nested model makes the cost of the wrong question visible

Munzner (2009) turned the ordering into four nested layers. Characterise the task and data in the vocabulary of the problem domain. Abstract into operations and data types. Design the encoding and interaction. Create the algorithm. Output from a level above is input to the level below. The consequence is stated plainly:

An upstream error inevitably cascades to all downstream levels.

At the problem level the users may not do the thing the design assumes. At the abstraction level the designer may be showing them the wrong thing. If the designer chose the abstraction badly, the paper says, even perfect encoding and algorithm design will not produce a system that solves the intended problem. That is why naming the question is a commissioning obligation.

A commissioner has not yet chosen among ten low-level tasks

Amar, Eagan and Stasko (2005) supply the vocabulary for those outer levels. The task taxonomies of the day described systems rather than people. They collected nearly two hundred questions from students about how they would analyse five datasets from different domains. The questions fell into ten low-level tasks: retrieve value, filter, compute derived value, find extremum, sort, determine range, characterise distribution, find anomalies, cluster and correlate. A commissioner who asks where money is being lost has not said whether the reader must find an extremum, characterise a distribution or correlate two attributes. Those are three different charts. The questions came from students, so the vocabulary describes a fluent reader in front of a screen; an older reader, or one reading in a second language, meets the same tasks differently.

Why comes before how, and the ends of a task are not its means

Brehmer and Munzner (2013) close the gap Amar and colleagues left open. Earlier work, they argue, covered either low-level tasks and interactions or high-level tasks. It never separated the ends of a task from its means. Their typology is organised by why the task is performed, how it is performed, and what it takes in and produces. Practitioners first identify why and only then how. That is how a domain problem becomes an abstract task and then a design choice.

Task and data are separate axes, so no question follows from the data

Shneiderman (1996) had made the crossing much earlier. Seven data types are set against seven tasks: overview, zoom, filter, details-on-demand, relate, history and extract. Shneiderman adds the mantra that a reader wants overview first, then zoom and filter, then details on demand. The two axes are independent, so no task can be read off the data alone. One reader moves between tasks within a session, so the mantra is a claim about a reader's question over time. A chart that answers only the first has answered a question already finished.

Commissioning as a stage, not a formality

Sedlmair, Meyer and Munzner (2012) describe the professional process that turns a domain problem into a design problem. The framework has nine stages in three groups: learn, winnow and cast; discover, design, implement and deploy; reflect and write. Winnow decides whether a design study is the right instrument. The authors judge it against two axes: the clarity of the task and the location of the information needed to solve it. A crisp task with the data to hand may suit automation. A fuzzy task whose knowledge sits in domain experts' heads calls for a study. It concedes that the question is sometimes unclear at commissioning. It does not concede that the question can be skipped.

Their own record is not a list of successes. Of the twenty-one design studies the authors reflect on, they count at least three as failures. One example is the relationship between ballot design and voting behaviour. Nothing can be encoded there until someone sharpens the question.

The named question decides what counts as evidence

North (2006) argued that the purpose of visualisation is insight. Evaluating one therefore means asking whether it generates insight. He argued that the standard controlled experiment with benchmark tasks measures insight poorly, because a benchmark task supplies the experimenter's question rather than the reader's. Lam, Bertini, Isenberg, Plaisant and Carpendale (2012) turn the point into seven scenarios. They derived the scenarios from a systematic analysis of 850 visualisation publications, 361 of them carrying an evaluation, to help a practitioner set goals and questions before choosing a method.

The question is held by a person, not by the data

The pressure against a task-first account comes from visual analytics. Pike and colleagues (2009) argue that interaction and inquiry are inextricable. Knowledge, they argue, is constructed, tested, refined and shared through the interactive manipulation of an interface, which they call the analytic discourse. Sacha and colleagues (2014) formalise the human side as exploration, verification and knowledge generation loops. On their definition, no system can automate the passage from insights to knowledge, or from knowledge to new hypotheses.

The authors then tested the model on a named system. Sacha and colleagues walked Jigsaw, Stasko's visual analytics environment for investigative analysis of document collections, through the three loops. They applied the same test to KNIME, Tableau and HARVEST. Jigsaw supported every action the model names. The authors still found a specific gap: integrating the system's tablet view more tightly would strengthen the verification loop.

The contested question: is the task settled at commissioning or discovered in use?

The disagreement is not whether questions matter but when the question can be known. On one side the task is an input to be specified. Munzner's (2009) outer layers take shape before anyone encodes anything. Brehmer and Munzner (2013) place why before how, as the order practitioners follow. Shneiderman's (1996) table assumes a choice between seven tasks. Sedlmair and colleagues (2012) place winnow before discover.

On the other side the visual analytics work treats the question as emergent. Pike and colleagues (2009) locate knowledge in the manipulation of an interface rather than in the specification that preceded it. Sacha and colleagues (2014) place hypothesis formation inside the analysis loop. North (2006) developed insight-based evaluation precisely because a benchmark task supplies the evaluator's question. Amar and colleagues (2005) built their taxonomy from questions people actually posed.

Some problems arrive crisp and others fuzzy, so the positions are conditional rather than contradictory. The decisive study would follow commissioned charts from the written brief to the use made of them. It would record the question named at commissioning and the tasks readers performed, then code how often the two diverge. A companion experiment would compare the insight reached from a fixed-brief chart with an exploratory view of the same data. Until that evidence exists, the defensible position is the weaker one. A task-level error cannot be repaired downstream, so the question must be named and its assumptions stated. At the same time, the work must be built to expect the question to move.

The two sides rest on different evidence. The commissioning side draws on a design framework and a typology tested against a case study (Munzner, 2009; Brehmer and Munzner, 2013). The emergent side rests on a research agenda and a model checked against four named systems (Pike and colleagues, 2009; Sacha and colleagues, 2014). While the dispute is open, a designer should name the question and its assumptions at commissioning, then build the view so the question can move without a rebuild. The strongest objection is that the nested model is an argument from practice, not a measurement, so the claim that a late question cannot be retrieved is untested.

Where this is heading

The direction of travel is towards making the question explicit. Brehmer and Munzner's (2013) typology is the clearest instrument for that, describing a task abstractly enough to compare visualisations across domains. Sacha and colleagues' (2014) model is the counterweight, since it locates the question in the human. Lam and colleagues' (2012) scenarios convert a named question into a choice of evaluation.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked on 2 November 2026.

• Amar, R., Eagan, J. and Stasko, J. (2005). Low-level components of analytic activity in information visualization. IEEE Symposium on Information Visualization, 2005. INFOVIS 2005, 111-117. doi:10.1109/infvis.2005.1532136 Collected nearly two hundred questions from students about how they would analyse five datasets from different domains, and grouped them into ten low-level tasks, from retrieve value to correlate, offered as a vocabulary and a checklist. It judges visualisation by analytic activity rather than representational quality, which is the unit's opening warrant.
• Brehmer, M. and Munzner, T. (2013). A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 19(12), 2376-2385. doi:10.1109/tvcg.2013.124 Argues that earlier classifications covered either low-level tasks and interactions or high-level tasks, leaving the gap unaddressed and conflating the ends and means of a task. Contributes a typology organised by why, how and what, orders why before how because that reflects how a domain problem becomes a design choice, and tests it against a case study and compares it with the over two dozen extant classification systems and theoretical frameworks it reviews.
• Lam, H., Bertini, E., Isenberg, P., Plaisant, C. and Carpendale, S. (2012). Empirical studies in information visualization: seven scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1520-1536. doi:10.1109/tvcg.2011.279 Derives seven evaluation scenarios, from visual data analysis and reasoning to collaborative data analysis, through a systematic analysis of 850 publications, 361 of them carrying an evaluation, and states its purpose as helping a practitioner set evaluation goals and questions before choosing a method. It shows that the question decides what counts as evidence about a chart.
• Mackinlay, J. (1986). Automating the design of graphical presentations of relational information. ACM Transactions on Graphics, 5(2), 110-141. doi:10.1145/22949.22950 Reports a tool that designs graphical presentations automatically, and codifies the design criteria it needs as expressiveness criteria, which determine whether a graphical language can express the desired information, and effectiveness criteria, which determine whether a presentation is easily understood. The first constrains the information rather than the form, so even an automatic designer must be told what is to be presented.
• Munzner, T. (2009). A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics, 15(6), 921-928. doi:10.1109/tvcg.2009.111 Sets out four nested layers, from characterising the problem domain through abstracting into operations and data types and designing encoding and interaction to implementing algorithms, and argues that an upstream error cascades to every downstream level. Recommends that authors state their assumptions about the levels above their own. It is the source of the unit's central claim that a task-level error cannot be repaired later.
• North, C. (2006). Toward measuring visualization insight. IEEE Computer Graphics and Applications, 26(3), 6-9. doi:10.1109/mcg.2006.70 Argues that the purpose of a visualisation is insight and that evaluating one means determining whether it generates insight, then argues that the controlled experiment with benchmark tasks measures insight poorly because it supplies the evaluator's question. Proposes an approach adapted to capture insight as it arises, which connects the named question to the choice of evaluation.
• Pike, W. A., Stasko, J., Chang, R. and O'Connell, T. A. (2009). The science of interaction. Information Visualization, 8(4), 263-274. doi:10.1057/ivs.2009.22 Argues that interaction and inquiry are inextricable, and that knowledge is constructed, tested, refined and shared through the interactive manipulation of an interface, which the authors call the analytic discourse. Sets out seven priority areas for visual analytics research over the following five years, all concerned with human and information interaction. It is the strongest counter-pressure to a task-first account.
• Sacha, D., Stoffel, A., Stoffel, F., Kwon, B. C., Ellis, G. and Keim, D. A. (2014). Knowledge generation model for visual analytics. IEEE Transactions on Visualization and Computer Graphics, 20(12), 1604-1613. doi:10.1109/tvcg.2014.2346481 Proposes a model with a computer part and a human part, in which the human side is a reasoning process of exploration, verification and knowledge generation loops, tested against a real-world visual analytics system. States that automating the passage from insights to knowledge, or from knowledge to new hypotheses, is not possible on their definition.
• Sedlmair, M., Meyer, M. and Munzner, T. (2012). Design study methodology: reflections from the trenches and the stacks. IEEE Transactions on Visualization and Computer Graphics, 18(12), 2431-2440. doi:10.1109/tvcg.2012.213 Draws on twenty-one design studies conducted by the authors to propose a nine-stage framework in three groups, learn, winnow and cast, then discover, design, implement and deploy, then reflect and write, and identifies thirty-two pitfalls. Judges when a design study is the right instrument using the clarity of the task and the location of the information needed, which is the unit's account of commissioning as a professional stage.
• Shneiderman, B. (1996). The eyes have it: a task by data type taxonomy for information visualizations. Proceedings of the 1996 IEEE Symposium on Visual Languages, 336-343. doi:10.1109/vl.1996.545307 Proposes a taxonomy of seven data types against seven tasks, from overview to extract, with the visual information seeking mantra of overview first, zoom and filter, then details on demand, presented as a starting point. It shows that task and data are separate axes, so a question cannot be read off the data, and that a reader's question changes within one session.