Direct manipulation trades invisibility for immediacy, and gesture pays the discoverability debt

Year 2 / Module 15 Interaction techniques and usability evaluation / Unit 4 / Direct manipulation trades invisibility for immediacy, and gesture pays the discoverability debt

Direct manipulation trades invisibility for immediacy, and gesture pays the discoverability debt

HCI2107 · Unit 4 of 10

Gestural interaction keeps the immediacy that made direct manipulation valuable while discarding the visible and labelled object that made it learnable, so a gesture a user cannot invent or remember imposes a discoverability debt that a visible control avoids, and elicitation studies measure that debt rather than removing it.

Direct manipulation trades invisibility for immediacy, and gesture pays the discoverability debt

Direct manipulation is praised for its immediacy. The object responds at once, the effect is visible and reversible, and the user feels they act on the thing itself. This fourth of ten units argues that the immediacy was bought with a visible, labelled object. Gesture kept the immediacy and gave up the object, leaving a discoverability debt. Elicitation studies measure that debt rather than repay it, under a condition that removes the part most users face: the method shows participants the effect first.

The object gave gestures their immediacy and their debt

Shneiderman (1983) described direct manipulation through three properties: continuous representation of the object of interest, physical actions or labelled button presses instead of a command syntax, and rapid, reversible operations whose effect is immediately visible. The interface moved from a language the user had to learn to objects the user could see. Hutchins, Hollan and Norman (1985) added that directness is a relation between the user's model and the interface, not a device property.

Norman (2010) attacked the word natural in natural user interfaces. Gestures must be learned, they have no standard, and they often violate good feedback and discoverability, so what is called natural is usually familiar. Familiarity is a convention a population already holds, such as a pinch to zoom, and once established it behaves like a label. A gesture carries the immediacy but no sign for a user who lacks the convention.

The field turned guessability into a measurement

The field turned the complaint into a measurement. Wobbrock, Aung, Rothrock and Myers (2005) gave a unified approach to the guessability of symbolic input, with formulae for guessability and agreement among guesses, and improved the EdgeWrite unistroke alphabet from 51.0 to 80.1 per cent by users alone. Participants entered it with 81.6 and 94.2 per cent accuracy after one and five minutes, against 78.8 and 90.2 for the original, so the user-led repair closed a gap rather than opening one. The debt is quantitative and reducible, but the measure is between people: the method shows the referent, and asks whether the next person recovers one person's symbol, which is agreement about a sign rather than discovery of an interaction.

Agreement is measured among people shown the effect first

Wobbrock, Morris and Wilson (2009) built the method this unit's debt rests on. They elicited gestures from 20 participants for 27 tabletop commands, logging 1080, by portraying each command's effect and asking for its cause. One hand was preferred to two, and desktop idioms shaped the mental models participants brought. Some commands elicited little agreement, which the authors read as a reason to place on-screen widgets beside those gestures. The finding is easy to overstate: the debt is command-dependent, not proof that gestures are undiscoverable, and every participant saw the effect first, so it measures reassignment of a seen gesture rather than the guessing of an unseen one, and the sample itself leaves a screen reader user no gesture at all.

Vatavu and Wobbrock (2015) showed a single agreement score is not a sufficient statistic, proposing a disagreement rate for referents and a coagreement rate between referents. They then added the dissimilarity-consensus method (Vatavu and Wobbrock, 2019), which models consensus over gesture articulation against numerical dissimilarity, using dynamic time warping or Hausdorff distances, demonstrated on 1312 whole-body gestures from 30 children. It separates users disagreeing about what to gesture from users unclear about what the referent meant, so a study can report low agreement for a gesture every user would find on screen.

A settled method measures agreement, not memory

The corpus is now large enough to describe the method rather than a study. Villarreal-Narvaez, Sluyters, Vanderdonckt and Vatavu (2023) reviewed 267 elicitation studies, covering 187,265 gestures from 6,659 participants across 4,106 referents, classified their contexts, referents, gestures and modalities, and proposed guidelines for the next one. Almost all end in a consensus set chosen by agreement analysis, which makes the debt systematic rather than incidental.

Nacenta, Kamber, Qiang and Kristensson (2013) compared user-defined, designer-defined and random gesture sets across three studies with 33 participants. User-defined gestures were easier to remember immediately and the next day, with up to a 24 per cent recall advantage, mostly through association rather than form errors. A set can therefore be easy to invent and hard to remember, or memorable once known and unguessable beforehand.

The debt deepens off the surface, and the setting changes the gesture

A designer can anchor a surface gesture to a plane. Piumsomboon, Clark, Billinghurst and Cockburn (2013) extended it to augmented reality, deriving a set for AR commands with agreement measures alongside, and Vatavu (2012) reports the first study of free-hand television gestures, proposing a set for basic television tasks. Users converged on some gestures and not others, so each set is specific to the domain that produced it.

Liu, Li, Chen, Wu and Liang (2024) carried the paradigm into virtual reality with an elicitation study for free-hand gestures in a VR museum. Two experiments used an enhanced real-time Wizard of Oz technique to help participants understand the referents. Average agreement for first-choice gestures rose from 0.211 to 0.412 between the experiments, and for second-choice gestures from 0.160 to 0.284. Refining the referents can raise agreement, but the refinement was a change to the study.

Hinrichs and Carpendale (2011) provided the ecological check. Their field study at the Vancouver Aquarium found interaction and social context shaping gesture choice and use, with gestures coming in sequences and serving social encounters as well as manipulation. An elicitation study asks one person, in a laboratory, for one command at a time, so its agreement is measured under conditions the setting does not reproduce.

Contested question: does high agreement mean a gesture should ship?

The literature divides on what an elicitation study is for. Nielsen, Storring, Moeslund and Granum (2004) treat it as a design method, combining elicitation of novice gestures with usability criteria for selecting among them. Wobbrock, Morris and Wilson (2009) license a decision rule, reading low agreement as the signal that a visible widget belongs beside the gesture.

Vatavu and Wobbrock (2019) undercut the criterion from within the paradigm. An unclear referent can cause a low rate rather than a bad gesture, so the number cannot license the decision until a study separates the two causes. Hinrichs and Carpendale (2011) show the conditions of measurement differ from the conditions of use, and Nacenta and colleagues (2013) that a set can be memorable without being guessable.

What would settle the dispute is a study of the condition every elicitation study omits: no effect shown, the interface presented without instruction, and a measure of whether the participant produces the correct gesture on the first attempt and retains it after a delay, in the setting of use. None of the studies here does it, and the pattern is systematic, because showing the effect is how the method obtains its referents at all.

The evidence for shipping on a rate is thin. Wobbrock, Morris and Wilson (2009) license the decision from one elicitation study of 27 commands with 20 participants, while Vatavu and Wobbrock (2019) argue from a method demonstrated on 1312 gestures that a low rate has two causes, and Hinrichs and Carpendale (2011) show from a field study that the setting changes the behaviour. The strongest objection is that the unit's thesis is partly circular, since it faults elicitation for showing the effect first yet cites no method that measures discovery of an unseen gesture. A designer should ship a visible control beside the gesture while the dispute is open.

Where the argument is heading

The field is heading towards treating discoverability as a measurable property of a population in a context rather than of a technique. Vatavu and Wobbrock (2019) supply the tool that splits the two causes of a low agreement rate, and Liu and colleagues (2024) show the debt is reducible when a study refines the referents. Norman (2010) named the mechanism that discharges the debt in practice: convention, a gesture becoming familiar until it functions as a label. The likely settlement is a design that ships a visible control beside the gesture until the convention takes hold.

The sources this unit rests on

Hinrichs, U. and Carpendale, S. (2011). Gestures in the wild: studying multi-touch gesture sequences on interactive tabletop exhibits. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1978942.1979391 A field study at the Vancouver Aquarium observed visitors using a large multi-touch table and found gesture choice and use were shaped by the interaction and social context, with gestures linked into sequences and serving social encounters as well as manipulation. It shows laboratory agreement is measured under conditions the setting does not reproduce.
Hutchins, E. L., Hollan, J. D. and Norman, D. A. (1985). Direct Manipulation Interfaces. Human-Computer Interaction. doi:10.1207/s15327051hci0104_2 The authors give a cognitive account of direct manipulation through the gulfs between the user's goals, the system's expression and its output, locating directness in the user's relation with the interface rather than in the device. It is why discoverability is a property of a user's model, not of a mechanism.
Liu, Q., Li, Y., Chen, B., Wu, H. and Liang, H. N. (2024). User-Defined Gesture Interactions for VR Museums: An Elicitation Study. 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). doi:10.1109/ismar62088.2024.00078 The authors ran an elicitation study for free-hand gestures in a VR museum across two experiments, with average agreement for first-choice gestures rising from 0.211 to 0.412 and for second-choice gestures from 0.160 to 0.284. It shows the debt is reducible by refining referents, though the refinement changed the study.
Nacenta, M. A., Kamber, Y., Qiang, Y. and Kristensson, P. O. (2013). Memorability of pre-designed and user-defined gesture sets. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/2470654.2466142 Three studies with 33 participants compared user-defined, designer-defined and random gesture sets and found user-defined gestures easier to remember immediately and the next day, with up to a 24 per cent recall advantage, mostly through association errors rather than form errors. It splits the debt into inventing, guessing and remembering.
Nielsen, M., Storring, M., Moeslund, T. B. and Granum, E. (2004). A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. Lecture Notes in Computer Science. doi:10.1007/978-3-540-24598-8_38 The authors propose a procedure that combines elicitation of novice gestures with usability criteria for selecting among them, formalising the move from proposed gestures to a design. It is the clearest statement of elicitation as a design method rather than only a measurement.
Norman, D. A. (2010). Natural user interfaces are not natural. Interactions. doi:10.1145/1744161.1744163 Norman argues that gestures and body movements must be learned, have no standard, and often violate the principles of good feedback and discoverability, so naturalness is usually familiarity. It names the debt in this unit's thesis and identifies convention as the mechanism that can discharge it.
Piumsomboon, T., Clark, A., Billinghurst, M. and Cockburn, A. (2013). User-Defined Gestures for Augmented Reality. Lecture Notes in Computer Science. doi:10.1007/978-3-642-40480-1_18 The authors conducted a user-defined gesture study for augmented reality, deriving a gesture set for AR commands and reporting agreement measures with it. It extends elicitation from surface to mid-air interaction, the point at which the gestures leave the surface behind.
Shneiderman, B. (1983). Direct manipulation: a step beyond programming languages. Computer. doi:10.1109/mc.1983.1654471 Shneiderman describes direct manipulation through continuous representation of the object of interest, physical actions or labelled button presses instead of a command syntax, and rapid, reversible operations with immediately visible effects. It is the origin of the properties gestures inherited in part and gave up in part.
Vatavu, R. D. (2012). User-defined gestures for free-hand TV control. Proceedings of the 10th European Conference on Interactive TV and Video. doi:10.1145/2325616.2325626 The paper reports the first study of users' preferences for free-hand television gestures, using an agreement analysis to propose a set for basic television tasks with guidelines from observed behaviour. Users converged on some gestures and not others, so the set is domain-specific.
Vatavu, R. D. and Wobbrock, J. O. (2015). Formalizing Agreement Analysis for Elicitation Studies. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. doi:10.1145/2702123.2702223 The authors propose a disagreement rate for referents and a coagreement rate between referents to accompany the widely used agreement formula, with a significance test for comparing agreement rates. The pair replaces a single score with measures that expose structure between referents.
Vatavu, R. D. and Wobbrock, J. O. (2019). The dissimilarity-consensus approach to agreement analysis in gesture elicitation studies. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3290605.3300454 The authors introduce the dissimilarity-consensus method, modelling consensus over gesture articulation against numerical dissimilarity using dynamic time warping or Hausdorff distances, demonstrated on 1312 whole-body gestures from 30 children. It separates confusion about what to gesture from confusion about what a gesture means, so a low rate cannot be read as a verdict on the gesture.
Villarreal-Narvaez, S., Sluyters, A., Vanderdonckt, J. and Vatavu, R. D. (2023). Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation Studies. ACM Computing Surveys. doi:10.1145/3636458 A systematic review of 267 elicitation studies covering 187,265 gestures from 6,659 participants and 4,106 referents, classifying contexts, referents, gestures and modalities. It documents that the great majority end in a consensus set derived by agreement analysis, so the debt belongs to a settled method.
Wobbrock, J. O., Aung, H. H., Rothrock, B. and Myers, B. A. (2005). Maximizing the guessability of symbolic input. CHI '05 Extended Abstracts on Human Factors in Computing Systems. doi:10.1145/1056808.1057043 The authors present a unified method for eliciting and evaluating the guessability of symbolic input, with formulae for guessability and agreement among guesses, and improved the EdgeWrite alphabet from 51.0 to 80.1 per cent guessability by users alone. It turns discoverability into an empirical quantity, measured among people shown the referent.
Wobbrock, J. O., Morris, M. R. and Wilson, A. D. (2009). User-defined gestures for surface computing. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1518701.1518866 The authors elicited gestures for 27 tabletop commands from 20 participants by portraying each command's effect and asking for its cause, logging 1080 gestures. One hand was preferred to two and some commands elicited little agreement, read as a case for on-screen widgets.

Check your understanding & apply it · 4 questions, 1 essay

1An elicitation study reports low agreement for a share gesture. A designer concludes that gesture is undiscoverable for that command and removes it from the design. Which diagnosis fits the unit?

2A team runs its elicitation study twice, the second time with the referents refined so that participants understand each command's effect. What does the unit predict for the agreement rate on first-choice gestures?

3A product ships a three-finger swipe for a destructive action, with no visible control and no tutorial. Which judgement does the unit's argument license?

4A team shows participants a set of gestures, tests recall a day later, finds the set memorable and concludes that the gestures are discoverable. Which diagnosis fits the unit?

5Essay Choose a gestural interaction you can observe or read about, for example a two-finger swipe to delete a row, a pinch to zoom, a mid-air gesture in a headset, or a free-hand gesture for a television. Design the evaluation that would tell you whether the gesture is discoverable, and then argue what your result would and would not license. State who your participants are and what they are told before they act, what you measure on a first attempt and after a delay, whether you test in the laboratory or the setting of use, and how you would report agreement. Use Shneiderman (1983), Hutchins, Hollan and Norman (1985) and Norman (2010) for what direct manipulation's visible object provided and what a gesture loses; Wobbrock, Aung, Rothrock and Myers (2005) for guessability as a measured quantity; Wobbrock, Morris and Wilson (2009) for the elicitation method and its shown-the-effect limit; Vatavu and Wobbrock (2015) and (2019) for what an agreement rate does and does not separate; Nacenta, Kamber, Qiang and Kristensson (2013) for the difference between inventing and remembering; Piumsomboon, Clark, Billinghurst and Cockburn (2013), Vatavu (2012) and Liu, Li, Chen, Wu and Liang (2024) for the move away from a surface; and Hinrichs and Carpendale (2011) for what the setting adds. Close by naming the comparison that would show whether your conclusion transfers to another command or another setting.

Fix what the participant is told before they act, because that is the whole argument. Every elicitation study in the unit portrays the effect of a command and then asks the participant to perform its cause, which is why Wobbrock, Morris and Wilson (2009) can report agreement at all: the referent is already understood when the gesture is proposed. That design measures reassignment, and the only defensible way to measure discovery is to present the interface without instruction and observe whether the correct gesture is produced on the first attempt. So I would run two conditions on the same command, one where the effect is portrayed and one where it is not, and I would report the first-attempt success rate separately from the agreement rate, because the two answer different questions. I would take the gesture set from an elicitation study rather than from my own preference, since Wobbrock, Aung, Rothrock and Myers (2005) show that guessability is a measurable quantity that can be improved, and I would measure it between people rather than within a person. For the gesture set itself I would need more than one number. Vatavu and Wobbrock (2015) add a disagreement rate for referents and a coagreement rate between referents, and Vatavu and Wobbrock (2019) show that a low rate can come from an unclear referent rather than an unguessable gesture, so I would report the pair and check referent clarity before reading a low rate as a verdict. I would measure retention after a delay as well, because Nacenta, Kamber, Qiang and Kristensson (2013) find user-defined gestures easier to remember, with up to a 24 per cent recall advantage, and that advantage begins only after learning, so it is a different property from first-encounter discovery. The setting matters to what I could conclude. Hinrichs and Carpendale (2011) found gesture use shaped by social and interaction context, with gestures forming sequences, and an elicitation study asks one person for one command at a time, so a laboratory result would describe a laboratory. If the interaction is mid-air I would expect the debt to be larger, since Piumsomboon, Clark, Billinghurst and Cockburn (2013) take the method off the surface, and Vatavu (2012) shows the resulting set is specific to its domain, while Liu and colleagues (2024) show agreement can be raised by refining referents, which means a poor result may indict my referents rather than the gesture. Finally I would be clear about what the result licenses. Shneiderman (1983) and Hutchins, Hollan and Norman (1985) locate the advantage of direct manipulation in a visible object and a short distance between the user's goal and the interface's expression, and Norman (2010) shows that a gesture supplies neither until familiarity makes it behave like a label. So a high first-attempt rate would license shipping the gesture alone for that command in that setting; anything less would licence shipping it as an accelerator beside a visible control. To test transfer I would repeat the no-instruction condition on a second command and in a second setting, because the unit's own evidence says agreement is command-dependent and context-dependent.