Direct manipulation and the costs it lowers

Year 1 / Module 1 Foundations of human-computer interaction / Unit 6 / Direct manipulation and the costs it lowers

HCI1101 · Unit 6 of 10

Direct manipulation's case rests on the cost of learning and the cost of error rather than on speed, and the field collected evidence about speed while leaving that case largely untested.

Direct manipulation and the costs it lowers

Introductory accounts of graphical interfaces repeat one claim as a finding. A system that keeps the object of interest visible and lets the user act on it directly is easier to learn and safer to use than a command language. Shneiderman (1983) made that claim as a thesis. The features were plausible, and the causal link was a proposal.

This unit argues that the case for direct manipulation rests on the cost of learning and the cost of error rather than on speed. The field collected evidence about speed while leaving the other two costs largely untested. The two fail differently. A speed claim can be shown wrong by a slower interface. A learning or error claim depends on who the user is, how often they return and whether the work can be undone, so it survives a stopwatch test. The next unit asks whether usability can name those costs.

What Shneiderman measured and what he asserted

The founding paper draws on display editors, the VisiCalc spreadsheet, spatial data management and video games. It derives four principles. The first is continuous representation of the object of interest. The second is physical actions or labelled buttons instead of complex syntax. The third is rapid, incremental and reversible operations whose impact is immediately visible. The fourth is layered learning that permits use with minimal knowledge (Shneiderman, 1983). The paper says that novices learn basic functionality quickly, that intermittent users retain operational concepts, and that users feel less anxiety because actions are reversible. Those are learning and error claims.

The speed claim carries a study: display editors took half the overall performance time of line-oriented editors. The learning and error claims carry no trial; they rest on users' enthusiasm. The paper's own problems section cuts against it. Flowcharts did not improve comprehension, debugging or modification over code alone, icons can take as long to learn as words, and a graphic can mislead. Graphics also consume space: fifty document names in a table beat ten abbreviated icons for experienced users. The paper asks for research on each feature's limitations, and left that testing to others.

The accounts explain the feeling, and the numbers the field collected measure time

Hutchins, Hollan and Norman (1985) sought a cognitive account of the advantages and disadvantages together. Engagement is the perceived locus of control, the difference between acting and instructing a hidden intermediary. Distance is the mental effort of turning a goal into an interface action and reading the response back. Directness is said to be their sum. Neither of the two costs appears in the model. Frohlich (1997) records the gap: the account promotes manual action over language and cannot represent mixed systems.

The figures that migrated into practice measure latency. Nielsen (1993) assembles them into three limits and attributes the advice, unchanged for thirty years, to Miller (1968) and Card, Robertson and Mackinlay (1991). At about 0.1 second the user feels the system reacts instantaneously, which Nielsen states is the limit for feeling that one is directly manipulating objects. At about 1 second the flow of thought continues. At about 10 seconds attention leaves the dialogue. Each of the three assumes a person watching a screen for a change, so the figure moves with the device and the connection, and a user who hears the response is outside them all. Nothing in the three limits measures what a user would have learned, or what error avoided, through a command line.

Delegation is the standing alternative

Maes (1994) argues for indirect management, in which autonomous agents hide complexity, act on the user's behalf and monitor events. In a recorded exchange, Maes argues that agents should take over routine work, and Shneiderman that users keep control only when they can see and reverse what happens (Shneiderman and Maes, 1997). They agree the system must be predictable and disagree about who initiates action.

Gentner and Nielsen (1996) invert each Macintosh principle:

Metaphors become reality
Direct manipulation becomes delegation
See and point becomes describe and command
Consistency becomes diversity
WYSIWYG becomes representing meaning
User control becomes shared control
Feedback and dialog become the system handling details
Forgiveness becomes modelling user actions
Aesthetic integrity becomes graphic variety
Modelessness becomes richer cues

Their objection is scale: direct manipulation suits a simple action on a few objects, but as objects and actions multiply it becomes repetitive drudgery, reducing the user from an executive to an assembly line worker. Horvitz (1999) frames the choice of acting, waiting or asking in decision-theoretic terms, and prices the cost of a wrong action.

The sharpest critique came from people who helped build the style

Raskin (2000) argues that interfaces should be judged by whether habits transfer and actions can be undone. He identifies modes, states in which the same action produces different results, as a primary avoidable source of error.

The error cost that Raskin argues from has been documented where modes are unavoidable. Sarter and Woods (1995), studying pilot interaction with glass-cockpit automation, argued that proliferating modes without supporting the new monitoring demands they create produces new mode-related error forms. The pilot must track which mode the automation is in and what it is doing. The accident at Strasbourg in January 1992 shows the mechanism in one episode. The crew began the descent with the flight control unit in heading and vertical speed. They selected 33 in its window, intending a 3.3 degree flight path angle, and the aircraft remained in vertical speed. It came down at about 3,300 feet per minute where roughly 800 would have held the intended path, and it struck the ridge, killing 87 people. The inquiry's simulations traced the recorded path to the speed mode rather than the angle mode. The two differed on the unit by a mode annunciation and a decimal point (Bureau d'Enquetes et d'Analyses, 1992). Reversibility would not have helped: the mode was wrong from the start and the display did not say so.

The evidence was thin in both directions, and the metaphor was reified

Frohlich (1997) converts that instinct into an audit: a manual mode, he concludes, does not always lead to direct or usable interfaces. His count matters: limitations as well as benefits rest on a small number of tests showing disadvantages on certain tasks, and naturalistic choice of alternatives on fewer still. He also reports Whittaker's finding that continuous representation makes objects hard to find in a large system, and that incremental reversal slows compound actions. Blackwell (2006) supplies the mechanism behind that thin testing. Treating metaphor as a design tool reified a loose analogy into a specification, so designers sought the wrong kind of correspondence between screen and task.

Instrumental interaction recast the style as a design space

Beaudouin-Lafon (2000) proposes instrumental interaction, in which a user acts on domain objects through interaction instruments. He presents it as a generalisation of direct manipulation covering graphical, two-handed and augmented reality techniques, converting a style into a design space. Beaudouin-Lafon (2004) draws the consequence: computing power rose a thousandfold in twenty years while interfaces stayed essentially the same. Research should therefore design interaction rather than interfaces, with principles such as immediate feedback attached to it. The vocabulary stayed unsettled: Hornbaek and Oulasvirta (2017) find that interaction is field-defining yet confused. The concepts in use, dialogue, transmission, optimal behaviour, embodiment and tool use, carry different scopes and accounts of causality. Their desiderata ask for propositions strong enough to fail.

Contested question: is the advantage speed, learning, or error, and who has tested it?

The literature divides on which cost is decisive. The speed position is the best evidenced: Nielsen (1993) states directness as a sub-second timing property, and Shneiderman (1983) reports the study that halved task time. The learning and error position carries the design consequences but not the trials: Shneiderman (1983) asserts it from enthusiasm, Hutchins, Hollan and Norman (1985) model it without measuring it, and Raskin (2000) argues it from habit and error. The delegation position denies the premise, since Maes (1994) and Gentner and Nielsen (1996) hold that manipulation becomes the cost at scale. Horvitz (1999) shows the trade-off can be priced. The strongest counter is Gentner and Nielsen's: at scale, manipulation itself becomes the cost.

Frohlich (1997) is the arbiter the field has not replaced. What would settle the disagreement is a comparison that varies the cost of error independently of speed, holding task and latency constant. If the learning and error account is right, a reversible interface should show fewer unrecovered errors or faster second attempts than an irreversible one at the same latency. If speed is the whole story, matching latencies should erase the difference. The tests Frohlich reports were not designed that way.

While it is open, design for the costs that can be designed away. Reversibility, immediate feedback and few modes answer the error case without settling whether it is the cause of the benefit. To settle that in a product, log unrecovered errors and second-attempt times, since latency alone cannot show whether an interface taught or was merely quick.

Where the argument is heading

The move now is to treat direct manipulation as one instrument among several. Beaudouin-Lafon (2000, 2004) makes interaction the design object, Horvitz (1999) shows the trade-off between acting and asking can be priced rather than argued, and Hornbaek and Oulasvirta (2017) set out the definitions such work needs first. Raskin (2000) remains the caution that the field's most successful style was never held to the standard it set itself.

The sources this unit rests on

Shneiderman, B. (1983). Direct manipulation: a step beyond programming languages. Computer 16(8), 57-69. doi:10.1109/MC.1983.1654471 Named the style, its four principles and its problems. Its speed evidence is a citation to Roberts; its learning and error claims rest on enthusiasm.
Hutchins, E. L., Hollan, J. D. and Norman, D. A. (1985). Direct manipulation interfaces. Human-Computer Interaction 1(4), 311-338. doi:10.1207/s15327051hci0104_2 Gave the cognitive account of directness through engagement, the perceived locus of control, and distance, the mental effort between goal and action, stated as their sum.
Nielsen, J. (1993). Response times: the three important limits. In Usability Engineering, chapter 5, Morgan Kaufmann. https://www.nngroup.com/articles/response-times-3-important-limits/ Assembles the advice on response time into three limits, 0.1 second, 1.0 second and 10 seconds. The figures are attributed to Miller (1968) and Card, Robertson and Mackinlay (1991) and packaged rather than tested.
Miller, R. B. (1968). Response time in man-computer conversational transactions. AFIPS Fall Joint Computer Conference, 267-277. doi:10.1145/1476589.1476628 The source of the three response-time limits Nielsen packages, proposed from experience rather than measured.
Maes, P. (1994). Agents that reduce work and information overload. Communications of the ACM 37(7), 30-40. doi:10.1145/176789.176792 Argues for indirect management and reports prototype agents for meeting scheduling, email handling, news filtering and entertainment. Delegation rather than manipulation is its answer to growing digital work.
Shneiderman, B. and Maes, P. (1997). Direct manipulation vs. interface agents. Interactions 4(6), 42-61. doi:10.1145/267505.267514 A recorded exchange in which Maes argues that agents should take over routine work and Shneiderman that users keep control only when they can see and reverse what happens.
Gentner, D. and Nielsen, J. (1996). The Anti-Mac interface. Communications of the ACM 39(8), 70-82. doi:10.1145/232014.232032 Inverts each Macintosh principle to derive a coherent alternative, pairing direct manipulation with delegation and forgiveness with modelling user actions, on the objection that at scale manipulation reduces the user to an assembly line worker.
Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99), 159-166. doi:10.1145/302979.303030 Sets out a decision-theoretic framework for when an automated service should act, wait or ask, demonstrated in a scheduling system where the interventions judged useful were those with a low expected cost of a wrong action.
Raskin, J. (2000). The Humane Interface: New Directions for Designing Interactive Systems. Addison-Wesley. https://archive.org/details/humaneinterfacen0000rask Argues from habit and error that interfaces should let habits transfer and actions be undone, and identifies modes as a primary avoidable source of error. This note rests on published descriptions and reviews rather than the text.
Frohlich, D. M. (1997). Direct manipulation and other lessons. Handbook of Human-Computer Interaction, 2nd edition (Helander, Landauer and Prabhu, eds.), North-Holland, 463-488. doi:10.1016/B978-044481862-1.50087-X Audits the evidence fifteen years on and concludes that a manual mode does not always lead to direct or usable interfaces. It reports the limitations as suggested by a small number of tests showing disadvantages.
Beaudouin-Lafon, M. (2000). Instrumental interaction: an interaction model for designing post-WIMP user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '00), 446-453. doi:10.1145/332040.332473 Proposes that interaction be described as the use of instruments on domain objects, and presents the model as a generalisation of direct manipulation covering graphical, two-handed and augmented reality techniques.
Beaudouin-Lafon, M. (2004). Designing interaction, not interfaces. Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '04), 15-22. doi:10.1145/989863.989865 Argues that computing power rose a thousandfold in twenty years while interfaces stayed essentially the same, so improvement requires designing interaction rather than interfaces.
Hornbaek, K. and Oulasvirta, A. (2017). What is interaction? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '17), 5040-5052. doi:10.1145/3025453.3025765 Observes that interaction is field-defining yet confused, and extracts concepts including dialogue, transmission, optimal behaviour, embodiment and tool use, which carry different scopes and accounts of causality.
Blackwell, A. F. (2006). The reification of metaphor as a design tool. ACM Transactions on Computer-Human Interaction 13(4), 490-530. doi:10.1145/1188816.1188820 Traces the history of the ideas behind interface metaphor and adds a study of researchers' attitudes thirty years later, arguing that treating metaphor as a design tool turned a loose analogy into a specification.
Sarter, N. B. and Woods, D. D. (1995). How in the world did we ever get into that mode? Mode error and awareness in supervisory control. Human Factors 37(1), 5-19. doi:10.1518/001872095779049516 Argues from studies of pilot-automation interaction in glass cockpits that proliferating modes, without support for the new monitoring demands they create, produces mode-related error forms.
Bureau d'Enquetes et d'Analyses. (1992). Accident to the Airbus A320 registered F-GGED operated by Air Inter, 20 January 1992 (English translation). BEA, Paris. https://www.bea.aero/uploads/tx_elydbrapports/F-GGED.pdf The official inquiry: the crew selected 33 expecting a 3.3 degree path while the aircraft was in vertical speed mode, giving about 3,300 feet per minute instead of the roughly 800 the path required. Eighty-seven people died.

Check your understanding & apply it · 4 questions, 1 essay

1A cockpit field takes a single number entered in one window. Depending on a mode the crew must remember, that number means a flight path angle or a vertical speed, and a small annunciation is the only sign of which mode is active. A crew entering 33 for a 3.3 degree path got about 3,300 feet per minute instead of about 800. Which reading does this unit support?

2This unit separates the speed claim made for direct manipulation from the learning and error claims. Which grading does it give them?

3A team cites Hutchins, Hollan and Norman to argue that its new interface will be easier to learn. What does this unit say the account supplies, and what is missing?

4This unit's contested question asks what would settle whether the case for direct manipulation is about speed, learning or error. Which evidence does it propose?

5Essay A hospital trust is replacing the file browser in its electronic records system. The design brief says the new interface will be easier to learn and safer to use because it is direct: clinicians will drag a record into a ward folder rather than type a filing code, and every action will show its result at once. Using Shneiderman (1983) on the founding principles and their problems, Hutchins, Hollan and Norman (1985) on engagement and distance, Nielsen (1993) on response time limits, Maes (1994), Shneiderman and Maes (1997), Gentner and Nielsen (1996) and Horvitz (1999) on delegation, mixed initiative and the cost of a wrong action, Raskin (2000) on habit, modes and reversal, Frohlich (1997) on the evidence for the style's advantages, and Blackwell (2006) on metaphor, write an assessment that (a) separates the brief's claims into speed claims, learning claims and error claims, saying which of them the cited literature supports and how, (b) says what the trust must measure to test the learning and error claims separately from speed, naming the comparison and the outcome that would count against the brief, and (c) states the strongest case for replacing the drag gesture with a delegation or mixed-initiative design, and what evidence would decide between the two.

The brief makes three kinds of claim and the literature treats them very differently. The speed claim is the best supported, since Shneiderman (1983) reports that display editors took half the overall performance time of line-oriented editors and Nielsen (1993) gives 0.1 second as the limit for feeling that one is directly manipulating objects, but Nielsen also shows those figures were packaged from work done thirty years earlier rather than tested for the chapter. The learning claim, that the interface will be easier to learn, rests on Shneiderman's assertion that novices learn basic functionality quickly, not on a trial, and Shneiderman's own problems section warns that icons can take as long to learn as words. The error claim, that the interface will be safer, is the one the trust most needs and the least supported: Shneiderman says users feel less anxiety because actions are reversible, Hutchins, Hollan and Norman (1985) explain the feeling of directness through engagement and distance without measuring error, and Raskin (2000) argues from habit that modes, in which the same action produces different results, are a primary avoidable source of error, which is exactly what a folder-drag can create. Frohlich (1997) reports the supporting tests as a small number showing disadvantages and an even smaller number showing naturalistic choice of alternatives. To separate the claims, the trust should compare a reversible drag version with an irreversible one at the same response latency, and count unrecovered filing errors and second-attempt times, especially for clinicians returning after a gap; if matching latency erases the difference, the speed account is sufficient. The strongest case for delegation is Gentner and Nielsen's (1996) argument that with many objects direct manipulation becomes repetitive drudgery, reinforced by Maes (1994) on agents that take over routine work; Horvitz (1999) supplies the way to decide, by comparing the expected cost of a wrong automatic filing against the cost of asking, and Shneiderman and Maes (1997) show the two positions disagree about who initiates rather than about predictability. Evidence would decide it: if mis-filings would harm patients, the expected cost favours asking; if the gesture is rare and cheap to reverse, manipulation wins.