Interaction is a relation, not a function: state, mode and the next-state map

A drawing tool has a pointer and a pen: drag with one and you move the shape under the cursor, drag with the other and you draw a line, and the next state differs although the input does not. This unit argues that an interactive system is a relation from a state and an input to the next state. A model that takes the input alone mispredicts exactly where modes exist. The repair is to make the mode visible rather than to train the user to remember it. Unit 3 asks what it costs to move around the space that map generates.

A function of the input alone fails wherever a mode exists

A function gives exactly one output for each input, so a model that takes the keypress alone asserts that the effect of a keypress is a function of the keypress. That assertion is testable and it fails, since the effect depends on a second argument the model has discarded. Stated properly, the machine is a next-state map from a state and an input to a next state. Where each state and input admits at most one next state, the map is a partial function and the system is deterministic. Where more than one next state is possible, it is a relation in the strict sense.

Two careless readings follow, and both are properties of the relation: if the map is not total, some gesture reaches a state where it has no meaning and quietly does nothing, and if it is not injective, two gestures carry the same command and the design cannot tell them apart.

The text editor vi is the standard case: in command mode the key x deletes the character under the cursor, and in insert mode the same key inserts the letter x. A model written as a function of the keypress therefore fails precisely at the states the editor calls modes; written as a next-state map the ambiguity vanishes, since the state carries the mode and the pair of mode and key fixes the buffer.

Statecharts give the relation a notation

Enumerating every combination of independent aspects yields a state for each one, so the machine needs a notation. Harel's statecharts supply one, extending state-transition diagrams with hierarchy, concurrency and communication to keep a small diagram modular. Hierarchy lets a mode contain sub-states, and concurrency lets independent aspects sit side by side rather than multiplied into one flat set. A nondeterministic transition is a decision rather than an accident: a specification that admits two next states for one state and event has not decided what the interface does.

A toolbar makes the notation concrete. A drawing tool with a current tool and a current selection is two orthogonal regions inside one state: a tool region cycles among pen, shape and text, and a selection region is empty or holds an object. Concurrency keeps them as two regions rather than six flat states, and a transition can then be guarded on their combination, so delete is enabled only while the selection is non-empty.

Modes are in the state, not in the person

The empirical case that a mode error belongs to the state comes from Sellen, Kurtenbach and Buxton. They counted mode errors in a text-editing task, such as a navigation command issued in insert mode, and crossed kinesthetic feedback with a visual mode indicator: both reduced errors, the kinesthetic manipulation more effectively, and it lowered the cognitive load of changing mode. A second experiment found a nonlatching foot pedal, held active by the user, a more salient indication of state than a latching one. The authors concluded that user-maintained mode states prevent mode errors more effectively than system-maintained ones. The repair generalises beyond sight: a mode signalled only by a visual indicator is imperceptible to a user reading the interface through a screen reader, so make the mode perceptible in a channel the user can read, moving the burden of conforming to the system (Wobbrock and colleagues, 2011).

Norman supplies the account that stops the analysis blaming the person: from action slips he proposed a schema theory in which a parent schema activates child schemas carrying triggering conditions and activation values, so a correctly formed intention can fire a wrong sub-action. In his taxonomy a mode error is a subcategory of errors in forming the intention, distinct from a capture error. The plan is well formed and the state was misread.

Kieras and Polson turn the same structure into a predictor rather than a diagnosis: they represent the user's procedural knowledge and the device's behaviour formally, so the knowledge required and the cognitive load can be quantified and simulated, and if learning an interface can be read off its transition structure the relation predicts which version is easier to learn.

The relation is an evaluation instrument, not only a specification

Degani and Heymann make the criterion explicit: they build formal models of an automated machine and of the operator's procedure, then verify on an autopilot case study whether the interface supplies the information the operator needs, training material included. Their definition is the one this unit needs: the display is adequate when the operator's model can track the machine's state.

Fields, Harrison and Wright formalise the same demand from the other side, specifying systems so that the fit between information presented and information required can be reasoned about, with the shortfall expressed as a requirement on the user's memory, so a shortfall becomes a located defect at a state rather than a complaint. Paterno's book states the broader tradition, setting out task models, abstract interaction objects and dialogue models centred on ConcurTaskTrees and keeping the interaction model distinct from the implementation.

Continuous state is the case the relation has to absorb

The counter-case the argument must absorb is the non-WIMP interface, where the interesting state is not a set of discrete screens. Jacob, Deligiannidis and Morrison argue that a non-WIMP dialogue such as a virtual environment is a set of continuous relationships, most temporary, described by a data-flow component and enabled and disabled by an event-based one. A drag is the smallest instance: while the pointer button is held, the interaction is a continuous relationship between pointer position and object position, and release ends it with one discrete event. Sampling the pointer at the display refresh rate makes that relationship discrete, but not a finite set of states, so the argument from a small state machine gives way to a question about how finely a continuous quantity must be sampled.

The relation can be recovered from the artefact

If the relation is the right model, the practical objection is the cost of writing one by hand. Memon, Banerjee and Nagarajan answered it with GUI ripping. That process traverses a running interface and reconstructs a model as a GUI forest and event-flow graphs with little human intervention. Memon consolidated the models used for GUI testing into one scalable event-flow model, in which events and their interactions form a directed graph that algorithms can reverse-engineer, generate tests from and measure coverage on.

Santiago Junior, Ozcan and Balera show where this practice has arrived, combining search-based with model-based testing. Test cases are generated against functional and non-functional objectives at once, with the event flow graph read from source or built from use cases, and over thirty-two problem instances NSGA-II performed best overall. The premise rather than the result is the point: the transition graph is assumed available and cheap.

Amalfitano and colleagues supply the caution, recasting fully automatic online Android testing techniques into one framework: the parameters of the technique, not only of the application, influence coverage and cost, so no single recovered transition model is ground truth.

Contested question: signal the mode, or remove it?

The literature does not agree on what to do once the state is on the table. The first position is this unit's: the mode is a real state, and the repair is to make it perceptible, on the evidence of a two-experiment laboratory study of text editing and a formal analysis of one autopilot case (Sellen, Kurtenbach and Buxton, 1992; Degani and Heymann, 2002), which shows feedback working in one editor and one aircraft, narrower than the claim it is asked to support. The second position is Raskin's, and it is more radical: modes are a design defect, and unavoidable state should be a quasimode that persists only while the user holds an action, as the shift key does, paired with universal undo. Its evidence is a design treatise and a working prototype rather than a controlled comparison, so it is the stronger prescription and the weaker warrant.

The disagreement is sharper than it first appears, because Raskin and Sellen share a mechanism while differing on the prescription. Raskin's quasimode is a user-maintained state, and Sellen's second experiment found a held pedal a more salient indication of state than a latching one, so their most error-resistant condition was user-maintained; the data can be read as support for removing or self-maintaining the state rather than signalling it. A designer who takes making the mode visible as the whole lesson has taken half of Sellen's result. A second disagreement sits underneath: whether the enumerated state relation is the right object at all, since Jacob's continuous dialogue and Raskin's modeless programme both suggest that some states an interface contains are artefacts of its present design.

What would settle it is a comparison rather than an argument. The decisive study would implement one command set three ways: a system-maintained mode with a display, one with kinesthetic feedback, and a user-maintained quasimode. It would measure error rate, recovery cost and time on task across expertise and over repeated sessions.

That study has not been run. Meanwhile, treat a mode as guilty until it is either made perceptible or removed: inventory the states, and prefer a quasimode wherever the state can be held rather than latched.

Where the argument is heading

The direction of travel is to make the relation cheap to obtain and cheap to check. Bolton, Jimenez, van Paassen and Trujillo note that model checking human-automation interaction has required an analyst to formulate the specification properties by hand, work that is itself error-prone, so they generate the properties mechanically from the task model. With the transition graph recovered automatically in Memon's line of work and in Santiago Junior's, both ends are automated, and two questions remain: are the states recovered by crawling the ones that matter to a user, and how is the relation carried into systems whose state is continuous?

The sources this unit rests on

• Harel, D. (1987). Statecharts: a visual formalism for complex systems. Science of Computer Programming, 8(3), 231-274. doi:10.1016/0167-6423(87)90035-9 Hierarchy, concurrency and communication give state diagrams a compact semantics.
• Sellen, A. J., Kurtenbach, G. P. and Buxton, W. (1992). The prevention of mode errors through sensory feedback. Human-Computer Interaction, 7(2), 141-164. doi:10.1207/s15327051hci0702_1 Both feedback kinds cut mode errors, the kinesthetic more. A user-maintained mode state was the most salient of the conditions tested.
• Norman, D. A. (1981). Categorization of action slips. Psychological Review, 88(1), 1-15. doi:10.1037/0033-295X.88.1.1 Mode errors are errors in forming the intention, not carelessness. The failure is in the operator's model of state.
• Kieras, D. and Polson, P. G. (1985). An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies, 22(4), 365-394. doi:10.1016/S0020-7373(85)80045-6 Formal models of knowledge and device allow the learning complexity of an interface to be quantified.
• Degani, A. and Heymann, M. (2002). Formal verification of human-automation interaction. Human Factors, 44(1), 28-43. doi:10.1518/0018720024494838 Verifies whether an interface supplies the information a task needs unambiguously, including training material.
• Fields, B., Harrison, M. and Wright, P. (1995). Modelling interactive systems and providing task relevant information. In Paterno (ed.), Interactive Systems: Design, Specification, and Verification, 253-266. Springer. doi:10.1007/978-3-642-87115-3_15 Reasoning about presented versus required information supports design comparison. A shortfall is a defect at a state.
• Paterno, F. (2000). Model-Based Design and Evaluation of Interactive Applications. Springer. doi:10.1007/978-1-4471-0445-2 Task models and dialogue models support evaluation before construction: the unit's tradition in book form.
• Jacob, R. J. K., Deligiannidis, L. and Morrison, S. (1999). A software model and specification language for non-WIMP user interfaces. ACM Transactions on Computer-Human Interaction, 6(1), 1-46. doi:10.1145/310641.310642 A non-WIMP dialogue is continuous relationships enabled and disabled by discrete events.
• Memon, A. M., Banerjee, I. and Nagarajan, A. (2003). GUI ripping: reverse engineering of graphical user interfaces for testing. Proceedings of the 10th Working Conference on Reverse Engineering (WCRE), 260-269. doi:10.1109/WCRE.2003.1287256 Traversing a running interface rebuilds it as a GUI forest and event flow graphs.
• Memon, A. M. (2007). An event-flow model of GUI-based applications for testing. Software Testing, Verification and Reliability, 17(3), 137-157. doi:10.1002/stvr.364 Consolidates GUI models into a scalable event-flow model. The relation becomes a graph a tool holds.
• Bolton, M. L., Jimenez, N., van Paassen, M. M. and Trujillo, M. (2014). Automatically generating specification properties from task models for the formal verification of human-automation interaction. IEEE Transactions on Human-Machine Systems, 44(5), 561-575. doi:10.1109/THMS.2014.2329476 Specification properties are generated mechanically from task models.
• Santiago Junior, V. A. de, Ozcan, E. and Balera, J. M. (2022). Many-objective test case generation for graphical user interface applications via search-based and model-based testing. Expert Systems with Applications, 208, 118075. doi:10.1016/j.eswa.2022.118075 GUI test cases are generated against several objectives at once; the event flow graph comes from source or use cases.
• Amalfitano, D., Amatucci, N., Memon, A. M., Tramontana, P. and Fasolino, A. R. (2017). A general framework for comparing automatic testing techniques of Android mobile apps. Journal of Systems and Software, 125, 322-343. doi:10.1016/j.jss.2016.12.017 Automatic online Android testing techniques are recast into one framework and evaluated on common subjects, with parameter values influencing coverage and cost.
• Raskin, J. (2000). The Humane Interface: New Directions for Designing Interactive Systems. Addison-Wesley. https://www.informit.com/store/humane-interface-new-directions-for-designing-interactive-9780201379372 Modes are a design defect; unavoidable state should be a held quasimode with universal undo.
• Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S. and Froehlich, J. (2011). Ability-based design: concept, principles and examples. ACM Transactions on Accessible Computing, 3(3), Article 9. doi:10.1145/1952383.1952384 Argues that accessible computing should address ability in context and move the burden of conforming to the system.