Interaction techniques and usability evaluation

Two questions run through the study of interaction. The mechanical one asks what a person does when they point, type or make a gesture, and how fast and how accurately. The evidential one asks how you would know whether a design is any good, and how much weight a number from an evaluation should carry.

A decision about target size or menu depth is a prediction about behaviour, and a prediction can be tested. An evaluation is worth its cost only when the method can support the claim made from it. The contested question here is whether usability evaluation is measurement or judgement.

Pointing is a channel with a measurable capacity

Fitts's law treats a movement to a target as the transmission of information. Difficulty is measured in bits by the index of difficulty, the logarithm to base two of twice the distance to the target divided by its width. Doubling the distance adds one bit; halving the width adds another (MacKenzie, 1992).

It is a predictive model. Place a frequent control twice as far from the pointer and you can estimate the cost before you build it. Throughput, the rate of information transfer, compares two devices on one scale.

It breaks down where the interesting cases live. MacKenzie's reanalysis shows observations curving above the regression line at low difficulty, and the model failing when difficulty is small: Fitts's own throughput of 5.56 bits per second at one bit sits more than three standard deviations from the mean of 10.10. Narrow targets are the second problem, costing disproportionately more time than extra distance and raising error rates. The repair replaces the nominal width with an effective width taken from where the hits landed (MacKenzie, 1992).

Take a toolbar. Moving a frequent action into an overflow menu makes the target smaller and further, so the prediction is a large increase in difficulty. Measure the same task with the same participants before and after, and the argument becomes a measurement rather than a preference. Across-study comparisons are inconsistent, so a throughput figure means only what its conditions allow.

Text entry has to be scored on speed and accuracy at once

Text entry is reported as a speed in words per minute and an accuracy, and accuracy is the harder half. Soukoreff and MacKenzie showed that both standard measures fail alone. The minimum string distance error rate counts the edits needed to turn what the participant produced into what was asked for. Keystrokes per character counts effort: error-free typing on a physical keyboard sits near 1.0, while multi-tap entry on a keypad averaged 2.03 (Soukoreff and MacKenzie, 2003).

The two move in opposite directions. A participant who leaves an error uncorrected lowers keystrokes per character and raises the string distance error rate; one who repairs everything does the reverse. One number therefore reports a choice the participant made. Their framework separates corrected from uncorrected errors, adds them into a total error rate, and adds correction efficiency and conscientiousness (Soukoreff and MacKenzie, 2003).

Compare a soft keyboard with an eye-gaze keyboard. Report words per minute, both error rates and the phrase set used, because speed and accuracy trade off and one figure can improve by trading the other away. The field has no single agreed measure, so a study that invents its own cannot be compared.

Menus, breadth and depth, and a result that depends on the model you assume

A hierarchy can be a few levels with many choices each, or many levels with few choices each. Landauer and Nachbar measured selection from touch menus and found mean response time per screen well described by a function of the logarithm of the number of alternatives at each step, a decision component in series with a movement component. On that function, breadth beats depth (Landauer and Nachbar, 1985).

The paper is careful about its reach: this function favours breadth, and other functions might not. The result is a prediction from an assumed cost per decision and per movement, and it applies to reaching a known item.

Take a settings screen. A tree of three levels with four items each costs fewer movements and more decisions than two levels of eight, so the shallower structure is predicted to be faster and can be tested. It says nothing about finding an item whose name the user does not know, or about label clarity. Breadth also demands better labels, and label quality is rarely manipulated in menu studies.

Direct manipulation, gesture and the discoverability debt

Direct manipulation works because objects are visible, actions are physical or labelled, and effects are immediate and reversible. Gestural input keeps the first property and loses the second: a gesture carries no label, and on a surface with no chrome there is nothing to press if the user does not already know what to do.

Wobbrock, Morris and Wilson put a number on this. They showed 20 participants the effect of each of 27 tabletop commands and asked them to perform the cause, logging 1080 gestures. Agreement varied greatly by command: some produced a shared gesture, others almost no consensus, and the authors concluded that those need on-screen widgets. One hand was preferred to two, and desktop idioms shaped what participants tried (Wobbrock, Morris and Wilson, 2009).

Take a two-finger swipe to delete a row. If a user-defined study shows weak agreement, shipping the gesture alone imposes a discoverability debt on every new user; offer a visible control as well and keep the gesture as an accelerator. Whether a gesture set should follow what users invent or what the platform already does is a product decision rather than a matter of evidence.

Feedback you can feel, and the latency at which interaction stops feeling direct

Haptic output arrives through vibration or force and is felt rather than seen, so it survives a phone in a pocket, a headset that fills the view and a glance-free control. The channel is coarse: vibrotactile patterns must stay distinguishable when brief, and a pattern means nothing until it has been learned. Haptics suits notification, confirmation and guidance, and fails as the only way to carry a large vocabulary of states.

Consider a wearable that steers a walker with three vibration patterns: left, right and stop. Add twelve patterns for points of interest and the channel collapses, because the wearer cannot separate them under load.

Latency has published limits. Nielsen reported three widely cited thresholds: 0.1 second for the user to feel that the system is reacting instantaneously, 1 second for the flow of thought to stay uninterrupted, although the user loses the feeling of operating directly on the data, and 10 seconds for attention to stay on the dialogue (Nielsen, 1993).

Those numbers describe perception, not permission. A server round trip of 1.2 seconds is under the attention limit and over the directness limit, so acknowledge the action at once and finish the work behind it. Users adapt to fast systems and then judge slower ones harshly, so measure the delay your own users notice in your own product.

Inspection methods do not agree with themselves

Heuristic evaluation and cognitive walkthrough are inspection methods. A person examines an interface against general principles, or works through a task step by step asking whether a user could tell what to do next, and returns a list of predicted problems. Both are cheap, and both produce a list rather than a measurement.

The reliability problem is documented. Hertzum and Jacobsen reviewed eleven studies of heuristic evaluation, cognitive walkthrough and thinking-aloud studies and found that two evaluators using the same method on the same interface agreed on between 5 per cent and 65 per cent of the problems they reported. The effect held for novices and experts, for cosmetic and severe problems, and for detection and severity judgement alike. A single evaluator's report is therefore highly questionable as an authoritative statement about an interface, and the practical response is several evaluators and a merge, with the effect managed rather than removed (Hertzum and Jacobsen, 2001).

Take a consultant's report. One evaluator produces forty problems and ranks them by severity, and the client acts on the top five. The ranking carries the authority of a number and the reliability of one person's attention. Three independent evaluators would produce a different list, and their union describes the interface better than any one of them. Cognitive walkthrough is proposed as the more disciplined alternative, yet the review found the effect persisting there too (Hertzum and Jacobsen, 2001).

Testing with people, and the observer who changes what is observed

Testing can be moderated, with a facilitator present, or unmoderated, with the participant alone and the session recorded, in a laboratory or remotely in the participant's own setting. Moderation lets the facilitator probe, which produces richer explanation and introduces a person who affects what happens. Unmoderated running trades that for reach, and reduces the risk that the researcher steers the participant.

Greenberg and Buxton argued that evaluation is treated as obligatory when it should be conditional. Evaluation done by rule rather than by thought harms the process: premature evaluation can kill promising ideas or stop a team pursuing several competing ones; testing an immature invention measures the technology rather than the idea; and a confirmatory test that shows a prototype works is weak science, because it is an existence proof rather than an attempt at refutation. The case is not against evaluation, but for choosing the method to fit the question (Greenberg and Buxton, 2008).

Think-aloud is the standard technique, and it changes what it measures. Hertzum, Hansen and Andersen compared silent performance with two regimes. Classic thinking aloud had little effect on behaviour apart from lengthening tasks. Relaxed thinking aloud, which is what practitioners usually mean, produced longer tasks, more widely distributed visual attention, more navigation commands, and higher reported mental workload on the task load index (Hertzum, Hansen and Andersen, 2009).

Take a timed comparison of two designs. Under relaxed think-aloud the timings carry the cost of talking and are not comparable with unmoderated data. Either label the timing as an outcome of the session as run, or run the comparison silently and collect the reasoning afterwards.

Metrics, sample size and the five-user rule

A test produces standard measures: completion, time on task, error count, and what the user thought. The System Usability Scale is the common attitudinal measure: ten statements scored into one value from 0 to 100, and that number is not a percentage. Bangor, Kortum and Miller added a seven-point adjective rating as an eleventh question to nearly a thousand surveys and found it correlated strongly with the score, at 0.822. Scores in the seventies were acceptable, and anything below 70 gave cause for concern (Bangor, Kortum and Miller, 2009).

Workload is measured separately, usually with the NASA Task Load Index, the subjective instrument that Hertzum and colleagues used to detect the extra load that thinking aloud imposed (Hertzum, Hansen and Andersen, 2009). A workload score is a finding about the task, not about preference.

Sample size is where the field is least careful. Nielsen's argument begins from a model in which each user independently finds a fixed proportion of the problems, typically 31 per cent, and it follows that five users find about 85 per cent of them. Nielsen notes that the formula holds only for comparable users and that distinct user groups need more participants (Nielsen, 2000).

The model is the weak part. It assumes that every user has the same chance of meeting each problem, independently of what the others found, and real problems range from those almost everyone hits to those one person in twenty meets. The same study can therefore find almost everything or miss a great deal, and the rule does not tell you which you got.

Reporting that a client can act on

Findings earn their cost when they can be acted on. That needs three things kept apart: what was observed, naming the task and the participant; what it is thought to mean, marked as inference; and what should change, with the smallest change that would test the interpretation. Severity should be defined before the study, not assigned after it, and the participant count belongs beside every finding: a problem seen once in five sessions and one seen in all five are different claims.

That discipline, and the engineering that follows it, is where this material hands on next: turning a design that has survived evaluation into a product that can be built, tested automatically and changed in production without breaking.