Text entry is two measurements in tension, so one speed figure reports the participant's choices

Year 2 / Module 15 Interaction techniques and usability evaluation / Unit 2 / Text entry is two measurements in tension, so one speed figure reports the participant's choices

Text entry is two measurements in tension, so one speed figure reports the participant's choices

HCI2107 · Unit 2 of 10

Speed, accuracy and correction effort in text entry trade against one another, so any single number converts a participant's decision about whether to repair an error into a performance score, and the field's disagreement about which metric to use is really a disagreement about which behaviour it wants to count; the large-sample studies settled the magnitude of the trade but not the measure.

Text entry is two measurements in tension, so one speed figure reports the participant's choices

A text entry study reports a speed, and nearly always an accuracy beside it, as though the two together described a technique. This second of ten units stays on one channel and argues something narrower: they describe a participant's decision. One volunteer repairs every mistake and another types over them, and the standard measures price the two policies differently. A single figure therefore reports a policy as much as a technique. Which metric to use is a disagreement about which behaviour to count.

Effort and accuracy are separate quantities

MacKenzie (2002) defined keystrokes per character as the average keystrokes needed to produce one character. He computed it across techniques, reporting values from about 10 for cursor-key selection to about 0.5 for word prediction, with error-free Qwerty typing near 1.0 and multi-tap keypad entry at 2.03. A designer can compute the figure from the technique and the language alone, before running any participant, though not for a person writing in a second language or on a predictive keyboard. Effort is therefore a quantity separate from accuracy, not a part of it.

Accuracy is not one quantity either. MacKenzie and Soukoreff (2002) described a sequence-alignment technique. It derives the optimal alignments between the presented and the transcribed text. It classifies each per-character error as an insertion, a substitution or a deletion. It replaces an aggregate rate with a description of what went wrong where.

The metrics mix two decisions and need a common text

Soukoreff and MacKenzie (2003) examined the two statistics the field had settled on, the minimum string distance error rate and keystrokes per character, and found each confounds quantities a study needs to keep apart. The two also move in opposite directions. The participant who corrects everything adds backspaces, which raises keystrokes per character and leaves a transcript close to what was asked for, which lowers the error rate. The participant who presses on does the reverse. Neither number is in error, since each faithfully reports a decision about whether an error was worth repairing.

Their repair combines the presented text, the input stream and the transcribed text. It derives a unified total error rate from a corrected and a not corrected error rate and adds correction efficiency and conscientiousness.

None of these measures travels unless the text does. MacKenzie and Soukoreff (2003) published 500 phrases for text entry evaluation, with utility software for their statistical properties, arguing a shared stimulus is what makes results generalisable. The set became the field's default, and a study that invents its own text cannot join the literature.

Techniques buy speed by trading accuracy or memory

Zhai and Kristensson (2003) built SHARK, adding a shorthand gesture for each word defined by its movement pattern on a stylus keyboard, so familiar words are written as one gesture and unfamiliar ones tapped, which buys speed with a memory load. SHARK2 (Kristensson and Zhai, 2004) removed the mode switch, so any word in a vocabulary of 10,000 to 20,000 could be written as a single gesture, moving the technique from tracing to recall-based gesturing. A speed figure recorded after training averages the two strategies and hides the transition.

EdgeWrite (Wobbrock, Myers and Kembel, 2003) recognises letters from the sequence of corners hit while a finger traverses the edges and diagonals of a square hole. Recognition is therefore deterministic rather than statistical. Able-bodied users were 18 per cent more accurate than with Graffiti, a difference that did not reach significance. Of four participants with motor impairments, all could use EdgeWrite while some could not use Graffiti.

EdgeBraille (Mattheiss, Regal, Schrammel, Garschall and Tscheligi, 2015) enters six-point Braille by swiping one finger along a touchscreen's edges. A laboratory study with 14 participants compared it with a talking keyboard, then two weeks of daily training with seven. Words per minute and error rate showed no significant difference. Where a null result is the finding, a metric that ranks techniques by entry rate ranks this one as ordinary.

The large samples settled the trade

Dhakal, Feit, Kristensson and Oulasvirta (2018) analysed 136 million keystrokes from 168,000 volunteers. Letter pairs typed by different hands or fingers predict speed better than letter repetition, and rollover typing is prevalent. Clustering divides most users into eight groups differing in performance and accuracy. Palin, Feit, Kim, Kristensson and Oulasvirta (2019) ran a web-based transcription task with 37,370 volunteers, reporting an average mobile entry rate of 36.2 words per minute with 2.3 per cent uncorrected errors. Auto-correct use correlated positively with entry rate, and word prediction use correlated negatively.

The trade is now established at a scale no laboratory could reach, though neither result supplies a measure and the second cuts at the effort proxy directly. Quinn and Zhai (2016) had shown why: in a transcription experiment, greater assertiveness reduced keyboard actions and was preferred, but attending to suggestions impaired average time.

Contested question: should the field unify the measure or keep the quantities apart?

Soukoreff and MacKenzie (2003) hold that the problem is soluble by a better metric. Separate corrected from uncorrected errors, report the total beside correction efficiency and conscientiousness, and policy and cost can be read off together. MacKenzie (2002) holds the opposite. He treats keystrokes per character as a characteristic computed a priori rather than an error metric, so folding it into a unified rate compresses two things that ought to stay apart. Quinn and Zhai (2016) hold that the proxy underneath both is wrong, since attention is a cost the keystroke count omits.

What would settle it is not another framework but a study with the correction policy as a manipulated factor: one technique, one phrase set from the shared pool, participants assigned to repair everything or to press on, and words per minute, corrected and uncorrected error rates, correction efficiency and task time reported so the technique by policy interaction can be read off. The question is whether rankings survive the change of policy, and no such study appears in this unit's list. Komninos (2023) proposes instead a simulator generating reproducible synthetic users, which would settle the question once its fidelity to human data is established.

The evidence is unevenly distributed. Soukoreff and MacKenzie (2003) rest on a reanalysis of the two field standards and on a worked framework, Quinn and Zhai (2016) on a controlled experiment that manipulated assertiveness, and Komninos (2023) on a simulator whose fidelity is argued rather than measured. The strongest objection is that the unit's own reporting norm is untested, since no study in this list assigns an error-repair policy while holding the technique and the phrase set fixed. A designer should report the correction policy beside the speed figure while the dispute is open.

Where the argument is heading

Recent work points away from the single figure and towards reporting the policy. Dhakal and colleagues (2018) show typing performance clusters into groups, which makes a population mean a poor description of any individual. Palin and colleagues (2019) report speed, uncorrected error rate and the direction of two intelligent-entry features that pull against each other. Komninos (2023) attacks the small laboratory study as replicable but not reproducible. Kristensson and Vertanen (2014) supply the goal the field measures against, an inviscid entry rate of 67 words per minute where the user rather than the technique is the bottleneck, which is a target and not a ranking because it describes people. The likely settlement is a reporting norm rather than a universal metric: state the correction policy, the phrase set and the keyboard's features, and treat the speed as an outcome of those choices.

Chen and colleagues (2024) add a technique whose effort no longer sits against its accuracy. Three large language model assisted methods for entry in extended reality cut manual keystrokes by 16.4, 49.9 and 43.7 per cent at the word, grammar and sentence levels, and, unlike the techniques above, they did not increase manual corrections, while reported physical, mental and temporal load fell. A rate recorded for such a method reports how much of the writing the model supplied, so the policy the figure carries is the model's rather than the participant's.

The sources this unit rests on

Chen, L., Cai, Y., Wang, R., Ding, S., Tang, Y., Hansen, P. and Sun, L. (2024). Supporting Text Entry in Virtual Reality with Large Language Models. 2024 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). doi:10.1109/vr58804.2024.00073 Three language model assisted methods for entry in extended reality cut manual keystrokes by 16.4, 49.9 and 43.7 per cent at the word, grammar and sentence levels and did not increase manual corrections, while reported physical, mental and temporal load fell. An entry rate for such a method reports how much the model supplied.
Dhakal, V., Feit, A. M., Kristensson, P. O. and Oulasvirta, A. (2018). Observations on Typing from 136 Million Keystrokes. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3173574.3174220 The authors analysed 136 million keystrokes from 168,000 volunteers, found letter pairs typed by different hands or fingers predict speed better than repetition, and clustered most users into eight groups differing in performance and accuracy, so a population mean describes no one.
Komninos, A. (2023). RoboType: Realistic Mobile Text Entry Evaluations with Synthetic Users. Proceedings of the 25th International Conference on Mobile Human-Computer Interaction. doi:10.1145/3565066.3608693 The authors present an open-source simulator that generates mobile text entry evaluations with synthetic users, so results are cheap and reproducible. Its fidelity to human data is argued rather than established.
Kristensson, P. O. and Vertanen, K. (2014). The inviscid text entry rate and its application as a grand goal for mobile text entry. Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services. doi:10.1145/2628363.2628405 They define the inviscid rate as the point where the user's creativity rather than the technique is the bottleneck, and estimate its population mean from a proxy at 67 words per minute, far above published mobile methods, which makes it a goal and not a ranking.
Kristensson, P. O. and Zhai, S. (2004). SHARK2: a large vocabulary shorthand writing system for pen-based computers. Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology. doi:10.1145/1029632.1029640 SHARK2 removed the need to alternate between gesture and tapping, so any word in a vocabulary of 10,000 to 20,000 could be written as one gesture, supporting a transition from tracing to recall-based gesturing. It moves speed and error together.
MacKenzie, I. S. (2002). KSPC (Keystrokes per Character) as a Characteristic of Text Entry Techniques. Lecture Notes in Computer Science. doi:10.1007/3-540-45756-9_16 MacKenzie defines keystrokes per character as the average keystrokes needed to produce one character, reporting values from about 10 for cursor-key selection to about 0.5 for word prediction, with multi-tap keypad entry at 2.03. Being computable without participants makes effort separate from accuracy.
MacKenzie, I. S. and Soukoreff, R. W. (2002). A character-level error analysis technique for evaluating text entry methods. Proceedings of the second Nordic conference on Human-Computer Interaction. doi:10.1145/572043.572056 They use sequence comparison to align the presented and the transcribed text and classify per-character errors as insertions, substitutions or deletions. It replaces an aggregate rate with a taxonomy, the detail behind the claim that the metric decides what counts as an error.
MacKenzie, I. S. and Soukoreff, R. W. (2003). Phrase sets for evaluating text entry techniques. CHI '03 Extended Abstracts on Human Factors in Computing Systems. doi:10.1145/765891.765971 The authors publish 500 phrases with utility programs for computing a phrase set's statistics, arguing a pre-defined stimulus is what makes results generalisable. It is the comparability instrument a cross-study claim needs.
Mattheiss, E., Regal, G., Schrammel, J., Garschall, M. and Tscheligi, M. (2015). EdgeBraille: Braille-based text input for touch devices. Journal of Assistive Technologies. doi:10.1108/jat-10-2014-0028 EdgeBraille enters six-point Braille characters by swiping one finger along a touchscreen's edges, and was compared with a talking keyboard in a laboratory study with 14 participants and two weeks of daily training with seven. Words per minute and error rate showed no significant difference, so this is the case where entry rate is not the governing measure.
Palin, K., Feit, A. M., Kim, S., Kristensson, P. O. and Oulasvirta, A. (2019). How do People Type on Mobile Devices? Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services. doi:10.1145/3338286.3340120 A web-based transcription task with 37,370 volunteers produced an average mobile typing speed of 36.2 words per minute with 2.3 per cent uncorrected errors, with auto-correct use correlating positively and word prediction use negatively with entry rate.
Quinn, P. and Zhai, S. (2016). A Cost-Benefit Study of Text Entry Suggestion Interaction. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. doi:10.1145/2858036.2858305 The authors controlled interface assertiveness, presenting suggestions always, never, or gated by a probability threshold. Greater assertiveness reduced keyboard actions and was preferred, but attending to suggestions impaired average time, so fewer keystrokes did not mean less effort.
Soukoreff, R. W. and MacKenzie, I. S. (2003). Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/642611.642632 The authors identify shortcomings in the minimum string distance error rate and keystrokes per character and propose a framework analysing the presented text, the input stream and the transcribed text together, deriving a unified total error rate with correction efficiency and conscientiousness.
Wobbrock, J. O., Myers, B. A. and Kembel, J. A. (2003). EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion. Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology. doi:10.1145/964696.964703 EdgeWrite recognises letters from the sequence of corners hit while a finger or stylus traverses the edges and diagonals of a square hole, so recognition is deterministic. Able-bodied users were 18 per cent more accurate than with Graffiti, a non-significant difference; of four participants with motor impairments all could use EdgeWrite.
Zhai, S. and Kristensson, P. O. (2003). Shorthand writing on stylus keyboard. Proceedings of the conference on Human Factors in Computing Systems. doi:10.1145/642629.642630 SHARK adds a shorthand gesture for each word defined by its movement pattern on a stylus keyboard, so familiar words are written as one gesture and unfamiliar ones tapped. A user study demonstrated a technique trading tapping effort against recall.

Check your understanding & apply it · 4 questions, 1 essay

1A study of a new soft keyboard instructs participants to repair every error. It reports a high entry rate beside a very low uncorrected error rate, and its authors present both as properties of the keyboard. Which diagnosis does the unit support?

2Two groups evaluate the same technique on the same shared phrase set. One group instructs participants to repair every error, the other tells them to press on. What does the unit predict for the two standard measures?

3A team evaluates a predictive keyboard in which auto-correct silently replaces mistyped words. It must decide what to publish. Which set does the unit's argument license?

4A team wants one number to rank five keyboards. Its protocol invents its own stimulus text, and it leaves the correction policy unstated because it wants participants to behave naturally. Which diagnosis fits the unit?

5Essay Choose one text entry technique you can observe or read about, such as a phone soft keyboard, a stylus shorthand method, an on-screen Braille method or a predictive keyboard. State how you would evaluate it: the phrase set you would use, the correction policy you would impose on participants, and the measures you would report. Then argue whether your numbers would report the technique, the participant or the choices you made as experimenter. Use MacKenzie (2002) and Soukoreff and MacKenzie (2003) for what the standard measures count and how they move, MacKenzie and Soukoreff (2003) for the comparability instrument, Zhai and Kristensson (2003) and Kristensson and Zhai (2004) or Wobbrock, Myers and Kembel (2003) for a technique that trades the two measurements against each other, Quinn and Zhai (2016) and Palin and colleagues (2019) for the cost that a keystroke count leaves out, and Kristensson and Vertanen (2014) for the target your speed figure will be judged against. Close by naming the comparison that would settle whether your measure transfers to another technique.

Begin by fixing the stimulus, because everything that follows depends on it. Use the 500-phrase set of MacKenzie and Soukoreff (2003) rather than text of your own invention: their argument is that a shared pool is what makes results generalisable, and without it your numbers cannot be compared with the literature you want to join. Then fix the correction policy, and say so in the method, because a participant who repairs every error and one who presses on are performing different tasks. Soukoreff and MacKenzie (2003) show the mechanism: repairs add backspaces, which raises keystrokes per character, and leave a transcript close to the target, which lowers the minimum string distance error rate, while a participant who presses on does the reverse. If you impose one policy and report one error rate, you have measured the policy as much as the keyboard. So instruct participants explicitly, or measure the policy instead of imposing it, and report the corrected and uncorrected error rates separately with correction efficiency, as that framework provides. State the effort term properly as well. MacKenzie (2002) computes keystrokes per character a priori, from the technique and the language, with values for English ranging from about 10 for cursor-key selection to about 0.5 for word prediction, and that range is why the measure is a characteristic rather than a verdict: a predictive keyboard reaches a low value by moving effort into attention, and Quinn and Zhai (2016) found that assertive suggestions cut keyboard actions and were preferred while average time got worse, a result Palin and colleagues (2019) echo in the negative correlation between word prediction use and mobile entry rate. Now pick a technique and make the trade visible. SHARK (Zhai and Kristensson, 2003) and SHARK2 (Kristensson and Zhai, 2004) write familiar words as single gestures and leave unfamiliar ones to be tapped, so the same technique behaves differently before and after learning; EdgeWrite (Wobbrock, Myers and Kembel, 2003) buys stability and accuracy rather than speed, and its 18 per cent accuracy advantage did not reach significance. Report a speed figure for either of those without saying which strategy participants had reached and you have averaged the design away. Finally, judge the number against something. Kristensson and Vertanen (2014) estimate the inviscid entry rate, where the user rather than the technique is the bottleneck, at 67 words per minute and find most published methods far below it, which is a useful goal and a poor ranking, because the target is a property of people. The comparison that would settle whether your measure transfers is the one nobody in this literature has run: take two techniques, hold the phrase set fixed, and vary the correction policy as a factor, so that the technique by policy interaction can be read off directly. If the ranking survives the change of policy, your measure describes the technique; if it flips, it described the experimenter's instructions.