Direct and mid-air input trade precision and endurance for naturalness

Year 3 / Module 19 Immersive, tangible and spatial interaction / Unit 7 / Direct and mid-air input trade precision and endurance for naturalness

Direct and mid-air input trade precision and endurance for naturalness

HCI3103 · Unit 7 of 10

Direct and mid-air input are not automatically better than indirect input, because they cost precision and endurance, and the relevant test is the sustained session rather than the demonstration.

Direct and mid-air input trade precision and endurance for naturalness

The case for taking the interface off the desk is always the same. Point with your hand and the computer should understand. The case is a trade: direct and mid-air input buy a mapping the body already knows, and pay for it in pointing accuracy and in the endurance of an unsupported arm. What matters is not whether the interaction looks natural for thirty seconds, but whether it survives a task that sets the duration.

Two measures let the price be stated. Fitts (1954) priced pointing in bits, and Hincapie-Ramos and colleagues (2014) priced shoulder effort as a ratio computable while the user works. This seventh of ten units moves from travelling to pointing.

Fitts priced pointing, and the price is set by distance and width

Fitts (1954) had participants tap between two target areas, and found movement time rising linearly with an index of difficulty. That index comes from the amplitude of the movement and the width of the target, counted in bits.

Lubos, Bruder and Steinicke (2014) apply the measure to head-mounted selection in its effective form, resizing targets so the error rate sits near four per cent. Pointer performance has a common currency: a claim that an interface is natural says nothing about bits per second. Bowman, Johnson and Hodges (2001) measure 6.57 seconds per trial with Go-Go against 3.28 seconds with ray casting.

Directness pays where the work is two-handed

Hinckley and colleagues (1994) argued that a physical object carries meaning a screen cannot. Their neurosurgical interface gave the surgeon a tracked head model and a cutting plane prop, so a section could be specified by holding a plate against the model. The props let two hands express a spatial relation as one cognitive chunk. The evidence was an informal evaluation in which surgeons who had never seen the interface used it after a cursory introduction: learnability demonstrated rather than compared.

Forlines, Wigdor, Shen and Balakrishnan (2007) ran the comparison on a tabletop: direct touch beat two mice on a bimanual task and was preferred by ten of twelve participants, but committed more errors, 18.9 per cent against 9.7 per cent, and the mouse was judged better for single point work.

Mid-air input spends the endurance of an unsupported arm

Hincapie-Ramos and colleagues (2014) named the gorilla arm effect and made it measurable. Consumed Endurance is the ratio of time spent interacting to the time the shoulder can hold the contraction before it needs rest, computed from tracked arm posture. Validated against the Borg CR10 rating of perceived exertion, it found the least endurance consumed with the arm bent midway between shoulder and waist.

The trade reappears in the techniques meant to replace the direct hand. Bowman, Johnson and Hodges (2001) report that occlusion selection caused significantly higher arm strain than ray casting. They advise a two degree of freedom ray technique where remote-selection speed matters. Indirect input also costs, but it moves the cost from the shoulder to the wrist.

In a headset the error sits in the eye rather than the hand

Lubos, Bruder and Steinicke (2014) tested where direct selection in a head-mounted display goes wrong. Their Fitts' law experiment placed targets in different regions of the space in front of the user. Hypothesising that errors would concentrate along the line of sight rather than the hand's movement, they found errors highest along the view axis and marginal across the orthogonal plane, about 1.2 centimetres against 1.1. Perception therefore dominates rather than motor control, so they derive an ellipsoid cursor tolerant along the view axis and a rule that high precision work belongs close to the eyes.

A headset grants direct interaction while denying the cues a hand relies on: no haptic contact, an invisible body, and focus and convergence at different distances (Lubos, Bruder and Steinicke, 2014).

The deficit is also a design variable

Bacim, Kopper and Bowman (2013) treat the precision problem as something to design around. Their technique casts a volume containing the target and refines through a four item menu, so no step demands a precise action. It was more accurate than ray casting and faster for small targets in less cluttered environments, so precision is exchanged for steps, and the exchange wins where targets are small.

Shoemaker, Tang and Booth (2007) design the same exchange into the display: their Shadow Reaching technique projects the user's shadow onto a very large wall. A perspective transform makes stepping towards or away from the surface change the shadow's size, and therefore the user's reach. The authors state the cost plainly: widening the reach coarsens the resolution of the interaction.

Naturalness is a claim about the world the interface borrows

Jacob and colleagues (2008) supply the vocabulary. Their reality based interaction framework groups the styles by naive physics, body awareness and skills, environment awareness and skills, and social awareness and skills, and argues that borrowing which supplies a skill also imports the limits of the world it came from. They list the trade-offs: realism against power, efficiency, ergonomics and practicality, and predict that a direct technique will be easy to learn while inheriting the physical world's costs, not that it will be better.

Two studies show the costs landing. Cockburn and McKenzie (2002) found location memory slowing and clutter ratings worsening as a third dimension was added, with the reliable difference in the physical models at 2.6 against 3.6 seconds. Masurovsky and colleagues (2020) found a handheld controller ahead of hand tracking on a grab and place task with thirty two participants, on performance, usability and preference. Their design work cut accidental drops from 4.75 to 2.41 per session against 0.125 for the controller, so the results do not support higher naturalness for hand tracking.

The contested question: is the deficit in the body or in the technique?

The literature divides over where the deficit belongs. On the technique position, the deficit is an engineering problem that is being solved. Progressive refinement makes a difficult target easy without demanding precision (Bacim et al., 2013), and score based selection keeps improving on ray casting for the targets that defeat it (Kruger et al., 2024). Virtual pointing is reported to outperform the virtual hand across a body of studies (Argelaguet and Andujar, 2013), so the costs are historical and falling.

On the body position, the costs follow from the situation rather than the implementation. Fitts' law fixes what a pointing movement costs once its amplitude and target width are settled, whatever performs it (Fitts, 1954). Headset errors arise in perception, along the view axis (Lubos et al., 2014). The finger occludes and the arm is short, which is why a mouse holds the single point case (Forlines et al., 2007). Hand tracking lost to a controller on accuracy, speed, drops and preference (Masurovsky et al., 2020), and more freedom in the third dimension did not improve location memory (Cockburn and McKenzie, 2002).

What would settle the dispute is one within-subject experiment rather than another technique, fixing the task and the participants. It would compare a direct, a mid-air and an indirect technique. It would report effective throughput in bits per second together with consumed endurance over a session as long as the deployment implies. No study in this unit's reading list runs that design, though Lou and colleagues (2024) add a fatigue term to Fitts' law for repeated pointing blocks, bringing endurance into the model for hand tracking alone. The pointing experiments use short blocks of trials, and the strongest hand tracking comparison covers grab and place (Masurovsky et al., 2020).

The unit's thesis has a serious counter-argument. If direct and mid-air input must trade precision and endurance for naturalness, the trade should survive every comparison, yet progressive refinement makes a small target easy (Bacim, Kopper and Bowman, 2013) and ray casting beat Go-Go and occlusion selection on speed and strain (Bowman, Johnson and Hodges, 2001). The evidence on each side is uneven: the body position rests on Fitts's model and on endurance ratios measured at fixed angles rather than during interaction. What a designer should do: price precision by effective throughput in bits per second (Lubos, Bruder and Steinicke, 2014), price effort by consumed endurance over a session as long as the deployment implies, choose by the dominant cost, price the hand as well, since a tremor or an ageing hand changes which cost dominates, and treat naturalness as a design detail.

Where this is heading

The direction of travel is towards techniques that model intent rather than require precision. Kruger and colleagues (2024) extend score based selection, which infers the target from gesture history rather than demanding an accurate ray. Fatigue is now an input computable in real time (Hincapie-Ramos et al., 2014), and effective throughput in bits per second, with the error rate near four per cent, is what Lubos and colleagues (2014) apply. The next useful contribution is unlikely to be another technique, but one protocol reporting throughput and endurance together over a session long enough for the second to matter.

Sources

Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology. doi:10.1037/h0055392 Participants tapped between two target areas as amplitude and width varied. Movement time rose linearly with an index of difficulty in bits, pricing a movement before any device is chosen.
Hinckley, K., Pausch, R., Goble, J. C. and Kassell, N. F. (1994). Passive real-world interface props for neurosurgical visualization. CHI '94. doi:10.1145/259963.260443 Built a neurosurgical planning interface from tracked passive props, a head model and a cutting plane used with two hands. Surgeons who had never seen it used it after a cursory introduction, so the claim rests on demonstration rather than comparison.
Bowman, D. A., Johnson, D. B. and Hodges, L. F. (2001). Testbed evaluation of virtual environment interaction techniques. Presence. doi:10.1162/105474601750182333 Evaluated selection and travel techniques in a reusable testbed. Go-Go took 6.57 seconds per trial against 3.28 for ray casting, and occlusion selection caused significantly higher arm strain, moving speed and effort together.
Cockburn, A. and McKenzie, B. (2002). Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. CHI '02. doi:10.1145/503376.503413 Compared location memory for web page images in 2D, 2.5D and 3D arrangements. Retrieval slowed as the third dimension was added, with the reliable difference in the physical models at 2.6 against 3.6 seconds.
Forlines, C., Wigdor, D., Shen, C. and Balakrishnan, R. (2007). Direct-touch vs. mouse input for tabletop displays. CHI '07. doi:10.1145/1240624.1240726 Compared direct touch with mouse input on unimanual and bimanual tabletop tasks. Touch won the bimanual task and was preferred by ten of twelve participants but produced more errors, 18.9 against 9.7 per cent.
Jacob, R. J. et al. (2008). Reality-based interaction: a framework for post-WIMP interfaces. CHI '08. doi:10.1145/1357054.1357089 Proposed a framework grouping interaction styles by naive physics, body awareness and skills, environment awareness and skills, and social awareness and skills. Borrowing from the real world supplies skill and imports its limits, with trade-offs against power, efficiency and ergonomics.
Bacim, F., Kopper, R. and Bowman, D. A. (2013). Design and evaluation of 3D selection techniques based on progressive refinement. International Journal of Human-Computer Studies. doi:10.1016/j.ijhcs.2013.03.003 Designed SQUAD, which casts a volume around the target and refines through a four item menu so no step requires precision. It was more accurate than ray casting and faster for small targets, trading precision for steps.
Shoemaker, G., Tang, A. and Booth, K. S. (2007). Shadow reaching: a new perspective on interaction for large displays. UIST '07. doi:10.1145/1294211.1294221 Projects a perspective-transformed shadow of the user onto a very large wall so that movement towards or away from the screen changes the effective reach. The first prototype cast a real shadow from a lamp ten feet from the screen, and the authors note that widening the reach coarsens interaction resolution.
Argelaguet, F. and Andujar, C. (2013). A survey of 3D object selection techniques for virtual environments. Computers and Graphics. doi:10.1016/j.cag.2012.12.003 Surveyed 3D selection, separating the virtual hand from virtual pointing metaphors. A number of studies find virtual pointing more effective than competing metaphors, while effective selection with pointing gestures remains open.
Hincapie-Ramos, J. D., Guo, X., Moghadasian, P. and Irani, P. (2014). Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. CHI '14. doi:10.1145/2556288.2557130 Defined Consumed Endurance as the ratio of interaction time to the time the shoulder can sustain the contraction. Validated against the Borg CR10 scale, it found least endurance consumed with the arm bent between shoulder and waist.
Lubos, P., Bruder, G. and Steinicke, F. (2014). Analysis of direct selection in head-mounted display environments. IEEE Symposium on 3D User Interfaces. doi:10.1109/3dui.2014.6798834 Ran a Fitts' law experiment on direct selection within arm's reach in a head-mounted display. Errors were highest along the view axis, about 1.2 centimetres against 1.1 across it, so perception rather than motor control dominates.
Masurovsky, A., Chojecki, P., Runde, D., Lafci, M., Przewozny, D. and Gaebler, M. (2020). Controller-free hand tracking for grab-and-place tasks in immersive virtual reality. Multimodal Technologies and Interaction. doi:10.3390/mti4040091 Compared a customised hand tracking interface and the standard one with a handheld controller on a grab and place task. The controller led on performance and preference, and the customised interface cut accidental drops from 4.75 to 2.41 against 0.125.
Kruger, M., Gerrits, T., Romer, T., Kuhlen, T. and Weissker, T. (2024). IntenSelect+: enhancing score-based selection in virtual reality. IEEE Transactions on Visualization and Computer Graphics. doi:10.1109/tvcg.2024.3372077 Presents an enhanced score based selection by volume technique. The original improved on ray casting but chiefly for small point-like objects and was hard to parameterise, which the enhanced version addresses.
Lou, X., Song, X., Hu, X., Ma, M., Fu, L., Zhuang, X. and Fan, X. (2024). An extended Fitts' law model for hand interaction evaluation in virtual reality. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2024.2443708 Ran a target-pointing experiment comparing translation and angular rotation mapping with a hand controller and freehand interaction. Arm fatigue affected movement time exponentially with task duration, and the authors derive model formulations including the spatial variables and the fatigue term the standard model omits.

Check your understanding & apply it · 4 questions, 1 essay

1A manufacturer builds a head-mounted assembly guide for technicians who select and rotate small components over thirty minute shifts. Half the selections are single-point and half are two-handed. Which advice does this unit support?

2A team plans to price the effort of its mid-air technique by counting the gestures a participant performs before requesting a break. Which correction does this unit support?

3One engineer says the pointing deficit lies in the body; another says it lies in the technique. Which experiment would separate the two positions, as this unit frames it?

4A team reports that surgeons who had never seen a prop-based interface used it after a cursory introduction, and concludes that the interface beats a graphical one. Which diagnosis does this unit support?

5Essay A manufacturer is building a head-mounted assembly guide for technicians who select and rotate small components over thirty minute shifts. Half the selections are single point, choosing one bolt from a cluster; the other half are two-handed, holding a part steady while rotating a tool. Using this unit's sources, advise the team on which selection techniques to use for each half of the task, state the trade each choice accepts, and design the comparison that would let the manufacturer decide rather than rely on a demonstration. Cite the sources you rely on at each step.

The unit's evidence does not support a single technique for both halves of the task. The starting point is that pointing has a price set by the geometry of the task, since Fitts (1954) found movement time rising linearly with an index of difficulty built from amplitude and target width, and Lubos, Bruder and Steinicke (2014) apply that measure to head-mounted selection with the error rate held near four per cent. The first consequence is that the team cannot buy accuracy with naturalness. For the single point selections, where a small bolt must be chosen from a cluster, the evidence points away from a bare finger. Forlines, Wigdor, Shen and Balakrishnan (2007) found direct touch committed more selection errors than two mice, 18.9 per cent against 9.7 per cent, and judged the mouse better for single point work; Masurovsky and colleagues (2020) found a handheld controller ahead of controller free hand tracking on accuracy, speed and preference, with about seven millimetres of error against sixteen. If motor precision is the requirement, the team should either use a controller or use a refinement technique, since Bacim, Kopper and Bowman (2013) made selection much more accurate than ray casting by refusing to demand precision at any step, and Kruger and colleagues (2024) continue that line with score based selection. For the two-handed part, where a part is held while a tool is rotated, direct two-handed interaction has the stronger case, because Hinckley and colleagues (1994) argued that physical props let two hands express a spatial relation as one cognitive chunk, and Forlines and colleagues (2007) found direct touch beat two mice on their symmetric bimanual task and was preferred by ten of twelve participants. The trade must be stated: direct two-handed work costs shoulder endurance, which Hincapie-Ramos and colleagues (2014) show is predictable from posture and least when the arm is bent on the plane midway between shoulder and waist, so the team should specify that working plane. A controller would move the cost to the wrist instead. One further limit belongs in the brief. Lubos and colleagues (2014) found errors concentrated along the view axis and concluded that perception rather than motor control dominates, so the layout should keep small targets close to eye level and use a tolerant cursor rather than assuming a fingertip will land where it looks. Cockburn and McKenzie (2002) add a caution about claiming spatial memory benefits, since the third dimension made retrieval slower in their physical models and no statistically reliable advantage appeared among their virtual arrangements. To decide rather than demonstrate, the manufacturer should run one within-subject study with technicians performing both halves of the task under direct, mid-air and indirect selection, reporting effective throughput in bits per second alongside consumed endurance over sessions of the full thirty minutes, and testing component recall afterwards. No study in this unit's reading list runs that design, since the pointing experiments use short blocks of trials and the endurance work holds static postures.