Immersive, tangible and spatial interaction

Spatial interaction has been sold on a promise for thirty years. Put on the headset and you will be somewhere else. Pick up the object and the computer will understand your hands. The technology can deliver that promise under the right conditions, but as a general claim about benefit it remains unproven. The contested question is therefore narrow and testable. For a specified task, does a more immersive or more tangible system produce a better outcome, and what evidence would settle it?

The discipline that follows is straightforward. Work upward from the hardware, because it sets the ceiling on everything designed above it. Weigh each claimed benefit against the evidence offered for it, and treat a vision statement as a hypothesis until it is tested.

What the hardware fixes before any design begins

A system's immersion is a property of the machinery, not of the person using it. Slater and Sanchez-Vives (2016) define an immersive system as one that lets a person perceive through natural sensorimotor contingencies. In plain terms, the display updates your view as your head and body move. That depends on tracking, field of view, resolution and latency. Nothing designed above that layer can repair a tracker that lags behind the head.

Tracking is the first constraint, and Azuma (1997) treats it as one of the two biggest problems in augmented reality. Registration is the alignment of a virtual object with the real scene it is drawn onto, and its tolerance is the eye's. Azuma also names sensing error, which is the error in knowing where the head and the camera actually are. A display can be bright, sharp and comfortable and still fail, because the virtual object swims against the real one.

Take a concrete case: optical see-through guidance for a repair. If the tracker drifts by a few millimetres, the drawn arrow points at the wrong screw. The technician then learns to distrust the whole overlay. This is why the honest question about a new headset is not its resolution but its motion-to-photon latency, the delay between a head movement and the matching change on the display. The literature does not agree on a single threshold at which that delay becomes visible. A designer should measure it for the task rather than quote a number.

Immersion is a property of the system; presence is inferred from the person

Slater and Sanchez-Vives (2016) separate two terms that marketing uses interchangeably. Immersion is the technical capability of the system, which they describe as the physics of the system. Presence is the subjective correlate, the illusion of being there. They divide it further into place illusion, the sense of being in the rendered place, and plausibility illusion, the sense that what is happening there is really happening.

The separation matters because the two are measured in different ways. Immersion is read from a specification. Presence has to be inferred from the person, and the strongest evidence in their account is behavioural rather than verbal. They report that when participants were confronted by a virtual fire, about ten per cent ran out of the room. The fire did not look realistic, and the participants knew it was not real.

That single figure states the measurement problem plainly. If you asked those participants, they would say the fire was not real. Their bodies acted otherwise. Self-report about presence therefore tells you what a person believes about an experience, which is not the same as how they responded to it. Physiological and behavioural measures are stronger, and both are expensive and easy to disturb.

The mixed reality continuum decides what the user can still see

Milgram and Kishino (1994) replace the binary of real and virtual with a continuum. At one end is a purely real environment and at the other a purely virtual one. Augmented reality sits between them, where an otherwise real environment is augmented with computer generated objects. Augmented virtuality is the converse case, where a virtual scene is augmented with real objects. They then add three dimensions: what the system knows about the world, how faithfully it reproduces it, and how far it creates the illusion that the observer is inside it.

Azuma (1997) gives the definition the field still uses, and it is deliberately technology independent. A system is augmented reality when it combines real and virtual, is interactive in real time, and is registered in three dimensions. The third condition excludes a flat overlay on live video, and it excludes film, because film is not interactive.

Choosing a position is a design decision, and it turns on what the real world contributes. Where the real environment carries information the system cannot reproduce, such as the state of a machine under repair, augmented reality is the right choice. Where the real world is the wrong place to practise, a fully virtual environment is better, because a dangerous task can be reset and observed. Choose the position from the task, not from the novelty of the hardware.

Depth cues conflict, and the eye pays for it

Stereo displays create a conflict that the visual system cannot resolve. In the real world, the eyes converge on an object and the lens focuses at the same distance, so vergence and accommodation agree. A conventional stereo display puts every object on one physical surface. The lens must then focus at the screen while the eyes converge at the depicted depth. Hoffman, Girshick, Akeley and Banks (2008) call this the vergence and accommodation conflict, and their four experiments test what it costs.

They built a display that presented focus cues correct or nearly correct for the depicted scene. With it, participants identified a stereoscopic stimulus faster and showed better stereoacuity under time pressure. They also perceived less distortion in depth and reported less fatigue. The ordinary display produces the opposite: flattened depth, fusion failures and tired eyes.

The evidence here is stronger than for most claims in this area, because the manipulations are controlled and the outcomes are perceptual. Its limit is that the corrective display was a laboratory instrument, not a product. The prediction to carry forward is that any stereo display ignoring focus cues will be less comfortable and less accurate than its specification suggests.

Comfort is a design constraint, and locomotion is where it bites

Sickness is the failure mode that ends a session, and the evidence about it is now good enough to design against. Saredakis and colleagues (2020) reviewed fifty five studies of sickness after head-mounted display use, covering just over three thousand participants. The pooled symptom score across all content was 28.00 on the Simulator Sickness Questionnaire, a scale whose recommended cut-off sits below that figure. Gaming content scored highest, at 34.26. Content type, visual stimulation, locomotion style and exposure length all moved the score.

Two findings are directly actionable. The review reports that physically walking reduces sickness compared with controller based navigation. This argues for room-scale walking where space allows. Exposure time also matters. One rollercoaster study included in the review found that about two thirds of participants could not complete a fourteen minute exposure. The design responses follow: keep sessions short, prefer locomotion that matches the body, avoid sustained acceleration, and offer a seated alternative.

The honest limit is the measure itself. The review notes that almost all of the evidence rests on self-report questionnaires, and that very few studies report objective physiological measures. A score of 28 is a statement about what people said. It will also miss the participants who dropped out before they answered.

Three-dimensional input carries a fatigue bill

Direct input is not automatically better than indirect input, and the comparison has been measured. Forlines, Wigdor, Shen and Balakrishnan (2007) compared direct touch with a mouse on a tabletop display in two experiments. Direct touch won for bimanual tasks, where two hands work at once. For a single user doing single point tasks, the mouse was often the better instrument. It needs less reach and supports more precise pointing.

Mid-air input has the same problem in a sharper form. Hincapie-Ramos and colleagues (2014) name the gorilla arm effect and give it a number. Their metric, Consumed Endurance, is the ratio of interaction time to the time the shoulder can hold that contraction before it needs rest. They validated it against the Borg CR10 rating of perceived exertion and used it to compare designs. Holding the arm bent, on a plane midway between shoulder and waist, consumed the least endurance. Dwell selection consumed less than a press.

The consequence is a rule for any spatial interface. Fatigue is a function of posture, reach and duration, and it can be predicted before anyone touches the system. An interaction that feels natural for thirty seconds can be unusable for ten minutes. The design test is therefore the long session rather than the demonstration.

Tangible and embodied interaction is a stance, not a widget

Ishii and Ullmer (1997) opened a line of work by arguing that the graphical interface had thrown away something valuable. They proposed coupling digital information to graspable objects, turning architectural surfaces into interfaces, and using ambient media such as light, sound and airflow for background awareness. They built three systems to explore it, including a table that reads the position of physical objects and an ambient room.

Dourish (2001) generalised the argument into a philosophical position. Embodied interaction, in his account, treats meaning as arising from a person's engaged, skilled practice in a physical and social setting. His strongest claim is that tangibility and social interaction are two views of one phenomenon, because both depend on a shared physical context.

Here the evidence is thinner than the argument. Tangible Bits is a vision paper illustrated with prototypes, and it proposes research questions rather than reporting controlled comparisons. The contested question is whether giving an interface physical form produces a measurable benefit over a good graphical one. The honest answer is that this depends on the task and has often not been tested. A thirty year old vision can still be right, but treat it as a hypothesis rather than a settled result.

Physical computing: what a cardboard prototype can prove

O'Sullivan and Igoe (2004) supply the practical route from sensor to behaviour, moving from circuits through microcontrollers and programming to sensing and making movement. That is the skill set a physical prototype requires, and the technique matters because such a prototype can answer questions a screen prototype cannot pose. Fabrication matters too: a printed enclosure holds the sensor where the design assumed it would be.

Take a simple case: a desk lamp controlled by a light sensor and a microcontroller, built from cardboard and a board. The prototype can show that the chosen sensor distinguishes the states it must distinguish. It can show that the behaviour is legible to someone who did not build it. It can show that the physical form fits the hand and the desk. Those findings are cheap to obtain.

What the same prototype cannot show is equally important. It cannot show that the sensing survives dust and daylight over a year. It cannot show that the enclosure can be manufactured. It cannot show that the interaction is better than a switch. A prototype such as this proves a mechanism and a behaviour, not a benefit. The gap between those two claims is where physical computing projects most often overreach.

A controlled study can destroy the thing it measures

Spatial interaction is unusually vulnerable to its own measurement. Presence depends on a person responding to a situation as if it were real, and the apparatus of a careful experiment works against that. Questionnaires between trials, interruptions, a researcher in the room and a fixed task script all pull attention back to the laboratory. Slater and Sanchez-Vives (2016) make the point indirectly. They note that people respond with reflex and physiology before they reason about the situation. The interesting response therefore happens in the first moments, before any instrument is deployed.

The practical position is to measure what the setting allows and to say what it cost. Behavioural and physiological measures preserve more of the phenomenon than self-report, and they are harder to run. Where a controlled comparison is possible, as it is for menu layout or pointing technique, take it, because those results travel. Where it is not, use observation and report the loss. The next stage of this programme takes interaction off the desk and into the field, where context, sensing and deployment replace the laboratory, and where these compromises become the normal conditions of the work.