The display pipeline sets the ceiling on immersion

A head-mounted display is bought on its specification, and the specification leads with resolution. That number is the least informative on the sheet. What a system can convincingly present is settled earlier, by how accurately it knows where the head is, how much of the visual field it covers, and whether the renderer can update that field fast enough to follow a moving head. Azuma (1993) set out the tracking requirements for augmented reality, and the display panel appears nowhere in his arithmetic. Immersion has a ceiling set by three coupled budgets. Resolution is therefore a demand on those budgets rather than a setting of them. The thresholds that matter belong to a task and a motion profile, not to a data sheet. This third of ten units leaves the person for the display that sets the ceiling.

Tracking is the first constraint, and the eye sets its tolerance

Azuma (1993) separates the virtual case from the augmented one, and the separation is why tracking is hard. In a fully virtual environment approximate head position is enough, because the visual sense overrides conflicting vestibular and proprioceptive signals. In augmented reality the virtual objects supplement the real world, and even tiny registration errors are detected at once. A tracker must therefore be accurate to a small fraction of a degree in orientation and a few millimetres in position. Azuma's demonstration: an angular error of 1.5 degrees in head orientation moves a virtual cup two metres away by about 52 mm. Holding that cup within 1 to 2 mm leaves no room for a larger positional error.

Latency is a pipeline total, not a component specification

Azuma's (1993) second requirement falls on the combined latency of tracker and graphics engine, the delay from the moment the tracker takes its measurements to the moment the matching images appear. That definition is end to end, which is why a quoted refresh rate tells you little. Steed (2008) makes the total measurable with an ordinary video camera. He films a tracker on a pendulum and an image driven by the tracker positions. The phase difference between sine curves fitted to the two motions resolves latency changes smaller than the camera frame rate. The figure a team needs is a measurement of its own pipeline, not a component specification.

The threshold is a number, and it is not one number

The quoted advice is a single latency budget, so it is worth asking what the measurements support. Mania, Adelstein, Ellis and Hill (2004) measured latency discrimination in a head-tracked stereoscopic display, across environments from simple geometry to a radiosity-rendered pair of rooms. They concluded that the just noticeable difference for trained observers averages about 15 ms or less, independent of scene complexity. Ellis, Mania, Adelstein and Hill (2004) tested three environments with 13 subjects: one containing only a foreground object, one only a background, and one both. They reported that perceptual stability requires latencies below 16 ms.

Azuma (1993) had already made the requirement a function of how fast the user turns. Many systems of his time carried combined latency over 100 ms, and at a moderate 50 degrees per second that latency produces 5 degrees of angular error. At 300 degrees per second, holding error below 0.5 degrees requires latency under 2 ms. Jerald and Whitton (2009) turn that arithmetic into a tested model relating latency, head motion, scene motion and perceptual thresholds, measured with six subjects.

Field of view buys presence and sells comfort

Field of view is the second budget, and the clearest case of a specification that is not a free upgrade. Lin, Duh, Parker, Abi-Rached and Furness (2002) collected data from 10 subjects at four fields of view, 60, 100, 140 and 180 degrees. Presence and sickness both rose as the field widened and approached asymptotes beyond 140 degrees, and presence correlated positively with sickness. Bowman and McMahan (2007) take that position: full immersion is not always necessary, and the question is how much a task requires. A wider field buys presence with sickness on the same purchase, and the shading budget below pays for it again.

The shading budget limits the others, and foveation adds a budget of its own

The third budget limits the other two, and it is a shading budget rather than a pixel count. Patney and colleagues (2016) designed a foveated renderer that shows high detail at the fixation point and progressively less in the periphery, so frame rate and resolution outpace conventional real-time renderers. Filtering peripheral regions reduces contrast and induces tunnel vision, but with a contrast enhancement subjects tolerated up to twice the blur radius before detecting a difference from a non-foveated ground truth. Their system reduces the number of shades by up to 70 per cent and coarsens shading up to 30 degrees closer to the fovea than the method compared.

Albert, Patney, Luebke and Kim (2017) measured how much latency foveation tolerates. Added eye-tracking latency of 80 to 150 ms significantly reduced the acceptable amount of foveation, while 20 to 40 ms did not. That suggests a total system latency of 50 to 70 ms could be tolerated.

Optics move the ceiling, and comfort moves it back

Maimone, Georgiou and Kollin (2017) present near-eye designs using phase-only holographic projection, with per-pixel focal control and holograms computed in real time at 90 Hz or more. They use the correction to enable compact, eyeglasses-like displays with fields of view of 80 degrees, and state plainly that combining every feature in a single device remains a challenge. Koulieris and colleagues (2019) survey the same territory and frame the outstanding challenges as ergonomics, comfort, visual quality and natural interaction. On their reading, comfort limits whether the display is worn at all.

The multi-plane route pays in the same currency. Akeley, Watt, Girshick and Banks (2004) built the prototype, stacking beam splitters to divide one liquid crystal panel into three sub-panels and presenting three focal planes 0.67 diopters apart. Zhan and colleagues (2020) report its panel measured 47.8 by 29.9 centimetres, and judge it a proof of concept rather than a product, because the volume it occupies grows further if a wider field of view is wanted.

The contested question: which constraint binds, and against what motion?

The first disagreement is whether a single latency threshold exists. Mania and colleagues (2004) and Ellis and colleagues (2004) report a stable figure near 15 to 16 ms for head tracking, and Mania and colleagues found it independent of scene complexity; that side rests on controlled psychophysics with ten to thirteen trained observers. Azuma (1993), Jerald and Whitton (2009) and Albert and colleagues (2017) hold the threshold to be conditional, but their side rests on arithmetic and a model fitted to six subjects, so neither carries a large-sample test. It falls as head velocity rises and differs by subsystem: the foveation budget tolerates a total system latency of 50 to 70 ms (Albert, Patney, Luebke and Kim, 2017), against a head-tracking requirement an order of magnitude smaller. Evidence that would settle it: one shared psychophysical procedure, thresholds reported with the head-motion profile and the subsystem charged, and end-to-end latency measured on the assembled system (Steed, 2008).

The second disagreement is which constraint binds first. On the tracking position, registration error is the ceiling and everything else is downstream (Azuma, 1993). On the rendering position, the shading budget is what makes a wide field and a high frame rate unaffordable together (Patney et al., 2016). On the ergonomic position, comfort limits adoption, so a display that wins on optics can lose on wearability (Koulieris et al., 2019). The thesis of this unit is the first of these positions, and the ergonomic position is its strongest counter-argument: if the device is not worn, no rendering budget is felt at all. A second counter is field of view, which raises presence and simulator sickness together (Lin et al., 2002), so part of the ceiling a designer meets is perceptual rather than a hardware limit.

What a designer should do while the dispute is unresolved. Quote any latency threshold with the head-motion profile and the subsystem it charges, and measure end-to-end latency on the assembled system rather than trusting a component figure (Steed, 2008). Fix one budget and vary another while holding the task constant, which turns the dispute into a testable comparison (Bowman and McMahan, 2007). Treat comfort as a ceiling alongside optics throughout, and fit as part of it, since a headset that will not sit over glasses or on a given face is not worn at all.

Where this is heading

Ishihara and colleagues (2023) build a perspective-corrected video see-through head-mounted display with edge-preserving occlusion and low latency. It reconstructs captured images to match the user's view. Accurate depth maps are computationally difficult and lengthen latency, so they generate maps rapidly by targeting edge smoothness and disocclusion rather than full accuracy. The advance is not the removal of the trade, but a more informed choice about which error to accept.

Sources

• Azuma, R. (1993). Tracking requirements for augmented reality. Communications of the ACM. doi:10.1145/159544.159581 A requirements paper setting the accuracy and latency augmented reality demands of a tracker. A head-orientation error of 1.5 degrees moves a cup two metres away by about 52 mm, and 100 ms of latency yields 5 degrees of error at 50 degrees per second.
• Lin, J. W., Duh, H., Parker, D., Abi-Rached, H. and Furness, T. (2002). Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. Proceedings IEEE Virtual Reality 2002. doi:10.1109/vr.2002.996519 Measured presence and simulator sickness in 10 subjects at four fields of view, from 60 to 180 degrees. Presence and sickness both rose with the field and levelled off beyond 140 degrees, so a wider field trades comfort against presence.
• Mania, K., Adelstein, B. D., Ellis, S. R. and Hill, M. I. (2004). Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity. Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization. doi:10.1145/1012551.1012559 Measured latency discrimination across scenes from simple geometry to a radiosity-rendered room. The just noticeable difference for trained observers averaged about 15 ms or less, independent of scene complexity.
• Ellis, S. R., Mania, K., Adelstein, B. D. and Hill, M. I. (2004). Generalizeability of latency detection in a variety of virtual environments. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. doi:10.1177/154193120404802306 Tested latency sensitivity in three environments with 13 subjects, using only a foreground object, only a background, and both. They report that perceptual stability requires latencies below 16 ms, and attribute a disagreement with an earlier study to the type of psychophysical judgement rather than the scene.
• Bowman, D. A. and McMahan, R. P. (2007). Virtual reality: how much immersion is enough? Computer. doi:10.1109/mc.2007.257 Argues that full immersion is not always necessary and that components of immersion contribute differently to what a system delivers. It asks how much immersion a task requires.
• Steed, A. (2008). A simple method for estimating the latency of interactive, real-time graphics simulations. Proceedings of the ACM Symposium on Virtual Reality Software and Technology. doi:10.1145/1450579.1450606 Measures end-to-end latency with an ordinary video camera by reading the phase difference between sine curves fitted to a pendulum and to a simulated image driven by the tracker. The method resolves latency changes smaller than the frame rate.
• Jerald, J. and Whitton, M. (2009). Relating scene-motion thresholds to latency thresholds for head-mounted displays. IEEE Virtual Reality Conference. doi:10.1109/vr.2009.4811025 Derive a model relating latency, head motion, scene motion and perceptual thresholds, and measure thresholds with six subjects. Their latency thresholds fall in the same range as earlier work and move as head velocity changes.
• Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D. and Lefohn, A. (2016). Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics. doi:10.1145/2980179.2980246 Built a renderer showing high detail at fixation and less in the periphery. With a contrast enhancement subjects tolerated up to twice the blur radius, and the system cuts shading work by up to 70 per cent.
• Albert, R., Patney, A., Luebke, D. and Kim, J. (2017). Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception. doi:10.1145/3127589 Evaluated the detectability of foveation artefacts under added latency for three techniques and three foveal radii. Latency of 80 to 150 ms on the eye-tracking signal cut the acceptable foveation while 20 to 40 ms did not.
• Zhan, T., Xiong, J., Zou, J. and Wu, S. T. (2020). Multifocal displays: review and prospect. PhotoniX. doi:10.1186/s43074-020-00010-0 Reviews multifocal displays, which present several focal planes so the stimulus to accommodation varies with depicted depth. It judges Akeley's three-plane prototype too large in volume for a product.
• Akeley, K., Watt, S. J., Girshick, A. R. and Banks, M. S. (2004). A stereo display prototype with multiple focal distances. ACM SIGGRAPH 2004 Papers. doi:10.1145/1186562.1015804 Built a prototype from two volumetric displays that generate near-correct focus cues without tracking eye position, demonstrated with three image planes. It is the primary design behind the multi-plane route, offered as a proof of concept rather than a product.
• Maimone, A., Georgiou, A. and Kollin, J. S. (2017). Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics. doi:10.1145/3072959.3073624 Presents near-eye designs using phase-only holographic projection, with per-pixel focal control and holograms computed in real time at 90 Hz or more. The correction enables compact, eyeglasses-like displays of 80 degrees, though combining every feature remains a challenge.
• Koulieris, G. A., Aksit, K., Stengel, M., Mantiuk, R. K., Mania, K. and Richardt, C. (2019). Near-eye display and tracking technologies for virtual and augmented reality. Computer Graphics Forum. doi:10.1111/cgf.13654 Surveys the perception, display engineering and tracking behind near-eye displays. It frames the outstanding challenges as ergonomics, comfort, visual quality and natural interaction.
• Ishihara, A. and colleagues (2023). Integrating both parallax and latency compensation into video see-through head-mounted display. IEEE Transactions on Visualization and Computer Graphics. doi:10.1109/tvcg.2023.3247460 Built a perspective-corrected video see-through display that reconstructs captured images, occludes virtual objects with nearer real objects, and reprojects both scenes to keep pace with head motion. Because accurate depth maps are expensive and lengthen latency, they generate rapid maps targeting edge smoothness and disocclusion.