For augmented reality the binding constraints are registration and occlusion, not the overlay

Augmented reality draws onto a live view of the world, and the drawing should sit where the world says it should. The reality-virtuality continuum, the field's founding vocabulary, described the mixtures between real and virtual, but it does not predict where a system will work. This eighth of ten units argues that the constraints which decide use are registration and occlusion, not the brightness of the overlay: registration aligns a virtual object with the real scene, and occlusion is its correct hiding by the other.

The continuum gave the field its vocabulary

Milgram and Kishino (1994) placed augmented reality on a virtuality continuum from the completely real to the completely virtual, identified six classes of hybrid display, and proposed three dimensions: extent of world knowledge, reproduction fidelity and extent of presence metaphor. The contribution replaced a binary with a description, and its terms are still in use.

Azuma (1997) gave the definition the field still tests. A system is augmented reality when it combines real and virtual, is interactive in real time, and is registered in three dimensions. The third condition does the work, because it excludes a flat overlay on live video and it excludes film.

Bajura, Fuchs and Ohbuchi (1992) built one of the earliest working examples, a video see-through head-mounted display. Its camera image was composited with live ultrasound slices reconstructed to the observer's viewpoint. The slices then appeared fixed in three dimensions inside a pregnant woman as the observer walked around her. Two magnetic trackers, one on the headset and one on the probe, were related by a calibration procedure, and the first human experiment ran in January 1992. What decided whether the system was usable was the registration error between those systems, not the richness of the overlay.

The definition writes registration in, and registration is binding

Registration is the alignment of a virtual object with the real scene it is drawn onto. Its origin is Sutherland's (1968) argument for the head-mounted display. Azuma (1997) names registration and sensing errors as two of the biggest problems, sensing error being the error in knowing where the head and camera are. Marchand, Uchiyama and Spindler (2016) survey the pose estimation registration depends on and note that a tracker accurate in a prepared room may be brittle on a factory floor. Kress and Cummings (2017) describe the HoloLens headset, whose inside-out sensor fusion performs the tracking and mapping itself. Across four decades the display improves and registration remains the difficulty.

Occlusion is the second binding constraint

A correctly placed virtual object can still be wrong: it must be hidden by nearer real objects and must hide them in turn. Wloka and Anderson (1995) put the problem plainly, that overlaying computer-generated visuals on real imagery fails when data are shown in three dimensions because occlusion between the real and generated objects is not addressed. Azuma (1997) folds the same issue into the trade-off between optical and video blending. A video see-through system can resolve occlusion in the composited image where a conventional optical one cannot, though Han, Lee and Park (2025) now build occlusion optics into a holographic optical see-through display, masking part of the real world so translucent images read as opaque.

Depth perception in a see-through display is the measured form of the problem. Swan and colleagues (2006) built a perceptual matching task to measure egocentric depth judgements in optical see-through augmented reality, at five to forty five metres. They varied field of view and an x-ray vision condition, asking a user to perceive a graphic behind an opaque surface, and report systematic error in perceived depth relative to the real scene. A system that places an annotation behind a machine part is therefore asking for exactly the judgement the display distorts.

A bright overlay does not decide whether the system works

The optical engineering is real, since Kress and Cummings (2017) describe field of view, image quality, eye box and form factor as mutually limiting in the HoloLens architecture. Brightness and field of view are genuine design variables. The claim is narrower: once a display is good enough to be seen, what decides whether an application works is the alignment of the drawn object and the handling of occlusion.

Baird and Barfield (1999) had fifteen participants assemble a computer motherboard using a paper manual, computer-aided instruction, an opaque augmented reality display and a see-through one. The augmented reality conditions beat the manual and the computer-aided instruction, and the see-through display was fastest. Participants made fewer errors, though usability problems remained. The gain came from placing the instruction in the position of the work, an alignment benefit that arrived with a cost the overlay did not remove.

Field studies isolate alignment as the active ingredient

Henderson and Feiner (2011) built an augmented reality application for military mechanics maintaining equipment inside an armoured vehicle turret and ran a within-subject study in which mechanics completed eighteen common tasks with a tracked head-worn display, the same display without tracking, and a fixed flat panel. The tracked condition reduced time and errors on some tasks, and the paper sets out where the benefit did not appear. The untracked display keeps the overlay and loses the registration, isolating alignment as the active ingredient.

A sceptical audit sits beside it: Fite-Georgel (2011) asked how much industrial augmented reality had left the laboratory and found sustained deployment rare despite progress in display and tracking. Howard and Davis (2023) meta-analysed more than two hundred and fifty studies of augmented reality programmes for education and training, finding a learning advantage of about three fifths of a standard deviation but significantly smaller effects for head-mounted displays than for smartphones and tablets. The larger, cheaper display won, which sits badly with any account that puts the head-mounted overlay at the centre of the benefit. It is also the device a student already owns, on a metered or shared connection.

The contested question: registration and occlusion, definition, or deployment?

The literature divides over what limits a system in use. On the technical position, the constraint is registration and occlusion, and it is an engineering problem that is being reduced. Its holders are Azuma (1997), for whom registration and sensing errors are the principal obstacles. Wloka and Anderson (1995) hold that three-dimensional occlusion is unresolved by construction in an overlay. Marchand and colleagues (2016) catalogue the tracking methods and their trade-offs, and Swan and colleagues (2006) measure the residual depth error. On this reading a system fails because the tracker or the depth handling is not yet good enough.

On the definitional position, the constraint is the vocabulary. Speicher, Hall and Nebeling (2019) interviewed ten experts and surveyed sixty eight papers, finding no agreed definition of mixed reality but six competing notions. Skarbez, Smith and Whitton (2021) found the continuum discontinuous and broader than current practice, so mixed reality now encompasses conventional virtual reality.

On the deployment position, the constraint is neither: the technology has been good enough for some time. Its holder is Fite-Georgel (2011), whose survey found deployment rare after exactly the tracking and display progress the technical position predicted would help, and Howard and Davis (2023) supply the quantitative version: the benefit is real but modest, and larger on the cheaper hardware. On this reading what decides use is the fit between the task and the medium, together with the maintenance, calibration and workflow costs a laboratory demonstration never pays.

What would settle the dispute is a study that varies the variable the technical position names while holding the others fixed. It would measure registration error and occlusion handling in one system, vary brightness and field of view, and report the task outcome. If brightness moves performance once alignment is controlled, the technical position is too narrow. A second study would test a taxonomy prospectively, predicting which tasks a system will support and checking the prediction in the field. No study in this unit's reading list runs either design.

The unit's own thesis is the first position, and its strongest counter-argument is the deployment one: Howard and Davis (2023) pool more than two hundred and fifty studies and find head-mounted displays produce smaller learning effects than smartphones and tablets, so the fidelity that registration and occlusion deliver does not decide whether an application works. The evidence on each side is uneven: the technical position rests on a 1992 prototype, a definitions paper and small depth studies, the deployment position on a survey and a large meta-analysis. What a designer should do: measure registration and depth error in your own setting, choose the display by task fit rather than fidelity, and budget for calibration and maintenance.

Where this is heading

The direction of travel is towards taxonomies that predict rather than describe. Skarbez, Smith and Whitton (2021) rework the continuum to account for the role of the user. Speicher, Hall and Nebeling (2019) replace the single axis with seven dimensions. The useful contribution now is unlikely to be a brighter display, but a study that measures registration and occlusion, varies them deliberately, and reports which property actually decides use.

Sources

• Sutherland, I. E. (1968). A head-mounted three dimensional display. AFIPS Fall Joint Computer Conference. doi:10.1145/1476589.1476686 Described the first head-mounted three-dimensional display and argued that the image must change exactly as a real object's would for the same head movement, more important to depth than stereo. Registration is written into the founding design.
• Milgram, P. and Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems. url:https://web.archive.org/web/2019/https://search.ieice.org/bin/summary.php?id=e77-d_12_1321 Set out the virtuality continuum from completely real to completely virtual, identified six classes of hybrid display, and proposed dimensions of world knowledge, reproduction fidelity and presence metaphor. It classifies what the eye receives, not where a system will work.
• Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments. doi:10.1162/pres.1997.6.4.355 Surveyed augmented reality and gave the definition the field still uses: combine real and virtual, be interactive in real time, be registered in three dimensions. It names registration and sensing errors as two of the biggest problems. The definition already contains the constraint the unit treats as binding.
• Wloka, M. M. and Anderson, B. G. (1995). Resolving occlusion in augmented reality. Symposium on Interactive 3D Graphics. doi:10.1145/199404.199405 Argued that overlaying computer-generated visuals on real imagery fails when data are shown in three dimensions, because occlusion between the real and the generated objects is not addressed. Occlusion, not the overlay, is the second binding constraint.
• Bajura, M., Fuchs, H. and Ohbuchi, R. (1992). Merging virtual objects with the real world: seeing ultrasound imagery within the patient. ACM SIGGRAPH Computer Graphics. doi:10.1145/133994.134061 Composited video from a camera on a head-mounted display with live ultrasound slices transformed to the observer's viewpoint, so the slices appeared fixed in three dimensions inside a pregnant subject. Two magnetic trackers and a calibration procedure related headset and probe, and the first human experiment ran in January 1992.
• Baird, K. M. and Barfield, W. (1999). Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality. doi:10.1007/bf01421808 Had fifteen participants assemble a motherboard with a paper manual, computer-aided instruction, an opaque display and a see-through display. The augmented conditions were more effective and the see-through display fastest, with fewer errors but remaining usability problems.
• Swan, J. E. et al. (2006). A perceptual matching technique for depth judgments in optical, see-through augmented reality. IEEE Virtual Reality Conference. doi:10.1109/vr.2006.13 Built a perceptual matching task to measure egocentric depth judgements in optical see-through augmented reality at five to forty five metres, varying field of view and an x-ray vision condition. They report systematic error in perceived depth relative to the real scene.
• Henderson, S. and Feiner, S. (2011). Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Transactions on Visualization and Computer Graphics. doi:10.1109/tvcg.2010.245 Ran a within-subject field study in which military mechanics completed eighteen tasks with a tracked head-worn display, the same display without tracking, and a flat panel. The tracked condition reduced time and errors on some tasks.
• Fite-Georgel, P. (2011). Is there a reality in industrial augmented reality? IEEE International Symposium on Mixed and Augmented Reality. doi:10.1109/ismar.2011.6092387 Asked how much industrial augmented reality had left the laboratory, and found sustained deployment rare despite progress in display, rendering and tracking.
• Marchand, E., Uchiyama, H. and Spindler, F. (2016). Pose estimation for augmented reality: a hands-on survey. IEEE Transactions on Visualization and Computer Graphics. doi:10.1109/tvcg.2015.2513408 Surveyed the vision-based pose estimation that aligns virtual objects with an image sequence, and reported the trade-offs between marker-based and markerless methods in robustness, accuracy and setup cost.
• Kress, B. C. and Cummings, W. J. (2017). Optical architecture of HoloLens mixed reality headset. SPIE Proceedings. doi:10.1117/12.2270017 Set out the display requirements for the HoloLens headset and the optical hardware chosen, including untethered inside-out sensor fusion for head tracking and mapping, and report field of view, eye box and form factor as mutually limiting.
• Speicher, M., Hall, B. D. and Nebeling, M. (2019). What is mixed reality? CHI Conference on Human Factors in Computing Systems. doi:10.1145/3290605.3300767 Interviewed ten experts and surveyed sixty eight papers, finding no agreed definition of mixed reality but six competing notions, and proposed a seven-dimension framework.
• Skarbez, R., Smith, M. and Whitton, M. C. (2021). Revisiting Milgram and Kishino's reality-virtuality continuum. Frontiers in Virtual Reality. doi:10.3389/frvir.2021.647997 Reopened Milgram and Kishino's continuum and argued that it is discontinuous and that mixed reality is broader than the original account, encompassing conventional virtual reality.
• Howard, M. C. and Davis, M. M. (2023). A meta-analysis of augmented reality programs for education and training. Virtual Reality. doi:10.1007/s10055-023-00844-6 Meta-analysed more than two hundred and fifty studies and found an average learning advantage of about three fifths of a standard deviation, with head-mounted displays producing significantly smaller effects than smartphones and tablets.
• Han, W., Lee, C. and Park, J. (2025). Enhancing realism in holographic augmented reality displays through occlusion handling. Laser and Photonics Reviews. doi:10.1002/lpor.202501052 Built an occlusion-capable holographic optical see-through display: a folded four-f system with a digital micromirror device masks part of the real world and doubles as an active Fourier filter. It turns translucent holographic images perceptibly opaque, showing the occlusion limit is an engineering problem even for optical see-through.