Situation awareness and the operator out of the loop: a measurable construct and its critiques

Endsley's three-level model turned situation awareness from a phrase into something a safety-critical system could be designed around and measured against. It defines awareness as the perception of elements in the environment, the comprehension of their meaning and the projection of their status in the near future (Endsley, 1995a). A measure that fixes its questions in advance measures the operator's fit to the analyst's model of the task. The automation the model was built to support removes the operator from the loop in which awareness is formed.

This third of ten units takes up the measurement the second left open, and asks whose model a measure of awareness encodes.

A construct built to be designed for

The theory paper names the three levels, separates awareness from decision making and from performance, and grounds the account in a descriptive view of how operators decide (Endsley, 1995a). An accurate picture can still yield a poor decision, and a poor picture a sound one. Awareness is therefore a thing to be measured rather than a proxy for outcome. Attention and working memory are the binding limits, and mental models relieve them. Each level names something a designer can support or a query can ask, and that translation decides in advance what will count as awareness when the analyst measures it.

Making an internal state a number

Endsley's companion paper reports two studies of a query-based technique (Endsley, 1995b). The technique interrupts a running simulation to ask the operator about the state of things. The studies found that the interruption did not disturb normal behaviour and that recall was not memory-limited, and the paper offers both as validity evidence. The technique freezes the task, and the analyst must write the query set before the operator answers. The query is spoken, so it also assumes an operator who can answer in the analyst's language, whether or not that is their first language. The validity evidence comes from aviation simulation. What the technique returns is the operator's agreement with a task model the experimenter built.

The query measures the analyst as well as the operator

Flach set out the sceptical case in the same year, using Underwood's distinction between a phenomenon description and a causal agent (Flach, 1995). As a description of the coupling between operator and environment it is useful. As an object inside the head offered as the cause of performance it risks circular reasoning. Dekker and Hollnagel generalise the charge, using automation-induced complacency as their guiding example (Dekker and Hollnagel, 2004). Folk models substitute one label for another rather than decomposing a construct into measurable specifics. They resist falsification and they are overgeneralised. The model decides what counts as essential performance, so a folk model produces a measurement of itself.

Wickens' appraisal is the balanced one: the construct is viable while still carrying controversy over measurement (Wickens, 2008).

Where awareness lives

Stanton and colleagues set out the alternative, starting from the interactions between agents, human and non-human, within subsystems (Stanton et al., 2006). Each agent holds its own awareness, compatible with but different from the others'. Sharing is not always desirable, because the agents have different purposes, so awareness becomes a collaborative process. Gorman, Cooke and Winner treat team awareness as an interaction-based phenomenon (Gorman et al., 2006). Team awareness still has to hold when a teammate reads the display through a screen reader rather than watching it. They outline a process-based measure that looks at what the team does, the coordinated awareness of situations by teams.

The operator out of the loop

The second half concerns the work as well. Bainbridge's argument from process control is its origin (Bainbridge, 1983). Automating industrial processes may expand rather than eliminate problems with the human operator, because the usual approach leaves the operator responsible for abnormal conditions. The residual role is the hard part, and it arrives after the routine that built the skill has gone to the machine.

Parasuraman and Riley (1997) separate use, misuse, disuse and abuse of automation. Misuse is over-reliance, and it shows up as failures to monitor and as decision biases. Workload, automation reliability and the salience of its state indicators all shape monitoring. Disuse is the opposite failure, the neglect of automation that works, and the paper attributes it most often to false alarms. The paper is an analytic review, so it names mechanisms without estimating their size.

Endsley and Kiris give it empirical form, automating a navigation task with an expert system and measuring awareness and takeover (Endsley and Kiris, 1995). Low awareness corresponded with out-of-the-loop decrements in decision time after the expert system failed. The level of operator control moderated the loss, and the authors judged a shift from active to passive processing the most likely cause. Proliferating modes, without support for the new cognitive demands, create new mode-related error forms (Sarter and Woods, 1995).

The construct reaches the ward

The test of a safety construct is what happens when it measures clinical work. Ghaderi and colleagues reviewed the measurement properties of situation awareness instruments for health care providers using the COSMIN methodology (Ghaderi et al., 2023). None presented all the measurement properties. Content validity and internal consistency were each reported in twelve studies. Cross-cultural validity and responsiveness were not investigated in any, so none has been shown to work for a clinician who reads the display in a second language. Evidence quality was not high in any instrument. None could be recommended, and the reviewers classified all of them as promising instruments in need of further psychometric assessment.

The contested question: construct or slogan, individual or system?

The first of the positions holds that situation awareness is valid and improvable, with a disputed but tractable measurement. Its evidence is the model's productiveness (Endsley, 1995a), Wickens' verdict that the construct is viable (Wickens, 2008), and fifteen instruments in active use (Ghaderi et al., 2023). Its test is an instrument with demonstrated predictive validity for a safety outcome in the setting where it is used.

The second holds that the instrument measures the analyst as much as the operator. Its evidence is the query technique, whose questions the analyst fixes before the operator answers (Endsley, 1995b). A construct offered as a causal agent invites circular reasoning (Flach, 1995). A folk model becomes the basis for its own measurement (Dekker and Hollnagel, 2004). Its test is discriminant validity: the measure is not a restatement of the task analysis used to write it.

The third holds that the unit of analysis is wrong. Its evidence has three parts. Awareness is a system property (Stanton et al., 2006), and team awareness is an interaction-based phenomenon (Gorman et al., 2006). Distributed approaches are the ones suited to collaborative work (Salmon et al., 2008). Its test is a head-to-head comparison, in which a team tries individual and system-level measures of the same task against the same outcome.

The fourth splits the field over automation. One side attributes the out-of-the-loop decrement to automation itself (Endsley and Kiris, 1995). The other attributes it to design. Sarter and Woods show designers proliferating modes without supporting the new cognitive demands (Sarter and Woods, 1995). Norman argues that the problem is not automation but its inappropriate design. At an intermediate level of intelligence, automation takes over control without providing the continual feedback that human operators give one another (Norman, 1990). What would settle this one is a comparison of two systems at the same level of automation with different feedback, because only then are level and feedback separated. None of those studies is reported here: the automation finding is real while its cause remains contested.

What to do while the dispute is open

The evidence on the two sides is unequal. That the construct is valid rests on a productive model and on instruments in use. That the measure returns the analyst rests on the query technique, whose questions are fixed before the operator answers, and on an argument from theory. The strongest objection to this unit's thesis is that the clinical review judges instruments, not the construct, so its finding may be a verdict on the scales rather than on the idea. While the dispute is open, a designer should say whose task model a probe encodes, measure the operator and the team, and treat agreement as an input rather than an outcome.

Where this is heading

The direction of travel is from one individual measure towards psychometrically evaluated and system-level instruments. Ghaderi and colleagues (2023) call for the development and psychometric evaluation of instruments for clinical settings, and their review is the benchmark any new instrument must beat. The load-bearing test on the automation side is Norman's (1990) reframing.

Sources

• Endsley, M. R. (1995a). Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors. doi:10.1518/001872095779049543 The defining paper: defines situation awareness across three levels, separates it from decision making and performance, and identifies attention and working memory as the binding limits with mental models as the relief. It is the model the unit interrogates.
• Endsley, M. R. (1995b). Measurement of Situation Awareness in Dynamic Systems. Human Factors. doi:10.1518/001872095779049499 Reviews the methods for measuring situation awareness and reports two studies of a query technique that interrupts a simulation to ask the operator about the state of things. The studies found the interruption did not disturb behaviour and recall was not memory-limited, so its pre-fixed query set is the unit's point of attack.
• Flach, J. M. (1995). Situation Awareness: Proceed with Caution. Human Factors. doi:10.1518/001872095779049480 Uses Underwood's distinction between a phenomenon description and a causal agent, arguing that situation awareness is valuable as a description of operator-environment coupling but risks circular reasoning as the cause of performance. It is the earliest statement of the measurement worry.
• Dekker, S. and Hollnagel, E. (2004). Human factors and folk models. Cognition, Technology and Work. doi:10.1007/s10111-003-0136-9 Argues that human factors is susceptible to folk models, using automation-induced complacency as the guiding example: such models substitute one label for another, resist falsification and are overgeneralised. It shows why a construct can end up measuring its own assumptions.
• Wickens, C. D. (2008). Situation Awareness: Review of Mica Endsley's 1995 Articles on Situation Awareness Theory and Measurement. Human Factors. doi:10.1518/001872008x288420 Reviews Endsley's two 1995 papers and the work they generated, finding the construct applied to training, error analysis, design, selection, teamwork and automation. Concludes it is viable and important while still carrying controversy over measurement.
• Salmon, P. M., Stanton, N. A., Walker, G. H., Baber, C., Jenkins, D. P., McMaster, R. and Young, M. S. (2008). What really is going on? Review of situation awareness models for individuals and teams. Theoretical Issues in Ergonomics Science. doi:10.1080/14639220701561775 Reviews and compares the prominent individual and team models, finding the construct remains predominantly individual and that team awareness is less developed. Concludes that systems-level distributed approaches suit collaborative environments.
• Stanton, N. A., Stewart, R., Harris, D., Houghton, R. J., Baber, C., McMaster, R., Salmon, P., Hoyle, G., Walker, G., Young, M. S., Linsell, M., Dymott, R. and Green, D. (2006). Distributed situation awareness in dynamic systems: theoretical development and application of an ergonomics methodology. Ergonomics. doi:10.1080/00140130600612762 Proposes foundations for a theory built on interactions between human and non-human agents, arguing each agent holds its own awareness, compatible with but different from others'. It is the basis for measuring awareness at system level.
• Gorman, J. C., Cooke, N. J. and Winner, J. L. (2006). Measuring team situation awareness in decentralized command and control environments. Ergonomics. doi:10.1080/00140130600612788 Re-examines team situation awareness beyond an individual knowledge-based construct, treating it as a team interaction-based phenomenon and outlining a process-based measure, coordinated awareness of situations by teams.
• Ghaderi, C., Esmaeili, R., Ebadi, A. and Amiri, M. R. (2023). Measuring situation awareness in health care providers: a systematic review of measurement properties using COSMIN methodology. Systematic Reviews. doi:10.1186/s13643-023-02220-6 Reviewed the measurement properties of situation awareness instruments for health care providers using COSMIN, across twenty-five studies and fifteen instruments. No instrument reported all properties and evidence quality was not high in any; content validity and internal consistency were each reported in twelve studies, and cross-cultural validity and responsiveness in none. The measurement problem is unresolved in the ward.
• Bainbridge, L. (1983). Ironies of automation. Automatica. doi:10.1016/0005-1098(83)90046-8 Argues that automating industrial processes may expand rather than eliminate problems with the human operator, and that the classic approach leaves the operator responsible for abnormal conditions. It is the origin of the argument that automation removes the routine that built the skill the residual role depends on.
• Endsley, M. R. and Kiris, E. O. (1995). The Out-of-the-Loop Performance Problem and Level of Control in Automation. Human Factors. doi:10.1518/001872095779064555 Automated a navigation task with an expert system and measured awareness and takeover. Found low awareness corresponded with out-of-the-loop decrements in decision time after the expert system failed, that the level of operator control moderated the loss of awareness, and that a shift from active to passive processing was the likely cause.
• Sarter, N. B. and Woods, D. D. (1995). How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control. Human Factors. doi:10.1518/001872095779049516 Draws on studies of pilot-automation interaction in commercial glass cockpits to extend mode error, showing that proliferating modes without supporting the new monitoring demands creates new mode-related error forms and failure paths. It is the unit's example of the operator losing the model of the automation rather than the automation failing.
• Norman, D. A. (1990). The problem with automation: inappropriate feedback and interaction, not over-automation. Philosophical Transactions of the Royal Society of London B. doi:10.1098/rstb.1990.0101 Argues the harm is not automation but its inappropriate design, at an intermediate level of intelligence that takes over control without providing the continual feedback human operators give one another. It is the unit's strongest counter to the claim that the operator is out of the loop by necessity.
• Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors. doi:10.1518/001872097778543886 Separates use, misuse, disuse and abuse of automation and gives each a mechanism. Misuse is over-reliance producing monitoring failures and decision biases, while disuse follows false alarms, because the base rate of the condition is left out of the alarm trade-off.