A test measures the interaction and changes it: moderation, think-aloud and setting are part of the finding

A product team runs a moderated laboratory study of two prototypes. Ten participants think aloud, an observer notes where they hesitate, and the report carries times, a failure rate and a list of difficulties. The numbers read as properties of the two interfaces. This ninth of ten units argues that they are properties of the session that produced them.

Every usability test intervenes in the behaviour it records. The facilitator's presence, the instruction to talk, the room or the street, and the analyst who decides which hesitation counts as a finding all shape what reaches the record, and a participant who cannot speak, or who speaks a second language, never reaches it at all. The interference is not a defect that better practice removes. It is the price of watching a person use an artefact: a timing or a failure rate means only what the session as run allows.

The session manufactures the behaviour it counts

Duh, Tan and Chen (2006) ran the same mobile phone evaluation in a laboratory and in real life. The settings differed significantly in the frequency and severity of the problems found, in user behaviour and in subjective responses. The setting did not hold the interaction still. Higher fidelity is not automatically more informative: Marcilly and colleagues (2024) compared two levels of ecological validity in usability testing of a health information technology and found the same types of use error in both, differing modestly in count and not consistently by severity.

The laboratory is the field's default. Kjeldskov and Graham (2003) classified 102 mobile HCI publications, and of the 41 per cent that involved evaluation, 71 per cent were laboratory experiments and 19 per cent field experiments.

Cost explains the choice. Kjeldskov, Skov and Stage (2004) ran conventional video analysis and Instant Data Analysis independently on the same hospital software, and the faster method identified 85 per cent of the critical problems in 10 per cent of the time. The analysis cost decides the setting, the participants and the number of sessions a project can afford.

Think-aloud is a treatment, and the theory licenses a narrower method than practice uses

Ericsson and Simon (1980) argued that verbal reports are data whose validity depends on the mechanisms and instructions that generate them. Concurrent reports of information currently attended to can be valid. Probing and retrospective requests are more suspect. That is the condition a protocol must meet to count as evidence.

Boren and Ramey (2000) set the theory against the practice that cites it, and found the practice diverging from the model. They propose speech genre instead. An evaluator who acknowledges, encourages and asks for clarification is running a different method from the one the theory licenses.

Hertzum, Hansen and Andersen (2009) measured what that does. Against silent performance, classic thinking aloud had little or no effect apart from prolonging tasks. Relaxed thinking aloud lengthened tasks, moved more time into distributed visual behaviour, increased navigation commands, and raised reported mental workload. A duration recorded under the relaxed regime is the duration of doing the task while explaining it.

The comparison evidence is weaker than the reactivity claim

Van den Haak, de Jong and Schellens (2004) compared concurrent think-aloud, retrospective think-aloud and constructive interaction on an online library catalogue, across the problems found, their relevance, task performance and participant experience. The methods proved very much comparable on the problems surfaced. They differed mainly in how problems were detected.

Krahmer and Ummelen (2004) compared a rigid Ericsson and Simon procedure with a Boren and Ramey based alternative. Thinking aloud was unaffected, and more tasks were completed under the second protocol.

Alhadreti and Mayhew (2018) compared concurrent, retrospective and hybrid think-aloud on a library website: concurrent detected more problems than retrospective, showed no reactivity, and cost far less evaluator time.

Together they support a narrower thesis: the protocol is part of a timing and of what participants do, far less clearly part of the problem list.

The analyst is part of the apparatus

Hornbaek and Frokjaer (2008) had 50 novice evaluators test usability. They compared the resulting problems, individually and in teams, with the problems ten independent HCI experts found in the same material. Agreement was about 40 per cent, and team matching improved it. All three groups produced effects of similar size while disagreeing about which problems were similar. Matching, the decision that two reported difficulties are the same problem, is a key determinant of the effect, so a single correct matching cannot be assumed.

Nothing about the participants changed here. Two studies of the same interface can report different problems because the analysts classified differently, invisibly. Nielsen, Clemmensen and Yssing (2002) asked what researchers think they obtain when they ask people to think aloud. They answered that the method presupposes explicit statements of design and analytic framework.

What the test is for is a design decision

Greenberg and Buxton (2008) argue that practice treats evaluation as critical to every design process when it should be conditional. Done by rule rather than by thought, it can kill creative ideas in brainstorming, kill an inspired vision by exposing early design problems, and validate a prototype instead of critiquing its adoption. Early-stage and culturally sensitive systems need other methods. Cotton and Gresty (2005) reflected on think-aloud in an e-learning study. They name the guidance given to participants, observer influence and the complexity of analysis as the concerns that arose, and they judge the richness of the data to outweigh them.

Contested question: does the intervention make a test number meaningless?

The first position says the interference is fatal to behavioural measures. Relaxed thinking aloud changes task time, visual behaviour, navigation commands and workload (Hertzum, Hansen and Andersen, 2009). The setting changes the frequency and severity of the problems reported (Duh, Tan and Chen, 2006). A difference measured under one regime is therefore not a difference between the interfaces.

The second position says the interference is bounded. Van den Haak and colleagues (2004) found the protocols comparable on the problems surfaced. Krahmer and Ummelen (2004) found the navigation problems unchanged. Alhadreti and Mayhew (2018) report no reactivity alongside better detection and lower cost. If the problem list is the measure of interest, these studies bound the interference at close to zero.

The third position says the dispute is mis-specified. Hornbaek and Frokjaer (2008) show that agreement figures are computed by matching one analyst's problems to another's, and that the matching itself varies. A rate of about 40 per cent describes an interpretive act rather than a property of the interface.

The positions reconcile in part, though the reconciliation is an inference. The behavioural effects are concentrated in the relaxed regime that Boren and Ramey (2000) show practice uses. The comparison reporting no reactivity recommends a disciplined protocol closer to the original guidance. Reactivity may belong to the relaxed method rather than to thinking aloud as such. Neither paper crosses the other's factor, so nothing here tests that reading.

What would settle it is a design none of these studies runs. A study would observe the same participants and interface in silence, under a classic protocol and a relaxed one, in the laboratory and in the field, with measures fixed in advance and the problem inventory matched by a panel blind to condition. Until then, the honest report states the regime and setting beside the number.

The evidence is unevenly split. The reactivity case rests on one study, Hertzum, Hansen and Andersen (2009), which compared a silent, a classic and a relaxed protocol, while the bounding case rests on three comparisons, Van den Haak and colleagues (2004), Krahmer and Ummelen (2004) and Alhadreti and Mayhew (2018), each finding the problem list changed little. The strongest objection is that the unit's thesis proves too much, because those three comparisons treated sessions as comparable enough to compare, which the claim that every number is session-bound would forbid. A designer should report the regime and the setting beside every number while the dispute is open.

Where the argument is heading

The direction is to treat the session as part of the result. Doi (2021) makes the analysis procedure explicit and repeatable. Alhadreti and Mayhew (2018) put evaluator cost beside problem detection. Hornbaek and Frokjaer (2008) turn agreement into a description of an interpretive act. Kjeldskov, Skov and Stage (2004) attack the analysis cost that forces the setting choice. The gaps match the direction: the behavioural evidence dates from 2009, and the laboratory dominance of 2003 needs re-auditing for mobile and remote tools. The claim to make stays narrow: here is the behaviour, here is the regime and setting that produced it, and here is what only a second session, run differently, could show.

The sources this unit rests on

• Alhadreti, O. and Mayhew, P. (2018). Rethinking thinking aloud: a comparison of three think-aloud protocols. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. doi:10.1145/3173574.3173618 Compared concurrent, retrospective and hybrid think-aloud on a library website across task performance, problems found and method cost. Concurrent found more problems than retrospective and cost less evaluator time, with no reactivity observed.
• Boren, T. and Ramey, J. (2000). Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication. doi:10.1109/47.867942 Compares the thinking aloud described in the usability literature with the Ericsson and Simon model it cites, and proposes speech genre instead. Practice diverges from the model, so the relaxed form in common use is not the method whose validity was argued for.
• Cotton, D. and Gresty, K. (2005). Reflecting on the think-aloud method for evaluating e-learning. British Journal of Educational Technology. doi:10.1111/j.1467-8535.2005.00521.x Reflects on their own use of think-aloud to study how students use electronic resources. The concerns that arose were the guidance given to participants, observer influence and the complexity of analysis, which they judge the richness of the data to outweigh.
• Doi, T. (2021). Usability textual data analysis: a formulaic coding think-aloud protocol method for usability evaluation. Applied Sciences. doi:10.3390/app11157047 Proposes a formulaic procedure for analysing think-aloud text, using step coding and 70 design items, applied to a usability case study. The author reports that the explicit procedure made problem extraction less ambiguous and less dependent on analyst skill.
• Duh, H. B. L., Tan, G. C. B. and Chen, V. H. H. (2006). Usability evaluation for mobile device: a comparison of laboratory and field tests. Proceedings of the 8th conference on Human-Computer Interaction with Mobile Devices and Services. doi:10.1145/1152215.1152254 Ran the same mobile phone evaluation in the laboratory and in real life. The settings differed significantly in the frequency and severity of the problems found, in user behaviour and in subjective responses.
• Ericsson, K. A. and Simon, H. A. (1980). Verbal reports as data. Psychological Review. doi:10.1037/0033-295x.87.3.215 Sets out the theory under which verbal reports count as data, tying validity to the mechanisms and instructions that generate a report. Concurrent reports of attended information can be valid; requests for information never heeded force inference.
• Greenberg, S. and Buxton, B. (2008). Usability evaluation considered harmful (some of the time). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1357054.1357074 Argues that evaluation is treated as obligatory when it should be conditional: done by rule, it can kill ideas in brainstorming, kill a vision in early design, and validate a prototype instead of critiquing its adoption. Early-stage and culturally sensitive systems need other methods.
• Hertzum, M., Hansen, K. D. and Andersen, H. H. K. (2009). Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? Behaviour and Information Technology. doi:10.1080/01449290701773842 Compared silent performance with classic and relaxed thinking aloud. Classic thinking aloud had little effect apart from prolonging tasks, while the relaxed regime lengthened tasks, spread visual attention, increased navigation commands and raised mental workload.
• Hornbaek, K. and Frokjaer, E. (2008). A study of the evaluator effect in usability testing. Human-Computer Interaction. doi:10.1080/07370020802278205 Had 50 novice evaluators test usability and analysed the resulting problems individually and in teams alongside ten independent HCI experts. Agreement was about 40 per cent, and the matching of problems was a key determinant of the effect.
• Kjeldskov, J. and Graham, C. (2003). A review of Mobile HCI research methods. Lecture Notes in Computer Science. doi:10.1007/978-3-540-45233-1_23 Classified 102 mobile HCI publications by research method and purpose. Of the 41 per cent involving evaluation, 71 per cent used laboratory experiments, 19 per cent field experiments.
• Kjeldskov, J., Skov, M. B. and Stage, J. (2004). Instant data analysis: conducting usability evaluations in a day. Proceedings of the third Nordic conference on Human-Computer Interaction. doi:10.1145/1028014.1028050 Ran conventional video analysis and Instant Data Analysis independently on the same hospital software evaluation. Instant Data Analysis identified 85 per cent of the critical problems in 10 per cent of the time.
• Krahmer, E. and Ummelen, N. (2004). Thinking about thinking aloud: a comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication. doi:10.1109/tpc.2004.828205 Compared a rigid Ericsson and Simon procedure with a Boren and Ramey based alternative. Thinking aloud was unaffected and more tasks were completed under the second protocol, but the number of navigation problems detected did not differ.
• Marcilly, R., Monkman, H., Pelayo, S. and Lesselroth, B. J. (2024). Usability Evaluation Ecological Validity: Is More Always Better? Healthcare, 1417. doi:10.3390/healthcare12141417 Compared two levels of ecological validity in usability testing of a health information technology and found the same types of use error in both conditions, with only a modest difference in error count and no consistent pattern by severity.
• Nielsen, J., Clemmensen, T. and Yssing, C. (2002). Getting access to what goes on in people's heads? Reflections on the think-aloud technique. Proceedings of the second Nordic conference on Human-Computer Interaction. doi:10.1145/572020.572033 Surveys the names, uses and modifications of the classic think-aloud technique and asks what researchers think they obtain when they ask people to think aloud. It argues that the technique presupposes explicit statements of the design, procedure and analytic framework.
• Van den Haak, M. J., de Jong, M. D. T. and Schellens, P. J. (2004). Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Interacting with Computers. doi:10.1016/j.intcom.2004.07.007 Compared concurrent think-aloud, retrospective think-aloud and constructive interaction on an online library catalogue across problems found, task performance and participant experience. The methods proved very much comparable on the problems surfaced.