Emotion in interaction: affect, experience and the problem of measuring it

Two checkouts can take the same time while one leaves the user anxious and the other confident. A report carrying only completion time and errors cannot tell them apart. Emotion belongs to the measurement of interaction rather than being a decoration laid over it. The field still disagrees about what emotion is and how to measure it, so an affective claim must name both its instrument and its theory. The nine units before it traced the limits of the mind. This one asks how the interaction feels. The strongest counter is that measuring affect as information reproduces the model the affective turn set out to challenge, because emotion is made in interaction rather than detected inside a head (Boehner et al., 2007). Three models circulate in the psychology that interaction research borrows: a small set of kinds given by biology, a position on two continuous dimensions, and a construction from a bodily state and a concept.

Three models compete over what an emotion is

Ekman (1992) gives the first model its clearest statement. He argues that emotions evolved for their value in dealing with fundamental life tasks, each carrying a unique signal, a distinct physiology and a class of antecedent events. Those features are what make anger or fear a natural kind, a category with a boundary in nature rather than in language.

Russell (1980) scaled twenty-eight emotion-denoting adjectives in four ways and analysed the self-reports of 343 participants. He found the terms arranged on a circle in a two-dimensional bipolar space, with pleasure and displeasure on one axis and arousal and sleep on the other. An emotion is a position rather than a kind, and the distance between two states is continuous.

Barrett (2006) finds the evidence accumulating against boundaries carved in nature. Lindquist and colleagues (2012) meta-analysed the neuroimaging literature. They report little evidence that discrete categories localise consistently and specifically to distinct brain regions. They find instead that interacting regions, engaged in ordinary emotional and non-emotional operations, are active across many categories. Barrett (2017) states the positive alternative as a brain-based computational account, the theory of constructed emotion.

Work the meta-analysis. Lindquist and colleagues (2012) pooled the neuroimaging studies that had tested discrete emotion categories and asked whether each maps to a consistent, specific region. The mapping did not hold: regions engaged by one category were also engaged by others and by ordinary non-emotional operations. The case is the strongest single piece of evidence against natural kinds. It does not show that no neural differences exist, since a meta-analysis averages across studies that differ in method.

Every instrument carries the theory it was built from

Russell's structure came from scaling words by self-report, so the measure assumes a reader of English adjectives: the two dimensions are an output of the instrument rather than an independent discovery (Russell, 1980).

Bradley and Lang (1994) turn the dimensional model into a portable instrument. Their Self-Assessment Manikin is a non-verbal picture scale asking for three judgements, pleasure, arousal and dominance. They compared it against a semantic differential of eighteen ratings. The two methods correlated highly for pleasure and arousal and differed on dominance.

Mandryk, Inkpen and Calvert (2006) used physiological signals recorded during gameplay to distinguish one play condition from another. The physiological response mirrored what participants reported. They argue that such measures complement self-report rather than replace it.

Lottridge, Chignell and Jovicic (2011) survey the territory for human factors, reviewing the concepts and the measurement techniques and recommending which suit which setting. Their gap is this unit's: the theory of emotion and the methods used to measure it are not yet fitted to each other.

Self-report recovers categories the dimensions miss

Cowen and Keltner (2017) tested the dimensional model against categories directly, analysing reports of responses to 2,185 short videos and found twenty-seven varieties of experience, with categories such as amusement capturing the reports better than valence and arousal ratings did.

Using deep neural networks, Cowen and colleagues (2020) examined sixteen types of facial expression across thousands of contexts in six million videos from 144 countries. They found context associations seventy per cent preserved across twelve world regions. The result supports many categories, though its labels are model outputs. The camps differ in what they count as evidence: the constructionist case rests on a pooled meta-analysis, the dimensional case on the structure of self-report, and the basic-emotion case on a signal and a physiology that the meta-analysis did not confirm.

Emotion is made in interaction, not detected inside a head

Boehner, DePaula, Dourish and Sengers (2007) attack the informational premise from inside HCI. Affective computing, they observe, often treats affect as information, discrete internal states transmitted without loss from person to system and back. They propose instead that emotion is interactional, dynamic and culturally mediated, made in action rather than detected.

Work the informational premise on a real kind of system. A learning platform infers frustration from the speed and pressure of clicks, logs a score against each learner, and adjusts the difficulty. Boehner and colleagues (2007) would ask what the score measures: the interpretation the learner gives the task, the setting and the other people in it is not in the log. The case shows what the premise assumes, and not that the system is useless. One assumption is a typical body, since an expression classifier is least tested on faces outside its training range.

McCarthy and Wright (2004) reach a compatible conclusion from a different direction. In Technology as Experience they argue that felt experience is emotional, intellectual and sensual at once, and that people live with technology rather than merely use it. Their account is interpretive rather than measured, so it cannot settle a measurement question. It explains why a convenient measure can miss what a person was doing.

Design for experience, and say whose experience

Hassenzahl (2004) separates the pragmatic from the hedonic in judgement. Two studies of four MP3-player skins measured perceived usability, hedonic attributes such as stimulation and identification, goodness and beauty. Goodness depended on both perceived usability and the hedonic attributes, while beauty depended largely on identification. Perceived usability and goodness moved with experience, whereas the hedonic attributes and beauty stayed stable. A design that optimises only the pragmatic attributes optimises one input to a judgement with at least two.

Contested question: what kind of thing is an emotion, and what would settle it?

The literature disagrees about whether emotion categories exist in nature, and the disagreement is not verbal.

• Natural kinds. Ekman (1992) argues for a small set of basic emotions with unique signals, and Cowen and Keltner (2017, 2020) find reported experience and facial expression both sort into many categories with cross-cultural consistency.
• Construction. Barrett (2006) reviews the evidence inconsistent with natural kinds, and the meta-analysis by Lindquist and colleagues (2012) finds no consistent one-to-one mapping between discrete categories and brain regions.
• Dimensions. Russell's (1980) two-axis account survives as a description of the space, but the categories it omits keep reappearing in the data (Cowen and Keltner, 2017).

A parallel dispute runs through interaction research: Boehner and colleagues (2007) treat emotion as made in interaction, while the programme Picard (1997) set out treats it as a state a machine can recognise and express.

Van Heijst, Kret and Ploeger (2023) dissected both theories against Tinbergen's four questions and proposed that they explain different phenomena, basic emotion theory the body's regulatory states and the constructed theory the feelings that arise from them, so the two might be supplementary. Barrett and colleagues (2025) rejected that reading. They argue the split misrepresents the constructed theory and that the two rest on incommensurate assumptions requiring different methods of evaluation, which they call intrinsically irreconcilable. On that reply no single converging test settles both, and a study must declare which theory it is testing.

A converging test is what the constructionist side asks for. A study would predict a specific category from brain activity, physiology, behaviour and self-report in the same participants across cultures, and recover the same boundary from each measure. The absence of that neural mapping in the pooled literature is the strongest current evidence against it (Lindquist et al., 2012). The measurement dispute would be settled by an instrument that declares its theory and is validated against a criterion it did not build in. Until then, a label such as "frustrated" is a claim about a person that a particular apparatus produced.

While the dispute is open, a designer can act on what all three readings allow. Name the instrument and the theory behind any affective claim, and report affect beside task performance rather than instead of it. Treat a detected state as a hypothesis about a person, and let the person confirm or correct it.

Where the argument is heading

The direction of travel is towards measurement that states its commitments, at scale. Picard (1997) set the agenda by arguing that computers should recognise, understand and express emotion, and Cowen and colleagues (2020) have shown the scale the programme can reach, classifying facial expression in six million videos from 144 countries. Barrett (2017) offers the opposite pole as a computational account of construction. The two converge on one demand: say what the categories are before reporting what was measured.

Boehner and colleagues (2007) keep the counter-current alive, arguing that systems should support interpretation rather than detection.

The sources this unit rests on

• Barrett, L. F. (2006). Are emotions natural kinds? Perspectives on Psychological Science, 1(1), 28-58. doi:10.1111/j.1745-6916.2006.00003.x Reviews the evidence inconsistent with the view that emotions are kinds with boundaries carved in nature, and asks what follows if they are not.
• Barrett, L. F. (2017). The theory of constructed emotion: an active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(1), 1-23. doi:10.1093/scan/nsw154 Sets out a brain-based computational account in which an emotion instance is built from interoception and categorisation rather than detected as a fixed kind.
• Barrett, L. F. and colleagues (2025). The theory of constructed emotion: more than a feeling. Perspectives on Psychological Science, 20(3), 392-420. doi:10.1177/17456916251319045 Answers a 2023 attempt to reconcile basic emotion theory with the constructed theory, arguing that the reconciliation misstates the constructed theory and that the two rest on incommensurate assumptions requiring different methods of evaluation.
• Boehner, K., DePaula, R., Dourish, P. and Sengers, P. (2007). How emotion is made and measured. International Journal of Human-Computer Studies, 65(4), 275-291. doi:10.1016/j.ijhcs.2006.11.016 Argues that affective computing treats affect as information, discrete internal states transmitted without loss, and so reproduces the information-processing model it set out to challenge.
• Bradley, M. M. and Lang, P. J. (1994). Measuring emotion: the Self-Assessment Manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry, 25(1), 49-59. doi:10.1016/0005-7916(94)90063-9 Introduces the Self-Assessment Manikin, a non-verbal scale of pleasure, arousal and dominance, and compares it with an eighteen-rating semantic differential; correlations were high for pleasure and arousal and differed for dominance.
• Cowen, A. S. and Keltner, D. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38), E7900-E7909. doi:10.1073/pnas.1702247114 Analysed self-reports of responses to 2,185 short videos and found twenty-seven varieties of reported experience, with categories such as amusement capturing the reports better than valence and arousal ratings.
• Cowen, A. S., Keltner, D., Schroff, F., Jou, B., Adam, H. and Prasad, G. (2020). Sixteen facial expressions occur in similar contexts worldwide. Nature, 589(7841), 251-257. doi:10.1038/s41586-020-3037-7 Used deep neural networks to examine sixteen facial expressions across thousands of contexts in six million videos from 144 countries, and found context associations seventy per cent preserved across twelve world regions. Its labels, however, are model outputs.
• Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3-4), 169-200. doi:10.1080/02699939208411068 Argues that emotions evolved for their value in fundamental life tasks, each with a unique signal, physiology and class of antecedents and shared response features.
• Hassenzahl, M. (2004). The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 19(4), 319-349. doi:10.1207/s15327051hci1904_2 Two studies of four MP3-player skins measured perceived usability, hedonic attributes, goodness and beauty. Goodness depended on perceived usability and the hedonic attributes, beauty largely on identification, and usability and goodness moved with experience while hedonic attributes and beauty stayed stable.
• Van Heijst, K., Kret, M. E. and Ploeger, A. (2023). Basic emotions or constructed emotions: insights from taking an evolutionary perspective. Perspectives on Psychological Science, 20(3), 377-391. doi:10.1177/17456916231205186 Dissects both theories against Tinbergen's four questions and proposes they explain different phenomena, the body's regulatory states against the feelings arising from them, so the two might be supplementary. A conceptual analysis, not a test.
• Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E. and Barrett, L. F. (2012). The brain basis of emotion: a meta-analytic review. Behav. Brain Sci., 35(3), 121-143. doi:10.1017/s0140525x11000446 Meta-analysed the neuroimaging literature and compared the locationist account, in which discrete categories correspond to distinct regions, with the constructionist account. It found little evidence that discrete categories localise consistently and specifically, and evidence for general networks active across categories.
• Lottridge, D., Chignell, M. and Jovicic, A. (2011). Affective interaction: understanding, evaluating, and designing for human emotion. Reviews of Human Factors and Ergonomics, 7(1), 197-217. doi:10.1177/1557234x11410385 Reviews emotion concepts, the arousal and valence dimensions, surveys the measurement techniques, and recommends which suit which setting.
• Mandryk, R. L., Inkpen, K. M. and Calvert, T. W. (2006). Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour and Information Technology, 25(2), 141-158. doi:10.1080/01449290500331156 Ran two experiments testing physiological measures as evaluators of entertainment technology, and found a physiological response that distinguished the play conditions and mirrored what participants reported.
• McCarthy, J. and Wright, P. (2004). Technology as Experience. MIT Press. doi:10.7551/mitpress/6687.001.0001 Argues that any account of user experience must take in the emotional, intellectual and sensual aspects of interaction, that people live with technology rather than use it, and builds a framework from Dewey and Bakhtin.
• Picard, R. W. (1997). Affective Computing. MIT Press. doi:10.7551/mitpress/1140.001.0001 Argues that computers should recognise, understand and express emotion if they are to interact naturally with people, making emotion a computational object.
• Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161-1178. doi:10.1037/h0077714 Scaled twenty-eight emotion-denoting adjectives in four ways and analysed 343 self-reports, finding the terms arranged on a circle in a two-dimensional bipolar space of pleasure and arousal.