What a number is a number of: measurement and construct validity

A usability score, an engagement rate or a completion time is not the thing it names. Each is a reading taken with an instrument, and the instrument is a decision about what to count. This first of ten units argues that the characteristic failure of quantitative human-computer interaction is not an unreliable instrument but an unexamined construct. The field measures what is convenient, it gives the number a name borrowed from the construct it resembles, and it lets the name carry an inference the measurement cannot support.

The claim is about the construct, not the instrument

Cronbach and Meehl (1955) distinguished construct validity from predictive and content validity. They argued that a test measures an attribute only within a nomological network, a body of theory that specifies how the attribute relates to other constructs. Campbell and Fiske (1959) turned the demand into a test: their multitrait-multimethod matrix requires convergence across methods that measure the same trait and discrimination between traits. A single instrument supplies neither comparison, however reliable it is, so a study that reports one questionnaire has established no construct. Kaptein and Robertson (2012) re-analysed public CHI 2010 data and argued that canned p values build weak theories from vaguely specified hypotheses. The case shows a decision reconsidered on published data; it does not show that the effects were large.

The scale decides which arithmetic is admissible

Stevens (1946) defined measurement as the assignment of numerals to objects or events according to rules, and separated nominal, ordinal, interval and ratio scales by the rules used. The arithmetic admissible depends on which kind the scale is. Counting errors or seconds gives a true zero and equal intervals, so a mean is meaningful. Ranking a preference as poor, fair, good or excellent gives order without equal spacing, so the median is honest and the mean is unsupported. His own illustration, numbering football players, is nominal. A seven-point item places a product between two adjectives, giving order without equal spacing, so a study that reports its mean has taken on the interval assumption whether or not it states it. The same item, read in a second language, through a screen reader or by an older reader, measures the reading as well as the product.

The field measures parts, not the construct

Hornbaek (2006) reviewed usability measures from 180 studies in core HCI journals and proceedings, sorting them against the ISO 9241 components of effectiveness, efficiency and satisfaction. Of those studies, 92 per cent measured at least one of task completion time, accuracy or satisfaction, but only 13 per cent measured all three, and in many the reasoning about measures fell short of a valid account of usability. Bargas-Avila and Hornbaek (2011) found the same drift in the newer user-experience literature, reviewing 51 publications covering 66 studies and finding many self-developed questionnaires that lacked validation.

The components are not interchangeable, and the standard revised its own definition

The strongest HCI-specific evidence that the construct is the problem comes from the measures themselves. Hornbaek and Law (2007) calculated correlations among usability measures from the raw data of 73 studies and found them low: effectiveness and efficiency at .247, with a 95 per cent interval of plus or minus .059, efficiency and satisfaction at .196, plus or minus .064, and effectiveness and satisfaction at .164, plus or minus .062. Hassenzahl (2004) shows what pooling them loses. In two studies using four MP3-player skins he found that goodness, meaning satisfaction, depended on both perceived usability and hedonic attributes, while beauty depended largely on identification. Perceived usability and goodness were affected by experience, and beauty stayed stable. A single satisfaction score merges constructs that respond to different things and move on different schedules.

The definition the field measures against is itself provisional. Bevan, Carter and Harker (2015) describe the revision of ISO 9241-11, whose 1998 version defined usability as effectiveness, efficiency and satisfaction in a context of use and framed it as an outcome of interaction rather than a property of a product. The revision widened satisfaction and redefined efficiency.

The gap is documented, quantified and general

The gap has been counted inside HCI as well as outside it. Perrig and colleagues screened four years of CHI proceedings, 2019 to 2022, and found 60 eligible studies that used survey scales, carrying 85 scales for 172 distinct constructs, most used once. Only 36.7 per cent reported any investigation of scale quality, and only 20 per cent gave a complete rationale for choosing a scale (Perrig et al., 2024). Nor is it unique to HCI. Hussey and Hughes (2020) assessed structural validity for fifteen self-report measures across 144,496 experimental sessions. Judged by internal consistency alone, 88 per cent appeared valid; judged comprehensively, by test-retest reliability, factor structure and measurement invariance, only 4 per cent were. Flake and Fried (2020) define questionable measurement practices as decisions that raise doubts about the validity of a measure and about a study's conclusions. They argue that a lack of transparency makes the threats to a study's inferences impossible to judge. Chester and Lasko (2020) extend the diagnosis to the input, coding 348 experimental manipulations from the 2017 issues of the Journal of Personality and Social Psychology. The vast majority were unvalidated before use, and roughly two fifths relied solely on face validity. A study can measure its outcome carefully and still have moved the wrong thing. The construct must therefore be examined on both sides of the hypothesis.

Contested question: repair the measure, or abandon the construct?

The literature agrees that something is wrong, and it disagrees about what. One position holds that the construct is sound and the practice is not. Cronbach and Meehl (1955) and Campbell and Fiske (1959) supply the standard, and Hussey and Hughes (2020) and Flake and Fried (2020) show that it is unmet. Hornbaek (2006) sets the task of validating and standardising satisfaction questionnaires, and Bevan and colleagues (2015) keep measurement and add the rest of evaluation.

The opposing position holds that the construct is the problem. Tractinsky (2017) argues that usability is an umbrella construct whose relations to its measures have been conceptualised formatively but tested reflectively, and concludes that it should be unbundled into well-defined constructs. Borsboom, Mellenbergh and van Heerden (2004) put the objection at its root. A test is valid for measuring an attribute only if the attribute exists and variation in it causally produces variation in the outcomes. No accumulation of correlations can rescue a construct that names no such entity. Borsboom (2006) adds that the operationalist habit of treating a construct as whatever the test measures is why psychology and psychometrics have failed to integrate.

What would settle the disagreement is a head-to-head predictive test the literature has not run. Operationalise usability both ways, measure each component with a validated instrument, and compare how well each version predicts outcomes such as adoption, error and preference. The repair position predicts that the umbrella score will out-predict its parts; the unbundling position predicts the reverse, and that the umbrella score will prove unstable when the context changes. Hornbaek and Law (2007) give the strongest indirect evidence for unbundling. No study in the unit's literature reports that comparison.

The two sides do not carry equal evidence. The repair position rests on audits of published practice, which count how common a fault is and cannot test a remedy. The unbundling case rests on an argument about what validity means and one reanalysis reporting low correlations.

The strongest objection is that the fault may be a reporting fault. Hussey and Hughes judged what studies published, so a measure that was never assessed counts as invalid, and the figure is a floor rather than an estimate.

What to do while the dispute is open

While the dispute is open, treat every score as a proxy for the construct rather than as its name. State the construct and the instrument beside the number, and report the parts with the total.

Where the argument is heading

The named work points one way: measurement decisions are becoming part of the method that must be reported. Flake and Fried (2020) would make the questions about a measure a required part of a paper, Hussey and Hughes (2020) would make comprehensive validity reporting the norm, Chester and Lasko (2020) would have manipulations validated before use, and Tractinsky (2017) would have the umbrella construct unbundled into constructs that can be validated at all. The burden has shifted from the precision of the test to the definition of the construct.

The sources this unit rests on

• Bargas-Avila, J. A. and Hornbaek, K. (2011). Old wine in new bottles or novel challenges. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1978942.1979336 They reviewed 51 user-experience publications from 2005 to 2009, covering 66 studies, and found the measured dimensions inconsistent and many studies using self-developed questionnaires without validation. It shows the construct in flux.
• Bevan, N., Carter, J. and Harker, S. (2015). ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? Lecture Notes in Computer Science. doi:10.1007/978-3-319-20901-2_13 They describe the revision of ISO 9241-11, whose 1998 definition framed usability as effectiveness, efficiency and satisfaction in a context of use, and record that measurement alone is now insufficient. It is the standards body's own concession.
• Borsboom, D. (2006). The Attack of the Psychometricians. Psychometrika. doi:10.1007/s11336-006-1447-6 He analyses why psychology and psychometrics have not integrated, blaming an operationalist habit of treating a construct as whatever the test measures. It carries the objection that a proliferation of named scores repeats a documented error.
• Borsboom, D., Mellenbergh, G. J. and van Heerden, J. (2004). The Concept of Validity. Psychological Review. doi:10.1037/0033-295X.111.4.1061 They advance a realist conception: a test is valid for measuring an attribute if the attribute exists and variation in it causally produces variation in the outcomes. It is the counter-position to validity as an accumulation of correlations.
• Campbell, D. T. and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin. doi:10.1037/h0046016 They set out the multitrait-multimethod matrix, requiring convergence across methods measuring the same trait and discrimination between traits. It supplies the standard against which the unit measures the field.
• Chester, D. S. and Lasko, E. N. (2020). Construct Validation of Experimental Manipulations in Social Psychology. Perspectives on Psychological Science. doi:10.1177/1745691620950684 They coded 348 experimental manipulations from the 2017 issues of the Journal of Personality and Social Psychology and found the vast majority created ad hoc and not previously validated, with roughly two fifths relying solely on face validity. It extends the unit's claim from outcomes to inputs.
• Cronbach, L. J. and Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin. doi:10.1037/h0040957 They distinguish construct validity from predictive and content validity and argue that a test measures an attribute only within a nomological network relating it to other constructs. It names the unit's problem.
• Flake, J. K. and Fried, E. I. (2020). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245920952393 They define questionable measurement practices as decisions that raise doubts about the validity of a measure and therefore of a study's conclusions, and argue that a lack of transparency hides a source of researcher degrees of freedom. It turns an abstract worry into a checkable gap.
• Hassenzahl, M. (2004). The Interplay of Beauty, Goodness, and Usability in Interactive Products. Human-Computer Interaction. doi:10.1207/s15327051hci1904_2 In two studies with four MP3-player skins he found goodness depended on perceived usability and hedonic attributes while beauty depended largely on identification, and that perceived usability and goodness were affected by experience while beauty stayed stable. It shows one score merging constructs that behave differently.
• Hornbaek, K. (2006). Current practice in measuring usability. International Journal of Human-Computer Studies. doi:10.1016/j.ijhcs.2005.06.002 He reviewed usability measures from 180 studies in core HCI journals and proceedings and found a large, inconsistent vocabulary, with 92 per cent measuring at least one of completion time, accuracy or satisfaction but only 13 per cent measuring all three. It documents what HCI actually measures.
• Hornbaek, K. and Law, E. L. (2007). Meta-analysis of correlations among usability measures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1240624.1240722 From the raw data of 73 studies they found correlations among usability measures low: .247 between effectiveness and efficiency, .196 between efficiency and satisfaction, and .164 between effectiveness and satisfaction. It is the evidence that the numbers are not proxies for one another.
• Hussey, I. and Hughes, S. (2020). Hidden Invalidity Among 15 Commonly Used Measures in Social and Personality Psychology. Advances in Methods and Practices in Psychological Science. doi:10.1177/2515245919882903 Across 144,496 experimental sessions they assessed structural validity for fifteen questionnaires and twenty-six scales, finding 88 per cent valid by internal consistency alone but only 4 per cent valid comprehensively. It shows the gap is general, not an HCI idiosyncrasy.
• Perrig, S. A. C., Aeschbach, L. F., Scharowski, N., et al. (2024). Measurement practices in user experience (UX) research: a systematic quantitative literature review. Frontiers in Computer Science. doi:10.3389/fcomp.2024.1368860 They screened CHI 2019 to 2022 and found 60 eligible studies using 85 scales for 172 constructs, most used once, with scale quality investigated in 36.7 per cent and a complete rationale in 20 per cent.
• Stevens, S. S. (1946). On the theory of scales of measurement. Science. doi:10.1126/science.103.2684.677 He defined measurement as the assignment of numerals to objects or events according to rules and separated nominal, ordinal, interval and ratio scales, arguing that the arithmetic admissible depends on the scale. It establishes the claim that a mean on an ordinal scale is a category error.
• Tractinsky, N. (2017). The Usability Construct: A Dead End? Human-Computer Interaction. doi:10.1080/07370024.2017.1298038 He argues that usability is an umbrella construct whose relations to its measures have been conceptualised formatively but tested reflectively, and concludes that it should be unbundled into well-defined constructs. It is the strongest form of the unit's objection.
• Kaptein, M. and Robertson, J. (2012). Rethinking statistical analysis methods for CHI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/2207676.2208557 They re-analysed public CHI 2010 data, arguing that canned p values build weak theories and should yield to effect magnitude and practical importance.