Validity, measurement and what a finding does not show

A project can be well argued, adequately powered and honestly reported and still support less than its abstract claims. The limit is usually distance: it runs between what the project wanted to know and what it measured. It runs again between the participants it studied and the people its conclusion quietly covers.

The seventh of ten units argues that a claim is bounded by the validity of its construct, its measurement and its sample, and the eighth asks how the bound is written down. The paper should therefore state where its knowledge stops. Validity is not a badge a method carries. It is a property of the inference a reader is asked to accept.

Validity belongs to the interpretation, not the instrument

Messick (1995) argued that dividing validity into content, criterion and construct types is fragmented and incomplete. In his unified account the interpretation and use of scores are what get validated, weighed across six aspects: content, substantive, structural, generalisation, external and consequential (Messick, 1995).

Kane (2013) makes that a procedure. Write the claim as an argument, state each inference and its assumptions, then appraise the argument's coherence and completeness. A validation covers the proposed interpretations and uses, not the test, and ambitious claims need more support than modest ones (Kane, 2013). A claim that participants trusted the system decomposes into a chain: item responses to a score, score to trust, and these participants to a population.

Construct validity has to be specified before it can be checked

Cronbach and Meehl (1955) asked how a test can be shown to measure an attribute that no single criterion defines. A construct, they argued, is scientifically admissible only if it occurs in a nomological net. That net is a system of laws relating observables to one another and to theoretical terms. The investigator who proposes a test as a measure of a construct must specify that network. Others must be able to accept or reject it (Cronbach and Meehl, 1955). Their worked example shows the mechanism. Candidate test X correlates about .50 with the palmar sweating a student shows after being told they failed an exam. It correlates .55 with intellectual inefficiency induced by electric shock, and .68 with the Taylor Anxiety scale. It correlates negligibly with social class, vocational aim and value orientation. The pattern supports reading X as a measure of anxiety proneness and rules out academic aspiration. That reading holds only because the theory of anxiety predicts the first set of links and not the second.

Flake, Pek and Hehman (2017) measured practice against that demand. They found construct validity evidence lacking, with coefficient alpha often the only psychometric evidence reported. Flake and Fried (2020) name such failures questionable measurement practices, decisions that raise doubt about the validity of the measures and so of the conclusions. The remedy is a writing obligation rather than a second study.

The measure decides what the construct can mean

Reliability is the easiest part of measurement to report and the easiest to report badly. A high alpha says only that the items hang together, not that the scores mean what the project says they mean.

Hussey and Hughes (2020) show what the gap costs. They assessed structural validity for 15 widely used questionnaires, 26 scales, across 144,496 experimental sessions. On internal consistency alone, 88 per cent of the scales appeared valid. Assessed comprehensively, including factor structure, test-retest reliability and measurement invariance, only 4 per cent did. They call the room for manoeuvre in this reporting validity hacking. Only the Need for Cognition scale passed all four checks. The Big Five Inventory subscales passed internal consistency and test-retest reliability while failing measurement invariance. The same items therefore did not behave the same way for respondents of different ages, genders, languages or abilities.

Each validity type carries its own threat

Campbell and Stanley (1963) named the threats to internal and external validity, and Shadish, Cook and Campbell (2002) supply the framework a limitations section can be written against. Statistical conclusion validity concerns the inferences that cause and effect covary, and how strongly. Internal validity concerns whether the observed covariation reflects a causal relationship from A to B in the form in which the variables were manipulated or measured. Construct validity is the degree to which inferences are warranted from the observed persons, settings and operations to the constructs they might represent. External validity concerns whether the relation holds over variation in persons, settings, treatment variables and measurement variables, and persons vary in age, ability, language and connection (Shadish, Cook and Campbell, 2002).

An order effect threatens internal validity. A five-item questionnaire written for the study threatens construct validity. A sample of students on one task in one laboratory, without older or disabled participants, threatens external validity. A measure taken once cannot speak to learning. Writing those sentences is the difference between a boundary the project states and one the reader supplies.

HCI measures many things under one name

Hornbaek (2006) reviewed the usability measures used in 180 studies from core HCI journals and proceedings. He classified them against the ISO 9241 grouping of effectiveness, efficiency and satisfaction. He concluded that in many studies the choice of and reasoning about measures fall short of a valid account of usability as quality in use (Hornbaek, 2006).

Nielsen and Levy (1994) supply the case in which two plausible measures answer different questions. They pooled 113 comparisons from 57 studies in which interfaces for the same task had been evaluated on both stated preference and measured performance. They found the two measures favoured the same interface in about three quarters of cases. In roughly a quarter, users preferred the interface on which they performed worse. A preference result does not license a performance claim.

The setting and the sample bound the claim

Carter, Mankoff, Klemmer and Matthews (2008) surveyed the literature and interviewed 28 developers on what blocks ecologically valid design in ubiquitous computing. They found that sensing and scale make systems resist iteration and prototyping. A controlled setting does not remove context; it substitutes one of its own.

Cronbach (1975) generalises the point. He argues from a programme of aptitude by treatment interaction studies that generalisations decay. A conclusion valid when it was written may therefore later account for little of the variance. He argues that proper weight to local conditions makes any generalisation a working hypothesis rather than a conclusion. Shadish, Cook and Campbell (2002) ask the same question of the sample. External validity concerns whether a relation holds over variation in persons, and a project that recruited through a single channel has sampled persons in one way only.

The contested question: whose property is validity?

Two positions are live, and they disagree about what validity is a property of. For Messick (1995) validity attaches to the inferences and actions based on scores, and consequences of use count among the evidence. Borsboom, Mellenbergh and van Heerden (2004) reject that as a move from ontology to epistemology. A test is valid for measuring an attribute if the attribute exists and variations in it causally produce variation in the measurement outcome. Validation should study the processes conveying the attribute's effect on scores.

Messick obliges a project to argue for its interpretation and weigh the consequences of the use. Borsboom and colleagues oblige it to show that the attribute stands in a causal relation to the responses. In practice that means tracking or manipulating the attribute and watching the measure move. Kane's (2013) argument-based approach reads as a working settlement, keeping Messick's frame while demanding evidence for each inference.

What would settle the dispute is hard to state. It would take a measure with a demonstrated causal relation to its attribute, which the Borsboom account calls valid. Its use would also have to produce reliably bad consequences, so that the Messick account called the interpretation into question. It would then need a ruling on which verdict governs. Because the accounts disagree about what is being evaluated, no single study settles them.

Read by kind, both accounts are arguments about the concept and neither carries a measurement. Messick rests on a programme of validation practice, and Borsboom and colleagues on a causal definition of validity. While the dispute is open, a project should write its claim as a chain of inferences, state the assumption at each link, and report evidence for the measure in its own sample rather than by citation. The strongest objection to this unit's thesis is that a bounded claim is not automatically a better one, since a project may list every inference, make one the data do not support, and document a gap rather than close it.

Where this is heading

The measurement side of the credibility reform is younger than the statistics side. Three lines pull the same way. They are the reporting questions of Flake and Fried (2020), the demonstration by Hussey and Hughes (2020) that widely used scales can fail checks nobody ran, and the argument-based validation of Kane (2013). Carter and colleagues (2008) turn ecological validity into a design constraint. What remains untested is whether a measurement section that documents its inferences changes what reviewers and examiners accept.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked against the Crossref and OpenAlex records on 31 October 2026.

• Cronbach, L. J. and Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. doi:10.1037/h0040957 Set out construct validity for an attribute that no single criterion defines, and introduce the nomological net, a system of laws relating observables to one another and to theoretical terms. A construct is admissible only inside such a net, which the investigator must specify for others to accept or reject.
• Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. doi:10.1037/0003-066X.50.9.741 Argues that splitting validity into content, criterion and construct types is fragmented, and that validity belongs to the inferences and actions based on scores, with consequences of use counted as evidence.
• Borsboom, D., Mellenbergh, G. J. and van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071. doi:10.1037/0033-295X.111.4.1061 Argue that a test is valid for measuring an attribute when the attribute exists and variations in it causally produce variation in the measurement outcome, and that validity theory overweights meaning and correlation over reference and causality. Validation should study the processes conveying the attribute's effect on scores.
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73. doi:10.1111/jedm.12000 Sets out argument-based validation, in which claims made from scores are written as an argument with stated inferences and assumptions, and validation appraises its coherence, completeness and plausibility. It is the proposed interpretations and uses that are validated, not the test.
• Flake, J. K., Pek, J. and Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370-378. doi:10.1177/1948550617693063 Reviewed a sample of articles in the Journal of Personality and Social Psychology and found construct validity evidence lacking, with coefficient alpha often the only psychometric evidence reported.
• Flake, J. K. and Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456-465. doi:10.1177/2515245920952393 Define questionable measurement practices as decisions that raise doubt about the validity of the measures and hence of the conclusions. Thin reporting hides a large source of researcher degrees of freedom, and they offer reporting questions in place of a rule.
• Hussey, I. and Hughes, S. (2020). Hidden invalidity among 15 commonly used measures in social and personality psychology. Advances in Methods and Practices in Psychological Science, 3(2), 166-184. doi:10.1177/2515245919882903 Assessed structural validity for 15 widely used questionnaires, 26 scales, across 144,496 experimental sessions. On internal consistency alone 88 per cent looked valid; assessed comprehensively, only 4 per cent were, a practice they call validity hacking.
• Campbell, D. T. and Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research. Rand McNally. https://archive.org/details/experimentalquas0000camp Names the threats to internal and external validity, including history, maturation, testing, instrumentation and selection. It is the primary origin of the unit's threat catalogue.
• Shadish, W. R., Cook, T. D. and Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin. https://www.cengage.com/c/experimental-and-quasi-experimental-designs-for-generalized-causal-inference-1e-shadish/9780395615560/ Sets out four validity types for causal claims, statistical conclusion, internal, construct and external, with the threats each must survive. Construct validity is defined as the warrant for inferences from observed persons, settings and operations to the constructs they represent.
• Hornbaek, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102. doi:10.1016/j.ijhcs.2005.06.002 Reviewed the usability measures used in 180 studies from core HCI journals and proceedings, classified against effectiveness, efficiency and satisfaction. He finds that in many studies the choice of and reasoning about measures fall short of a valid account of usability as quality in use.
• Nielsen, J. and Levy, J. (1994). Measuring usability: Preference vs. performance. Communications of the ACM, 37(4), 66-75. doi:10.1145/175276.175282 Pooled 113 comparisons from 57 studies in which interfaces for the same task had been evaluated on both stated preference and measured performance. The two measures favoured the same interface in about three quarters of cases and disagreed in about a quarter.
• Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116-127. doi:10.1037/h0076829 Argues from a programme of aptitude by treatment interaction studies that generalisations decay in a world of interacting effects, so a conclusion may later account for little variance. Proper weight to local conditions makes any generalisation a working hypothesis rather than a conclusion.
• Carter, S., Mankoff, J., Klemmer, S. R. and Matthews, T. (2008). Exiting the cleanroom: On ecological validity and ubiquitous computing. Human-Computer Interaction, 23(1), 47-99. doi:10.1080/07370020701851086 Surveyed the literature and interviewed 28 developers on what blocks ecologically valid design in ubiquitous computing. Sensing and scale make systems resist iteration and prototyping, and evaluation struggles when critical events are infrequent or the evaluation interferes with use.