Warranting the claim: validity, measurement and the limits of inference

A proposal is judged twice on its method: whether the design can answer the question, and whether the answer will bear the conclusion drawn from it. This unit argues that a finding is only as good as the inference drawn from it.

This fifth of ten units argues in four moves. Validity attaches to the inference rather than the apparatus (Messick, 1995; Cronbach and Meehl, 1955), and reliability is not validity (Bland and Altman, 1997). The classical threats name where the leap from data to claim fails, and they trade against one another (Campbell, 1957; Bracht and Glass, 1968; Schmuckler, 2001). Measurement error and response bias are properties of the design, not noise (Furnham, 1986; Podsakoff, MacKenzie, Lee and Podsakoff, 2003; Kitchenham, Pfleeger and Fenton, 1995; Bulbulia, 2024).

The warrant is the inference, not the instrument

Messick (1995) argues that validity is an integrated evaluative judgement of the adequacy and appropriateness of inferences and actions based on test scores. The older division into content, criterion and construct validities is fragmented, he argues, because it leaves out the value implications of score meaning and the consequences of score use. His unified account gives construct validity six aspects, content, substantive, structural, generalisability, external and consequential.

Cronbach and Meehl (1955) supplied the ground. They set out construct validity as the central problem of psychological measurement. The network of theoretical relations in which a test participates validates it, rather than any single correlation. Naming a construct is then a theoretical commitment: a project measuring trust or workload has joined a literature that argues about what the term means.

Reliability is not validity, and internal consistency is neither

Bland and Altman (1997) explain Cronbach's alpha as internal consistency for items summed into a scale. They interpret it as the expected correlation between two random k-item samples of the available items. They give rules of thumb of 0.7 to 0.8 for scales used to compare groups, and 0.90 where an individual's score carries clinical weight. Their worked scale is the mini-HAQ, ten items on the difficulty of daily activities for patients with cervical myelopathy. Each item is scored from 1 for no difficulty to 4 for cannot do, and the ten scores are summed on a form the patient must read.

Alpha describes whether the items hang together; it does not establish that the summed score represents the attribute the project names.

The classical threats name where the inference breaks, and case study work inherits them

Campbell (1957) analysed the extraneous variables in social experiments and separated seven categories, history, maturation, testing, instrument decay, regression, selection and mortality, from their effects. Their main effects jeopardise internal validity, and standard designs control them; their interactive effects limit generalisability. Closing one route for a rival explanation therefore restricts the conditions to which the result generalises.

Bracht and Glass (1968) define external validity as the extent and manner in which results generalise to different subjects, settings, experimenters and tests. They divide the threats into population validity and ecological validity. Population validity concerns generalisation from the sample to the target population; ecological validity concerns the environment of the treatment, with novelty, experimenter and sensitisation effects among its threats.

Schmuckler (2001) argues that researchers use the term too loosely for that work. Concerns arise across the setting, the stimuli and the observer's response used as the measure, and no criteria have been published for applying it. That makes the objection available against almost any experiment. He proposes a dimensional account distinct from external validity.

The same four categories carry into qualitative work. Runeson and Host (2008) compile guidelines for case study research in software engineering from social science and information systems handbooks and adapt them to the field. They set out the categories in which threats to validity are analysed: construct, internal, external and reliability. Their answer to the sceptic is protocol and reporting.

Measurement error and response bias are designed in, not averaged out

Furnham (1986) reviewed response bias and dissimulation in self-report data. He found growing evidence that social desirability is a relatively stable, multidimensional trait rather than a situationally specific response set. A stable property of a subgroup of respondents distorts comparisons between subgroups, so it must be measured or designed against rather than assumed to cancel.

Podsakoff, MacKenzie, Lee and Podsakoff (2003) identify the sources of method bias. They distinguish biases arising from the measurement context, the item context and the respondent. They also evaluate the procedural and statistical remedies. A correlation between two variables measured on the same instrument in one sitting is partly a correlation between the instrument and itself.

Kitchenham, Pfleeger and Fenton (1995) carry the argument into computing. Their framework for validating software measurement rests on a measurement structure model and further models covering units, instrumentation, attribute relationships, protocols and entity populations. It found problems with the construction of function points, a measure already in use.

Bulbulia (2024) gives the structural version. He uses causal diagrams to set out measurement error bias and bias from restricting the target population, which he calls selection restriction. He names studies exhibiting either bias, or confounding, as weird. He argues that configural, metric and scalar invariance tests cannot address those biases, and offers workflows as preflight checklists.

The decision rule is not the inference

Cohen (1994) attacked the ritual of null hypothesis significance testing and catalogued its misinterpretations. The p value is widely read as the probability that the null hypothesis is false, when the test gives the probability of the data given the hypothesis and not the reverse. He recommends effect sizes with confidence intervals. He names two further misreadings alongside it: that the complement of p is the probability of a successful replication, and that rejecting the null confirms the theory that prompted the test. A plan that says a difference will be significant at p < 0.05 states a decision rule and not a claim, and must add what size of difference would matter.

A worked case: what a small model can and cannot warrant

Nielsen and Landauer (1993) fitted a Poisson model to eleven studies, thirteen datasets, of user testing and heuristic evaluation. They found problem detection as a function of the number of users or evaluators well modelled by that process. Single-evaluator detection ranged from 16 to 60 per cent across the studies, mean 33 per cent and fitted mean detection probability 0.31.

Their stated assumptions are the part a proposal should carry. Detection is independent of whether a problem has been found already, and the problems are independent of one another. They call that a simplification, since some difficulties co-occur. A plan that writes that five users find most problems claims one interface, one detection probability and independent detection. For a medium case it estimates that sixteen evaluations are worth their cost, with the best ratio of benefit to cost at four. That is a statement about planning an evaluation, not about how many users find how many problems.

Contested question: whose property is validity, and is ecological validity a threat at all?

Messick (1995) holds that it is a unified evaluative judgement of the adequacy and appropriateness of inferences and actions based on scores. The consequences of score use are one of its six aspects. Borsboom, Mellenbergh and van Heerden (2004) hold that a test is valid for measuring an attribute only if the attribute exists and variations in it causally produce variation in the outcomes. They conclude that validation should target the processes conveying the attribute's effect on the scores. Messick makes consequences part of validity; they make them irrelevant to it.

What would settle it is evidence about the causal route. A study that manipulated the attribute and compared the observed score with the conventional correlational and consequential evidence would show whether the additional aspects change the verdict. Neither side supplies it, so a proposal must say which conception it uses and stay consistent.

Bracht and Glass (1968) treat ecological validity as one of two classes of threat to external validity, answered by generalising the finding to other environments. Schmuckler (2001) argues that concerns arise in the setting, the stimuli and the response measure at once, and that no criteria exist for applying the term. A critic can therefore raise the objection against almost any experiment. What would settle that is a published set of criteria applied blind by independent coders to a sample of studies. No such instrument appears in this unit's literature, so the term must be operationalised on the spot.

Read by kind, both disputes are argued rather than demonstrated. Messick (1995) and Borsboom, Mellenbergh and van Heerden (2004) reason from the concept of validity, Schmuckler (2001) re-reads the history of a term, and no study here manipulates an attribute to compare the two accounts. The one fitted model in the unit, Nielsen and Landauer (1993), tests problem detection and not validity theory. While the dispute is open, a student should name the conception of validity the study uses, keep to it, and specify ecological validity by setting, stimuli and response. The strongest objection is Borsboom's: if validity turns on whether an attribute exists and causally produces the score, a well-bounded inference can pass the unit's audit while resting on a causal structure the study never tested.

Where the argument is heading

The direction is from a list of labels to a structural account of bias. Bulbulia (2024) draws measurement error and restricted sampling as features of a causal diagram and turns the analysis into preflight checklists. Schmuckler (2001) points the same way on the narrower term, replacing a binary verdict with dimensions specified in advance. Kitchenham, Pfleeger and Fenton (1995) supply the computing instance, where validating a measure is a multi-stage process.

The sources this unit rests on

Each note states what the work did, what it found and how it bears on the argument. Every identifier was resolved on 31 October 2026.

• Bland, J. M. and Altman, D. G. (1997). Statistics notes: Cronbach's alpha. doi:10.1136/bmj.314.7080.572 Explains alpha as internal consistency for summed scales, read as the expected correlation between two random k-item scores, with 0.7 to 0.8 satisfactory for comparing groups. Separates the coherence of an instrument from the meaning of the score it yields.
• Borsboom, D., Mellenbergh, G. J. and van Heerden, J. (2004). The concept of validity. doi:10.1037/0033-295x.111.4.1061 Argues that a test is valid for measuring an attribute only if the attribute exists and causally produces the measurement outcomes, and that validity theory has favoured correlation over causality. Redirects validation to the processes conveying the attribute's effect on the scores.
• Bracht, G. H. and Glass, G. V. (1968). The external validity of experiments. doi:10.2307/1161993 Defines external validity as the extent and manner in which results generalise to different subjects, settings, experimenters and tests, dividing the threats into population and ecological validity. Catalogues the threats to each.
• Bulbulia, J. A. (2024). Methods in causal inference. Part 3: measurement error and external validity threats. doi:10.1017/ehs.2024.33 Uses causal diagrams to set out measurement error bias and bias from restricting the target population, and argues that invariance tests cannot address those structural biases. Offers workflows as preflight checklists.
• Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. doi:10.1037/h0040950 Analyses the extraneous variables in social experiments and separates seven categories, history, maturation, testing, instrument decay, regression, selection and mortality, by how they act. Finds that main effects threaten internal validity while interactive effects limit generalisability.
• Cohen, J. (1994). The earth is round (p < .05). doi:10.1037//0003-066x.49.12.997 Reviews the problems with null hypothesis significance testing, including the misreading of p as the probability that the null hypothesis is false. Recommends effect sizes with confidence intervals and replication for generalisation.
• Cronbach, L. J. and Meehl, P. E. (1955). Construct validity in psychological tests. doi:10.1037/h0040957 Sets out construct validity as the central problem of psychological measurement, arguing that a test is validated by the network of theoretical relations it participates in rather than by any single correlation. Distinguishes it from the predictive, concurrent and content categories.
• Furnham, A. (1986). Response bias, social desirability and dissimulation. doi:10.1016/0191-8869(86)90014-0 Reviews response bias and the dissimulating of socially desirable answers to self-report data, and reports growing evidence that social desirability is a relatively stable, multidimensional trait rather than a situational response set. Puts the bias in respondents, so it must be designed against.
• Kitchenham, B. A., Pfleeger, S. L. and Fenton, N. (1995). Towards a framework for software measurement validation. doi:10.1109/32.489070 Proposes a framework for validating software measurement, built on a measurement structure model and further models covering units, instrumentation, attribute relationships, protocols and entity populations, and finds problems with the construction of function points. Recommends measurement vectors over contrived scalars.
• Messick, S. (1995). Validity of psychological assessment. doi:10.1037/0003-066x.50.9.741 Reformulates validity as an integrated judgement of the adequacy of inferences and actions based on scores, with content, substantive, structural, generalisability, external and consequential aspects. Argues that validity belongs to inferences, not instruments.
• Nielsen, J. and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. doi:10.1145/169059.169166 Fits a Poisson model to eleven studies of user testing and heuristic evaluation and finds problem detection well described by it. Reports single-evaluator detection from 16 to 60 per cent, mean 33 per cent and fitted mean 0.31.
• Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y. and Podsakoff, N. P. (2003). Common method biases in behavioural research: a critical review of the literature and recommended remedies. doi:10.1037/0021-9010.88.5.879 Identifies the sources of method bias, distinguishes biases of the measurement context, the item context and the respondent, and evaluates remedies for them. A single instrument used once measures itself as well as its target.
• Runeson, P. and Host, M. (2008). Guidelines for conducting and reporting case study research in software engineering. doi:10.1007/s10664-008-9102-8 Compiles guidelines for case study research in software engineering from social science and information systems handbooks, with recommended practices and checklists. Sets out the categories for analysing threats to validity, construct, internal, external and reliability.
• Schmuckler, M. A. (2001). What is ecological validity? A dimensional analysis. doi:10.1207/s15327078in0204_02 Examines the history of the concept and finds concerns arising across the setting, the stimuli and the observer's response used as the measure, with no published criteria for applying it. Proposes a dimensional account distinct from external validity.