Instruments: reliability is necessary, not sufficient

Ask a team how it knows its questionnaire works, and the answer is usually a statistic. The team reports coefficient alpha as the instrument's credential, and readers take it as a pass. Three instruments carry most of HCI's subjective measurement: the System Usability Scale (SUS), the NASA Task Load Index (NASA-TLX) and the User Experience Questionnaire (UEQ). Alpha is a lower bound on reliability under assumptions that these instruments do not meet. It says nothing about how many constructs the items carry. This second of ten units argues that a reliable instrument is not yet a measured construct. The instruments remain useful. Their psychometric credentials are weaker than their ubiquity implies, and a reported alpha licenses a narrower claim than the literature usually draws from it.

An instrument arrives with a design argument, not a proof

Each of the three arrived with its purpose stated and its limits named. Brooke (1996) offered the ten-item SUS as a quick and dirty questionnaire, and the phrase was his own. The SUS trades diagnostic detail for a single score, and its alternating positive and negative items reduce response bias rather than establish structure, though they ask more of an older respondent than a single-direction scale does. Hart and Staveland (1988) produced NASA-TLX, a six-subscale instrument whose weights are set by each participant's own paired comparisons. That is a design feature with no independent reliability evidence attached. Laugwitz, Held and Schrepp (2008) built the UEQ as a semantic differential with six scales, and they deliberately kept the dimensions apart. The field has since treated ubiquity as evidence.

NASA-TLX carries the most elaborate design of the three, and its author's retrospective is the clearest evidence that design sophistication is not psychometric evidence. Hart (2006) surveyed 550 studies in which NASA-TLX had been used or reviewed. He reported that the raw ratings often carry the same information as the weighted index. Hart and Staveland (1988) presented the paired-comparison weighting as part of the instrument's rationale.

Alpha is a floor, not a structure

Cronbach (1951) showed that alpha is the mean of all split-half reliabilities from different splits of a test. It estimates the correlation between two random samples of items drawn from a universe like those in the test. Alpha is an index of equivalence and, except for very short tests, of first-factor concentration. Cronbach advised that a test divisible into distinct subtests should be divided before the formula is applied, because the statistic presupposes that the items measure one thing. Alpha is also a mean of split halves, so it is a lower bound on reliability rather than its value.

Green, Lissitz and Mulaik (1977) showed that alpha is a complexly determined statistic, in which item homogeneity is only one influence. Alpha can therefore be high for a heterogeneous item set. Sijtsma (2009) put the case at its sharpest. Alpha always takes a value, and it is unrelated to the internal structure of the test. Sijtsma and Pfadt (2021) revisit the lower bound. They show that alpha can be a poor estimate of reliability under conditions common in practice, including failure of essential tau-equivalence and correlated errors. A high alpha cannot distinguish items that measure one construct from items that measure two. It sets only a floor under reliability, not a level. Reporting alpha certifies less than the word suggests, and much less than the construct claim that the score is then asked to carry. Schrepp, Hinderks and Thomaschewski (2017) derived an eight-item short form of the twenty-six-item User Experience Questionnaire, the UEQ-S, where the full instrument is impractical. They report that the short form reproduces the longer one closely enough for practical use. With two items there are few split halves to average, and a high value cannot separate one construct from two. What it does not carry is evidence that its scales are unidimensional.

Reliability is not validity, and the SUS shows why

The SUS is the test case, because it is the most widely used standardised questionnaire for the assessment of perceived usability (Lewis, 2018). Lewis and Sauro (2009) applied factor analysis to two independent SUS datasets, and they found that the scale is not unidimensional. A usable factor rested on eight items, and a learnable factor rested on two. The score that carries the famous alpha therefore averages two things. Reliability concerns whether a score is consistent. Validity concerns whether it is a score of the construct named. No value of alpha could have revealed the two-factor structure (Green, Lissitz and Mulaik, 1977; Sijtsma, 2009). Nor would it reveal the shift when an item is translated, read aloud by a screen reader, or reached on a metered connection, because the reader then answers different words. Lewis (2018) reaches the same open conclusion. He records that the SUS proved quick but not dirty, he recommends its use, and he lists the outstanding questions about what it measures.

The norms that outrun the evidence

The interpretation layer built on the SUS rests on the same score. Bangor, Kortum and Miller (2008) analysed nearly ten years of SUS data from numerous products. They added an adjective rating that correlates with a score, plus acceptability ranges. Kortum and Bangor (2013) extended the norms to fourteen everyday products, with more than a thousand users who rated them from integrated experience without performing tasks. Users judged some products as a class, such as microwaves. A benchmark that compares a system against a remembered household appliance is a memory judgement rather than a measurement of interaction. The question is what they are norms of, given a score whose structure is contested (Lewis and Sauro, 2009) and whose reliability is a lower bound (Sijtsma and Pfadt, 2021).

Contested question: one SUS score or two, and does alpha decide it?

The literature disagrees about what a high alpha licenses, and the disagreement is clearest over the SUS. One position holds that the score is a single measure. Bangor, Kortum and Miller (2008) treat the SUS as robust and versatile and supply the norms. Kortum and Bangor (2013) extend them. Lewis (2018) recommends the SUS for anyone needing a measure of perceived usability. On this account the two-factor finding is a curiosity, because the factors correlate so highly with the total that the total is a fair summary.

The opposing position holds that the two-factor finding is the finding. Lewis and Sauro (2009) show that the single score averages two constructs. Green, Lissitz and Mulaik (1977) and Sijtsma (2009) establish that no value of alpha could have contradicted this. Sijtsma and Pfadt (2021) add that treating alpha as the reliability carries a further error. On this account the field reports a statistic that cannot detect the problem it is used to rule out. It also reports a score that merges a product's current usability with its learnability.

The two positions can be tested, and the literature has not run the test. What would settle it is a study that fits the factor structure and then asks whether the learnable factor predicts learning-related outcomes independently of the total score. Such a study would report reliability as omega total or the greatest lower bound alongside alpha (McNeish, 2018), so that the gap between the lower bound and a defensible estimate is visible. Lewis and Sauro supply the structure but not the predictive comparison.

The two sides rest on different kinds of evidence. The one-score position rests on norms from large archives of ratings, which describe a distribution rather than a structure. The two-factor position rests on a factor analysis of two datasets, which is a structural test.

The strongest objection is that the unit asks for a property no design decision needs. The one-score case does not claim unidimensionality. It claims the total predicts what a team must decide, and Lewis and Sauro found both factors correlating highly with the total.

What to do while the dispute is open

While the dispute is open, report alpha as a floor and name a second estimate beside it. Say which construct the score stands for, and report the parts as well as the total where a scale has two.

Where the argument is heading

The named work points one way. Reliability is being demoted from a passport to one reported property among several. McNeish (2018) sets out alpha's problems in terms that a practitioner can follow. He notes that methodological criticism has not appreciably changed how studies report reliability, and he recommends alternatives that make less rigid assumptions, including omega total and the greatest lower bound. Sijtsma and Pfadt (2021) continue the technical argument about lower bounds and correlated errors rather than closing it in alpha's favour. Boateng and colleagues (2018) supply the constructive counterpart, a validation procedure in three phases and nine steps that reaches reliability and validity testing only at the end. Lewis (2018) sees continued use of the SUS alongside an unfinished account of what it measures. Laugwitz and colleagues (2008) built the UEQ from the start as dimensions rather than a single score.

The sources this unit rests on

• Bangor, A. W., Kortum, P. T. and Miller, J. (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. doi:10.1080/10447310802205776 Analysed nearly ten years of SUS data from numerous products and added an adjective rating that correlates with a score plus acceptability ranges. It supplies the norms and the stability evidence on which the one-score position rests.
• Boateng, G. O., Neilands, T. B., Frongillo, E. A., et al. (2018). Best Practices for Developing and Validating Scales for Health, Social, and Behavioural Research: A Primer. Frontiers in Public Health. doi:10.3389/fpubh.2018.00149 Set out scale development in three phases and nine steps, from item generation and content validity to reliability and validity testing. It is the constructive counterpart, placing reliability late in the sequence rather than at its start.
• Brooke, J. (1996). SUS: A 'Quick and Dirty' Usability Scale. Usability Evaluation in Industry. doi:10.1201/9781498710411-35 Introduced the ten-item SUS as a quick and dirty questionnaire trading diagnostic detail for a single score, with alternating positive and negative items. The design is the author's own, so the instrument arrived as a decision rather than a validated construct.
• Cronbach, L. J. (1951). Coefficient Alpha and the Internal Structure of Tests. Psychometrika. doi:10.1007/bf02310555 Showed that alpha is the mean of all split-half reliabilities of a test, an estimate of equivalence and, except for short tests, of first-factor concentration. He advised dividing a test into subtests before applying the formula, which states the unidimensionality assumption.
• Green, S. B., Lissitz, R. W. and Mulaik, S. A. (1977). Limitations of Coefficient Alpha as an Index of Test Unidimensionality. Educational and Psychological Measurement. doi:10.1177/001316447703700403 Argued that alpha is a complexly determined statistic in which item homogeneity is only one influence, and that using it as a homogeneity index is a misuse. It is the technical evidence that a high alpha and a multidimensional scale can coexist.
• Hart, S. G. (2006). NASA-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. doi:10.1177/154193120605000909 Surveys 550 studies in which NASA-TLX was used or reviewed and reports that the raw ratings often carry the same information as the weighted index. A canonical design feature is questioned by the instrument's own author.
• Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology. doi:10.1016/s0166-4115(08)62386-9 Reports the multi-year programme that produced NASA-TLX, a six-subscale instrument whose weights are set by the participant's own paired comparisons. It introduces the design feature that the later review questions.
• Kortum, P. T. and Bangor, A. (2013). Usability Ratings for Everyday Products Measured With the System Usability Scale. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2012.681221 Measured the SUS for fourteen everyday products with more than a thousand users, who rated products from integrated experience without performing tasks, some judged as a class. The benchmarks compare systems against remembered products rather than measured interaction.
• Laugwitz, B., Held, T. and Schrepp, M. (2008). Construction and Evaluation of a User Experience Questionnaire. Lecture Notes in Computer Science. doi:10.1007/978-3-540-89350-9_6 Report the construction and evaluation of the UEQ, a semantic differential with six scales from attractiveness and perspicuity to stimulation and novelty. It is the multidimensional alternative to a single usability score.
• Lewis, J. R. (2018). The System Usability Scale: Past, Present, and Future. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2018.1455307 Reviews the SUS from inception to recent research, finds it proved quick but not dirty, recommends it for measuring perceived usability, and lists the questions about what it measures. The field's most studied instrument remains not fully understood.
• Lewis, J. R. and Sauro, J. (2009). The Factor Structure of the System Usability Scale. Lecture Notes in Computer Science. doi:10.1007/978-3-642-02806-9_12 Factor analysis of two independent SUS datasets found the scale is not unidimensional: a usable factor of eight items and a learnable factor of two, both reliable and highly correlated with the total. The single score therefore averages two constructs.
• McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods. doi:10.1037/met0000144 Reviews alpha's problems and its unrealistic assumptions in terms practitioners can follow, and reports that methodological criticism has not changed reporting practice. It recommends alternatives including omega total and the greatest lower bound.
• Sijtsma, K. (2009). On the Use, the Misuse, and the Very Limited Usefulness of Cronbach's Alpha. Psychometrika. doi:10.1007/s11336-008-9101-0 Argues that alpha always has a value, is unrelated to the internal structure of the test, and is of very limited use as a reliability estimate. It shows no value of alpha could detect the multidimensionality Lewis and Sauro found.
• Sijtsma, K. and Pfadt, J. M. (2021). Part II: On the Use, the Misuse, and the Very Limited Usefulness of Cronbach's Alpha: Discussing Lower Bounds and Correlated Errors. Psychometrika. doi:10.1007/s11336-021-09789-8 Revisits alpha as a lower bound on test-score reliability, considers the criticism that it requires essential tau-equivalence, and examines correlated errors. It shows alpha can poorly estimate reliability under conditions common in practice.
• Schrepp, M., Hinderks, A. and Thomaschewski, J. (2017). Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence. doi:10.9781/ijimai.2017.09.001 They derive an eight-item short version of the twenty-six-item UEQ and show it reproduces the full instrument closely enough for practical use while reducing completion time.