Usability, user experience and the standards that fix the words

A specification that calls a product usable without saying for whom, at what, or under what conditions makes a claim that cannot be wrong. ISO 9241-11:2018 removes that freedom. It defines usability as the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. It states that usability is not an attribute of a product (ISO, 2018).

This unit argues that the standard converted usability from a property of a product into a measurement contract with named conditions, and that user experience escaped that discipline by widening what counts as evidence. Part 11 restricts a claim until it can be tested. Part 210 names an object so wide that the restrictions fall away. A term defined by perception can be measured with almost any instrument. The next unit turns to who holds control.

Every qualifier in the definition removes a claim

Four clauses carry the work. Specified users excludes the general public. Specified goals excludes whatever the user is doing instead. Effectiveness, efficiency and satisfaction fixes three components and no more. A specified context of use covers users, goals and tasks, resources and environment, which in practice means the device, the connection, the software the user works through, assistive technology among it, and the language the user reads. Remove one and the sentence becomes a preference. The document describes no methods, so it fixes words and leaves the evidence to whoever measures (ISO, 2018).

A knife is sharp as a property of the object. Usability belongs to a use, so the same interface is usable for a clinician at work and unusable at three in the morning with a queue waiting. Brooke (2013) records the settlement: no specific property of an artefact can be called usability.

The practitioner half names attributes, and the instrument decides what the claim covers

Nielsen (1993) supplied the practitioner half of the contract, setting usability out as attributes that can be specified and measured: learnability, efficiency, memorability, errors and satisfaction. His list does not match the standard's: memorability and errors are not its words, so the instrument decides what the claim covers.

Hart and Staveland (1988) built a workload measure on six subscales combined through a weighting drawn from fifteen pairwise comparisons. They reported that the weighted combination reduced between-subject variability and gave a more stable measure than a single global scale. The instrument's later history shows what happens to such a construction. Twenty years on, NASA-TLX had been used or reviewed in 550 studies, in applications far beyond its original one of aviation crew complement and beyond English (Hart, 2006). The weighting procedure travels with the measure, and the analyst who reads a score usually reads only the score. Brooke (1996) built the System Usability Scale as ten items producing one score, quick to administer and workable with small samples.

The instruments carry claims their construction does not

Brooke (2013) reports that the scale was developed in 1986, cited in more than 1,200 publications, and never through a formal standardisation process. He also reports that its nought to one hundred range was a marketing strategy, not a scientific one. Bangor, Kortum and Miller (2008) analysed a large archive of scale data across many products and development stages and concluded that the scale is robust and versatile. A later study added a seven-point adjective scale as an eleventh question to 964 surveys, where the ratings correlated with the scale scores at r = 0.822 and the mean score was 50.9 for OK, 71.4 for Good and 85.5 for Excellent (Bangor, Kortum and Miller, 2009). The instrument was built to produce one number; what the number means was settled twenty years later, by a second study.

What no definition can supply is whether its components move together. Hornbaek (2006) reviewed the measures used in 180 studies. He found that 57 per cent measured task completion time, 92 per cent measured at least one of task completion time, accuracy or satisfaction, and only 13 per cent measured all three. Hornbaek and Law (2007) computed the relation from the raw data of 73 studies, and found the correlations low. They were 0.247 between effectiveness and efficiency, 0.196 between efficiency and satisfaction and 0.164 between effectiveness and satisfaction. The components do not carry each other's evidence, so a usability claim has to name its measure.

The second standard widened the object

ISO 9241-210:2019 defines user experience as a person's perceptions and responses resulting from the use or anticipated use of a system, product or service, including emotions, beliefs, preferences and responses before, during and after use. It states that it does not cover methods in detail (ISO, 2019). Part 11 names users, goals, components and context. Part 210 names a person and a span of time opening before use and closing after it. Anticipation and recollection sit inside the object, and neither can be observed in a session. Part 11 already lists user experience and avoidance of harm as outcomes of use beside usability. The result is a scope set by whoever answers, not by a clause.

The concept was argued before it was measured

Hassenzahl and Tractinsky (2006) treat the term as a proposal rather than a finding: their agenda notes that user experience had become a buzzword promising change without being specific about its meaning. The paper reports Hassenzahl and Sandweg's result that summary judgements of usability depend heavily on the problems met at the end of an episode, so one post-session score samples an experience poorly.

Law, Roto, Hassenzahl, Vermeeren and Kort (2009) put the disagreement to the field: their survey of 275 researchers and practitioners found most agreeing that the concept is dynamic, context-dependent and subjective. The authors propose to delineate it as individual rather than social. Hassenzahl (2004) gives a measured reason to keep the wider term. Participants rated four MP3 player skins on pragmatic quality, hedonic attributes, goodness and beauty. After use, perceived usability strongly determined how good a product seemed. Beauty depended largely on identification, and the hedonic attributes stayed stable as perceived usability moved with the problems encountered. The two dimensions are separable in judgement, so a usability score is not a summary of the experience.

Practice measured the dimensions the definitions left out

Two inventories show what arrived. Bargas-Avila and Hornbaek (2011) coded 51 publications from 2005 to 2009, reporting 66 empirical studies. They found a shift from work towards leisure and from controlled tasks towards open use. Context of use and anticipated use, the factors the definitions call central, were rarely researched, and emotions, enjoyment and aesthetics were the dimensions most assessed. The methods were mostly qualitative and inherited from usability practice, and many studies used self-developed questionnaires with no items or validation.

Vermeeren, Law, Roto, Obrist, Hoonhout and Vaananen-Vainio-Mattila (2010) collected 96 evaluation methods and classified them by development phase and period of experience. They report that no clear overview had existed, and they name early-stage methods, social and collaborative evaluation, and scientific quality as development needs.

Contested question: has the wider term bought knowledge or spent it?

Hassenzahl (2004) and Hassenzahl and Tractinsky (2006) argue that a narrow focus on products as tools misses what technology use has become. Bargas-Avila and Hornbaek (2011) accept the ambition and audit the record, finding that studies which borrowed usability methods never addressed the factors the definitions make central. The evidence is not evenly weighted. Hornbaek's reviews audit the measures actually used and pool raw correlations, and the widening case rests on a survey of the field and on agenda papers. Neither ISO document prescribes a method, so only the measures can settle it. Hornbaek (2006) and Hornbaek and Law (2007) supply the sharpest sceptical case from inside measurement. If the components of usability do not correlate, widening the construct multiplies what must be measured independently.

What would settle the disagreement is not more discussion of the word. Progress would look like three things. The first is a construct narrow enough to yield a prediction that could fail. The second is instruments with published items and reported reliability, applied at more than one point. The third is convergent evidence, in which two instruments built on different theories agree about the same product. That is the test Hornbaek and Law (2007) applied to usability and found wanting. Without those three, the wider term names a class of concerns the instruments this unit reviews have not yet measured. The strongest counter to this unit's view is that the definitional clause did not change practice. Bangor and colleagues benchmark scale scores across unrelated products, so usability is treated as a comparable product property whatever the standard says.

While it is open, name the measure in the sentence that makes the claim, and treat a user experience score as evidence about one construct rather than about the product. Where the claim is comparative, use an instrument whose items and reliability are published.

Where the argument is heading

The direction of travel is convergence on the instrument. Vermeeren and colleagues (2010) set out the development needs. Bargas-Avila and Hornbaek (2011) ask for items and validations to be published, and Hassenzahl and Tractinsky (2006) asked for experience to be studied over time. Against that, Brooke (2013) shows what happens when an instrument acquires authority without a standard. The standards fixed the words; the unfinished work is the methods sentence that part 11 left out.

The sources this unit rests on

• ISO. (2018). ISO 9241-11:2018. Ergonomics of human-system interaction. Part 11: Usability: definitions and concepts. ISO, Geneva. https://web.archive.org/web/2024/https://www.iso.org/standard/63500.html Defines usability as an outcome of use, not an attribute of a product, and describes no methods. Each qualifier therefore restricts what may be claimed.
• ISO. (2019). ISO 9241-210:2019. Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. ISO, Geneva. https://web.archive.org/web/2025/https://www.iso.org/standard/77520.html Defines user experience as perceptions and responses from use or anticipated use, before, during and after use. It is the wider term with no restricting clause.
• Nielsen, J. (1993). Usability Engineering. Morgan Kaufmann. doi:10.1016/C2009-0-21512-1 Sets usability out as attributes that can be specified and measured, learnability, efficiency, memorability, errors and satisfaction.
• Brooke, J. (1996). SUS: a quick and dirty usability scale. Usability Evaluation in Industry (Jordan, Thomas, Weerdmeester and McClelland, eds.), Taylor and Francis, 207-212. doi:10.1201/9781498710411-35 Introduces ten items producing one score for perceived usability, quick to administer and workable with small samples.
• Brooke, J. (2013). SUS: a retrospective. Journal of Usability Studies 8(2), 29-40. https://web.archive.org/web/2021/https://uxpajournal.org/sus-a-retrospective/ Reports that the scale was developed in 1986, cited in over 1,200 publications, never through a formal standardisation process, and that its nought to one hundred range was a marketing decision.
• Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Human Mental Workload (Hancock and Meshkati, eds.), North-Holland, 139-183. doi:10.1016/S0166-4115(08)62386-9 Develops a workload measure with six subscales combined through a weighting procedure drawn from fifteen pairwise comparisons, and reports that the weighted combination reduced between-subject variability.
• Hassenzahl, M. (2004). The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction 19(4), 319-349. doi:10.1207/s15327051hci1904_2 Participants rated four MP3 player skins: usability strongly determined goodness after use, and beauty depended largely on identification.
• Hassenzahl, M. and Tractinsky, N. (2006). User experience: a research agenda. Behaviour and Information Technology 25(2), 91-97. doi:10.1080/01449290500330331 Proposes a research agenda, noting that the term had become a buzzword promising change without specifying its meaning, and that research divides into three threads.
• Law, E. L. C., Roto, V., Hassenzahl, M., Vermeeren, A. P. O. S. and Kort, J. (2009). Understanding, scoping and defining user experience: a survey approach. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 719-728. doi:10.1145/1518701.1518813 Surveyed 275 researchers and practitioners, most agreeing that user experience is dynamic, context-dependent and subjective, and proposes to delineate it as individual rather than social.
• Bargas-Avila, J. A. and Hornbaek, K. (2011). Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 2689-2698. doi:10.1145/1978942.1979336 Coded 51 publications from 2005 to 2009 reporting 66 studies. It found a shift from work towards leisure and open use, and that context of use and anticipated use were rarely researched.
• Hornbaek, K. (2006). Current practice in measuring usability: challenges to usability studies and research. International Journal of Human-Computer Studies 64(2), 79-102. doi:10.1016/j.ijhcs.2005.06.002 Reviews measures in 180 studies in core HCI journals and proceedings. It reports that 57 per cent measured task completion time, 92 per cent at least one of time, accuracy or satisfaction, and 13 per cent all three.
• Hornbaek, K. and Law, E. L. C. (2007). Meta-analysis of correlations among usability measures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07), 617-626. doi:10.1145/1240624.1240722 Pools correlations among usability measures from 73 studies. Correlations were low, 0.247 between effectiveness and efficiency, 0.196 between efficiency and satisfaction and 0.164 between effectiveness and satisfaction.
• Vermeeren, A. P. O. S., Law, E. L. C., Roto, V., Obrist, M., Hoonhout, J. and Vaananen-Vainio-Mattila, K. (2010). User experience evaluation methods: current state and development needs. Proceedings of the 6th Nordic Conference on Human-Computer Interaction (NordiCHI '10), 521-530. doi:10.1145/1868914.1868973 Collected 96 user experience evaluation methods and classified them by development phase and period of experience studied. The authors report that no clear overview had existed.
• Bangor, A., Kortum, P. T. and Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 24(6), 574-594. doi:10.1080/10447310802205776 Analysed nearly ten years of System Usability Scale data across many products and development stages, concluding that the scale is robust and versatile.
• Bangor, A., Kortum, P. and Miller, J. (2009). Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies 4(3), 114-123. https://web.archive.org/web/20231215212550/https://uxpajournal.org/determining-what-individual-sus-scores-mean-adding-an-adjective-rating-scale/ Added a seven-point adjective scale as an eleventh question to 964 surveys, mapping scores to verbal labels from OK to Excellent.
• Hart, S. G. (2006). NASA-Task Load Index (NASA-TLX): 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50(9), 904-908. doi:10.1177/154193120605000909 Surveys 550 studies in which NASA-TLX was used or reviewed, and reports that its use spread far beyond its original application of aviation, focus of crew complement and language of English.