Quality attributes as requirements: accessibility, performance and observability

The tenth and last unit turns to the properties that cut across every unit before it. Some requirements no feature can satisfy and no later audit can install. Accessibility, performance and observability are properties of the whole product: they hold across every screen, every request and every failure, or they do not hold at all. The thesis of this unit is that such properties are requirements in the ordinary sense. They arrive as testable acceptance criteria attached to the work that creates them, not as inspections performed after it ships.

A quality attribute is a property of the whole, which is why the word non-functional misleads

Glinz (2007) surveyed definitions of the non-functional requirement. He found that after twenty years the community still had no consensus on what it is, or how to elicit, document and validate it, while agreeing that it matters. He argues that the residual definition, everything that is not functional, is incoherent. He proposes classifying a requirement by whether it describes a system's behaviour or an attribute of it.

Mairiza, Zowghi and Nurmuliani (2010) reached the same verdict. Their systematic analysis across definition, types and domain found no general agreement, while cataloguing recurring types. Two systematic reviews agreeing is strong evidence that the vocabulary is unsettled.

Chung, Nixon, Yu and Mylopoulos (2000) supply the constructive half. Their NFR Framework treats a quality attribute as a softgoal. The framework refines it and inter-relates it with other softgoals in a graph that captures tradeoffs and synergy, and a team satisfies it sufficiently, not absolutely. If a team can buy accessibility at the cost of performance, the two are one decision, and a decision belongs in the work.

Accessibility arrives as a specification, and the specification has a human half

WCAG 2.2 is a W3C Recommendation of success criteria written as testable statements that are not technology-specific; it was first published on 5 October 2023 and reissued on 12 December 2024. The criteria are organised under four principles: perceivable, operable, understandable and robust (World Wide Web Consortium, 2024). Understandable is the principle a reader with low literacy depends on most. The standard also states its limits: it will not address every user need, and it is tested with a combination of automated and human evaluation.

The standard is also versioned, so conformance is a target rather than a state. A working group initiated WCAG 2.1 to improve guidance for users with cognitive or learning disabilities, low vision, and disabilities on mobile devices. It adds criteria including Orientation, Reflow and Text Spacing (World Wide Web Consortium, 2018). A team claiming conformance has to name a version it can be held to.

Conformance is necessary, and measurably not sufficient

The case that a criterion is not the outcome rests on measurement. Power, Freire, Petrie and Swallow (2012) had 32 blind users attempt tasks on 16 websites, and they collected 1,383 instances of user problems. WCAG 2.0 criteria covered only 50.4 per cent of them. For covered problems, 16.7 per cent of websites had implemented the recommended techniques without solving the problem. A conformance claim describes a subset of the barriers, and occasionally one that is still there.

Vigo, Brown and Conway (2013) measured the automation rather than the standard. Six tools showed coverage of at most 50 per cent of success criteria, completeness between 14 and 38 per cent, and correctness between 66 and 71 per cent. Reliance on automated tests alone, they conclude, means one of two criteria is never analysed, and of those analysed only about four in ten are caught.

The longitudinal record is no better. Hanson and Richards (2013) examined more than 100 top-traffic and government websites across the 14 years from 1999 to 2012. They found violations still high, although the sites improved. They attribute the improvement plausibly to changing web technologies rather than to attention to accessibility.

Performance is a design property with recurring failure modes

Woodside, Franks and Petriu (2007) set out the agenda of software performance engineering. They distinguish early-cycle predictive, model-based work from late-cycle measurement, and they argue that the two must converge to cover the whole development cycle. Performance, they write, is a pervasive quality that everything affects. Waiting to measure it is therefore waiting until the design has already decided it.

Jin, Song, Shi, Scherpelz and Lu (2012) give the empirical case that performance defects are systematic. They studied 110 performance bugs sampled from five open-source suites: Apache, Chrome, GCC, Mozilla and MySQL. They extracted efficiency rules from 25 patches. A detector built on those rules found 332 previously unknown performance problems, 219 of them by applying rules across applications. A failure mode that recurs across unrelated products can be written down as a criterion.

Observability is a requirement the product is built to meet

The third attribute has the thinnest research base and the clearest practitioner account. Beyer, Jones, Petoff and Murphy (2016) define three terms. A service level indicator is a quantitative measure of some aspect of a service. A service level objective is a target value or range for it. A service level agreement is a commitment with consequences, and the authors treat latency, error rate and availability as measures of reliability. An objective with a number in it is a requirement.

An objective of 99.9 per cent availability over a thirty-day month permits 43.2 minutes of unavailability. That is a quantity a team can spend deliberately on a risky release, and then account for when it is gone (Beyer and others, 2016). Stated as an intention rather than a budget, the same sentence commits nobody.

Yuan, Park and Zhou (2012) show that the reporting itself must be designed. They studied log messages in four large open-source projects. Logging was pervasive but arbitrary, with messages often wrong on the first attempt and repaired later as afterthoughts. A checker built from one finding detected 138 new problematic logging statements, 24 confirmed and fixed.

The practice this thesis indicts

If the argument were wrong, it would be wrong for a reason visible in practice. Ameller, Ayala, Cabot and Franch (2012) interviewed 13 software architects about how non-functional requirements are decided, documented and validated. The architects usually elicited the requirements themselves, and they usually did not document them. Performance and usability were the most important technical attributes, and compliance was rarely verified. The gap is not ignorance of the attributes. It is the absence of written, testable criteria.

Contested question: can a quality attribute be specified before the product exists, or only measured afterwards?

The disagreement is not whether quality attributes matter. It is whether a criterion written in advance can stand for the outcome. Chung and others (2000) hold the design-time position, trading softgoals off before construction. Woodside and others (2007) extend it to performance by predicting from models. WCAG 2.2 is the strongest case, a normative document of testable criteria existing before any particular product (World Wide Web Consortium, 2024).

The measured camp reads the same facts the other way. Power and others (2012) found that half of the problems blind users met lay outside the criteria. Vigo and others (2013) found that tools analyse at most half the criteria and catch only part of those. A tool-checked criterion is therefore a claim about the tool as much as about the user. Glinz (2007) and Mairiza and others (2010) show the vocabulary is loose enough for two teams to agree on the words and still disagree on the threshold.

One study would settle it, and it would treat each attribute as a first-class requirement. Record the criterion and its threshold when the work is written, and record whether the team met it at each release. Then measure the user-facing outcome independently, whether task success for disabled users, latency on target devices, or time to detect a failure. Power and others (2012) and Vigo and others (2013) predict divergence. Iannuzzi and others (2023) show the measurement half is now feasible at a scale that makes such a study affordable.

What to do while the dispute is open

The evidence is stronger on the measured side. The design-time case is largely normative: a framework for trading softgoals (Chung and others, 2000), a research agenda (Woodside and others, 2007) and a standard its own authors say will not cover every need (World Wide Web Consortium, 2024). The measured side has task trials with 32 blind users and a benchmark of six tools (Power and others, 2012; Vigo and others, 2013). The strongest objection to this unit's thesis is that a criterion fixed early can freeze a threshold the user experience later contradicts. While the dispute is open, a designer should attach each attribute to a criterion and to an independent outcome check, and let the check overrule the criterion.

Where the argument is heading

The direction of travel is instrumentation at the scale at which the attributes are delivered. Iannuzzi and others (2023) validated about 2.7 million pages of Italy's public administration sites, whose readers include older citizens. Yuan and others (2012) built a checker from a logging study. Power and others (2012) call for a design-principle rather than problem-based approach to accessibility, the movement this unit argues for across all three.

The sources this unit rests on

• World Wide Web Consortium (2024). Web Content Accessibility Guidelines (WCAG) 2.2. W3C Recommendation, first published 5 October 2023 and reissued 12 December 2024. https://www.w3.org/TR/WCAG22/ The normative specification of testable, technology-independent success criteria under four principles, conceding that it will not address every user need and needs both automated and human evaluation. It is the unit's instance of a specified quality attribute.
• World Wide Web Consortium (2018). Web Content Accessibility Guidelines (WCAG) 2.1. W3C Recommendation, 5 June 2018, reissued 21 September 2023, 12 December 2024 and 6 May 2025. https://www.w3.org/TR/WCAG21/ The version initiated to improve guidance for users with cognitive or learning disabilities, low vision, and disabilities on mobile devices, adding criteria such as Orientation, Reflow and Text Spacing. Conformance has to name a version.
• Power, C., Freire, A. P., Petrie, H. and Swallow, D. (2012). Guidelines Are Only Half of the Story. In CHI 2012. New York: ACM, 433-442. doi:10.1145/2207676.2207736 A task-based study with 32 blind users on 16 websites producing 1,383 user problems, of which only 50.4 per cent were covered by WCAG 2.0 criteria and the recommended techniques failed in 16.7 per cent of websites. Conformance and accessibility differ.
• Hanson, V. L. and Richards, J. T. (2013). Progress on Website Accessibility? ACM Transactions on the Web 7(1), Article 2. doi:10.1145/2435215.2435217 A longitudinal analysis of more than 100 top-traffic and government websites from 1999 to 2012 that found violations high although the sites improved, attributing the improvement plausibly to changing web technologies rather than accessibility. It is the measured state of practice.
• Vigo, M., Brown, J. and Conway, V. (2013). Benchmarking Web Accessibility Evaluation Tools. In W4A 2013. New York: ACM, Article 1. doi:10.1145/2461121.2461124 A benchmark of six automated tools that found coverage of at most 50 per cent of success criteria, completeness between 14 and 38 per cent, and correctness between 66 and 71 per cent. It quantifies the limit of automated accessibility testing.
• Glinz, M. (2007). On Non-Functional Requirements. In RE 2007. Piscataway, NJ: IEEE, 21-26. doi:10.1109/RE.2007.45 A survey of definitions that found no consensus on what a non-functional requirement is or how to elicit, document and validate it, arguing that everything not functional is an incoherent definition. It provides the definitional clarity the thesis needs.
• Chung, L., Nixon, B. A., Yu, E. and Mylopoulos, J. (2000). Non-Functional Requirements in Software Engineering. Boston, MA: Kluwer Academic. https://link.springer.com/book/10.1007/978-1-4615-5269-7 The book presents the NFR Framework, in which quality attributes are softgoals refined and inter-related in graphs capturing tradeoffs and synergy and satisfied sufficiently rather than absolutely. It is the foundational text treating attributes as design objects.
• Woodside, M., Franks, G. and Petriu, D. C. (2007). The Future of Software Performance Engineering. In FOSE 2007. Piscataway, NJ: IEEE, 171-187. doi:10.1109/FOSE.2007.32 A research agenda distinguishing early-cycle predictive, model-based performance work from late-cycle measurement and arguing that the two must converge across the development cycle, since performance is affected by every layer. It is the design-time position for performance.
• Jin, G., Song, L., Shi, X., Scherpelz, J. and Lu, S. (2012). Understanding and Detecting Real-World Performance Bugs. In PLDI 2012. New York: ACM, 77-88. doi:10.1145/2254064.2254075 A study of 110 performance bugs sampled from five open-source suites, from which rules were extracted from 25 patches; a detector built on them found 332 previously unknown performance problems, 219 across applications. Performance defects are systematic and specifiable.
• Beyer, B., Jones, C., Petoff, J. and Murphy, N. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA: O'Reilly Media. https://sre.google/sre-book/table-of-contents/ The practitioner account defining service level indicators, objectives and agreements and treating reliability as a target, with latency, error rate and availability among its measures. It is the source of the observability strand, though its evidence is operational experience.
• Ameller, D., Ayala, C., Cabot, J. and Franch, X. (2012). How Do Software Architects Consider Non-Functional Requirements? In RE 2012. Piscataway, NJ: IEEE, 41-50. doi:10.1109/RE.2012.6345838 Thirteen interviews found architects usually elicited non-functional requirements themselves, usually did not document them and validated them only partially, with compliance rarely verified. It supplies the practice evidence the prescription answers.
• Mairiza, D., Zowghi, D. and Nurmuliani, N. (2010). An Investigation into the Notion of Non-Functional Requirements. In SAC 2010. New York: ACM, 311-317. doi:10.1145/1774088.1774153 A systematic analysis of the literature across definition, types and domain that found no general consensus on the notion while cataloguing recurring types. It corroborates Glinz and shows the vocabulary is unsettled.
• Yuan, D., Park, S. and Zhou, Y. (2012). Characterizing Logging Practices in Open-Source Software. In ICSE 2012. Piscataway, NJ: IEEE, 102-112. doi:10.1109/ICSE.2012.6227202 A study of log messages in four large open-source projects that found logging pervasive but arbitrary, with messages often wrong on the first attempt; a checker detected 138 new problematic logging statements, 24 confirmed and fixed. Observability must be designed, not assumed.
• Iannuzzi, N., Manca, M., Paterno, F. and Santoro, C. (2023). Large Scale Automatic Web Accessibility Validation. In GoodIT 2023. New York: ACM, 307-314. doi:10.1145/3582515.3609549 The authors extended an accessibility validation tool and validated about 2.7 million pages of Italian public administration sites. It shows what automated monitoring can detect at population scale.