WCAG and the conformance regime: what a standard can and cannot require

Year 2 / Module 14 Accessibility, inclusive design and assistive technology / Unit 4 / WCAG and the conformance regime: what a standard can and cannot require

WCAG and the conformance regime: what a standard can and cannot require

HCI2106 · Unit 4 of 10

WCAG is a genuinely testable instrument whose reach was set by what could be agreed and measured, so conformance is a floor that under-represents cognitive, language and learning needs.

WCAG and the conformance regime: what a standard can and cannot require

A standard that cannot be tested cannot be enforced, and a standard written to be tested covers only what the test can see. WCAG is the most consequential artefact in digital accessibility. This fourth of ten units argues that its testability is both its achievement and the source of its narrowness. Its success criteria exist because a working group could agree on them and because an evaluator could decide whether they had been met.

Hold the two questions the module set apart: what the standard requires, and what the disabled user experiences. WCAG answers the first, and is candid that it does not answer the second on its own (W3C, 2023). Conformance is a floor poured to the shape of measurement, not of need.

The standard was rebuilt around what an evaluator could decide

WCAG 1.0, published in 1999, was written around checkpoints tied to the web technologies of its day. When the web moved, the standard had to be rebuilt, and the rebuild was an argument about form. Caldwell and colleagues (2008) describe WCAG 2.0 replacing technology-specific checkpoints with technology-neutral success criteria under four principles and three conformance levels. Reid and Snow-Weaver (2008), inside the working group, give the standard's own justification. Criteria had to be testable and technology-neutral to keep pace with the web.

Only the success criteria are normative and testable: four principles at the top, thirteen guidelines underneath, the criteria below those, each carrying a level (W3C, 2023). Nothing can be enforced where the criteria stop.

What the standard requires, and what it declines to

The conformance model is a pass or fail judgement per criterion, at a level, for a defined scope. Level A is the floor, and AA is the level most policy names. WCAG 2.1 added seventeen criteria to 2.0, mainly for mobile, low vision and cognitive needs (W3C, 2018). WCAG 2.2 added nine more, including focus not obscured, dragging movements and target size minimum. It removed 4.1.1 Parsing, now obsolete (W3C, 2023). The standard grows by accretion while its model of conformance holds still. One of the nine added in 2.2 makes the point. Target size minimum requires an interactive target to be at least 24 by 24 CSS pixels, or to carry enough spacing around it, a floor an ageing hand needs as much as a diagnosed one. An evaluator can measure that value from the rendered page without asking anyone what they experienced.

The standard is explicit about its limits, and that admission is this unit's evidence. It states that it addresses some accommodation for learning disabilities and cognitive limitations but not every user need. It records the difficulty in defining criteria for cognitive, language and learning disabilities, including reaching consensus on testability (W3C, 2023). It also states that it is testable with a combination of automated testing and human evaluation. The agreement it could reach about what counts as a measurement bounds its reach.

Conformance is not access: the components argument

The sharpest case against equating conformance with access predates WCAG 2.0 and comes from the working group itself. Chisholm and Henry (2005) argue that web accessibility depends on interchangeable components working together. Those components are the content, the browsers and other user agents, and the authoring tools, with guidelines for each, WCAG, UAAG and ATAG. A system can satisfy a standard governing content alone and still exclude the user. The failure sits in a component the criterion does not govern. HTML defines the alt attribute. The authoring tool has to make it easy to supply, the browser has to expose it, the assistive technology has to speak it, and the user has to know how to retrieve it. If a content management system offers no field for the text, the author cannot supply it at all.

Kelly and colleagues (2005) argued that strict application of the WCAG can damage usability. They proposed a holistic framework in which context of use, audience capability and access environment shape how the criteria are applied. Brajnik (2008) adds that conformance testing and user-based testing answer different questions and detect different problems, so neither alone establishes accessibility. A conformance claim is a partial claim whose scope is the scope of the test behind it.

Conformance covers half of the problems users meet

Power and colleagues (2012) ran task-based evaluations with 32 blind users across 16 websites and logged 1,383 instances of user problems. WCAG 2.0 success criteria covered only 50.4 per cent of those problems. Where a problem was covered, 16.7 per cent of websites had implemented the recommended technique in a way that did not solve it. A site can conform and still leave half of the encountered problems unaddressed. A problem-based list can cover only the problems someone anticipated, so they argued for moving towards design principles.

What actually drives conformance

If the standard's text produced access, the measured gap would be smaller. Yesilada, Brajnik, Vigo and Harper (2012) asked over 300 people to discuss their definitions of accessibility. They found that social rather than economic factors shaped their positions, and that definitions used by standards and regulatory bodies were the most accepted. They found that forcing adoption did not encourage an accessibility ethos, while evidence of benefit to all did.

Inal, Rizvanoglu and Yesilada (2018) give the professional version. Their survey of user experience professionals in Turkey found awareness of accessibility high but knowledge and routine practice shallow. Respondents were unfamiliar with accessibility standards and assistive technologies, and they did not involve disabled people in usability work. Awareness is not the capability to apply.

Contested question: is testability the standard's strength or its ceiling?

The literature disagrees about whether the standard's testable form is what makes it work or what limits it, and both positions draw on the same artefact. The testability-first position holds that the criteria are the reason accessibility became enforceable and procurable at all. Caldwell and colleagues (2008) treat testable, technology-neutral criteria as the fix that let the standard survive the web's change, and Reid and Snow-Weaver (2008) present testability as the enabling property. On this reading the growth from 2.0 to 2.2, as criteria became testable, is how scope increases (W3C, 2018; W3C, 2023).

The testability-as-ceiling position holds that the form of the standard encodes a partial view of disability and therefore produces partial access. Chisholm and Henry (2005) locate access in components the criteria do not govern. Power and colleagues (2012) measure the shortfall at half of the encountered problems. Lewthwaite (2014) argues that the values carried by legislation and standards can be counter-productive outside Western accessibility norms. The disagreement is definitional before it is empirical. Conformance tests a property of a resource, while the unified definition of Petrie, Savva and Power (2015) tests whether a goal was achieved. That definition was built from 50 definitions drawn from 1996 to 2014 in 21 countries.

The two sides rest on evidence of different kinds. The testability-first case rests on the working group's own account of why it chose the criteria, which is testimony about intent rather than a measurement of effect. The ceiling case rests on a task-based evaluation with 32 blind users that logged 1,383 problems and measured how many the criteria covered, and on a component analysis. Only one side reports a rate.

The strongest objection is that the gap has no counterfactual. Power and colleagues measured the problems the criteria missed, and the unit cannot say what an untestable alternative would have caught in their place. A criterion nobody can check cannot be shown to help either, so the measured shortfall is a floor on one form of the standard rather than a demonstration that another form would do better.

The test that would settle it has not been run. The test is a standard that added testable criteria for cognitive, language and learning needs. It would be evaluated for whether users' problem rates fell as well as whether sites passed, against a control given the same guidance as non-testable aims. Power and colleagues (2012) supply the outcome measure and the baseline.

What to do while the dispute is open

While the dispute is open, treat conformance as a floor and report it as one. Then test the tasks users came to do, and record the problems a conformance report cannot see, including the ones an older reader and a slow connection produce.

Where the argument is heading

The named work points at changing the instrument rather than adding to it. The WCAG 3.0 working draft (W3C, 2026) proposes outcome-based guidelines supported by more granular requirements and assertions. Its September 2026 draft rebuilds the conformance model around requirement tags, physical harm, risk, barrier and friction, and reporting tiers, so a site reports progress towards conformance rather than a single level. It states that the final requirements will differ and that several years of work remain. Petrie and colleagues (2015) supply the target such a model would serve: accessibility as achieving goals in context rather than a property a checker confirms. The next unit asks whether a broader, less binary instrument can be measured at all.

The sources this unit rests on

Chisholm, W. and Henry, S. L. (2005). Interdependent components of web accessibility. Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A). doi:10.1145/1061811.1061818 Argues that web accessibility depends on interchangeable components, the content, the user agents and the authoring tools, governed by WCAG, UAAG and ATAG. A content standard governs one component and can be satisfied while the system still excludes the user.
Kelly, B., Sloan, D., Phipps, L., Petrie, H. and Hamilton, F. (2005). Forcing standardization or accommodating diversity? A framework for applying the WCAG in the real world. Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A). doi:10.1145/1061811.1061820 Argues that inappropriate reference to the WCAG in evaluation and policymaking causes practical difficulties, and proposes a holistic framework shaped by context of use and audience capability. It is the early statement that strict conformance and usable access can pull against each other.
Brajnik, G. (2008). Beyond Conformance: The Role of Accessibility Evaluation Methods. Lecture Notes in Computer Science. doi:10.1007/978-3-540-85200-1_9 Examines when conformance testing and user-based testing are each appropriate and what each can detect, arguing that the two answer different questions and that neither alone establishes accessibility. It frames a conformance claim as a partial claim.
Caldwell, B., Cooper, M., Reid, L. G. and Vanderheiden, G. (2008). Web Content Accessibility Guidelines (WCAG) 2.0. W3C Recommendation. https://www.w3.org/TR/WCAG20/ The recommendation that replaced technology-specific checkpoints with technology-neutral success criteria under four principles and three conformance levels, so the guidance could outlive any single technology. It is the unit's history of how the standard became testable.
Reid, L. G. and Snow-Weaver, A. (2008). WCAG 2.0: a web accessibility standard for the evolving web. Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A). doi:10.1145/1368044.1368069 Describes the problems a changed web raised for WCAG 1.0 and the approaches developed to address them in WCAG 2.0, the working group's own account of why testable, technology-neutral criteria were needed. It gives the standard-writers' justification the unit must weigh against the critics.
Power, C., Freire, A., Petrie, H. and Swallow, D. (2012). Guidelines are only half of the story: accessibility problems encountered by blind users on the web. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). doi:10.1145/2207676.2207736 Evaluations with 32 blind users on 16 websites yielded 1,383 instances of user problems, only 50.4 per cent of which were covered by WCAG 2.0 success criteria; where a problem was covered, 16.7 per cent of websites had implemented the recommended technique without solving it. It measures the gap between conformance and experience.
Yesilada, Y., Brajnik, G., Vigo, M. and Harper, S. (2012). Understanding web accessibility and its drivers. Proceedings of the International Cross-Disciplinary Conference on Web Accessibility. doi:10.1145/2207016.2207027 Asked over 300 people to discuss their definitions of accessibility, finding that social rather than economic factors shaped their positions and that definitions used by standards and regulatory bodies were the most accepted. It shows the standard's authority to be institutional.
Lewthwaite, S. (2014). Web accessibility standards and disability: developing critical perspectives on accessibility. Disability and Rehabilitation. doi:10.3109/09638288.2014.938178 A commentary arguing that dominant standards do not respect disability as a complex and culturally contingent interaction, and that the values carried by legislation promote universal, statistical and automated approaches that fix the relation between technology and disability irrespective of context. It warns this can be counter-productive outside Western accessibility conventions.
Petrie, H., Savva, A. and Power, C. (2015). Towards a unified definition of web accessibility. Proceedings of the 12th International Web for All Conference. doi:10.1145/2745555.2746653 Analysed 50 definitions of web accessibility from 1996 to 2014 and 21 countries, extracted six core concepts, and proposed a unified definition in which people can use websites across contexts of use with mainstream and assistive technologies. It gives the unit a goal-centred definition against which conformance is measured.
Inal, Y., Rizvanoglu, K. and Yesilada, Y. (2018). Web accessibility in Turkey: awareness, understanding and practices of user experience professionals. Universal Access in the Information Society. doi:10.1007/s10209-017-0603-3 An online survey of user experience professionals in Turkey found awareness of accessibility high but knowledge and routine practice shallow, with respondents unfamiliar with accessibility standards and assistive technologies and not considering accessibility in projects. Awareness of a standard does not produce conformance.
World Wide Web Consortium (2018). Web Content Accessibility Guidelines (WCAG) 2.1. W3C Recommendation. https://www.w3.org/TR/WCAG21/ Added seventeen success criteria to WCAG 2.0, mainly for mobile, low vision and cognitive needs, while keeping the same conformance model. It is the reference for most contemporary policy that names WCAG.
World Wide Web Consortium (2023). Web Content Accessibility Guidelines (WCAG) 2.2. W3C Recommendation. https://www.w3.org/TR/WCAG22/ States that it addresses some accommodation for learning disabilities and cognitive limitations but not every user need, and records the difficulty of reaching consensus on cognitive, language and learning criteria. It is the primary standard the unit reads.
World Wide Web Consortium (2026). W3C Accessibility Guidelines (WCAG) 3.0. W3C Working Draft, September 2026. https://www.w3.org/TR/wcag-3.0/ Proposes outcome-based guidelines supported by more granular requirements and assertions, and a conformance model built on requirement tags and reporting tiers. It states that the draft will change and that several years of work remain, so it carries the unit's claim that the conformance model itself is under revision.

Check your understanding & apply it · 4 questions, 1 essay

1A vendor claims full accessibility after its page passes all automated checks for WCAG 2.2 AA, though a captcha on the page cannot be completed with a screen reader. Which diagnosis does the unit support?

2A working group proposes a success criterion requiring that a form not place unreasonable cognitive demands on the user. What does the unit's account of the standard predict?

3A regulator asks whether replacing testable criteria with outcome-based aims would raise the access users receive. Which reading does the unit's evidence support?

4A browser drops a landmark role that the page marks up correctly, so the page structure is not announced, though the content meets every applicable success criterion. Which diagnosis does the unit's components argument support?

5Essay A public body publishes an accessibility statement claiming WCAG 2.2 AA conformance, supported by an automated scanning tool that reported no failures. A panel of disabled users then reports that they cannot complete the body's online application form with a screen reader. Assess the conformance claim. In your answer, state what the claim does and does not establish, evaluate the evidence behind it, give the strongest argument the body could make in its defence, and specify what further evidence would settle whether the form is accessible. Name the unit's sources at each step and use the numbers it reports.

Begin by separating the two claims the statement runs together. A conformance claim is a claim about a resource against a set of testable criteria; it is not a claim that a person can achieve a goal. Petrie and colleagues (2015) analysed 50 definitions of web accessibility and proposed one centred on people being able to use websites across contexts of use with mainstream and assistive technologies, which is a goal-centred definition rather than a property a checker confirms. On that definition the body has not shown what it asserts. The evidence behind the claim is weak for the same reason. An automated scan can test only the criteria a tool can decide, and the standard itself describes accessibility as testable with a combination of automated testing and human evaluation (W3C, 2023), so a clean scan is a partial result presented as a complete one. Brajnik (2008) explains why the two methods are not interchangeable: conformance testing and user-based testing answer different questions and detect different problems. The panel's report is the user-based half the body omitted. The body could defend itself by narrowing its claim, and the arguments are real. It can say that conformance is what procurement and policy demand, that a verifiable floor is better than an unverifiable aspiration, and that Chisholm and Henry (2005) show access depends on several components, so a failure in the content or in the assistive technology is not necessarily a failure of its content. Caldwell and colleagues (2008) and Reid and Snow-Weaver (2008) support the position that testable criteria are what made the standard enforceable at all. What settles the question is evidence the statement did not supply: task-based evaluation with screen reader users on the form, recording whether they completed the task, where they stopped and which of the failures fall inside the criteria. Power and colleagues (2012) provide the design and the baseline, having logged 1,383 user problems with 32 blind users and found only 50.4 per cent covered by WCAG 2.0 success criteria, with 16.7 per cent of websites implementing a recommended technique that did not solve the problem. If the panel's failures fall inside the criteria, the scan was inadequate; if they fall outside, the standard was, which is the outcome Lewthwaite (2014) predicts when universal and automated approaches are applied irrespective of context. A defensible statement should therefore report what was tested, how, and by whom, and should not translate conformance into access.