Conformance is a floor: WCAG, the user and the automated checker

Accessibility is the one design property here that arrives written down as a numbered, testable list. WCAG 2.2 is a W3C Recommendation, and it states its criteria as testable statements under thirteen guidelines and four principles (W3C, 2023). That gives a designer a target someone else can check. The target is a floor, not a definition. The standard says so of itself, and the measurement literature says it more sharply. The criteria do not name about half of what users meet, and automated checkers see only a slice of what the criteria require. The strongest counter to this unit's thesis is that an enforced floor is worth more than an unenforceable definition. Kous and colleagues (2020) found a procurement-linked standard raised full compliance among Slovenian municipalities from none to a third within a year. This unit asks what standard holds that work to account. The next two ask what the page costs the reader, and what it can do to them.

Conformance is a claim about a document against a list

WCAG 2.2 arranges its guidance in layers: four principles, thirteen guidelines beneath them, and testable success criteria beneath those (W3C, 2023). Conformance means satisfying the success criteria of a level. The standard says its criteria are testable with a combination of automated testing and human evaluation, and the same document concedes its boundary: its abstract says following the guidelines will not address every user need, and its introduction records difficulty in agreeing criteria for cognitive, language and learning disabilities, and no criterion names a reader's age. A conformance claim is therefore about a document against a list, not a person against a task.

Work the arithmetic on two greys. The criterion for normal text is a contrast ratio of at least 4.5 to 1 against the background, computed from the relative luminance of the two colours (W3C, 2023). On a white background the grey #767676 reaches 4.54 to 1 and passes. One step darker, #777777, reaches 4.48 to 1 and fails. Two shades most readers cannot reliably tell apart sit on opposite sides of the line. That is what makes the criterion testable, and a line rather than a spectrum.

The criteria do not name about half of what users meet

The strongest evidence against equating the two is a user study. Power and colleagues (2012) ran task-based evaluations with 32 blind users on 16 websites, logging 1,383 user problems. Only 50.4 per cent of those problems were covered by a WCAG 2.0 success criterion at all. Where a site had implemented a technique the guidelines recommended, it still failed to solve the problem in a reported 16.7 per cent of cases. That is one study on 16 sites, and mapping a problem onto a criterion takes judgement. A task-level measure is nevertheless stronger evidence than an audit.

The instrument decides the number, and the ceiling has fallen

If conformance is measurable, the measurement deserves equal scrutiny. Vigo, Brown and Conway (2013) benchmarked six state-of-the-art tools against WCAG 2.0 on coverage, completeness and correctness. Coverage was narrow: at most 50 per cent of the success criteria were checked by any tool of the day. Completeness is the share of violations present that a tool found, and it ranged from 14 to 38 per cent. Tools with higher completeness had lower correctness, between 66 and 71 per cent, because catching more meant more false positives. One criterion in two will not be analysed at all. Of those analysed, perhaps four in ten will be caught, with false positives on top.

The ceiling has fallen since. Fischer, Lundell and Gamalielsson (2025) ran six engines over Swedish public sector sites: each reported about a sixth of the success criteria, the six together 39 per cent, and 31 of 78 criteria none at all.

Kelsey-Adkins and Thompson (2022) ran four command-line checkers over the same pages and compared them with Gwet's alpha coefficient, a measure of agreement between raters. Agreement was very poor, so an audit result is a property of the instrument. The WebAIM Million (2026) ran the WAVE engine over the top one million home pages. It detected WCAG 2 failures on 95.9 per cent of them, up from 94.8 per cent, with six failure types accounting for 96 per cent of all errors found. Its authors warn that not every conformance failure can be detected automatically, so absent errors do not mean a page is accessible.

What automation can repair, and what it cannot

The harder question is whether a script can repair a page at run time. Hartman and Gorichanaz (2026) built two sites with different layouts and planted known WCAG violations. They added three leading overlays, and tested the sites automatically and by hand, with keyboard and screen reader, across devices and platforms. Automated testing reported improved accessibility, but manual testing found numerous severe errors surviving behind a report of full compliance. The overlays corrected only moderate errors with simple solutions, such as alternative text and form field labelling.

The Overlay Fact Sheet sets out the case against run-time repair as a position rather than a result. Its argument is structural: conformance means meeting the requirements of a standard, the products cannot repair all possible issues, so they cannot deliver conformance (Groves, n.d.). Karl Groves published it and more than a thousand named practitioners endorsed it. It lists repairs it calls unreliable, among them text alternatives, form labels, error handling and keyboard access. It also argues that detecting assistive technology reveals that a user has a disability. The sheet is advocacy rather than a controlled study, and it states one side most clearly.

The method and the sample decide which pages you find

One more layer sits under the tool: how the pages audited were chosen. Hambley and colleagues (2025) argue that selecting pages for evaluation is unsystematic and resource-intensive, and they propose treating a website as a population to be sampled. Their framework clusters pages that share accessibility characteristics and selects representatives from each cluster. It adds six metrics to the WCAG Evaluation Methodology, and server logs, they report, gave better coverage than crawling. A site-wide conformance percentage is therefore a statistic computed from a sample of pages, whether or not the audit declares which pages those were.

Contested question: what conformance is for, and whether a tool can deliver it?

The disagreement is not about whether WCAG works but about what it is for. On one side, Power and colleagues (2012) show a task-level measure and a conformance measure diverging on the same sites. Olalere and Lazar (2011) show a compliance claim coexisting with an inaccessible site. They evaluated 100 United States federal home pages with human and automated methods, and found the pages still inaccessible and the statements attached to them saying nothing useful. On the other side, a standard is the only instrument that can be contracted for, audited and enforced. Kous and colleagues (2020) supply the counter-evidence. They evaluated 189 Slovenian municipal websites in 2017 and again in 2018, around the adoption of European Standard EN 301 549. None was fully compliant in 2017, and 62, or 33 per cent, were by the end of 2018, so the standard changed what organisations built. Both positions can hold at once, so the dispute is about priority rather than truth. What would settle it is one study measuring both quantities on the same sites over time: conformance level and disabled users' task success, before and after a procurement requirement. If conformance rose and task success did not, the floor is real and insufficient. If both rose, conformance is a workable proxy.

The second dispute is sharper because the sides make incompatible claims about the same products. Hartman and Gorichanaz (2026) found severe errors surviving behind a "100 per cent compliant" report. The Overlay Fact Sheet argues that run-time repair cannot reach conformance by definition rather than by measurement (Groves, n.d.). Against them stand the vendors, whose marketing is the object of study, and a genuine set of small successes. Automation is strong where a problem is local and its repair is checkable. It is weak where the repair needs to know what the content means or how the page will behave. What would settle it is a pre-registered test using errors of known severity, in which the repairs are verified by disabled users on real tasks and the tool's own compliance report is scored for false claims.

While the question is open, a designer should treat conformance as a floor and report it as one. Meet the criteria, then test the tasks the criteria do not name with disabled users and older readers, and record what the manual pass found that the tool did not. Declare the sampling frame behind any site-wide percentage, and state in the accessibility statement which method produced it. Do not accept a vendor's compliance claim as evidence of access.

Where the argument is heading

Two directions are visible in named work. The first is that the standard itself is being rebuilt around the gaps rather than the criteria. The WCAG 3.0 Working Draft, updated on 10 September 2026, restructures the guidance as user needs, requirements and assertions, and proposes new tiers of conformance and reporting. It states that WCAG 2 remains in force with several years of work still to come (W3C, 2026). The second is that the audit is becoming a sampling problem rather than a page problem, the direction Hambley and colleagues (2025) set out. On that view a defensible accessibility claim will be about a population of pages rather than a page.

The sources this unit rests on

• W3C (2023). Web Content Accessibility Guidelines (WCAG) 2.2. W3C Recommendation, revised 2024. https://www.w3.org/TR/WCAG22/ The recommendation states the four principles, thirteen guidelines and testable success criteria that a conformance claim is measured against, and records the difficulty of agreeing criteria for cognitive, language and learning disabilities. Its abstract concedes that it will not address every user need.
• Power, C., Freire, A. P., Petrie, H. and Swallow, D. (2012). Guidelines are only half of the story. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/2207676.2207736 Task-based evaluations with 32 blind users on 16 websites logged 1,383 user problems; only 50.4 per cent fell under a WCAG 2.0 criterion, and recommended techniques still failed in a reported 16.7 per cent of cases. The central evidence that the two measures diverge.
• Olalere, A. and Lazar, J. (2011). Accessibility of U.S. federal government home pages. Government Information Quarterly 28(3). doi:10.1016/j.giq.2011.02.002 One hundred United States federal home pages were evaluated with human and automated methods and their accessibility statements analysed. The pages remained inaccessible and the statements often said nothing useful.
• Kous, K., Kuhar, S., Pavlinek, M., Hericko, M. and Pusnik, M. (2020). Web accessibility investigation of Slovenian municipalities' websites before and after the adoption of EN 301 549. Universal Access in the Information Society 20. doi:10.1007/s10209-020-00732-9 The 189 municipal websites were evaluated in 2017 and again in 2018 around the standard's adoption. No site was fully compliant in 2017; by the end of 2018, 62 sites, 33 per cent, were. A standard tied to procurement changed what organisations built.
• Vigo, M., Brown, J. and Conway, V. (2013). Benchmarking web accessibility evaluation tools. Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. doi:10.1145/2461121.2461124 Six tools were benchmarked against WCAG 2.0 on coverage, completeness and correctness. Coverage reached at most 50 per cent of the success criteria, completeness ranged from 14 to 38 per cent, and correctness from 66 to 71 per cent. It carries the claim that automation sees a fraction of what conformance requires.
• Fischer, T., Lundell, B. and Gamalielsson, J. (2025). Coverage of web accessibility guidelines provided by automated checking tools. Universal Access in the Information Society, 24(4), 3615-3637. doi:10.1007/s10209-025-01263-x Six engines over Swedish public sector sites each reported violations for about a sixth of the success criteria, 39 per cent between them, and 31 of 78 criteria for none. It lowers the coverage ceiling the 2013 benchmark set.
• Kelsey-Adkins, E. R. and Thompson, R. H. (2022). Inter-rater Reliability of Command-Line Web Accessibility Evaluation Tools. Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. doi:10.1145/3517428.3550395 Four command-line checkers were run over the same pages and compared with Gwet's alpha coefficient, a measure of agreement between raters. Agreement was very poor. An automated result is a property of the instrument.
• WebAIM (2026). The WebAIM Million: the 2026 report on the accessibility of the top 1,000,000 home pages. https://webaim.org/projects/million/ The WAVE engine assessed the top one million home pages and detected WCAG 2 failures on 95.9 per cent of them, up from 94.8 per cent in 2025, with six failure types accounting for 96 per cent of all errors found. Its authors state that absence of detected errors does not mean a page is accessible.
• Hartman, P. and Gorichanaz, T. (2026). Future Imperfect? Evaluating Whether AI-Powered Web Accessibility Overlays are Worth the Hype. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3805689.3812259 Two sites were built and planted with known WCAG violations, then fitted with three leading overlays and tested automatically and manually across devices. Moderate errors with simple solutions were corrected while numerous severe errors remained, even where an overlay reported the page as fully compliant.
• Groves, K. (n.d.). Overlay Fact Sheet. https://overlayfactsheet.com/en/ The sheet argues that run-time overlays sit in the wrong part of the stack and cannot deliver conformance by definition, because conformance means meeting all requirements of a standard. It lists repairs it calls unreliable, among them text alternatives, form labels, error handling and keyboard access. It is advocacy, not a controlled study.
• Hambley, A., Yesilada, Y., Vigo, M. and Harper, S. (2025). Optimising web accessibility evaluation: Population sourcing methods. International Journal of Human-Computer Studies 198. doi:10.1016/j.ijhcs.2025.103472 The authors argue that choosing pages for evaluation is unsystematic, and propose sampling a site as a population, clustering pages and selecting representatives, with server logs covering more than crawling. Their framework adds six metrics to the WCAG Evaluation Methodology.
• W3C (2026). W3C Accessibility Guidelines (WCAG) 3.0. W3C Working Draft, 10 September 2026. https://www.w3.org/TR/wcag-3.0/ The draft successor restructures the guidance as user needs, requirements and assertions and proposes new tiers of conformance and reporting. It states that WCAG 2 remains in force and that several years of work remain.