Measuring accessibility: automated tools, audits and the conformance-experience gap

An accessibility audit reports a number, and the number is read as a fact about the user, not about whether a reader in a second language, or late in life, could finish it. This fifth of ten units argues that the number is a fact about an instrument and a judge, and that few audits name either. Automated testing finds only a minority of the barriers between a disabled user and a task, and expert judgement varies with who holds it. A clean audit is evidence about the tool and the evaluator before it is evidence about the person.

Keep two questions apart: a conformance result answers whether a resource meets testable criteria, a measurement of accessibility whether a person achieves a goal.

The instrument problem: what a tool can see

The strongest case against trusting a scan is a benchmark. Vigo, Brown and Conway (2013) tested six state-of-the-art tools against WCAG 2.0. Coverage was narrow: at most 50 per cent of success criteria were covered, completeness ranged from 14 to 38 per cent, and the tools with the highest completeness produced lower correctness, 66 to 71 per cent, because catching more violations brings more false positives. Their summary is the thesis: automated tests alone leave one of every two criteria unexamined, and of those examined only about four in ten are caught. That verdict describes rule-based tools: Lopez-Gil and Pereira (2024) automated three criteria the field treats as manual, with detection above 85 per cent.

Brajnik (2008) compared a barrier walkthrough with conformance testing as the control, with twelve novice evaluators. The walkthrough improved correctness by 9 to 60 per cent and led evaluators to identify 45 to 127 per cent more correctly judged barriers, at slightly lower reliability, 0.62 against 0.69. The method changes what the evaluator finds and how consistently, so it belongs in the result.

The judge problem: expertise as part of the instrument

If the method is part of the result, so is the person holding it. Brajnik, Yesilada and Harper (2011) ran a barrier walkthrough with 19 expert and 57 non-expert judges: validity and reliability dropped when non-experts judged, and more judges produced less reproducible results.

Mankoff, Fait and Juang (2005) built EASE to simulate low vision, colour blindness and motor impairment, comparing simulated performance with real users, treating simulation as a way to find problems early in design rather than a substitute for user testing, and their word-prediction study found the technique most useful at 5 to 8 words per minute.

What the audits find: failure is the rule

Kuzma (2010) applied an automated tool to 130 UK parliamentary e-government websites. The vast majority failed the criteria despite the duty set by the Disability Discrimination Act of 1995, errors concentrated in recurring checkpoints. Acosta-Vargas and colleagues (2016) assessed twenty universities' websites against WCAG 2.0, and most did not reach an acceptable level of compliance: academic prestige was no guide to accessibility.

The longest-running measurement is more careful about attribution. Hanson and Richards (2013) analysed over 100 top-traffic and government websites in the United States and the United Kingdom from 1999 to 2012, using automated tests of WCAG 2.0 Level A criteria. Violation rates were high but improved, government sites doing better, and the authors caution that part of the change may reflect web technology and coding practice.

Campoverde-Molina, Lujan-Mora and Valverde (2020) reviewed 25 empirical studies of educational websites: 20 used automated tools alone, two used real users and three combined methods, and every site analysed needed corrections. Csontos and Heckl (2021) found that of 25 Hungarian government websites none fully met WCAG, many failing accessibility, usability and security together. The narrowest instrument dominates the field's measurement.

Conformance and experience disagree

Petrie and Kheir (2007) studied two websites with six blind and six sighted participants. Their problem sets overlapped by roughly 15 per cent, blind participants rated the problems significantly more severely on one site, and the participants did not agree with the importance ratings the guidelines gave the same problems.

Romen and Svanaes (2012) tested whether conformance to WCAG 1.0 and 2.0 predicted usable access for disabled participants and found the relationship weak: many compliant pages still posed barriers, and visually impaired users reported redundant, indistinguishable links that WCAG 2.0 does not address.

The most precise version separates the two measures. Aizpurua, Harper and Vigo (2016) related perceived accessibility and WCAG 2.0 conformance to 35 user-experience attributes. Perceived accessibility correlated significantly with 27 of them; conformance with only three hedonic attributes.

The metric and the standard both lag what they measure

A metric reduces the audit to a scalar, and the literature warns about the reduction. Vigo and Brajnik (2011) applied seven published metrics to more than 1,500 web pages within a framework of validity, reliability and sensitivity, and hold that a score which does not report its validity is a number, not a measurement. Onay Durdu and Soydemir (2022) reviewed 30 later metric studies, identifying fourteen metrics and recording each one's guideline set and validation with users.

A second limit is the platform. A measurement regime is only as current as the platform it was written against. Alajarmeh (2021) worked with 16 participants with visual impairment on mobile touchscreen devices and compiled 34 major problems, some lacking success criteria in WCAG 2.1 and others sitting at conformance levels that did not match their severity. A new form of use spends its early years unmeasured.

Contested question: is a clean automated audit worthless, or the only evidence that scales?

The literature disagrees about what a clean automated scan is worth, and the disagreement concerns scale as much as accuracy. The critics hold that a clean scan is close to worthless as evidence about the user. Vigo, Brown and Conway (2013) show that tools cover at most half the criteria and catch about four in ten of those they examine; Brajnik (2006) shows that the method determines the verdict, and Brajnik, Yesilada and Harper (2011) that the judge does. A scan pre-empts both variables and reports neither.

The pragmatists hold that automated testing is the only method that scales and runs in a build pipeline, so the fault lies in the claim rather than the tool. Hanson and Richards (2013) measured fourteen years of change across more than a hundred sites, and Campoverde-Molina and colleagues (2020) show that twenty of twenty-five educational studies rested on tools. A third position treats the dispute as a category error: Brajnik (2006) argues that conformance review and barrier-based evaluation answer different questions, and Petrie and Kheir (2007) supply the outcome a claim about either one needs, namely whether the problem occurred and how severely the user rated it.

The two sides rest on different objects. The critics rest on a benchmark of six tools that measured coverage, completeness and correctness, and two controlled experiments about method and judge. The pragmatists rest on a longitudinal analysis of over a hundred sites and a review of 25 studies, which measure what the instrument has produced, not how well it performs.

The strongest objection is that no instrument has replaced the scan. The unit can show that an automated test misses barriers, and it names no method that finds them at the same scale. It also argues that the field's own evidence about failure is built on the instrument it distrusts, so the critique removes the baseline as well as the number.

The study that would settle it has not been run for a modern tool and a modern standard. It would take a named tool, report its coverage, completeness and correctness on a defined scope, cross those numbers against task-based outcomes for disabled users on the same pages, and count false negatives as named user failures. Vigo, Brown and Conway (2013) supply the instrument and Romen and Svanaes (2012) the outcome.

What to do while the dispute is open

While the dispute is open, publish what the tool covered and what it could not see beside the count, pair the scan with a task-based test on the pages that matter, run in later life, over a slow connection and on a shared device, and count each miss as a named user failure.

Where the argument is heading

The named work points towards making the user's success the measured quantity rather than an inferred one. Vigo and Brajnik (2011) set the properties a defensible metric needs, and Onay Durdu and Soydemir (2022) record fourteen metrics proposed in a decade and how far each was validated. Aizpurua, Harper and Vigo (2016) point at perceived accessibility as the measure that tracks experience. The next unit asks where assistive technology fits, because the system a disabled user operates includes the technology they choose and sometimes abandon.

The sources this unit rests on

• Mankoff, J., Fait, H. and Juang, R. (2005). Evaluating accessibility by simulating the experiences of users with vision or motor impairments. IBM Systems Journal. doi:10.1147/sj.443.0505 Simulated low vision, colour blindness and motor impairment with EASE and compared simulated performance with real users, capturing some but not all barriers. It bounds what simulation can replace.
• Brajnik, G. (2006). Web Accessibility Testing: When the Method Is the Culprit. Lecture Notes in Computer Science. doi:10.1007/11788713_24 Describes a barrier-based heuristic walkthrough and reports experimental data on its validity and usefulness against a standards review with novice evaluators. The method changes the verdict.
• Brajnik, G. (2008). A comparative test of web accessibility evaluation methods. Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). doi:10.1145/1414471.1414494 Ran a laboratory experiment with twelve novice accessibility evaluators, comparing a barrier walkthrough with conformance testing as the control. The walkthrough raised correctness by 9 to 60 per cent and identified 45 to 127 per cent more correctly judged barriers, at slightly lower reliability, 0.62 against 0.69.
• Petrie, H. and Kheir, O. (2007). The relationship between accessibility and usability of websites. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1240624.1240688 Studied two websites with six blind and six sighted participants whose problem sets overlapped by roughly 15 per cent, and found no agreement between participants and the guidelines' ratings. Priority is not severity.
• Kuzma, J. M. (2010). Accessibility design issues with UK e-government sites. Government Information Quarterly. doi:10.1016/j.giq.2009.10.004 Applied an automated tool to 130 UK parliamentary e-government websites; the vast majority failed despite the Disability Discrimination Act duty. Legal duty does not produce compliance.
• Brajnik, G., Yesilada, Y. and Harper, S. (2011). The Expertise Effect on Web Accessibility Evaluation Methods. Human-Computer Interaction. doi:10.1080/07370024.2011.601670 Ran a barrier walkthrough with 19 expert and 57 non-expert judges; validity and reliability fell with non-experts and more judges produced more diverse outputs. The judge is part of the instrument.
• Vigo, M. and Brajnik, G. (2011). Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers. doi:10.1016/j.intcom.2011.01.001 Applied seven metrics to more than 1,500 pages within a framework of validity, reliability and sensitivity; three ranked highest but reliability and validity-in-use went untested. A metric must report its validity.
• Romen, D. and Svanaes, D. (2012). Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users. Universal Access in the Information Society, 11(4), 375-385. doi:10.1007/s10209-011-0259-3 Tested whether conformance to WCAG 1.0 and 2.0 predicted usable access for disabled participants; the relationship was weak and some reported problems WCAG 2.0 does not address. Conformance does not track success.
• Hanson, V. L. and Richards, J. T. (2013). Progress on Website Accessibility?. ACM Transactions on the Web. doi:10.1145/2435215.2435217 Analysed over 100 United States and United Kingdom government and top-traffic sites from 1999 to 2012; violations fell, though the authors attribute part of the change to technology. It is the longitudinal test of the regime.
• Vigo, M., Brown, J. and Conway, V. (2013). Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A). doi:10.1145/2461121.2461124 Benchmarked six tools against WCAG 2.0: coverage reached at most 50 per cent of criteria, completeness 14 to 38 per cent and correctness 66 to 71 per cent. Tools alone leave most barriers unfound.
• Aizpurua, A., Harper, S. and Vigo, M. (2016). Exploring the relationship between web accessibility and user experience. International Journal of Human-Computer Studies. doi:10.1016/j.ijhcs.2016.03.008 Related perceived accessibility and WCAG 2.0 conformance to 35 user-experience attributes; perceived accessibility correlated with 27, conformance with only three. Experience tracks perception, not conformance.
• Campoverde-Molina, M., Lujan-Mora, S. and Valverde, L. (2020). Empirical Studies on Web Accessibility of Educational Websites: A Systematic Literature Review. IEEE Access. doi:10.1109/access.2020.2994288 Reviewed 25 empirical studies of educational websites, 20 of which used automated tools alone; every site analysed needed corrections. The field's evidence rests on the narrowest instrument.
• Acosta-Vargas, P., Lujan-Mora, S. and Salvador-Ullauri, L. (2016). Evaluation of the web accessibility of higher-education websites. ITHET 2016. doi:10.1109/ithet.2016.7760703 Assessed twenty universities' websites worldwide against WCAG 2.0 with the WCAG-EM methodology, finding that most did not reach an acceptable level of compliance and that prestige did not predict accessibility.
• Alajarmeh, N. (2021). The extent of mobile accessibility coverage in WCAG 2.1. Universal Access in the Information Society. doi:10.1007/s10209-020-00785-w Worked with 16 participants with visual impairment on mobile touchscreens and compiled 34 major problems, some lacking success criteria in WCAG 2.1. The standard lags the platform.
• Csontos, B. and Heckl, I. (2021). Accessibility, usability, and security evaluation of Hungarian government websites. Universal Access in the Information Society. doi:10.1007/s10209-020-00716-9 Evaluated 25 Hungarian government websites on accessibility, usability and security; none fully met WCAG and many failed all three. The three qualities do not move together.
• Onay Durdu, P. and Soydemir, O. N. (2022). A systematic review of web accessibility metrics. Advances in Web Technologies and Engineering. doi:10.4018/978-1-7998-7848-3.ch004 Reviewed 30 metric studies published since 2008 and identified fourteen metrics proposed in that period, recording the guideline set each used, its coupling with the guidelines and how far it had been validated with users. The problem is current.
• Lopez-Gil, J. M. and Pereira, J. (2024). Turning manual web accessibility success criteria into automatic: an LLM-based approach. Universal Access in the Information Society. doi:10.1007/s10209-024-01108-z Used a large language model to evaluate three success criteria that normally require manual checks, reporting accuracy above 85 per cent against prepared test cases. It shows the automated and manual boundary moving, so a coverage figure dates as fast as the tools it measured.