Comparing Manual and Free Automated WCAG Reviews

Automated accessibility testing tools cannot test 100% of WCAG. This position is not controversial. Other than overlay vendors, no automated tool maker makes that claim. This is partly because WCAG is targeted at humans, not code, and so nuance and context apply.

Free automated accessibility testing tools may have even fewer features available to users, which makes sense. My experience is that the majority of people who are testing against WCAG are starting or supplementing with free automated checkers. There is nothing wrong with that. We should want the easy bits handled to free us up for the more nuanced work.

My concern is that too many managers, bosses, stakeholders, and even testers, may do no more than run a free automated tool against a site or page and consider that sufficient. This post shows why it may not be sufficient. Bear in mind that manual testers are not free, so this is not a one-to-one comparison between humans and tools (nor should it be).

Again, this not a criticism of the tool makers. This is a function of WCAG itself and its correct and appropriate focus on people over code.

The Process

I picked a reasonably popular site and, using only its home page, performed a review against WCAG 2.1 Levels A & AA using the following:

• Manual using bookmarklets, assorted contrast checkers, dev tools, and assistive technology.
• axe DevTools v4.47.0 browser extension (using axe-core v4.6.2) for Chrome and Firefox.
• ARC Toolkit v5.4.2 browser extension for Chrome.
• WAVE Evaluation Tool v3.2.2.0 browser extension for Chrome and Firefox.
• Equal Access Accessibility Checker (EAAC) v3.1.42.9999 browser extension for Chrome and Firefox.

I performed the testing on Saturday, 14 January 2023 against a live version of the site. I also archived it to the Wayback Machine when I started testing.

For all tests, I used screen reader pairings of Chrome/JAWS, NVDA/Firefox, and VoiceOver/Safari on desktop. I did not test on mobile, but I did test across viewport sizes. I tested in private/incognito mode and not, making sure to clear my cache between reloads. The page had a reCAPTCHA component which I arbitrarily declared out of scope and did not test.

Highlights

A quick, if a bit insincere, comparison is to count against how many Success Criteria (SCs) each tool found a violation:

Some Success Criteria can be failed multiple times. Some failures can be logged against multiple Success Criteria. These are the counts for total number of unique failures. These also exclude the same instance of an issue replicated against analogous nodes (so different duplicated ids might happen a dozen times, but I count it once):

Some tools also provide warnings or alerts for the reviewer to test manually. The value of these is very much dependent on the site, the reviewer, and the context. These sometimes include WCAG Success Criteria for reference. The number of these may or may not be useful to you or your team, so do not read too much into non-zero numbers.

For some interesting differences between the substance of the reporting:

• Both ARC and I failed an SVG for no img role, though I did so under 4.1.2 Name, Role, Value and ARC did it under 1.1.1 Non-text Content.
• Both axe and I failed the still-functional link with aria-hidden under 4.1.2 Name, Role, Value. ARC failed it under 2.1.1 Keyboard. EAAC failed it under 1.3.1 Info and Relationships.
• Most called out links given a listitem role as not belonging to a list. Axe and I failed it under 1.3.1 Info and Relationships, ARC and EEAC under 4.1.2 Name, Role, Value.

In my manual review I found almost seven-and-a-half times (7½×) more issues than the tool with the next highest set of found issuess across three times (3×) as many Success Criteria.

It is possible I was overzealous, as many accessibility reviewers can understandably be. I had no QA review (often helping me cut 3 or 4); nobody looked over these to make sure I was not being too aggressive. However, I try to adhere to my own take on Postel’s Law in my WCAG reviews — be liberal in what you flag, conservative in what you fail. I also no longer fail issues using SC 4.1.1. My results are in the tables so you can judge for yourself.

Raw Results

These sections contain the output from a manual review and from the tools. Not only the WCAG failures, but also the alerts and warnings the tools issue for further manual review.

WCAG Failures

The following two tables are WCAG 2.1 Success Criteria cross-referenced with the five testing approaches. The first table only covers SCs at Level A and the second table at Level AA.

Each cell identifies if the sample passes, does not apply, or fails against the SC. If the sample fails, a bulleted list describes the issue and, where appropriate, a path is provided to allow you to navigate to the affected node or nodes using your browser’s developer tools (I use XPath).

Each issue description in the first column is followed by a severity of low, medium, high, or critical. These four priorities break down as follows:

Low

Users can accomplish the task with no or very minor workarounds.

Medium

Users can accomplish the task with workarounds.

High

Users can accomplish the task with difficulty and/or significant workarounds.

Critical

It is impossible for some users to accomplish the task.

For brevity I abbreviate “accessible name” to accName and “accessible description” to accDesc.

It is possible that in my manual review I missed other issues. I may have also over- or under-estimated the impact of an issue. With no insight into the audience, no independent QA team, and no interaction with the development team I have no sense of what may be a true barrier or problem for the site’s users. There may be cases where extensive testing with users shows something I flagged as a problem is preferred by its users.

Where cells are blank, the automated checker returned no failures. Tools do not indicate a pass nor that an SC was not applicable, nor should they be expected to.

Warnings

Three of the four automated tools provide warnings. These are cues to the reviewer to dive further into the sample and identify if the thing is really a WCAG failure or a problem for users.

A manual review should result in binary pass/fail, so it uncommon to find warnings in one. You may find “best practices” instead, which I address later.

These lists represent the warnings from each tool. Each tool handles this differently, which is neither bad nor good, but I have done my best to normalize them for consistency here. I did not include specific paths from the tools when provided because I ran out of steam.

• Manual alerts
  1. None; manual reviews should generally get you a binary pass/fail
• axe DevTools warnings
  1. None; the “needs review” category is not in this release
• ARC Toolkit alerts
  1. aria-hidden used, 8 instances, rule ARIAHiddenUsed.
  2. Unable to determine text contrast against image background, 8 instances, rule textWithBackgroundImage.
  3. Heading level skipped, 2 instances, rule headingLevelSkipped.
  4. Multiple naming techniques used, 4 instances, rule multipleLabellingTechniquesUsed.
  5. Autocomplete missing, 7 instances, rule autocompleteMissing.
  6. Multiple header landmarks, 2 instances, multipleHeaderLandmarks.
  7. Empty list, 5 instances, rule emptyList.
• WAVE Evaluation Tool alerts
  1. Long alternative text, 1 instance, citing 1.1.1 Non-text Content.
  2. Skipped heading level, 2 instances, citing 1.3.1 Info and Relationships, 2.4.1 Bypass Blocks, 2.4.6 Headings and Labels.
  3. Redundant link, 3 instances, citing Link Purpose (In Context).
  4. Noscript element, 1 instance.
  5. HTML5 video or audio, 1 instance, citing 1.2.1 Prerecorded Audio-only and Video-only, 1.2.2 Captions (Prerecorded), 1.2.3 Audio Description or Media Alternative (Prerecorded), 1.2.5 Audio Description (Prerecorded), 1.4.2 Audio Control.
  6. YouTube video, 1 instance, citing 1.2.1 Prerecorded Audio-only and Video-only, 1.2.2 Captions (Prerecorded), 1.2.3 Audio Description or Media Alternative (Prerecorded).
• Equal Access Accessibility Checker needs review messages
  1. Confirm Windows high contrast mode is supported when using CSS to include, position or alter non-decorative content, 1 instance, cites 1.1.1 Non-text Content.
  2. Verify that captions are available for any meaningful audio or provide a caption track for the <video> element, 1 instance, cites 1.2.1 Audio-only and Video-only (Prerecorded).
  3. Verify that captions are available for any meaningful audio or provide a caption track for the <video> element, 1 instance, cites 1.2.2 Captions (Prerecorded).
  4. Verify that captions are available for any meaningful audio or provide a caption track for the <video> element, 1 instance, cites 1.2.4 Captions (Live).
  5. Verify the ‘::before’ and ‘::after’ pseudo-elements do not insert non-decorative content, 2 instances, cites 1.3.1 Info and Relationships.
  6. If the following text is a quotation, mark it as a <q> or <blockquote> element: “Building a faster YouTube on web”, 1 instance, cites 1.3.1 Info and Relationships.
  7. Verify that this ungrouped checkbox input is not related to other checkboxes, 2 instances, cites 1.3.1 Info and Relationships.
  8. If the following text is a quotation, mark it as a <q> or <blockquote> element: “aside flow bg-state-warn-bg col … -12-3v-2h2v2h-2zm0-4h2v-4h-2v4z”, 1 instance, cites 1.3.1 Info and Relationships.
  9. If the word(s) ‘background-clip’ is part of instructions for using page content, check it is still understandable without this location or shape information, 1 instance, cites 1.3.3 Sensory Characteristics.
  10. If the word(s) ‘top’ is part of instructions for using page content, check it is still understandable without this location or shape information, 1 instance, cites 1.3.3 Sensory Characteristics.
  11. Verify color is not used as the only visual means of conveying information, 1 instance, cites 1.4.1 Use of Color.
  12. Verify that text sized using viewport units can be resized up to 200%, 1 instance, cites 1.4.4 Resize text.
  13. Verify the <div> element with “listbox” role has keyboard access, 1 instance, cites 2.1.1 Keyboard.
  14. Verify media using <audio> and/or <video> elements have keyboard accessible controls, 1 instance, cites 2.1.1 Keyboard.
  15. Verify <frame> content is accessible, 1 instance, cites 2.4.1 Bypass Blocks.
  16. Verify that using the filename as the page <title> value is descriptive, 1 instance, cites 2.4.2 Page Titled.
  17. Component with “combobox” role does not have a tabbable element, 1 instance, cites 2.4.3 Focus Order.
  18. Confirm the element should be tabbable, and is visible on the screen when it has keyboard focus, 1 instance, cites 2.4.7 Focus Visible.
  19. Verify the <form> element has a submit button or an image button, 1 instance, cites 3.2.2 On Input.
  20. The input element does not have an associated visible label, 1 instance, cites 3.3.2 Labels or Instructions.

The axe DevTools release I used for this post no longer has a “needs review” category. However, I have a copy of 4.36.2 which uses axe-core 4.4.2 and which returned these items flagged as “needs review”:

1. Elements must only use allowed ARIA attributes, 8 instances, rule aria-allowed-attr (for the aria-label on the <div>s).
2. Text elements must have sufficient color contrast against the background, 8 instances, rule color-contrast.
3. <video> elements must have a <track> for captions, 1 instance, rule video-caption.

I kept it separate because it is not using the same axe-core and because users on the current release no longer have access to “needs review” entries.

Bonus: My Best Practices

Each of the tools provide best practices as well. They can address known bugs, anti-patterns, or opinions from the developers, but generally should not represent WCAG failures (though they sometimes do, from experience). For this page multiple <h1>s, redundant roles, or unnecessary HTML attributes were referenced. I am not logging them because it was just taking me too long to write this already.

I am including my own best practices for the page, but I did not include recommendations (I reserve those for paying clients):

1. Visible non-interactive content that is not repeated elsewhere has aria-hidden, which can make for an odd experience for sighted screen reader users.
   //span[@class="hero__eyebrow"][@aria-hidden="true"]
2. If an image has a blank alt (<img alt="">), then aria-hidden is redundant and unnecessary and redundant.
   //img[@alt=""][@aria-hidden="true"]
3. The primary and footer navigation announce as “Main navigation navigation” and “Footer navigation navigation” because the word “navigation” is included in the aria-label.
   //nav[contains(@aria-label,"navigation")]
4. The placeholder for the search does not fail 3.3.2 because it is a common pattern, however consider a persistent visible label regardless.
   //input[@placeholder="Search"]
5. Many of the links are verbose. A couple links are duplicated but have differing link text. Combined these make for a complex page to navigate by link text alone.
   //a[@href="/interop-2022-wrapup/"], //a[@href="/web-platform-12-2022/"]
6. The centered all-caps “CHROME DEVELOPERS” text is a link among other centered text. It is not immediately apparent as a link and in dark mode its background (#2c333f) against the page background (#303136) has a 1:1 contrast ratio.
   //section[contains(@class,"homepage__developers")]//a[@data-type="primary"]
7. The country select uses mixed languages and character sets within options. Because you cannot break up text in an <option> to give parts a lang, consider not mixing these.
   //select[@id="sub-country"]
8. 4.1.3 is mooted on the cookie consent since it appears first in the DOM and is drawn at page load. It may not need to be a live region (nor did it announce as one).
   //div[@class="web-snackbar__label"][@role="status"]

Takeaways

Automated accessibility checkers lack the context of a page and user. They can only run against the code in the current state of the page. This means you have to run and re-run them while checking assorted viewport sizes, orientations, states, and whatever else may be a factor.

They may also disagree on which Success Criterion best matches an issue they identify in the code. This post makes no judgment on which tool is right or wrong in that regard (if some of these can be wrong).

This does not mean you should avoid automated tools. Like any tool, it means they have a place in your toolbox. When used correctly they can be extremely helpful. When used in the hands of a novice they can result in a sense of complacency. When not used at all they can be a missed opportunity.

Please remember that WCAG itself is also the bare minimum of accessibility. Conforming to WCAG does not guarantee something is accessible. It does not even guarantee something is usable. All WCAG does is provide you with a starting point. Lots of WCAG failures suggest the page has not even made it to the starting line.

Related posts:

• Speech Viewer Logs of Lies, August 2020
• XPath for In-Browser Testing, April 2021
• Beware False Negatives, September 2021
• What Does X% of Issues Mean?, July 2022
• The 411 on 4.1.1, December 2022 (repeated from above)

Update: 22 January 2023

Rachele DiTullio ran the same page through the Access Assistant extension for Chrome v8.10.0.11 (which I refer to above as AA) and recorded results to match the format I used in this post: Comparing Level Access automated tools to manual accessibility testing

It found 4 issues and logged them against 5 Success Criteria (so one was logged to two SCs). For the sample (tested a few days after my test and seemingly still the same code), Access Assistant found more issues than WAVE Evaluation Tool, axe DevTools, and ARC Toolkit, and fewer than Equal Access Accessibility Checker.

Update: 23 January 2023

I have been asked about Microsoft Accessibility Insights and Google Chrome Lighthouse and how they compare. I did not test either of them because they use axe-core, the same engine as axe DevTools. Accessibility Insights and Lighthouse both use axe-core 4.4.1, which is an older version than axe DevTools uses.

That being said, each returns more issues that need review than axe DevTools does (which returns none in its current release).

Sales Pitch: 27 April 2023

Nobody asked me to make the computer image in this post into a branded product, so I did it anyway.

Bring a red or yellow notebook to a meeting so folks know you are serious. Nring a matching mug and laptop sticker. Get a blue t-shirt with the logo on the chest to cosplay as a weird accessibility first responder (or lighter blue with black text or any color tee you want).

Thank you for humoring me on my first ever product pitch on this site.

Automated Tools Comparison: 15 April 2024

Equal Entry has published A Comparison of Automated Testing Tools for Digital Accessibility, which takes a slightly different approach than I do here:

• It tests six tools (including two from overlay vendors AudioEye and Level Access) with two or three overlapping the tools I tested (maybe three since since Equal Entry does not name the tools);
• It does not provide its test results;
• It created a 31 page site with 104 guaranteed WCAG violations (I used a single public page);
• It does not provide a URL for the test site;
• It has no manual testing results as a control;
• It performed the comparison as contract work for Evinced.

We know only about a quarter to a third of WCAG is automatable. As such, every tool is going to have an upper limit against a test of all SCs.The 6 automated checkers caught 3.8–10.6% of the issues. In contrast (or support), my own tests above showed a hit rate of 0–13.5%, which is arguably similar.

Without seeing the test site it is hard to understand how to read these results. For example, I can create a 1.1.1 Non-text Content issue that gets caught by all tools or I can create one that gets missed by all tools. I can even create a 1.1.1 issue that I know will be caught in some and missed in others.

One minor concern about reporting, but not the tests, is this statement: If a product links to a WCAG technique that is a published standard, then it is easier to ensure that the issues reported are consistent between products. A WCAG Technique is not a standard. Each is an example of how to meet an SC, but they are not normative (not standards). Some Techniques may even recommend things that simply don’t work.

Regardless, it is still an interesting report and approach.