Beware False Negatives

Automated accessibility testing tools are rightly wary of giving false positives. You don’t want to flag something as an error in a test only to have the team rebuild an entire thing for no reason (never mind the risk of introducing errors).

We know automated accessibility testing tools can really only capture about 30% of issues. That number is, of course, highly variable based on factors specific to the thing being reviewed. As more complex widgets are built that lean on novel ways of twisting HTML and CSS, it can be harder to validate them with a tool. Especially through all their states.

But what happens when the tools themselves are missing things they should catch? Developers who build a broken thing, but do not have the necessary testing or even standards experience might rely on automated tools and produce problematic content as a result.

This is not new, of course.

Straight-up HTML Violations

Take the following construct:

<button>
  <h2>A <code>&lt;button&gt;</code> Holding a Heading</h2>
</button>

This will not trigger a warning from the major accessibility testing tools — WAVE, Axe, and ARC. If you don’t believe me, I made a demo, which you can also load directly in debug mode and test yourself.

See the Pen What Happens to Headings in Buttons by Adrian Roselli (@aardrian) on CodePen.

This code will, however, generate an error in the HTML validator. Ideally a code linter would catch it as well. And it rightly should, because this is not only invalid HTML, but it is a 4.1.1 Parsing error under WCAG.

And not only that, it doesn’t act as a heading in Safari with VoiceOver on iOS 14, iOS 15, or macOS 11.4; nor in Chrome with TalkBack on Android 11; nor in Chrome with JAWS 2020 or 2021. NVDA 2021 with Firefox announces the heading, as does Narrator with Edge, both papering over the developer’s mistake. Going off the WebAIM 2019 screen reader survey, that means 70% of users will encounter a broken pattern.

While accessibility practitioners generally try to avoid 4.1.1 failures for stuff that has no impact on users, this has an impact on users.

A developer knowledgeable in HTML could catch this at a glance, and probably would know to never use this construct even without having to confirm its brokenness across a range of assistive technology. An accessibility practitioner should be able to take it further and test it in the common platforms.

Abusing Roles and Not Knowing It

Take the following construct:

<custom-element role="heading" aria-level="2">
  <div role="group" aria-label="Expanded region">
    <custom-label role="heading" aria-level="3">
      A Fake Level 3 Heading
    </custom-label>
  </div>
</custom-element>

It is not uncommon to see ARIA constructs used to get around linting and validation errors. The HTML validators do not check ARIA (or all of ARIA), so if your goal is to avoid HTML validation errors then ARIA is a great way to game the system. Try it yourself with this debug version of the following demo.

See the Pen by Adrian Roselli (@aardrian) on CodePen.

As predicted, the HTML validator fails to flag the invalid ARIA construct. It does, however, note the broken HTML equivalent.

Take a moment to guess how the major accessibility testing tools (WAVE, Axe, and ARC) perform. After all, there is invalid ARIA nesting happening as well as a broken HTML structure on the page that we know the HTML validator catches.

These examples are 4.1.1 Parsing errors. While it is unclear if the author wanted a heading level 3, level 2, a group, or some other combination, it is clear that a screen reader is left to apply its own rules to try to make sense of it.

A related example is when a well-known name in the web dev industry shared a broken drop-cap approach that leaned on a non-standard role (only recognized by VoiceOver) and misused ARIA accessible name techniques. Someone who knew ARIA or tested on more assistive technologies than their dev machine’s default browser would have caught this.

But people trusted this person and integrated it into their work. And when this person revised the code (basing it on mine) and did not acknowledge why they made the change, it became not just de facto gaslighting, but failed to signal to those who had used the broken code that they should maybe go update it.

Abusing Roles and Knowing It

Some of these developer mistakes may be genuine errors from misunderstanding, lack of time to test, lack of understanding of the impact it can have on users, or even just laziness.

But then there is Facebook.

If you have not read the latest example of abusive behavior of this company, then read Facebook Rolls Out News Feed Change That Blocks Watchdogs from Gathering Data and see how Facebook capitalizes on affordances for disabled users to game the system.

The changes, which attach junk code to HTML features meant to improve accessibility for visually impaired users, also impact browser-based ad blocking services on the platform. The new code risks damaging the user experience for people who are visually impaired, a group that has struggled to use the platform in the past.

The updates add superfluous text to news feed posts in the form of ARIA tags, an element of HTML code that is not rendered visually by a standard web browser but is used by screen reader software to map the structure and read aloud the contents of a page. Such code is also used by organizations like NYU’s Ad Observatory to identify sponsored posts on the platform and weed them out for further scrutiny.

Generally I do not hold it against individuals who work for a demonstrably amoral company, especially when they are trying to improve accessibility for those few workers and users who have little choice. In this case, however, I rather hope anyone who works in accessibility at Facebook strongly considers leaving before their skills are used to further harm users, especially disabled users. It seems counter to the reason one gets into accessibility work.

Contrary to the title of this post, Facebook is not a false negative. Facebook is a net negative. Block it at the router. Or the hosts file.

Wrap-up

We all make honest mistakes, and sometimes rely too much on our tools to catch them. But they won’t. This doesn’t mean the tools are worthless, but the people behind those tools may have challenges ranging from strong (questionable) opinions to genuine technical difficulties addressing some of this. If it was easy, accessibility overlays might actually be useful instead of harmful.

Your options are straightforward — learn HTML well, understand the impact of ARIA, and test with more assistive technology than is on your phone. Then pass that knowledge to your team.

Alternatively, if you are intentionally trying to game the system to pass code checks or something as simple as skirt watchdogs in a brazen attempt to subvert democracy while knowingly harming your disabled audience, then you can get into the sea.

Related

• WCAG 2.1 parsing error bookmarklet (2012 and 2019 by Steve Faulkner)
• The HTML Star Is Ignored (and Shouldn’t Be) (2014)
• Uncanny A11y (2019)
• Building the most inaccessible site possible with a perfect Lighthouse score (2019 by Manuel Matuzović)
• Speech Viewer Logs of Lies (2020)

Update: 26 September 2021

Hidde asked a good question about automatically making certain roles focusable, a narrowing of the request to automatically fix detectable accessibility issues:

Accessibility friends… what if browsers would take role=button/role=link as a hint to add an element into tab order and allow activation with keyboard? What would be pros and cons?

Steve Faulkner asked this exact question in August in his post Why not? and got some thoughtful replies.

As this post outlines, developers already mis-use roles, including interactive roles. Some do not understand broken nesting, some choose to do so to bypass linters, and some (Facebook) intentionally mis-use ARIA.

Scott lays out considerations in a response thread:

doing this would also take away legitimate use cases where controls need to be programmatically exposed to AT, but DO NOT need to be in the focus order (and shouldn't need a tabindex=-1) because there are alternative focusable elements to perform the same function

And drops this nugget, which I embed after the tweet:

For example, here's just a QUICK demo of where things 'look' relatively fine with invalid markup, but there can be minor to major impacts for assistive technologies
codepen.io/scottohara/pen/oNwQLNr
twitter.com/scottohara/status/1442134…

Because browsers are papering over poor mark-up, and accessibility testing tools fail to flag it, screen reader users in particular are harmed by the level of correction browsers already provide.

See the Pen invalid markup. what could possibly go wrong? by Scott (@scottohara) on CodePen.

And Pat points out why even relying on what browsers have in place today can lead to “best viewed in” messages on sites.

the problem was that without a documented error correction algorithm, browsers all applied their own mysterious heuristics to fixing borked markup. and unless a developer then tested in all browers (rather than just Chrome or whatever) that led to a myriad of bugs for real users

When developers or even spec writers say WCAG Success Criterion 4.1.1 Parsing (A) is no longer a thing to worry about, they are simply wrong. Especially when self-described accessibility experts promote patterns without testing, without understanding, and without the necessary skill.

Update: 3 October 2021

Since I wrote this post, the ARIA in HTML specification has moved to Proposed Recommendation. Unlike some W3C documents, such as Notes, which exist only as informational documents, ARIA in HTML is clear in its purpose:

[…] This specification’s primary objective is to define requirements for use with conformance checking tools used by authors (i.e., web developers). These requirements will aid authors in their development of web content, including custom interfaces/widgets, that makes use of ARIA to complement or extend the features of the host language [HTML].

Until conformance checkers (such as the ones I cited in this post) implement testing for invalid nesting as in my examples, I encourage all developers to read the section 4. Allowed descendants of ARIA roles. It is arguably easier to read than the ARIA spec itself. For the scope of the examples in this post, the table shows that headings should not be in buttons, and headings cannot contain other headings.

Read Scott O’Hara’s (one of the spec authors) description in his post ARIA in HTML.