Baseline Does Not Really Cover Baseline Support

Yeah, that’s not exactly a helpful title.

The relatively new Web Platform Baseline offering does not track browser support for accessibility features built into the web platform. If you need to understand whether browsers support accessibility features as your own base level set of requirements, for legal or other compliance reasons, then Web Platform Baseline does not represent a baseline.

Background

In May 2023, Google announced its Web Platform Baseline initiative. This came out of efforts from the WebDX Community Group, chartered to have two workstreams:

• The research workstream focuses on shared research to make available quantitative and qualitative data to all aspects of the web platform feature life cycle.
• The feature set workstream explores the set of interoperable features of the web platform.

In December 2023, Web Platform Baseline amended its framing, partly as a function of developer feedback (emphasis mine):

Web Platform Baseline gives you clear information about which web platform features are safe to use in your projects today. When reading an article, or choosing a library for your project, if the features used are all part of Baseline, you can trust the level of browser compatibility. By aligning with Baseline, there should be no surprises when testing your site.

Baseline is rolling out on MDN, Can I Use, web.dev, and we will be providing the tools for you to show that features described in an article or library are all in Baseline.

[…]

Baseline has two stages:

• Newly available: The feature becomes supported by the last of the core browsers, and is therefore interoperable.
• Widely available: 30 months has passed since the newly interoperable date. The feature can be used by most sites without worrying about support.

Some of the Web Platform Baseline material has started to appear in MDN, such as this disclosure widget on the <dialog> page:

The detailed Web Platform Baseline definition at MDN gives a bit more context to source data:

Baseline is a community effort of the W3C WebDX Community Group and relies on MDN’s open source browser compatibility data.

In addition to the MDN data, Web Platform Baseline uses browser release notes to supplement its information. I have not waded into its repo nor discussions to understand the processes that it follows to include that information.

Because Web Platform Baseline was being promoted again this week, I finally got around to looking into it.

Accessibility

For me (and my clients), the base level of support (I had to phrase it that way to avoid confusion) of anything we implement must include accessibility considerations. This is often for legal compliance, but also to meet contract requirements.

If a developer wants to deploy a new feature, but only one browser exposes that feature to accessibility APIs, then the base level of support for that feature is insufficient. Think of years of workarounds and polyfills for the <dialog> element or inert attribute to get around that lack of basic support.

In the initial announcement from Google Chrome I found no mention of accessibility as a metric for feature support. I also found none in the follow-up announcement from Google Chrome. The MDN Baseline announcement makes no mention of accessibility. The detailed Baseline definition at MDN makes no mention of accessibility.

I saw nothing in the Web Platform Baseline documentation nor goals about how to test, track, nor report whether or not something is (at least) accessibility supported (nor what level of support across versions and assistive technology / browser combinations).

MDN’s open source browser compatibility data has no structure to contain accessibility information, so it isn’t going to come from there.

Browser makers are not perfect witnesses to the accessibility of their own implementations. Sometimes regressions happen and it takes a couple versions to fix them. Sometimes a browser asserts something works when it doesn’t. Sometimes twice. Sometimes thrice. And sometimes it files PRs at Can I Use in a form of revisionist history to overwrite documented bugs in its browser.

Essentially the source data feeding Web Platform Baseline is insufficient, inaccurate, or corrupt when it comes to accessibility support and exposure for web platform features.

From a software development perspective, if a thing does not work for its target audience then it is buggy. The current model for Web Platform Baseline is missing a critical signal to identify this. Which means it is setting a false impression and opening its users (developers) up to risk.

Since this is a Developer Experience (DX) initiative, I think we need to acknowledge that telling developers which web platform features are safe to use in your projects and being wrong about it is, in fact, a terrible developer experience.

A Quick Example

Let’s look at the MDN browser compatibility table for <datalist>, the primary source for Web Platform Baseline (the badge is not on the MDN page, so Web Platform Baseline is not yet asserting anything).

Chrome 20+ lists full support

Yet: Issue 1322462: Form control validation errors don’t change zoom when zooming in/out on the page (CMD +/- on MacOS), open since 4 May 2022 with no action other than lowering its priority a year later (after threatening to close it)

Safari 12.1+ lists full support

Yet: Bug 240077 – AX: Default field validation messages do not zoom with page, open since 4 May 2022 with no action

Firefox 4+ lists partial support (options do not appear on input types date, time, range, and color)

Yet does not mention: Bug 1756203: Users need a way to increase the size of overlays/menus that come from web content but are styled as browser UI (e.g. datetime-picker, text-input datalist, alert(), prompt(), tooltip), open since 18 February 2022 with no activity in 5 months and none pending.

MDN’s support chart, which feeds Web Platform Baseline, indicates <datalist> is good to use except for a few input types.

A dev may not know these bugs when weighing whether or not to use a native web platform feature or wire up a custom ARIA construct. A user doesn’t care about the charts when filing a complaint against your site.

Who will update <datalist> support information for Web Platform Baseline?

Browser makers clearly aren’t doing so. Can I Use <datalist> support notes are less accurate, so that’s no good as a comparison.

Am I expected to file an MDN web compat PR for every browser bug I file? That’s a heavy burden and requires devs to both know to do it and be willing to provide free labor for browser makers.

In addition, reporting documented and confirmed browser bugs with MDN web compat doesn’t mean anything will happen, whether after only a few months or more than a year.

Web Platform Baseline is built on this foundation of sand.

I Filed an Issue

Some of what I learned was a function of filing an issue, #498 Is accessibility support included?

I got confirmation that Web Platform Baseline is essentially an aggregator, which then applies the rules I outline above to tell developers a feature is “supported”. The resources they are aggregating simply don’t provide that data, and the team does not have the background nor capacity to evaluate it.

When I was asked for sources it could integrate, I offered:

• a11ySupport.io, which has robust tests but is community supported and likely not as current as a result; and
• ARIA and Assistive Technologies Community Group, though it only tracks screen readers and only cares about ARIA.

Because evaluating those resources along with planning how to integrate them into Web Platform Baseline will take time (assuming they can be integrated), I suggested a warning or disclaimer in the short term added as a third sentence to what is there now (emphasis is my sentence):

Since March 2022, this feature works across the latest devices and browser versions. This feature might not work in older devices or browsers. There is no guarantee this feature is accessibility supported , however, so support testing is recommended.

If you have better, briefer, clearer language then please offer it at the issue.

Related

I missed that M⁠аth​ias Sc​hä⁠fer (molily) had raised broader concerns in May in the post On browser compatibility and support baselines:

Google Baseline does not answer crucial browser compatibility questions. Baseline merely states that the latest version and the last but one version of certain evergreen browsers support a feature interoperably. This is good to know, but not enough to use the feature on your site in a safe manner.

[…]

A better browser support box should center on practical and safe usage. A concise text should answer these questions: What is the percentage of users with browsers supporting the feature? How do old browsers deal with the feature? Can it be used as an enhancement? How to detect the feature? How to develop a fallback? Do polyfills exist?

Posts of mine that may be related because they cover browser bugs for web platform features that may be reported as “supported” but which will create barriers for users and WCAG violations in practice:

• Avoid Default Browser Focus Styles, 4 February 2017 (browsers have mostly fixed this)
• Display: Contents Is Not a CSS Reset, 1 May 2018 (though this is primarily the fault of the spec, browser makes didn’t think to flag how devs would use it)
• Avoid Default Field Validation, 20 February 2019
• Column Headers and Browser Support, 20 February 2022
• It’s Mid-2022 and Browsers (Mostly Safari) Still Break Accessibility via Display Properties, 7 July 2022
• Brief Note on aria-readonly Support, 20 November 2022
• Avoid Spanning Table Headers, 20 February 2023
• No, APG’s Support Charts Are Not ‘Can I Use’ for ARIA, 29 April 2023
• Under-Engineered Comboboxen?, 29 June 2023
• Splitting within Selects, 23 October 2023

As always, do not hassle WebDX Community Group members. They are trying to fill a demonstrable gap in what little time they have with the background and skills they bring to the table. I would hope I don’t have to say this, but just as I have asked people not to pick on Apple and Google people for user-hostile decisions from their employer, well, here we are.