Lessons from Gutenberg

When Rian Rietveld resigned from the WordPress accessibility team on October 9, I decided to track the fallout in an ongoing Twitter thread.

The subsequent weeks and days proved to be wonderful insight into how a project can suffer when accessibility is not built in from the start. When subject matter experts are only included to mop up what few others consider to be bugs. When sprints and deadlines make no consideration of what are now remediation efforts.

You can read my Twitter thread and all the sources I link, but Andy Bell did a great job mining it for his post What Can Be Learned From The Gutenberg Accessibility Situation? over at Smashing Magazine.

Instead, I think there are lessons to be learned from Gutenberg that, thanks to its open nature, we can apply to other projects. Ideally, you want to make your next project the Anti-Gutenberg.

• Objective
• Resources
• Training
• Process
• Timeframes
• Messaging
• Leadership
• Wrap-up

Objective

When Gutenberg was first formally announced at WordCamp Europe 2017, I wrote about what I heard from Matt on stage. The scope and objective of Gutenberg were not very clear.

Matt mostly tried to convey airy ideas of a modern interface, all while exposing his fear of Wix, SquareSpace, and Weebly. In other words, risks to Automattic’s business, but not WordPress itself.

With no clarity on how fundamental features of WordPress would be supported, no public release schedule, and no major community involvement to that point, it was clear that Gutenberg was generally un-scoped and those parts that might be scoped would suffer from scope creep.

The Lesson

Identify the output objective and the scope before committing any code, let alone any planning resources. Make sure stakeholders understand it. Publish it. Ensure a barrier to scope creep and a process to allow scope expansion. If accessibility is part of it (it should be) identify to what level and standard (eg. WCAG 2.1 AA).

Resources

Gutenberg has been framed as a community project, but Automattic has committed at least 40 employees. There are dozens of other volunteers, in many cases funded by their own employers. They are essentially free labor for Automattic’s agenda.

Gutenberg was never staffed for accessibility to be successful. We know that the accessibility team is small, over-taxed, and has little prior skill in React. That niche intersection of skills, React and deep accessibility knowledge, is hard to find and the Gutenberg project expects those people to volunteer. Accessibility scapegoating was (is) a very real concern.

Even the accessibility release lead Automattic provided for Gutenberg is not an accessibility practitioner. He has been thrust into a role overseeing an aspect with which he has little experience or training. At WordCamp US, Matt suggested that if you cannot get hired for a job doing the skill you want to contribute, then maybe you are not qualified to contribute. That message cuts both ways.

The Lesson

Training can get basic accessibility skills across a team. Deeper accessibility skills are tougher to find and typically come at a premium; pairing those with deeper technology stack skills compounds that. Given what it takes to get to that level, expect to pay good money for professionals, open source or not, and build them into the budget from day one.

Training

Gutenberg is built on React. Much of the rest of WordPress is static HTML generated from PHP. The React developers overall had little or no accessibility experience. The accessibility team had little or no React experience. No effort appears to have been made to train either team to try to bridge that gap.

That meant that in many cases developers would be talking past each other. The accessibility team could identify what the HTML needed to look like to address an issue, but the React team might not have the needed control over the HTML output of React.

It is fair to expect the average developer to have little experience with screen readers or dictation software. But the developers did not even have the most fundamental training — testing keyboard accessibility during development.

The Lesson

Identify the skills you need for the project. If you can, build a team from your internal resources who have those skills. If you don’t have those skills on your team, train them. The fundamentals of accessibility can be easy to convey to a team, so do it.

Process

Gutenberg was ultimately driven by engineering priorities. Release schedules, developer feedback, code commits, all moved at a timeline that was too aggressive for structured testing, let alone thoughtful development.

In many cases, new code was committed before requested accessibility feedback was provided. Anyone who wanted to review the accessibility was working on a constantly shifting codebase. In addition, breaking changes were constant. The accessibility team might fix an issue one day only to see it reappear a week or two later. With no process to prevent this, the accessibility team kept fighting the same fires, killing morale and leaving other parts of the project exposed.

Most fundamentally, accessibility was not incorporated from the start. Not in the project specification, not in the design phase, and not in the definition of done. Gutenberg was always going to be playing catch-up with accessibility, forcing it into a pain point of the project owners’ own creation.

The Lesson

Incorporate accessibility from the start. At planning, design, scoping, scheduling, testing, and every other step along the way accessibility must be incorporated. It saves time in the end and prevents a backlash against accessibility requirements and team members.

Timeframes

Gutenberg always had an artificial timeline. Each announcement of a new date was tied to an event, not to a discrete list of features or quality standards. With no clear definition of done, the target date became the only visible success metric and it was often unknown. It slipped multiple times, most recently November 19, then November 27, then no date, then abruptly December 6 (with just 4 days notice). The accessibility team did not know about the November 19 date until 6 weeks prior.

When you are targeting an artificial date, one that keeps shifting, that can undermine good development practices. Test cycles are reduced, decisions are made for expediency not user experience, and generally people are put under unnecessary pressure.

Note that repeated calls to delay the release until after an accessibility audit were dismissed. Even after conceding an audit might be useful, the community was told the audit will not affect release timing and that it would be more prudent to explore an audit on a less rushed timeline in the future.

The Lesson

Target dates are critical. Without them, projects may never end. However, target dates must be chosen based on scope, objective, and resources. They also must be allowed to slip when critical bugs are brought to light (such as the legal risk of an inaccessible product). Otherwise they become pain points that can justify poor decisions.

Messaging

Messaging was disjointed and inconsistent. Sometimes news came on the Make-dot site, sometimes in comments on other’s posts, sometimes in Matt’s posts on his own blog. Finding a clear plan with updates became tricky. Expecting WordPress community members to follow every passing tweet, GitHub comment, or Slack missive is a very high bar.

As Matt was making assertions that he believe Gutenberg was more accessible, he would never answer requests for evidence. When Automattic committed to an accessibility audit via GitHub, it later dropped it, then Matt suggested the audit was never off via a (petty) tweet (now deleted), then announced the audit on his own site.

All the while, Matt and others from Automattic stressed that Gutenberg was a community project. They constantly used the word we, even though it was clear we had a shifting meaning. This was enough of a question that Matt was asked at WordCamp US to clarify who we are.

The Lesson

Somebody needs to be the sole communicator, and that communication should come via, or funnel to, a single platform. Avoid words like we when the bulk of the team, community, public, etc. feel their voices have not been heard. Do not dismiss genuine concerns from the community and/or experts, and certainly do not threaten them.

Leadership

Arguably, the biggest problem with Gutenberg’s accessibility is not code. It is lack of leadership. The tone-deaf response to real concerns compounds it. Much of the messaging confusion came from the top, and much of the public frustration with the project came from Matt’s public behavior.

With Matt’s role in the tech industry on the whole (not just helming a billion-plus three billion dollar company), and the power dynamic that brings, he spent too much time punching down (he deleted this tweet), making false assertions, and making community promises and personal commitments he has so far failed to keep. I am not the only one who will believe his commitment to fund WPCampus’s accessibility audit only after the payment has cleared, and I hope this motivates him to finally keep one of his commitments.

This is not how a community nor project leader behaves. This is how someone behaves who cares more about an artificial deadline driven by fear of competitors’ impact to the bottom line. With the exception of the people who need something from him (such as a paycheck), he has burned a lot of goodwill from the very people he needs to move his agenda forward.

Addendum: December 14, 2018

My 404 handler notified me that Matt deleted the tweet I linked. The tweet where he effectively tried to cost a women in tech her job for a tweet that did not mention him nor was inflammatory. He did no such thing to any men in tech for far more inflammatory tweets. He also failed to publicly apologize. I don’t believe people with that kind of influence should get away with erasing history. Or be in charge of a project with such a large influence over the web.

The Lesson

Whomever is in charge needs to check his or her ego and listen to the team, as well as the customers and users. That person must also honor any commitments he or she has made, as doing otherwise will damage the credibility of not just leadership, but the project as a whole.

Wrap-up

I may have spent more time cutting this post after writing it in bits and parts over the last month. There are just so many examples of how this project’s failure to address accessibility has damaged the image of the flagship feature of WordPress 5.0 that had I included them then my original Twitter thread would have been light reading by comparison.

This post paints broad strokes for a reason. Lots of people gave their time and effort and their work should not be diminished. A broken project, however, can make everyone resent it and the people behind it.

I hope that Gutenberg will be successful, but it and other projects cannot be when they are mismanaged at the most basic level.

Update: 1 September 2020

Joe Dolson from the Make WordPress Accessible team gave an update at WPCampus Online on the ongoing work on the results from the Gutenberg accessibility audit (commissioned by WPCampus in late 2018 and delivered on 1 May 2019).

In 15 months the team has managed to work through two-thirds of the tickets, or 66 of them. While an impressive amount of work has been done by such a small team, let’s not forget how this is all clean-up from the two years of warnings prior to the Gutenberg launch that this would happen.

Automattic did not fund this audit, though it did donate some money, and Automattic is still relying on volunteer accessibility experts to do the work to make its platform legally viable for many of its customers. Remember, Automattic is a three billion dollar company.

A full transcript is available at the session page.

You can see open issues in Trac and on Github.

Update: September 25, 2020

As the W3C has embarked on a full web property rebuild, its vendor (Studio24) indirectly announced earlier this month that it had dropped WordPress from consideration as a CMS. WPTavern took issue with this yesterday, and Studio24 responded today, (politely) pointing the finger squarely at Gutenberg.

Part of WPTavern’s argument was that the selected platform (Craft CMS) is not open source software (OSS), though WordPress itself is not completely open source in practice (Slack versus IRC or MatterMost being the easiest example). More importantly, Gutenberg itself was not truly open source in process — Automattic drove it as a business decision reaction to the growth of Wix, Weebly, and SquareSpace.

Studio24 cited accessibility as the deciding factor. Automattic has already signaled accessibility is not a priority for it or its investment in Gutenberg. The community had to crowd-fund an audit, all while Gutenberg was pushed to final public release with dozens of known accessibility issues. The entire time, all the accessibility work was done by volunteers, not supported by Automattic’s $3 billion bulk in any meaningful way.

As Gutenberg works to replace some common plug-ins with its own features (think ACF), any code written today that uses the features of those plug-ins has a built-in expiration date. Even as Gutenberg’s scope continues to shift, the same confusion we saw over meta boxes will likely appear in other core WP features, warranting unanticipated code changes. This is an unnecessary and unknown future cost to the W3C.

By using a smaller platform (which has already made a public commitment to WCAG), the W3C has more leverage to force WCAG / ATAG compliance. It can do this in the contracts, where payments are tied to compliance. With WordPress, there is no way to do that. All accessibility work comes from the community; there is no way to enforce it.

Given the past behavior of Automattic (which predicts future actions) versus the published (and contractually obligated) commitments from Craft CMS, the W3C made the right call. It is an unfortunate black eye for OSS, and an unforced error on the part of Automattic that got us to this point, but here we very publicly are.

Related from me that informed this hot take:

• This post you are reading;
• First Reactions to Gutenberg (2017);
• A Model for WordPress Accessibility (2019).

Update: November 2, 2020

At the start of October, the WordPress Accessibility Team hosted WordPress Accessibility Day, a live-streamed remote conference. Raghavendra Satish presented Gutenberg Accessibility: A Screen Reader Users Perspective and I have embedded the video below. If the embed does not work, you can jump to his session in the larger block at YouTube.

Update: January 10, 2022

Joe Dolson calls out a cavalier tweet of mine in his post Don’t judge progress by the number of open issues.

He is right. Error density, changes in code over time, plus overall complexity changes, all factor into progress that raw numbers do not show. And as he notes, closing an issue does not necessarily mean fixing an issue.

My attempt at a joke was that Gutenberg was a more accessible product when it did not exist, as supported by my spurious math suggesting time flows backward. But it was also a shot at Automattic’s failure to support (in dollars and developers) the accessibility of the tool it forced through the process. Where the progress it makes is on the backs of OSS developers. And where Joe confirms in his post that it is still grimly inaccessible.

Gutenberg has been with us for 5 years (4½ since its announcement) and this (with my other posts) suggest Automattic’s 3 billion valuation dollars are still not being used to hire accessibility pros and assign them to the project and/or train the community on which it relies to support its product.

I have consistently judged progress by actions, and I still haven’t seen any from Automattic suggesting it has made any.

Update: August 13, 2023

From Alex Stine’s #WordPress Gutenberg 6.3 state of #accessibility video:

…the other big problem in Gutenberg development moves so fast we’re already talking about collaborative editing before screen reader users have a fair experience to edit by themselves. And don’t you all think that’s a little bit backwards? Because I sure do.

Lessons still not learned.