My Thoughts on the State of Surveys

Every few months there’s another State of Something survey. As of this week, the State of CSS 2025 survey results have just become available and the State of HTML 2025 survey is wrapping up. This post is skewed to those, touches on more, and is rather disjointed.

I should disclose that I am broadly wary of surveys. I am also not a researcher. My thoughts here are mine alone and driven as much by gut as anything else.

Demographics

The 2025 State of CSS Survey results came out this week. I went to the demographics section to get a sense of the participants:

• A majority of respondents are white, male, and abled (no disability). A plurality are American.
• The average respondent age is 36, with 13.4 years of experience and working for a 337 person company.

When looking at the results, consider how much those categories reflect you. Maybe consider that of the people named as resources (including me). This is not a judgment, but an observation.

I suspect that some of the low participation from disabled folks may be related to the dozen still-open accessibility bugs I’ve filed against the surveys (I stopped filing).

If you go back through other industry surveys, the breakdowns don’t differ much. Chris Ferdinandi identified this in the State of JavaScript 2024 survey. Miriam Suzanne talked about this in her November post What do survey demographics tell us?. I noted the Stack Overflow survey didn’t ask about disability at all.

Accessibility Data

In the State of CSS 2025 Interactions Pain Points section, the survey asked What are your top CSS pain points related to interactions? Respondents were offered five freeform text fields.

24 out of 321 responses (7%) included accessibility. 8 of those named CSS carousels, with a couple angry. 8 cited a failure of the CSS Working Group.

When you go to the section covering All Features and filter by only “accessibility” nothing shows up. There were no questions about accessibility. It’s interesting (to me) there is that much about accessibility (and CSSWG process) given the demographic breakdown I cited

This is despite an overwhelmingly abled respondent pool.

So I’m pleased to see accessibility still percolate up in these results, even if it wasn’t asked. I want to see that happen more.

Selection Bias

If you want your general developer survey to produce lots of content on “AI”, for example, make “AI” 23% of all questions, as Stack Overflow did. Reading the results, I would like to know if the drop in positive sentiment toward “AI” was higher among disabled devs, but Stack Overflow didn’t ask about disabilities.

The State of Something surveys are driving browser and standards work based on the questions they choose to ask as much as, or more than, the answers.

I alluded to that in the 2025 State of HTML survey when I noted that half the questions (3½ questions out of 7) in the Forms section are on color pickers.

It also asserts Chrome DevTools & WAVE are screen readers. It forgets Firefox’s or Safari’s dev tools. It lumps blindness into low-vis. It listed “Arc” as an accessibility testing tool, without clarifying if it meant TPGi’s ARC (which is) or the Arc browser (which isn’t).

I often have to dismiss accessibility results from surveys. Poor framing, context, and sometimes understanding make the data useless to me. Though there is a risk in what conclusions others will draw.

Sampling Bias

I understand selection and sampling bias are not the best breakdowns here. I’m still not a researcher.

Don’t treat survey results as absolute truths. I discuss this with the WebAIM 10th Screen Reader User Survey:

This was all from a self-selected uncontrolled survey. If a particular screen reader vendor (Freedom Scientific / Vispero) once again prompted its customers to fill it out, that could have skewed some of the results.

One year NVDA had claimed the top spot as primary screen reader. The next year, the other vendor moved back to the top spot.

For the State of Something surveys, if you’re not plugged into the top sites or influencers, you may be unaware. If your focus on these topics ends with your workday, you may not go looking. If you have survey fatigue, well, me too.

There’s also a first-mover bias:

While the survey will be open for about a month, responses entered within the first two weeks (until end of July) will have a much higher impact on the Web, as preliminary data will be directly used to inform Interop 2026.

Add the accessibility bugs in the surveys themselves, and when you get a disabled population in front of a survey they may simply close the tab.

Wrap-up

Fill out surveys. These are one more channel where your voice can be heard. But balance that against any clear biases and barriers. Hopefully people running these will recognize that these are broadly qualitative and produce only anecdata that can inform more rigorous efforts.