Automated accessibility testing, a few scenarios

What can you test automatically? Unfortunately, not everything—at least not yet.
It is estimated that, at the moment, automated accessibility testing will help you
root out about 30% of the errors in your code. The exact number will, of course,
depend on what your code is meant to do.

Thirty percent is not that much. Why is the number not higher? Well, accessibility
is about more than just correct code. What you can’t test properly is the story you
tell in the browser-generated code.

More specifically, here are a few items you cannot test automatically at the
moment:

As with all tests, you test against standards.
For accessibility testing, we use the Web Content Accessibility Guidelines (WCAG)
put together by the W3C. The latest version is 2.1. There are three levels: A is for
basic accessibility, AA is the global standard, and AAA is the strictest level, for
special applications.

In the European Union, we need to comply with WCAG 2.1 AA for the web. In the
United States, WCAG 2.0 AA is the legal standard at the moment.

Error-free, semantic HTML is also important alongside accessibility guidelines. The
specifications of the HTML Living Standard are applicable here. They are
published in an HTML Standard document that is, unfortunately, not very
accessible or easy to understand. Fortunately, the MND Web Docs contains a lot
of easy-to-understand documentation that is based on the Living Standard.

Testing for accessibility is different from testing for correct PHP or JavaScript
syntax, for instance.

You can test in two ways: based on the static code, and according to the
generated document object model (DOM), which represents how a browser
processes and presents your website.
With static code, you test based on parts and not on a generated page or
component. Static tests do not catch many errors, because they cannot see the
whole picture.
The best accessibility testing is done according to the DOM.

Axe by Deque is currently the best software for web developers. Many tools run
on the axe-core engine, such as Axe DevTools, Google Lighthouse, Microsoft
Accessibility Insights, Pa11y, and PayPal’s Automated Accessibility Testing Tool.
How the reporting is done and the focus of the error messages may differ, but the
underlying engine is the same.

And if you want to check only the HTML, there is the Nu Html Checker or HTML
CodeSniffer.

And then there are countless npm modules for React and Vue, for instance. The
search engine is your friend.
These are all free, open-source solutions that you can use during development.

There are other tools, too, for monitoring entire running production
environments, such as SiteImprove, which offers more than just accessibility
checks, Deque’s Axe Monitor, and the excellent Tenon.io. Tenon is also available as
a WordPress Access Monitor plugin and the Drupal project Tenon IO.

These tools are also useful for checking both the content that has been entered
and updates to modules and plugins. These are all paid solutions, which monitor
entire web pages and websites.

The tests are performed, and the data is stored, not locally, but on the servers of
the relevant companies. Whether you want that can be a consideration.

You might be wondering whether all these tools give the same results. And the
truth is, it’s hard to tell since the presentation of results is not standardised across
tools. At the moment a number of companies are working together on a project
to come up with a single format, Accessibility Conformance Testing Rules Format
1.0.

That being said, it’s important to cast a critical eye over any feedback, regardless
of how it’s presented, bearing in mind that it’s the result of an automated check
and cannot be complete. As we noted above, this method can help you root out about 30% of the errors in your code. You will also need to perform manual tests
with your keyboard, for instance.

Checking for error messages in your code editor is the best place to start. Writing
accessible code involves for the most part the proper use of semantic HTML5.
Some of that can be checked in your code editor.

PHPStorm checks for syntax errors, including in HTML—a form input without a
label, for instance. It also offers autocompletes for ARIA attributes. That’s really
useful for attributes such as aria-labelledby or aria-describedby, which are often
misspelled.

For Visual Studio Code, you can install axe-linter—see Shift further left with
Deque’s ax-linter for VS Code.
If you’re using another editor, it is useful to see early on whether there are any
add-ons or settings that allow you to do checks of these kinds.

A command-line linter allows you to test in the command line with tools such as:

In addition, reactjs.org offers information about React and accessibility that also
includes automated testing. Vue has a website dedicated to accessibility, vue-
a11y.com, while on axe.vue-a11y.com you can find out how to run automatic
checks with axe-core.

This may well be the most efficient way of testing. During development you can
keep checking what you are building. Does it work? Are there errors? And while

you’re at it, you can also test for accessibility errors and do some manual testing
with the keyboard as well as the mouse.

The go-to browser add-on for Chrome, Firefox, and Edge is—you guessed it—Axe
DevTools, which adds a tab in the browser inspector with the option to perform a
test on the web page.

Another popular tool is WAVE, which provides information suitable for non-
techies

HTML checks can be done with HTML_CodeSniffer, a JavaScript-based
bookmarklet.

There are many more tools, such as Storybook’s accessibility options, that allow
you to test per component. Developments are taking place at lightning speed,
and it is both interesting and useful to keep abreast of them.

Once you’ve finished a piece of code, you can push it to GitHub. Via GitHub
actions you can do a number of checks on the quality of your code before it can
be merged—Psalm, Unit tests, PHP coding standards, WordPress coding
standards, Javascript linters, you name it. You can even run accessibility checks.

Adrián Bolonio describes how to set up accessibility checks step by step for React
with GitHub Actions, in a number of excellent posts—see Automating the
accessibility tests of your source code with GitHub Actions. He uses jest-axe as an
example to set up accessibility unit tests for React.

There is one big disadvantage to using an accessibility check on a pull request to
GitHub. At the moment, you can’t definitively approve or disapprove a pull
request that comes with an accessibility warning or error. The error messages
aren’t yet reliable enough for that. You will still have to check manually.

Again, it is always a good idea to do accessibility checks throughout the
development process, and not just at the end, in your code editor, in your
browser, with your build or run, and, if you want, via GitHub actions.

But you will still need to do additional tests, such as the following:

If you test only during a commit to GitHub, you’re too late. It can be a nice extra
check, but it is expensive and too time-consuming to wait for.

Testing during development, in your own browser, is faster and offers you more
options. Moreover, you see the whole story, not only in code, but also in terms of
the user experience, and you can quickly perform additional manual tests.

That’s how things stand at the moment, at least. Maybe in five years’ time there
will be a completely different story. Developments happen so fast, and just
keeping abreast of them is exciting in itself—so stay tuned.