Skip to content

End-to-End Tests With Playwright: Frontend, Backend and Database · Part 12 of 14 in the series

Part 12: Accessibility Checks With axe

Let axe look at four states of the quest board on every run, fix what it finds, and then fix what it could not find: a field whose label nobody ever sees. An ARIA snapshot pins down what a screen reader is told.

Quest objectives

  • Know what axe checks, and why that is a share of the problems and not all of them
  • Write a fixture that hands the test a function, so the test decides when to check
  • Check several states of a page instead of the first one it shows
  • Read a finding of axe and fix it
  • Spot a problem no rule reports, with a hint from your own locators
  • Pin down roles and names of a page with an ARIA snapshot

The suite knows how to log in, and all four sections are green again. Before the series ends in a pipeline, two more kinds of checks move into the frontend section. Both need a browser and a page in a state the test has chosen, and that section has both already. This part asks whether everyone can use the board. The next one asks how it looks.

A second pair of eyes for the board: a machine measures what can be measured, such as contrast, and the rest stays a job for people.
A second pair of eyes for the board: a machine measures what can be measured, such as contrast, and the rest stays a job for people.

axe is an engine of rules for common accessibility problems: text with too little contrast, form fields and buttons without a name, images without a text alternative, and many more. The package @axe-core/playwright runs that engine inside the page your test has open and hands the findings back to the test.

Rules can only check what a machine can decide. Whether a text has enough contrast is arithmetic. Whether a label says something useful, whether the order of the page makes sense, whether you get through a form with the keyboard alone: that takes a person. So axe finds a share of the problems, not all of them. It is the share that costs nothing to look for on every run, and that is reason enough to let a test do it.

Install the package:

questboard
$ npm install --save-dev @axe-core/playwright

A fixture that is a function

Every test that checks accessibility does the same two things: run axe on the page, and look at what it found. That belongs in one place, the fixtures of the frontend section. Open tests/frontend/fixtures.ts and add an import as its first line:

tests/frontend/fixtures.ts
import AxeBuilder from '@axe-core/playwright';

The type of the fixtures gets a third entry:

tests/frontend/fixtures.ts
type Fixtures = {
  board: QuestBoard;
  /** Whether the mocked API knows the visitor as a member. `test.use({ loggedIn: false })` turns it off. */
  loggedIn: boolean;
  /** Checks the page as it is right now, and returns one line per problem axe finds. */
  findA11yProblems: () => Promise<string[]>;
};

And the fixture itself goes into base.extend, after board:

tests/frontend/fixtures.ts
findA11yProblems: async ({ page }, use) => {
  await use(async () => {
    const { violations } = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
      .analyze();
    return violations.flatMap((violation) =>
      violation.nodes.map((node) => `${violation.id}: ${node.target.join(' ')}`),
    );
  });
},

The fixtures so far handed the test a thing: a page, a board. This one hands it a function. The reason is timing. A fixture is built before the test starts, when the page is still empty, and axe checks the page as it is at the moment it runs. So the test has to say when: it brings the page into the state it cares about and calls findA11yProblems() then.

Inside, new AxeBuilder({ page }) prepares a check of this page, and analyze() runs it. withTags picks the rules. The four tags stand for the levels A and AA of WCAG 2.0 and 2.1, the Web Content Accessibility Guidelines, which is the standard that accessibility requirements usually refer to.

What comes back is a list of violations. Each one names the rule that was broken (id) and the elements that break it (nodes), each with its selector (target). The last lines turn that into one line of text per element, in the form rule: selector. A test can then expect an empty list, and when the list is not empty, the failure message is the list itself: which rule, and where.

Check states, not only the first page

Create tests/frontend/accessibility.spec.ts:

tests/frontend/accessibility.spec.ts
import { DRAGON, SWORD, expect, mockApi, test } from './fixtures.ts';

test('the board of a member', async ({ page, board, findA11yProblems }) => {
  await mockApi(page, 'GET', '/quests', { json: [SWORD, DRAGON] });
  await board.open();
  await expect(board.quests).toHaveCount(2);

  expect(await findA11yProblems()).toEqual([]);
});

test('the empty board', async ({ page, board, findA11yProblems }) => {
  await mockApi(page, 'GET', '/quests', { json: [] });
  await board.open();
  await expect(page.getByText('No quests yet.')).toBeVisible();

  expect(await findA11yProblems()).toEqual([]);
});

test('the board that could not be loaded', async ({ page, board, findA11yProblems }) => {
  await mockApi(page, 'GET', '/quests', { status: 500 });
  await board.open();
  await expect(board.problem).toBeVisible();

  expect(await findA11yProblems()).toEqual([]);
});

test.describe('for a visitor', () => {
  test.use({ loggedIn: false });

  test('the board with the login form', async ({ page, board, findA11yProblems }) => {
    await mockApi(page, 'GET', '/quests', { json: [SWORD, DRAGON] });
    await board.open();
    await expect(board.loginButton).toBeVisible();

    expect(await findA11yProblems()).toEqual([]);
  });
});

Four tests, four states of the same page: the board of a member with an open and a finished quest, the empty board, the board whose quests could not be loaded, and the board a visitor sees, with the login form. Problems like to sit in the states you rarely look at, such as an error message, and with a mocked API each of them is one line away. The visitor's state uses the option from the last part: test.use({ loggedIn: false }) makes the mocked /api/me answer 401 for the tests of this group.

Each test has one line between opening the page and checking it. It waits for something that is only there in the state the test means: two quests in the list, the sentence of the empty board, the alert, the login button. goto comes back when the page has loaded, and that can be before the answer of the API is on it. Without the waiting line axe might check a page that still says "Loading quests…", find nothing wrong with it, and the test would pass without having looked at what its name promises.

Run the new file. The word at the end of the command filters the test files by name:

questboard
$ npx playwright test --project=frontend accessibilityRunning 4 tests using 4 workers  ✓  1 [frontend] › tests/frontend/accessibility.spec.ts:19:1 › the board that could not be loaded (546ms)  ✓  3 [frontend] › tests/frontend/accessibility.spec.ts:11:1 › the empty board (555ms)  ✓  4 [frontend] › tests/frontend/accessibility.spec.ts:30:3 › for a visitor › the board with the login form (549ms)  ✘  2 [frontend] › tests/frontend/accessibility.spec.ts:3:1 › the board of a member (673ms)  1) [frontend] › tests/frontend/accessibility.spec.ts:3:1 › the board of a member    Error: expect(received).toEqual(expected) // deep equality    - Expected  - 1    + Received  + 3    - Array []    + Array [    +   "color-contrast: .done > label",    + ]       6 |   await expect(board.quests).toHaveCount(2);       7 |    >  8 |   expect(await findA11yProblems()).toEqual([]);         |                                    ^       9 | });      10 |      11 | test('the empty board', async ({ page, board, findA11yProblems }) => {        at /home/robin/questboard/tests/frontend/accessibility.spec.ts:8:36  1 failed    [frontend] › tests/frontend/accessibility.spec.ts:3:1 › the board of a member  3 passed (2.1s)

Three states pass and one does not, and the message says what and where: the rule color-contrast, on the element .done > label. Those are the finished quests. The page strikes them through and dims them, and it dims them so far that the text no longer stands out enough against the background. Level AA asks for a contrast of at least 4.5 to 1 for text of this size.

The visitor's board shows the same dim quest and passes all the same. A visitor's checkboxes are disabled, and the contrast rule of axe leaves disabled controls and their labels out.

The fix is one value in web/src/style.css. The colour was #5b4a78. A lighter one is readable, and the quest still looks finished, because it stays struck through:

web/src/style.css
li.done label {
  color: #a99bc9;
  text-decoration: line-through;
}

Run the command again, and all four tests are green. There is nothing to rebuild by hand: Playwright builds the frontend anew for every run.

What axe did not find

Look at the form of the board. The text field has a label, "New quest". The number field beside it has none. It has a placeholder, "Reward", and a placeholder shows only while a field is empty. This field never is: it starts at 10 and goes back to 10 after every new quest. So what people see is a box with a number in it, and nothing that says what the number is for.

axe does not complain, and by its rules it is right. A field must have a name that assistive technology can announce, and the placeholder gives it one. The machine is satisfied. A person looking at the page is not.

The test code has been giving a hint all along. The page object from part 7 finds the title field the way a person does, by its label. For this field it had to write page.getByPlaceholder('Reward'). When a test has to reach for something other than a role or a label to find an element, ask why: often the page lacks something a person would need too.

Give the field a label that is always there. In web/src/App.vue, the form for a new quest becomes:

web/src/App.vue
<form @submit.prevent="add">
  <label>
    New quest
    <input v-model.trim="title" maxlength="80" required />
  </label>
  <label class="narrow">
    Reward
    <input v-model.number="reward" type="number" min="0" max="1000" required />
  </label>
  <button>Add quest</button>
</form>

The number field moves into a label of its own with the visible text "Reward", and the placeholder goes. The labels of this form share the width of the row between them. The class narrow keeps the new one as wide as its field. Add the rule to web/src/style.css, below the one for form label:

web/src/style.css
form label.narrow {
  flex: none;
}

Now every test that fills in a reward would fail, because nothing on the page has the placeholder "Reward" any more. This is where the page object pays off. None of those tests says how the field is found, so one line in tests/support/board.ts is the whole repair:

tests/support/board.ts
this.reward = page.getByLabel('Reward');

The page without the picture

axe checks rules. There is a second, more direct question: what does someone get who does not see the page at all? A screen reader does not work with pixels. It works with what the browser derives from the page for it: the elements that matter, each with a role (heading, text field, button, list) and a name. Playwright can write that down as text, an ARIA snapshot, and compare it with a template in the test. Add one more test at the end of tests/frontend/accessibility.spec.ts:

tests/frontend/accessibility.spec.ts
test('the board reads well without the picture', async ({ page, board }) => {
  await mockApi(page, 'GET', '/quests', { json: [SWORD, DRAGON] });
  await board.open();

  // What a screen reader gets to work with: roles and names, in the order of the page.
  await expect(page.getByRole('main')).toMatchAriaSnapshot(`
    - heading "Quest Board" [level=1]
    - textbox "New quest"
    - spinbutton "Reward"
    - button "Add quest"
    - list "Quests":
      - listitem:
        - checkbox "Find the lost sword"
        - button "Delete Find the lost sword"
      - listitem:
        - checkbox "Feed the dragon" [checked]
        - button "Delete Feed the dragon"
    - paragraph: 1 of 2 quests open
  `);
});

The template inside toMatchAriaSnapshot reads like an outline of the page: a heading of level 1 called "Quest Board", a text field "New quest", a spinbutton "Reward" (that is the role of a number field), the button "Add quest", then a list called "Quests" with two items. Each item has a checkbox named after its quest and a button "Delete …", the second checkbox is [checked], and below the list stands the count. Roles and names, in the order of the page. They are the same roles and names your locators have used since part 2, which is why tests written with getByRole and getByLabel have been saying something about accessibility all along.

The comparison is strict about what the template lists and silent about what it leaves out. The line "Logged in as robin", the button "Log out" and the gold of each quest are on the page and not in the template, and the test does not mind. So a template can stay with what matters. Here that is: the fields of the form have names, every quest can be ticked and deleted by its name, and a finished quest is announced as checked. If someone removes the aria-label from the delete buttons, all of them are called "✕", and this test says so.

Like the other assertions on locators, this one retries until the page matches, so the test needs no waiting line of its own. Run the file once more:

questboard
$ npx playwright test --project=frontend accessibilityRunning 5 tests using 5 workers  ✓  3 [frontend] › tests/frontend/accessibility.spec.ts:39:1 › the board reads well without the picture (217ms)  ✓  2 [frontend] › tests/frontend/accessibility.spec.ts:30:3 › for a visitor › the board with the login form (578ms)  ✓  4 [frontend] › tests/frontend/accessibility.spec.ts:11:1 › the empty board (557ms)  ✓  5 [frontend] › tests/frontend/accessibility.spec.ts:19:1 › the board that could not be loaded (564ms)  ✓  1 [frontend] › tests/frontend/accessibility.spec.ts:3:1 › the board of a member (603ms)  5 passed (2.0s)

What you have now

The frontend section checks four states of the board against the rules axe knows, and one more test pins down what assistive technology is told about the page. On the way the board itself got better twice: finished quests are readable, and the reward field says what it is.

All of that is about what the page means. Whether it still looks right after a change to the CSS, no test so far can tell. In the next part the suite takes pictures of the board and compares them.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.