Playwright & CI guide

What this is

The extension's interface lives in a DevTools panel, which test runners cannot drive. But the scanning engine underneath is ordinary code that reads the page it runs in. The CI bundle, a11y-scanner.js, is that engine as a single file: no extension, no browser-extension APIs. Inject it into a page from Playwright (or Puppeteer, Selenium or any other runner), call a global function, and assert on the results.

It is the same engine and the same code path as the panel. A failure in CI has the same rule id, selector and suggested fix you would see in DevTools, so you can reproduce it by opening the panel on the same page.

1. Get the bundle

The bundle is one file, a11y-scanner.js. Put it in your test project, for example:

tests/
  a11y/
    a11y-scanner.js
  dashboard.spec.ts

It is an IIFE script (not an ES module), so it works with page.addScriptTag({ path }) without a server or an import map. Once injected, it exposes window.a11yScanner.

If you build the extension from source, the bundle is produced by:

npm run build:ci     # writes dist-ci/a11y-scanner.js

2. Your first test

import { test, expect } from '@playwright/test';
import path from 'node:path';

const SCANNER = path.resolve(__dirname, 'a11y/a11y-scanner.js');

test('the dashboard has no accessibility violations', async ({ page }) => {
  await page.goto('https://staging.example.com/dashboard');

  // Wait for the page to settle (see step 3).
  await page.waitForLoadState('networkidle');

  await page.addScriptTag({ path: SCANNER });
  const violations = await page.evaluate(() => window.a11yScanner.violations());

  expect(violations).toEqual([]);
});

Assert on the array, not its length. When the test fails, Playwright then prints exactly what is wrong. A test that only says "expected 0, got 7" is a test people learn to skip.

Each violation has this shape:

{ rule, severity, wcag, element, message }

• rule: the rule id, for example color-contrast or a11y-icon-button-name
• severity: critical, serious, moderate or minor
• wcag: the success criteria, for example 1.4.3 AA, or best practice
• element: a CSS selector for the element
• message: what is wrong

violations() returns confirmed failures only. It leaves out "needs review" findings (cases the engine cannot decide) and anything you have marked as ignored.

3. Scan a settled page

The largest source of false positives is scanning a page that has not finished loading. Skeleton loaders are grey-on-grey by design, and a background image that has not arrived yet can turn a contrast result from "undecidable" into "failing".

Wait for something that proves the page is ready, not just for the network:

await page.goto('/orders');
await page.getByRole('heading', { name: 'Orders' }).waitFor();
await expect(page.getByTestId('orders-table')).toBeVisible();
await expect(page.locator('.skeleton')).toHaveCount(0);

4. Scan every frame

The bundle scans the document it is injected into. Pages with embedded checkouts, video players or widgets have more than one document, and Playwright gives you each of them. Scan them all and combine the results, which is what the extension does internally:

const all = [];
for (const frame of page.frames()) {
  await frame.addScriptTag({ path: SCANNER });
  const found = await frame.evaluate(() => window.a11yScanner.violations());
  all.push(...found.map((v) => ({ ...v, frame: frame.url() })));
}
expect(all).toEqual([]);

5. A reusable helper

Put the scan in one place so every test scans the same way:

// tests/a11y/scan.ts
import { expect, type Page } from '@playwright/test';
import path from 'node:path';

const SCANNER = path.resolve(__dirname, 'a11y-scanner.js');

export async function expectNoViolations(page: Page, options = {}) {
  const all = [];
  for (const frame of page.frames()) {
    await frame.addScriptTag({ path: SCANNER });
    const found = await frame.evaluate((opts) => window.a11yScanner.violations(opts), options);
    all.push(...found.map((v) => ({ ...v, frame: frame.url() })));
  }
  expect(all).toEqual([]);
}

import { expectNoViolations } from './a11y/scan';

test('checkout is accessible', async ({ page }) => {
  await page.goto('/checkout');
  await expectNoViolations(page, { standard: 'wcag21aa' });
});

6. Options

Both violations(options) and scan(options) take the same options:

Option	Default	What it does
standard	'wcag22aa'	The conformance target (see below)
bestPractice	true	Include axe-core's best-practice rules, beyond what WCAG requires
needsReview	true	Include findings the engine could not decide. They are never counted as violations
passedChecks	false	Record what passed. Off by default because it adds about a third to scan time
contrastFixer	true	Work out the nearest passing colour for contrast failures
scanShadowDom	true	Walk open shadow roots, so web components are checked
ignoredRules	[]	Rule ids to mark as ignored rather than failing
ignoredSelectors	[]	CSS selectors whose findings are marked as ignored

Conformance targets

standard value	Target
wcag20a, wcag20aa	WCAG 2.0 Level A, AA
wcag21a, wcag21aa, wcag21aaa	WCAG 2.1 Level A, AA, AAA
wcag22a, wcag22aa, wcag22aaa	WCAG 2.2 Level A, AA (default), AAA
section508	Section 508 (Revised): the WCAG 2.0 AA set it adopts
en301549	EN 301 549: the WCAG 2.1 AA set clause 9 adopts
ttv5	Trusted Tester v5
rgaa4	RGAA 4
all	Every available check, including experimental and AAA

The standard changes which checks run, not what is filtered afterwards, exactly as in the extension.

7. Known issues and severity gates

Adopting accessibility tests on an existing app usually means starting with some known problems. Mark them as ignored, and they are reported as ignored rather than failing the build:

await page.evaluate(() => window.a11yScanner.violations({
  ignoredRules: ['color-contrast'],                 // tracked in TICKET-123
  ignoredSelectors: ['#legacy-widget', '.third-party-chat'],
}));

To fail the build only on the most serious problems at first, filter in your test:

const violations = await page.evaluate(() => window.a11yScanner.violations());
const blocking = violations.filter((v) => ['critical', 'serious'].includes(v.severity));
expect(blocking).toEqual([]);

Tighten the gate over time by removing ignores and adding severities.

8. Save the full report

scan() returns the full report, in the same shape the extension produces and exports:

{ url, title, timestamp, framework, target, scope, summary, issues, passes, notes, durationMs }

Each entry in issues carries a kind (a confirmed violation, or needs review), a status (open or ignored), a severity, the ruleId, the selector, the message and the WCAG criteria.

Attach it to the test result, so every CI run keeps a readable record even when it passes:

test('dashboard report', async ({ page }, testInfo) => {
  await page.goto('/dashboard');
  await page.addScriptTag({ path: SCANNER });
  const report = await page.evaluate(() => window.a11yScanner.scan());

  await testInfo.attach('a11y-report.json', {
    body: JSON.stringify(report, null, 2),
    contentType: 'application/json',
  });

  // Same rule as violations(): open, confirmed violations only (not needs-review or ignored)
  const critical = report.issues.filter((i) => i.kind === 'violation' && i.status === 'open' && i.severity === 'critical');
  expect(critical).toEqual([]);
});

9. Test states, not just pages

A page-load scan only sees the first state of the page. Drive the page into the states that matter and scan each one: an open menu, a dialog, a validation error.

test('the new-project dialog is accessible', async ({ page }) => {
  await page.goto('/projects');
  await page.getByRole('button', { name: 'New project' }).click();
  await expect(page.getByRole('dialog')).toBeVisible();

  await expectNoViolations(page);
});

test('form errors are accessible', async ({ page }) => {
  await page.goto('/signup');
  await page.getByRole('button', { name: 'Create account' }).click();   // submit empty
  await expect(page.getByText('Email is required')).toBeVisible();

  await expectNoViolations(page);
});

In the extension, User flows do the same thing interactively: record a journey once and replay it with every state scanned.

10. Many pages

const pages = ['/', '/pricing', '/docs', '/account/settings'];

for (const url of pages) {
  test(`no violations on ${url}`, async ({ page }) => {
    await page.goto(url);
    await page.waitForLoadState('networkidle');
    await expectNoViolations(page);
  });
}

For pages behind a login, sign in once with Playwright's storageState and reuse it in every test.

11. Run it in CI

The scan runs inside the browser Playwright already controls, so there is nothing extra to install. A GitHub Actions workflow looks like any Playwright workflow:

name: accessibility
on: [pull_request]
jobs:
  a11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test tests/a11y
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/

The attached JSON reports (step 8) end up in the uploaded Playwright report.

12. Other runners

Any tool that can add a script to a page and evaluate code works the same way.

Puppeteer:

await page.addScriptTag({ path: 'tests/a11y/a11y-scanner.js' });
const violations = await page.evaluate(() => window.a11yScanner.violations());

Selenium WebDriver (JavaScript):

const fs = require('node:fs');
await driver.executeScript(fs.readFileSync('tests/a11y/a11y-scanner.js', 'utf8'));
const violations = await driver.executeScript('return window.a11yScanner.violations()');

Troubleshooting

Symptom	Cause and fix
window.a11yScanner is undefined	The script was blocked, usually by a strict Content Security Policy. In Playwright, set test.use({ bypassCSP: true }) for accessibility tests
Contrast failures on placeholders or skeletons	The page was scanned before it finished loading. Wait for real content (step 3)
A widget's problems are missing	It is inside an iframe. Scan every frame (step 4)
Results differ from the panel	Check the standard and other options match the panel's settings. The panel's default target is WCAG 2.2 AA, the same as the bundle's

What a green build means

Automated checks find a meaningful subset of accessibility problems, commonly cited as a third to a half of WCAG failures, not all of them. A passing test means no machine-detectable failures, not an accessible page. Captions, meaningful alt text, reading order and whether an interaction makes sense still need a person, which is what the extension's guided manual checks are for.