Our accessibility testing methodology – and where it stops

Reviseberg tests websites with five kinds of evidence: automated rules (axe-core) on every crawled page, a keyboard agent that presses real keys, checks of our own that re-render the page and, on a crawl from an account, operate it, results people record by hand, and documented decisions from your team. From these, each of the 55 WCAG 2.2 Level A and AA success criteria gets a status: passed, failed, partial, untested or not applicable. Anything a machine cannot decide stays "partial" or "untested" – it is never quietly counted as a pass.

This page explains how that works, how the score is calculated and where automated accessibility testing reaches its limits. Every number on it comes from the same code the product tests and scores with.

The methodology at a glance

The methodology at a glance
Source of evidenceWhat it testsWhat it cannot do
Rules (axe-core)Measurable facts in the rendered page: missing text alternatives, contrast, missing names and roles, page languageJudge meaning – whether alt text actually describes the image, for instance
Keyboard agentReachability by Tab, visible focus, skip links, keyboard trapsDecide whether an order makes sense
Our own checks (probes)The page at 320 px, at 200% zoom and with wider text spacing; contrast against the pixels behind text; on a crawl from an account, controls, menus, hover content and forms operated with test dataProve a pass: each reports a failure only where it measured one, and a question where a person has to decide
Manual and guided testsEverything that needs human judgement– (valid for 12 months, then retest)
Your team's decisionsFalse positives, deliberately ignored findings, findings that can't be fixed – with a reason–

Step 1: Crawl and rule-based testing with axe-core

Reviseberg crawls your site and runs axe-core, the widely used open-source testing library (currently version 4.14.0), on every page it finds. It tests the rendered page in a real browser, not the source files – and it tests it twice: at 1,280 pixels wide with every WCAG 2.2 A and AA rule, and again at 360 pixels with the rules that only fail once the layout reflows: color-contrast, target-size, meta-viewport and scrollable-region-focusable. Running everything twice would report the same markup problems twice; running these twice catches the contrast, target sizes and overflow a phone actually gets.

Besides "violation" and "pass", axe-core has a third outcome: cases a rule cannot settle, such as text over a background image. These go into the Potential issues list, where a person decides. We report them neither as failures nor as passes, and they cost no points.

Identical violations are grouped into one issue: one rule, the WCAG criteria it affects, the number of occurrences and pages. An issue counts as fixed when its findings stop appearing on a new crawl – not when someone ticks a box. That is why there is no "done" status.

Step 2: The keyboard agent

Rules see a single state of a page. Whether a page can be operated by keyboard only shows when someone operates it. So our keyboard agent walks the pages you name as the site's journey – from basket to checkout, say – with Tab, Shift+Tab and Escape. It drives Chromium with real key events, never a scripted focus call: a scripted call reaches elements the tab order does not, and that gap is exactly the bug it is looking for.

It reports things that behave like controls but no Tab reaches (in tabs, radio groups, toolbars, menus and listboxes, only after pressing the arrow keys from the widget's one Tab stop), focus that lands on something with no size or off-screen, focus that changes nothing visible, missing skip links and keyboard traps. It only calls something a trap once Tab, Shift+Tab and Escape have confirmed it. For every step it keeps a trail as evidence: the key pressed, the element that received focus with its role and name, the verdict with a note saying why and, on failure, a screenshot. Where the styles cannot show whether focus is visible, the verdict is "human" rather than a guess.

This one agent is the only one running today. The further agents on the agents page (screen reader, zoom and reflow, forms, voice control and others) would use a page the way a person with that need does; they are in development, labelled as such, and feed into no status. What runs beside the keyboard agent today is narrower: the checks in the next step.

Step 3: Our own checks beside axe

Rules read the page as it loaded, and the agent walks one journey. Much of WCAG only shows when a page is changed or used, so a crawl also runs checks of our own, which we call probes. Each reports a failure only where it measured one, and a question where a person has to decide:

  • Rendering, on every page: the page at 320 pixels wide (1.4.10), at 200% zoom (1.4.4) and with WCAG's text spacing (1.4.12), read for sideways scrolling, text cut off or running into other text, and content that disappears; both screen orientations (1.3.4); contrast against the pixels behind the text and in hover and focus states (1.4.3); the edges of form fields and icon-only controls (1.4.11); small markers told apart by colour alone (1.4.1, a question); a way past the navigation (2.4.1); targets in menus that open on hover or focus (2.5.8); and whether the declared language is the one the page is written in (3.1.1).
  • Interaction, on a crawl from an account, on one page of each template the crawl finds (up to a limit per crawl): the keyboard walk, and what pressing each control does; content that appears on hover or focus (1.4.13); single-key shortcuts (2.1.4); gestures, dragging, pointer cancellation and device motion (2.5.1, 2.5.2, 2.5.4, 2.5.7) and visible labels in accessible names (2.5.3); focus hidden under sticky bars (2.4.11), focus order (2.4.3), reading sequence (1.3.2) and changes on focus (3.2.1); names, roles and states (4.1.2) and status messages (4.1.3).
  • Time and media: motion and carousels with no pause (2.2.2), time limits (2.2.1), flashes (2.3.1), sound that plays on its own (1.4.2), and captions, transcripts and audio description where the page's own player or an embedded player's public information says (1.2.1 to 1.2.5) – most of these are questions. A third party's video is never downloaded or played.
  • Forms, on a crawl from an account: labels and instructions (3.3.2), input purpose (1.3.5), changes on input (3.2.2), how errors are shown and what they suggest (3.3.1, 3.3.3), error prevention (3.3.4), redundant entry (3.3.7) and sign-in (3.3.8). Only test data is typed, and every write is refused unless an Admin has allowed form submissions, which is meant for a staging copy of the site. What the server answers is only seen then; otherwise that part stays a question.
  • The whole site: navigation, names and help kept in the same order and place across pages (3.2.3, 3.2.4, 3.2.6), titles that are unique and say what the page is (2.4.2), and more than one way to reach a page (2.4.5).

The free scan runs only the checks that look – rendering, time and media, the whole site. The ones that press, hover or type run only on a crawl from an account. An AI review that asks a model about alt text, link purpose, headings and labels and the like is built but off: it needs a model configured for the service, a paid plan and a site Admin who turns it on, and every answer it gives is a question, never a failure.

Questions – axe's, these checks' and the AI review's – go to Potential issues, where a person answers "Fails" or "Passes". The answer is kept for that element and applied again on every later crawl until the element changes: "Fails" makes it a failure, and "Passes" takes it off the list without ever making the criterion pass.

How far each check reaches is measured rather than asserted. A corpus of test pages fails each criterion in the ways we know of – including ways we know we cannot catch – or passes it while looking as if it fails, and a criterion's label in the WCAG 2.2 checklist has to be earned there. A check that finds nothing does not make a criterion pass.

Step 4: A status for every WCAG criterion

From rules, agent, our own checks, manual results and decisions we calculate a status for each of the 55 WCAG 2.2 Level A and AA success criteria:

  • Passed: no open violation, and either an automated check can decide the criterion completely or a person has recorded a pass.
  • Failed: at least one open violation. A crawl that finds a violation outranks a pass recorded by hand – the evidence disagrees with the tick.
  • Partial: the automated checks found nothing, but they only cover part of the criterion. The rest needs human judgement.
  • Untested: no automated check reaches the criterion and no valid human result has been recorded.
  • Not applicable: the content type does not occur (no video, for example) – only a person can record that.

Our WCAG 2.2 checklist sorts every criterion into one of three classes: automated (0 criteria), partly automated (52) and manual only (3). The class is the measurement from step 3, not a guess, and it moves when the measurement does.

How far an automated check gets, by WCAG principle Of the 55 WCAG 2.2 success criteria at level A and AA, an automated check can decide 0, can decide 52 only in part, and can say nothing useful about 3.
Show the numbers as a table
How far an automated check gets, by WCAG principle
PrincipleAutomatedPartly automatedManual onlyTotal
Perceivable018220
Operable019120
Understandable013013
Robust0202
Screenshot of the WCAG coverage screen in the Reviseberg dashboard: a card for each of the four principles with a percentage and the number of failed, partial, untested and passed criteria, then filters and a legend of the statuses.
The criteria overview in the dashboard: what nobody has checked is shown as “Untested”, never as passed. Example data from the demo dashboard

Some axe-core rules test no success criterion at all but a good practice – landmarks, heading hierarchy, a page with no h1. We show those findings, because they are real obstacles, but as best practice: they fail no criterion, never reach the statement, and cost less in the score than a WCAG failure (see below).

Why "untested" is never "passed"

Many scanners end with a green number even though they never looked at half the requirements. A blank then reads like a pass – convenient, wrong, and expensive once an accessibility statement is built on it. A scanner that reports a manual-only criterion as passing is really reporting that it did not look. That is why Reviseberg shows how many criteria are untested, and why the accessibility statement can only claim what the evidence supports. While any criterion is partial or untested, the status reads "partially compliant" – never "fully compliant".

Step 5: Manual and guided tests

For criteria no automation can settle, Reviseberg offers two routes to the same evidence:

  • Manual test: someone with the expertise records a result – for 1.2.2 Captions or 1.3.2 Meaningful Sequence, for example. A method is required ("NVDA + Firefox", "keyboard only"), and a failure has to say what failed.
  • Guided test: 34 criteria – the ones a crawl reaches least far, or not at all – each have a step-by-step procedure: when the criterion applies at all, what to do, what to look for and what counts as a failure. Written for people who are competent but not accessibility specialists.

Every recorded result carries a name and a date, is added to rather than overwritten, and expires after 12 months. Websites change; a judgement from two years ago is not evidence for today. An expired result stays visible in the history, but the criterion falls back to what the crawl alone shows – "partial" or "untested" – and the status in reports and the statement updates accordingly.

How the score is calculated

The accessibility score answers one question: how many automatically detected barriers are open right now, and how serious are they? Every open issue costs points, and the score is what is left of 100 once they come off:

  • Points = 7.4 × severity × WCAG level × reach
  • Score = 100 × e^(−P ÷ 100), rounded, where P is the sum of the points of all open issues
How the score is calculated
FactorWeighting
Severity (the axe-core impact, or the agent's rating)critical 1 · serious 0.68 · moderate 0.4 · minor 0.18
WCAG levelA 1 · AA 0.85 · best practice (no criterion) 0.3
Reach (n = occurrences)0.4 + log₁₀(1 + n) ÷ log₁₀(1 + 1,400)

A best-practice finding weighs less than a AAA failure: it is a real obstacle, but a tool's advice rather than a requirement the W3C wrote down. Reach grows on a log scale: a template error on every page weighs more than a one-off, but not a thousand times more. It stops growing at 1,400 occurrences.

In words: each point takes about one percent of what is left. At the top that is nearly the same as plain subtraction – open issues costing 5 points leave a score of 95 – but the score falls ever more slowly and never sticks at 0. Up to v3 the points were simply subtracted, never below 0, and a site with a great deal open stayed at 0 whatever it fixed. On the curve every fix moves the score: issues costing 130 points leave 27. It rounds to 0 only past about 530 points.

For every issue we show the points back: how far the score rises when that one issue is fixed, from where it stands today. The issue list is sorted by them, so you can see what moves the score most per hour of work. Because of the curve, the same fix gives back less the more else is open. An example: "Focus not visible" (2.4.7, AA, serious) with 17 occurrences costs 3.42 points; with nothing else open, fixing it gives back 3.36.

Every stored score carries the version of the formula, currently v4. When a weight changes, the version changes – v2, for instance, moved best-practice findings from Level A to their own, lower weight – so a trend shows where the formula moved instead of pretending the site did. v4 brought the curve, and the scores stored before it were recomputed from each crawl's findings, so a trend is not a row of zeros no fix could move. Quality (broken links, spelling, readability) and SEO are scored separately, with the same arithmetic minus the WCAG level; they never move the accessibility score.

Since v3, linked PDFs get a documents score of their own, by the same arithmetic, WCAG level included; the accessibility score covers the pages. The people who fix a PDF are rarely the people who fix a template, and we open a sample of the documents a site links, so a mixed score moved with the sample. The criteria a PDF fails still count in the criteria overview and the statement: a document your site offers is part of its conformance.

The score is not a conformance claim. A site scoring 100 can still fail criteria only a person can check. That is why the criteria overview, with every "untested" and "partial" entry, always sits next to the score.

Decisions with a reason

Not every finding is correct or fixable straight away. Your team can mark findings as a false positive, ignored or can't fix – for one element, one page or the whole site, until a date if you like, and always with a reason. Who decided and when is recorded; a Viewer cannot record a decision.

A finding decided this way costs no points and no longer fails its criterion. That is exactly why the reason is required: the decision is part of the evidence, and it is the first thing an auditor will ask about.

Mapping to EN 301 549 and the BITV-Test

EN 301 549 is the European standard for accessible information and communication technology. Its chapter 9 (web) adopts the WCAG success criteria under its own numbering: WCAG 2.1.1 Keyboard becomes clause 9.2.1.1. Reviseberg therefore also reports clause by clause against EN 301 549 V3.2.1, from the same criteria [2].

V3.2.1 is based on WCAG 2.1. The criteria new in WCAG 2.2 – 2.4.11, 2.5.7, 2.5.8, 3.2.6, 3.3.7 and 3.3.8 – have no clause in it and appear in our reports as WCAG results without a clause number. In September 2026 ETSI published V4.1.1, which adopts WCAG 2.2 [3][4]. Our reports are not mapped to it yet.

Many German auditors work with the BIK BITV-Test, which splits requirements into finer test steps (Prüfschritte) [5]. Reviseberg does not map its findings to those steps. A BITV-Test is a manual expert evaluation; Reviseberg doesn't replace it – it covers what can be automated and keeps the evidence current between audits.

The limits of automated testing

We are clear about what our automation does not do:

  • Meaning: whether alt text, a heading or a link text fits its content is a human call. AI suggestions for alt text exist – only what a person accepts is kept, and nothing is applied automatically.
  • Comprehension: the quality of plain language and editorial intent.
  • Media: the quality of captions, audio description and sign language.
  • Real transactions: flows that need real payment or real identity documents.
  • Assistive technology: no agent checks how a screen reader actually announces the page today. The screen reader agent is in development.
  • Documents: we check the PDFs your site links to for their structure only: whether they are tagged, whether the tags hold content, the document language, a displayed title and figure alternative text. Whether the reading order is right, tables, contrast and form fields inside a PDF are not checked, and Office files are not opened.
  • Legal sign-off: the accessibility statement is yours to sign; we are not a law firm.

Run a free scan and see the status of your criteria

Who is behind the methodology

One of our founders has worked as a UX designer and frontend developer since 2013 and has spent years testing websites against WCAG, BITV and BFSG for agency clients in tourism, the public sector and ports. The methodology above is how he would test by hand – only on every page instead of a sample. Questions or objections about the methodology: hello@reviseberg.com.

Changes to the methodology

When a weight, the classification of a criterion or the status logic changes, it is recorded here – with the date and the scoring version it applies from.

Changes to the methodology
DateScoring versionWhat changed
v4Scores come off 100 along a curve instead of a straight line: each point lost takes about one percent of what is left, in every module. Healthy sites read about what they did — 5 points lost is still 95 — but a site with a great deal open no longer sits at 0 while it fixes things. Points back are now what one fix gives back from where the score stands today. Scores stored before the change were recomputed from each crawl’s findings.
v3Beside axe-core and the keyboard agent, a crawl runs checks of our own: every page at 320 px, at 200% zoom and with wider text spacing, contrast against the pixels behind text, media and time limits, and the site’s pages compared with each other; on a crawl from an account also the keyboard walk on one page per template, and controls, hover content and forms operated with test data. Each reports a failure only where it measured one, and a question where a person has to decide. How far a check reaches each criterion is now measured on a corpus of failing and passing test pages rather than assigned: the criteria these checks reach moved from manual to partial, and the four labelled automated (1.3.4, 1.4.3, 2.4.2, 3.1.1) are partial, because finding nothing never proved them met. No criterion is labelled automated.
v3Linked PDFs get a documents score of their own, by the same formula, and no longer move the accessibility score, which now covers the pages. Their issues are still listed with the others, and the criteria they fail still count in the criteria overview and the accessibility statement.
v2Methodology published: axe-core at two widths, the keyboard agent, a status for every criterion in the catalogue, manual results valid for twelve months, and the score formula on this page, including best-practice findings at their own, lower weight.

Frequently asked questions

How much of accessibility can an automated tool test?

Less than most scanners suggest. Our checks reach 52 of the 55 criteria at Levels A and AA, and none of them completely: a check can find a failure, never prove a pass. 3 are out of their reach altogether. Reviseberg shows which class each criterion falls into and what evidence exists.

Why does my site score 100 but read "partially compliant"?

The score measures open, automatically detected barriers. Conformance also depends on criteria only a person can check. Until those are recorded they remain "partial" or "untested", and the statement reads "partially compliant".

What does "points back" mean?

How far your score rises when you fix a particular issue, from where it stands today. The lower the score, the less one fix moves it – but it never sticks at 0 while you are fixing things. The issue list is sorted by it, so you start with what matters most.

Why do manual test results expire after 12 months?

Websites change constantly. An old judgement is not evidence of today's state. After 12 months the criterion falls back to "partial" or "untested" until it is tested again. The result itself stays in the history.

Does Reviseberg replace a BITV-Test or an expert audit?

No. Reviseberg covers what can be automated, guides you through manual tests and keeps results current between audits. An expert review of the criteria that need judgement is still worth doing.

Do you test against WCAG 2.1 or 2.2?

WCAG 2.2, all 55 success criteria at Levels A and AA, with the EN 301 549 clause for every criterion V3.2.1 already contains. WCAG 2.2 includes every WCAG 2.1 criterion except 4.1.1 Parsing, which was removed.

Sources

  1. W3C: Web Content Accessibility Guidelines (WCAG) 2.2 – https://www.w3.org/TR/WCAG22/
  2. ETSI: EN 301 549 V3.2.1 (2021-03) – https://www.etsi.org/deliver/etsi_en/301500_301599/301549/03.02.01_60/en_301549v030201p.pdf
  3. ETSI: EN 301 549 V4.1.1 (2026-09) – https://www.etsi.org/deliver/etsi_en/301500_301599/301549/04.01.01_60/en_301549v040101p.pdf
  4. AccessibleEU: "The European accessibility standard EN 301 549 has been updated" (7 Sep 2026) – https://accessible-eu-centre.ec.europa.eu/content-corner/news/european-accessibility-standard-en-301-549-has-been-updated-2026-09-07_en
  5. BIK BITV-Test: test steps BIK BITV-Test + WCAG 2.2 (Web) – https://bitvtest.de/pruefverfahren/bitv-20-plus-web
  6. Deque: axe-core rule descriptions – https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md
  7. W3C: Website Accessibility Conformance Evaluation Methodology (WCAG-EM) – https://www.w3.org/TR/WCAG-EM/

See what a scan finds on your site

One page in about thirty seconds, no email needed. The full report covers up to 100 pages and a keyboard journey.