Book a call

Testing an Accessible AI Chatbot Before Launch

An accessible AI chatbot has to work as a complete conversation. Test the launcher, dialogue, changing messages, input, recovery and alternate route before anyone depends on it.

Accessibility has to cover the whole conversation

The chatbot looks ready in a desktop demo. Its launcher has a label and an automated scan reports no obvious error. Then a keyboard user opens it and loses their place. A screen reader misses the reply. At 400 per cent zoom, the message field sits behind the on-screen keyboard. A timeout clears the person's work before they can finish.

Those failures sit inside the conversation, not only in the widget's static markup. The test boundary has to run from the launcher through opening, reading, typing, waiting, correcting, escalating and closing. It also includes the host page and the alternate route offered when chat cannot serve the person.

This guide assumes you've already selected a text chatbot. If you're still deciding, start by choosing between a chatbot, guided form and live chat. The wider page structure, content and forms still need strong website accessibility foundations. Here, we're testing the conversational component and its immediate joins.

WCAG 2.2 defines testable success criteria for accessible web content across desktop and mobile devices. It also makes clear that conformance does not address every user need. The Australian Human Rights Commission's digital accessibility guidelines identify WCAG 2.2 as the relevant international technical standard for assessing digital access in Australia. An organisation should assess its own obligations and seek advice suited to its circumstances.

Turn standards into observable requirements

A delivery team cannot release against “make the chatbot accessible”. It needs requirements that name a trigger, an expected behaviour, the evidence to collect and the consequence of failure.

WCAG success criteria are normative requirements within the standard. W3C Understanding documents, techniques and the WAI-ARIA Authoring Practices Guide explain possible ways to meet accessibility needs, but they are informative. Your widget configuration and code are implementation choices. Keyboard runs, screen-reader output, screenshots and user observations are test evidence.

Keep those layers separate. A requirement might say that activating the chat launcher by keyboard moves focus to a meaningful location inside the dialogue. The implementation might use a native button and a modal dialogue pattern. The evidence might be a recorded keyboard run in named browser and assistive-technology combinations. A single result or checklist doesn't prove WCAG conformance.

Translate the relevant WCAG 2.2 requirements into observable outcomes for keyboard operation, focus order and visibility, names and labels, error identification, adjustable timing, text resize, reflow and status messages. Add product requirements where the standard does not prescribe the experience, such as retaining a draft when the connection fails.

Make opening and closing predictable

Start with the launcher. Its accessible name should explain its purpose, and its visible label should agree with that name. A person must be able to reach and activate it using a keyboard without triggering an unexpected change.

Many chat widgets open as modal dialogues. The informative WAI-ARIA modal dialogue pattern says a modal needs dialogue semantics and an accessible name. Focus moves into it when it opens, Tab and Shift+Tab remain within its operable controls, and Escape generally closes it. When it closes, focus usually returns to the control that opened it or another logical location.

Test the behaviour rather than accepting the presence of ARIA attributes as proof. Open the chat with Enter and Space. Confirm that focus lands on the dialogue heading, introductory content or first suitable control according to the content. Move forward and backward through every interactive element. Check that focus remains visible and is not hidden behind the widget edge or a sticky element. Confirm that background content is not presented as operable when the chat is modal.

Then close the chat with its visible control and any documented keyboard command. Focus should return to the launcher unless the launcher's removal or the completed task makes another destination more logical. Reopen it and check whether the promised conversation state remains available.

A worked acceptance case can stay simple. Given focus is on the closed launcher, when the user presses Enter, the named chat dialogue opens and focus moves inside it. When the user closes the dialogue, focus returns to the launcher. Failure blocks release because the person can lose their place or become trapped.

Keep conversation updates perceivable and focus stable

A conversation changes while the person reads or types. Sent-message confirmation, processing states, new replies, validation errors, connection failures and completion all need to be perceivable. Moving focus to every update would interrupt reading and make typing unstable.

The W3C explanation of Status Messages under WCAG 2.2 covers changes that assistive technology should detect without receiving focus. It includes results, waiting states, errors and progress information. For a chatbot, that means the implementation needs programmatically determinable status behaviour for important updates while leaving the person's working position intact.

Send one short message with a screen reader running. Confirm the interface indicates that it was sent. Listen for a processing state if that state matters. Confirm the reply is announced once, in a useful order, without reading the entire transcript again. Keep focus in the input unless the interaction genuinely requires a move.

Repeat the test with a long reply, rapid consecutive replies and an error. Look for silence, duplicate announcements and excessive verbosity. A live region that announces every token can make a streamed answer unusable. The product team needs to decide when a response is stable enough to announce and how someone can navigate back through earlier messages.

Protect reading input and recovery

People need to read the conversation and keep control of what they've entered. Test text resizing and reflow at 200 and 400 per cent zoom where the relevant WCAG criteria apply. At narrow widths, the launcher, close control, transcript, input and send control should remain available without two-dimensional scrolling for ordinary text content.

Run the same task on a small mobile viewport with the on-screen keyboard open. Check that the current message, input, error and send control stay visible or can be reached without fighting the page. Rotate the device where the experience supports both orientations. Increase text spacing and confirm that labels and controls don't overlap or disappear.

Errors need a clear identification and a recovery path. Submit an empty message, exceed any disclosed input limit and simulate a network failure. The interface should explain what happened in text, associate the message with the affected input where appropriate and preserve the person's draft whenever it can. Correction should not require starting the whole conversation again.

Timing needs an equally deliberate test. If inactivity will end the session or remove information, warn the person and offer enough control to extend it where required. Test what happens when the warning receives no response, when the extension is activated by keyboard and when the connection returns after interruption. A transcript or recovery option must not expose another person's conversation on a shared device.

Keep disclosure privacy and human help accessible

People should be able to perceive that they're interacting with AI before relying on its answer or deciding what to disclose. The identification, privacy information and alternate route must work with the same keyboard, focus, zoom and screen-reader conditions as the conversation.

The OAIC guidance on commercially available AI products says public-facing AI tools such as chatbots should be clearly identified. It also recommends due diligence, privacy impact assessment, clear privacy information, testing, monitoring and human oversight when personal information is involved. Coverage depends on the organisation, activity and information handled, so obtain privacy advice for your circumstances.

Check that a person can find the privacy explanation before entering personal information. Explain whether a transcript is stored, why it's used, who may receive it and how long it's kept in language that fits the actual practice. Test transcript download or deletion controls if the product offers them. Keep accessibility evidence lean so testing doesn't create a new store of personal conversation content.

The Australian Government's Guidance for AI Adoption implementation guidance asks organisations to tailor transparency to diverse audiences, including people with accessibility needs, and to preserve meaningful human control. In practice, the non-chat route must be findable and operable by the people most likely to need it.

Trigger the fallback from a normal reply, an error and an unsupported request. Confirm the person can reach it without a pointer, understand whether they will contact a person or leave a message, and carry forward only the context they have agreed to share. The specialist guide covers designing the human handoff itself. This test checks whether people can perceive and operate that route.

Run the tests as real conversations

Automated scanning can detect some markup, name, contrast and structural problems. It cannot prove that focus returns logically, that a reply is announced once, that the on-screen keyboard leaves the input usable or that a failed conversation has a workable alternate route. W3C describes accessibility evaluation as a combination of automated testing and human evaluation.

Choose representative tasks from real intended use. Run each from a defined starting state with keyboard-only interaction, supported screen readers and browsers, zoom or reflow, narrow mobile layouts, timing changes, errors, correction, transcript handling and fallback. Include people with relevant access needs where feasible. Their experience can reveal barriers that a standards-based test case misses.

Use the same evidence discipline as the wider pre-launch test and release method. Record the actual result, not just pass or fail. Screenshots can show layout. A short recording can show focus order. A screen-reader log can capture announcements. A tester's observation can explain why technically present information was still hard to use.

Use failures to define the next piece of work

Sort each failure by the part of the experience that owns the fix. Vendor configuration may control dialogue semantics, status announcements or timeouts. The host page may obscure focus or break reflow. Conversation content may use unclear prompts and errors. The fallback may fail because the form, phone route or human queue is inaccessible.

That ownership matters more than a vague accessibility score. Block release when a person cannot start, complete, correct or leave the interaction and no verified accessible route exists. Fix and retest bounded defects when the expected behaviour is clear. Record any accepted limitation with its owner, user impact, alternate route and review date.

If the chatbot is already live, use the broader method for auditing the live AI customer experience to examine production journeys and ongoing controls. When failures cross the widget, host page, content and lead journey, isolated configuration changes won't be enough. The next piece of work is website design that fixes the whole interaction, supported by the test evidence and a named release decision.