Start with the customer outcome that looks wrong
A customer completes a conversation with a website assistant and receives a confident answer. The dashboard marks the interaction resolved. Two days later, the same person emails the business because nothing happened.
The transcript may look fluent. The customer journey did not work.
A live AI customer experience audit starts with evidence of the outcome rather than how polished the conversation sounds. Look for repeat contact, reopened cases, abandoned chats, complaints, staff corrections, missing records, or actions that never reached the person responsible. These signals reveal customer effort and consequences hidden by a tidy automation metric.
Choose one suspect journey and audit it end to end. The output should be an owned decision to scale, repair, narrow, or stop that experience. It should also name the evidence, remaining uncertainty, responsible person, action, and review date.
This work begins after launch or during a live pilot. If the system hasn't reached customers, use the guide to test an AI workflow against acceptance criteria before launch.
Set the audit boundary around one real journey
Define the journey tightly enough that someone else could reproduce the review. Record the use case, channel, customer group, time window, intended outcome, and accountable owner. Mark where the journey begins and where it genuinely ends. For an enquiry assistant, that may be a correctly routed CRM record and a useful staff response beyond the final chatbot message.
This boundary follows the use-case-specific logic in NIST's TEVV-Athlon Framework for Evaluating AI Systems. NIST describes a flexible approach to testing, evaluation, verification, and validation tailored to organisational goals, real-world impacts, and the application being assessed. TEVV-Athlon is an initial public draft announced on 7 August 2026, so treat it as developing guidance rather than a settled standard.
Write the expected outcome in terms a customer and operator would recognise. "Chat completed" is a system event. "The customer received an accurate answer and the promised follow-up reached the correct queue" is an auditable outcome.
Build evidence from what customers actually experienced
Bring the conversation and the surrounding journey into one evidence record. A compact working table is enough.
| Touchpoint | AI action | Customer outcome | Human action | System record |
|---|---|---|---|---|
| Website enquiry | Answered and collected details | Customer expected a callback | Sales queue checked the request | Transcript and CRM event |
| Follow-up | Stated a next step | Customer contacted the business again | Staff reconstructed the enquiry | Email and correction note |
Populate the record with real transcripts and events, then add complaints, abandonments, repeat contacts, corrections, and a sample of verified outcomes. Verification means checking whether the promised result happened. A closed status or positive-sounding final message needs evidence from later in the journey.
Only collect evidence needed for the audit. Restrict access, redact unnecessary identifiers, and record where copies are stored. When personal information is involved, the OAIC guidance on commercially available AI products calls for due diligence, data minimisation, transparent notices, accuracy controls, and ongoing review where the Privacy Act applies. Coverage and duties depend on the entity and use case. Confirm your obligations and seek specialist advice where needed.
Sample by consequence and uncertainty
A random sample can show ordinary behaviour, but it may miss rare failures with serious consequences. Build a risk-weighted sample that includes four kinds of evidence.
- High-consequence journeys where a wrong answer, disclosure, or missed action could materially harm the customer
- Recurring failures such as repeat contact or staff correction
- Hard-to-detect outcomes where the transcript can look successful while the downstream action fails
- Edge cases involving ambiguity, accessibility, sensitive information, urgency, or an unusual route
Keep a baseline random sample beside these cases to represent ordinary performance as well as the worst examples. Choose its size according to journey volume, variation, consequence, evidence quality, and the confidence required for the decision.
The Australian Government's October 2025 Guidance for AI Adoption implementation guidance supports risk and impact assessment, testing, monitoring, and continual evaluation across the AI lifecycle. It's voluntary guidance, not a new legal duty. Its risk-proportional approach is useful here because the effort spent reviewing a journey should reflect what failure could do to a customer and the business.
The earlier 10 voluntary AI safety guardrails are predecessor guidance. The department says they evolved into the updated and simplified Guidance for AI Adoption published in October 2025.
Reconstruct the journey before scoring the conversation
Read each selected case as a chain of evidence. What did the customer ask? What did the AI infer? Which source supported its answer? What action did it trigger? Did a person become involved? What did the CRM, ticketing system, or inbox record? What outcome can you verify?
Consider a composite website enquiry. A customer asks whether a service is available in their suburb and explains a timing constraint. The assistant answers from an old service-area page, collects contact details, and says the team will respond. The transcript ends politely. The CRM receives the contact details without the suburb or timing constraint, routes the lead to the wrong queue, and records the chat as completed. The customer follows up by email and repeats the request.
Conversation scoring alone might find an appropriate tone and a clear closing message. Journey reconstruction finds a stale source, lost context, incorrect routing, false resolution, and extra customer effort. It also shows that changing the prompt would address only part of the failure.
Classify the broken layer
Turn each observed symptom into a diagnosis that an owner can inspect. The same symptom may cross several layers, so record uncertainty rather than forcing a single cause.
| Symptom | Evidence to inspect | Likely layer | Owner |
|---|---|---|---|
| Customer abandons | Events, accessibility checks, alternate routes | Interface | Digital or CX lead |
| Answer is stale or unsupported | Retrieved source, version, permissions | Knowledge | Content or knowledge owner |
| Promised action fails | Integration event, CRM fields, queue status | Workflow | System or process owner |
| Staff can't pause the journey | Incident record, approval and stop authority | Governance | Accountable AI owner |
| Customer repeats their story | Transfer summary, consent, queue acceptance | Human service | Service team lead |
Route the repair to the layer instead of reteaching every solution inside the audit. An interface that doesn't fit the enquiry job may need a different choice between chatbots, guided forms, and live chat. Stale, conflicting, or poorly retrieved answers need ongoing knowledge-base quality monitoring.
For intake, consent, qualification, CRM, or routing failures, review how to design AI lead qualification as an owned website workflow. Failed escalation, lost context, or no usable human route belongs in the guide to designing a safe human handoff for AI customers.
Unclear accountability, privacy boundaries, incident handling, or stop authority requires broader AI governance controls for the business. Keep each destination focused on repair while this audit remains focused on diagnosis.
Check trust where the journey is most exposed
Inspect what the customer could see and do at the moments where the AI reached a limit, collected information, influenced a decision, or transferred control. Check whether the interface identifies AI involvement, explains relevant limitations, provides a usable alternate channel, supports accessibility, accepts feedback, and makes human takeover conditions observable.
The Australian Government AI Technical Standard Statement 10 turns these ideas into concrete criteria for disclosure, limitations, alternate channels, accessibility, feedback, and human control. The standard applies to Australian Government agencies. A private business can use it as a transferable inspection benchmark while distinguishing it from a universal private-sector requirement.
Privacy needs its own evidence. Where the Privacy Act applies, OAIC guidance explains that collecting personal information through a customer-facing AI system can engage Australian Privacy Principle requirements covering collection and notice. It also addresses accuracy, use, and disclosure. Inspect what the customer was told, which information was necessary, where it went, who could access it, and whether generated personal information was corrected when wrong. This audit isn't legal advice.
The operating consequence matters alongside compliance. A hidden AI interaction can weaken informed choice. An unusable alternate channel traps the customer. A vague takeover rule leaves staff unable to intervene before harm or frustration grows.
Choose whether to scale, repair, narrow or stop
Make the decision using two assessed axes. Verified outcome quality asks whether sampled customers achieved the intended result. Consequence of failure asks what the observed or plausible failure can do to the customer and business. These axes create four distinct operating areas and make the next move easier to defend.
Outcome quality and consequence determine the next move
Verified outcome quality increases
Scale
Strong verified outcomes with bounded consequence
Repair
Strong outcomes exist but higher consequence needs tighter controls
Narrow
Weak outcomes justify a smaller safer scope
Stop
Weak outcomes and high consequence make continued use indefensible
Consequence of failure increases
Scaling still requires monitoring. Repair means an identified layer and owner can address a controllable failure. Narrowing can remove a topic, customer group, action, or channel until evidence improves. Stop when weak outcomes and high consequence cannot be controlled within an acceptable boundary.
Read speed and automation rate alongside stronger evidence. The Guidance for AI Adoption describes internal, external, risk, and impact measures within its implementation approach. Off Piste's practical interpretation is to compare operational measures with verified customer outcomes. A fast response has little value when the customer must return or a promised action never occurs.
Turn findings into a repair queue
Prioritise findings by customer consequence, recurrence, detectability, and diagnostic confidence. High consequence and hard-to-detect failures deserve attention even when they appear less often. Frequent low-consequence friction may also justify early repair when it adds substantial customer and staff effort.
Write each repair as a testable change. The composite enquiry might produce separate work for the service-area source, CRM field mapping, queue rule, and completed-status definition. Convert each finding into an acceptance test and regression case before redeployment, following the existing pre-launch AI workflow testing method.
Use the broader framework for measuring AI workflow ROI and reliability to maintain baselines, verified outcomes, exception rates, review effort, and operating cost after the immediate audit. Feed it the audit findings and keep the measurement method in one place.
Before a change enters the queue, leave a record another person can verify.
End with an owned decision
A useful AI customer experience audit ends with a bounded journey, defensible evidence, diagnosed layers, and named owners. Record the decision, the uncertainty that remains, the person authorised to act, and the date the evidence will be reviewed again.
Some audits find one contained repair. Others show that content, interface, forms, accessibility, analytics, CRM capture, and human follow-up fail at the joins. When the evidence crosses those layers, a scoped website and customer journey review can resolve the system as one experience.