Build the record before the workflow goes live
A website enquiry arrives. A classifier identifies the likely service, a retrieval step supplies approved business information, a model prepares a summary, and a connector proposes a CRM update. A person approves the action, the record is written and a customer response is sent. Two days later, the classification looks wrong. The team can see the final CRM record, but it can't reconstruct which version, sources, approval or tool result produced it.
AI workflow observability is the run-level evidence that shows what happened across that transaction. Design the audit trail before a failure. It records observable inputs, outputs, versions, sources, tool calls, approvals and business actions. Its scope is observable evidence. A model's private chain of thought and a complete causal explanation of its behaviour remain outside that scope.
The Australian Government National AI Centre implementation guidance recommends clear accountability, documented monitoring requirements and records of governance decisions, tests, incidents and ongoing monitoring. For an operator, that means deciding who owns the trace and what it must prove before the workflow enters routine production.
Start with the business transaction
Define the unit of work in business language. In this example, one unit begins when the website accepts an enquiry and ends when the CRM record and approved response reach their intended states. Name the workflow owner, the expected outcome and the systems inside the boundary.
Carry forward the acceptance criteria and identifiers used while testing the AI workflow before launch. The production trace should show whether the same required fields, approval gates, prohibited actions and outcome checks still hold under live conditions.
The workflow boundary includes the form, classifier, retrieval service, model interaction, tool request, approval interface, CRM write and response service. An external email or CRM platform may keep its own logs, but the workflow still needs a reference that connects its event to the same transaction.
Give every run one correlation ID
Create a random, non-sensitive correlation ID when the enquiry enters the workflow. Pass that value through every component and include it in each event. Keep the person's name, email address, customer number and message content out of the identifier.
The correlation ID answers, “Which events belong to this business transaction?” Timestamps order those events. A component span ID can connect a request with its response or retry. A stable outcome reference can point to the CRM record without copying the record into the log.
For the running example, the form creates run_7f3.... The classifier, retrieval step and model call record it. The approval interface uses it when a person accepts or rejects the proposed action. The CRM connector returns the same ID with the write result, and the response service records it when the approved message is sent. A retry keeps the correlation ID and receives a new event or span ID so it doesn't look like a separate enquiry.
Record the evidence that explains the run
The OpenTelemetry generative AI semantic conventions provide vendor-neutral attributes for operations, models, messages and data sources. Use them as a starting vocabulary and collect only fields with a defined purpose. The conventions were checked on 17 October 2026 and should be reviewed as they evolve.
Keep the required record compact enough to implement consistently. Stable metadata should explain the path and result. Prompt text, output text and retrieved content remain optional, sensitive fields that need a separate justification.
| Field | Purpose | Required | Retention |
|---|---|---|---|
| Correlation ID | Join one transaction across systems | Yes | Operational window |
| Event time and span ID | Order steps, retries and responses | Yes | Operational window |
| Workflow and component version | Identify the deployed release | Yes | Release evidence period |
| Operation and status | Show the attempted step and result | Yes | Operational window |
| Model and provider ID | Identify the model interaction | When used | Release evidence period |
| Source or data reference | Point to retrieved evidence safely | When used | Match source review need |
| Tool request and result | Show the requested and completed action | When used | Business record need |
| Approval state and approver role | Prove the human gate was applied | When required | Approval evidence period |
| Outcome reference and error code | Connect the final state or failure | Yes | Operational window |
| Owner and access class | Assign review and restrict visibility | Yes | Life of the record |
Retention labels need actual periods in the organisation's system policy. Set them from the evidence purpose, business-record obligations, privacy risk and incident needs. Give every record a deliberate expiry or documented preservation requirement.
Trace versions, sources, tools and approvals
A useful trace identifies the complete release unit. Record workflow, prompt, model, source-set, connector, schema and business-rule versions where they can change the result. This connects the run to the separate discipline for controlling changes to a live AI workflow without rebuilding its change record inside the log.
Store source references that a reviewer can resolve, such as a document ID and revision, rather than copying a knowledge base into every trace. When those references reveal gaps in coverage, freshness, permissions or answer support, use the specialist process for monitoring AI knowledge base quality.
Tool events should distinguish a proposed action from an authorised and completed action. OWASP's guidance on excessive agency recommends minimising permissions, requiring approval for high-impact actions and logging activity. For the CRM write, record the action requested, the permission or approval state, the connector result and the affected record reference. The connector result confirms whether the write succeeded.
Match evidence capture to sensitivity
Observability can create a new exposure if the team copies full enquiries, prompts, outputs and retrieved documents into a broadly accessible log. OWASP's sensitive information disclosure guidance identifies personal, financial, confidential and proprietary information as material risks. OpenTelemetry also warns that message content attributes may contain sensitive information.
Default to metadata, categories, hashes, token counts and resolvable references. Use a redacted excerpt only when it answers a defined review question. A controlled content sample needs named access roles, an approved purpose, a short retention rule and an auditable deletion route. Operators may need event metadata while a privacy or security reviewer controls access to captured content.
Two questions shape the decision. How much operational evidence would the field provide, and how sensitive is the content? Together they create four distinct handling areas.
Evidence value and sensitivity determine what the trace keeps
Content sensitivity increases
Redact or reference
High sensitivity with lower evidence value. Keep a safe identifier, category or hash instead of the content.
Controlled sample
High sensitivity with high evidence value. Capture only under explicit access, minimisation and retention controls.
Discard low-value data
Low sensitivity with lower evidence value. Extra data adds cost and exposure without helping a decision.
Retain metadata
Low sensitivity with high evidence value. Keep structured fields such as versions, timings, status and action references.
Operational evidence value increases
Apply the decision field by field. A source revision ID may be valuable and low sensitivity. A full customer message may be sensitive even when it helps an investigation, so a short controlled sample or a protected source reference is safer than default capture.
Turn important events into owned alerts
An audit log helps reconstruction. Production monitoring also needs alerts for events that require timely attention. The NIST AI RMF Playbook Measure guidance connects post-deployment monitoring with pre-deployment test results and calls for mechanisms that identify when human intervention is needed.
Tie alerts to the accepted boundary. Trigger review for repeated action failure, a missing required approval, unexpected tool use, a deployed version mismatch, a latency breach or a sensitive-data signal. Each alert needs an owner, a response window, the evidence link and a route for acknowledgement.
When a threshold indicates possible material harm, privacy or security impact, or business continuity risk, hand coordination to the AI incident response playbook. The trace supplies evidence. The playbook owns containment, communication and recovery.
Check that the trace works before launch
Test the evidence path as part of release readiness. A log that works only on the happy path will be least useful when a dependency fails.
- Run representative, boundary and denied test enquiries through the complete production-like path.
- Confirm the correlation ID survives every boundary, retry and asynchronous handoff.
- Verify that version, source, tool, approval, outcome and error fields resolve to the expected records.
- Trigger each alert and confirm the owner receives enough context to act.
- Inspect redaction and access with an account that should see metadata but not protected content.
- Prove that records and controlled samples are deleted when their retention periods end.
Keep these checks after launch. NIST AI 800-4 explains why monitoring deployed AI remains difficult as providers, releases, system features and oversight needs change. Review the schema, integrations, thresholds and access after a material workflow change, and assign an owner to maintain them.
Route audit evidence to the next operating decision
Run events are evidence for other operating decisions. Aggregated outcomes can feed the separate framework for measuring AI workflow ROI and reliability. A preserved failed run can enter the guide for diagnosing failures in a live AI workflow. Neither job belongs inside the trace design itself.
Governance owners can use the audit trail to confirm that registered boundaries, approval and retention controls operate as intended. Change owners can identify which release handled an enquiry. Knowledge owners can inspect a referenced source. Each person receives the evidence needed for their decision without giving every role access to captured content.
A trace should make the next decision possible
Select one transaction and try to reconstruct it. You should be able to see which workflow and component versions ran, which sources were referenced, what actions were requested and completed, whether the required approval occurred, where the outcome lives and when each piece of evidence will expire.
If missing form fields, CRM identifiers, approval interfaces, integrations or analytics make that impossible, the constraint sits in the connected system design. Designing the website and CRM journey around stable identifiers and visible handoffs can make the next run reviewable before the business has to investigate it under pressure.