A routine update is still a production release
An enquiry-to-CRM workflow is already live. It reads a website form, classifies the service need, prepares a summary, writes the CRM record and leaves a follow-up draft for a person to approve. The provider is retiring the model it uses, so the team must move to a newer one.
The replacement looks better in a few manual trials. That doesn't answer the operational questions. Could it classify mixed enquiries differently, omit a required field, call the connector with a changed value or increase review time? Who can accept those changes? How quickly can the team restore the current version?
Change control starts after the workflow has passed its first release and gained an acceptance suite. If yours hasn't, begin with testing an AI workflow before launch. Then compare each candidate with a known-good production version, approve a bounded release and keep a recovery route open.
A passing regression run reduces uncertainty. It cannot establish safe behaviour across every live condition. NIST AI 800-4 explains that pre-deployment evaluations take place mainly in controlled environments, while post-deployment monitoring is needed to detect unforeseen outputs and consequences in real use. Every changed workflow therefore needs both release evidence and observation after release.
Treat the workflow as one connected release unit
An AI workflow's behaviour comes from more than its model. Prompts shape the request. Retrieval sources shape the available facts. Permissions determine what the system can see and change. Connectors and schemas determine what moves between systems. Deterministic rules decide when to route, block or escalate.
A locally small edit can travel through that chain. Renaming a CRM field can cause a valid classification to land in the wrong place. Tightening source permissions can leave retrieval without the information needed to draft an accurate reply. A clearer prompt can produce longer text that exceeds a downstream field limit.
Record the complete release unit for the enquiry workflow. That means the model identifier, prompt version, approved source set, connector version, field mapping, service taxonomy, permissions and approval rules that jointly produced the known-good result. A version number that identifies only the prompt leaves the team unable to reproduce the business outcome.
This inventory also reveals ownership. If nobody can identify the person responsible for the workflow outcome and the people responsible for its connected systems, return to the work of choosing and mapping the workflow before making a material change.
Record the known-good version before changing anything
The current production version is the comparison point and the recovery target. Preserve enough of it to restore the working path. Capture component versions and configuration, representative inputs, expected outcomes, recent operating measures and any accepted limitations.
For the enquiry workflow, keep examples of straightforward, incomplete and mixed-service enquiries. Record the expected classification, required CRM fields, correct review state and prohibited actions. Add current measures that would reveal operational movement, such as successful record creation, duplicate creation, human correction, review time, failed connector calls and unauthorised sends.
Choose measures because they protect the workflow's purpose. The guide to measuring AI workflow ROI and reliability shows how to combine outcome, reliability, review effort and cost. Output quality alone misses too much.
Save the actual restorable release as well as its description. That may include a deployable configuration, the previous prompt and model selection, connector mappings, permission settings and the instructions for switching traffic back. A screenshot of settings is useful evidence, though it isn't a dependable rollback mechanism.
The baseline creates a fair test. The candidate must be compared with the same cases and measures where possible. If the team changes the model, prompt and CRM mapping at once, it becomes much harder to explain a moved result. Keep the release narrow unless the components genuinely have to change together.
Classify the change by its likely impact
Change classification should determine the evidence and authority required. It doesn't need a decorative risk score. Ask what changed, which dependencies it touches, what data and actions it can reach, how visible the result is to a customer and how easily the team can reverse it.
- Prompt edits can move tone, extraction, classification and tool choice. Rerun critical outcome cases, ambiguous inputs, prohibited actions and downstream format checks.
- Model migrations can move interpretation, consistency, latency, cost and tool behaviour. Compare the candidate with the known-good model across the full regression set in a non-production environment.
- Retrieval updates can change the facts available to the workflow. Check source coverage, freshness, access boundaries, unsupported-answer behaviour and citations. Use the specialist method for monitoring AI knowledge base quality when retrieval is a material dependency.
- Connector or schema changes can corrupt an otherwise good result. Test field mappings, types, required values, duplicates, retries, partial writes and reconciliation after failure.
- Permission changes can either block necessary work or expand the blast radius. Test the configured production identity against allowed and prohibited records, fields and actions.
- Business-rule changes can alter routing, thresholds and human handoffs. Test boundary values, conflicting rules, exception paths and the authority needed to override a decision.
Apply more scrutiny when several of these conditions combine. A customer-facing model migration with write access to the CRM and weak reversibility deserves a broader suite, a smaller first cohort and a more senior approver than an internal wording change with no tool access.
In the running example, the model migration touches classification, summary quality and tool arguments. The connector is unchanged, yet its calls remain in scope because a different argument can still produce a bad CRM record.
Select regression cases from the change
Reuse the representative acceptance suite from the first release. Don't redesign it around the candidate or remove inconvenient cases. Rerun every critical case whose failure could cause harm, corrupt a record or bypass an approval, even when the change appears unrelated.
Then add targeted cases for the affected paths. For the enquiry model migration, include common and mixed-service enquiries, missing details, prompt injection inside form text, protected CRM fields, unavailable dependencies and requests that should be sent to a person. Compare both the final business state and the intermediate tool calls.
Some checks should be deterministic. Required fields, allowed categories, JSON shape, permission boundaries and duplicate prevention can have explicit pass rules. Model judgement needs reviewed examples and declared tolerances. Record disagreements and accepted exceptions so an average score can't hide them.
Every repaired production failure belongs in the permanent regression set. If a previous version attached an enquiry to the wrong customer after a retry, the exact repaired pattern should be tested before this candidate moves forward. When the cause is unclear, use the process for diagnosing failures in a live AI workflow to preserve the failed unit of work and trace it through the system.
Build one reviewable change record
Test results become useful release evidence when an approver can connect them to the exact candidate, its likely impact and its recovery plan. Keep those decisions together. Scattered tickets, spreadsheets and chat threads make it too easy to approve one version and deploy another.
For the enquiry workflow, the record might say that the model is changing because the current version is being retired, while the prompt, connector and rules remain fixed. It would attach the baseline comparison, identify mixed-service classification and tool arguments as the most likely failure paths, and name a limited internal cohort for the first release window.
This is an Off Piste operating template derived from the Australian Government guidance, NIST AI RMF Core and the attributed AWS and Microsoft implementation examples. Adapt its depth to the workflow's reach and risk while retaining a single chain from reason to evidence to authority.
Approve against declared release gates
An approver needs evidence and authority. Evidence shows how the candidate performed against the acceptance criteria and baseline. Authority determines who may accept the remaining uncertainty for this use, data and customer exposure.
The Australian Government National AI Centre implementation guidance calls for clear accountability across deployment, testing and ongoing oversight. It also calls for records covering acceptance criteria, test results, risk treatment and review dates, plus reassessment when a significant change occurs.
Translate that into declared gates. The model migration cannot create unauthorised sends, expose another customer's data or write protected CRM fields. It must meet the accepted result for every critical case. Any agreed tolerance for classification quality, review effort, connector failure or cost must be recorded before testing.
The workflow owner can recommend release. The person authorised by the organisation's AI governance policy accepts the residual risk at the appropriate tier. Escalate when the change expands the use, data access, autonomy or possible harm beyond the registered approval. A material incident follows the organisation's incident response route as well as this release process.
Release in stages and watch the same measures
Move the candidate through environments and cohorts that limit exposure. Compare it with the baseline in non-production first. Then use an internal group, a small traffic share or a short reversible window where the team can inspect results quickly. Expand only after the declared gates hold.
AWS's Generative AI Lifecycle Operational Excellence framework describes versioned experiments and evaluation data, testing, deployment strategies, continuous monitoring, iterative deployment and rollback. These are useful lifecycle patterns. Its exact architecture remains specific to AWS.
Microsoft's guide to validating a Copilot Studio model migration gives a platform-specific example of the same loop. It establishes a baseline, runs offline comparisons, requires approval, stages deployment, watches gates and adds discovered failures to the regression set.
For the enquiry workflow, start with staff-submitted test enquiries that use the production path without contacting customers. Then allow a limited set of normal enquiries while a named reviewer checks the CRM write and draft before approval. Monitor the same outcome, reliability, review-effort, cost and harm indicators captured in the baseline.
The NIST AI RMF Core connects measurement with benchmarks and documented results, then places monitoring alongside recovery and change management. This continuity matters. Changing the metric after release can disguise movement that the baseline was designed to reveal.
Decide rollback before the release
Rollback is an operating decision made before an emergency. Before approval, name the version to restore, the person who can trigger restoration, the conditions that trigger it and the work needed to reconcile data already written by the candidate.
For the model migration, a rollback trigger might be any unauthorised action, any protected-field write or a sustained breach of an agreed reliability gate during the monitoring window. The owner restores the known-good model and configuration, pauses new work if restoration isn't dependable, and routes urgent enquiries to the human process.
Restoring software doesn't repair every consequence. Identify CRM records created or changed during the release window. Preserve logs and affected units of work. Correct duplicates, invalid classifications or missed follow-ups, then confirm the customer-facing state. If there may be material harm, privacy impact or a reporting obligation, activate the AI incident response playbook.
Diagnose the failure before preparing another candidate. Add the repaired case to the permanent suite, update the change record and seek approval again. This closes the loop without treating a rollback as proof that the underlying problem has gone away.
A controlled change should leave better evidence behind
A release is complete when the team has recorded the deployed version, approval, monitoring result, exceptions, reconciliation work and any new regression cases. The workflow should be easier to reproduce and recover than it was before the change.
Start with the next proposed model, prompt, source, connector, permission or rule change. Identify the complete release unit, preserve the known-good baseline, target regression cases to the likely impact and agree the rollout and rollback decisions before production exposure.
If the website form, CRM mapping, approval interface or analytics cannot show which version handled an enquiry, release control has become a systems-design problem. Designing the connected website and CRM path around version visibility makes future changes easier to review and recover.