The pilot is live, but is it working?
The workflow is processing enquiries, drafting follow-ups, updating records, or preparing content. The dashboard proves activity. People say it feels faster. The next investment decision needs stronger evidence.
A useful answer needs evidence that the intended outcome happened, the output was reliable enough for its context, the human review burden stayed visible, and the total operating cost made commercial sense. That evidence supports the next decision. Scale the workflow, fix a defined weakness, or stop putting time and money into it.
This is a measurement problem at the level of one real workflow. The unit might be a qualified enquiry followed up correctly, a support request resolved, a CRM record completed, or an article approved for publication. Raw AI activity isn't the unit.
Start with the decision this workflow must improve
Write down the workflow, its owner, the unit of work, the intended outcome, and the business consequence. For an enquiry workflow, the unit could be one submitted enquiry. The intended outcome might be an accurate CRM record and an appropriate response approved within one business hour. The commercial consequence could be more qualified conversations without adding administrative load.
If the team hasn't chosen a suitable workflow or mapped its inputs and trust gates, start with the guide to deciding what to automate first. Once a pilot is live, keep measurement tied to the decision it was meant to improve.
The Australian National AI Centre's guidance on measuring return on AI investment starts with the problem, expected outcome, and signs of progress. It also includes quality, capacity, confidence, staff and customer satisfaction, and decision-making alongside financial return. That's a useful commercial boundary. The workflow has to improve something the business values, and the team needs to be able to see the change.
Build the baseline before claiming a gain
A before-and-after comparison is only useful when both periods represent comparable work. Choose a pre-automation window that captures normal volume and a reasonable mix of straightforward and difficult cases. Record seasonal promotions, staffing changes, major website changes, and other events that could distort the comparison.
For each unit of work, capture the volume, elapsed time from start to outcome, active labour time, queue time, rework, exceptions, approvals, completion or conversion outcome, and operating cost. Use medians or ranges when a small number of extreme cases would make an average misleading. Mark missing data and leave the gap visible.
This distinction matters. A workflow may cut hands-on drafting from 20 minutes to five while the item waits six hours for approval. It may process twice the volume because demand increased, while conversion quality falls. It may save a coordinator's time and add review work for a more expensive manager.
Clean intake makes the baseline measurable. Forms need to capture service, urgency, source, consent, and a stable identifier so the workflow can connect each input to its later customer outcome. The website redesign checklist covers the forms, analytics, lead capture, and CRM handoffs that often need attention before clean measurement is possible.
Define a verified successful outcome
Define success before calculating a rate. A verified successful outcome has reached the intended end state and passed the review required for that workflow's risk.
For an enquiry workflow, an attempt is any enquiry sent into the system. A technical completion means the automation ran without an integration error. An accepted output means the summary, CRM fields, and response passed review. A customer outcome means the right person received an appropriate response and moved to the next relevant stage. Those counts answer different questions.
Use this core formula:
Verified task success rate = verified successful outcomes ÷ eligible workflow attempts × 100
Define “eligible” in writing. Spam, duplicate submissions, test records, and enquiries outside the service area might be excluded, but the exclusions must be logged. Otherwise a team can improve the rate by quietly removing hard cases from the denominator.
Silent failures deserve their own check. These outputs look complete but contain an incorrect field, unsupported claim, missed escalation, or unsuitable response. They sit outside integration error logs, so finding them usually requires a representative human review sample.
Use a four-layer workflow scorecard
The Off Piste scorecard keeps four connected views of the same workflow. Each layer answers a different operating question.
| Layer | Question | Example metrics |
|---|---|---|
| Business outcome | Did the decision or customer result improve? | qualified conversations, completion, conversion, capacity used, satisfaction |
| Workflow performance | Did work move through the process better? | verified success, cycle time, queue time, throughput, exception rate |
| Quality and risk | Was the result dependable for its consequence? | approval, rework, review coverage, silent failures, failure severity |
| Total cost | Was the verified outcome worth producing? | labour, review, rework, platform, integration, maintenance, incident cost |
This interpretation aligns with the US GAO AI Accountability Framework, which organises accountability around governance, data, performance, and monitoring. For a growing business, the implementation can stay light. The owner still needs to inspect source data, operating performance, responsibility, and ongoing evidence together.
Google Cloud's vendor-authored framework for production AI agent KPIs similarly groups operational performance, adoption, and business value. Its terminology is aimed at production agents, but the practical categories translate to a bounded AI-assisted workflow. Use them as practitioner guidance. Thresholds remain specific to the workflow.
Define the metrics people will actually use
Every selected metric needs a unit, numerator, denominator, data source, owner, review cadence, and known gaming risk. A short definition sheet prevents three people from reporting three versions of “success”.
| Metric | Calculation | Source and owner | Cadence and gaming risk |
|---|---|---|---|
| Verified success | accepted outcomes ÷ eligible attempts | workflow log plus review record, workflow owner | weekly during pilot. Weak exclusions inflate it |
| Exception rate | cases leaving the designed path ÷ eligible attempts | exception log, operations owner | weekly. Unlogged manual work hides exceptions |
| Rework rate | accepted outputs requiring material correction ÷ reviewed outputs | review record, reviewer | weekly. Vague “material” edits suppress the rate |
| Review coverage | outputs reviewed ÷ outputs completed | approval log, accountable owner | weekly. Clicks without inspection overstate coverage |
| Cycle time | outcome timestamp minus intake timestamp | CRM or system log, operations owner | weekly median and range. Excluding queue time flatters speed |
| Cost per verified outcome | total workflow cost ÷ verified outcomes | finance data plus time logs, workflow owner | monthly. Raw attempts in the denominator understate cost |
Failure severity belongs beside frequency. Ten corrected formatting errors and one unapproved pricing promise shouldn't be treated as equivalent. Use a small workflow-specific classification based on customer, financial, privacy, legal, or operational consequence. Record who assigns severity and how disagreements are resolved.
Calculate net value from evidence the business can defend
Start with a transparent value model grounded in the business's own evidence.
Gross capacity value = verified hours released × relevant loaded hourly cost
Net verified value = gross capacity value + evidenced revenue or avoided cost − review labour − rework labour − platform cost − integration and maintenance cost − incident cost
ROI = (net verified value − implementation cost) ÷ implementation cost × 100
Payback period = implementation cost ÷ average monthly net verified value
Only count hours released after review and rework. Then explain what happened to that capacity. If the team served more customers, avoided a hire, reduced overtime, or moved a constrained specialist into higher-value work, the business can describe the effect. If saved minutes simply disappeared into a busy day, report capacity released. A cash-saving claim needs evidence of a financial effect.
The Australian Government guidance makes the same practical distinction. It says time saved creates value when it is redirected to useful work, and asks organisations to include testing, data preparation, change management, governance, and ongoing oversight in the cost picture.
If credible first-party figures aren't available, don't invent a case study. Use the formulas with labelled assumptions. For example, a team can enter its own eligible volume, median labour time before and after, reviewer minutes, rework minutes, loaded rates, software fees, maintenance hours, and verified outcome count. The calculation is hypothetical until logs and finance records replace those inputs.
Measure review and exceptions as part of the system
Human review isn't an external tax on the automation. It's one of the workflow steps. Measure who reviews, which cases they see, how long review takes, what they change, and what they escalate.
Use full review where the consequence demands it. For lower-risk work, use a representative sample that covers normal cases, edge cases, different input sources, different times, and recent system changes. Increase coverage after a prompt, model, integration, source, or policy change. Increase it again when failure severity rises or a new exception pattern appears.
The NIST AI Risk Management Framework Core supports quantitative, qualitative, or mixed measurement, testing before deployment and during operation, documented human oversight, and assessment under conditions similar to deployment. It doesn't prescribe one accuracy target. The acceptable threshold depends on the workflow's context, risk tolerance, and cost of failure.
Content production shows why review metrics matter. In an AI-assisted content workflow, useful measures include expert review minutes, material revision rate, evidence failures, approval rate, and publishable-output rate. A fast first draft has limited value when the expert has to rebuild the argument or replace unsupported claims.
Exception logs should capture the input, stage, detected problem, severity, reviewer action, root cause, and resolution. Group recurring causes such as missing form fields, stale source material, ambiguous business rules, integration failures, and cases outside scope. That turns failure into a redesign queue.
Set acceptance criteria before the next review
An acceptance criterion tells the team what evidence will support a decision. Write the metric, threshold or range, evidence source, accountable owner, review date, and required response. Set it before opening the next dashboard.
The Australian Government's Voluntary AI Safety Standard guardrails call for accountable ownership, measurable acceptance criteria, testing against those criteria, ongoing monitoring, and documented evidence. Applied to one workflow, that could mean a CRM owner checks verified success weekly, a sales lead reviews every high-consequence response, and the system pauses when a defined failure pattern appears.
Qualitative evidence belongs here too. Staff may report that exceptions are harder to recognise, customers may find responses less useful, or reviewers may feel pressure to approve quickly. Record who was asked, what question they answered, and when. NIST explicitly allows mixed methods because some impacts remain invisible in a numeric log.
Microsoft's vendor-backed 2026 Work Trend Index reports associations between organisational conditions and reported AI impact, and broadens impact beyond output speed to quality, capability, control, collaboration, and higher-value work. Its survey findings include self-reported measures and don't prove that a particular workflow caused an outcome. They are still a useful prompt to ask whether the surrounding work design helps people use the system well.
Decide whether to scale, fix, or stop
Use verified business value and verified reliability as the two decision axes. Define both for the workflow before placing it on the matrix.
Scale when the workflow creates worthwhile outcomes, meets its reliability criteria under representative conditions, and has ownership for monitoring the larger volume. Expand in stages and keep sampling after each material change.
Fix when the outcome is valuable and the failures have tractable causes. Better source material, clearer intake, narrower scope, stronger escalation, or a different review gate may solve the problem. If form architecture, instrumentation, or CRM handoff is the constraint, that is a specific website and systems design problem.
Stop when the commercial value remains weak, evidence stays opaque, severe failures recur, or the review and operating burden exceeds the gain. A technically reliable automation can still be the wrong investment if it accelerates low-value work.
Make measurement part of the workflow
Assign one accountable owner. Capture a comparable baseline. Define the unit of work and verified outcome. Log attempts, approvals, exceptions, rework, costs, and customer results. Review a representative sample. Set acceptance criteria and a decision date. Then record whether the evidence supports scale, fix, or stop.
Build measurement into the workflow. Instrument one outcome from intake to completion, then schedule the review that will decide what happens to the system next.