AI agents from pilot to production: a controlled rollout plan

Define the questions the pilot must answer

Write down what you need to learn. Can the agent produce a useful draft for the supported product category? How much correction remains? Where is information missing? Do approvals reach the right owner? Connect these questions to acceptance criteria from the process brief.

For the hypothetical distributor, the question is not whether AI can write quotations in general. It is whether the agreed task works in its operating environment. Success with a simple catalog does not establish readiness for special pricing agreements or another product range.

Name the decision owner and possible conclusions: expand, continue a bounded trial, redesign the task or stop. If production rollout is the only acceptable conclusion, the pilot has become a demonstration of a decision already made rather than a source of useful evidence.

Begin with historical cases and known outcomes

Before touching live work, use cases with an established expected result. Include ordinary tasks and those needing clarification, handoff or rejection. Cover the intended launch scope rather than only the inputs the prototype already handles well.

Separate development cases from the acceptance set. Tuning instructions on the same cases later used to declare success can overstate readiness. Add unseen cases and retain a change record so that the team can understand what improved between versions.

Anthropic’s agent evaluation guidance addresses multi-step behavior and changes in the environment. Your initial checks should therefore cover the draft record, correct handoff and response to failure, not only the text of the final answer.

Use shadow mode to separate observation from execution

In shadow mode, the agent processes cases and prepares proposals while the established process still determines the business outcome. The purpose is comparison under more realistic conditions. A shadow proposal should not quietly become an operational change.

Enforce that separation in the implementation rather than labeling the environment “test.” Keep trial drafts distinct, disable out-of-scope operations and define who can see results. If both processes create real quotations or change the same case, observation is no longer separate from execution.

Compare the proposal with the human-completed result and investigate differences. Disagreement may reflect agent error, an unreliable source or an undocumented rule. Turn those findings into specific improvements rather than relying only on an overall satisfaction score.

Give the live pilot explicit boundaries

Select a defined set of cases, users and operations. For example: one supported product category, named salespeople and draft preparation only. Document the selection so that later conclusions remain attached to the scope actually tested.

Google SRE’s canary-release guidance describes limited exposure to a change with relevant monitoring. Apply the principle as controlled expansion of a business workflow. Technical success in a small group does not, by itself, establish a business effect across the company.

The pilot still needs varied cases within its boundary. Sending only perfectly structured requests will not reveal whether incomplete inputs are handed off properly. Limit scope while preserving a realistic sample of the situations that scope contains.

Measure accepted work and remaining human effort

The primary unit should be a task resolved against agreed criteria. Track accepted outputs, active review time, corrections and open exceptions. Model calls and generated-document counts can grow while the operating team spends more time fixing work.

For an operational cost calculation, include model execution, tools, infrastructure and review and correction effort. Divide that total by accepted tasks within the same scope and period. The result supports version comparisons but does not alone prove savings or additional revenue.

Distinguish gross task-time reductions from capacity the team can actually use. Five shorter tasks may create five new interruptions for a reviewer. Examine whether released time supports other agreed work, and account for the coordination effort introduced by the new process.

Set expansion and stop conditions

Thresholds should reflect process importance and acceptable ways of working. Define separate conditions for output quality, technical reliability and operational load. One universal accuracy rate is insufficient: an incorrect product code has different consequences from an awkward sentence.

Google SRE’s guidance on service objectives connects indicators to the expected service level. For an agent workflow, define usefulness from the operator’s perspective. An available draft with correct status is a stronger service outcome than a successful intermediate API response.

Where practical, expand one significant dimension at a time: a product category, more users or more autonomous operations. Changing all three makes regressions harder to diagnose. Before each expansion, also check whether the approval queue has enough capacity to handle the new volume.

Rolling back software does not undo business actions

A stop plan explains how new intake is paused, what happens to open cases and who takes over manually. Restoring the previous system version does not erase quotations or other effects already created in external systems.

Separate technical rollback from business correction. Already executed actions need an affected-case ledger and a reconciliation procedure. Assign responsibility before the pilot when correction requires a person. Waiting until an incident occurs leaves the most important operational question unanswered.

Rehearse the stop procedure using test cases. Verify that operations can find open work, new cases do not accumulate without owners and completed actions are not repeated after restart. This exercise reveals gaps that ordinary output-quality checks may miss.

Operational acceptance includes people and documentation

Provide concise instructions for normal tasks, status meanings, exception handling and the responsible contact. Employees should distinguish a waiting case from a completed one without asking the development team. That understanding is necessary for everyday use.

Record the model, instruction, tool and data-rule versions that make up the release. Apply the agreed checks when any component changes. A model update is not necessarily behaviorally neutral just because the visible interface stays the same.

Schedule a regular review of failures and improvement requests with the process owner. Choose the frequency according to volume and importance. The goal is controlled removal of recurring causes while retaining comparable results, rather than constant changes that make every observation obsolete.

Worked example: a pilot with mixed results

Suppose ordinary quotation drafts pass review but exceptions are handed off too late. Average preparation time improves while the age of incomplete cases increases. This does not support unrestricted expansion. It suggests that basic drafting works while exception ownership still needs attention. This is a hypothetical decision example, not a reported client result.

The next decision may retain the existing scope and improve handoff routing. Add an accountable receiver, a clear waiting reason and a reminder rule, then repeat relevant cases. If missing information causes the problem, target intake or the data source rather than automatically replacing the model.

Record the observation, interpretation, chosen action and evidence to examine next. Retain unresolved hypotheses too. A small sample may expose a clear defect without reliably estimating every rare failure rate. The strength of the conclusion should match the evidence actually collected.

After the fix, retain metric definitions and add fresh cases. Changing the model, task scope and handoff policy simultaneously would obscure what helped. Running the pilot therefore includes change management, not merely watching a dashboard or collecting individual impressions.

Define readiness for the next scope

The system is ready for a broader scope when tasks are verified, integrations work, handoffs are useful and a maintainer understands operational state. This does not mean errors disappear. It means the business can detect, contain and handle them without losing control of work.

NUMEDIA treats agent implementation as a connection between process design, data, development and operations. The series provides a foundation for that project: workflow architecture, the process brief, integration contracts, human handoffs and the rollout method.

For an initial discussion, bring one task, representative cases and an accountable owner. We can use them to define a feasible pilot and identify prerequisites. A clearly bounded task is a more useful starting point than a list of agents without a shared business outcome.

Sources and methodology

1. Canarying releases (Google SRE, accessed 2026-10-19)
2. Implementing SLOs (Google SRE, accessed 2026-10-19)
3. Demystifying evals for AI agents (Anthropic, accessed 2026-10-19)