AI Agent Shadow Mode: A Practical Evaluation Plan

AI agent shadow mode evaluates proposed work alongside the real process while preventing the candidate from making production changes. It helps reveal missed evidence, inappropriate recommendations and reviewer effort before broader authority is granted. Its value depends on isolation and a credible comparison: an agent that secretly triggers writes, sees future outcomes or receives only easy cases is not being evaluated honestly.

This plan is for operations teams introducing a worker into case preparation or exception handling. It produces a comparison dataset and a promotion decision. It does not establish that a model is safe for every process or that shadow performance will transfer unchanged to production. Define the exact procedure, case population and permissions before collecting results.

Define the shadow-mode boundary

Run the candidate with a separate execution identity and a tool configuration that cannot change production state. Restrict reads to approved sources and purposes. A hidden user interface or an instruction saying “do not execute” is not an enforcement boundary. Check the connectors, queues, callbacks and scheduled tasks through which a proposal could become an action.

Keep the operational process authoritative. The human team should continue its normal decision path, and the candidate's output should be clearly separated from the case of record. Decide whether reviewers may see the candidate suggestion before making their own decision. If they do, the comparison measures assisted work, not an independent baseline.

The NIST AI RMF Playbook discusses measurement and management as part of ongoing risk work. Our shadow plan turns that general need into a concrete process exercise. NIST does not prescribe the sample sizes, acceptance thresholds or promotion steps proposed here.

Freeze the comparison before starting

Select cases by a documented sampling rule. Include routine work, incomplete evidence, conflicting sources, unusual exceptions and cases outside the worker's scope. Record exclusions and why they were necessary. If the business cannot provide representative cases, report that limitation rather than presenting a convenient sample as the entire workload.

Keep information available at the decision time separate from information learned later. A historical case may contain a final resolution that the original team did not know. Exclude that resolution from the candidate input. Otherwise the evaluation rewards hindsight rather than the ability to prepare a useful next step.

Score the work reviewers actually need

For an audit-preparation task, score whether the draft identifies the relevant records, distinguishes missing evidence, cites the correct source versions and routes unresolved questions. Assess unsupported conclusions separately from writing quality. A polished narrative with one invented source may create more work than a plain draft that accurately reports a gap.

Use a rubric with observable criteria. “Helpful” is too broad for diagnosing a regression. “The reviewer can find the source for each material assertion without another search” is more specific. Let reviewers record disagreement and uncertainty. Adjudication should explain the contested point, not merely overwrite one review with another number.

Record review effort as a distribution, including difficult cases. A short median can coexist with a long tail that exhausts specialist capacity. Do not claim savings by subtracting model response time from total manual handling time. Compare equivalent work, including validation, correction, coordination and the cost of unresolved cases.

Test leakage and side effects deliberately

Insert a harmless synthetic instruction into a retrieved document asking the worker to send its case elsewhere. The candidate should treat the document as evidence, not as permission. Observe proposed actions as well as actual dispatch attempts. A blocked attempt reveals a behavior problem even when the isolation layer prevents harm.

Attempt a direct call to the write connector with the shadow identity. Verify denial at the receiving boundary, not just in the application interface. Test a queued action after the run has finished and a callback after the candidate has been disabled. Shadow isolation must cover delayed paths, not only the synchronous request.

If a write occurs, stop the exercise and determine its scope before continuing. Preserve the run, connector configuration and affected transaction references. Repair the enforcement path and repeat the isolation tests. Removing the offending row from the quality report does not repair the experiment.

Decide when to move to a bounded canary

Promotion should be a separate decision with new permissions and explicit limits. Review unresolved failures, representative coverage, reviewer capacity and operational ownership. Define which cases remain excluded. A canary may allow preparation for one team while retaining human approval for every consequential action; it need not begin with autonomous execution.

Use the same candidate identifier in the promotion record and the deployed configuration. If a developer changes the procedure after the evaluation, assess whether the evidence still applies. Preserve the old result as history rather than relabelling it as evidence for the new candidate.

Repeat the exercise when the workload changes

Shadow evaluation is also useful after a source-system change or a new procedure version. Retain a fixed regression set and add fresh cases from the current workload. Report these populations separately. A fixed set makes releases comparable; a fresh sample helps identify changes in the real process that the old set cannot represent.

For the worksheet, start with five case categories and record one expected failure behavior for each. Run the candidate twice on the same frozen inputs to observe variation. This is a diagnostic exercise, not a statistically representative benchmark. The production acceptance plan must justify its own sampling, thresholds and confidence claims.

The final report should state what the candidate saw, what it proposed, what was blocked, what reviewers changed and which claims remain untested. That report gives the process owner a basis for a limited next step. A single accuracy percentage cannot explain those boundaries.

Use the worksheet and related implementation guides

Use the parent guide for the wider architecture, then apply this article to one bounded implementation. The adjacent articles cover related decisions without replacing this worksheet.

• Parent guide
• Related implementation
• Adjacent control

Download the editable worksheet. Next step.

Engineering draft, 18 October 2026. Primary sources checked on this date; specialist review remains pending. The worksheets describe proposed tests, not measured customer results. OATI remains a developer preview. Local implementation evidence does not establish independent assurance, customer acceptance or production readiness for a particular deployment.