AI Agent Production Readiness: Evidence Gates
Assess AI agent production readiness with candidate-bound evidence gates, failure exercises, accountable owners and a downloadable release worksheet.
AI agent production readiness depends on evidence for one exact deployment: its procedure, data permissions, tool access, evaluation results, operational owner and recovery path. A successful demonstration is useful, but it does not establish that the same controls will hold with customer data, an unavailable dependency or an interrupted approval. Make the release decision against a versioned evidence package.
This article is for enterprise architects and process owners preparing a bounded first deployment. Its deliverable is a release worksheet that separates demonstrated behavior from unresolved conditions. The example concerns an agent preparing an internal audit case; the accountable reviewer still decides whether the case is acceptable. It is a proposed engineering method, not a certification scheme.
What should a production-readiness gate decide?
A gate answers a specific question with an accountable decision. “Security complete” cannot be verified without knowing the threat, environment and evidence. “The candidate cannot submit an external message while approval is unavailable” can be tested against a named connector and deployment. Record the expected result before running the exercise, and retain the actual result even when it blocks release.
Separate scope approval, technical verification and business acceptance. A platform engineer can establish that a stop control works; the process owner decides whether the remaining workflow is suitable. Neither decision supplies missing data-use permission. The NIST AI RMF Playbook provides suggested risk-management actions. The release worksheet below is our implementation proposal, not a mandatory NIST checklist or proof of conformity.
Bind every item to the same candidate
Create a release identifier before assembling evidence. Bind it to the procedure revision, worker configuration, approved tools, data scope, environment and evaluation dataset. A result from a different candidate can inform investigation, but it cannot silently satisfy the current gate. Record any accepted transfer of evidence and the reason it remains relevant.
| Gate | Evidence to collect | Who evaluates it | Condition that blocks release |
|---|---|---|---|
| Process scope | Named owner, allowed outputs and excluded actions | Process owner | The agent can decide something reserved for a person |
| Data access | Approved sources, purpose and tested access boundaries | Data owner | A case can retrieve another team's restricted material |
| Evaluation | Frozen case set, scoring rules and unresolved failures | Independent evaluator | A critical failure is averaged into an acceptable score |
| Human handover | Exact review object, assignment and expiry behavior | Operations lead | Approval can be reused after the prepared action changes |
| Recovery | Stop, restore and reconciliation exercise records | Service owner | Restart can repeat an action with an unknown outcome |
| Acceptance | Decision on this candidate and its remaining limitations | Accountable sponsor | Approval refers to a different version or environment |
An evidence reference should resolve to a controlled record with an owner and date. Do not copy sensitive case material into a management spreadsheet. Store a reference, the relevant finding and access instructions. A dashboard showing six green boxes is not a replacement for the underlying records.
Walk one synthetic case through the gates
Prepare a case containing an approved procedure, two source records and one deliberately missing record. Run the candidate with external write actions disabled. The expected result is a draft that identifies the missing evidence and routes the case to the correct reviewer. It must not invent the absent record or mark the case accepted.
Next, make the reviewer unavailable. Observe whether the case remains pending, moves to an authorised backup or becomes overdue. Then replace the prepared draft after an approval has been recorded. The system should require review of the changed object according to the agreed approval policy. Record the object versions and decision references rather than relying on screenshots alone.
Finally, interrupt the connector after it may have accepted a submission but before the worker receives confirmation. The expected outcome is uncertainty followed by reconciliation with the receiving system. Restarting the worker should not be treated as permission to submit again. This exercise tests a different boundary from a clean denial before submission.
Keep blockers separate from accepted limitations
Use three decision states: satisfied, blocked and accepted limitation. A limitation needs a bounded impact, compensating control, owner and expiry. It must not become a way to rename a failed critical test. For example, a reviewer may accept a restricted document format during a pilot; missing isolation between customer cases is a different class of problem.
Avoid universal numeric thresholds. The tolerable error rate depends on the task, the cost of mistakes and what the human reviewer can detect. Define critical failures separately from routine quality scores, and report both the numerator and denominator. Ten successful easy cases say little about a rare but consequential exception that was never sampled.
Reopen the decision when its assumptions change
A release decision has a scope and a lifetime. Reconsider it when the model configuration, procedure, connector, source permissions or authority boundary changes. Some changes need a focused regression test; others invalidate the entire evaluation. Define that distinction before the first production incident rather than during it.
If a gate becomes invalid after release, identify affected cases, stop new work within the relevant scope and preserve existing evidence. Decide whether in-flight cases can safely finish under the old configuration. A rollback of software does not undo a message already sent or restore a deleted record. Recovery needs a business-state decision as well as a deployment command.
Verify that the worksheet can actually block a release
Run a tabletop exercise with one missing evidence item, one expired approval and one result from another candidate. Ask an evaluator who did not build the worker to decide whether the package is sufficient. If they cannot determine the candidate, scope or outstanding conditions without interviewing the implementer, improve the package.
Keep failed exercises and their corrections. A replacement test result should point to the failed run and explain what changed. Record who accepted the correction. This makes the release history useful during incident reconstruction and prevents an apparently complete folder from concealing the sequence that led to acceptance.
The worksheet is ready when every gate has a decision rule, evidence location and accountable owner, and when a missing prerequisite produces a visible stop. It is not ready merely because all columns contain text. Start with one procedure and one environment; expand only after the same evidence discipline can be maintained for the larger scope.
Use the worksheet and related implementation guides
Use the parent guide for the wider architecture, then apply this article to one bounded implementation. The adjacent articles cover related decisions without replacing this worksheet.
Download the editable worksheet. Next step.
Engineering draft, 18 October 2026. Primary sources checked on this date; specialist review remains pending. The worksheets describe proposed tests, not measured customer results. OATI remains a developer preview. Local implementation evidence does not establish independent assurance, customer acceptance or production readiness for a particular deployment.