Fail Open vs Fail Closed for Enterprise AI Agents

13 minute read

Fail open vs fail closed is the wrong question when it is treated as one global switch. An enterprise AI agent asks to rotate a database credential at 02:13. The gateway can verify the request signature, but the trust lookup service is timing out. Its revocation cache is seven minutes old. The replay store has lost quorum. The maintenance window ends in 12 minutes.

Should the action run?

This is not one availability decision. It is several decisions about different kinds of state: identity, authority, policy, revocation, replay, usage and evidence. A single failOpen: true switch cannot express the risk.

The useful design question is: what is the minimum fresh state required to authorize this exact action, and which missing dependency makes the answer unknowable?

Fail open vs fail closed starts with separate planes

The control plane manages slow-changing objects and administration:

• issuer and key publication
• Passport and mandate lifecycle
• policy authoring and bundle distribution
• revocation
• trust-anchor configuration
• approval workflows

The data plane handles live requests:

• signature and audience verification
• request-context binding
• replay claims
• deterministic policy evaluation
• usage reservation
• upstream execution
• receipt generation

If every data-plane request calls a central SaaS service, a network partition becomes a business outage. It also creates a tempting bypass: operators under pressure may disable the gateway to restore service.

Local verification avoids that dependency, but it does not remove freshness questions. A cached key may have been compromised. A mandate may have been revoked. A policy bundle may have been replaced. Local enforcement must know how stale each input may become for each risk class.

Classify actions before the outage

Do not invent outage behavior during an incident. Put each operation into an explicit class.

action_classes:
  public_read:
    materiality: low
    trust_unavailable: allow_with_stale_cache
    max_revocation_age: 30m
    receipt_required: true

  internal_sensitive_read:
    materiality: medium
    trust_unavailable: deny
    max_revocation_age: 5m

  credential_rotate:
    materiality: high
    trust_unavailable: deny
    replay_unavailable: deny
    usage_unavailable: deny
    approval_required: true

  supplier_payment:
    materiality: high
    trust_unavailable: deny
    reconciliation_unavailable: hold

The names are yours. The important part is that the behavior is narrow, reviewable and testable. A fail-open rule should normally cover only low-risk reads with bounded data exposure. It should never emerge from a caught exception deep in middleware.

Package enough trust state for local decisions

A signed policy bundle can give a gateway a stable local basis for decisions:

{
  "bundle_id": "policy:prod-eu:2026-08-10.4",
  "tenant_id": "tenant:acme",
  "valid_from": "2026-08-10T00:00:00Z",
  "refresh_after": "2026-08-10T00:05:00Z",
  "expires_at": "2026-08-10T00:20:00Z",
  "trust_anchors": ["oati:issuer:intelliger:root-1"],
  "actions": {
    "credential.rotate": {
      "required_assurance": "production",
      "max_proof_age_seconds": 120,
      "approval_roles": ["security_on_call"],
      "on_trust_unknown": "deny"
    }
  },
  "digest": "sha256:...",
  "proof": { "type": "DataIntegrityProof", "...": "..." }
}

Verify the bundle when it arrives, store it atomically, and retain the last known good version until its hard expiry. refresh_after tells the gateway to seek a newer bundle. expires_at tells it when the bundle is no longer a valid basis for material authorization.

This distinction matters. A refresh failure is a warning. Expiry is a policy event.

The local package may include issuer chains, verification keys, signed service profiles and compact revocation state. Keep private payload data out of it. Sign each artifact or sign a manifest that binds every artifact digest.

Revocation freshness is action-specific

Short-lived credentials reduce exposure, but they do not eliminate revocation. An employee can leave, an agent key can be compromised, or a mandate can be withdrawn before expiry.

Define a maximum revocation age per action class. The gateway records when it last obtained an authenticated status view, not merely when a cache entry was read.

function revocationUsable(action: ActionClass, cache: RevocationSnapshot, now: Date) {
  if (!verifySnapshot(cache)) return false
  const ageMs = now.getTime() - Date.parse(cache.observed_at)
  return ageMs <= action.maxRevocationAgeMs
}

Negative caching needs care. "Not revoked" is a statement with a freshness limit, not a permanent property. Positive revocation can usually be cached through the object's natural expiry.

When connectivity returns, invalidate affected objects by target ID. Rotating one key should not require flushing every tenant's trust cache, but leaving a compromised key in process memory is worse than a broad cache miss.

Replay storage is live authorization state

A gateway can verify signatures and policies offline, but replay protection needs coordination when multiple replicas serve the same audience. If the replay store is unavailable, each replica can accept the same nonce.

For material actions, deny when an atomic nonce claim cannot be made. A process-local fallback is unsafe unless routing guarantees that all requests for the relevant scope reach one process, and that guarantee survives failover. It usually does not.

const claim = await replayStore.claim({
  key: `${verificationMethod}\0${audience}\0${nonce}`,
  expiresAt: proof.expires
})

if (claim === 'duplicate') return deny('REPLAY_DETECTED')
if (claim === 'unavailable') return deny('REPLAY_STATE_UNAVAILABLE')

Low-risk reads may use a different policy, such as a local bounded cache plus a response marker. Be honest about the assurance downgrade. The receipt should record that the decision used stale or degraded trust state.

Outage behavior belongs in the decision result

A generic 503 Service Unavailable tells an agent too little. Return a structured denial without exposing sensitive infrastructure detail:

{
  "decision": "deny",
  "reason_code": "TRUST_STATE_UNAVAILABLE",
  "transaction_id": "urn:oati:txn:01K...",
  "retryable": true,
  "retry_after_seconds": 30,
  "safe_to_retry_with_same_idempotency_key": true,
  "correlation_id": "req-7f2c..."
}

Differentiate an unavailable decision from a policy denial. Agents should not keep retrying a forbidden action. They may retry a temporary trust failure, but only with the same business idempotency key.

Do not return a detailed list of missing internal services to an untrusted caller. Operators can retrieve that through the correlation ID.

Partial execution is the harder outage

The gateway may authorize an action, reserve its usage, send it upstream and lose connectivity before receiving the result. At that point, "deny" is no longer available. The action may already have happened.

Model the state explicitly:

RECEIVED
  -> VERIFIED
  -> AUTHORIZED
  -> RESERVED
  -> DISPATCHED
  -> SUCCEEDED | FAILED | UNCERTAIN
  -> RECONCILED

If a dispatched write becomes uncertain, keep its budget or one-time authority reserved. Query the upstream system using the original idempotency key or provider reference. Do not ask the language model whether a retry seems safe.

Receipt generation has a similar split. You can issue a signed receipt that says the request was authorized and dispatch was attempted. You cannot truthfully mark the business result as successful until the upstream result is known.

{
  "result_status": "uncertain",
  "authorization_decision": "allow",
  "execution": {
    "dispatched_at": "2026-08-10T02:13:41Z",
    "provider_reference": null
  },
  "degraded_dependencies": ["upstream_response_path"]
}

An evidence record should not make an uncertain event look complete.

Recovery needs monotonic rules

When the control plane returns, newer state can invalidate assumptions made during the outage. Recovery code should not blindly overwrite local state.

Use versions or epochs for policy and revocation snapshots. Reject rollback unless an authorized rollback record exists. Apply object status changes monotonically where possible: a revoked mandate should not become active because an older cache snapshot arrived late.

Then reconcile every transaction left in RESERVED, DISPATCHED or UNCERTAIN:

1. Fetch the provider result by idempotency key or reference.
2. Confirm the exact request fingerprint.
3. Commit or release the reservation according to policy.
4. Issue a final receipt or a linked correction record.
5. Alert when no authoritative outcome can be established.

Never mutate a signed receipt. Append a new signed record that references the previous one.

Run failure drills as acceptance tests

Unit tests for cache helpers are not enough. Exercise the whole path with injected faults.

Measure more than availability. Record how many actions ran on stale state, how old that state was, how many executions became uncertain and how long reconciliation took.

Keep the emergency path inside the model

Some organizations need a break-glass route. It should use stronger identity, narrower commands, short expiry and separate evidence. Bypassing the gateway with a shared administrator token destroys the audit chain at the moment it matters most.

A practical emergency mandate can bind:

• the incident or change ticket
• a named operator and accountable organization
• one action on one resource
• a five-minute window
• a second approver
• a proof-bound temporary credential
• mandatory post-action review

The emergency path should be tested during normal operations. An untested break-glass procedure is a story, not a control.

Related engineering guides

• Use just-in-time access for production AI agents to constrain the capability that survives authorization.
• Test degraded uniqueness guarantees with the replay attack prevention guide.

What OATI supports today

OATI's developer preview includes local signature and trust verification, audience and time checks, replay protection, deterministic mandate evaluation, target-specific revocation and a reference Envoy authorizer path. Public lookup and discovery have a deployed vertical slice. Signed receipt primitives can be verified offline.

The full production claim is deliberately narrower. A production gateway outage and cache-failure drill remains pending. The durable evidence and dispute worker is incomplete, and the independent cryptographic and protocol review is still open. The policy compiler and bundle service are target-state components rather than a finished production subsystem.

Implementation checklist

• Classify every action by materiality before deployment.
• Define hard expiry and refresh timing for local trust artifacts.
• Sign bundles and bind every included artifact by digest.
• Set revocation freshness by action class.
• Deny material actions when trust, replay or usage state is unknown.
• Make any fail-open behavior explicit and limited to bounded reads.
• Return structured, retry-aware decisions with correlation IDs.
• Carry the same idempotency key across safe retries.
• Preserve reservations after uncertain dispatch.
• Reconcile against the authoritative upstream system.
• Append correction evidence instead of mutating receipts.
• Reject policy and revocation rollback.
• Keep emergency authority narrow, short-lived and approved.
• Inject lookup, cache, replay, signer and upstream failures in acceptance tests.

The goal is not to make the trust control plane impossible to lose. Distributed systems do not offer that bargain. The goal is to make every degraded decision predictable, bounded and reconstructable before an outage forces the issue.