AI Incident Response Starts Before the Incident

The worst time to discover how to stop an agent is after it begins doing the wrong thing quickly.

By then, teams are searching for the owner, reconstructing which credentials were active, debating whether shutting down the worker will interrupt an external action, and trying to preserve evidence without copying sensitive data into an ungoverned folder.

Traditional incident response already teaches that preparation matters. Agent systems raise the stakes because behavior can cross models, tools, sandboxes, external services, and human approvals in one operation. The failure may not look like a server outage. The system can remain healthy while making coherent, authorized, and wrong decisions.

Incident response has to be designed into the agent harness before launch.

Define What an AI Incident Is

Not every incorrect answer is an incident, and not every incident begins with a model error.

Useful categories include unauthorized effects, sensitive-data exposure, repeated or duplicate actions, policy bypass, corrupted memory, retrieval of cross-tenant data, compromised tools, unsafe model behavior, and failures that cannot be reconstructed.

Set severity by consequence and spread, not by how dramatic the transcript sounds. A quiet data leak may be more severe than a spectacular sandbox failure with no external access.

Define thresholds in advance. Which event pages security? Which pauses one workflow? Which disables a capability across the platform? Which requires customer or regulator communication?

Clear categories prevent the response from being negotiated under pressure.

Give Every Operation a Kill Path

A kill switch is only useful if the team knows what it stops.

Terminating a model invocation may leave tool calls in flight. Revoking a token may not undo a queued external action. Disabling one agent may leave delegated workers active. A long-running task may resume automatically from durable state after the process is killed.

Design containment at several levels: invocation, step, operation, capability, tenant, model route, and platform. Each control should have a documented effect and an observable acknowledgement.

Containment should also freeze automatic continuation. A task marked suspicious must not be picked up by a recovery worker that interprets the missing lease as permission to proceed.

The goal is not one giant red button. It is the ability to stop the smallest sufficient part of the system without pretending downstream effects vanished.

Preserve Evidence at Creation Time

Incident responders cannot reconstruct a decision if the system never recorded the context, policy, tool result, or artifact identity that shaped it.

Capture the operation hierarchy, context manifest, model and harness versions, tool calls, policy decisions, approvals, external operation identities, and verification evidence as the run proceeds. Store sensitive payloads under governed access and retain hashes or references in the operational trace.

Use append-only or tamper-evident storage for high-consequence events. Synchronize clocks and stable identifiers across services. Document retention and deletion requirements before traces become an accidental second customer database.

Decision reconstruction is an incident-response capability, not merely a debugging convenience.

Separate Containment From Erasure

The urge to clean up can destroy the evidence needed to understand the event.

Do not immediately delete the workspace, rerun the task, or overwrite the artifact. First isolate the environment, revoke authority, snapshot relevant state under policy, and record known external effects.

At the same time, do not keep sensitive material indefinitely because it might be useful. Incident evidence needs explicit custody, access control, and a retention clock. The response plan should balance forensic value with privacy and legal obligations.

Contain first. Preserve deliberately. Remediate after the state is understood.

Reconcile the Outside World

An agent incident is rarely contained entirely inside the AI platform.

Messages may have been delivered. Code may have been pushed. Credentials may have been used. Records may have changed. A payment may be pending despite a timed-out response.

Build an effect ledger keyed by durable operation identity. During response, query authoritative systems to classify each intended or attempted action as not started, pending, completed, reversed, or unresolved.

This is where idempotency becomes forensic infrastructure. It lets responders distinguish one authorized intent with several attempts from several distinct actions.

Do not assume the transcript is the external state. Reconciliation must read the systems that experienced the effect.

Prepare Capability-Specific Playbooks

“Disable the AI” is not a useful runbook.

The response to a compromised research connector differs from the response to unauthorized repository writes or leaked customer context. For each consequential capability, document owners, containment controls, credentials, external dependencies, evidence sources, reversal options, and customer impact.

Include expected safe degradation. If the model route is disabled, does the product queue work, fall back to read-only assistance, or stop entirely? If retrieval is suspected, can users continue with a clean source set? If one tool is compromised, can the rest of the workflow operate without it?

Playbooks should reflect how the system is actually assembled, not an architecture diagram from launch week.

Put Humans Into Named Roles

During an incident, everyone should not review everything.

Assign an incident commander, technical containment owner, evidence lead, domain owner, communications lead, and decision authority appropriate to the severity. Smaller events can combine roles, but ownership should remain explicit.

The model may assist with timeline construction, log classification, and hypothesis generation. It should not be the sole authority interpreting an incident in which its own behavior is part of the evidence.

Human decisions should be recorded with the information available at the time. That history prevents later certainty from rewriting why a reasonable containment choice was made under ambiguity.

Rehearse Ambiguous Incidents

Teams like drills with a clear bad event and an obvious fix. Agent incidents are often stranger.

The model may have followed a stale instruction from an approved source. A tool may report success while the effect remains unknown. An approval may be valid but attached to a different artifact version. An attacker may have used a permitted destination as an exfiltration channel.

Run tabletop exercises where the evidence conflicts. Practice stopping durable workflows, rotating scoped credentials, locating delegated workers, reconciling external state, and communicating uncertainty.

Also practice normal operations after containment. Recovery under pressure can create a second incident if teams restore broad access or replay unresolved work.

Recovery Is a New Release

Restarting the same configuration after deleting a bad run is not recovery.

Identify the violated invariant and change the control that should enforce it: retrieval admission, tool scope, sandbox egress, approval binding, verification, or model route. Add a representative regression case. Validate the fix against normal work and the incident scenario. Roll out gradually with heightened monitoring.

Review durable memory and derived artifacts for contamination. A malicious or incorrect fact may persist after the original environment is gone. Trace which later outputs consumed the affected state.

Recovery should produce evidence that the system is safe to trust again, not merely available.

Learn Without Reducing Everything to “Model Error”

The model is part of the causal chain and rarely the whole explanation.

Ask why the content entered context, why the tool was available, why policy permitted the action, why verification missed it, and why containment did or did not work. A model-level fix may reduce recurrence while leaving the architecture vulnerable to the next variation.

Post-incident reviews should produce concrete changes in controls, evaluation cases, playbooks, and ownership. Track whether those changes are implemented and exercised.

The most valuable lesson is the one the system can enforce.

Preparedness Is Part of Autonomy

Autonomous systems are sometimes sold as needing less operational involvement. In reality, greater delegated authority requires better preparation for the moments when behavior diverges from intent.

Define incidents. Build layered kill paths. Preserve governed evidence. Reconcile external effects. Write capability-specific playbooks. Assign human roles. Rehearse ambiguity. Treat recovery as a new release.

The incident plan is not an admission that the agent will fail.

It is evidence that the organization understands what the agent is capable of doing.