MCP Server Incident Response Runbook

An MCP incident is any production condition that prevents supported clients from discovering, authorizing, calling, or completing important work through an MCP server. Use this runbook to classify the failure, isolate the first broken step, mitigate user impact, and add a regression check before closing the incident.

1. Confirm the blast radius

• Which server, environment, region, and deployment version are affected?
• Which clients, tenants, users, tools, and workflows are affected?
• Did the condition begin after a deployment, schema change, credential change, dependency incident, or client update?
• Is the issue a complete outage, a degraded path, or a low-volume isolated failure?

2. Classify the first failing layer

• Transport: the endpoint cannot be reached or returns unexpected protocol responses.
• Authorization: clients receive 401, 403, discovery, issuer, audience, or scope errors.
• Catalog: expected tools, resources, or prompts are missing, renamed, stale, or malformed.
• Tool execution: a handler, schema, dependency, timeout, or tool-level error is failing.
• Workflow: individual calls succeed but the agent retries, stalls, or never reaches the outcome.

3. Inspect the first broken event

Reconstruct one affected session or trace. Identify the last successful event, the first failure, and what the client did next. Compare the client, tool, method, server version, environment, duration, outcome class, and deployment timestamp. A later error may be a consequence of the first one.

4. Apply the smallest safe mitigation

• Rollback the deployment if a recent release changed the failing path.
• Disable or gate one broken tool if the rest of the catalog is healthy.
• Rate-limit a retry loop before it exhausts dependencies or increases cost.
• Restore or rotate credentials without exposing them to logs or traces.
• Use a dependency fallback or maintenance response when the upstream service is unavailable.
• Communicate affected clients, expected impact, and the next update time.

5. Investigate common MCP failure patterns

• A green HTTP status with a tool-level error: parse the result and inspect the error state.
• Repeated identical arguments: inspect schema shape and error recovery guidance.
• High p95 with normal average latency: find the slow dependency behind the tail.
• One client failing: compare protocol version, catalog cache, authorization, and argument behavior.
• Connected sessions with no tool calls: inspect discovery, descriptions, and client permissions.

6. Verify recovery

Run a clean check through discovery, authorization, a critical tool, and a representative workflow. Confirm that the original symptom is gone, the mitigation did not introduce a security or catalog regression, and new telemetry shows healthy behavior for the affected client and tool.

7. Close with a durable fix

• Write a timeline with exact timestamps and versions.
• Record the root cause separately from contributing conditions.
• Add a unit, integration, synthetic, schema, or client-compatibility regression test.
• Update the alert or dashboard that should have shortened detection.
• Document any catalog, auth, transport, or client migration required.
• Review whether sensitive data appeared in logs, traces, or incident artifacts.

Frequently asked questions