Runbook: MCP Hangar not responding

Name: MCP Hangar
Author: MCP Hangar

Alerts: MCPHangarNotResponding (critical) · MCPHangarAllProvidersDown (critical)

What it means

Prometheus cannot scrape the Hangar (up{job="mcp-hangar"} == 0) for 1m, or no MCP server reports up while at least one is DEAD (4), for 1m. A pool that is merely all cold does not fire it. The gateway is effectively unavailable to clients.

Impact

All tool calls fail. This is a user-facing outage.

Diagnose

kubectl -n <ns> get pods -l app.kubernetes.io/name=mcp-hangar
kubectl -n <ns> describe pod <pod>          # crashloop / OOMKilled / probe failures
kubectl -n <ns> logs <pod> --tail=200
kubectl -n <ns> get endpoints <svc>         # is the Service backed by a ready pod?

up{job="mcp-hangar"}
sum(mcp_hangar_mcp_server_up)               # 0 ...
count(mcp_hangar_mcp_server_state == 4)     # ... while at least one server is DEAD?

Check readiness: GET /health/ready does not wait for a warm server. It fails when a configured durable event store fell back to memory, or, on a front door with tool_access.required_catalogue, while the listed servers are not yet projected. A not-ready pod is pulled from the Service.

Remediate

Crashloop → fix the failing dependency shown in logs; roll back the last deploy if it correlates.
OOMKilled → raise memory limits (see MCPHangarHighMemoryUsage).
All-providers-down → the servers are the problem, not Hangar; work provider-dead.
Scrape-only failure (app healthy) → check the ServiceMonitor/NetworkPolicy allowMonitoring.

Escalate

If not resolved in 15m or the root cause is infrastructural (node, network, registry), page the platform on-call.