Runbook: MCP server unhealthy / degraded

Alerts: MCPHangarProviderDegraded (warning, mcp_hangar_mcp_server_state == 3 for 5m). The chart's MCPHangarProviderNotSeenHealthy also links here; it is described in provider-dead.

What it means

An MCP server's consecutive failures, usually failed health checks, reached max_consecutive_failures (default 3), and it is DEGRADED. While it is degraded, Hangar refuses calls to it with CircuitBreakerOpen (see circuit-breaker), and the recovery saga restarts it with a growing delay.

Diagnose

mcp_hangar_health_check_consecutive_failures
mcp_hangar_mcp_server_state                                # 3 = DEGRADED
histogram_quantile(0.95, sum by (le, mcp_server) (rate(mcp_hangar_health_check_duration_seconds_bucket[5m])))

curl -s -H "X-API-Key: $KEY" "<hangar>/api/mcp_servers/<id>/health"
curl -s -H "X-API-Key: $KEY" "<hangar>/api/mcp_servers/<id>/logs?lines=200"

Hangar's log shows the path: mcp_server_degraded_by_health_check: <id>, then McpServer <id> degraded, scheduling retry 1/3 in 5.0s, and either McpServer <id> recovered successfully after <n> retries or mcp_server_given_up.

Remediate

  • Container mode: check the pod/process — crashloop, bad image, missing env/secret.
  • Remote mode: check reachability/TLS/auth to the upstream endpoint (MCPHangarRemoteProviderUnreachable). Since 2.25.0 a refused connection fails a remote server's health check like a timeout: with the defaults the member leaves its group's rotation after 2 failed checks (health.unhealthy_threshold) and the server degrades after 3 (max_consecutive_failures), then reads DEAD once the recovery retries fail. The group circuit (circuit_breaker.failure_threshold, 10) does not open from health checks alone with these defaults: a degraded server is not probed again, so the group's count stops short of it. On 2.24.0 and earlier a refused connection did not fail the check at all (#1698), so a stopped remote upstream stayed READY; its tool-call errors were the signal.
  • Transient → a restart by the recovery saga succeeds and the server returns to READY.
  • Not transient → when the saga runs out of retries the server reads DEAD (4), MCPHangarProviderDegraded resolves and MCPHangarProviderDead fires; work provider-dead.

Escalate

Server owner if it's a specific integration; platform on-call if many servers degrade at once (shared dependency / DNS).