Runbook: batch high failure rate
Alert: MCPHangarBatchHighFailureRate (critical) — batch failure ratio exceeds the threshold for 3m.
What it means
rate(mcp_hangar_batch_calls_total{result="failure"}) / rate(mcp_hangar_batch_calls_total)
is high — batched multi-tool requests are failing as a whole.
Diagnose
sum by (result) (rate(mcp_hangar_batch_calls_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(mcp_hangar_batch_duration_seconds_bucket[5m])))
sum(rate(mcp_hangar_batch_truncations_total[5m])) # oversized batches?
sum(rate(mcp_hangar_batch_cancellations_total[5m])) # cancelled / timed out?
Remediate
- Truncations high → clients sending oversized batches (
MCPHangarBatchSizeTooLarge); advise smaller batches or raise the limit. - Cancellations high → per-call timeouts or concurrency starvation (
MCPHangarConcurrencyQueueBuildup). - Failures concentrated on one server → treat as
high-error-ratefor that server.
Escalate
If batch failures coincide with a circuit breaker (circuit-breaker), follow that runbook first.