13 -- Production Checklist
Before you go live, walk through this list. Concept: What Hangar costs — and where it stops
Security
- TLS termination configured (reverse proxy or load balancer)
-
auth.enabled: trueandauth.allow_anonymous: false - API keys created for each service principal
- RBAC roles assigned with least-privilege
- Tool access policies set for sensitive tools
- Secrets use environment variable interpolation (
${VAR}), not plain text in config - Docker MCP servers use
read_only: trueandnetwork: nonewhere possible
Reliability
- Health checks enabled on all MCP servers (
health_check_interval_s) - Circuit breaker thresholds tuned (
max_consecutive_failures) - MCP Server groups configured for critical MCP servers (at least 2 members)
-
min_healthyset to match your SLA requirements - Idle TTL set appropriately (300s for subprocess, 600s for containers)
- Rate limiting enabled to prevent overload
- Storage decided once:
persistence.backendset tosqlite(durable volume) orpostgresql-- since 2.5.0, a backend serves every persisted concern or startup is refused. On 2.4.0 and earlier:event_store.driver: sqliteplusauth.storage.driver, chosen separately - If
postgresql: the driver installed -- it is an extra, not a base dependency, so a pip install needspip install "mcp-hangar[postgres]". The published image already carries it
Observability
- Prometheus scraping
/metricsendpoint - Grafana dashboards shipped by the chart (
dashboards.enabled=true), or imported by hand frommcp-hangar/files/dashboards/ - Alertmanager rules configured for:
- MCP server state transitions to DEAD
- Circuit breaker OPEN events
- Health check failure rate above threshold
- Tool call error rate above threshold
- Structured JSON logging enabled (
MCP_JSON_LOGS=true) - Log level set to
INFOfor production (MCP_LOG_LEVEL=INFO)
Configuration
- Config file reviewed for correctness (no
validatesubcommand exists) - Hot-reload tested via the
add/removeAPI and viaSIGHUP(graceful config reload) - Environment-specific configs separated (dev/staging/prod)
Deployment
- Running behind a reverse proxy (nginx, Caddy, Envoy)
- Health probe endpoints exposed for orchestrator (
/health/live,/health/ready,/health/startup) - Graceful shutdown configured (SIGTERM handling)
- Resource limits set (memory, CPU) for container deployments
- Persistent volume for event store SQLite database
- Docker image pinned to specific version tag, not
latest
Kubernetes (if applicable)
The MCP-Hangar Operator is an external component shipped from hangar-operator. See Recipe 11 for install instructions.
- MCP-Hangar Operator installed (see Recipe 11 prerequisites)
- CRDs applied (
MCPServer,MCPServerGroup,MCPDiscoverySource) - RBAC (Kubernetes) configured for operator service account
- Network policies restricting MCP server-to-MCP server communication
- Resource requests and limits in Helm values
- PodDisruptionBudget for Hangar deployment -- meaningful only with more than one replica, which needs the row below
More Than One Replica (if applicable)
Since 2.5.0. On 2.4.0 and earlier, run a single instance: replicas there disagree with each other and the failure is silent.
- One PostgreSQL every replica shares (
persistence.backend: postgresql), with the driver installed on every replica --pip install "mcp-hangar[postgres]"; the published image already carries it - A
coordination:block, with the samelease_ttl_son every replica -- the tenure in force is written by whoever holds the lease, so one stale ConfigMap sets the failover window for the whole set - Every server in
remotemode;subprocess/dockerare single-instance - Discovery configured on every replica, not one -- it runs on the lease holder, and the holder can be any of them
- Verified pod by pod, not through the Service: exactly one answers
manages_fleet: trueatGET /api/system - Fleet-wide request cap at the ingress -- Hangar's own limit is per pod
- Rolling update rehearsed: two versions run against one database for its duration
See 25 -- Running More Than One Replica.
Testing
- Failover tested: kill a primary MCP server, verify backup takes over
- Cold start tested: invoke a tool on a cold MCP server, verify latency
- Rate limit tested: flood API, verify 429 responses
- Auth tested: invalid key returns 401, insufficient role returns 403
- Config reload tested: edit config.yaml, verify changes apply
- Recovery tested: kill all MCP servers, verify they reinitialize
Runbook
- Incident response documented
- MCP Server restart procedure documented
- Config rollback procedure documented
- Contact list for MCP server owners maintained