Observability Guide

New to this? Governance observability is the concept behind this page.

This guide covers MCP Hangar's observability features: metrics, tracing, logging, and health checks.

Table of Contents

• Quick Start
• Monitoring Stack
• Metrics
• Grafana Dashboards
• Alerting
• Tracing
• Langfuse Integration
• Logging
• Health Checks
• SLIs/SLOs
• Troubleshooting
• Best Practices

Quick Start

Prerequisites

# Core package
pip install mcp-hangar

# For full observability support -- these are two separate extras
pip install mcp-hangar[opentelemetry,langfuse]

Ship the Dashboards and Alerts

MCP Hangar does not ship a monitoring stack. Bring your own Prometheus and Grafana; what Hangar maintains is the payloads that go into them, and those ship with the Helm chart:

helm upgrade --install hangar mcp-hangar/mcp-hangar \
  --set serviceMonitor.enabled=true \
  --set prometheusRule.enabled=true \
  --set dashboards.enabled=true

See Monitoring Stack for what each value renders and where the payloads live.

Start MCP Hangar with Metrics

# HTTP mode (exposes /metrics endpoint)
mcp-hangar serve --http --port 8000

# With custom config
MCP_CONFIG=config.yaml mcp-hangar serve --http --port 8000

Verify metrics are exposed:

curl http://localhost:8000/metrics | grep mcp_hangar

Monitoring Stack

Architecture

flowchart LR
    hangar["MCP Hangar<br/>:8000/metrics"]
    prom["Prometheus<br/>:9090"]
    grafana["Grafana<br/>:3000"]
    am["Alertmanager<br/>:9093"]

    hangar -->|scrape| prom
    prom -->|query| grafana
    prom -->|alerts<br/>from alert rules| am

What Hangar Ships, and What You Bring

Prometheus, Grafana and Alertmanager are yours to run. Hangar maintains the three payloads that go into them, and the mcp-hangar Helm chart is the only place they ship from:

Everything else is your own: a raw Prometheus scrape config, Alertmanager routing, and Grafana provisioning are not shipped in any form. The examples below are illustrations of what to write, not files you can copy from this project.

Prometheus Configuration

Without the Prometheus Operator, scrape the gateway directly:

scrape_configs:
  - job_name: 'mcp-hangar'
    static_configs:
      - targets: ['host.docker.internal:8000']
        labels:
          service: 'mcp-hangar'
          tier: 'application'
    metrics_path: /metrics
    scrape_interval: 10s
    scrape_timeout: 5s

On Kubernetes, prefer serviceMonitor.enabled=true. If you are not running the Prometheus Operator, use service discovery:

scrape_configs:
  - job_name: 'mcp-hangar'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: mcp-hangar
        action: keep

Metrics

MCP Hangar exports Prometheus metrics at /metrics. All metrics use the mcp_hangar_ prefix.

Currently Exported Metrics

Tool Invocations

Example queries:

# Tool call rate by mcp_server
sum(rate(mcp_hangar_tool_calls_total[5m])) by (mcp_server)

# P95 latency by tool
histogram_quantile(0.95, sum(rate(mcp_hangar_tool_call_duration_seconds_bucket[5m])) by (le, tool))

# Error rate
sum(rate(mcp_hangar_tool_call_errors_total[5m])) / sum(rate(mcp_hangar_tool_calls_total[5m]))

Batch Invocations

Example queries:

# Batch success rate
sum(rate(mcp_hangar_batch_calls_total{result="success"}[5m]))
/ sum(rate(mcp_hangar_batch_calls_total[5m]))

# Average batch size
rate(mcp_hangar_batch_size_sum[5m]) / rate(mcp_hangar_batch_size_count[5m])

Health Checks

Example queries:

# Unhealthy mcp_servers (>2 consecutive failures)
mcp_hangar_health_check_consecutive_failures > 2

# Health check success rate
sum(rate(mcp_hangar_health_checks_total{result="healthy"}[5m])) by (mcp_server)
/ sum(rate(mcp_hangar_health_checks_total[5m])) by (mcp_server)

MCP Server Lifecycle

Discovery

HTTP Transport

Messages (stdio + HTTP)

Rate Limiting

Approval Gate

Since 2.7.0.

Example queries:

# Armed and unmanned: the gate is holding calls and nobody is being told.
sum(rate(mcp_hangar_approval_requests_total[15m])) by (channel)
  - sum(rate(mcp_hangar_approval_deliveries_total{outcome="sent"}[15m])) by (channel)

# The same story from the other end: holds ending in expiry rather than a decision.
sum(rate(mcp_hangar_approval_decisions_total{decision="expired"}[1h])) by (channel)
  / sum(rate(mcp_hangar_approval_decisions_total[1h])) by (channel)

# A configured adapter that is failing rather than absent.
sum(rate(mcp_hangar_approval_deliveries_total{outcome="failed"}[5m])) by (channel)

outcome="not_notified" tracking requests one-for-one means the resolved channel reaches nothing outside the process. That is not an outage — held calls still expire closed and stay resolvable over REST — but every gated call waits out its timeout first, which from the client side looks like a broken gateway. The startup check reports the same condition at boot; see Configuration → approvals.

GC (Garbage Collection)

Grafana Dashboards

Four maintained dashboards ship with the Helm chart. dashboards.enabled=true renders each as a ConfigMap labelled grafana_dashboard, which the Grafana sidecar imports on its own; the JSON lives in mcp-hangar/files/dashboards/.

Overview Dashboard

File: overview.json URL: http://localhost:3000/d/mcp-hangar-overview

Provides high-level system health:

• Request rate and error rate trends
• Latency percentiles (P50, P95, P99)
• MCP Server health status
• Batch invocation success/failure rates
• Health check results
• GC cycle performance

MCP Server Details Dashboard

File: provider-details.json URL: http://localhost:3000/d/mcp-hangar-provider

Deep dive into individual MCP servers:

• Tool call breakdown by tool name
• Per-tool latency histograms
• Error distribution by type
• Health check history
• Consecutive failure tracking

Alerts Dashboard

File: alerts.json URL: http://localhost:3000/d/mcp-hangar-alerts

Alert monitoring and trends:

• Active alerts by severity
• Alert condition trends (error rate, latency, health)
• Historical alert timeline

Governance Dashboard

File: governance.json URL: http://localhost:3000/d/mcp-hangar-governance

MCP Hangar 1.4.0 adds a governance dashboard for tenant and policy operations:

• Cost attribution by MCP server, tool, and cost model
• Capability violations, tool schema drifts, detection rule matches, and enforcement actions
• Tool access denials, filtered tools, and active tool-access policies
• Batch in-flight calls, concurrency queueing, P95 wait time, and circuit-breaker state

Importing Dashboards Manually

If you are not running the Grafana sidecar:

1. Download the JSON from mcp-hangar/files/dashboards/ in the helm-charts repo
2. Open Grafana at http://localhost:3000
3. Go to Dashboards > Import
4. Upload the JSON file
5. Select Prometheus data source
6. Click Import

Alerting

Alert Configuration

The 30 maintained alert rules ship with the Helm chart. prometheusRule.enabled=true renders them as a PrometheusRule CR, which needs the Prometheus Operator CRDs installed; the source is mcp-hangar/files/prometheus-alerts.yaml. They are organized by severity:

Critical Alerts (Page On-Call)

Warning Alerts (Investigate)

Governance and Availability Alert Groups

1.4.0 adds two dedicated Prometheus groups:

• mcp-hangar-governance -- security, policy/enforcement, and concurrency saturation signals. (Cost is tracked on the governance dashboard but is not alerted.)
• mcp-hangar-availability -- MCP server state, discovery health, remote transport errors, and runtime rate limiting.

Use these groups when routing alerts to different teams; for example, security teams can subscribe to governance alerts while platform on-call owns availability and transport alerts.

Info Alerts (Tracking)

Alertmanager Configuration

Hangar ships no Alertmanager configuration — routing is yours. The rules above carry severity labels, so a routing tree keyed on them works out of the box:

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://your-webhook-endpoint'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<your-service-key>'

  - name: 'slack'
    slack_configs:
      - api_url: '<your-slack-webhook-url>'
        channel: '#mcp-hangar-alerts'
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

Testing Alerts

Verify alert rules are loaded:

# Check Prometheus rules
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[].name'

# Check for firing alerts
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'

Tracing

OpenTelemetry Integration

MCP Hangar supports distributed tracing via OpenTelemetry. Every tool invocation produces an OTEL span carrying MCP governance attributes (mcp.server.id, gen_ai.tool.name, mcp.tool.status, enforcement context, and identity context when available).

For the full MCP attribute taxonomy, partner backend recipes (OTEL Collector, OpenLIT, Langfuse, Grafana), and reference docker-compose setups, see: OpenTelemetry Integrations.

from mcp_hangar.observability import init_tracing, trace_span

# Initialize once at startup
init_tracing(
    service_name="mcp-hangar",
    otlp_endpoint="http://localhost:4317",
)

# Create spans for operations
with trace_span("process_request", {"request.id": req_id}) as span:
    span.add_event("checkpoint_reached")
    result = do_work()

MCP Governance Attributes on Spans

TracedMcpServerService automatically creates an OTEL span for each tool invocation with standard MCP governance attributes via set_governance_attributes():

from mcp_hangar.observability.conventions import McpServer, MCP, set_governance_attributes

# set_governance_attributes(span, ...) sets all applicable attributes in one call.
# None values are omitted -- no empty strings pollute OTLP backends.
set_governance_attributes(
    span,
    mcp_server_id="math",
    tool_name="add",
    user_id="alice",           # optional
    session_id="sess-42",      # optional
    policy_result="allow",     # optional
    enforcement_action=None,   # omitted from span
)

OTLP Audit Export

Security-relevant domain events (tool invocations, MCP server state transitions) are automatically exported as OTLP log records when OTEL_EXPORTER_OTLP_ENDPOINT is set. This is handled by OTLPAuditExporter and OTLPAuditEventHandler -- no additional configuration needed.

Events exported:

• ToolInvocationCompleted / ToolInvocationFailed -- with MCP server, tool, status, duration, caller identity, cost attribution
• McpServerStateChanged -- with MCP server, from_state, to_state

Caller identity attributes (mcp.caller.type, mcp.caller.id, mcp.caller.roles) are automatically propagated from the event's identity_context when available.

Cost attributes (mcp.cost.cents, mcp.cost.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) are included when cost attribution is configured.

Compliance Export Formats

MCP Hangar can export audit events in SIEM-compatible formats alongside OTLP. Available exporters (in src/mcp_hangar/compliance/):

All exporters implement the IAuditExporter protocol and output to file, callback, or stderr (for container log collection). Configure via the compliance bootstrap.

Environment Variables

Trace Context Propagation

W3C TraceContext is automatically propagated across agent -> Hangar -> MCP server boundaries:

• Inbound: BatchExecutor extracts traceparent from call metadata, creating child spans linked to the agent's root trace.
• Outbound: HttpClient injects traceparent into outbound HTTP headers when calling remote MCP servers.
• Stdio: Not supported (JSON-RPC over stdin/stdout has no header mechanism).

Manual propagation is also available:

from mcp_hangar.observability import inject_trace_context, extract_trace_context

# Inject into outgoing requests
headers = {}
inject_trace_context(headers)

# Extract from incoming requests
context = extract_trace_context(request_headers)

Langfuse Integration

MCP Hangar integrates with Langfuse for LLM-specific observability.

Configuration

export MCP_LANGFUSE_ENABLED=true
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=https://cloud.langfuse.com

Or via config.yaml:

observability:
  langfuse:
    enabled: true
    public_key: ${LANGFUSE_PUBLIC_KEY}
    secret_key: ${LANGFUSE_SECRET_KEY}
    host: https://cloud.langfuse.com
    sample_rate: 1.0

Trace Propagation

from mcp_hangar.application.services import TracedMcpServerService

result = traced_service.invoke_tool(
    mcp_server_id="math",
    tool_name="add",
    arguments={"a": 1, "b": 2},
    trace_id="your-langfuse-trace-id",
    user_id="user-123",
    session_id="session-456",
)

See ADR-007 for architectural details.

Logging

Structured Logging

MCP Hangar uses structlog for structured JSON logging:

{
  "timestamp": "2026-02-03T10:30:00.123Z",
  "level": "info",
  "event": "tool_invoked",
  "mcp_server": "math",
  "tool": "add",
  "duration_ms": 150,
  "service": "mcp-hangar"
}

Configuration

logging:
  level: INFO          # DEBUG, INFO, WARNING, ERROR
  json_format: true    # JSON output for log aggregation

Environment variable:

MCP_LOG_LEVEL=DEBUG mcp-hangar serve --http

Log Correlation

Include trace IDs for correlation with distributed traces:

from mcp_hangar.observability import get_current_trace_id
from mcp_hangar.logging_config import get_logger

logger = get_logger(__name__)
logger.info("processing", trace_id=get_current_trace_id())

Health Checks

HTTP Endpoints

Response Format

{
  "status": "healthy",
  "checks": [
    {
      "name": "mcp_servers",
      "status": "healthy",
      "duration_ms": 1.2
    }
  ],
  "version": "0.6.3",
  "uptime_seconds": 3600.5
}

Kubernetes Configuration

livenessProbe:
  httpGet:
    path: /health/live
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8000
  initialDelaySeconds: 10
  periodSeconds: 5

SLIs/SLOs

Service Level Indicators

Recommended SLOs

PromQL Queries

# Availability (service up ratio over 30d)
avg_over_time(up{job="mcp-hangar"}[30d])

# Error budget remaining
1 - (
  sum(increase(mcp_hangar_tool_call_errors_total[30d]))
  / sum(increase(mcp_hangar_tool_calls_total[30d]))
) / 0.01

# P95 latency
histogram_quantile(0.95,
  sum(rate(mcp_hangar_tool_call_duration_seconds_bucket[5m])) by (le)
)

# Batch success rate
sum(rate(mcp_hangar_batch_calls_total{result="success"}[5m]))
/ sum(rate(mcp_hangar_batch_calls_total[5m]))

Troubleshooting

Metrics Not Visible

1. Verify endpoint:

   curl http://localhost:8000/metrics | head -20

2. Check Prometheus targets at http://localhost:9090/targets

3. Verify network connectivity (use host.docker.internal for Docker on Mac/Windows)

Alerts Not Firing

1. Check alert rules loaded:

   curl http://localhost:9090/api/v1/rules | jq '.data.groups[].name'

2. Verify metrics exist for alert expressions

3. Check Alertmanager connectivity:

   curl http://localhost:9093/api/v1/status

High Consecutive Failures

If MCPHangarHighConsecutiveFailures fires:

1. Check MCP server logs for errors

2. Verify MCP server command/configuration

3. Restart the MCP server by restarting Hangar or invoking the MCP server (the first tool call triggers a cold start):

   mcp-hangar status

MCP Server Start Errors

Common patterns and fixes:

Best Practices

Metrics

1. Monitor the right things - Focus on user-facing SLIs
2. Set appropriate retention - 15 days for metrics, 7 days for traces
3. Avoid high cardinality - Don't use unbounded values as labels

Alerting

1. Create runbooks - Document response procedures
2. Start conservative - Tune thresholds based on baseline
3. Test regularly - Verify notification channels work
4. Use severity correctly - Critical = page, Warning = ticket

Dashboards

1. Layer information - Overview -> Details -> Debug
2. Include time selectors - Allow drilling into incidents
3. Add annotations - Mark deployments and incidents

Production Readiness Checklist

• Prometheus scraping MCP Hangar metrics
• Grafana dashboards imported and working
• Alertmanager configured with notification routes
• Critical alerts tested (e.g., stop service, verify page)
• Runbooks created for each alert
• Log aggregation configured (ELK, Loki, etc.)
• Tracing enabled and traces visible in Jaeger/Langfuse