OpenTelemetry Integrations

Hangar is the runtime governance layer for MCP servers. It is not an observability platform. Hangar exports governance telemetry -- enforcement decisions, MCP server lifecycle events, capability violations, and identity-aware audit trails -- through the OpenTelemetry (OTEL) interoperability contract. Partner backends visualize it.

Agent --> Hangar (governance) --> OTLP --> [OTEL Collector | OpenLIT | Langfuse | Grafana]

This page covers how to connect Hangar to each supported backend and what governance data flows through.

MCP Attribute Taxonomy

Hangar's spans and audit records carry MCP-specific attributes defined in src/mcp_hangar/observability/conventions.py. These attributes form a stable contract that partner backends consume without Hangar-specific plugins.

Not every key in the tables below is emitted. In 2.24.0 nothing sets mcp.server.group_id, mcp.server.image, mcp.server.has_capabilities, mcp.server.enforcement_mode, mcp.tool.cold_start, mcp.tool.args_hash, mcp.tool.response_tokens, mcp.cost.currency, mcp.health.result, mcp.health.duration_ms, or any mcp.enforcement.*, mcp.audit.*, mcp.risk.* or mcp.behavioral.* key: they are reserved names. mcp.server.state, mcp.tool.status, mcp.tool.duration_ms, mcp.caller.roles, mcp.cost.* and gen_ai.usage.* appear on audit records only. Refusals are described by hangar.call.outcome, hangar.refusal.* and hangar.l7.* instead: see the tracing diagnosis runbook.

MCP Server attributes

The hangar.route.* attributes above and the hangar.l7.* attributes under Enforcement attributes were added in 2.24.0. In 2.23.0 and earlier, mcp_server.cold_start and command.send.InvokeToolCommand carried the selected group member in mcp.server.id; since 2.24.0 they carry the group, and the member is hangar.route.backend. The lifecycle spans mcp_server.launch and mcp_server.startup_wait still name the member they start. The tracing diagnosis runbook shows how to read both.

Tool invocation attributes

Enforcement attributes

Audit attributes

Behavioral attributes

Roadmap / planned — not emitted in the current release. These attributes are reserved for the behavioral/anomaly-detection path, which is held back. Deterministic enforcement does not populate them today; the table documents the reserved contract for a future release.

Caller attributes (identity propagation)

The identifiers in this taxonomy (mcp.caller.id, mcp.user.id, mcp.agent.id, mcp.session.id) are off spans by default and need the opt-in described under Effective tracing configuration. Audit records carry the caller regardless.

Cost attributes (FinOps)

Risk attributes (semantic analysis)

Attributes in the mcp.risk.* namespace carry detection rule match signals from the semantic analysis engine. The engine evaluates multi-step tool call sequences per agent session against detection rules (e.g. credential exfiltration, privilege escalation). Partner backends such as OpenLIT, Grafana, and SIEM tools can filter spans by mcp.risk.severity = critical to surface high-risk events.

CapabilityViolationDetected signals are emitted today (enforcement is deterministic — rule- and threshold-based). DetectionRuleMatched is not: the semantic analysis engine that would produce it is not shipped yet. The mcp.risk.score and mcp.risk.session_anomaly_score attributes are reserved for the aggregate behavioral anomaly-scoring path, which is not yet enabled: these session-level scores are not populated in the current release.

Health attributes

Prometheus metric names

OTEL Collector

The OTEL Collector is the recommended entry point for governance telemetry. It receives OTLP from Hangar and routes spans, metrics, and logs to any supported backend -- Prometheus, Jaeger, OpenLIT, Datadog, or a custom pipeline.

Example: examples/otel-collector/

Getting started

Set these environment variables before starting Hangar:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_SERVICE_NAME=mcp-hangar
MCP_TRACING_ENABLED=true

Start the reference stack:

cd examples/otel-collector
docker-compose up

This starts:

The collector config (otel-collector-config.yaml) receives OTLP on both gRPC and HTTP. It writes spans and audit records to stdout (the debug exporter) and to a JSON-lines file (the file exporter). Hangar exports no OTLP metrics, and Prometheus scrapes its /metrics endpoint directly. Replace the debug exporter with your production backend.

What flows through the collector

• Traces: one trace per request. The governance span batch.call.<tool> carries mcp.server.id, gen_ai.tool.name, the caller attributes the bound identity has, one hangar.gate.decision event per gate, and hangar.call.outcome. Since 2.24.0, it also carries the route (hangar.route.*) and any L7 verdict (hangar.l7.*). The upstream call is the CLIENT span execute_tool <tool>. The tracing diagnosis runbook shows the full span tree.
• Logs: audit records under scope mcp_hangar.audit, for tool invocations (mcp.tool.status, mcp.tool.duration_ms), refused calls included since 2.24.0, and MCP server state transitions.
• Metrics: not sent over OTLP. Prometheus scrapes Hangar's /metrics endpoint.

Effective tracing configuration

Hangar builds its tracer provider itself, so it resolves the standard variables in its own code. For each setting, the first source that is set wins:

Current limitations:

• Hangar does not read OTEL_TRACES_EXPORTER or OTEL_PROPAGATORS. It adds the OTLP exporter itself, and it propagates W3C traceparent and tracestate only, never baggage.
• Audit records are exported only when an endpoint is set explicitly, in OTEL_EXPORTER_OTLP_ENDPOINT or observability.tracing.otlp_endpoint. The signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_LOGS_ENDPOINT do not turn audit export on by themselves. MCP_AUDIT_EXPORT_ENABLED=false turns it off.
• No semantic-convention schema version is declared or pinned. The request-entry SERVER span, with mcp.method.name, gen_ai.operation.name and gen_ai.tool.name, is emitted by the pinned MCP SDK (mcp==2.0.0), not by Hangar.

Supported versions. The mcp-hangar[opentelemetry] extra requires opentelemetry-api, opentelemetry-sdk and opentelemetry-exporter-otlp at 1.35.0 or later, the lowest release that installs beside the core dependencies. The lockfile pins 1.45.0; CI installs with pip, unpinned, so it tests the newest release. The container image installs these packages unpinned when it is built, so check the installed versions on a live install.

Without a collector. Tracing is on by default and exports to localhost:4317. With nothing listening, export fails in the background and never blocks a call, but every failed batch is logged and counted. Set MCP_TRACING_ENABLED=false to turn Hangar's tracing off. To read spans locally instead, set MCP_TRACING_CONSOLE=true, which adds an exporter that prints to stderr, and set OTEL_EXPORTER_OTLP_ENDPOINT to an empty value so that no OTLP exporter is added. Stderr is used because stdout carries the protocol on the stdio transport.

Tracer provider ownership

Hangar registers its own tracer provider only if no other provider was registered first. If a host application or an instrumentation agent registered one, including through OTEL_PYTHON_TRACER_PROVIDER, Hangar uses it and builds nothing. It logs tracing_external_provider_in_use, and never replaces, flushes or shuts that provider down. Hangar's sampler, length limits, resource, OTLP exporter and mcp_hangar_otlp_export_failures_total counter then do not apply: the owner's configuration does. MCP_TRACING_ENABLED=false still keeps Hangar's spans out of a provider that someone else registered. Audit records follow the same rule for the logger provider (audit_log_external_provider_in_use).

Correlating logs and audit records with traces

• Structured logs. A line written inside a span carries trace_id and span_id. batch_call_refused is one such line, so a refusal found in the log leads to its trace.
• OTLP audit records. A record emitted inside a span carries that span's trace and span IDs, even when the span is not sampled. A record emitted outside any span, such as some server state changes, carries none. The core live test checks that the tool_invocation record arrives, but not that its trace ID matches the call's trace. Treat the audit-to-trace join as best-effort, not as guaranteed navigation.
• Both signals share the resource attributes service.name and service.instance.id, so a replica's records and spans can be matched. One exception: after a restart, events the previous process stored but did not deliver are delivered again (hangar.event.delivery_mode=recovered), and their audit records carry the new process's service.instance.id.

OpenLIT

OpenLIT provides a trace explorer, session analytics, and cost attribution UI. Hangar exports governance telemetry; OpenLIT visualizes it. They connect through the OTEL Collector -- Hangar does not send data directly to OpenLIT.

Example: examples/openlit/

Getting started

cd examples/openlit
docker-compose up

This starts Hangar, an OTEL Collector, and OpenLIT. Open the OpenLIT dashboard at http://localhost:3000.

Environment variables for Hangar (already set in the docker-compose):

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_SERVICE_NAME=mcp-hangar
MCP_TRACING_ENABLED=true

What governance data is visible in OpenLIT

In the OpenLIT trace explorer, filter on MCP governance attributes:

• By MCP server: mcp.server.id = "math-server"
• By tool: gen_ai.tool.name = "add"
• By user: mcp.user.id = "alice" (only with MCP_TRACING_CALLER_IDS=true)
• By refusal: hangar.call.outcome = "deny", and the refusing gate in hangar.refusal.gate
• By L7 verdict: hangar.l7.verdict = "deny"

MCP Server lifecycle events (cold, initializing, ready, degraded, dead) appear as audit log records with mcp.server.state attributes.

Langfuse

Langfuse provides LLM-specific observability: input/output recording, token counting, user session tracking, and evaluation workflows. It accepts OpenTelemetry traces on an OTLP/HTTP endpoint, so Hangar sends it the same spans any OTLP backend receives -- no Langfuse package and no Langfuse-specific setting.

Hangar shipped a Langfuse adapter until 2.24.0. Nothing called it after 2.22.0, and 2.25.0 removes it with observability.langfuse, MCP_LANGFUSE_*, HANGAR_LANGFUSE_* and the langfuse extra (mcp-hangar#1683). A scrub setting from it refuses the boot; the other settings are named in a warning. See the upgrade guide.

Example: examples/langfuse/

Getting started

You need a running Langfuse instance (v3.22.0 or later) -- either Langfuse Cloud or a self-hosted deployment -- and Hangar installed with the opentelemetry extra. Langfuse takes OTLP over HTTP only, so set the protocol as well as the endpoint; the credential is HTTP Basic auth over public_key:secret_key:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
AUTH_STRING=$(printf '%s' "${LANGFUSE_PUBLIC_KEY}:${LANGFUSE_SECRET_KEY}" | base64 | tr -d '\n')

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://cloud.langfuse.com/api/public/otel/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic%20${AUTH_STRING},x-langfuse-ingestion-version=4"

• Self-hosted: https://<your-host>/api/public/otel/v1/traces.
• Use the TRACES_ variables, not the generic OTEL_EXPORTER_OTLP_ENDPOINT: the generic one also turns on Hangar's OTLP audit log export, and Langfuse does not accept logs.
• To send to Langfuse and another backend, or to redact span attributes before they leave your network, run an OTEL Collector with Langfuse as an otlphttp exporter.

!!! warning "Secret handling" LANGFUSE_SECRET_KEY is a secret, and so is the header built from it. Use environment variables, HashiCorp Vault, or Kubernetes secrets. Never commit either to config files or source control.

What Langfuse receives

Hangar's spans, with the attributes described on this page: names, ids, outcomes and durations, and the trace context propagated from the caller. Tool arguments and results are not on spans, and caller identifiers are only when MCP_TRACING_CALLER_IDS (or observability.tracing.caller_ids) is on.

Grafana

Hangar exposes Prometheus metrics at the /metrics HTTP endpoint. Grafana can scrape these directly or consume them through the OTEL Collector's Prometheus exporter.

Four maintained Grafana dashboards ship with the mcp-hangar Helm chart, source in mcp-hangar/files/dashboards/:

• Overview dashboard: MCP server states, tool call rates, error rates
• Provider details dashboard: Per-MCP server metrics, health check history
• Alerts dashboard: Circuit breaker state, violation counts
• Governance dashboard: Cost attribution, policy violations, enforcement actions

Getting started

Prometheus and Grafana are yours to run — Hangar ships no stack. On Kubernetes, dashboards.enabled=true renders the four as ConfigMaps for the Grafana sidecar to auto-import; elsewhere, import the JSON by hand from the link above.

Start Hangar in HTTP mode so the /metrics endpoint is available. Without authentication, Hangar binds only to a loopback address:

mcp-hangar serve --http --host 127.0.0.1 --port 8000

Key Prometheus queries

# Tool call rate per mcp_server (last 5 minutes)
rate(mcp_hangar_tool_calls_total[5m])

# 95th percentile tool call latency
histogram_quantile(0.95, rate(mcp_hangar_tool_call_duration_seconds_bucket[5m]))

# MCP servers currently in DEGRADED state (3=degraded)
mcp_hangar_mcp_server_state == 3

# Groups whose circuit is open (one series per state; 1 marks the current one)
mcp_hangar_circuit_breaker_state{state="open"} == 1

Integration stance

Hangar is not trying to become an observability platform. OpenTelemetry-compatible tools -- OpenLIT, Langfuse, Grafana, OTEL Collector, Datadog, Honeycomb -- are extensions to the visibility layer around Hangar. Hangar defines the governance telemetry contract (the MCP attribute taxonomy above) and exports it via OTEL. Partner tools consume, correlate, alert, and visualize.

This separation means:

• Hangar focuses on runtime security enforcement, lifecycle management, and governance policy.
• Partner tools focus on visualization, alerting, session analytics, and cost attribution.
• The OTEL interoperability contract ensures any OTEL-compatible tool works without Hangar-specific plugins.