ADR-029: One Enrichment Boundary and a Bounded Decision Vocabulary for Traces

Status: Accepted Date: 2026-09-19 Authors: MCP Hangar Team

Context

The analysis below records the historical 2.21.0 baseline that motivated this contract. The enrichment and bounded decision vocabulary subsequently shipped across 2.22.0 and 2.24.0; the old missing-path descriptions are not claims about today's serving paths. Acceptance of this contract is separate from completion of each instrumentation task, including peer-event causality (mcp-hangar/mcp-hangar#1288).

Hangar's traces became structurally correct in 2.19 (#1308–#1331): one request is one trace entered through the mcp SDK's SERVER span, batch.call.<tool> nests under batch.execute, the upstream CLIENT span carries its own carriers and outcome, the concurrency wait is measured on its own, caught failures end in ERROR with a bounded error.type, and export is proven end to end against a real OTLP receiver (#1293).

What traces still cannot do is explain decisions: why a gate refused, which backend was chosen, what an L7 policy said, or how background work relates to the call that caused it. Every task that would add this (#1278, #1285, #1286, #1295, #1297, #1279, #1281, mcp-hangar/mcp-hangar#1288, #1296) needs the same answers first. Without one contract, each would invent its own.

The state of the published 2.21.0 wheel, re-checked on main, that forces the decision:

• The request-entry span is not Hangar's, and it is not always a trace root. The mcp SDK's OpenTelemetryMiddleware is on by default and never removed. It opens one SERVER span per inbound message, named {method} {target}, with context=extract_trace_context(ctx.meta) — so a caller that sends a valid W3C traceparent in _meta makes that span the child of a remote span, and it is a trace root only when no valid remote parent is extracted. It sets mcp.method.name, mcp.protocol.version, jsonrpc.request.id, gen_ai.operation.name=execute_tool and gen_ai.tool.name, and declares no semconv schema.
• record_exception=False on that span is narrower than it sounds. It disables the context manager's automatic recording only. The middleware still writes status descriptions by hand: set_status(ERROR, e.error.message) for an MCPError, and record_exception(e) plus set_status(ERROR, str(e)) for anything else — a ValidationError is the single sanitised case. That span is not opened through Hangar's _TextFreeTracer, so Hangar's text-free discipline does not reach it.
• The gate chain is thirteen stages. _GATES runs cancelled_before_execution, global_timeout, resolve_target, tool_access, withdrawal, digest_pin, circuit_breaker, validators, tenant_budget, approval, cold_start, deferred_digest_pin and cancelled_after_cold_start. rate_limit.check is not one of them: it is command-bus middleware.
• No semantic-convention pin exists. There is no SEMCONV_VERSION in src/; this ADR establishes one.
• Governance enrichment is dead-wired. set_governance_attributes has one caller, TracedMcpServerService, which nothing in src/ constructs, so the decorator ADR-007 prescribes traces nothing on the real call paths.
• A refusal looks like a failure, on two different paths. A refusing gate in the executor's chain returns a CallResult with success=False, and executor.py:1556-1561 marks the span ERROR for any such result — the same status an upstream failure gets. Two classes of refusal never take that path: an L7 deny or approval requirement raises (EgressPolicyDeniedError, EgressPolicyApprovalRequiredError), and a rate-limit refusal raises RateLimitExceeded in command-bus middleware. _TextFreeTracer.start_as_current_span marks a span ERROR on any escaping exception, so one L7 deny also ends rate_limit.check, handler.InvokeToolCommand, command.send.InvokeToolCommand and invoke_with_retry in ERROR. Both paths contradict the #1272 rule that only operational failures are ERROR. A refused flat call has a third shape: _flat_call_tool returns CallToolResult(isError=True), which the SDK middleware matches to set its SERVER span ERROR with error.type=tool_error.
• mcp.server.id changes meaning within one trace. On the live tool-call path: logical target on batch.call.<tool>, concurrency.acquire, policy.check_access, approval_gate.check, approval_gate.flow and invoke_with_retry; selected member on mcp_server.cold_start and command.send.InvokeToolCommand. Of the spans on the executor's call path, only that member-valued pair has to change. Two lifecycle spans that landed after this ADR was first written (#1547, #1571) also name the member inside a group call's trace: mcp_server.startup_wait (infrastructure/observability/startup_spans.py:170-171) and mcp_server.launch (infrastructure/launchers/base.py:38-40). mcp_server.startup_wait is opened by the startup observer for a caller waiting on a start, through single flight or the aggregate's readiness wait (startup_spans.py:99-103, :130-140), and mcp_server.launch by the launcher. Both act on one concrete server process keyed by its own id and shared across calls (server/tools/batch/executor.py:2360-2367). Line references in the route and lifecycle passages of this ADR are to src/mcp_hangar/ on mcp-hangar main at c607e838.
• No causal envelope exists. Task events carry correlation_id and tenant_id, peer events carry produced_by, and none carries an origin trace context.
• Revision provenance exists only for L7, where policy_id is a content hash. No configuration revision exists anywhere in src/.
• The domain is already SDK-free. domain/ imports no OpenTelemetry or tracing module.

Decision

1. Signal ownership

2. Parent versus link

• Parent only when the child's work runs synchronously on behalf of the parent, within the same request lifetime.
• Link when: many waiters share one cause (single-flight cold start, #1279); the work outlives the request (governed tasks, #1281); the cause is on another instance (tailed peer events, mcp-hangar/mcp-hangar#1288); the cause is a timer or scheduler (#1296); an inbound carrier names a different trace than the ambient span (already implemented, #1316).
• Never: correlation_id as a parent; a span held open across a multi-hour task; an invented parent or link when the origin is unknown or malformed.

3. The SDK SERVER span is the request-entry span, not necessarily the trace root

Hangar never opens a parallel request span. The SDK's SERVER span is where a request enters Hangar and is the topmost span of that request inside it; it is a trace root only when no valid remote parent was extracted from _meta. With a valid inbound carrier the trace continues from the caller's span, and Hangar's spans are descendants of it.

hangar_call is the SERVER span's child, batch.execute is hangar_call's child, batch.call.<tool> is batch.execute's child. Flat tools enter as SERVER → batch.execute → batch.call.<tool>. execute_tool <tool> (CLIENT) is the only upstream call span. Names and existing attributes are kept; the one exception is named in Decision 5.

4. One governance enrichment boundary

• BatchExecutor._execute_call's batch.call.<tool> span is the single enrichment point for MCP tool calls; batch and flat-tool invocations both pass through it. Caller and tenant attributes come only from the bound identity context, never from baggage, and unknown values are omitted. The per-gate outcome summary lives here.
• Management and REST commands that do not reach the executor get the dispatch span of #1297, with the same outcome vocabulary.
• TracedMcpServerService is retired (#1278 deletes it). Reviving it would rebuild a parallel pipeline on a path nothing constructs.
• Vendor-neutral export replaces the removed Langfuse adapter. The process-local enrichment boundary is not a vendor SDK adapter. Core mcp-hangar/mcp-hangar#1734 removed that adapter and its settings in 2.25.0; ADR-007 is Deprecated. Langfuse consumes the existing OTLP trace export like another receiver. Every emitted value remains subordinate to the #1276 data-handling contract. This record neither revives the adapter nor reintroduces its scrub settings.

5. A bounded decision vocabulary under hangar.

Four vocabularies, one per layer that can stop or steer a call, because those layers have different owners and lifecycles. A term never crosses layers.

A gate is a stage of _GATES, and nothing else. rate_limit.check is command-bus middleware and belongs to hangar.dispatch.*; an L7 verdict is raised from the aggregate and belongs to hangar.l7.*. Naming them gates would make the word mean "anything that can stop a call", which is not a set the code has.

Gate decisions are span events, not scalar attributes

One call passes many gates. A scalar attribute holds one value per key, so the last gate to write would erase every decision before it. Each gate that runs, or is explicitly skipped, records one bounded event on batch.call.<tool>:

A deferred digest pin therefore records {name=digest_pin, outcome=deferred} and, on the same span, later records {name=deferred_digest_pin, outcome=allow|deny|error}. Neither overwrites the other.

The call's summary is separate and scalar. It never carries a history.

Two route cases are settled here rather than by a new value. A tenant pinned to a member that is out of rotation reads hangar.route.reason=canary_fallback: pins are configured under the group's canary: block (server/config.py:1331-1353), resolved by the same CanaryPolicy as the canary split (domain/model/mcp_server_group.py:41-50), and fall back to the load balancer on the same path (domain/model/mcp_server_group.py:590-604), so there is no separate pin-fallback value. A standalone call carries hangar.route.backend equal to mcp.server.id, so the key is present whenever a backend was chosen and a query by backend needs no fallback to mcp.server.id.

Span status

Rules:

• This binds Hangar-owned spans only. ERROR-for-operational-failures governs the spans Hangar opens through _TextFreeTracer. The SDK-owned SERVER span follows MCP SDK response semantics and stays ERROR for a CallToolResult(isError=true) refusal, so a governance denial on the flat path produces a trace whose entry span is ERROR above a batch.call.<tool> that is UNSET. That deviation is accepted and recorded here rather than left to be discovered. Overriding it would mean writing to a span Hangar does not own, and OpenTelemetry does not let a status return to UNSET — the only available override is OK, which asserts more about an SDK-owned response than Hangar is entitled to claim.
• A refusal carried by an exception leaves no ERROR behind it. L7 denials, approval requirements and command-bus rate limits are raised, and _TextFreeTracer marks every span they escape as ERROR. Those spans — rate_limit.check, handler.*, command.send.*, invoke_with_retry — are UNSET for a refusal. The tracer therefore has to tell a refusal exception from a failure instead of treating every escape alike; that is part of #1285 and #1295, not an implementation detail left to each.
• The first refusing gate is identifiable. No downstream gate, cold-start or invoke span appears after it, and it is what hangar.refusal.gate names.
• No invented reasons. An engine returning only a boolean gets an extended port result. If it cannot say why, hangar.gate.reason is omitted.
• mcp.server.id means the logical target on every span on the executor's call path: batch.call.<tool>, policy.check_access, approval_gate.check, approval_gate.flow, concurrency.acquire, invoke_with_retry, mcp_server.cold_start and command.send.InvokeToolCommand. One attribute that means two things within a call cannot be queried; the concrete member is hangar.route.backend. The logical target is what the caller named and the scope the call is governed under (the selected member's own policy is added to it, server/tools/batch/executor.py:1956-1961), and audit is to be keyed on it too (#1594), so it is the identity that has to be constant across the call. The member is a routing outcome, a separate fact: calls to one group spread across its members by load balancing, canary split and failover, and a group call refused with no_available_member has no member at all. The executor selects the member once per call, in _gate_resolve_target (server/tools/batch/executor.py:1939-1945), and fixes the dispatch target before retry_sync runs (executor.py:2517, :2559), so every retry attempt of one call reaches the same member. A failure reported against that member can take it out of rotation for the next call's selection (executor.py:2202-2203, domain/model/mcp_server_group.py:733, :583), not for a retry of the current one. This is a compatibility change, not an additive one: mcp_server.cold_start and command.send.InvokeToolCommand carry the member today, and queries reading them as the concrete backend must move to hangar.route.backend. batch.call.<tool> carries both hangar.route.backend and hangar.route.reason; mcp_server.cold_start and each command.send.InvokeToolCommand attempt carry hangar.route.backend only; a no_available_member refusal carries the reason and no backend. #1286 owns it and ships it with a changelog and migration note, and that note also states that the lifecycle spans below keep naming the member.
• Lifecycle spans name the server they act on. mcp_server.launch and mcp_server.startup_wait carry the concrete server in mcp.server.id, which for a group call is the selected member. The launcher and the startup observer act on one concrete server process keyed by its own id and shared across calls (server/tools/batch/executor.py:2360-2367, infrastructure/launchers/base.py:38-40, infrastructure/observability/startup_spans.py:170-171), and a member can belong to several groups (fastmcp_server/flat_tool_projection.py:1150-1153), so one start can serve calls that named different logical targets and has no single one to name. The member of a group call stays discoverable through hangar.route.backend on batch.call.<tool>, mcp_server.cold_start and command.send.InvokeToolCommand.
• The log side widens to match the gate vocabulary. _log_call_failure keys refused on two error types only (EgressPolicyDeniedError, EgressPolicyApprovalRequiredError), so a gate denial would read hangar.gate.outcome=deny on the span and batch_call_failed in the log. Any expected refusal logs as batch_call_refused at warning with its bounded reason, so one denial is one word in both places. This raises warning volume above today's L7-only behaviour, and that is accepted.

6. Revision provenance

7. The domain stays SDK-free

A decision made inside an aggregate leaves through a narrow domain port with a no-op default, or a defaulted observational field on the existing Decision. An application or infrastructure adapter writes the span attributes. Existing domain events such as EgressPolicyViolationObserved are the preferred carriers. Telemetry never runs a second policy evaluation; #1322 is the precedent.

8. Causal envelope

• Stored as an optional bounded value: the W3C traceparent string of the originating span, as data and never as an SDK object — the event metadata column (mcp-hangar/mcp-hangar#1288), the task ledger entry (#1281), the timer payload (#1296).
• Consumed by opening a new span with one link to that origin. Many waiters produce many links, never one shared parent.
• Missing or malformed origin: no link.
• Replication. Tailed delivery stays projection-only (ADR-020 §4). Replicating trace context never replays effects.

9. Semantic conventions

• Hangar pins the MCP and GenAI attribute definitions that mcp==2.0.0's middleware emits, introduced in OpenTelemetry semantic conventions v1.39.0 and maintained since v1.42.0 in semantic-conventions-genai. The pin is recorded as SEMCONV_VERSION in observability/conventions.py, which does not exist yet. The compatibility baseline is reviewed on every mcp dependency bump, and the pin may also be advanced independently when Hangar deliberately adopts a newer semantic-convention version; either way the change reviews the GenAI/MCP changelog for attribute and span-semantic changes.
• Documented deviations, not renamed: batch.call.<tool>, mcp.server.id, deployment.environment, audit's mcp.error.type.
• Additive, with one named exception. Every hangar.* key above is new surface added beside what exists. The one change to an existing attribute is mcp.server.id on mcp_server.cold_start and command.send.InvokeToolCommand, owned by #1286 and shipped with a migration note. Live code uses the conventions.py constants instead of string literals; unused constants are deleted or marked reserved.

10. Privacy and sampling

Every attribute, reason code and link above passes the data-handling allowlist decided in #1276 before it is emitted: caller user and agent ids on spans are opt-in only, span attributes are limited to 256 characters, and status descriptions and exception events are bounded by that same contract. Sampling never decides audit durability.

11. What the contract produces on the live paths

What the rules above produce on the paths that exist today. Each row is the contract #1278, #1285, #1286, #1295 and #1297 implement, not a separate decision any of them may take.

Consequences

Positive

One boundary, one vocabulary and one parent/link rule unblock #1278, #1285, #1286, #1295, #1297 and #1279, #1281, mcp-hangar/mcp-hangar#1288, #1296 — none has to invent its own. Denials stop inflating error rates on the spans Hangar owns, and the first refusing gate becomes queryable by name. Recording each gate as a span event means a call's full decision history survives instead of the last gate overwriting the rest. Five implementers can take #1278, #1285, #1286, #1295 and #1297 and produce compatible telemetry without another semantic decision. The dead path ADR-007 pointed at is retired rather than revived.

Negative

The status of refused spans changes across the Hangar-owned tree, not only on batch.call.<tool>; dashboards and alerts keyed on ERROR must be reviewed, including the Helm-owned ones in mcp-hangar/helm-charts. mcp.server.id changes value on two spans — a compatibility change, not an additive one, and queries reading it there as the concrete backend must move to hangar.route.backend. It is constant across the spans on the executor's call path, not across the whole trace: mcp_server.launch and mcp_server.startup_wait still name the member inside a group call's trace, so a query by mcp.server.id over every span of a trace sees two values unless it is scoped to the call's spans or uses hangar.route.backend. A trace whose entry span is ERROR above an UNSET batch.call.<tool> is now an expected shape on the flat path, and anyone reading the entry span alone will still count governance denials as errors. Telling a refusal exception from a failure inside the tracer is new work that #1285 and #1295 both depend on. hangar.* is new surface to keep stable, and revision stays absent for most decision types.

Neutral

Hangar depends on the SDK middleware staying on by default, and nothing asserts that today. The integration trace tests observe a tools/call hangar_call span, which depends on the middleware only incidentally; a contract test that fails by naming the middleware still has to be written.

The #1276 data-handling contract reaches Hangar-owned spans through _TextFreeTracer. The SDK-owned SERVER span does not go through it: record_exception=False disables only the context manager's automatic recording, while the middleware still sets status descriptions by hand and calls record_exception for a generic exception. So #1276's guarantees apply directly to Hangar's spans, and to the SERVER span only if it is separately adapted or verified. This ADR does not assume it is.

Alternatives Considered

1. Revive TracedMcpServerService and wire it in — rejected: it decorates a service no path constructs, so reviving it rebuilds a parallel pipeline.
2. A Hangar-owned request span beside the SDK's — rejected: two roots per request, and the SDK span already parents from _meta.
3. Reuse the existing mcp.* and policy.* keys — rejected: needs renames or overloads existing semantics; the hangar.* additions keep old queries working.
4. Parent tasks and peer events on the originating span — rejected: long-lived or cross-process parents, and one parent for many waiters. Links are the OpenTelemetry-correct form.
5. Keep ERROR for refusals, or phase it over two releases — rejected: keeping ERROR leaves denials inflating error rates; a two-release transition doubles the work for a change a changelog note covers.

References

• mcp-hangar/mcp-hangar#1275 (this decision's issue), #1276 (the data-handling contract), #1264 (the tracing epic), #1286 (route telemetry, where the scope of the logical-target rule and the one selection per call were settled).
• ADR-007, ADR-013, ADR-014, ADR-016, ADR-020.
• OpenTelemetry semantic conventions v1.39.0 (open-telemetry/semantic-conventions#2043, #2083); open-telemetry/semantic-conventions-genai.