Upgrade Guide

This guide covers user-visible migration steps between MCP Hangar releases.

Upgrade to 2.27.5

init writes absolute paths; state follows a configured event_store.path

When event_store.path is absolute, the saga state file (saga_state.db) and the approvals database (mcp_hangar.db) now live in the same directory as the event store instead of in data/ under the working directory. Configurations without an absolute event_store.path are unchanged.

Entries written by an earlier mcp-hangar init use a bare mcp-hangar command and a relative data/ path, which fails when the client spawns Hangar with a minimal PATH or an unwritable working directory. Rerun mcp-hangar init to refresh them. If you move to the new location, copy any existing data/*.db files next to the new event_store.path first.

The MCP SDK is 2.3.0

mcp-hangar now requires mcp==2.3.0 and mcp-types==2.3.0; it was 2.0.0 for both. Both stay pinned exactly. Nothing in the configuration changes.

• A project that pins mcp itself, or installs mcp-hangar beside a package that does, needs mcp==2.3.0.
• The SDK's own defaults changed with it: a 4 MiB request-body limit, a 30-minute idle expiry and a cap of 10,000 stateful sessions. Hangar does not override them.
• Mcp-Param-* headers of projected tools are still checked against a call's arguments. 2.3.0 stopped checking them for any tool the SDK had not registered itself. Hangar answers for its own tools since 2.27.4, which this release includes.

Upgrade to 2.27.4

An access policy with a key nothing reads refuses the configuration

A policy block may carry only allow_list, deny_list, approval_list, approval_timeout_seconds and approval_channel (and access in a tool_access.member.<tenant> entry). Any other key now refuses startup and a reload, naming the path and the key. Before, such a block was read as no policy at all, so the tools it meant to restrict were served.

• If you wrote tools: {allow: [...], deny: [...]}, as the README did: you have had no restriction. Rename the keys to allow_list and deny_list, and check what the gateway served in the meantime.
• Run mcp-hangar config check <file> before upgrading: it reports the same keys and exits 1.
• A reload that hits this keeps the policy that was in force.

Upgrade to 2.27.0

Trusting private networks for servers registered over the API

Nothing changes unless you add the new section. POST /api/mcp_servers still refuses every private endpoint by default. To let the Kubernetes operator register a server it runs in the cluster, list the cluster's Service range in config.yaml:

ssrf:
  trusted_networks: ["10.96.0.0/12"]

• The list applies to every caller allowed to register a server, not only the operator. Keep it as narrow as the servers need.
• The Service range also holds the kube-apiserver's ClusterIP and every other Service in the cluster. Add the pod range only for headless Services.
• Loopback, link-local (including the cloud metadata endpoint) and the unspecified ranges are refused at startup and on reload, and so is a CIDR with host bits set.

See ssrf in the configuration reference.

Upgrade to 2.26.0

deleting a configured server over the API is refused

DELETE /api/mcp_servers/{id} on a server that the configuration file declares now answers 409 with "error": "declared_in_config". The server keeps running. Before, the delete stopped it and removed it from the pool, and the next reload or restart brought it back.

• To remove a configured server, delete it from the configuration and reload.
• To take it out of service for a while, stop it instead of deleting it.

Servers added with POST /api/mcp_servers, and servers that discovery registers, are deleted as before.

The Kubernetes operator sent this delete whenever an MCPServer CR was deleted, including a CR that mirrored a configured server. Operator versions that include mcp-hangar/mcp-hangar-operator#202 no longer do.

Upgrade to 2.25.0

An invalid access policy refuses the configuration

A tools: access policy (allow_list / deny_list / approval_list) on a server, a group, a group member or a tool_access.member tenant entry, and an access: block, is now refused when one of its fields is invalid: an approval_timeout_seconds that is not a positive integer, an empty or non-string pattern, a whitespace-only approval_channel, or a list given as a number.

Before, the gateway logged invalid_tools_access_config (or the group, member, tenant or access variant) at warning and booted with no policy for that scope. Now the boot fails, and a reload is refused with the previous policy kept in force, with a ConfigurationError naming the scope and the field, for example:

Invalid tools access policy for mcp_server 'calc': Invalid approval_timeout_seconds: 0

A configuration that booted with one of those warnings in its log will not boot on this release. Fix the field the error names. A policy you meant to have no effect is removed by deleting the block, not by leaving it invalid.

The actor of an auth mutation is the caller; *_by body fields are refused

The auth REST routes used to accept the actor of a change in the request body: created_by on POST /api/auth/keys and POST /api/auth/roles, revoked_by on DELETE /api/auth/keys/{key_id} and DELETE /api/auth/roles/revoke, assigned_by on POST /api/auth/roles/assign, and updated_by on PATCH /api/auth/roles/{role_name}. Each was optional and defaulted to "system", and whatever it said was recorded in the domain event, the log line and the response.

The actor is now always the principal that authenticated the request. A body that still carries one of those fields -- whatever its value, including null or the caller's own id -- is refused with 422 and a ValidationError naming the field, and nothing is changed. DELETE /api/auth/roles/{role_name} records the caller as deleted_by instead of "system", and POST /api/auth/keys returns created_by alongside the key.

Old:

{"principal_id": "user:bob", "role_name": "developer", "assigned_by": "user:alice"}

New -- drop the field; the assigner is whoever the key or token belongs to:

{"principal_id": "user:bob", "role_name": "developer"}

A script or client that sent these fields must stop sending them. With auth disabled no principal is attached to a request, and the actor is recorded as anonymous, never system; in practice the key and role routes have no store to write to with auth off, so this only names the value should one be wired.

A missing configuration file stops the gateway

There is no built-in demo configuration any more. Before, when the configuration file was not there, Hangar logged config_not_found_using_default at INFO and booted a math_subprocess backend with every hangar_* tool -- stop, load and reload_config included -- and none of your pins, policies or auth. That happened for a mistyped --config, an MCP_CONFIG naming a missing file, a config_path given to bootstrap() or Hangar.from_config(), and a start with no configuration file anywhere.

Now each of those refuses to start. mcp-hangar serve, a bare mcp-hangar and python -m mcp_hangar.server exit 1 with the path and where it came from on stderr, for example:

Error: Configuration file /etc/hangar/confg.yaml (named on the command line) does not exist. Nothing is served without the configuration that was asked for.

bootstrap() and Hangar.start() raise MissingConfigFileError (a ConfigurationError). A path that is a directory or is not readable is refused the same way.

With no flag, no MCP_CONFIG, no ./config.yaml and no ~/.config/mcp-hangar/config.yaml, the message says to run mcp-hangar init.

What to do: if a deployment logged config_not_found_using_default, it was running the demo configuration. Fix the path, or create the file with mcp-hangar init. For a gateway with no servers, use a file holding mcp_servers: {}. From Python, a configuration with no file goes in bootstrap(config_dict=...) or Hangar.from_builder(...).

Every command reads the configuration file by one rule

init, status, add, remove, pin, serve, a bare mcp-hangar, config check and python -m mcp_hangar.server now resolve the configuration file the same way, highest first:

1. a path on the command line -- the command's own flag or argument, then the global --config;
2. $MCP_CONFIG;
3. ./config.yaml, if the working directory has one;
4. ~/.config/mcp-hangar/config.yaml, the file init writes.

What changes for you:

• serve with no flag, no MCP_CONFIG and no ./config.yaml used to boot the built-in demo config. It now reads ~/.config/mcp-hangar/config.yaml if init wrote one, and refuses to start if there is none (see "A missing configuration file stops the gateway"). A ./config.yaml beside the process still wins, so a setup that ran serve next to its file is unchanged, and so is the container image (its working directory is /app).
• init, add, remove and status in a directory with a ./config.yaml used to write or report ~/.config/mcp-hangar/config.yaml. They now use ./config.yaml, the file serve there reads. init -y backs that file up and replaces it, and add and remove edit it in place -- so run them in a directory whose config.yaml is Hangar's, or pass --config ~/.config/mcp-hangar/config.yaml (or init --config-path) to keep the old target.
• status with a named file that does not exist used to report whichever default file it found next. It now reports that no configuration was found at the named path.
• MCP_CONFIG no longer outranks a global --config. MCP_CONFIG=a.yaml mcp-hangar --config b.yaml serve used to run a.yaml, because serve's own --config read the variable. It now runs b.yaml.
• pin and config check honour the global --config. mcp-hangar --config X pin --check used to exit 2 looking for ./config.yaml; it now checks X.
• Reload works on a gateway started without --config. The watcher, SIGHUP, the reload tool and POST /api/config/reload used to fail with "No configuration path specified"; they now reload the file the gateway booted from.

The Langfuse adapter and its settings are removed: send spans to Langfuse over OTLP

Nothing called the Langfuse adapter after 2.22.0, so these settings sent nothing to Langfuse. They are removed:

• observability.langfuse in config.yaml (enabled, public_key, secret_key, host, sample_rate, scrub_inputs, scrub_outputs)
• MCP_LANGFUSE_ENABLED, MCP_LANGFUSE_SAMPLE_RATE, MCP_LANGFUSE_SCRUB_INPUTS, MCP_LANGFUSE_SCRUB_OUTPUTS
• HANGAR_LANGFUSE_ENABLED, HANGAR_LANGFUSE_SAMPLE_RATE, HANGAR_LANGFUSE_SCRUB_INPUTS, HANGAR_LANGFUSE_SCRUB_OUTPUTS
• the langfuse extra (pip install mcp-hangar[langfuse])
• in Python, create_runtime(observability_config=...), Runtime.observability, Runtime.observability_config, mcp_hangar.bootstrap.runtime.ObservabilityConfig, the mcp_hangar.integrations package, ApplicationContext.observability_adapter; init_observability() now returns only the parsed config, and shutdown_observability() takes no argument

What happens to a configuration that still sets them:

• A scrub setting (MCP_LANGFUSE_SCRUB_*, HANGAR_LANGFUSE_SCRUB_*, or observability.langfuse.scrub_*), whatever its value, refuses the boot with a ConfigurationError naming it. Before, it was accepted and did nothing. Delete it. Hangar's spans carry no tool arguments or results; to redact what they do carry, put an OpenTelemetry Collector with an attributes or redaction processor in front of Langfuse.
• The other environment variables are named in a langfuse_settings_removed warning, and the boot continues.
• The rest of an observability.langfuse block is named as removed by the unknown-key check: a warning, or a refusal under HANGAR_CONFIG_STRICT and in mcp-hangar config check.

LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are the Langfuse SDK's own variables and are left alone.

To send Hangar's spans to Langfuse, use the OTLP exporter (Langfuse accepts OTLP over HTTP, not gRPC):

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://cloud.langfuse.com/api/public/otel/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic%20<base64 of public_key:secret_key>,x-langfuse-ingestion-version=4"

Use the TRACES_ variables: the generic OTEL_EXPORTER_OTLP_ENDPOINT also turns on OTLP audit log export, which Langfuse does not accept. The full recipe is examples/langfuse/README.md.

A pinned server's catalogue is re-listed every 60 seconds

Before, the catalogue a digest pin is checked against was refreshed only when a server started and when its upstream sent tools/list_changed. An upstream that changed a pinned tool's schema without announcing it was served under the old pin until the gateway restarted.

Now every READY server that a pin covers -- a tool_projection.pins or tenant_overrides.<tenant>.pins entry on the server itself or on a group it is a member of -- is re-listed on an interval, set by a new top-level key:

tool_projection:
  pin_recheck_interval_s: 60   # the default; 0 turns the re-check off

A non-zero value must be between 5 and 3600; anything else, a string or a boolean included, refuses the boot. The value is read at start, so a reload does not change it. Servers that are cold, starting, degraded or DEAD are not listed and not started.

What to expect:

• A drifted pinned tool is refused (ToolDigestMismatchError, gate reason digest_mismatch) at most one interval, plus the time the pass takes, after the upstream changed it. The tool_digest_pin_drift_detected warning and one DigestMismatchEvent are emitted when the pass first sees it.
• One tools/list request per pinned server per interval reaches its upstream, from every replica.
• A re-list also refreshes the rest of the catalogue. A tool the upstream added without announcing it is now seen within one interval as well. On a pinned server whose capabilities enforcement blocks or quarantines a tool outside expected_tools, that server is now blocked at its next call instead of running on until a restart.

A SIEM export the gateway cannot perform refuses startup

Before, an unknown MCP_COMPLIANCE_FORMAT logged unknown_compliance_format at warning and the gateway started with no export. An MCP_COMPLIANCE_OUTPUT whose directory did not exist, or could not be written, started too, and every record was logged as an error and dropped.

Now each refuses the boot with a ConfigurationError naming the value:

Unknown MCP_COMPLIANCE_FORMAT 'cefx'; expected one of: cef, json-lines, jsonlines, leef, syslog
MCP_COMPLIANCE_OUTPUT '/var/log/hangar/audit.cef' cannot be appended to: [Errno 2] No such file or directory: ...

So does a set MCP_COMPLIANCE_FORMAT whose exporter cannot be imported, which used to log compliance_exporter_unavailable and start without export. The format is now matched after trimming whitespace, so " cef" is read as cef.

A deployment that started with one of those in its log will not start on this release. Fix the format, create the output directory writable by the gateway's user, or unset MCP_COMPLIANCE_OUTPUT to export to stderr. To run with no SIEM export, unset MCP_COMPLIANCE_FORMAT.

A write that fails after startup does not stop the gateway or refuse calls. It is counted in mcp_hangar_compliance_export_failures_total (labels format and reason), and /health/ready and hangar_health carry a compliance_export field whose status is degraded until a write succeeds again; /health/ready keeps answering 200. Alert on the counter.

Approving a held call whose batch was cancelled is refused with 409

When a call is held for approval and its hangar_call batch deadline passes before anyone answers, the call will not run. Before, approving it afterwards answered 200 with state: "approved", the record said approved, and ToolApprovalGranted was published naming the approver in decided_by.

Now POST /api/approvals/{id}/resolve with "decision": "approve" answers 409:

{"error": "Approval refused: the held call was cancelled and did not run", "state": "cancelled"}

The approval record moves to a new terminal state, cancelled (GET /api/approvals?state=cancelled lists them), and the gate publishes ToolApprovalCancelled, whose attempted_by names the approver, instead of ToolApprovalGranted. The mcp_hangar_approval_decisions counter gains the decision="cancelled" value. On a cancelled record, decided_by names who tried to approve the call, not who let it through: nothing did.

A denial after the deadline is still recorded as a denial, and an unanswered hold still expires. An approval tool or dashboard that treats every 409 from resolve as "already resolved" should read state; an audit consumer that enumerates approval events or states should add the new ones. An approval that lands on a different gateway instance than the held call still answers 200 there; the instance holding the call records it cancelled and publishes ToolApprovalCancelled, not a grant.

A tool access list written as a string or a mapping refuses the configuration

allow_list, deny_list and approval_list in a tools: policy (on a server, a group, a group member or a tool_access.member tenant entry) and in an access: block must now be lists.

Before, a string was split into its characters and a mapping was read as its keys, and the gateway booted. deny_list: add became the patterns a, d, d, so add was allowed. Now the boot fails, and a reload is refused with the previous policy kept in force, with a ConfigurationError naming the scope and the field, for example:

Invalid tools access policy for mcp_server 'calc': Invalid deny_list: expected a list of patterns, got str 'add'

Old form:

tools:
  deny_list: add

New form:

tools:
  deny_list: [add]

A configuration that used the old form booted with a different policy from the one written; it will not boot on this release until the list is fixed. hangar_load now refuses a policy that fails to parse (an empty pattern in allow_tools, deny_tools or approval_tools, or a value that is not a list) before anything is installed. It used to install and start the server first, and leave it loaded with no policy.

A tool access policy block that is not a mapping refuses the configuration

A policy block that is present must now be a mapping. This covers tools: on a server, a group or a group member, access: and each access.<kind>, and tool_access:, tool_access.member and each tool_access.member.<tenant> entry on a server. On a server, and on a group member that defines its server inline, tools: may still be a list of tool schemas, and each item must be a mapping.

Before, a block of any other shape was skipped and the gateway booted with no policy for that scope: tools: add left every tool allowed. A key with no value (YAML null) was skipped the same way. Now the boot fails, and a reload is refused with the previous policy kept in force, with a ConfigurationError naming the scope, for example:

Invalid tools access policy for group 'pool': expected a mapping, got str 'add'

Old form:

tools: add
access:
  prompt:

New form:

tools:
  deny_list: [add]

tools: [add] on a server was read as a tool schema list and stopped the boot with an AttributeError; it is now a ConfigurationError that says a policy is a mapping. A list under tools: on a group, or on a member that names a server declared under mcp_servers, used to be ignored and is now refused. hangar_load takes no policy block, only the allow_tools, deny_tools and approval_tools lists, which #1718 already refuses when they are not lists.

Upgrade to 2.24.0

mcp.server.id names the group on every span of a group call; the member is hangar.route.backend

mcp_server.cold_start and command.send.InvokeToolCommand used to carry the selected group member in mcp.server.id, while every other span of the same call carried the group. They now carry the group too, and the member moves to hangar.route.backend.

A trace query, dashboard or alert that selects those spans by member, such as name = "command.send.InvokeToolCommand" AND mcp.server.id = "<member>", finds nothing after the upgrade for calls routed through a group. Change it to hangar.route.backend = "<member>". A query by group keeps working and now matches those spans as well. Calls to a server outside any group carry the same value in both keys, so their queries need no change.

The lifecycle spans mcp_server.launch and mcp_server.startup_wait are not opened by the call, and keep naming the member they start in mcp.server.id, so a query over every span of a group call's trace still sees the member there.

An L7 refusal logs bounded verdict fields, and an uninspectable call is an error trace

batch_call_refused for an L7 egress refusal (error_type EgressPolicyDeniedError or EgressPolicyApprovalRequiredError) no longer carries reason, which held the policy's reasons joined into one free-text string, such as tool 'x' matched a deny rule. It carries l7_verdict (deny or require_approval), l7_mode (enforce), l7_rule_kind (tool, argument or header) and l7_inspection_failed, next to the policy_id it already had. A log query or alert matching text in reason for these lines has to move to those fields. The reasons themselves are still in the egress_policy_enforced warning and in the EgressPolicyEnforced event.

A call the policy refused because it could not inspect the arguments (they could not be serialized, or the inspection raised) is now hangar.call.outcome=error and ends batch.call.<tool> ERROR, where it used to read deny and UNSET: the verdict is a denial, but the evaluator broke. Its log line is batch_call_failed at warning with l7_inspection_failed=true, not batch_call_refused. The spans inside the call stay UNSET. The caller is refused exactly as before. In Audit mode nothing changes on the span; the egress_policy_violation_observed warning carries inspection_failed.

GET /mcp: a list_changed stream on a front door, 405 in egress

On the handshake era (2025-11-25 and earlier), GET /mcp used to open an empty SSE stream that nothing wrote to. It stayed open until the client left. It now does one of two things:

• On a front_door gateway, it is the stream that carries notifications/tools/list_changed, and initialize answers tools.listChanged: true. A client that re-lists on the notification now sees upstreams that arrive after it connected, without reconnecting.
• In egress, it answers 405 Method Not Allowed. MCP clients treat a 405 on this GET as "no stream offered".

A proxy in front of the gateway should pass text/event-stream responses unbuffered and allow an idle interval longer than 15 s, the ping interval. On each replica, one principal may hold 32 open streams and one tenant 256. A client past either gets 429 on the GET, while its POSTs are unaffected.

A stream ends when its principal's API key or a role is revoked, when a JWT's exp passes, and after an hour at the latest. The client then reconnects and authenticates again. A stream is told about the catalogue of the replica it is connected to (#877).

To confirm that streams get through a proxy, watch mcp_hangar_tool_list_changed_streams (open streams per replica) and mcp_hangar_tool_list_changed_notifications_total{transport="http"}. A cap being hit shows in mcp_hangar_tool_list_changed_streams_refused_total{reason}.

caller user, agent and session ids are off spans unless observability.tracing.caller_ids is on

Since 2.22.0 every batch.call.<tool> span carried mcp.caller.id, mcp.user.id, mcp.agent.id and mcp.session.id for an authenticated caller. They are now left off by default, as the telemetry data contract requires. mcp.caller.type, mcp.caller.tenant_id and mcp.correlation_id are still set. The attribute names are unchanged: only whether they are emitted changed.

A dashboard, alert or trace query that selects spans by mcp.caller.id or mcp.user.id finds nothing after the upgrade. Either move it to OTLP audit records, which carry caller identity whatever this setting says, or turn the ids back on:

observability:
  tracing:
    caller_ids: true

MCP_TRACING_CALLER_IDS=true does the same, and wins over the file.

batch_call_refused from a gate no longer carries error

Since 2.22.0 a call refused by a batch gate logged batch_call_refused with an error field holding the message the caller was told, which could be an approver's own reason. That field is gone. A log query or alert that matches on the text of error in these lines finds nothing after the upgrade: match on reason (a bounded code such as approval_denied or tool_withdrawn), gate or error_type instead, which are unchanged. The refusal message still reaches the caller in the tool result, and approval decisions keep their reason in the event store.

Audit and compliance feeds now include refused tool calls

A tool call a gate, the tool:invoke check or an L7 policy refuses now produces a tool_invocation audit record with mcp.tool.status=denied, where it produced none before. The compliance exporters write it under a new event type, ToolInvocationDenied: CEF signature 103 ("Tool Invocation Denied", severity 5), LEEF event id 103, syslog MSGID 103 at warning, and "event_type": "ToolInvocationDenied" in JSON lines. Nothing is removed or renamed. A SIEM rule, parser or dashboard that counts every tool_invocation record as a call that ran, or that keys on the known event types, should filter on the status or add the new type. The refusal's reason is in the bounded fields gate and gateReason (gate_reason in JSON lines), or l7Verdict, l7Mode, l7RuleKind and l7PolicyId for an L7 refusal.

A group call's audit record names the group in mcp.server.id; the member is hangar.route.backend

The tool_invocation audit record of a call routed through a server group used to carry the selected member in mcp.server.id. It now carries the group the caller named, the same value its spans carry since #1286, and the member moves to hangar.route.backend.

An audit query, dashboard or alert that selects group calls by member finds nothing after the upgrade. Change it:

• Old: mcp.event.name = "tool_invocation" AND mcp.server.id = "<member>"
• New: mcp.event.name = "tool_invocation" AND hangar.route.backend = "<member>"

A query by group, mcp.server.id = "<group>", used to need the group's membership at the time of each call and now matches directly. Calls to a server outside any group carry the same value in both keys, so their queries need no change.

The compliance exporters follow the same rule. For a group call, the server field now names the group: CEF cs1 (cs1Label=ProviderID), LEEF src, syslog provider and JSON lines provider_id. The member moves to a new field: CEF flexString1 (flexString1Label=RouteBackend), LEEF routeBackend, syslog routeBackend and JSON lines route_backend. A SIEM rule that matched the member in the server field, such as CEF cs1=<member>, should match flexString1=<member> instead.

Records written before the upgrade are not rewritten. Replayed events from before it carry no group, so their records fall back to the member in both keys.

L7 header rules no longer match on hangar_call

An MCPEgressPolicy header rule (headers.allow, headers.deny or headers.requireApproval) is no longer consulted for a call made through hangar_call. That surface declares no x-mcp-header, so the SDK checks none of its Mcp-Param-* headers against the body, and ADR-025 says a selector must not match a header nothing checked. The call is decided by the policy's tool rules and its defaultAction, and the verdict's reasons include "header rules not consulted". A policy that relied on a header allow rule to let a hangar_call through a tool-name default-deny now denies it.

The same holds on the front door for an Mcp-Param-* header the called tool does not declare: only declared headers are matched.

To select on a header, call the tool on the front door (tool_access.mode: front_door) with the tool declaring the header through x-mcp-header on the argument it mirrors. That header is checked against the body before dispatch and still matches. Otherwise, express the rule as a tool-name rule.

headers.param_validation.required now refuses hangar_call with Mcp-Param-* headers

With headers.param_validation.required: true, a hangar_call request on a modern protocol version that carries any Mcp-Param-* header is now refused with HEADER_MISMATCH (-32020) and the message "the request's Mcp-Param-* headers could not be validated against its body". Before, it was served. hangar_call declares no x-mcp-header, so none of its Mcp-Param-* headers is ever checked against the body, and none reaches an L7 selector either.

The same refusal now applies on the front door to a call carrying an Mcp-Param-* header the called tool does not declare, next to a declared one. Before, only a failed pre-dispatch listing was refused.

A request with no Mcp-Param-* header is not affected, and neither is any deployment that leaves required at its default, false. A handshake-era (legacy protocol version) call is not refused either, on the front door or on hangar_call: its client does not validate headers, by design, so its Mcp-Param-* headers are ignored and never reach an L7 selector.

To keep a modern call working under required, stop sending Mcp-Param-* headers on hangar_call, or call the tool on the front door with the tool declaring the header through x-mcp-header.

An upstream response over the size limit fails the call instead of coming back dropped

Until this release a hangar_call result over 10 MB (10 485 760 bytes) was read whole, then dropped: its entry came back with success: true, result: null, truncated: true, truncated_reason: "response_size_exceeded" and original_size_bytes, the trace carried a hangar.shaping.drop event and mcp_hangar_batch_truncations_total{reason="per_call"} went up. A flat call and the facade's invoke were not cut.

The limit is now enforced while the upstream response is read, over stdio and HTTP, for every surface, and its default is 32 MiB (33 554 432 bytes). A response over it is not read to the end, and the call fails with error_type: ResponseTooLarge and the message The upstream response exceeded the limit of <n> bytes and was not read., the same on hangar_call, on a flat call (a tool error, isError: true) and on the facade. The call span ends in ERROR with error.type=ResponseTooLarge. Nothing that was served before is refused at the default, since every response under 10 MB is under 32 MiB.

• A client or alert that looked for truncated_reason: "response_size_exceeded" should look for error_type: "ResponseTooLarge" instead. The reason="per_call" series of mcp_hangar_batch_truncations_total is no longer written; reason="batch_budget" is unchanged, and so is batch truncation, with its continuation.
• To change the limit, set execution.max_response_bytes in the config file, or MCP_MAX_RESPONSE_BYTES in the environment, which wins over the file. A server can set its own max_response_bytes in its mcp_servers entry, which wins over both. Each must be a positive whole number of bytes; any other value refuses the config.

A front-door caller without tool:invoke can no longer call upstream tools

With auth on, the front door's flat tools/call now requires the tool:invoke permission, as hangar_call has since #389. A principal whose roles lack it, such as one holding only viewer, gets a tool error (Not authorized to invoke tool '<tool>': tool:invoke permission required) where it used to be served. An unauthenticated or anonymous caller is refused as Authentication required to invoke tools.

Grant a role that holds tool:invoke (developer, service-account, admin) to every principal that calls tools through a front door, for example in auth.role_assignments:

auth:
  role_assignments:
    - principal: "group:agents"
      role: service-account
      scope: global

Listing is unchanged: such a caller still sees the tools its tool-access policy allows, and is refused when it calls one.

With auth off (the default, and --unsafe-no-auth) nothing changes.

Over stdio with auth.enabled: true, both invoke paths now decide tool:invoke for the principal auth.stdio.principal declares, on the roles it declares. The default declaration is roles: [viewer], which does not hold tool:invoke, so its flat calls are now refused where they used to be served. Declare developer or service-account where a stdio session must call tools:

auth:
  stdio:
    principal:
      id: local-user
      tenant_id: local
      roles: [developer]

The same declaration also makes hangar_call over stdio with auth on serve a caller it used to refuse as unauthenticated. The declared roles are the principal's only role source: the configured role store and any OPA policy are not consulted for it.

set_governance_attributes is removed

The 2.22.0 upgrade note said set_governance_attributes in mcp_hangar.observability.conventions was kept for an ADR-007 adapter. It is now removed: it wrote mcp.caller.roles onto a span, which the telemetry data contract never allows, and the caller's own ids without the observability.tracing.caller_ids opt-in. Nothing in Hangar called it.

An adapter that called it sets the attributes it needs itself, with the constants that stay in mcp_hangar.observability.conventions:

# before
from mcp_hangar.observability.conventions import set_governance_attributes
set_governance_attributes(span, mcp_server_id="math", tool_name="add", caller_type="human")

# after
from mcp_hangar.observability.conventions import Caller, GenAI, McpServer
span.set_attribute(McpServer.ID, "math")
span.set_attribute(GenAI.TOOL_NAME, "add")
span.set_attribute(Caller.TYPE, "human")

Leave caller roles off: they belong on the audit record. Caller ids go on a span only when the operator turned observability.tracing.caller_ids on.

Upgrade to 2.23.0

a stdio front door advertises tools.listChanged

A front_door gateway served over stdio used to answer initialize with tools.listChanged: false, so a client listed once and kept that answer until it reconnected. It now answers tools.listChanged: true on protocol 2025-11-25 and earlier, and sends notifications/tools/list_changed when what that client is projected changes.

Nothing needs changing for a client that follows the spec: it re-lists on the notification. A client that cached on the strength of the old false and ignores the notification behaves as before. front_door over HTTP and egress still advertise false, and a client pinned to one replica is notified by that replica about its own catalogue (#877).

a group with no member left in rotation starts its members again

A group whose members had all failed out of rotation and then gone cold or DEAD (given up on, start failed or crashed) used to stay that way until an operator ran hangar_start on it. Each replica now starts those members itself, every 30 s at first and backing off to once every 10 minutes while the upstream stays down. Expect group_recovery_probe_* log lines and start attempts against an upstream that is still down. The same applies to members whose start failed when an auto_start group loaded. A member stopped on purpose (hangar_stop on it or on its group, or a detection block), one never started in an auto_start: false group, and one blocked for a capability violation are not started.

The first failed start of a member reaped for idling can degrade it, and the recovery saga then retries it up to max_retries times before it gives up, as it does for any degraded server. After that each failed probe start gives up at once, and only the probe's backoff spaces the attempts.

hangar_group_rebalance changed with it. It used to take every member that was not ready out of rotation. It now keeps, or puts back, a cold member, and one DEAD because its process crashed or its start failed, when that member is in rotation or left it on a failure: the call that selects such a member starts it. Members Hangar gave up on, degraded ones and ones blocked for a capability violation still leave rotation; the recovery worker, or hangar_start, brings those back.

Upgrade to 2.22.1

a pushed L7 policy survives a restart only when the gateway keeps its fleet

The operator delivers an MCPEgressPolicy to the gateway as a compiled L7 policy, over POST /api/mcp_servers/{id}/l7_policy, and re-delivers it only on its next reconcile. Whatever the gateway does not keep across a restart is therefore not enforced between the restart and that reconcile -- 2h16m in the report behind #1306 -- while the CR still reads Compiled. The network backstop is not affected: it lives in the cluster, not in the gateway.

Until this release, a restart dropped the policy of every server config.yaml declares, whatever the backend. Now whether a restart keeps it depends only on how the gateway stores its fleet:

This covers servers declared in config.yaml and servers registered over the REST API alike. Deleting the policy is kept the same way, so a restart does not bring back a policy the operator removed.

Two failures at start are not covered by the table. A stored policy that no longer parses fails closed: the server denies every tool until the operator delivers its policy again, and the gateway logs l7_policy_unreadable_denying_all at error. A database that cannot be read at start fails open: the gateway starts anyway, logs the failure at error, and serves its config.yaml servers without the stored policies until the operator delivers them again.

To keep the policy across restarts, select a durable backend in the chart:

persistence:
  backend: postgresql

or backend: sqlite with persistence.sqlite.persistentVolume.enabled: true. Multiple replicas need PostgreSQL in any case.

To find a deployment that does not keep it, look for l7_policy_not_persisted in the gateway log, or for "persisted": false in the response the operator receives. Both appear on every push, so a deployment that has one has it on every reconcile.

Upgrade to 2.22.0

TracedMcpServerService is removed; governance attributes now come from the call path

mcp_hangar.application.services.TracedMcpServerService and its legacy alias TracedProviderService are gone, along with the application.services.traced_provider_service module alias.

Nothing in Hangar constructed the class, so removing it changes no behaviour of the gateway: the decorator wrapped a service the real invoke path never used, which is why governance attributes did not appear on tool-call spans. ADR-029 retires it rather than reviving it, because wiring it in would have rebuilt a second pipeline beside the one that actually runs.

If you imported it, you were tracing a path your calls did not take. The attributes are now set by the gateway itself on batch.call.<tool>, so there is nothing to construct and nothing to wire:

# before -- traced nothing your callers reached
from mcp_hangar.application.services import TracedMcpServerService
service = TracedMcpServerService(mcp_server_service=inner, observability=adapter)

# after -- delete it; the attributes are on the span already
service = inner

set_governance_attributes in mcp_hangar.observability.conventions is kept. An ADR-007 observability adapter may still call it with its own process-local invocation data. Hangar's own boundary does not, because that helper also asserts gen_ai.operation.name=execute_tool and mcp.method.name=tools/call, which name the upstream call and stay on the execute_tool <tool> CLIENT span.

Langfuse scrubs tool inputs and outputs by default on the environment path too

If you enable Langfuse with HANGAR_LANGFUSE_ENABLED=true, tool inputs and outputs are now scrubbed unless you say otherwise: Langfuse receives {"scrubbed": true, "keys": [...]} in place of input_params and output (for an output that is not a mapping, its type in place of its keys). Before this release that path sent both in full, because it defaulted HANGAR_LANGFUSE_SCRUB_INPUTS and HANGAR_LANGFUSE_SCRUB_OUTPUTS to false, overriding the scrubbing default that LangfuseConfig and the config.yaml example already stated.

Deployments that configure Langfuse through config.yaml or the MCP_LANGFUSE_* variables already scrubbed by default and see no change.

If you rely on raw payloads reaching Langfuse, opt out explicitly:

# before -- raw payloads were sent with nothing set
HANGAR_LANGFUSE_ENABLED=true

# after -- say so deliberately
HANGAR_LANGFUSE_ENABLED=true
HANGAR_LANGFUSE_SCRUB_INPUTS=false
HANGAR_LANGFUSE_SCRUB_OUTPUTS=false

Only false, 0, no or off opts out. Any other value, including a typo, keeps scrubbing on.

OIDC signing keys are fetched over https only, and old pyjwt / cryptography / python-multipart are upgraded

Dependency floors

Upgrading raises pyjwt to at least 2.13.0 (now with its [crypto] extra), cryptography to at least 50.0.0, python-multipart to at least 0.0.31 and pydantic to at least 2.12.0. An environment that already had older versions installed has them upgraded by the same pip install --upgrade mcp-hangar. If you pin any of these yourself, for example in a constraints file or a lockfile of your own, raise the pin, or the install fails to resolve.

A non-https jwks_uri or discovery issuer no longer starts

Before, an auth.oidc issuer or jwks_uri that was not https:// logged oidc_issuer_not_https or jwks_uri_not_https and was used anyway. Whoever can answer a plain-HTTP request to that URL chooses the keys every token is verified against, so the gateway now refuses to start with a ConfigurationError that names the field and the value.

What is checked is the URL the keys come from: jwks_uri when it is set, otherwise the issuer, whose /.well-known/openid-configuration names the JWKS. When jwks_uri is set, the issuer is not fetched, only matched against the token's iss, and is not checked. A jwks_uri returned by discovery that is not https is refused as well. That check happens at the first token, not at startup, so it shows as 401 responses with jwks_uri_not_https in the log.

http://localhost, http://127.0.0.1 and http://[::1] are still accepted, for an IdP on the same host during development. Every other http:// host is refused, including a service name inside a compose network or a cluster, as are all other schemes (file:// included).

To move across, serve the IdP over TLS and use its https URL:

auth:
  oidc:
    enabled: true
    issuer: https://keycloak.example.com/realms/mcp-hangar
    audience: mcp-hangar

a refused call is no longer an error trace

Spans Hangar owns end UNSET when a call was refused on purpose: a gate deny, an egress policy denial, a call routed to approval, a spent rate-limit budget. They ended ERROR before.

If you count refusals by looking for error traces, that query stops matching them. What to use instead, all of it already on the span:

# before -- everything that failed OR was refused
status = ERROR

# after
status = ERROR                      # failures only
hangar.call.outcome = "deny"        # a refused batch call
hangar.refusal.gate, hangar.refusal.reason   # which gate refused it, and why
hangar.dispatch.outcome = "rejected"         # a refusal at the command bus

The bounded error.type is unchanged and still names what refused (for example EgressPolicyDeniedError), so a query keyed on it keeps working. The exception event is no longer added for a refusal.

An alert on Hangar's error rate now excludes refusals. That is the point of the change, but it does mean a dashboard that looked busy under a denying policy will read lower without anything having changed upstream.

hangar.dispatch.outcome is rejected for every refusal that reaches the command bus, not only a rate limit; an L7 denial reported error there before. The SDK's own SERVER span is untouched and still ends ERROR for a refused call, as ADR-029 s5 records.

a refused call logs batch_call_refused, and the per-gate lines are debug

A batch call refused by a gate now logs exactly one warning:

batch_call_refused  gate=withdrawal reason=tool_withdrawn tool=... mcp_server=...
                    tenant_id=... call_id=... error_type=ToolWithdrawnError elapsed_ms=...

gate and reason are the values the call's span carries as hangar.refusal.gate and hangar.refusal.reason.

If you alert on the per-gate lines, move those alerts to this one. These four were emitted at info and are now debug, which a default deployment does not print, because each only restated a refusal that is now reported once:

• tool_withdrawn_rejected
• tool_access_denied
• tool_digest_pin_rejected
• tool_digest_pin_unresolvable

tenant_quota_exceeded is unchanged: it carries the budget and the approval it spent, which the refusal line does not.

Warning volume rises on a deployment that refuses calls routinely -- a strict tool-access policy, a withdrawn tool still being called. That is the intended trade (ADR-029 s5): one refusal is one warning, in the log and in the trace.

Upgrade to 2.21.0

giving up on a server is recorded as a stop

When the recovery saga gives up on a server, the server now records McpServerStopped with reason: max_retries_exceeded, and right after it the McpServerStateChanged that moves it to dead. The server still ends dead, live and when its stream is replayed.

• mcp_hangar_mcp_server_stops_total{reason="max_retries_exceeded"} counts each give-up once, as in 2.20.0. It is now counted from the event, the way idle and shutdown are. To alert on a give-up, use increase(mcp_hangar_mcp_server_stops_total{reason="max_retries_exceeded"}[1h]) > 0. A reason!="idle" rule also matches every operator stop.
• The reason label is a closed set, and the metric's HELP line lists it: idle, shutdown, user_request, manual, failback, compensation, detection_enforcement:block and max_retries_exceeded. A REST stop whose body names any other reason is now counted as manual. The response still returns the reason it was given.
• The audit log and the event store hold the new McpServerStopped record. A consumer that reads every McpServerStopped as cold should read the state change that follows it.
• The alert handler's warning for an unexpected stop now fires on a give-up.
• A failover whose backup is given up on ends, as it does when the backup is stopped. The primary's recovery no longer schedules a failback stop that would have turned the dead backup cold.

a front-door call the gateway cannot answer is a tool error, not -32602

On a front door, a flat tools/call whose upstream answer could not be turned into a valid CallToolResult -- one with no content -- reached the client as a JSON-RPC error with code -32602 ("Invalid params"), which blamed the caller's arguments for an upstream's answer. The gateway's own per-call log line said outcome: ok for that same call, so the log and the client disagreed about it.

Such a call is now answered the way every other failure on this surface is: a result with isError: true, whose text says the upstream result could not be returned. The upstream's payload is not quoted in it, and a valid result is unaffected.

• A client that watched for the -32602 now sees an ordinary tool error, and has to read isError to notice the failure.
• The front_door_tool_call line for that call now reads outcome: tool_error with reason: invalid_result. An alert that counts non-ok outcomes will see calls it used to count as ok -- they were already failing for the client.

an upstream task is handed only to a caller that can poll it

When an upstream answers a tools/call with a task, Hangar now hands the task only to a caller that can poll it: a client on the 2026-07-28 revision that declared the io.modelcontextprotocol/tasks extension. That is the caller tasks/get, tasks/update and tasks/cancel already served.

• Any other caller gets a tool error, not a task. On the front door's flat tools/call it is a result with isError: true whose text names the extension to declare. In a hangar_call batch the call fails with error_type: "TasksNotNegotiated". Before, such a caller was handed the task, and tasks/* then refused it (-32021 or -32601), so it could never read the task.
• The upstream has still made the task. Hangar does not record it and does not cancel it.
• An upstream task in the current flat shape now works. A task an upstream answers with as resultType: "task" (SEP-2663) is recorded and relayed like one in the older nested shape. On hangar_call the flat handle used to come back unrecorded, so tasks/* answered "Task not found", and the front door answered -32602.

To keep receiving tasks, declare io.modelcontextprotocol/tasks under clientCapabilities.extensions on each request.

a group member is judged by its own health

A call through a group used to count against the member that took it whenever the call did not succeed. It now counts only when the outcome is evidence the member is unwell. Nothing to change in configuration. What you may see:

• Fewer members leave rotation, and a group's circuit opens less often. mcp_hangar_group_circuit_open is at 1 less often, and hangar_group_list shows consecutive_failures at 0 for a member that only answered bad requests.
• A member that answered the request counts as healthy, as a success does: a tool result with isError: true, a JSON-RPC error -32602 (invalid params) or -32601 (method not found), and a JSON-RPC error whose code is outside the reserved range -32768 to -32000, such as a tool's own -1.
• A refusal Hangar makes before asking the member counts as nothing: the command bus's rate_limit, an egress rule, or a tool the member does not list. Refusals by Hangar's gates (tool access, pins, validators, approval, the tenant's budget) already counted as nothing.
• These still count against the member, as before: a transport failure, a timeout, a failed start, the member's own backoff or dead state, and a JSON-RPC error in the reserved range, such as -32603 (internal error), -32700 (parse error) or -32000 (server error). An HTTP upstream's 4xx or 5xx status reaches the group as -32000, so it counts.
• An alert that fired when callers sent bad requests through a group no longer fires for them. Alert on the tool errors themselves instead.

a reload restarts only the servers whose settings changed

A configuration reload used to restart every server whose entry left resources out, which is most of them, even when nothing in the file had changed. That applied to a reload over POST /api/config/reload, SIGHUP or the file watcher alike. Each reload stopped those servers and dropped their in-flight calls, and their next call started a new process.

A reload now keeps a server running, with the same process, sessions, health and circuit state, unless the file changes something the server is built from: mode, command, args, image, build, endpoint, env, volumes, resources, network, read_only, user, description, idle_ttl_s, health_check_interval_s, max_consecutive_failures, a list of predefined tools, auth, tls, http or capabilities. Defaults count as set: leaving a setting out and spelling out its default are the same server. A group's inline members follow the same rule.

What changes for you:

• A governance change no longer restarts a server. A reload applies a change to a server's tools access block, access, tool_access, tool_projection or header_exposure, or to a member's weight or priority in its group, and the server keeps running. Before, such a change restarted the server.
• mcp_servers_updated lists only the servers the reload restarted, in the reload response and in the ConfigurationReloaded event. The servers it kept are in mcp_servers_unchanged.
• A reload no longer restarts a server whose settings did not change. To restart one, stop it with hangar_stop or POST /api/mcp_servers/{id}/stop. Its next call starts it again.
• A server whose env, description or intervals were edited over the REST API is still restarted with the file's values, as before.

the Python facade stops idle servers and health-checks them

Hangar and SyncHangar now start the background workers mcp-hangar serve starts, and stop them on stop() or at the end of the async with or with block. No code change is needed, but an embedded gateway now does what a served one does:

• A server idle for longer than its idle_ttl_s is stopped by the GC worker, which runs every 30 seconds. HangarConfig.add_mcp_server() defaults idle_ttl_s to 300. The next call starts the server again, and pays its start-up time. To keep a server running for the life of the host, give it a larger idle_ttl_s, up to 86400.

• Every running server is health checked by the health-check worker, every 60 seconds, so a failing server is noticed, and one that recovers is returned to rotation, without a call.

• The metrics snapshot worker records metrics history under ./data, as it does under serve.

• A facade started from a config file, Hangar.from_config() or SyncHangar.from_config(), watches that file, and a change to it reloads the configuration. To keep the file from being reloaded, set:

  config_reload:
    enabled: false

stop() now waits for the workers' threads to end, up to 10 seconds in total.

a group member is its top-level server, whatever the order in the file

A group member whose id names a top-level server in mcp_servers is now that server, wherever the group appears in the file, at startup and on a reload. Before, a group listed above the server got a member built from the member entry alone. With only an id, that was a server with no command, which could not start, and not the server the rest of Hangar knew by that id.

Two configurations now load differently:

• A member that names no server is refused. A member whose id is not a top-level server, and whose entry does not say how to run one, fails the load: Group 'pool' member 'm1' names no server. Before, it loaded and failed only when a call was routed to it. A reload with such a member is refused and changes nothing. Declare the server under mcp_servers, or give the member entry what its mode needs: command for subprocess, image or build for docker, endpoint for remote.
• A member entry cannot redefine a top-level server. When a member entry sets server fields such as command, env or mode, and a top-level server has the same id, the member is the top-level server and the entry's server fields are ignored. Before, which of the two the group held depended on the order in the file. The ignored fields are named in a group_member_entry_settings_ignored warning. weight, priority and tools on a member entry still apply.

per-tenant execution budgets

execution.max_concurrency bounds the whole process, so one tenant's burst could take every execution slot. The new, optional execution.tenant_limits section bounds each tenant on its own. With no tenant_limits section, nothing changes.

execution:
  max_concurrency: 50
  tenant_limits:
    "tenant:a": {max_concurrency: 4, rps: 10, burst: 20}
    "*": {max_concurrency: 2, rps: 5, burst: 10}

• max_concurrency is how many of the tenant's calls may be executing at once. rps and burst are a token bucket: calls start at rps per second on average, and at most burst at once. All three are required. max_concurrency and burst are integers from 1 to 1000000000, and rps is above 0 and at most 1000000. A misspelt or extra key, or a value out of range, refuses the configuration.
• A tenant that is not listed gets a budget of its own, built from the "*" entry. "*" is a template, not a pool that unlisted tenants share. Callers with no tenant share one such budget. With authentication off, no caller has a tenant, so every caller shares that single "*" budget.
• With budgets configured and no "*" entry, an unlisted tenant and a caller with no tenant are refused. If you add tenant_limits for a few tenants, add a "*" entry too, unless refusing everyone else is what you want.
• none cannot be a tenant id in tenant_limits: it is the metric's label for a call that no entry applied to.
• Where the budget is taken. Both parts come after the policy gates (tool access, withdrawal, pins, the circuit breaker and validators), so a call they refuse spends nothing.
  • The token, and the check that the caller has a budget at all, come before the approval hold and the cold start. A caller with no budget, or over its rate, is refused before anyone is asked to approve the call, and before the call starts a stopped server. On a server that has not started yet, a pinned tool's pin can only be checked once the server starts, so such a caller is refused with TenantQuotaExceeded, not a pin mismatch. A call refused after that point (denied or expired at approval, no longer valid after the hold, or its server failing to start) gets its token back.
  • The slot comes last, just before the upstream call. A call held for approval, or waiting on a cold start, holds no slot. The slot is given back when the call returns. A call that the upstream answers with a task handle returns with the handle, so a task the upstream keeps running does not hold a slot.
• An approved call can still be refused if all of its tenant's slots are taken when it is dispatched. The approval is then spent: running the call again needs a new one. The tenant_quota_exceeded log line, a warning in this case, names the approval. Leave room in max_concurrency for tools that need approval.
• A tenant at its concurrency limit can still start a stopped server, with a call that is then refused.
• A call over its budget is refused at once, never queued or retried, with the error type TenantQuotaExceeded. The front-door call log records it as denied, and mcp_hangar_tenant_quota_refusals_total{budget,reason} counts it, with reason one of no_budget, concurrency or rate.
• Budgets are counted per process, like execution.max_concurrency. With N replicas a tenant can run up to N times its budget, so size each budget for your replica count.
• A reload keeps the budget of every tenant whose limits did not change, with its calls in flight and its spent tokens. A tenant whose limits changed keeps counting the calls it has in flight, so lowering a limit never lets more than the new limit start. A tenant removed from tenant_limits is refused from then on, and calls it still has running are counted again if it is added back. Calls already running when budgets are first turned on are not counted.

a front door can wait for its catalogue before it is ready

Opt-in: nothing changes unless you add tool_access.required_catalogue, and it only takes effect with tool_access.mode: front_door.

tool_access:
  mode: front_door
  required_catalogue:
    servers: [payments, search-pool]
    retry_for_s: 600

• retry_for_s, 600 by default, is a window that opens when the replica first applies its configuration. Inside it, /health/ready answers 503 until the boot warm-up has projected every listed server once. When they all have been, or when the window ends, readiness stops depending on the catalogue and goes back to today's rule. A replica is held out of the Service for at most retry_for_s, even if a listed backend never comes back.

• The window counts from that first apply, before the rest of boot and the warm-up, so set retry_for_s to cover boot plus the warm-up. If they outlast it, the retry still gives each missing server one attempt before it ends; readiness does not wait for that attempt.

• The readiness endpoint is unauthenticated, so its catalogue field reports counts and state only: complete, holds_readiness, required, projected, missing_count, not_retried_count, and retry.

• The missing ids, and the reason the retry will not start a server, are logged in a required_catalogue_waiting line each time they change, and hangar_health returns them under catalogue. Both keep reporting after the window ends.

• Within the window, a server the warm-up could not start is retried. The retry starts a server the way a call does, so a dead server waits out its backoff. It never starts a server that is dead for given_up or capability_blocked, it leaves a degraded server to the recovery saga, and it never starts a server this replica has already projected, so a server stopped for being idle stays stopped. Each attempt writes a required_catalogue_retry log line and one sample of mcp_hangar_catalogue_retries_total{mcp_server, outcome}.

• The retry ends in a final state: finished once the list is met, blocked as soon as every server still missing is one it may not start, exhausted when the window ends, and stopped at shutdown or when a reload removes the list. After blocked, readiness still waits for the rest of the window.

• Once every listed server has been projected, readiness never depends on the catalogue again: a backend that stops, goes idle or fails later does not make the replica not ready.

• A group id is satisfied once any one of its members has been projected. A member defined only inline in its group can be listed too.

• The block is checked at load, from a file and from a dict alike. These refuse the configuration:

  • an id that is not a server or group in mcp_servers;
  • a group with no members;
  • a key other than servers and retry_for_s;
  • a retry_for_s that is not a number above 0;
  • with a coordination: block or a shared storage backend, a listed server in a local mode (subprocess, docker, container, podman), or a group whose members all are. Only the replica holding the management lease may start one, so the others could never project it. Use remote mode for a server every replica must serve.
  • A persistence backend registered by a plugin is treated as shared, so a local-mode server is refused there too, as a precaution.
  • A single-replica deployment on a shared backend, such as postgresql, cannot require a local-mode server either: the configuration cannot know how many replicas will run it.

  In egress it is checked and then ignored.

• A reload checks the block like any other key, and never moves the window. Inside the window, a reload replaces the list, and one that removes the block releases the wait. A replica that has met its list stays ready. After the window has ended, or on a replica that booted without a list, a reload that adds a required server does not hold readiness and does not start a retry: the server is reported in the log and in hangar_health, and is started by a call or a deliberate start.

A call's cold start now follows the call rules, in every topology. The batch executor starts a cold or dead server as a call, not as a deliberate start, so a server that became capability-blocked, or went back into its backoff, after the executor checked it is not started by the call. The call is refused with the code the executor's own check gives the same condition: CircuitBreakerOpen inside a backoff, CannotStartMcpServerError for a capability block.

If readiness stays 503. Read the required_catalogue_waiting log line, or hangar_health, for the ids. A server under not_retried is one Hangar gave up on or blocked for a capability drift. Fix it, then start it deliberately (hangar_start, or a start through the REST API), or take it off the list; otherwise readiness falls back when the window ends.

Probe timings. A replica can now stay not ready for up to retry_for_s after it starts. A failing readiness probe does not restart a pod, but a rollout waits for it, so keep the Deployment's progressDeadlineSeconds above retry_for_s.

the HTTP graceful-shutdown bound can be set

serve --http reads a new key, http.graceful_shutdown_timeout_s. It is how many seconds a stop waits for the requests already in flight before it cancels them.

http:
  graceful_shutdown_timeout_s: 90

Nothing changes unless you set it. Unset, Hangar passes uvicorn its own default, None, which waits for in-flight requests without a bound. The process then ends when they finish, or when something kills it. In Kubernetes that is the kubelet's SIGKILL at the end of the pod's terminationGracePeriodSeconds, 30 seconds by default.

• The value is a positive whole number of seconds. Any other value, or an http that is not a mapping, refuses to start. A reload with such a value is refused too, and everything keeps running as it was.
• The bound is read when the HTTP server starts. A reload checks it, but the running server keeps the bound it started with. Restart to change it.
• Stdio mode has no HTTP server, and ignores the key.
• starting_http_server logs the bound in force as graceful_shutdown_timeout_s, and logs null when it is unset.

In Kubernetes, the bound only helps if the pod lives long enough to use it. The kubelet counts the grace period from the start of the preStop hook, so set terminationGracePeriodSeconds longer than the preStop delay plus the bound, with room for Hangar's own cleanup after it. The mcp-hangar Helm chart sets all three from its shutdown values, and refuses to render a grace period that is too short.

the Python facade's invoke applies the configured controls

Hangar.invoke and SyncHangar.invoke called the server directly, so none of the call-time controls in your configuration applied to them. They now run the call through the same executor as hangar_call: tool access and withdrawals, digest pins, validators and interceptors, approval, the global and per-server concurrency limits, and tenant budgets all apply.

Pass the caller. invoke takes a new, optional principal=. The call is authorized for tool:invoke as an authenticated hangar_call caller is, by the roles your configuration gives that principal id and its groups. The principal's tenant_id is the tenant the per-tenant controls are applied for.

from mcp_hangar import Hangar
from mcp_hangar.domain.value_objects import Principal, PrincipalId, PrincipalType

caller = Principal(
    id=PrincipalId("agent-1"),
    type=PrincipalType.SERVICE_ACCOUNT,
    tenant_id="team-a",
)

async with Hangar.from_config("config.yaml") as hangar:
    result = await hangar.invoke("math", "add", {"a": 1, "b": 2}, principal=caller)

SyncHangar.invoke takes the same principal=.

Nothing verifies the principal: your application vouches for its id, groups and tenant, as an authenticator does for a request. Principal.system() is refused with ValueError, because authorization grants the system principal every permission.

Without a principal, the call is an anonymous caller's, the same as an unauthenticated hangar_call:

• With authentication configured, it is refused with the code AuthorizationDenied.
• It carries no tenant. With execution.tenant_limits set, it shares the budget of callers with no tenant, built from the "*" entry, and is refused with TenantQuotaExceeded when there is no "*" entry.

There is no way to make an unchecked call. If your configuration refuses anonymous callers, pass a principal.

What a call that does not succeed raises.

• A call a control refuses, or one that fails upstream, raises the new ToolCallFailedError. It is a ToolInvocationError, so an except ToolInvocationError still catches it. Its code is the error_type that hangar_call reports for the same call: for example ToolAccessDeniedError, ToolWithdrawnError, ValidatorDenied, TenantQuotaExceeded or CircuitBreakerOpen. Its message is the text hangar_call reports.
• An upstream failure that invoke let through as its own exception type, such as ToolTimeoutError or ClientError, is now a ToolCallFailedError whose code names that type.
• McpServerNotFoundError, ToolNotFoundError and TimeoutError are raised as before.
• A server that fails to start used to raise McpServerStartError. It now raises ToolCallFailedError with code == "McpServerStartError", so an except McpServerStartError no longer catches it.
• The result is still returned whole. The per-call size limit (10 MB) and a truncation: section cut hangar_call results, not the results invoke returns, and no continuation is stored for an invoke call.

A tool that needs approval.

• invoke raises TimeoutError at timeout_s, and the event loop is not blocked while the call waits.
• The call still holds one of the facade's pool threads until the approval is decided or expires (approval_timeout_seconds, 300 seconds by default). An approval given after invoke timed out is refused, so the tool does not run.
• That pool also runs stop() and health(), and each pending approval takes one of its threads. Size it with HangarConfig().max_concurrency(...) for the approvals that can be pending at once.
• SyncHangar.invoke blocks the calling thread for up to timeout_s.

Also:

• invoke accepts a group id, as hangar_call does, and the call goes to the member the group selects.
• timeout_s still bounds the wait. The call itself is given timeout_s clamped to 1-300 seconds, as hangar_call clamps its timeout.
• A facade call now writes the hangar_call span and log lines, and is counted in the batch metrics.
• A call through invoke has no session and no request headers: session suspension does not apply to it, and an L7 rule that selects on Mcp-Param-* does not fire, as for hangar_call over stdio.

a cluster refuses mode-less servers and local inline group members

A configuration with a coordination: block is refused at startup when it declares a server that runs as a child process of one gateway, because only the replica holding the management lease can run it. The check read only top-level mcp_servers entries whose mode was written as subprocess, docker or container. It now also refuses:

• a server with no mode. The loader builds it as subprocess, with or without an endpoint;
• a group member whose id names no server under mcp_servers, when its own entry in the group's members: list defines a server with a local or missing mode. The error names it <group>/<member>. An entry that says neither is still refused by the loader as naming no server;
• mode: podman. That configuration already failed to load, later, with 'podman' is not a valid McpServerMode.

To fix a refused configuration, give each server it names mode: remote and an endpoint. For a group member, do that in the member's entry, or declare the server under mcp_servers and name it by its id. If this is one gateway, remove the coordination: block. A configuration without coordination: is unaffected.

docker discovery no longer waits for Docker

With docker discovery enabled and Docker unreachable, serve and Hangar.start() no longer wait out the docker source's connection retries, and a stop no longer waits for a connection attempt in flight. Nothing needs changing. Two things read differently:

• The docker source's is_healthy in GET /discovery/sources and hangar_sources reports whether discovery's last connection or scan reached Docker. It no longer connects and pings from the request. It is false until the first connection completes, and it follows a Docker that goes away or comes back at the next discovery cycle, not at the next listing.
• Each connection attempt times out sooner. Its requests, the API version check and the ping, time out after 5 s instead of the Docker client's 60 s. A Docker that takes longer than that to answer is treated as unreachable. Calls after the connection keep the client's default timeout.

the Python facade takes the management lease and warms a front door

Hangar.start() and SyncHangar.start() now start two things that serve starts and the facade did not:

• Under a coordination: block, the management lease keeper and the event tailer. An embedded gateway now takes and renews the lease, and follows the shared event log, as a served replica does. stop() stops the tailer, then shuts down, then releases the lease.
• In front-door mode (tool_access.mode: front_door), the catalogue warm-up. Every configured server is started at start(), on a thread of its own, so start() does not wait for it. With tool_access.required_catalogue set, the required-catalogue retry runs after it. stop() stops the retry and waits up to 10 s for the warm-up to finish.

In egress mode, and without a coordination: block, nothing changes.

What to check. An embedded front door now starts every configured server when it starts, not on first use, as serve does. stop() can take up to 10 s longer while a warm-up is still starting servers.

Concurrent starts. Concurrent start() calls now share one bootstrap: the later calls return once the first has finished. A stop() made while a start() is in flight waits for it, then stops everything it started. SyncHangar.start() and SyncHangar.stop() are serialised across threads in the same way. Before, two concurrent starts could both bootstrap, and the first context was never stopped.

each server stop is counted once

mcp_hangar_mcp_server_stops_total counted some stops twice. An idle reap added 2 under reason="idle". A stop through hangar_stop, the REST stop or block, or the failover saga added 1 under its own reason and 1 more under reason="shutdown". Each stop now adds 1, under the reason it was made for.

Stop rates drop after the upgrade, with no change in how often servers stop:

• reason="idle" halves.
• reason="shutdown" no longer counts a stop made through the stop command. It counts the stops Hangar makes by itself: a reload, an unload or delete, a group's stop_all, process exit.
• user_request, manual, failback, compensation, detection_enforcement:block and max_retries_exceeded count as before.
• A sum over every reason drops by one for each stop that was counted twice.

Review the alerts and recording rules on this counter, such as a rate(mcp_hangar_mcp_server_stops_total[5m]) threshold or a ratio against reason="shutdown".

McpServerStopped carries the stop command's reason. A stop through hangar_stop is now recorded as reason: user_request instead of shutdown. A REST stop is recorded under the reason its body names when the counter lists that reason, and as manual otherwise; the response still repeats the reason as given. A failback is recorded as failback, a compensation as compensation, a block as detection_enforcement:block. The event store, the audit log and the event stream show these values. The alert handler still raises no alert for these stops, and the recovery saga still clears a server's retry state after them.

each caller can have its own command-bus rate limit

rate_limit keeps its meaning: rps and burst are the budget all callers share, one per command type. Without the new key below, it admits and refuses what it did before, except for the read-only tools. Nothing needs changing. What is new:

• rate_limit.per_caller gives each caller a budget of its own, per command type, under the shared one. A caller is its tenant and its principal. Callers with neither are one caller and share one budget: anonymous callers, every caller when authentication is off, and work no request started. A caller that has used up its own budget is refused without spending the shared one, so the other callers keep theirs. Both keys are required, and a malformed per_caller stops startup. It is read at startup and counted per replica, as rate_limit is. Set it below the shared budget: at or above it, it never refuses a call the shared budget would not.
• The read-only tools are never refused by the rate limit: hangar_list, hangar_status, hangar_details, hangar_group_list, hangar_health, hangar_metrics, hangar_discovered and hangar_quarantine. They read Hangar's own state and call nothing outside it. hangar_tools is still limited, because it may start a stopped server. So is hangar_sources, because it runs every discovery source's health check, and the Kubernetes source's check is a call to the cluster's API.
• A refusal reads the same on every path. Its message names the code, the budget, the limit and when to retry, as in the example below. A hangar_call result has error_type RateLimitExceeded. The HTTP API answers 429 with a Retry-After header, and its details hold retry_after and scope (caller or all_callers). A client or an alert that matched the old text, Rate limit exceeded: N requests per Ms, needs the new one.

rate_limit:
  rps: 50         # all callers together, per command type
  burst: 100
  per_caller:     # each caller, per command type
    rps: 5
    burst: 10

RateLimitExceeded: this caller's rate limit for InvokeToolCommand is used up (10 at once, refilled at 5 per second). Retry after 0.20s.

per_caller is a key of its own so that no deployment admits more than it did. Making rate_limit itself per caller would have multiplied what a gateway admits in total by its number of callers, without the operator asking for it.

a failed start's degraded event and initialize log carry no upstream text

• When an upstream process answers initialize with an error, Hangar logs one mcp_server_initialize_refused line with mcp_server_id, exit_code and stderr_bytes. The mcp_server_process_exit_code: <code> and mcp_server_stderr: <text> lines are gone. The stderr still reaches the caller, in the McpServerStartError the start raises. A log query or alert that matched the old lines needs updating.
• McpServerDegraded.reason for a failed start is the error's type, such as McpServerStartError or ConnectionError, where it was the error's text. The event store, the audit log, the alert handler's details.reason and the security handler's details.reason carry the new value. A server that health checks degraded still reads health_check_failures. Events recorded by an earlier release keep the text they were written with.
• A failed tool call's security record (log_validation_failed, field tool) has the error's type as its message, where it was <type>: <message>.

a task's follow-ups follow its tool's current access

A relayed task's follow-ups are checked against the tool access in force when they arrive, as a new call of the task's tool is: the tool policy, the withdrawals and the caller's tenant. Nothing needs changing in configuration. What a client may see when a tool is withdrawn, or the policy stops allowing it for the caller's tenant, while its task runs:

• tasks/update is refused with -32602, the call's message, and data.error_type set to the call's refusal: ToolWithdrawnError or ToolAccessDeniedError. The upstream is not sent the input.
• tasks/get still answers with the task's status. A poll that would hand over what the tool produced, a result, an error or input requests, is refused the same way.
• tasks/cancel is always served, so a task whose tool was taken away can still be stopped.

A tool that is still allowed is followed up as before.

a member of several groups is governed by each of them on the front door

On the front door, a server that is a member of several groups is now governed by every one of those groups, as hangar_call naming that server already was. Before, the front door kept one group per member, whichever group the file declared last. The other groups' tools policy, tool_projection withdrawals and pins, and header_exposure block did not apply to that member there. A server in one group, or in none, is unaffected. For a server in several groups:

• It may now be refused a tool it was served before. A tool that any of its groups denies or withdraws, for every tenant or for the caller's, is no longer listed for it, and a call to it is refused. A call to a tool any of its groups pins is checked against that pin. This is the decision hangar_call already gave for the same config.
• A call is routed to the server itself, not through one of its groups, so no group's strategy picks the member that answers. A server in one group is still routed through its group.
• Its tools are counted under its own id in mcp_hangar_projected_upstream_bytes, not under a group's.

To serve such a tool again, allow it in every group the server is a member of, or take the server out of the group that refuses it.

a hangar_* tool call is charged to one rate-limit budget

rate_limit and rate_limit.per_caller keep their keys and their meaning. Nothing needs changing in a configuration. What they admit changes in three ways, each toward the configured numbers:

• A server or group a call names no longer has a budget of its own. The limited tools used to keep one per id, so starting four servers admitted four times the budget. Now a tool whose work is a command is charged at the command bus, once per command type: hangar_start and hangar_stop of a server, hangar_tools, hangar_warm and hangar_reload_config, whose budgets are StartMcpServerCommand, StopMcpServerCommand and ReloadConfigurationCommand. A tool whose work never reaches the bus is charged once to a budget named after it, whatever it names: hangar_load, hangar_unload, hangar_approve, hangar_discover, hangar_sources, hangar_group_rebalance, the continuation tools, and hangar_start and hangar_stop of a group.
• A call is charged once. The limited tools were charged by the deprecated check_rate_limit() and again by the command bus. A hangar_start of a server now spends one token, not two, and a refusal names the command, as in ... rate limit for StartMcpServerCommand is used up .... A call that names no server spends nothing, since it does no work.
• Waiting no longer refills a bucket past its rate. A bucket idle for the cleanup window (60 seconds) was dropped and came back full. It is now dropped only once it has refilled, so after a wait a caller gets what rps refilled in that time.

mcp_hangar.server.validation.check_rate_limit() is removed. It was deprecated and nothing in Hangar calls it now. Code that called it can call charge_tool(<tool name>) from the same module, which charges the caller's budget and the shared one for that name.

a front door relays tasks from a current-spec upstream

A front door's flat tools/call now forwards the caller's io.modelcontextprotocol/tasks declaration to the upstream, as hangar_call already did. SEP-2663 leaves task creation to the upstream and gates it on the caller having declared the extension, so an upstream that follows the spec used to answer a front-door call with an ordinary tool result and never a task. Such a call can now come back as a task result (resultType: "task"), which the caller polls with tasks/get.

• What changes for a caller. A client that declares the extension under clientCapabilities.extensions on each request may now receive a task where it previously received a tool result, from upstreams that offer one. A client that declares nothing sees no change. Declaring the extension has always been the opt-in; on a front door it now has effect.
• The same request context also carries two things it did not. The caller's W3C trace context (traceparent / tracestate in params._meta) now parents the flat call's spans, so a front-door call joins the caller's trace instead of starting its own; and the request's Mcp-Param-* routing headers reach the L7 egress evaluator on this path, so a policy selecting on them can now fire for a flat call.
• A caller on an older revision is no longer spoken for. Whether the caller declared the extension is read once per request, together with whether its protocol revision has tasks/* at all. A caller without tasks/* has no declaration forwarded on its behalf, since it could not poll the resulting task.

a task no caller is handed is cancelled upstream

When an upstream answers a call with a task that Hangar will not hand over -- the caller cannot poll one -- Hangar now sends that upstream a best-effort tasks/cancel for it. The caller still gets the same refusal it got before, and it gets it without waiting: the cancel is sent off the request path, with one attempt and a bounded timeout, and its outcome is logged (cancelled, refused or failed) rather than surfaced.

Cancellation is cooperative under SEP-2663, so an upstream may decline. An upstream that does honour it will see tasks/cancel arrive for a task it just created, shortly after creating it, with no tasks/get in between. Before, that task was left to run until its own TTL with nobody able to collect the result.

error_type names the failure in every MCP tool error payload

A tool error payload used to name the failure under type, while a hangar_call result named it under error_type. Both now use error_type, so a client reads one key whichever tool answered.

{"error": "...", "error_type": "RateLimitExceeded", "details": {}}

A client that matched type on a tool error payload has to read error_type instead. The error and details keys are unchanged. This applies to every tool error, not only a rate-limit refusal. A hangar_call result already used error_type and does not change.

A rate-limit refusal also had more than one shape, depending on which limiter refused the call:

• The tool wrapper's own check (hangar_load, hangar_fetch_continuation and the other tools charged by name) raised out of the wrapper as an MCP error, so the call came back as a JSON-RPC error rather than a tool result. It is now the same error payload every other path returns. A client that caught a JSON-RPC error to detect this refusal now sees a normal tool result whose error_type is RateLimitExceeded.
• A refusal raised inside a tool body -- by the command bus, or by charge_tool for work a tool does itself -- was already a payload, and keeps being one.

details now carries the refusal's own fields on every path, where before a tool payload dropped them and left only the message text:

{
  "error": "RateLimitExceeded: this caller's rate limit for hangar_load is used up (2 at once, refilled at 5 per second). Retry after 0.20s.",
  "error_type": "RateLimitExceeded",
  "details": {
    "limit": 2,
    "window_seconds": 1,
    "retry_after": 0.2,
    "scope": "caller",
    "key": "hangar_load",
    "rps": 5
  }
}

scope is caller when the caller's own budget was used up and all_callers when the shared one was, as it already was over the HTTP API. Only a rate-limit refusal carries details; every other tool error still has details: {}.

Security log. A refusal by the command bus's limiter now reaches the security handler, which recorded only the tool-level ones before. Each refusal is recorded once, as rate_limit_exceeded, and its details gained scope, the key_kind (tool or command) and the key it names. These are values Hangar chose: no argument value and no caller's own text is recorded. A refusal is no longer also recorded as a validation_failed event.

an L7 egress policy set over the API survives a reload

An L7 egress policy is set at runtime, through POST /api/mcp_servers/{id}/l7_policy or the fleet projection, and no configuration file declares one. A reload that rebuilt a server -- because a setting it is built from, such as env, changed -- dropped that policy, and reported success.

A rebuild now carries the running server's policy onto its replacement, and logs l7_policy_carried_to_rebuilt_mcp_server when it does. A deployment that relied on a reload clearing a policy must now clear it explicitly, with DELETE /api/mcp_servers/{id}/l7_policy.

a call naming a group waits for the approver its member's L7 policy asks for

A call that names a group is routed to one of the group's members. When that member's L7 egress policy (MCPEgressPolicy) is in Enforce mode and routes the tool to requireApproval, the call is now held and an approval is raised, exactly as for a call that names the member directly.

Before, the approval gate looked the L7 policy up by the id the call named. A group id is not a server id, so a call naming a group found no policy and nobody was asked; the member's own check then refused the call at invoke with EgressPolicyApprovalRequiredError.

A call that was refused immediately may now wait. With an approval gate configured, such a call blocks until someone decides it, or until the timeout:

• approved -- the call runs, and its result comes back as usual;
• denied -- it is refused with approval_denied;
• nobody answers -- it is refused with approval_timeout after approval_timeout_seconds, which is 300 seconds when the rule comes from the L7 policy alone and no tool-access policy narrows it.

If a caller of yours relied on the immediate refusal, give it a timeout, or change the rule from requireApproval to deny where you meant "refuse" rather than "ask". Check as well that the configured channel reaches a person (approvals.channel): on noop the hold is real and nobody is told, so the call runs out its timeout.

Nothing changes without an approval gate. With approvals disabled, or with no gate wired, the call is still refused at invoke with EgressPolicyApprovalRequiredError, as before. The member's own invoke-time L7 check is untouched and still refuses first, so deny still wins and Audit mode still asks nobody. A call that names the member directly behaves as it did.

every error payload a tool answers with is error, error_type, details

a failed hangar_reload_config answers the tool error payload

A reload that failed used to answer a layout of its own:

{
  "status": "failed",
  "message": "Configuration reload failed: ...",
  "error_type": "ConfigurationError"
}

It now answers the payload every other tool failure uses:

{
  "error": "Configuration reload failed: ...",
  "error_type": "ConfigurationError",
  "details": {}
}

A client that detected a failed reload by status == "failed" reads error (or error_type) instead, and takes the message text from error rather than message. The text itself is unchanged, and error_type still names the exception class.

A successful reload is untouched: it still answers status: "success", message, the four mcp_servers_* lists and duration_ms.

MCPError.to_dict() is removed

mcp_hangar.MCPError -- and every domain exception under it -- carried a to_dict() that rendered:

{
  "error": "...",
  "mcp_server_id": "...",
  "operation": "...",
  "details": {},
  "type": "ConfigurationError"
}

Nothing in Hangar called it. The REST API builds its own envelope ({"error": {"code", "message", "details"}}) from the exception's attributes, and an MCP tool error is the three-key payload above, so this was a second serializer that named the failure under type where the answering surfaces named it code and error_type.

An embedder that called exc.to_dict() builds the dict itself:

{
    "error": exc.message,
    "mcp_server_id": exc.mcp_server_id,
    "operation": exc.operation,
    "details": exc.details,
    "error_type": type(exc).__name__,
}

The attributes it read -- message, mcp_server_id, operation, details -- are unchanged, as is every exception class and what raises it.

Upgrade to 2.20.0

block and quarantine stop a server whose tools drift

A server with capabilities.enforcement_mode set to block or quarantine whose upstream serves a tool that is not in capabilities.tools.expected_tools now serves nothing. Before, block mode detected the drift and marked the server dead, but the call that started it still ran the tool it asked for, and the upstream process was left running. Quarantine did not act on the drift at all and served the server as alert does.

What happens now, in both modes:

• The start that finds the drift fails with CapabilityBlockedError. Hangar closes its connection, records CapabilityViolationDetected, and moves the server to dead for a capability block (capability_blocked). It records no McpServerStarted. Quarantine also records McpServerCapabilityQuarantined.
• No call starts the server again: every later call to any of its tools, declared ones included, is refused with CannotStartMcpServerError. The recovery saga does not retry it, and a group does not put it in rotation.
• A tool that appears after a clean start, through a refresh or tools/list_changed, blocks the server in the same way at its next call. Hangar closes the connection at once, so a call already in flight on it fails.
• alert mode is unchanged.

If you set quarantine expecting it to keep serving, as it did, it no longer does: use alert for that.

The error does not name the undeclared tools, because the caller it reaches is not the operator who has to act on them. The capability_drift_detected warning and the CapabilityViolationDetected event name them.

To bring a blocked server back, fix the upstream or add the tool to expected_tools, then start the server deliberately: hangar_start, hangar_warm naming it, or a start through the REST API. That start checks the tools again, and fails the same way while they still drift. A stop leaves the server cold, and the next start, a call's included, checks them too.

Prompts, resources and task relays to a server now need it to be ready, the same as a tool call. Before, they were forwarded to any server with a live connection, a degraded one included.

egress calls are governed with their group and tenant scope

A hangar_call that names a group member by its own server id, instead of naming the group, is now governed by that group. Before, it was governed as if the member were a standalone server.

For each group that owns the member, a call naming the member is now checked the way a call naming that group and routed to that member is checked:

The call is still sent to the member it names. The group's member selection does not run. Everything the member's own server id declares applies as it did before.

A server that is a member of more than one group is governed by all of them, and deny wins. A tool that any one of them denies, withdraws, or pins to a digest the tool does not match is refused.

Approval lists declared on groups or tenants now take effect. The approval gate now reads approval lists with the same scope as the access policy. Before, it read only the approval_list of the server a call named, and asked without the caller's tenant.

A tool on any list that applies needs approval. The first list that applies supplies the timeout and channel. Approval routed by an L7 egress policy's requireApproval is unchanged.

Withdrawal is re-checked after an approval hold. A tool withdrawn while a call waits for approval, on the server, on an owning group, for every tenant or for the caller's, is now refused with ToolWithdrawnError once the call is approved. Before, it ran.

What to check.

• A caller that relied on naming a member to reach a tool its group denies, withdraws or pins now gets ToolAccessDeniedError, ToolWithdrawnError or ToolDigestMismatchError, as a call naming the group does. If that caller should reach the tool, allow it on the group, or move the server out of the group.
• If you declared an approval_list on a group, on a group member, or for a tenant, or you run front_door with approval lists, the tools on those lists now wait for approval. With no approval channel configured, a held call times out after approval_timeout_seconds. Remove a list you do not want enforced.
• On front_door, a call has a fixed 30 s budget, so a held front-door call must be approved within 30 s. An approval that arrives later is refused, and the call does not run.

Calls that name a group are governed as before, apart from their approval lists. So are servers that are in no group.

a continuation answers only the caller that made the call

With response truncation on (truncation.enabled), the rest of a truncated hangar_call result is kept in the continuation cache. hangar_fetch_continuation and hangar_delete_continuation now serve it only to the caller whose hangar_call produced it: the same tenant and the same principal. Anyone else who presents the id gets the answer for an id that does not exist.

What may need a change.

• A client that fetched a continuation as a different principal from the one that made the call now gets not found, even within the same tenant. Fetch it with the credentials that made the call.
• A caller with no tenant does not match a continuation stored with one, and the reverse.
• The result_truncated log line no longer carries continuation_id. It has batch_id, call_index and continuation_advertised. The cache and continuation-tool log lines carry continuation_ref in place of continuation_id: the id without its random suffix.
• A fetch or delete of another caller's continuation logs a continuation_owner_mismatch warning naming the owner's and the caller's tenant.

With auth off, neither the caller that makes the call nor the caller that fetches has an identity. Both are anonymous, and continuations work as before. Nothing authenticates a caller there, so this is not a boundary. With the memory cache, a continuation is still fetchable only on the replica that truncated the result.

Upgrading with the Redis cache. Values are now stored with their owner. A value written before the upgrade has no owner and is read as anonymous: with auth off it is still served, and with auth on it is not found. Either way it expires within cache_ttl_s. Until every replica runs this version, a replica on an older version still serves any continuation to any caller that holds its id, and returns a value written by an upgraded replica with the owner in front of the payload.

tool_access.rules is refused as a key nothing reads

tool_access.rules was never read. The config schema listed it next to tool_access.mode, so a rules: block passed mcp-hangar config check and HANGAR_CONFIG_STRICT=1, loaded without a warning, and restricted nothing. No document or example described it (#1422).

Delete it:

tool_access:
  mode: front_door
  rules: []   # delete this key, and anything nested under it

A config that still sets it loads, and logs unknown_config_key saying the key was never read. HANGAR_CONFIG_STRICT=1 and mcp-hangar config check refuse it, as they refuse any key nothing reads: under strict mode a gateway whose config still sets it does not start, so delete the key before upgrading. This applies to a file and to bootstrap(config_dict=...) alike.

Nothing that restricts a tool changes. Tool access is set by the tools: allow and deny lists of a server, a group and a group member, and tool_access.mode still selects the egress or front_door topology.

a reload applies the whole configuration, and keeps the topology mode

This affects every configuration reload: POST /api/config/reload, hangar_reload_config, SIGHUP, and the config file watcher.

A reload used to reset tool_access.mode to egress and apply only mcp_servers. A reload now applies every section startup applies from the configuration:

• mcp_servers, as before
• interceptors.validators
• ui_resources, the ui:// allow list
• headers.param_validation
• resource_links
• execution, the concurrency limits

A section you delete from the file goes back to its default when you reload. Before, it stayed in force until the next restart. Every section is checked before any server is stopped. A bad value refuses the reload, and nothing changes.

Sections that startup reads only once, such as auth, persistence, event_store, discovery and logging, still need a restart, as before.

A reload that changes tool_access.mode is refused

A reload keeps the mode the gateway started with. If the file sets a different tool_access.mode, the reload is refused and nothing changes, because the front-door tool surface is built at startup. Restart the gateway to change the mode.

The workaround for a front_door gateway, config_reload.enabled: false and a restart for every change, is no longer needed.

Policies set at runtime

Before, a reload removed every tool-access policy set at runtime. Now:

• The policies stored by the REST policy endpoint are replayed after the file, as a restart does. On a scope that both the file and the REST endpoint define, the stored policy applies, after a reload as after a restart. If the policy store cannot be read, the reload is refused and nothing changes; POST /api/config/reload answers HTTP 503.
• Any other policy set at runtime, such as one hangar_load set, is kept unless the file now defines the same scope. The file's policy then replaces it.
• A server that the reload removes takes its policies with it, as hangar_unload does. A policy the REST endpoint stored for it comes back at the next restart, as before.

A group's inline members count as declared by the file. A reload no longer stops them as removed servers or strips the policies set on them at runtime.

The groups, and the policies, withdrawals, pins and header_exposure blocks, are replaced rather than cleared and registered again, so a call made during a reload never finds them empty. A reload builds and checks every server and group before it stops any, so a bad block refuses the reload and changes nothing.

a config dict gets every setting it passes

This affects code that calls bootstrap(config_dict=...) directly, such as embedders and test harnesses. Hangar.from_config() and mcp-hangar serve read a file and are unchanged. Hangar.from_builder() passes a dict and is covered at the end of this section.

A dict is now applied the same way as the same document in a file. Before, the dict path dropped these settings without logging anything:

• tool_access.mode, so a dict that asked for front_door came up in egress
• interceptors.validators, so no parameter validator ran
• ui_resources, the ui:// allow list
• headers.param_validation
• resource_links
• execution, the concurrency limits

A dict now gets all of them. If a harness passed one of these and relied on it being ignored, remove it from the dict.

The schema check now runs on a dict too. An unknown or removed key logs unknown_config_key, and under HANGAR_CONFIG_STRICT=1 the boot refuses, as it does for a file.

Four more cases used to be accepted without a word:

Relative paths in a dict resolve against the working directory, as they do in a file.

Hangar.from_builder() no longer passes its own max_concurrency to the gateway, which never read it. It still sizes the facade's thread pool. A builder that calls enable_discovery(), or adds a server with mode="remote" and url=..., produces keys the gateway does not read. Those settings were never applied. They now log unknown_config_key, and under strict mode the boot refuses.

remote servers and discovery from the builder take effect

This affects code that builds its configuration with HangarConfig and runs it with Hangar.from_builder(). Code that calls Hangar.from_config() on a file that enables discovery is affected by the Hangar.start() change below.

Two builder features wrote keys the gateway does not read, so neither was ever applied. Both now take effect:

• add_mcp_server(..., mode="remote", url=...) writes the address as endpoint, the key the gateway reads. A remote server now boots with its address and answers calls. The argument is still called url=.
• enable_discovery(...) writes discovery: {enabled: true, sources: [...]}, one additive source per requested type. Additive sources add the servers they find and never remove one.

Hangar.start() now runs discovery when the configuration enables it, and Hangar.stop() stops it. bootstrap() builds the discovery sources and starts nothing, and only mcp-hangar serve used to start them. So under the facade, a discovery section built its sources and never ran them, whether it came from the builder or from a file. If you enabled discovery and relied on it doing nothing, remove the call or the section: discovery now registers the servers its sources report.

The builder now raises ConfigurationError on these calls. Each used to be stored and never applied:

build() now checks the configuration against the gateway's schema, and raises on a key the gateway does not read. to_dict() no longer includes max_concurrency, which sizes the facade's thread pool and is not a gateway setting. HangarConfigData no longer has gc_interval_s or health_check_interval_s.

a server Hangar gives up on reads dead, not cold

When the recovery saga runs out of retries, the server now goes to dead. Before, giving up was a stop, so the server went to cold: the state of a server nobody has called yet. Two other failures already reached dead but published no state change, so the gauges kept their last values. A crashed process read ready and up 1 until something called it, and a start that failed below max_consecutive_failures read initializing. All three now read dead.

This affects anything that reads mcp_hangar_mcp_server_state, mcp_hangar_mcp_server_up or mcp_hangar_mcp_server_initialized, and anything that reads a server's state from hangar_list, hangar_status or GET /api/mcp_servers.

An alert on mcp_hangar_mcp_server_state == 0 no longer fires for a server Hangar gave up on. 0 now means only that the server is not running and is not failing. Use one of these instead:

• mcp_hangar_mcp_server_state == 4 fires when a server is dead.
• The new gauge mcp_hangar_mcp_server_last_healthy_timestamp_seconds holds when Hangar last saw the server working: a passing health check, a completed start or a successful tool call. It is kept when the server goes cold or dead.

time() - mcp_hangar_mcp_server_last_healthy_timestamp_seconds > 900
  unless mcp_hangar_mcp_server_state == 0

Keep the unless, and keep its default matching. A cold server is not probed, so without the unless the rule also fires for every server reaped for being idle more than 15 minutes ago. unless on(mcp_server) would drop the instance label, so with more than one replica a server that is cold on one replica would hide it being dead on another. A server that was never healthy has no series, so pair the rule with state == 4.

sum(mcp_hangar_mcp_server_up) == 0 can newly fire. A crashed server used to keep reading up 1; it now reads 0. A pool whose only servers reading up had crashed now reads 0.

What starts a dead server again

Why it died decides. A call is refused while the server's backoff lasts and is told how long to wait: through hangar_call, CircuitBreakerOpen with the time to retry. Once the backoff has passed, the call starts it.

The deliberate starts are hangar_start on the server or on its group, POST /api/mcp_servers/{id}/start, hangar_warm naming the server, a group adding the server with auto-start, and a failover saga starting the backup it was configured with.

Nothing else starts a dead server:

• The health worker does not check it. It used to, every 60s, and counted mcp_hangar_health_checks_total{result="unhealthy"} for a check that sent nothing, so that series stops moving while a server is dead.
• The recovery saga cancels the restarts it has scheduled when it gives up, and when the server starts or stops.
• The bulk warm-ups skip it: the front door's at boot, and hangar_warm with no names, which now lists it under skipped_dead.
• hangar_tools lists a dead server, or a group's dead member, without starting it.
• The GC acts only on servers that are ready.

A group does not count a dead member as healthy. A member Hangar gave up on, or one a capability block stopped, leaves rotation, and a successful start puts it back, subject to the group's healthy_threshold. A member whose process crashed stays in rotation, so the next call through the group restarts it, as it always did. When that restart fails, or the call is refused inside the member's backoff, the call counts as the member's failure, so a member whose restart keeps failing leaves rotation and the group fails over.

Other changes

• A give-up no longer also counts mcp_hangar_mcp_server_stops_total{reason="shutdown"}. It still counts reason="max_retries_exceeded".
• hangar_stop on a dead server makes it cold.
• A server that is deleted, unloaded or reloaded away loses its lifecycle gauges, so a removed dead server stops reading 4. A deletion removes them on every replica. Its counters stay.
• With a durable event store, a server restored dead or degraded reads so from boot. A server restored ready has no connection in the new process: it has no series until its first health check or call, which make it cold, not dead.

a group's circuit_breaker.reset_timeout_s is removed

It never did anything. An open group circuit did not half-open once the timeout passed, however long it waited: a breaker half-opens only when asked whether to let a request through, and a group never asks. The circuit closes once min_healthy members are back in rotation, after a passing health check or a successful call, and that is unchanged. A timed probe would have been a second way out, competing with that one, so the option was removed rather than honoured (#1398).

Delete it from every group:

mcp_servers:
  pool:
    mode: group
    circuit_breaker:
      failure_threshold: 10
      reset_timeout_s: 60   # delete this line

A config that still sets it loads, and logs unknown_config_key naming the group and the key. HANGAR_CONFIG_STRICT=1 and mcp-hangar config check refuse it, as they refuse any key nothing reads: under strict mode a gateway whose config still sets it does not start, so delete the key before upgrading. The flat spelling circuit_reset_timeout_s is reported the same way. In Python, McpServerGroup(...) no longer accepts circuit_reset_timeout_s: passing it raises TypeError.

a group's healthy_count counts members that are ready

A group's healthy_count used to count every member in rotation that was not dead, cold ones included. It now counts the members that are ready and in rotation. The number of members in rotation, in any state, is a new field, members_in_rotation_count: the length of the members_in_rotation list that hangar_group_rebalance returns.

healthy_count changes meaning, so its value drops for any group with a member in rotation that is not ready. The common case is a group whose members the GC reaped for being idle. Before, it read healthy_count: 2 with nothing running; it now reads healthy_count: 0 and members_in_rotation_count: 2 until a call through the group starts a member.

This affects anything that reads a group's healthy_count from GET /api/groups, GET /api/groups/{id}, hangar_details, hangar_group_list, hangar_list, hangar_start or hangar_group_rebalance, or its healthy_members from hangar_status, hangar_health or hangar_metrics. The GroupStateChanged event's healthy_count changes the same way and gains members_in_rotation_count. No metric reports either count.

A check that treats healthy_count: 0 as a group that cannot serve now fires for idle groups that serve fine. A group routes as long as is_available is true. To ask whether a group can take a call, read is_available. To ask whether members are in rotation, read members_in_rotation_count.

What the group decides is unchanged. is_available, the group state and the min_healthy rule that closes an open circuit count the members in rotation that are not dead, as healthy_count did before.

Upgrade to 2.19.1

a suspended session is refused

POST /api/sessions/{id}/suspend now blocks the session it names. Before, it answered 200 and replicated the suspension to every replica, and no request path read it, so the session went on calling tools.

A request that carries a suspended session id is refused before Hangar does anything for it: no validation, authorization, approval, cold start or upstream call. A replica refuses the session once it has read the suspension from the shared event log. DELETE /api/sessions/{id}/suspend lifts it. Each refusal logs a session_suspended_call_refused warning naming the session.

Where a caller's session id comes from. A suspension refuses only a caller that carries the session id it names. Hangar reads it from one of two places, in this order:

If your identity provider puts the session id in another claim, name it with auth.oidc.session_id_claim, or per issuer with session_id_claim on an auth.oidc.issuers entry. A session id must match [A-Za-z0-9_-]{1,128}, the shape the suspend route accepts. Anything else is ignored. Mcp-Session-Id is not read.

To suspend API-key callers by session, put a proxy in front of Hangar that sets x-session-id and removes any the client sent. List the proxy's address in MCP_TRUSTED_PROXIES, which defaults to 127.0.0.1,::1.

What is not covered.

• A caller with no session id carries nothing to match, and is not refused. Suspension does not cut off a principal. To do that, revoke its API key or disable it at the identity provider.
• A stdio session has no session id: a pipe carries neither source.
• With auth disabled, nothing is refused. The suspend route is then open to every caller, so it was never a control there.
• tools/list, the REST API and /ws/events do not check suspensions. The tool listing answers from Hangar's own catalogue and reaches no upstream.
• A replica that starts after a suspension does not learn of it.
• A suspension expires after 24 hours. Each replica holds at most 10,000.

only MCP_TRUSTED_PROXIES decides the forwarded client address

Hangar now starts its HTTP server with uvicorn's forwarded-header handling turned off. Before, uvicorn rewrote the client address from X-Forwarded-For, and the scheme from X-Forwarded-Proto, for any peer in FORWARDED_ALLOW_IPS (default 127.0.0.1), before Hangar saw the request. Hangar then applied its own MCP_TRUSTED_PROXIES to what was left.

Now Hangar sees the address that actually connected. It believes X-Forwarded-For only when that address is in MCP_TRUSTED_PROXIES (default 127.0.0.1,::1). The client address is the rightmost forwarded entry that is not itself a trusted proxy, which is the rule uvicorn applied. Before, behind a proxy outside FORWARDED_ALLOW_IPS, Hangar took the leftmost entry, which the client can write.

What to do:

• If you set FORWARDED_ALLOW_IPS for Hangar, put the same addresses in MCP_TRUSTED_PROXIES. Hangar no longer reads FORWARDED_ALLOW_IPS.
• If requests pass through more than one proxy, list every proxy in MCP_TRUSTED_PROXIES. An unlisted hop is taken to be the client.
• A proxy on loopback needs nothing, because loopback is trusted by default. Its x-session-id is now honoured even when it also sends X-Forwarded-For. Before, the rewritten address made Hangar ignore it.
• The protected-resource metadata already read X-Forwarded-Proto itself, so the scheme it advertises is unchanged.

traces, the security log and Langfuse carry no error text

Hangar's telemetry no longer says what a failure said: a tool's error message can hold whatever the tool returned. A failed span ends in ERROR with an empty status description and a bounded error.type. Where an exception escaped the span, or a fault barrier handled one, the span has an exception event whose only attribute is exception.type, with no exception.message and no exception.stacktrace.

An error type that does not look like a class name or a code is recorded as _OTHER. Langfuse still receives tool arguments and results.

If a dashboard or alert matched on span status descriptions, on exception.message or exception.stacktrace, on the security log's details.error, or on the Langfuse error output, match on error.type or exception.type instead. For the message itself, look up the call's ToolInvocationFailed event in the event store, or the result the caller received. Neither has changed.

scrub_baggage_for_tenant is removed, and baggage is not forwarded

scrub_baggage_for_tenant is removed from mcp_hangar.observability, with no replacement. If you call it, delete the call. There is nothing to do instead: Hangar forwards no baggage.

Hangar propagates traceparent and tracestate only, over HTTP and stdio alike, and ignores baggage on inbound requests. Hangar sets no baggage itself, so this changes something only where host instrumentation or an embedding application attached baggage to the context, or a request carried a baggage key in _meta. Those entries no longer reach upstream servers.

Upgrade to 2.19.0

tenant-scoped role grants are limited to their tenant

A role bound at tenant:<id> is now a grant within that tenant only. That covers an auth.role_assignments entry with scope: "tenant:<id>", assign-role --scope, and POST /api/auth/roles/assign with a scope. Before, such a grant authorized every REST route, the /ws/events stream and the hangar_* management tools as if it were global.

hangar_call and the continuation tools are unchanged.

If a principal needs fleet-wide access, bind its role at global scope. That includes listing, starting and stopping servers, and reading configuration. It also includes managing groups, discovery or credentials.

auth:
  role_assignments:
    - principal: "group:platform-engineering"
      role: developer
      scope: global        # was: tenant:platform

Global grants, and deployments with auth disabled, behave exactly as before.

tool-call metrics are no longer counted twice

Every call through hangar_call or a projected tool was observed once by the invocation handler and again by the metrics event handler. mcp_hangar_tool_calls_total, mcp_hangar_tool_call_errors_total and the count of mcp_hangar_tool_call_duration_seconds now move by one per call, so every call-volume series halves after upgrading. Review any alert or dashboard threshold tuned against the doubled series. The latency histogram now holds only the upstream round trip; the dropped observation also timed policy checks and cold starts. The ToolInvocationError series stops growing.

Four metrics that nothing has written to since 2.10.0 are gone as well, each a # TYPE header with no sample: mcp_hangar_behavioral_deviations_total, mcp_hangar_tool_schema_drifts_total, mcp_hangar_detection_rule_matches_total and mcp_hangar_enforcement_actions_total are no longer exposed. Drop any panel or rule that reads them.

a scheme-less OTLP endpoint now uses TLS

The trace exporter and the audit log exporter both honour the standard OTLP environment now, so an endpoint such as collector:4317 follows the OpenTelemetry SDK default and connects with TLS. It used to be plaintext.

observability:
  tracing:
    otlp_endpoint: http://collector:4317   # was: collector:4317

OTEL_EXPORTER_OTLP_INSECURE=true (or the per-signal OTEL_EXPORTER_OTLP_TRACES_INSECURE / OTEL_EXPORTER_OTLP_LOGS_INSECURE) does the same. https:// always uses TLS, and the default http://localhost:4317 stays plaintext.

audit records really reach the OTLP endpoint

Audit records for tool calls and server state changes used to be handed to the OpenTelemetry API's placeholder and dropped. Hangar now registers its own logger provider and exports them, so a configured OTLP endpoint receives the logs signal as well as traces. Point it at a collector that accepts logs; one that only ingests traces will reject the batches, counted in mcp_hangar_otlp_audit_export_failures_total.

To keep exporting traces but not audit records:

observability:
  audit:
    enabled: false

or MCP_AUDIT_EXPORT_ENABLED=false, which wins over the file. Switched off, no audit records are exported and none are written to the structured log either: the switch selects the null exporter, so the records are never built. That is the same visibility 2.18.x had, since it dropped them. The structured log carries them only when export is on and there is no usable logger provider — no OpenTelemetry SDK installed, or a provider already shut down — where they appear as audit_event lines.

The in-process audit trail and the compliance feed selected by MCP_COMPLIANCE_FORMAT are unaffected either way.

the domain_event log line carries only identifiers

LoggingEventHandler wrote the whole event into every line, so the structured log held each event's identity context and its free-text fields at INFO. The line now carries event_type, event_id, mcp_server_id, tool_name, error_type, tenant_id and correlation_id. The full event moved to a domain_event_detail line at DEBUG. A consumer that parses event payloads out of INFO or WARNING lines must enable DEBUG or read the event store.

recorded values are truncated

Span attributes are cut at 256 characters, OTLP audit attributes at 256, structured-log strings at 2048, and a domain event's free-text fields, such as error_message and reason, at 4096. A cut value ends with …[truncated N], except span attributes, which the OpenTelemetry SDK cuts without a marker. Set MCP_SPAN_ATTRIBUTE_LENGTH_LIMIT, MCP_AUDIT_ATTRIBUTE_LENGTH_LIMIT, MCP_LOG_FIELD_LENGTH_LIMIT or MCP_EVENT_TEXT_LENGTH_LIMIT to change a limit. A domain event is cut once, when it is constructed, so the event store, /ws/events, the audit trail and the logs all hold the same value.

Upgrade to 2.18.0

adopting the enforcement init now writes

Nothing is required. An existing configuration is served exactly as it was in 2.17.1: no identity is declared, so a stdio caller stays anonymous, and no pins are written, so no tool is checked against one. What follows is how to adopt what the new init writes, on a configuration you already have.

Pin what your servers serve

mcp-hangar pin --check     # what does the file claim, and what do the servers serve?
mcp-hangar pin --write     # adopt what they serve now

--write rewrites the file through PyYAML, which keeps values but not comments or key order, and leaves the previous file beside it as <config>.bak. Bare mcp-hangar pin prints the same digests if you would rather paste them into a file you maintain by hand.

digest_enforcement already defaults to block, so a pinned tool whose schema or description changes is refused from the moment you write the pin. That is the point, and it is also the one thing to know before writing pins on a busy gateway: an upstream that changes a tool stops serving it through Hangar until you re-pin.

declaring who calls you over stdio

tool_access.mode: front_door serves the upstreams' own tool names instead of Hangar's hangar_* API. It is fail-closed on identity, and identity arrives through HTTP middleware that a stdio process never enters -- so over stdio it needs a declared caller, or it serves an empty list:

tool_access:
  mode: front_door
auth:
  stdio:
    principal:
      id: local-user
      tenant_id: local
      roles: [viewer]     # read-only; `[]` projects no hangar_* tools at all

Nothing is checked here and nothing can be: a stdio server is not listening on anything, so the process that spawned it is the trust boundary (ADR-026). Over HTTP the block is ignored and the credential channel is unchanged.

Per-tenant pins for the declared tenant_id become matchable, so a configuration that previously refused to boot with PinnedToolsNeedAnIdentityError for that tenant now starts.

the CLI stopped printing mcp_server

The CLI no longer prints mcp_server at people: help text says "MCP server", and init --mcp_servers is now init --servers. The old flag still parses, so scripts keep working. If you assert on --help output, it changed.

a config generated by an older init

Two more things moved. init no longer writes a health_check: block -- nothing has ever read it, so every generated config logged unknown_config_key and tuning interval_s tuned nothing. And the servers it configures now use their real npm packages where no official PyPI distribution exists; if you have a config from an earlier init with command: [uvx, "mcp-server-filesystem"] or mcp-server-memory, those never resolved -- replace them with [npx, -y, "@modelcontextprotocol/server-filesystem"] and [npx, -y, "@modelcontextprotocol/server-memory"].

Upgrade to 2.17.1

A patch release, but four settings that previously did nothing now do something, and two payloads an operator may be reading changed shape. Nothing here needs a configuration change; all of it is worth knowing before the upgrade.

The retry: block starts applying (#1162)

The section was parsed, merged and logged at startup, and then no code read it back: every hangar_call used the attempt count from the tool argument alone, and backoff, initial_delay, max_delay, retry_on and jitter were always the built-in defaults. If your config.yaml carries a retry: block, the gateway now honours it -- a default_policy.max_attempts: 5 means calls that used to be attempted once are attempted up to five times, against the upstream.

Two rules that are new and worth reading if you set the block:

• The configured attempt count is a ceiling. A caller's max_attempts can lower it, never raise it.
• Refusals are never retried, whatever retry_on says: access and egress denials, approval-required holds, authn/authz failures, rate limits and validation errors. retry_on is matched as a substring of both the error type and its message, so a broad entry such as Error used to cover every denial there is -- and an approval gate re-driven once per attempt holds a human decision open that many times.

A deployment with no retry: block is unchanged: one attempt unless the caller asks for more.

HTTP upstreams retry the statuses they always claimed to (#1163)

max_retries, retry_backoff_factor and retry_status_codes on an http: block described behaviour that did not exist -- only connection failures were retried, by the underlying library, on a fixed backoff. A remote upstream returning 502/503/504 during a rollout is now retried under your configured backoff instead of failing the call. mcp_hangar_http_retries_total starts producing samples, so the Grafana panel shipped with the Helm chart draws for the first time.

If you deliberately relied on a fast failure from a remote upstream, set max_retries: 1.

Tool arguments are redacted in the audit trail (#1168)

ToolInvocationRequested carried the caller's arguments verbatim into the event store and the /ws/events stream, so a secret passed to a tool sat in SQLite/Postgres for the retention of the log and was served to every audit:read holder. Arguments are now redacted the same way the approval record has redacted them since 2.10 -- by key name at every depth, and by value shape.

If you query the event log for argument values, they are [REDACTED] from this release on. The event gains arguments_hash (SHA-256 over the raw payload, taken before redaction), which is what to correlate on instead: it still tells you whether two calls carried the same arguments. Rows written before the upgrade are unchanged.

The config export no longer contains secrets (#1169)

POST /config/export and GET /config/diff returned a subprocess server's env verbatim, and the auth, discovery and other pass-through sections with ${VAR} references already resolved -- readable by any principal holding config:read. Both are redacted now.

An export is therefore no longer a lossless round-trip of secrets, which was already true of the remote auth block and is now true of env too. If you feed the export back in, restore the secret-bearing values from your secret store; the structure and every non-sensitive value are unchanged.

A withdrawal now reaches the whole fleet (#1165)

POST /admin/tools/{server}/{name}/withdraw wrote to the memory of the replica that served it. On more than one replica the withdrawn tool, prompt or resource stayed listed and callable everywhere else, and a rolling restart lifted the withdrawal entirely. It is recorded now, applied on every replica within one event-tail interval, and rebuilt at startup.

Nothing to do, but two consequences: withdrawals made before this upgrade are not recovered (there was no record to read), so re-issue any that should still be in force; and a withdrawal now outlives the process, so lifting one means calling .../restore rather than restarting.

Smaller changes

• ui_resources: is a known configuration section, so HANGAR_CONFIG_STRICT=1 no longer refuses a config that declares it (#1167). If you worked around this by leaving strict mode off, you can turn it back on.
• The first call to a pinned tool through a group after a gateway restart is no longer refused as unverifiable (#1166).
• A group member's own tools: policy is enforced for calls routed through the group (#1164). A deny_list on a member that silently did nothing now denies -- if a call starts being refused after the upgrade, that policy is why, and it was doing nothing before rather than being absent.
• mcp_hangar_discovery_conflicts is removed. It was registered and never incremented, so it only ever exposed a TYPE header with no samples; nothing can have been alerting on it. mcp_hangar_build and mcp_hangar_process_start_time_seconds now carry values (#1163).

Upgrade to 2.17.0

A prompt or resource can be withdrawn at runtime, and the endpoint says which (#1137)

POST /api/admin/tools/{server}/{name}/withdraw and .../restore accept a kind in the JSON body -- tool (default), prompt or resource. Until now both endpoints wrote the tool overlay whatever was named: withdrawing a prompt returned {"withdrawn": true}, left the prompt served, and -- if a tool shared the name -- silently withdrew that tool instead. Prompts and resources could be withdrawn from config.yaml and from nowhere else.

Three things to check:

• A caller that sends no kind is unchanged: it withdraws a tool and the response carries the same keys as before, plus kind.
• An unrecognised kind is a 400 invalid_kind and writes nothing. There is no fallback to tool, because that fallback is the bug.
• A resource is named by its upstream URI (demo://doc/1, file:///data/x.txt) -- the form withdrawn_resources: reads -- not the projected hangar://<upstream>/<uri>. The {name} path segment now accepts slashes so such a URI can ride it.

ToolWithdrawn and ToolRestored carry kind at schema version 2. Rows written by an earlier gateway have no kind and replay as tool, which is all they could have been. An event consumer that pins schema versions should accept 2.

Unloading a server retires its per-tenant policies (#1138)

Hot-unloading an mcp_server removed its own tool_access policies (every kind, since 2.13.0) and left its tool_access.member.<tenant> overrides behind. A server loaded later under the same id inherited them: a tenant its predecessor denied stayed denied, and a stale per-tenant allow_list restricted the successor to whatever it named.

After this upgrade an id that comes back is a clean id. If a deployment relied on the inherited rules -- unload, reload under the same id, expect the old per-tenant scoping to persist -- that scoping must now be declared on the new server's own tool_access: block. There was never a documented way to depend on it; it is called out because the effect is a widening.

The handed-out resource_link map is bounded per tenant (#1139)

The front door remembers every resource_link it hands out so that following one resolves. That memory was one process-wide map capped at 4096 entries and evicted oldest-first across all tenants: one tenant handing out 4096 links flushed everyone else's, and the victims' resources/list silently shrank.

Each tenant now has its own 4096-link budget, evicted only by that tenant's own traffic, and the number of tenant maps is itself capped at 1024 (least recently used first). Two new surfaces:

• mcp_hangar_resource_links_evicted_total{reason} -- tenant_cap or tenant_map_cap. Until now an eviction left no record at all. No tenant label, for the same cardinality reason as mcp_hangar_empty_projection_total.
• resource_links.max_per_tenant in config.yaml -- default 4096, so nothing changes on upgrade. Raise it when the counter climbs for tenants that legitimately hand out more. A value that is not a positive integer refuses to start; the key is known to validate_config, so a typo is reported and refused under HANGAR_CONFIG_STRICT=1.

resource_links:
  max_per_tenant: 8192

The map is still per-replica and in-memory; restart and cross-replica behaviour are unchanged.

uv.lock now tracks the release version (#1151)

Cosmetic, but it stops a recurring nuisance: the lock's own mcp-hangar entry lagged pyproject.toml by however many releases had happened since the last dependency bump, so every fresh uv sync dirtied the tree. From this release the version bump lands in the lock in the same release commit. If you carry a fork with your own uv.lock, expect a one-line conflict on version = "2.17.0".

Upgrade to 2.16.0

A header selector stops matching a header nobody validated (#1053)

SEP-2243's Mcp-Param-* check is fail-open in the SDK by design: when the pre-dispatch listing that resolves the called tool's schema fails, validation is skipped and the call is dispatched anyway. Until now an MCPEgressPolicy headers.allow / deny / requireApproval selector matched on such a request regardless, which is SEP-2243's own failure mode -- route on one value, execute another -- inside the policy engine.

A request whose validation was skipped now satisfies no header selector. It falls through to the tool rules and the policy default, exactly as a handshake-era request already did, and the verdict reason says the rules were not consulted rather than that none matched.

This can change a verdict. A call that a headers.allow selector used to permit against a defaultAction: Deny policy is now denied by that default when its headers could not be validated; a call a deny selector used to refuse now falls to the tool ladder. Nothing changes for a policy that writes no headers.* rules. Watch mcp_hangar_param_header_validation_skipped_total{reason="listing_failed"} -- if it is flat, no request on your gateway is affected.

Optionally, refuse such a call outright instead of serving it unvalidated:

headers:
  param_validation:
    required: true      # default: false

Off by default, because it turns an upstream listing failure into a client-visible refusal (-32020) for every call carrying header parameters. See ADR-025.

An enforced egress refusal now produces an event and a metric (#1128)

Enforce mode refused calls without recording anything: no event, no metric, and a debug-level log line carrying the generic caller-facing message. A refusal now publishes EgressPolicyEnforced, increments mcp_hangar_egress_policy_enforced_total{mcp_server,action,rule_kind} and logs at warning with its reasons.

Two consequences worth planning for:

• A SIEM export or event consumer sees a new event type. Its fields mirror EgressPolicyViolationObserved plus the applied action, so a handler written for the Audit-mode event works unchanged on this one.
• Log volume rises on a deployment that refuses calls routinely. The batch fault barrier now logs a refusal as batch_call_refused at warning; upstream failures keep the debug level they always had.

There is no flag to restore the silence. Alert on the new counter rather than on the absence of one -- and note that a refusal recorded by a 2.15.0 or earlier gateway is in no event stream at all, so an export that spans the upgrade is not comparable across it.

Every egress verdict names the policy that produced it (#1129)

Decision, EgressPolicyEnforced, EgressPolicyViolationObserved, EgressPolicySet and both refusals now carry policy_id: a content hash of the compiled rules (sha256: plus 16 hex digits), so a verdict and a policy change join on a value rather than on adjacent timestamps. GET /api/mcp_servers/{id}/l7_policy returns it as policyId beside the rules.

The id covers the policy's mode, so flipping Audit to Enforce yields a different id for the same rules. That is deliberate -- two verdicts either side of the flip did not come from the same policy.

Breaking, for code embedding Hangar: PolicyEvaluationResult.policy_id and the policy_id= argument of its allow() / deny() constructors are removed. The field was documented as an audit identifier and was never set by anything, so an implementation passing it was passing a value nobody read; the call now raises TypeError. Nothing in a deployment that does not implement IToolAccessPolicyEnforcer in Python is affected.

Approval records redact nested secrets (#1130)

Argument redaction in the approval gate matched sensitive key names at the top level only, so {"config": {"password": "..."}} -- and the same key inside a list of records -- was persisted to the approval store and served to every approval:read holder verbatim. The check now applies at every level, a sensitive key hides its whole subtree, and anything past the walk's depth cap is replaced rather than passed through.

The shared redactor also learned URL userinfo credentials (postgres://user:pw@host/db), which applies everywhere it runs -- the log pipeline and stderr capture included. Only the userinfo is replaced; the scheme and host stay readable.

Two things this does not change: arguments_hash is still computed over the raw arguments, so the dispatch-time substitution check is unaffected; and approval records written before this release are not rewritten. A record persisted by an earlier gateway may still contain a secret -- treat the store's history accordingly.

Upgrade to 2.15.0

A merged tool-access policy narrows to what it always said (#1106)

Merging a narrower scope onto a broader one could widen access. A tool denied at the server scope came back allowed as soon as a group scope declared an allow list -- for example a server-level allow: ["*"], deny: ["drop_*"] merged with a group-level allow: ["*"] allowed drop_db. merge() dispatched on which lists were populated and each branch rebuilt part of the deny > approval > allow > default ladder from the lists its own condition named, dropping the rest.

A tool that has been callable may stop being callable after this upgrade. That is the correct behaviour -- the policy always said so -- and it is still a surprise if you have been relying on the widened set. Check the effective policy for each identity before upgrading:

hangar tools list --identity <principal>

Anything that disappears was denied by one of the scopes all along.

Deeply nested tool arguments are denied instead of crashing (#1102)

evaluate() could propagate a RecursionError out of the policy plane: JSON nested deeper than the encoder can walk (about 992 levels on CPython 3.11, ~7 KB of payload) raised instead of returning a verdict. maxPayloadBytes was no defence, because the size check reads the string the serializer produces.

In Enforce the call was refused before, but by accident of ordering rather than by decision -- no EgressPolicyDeniedError, no observation, no audit entry. It is now a DENY with a reason, and the refusal is recorded. In Audit the exception aborted the call before dispatch, which is the opposite of what ADR-013 documents as the safe adoption path; audit now records and lets the call through, as it always should have.

If you have a legitimate caller sending deeply nested arguments, it will now see a denial with arguments could not be serialized for policy inspection rather than an unattributed failure. Flatten the payload or raise nothing -- there is no flag to restore the old behaviour, because the old behaviour was a crash.

The front door advertises two more capabilities (#1027, #1026)

front_door now serves subscriptions/listen and completion/complete for prompt arguments, so clients see resources.subscribe, the three listChanged flags and completions where they previously saw nothing. This is additive -- no configuration changes -- but a client that adapts to advertised capabilities will start using surfaces it skipped before.

Outside front_door mode the subscriptions/listen handler is now withdrawn rather than advertised: nothing published to it, and the modern protocol derives all four subscription flags from its presence. A client that called it in registry mode and got an empty stream now gets method-not-found, which is the honest answer.

If you built the demo images by hand (#1095, #1097)

docker/Dockerfile.{fetch,filesystem,memory,sqlite,math} and examples/discovery/Dockerfile.test-provider are gone. Nothing in the repository built them, and scripts/containers.sh referenced images no command produced while sending every failure to /dev/null.

The replacements are the official servers from modelcontextprotocol/servers; the guide says which to use for each of the three wiring shapes. There is no official SQLite server -- the upstream one is archived -- so if you ran that image, it has no direct replacement.

Upgrade to 2.14.0

A front-door tool can disappear, with no config change (#1056)

SEP-2243 makes it a client-side MUST: a client drops any tool whose x-mcp-header annotations are invalid. Hangar forwarded the upstream definition verbatim, so it advertised such a tool and counted it as surface delivered while every conforming client silently dropped it.

Since this release the tool is withheld at the projection instead: absent from tools/list, and -32601 on the call. If an upstream ships a malformed annotation, a tool that was listed yesterday is gone today, and nothing in your configuration changed. The reason is in the log (tool_withheld_invalid_x_mcp_header, naming the tool and what was wrong) and on mcp_hangar_projection_withdrawals_total{reason="invalid_x_mcp_header"}. Scrape that counter after the upgrade if you want to know before your users do.

An annotation is valid only when it sits on a property reachable through a pure properties chain, names an RFC 9110 token, is on an integer/string/ boolean property, and is unique across the schema. The fix belongs upstream; Hangar does not edit a projected schema, because that would move its digest and break every pin.

A legacy-era header/body mismatch now answers -32020 (#1051)

The handshake-era front door refused a mismatch with -32600 (InvalidRequest) while tasks/* used HEADER_MISMATCH (-32020) for the same class of failure. Both now answer -32020; the HTTP status stays 400 and nothing that was refused before is accepted now. Update any client branch or log query keyed on -32600 for this case. Modern-protocol clients are unaffected — the SDK ladder already answered -32020 there.

Eight metrics appear on /metrics for the first time (#1059, #1049)

These were defined and incremented on the live path, and never registered, so they were absent from every scrape — indistinguishable from a feature that was never built:

mcp_hangar_approval_requests_total, mcp_hangar_approval_deliveries_total, mcp_hangar_approval_decisions_total, mcp_hangar_egress_policy_violations_observed_total, mcp_hangar_projected_tools, mcp_hangar_empty_projection_total, mcp_hangar_param_header_validation_skipped_total, mcp_hangar_projection_withdrawals_total.

A panel or alert that has been quietly returning no data starts returning rows. The three approval counters have been dead since 2.10.0, and the queries in the observability guide were written against them — those work now. The Audit-mode egress counter is the signal ADR-013 calls the safe adoption path for an MCPEgressPolicy, so an Audit rollout is measurable for the first time.

Nothing to change; expect graphs that were flat to start moving.

Optional: govern what an upstream may ask a client to expose (#1057)

A new header_exposure: block sits beside tool_projection: on an mcp_server or a group. It is off unless you configure it, and its default action is warn, so adopting it does not change what any client sees until you say otherwise.

mcp_servers:
  payments:
    header_exposure:
      deny_annotated: ["*token*", "*secret*", "*password*", "api_key", "*_key"]
      on_violation: warn          # warn | withdraw | refuse_boot

refuse_boot refuses to serve the catalogue, not to start the process: the violation is only knowable after discovery. An unknown on_violation is refused at parse rather than defaulted.

Optional: Mcp-Param-* selectors in an egress policy (#1058)

L7Policy can now select on SEP-2243 Mcp-Param-* headers with the same glob precedence as the tool-name rules. A request whose MCP-Protocol-Version predates mandatory header-body validation never satisfies such a selector.

In this release the selector is reachable through the REST channel (PUT /api/mcp_servers/{id}/l7_policy) only. The matching MCPEgressPolicy CRD field ships separately in the operator (operator#160), so a GitOps deployment cannot express it yet.

Upgrade to 2.13.1

One thing can stop a gateway that started yesterday, and it is a configuration refusal rather than a runtime failure. Everything else here is a control that begins working; read it if you run groups, because things that were served may stop being served -- which is the point.

approval_list under access: now refuses the load

A config that declares approval_list inside an access.prompt or access.resource block is refused when it is read:

ConfigurationError: access.resource.approval_list on mcp_servers.docs_server asks
for a human approval hold that no resource path performs: the gate runs on tool
calls only …

2.13.0 accepted the key, registered it, and enforced nothing -- the item was listed and served with no hold -- while the startup reachability check read the same key and refused the boot over it if no approval gate was wired. One configuration, fail-open at request time and fail-closed at boot.

Find it before you upgrade:

grep -rn -A5 "access:" /etc/mcp-hangar/ | grep -n "approval_list"

Replace it with deny_list, which hides the item from every listing and answers a fetch the way a nonexistent one is answered. A hold on a fetch is decided against in ADR-024; tools.approval_list is unaffected and keeps holding tool calls.

The mirrored half is a relief rather than a step: a [prompt] / [resource] policy no longer demands an approval gate at startup, so a deployment that set approvals.enabled: false and carried such a policy stops being refused for a gate it does not need.

A group's policies and withdrawals now enforce

A group and its members are one logical server, and every governed lookup had two possible keys -- the group id and the member id -- with different surfaces picking different ones. Four declarations that parsed and did nothing now do what they say:

So after the upgrade a tenant may see fewer prompts and resources, and a group-routed tool call may be refused with ToolWithdrawnError or ToolDigestMismatchError where it previously ran. Every one of those refusals is a control an operator had already written. If you want the old behaviour for a particular item, remove the declaration rather than leaving it in place expecting it to be ignored.

Check what a group actually enforces before upgrading by reading the same declarations back:

grep -rn -A10 "mode: group" /etc/mcp-hangar/ | grep -E "access:|tool_projection:|withdrawn"

Human approval works again in front_door

Every approved tool call was refused at dispatch in front_door mode -- Approval no longer valid at dispatch: tool is no longer allowed by policy -- because the post-hold re-check resolved policy without the caller's tenant, which is the fail-closed missing-identity branch. If you turned approval_list off to get past it, you can turn it back on.

New: ui_resources:

SEP-1865 ui:// resources can now be allowlisted and consented to instead of only denied. A deployment that does not configure it sees no change -- every ui:// resource stays denied, as it has been.

ui_resources:
  tenants:
    "tenant:a":
      allowlist: ["ui://reports/"]

Both gates must pass: the allowlist, and a human consent on resources/read delivered through the approval gate. A deployment with no approval gate wired cannot deliver a ui:// resource at all -- consent is mandated by the spec and cannot be switched off from the file. See ui_resources.

Upgrade to 2.13.0

No migration steps are required. Two changes are worth knowing about before you upgrade, both in front_door mode only -- egress mode is untouched.

An upstream's prompts and resources are now served

The gateway previously projected tools and nothing else: an upstream's prompts were unreachable, and resources/list answered with only the resource_links a tenant had already been handed. Both surfaces are now proxied in full, so after the upgrade a tenant sees prompts and resources it could not see before -- from its own upstreams only; cross-tenant isolation is unchanged.

If you want either surface restricted, they are governed by the same policy surface tools use, keyed by kind:

mcp_servers:
  docs_server:
    access:
      prompt:   {deny_list: ["draft_*"]}
      resource: {allow_list: ["docs://*"]}
    tool_projection:
      withdrawn_prompts: [retired_prompt]
      withdrawn_resources: ["demo://gone/1"]

allow_list / deny_list mean exactly what they mean for tools, and an undefined block leaves that kind unrestricted -- the same rule tools have always followed. A resource is matched by its upstream uri (demo://doc/1), not the projected form below.

approval_list is not among them. The human approval gate runs on tool calls and nowhere else, so an approval-listed prompt or resource is served immediately; 2.13.0 said otherwise and was wrong. 2.13.1 refuses the key at load rather than accepting it silently -- see Upgrade to 2.13.1. Use deny_list for anything a prompt or resource surface must not hand out.

resource_link uris carry the owning upstream

A projected resource uri is now hangar://<upstream id>/<upstream uri>, so two upstreams using the same scheme -- or the same uri -- no longer collide. The gateway translates back on resources/read, so nothing changes for a client that uses the links it is handed.

This is not a migration step: the link map lives in a replica's memory, so an upgrade re-issues every link regardless. It matters only if you have tooling that parses these uris rather than passing them back verbatim.

Upgrade to 2.12.0

truncation.cache_driver: redis now fails closed (#1007)

If Redis cannot actually serve the continuation cache -- the redis package is missing, the URL does not parse, or the server cannot SETEX (a Sentinel listen port) -- the gateway now refuses to start instead of silently falling back to the per-replica memory cache. If your deployment booted with cache_driver: redis before this release, Redis was never actually in use; either fix the connection (the image now ships the redis extra, #1008) or set cache_driver: memory explicitly. A truncated response no longer carries a continuation_id unless the full payload was actually stored.

pip install mcp-hangar[redis] provides the client; it is deliberately not part of the base install or the full extra.

Upgrade to 2.11.0

the unused-surface sweep (#969)

Nine verified-dead surfaces left over from the factory cut are gone. None had a caller in src/; if you imported them in your own code, the replacements are listed:

• HangarError / Rich* errors, factories, ErrorClassifier (mcp_hangar.errors, also re-exported from the package root). The live hierarchy is mcp_hangar.domain.exceptions; is_retryable stays and keeps matching timeout/connection-style exceptions by pattern.
• ProgressTracker / create_progress_tracker (mcp_hangar.progress, module deleted). MCP progress notifications are a different, live feature.
• HealthEndpoint / HealthCheck / get_health_endpoint (mcp_hangar.observability). The live probes are the /health/* routes; event-store durability get/set remains in observability.health.
• mcp_hangar.domain.bundles (starter/developer/data bundle catalog). Hot-loading from the registry (hangar_load) is the live path.
• AuditService (domain.services). Live audit is AuditEventHandler over IAuditRepository.
• Tenant/catalog/package exception cluster (TenantNotFoundError, QuotaExceededError, CatalogItemNotFoundError, PackageVerificationError, ...) plus McpServerEntry and CatalogItemId. There is no catalog API these could describe.
• HangarLoadResult / HangarUnloadResult and REST serialize_tool_info / serialize_health_info; the tools return dicts and REST serializes via .to_dict().
• Metrics helpers init_metrics, timed, record_* for detection/behavioral features that never shipped producers.
• initialize_runtime / shutdown_runtime (bootstrap.runtime) and the trace_tool_invocation decorator; create_runtime, init_tracing and get_tracer stay.

Upgrade to 2.10.0

config.yaml warns about a key nothing reads

Unknown keys were kept and ignored at every level. They are now reported, with the offending key and the allowed set named:

auth has unknown key(s) ['enabledd']; allowed keys: ['allow_anonymous',
'api_key', 'enabled', 'oidc', 'opa', 'rate_limit', 'role_assignments', 'storage']

This release warns and starts anyway. Refusing is correct -- a misspelled auth key is a gateway that believes it enabled authentication -- and is also a breaking change for anyone carrying a stale key, so it gets a release of notice instead of arriving in a patch.

• HANGAR_CONFIG_STRICT=1 refuses now, which is what to set in CI.
• the default becomes refusal in 3.0.0. Any unknown_config_key warning in your logs today is a config that will not load then.

Checked: top-level section names, the direct keys of each section, and the keys of an mcp_servers.<id> spec. Not checked: anything deeper. That is where a single reader exists to enumerate from -- below it the keys live in around twenty modules, and a schema hand-copied from twenty readers drifts into rejecting valid configuration, which is worse than accepting a typo.

New: mcp-hangar config check [path]. Answers the same question without starting a gateway, and is always strict. Exit 0 clean, 1 unknown key, 2 the file is missing or is not YAML. It defaults to $MCP_CONFIG, then config.yaml.

$ mcp-hangar config check config.yaml
FAIL config.yaml: 1 key(s) nothing reads:

  mcp_servers.math has unknown key(s) ['commandd']; allowed keys: [...]

Upgrade to 2.9.0

Drop-in for every deployment that runs the gateway. Nothing about a served Hangar changes. One tool starts working, and a Python API that no shipped code path ever executed is gone.

hangar_load can now succeed, and wants uvx or npx on PATH

Hot-loading has been enabled by default and unable to complete since it shipped: bootstrap handed its resolver a runtime table with every entry False and an empty installer list, so every call answered

{"status": "failed", "message": "No compatible package found (missing runtime?)",
 "warnings": ["Available runtimes: []"]}

It now resolves pypi packages through uvx and npm packages through npx, and reports availability from the installers rather than from a hardcoded table.

The runtime has to be on the gateway process's PATH, which is the part to check before expecting different behaviour:

• the published container image carries neither uvx nor npx, so hot-loading still fails there — now with a message naming what is missing rather than an empty list. Derive your own image from ours and add uv, Node, or both if you want it working in a container.
• running from a pip install, install uv for PyPI-published servers and Node for npm-published ones. Either alone is fine.

oci and mcpb packages remain unloadable, deliberately: OCI needs a container runtime the image does not ship, and mcpb has no defined install path. Both are now reported unavailable instead of being selected and then dropped.

Nothing to do if you do not use hangar_load. hot_loading.enabled: false keeps the tool switched off.

The fastmcp_server factory stack is gone

Removed from mcp_hangar.fastmcp_server: MCPServerFactory with its builder() and create_asgi_app(), MCPServerFactoryBuilder, HangarFunctions, ServerConfig, the thirteen Hangar*Fn protocols, and the ASGI combiners create_health_routes, create_combined_asgi_app and create_auth_combined_app.

Nothing about a running Hangar changes. No shipped code constructed any of it. serve --http builds its MCP server in mcp_hangar.server.bootstrap and its ASGI app in mcp_hangar.server.lifecycle.mcp_app_for_serving, and has never gone through the factory. The two assemblies had drifted far enough to prove it: the factory mounted flat /health and /ready, while a running Hangar serves /health/live, /health/ready, /health/startup and /metrics.

Keeping a second construction path that looked serviceable is what made four bugs possible (#592, #594, #595, #596): each was a capability wired into the factory, which made it appear wired and shipped it dead.

If you were embedding through the factory there is no drop-in replacement, because the factory was never how the product ran. Either run the gateway (mcp-hangar serve --http) and drive it over MCP or the REST API, or call the composition root the CLI itself uses: server.bootstrap to build and register, lifecycle.mcp_app_for_serving for the ASGI app, and server.api.middleware.create_auth_enforced_app to apply the same authentication. Those are tested on every PR and are what a released Hangar executes.

HANGAR_SERVER_NAME is unchanged and still exported from mcp_hangar.fastmcp_server. The v0.4.0 note further down names the factory as the successor to setup_fastmcp_server(); it describes what that release did and stays as history.

Upgrade to 2.8.0

Two things can break a build rather than a deployment: an extra that no longer exists, and a bundled monitoring stack that has moved to the Helm chart.

pip install mcp-hangar[containers] now fails

The containers extra is gone. It installed testcontainers for a test tier that never ran — those tests were gated behind --run-containers / --run-slow and no CI job, Makefile target or script ever passed either flag, so every one of them reported skipped on every run. Nothing in the shipped package imported it.

Drop [containers] from your install line. If you depended on testcontainers yourself, depend on it directly.

The bundled compose monitoring stack is gone

monitoring/ and docker-compose.monitoring.yml are removed from the repository. The four Grafana dashboards and the 30 Prometheus alert rules ship with the Helm chart instead: dashboards.enabled renders them as sidecar-labelled ConfigMaps, prometheusRule.enabled renders a PrometheusRule.

Instrumentation is untouched — /metrics, tracing and the OTLP exporter are unchanged; only bundled config moved. There is no one-command local Grafana any more. If you were running it, either use the chart or keep a copy of the compose file from the 2.7.0 tag.

The published container runs Python 3.14

pip install still supports 3.11 through 3.14, and 3.14 is now a required CI citizen rather than an advisory one. Relevant only if you build on top of our image and pin something against the interpreter version.

Three unused symbols left the application layer

CallbackAlertSink and LogAuditStore are gone from mcp_hangar.application.event_handlers, and detect_runtime_availability with its IRuntimeChecker protocol from mcp_hangar.application.services. None had a caller outside this repository's own tests.

• CallbackAlertSink — production get_alert_handler() builds a LogAlertSink. To capture alerts in your own code, implement the ABC:

  from mcp_hangar.application.event_handlers.alert_handler import Alert, AlertSink
  
  class CapturingSink(AlertSink):
      def __init__(self) -> None:
          self.alerts: list[Alert] = []
  
      def send(self, alert: Alert) -> None:
          self.alerts.append(alert)

• LogAuditStore — it could not have served as an audit store: query() raised NotImplementedError, because a log sink cannot answer a query. Write the sink you want against the AuditStore ABC, or use the OTLP exporter path (OTLPAuditEventHandler / IAuditExporter), which is built for shipping audit records off the box.

• detect_runtime_availability — no replacement, deliberately. Ask the installer you care about (is_runtime_available()) and construct the RuntimeAvailability yourself, which is all the removed function did, in a fixed order, for a list it did not validate.

AlertSink, Alert, LogAlertSink, AlertEventHandler, get_alert_handler, AuditRecord, AuditStore, InMemoryAuditStore, AuditEventHandler, get_audit_handler, PackageResolver and RuntimeAvailability are unchanged.

Upgrade to 2.7.0

Drop-in for most deployments. Two behaviours change without a config change: approval_channel, which was recorded and ignored, now selects where approvals are notified; and the MCP endpoint stops handing out session ids. Read the approval_channel section if any of your policies set it, and the session section if anything in front of your pods pins traffic.

The MCP endpoint no longer hands out a session id

initialize returns no Mcp-Session-Id, and no request needs one.

A session lived in one replica's memory, so a client that initialized against one pod and called against another was told Session not found -- 13 of 15 attempts through a three-replica Service. Session affinity papered over that and could not fix it: a pin does not outlive its pod, so a rolling restart or a scale-down took the session with it.

The last row is the only one that can surface in a client's logs. There is no session to terminate, so teardown is refused rather than acknowledged.

What this does not change. Nothing about the 2026-07-28 revision, which has no sessions at all and was already served this way. Nothing about session suspension (/api/sessions), which keys on the caller identity from x-session-id or the JWT sid claim and never on the transport. Nothing about authorization, which is per request.

Deployments. Sticky routing is no longer a requirement for a replica set -- see running more than one replica. Existing pinning is now merely unhelpful rather than wrong, so there is no rush to remove it; leave it if you still run an older gateway behind the same ingress.

front_door no longer serves an empty tool list after a restart

Also fixed here, and worth knowing whether it happened to you. In tool_access.mode: front_door, tools/list is the per-tenant projection, and the projection was built from whatever that replica had started. A replica that had started nothing served an empty list to a valid tenant, with no client- reachable way to fix it, and two replicas that had warmed different servers answered the same tenant differently.

A front_door gateway now starts every configured mcp_server at boot, on its own thread so readiness never waits on a backend handshake. A backend that fails to start is logged as front_door_warmup_failed rather than costing the others their projection. egress is unchanged: backends still start lazily on first use.

A consent gate no longer disappears on restart

Fixed, not a migration step — but worth knowing whether it happened to you.

The tool-access-policy store held allow_list and deny_list and nothing else, and the startup replay rebuilt policies from those two fields, assigning over whatever the YAML had already registered. A server with tools.approval_list in its config and any prior policy update over the REST API came back ungated after a restart: the tools it named ran without being held, and the startup check that guards this class saw no approval_list left to demand a gate, so the boot was clean.

The store now persists the approval fields and the replay hands back whole policies. An existing database is widened in place on first open. A row written by an older build carries no approval columns; rather than let that erase a gate one last time, the replay carries the in-force gate forward and logs tap_replay_carried_approval_gate.

Nothing to do. If a gate was lost to this, it is back on the next restart — the YAML declaration was never what went missing. If you keep audit records, calls to approval_list tools between an affected restart and this upgrade ran without a human decision.

approval_channel now routes, and the built-in channel is renamed

approval_channel was documented as a policy's delivery channel and merged carefully across scope narrowing — and dispatched nowhere. One delivery, built from the global approvals.channel, handled every approval whichever policy raised it. A config that set approval_channel: slack on one server and something else on another got one channel, silently.

They now route as written. Check your policies before upgrading: if two servers name different channels and only one adapter is installed, the other now degrades to noop where it previously borrowed the global channel.

The core channel formerly called dashboard is now event_stream. It was named after a management UI that shipped with the Hangar Cloud tier and was archived with it, and it never pushed to that UI anyway — its send wrote a log line while its docstring claimed a WebSocket integration that was never wired. The new name points at the surface that does carry the notification: the ToolApprovalRequested domain event on /api/ws/events.

channel: dashboard still resolves, to the same delivery, and logs approval_delivery_channel_renamed once at boot. No config change is required.

An armed gate now says when nobody is listening

A policy that gates a tool while its channel reaches nothing outside the process — noop, or a vendor name no installed package claims — is now reported at startup:

subsystem_configured_but_unreachable
  subsystem=approval_delivery
  required_by="tools.approval_list on mcp_server:payments (channel 'slack')"
  fail_closed=False

The gateway still starts. The gate is fail-closed by timeout, so what is missing is a signal rather than enforcement, and refusing the boot over a notification channel would trade a degraded notify path for an outage. A deployment that wants the refusal opts in:

approvals:
  delivery:
    required: true

Three metrics land with it — mcp_hangar_approval_requests, mcp_hangar_approval_deliveries and mcp_hangar_approval_decisions, all labelled by channel. See Observability → Approval Gate.

Removed

hangar_approve_prompt, an MCP tool nothing registered, whose docstring pointed at an approvals.channel: mcp_prompt that no builtin or entry point has provided since 2.0. If you were calling it, you were getting a tool not found.

Upgrade to 2.6.0

Not drop-in. Two changes can stop a gateway that works today, and both are the same shape: enforcement that was advertised and did not run now runs. Read the first two sections before you upgrade.

A gateway with authentication off is unaffected by everything here except the boot refusal in the first section.

Per-tenant digest pins now refuse to boot without authentication

A digest pin could only be addressed to a tenant, and the tenant reaches the enforcement path from the authenticated principal and from nowhere else. So on a gateway with auth.enabled: false, where every caller is anonymous and carries no tenant, no pin was ever matched: drift stayed computable and nothing stopped it, while initialize advertised io.mcp-hangar.digest-pinning with all three enforcement modes. The same miss reached the task path, where nothing bound a relayed task to a digest, so the fail-closed re-verification on result retrieval had nothing to check.

Such a configuration now fails the boot, naming the pins it found and the auth setting that makes them unmatchable, rather than serving a guarantee it cannot keep.

Two ways forward. Turn authentication on, so callers arrive carrying the tenant the pins name — or move the pins to the all-tenants block added in this release, which holds every caller including an anonymous one:

mcp_servers:
  payments:
    tool_projection:
      digest_enforcement: block
      pins:                       # every caller, including an anonymous one
        refund: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
      tenant_overrides:
        "tenant:a":
          pins:                   # this tenant only; wins over the block above
            refund: fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210

Both forms can be used together; resolution is narrowest first. A gateway with authentication on is unaffected — its per-tenant pins were being enforced and continue to be.

The hangar_* tools now require permissions

hangar_call authorized every call it dispatched. The other twenty-one hangar_* tools authorized nothing, so with authentication on any valid credential could stop a server, load one, reload the configuration or approve a discovered upstream over MCP — while the same operations over the REST API were refused for the same identity in the same process.

Authorization is now resolved from the tool name, mirroring the REST route that performs the same operation. No permission was invented and no built-in role changed.

Check any API key or token that drives the fleet over MCP. If it worked because MCP asked for nothing, it now needs the role its REST equivalent has always needed. Two combinations are not guessable from the role names:

• provider-admin cannot start, stop, warm, load, unload or reload. It holds mcp_servers:read and neither :write nor :lifecycle, and not config:reload. That is deliberate and it could not perform those operations through the REST API either. An operator key that needs lifecycle wants admin or a custom role.
• developer cannot approve, quarantine, trigger discovery, rebalance a group or read metrics. It holds fleet read, write and lifecycle and none of those.

A refusal names the permission it wanted, so the log says what to grant:

Not authorized to call 'hangar_stop': mcp_servers:lifecycle permission required

hangar_call is unchanged and still checks tool:invoke for each call in the batch. A gateway with authentication off allows every call exactly as before, so --unsafe-no-auth is unaffected.

metrics:read is now enforced on hangar_metrics. The unauthenticated /metrics scrape endpoint is untouched and remains on the auth skip list.

front_door now shows an operator a control plane

Additive; nothing to do. The mode served flat upstream names and no hangar_* to anybody, so an operator on a front door had no management surface over MCP and had to run a second instance in egress to get one.

A management tool is now listed exactly when the caller is authorized to call it, by the same table and the same authorizer as the section above. An agent principal sees what it saw before; an operator's list grows by what its role permits. Nothing is shown that cannot be called, and a name that is not shown is still answered -32601.

With authentication off the management surface stays empty — stricter than the invoke path on the same gateway, deliberately: a front door that shows an unauthenticated caller nothing today does not start showing it a control plane.

egress is untouched and still serves every caller the whole meta-API. There it is not a management surface that happens to be visible, it is the surface: a client without hangar_call reaches no upstream tool at all. See ADR-022.

New metric: mcp_hangar_projected_tools, a histogram of the tools a front-door tools/list returned, labelled kind=governed|management.

One log line per config-file remote upstream

A remote server declared in config.yaml is outside the SSRF policy: it gets neither the registration check that answers 400 ssrf_blocked on the REST path, nor the connect-time re-resolution added in 2.5.0 that closes DNS rebinding. That exclusion is deliberate — the configuration file is trusted input, see ADR-021 — and has not changed. What changed is that startup says so, once per such upstream, naming it and its endpoint.

Nothing is refused and no request path is affected. To have an endpoint checked, register it through the REST API rather than the file.

Upgrade to 2.5.3

Drop-in from 2.5.2: nothing you wrote has to change. It is six fixes, all of them defects, and two are visible from outside the gateway — a patch number promises they are not, so they are named here.

prompts/list and resources/list now answer -32601

The gateway advertised the prompts and resources capabilities on every deployment and served neither. Nothing hard-coded that claim: the SDK derives each capability from whether its handler is registered, and the framework registers both unconditionally, empty or not.

The cost was not the missing feature, it was the false statement. {"prompts": []} tells a conformant client this server has no prompts, which is a different thing from this gateway does not carry prompts — and nothing on the wire distinguished them. A registered upstream's whole prompt and resource surface was invisible, with no error anywhere.

Who has to do anything. A client that reads the advertised capabilities before calling — which the specification tells it to do — sees no prompts capability and does not call. Nothing to change. A client that calls these methods unconditionally and treats a JSON-RPC error as fatal will now fail where it previously received an empty list; it needs to check capabilities first.

This is derived rather than inverted: when the gateway proxies an upstream's prompts and resources, the capabilities come back on their own.

An upstream's tool catalogue may grow

The gateway never finished the MCP handshake. It sent initialize and went straight to tools/list, skipping the notifications/initialized the lifecycle requires — so every upstream, in every mode, sat permanently mid-handshake. A server is entitled to defer work until that notification arrives, and servers do: against the official reference server, a tool registered in its oninitialized handler was neither listed nor callable through Hangar.

What to expect. If your upstream registers tools on initialization, this release discovers them for the first time and its catalogue legitimately grows after the upgrade. Anything asserting on a tool count will notice. Tool digests are unchanged, so no existing pin moves — including pins on tools that now carry the forwarded title, annotations, execution, icons and _meta, which earlier releases discarded on the way through.

The notification is best-effort: an upstream that mishandles it produces a warning in the log, not a failed start.

Also in this release, with nothing to do

• The gateway identifies itself to upstreams as mcp-hangar at its running version, rather than mcp-registry / 1.0.0 — a product name that has not existed for a long time. If you filter upstream logs or key server-side workarounds on mcp-registry, those match on mcp-hangar from now on.
• A front_door gateway that serves no tools now says why, at WARNING, and counts it under mcp_hangar_empty_projection_total{reason=...}. The three causes — no caller identity, nothing discovered yet, everything filtered by policy — used to be indistinguishable from outside.

Upgrade to 2.5.2

Drop-in from 2.5.1: nothing you wrote has to change. It is five fixes, and the first one decides whether a gateway starts at all.

An auth-enabled gateway could not start on a selected storage backend

persistence.backend made storage one decision (ADR-019), and the branch implementing it handed the API-key, role and tool-access-policy stores out without creating their tables. Startup reached the auth bootstrap and died:

Unexpected error: relation "roles" does not exist

or, with no auth.role_assignments configured to trip that, on tool_access_policies a few lines later. Both backends were affected -- SQLite failed the same way with no such table: roles -- so this was never a PostgreSQL problem, it was the handoff.

That is the configuration more than one replica requires, so the documented multi-replica deployment could not start on 2.5.0 or 2.5.1.

What to do: upgrade. The tables are created when the stores are built, the same way the event store already did it. Nothing to run by hand, and an existing database is unaffected -- the DDL is CREATE TABLE IF NOT EXISTS.

If you worked around it by also naming the backend under auth.storage:, that block is now redundant. It is still legal (naming the same backend twice always was), so you can leave it or drop it.

auth bootstrap-admin refused to run on that same configuration

The command reads the durable backend to claim the initial administrator on, and it consulted only auth.storage.driver -- whose default is memory -- so on a one-storage deployment it answered:

Error: Auth storage driver 'memory' is not durable

Since every /api/auth/** route requires an admin principal, with no carve-out for the first call, there was no other way to mint the first key. It now uses the backend persistence.backend selected.

Related, and worth knowing if you scripted around it: bootstrapping a principal that auth.role_assignments also grants global admin used to fail on a duplicate key. It no longer does.

provider-admin could not deliver an egress policy

/api/mcp_servers/{id}/l7_policy is mapped to policy:write, which is what provider-admin holds and why the role exists -- but the handlers demanded mcp_servers:write on top, which it does not hold. The operator's push answered 403 while the MCPEgressPolicy CR still reported Compiled and BackstopApplied, so a policy enforced its network half and silently dropped its L7 half.

What to check. If you run the operator with an API key scoped to provider-admin, look for L7PushFailed events on your MCPEgressPolicy objects. If you widened that key to admin or developer to get past it, you can narrow it back after upgrading.

MCP_TRUSTED_HOSTS did not reach the MCP endpoint

It governed the REST API only. The MCP endpoint used the SDK's own DNS-rebinding guard, built from the SDK's default bind host, so /mcp answered 421 Invalid Host header to the gateway's own Service DNS name and to every Ingress host -- while the same names were listed in MCP_TRUSTED_HOSTS and accepted by /api/** on the same process.

What to check. If you worked around this by rewriting the upstream Host at your proxy (an nginx upstream-vhost, for example), you can drop that once the hostname is in MCP_TRUSTED_HOSTS. Entries now match with and without a port.

One thing this does not change, and that is worth knowing when you put a Service or an Ingress in front of a replica set: a Streamable HTTP session lives in one replica's memory. A round-robin load balancer can send the second request of a session to a pod that never saw the first. Pin it -- sessionAffinity: ClientIP on the Service, or a client-address hash at the Ingress.

front_door projected no tools to any tenant

tool_access.mode: front_door returned an empty tools/list to every authenticated tenant over Streamable HTTP. The per-request context that carries the caller was handed to the projection handlers and dropped, so the resolver saw no tenant and took its deny-all branch -- correctly, for a caller it could not identify. An empty list is indistinguishable from "no tools configured", which is why this was quiet.

What to check. If you concluded that front-door mode needed a policy you had not written yet, it did not. Per-tenant tool_access.member policies and tool_projection withdrawals now apply as documented in front-door multi-tenant.

Still not working, and not fixed here: a group under front-door mode. Its members expose the same tool names by definition, the flat projection drops any name it finds in two servers, and so the group contributes nothing -- and takes co-located servers sharing those names down with it. Recipe 19 pairs the two; that combination is still open.

Upgrade to 2.5.1

Drop-in from 2.5.0 in the sense that nothing you wrote has to change. Two things are worth acting on rather than reading past.

The connect-time SSRF guard was not surviving a restart

2.5.0 added a second SSRF check on every outbound connection, with the connection pinned to the validated address, so a hostname that passed registration could not be re-pointed at 169.254.169.254, 10.x or 127.0.0.1 before the next tool call. The flag that arms it was never written to the stored server record, so any server rebuilt from that record came back without it -- after every restart, and on every replica that learned of the registration from the shared log rather than performing it.

What that means for a gateway running 2.5.0. If it has restarted since a remote server was registered through the REST API or by discovery, that upstream has been reached with registration-time validation only. The endpoint was still checked once, when it was registered; what lapsed is the re-check that defends against the name being re-pointed afterwards. In a replica set, only the replica that handled the registration ever had the guard.

What to do: upgrade and restart. The guard is restored for servers already in the store -- no re-registration, no edit to the database. One deliberate exception: a stored endpoint that is a private literal keeps 2.5.0's behaviour, because such a row can only have come from discovery reporting a container address, and arming the strict policy over it would refuse an upstream that works today. Re-registering such a server writes the provenance that scopes the guard correctly.

One behaviour to expect once it is armed. Guarding is also pinning: a guarded connection goes to one validated address rather than letting the client walk a multi-address DNS answer, so a dead address behind a healthy name fails the call instead of being skipped. That shipped in 2.5.0; what changes here is how much of your fleet it covers.

A coordination: block with no persistence.backend is now refused

2.5.0 refused a declared cluster on a backend the replicas cannot share, and said nothing when no backend had been selected at all. The outcome is the same either way -- no lease keeper, every replica managing the fleet, every one reporting manages_fleet: true -- so it is now refused too.

Who is affected: a configuration carrying coordination: while storage is still configured through the legacy per-subsystem keys (event_store.driver, auth.storage.driver). That deployment may well share one PostgreSQL, and it was never coordinating through it. It booted on 2.5.0 and will not boot on 2.5.1 until it says where it persists:

this gateway is configured as part of a cluster (`coordination:`), and no
storage backend has been selected. ... Set `persistence.backend: postgresql`,
or remove the `coordination:` block to run this as a single gateway.

Both ways out are in the message. A single gateway that never declared coordination: is unaffected, whatever its storage.

Upgrade to 2.5.0

Nothing changes until you select a storage backend. The release adds persistence.backend and multi-replica coordination, and a configuration that sets neither is unaffected -- omitting persistence keeps per-subsystem storage exactly as it was. That is deliberate: a storage rewiring must not change what a running deployment does.

Six things below apply to every deployment regardless: interpolation, the bootstrap command, TLS, the backup endpoint, the connect-time SSRF re-check, and the Preview header on discovery source management. Read those even if you are not opting in -- the SSRF section in particular, since it changes a security behaviour and describes a lapse to plan for.

Selecting a backend takes over every persisted concern

persistence.backend: sqlite | postgresql chooses storage for all of it at once: the event log and its delivery mark, server configuration, the audit trail, saga state, approvals, API keys, roles, tool-access policies, metric history and the management lease. A backend serves every one of them or the selection is refused -- which is what makes the half-configured deployment unrepresentable. Before 2.5.0 you could select the PostgreSQL auth driver and silently lose tool-access policy management with it.

Two consequences to check before you roll out:

• A per-subsystem key naming a different backend now refuses startup. auth.storage.driver and event_store.driver are compared against your selection, and a contradiction fails the boot rather than being resolved by a precedence rule. Whichever way such a rule fell, half of what you wrote would be ignored -- and the half that loses is the one written most recently. memory is exempt: it is a testing choice, not a storage backend.
• event_store.allow_memory_fallback no longer has anything to decide. With a backend selected, the log and its delivery mark come from it, and a backend is durable as a whole. Keep the key if you are not selecting a backend; it still fails a non-durable store fast there.

There is no migration between backends. Selecting PostgreSQL on a gateway that has been running on SQLite starts an empty database -- it does not move what is in the file.

persistence.backend: postgresql needs the [postgres] extra

A plain pip install mcp-hangar carries no PostgreSQL driver, so a gateway that selects the backend without the extra fails the moment it opens a connection: psycopg2 is required for PostgreSQL. Install it with the backend:

pip install "mcp-hangar[postgres]"

The extra used to install asyncpg, which nothing in the codebase imports -- the stores use psycopg2 -- so an install that looked right still could not start. It now installs psycopg2-binary. The published image installs the extra already, so this applies to a pip install only. It applies to every replica set too, since a coordination: block requires PostgreSQL.

Selecting PostgreSQL turns coordination on, at one replica

This is the one that can surprise a single-node deployment. Coordination keys off whether the storage can be shared, not off how many replicas you run, so a single gateway on PostgreSQL takes a management lease and reports coordinates_with_peers: true. It manages the fleet, because it is the holder -- nothing stops working.

What does change: registering a subprocess, docker or container server through the API is refused (HTTP 422). Those modes attach a child process's stdio to one gateway, and any peer that learned of such a server would start its own copy. Servers already declared in config.yaml keep working -- the refusal is on the registration path, not the startup one.

If that deployment is genuinely single-node and wants to keep registering local modes at runtime, stay on persistence.backend: sqlite, which is not shareable and therefore not coordinated.

A declared cluster refuses a child-process server outright

The paragraph above is about runtime registration. Servers declared in config.yaml take a different path, and when the deployment declares a coordination: block they are refused at startup, naming every offender at once:

this gateway is configured as part of a cluster (`coordination:`), and
'reports' is 'subprocess'. ... Use `remote` mode for servers several replicas
must serve, or remove the `coordination:` block to run this as a single gateway.

Without the block nothing here fires, which is the point of asking on that axis: a single gateway that merely uses PostgreSQL keeps running its child processes exactly as before.

A coordination: block requires PostgreSQL

Adding coordination: is the statement that these replicas are meant to be one gateway. On a file-backed backend it refuses to start, because replicas that cannot share storage are not a cluster -- each would hold its own fleet and its own lease and never notice the others. Not hypothetical: three replicas on SQLite each reported manages_fleet: true, with every health check green.

Running many pods each with their own storage stays legitimate -- that is many gateways. What is refused is calling them one.

If you already run more than one replica

Through 2.4.0 the documentation said not to, and the failure was silent rather than loud. To make a replica set safe on 2.5.0 you need all three of: one PostgreSQL every replica shares, a coordination: block, and remote-mode servers. Then check it pod by pod rather than through the Service -- exactly one should answer manages_fleet: true at GET /api/system.

Two costs are worth knowing before the rollout rather than after: rate limits are counted per instance (three replicas admit three times the configured rate -- a fleet-wide cap belongs at the ingress), and anything travelling by the shared log reaches peers within a poll interval rather than immediately.

Full recipe: running more than one replica. The decisions and their failure modes are in ADR-020.

${VAR} is interpolated everywhere, and an unset one now fails the boot

Interpolation used to work inside mcp_servers.<id>.auth and nowhere else, while the documentation described it as a property of configuration. If you kept a secret out of the file the way the production checklist says to, and it silently arrived as the literal characters ${...}, this is why.

The refusal moved with it. A ${VAR} with no value and no :-default has always been fail-closed, and now fails the whole boot rather than only the auth sub-block:

ConfigurationError: Required environment variable '${HANGAR_DB_PASSWORD}' is not
set and has no default. Use '${HANGAR_DB_PASSWORD:-default}' to provide a
default value, or '${HANGAR_DB_PASSWORD:-}' to explicitly allow an empty value.

So check the keys you never had to set before -- ${VAR:-} allows an empty value explicitly. A value that contains a literal ${...}, such as a generated password, is safe: the document is interpolated once, so substituted text is never rescanned.

auth bootstrap-admin requires --show-key when API keys are the only way in

On a deployment with no trusted OIDC issuer, omitting --show-key is refused before anything is written:

Error: Nothing could use this administrator: API keys are the only
authenticator, and the key's secret would not be printed.

Who is affected: anything that scripts mcp-hangar auth bootstrap-admin against a config without an auth.oidc block. Add --show-key and capture the secret the run prints.

Why it refuses rather than warns. The claim is one-shot and the key it mints is stored hashed, so a run that ends without printing the secret can be neither repeated nor recovered from. It used to end by advising a re-run with the flag, at the moment re-running had become impossible -- the second run answers the initial administrator has already been bootstrapped, and bootstrap-admin is the only subcommand in the auth CLI. The refusal costs one command; the advice cost the deployment.

Nothing changes for a deployment that trusts an OIDC issuer: the principal authenticates on its own identity and needs no secret. A store whose claim was already spent with the secret discarded is recovered by clearing its initial_admin_bootstrap row, or starting from a fresh auth store, and re-running with the flag. See recipe 12.

Per-server TLS settings now reach the connection

mcp_servers.<id>.tls.verify_ssl and tls.ca_cert_path were accepted and silently discarded before 2.5.0 -- the HTTP client passes an explicit transport for retries, and the transport was built without them.

• ca_cert_path failing was pure loss: an upstream behind your own CA was unreachable with no way to fix it from configuration. It works now.
• verify_ssl: false failed closed, so it looked like a stubborn certificate. It now does exactly what it says. If you left that line in a configuration expecting it to be inert, verification really is off now -- the gateway logs a warning naming the endpoint on every such upstream at startup. Trusting a private CA through ca_cert_path is verification and stays quiet.

POST /api/config/backup answers 503 instead of 500

The backup is written beside the configuration file, so it fails wherever that directory is not writable -- which is every deployment on the published image, where /app is owned by root and the gateway runs as hangar. The caller used to get 500 and An internal server error occurred., with the real reason in the log only.

It now answers 503 naming the path and the reason. Anything monitoring this endpoint on status code should expect 503 for a filesystem refusal; mount a writable directory and point --config at it if you need it to succeed.

An endpoint registered through the API is re-checked on every connection

Who is affected: remote servers registered through the REST API or by a discovery source. The SSRF check used to run once, at registration, against the addresses the hostname resolved to then -- and the HTTP client then re-resolved that name at every connect with no second check. A name that passed once could be re-pointed at 169.254.169.254, 10.x or 127.0.0.1 afterwards, and every later tool call followed it there.

Now each request on such an upstream resolves the host again, applies the same policy the registration check used, and connects to the validated address, with the Host header and the TLS SNI/certificate check still carrying the original name. A refused address surfaces as a connection failure rather than a new error class.

What to do: go to 2.5.1 rather than stopping here. On 2.5.0 this check does not survive a restart. The record a registered server is rebuilt from does not carry the enforcement flag, so after any restart -- and on every follower replica in a coordinated deployment, which builds its fleet from the same records -- those servers come back registration-time validated only, with the connect-time guard off, and no re-registration short of DELETE /api/mcp_servers/<id> plus a fresh one restores it. 2.5.1 carries the flag on the record and derives it for servers registered while 2.5.0 was running, so upgrading and restarting is the whole remedy. If you are staying on 2.5.0, the defence between restarts is the registration-time check plus a resolver you control -- which is what 2.4.0 and earlier already ran on.

One thing does change immediately: an upstream you registered through the API that resolves to an address the policy refuses now fails on every call, instead of working after a registration that passed. Servers declared in config.yaml are not re-checked at all, so a remote server deliberately pointed at an internal address in configuration keeps connecting as before.

Discovery source management ships as Preview

The five mutating source routes -- POST /api/discovery/sources, PUT and DELETE on /api/discovery/sources/{id}, POST .../scan and PUT .../enable -- now answer with a response header:

X-Hangar-Preview: discovery-source-management

It gates nothing. The routes work, no request header is required, and nothing is refused for omitting one. It marks a surface that was broken end to end as late as 2.5.0-rc.4 and whose behaviour may still change. The read-only discovery flow -- listing sources, pending and quarantined servers, approve and reject -- is stable and carries no such header.

What to do: if you script these five, log the header rather than assert on it, and expect their shapes to move; if you only read discovery state, nothing changes.

Upgrade to 2.4.0

Drop-in for a default deployment. Everything below affects deployments that run discovery, which is off unless you enabled it.

A third-party discovery source must report the addresses it found

Discovery now registers through the same command a REST caller uses, so a discovered endpoint goes through the SSRF check. A container or pod address is private by definition, so that check is scoped by provenance: a discovered endpoint may resolve to a private address only where the container runtime reported it for that container or pod.

The built-in sources report those addresses. A third-party source written against the entry-point group must put them in the discovered server's metadata:

metadata = {
    # ... your own metadata ...
    "runtime_addresses": ["10.88.0.7"],   # what the runtime says, never a label
}

Without them the source is treated as untrusted input and its servers stop registering, with SSRF blocked: endpoint resolves to private address in the log. That is the safe direction to fail, and it is a change in behaviour for a source that used to bypass the check entirely.

Link-local, loopback and cloud metadata hostnames stay refused whatever the runtime claims.

Kubernetes discovery needs its client installed

pip install mcp-hangar[kubernetes]

The published image ships it. Before 2.4.0 the extra did not exist and the source could not be constructed at all, so this enables a feature rather than breaking one -- but a deployment that expected it to work silently was getting discovery_source_unavailable and no servers.

The event store now records more, and starts earlier

EventBus.publish persists an event that names an aggregate, where it previously delivered without storing. Two consequences worth knowing before you upgrade a long-running gateway:

• data/events.db grows where it did not before -- registrations, discoveries, quarantines and lifecycle transitions all land in it now.
• A server's stream begins with McpServerRegistered rather than with its first edit. Streams written before 2.4.0 keep the shape they had; nothing is backfilled, and replay before a stream's first row is not claimed.

Handlers must be idempotent on event_id. They always should have been -- the startup sweep re-delivers anything the previous process did not finish -- but until 2.4.0 far fewer events reached that path.

Upgrade to 2.3.0

Drop-in for a default deployment. Two things need checking, and each affects a narrow case: code that imports the concrete launchers from the domain layer, and deployments running auth.storage.driver: event_sourcing.

Written first against a planned 2.2.2. That release was never cut -- it became 2.3.0 when the launcher removal landed, so everything below ships in 2.3.0.

The deprecated launcher import paths are gone

Only affects code importing the concrete launcher classes from the domain layer. If you import them from mcp_hangar.infrastructure.launchers, which is where they live and what the deprecation warning has said since v1.0.2, nothing changes.

# Both of these now raise.
from mcp_hangar.domain.services.mcp_server_launcher import DockerLauncher
from mcp_hangar.domain.services import DockerLauncher

# This is the one to use, and always was:
from mcp_hangar.infrastructure.launchers import DockerLauncher

The same applies to SubprocessLauncher, ContainerLauncher, HttpLauncher, ContainerConfig, McpServerLauncher and get_launcher.

mcp_hangar.domain.services still exports the launcher port, IMcpServerLauncher, along with LaunchResult and TransportClient. It is the concrete implementations that moved out -- a domain package re-exporting infrastructure classes is what the deprecation was about.

The shim emitted a DeprecationWarning on import from v1.0.2 onward, so one run of your test suite with warnings fatal lists every call site:

python -W error::DeprecationWarning -m pytest

Removing it also broke a real import cycle: the domain reaching for the concrete launchers is what forced two sagas to import their saga manager inside a function body rather than at module level.

If you run auth.storage.driver: event_sourcing, read this before you upgrade

On that driver, API keys and role assignments were written to the event store correctly and could not be read back. The store's writer accepts any domain event -- it serialises whatever it is handed -- while the reader looked the class up in a table maintained by hand, which listed 30 of the 116 event types in the codebase. All five that the API-key and role aggregates emit were missing from it, so the first read after a restart raised EventSerializationError. In practice:

• every API key stopped authenticating across a restart, and
• role assignments were invisible after a restart.

Affected from 1.2.2, when the driver landed, through 2.2.1. The default driver is memory, and the sqlite and postgresql drivers were never affected -- this only ever reached deployments that set event_sourcing explicitly.

No data was lost. The events were written correctly the whole time; only the read path failed. That is good news with a consequence worth planning for:

Credentials and role assignments you believed were gone will start working again the moment you upgrade.

If you worked around the failure by re-issuing keys after each restart, the older keys are still live and will come back. So will every role assignment ever made on that store that was not explicitly revoked -- including admin.

Check what comes back, before you upgrade

Run these against your configured event_store.path (default data/events.db):

# How much is dormant, by event type.
sqlite3 data/events.db "
  SELECT event_type, COUNT(*) FROM events
  WHERE stream_id LIKE 'api_key:%' OR stream_id LIKE 'role_assignment:%'
  GROUP BY event_type ORDER BY event_type;"

# Exactly which principals get which roles back, oldest first.
sqlite3 data/events.db "
  SELECT json_extract(data, '\$.principal_id') AS principal,
         json_extract(data, '\$.role_name')    AS role,
         event_type, created_at
  FROM events WHERE stream_id LIKE 'role_assignment:%'
  ORDER BY created_at;"

Revocations are events too and they replay in order, so a key or role you revoked before the upgrade stays revoked. What returns is what was never revoked. Revoke anything you do not want live, then upgrade.

If the counts are larger than you expect, that is the measure of how long the store had been unreadable -- every restart since 1.2.2 left its writes behind.

Also in this release

Events written before the provider -> mcp_server rename replay again. The rename landed after 1.0.1, so an event store from 1.0.1 or earlier holds rows typed ProviderStarted, ProviderDiscovered and so on. Replaying one was a silent no-op: it reconstructed into the deprecated alias class, and handlers are registered against the modern class, so it reached nothing. No error, no warning. If you have carried an event store across that rename, expect replay to start producing events it previously swallowed.

A datetime field on a persisted event comes back as a datetime. JSON has no datetime, so it was written as an ISO string and nothing parsed it back -- consumers received a str. Only PolicyPushRejected.timestamp was affected.

Neither of these needs any configuration change.

Upgrade to 2.2.0

Read this one before upgrading. Unlike 2.1.0 and 2.1.1, this is not a drop-in release. It is a security release cut as a minor rather than a patch, deliberately: three of the changes break a working deployment, two of them silently, and ~=2.1.1-style constraints would have pulled a patch in unattended. Three items need action before you roll it out.

Action required

1. Check the role on your operator's API key.

The compiled-egress-policy channel, POST/DELETE /api/mcp_servers/{id}/l7_policy, now requires policy:write instead of mcp_servers:write. mcp_servers:write is held by the developer role, so any developer token could clear a compiled MCPEgressPolicy — ADR-013 makes that channel privileged.

provider-admin has been given mcp_servers:read and policy:write so it is the least-privilege home for an operator key. It previously held only the pre-rename provider:* permissions, which the REST API checks against nothing, so it could not make any of the operator's calls.

If your operator key is a developer token, move it to provider-admin before upgrading. The failure is silent otherwise: the CRD still reconciles, the status still reports Compiled, and the policy never reaches the enforcement point.

2. Check that your OPA policy returns a boolean.

OPAAuthorizer compared the verdict for truthiness. A Rego rule returning an object ({"result": {"allow": true, ...}}), a string ({"result": "deny"}) or an array was therefore treated as allow — including the one that says deny. A non-boolean verdict is now a denial (opa_error:non_boolean_result), and a missing result key — what OPA returns for an undefined rule, e.g. a wrong policy_path — is reported separately as opa_error:undefined_result.

If your policy returns anything other than a bare boolean, it flips from allowing everything to denying everything. Query the rule directly and confirm the response body is {"result": true} or {"result": false}.

3. Check tool_access.mode for typos.

An unrecognised value used to resolve to egress with a warning, which handed a deployment that had written front_door — but misspelled it — the permissive topology. The server now refuses to start on an invalid value. An absent key still means egress; only a present-but-invalid one is fatal.

Other behaviour changes

• REST authorization is enforced on every route. /config, /discovery, /groups, /sessions, /tools, the /approvals reads and the whole /auth subtree previously authenticated callers but made no authorization decision — any valid credential could POST /api/auth/roles/assign and grant itself admin. Authorization is now resolved from the route, and a route not in the permission table is denied. Clients that relied on the gap will start receiving 403.
• POST /api/config/reload no longer accepts config_path. It reloads the server's own configuration file. A request still sending the field gets 422 rather than being silently ignored. Reload loads whatever path it is given and an mcp_servers entry carries command/args, so the old behaviour was a remote "load an arbitrary file and start what it describes" primitive.
• Approvals pending across the upgrade will be refused. The dispatch-time integrity hash moved from the redacted copy of the arguments to the raw ones, so records written by the old version no longer revalidate. Re-request them; this is the fail-closed direction.
• An unknown secretPatterns group is now rejected. A misspelled group name (github-token for github-tokens) used to be skipped silently, leaving the policy reporting as enforcing with that detector off. The policy is now refused at parse time, on both the operator and REST channels.
• Approval arguments are redacted by value, not only by key name. A secret under an innocuous key ({"body": "Authorization: Bearer ..."}) no longer reaches the SQLite record or the REST DTO.

New audit events

EgressPolicySet and EgressPolicyCleared are emitted when an L7 egress policy is attached, replaced or removed. Consumers of the event stream that enumerate event types exhaustively should add them.

Upgrade to 2.1.1

A drop-in security patch on 2.1.0 — no new configuration keys, no API changes. It closes two issues found by red-teaming 2.1.0, both fail-closed:

• The L7 egress jwt secret pattern now catches short-header JWTs that slipped past the previous matcher (#687).
• The approval surface is now scoped by tenant, so an approver in one tenant can no longer see or resolve another tenant's approvals (#688).

Nothing to change in your configuration — upgrade in place.

Upgrade to 2.1.0

Drop-in from 2.0.x for almost everyone: the new configuration keys are opt-in, no key moved, and no API shape changed. What 2.1.0 adds is that the human-in-the-loop approval gate is reachable for the first time. It was documented, unit-tested in isolation and wired nowhere on any shipped path — no config key could put a tool behind it, the gate service was never constructed, and GET /api/approvals answered 500 while a call the policy said to hold executed immediately (#678, fixed in #684).

The one thing that can bite: a config that already carries approval_list. The key existed on the internal policy object, so a tools: block naming it parsed as a block with no access policy and the pattern was silently dropped — those tools ran ungated. From 2.1.0 the same file is honoured: matching calls are held for a human decision, for approval_timeout_seconds (default 300), and a call nobody decides is refused rather than executed. Grep your configuration before you upgrade:

grep -rn "approval_list" /etc/mcp-hangar/

If a hit is not something you want gated, remove it. If it is, make sure someone is watching GET /api/approvals — or a delivery adapter is installed — before you roll it out, because otherwise every matching call now stalls for five minutes and then fails.

Approvals are on by default and inert until a policy gates a tool. Set approvals: {enabled: false} to opt out entirely.

The server may now refuse to boot

2.1.0 adds a startup reachability check. At the end of bootstrap() — the funnel serve, serve --http and the facade all pass through — it asks, for each subsystem a configuration can demand, whether the runtime object that serves it is actually present. A demand met by absence is no longer silent.

A tool on approval_list with no gate service refuses the boot: a gateway that cannot hold a call is a gateway executing it unapproved, and starting anyway is failing open. Everything else logs at ERROR naming the subsystem and what asked for it. The refusal is a ConfigurationError reading Configured subsystem is not reachable on this server: ....

startup_checks:
  enforce: false      # downgrade the refusals to error logs

There is deliberately no switch that makes an unreachable subsystem silent.

The full key reference is in Configuration → tools dual format.

Upgrade to 2.0.1

Drop-in from 2.0.0 — nothing to decide, no config key moved, no API shape changed. A single security fix: the approval gate now re-establishes an approval's validity at dispatch rather than only at decision, re-checking state, expiry and the argument hash, re-resolving the effective policy, and re-running the digest pin after the hold. Behaviour changes in one direction only — a call whose world moved while its approval was pending (tool withdrawn, policy tightened, arguments rewritten, approval expired) is now refused where it previously executed. An expired approval resolves 409 instead of minting a false APPROVED record (#674).

Scope it honestly: at 2.0.1 the approval gate was not reachable on a stock serve --http (#678), so this was the guard that had to be in place before that wiring landed, not a patch to a live exposure. The wiring landed in 2.1.0, and this fix now guards a path a deployment can actually enter.

Upgrade to 2.0.0

MCP Hangar 2.0.0 moves the gateway onto the MCP 2026-07-28 protocol generation and the stable mcp==2.0.0 SDK, and it removes the last vendor integration from core. Four things need a decision before you upgrade; the rest is drop-in (pip install -U mcp-hangar, or pull ghcr.io/mcp-hangar/mcp-hangar:2.0.0).

Slack approvals need an adapter (breaking, act before upgrading)

Core no longer knows any approval vendor. The resolve route dropped its X-Slack-Signature branch, and delivery/slack.py left the tree. If your config sets approvals.channel: slack, the channel silently degrades to noop on 2.0.0 — approvals still queue and stay resolvable over REST, but nobody is notified. That degradation is deliberate: refusing to boot over a notification channel turns a degraded path into an outage.

The replacement is an adapter you run yourself. It terminates the Slack webhook, verifies the signature, maps the Slack identity onto a Hangar principal, and calls POST /approvals/{id}/resolve with an ordinary token. A reference adapter ships in Approval delivery adapters, which walks the Slack case end to end — outbound notification and inbound resolution.

Why the change: both authentication branches on the old route were individually sound, but an unauthenticated caller chose which one ran. One chokepoint, one mechanism (ADR-016).

Provenance changes with it. decided_by was slack:<user-id>; it now names a Hangar principal. Anything parsing that field — audit queries, dashboards, exports — needs updating.

Approval resolution is authorized now (breaking for API callers)

approval:resolve was defined, granted to a role, and checked nowhere: any principal with a valid token could decide any approval given its id. It is enforced from 2.0.0, so a caller without the permission gets 403 where it previously got 200.

The x-principal-id request header no longer sets identity. It used to be the only path that set decided_by, including on authenticated requests, and a client-supplied header landing in a provenance chain is not attribution. Identity now comes from the authenticated principal; with auth disabled the decision is attributed to the system principal.

tasks/result and tasks/list are gone (breaking for task clients)

The task relay serves the SEP-2663 wire. tasks/get now inlines the outcome — the round trip through tasks/result is what SEP-2663 removed — and both tasks/result and tasks/list answer -32601. tasks/get|update|cancel require the mandatory Mcp-Name: <taskId> header over HTTP.

The synchronous 2025-11-25 mid-flight consent flow is removed with it: Hangar no longer issues an elicitation/create prompt inside tasks/get. On the 2026-07-28 wire the client resolves its own input by driving tasks/update, which is governed and still fail-closed.

The SDK pin moves to mcp==2.0.0

If your environment installs mcp alongside Hangar, it moves to the v2 line. Your upstream MCP servers do not have to. A connection that negotiates 2025-11-25 keeps working: the handshake records the negotiated era and withholds the modern _meta envelope on legacy connections.

Upgrade to 1.6.0

MCP Hangar 1.6.0 is an observability-hardening release: tool-invocation telemetry now follows the OpenTelemetry GenAI/MCP semantic conventions, the transport message metrics are wired, and it ships the L7 egress-policy (MCPEgressPolicy) enforcement plane end to end. Upgrade is drop-in (pip install -U mcp-hangar==1.6.2, or pull ghcr.io/mcp-hangar/mcp-hangar:1.6.2); the notes below cover what changed for telemetry consumers.

Span attributes moved to OTel semantic conventions (breaking for trace consumers)

Tool-invocation spans now use the OTel GenAI/MCP semconv names. If you query, filter, or alert on Hangar's traces or OTLP audit records by attribute, update:

• mcp.tool.name → gen_ai.tool.name
• mcp.cost.input_tokens → gen_ai.usage.input_tokens
• mcp.cost.output_tokens → gen_ai.usage.output_tokens
• the application span name tool.invoke.{tool} → execute_tool {tool}; the outgoing transport call is now a SpanKind.CLIENT span carrying gen_ai.operation.name and mcp.method.name.

The Hangar-specific governance namespaces (mcp.enforcement.*, mcp.risk.*, mcp.audit.*, mcp.cost.cents/model/currency, mcp.session.id) are unchanged. OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG are now honored.

Metrics: new transport message metrics; three dead metrics removed

New, labeled per upstream server: mcp_hangar_messages_sent_total, mcp_hangar_messages_received_total, and the mcp_hangar_message_size_bytes histogram. Removed (they were never emitted): mcp_hangar_http_connection_pool_size, mcp_hangar_http_sse_streams_active, and mcp_hangar_http_sse_events_total — drop any dashboard panel or alert that still references them.

Upgrade to 1.5.0

MCP Hangar 1.5.0 adds a one-time admin bootstrap, a configurable command-bus rate limit, the interceptor invocation surface with phase-aware hooks, task-lifecycle audit events, and a per-tenant discovery entry point. It also fixes OIDC bearer authentication over the HTTP surface. Upgrade is drop-in (pip install -U mcp-hangar==1.5.0, or pull ghcr.io/mcp-hangar/mcp-hangar:1.5.0); the notes below cover the behavior changes worth reviewing.

OIDC bearer auth over serve --http now works

If you configured OIDC/JWT front-door auth (auth.oidc) on the HTTP server in 1.4.x and every request returned 401 with auth_method: none even for a valid token, that was a header-casing bug in the JWT authenticator -- it is fixed in 1.5.0. No config change is needed; existing auth.oidc config now authenticates bearer tokens as intended.

Bootstrap the initial admin

A fresh durable auth store with anonymous access disabled could not create its first administrator through the protected API. mcp-hangar auth bootstrap-admin --config PATH --principal PRINCIPAL now grants the one-time global admin role to an existing external (OIDC) principal using the server's own durable backend. It fails closed when auth is disabled, anonymous access is allowed, or the store is non-durable (memory / event_sourcing), and a second run is refused without mutating storage. No secret is printed.

Behavior changes to review

• Tool isError results now count as failures. A backend MCP tool result with isError: true is treated as a tool failure -- reflected in the per-call result, batch succeeded/failed counts, health, and ToolInvocationFailed events. If you previously treated error results as successes, expect failure counts to rise.
• The SQLite event store fails fast. When a durable event store cannot be initialized (path not writable / backend unavailable), Hangar now refuses to start instead of silently degrading to a non-durable in-memory store. Opt into the fallback with event_store.driver: memory or event_store.allow_memory_fallback: true. /health/ready returns 503 if the store degraded to in-memory while a durable driver was configured.
• Group circuit breaker. A tripped circuit breaker on one group member no longer blocks a healthy remaining member from serving.
• Command-bus rate limit is configurable. The previously-fixed command-bus rate limit can now be tuned in config.yaml; review the configuration reference if you relied on the old fixed value.

Upgrade to 1.4.0

MCP Hangar 1.4.0 builds on the 1.3 front-door release. It adds tenant-scoped digest pins, multi-issuer OIDC trust, resource-bound JWT audiences, and per-tenant canary routing for MCP server groups.

Review OIDC audience binding

If auth.oidc.resource_uri is set, it now becomes the expected JWT aud value for every trusted issuer. This aligns token validation with the RFC 9728 Protected Resource Metadata resource value and RFC 8707 resource indicators.

Before upgrading production front-door deployments:

• Confirm the authorization server issues tokens with aud equal to auth.oidc.resource_uri.
• If you need legacy audience values per issuer, leave resource_uri unset and configure audience on each issuer instead.
• Prefer setting resource_uri behind proxies; otherwise Hangar derives the resource from the incoming request scheme and host.

Move multi-issuer deployments to auth.oidc.issuers

Single-issuer config still works:

auth:
  oidc:
    enabled: true
    issuer: https://issuer-a.example.com
    audience: mcp-hangar

Use auth.oidc.issuers when one Hangar instance trusts multiple authorization servers:

auth:
  oidc:
    enabled: true
    resource_uri: https://hangar.example.com
    tenant_claim: tenant_id
    issuers:
      - issuer: https://issuer-a.example.com
        audience: https://hangar.example.com
        jwks_uri: https://issuer-a.example.com/jwks
      - issuer: https://issuer-b.example.com
        audience: https://hangar.example.com
        jwks_uri: https://issuer-b.example.com/jwks
        groups_claim: roles

Tokens with a missing, empty, non-string, or untrusted iss claim now fail closed with a 401 instead of reaching any issuer validator.

Add tenant-scoped digest pins intentionally

1.4.0 can enforce schema pins per tenant on the live invocation path:

mcp_servers:
  payments:
    mode: remote
    endpoint: https://payments.example.com/mcp
    tool_projection:
      digest_enforcement: block
      tenant_overrides:
        "tenant:a":
          pins:
            refund: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

digest_enforcement is scoped per MCP server. If unset, pin mismatches default to block. Roll out pins with audit or warn first when you are recomputing digests or onboarding a new tenant.

The deprecated allow_degraded unknown-tool policy is still accepted with a DeprecationWarning in 1.4.0, but allow_unverified remains the canonical value. Do not add new allow_degraded config.

Gate canary routing by tenant identity

Group canary routing only applies when Hangar has a tenant_id for the caller. Explicit tenant pins win first, then the sticky percentage split, then the regular load-balancing strategy.

mcp_servers:
  search:
    mode: group
    strategy: weighted_round_robin
    canary:
      member: search-v2
      split_pct: 10
      pinned_tenants:
        "tenant:beta": search-v2
    members:
      - id: search-v1
        mode: remote
        endpoint: https://search-v1.example.com/mcp
      - id: search-v2
        mode: remote
        endpoint: https://search-v2.example.com/mcp

Invalid canary targets are skipped with a warning. If a pinned or canary member is not in rotation, Hangar falls back to the group load balancer instead of routing traffic to an unhealthy member.

Upgrade to 1.3.0

MCP Hangar 1.3.0 relicenses the project to MIT, folds the former enterprise package into the main mcp_hangar package, and tightens digest-pinning canonicalization.

Recompute pinned tool digests

compute_tool_digest() now uses RFC 8785 JSON Canonicalization Scheme (JCS) instead of json.dumps output. Existing pinned digests produced by earlier versions may no longer match and must be regenerated before enforcement mode is enabled.

When recomputing digests, note the v1.3 normalization rules:

• None, {}, [], and "" are treated as absent values.
• Tool entries with a missing, empty, or non-string name field are rejected.

Recommended rollout:

1. Upgrade one environment with digest enforcement set to audit or warn.
2. Collect the new expected digests from the audited tool inventory.
3. Replace old pins with the RFC 8785/JCS digests.
4. Re-enable block only after the audited drift is resolved.

Rename ALLOW_DEGRADED to ALLOW_UNVERIFIED

DigestUnknownPolicy.ALLOW_DEGRADED was renamed to DigestUnknownPolicy.ALLOW_UNVERIFIED for clarity.

If your YAML or code uses the string value allow_degraded, change it to allow_unverified. MCP Hangar 1.3 still accepts allow_degraded with a DeprecationWarning; 1.4.0 still accepts the alias, but new configuration should use only allow_unverified.

Remove license-tier assumptions

The former BSL/enterprise split is gone. All MCP Hangar features are now available under the MIT license.

Required cleanup for integrations and deployment manifests:

• Stop setting HANGAR_LICENSE_KEY; v1.3 ignores it and emits a DeprecationWarning when present.
• Remove checks for LicenseTier, LicenseValidation, or ApplicationContext.license_tier.
• Update imports that referenced the former enterprise/ package; auth, compliance, approvals, integrations, and persistence modules now live under src/mcp_hangar/.

Verify interceptor discovery clients

interceptors/list now returns unique instance names required by SEP-1763:

• mcp-hangar-validator
• mcp-hangar-mutator

If a client keyed both entries by the previous shared name mcp-hangar, update it to handle the two explicit instance names.