ADR-030: An Upstream Is Pinned by Its Key at the Connection, and Attestation Is a Declared Source of the Key
Status: Accepted Date: 2026-10-19 Authors: MCP Hangar Team
Context
State of main on 2026-10-19: core 2.31.0 (40f448a1), operator 6d5d7e6 with CRD v1alpha2.
Hangar already proves the integrity of a tool's contract, and nothing about the code behind it. The JCS digest over {name, description, inputSchema, outputSchema} (ADR-004, domain/services/digest_validator.py) is enforced by _gate_digest_pin and _gate_deferred_digest_pin (server/tools/batch/executor.py:2234, _GATES at :3104-3117). Since 2.28.0 (#1802) a digest pin is bound to McpServer.upstream_identity() (domain/model/mcp_server.py:578-597) and the binding is checked at call time (executor.py:473-476, digest_validator.py:77). Digest-pin history lives in the pin_ledger:<server> stream as ToolPinRecorded, a log derived from configuration (domain/events/enforcement.py:351-357), not the other way round.
For every operator-managed server, upstream_identity() is a pod IP and port, not an image. The operator annotates the pod (operator/pkg/provider/builder.go:192-195: mcp-hangar.io/enabled|name|mode=http|port); core discovers it from pod.status.pod_ip (infrastructure/discovery/kubernetes_source.py:250-287) and registers it as mode: remote, so the identity is remote:http://<pod_ip>:<port>. The image: branch of upstream_identity() serves Hangar's own Docker mode only. Core never learns the image digest of an operator pod. A digest pin, an audit record or any other fact bound to upstream_identity() for an operator pod is bound to an IP the CNI will reuse, and nothing in Hangar can tell the pod it meant from the pod that inherited the address.
Nothing in core or the operator attests a workload. The only uses of attestation vocabulary are build provenance of Hangar's own artefacts (actions/attest-build-provenance@v4 in core release.yml:232; cosign sign/cosign attest for SBOM and SLSA in the operator and chart release workflows; tests/ci/test_a_release_carries_verifiable_artifacts.py); the phrase "either client-attested or "unknown"" in approvals/api/routes.py:257-258, where attested means self-asserted; and docs/operations/COMPLIANCE_POSTURE.md, which says Hangar is not an attestation in the SOC 2 sense.
The operator cannot yet place an upstream in a Confidential Container. MCPServerSpec (operator/api/v1alpha2/mcpserver_types.go) passes nodeSelector, tolerations, affinity, priorityClassName, serviceAccountName, automountServiceAccountToken, podSecurityContext and imagePullSecrets into the PodSpec (pkg/provider/builder.go:83-107); it has no runtimeClassName, and without runtimeClassName: kata-qemu-snp|kata-qemu-tdx the pod does not start in a confidential VM. That field is needed independently of attestation (gVisor and plain Kata need it too) and is a separate operator change (mcp-hangar/mcp-hangar-operator#355); this record depends on it for container-mode CRs that declare issuance: kbs-attested and does not decide it.
The HTTP client already trusts a per-upstream CA. infrastructure/launchers/http.py:35-51,172-212 accepts tls.ca_cert_path from the server's tls block (server/config.py:1279) and warns (tls_verification_disabled), naming tls.ca_cert_path, on the verify_ssl: false + no-CA combination; it does not refuse it. When the server also has an http block, the launcher rebuilds the client configuration and copies only verify_ssl and ca_cert_path from tls (launchers/http.py:176-194). The CRD has no TLS block; operator pods are discovered as plain http.
Image integrity is already an admission policy. operator/config/admissionpolicy/image_digest.yaml + params.yaml give imageDigestPolicy: off|warn|block (default warn) as a ValidatingAdmissionPolicy whose level comes from a ConfigMap; a missing ConfigMap admits. A per-object opt-out, the annotation hangar.io/allow-mutable-image, is honoured (image_digest.yaml:35-36). internal/webhook/image_digest.go:9-21 (IsImageDigestPinned) is the controller's egress coupling plus a VAP mirror — the webhook copies are gone (#312). Intra-object consistency rules in this CRD family are expressed as CEL XValidation on the type (api/v1alpha2/mcpdiscoverysource_types.go:31-32,47); CRD CEL cannot read metadata.annotations. An MCPEgressPolicy upstream match (UpstreamMatch) carries toolSchemaDigestRef and an imageDigest mode, required|inherited (mcpegresspolicy_types.go:103-119).
Other primitives that matter, as they are. HandlerKind (domain/contracts/event_bus.py:13-68): PROJECTION runs on every replica for every event, must be built from the event payload or the shared store and must not publish; EFFECT touches the world and runs only on the producing instance; LOCAL_VIEW keeps a view of local state and runs only on the producing instance (:51); McpServerStarted is produced by every replica, because each starts its servers itself (event_bus.py:60-66, ADR-020 "what stays local"). The fleet snapshot (snapshot_of, domain/services/fleet_snapshot.py:27-76; server_from_snapshot, :124-178) is configuration: a server rebuilt from it starts COLD, enforce_ssrf is derived, not trusted on read (:79-122, :174), and it carries no tls, auth or http field. The fleet projection (application/event_handlers/fleet_projection.py:78-86) applies McpServerRegistered, McpServerDeregistered and L7 policy changes on every replica; a registration for a server the replica already holds is ignored (:131-135), and McpServerUpdated, which the aggregate records on a configuration update (mcp_server.py:2713), is not applied by peers. OPA authorization for hangar_call runs per call before BatchExecutor.execute with no context filled (server/tools/batch/__init__.py:245; ADR-028 §2). The operator delivers L7 policy through POST /api/mcp_servers/{id}/l7_policy (server/api/mcp_servers.py:526-528, operator pkg/hangar/client.go:397-401) under the kubernetes-operator role, which holds policy:write (auth/roles.py:176-195). ADR-029 gives hangar.gate.decision span events with a bounded hangar.gate.reason (observability/conventions.py:145-170); an expected refusal leaves Hangar-owned spans UNSET (ExpectedRefusal, errors.py:51); mcp-hangar why (2.28.0) reads refusals from the event store.
External forces. Attestation of MCP servers is being named as a gap by the confidential-computing community, and TEE-hosted MCP gateways now position themselves against software-only governance: a policy engine inside a TEE with its policy hash in the attestation report, without a fleet, Kubernetes or contract digests. Research on MCP security shows that an MCP server an intermediary cannot authenticate is easy to impersonate.
Boundaries this record does not move. Deterministic enforcement, no ML; no SaaS (ADR-010); L7 egress with a network backstop and no MITM/eBPF (ADR-013); in-cluster MITM outside the threat model; the event log signed by the operator's key (the circular trust named in docs#252) stays as it is.
The question. How Hangar can know that the upstream answering a call is the one the operator meant, rather than whatever holds its address today; and, when the operator runs that upstream in a Trusted Execution Environment, where the fact "this upstream runs attested code" enters Hangar's behaviour — and what Hangar must refuse to become in the process.
Decision
An operator may give a remote upstream a peer identity: a stable name for a key lineage and a list of SPKI digests. Hangar then requires every connection that carries traffic to that upstream to terminate at a key in the list, verified inside the TLS handshake and before the first byte, and records the observed key in the audit trail from the socket. Where the keys come from is a separate, declared fact: the operator may declare that a KBS released them only to attested guests, and Hangar records that declaration as a declaration. Hangar does not verify evidence, trust a CA, issue nonces, or become a subject of attestation; an attested source of keys means attestation-gated issuance (Guarantee A), never continuous runtime attestation (Guarantee B).
In this record a key pin is one SPKI digest in the key list of a server, and a key-pinned server is a remote server with tls.peer_identity. A digest pin is the existing pin of a tool's contract (ADR-004); the two are never called just "pin".
1. An upstream's identity can be its key
A key-pinned server terminates TLS with a leaf key whose SPKI digest is in its key list (peer_identity.spki_sha256, a list to allow rotation windows). No CA is a trust input for a key-pinned server: a CA kept at an issuer makes the SAN a statement of issuance policy Hangar cannot see, a CA private key handed to workloads would let any of them mint a certificate for any name, and hostname matching in the client accepts DNS/IP SANs only, so a SPIFFE URI SAN would not survive the handshake and a static leaf cannot carry a CNI-assigned pod IP. Pinning the key removes all three problems: the chain is irrelevant (a self-signed leaf is fine), the name is irrelevant, and what Hangar records is this connection terminated at key K, which the operator put in the key list. The lineage of keys is named in the declaration as peer_identity.key_ref, a stable name that does not change when the key rotates (§7). The binding holds for the life of the connection; Hangar does no TTL, nonce or replay reasoning.
The mechanism does not depend on where the key came from. An operator without a TEE gains channel binding against IP reuse, a stable upstream identity across reschedules, and an audit record of which key answered.
2. Where the keys come from is a declared fact
peer_identity.issuance names how the keys in the list were issued. It is a declaration by whoever wrote the key list, recorded as such, and Hangar verifies none of it:
- absent — the operator manages the keys. The record establishes only that the connection ended at a key the operator listed.
kbs-attested— the operator declares that a Key Broker Service released each key only to an attested guest for the workload it was provisioned for (Guarantee A, below).
The set is closed; another value is refused at load and by the CRD. A new source of keys, or a new kind of peer identity, is a later decision (last paragraph of this section).
Two independent policy enforcement points for kbs-attested; Hangar is the first and only the first. Hangar answers allow(principal, tenant, tool, args). A Key Broker Service (Trustee KBS or equivalent) answers release(secret, measurement, nonce, pubkey, policy) to an attested guest. Neither replaces the other: a Hangar refusal means the call never reaches the upstream; a KBS refusal means the upstream has nothing to execute it with. Hangar does not hold downstream secrets, does not release keys and does not become a KBS.
What kbs-attested assumes. The key pair is provisioned as a KBS resource before the workload starts, so the operator knows its SPKI before the pod exists and reads it from that resource; key_ref names that resource (for example the KBS resource path). A key generated inside the guest qualifies only if the guest registers its public key with the KBS as part of attestation and the operator reads the SPKI from the KBS; an SPKI read from the running pod never qualifies, because that would put whatever answered into the key list. Whether Trustee supports the per-workload release this needs is checked outside this record (mcp-hangar/mcp-hangar#1936); if it does not, only this value of issuance is affected.
Two guarantees are kept apart, and kbs-attested provides only the first. Guarantee A — attestation-gated credential issuance: key K was released under a policy that required a successful attestation of the guest for the workload K was provisioned for. Guarantee B — continuous runtime attestation: the process answering now is in a verified TEE state. TLS proves possession of K, not where K lives today: a key exfiltrated from a compromised guest, or a guest whose measurement drifted after release, still completes the handshake. A revocation at the KBS does not reach Hangar: a key stays acceptable to Hangar for as long as it is in the key list, whatever the KBS has since decided. The issuer-side mitigations are short-lived keys and re-release tied to re-attestation; the Hangar-side control is the key list itself (§8).
Extension points, not decided here. A second kind of peer identity — for example RA-TLS, where the client verifies an attestation quote inside the handshake (Alternative 7) — would plug into the same handshake check (§5) and the same audit fields (§6), and would make Hangar a verifier, which §3 rules out for this record. A key list written by someone other than its registrant — an independently administered store, or a signed admission statement such as the one MCP SEP-2809 proposes — would change who writes peer_identity, not how it is enforced (mcp-hangar/mcp-hangar#1951).
3. Hangar is a relying party only through TLS; it is not a verifier
Hangar does not parse or verify Evidence, does not talk to an Attestation Service, holds no Endorsements or Reference Values, ships no attestation client, and does not evaluate X.509 chains for key-pinned servers. The SPKI digest is computed with the cryptography package already in the dependency set (pyproject.toml:70). The domain stays free of platform SDKs.
4. Declaration and its carriage
Config file. The tls block gains peer_identity:
mcp_servers:
payments:
mode: remote
endpoint: https://10.244.3.17:8443 # pod IP, Service name, anything: the name is not checked
tls:
peer_identity:
key_ref: kbs:///default/payments/tls-key # stable name of the key lineage; survives rotation
spki_sha256:
- sha256:9f2c... # current key
- sha256:4b10... # next key during rotation
issuance: kbs-attested # optional declaration of where the keys came from (§2)
max_connection_age: 1h
A key pin has one canonical form: sha256: followed by 64 lowercase hexadecimal characters, the SHA-256 of the DER-encoded SubjectPublicKeyInfo. Any other form (base64, uppercase, another algorithm) is refused at load and by the CRD pattern, so an encoding difference can never surface as a key mismatch.
Load-time refusals (ConfigurationError naming both halves, the #841/#902 pattern): peer_identity on mode: subprocess or Hangar's own container mode (no connection to bind); with an http:// endpoint; with an empty spki_sha256 list, a key pin not in canonical form, or no key_ref; with an issuance outside the closed set; with verify_ssl: false (meaningless, but a sign of a misread). ca_cert_path may be present for servers without a peer identity and is ignored for key-pinned ones with a warning. The http block keeps the key list: building an HTTP client for a key-pinned server without its key list raises instead of producing a client that verifies nothing.
CRD. MCPServerSpec gains tls: {peerIdentity: {keyRef, spkiSha256: [], issuance}, maxConnectionAge}. SPKI digests are public values, so no Secret or ConfigMap reference is needed and the operator's least-privilege RBAC (rbac_least_privilege_test.go:22-33: no Secret reads) is untouched. CEL XValidation (mcpdiscoverysource_types.go:31-32 precedent): peerIdentity requires a non-empty spkiSha256 whose items match the canonical pattern, and a keyRef; issuance is an enum. Rules that only make sense for an attested source apply only when issuance is kbs-attested: once MCPServerSpec has runtimeClassName, a container-mode CR with issuance: kbs-attested requires it (an operational precondition, not evidence: gvisor satisfies the rule and attests nothing); and, because CRD CEL cannot read annotations and the image-digest policy is parameterised with a default of warn, a separate ValidatingAdmissionPolicy with an unparameterised Deny binding refuses hangar.io/allow-mutable-image on any CR whose issuance is kbs-attested, whatever imageDigestPolicy says and whether or not its ConfigMap exists. Remote-mode CRs need no runtime class.
Carriage: operator to core is the only declaration path, and it does not exist today for container mode. The operator registers only remote-mode CRs with core (internal/controller/mcpserver_registration.go:222-224, body {mcp_server_id, mode, endpoint}, pkg/hangar/client.go:485); container-mode pods reach core solely through core's own annotation discovery, which hardcodes http:// (server/bootstrap/discovery.py:235) and skips unknown modes (:186-196); the REST create route reads no tls (server/api/mcp_servers.py:319-360) and CreateMcpServerCommand has no such field. This record therefore requires:
- (a) the operator registers a key-pinned CR before its pod exists. It POSTs the name with the
tlsblock and no endpoint; such a server cannot start until it has one. When the pod has an IP, the operator sets the endpoint; on observing the pod's deletion the operator clears the endpoint, and it sets the new one when the new pod has an IP; the operator cannot order this against the CNI's reuse of the old IP, and does not need to, because a pod that inherits the IP does not hold the listed key. Because the name exists in core before any pod carries it, discovery never wins a race for it; - (b) the REST create/update body,
CreateMcpServerCommandandUpdateMcpServerCommandgain atlsfield, the update gainsendpoint, and create accepts a key-pinned remote server without an endpoint; settingtlsor the endpoint of a key-pinned server requiresmcp_servers:peer_identity(§9). An endpoint set over REST is aHUMAN-provenance endpoint (domain/value_objects/provenance.py) under the SSRF policy, so the pod CIDR must be inssrf.trusted_networks(server/config.py:1843, #1752); the operator and chart documentation say so, because a CR moved from discovery to key-pinned registration meets the SSRF policy for the first time. An endpoint change on a server whosekey_refis unchanged resets that server's start backoff and circuit breaker, so failures against an old pod IP do not hold the new pod back; - (c) the operator emits no discovery annotations for a key-pinned CR, and core discovery never creates, replaces or downgrades a server whose name is in the peer-key name record (§9). The check reads the shared table and the names the lease holder's configuration file declares, not the replica's local fleet, because discovery runs on the lease holder (
server/bootstrap/discovery.py:54); a refused name is logged asdiscovered_peer_key_upstream; - (d)
MCPDiscoverySourcesources with apeerIdentityin theirproviderTemplateare refused by the CEL for the source: the template is deep-copied into every server the source yields (mcpdiscoverysource_controller.go:751-753), so they would all share onekeyRefand one key list, which §7 forbids; - (e) the operator registers a key-pinned CR only with a core that advertises the capability, and only over an
https://hangar.url; otherwise it reports the CR as not ready. A core that does not know the field would ignoretlsin the REST body (server/api/mcp_servers.py:342-360), and its launcher reads onlyverify_sslandca_cert_path(launchers/http.py:176-182), so a peer identity declared on an older core would be silently served as a plain server. The same would hold for an older replica during a rolling upgrade:McpServerConfigSnapshot.from_dictdrops keys it does not know (domain/contracts/persistence.py:192-204), so an older replica tailing the registration would rebuild the server withouttls. Core has no record of which instances are live, so it cannot gate on the fleet; instead the row of a key-pinned server is stored in a form the previous release declines rather than reads partially. The declared mode staysremote, andMcpServerModegains no member; only the stored row encodes it:snapshot_ofwritesremote-peer-keyto the row'smodecolumn and toconfig_json.modewhentls.peer_identityis present, and the current release'sfrom_dictmaps it back toremoteand refuses such a row that has notls.peer_identityblock. The previous release'sserver_from_snapshotraisesValueErroron that value (McpServerMode.normalizehas no fallback,domain/value_objects/mcp_server.py:84-89, called atfleet_snapshot.py:120andmcp_server.py:341); in the projection the event bus's handler fault barrier (infrastructure/event_bus.py:393) catches it and the tail moves on, and at startup recovery skips that row (infrastructure/persistence/recovery_service.py:151), so the older replica does not hold the server at all. An older replica can still write: its create passes its local check and overwrites the row with a plain one through the old upsert. The current release therefore refuses, at recovery and in the projection, any row whose name is in the peer-key name record unless it declarestls.peer_identitywith the samekey_ref, so a downgraded row is never served; and the upgrade note forbids declaring a peer identity until no replica of the previous release runs. This is the "decline rather than invent" rule of ADR-020 §8 applied to key-pinned rows, and it relaxes that section's rule that a row written by the new version must be readable by the old one, for key-pinned rows only and for one release; the upgrade note says so. A configuration file that setspeer_identityhas no such guard: an older core serves it as a plain server, and the upgrade note must say that too. - (f) a server that already exists from discovery, with
DISCOVERYprovenance, is not created again but updated: undermcp_servers:peer_identitythe operator sets itstlsblock and endpoint, its provenance becomesHUMAN, its identity changes (§7) and its digest pins are recorded again, and its name enters the peer-key name record (§9), after which discovery ignores the old annotated pod;
Integrity of the declaration. The key list travels operator to core over hangar.url with X-API-Key (pkg/hangar/client.go:582-583); the chart's default hangar.url is empty and its documented example is plaintext (helm-charts/.../values.yaml:177-180). The declaration is public data; its integrity is what matters, and it is protected exactly as far as that channel is. The operator therefore refuses to register a key-pinned CR over anything but https:// (§4e), with the core's certificate verified against hangar.tls.ca. Delivery of the key list to core is inside the trust model; delivery of calls to the pod is not.
5. Enforcement inside the handshake, before the first byte, fail-closed
Where the check runs. For a key-pinned server, the client builds an ssl.SSLContext with minimum_version = TLSv1_3, check_hostname=False and verify_mode=CERT_NONE for chain purposes, and the check runs inside that context's handshake. TLS 1.3 binds the server certificate and its CertificateVerify to the handshake transcript and has no renegotiation, so the key checked is the key the channel was established with; an upstream that offers only TLS 1.2 fails the handshake. The context's sslsocket_class overrides do_handshake: it completes the handshake, reads the peer certificate (getpeercert(binary_form=True), which returns the DER certificate under CERT_NONE), hashes its SPKI, keeps the digest on the socket for audit (§6), and raises when the digest is not in the key list or no certificate was presented; it keeps the signature do_handshake(self, block=False). Because the raised type is neither OSError nor ValueError, SSLSocket._create does not close the socket, so the override closes it before raising. The connection is therefore refused before the HTTP library returns it to the pool and before any request byte is written. httpcore's trace extension reports the end of start_tls, but only for a request that asked for it; the handshake route covers every connection, including the GET stream. Every upstream client is synchronous today (the only AsyncClient in core is infrastructure/registry/client.py); an asynchronous upstream path, if one is added, uses the same context with an sslobject_class that performs the same check. This is new TLS verification code in the transport (http_client.py:425-495 area), merged with _SsrfGuardedTransport so there is one transport, and it applies to every new connection including retries, reconnects, pool replacement and the GET stream.
What is refused and how it is coded. One outcome is a refusal, with one bounded code and one dedicated exception type: upstream_peer_key_mismatch (the handshake completed and the key is not in the list, or no certificate was presented). Everything else that can go wrong before the key is seen — a peer that does not speak TLS, a peer that offers only TLS 1.2, a reset, a timeout, an alert — is an ssl.SSLError or another OSError, becomes httpcore's ConnectError, and is retried and recorded as a transport error, as today: telling a plaintext peer from a protocol error would need OpenSSL's version-specific SSLError.reason strings, and both are fail-closed either way, so no second code is defined. The refusal type derives from ClientError, not from OSError, so httpcore does not map it to ConnectError, and every aggregate I/O site that already catches ClientError (mcp_server.py:2010, 2072, 2265, 2316, 2361, 2404) catches it; those sites record the failure and then re-raise the refusal type unwrapped, where today invoke_tool raises ToolInvocationError after recording (:2099-2104) and relay_request wraps without recording (:2265-2270). The type carries failure_code and observed_peer_spki and the ExpectedRefusal marker (errors.py:51).
How the type survives each layer.
HttpClient.call(http_client.py:816-827) andHttpClient.notify(:968) re-raise the refusal type unchanged instead of wrapping it;_post_with_retrydoes not retry it; the executor's retry (executor.py:2765) does not retry it either._send_initialized_notification(mcp_server.py:1455-1462), the one caller ofnotify, swallows every failure so that a missed notification does not fail a start; it re-raises the refusal type, and keeps logging everything else, so a refusal on the connection that carriesnotifications/initializedfails the start instead of finalising itREADY.McpServerstart re-raises it unwrapped, as it already re-raisesMcpServerNotHereError(mcp_server.py:951-961), still recording the failure with_handle_start_failureas that branch does, instead of wrapping it inMcpServerStartError(:969-985). A caller waiting on that start inensure_readyreceives the stored exception itself, not a new genericMcpServerStartError(:917-921)._gate_cold_startrefuses it with the type's own name instead of the literalMcpServerStartError(executor.py:2558), and as an expected refusal: norecord_handled_failure, span UNSET, onebatch_call_refusedline.- MCP session renegotiation (#1852,
_call_with_session_recovery,mcp_server.py:1307) lets it through instead of turning it into an RPC error without a code. - The GET-stream loop (
http_client.py:1035) treats it as terminal for the current pool: it stops retrying, logs, and counts the metric; the stream is re-opened when the pool or the server is next replaced (§8).
_GATE_REASONS is keyed by the refusal's error_type string, which the gate sets (executor.py:2908-2911); the type maps to the code with outcome deny, not error. ADR-029 §5 already requires a bounded code list per gate, defined next to the gate; adding this code extends that list and does not amend ADR-029.
Cold and warm agree. On a COLD or DEAD server the first contact happens in _gate_cold_start (executor.py:2512), which records the gate decision with the specific reason and publishes the ToolCallRefused that every gate refusal publishes. On a warm server the aggregate records a ToolInvocationFailed with failure_code and re-raises the refusal type, which joins _REFUSED_AT_DISPATCH (executor.py:147, keyed by the raised type's name and tied to the ExpectedRefusal marker), so hangar.call.outcome is deny and the span is UNSET, as on the cold path. For a group, a member's refusal counts as that member's failure and selection moves on, as a member's failed start does today. A refusal counts as a failed start for backoff and the circuit breaker, like any failed start; the refusals that follow it keep their own reasons (not_revived_by_calls, circuit_open), the server's last start error is the refusal, and an endpoint change resets both (§4b). No new entry in _GATES.
6. Audit, from the socket, with explicit semantics
ToolInvocationCompleted and ToolInvocationFailed gain optional upstream_peer_spki: str | None; ToolInvocationFailed gains failure_code: str | None (default None, so stored events read back unchanged); and ToolInvocationFailed and ToolCallRefused gain optional observed_peer_spki, so the observed key is kept on the cold-start path too. The value is the digest the handshake kept on the socket of the connection that served this attempt, read through response.extensions["network_stream"].get_extra_info("socket") while the response is still open (the request is sent with stream=True, so the stream is read before the body is consumed) — not get_extra_info("ssl_object"), which is gone once the server closes the connection — and never from configuration, so a retry on a different connection records that connection's key and a group member's key is never attributed to a call another member served.
ToolInvocationCompleted and ToolInvocationFailed also gain optional declared_key_issuance, copied from the server's peer_identity.issuance at the time of the call and set only on a verified connection (table below). It is configuration, not an observation, and its name says so: it records what the operator declared about the keys, never that Hangar verified it.
| Case | upstream_peer_spki | Other fields |
|---|---|---|
| Refusal before any connection | absent | — |
| Transport failure (plaintext peer, TLS version, reset, timeout, protocol error) | absent | recorded as today |
| Key mismatch or no certificate | absent | observed_peer_spki set when a certificate was presented |
| Verified connection, call succeeded or failed at the application level | the verified key | declared_key_issuance when declared |
Exporters JSON/CEF/LEEF/syslog carry the fields next to caller_id; spans carry hangar.upstream.peer_spki. mcp-hangar why already reads the ToolCallRefused a cold-start refusal publishes; it is extended to read ToolInvocationFailed rows whose failure_code is upstream_peer_key_mismatch (refusal_log.py:26-32 today lists five refusal types and no invocation failures), and prints the expected key list and the observed key for both. It does not synthesise a ToolCallRefused for a warm refusal. Metric: mcp_hangar_upstream_peer_key_connections_total{server, outcome}, registered in _register_all_metrics() (#1059), counting every handshake of a key-pinned server, including the GET stream's.
7. The upstream identity of a key-pinned server is its key lineage
McpServer.upstream_identity() returns peer-key:<key_ref> for a key-pinned server, instead of remote:<scheme>://<host>:<port>. The identity must be the same on every replica and stable across a rotation, and it must be computable from configuration so that a server that has never connected has it too. A value taken from the connection, or derived from the key list, fails this: the list changes when the next key is added and again when the old one is dropped. key_ref is the operator's stable name for the key lineage, so a rescheduled pod with a new IP keeps its digest pins, and a key rotation keeps them too. Nothing is lost by not binding to the key itself: a key outside the key list is refused at connect (§5) before any tool is served. Changing key_ref is a different upstream, and its digest pins are recorded again. Giving an existing server a peer identity changes its identity the same way, so all its digest pins are recorded again; the upgrade note says so. issuance is not part of the identity: declaring or changing it does not touch digest pins.
Because key_ref is a label the declarer chooses, one rule keeps it from carrying digest pins it should not: key_ref is unique across servers, enforced by the peer-key name record (§9), not by a replica's local fleet, so two replicas registering concurrently cannot both succeed. A second rule is a guard against mistakes, not against the declarer: a single update whose key list shares no key with the stored list must change key_ref or is refused, so a wholesale swap cannot happen by accident in one step; a stepwise replacement ([A] → [A, X] → [X]) passes it, because the holder of mcp_servers:peer_identity is a trust root either way (§9b). For a configuration-file server the previous list is the one from that replica's previous load; the first load has none. Servers without a peer identity are unchanged.
8. A key-list change reaches every replica
Three rules make it safe:
- Every write of a key-pinned server's row carries the version it read, and a stale write is refused. Today every writer of the row saves a snapshot of the replica's whole local copy: the update handler for any field (
application/commands/crud_handlers.py:257-270), the L7 policy handler (:331), which the operator's policy delivery goes through, and the repositories'save()reads the current version and overwrites it in the same call (infrastructure/persistence/config_repository.py:119-150, and its PostgreSQL peer), so a replica whose copy is behind writes a removed key back and always wins. The three writers of a row are create, update and L7 policy (crud_handlers.py:163, 270, 331); configuration-file load, reload and gc write none, and discovery writes through create and delete. For a key-pinned server each of them instead reads the row with its version (get_with_version, which both backends already have,config_repository.py:371, joinsIMcpServerConfigRepositoryandIFleetWritergains a read), builds the new snapshot from that row plus only the fields it changes, and saves it withexpected_versionin one statement (UPDATE ... SET version = version + 1 WHERE mcp_server_id = ? AND version = ?, refused when no row matched). Create saves withexpected_version = None, meaning insert only: today create checks only the replica's local fleet (crud_handlers.py:119) andsave()falls through to an update when a row exists, including a soft-deleted one (config_repository.py:256), so a replica that does not hold the server overwrites its row. A soft-deleted row counts as absent only when its name is not in the peer-key name record (§9), or when the creator holdsmcp_servers:peer_identityand declares the storedkey_ref. The aggregate method is called only after the write succeeds, so a refused write leaves neither a changed local copy nor an uncollected event. A refused write is aConcurrentModificationError, which the API maps to 409; the operator retries its endpoint and policy deliveries on it. A removed key can therefore not be brought back by any concurrent write on another replica. The in-memory repository has no fleet writer (server/bootstrap/cqrs.py:73); §9 says what a key-pinned server needs there. - Every replica, including the one that handled the update, replaces its copy. The update publishes
McpServerUpdatedafter the row is written (crud_handlers.py:270-274), and it is persisted and tailed like any aggregate event. The fleet projection subscribes to it and, for a key-pinned server, when the row'stlsblock or endpoint differs from the configuration the local copy's client was built with, builds aCOLDcopy from the row (server_from_snapshot) and puts it in the local fleet in place of the old one. The displaced copy is retired: it refusesensure_readyand calls, and the replica's background worker closes its client and GET stream, so the projection itself publishes nothing and does no I/O beyond the row read, asHandlerKind.PROJECTIONrequires. This differs from a registration, which never replaces a running copy (fleet_projection.py:131-135), because a key-pinned server's key list is the one part of its configuration that must not wait for a restart. The replica that handled the update compares against the built client, not against its in-place-updated aggregate, so its own pooled connections are not left on the old list. - An unreadable row fails closed. Where the L7 precedent keeps the local copy when the row cannot be read (
fleet_projection.py:101-111,:161-166), a key-pinned server's copy is retired and its calls are refused until a later read succeeds, because the tail has moved past the event and would not deliver it again.
No new event type is needed. Versioned row writes and the projection of McpServerUpdated are general mechanisms that this record introduces for key-pinned servers only; applying them to every server is a separate decision. Exposure to a key that should no longer be trusted is therefore bounded by the time until the operator removes it from the key list plus the event-tail delay to each replica, not by anything an issuer such as a KBS does. This bound is for servers registered through REST or the operator; a configuration-file server changes on each replica when that replica reloads its configuration (application/commands/reload_handler.py).
tls.max_connection_age (default 1 h) caps the life of a connection. httpcore has no maximum connection age, only an idle keepalive_expiry, so the cap is a scheduled replacement of the connection pool. It bounds how long one connection serves without a fresh handshake against its replica's current list; it does not replace the propagation above: every max_connection_age the httpx.Client inside the server's HttpClient is replaced, the old one is closed once its in-flight requests complete, and the MCP session (_mcp_session_id) is kept, because it lives on HttpClient and not on the pool. The streamable-HTTP GET stream, which has no read timeout and is the connection most in need of the cap, is closed and re-opened on the new pool.
9. Permission, the peer-key name record and trust roots
A new permission mcp_servers:peer_identity, held by kubernetes-operator and admin only, is required to set or change peer_identity, to change the endpoint of a key-pinned server, to delete one, and to create a server under a name in the peer-key name record. mcp_servers:write and mcp_servers:lifecycle, which developer holds (auth/roles.py:197-203; delete needs lifecycle, server/api/mcp_servers.py:419), are not enough for any of these. The commands carry the authenticated caller, not a flag: CreateMcpServerCommand, UpdateMcpServerCommand and DeleteMcpServerCommand gain principal: Principal | None, which the REST routes fill from the request (server/api/mcp_servers.py:348, 392, 423; they build commands field by field, so a request body cannot supply it). The handler asks the authorizer for mcp_servers:peer_identity before any creation of a name in the peer-key name record, any tls or endpoint change of a key-pinned server, and any deletion of one; with no principal the change is refused. A boolean set by the caller would not be a boundary, because any in-process code can build a command. The only other dispatchers are core discovery and discovery approval (server/bootstrap/discovery.py:272, 318; hangar_approve reaches the same path through orchestrator.on_register, :92), which have no principal and are therefore refused, fail-closed; discovery's repeated attempt to delete a server that §4f moved is logged at info, not as an error. With authentication disabled the authorizer allows everything (mcp_servers.py:67-68), so on such a gateway any client of the API can set or change a peer identity; the documentation says so. hangar_load (which needs only mcp_servers:write, tool_permissions.py:94) dispatches to LoadMcpServerHandler, which writes the runtime store and has no access to the configuration store; it is given read access to the peer-key name record and refuses to hot-load a name in it.
The peer-key name record. The record holds one entry per name given a peer identity, with its key_ref. It has two sources:
- The configuration file. Every name the file declares with
peer_identityis in the record on every backend, derived from the file at load and at reload, because the file is the operator-trusted source of those servers (ADR-021). This is what lets a single gateway with an in-memory repository serve key-pinned servers from its configuration file. - REST and the operator. A name given a peer identity through a command is stored in its own table (
peer_key_names(name PRIMARY KEY, key_ref UNIQUE)): deleting a server only disables its configuration row (config_repository.py:256), but a disabled row is rewritten by the next create of that name, so the record cannot live in it. The entry is written together with the configuration row by a new two-statement write onIMcpServerConfigRepository, implemented by each backend in one transaction (ADR-019: the existingTransactionalMcpServerConfigRepository,infrastructure/persistence/unit_of_work.py:20, is SQLite-only and unused on the fleet path, so it is not the mechanism): an insert that does nothing on conflict, then a read of the existing row to confirm thekey_ref. A failed write refuses the declaration rather than serving it. The two sources are checked against each other: a command whose name orkey_refmatches an entry derived from the configuration file is refused, and a file that declares a storedkey_refunder another name is refused at load. The entry survives the deletion of the server and is removed only by a holder ofmcp_servers:peer_identity. On a shared backend every replica reads it from the shared store, so it survives restarts and is the same on every replica. An in-memory repository has no projection and no tail (server/bootstrap/event_handlers.py:202-203,server/bootstrap/coordination.py:142-152) and keeps nothing across a restart, so declaring a peer identity over REST or from the operator is refused there.
Configuration-file load and reload write no row and bypass the handlers (application/commands/reload_handler.py:197-210), so they check the stored entries themselves: an entry that declares a server without a peer identity under a stored name is refused at load and at reload. The record is what keeps a deleted key-pinned server from being re-created as a plain one, what keeps discovery off a key-pinned name (§4c), and what makes key_ref unique (§7).
Trust roots, named:
- (a) for
issuance: kbs-attested, the KBS operator and its reference values; - (b) whoever holds
mcp_servers:peer_identityor writes the config file — they can list a key they control and declare anyissuance, so this is operator-declared trust, not independently administered trust, and the audit record says what the operator declared, not what a third party verified; - (c) for
issuance: kbs-attested, the guest TCB after key release — a compromised guest can exfiltrate K; - (c') for keys the operator manages, wherever the operator keeps them;
- (d) the operator-to-core channel for the integrity of the key list (§4);
- (e) write access to the shared configuration store and the event store, which hold the key lists, the peer-key name record and the rows every replica rebuilds from.
Independently administered trust (a key list the registrant cannot write) is a later decision and is not made here.
10. Defaults, compatibility, naming
No behavioural change for servers without tls.peer_identity; existing TLS behaviour, including the verify_ssl: false warning (launchers/http.py:196-210), is untouched. The fleet snapshot carries no tls today (fleet_snapshot.py:27-76, :124-178): a server rebuilt from it on restart or on a peer replica would come back without its peer identity and be served as a plain one. McpServerConfigSnapshot therefore gains the tls block (it holds no secret), server_from_snapshot restores it, and a row whose tls.peer_identity cannot be restored is refused, not rebuilt as a plain server. Additive CRD fields, no API version bump. McpServer.provenance (HUMAN | DISCOVERY, domain/value_objects/provenance.py) keeps its SSRF meaning (ADR-021). The word "attested" appears in configuration and audit only as the declared issuance value kbs-attested; nothing is named attestation result, measurement or verifier.
The change spans config and validation; a new transport verification path and its propagation through the start, call, renegotiation and stream paths; the registration body, command and REST fields; the fleet snapshot, versioned writes and the projection of McpServerUpdated; the peer-key name table on both backends; the operator's registration flow and capability check; one admission policy for issuance: kbs-attested; discovery refusal; event schema and exporters; why; one metric; one permission; the tests below.
Consequences
Positive
- A call to a key-pinned server is carried by a connection that provably ends at one of the operator's listed keys, verified before any byte is sent, and the audit record names the key read from the socket — a claim Hangar can stand behind, unlike a verifier result with a shelf life or a CA-anchored name whose meaning lives in an issuance policy Hangar cannot see.
- The binding is useful without a TEE: it closes IP reuse by the CNI, keeps digest pins across reschedules, and records which key answered, for any remote upstream that serves TLS with a stable key.
- Attestation enters as one declared source of keys. Its specific rules (the KBS premise, Guarantee A, the runtime-class and mutable-image rules) apply only to
issuance: kbs-attested, so a failure of that premise changes one value, not the mechanism. - RA-TLS and independently administered key lists have a defined place: a second peer-identity kind behind the same handshake check, and a different writer of the same key list.
- No new gate in
_GATES, no new event type, no verifier client, no CA handling, no Secret reads in the operator. For operator-managed keys the trust roots are who may write the key list, the channel that carries it and the stores that hold it (§9b, d, e). - Nothing changes for servers without
tls.peer_identity.
Negative
- For
issuance: kbs-attested, the KBS operator and its reference values are a trust root of Hangar's audit trail. A KBS policy that releases a leaf key to the wrong workload makes that server's declared issuance meaningless, and Hangar cannot detect it. - Operator-declared trust only: whoever holds
mcp_servers:peer_identityor writes the config file can list a key they control and declare anyissuance. The record names it and narrows it (developercannot), but does not close it; independent administration is a later decision. - For
issuance: kbs-attested, Guarantee A, not B: a key exfiltrated from a compromised guest or a guest whose measurement drifted after release keeps working against Hangar until the operator removes it from the key list; a revocation at the KBS alone changes nothing on Hangar's side. Documented; not solvable on Hangar's side without RA-TLS. - The key list's integrity depends on the operator-to-core channel and on the shared stores. The operator refuses plaintext
hangar.urlfor key-pinned CRs, but the stores remain a trust root. - A key-pinned server whose key is not in the list is unavailable, fail-closed, with no "temporary" plaintext mode. Rotation requires the operator to publish the next key before switching; a missed step takes the upstream down. A change to the
tlsblock or the endpoint of a key-pinned server stops it toCOLDon every replica. - Upstreams must serve TLS 1.3 with a stable key; a plaintext or TLS 1.2 upstream is a transport error, indistinguishable in the audit trail from an unreachable one. For
kbs-attested, serving TLS with the KBS-released key is integration work on the upstream image, outside Hangar (CoCo CDH supports resource retrieval). - Giving an existing server a peer identity re-records all of its digest pins.
- A key-pinned server declared through REST or the operator needs a shared, durable backend; one declared in the configuration file does not. During a rolling upgrade an older replica does not hold a key-pinned server at all, by design.
- A name once given a peer identity through a command keeps its
key_refreserved after the server is deleted, until a holder ofmcp_servers:peer_identityremoves the entry. - Every write of a key-pinned server's row is versioned, so a writer with a stale copy is refused and must re-read.
- New operator work: registration before the pod, endpoint maintenance across reschedules, the capability and
https://checks, one admission policy forissuance: kbs-attested, and, for container-mode CRs declaringkbs-attested, the dependency onruntimeClassName. New core work: transport verification and its propagation, event fields, the fleet snapshot and projection, the peer-key name record,why, permission, REST/command fields. - Three things a reader will call "attestation" must stay apart in every document: build provenance of Hangar's artefacts, the digest of a tool contract, and the declared issuance of an upstream's key.
Neutral
- Every acceptance test except T26 runs without hardware (a self-signed leaf, a local TLS server, envtest); the rows that need two replicas over PostgreSQL belong to
tests/live, which the unit and integration suites do not run; no security statement aboutkbs-attestedmay rest on a run without a TEE, and the documentation must say so. - The verifier-result design is rejected in the form described in Alternative 1; a design with a different binding would need its own record.
- Hangar-in-TEE (Alternative 5) remains possible as a separate record.
What this decision does not establish
Rows for docs/security/VERDICT_LIMITS.md:
| Verdict | Establishes | Does not establish | Operator's side |
|---|---|---|---|
| Verdict on a key-pinned server | The connection that carried this call terminated at a private key whose SPKI digest was in the replica's key list at connect time. The observed key is recorded from the socket of the connection that served the attempt. | Where the key came from or who else holds it; that every replica has already applied the latest key list; that the key list was written by anyone other than the holder of mcp_servers:peer_identity, the config file or a writer of the shared stores (operator-declared trust); that Hangar's own policy is correct; that the upstream's code is benign or its outputs trustworthy; where the upstream runs; what Kubernetes privileges or runtime configuration its pod has; that the operator-to-core channel carrying the key list was not tampered with; that a tool listed in tools/list has a currently reachable key-pinned upstream (catalogues are cached; only connections are checked); that Hangar itself runs in a trusted environment; that the event log is independent of the operator. | Keeps the private keys where only the upstream can use them; keeps the key list to current+next during rotation and removes a key as soon as it is retired or suspected; sets tls.max_connection_age to the longest time a single connection may go without being re-checked; runs hangar.url over TLS; puts the pod CIDR in ssrf.trusted_networks; never runs the same server as plaintext "temporarily"; separates the identity holding mcp_servers:peer_identity from the identity managing the keys where the threat model requires it. |
Additionally, when the server declares issuance: kbs-attested | Only that the operator declared that a KBS released those keys only after attestation of the workload they were provisioned for (Guarantee A); the event records this as declared_key_issuance. Hangar verified none of it. | That the declaration is true; that the guest is now in a verified TEE state (Guarantee B); that the key has not been exfiltrated from the guest since release; that a key revoked at the KBS is refused while it is still in the key list; that the KBS policy or reference values are correct or current; that the application container was in the hardware measurement. | Operates or trusts a KBS that releases per-workload leaf keys (never a CA key) only after attestation, with short validity and re-release tied to re-attestation; provisions the key pair as a KBS resource and reads the SPKI from it, not from the running pod; removes a key from the key list as soon as the KBS revokes it; never sets hangar.io/allow-mutable-image on the CR. |
Acceptance tests
Each row is an acceptance criterion of the clause it names. All rows except T26 run without hardware: a self-signed leaf generated in the test, a local TLS 1.3 server that records what it received, and envtest for the operator.
| # | Test | Clause | Layer | Needs hardware |
|---|---|---|---|---|
| T1 | Key-pinned server, peer key in the key list: call succeeds; ToolInvocationCompleted.upstream_peer_spki equals the key kept at handshake, not the config, also when the server answers with Connection: close | §5, §6 | integration (local TLS server) | no |
| T2 | Peer key not in the key list, chain otherwise valid (signed by a CA given in ca_cert_path, which is ignored): upstream_peer_key_mismatch; a peer presenting no certificate: the same code; a plaintext peer and a TLS 1.2-only peer: ConnectError, retried and recorded as a transport error, never as a refusal; the test server asserts zero application bytes received in every case | §1, §5 | integration | no |
| T3 | Group: member A presents a wrong key, member B serves; A's refusal records observed_peer_spki, B's call records B's key; no cross-attribution; neither _post_with_retry nor the executor retry retries the refusal; a connection reset during handshake is retried as ConnectError and recorded as a transport error | §5, §6 | integration | no |
| T4 | endpoint: http:// with peer_identity: refused at load naming both halves | §4 | unit | no |
| T5 | peer_identity with an empty spki_sha256, a key pin not in canonical form (base64, uppercase, other algorithm), no key_ref, mode: subprocess/own-container mode, or verify_ssl: false: refused at load; a server with tls and http blocks keeps its key list; building a client for a key-pinned server without its key list raises | §4 | unit | no |
| T6 | Pod IP reused by a different server with a different key: refused on the next connect with upstream_peer_key_mismatch; the same during MCP session renegotiation (#1852), recorded with that code, not as an RPC error | §1, §5 | integration | no |
| T7 | Through the command bus against a real local TLS server, not a mocked start: COLD server, key mismatch at first contact: _gate_cold_start records hangar.gate.outcome=deny, hangar.gate.reason=upstream_peer_key_mismatch, never start_failed; ToolCallRefused.observed_peer_spki set; span UNSET; one batch_call_refused line; a second caller waiting on the same start receives the same code; the same when the refusal happens on the connection that carries notifications/initialized, which fails the start instead of finalising it | §5, §6 | integration | no |
| T8 | Warm server whose peer changes key: invoke_tool records ToolInvocationFailed with failure_code and observed_peer_spki and re-raises the refusal type, not ToolInvocationError; hangar.call.outcome=deny, span UNSET, one batch_call_refused line; relay_request (prompt, resource), the tools refresh, recheck_catalogue and health_check record the failure and re-raise the same type | §5, §6 | integration | no |
| T9 | GET stream to a peer with a wrong key: the loop stops retrying on that pool, logs and counts the metric; it re-opens after the pool or server is replaced | §5, §8 | integration | no |
| T10 | Rotation: key list [old, new], server switches key: calls continue and digest pins are still honoured; max_connection_age elapsed: the pool is replaced, an in-flight request completes, the MCP session id is kept and its 404 recovery still works, the GET stream re-opens on the new pool | §7, §8 | integration (local TLS server, clock) | no |
| T11 | Two replicas over one shared PostgreSQL store: a key-list, tls or endpoint update on A records McpServerUpdated; B's projection replaces its copy with a COLD one built from the row and publishes nothing; the retired copy refuses ensure_ready; B's next call to a peer on the removed key is refused; on A, the replica that handled the update, a pooled connection on the removed key is not reused | §8 | live (two replicas, PostgreSQL) | no |
| T11a | Concurrent writes on two replicas: A removes a key while B, from a stale copy, updates the description, delivers an L7 policy, or updates the endpoint; every stale write is refused by the row version with 409 and changes neither the row nor B's local copy, and the removed key is never trusted again on any replica; a create on a replica that does not hold the server, and a create after a delete without mcp_servers:peer_identity and the stored key_ref, are refused instead of overwriting the row; with both, the create after a delete succeeds | §8 | live (two replicas, PostgreSQL) | no |
| T11b | The projection cannot read the row of a key-pinned server: its copy is retired and calls are refused until a later read succeeds | §8 | unit | no |
| T11c | A row stored with mode: remote-peer-key given to the previous release's server_from_snapshot raises ValueError, the event bus's handler fault barrier drops the delivery and recovery skips the row; the current release reads it back as remote with tls.peer_identity, and refuses one without a tls.peer_identity block; a plain row under a name in the peer-key name record is refused at recovery and in the projection; the current release advertises the capability the operator checks (§4e) | §4 | unit | no |
| T12 | Server without tls.peer_identity keeps today's TLS behaviour including the verify_ssl: false warning; no audit field, no metric series; peers' handling of updates to it unchanged; existing TLS tests unchanged | §10 | unit | no |
| T13 | _GATES length and order unchanged; _GATE_REASONS contains the one new code and nothing else new; the type is not in _GATE_ERRORS; it derives from ClientError and not from OSError, carries ExpectedRefusal, and is in _REFUSED_AT_DISPATCH; the key-pinned context has minimum_version TLS 1.3 | §5 | unit | no |
| T14 | CRD: peerIdentity with empty spkiSha256, a non-canonical key pin, or no keyRef: rejected by CEL; issuance outside the enum: rejected; issuance: kbs-attested with mode: container and no runtimeClassName (once the field exists): rejected; mode: remote without runtimeClassName: admitted; operator RBAC still reads no Secrets | §4 | envtest + existing RBAC test | no |
| T15 | A CR with issuance: kbs-attested carrying hangar.io/allow-mutable-image is denied with imageDigestPolicy at off, at warn, and with no ConfigMap; a key-pinned CR without issuance carrying it is admitted as today | §4 | envtest | no |
| T16 | Operator registers a key-pinned CR before its pod exists, without an endpoint, and sets the endpoint once the pod has an IP; clears it on observing pod deletion and sets the new one after a reschedule; updates, not creates, a server that already exists from discovery, and its provenance becomes HUMAN; emits no discovery annotations for it; does not register against a core that does not advertise the capability or over a non-https:// hangar.url | §4 | envtest | no |
| T17 | Core discovery on the lease holder refuses a pod under a name in the peer-key name record, whatever its annotations, before and after the pod exists and after a replica restart, logging discovered_peer_key_upstream; an endpoint update with the same key_ref resets backoff and the circuit breaker | §4 | core unit | no |
| T18 | MCPDiscoverySource with a peerIdentity in its providerTemplate is rejected by CEL, for Pod- and Namespace-type sources | §4 | envtest | no |
| T19 | Exporters JSON/CEF/LEEF/syslog write upstream_peer_spki, failure_code, observed_peer_spki and declared_key_issuance; absent when not set; a stored ToolInvocationFailed without failure_code reads back; CEF mapping documented | §6 | unit | no |
| T20 | Metric registered in _register_all_metrics() and on /metrics after one connect, with GET-stream handshakes counted | §6 | unit | no |
| T21 | Provenance enum and serialisation unchanged; the tls block round-trips through snapshot_of/server_from_snapshot, and a second replica rebuilt from the snapshot serves the server with the same peer identity; a snapshot row whose tls.peer_identity cannot be restored is refused | §10 | unit | no |
| T22 | import-linter: no attestation SDK imported anywhere; SPKI hashing and peer verification live only in http_client.py | §3 | lint | no |
| T23 | A developer key cannot set or change peer_identity, change the endpoint of a key-pinned server, delete it, or create a server under a name in the peer-key name record — over REST, through hangar_load, or through discovery approval; a command built in-process with no principal is refused; a config-file entry declaring a server without a peer identity under a stored name is refused at load and at reload; kubernetes-operator and admin can; a stored entry survives the deletion and a restart and is seen by a second replica; hangar_load of a name in the record is refused; on an in-memory repository, declaring a peer identity over REST is refused and a config-file server with a peer identity is served | §9 | unit + live (two replicas, PostgreSQL) | no |
| T24 | upstream_identity() returns peer-key:<key_ref> before any connection, after a rotation and on a second replica; digest pins bound to it survive an endpoint change and a rotation; a different key_ref behind the same endpoint is a different identity; giving an existing server a peer identity changes its identity; changing issuance does not; two replicas registering the same key_ref concurrently on PostgreSQL: exactly one succeeds; a single update whose list shares no key with the stored list is refused unless key_ref changes | §7, §9 | unit + live (two replicas, PostgreSQL) | no |
| T25 | mcp-hangar why lists a cold-start refusal (ToolCallRefused) and a warm refusal (ToolInvocationFailed) of a key-pinned upstream, each with the expected key list and the observed key; it never shows a synthesised ToolCallRefused for a warm refusal | §6 | unit (SQLite) | no |
| T26 | End-to-end on kind with kata-qemu-coco-dev, Trustee sample attester, KBS releasing a pre-provisioned leaf key to the guest; operator lists its SPKI with issuance: kbs-attested: T1 passes and the event records declared_key_issuance; the docs page states this proves integration, not security | §2 | nightly e2e (KVM) | yes |
| T27 | issuance outside the closed set is refused at load; declared_key_issuance is recorded on invocation events only when declared, and never on a server without it; no field, log line or span names the declaration as verified | §2, §6 | unit | no |
Alternatives Considered
1. Verifier result as a verdict input with TTL
- Rejected: Hangar would consume an Attestation Result (EAT) from Trustee AS, bound to
upstream_identity(), withmaxAge, a new_gate_attestationin_GATES, anAttestationResultRecordedevent inpin_ledger, persistence on the fleet snapshot and propagation via the event tail. The defects that decide it: (a) for operator podsupstream_identity()isremote:http://<pod_ip>:<port>, so the binding is to an IP the CNI reuses and the snapshot/replay machinery is dead weight; (b) with TTL and no channel binding, a pod-create attacker who inherits the IP insidemaxAgereceives calls audited asattestation_status=ok, a claim Hangar could not stand behind; (c) verification as aPROJECTIONhandler onMcpServerStartedviolatesHandlerKind(network I/O, publishes), and as anEFFECTit runs once per replica because every replica emitsMcpServerStarted; (d) a_GATESentry cannot govern prompts/resources or group listings, which are decided elsewhere (is_governed_allowed,fastmcp_server/flat_tool_projection.py:235); (e)referenceValues[kind=image-digest]is not something an SNP/TDX verifier attests.
2. KBS-only: the attested guest gets its downstream credential from the KBS, Hangar is unchanged
- Rejected as sufficient alone; kept as the baseline for attested sources: §2 preserves it. It gives no channel binding: Hangar still dials an IP and cannot tell an attested pod from one that inherited its address, and the audit trail cannot say which it was. The decision adds exactly one thing for this case: the KBS also releases the upstream's TLS leaf key, and Hangar checks it against the key list.
3. CA-anchored identity: pin an issuer CA and match a SAN
- Rejected: three independent defects. A CA private key handed to workloads lets any of them mint any name; a CA kept at an issuer makes the SAN a statement of an issuance policy Hangar cannot inspect, so
peer_identity.sanwould mean whatever the issuer's rules mean; and the client's hostname verification (sni_hostname,http_client.py:267-270; CPython matches DNS/IP SANs only) rejects a URI SAN and cannot match a static leaf against a CNI-assigned pod IP. Key pinning (§1) needs no chain, no name and no issuance policy on Hangar's side, and shrinks the trust root to "who may write the key list". Cost: rotation is a key-list change, and the operator must know the SPKI before the upstream serves it.
4. Sidecar or mesh mTLS termination with SPIFFE policy, Hangar untouched
- Rejected as the mechanism; compatible for enforcement, loses the audit field: a mesh that terminates mTLS beside the upstream and admits only the SVIDs it should enforces the same property without any Hangar change; Hangar then sees plaintext to a local sidecar and cannot record which key answered. A deployment that already runs such a mesh may prefer it; this record does not stand in its way and does not claim its audit guarantees for it.
5. Hangar itself in a TEE, policy measured, log signed by an enclave key
- Rejected: it changes the product from a fleet-level policy enforcement plane into an attested single-process gateway — the same widening-under-pressure that ADR-010 refused for SaaS. It needs an attestation provider abstraction, key release, freshness and rollback protection for Hangar's own log. HA (ADR-020) assumes N replicas over one log; enclave-held signing keys per replica need either a migratable sealed key or N keys and a verifier that knows them. Nothing here forbids a later record on it; it must supersede this clause explicitly.
6. Hangar issues the nonce and verifies Evidence
- Rejected: MCP has no in-band channel from an intermediary to the upstream's Attestation Agent; a nonce would travel out of band (
_meta, a header), which is a protocol change under discussion elsewhere, not a Hangar change. SNP/TDX quote verification in core duplicates Trustee and Veraison and brings platform cryptography into the domain.
7. RA-TLS (X.509 with the quote in an extension) to the upstream
- Deferred: it is the stronger form of an attested source: the client verifies the quote inside the handshake rather than trusting a KBS-gated key release. It needs an RA-TLS-aware client and a verifier at handshake time, which §3 rules out for this record. The KBS-brokered form gives the same channel binding with the client Hangar already has, at the cost of trusting the KBS operator (§9a). If an upstream ecosystem standardises RA-TLS, a record can add it as a second
peer_identitykind behind the same handshake check (§5) and audit fields (§6), superseding §3 for that kind.
8. OPA-only: the fact in context, policy in Rego
- Rejected: OPA authorization for
hangar_callruns before target resolution and with an empty context (server/tools/batch/__init__.py:245, ADR-028 §2); filling it would mean restructuring_authorize_callsto run after resolution, which is an ADR-028 amendment, and OPA is optional, so an installation without it would have no enforcement at all. Connect-time enforcement works in every configuration.
9. SPIRE with a TPM/TEE node attestor issuing SVIDs
- Rejected as a separate mode; compatible under a condition: a SPIRE-issued SVID is a key like any other; if the operator lists its SPKI, nothing in §1 changes. Two caveats: declaring
issuance: kbs-attestedfor SPIRE-issued keys is wrong unless SVID issuance is gated by an attestation policy that satisfies the declared trust model — a TPM node attestor proves the node booted a measured stack, not that this process runs in a TEE; and SVID rotation is frequent, so the key list must be fed from the SPIRE trust bundle or the deployment must pin the SPIRE CA instead — which is Alternative 3 with its SAN problems. The record keeps key pinning as the only mode and lets SPIRE deployments decide whether the rotation cost is acceptable.
10. Admission-time-only verification (Kata policy hash gates the KBS; Hangar unchanged)
- Rejected as sufficient alone: same reason as Alternative 2. It secures what runs; it does not bind what Hangar dials to what runs.
11. Peer identity declared on MCPServerGroup
- Rejected: the connection is per upstream; a group is a projection over members and a declaration there would have to be fanned out — the key-mismatch class of #1036/#1040. A group may contain members with and without a peer identity; each member's
tlsblock decides.
12. A new CRD API version for the new fields
- Rejected:
tlsis additive and optional; absent means today's behaviour. Optional fields added to a served version are backward compatible for existing objects and clients, which is all an API version bump would buy.
13. Defer the decision until a deployment asks
- Rejected: deciding the contract before a deployment depends on it keeps that deployment from re-deriving it under pressure.
14. Propagate key-list changes only through replica restarts
- Rejected: a removed key would stay acceptable on every replica that has not restarted, with no bound at all. The fleet projection already applies L7 policy on every replica; applying
McpServerUpdatedthe same way bounds the window by the event tail.
15. Offer the key binding only to attested upstreams
- Rejected: the handshake check, the audit from the socket, the stable identity and the propagation rules do not depend on where the key came from. Restricting them to attested sources would deny channel binding against IP reuse to every deployment without a TEE, put an attestation label on records whose guarantee is the key binding, and make the whole mechanism wait on a premise about one issuer that the mechanism itself does not use.
References
- ADR-004, ADR-010, ADR-013, ADR-018, ADR-019, ADR-020, ADR-021, ADR-028, ADR-029 (§5 adds one reason code under its rule)
- mcp-hangar/mcp-hangar:
domain/model/mcp_server.py:578-597(upstream_identity),1455-1462,2099-2104,2265-2270,server/tools/batch/executor.py:147, 473-476, 2234, 2512, 2558, 2908-2911, 3104-3117,domain/contracts/event_bus.py:13-68,domain/services/fleet_snapshot.py:27-178,application/event_handlers/fleet_projection.py:78-166,application/commands/crud_handlers.py:257-274,domain/contracts/persistence.py:192-204,infrastructure/persistence/unit_of_work.py:20,infrastructure/persistence/config_repository.py:119-150, 256, 371,infrastructure/persistence/recovery_service.py:151,domain/value_objects/mcp_server.py:84-89,server/bootstrap/discovery.py:92, 272, 318,server/bootstrap/cqrs.py:73,application/commands/reload_handler.py:197-210,infrastructure/event_bus.py:393,domain/value_objects/provenance.py,server/config.py:1843,domain/events/enforcement.py:351-357,infrastructure/discovery/kubernetes_source.py:250-287,infrastructure/launchers/http.py:35-51, 172-212,http_client.py:425-495, 816-827, 968, 1035,server/config.py:1279,server/tools/batch/__init__.py:245,server/tools/tool_permissions.py:94,auth/roles.py:176-203,server/api/mcp_servers.py:319-360, 419, 526-528,observability/conventions.py:145-170,errors.py:51,approvals/api/routes.py:257-258,.github/workflows/release.yml:232,tests/ci/test_a_release_carries_verifiable_artifacts.py - mcp-hangar/mcp-hangar-operator:
api/v1alpha2/mcpserver_types.go,api/v1alpha2/mcpegresspolicy_types.go:103-119,api/v1alpha2/mcpdiscoverysource_types.go:31-32,47,pkg/provider/builder.go:83-107, 192-195,pkg/hangar/client.go:397-401, 485, 582-583,internal/webhook/image_digest.go:9-21,internal/controller/mcpserver_registration.go:222-224,internal/controller/mcpdiscoverysource_controller.go:751-753,config/admissionpolicy/image_digest.yaml,config/admissionpolicy/params.yaml - mcp-hangar/docs:
security/VERDICT_LIMITS.md,operations/COMPLIANCE_POSTURE.md - RFC 9334 — RATS Architecture; RFC 9711 — Entity Attestation Token; RFC 8446 — TLS 1.3
- Confidential Containers — Trustee; Confidential Containers — architecture
- MCP SEP-2809 — Attested Tool-Server Admission (modelcontextprotocol/modelcontextprotocol#2809)
- Forough, Kogias, Haddadi — When Agents Handle Secrets; Metere — Attested Tool-Server Admission