Stop a rescheduled pod from answering as your MCP server

The problem

Your gateway reaches an MCP server at an address. In a cluster, that address is a pod IP, and the next pod to be scheduled can inherit it. Whatever answers there gets the tool calls, with their arguments. A certificate from a CA does not help: the IP has no name for a CA to vouch for, and many workloads can hold a certificate the CA would sign.

Pinning the server by its key settles who answers. This walkthrough shows the refusal on one machine first, then declares the same pin on Kubernetes. The concepts are in Pin an MCP server by its key.

Before you start

One Python environment, with Hangar and the MCP SDK it brings along (2.x):

python -m venv .venv && . .venv/bin/activate
pip install "mcp-hangar==2.32.0"

Run every command below in that environment. The steps were run against mcp-hangar 2.32.0, with the MCP SDK 2.3.0 it installs.

1. Two keys, one address

Make two keys and self-signed certificates for 127.0.0.1. One belongs to your server; the other plays the pod that inherits its address. Self-signed is fine: for a key-pinned server, Hangar consults no CA.

for k in served other; do
  openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
    -pkeyopt ec_param_enc:named_curve -nodes \
    -keyout $k.key -out $k.crt -days 2 -subj "/CN=payments" \
    -addext "subjectAltName=IP:127.0.0.1"
done

The key list holds the SHA-256 of each certificate’s SubjectPublicKeyInfo, written as sha256: and 64 lowercase hex characters. Print the one for served.crt:

echo "sha256:$(openssl x509 -in served.crt -pubkey -noout \
  | openssl pkey -pubin -outform der | openssl dgst -sha256 -hex | awk '{print $NF}')"

2. A server, and a gateway that knows its key

Any MCP server over HTTPS will do. This one has a single echo tool, on the MCP SDK 2.x (on 1.x the class is FastMCP):

# server.py: python server.py served   (or: python server.py other)
import sys
import uvicorn
from mcp.server.mcpserver import MCPServer

mcp = MCPServer("payments")

@mcp.tool()
def echo(text: str) -> str:
    """Echo the text back."""
    return text

uvicorn.run(mcp.streamable_http_app(), host="127.0.0.1", port=8443,
            ssl_keyfile=f"{sys.argv[1]}.key", ssl_certfile=f"{sys.argv[1]}.crt")

Give the gateway the server and its key list. Put in the hash you printed for served.crt; yours will differ from any shown on this page. The event store is there so that mcp-hangar why can read the refusal later.

# demo.yaml
mcp_servers:
  payments:
    mode: remote
    endpoint: https://127.0.0.1:8443/mcp
    tls:
      peer_identity:
        key_ref: payments-tls
        spki_sha256:
          - sha256:<the hash you printed for served.crt>
event_store:
  driver: sqlite
  path: ./events.db

python server.py served &
mcp-hangar --config demo.yaml serve --http --port 8099 &

Call echo through the gateway’s hangar_call tool. Any MCP client pointed at http://127.0.0.1:8099/mcp will do; this one uses the SDK you already have:

# call.py
import asyncio
from mcp.client.session import ClientSession
from mcp.client.streamable_http import streamable_http_client

async def main():
    async with streamable_http_client("http://127.0.0.1:8099/mcp") as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            result = await session.call_tool("hangar_call", {"calls": [
                {"mcp_server": "payments", "tool": "echo", "arguments": {"text": "hello"}},
            ]})
            print(result.content[0].text)

asyncio.run(main())

python call.py

It is allowed:

"success": true   "text": "hello"

3. Another key at the same address. The same call is refused

Stop the server, and start the impostor on the same address and port:

kill %1
python server.py other &

Run python call.py again. The same call, with nothing about it changed:

"success": false
"error": "upstream TLS key is not in the key list"
"error_type": "UpstreamPeerKeyMismatchError"

The gateway refused inside the TLS handshake, before it wrote a byte of the request. The impostor received a TLS handshake and nothing else: never the call, never its arguments.

4. See which key answered

mcp-hangar why reads the refusal from the event store. Its output, trimmed to fit:

$ mcp-hangar --config demo.yaml why --db events.db
│ Time (UTC)          │ Event    │ Server   │ Tool │ Gate         │ Reason                      │
│ 2026-10-20 20:41:11 │ 9000edaf │ payments │ echo │ upstream_tls │ upstream_peer_key_mismatch  │
  9000edaf  upstream presented sha256:05e1fe90...0cb87b  listed now sha256:06eeb576...4e3901

It names the key the peer presented next to the keys your configuration file lists now. The audit record carries the same key in every SIEM export, under each format’s own field name; in JSON lines it is observed_peer_spki.

The gateway’s metrics count the handshakes by outcome, one per connection, and a call can open more than one:

curl -s http://127.0.0.1:8099/metrics | grep upstream_peer_key

mcp_hangar_upstream_peer_key_connections_total{...,outcome="verified"} 2.0
mcp_hangar_upstream_peer_key_connections_total{...,outcome="refused"} 2.0

5. The same pin on Kubernetes

With the operator (0.17.32 or newer), the key list is part of the MCPServer:

apiVersion: mcp-hangar.io/v1alpha2
kind: MCPServer
metadata:
  name: payments
spec:
  mode: remote
  endpoint: https://payments.payments.svc:8443/mcp
  tls:
    peerIdentity:
      keyRef: payments-tls
      spkiSha256:
        - sha256:<your server key's hash>
    maxConnectionAge: 1h

The apiserver refuses a malformed hash, a missing keyRef and an http endpoint, with no webhook involved. For a mode: container server, the operator runs the pod and registers it at https://<pod IP>:8080. The pod gets no discovery annotations, so core never finds it as a plain server. The server must serve TLS on port 8080 with a listed key.

The operator registers a key-pinned server only when these hold. A failure shows as condition Degraded on the CR, with this reason:

An operator started without --hangar-url cannot register anything. It holds a key-pinned container CR with Ready=False and reason PeerIdentityNotCarried, and creates no pod.

Core checks every endpoint registered over the API against its SSRF policy, so list the private networks it may resolve into. A remote server addressed by a .svc name resolves to a ClusterIP in the Service range; a mode: container server is reached at its pod IP in the pod range. The ranges below are common defaults; use your cluster’s:

ssrf:
  trusted_networks:
    - "10.96.0.0/12"    # Services, for mode: remote
    - "10.244.0.0/16"   # pods, for mode: container

6. Rotate the key

Rotation is two edits of spkiSha256:

1. Add the next key beside the current one. Roll out the workload with the new key.
2. Remove the old key.

Each replica picks up an edit without a restart, when the change reaches it. It then rebuilds its copy of the server and closes the connections made under the old list. One edit cannot replace the whole list: a list that shares no key with the current one is refused, so a typo cannot cut every trusted key at once. A new keyRef starts a new lineage, and digest pins bound to the old one have to be recorded again.

What you just did

You pinned a server by its key, then let something else take its address, and watched the gateway refuse it in the handshake. The call was the same both times, and so was the address. Only the key changed.

The mechanism has no model in it. A key is on the list or it is not. That is also its limit: whoever writes the list decides which keys are trusted, and a listed key that is stolen is trusted until you remove it. The concept page says what a key pin does not establish.