Proxy HA — the last data-plane SPOF

Design 2026-08-11; built, deployed and rehearsed the same day. Status: live — two proxies on two nodes behind the health-followed VIP. 2026-08-13: failover moved from the 10 s reconcile tick to a 1 s watchdog — measured 1.2 s (node tables) / 1.7 s (VIP table) from socket death, both directions of the recovery proven, see the second rehearsal record below.

Rehearsal record (2026-08-13, fast watchdog, players absent)#

• Kill: systemctl stop mc on the incumbent (CT 200) issued at node-clock …793823; systemd confirmed the unit down by …794600 (socket dead somewhere in that 780 ms).
• Node table flip (core-01's own ip conduit, local 200 → survivor 226): …795794 — 1.2 s after socket death, sampled at 120 ms.
• VIP table flip (inside CT 190): …796291 — 1.7 s after socket death. Both layers moved in one veto event; the difference is pct-exec cost on the replica apply.
• Recovery: systemctl start mc at …820471; node table returned to local preference at …842066 (+21.6 s = velocity boot to port-bind plus one clearing probe). The VIP monitor stayed silent throughout recovery — sticky held, the survivor kept the port.

Mechanism (lib/firewall.ts): the leader stashes each firewall pass's exact inputs and runs a 1-second watchdog over the candidate instances of multi-candidate public ports. Two consecutive failed probes (700 ms cap) veto the instance IP; the veto triggers an immediate rewrite of the VIP table and every node table, and desiredForwards honors the same veto so the next reconcile agrees instead of re-installing the corpse — Proxmox still reports a running container when only the process died. Recovery clears the veto and the node tables return to local preference; the VIP stays sticky by incumbency. The veto has a 30 s TTL, so a dead watchdog degrades to the old 10-second behaviour rather than pinning a stale opinion. Every move raises a proxy.failover alert carrying its measured time.

Rehearsal record (2026-08-11, players absent, planned-downtime flag set)#

• network-proxy scaled 1 → 2. Anti-affinity placed the sibling (CT 226) on core-03 with the incumbent (CT 200) on core-01 — first live use of the placement rule. Parity verified: same velocity jar (md5-identical), same rendered toml content, both listening on 25565. The per-vmid /opt/shared bind mounts rule out cross-task config collisions by construction.
• 09:37:41 stopped velocity on the incumbent → 09:37:51 the leader logged ~ vip ingress: port 25565 → 10.0.0.17:25565 (network-proxy (#226); was 10.0.0.29:25565) and all three replicas' tables carried the new target. Failover: 10 seconds. A TCP connect to VIP:25565 answered through the survivor.
• 09:38:16 the stopped proxy recovered; the target was watched for 40 s and did not move — sticky held, zero churn on recovery.
• Reverse: stopped the new incumbent → 09:39:19 target back to CT 200, again on all three replicas, again ~10 s, VIP connect answering. Both proxies then restored; final state healthy with the target sticky on the current incumbent.

Not exercised and said plainly: an in-game client riding through a failover (needs a player), and voice re-handshake across a retarget — both are user-verify items on the next in-game session.

The gap#

The panel is three replicas behind a keepalived VIP with unattended repair. Storage is replica-3 GlusterFS. Backends are elastic. And every player connection flows through one Velocity container — network-proxy, CT 200, on one node. If that container or its node dies, the panel survives, the worlds survive, and nobody can play on any of them.

What the recon found (all measured, not inferred)#

The ingress is already HA; only its target is not:

• Players enter via the panel VIP (10.0.0.50). A dedicated conduit-vip nft table is pre-installed in all three panel replicas and sits inert until keepalived promotes the holder (lib/firewall.ts, vipIngressPass). A panel failover already keeps the game port answering.
• That table DNATs the game port to one address: vipNftScript collapses per-port to the first forward in array order — CT 200, always. No health in the choice. This one line is the SPOF.
• desiredForwards is per-node — "each node forwards its public IP to the first running instance of the service on that node" — so per-node tables already do the right thing with a proxy per node.
• velocityPass already renders every ready instance of each proxy task from the same backend set and try-order. Config parity across two proxies is not new work; it is the existing loop.
• Provisioning is per-task, so the second instance gets the same blueprint, custom/patched jar, -Dseam.pairs, JVM args and connector automatically. The forwarding secret is network-global.
• Voice (proxy-mode) tunnels its UDP through the proxy's own public port via the SVC velocity plugin, so it follows whatever the game-port forward targets.
• pickNode honors pins, allow/deny lists and node groups — but has no anti-affinity: two instances of the same task can land on one node, which would make a "second proxy" decorative.

150

holds the VIP

100

backup

Add a replica→ 50 Reset2 · step 50

Below every peer, so a cold replica cannot steal the VIP

Within 60 of the one above, so it can still take over when that one fails

The step halves once a full 50 would fall below the floor. A fixed step declared the ladder full at three replicas while forty rungs remained — that was a real bug.

Design: leader-retargeted VIP, not a second VRRP domain#

Two proxies on two nodes, both always running, both fully configured. The leader chooses the VIP DNAT target by health and rewrites the conduit-vip table (on all three replicas, as today) when — and only when — the current target stops answering.

Explicitly rejected: keepalived inside the proxy CTs (a proxy VIP). It buys nothing here and costs a second VRRP peering domain with every trap we have scars from (VRID collisions, auth_type PASS; parsing, the 8-char v2 password truncation, split-brain on config drift). The reason it buys nothing is conntrack: established flows are pinned to their DNAT target — rewriting the table only moves new connections. A sub-second VIP move cannot save a TCP session whose endpoint process died; the players on the dead proxy reconnect either way. So the honest failover unit is "how fast do new connections reach the survivor" — originally the 10 s leader tick, since 2026-08-13 the 1 s watchdog (measured 1.2–1.7 s; see the rehearsal record above). The machinery it reuses — health sweep, signature-cached nft rewrite, three pre-installed replicas — is exactly the machinery the panel VIP already proved.

Sticky by design: the leader keeps the current target while it is healthy, even after the other proxy recovers (choice memo in a global.__conduit* singleton — per-bundle rule). Moving only on failure means a recovering proxy causes zero churn; conntrack would make flap-back mostly harmless, but "mostly harmless" is not a design.

What a failover means for a player, stated plainly: the players on the dead proxy are disconnected once and reconnect from the multiplayer screen, landing on the survivor with the same server list and try-order. World and inventory state live on the backends and never notice. The limbo auto-reconnect cannot soften this one — it lives in the proxy that just died.

Implementation stages#

1. Anti-affinity in pickNode — instances of the same ingress/proxy task prefer nodes not already hosting one (soft: falls back rather than refusing when only one node is eligible). Small, general, benefits any multi-instance ingress service.
2. Health-ranked, sticky VIP target — rank game-port forward candidates by "is this proxy's port answering" (the sweep's verdict, with a direct TCP probe as tiebreak), keep the incumbent while healthy, rewrite conduit-vip only on change. The signature cache already makes the rewrite idempotent.
3. network-proxy → instances: 2 and let reconcile provision the sibling (clone-fast path applies). Verify config parity byte-for-byte and connector/voice registration on both.
4. Rehearsal, players absent — the panel-VIP playbook pointed at the proxy: stop velocity on the incumbent → watch the vip tables retarget on all three replicas → MC status ping through the VIP answers via the survivor → restart → confirm the choice stays (sticky) — then kill the survivor and watch it move back. Record timings in this doc like the VIP note did.

What the scale-up taught us (2026-08-13, the hard way)#

• A TCP probe cannot see auth breakage. CT 226 was provisioned honestly from the store and answered every probe — and rejected every real login with "Unable to verify player details", because the store's network.forwardingSecret had rotated away from what the whole live fleet runs. Provisioning is write-once, so the rotation never reached installed members; the fleet kept working pairwise on the old value until the first new member joined it. Hotfix: 226's secret file copied from CT 200 (both live on the shared volume). Durable fix: lib/secret-parity.ts compares every instance's forwarding secret against the store each reconcile and alerts on split-brain; convergence itself stays a coordinated operator flip, because pairwise-matching secrets mean one-at-a-time convergence breaks logins progressively. The flip ran the same evening (network empty, owner blocked on exactly this): all fourteen June-island members rewritten from the store value — the limbo's copy lives in-container at /opt/mc/settings.yml, not on the shared dir, and the redis CT carries a stale leftover paper-global.yml that is not a member at all — then eight running backends and both proxies restarted. Post-flip sweep: twenty service dirs, one hash, one island; the fast watchdog carried the VIP through both proxy restarts on its own, ~2 s per move.
• The parity check must include the secret file, not just the toml and the jar — the 08-11 record above says "parity verified" and it was, for everything except the one file that mattered.
• Node-local preference means a scale-up changes where existing addresses land: the moment core-03 had its own proxy, every client using that node's address moved from 200 to 226 without anything being "failed over". Placement changes are routing changes.

1. 0.0sProxy socket dies on core-01
2. 0.2sWatchdog misses its probe (1 s interval)
3. 0.6sSecond miss — the node is declared down
4. 0.8sIP veto raised; the dead proxy is removed from every table
5. 1.0sNode tables rewritten to prefer the surviving local proxy
6. 1.2sTraffic restored — new connections land on proxy-02
7. 1.7sVIP move complete (the slower of the two paths)

Play ResetReal time — 1.2 s takes 1.2 s.

Out of scope, noted honestly#

• Nodes without a local proxy still have no node-level game-port forward (pre-existing; the VIP is the public entry). A later pass could point proxy-less nodes at any healthy proxy.
• Active-active (players spread across both proxies) — needs a stable public L4 spread; the active-passive VIP is the correct first step and does not preclude it.
• The seam test proxy (network-seam-proxy, CT 222) stays single — it is a test rig, not ingress.

Drain-to-update: enabling it, and what is still unproven (2026-08-17)#

Auto-hotfix only ever restarted instances with zero players, and a proxy's count is never zero, so the fleet's front door was the one thing it never touched — silently. lib/proxy-roll.ts holds a proxy out of the routing tables instead, waits up to five minutes for it to empty, swaps the jar, waits for it to re-register, and always releases the hold in a finally (a proxy left drained carries no traffic, which is the outage this exists to avoid). lib/fleet-roll.ts decides who may go, and refuses both to take the last instance serving and to drain with no sibling to drain onto.

Enable it per task with the drain to update checkbox on the version card, beside auto-hotfix. It is refused on non-proxies, with the reason. Currently on for network-seam-proxy, off for network-proxy — turning it on for the front door is an owner decision, not a default.

Proven live: the flag persists, the non-proxy refusal fires, and the drain hold is honoured by desiredForwards and fastSwapPlan. The veto and drain holds are separate maps unioned in vetoedIps(), and they must stay separate: the 1 s watchdog clears a veto the instant a socket answers, and a draining proxy answers perfectly well.

Not proven live: a complete drain-and-roll. network-seam-proxy has one instance, so the no-sibling guard correctly refuses, and its auto-update resolves through the seam repository's release feed, which was returning 403 during a GitHub incident. Closing this needs either a two-instance non-production proxy or the flag on network-proxy. If you scale the seam proxy to do it: that task is ingress: false and every firewall forward belongs to network-proxy, voice or the ssh gateway, so adding an instance is routing-neutral — unlike 2026-08-13, when creating CT 226 silently moved everyone using core-03's address, because node tables prefer the local proxy.