Federation — more than one cluster, and the link between them

Design 2026-08-21. Status: not built. This document exists so that the parts which are already true are written down, and so the parts that are not are described precisely enough to argue with before anyone writes them. The first-run wizard (/welcome) says "not yet" for exactly the reasons below.

Why this is a document and not a feature#

Two panels sharing one view is easy to draw and hard to be honest about. Every hard question is about trust and authority, not about protocol:

• Conduit's store is one JSON document on /etc/pve/conduit/conduit.json, replicated by corosync inside one Proxmox cluster. Corosync is a LAN protocol with a low latency budget; it is not something to stretch over the internet, and Proxmox says so.
• The panel holds a root@pam API token with privilege separation off, because bind mounts require it (see install.md). Anything that joins clusters is therefore joining two things that can each create a root-capable container on the other's hardware.
• The agent token is cluster-wide. Today, "reaching a node" and "being trusted by the cluster" are the same fact. Federation needs those separated before it needs anything else.

So the order is: make the trust boundary explicit, then read across it, then write across it. Anything that writes before the boundary is real is a way to lose two clusters at once.

What is already true#

• One cluster spans many nodes. core-01/02/03 are one Proxmox cluster and one store. Adding a node is a Proxmox operation followed by a re-run of the idempotent installer.
• Clusters are already independent. edge-01 (Hetzner) and edge-02 (24fire) each run their own panel, store and secrets. They are not federated and nothing pretends otherwise.
• Provider labels group nodes within one cluster (nodeMeta.provider, /nodes). Seeing edge-01 beside the core trio is federation, not that feature — which is why /nodes stays a flat grid on a single-provider cluster rather than inventing groups.
• The hub already federates one thing: releases. Every cluster pulls the same feed, and its half-hourly ping is the only cross-cluster traffic that exists today.

Stage 1 — a private link between nodes (WireGuard)#

The transport, on its own, with nothing depending on it yet. Useful immediately even without federation: it is also the answer to "the panel container reaches its node over a public IP", which on a datacenter host means cluster traffic crossing the provider's network in the clear.

• One wg0 per node, 10.43.<cluster>.<node>/32 peers, keys generated on the node and never leaving it — the panel stores public keys only. Same shape as the managed container network (lib/managed-network.ts), a different /16 so the two can never overlap.
• Peer configuration is derived, not hand-written: the store holds the peer list, a leader job renders and applies it, exactly as managedNetworkKeepInSync does for bridges. The rule learned today applies — decide over the Proxmox API, apply over ssh, never let a failed probe read as "leave it alone" (see the .88 commit).
• Explicitly out of scope for stage 1: routing corosync over it. Proxmox supports a separate corosync ring, but changing the ring of a live cluster is a coordinated flip with a split-brain failure mode, and it earns nothing until stage 3.

Done when: every node can reach every other node's wg0 address, conduit doctor checks it, and killing the link degrades to today's behaviour rather than breaking anything.

Stage 2 — read-only federation, brokered by the hub. Built, then removed#

This shipped and was taken out again on 2026-08-31. Recorded here rather than deleted, because the reason is the useful part.

What it was. Each cluster pushed a summary to the hub on its existing telemetry ping. The hub served GET /v1/fleet to anyone holding HUB_FLEET_TOKEN, and the panel rendered a Federation page listing every cluster on that hub with its node, service, instance and player counts, cores, memory, software inventory and version.

Why it went. The owner hosts the public hub. Other people point their installs at it to receive releases, and in exchange the hub had quietly become a census of their infrastructure. Three specific faults, all measured:

1. The hub's dashboard had no login. GET / and GET /v1/stats were open to anyone who could reach the host, and the page rendered a per-cluster table: id, version, nodes, services, instances, players, OS, country, first and last seen.
2. Nothing was ever forgotten. Only telemetry.daily was pruned. telemetry.clusters grew forever.
3. A reinstall produced a ghost. clusterId is minted once into cluster state, so wiping a box and reinstalling registered it as a new cluster and orphaned the old row permanently. Five rows existed for three machines.

The token gate did work: without HUB_FLEET_TOKEN the hub answered bad token and a stranger's panel showed "not configured". The exposure was the public dashboard, not the token path. But the feature was still the wrong shape. A release feed does not need to know how many cores you have.

What replaced it. The hub keeps serving releases, with its human-facing pages behind a login. Telemetry is still collected and still opt-out, but stale clusters are archived after 30 days and a hardware fingerprint lets a reinstalled host continue its own record instead of leaving a ghost. There is no cross-cluster view in the panel any more.

What survives. Stage 1 below is live and unaffected: the WireGuard mesh carries Conduit's own traffic between nodes of one cluster. Peer panels still appear read-only on /nodes as estates, which is a link the operator configures themselves rather than something a hub hands out.

Stage 3 — one cluster over two Proxmox clusters. Shipping, in stages#

Built after all, once two joined clusters existed (Hetzner and 24fire) and the first attempt at control — one panel relaying a start button to the other panel — showed both of its faults in an afternoon: it died with the far panel, and it kept two service lists. Three rules replaced it.

Ownership-partitioned replication, never a shared document. Every logical record (a service, a group, a label, a schedule, a rule, a custom blueprint) carries ownerStore, and the owner's store is the only place it is written. Every joined cluster holds a read-only replica of the others' logical partitions, refreshed on the fleet tick from the owner's node agents over the mesh, with the owner's panel as the fallback. So the far cluster's services are listed here while its panel is down. Instances are never replicated at all — they are discovered from Proxmox tags on whichever cluster runs them — and secrets never leave the store they are in. A join is a verdict, not a merge: two records with one id are merged only when their logical content is equal, and surfaced with the diff otherwise. A delete is a tombstone that travels with the snapshot; nothing is destroyed on absence.

Reach: control goes to the far cluster's nodes, not its panel. Switching "manage together" on for a joined cluster makes this cluster's leader mint, for that cluster alone: a token this cluster's node agents accept only from the mesh, a Proxmox API token on conduit-fleet@pve with the guest roles and nothing else, and this cluster's CA. Those land as files on every one of our nodes, and our node agent serves them to exactly one asker — the one whose source address lies in that cluster's mesh segment, which is an identity because it can only have come out of that cluster's WireGuard tunnel. So the far panel fetches its own credentials from any of our nodes, with our panel down, and nothing secret ever crosses the plain-HTTP path the two panels talk over. Power, console, exec and files on a remote instance then go straight to its node over the mesh. Switching manage off removes the Proxmox token and the files on the next tick; the far side reads a 404 and drops what it held. What crosses is root on our nodes — the parity the owner chose — and what the design changes is revocability and attribution: conduit-fleet@pve!fleet-<cluster> in our task log, the peer label in our agent journal.

One tree. The servers page on either panel lists every joined cluster's groups and services in one tree, each row saying which cluster it lives on; a foreign service's instances are read from that cluster's Proxmox over the mesh, and a cluster that does not answer keeps its last rows marked unreachable rather than vanishing. A group with the same id on both clusters is one group.

Edits go through the owner. A change to a foreign-owned record is forwarded to the panel that owns it, and refused with a sentence when that panel is not answering — power and console still work meanwhile, because they do not need it. This is the only thing the relay still does, and the list of what it carries is explicit: a method and a path, with a deny list that wins over it and a test that refuses any listed route which can reach a shell.

The fleet pages are one list too. Worlds, players, the agent census, log search and the version card each ask this panel and, beside it, every joined cluster that has issued reach, through the relay entry the route lists; the answers are merged with every far row tagged by its cluster, keyed by cluster and vmid because both clusters run a 204. An action is addressed back to the panel that answered: pre-generation through the owner's console, a kick or a move on the owner's proxy, a converge through the owner's install path. These reads do depend on the far panel, unlike power and console; a cluster that does not answer is named above the list with the sentence it failed on, and the log search counts what it could not search, per cluster. Two things were learned the hard way: the asking side of a relayed route has to forward, or it answers from its own records for a request that named another cluster, and both clusters' system containers carry the same ids, so the wrong answer looked right; and a deny-list prefix beats an explicit entry under it, so the three registry routes the pages needed were refused until the connector's channel was denied route by route. A structural test walks the relay list and refuses a route that does not forward.

Measured on the live pair: a marker bundle on one cluster's node was read from inside the other cluster's panel container over the mesh, refused from the issuing node's own loopback and mesh address, and unanswered from the internet; the read was attributed in the issuing node's journal to the peer's tunnel address.

What "join a cluster" means, and what it does not#

Joining means this panel and that one exchange node lists, and — when both sides switch it on — mesh identity, read-only replicas of each other's service definitions, and reach. It does not mean two panels sharing one store: each record has one owner and one writer, and the other side holds a copy it cannot change except by asking the owner. That is the distributed-systems commitment this document once refused (quorum, split-brain, a write path over the internet), kept refused; what replaced it costs nothing on a standalone cluster, where the fleet's task list is exactly the local one.

Adding capacity stays what it is today and works now: another node in the same Proxmox cluster, then the installer. That is the answer to "I need more machines". Federation is the answer to "I have separate estates and want to see them together and act on them from one screen", and conflating the two is how this would get built wrong.

Measured: what stops edge-02 and edge-01 becoming one Proxmox cluster#

Asked directly on 2026-08-23 — can the 24fire dev node and the Hetzner production box be joined into a single Proxmox cluster, so provider-wide features work across both? Both were inspected rather than reasoned about, and the answer is no for three separate reasons, any one of which is on its own decisive.

The link is not one of them. From edge-02 to edge-01, twenty packets: 13.7/14.0/14.3 ms min/avg/max, mdev 0.157 ms, no loss. That is a better corosync path than plenty of production clusters have — both sites are in Germany and the jitter is remarkably low. Latency was the reason this was expected to fail, and it is the one thing that would have worked.

What actually stops it:

1. The VMIDs collide. Both boxes run 190, 200, 201, 202 and 203 — the panel and the four core services — because the installer assigns the same numbers on every fresh cluster. edge-02 also has 204 and 205. VMIDs are cluster-wide unique in Proxmox, so five guests would have to be renumbered before a join could even be attempted. This is a consequence of the install being reproducible, which is otherwise a feature.
2. pvecm add refuses a node that already has guests. Both have them and both are in use. The documented path is to join empty nodes and then create guests, which is the opposite order.
3. Quorum, and this is the one that would hurt. Two nodes means two votes and a quorum of two, so either node going away — or just the link blinking — drops the survivor below quorum and /etc/pve goes read-only. Conduit's store lives at /etc/pve/conduit/conduit.json, so a read-only /etc/pve is a panel that cannot write anything at all. edge-01 has its own quorum today, on its own, and joining would trade that for a dependency on a dev VPS staying reachable. The usual escape, two_node: 1 with wait_for_all: 0, buys availability by permitting split-brain: both sides keep writing their own copy of the store with nothing to reconcile them afterwards. This project has already lost a store once and does not need a mechanism for doing it deliberately.

It was not a prediction — someone had already tried it#

Re-inspected 2026-08-31, and the reasoning above turned out to be describing something that already existed. Both boxes carried a corosync.conf naming each other as a two-node cluster, with exactly the two_node: 1 + wait_for_all: 0 escape this document argues against. The cluster had never formed: each reported Nodes: 1, each saw only itself in membership, and each was quorate alone — so both had been writing their own copy of the store for weeks. Their conduit.json files had already diverged (6a6db824… against 5f114bfa…). The WireGuard tunnel handshakes and passes no traffic, which is the only reason it stayed harmless: the moment the two could talk, pmxcfs would have had to reconcile two divergent /etc/pve trees.

Both are now genuine standalones, guests untouched, and they appear together on /nodes as estates — which is the thing that was actually wanted, without a consensus protocol spanning two providers.

Two mechanical details worth keeping, because neither is in the Proxmox documentation:

• Removing corosync.conf does not clean /etc/pve/nodes/. Each box kept a directory for the other, holding that box's guest configs — which is why pct create 190 on one of them refused with "CT 190 already exists on edge-01". Remove the foreign node directories too.
• killall pmxcfs returns before the process dies, so pve-cluster then fails to start with unable to acquire pmxcfs lock for ever. Kill by pid and wait for it, then fusermount -u /etc/pve before starting the service.

And the store-level residue outlives the split: both boxes kept each other's tasks and node groups, so edge-01 held a service pinned to a node group whose only member was edge-02 — wanting an instance for ever, getting nothing, silently. That is now watched by lib/placement-health.ts.

There is also a fourth thing, which matters for stage 1 rather than for joining: every standalone cluster's first node is 10.43.0.1, because the mesh range is assigned per cluster from the same base. edge-02 already holds it. Any future link between two clusters has to renumber one side or carry a per-cluster offset, and that belongs in the design before anything is built on top of it.

None of this blocks what was actually wanted. "Provider grouping and features that work across provider nodes" is stage 2 — one pane of glass over separate estates — and it needs no shared quorum, no shared VMID space and no corosync over the internet. Merging the clusters would be a distributed-systems commitment taken on to solve a reporting problem.

Credentials between joined clusters (2026-09-05)#

• A peer address typed without a scheme is https://. An explicit http:// peer keeps working, and its row on the Nodes page says "credentials in the clear": the grant secret (or a hand-added peer's machine token) is a bearer on every tick to that address. Put the peer behind TLS.
• Reach (the agent token and the scoped Proxmox token a cluster hands to one it lets "work together") is re-minted every 30 days. The old pair stays honoured for a day, because the far side learns the new bundle by reading our nodes on its own tick.
• Detaching a cluster withdraws its reach in the same request, drops the replica of its store and its remote mesh entry. A bundle naming a different issuer than the cluster it was read from is refused.
• "Work together" is root-equivalent on the other cluster's nodes through the node agent; the Secrets page names every cluster that holds it.

Joins over TLS: the cluster address (2026-09-05)#

A Domains entry of kind Cluster address (edge-01.example.com, one per cluster, Cloudflare-proxied and covered by the origin certificate like the panel's other names) is this panel's own https address for the clusters joined to it. Once it is proven, /api/fleet/peer advertises it, a join started from this side sends it as our address, and every joined cluster moves from the address it was joined at (http://<ip>:3001, the "credentials in the clear" row) to the https name, after that name answered the same call. A peer can never move us down to http by advertising one, and an address we already left is not offered back. A cluster address never fails over; that is what the panel hostname marked failover is for.

Node doors: a hostname and a public certificate per node (2026-09-05)#

The mesh carries consoles, files and power between the clusters, and a tunnel that is down takes all of them with it. A Node entry on Domains and TLS (play.example.com, one per node) gives the node a name of its own: a DNS-only A record to that node's public address, and a Let's Encrypt certificate proven through the zone's DNS (lib/acme.ts, lib/node-tls.ts) that the panel pushes to the node. The node agent (0.3.4) serves the same API on 8443 with it, and reloads a renewed certificate on its own. A joined cluster's panel learns the door from the owner's replicated partition and uses it only when the mesh address does not connect; the reach token authenticates on both paths, TLS is what makes the second one safe. The firewall opens 8443 to the joined clusters' node addresses and to nobody else. Consoles stay on the mesh.

The shared front door (2026-09-05)#

One panel hostname, served by the cluster that owns it, taken over by a joined cluster when the owner falls silent, handed back when it answers again.

• Owner side. On Domains and TLS, a panel hostname gets a Failover switch, and the zone card a Share these credentials with joined clusters switch. Sharing puts the zone's API token and Origin CA key into the reach bundle every cluster that works together with this one reads from our nodes; switching it off withdraws them on the next tick. The hostnames marked failover are published in the owner's replicated partition (frontDoor.hostnames), with the address the owner answers at.
• Stand-in side. A joined cluster holding the credentials issues an origin certificate for the owner's hostnames ahead of time and installs a server block for them on its own TLS front, so a takeover moves only a DNS record. It probes the owner's panel every tick on two paths: from itself on the owner's public address, and, when that fails, from the owner's own node over the mesh. After three minutes of silence on both it points the A record at itself and publishes a claim (frontDoor.claims, owned by the claimant, replicated like any record); after two minutes of the owner answering it points the record back and withdraws the claim. A claim not refreshed for ten minutes is a dead claimant, and the owner takes its door back.
• What it means for people. Browsers follow the record within a minute (TTL 1). Sessions are per cluster, so a person signs in again on the stand-in; their account is there because identity replicates. The stand-in drives the owner's servers over the mesh; settings that live in the owner's store wait for it.
• Limits. A partition between the two clouds that also cuts the mesh reads as an outage: the stand-in takes the door, and hands it back two minutes after the partition heals. The owner's DNS reconcile leaves a hostname alone while a fresh claim on it stands.

Open questions#

1. Which cluster owns a player who exists on both? Probably neither — identity is per-network, and pretending otherwise imports LuckPerms and player-sync into the trust boundary.
2. Does the hub become infrastructure people must run themselves? If federation depends on it, the answer is yes, and it needs the same treatment as the panel: HA, backups, an update path.
3. Is stage 1 worth shipping alone? Probably — encrypting node-to-node traffic on a datacenter host is worth having whether or not anything federates.