Architecture

How Conduit turns a Proxmox VE cluster into a self-operating game network.

Everything on this page is derived from the implementation. File references point at the source of truth; when in doubt, the code wins.

System overview#

Three moving parts, one direction of truth:

• The panel is a single Next.js app: UI, ~130 API routes, and the controller in one process (dashboard/src/instrumentation.ts boots it; CONDUIT_CONTROLLER=off disables it).
• The node agent is a small Node service on every Proxmox host (agent/src/index.mjs): authenticated pct exec, sandboxed access to the shared store, and the WebSocket console bridge. Entering a container costs a fixed ~0.40s however small the command, so it holds one pct exec <vmid> -- bash -s open per container and feeds commands to that.
• The container agent is a single-file stdlib Python 3 daemon inside each LXC (agent/ctagent/), installed and kept current by a reconcile pass. Typed operations only — no generic exec — reached on a per-container token derived from a value containers never see. It is an optimisation with a fallback: every caller reverts to the node agent when it is absent, stopped or version-skewed, so nothing in the panel depends on it existing.
• Reads go through the container agent, writes and anything genuinely shell-shaped do not. lib/ct-inspect.ts answers "does this path exist", "what is in this directory" and "what does this small text file say" over the agent's fs/list and fs/read, because those were the questions the reconcile passes asked thousands of times a day through pct exec — which costs 0.40 s and about 130 MB per call, being a Perl program that loads the entire PVE library stack. It stays typed deliberately: a generic shell reachable over HTTP inside every container is a worse trade than a slower reconcile. Two rules for anything added here — bytes stay on the shell (the agent's read decodes UTF-8 with replacement, so a PEM or a jar comes back corrupted), and "could not tell" is not "absent" (several checks default to assume installed on failure, because guessing absent reinstalls a working plugin on every pass).
• The connector is one jar loaded by every Paper and Velocity server (plugin/), reporting state up and applying config pushed down.

Control loop#

instrumentation.ts runs a tick every CONDUIT_INTERVAL_MS (default 10s). Only the panel holding the cluster VIP acts (see High availability); the others stay warm.

Each tick, in order: reconcileAll(), scheduled actions, then the flush/sweep family — activity feed, operator audit (+ SIEM forward), outbound webhooks, alerts, login history, player audit (+ GDPR retention purge), the uptime sweep, automation rules, maintenance and event windows, the weekly digest, and dev-workspace expiry.

reconcileAll() (dashboard/src/lib/engine.ts) is a sequence of idempotent passes:

Every pass converges from observed state — a panel restart loses nothing, and manual drift (edited config, missing jar, wrong password) is repaired on the next pass.

State#

One JSON document is the whole desired state (dashboard/src/lib/store.ts):

• file backend (default): dashboard/data/conduit.json — development.
• agent backend (CONDUIT_STATE_BACKEND=agent): read/written through a node agent into /etc/pve/conduit/conduit.json — replicated by Proxmox corosync, so all three panels share one quorum-backed copy.

The document holds groups, tasks (services), network-wide settings (secrets, alerting, domains, voice, S3…), blueprints, schedules, users, monitors, automation rules, packs, firewall rules, and more. Mutations go through mutate() and bump the live bus (lib/live-bus.ts) — an SSE version counter per topic that tells open dashboards to refetch (/api/live).

Two things deliberately live elsewhere:

• The language catalogs sit in languages.json on the shared volume (lib/lang-store.ts), not in the store. They were 817 KB of an 835 KB document — corosync replicated them on every unrelated write, and every getDB() parsed them to answer questions about 20 KB of orchestration state. Reads fail open (stale cache beats a missing translation), edits fail closed (never save defaults over data).
• Writes carry a monotonic rev. The agent compares an x-conduit-if-rev header inside its write queue and answers 409 when the document moved underneath, and mutate() replays its mutator against fresh state (up to 5×). Three replicas all serve HTTP, so two interleaved read-modify-writes used to lose one silently; now the loser retries. Mutators must therefore be idempotent re-runs over the state they are handed — they can execute more than once.

Provisioning lifecycle#

Instances launch from prepared templates (pre-provisioned clones) when available, so a dynamic lobby scales in seconds, not minutes. The install log is streamed to the UI (/api/services/[vmid]/install-log, SSE).

Connector protocol#

The connector (plugin/src/main/java/dev/admin/conduit/ConduitClient.java) speaks plain HTTPS to the panel:

Design principles baked into the protocol:

• Hash-deduplicated config — the heartbeat response omits an unchanged config block.
• Acknowledged actions — the connector reports ackActionId; the panel only delivers newer actions, giving effectively-once execution across reconnects.
• One-shot payloads with a resend gate — bulky data (the command map) is sent once; if the panel restarts and loses it, the response carries needCommands: true and the connector re-publishes past its hash gate.
• Redis for the data plane — chat relay, voice frames, and sharding sync go over Redis directly; HTTP is the control plane.

High availability#

• Three panel replicas, one per node, behind a keepalived VIP.
• Leader election is the VIP: each tick checks whether the VIP is bound locally (os.networkInterfaces()); only the holder reconciles. Failover is VRRP-fast with no coordination protocol of its own.
• The VIP follows a panel that answers, not a container that booted. A keepalived track_script curls /api/health on loopback every two seconds and drops the replica's priority after three failures, so a leader that comes up into a broken build loses the address in about eight seconds. It tests nothing shared — a check that read the store would fail on all three replicas at once and move the VIP in a circle without fixing anything. See install/lib/keepalived.sh and agent/conduit-panel-health.sh.
• State survives anything short of quorum loss — corosync for the store, GlusterFS for artifacts (jars, overlays, packs, uptime rollups, backups of the store itself).
• The game ingress is HA too: two Velocity proxies on two nodes; a 1-second watchdog on the leader vetoes a dead proxy and rewrites the VIP table and every node table at once — measured 1.2 s (node entries) / 1.7 s (VIP) from socket death, sticky through recovery. Established connections die with their proxy; what the watchdog buys is that reconnects land on the survivor immediately. docs/proxy-ha.md has both rehearsal records.
• Zero-downtime upgrades: install/update.sh rolls the bundle replica by replica, VIP holder last, after snapshotting the store — and the panel updates itself the same way from the hub's release feed, with a boot finalizer that rolls a failed replica back unattended (docs/updates.md). scripts/upgrade-live.sh survives as a shim.
• The console WebSocket bridge (conduit-console) runs on every replica — consoles work against whichever panel you reached.

Shared storage#

/var/lib/conduit is a GlusterFS replica-3 volume with one brick per node. It holds the per-service config dirs (services/<vmid> — bind-mounted into each container as /opt/shared, with configs and plugins relocated onto it), the canonical connector jar, overlays, packs, uptime rollups and store backups. Hot data — worlds, game state — deliberately stays on each container's local disk for I/O; a cross-node move ships it explicitly (detach → migrate → re-attach). The panel reads and writes the volume through the node agent's sandboxed file API rather than mounting it itself — a fact worth internalizing before writing any panel code that touches those paths with plain fs: it will read nothing, error nothing, and conclude whatever silence implies.

lib/storage-health.ts reads the volume every five minutes from whichever node answers first — volume info, volume status, peer status, heal info and heal info split-brain, all in one --xml round trip — and asks each node separately whether it still has the volume mounted, since a node whose mount has dropped looks perfectly healthy from anywhere else. Findings become alerts and render on Storage. It is strictly read-only: it never runs a heal and never touches a brick, because choosing which copy of a file wins is a judgement call with data on the line.

Pending heals are normal; heals that do not finish are not. During any ordinary write there are entries pending, so alerting on their presence would be constant noise. cluster.heal-timeout is 600s, so the watch remembers when it first saw each entry and reports only the ones still pending after thirty minutes — three full self-heal passes.

A backlog means two different things, and which one depends on whether a copy is missing. With every brick online, an entry that will not clear has somewhere to heal to and is not doing it — that is the GFID case below. With a brick offline, every write since it went leaves an entry that cannot heal, because the destination is not there; the count climbs for as long as the outage lasts and drains on its own afterwards. Both look identical through heal info, so the only thing that separates them is the brick and peer state, and the watch reads it before choosing what to say. It got this wrong until 2026-08-27: with core-03 down it reported 141 files "waiting to heal" and offered the GFID recipe below, which would have sent somebody comparing extended attributes on a brick that was not in the pool.

One limit worth knowing: the clock is in memory. The map of when each entry was first seen lives in the panel process, so a restart resets it and the thirty-minute window starts again. That is harmless for a real fault, which outlives any number of restarts — but a cluster restarting more often than every thirty minutes (a rolling update series, or a crash loop) would never accumulate enough age to report one. Recorded rather than fixed, because persisting it means either a write to the corosync store every five minutes or per-replica local state that leadership can move away from.

That threshold exists because of a real failure. On 2026-08-05 three files under services/213 and services/216 were found unreadable — Input/output error through the mount on all three nodes — while the bytes sat healthy on two of the bricks. The same path had a different GFID on two bricks, so the client could not pick an authority; one brick was missing the files entirely, meaning the volume had been running on two copies instead of three since 2026-07-26. Ten days, no signal.

The important part for anyone debugging this again: heal info split-brain reported 0 entries the whole time. A GFID mismatch is not split-brain. The self-heal daemon logged errno=2 against the same three GFIDs every ten minutes and never completed, and that — an entry that stays pending — is the only symptom the volume actually offers.

To confirm it, compare the GFIDs directly on each node:

getfattr -n trusted.gfid -e hex --absolute-names /data/conduit-brick/<path>

Directory GFIDs can match while the child files differ, so check the files themselves. The fix is to back up the file, then on the minority brick remove both the entry and its hardlink under .glusterfs/<g0:2>/<g2:2>/<dashed-gfid> — removing only the entry leaves an orphan the heal daemon chases forever — then gluster volume heal <vol> full. Verify by reading the file through the mount on every node, not on the bricks; the bricks looked fine throughout.

The volume's own backup layer#

vzdump excludes bind mounts from container backups, and the panel replicas never mount the shared volume — so for a long time the tree holding every relocated plugin directory (Skript scripts and variables, Citizens saves, plugin configs), the language catalogs, overlays and store history had no backup layer at all. The TimeSMP servers proved it: variables.csv came back from a crash truncated to a header on four servers, and the only recovery source turned out to be Skript's own backups folder — on the same volume, by luck.

lib/services-capture.ts closes that. Nightly, the leader picks a node whose fuse mount actually shows the services tree (an empty mount is a gluster failure, not a small backup), tars /var/lib/conduit minus what is reproducible — assets/hytale, updates/, every *.jar — and streams it through a presigned PUT to object storage under shared-capture/, with a file manifest beside each tarball so a restore can start from a listing instead of a download. The far end is verified with a stat after upload, a capture implausibly small refuses to count as success, failures alert (capture.failed) and retry within the hour, and five nightly pairs are kept.

Networking & firewall#

dashboard/src/lib/firewall.ts renders one dedicated nftables table (ip conduit) per node and applies it atomically. Nothing outside that table is ever touched; disabling the firewall deletes the table and reverts every trace.

• Service-driven DNAT — ingress-role services (proxy, web) get node forwards automatically; they appear and disappear with the service.
• Hairpin SNAT — LAN clients work because only DNATed connections are masqueraded.
• Access control — allow/block lists by IP/CIDR or ASN (prefixes resolved via RIPEstat), enforced in prerouting before any backend sees the connection.
• Node protection — optional default-drop for internet-sourced connections to node-terminated services (Proxmox UI etc.); private space, SSH, ping, established and every forwarded port keep working. Lockout-safe by construction.
• Voice forwards — per-instance UDP ports (direct mode) or a single UDP forward on the proxy port (proxy mode).

Voice architecture#

Managed Simple Voice Chat with two routing modes (lib/voice.ts, ConduitVoice.java):

• Proxy routing (live-parity): the official SVC Velocity plugin is installed on selected proxies; every backend binds voice on its Minecraft port and all voice UDP tunnels through the proxy's own port — one public port for the network.
• Group layer on top: rosters sync instantly over Redis (create/remove events → mirrored groups with the origin's UUID everywhere), your group follows you across server switches, and optional halls namespace voice per Conduit group. Proximity voice remains native and per-server — no position streaming, no added latency.

The trade-off against a central voice server (a dead backend only kills its own voice; Redis loss degrades to proximity; no new single point of failure) is documented in the design notes and was chosen deliberately.

Security model#

See security.md for the full treatment. In brief:

Updating itself#

The panel is a service like any other it manages, and it manages itself the same way: it knows which build it is running (version.json, stamped into the bundle at build time, read out of the process's working directory), compares that against a feed it is pointed at, and can replace all three replicas — the leader last, detached, with the run finished by whatever boots afterwards.

The mechanics that make that survivable are the same primitives as the rest of the control plane: the shared store carries the run so any replica can report it, the VIP decides who orchestrates, and ctExec moves the bundle exactly the way install/update.sh always has. Full walkthrough, including running your own hub: updates.md.

Frontend#

Next.js 16 + React 19, Tailwind 4 with OKLCh design tokens, framer-motion for restrained entrance choreography (150–250ms, reduced-motion aware). The design system is the Cloudflare dashboard school: flat panels, hairline borders, band layouts, one blue. Consoles are xterm.js over the agent's WebSocket with an SSE fallback. Live updates ride the SSE live bus rather than polling where it matters.