Install

Two shapes: a single-process development setup, and the production 3-node HA topology this project actually runs in. A guided installer under install/ drives both and verifies every stage; the underlying scripts/ stay usable on their own. Nothing here is hypothetical — these are the flows that operate the live cluster.

Quick start#

curl -fsSL https://raw.githubusercontent.com/admin-dev/Conduit/master/install/bootstrap.sh | bash

Clones (or updates) the repo and hands over to install/install.sh: system detection, a short wizard for the things it can't work out (node IPs, panel CT ids, VIP, Proxmox API token), then shared storage → node agents → panel replicas → a full health pass, verifying each stage before the next. Everything it decides lands in install/cluster.conf (mode 600, gitignored) — the single source of truth every later command reads.

Requirements#

The machine the installer runs from barely matters — any Linux with ssh, curl, git and Node ≥ 20 can drive a cluster over ssh. The requirements live on the nodes:

• Debian, Ubuntu or Proxmox VE on every node. Other distros are detected and named honestly, not pretended to work.
• At least a VM — ideally dedicated hardware. Conduit's firewall is a per-node nftables table (DNAT ingress, hairpin SNAT, node protection); inside an LXC or other container that kernel-layer ip routing is not really usable. Containers are the right tool for the panel and the game instances, not for node roles. The installer detects this (systemd-detect-virt) and refuses container nodes unless forced.
• A public IP with an open routing table is recommended for the full use case — direct game ingress, firewall DNAT, voice UDP forwards. Behind NAT everything else works: forward the game and voice ports at the upstream router, and the doctor keeps pointing out which half is missing.
• net.ipv4.ip_forward=1 and nftables present, for DNAT'd game and voice traffic. Both are checked, and the exact fix is printed when they aren't.

install/validate.sh verifies all of this read-only before anything is touched.

How much machine#

Conduit itself is small; the game servers are what you are actually sizing for. What the control plane costs, from scripts/deploy-panel.sh and confirmed against the running cluster:

So a single-node install wants roughly 2 cores and 2 GB spare on top of whatever the servers need, and three-node HA wants that on each node. The reference cluster runs 16 cores / 43 GB per node carrying about a dozen instances, which is comfortable rather than minimal.

The 2 GB on the panel container is not arbitrary and is worth not shrinking: V8 sizes its default old-space from the cgroup limit, so a smaller container gives the panel a smaller heap ceiling than the one lib/heap-watch.ts is calibrated against.

Game containers are sized per blueprint when you deploy them — a Paper server on the reference cluster gets 2 cores / 4 GB — so plan those against your player counts, not against Conduit.

Proxmox VE 8 or 9. The installer drives pct, pveam and pveum, which are stable across both; the reference cluster runs 9.2.3. It does not need a subscription, and install/modules/pve.sh handles the ISO default of subscription-only apt repositories, which otherwise makes apt-get update fail before anything else can start.

How long it takes depends almost entirely on one step: on a node with an empty template store the installer downloads an LXC template first, and that download dominates everything after it. The stages themselves are minutes of ssh and pct, each verified before the next begins, and --dry-run prints the whole plan without executing any of it.

Seeing the install without doing one#

scripts/demo-install.sh runs the real installer's --dry-run against a throwaway config on documentation addresses, so you can read the plan — detection, warnings and all — without a cluster and without touching install/cluster.conf:

scripts/demo-install.sh          # single node
scripts/demo-install.sh --ha     # three nodes + a floating VIP

The plan differs by shape, which is the interesting part: a single node has nothing to replicate with, so it skips the GlusterFS volume and comes out a stage shorter.

Updates, and what talks to the hub#

A fresh install points at the public hub, https://hub.example.com, so it can tell you when a newer release exists. Without a hub an installation is simply blind to updates — the version card, the update dialog and the self-update all have nothing to ask — and that silence is worse than a default. A cluster with its own hub keeps it: the store wins, then CONDUIT_HUB_URL, then this.

Reading the feed and pinging the hub are independent, but note that having a hub at all enables the ping, which is opt-out in Settings → Hub rather than opt-in. It carries a random cluster id, the running version and a handful of counters — no names, addresses or domains. To make no outbound calls whatsoever:

CONDUIT_HUB_URL=off        # in the panel env; silences both the feed and the ping

What the installer does to Proxmox itself#

Nothing needs preparing by hand. On a node that has never built a container the installer refreshes the appliance catalog and downloads the Debian LXC template (falling back to the newest debian-12-standard if the pinned point release has aged out), then creates each panel container from it — unprivileged, nesting on, onboot, tagged conduit-panel, on the storage and bridge the node actually has (DHCP on vmbr0 where a LAN exists; see the two sections below when it does not).

The Proxmox API credential is minted the same way when you don't already have one:

pveum user token add root@pam conduit --privsep 0     # what the installer runs for you

root@pam with privilege separation off is deliberate, not laziness. Bind-mounting a host path into a container — which every game server gets, for the shared /var/lib/conduit — is a root@pam-only operation in Proxmox. A token on a dedicated PVEAdmin user passes every check at install time and then fails much later, as a container that provisions but has no shared volume.

The secret is displayed by pveum exactly once, at creation. So if a conduit token already exists and its secret isn't known, the installer mints a differently-named one rather than rotating a credential a running panel may still be using.

Where containers are placed#

Conduit asks each node which of its storages can hold a container root filesystem (content includes rootdir) and uses that. local-lvm still wins wherever it exists, so an existing cluster keeps putting new containers exactly where its old ones live.

This is worth stating because it used to be assumed. A cluster built from the Proxmox ISO always has an LVM thin pool called local-lvm, and that name was written into the code as if it were a property of Proxmox rather than of one installer. A host imaged another way — Hetzner's installimage puts root on mdraid and creates no thin pool — has only the directory storage local, and every pct create failed there with storage 'local-lvm' does not exist. Same story for the network bridge: see the managed network below.

If you want a specific storage, set it per node; detection only fills in the blank.

Dedicated servers from a provider that images the disk for you#

Two things differ from a stock Proxmox ISO install, and both were learned the hard way on a Hetzner box (verified across a real cold reboot: back in 75s, all RAID [UU], systemctl is-system-running = running, web UI answering from outside).

You do not need to create vmbr0. The Hetzner wiki has you enslave the physical NIC into a bridge; on a MAC-filtered switch port that drops every frame from a MAC the provider has not assigned, which is how people lose a box they can no longer reach. Conduit never does it. Where a node has no private LAN, the installer builds a portless vmbr1 with its own /24, NAT for egress, and a DNAT of the node's public :443/:3001 to the panel container. The physical NIC is untouched and no extra provider IP or MAC is required.

Two boot hangs worth pre-empting, both of which wedge the machine before networking starts, so they look like a dead server rather than a boot problem:

The same class of fault bites KVM guests, not just bare metal. core-03 — a virtual Proxmox node, not a physical one — had been dropping out of the cluster every few days and staying down until somebody started it by hand. Its journal simply stops mid-write, with no panic, no OOM and no shutdown sequence, which reads like the host killed it. It was not. Thirteen seconds before the last one:

00:54:22  WARNING drivers/gpu/drm/drm_vblank.c:1320  drm_crtc_wait_one_vblank        CPU#1
00:55:10  WARNING drivers/gpu/drm/drm_atomic_helper.c:1921  wait_for_vblanks          CPU#8
00:55:11  corosync: token has not been received in 2668 ms
00:55:12  pve-firewall: firewall update time (10.179 seconds)     ← normally milliseconds
00:55:23  journal ends mid-line

The guest sees QEMU standard VGA (1234:1111) and had two drivers bound to it: bochs, the DRM driver that belongs to that hardware, and vga16fb, the legacy 16-colour framebuffer driver. With both loaded the DRM layer waits on vblank interrupts that never arrive, which blocks kworkers, starves corosync of its token and eventually wedges the machine. Nothing on a Proxmox node needs a framebuffer console — it is administered over ssh and the web UI — so the fix is the same shape as the nvidiafb one above: blacklist vga16fb and vgastate in /etc/modprobe.d/, then update-initramfs -u. Both modules had a use count of 0, so they could also be rmmod-ed live without a reboot.

Worth checking on any Proxmox node, physical or virtual: lsmod | grep -E 'vga16fb|.*fb$' next to the real DRM driver. Two drivers on one display device is the pattern.

Never let the store be written while /etc/pve is unmounted#

The nastiest trap this project has hit, and it locked two nodes out within an hour on 2026-08-23.

pmxcfs stops during any dist-upgrade, pvecm create, pvecm add, or a plain systemctl restart pve-cluster. While it is down /etc/pve is an ordinary directory on the root filesystem — and Conduit's node agent, which writes /etc/pve/conduit/conduit.json, recreates that directory in the underlay within seconds because the panel keeps writing. FUSE refuses to mount over a non-empty directory, so pmxcfs then never starts again:

fuse: if you are sure this is safe, use the 'nonempty' mount option
[main] crit: fuse_mount error: File exists

That locks you out of the node, because /root/.ssh/authorized_keys is a symlink into /etc/pve/priv — no mount, no keys, no ssh — while pveproxy blocks forever on the same directory, so :8006 accepts the connection and never answers. One cause, three symptoms, and it reads exactly like the hypervisor killed the guest.

The agent refuses now (2026.8.111): before writing it checks that /etc/pve reports the FUSE superblock magic (0x65735546) and answers with a sentence naming the problem instead of creating the directory. It fails open when it cannot tell — a missing or throwing statfs allows the write, because blocking every store write over an unrecognised filesystem call would be a worse outage than the one it prevents. Losing a write for a few seconds is survivable; the panel retries and the store is the same file on every node. Losing the node is not.

Recovery, if you meet this on a node running an older build:

pct stop <panel-vmid>          # stop the writer FIRST or it recreates the directory in seconds
cp -a /etc/pve/conduit /root/  # keep it: this is the store written while unmounted
rm -rf /etc/pve/conduit
systemctl reset-failed pve-cluster && systemctl start pve-cluster
stat -f -c %T /etc/pve         # want "fuse", not "ext2/ext3"
mkdir -p /etc/pve/priv && cat >> /etc/pve/priv/authorized_keys   # your key back
systemctl start corosync pveproxy pvedaemon pvestatd

The pmxcfs database is fine — the store returns intact from config.db, and the underlay copy is the stale one. Read journalctl -u pve-cluster for the real cause rather than assuming a corrupt database. Two related traps: pct fails silently while pmxcfs is down, so a stop/destroy loop prints nothing and does nothing; and a hostname rename needs a reboot, or pvestatd keeps publishing under the old name and the node reads status=unknown forever.

Two caveats to know rather than fix. Removing the Debian kernel — which the Proxmox wiki requires — can leave you with one kernel and no fallback if an upgrade breaks boot; Proxmox keeps the previous PVE kernel on the next upgrade, so don't hand-prune back to one. And the console-setup override is the one piece resting on reasoning rather than an observed boot: a reboot after dpkg-reconfigure console-setup has not been tested.

Finally: a provider that gives you a public IP gives the whole internet a route to :8006. Put the Proxmox firewall or a VPN in front of it before you leave the box running.

Development#

Requirements: Node.js ≥ 20. A reachable Proxmox host only once the controller is on.

git clone https://github.com/admin-dev/Conduit.git
cd Conduit
install/install.sh --dev     # writes dashboard/.env.local (controller off, fresh session secret)
cd dashboard
npm install
npm run dev                  # http://localhost:3000 (production runs on 3001)

The same setup by hand: cp .env.example .env.local and fill in the essentials.

PROXMOX_HOST=10.0.0.10
PROXMOX_TOKEN_ID=root@pam!conduit
PROXMOX_TOKEN_SECRET=…
PROXMOX_SSH_KEY=/home/you/.ssh/id_ed25519

Useful switches while developing:

CONDUIT_CONTROLLER=off      # UI/API only — no reconcile against your cluster
CONDUIT_INTERVAL_MS=10000   # reconcile cadence when the controller is on

The first page visit bootstraps the admin account. State lands in dashboard/data/conduit.json (file backend).

Environment reference#

The variables the code actually reads:

Production (HA)#

Target shape: three Proxmox nodes, one panel LXC each behind a keepalived VIP, a per-node agent, GlusterFS for shared artifacts, corosync for the store.

install/install.sh runs the whole chain and stops at the first stage that fails verification. The topology it works from:

# install/cluster.conf — written by the wizard, sourced by everything
CONDUIT_NODES="10.0.0.1:190:150,10.0.0.2:191:100,10.0.0.3:192:100"
VIP="10.0.0.50"
SSH_KEY="$HOME/.conduit/conduit_ed25519"
KEEPALIVED_PASS="…"        # generated per cluster
KEEPALIVED_ROUTER_ID="51"

Secrets: the agent token is generated for new clusters (or recovered from a node when re-provisioning an existing one — never regenerated over a live cluster), the ssh keypair is created when missing, keepalived gets a random password per cluster, and env files everywhere are mode 600. The summary names where each secret lives; none are echoed in full.

The modules wrap the proven scripts/, which remain usable on their own:

All of them read the topology from install/cluster.conf when present (CONDUIT_PANEL_NODES / CONDUIT_NODE_IPS / KEEPALIVED_* overrides) and fall back to their built-in defaults otherwise.

Production panels run with CONDUIT_STATE_BACKEND=agent and CONDUIT_VIP=<vip>. scripts/deploy-panel.sh also writes CONDUIT_ENFORCE_MACHINE_AUTH=1, which since 2026.8.304 means the same as leaving it unset. Connectors need no manual install — the reconcile provisions the jar into every game container and keeps it current.

Joining a second cluster#

Two Conduit installs at two providers stay two Proxmox clusters (why is measured in federation.md) and become one fleet: one servers tree, one account list, one permissions page, and control of the other cluster's instances that keeps working while its panel is down. This is what a join does, in the order it happens, and what it hands over.

Before you start. Each panel must be reachable from the other over the internet at the address the join uses, http://<ip>:3001 at first or, better, an https hostname. Every cluster needs an id of its own: two installs cloned from one image share theirs, and the panel refuses to link them until one is re-minted (the estates card says so rather than renumbering the mesh). Give both panels a public address to advertise, CONDUIT_PUBLIC_URL or a proven cluster hostname, or the join comes out one-way.

1. Mint an invite on the first panel. Nodes, estates card, invite. POST /api/fleet/invite {label} answers with a code like K7M2-QX4T-9BRD-HFNW, shown once. It is good for one hour, works once, and only its sha256 is stored, so a copy of the store does not contain a working invite. Anyone holding the code can redeem it, so it travels like a password.

2. Join from the second panel. Nodes, estates card, the first panel's address and the code. POST /api/fleet {url, code, label?} does the rest: a bare address becomes https://; the panel mints a grant for the other side and sends PUT /api/fleet/invite there with the code, its own address and that grant. The far side compares the code in constant time, spends it, records a grant for this panel, checks the address it was handed (public addresses only, since anyone with an invite code could have typed one), probes it, and stores this panel as a peer too. One action links both sides; the reply says mutual: "linked" or "one-way", and one-way is repaired by adding the missing peer by hand with a token.

3. Manage together is on from the first tick. A new peer starts with the mesh link offered and management allowed, in both directions. On the next leader tick the grant widens to the mesh paths; the link reads linked once the far side has offered too; the tick after that each cluster issues the other reach: a token its node agents accept from the mesh only, a Proxmox token on conduit-fleet@pve with the guest roles and nothing else, and its CA, written as files on every node and read by the far panel over the mesh. From then on power, console, files and exec on the other cluster's instances go straight to its nodes, and edits to its records go to its panel through the relay. Both switches stay on the Nodes page: turning manage off narrows the grant, removes the Proxmox token and rewrites the files on the next tick; detach withdraws reach at once and drops the replica of that cluster's records and its mesh entry.

4. The mesh. Each cluster derives a /24 inside 10.43.0.0/16 from its own id, so the two sides have nothing to negotiate. A collision is detected on both sides; the lower id keeps its range and the other is told which one to move to.

5. Move the join to https. A join made at http://<ip>:3001 sends the grant secret as a bearer on every tick, in the clear, and the estates card marks that row "credentials in the clear". On each panel add a hostname of kind Cluster address on the Domains page (edge-01.example.com), let it be proven, and GET /api/fleet/peer starts advertising it. Every joined cluster moves to that address once it has answered the same call there, and never moves back down to http whatever a peer advertises. Details in domains.md.

What crosses, and what does not. Accounts replicate (the password hash, API token hashes, role, extras, scope and public ssh keys), so the same account signs in on either panel. Sessions do not: you sign in again on the other panel, and an account protected by TOTP cannot sign in with a password alone on a panel that has no secret to check the code against. Services, groups, labels, blueprints, schedules and rules replicate read-only, each edited on the cluster that keeps it. Nothing under network crosses, except the zone credentials while you share them for the front door. What "work together" does hand over is root on the other cluster's nodes through its node agent, and the Secrets page names every cluster that holds it. The credentials, and how each reader is authenticated, are listed in security.md.

Operations#

Every run writes a timestamped plain log and prints its path. The doctor distinguishes a dead service from a dead path — if a replica answers in-cluster but the VIP doesn't answer from your workstation, it says so instead of blaming keepalived.

Upgrades#

install/update.sh

Shows the plan, builds the bundle, snapshots the store to /var/lib/conduit/backups/ — and refuses to continue if the live store looks wiped — then ships bundle + connector jar to every node, restarts the replicas, and verifies each one: units active, panel answering, VIP held, connector build markers matching. Environment files are never touched. Backends pick the new connector up on their next restart — or push it fleet-wide from the Plugins page.

--dry-run previews, --no-build reuses the existing .next build, --yes skips the confirmation. scripts/upgrade-live.sh remains as a shim to this flow.

From the panel instead#

Point the cluster at a hub (Settings → Conduit Hub) and it can install releases itself: the sidebar's cluster card shows the version and turns brand-coloured when the hub has a newer one, the dialog runs seven preflight checks, and the rollout replaces each replica in turn — the one serving you last. install/update.sh stays the fallback and the bootstrap: a cluster deployed before the pipeline existed picks up its version stamp and replica inventory on its next run of it.

To host the hub, clone this repository wherever it should live and run hub/:

useradd -r -s /usr/sbin/nologin conduit-hub
mkdir -p /opt/conduit/hub/data && chown conduit-hub /opt/conduit/hub/data
printf 'HUB_PUBLISH_TOKEN=%s\n' "$(openssl rand -hex 24)" > /etc/conduit/hub.env
cp /opt/conduit/hub/conduit-hub.service /etc/systemd/system/
systemctl enable --now conduit-hub

Then cut releases from a clean checkout with HUB_URL=… HUB_PUBLISH_TOKEN=… hub/release.sh. It builds the same bundle update.sh ships, so what clusters install is what a manual rollout would have deployed. The whole system — preflight, rollout, recovery, telemetry — is walked through in updates.md.