Updates & the Hub

How a Conduit cluster learns that a new version exists, how it installs one on itself, and what — if you leave telemetry on — it says about itself while doing so.

There are two halves. The hub is a small server you host: it holds published releases and serves a feed. The panel watches that feed and can install what it finds, replacing all three of its own replicas including the one you are looking at. Neither half needs the other to exist: a cluster with no hub configured simply never mentions updates, and a hub with no clusters is a web page with nothing on it.

Installing an update#

The bottom of the sidebar shows which build this panel runs. When the hub offers a newer one the row turns brand-coloured, gains a pulsing dot and reads Update to 2026.8.26. You do not have to reload to see it: the panel that holds the VIP re-reads the feed every few seconds while any browser has the panel open, and pushes the change over the live stream — a release appears within about ten seconds of being published. When nobody is watching it backs off to ten minutes, because nothing is waiting on the answer.

Click the row and a dialog opens with what is about to be installed — version, publish date, bundle size, the artifact's sha256 — and the release notes.

1. Run checks. Seven of them, each reporting what it found. Nothing has been changed yet.
2. Update to stays disabled until all seven are green.
3. The rollout replaces each replica in turn, narrating as it goes. The replica serving you goes last; when it restarts the panel is briefly unreachable and then comes back on the new build. If it does not come back, the VIP moves and the panel returns on a sibling instead — see below.
4. The dialog reports the result once, to whoever was watching, and asks you to reload — your browser is still holding the previous bundle's JavaScript.

install/update.sh from a checkout does the same rollout from a terminal and remains the fallback. It is also the bootstrap: a cluster installed before this pipeline existed picks up its version stamp and replica inventory on its next run of that script.

Several versions behind#

It goes straight to the newest. One install, never a chain. A cluster on 2026.8.1 with releases up to 2026.8.33 published is offered 2026.8.33, and installs exactly that — the thirty-one bundles in between are never downloaded, extracted or run. The hub's /v1/feed returns a single latest rather than a list, and runUpdate takes that version as its target and nothing else.

That is a deliberate property of the artefact, not a shortcut. The panel ships as one self-contained standalone bundle with no incremental state inside it, so replacing .1 with .33 is the identical operation to replacing .32 with it. Stepping through the intermediates would mean thirty-one restarts of every replica to reach the same bytes.

What makes it safe is where migrations live. lib/migrate.ts runs on the leader at boot, each migration is idempotent and only touches records that still need it, and the newest build carries every migration ever written. So the build being installed is the one that knows how to fix up whatever older shape it finds. The rule that follows, and it is worth stating because nothing enforces it:

A migration must be able to run against any older state, not merely the one release before it. Since a cluster can arrive from arbitrarily far back, "the previous version already did X" is never a safe assumption to build on.

Two consequences of only ever offering latest. The panel cannot install or pin a specific version — the hub does serve /v1/releases, but nothing in the panel calls it. And rollback is the .old tree on disk, which is one version back, not an arbitrary one. If a bad release goes out, the recovery is to publish a good one rather than to select an old one.

What the preflight checks#

A green set expires after five minutes, and starting a run re-checks that the feed still offers exactly the version and hash that were verified.

How the rollout works#

The bundle is pushed into each container and extracted beside the live tree, never over it:

/opt/conduit-panel.new     ← extracted here, checked for server.js and the version it claims
/opt/conduit-panel         ← two renames swap it in
/opt/conduit-panel.old     ← what was displaced, kept

install/update.sh unpacks over the live directory, which is fine with a person watching a terminal and not fine for something running unattended — a crash mid-extract would leave a half tree that systemd would restart into. Here the window in which a container has no panel directory is two renames, and the previous build stays one move from being restored.

Order matters: siblings first, the replica serving you last. By the time it replaces its own tree the identical bundle has already booted and answered on two other machines. Its own restart is handed to systemd-run detached, because a plain systemctl restart would kill the panel's own process group — including the SSH client carrying the command. The run is therefore finished by the process that boots afterwards, reading what its predecessor left in the store.

Progress lives in the shared store rather than in memory, for the same reason: the machine writing it deliberately stops existing before the end. That is also why the log you see in the dialog is persisted (and short — the store lives on pmxcfs).

When it goes wrong#

• A replica does not come back on the new version. It is rolled back from .old, verified, and the run stops. The rest of the fleet stays on the build it was serving — consistent-old beats mixed-broken.

• The leader dies mid-run. Its heartbeat goes stale; the next leader marks the run failed and alerts. Replicas already updated stay updated. Re-running converges.

• The leader restarts into a broken build. keepalived health-checks the panel port, so the VIP leaves a leader that stops answering: three failed probes two seconds apart drop its priority below the backups', and a sibling has the address about eight seconds after the panel went quiet — measured on the live cluster, not estimated. The run is then finished, and reported failed, by whichever replica picked it up, instead of by nobody.

  That replica also puts the failed one back. There is not much to deliberate about by then: the container is not answering, the VIP has already left it, and the build it was serving minutes ago is on its disk. So the finalizer runs the rollback itself rather than printing a command and waiting for someone to read it. One attempt, no retries, and the tree that failed is kept as /opt/conduit-panel.failed — automatic recovery that destroyed the evidence of what went wrong would be a bad trade. It reads what is actually in .old first, because a second bad release in a row can leave the previous build being the one that just failed, and rolling onto it would restart the container to no purpose while reporting a recovery.

  End to end on the live cluster, from a leader restarting into a bundle whose server.js had been removed: panel dead at 13:05:33, VIP on a sibling at 13:05:40, rolled back and answering again at 13:07:35, VIP home at 13:07:56. Two minutes and twenty-three seconds, nobody asked.

  Only when the rollback cannot be done or does not take does the alert fall back to asking for hands, and then it carries the command:

  Copy

  pct exec <vmid> -- bash -c 'rm -rf /opt/conduit-panel && mv /opt/conduit-panel.old /opt/conduit-panel && systemctl restart conduit-panel'

  You are reading that alert on a different replica; the VIP has already moved.

  What the check deliberately does not test is anything the three replicas share. The store is one file replicated by corosync — a check that read it would fail on all three at once and pass the VIP round a circle without repairing anything, because the fault would follow it. So it asks one local question, GET /api/health on loopback, which touches no state at all: is the panel process in this container answering. That is the only question a failover can answer. A 401 counts as a yes (the auth wall answering is an answer), and so does a 404, which is what an older bundle returns — a guard that a rename could break must fail safe.

  The window is sized against a measurement rather than a guess: a legitimate systemctl restart conduit-panel is answering again in well under a second, so at one probe every two seconds a planned restart can darken at most one check, and it takes three to move anything. Coming back is deliberately slower than going away — a recovered leader must be healthy for twenty unbroken seconds before it takes the VIP back, which a crash-looping panel can never manage.

Both outcomes raise an alert and an activity entry.

Running your own hub#

hub/ is a single zero-dependency Node process. Give it its own small host — it holds every release artifact you have ever published, so it wants disk and a life independent of the cluster it serves. A plain container is plenty: the reference one idles at about 10 MB.

# Node >= 18 for global fetch. Debian/Ubuntu ship something older, so take it from NodeSource.
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && apt-get install -y nodejs

useradd -r -s /usr/sbin/nologin conduit-hub
git clone https://github.com/admin-dev/Conduit /opt/conduit
mkdir -p /opt/conduit/hub/data && chown -R conduit-hub /opt/conduit/hub
install -m 600 /dev/null /etc/conduit/hub.env
printf 'HUB_PUBLISH_TOKEN=%s\n' "$(openssl rand -hex 24)" > /etc/conduit/hub.env
cp /opt/conduit/hub/conduit-hub.service /etc/systemd/system/
systemctl enable --now conduit-hub

Moving an existing hub is an rsync of hub/data/ and nothing else — releases, artifacts, telemetry and crash signatures all live there, and the feed keeps its history rather than starting over at whatever you publish next.

Point the cluster at it in Settings → Conduit Hub, in the panel. Not through the API: /api/hub requires a session on purpose, because it also carries the cluster's own id.

Config is four environment variables: CONDUIT_HUB_PORT (8787), CONDUIT_HUB_HOST, CONDUIT_HUB_DATA, and HUB_PUBLISH_TOKEN — with the token unset, publishing is disabled and the read side still works. Put TLS in front of it; the hub honours X-Forwarded-Host and -Proto when building artifact URLs.

The URL is validated by actually reading its feed, not by looking like a URL — and install/doctor.sh re-checks it from a node on every run, because a hub reachable from your workstation but not from the cluster is exactly the failure that would otherwise go unnoticed until an update silently never appeared.

What the dashboard shows#

GET / is a page anyone can read. It leads with releases — every published version with its age, commit, size, notes and a download link, plus how many installations run each one — and an adoption figure for what fraction of the network is on the newest release. Under that: active installations, servers, players, hardware totals, a 30-day activity chart, the server-software and feature mixes, and the week's crash signatures. It refreshes every thirty seconds.

Cutting a release#

Versions are calver in the repo-root VERSION file (2026.8.26). Bump it, commit, then:

HUB_URL=https://hub.example.com HUB_PUBLISH_TOKEN=… hub/release.sh

It refuses a dirty tree, because the artifact records a commit and that commit had better be checkoutable. It builds the bundle with the same bundle_build install/update.sh uses, so what clusters install is byte-for-byte what a manual rollout would have deployed. Release notes default to the commit subjects since the last released tag. On success it tags v<version> locally; pushing the tag is your call.

The hub keeps the highest version ever published as latest, so re-publishing an older one to replace a bad artifact does not roll the feed backwards.

Publish to the origin, not through a CDN. If the hub sits behind Cloudflare, the 20 MB PUT comes back 403 Request forbidden by administrative rules from the edge — the hub never sees it. Point HUB_URL at the hub's own address for the publish (http://10.0.0.8:8787 here); clusters still read the feed over the public hostname, and the artifact they download is the same file.

If the hub is gone#

The hub is one box, and until recently that was the whole story: with it down, a cluster could see nothing newer and had no way to install anything, even though every byte of the release existed on a machine somewhere.

So release.sh also puts the tarball on GitHub Releases, and the panel falls back to reading it from there. Not a peer — a reachable hub always wins, because it is what the release was published to — and a mirrored read still records the hub as failed, so the "hub unreachable" alert and the settings card keep telling the truth about it.

The metadata rides in the release body as an HTML comment, which GitHub renders as nothing:

<!-- conduit-release {"version":"2026.8.37","commit":"…","sha256":"…","size":20628886} -->

One description of a release, rather than a sidecar file that can drift from the artifact beside it. lib/github-mirror validates before it believes any of it: calver version, 64-hex sha256, a tag that agrees with the trailer, a <version>.tgz asset of exactly the stated size, not a draft, not a prerelease. Anything it cannot fully verify is skipped rather than guessed at. The sha256 is the hub's own, and the update preflight hashes the download against it on every node, so installing from the mirror is exactly as safe as installing from the hub.

Mirroring never fails a release: the release is published the moment the hub has it. A missing gh, no network, or a declined push prints a warning and the command to do it later.

A private repo needs a token, and Conduit's own is private today. Anonymous callers get a flat 404 from the API — which reads as "no such repo" and sends you looking for a typo — and a private asset is not served over browser_download_url at all; it has to be fetched from the API endpoint with Accept: application/octet-stream. The panel uses the same GitHub token the source builds use, set in Settings → Integrations, and addresses the asset through the API whenever it has one. Credentials are only ever attached to api.github.com, never to a redirect target. With no token and a public repo, everything works anonymously.

Unauthenticated API reads are capped at 60/hour per address; a token lifts that to 5000. The panel asks at most once every 15 minutes and only when the hub is already failing, so neither is close.

Proven end to end on 2026.8.37: the mirrored asset downloaded to 20,658,006 bytes with sha256 b52f1c3f7cde326d335e…, byte-identical to what the hub published, and every node can reach api.github.com.

A version bump is not a release#

Worth saying plainly, because it cost a day. 2026.8.145 was built, tested, committed, version-bumped and tagged, and the publish step never ran. Everything local agreed it had shipped — git log had the bump, VERSION read .145, the tree was clean, the tests were green — and the fleet sat on .144 with two reported bugs still live. The panel could not have known: it compares itself against the feed, and the release it was missing had never been put there.

Two places now say so, both reading the same install/lib/release-state.sh:

• hub/release.sh, in its Plan section before the confirm, names any version this repo declared that the hub never received — and warns if the version you are publishing is already on the hub from a different commit, since clusters key downloads on the version string and one that already has it will not fetch it again.
• install/doctor.sh compares this checkout's VERSION against the hub's newest and says when the checkout is ahead.

Both stay quiet when they cannot see the hub, and when the feed has no version in common with this repo's history — a stranger hub or a typo'd URL produces no finding rather than a list of every version you ever had. When that check was first run against the real feed it found three more: 2026.8.129, 2026.8.111 and 2026.8.87 had all been bumped and never published. Those did no harm because a later release followed within hours; .145 was the last one, so nothing covered it.

Rolling it out#

install/update.sh still exists and still works — it is what you want when the panel is too broken to update itself. Everything else goes through the hub: publish, then either wait for the cluster to notice within the hour or press Update in the panel. That path runs seven preflight checks first — this replica holds the VIP, the release is genuinely newer, the store reads and writes and has just been snapshotted, every replica is reachable with disk to spare, the bundle matches the hash the hub published, every node hashes it identically, and nothing else is in flight — and then takes the replicas one at a time, the one serving you last.

Measured on the live three-node cluster, twice, probing every 200 ms throughout a full rollout (scripts/probe-availability.sh):

The VIP does not move. Each replica restarts in place and nginx sends the request to a sibling, so the served path never breaks. :3001 is the panel process itself with no nginx in front of it, so it is down for exactly as long as that one process takes to restart — which is what the second row measures, and it is the same 0.6 s both times.

A single-node install has no sibling to fall through to. Its three blips are 0.2 s each and more than a minute apart, not one outage; where nginx is in the path, a browser gets a small self-refreshing "Conduit is restarting" page instead of a 502, and it comes back on its own.

Telemetry#

On by default, with the switch in Settings → Conduit Hub, which spells out the entire payload rather than linking to a policy. Every half hour the leader sends:

• a random uuid minted once for the installation, and its secret
• the version and commit it runs, and how long it has been up
• counters: nodes, services, servers, players
• the platform string (linux/x64, Node version)
• the software mix as kind:version → count — paper:1.21.11 → 11
• which Conduit features are switched on
• hardware totals: cores, memory, containers, disk

Every value is a count, a boolean or a short enum. The software map is keyed by kind and version rather than by what anything is called, so it answers questions about Conduit without answering any about you. There are no hostnames, no addresses, and no group, service, world, domain or player names. Country comes from an edge proxy's header if there is one — the hub never geolocates, so an address stays something the socket knows and then discards. Source IPs live only in an in-memory rate limiter.

The rule for adding a field is not "is this useful" but "would an operator be surprised to find it on a public web page" — because that is exactly where it goes.

Crash signatures#

Split in two, and the split is the point.

A breadcrumb — the last forty lines a dying server said — is exactly what its own operator needs and exactly what nobody else may see. So before anything leaves the cluster it is reduced to a signature: the exception class, the first stack frame that is neither the JDK nor the server itself, and the software version. That is public, and it is what lets the hub say

NoClassDefFoundError out of net.essentialsx on Paper 1.21.11 — seen by 12 clusters this week

which is the shape of a bad plugin-and-build pairing without a line of anyone's log. Aggregates count distinct clusters, so one server in a restart loop cannot manufacture a trend.

The signature is null when the lines do not describe a crash at all — a server stopped on purpose leaves a breadcrumb too, and inventing an exception for it would poison the aggregate.

The detail travels separately, into your own cluster's private area on the hub, readable only by whoever holds that cluster's secret (GET /v1/diagnostics). The hub is holding someone else's logs there, so it keeps few of them, ages them out, and never lets them near a public route.