4. Network & gameplay features#

Each entry: what/why → how to steer → integration class. Classes: NATIVE = connector Java (migrated off Skript for perf/reliability) · ADOPTED = live Skydinse Skript running unchanged · GLUE = small Conduit-only Skript adapter.

4.1 Chat — NATIVE#

Live layout [extraprefix][rank prefix][head][name] : msg rendered by ConduitChat.

• Format template: network.chatFormat (Settings page / POST /api/network), placeholders {extraprefix} {prefix} {head} {namecolor} {name} {message} {server}. Applied live via heartbeat.
• Name colour comes from the LP prefix (after the last |) → edited in the Permissions UI.
• Extra prefix (the pre-prefix before the rank): LP meta extraprefix (meta.extraprefix.<v>), resolved cross-server by GET /api/luckperms/effective. On TimeSMP it is auto-set to the player's faction name by conduit-faction-tag.sk (GLUE) — live-identical [Faction] [Rank] Name.
• Global chat: network.globalChatGroups — group IDs whose servers share one cross-server chat channel over redis (live's globalchat). Others stay local (minigames must).
• Colour perms: conduit.color, conduit.color.hex; slowmode/filter bypass conduit.chat.bypass.

4.2 Chat filter — NATIVE#

ConduitFilter = the live newfilter (regex + spam heuristics) migrated to Java. Patterns are served from the panel (filterPatterns in the heartbeat; managed on the Moderation/Settings pages) — edit patterns in the panel, applied live, no restart.

4.3 Tablist — NATIVE#

• Backend tab: network.tablistHeader/Footer templates (+ per-locale variants auto-derived; the Blossoms gradient is re-lettered per language), LP prefix+name+suffix entries (network.tablistPrefixes), per-player locale from the LP lang meta (case-insensitive).
• Live placeholders: {online} {server} {ping}/{pingcolor} {name} {prefix} {blossoms} {logo} plus {tps} (local, one decimal), {node} (the Proxmox node this instance runs on) and {event} (active event-window title, empty when none) — the last three fed per heartbeat by the panel, re-rendered every tab tick.
• Cross-server tab: network.tablistGroups — sets of tasks whose players all see each other (live TAB global-playerlist); injected by the proxy connector. Vanished players are hidden.

4.4 Bossbar — NATIVE#

The live lobby bossbar, connector-rendered: network.bossbar = enabled + slides (each = lang catalog+key, colour, style, interval) + targets (task names). Text resolves per-player locale. Edited on the Settings page; applied live. The old lobby/visual/bossbar.sk stays disabled.

4.5 Resource packs — NATIVE#

Hosted packs with a stable /api/packs/<id>/latest URL (replace-in-place keeps the URL, bumps SHA-1). Per pack: scope network (proxy sends) or backend (Paper sends on join), targets, required, prompt, order, enabled. The lobby's required pack is additionally delivered config-phase via server.properties (delivery:"config" — panel writes the RP fields; prompt shows on the join screen like live). Manage on the Packs page / …/api/packs.

The URL is built from a public hostname, and that is the difference between a pack that works and one that only appears to. It used to come from the cluster VIP — a private LAN address — so a pack applied on a test client inside the cluster and silently failed for everyone on the internet, with nothing in the panel to say so. The endpoint itself was never the problem: /api/packs/<id>/latest is auth-exempt and always answered over the domain. Set the hostname on the Packs page (…/api/packs/domain): mirror follows the primary domain, or give packs a subdomain of their own so a pack URL does not advertise the panel's. The DNS record is created for you, proxied — a pack download is ordinary HTTPS and the proxy keeps the origin private. It is used only once a real request to a real pack URL comes back OK over the public path — not merely once DNS resolves. That distinction cost an outage: packs.<zone> resolved, was proxied and pointed here, and every download still failed 526, because the panel's Origin CA certificate is issued for the panel hostname alone and Cloudflare will not accept a certificate that does not cover the name it asked for. DNS proves a name points here; it says nothing about whether this host can serve TLS for it. The probe uses a pack id that cannot exist, so a working path answers 404 from the app — which proves DNS, TLS and routing all the way to the route rather than just the first of the three. A hostname that later stops serving is demoted rather than deleted, so it resumes by itself once the cause is fixed, and the base falls back to the primary domain — public, already covered by the certificate, same auth-exempt route — before the VIP is ever considered. A record that resolves DNS-only is called out too: an Origin CA certificate is only trusted through Cloudflare, so an unproxied record fails TLS in the client.

Using a subdomain of its own therefore needs the origin certificate to cover it. The cluster's Origin CA certificate is now issued for example.com and *.example.com (previously the panel hostname alone), so any subdomain works without reissuing per name. It lives at /etc/conduit/tls/{cert,key}.pem on each panel replica, behind nginx. Replacing it: back up first, install on a non-VIP replica, gate the reload on nginx -t, verify, then roll — VIP holder last, and reload rather than restart so live connections are not dropped. Getting it wrong takes down the panel itself, not just the feature being fixed. Changing the hostname rewrites server.properties for config-phase packs; the SHA-1 is untouched, so nobody re-downloads.

4.6 Moderation: bans/mutes/kick — NATIVE + panel#

• In-game: /ban /tempban /unban /mute /tempmute /unmute /kick (ConduitBans, perms conduit.ban|mute|kick) → panel /api/bans → gamestats conduit_punishments. Enforced at pre-login (ban) and chat (mute). Live punishment history was imported.
• Panel: Moderation page = list + issue + lift (typed-name confirm) + action log.
• Player reports with follow-through: /report <player> <reason> (rate-limited) lands in the Moderation staff inbox and raises a staff alert; resolving/dismissing queues a store-backed notice, and the reporter gets an in-game "your report #ab12 was handled — thanks!" on whatever server they're on (connector delivers once, acks via the heartbeat; survives panel restarts). lib/reports, GET/PATCH /api/reports.
• Staff alert (live punishalert): conduit-punishalert.sk (GLUE, all servers) watches conduit_punishments and fires a localized, perm-gated network announce (sd.sup) — catches both in-game and panel-issued punishments. Catalog key conduit/PUNISH.alert.

4.7 Friends & Parties — NATIVE#

/friend add|accept|deny|remove|list and /party invite|accept|leave|kick|disband|list (ConduitFriends/ConduitParty) → panel /api/friends /api/parties → gamestats DB. Cross-server online status from the live roster; invites notify across servers. Live friends data imported. Panel Social page shows friendships + active parties. (The old Skript friendsystem stays only for its lobby joinbanner visuals; parties.sk is replaced by the connector.)

4.8 Private messages — NATIVE#

/msg + /r (ConduitMsg) — cross-server via the conduit:msg redis channel; social-spy perm conduit.msg.spy. Live bungeesystem/msg equivalent.

4.9 Announces & broadcasts — NATIVE + GLUE#

• The connector detects network join/leave/switch and pushes localized announces (catalog conduit: NETWORK.join/leave/switch), perm-gated (conduit.networkswitch.info, …), rendered per-player locale. Backends poll /api/connector/broadcasts (~0.25s) so it feels like redis pub/sub.
• Team presence (live accountmgr): conduit-team-notice.sk announces TEAM.online/offline to sd.team holders.
• Skript can announce anything: conduit announce permission "<perm>" "<catalog>", "<KEY>", <args…>.
• Autobroadcasts (the rotating tips): live Skript adopted (conduit-broadcasts.sk + essential/autobroadcasts).

4.10 Heads & identity — NATIVE#

• ConduitHeads: the live RP-font player-head glyph (conduit head glyph of "<name>"), async skin fetch + cache (scoreboards re-render on join via a head-refresh GLUE).
• PlayerIdentityCache: non-blocking uuid↔name (conduit name of / conduit uuid of) — replaces main-thread .dat reads; persisted playernames.json, seeded with 26,765 live names. Powers Orders/AuctionHouse/etc. offline-name rendering.

4.11 RediSkript (redis bridge) — NATIVE#

Full RediSkript-compatible syntax on the connector's Redis client (replaces Skyd.Redis.jar): send redis message … to channel …, on redis message: (+ redis channel / redis message), redis strings/hashes/lists (§5). PSUBSCRIBEs * so every channel a script uses is received (essential, ashlyn, bungeesystem, playerdata, custom). Endpoints+password arrive via heartbeat from the panel-managed Redis egg.

4.12 Language system — panel-owned#

Single source of truth = panel Languages page (…/api/languages): base locale, locales with fallbacks, message catalogs (imported from the live Language repo + Conduit-native catalogs). Runtime: the panel generates conduit-lang-data.sk (populates {l.<system>::<locale>::KEY}) and pushes it per-host group-scoped (POST /api/languages/lang-data). Player locale = LP lang meta (set by the ported language GUI, synced network-wide); lang()/conduitSetLang() in conduit-lang.sk (GLUE). Formatting quirks of live (;→newline, &s/&g/&i, &#hex) are baked at import/format time. Announce/bossbar text resolves from the same catalogs.

4.13 LuckPerms — panel-owned#

Panel Permissions page = full LP editor (groups, users, tracks, weights, prefixes, meta) on the Postgres backend, synced network-wide via LP messaging. GET /api/luckperms/effective serves prefix/suffix/extraprefix to connectors for chat/tab. Managed install set via /api/luckperms/install; credentials kept in sync by the reconcile. Live sd.*/rank perms were imported, so live staff keep their nodes. Conduit-specific nodes: see CONDUIT-PERMISSIONS.md.

Connection budget (learned the hard way): every LP backend defaults to a Hikari pool of 10 max / 10 MIN-IDLE — ~10 servers ≈ 100 idle conns = stock Postgres max_connections → "remaining connection slots are reserved…" breaks the panel LP APIs, the Databases page and new LP boots. Conduit therefore writes data.pool-settings: {maximum-pool-size: 4, minimum-idle: 2} into every managed LP config, provisions Postgres with max_connections = 200, and the panel itself uses ONE bounded pg pool (max 8) behind lpClient(). Self-healing drift: the reconcile's keep-in-sync greps each server's actual config (per-platform path — Bukkit plugins/LuckPerms/, Velocity plugins/luckperms/!) and repairs a stale redis password (rewrite + restart, messaging is broken anyway) or missing pool-settings (rewrite only — applies on the next natural restart, no network-wide kick). What drifted is surfaced as a small amber notice on the Permissions page (lpDrift() → /api/luckperms/status.drift).

4.14 Vanish — NATIVE + GLUE#

/vanish (conduit-vanish.sk, GLUE) toggles vanished metadata + live-style feedback; the connector hides vanished players from anyone without conduit.vanish.see (tab + world), across servers (proxy tab included). Redis-persisted across relogs. vanishTick is state-tracked (acts only on a hide↔show transition, so it never re-styles the tab each tick) and self-heals stuck hides: on servers where vanish is the only legitimate hide reason (i.e. not mid-minigame-round, which hides players deliberately under the same key), it force-shows any player a viewer should see but can't — repairing the "tab count says N but the entry is missing" client desync after an unvanish.

4.15 Games / rounds (Ashlyn) — NATIVE host + ADOPTED gamemodes#

• ConduitGames (Java) replaces ashlyn-server.sk: hosts multiple virtual rounds per server (per-task Games config: registername, roundnames, maxgames, minServiceCount, startAt, lobbyCountdown, modes, hooks, maps), reports them in the heartbeat, and the panel keeps a network round registry.
• Gamemodes (MM/BP minigame-basics + mode Skript) are ADOPTED; conduit-games-bridge.sk + conduit-ashlyn-mirror.sk (GLUE) translate the hook contract: rounds mirror into {ashlyn.data::…}, ashlynFindGame→conduit find game, ashlynConnect→conduit connect. Skript sees conduit game boot/start/join/leave/end/shutdown events (§5).
• Maps: SchemFlow + S3 (all 14 live maps under the SF_ prefix); conduit map <world> is ready condition + conduit map ready event for async load. WDB SQL is retired.
• The lobby game GUI + game NPC holograms read the mirrored registry; portal/NPC status is live.
• Steering: per-task Games section in the panel; rounds visible via GET /api/connector/rounds.

4.16 Susi (network routing in Skript) — GLUE#

conduit-susi.sk reimplements the live Susi selection on connector data: {susi.*} registry mirrored from conduit network servers, susiconnect/susiConnectBest (lowest / upto:N) via send … to conduit server, susiconnectlobby, player-count/online strings for portals + NPC holos. Susigroup alias: Conduit tasks (timesmp-spawn) map to live cross-task susigroups (TIMESMP) via {susi.groupalias::<task>} — portals, seamless-spawn and playercounts key off it.

4.17 World sharding — NATIVE (connector-owned)#

• ConduitSharding: panel-computed X-strip grid, vanilla border + seam warnings, boundary handoff keeping exact position (/api/connector/transfer + pending), Redis player-data sync. Grid view on the task's World tab. Strip geometry is also readable from Skript (conduit shard strip min|max of <world>, half extent, index, server at, border distance at, servers) so game content can use the panel's numbers instead of recomputing them; each yields nothing when there is no grid, so callers can test for unset and keep their own fallback.
• Enabling it: use PATCH /api/tasks/:id with a sharding object. POST /api/tasks/:id/sharding — which is what the panel's "Enable sharding" button calls — is destructive: it runs regenWorldWithSeed, which rm -rfs world, world_nether and world_the_end on every region to rebuild them on a shared seed. That is correct for a brand-new shard and catastrophic for a populated one. The button's OFF path is the plain, safe PATCH.
• serverlogic is retired (2026-07-25). serverlogic.sk used to own TimeSMP's strips and the two collided — both crossing players on the same 10-tick loop with the same coordinate format. The connector now owns them: {smp.conf::extension.serverlogic.enabled} is false, which makes the whole script inert (everything in it gates on …serverlogic.this, only set by LoadScaling()), and rtp.sk reads the shard expressions instead of {serverlogic.conf::*}. The file is left on disk, so reverting is that one flag plus a restart. Boundaries were diffed first and did not move: legacy's own IsServerOuterServer reported 216 centre / 217 east / 218 west, matching the panel's strips exactly, and live borders were already 120000/15000/120000 on all three.
• Three things worth knowing, all found by reading the two implementations side by side:
  • The seam buffer must measure from the strip that owns a coordinate, not from ours. A crossing player may stand up to HARD_CAP (25) blocks past their boundary while the transfer lands — inside the neighbour's 30-block buffer — so measuring from our own strip let edits through there.
  • Inventory-share stays on under sharding. It used to disable itself, on the reasoning that sharding syncs its own instances; but the timesmp-cluster group spans the sharded regions and the unsharded spawn, so disabling left the regions ignoring a key spawn was still writing. Only a quit that is itself a shard handoff is skipped now.
  • Strip assignment must not follow live membership. Members are derived from reporting instances, and liveServers() drops anything unseen for 30 s — so a region restart recomputed the grid one member short and moved every boundary. Members are remembered now, and a missing one keeps its strip until the task is back at its desired count. Verified by stopping a region for 60 s: the grid held three strips with the member marked unreachable and borders unchanged.
• Time & weather sync across every strip — automatic, no toggle. A sharded world is one world on several servers, but each ran its own day/night and weather cycle, so a seam crossing could step from noon into dusk. Now one shard is the lead clock and the rest mirror it whenever a world is sharded. The lead is the lowest live strip: each shard renews a 3 s-TTL key conduit:shard:alive:<taskId>:<i> in Redis every second and the lead is whichever lowest index still has one, so a dead lead is replaced within a few seconds with no coordination. The lead keeps the vanilla cycles running (its world is the master) and publishes each overworld's {fullTime, storm, thunder} to conduit:shard:env:<taskId>:<world> once a second; followers freeze their own cycles and apply it — time every tick (a 20-tick jump per second reads as the natural sky speed), weather only on a real change. /time and /weather work from any strip: the command runs locally and, from a follower, is forwarded over conduit:shard:cmd:<taskId> to the lead, which re-runs it on the master so the change comes back to everyone within a second — but only for a sender who could already run it (the relay checks minecraft.command.time/.weather first, so it can't grant the permission by side effect). Redis is blocking, so the whole tick runs async and only touches the world on the global region scheduler, the same Folia-safe path quietWorld() uses. Redis down or no lead ⇒ each strip falls back to its own cycle and re-syncs when Redis returns. The Shards tab shows a read-only "time & weather synced" chip naming the current lead. Namespaced by the new group field (task id) on the shard config block. ConduitSharding.envTick() + ConduitPaperPlugin command handlers; lib/shard-state.ts sends group.
• Seamless switch — no loading screen. Adjacent shards share the seed, so the terrain either side of a seam is identical, position is carried, and inventory/HP/XP ride Redis (ConduitInvShare). What used to spoil it was the switch itself, and the cause was not JoinGame as this said for months: on any client from 1.20.2 up Velocity routes a backend change back through the configuration state, and that round trip is what makes the client throw its level away — JoinGame only arrives afterwards to rebuild what is already gone. The patched proxy (§2.15) therefore stands in for the client through the destination's configuration phase, so the client never leaves play and never drops the world. Armed per server pair with -Dseam.pairs=<task>-*; unset, the jar behaves exactly like stock Velocity.
• Entity ids derived from the player UUID. Keeping the client in play means it also keeps the entity id the previous server gave it — the destination's JoinGame, which would re-teach it, is exactly what the seam suppresses. Usually invisible, until the kept id happens to name a real entity on the destination: then packets about that entity are applied by the client to itself and it dies with "network protocol error". Measured across three crossings — 326→197 fine, 347→500 fine, then 500→497 and the client was gone a second later. So both ends now compute the id from the UUID and agree by construction: no packet rewriting, nothing pinned to a protocol version. The value sits in the top quarter of the positive range ([2^30, 2^31)), which a server would have to spawn a billion entities in one run to reach, so a collision is a practical impossibility while every id stays positive the way server internals expect. Bukkit has getEntityId() and no setter, so the field is set on the NMS entity in PlayerLoginEvent — the last moment before the level indexes the player by id. It verifies the write by reading the id back and logs once whether it took; on failure nothing changes and the seam behaves as before. Only on servers whose instances share one world. SeamEntityId.
• Design history, the corrected mechanism and the phasing are in docs/seamless-transfer.md.

4.17a Env groups — one sky, and optionally one position, without sharding — NATIVE#

Sharding is not the only reason several servers should agree about the world. Two lobbies look wrong under different skies; a pair of servers generated from the same seed is a way to hand players between identical worlds without splitting one. So the clock and the position carry were lifted out of sharding into an env group: any task whose instances should share this, sharded or not.

Two separate opt-ins, and separate on purpose — sharing a sky is not the same wish as sharing coordinates. A lobby pair wants the first and never the second, because landing a player on the coordinates they had on another server is exactly wrong when you teleport them in normally.

• Sync time & weather. The same lead-election and mirror described in §4.17, keyed by the task id instead of the grid. Sharded tasks do this implicitly and ignore the flag.
• Carry exact position. A switch between the group's instances resumes at the same spot, facing the same way — position, world, yaw and pitch, airborne/gliding state and velocity.
• Keep gamemode — the one that is on by default. Walking out of survival and arriving in creative, flying and able to break anything, was possible because the group synced the sky and the coordinates but never the player; the destination's own default won. The mode now rides in the same stash as the position, appended as a trailing field so a mid-rollout crossing between two connector versions degrades instead of breaking, and applied a tick after join because the destination sets its default as part of bringing the player in. On by default because the surprising outcome is a crossing that changes your mode, not one that keeps it — but still an option, since a creative build world beside a survival one is a legitimate pair. Stored only when switched off, so "absent" keeps meaning on for tasks saved by an older panel.

Both are set in the deploy wizard's Advanced options and on an existing server in its edit dialog, Paper-family only (a proxy has no world, a database no sky). Off by default.

The position carry cannot work by stashing on quit, and the ordering is what makes it impossible rather than merely racy: a switch runs destination creates the player and asks where to spawn it → destination reports login success → proxy releases the origin, so the destination has already chosen before the origin has written anything. Every arrival landed on the world spawn. Sharding escapes this by announcing at the crossing, long before the handoff; a plain group has no crossing to announce, so the position is kept current in Redis while the player is online instead — refreshed once a second, written only when it actually changed, so a standing player costs nothing. Consumed once on arrival, TTL'd, namespaced per group, and only honoured for a world that server actually has.

Arrival fall damage is swallowed for a player who was standing when they left: placing them at their exact previous Y lets them drop for a tick or two before the destination has that chunk loaded, and they land taking damage they never earned. Someone genuinely mid-air keeps their fall.

ConduitSharding.envPosTick() / updateEnv(); envConfigForServer() in lib/shard-state.ts sends {group, ordinal, carryPos, clock} on the heartbeat. POST /api/tasks/<id>/env {time, weather} sets either from the panel or the API — it runs the ordinary command on one live instance and lets the existing relay propagate it, so no new connector API was needed.

4.18 Inventory share (HuskSync replacement) — NATIVE#

ConduitInvShare: named groups of services sharing full player state (inventory/EC/HP/XP/effects/ gamemode) via Redis stash-on-quit/restore-on-join. Configure network.invShareGroups (POST /api/network). TimeSMP: group timesmp-regions spans spawn + regions (live HuskSync cluster_id tm2 equivalent). Suppressed automatically where connector-sharding is active.

4.19 Stats & databases#

• gamestats MariaDB (panel-managed): all ported systems repointed via conduit-stats-config.sk
  • conduit gamestats host/port/… bridge exprs — db-ids global, minigames1..3, waitinggames, smps1, citybuild all resolve to gamestats. Includes live imports (friends, punishments, JnR, FFA lootpool, filter patterns…).
• Blossoms (currency): adopted Skript + GET /api/blossoms?uuid= for the panel.
• Minigames→Postgres mirror: …/api/minigames/sync (stats exploration on CT208 PG).
• Action log: conduit log <uuid>,<type>,<title>,<desc> (bridge) → conduit_logs → Moderation page.
• Managed databases (DBaaS) — the Databases page provisions an isolated database + scoped login on a running system engine (MariaDB or PostgreSQL) for a website/bot/API, and (/databases → Managed databases, sudo-unlock gated) now browses and edits them in place. The Browse button opens the record in a viewer that connects as that database's own scoped login into its own database (?dbId=), not the system LuckPerms Postgres: Postgres tenants get the full editor (rows, columns, tables, arbitrary SQL — the reusable TableBrowser keyed by endpoint+engine), MariaDB tenants a mirroring path (rows + SQL, no DDL) at /api/db/mysql; the LuckPerms Postgres browser is the same component with no dbId. Every managed DB row shows its connection strings — internal URI + JDBC, plus external URI + JDBC when exposed — with copy buttons (connectionStrings()).
• Public exposure — a per-database "expose publicly" toggle (PATCH /api/databases) DNATs the engine's port on its node through the firewall so the database is reachable off-cluster. Off by default; the UI warns it opens the whole shared engine port (every tenant + system schema on that instance), the tenant login stays scoped to its own database, writes stay logged, and the firewall access CIDR list is the real gate — the forward is added in desiredForwards(), deduped by node:port so N exposed tenants on one engine collapse to one rule. Endpoints: /api/db/pg?dbId=, /api/db/mysql, /api/databases (POST/PATCH/DELETE). lib/db-viewer.ts, lib/user-databases.ts. With the compose-apps + generic egg above this is the full self-hosting story: app + scoped db in a few clicks.

The databases pages (2026.8.306). /databases is four tabs: Managed databases, PostgreSQL (the LuckPerms store), MariaDB (gamestats) and Redis (the Valkey handoff cache), with deep links #managed, #pg, #maria and #redis for the palette. One editor serves both SQL engines: components/db/table-browser.tsx against /api/db/pg or /api/db/mysql, and both route files are a two-line call into lib/db-route.ts dbRoute(engine), so the switch on action exists once. What differs per engine is confined to lib/db-dialect.ts (quoting, limits, the system database a dbId-less request opens). The one policy in the dispatcher is about who is connected, not about the SQL: the four schema-shaping actions (addColumn, dropColumn, createTable, dropTable) are refused with a 409 on a system connection, because reshaping the LuckPerms Postgres or the gamestats MariaDB is reshaping a schema other services depend on, and open on a tenant's own managed database. Reads, cell edits, inserts, deletes and the SQL console stay open on both; every exec is recorded in the operator audit log under the requester's name and path.

Every managed database has a page of its own, /databases/<id>: the editor connected as that database's own scoped login (the isolation story: this connection reaches exactly this database and nothing else on the engine), its credentials, live usage and the two ways to lose it. The connection strings (internal and external URI and JDBC) arrive masked from the server (*** in place of the password) and are unmasked by Reveal, which is the sudo re-auth, the second password that stays unlocked for about five minutes; Rotate password sits behind the same re-auth, and one re-auth covers both the list page and this one because they share a poll and a revealed flag. Rotating replaces the login's password at once and says so: anything using the old string fails until it is updated. Usage (size on disk, tables, open connections) is its own small route polled every 30 s. "Expose publicly" opens the shared engine port on the node (the login stays scoped to this database, the firewall access list is the real gate) and "Drop" asks for the database name typed back. What this page does not do: it never shows the system engines' credentials; those stay in the credentials vault under Settings.

The Valkey build resumes where it stopped (2026.8.310). There is no Valkey package on the Debian 12 base, so lib/provision.ts redisScript builds it from source (latest GitHub release, pin 9.1.1 when the lookup fails), after first trying the MinIO cache under conduit-artifacts/valkey/<version>/, which the first successful build of a version seeds so every later instance downloads instead of compiling. On a one-core container the build took longer than the ten-minute provisioning window, and because each attempt threw the tree away it could never finish. The tree now stays in /tmp/valkey-src with a .conduit-version marker between attempts, make skips what is already compiled, and only a tree of another version is discarded; the window is thirty minutes and is a bound, not a wait. Nothing else moved: the unit is still redis-server, the config still /etc/redis/redis.conf, and redis-cli is still a symlink to valkey-cli, so the password rotation and replication paths are untouched.

4.20 SchemFlow + S3 — panel-owned#

Node-wide S3 (MinIO) config on the panel (…/api/s3), object browser/upload for schematics + worlds, managed SchemFlow install per service (…/api/schemflow/install), presigned world pulls (/api/worlds/pull). schemflow console command is lowercase.

4.21 Proxy behaviors — NATIVE#

Fallback/try order (panel-editable, single source proxyFallbackOrder), MOTD editor, maintenance (+per-task), pause-on-restart (network.pauseOnRestart: backend dies → players park on limbo/lobby and auto-return when it re-registers), live "Server closed" kick suppression → smooth fallback, per-player RP offers, cross-server tab injection. Limbo (CT209) = NanoLimbo with the live "loading bubble" theme.

4.22 Staff tools — ADOPTED (+ panel)#

• In-game staff GUI (/conduit gui, perm conduit.gui, NATIVE): a chest-UI mirror of the panel's network view — every routable server with live player counts (click = connect yourself via the proxy) plus the open-report count from the staff inbox. Renders from the heartbeat config the connector already holds, so opening it costs no HTTP round-trip.
• supsystem (basicsystems/essential/supsystem.sk): /sup, /spectator, /invsee, /tpo (+/tp intercept for online players), /fly — perm sup.use. Dep teleportPlayer lives in conduit-perm-helpers.sk (GLUE).
• Cross-server teleport (cross-server-teleport.sk, ADOPTED): the live CST redis handshake on channel essential — carries join-actions (e.g. custom:rtp:world_nether) so /rtp dimension routing + region entry work exactly like live.
• Team-perm management (live /teamperms) is replaced by the panel Permissions page.

4.23 Web application management — panel-owned#

An nginx or apache service (either web-role blueprint) gets a Web tab on its service page. Edit the site config in the browser, validate it (nginx -t / apache2ctl configtest) and reload the server live — all via ctExec in the container (operator+). A live preview of the served site renders in an iframe alongside the editor: the bytes are fetched through a panel proxy (/api/services/:vmid/proxy) because the panel can reach the container's LAN IP a workstation browser can't. The preview is fully sandboxed — served under a sandbox CSP with no allow-scripts (an opaque, script-less context however it's loaded, direct link or iframe) and the iframe drops allow-same-origin — so untrusted tenant HTML can never run script on the dashboard origin or touch the session cookie. Linked domains that point at the service (from /domains) are listed with a jump to manage them. Apache installs like nginx: apt apache2, the same /opt/www web root (so the panel surface is uniform), config symlinked into the /opt file sandbox (/opt/apache), and an aliased console shell. Both engines share /api/services/:vmid/web (load/save/validate/reload). provision.installApache, lib/runtime.ts, service page Web tab.

4.23a Node mesh — the private link between nodes#

Conduit assumed for a long time that its nodes share a LAN, because the first three did. The moment a node lives at a provider that hands it one public address, that stops being true: agent traffic, the container-network NAT and every nodeExec cross the provider's network, and on a shared segment that is other people's network too.

Node mesh (/nodes, off by default) gives every node a WireGuard interface and peers each with all the others. Addresses are assigned, not leased — the nth node takes the nth address out of 10.43.0.0/16 (deliberately neither the container range nor the reference cluster's LAN), so they survive a reboot and can be read out of a config instead of asked for. A node that already has an address never gets renumbered, because every other node's config names it and they are all rewritten together — one moved address would break the mesh at once rather than one link at a time.

Private keys are generated on the node with wg genkey, written 0600 and never read back; only the public half travels. The rendered config references the key by path and never inlines it, so the config can be logged and diffed without ever holding a secret. Every AllowedIPs is a /32: this is a mesh of hosts, and a wider mask would have one node quietly claim traffic meant for another. Peers render in name order so an unchanged mesh is byte-identical, and the apply step only restarts the interface when the file actually changed — wg-quick down/up drops every tunnel on the node, which is not something to do once a tick for nothing.

The card keeps two claims apart that look identical if you let them: configured means the panel wrote a file; connected means a node has heard from another. Every peer row shows its last handshake, a peer that has never handshaken says so, and a node missing a key or a reachable address is listed as a gap rather than silently left out of the mesh.

There is a third claim the card makes now, because two were not enough: carrying traffic. A handshake says the two ends can reach each other, not that anything uses the link, and an idle mesh looks exactly like a busy one until somebody shows the counters. Each row carries rx/tx throughput and a running total, the header sums them next to a sparkline of the last forty readings, and the headline is "n of m linked" rather than a node count — those two numbers are not always the same, and the difference is the point.

The counters come from one wg show <iface> dump per node rather than three separate wg show calls, with one detail that is load-bearing rather than tidy: the first line of a dump is the interface, and it begins with the private key. The command drops that line on the node (tail -n +2) so it never crosses the ssh boundary, and the parser refuses to read a four-field interface line as a peer even if one somehow arrives — a private key rendered in the UI as a peer's public key would be a genuine leak, so there is a test that plants one. Rates are worked out against the previous poll and report null rather than zero when they cannot honestly be known: a first reading has nothing to compare with, and WireGuard's counters reset whenever the interface restarts, which Conduit does itself every time the peer set changes. Clamping that to zero would show an idle link during exactly the moment the mesh was being reconfigured.

lib/wireguard.ts decides and renders (pure, tested), lib/wireguard-traffic.ts parses and rates (pure, tested), lib/wireguard-sync.ts applies — the same split as the managed container network, and following the same two rules learned this month: decide over the Proxmox API, apply over ssh, and never let a failed probe read as a negative answer. What this is not: the thing that joins two clusters. Corosync still replicates the store inside one Proxmox cluster, and docs/federation.md now records what happened when joining two live boxes was actually attempted — the link measured 14 ms with almost no jitter, and it failed on colliding VMIDs and on quorum instead.

4.23c One cluster over joined clusters#

Two Proxmox clusters cannot be merged here. That was measured, not argued, and it is written up in docs/federation.md: the link between the two live boxes was fine at 14 ms, and the join failed on five colliding VMIDs, on pvecm add refusing a node that has guests, and on quorum, where two nodes over the internet take /etc/pve read-only the moment either side blinks, which is where the store lives.

What shipped instead, between 2026.8.289 and 2026.8.318, is one Conduit cluster over two Proxmox clusters. Each keeps its own store, its own quorum and its own VMID space. The panels agree on who owns what, act on each other's containers directly, and hold one account list and one service list. The live pair is edge-01 (Hetzner, one node) and edge-02 (24fire, one node); "edge-01" and "edge-02" below. This section is the state at 2026.8.318, and it says which parts are not there yet. docs/federation.md keeps the design history and the measurements.

Three rules sit under everything:

• Every logical record has one owner and one writer. Every other cluster holds a read-only replica. Nothing is merged; a clash is shown.
• Control of a joined cluster's containers goes to its nodes, never through its panel. Power, console, files and uploads cross the WireGuard mesh with credentials that cluster issued and can withdraw in one tick, so they keep working while that panel is down.
• An edit of a record goes to the panel that owns it, as the person who asked. It is refused with a sentence when that panel is not answering.

flowchart LR
    subgraph edge-01["edge-01 (Hetzner)"]
        PA["panel"]
        NA["node: agent :8800, Proxmox :8006"]
    end
    subgraph edge-02["edge-02 (24fire)"]
        PB["panel"]
        NB["node: agent :8800, Proxmox :8006"]
    end
    PA -->|"https relay: store edits, as the person"| PB
    PB -->|"https relay: store edits, as the person"| PA
    PA -->|"mesh 10.43.0.0/16: power, console, files, replica of the store"| NB
    PB -->|"mesh 10.43.0.0/16: power, console, files, replica of the store"| NA
    NB -.->|"reach bundle, served to edge-01's segment only"| PA
    NA -.->|"reach bundle, served to edge-02's segment only"| PB

Joining and the mesh

/nodes, the Joined clusters card. Two ways in. Invite mints a one-time code on one panel: twenty characters of a 31-letter alphabet with no O/0 or I/1/l (about 98 bits), shown once, stored as a sha256 hash, compared in constant time, good for one hour, spent on first use, and kept for seven days afterwards so "who joined, and when" can be answered. Join on the other panel takes that panel's address and the code, checks the code without spending it, and redeems it for a credential of this cluster's own (a grant), never for the far panel's machine token. The joining side sends its own address and a grant of its own in the same request, so the join is mutual in one step (app/api/fleet/route.ts, lib/fleet-invite.ts). Pasting a machine token still works and is the worse option: it is admin over there and never expires. A peer is probed before it is saved, so a typo is a sentence in the form.

An address typed without a scheme is https://. An explicit http:// peer keeps working, and its row says credentials in the clear: the grant secret is a bearer on every tick to that address. The cluster address kind under Domains (below) is what moves such a row to https on its own.

Grants carry caps. Every grant may read the node list (/api/nodes). A mesh link adds /api/fleet/mesh, /api/fleet/node and /api/fleet/partition; "work together" adds /api/fleet/services and /api/fleet/relay plus role: "admin". Caps are recomputed from the peer's two switches on every fleet tick (reconcileGrants, lib/fleet-relay.ts) and are never a snapshot, because a snapshot is how a cap added in code once failed to reach any existing grant and the far side answered 401 for ever while the job reported green. Every grant shows when it was last presented (written at most once per five minutes) and can be revoked on the card.

The mesh across clusters. Every cluster claims a /24 out of 10.43.0.0/16, derived from its own cluster id (deriveSegment, lib/fleet.ts); segment 0 is never derived, because every cluster built before segments existed holds 10.43.0.x and must keep it. Two clusters that derive the same segment get the same verdict on both sides with nothing exchanged: the lower id keeps it, the other is told which segment to move to. Two clusters presenting one cluster id (a cloned store) are told that instead, because renumbering a mesh does not fix an identity clash. Mesh identity (each node's public key, mesh address and candidate endpoints) crosses through /api/fleet/mesh once both sides have offered; offered becomes linked when each side has seen its own id in the other's linked list, and only then is a far node written into wg0, since a peer configured before the far side has configured us reads as permanently dead. The far cluster's container range is routed through the tunnel.

Joined means working together. Since 2026.8.309 a new peer starts with the mesh offered and "work together" (manage) on, in both directions, because a pair that cannot act for each other is not working together. The link becomes linked once the far side has offered too, and reach (below) is issued the tick after. What each side lets the other do is stated in a dialog, not a paragraph: open consoles, move files, start and stop, deploy, take backups, act as an admin. Detach is one deliberate action on the card: it withdraws reach in the same request (a detached cluster with root on these nodes for another ten seconds is ten seconds too many), drops the replica of that store and its remote mesh entry, and narrows every grant back to the node list. Its segment is kept aside for a return. Health is a word, not a timestamp: ok within three minutes of an answer, stale, unreachable (answered once, stopped), never (wrong address or credential). A cluster that stops answering keeps its last known nodes and servers on screen, marked, and raises "Joined cluster X is not answering", which also says that nothing here depends on it.

Reach: what a cluster hands to one it works together with

lib/fleet-reach.ts. While "work together" is on and the mesh is linked, this cluster's leader mints, for that cluster alone: an agent token (32 random bytes) its node agents accept only from the mesh; a Proxmox API token on conduit-fleet@pve with PVEVMAdmin,PVEAuditor at / and nothing else (not storage, network, users or the host); and the cluster CA (/etc/pve/pve-root-ca.pem), so the far side verifies our Proxmox before it sends a token. These land as files on every one of our nodes: /etc/conduit/agent-peers.json (token and label per peer) and /etc/conduit/reach/<cluster-id>.json (the bundle: tokens, ports, segment, CA, issued-at), mode 600, written by rename, and converged declaratively so a revoked peer's file disappears without anyone remembering to remove it.

Read by the peer over the mesh, by source segment. The far panel fetches its own bundle from any of our node agents at http://<mesh address>:8800/v1/reach, and the agent serves the bundle whose segment contains the requesting address. That address is an identity because it can only have come out of that cluster's WireGuard tunnel, which the firewall's anti-spoof rule is what makes true. So the far panel gets its credentials with our panel down, and nothing secret ever crosses the https path the two panels talk over. Measured on the live pair: a marker bundle was read from inside the other cluster's panel container over the mesh, refused from the issuing node's own loopback and mesh address, and unanswered from the internet.

What it means. Root-equivalent on those nodes through the node agent: exec, files and the store on the host, plus every guest action on that Proxmox. That is the parity the owner chose; what the design changes is revocability (one switch, one tick, no restart) and attribution (conduit-fleet@pve!fleet-<cluster> in the Proxmox task log, the peer label in the agent journal). The Secrets page says it out loud (below).

Re-minted every 30 days, withdrawn at once. A credential with no end has unbounded exposure, so the pair is minted again after 30 days, and the pair it replaced stays honoured for one day as a second :prev entry in the peers file, because the far side learns the new bundle by reading our nodes on its own tick and not by being told; the retired Proxmox token is removed by name when the day is up. A 404 from a node that answered is that cluster's decision and the far side drops what it held; no node answering is a path problem, and the last bundle stays with the reason beside it. A bundle naming a different issuer than the cluster it was read from is refused. Switching "work together" off, or detaching, removes the Proxmox token with pveum and rewrites the files without it.

Spending it. lib/sites.ts turns ?cluster=<id> on an instance action, or on anything under /api/services/<vmid>/, into a Target with the same names a route uses for the local cluster: a Proxmox client over the mesh with the token and CA, an agent client with the token, and nodeExec that is the node agent rather than ssh (no key of ours is on their nodes, by design). Which node holds a container is asked of the far Proxmox and kept for 30 s. Only an admin or an operator may act on another cluster from here; an account scoped to particular servers is refused, because its scope is defined here and cannot be checked there; a ?cluster= that is present but empty is a 400 and never "local", since acting on the local container with that number is the one confusion this mechanism exists to prevent.

The relay: one panel asking another to edit its store, as the person

lib/fleet-relay.ts, app/api/fleet/relay/route.ts, lib/fleet-relay-client.ts, lib/requester.ts. A local route reads ?estate=<cluster id> and forwards instead of acting: the method, the path, the body byte for byte, and the id of the account that pressed the button, to that cluster's POST /api/fleet/relay with the grant it gave us. Its answer comes back untouched, status and stream included. A request that arrived through a relay never forwards again.

The serving side decides in this order:

1. The grant must carry the relay cap and role: "admin", both written by the same switch. An unknown credential is 401 ("re-join with a fresh invite"); a known one without the role is 403 ("turn on Manage together here, under Nodes").
2. The path in canonical form: no percent-encoding in a path segment, no .., no //, no absolute URL smuggled in, and query values are refused when they carry a quote, a backslash, $(, ;, NUL, a line break or .., judged after decoding on exactly the string the far handler reads. The first version checked the spelling and handed the string to a normalising client, so /api/tasks/%2e%2e/users passed as a task path and ran as /api/users.
3. The hop counter: the asking side stamps 1, the serving side stamps 2 on its loopback request, and anything arriving already stamped is refused with 409. A relay does not relay.
4. The deny list, which wins over the allow list: /api/fleet, /api/auth, /api/sso, sessions, logout-all, TOTP, console sessions and email verification under /api/account, /api/system-credentials, /api/secrets, /api/offsite, /api/s3, /api/update, /api/hub, /api/setup, /api/welcome, /api/op-audit, /api/store-history, /api/internal, /api/ctagent, /api/db, /api/domains, /api/wireguard, /api/network, /api/blueprints, /api/conduit/state, and the connector's own channel route by route (heartbeat, register, report, event and the rest), because a relayed report would be written with this panel's machine token as if a server of this cluster had sent it.
5. The explicit list: 79 method-and-path entries, no prefixes. Creating, editing and deleting services, groups and subgroups; a service's motd, env, update, resync, portals, prewarm, replay, canary, sharding, shards, time machine and reachability; the reads only the owner's heartbeat registry can answer (players, metrics, console history and completion, config, logs, proxy, regions, stats, tps); the fleet-page reads and actions (worlds, log search, agents, versions, the connector registry, move targets, a queued player action, balances); the LuckPerms editor (groups, tracks, users, permissions); and accounts, roles and your own password, tokens and ssh keys. Power, console, exec and files are not on it: those go to the nodes. import-bundle and bundle under blueprints must never be, because both splice a body into a shell. fleet-relay-shell.test.ts walks the list and refuses any listed handler that can reach a shell; relay-asking-side.test.ts refuses any listed route that does not forward on the asking side, which is how two registry routes were once answered from local records for a request that named another cluster.
6. The person. The serving panel resolves the sent id over its own accounts and every replica's (resolveActor, lib/identity.ts): an id it has not seen yet is 409 "identity replicates within a minute", a disabled account is 403, no id at all names the upgrade. Then it re-checks the capability the route needs against ITS OWN manifest and roles (actorMay), whatever the asking panel's manifest said.

Then it executes locally over loopback with its own machine token and a signed hop (x-conduit-relay-hop: 2, HMAC over hop, path, timestamp and the person's id, five minutes of validity), so the local route's requester() answers with that person and not with "machine": the route's own gate, scope check, audit and error shape all apply as if the person had used a browser here, and a hop header somebody added by hand is worth nothing. Bodies are capped at 1 MiB before they are buffered. Streams (the console) follow the far browser; anything else has 120 s. Both sides audit: the asking side records "sent to edge-02", the serving side records the action under the person's name with "relayed from edge-01", and relayed reads are recorded once per person and path per five minutes, because a relayed file download is a read. A 5xx is never sent back over the internet (Cloudflare replaces its body); a failure to run is a 409 with the reason.

Identity: one account list

lib/identity.ts, lib/fleet-partition.ts. An account is written only by the cluster it was created on; every other cluster holds the projection, built field by field so a field added to the raw record later must be named to cross. What crosses: id, name, email, role and role id, extras and scope, the scrypt password hash (s1.<salt>.<hash>, one-way, the owner's decision), each API token's sha256 with its lock (read-only, expiry, scopes), public ssh keys, disabled, the session epoch, the SSO link (provider and subject), and the fact that TOTP is on. What never crosses: sessions, passkeys, TOTP secrets, pending email and reset codes, device trust (deviceAuth), profiles. fleet-partition.test.ts plants a user with every forbidden field and checks each one is stripped.

Signing in on any joined panel works with the same password or API token, and the same rights. A joined panel refuses password-only sign-in for an account whose home cluster protects it with TOTP, because it holds no secret to verify a code against; such an account signs in at home. A foreign account's sessions on this panel live in db.foreignSessions with the same row shape as local ones, so /account lists and revokes them with the code it already has. The session epoch is the owner's: "sign out everywhere" bumps it at home and every panel refuses the older cookies after the next replication tick. Name clashes are two accounts, never one: both live clusters were installed with an admin called admin, with different ids and passwords. Signing in prefers this cluster's, /users shows both with their cluster, and an alert fleet.identity:<name> is raised on the tick a replica first brings the name in and resolved from what is open on the tick it stops (2026.8.318: judged from the open alerts, because a deletion made here ended the collision before the sweep and the alert stood for ever on the cluster that removed its own copy). Edits go to the owning cluster: a row on /users carries its cluster, and an edit, a token or a delete for an account kept elsewhere goes through the relay as you (withEstate), which also covers your own password, tokens and ssh keys on /account when your account lives on the other cluster.

One service list

lib/fleet-partition.ts, lib/fleet-view.ts, lib/cluster-params.ts, lib/sites.ts. What replicates is a projection of the store: groups, tasks (their intent fields only; the fields the owner's reconcile writes every tick, such as node, allowedNodes, installedBuild, canary, stay home so an owner's physical writes never churn replication), labels, blueprints, schedules, automation rules, identity, front-door records and tombstones. What never replicates: users and roles raw, fleet, wireguard, network, managedNetwork, firewall, plugin and pack libraries, databases, and every secret. A structural test asserts every top-level store key is in exactly one of the three sets, so a new key cannot be added without somebody deciding whether it crosses. System groups and the four conduit-* system tasks never cross (each cluster needs its own databases), by the system flag and never by id; dev workspaces never cross.

How it replicates. On the fleet tick, after the sweep, for every linked peer: its node agent over the mesh first (that path survives the far panel dying, and every node in that cluster serves it), its panel second; the agent's raw keys are run through the same projection the panel would send, so a system task never arrives twice. An ETag of logicalRev:sig makes an unchanged partition a 304. A snapshot is accepted only when every record is owned by the cluster that sent it (a replica of a replica is refused), when its schema is not newer than ours, and never when N records became 0 with no tombstones to explain it, which is a glitch and not a deletion. A fetch that fails keeps the previous replica, marked with when and by which path it last arrived.

The join is a verdict. For each id present on both sides, equal logical content (with createdAt stripped, since a cloned store carries identical timestamps) is one record, owned by the lower store id, decided once. Different content raises fleet.join:<kind>:<id> with the fields that differ, and a person settles it under Nodes: keep this cluster's, keep the other's, or rename one. Nothing is merged on its own. A delete is a tombstone that travels with the snapshot, kept for 30 days and never shorter than the longest peer silence, and a live tombstone refuses the id for re-use because containers match on the t-<id> tag alone. Orphan GC destroys a container only on a tombstone; absence without one is an alert and nothing is destroyed, because with a twenty-second lag absence-then-destroy would wipe a persistent world over a hiccup on another provider.

Instances are never replicated. They are discovered from Proxmox tags on whichever cluster runs them, read over the mesh with that cluster's reach, and keyed by cluster and vmid, because a VMID is only a name inside its own cluster (both live clusters run a 204 and a 205). So the tree on /groups is the same tree on either panel, each row saying which cluster it lives on, and a group with the same id on both clusters is one group. Consoles, files, power, uploads and the SFTP gateway reach a joined cluster's containers over the mesh with ?cluster=; a container's page is /services/<vmid>?cluster=<id>, since /services/205 alone would show the local 205. A cluster that cannot be reached still contributes the instances it last reported, marked unreachable, because a list that shrinks as things break reads as healthier exactly when it should not. A deploy can be sent to a joined cluster (POST /api/tasks is on the relay list): the service is created in that cluster's store and its own reconcile places it on its nodes. Migration between clusters is refused on both sides: it runs on the cluster that owns the service. The fleet pages (worlds, players, agents, log search, versions, permissions) ask every reachable cluster through the relay and merge the answers with each far row tagged; a cluster that did not answer is named above the list with the sentence it failed on, and the log search counts what it could not search per cluster (lib/fleet-pages.ts).

The panel coverage card

lib/replica-coverage.ts, 2026.8.309. The card on /nodes counts every panel that can run this cluster's servers: this cluster's replicas, plus each joined cluster's panel that holds reach, each counted only while it answers (a replica by a curl of /api/health from inside its container, a joined panel by the fleet poll's own stale window). On the single-node live clusters it reads "2 panels working together", with two labelled lines because they are two questions: Panel: the other cluster runs power, console, files and worlds over the mesh while this panel is down; settings wait for this panel, or for an adopt. Node: one machine here, so its containers go down with it; a joined cluster cannot take them over until Relocate lands. The replica floor (two answering, or one on a single-node cluster) is still judged on replicas alone, and losing a node never moves the verdict in the good direction: a node that is down is counted in the target, named in the sentence, and never listed as a node missing a panel.

The secrets register

lib/secrets-registry.ts, /secrets, 2026.8.314. Every credential the cluster holds, by its full store path (fleet.reach[].agentToken, network.domains.cfToken, frontDoorCerts[].keyPem, CONDUIT_AGENT_TOKEN (env, not the store)), with what it protects, its blast radius written for somebody deciding how alarmed to be, its rotation class (self-service: the panel rotates it; coordinated: consumers move together, needs a plan; external: lives in somebody else's console, with a runbook and a button to record that you replaced it), the first eight hex of its sha256 as a fingerprint (enough to see that a rotation happened and to compare replicas), and the date a replacement was last recorded. 39 entries at 2026.8.318, worst first; three join them in 2026.8.319 (the Let's Encrypt account key, the certificate's key, and the node certificates' keys). Nothing on the page is a value, and /api/secrets returns none: copying a password is the credentials vault's job, behind the sudo re-auth.

The register is checked against the store, not maintained beside it. secrets-registry.test.ts walks the DB type through every type it reaches into and builds the full path of every field whose name looks like a credential; each path must be declared here or exempted with a stated reason. Keyed by full path since 2026.8.314, because a name is not an address: fleet.peers[].token (admin on another install) had been absorbed by the exemption written for the single-use device-auth token, and the coverage test reported it covered.

The disclosure. When "work together" is on, the page says so: Work together is on: edge-02 can read all of this. Each such panel holds a reach token for this cluster's nodes; through the node agent that token runs commands on the host, so it reaches the store and every value listed, the Cloudflare token and the origin key included. That is what working together means, and it is mutual. Turning it off, or detaching, on the Nodes page withdraws the reach at once; the pair is re-minted every 30 days on its own. The header carries the posture chip: machine auth opted out appears only when CONDUIT_ENFORCE_MACHINE_AUTH turns enforcement off.

Domains and TLS

app/domains/page.tsx, lib/hostnames.ts, lib/domains-dns-sync.ts, lib/domains-sync.ts, 2026.8.307 and .316. One hostname list, and every row has a kind: panel (the dashboard through the TLS front; may be marked failover), cluster address (this cluster's own https address for the clusters joined to it; one per cluster; never moves), ssh gateway (DNS only, raw SSH cannot cross the Cloudflare proxy), packs (pack downloads, served by the panel, proxied), custom (any ip:port behind the front), and, from 2026.8.319, node (a DNS-only name for one Proxmox node, its TLS door, below). Three settings used to hold hostnames on three pages with three mode switches and three ways of writing a DNS record; they are one table now, and the old gateway and packs settings are moved into rows once, by a migration on the leader's first tick. Consumers ask hostnameFor(kind); a kind with no dedicated row falls back to the activated primary, which is what "mirror" used to mean, and the cluster kind answers only once its row is proven.

The live pair, on the zone example.com, with one Cloudflare token (DNS edit) that both clusters share with each other:

A records follow the address. A record used to be written once, when the hostname was added, with whatever address the route had at that moment. lib/domains-dns-sync.ts reconciles instead: on the leader tick every hostname's A record is compared with the panel's public address (an external echo service refreshed every five minutes, CONDUIT_PUBLIC_IP, or an override typed on the page), a signature over the address, the hostname list and the held set makes an unchanged tick free, and a write touches only the records Cloudflare has wrong. An ssh row is always DNS-only whatever its flag says. What it will not do: write an empty address (detection can fail, and an A record with no content is an outage, not a fallback), retry a refused write every ten seconds (five minutes of backoff), or touch a hostname a joined cluster is currently serving for us.

Certificates at 2026.8.318. One Cloudflare Origin CA certificate covers every hostname on the list, issued with the Origin CA key or a token carrying Zone → SSL and Certificates → Edit. The leader installs it on every panel CT as nginx (/etc/conduit/tls/cert.pem, one vhost per HTTP hostname, the local panel first and its siblings as backup, per-CT applied record so one dead node never re-rolls the others), and renews it inside the last 30 days; when it cannot for lack of credentials it raises an alert, since the expiry date on a page nobody reads daily is how a certificate expiring should not be discovered. An Origin CA certificate is trusted by the Cloudflare edge alone, which is what SSL mode Full (strict) needs and nothing else.

Certificates from 2026.8.319: Let's Encrypt. lib/acme.ts (on acme-client). With a DNS-edit token and no Origin CA key the panel issues a publicly trusted certificate through the DNS-01 challenge: it registers an ACME account on first use and keeps the account key in the store (network.domains.acme.accountKeyPem), writes a TXT record under _acme-challenge.<hostname> (TTL 120) into the zone for each name, lets Let's Encrypt read it, removes it, and stores one certificate for every hostname on the list (network.domains.leCert) with a fresh key pair on every issuance. The same tick that renews the origin certificate renews this one 30 days before expiry. When both certificates exist the Let's Encrypt one is served, because it also works for a DNS-only hostname and for a client that reaches the origin directly; the origin one stays as the fallback for a zone whose owner gave the Origin CA key and nothing else. The card names its issuer; POST /api/domains {issueCert: true} picks Let's Encrypt when the zone token is present and Origin CA otherwise, and kind forces either. No port has to be open for the challenge, which is why the same path can issue a certificate for a node whose only reachable service is the agent (below). Two register entries come with it: the account key (coordinated: remove acme under network.domains and the next issuance registers afresh) and the certificate key (self-service: Renew).

The cluster address moves joins to https (2026.8.316). lib/peer-url-upgrade.ts. Once the cluster-address row is proven, /api/fleet/peer advertises https://<name>, a join started from this side sends it as our address, and every cluster already joined moves from the address it was joined at (http://<ip>:3001, the "credentials in the clear" row) to the name, after the name passed the same vetting an invite address gets and answered the same call. A peer can never move us down to http by advertising one, an address we already left is not offered back, and a name that did not answer is retried after ten minutes and shown on the row meanwhile. On the pair, edge-02 reaches edge-01 at https://play.example.com and edge-01 reaches edge-02 at https://play.example.com. A cluster address never fails over; that is what a panel hostname marked failover is for.

Checks. Every row has a Check with a visible running state: a panel or custom row is proven by the admin's browser fetching the challenge over the internet (DNS, the :443 forward, the certificate and the Host header all have to work), an ssh or packs row server-side. Activation as the primary still hard-gates on a fresh proof.

The shared front door

lib/front-door.ts (the decision, pure), lib/front-door-sync.ts (the job), 2026.8.315 and .318. One panel hostname, served by the cluster that owns it, taken over by a joined cluster when the owner falls silent, handed back when it answers again. On the pair: play.example.com is edge-01's, failover is on, and edge-02 stands in.

Owner side. Each panel hostname gets a Failover switch, and the zone card a switch Share these credentials with joined clusters. Sharing puts the zone's API token and Origin CA key into the reach bundle every cluster this one works together with reads from our nodes (see Reach); switching it off withdraws them on the next tick. The failover hostnames are published in the owner's replicated partition (frontDoor.hostnames) with the address the owner answers at, the addresses its own node can probe its panel at (http://<replica ip>:3001/api/fleet/peer), and the stores it shares with, which are the only ones that may claim.

Stand-in side. With the credentials it issues an origin certificate for the owner's hostnames ahead of time (frontDoorCerts, one per owner, renewed 30 days before expiry, files named by a hash of the owner's store id and never the id) and installs a server block on its own TLS front, so a takeover moves only a DNS record. It probes the owner's panel every tick on two paths: from itself, the peer route with its credential on the owner's public address (any composed answer counts, a 401 included; only no answer or a gateway's 5xx is silence), and when that fails, the owner's own node, over the mesh with the reach it issued, curling the owner's panel at its LAN addresses. The second path exists because the public address does not hairpin from the node, so the first version's second probe always failed and turned any blip into a takeover (found live, 2026.8.318). After three minutes of silence on both it points the A record at itself (the record has a one-minute TTL at Cloudflare) and publishes a claim (frontDoor.claims, owned by the claimant, replicated like any record); after two minutes of the owner answering it points the record back at the owner's published address and withdraws the claim. A claim is refreshed every minute; one not refreshed for ten minutes was made by a claimant that is itself gone, and the owner takes its door back, once per stale claim. Two claimants settle on the lower store id. Every reason to hold is a sentence ("owner silent for 40s, moves at 180s", "edge-02 is first in line").

What a person sees. Browsers follow the record within about a minute. The stand-in asks them to sign in again: sessions are per cluster, and their account is there because identity replicates. From there they drive the owner's servers over the mesh; settings that live in the owner's store wait for the owner's panel. Both sides raise an alert, "play.example.com is being served by edge-02" on the owner and "play.example.com now points at this cluster" on the stand-in, resolved on hand-back.

Limits. A partition between the two clouds that also cuts the mesh reads as an outage: the stand-in takes the door, and hands it back two minutes after the partition heals. The owner's DNS reconcile leaves a hostname alone while a fresh claim on it stands. The stand-in certificate is an Origin CA certificate, so the taken-over hostname is served through the Cloudflare proxy only.

Node hostnames and node TLS (shipping in 2026.8.319)

Not in 2026.8.318, the baseline of this section: there the node agent is 0.3.3, listens on 8800, and is reached in plain HTTP inside the WireGuard tunnel only. It ships in 2026.8.319 as lib/node-tls.ts and agent 0.3.4, and this is what that code does.

A hostname row of kind node names one Proxmox node, such as play.example.com. Its A record points at that node's own public address, read from the node's interfaces, never at the cluster's address and never proxied, because the agent's port is not one Cloudflare carries; a node whose address is not known gets no record. On the leader tick nodeTlsTick issues a Let's Encrypt certificate for the name through the same DNS-01 path as the panel's (it needs the zone token, and says so once when it is missing), keeps it under network.domains.nodeCerts[<node>], pushes cert.pem and key.pem to /etc/conduit/agent-tls on the node (written by rename, key mode 600), records an applied signature per node so a tick is idempotent, and renews 30 days before expiry. A node that cannot be reached keeps its old files and is retried next tick. The agent watches the two files every minute and serves the same handler on 8443 (CONDUIT_AGENT_TLS_PORT), reloading the certificate in place when the files change; the firewall pass opens 8443 on the node only to a computed source list (conduit:protect-agent-tls), never to the internet at large. The door is published to joined clusters through the front-door projection (frontDoor.nodes: node, hostname, port, expiry), so they learn the name without being told.

What it is for: the mesh stays the first path and needs no certificate, but a tunnel that is down takes consoles, files and power with it, and a panel that can still reach the far node's public address had no way to trust what answers there. Now the agent client tries the mesh address first and, only when that fails to connect (an HTTP error is the agent speaking, not the path failing), sends the same request to https://<node hostname>:8443, verified against the system trust store. The reach token authenticates on both paths; TLS is only what makes the second one safe. Consoles stay on the mesh, since a websocket is not retried this way.

What is not there

• No shared backups, plugins or voice across clusters yet. Each cluster's PBS jobs, plugin library and voice halls are its own, and the plugin, pack and database libraries do not replicate.
• No adopt. A joined cluster cannot take over a dead cluster's servers: the containers are on the dead machine, and settings in that store wait for its panel. Reach is a second panel for the same nodes, not a second set of nodes.
• A partition that cuts the mesh reads as an outage, on the front door and on every "unreachable" row. Two vantage points are what there is.
• Two clusters are never one store. No shared quorum, no shared VMID space, no corosync over the internet.
• Only admins and operators act across clusters. A scoped account is refused, because its scope is defined on one cluster's records and the other cluster's containers may carry the same numbers.

4.23b First run — /welcome#

A brand-new cluster spends its first minutes building four containers nobody asked for: a Valkey, a Postgres, a MariaDB and a MinIO, because Conduit needs them to be Conduit. That used to happen in silence — the services appeared in the list marked "starting" and either became ready or, if anything was wrong, said the same word forever. The owner of the first outside install watched exactly that, and nothing on the screen could tell a slow install from a dead one.

Creating the first admin account now lands on /welcome (a normal sign-in never does), which shows the bring-up while it happens: one row per service moving through waiting → creating → installing → ready, a bar that fills, and the reconcile's own narration underneath — the same lines it already writes, filtered to the ones about this. A service that has wanted a container for several minutes and has none is reported failed, with the reason when there is one, because silence is not progress. lib/first-boot.ts computes it from tasks, discovered instances and events — no new state — and is pure enough to test without a cluster (first-boot.test.ts).

The wizard then asks the two things a new cluster has to answer. The hostname it will live on — recorded as an intention in network.onboarding, deliberately not written into domains, because a hostname only becomes real once /domains has pointed DNS at the panel, proved the path end to end and issued a certificate; claiming it earlier would have the panel assert a domain it cannot serve. And what this cluster is: standalone (what you have — adding a node means re-running the idempotent installer), or joining another Conduit cluster, which is described honestly as not built rather than shown as a button that does nothing. Two panels sharing one store across the internet needs a private link between nodes and a decision about which cluster is authoritative; it moves the trust boundary, so it is being designed first.

Finishing or skipping is recorded with who and when — skipped is stored, not hidden, so a later reader can tell "set up deliberately" from "clicked past it". That flag is the only thing that stops the wizard appearing.

4.24 The update pipeline — the panel updates itself#

Conduit knows which build it runs, because install/modules/bundle.sh stamps version.json into the bundle root and the unit's WorkingDirectory is that root — so every replica answers for itself, which is also how a rollout proves one came back on the new build. lib/version.ts reads it; lib/hub.ts compares it against the configured hub's feed (updateStatus is strictly-greater, so a release pulled from the hub can never invite a cluster to update downwards). The sidebar carries the version as a thin row that turns brand-coloured when something newer exists, and opens the update dialog.

Noticing is push, not poll. The leader re-reads the feed on its own five-second clock — not inside the reconcile tick, which takes the better part of a minute and would put a floor under how fast anything could be seen — and bump("hub") on the live bus reaches every open panel in under a second. That watch is fast (4s) only while a browser actually holds a live stream open and drops to ten minutes when none does, so a hub serving many clusters is not asked constantly by all of them forever. Every step of a rollout bumps the same topic, so replicas flip in front of you. The first look after a restart announces too: staying quiet there meant a release published while the panel was down — which includes every panel restart caused by installing an update — was absorbed silently.

The dialog shows what is being installed (version, publish date, size, hash), the release notes, the replicas it will touch, the seven checks, and a log fed by the same narration the leader persists. The result is shown once, to the page that watched the rollout finish, and closing the dialog acknowledges it — a finished run is news for exactly as long as somebody is waiting on it, not for a window of minutes afterwards. It also yields immediately if the hub offers something newer still. The log folds itself open when a run starts and away when it ends. Whether the run view shows is decided by the run's own phase, not by the hub agreeing — the hub echo lags the finish by a poll cycle, and gating on it made the dialog fall back to the "about to update" screen for those seconds before jumping to the result. Success ends in a visible countdown, not a button: the replicas just replaced the very bundle this page was served from, so a reload is not optional — the dialog says "reloading in 5s" and does it, clicking the countdown skips the wait, closing the dialog cancels it. A failed run keeps the full screen too, and dismissing the dialog animates out on its own slower clock instead of the stock 100ms blink.

Preflight (lib/self-update.ts, seven checks, all must pass before the button lives): this panel holds the VIP · the hub offers a genuinely newer release with an artifact hash · the store reads, writes (a real round-trip) and has just been snapshotted with update.sh's keep-20 semantics · every replica is reachable, its units active, /opt has twice the bundle free, and /var/lib/conduit is mounted and writable · the artifact downloads to the shared volume and matches size + sha256 · every node hashes its own copy (gluster replicating a file is a claim worth checking) · no other run is in flight. Results stream through global.__conduitUpdate — the leader's memory, so a non-leader answers "the checks run on the VIP holder".

The run is staged, never in place: the bundle is pct pushed into each container, extracted to /opt/conduit-panel.new, checked for a server.js and the version it claims, then two renames swap it — the displaced tree stays as .old. install/update.sh unpacks over the live directory, which is fine with a human watching and not fine unattended. Siblings go first and the VIP holder last; a sibling that does not answer on the new version is rolled back from .old and the run stops rather than leaving a mixed-broken fleet. The leader's own restart is handed to systemd-run detached, because a plain restart would kill the ssh client carrying the command, and the run is finished by the process that boots afterwards (finalizeUpdateRun, called before the reconcile loop starts). Progress lives in db.updateRun because the UI polls through the VIP and any replica may answer; a heartbeat older than ten minutes means the leader died, and the row is failed rather than left as a lock nobody can clear. Alerts + activity entries on both outcomes; the failure carries the exact per-container rollback one-liner.

The VIP follows a panel that answers, not a container that booted (install/lib/keepalived.sh, agent/conduit-panel-health.sh). A keepalived track_script curls /api/health on loopback every two seconds and takes 60 off the replica's priority after three consecutive failures, so a leader that restarts into a broken build loses the address — measured live at eight seconds from the panel going quiet to a sibling answering, where before it was however long it took a human to notice. Coming back is deliberately slower than going away: twenty unbroken seconds of health before a recovered leader preempts, which a crash-looping panel can never string together, and which is what stops the address oscillating. A weight rather than a bare track_script on purpose — without one a failing check drives the instance into FAULT and it stops advertising entirely, so a fault common to all three would leave nobody holding the VIP, which is strictly worse than not checking. The check reads nothing shared, because the three replicas share a store and a check that touched it would fail on all of them at once and pass the VIP round a circle without repairing anything; /api/health therefore touches no state at all, and 401 (auth wall answering) and 404 (an older bundle without the route) both count as healthy so the guard fails safe. This also cost finalizeUpdateRun an assumption it used to be able to make — the process that finishes a run is no longer necessarily the replica that restarted, so the run records which replica took the self step and the finalizer compares that against its own identity before believing its own version number. install/doctor.sh checks that the guard is still armed, unpinned, currently passing, and not flapping, since every way it can fail leaves the cluster quietly back where it started.

Disk pressure, before it bites (lib/disk-watch.ts, agent /v1/disk/usage). A full disk is the most quietly destructive thing that can happen to a service: it does not crash cleanly. A Paper server keeps running and stops saving chunks; a database refuses writes; a log rotates into nothing. The backup server demonstrated the shape of it at the worst scale on 2026-08-03 — its datastore filled, three nights of backups failed, and because the same filesystem also held the task logs, prune and garbage collection could not run either. A store that is completely full cannot clean itself up. This watches for that one level down, across the fleet.

It costs almost nothing: the container agents already report disk usage in every heartbeat, so the check reads memory the panel is already holding — no polling, no pct, no SSH. The expensive question is asked only of a container already over the line, and only then: du walks the tree, so "what is eating the space" is a diagnosis rather than a metric to sample. That distinction is the whole design — warn at 85%, error at 95%, keyed per container per day so a disk sitting at 88% for a week says so once a day rather than every ten seconds.

The shared filesystem, watched while it runs (lib/storage-health.ts, Storage page). Every server's working directory lives on a GlusterFS replica-3 volume at /var/lib/conduit. doctor.sh has always checked it — but doctor is a thing a human runs, and in August a volume quietly serving from two copies instead of three, with three files returning Input/output error on every node, went unnoticed for ten days. A five-minute leader tick now reads bricks, peers, heal state, split-brain and each node's mount, and the Storage page opens with the answer to the only question worth asking: healthy, degraded but still serving, or broken.

Pending heals are normal — heals that never finish are the signal. There are entries pending during any ordinary write, so alerting on their presence would be noise on a live cluster. The watch remembers when it first saw each entry and reports only what is still pending thirty minutes later, three full self-heal passes. The threshold is not arbitrary: the August fault was a GFID mismatch, where the same path carries different GFIDs on two bricks and the client refuses to choose. Its one observable symptom is an entry that stays pending — heal info split-brain reported zero the entire time, so the obvious check would have missed it completely. A backlog while a brick is offline is a different thing entirely, and the watch says so rather than repeating the GFID story: with a copy missing every write leaves an entry that cannot heal until the brick returns, so the finding names the offline node, says the count will keep climbing, and drops the forensics. Both look the same through heal info; only the brick and peer state tells them apart. Every finding carries the command to run next, and the whole thing is read-only: deciding which copy of a file wins is not a decision to automate. Parsing is verified against real --xml output captured from a deliberately broken throwaway volume, including the offline brick that reports its entry count as - rather than a number. See docs/architecture.md → Shared storage.

And the alert says where the space went. "Disk 91% full" sends someone looking; "91% full, /opt/mc/logs is 12 GB" does not. An alert that only states a number has moved the work rather than done it. Verified against its own thresholds and content, including that a healthy disk stays silent.

Fleet-wide log search (/logs, GET /api/logs/search, agent /v1/logs/search). "Which server said this, and when" was answerable only by opening twenty-one consoles and reading. Now it is one box. The panel fans out to every running container's agent at once and each greps its own logs in place — 351 matching lines across 12 servers came back in 207ms, because a kilobyte of answers moves instead of gigabytes of log.

What it searches is the part that had to be got right: not the journal, for a game server. mc.service starts tmux, so journald holds Starting mc.service and none of the game's output — a journal-only search would answer "nothing found" to every question worth asking, which is the same trap as the hardcoded ?unit=mc that once broke the log tail on every database container. So it reads Paper's own latest.log and the console mirror first, the unit's journal second (databases and web servers really do use it), and every hit says which it came from. Case follows the query the way editors do it — an all-lowercase needle matches anything, capitals mean them — and the regex toggle is ERE, because plain grep is basic regex where (a|b) is a literal string and a toggle that silently means a different dialect is worse than no toggle at all.

Above the results, not below: the containers it could not search, each named with why. A search that quietly skips three servers and reports "no results" is indistinguishable from one that found nothing, and an operator mid-incident will believe it. Same failure this codebase already shipped once, in a backup page that stayed green through three nights of failed jobs.

Copies that leave the building (lib/offsite.ts, lib/offsite-crypto.ts, Settings → Offsite copies, /api/offsite). Every backup plane Conduit had lands in the same building it protects: PBS is one box on the rack, the object store is a container on the same three nodes. That answers "a container broke" and not "the site did". This ships a daily encrypted copy of the cluster store — every service, group, route, domain, schedule and credential — to any S3-compatible bucket, and nothing else, deliberately: it is a few hundred kilobytes, it is the only artefact that cannot be rebuilt from something else, and it is what turns a rack of blank containers back into this network. Worlds are large and already have two planes of their own; vzdump archives are larger still and live in PBS's chunk format, which cannot be shipped without restoring it first. Sending the small irreplaceable thing daily beats sending the big reproducible thing never.

The provider is not trusted. Objects are encrypted client-side (AES-256-GCM) with a key held only in this cluster's store, so a leaked bucket, a seized account or a curious provider gets nothing — the key is shown once, at the moment it is minted, with the trade said plainly: keep it somewhere that is not the cluster it protects, because it is also what makes the copies readable. Ciphertext is framed in 1 MiB chunks, each authenticating its own ordinal, ending in a terminator that authenticates the chunk count. That is aimed squarely at truncation — the ordinary failure of an interrupted upload — so a copy cut short refuses to decrypt instead of restoring as a plausible prefix of itself. A backup that half-restores is worse than one that refuses to, because only one of those gets noticed. Verified against wrong keys, mid-chunk and on-boundary truncation, flipped bits, appended bytes, reordered chunks and an unencrypted file wandering into the bucket.

And it is read back after every write: offsiteVerify() downloads the newest object, decrypts it and insists it parses into a store with tasks in it — a weaker check would pass well-formed JSON of the wrong shape. Same principle as the restore drills, pointed at the far end. Live-verified end to end against a stand-in bucket, including reading the stored bytes off the destination's disk to confirm none of the store's contents survive in the clear.

The plane watches its own outcome, because the generic job wrapper cannot: a failed attempt stamps lastRun and the daily gate then rests, so the three-strike counter meets a clean no-op tick in between and resets — the wrapper would watch this job fail forever and stay green (the live cluster demonstrated the shape: a copy quietly eight days old behind a green job row). So the tick judges the age of the newest good copy every pass and raises offsite.stale when replication is enabled but no copy has landed in two days, retries hourly after a failure instead of resting a day on it, and posts a resolved note when copies land again. The Backups page carries an Offsite copy band — state, last copy age and size, last read-back, copy count, destination — with disabled shown as a stated choice ("no copy of the store leaves the cluster while this is off"), not an alarm.

Logical database backups (lib/db-dumps.ts, nightly on the leader tick, proven by the Verify sweep). vzdump images the database containers while MariaDB and Postgres are running — crash-consistent at best, and "usually recovers on restore" is not a doctrine this panel accepts anywhere else. So every managed database also gets a real dump each night: per-database mariadb-dump --single-transaction / pg_dump taken inside the engine's own container over the unix socket (no database password ever travels), compressed and streamed to object storage under dbdumps/<task>/<db>/, fourteen dailies kept. Per-database on purpose rather than --all-databases: a single-database dump carries no CREATE DATABASE and no roles, which is exactly what lets the Verify drill restore it into a scratch database beside the live one — same engine, same container — count tables and rows, and drop it. Roles and grants lose nothing by the choice; the panel derives and re-asserts them continuously, so the data is the only part that cannot be rebuilt, and the data is what gets dumped. Dump failures alert per database on the outcome, a failed night retries hourly instead of resting until tomorrow, and each database appears as a row in Verify, auto-proven weekly like every other S3-backed artifact.

Shared volume capture (lib/services-capture.ts, nightly on the leader tick). The shared gluster volume is a bind mount inside every game container, and vzdump excludes bind mounts — so the tree holding every relocated plugin directory (Skript scripts and variables, Citizens saves, plugin configs), the language catalogs and the store history had no backup layer at all until a crash proved it by truncating variables.csv on four servers. Now a node tars the volume nightly — minus what is reproducible (assets/hytale, staged update bundles, every jar) — and streams it through a presigned PUT to object storage under shared-capture/, a manifest beside each tarball. The upload is verified against the bucket, an implausibly small capture counts as failure rather than success (an empty fuse mount must not read as a good backup), failures alert and retry hourly, five nightly pairs are kept, and the newest capture shows on /backups under Object storage with amber when it goes stale. And because a backup that has never been restored is a hope, the Verify sweep drills it weekly like every other artifact: fetch the manifest, pick the canary the volume would be poorest without (the language catalogs, else the largest Skript source), stream it back out of the tarball on a node and compare byte-for-byte — the real restore path end to end, without touching the live volume.

Config history (lib/store-history.ts, /store-history, nav under Data). The store had three disconnected shards of history nobody could browse — a monotonic rev on every write, predeploy .bak files, daily offsite copies — and none of them answered the operator's questions: what changed, when, by whom, and can I have the old one back. The leader now samples the store each tick and appends a snapshot to the shared volume whenever the tracked view moved: the store minus its operational keys (job verdicts, breadcrumbs, verify results, sweep claims) and minus nested fields that churn on their own clock (offsite lastRun, the packs hostname's verifiedAt), so the changelog reads like a changelog rather than a write log. Each entry is its own file beside a small index — one file per change stays under the fs read cap forever. The page lists entries newest-first with the human diff each change made (diffConfig, secrets masked) and per-entry attribution from the op-audit trail; reverting merges the snapshot's tracked keys back through mutate() — CAS-protected and op-audited — with the consequence stated rather than softened: reconcile will enforce the old configuration, creating and destroying to match. Live-proven the honest way round: a task change was recorded and attributed within a tick, the revert restored the task and appeared in the history as its own entry.

Drift report (lib/drift.ts, /drift, nav under Monitoring). The reconcile loop re-asserts desired state forever and used to say almost nothing about it — each tick's action lines went to a console log nobody reads. That silence has a cost: the LuckPerms redis-password drift burned days precisely because self-healing hid the pattern, re-applying the fix every pass while the cause sat unfixed. Every acted line is now normalised into the KIND of action it is (digit runs and bracketed lists collapse, so the same repair on different instances counts together) and counted per day, seven days kept on the shared volume. The page ranks kinds by volume with a per-day bar strip and marks the rows active every single day in amber — steady low rates are routine re-assertion, but a repair that never stops being needed is a bug wearing a bandage, and this is the page that takes the bandage off.

Provider labels + grouping (/nodes, the small label beside each node's name). A node's hosting provider is an operator fact the cluster cannot discover about itself, so it is a label you set — click it, type "Hetzner" or "24fire", done (nodeMeta in the store, PATCH /api/nodes/<node> with {provider}). The page groups by it only once TWO providers are named: a single-provider fleet renders exactly as it always has, so the common case pays nothing. Grouped, each provider gets a header with its rollup — nodes, cores, memory, containers — which is the at-a-glance answer to "what do we run where" for a fleet that spans hosts. Purely a label today; the cross-provider overlay (nodes joining over WireGuard rather than a shared LAN) is designed separately before it is built, because grouping a view is an afternoon and stretching the cluster's trust boundary is not.

Node maintenance mode (lib/node-maintenance.ts, wrench on each /nodes card, admin). Planned downtime one level up from tasks, for the day a node needs a kernel, a disk, or a reseat. Flag the node and three things follow: every service whose instances all live there counts as planned — the same suppression path task-level planned downtime uses, so no incidents, no pages, no subscriber mail — the controller stops fighting the work (no auto-restart of instances on the node, no new placements onto it), and the node card says so in amber with the reason and who set it. Services with a surviving instance elsewhere — the HA proxy — stay monitored, because they are still expected to answer. Windows can carry an until and expire themselves with a hand-back note in the feed, or run open-ended until ended from the card.

Connector jar drift sync. The jar a container RUNS used to meet the jar a deploy SHIPPED only at provision time — the fleet quietly ran three different July builds while every deploy staged a fresh one, and the Plugins page couldn't see it because it compared instances against the fleet's newest running build, which is self-referential. A reconcile pass now compares each instance's live build (already in its heartbeat) against the canonical sidecar and pushes the jar where it is behind — push only, never a restart: restarts kick players and stay a person's decision, and the Plugins page judges pending-restart against the canonical build, so the whole fleet tells the truth the moment something newer is staged. Every staging is logged, which also lands it in the drift report.

Hub reachability watch. The release hub is one internet-facing container and updates fail soft when it dies — which meant it could be dead for a week before anyone noticed the feed had gone quiet. The hub watch now pages once after half an hour of failed checks and leaves a resolved note when it answers again; the cluster keeps running throughout, it just says so.

Secrets inventory (Settings, admin). Every credential the cluster runs on in one honest list — forwarding secret, derived and overridden service passwords, S3 and offsite keys, the offsite encryption key, Cloudflare token, GitHub token, Anthropic key, SMTP, hub cluster secret, session secret and agent token from the env, and each configured SSO provider. Values never leave the server: each row is an 8-character sha256 fingerprint — enough to see that a rotation actually happened and to compare two replicas' envs, useless for anything else — plus what it protects and how to rotate it. Ages are deliberately absent; most of these predate tracking, and an invented age is worse than none. The config changelog records store-secret changes from now on, masked.

Nothing recurring fails silently (lib/job-health.ts, card on /activity, GET /api/jobs). The leader ticks about twenty background jobs and every one of them used to be called as await someTick().catch((e) => console.error(e)). That line is load-bearing in the wrong direction: it correctly stops one broken job taking down the other nineteen, and it puts the reason in a log nobody reads. A job could throw on every tick for weeks and no surface would say so — which is exactly how three nights of failed backups passed unnoticed in August. Each job is now wrapped so its outcome is recorded, alerted after three consecutive failures (one is a blip, three is not), and resolved out loud when it recovers. The wrapped set deliberately includes the quiet plumbing as well as the obvious work: TLS renewal, world captures, scheduled restarts, pre-warm, and the outbound webhook pump — that last one being the delivery path for every other alert, so its silent failure would have disconnected the whole alerting system from Discord and ntfy while the panel went on looking healthy. Verified against its own failure path rather than just its happy one: returns null instead of throwing, stays quiet for two failures, alerts on the third with the cause attached, does not nag for the next twenty, resolves on recovery, and counts per job rather than globally. The unit is the invocation, not the work — most of these are asked every tick and return immediately unless due — and the card says so rather than letting a reader infer the stronger claim. The card itself is a verdict band (healthy count, last tick, slowest job, alert threshold) over a dense per-job grid — each cell carries its age and a duration bar against the slowest, failures widen to a full row with their reason, jobs with no run since boot show as waiting (the API serves the union of live and persisted state, so the list never shrinks after a restart), and a hairline refresh bar fills over the poll interval so a live surface and a dead poller can never look alike.

And the replica that failed gets put back. Moving the VIP made the panel honest about a broken leader; it did not make the cluster whole. So the finalizer runs the rollback itself — .old is on that container's disk and it was serving from it minutes ago. Deliberately dull: one attempt, no retries, the failed tree preserved as /opt/conduit-panel.failed, and a check of what is actually in .old first, because a second bad release in a row can leave the previous build being the one that just failed. Proven end to end by publishing a release and deleting server.js from the tree the leader had just swapped in — panel dead at 13:05:33, VIP on a sibling at 13:05:40, rolled back and answering at 13:07:35, VIP home at 13:07:56, and the run recorded phase: failed with CT 190: rolled-back rather than the "all replicas verified" it would once have claimed.

4.25 The hub — release feed and the live network of installations#

hub/server.mjs, one zero-dependency Node process, hosted wherever you like (systemd unit + README in hub/). GET /v1/feed (schema, latest version, notes, bundle url + sha256 + size) · PUT /v1/releases/<version> streams a bundle straight to disk while hashing it, bearer-authed · GET /v1/artifacts/<v>.tgz · POST /v1/ping · GET /v1/stats · GET / the public dashboard. The newest version ever published stays latest even if an older one is re-published.

hub/release.sh cuts a release from a clean tree using the same bundle_build a rollout ships, so what clusters install is byte-for-byte what install/update.sh would have deployed; notes default to the subject lines since the last released tag. Versions are calver from the repo-root VERSION.

Telemetry is on by default with the switch in Settings → Conduit Hub, and the card spells out the whole payload rather than linking to a policy: a random cluster uuid, version and commit, uptime, four counters (nodes, services, servers, players), the platform string, the software mix as kind:version → count, which Conduit features are switched on, and hardware totals (cores, memory, containers, disk). Every value is a count, a boolean or a short enum, and the software map is keyed by kind and version rather than by what anything is called — so it answers questions about Conduit without answering any about the operator. No hostnames, no addresses, no group / service / world / domain / player names. The rule for extending it is not "is this useful" but "would an operator be surprised to find it on a public web page", because that is where it goes. Country comes from the edge proxy's header when there is one; the hub never geolocates, so an address stays something the socket knows and we discard. The hub clamps every string it accepts and keeps source IPs only in an in-memory rate limiter.

Crash reporting splits in two, and the split is the feature. A breadcrumb — the forty lines a dying server last said — is exactly what its operator needs and exactly what nobody else may see, so before anything leaves the cluster it is reduced to a signature: the exception class, the first stack frame that is neither the JDK nor the server itself, and the software version. That is public, and it is what lets the hub say "NoClassDefFoundError out of net.essentialsx on Paper 1.21.11, seen by 12 clusters this week" — the shape of a bad plugin-and-build pairing without a line of anyone's log. The signature is null when the lines do not describe a crash at all, because a server stopped on purpose leaves a breadcrumb too and inventing an exception for it would poison the aggregate. The detail travels separately into the sending cluster's own private store on the hub, accepted and returned only against that cluster's secret — useful precisely when the panel holding the local copy is what died.

The public dashboard (GET / on the hub) leads with the thing a release hub is for: every published release, newest first, with its age, commit, size, a download link, the notes, and how many installations are actually running that build. That last number is the one only a hub can answer — the changelog says what shipped, adoption says what landed — and it gets its own figure beside the activity chart: what fraction of the network is on the newest release, what is behind, and what is on a build the hub never published. The version-bump commit is filtered out of every release's notes, since it appears in all of them and says only what the heading already says.

Below that, what the network looks like: active installations, servers and players, cores and memory, an active-installations sparkline, bar breakdowns of server software and feature adoption, where they are, and the week's top crash signatures ranked by how many distinct clusters hit them. It refreshes itself every thirty seconds and says in the header whether it is actually live.

It is built in the panel's design system rather than a look of its own — §2d applies to it exactly as it does to the panel. The tokens are copied from globals.css (hex, upgraded to OKLCH behind an @supports, because this page is public and the browsers without it are real), so the two surfaces cannot drift apart by being maintained separately. The KPI strip and the hardware totals are bands — one panel, hairline showing through 1px gaps — and every cell renders its sub-line even when there is nothing to say, so a row stays level whichever numbers are missing. Status is a solid mark and plain text; there are no tinted pills and no washes behind text. Inter and JetBrains Mono are vendored as latin woff2 and served from the hub itself, because a page whose footer is a privacy statement should not hand every visitor's address to a font CDN to render it.

The page lives in hub/public/ as three files rather than a template literal inside server.mjs, so a stylesheet can no longer be a syntax error in the process that serves the release feed.

4.26 The agents — one on every node, one in every container#

Two daemons, in two positions, for two different reasons.

The node agent (agent/src/index.mjs) runs on each Proxmox host, where pct lives. Entering a container from outside costs a fixed 0.40s — measured, every time, whatever the command — and the web terminal used to pay it on every submitted line, which is exactly what made typing into it feel remote. It now holds one pct exec <vmid> -- bash -s open per container and feeds commands to it: twenty sequential pct exec calls take 7.9s, the same twenty through a held-open shell take 0.405s, and nearly all of that is the single entry at the front.

That 0.40s was only half the price. pct is a Perl program that loads the whole PVE library stack, so each call is also ~130 MB of resident memory — unmeasured until a node reset itself twice in two days. The reconcile passes were spending it on questions no shell is needed for: does this jar exist, what is in this directory, what does this properties file say, once per plugin per instance. Sampled on one node, the panel was opening an ssh session a minute, each burning one to two minutes of CPU and up to 850 MB peak; under that pressure the agent's own madvise triggered a TLB shootdown that spun 22 seconds with preemption off, the kernel logged soft lockups on three CPUs, corosync dropped both links, and the guest was reset. Reads now go through the container agent's typed filesystem operations (lib/ct-inspect.ts), leaving the shell for writes and for work that genuinely is shell-shaped. Fleet-wide, distinct pct exec calls per 300s went 97 → 9, and the heaviest session's memory peak from 850 MB to ~118 MB.

A caller's script goes over base64'd (so a newline or an unbalanced quote cannot desync the control channel) and runs in a subshell with stdin, stdout and stderr redirected — a cd, an exit or a stray & then cannot leak into the channel or outlive its reply, and a forged sentinel in a command's output arrives as data rather than framing. A command that overruns its timeout retires the shell instead of trying to resync one we can no longer predict.

The container agent (agent/ctagent/conduit-ctagent.py) runs inside each LXC — the Docker-daemon / Wings position. Single-file Python 3, standard library only, no build step: every Conduit container already has python3, and a daemon that needs installing is a daemon that will be missing somewhere. A reconcile pass installs and upgrades it, re-confirming every five minutes so an agent stopped by hand is noticed rather than assumed (systemd's Restart=always covers a crash; it does not cover a deliberate stop). It idles at about 12 MB.

There is deliberately no generic exec endpoint. Every operation is a named, validated call — list a directory, read a file, tail a log, restart an allowlisted unit — so a stolen token buys those and nothing more. Paths are confined to a root allowlist after realpath, so a symlink inside the tree is not a way out of it; archive extraction drops any member that would land outside the target, and symlink and hardlink members entirely; console input travels over stdin to tmux load-buffer rather than on a command line. Each container's token is derived from the node agent's, a value containers never see, so reading one container's token says nothing about any other — verified live: 213's token against 220 is 401, and so is the panel's own machine token, which is not a container-agent credential either.

Nothing depends on it. Every caller asks ctagentReady() first and falls through to the node-agent/pct path on absence, on a stopped agent, on a version skew, or on any error at all. A fresh container, a foreign template or a crashed agent changes the speed and nothing else. What it bought, measured on CT 213:

The console reaches it through console-proxy.mjs, which gained a second upstream mode: browser WebSocket in, the agent's SSE stream out, translated into the same frames the terminal already reads. The agent speaks SSE because the standard library has no WebSocket server and does not need one — the browser never talks to it directly. The container's address comes from the panel rather than the browser's query string, because that address is where a per-container token gets sent.

Vitals, on a service's Metrics tab: CPU and memory as sparklines, then disk, network rates, process count and load, read from the container's own cgroup and /proc every two seconds — different in kind from the Proxmox RRD graphs below, which are host-side and coarse. Memory deliberately excludes reclaimable page cache: memory.current counts it, the kernel hands it back the moment anything needs it, and reporting it as used put every Java container permanently at 98% when real pressure was 79%. Under that, collapsed, the unit's systemd journal — the console shows what the server printed, this shows what systemd saw, which is the half that survives a server dying before it could say anything — and its unit is never named by the panel, because knowing whether a service is a Paper server, a Postgres or an nginx is the container's job, not the caller's.

The graph has shape before its first poll, because the heartbeats are kept: each one appends to a half-hour ring per container, which costs an array push and no requests at all. The line is that 10s series; the 2s poll drives only the numbers, since feeding one into the other would make the time axis lie.

Coverage is visible in four places, because an agent that quietly goes missing makes things slower rather than broken and would otherwise never be mentioned. install/doctor.sh names the vmids that are silent — a warning, not a failure, since the panel falls back to pct — and reports whether a configured hub actually answers from a node, which is the case a workstation check would miss. conduit doctor asks the panel the same questions through GET /api/ctagent, answered from heartbeats already in memory.

/agents is the fourth, and the one an operator reads. Every node agent and every container agent on one page: what answers, on what address, in how many milliseconds, what version it runs against what this panel ships, and what unit it is watching. A container with no agent says what that costs — 393 ms per question instead of 9 — and a container that cannot host one says why, because "no agent" and "no agent on purpose" looked identical from outside until the install pass started recording the reason it skipped. Admin only, and a live sweep behind a 20-second cache rather than a leader job: an agent answers in 9 ms and every replica can reach every container, so probing the fleet costs less than drawing the page, and the page says when it looked.

Two columns carry the weight, and the second exists because of a specific failure. CTAGENT_VERSION is typed by hand, and once it was not: a release rewrote the daemon, left the constant at its previous value, and the install pass compared 0.1.7 against 0.1.7, concluded the fleet was current, and pushed the new file to zero containers. Nothing anywhere reported a problem, because every check in that chain was comparing one unchanged string to itself. So both agents now report a hash of the file they are actually running alongside the version they claim. The version still gates fallback, since it is what says which endpoints exist; the hash is what catches right version, different bytes, and it cannot be forgotten because nobody types it. The install pass re-installs on a hash mismatch even at a matching version — precisely the case it used to wave through forever. An agent too old to report a hash, or a bundle that cannot read its own agent source, is judged on the version alone: unknown is not a mismatch, and dressing it up as one would paint every row red during the very rollout that introduces the field.

Converge and restart sit on each row, and both go through the same install path the reconcile loop uses — a second way to write those files would drift from the first the moment either changed, and the difference would surface only as a container behaving unlike its neighbours for reasons nobody could see. Converge is always forced, because the button exists for the cases where the version agrees and something is still wrong. Restart goes through pct rather than through the agent: the agent's unit allowlist deliberately excludes itself, and a daemon that is misbehaving is the one you cannot ask to restart itself. Node agents are listed and not driven — install/ owns those, and a button that could not work would be worse than the sentence saying so.

The heartbeat runs the other way: each agent tells the panel how it is every ten seconds, and says so in the same second when its unit fails. It shares an alert key with the reconcile sweep, so whichever notices first wins and the other is a no-op. It fires only on a unit that actually failed, never on one merely stopped — a deliberate systemctl stop is not a crash, and an agent cannot tell an operator's stop from a death.

And the loop believes it. Agents post to the VIP and the reconcile runs on the VIP holder, so both the ctagent install pass and the crash sweep read a container's reported version and unit state instead of shelling in to re-derive them: twenty-one containers volunteering the answer every ten seconds made asking them over SSH the most frequent remaining use of pct in the loop. A container that is not heartbeating still gets probed — which is exactly the set that might need repairing. Both halves verified live: with the panel freshly restarted (so the in-memory skip cache is empty and cannot be doing the work) six consecutive ticks touched nothing, and an agent that was stopped with its version marker deleted was probed, reinstalled and restarted unattended.