3. Platform features (orchestration)#

Everything here is steered from the panel UI or the HTTP API (full registry: dashboard/src/lib/api-registry.ts, browsable on the panel APIs page — 90+ endpoints with descriptions and a try-it runner). Highlights and what they're for:

• Groups / subgroups — org units → Proxmox pools; per-group maintenance (proxy denies join unless conduit.maintenance.bypass[.<task>]), slot limits, group-wide console broadcast. GET/POST/PATCH /api/groups…. One group is the platform's own: Conduit (system: true), created on a fresh cluster to hold the databases Conduit runs for itself. Only an admin can change anything in it — operators included, since a rename or a slot limit there is a change to the platform rather than to the network — and nobody can delete it, because deleting a group destroys every server in it. The panel marks it with a cog and says so; the id conduit is reserved so a group created under that name can never end up holding the system databases without the flag that protects them.

• Tasks (servers) — blueprint + role (the job the server does — see below) + mode (static persistent / dynamic autoscaled) + min/max/desired + node placement (pin/allow/deny) + per-task JVM args (task-level overrides group-level, panel-editable) + per-task seed (worldUrl/configUrl/plugins for fresh provisions). POST/PATCH /api/tasks…, scaling via {desired} or {delta}. JVM args are shell-escaped into the systemd ExecStart (shSafeJvm): the unit launches the server through tmux new-session '<java … -jar …>' and tmux runs that via sh -c, so a ; in a flag used to split the command — java ran without -jar, printed usage and exited, leaving no server, no tmux session and no log to say why. Applied on every MC launch path, both at first provision and when args are edited and the instance restarted.

• Blueprints (eggs) + golden images — per-egg LXC template per node, linked-clone provisioning in seconds, optional warm pool. GET /api/blueprints, GET /api/images. Blueprints are named for the software they install, not a use case: Velocity, Paper, Folia, Purpur, CanvasMC, Limbo, PostgreSQL, MariaDB, Valkey, MinIO, Nginx, Apache (+ Hytale, generic). Folia/Purpur/CanvasMC are Paper flavors (software.flavor), not new kinds — they install and behave exactly like Paper (server.jar, Velocity forwarding, connector, plugins dir); only the jar source differs (purpurmc API / Fill folia / the canvasmc.io /api/v2/builds/latest API), so every kind === "paper" gate keeps working unchanged. CanvasMC is Folia-based, so it's regionised like Folia and takes Folia-compatible plugins. The two old Paper presets ("Paper Lobby"/"Paper SMP") collapsed into one Paper blueprint; their retired ids still resolve via LEGACY_BP_ALIAS and stay reserved in BUILTIN_IDS.

• Folia support (the connector runs on it) — Folia ticks independent world regions on separate threads and rejects the Bukkit scheduler, which is why it refused to load the connector at all. Every scheduling call now goes through plugin/.../paper/Sched.java, which targets the regionised API: global work on the global region, per-player work on the thread that owns that player, world work on the location's region, off-thread work on the async scheduler. Paper implements the same API, so it's one build for both — no platform branch. Player-state restores, message delivery and the sharding seam handoff run on the player's own thread, and every teleport is teleportAsync (a cross-region teleport can't be synchronous). plugin.yml declares folia-supported: true. Verified on a live CanvasMC server and on Paper. Third-party plugins still need their own Folia support — that's on them, not Conduit.

• Regions view (Folia only) — Folia doesn't publish its region split, so the connector infers it: a task run on a player executes on the thread ticking their region, so the thread name identifies the region and players sharing one share a region. The service page grows a Regions tab — region count, players, TPS, worlds, a chunk map with each player coloured by region, and the region roster. Regions holding nobody are invisible by construction, and the page says so. FoliaRegions.java, GET /api/services/:vmid/regions.

• Role (a task property) — what a server does on the network — proxy · lobby · smp · game · hub · web · db · generic — is now a task property, independent of the software: a Paper server can be a lobby, an smp, a minigame game host or a plain backend. Picked in the deploy wizard (with a one-line hint per role) and editable on a running server (PATCH /api/tasks/:id). Resolved by effectiveRole(task, bp) — task override → blueprint default → generic — which is the single source of truth every behaviour gate reads (proxy fallback order, autoscale defaults, firewall ingress, MOTD, language group, metrics, instance tags), so the raw blueprint role is no longer consulted directly. A blueprint still ships a sensible default role. The stock flat-lobby world + rules is a deploy-time preset (LOBBY_SEED), offered when a Paper-family server is given the lobby role (it used to be baked into the retired "Paper Lobby"). A one-time idempotent leader migration (migrateBlueprintIds, lib/migrate.ts) back-fills role on existing paper-lobby/paper-smp tasks so nothing changes for them. lib/blueprints.ts.

• Software versions — pinned line per task, hotfix-to-latest-build, explicit full-version switches, optional auto-hotfix, and rolling (zero-downtime) updates: with rolling:true (the version card's default toggle) a multi-instance task is updated one instance at a time, waiting for each to re-register before the next, so the service never fully drops — pairs with the proxy's pause-on-restart (players on the restarting instance park on a sibling + auto-return). GET /api/versions/status, POST /api/tasks/:id/update. Shared helper lib/rolling.ts also backs the bulk rolling restart action.

• Proxies update themselves too, without kicking anyone. Auto-hotfix used to touch only instances with zero players, which is right for a backend and a deadlock for a proxy — every player on the network is connected through it, so its count is never zero and it was skipped forever. The fleet's front door was the one thing that never got patched automatically. It now gets drained instead: the proxy is held out of the public ports (lib/firewall moves the DNAT to a sibling in about a second, the same machinery that already covers a crash), its remaining players are left to drift off in their own time, the jar is swapped once it is empty, and the hold is released. If it does not empty within five minutes nothing happens and the next pass tries again — a background update has no deadline, so waiting for a quiet minute costs nothing and interrupts nobody. At least one proxy always keeps serving: the floor is a property of the fleet's size, so a pair mid-roll still refuses to give up its last instance, and draining with no sibling to drain onto is refused outright. Opt-in per task: the drain to update checkbox on the version card, next to auto-hotfix and shown once that is on, because the first thing this touches is the front door of a live network — a decision somebody makes rather than inherits from an upgrade. Left off, a proxy behaves exactly as it always has, except the log now says it is being skipped and names the control instead of silently doing nothing forever. Refused on anything that is not a proxy, with the reason: a backend that has players on it is skipped and retried, which is already right. (It shipped without that checkbox or any route, so for a day the flag was read by the reconcile loop, named in its own log message, and impossible to set. lib/reachable-flags.test.ts now fails the build if any task flag the engine branches on has no route that writes it.)

  Replace to update is the third way, and the gentlest: blue-green. On a new build, Conduit provisions a NEW proxy instance first (a freshly provisioned instance installs the newest build as a matter of course), and only once it is serving does the old one start being eased out — held out of ingress so no new player lands on it, while everyone already on it stays, unkicked, until they leave on their own. At zero players it is stopped and destroyed and desired returns to normal; after a day of patience it is retired anyway, and the network's auto-reconnect lands the stragglers on the replacement. The run itself lives in the store (task.replaceRun), not in process memory, so a panel restart mid-replacement resumes exactly where the persisted run says it is — and every step is re-derived each pass from live facts by lib/proxy-replace, the pure decision core, which refuses the one veto that must never happen: holding the old proxy out of ingress while the new one is not serving. Static proxy tasks only (on autoscale the reconcile owns the instance count), opt-in via the replace to update checkbox beside drain, and it takes precedence over drain when both are set. The trade against draining: capacity first, nobody rushed. lib/fleet-roll.ts (which instance may go, pure + tested), lib/proxy-roll.ts (drain, apply, return).

• Bedrock players, kept current — Geyser (Bedrock clients on a Java network: phones, consoles, Windows) and Floodgate (they get in without owning a Java account) are built-in managed plugins alongside LuckPerms and Chunky, tracked against GeyserMC's own feed and auto-updatable from the same page. They install on the proxy, not on each backend — one install covers everything behind it, and their updates ride the drain-and-roll above, so adding or updating Bedrock support costs nobody their session. Both ship with no targets: Geyser binds a UDP port and changes who can reach the network, so it is opt-in. Conduit places both jars (each checked against the sha256 the feed publishes), writes Geyser's config once — port 19132, auth-type following whether Floodgate is actually installed — and emits the UDP forward on every node so a phone can reach it. lib/bedrock.ts, lib/geyser.ts. Geyser is pinned to 2.10.1, and the pin is load-bearing. Geyser does not shade Adventure — it links against whatever the proxy provides — so "latest" is a fact about GeyserMC's build server and not about whether Bedrock works on your proxy. Measured here, swapping the jar under a test proxy for each run:

  Geyser	Velocity	result
  2.10.1 b1184	3.4.0 (stock)	Started Geyser on UDP port 19132
  2.10.1 b1184	3.5.1 (source-built)	Started Geyser on UDP port 19132
  2.11.0 b1205	3.5.1	NoSuchMethodError: GsonComponentSerializer.toBuilder() — never binds
  2.11.1 b1225	3.4.0	same — never binds
  2.11.1 b1225	3.5.1	same — never binds

  2.11.0 is where it broke. The pin tracks the newest build within the 2.10.1 line, so hotfixes still arrive and only the version bump is held back; lift it once a newer Geyser is confirmed to start on the Velocity your fleet runs. Independently of the pin, Conduit checks that Geyser is actually bound to the Bedrock port after a restart and raises bedrock.silent if it is not — a staged jar is not a working feature, and that check is what caught this in the first place.

  Both halves, or nobody joins. Geyser hands a Bedrock player through with a Floodgate token rather than a Mojang-signed profile, so a backend without Floodgate refuses them at whichever server they land on. Conduit places the spigot flavour of the same Floodgate build on every covered backend and copies the proxy's key.pem to each one — the same key on both ends is what signs the token and what verifies it. One target list, split by role, because two lists is an invitation to configure the proxy, forget the backends, and end up with Bedrock that looks switched on and rejects everybody. The key never travels through /var/lib/conduit (the file manager serves that volume, and this file would let someone impersonate any Bedrock player); it is read from a proxy and written straight into each backend, compared by hash. Floodgate mints it on its own first start, so before the proxy has run once this reports a wait rather than an error.

• Overlays / file templates — layered file distribution (egg overlay → _global/<kind> → named global templates with hand-picked members → per-task overlay). templateSync:true on a task auto-resyncs on overlay change; POST /api/tasks/:id/resync re-applies now.

• Managed plugins (Extensions) — LuckPerms/connector/SchemFlow/Chunky/Geyser/Floodgate built-ins

  • user plugins from Modrinth/GitHub/URL/GeyserMC with server scoping and update status; jar-drift visibility (/api/plugins shows pending-restart when a deployed jar differs from the store).

• Schedules — daily restarts (warn minutes, only-when-empty), console commands, broadcasts, backups; targets = group/subgroup/task/instance. …/api/schedules.

• Backups — PBS on-demand + scheduled jobs + one-click restore. …/api/backups.

• Firewall — dynamic nftables, master toggle, per-service ingress overrides, manual forwards, with a packet counter on every forward. …/api/firewall.

  There are two forwarding layers, because there are two places public traffic can land. Every node carries ip conduit for an edge that sends the public address to a node's own IP. The panel replicas carry ip conduit-vip for an edge that hands it to the VIP instead — a GRE tunnel terminating there, or a router forwarding to it. The second is not optional cleverness: a packet addressed to the VIP is delivered straight into the VIP-holding container and never sees a node's prerouting, so no node rule can match it. The panel and console answer on that path only because they listen locally; a game port had nothing there and was refused while looking perfectly configured in every view. Installed on all replicas since the VIP moves — only the holder sees traffic, which is what makes a failover keep the network up rather than dropping every player. Re-applied periodically, because a signature that tracks intent cannot notice that someone flushed the table.

  Ingress forwards land on every node, not only ones hosting the service. Which node the public address reaches is decided upstream and Conduit cannot see that config, so following the instances quietly tied the entrypoint to placement — the network would have gone unreachable the first time a proxy was rescheduled. Each node prefers a local instance and otherwise forwards to the lowest-numbered healthy one.

  With two proxies (proxy HA), the VIP table picks one winner instance per public port — health-probed, sticky by incumbency (a recovering proxy causes zero churn), and every protocol rule for a port moves together so voice UDP never splits from game TCP. Failover is not left to the 10 s reconcile: the leader runs a 1-second watchdog over the candidates of every multi-candidate port; two consecutive failed probes veto the dead instance's IP and rewrite the VIP table and every node table immediately — measured live at 1.2 s (node) / 1.7 s (VIP) from socket death. desiredForwards honors the same veto, so the next reconcile agrees instead of re-installing a crashed process whose container still reports running; the veto expires after 30 s so a dead watchdog degrades to tick-cadence rather than pinning a stale opinion. Each move raises a proxy.failover alert carrying its measured time. Established TCP sessions die with their proxy — physics, not policy — so the number that matters is how fast reconnects land on the survivor. lib/firewall.ts (vipChooseForwards, fastSwapPlan, startVipFastWatch).

  The forwarding-secret parity watch (lib/secret-parity.ts) closes the trap the second proxy shipped into: provisioning is write-once, so a store-side secret rotation never reaches installed members — the fleet keeps working pairwise on the old value until the first fresh provision joins it and every login through the mixed pair dies with "Unable to verify player details", while TCP health probes see nothing wrong. Every ten minutes the leader sweeps the shared services dir on a node (the panel replicas don't mount it — the first version read panel-locally, found nothing, and reported "no drift"; a detector's zero is worthless until it has seen a known-bad): one exec hashes every proxy forwarding.secret and Paper paper-global.yml secret and reports only mismatching vmids. Only the store secret's SHA-256 leaves the panel; no value is printed anywhere. Raises secret.parity on split-brain. Deliberately no auto-fix: secrets must match pairwise, so one-at-a-time convergence breaks logins progressively — convergence is a coordinated operator flip.

• Command palette finds features, not just pages (⌘K) — a page list only helps once you already know where something lives, which is the opposite of what you need when hunting for it. Recent additions are indexed by what they do and keyed on the words someone types in the moment: "nobody can join", "526", "crashed", "revoke". Entries can address a section, not just a route — /packs#packs-domain scrolls to the download-host card and rings it briefly, because landing at the top of a long page having asked for something specific leaves you hunting, which is what the palette exists to avoid. The scroll/highlight is mounted once in the layout, so any element with an id is addressable and a new section becomes linkable just by giving it one; it polls briefly for the target because pages often render a beat after navigation. A fragment must match a real element — features that live inside a settings dialog carry no hash and name the dialog instead, since a hash matching nothing scrolls nowhere and looks broken.

• The Elastic World — split one strip in two and leave everyone else's boundaries alone. The resize re-tiles (width×count divides the whole axis), so adding capacity moves every boundary and restarts every instance — the most disruptive answer at exactly the moment one strip is drowning. A split is the surgical one: the hot strip's range is cut at a chosen line, the outer half streams to a spare instance through the same tar→S3 pipe the resize migration uses, and only the strip being split stops, briefly. The price is paid once and said out loud: the first split ends the uniform grid — every strip's bounds are pinned as an explicit partition, and width-based re-tiling is refused from then on. From there the world grows by splitting and narrows by removing the last strip (its range absorbed by the spatial neighbour). Safety is in the order of operations: snapshot first (drill-proven), geometry written only after files land, the split strip restarted in a finally. When one live strip holds most of a world's players, its row gets a quiet chip suggesting the split — the panel points, a person clicks. Proven live: strip 1 of the TimeSMP region world split at x=0, 636 region files to the new instance, strips 2 and 3 never moved a block.

• Beyond Minecraft — the catalog's generic runtime makes any game server a first-class service: a blueprint declares apt packages, an install script and a start command, and the panel provides the rest — tmux console with two-way commands, TCP uptime, vzdump + restore drills, the map, the works. Mindustry ships as the proof: deployed from its blueprint like any Paper server, it answered status through the panel console on first boot ("Playing on map Islands / Wave 1 · 60 FPS"). Blueprints ask for default-jre-headless rather than a pinned openjdk-NN, because a pinned name rots the day the distro moves on — the first provision proved that too.

• Hytale — the one catalogued server that is not Minecraft and not merely generic. Its own blueprint (Java 25, UDP 5520, 8 GB) plus a shared-asset model, because the distribution is a 3.4 GB Assets.zip and copying that per instance would be absurd: the official downloader fetches it once per cluster into a read-only /assets/hytale, every instance bind-mounts the same copy, and only /opt/hytale/data is writable per server. A Hytale build of the connector (plugin-hytale/, loaded as a mod) registers, heartbeats every 3 s, tracks joins and leaves, and executes move, message, broadcast and kick, translating Minecraft colour codes into Hytale's styled Message API so one message catalog serves both platforms. Proven live on 2026-08-17 against a server that had been up for 49 hours: the mod loads (Enabled plugin dev.admin:conduit-connector), the panel lists it as env: "hytale", ready: true, max: 100, heartbeating; the UDP 5520 forward installs correctly on all three nodes and carries traffic; and the action delivery path acks — a queued broadcast moved ackActionId from 0 to the action's id within two heartbeats, with nothing thrown in the server log. What remains unproven is only the visible effect of an action in-game, which needs a player.

• Hytale login state, surfaced — a Hytale server needs its own account tokens before it will serve anyone, and without them it boots indistinguishably from a working one: process up, port bound, connector heartbeating, a green row. Every join is refused. The only evidence is one WARN in a boot log, and a server on this cluster sat in exactly that state for days unnoticed. A slow pass now asks each Hytale service auth status over the console it already has, parses the reply, and raises an error naming the server when its tokens are missing — resolving itself when somebody logs it in. The parser is careful about one thing in particular: an unauthenticated reply's first line reads Connection Auth: Authenticated (mTLS + JWT), so anything matching "Authenticated" against the blob would call an unusable server healthy. Both the session and identity tokens must be present, and an unreadable console answer is logged rather than alerted — it says nothing about the tokens either way. Logging in is still a console action (auth login device prints a code and a URL, and the code expires in ten minutes, so it wants a browser already open); the panel tells you it is needed. Not built: a Hytale proxy — the plan is to adopt a QUIC relay as the Velocity equivalent, so Hytale services join the same group and routing model as Minecraft — and a panel button for the login flow itself. Neither is blocked on the runtime any more, which is what previously held them.

• Backup coverage — /backups is the data-safety instrument: one report joins the three backup planes (vzdump, strip snapshots, Time Machine) per world. The verdict band answers "would it come back?" in five numbers — newest vzdump age, worlds covered, proven restores, S3 footprint, reclaimable staging — and the coverage matrix shows every world's planes side by side with a verdict whose tooltip names its reason. The vzdump age is the oldest of each container's newest archive, so one freshly backed-up container cannot mask two naked ones. All S3 facts come from ONE bucket walk (key names carry the task, strip and capture time — no manifests, no SSH), memoised a minute; a failed walk reports "unreachable", never zero, because "your snapshots are fine" and "I could not see your snapshots" are different sentences. Scheduled jobs wear the one fact a schedule cannot hide (the newest backup its storage actually received), the archive table leads with colour-coded ages and flags stale guests, a twelve-week sparkline draws the cadence, and an object-storage band shows what sits in S3 by purpose — including transfer staging left behind by splits/resizes, called out as reclaimable with the button that reclaims it. Naming a number nobody can act on just moves the work to whoever reads it: 1.3 GB from a July split was still in the bucket in August, riding along in every nightly backup of the object store, because "reclaimable" was the end of the sentence. Splits now delete their own buffer on the way out — success or failure — and the page can clear whatever a move died too hard to tidy. It is scratch either way: the source of a move never deletes the files it pushed, so nothing there is the only copy of anything. First run on the live cluster the page surfaced what nobody knew: five worlds had never been backed up on any plane, and no backup schedule existed at all.

• Verify — a backup that has never been restored is a hope, not a backup. The Backups page lists every backup-shaped artifact — vzdump archives, strip snapshots, Time Machine captures — with its last verification, and "never run" is styled as loudly as a failure because it is one. Having nothing to verify is not a failed verification, and for a while it was recorded as one: four strips with no snapshots yet sat on the page as red failures, each with its own alert, each holding a slot in the proven/total ratio it could never fill. There are three outcomes now — verified, failed, and nothing to verify — and only the middle one is an incident. A verification runs the real read path into a scratch area: strip drills stream through the same S3 preamble and pipe a restore uses and must yield a level.dat and region files; Time Machine drills are the strictest — the extracted file count must match the manifest exactly, so a truncated upload cannot pass; vzdump drills do a full restore into a scratch container that is never started (a booted restore would walk into the proxy wearing a live server's identity), mount it, check for the shape the container should have, then destroy it. That check used to know two kinds out of eleven: Paper wanted a jar, a world and its unit, Velocity its proxy jar, and everything else passed on "os-release exists and the filesystem is over 50 MB" — recorded, in those words, as proven. A Postgres archive restored with an empty data directory produced the identical sentence, because neither question asked was about a database. Every kind now carries an expectation read out of a running container rather than reasoned about: postgres its data directory (version globbed, so a major upgrade is not a fleet of false failures), mariadb its ibdata1, redis its data directory and not dump.rdb — an append-only server has no snapshot — and hytale its start.sh, because the obvious guess, a jar in /opt/hytale, is not there. Where Conduit genuinely has no expectation the verdict says restored, not proven, and says plainly that it proves the archive restores rather than that the service returns. One thing it now states out loud instead of leaving to be assumed: nginx and Apache keep their web root on a bind mount, which Proxmox does not put in a container archive at all, so that content is covered by the shared-volume capture on a different schedule — a passing nginx drill would otherwise read as "the site is safe". Being lenient with an unfamiliar kind is still right, and failing a redis backup for not being a Minecraft server would teach people to ignore drills; calling the lenient answer proven was the part that was wrong. S3-backed drills also run themselves, one artifact per week, store-claimed across replicas; vzdump drills stay a button because they create and destroy real containers. Failures raise alerts; a disk-space guard runs before any extraction.

• Off-host — a backup on the same machine as the thing it protects is a copy, not an insurance policy. It covers the failures people plan for (a bad upgrade, a deleted world, a corrupted config) and none of the one that takes the host with it, and it is invisible in every view that shows only "backed up 1d ago". Coverage now judges where an archive lives, not just how old it is: Proxmox's own shared flag is believed over the storage type, an unknown storage is never allowed to read as safe, and one off-host copy is enough however many local ones there are — nine local archives and no remote one is the same position as none. This cluster passes and the check was written knowing that, which is worth saying because a check written against data that satisfies it is a check nobody has watched fail: all 296 archives live on an external PBS host that is not one of the three nodes, including the containers stranded on a node that is down. The value is not the finding, it is that nothing in the product asserted this before — so the day somebody adds a fast local dir storage, something says so.

• Ghost Replay — where everybody was, a day back. The connector already sampled every player's position for the region map and the panel already received it on every heartbeat; nothing was keeping it. Now a coarse trail (one sample per server every few seconds) is kept per task, with a scrubber under the strip map: drag to a moment and the bar shows who was standing where then, nearest sample per player so a person is a dot and not a smear. It pairs with the Time Machine on purpose — that one answers "undo the grief", this one "who did it", and either alone is half a story. Off by default: it is a record of people's movements, so switching it on should be a decision someone made, and switching it off stops recording while keeping what exists (deleting someone's history is a separate act, not a side effect of a switch). The position sampler used to be Folia-only because the region map was its only consumer; a position means the same on both platforms, so it now runs everywhere. Trail lives on the nodes' shared volume, written through the agent — a panel replica has no shared filesystem of its own, and three container-local trails would each look complete while omitting whoever another replica recorded.

• Selection wand — /conduit wand (permission conduit.wand) gives an axe: left-click one corner, right-click the other, and the box appears in the panel as a chip to click wherever a region is asked for. Typing corners off F3 works, but you are standing in the world looking at the two blocks you mean — the wand removes the alt-tab. Deliberately not a protection plugin or a region database: it produces one short-lived selection per player and forgets it, so the panel decides what a box is for while this only says where it is, which is what makes it the input for everything region-shaped rather than a portals-only detail. The handler cancels the click, so marking a corner never also breaks the block; only real block clicks count (an air click has no coordinates); and a box spanning two worlds is refused with a reason rather than sent as nonsense.

• Ask Conduit — /ask answers operator questions from the panel's own evidence. One question, one model call (Claude, via the Anthropic SDK): the panel assembles a bounded bundle of what it already knows — fleet state, uptime verdicts, backup coverage, shard geometry, alerts, events, crash breadcrumbs, drill history — and the model is instructed to answer only from that bundle, quote numbers verbatim, and end every answer naming which evidence sections it used, so anything it says is checkable against the page that shows the same numbers. When the evidence doesn't contain the answer, the honest reply is "the evidence doesn't say", not a guess; and the whole surface is read-only — it describes where an action lives, it never performs one. No chat history on purpose: each question gets a fresh bundle, because a stale answer about live infrastructure is worse than none. The API key is a Settings card next to the GitHub token, stored the same way (cluster config, so it survives redeploys; the env var honoured second) and validated against the API before it is saved — a bad key fails loudly at paste time instead of quietly on the first question.

• Map studio — the live network's world-authoring workflow, as one tool instead of three. On the network this replaces, building a minigame map took a build-server skript for the world, per-game setup commands for the spawn points, and SchemQL to push the blocks into MySQL — three systems held together by staff lore. /conduit map is the whole flow: create <group> <name> makes the same void world the old Bauserver did (group tab-completed from the panel's games-enabled tasks), and the primary authoring flow is walking: stand somewhere and add spawn appends the family's next number — spawn-1, spawn-2, … — exactly the rhythm of the live /setup tools, with undo taking back the last point and set <id> / remove <id> for exact names (families and ids tab-complete from what the map already uses). bounds takes the wand box as the map's extent, and scan <family> <block> is the bulk tool that walks the box turning every matching block into a numbered location — which is how a thousand dance-floor blocks ever get authored without someone going mad. The metadata lives in conduit-map.json inside the world folder, so the map is self-describing wherever the folder goes; upload tars the world and PUTs it straight to the cluster's S3 under the same worlds/ prefix game hosts already pull from. The panel registers it in the map library, merges it into every matching games config at heartbeat delivery — hand-written maps keep priority — and mirrors the rows into the WDB tables so the unchanged live-network gamemode skripts see it too. Setting a spawn IS writing the config; nobody types coordinates into a form. The library shows in the Games dialog with an in-rotation toggle; de-listing keeps the tarball (deleting stored world data is a bigger decision than de-listing a map).

• Cross-server portals — walk into a drawn region on one server, arrive at exact coordinates on another. The seam handoff generalized: the same panel-staged coords and proxy move the shard seam uses, with only the trigger (a region instead of a strip boundary) and the destination (explicit coordinates) changed. The arrival side needed no new code — the pending-coords fetch was never gated on sharding, and restore() already moves any arrival not standing in the server's own strip, which on an unsharded server is everyone. A portal names a destination task; the panel resolves it to a live instance every heartbeat, so it survives the destination restarting, and a destination with no live instance leaves the region cold rather than teleporting people into a void. Arrivals get a three-second grace so a return portal cannot bounce them straight back. Managed from the Portals card on the World tab; corners are the numbers off F3.

• Ephemeral event servers — an event window can own a server. When the window goes live, the event's blueprint becomes an ordinary one-instance task and the reconcile provisions it: an event is not a special way of running a server, it is a normal server with a lifecycle attached. The join announce waits until the server's connector actually registers — handing players a /server name that answers nothing teaches them the announcement lies. After the event, the server lives exactly as long as someone is on it: empty for lingerMin, then decommissioned, container and all; players still playing hold the teardown, and one returning player resets the clock. The blueprint is validated at scheduling time, when a person is watching, not when the event goes live with a crowd waiting. …/api/event-windows with a server: { blueprintId, lingerMin } block.

• Canary deploys — one instance tries the new build first; the fleet follows only if it survives. The "Canary first" button on the version card sends the jar to a single instance and watches it against its siblings — the crash detector's verdict, the connector going quiet, TPS falling away from the siblings' median, all strikes-based so one bad sample cannot kill a good build. The watch ends in a verdict: promote to the rest (rolling, never the whole task at once) or roll back — and rollback restores the exact previous bytes from an aside copy kept on disk before the update, immune to the upstream feed having moved on. The resolvers answer "what is newest"; a rollback needs "what was here", and only the disk knows that. Verdicts land as alerts; promote/cancel early from the in-flight chip. …/api/tasks/:id/canary.

• Predictive pre-warm — scale before the evening peak, not on it. The analytics heatmap already knows the network's weekly rhythm (average concurrent players per weekday × hour, from presence-minutes in the audit trail); the autoscaler only ever reacts to it. Pre-warm reads the rhythm instead: fifteen minutes before an hour that history says is busy, an opted-in task's floor rises to what that hour has actually needed, so instances are up and quiet before anyone joins. The floor is a floor — it never lowers anything, expires on its own (a wrong forecast costs at most one over-provisioned hour, in the safe direction), and is capped by the task's own max, so pre-warm never grants capacity the operator did not. Opt in per task; in practice that means the entry points, where arriving players land. …/api/tasks/:id/prewarm.

• World Time Machine — continuous snapshots of a live world, and the way back. Every 15 minutes (and on demand) the changed region files are captured — after a save-all flush, without stopping the server — into the cluster's S3, riding the same tar→zstd→MinIO transport as the strip snapshots. Rewind works at two grains from the World tab: the 512×512 region a grief happened in (entered as world coordinates, the numbers off F3) or the whole world, to any captured moment; the needed version of each file is the newest copy at or before the chosen time, found by walking the manifest chain. Nothing is deleted on restore — everything replaced moves to .tm-displaced/ first, so a rewind is itself reversible. Off by default: it costs S3 space and a flush per capture, and a rewind button is only trustworthy if its price was a choice. Honest about live capture: a region written at the exact tar moment can be torn; the flush narrows the window and the next capture self-heals it, and the guaranteed-cold copy remains the strip snapshot, which stops the server on purpose. Chunk-grain restore (Anvil surgery) is deliberately deferred.

• Network map (/map) — the network as a picture: edge → services → nodes in three fixed lanes, players as wire weight, placement on the right, moving live. Fixed lanes rather than a force graph, because a layout that reshuffles between visits builds no spatial memory. Up/down comes from the uptime sweep — the one place that knows what "up" means per kind (TCP for a database, HTTP for a website, heartbeat for Paper) — with amber layered on only when the connector data itself shows a split. Judging everything by connector reachability was the first version, and it painted every healthy database red.

• World strip map — on a sharded task's World tab, every strip drawn to scale on the X axis: owner and live player count inside each band, seams as the hairlines between them. The rows below say the same things one strip at a time; the bar says them spatially. Colours follow the row semantics exactly, so the two can never tell different stories.

• Platform endpoint health — the panel domain, the packs URL and the SSH gateway are probed by the same uptime sweep as every managed service and shown as a "Platform" band on /uptime. They used to be checked only when someone opened their settings page — health-checking as a page-load side effect, which is how a broken pack hostname stayed invisible until a player hit it. The packs promote/demote logic lives in shared code (lib/endpoint-health) called by both the sweep and the settings page, and both transitions raise an alert.

• Last words (crash breadcrumbs) — the connector's shutdown path posts a breadcrumb: log tail, TPS, heap, who was online. The journal for an OOM'd JVM is often just "Deactivated"; the server itself knew more right up to the end. The console shows it under the journal (--- last words from the server ---), the crash alert says it is there, and every clean stop posts one too — "stopped cleanly with 3 players" is also what you want to see when a server is missing. Best-effort and bounded: a panel that is down never holds a shutdown hostage.

• /conduit find + alerts — in-game: find <player> answers from the connector's own registry; alerts shows the panel's open alerts in chat, with down/recovered pairing so last night's blips do not read as burning now. Read-only on purpose — acting on alerts stays where the audit trail lives.

• conduit doctor, second ring — after the client-path checks, the CLI asks what the INTERNET sees: every exposed service through the panel's own reachability verdict, plus the packs URL, in the panel's wording so the two can never disagree. --json for scripts; bounded because each verdict costs an external probe.

• Network messages — the lines players read when the network moves them around: sent elsewhere during a restart, waiting while one comes back, a queue slot opening, /server naming something that is not up. Every one was hardcoded English in the proxy, so changing the wording meant editing Java and rolling a jar; two were overridable and the rest were not, for no reason, and one (restartHoldMessage) was read by the connector and never sent by the panel, so its override could not be used at all. All of them now go through one lookup with {task}, {server} and {player} filled in, overridden per group beside the existing full-network message, and spread flat into the proxy config so the connector needed no new shape. An unset key keeps the built-in wording — nothing has to be configured for the network to keep speaking. The catalog (lib/network-messages.ts) is the single source of truth: the dialog renders from it and the panel sends every key with defaults resolved, so improved wording reaches players on the next heartbeat rather than the next jar roll — the connector's inline defaults only cover the seconds before the first heartbeat. Unknown keys are rejected at the PATCH, so a typo is an error instead of a mystery. The editor shows each default as the field's placeholder rather than pre-filling it, which is the difference between "keep whatever Conduit says" and "pin today's wording forever": a pre-filled box would silently freeze the current text the first time anyone opened the dialog. Each field previews its rendered colours live.

• Network map (/map) — the network as a picture: edge → services → nodes in three fixed lanes, players as wire weight, placement on the right, all live off the bus. Fixed lanes rather than a force graph, because a layout that reshuffles between visits builds no spatial memory. Health comes from the uptime sweep — the sweep already knows what "up" means per kind (TCP for a database, HTTP for a website, heartbeat for Paper), and the map's first version re-derived it from connector reachability, which painted every healthy database red. Amber is layered on only when the connector data itself shows a split. Cards link to their pages.

• Strip map (World tab) — every shard strip drawn to scale on the X axis: owner, live player count, seams as the hairlines between segments, bounds labelled. The rows below say the same things one strip at a time; the bar says them spatially — "strip 2 is half the world and holds everyone" is a shape, not a table read. Colours follow the row semantics exactly so bar and rows cannot disagree.

• Last words (crash breadcrumbs) — the connector's shutdown path posts a breadcrumb: log tail, TPS, heap, players online, and whether the stop was clean. The journal for an OOM'd JVM is often just "Deactivated"; the server itself knew more right up to the end. The console diagnostics append it under the journal, the crash alert says it is there, and only the newest per service is kept — its job is to explain the current absence, not to be a log archive. Best-effort and bounded: a panel that is down must not hold a server's shutdown hostage.

• Platform self-monitoring — the panel domain, the packs URL and the SSH gateway are probed by the same uptime sweep as everything else and shown as a "Platform" band on /uptime. These were checked only when someone opened their settings page — which is how a broken pack hostname stayed invisible until a player hit it. The packs promote/demote now rides the sweep tick and alerts on both transitions (lib/endpoint-health), shared with the settings GET so there is one definition of "does the packs URL work".

• /conduit find + /conduit alerts — in-game: which server a player is on (answered from the connector's own registry), and the panel's open alerts in chat. Read-only on purpose — seeing what is paging belongs in-game; acting on it stays where the audit trail lives.

• conduit doctor, outward-facing — a second ring after the client checks: every exposed service through the panel's own reachability verdict plus the packs URL, in the panel's wording so terminal and panel can never disagree. Bounded and skippable (--skip-network) because each verdict costs an external probe.

• Labels — your own grouping: a name and a colour, attached to any service, shown as a small dot that expands to its name on hover. Proxmox-style, and for the same reason — a fleet list is mostly names and numbers, so a row of text chips would compete with all of it, while a 6px dot is enough to see that two services belong together. Deliberately not the same thing as a role: a role is behaviour (proxy routes, web gets ingress, lobby is a fallback target), so that set is closed and inventing one would mean inventing a behaviour that does not exist. A label means nothing to the control plane and never will, which is exactly what makes it safe to let anyone create — and it is most useful precisely where the role says nothing, since a database is just a database. Services store label ids, so a rename or recolour is one edit rather than a sweep; deleting detaches everywhere, because a dangling id is invisible and would silently reattach if a later label slugged to the same string. New labels take the least-used palette colour, so a handful stay distinguishable instead of landing on the same blue twice. …/api/labels.

• Crash detector — a server can die without its container dying: the JVM crashes, the unit stops, the CT keeps running and every view stays green. Nothing caught that before — the instance is not "stopped", so the self-heal never touched it — and it is the failure noticed last precisely because the page looks fine. The reconcile now raises an alert when the connector has gone quiet and the service unit confirms it is not active; either signal alone is a false alarm waiting to happen, since a server can be silent while saving a world and a unit can be briefly inactive during a planned restart. A container mid-move is skipped (stopped on purpose), and the guard is the persisted alert feed rather than a Map, so a deploy cannot re-alert for a crash already reported. The instance count separates degraded from down — amber when some answer and others do not, red when none do — with red held back until booting is implausible, because going red on every boot teaches people to ignore the colour.

  The console shows the reason. With no tmux pane it used to guess ("may still be provisioning or stopped") at the exact moment the answer mattered; it now shows the unit's state, exit status and the last forty journal lines instead. Both console routes share one implementation — they each carried their own guess, and a reader has no idea which one answered them.

• Reachability check — is this server actually online, and what does the world see? GET /api/tasks/:id/reachability answers in one request what otherwise takes an evening: a real Minecraft status ping (MOTD, version, player count, latency — the only thing that proves a player could see it), TCP reachability probed from the internet (the panel cannot test its own public address; NAT hairpin makes a local connect meaningless), and the DNAT counters. A counter still at zero after a failed external probe is the proof that the traffic never arrived — the fix is a port forward upstream, not anything in Conduit, and every other view will keep insisting the service is healthy. Non-HTTP ports are tested at the raw IP, since a Cloudflare-proxied hostname only carries HTTP and testing a game port against one produces a confident false negative. The verdict names the conclusion; a service that is deliberately internal is not probed from outside at all.

• Move a container between nodes, including ones Proxmox refuses to migrate. A CT with a bind mount is declined outright — Proxmox will not reason about a host path — but every bind mount Conduit creates lives under the replicated GlusterFS volume, so the data is already on the target before the move starts and only the declaration is in the way. Conduit stops the container, detaches the mounts, migrates, re-attaches on the far side and starts it: seconds instead of a backup and restore of a multi-GB rootfs, and at no point is the only copy of the data in a temporary archive. A mount that is genuinely node-local is refused by name, because moving without it would start a server with an empty directory where its world used to be. Any failure after the detach restores the mounts and starts the container where it currently is — a server that never moved beats one left half-moved — and the restore is verified, since a container running without its mount looks perfectly healthy while its config directory is empty. Mount changes go through pct over SSH: Proxmox forbids an API token from touching a bind mount at all. Progress (shutting down… → migrating… → re-attaching mounts… → starting…) lives in the shared store and is pushed over the live bus, so it shows on whichever replica the UI is talking to, in the same vocabulary as every other lifecycle action, and lands in the activity log rather than only in a live indicator. And because a verification inside a process that can die mid-move is not a guarantee — a panel deploy once killed a migration between detach and re-attach, stranding a container to crashloop on a broken /opt/shared symlink — a recurring mount guard re-derives the expectation from first principles: a service directory on the volume plus a share-carrying software kind means the bind must exist. A stranded container is re-attached (applying at its next start), one mid-migration is left alone, and an attach waiting behind a restart raises an alert instead of pretending the fix is live.

• Metrics / uptime / monitors — live player counts, Proxmox RRD CPU/mem series (5m→30d), per-service health with uptime % + incidents, and user-defined TCP/UDP/HTTP/ping monitors.

• Players page — live SSE network list (skins, MC/Hytale split) with move (compatible targets only), message (&-code preview), kick. POST /api/connector/action. The connector also reports an afk flag per player (5 minutes without look/move/chat/command/ interact input) riding the heartbeat roster — honest active-vs-parked concurrency.

• Player 360 (/players/:id, from the row context menu) — one page aggregating a player's identity + online status, rank/prefix, Blossoms balance, active punishments, friends, and 30-day session/action history. GET /api/players/:id (name or uuid; each source degrades independently).

Voice · managed Simple Voice Chat (/voice) — system-managed network voice. Conduit installs the SVC plugin on chosen servers (whole Conduit groups and/or individual services via the target picker), writes their config, and auto-opens the per-instance UDP voice port through the node firewall — voice_host is set per a deterministic per-vmid port (24700–25499) so voice works through NAT with zero manual port setup. An endpoint mode toggle picks what host that port is advertised on: Public (the configured public IP/domain — real players; forward the UDP ports on your edge) or Local (LAN) (the node's own LAN IP — reachable on-LAN with no port-forward, for testing). The page shows the live voice endpoint + per-server state (live / pending restart / offline) for every covered server, and warns when the public host isn't set (public mode only). Saving any setting auto-applies to every covered running server (re-writes voicechat-server.properties + voicechat reload live); range/codec changes take effect on reload, while voice_host/port changes are flagged as needing a restart. An Apply to all button re-pushes on demand. voiceKeepInSync in the reconcile is the self-correcting backstop (config-diff based; re-applies on drift/enable). Proximity voice stays per-server (native SVC — you always hear nearby players on your own server). Cross-server voice GROUPS are bridged by the connector (ConduitVoice): when a player speaks while in an SVC group, their Opus frame is relayed over Redis to the other servers and injected to local players in a same-named group — so friends on different backends share a group. Group AUTO-SYNC (create once → joinable everywhere): every server publishes its SVC group roster over Redis; peers create mirror groups with the origin's UUID (persistent → joinable while empty, same type/hidden), pruned when the origin dies and nobody local remains. Event-driven + instant: CreateGroupEvent/RemoveGroupEvent push +/- deltas applied in ~ms (same-second mirror, <1s prune); the 2s roster tick is only the crash backstop. Password groups stay manual (the SVC API can't read passwords). Ops/debug: conduit voicetest create|remove|list from any console. Routing toggle: Via proxy (live-parity, default here) — the official SVC velocity plugin is installed on the network proxy (Modrinth, version-matched to the bukkit SVC), backends bind voice on their MC port (port=-1), and ONE UDP forward on the proxy's own port carries all voice (follows server switches); with multiple proxies a Voice proxies chip selector scopes which proxies carry voice (voice.proxyTaskIds, none = all). Direct — the original per-vmid forwarded ports. Separate halls per group (voice.hallsPerGroup): the relay + group-sync Redis channels are namespaced by the server's Conduit group (heartbeat voice.hall), so halls don't see/hear each other's groups. Clicking an already-active toggle is a no-op (no redundant re-apply). lib/voice.ts (endpointMode/routing/voiceProxyTasks), …/api/voice (applyVoiceToServers), provision.installVoiceChat(+Velocity), firewall UDP DNAT (lib/firewall.ts), engine.voiceKeepInSync, connector ConduitVoice.java (soft-depends the SVC API; heartbeat voice.crossServerGroups + voice.hall).

Firewall · node protection (/firewall) — the built-in "ufw": with forwarding on, a conduit:protect input chain drops internet-sourced NEW connections to node-terminated services (Proxmox UI 8006, node ports). All private source space (10/8, 172.16/12, 192.168/16 — LAN, cluster ring, WireGuard) stays fully open; SSH (22), ping, established traffic and every DNAT-forwarded game port keep working from anywhere — lockout-safe by construction (DNAT'd traffic never traverses the input hook). Toggle card on the firewall page (firewall.protectNodes, default on); part of the same atomic per-node ip conduit nft table.

Passkeys (WebAuthn) (/account + /login) — passwordless sign-in. Register any platform authenticator or security key on the account page (named, listed with created/last-used, synced badge for backed-up credentials, removable); the login page offers Sign in with a passkey (usernameless discoverable-credential flow) next to SSO. A passkey login skips TOTP by design — the authenticator's own user verification is the second factor — and issues the exact same device-registered session as a password login (sessions list, per-device revoke, login history entry with reason passkey). Challenges are single-use, 5-min TTL, parked in a global map; rpID/origin derive from the request host so the HTTPS domain just works. Insecure origins (plain http on the LAN IP) hide the login button and show a hint on the account card — browsers only run WebAuthn in secure contexts. @simplewebauthn/{server,browser} · lib/passkeys.ts · api/auth/passkeys{,/register,/login} · verified end-to-end on the live domain with a CDP virtual authenticator (register → sign out → passkey sign-in → session).

Resource packs · GitHub auto-pull (/packs) — a pack can name a GitHub repo as its source: the reconcile polls every 5 min and republishes automatically when a new push or release lands (manual "Pull now" too). Two publishing styles auto-detected: a repo whose latest release carries a .zip asset uses that build; otherwise the branch head zipball (repo root = pack root, pack.mcmeta required — GitHub's wrapper folder is stripped, .git* excluded). Same flow as a manual upload: stable /latest URL, sha1 bump, connectors resend on heartbeat. Failures surface as a red sync error on the pack card. Per-proxy targeting already exists: network-scope packs pick their proxies (or groups) in the target picker. lib/resource-packs-git.ts · engine.packsGitKeepInSync · pack fields gitUrl/gitBranch/gitSha.

Setup checklist (Overview → Get started) — a zero→production onboarding card that appears until the install is fully set up, then auto-hides (and is dismissible meanwhile). Eight steps computed from live config — deploy a server, connect an alert channel, SMTP, activate a domain + TLS, schedule backups, enable 2FA (reflects the signed-in user), invite a teammate, turn on the weekly digest — each todo links to the page that completes it. lib/setup.ts · GET /api/setup.

Fleet pulse (Overview hero) — the "is the network fine?" glance. A composite health score (0-100) per group AND network-wide, from signals the panel already has: Readiness (desired instances actually running + heartbeating — MC needs a fresh connector beat), Uptime 24h, Headroom (cpu/mem peak, penalized only above 75%), and Incidents (recent unresolved warn/error alerts matched to the group's task ids). Rendered as an SVG score ring + Healthy/Fair/Degraded/Down status + a trend chip (vs ~20 min ago, from an in-memory score ring) and per-group cards whose four dimension bars click through to /groups · /uptime · /activity. lib/pulse.ts · GET /api/pulse (any signed-in user). Weighted 0.5 readiness / 0.2 uptime / 0.15 headroom / 0.15 incidents; instance-weighted for the network overall.

Right-sizing (Overview → Right-sizing, appears only when actionable) — capacity advisor that compares each task's allocation to its real 7-day P95 cpu/mem (Proxmox weekly RRD, the task's worst instance sets the bar) and recommends reclaim/grow with one-click apply. Cores move both directions (cpu is a clean fraction of allotted cores). Memory is reclaim-only — LXC RRD mem is cache-inclusive (it fills free RAM, even reads above the cgroup limit), so a "high usage → grow" signal is unreliable; a starved server OOMs and trips the down/uptime watchdogs instead. Apply is safe: cores hot-plug onto the live CT, but a memory decrease is written to the task (effective next restart, when the JVM Xmx is recomputed) and is never forced onto a running server — shrinking the cgroup under a live -Xmx would OOM-kill it. Persisted on the task so it survives reprovision/clone. lib/rightsizing.ts · GET /api/rightsizing (any user) · POST (admin/operator).

Capacity forecasting. An Overview card projects, per Proxmox node, when CPU / memory / disk run out of headroom — a least-squares trend line fit to the 30-day node RRD gives the current utilisation, the trend per day, a 30-day projection, and a days-to-90% runway. Colour-coded bars (green→amber→red by urgency) with a faint projection extension, and a cluster headline naming the single soonest exhaustion so you add capacity before it bites. lib/forecast, /api/forecast.

Offline-node visibility. An offline Proxmox node drops its guests from /cluster/resources entirely — which would make its services silently vanish from the panel. discoverInstances() keeps a global last-seen CT cache and synthesizes node-offline ghost instances for cached CTs whose node is currently offline, so the Overview services table shows a red "node offline" status instead of nothing, and the Nodes strip renders the offline node as a red unreachable card (not empty bars). Ghosts are reconcile-inert (auto-restart keys on stopped) so nothing double-provisions while a node is away. Live updates on Overview ride the state/activity/alerts live-bus topics (instant) with the 5s poll as the safety net.

Automation · self-healing + resource watchdogs — event-driven rules (Settings → Automation). Triggers: service down (N consecutive failed checks — the original self-heal) · CPU high / memory high (any RUNNING instance of the task sustained ≥ threshold%, per-instance from /cluster/resources; cpu = % of the CT's allotted cores) · players high (task-wide online count from connector heartbeats sustained ≥ threshold — the pre-emptive "scale before it's full") · TPS low (any RUNNING MC instance's TPS from the connector heartbeat sustained ≤ threshold, default 15 — the lag detector; fires a 30s spark auto-profile before acting, like cpu/mem) · node disk full (a Proxmox node's rootfs ≥ threshold%; watches every node, alert-only by design). Actions: restart (mc only, least-disruptive) · alert only · scale up +1 (dynamic tasks: desired+1 clamped to max, the reconcile loop provisions it; degrades to an alert with the reason otherwise). Resource triggers use sustained windows (condition must hold continuously for N minutes — spikes don't fire; the window resets on every dip and after each fire) on top of the per-rule cooldown, opt-in per rule, and every action raises an alert. Evaluated on the leader tick after the uptime sweep; one /cluster/resources call per tick covers cpu+mem+disk. …/api/automation. lib/automation.ts.

TPS history (lib/tps-history.ts) — every connector heartbeat's TPS lands in a per-server minute-bucket ring (48h, in-memory on the leader; buckets keep the minute's MINIMUM — the worst sample is what lag looks like). Surfaces: a TPS card (current value + 60-min sparkline, green ≥18 / amber ≥15 / red <15) on the service page's Metrics tab (GET /api/services/:vmid/tps?min=), a colored N.N tps chip on groups instance rows (via /api/metrics), and the automation tps trigger above. Live-fired 2026-07-15: a ≤20 test rule alerted "TPS at 20.0 ≤ 20 for 1m" with a watchdog spark profile auto-started and collected.

Spark profiler (Containers page, admin/operator) — the one-click "why is the server lagging" answer. Right-click a running Paper server → Profile (spark) → 15s/30s/1m/2m (spark rejects ≤10s): the panel drives Paper's bundled spark through the tmux console (spark profiler start --timeout N via sendKeys — nothing to install on 1.21+), then collects the spark.lucko.me/<code> viewer URL from latest.log and lists runs (running/done/failed + who/when) in a Spark profiles panel on the same page; opening the page finalizes pending runs (no background job). Guarded end-to-end in startProfile: MC-only via the blueprint's software.kind (not role — limbo is role=lobby but NanoLimbo; Velocity doesn't bundle spark, live-tested), availability check (Paper ≥1.21 bundled, else the spark plugin must be in /opt/mc/plugins — actionable error pointing at the Plugins page), one-run-per-instance, and a baseline snapshot of pre-existing URLs at start so a stale console-run URL can never be mis-claimed as a result (no candidate → honest grace-fail after 3min). Watchdog tie-in: cpu/ mem automation hits auto-start a 30s profile on the exact instance BEFORE alerting, so the alert carries evidence. Runs persist in the store (capped 50). GET/POST /api/profiler. lib/profiler.ts. Live e2e 2026-07-14 (12 API checks + full browser path).

Compose apps (containerized tasks) — the generic egg runs ANY docker-compose app as a normal Conduit task: set custom.composeYaml on the template and provisioning installs Docker inside the CT (every LXC already runs with nesting=1), writes /opt/app/docker-compose.yml and supervises docker-compose up as the app unit — the Console tab streams the aggregated container logs, ExecStop performs a real docker-compose stop, and the {{port}}/{{memory}}/ {{vmid}}/{{name}} placeholders resolve per instance. Uptime, backups, domains custom-targets and the firewall all apply since it's a regular service. With managed databases (below) this is the complete self-hosting story: app + db in a few clicks. Live-verified end-to-end (whoami container served HTTP from a freshly provisioned CT).

Custom-template configurability & egg sharing — generic templates are real app recipes: apt packages → asset pulls → install script → supervised start command, now with placeholders ({{memory}} MB · {{port}} · {{vmid}} · {{name}}, resolved per instance at provision) and environment variables (custom.env → written to /etc/conduit/app.env, loaded by the app unit via EnvironmentFile; values take placeholders too). Eggs export/import as JSON: every card has an Export action (downloads <id>.egg.json) and the Templates header has Import (file-pick → POST /api/blueprints, which accepts the exported shape unchanged — share templates between Conduit installs). Creation/import validation is fail-fast with actionable errors (cores/memory/disk/port ranges, http(s)-only asset URLs, env key names, builtin-id collisions). e2e 2026-07-15: 5 validation cases + a full export → delete → import round-trip (env + placeholders intact).

Per-kind runtime identity (lib/runtime.ts) — what runs in a container and how to reach it, resolved per software kind instead of the historical everything-is-mc. Game kinds (paper/velocity/limbo/hytale) keep the classic mc unit+tmux. System kinds carry their REAL service unit (mariadb / redis-server / postgresql / nginx / minio) — automation self-heal, restart buttons and bulk/rolling restarts act on the actual service now (previously they restarted a cosmetic tmux shell on DB tasks and healed nothing) — plus an aliased console: a conduit-shell unit runs tmux -L mysql|redis|postgres|web|minio whose session drops straight into the native client (mysql · redis-cli with REDISCLI_AUTH · psql; exit lands in bash), and config paths are symlinked into the files sandbox (/opt/mysql → /etc/mysql, /opt/postgres, /opt/redis). Custom generic apps run as unit/socket app in /opt/app. Consumers all resolve through runtimeFor(kind) / runtimeForVmid(vmid) (15s memo): ops.sendKeys/restarts, console GET/POST/SSE, the browser WS console chain (/api/services/:vmid/agent returns sock, the console-proxy + node agent accept ?sock=, validated), bulk restart/rolling. Existing system CTs were migrated in place by runtimeAliasPass in the reconcile loop (idempotent, memoized; the real service is never restarted). Console sends use tmux load-buffer + paste-buffer — send-keys re-parses its args and treats ; as a command separator, which broke any SQL line (latent for MC too). Live-verified 2026-07-15: SELECT 42; executes in the mysql REPL through the panel console, redis PING→PONG authenticated, MC consoles unchanged.

Source-built software · upstream release tracking (lib/source-build.ts, lib/provision.ts, the Build pipeline card on a service page). A task's jar can come from a repository instead of a vendor feed — the reason being the seamless-transfer proxy, which needs a Velocity upstream does not publish. Two shapes: GitHub releases (CI builds and publishes, Conduit downloads — the tag plays the version line and the monotonic release id plays the build, so it slots into the same "newer build available" comparison Paper uses) and a git checkout built on a build host into S3, keyed by commit. Nothing compiles in the cluster either way. Gated entirely on a GitHub token: with none the blueprints are filtered out and the card renders nothing, rather than offering something that can only fail. Contents: Read is the minimum, Actions: Read adds the runs list, and Contents: Write — not Actions: Write — enables the watcher below to trigger a build. GitHub files POST /repos/{owner}/{repo}/dispatches under the Contents permission despite it starting a workflow; the endpoint's own reference page doesn't say so, only the fine-grained permissions table does, and setting Actions: Write instead leaves the dispatch failing with a 403 that names no permission at all.

The token is set in the UI (Settings → Data & Credentials → GitHub token, collapsed by default since most networks never need one) and lives in the store as network.github.token, alongside the Discord and mail secrets — not in the panel environment. That matters because /etc/conduit/panel.env is rewritten from scratch by a full deploy-panel.sh, so a token pasted there by hand vanished on the next deploy and the only symptom was the whole feature quietly disappearing. The store is pmxcfs-replicated, so one paste covers all three replicas and survives updates. Saving verifies the token against GitHub first and refuses one it rejects — an expired PAT is otherwise indistinguishable from an absent one — and the card reports which account it belongs to. The CONDUIT_GITHUB_TOKEN env var is still honoured as a fallback for an existing install, with the stored one winning; and a full deploy now preserves unmanaged keys in panel.env rather than dropping them. githubToken() in lib/source-build.ts is the single resolution point; the token is never returned by any API (/api/github reports only whether one is set and whether it still works).

The tracking is the interesting half, because Velocity is genuinely hard to watch: it publishes no git tags and no GitHub releases, so there is no ref that means "the current release" and no event to subscribe to (a webhook needs a repository you own; GitHub has no cross-repo release event). What it has is PaperMC's Fill API, which groups versions into families and — the part that makes this work — names the exact upstream commit behind every build. So a release resolves to a commit, and upstreamPolicy() reads the source repo's own upstream.txt policy (track=3 family=3.0.0 channel=recommended) and resolves it the same way that repo's CI does. upstreamWatchPass (engine, 5-minute throttle) then dispatches a build for any tracked line whose release commit has no release yet. The release tags are the state — they encode (track, patch commit, upstream commit) — so there is no bookkeeping to keep in sync and a restart mid-build costs at most one redundant dispatch, which CI's own gate declines. Without the token's write scope the dispatch fails, is logged, and CI's 6-hourly cron still covers it.

Several release lines share one feed and are selected by asset-name glob alone (velocity-seam-v3-*.jar): a release with no matching asset drops out of the list entirely, so following one line needs no new config. Each release carries its facts in a <!-- conduit --> block in the body — version, upstream commit, required JRE, whether the patches were verified — and runJava() reads the JRE from the artifact, not the blueprint, because Velocity 4.x needs 25 where 3.x needs 21 and a blueprint's javaMajor is a human's memory of what was true when it was written. Build failures are classified per step (githubRunFailure) so the runs band reads failure · patch — a patch someone must rewrite — rather than an undifferentiated failure, and raise source.patch.stale / source.build.failed through the normal alert channels, suppressed for 24 h per upstream commit (a new release is a new problem; the 6-hourly retry of the same one is not). CI itself gates a publish on the jar booting and on an effect assertion that the patches still reach compiled, called code — a patch that applies cleanly into code upstream no longer calls otherwise ships as an inert jar that looks identical to a working one. That assertion covers every hook a patch has, not one: the seamless patch hooks three places, and the hook that is most obvious to assert is the one that would keep passing if the other two stopped applying. Both release lines are built from the same patch file (patches/v4/ symlinks patches/v3/), so track 4 is a standing answer to "does this still apply to the next major" rather than a copy that drifts.

Deliberately not automatic on the proxy: autoUpdatePass skips instances with players online, so on the one component every player traverses it would fire only when the proxy happened to be empty and drop whoever arrived mid-restart. The panel surfaces "newer build ready"; the rollout is a human decision through the rolling-update path.

Audit / SIEM export (Settings → Data & Credentials → Audit export, admin) — stream the operator audit trail (every panel mutation: user, role, method, path, action, status, ip) to an external HTTP collector as JSON, for long-term retention or SIEM ingestion (Splunk HEC, Elastic, a Logstash HTTP input, a generic webhook). The leader POSTs new entries each tick; the cursor seeds to the current newest id on enable/restart so it streams from NOW forward and never re-floods the backlog. A "Send test event" button posts a probe. lib/siem.ts · GET/POST /api/siem.

Event windows (/schedules → Event windows, admin) — the positive twin of maintenance windows for hyping a double-XP weekend, drop party or season launch. Schedule one and the leader drives it: T-30/15/10/5/1-min countdown broadcasts → at start it goes live (optional proxy MOTD flair swapped in, connector-applied) with an "event live" fuchsia banner on the public status page → at end it auto-restores the MOTDs exactly. Never worsens the overall status (it's positive flair); the panel does the announcing, staff run the mechanics. lib/event-windows.ts (mirrors maintenance-windows) · GET/POST/DELETE /api/event-windows · status API event/ eventUpcoming.

Weekly digest (Settings → Alerts → Weekly digest) — a scheduled network summary pushed through the same channels as alerts (Discord / ntfy / webhook / email). Composed from what the panel already computes: unique/peak players + playtime + top gamemode + busiest hour (from the analytics heatmap), 7-day avg uptime + the lowest-uptime service, incident count, the fleet-pulse score, and TLS-cert days-left. The leader fires it once per week at the configured weekday/hour (server tz) via weeklyDigestTick — a last-occurrence guard means it sends exactly once per slot and never double-fires. The card shows a live preview of this week's digest + a Send digest now button. lib/digest.ts (+ deliverBroadcast in lib/alerts) · GET/POST /api/digest. (HTTP header values must be ASCII — the ntfy Title strips emoji/non-latin1 so fetch can't throw; the same hardening was applied to the alert ntfy path.)

Webhooks (/hooks, Developer, admin). Inbound URLs that fire a panel action — turn external events into Conduit operations. Each hook has an unguessable token (POST /api/hooks/<token>, the URL IS the auth like a GitHub/Jenkins hook), an action (restart a task/group · scale-up/down a dynamic task · broadcast · reconcile now), an optional HMAC secret (require a valid X-Hub-Signature-256 (GitHub) or X-Conduit-Signature over the body) and a per-hook cooldown. The endpoint is public (isPublicPath; external senders have no session); GET returns a harmless info ping, POST fires. Every fire records hits/last-fired, raises a webhook.fired alert + event. Admin CRUD + a manual test-fire (ignores cooldown) on the page. lib/webhooks.ts reuses the scheduler's ops helpers. e2e 15/15 (token auth, GitHub HMAC valid/invalid, cooldown 429, disable 403, unknown 404, scale/restart target validation, admin-only CRUD). Example: point GitHub's push webhook (with the repo secret) at a restart dev-group hook → auto-redeploy on push.

Maintenance windows (/schedules → Maintenance windows, admin). Announce downtime like a pro: schedule a window (title, group or whole network, start + duration, message, optional maintenance MOTD) and the leader drives the lifecycle — countdown broadcasts to the targeted servers at T-30/15/10/5/1 min (only the closest due mark announces; marks persist across ticks/failover), at start the targeted groups flip maintenance=true (prior state captured), their proxy tasks optionally get the maintenance MOTD (connector applies live), a warn alert fires and the public status page shows an amber banner (overall becomes maintenance — planned work, not an incident; an upcoming window ≤24h shows a heads-up row). At start+duration everything auto-lifts: flags/MOTDs restored exactly (a manual admin change mid-window wins), end broadcast

• resolved alert. Cancelling an active window restores immediately. /api/maintenance-windows (GET/POST/DELETE=cancel), engine lib/maintenance-windows.ts. Live-fire verified end-to-end (countdowns → flag+alert → auto-lift+restore).

Player analytics (/analytics, Players category). Joins, unique players, playtime, peaks and retention aggregated from the player audit trail (so the horizon = the audit retention setting): range switch 7/14/30d, a connected KPI band (uniques · joins · playtime · avg session · peak online · returning-today %), a daily joins/uniques bar chart with per-day peak tooltips, playtime by server (proxy instance-ids and backend short-names normalized onto task display names and merged) and a top-players board (rank, sessions, playtime). The aggregator (lib/audit auditAnalytics) dedupes the proxy+backend double-reports (same player+type within 2.5s, keeping the richer entry), reconstructs sessions join→switch→quit with a 6h runaway cap, attributes spans per server, and sweeps concurrency for daily peaks. GET /api/analytics?days= (60s cache; players token scope). Math verified against a synthetic trail (dedupe, spans, still-online close-out, peak) + rendered page e2e. A busy-hours heatmap (weekday×hour avg-concurrent, from the session spans — presence-minutes per cell normalised by how often that weekday fell in range, tz-labelled) shows when the network is actually busy for event/autoscale/maintenance timing.

Appearance — per-browser accent theme (Settings → Appearance): pick the --brand accent colour. Service rows show an auto role/kind glyph (proxy/lobby/smp/game/db/web) so you see what's running at a glance.

• Audit (DSGVO) — per-player history (joins/quits/switches/operator actions), retention window, hard erase. …/api/audit.
• Consoles + files — live tmux console stream + in-container file manager per service; shared store file API. Plus a Config tab (service page) — drift detection: every text config the template overlay chain defines is tagged in-sync / drifted / missing vs the live file, with a green/red inline diff and one-click Restore-from-template · Save-to-live · Save-to-template (adopt the live edit into the task overlay). lib/config-drift, /api/services/:vmid/config.
• Multi-console (/consoles, Monitoring) — the incident view: pick up to 4 running servers and watch their consoles side by side (proxy + lobby + a backend at once). Each tile is a lightweight ANSI-rendered live tail off the same SSE stream (base64 pane snapshot per frame → UTF-8 → HTML), far cheaper than N xterm canvases; a shared input bar with a broadcast toggle (all open consoles, or the focused one) sends a command to every target at once (admin/operator; the console POST enforces it). Verified a broadcast reaching two backends on different nodes at the same instant. …/api/services/:vmid/…, …/api/files.
• Honest lifecycle labels + auto-restart-aware power menus — an instance whose CT runs but whose server isn't heartbeating shows what's actually happening: the actor's last action ("rebooting… / shutting down… / restarting… / starting…", per-browser memory, 150s TTL), else "booting…" on a fresh CT (uptime < 2min), else "not responding" — never a blanket "restarting…". And wherever a task's Auto-restart is on (default; the reconciler restarts stopped instances), every Shutdown control (groups instance/bulk menus + bulk bar + task Stop server, containers hover + context menu) is disabled with the reason "auto-restart on" instead of pretending to work — Force stop stays as the destructive escape hatch. /api/containers rows carry managed/autoRestart for this.
• System credentials — Redis/Postgres/MariaDB/MinIO passwords derived from the network secret, rotatable with automatic re-sync to all consumers. …/api/system-credentials.
• DB browser — read/write Postgres explorer + read-only Redis browser. Plus managed databases (DBaaS): one-click provision an isolated database + scoped login on the running system MariaDB/Postgres — copyable connection string, admin-drop with full cleanup; each tenant is GRANTed only its own database (verified isolation). lib/user-databases, /api/databases. …/api/db/pg, …/api/db/redis. The page opens with a system-engines band: live connection usage per engine (used/max with a pressure gauge, lib/db-stats) — pool squeezes are visible before they surface as "connection slots reserved" in a server log. Postgres provisioning sizes max_connections/shared_buffers dynamically from the container's memory, and the LuckPerms installer keeps every backend on a small (4) connection pool, healing value drift each reconcile pass.

Safety: the engine only touches VMIDs 200–999 tagged conduit; persistent instances are never auto-destroyed; the store is fail-closed (refuses to start from an empty/corrupt conduit.json).

---