Development

How to work on Conduit without breaking a live network.

Ground rules#

1. Verify before you ship. Every change is proven against the running system — panel features through the browser (screenshots, DOM assertions), connector changes through live server logs, infrastructure through the actual nodes. "It compiles" is not done.
2. The implementation is the source of truth. Docs describe what exists; if they disagree with code, fix the docs or the code — never leave both.
3. Design system is law. Every panel surface follows the Cloudflare school: flat panels, hairline borders, band layouts, the one brand blue, entrance animations via the shared motion primitives, motion-safe gating. No drop shadows, no glow.

Components#

Panel (dashboard/)#

cd dashboard
npm install
CONDUIT_CONTROLLER=off npm run dev   # UI/API without touching the cluster
npx tsc --noEmit                     # types
npm run build                        # production build (Turbopack)

• App Router, React 19, Tailwind 4 (OKLCh tokens in src/app/globals.css).
• Shared UI in src/components (motion primitives in components/motion).
• All server logic in src/lib — the reconcile engine (engine.ts), store (store.ts), Proxmox client, firewall renderer, and ~40 focused modules.
• Gotcha: cross-route library state must live on global.__conduit* — Next bundles libs per route, so module-local singletons silently split per bundle.

Connector (plugin/)#

bash plugin/build.sh     # plain javac + jar; fetches API jars into plugin/lib once

One jar serves Paper and Velocity (entry points paper/ConduitPaperPlugin.java, velocity/ConduitVelocityPlugin.java). Soft-depends (Skript, LuckPerms, voicechat) are compile-only — never shaded. Deploy to a live fleet via install/update.sh (builds the panel, snapshots the store, ships bundle + jar into the node stores, verifies every replica; scripts/upgrade-live.sh remains as a shim). From there a reconcile pass stages the canonical jar on any instance whose heartbeat reports an older build — activation still waits for a restart, which stays a human decision; the Plugins page shows exactly who is pending. The lesson behind that pass: the jar used to reach containers only at provision time, and the fleet quietly ran three different month-old builds while every deploy staged a new one.

When a heartbeat field you added doesn't arrive, read ConduitClient's serializer before theorizing — the player-map serializer once whitelisted three keys and silently dropped every field added after it, which kept a whole panel feature dead with zero log lines.

Two versions in build.sh are worth understanding before changing them:

• JAVA_TARGET (21) is the bytecode level, and because the same jar loads on both platforms it can only be as new as the oldest JRE in the fleet. JAVA_FLOOR in lib/provision.ts is the other half of that contract — it stops any service being provisioned below the target. Raise them together, and only once every running instance is on the newer JRE.
• PAPER_MC is the API compiled against, and it should track what the fleet runs. Mojang renames constants between minor versions, so an older API pins names that no longer exist at runtime — the jar builds fine and then throws NoSuchFieldError on first use. Constants known to have moved live in paper/Compat.java, resolved by name so both the old and new spelling work; anything it can't find is logged at load and the dependent behaviour turns itself off.

GameRule.RANDOM_TICK_SPEED, DO_DAYLIGHT_CYCLE and DO_MOB_SPAWNING (used by the minigame world setup) are deprecated for removal upstream. They still exist on 1.21.11, so they only warn, but they will need the Compat treatment or a registry lookup whenever Paper actually drops them.

Software built from source (velocity-source, builder)#

Some software has no upstream build you can use — a patched proxy, most obviously. A blueprint can declare software.source = { type:"git", repo, ref, buildCmd, artifact } instead of relying on a vendor feed, and the panel then tracks it like any other version:

Two things differ from a vendor feed, both deliberate:

• Builds compare by identity, not order. Vendor build numbers only increase, so latest > installed is safe. A ref can be force-pushed or reverted backwards and that is still a change worth rolling out, so the test is latestSha !== installedSha.
• Artifacts are content-addressed by commit (conduit/builds/<name>/<sha>.jar in object storage). Rebuilding an unmoved ref is free, rollback is picking an older sha, and the sha recorded on the task answers "what is actually running" long after the fact.

Compiling happens on a build server — an ordinary task from the builder blueprint (generic kind, git + JDK, no start command, idle between builds). It is not done on the panel: a gradle build wants a JDK and a core for minutes, and neither belongs in the control plane. The panel resolves refs over plain HTTPS using git's smart-HTTP ref advertisement (no git binary, no clone), then hands the builder a presigned PUT for exactly one object — so the builder holds no bucket credentials, and provisioning pulls the jar with a presigned GET. (The in-game map studio uploads worlds the same way: the connector asks the panel, gets a presigned PUT for exactly one tarball, and the panel verifies the object exists before registering the map — no S3 credentials ever reach a game server.)

Because it is a fork, the honest cost stands: re-patch per upstream release, re-validate per protocol bump, on the component every player connects through. Keep it per-task, prove it on a test proxy, and remember the stock jar is one PATCH away (drop source and the Fill resolver returns).

Agents (agent/)#

Two of them. agent/src/index.mjs is the node agent: plain Node ≥ 18, no build step, shipped by scripts/deploy-agent.sh, systemd unit conduit-agent. Bump its VERSION constant when the protocol changes. Also here: console-proxy.mjs (the panel CTs' terminal bridge) and cli-gateway.sh (the SSH gateway's forced command).

agent/ctagent/ is the container agent — single-file Python 3, standard library only, no build step and no dependencies to install. It travels inside the panel bundle (see install/modules/bundle.sh) and a reconcile pass pushes it into containers, so shipping a change means bumping VERSION in the .py and CTAGENT_VERSION in dashboard/src/lib/ctagent.ts to match. They are compared on every probe: a mismatch makes the panel treat that agent as absent and fall back to pct, which is what makes a rollout safe mid-flight — and it is also what upgrades the fleet, since a skewed agent is reinstalled.

CLI (cli/)#

Zero-dependency Node ESM; no build step. cli/conduit is a CJS-safe bootstrap (the panel replicas run Node 20, which parses extensionless files as CommonJS) that hands to cli/main.mjs; commands are auto-discovered from cli/commands/*.mjs — drop a file in, default-export { name, summary, usage, run }, done. The panel bundle ships the whole directory for the SSH gateway.

Installer (install/)#

Plain bash, bash -n clean. Entry points share lib/ (ui, core, detect, checks) and modules/ wrap the proven scripts/; topology comes from install/cluster.conf (gitignored — cluster.conf.example is the template). install/update.sh is the standard deploy path and doubles as the panel's TypeScript gate.

Skript content (skript/)#

Ported game content and the glue that bridges it to the connector's Skript API. Start with skript/conduit-games/PORTING-GUIDE.md.

Six rules about silence#

Every one of these came from a fault that ran for weeks or months in production without anything reporting a problem. They are here because the code enforcing them is spread across a dozen files, and the reasoning is worth more than any single guard.

A swallowed read must not become a recorded verdict. await thing().catch(() => []) stops one failing call taking a whole page down, and is right more often than not — the empty array it produces is also indistinguishable from a genuine empty answer. That is fine where the result is rendered and forgotten. It is not fine where something writes the result down. backupStorages() had always failed; the restore drills caught it, found no archive for any container, and recorded "no vzdump backup exists yet" — a verdict deliberately classified as benign. The thing whose only job is proving backups restorable was blind for three months and its blindness was filed as health. Use lib/readable where a read feeds a verdict, and return an outcome that records nothing when you could not look.

A derived view may decide what is shown; it must never hide or delete what somebody wrote. Deriving the per-kind overlay list from the running fleet is the obvious design and it is wrong: delete the last Postgres server and a whole layer of configuration vanishes from the UI while the files sit on disk, ready to apply again. Derive as a union of live state and what already exists.

A verdict type needs a state for "I cannot tell". With only good/bad to choose from, uncertainty gets encoded as one of them, and it is always the benign one. Coverage taints its reasons when S3 is unreachable; the drills have unasked; the secrets register shows "never recorded" rather than inventing a date.

Plant a known-bad before trusting a green check. Used about a dozen times in one day and it caught its own detector being wrong four times — a regex anchored to line starts that missed every inline field, a 400-character window that truncated before the evidence, a toContain that matched its own doc comment, and an ordering check comparing positions in an import line. A check that has never been watched to fail is worth nothing.

A grep negative is not evidence of absence. serviceDir looked absent because the copy was private in another file; a whole secrets inventory looked absent because it lived under a name the search did not cover. Two independent sources beat one clever query.

Refusals are not failures. A 403 for a role that may not do something, or a 404 for a name that does not exist, is the endpoint working. Counting those as failures produces an alert that fires during normal use, and an alert that fires during normal use is one people learn to close.

Testing#

cd dashboard
npm test        # vitest — the invariant suite, ~1s

The suite covers logic where a regression would be quiet and expensive: the store's compare-and-swap under a two-instance race (vi.resetModules() + a globalThis-backed store gives two "replicas" in one process), the offsite crypto truncation matrix, the VIP chooser's stickiness and the fast-failover swap planner, backup coverage verdicts, byte-capped tails, the drift normaliser, node-maintenance window semantics, the secret-parity extraction (which runs the real shell pipeline through bash rather than a TypeScript copy that would drift).

A few tests assert things about the source rather than about behaviour, because the bug they guard against has no failing behaviour to catch — it produces silence:

• agent-deadlines.test.ts — no fetch in the agent client without a deadline. A hang raises nothing, logs nothing and returns never; there is no assertion that fails, only a control loop that stops. Four calls had no timeout, two of them the ones getDB() uses, which is how a panel answered /api/health in 0.7 ms with an eleven-minute-old reconcile tick.
• reachable-flags.test.ts — every optional boolean on Task that engine.ts branches on must be assigned by some route. Twice in one day a feature was gated on a field nothing could write: /api/network silently dropped geyserTargets while answering {"ok":true}, and drainOnUpdate was read by the reconcile loop and named in its own log message with no route anywhere. Every unit was correct; the seam between them was not.

A detector that has never failed is a detector whose pass means nothing. Both of the above carry a planted-failure case that feeds them a deliberately broken input and asserts they notice. That is not ceremony: the plant in reachable-flags immediately caught that t.replay === true matched the "is this assigned" regex through the first = of ===, so a flag that was only ever compared looked wired up.

Write new invariants as *.test.ts beside the lib they test — not as throwaway harnesses. CI (GitHub Actions) runs typecheck + the suite on every push to both branches. The suite complements, not replaces, the verification culture above: live behavior is still proven live.

Conventions#

• Commits: plain human prose. No AI-styled attributions ("user asked…"), no all-caps emphasis, no co-author trailers. Subjects in the conventional area: what changed shape where it helps.
• Branches: work lands on master; feat/conduit-ha-fleet-platform mirrors it. Direct pushes — no pull requests.
• Comments: explain why, at the density the surrounding file already uses. Write them as prose, not emphasis — no shouted words (NEVER, CRITICAL, ONLY) and no NOTE:/IMPORTANT: labels; if a point matters, say why it matters. Acronyms (API, UDP, RESP) and literals (WRONGPASS) stay uppercase, obviously. Skip comments that only restate the line below them.
• Docs: user-visible behavior changes update FEATURES.md and the relevant docs/ page in the same commit.