Architecture review — 2026-08-06

A structural read of the panel ahead of V1, measured against the live cluster rather than inferred from the code alone. Findings are ordered by likelihood × blast radius. Where the design is sound it says so; the point of a review is not to produce a list.

Everything below was checked against the running system. Where a claim is reasoning rather than measurement, it says which.

1. The store is 98% one key, and it is not the one you would guess#

Measured, /etc/pve/conduit/conduit.json, 835.2 KB:

The orchestration state this whole system exists to manage — every task, group, route and schedule — is under 25 KB. The other 97% is three language catalogs.

This matters because the store lives on pmxcfs, the corosync-replicated filesystem. Every write serialises the entire document and replicates it to all three nodes. That is the right home for 25 KB of cluster-critical configuration and the wrong home for 800 KB of translation strings, which change rarely, are not cluster-critical, and are already the kind of thing /var/lib/conduit (GlusterFS) exists to hold.

Recommendation. Move languages out of the corosync store and onto the shared volume, leaving the store as what it claims to be. This is the single highest-leverage change in this document: it makes every store write ~33× smaller, and it removes the only component whose growth is driven by content rather than by fleet size.

Done 2026-08-10 — lib/lang-store.ts, migrated live by the leader's boot pass. The store went 850,691 → ~100,000 bytes, and all five language-serving API responses hashed byte-identical before and after (after fixing a pre-existing agent readBody bug the comparison itself uncovered: bodies were decoded per stream chunk, so a multi-byte character split across chunks became U+FFFD).

Do not generalise this into "prune the store". See the next finding.

2. The growth shape is better than it looks — the keyed maps are bounded by fleet, not time#

It is easy to read drills, jobHealth and breadcrumbs as append-only logs that grow forever. They are not. Each is a map keyed by identity — strip:<task>:<index>, the job key, the service id — so a new entry appears only when a new thing appears, and re-running overwrites in place. Their ceiling is the size of the fleet, not the age of the cluster.

Measured: 8 drill records for 26 candidates, 24 job-health records for 24 jobs. A cluster running for five years has the same number as one running five days.

The genuinely unbounded structures — activity events, operator audit, alerts — are already capped and already live outside the store, flushed to the shared volume with ring buffers in front.

One number does deserve attention: breadcrumbs are ~8 KB each. Keyed by service, so 21 services is ~170 KB, which is fine — but it is the one keyed map where a single entry is large enough that a much bigger fleet would be felt. Worth a size cap on the captured log tail rather than a count cap.

Done 2026-08-10 — the tail is byte-capped at 6 KB per breadcrumb, newest lines kept and counted in real bytes (a stack trace full of multi-byte characters is exactly when a character count lies). Proven live: a demo-server restart posted 80 captured lines and the store kept 66 at 6,100 bytes. The pathological case — a hundred 300-char lines, 30 KB — now lands at 20 lines.

3. mutate() has no cross-replica coordination#

Verified in lib/store.ts. The write path is:

const db = await ensure();     // read
const result = fn(db);         // modify
await saveDB(db);              // write

There is no version, no compare-and-swap, no lease. Writes are serialised within a process, but all three replicas serve HTTP and all three can write. Two writes that interleave read-modify-write lose one of them silently, and the loser is not told.

How much this matters in practice is smaller than it sounds, for three reasons that are worth stating rather than assuming:

• the reconcile loop only runs on the VIP holder, so the highest-frequency writer is a single process;
• reconcile writes are re-assertions of desired state, so losing one is corrected on the next tick;
• operator writes are human-paced and rare.

So the exposure is specifically: two humans on two replicas, or a human racing the leader tick, in a window of roughly a hundred milliseconds. That is unlikely, not impossible, and it fails silently, which is the part worth fixing.

Recommendation. Add a monotonic rev to the document. mutate() reads it, and saveDB refuses the write if rev changed underneath, retrying the callback against fresh state. That converts a silent lost update into a retry, costs one integer, and needs no locking. It also gives the offsite copy and the restore path a version to talk about.

Done 2026-08-10 — the agent compares x-conduit-if-rev inside its write queue (the single serialisation point) and answers 409; mutate() replays against fresh state up to 5×. Rollout is order-free: an old agent ignores the header, a headerless PUT is accepted. Proven three ways: an offline two-instance race harness (21 induced conflicts, zero lost updates, rev exact), a live parallel same-rev race on the agent (exactly one 200, one 409), and the live store advancing rev-per-write under the leader tick with clean journals.

4. The leader tick is a single sequential chain#

~24 jobs run one after another inside one setInterval, guarded against re-entry, and the loop "routinely takes half a minute" by its own comment. Two consequences follow:

• One slow job delays every job behind it. This already bit: the storage watch had to be self-throttled to five minutes precisely because a gluster call inside the chain would otherwise extend the tick for everything else. runJob records duration, so the evidence is already being collected — it is simply not acted upon.
• The recorded unit is the invocation, not the work. A self-throttled job reports "ok, 0 ms" on every tick it declines to run, which reads as healthy and is (see conduit-job-health).

This is not urgent — the jobs are individually cheap and the guard prevents pile-up. But the pattern does not survive many more jobs, and the fix is small: give the slow-and-infrequent jobs their own interval, as the hub watch already has, rather than adding them to the chain.

5. Module structure is in good shape#

Checked and found genuinely healthy, which is worth recording so it is not re-litigated:

• No import cycles. The lib/cache.ts extraction that exists specifically to break one is still doing its job, and the backup-coverage → restore-drills edge added today does not create a new one (restore-drills does not import back).
• Every global.__conduit* is legitimately global. All of them are cross-route runtime state that would be a different instance per Next bundle if module-local. No stragglers. (There were ~25 when this was written and there are 88 now, which is growth in the fleet's surface rather than in the pattern — and it is what makes GET /api/diagnostics/memory possible, since the whole retained set is enumerable.)
• The server/client boundary is clean. No client component imports a lib module that reaches the store, the Proxmox API, or node: builtins.

The two files worth watching are engine.ts (2273 lines) and provision.ts (2158). provision.ts is long but coherent — one install recipe per function. engine.ts is the one that is starting to do orchestration and per-software provisioning detail; the next new software kind is the moment to extract a provider registry rather than add another branch.

6. Single points of failure, stated plainly#

• The hub is one LXC. If it dies, no installation can see updates. They keep running, so this is correctly a low-severity SPOF — but it is now internet-facing and worth a health check of its own.
• MinIO (CT 210) holds strip snapshots and Time Machine captures on one container's disk. It is backed up nightly, but there is no second copy of the object store itself.
• PBS was the acute one and is now materially better: its datastore has its own disk, so a full store can still write task logs and clean itself up. The remaining exposure is that it is one box in one building — which is exactly what the offsite copy addresses for the store, and does not address for the archives.

7. Should the control plane be its own process — or a different language? (2026-08-15)#

Asked directly: is the Next.js API fine, or should the panel become something "detached, like Rust"? The reason behind the question turned out to be availability — after a node reboot the API was not immediately back — so the answer has two halves.

The availability half was not architectural. Every replica's nginx proxied to 127.0.0.1:3001 and nowhere else, so while its own panel restarted it served 502s with two healthy panels sitting on the other nodes. Measured on the public host at five probes a second, with the VIP holder's panel stopped for twelve seconds: 23 failed requests before, 0 after, and 0 through a full update.sh deploy. That was a nginx upstream, not a rewrite.

Do not rewrite in Rust. The numbers point elsewhere: API routes answer in 70–100 ms, the VIP health check in 4 ms in-cluster, and the floor under anything that must enter a container is pct exec plus the Proxmox API. Those are I/O costs; a faster language does not move them. The August outage was a memory leak, and leaks happen in every language — a rewrite would trade this failure mode for different ones, at the cost of ~30,000 lines that currently run a live network.

That paragraph used to say the floor was under every real operation, and 2026-08-17 disproved it in the most useful direction: most of what the reconcile loop was doing did not need to enter a container at all. pct is a Perl program carrying the whole PVE library stack — 0.40 s and ~130 MB per call, the second number never having been measured before — and the passes were spending it on "does this jar exist" and "what does this properties file say", once per plugin per instance. Under that load a node agent's madvise triggered a TLB shootdown that spun 22 s and the guest was reset, twice in two days. Routing those reads through the container agent's existing typed fs operations (lib/ct-inspect.ts) took the fleet from 97 to 9 pct exec calls per 300 s and the heaviest ssh session's peak from 850 MB to ~118 MB. The lesson generalises past this codebase: before optimising the expensive call, count how many of them are asking a question that does not need it.

Splitting the controller into its own process is defensible, and cheaper than it looks. instrumentation.ts, engine.ts, store.ts, connector.ts, provision.ts and firewall.ts import nothing from next/ or react — checked, not assumed. The precedent exists twice over: console-proxy.mjs and discord-bot.mjs already ship from the same bundle as their own systemd units. The store is HTTP plus revision CAS, so a second writer on one host is already the normal case across three replicas.

Two honest complications, which is why this stays a recommendation rather than a task. Connector heartbeats arrive at a web route while the controller consumes them, so that ingest would have to move with it — arguably correct anyway, since machine traffic and operator traffic are different surfaces. And the in-memory live-bus that drives SSE would need a notify path across the boundary.

Judge it after the availability fix has had time to prove itself. Most of what the split would have bought is already banked.

Suggested order#

1. Move languages out of the corosync store. Biggest effect, lowest risk, no behaviour change. Done 2026-08-10.
2. Add rev + retry to mutate(). Turns a silent failure mode into a retried one. Done 2026-08-10.
3. Cap the breadcrumb log tail by bytes. Small, and the one keyed map with a large per-entry cost. Done 2026-08-10.
4. Split the slow jobs out of the reconcile chain when the next one is added, not before. Done 2026-08-15 — six of them (world capture, restore drills, database dumps, shared-volume capture, storage watch, offsite) moved to their own minute interval; 24 stayed on the chain.
5. Extract a provider registry from engine.ts when the next software kind arrives.

Nothing here is on fire. Items 1 and 2 are the ones that would be embarrassing to explain after the fact; the rest are shape, not risk.

Nodes at different providers (measured 2026-08-15)#

The question is whether a second provider joins the Proxmox cluster or stays its own. Measured on the real path — a LAN node to the 24fire VPS:

That is a good internet link and still sixty times the LAN. Do not extend the Proxmox cluster over it. Corosync can survive 22 ms; /etc/pve cannot enjoy it — pmxcfs is a synchronously replicated filesystem where every write is a cluster-wide transaction, and Conduit writes the store on every mutate(). Worse than the latency is the failure mode: a provider's maintenance window or a BGP reroute becomes a fencing event, and HA starts moving containers on both sides of a partition. The day this was measured a node died and the cluster handled it cleanly because it was quorate at 2/3 on a sub-millisecond LAN. A WAN member makes that worse, not better.

So: one Proxmox cluster per provider, federated above them by Conduit.

Two pieces of that already exist, which is the encouraging part:

• The store is already remote-capable. The live panels run CONDUIT_STATE_BACKEND=agent against CONDUIT_STATE_AGENT=10.0.0.103 — HTTP with revision CAS, not pmxcfs reads. A second site points at the same agent. Writes are infrequent, so the round trip is affordable.
• The addressing is already unique per node. A /24 per node out of 10.42.0.0/16, address matching vmid, does not collide across providers any more than across a rack.

Proven end to end over a WireGuard link (10.43.0.0/30, edge-02 listening, the LAN node dialling out):

• LAN node → a container on the VPS by its managed address: 23 ms, 0% loss
• LAN node → that container's Minecraft port: 25565 open cross-provider
• VPS → the cluster's state agent: HTTP 401 in 47 ms — connection good, service answering, correctly refusing an unauthenticated caller

AllowedIPs is the access-control list, and it is deliberately narrow: the VPS can reach exactly 10.0.0.103, not the LAN. Widen it only for hosts that need to be reachable.

What still has to be written, cheapest first:

1. lib/proxmox.ts learns about sites — one endpoint and one token become a map. This is the real refactor; everything calling api.nodes(), pct exec and nodeIp() goes through it.
2. A site field on Task/Instance, and site-aware placement (the node pin/allow/deny idea, one level up).
3. Per-site entry points. keepalived needs L2, so each site keeps its own VIP and proxy; DNS or a routing tier decides which a player reaches.
4. Site-aware UI.

What cannot cross and should not be made to: GlusterFS (/var/lib/conduit stays site-local, S3 for anything shared), keepalived, and live migration (no shared storage — offline moves only).

One rule for players: a session stays inside one site. 22 ms added to every packet is fine for a lobby and wrong for PvP. Sites are for capacity and redundancy, not for spreading one world.