2e. The panel's own health, and its own network#

Two things Conduit did not watch or provide for itself until August 2026.

The process hosting everything. The job-health layer covers ~30 recurring jobs, but the panel process itself was uncovered — on 14 August the replica holding the VIP was killed by V8 with ~1 GB of live objects and raised nothing before, during or after. lib/heap-watch.ts now warns at 80% of the ceiling V8 itself reports (it derives from machine RAM, so it differs per node and a hardcoded number would be wrong exactly when it mattered), with hysteresis on the way back down, and runs before the leader gate — a passive backup quietly eating its ceiling is the replica the VIP is about to move to. It reports only: a panel that restarts itself on a memory reading is a worse failure than one sitting near its limit. GET /api/diagnostics/memory answers "holding what?" in milliseconds without pausing anything, and the unit runs node with --heapsnapshot-signal=SIGUSR2 for when a real snapshot is wanted (it freezes that replica for ~20s, so never the VIP holder).

The peak survives the process now, which is why this stayed unsolved so long. The reading was kept in global.__conduitHeapWatch — the heap of the process V8 kills — so the one event worth explaining destroyed its own evidence, and every restart put the record back to zero. A month of watching could rule things out (not a slow leak at +0.96 MB/h, not any registry at ~200 KB across all of them) but could never name a culprit. lib/heap-peak-store.ts writes the peak, and what was running when it was set, to a container-local file the moment it moves: local rather than the GlusterFS mount because doing network I/O from the path that runs while the process is at its ceiling turns a diagnostic into an outage; synchronous because a write scheduled at 0.98 of the limit may never get its turn; and gated on a 25 MB rise so climbing to a gigabyte costs about forty small writes rather than one per sample. It commits by rename, so a torn write cannot come back as "nothing was ever recorded" — the exact answer that kept this open. On the next boot the panel reads it and, if the previous process died above 75% of the ceiling, raises an alert naming the reading and the jobs in flight.

The other half was that the heavy paths wore no label. runJob wraps the recurring set, but the self-update is user-triggered and was outside it — and the first reading taken after peaks began naming their cause showed a peak set ten seconds after an update finished, attributed to nothing at all. inFlightAs() labels a stretch of one-off work without the alert-on-third-failure contract that would leave a permanent red row for an action nobody runs on a schedule.

And the request paths wore no label either. runJob covers the recurring set, but an API route that answers every single request with a failure raised nothing at all — /api/backups did exactly that from 2026-06-06 to 2026-08-29, and three operations nobody had noticed were broken (archive listing, restore, delete-backup) went with it. Not for want of watching: the panel answers a failure as a 4xx or a 200 carrying {ok:false}, deliberately, because Cloudflare replaces the body of an origin 5xx with its own page and the explanation never arrives — and every operator mechanism then keys on precisely what that convention avoids. The audit log records mutations, the journal needs an explicit console call, an alert needs somebody to raise one; a 200 carrying an error is indistinguishable from success to all three. So the same question runJob asks about background work is now asked about routes, from the shared failure helpers rather than from each handler: not "did this fail", since plenty of failures are ordinary, but "has it failed enough times in a row that it cannot be a blip". Refusals are excluded on purpose — a 403 for a role that may not do something, or a 404 for a name that does not exist, is the endpoint working, and an alert that fires during normal use is one people learn to close. Numeric path segments collapse, so a fleet-wide outage is one failing endpoint rather than a dozen unrelated single failures.

A panel on every node, and a fallback that is checked rather than assumed. The failover machinery has worked since HA landed — keepalived moves the VIP in about six seconds and each replica's nginx lists its siblings as backup — but nothing ever compared the replica list to the cluster. That list is CONDUIT_NODES in install/cluster.conf, written once at install time, so a node that joined later got no panel and no part of the system noticed: check_panels() iterated the configured list and reported "panel + console units active on all 2 replica(s)", which is true, complete, and about the wrong set. The rule is one replica per node with a floor of two running (one on a single-node cluster, where two is impossible and demanding it would be an alert nobody can satisfy). lib/replica-coverage.ts now measures it against the live node list, and the installer's check asks the cluster instead of the config file.

The first version of that check scored a shrinking cluster as a healthy one. It filtered nodes to status === "online" before judging anything, reasoning that a dead node should not also be reported as a node missing a panel; on a three-node cluster with one node down it read "2 of 2 replicas answering" — a full ratio, no gaps named, nothing on the screen false. The denominator had quietly shrunk to match the numerator. A node that is down is now its own category: counted in the target, named in the sentence, and never listed as missing a panel, because it has one and it is on the floor with the node. The invariant is that losing a node must never move the verdict in the good direction.

Provisioning the missing one. When a node has no panel the leader tick builds it: create the container, wait for a lease, install the runtime, push the bundle from the node's shared volume, copy the cluster identity, start the unit, and confirm the panel answers on :3001 — not that the container started, since one that boots and then 502s is a machine that looks like a fallback. The identity is copied from a serving replica rather than minted, because a freshly generated session secret would mean the new panel rejects every cookie the others issued: everyone logged out, by a machine added for resilience.

The delicate part is the VRRP priority, and "put it lower" is only half the rule. The health check runs at weight -60, so a failing replica advertises sixty below its configured priority — which is why the live ladder is 150/100 and not 150/40, so that a healthy 100 beats a failing 150. A new replica must therefore sit strictly below every peer and no more than 59 below the nearest one: a window, not a direction. A step of 70 "to be safe" inverts it and the health check stops meaning anything. The step is therefore the full 50 while the ladder has room for another rung after it and half of what is left below that, giving 150 · 100 · 50 · 30 · 20 · 15 · 13 · 12 · 10 — ten replicas, where a fixed step of 50 declared the ladder full at three and left forty legal rungs unused. It refuses only when refusal is forced, at the floor, because widening past that means renumbering replicas that already exist and a reconciler that only ever adds is not allowed to do that. A ladder it cannot read — a priority that is not an integer in 1..254 — is refused rather than filtered, since filtering the bad entries away empties the list and lands a cold replica at 150, above every serving peer, through a change that looks defensive.

Every other guard has the same shape: never while an update run is in flight, since that run is already the authority on these containers; never onto a node that is down or in maintenance, because provisioning onto a host somebody is repairing is the same mistake as updating one underneath them; exactly one per pass, so a four-node gap is four calm decisions; and an unreadable store counts as "a run may be in flight", so failing to see the cluster means building nothing rather than assuming the coast is clear.

What the first real run taught, which the drill could not. Destroying a replica and letting the reconciler rebuild it produced a container with a runtime and a bundle and nothing else. Both the identity copy and the VRRP config searched the target node for a serving replica to copy from — and a node that needs a panel is, by definition, a node with no panel on it, so the step could only ever succeed where it was not needed. The scratch drill had passed because it runs on a node that already has a replica: it exercised the single configuration in which the code was correct. A drill that can only run where the bug is absent is not a drill. Both steps now find a serving peer anywhere in the cluster and read from that peer's node.

The same run exposed a second fault, and it is the more instructive one: the half-built container blocked its own repair. It existed, so the node read as covered, so coverage reported "every node already has a panel" and nothing tried again — while nothing on that node served. Wreckage from one failed attempt made every future attempt unnecessary. Provisioning now refuses only a container that is actually answering, resumes one that exists but is unfinished, and does not run pct start against a container that is already running. Rebuilt in 59 seconds, keepalived active as BACKUP, VIP never moving.

A service pinned to a node this cluster does not have is the same class of silence one layer up. pickNode returns null, the caller logs a line into the reconcile output, and the task wants an instance for ever without anything saying so. The check cannot simply ask "could it be placed just now" — a node rebooting also leaves nowhere to place, and alerting on that would fire during every reboot. It asks whether the cluster has the nodes the placement names at all, counting offline nodes as present: a node that is down is still a node this cluster has, and reporting its services as misconfigured would turn one outage into a second, wrong finding.

Provisioning that has never run is a hope, so it is drilled the way backups are: POST /api/panel/replicas/drill builds a throwaway replica in the vmid range the restore drills already use, tagged apart so nothing adopts it as fleet, with no keepalived — it never becomes a VRRP peer — then destroys it, refusing any vmid outside that range. The first live drill went from nothing to a serving panel in 22 seconds and still found a defect no reading of the code had: it installed a build nine versions behind the fleet, because the staging directory only ever receives bundles from panel-driven self-updates and the step took the newest file. Every one of its eight steps had reported success. The bundle is now chosen by version, preferring the one the panel is itself running, and when it has to settle for an older one it says so as its own step instead of letting the number pass unmentioned.

A network of its own. Containers were created with bridge=vmbr0,ip=dhcp, which assumes a router is out there handing out leases — true in a rack, false on a VPS, where the container comes up with no address and could not reach the internet even with one, since the only masquerade rule covered inbound DNAT for hairpin. In managed mode Conduit brings the network itself: a private bridge per node, a /24 carved per node from a /16 base, an address assigned per container (matching the vmid where it can, so CT 201 becomes …0.201 and every rule reads without a lookup), and masquerade outbound with an exemption for traffic staying inside the subnet. A subnet that overlaps a route the node already has is refused rather than applied. Off wherever a LAN exists; a fresh install with no private address on any node and no containers yet turns it on itself, because there it is the only thing that could work.

A container outside its node's subnet is moved into it (2026.8.308). Found on edge-02: its store had been cloned from edge-01's, so it carried edge-01's node with the first /24 reserved, and edge-02's own node was adopted into the second. The four system containers created before that kept their leases in the first /24; a later pass rewrote their gateway and left their address, so each had a gateway it could reach and a source address the node would not masquerade. No route out: apt could not resolve a name, and the reconcile failed the same provisioning step every tick with a warning nobody read. Two changes. planAddress (lib/managed-network.ts) now keeps a container's old lease only while it still lies inside the subnet its node has; otherwise it plans a fresh one, matching the vmid where that address is free. And strayContainers names every container whose address is outside its node's /24, with the address a re-plan gives it; it is pure and decides only. renumberStrays in lib/managed-network-sync.ts applies it on the leader tick: pct set <vmid> -net0 …, pct reboot when the container was running, the lease recorded, the firewall signature invalidated so DNAT forwards follow on the next firewall pass, and an info alert "CT n moved to a.b.c.d" raised already resolved, because a container getting a new address is an event an operator must be able to see and not an incident that stays open. What it will not touch: the panel's own container. A stray panel replica is named in the log line with the exact pct set to run by hand, since rebooting the panel from inside its own tick is how a leader stops mid-write. A container on a node the plan does not know, or with no address at all, is not a stray: there is nothing to move it into.