Measured, on the live cluster

Numbers Conduit's design claims rest on, taken from the running fleet rather than from a synthetic harness. Where a claim is structural — true because of how something is built, not because of what was loaded that evening — it says so; where it depends on the fleet's state, it says that too.

The CloudNet comparison lives in COMPARISON.md and is a different exercise: two products, one evening. This file is about Conduit against its own earlier self.

---

Asking a container a question#

Measured 2026-08-25 on core-01, five runs each against the same container and the same question. 44×, and the interesting part is not the speed.

The other four rows on that chart are from CT 213 and are the same shape, except for the two at the bottom. Those are close because the console path already went through a shell the node agent keeps open — that fix landed first, and it is the reason a warm console never felt slow while everything else did.

pct exec is a ~130 MB Perl process. The reconcile loop was firing roughly 19 of them a minute just to ask whether files existed, and the memory churn produced a 22-second TLB-shootdown soft lockup that reset a guest — twice in two days. Moving container inspection into lib/ct-inspect.ts cut it from 97 to 9 per 300 s, and the agent removes the rest. So the latency figure is the visible half of a fix whose real subject was node stability.

Live on 27 of 30 running containers. The three without an agent are the panel replicas, which do not need one.

Structural. Entering a namespace costs what it costs; a warm HTTP listener costs a LAN round trip. Neither number moves with load.

---

Moving a file into a container#

base64 is 1.33× the bytes — measured, not the textbook 4/3 — and the old upload path held the incoming body, the parsed value and the re-serialised outgoing body at the same time. So the peak was about 4× the file, and a file large enough to multiply past the heap ceiling killed the process.

The ceiling is now a decision rather than an accident: the unit passes --max-old-space-size=1536, and the live panel reports V8's own limit as 1560 MB. When the 2026-08-14 kill happened it was whatever V8 derived from a 2 GB container — about 1 GB — so a ~250 MB upload was enough. At 1536 MB it takes about 384 MB. Neither number is comfortable.

A 400 MB world is not an edge case for a Minecraft host, and 4× of it clears both ceilings. It was simply impossible.

The streamed path carries one chunk at a time, so the peak is the chunk — 32 KB when an SFTP client asks, ~1 MB on the panel's own upload — and does not grow with the file. This is why ctFsDownloadRange takes an offset and a length rather than returning a file: SFTP reads in pieces, and a range that fetched the whole file each time would have reproduced the original bug inside the fix.

Structural. The ratio is arithmetic. Only the ceiling is environmental, and it follows the CT's memory.

---

Reading a file over SFTP#

Measured 2026-08-26 through a real sftp client against the ingress gateway, three runs after the change; every run returned all 57,698,070 bytes with sha 52d5fd9f…, identical to the file inside the container. ~140×, and it is worth being precise that this is not an optimisation — it is a fault that had been shipped.

Where the client sits decides the number, so it belongs in the table. Re-run on 2026-08-27 from a workstation on the far end of a WireGuard tunnel, the same file came back in 3.6–4.8 s — 11–15 MB/s, three runs, same sha every time. That is not a regression: raw ssh … dd over the same tunnel, carrying no SFTP at all, managed 17–22 MB/s. SFTP is running at roughly two thirds of whatever the path can carry, which is the honest way to state it. The 48–61 MB/s figure is what that leaves on a client near the cluster; quote it without saying where the client was and somebody on a VPN will reasonably think something broke.

Two things compounded. The route resolved the fleet on every request, and doing so calls Proxmox's /cluster/resources, which has no cache of its own. SFTP clients read 32 KB at a time, so a 400 MB world was costing roughly 12,800 Proxmox round trips and nearly all of the transfer time was the panel re-deriving something that had not changed. Separately the protocol adapter answers one packet at a time, so openssh keeping 64 reads outstanding bought nothing — each queued behind the last.

The fix is a ten-second memo on the fleet (cached unfiltered, because identity is applied per request, so it cannot carry one person's scope to another) and a 4 MB read-ahead window in the adapter, which turns ~128 round trips into one.

That is the whole story in one picture: 1,761 requests for one 57.7 MB file, each one a full /cluster/resources call, against 14. The client still reads 32 KB at a time — nothing about it changed. It is now served from a window that had already been fetched.

Not structural. The ceiling here is the LAN and the agent, and the numbers move with both. What is structural is the shape: cost per window rather than per 32 KB read.

Writes are not in the table because I have no before to compare against — they did not work at all until the same week. A 3 MB binary round-trips byte-identical; the sustained figure for a large upload is not yet measured, and I would rather leave that blank than estimate it.

---

Failover#

From the proxy-HA work (2026-08-13) and the VIP health check (2026-08-03), each timed from a real socket death rather than a clean shutdown. The panel figures come from a sabotage drill: dead at 13:05:33, VIP moved 13:05:40, replica repaired 13:07:35, VIP home 13:07:56.

The axis is logarithmic because the four numbers span two orders of magnitude, and they should: the first two are a watchdog rewriting a port table, the last one is a build being rolled back and a service coming up healthy. Compressing them onto a linear axis would hide the two that matter to a player.

---

Taking a permission away#

Each panel caches the account list for five seconds so the gate is not a store read per request, and the cache is per process. Revoking a token and disabling an account always forced a fresh read; changing a role, narrowing a scope or removing a capability did not — those were honoured at once on whichever replica served the write and up to five seconds later on the other two.

Measured 2026-08-29 by demoting a throwaway account from admin to viewer on CT 190 and polling an admin-only endpoint on CT 191 until it stopped answering 200. Every run was preceded by a control confirming CT 191 answered 200 while the account was still an admin — without it a broken probe would refuse everything and look like a fast revocation.

The comparison that makes the number mean something: under the old behaviour this wait was the sibling's own TTL expiring, uniform in [0, 5000] ms. Five samples all under 65 ms has a probability of about 4 × 10⁻¹⁰ under that distribution, so the mechanism is doing the work rather than luck.

The remaining ~50 ms is not the notice. It is the PATCH round trip, the store write with its compare-and-swap, and the sibling's own re-read — the notice itself is one LAN hop.

---

A planned update, on the other hand#

A self-update moves none of that — replicas restart in place and nginx sends traffic to a sibling. Measured through a full three-replica update, probing every 200 ms. The first two rows were run twice and came out the same both times (861 and 1417 requests on the second pass, same verdicts):

The difference between the first row and the second is nginx. :3001 is the panel process itself, so it is unavailable for exactly as long as one process takes to restart; where nginx is in the path there is another replica to answer, and nobody sees anything. A single-node install has no sibling, which is why its three blips exist — 0.2 s each and more than a minute apart, so three small ones rather than one outage.

---

What is deliberately not here#

A number I would have preferred not to publish. This section used to say there was no honest SFTP figure to give. When I finally measured one, it was 0.34 MB/s — 57.7 MB in 160 seconds. See the section below; the disclaimer had been standing in for a fault.

Anything about game performance. Conduit does not touch the tick loop. Comparing worlds would be comparing plugin stacks and player counts, which says nothing about the control plane.

Heap-per-panel over time. Watched, not settled. Two windows now agree that there is no leak worth the name: 18.8 hours measured +0.96 MB/h, and a fresh 16-hour window on 2026-08-27 measured +0.71 MB/h across its last eight hours. At that rate the leader would need about three months of uninterrupted uptime to reach its own ceiling, and it never gets one.

Which leaves the 2026-08-14 kill (#188) still unexplained, but narrowed. It was a spike, not a drift. The strongest candidate was the base64 upload path above, and that path no longer exists — it streams. Since the panel started recording its own peaks, the highest any process has reported is 287 MB of 1560, with none of the recurring jobs in flight.

Worth one caution, because it nearly caught me: the endpoints of that 16-hour window are 69 MB and 127 MB, which subtract to +3.6 MB/h. That number is wrong. Most of it is a freshly restarted process filling its caches, and the rest is GC sawtooth of about ±23 MB — a least-squares fit over the post-warm-up half gives 0.71. Two points on a sawtooth are not a trend.

---

Method#

• Agent latency — pct exec 200 -- true and curl /v1/health against the same container's agent, five iterations each, wall time by date +%s%N on the node itself so no workstation round trip is included. Agent coverage counted by probing port 8802 in every running container on all three nodes.
• Transfer memory — base64 and JSON expansion measured on a 1 MB buffer in Node, multiplied out; the ×3 concurrency is the three simultaneous copies in the old handler, read from the code path. Chunk sizes are the constants the two callers actually use.
• Per-operation latency — the 393/9 ms pair on core-01 as above; the other four rows on CT 213, timed at the panel's own client rather than inside the agent, so the LAN round trip is included in both columns. The relay column for the two console rows is the node agent's persistent shell, not a cold pct exec — which is exactly why they are close, and why the chart says so.
• Failover — timestamps from the panel journal and the fast-watch ring, cross-checked against a 200 ms external probe.
• Rollout availability — scripts/probe-availability.sh, every 200 ms for the length of a full rollout, run twice. On the strip chart a failed probe is about a third of a percent of the width, so the failures are drawn at a minimum visible size; the counts in the table are the real ones.
• Heap — GET /api/diagnostics/memory on the leader, which keeps its own 24-hour ring at one sample per five minutes (lib/heap-watch.ts). The growth figure is a least-squares fit over usedMB, not a difference of endpoints. Its growing list is a starting point and not a verdict: on this run the two fastest climbers were the GlusterFS heal backlog and the storage snapshot, both of which are sized by live cluster state — they were large because a node was down, and both prune themselves. Reading the code beat reading the number.

Permission convergence. A throwaway admin account plus an API token, created on CT 190. Each run: promote to admin, poll CT 191's /api/users until it answers 200 (the control), then PATCH {roleId:"viewer"} on CT 190 and poll CT 191 until it stops. Wall time via date +%s%N around the PATCH. Note that role alone does not demote — the gate reads roleId — which is the sort of thing that turns a measurement into a fiction if you do not check the control.