2b. Panel auth, roles & dev workspaces

2b. Panel auth, roles & dev workspaces#

The panel is login-gated with role-based access and self-service dev environments.

Accounts & login. First run offers create the first admin at /login; after that every page and API needs a signed session (HMAC cookie, 7d). Passwords are scrypt-hashed. Admin manages accounts at /users (create, disable, reset, delete — the last admin is protected). Sign-in flow: /api/auth/{me,login,logout,setup}.

The gate runs in the Node runtime (src/middleware.ts) so it resolves the acting user against the live store per request, not when the 7-day cookie expires. The cookie's role is advisory; the store role is authoritative.

The account list is cached for 5s per replica so the gate is not a store read per request, and that cache used to be the honest answer to "how fast does a change take effect": immediately on the panel that served the write, up to 5s later on the other two. It no longer is. The writing replica sends its siblings the store revision it landed on, and a cache read before that revision is refused (lib/identity-sync) — measured at ~50ms across three replicas, with the 5s TTL still there as the floor if a notice is lost. Revoking a token and disabling an account additionally force a fresh read, so they never depended on either.

What the gate asks. Every page and endpoint declares the capability it needs, and the middleware enforces from that declaration rather than from path patterns. A capability is servers.power, backups.restore, files.write — about eighty across twelve areas in lib/capabilities.ts, mapped to routes in lib/route-permissions.ts, with the four roles written as capability sets in lib/roles.ts. Coarse and fine are the same mechanism: a role may hold a whole area (servers) or pick verbs out of it (servers.power, servers.console, not servers.delete), and an area grant keeps meaning when a verb is added later because the expansion happens at check time, never at save.

Before this it was two hand-maintained lists of path prefixes, and the failure mode was permissive — add a surface, forget the list, and it was open to everybody. Now three tests decide coverage rather than diligence: one walks every route.ts and one every page.tsx, failing while anything is undeclared, and a third refuses any route that exports a mutating handler without declaring a write. An unmapped path is refused, which is only safe because those walkers exist. Writing the role sets found five wrong mappings and one real hole before any of it shipped, including a read-only route that would have accepted a POST from a viewer.

Denials say what was missing instead of "admin only" for everything: 403 {"error":"you need \"users.view\" — see panel users and their roles"}.

Capability and scope stay separate questions. The capability is which verb; which servers is answered in-route by lib/rbac.ts, because that needs the task behind the id in the URL. Conflating them is how developer became a special case threaded through the routes by hand.

Self-service (/account, any signed-in user incl. viewers): change your own password + view / mint / revoke your own API tokens, see your sign-in history (IP + success/failure), and sign out of all other sessions. /api/account, /api/account/{password,tokens,logout-all}.

Scoped API keys ("fully locked APIs"). Tokens mint with optional read-only (GET/HEAD only), a hard expiry (7/30/90 days or never), and area scopes (lib/api-scopes.ts — 8 areas: State & metrics · Servers & scaling · Players & moderation · Content & files · Gameplay systems · Automation & alerts · Infrastructure · Administration; nothing ticked = full owner power). All three enforced at the single auth choke point (lib/auth currentUser), so the middleware and every route see the same decision. The path→area map is fail-closed (unmapped endpoints resolve to Administration) and scoped keys can never call /api/account*//api/auth* — a locked key cannot mint itself an unlocked one. Tokens are editable in place on /account (pencil → rename, read-only, expiry, scopes; enforcement updates within ~5s, the secret never changes); rows show scope-count/read-only/expiry badges + last-used.

Login security. Every sign-in attempt is recorded (lib/login-history.ts, IP + result); 5+ failures for a name in 10 min raises a brute-force alert (into the alerting channels). "Sign out everywhere" bumps a per-user session epoch baked into the signed cookie — every older cookie is rejected on the next request (stateless revocation), while the caller's session is re-issued so they stay in. Cookies are v2 (v2.<uid>.<role>.<epoch>.<exp>.<sig>); legacy v1 cookies still verify (epoch 0).

TOTP 2FA (self-service, /account → Two-factor authentication): RFC-6238 codes, hand-rolled on node crypto (lib/totp.ts — SHA-1/6-digit/30s, ±1 step drift; secrets 20B base32). Enroll = QR (otpauth://, rendered server-side) or manual secret → verify a code → 8 one-time recovery codes (shown once; stored as sha256, consumed on use). Login flow: password OK on a 2FA account → 401 {totp:true} → the login page shows the code step and re-submits with code (TOTP or recovery). Disable requires the password. Failed codes hit the same uniform-delay + login-history path. /api/account/totp (GET state · POST setup/enable/disable).

Sudo mode (re-auth elevation). Sensitive actions require re-entering YOUR password even inside a valid admin session: POST /api/auth/reauth {password} mints a second short-lived signed cookie (conduit_sudo, 5 min, uid-bound, stateless → works across the HA panels). The credentials vault (Settings → Data & Credentials) requires it to REVEAL (?reveal=1) or ROTATE — the API answers 403 {error:"reauth-required"} and the UI prompts for the password (masked dialog) and retries.

Secrets register (/secrets, capability settings.secrets) — WHAT the cluster holds. Every credential in one place with three facts the vault does not carry: what it protects, what a leak would cost, and what replacing it takes. Values never appear and the API never returns one; each row shows an 8-char sha256 fingerprint, which is enough to confirm a rotation really happened and to compare the same secret across two replicas, and useless otherwise. Rotation is classed self-service (the panel changes it, reconcile re-applies it), coordinated (consumers must flip together — a mismatch fails auth with no useful error), or external (another vendor's console; the panel offers the runbook and records that you did it, because a Rotate button that cannot rotate turns a two-minute job into a bug report).

The list is derived, not maintained: secrets-registry.test.ts reads the field declarations in lib/store.ts, takes every credential-shaped name, and fails if one is neither registered nor exempted with a written reason — so a secret cannot enter the store without saying what it guards. It replaced a hand-written inventory that had gone quietly stale (no rows for the webhook secrets, the per-service database logins, either Discord token, or the Origin CA private key), and the check found the Origin CA key on its first widened run.

Operator audit log (/op-audit, admin-only) — WHO did WHAT on the panel. The Node middleware is the single chokepoint that already resolves the actor, so every authenticated mutating request (all 78 routes) is logged with {user, role, method, path, action, ip, status} — including denied attempts (403). Human-readable actions ("Deleted server timesmp-spawn"), live via the opaudit topic, persisted to the shared store (op-audit.json, bounded ring like the activity feed). GET /api/op-audit. This is the operator complement to the player audit (§4.10-ish /api/audit). Config diffs: settings mutations record the concrete change, not just the endpoint — POST /api/network snapshots the network config before/after and audits diffConfig(before, after) as a human detail ("alerts.minLevel: warn → error", "tablistHeader = …"). Secret-ish keys (webhook/password/token/ secret/*key) mask their value as ••• but still show as changed (never leaked to the log).

Command palette (⌘K / Ctrl-K, or the sidebar "Search…" pill) — a global launcher: fuzzy-jump to any page or straight to a server/container. Keyboard-driven (↑/↓/↵, Esc), client-only. v2 — action mode: typing a verb (restart / profile / broadcast / scale / say) switches the palette into argument-parsing — e.g. restart lobby, profile 211 45, broadcast network <msg>, scale <task> +1, say <server> <cmd> — with fuzzy task/instance resolution, admin/operator gating (viewers get navigation only; the underlying APIs enforce it too), inline confirm for disruptive verbs, and the verb templates shown idle for discoverability. Runs against the existing bulk-restart/profiler/broadcast/task-PATCH/console endpoints. v3 — entity search: GET /api/search?q= matches live entities server-side — online players (→ Player 360 deep link), groups, blueprints/templates, and panel users (admin-only) — debounce-merged above the nav results with type icons; any Minecraft-name-shaped query always offers a "Player 360 · " row (works for offline players too).

Notification center (the bell, top-right; mounted once, hidden on /login + /status) — a global live alert feed (service/node down + recovery, backups, watchdogs, mail announcements) via the alerts live-bus topic (instant) with a 60s poll as the safety net. v2: severity filter chips (All / Problems / Resolved), consecutive same-title runs coalesced into one ×N row (a flapping service can't flood it), unread rows get a brand band + explicit Mark all read, a View incidents jump, and per-alert desktop notifications (opt-in, native Web Notifications). Motion: AnimatePresence panel + staggered rows, reduced-motion respected.

Incidents (/incidents, Monitoring → Incidents) — the flat alert feed correlated into incidents per subject (30-min join gap; a recovery alert closes one; peak severity + duration; ongoing = pulsing). Operator-audit response actions in the window are overlaid ("what happened AND what we did"). v2 is an incident-command surface: a summary stat band (ongoing / 7-day count / 7-day downtime / planned), a 7-day time-strip (per-subject lanes, severity-colored draw-in bars, day gridlines), day-grouped cards with left severity bands + log-scale impact bars, an expandable draw-in timeline, and per-incident Markdown postmortem export (?id=&format=md). Pure derivation over the alert ring + op-audit — no new state. Planned downtime: task.plannedDown {at,by,reason,until?} (set via PATCH /api/tasks/:id, until auto-expires in the uptime sweep with a service.planned.end info alert) marks a service intentionally down → no service.down alerts / status-subscriber mail, planned (neutral sky) status on /uptime + the public status page (never degrades overall), neutral incidents. A scheduler panel on /incidents sets service + reason + duration with an End-now action. Maintenance is wired to the same suppression: a maintenance-flagged task (own flag, or cascaded from its subgroup chain / group) is auto-included as planned — shown with a black/yellow caution-tape band + MAINTENANCE chip and a Disable-maintenance action.

Outbound webhooks (/hooks → "Outbound webhooks", admin-only) — the inverse of inbound hooks: subscribe an external URL to the alert bus with a per-subscription event kind-prefix filter. Every delivery is HMAC-SHA256-signed (x-conduit-signature: sha256=… over the raw body; the signing secret is shown once at creation), retries with backoff (30s → 2m → 10m → 30m, 5 attempts) off the leader tick, and a rolling in-memory delivery log feeds the UI. lib/outbound-hooks + GET/POST/PATCH/DELETE /api/outbound-hooks (+ test-fire).

Prometheus exporter (GET /api/metrics/prometheus) — standard text exposition, 17 metric families, of everything Conduit already measures. This is the whole Grafana story — there is no InfluxDB and no second collector, because Conduit is already the thing that measures the fleet and Prometheus is the format Grafana speaks. Anything shipping metrics to a separate time-series database would be copying data sideways to say the same thing twice.

It's an isMachinePath, so it's open when machine-auth isn't enforced and needs a bearer token when it is (which is the case here — verified: an unauthenticated scrape gets 401). The token is any of CONDUIT_AGENT_TOKEN / CONDUIT_CONNECTOR_TOKEN / CONDUIT_SESSION_SECRET from /etc/conduit/panel.env on a replica. Scrape the VIP, not a replica, so it follows the leader:

scrape_configs:
  - job_name: conduit
    metrics_path: /api/metrics/prometheus
    authorization: { credentials: "<CONDUIT_AGENT_TOKEN>" }
    static_configs: [{ targets: ["10.0.0.50:3001"] }]

health — conduit_service_up, conduit_service_instances, conduit_containers_total, conduit_alerts_total{level}
players — conduit_service_players, conduit_network_players (from the proxies, so no double-counting), conduit_network_capacity
performance — conduit_service_tps, labelled per instance (vmid), not per service: averaging two shards hides the one that is struggling, which is the only reason anyone looks at TPS
availability — conduit_service_uptime_ratio{window="24h|7d|30d"} and conduit_service_latency_seconds, taken from the panel's own uptime ring rather than left for a dashboard to derive from scrapes of conduit_service_up — which would cover only the window Prometheus itself has been running and would disagree with the figure the panel shows for the same service
capacity — conduit_node_up, conduit_node_cpu_ratio, conduit_node_memory_bytes{state}, conduit_node_disk_bytes{state}, plus per-container conduit_instance_cpu_ratio and conduit_instance_memory_bytes{state} for "which service is eating the node"

Base units throughout (seconds, bytes, 0–1 ratios) per Prometheus convention — a dashboard that assumes milliseconds is out by a factor of a thousand with nothing to give it away. Degrades gracefully if Proxmox is momentarily unreachable: node and container metrics drop out, service health continues from the uptime ring and the connector heartbeat.

Public status page (/status, opt-in) — a shareable, no-login status page: per-service up/down + 24h uptime + an overall banner. Off by default (Settings → Public status page); when enabled, /status and GET /api/status are public but expose only friendly names + status (no ids/ips/nodes/counts). Auto-refreshes. Email subscriptions: the page also carries a double-opt-in subscribe box (shown only when SMTP is configured) — visitors confirm an address via a mailed link, then get emailed when a shown service goes down or recovers, with an unsubscribe link in every message. lib/status-subscribers, public /api/status/subscribe (POST subscribe · GET confirm/unsubscribe), fan-out hooked into the uptime sweep. Both public faces — /status and the login page — carry a quiet ambient layer (components/ambient-field.tsx): a slow constellation of the brand mark's node dots joined by distance-faded lines, plain canvas 2D (no dependency, renders everywhere), reduced-motion gets a single static frame and a hidden tab stops the loop.

Alerts & notifications. Conduit turns operational events into a persisted alert feed + pushes them to channels. The uptime sweep raises an alert on every service/node status transition (DOWN = incident, UP = recovery), de-duplicated by key (a persistent-down service pages once, not every tick). Channels (Settings → Alerts · notification channels): Discord webhook, ntfy topic, a generic webhook (gets the full alert JSON), and email (comma-separated recipients, delivered via the panel's SMTP server — Settings → Email · SMTP server, nodemailer-backed, password is write-only and saving/testing requires sudo re-auth) — each gets every alert at/above a configurable minimum level. A notifications bell (top-right, all pages) shows the live feed with an unread badge (live via the alerts topic), with a clear, browser-permission-gated desktop-notification opt-in — when enabled, new incidents fire a native OS notification (no backlog replay on enable). GET /api/alerts (feed) · POST /api/alerts {test:true} (fire a test through the channels). lib/alerts.ts (raise/dedup/deliver/persist, same ring pattern as the audit logs).

Panel-managed Discord bot. A two-way Discord bot, distinct from the one-way alerts.discordWebhook channel above. It runs as a sidecar (agent/discord-bot.mjs, conduit-discord.service) in the panel LXCs next to the console proxy and follows the VIP — only the replica holding it runs the gateway, so a command never double-fires. It registers guild slash commands and answers them against the panel API: /status, /servers, /scale, /players, /incidents; it can also mirror panel alerts into an ops channel. Transport is the raw Discord Gateway v10 + REST over the already-bundled ws dep (no discord.js), best-effort and self-healing (any error logs and retries; never crashes). Configured entirely from Settings → Alerts (Discord bot card, admin): bot token, application id, guild id, ops channel, a pasted Conduit API token the bot drives ops with — its role and scopes bound what the bot can do — an optional Discord role-id allowlist that gates who may run ops commands, a per-command enable set, and the alert-mirroring toggle. The bot reads the full config (incl. secrets) with the machine token; the admin UI reads a masked view. Secrets live vault-style in the store (network.discord); sending "" keeps the current secret, "-" clears it. The service is bundled and enabled across install/update/repair/uninstall. GET/POST /api/discord/config.

Per-user notification mails. Each user opts in on /account (profile → Email me: incidents / warnings, requires a verified recovery email): incidents = service-DOWN + recovery mails, warnings = warn-level alerts — delivered by deliver() alongside the channel fan-out, independent of the channel minLevel, deduped against the global email-channel list, and reset automatically when the recovery email is removed.

Web-push notifications (PWA). The panel is an installable PWA (public/sw.js service worker + manifest.webmanifest) and can deliver alerts as real OS push notifications on browser and mobile — not just the in-tab bell. Each device opts in on /account (Push notifications → Enable on this device): a VAPID keypair is auto-generated once and persisted in network.vapid (shared across the HA panels), the browser subscribes via the service worker, and the subscription is stored on the user record. Alerts fan out to every subscription of opted-in users through the same deliver() path and the same incidents/warnings toggle as the mails; dead endpoints (404/410/400/403) are pruned automatically, tapping a notification focuses/opens the panel at the relevant page. lib/webpush.ts, GET/POST/DELETE /api/push (public key / subscribe / test / unsubscribe). Needs HTTPS to activate in the browser (the panel domain provides it).

Announcement mails. Admins compose subject+message on /users (header → Announcement) — delivered to every user's verified recovery email via the panel SMTP (POST /api/mail/announce, admin + sudo re-auth; optional explicit to:[…] list; per-recipient results, [Conduit] subject prefix + sender attribution; the op-audit middleware records who sent what). Every announcement is ALSO mirrored into the notifications bell (mail.announce alert kind, mail icon, live via the alerts topic) so panel users see it in the UI, mailed or not.

Mail extras. The SMTP card also manages DKIM signing (send-only hardening): one click generates an RSA-2048 keypair for the From-domain (conduit._domainkey.<domain>), the private key stays write-only in the store, outgoing mail carries a DKIM-Signature header, and the card shows the DNS TXT record to publish (name/value with copy buttons). Removing the key stops signing. SPF/DMARC stay plain DNS records on the domain (nothing to configure panel-side).

Password-reset mails. Every user can set a recovery email on /account (profile card, inline edit) — setting/changing/removing it re-checks the password, and a new address is pending until verified: a 6-digit code is mailed there (30-min expiry, 5 wrong tries voids it, 60s resend limit) and only POST /api/account/verify-email activates it. Reset mails only ever go to the verified address (an unverified change never hijacks resets — the old verified address stays active until the new one confirms). The login page gets Forgot password? → POST /api/auth/reset-request mails a one-time link (/login?user=…&reset=<token>; token stored sha256-only, 30 min expiry, 5-min rate-limit, oracle-safe: uniform timing + identical response whether or not the account/email exists). The link opens a Set a new password form → POST /api/auth/reset-complete verifies + consumes the token, re-hashes, and bumps sessionEpoch so every existing session dies. Both endpoints are public (login-page paths) and feed the login history (reset mail sent / password reset via mail).

Live sessions. Logins issue v3 cookies carrying a session id registered per device (PanelUser.sessions[]: ip, user-agent, created, last-seen — last-seen touched at most every 5 min). A re-login from the same ip+ua prunes that device's older rows (the browser overwrote its cookie anyway — no duplicate "two Linux sessions" from one machine, and a copied old cookie dies too). /account shows Cloudflare-style per-device session cards (device title + current session chip or Revoke button, then icon detail lines: Browser · IP, Signed in · Last seen); revoke = POST /api/account/sessions {id} — that cookie dies within ~5s. A console/CLI session (…/api/account/console-sessions) is also disconnected, not merely invalidated: deleting the record only made the session's next API call fail, so a shell sitting at a prompt stayed open — not what revoking a credential you have stopped trusting should mean. The gateway shell reports its pid, and revoke signals it on the replica that minted it, after reading /proc to confirm the process really is that session's shell (pids get recycled, and a panel container is the last place to send a blind kill). The CLI also exits on a 401, so the revoke lands even when the process cannot be signalled. "Sign out other sessions" bumps the epoch AND resets the registry to just the caller; plain logout prunes its own row. Legacy v1/v2 cookies stay valid but unlisted until that device's next sign-in. Sign-in history shows the 5 newest with a Show all expander.

Account page (CF-style). /account mirrors Cloudflare's profile UX: identity in the page header (username big, email · Member since <date> subline), a profile panel with an avatar header row (role badge, 2FA badge, last sign-in) and hairline-divided settings rows (title+description left, control right: recovery email, notification opt-ins), Password and 2FA side by side as equal-height cards sharing one header recipe (title+description left, ONE action right) — the password form is collapsed and height-animates open/closed (framer, CF ease, the header button flips to Cancel; reduced-motion snaps), then tokens + sessions/history.

Profile picture. Fallback chain: custom upload → Gravatar → initials. Hovering the avatar reveals an upload overlay (camera icon → file picker; trash appears when a custom picture is set). Uploads are canvas-resized client-side (center-crop square, 128px, webp) so the store only carries a tiny data URL (PATCH /api/account {avatar} — validated mime + ≤200KB; "" removes and falls back to Gravatar-if-email-else-initials). The Gravatar URL is the SHA-256 of the VERIFIED recovery email with d=404, so a missing Gravatar cleanly falls through to initials client-side.

Roles.

admin — everything, incl. user management, system credentials, DB browser.
operator — full operations, no admin surface.
developer — operations scoped to selected groups only (enforced per-route via lib/rbac on task/group/container/console/file mutations).
viewer — read-only (mutations blocked in the gate; own-account self-service is exempt).

API tokens. Personal bearer tokens (Authorization: Bearer cu_…) — mint your own on /account (or an admin mints for anyone on /users), scoped to that account's role/groups, for scripts/CI without a session. The Node gate validates them against the store, so a revoked token stops working immediately.

SSH access. /account holds ssh keys for the console gateway — a DEDICATED sshd on every panel CT (default :2202, own host keys, conduit-ssh unit). ssh -p 2202 conduit@<vip> opens the Conduit console as your panel user: registered/trusted keys go straight in (live per-connection lookup — revocation is instant), any other key lands in the username sign-in (approve the shown code on /approve in the browser, or type your authenticator code; then optionally trust the key). Sessions run on a 12h token ("ssh session" under API tokens). No passwords anywhere, no forwarding, and no system shell — the only exception is the console's system shell, admin-role-checked and confirmed. /api/account/ssh-keys, /api/auth/device + /approve, lib/ssh-gateway, agent/cli-gateway*.sh.

Browser-approved CLI sign-in. conduit auth login never asks for a password: it shows a short one-shot code, the bound user approves it on /approve (only the named account can even see the request), and the terminal receives a personal token. /api/auth/device.

Machine callers. The ops scripts, the installer, the CLI and the Discord bot authenticate with the panel machine token (CONDUIT_AGENT_TOKEN, full access, compared in constant time). The connector-facing paths (/api/connector/*, friends/parties/bans/logs/blossoms, LP-effective, world pulls, lang-data, the Prometheus scrape) require a token by default since 2026.8.304 (lib/auth-core.ts enforceMachineAuth): a tokenless call answers 401. CONDUIT_ENFORCE_MACHINE_AUTH only ever turns enforcement OFF, with 0, false or off; unset, or any other value, enforces. It was the other way round until 2026-09-03 (opt-in with =1, off by default, on the argument that turning panel login on must never 401 a live heartbeat), and that default cost two live installs: on 2026-08-23 both public boxes, each with :3001 open to the internet, were found running without the flag, so /api/bans, /api/friends and /api/connector/servers answered anonymous callers and POST /api/connector/action reached the player action queue. A default that fails open is the bug, not the flag. The opt-out is kept for the migration case and is loud while set: /secrets shows an amber "machine auth opted out" chip and the middleware logs a [machine-auth] tokenless hit warning (one per path and address per ten minutes) for every tokenless call it lets through. Sessions still pass those paths through the normal user gate. The connector's own token is deliberately not a panel-wide credential: it lives in a file inside every game container, so it authenticates the connector endpoints and nothing else (lib/connector-auth.ts).

What signs a session, and what compares a secret (2026.8.304). The session key is resolved in order: CONDUIT_SESSION_SECRET, used byte for byte as it always was; otherwise a key DERIVED from CONDUIT_AGENT_TOKEN as HMAC-SHA256(key = token, data = conduit-session-key), so the full-admin machine bearer is no longer the string every cookie is signed with and a copy of one is not a copy of the other (an install on this path got exactly one forced re-login when it took the release); otherwise a per-boot random with a console warning, and sessions die on restart. Every place a presented credential meets a stored one compares in constant time: safeEqual runs over the longer of the two byte strings and folds the length difference into the result, so neither the position of the first wrong byte nor the length of the secret shows up as time; invite codes, grant secrets and the relay's signed hop use timingSafeEqual. Cookies carry Secure when the request arrived over TLS (x-forwarded-proto: https from nginx or Cloudflare, or the request itself https) and not otherwise, because a Secure cookie on a plain-http LAN panel is one the browser silently refuses to store, and a login that "succeeds" into a blank page is worse than a cookie without the flag (cookieSecure, applied at login, setup, re-auth, passkey login and the SSO callback). Single sign-on links or creates an account by email only when the provider vouches for the address (Google's email_verified, Discord's verified, GitHub's verified-emails list, read whenever GitHub has one); an unverified address is refused with "sign in with an email that X has verified, or ask an admin to link the account", because it is a string anyone can type into a profile. An identity already linked by provider and subject wins regardless. And a role cannot hand out more than its holder has: grantsBeyond (lib/permissions.ts) is checked on what a grant EXPANDS to, so servers.* is beyond reach for an account holding only servers.view, and POST/PATCH /api/roles refuses it; an admin holds * and is never limited by it.

Dev workspaces (/workspaces). One click clones a source server (world + plugins + config from its blueprint/seed) into a temporary personal copy in the shared dev group — a single instance with a TTL (default 24 h, max 7 d). The leader tick auto-tears-down expired workspaces. Developers create/see only their own (and only from servers in their scope — the source picker is served already filtered); operators/admins see all. This is the Skydinse-team onboarding path: spin up an isolated TimeSMP (or any server) copy to work on, that cleans itself up.

Workspaces are fully separated from production: the state route keeps the dev group out of the servers tree, the players strip and the topology map (they travel as their own devWorkspaces field), the uptime sweep never monitors or pages for them, and backup coverage, Verify and the nightly database dumps all skip them — a booting or expiring clone can't cry wolf anywhere. The /workspaces page is the one place they live, and it is a real server list: live status dot (provisioning pulses amber), owner, source, the actual container (#vmid · node · ip · uptime), and per-row Console (the full /services/[vmid] surface, which wears a violet workspace banner there), Restart, +24 h (TTL extension measured from now, capped at 7 days however often it is pressed) and Tear down. The containers inventory keeps listing the CTs, marked with a dev chip. RBAC passes a developer for workspaces they own — the shared dev group is in nobody's group scope, so without that pass a developer could create a copy and then be locked out of its console. /api/dev-workspaces (+ /:id PATCH extend · DELETE).

Env: CONDUIT_SESSION_SECRET signs sessions; without it the key is derived from CONDUIT_AGENT_TOKEN, and without either a per-boot random is used and sessions die on restart. CONDUIT_ENFORCE_MACHINE_AUTH is not needed to enforce: enforcement is the default, and setting it to 0, false or off is the only thing it does.

Domains & TLS (/domains, Data category — admin + sudo for every mutation). Cloudflare- integrated HTTPS for the panel (and any HTTP service):

Cloudflare connection: zone + a scoped API token (Zone.DNS edit — verified live by resolving the zone) + optionally the Origin CA key; both are write-only in the store (vault pattern). The Origin CA key is a dashboard-only artifact (CF has no API to create/fetch it) — but cert issuing FALLS BACK to the API token, so a token granted Zone → SSL and Certificates → Edit makes the origin key unnecessary (one credential total).
Hostnames: entries like panel.example.com → target panel (the HA front) or custom ip:port (any internal service — future BlueMap/status hosting). Adding/removing an entry upserts/deletes the CF A-record (proxied orange-cloud or DNS-only per entry, pointing at the network's public IP). Hostnames are validated against the zone.
Origin CA certificate: one click issues a CF-signed origin cert covering all entry hostnames (15-year validity; CSR built in-process — no openssl dependency; private key never leaves the store). This is what lets the orange-cloud run SSL mode Full (strict).
TLS front rollout: the reconcile leader (lib/domains-sync.ts) installs nginx on the 3 panel LXCs, writes cert+key + one vhost per hostname (SSE-friendly proxying, HSTS, 512m uploads, websocket upgrade), validates (nginx -t) and reloads — re-applied only when the config hash drifts (cert re-issue, entry changes). Panel entries proxy 127.0.0.1:3001 on each CT (any panel node serves the domain — keepalived VIP or CF load-balancing both work in front).
e2e-tested against a mock CF API (16 checks: token verify, record upsert/dedupe/delete, zone validation, CSR verified with openssl, key never exposed). CLOUDFLARE_API_BASE overrides the endpoint for tests.
Connectivity wizard + activation gate (/domains → Connectivity & activation): before a domain goes live the panel PROVES it reaches this panel. POST /api/domains/check {hostname} runs server-side DNS diagnostics (resolves? Cloudflare-proxied? points at our public IP?) and mints a one-time challenge token bound to the hostname; the admin's browser then fetches https://<hostname>/api/domains/challenge?token=… over the public internet — for that to succeed DNS, the edge :443 forward, the Origin cert (TLS handshake) and the Host header all have to work. The public challenge endpoint (isPublicPath, CORS-open) stamps the entry verifiedAt only when the token AND Host match — the browser's word is never trusted. POST /api/domains {setPrimary} HARD-GATES on a recent stamp (409 otherwise), so a missing port-forward / wrong public IP / bad cert can't leave you locked out. The wizard shows a live checklist (DNS · reachability+TLS · ready-to-activate) with exact fix hints, and activation sets network.domains.primary. e2e 11/11 (gate blocks unverified, wrong-token 403, wrong-Host 409, right-token+Host stamps + one-time, activate allowed only after).
Setup for panel.example.com: paste the CF token + Origin CA key, add the hostname (target: panel, proxied), issue the cert, wait one reconcile tick, set the CF zone SSL mode to Full (strict) → done. A guided first-visit onboarding wizard with reachability verification is queued in ideas.md.