From 3d7215593cd0a042bd2cbe27e1dde98f63daeda2 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 16:55:47 -0300 Subject: docs/next-daemon.md: Phase 5 groundwork notes: the replica socket's decisions, the round trips locally, through the relay and to FreeBSD, and open points 3 and 5 decided (walker copy for rollback and for join, no fork) A ping through `ssh -L` to a session on the FreeBSD box costs its RTT (median 205 to 210 ms against a TCP connect's 206 to 208) and a hello two (411 to 476 ms: ssh confirms the channel before it forwards a byte); through the relay at 30, 80 and 207 ms they are the RTT plus 0.3 ms. A rollback copy is about 80 KB (typical) or 96 KB (big) once file texts and undo snapshots are pinned, 0.02 to 0.045 ms; a fork of 1 GB stalls the authority 13 to 20 ms here and 20 on FreeBSD, and every page it writes while the child lives costs ~1.3 us more, so a join takes the same walker copy and a worker streams it. The tty A/B against phase 1's tip is within its A/A runs. Still open marks points 3 and 5 decided. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 199 insertions(+), 2 deletions(-) diff --git a/docs/next-daemon.md b/docs/next-daemon.md index e7ddcf5f..efe5214a 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -163,12 +163,17 @@ open. ordered stream with no per-read round trip; 9P stays for scripting and the relay] 3. Rollback by walker copy or by arena copy-on-write. [whichever - measurement of the mutable state's size decides] + measurement of the mutable state's size decides] Decided: the walker + copy, about 100 KB and 0.05 ms once file texts are pinned (Phase 5 + groundwork notes). 4. How the daemon starts: spawning itself or socket activation. [itself, by double fork; FreeBSD has no systemd. A user unit can be added if a terminal's scope ever kills it] 5. The join writer: a fork child, or a metadata copy serialised on a - worker. [fork first, measure the page-table copy] + worker. [fork first, measure the page-table copy] Decided: the copy on + a worker. A fork of 1 GB stalls the authority 13 to 20 ms, and + copy-on-write taxes it for as long as the join streams (Phase 5 + groundwork notes). ## Phase 3 notes @@ -941,3 +946,195 @@ against that tty, is the Tty throughput notes' keystroke-to-present. - A replica pushes a flood's rows into its history one at a time. - The integration into the stream and the cores (a replica pane's State fed by `apply` instead of `.output`) is the next piece. + +## Phase 5 groundwork notes + +Built 2026-10-29: the delay relay (`test/delay_relay.py`), the replica +socket (`src/daemon.zig`, `src/detached/wire.zig`), and two measurements +for open points 3 and 5 (`zig build rollback-bench`, which runs +`test/rollback_bench.zig`, and `test/fork_stall.py`). Decisions taken at +the forks, with the ponytail lens: + +- **The delay relay is a Python script.** Each direction is a reader that + stamps every chunk with a due time and a writer that sends it then. The + due time is the chunk's arrival plus the delay. With a rate cap, it is + the moment the link has carried what came before, plus the delay. So + throughput is not serialised behind the delay, and the bytes stay in + order. Either side is a Unix or TCP socket. ponytail: its buffer is + unbounded, which makes a link with an infinite bandwidth-delay product; + on a real link, ssh's window is the bound. +- **Two sockets, where the design had "the daemon's one".** The + fd-carrying SEQPACKET socket stays local. Once the daemon holds its + lock, it binds `pardes-replica.sock` (SOCK_STREAM) beside it, and + unlinks it at exit. `ssh -L` forwards that one. +- **The daemon judges the hello, so a session never sees one it + refuses.** + - Every session's registration carries its semantics hash (daemon + protocol 2), and the daemon compares the client's with the session's. + - It refuses in words: another protocol; another build, naming both + hashes and pointing at `pardes --version` to compare them; or a + session nobody has. + - It reads exactly the hello's length (the header, then the payload), + so whatever the client sends after it is left for the session. + - Up to 16 hellos may be on their way at once. ponytail: a client that + never sends one holds its slot until it hangs up. +- **The welcome comes from the session**, with its pid and name, over the + connection it was handed. That tells the client the handoff happened. + The test checks that the session's fd count went up and the daemon's + did not. +- **The semantics hash** (`wire.semantics`) is computed at comptime and + covers: + - the protocol, the release and the commit; + - the platform and the build options; + - every dependency the manifest pins, by its content hash (build.zig's + `deps` option). The design said grammars, but ghostty and the + Unicode tables also make a core step differently, so it takes all of + them; + - the walker codec's assumptions (pointer width, byte order). + + It leaves out the OS. The main test is a Linux client of a FreeBSD + session, and what differs between those is the host's, which the tape + carries. The two builds of one tree hash alike (1087257bf166588a on + both), and the box's session welcomed the client from here. ponytail: + an unstamped build (Debug with `-Dplatform`, so every test build) has + no commit, so two dev trees with one release and the same options hash + alike. +- **The hello lost its grid**: a replica's view takes a resize like any + other input. A ping (client tag 0x03) and three server tags (welcome, + refused, pong) are new. The wire is version 14. +- **The session's half is a stub**: a thread per replica that sends the + welcome, answers pings, and ends at bye or a hangup. + +Measured. Timings ran alone through the perf lock, pinned to cores 6-7 +while builds ran niced on the others. The local A/B ran interleaved, with +an A/A run beside it. + +- **The relay's self-check** added this much over a direct connection, + at each delay (median RTT, then throughput through it): + + | one-way delay | wanted | added | throughput | + |---|---|---|---| + | 15 ms | 30 ms | 30.26 ms | 3.2 GB/s | + | 40 ms | 80 ms | 80.27 ms | 2.3 GB/s | + | 103.5 ms | 207 ms | 207.26 ms | 2.3 GB/s | + + Direct, the same pair gave 9.9 GB/s. The extra 0.13 ms each way is + the threads' wake-ups. A 20 MB/s cap held at 19.98 MB/s. +- **A replica's round trips.** Each row is a connect and hello, then 50 + pings, to a detached session of this build. The session's thread + answers each ping. + + | path | hello | ping min / median / p90 | + |---|---|---| + | direct (Unix socket) | 0.26 ms | 0.01 / 0.01 / 0.01 ms | + | relay, 30 ms RTT | 30.95 ms | 30.26 / 30.34 / 30.40 ms | + | relay, 80 ms RTT | 80.73 ms | 80.25 / 80.31 / 80.39 ms | + | relay, 207 ms RTT | 207.62 ms | 207.22 / 207.35 / 207.41 ms | + | FreeBSD box, `ssh -L` | 411 to 476 ms | 203.2 / 204.9 to 210.4 / 210 to 221 ms | + + The box's rows cover six runs over two passes. In the same passes, a + TCP connect to the box's port 22 took 203.8 to 205.3 ms at best and + 206.4 to 208.3 ms at the median. The relay rows are the second pass; + the first pass had p90s up to 5 ms worse. So a ping through `ssh -L` + and the daemon's handoff costs the RTT and nothing more measurable. + The hello costs two RTTs: + ssh confirms the channel before it forwards a byte. A refused hello + over the link came back in 412 ms, also two RTTs. That is the floor + phase 5 builds on. A replica connects once, so it pays the second RTT + once. +- **Open point 3, rollback: the walker copy.** `zig build + rollback-bench -Doptimize=ReleaseFast` builds two sessions in process: + - typical: a shell with 1 MB of scrollback, three 60 KB files and a + 3 MB one, 40 edits; + - big: 14 panes, an 18 MB file, four 3 MB files and eight small ones, + 8 MB of scrollback, 200 edits. + + For each, it measures the state the digest walk visits (what + speculation can write) and the share of it that is blobs. Blobs are a + file's text and its undo snapshots. `File.setContentSpan` puts a new + buffer in place of the old one at every edit, and a snapshot is a + copy, so both are immutable once made and a copy can pin them. The + walk is then timed again with them taken out. + + | session | walked | of it blobs | left to copy | walk | walk, blobs out | copy (walk + memcpy) | + |---|---|---|---|---|---|---| + | typical | 36.3 MB | 36.2 MB | 80 KB | 3.1 to 3.8 ms | 0.019 to 0.023 ms | 0.020 to 0.024 ms | + | big | 282.5 MB | 282.4 MB | 96 KB | 23 to 27 ms | 0.043 to 0.044 ms | 0.044 to 0.045 ms | + + A copy is about 0.1 to 0.3% of a 16.7 ms frame, so the walker wins: + arena copy-on-write would buy nothing. Three conditions come with it: + - a blob a copy holds must not be freed until the copy is let go, + whether by a refcount or by a free list drained at confirmation; + - caches derived from a blob (`line_starts`, highlights) are also made + anew at each edit, so they can be pinned with it, or remade on + demand, since the geometry is keyed by revision; + - a terminal's history is not in the walk, because under screen-state + terminals it is the authority's pinned rows. + + The walk with the blobs out is the stand-in for a copy walk: phase 2's + dump walk is the real one, and this step re-measures it when it lands. +- **Open point 5, the join writer: not a fork.** The authority's stall, + as median fork time, with 64 MB written by the authority while the + child lives: + + | written memory | Linux 7.2 (THP always) | FreeBSD 15.1 | + |---|---|---| + | 100 MB | 1.8 to 2.7 ms | 4.1 ms | + | 600 MB | 8.6 to 10.8 ms | 14.0 ms | + | 1 GB | 13.3 to 19.6 ms (max 24) | 20.4 ms (max 26) | + | 4 GB (one Linux pass) | 62 ms (4 KiB pages: 162) | 52 ms | + | then 64 MB written under the child | 26 to 32 ms | 19 to 27 ms | + + Linux is two passes on the pinned cores, beside the builds; the box was + idle. The big session forked in process at 431 MB RSS took 3.4 and + 8.3 ms. The user's windows hold 450 and 630 MB. + + The gate, "a join costs the authority under one frame", fails twice: + - fork itself, at 1 GB on both systems (and at 600 MB once a frame is + 8.3 ms, at 120 Hz); + - copy-on-write after it. Every page the authority writes while the + child streams costs a fault, about 1.3 µs per 4 KiB page. The child + lives as long as the dump takes to cross, which is seconds at + 207 ms. + + So the join is the same walker copy as rollback: tens of microseconds + at a step boundary, blobs pinned. A worker thread then serialises the + copy and streams the pinned blobs, which are immutable and so need no + lock, while the authority buffers the events after it for the joiner. + This is the coordinator's alternative, "a metadata copy serialised on + a worker". The authority pays the copy, and the blobs stay pinned + while they stream. + +- **No local change.** tty startup and keys, the phase 1 tip (a4b85cb7, + ReleaseFast) against this, both live at once with their samples + alternating; the p50 ratio new/base: + + | | three A/B runs | two A/A runs | + |---|---|---| + | startup | 1.040, 0.965, 1.024 | 1.053, 0.978 | + | key in a file, to its frame | 1.084, 1.025, 1.027 | 1.025, 1.013 | + | key echoed by bash, to its frame | 0.996, 0.972, 1.010 | 1.015, 0.998 | + + The daemon and the hello are off the hot path, and every ratio is + within the spread of the A/A runs beside it, whose second slot reads 1 + to 5% slow against the same binary. So there is no measurable change. + The absolute times are not comparable with phase 3's 4.73 ms: the + startup p50 was 6.5 to 26 ms here, on two cores shared by both live + sessions and the driver, with builds running on the others. + +What proved wrong in the design: +- "`ssh -L` of the daemon's one socket": there are two. The one that + carries fds cannot cross ssh, and no fd would cross anyway. +- "The authority forks a writer child": fork's page-table copy is over a + frame at 1 GB on both systems, and the copy-on-write tax lasts as long + as the join streams. Once blobs are pinned, a copy of what speculation + can write is about 100 KB, small enough for a join to take one too. +- "When the first speculative event goes pending, the replica copies + views, buffers, ...": a buffer's text never needs copying, since every + edit replaces it. Only its handle and the views are copied. +- "The hello carries a semantics hash (version, features, grammars)": it + hashes every dependency, and leaves the OS out on purpose. +- Pre-existing on the phase 1 tip, and not this line's: test/fs.py's + `mounted-startup` session hangs at startup (its main thread waits on a + futex), the same with the base binary. A ReleaseFast `exit_answers` + run loses its Rclunk race about 1 time in 7 on base, and 1 in 40 here. -- cgit v1.3