summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--docs/next-daemon.md201
1 files changed, 199 insertions, 2 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md
index e7ddcf5f..efe5214a 100644
--- a/docs/next-daemon.md
+++ b/docs/next-daemon.md
@@ -163,12 +163,17 @@ open.
ordered stream with no per-read round trip; 9P stays for scripting and
the relay]
3. Rollback by walker copy or by arena copy-on-write. [whichever
- measurement of the mutable state's size decides]
+ measurement of the mutable state's size decides] Decided: the walker
+ copy, about 100 KB and 0.05 ms once file texts are pinned (Phase 5
+ groundwork notes).
4. How the daemon starts: spawning itself or socket activation. [itself,
by double fork; FreeBSD has no systemd. A user unit can be added if a
terminal's scope ever kills it]
5. The join writer: a fork child, or a metadata copy serialised on a
- worker. [fork first, measure the page-table copy]
+ worker. [fork first, measure the page-table copy] Decided: the copy on
+ a worker. A fork of 1 GB stalls the authority 13 to 20 ms, and
+ copy-on-write taxes it for as long as the join streams (Phase 5
+ groundwork notes).
## Phase 3 notes
@@ -941,3 +946,195 @@ against that tty, is the Tty throughput notes' keystroke-to-present.
- A replica pushes a flood's rows into its history one at a time.
- The integration into the stream and the cores (a replica pane's
State fed by `apply` instead of `.output`) is the next piece.
+
+## Phase 5 groundwork notes
+
+Built 2026-10-29: the delay relay (`test/delay_relay.py`), the replica
+socket (`src/daemon.zig`, `src/detached/wire.zig`), and two measurements
+for open points 3 and 5 (`zig build rollback-bench`, which runs
+`test/rollback_bench.zig`, and `test/fork_stall.py`). Decisions taken at
+the forks, with the ponytail lens:
+
+- **The delay relay is a Python script.** Each direction is a reader that
+ stamps every chunk with a due time and a writer that sends it then. The
+ due time is the chunk's arrival plus the delay. With a rate cap, it is
+ the moment the link has carried what came before, plus the delay. So
+ throughput is not serialised behind the delay, and the bytes stay in
+ order. Either side is a Unix or TCP socket. ponytail: its buffer is
+ unbounded, which makes a link with an infinite bandwidth-delay product;
+ on a real link, ssh's window is the bound.
+- **Two sockets, where the design had "the daemon's one".** The
+ fd-carrying SEQPACKET socket stays local. Once the daemon holds its
+ lock, it binds `pardes-replica.sock` (SOCK_STREAM) beside it, and
+ unlinks it at exit. `ssh -L` forwards that one.
+- **The daemon judges the hello, so a session never sees one it
+ refuses.**
+ - Every session's registration carries its semantics hash (daemon
+ protocol 2), and the daemon compares the client's with the session's.
+ - It refuses in words: another protocol; another build, naming both
+ hashes and pointing at `pardes --version` to compare them; or a
+ session nobody has.
+ - It reads exactly the hello's length (the header, then the payload),
+ so whatever the client sends after it is left for the session.
+ - Up to 16 hellos may be on their way at once. ponytail: a client that
+ never sends one holds its slot until it hangs up.
+- **The welcome comes from the session**, with its pid and name, over the
+ connection it was handed. That tells the client the handoff happened.
+ The test checks that the session's fd count went up and the daemon's
+ did not.
+- **The semantics hash** (`wire.semantics`) is computed at comptime and
+ covers:
+ - the protocol, the release and the commit;
+ - the platform and the build options;
+ - every dependency the manifest pins, by its content hash (build.zig's
+ `deps` option). The design said grammars, but ghostty and the
+ Unicode tables also make a core step differently, so it takes all of
+ them;
+ - the walker codec's assumptions (pointer width, byte order).
+
+ It leaves out the OS. The main test is a Linux client of a FreeBSD
+ session, and what differs between those is the host's, which the tape
+ carries. The two builds of one tree hash alike (1087257bf166588a on
+ both), and the box's session welcomed the client from here. ponytail:
+ an unstamped build (Debug with `-Dplatform`, so every test build) has
+ no commit, so two dev trees with one release and the same options hash
+ alike.
+- **The hello lost its grid**: a replica's view takes a resize like any
+ other input. A ping (client tag 0x03) and three server tags (welcome,
+ refused, pong) are new. The wire is version 14.
+- **The session's half is a stub**: a thread per replica that sends the
+ welcome, answers pings, and ends at bye or a hangup.
+
+Measured. Timings ran alone through the perf lock, pinned to cores 6-7
+while builds ran niced on the others. The local A/B ran interleaved, with
+an A/A run beside it.
+
+- **The relay's self-check** added this much over a direct connection,
+ at each delay (median RTT, then throughput through it):
+
+ | one-way delay | wanted | added | throughput |
+ |---|---|---|---|
+ | 15 ms | 30 ms | 30.26 ms | 3.2 GB/s |
+ | 40 ms | 80 ms | 80.27 ms | 2.3 GB/s |
+ | 103.5 ms | 207 ms | 207.26 ms | 2.3 GB/s |
+
+ Direct, the same pair gave 9.9 GB/s. The extra 0.13 ms each way is
+ the threads' wake-ups. A 20 MB/s cap held at 19.98 MB/s.
+- **A replica's round trips.** Each row is a connect and hello, then 50
+ pings, to a detached session of this build. The session's thread
+ answers each ping.
+
+ | path | hello | ping min / median / p90 |
+ |---|---|---|
+ | direct (Unix socket) | 0.26 ms | 0.01 / 0.01 / 0.01 ms |
+ | relay, 30 ms RTT | 30.95 ms | 30.26 / 30.34 / 30.40 ms |
+ | relay, 80 ms RTT | 80.73 ms | 80.25 / 80.31 / 80.39 ms |
+ | relay, 207 ms RTT | 207.62 ms | 207.22 / 207.35 / 207.41 ms |
+ | FreeBSD box, `ssh -L` | 411 to 476 ms | 203.2 / 204.9 to 210.4 / 210 to 221 ms |
+
+ The box's rows cover six runs over two passes. In the same passes, a
+ TCP connect to the box's port 22 took 203.8 to 205.3 ms at best and
+ 206.4 to 208.3 ms at the median. The relay rows are the second pass;
+ the first pass had p90s up to 5 ms worse. So a ping through `ssh -L`
+ and the daemon's handoff costs the RTT and nothing more measurable.
+ The hello costs two RTTs:
+ ssh confirms the channel before it forwards a byte. A refused hello
+ over the link came back in 412 ms, also two RTTs. That is the floor
+ phase 5 builds on. A replica connects once, so it pays the second RTT
+ once.
+- **Open point 3, rollback: the walker copy.** `zig build
+ rollback-bench -Doptimize=ReleaseFast` builds two sessions in process:
+ - typical: a shell with 1 MB of scrollback, three 60 KB files and a
+ 3 MB one, 40 edits;
+ - big: 14 panes, an 18 MB file, four 3 MB files and eight small ones,
+ 8 MB of scrollback, 200 edits.
+
+ For each, it measures the state the digest walk visits (what
+ speculation can write) and the share of it that is blobs. Blobs are a
+ file's text and its undo snapshots. `File.setContentSpan` puts a new
+ buffer in place of the old one at every edit, and a snapshot is a
+ copy, so both are immutable once made and a copy can pin them. The
+ walk is then timed again with them taken out.
+
+ | session | walked | of it blobs | left to copy | walk | walk, blobs out | copy (walk + memcpy) |
+ |---|---|---|---|---|---|---|
+ | typical | 36.3 MB | 36.2 MB | 80 KB | 3.1 to 3.8 ms | 0.019 to 0.023 ms | 0.020 to 0.024 ms |
+ | big | 282.5 MB | 282.4 MB | 96 KB | 23 to 27 ms | 0.043 to 0.044 ms | 0.044 to 0.045 ms |
+
+ A copy is about 0.1 to 0.3% of a 16.7 ms frame, so the walker wins:
+ arena copy-on-write would buy nothing. Three conditions come with it:
+ - a blob a copy holds must not be freed until the copy is let go,
+ whether by a refcount or by a free list drained at confirmation;
+ - caches derived from a blob (`line_starts`, highlights) are also made
+ anew at each edit, so they can be pinned with it, or remade on
+ demand, since the geometry is keyed by revision;
+ - a terminal's history is not in the walk, because under screen-state
+ terminals it is the authority's pinned rows.
+
+ The walk with the blobs out is the stand-in for a copy walk: phase 2's
+ dump walk is the real one, and this step re-measures it when it lands.
+- **Open point 5, the join writer: not a fork.** The authority's stall,
+ as median fork time, with 64 MB written by the authority while the
+ child lives:
+
+ | written memory | Linux 7.2 (THP always) | FreeBSD 15.1 |
+ |---|---|---|
+ | 100 MB | 1.8 to 2.7 ms | 4.1 ms |
+ | 600 MB | 8.6 to 10.8 ms | 14.0 ms |
+ | 1 GB | 13.3 to 19.6 ms (max 24) | 20.4 ms (max 26) |
+ | 4 GB (one Linux pass) | 62 ms (4 KiB pages: 162) | 52 ms |
+ | then 64 MB written under the child | 26 to 32 ms | 19 to 27 ms |
+
+ Linux is two passes on the pinned cores, beside the builds; the box was
+ idle. The big session forked in process at 431 MB RSS took 3.4 and
+ 8.3 ms. The user's windows hold 450 and 630 MB.
+
+ The gate, "a join costs the authority under one frame", fails twice:
+ - fork itself, at 1 GB on both systems (and at 600 MB once a frame is
+ 8.3 ms, at 120 Hz);
+ - copy-on-write after it. Every page the authority writes while the
+ child streams costs a fault, about 1.3 µs per 4 KiB page. The child
+ lives as long as the dump takes to cross, which is seconds at
+ 207 ms.
+
+ So the join is the same walker copy as rollback: tens of microseconds
+ at a step boundary, blobs pinned. A worker thread then serialises the
+ copy and streams the pinned blobs, which are immutable and so need no
+ lock, while the authority buffers the events after it for the joiner.
+ This is the coordinator's alternative, "a metadata copy serialised on
+ a worker". The authority pays the copy, and the blobs stay pinned
+ while they stream.
+
+- **No local change.** tty startup and keys, the phase 1 tip (a4b85cb7,
+ ReleaseFast) against this, both live at once with their samples
+ alternating; the p50 ratio new/base:
+
+ | | three A/B runs | two A/A runs |
+ |---|---|---|
+ | startup | 1.040, 0.965, 1.024 | 1.053, 0.978 |
+ | key in a file, to its frame | 1.084, 1.025, 1.027 | 1.025, 1.013 |
+ | key echoed by bash, to its frame | 0.996, 0.972, 1.010 | 1.015, 0.998 |
+
+ The daemon and the hello are off the hot path, and every ratio is
+ within the spread of the A/A runs beside it, whose second slot reads 1
+ to 5% slow against the same binary. So there is no measurable change.
+ The absolute times are not comparable with phase 3's 4.73 ms: the
+ startup p50 was 6.5 to 26 ms here, on two cores shared by both live
+ sessions and the driver, with builds running on the others.
+
+What proved wrong in the design:
+- "`ssh -L` of the daemon's one socket": there are two. The one that
+ carries fds cannot cross ssh, and no fd would cross anyway.
+- "The authority forks a writer child": fork's page-table copy is over a
+ frame at 1 GB on both systems, and the copy-on-write tax lasts as long
+ as the join streams. Once blobs are pinned, a copy of what speculation
+ can write is about 100 KB, small enough for a join to take one too.
+- "When the first speculative event goes pending, the replica copies
+ views, buffers, ...": a buffer's text never needs copying, since every
+ edit replaces it. Only its handle and the views are copied.
+- "The hello carries a semantics hash (version, features, grammars)": it
+ hashes every dependency, and leaves the OS out on purpose.
+- Pre-existing on the phase 1 tip, and not this line's: test/fs.py's
+ `mounted-startup` session hangs at startup (its main thread waits on a
+ futex), the same with the base binary. A ReleaseFast `exit_answers`
+ run loses its Rclunk race about 1 time in 7 on base, and 1 in 40 here.