From 1f5ef5bc77f12b721864c2785d9daedc82947e57 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 03:32:02 -0300 Subject: docs/next-daemon.md: phase 5 part B: the decisions at its forks (the stream, terminals as screen state in rterm's order and the wait's split, the taped prompt, predicted echo wired, the join's state in the step and its blobs streamed pinned, the texts hashed first, the rejoin, an Upgrade's words, ssh -C), the main test on the FreeBSD box, keystroke to present at 0/30/80/207 ms and through the box, echo with and without prediction, the visible rollback rate, the join's cost before and after, the join through the box plain and compressed, the step while a client lags, the local A/B against base and main, the gates, what was found and what is not done (a GUI client first) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 373 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 372 insertions(+), 1 deletion(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 5039c81e..f6400770 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1870,6 +1870,376 @@ Follow-ups (2026-10-29): p99 0.27 ms (0.22 before, within the run's noise); none diverged, none unequal, no delta wrong. +## Phase 5 part B notes + +Built 2026-10-29 and 30: the stream over the replica socket (src/replica.zig +`Peers`, and its client half, `dial` and `readStream`; detached/wire.zig +version 15), the client process (`pardes --attach=:` and +`--attach=:`, tty.zig `remote`), the session's side of it in +the tty host, rterm's screen-state terminals and the echo agent's +predicted echo carried by it, a client's rejoin, and test/daemon.py's +replica clients. Decisions taken at the forks, with the ponytail lens: + +- **A client's input is a unit of its own, settled by `step`.** On the + authority, at the step boundary after each pump (`Peers.boundary`), each + input that came is `toView(its view)`, `update(it)` and `step()`: the + middle of a pump (queued inputs and effects), with no wait, no clock + read and no frame. The client's speculation is the same three calls, so + the two tapes read alike and the fast path takes the unit as it came. + With the pump in its place (part A's harness) every speculation stopped + at the tty host's `wait_input` and read the clock in ns. +- **The stream.** Framed as wire.zig frames everything, a tag and a u32le + length: + - the session's: 0x01 welcome (pid, name, the client's id and view, the + dump's length, the session's offset from UTC, its digest as dumped + and its build's semantics), 0x02 refused (words), 0x04 a piece of the + dump (1 MiB), 0x05 a unit (origin, seq, the authority's realtime and + monotonic clocks when it was made, its tape), 0x06 a terminal's + message (rterm's: a delta or rows), 0x07 a check's answer; + - the client's: its inputs as the existing event tags (their seq is the + n-th on the connection), 0x04 a terminal's message (rterm's: an + acknowledgement, rows asked, a reset), 0x05 check. + + A unit carries one digest, at its end (`unit_digests`); under screen + state it is one word (`whole_digest`, PARDES_REPLICA_TRACE for one a + field). Blobs are hashed once and folded in as a word (main's + walk.zig), so a digest a unit is cheap. The draft's lengths-only digest + went with that. +- **Terminals cross as screen state, in rterm's order.** The session's + Peers keep a `Screens`, a peer each. At each boundary: the rows the + pump's steps read that a client holds as holes (`fill`, then `filled`), + the pump's unit, then the deltas (`flush`). Before each client input's + step its deltas again, after it the rows it read, then its unit. A + joiner's terminals start afresh after its dump. The client applies a + delta or rows in stream order, acknowledges, and asks for the history + rows a frame shows that it lacks. Units carry no pty bytes. + - **A wait takes one step of a person's input, and none after a pty's + output** (tty.zig `split`), while clients replicate: a step reads the + terminals as the deltas after each unit bring them, and in a pump that + took a pty's output and then a key, the key would read on the session + what the client has only after the unit. The next is held for the + next wait. Only while a client is attached. + - **A step that reads a terminal's prompt after its output is taped** + (tape.zig `.prompt`): the first prompt's greeting (`ls`, after a pty + resize) and a command waiting for the prompt. Found at once by a + client opening a shell of its own: the session's layout after the + shell's first output saw the prompt, and the client's replay, its + emulator a delta behind, did not. + - A dump-loaded client's emulators are on the C heap, as + `terminal.create` makes a screen replica's (walk.zig `.alloc` of a + pane's `terminal`, dump.zig `terminal_gpa`). +- **Predicted echo is the echo agent's** (src/echo.zig), wired as its + notes say: the client's core points at its replica's `echo`, a frame + judges it before drawing, and a wake comes a frame later while + predictions are out. PARDES_NO_ECHO turns it off. +- **Join, at a step boundary, the step paying for the mutable state + alone.** The view is opened and the session dumped without its blobs + (walk.zig `Dump.deferred`: a `.blob`'s texts of 4 KiB and up, noted + where they go) and without its terminals' emulators (their size: the + first delta brings each). The client's writer thread writes the + welcome, then the dump with the blobs put back where they go: what the + client reads is `dumpBytes`' dump byte for byte but for the terminals + (a test splices one of a session without any and compares), and the + client is unchanged. Each blob is pinned until it is + written (memory.zig `pin`: its free is put off; a blob is never written + in place and is freed raw). The texts the session never digested (it + records nothing until a client comes) would be hashed by the join's + first digest, 250 MB for a 100 MB file and its undo: they are hashed + first on a thread of their own (walk.zig `unhashed`, `makeWords`), + pinned meanwhile, and the join waits a boundary for them; a second + round joins whatever is left. +- **A joiner checks its load.** The welcome carries the session's digest + as dumped; a client whose load differs says so and ends. The first unit + after a join says the host and its writes again (`tape.resync`), which a + core made from a dump has not. Found on the way: the dump's reader took + the first unit with the dump's last piece and dropped it (the client + diverged at its first digest about one join in three); what it read past + the dump is the stream's now (`Framer`). +- **A diverged client joins again**, the same join as the first: its link + ended, the session dialled again, its dump loaded, its view's size sent, + in the same terminal, and a line over its frame until its next key says + why ("joined again: this client diverged from the session (...)"). It + gets a new view: the old one went with its link. PARDES_TEST_DIVERGE= + puts a client's core off the session's (a digested counter) once quiet + after its n-th unit. A divergence shows only at a unit the client + replays (another's input, or its own it did not apply): the fast path + compares records, not digests. +- **An Upgrade lets its clients go in words** (`Peers.refuseAll`, before + the dump): each is sent `refused` ("the session is upgrading to another + build of pardes") after what it was sent, for up to 200 ms, its view + closed. The client says it in its line and dials again, every 100 ms + for 20 s (the exec, the load, the daemon's own exec). A daemon's + refusal for another build, or a welcome whose semantics are not its + own (the daemon judged the hello by the hash it last heard, which an + upgrade under way may not have said yet), ends it with "session + upgraded to a newer pardes: upgrade this client too". "A newer pardes + is installed" is now a Host call the core makes in its focus step + (`newer_build`), so a session's clients take the line from the tape. +- **The session's clock, guessed.** A unit carries the authority's clocks + when it was made; a speculation's clock reads answer the last unit's + reading plus the time since it came plus the client's round trip + (`tape.Guess`): a file's mtime (realtime seconds), a message's time of + day (in the session's zone, its offset in the welcome) and a message's + monotonic ms. Never the client's own wall clock. +- **Mostly no rollback for the session's own pumps.** Every client input + woke the session's pump before its own unit, and that pump advanced the + clock: a foreign unit, a rollback a key. Two changes: a pump woken only + by a replica's post reads no new time (`mail_only`: the tty's `now` + answers the core's own), and a replica keeps what it applied over a + unit of another's that ends on the digest the last one ended on + (`neutral`; such a unit is replayed when the copy is next put back). A + unit that says what the host wrote, or which host it is, is never + neutral. +- **The copy is taken while idle.** The client takes its rollback copy + after drawing, when nothing is queued (`Replica.ready`), and lets it go + when a unit comes first: a key no longer pays a megabyte's copy (memcpy + was a third of the client's profile). The epoch's and the copy's maps + hash an address by a multiply, not Wyhash (the other sixth). +- **Each client a view, drawn by its own process.** `Pardes.undrawn` marks + a replica client's view: the session's pump and `drawOthers` skip it. + The client draws its own view between steps (`views.drawOne`), presenting + through the tty Shell's own `present`; what that writes in the core (a + frame skipped, one owed) is put back. A drawn parked view gives its + selections back to its slot (`drawParked`). +- **The client is the tty host with its inputs fed elsewhere.** Shell + gains `remote`; `feed` sends a terminal's input to the replica engine + and the session instead of to the core; quit, hangup and the size are + the link's. Its replica core's allocator is the rollback epoch; images' + and PDFs' are the C heap's, whose workers allocate on threads of their + own. ponytail: a picture's pixels freed while a copy is held are not + put off. +- **Every kind of input is speculated** but one into a terminal (rterm's + rule: its step reads the emulator), a view's own (a resize, the + pointer) too. +- **`unattended` is a watched field**, and a session with a client is + attended: its messages go by keys, not the clock. +- **Detach in a client ends that client**, a client's death too (its + link's end); the session and the others carry on. A Restore lets every + client go (they attach again). +- **A lagging client never holds the session up.** Each client has a + writer thread and a queue the session appends to (units coalesce there + into one write); one whose queue passes 64 MiB is cut off. rterm's + deltas skip states within that (32 in flight). +- **ssh: a terminal asked for, not -N.** `--attach=:` runs + `$PARDES_SSH -tt -q -L : 'exec sleep ...'`: only an + interactive connection gets TCP_NODELAY from ssh and sshd. With -N, + Nagle held each key of a burst for the last one's ACK, and a shell's + echo through the box took two round trips (413 ms) where it takes one. + `$PARDES_REMOTE_RUNTIME` names the remote runtime directory when it is + not this one's path. +- **`ssh -C` by default** when `$PARDES_SSH` is not set (it is then the + whole command). Through the box a 3.9 MB join took 7.8 to 7.9 s plain + and 5.4 to 5.8 s compressed. Of a join's time ssh's own forward is 2.0 s + (connect, key exchange, authentication, a pty) and opening a channel + 380 ms; a ControlMaster in the user's ssh config would skip the first. + A rejoin keeps its forward. + +Measured. ReleaseFast; timings alone on the perf cores (lock.sh), the +box's through its own link (207 ms, 170 to 400 KB/s from it). The harness +scripts are in `.scratch/daemon/remote/t2/`. + +- **The main test** (`e2e_box2.py`): a session on the FreeBSD box with a + 400-line file, two replica clients here through `ssh -L` each, and a + terminal on the box itself attached as a view (`pardes --attach` there + over `ssh -tt`). Each types into the file; A and B move and scroll; A + opens a shell, runs a command in it and exits it; B is hung up; A and + the box's terminal detach. A's and B's digests were the session's at + every quiet point (every check agreed), the session's text had + all three's words, the shell's answer was on A's screen 214 to 224 ms + after Enter, and nothing was left on the box once its idle daemon had + gone (a few seconds). The clients' first frames + came 3.4 to 4.0 s after they started (ssh's forward, below). +- **Keystroke to present on a client, local-only actions** (base's + `pardes-perf-live --live`, its keys on a 50k-line file; the client of a + session detached on that file, through test/delay_relay.py at each RTT + and through the box's link; two rounds interleaved; p50 / p99, µs): + + | an idle machine, p50 (p99), µs | key-down | key-right | edit-char | page-down | wheel | + |---|---|---|---|---|---| + | a local session, this line | 2070 / 2067 (2926) | 1507 / 1502 (1895) | 4744 / 4708 (7141) | 1997 / 2033 (5486) | 2098 / 2126 (2885) | + | a local session, base | 2059 / 2068 (2678) | 1517 / 1536 (2391) | 4755 / 4695 (7464) | 1916 / 1927 (5441) | 2100 / 2108 (2642) | + | client, direct | 1903 / 1917 (6813) | 1644 / 1650 (2136) | 4477 / 4413 (7912) | 1994 / 2244 (7405) | 1934 / 2063 (6793) | + | client, relay 30 ms | 1849 / 1838 (6910) | 1576 / 1557 (2695) | 4465 / 4300 (15560) | 1755 / 1712 (7682) | 1860 / 1757 (7163) | + | client, relay 80 ms | 1799 / 1753 (7281) | 1532 / 1501 (2034) | 3085 / 2654 (13705) | 1778 / 1743 (6386) | 2106 / 1870 (7692) | + | client, relay 207 ms | 1778 / 1814 (6721) | 1453 / 1478 (1887) | 4309 / 4333 (18092) | 4640 / 1745 (7562) | 1963 / 1978 (7315) | + | client, the box, ssh -C | 1773 (6947) | 1471 (2138) | 4441 (14018) | 3219 (7503) | 2163 (7692) | + + The same harness while builds ran on the other cores (the line before + its last two changes, which do not touch a client's keys; one round): + + | builds on the other cores, p50 (p99), µs | key-down | key-right | edit-char | page-down | wheel | + |---|---|---|---|---|---| + | a local session, this line | 464 (574) | 343 (417) | 1190 (1714) | 370 (1251) | 471 (638) | + | a local session, base | 463 (629) | 349 (499) | 1188 (1663) | 399 (2078) | 2096 (2591) | + | client, direct | 433 (1266) | 389 (441) | 1381 (2071) | 1183 (1400) | 451 (1360) | + | client, relay 30 ms | 413 (1243) | 364 (537) | 1332 (4083) | 1163 (1254) | 421 (1293) | + | client, relay 80 ms | 421 (1379) | 357 (500) | 1104 (1815) | 1143 (1359) | 1112 (1487) | + | client, relay 207 ms | 398 (1228) | 339 (444) | 1145 (6882) | 1151 (1263) | 615 (1469) | + + - **A client's local-only key is as fast as a local session's at any + RTT, the box's link included**: its p50s are the local session's, + whatever the delay. Its p99 is 1 to 3x the local one's: the rollbacks + (another's unit before its pending inputs; p99 1.6 to 17 ms in the + client's own count) land between some keys and their frames. + - **The machine's state moves every cell together** by up to 4.5x: on + an idle machine the cores sleep deeply between keys (the `powersave` + governor), and with builds on the other cores they do not; in a run + made while one build ran, half the cells were in each state. Within + one run, base and this line's local session read alike, and the + clients alike with them. + - Page-down on a client costs 1.15 ms where a local one costs 0.37 when + the machine is awake: a client draws outside the pump, so the render's + paint-ahead (`Stream.askAhead`, asked of the host after a pump's + present) is never asked, and the rows a page-down brings are + highlighted in its frame. + +- **Terminal echo** (`echo2.py`: a key written to the client's terminal, + to the first output holding its glyph; 30 keys; p50 / p90, ms): + + | | file | shell, predicted | shell, not predicted (PARDES_NO_ECHO) | + |---|---|---|---| + | a local session, base | 0.5 / 0.5 | | 0.8 / 0.8 | + | a local session, this line | 0.5 / 0.5 | | 0.8 / 0.8 | + | client, relay 30 ms | 0.6 / 0.6 | 0.8 / 0.9 | 31.4 / 31.4 | + | client, relay 80 ms | 0.5 / 0.5 | 0.5 / 0.8 | 81.4 / 81.6 | + | client, relay 207 ms | 0.5 / 0.5 | 0.5 / 0.5 | 208.4 / 208.6 | + | client, the box, ssh | 0.5 / 0.6 | 0.5 / 0.6 | 206.9 / 207.4 | + | client, the box, ssh -C | 0.5 / 0.5 | 0.5 / 0.6 | 215.8 / 216.9 | + + Predicted, a key into a shell at a prompt shows as fast as a local one + (it is drawn by the client's own frame); not predicted, it is one round + trip and a millisecond and a half. Before this round (the draft's) the + echo through the box was 211 ms p50. +- **The visible rollback rate.** A frame drawn after one of the client's + own inputs was recorded otherwise is counted as a rollback that may have + been seen (an upper bound: the frame shows the inputs after it too). + In the keys harness, of 669 inputs a run (two runs each): none at 0 + and 30 ms; 2 and 3 (0.4%) at 80 ms; 11 and 13 (1.6 to 1.9%) at 207 ms, + of 25 and 29 mispredicted (a key the session recorded otherwise than + the client applied it). Every other rollback (another's unit before the + client's pending inputs) is put back and replayed before the next frame. +- **Join cost on the session** (`join2.py`: three clients join a session + one after another, through the Unix socket; the session's worst + boundary, which holds the join, and the client's first frame): + + | session | its dump | of it, made in the step | worst step, before | worst step, after | first frame | + |---|---|---|---|---|---| + | typical: 4.1 MB of files, 1 MB of scrollback, 6 panes | 8.1 MB | 113 KB | 12.4 ms | 1.6 ms | 22 to 46 ms | + | big: 38.6 MB of files, 8 MB of scrollback, 14 panes | 75.6 MB | 122 KB | 51.3 ms | 1.5 ms | 87 to 117 ms | + | a 100 MB file beside typical's (130 MB of files) | 254 MB | 114 KB | 172.5 ms | 1.6 ms | 247 to 288 ms | + + Each join itself took 0.3 to 1.4 ms of a step. The dump is about twice + the files: their undo snapshots are blobs too. +- **Join through the box** (`boxjoin.py`: a session there with five of the + tree's source files open, 3.5 MB, and a shell with 1 MB of scrollback; + a 3.9 MB dump; two joins each): the client's first frame 7.8 and 7.9 s + through plain ssh, 5.4 and 5.8 s through `ssh -C` (the default now); the + session's worst step 0.9 ms. A 16 KB session's first frame takes 3.4 to + 4.0 s of which ssh's forward is 2.0 s and its channel's opening 380 ms. +- **The session's step while a client lags** (`lag2.py`: a shell printing + a line a millisecond for 12 s, client A typing a key every 30 ms, client + B stopped with SIGSTOP and cut off at a 256 KB lag; µs, p50 / p99 / + max): A's inputs 47 / 150 / 213 and the boundaries 66 / 208 / 681 with B + stopped (cut off: yes), against 60 / 134 / 309 and 69 / 276 / 4072 with + B running. +- **Digests at quiescence**: every client of every test above ended on + the session's digest (PARDES_REPLICA_CHECK): the main test's, the + clients test's four (one rejoined after a forced divergence), and the + upgrade test's client that joined again. + +- **No local regression.** With no client attached the recorder costs an + atomic load a pump (`Peers.boundary`), a flag test an event (`split`) + and one a raw free (`memory.pinned`); a session's own core has no echo. + ReleaseFast, interleaved base new base new base on the perf cores, the + median of each side's runs, new/old, beside an A/A of the old: + - keystroke to present (base's `pardes-perf-live --live`, against base, + present / core): + + | | new/base | A/A | + |---|---|---| + | key-down | 0.98 / 0.98 | 0.99 / 1.01 | + | key-right | 0.86 / 0.84 | 1.00 / 0.99 | + | edit-char | 0.76 / 0.86 | 1.01 / 1.00 | + | page-down | 0.81 / 0.81 | 0.91 / 0.93 | + | wheel | 0.89 / 0.88 | 0.96 / 1.00 | + | `yes \| head -c 32M` wall | 0.98 | 1.06 | + | a 1M-line open | 1.03 | 1.32 | + + - the core in process (`pardes-perf --reps 3`, each tree's own build of + it), against main (ykrsossp): the keys, page-down and wheel 0.92 to + 1.08 over the small, medium and large files; the edits 0.91 to 1.05; + the opens 0.93 to 1.02; body-2m 1.02 to 1.26 (small's 1.26 alone); the + scrollback cells 0.93 to 1.05. The A/A read 0.79 to 1.46 (most within + 0.9 to 1.1). Against base the keys read 1.11 to 1.20 and the + scrollback renders 1.68 to 2.18: main's since base (this line is level + with main), and base's harness is an older perf.zig, so that pair + measures the harness too. + +The gates, once at the end (Debug unless said): +- `zig build unit-test -Dplatform=tty` passes, its perf gate too; + `-Dplatform=gui` 1563 of 1575, 10 skipped, and the two that fail on main + already (`tagline pointer mapping...`, `a tag caret past its row's + end...`). +- `zig build snap`: 113 of 113. `zig build replay`: the 113 scripts and 20 + monkey seeds, none panicked or diverged. `zig build replicas`: 4 seeds + at each RTT, none diverged, none unequal, no delta wrong. +- `test/daemon.py` (20 cycles, its replica clients' test included: four + clients, one through the relay, one rejoined after a forced divergence, + one a shell's, a terminal attached while a client's view was live), + `test/daemon.py --upgrade` (an unstamped build to a stamped one, its two + replica clients included) and `test/fs.py` (49 s) pass. +- core-test's new tests: a join's dump with its blobs spliced back is the + session's dump byte for byte; a pinned blob's free is put off until its + last pin goes; and the replica tests (`-Dtest-filter=replica`). + +Found on the way: +- **A terminal attached while a client's view is live drew nothing.** + The tty's attach took the live view's screen, and a client's view has + none in the session: it returned without a word, its stub waiting (the + main test's terminal on the box). It takes the session's own now; the + clients test attaches one while a client's view is live. +- **The session's layout read a terminal's prompt after its output** + (above, `.prompt`): a client opening a shell diverged at its first + prompt. +- **A divergence shows only at a unit the client replays.** The fast path + compares a speculation's records with the unit's, not digests, so a + client off the session's state that only types on stays off until a + unit it did not apply comes (another's input, or its own it waited + for). The forced-divergence test has another client type. +- **Harness traps** (the scripts, not pardes): a terminal on the box + through `ssh -tt` from a pty must have its size before ssh starts, else + the remote pty is 0 by 0 and the session draws nothing in it; an Esc + and the next key written within the input parser's lone-Esc window read + as an Alt chord; a ReleaseFast build stamped with another commit has + another semantics hash than the box's of the same tree (`-Dstamp-commit + =false` for both); keys a test types into a client whose live pane is a + shell in tty mode go to the shell (end it with a hangup). + +Not done, and why: +- **A GUI client**, and a window session serving replicas (refused in + words). The GUI host feeds its core from 52 places, each to route + through the replica as the tty's `feed` does; its frame loop is the + pump's (the client draws between steps, outside it); and its own + presentation (the window's surface, its animations) would have to come + out of the core's step as the tty's present does. A window client of a + tty session is the main test's other half; it is the next piece. +- Paint-ahead on a client (above: page-down 1.15 ms where a local one is + 0.37): its draw would ask for it as a pump does, the worker's rows taken + in as the client's own events, outside the session's tape. +- A rejoin gets a new view: the old one's layout and focus go with its + link. A rejoin that kept the view would take the view's number in the + hello. +- A notice of the session's own while clients replicate goes through the + tape (the upgrade's line); a host that writes core state outside a step + (the GUI's notice) would diverge them, and the GUI serves none. +- The join waits for the session's texts to be hashed once a session (its + first join, or after new texts): the joiner pays some milliseconds + more, never the session. +- A held paste (`split`) is dropped if the session ends with it held. +- A `--attach=:` client ended by a hangup leaves its forward's + socket file in $TMPDIR (its ssh goes with the terminal; the unlink is on + the normal way out only). + ## Phase 6: upgrade notes Built 2026-10-29 (`src/upgrade.zig`, the tty and GUI hosts, `daemon.zig`, @@ -2543,4 +2913,5 @@ Not done, and why: harness checks in the rendered surface, and its build compiles and passes the echo tests. - The remote client (the other line) wires it: `initDump`'s core, the - frame hook, its `moved`. + frame hook, its `moved`. (Done since: Phase 5 part B notes, with the + echo through the FreeBSD box's link.) -- cgit v1.3