diff options
Diffstat (limited to 'docs/next-daemon.md')
| -rw-r--r-- | docs/next-daemon.md | 196 |
1 files changed, 196 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 28449487..8ec76742 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3653,6 +3653,19 @@ same scripts (`.scratch/daemon/final/final2.sh`). ## The remote bugs the user hit +> due to the ping I know it won't be possible to have the same snappy +> feeling as the local version, but these kinds of errors have nothing to +> do with this limitation, you should do it much better, and measure it +> too since I don't want to catch those myself + +The user's window client of the box blinked, lost its panes and jumped its +cursor: each was a divergence, and the rejoin after it opened a fresh +view. An adversarial soak of replica clients (`test/soak.py`, a session +here or on the box, terminal and window clients through a jittered delay +relay, a seeded monkey of a person's actions) found the causes; the fixers +took them by root cause, each with a test that fails before its fix. Ids +are the hunt's (`.scratch/daemon/hunt/findings.md`). + ### fix3: what the replica stream carried in its build's own words A unit of the replica stream is its session's tape, written by @@ -3729,3 +3742,186 @@ Found by the variant hunt (V2, E), with real ReleaseSafe executables. replay even by its own build, on main too: pane 0's message differs a step after a failure is said (seven times in the first runs, none in 15 since). cross_replay.py leaves the setting out. + +### fix2: what a step read that only one side had, and units that ended undrained + +Each of these made every client of a session diverge at the same unit, so +the session's step and its replicas' replay disagreed. None was a lag. + +- **A step read what only the authority holds** (H4, H8, H11, H12, H13, and + the variant hunt's list). The store's closed panes and Recent are the + authority's (`.authority`: every window of the user writes the store's + file); a replica keeps its own, which has no file and only what it + replayed since it joined. Steps made digested state of them: + - the Panes picker's closed rows and Recent's rows (H4: every client + diverged at the key that opened them; H13: a client on another machine + read its own store); + - `trackJump`, at every step, dropped a closed file's jumps when Recent + did not know the file (H11: once a pane had closed, every client that + joined diverged at its first unit, forever, in `jumps njumps jcur`); + - `noteClosed` looked in Recent before it asked the host whether the file + is there (`recentKeeps`, a taped stat), so the session skipped a + syscall the replica made (H12: every Del of a file crashed the client); + - a closed pane brought back (`Store.reopen`: a Look at `@pN`, a Panes + row), a jump to a closed file and the +Jumps rows (`recentByHash`); + - and, the same class: a Look or exec handed to a script reading a pane's + events (`pane.fs.readers`, the 9P server's), Tty9p's socket check, and + the session's config directory and startup file (`opts`, `.local`: + ThemeFile, DumpThemes, Config, DumpConfig, an init file saved). + + **The model:** what only the authority holds stays the authority's, and a + step reads it through `tape.authority` (src/tape.zig): a `value` read, + its answer taped; a replay takes the answer, and a speculation that + reaches one waits for it, as at a host call (one RTT, never wrong). A + replica's own store and Recent are never read by a step. Rejected: the + store and Recent as session state (dumped, digested, copied), which + needs the store's file I/O on the tape call by call (it reads other + windows' records at every step that writes), for the same rows. + - `trackJump` no longer reads Recent: a closed file's jump stays, and the + jump to it drops that file's entries when Recent has forgotten it, as + when the file is gone. + - `noteClosed` asks the host first, on every side. + - The tty toggle, read on every key (which cannot wait a round trip), is + session state: `Pardes.tty_toggle`, from the options when the core is + made, dumped. +- **A rollback put back pointers to freed memory** (H8, the segfault in the + Panes picker). The copy saves a struct's bytes field by field, the + `.authority` ones included, but walks only what the digest walks. The + store and Recent live on the host's heap (`recentGpa`), whose frees the + epoch never puts off: a speculated Del grew the closed list, the + rollback put the old header back, and the next listing read a freed + name. `store` and `recent` are never saved now (rollback.zig `unsaved`); + a step never reads a replica's own. +- **A unit taped outside a step ended with effects queued** (H6, H10). A + leave is the session's `closeView`, whose sync sizes again a terminal the + closed view sized and queues its pty's resize; the session did it at its + next pump, in another unit. A replica that replayed the close kept the + effect queued, and its next idle copy asserted (H10: a client killed when + another left). A client joining at that boundary was dumped with it + queued, which a loaded core has none of, and diverged at the session's + next pump (H6: `the core called poll_frame where the tape has a host + (12)`). `closeView` and `openView` settle inside their own input now. +- **The lowered copy of a searched text held heap bytes** (H5's and H17's + `places.hits|hay`). A smart-case search lowers the text a piece at a time + into `Places.hay`, made with `memory.unfilled`; the digest and a join's + dump take the whole slice, so the pieces not searched yet were whatever + each process's heap held. Zeroed where it is made: 20 µs for 1 MB, + 0.4 ms for 10 MB, 10 ms for 100 MB in one frame (ReleaseFast, page + faults included; lowering the same text is four times that, over + frames), the order of the copy an edit of that text already makes. +- **A view drawn outside a step left the live slot aliasing** (H5's + hypothesis, in a path it did not name). `withParked` (a replica client + drawing or reading its own view while another is live) parked the live + view and copied it back without emptying its slot or its panes' shares + of selections, which the rollback copy walks. Not the cause of H5's + signatures, which were the jumps (Recent) and the places (the lowered + copy) above; the slot was saved and put back all along. + +Tests, each failing before its fix: src/rollback.zig ("a rollback leaves the +store and Recent as a speculation left them", "a view drawn outside a step +leaves the live view's slot holding nothing"), src/replica.zig ("a replica +joined after a pane closed", with the variant hunt's Recent written by +another window; "a replica's Del of a file its session's Recent knows"; "a +client leaves, and one joins"), src/Places.zig ("a lowered copy's pieces not +searched yet are the same in every process"). The hunt's repros +(`panes_closed.py`, `del_file.py`, `recent_store.py` with the client's own +XDG_STATE_HOME, `leave_resize.py`) agree on the fixed builds; on the +installed 0.24 `panes_closed.py` still rejoins 16 times. `zig build +replicas` now looks at files, closes panes, opens the Panes and Recent +pickers and jumps. + +The gate, on fix2's five changes (`.scratch/daemon/fix2/gate.sh`): + +- **Debug:** `core-test` tty; `core-test` gui and `unit-test -Dplatform=gui` + fail only on main's two (`tagline pointer mapping...`, `a tag caret past + its row's end...`); `unit-test` tty, `snap` (113/113), `replay`, + `replicas`, `test/daemon.py` 20, `--far`, `--manage`, `--upgrade`, + `--window-clients`, `--gui` 5 and `test/fs.py` pass. +- **ReleaseSafe:** `test/daemon.py` 20, `--far`, `--window-clients`, + `--gui` 5, `--upgrade` from Debug and `test/fs.py` pass. +- **Rebased onto fix3's line** (myulmrpt; its `tape.results` now lists + what `tape.authority` returns): `core-test` and `unit-test` tty, + `snap`, `replicas`, `test/daemon.py` 20, `--far`, `--window-clients`, + `--gui` 5 and `test/fs.py` pass again in Debug; `test/daemon.py` 20, + `--window-clients`, `--gui` 5 and `test/fs.py` in ReleaseSafe. +- **The copy** (`zig build rollback-bench`, ReleaseFast, main's + rollback.zig against fix2's): a typical session's 980.3 KB to 980.1 KB, + a big one's 1487.9 to 1487.8 (the Store's and Recent's own bytes; their + heap was never saved). Its take 0.045 and 0.063 ms, 0.075 and 0.077; + its restore 0.036 and 0.034 ms, 0.053 and 0.054: noise, one run each. + +The soak (`test/soak.py local --minutes 5 --rtts 0,30,80,207 --clients +tty,win --trace`, ReleaseSafe, the mouse on, Upgrades, resizes, detaches +and scripts among the moves), twice: + +| cell | client | min | inputs | links | visible | mispred | p50 / p99 ms | cut short by | +|---|---|---|---|---|---|---|---|---| +| 0 ms | tty | 1.4 | 330 | 2 | 0 | 0 | 1.5 / 1298 | H7 (session) | +| 0 ms | window | 1.4 | 225 | 3 | 6 | 2 | 4.5 / 12.4 | H7 | +| 30 ms | tty | 0.6 | 101 | 1 | 0 | 0 | 43.2 / 1283 | H7 | +| 30 ms | window | 0.6 | 56 | 1 | 0 | 0 | 4.8 / 25.4 | H7 | +| 80 ms | tty | 5.1 | 962 | 1 | 6 | 30 | 14.4 / 1570 | | +| 80 ms | window | 5.1 | 911 | 3 | 63 | 35 | 5.9 / 117 | | +| 80 ms | window, attached later | 0.8 | 35 | 0 | 0 | 0 | 2.0 / 3.6 | its replay panic (H9) | +| 207 ms | tty | 0.8 | 98 | 1 | 0 | 0 | 42.3 / 1879 | H7 | +| 207 ms | window | 0.8 | 84 | 1 | 0 | 0 | 6.3 / 18.7 | H7 | + +- Every divergence of that run was the pointer's: `pointer_tag_hit + look_hover_wait` (H3, fixer 1's: a tag hit made from the process's own + frame), which takes every client of the session at once, and a window + client's replay that met the `pointer` site where the tape had none (H9's + panic, the same class). None of this section's classes. +- H7 (unassigned: `integer overflow` in ghostty's PageList under + `Screens.flush`, after a unit with a `pty_resize`) ended the session in + three of four cells; the first run's cells, of the same seeds, did not + meet it. +- The first run (its table lost to the harness, which raised when a session + with unsaved text would not Exit; fix2's copy now ends it by pid and + counts a breach): 0 ms, three rejoins on H3 alone; 30 ms, one window + client's replay panic at the `pointer` site; 80 ms, three on H3 and three + of H18's (`jumps … pane N.body` at units with a `pty_resize`, every + client together). +- H17's own configuration (30 ms, seed 22, 4 minutes, where both clients + had died at `n` on different places): no divergence in `places`. Its + rejoins were H3's (five) and H18's (two units, `jumps … pane 0.body` + and `pane 0.body pane 0.ovl`, each with a `pty_resize`). + +Not done, and found on the way: + +- **H18, handed back: a terminal's resize leaves a replica's screen state + stale for a round trip.** Two signatures, both on every client at one + unit (`to_view`, a key, a `step` with a `pty_resize`): + - `pane 2.ovl` (the hunt's seed 5 at 80 ms, deterministic): an edit copy + extended over a three-row terminal read history rows that both + replicas held as blank holes where the session read text, and the + session sent none before the unit (`Screens.fill`). The resize in that + unit was another pane's. An instrumented build showed the step read + the rows through the core's cache (`shellRows`), which marks a + terminal read only when it builds the rows, so `fill` skips it; and + with that fixed the unit still diverged. + - `jumps … pane N.body` (this table's 80 ms cell, H5's own signature). + - The window: a resize renumbers a terminal's rows (`Rows.epoch`); the + session's next delta is against the old base, a replica refuses it + (`Renumbered`) and is started afresh a round trip later, while + `noteSent` took it as applied and expects the replica's holes where + they were. A step that reads the terminal meanwhile, or in the step + that resized it after the reflow, reads another grid on each side. + - The likely fix, untried: a delta across a renumbering is a reset + (`encode` with no base when the base's epoch is another), so the peer + is afresh at once and `fill` knows its holes; `shellRows` marks a read + on a kept hit too; and, with screen-state replicas, the session + reflows at the cut of the step's unit, not inside it, so the step + reads one grid on both sides. A test needs Screens with two replicas + of different sizes and a short terminal with a history. +- **Two edges of the same class, left:** a pipe or an Edit in flight when + a client joins (`Pipe.wait`, `.authority`, not dumped: the joiner drops + its answer), and a 9P batch held at a join (`fs.batch`). Both need a + join in the instant between a request and its answer. +- **A Debug session with `PARDES_REPLICA_TRACE` is no soak:** its per-field + digests rehash every blob at every unit (`walk.rehash`, Debug's check), + a whole undo history of a big file a unit, and a boundary took 62 s. + Soak ReleaseSafe, or Debug without `--trace`. +- **std's segfault handler panics while it symbolizes** (H8's S3: + `SelfInfo.Elf.findModule`, index out of bounds), so a crash record says + that and not the segfault. It is std's, not pardes's (`crash.zig` takes + panics only). |
