summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--docs/next-daemon.md196
1 files changed, 196 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md
index 28449487..8ec76742 100644
--- a/docs/next-daemon.md
+++ b/docs/next-daemon.md
@@ -3653,6 +3653,19 @@ same scripts (`.scratch/daemon/final/final2.sh`).
## The remote bugs the user hit
+> due to the ping I know it won't be possible to have the same snappy
+> feeling as the local version, but these kinds of errors have nothing to
+> do with this limitation, you should do it much better, and measure it
+> too since I don't want to catch those myself
+
+The user's window client of the box blinked, lost its panes and jumped its
+cursor: each was a divergence, and the rejoin after it opened a fresh
+view. An adversarial soak of replica clients (`test/soak.py`, a session
+here or on the box, terminal and window clients through a jittered delay
+relay, a seeded monkey of a person's actions) found the causes; the fixers
+took them by root cause, each with a test that fails before its fix. Ids
+are the hunt's (`.scratch/daemon/hunt/findings.md`).
+
### fix3: what the replica stream carried in its build's own words
A unit of the replica stream is its session's tape, written by
@@ -3729,3 +3742,186 @@ Found by the variant hunt (V2, E), with real ReleaseSafe executables.
replay even by its own build, on main too: pane 0's message differs a
step after a failure is said (seven times in the first runs, none in 15
since). cross_replay.py leaves the setting out.
+
+### fix2: what a step read that only one side had, and units that ended undrained
+
+Each of these made every client of a session diverge at the same unit, so
+the session's step and its replicas' replay disagreed. None was a lag.
+
+- **A step read what only the authority holds** (H4, H8, H11, H12, H13, and
+ the variant hunt's list). The store's closed panes and Recent are the
+ authority's (`.authority`: every window of the user writes the store's
+ file); a replica keeps its own, which has no file and only what it
+ replayed since it joined. Steps made digested state of them:
+ - the Panes picker's closed rows and Recent's rows (H4: every client
+ diverged at the key that opened them; H13: a client on another machine
+ read its own store);
+ - `trackJump`, at every step, dropped a closed file's jumps when Recent
+ did not know the file (H11: once a pane had closed, every client that
+ joined diverged at its first unit, forever, in `jumps njumps jcur`);
+ - `noteClosed` looked in Recent before it asked the host whether the file
+ is there (`recentKeeps`, a taped stat), so the session skipped a
+ syscall the replica made (H12: every Del of a file crashed the client);
+ - a closed pane brought back (`Store.reopen`: a Look at `@pN`, a Panes
+ row), a jump to a closed file and the +Jumps rows (`recentByHash`);
+ - and, the same class: a Look or exec handed to a script reading a pane's
+ events (`pane.fs.readers`, the 9P server's), Tty9p's socket check, and
+ the session's config directory and startup file (`opts`, `.local`:
+ ThemeFile, DumpThemes, Config, DumpConfig, an init file saved).
+
+ **The model:** what only the authority holds stays the authority's, and a
+ step reads it through `tape.authority` (src/tape.zig): a `value` read,
+ its answer taped; a replay takes the answer, and a speculation that
+ reaches one waits for it, as at a host call (one RTT, never wrong). A
+ replica's own store and Recent are never read by a step. Rejected: the
+ store and Recent as session state (dumped, digested, copied), which
+ needs the store's file I/O on the tape call by call (it reads other
+ windows' records at every step that writes), for the same rows.
+ - `trackJump` no longer reads Recent: a closed file's jump stays, and the
+ jump to it drops that file's entries when Recent has forgotten it, as
+ when the file is gone.
+ - `noteClosed` asks the host first, on every side.
+ - The tty toggle, read on every key (which cannot wait a round trip), is
+ session state: `Pardes.tty_toggle`, from the options when the core is
+ made, dumped.
+- **A rollback put back pointers to freed memory** (H8, the segfault in the
+ Panes picker). The copy saves a struct's bytes field by field, the
+ `.authority` ones included, but walks only what the digest walks. The
+ store and Recent live on the host's heap (`recentGpa`), whose frees the
+ epoch never puts off: a speculated Del grew the closed list, the
+ rollback put the old header back, and the next listing read a freed
+ name. `store` and `recent` are never saved now (rollback.zig `unsaved`);
+ a step never reads a replica's own.
+- **A unit taped outside a step ended with effects queued** (H6, H10). A
+ leave is the session's `closeView`, whose sync sizes again a terminal the
+ closed view sized and queues its pty's resize; the session did it at its
+ next pump, in another unit. A replica that replayed the close kept the
+ effect queued, and its next idle copy asserted (H10: a client killed when
+ another left). A client joining at that boundary was dumped with it
+ queued, which a loaded core has none of, and diverged at the session's
+ next pump (H6: `the core called poll_frame where the tape has a host
+ (12)`). `closeView` and `openView` settle inside their own input now.
+- **The lowered copy of a searched text held heap bytes** (H5's and H17's
+ `places.hits|hay`). A smart-case search lowers the text a piece at a time
+ into `Places.hay`, made with `memory.unfilled`; the digest and a join's
+ dump take the whole slice, so the pieces not searched yet were whatever
+ each process's heap held. Zeroed where it is made: 20 µs for 1 MB,
+ 0.4 ms for 10 MB, 10 ms for 100 MB in one frame (ReleaseFast, page
+ faults included; lowering the same text is four times that, over
+ frames), the order of the copy an edit of that text already makes.
+- **A view drawn outside a step left the live slot aliasing** (H5's
+ hypothesis, in a path it did not name). `withParked` (a replica client
+ drawing or reading its own view while another is live) parked the live
+ view and copied it back without emptying its slot or its panes' shares
+ of selections, which the rollback copy walks. Not the cause of H5's
+ signatures, which were the jumps (Recent) and the places (the lowered
+ copy) above; the slot was saved and put back all along.
+
+Tests, each failing before its fix: src/rollback.zig ("a rollback leaves the
+store and Recent as a speculation left them", "a view drawn outside a step
+leaves the live view's slot holding nothing"), src/replica.zig ("a replica
+joined after a pane closed", with the variant hunt's Recent written by
+another window; "a replica's Del of a file its session's Recent knows"; "a
+client leaves, and one joins"), src/Places.zig ("a lowered copy's pieces not
+searched yet are the same in every process"). The hunt's repros
+(`panes_closed.py`, `del_file.py`, `recent_store.py` with the client's own
+XDG_STATE_HOME, `leave_resize.py`) agree on the fixed builds; on the
+installed 0.24 `panes_closed.py` still rejoins 16 times. `zig build
+replicas` now looks at files, closes panes, opens the Panes and Recent
+pickers and jumps.
+
+The gate, on fix2's five changes (`.scratch/daemon/fix2/gate.sh`):
+
+- **Debug:** `core-test` tty; `core-test` gui and `unit-test -Dplatform=gui`
+ fail only on main's two (`tagline pointer mapping...`, `a tag caret past
+ its row's end...`); `unit-test` tty, `snap` (113/113), `replay`,
+ `replicas`, `test/daemon.py` 20, `--far`, `--manage`, `--upgrade`,
+ `--window-clients`, `--gui` 5 and `test/fs.py` pass.
+- **ReleaseSafe:** `test/daemon.py` 20, `--far`, `--window-clients`,
+ `--gui` 5, `--upgrade` from Debug and `test/fs.py` pass.
+- **Rebased onto fix3's line** (myulmrpt; its `tape.results` now lists
+ what `tape.authority` returns): `core-test` and `unit-test` tty,
+ `snap`, `replicas`, `test/daemon.py` 20, `--far`, `--window-clients`,
+ `--gui` 5 and `test/fs.py` pass again in Debug; `test/daemon.py` 20,
+ `--window-clients`, `--gui` 5 and `test/fs.py` in ReleaseSafe.
+- **The copy** (`zig build rollback-bench`, ReleaseFast, main's
+ rollback.zig against fix2's): a typical session's 980.3 KB to 980.1 KB,
+ a big one's 1487.9 to 1487.8 (the Store's and Recent's own bytes; their
+ heap was never saved). Its take 0.045 and 0.063 ms, 0.075 and 0.077;
+ its restore 0.036 and 0.034 ms, 0.053 and 0.054: noise, one run each.
+
+The soak (`test/soak.py local --minutes 5 --rtts 0,30,80,207 --clients
+tty,win --trace`, ReleaseSafe, the mouse on, Upgrades, resizes, detaches
+and scripts among the moves), twice:
+
+| cell | client | min | inputs | links | visible | mispred | p50 / p99 ms | cut short by |
+|---|---|---|---|---|---|---|---|---|
+| 0 ms | tty | 1.4 | 330 | 2 | 0 | 0 | 1.5 / 1298 | H7 (session) |
+| 0 ms | window | 1.4 | 225 | 3 | 6 | 2 | 4.5 / 12.4 | H7 |
+| 30 ms | tty | 0.6 | 101 | 1 | 0 | 0 | 43.2 / 1283 | H7 |
+| 30 ms | window | 0.6 | 56 | 1 | 0 | 0 | 4.8 / 25.4 | H7 |
+| 80 ms | tty | 5.1 | 962 | 1 | 6 | 30 | 14.4 / 1570 | |
+| 80 ms | window | 5.1 | 911 | 3 | 63 | 35 | 5.9 / 117 | |
+| 80 ms | window, attached later | 0.8 | 35 | 0 | 0 | 0 | 2.0 / 3.6 | its replay panic (H9) |
+| 207 ms | tty | 0.8 | 98 | 1 | 0 | 0 | 42.3 / 1879 | H7 |
+| 207 ms | window | 0.8 | 84 | 1 | 0 | 0 | 6.3 / 18.7 | H7 |
+
+- Every divergence of that run was the pointer's: `pointer_tag_hit
+ look_hover_wait` (H3, fixer 1's: a tag hit made from the process's own
+ frame), which takes every client of the session at once, and a window
+ client's replay that met the `pointer` site where the tape had none (H9's
+ panic, the same class). None of this section's classes.
+- H7 (unassigned: `integer overflow` in ghostty's PageList under
+ `Screens.flush`, after a unit with a `pty_resize`) ended the session in
+ three of four cells; the first run's cells, of the same seeds, did not
+ meet it.
+- The first run (its table lost to the harness, which raised when a session
+ with unsaved text would not Exit; fix2's copy now ends it by pid and
+ counts a breach): 0 ms, three rejoins on H3 alone; 30 ms, one window
+ client's replay panic at the `pointer` site; 80 ms, three on H3 and three
+ of H18's (`jumps … pane N.body` at units with a `pty_resize`, every
+ client together).
+- H17's own configuration (30 ms, seed 22, 4 minutes, where both clients
+ had died at `n` on different places): no divergence in `places`. Its
+ rejoins were H3's (five) and H18's (two units, `jumps … pane 0.body`
+ and `pane 0.body pane 0.ovl`, each with a `pty_resize`).
+
+Not done, and found on the way:
+
+- **H18, handed back: a terminal's resize leaves a replica's screen state
+ stale for a round trip.** Two signatures, both on every client at one
+ unit (`to_view`, a key, a `step` with a `pty_resize`):
+ - `pane 2.ovl` (the hunt's seed 5 at 80 ms, deterministic): an edit copy
+ extended over a three-row terminal read history rows that both
+ replicas held as blank holes where the session read text, and the
+ session sent none before the unit (`Screens.fill`). The resize in that
+ unit was another pane's. An instrumented build showed the step read
+ the rows through the core's cache (`shellRows`), which marks a
+ terminal read only when it builds the rows, so `fill` skips it; and
+ with that fixed the unit still diverged.
+ - `jumps … pane N.body` (this table's 80 ms cell, H5's own signature).
+ - The window: a resize renumbers a terminal's rows (`Rows.epoch`); the
+ session's next delta is against the old base, a replica refuses it
+ (`Renumbered`) and is started afresh a round trip later, while
+ `noteSent` took it as applied and expects the replica's holes where
+ they were. A step that reads the terminal meanwhile, or in the step
+ that resized it after the reflow, reads another grid on each side.
+ - The likely fix, untried: a delta across a renumbering is a reset
+ (`encode` with no base when the base's epoch is another), so the peer
+ is afresh at once and `fill` knows its holes; `shellRows` marks a read
+ on a kept hit too; and, with screen-state replicas, the session
+ reflows at the cut of the step's unit, not inside it, so the step
+ reads one grid on both sides. A test needs Screens with two replicas
+ of different sizes and a short terminal with a history.
+- **Two edges of the same class, left:** a pipe or an Edit in flight when
+ a client joins (`Pipe.wait`, `.authority`, not dumped: the joiner drops
+ its answer), and a 9P batch held at a join (`fs.batch`). Both need a
+ join in the instant between a request and its answer.
+- **A Debug session with `PARDES_REPLICA_TRACE` is no soak:** its per-field
+ digests rehash every blob at every unit (`walk.rehash`, Debug's check),
+ a whole undo history of a big file a unit, and a boundary took 62 s.
+ Soak ReleaseSafe, or Debug without `--trace`.
+- **std's segfault handler panics while it symbolizes** (H8's S3:
+ `SelfInfo.Elf.findModule`, index out of bounds), so a crash record says
+ that and not the segfault. It is std's, not pardes's (`crash.zig` takes
+ panics only).