From a73697397ee3f2b7fa7cba08c96f6f81ff7ee04c Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 10:03:35 -0300 Subject: docs/next-daemon.md: the daemon and multi-client design, argued 2026-10-29 (deterministic replicated core, kakoune-style views, one daemon per user, phases) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 docs/next-daemon.md (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md new file mode 100644 index 00000000..8bc9b38c --- /dev/null +++ b/docs/next-daemon.md @@ -0,0 +1,171 @@ +# The daemon: one pardes per machine, many clients per session + +> I want to have a global pardes daemon ... holding file descriptors for +> ptys, files, etc. ... like kakoune or emacs, or as in plan9 terms a +> cpu+file server ... the main test: multiple clients to a single session, +> and a client here and the session on freebsd and very deep and thorough +> performance testing and optimizations to make it feel like its local and +> snappy. + +Designed 2026-10-29 in discussion (the user, the coordinator, a survey +agent). Nothing here is built yet. Paths are relative to the repo. + +## Decided by the user + +- **A deterministic core, replicated.** The session is the authority: it + orders every client's input and every host result, and only it performs + effects. Each client runs a full core replica, applies its own input at + once, and rolls back and replays when the order differs. Visible rollback + is acceptable. +- **Kakoune-style clients.** Each client has its own view (layout, focus, + selections, scroll) over shared buffers and terminals. Registers are + session-wide. A terminal shown at two sizes is sized by its last typist. +- **Performance order.** First, the regular tty and GUI get no slower. + Then the detached and remote clients reach that local performance, with + the session on the FreeBSD box included. +- **A host step may show up one RTT late, but is never wrong.** This + covers Look, open and exec, as sam's hostlock does. +- **Join is complete.** The joining client pays for it in time; the + authority and the other clients do not. +- **One long-lived daemon per user per machine.** It supervises one child + process per session, postgres-style. Nothing is left dangling, and the + core is not de-globalised. + +## What today's seam is, and why it goes + +`src/detached/server.zig` is a third Host: it runs the core and the ptys +in a single poll loop and sends cell frames to the frontends, at the +smallest common grid across them. + +- **The frames are mostly not diffed.** Every layer's cells are re-sent on + each frame (wire.zig:734, :755). +- **Several things never cross the wire.** Images, PDF pixels, rail ticks + and pane animations stay behind (surface.zig:483-509). +- **Every keystroke is a round trip,** which no amount of optimisation can + make feel local at 30-80 ms. +- **Each remote 9P operation dials afresh,** costing about 6 RTTs + (9p_io.zig:2361-2385). + +## The design + +- **Processes.** + - The daemon is subreaper, spawner, fd holder, registry and 9P root. + - Every session is a child of the daemon: a GUI window, a tty session + behind a stub, or a headless session. + - A session is always an in-process authority with its own local view, + so the regular path does not change. Replicas exist only for extra + clients or remote ones. +- **The tty stub.** + - `pardes` in a terminal passes its terminal fd to the daemon + (SCM_RIGHTS), along with cwd, env and argv. + - The session child drives that fd directly. The stub forwards SIGWINCH + and exits with the session's status. + - Detach: the session drops the fd and the stub exits. No fork happens. +- **GUI detach.** The window is destroyed and the process carries on + headless. Closing a window that has no other client attached quits it. +- **Shells survive a session crash.** After each `forkShell`, the session + sends the daemon a dup of the pty master, so the shells do not get + SIGHUP. Being subreaper only decides who reaps them. +- **The recorder.** It is null when no client is attached. When one is, + it appends what the authority consumed, in order: + - every step's host tape (the results of the ~30 `turn.yield()` sites); + - each `serveFs` mutation. + + Turn order is log order, so the local path changes by one null check. +- **The protocol.** One ordered stream: events, host tapes, digests and + terminal state. Framing, the hello and the Event codec are kept from + wire.zig; the frames go. +- **Terminals cross as authoritative screen state,** not as pty bytes: + - metadata plus immutable row blobs, assigned in absolute terms; + - scrollback fetched on demand. + + A `cat` of 1 GB costs screens, not bytes. ghostty stays exact on the + authority, and the local path keeps reading ghostty directly. +- **The reflective walker.** One comptime `@typeInfo` walk gives: + - the digest; + - the complete dump, which also serves join, Restore and Handover; + - the rollback copy. + + Fields are annotated local, authority-only or derived. A hash of the + type graph versions it. The hello carries a semantics hash (version, + features, grammars), and a mismatched build is refused. +- **Rollback.** + - When the first speculative event goes pending, the replica copies what + speculation can write: views, buffers, registers, counters and terminal + metadata. Blobs are pinned rather than copied. + - On a divergence it restores that copy, then replays the confirmed + events and its own pending ones. + - The authority never rolls back. +- **Join.** The authority forks a writer child at a step boundary; the + child streams the dump. The authority buffers the events after it for + the joiner. +- **Remote.** + - `ssh -L` of the daemon's one socket. + - The daemon hands a replication connection's socket to its session. + - A remote daemon mounts as `/n/`. + +## Determinism leaks (the survey's census) + +| Leak | Where | Fix | +|---|---|---| +| Render writes core state | body_layer.zig:181-187, File.zig:1923-1957 | view geometry in a lazy cache, render pure; the worst leak | +| 9P mutating between steps | pardes.zig:6060, pump :6016 | sequenced; turn order is log order | +| Host syscalls in a step | the ~30 `turn.yield()` sites | host tape | +| Wall-clock budgets | Names.zig:150, Places.zig:480, Tables.zig:519 | tape the work done | +| Ticks off the host clock | pardes.zig:6014/7850 | session ticks stamped, presentation ticks local | +| Clock and TZ in messages | Messages.zig:129-145 | event stamp, session TZ | +| Worker results polled in render | image.zig:1103, pdf_view.zig:1991, Stream.zig:482 | consumed at step boundaries, sequenced | +| getenv in the core (~30) | e.g. fs.zig:798 | env captured once into session state | +| Frontend kind / build flags | `windowed`/`native_images`; 223 comptime branches | moved into View; semantics hash | +| ghostty media reads, its pages outside the arenas | terminal.zig:629, PageList.zig:580 | authority-only under screen-state terminals | + +## Phases + +Each phase ships on its own and keeps `zig build perf --base` level (with +an A/A run). All 110 snapshots stay green throughout. + +1. **Determinism and the replay-divergence harness.** + - Work: the recorder and tape; the leak fixes, the render one first; the + semantics hash; the walker's digest. + - Check: replay each recorded log into two cores at different addresses + and one headless core. + - Gate: zero divergences on the snapshot scripts and on N monkey seeds. +2. **A complete dump on the walker.** It replaces dump.zig for Restore and + Handover. + - Gates: `digest(load(dump(x))) == digest(x)`, and dump is a fixpoint. +3. **The daemon and the tty stub.** + - Work: in-place detach and reattach; pty master dups; the frame wire + deleted. + - Gates: keystroke-to-present unchanged; a session `kill -9`'d leaves + its shells adoptable; no stale sockets after N cycles. +4. **Views.** A second window on the same session, in one process. + - Gate: logs with several views replay without divergence. +5. **Replicas.** + - Work: speculation, rollback, screen-state terminals, join, socket + handoff, `ssh -L`. + - Gates: + - a 30/80 ms delay relay keeps remote within noise of local; + - the rollback rate stays low under the fuzzer; + - a lagging client leaves the authority flat; + - a join costs the authority under one frame. +6. **Remote amortisation.** A content cache by hash, prefetch, zstd, + predicted echo in TTY mode, and session upgrade by re-exec. + +New measurements to add before phase 1 lands: keystroke-to-present in the +tty and the GUI, `yes | head -c 1G` through a real pty, and a big-file +open. + +## Still open (the coordinator's position in brackets) + +1. Delete the frame wire at phase 3, leaving no remote attach until + phase 5. [yes: it was never used daily] +2. The stream as its own framed protocol, or as a 9P file. [framed: one + ordered stream with no per-read round trip; 9P stays for scripting and + the relay] +3. Rollback by walker copy or by arena copy-on-write. [whichever + measurement of the mutable state's size decides] +4. How the daemon starts: spawning itself or socket activation. [itself, + by double fork; FreeBSD has no systemd. A user unit can be added if a + terminal's scope ever kills it] +5. The join writer: a fork child, or a metadata copy serialised on a + worker. [fork first, measure the page-table copy] -- cgit v1.3 From 9be47b76e4edf4ddb56e4d572c14b54bf45fdd39 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 10:48:56 -0300 Subject: docs: the daemon, Detach, --attach and Adopt; phase 3 notes in docs/next-daemon.md building.typ's detached sessions are the daemon and its sessions; the guide, cheatsheet and reference say what Detach, a closing terminal, --attach of each kind and Adopt do, and that attaching from another machine waits for phase 5. docs/next-daemon.md records the forks taken (sessions register and are not the daemon's children, and why; how it starts; masters; Adopt as a word; the stub's signals and job control; --detach; the Attach word; the harness switch), the known gaps, and the termios trap. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++ docs/typ/building.typ | 55 ++++++++++++++++------------ docs/typ/reference.typ | 22 ++++++------ src/CHANGELOG.md | 23 ++++++++++++ 4 files changed, 165 insertions(+), 33 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 8bc9b38c..d164eced 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -169,3 +169,101 @@ open. terminal's scope ever kills it] 5. The join writer: a fork child, or a metadata copy serialised on a worker. [fork first, measure the page-table copy] + +## Phase 3 notes + +Built 2026-10-29 (`src/daemon.zig`, the tty and GUI hosts, `main.zig`). +Decisions taken at the forks, with the ponytail lens: + +- **Sessions are not the daemon's children; every one registers.** The + design had the daemon spawn a terminal's session from the fd, cwd, env + and argv the stub passes. That proved wrong here: + - the user's terminals each run in a `9ns --mntgen` user and mount + namespace (`unshare(CLONE_NEWUSER|CLONE_NEWNS)`); a session the daemon + spawned would run in whichever namespace the daemon was started from, + its `/mnt/9p` served by a 9ns that may be gone. Its cgroup, rlimits, + umask and stdin (`pardes -`) would be the daemon's too, or have to be + carried and re-execed; + - startup would pay an IPC round trip and an exec of a 100+ MB binary; + - the snapshot harness's idleness probe walks the launched process's + tree, and a daemon-spawned session is outside it. + + So `pardes` in a terminal forks its session off itself before anything + starts (`daemon.forkSession`): the child calls `setsid` and draws on the + `/dev/tty` the parent opened; the parent stays as its stub. GUI windows + and `--detach` sessions just register. The daemon spawns nothing, so it + is no subreaper: a detached session is reparented to the system's reaper + (systemd --user or init), which reaps it, as it reaps a crashed + session's shells. Nothing dangles: the daemon lists every session (its + connection) and holds every master, and exits a second after it holds + neither. +- **Starting the daemon.** The first session takes the flock, binds and + listens itself and queues its own connection; `pardes --daemon` is then + double-forked and exec'd onto that listener and lock (fds 3 and 4), by + a terminal's stub after the session's fork, so no session waits for it: + its hello sits queued until the daemon runs, and is not waited on (the + daemon's `ok` is the reader thread's; Detach waits for it, up to 2 s). + With the spawn before the fork, a cold start measured +0.38 ms (+7.6%). + The daemon's reader thread likewise starts after the first frame. + Another starter waits on the lock; a stale socket is the lock holder's + to replace. Messages are `SOCK_SEQPACKET` datagrams, a line of words and + the fds it carries. +- **pty masters.** `forkShell` sends a dup; a host's close sends `close`; + a session's end sends `quit` (its shells hang up); a master whose shell + is gone (`POLLHUP`, polled with no events) is dropped. TERM, HUP and INT + handlers send `quit` too (one `send`, async-signal-safe), so only a + session killed outright or crashed leaves shells behind. +- **Adoption is a word, `Adopt`,** calling the host through a vtable entry + (no Effect): the daemon hands over every orphaned master, and each + becomes a pane through the `Tty` word, its spawn taking the pty instead + of forking; a 1x1 size then the pane's makes it redraw. Adopting at + startup was rejected: an `EDITOR='pardes --wait'` session would take the + shells and hang them up when the editor exits. +- **The stub.** It forwards WINCH, TERM, INT, QUIT and HUP by `kill`, and + ends with the session's status (a forked stub `waitpid`s; an attaching + one reads `x`). A HUP is the terminal's end: kept by a daemon, a + session takes HUP on its signal thread (sigwait, beside WINCH) and + detaches, as on EIO; HUP never ends a kept session, since the terminal's + own EIO and HUP arrive in either order. Ctrl-Z stays a key for the pane + (the terminal is raw, as before); a TSTP sent to the stub is ignored, + since a stopped stub hands the terminal back under a session still + drawing; a background start stops at a `tcsetattr` before the fork, as + the old raw mode did. `--wait`, nested forwarding and `--tty` run before + any of this and are unchanged. +- **`--detach` is a session started detached:** the tty host with no + terminal, the GUI with a hidden window closed at once. With the frames + gone, `detached/server.zig` was a third host with nothing left to do + that the other two could not. +- **`Attach` (the word) is deleted:** it switched a session's own terminal + to another session through the frames. +- **Harnesses.** `PARDES_DAEMON=off` (the e2e harness sets it) runs a + session as before: in the terminal's own process, ended by its hangup, + for a harness that kills by pid and closes the terminal. A GUI test + window (`PARDES_TEST`) has none unless `PARDES_DAEMON=on`. +- **Known gaps.** A stub killed with SIGKILL while its terminal stays open + leaves the session drawing there until the terminal closes or Detach. A + reattached terminal keeps the first one's environment (TERM, COLORTERM); + vaxis queries its capabilities again. A daemon killed outright leaves + its sessions unregistered, and Detach then refuses. A daemon of an older + protocol answers `version`, and a session then runs without one. A + GUI started in a terminal's foreground stays that terminal's job. + +Measured (ReleaseFast, base main a7369739 against these changes, both +live at once and their samples alternating A, B, A, B; A/A the base +against itself; p50 ratio new/base): + +| | tty | A/A | GUI (test mode) | A/A | +|---|---|---|---|---| +| startup, no daemon running | 1.017 (4.73 to 4.81 ms) | 0.998 | 0.982 (295 to 290 ms) | 0.999 | +| startup, daemon running | 1.014 (4.66 to 4.72 ms) | | 1.014 (noisy) | | +| key in a file, to its frame | 0.992 | 1.009 | 1.003 | 0.986 | +| key echoed by bash, to its frame | 1.016 | 1.005 | | | + +What is left of the tty's startup is the stub's fork (about 70 us), which +the design cannot do without: the job the shell waits on must be able to +end while the session runs on. + +Trap found on the way: glibc's `tcgetattr` writes its 60-byte `struct +termios` into Zig's `std.c.termios`, which is 44 (the kernel's layout), +and overran the stack, clobbering a neighbouring local in the forked +child; `std.posix.tcgetattr` (an ioctl on that layout) is the right one. diff --git a/docs/typ/building.typ b/docs/typ/building.typ index d551c251..152be838 100644 --- a/docs/typ/building.typ +++ b/docs/typ/building.typ @@ -1,6 +1,6 @@ // Building pardes, for contributors: platforms, options, tests, the // release gates and sending patches, then how the pieces fit: the core -// and its shells, the threads, detached sessions, the 9P engine, the +// and its shells, the threads, the daemon and its sessions, the 9P engine, the // listeners and mounts. #import "style.typ": key, keys, btn, chord, word, tag, addr, file, cmd, doc, pairs @@ -177,8 +177,9 @@ Code map: `src/panes.zig` and the pane kinds it names (`src/File.zig`, `src/Terminal.zig`, ...), `src/layout.zig`, `src/fs.zig` (host access, mounts and resolution) and `src/pardes.zig` (input), with one file per thing beside them (`edit.zig`, `normal.zig`, `look.zig`, `exec.zig`, -`mouse.zig`, ...). `src/ninep/` is the control tree, `src/detached/` the -wire, the detached core and the frontend client, `src/lsp/` the +`mouse.zig`, ...). `src/ninep/` is the control tree, `src/daemon.zig` the +per-user daemon and a terminal's stub, `src/detached/wire.zig` the input +wire a remote client will speak, `src/lsp/` the language-server client; `test/` holds harnesses, goldens and helix cases, `build/` the snapshot suite's build step. @@ -197,26 +198,34 @@ reading the live core, and a request id, the pane's serial and (for an edit) the file's revision reject a stale reply. A process that never calls `turn.start` (the tests, the ESP32-P4, the browser) has no second thread. -== Detached sessions - -One poll loop in `src/detached/server.zig` owns core mutation, frontend -connections, pty I/O and file watches. The frontend socket is -`pardes-detached-.sock` beside the session's 9P socket; a second -session cannot take a live name. Up to 32 frontends attach; frames go to -every one, while clipboard reads, browser opens and #word("Detach") go to -the frontend that asked (else the first attached). Frames are full grids -or changes against what each frontend last received, never queued: a -frontend with unsent bytes skips that frame, and one that stops draining -past 1 MiB of control backlog is closed, its peers untouched. Frontends -never spawn shells, write session files or watch files. #word("Attach") -connects first and swaps second: only after the handshake does a shell -give up its own core. #word("Restore") builds the replacement core before touching -the current one. - -`src/detached/wire.zig` is the versioned protocol (version 10): fixed-width -little-endian fields, a 5-byte header (tag, then a u32 length), payloads up -to 16 MiB and grids up to 512 by 128. A comptime hash of `pardes.Chrome` -forces a version bump when that struct changes. +== The daemon and detached sessions + +One `pardes --daemon` per user (`src/daemon.zig`) listens on +`pardes-daemon.sock` in the runtime directory, its `.lock` flocked for its +life; the first session that finds none starts it (a double fork and an +exec) and it exits a second after it holds nothing. It is nobody's parent: +a session registers on one connection, sends it a dup of every pty master +it forks (so a session killed outright never hangs its shells up, and +#word("Adopt") takes them into another), says whether a terminal or a +window shows it, and is handed a terminal (`pardes --attach`) or a window +request (`pardes-gui --attach`) through it. Messages are `SOCK_SEQPACKET` +datagrams: a line of words and the fds it carries. + +`pardes` in a terminal forks its session off before anything starts: the +session, in a session of its own (`setsid`), draws on the terminal it +opened, and the process the shell waits on stays as its stub, passing it +`SIGWINCH`, `SIGTERM`, `SIGINT` and `SIGQUIT` and ending with its status. +#word("Detach"), or the terminal's end (EIO, a hangup), puts +the terminal back and lets it go; the stub ends (exit 0) and the session +runs on with nothing drawn, until `pardes --attach` hands it another +terminal. A window's #word("Detach") closes the window and keeps the +process; `pardes-gui --attach` opens one again. `PARDES_DAEMON=off` (the +snapshot harness) runs a session as before the daemon: in the terminal's +own process, ended by its hangup. + +`src/detached/wire.zig` is the input half of the protocol a remote client +will speak (version 13): fixed-width little-endian fields, a 5-byte header +(tag, then a u32 length), the hello and its version, the input events. = 9P diff --git a/docs/typ/reference.typ b/docs/typ/reference.typ index b83b4a83..c95a574a 100644 --- a/docs/typ/reference.typ +++ b/docs/typ/reference.typ @@ -386,9 +386,9 @@ Writes take settings and session builtins (`scope = .session` in record for why. A #word("ThemeFile") the dump names that fails to load changes nothing. - #word("Kill") `[word...]` (above), #word("Mount") `name dial`, #word("Unmount") `name`, #word("Theme") `x`. -- `size C R` sizes a `--detach` session no frontend is attached to (160x50 - until then): from 20x6 to 4096x4096, else `invalid size`; refused while a - frontend owns the size, and when a column would lose its panes' minimum +- `size C R` sizes a session no terminal or window is attached to (160x50 + for a `--detach` one until then): from 20x6 to 4096x4096, else `invalid + size`; refused while a terminal or window owns the size, and when a column would lose its panes' minimum rows (`size: too small for the panes, each its tag and 2 rows`). A bad `size` is refused quoting the value it got (`"5 2"`). @@ -454,8 +454,7 @@ turning it off: #word("PanelSlide"), #word("PanelZoom"), #word("PanelDissolve"), #word("PanelAscii"), #word("PanelVertical"), #word("PanelEdges"), #word("PanelFall"), #word("PanelWave"), #word("PanelCurtain"), #word("PanelScramble"), #word("PanelType"); all -start off, and the web shell has none. Scene passes (the SDL window, and -one attached to a detached session), each at a level 0-3 (`on` is 2), all +start off, and the web shell has none. Scene passes (the SDL window), each at a level 0-3 (`on` is 2), all off: #word("Crt"), #word("Bloom"), #word("Vignette"), #word("Grain"). #word("Shader") `` adds a Shadertoy file written for ghostty to the chain (#word("Shader") `off` removes it; it recompiles when saved); @@ -1011,11 +1010,14 @@ to its end before the write is answered, and one that finds nothing fails, git's `a/` `b/` (and `c/ i/ w/ o/`) prefixes are dropped; a `--no-prefix` diff's paths are kept. - *Sessions.* Bare `--detach` names the session after its pid; bare - `--attach` needs exactly one session; a second `--detach=NAME` while NAME - runs (a plain session is named by its pid) says so in words and exits 1. Frontends share the smallest common size. With - none attached, `size C R` sets the screen, messages clear by the clock, - and a pty reports its size in pixels at the last frontend's cell size - (8×16 before any), as CSI 14, 16 and 18 t do. + `--attach` needs exactly one detached session of its kind (a terminal's + for `pardes`, a window's for `pardes-gui`); a second `--detach=NAME` + while NAME runs (a plain session is named by its pid) says so in words + and exits 1. One terminal or window shows a session at a time. With none + attached, `size C R` sets the screen, messages clear by the clock, and a + pty reports its size in pixels at the last cell size (8×16 before any), + as CSI 14, 16 and 18 t do. Attaching from another machine is gone until + the replicated clients (docs/next-daemon.md, phase 5). - *Config.* The startup file is `$XDG_CONFIG_HOME/pardes/init` when that is absolute, else `~/.config/pardes/init` (macOS: `~/Library/Application Support/pardes/init`). A line that fails is a diff --git a/src/CHANGELOG.md b/src/CHANGELOG.md index 4466ad26..10cc451d 100644 --- a/src/CHANGELOG.md +++ b/src/CHANGELOG.md @@ -2,6 +2,29 @@ ## Unreleased +- One `pardes --daemon` per user keeps sessions, started by the first + session and gone once it keeps nothing. `pardes` in a terminal forks + its session off and stays as its stub: `Detach` (`SPC s D`), or a + terminal that closes, puts the terminal back and leaves the session + running in place (no fork of the core, no dump), and the stub ends. + `pardes --attach[=name]` in any terminal takes it back there. A + window's Detach closes the window and keeps the process; + `pardes-gui --attach[=name]` opens it again. A terminal session takes a + terminal and a window session a window, and the other is refused in + words. The daemon holds a dup of every shell's pty: a session killed + outright leaves its shells running, and `Adopt` in another session + takes them in as terminal panes (what they showed is gone; a resize + makes them redraw). A hangup never ends a session the daemon keeps (it + is the terminal's end); TERM and INT do, its shells with it. + `PARDES_DAEMON=off` runs a session as before. +- The frame wire goes: `src/detached/server.zig`, `client.zig` and the + cell frames of `wire.zig`, the frontends that drew a session's screen + from them, the smallest-common-grid rule, and the `Attach` word that + switched a session's own terminal to another. A `--detach` session is + now a terminal (or window) session with none attached yet. Attaching + from another machine is gone until the replicated clients + (docs/next-daemon.md, phase 5); `wire.zig` keeps the framing, the hello + and the input events for them. - One PNG and one JPEG decoder in a binary: images decode through wuffs, the decoders ghostty uses (ghostty-vt already links them), and stb_image keeps only the other formats. stb's own inflate and image writer go, as -- cgit v1.3 From f1f52054e3fe82f94697561d6fe9f9c14774f5de Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 12:32:23 -0300 Subject: docs/next-daemon.md: FreeBSD notes, for phase 5 (building, deploying and running on the box, what differs there, what 207 ms costs) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index d164eced..e63f69c6 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -267,3 +267,136 @@ Trap found on the way: glibc's `tcgetattr` writes its 60-byte `struct termios` into Zig's `std.c.termios`, which is 44 (the kernel's layout), and overran the stack, clobbering a neighbouring local in the forked child; `std.posix.tcgetattr` (an ioctl on that layout) is the right one. + +## FreeBSD notes + +Groundwork for phase 5's main test, a client here and the session on the +FreeBSD box (2026-10-29): the tty pardes is built here for FreeBSD and +runs there. The GUI is not built for it, since frontends stay local. + +The box runs FreeBSD 15.1 amd64 at 65.109.81.103, 207 ms away. Port 22 is +the host's root shell; port 2222 is a git jail and is never touched. +Everything of ours lives in `/root/pardes-test`, nothing outside it. + +- **Building.** Everything, the C deps included, cross-compiles here: + + ```sh + export PATH=$HOME/.local/bin:$PATH + E='env -u PARDES_PID -u PARDES_PANE -u PARDES_9P' + $E zig build -Dplatform=tty -Dtarget=x86_64-freebsd.15.0 >log 2>&1 + # zig-out/bin/pardes-freebsd-x86_64 + $E zig build test-install -Dplatform=tty -Dtarget=x86_64-freebsd.15.0 >log 2>&1 + # zig-out/test/{main,panes,kitty,9p,fonts,...}, named for their root files + ``` + + The `.15.0` matters: Zig's default FreeBSD (14.0) links + `libutil.so.9`, and 15 ships only `.so.10`. `test-install` installs + the test-build programs instead of running them. A test binary takes + no filter when it runs, so running one test alone means a rebuild + with `-Dtest-filter=` (and `--prefix ` to keep the full set). +- **Deploying.** `-F /dev/null` is required here, because /etc/ssh is + owned by nobody: + + ```sh + S='-F /dev/null -o BatchMode=yes -o IdentitiesOnly=yes -o UserKnownHostsFile=/home/goblin/.ssh/known_hosts -i /home/goblin/.ssh/id_ed25519' + zstd -q -f -T0 zig-out/bin/pardes-freebsd-x86_64 -o p.zst # 108 MB Debug; ~11 s in all + scp $S p.zst root@65.109.81.103:/root/pardes-test/bin/pardes.zst + ssh $S root@65.109.81.103 'cd /root/pardes-test/bin && zstd -q -d -f pardes.zst && rm pardes.zst' + ``` + + The same steps are scripts in `.scratch/daemon/bsd/`: `zb.sh + ` (a build, PARDES_* stripped, logged there), `deploy.sh + [name]` and `bx.sh ''` (a command run there with env.sh + sourced). `sshcheck.py` drives a session through `ssh -tt` from a + local pty, and `fwd9p.py` times 9P through `ssh -L`. +- **The box's layout.** Under `/root/pardes-test`: + - `bin/`: pardes, and `python3` linked to python3.12; + - `run/`: the runtime directory (sockets, the daemon's lock); + - `home/` and `tmp/`; + - `src/`: the tracked tree, which the tests need: they read docs/typ, + and make their temporary directories in `.zig-cache/tmp` under the + working directory; + - `test-bin/test/` and `logs/`; + - `env.sh`, which points every path pardes uses there (HOME carries + the XDG defaults): + + ```sh + T=/root/pardes-test + export HOME=$T/home XDG_RUNTIME_DIR=$T/run TMPDIR=$T/tmp + export PATH=$T/bin:$PATH SHELL=/usr/local/bin/bash TERM=${TERM:-xterm-256color} + ``` +- **Running.** + - A session: `ssh -t $S root@65.109.81.103 '. /root/pardes-test/env.sh; + pardes notes.txt'`, and `pardes --attach=` the same way. + - The unit tests: `cd /root/pardes-test/src && ../test-bin/test/main`. + Some tests name `/tmp/pardes-*` outright and leave seven empty + directories there (`pardes-arg-save` and the like): rmdir them after. + - The e2e scripts, from `src/`: `python3 -B test/daemon.py + /root/pardes-test/bin/pardes 20`, and `test/fs.py` the same way. + - Kill every process you start there by its pid, and check `ps -axww` + afterwards. +- **What FreeBSD does differently.** Each item was found on the box, and + each is fixed where pardes met it unless it says otherwise: + - A write to a terminal that is gone fails with ENXIO, not EIO. std + does not expect that errno, and a Debug build traced it to stderr, + which is the same dead terminal, recursing until the stack ran out: + a session whose ssh dropped hung. Tracing is off on FreeBSD + (main.zig). + - A signal whose action is to be ignored is dropped when it is sent, + sigwait or not, so WINCH never reached the session's watcher. A + handler that does nothing keeps it (tty.zig). + - waitid's P_PID is 0, WEXITED 16 and WNOWAIT 8, and its id is 64 + bits. With Linux's numbers a command pane's exit was never seen. + - There is no /proc. A shell's directory and program come from sysctl + instead (kern.proc.cwd, kern.proc.pathname). + - `ttyTaken` answers taken for any foreground group but the + shell's, so a nested shell counts as a job. + - `signalSession` stays Linux-only, so Kill leaves a command's `&` + jobs running there. + - File watching uses FreeBSD 15's inotify: Linux's events, with its + own IN_CLOEXEC and IN_NONBLOCK. + - `shm_open` refuses O_NONBLOCK; its names are not files, so there is + no FIFO to fear. A `write` does not grow a shared memory object. + - A kitty `t=f` naming a FIFO hung the open everywhere but Linux; a + stat by path now comes first. + - A Unix socket whose listen queue is full refuses a connect with + ECONNREFUSED, exactly as a socket nobody listens on does. The + registry sweep reads that refusal as a dead session, so it could + remove a live session's socket while its queue overflows. **Open:** + the unit test "the registry sweep takes the dead entries" fails on + it. +- **The tests there.** The box runs them as root, with no zls, fish or + fonts installed. + - main: 1138 passed, 36 skipped, 4 failed: the sweep above; the LSP + format retry (no zls); and two tests that make a file or directory + unreadable, which root reads anyway. + - panes: 224 passed, 1 failed (`Shell fish`: no fish). kitty: 43 + passed. The small programs all pass but fonts (no fonts). + - Off Linux, the parts of tests whose refusing path is Linux's + /proc/1 are skipped, and so is the builtins JSON check, since the + committed lists are Linux builds' (Tty9p included). + - test/daemon.py passes there at 20 cycles. One earlier run failed at + cycle 3, one process short in its count (most likely ps -e failing to + read an environment). + - test/fs.py runs as far as its `pardes -` +Pager checks. Those expect + one +Pager reused, which has not been true on Linux either since + 2026-10-22 (a new one each run). +- **At 207 ms, for the replicas.** + - In today's model every key is a round trip. Measured through `ssh + -tt` (min, median and max): + - a key typed into a file was on screen in 213, 229 and 346 ms; + - a key typed into a shell pane in 218, 219 and 224 ms. + + That is one RTT and about 15 ms. + - 9P over `ssh -L` of the session's socket costs one RTT a message: + 670 ms to connect, 1035 ms for a file read (walk, open, read, read, + clunk), 830 ms for a ctl write. A replica cannot fetch its state + file by file. + - The daemon's socket is SOCK_SEQPACKET and carries fds, so `ssh -L` + cannot forward it (tried: EPROTOTYPE here, connect failed there), and + no fd would cross anyway. The replication connection has to be a + stream of its own, as the 9P socket is, which the daemon hands on + locally. + - A dropped connection is a hangup: the session detaches, and + `pardes --attach` takes it back, as after a Detach. + - sun_path is 104 bytes there, not 108. -- cgit v1.3 From e9328a417ef82d72a73f5e3e7d182591616137de Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 13:27:52 -0300 Subject: 9p_io.zig: a session's liveness is a flock on its socket's .lock, not a connect, which FreeBSD refuses to a full listen queue as to a dead socket The owner holds `.lock` flocked from before it binds until after it unlinks (claim, release; the fatal-signal path unlinks it too). The startup sweep, the registry sweep, a registry entry's replacement and `taken` ask that lock: held is live, free is dead. They unlink only while holding it, so no new owner comes between the judging and the unlink, and two starters of one name are told apart by who took it. A socket with no lock, a pardes from before the locks, is still judged by a connect. The daemon's own start was already decided by its flock. The sweep test gains a live session whose socket refuses connects on every system (bound, not listening, its lock held) and a dead one's free lock, which goes with it; it passes on the FreeBSD box now. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 19 ++++--- src/9p_io.zig | 156 ++++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 149 insertions(+), 26 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index e63f69c6..159e66e7 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -360,16 +360,19 @@ Everything of ours lives in `/root/pardes-test`, nothing outside it. - A kitty `t=f` naming a FIFO hung the open everywhere but Linux; a stat by path now comes first. - A Unix socket whose listen queue is full refuses a connect with - ECONNREFUSED, exactly as a socket nobody listens on does. The - registry sweep reads that refusal as a dead session, so it could - remove a live session's socket while its queue overflows. **Open:** - the unit test "the registry sweep takes the dead entries" fails on - it. + ECONNREFUSED, exactly as a socket nobody listens on does, so a sweep + that judged by connecting could remove a live session's socket while + its queue overflowed. A session's liveness is now its lock: it holds + `.lock` flocked from before it binds until after it unlinks + (9p_io.zig `claim`), the sweeps and `taken` ask that lock, and they + unlink only under it. A connect judges only a socket with no lock, one + of a pardes from before the locks. The daemon's own start was already + decided by its flock. - **The tests there.** The box runs them as root, with no zls, fish or fonts installed. - - main: 1138 passed, 36 skipped, 4 failed: the sweep above; the LSP - format retry (no zls); and two tests that make a file or directory - unreadable, which root reads anyway. + - main: 1139 passed, 36 skipped, 3 failed: the LSP format retry (no + zls), and two tests that make a file or directory unreadable, which + root reads anyway. - panes: 224 passed, 1 failed (`Shell fish`: no fish). kitty: 43 passed. The small programs all pass but fonts (no fonts). - Off Linux, the parts of tests whose refusing path is Linux's diff --git a/src/9p_io.zig b/src/9p_io.zig index 174d3f89..5f22b326 100644 --- a/src/9p_io.zig +++ b/src/9p_io.zig @@ -195,6 +195,9 @@ pub const Listener = struct { paused_ms: i64 = 0, path_buf: [sun_path_len]u8 = undefined, path_len: usize = 0, + /// The socket's lock (`claim`), held for the listener's life; -1 when none. + lock: c_int = -1, + lock_buf: [lock_path_len:0]u8 = undefined, /// The registry entry this editor posted /// (`$XDG_RUNTIME_DIR/9p/pardes/`), zero when it did not. posted_buf: [sun_path_len]u8 = undefined, @@ -939,6 +942,8 @@ pub const Listener = struct { z[l.path_len] = 0; _ = libc.unlink(z[0..l.path_len :0]); } + // The lock after the socket: a socket there never lacks a held lock. + if (l.lock >= 0) release(l.lock, std.mem.sliceTo(&l.lock_buf, 0)); freeRunner(l.runner); gpa.destroy(l); } @@ -961,18 +966,22 @@ fn freeRunner(runner: *Runner) void { /// The socket this process listens on, kept where a fatal signal's handler /// can reach it without allocating (unlinkOwnSocket); empty when none. -var own_socket: [sun_path_len:0]u8 = undefined; +var own_socket: [lock_path_len:0]u8 = undefined; var own_socket_len: usize = 0; -/// Removes the socket this process listens on. Async-signal-safe (one -/// unlink): SIGTERM and SIGHUP call it before the signal's own death, so a -/// killed editor leaves no socket behind. +/// Removes the socket this process listens on, then its lock (the death +/// lets go of it). Async-signal-safe (two unlinks): SIGTERM and SIGHUP call +/// it before the signal's own death, so a killed editor leaves no socket +/// behind. pub fn unlinkOwnSocket() void { if (comptime !supported) return; const n = own_socket_len; if (n == 0) return; own_socket_len = 0; _ = libc.unlink(own_socket[0..n :0]); + own_socket[n..][0..lock_suffix.len].* = lock_suffix.*; + own_socket[n + lock_suffix.len] = 0; + _ = libc.unlink(own_socket[0 .. n + lock_suffix.len :0]); } /// SIGTERM, SIGHUP and SIGINT with no handler of their own (a GUI's, a @@ -1027,10 +1036,76 @@ pub fn sweepDeadSockets(io: std.Io, dir: [:0]const u8) void { const facts = statNoFollow(path) orelse continue; if (facts.mode & 0o170000 != 0o140000 or facts.uid != libc.getuid()) continue; if (probe(path) == .live) continue; - _ = libc.unlink(path); + removeDead(path); } } +/// A socket's lock, `.lock` beside it: flocked by the socket's owner +/// from before it binds to after it unlinks (`claim`, `release`), so held +/// is alive and free is dead. A connect cannot say that: FreeBSD refuses one +/// to a listener whose queue is full exactly as to a socket nobody listens +/// on, and a starting owner's socket refuses until it listens. +const lock_suffix = ".lock"; +const lock_path_len = sun_path_len + lock_suffix.len; + +fn lockPath(buf: *[lock_path_len:0]u8, sock: []const u8) ?[:0]const u8 { + return std.fmt.bufPrintSentinel(buf, "{s}" ++ lock_suffix, .{sock}, 0) catch null; +} + +/// `sock`'s lock taken (its path in `buf`), made when it is missing: null +/// while someone else holds it. A probe holds it shared for a moment, so a +/// refusal is asked again for a while before it counts. One taken on a file +/// its owner unlinked as it let go is taken again on the one at the path. +fn claim(sock: []const u8, buf: *[lock_path_len:0]u8) ?c_int { + const path = lockPath(buf, sock) orelse return null; + var tries: u32 = 0; + while (tries < 50) : (tries += 1) { + const fd = libc.open(path, .{ .ACCMODE = .RDWR, .CREAT = true, .CLOEXEC = true }, @as(libc.mode_t, 0o600)); + if (fd < 0) return null; + if (libc.flock(fd, std.posix.LOCK.EX | std.posix.LOCK.NB) != 0) { + _ = libc.close(fd); + Client.nap(1); + continue; + } + const io = std.Io.Threaded.global_single_threaded.io(); + const held = (std.Io.File{ .handle = fd, .flags = .{ .nonblocking = false } }).stat(io) catch null; + const named = std.Io.Dir.cwd().statFile(io, path, .{ .follow_symlinks = false }) catch null; + if (held != null and named != null and held.?.inode == named.?.inode) return fd; + _ = libc.close(fd); + } + return null; +} + +/// A claim let go, its lock file removed while still held: nobody else +/// unlinks it, and a claim on the unlinked file is taken again (`claim`). +fn release(fd: c_int, path: [:0]const u8) void { + _ = libc.unlink(path); + _ = libc.close(fd); +} + +/// What `sock`'s lock says of its owner: held is `.live`, free `.stale`, and +/// none at all `.none`, a socket of a pardes from before the locks, which a +/// connect judges. Anything unclear is `.live`. +fn lockState(sock: []const u8) Probe { + var buf: [lock_path_len:0]u8 = undefined; + const path = lockPath(&buf, sock) orelse return .live; + const fd = libc.open(path, .{ .ACCMODE = .RDONLY, .CLOEXEC = true }, @as(libc.mode_t, 0)); + if (fd < 0) return if (libc.errno(fd) == .NOENT) .none else .live; + defer _ = libc.close(fd); + return if (libc.flock(fd, std.posix.LOCK.SH | std.posix.LOCK.NB) == 0) .stale else .live; +} + +/// A dead owner's socket removed, and its lock, under that lock: no new +/// owner can come between the judging and the unlink. Under the lock, an +/// owner that took it is gone, and a socket of a pardes from before the +/// locks still has its connect to answer. +fn removeDead(sock: [:0]const u8) void { + var buf: [lock_path_len:0]u8 = undefined; + const fd = claim(sock, &buf) orelse return; + if (connectProbe(sock) == .stale) _ = libc.unlink(sock); + release(fd, std.mem.sliceTo(&buf, 0)); +} + /// The names in `dir` that `keep` takes (all with none), into `names` as a length byte and /// the name each, until it is full; the bytes staged. They are staged /// before anything is unlinked, so a sweep never asks a directory to keep @@ -1108,6 +1183,15 @@ pub fn listen(io: std.Io, gpa: std.mem.Allocator, core: *pardes.Pardes, named: [ .greet_timeout_ms = Listener.greet_deadline_ms, }); const p = socketPath(&l.path_buf, dir, entry_name).?; + // The name's lock first, held from here to the end: whoever else holds + // it is alive, however its socket answers a connect. Held, a socket + // there is a dead owner's, or one from a pardes before the locks, which + // the connects below still ask. + l.lock = claim(p, &l.lock_buf) orelse { + log.warn("something is already listening on {s}", .{p}); + l.deinit(gpa); + return null; + }; // Listened on under a name of its own, then renamed into place: a client // that waits for the socket to appear finds it listening already. Bound // at its own name, it was there a moment before listen(2), and a connect @@ -1215,7 +1299,7 @@ pub fn taken(name: []const u8) bool { const dir = socketDir(&dir_buf) orelse return false; var buf: [sun_path_len]u8 = undefined; const p = socketPath(&buf, dir, name) orelse return false; - return statNoFollow(p) != null and alive(p); + return probe(p) == .live; } /// What is at a socket path, in cloud9's three answers (`post.Probe`). @@ -1235,7 +1319,18 @@ const Probe = enum { none, stale, live }; /// `fcntl` instead. const probe_socket_kind: c_uint = libc.SOCK.STREAM | (if (darwin) 0 else libc.SOCK.NONBLOCK); +/// Whether whoever made the socket at `path` (or the one a registry entry +/// there names) is alive: its lock decides (`lockState`), and a connect +/// only for a socket that has none. fn probe(path: [:0]const u8) Probe { + var link_buf: [sun_path_len]u8 = undefined; + const n = libc.readlink(path, &link_buf, link_buf.len); + const sock = if (n > 0 and n < link_buf.len and link_buf[0] == '/') link_buf[0..@intCast(n)] else path; + const locked = lockState(sock); + return if (locked == .none) connectProbe(path) else locked; +} + +fn connectProbe(path: [:0]const u8) Probe { if (path.len + 1 > sun_path_len) return .live; // cannot ask; assume occupied var addr: libc.sockaddr.un = .{ .path = @splat(0) }; @memcpy(addr.path[0 .. path.len + 1], path[0 .. path.len + 1]); @@ -1275,7 +1370,8 @@ fn probe(path: [:0]const u8) Probe { /// the somebody. It is only ever a symlink that is followed or /// unlinked, because that is the only thing `postToRegistry` makes and /// anything else under a name belongs to whoever put it there; and only -/// a definite refusal counts as gone. The socket a stale entry points +/// a lock let go (or, with none, a definite refusal) counts as gone +/// (`probe`). The socket a stale entry points /// at goes too, but not before a stat agrees it is a socket of ours: a /// plain file answers a connect with the same refusal. /// Is this the kind of path this editor is allowed to delete? A registry @@ -1324,12 +1420,10 @@ fn sweepRegistry(io: std.Io, svc: [:0]const u8, sockets: []const u8) void { if (!ourSocket(target, sockets)) continue; const t = statNoFollow(target) orelse continue; if (t.mode & 0o170000 != 0o140000 or t.uid != libc.getuid()) continue; - // Ask again, immediately before deleting. The first probe was of the - // ENTRY and is by now several syscalls old; a socket that is bound but - // has not reached listen(2) yet answers ECONNREFUSED exactly like a - // dead one, and that window is every server's startup. - if (probe(target) != .stale) continue; - _ = libc.unlink(target); + // Judged again, under its lock: the first probe was of the ENTRY and + // is by now several syscalls old, and a new owner may have claimed + // the name since. + removeDead(target); } if (reaped != 0) log.info("reaped {d} stale registry entries under {s}", .{ reaped, svc }); } @@ -1428,21 +1522,42 @@ test "the registry sweep takes the dead entries and leaves everything else" { const outsider_sock = try std.fmt.bufPrintSentinel(&paths[5], "{s}/sweep-outsider-{d}.sock", .{ base, pid }, 0); var outsider_buf: [sun_path_len:0]u8 = undefined; const outsider = try std.fmt.bufPrintSentinel(&outsider_buf, "{s}/outsider", .{svc}, 0); + // Live, its lock held, but bound and not listening: a connect is + // refused, on every system, as FreeBSD refuses one to a live session + // whose queue is full. Its lock is what says it lives. + var starting_paths: [2][sun_path_len:0]u8 = undefined; + const starting_sock = try std.fmt.bufPrintSentinel(&starting_paths[0], "{s}/pardes-9p-sweepstarting-{d}.sock", .{ base, pid }, 0); + const starting = try std.fmt.bufPrintSentinel(&starting_paths[1], "{s}/starting", .{svc}, 0); + var locks: [3][lock_path_len:0]u8 = undefined; + const dead_lock = lockPath(&locks[2], dead_sock).?; defer { - for ([_][:0]const u8{ live_sock, dead_sock, live, dead, stranger, outsider_sock, outsider }) |p| _ = libc.unlink(p); + for ([_][:0]const u8{ live_sock, dead_sock, live, dead, stranger, outsider_sock, outsider, starting_sock, starting, dead_lock }) |p| _ = libc.unlink(p); _ = libc.rmdir(svc); } + const live_lock = claim(live_sock, &locks[0]) orelse return error.LockFailed; + defer release(live_lock, std.mem.sliceTo(&locks[0], 0)); const listening = bindSocket(live_sock); try testing.expect(listening >= 0); defer _ = libc.close(listening); try testing.expectEqual(@as(c_int, 0), libc.listen(listening, 1)); - // Bound and then dropped: the file stays, and nobody answers it — - // exactly what an aborted test leaves behind. + const starting_lock = claim(starting_sock, &locks[1]) orelse return error.LockFailed; + defer release(starting_lock, std.mem.sliceTo(&locks[1], 0)); + const bound = bindSocket(starting_sock); + try testing.expect(bound >= 0); + defer _ = libc.close(bound); + try testing.expectEqual(@as(c_int, 0), libc.symlink(starting_sock, starting)); + + // Bound and then dropped, its lock let go: the files stay, and nobody + // answers it -- exactly what a killed session leaves behind. const abandoned = bindSocket(dead_sock); try testing.expect(abandoned >= 0); _ = libc.close(abandoned); + const dropped = libc.open(dead_lock, .{ .ACCMODE = .RDWR, .CREAT = true, .CLOEXEC = true }, @as(libc.mode_t, 0o600)); + try testing.expect(dropped >= 0); + try testing.expectEqual(@as(c_int, 0), libc.flock(dropped, std.posix.LOCK.EX)); + _ = libc.close(dropped); try testing.expectEqual(@as(c_int, 0), libc.symlink(live_sock, live)); try testing.expectEqual(@as(c_int, 0), libc.symlink(dead_sock, dead)); @@ -1499,6 +1614,9 @@ test "the registry sweep takes the dead entries and leaves everything else" { try testing.expect(statNoFollow(dead) == null); try testing.expect(statNoFollow(dead_sock) == null); + try testing.expect(statNoFollow(dead_lock) == null); + try testing.expect(statNoFollow(starting) != null); + try testing.expect(statNoFollow(starting_sock) != null); try testing.expect(statNoFollow(live) != null); try testing.expect(statNoFollow(live_sock) != null); try testing.expect(statNoFollow(stranger) != null); @@ -1630,8 +1748,9 @@ test "a listening editor posts itself into the 9P registry and unposts on stop" // adopting the registry moves nothing, it only advertises. try testing.expect(statNoFollow(sock) != null); // It appeared already listening, renamed into place from a name of - // its own, which is gone. + // its own, which is gone; its lock held all the while. try testing.expect(alive(sock)); + try testing.expectEqual(Probe.live, lockState(sock)); var tmp_buf: [sun_path_len:0]u8 = undefined; try testing.expect(statNoFollow(try std.fmt.bufPrintSentinel(&tmp_buf, "{s}.{d}", .{ sock, libc.getpid() }, 0)) == null); // And the registry holds a symlink to it one directory down, so @@ -1643,8 +1762,9 @@ test "a listening editor posts itself into the 9P registry and unposts on stop" try testing.expectEqualStrings(sock, link[0..@intCast(n)]); } // Stopping takes the name back out, so the next editor of that name - // is not refused by its own corpse. + // is not refused by its own corpse; the lock goes with the socket. try testing.expect(libc.readlink(entry, &link, link.len) < 0); + try testing.expectEqual(Probe.none, lockState(sock)); } test "Unix TCP and QUIC share one listener through reads writes reconnects and reset" { -- cgit v1.3 From 7bd678868901ec9ae7c8e4369c16c53d30282e41 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 14:27:42 -0300 Subject: test/fs.py: a new +Pager each run, its colours through the theme until Filter, SYSTEMD_PAGER read by the shell, and no pager of the caller's Since 046979b8 (committed 2026-10-22) `pardes -` makes a new +Pager for each paged text, one already there keeping its page, and maps its colours through the theme as a terminal's until Filter shows them as sent; the test still expected one +Pager refilled and the colour as given. coreutils 9.12's printenv quotes a value with a blank on a terminal, so the shell prints SYSTEMD_PAGER instead. And a session's shells no longer inherit the caller's pagers: a PAGER naming the user's own pardes-gui ran instead of the binary under test. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 8 +++++--- test/fs.py | 56 +++++++++++++++++++++++++++++++++-------------------- 2 files changed, 40 insertions(+), 24 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 159e66e7..f60e32ab 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -381,9 +381,11 @@ Everything of ours lives in `/root/pardes-test`, nothing outside it. - test/daemon.py passes there at 20 cycles. One earlier run failed at cycle 3, one process short in its count (most likely ps -e failing to read an environment). - - test/fs.py runs as far as its `pardes -` +Pager checks. Those expect - one +Pager reused, which has not been true on Linux either since - 2026-10-22 (a new one each run). + - test/fs.py runs past its `pardes -` +Pager checks now, as far as the + SIGTERM and SIGHUP of a `--tty` stub: that test stops reading the pty + after the first frame, FreeBSD's pty fills before the session's frame + and reset are written, and the session waits in `ttyout` (a binary + from before these fixes too). Open. - **At 207 ms, for the replicas.** - In today's model every key is a round trip. Measured through `ssh -tt` (min, median and max): diff --git a/test/fs.py b/test/fs.py index b69ecf57..e351d295 100644 --- a/test/fs.py +++ b/test/fs.py @@ -22,8 +22,10 @@ from ninep import Client @contextlib.contextmanager def session(binary, root, name, *options, socket_name=None, file=None, tty=False, network=None, launch=None, inherited=None, terminal=None): + # No pager of the caller's (a session's own PAGER, SYSTEMD_PAGER...): + # its shells get the session's defaults, the ones under test. env = {key: value for key, value in os.environ.items() - if not key.startswith('PARDES_')} + if not key.startswith('PARDES_') and 'PAGER' not in key} env.update(HOME=str(root), XDG_RUNTIME_DIR=str(root), XDG_CONFIG_HOME=str(root / 'config'), PARDES_NOTIME='1', PARDES_DUMP=str(root / (name + '.dump.zon'))) @@ -519,8 +521,10 @@ def run_file(binary): assert b'\x1b' not in client.read(f'/pane/{pager[0]}/body') client.remove(f'/pane/{pager[0]}') # systemd's tools page through their own variable: set too, its - # secure mode off, the path unquoted where it needs none. - systemd_pager = run(client, term, b'printenv SYSTEMD_PAGER; printenv SYSTEMD_PAGERSECURE\n') + # secure mode off, the path unquoted where it needs none. Read + # by the shell: coreutils 9.12's printenv quotes a value with a + # blank on a terminal. + systemd_pager = run(client, term, b'printf "%s\\n" "$SYSTEMD_PAGER" "$SYSTEMD_PAGERSECURE"\n') assert systemd_pager.startswith(b'exit 0\n') and systemd_pager.rstrip().endswith(b' -\n0'), systemd_pager if shutil.which('journalctl'): assert run(client, term, b'journalctl -n 3 --no-hostname\n').startswith(b'exit ') @@ -1762,34 +1766,43 @@ def test(binary, quic=False): assert f'\nnew {made_now} {root}/+Pager\n' in '\n' + log_now, log_now[-400:] assert f'\nrename {made_now} ' not in '\n' + log_now # Its colours are the pane's, drawn and never read: the body is - # the plain text, and the screen shows `red` in the theme's red - # and `rgb` in the colour given, the rest in the pane's ink. + # the plain text, and the screen shows `red` in the theme's red, + # the rest in the pane's ink. Mapped through the theme, as a + # terminal's are; Filter shows them as sent, `rgb` in the colour + # given. + # Each run is its own +Pager; one there already keeps its page. fed = subprocess.run([binary, '-'], cwd=root, env=env, input=b'\x1b]0;title\x07\x1b[31mred\x1b[m \x1b[38;2;1;2;3mrgb\x1b[m plain\n', stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=10) assert fed.returncode == 0, fed.stderr - assert client.read(f'/pane/{made_now}/body') == b'red rgb plain\n' - screen = client.screen() - for row in range(screen['rows']): - cells = screen['cells'][row * screen['cols']:(row + 1) * screen['cols']] - text = ''.join(cell[0] for cell in cells) - at = text.find('red rgb plain') - if at >= 0: - break - else: + coloured, = serials() - before - {made_now} + assert client.read(f'/pane/{coloured}/name') == f'{root}/+Pager\n'.encode() + assert client.read(f'/pane/{coloured}/body') == b'red rgb plain\n' + assert client.read(f'/pane/{made_now}/body') == b'bold text\n' + def inks(): + screen = client.screen() + for row in range(screen['rows']): + cells = screen['cells'][row * screen['cols']:(row + 1) * screen['cols']] + at = ''.join(cell[0] for cell in cells).find('red rgb plain') + if at >= 0: + return [screen['styles'][cells[at + i][1]]['fg'] for i in (0, 4, 8)] raise AssertionError(('the paged text is not on the screen', screen)) - ink = [screen['styles'][cells[at + i][1]]['fg'] for i in (0, 4, 8)] + ink = inks() + assert ink[0] != ink[2] and ink[1] != {'rgb': [1, 2, 3]}, ink + client.write(f'/pane/{coloured}/exec', b'Filter\n') + ink = inks() assert ink[1] == {'rgb': [1, 2, 3]}, ink assert ink[0] != ink[2] and ink[0] != ink[1], ink - # The directory's one +Pager: the next paged text refills it. + # A progress line keeps its last state. fed = subprocess.run([binary, '-'], cwd=root, env=env, input=b'again\r\nprogress 10%\rprogress 100%\n', stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=10) assert fed.returncode == 0, fed.stderr - assert serials() - before == {made_now}, (serials(), before) - assert client.read(f'/pane/{made_now}/body') == b'again\nprogress 100%\n' - client.remove(f'/pane/{made_now}') + again, = serials() - before - {made_now, coloured} + assert client.read(f'/pane/{again}/body') == b'again\nprogress 100%\n' + for serial in (made_now, coloured, again): + client.remove(f'/pane/{serial}') # More than one 8 MiB write's worth arrives whole, byte for byte, - # in the one +Pager (each write once replaced the last). + # in its +Pager (each write once replaced the last). big = b''.join(b'line %08d of the paged text\n' % i for i in range(400000)) # ~11 MiB assert len(big) > 8 * 1024 * 1024 fed = subprocess.run([binary, '-'], cwd=root, env=env, input=big, @@ -1804,7 +1817,8 @@ def test(binary, quic=False): stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=600) huge.wait() assert fed.returncode == 0, fed.stderr - assert serials() - before == {big_pane} + client.remove(f'/pane/{big_pane}') + big_pane, = serials() - before got = client.read(f'/pane/{big_pane}/body') note = b'\n[pardes: the rest of stdin is cut: a pane holds at most a file\'s limit]\n' assert got.endswith(note) and len(got) == cap + len(note), (len(got), got[-120:]) -- cgit v1.3 From 791d0447aa2bdd4fe4ae0c76052e09acca4c61f2 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 14:27:42 -0300 Subject: daemon.zig: a forked session takes its stub's death, a SIGKILL too, as the terminal's HUP and detaches PR_SET_PDEATHSIG (FreeBSD: PROC_PDEATHSIG_CTL) in the session child, cleared again at Detach so the stub's own exit then is no hangup; a stub dead before it was set hangs up at once. It is the HUP the signal thread already takes (blocked in every thread, sigwaited), and setsid does not touch it. An --attach stub is no parent of its session: one killed so still leaves the session drawing until the terminal closes or Detach. test/daemon.py kills a session's stub outright under a shell that keeps the terminal (on FreeBSD the session leader's exit revokes it), waits for the session to put the terminal back and detach, and attaches it again; without the death signal it times out. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 10 +++++++--- src/daemon.zig | 24 +++++++++++++++++++++++- test/daemon.py | 28 ++++++++++++++++++++++++++-- 3 files changed, 56 insertions(+), 6 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index f60e32ab..500b53fe 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -240,8 +240,11 @@ Decisions taken at the forks, with the ponytail lens: session as before: in the terminal's own process, ended by its hangup, for a harness that kills by pid and closes the terminal. A GUI test window (`PARDES_TEST`) has none unless `PARDES_DAEMON=on`. -- **Known gaps.** A stub killed with SIGKILL while its terminal stays open - leaves the session drawing there until the terminal closes or Detach. A +- **Known gaps.** A forked session takes its stub's death, a SIGKILL + too, as a HUP (PR_SET_PDEATHSIG; FreeBSD's PROC_PDEATHSIG_CTL) and + detaches; a `pardes --attach` stub is no parent of its session, and one + killed with SIGKILL while its terminal stays open still leaves the + session drawing there until the terminal closes or Detach. A reattached terminal keeps the first one's environment (TERM, COLORTERM); vaxis queries its capabilities again. A daemon killed outright leaves its sessions unregistered, and Detach then refuses. A daemon of an older @@ -378,7 +381,8 @@ Everything of ours lives in `/root/pardes-test`, nothing outside it. - Off Linux, the parts of tests whose refusing path is Linux's /proc/1 are skipped, and so is the builtins JSON check, since the committed lists are Linux builds' (Tty9p included). - - test/daemon.py passes there at 20 cycles. One earlier run failed at + - test/daemon.py passes there at 20 cycles, its killed stub included + (procctl's death signal). One earlier run failed at cycle 3, one process short in its count (most likely ps -e failing to read an environment). - test/fs.py runs past its `pardes -` +Pager checks now, as far as the diff --git a/src/daemon.zig b/src/daemon.zig index c7729e59..0f770514 100644 --- a/src/daemon.zig +++ b/src/daemon.zig @@ -511,9 +511,25 @@ pub fn stubWait(session: libc.pid_t, child: bool, pipe: c_int) noreturn { /// The session's half of its stub: what `stubWait` reads. pub var stub: c_int = -1; -/// Detached: the stub is told (and goes), and the pipe let go. +extern "c" fn procctl(idtype: c_int, id: i64, cmd: c_int, data: ?*anyopaque) c_int; + +/// A HUP when this process's parent dies (a forked session's stub), or no +/// longer. Linux keeps it per thread: set and cleared on the main one. +fn hupOnStubDeath(on: bool) void { + var sig: c_int = if (on) @backingInt(posix.SIG.HUP) else 0; + switch (builtin.os.tag) { + .linux => _ = std.os.linux.prctl(@backingInt(std.os.linux.PR.SET_PDEATHSIG), @intCast(sig), 0, 0, 0), + // procctl(P_PID, 0, PROC_PDEATHSIG_CTL, &sig) + .freebsd => _ = procctl(0, 0, 11, &sig), + else => {}, + } +} + +/// Detached: the stub is told (and goes), and the pipe let go. Its end is +/// no hangup now (hupOnStubDeath). pub fn tellStubDetached() void { if (stub < 0) return; + hupOnStubDeath(false); var buf: [160]u8 = undefined; const said = std.fmt.bufPrint(&buf, "d{s}", .{name()}) catch "d"; _ = libc.write(stub, said.ptr, said.len); @@ -549,6 +565,7 @@ pub fn forkSession(conn: c_int) ?void { if (libc.pipe(&pipe) != 0) return null; ninep_io.setCloexec(pipe[0]); ninep_io.setCloexec(pipe[1]); + const parent = libc.getpid(); const pid = libc.fork(); if (pid < 0) { _ = libc.close(pipe[0]); @@ -558,6 +575,11 @@ pub fn forkSession(conn: c_int) ?void { if (pid == 0) { _ = libc.close(pipe[0]); _ = libc.setsid(); + // The stub's death, however it comes (a SIGKILL too), is a HUP: + // the terminal's end, which the session's signal thread takes and + // detaches on. One already dead hangs up at once. + hupOnStubDeath(true); + if (libc.getppid() != parent) _ = libc.raise(.HUP); stub = pipe[1]; // The listener and lock are the daemon's, which the stub starts. startPending(false); diff --git a/test/daemon.py b/test/daemon.py index f4f46286..9e9ea600 100644 --- a/test/daemon.py +++ b/test/daemon.py @@ -217,15 +217,39 @@ def main(binary, cycles): time.sleep(.3) assert alive(shell_pid), 'a killed session\'s shell is not hung up' assert alive(daemon) + # Started from a shell, which leads the terminal's session and + # outlives the stub, as a terminal's shell does (on FreeBSD the + # leader's exit revokes the terminal for every process). term = Term() - stub = term.spawn([binary, '--tty', '--9p=fresh'], env) - started.append(stub.pid) + stub_pid = root / 'stub.pid' + shell = term.spawn(['sh', '-c', '"$@" & echo $! > "$0"; wait; exec sleep 600', + str(stub_pid), binary, '--tty', '--9p=fresh'], env) + started.append(shell.pid) sock = root / 'pardes-9p-fresh.sock' wait_for('the second session', sock.exists) + stub = int(stub_pid.read_text()) + started.append(stub) client = Client(sock) status = dict(line.split(maxsplit=1) for line in client.read('/status').decode().splitlines()) session = int(status['pid']) started.append(session) + # Its stub killed outright, the terminal left open: the session + # (the stub's child) takes that as the terminal's end and detaches. + wait_for('the second session\'s first frame', lambda: b'\x1b[?1049h' in term.text()) + drawn = len(term.text()) + os.kill(stub, signal.SIGKILL) + wait_for('the stub gone', lambda: not alive(stub)) + wait_for('the session detached from its killed stub', lambda: sized(client)) + assert b'\x1b[?1049l' in term.text()[drawn:], 'the terminal is put back' + assert alive(session) + shell.kill() + shell.wait(timeout=10) + term.close() + term = Term() + stub = term.spawn([binary, '--attach=fresh'], env) + wait_for('the attached terminal\'s first frame', lambda: b'\x1b[?1049h' in term.text()) + wait_for('the session sized to its new terminal', lambda: not sized(client)) + print('daemon: a session whose stub was killed outright detached, and was attached again') before = set(terminals(client)) client.write(f'/pane/{min(before)}/exec', b'Adopt\n') adopted = wait_for('the adopted pane', lambda: set(terminals(client)) - before) -- cgit v1.3 From cb747d385019564e2141f7d9b01d03b394b469b8 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 12:22:50 -0300 Subject: docs/next-daemon.md: Phase 1 notes: the forks decided, what the census missed, what is deferred, and the gate's numbers How the tape and the walker came out (a tree at the host boundary, the replay a mode of the same binary, three replicas, digests per field), the annotations the walker needed (`len` besides local, derived and authority), the render-time writes moved into the step, the clock and host facts taped, the unit tests that had passed on a stale frame, and the two per-frame costs the A/B found and that are gone (the geometry key fed field by field, the paint-ahead worker started inside the render). The gate's numbers close it. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 146 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 500b53fe..903a80ad 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -409,3 +409,149 @@ Everything of ours lives in `/root/pardes-test`, nothing outside it. - A dropped connection is a hangup: the session detaches, and `pardes --attach` takes it back, as after a Detach. - sun_path is 104 bytes there, not 108. + +## Phase 1 notes + +Built 2026-10-29 (`src/walk.zig`, `src/tape.zig`, the hooks across the +core; `pardes-snap --tape`, `pardes-monkey --tape`, `zig build replay`). +Decisions taken at the forks, with the ponytail lens: + +- **The tape is a tree recorded at the host boundary, both ways.** Each + call the host makes into the core (`tape.Input`: update, serveFs, pump, + perform, the shell's acknowledgements and failures, a Restore...) is + recorded with its arguments; each call the core makes out -- a Host + function, wrapped generically over the vtable's fields, or a syscall of + its own (`tape.site`, which yields around it; every old `turn.yield()` + is one) or a read of the clock or environment (`tape.value`) -- with its + result. `done` and `ret` close each, so a 9P request served while a + step is out lands inside that step's syscall: turn order is tape order. + A replica drives the same functions and its host answers from the tape; + one that makes a call the recording did not diverges on the spot, + before any digest. `turn.yield` panics in a replica, so an untaped + syscall is a stack trace. +- **The replay is a mode of the same binary** (`PARDES_REPLAY= + pardes`), not a harness of its own: an error travels as this build's + number, and the code replayed is the code that recorded. Three replicas + run one after another: two that draw (the first's memory held while the + second runs, so it lies elsewhere) and one that never renders. The + headless one steps over the presents it does not make but takes what + the host told the core inside them (`acknowledge_panel_presentation`). +- **Fields a host writes between calls** (`native_images`, + `present_skipped`, `active`...) are found by a diff of a short list at + every crossing, not by touching the frontends. +- **The digest** is taken after every call the host made at the top (a + pump, in the tty), as one per field of the core, its stream and each + pane, its file and places, so a divergence names the field. + `PARDES_RECORD_EVERY` digests after every input, nested ones too, to + find the step one starts at. +- **The walker needed a fourth annotation, `len`**: a fixed array written + only as far as a length field says (`jumps`/`njumps`, a message's text) + is undefined beyond it, 0xAA in a safe build and a fault where it holds + a slice. Integers are hashed by value (a `u21`'s padding bits are not + its value). +- **View geometry** is `File.geometry`, a lazy cache keyed by a hash of + what it is made of: revision, length, path, scroll, hscroll, the pane's + size, tags and query, tables as far as read, context declarations, the + output origin, and the pane's `view` (wrap, layered context, row + metrics, rect height), which `sync` sets. A hundred readers a frame ask + for it, so the key is one hash of those facts laid out as words (fed + field by field it cost 4% of every frame). A step reading it (`wrapAt`, + `wrapRow`, `ensureCursorVisible`, a body drag, `zz`) makes it on demand, + and the render draws from what the step made: made afresh for every + frame as well (to show an input the key missed only as a headless + divergence), it cost a scroll a third walk of the rows, 4% of a + PageDown. An input the key misses shows as a stale frame in a golden. +- **Render wrote more than the census found**, all moved into the step: + context declarations, the notice bands and their chips' columns (a + click on a chip was routed by the last frame's), the header's caret + reveal (`tagnav.golden` moved: after `g l B B` typed in one burst the bar + stays scrolled, as a human typing it sees it), the tables' reading (a + first look of 4096 lines, a count, wherever it is asked; the rest a + frame's taped clock budget a tick), the tty filter palette and body + chrome (derived). A view's paint-ahead stays the render's own prefetch: + instead of queuing an effect it is asked of the host once the frame is + presented (`Stream.askAhead`, in the pump), so a replica that draws + nothing asks for none; query rows' paints count apart from it. (Asked + inside the render, its worker's thread start cost a held `j` 10% in the + live tty measurement.) +- **Clock.** `advance`'s wakes are the recording session's + (`tape.value(.wake, nextWake)`): a picture decoding or a fade asks a + frame, and a replica that draws none must still take the session's + ticks. The budgets of Places, Names and Tables are taped verdicts + (`tape.over`), as are a message's time of day (TZ included), File.mtime + and the pump's clock read (made whether or not it waits). A message's + stamp and an unattended session's expiry of it read the clock through + the tape too, not the session clock: that is the last pump's, and + stamped by it a message written over 9P after an idle spell was as old + as the spell and went at once (fs.py's `unattended_messages_expire`). +- **A pointer seen through a sliding pane** is taped + (`Pardes.pointerCell`): what was under it is the frame the user saw. +- **The host's facts**: the session directory (getcwd, asked once at + init), `$SHELL` and whether it is there, `$HOME` for `~`, a `Shell` that + is not there, a missing interpreter, a directory refused above, a file + deleted on disk (`File.deleted` is an input). The rest of the core's + getenv (PARDES_DUMP and the XDG dirs in dump.zig, TMPDIR for kitty + media, PARDES_TEST_CLOCK in the render) is the host's or authority-only. + Per read rather than "captured once into session state": less code, and + the same for a replica. +- **Authority-only**: worker futures and their Io (a future's presence + too), the 9P server's state (`fs`: fids, opens, the log; serveFs itself + is replayed in turn order, 9p_io's own writes into `fs` are not), the + store file every window of the user shares, which a replica keeps in + memory, and Recent read from it. **Local**: the surface, presentation, + frame stats, animations, the effect and event queues (drained by every + pump before its digest). **Derived**: the geometry, notices, highlights, + line index, tables, node trees, rail ticks, a compiled regex, an image's + decoded pixels, PDF rasters. +- **Terminals** replay from their taped pty bytes (they are `output` + events) and are digested as their active screen, cursor and scroll + numbers; their kitty images are left out until phase 5. +- **Worker results**: the census's render-polled ones (image decode, PDF + lanes) are derived state; the rest already arrive as events. + +- **Unit tests that mutate and render** without a step now step + (`p.sync()`) first, as every host does; three of them had passed on the + last frame's stale state: a prompt chip a body click had already closed, + a `.center` landing measured on no wrap map at all (it centres by lines, + so under wrap a line above can now push the cursor down where + `ensureCursorVisible` takes the view back: that test is unwrapped now), + and a context row count a scroll had not refreshed. The perf gate's + `deep-sel` cell was one more: its view, placed from the last frame's + map, stopped six lines short of the 2 MB line it selects in. In view, + that line showed `File.rawAtDisplay` measuring a line's whole rest + before walking to a column (20 ms a frame, Debug); it walks to the + column only now, and the cell is the baseline's (91 us against 106). + +What proved wrong or missing in the design: +- The census missed the host writes between calls, the pointer through + the presentation, the render-time writes above, `Stream.start` (host + code doing core work, made harmless by the futures being authority's), + getcwd, `access(2)` for `$SHELL`, File.mtime's clock, and a deadlock in + vaxis's queue (a `yes` stopped being read; fixed on this line, found by + the new measurement). +- `windowed`/`native_images` are watched host writes for now, not View + state (phase 4); the semantics hash in the hello is not done (there is + no hello in phase 1), so a tape from another build is not refused yet. +- Deferred: the GUI's own host-to-core calls (font requests, opacity, + `core.advance`/`render` are inputs, the rest not yet), PDF lanes and + Handover yields (never reached by the gate; a replica would panic on + them, which is the point), `Stream.runHere`'s yield (a host with no + workers). + +The gate on the final line (on main vukknslx): `pardes-snap --tape`, +110/110 scripts, 128 tapes, 384 replica runs, 33,093 digests, no +divergence; `pardes-monkey --tape`, 50 seeds of 70 steps, 150 replica +runs, 37,800 digests, none; `zig build replay` (the 110 scripts and 20 +seeds) clean; `snap` 110/110 with no golden changed but `tagnav`; +unit-test tty, pane-test, 9p-io-test and fs-test pass, unit-test gui +fails only its two known tests. Perf, base (main with perf --live and +the queue fix) against this line, ReleaseFast, interleaved A B A ... A, +medians of runs, the base's own spread as the A/A noise. The perf +harness's gesture cells (render, keys, PageDown, wheel, edit, deep-sel) +0.97-1.03 at an A/A of 2-7%, and on the final line, measured on the +shared perf cores, 0.91-0.97 at 10-31%. The live tty on a quiet machine: +core time p50 0.96-0.97, keystroke-to-present p50 0.98-1.03 (A/A 1-23%), +`yes | head -c 1G` 0.96, a million-line open 1.01. The live GUI's core +p50 1.00-1.04 (A/A 2-18%), present-bound at the window's 30 ms. Two +costs the A/B found are gone (the geometry key, the paint-ahead's +thread start in the render). -- cgit v1.3 From 97bb0e816513b805ca13358d8a14ba117f586403 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 17:01:35 -0300 Subject: tty: a flooded pane's output is drawn a frame a tick at most, two of its reads queued at most, so it costs no frame a 4 KB read and holds no key or echo behind it Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++ src/tty/tty.zig | 57 ++++++++++++++++++++++++++++++-- 2 files changed, 148 insertions(+), 2 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 903a80ad..a39484b8 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -555,3 +555,96 @@ core time p50 0.96-0.97, keystroke-to-present p50 0.98-1.03 (A/A 1-23%), p50 1.00-1.04 (A/A 2-18%), present-bound at the window's 30 ms. Two costs the A/B found are gone (the geometry key, the paint-ahead's thread start in the render). + +## Tty throughput notes + +Found and fixed 2026-10-29 (`src/tty/tty.zig`, `waitInput`). + +- **What the 1 GB `yes` cell measures is mostly the kernel.** A pty's + output processing (ONLCR, a `\n` made `\r\n`) costs the writer about + 0.4 us a newline here, and `yes` writes one every two bytes. Every reader + of a pty is capped by it: `script(1)` drains 100 MB of `yes` at 8.6 MB/s + with OPOST on and at 650 MB/s with it off, a Python drain at 8.0 MB/s, + and tmux at 5.4 MB/s. The tty pardes was at 5.7 to 6.1 MB/s, three + quarters of that ceiling, before any change. The 179 s were the kernel's. +- **Where pardes itself was slow:** with 200-byte lines (`yes` of a long + word), the ceiling rises to 264 MB/s, tmux reaches 55 MB/s, and pardes + managed 18.6 MB/s. A flat profile (perf at paranoid 2 allows no call + graphs) put 98.7% of the samples in pardes: `Vaxis.render` 29%, + `body_layer.renderBody` 17%, `Shell.present` 10%, memcpy 8%, + ghostty's parse 1.5%. The tty drew a frame for every pty read, which + is 4 KB, nearly all of them overwritten before anyone could see them + (about 690k frames for the 1 GB cell). +- **The fix: output is drawn once per frame tick at most** (16 ms), with + whatever came in until then, as the GUI drains its queue once per vsync. + Two things are drawn at once, as before: + - an input's frame; + - the focused pane's output within a second of an input (its echo; a + remote shell's arrives an RTT later). + Neither waits while it is undrawn, including when FrameAck held its + frame for the terminal's answer. Snapshot runs (the virtual clock) are + not paced. +- **And a pane has two reads queued at most** (`readPty`): its reader + waits for the loop to take one before it reads on. ghostty parses `yes` + at about 9 MB/s, 0.45 ms a 4 KB read, slower than this machine's pty + delivers it. So a flood filled the loop's queue (512 events): a key and + its echo waited behind its reads, and which blocked writer a free slot + woke was luck, which gave half-second peaks. Paced but unbounded, the + wait was steady at about 20 reads, 9 ms for the key and 9 more for its + echo. A frame dump lined up with each key's write showed it (PARDES_FRAMES + and `flood.py` with FRAMES=). What a flood writes beyond two reads now + waits in its pty; the loop could not have kept up with it anyway. +- **Measured** (ReleaseFast, `.scratch/daemon/term/yes.py` through a real + 120x40 pty, all interleaved, with measurements pinned to two physical + cores and so four CPUs; medians of 3 rounds, MB/s): + + | | `yes`, 32 MiB | 200-byte lines, 256 MiB | + |---|---|---| + | base (before the rework) | 5.5 | 14.5 | + | phase 1's tip | 4.8 | 15.0 | + | pacing alone (the queue limit came after) | 4.6 | 117 | + | tmux | 5.4 | 47.6 | + | raw pty drain (Python) | 7.7 | 239 | + | GUI, test mode | 0.9 | | + + Unpinned, before the pinning: 200-byte lines 18.6 to 247 MB/s (1 GB in + 53.7 s and in 4.0 s), and `yes` 5.7 to 6.9 MB/s. The GUI could only be + measured in its test mode (a hidden window), which downloads every frame + from the GPU, so 0.9 MB/s says nothing of a real window. There is no + headless compositor here, and a real window must not open on the user's + session. In a later interleaved run, the change as committed against + pacing alone gave 4.8 and 6.0 against 4.8 and 5.4 MB/s for `yes`, and + 102 and 117 against 124 and 119 MB/s for long lines: the same within + the noise. `pardes-perf --live`'s `yes` cell drew 2.3k frames for 32 MiB + against 11k before, in the same time: it is kernel-bound. +- **A held key while another pane floods** (`flood.py`: a key every 30 + ms into `cat` in one pane of `-n 3`, timed from its write to the pty to + its echo on screen, with pyte in the loop; `yes` in another pane): + in one interleaved run of two rounds (p50, p90, worst, in ms): + + | | under the flood | idle | + |---|---|---| + | phase 1's tip | 3.0, 212 to 214, 445 to 470 | 2.2 to 2.3 | + | pacing alone | 10.3 to 10.6, 19 to 20, 26 to 33 | 2.3 | + | this change | 3.3 to 4.2, 4.3 to 5.6, 6.0 to 7.8 | 2.3 to 3.2 | + + The driver's own cost (pyte) is in every number. With 16 KiB queued a + pane instead of two reads, the flood's p50 was 7.5 to 7.8 ms for no + throughput gained, so two reads it is. +- **Keystroke-to-present** (`pardes-perf-live --live`, base and this + change interleaved, 5 and 4 runs; A/A between the base runs' halves): + p50 against base is 0.92 for key-down, 0.89 key-right, 0.85 edit-char, + 0.80 page-down and 0.93 wheel. The A/A was 0.73 to 0.92. The 32 MiB + `yes` took 1.00 the time with 0.18 the frames, and the big open 1.10 + (A/A 1.05). Two earlier runs of pacing alone, 3 and 5 rounds, were + within their A/A too. +- **Unrelated, found on the way:** `test/fs.py` failed + `unattended_messages_expire` (its first `shown()`) on phase 1's tip as + well; pmxpuolt on main (a message's stamp and expiry read the clock + again) is that fix. fs.py also needs a short TMPDIR: under + `.scratch/daemon/term/tmp`, `directory_panes`'s socket path is 108 bytes, + one past the limit, and the session starts with no 9P. Two GUI unit tests + fail on phase 1's tip as well (`tagline pointer mapping ...`, `a tag caret + past its row's end ...`). A suite started as a shell's `&` job runs with + SIGINT ignored, so four host_io tests whose Ctrl-C must stop a `sleep` + fail there; started otherwise, they pass. diff --git a/src/tty/tty.zig b/src/tty/tty.zig index d474e83a..760c0692 100644 --- a/src/tty/tty.zig +++ b/src/tty/tty.zig @@ -1407,7 +1407,10 @@ fn localSession( sh.inotify_fd = -1; } while (loop.tryEvent() catch null) |ev| switch (ev) { - .pty_read => |pr| gpa.free(pr.bytes), + .pty_read => |pr| { + tookRead(io, pr.id); + gpa.free(pr.bytes); + }, .paste => |b| gpa.free(@constCast(b)), .lsp_done => |d| if (d.rows) |rows| allocs.lsp.free(rows), .lsp_status => |text| allocs.lsp.free(text), @@ -1591,6 +1594,13 @@ const Shell = struct { pace_ns: u64 = 0, input_seen: bool = false, wake_to: u64 = 0, + /// waitInput's pacing of output: whether this wait read a pane's output, + /// and when the last frame was written, the last input was read and the + /// focused pane's output last was. + output_seen: bool = false, + drawn_ns: u64 = 0, + input_ns: u64 = 0, + focused_ns: u64 = 0, fn of(ctx: ?*anyopaque) *Shell { return @ptrCast(@alignCast(ctx.?)); @@ -1778,6 +1788,7 @@ const Shell = struct { wait = @intCast(@max(1, (tick -| now) / std.time.ns_per_ms)); } s.input_seen = false; + s.output_seen = false; // Paced, what is not an input (a frame's ack, a pty's output) is // taken in and the wait goes on to the tick: the acks alone ran the // loop at the terminal's round trip. @@ -1789,6 +1800,21 @@ const Shell = struct { wait = @intCast((tick - now) / std.time.ns_per_ms); } if (paced and s.input_seen) s.pace_ns = @intCast(std.Io.Clock.awake.now(s.io).nanoseconds); + // Output alone is drawn a frame a tick at most, what came in till + // then in one: a flood (`yes`, 4 KB a read) was drawn a read at a + // time, nearly all of its frames written over before they were + // seen. Never while what someone waits on is undrawn: an input + // (its frame held for the terminal's answer, FrameAck), or the + // focused pane's output soon after one, its echo (a remote shell's + // an RTT later). + while (!paced and s.output_seen and !s.input_seen and !s.core.quit and s.test_clock == null) { + if (s.input_ns > s.drawn_ns) break; + if (s.focused_ns > s.drawn_ns and s.focused_ns -| s.input_ns < std.time.ns_per_s) break; + const due = s.drawn_ns + pardes.animation.frame_ns; + const now: u64 = @intCast(std.Io.Clock.awake.now(s.io).nanoseconds); + if (now + std.time.ns_per_ms > due) break; + s.waitOnce(@intCast((due - now) / std.time.ns_per_ms)); + } } fn waitOnce(s: *Shell, wait: u32) void { @@ -1853,6 +1879,7 @@ const Shell = struct { .paste, .paste_start, .winsize => s.input_seen = true, else => {}, } + if (s.input_seen) s.input_ns = @intCast(std.Io.Clock.awake.now(s.io).nanoseconds); switch (event) { .nop => {}, .wake => if (s.test_clock) |*virtual| { @@ -1886,6 +1913,9 @@ const Shell = struct { core.update(.{ .resize = .{ .cols = ws.cols, .rows = ws.rows } }); }, .pty_read => |pr| { + tookRead(s.io, pr.id); + s.output_seen = true; + if (pr.id == core.active) s.focused_ns = @intCast(std.Io.Clock.awake.now(s.io).nanoseconds); if (s.gens[pr.id] == pr.gen) core.update(.{ .output = .{ .pane = @intCast(pr.id), .bytes = pr.bytes } }); s.gpa.free(pr.bytes); @@ -2139,6 +2169,7 @@ const Shell = struct { // vaxis's diff of the cells wrote. s.core.frames.bytes +|= @intCast(@min(std.math.maxInt(u32), s.tty.tty_writer.pos + s.tty.writer().end -| before)); if (s.tty.tty_writer.pos + s.tty.writer().end != before) FrameAck.ask(s.tty.writer(), @intCast(std.Io.Clock.awake.now(s.io).nanoseconds)); + s.drawn_ns = @intCast(std.Io.Clock.awake.now(s.io).nanoseconds); tz_render.end(); } @@ -2572,15 +2603,37 @@ fn wakeFs(ctx: ?*anyopaque) void { _ = loop.tryPostEvent(.fs_ready) catch {}; } +/// Each pane's reads posted and not yet taken in. Two at most: a flood the +/// loop cannot keep up with waits in its pty, not in the queue ahead of a +/// key and its echo (hundreds of 4 KB reads queued were half a second). +var reads_queued: [pardes.MAX_PANES]std.atomic.Value(u32) = @splat(.init(0)); + +/// A read of pane `id` taken in (or dropped): its reader may read on. Never +/// below zero, for an event no reader posted (a test's). +fn tookRead(io: std.Io, id: usize) void { + var queued = reads_queued[id].load(.acquire); + while (queued > 0) queued = reads_queued[id].cmpxchgWeak(queued, queued - 1, .acq_rel, .acquire) orelse { + if (queued == 2) io.futexWake(u32, &reads_queued[id].raw, 1); + return; + }; +} + fn readPty(io: std.Io, gpa: std.mem.Allocator, pty: std.Io.File, id: usize, gen: u32, loop: *Loop) anyerror!void { // One buffer, read into directly: a Reader around it only added a second // (and in a safe build, filled both as `undefined` on every start). var buf: [0x10000]u8 = undefined; - while (true) { + read: while (true) { + while (true) { + const queued = reads_queued[id].load(.acquire); + if (queued < 2) break; + io.futexWait(u32, &reads_queued[id].raw, queued) catch break :read; + } const n = pty.readStreaming(io, &.{&buf}) catch break; if (n == 0) break; const bytes = try gpa.dupe(u8, buf[0..n]); + _ = reads_queued[id].fetchAdd(1, .acq_rel); loop.postEvent(.{ .pty_read = .{ .id = id, .gen = gen, .bytes = bytes } }) catch { + tookRead(io, id); gpa.free(bytes); break; }; -- cgit v1.3 From c2b4f235e78735102c691677f08bd1b2403168df Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 17:01:35 -0300 Subject: screen_state.zig: a terminal crosses to a replica as its screen's state, ghostty's own records by content address, a delta a screen of row hashes and the blobs the peer lacks, history fetched by absolute row Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 98 +++++ src/pardes.zig | 2 + src/screen_state.zig | 1102 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1202 insertions(+) create mode 100644 src/screen_state.zig (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index a39484b8..e3be6af8 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -648,3 +648,101 @@ Found and fixed 2026-10-29 (`src/tty/tty.zig`, `waitInput`). past its row's end ...`). A suite started as a shell's `&` job runs with SIGINT ignored, so four host_io tests whose Ctrl-C must stop a `sleep` fail there; started otherwise, they pass. + +## Screen-state terminal notes + +Built 2026-10-29 (`src/screen_state.zig`), a pure encode and apply pair +over byte buffers for the phase 5 stream; no transport yet. Only tests +reference it, so the tty and GUI binaries carry none of it: no symbol of +it is linked into the tty built from this tree. The local path's A/B, base +against that tty, is the Tty throughput notes' keystroke-to-present. + +- **Both kinds of blob are ghostty's own snapshot records,** not a cell + format of ours. The terminal-wide state (modes, colours and palette, + title, pwd, tab stops, mouse modes, scrolling region) is a TERMINAL + record. A row is the PAGE record of a one-row page the row is cloned + into, which carries its styles, graphemes, hyperlinks, wide cells, wrap + and prompt marks exactly. The replica decodes them with ghostty's + decoders and writes a row with `Screen.clonePartialRowGrowCapacity`. +- **A row's address is a hash of what it shows** (Wyhash of its flags, + its width and its cells by value), not of its blob: ghostty keeps the + source page's style ids when it clones a row, so equal rows give blobs + that differ. The width is in the address, because a blank row is the + same at every width. +- **A delta** is a header (cursor, kitty keyboard flags, the active + area's first row as an absolute row, the history's extent), the TERMINAL + record's address, an address for each active row, an address for each + row that scrolled off since the acknowledged state (the tail, up to a + screen of them), and the blobs for the addresses the peer does not hold. + An unchanged row costs 8 bytes. +- **One delta per peer is in flight,** against the state it acknowledged, + as FrameAck keeps one frame in flight for the tty. A slow peer skips the + states in between. The peer holds exactly the blobs of the state it + acknowledged, at most two screens of rows and the meta, so the authority + knows what it holds without asking. At 207 ms this means at most about 5 + deltas a second to the FreeBSD box; phase 6's predicted echo is meant to + cover typing there. +- **Absolute rows** come from a tracked pin at the last active top. The + rows scrolled since are the distance to the new top; if the pin's page + was pruned, the count jumps past every row the replica could hold. A + resize renumbers (an epoch), and the peer starts afresh. Scrollback is + fetched by absolute row (`encodeRows`, `applyRows`). +- **The replica writes into a ghostty Terminal,** a terminal pane's own, + so `src/terminal.zig` draws it unchanged. A test renders a replica's + terminal pane and its authority's in two cores: the bodies are equal, + cell for cell. +- **Gates.** + - The soak drives an authority with pty bytes and lands deltas at + random, both skipping states and delaying a delta while the authority + moves on. Resizes and fetches of holes also come at random. After each + delta it checks the replica's screen against the authority's as + encoded, and every history row the replica holds outside its holes + against the authority's row at that number. + - It passed with synthetic shell, TUI and `yes` streams at 4 seeds (in + the unit tests, with a 64 KB scrollback so it prunes). It also passed + with 197 recorded streams at 3 seeds each (the test that + PARDES_SCREEN_STATE= turns on): the 192 pane outputs of the snapshot scripts' + tapes, plus real vim, nvim, htop, less and btop sessions recorded + through a pty. Its corpus and recorder are in `.scratch/daemon/term`. +- **Numbers** (ReleaseFast: `PARDES_SCREEN_STATE= zig build core-test + -Dplatform=tty -Doptimize=ReleaseFast -Dtest-filter=screen_state`, which + runs under std.testing's checking allocator): + - Encode, 200x60, every row new: p50 250 us, p99 487 us, 29.8 KB. + - Encode, 200x60, nothing new: p50 71 us, p99 139 us, 534 bytes. That + is the 60 rows hashed and the TERMINAL record encoded to hash it. + - Under `yes` as fast as ghostty takes it in-process (9 to 12 MB/s), + with a delta every 16 ms: 62 deltas a second, 63 KB/s on the wire, + 1 KB a delta (60 rows and the tail's 60, all held). That is bounded by + the frame rate, not the output. + - Under varied text at 36 to 55 MB/s: 355 KB/s on the wire, 5.6 KB a + delta. + - The replica applies a `yes` delta in 2.4 ms, most of it pushing the + rows scrolled past into its history (a row at a time). A delta of + varied text applies in 0.24 ms under the C allocator and 0.95 ms + under the checking one. +- **What proved wrong in the design.** + - "Scrollback fetched on demand" alone loses rows. A row that changed + after the last delta and then scrolled off would sit stale in the + replica's history. The tail fixes the rows that scrolled within a + screen. Older ones, which the replica never saw or saw before they + last changed, are holes: rows it must fetch before it shows them. + - The address had to include the row's width, as above. The soak found + this: a replica reused an 80-column blank row after a resize to 10. + - ghostty narrows pages lazily, so an active row can be wider than its + terminal: only the terminal's columns are hashed and encoded. +- **Not done, or a gap.** + - Kitty images in terminals. + - pardes's own state derived from the stream is not carried: its + `MarkedHandler` prompt phase, command-output pins and the replies it + writes to the pty. ghostty's prompt marks on rows do cross. + - The view's scroll offset is per client and is not sent. + - A replica's core must not resize its pane's terminal itself; the + authority sizes it. + - The replica's history extent follows its own pruning under the + authority's limit, so it can be a page off, and its oldest rows can + be ones the authority has dropped. + - A fetch sends every row as a blob (phase 6's cache will send + addresses). No zstd. + - A replica pushes a flood's rows into its history one at a time. + - The integration into the stream and the cores (a replica pane's + State fed by `apply` instead of `.output`) is the next piece. diff --git a/src/pardes.zig b/src/pardes.zig index b5e729bb..228d97fa 100644 --- a/src/pardes.zig +++ b/src/pardes.zig @@ -63,6 +63,7 @@ pub const pdf = panes.Pdf.pdf; pub const memory = @import("memory.zig"); pub const tape = @import("tape.zig"); pub const walk = @import("walk.zig"); +pub const screen_state = if (terminal_panes) @import("screen_state.zig") else struct {}; pub const image = @import("image.zig"); pub const dump = @import("dump.zig"); pub const lsp = @import("lsp/lsp.zig"); @@ -509,6 +510,7 @@ test { _ = @import("Tables.zig"); _ = @import("Handover.zig"); _ = @import("walk.zig"); + _ = screen_state; } pub const Builtin = builtins.registry.Builtin(); diff --git a/src/screen_state.zig b/src/screen_state.zig new file mode 100644 index 00000000..51c2ec30 --- /dev/null +++ b/src/screen_state.zig @@ -0,0 +1,1102 @@ +//! Screen-state terminals (docs/next-daemon.md, "Terminals cross as +//! authoritative screen state", phase 5). A replica is sent a terminal's +//! state, never its pty's bytes, so a `cat` of a gigabyte costs it screens. +//! +//! The authority encodes a delta for one peer: a header (the cursor, the +//! active area's first row as an absolute row, the history's extent), the +//! content address of the terminal-wide state, the content address of each +//! row of the active area, and the blobs behind the addresses the peer does +//! not hold. Both kinds of blob are ghostty's own snapshot records, so they +//! cross exactly: the terminal-wide state is a TERMINAL record (modes, +//! colours, title, tab stops...), and a row is the PAGE record of a one-row +//! page (its styles, graphemes, hyperlinks, wide cells, wrap and prompt +//! marks). A row's address is a hash of what the row shows, not of its blob, +//! whose style ids are its source page's. An unchanged row costs 8 bytes. +//! +//! One delta per peer is in flight, as FrameAck keeps one frame in flight +//! for the tty. The next delta is encoded against the state the peer +//! acknowledged, once it has, so a slow peer skips the states in between. +//! The peer holds the blobs of the state it acknowledged and no others. +//! +//! The replica writes each delta into a ghostty Terminal, which a replica's +//! terminal pane draws as a local one (src/terminal.zig reads only that). +//! Rows that scroll off its active area go into its history, made what the +//! authority's are by the delta's tail (their addresses, up to a screen of +//! them). Older ones it never saw, or saw before they last changed, are +//! holes there, fetched by absolute row on demand (`encodeRows`, +//! `applyRows`). +const std = @import("std"); +const vt = @import("ghostty-vt"); +const snapshot = vt.snapshot; +const Terminal = vt.Terminal; +const Page = vt.page.Page; +const Row = vt.page.Row; +const Pin = vt.Pin; +const Screen = vt.Screen; + +// ponytail: 64 bits: two of a million distinct rows collide about once in +// 4*10^7 sessions; phase 6's cache, shared across sessions, wants 128. +pub const Hash = u64; + +/// What a row shows, hashed: its wrap and prompt flags, and each of its +/// first `cols` cells up to the last one that is not blank, with its style, +/// grapheme and hyperlink by value. Not the page's style or hyperlink ids, +/// nor its derived flags. Its width is in it: a blob is a row that wide. +/// (A page can be wider than its terminal: ghostty narrows them lazily.) +pub fn rowHash(page: *Page, row: *Row, cols: u16) Hash { + var h = std.hash.Wyhash.init(0); + h.update(&.{rowFlags(row)}); + h.update(std.mem.asBytes(&cols)); + const all = page.getCells(row); + const cells = all[0..@min(cols, all.len)]; + var n = cells.len; + while (n > 0 and @as(u64, @bitCast(cells[n - 1])) == 0) n -= 1; + for (cells[0..n]) |*c| { + var word = c.*; + word.style_id = 0; + h.update(std.mem.asBytes(&@as(u64, @bitCast(word)))); + if (c.style_id != 0) h.update(std.mem.asBytes(&page.styles.get(page.memory, c.style_id).hash())); + if (c.hasGrapheme()) h.update(std.mem.sliceAsBytes(page.lookupGrapheme(c).?)); + if (c.hyperlink) if (page.lookupHyperlink(c)) |id| + h.update(std.mem.asBytes(&page.hyperlink_set.get(page.memory, id).hash(page.memory))); + } + return h.final(); +} + +fn rowFlags(row: *const Row) u8 { + return @as(u8, @intFromBool(row.wrap)) | @as(u8, @intFromBool(row.wrap_continuation)) << 1 | + @as(u8, @intFromEnum(row.semantic_prompt)) << 2; +} + +/// The authority's absolute row numbers for one terminal: its primary +/// screen's active area starts at `top`, and a row `d` above it in its +/// history is `top - d`. Kept by a tracked pin at the last active top, so +/// it counts the rows scrolled since. A resize renumbers (`epoch`). +pub const Rows = struct { + pin: ?*Pin = null, + top: u64 = 0, + epoch: u32 = 0, + cols: u16 = 0, + rows: u16 = 0, + + pub fn update(n: *Rows, t: *Terminal) !void { + const pages = &t.screens.get(.primary).?.pages; + const top = pages.getTopLeft(.active); + const y = pages.pointFromPin(.screen, top).?.screen.y; + if (n.pin) |p| { + if (t.cols != n.cols or t.rows != n.rows) n.epoch +%= 1; + // The last top was pruned away: every row the peer has is + // older than the oldest row here. + const was: ?u32 = if (p.garbage) null else if (pages.pointFromPin(.screen, p.*)) |pt| pt.screen.y else null; + n.top += if (was) |w| y -| w else @as(u64, y) + t.rows; + p.* = top; + } else { + n.pin = try pages.trackPin(top); + n.top = y; + } + n.cols = t.cols; + n.rows = t.rows; + } + + pub fn deinit(n: *Rows, t: *Terminal) void { + if (n.pin) |p| t.screens.get(.primary).?.pages.untrackPin(p); + n.* = .{}; + } + + /// The pin of absolute row `abs`, if the terminal still has it. + fn pinAt(n: *const Rows, t: *Terminal, abs: u64) ?Pin { + return pinOf(&t.screens.get(.primary).?.pages, n.top, abs); + } +}; + +fn pinOf(pages: anytype, top: u64, abs: u64) ?Pin { + if (abs >= top) return pages.pin(.{ .active = .{ .y = std.math.cast(u32, abs - top) orelse return null } }); + const d = std.math.cast(usize, top - abs) orelse return null; + return pages.getTopLeft(.active).up(d); +} + +/// The authority's scratch: a one-row page each blob is copied into +/// (fresh, so the blob holds that row's styles alone) and encoded from. +pub const Encoder = struct { + gpa: std.mem.Allocator, + scratch: ?Page = null, + meta: std.Io.Writer.Allocating, + pins: std.ArrayList(Pin) = .empty, + needed: std.ArrayList(struct { pin: Pin, hash: Hash }) = .empty, + + pub fn init(gpa: std.mem.Allocator) Encoder { + return .{ .gpa = gpa, .meta = .init(gpa) }; + } + + pub fn deinit(e: *Encoder) void { + if (e.scratch) |*s| s.deinit(); + e.meta.deinit(); + e.pins.deinit(e.gpa); + e.needed.deinit(e.gpa); + } + + /// The PAGE record of `row` alone, its first `cols` cells. + fn blob(e: *Encoder, page: *Page, row: *Row, cols: u16, w: *std.Io.Writer) !void { + var cap: vt.page.Capacity = if (e.scratch) |s| s.capacity else .{ .cols = cols, .rows = 1, .styles = 64 }; + if (cap.cols != cols) cap = .{ .cols = cols, .rows = 1, .styles = 64 }; + for (0..16) |_| { + if (e.scratch) |*s| { + if (std.meta.eql(s.capacity, cap)) s.reinit() else { + s.deinit(); + e.scratch = null; + } + } + if (e.scratch == null) e.scratch = try Page.init(cap); + const s = &e.scratch.?; + s.cloneRowFrom(page, s.getRow(0), row) catch { + // Room for this row: everything doubled until it fits. + cap.styles *|= 2; + cap.grapheme_bytes *|= 2; + cap.hyperlink_bytes *|= 2; + cap.string_bytes *|= 2; + continue; + }; + break; + } else return error.RowTooBig; + var rw: snapshot.record.Writer = .init(e.gpa, w); + defer rw.deinit(); + try snapshot.page.encode(&e.scratch.?, &rw); + } +}; + +/// What the authority knows of one peer. +pub const Peer = struct { + seq: u32 = 0, + in_flight: bool = false, + /// Never sent anything, or sent before a renumbering: the next delta + /// starts the replica afresh. + fresh: bool = true, + epoch: u32 = 0, + /// The active area's first row in the state it acknowledged. + top: u64 = 0, + /// The blobs the peer holds: those of the state it acknowledged. + known: std.AutoHashMapUnmanaged(Hash, void) = .empty, + /// Those of the delta in flight, which become `known` on its ack. + sent: std.AutoHashMapUnmanaged(Hash, void) = .empty, + + pub fn deinit(peer: *Peer, gpa: std.mem.Allocator) void { + peer.known.deinit(gpa); + peer.sent.deinit(gpa); + } + + pub fn ack(peer: *Peer, seq: u32) void { + if (!peer.in_flight or seq != peer.seq) return; + std.mem.swap(std.AutoHashMapUnmanaged(Hash, void), &peer.known, &peer.sent); + peer.in_flight = false; + } + + /// Start the peer afresh (it joined again, or lost its state). + pub fn reset(peer: *Peer) void { + peer.fresh = true; + peer.in_flight = false; + peer.known.clearRetainingCapacity(); + } +}; + +/// The delta's fixed header: u32 seq, u8 flags, u32 epoch, u64 meta, u16 +/// cursor x and y, u8 cursor style, u8 pending wrap, the kitty keyboard +/// flags, u64 top, u64 history rows; then u16 rows and a u64 per active +/// row, u16 and a u64 per row of the tail (those that scrolled off the +/// active area since the acknowledged state, up to a screen of them, oldest +/// first), and a u16 blob count and each blob as u64 hash, u32 length, bytes. +const flag_reset: u8 = 1; +const KittyFlags = @FieldType(Screen, "kitty_keyboard"); + +/// The delta for `peer` against what it acknowledged, onto `out`, or false +/// (nothing written) while its last delta is in flight. +pub fn encode(e: *Encoder, peer: *Peer, n: *Rows, t: *Terminal, out: *std.Io.Writer) !bool { + if (peer.in_flight) return false; + const gpa = e.gpa; + try n.update(t); + if (peer.epoch != n.epoch) peer.reset(); + const reset = peer.fresh; + peer.sent.clearRetainingCapacity(); + + e.meta.clearRetainingCapacity(); + { + var rw: snapshot.record.Writer = .init(gpa, &e.meta.writer); + defer rw.deinit(); + try snapshot.terminal.encode(t, &rw); + } + const meta = std.hash.Wyhash.hash(0, e.meta.written()); + try peer.sent.put(gpa, meta, {}); + + const screen = t.screens.active; + const pages = &t.screens.get(.primary).?.pages; + try out.writeInt(u32, peer.seq +% 1, .little); + try out.writeByte(if (reset) flag_reset else 0); + try out.writeInt(u32, n.epoch, .little); + try out.writeInt(u64, meta, .little); + try out.writeInt(u16, screen.cursor.x, .little); + try out.writeInt(u16, screen.cursor.y, .little); + try out.writeByte(@intFromEnum(screen.cursor.cursor_style)); + try out.writeByte(@intFromBool(screen.cursor.pending_wrap)); + // ponytail: raw bytes, as the hello refuses a peer of another build. + try out.writeAll(std.mem.asBytes(&screen.kitty_keyboard)); + try out.writeInt(u64, n.top, .little); + const history = pages.total_rows - pages.rows; + try out.writeInt(u64, history, .little); + try out.writeInt(u16, t.rows, .little); + + // The active rows, then the tail: what scrolled off since the state + // acknowledged, which may have changed after it was sent. + const tail: u16 = if (reset) 0 else @intCast(@min(n.top -| peer.top, t.rows, history)); + e.pins.clearRetainingCapacity(); + var it = screen.pages.rowIterator(.right_down, .{ .active = .{} }, null); + while (it.next()) |p| try e.pins.append(gpa, p); + for (0..tail) |j| try e.pins.append(gpa, pages.getTopLeft(.active).up(tail - j).?); + e.needed.clearRetainingCapacity(); + for (e.pins.items, 0..) |p, i| { + if (i == t.rows) try out.writeInt(u16, tail, .little); + const h = rowHash(p.node.page(), p.rowAndCell().row, t.cols); + try out.writeInt(u64, h, .little); + // Every address listed is held once this is acknowledged; a blob + // goes only for one not held already, once. + if ((try peer.sent.getOrPut(gpa, h)).found_existing or peer.known.contains(h)) continue; + try e.needed.append(gpa, .{ .pin = p, .hash = h }); + } + if (tail == 0) try out.writeInt(u16, 0, .little); + const meta_new = !peer.known.contains(meta); + try out.writeInt(u16, @intCast(e.needed.items.len + @intFromBool(meta_new)), .little); + if (meta_new) { + try out.writeInt(u64, meta, .little); + try out.writeInt(u32, @intCast(e.meta.written().len), .little); + try out.writeAll(e.meta.written()); + } + var blob: std.Io.Writer.Allocating = .init(gpa); + defer blob.deinit(); + for (e.needed.items) |need| { + blob.clearRetainingCapacity(); + try e.blob(need.pin.node.page(), need.pin.rowAndCell().row, t.cols, &blob.writer); + try out.writeInt(u64, need.hash, .little); + try out.writeInt(u32, @intCast(blob.written().len), .little); + try out.writeAll(blob.written()); + } + peer.top = n.top; + peer.seq +%= 1; + peer.in_flight = true; + peer.fresh = false; + peer.epoch = n.epoch; + return true; +} + +/// The primary screen's rows `[lo, hi)` the authority still has, as u32 +/// epoch, u32 count, then each as u64 row, u32 length, blob: a replica's +/// fetch of history it holds as holes. +// ponytail: every row its blob, none by address: phase 6's content cache. +pub fn encodeRows(e: *Encoder, n: *Rows, t: *Terminal, lo: u64, hi: u64, out: *std.Io.Writer) !void { + try n.update(t); + var blobs: std.Io.Writer.Allocating = .init(e.gpa); + defer blobs.deinit(); + var count: u32 = 0; + var abs = lo; + while (abs < hi) : (abs += 1) { + const p = n.pinAt(t, abs) orelse continue; + const start = blobs.written().len; + try blobs.writer.writeInt(u64, abs, .little); + try blobs.writer.writeInt(u32, 0, .little); + try e.blob(p.node.page(), p.rowAndCell().row, t.cols, &blobs.writer); + const len: u32 = @intCast(blobs.written().len - start - 12); + std.mem.writeInt(u32, blobs.written()[start + 8 ..][0..4], len, .little); + count += 1; + } + try out.writeInt(u32, n.epoch, .little); + try out.writeInt(u32, count, .little); + try out.writeAll(blobs.written()); +} + +/// A replica's terminal: what it was sent, written into a ghostty Terminal. +pub const Replica = struct { + seq: u32 = 0, + epoch: u32 = 0, + meta: Hash = 0, + top: u64 = 0, + /// The address of each active row as the Terminal holds it, a list per + /// screen (primary, alternate). + held: [2]std.ArrayList(Hash) = .{ .empty, .empty }, + /// The blobs of the last state applied: what the authority takes it to + /// hold. Owned copies. + blobs: std.AutoHashMapUnmanaged(Hash, []u8) = .empty, + /// History rows it cannot vouch for (written blank, or seen before + /// they last changed), `[lo, hi)` absolute, oldest first. + holes: std.ArrayList([2]u64) = .empty, + /// The one-row page each row blob is decoded into. + scratch: ?Page = null, + + pub fn deinit(r: *Replica, gpa: std.mem.Allocator) void { + for (&r.held) |*h| h.deinit(gpa); + var it = r.blobs.valueIterator(); + while (it.next()) |b| gpa.free(b.*); + r.blobs.deinit(gpa); + r.holes.deinit(gpa); + if (r.scratch) |*s| s.deinit(); + } + + /// Write one delta into `t`; the seq to acknowledge. + pub fn apply(r: *Replica, gpa: std.mem.Allocator, io: std.Io, t: *Terminal, bytes: []const u8) !u32 { + var in: std.Io.Reader = .fixed(bytes); + const seq = try in.takeInt(u32, .little); + const flags = try in.takeByte(); + const epoch = try in.takeInt(u32, .little); + const meta = try in.takeInt(u64, .little); + const cx = try in.takeInt(u16, .little); + const cy = try in.takeInt(u16, .little); + const style = try in.takeByte(); + const wrap = try in.takeByte(); + var kitty: KittyFlags = undefined; + @memcpy(std.mem.asBytes(&kitty), try in.take(@sizeOf(KittyFlags))); + const top = try in.takeInt(u64, .little); + const history = try in.takeInt(u64, .little); + const nrows = try in.takeInt(u16, .little); + const hashes = try in.take(@as(usize, nrows) * 8); + const tail = try in.takeInt(u16, .little); + const tails = try in.take(@as(usize, tail) * 8); + // The delta's blobs, borrowed from `bytes` while it applies. + var fresh: std.AutoHashMapUnmanaged(Hash, []const u8) = .empty; + defer fresh.deinit(gpa); + for (0..try in.takeInt(u16, .little)) |_| { + const h = try in.takeInt(u64, .little); + try fresh.put(gpa, h, try in.take(try in.takeInt(u32, .little))); + } + const blobOf = struct { + fn of(rr: *Replica, f: *const std.AutoHashMapUnmanaged(Hash, []const u8), h: Hash) ![]const u8 { + if (f.get(h)) |b| return b; + if (rr.blobs.get(h)) |b| return b; + return error.MissingBlob; + } + }.of; + + if (flags & flag_reset != 0 or meta != r.meta) { + var reader: std.Io.Reader = .fixed(try blobOf(r, &fresh, meta)); + var next = try snapshot.terminal.decode(&reader, try snapshot.record.nextHeader(&reader), io, gpa); + if (flags & flag_reset != 0) { + std.mem.swap(Terminal, t, &next); + next.deinit(gpa); + for (&r.held) |*h| h.clearRetainingCapacity(); + r.holes.clearRetainingCapacity(); + r.top = top; + r.epoch = epoch; + // The authority's history, blank, to be fetched. + const primary = t.screens.get(.primary).?; + try scrollUp(primary, history); + if (history > 0) try r.holes.append(gpa, .{ top -| history, top }); + } else { + defer next.deinit(gpa); + try takeMeta(t, &next, io, gpa); + } + r.meta = meta; + } + if (epoch != r.epoch) return error.Renumbered; + + // The primary screen scrolled `top - r.top` rows since the last + // delta: as many go up into the history here (no more than it and + // the screen hold). + // ponytail: the history's extent then follows this terminal's own + // pruning, under the authority's limit: a page more or less than the + // authority's, and the oldest rows may be some it has dropped. The tail is what they are now; any older than + // the tail are holes, never seen or seen before they last changed. + const primary = t.screens.get(.primary).?; + if (top > r.top) { + const k = top - r.top; + try scrollUp(primary, @min(k, @as(u64, t.rows) + history + 1)); + const held = &r.held[0]; + if (k >= held.items.len) held.clearRetainingCapacity() else { + std.mem.copyForwards(Hash, held.items, held.items[@intCast(k)..]); + held.shrinkRetainingCapacity(held.items.len - @as(usize, @intCast(k))); + } + if (top - tail > r.top) try r.holes.append(gpa, .{ r.top, top - tail }); + r.top = top; + } + for (0..tail) |j| { + const p = pinOf(&primary.pages, top, top - tail + j) orelse continue; + const h = std.mem.readInt(u64, tails[j * 8 ..][0..8], .little); + if (rowHash(p.node.page(), p.rowAndCell().row, t.cols) != h) try r.writeRow(gpa, primary, p, try blobOf(r, &fresh, h)); + } + // The authority's history went (an ED3): so does this one's. + if (history == 0 and primary.pages.total_rows > primary.pages.rows) { + primary.pages.eraseHistory(null); + r.holes.clearRetainingCapacity(); + } + r.trimHoles(primary); + + const screen = t.screens.active; + const held = &r.held[@intFromEnum(t.screens.active_key)]; + const blank = blankHash(t.cols); + while (held.items.len < nrows) try held.append(gpa, blank); + held.shrinkRetainingCapacity(nrows); + for (0..nrows) |y| { + const h = std.mem.readInt(u64, hashes[y * 8 ..][0..8], .little); + if (h == held.items[y]) continue; + try r.writeRow(gpa, screen, screen.pages.pin(.{ .active = .{ .y = @intCast(y) } }).?, try blobOf(r, &fresh, h)); + held.items[y] = h; + } + screen.cursorReload(); + screen.cursorAbsolute(@min(cx, t.cols - 1), @min(cy, t.rows - 1)); + screen.cursor.cursor_style = @enumFromInt(style); + screen.cursor.pending_wrap = wrap != 0; + screen.kitty_keyboard = kitty; + + // Hold the blobs of this state, and no others. + var keep: std.AutoHashMapUnmanaged(Hash, []u8) = .empty; + errdefer keep.deinit(gpa); + try keep.ensureTotalCapacity(gpa, nrows + tail + 1); + for (0..nrows + tail + 1) |i| { + const h = if (i == nrows + tail) meta else if (i < nrows) + std.mem.readInt(u64, hashes[i * 8 ..][0..8], .little) + else + std.mem.readInt(u64, tails[(i - nrows) * 8 ..][0..8], .little); + if (keep.contains(h)) continue; + const b = if (r.blobs.fetchRemove(h)) |kv| kv.value else try gpa.dupe(u8, try blobOf(r, &fresh, h)); + keep.putAssumeCapacity(h, b); + } + var old = r.blobs.valueIterator(); + while (old.next()) |b| gpa.free(b.*); + r.blobs.deinit(gpa); + r.blobs = keep; + r.seq = seq; + return seq; + } + + /// The holes in `[lo, hi)`: the rows to fetch before showing them. + pub fn missing(r: *const Replica, lo: u64, hi: u64, out: *std.ArrayList([2]u64), gpa: std.mem.Allocator) !void { + for (r.holes.items) |hole| { + const a = @max(lo, hole[0]); + const b = @min(hi, hole[1]); + if (a < b) try out.append(gpa, .{ a, b }); + } + } + + /// Write fetched history rows (`encodeRows`) into `t`. + pub fn applyRows(r: *Replica, gpa: std.mem.Allocator, t: *Terminal, bytes: []const u8) !void { + var in: std.Io.Reader = .fixed(bytes); + if (try in.takeInt(u32, .little) != r.epoch) return; // numbered before a resize + const primary = t.screens.get(.primary).?; + for (0..try in.takeInt(u32, .little)) |_| { + const abs = try in.takeInt(u64, .little); + const blob = try in.take(try in.takeInt(u32, .little)); + if (abs >= r.top) continue; + const p = pinOf(&primary.pages, r.top, abs) orelse continue; + try r.writeRow(gpa, primary, p, blob); + try r.fill(gpa, abs); + } + } + + fn inHole(r: *const Replica, abs: u64) bool { + for (r.holes.items) |hole| if (abs >= hole[0] and abs < hole[1]) return true; + return false; + } + + /// Row `abs` is written: out of the holes. + fn fill(r: *Replica, gpa: std.mem.Allocator, abs: u64) !void { + for (r.holes.items, 0..) |*hole, i| { + if (abs < hole[0] or abs >= hole[1]) continue; + if (abs == hole[0]) hole[0] += 1 else if (abs + 1 == hole[1]) hole[1] -= 1 else { + try r.holes.insert(gpa, i + 1, .{ abs + 1, hole[1] }); + r.holes.items[i][1] = abs; + } + if (r.holes.items[i][0] == r.holes.items[i][1]) _ = r.holes.orderedRemove(i); + return; + } + } + + /// Holes older than this history's oldest row are gone with it. + fn trimHoles(r: *Replica, primary: *Screen) void { + const oldest = r.top -| (primary.pages.total_rows - primary.pages.rows); + var i: usize = 0; + while (i < r.holes.items.len) { + const hole = &r.holes.items[i]; + hole[0] = @max(hole[0], oldest); + if (hole[0] >= hole[1]) _ = r.holes.orderedRemove(i) else i += 1; + } + } + + fn writeRow(r: *Replica, gpa: std.mem.Allocator, screen: *Screen, p: Pin, blob: []const u8) !void { + var reader: std.Io.Reader = .fixed(blob); + var dec: snapshot.page.Decoder = undefined; + try dec.init(&reader, try snapshot.record.nextHeader(&reader)); + const cap = dec.capacity(); + if (r.scratch) |*s| { + if (std.meta.eql(s.capacity, cap)) s.reinit() else { + s.deinit(); + r.scratch = null; + } + } + if (r.scratch == null) r.scratch = try Page.init(cap); + const src = &r.scratch.?; + try dec.decode(src, gpa); + if (src.size.cols != screen.pages.cols) return error.BadRow; + _ = screen.clonePartialRowGrowCapacity(p.node, p.y, src, src.getRow(0), 0, src.size.cols); + } +}; + +/// `k` blank rows onto the bottom of a screen's active area, its top `k` +/// going up into the history, as `k` linefeeds at the bottom would. +// ponytail: a row at a time, so a flood costs the replica up to its whole +// history's rows a delta (measured in `bench`); a bulk scroll if that shows. +fn scrollUp(screen: *Screen, k: u64) !void { + if (k == 0) return; + screen.cursorAbsolute(0, screen.pages.rows - 1); + for (0..@intCast(k)) |_| try screen.cursorDownScroll(); +} + +/// `rowHash` of a row never written. +fn blankHash(cols: u16) Hash { + var h = std.hash.Wyhash.init(0); + h.update(&.{0}); + h.update(std.mem.asBytes(&cols)); + return h.final(); +} + +/// The terminal-wide state of `next` (a decoded TERMINAL record) into `t`, +/// whose screens and pages stay: what `snapshot.terminal.decode` sets. +fn takeMeta(t: *Terminal, next: *Terminal, io: std.Io, gpa: std.mem.Allocator) !void { + if (t.cols != next.cols or t.rows != next.rows) try t.resize(gpa, .{ .cols = next.cols, .rows = next.rows }); + t.width_px = next.width_px; + t.height_px = next.height_px; + t.scrolling_region = next.scrolling_region; + t.status_display = next.status_display; + t.previous_char = next.previous_char; + t.cursor = next.cursor; + t.modes = next.modes; + t.flags.shell_redraws_prompt = next.flags.shell_redraws_prompt; + t.flags.modify_other_keys_2 = next.flags.modify_other_keys_2; + t.flags.mouse_event = next.flags.mouse_event; + t.flags.mouse_format = next.flags.mouse_format; + t.flags.mouse_shift_capture = next.flags.mouse_shift_capture; + t.flags.password_input = next.flags.password_input; + t.mouse_shape = next.mouse_shape; + t.colors = next.colors; + try t.setPwd(next.getPwd() orelse ""); + try t.setTitle(next.getTitle() orelse ""); + std.mem.swap(@TypeOf(t.tabstops), &t.tabstops, &next.tabstops); + const key = next.screens.active_key; + if (key != t.screens.active_key) { + if (t.screens.get(key) == null) _ = try t.screens.getInit(io, gpa, key, .{ .cols = t.cols, .rows = t.rows, .max_scrollback_bytes = 0 }); + t.screens.switchTo(key); + } +} + +// ---- checks: a replica against its authority, cell for cell ---- + +/// What a row shows, written out value by value (not `rowHash`'s walk): +/// its flags, then each cell's word without its page's ids, its style, its +/// grapheme and its hyperlink's URI. +fn dumpRow(page: *Page, row: *Row, cols: u16, w: *std.Io.Writer) !void { + try w.print("{d}:", .{rowFlags(row)}); + const all = page.getCells(row); + for (all[0..@min(cols, all.len)]) |*c| { + var word = c.*; + word.style_id = 0; + try w.print(" {x}", .{@as(u64, @bitCast(word))}); + if (c.style_id != 0) try snapshot.style.encode(page.styles.get(page.memory, c.style_id).*, w); + if (c.hasGrapheme()) for (page.lookupGrapheme(c).?) |cp| try w.print("+{x}", .{cp}); + if (c.hyperlink) if (page.lookupHyperlink(c)) |id| try w.print("@{s}", .{page.hyperlink_set.get(page.memory, id).uri.slice(page.memory)}); + } + try w.writeByte('\n'); +} + +/// The state a replica must match: the terminal-wide state, the cursor, +/// the active screen's rows. +fn dumpScreen(t: *Terminal, w: *std.Io.Writer) !void { + const screen = t.screens.active; + try w.print("{d}x{d} {s} cursor {d},{d} {s} {} title {s} pwd {s}\n", .{ + t.cols, t.rows, @tagName(t.screens.active_key), + screen.cursor.x, screen.cursor.y, @tagName(screen.cursor.cursor_style), + screen.cursor.pending_wrap, t.getTitle() orelse "", t.getPwd() orelse "", + }); + try w.writeAll(std.mem.asBytes(&t.modes.values)); + try w.writeAll(std.mem.asBytes(&screen.kitty_keyboard)); + try w.print(" mouse {s} {s} {s}\n", .{ @tagName(t.flags.mouse_event), @tagName(t.flags.mouse_format), @tagName(t.mouse_shape) }); + for (t.colors.palette.current) |c| try w.print("{x}{x}{x}", .{ c.r, c.g, c.b }); + try w.writeByte('\n'); + var it = screen.pages.rowIterator(.right_down, .{ .active = .{} }, null); + while (it.next()) |p| try dumpRow(p.node.page(), p.rowAndCell().row, t.cols, w); +} + +/// An authority and its replica, fed one stream of pty bytes, the deltas +/// and fetches between them at random: a delta encoded at a random step and +/// applied a random number of steps later (so the states between are +/// skipped), random resizes of the authority, random fetches of holes. +/// After each delta, the replica's screen is the authority's as it was +/// encoded, and every history row it holds outside its holes is what the +/// authority holds at that absolute row. +pub const Soak = struct { + gpa: std.mem.Allocator, + io: std.Io, + a: *Terminal, + stream: vt.TerminalStream, + rows: Rows = .{}, + enc: Encoder, + peer: Peer = .{}, + b: *Terminal, + rep: Replica = .{}, + prng: std.Random.DefaultPrng, + /// The delta in flight, the authority's dump when it was encoded, and + /// how many steps it waits before it lands. + delta: std.Io.Writer.Allocating, + expect: std.Io.Writer.Allocating, + got: std.Io.Writer.Allocating, + wait: u32 = 0, + resizes: bool = true, + stats: struct { deltas: u64 = 0, bytes: u64 = 0, fetched: u64 = 0, history_checked: u64 = 0 } = .{}, + + pub fn init(gpa: std.mem.Allocator, io: std.Io, cols: u16, rows: u16, scrollback: usize, seed: u64) !*Soak { + const s = try gpa.create(Soak); + errdefer gpa.destroy(s); + const a = try gpa.create(Terminal); + a.* = try Terminal.init(io, gpa, .{ .cols = cols, .rows = rows, .max_scrollback_bytes = scrollback }); + const b = try gpa.create(Terminal); + b.* = try Terminal.init(io, gpa, .{ .cols = 2, .rows = 2, .max_scrollback_bytes = scrollback }); + s.* = .{ + .gpa = gpa, + .io = io, + .a = a, + .stream = a.vtStream(), + .enc = .init(gpa), + .b = b, + .prng = .init(seed), + .delta = .init(gpa), + .expect = .init(gpa), + .got = .init(gpa), + }; + return s; + } + + pub fn deinit(s: *Soak) void { + const gpa = s.gpa; + s.stream.deinit(); + s.rows.deinit(s.a); + s.a.deinit(gpa); + gpa.destroy(s.a); + s.b.deinit(gpa); + gpa.destroy(s.b); + s.enc.deinit(); + s.peer.deinit(gpa); + s.rep.deinit(gpa); + s.delta.deinit(); + s.expect.deinit(); + s.got.deinit(); + gpa.destroy(s); + } + + /// `bytes` into the authority in random pieces, a step after each. + pub fn feed(s: *Soak, bytes: []const u8) !void { + var at: usize = 0; + const random = s.prng.random(); + while (at < bytes.len) { + const n = @min(bytes.len - at, 1 + random.uintLessThan(usize, 512)); + s.stream.nextSlice(bytes[at..][0..n]); + at += n; + try s.step(); + } + } + + pub fn step(s: *Soak) !void { + const random = s.prng.random(); + if (s.resizes and random.uintLessThan(u32, 200) == 0) { + try s.a.resize(s.gpa, .{ .cols = 10 + random.uintLessThan(u16, 200), .rows = 3 + random.uintLessThan(u16, 60) }); + } + if (!s.peer.in_flight) { + if (random.boolean()) return; // a state skipped + s.delta.clearRetainingCapacity(); + if (!try encode(&s.enc, &s.peer, &s.rows, s.a, &s.delta.writer)) unreachable; + s.expect.clearRetainingCapacity(); + try dumpScreen(s.a, &s.expect.writer); + s.wait = random.uintLessThan(u32, 4); + s.stats.deltas += 1; + s.stats.bytes += s.delta.written().len; + return; + } + if (s.wait > 0) { + s.wait -= 1; + return; + } + try s.land(); + if (random.uintLessThan(u32, 8) == 0) try s.fetch(); + } + + /// The delta in flight applied and acknowledged, and checked. + pub fn land(s: *Soak) !void { + if (!s.peer.in_flight) return; + s.peer.ack(try s.rep.apply(s.gpa, s.io, s.b, s.delta.written())); + s.got.clearRetainingCapacity(); + try dumpScreen(s.b, &s.got.writer); + if (!std.mem.eql(u8, s.expect.written(), s.got.written())) { + reportDiff(s.expect.written(), s.got.written()); + return error.ReplicaDiffers; + } + try s.checkHistory(); + } + + /// Every history row the replica holds, outside its holes and inside + /// what the authority still has, is the authority's (rows in history + /// never change; a resize renumbers and starts the replica afresh). + fn checkHistory(s: *Soak) !void { + // Numbered as of now: the authority may have scrolled on since. + try s.rows.update(s.a); + if (s.rep.epoch != s.rows.epoch) return; + const pages = &s.b.screens.get(.primary).?.pages; + const have = pages.total_rows - pages.rows; + var ra: std.Io.Writer.Allocating = .init(s.gpa); + defer ra.deinit(); + var rb: std.Io.Writer.Allocating = .init(s.gpa); + defer rb.deinit(); + var d: u64 = 1; + while (d <= have) : (d += 1 + d / 8) { + const abs = s.rep.top - d; + if (s.rep.inHole(abs)) continue; + const pb = pinOf(pages, s.rep.top, abs) orelse continue; + const pa = s.rows.pinAt(s.a, abs) orelse continue; + ra.clearRetainingCapacity(); + rb.clearRetainingCapacity(); + try dumpRow(pa.node.page(), pa.rowAndCell().row, s.a.cols, &ra.writer); + try dumpRow(pb.node.page(), pb.rowAndCell().row, s.b.cols, &rb.writer); + if (!std.mem.eql(u8, ra.written(), rb.written())) { + std.debug.print("history row {d} ({d} above the top) differs\n", .{ abs, d }); + reportDiff(ra.written(), rb.written()); + return error.ReplicaHistoryDiffers; + } + s.stats.history_checked += 1; + } + } + + /// Some holes fetched from the authority, as a replica scrolled back. + fn fetch(s: *Soak) !void { + if (s.rep.holes.items.len == 0) return; + const random = s.prng.random(); + const hole = s.rep.holes.items[random.uintLessThan(usize, s.rep.holes.items.len)]; + const lo = hole[0] + random.uintLessThan(u64, hole[1] - hole[0]); + const hi = @min(hole[1], lo + 1 + random.uintLessThan(u64, 80)); + var out: std.Io.Writer.Allocating = .init(s.gpa); + defer out.deinit(); + try encodeRows(&s.enc, &s.rows, s.a, lo, hi, &out.writer); + try s.rep.applyRows(s.gpa, s.b, out.written()); + s.stats.fetched += hi - lo; + try s.checkHistory(); + } +}; + +fn reportDiff(want: []const u8, got: []const u8) void { + var wl = std.mem.splitScalar(u8, want, '\n'); + var gl = std.mem.splitScalar(u8, got, '\n'); + var line: usize = 0; + while (true) : (line += 1) { + const a = wl.next(); + const b = gl.next(); + if (a == null and b == null) return; + if (a != null and b != null and std.mem.eql(u8, a.?, b.?)) continue; + std.debug.print("line {d} differs:\n authority {s}\n replica {s}\n", .{ line, a orelse "(none)", b orelse "(none)" }); + return; + } +} + +/// Drive one stream through a soak and land a last delta of its end. +pub fn soak(gpa: std.mem.Allocator, io: std.Io, bytes: []const u8, cols: u16, rows: u16, scrollback: usize, seed: u64, resizes: bool) !@FieldType(Soak, "stats") { + const s = try Soak.init(gpa, io, cols, rows, scrollback, seed); + defer s.deinit(); + s.resizes = resizes; + try s.feed(bytes); + try s.land(); + s.delta.clearRetainingCapacity(); + _ = try encode(&s.enc, &s.peer, &s.rows, s.a, &s.delta.writer); + s.expect.clearRetainingCapacity(); + try dumpScreen(s.a, &s.expect.writer); + try s.land(); + return s.stats; +} + +/// Every recorded pty stream in `dir` (a `-x` in its name sizes +/// it, 80x24 otherwise) through a soak at three seeds, then the encode's +/// cost and the bytes floods put on the wire: the test below, when +/// PARDES_SCREEN_STATE names the directory (numbers from a ReleaseFast +/// build: `zig build core-test -Doptimize=ReleaseFast -Dtest-filter=screen_state`). +fn perf(gpa: std.mem.Allocator, io: std.Io, dir_path: []const u8, w: *std.Io.Writer) !void { + var dir = try std.Io.Dir.cwd().openDir(io, dir_path, .{ .iterate = true }); + defer dir.close(io); + var it = dir.iterate(); + var files: usize = 0; + var failed: usize = 0; + while (try it.next(io)) |entry| { + if (entry.kind != .file) continue; + const bytes = try dir.readFileAlloc(io, entry.name, gpa, .unlimited); + defer gpa.free(bytes); + var cols: u16 = 80; + var rows: u16 = 24; + if (std.mem.lastIndexOfScalar(u8, entry.name, '-')) |dash| { + const size = std.mem.sliceTo(entry.name[dash + 1 ..], '.'); + if (std.mem.indexOfScalar(u8, size, 'x')) |x| { + cols = std.fmt.parseInt(u16, size[0..x], 10) catch cols; + rows = std.fmt.parseInt(u16, size[x + 1 ..], 10) catch rows; + } + } + files += 1; + for (0..3) |seed| { + const st = soak(gpa, io, bytes, cols, rows, 128 * 1024, seed, seed != 0) catch |err| { + try w.print("screen-state {s} seed {d}: {s}\n", .{ entry.name, seed, @errorName(err) }); + failed += 1; + continue; + }; + if (seed == 0) try w.print("screen-state {s} {d}x{d}: {d} bytes, {d} deltas, {d} wire bytes, {d} history rows checked, {d} fetched\n", .{ entry.name, cols, rows, bytes.len, st.deltas, st.bytes, st.history_checked, st.fetched }); + } + } + try w.print("screen-state corpus: {d} streams x 3 seeds, {d} failed\n", .{ files, failed }); + try w.flush(); + try bench(gpa, io, w); +} + +fn bench(gpa: std.mem.Allocator, io: std.Io, w: *std.Io.Writer) !void { + const t = try gpa.create(Terminal); + defer gpa.destroy(t); + t.* = try Terminal.init(io, gpa, .{ .cols = 200, .rows = 60, .max_scrollback_bytes = 16 << 20 }); + defer t.deinit(gpa); + var stream = t.vtStream(); + defer stream.deinit(); + var enc: Encoder = .init(gpa); + defer enc.deinit(); + var n: Rows = .{}; + defer n.deinit(t); + var out: std.Io.Writer.Allocating = .init(gpa); + defer out.deinit(); + var line: [1024]u8 = undefined; + // A full 200x60 screen, every row new to the peer: each a blob. + var full: [200]u64 = undefined; + var same: [200]u64 = undefined; + var full_bytes: usize = 0; + var same_bytes: usize = 0; + for (&full, &same, 0..) |*f, *u, rep| { + for (0..60) |y| { + var lw: std.Io.Writer = .fixed(&line); + try lw.print("\x1b[{d};1H\x1b[0;3{d};4{d}m{d} {d} ", .{ y + 1, rep % 8, y % 8, rep, y }); + while (lw.end < 190) try lw.writeAll("\x1b[1mbold\x1b[22m text "); + stream.nextSlice(lw.buffered()); + } + var peer: Peer = .{}; + defer peer.deinit(gpa); + out.clearRetainingCapacity(); + var t0 = std.Io.Clock.awake.now(io).nanoseconds; + _ = try encode(&enc, &peer, &n, t, &out.writer); + f.* = @intCast(std.Io.Clock.awake.now(io).nanoseconds - t0); + full_bytes = out.written().len; + peer.ack(peer.seq); + out.clearRetainingCapacity(); + t0 = std.Io.Clock.awake.now(io).nanoseconds; + _ = try encode(&enc, &peer, &n, t, &out.writer); + u.* = @intCast(std.Io.Clock.awake.now(io).nanoseconds - t0); + same_bytes = out.written().len; + } + std.mem.sort(u64, &full, {}, std.sort.asc(u64)); + std.mem.sort(u64, &same, {}, std.sort.asc(u64)); + try w.print("screen-state encode 200x60, every row new: p50 {d} us p99 {d} us, {d} bytes\n", .{ full[100] / 1000, full[198] / 1000, full_bytes }); + try w.print("screen-state encode 200x60, nothing new: p50 {d} us p99 {d} us, {d} bytes\n", .{ same[100] / 1000, same[198] / 1000, same_bytes }); + + // Floods: a second of output as fast as ghostty takes it, a delta every + // 16 ms when the last is acknowledged (by a replica applying it at once). + const yes = try yesStream(gpa, 1 << 16); + defer gpa.free(yes); + const shell = try shellStream(gpa); + defer gpa.free(shell); + for ([_]struct { name: []const u8, bytes: []const u8 }{ .{ .name = "yes", .bytes = yes }, .{ .name = "varied text", .bytes = shell } }) |flood| { + const s = try Soak.init(gpa, io, 200, 60, 16 << 20, 0); + defer s.deinit(); + const start = std.Io.Clock.awake.now(io).nanoseconds; + var next = start; + var fed: u64 = 0; + var deltas: u64 = 0; + var wire: u64 = 0; + var apply_ns: u64 = 0; + while (true) { + const now = std.Io.Clock.awake.now(io).nanoseconds; + if (now - start >= std.time.ns_per_s) break; + s.stream.nextSlice(flood.bytes); + fed += flood.bytes.len; + if (now < next) continue; + next += 16 * std.time.ns_per_ms; + s.delta.clearRetainingCapacity(); + _ = try encode(&s.enc, &s.peer, &s.rows, s.a, &s.delta.writer); + const a0 = std.Io.Clock.awake.now(io).nanoseconds; + s.peer.ack(try s.rep.apply(gpa, io, s.b, s.delta.written())); + apply_ns += @intCast(std.Io.Clock.awake.now(io).nanoseconds - a0); + deltas += 1; + wire += s.delta.written().len; + } + try w.print("screen-state flood of {s}: {d} MB/s of output, {d} deltas/s, {d} wire bytes/s ({d} a delta), replica apply {d} us a delta\n", .{ flood.name, fed / 1_000_000, deltas, wire, wire / @max(1, deltas), apply_ns / @max(1, deltas) / 1000 }); + } + try w.flush(); +} + +const testing = std.testing; + +test "recorded streams through replicas, and the encode's cost (PARDES_SCREEN_STATE=)" { + const dir = std.c.getenv("PARDES_SCREEN_STATE") orelse return error.SkipZigTest; + var buf: [4096]u8 = undefined; + // stderr: a test's stdout is the build runner's. + var out = std.Io.File.stderr().writer(testing.io, &buf); + try perf(testing.allocator, testing.io, std.mem.span(dir), &out.interface); +} + +/// A shell's: prompts marked (OSC 133), colours of the three kinds, wide +/// and combined graphemes, hyperlinks, tabs, a title and a directory, long +/// lines that wrap, and a clear (ED2, ED3) now and then. +fn shellStream(gpa: std.mem.Allocator) ![]u8 { + var out: std.Io.Writer.Allocating = .init(gpa); + errdefer out.deinit(); + const w = &out.writer; + try w.writeAll("\x1b]0;a shell\x07\x1b]7;file://host/home/someone\x07"); + for (0..400) |i| { + try w.print("\x1b]133;A\x07$ \x1b]133;B\x07ls -l {d}\r\n\x1b]133;C\x07", .{i}); + try w.print("\x1b[1;31mred\x1b[0m \x1b[4;38;5;{d}m256\x1b[0m \x1b[38;2;{d};100;200;48;5;17mtrue\x1b[0m 漢字 e\u{301} 👩\u{200d}💻 ", .{ i % 256, i % 256 }); + try w.print("\x1b]8;;https://example.com/{d}\x07link\x1b]8;;\x07\tx\r\n", .{i}); + if (i % 7 == 0) { + for (0..12) |_| try w.writeAll("a long line that wraps "); + try w.writeAll("\r\n"); + } + try w.writeAll("\x1b]133;D;0\x07"); + if (i % 97 == 96) try w.writeAll("\x1b[H\x1b[2J\x1b[3J"); + if (i % 50 == 0) try w.print("\x1b]2;title {d}\x07\x1b]4;1;rgb:{x:0>2}/20/30\x07", .{ i, i % 256 }); + } + return out.toOwnedSlice(); +} + +/// A full-screen program's: the alternate screen and its modes, a status +/// line, a scrolling region scrolled both ways, lines inserted and +/// deleted, cursor shapes, the kitty keyboard pushed and popped. +fn tuiStream(gpa: std.mem.Allocator) ![]u8 { + var out: std.Io.Writer.Allocating = .init(gpa); + errdefer out.deinit(); + const w = &out.writer; + try w.writeAll("before the program\r\n\x1b[?1049h\x1b[?25l\x1b[?1000h\x1b[?1006h\x1b[?2004h\x1b[?1h\x1b[>1u\x1b[4 q\x1b[H\x1b[2J"); + for (0..300) |f| { + try w.print("\x1b[1;1H\x1b[44;97m status {d} \x1b[K\x1b[0m", .{f}); + try w.print("\x1b[3;18r\x1b[18;1H\r\n\x1b[3{d}mline {d}\x1b[0m", .{ f % 8, f }); + if (f % 3 == 0) try w.writeAll("\x1b[3;1H\x1bM\x1b[2mreversed in\x1b[0m"); + if (f % 5 == 0) try w.writeAll("\x1b[6;1H\x1b[2L\x1b[9;1H\x1b[M"); + try w.writeAll("\x1b[r"); + try w.print("\x1b[{d};{d}H\x1b[7m*\x1b[0m\x1b[{d}X", .{ 2 + f % 20, 1 + (f * 7) % 70, f % 5 }); + if (f % 11 == 0) try w.writeAll("\x1b[?25h\x1b[6 q") else if (f % 11 == 5) try w.writeAll("\x1b[?25l\x1b[2 q"); + } + try w.writeAll("\x1b[ 10); + fetched += st.fetched; + history += st.history_checked; + } + } + // The holes were fetched, and history rows compared, at all. + try testing.expect(fetched > 0 and history > 0); +} + +test "under a flood a delta is a screen of hashes: an unchanged row costs its address alone" { + const gpa = testing.allocator; + const s = try Soak.init(gpa, testing.io, 200, 60, 1 << 20, 1); + defer s.deinit(); + s.resizes = false; + const yes = try yesStream(gpa, 10_000); + defer gpa.free(yes); + for (0..3) |_| { + s.stream.nextSlice(yes); + s.delta.clearRetainingCapacity(); + try testing.expect(try encode(&s.enc, &s.peer, &s.rows, s.a, &s.delta.writer)); + s.expect.clearRetainingCapacity(); + try dumpScreen(s.a, &s.expect.writer); + try s.land(); + } + // The last: 60 rows and the tail's 60, of hashes the replica holds, no + // blob; the header and counts around them. + const d = s.delta.written(); + const at = 4 + 1 + 4 + 8 + 2 + 2 + 1 + 1 + @sizeOf(KittyFlags) + 8 + 8; + const nrows = std.mem.readInt(u16, d[at..][0..2], .little); + const tail = std.mem.readInt(u16, d[at + 2 + nrows * 8 ..][0..2], .little); + const blobs = std.mem.readInt(u16, d[at + 4 + (nrows + tail) * 8 ..][0..2], .little); + if (d.len >= (nrows + tail) * 8 + 128) std.debug.print("delta {d} bytes: {d} rows, {d} tail, {d} blobs\n", .{ d.len, nrows, tail, blobs }); + try testing.expectEqual(@as(u16, 0), blobs); + try testing.expect(d.len < (nrows + tail) * 8 + 128); + try testing.expect(s.rep.holes.items.len > 0); +} + +test "a replica's terminal pane draws as its authority's does" { + const pardes = @import("pardes.zig"); + const gpa = testing.allocator; + const a = try pardes.Pardes.init(gpa, .{ .tty_only = true, .cols = 60, .rows = 16 }); + defer a.deinit(); + const b = try pardes.Pardes.init(gpa, .{ .tty_only = true, .cols = 60, .rows = 16 }); + defer b.deinit(); + while (a.nextEffect()) |_| {} + while (b.nextEffect()) |_| {} + var fa = std.heap.ArenaAllocator.init(gpa); + defer fa.deinit(); + var fb = std.heap.ArenaAllocator.init(gpa); + defer fb.deinit(); + // Laid out once, so both panes have their size before the first delta. + _ = try a.render(fa.allocator()); + _ = try b.render(fb.allocator()); + const ta = &a.panes[0].?.terminal.?.vt; + const tb = &b.panes[0].?.terminal.?.vt; + var enc: Encoder = .init(gpa); + defer enc.deinit(); + var peer: Peer = .{}; + defer peer.deinit(gpa); + var rows: Rows = .{}; + defer rows.deinit(ta); + var rep: Replica = .{}; + defer rep.deinit(gpa); + var delta: std.Io.Writer.Allocating = .init(gpa); + defer delta.deinit(); + var line: [256]u8 = undefined; + for (0..60) |i| { + const bytes = try std.fmt.bufPrint(&line, "\x1b[1;3{d}m{d}\x1b[0m 漢字 e\u{301} \x1b[48;2;{d};40;60mbg\x1b[0m and then some text that wraps past the width of the pane\r\n", .{ i % 8, i, i * 4 % 256 }); + a.update(.{ .output = .{ .pane = 0, .bytes = bytes } }); + delta.clearRetainingCapacity(); + try testing.expect(try encode(&enc, &peer, &rows, ta, &delta.writer)); + peer.ack(try rep.apply(gpa, terminalIoForTest(), tb, delta.written())); + b.needs_frame = true; + _ = fa.reset(.retain_capacity); + _ = fb.reset(.retain_capacity); + const sa = try a.render(fa.allocator()); + const sb = try b.render(fb.allocator()); + const r = a.rects[0]; + // The body: below the pane's tag row. + for (r.y + 1..r.y + r.h) |y| for (r.x..r.x + r.w) |x| { + const ca = sa.at(@intCast(x), @intCast(y)); + const cb = sb.at(@intCast(x), @intCast(y)); + if (!ca.visuallyEqual(cb)) { + std.debug.print("step {d}: cell {d},{d}: '{s}' vs '{s}'\n", .{ i, x, y, ca.grapheme(), cb.grapheme() }); + return error.ReplicaDrawsDifferently; + } + }; + } +} + +fn terminalIoForTest() std.Io { + return std.Io.Threaded.global_single_threaded.io(); +} -- cgit v1.3 From 74286e81e88a6df7586ead9e8bcd503d44446a85 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 17:58:06 -0300 Subject: docs: views in the guide and reference (a session attached already takes another view, up to four; Detach lets one go); phase 4 notes in docs/next-daemon.md; tty: Screen.hungUp reads a test terminal's fd as the test build has it Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 124 +++++++++++++++++++++++++++++++++++++++++++++++++ docs/typ/guide.typ | 6 ++- docs/typ/reference.typ | 11 ++++- src/tty/tty.zig | 3 +- 4 files changed, 140 insertions(+), 4 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index e3be6af8..b5939b9d 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -271,6 +271,130 @@ termios` into Zig's `std.c.termios`, which is 44 (the kernel's layout), and overran the stack, clobbering a neighbouring local in the forked child; `std.posix.tcgetattr` (an ioctl on that layout) is the right one. +## Phase 4 notes + +Built 2026-10-29 (`src/views.zig`, the tty and GUI hosts, `daemon.zig`, +the snapshot harness and the monkey). Decisions taken at the forks, with +the ponytail lens: + +- **A view is swapped in, not pointed at.** The ~1,900 references to the + core's layout, focus and pane state stay as they are: the core sees one + view, the live one, in the fields it always had. A view's share of the + session is a list of field names (`views.fields`, ~60) made into a struct + by one comptime `@Struct` (`Part`), and of each pane a struct beside it + (`PaneView`, in `Pane.pv`); going into view `k` copies the live one's out + (`park`) and `k`'s in (`unpark`). Copies, not swaps: a slot is stale + while its view is live and written whenever it goes out, so no two places + own a selection's heap. One view never parks: the single client pays + `views.several` (an AND and a compare) in the pump, the step's sync and an + edit, and one store a key (`typedInto`). Rejected: a `*View` the core + reads through (every reference rewritten, an indirection on every + access); a twin pane per view (Zerox exists only for files, and a + terminal's pty is its pane's slot). +- **Where each piece of state went.** + - Each view's own (`views.fields`): the screen size and cell metrics; + the columns, their weights, panes, serials, tags and the finder; rects; + `active`, `active_column` and the header focus; the jump list (kakoune's + is per client); the pointer, hover, drag, mouse grab, clicks and chords; + key prefixes (`"x` register name, leader, Ctrl-w, window pick); a + pending clipboard read; the presentation and the animations (lifts, + cursor glide); `windowed`, `native_images` and `native_image_patches`, + which the census had as watched host writes: a tty view on kitty and + one on a plain terminal differ. The 223 comptime platform branches stay + comptime: a session is one build, which the semantics hash says. + - Each pane's, each view's own (`PaneView`): `vweight`, `collapsed` and + `attached_to` (a layout's); `focus`; the body's mode, cursor, + selections, sticky column, pending normal command and find; a document's + scroll, `hscroll` and size; and what the step measured for that view + (`tag_rows`, `tag_top`, `tag_scroll`, `view`, `notices`), so a view drawn + without a step of its own draws its own geometry. + - The session's, every view's: the panes' text, undo, terminals and + their scroll (tmux shares a pane's copy mode too), tags (acme's are + per window; one tag per buffer is one text, its caret included), the + workspace tag, prompts, messages, registers (decided), the macro being + recorded, look-hover (one pointer at a time), mouse sweeps, query + panes, search state, settings and theme, the effect and input queues, + `needs_frame` (any change draws every view). +- **A terminal shown at two sizes** takes the size of the view that typed + into it last (`Pane.sized_by`, set by a key or a paste; a new pane's is + its maker's; a closed view's goes to the next to sync it). The others show + it at that size: `sync` resizes no pty for them, and `pane.cols/rows` + stay the terminal's own, so the terminal code's invariant (pane size is + the emulator's) holds. ponytail: a smaller view sees its top rows, cut; + panning to the cursor as tmux does is the upgrade, if a small view must + follow a shell it never types into. +- **Panes come and go under parked views.** A view keeps the serial of + each slot when it goes out; unparked, the ids whose serial changed leave + its layout and its keyboard moves to its first pane left. A pane focused + in a view that does not show it (a Look at a file another view opened, a + jump, a 9P focus) is placed in the view's active column (`views.show`, + kakoune's buffer shown in the window that asked). A text edited in one + view moves every other view's cursor and scroll with the splice, as a + Zerox twin's (`File.ParkedPlaces`); their selections go, as a twin's do. +- **Drawing the other views stays out of the digest.** The pump draws each + other view by unparking it, rendering and presenting, and restores the + live one exactly (`drawOthers`): no digested field moves, so the headless + replica, which draws nothing and skips it, agrees. What drawing a view + makes of its own (its animations, its presentation) is kept; what its + unpark normalised (a cursor clamped inside a changed text, a pruned + layout) is not, and is done again when it next goes live by an input, + in every replica alike. `parked` and `pv` are `.local`: a parked view is + checked when it is live again. Found by `pardes-monkey --views=3 --tape`: + a drawn view's `post_present` (acknowledgePanelPresentation, which moves + the pointer's hover) was applied by the headless replica to the live view; + other views take no post_present, their panels acknowledged when they + are live and drawn. +- **Inputs.** `toView`, `openView` and `closeView` are taped. A host goes + into a view before the input its client sent: the tty's terminal readers + post through a `Poster` that puts a `from` ahead of a terminal's input + whenever another terminal posted last (so one terminal posts none, and + coalesced motions never cross views); the GUI asks SDL which window an + event is for (`route`), which also takes a window's resize to its view + (the GUI sizes the live view from its window each frame). +- **The tty.** Each terminal is a `Screen` of its own: its vaxis, input + reader, frame answer (`FrameAck`, was global), kitty images and shm probe + (were global), glide, trail and flash, its stub and its Killed slot. The + primary's stub stays `daemon.stub`. A SIGWINCH asks every terminal its + size (attach stubs forward theirs; another view's terminal is no one's + controlling terminal); a SIGHUP asks each whether its terminal or stub's + pipe hung up, where it used to mean the one terminal. Detach, or a + terminal's end, with views left closes only that view and tells its stub + `d`; the session's end tells every stub. ponytail: a bracketed paste in + progress is the shell's, not a terminal's: keys from another terminal + during it join it. +- **The GUI.** One `Gui` (device, atlas, fonts, pipelines) and its + window's share swapped like the core's (`window_fields`: the SDL window + and its presenting, scroll and glide, scene and post chain, pointer, + blink, captures); `Shell.follow` keeps the Gui on the core's live view. + Closing a window while others stay closes its view; Restore closes the + others. ponytail: a second window's compositor hit test reads whichever + window's state is in the Gui (the workspace tag is in the same place in + every view). +- **The daemon.** `attach` names an attached session too: it takes another + view (a terminal, or a window). With no name, the detached sessions are + asked first, then the only session of that kind. +- **Harnesses, extended not new.** A snapshot script's `start ... views=N` + gives the app N-1 more ptys (PARDES_TEST_VIEWS, opened as attached ones + from the start), `view ` types into and captures terminal k, and every + wait reads all of them (a pty nobody reads would block the session's + frame); a `resize` there SIGWINCHes the app, as a stub would. + `pardes-monkey --views=N` switches among N terminals a quarter of its + steps, each with its own size; `zig build replay` runs its seeds at one + view and at three. The GUI's test feed takes `ESC]777;view;BEL`, and + PARDES_TEST_VIEWS=N opens N more hidden windows, captured under view-k. +- **What proved wrong in the design.** "Views" in the design were a + struct the core reads through; the swap made them cheaper and kept every + reference. Its rollback copy (phase 5) is the same list of fields. The + design's view-local state assumed a step per view; the step's + measurements (`tag_rows`, `notices`, the pane's geometry key) had to + become per view too, or a view drawn after another's step drew the + other's tag heights. And the census's single "frontend kind" leak is not + per process but per client: kitty graphics are a terminal's. + +Measured: PERF_TABLE + +Gates: GATES_TABLE + ## FreeBSD notes Groundwork for phase 5's main test, a client here and the session on the diff --git a/docs/typ/guide.typ b/docs/typ/guide.typ index 5dea0522..40802ed9 100644 --- a/docs/typ/guide.typ +++ b/docs/typ/guide.typ @@ -350,7 +350,11 @@ pardes-gui --attach=work a window session's window again leaves the session running, its panes, shells and files kept; so does a terminal that closes. `pardes --attach` takes it back in any terminal, `pardes-gui --attach` a window session's window: a terminal session takes -a terminal, a window session a window. A session is named by `--9p=` or +a terminal, a window session a window. Attached already, it takes another +view: each terminal or window has its own columns, focus, cursors and +scroll over the same panes, texts and registers, and a terminal shown in +two takes the size of the one that last typed into it. #word("Detach") in +one lets only that one go. A session is named by `--9p=` or `--detach=`, else its pid. A session killed outright leaves its shells running; #word("Adopt") in another takes them in as terminal panes, what they showed gone. One `pardes --daemon` per user keeps all of it, and diff --git a/docs/typ/reference.typ b/docs/typ/reference.typ index c95a574a..190adba7 100644 --- a/docs/typ/reference.typ +++ b/docs/typ/reference.typ @@ -1011,9 +1011,16 @@ to its end before the write is answered, and one that finds nothing fails, `--no-prefix` diff's paths are kept. - *Sessions.* Bare `--detach` names the session after its pid; bare `--attach` needs exactly one detached session of its kind (a terminal's - for `pardes`, a window's for `pardes-gui`); a second `--detach=NAME` + for `pardes`, a window's for `pardes-gui`), or with none detached the + only session of that kind; a second `--detach=NAME` while NAME runs (a plain session is named by its pid) says so in words - and exits 1. One terminal or window shows a session at a time. With none + and exits 1. A session attached already takes another view, up to four: + each terminal or window its own layout, focus, cursors, selections, + modes, scroll and jumps, over the panes, texts, tags, terminals, + messages and registers they share. A terminal pane shown in two takes the + size of the view that typed into it last; a smaller view shows its top + rows. #word("Detach"), or a terminal closing, lets that view go and + leaves the others. With none attached, `size C R` sets the screen, messages clear by the clock, and a pty reports its size in pixels at the last cell size (8×16 before any), as CSI 14, 16 and 18 t do. Attaching from another machine is gone until diff --git a/src/tty/tty.zig b/src/tty/tty.zig index f949d571..b56e74b1 100644 --- a/src/tty/tty.zig +++ b/src/tty/tty.zig @@ -1329,8 +1329,9 @@ const Screen = struct { /// Its terminal hung up, or the stub that waits on it is gone. fn hungUp(sc: *const Screen) bool { const stub = if (sc.primary) daemon.stub else sc.stub; + const handle = if (@TypeOf(sc.tty.fd) == std.Io.File) sc.tty.fd.handle else sc.tty.fd; var fds = [_]posix.pollfd{ - .{ .fd = sc.tty.fd.handle, .events = 0, .revents = 0 }, + .{ .fd = handle, .events = 0, .revents = 0 }, .{ .fd = stub, .events = 0, .revents = 0 }, }; _ = posix.poll(fds[0..if (stub >= 0) 2 else 1], 0) catch return false; -- cgit v1.3 From b3b2768cc82105fa2a1845b7a7a3767c51173afb Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 18:39:54 -0300 Subject: docs/next-daemon.md: phase 4's gates and perf (every cell inside its A/A), and what is not done; ponytail marks for the shortcuts left (a paste shared by every terminal, a second window's hit test, one display density per process) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++-- src/gui/gui.zig | 7 +++++- src/tty/tty.zig | 2 ++ 3 files changed, 72 insertions(+), 3 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index b5939b9d..3d4c8a08 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -382,6 +382,17 @@ the ponytail lens: steps, each with its own size; `zig build replay` runs its seeds at one view and at three. The GUI's test feed takes `ESC]777;view;BEL`, and PARDES_TEST_VIEWS=N opens N more hidden windows, captured under view-k. +- **Not done, and why.** + - A view of the other kind (a terminal on a window session, a window on + a terminal's): each process links one frontend; the daemon still + matches kinds. + - 9P acts in the live view, the one whose client spoke last: no view is + named in a path yet (a script's focus or layout is that view's). + - Every change draws every view (`needs_frame` is the session's); a + view's own dirtiness waits for a measurement that wants it. + - Prompts are the pane's: a `.name` prompt drafted in one view is + closed by another view's step, where that pane has not the keyboard. + - Dump, Restore and Handover keep one view, the live one. - **What proved wrong in the design.** "Views" in the design were a struct the core reads through; the swap made them cheaper and kept every reference. Its rollback copy (phase 5) is the same list of fields. The @@ -391,9 +402,60 @@ the ponytail lens: other's tag heights. And the census's single "frontend kind" leak is not per process but per client: kitty graphics are a terminal's. -Measured: PERF_TABLE +Measured: ReleaseFast, the base binaries from before the rework against +this line on phase 1's merge (vvmuwntp, before main's tty throughput +fix), interleaved base, new, base ... in one run on the perf cores, then +an A/A of the base against itself the same way; the ratio of the medians +of runs, new/base (4 runs of new, 5 of base; `--reps 25` for the core, +`--live` and `--live-gui` through the 32 MiB live driver): -Gates: GATES_TABLE +| | new/base | A/A | +|---|---|---| +| core, medium: render, keys, PageDown, wheel | 0.97-1.02 | 0.99-1.01 | +| core, medium: edit-char, edit-twin, deep-sel | 0.90-0.96 | 0.99-1.00 | +| core, medium: open, body-2m, body-appends | 0.94-1.02 | 0.88-1.08 | +| tty, keystroke to present p50 (5 gestures) | 0.94-1.11 | 0.90-1.02 | +| tty, the core's part of it p50 | 0.95-1.04 | 0.86-0.99 | +| tty, `yes \| head -c 32M` wall; 1M-line open | 0.96; 0.98 | 1.09; 1.02 | +| GUI, keystroke to present p50 (present-bound, 30 ms) | 0.98-1.01 | 1.03-1.10 | +| GUI, the core's part of it p50 | 0.84-1.01 | 1.01-1.19 | + +Every cell is inside its A/A; the widest, the tty's PageDown at 1.11 +(runs 386-506 us against 410-451), is 0.94 best against best. What one +view pays is a byte test in the pump, the step's sync and an edit +(`views.several`), a store a key, an index for the live terminal or +window, and a `from` never posted. A first A/B, of this line before the +rebase onto phase 1's merge (its render still made the geometry afresh +every frame), read the GUI's core at 0.89-1.16 with no A/A to set it +against; it is not the one reported. + +Two views (not gated): every frame draws both, so a change costs two +renders; a view switch copies ~60 fields and each pane's share, a few +microseconds. + +Gates, on the line rebased onto main's tty throughput fix (qzsouslm): + +- `zig build replay -Dplatform=tty`: 112 snapshot scripts pass (the 110, + views.snap and views-term.snap), their 130 tapes replay in all three + replicas (390 runs, 29,355 digests); 20 monkey seeds at one view (60 + runs, 14,010 digests) and 20 at three views (60 runs, 9,720 digests); + no divergence, no panic. views.snap: 75 digests a replica, + views-term.snap 35. +- Unit tests: tty 1478 of 1498 pass, 20 skipped; GUI 1549 of 1560, 9 + skipped and 2 failed, the same two that fail on phase 1's line (`tagline + pointer mapping retains physical grips...` and `a tag caret past its + row's end glides...`). The views' own: 7 (own cursor, scroll, mode, + layout and focus; an edit and a register shared; a terminal sized by its + last typist; views closing; the other views drawn leave the digest; a + view keeps its place in a text another edits). +- test/fs.py passes (with a TMPDIR short enough for the 9P sockets' + paths). test/daemon.py: 20 detach and attach cycles, then a second + terminal on the attached session, both drawn, detached from inside and + then closed outright, the first drawing on, and the kill and Adopt; + `--gui`: 10 window cycles, then a second window on the open one, + detached alone. +- `pardes-monkey --views=3 --tape` found the one divergence fixed here + (the drawn views' post_present). ## FreeBSD notes diff --git a/src/gui/gui.zig b/src/gui/gui.zig index 2d9124cc..a342cfe2 100644 --- a/src/gui/gui.zig +++ b/src/gui/gui.zig @@ -2333,6 +2333,9 @@ fn reopenWindow(g: *Gui) bool { g.window = window; if (g.window_slot) |slot| slot.* = window; updateDisplayScale(window); + // ponytail: another view's window asks the hit test of whichever + // window's share the Gui holds then (the workspace tag sits in the same + // place in every view); a test per window if one moves it. if (!g.capture) _ = c.SDL_SetWindowHitTest(window, hitTest, g); _ = c.SDL_StartTextInput(window); return true; @@ -9604,7 +9607,9 @@ fn rulePx(chrome: *const pardes.Chrome) f32 { /// by which every size given in logical pixels is drawn, as acme scales its /// Border, ButtonBorder and Scrollgap (scalesize). Cells, glyphs and what is /// measured in them are already the display's pixels and are not scaled. -/// ponytail: one window per process, so one global; per window if ever two. +/// ponytail: one per process: a session's windows (views.zig) on screens +/// of two densities draw at the density of the last one opened or moved; +/// a field of `window_fields` when that matters. var display_scale: f32 = 1; /// The window's display scale, or PARDES_TEST_SCALE (a harness forcing 2x); diff --git a/src/tty/tty.zig b/src/tty/tty.zig index b56e74b1..a7048645 100644 --- a/src/tty/tty.zig +++ b/src/tty/tty.zig @@ -1677,6 +1677,8 @@ const Shell = struct { pipe_tasks: PipeTasks = .{}, threads_ok: bool = false, check_files: bool = false, + // ponytail: one bracketed paste at a time for every terminal: keys + // another terminal sends while one pastes join its paste. in_paste: bool = false, /// PARDES_TEST_CLOCK's virtual time (host_io.testClock), and where the /// running-out wait moves it: the core's next wake. -- cgit v1.3 From 32b80ad0393321d44f929e501e5515bbf6e51c12 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 19:52:55 -0300 Subject: views: a terminal shown in a view shorter than it is passed down to its cursor, a shell's prompt never out of sight, an arrow in the gutter where rows are above; a name drafted in a tag is the view's that opened it A terminal takes the size of the view that typed into it last; a shorter view used to show its top rows, the prompt below them cut. It now passes over as many as keep the cursor's row in sight, as tmux pans a bigger window in a smaller client: `Pane.pan` (per view, derived) is made in the step's sync and again when a view is drawn out of turn (views.unpark), by terminal.panFor (0 in the view that sizes it, in one tall enough, or folded), and read where the body's rows start: `Pane.scroll`, the body walk in tty mode, the visible cursor. One view never pans: the view that sizes a terminal returns 0 at its first test. A `.name` prompt (a file's name drafted in its tag) is the view's that opened it (`Pane.prompt_view`): another view's step, where that pane has not the keyboard, no longer drops it; its view closing does. views-pan.snap: a shell sized for a 24-row view, its last lines and prompt in a 12-row one under the arrow, then typed into from the short view and sized for it. A unit test: the short view's frame has the cursor's row, and the line on it, inside the pane. docs: the notes, the reference. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 41 +++++++++++++++++++------------- docs/typ/reference.typ | 4 ++-- src/draw.zig | 3 +++ src/panes.zig | 11 +++++++-- src/pardes.zig | 11 +++++---- src/tagline.zig | 1 + src/terminal.zig | 24 ++++++++++++++++--- src/views.zig | 49 +++++++++++++++++++++++++++++++++++++- test/snapshots/views-pan.golden | 52 +++++++++++++++++++++++++++++++++++++++++ test/snapshots/views-pan.snap | 27 +++++++++++++++++++++ 10 files changed, 194 insertions(+), 29 deletions(-) create mode 100644 test/snapshots/views-pan.golden create mode 100644 test/snapshots/views-pan.snap (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 3d4c8a08..e7ddcf5f 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -314,15 +314,23 @@ the ponytail lens: workspace tag, prompts, messages, registers (decided), the macro being recorded, look-hover (one pointer at a time), mouse sweeps, query panes, search state, settings and theme, the effect and input queues, - `needs_frame` (any change draws every view). + `needs_frame` (any change draws every view). A `.name` prompt (a + file's name drafted in its tag) is the view's that opened it + (`Pane.prompt_view`): only that view's focus leaving the pane drops it, + and its closing. - **A terminal shown at two sizes** takes the size of the view that typed into it last (`Pane.sized_by`, set by a key or a paste; a new pane's is its maker's; a closed view's goes to the next to sync it). The others show it at that size: `sync` resizes no pty for them, and `pane.cols/rows` stay the terminal's own, so the terminal code's invariant (pane size is - the emulator's) holds. ponytail: a smaller view sees its top rows, cut; - panning to the cursor as tmux does is the upgrade, if a small view must - follow a shell it never types into. + the emulator's) holds. A view shorter than the terminal is passed down + its top rows as far as keeps its cursor's row in sight, as tmux shows a + bigger window in a smaller client (`Pane.pan`, terminal.panFor: per view, + made in the step and again when a view is drawn out of turn, read where + the body's rows start, `Pane.scroll` and the body walk), an arrow in its + gutter's first row saying rows are above; a shell's prompt, at the + bottom, is never out of sight. ponytail: a panned view's mouse reports to + the terminal's program and its kitty images are placed as unpanned. - **Panes come and go under parked views.** A view keeps the serial of each slot when it goes out; unparked, the ids whose serial changed leave its layout and its keyboard moves to its first pane left. A pane focused @@ -390,8 +398,6 @@ the ponytail lens: named in a path yet (a script's focus or layout is that view's). - Every change draws every view (`needs_frame` is the session's); a view's own dirtiness waits for a measurement that wants it. - - Prompts are the pane's: a `.name` prompt drafted in one view is - closed by another view's step, where that pane has not the keyboard. - Dump, Restore and Handover keep one view, the live one. - **What proved wrong in the design.** "Views" in the design were a struct the core reads through; the swap made them cheaper and kept every @@ -433,21 +439,24 @@ Two views (not gated): every frame draws both, so a change costs two renders; a view switch copies ~60 fields and each pane's share, a few microseconds. -Gates, on the line rebased onto main's tty throughput fix (qzsouslm): +Gates, on the line rebased onto main's screen-state terminals (mnlwlpvk): -- `zig build replay -Dplatform=tty`: 112 snapshot scripts pass (the 110, - views.snap and views-term.snap), their 130 tapes replay in all three - replicas (390 runs, 29,355 digests); 20 monkey seeds at one view (60 - runs, 14,010 digests) and 20 at three views (60 runs, 9,720 digests); - no divergence, no panic. views.snap: 75 digests a replica, - views-term.snap 35. -- Unit tests: tty 1478 of 1498 pass, 20 skipped; GUI 1549 of 1560, 9 +- `zig build replay -Dplatform=tty`: 113 snapshot scripts pass (the 110, + views.snap, views-term.snap and views-pan.snap), their 131 tapes replay + in all three replicas (393 runs, 29,436 digests); 20 monkey seeds at one + view (60 runs, 14,004 digests) and 20 at three views (60 runs, 9,666 + digests); no divergence, no panic. views.snap: 75 digests a replica, + views-term.snap 35. core-test: 1117 of 1135, 18 skipped. +- Unit tests (on qzsouslm): tty 1478 of 1498 pass, 20 skipped; GUI 1549 of 1560, 9 skipped and 2 failed, the same two that fail on phase 1's line (`tagline pointer mapping retains physical grips...` and `a tag caret past its - row's end glides...`). The views' own: 7 (own cursor, scroll, mode, + row's end glides...`). The views' own: 8 (own cursor, scroll, mode, layout and focus; an edit and a register shared; a terminal sized by its last typist; views closing; the other views drawn leave the digest; a - view keeps its place in a text another edits). + view keeps its place in a text another edits; a short view keeps a tall + terminal's cursor in sight). views-pan.snap: a shell sized for the tall + view, its prompt and last lines in the short one under an arrow, then + typed into from the short one and sized for it. - test/fs.py passes (with a TMPDIR short enough for the 9P sockets' paths). test/daemon.py: 20 detach and attach cycles, then a second terminal on the attached session, both drawn, detached from inside and diff --git a/docs/typ/reference.typ b/docs/typ/reference.typ index 190adba7..ce2a45f1 100644 --- a/docs/typ/reference.typ +++ b/docs/typ/reference.typ @@ -1018,8 +1018,8 @@ to its end before the write is answered, and one that finds nothing fails, each terminal or window its own layout, focus, cursors, selections, modes, scroll and jumps, over the panes, texts, tags, terminals, messages and registers they share. A terminal pane shown in two takes the - size of the view that typed into it last; a smaller view shows its top - rows. #word("Detach"), or a terminal closing, lets that view go and + size of the view that typed into it last; a shorter view shows the + rows down to its cursor, an arrow in the gutter when rows are above. #word("Detach"), or a terminal closing, lets that view go and leaves the others. With none attached, `size C R` sets the screen, messages clear by the clock, and a pty reports its size in pixels at the last cell size (8×16 before any), diff --git a/src/draw.zig b/src/draw.zig index 4f9c6c87..0cc5c80d 100644 --- a/src/draw.zig +++ b/src/draw.zig @@ -958,6 +958,9 @@ fn renderPane(p: *Pardes, s: *Surface, arena: std.mem.Allocator, pane: *Pane, r: } try body_layer.joinBody(p, s, arena, pane, r, id, active, rail.*); + // A terminal panned to its cursor in a view shorter than it: an arrow + // in the gutter's first body row says rows are above (views.zig). + if (pane.pan > 0) s.set(r.x, p.bodyTop(pane, r), "↑", .{ .fg = .{ .rgb = chrome.box } }); // A query's places' ticks, each in the track's cell of its row: chrome, // painted with the rail (a pixel shell draws its own from the same list). for (s.rail_ticks[0..s.nrail_ticks]) |tick| { diff --git a/src/panes.zig b/src/panes.zig index 71a04464..89f24eb6 100644 --- a/src/panes.zig +++ b/src/panes.zig @@ -30,7 +30,7 @@ pub const Pane = struct { /// the pane's opens and readers; the other views' share of it (views.zig). pub const replica = .{ .local = .{ "pv", "pv_known" }, - .derived = .{ "wrap_line", "wrap_col", "wrap_pad", "wrap_n", "body_wrapped", "context_rows", "body_rows", "body_visible_rows", "context_row_limit", "geometry_key", "geometry_len", "view", "notices" }, + .derived = .{ "wrap_line", "wrap_col", "wrap_pad", "wrap_n", "body_wrapped", "context_rows", "body_rows", "body_visible_rows", "context_row_limit", "geometry_key", "geometry_len", "view", "notices", "pan" }, .authority = .{"fs"}, .len = .{ .{ "msg", "msg_len" }, .{ "msg_older", "msg_older_len" }, .{ "question", "question_len" }, .{ "shell_tail", "shell_tail_len" }, .{ "sel_snap", "nsel_snap" } }, }; @@ -309,6 +309,13 @@ pub const Pane = struct { /// The view that last typed into it: a terminal shown at two sizes takes /// that view's (Pardes.sync). sized_by: u8 = 0, + /// A terminal shown in a view shorter than it (one that does not size + /// it): the rows its top is passed over by, so its cursor stays in + /// sight (terminal.panFor); 0 everywhere else. + pan: u16 = 0, + /// The view whose step a `.name` prompt is (tagline.draftName): only + /// that view's focus leaving the pane drops it. + prompt_view: u8 = 0, pub const MessagePhase = enum { entering, shown, lingering, leaving }; /// One message line's animation: falling in, shown, lingering after the @@ -543,7 +550,7 @@ pub const Pane = struct { pub fn scroll(pane: *Pane) i32 { if (pane.file) |f| return @intCast(f.scroll); if (comptime Pdf.enabled) if (pane.pdf) |pv| return @intCast(pv.text_scroll); - return pane.surfRow(terminal.gridOffset(pane)); + return pane.surfRow(terminal.gridOffset(pane) + pane.pan); } pub fn wrapAt(pane: *Pane, vr: i32) struct { line: i32, at: i32, pad: i32 = 0 } { diff --git a/src/pardes.zig b/src/pardes.zig index 59233347..75e91ee9 100644 --- a/src/pardes.zig +++ b/src/pardes.zig @@ -7729,7 +7729,7 @@ pub const Pardes = struct { // hands it back to its body, and a name drafted there is dropped. if (id != p.active or p.header_focus) { pane.focus = .body; - if (pane.prompt == .name) pane.closePrompt(); + if (pane.prompt == .name and pane.prompt_view == p.view) pane.closePrompt(); } // A sweep over the tag is the tag's while it has the keyboard. const swept = pane.sel[sel_slot]; @@ -7822,6 +7822,7 @@ pub const Pardes = struct { pane.rows = rows; if (pane.file) |*file| file.syntax_dirty = true; } + pane.pan = panes.terminal.panFor(p, pane, r); // A directory pane's entries follow its width, as acme's // columns do (Dir.fit). if (pane.file) |file| if (file.listing != null and r.h > pane.tag_rows) panes.Dir.fit(p, pane, pane.cols); @@ -7856,10 +7857,10 @@ pub const Pardes = struct { /// Whether the live view sizes terminal `pane`: a terminal shown at two /// sizes takes the size of the view that typed into it last, and any - /// view's once that one is gone. The others show it at that size. - // ponytail: a view smaller than the terminal shows its top rows, the - // rest cut; panning to its cursor, as tmux does, when a small view must - // follow a busy shell it never types into. + /// view's once that one is gone. The others show it at that size, a + /// shorter one passed down to its cursor (`Pane.pan`, terminal.panFor). + // ponytail: a panned view's pointer reports to a terminal's program and + // its kitty images are placed as the unpanned rows are. fn sizes(p: *Pardes, pane: *Pane) bool { if (pane.sized_by == p.view) return true; if (views.isOpen(p, pane.sized_by)) return false; diff --git a/src/tagline.zig b/src/tagline.zig index 79c2c09b..f669e98b 100644 --- a/src/tagline.zig +++ b/src/tagline.zig @@ -361,6 +361,7 @@ pub fn draftName(p: *Pardes, pane: *Pane, col: usize) void { const name = p.gpa.dupe(u8, f.path) catch return; pane.closePrompt(); pane.prompt = .name; + pane.prompt_view = p.view; pane.input = .{ .what = .input, .own = name, .mode = .insert, .cur_pinned = true, .cur_col = @intCast(@min(name.len, panes.File.rawAtDisplay(name, col))) }; } diff --git a/src/terminal.zig b/src/terminal.zig index 0afbe43a..e737a831 100644 --- a/src/terminal.zig +++ b/src/terminal.zig @@ -395,8 +395,26 @@ pub fn visibleCursor(pane: *const Pane) ?GridCursor { const cur = state.vt.screens.active.cursor; const sb = scrollbar(pane); const row = sb.total - sb.len + cur.y; - if (row < sb.offset or row - sb.offset >= sb.len) return null; - return .{ .x = @intCast(cur.x), .y = @intCast(row - sb.offset) }; + if (row < sb.offset + pane.pan or row - sb.offset >= sb.len) return null; + return .{ .x = @intCast(cur.x), .y = @intCast(row - sb.offset - pane.pan) }; +} + +/// The rows terminal `pane`'s top is passed over by in a view whose rect +/// `r` is shorter than it, one that does not size it (Pardes.sizes): as +/// many as keep its cursor's row in sight, as tmux pans a bigger window in +/// a smaller client. 0 in the view that sizes it, and in one tall enough. +pub fn panFor(p: *const Pardes, pane: *const Pane, r: pardes.Rect) u16 { + if (comptime !enabled) return 0; + if (pane.sized_by == p.view or pane.collapsed or !pane.isTerminal()) return 0; + const shown = r.h -| pane.tag_rows; + if (shown == 0 or shown >= pane.rows) return 0; + const state = pane.terminal orelse return 0; + const sb = scrollbar(pane); + // The cursor's row in the viewport; past its end (scrolled back), its + // last row. + const row = sb.total - sb.len + state.vt.screens.active.cursor.y; + const at: usize = if (row < sb.offset) 0 else @min(row - sb.offset, pane.rows - 1); + return @intCast(@min(pane.rows - shown, (at + 1) -| shown)); } /// Move the emulator's viewport by `delta` shell rows (negative scrolls back). @@ -1886,7 +1904,7 @@ const BodyWalk = struct { return .{ .pane = pane, .goff = goff, - .g = if (pane.body.mode == .tty) goff else pane.gridRow(off), + .g = if (pane.body.mode == .tty) goff + pane.pan else pane.gridRow(off), .skip = if (pane.ovl) |o| @intCast(@max(0, off - pane.surfRow(o.row))) else 0, }; } diff --git a/src/views.zig b/src/views.zig index e3ba5b4e..c847b987 100644 --- a/src/views.zig +++ b/src/views.zig @@ -53,7 +53,7 @@ pub const fields = [_][:0]const u8{ /// A pane's fields that are each view's own: where it is in the view's /// layout, which of its texts has the keyboard, how far it is scrolled, and /// what the view's step measured of it (its tag's rows, its bands). -const pane_fields = [_][:0]const u8{ "focus", "vweight", "collapsed", "attached_to", "hscroll", "tag_scroll", "tag_top", "tag_rows", "view", "notices" }; +const pane_fields = [_][:0]const u8{ "focus", "vweight", "collapsed", "attached_to", "hscroll", "tag_scroll", "tag_top", "tag_rows", "view", "notices", "pan" }; /// The body's: its cursor, selections and mode. Its undo stays the pane's. const body_fields = [_][:0]const u8{ "mode", "cur_pinned", "cur_row", "cur_col", "msel", "vsel", "sels", "nsel", "select", "sticky_col", "restore_cursor", "last_edit", "reg_wait", "normal", "find_op", "find_ch" }; @@ -72,6 +72,7 @@ pub const PaneView = struct { tag_rows: u16 = 1, view: Pane.SessionView = .{}, notices: Pane.Notices = .{}, + pan: u16 = 0, body: Part(Text, &body_fields) = .{}, /// A document's scroll and size; a terminal's size is its pty's, which /// every view shows (sized by its last typist, `Pardes.sync`). @@ -216,6 +217,10 @@ fn unpark(p: *Pardes, k: u8) void { }; } layout.compute(p); + // Its terminals' cursors moved while it was out: each pan follows. + for (p.panes, 0..) |s, id| if (s) |pane| { + pane.pan = pardes.panes.terminal.panFor(p, pane, p.rects[id]); + }; } /// The pane with the keyboard is one the live view shows: focused where its @@ -285,6 +290,8 @@ pub fn close(p: *Pardes, k: u8) void { if (sels.len > 0) p.gpa.free(sels); pane.pv[k].body.sels = &.{}; pane.pv_known &= ~(@as(u8, 1) << @intCast(k)); + // A name it was drafting goes with it. + if (pane.prompt == .name and pane.prompt_view == k) pane.closePrompt(); }; p.views_open &= ~(@as(u8, 1) << @intCast(k)); } @@ -555,3 +562,43 @@ test "a view not live keeps its place in a text another view edits" { try testing.expectEqual(scrolled + 3, pane.file.?.scroll); try testing.expect(std.mem.startsWith(u8, pardes.panes.File.sourceLine(pane, 103), "line 101")); } + +test "a terminal shown in a view shorter than it keeps its cursor in sight" { + const gpa = testing.allocator; + const p = try Pardes.init(gpa, .{ .tty_only = true, .cols = 80, .rows = 20 }); + defer p.deinit(); + const id = p.active; + const term = p.panes[id].?; + const k = p.openView().?; + p.toView(k); + p.update(.{ .resize = .{ .cols = 100, .rows = 50 } }); + p.update(.{ .key = .{ .cp = Key.right } }); + const rows = term.rows; + try testing.expect(rows > 30); + var out: std.ArrayList(u8) = .empty; + defer out.deinit(gpa); + for (1..rows) |i| try out.print(gpa, "row-{d}\r\n", .{i}); + try out.appendSlice(gpa, "LAST$ "); + p.update(.{ .output = .{ .pane = @intCast(id), .bytes = out.items } }); + // Sized for the tall view, it fits there whole: nothing is passed over. + try testing.expectEqual(@as(u16, 0), term.pan); + p.toView(0); + p.sync(); + try testing.expectEqual(rows, term.rows); + try testing.expect(term.pan > 0); + var arena: std.heap.ArenaAllocator = .init(gpa); + defer arena.deinit(); + const s = try p.render(arena.allocator()); + const r = p.rects[id]; + // The cursor's row, the last, is on screen in the short view, inside + // the pane, and so is the line it is on. + const cursor = s.cursor.?; + try testing.expect(cursor.y >= r.y and cursor.y < r.y + r.h); + var row_text: std.ArrayList(u8) = .empty; + defer row_text.deinit(gpa); + for (0..s.cols) |x| try row_text.appendSlice(gpa, s.cells[@as(usize, cursor.y) * s.cols + x].grapheme()); + try testing.expect(std.mem.indexOf(u8, row_text.items, "LAST$") != null); + // Back in the tall view, its top is where it was. + p.toView(k); + try testing.expectEqual(@as(u16, 0), term.pan); +} diff --git a/test/snapshots/views-pan.golden b/test/snapshots/views-pan.golden new file mode 100644 index 00000000..db63d29c --- /dev/null +++ b/test/snapshots/views-pan.golden @@ -0,0 +1,52 @@ +== snap tall grid=80x24 cursor=4,16 +| Newcol Joincol Find Grep Help Changelog Tutor Dump Themes Config Debug Exit +| New Tty Find Grep Joincol Delcol +|$ /tmp/pardes-snap/views-pan/cwd Tty+bash Save Mode Filter Collapse Del +| 18 +| 19 +| 20 +| 21 +| 22 +| 23 +| 24 +| 25 +| 26 +| 27 +| 28 +| 29 +| 30 +| $ +| /tmp/pardes-snap/views-pan/cwd/+New Save Tty Collapse Del +| 1 +== snap short-pans grid=60x12 cursor=4,8 +| Newcol Joincol Find Grep Help Changelog Tutor Dump Themes +| New Tty Find Grep Joincol Delcol +|$ /tmp/pardes-snap/views-pan/cwd Tty+bash Save Mode Filter C +| ollapse Del +|↑ 27 +| 28 +| 29 +| 30 +| $ +| /tmp/pardes-snap/views-pan/cwd/+New Save Tty Collapse Del +| 1 +== snap short-typed grid=60x12 cursor=4,8 +|4: 29 +|5: 30 +|6: $ echo typed-in-short +|7: typed-in-short +== snap tall-sees grid=80x24 cursor=4,7 +|3: 29 +|4: 30 +|5: $ echo typed-in-short +|6: typed-in-short +|7: $ +|8: +|9: +|10: +|11: +|12: +|13: +|14: +|15: +|16: diff --git a/test/snapshots/views-pan.snap b/test/snapshots/views-pan.snap new file mode 100644 index 00000000..3ba1ac5b --- /dev/null +++ b/test/snapshots/views-pan.snap @@ -0,0 +1,27 @@ +# A terminal shown in a view shorter than it (src/views.zig): sized for the +# tall view that typed into it, the short one passes over its top rows so +# its cursor and prompt stay in sight, an arrow in the gutter saying rows +# are above. +config Verbose off +start 24 80 views=2 +wait 8000 $ +stable 700 20000 +view 2 +resize 12 60 +stable 500 10000 +view 1 +text seq 1 30 +key enter +wait 8000 30 +stable 500 10000 +snap tall +view 2 +stable 500 10000 +snap short-pans +text echo typed-in-short +key enter +stable 500 10000 +snap short-typed +view 1 +stable 500 10000 +snap tall-sees -- cgit v1.3 From 3d7215593cd0a042bd2cbe27e1dde98f63daeda2 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 16:55:47 -0300 Subject: docs/next-daemon.md: Phase 5 groundwork notes: the replica socket's decisions, the round trips locally, through the relay and to FreeBSD, and open points 3 and 5 decided (walker copy for rollback and for join, no fork) A ping through `ssh -L` to a session on the FreeBSD box costs its RTT (median 205 to 210 ms against a TCP connect's 206 to 208) and a hello two (411 to 476 ms: ssh confirms the channel before it forwards a byte); through the relay at 30, 80 and 207 ms they are the RTT plus 0.3 ms. A rollback copy is about 80 KB (typical) or 96 KB (big) once file texts and undo snapshots are pinned, 0.02 to 0.045 ms; a fork of 1 GB stalls the authority 13 to 20 ms here and 20 on FreeBSD, and every page it writes while the child lives costs ~1.3 us more, so a join takes the same walker copy and a worker streams it. The tty A/B against phase 1's tip is within its A/A runs. Still open marks points 3 and 5 decided. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 199 insertions(+), 2 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index e7ddcf5f..efe5214a 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -163,12 +163,17 @@ open. ordered stream with no per-read round trip; 9P stays for scripting and the relay] 3. Rollback by walker copy or by arena copy-on-write. [whichever - measurement of the mutable state's size decides] + measurement of the mutable state's size decides] Decided: the walker + copy, about 100 KB and 0.05 ms once file texts are pinned (Phase 5 + groundwork notes). 4. How the daemon starts: spawning itself or socket activation. [itself, by double fork; FreeBSD has no systemd. A user unit can be added if a terminal's scope ever kills it] 5. The join writer: a fork child, or a metadata copy serialised on a - worker. [fork first, measure the page-table copy] + worker. [fork first, measure the page-table copy] Decided: the copy on + a worker. A fork of 1 GB stalls the authority 13 to 20 ms, and + copy-on-write taxes it for as long as the join streams (Phase 5 + groundwork notes). ## Phase 3 notes @@ -941,3 +946,195 @@ against that tty, is the Tty throughput notes' keystroke-to-present. - A replica pushes a flood's rows into its history one at a time. - The integration into the stream and the cores (a replica pane's State fed by `apply` instead of `.output`) is the next piece. + +## Phase 5 groundwork notes + +Built 2026-10-29: the delay relay (`test/delay_relay.py`), the replica +socket (`src/daemon.zig`, `src/detached/wire.zig`), and two measurements +for open points 3 and 5 (`zig build rollback-bench`, which runs +`test/rollback_bench.zig`, and `test/fork_stall.py`). Decisions taken at +the forks, with the ponytail lens: + +- **The delay relay is a Python script.** Each direction is a reader that + stamps every chunk with a due time and a writer that sends it then. The + due time is the chunk's arrival plus the delay. With a rate cap, it is + the moment the link has carried what came before, plus the delay. So + throughput is not serialised behind the delay, and the bytes stay in + order. Either side is a Unix or TCP socket. ponytail: its buffer is + unbounded, which makes a link with an infinite bandwidth-delay product; + on a real link, ssh's window is the bound. +- **Two sockets, where the design had "the daemon's one".** The + fd-carrying SEQPACKET socket stays local. Once the daemon holds its + lock, it binds `pardes-replica.sock` (SOCK_STREAM) beside it, and + unlinks it at exit. `ssh -L` forwards that one. +- **The daemon judges the hello, so a session never sees one it + refuses.** + - Every session's registration carries its semantics hash (daemon + protocol 2), and the daemon compares the client's with the session's. + - It refuses in words: another protocol; another build, naming both + hashes and pointing at `pardes --version` to compare them; or a + session nobody has. + - It reads exactly the hello's length (the header, then the payload), + so whatever the client sends after it is left for the session. + - Up to 16 hellos may be on their way at once. ponytail: a client that + never sends one holds its slot until it hangs up. +- **The welcome comes from the session**, with its pid and name, over the + connection it was handed. That tells the client the handoff happened. + The test checks that the session's fd count went up and the daemon's + did not. +- **The semantics hash** (`wire.semantics`) is computed at comptime and + covers: + - the protocol, the release and the commit; + - the platform and the build options; + - every dependency the manifest pins, by its content hash (build.zig's + `deps` option). The design said grammars, but ghostty and the + Unicode tables also make a core step differently, so it takes all of + them; + - the walker codec's assumptions (pointer width, byte order). + + It leaves out the OS. The main test is a Linux client of a FreeBSD + session, and what differs between those is the host's, which the tape + carries. The two builds of one tree hash alike (1087257bf166588a on + both), and the box's session welcomed the client from here. ponytail: + an unstamped build (Debug with `-Dplatform`, so every test build) has + no commit, so two dev trees with one release and the same options hash + alike. +- **The hello lost its grid**: a replica's view takes a resize like any + other input. A ping (client tag 0x03) and three server tags (welcome, + refused, pong) are new. The wire is version 14. +- **The session's half is a stub**: a thread per replica that sends the + welcome, answers pings, and ends at bye or a hangup. + +Measured. Timings ran alone through the perf lock, pinned to cores 6-7 +while builds ran niced on the others. The local A/B ran interleaved, with +an A/A run beside it. + +- **The relay's self-check** added this much over a direct connection, + at each delay (median RTT, then throughput through it): + + | one-way delay | wanted | added | throughput | + |---|---|---|---| + | 15 ms | 30 ms | 30.26 ms | 3.2 GB/s | + | 40 ms | 80 ms | 80.27 ms | 2.3 GB/s | + | 103.5 ms | 207 ms | 207.26 ms | 2.3 GB/s | + + Direct, the same pair gave 9.9 GB/s. The extra 0.13 ms each way is + the threads' wake-ups. A 20 MB/s cap held at 19.98 MB/s. +- **A replica's round trips.** Each row is a connect and hello, then 50 + pings, to a detached session of this build. The session's thread + answers each ping. + + | path | hello | ping min / median / p90 | + |---|---|---| + | direct (Unix socket) | 0.26 ms | 0.01 / 0.01 / 0.01 ms | + | relay, 30 ms RTT | 30.95 ms | 30.26 / 30.34 / 30.40 ms | + | relay, 80 ms RTT | 80.73 ms | 80.25 / 80.31 / 80.39 ms | + | relay, 207 ms RTT | 207.62 ms | 207.22 / 207.35 / 207.41 ms | + | FreeBSD box, `ssh -L` | 411 to 476 ms | 203.2 / 204.9 to 210.4 / 210 to 221 ms | + + The box's rows cover six runs over two passes. In the same passes, a + TCP connect to the box's port 22 took 203.8 to 205.3 ms at best and + 206.4 to 208.3 ms at the median. The relay rows are the second pass; + the first pass had p90s up to 5 ms worse. So a ping through `ssh -L` + and the daemon's handoff costs the RTT and nothing more measurable. + The hello costs two RTTs: + ssh confirms the channel before it forwards a byte. A refused hello + over the link came back in 412 ms, also two RTTs. That is the floor + phase 5 builds on. A replica connects once, so it pays the second RTT + once. +- **Open point 3, rollback: the walker copy.** `zig build + rollback-bench -Doptimize=ReleaseFast` builds two sessions in process: + - typical: a shell with 1 MB of scrollback, three 60 KB files and a + 3 MB one, 40 edits; + - big: 14 panes, an 18 MB file, four 3 MB files and eight small ones, + 8 MB of scrollback, 200 edits. + + For each, it measures the state the digest walk visits (what + speculation can write) and the share of it that is blobs. Blobs are a + file's text and its undo snapshots. `File.setContentSpan` puts a new + buffer in place of the old one at every edit, and a snapshot is a + copy, so both are immutable once made and a copy can pin them. The + walk is then timed again with them taken out. + + | session | walked | of it blobs | left to copy | walk | walk, blobs out | copy (walk + memcpy) | + |---|---|---|---|---|---|---| + | typical | 36.3 MB | 36.2 MB | 80 KB | 3.1 to 3.8 ms | 0.019 to 0.023 ms | 0.020 to 0.024 ms | + | big | 282.5 MB | 282.4 MB | 96 KB | 23 to 27 ms | 0.043 to 0.044 ms | 0.044 to 0.045 ms | + + A copy is about 0.1 to 0.3% of a 16.7 ms frame, so the walker wins: + arena copy-on-write would buy nothing. Three conditions come with it: + - a blob a copy holds must not be freed until the copy is let go, + whether by a refcount or by a free list drained at confirmation; + - caches derived from a blob (`line_starts`, highlights) are also made + anew at each edit, so they can be pinned with it, or remade on + demand, since the geometry is keyed by revision; + - a terminal's history is not in the walk, because under screen-state + terminals it is the authority's pinned rows. + + The walk with the blobs out is the stand-in for a copy walk: phase 2's + dump walk is the real one, and this step re-measures it when it lands. +- **Open point 5, the join writer: not a fork.** The authority's stall, + as median fork time, with 64 MB written by the authority while the + child lives: + + | written memory | Linux 7.2 (THP always) | FreeBSD 15.1 | + |---|---|---| + | 100 MB | 1.8 to 2.7 ms | 4.1 ms | + | 600 MB | 8.6 to 10.8 ms | 14.0 ms | + | 1 GB | 13.3 to 19.6 ms (max 24) | 20.4 ms (max 26) | + | 4 GB (one Linux pass) | 62 ms (4 KiB pages: 162) | 52 ms | + | then 64 MB written under the child | 26 to 32 ms | 19 to 27 ms | + + Linux is two passes on the pinned cores, beside the builds; the box was + idle. The big session forked in process at 431 MB RSS took 3.4 and + 8.3 ms. The user's windows hold 450 and 630 MB. + + The gate, "a join costs the authority under one frame", fails twice: + - fork itself, at 1 GB on both systems (and at 600 MB once a frame is + 8.3 ms, at 120 Hz); + - copy-on-write after it. Every page the authority writes while the + child streams costs a fault, about 1.3 µs per 4 KiB page. The child + lives as long as the dump takes to cross, which is seconds at + 207 ms. + + So the join is the same walker copy as rollback: tens of microseconds + at a step boundary, blobs pinned. A worker thread then serialises the + copy and streams the pinned blobs, which are immutable and so need no + lock, while the authority buffers the events after it for the joiner. + This is the coordinator's alternative, "a metadata copy serialised on + a worker". The authority pays the copy, and the blobs stay pinned + while they stream. + +- **No local change.** tty startup and keys, the phase 1 tip (a4b85cb7, + ReleaseFast) against this, both live at once with their samples + alternating; the p50 ratio new/base: + + | | three A/B runs | two A/A runs | + |---|---|---| + | startup | 1.040, 0.965, 1.024 | 1.053, 0.978 | + | key in a file, to its frame | 1.084, 1.025, 1.027 | 1.025, 1.013 | + | key echoed by bash, to its frame | 0.996, 0.972, 1.010 | 1.015, 0.998 | + + The daemon and the hello are off the hot path, and every ratio is + within the spread of the A/A runs beside it, whose second slot reads 1 + to 5% slow against the same binary. So there is no measurable change. + The absolute times are not comparable with phase 3's 4.73 ms: the + startup p50 was 6.5 to 26 ms here, on two cores shared by both live + sessions and the driver, with builds running on the others. + +What proved wrong in the design: +- "`ssh -L` of the daemon's one socket": there are two. The one that + carries fds cannot cross ssh, and no fd would cross anyway. +- "The authority forks a writer child": fork's page-table copy is over a + frame at 1 GB on both systems, and the copy-on-write tax lasts as long + as the join streams. Once blobs are pinned, a copy of what speculation + can write is about 100 KB, small enough for a join to take one too. +- "When the first speculative event goes pending, the replica copies + views, buffers, ...": a buffer's text never needs copying, since every + edit replaces it. Only its handle and the views are copied. +- "The hello carries a semantics hash (version, features, grammars)": it + hashes every dependency, and leaves the OS out on purpose. +- Pre-existing on the phase 1 tip, and not this line's: test/fs.py's + `mounted-startup` session hangs at startup (its main thread waits on a + futex), the same with the base binary. A ReleaseFast `exit_answers` + run loses its Rclunk race about 1 time in 7 on base, and 1 in 40 here. -- cgit v1.3 From a8d9eb9e5051521aaeb1a95584729794059b5d83 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 18:19:13 -0300 Subject: docs/next-daemon.md: Phase 5 part A notes: the copy put back where it was, speculation checked against the authority's tape, each client in its own view, what release builds read unwritten, and the numbers Three clients typing 100 inputs a seed at 30, 80 and 207 ms RTT, a view each: no divergence and every replica on the authority's digest at quiescence (Debug 12 seeds, ReleaseFast with a filled heap 20, ReleaseFast timed 20, and 20 in one shared view). ReleaseFast, p50/p99: a speculative apply 35-54/168-183 us, a copy (1.3 MB) 75-94/198-447 us, a rollback 342-569/818-1561 us, the worst a tenth of a frame; rollbacks per input 31%, 76% and 129%, nearly all for another client's unit first, which views do not change (they change what a rollback shows); 2-7% mispredictions, every one a message's HH:MM:SS read in another second. rollback-bench: a typical session's copy 980 KB in 0.044 ms, put back in 0.036; a big one's 1.49 MB in 0.071, 0.051. What proved wrong (a 100 KB copy, rare mispredictions), what is not done (transport, the view's events at once, echo prediction, a single-threaded epoch), and fs.py's mounted-startup hang still on main, shown with main's own binary. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 176 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 2 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index efe5214a..d4f9130e 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1136,5 +1136,177 @@ What proved wrong in the design: hashes every dependency, and leaves the OS out on purpose. - Pre-existing on the phase 1 tip, and not this line's: test/fs.py's `mounted-startup` session hangs at startup (its main thread waits on a - futex), the same with the base binary. A ReleaseFast `exit_answers` - run loses its Rclunk race about 1 time in 7 on base, and 1 in 40 here. + futex), the same with the base binary, and still on main after phase 4 + (Phase 5 part A notes). A ReleaseFast `exit_answers` run loses its + Rclunk race about 1 time in 7 on base, and 1 in 40 here. + +## Phase 5 part A notes + +Built 2026-10-29, in one process and with no transport: the rollback copy +(`src/rollback.zig`), the replica engine (`src/replica.zig`), the tape's +speculation mode (`src/tape.zig`), and an in-process harness +(`test/replicas.zig`, `zig build replicas`). `zig build rollback-bench` +now times the real copy. Decisions taken at the forks, with the ponytail +lens: + +- **The copy saves bytes and puts them back where they were.** Its walk + is the digest's (src/walk.zig): views, buffers, registers, counters and + terminal metadata. It saves the bytes of every allocation the walk + reaches, and puts each back over its allocation. That works because the + replica core's allocator (`rollback.Epoch`) changes three things while + a copy is held: + - a free of anything older than the copy is put off; + - growing such an allocation in place is refused, so the core moves it; + - anything allocated meanwhile belongs to the epoch. + + So every pointer in the saved bytes still points at what it pointed at + when the copy was taken. Putting the copy back frees the epoch's + allocations and forgets the put-off frees. Letting the copy go makes + them. Nothing has to know who owns what. +- **Three blobs are pinned instead of copied** (4 KiB and up): a file's + text, its line index and its undo snapshots. An edit puts a new one in + place of the old (File.setContentSpan) and never writes one, so the + copy keeps a reference and the put-off free keeps it alive. A safe + build checks every pin is unchanged when the copy goes. +- **More than the digest's fields are saved.** Within a struct the copy + saves every field: the derived caches, and this process's own state too. + It skips the queues a step drains (asserted empty) and the frame + statistics, and cuts an array at its `len`. Both were measured wrong + first: + - a safe build's `free` writes 0xAA over what it frees, so a cache the + speculation freed came back as that (a line index full of 0xAA, then a + crash); + - this process's own fields (the frame arena, the surface) point at what + the epoch then frees. + + For the same reason, file texts and line indexes are now freed raw + everywhere (`memory.freeRaw`), as setContentSpan already did. +- **A speculation is checked against the authority's tape of the same + input.** The replica applies its own input as the authority will (an + update, then a pump) and records it with its own tape. If the + authority's unit for that input has the same records (inputs, host calls + and their answers, digests aside), the head goes: that is the fast path. + The digests are left out of the comparison because the starting states + are equal by induction. A digest checks every unit a replica replays. +- **Each client has its own view** (src/views.zig). The authority goes + into the client's view before its input (`toView`, which is taped), and + the replica's speculation does the same. So a unit records the switch + only when another view was live, alike on both sides. The views not + live (`parked`, each pane's `pv`) are this process's own fields, and + the copy saves them with the rest. +- **A host call or a syscall stops a speculation, but cannot stop it + midway.** Zig has no unwinding. The call is answered with a harmless + value (an error of its set, null, zeroes, or a union's first zeroable + case), the input runs to its end, and then it is undone: the copy is + put back and the earlier pending inputs applied again. That input, and + those after it, wait for the authority's answer. A read of the clock or + the environment (`tape.value`) is a guess made locally, which the + comparison checks. `watched` is answered false, as the authority's host + says too: a replica draws outside the step. +- **An emulator is the authority's.** Its writers in terminal.zig (feed, + resize, scroll, follow, create, deinit) stop a speculation before they + write. A safe build asserts that a clean speculation left every + terminal's digest as it was. Tree-sitter's trees live on the C heap, + which the epoch cannot see. While a copy is held, `syntax.held_trees` + puts off their frees and frees the epoch's own. +- **The view's events wait** (a resize, the pointer): phase 4. +- **Messages are bytes.** A client's input is a u16 client id, a u32 + sequence number, and the event as detached/wire.zig frames it. A unit is + the input's origin (0 for the authority's own: a pty's output, a setup), + its sequence number, and the authority's tape of it. +- **The harness** is a discrete-event simulation. One authority and three + replicas share one view, each client typing monkey.zig's mix (keys, + typing, pastes, a few commands, resizes and clicks) at 50-200 ms with + pauses. Each link takes half the RTT plus up to 10% of it in jitter, in + order. The session is the tty's own boot (a raw shell over a text pane) + with three files looked at. The authority's pty echoes what it is + written 2 ms later, which makes units of its own. + +Found on the way: phase 1's replay ran Debug, whose 0xAA fill made every +read of unwritten memory alike in two cores. In a release build they read +what the heap had, and replicas diverged at once, from: +- `col_panes`, digested whole and undefined past each column's panes (now + zeroed); +- `restore_buf`, the same (now zeroed); +- `terminal.State.reply_dropped`, never set, since `create` leaves fields + as the heap had them; +- `Text.sels` and `PendingPipe.ranges`, digested past their length (a + `len` now cuts a slice too). + +`PARDES_REPLICA_FILL=0xff` fills a replica's allocations, so that what is +left shows at once. Any replica is a release build, so these mattered. + +Measured on main with views (lyntonns). The ReleaseFast runs were alone +through the perf lock; the correctness sweeps ran beside builds. + +- **Correctness.** Three clients each typed 100 inputs per seed, at RTTs + of 30, 80 and 207 ms. No replica diverged, and every replica ended on + the authority's digest, in all three runs: + - Debug: 12 seeds, a view each; + - ReleaseFast with `PARDES_REPLICA_FILL=0xff`: 20 seeds, a view each; + - ReleaseFast, timed: 20 seeds, a view each and in one shared view. +- **What happened to the inputs, and what it cost** (ReleaseFast, a view + each; times in µs, p50 / p99): + + | RTT | rollbacks per input (mispredicted + another's) | speculative apply | copy (1.3 MB) | rollback | + |---|---|---|---|---| + | 30 ms | 31% (106 + 1745) | 35 / 183 | 75 / 198 | 342 / 818 | + | 80 ms | 76% (257 + 4285) | 54 / 168 | 94 / 447 | 455 / 1318 | + | 207 ms | 129% (423 + 7335) | 50 / 174 | 91 / 373 | 569 / 1561 | + + - Of 6000 inputs a run, 259 to 348 stopped at a host call and waited, + and 434 to 449 were the view's own (resizes and clicks). + - In one shared view, the copy is half the size (0.66 MB), the costs + are half (copy 37 / 152, rollback 184 to 259 at p50), and the + rollbacks are the same: 30%, 70% and 121%. + - The worst rollback, 1.6 ms, is a tenth of a 16.7 ms frame. A + speculative apply is 0.2 ms at worst. +- **The copy alone** (`rollback-bench`, ReleaseFast, single view): + + | session | saved | pinned | take | put back after an edit | + |---|---|---|---|---| + | typical (5 panes) | 980 KB | 36.6 MB | 0.044 ms | 0.036 ms | + | big (14 panes) | 1.49 MB | 286.5 MB | 0.071 ms | 0.051 ms | + + Letting a copy go costs nothing measurable. + +What proved wrong: +- "Views cut the conflicts": they cut what a rollback shows, not how + often one happens. Any unit of another client's, in any view, comes + before this replica's pending inputs in the authority's order, so the + copy goes back and the pending inputs are applied again. Each costs a + third of a millisecond, a tenth of a frame at worst. Telling which + units commute with which pending inputs is the next lever, not a + needed one. +- "A copy of about 100 KB": it is 1 to 1.5 MB. The core struct itself is + most of it (the presentation, the surface, the settings and the column + tags), plus the views' parked state, now that the copy saves all of a + struct and not only what a digest takes. It is still under 0.1 ms. +- "A misprediction is rare": every key that leaves a message stamps it + with the time of day (`HH:MM:SS`). So a key confirmed in another second + than it was applied is recorded differently: 2 to 7% of inputs here. In + the harness both cores share one clock, so this is the floor. Across + machines an offset between the clocks raises it. Part B should guess + the authority's clock (its last confirmed reading, plus the time since + locally), not read its own. +- views.zig left a pane's slot for the live view unwritten, and a slot + holding the live body's selections. A pane closed before that view + parked again freed whatever its slot held. This is fixed in its own + change, before this one. Every replica switches views on every input, + and found it at once. + +Not done, and why: +- No transport: part B. +- The view's events (a resize, a click) still wait for the authority. + With views they could be applied at once. +- A pty write stops a speculation, so a shell's echo is shown late. + Phase 6 predicts the echo. +- The authority's own units (a shell's echo) roll back a replica that has + pending inputs. +- The epoch allocator is single-threaded. A replica core's workers + (images, search) must allocate elsewhere, or hold a lock, in part B. +- A replica keeps a timing sample for every input it applies (the + harness reads them). A long-lived one keeps only counters. +- test/fs.py's `mounted-startup` still hangs on main (lyntonns). The same + hang shows with main's own binary in a workspace of its own, so it is + not this line's. -- cgit v1.3 From b4c7b2ff828d258dde78ff2dcded3b226e6d874d Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 18:31:11 -0300 Subject: docs/next-daemon.md: phase 2 notes (the walker dump, crossing builds, load and revive, what the design missed); the GUI's Restore test keeps the dump's serial Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 121 ++++++++++++++++++++++++++++++++++++++++++++++++++++ src/gui/gui.zig | 3 +- src/walk.zig | 2 - 3 files changed, 123 insertions(+), 3 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index d4f9130e..7da2804f 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1310,3 +1310,124 @@ Not done, and why: - test/fs.py's `mounted-startup` still hangs on main (lyntonns). The same hang shows with main's own binary in a workspace of its own, so it is not this line's. + +## Phase 2 notes + +Built 2026-10-29 (`src/walk.zig` Dump/Load, `src/dump.zig`, terminal.zig's +dump hooks; `pardes-snap --dump`, `pardes-monkey --dump`, and both in +`zig build replay`). Decisions taken at the forks, with the ponytail lens: + +- **The dump is the digest's preimage.** It walks exactly what the digest + walks (local, derived and authority fields stay out), so `digest(load( + dump(x))) == digest(x)` is the definition of complete, and the gate. The + dump is `magic`, a table of every type the roots' walk reaches (each + struct's and union's fields by name with their types' places, each + enum's names and values, each number's width), then the values as the + tape lays them out but for an enum or union tag by its place in the + table, and a pointer or slice met again written as a reference to the + first. The table is a comptime graph of the roots: a session's is 324 + types in 23 KB, of a fresh session's 96 KB dump. +- **Crossing builds: by name, and positional when it need not be.** A + table byte-identical to this build's is read field after field, no name + asked (the join's same-build path: no separate strict format). Any other + is matched by name: a field added takes its default, one removed is + passed over, an enum value by its name (Builtin's ordinals shift with + every builtin added), numbers widened or narrowed by value, arrays as + far as they fit, one that came to hold a `len` read either way. A field + whose kind changed takes its default and is said; with no default the + value it is in is refused, and an element of an array of optional + pointers (the session's panes) is then left out with its place in the + layout (`dropMissing`), never more. What was let go is the session's + first message after the load. A pane whose terminal another build could + not read is a fresh terminal (`ensureKind`). +- **Roots, not wrappers.** A session's dump is three roots: the core, + read *into* a core `Pardes.create` made (its local, derived and + authority fields stay create's: no defaults to invent for an arena or a + host), its 9P mounts (state the user made, inside the 9P server's + authority `fs`), and which file panes' files were on disk when it was + made (one gone by the load is said deleted, so Del asks: its text is + nowhere else; one never on disk comes back as it was). A pane's dump + (the store's, a handover's) is a Pane alone. +- **Aliasing by reference.** Zerox twins share their text and undo again, + an `inherited` cwd points at its pane again. Keyed by address, length + and type, which is also what made twins dump their text once. +- **Terminals are ghostty 1.4's snapshot** (`terminal.State.dumpWrite`): + both screens, history, styles, links, and the kitty images with their + pixels and placements (the known gap in test/kitty.zig closed), plus a + walk dump of pardes's own parts (unsent replies, the view, the OSC 133 + command cycle, its output pins as screen points). The snapshot is at + the ground state: a sequence the program had begun at the dump's moment + is lost (ponytail: the stream would have to track its continuation, a + cost on every pty read). ghostty calls its format a work in progress: + a ghostty bump that changes it makes a terminal another build cannot + read, which degrades as above. +- **Clean files' text is in the dump.** A join cannot read the + authority's disk, and the session is what it holds. A file changed on + disk since comes back as the watch a load emits finds it (as on open: a + clean pane reloads, an unsaved one is marked changed), not by a read in + the load. The store keeps a closed pane's record under 8 MiB by a slim + form (no terminal history, no undo, a clean file's text read again on + reopen), as it kept terminals without history before. +- **Load and revive are two halves.** `load` makes the core and nothing + else (what the gate checks, and what a replica takes); `revivePane` and + `reviveSession` give it to a host: a new shell under each terminal's + history (alternate screen left, keys, mouse and paste modes reset, the + banner; the old shell's REPL, Tty+ word and sent command dropped; a + command pane finished), watches, PDFs opened again (MuPDF's document is + derived: `Document.handle` is optional now), Recent and the store, the + ThemeFile asked for again, the startup config. The screen stays the + dump's until the host resizes: setting this process's `opts` size first + reflowed twice and lost where a terminal's view was. +- **`opts` is local** (how this process was started: allocators, socket + names, store path), and so are `dump_out` and the effect and input + queues' counters (the queues were local, their heads and lengths were + walked: a loaded core read a garbage effect). Pipe's runs, an Edit's, + Handover's dump in flight and its carrier, and the host's fallback file + table are authority: host errands, not state. +- **Restore keeps serials.** Renumbering would rewrite every serial the + session holds (attachments, query owners, places, messages, LSP and + callstack requests); Restore hangs up every connection anyway. The next + serials are past the old session's; `restored`/`restoredcol` records + are gone. A Restore over an older session keeps the host's facts (cell + pixels, row metrics, native images) and the higher of the request + sequences. +- **A failed load leaves nothing**: the loader records each allocation it + made and frees them all (`rollback`), and a `dumpFree` for a custom + type's insides. ghostty's snapshot decode itself leaks when an + allocation inside it fails: the OOM Restore test runs without a + terminal. +- **Handover** dumps `Carried` (the kind, a column's tag, each pane's + dump, weights, owners) behind `pardes-carried\n`; a new window's `-l` + of it is an empty core that adopts the panes, its settings this + process's (the old carried dump brought the source window's theme and + settings along). Panes carried now keep their undo. +- **No v1 reader.** A ZON dump is refused before anything is replaced + (`not a pardes dump: it does not begin as one`), and a ZON record in + the store does not reopen. Dumps are `.dump` files now. + +What proved wrong or missing in the design: +- The walker's list rule (exactly `items` and `capacity`) never matched in + a safe build: Zig 0.17's ArrayList has a `pointer_stability` lock there, + so phase 1's digest hashed every list's capacity and a dump wrote it. + Both take a list's items now. +- "pty masters passed as fds where Handover passes them today": Handover + passes none (it refuses terminals: their shells are the window's + children), and phase 3 made Detach happen in place, so no dump carries + a master. A loaded terminal gets a new shell; the daemon's Adopt is the + way back to the old ones. +- There are no crash dumps: crash.zig records the panic message only. + None was added: a panic handler would walk state that may be what broke + and allocate inside the handler. The path that exists is Dump, rebuild, + Restore, which the by-name load is for (a session read by name is a + unit test over the whole type graph). +- Two behaviours of the old dump were its approximations, and goldens + moved to the real session: a terminal's view comes back where it was + (load.snap's old view was not the dumped one), its prompt rows stay + hidden in text mode (OSC 133 marks are in the snapshot), its Filter, a + pane's message and typed text over its history are kept, and a query + pane comes back a live query, not plain text. +- Not done: the web shell (`-Ddump=` embeds a dump) was not built; a + native dump should load there by name. A dump is read whole, not + decoded as it arrives (the join can stream it; decoding as it arrives + needs a reader over a stream, not a slice). Native byte order: every + host pardes runs on is little-endian. diff --git a/src/gui/gui.zig b/src/gui/gui.zig index a342cfe2..c550aa4c 100644 --- a/src/gui/gui.zig +++ b/src/gui/gui.zig @@ -1109,7 +1109,8 @@ test "GUI PTY Restore joins a real reader waiting for queue space" { core.deinit(); core = replacement; shell.core = replacement; - try std.testing.expect(core.panes[0].?.serial != old_serial); + // The dump's pane, its serial too; its old shell stopped all the same. + try std.testing.expectEqual(old_serial, core.panes[0].?.serial); try std.testing.expectEqual(@as(usize, 0), queue.waiting); try std.testing.expect(queue.readers[0] == null); try std.testing.expectEqual(@as(c_int, -1), libc.fcntl(child.file.handle, libc.F.GETFD)); diff --git a/src/walk.zig b/src/walk.zig index 304fc5aa..00868133 100644 --- a/src/walk.zig +++ b/src/walk.zig @@ -659,7 +659,6 @@ pub fn Dump(comptime roots: []const type) type { out: std.Io.Writer.Allocating, memo: std.AutoHashMapUnmanaged(Key, u32) = .empty, shared: u32 = 0, - next_root: usize = 0, const Key = struct { addr: usize, len: usize, ty: u32 }; @@ -686,7 +685,6 @@ pub fn Dump(comptime roots: []const type) type { pub fn put(d: *Self, comptime T: type, v: *const T) !void { try writeU32(&d.out.writer, comptime indexIn(types, Canon(T))); try d.value(T, v); - d.next_root += 1; } /// Whether the pointer or slice at `addr` was met before: its -- cgit v1.3 From ad1a8f4bc4c363e7c6e7a89e14fdab62cb914a82 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 18:54:12 -0300 Subject: docs/next-daemon.md: phase 2's gates and measurements Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 7da2804f..fc9bdf64 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1431,3 +1431,40 @@ What proved wrong or missing in the design: decoded as it arrives (the join can stream it; decoding as it arrives needs a reader over a stream, not a slice). Native byte order: every host pardes runs on is little-endian. + +The gates on the rebased line (on main vvmuwntp). `zig build replay +-Dreplay-seeds=50`, now with `--dump`: the 110 scripts (129 sessions, +32,373 replica digests) and 50 monkey seeds of 70 steps (38,022 digests), +no divergence, and every session's dump at its end loaded to its digest +(all 179, field by field) and dumped again byte for byte (13.9 MB and 5.4 +MB of dumps). The 110 snapshots pass with five goldens moved (above). +unit-test tty passes; unit-test gui fails only phase 1's two known tests; +test/fs.py and test/daemon.py (20 cycles) pass. The cross-build check is +walk.zig's test of two hand-written versions of a struct (a field added, +one removed, one renamed, enums renumbered, a width changed, a pane whose +required field changed kind left out), both ways, and dump.zig's of a real +session read by name. + +Measured (ReleaseFast, the base binaries from before the rework against +this line, interleaved A B A ... A on the perf cores, medians, the base's +own spread as the A/A): the perf harness's core cells 0.91-1.01 of base +(p90 0.95-1.08) at an A/A of 5-109%. A session of 16 small files, a +100 MB file and a terminal with `seq 1 3000000` of scrollback, through +9P in a real pty: + +| | base | new | A/A | +|---|---|---|---| +| Dump, the big file clean | 9.5 ms, 0.2 MB | 83 ms, 105 MB | 19% | +| Restore of it | 71 ms | 98 ms | 10% | +| Dump, the big file unsaved | 507 ms, 140 MB | 185 ms, 210 MB | 8% | +| Restore of it | 1841 ms | 316 ms | 2% | +| a closed pane kept: the terminal | 122 ms | 128 ms | 49% | +| the 100 MB file | 438 ms | 84 ms | 4% | +| a small file | 10.5 ms | 18 ms | 23% | + +A clean file's text is in the dump now (what makes it complete), and an +unsaved one's undo (the 100 MB file's one edit keeps the text before it); +binary, not base64 in ZON, so the rest is faster. A small pane's close +pays its type table (a pane's dump is self-describing) and the generic +walk: 8 ms. The live keystroke cells were not run (perf rule: `--live` +only for a change to terminal output); a dump is off the hot path. -- cgit v1.3 From ad0d3f61f310ca0cc3a1ab2c36edcafb5c44b97a Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 19:36:11 -0300 Subject: docs/next-daemon.md: phase 2's upgrade (what a user loses: ZON dumps; the store checked on a copy of a real one), the web shell, and the closes measured again Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 85 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 66 insertions(+), 19 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index fc9bdf64..95713b42 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1402,8 +1402,9 @@ dump hooks; `pardes-snap --dump`, `pardes-monkey --dump`, and both in process's (the old carried dump brought the source window's theme and settings along). Panes carried now keep their undo. - **No v1 reader.** A ZON dump is refused before anything is replaced - (`not a pardes dump: it does not begin as one`), and a ZON record in - the store does not reopen. Dumps are `.dump` files now. + (`not a pardes dump: it does not begin as one`); a ZON record in the + store is an old build's session's, which no new one lists (below). + Dumps are `.dump` files now. What proved wrong or missing in the design: - The walker's list rule (exactly `items` and `capacity`) never matched in @@ -1426,11 +1427,52 @@ What proved wrong or missing in the design: hidden in text mode (OSC 133 marks are in the snapshot), its Filter, a pane's message and typed text over its history are kept, and a query pane comes back a live query, not plain text. -- Not done: the web shell (`-Ddump=` embeds a dump) was not built; a - native dump should load there by name. A dump is read whole, not - decoded as it arrives (the join can stream it; decoding as it arrives - needs a reader over a stream, not a slice). Native byte order: every - host pardes runs on is little-endian. +- The web shell had not built on main for a while (u64 counts on a + 32-bit target, threads, stb_image and the tape's host calls). It builds + again and loads a native build's dump by name: `-Ddump=` embeds + test/web-snapshots/touch.dump, made by tools/web_fixture.py, in place + of the hand-written ZON fixture. A field this build compiled out (void + on wasm) is passed over, and the theme goes by its name, a fourth + session root, since the theme table differs between builds. web-snap's + touch check passes; its drift was the test's (13 tag-layer fields, + helix's retuned colours, and a pane resized by its grip, there being no + rule between panes to drag). The web shell's own `chromeTarget` still + calls an upper pane's last row a handle, which the mouse code no longer + resizes on. +- Not done: the website's replay (pardes-website/replays/00) still runs + its July wasm; the dump it was built from is gone. `node --test + test/web_driver_test.mjs` fails "CDP closes a stalled connection after + its deadline" every time, a test of the harness's CDP client, not of + pardes. A dump is read whole, not decoded as it arrives (the join can + stream it; decoding as it arrives needs a reader over a stream, not a + slice). Native byte order: every host pardes runs on is little-endian. + +Upgrading from a build before phase 2, a user loses: +- A ZON dump: Restore and `-l` refuse it, the session they are in + unchanged. A pane carried between an old window and a new one stays + where it was (a failed handover). +- Nothing in the store. Its framing and records are as they were; only a + closed pane's dump inside its record changed. A closed pane is listed + only by the session that closed it (a session's id is new each + process), so a new build never lists, opens or reports an old build's + records, and an old window still running brings back its own. Both + builds' compactions keep the records of every running session and drop + those of sessions that ended, as before. + +Checked on a copy of the user's real store (14.4 MB: Recent's 234 paths, +140 ZON records of 8 sessions, three of them running), the copy under the +workspace's scratch, the real one never opened: a new build starts on it +with no error or message, /recent lists the same 200 entries as the base +build's on the same copy, Panes shows none of the old records, a pane +closed and brought back comes back. A compaction forced by a record cut +short (as a writer that died leaves one) gives the same file in both +builds but for each one's own records: the running sessions' 72 records +byte for byte, Recent folded to the same 200, the ended sessions' 68 +dropped. The base build then runs on a store the new one wrote, with its +walker records in it, without an error. One cost moved: a closed small +pane's record is 14.5 KB, not 1.5 KB, most of it the type table each +self-describing dump carries (ponytail: bounded by 64 a session and the +compaction; a table once per store generation if it matters). The gates on the rebased line (on main vvmuwntp). `zig build replay -Dreplay-seeds=50`, now with `--dump`: the 110 scripts (129 sessions, @@ -1447,24 +1489,29 @@ session read by name. Measured (ReleaseFast, the base binaries from before the rework against this line, interleaved A B A ... A on the perf cores, medians, the base's -own spread as the A/A): the perf harness's core cells 0.91-1.01 of base -(p90 0.95-1.08) at an A/A of 5-109%. A session of 16 small files, a +own spread as the A/A): the perf harness's core cells 0.92-0.99 of base +(p90 0.80-1.03) at an A/A of 15-70%. A session of 16 small files, a 100 MB file and a terminal with `seq 1 3000000` of scrollback, through 9P in a real pty: | | base | new | A/A | |---|---|---|---| -| Dump, the big file clean | 9.5 ms, 0.2 MB | 83 ms, 105 MB | 19% | -| Restore of it | 71 ms | 98 ms | 10% | -| Dump, the big file unsaved | 507 ms, 140 MB | 185 ms, 210 MB | 8% | -| Restore of it | 1841 ms | 316 ms | 2% | -| a closed pane kept: the terminal | 122 ms | 128 ms | 49% | -| the 100 MB file | 438 ms | 84 ms | 4% | -| a small file | 10.5 ms | 18 ms | 23% | +| Dump, the big file clean | 8.8 ms, 0.2 MB | 83 ms, 105 MB | 21% | +| Restore of it | 69 ms | 105 ms | 4% | +| Dump, the big file unsaved | 514 ms, 140 MB | 191 ms, 210 MB | 1% | +| Restore of it | 1845 ms | 310 ms | 0% | +| a closed pane kept: the terminal | 131 ms | 143 ms | 4% | +| the 100 MB file | 433 ms | 127 ms | 3% | +| a small file, the step after that | 9.6 ms | 17 ms | 44% | +| a small file, alone | 1.0 ms | 1.0 ms | 7% | A clean file's text is in the dump now (what makes it complete), and an unsaved one's undo (the 100 MB file's one edit keeps the text before it); binary, not base64 in ZON, so the rest is faster. A small pane's close -pays its type table (a pane's dump is self-describing) and the generic -walk: 8 ms. The live keystroke cells were not run (perf rule: `--live` -only for a change to terminal output); a dump is off the hot path. +costs what it did: the 17 ms is the 100 MB pane's teardown (panes are +freed two syncs after they close, its undo's text with it) landing in +the next close's time; three small closes after two others take 1 ms in +both builds. The type table is made once a process (it was made at +every store record: 123 µs to 33 µs a small pane's record in Debug). The +live keystroke cells were not run (perf rule: `--live` only for a change +to terminal output); a dump is off the hot path. -- cgit v1.3 From d8e30740463dda5a87fb8a4f6e9755fbb84b7220 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 20:08:12 -0300 Subject: dump: a session's dump keeps every view, the ones not live parked again by load; a session revived for one client is its live view alone, as view 0 Co-Authored-By: Claude Opus 5.5 --- build/snap.zig | 2 +- docs/next-daemon.md | 39 +++++++++-- src/dump.zig | 33 ++++++++-- src/views.zig | 181 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 244 insertions(+), 11 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/build/snap.zig b/build/snap.zig index dad23c53..ece84ad4 100644 --- a/build/snap.zig +++ b/build/snap.zig @@ -93,7 +93,7 @@ pub fn addTty(b: *std.Build, opts: struct { const run_views_replay = b.addRunArtifact(monkey); run_views_replay.addArtifactArg(snap); run_views_replay.addArtifactArg(opts.exe); - run_views_replay.addArgs(&.{ b.fmt("{d}", .{seeds}), "70", "--views=3", "--tape=" ++ tapes ++ "/monkey-views", "--out=" ++ tapes ++ "/monkey-views" }); + run_views_replay.addArgs(&.{ b.fmt("{d}", .{seeds}), "70", "--views=3", "--tape=" ++ tapes ++ "/monkey-views", "--dump", "--out=" ++ tapes ++ "/monkey-views" }); run_views_replay.setCwd(b.path(".")); run_views_replay.has_side_effects = true; run_views_replay.step.dependOn(&run_monkey_replay.step); diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 95713b42..0462c445 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -403,7 +403,9 @@ the ponytail lens: named in a path yet (a script's focus or layout is that view's). - Every change draws every view (`needs_frame` is the session's); a view's own dirtiness waits for a measurement that wants it. - - Dump, Restore and Handover keep one view, the live one. + - Restore and `-l` keep one view, the live one; a dump has every view + (phase 2 notes), and Handover carries panes, which join the views of + the window they land in. - **What proved wrong in the design.** "Views" in the design were a struct the core reads through; the swap made them cheaper and kept every reference. Its rollback copy (phase 5) is the same list of fields. The @@ -1340,14 +1342,39 @@ dump hooks; `pardes-snap --dump`, `pardes-monkey --dump`, and both in layout (`dropMissing`), never more. What was let go is the session's first message after the load. A pane whose terminal another build could not read is a fresh terminal (`ensureKind`). -- **Roots, not wrappers.** A session's dump is three roots: the core, +- **Roots, not wrappers.** A session's dump is five roots: the core, read *into* a core `Pardes.create` made (its local, derived and authority fields stay create's: no defaults to invent for an arena or a host), its 9P mounts (state the user made, inside the 9P server's - authority `fs`), and which file panes' files were on disk when it was - made (one gone by the load is said deleted, so Del asks: its text is - nowhere else; one never on disk comes back as it was). A pane's dump - (the store's, a handover's) is a Pane alone. + authority `fs`), which file panes' files were on disk when it was made + (one gone by the load is said deleted, so Del asks: its text is nowhere + else; one never on disk comes back as it was), the theme's name, and + the views not live. A pane's dump (the store's, a handover's) is a Pane + alone. +- **Every view is in the dump** (decided after phase 4 merged). The live + view is the core's own fields, so `view` and `views_open` were dumped + already, and a session dumped with several views loaded into a core + that said so with no parked slot: its first view switch, close or + drawing of the others unwrapped a null. Leaving them out would have + needed `load` to fold them anyway, and the join (phase 5) needs them: + a replica applies the other clients' `toView`. So `views.dumped` writes + each open view but the live one (the live one's slot is stale by + design): its share of the session but what drawing keeps + (`Pardes.replica.local`, computed from `views.fields`: its panels as + presented, its animations), its slots' serials, and its share of every + pane (`pv_known` says which shares own their selections, not which + exist: a parked view has one of every pane). `views.undumped` parks + them again in `load`, last, so nothing after it rolls back what it + moved; a pane a view has no share of gets the live view's, a view the + core says open that the dump lacks is closed, a dump with no views root + (before this) has one view. They stay out of the digest as before + (parked views are checked when live again); the gate is the re-dump + byte for byte, now also on the three-view monkey seeds. A session + *revived* for one client (Restore, `-l`) is one view, the live one as + view 0 (`views.only`, what the hosts already assumed), and a pane + carried or reopened takes the view it lands in (`sized_by`, + `prompt_view`). Cost: the views root is empty for one view; with + several, a share is about 100 bytes a pane a view. - **Aliasing by reference.** Zerox twins share their text and undo again, an `inherited` cwd points at its pane again. Keyed by address, length and type, which is also what made twins dump their text once. diff --git a/src/dump.zig b/src/dump.zig index c33ac02c..e6a41f09 100644 --- a/src/dump.zig +++ b/src/dump.zig @@ -19,6 +19,7 @@ const MAX_COLS = pardes.MAX_COLS; const effect_path_cap = pardes.effect_path_cap; const Options = pardes.Options; const Pardes = pardes.Pardes; +const views = pardes.views; /// Where the next dump goes. $PARDES_DUMP wins verbatim (the snapshot harness /// pins it for deterministic goldens); else $XDG_DATA_HOME|~/.local/share @@ -144,11 +145,12 @@ pub fn defaultDirectory(buf: []u8) ?[]const u8 { /// the frame's or worked out again (`replica` in each type, src/walk.zig), /// then its 9P mounts, which live with the 9P server's own state, which of /// its panes' files were on disk when it was made (one gone by its load is -/// said deleted: its text is nowhere else now), and its theme's name: the +/// said deleted: its text is nowhere else now), its theme's name (the /// setting is a place in this build's ring of themes, another build's -/// ring is another. -const Session = walk.Dump(&.{ Pardes, []const filesystem.Mount, OnDisk, []const u8 }); -const SessionLoad = walk.Load(&.{ Pardes, []const filesystem.Mount, OnDisk, []const u8 }); +/// ring is another), and the views not live (views.zig: the core holds +/// the live one's). +const Session = walk.Dump(&.{ Pardes, []const filesystem.Mount, OnDisk, []const u8, []const views.Dumped }); +const SessionLoad = walk.Load(&.{ Pardes, []const filesystem.Mount, OnDisk, []const u8, []const views.Dumped }); const OnDisk = [MAX_PANES]bool; fn onDisk(pane: *const pardes.panes.Pane) bool { @@ -180,6 +182,13 @@ pub fn dumpBytes(p: *Pardes) ![]u8 { try d.put(OnDisk, &on_disk); const theme: []const u8 = colors.themes[@min(p.settings.theme, colors.themes.len - 1)].name; try d.put([]const u8, &theme); + const others = try views.dumped(p, p.gpa); + defer { + for (others) |o| p.gpa.free(o.panes); + p.gpa.free(others); + } + const kept: []const views.Dumped = others; + try d.put([]const views.Dumped, &kept); return d.finish(); } @@ -263,6 +272,18 @@ fn loadKnowing(gpa: std.mem.Allocator, opts: Options, bytes: []const u8, notes: for (p.panes) |slot| if (slot) |pane| try ensureKind(p, pane); if (notes) |n| try n.appendSlice(gpa, l.notes.items); } + // The views not live, last: what it moves into the core is never + // rolled back from under it, nothing after it failing. + const at = l.shared.items.len; + const others = l.get([]const views.Dumped) catch |err| switch (err) { + // A dump made before views were in it. + error.EndOfDump => &.{}, + else => return err, + }; + try views.undumped(p, others); + for (others) |o| gpa.free(o.panes); + gpa.free(others); + l.shared.shrinkRetainingCapacity(at); return p; } @@ -351,6 +372,9 @@ fn loadAndRevive(gpa: std.mem.Allocator, opts: Options, bytes: []const u8, previ /// panes): the store and Recent read (a Restore's carried from the session /// it replaces) and its ThemeFile asked for again. fn reviveSession(p: *Pardes, previous: ?*const Pardes) !void { + // ponytail: one client loads it, so it is one view, the live one's; a + // replica joining (phase 5) takes `load`'s, every view. + views.only(p); if (previous) |old| { // Never a serial or a request number the old session gave out. p.next_serial = @max(p.next_serial, old.next_serial); @@ -471,6 +495,7 @@ pub fn loadPane(p: *Pardes, id: usize, bytes: []const u8) !*pardes.panes.Pane { pane.command_from = 0; pane.places.from = 0; pane.collapsed = false; + pane.prompt_view = p.view; if (pane.file) |*f| if (f.content.len == 0 and (f.saved_len orelse 0) > 0) { // A clean file kept slim: its text is its file's. if (filesystem.read(p, f.path)) |text| { diff --git a/src/views.zig b/src/views.zig index e99e1694..329916a2 100644 --- a/src/views.zig +++ b/src/views.zig @@ -79,6 +79,9 @@ pub const PaneView = struct { scroll: usize = 0, cols: u16 = 1, rows: u16 = 1, + + /// In a dump (`Dumped`): what a pane's step works out again. + pub const replica = .{ .derived = .{ "view", "notices", "pan" } }; }; /// A view not live: its share of the session, and the serial of each pane @@ -88,6 +91,141 @@ pub const Parked = struct { serials: [MAX_PANES]u32 = @splat(0), }; +/// What a session's dump keeps of a view's share: all of it but what +/// drawing keeps (Pardes.replica's local fields: its animations, its +/// panels as presented). +const kept = blk: { + @setEvalBranchQuota(100_000); + var names: []const [:0]const u8 = &.{}; + for (fields) |name| { + const local = for (Pardes.replica.local) |l| { + if (std.mem.eql(u8, l, name)) break true; + } else false; + if (!local) names = names ++ &[_][:0]const u8{name}; + } + const out: [names.len][:0]const u8 = names[0..names.len].*; + break :blk out; +}; + +/// A view not live as a session's dump keeps it (dump.zig): its share of +/// the session, its slots' serials, and its share of each pane. The live +/// view is the core's own fields; its slot is stale, and no part of it. +pub const Dumped = struct { + k: u8, + view: Part(Pardes, &kept), + serials: [MAX_PANES]u32 = @splat(0), + panes: []const Share = &.{}, + + pub const Share = struct { id: u16, pv: PaneView }; +}; + +fn bit(k: usize) u8 { + return @as(u8, 1) << @intCast(k); +} + +/// The views not live, as a dump keeps them: the slices `gpa`'s, what +/// they hold the session's. +pub fn dumped(p: *const Pardes, gpa: std.mem.Allocator) ![]Dumped { + var out: std.ArrayList(Dumped) = .empty; + errdefer { + for (out.items) |d| gpa.free(d.panes); + out.deinit(gpa); + } + for (0..max) |k| { + if (k == p.view or !isOpen(p, @intCast(k))) continue; + const slot = p.parked[k] orelse continue; + var d: Dumped = .{ .k = @intCast(k), .view = undefined, .serials = slot.serials }; + copy(&kept, &d.view, &slot.view); + var shares: std.ArrayList(Dumped.Share) = .empty; + errdefer shares.deinit(gpa); + // A view not live has a share of every pane: written when it went + // out, made with one made since (`installed`). `pv_known` says + // which shares own their selections, not which there are. + for (p.panes, 0..) |s, id| if (s) |pane| { + try shares.append(gpa, .{ .id = @intCast(id), .pv = pane.pv[k] }); + }; + d.panes = try shares.toOwnedSlice(gpa); + try out.append(gpa, d); + } + return out.toOwnedSlice(gpa); +} + +/// The views a dump kept (`dumped`) parked again in `p`, a session just +/// loaded with its live view: each its slot, each pane its share in it +/// (one a view had none of as the live view has it). What they hold is the +/// session's then, the slices the caller's; a failure moves nothing. A +/// view the core says open that the dump has none of is not. +pub fn undumped(p: *Pardes, others: []const Dumped) !void { + var take: u8 = 0; + for (others) |d| if (d.k < max and d.k != p.view and isOpen(p, d.k) and take & bit(d.k) == 0) { + take |= bit(d.k); + }; + var made: u8 = 0; + errdefer for (0..max) |k| if (made & bit(k) != 0) { + p.gpa.destroy(p.parked[k].?); + p.parked[k] = null; + }; + if (take != 0) for (0..max) |k| if ((take | bit(p.view)) & bit(k) != 0 and p.parked[k] == null) { + const slot = try p.gpa.create(Parked); + slot.* = .{ .view = .{ .screen_w = p.screen_w, .screen_h = p.screen_h } }; + p.parked[k] = slot; + made |= bit(k); + }; + // Nothing fails from here. + p.views_open &= take | bit(p.view); + for (p.panes) |s| if (s) |pane| { + pane.pv_known |= bit(p.view); + }; + var left = take; + for (others) |d| { + var view = d.view; + if (left & bit(d.k) == 0) { + // Refused (another build's, or twice): what it holds goes. + for (&view.col_tags) |*t| t.deinit(p.gpa); + if (view.chord_arg) |arg| p.gpa.free(arg); + for (d.panes) |sh| if (sh.pv.body.sels.len > 0) p.gpa.free(sh.pv.body.sels); + continue; + } + left &= ~bit(d.k); + const slot = p.parked[d.k].?; + slot.serials = d.serials; + copy(&kept, &slot.view, &view); + for (d.panes) |sh| { + const pane = (if (sh.id < MAX_PANES) p.panes[sh.id] else null) orelse { + if (sh.pv.body.sels.len > 0) p.gpa.free(sh.pv.body.sels); + continue; + }; + if (pane.pv_known & bit(d.k) != 0) { + if (sh.pv.body.sels.len > 0) p.gpa.free(sh.pv.body.sels); + continue; + } + pane.pv[d.k] = sh.pv; + pane.pv_known |= bit(d.k); + } + for (p.panes) |s| if (s) |pane| if (pane.pv_known & bit(d.k) == 0) { + pane.pv[d.k] = fresh(pane); + pane.pv_known |= bit(d.k); + }; + } +} + +/// The live view alone, as view 0: every other closed. A session loaded +/// for one client (a Restore, `-l`) is one view. +pub fn only(p: *Pardes) void { + for (0..max) |k| if (k != p.view) close(p, @intCast(k)); + const k = p.view; + if (k == 0) return; + std.mem.swap(?*Parked, &p.parked[0], &p.parked[k]); + for (p.panes) |s| if (s) |pane| { + pane.pv[0] = pane.pv[k]; + pane.pv_known = if (pane.pv_known & bit(k) != 0) 1 else 0; + if (pane.sized_by == k) pane.sized_by = 0; + if (pane.prompt_view == k) pane.prompt_view = 0; + }; + p.view = 0; + p.views_open = 1; +} + /// The fields of `T` named, as a struct of their own, each of its type and /// default. fn Part(comptime T: type, comptime names: []const [:0]const u8) type { @@ -612,3 +750,46 @@ test "a terminal shown in a view shorter than it keeps its cursor in sight" { p.toView(k); try testing.expectEqual(@as(u16, 0), term.pan); } + +test "a session's dump keeps its views: loaded, each where it was; revived for one client, the live one alone" { + const gpa = testing.allocator; + const p = try Pardes.init(gpa, .{ .tty_only = true, .cols = 80, .rows = 24 }); + defer p.deinit(); + const text = try numbered(gpa, 200); + defer gpa.free(text); + const pane = try p.setTestFile(text); + const id = for (p.panes, 0..) |s, i| { + if (s == pane) break i; + } else unreachable; + const k = p.openView().?; + cursorTo(p, pane, 119); + p.toView(k); + p.update(.{ .command = "Newcol" }); + p.toView(0); + p.toView(k); + cursorTo(p, pane, 30); + const ncol = p.ncol; + const bytes = try pardes.dump.dumpBytes(p); + defer gpa.free(bytes); + const loaded = try pardes.dump.load(gpa, .{ .tty_only = true, .cols = 80, .rows = 24 }, bytes, null); + defer loaded.deinit(); + try testing.expectEqual(pardes.walk.digest(Pardes, p), pardes.walk.digest(Pardes, loaded)); + const again = try pardes.dump.dumpBytes(loaded); + defer gpa.free(again); + try testing.expectEqualSlices(u8, bytes, again); + const lp = loaded.panes[id].?; + loaded.toView(0); + try testing.expectEqual(@as(i32, 119), lp.body.cur_row); + try testing.expectEqual(ncol - 1, loaded.ncol); + loaded.toView(k); + try testing.expectEqual(@as(i32, 30), lp.body.cur_row); + try testing.expectEqual(ncol, loaded.ncol); + // For one client: the view live when dumped, as view 0. + const one = try pardes.dump.initFromDump(gpa, .{ .tty_only = true, .cols = 80, .rows = 24 }, bytes); + defer one.deinit(); + try testing.expectEqual(@as(usize, 1), count(one)); + try testing.expectEqual(@as(u8, 0), one.view); + try testing.expectEqual(ncol, one.ncol); + try testing.expectEqual(@as(i32, 30), one.panes[id].?.body.cur_row); + try testing.expectEqual(@as(?u8, 1), one.openView()); +} -- cgit v1.3 From 22a482a9f1d73472e646b4a83b55bafd9528f5a3 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 20:14:31 -0300 Subject: dump: savePane puts back only the fields it changes, not a copy of the whole 88 KB pane, and makes its empty undo history only for a slim record; the notes say what the small close after a big one measures Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 28 +++++++++++++++++++--------- src/dump.zig | 32 ++++++++++++++++++++++---------- 2 files changed, 41 insertions(+), 19 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 0462c445..0f008974 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1529,16 +1529,26 @@ own spread as the A/A): the perf harness's core cells 0.92-0.99 of base | Restore of it | 1845 ms | 310 ms | 0% | | a closed pane kept: the terminal | 131 ms | 143 ms | 4% | | the 100 MB file | 433 ms | 127 ms | 3% | -| a small file, the step after that | 9.6 ms | 17 ms | 44% | +| a small file, after the 100 MB one | 9.6 ms | 17 ms | 44% | | a small file, alone | 1.0 ms | 1.0 ms | 7% | A clean file's text is in the dump now (what makes it complete), and an unsaved one's undo (the 100 MB file's one edit keeps the text before it); -binary, not base64 in ZON, so the rest is faster. A small pane's close -costs what it did: the 17 ms is the 100 MB pane's teardown (panes are -freed two syncs after they close, its undo's text with it) landing in -the next close's time; three small closes after two others take 1 ms in -both builds. The type table is made once a process (it was made at -every store record: 123 µs to 33 µs a small pane's record in Debug). The -live keystroke cells were not run (perf rule: `--live` only for a change -to terminal output); a dump is off the hot path. +binary, not base64 in ZON, so the rest is faster. The small close after +the big one is not the small pane's: it is the kernel freeing the 100 MB +pane's memory on the 9P connection's thread (panes are freed two syncs +after they close), 208 MB against the base's 108, since the restored +unsaved file's undo holds a copy of its text (found with the access +tracer, perf and schedstat, .scratch/daemon/trace). A small close alone +is level (three after two others: 1.0 ms both; an isolated one, median +of ten: base 0.77 ms, new 0.92). Of the 0.15 ms left, two thirds were +outside the walk and are gone: savePane kept the whole 88 KB Pane aside +and put it back, and built a 41 KB empty undo history when it was not +slim; it saves only the fields it changes now. The type table is made +once a process (it was made at every store record: 123 µs to 33 µs a +small pane's record in Debug). ponytail: Store.close's CRC32 over the +record is most of what is left, and a record repeats its 14.6 KB type +table; a table once per store file would shrink both, a change to the +store's format left until a close's cost matters. The live keystroke +cells were not run (perf rule: `--live` only for a change to terminal +output); a dump is off the hot path. diff --git a/src/dump.zig b/src/dump.zig index e6a41f09..2cef46e2 100644 --- a/src/dump.zig +++ b/src/dump.zig @@ -457,22 +457,34 @@ fn revivePane(p: *Pardes, id: usize, pane: *pardes.panes.Pane, was_on_disk: bool /// file's text, which is read from its file again (a store's record past /// its bound). pub fn savePane(gpa: std.mem.Allocator, pane: *pardes.panes.Pane, slim: bool) ![]u8 { - const was = pane.*; - defer pane.* = was; - if (was.cwd == .inherited) { + // The fields it changes put back after, not the whole pane (88 KB) kept + // aside: a store record is made at every close. + const cwd = pane.cwd; + const attached_to = pane.attached_to; + const terminal = pane.terminal; + defer { + pane.cwd = cwd; + pane.attached_to = attached_to; + pane.terminal = terminal; + } + if (cwd == .inherited) { // Read before the union is written (its tag is set first). const dir = pane.cwdSlice(); pane.cwd = .{ .owned = @constCast(dir) }; } pane.attached_to = null; - var empty: pardes.panes.File.History = .{}; - if (slim) { - pane.terminal = null; - if (pane.file) |*f| { - f.history = ∅ - if (f.output == null and f.revision == f.saved_revision and filesystem.localPath(f.path) != null) f.content = &.{}; - } + if (!slim) return walk.save(pardes.panes.Pane, gpa, pane); + pane.terminal = null; + const f = if (pane.file) |*file| file else return walk.save(pardes.panes.Pane, gpa, pane); + const history = f.history; + const content = f.content; + defer { + f.history = history; + f.content = content; } + var empty: pardes.panes.File.History = .{}; + f.history = ∅ + if (f.output == null and f.revision == f.saved_revision and filesystem.localPath(f.path) != null) f.content = &.{}; return walk.save(pardes.panes.Pane, gpa, pane); } -- cgit v1.3 From 10a7e8768304478555dfe0b619423f5a4ad47d76 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 21:13:33 -0300 Subject: docs/next-daemon.md: phase 2 on main with views and phase 5 part A: a slice a len cuts, the drawn views' selections, and the gates again Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 0f008974..0f3a494c 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1437,7 +1437,19 @@ What proved wrong or missing in the design: - The walker's list rule (exactly `items` and `capacity`) never matched in a safe build: Zig 0.17's ArrayList has a `pointer_stability` lock there, so phase 1's digest hashed every list's capacity and a dump wrote it. - Both take a list's items now. + Both take a list's items now (`walk.isList`). +- Merged onto phase 5 part A (sqsnvoux): a `len` may cut a slice now (a + text's room for its ranges, made whole and used `nsel` deep). The dump + writes such a slice whole, room and all, and a load gives it the room + it had; only one of plain numbers may be cut so (every bit pattern a + value past the cut), else no build. +- Phase 4's other views lost their selections whenever they were drawn: + `unpark` hands a slot's selections to the body (main's fix for a double + free) and `drawOthers` never gave them back, so a drawing session took + them from every view not live while a headless replica kept them. The + three-view monkey seeds with the headless replay found it (seeds 5 and + 35 of 50 diverged at the next key typed at several cursors); + `drawOthers` gives them back after each frame. - "pty masters passed as fds where Handover passes them today": Handover passes none (it refuses terminals: their shells are the window's children), and phase 3 made Detach happen in place, so no dump carries @@ -1501,18 +1513,22 @@ pane's record is 14.5 KB, not 1.5 KB, most of it the type table each self-describing dump carries (ponytail: bounded by 64 a session and the compaction; a table once per store generation if it matters). -The gates on the rebased line (on main vvmuwntp). `zig build replay --Dreplay-seeds=50`, now with `--dump`: the 110 scripts (129 sessions, -32,373 replica digests) and 50 monkey seeds of 70 steps (38,022 digests), -no divergence, and every session's dump at its end loaded to its digest -(all 179, field by field) and dumped again byte for byte (13.9 MB and 5.4 -MB of dumps). The 110 snapshots pass with five goldens moved (above). -unit-test tty passes; unit-test gui fails only phase 1's two known tests; -test/fs.py and test/daemon.py (20 cycles) pass. The cross-build check is +The gates on the line rebased onto main with views and phase 5 part A +(sqsnvoux). `zig build replay -Dreplay-seeds=50`, with `--dump` in all +three parts: the 113 scripts (132 sessions, 29,430 digests over their +first, second and headless replicas), 50 monkey seeds of 70 steps +(37,722 digests) and 50 at three views (29,148), no divergence, and +every session's dump at its end loaded to its digest (all 232, field by +field) and dumped again byte for byte (14.4, 5.4 and 6.7 MB of dumps). +The snapshots pass with five goldens moved (above). `zig build replicas +-- --seeds 12` (Debug, three clients, 30/80/207 ms): none diverged or +unequal. unit-test tty passes; unit-test gui fails only phase 1's two +known tests; test/fs.py and test/daemon.py (20 cycles) pass; the web +shell builds and its touch check passes. The cross-build check is walk.zig's test of two hand-written versions of a struct (a field added, one removed, one renamed, enums renumbered, a width changed, a pane whose required field changed kind left out), both ways, and dump.zig's of a real -session read by name. +session read by name. The real-store check (above) holds on this line. Measured (ReleaseFast, the base binaries from before the rework against this line, interleaved A B A ... A on the perf cores, medians, the base's -- cgit v1.3 From 2d86687b98f389232b213dfa67cfa731b0280fbb Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 23:05:05 -0300 Subject: docs/next-daemon.md: Phase 5 part B terminal notes: what crosses and where in the stream, the authority's steps batched, deltas in flight against the version last sent, the replica's terminal the authority's to the row, what a step reads sent before it, and the numbers ReleaseFast, 20 seeds of three clients at 30, 80 and 207 ms, the shell running a TUI, a session with clears and 8 MB of yes: no divergence, every terminal the authority's at quiescence and 23,508 deltas each the authority's screen at its version (and with a filled heap). Under the flood 72-76 deltas a second a replica (one in flight would be 4.8 at 207 ms), 10 KB/s of deltas and 101 KB/s of units a peer, encode a frame 15-16 us at three peers and 13-15 at one, apply 34-37 us p50. Bench 200x60: yes 65 KB/s a peer, apply 65-72 us (was 2.4 ms); varied text 356 KB/s, apply 0.47-0.49 ms. Local A/B against main within its runs; against base page-down and wheel are main's since. What is not done: kitty images, the units' bytes under a flood, the first text-mode step after a flood sending its history, a stale view slot the rollback copy follows (part A's), echo prediction. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 227 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 0f3a494c..c7cb719d 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1568,3 +1568,230 @@ table; a table once per store file would shrink both, a change to the store's format left until a close's cost matters. The live keystroke cells were not run (perf rule: `--live` only for a change to terminal output); a dump is off the hot path. + +## Phase 5 part B terminal notes + +Built 2026-10-29: screen-state terminals wired into the replicas +(`src/screen_state.zig`, `src/replica.zig`, `src/tape.zig`, +`src/terminal.zig`; `zig build replicas`). Transport-free: bytes in, bytes +out, proven in part A's in-process harness; the remote agent's stream +carries them. Decisions taken at the forks, with the ponytail lens: + +- **What crosses.** The authority's units, as in part A, but a pty's + output is recorded without its bytes. Then, per peer, its terminals' + messages. Each one is u8 pane, u32 the pane's serial (a slot's next pane + is another), u8 kind, and the kind's bytes: + - to a replica (the remote's 0x06): 1, a delta; 2, rows (`encodeRows`); + - to the authority (0x04): 1, an acknowledgement (u32 seq); 2, a request + for rows (u64 lo, u64 hi, absolute); 3, a delta refused: start this + terminal afresh. +- **Where they go in the stream.** Each replica steps every unit on its + own core, and a step in a terminal pane reads the emulator (rows, + cursor, modes, marks). So before the authority steps any input that is + not a pty's output, and once a frame, it sends a peer its own steps since + the last unit (`Authority.pending`, below) and then what the peer is owed + of the terminals (`Screens.flush`); after a step that read a terminal's + history, before its unit, the rows the peer lacks (`Screens.fill`). A + replica takes them in that order between units, and a step it replays + reads the terminal its authority's step read. Nothing goes after an + output step: a flood is paced by the frame rate, not by pty reads. +- **The authority's own steps are batched.** One unit a pty read was + 2000 units a second under `yes`: each rolled back every replica that had + pending input, and each carried a digest of one word a field, about + 12 KB (28 MB/s a peer). `Authority.output` steps into an open batch + without a digest; `pending` sends the batch as one unit with one digest, + before any other input is stepped and once a frame. Under screen state a + digest is one word (the harness's trace mode keeps the fields, to name + one). Units under the flood went from 28 MB/s to 0.1 MB/s a peer. +- **Several deltas in flight, against the version last sent.** Each state + sent is a version of its terminal (`Track`), made once for every peer: a + hash of each active row, the TERMINAL record, and what pardes keeps of + the stream (the OSC 133 marks, `terminal.marksOf`). A peer's delta is + against the version it was last sent. The stream is ordered and loses + nothing, so the peer has that one by the time it applies the next, and + the replica holds exactly that version's blobs. Up to 32 are in flight (13 + at 207 ms and 60 Hz). A peer at the bound gets nothing until an + acknowledgement frees room, then the newest version, the ones between + skipped. Peers at the same base get the same bytes (one encode for all). + A delta against a base the replica does not have (one lost, or it started + afresh) is refused, and its acknowledgement asks for a reset. + - Rejected: mosh's base at the last *acknowledged* state with a + throwaway number. On a stream that loses nothing it only makes deltas + bigger and the replica keep more. +- **The replica's terminal is the authority's, to the row.** + - Its history is exactly as long as the authority's: no byte limit of its + own, trimmed to the authority's count at each delta. Row numbers and + the scrollbar then say the same on both sides, which steps read. + - It takes the authority's viewport, its cursor's semantic content (a + prompt's input, which `promptInputReady` reads) and its marks. + - Its emulator is written by `apply` and fetched rows alone. A replica's + core never ingests, resizes (the authority sizes it; a resize's own + core work, its pointer selections cleared, still runs) or replies to + its program; its pty writes are off the tape on both sides. + - No digest covers an emulator under screen state (`State.digest`); the + harness compares screens instead. +- **What an ingest tells pardes is taped.** Over a terminal, pardes keeps + positions: the text-mode cursor, selections, the edit copy, the pointer's + rows. An ingest moves them when it prunes or erases history or clears the + screen. The authority records what it told them (each tracked row's row + after it and whether it was erased, the clear, whether the pages lived: + `Site.positions`); a replica, which ingests nothing, takes that, and the + rest of the logic runs alike on both. +- **History a step reads is sent before its unit.** Scrolled past more than + a screen at once between deltas, rows are holes on the replica, blank + until fetched. A step in text mode reads a terminal's whole grid, and + the replica's replay of that step must read what the authority's read, + which a fetch an RTT later cannot give. The authority keeps each peer's + holes as the replica does (`Peer.holes`, from what it sent), and after a + step that read a terminal's rows (`State.read`, set outside the render) + sends each peer those it lacks, before the step's unit (`Screens.fill`). + A history row never changes, so each goes once. Scrolled into view, holes + are blanked and asked for (`Replica.screenRequest`); the answer fills + them an RTT later. +- **A flood costs the replica a counter, not its rows.** When a delta + scrolls more than the replica's screen and history hold, nothing it + holds is in the new state: its rows stay where they are, renumbered, the + active ones and the tail written from the delta, the rest holes. Growing + and trimming a history's rows one by one was the 2.4 ms a `yes` delta + took. +- **Speculation stays off terminals.** An input whose live pane is a + terminal is not speculated: its step reads the emulator, the last + delta's here and the pty's newer one there. It waits for the authority + (a key in tty mode already did, at its pty write). Phase 6 predicts the + echo. +- **Rollback never brings a terminal's past back.** + - The rollback copy never held an emulator (State is skipped), and a + replica's emulators, their State included, are allocated on the C heap, + outside its rollback epoch (`terminal.emulator_gpa`, `State.gpa`). + Deltas write them while a copy is held, and nothing of them is a copy's + to save, keep or put back. Without that, a rollback freed a delta's + pages (the test below crashes). A first fix, the epoch paused while a + delta was written, was not enough: the copy also saved emulator memory + reached through a stale pointer (below), and put its old bytes back. + - The core's cache of a terminal's rows is marked stale after a delta, + dropped when a copy is taken and emptied when one is put back: it is a + grid's worth of text, and the copy's would be of an older grid (without + the mark, the test reads it). + +Found on the way: +- **A view's slot that is live is stale, and the copy follows it.** views.zig + keeps a parked view's share as copies, stale while that view is live + (a column tag's text, `Text.own`, freed by the live view since). The + rollback copy walks every slot: a stale pointer that lands on a live + allocation saves it, and a rollback puts its old bytes back. Part A was + safe while every free under a held copy was put off, which holds again + now that a delta's writes are outside the epoch. A ReleaseFast harness + run found it (heap corruption after 20 seeds), a safe build's check that + named what was freed under the copy pinned it. ponytail: the copy still + follows the stale slots; skipping the live view's slot diverged (part of + it is read), and is the next fix in views.zig, not here. +- A pane's `cwd` inherited from a terminal's pane is a pointer, and the + walker follows it: a digest of a pane next to a shell digested the + shell's emulator. It mattered when a digest was taken outside the + authority's tape (`pending`'s first version). +- `resizeGrid` does core work before it resizes (the pointer selections + of a narrowed terminal): skipping it whole on a replica diverged. + +Measured. Correctness ran beside builds; the timings alone on the perf +cores (ReleaseFast). + +- **Correctness.** + - The soak (unit tests, and the 197 recorded streams at 3 seeds): several + deltas in flight, landing late and in order, one in 50 lost (refused, + then a reset), random resizes and fetches. After every delta the + replica's screen, history extent and viewport are the authority's as + encoded, and every history row it holds outside its holes is the + authority's. None failed, Debug or ReleaseFast. + - `zig build replicas`: three clients typing in their own views over + 30, 80 and 207 ms links while the shell runs a TUI, a shell's session + with clears, then 8 MB of `yes` read 4 KB at a time. In ReleaseFast, + 20 seeds at each RTT, there was no divergence, every replica's digest + and every terminal at quiescence was the authority's, and each of the + 23,508 deltas applied matched the authority's screen at its version. + The same held in Debug: 4 seeds at each RTT, and 20 at 30 and at + 80 ms; and in ReleaseFast with every replica allocation filled with + 0xff (`PARDES_REPLICA_FILL`), 20 seeds at each RTT. +- **The terminals in the harness** (ReleaseFast, 20 seeds, three clients; + a peer's numbers while the flood runs): + + | | 30 ms | 80 ms | 207 ms | + |---|---|---|---| + | deltas a replica applied a second | 76.3 | 75.7 | 71.7 | + | delta bytes a second | 10.3 KB | 10.5 KB | 10.0 KB | + | unit bytes a second | 101.6 KB | 102.0 KB | 101.5 KB | + | encode a frame, every peer, p50 / p99 | 15 / 36 us | 16 / 42 us | 16 / 31 us | + | the same, one peer | 13 / 28 us | 14 / 30 us | 15 / 30 us | + | apply a delta, p50 / p99 | 37 / 494 us | 35 / 442 us | 34 / 400 us | + | rows fetched on demand, all runs | 1103 | 649 | 786 | + | rows sent before a step that read them, all runs | 18.0 MB | 18.5 MB | 10.0 MB | + + One delta in flight would have been one an RTT: 4.8 a second at 207 ms. + The rate is the frame's (60) and the inputs' (a delta before each). +- **The bench** (`PARDES_SCREEN_STATE= zig build core-test + -Doptimize=ReleaseFast -Dtest-filter=screen_state`, 200x60, the floods + under the C allocator): + - encode a version, every row new: p50 273 us, p99 394 us, 29.8 KB; + nothing new: 71 / 95 us, 549 bytes; + - `yes` at 19-20 MB/s: 63 deltas a second, 65 KB/s a peer (1038 bytes a + delta), encode a frame 28 us at one peer and 29 at three, apply p50 65 + to 72 us and p99 202 to 265 us (it was 2.4 ms); + - varied text at 59-64 MB/s: 356 KB/s a peer, encode a frame 52 / 54 us, + apply p50 468 to 493 us. +- **Part A's costs, with terminals** (the harness, three clients): + speculative apply p50 28 to 42 us; copy p50 95 to 151 us (1.3 to 1.5 + MB); rollback p50 331 to 442 us, p99 1.6 ms at 80 and 207 ms. At 30 ms + the copy's p99 is 7.8 ms and a rollback's 17 ms: a register holding a + terminal's whole grid (`%` then `y` in a shell's text, 2 MB after the + flood) is copied whole by every speculation. Registers could be pinned + as file texts are; left as found. +- **No local regression.** `pardes-perf-live --live`, A and the new tty + interleaved A B A ... A (4 runs of the new one, 5 of A) on the perf cores; + the median of the runs' p50s, new/A, present and core: + + | | against main (kuovmsxw) | against base | A/A (base) | + |---|---|---|---| + | key-down | 1.02 / 1.02 | 0.94 / 0.94 | 1.00 / 1.01 | + | key-right | 1.02 / 1.03 | 1.00 / 0.95 | 0.99 / 1.01 | + | edit-char | 1.01 / 1.00 | 0.99 / 0.99 | 1.00 / 1.00 | + | page-down | 0.92 / 0.95 | 1.22 / 1.05 | 1.00 / 0.99 | + | wheel | 0.83 / 0.85 | 1.13 / 1.10 | 1.00 / 1.00 | + | `yes \| head -c 32M` wall | 0.92 | 1.12 | 0.94 | + | a 1M-line open | 1.03 | 1.14 | 0.97 | + + Against main, this line's own A/B, every cell's runs overlap main's. + Page-down and wheel against the base read 1.21 and 1.12 in an earlier + run too, and 0.92 and 0.83 against main: main's since the base, not + this line's. What the local path gained: a counter a pty read, a resize + and a scroll; a flag a frame and a store when rows are built; and, for a + terminal with positions over it, a taped value (a null check) and two + stack arrays. + +Not done, and why: +- Kitty images in terminals still do not cross. +- Units under a flood still cost about 0.1 MB/s a peer: an output step is + about 45 bytes of tape (its input, the pump, a host call), 2000 a second. + Phase 6's zstd takes most of it (each step's records are alike); or a + host could feed a pane's queued reads to the core as one. +- The first step in text mode after a flood sends its terminal's whole + history of holes, up to a few MB, ahead of its unit (ponytail: a step + reads every row, its motion surface). +- A 9P run waiting on a replica's marks (`noteMarks`) reads them from the + last delta, not from the output step; its writes are the authority's, but + `sent_command` could differ. The output pins (`out_start`, `out_end`) + stay the authority's. +- Text reads outside `buildRows` (a pointer selection's copy in tty mode, + 9P's screen reads) do not ask for the rows; 9P's are the authority's + anyway. +- **A hazard for part A, not fixed here.** The rollback copy follows a + view's stale slot (above). If a stale pointer lands on a live allocation + that a speculation then writes in place, the rollback puts the old bytes + back over that allocation's new owner. Emptying a slot's pointers when + its view goes live (views.zig `unpark`) would end it; the copy skipping + the live slot instead diverged, so part of it is still read. +- Echo prediction (phase 6): a key into a terminal waits a round trip. +- A replica steps every output step of the authority's (bytes-less), about + 2000 a second under a flood, each a core step. +- The GUI was not measured: its host is unchanged, and the core's terminal + path is the tty's. +- The other agent wires it in: the tty and GUI hosts, `Screens` per + attached peer, and a replica frontend drawing after `screen`. -- cgit v1.3 From bbb37c129126e9675d3ab6f7730083dba2d2becd Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 22:56:07 -0300 Subject: docs/next-daemon.md: phase 6's upgrade by re-exec: the decisions at its forks, what was found, the gates here and on the FreeBSD box, the startup A/B, and what is not done Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 260 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index c7cb719d..00f66cf8 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1795,3 +1795,263 @@ Not done, and why: path is the tty's. - The other agent wires it in: the tty and GUI hosts, `Screens` per attached peer, and a replica frontend drawing after `screen`. + +## Phase 6: upgrade notes + +Built 2026-10-29 (`src/upgrade.zig`, the tty and GUI hosts, `daemon.zig`, +`main.zig`, `dump.resumeFromDump`; `test/daemon.py --upgrade`). A running +session, and the daemon, move onto the pardes installed since they +started, in place: text, undo, views and layout, terminals with their +running shells, and their terminals, windows and daemon link. Decisions +taken at the forks, with the ponytail lens: + +- **The trigger is the word, `Upgrade`** (`SPC s U`, any tag, `/ctl` and + `/pane/N/exec` over 9P). It moves the daemon too: the session sends it + `upgrade` just before its own exec, so one word moves both, and no word + of the daemon's own was needed. Nothing upgrades by itself. A focus-in + (one comes back to the terminal or window from rebuilding elsewhere) + stats the start path, and a file there that is not the one the process + started from is said once, in a line: `a newer pardes is installed: + Upgrade (SPC s U) moves this session onto it`. Rejected: a watch on the + binary's directory (an inotify entry and an event source for a line), + and upgrading on its own (a build half-written, or one the user means + to test first, would take every session). +- **Which binary: the path this process was started from**, read once + just after its first frame (`std.process.executablePath`, a readlink, + and a stat for the notice: off a startup's way, and no build is + installed within a frame of a start), exec'd again at the upgrade. + `install(1)` and `zig build` put a new file at the path, renamed over + the running one, which `/proc/self/exe` then names `(deleted)`. The + record carries the path + exec'd, so a process that fell back to the old binary (whose own exe is + that deleted file) upgrades from the same path next time. Not argv[0] + and PATH: a session's PATH is the one it started with, and the readlink + is what the daemon's spawn already used. +- **Asked first, ` --version`**: run as it is, it must exit 0 with a + `pardes ` line within 10 s. A file not there, not a pardes, or one that + dies at start is found before anything is let go, and said ("Upgrade: + exited 3 at --version; this session stays as it is"). 9P is + served while it runs (the turn rested), so a client's clunk after its + `Upgrade` write, and anything else, is answered; after it, nothing is. +- **At a step boundary**: after a pump, the pty readers are canceled (a + canceled tty reader posts an end of file, passed over while upgrading), + the terminals' input readers stopped, the loop's queue applied, and the + core settled (`Pardes.settle`: queued inputs, then effects). In the GUI + the queue is drained, the readers canceled, and drained again (a reader + waiting for room gets it). Bytes read are never lost; keys typed during + the exec wait in the terminal's kernel buffer for the new process. +- **The dump goes to the DumpDir, `pardes-upgrade-.dump`**, not a + memfd and not the runtime directory: a refused load leaves it where a + relative `Restore` finds it, on a disk and not a tmpfs (a big file's + session dumps 100 MB and up). Never $PARDES_DUMP, which is a Dump's. A + successful load deletes it. +- **The record is one argument**, `--upgraded=`, ahead of the + arguments the session was started with, which configure it again + (`--9p=`, `--tty-toggle`, `--geometry`, `--nested`); its file, stdin, + directory, `--detach` check, forwarding and daemon fork are skipped. A + line a word: `link`, `stub`, `old` (fds), `window`, `pty + `, `tty `, `exe`, + `dump`, and `refused` in a fallback; a word another build adds is + passed over. Every fd it names is close-on-exec but across the one exec + (`Record.closeOnExec`), and the new process sets them back at once, so + no shell it forks inherits a master or a terminal. +- **The pid stays**: the shells stay the session's children, the stub + keeps waiting on it, the 9P name of a session named by its pid is the + same, and PDEATHSIG (set again) still means the stub's death. +- **Terminals.** A kept terminal pane is not revived (`resumeFromDump` + with `kept`): no reset, no banner, no new shell; ghostty's state is the + dump's, the program's own, and its pty keeps its size (no 1x1 nudge: + the screen is already right). The host puts the master back in the + pane's slot and reads on (a command pane's exit watched again). A + terminal pane with no live master (its shell gone) is revived as a + Restore revives it. +- **The tty's terminals stay raw, their cooked termios in the record**: + put back to cooked between the two processes, a key would echo and a + ^C (forwarded by the stub) could kill the new process before its + handlers. What is a stack is unwound: the kitty keyboard flags are + popped (the new process pushes them once its query is answered), its + kitty images deleted and its shm names unlinked (the pid is the same, + and so would the names be). The alternate screen, mouse and paste modes + stay on; the new process sets them again and draws everything, one + full frame over the old one in place. The old image's prompt rc files + (the running shells have read them) are swept by the new one, which + makes its own: at its sweep, any of its pid are the old image's. They + stay for an exec that failed. +- **Views come back whole.** `resumeFromDump` keeps every view the dump + has (no `views.only`) and does not run the startup config again (the + session's settings are the dump's). Each terminal's view is in its + `tty` line: the primary one is `run`'s, the others are entered as + `--attach` enters one, with their stubs; a view that lost its terminal + closes. The GUI opens a window for each open view, the live one first + (`openAnotherWindow(k)`): its windows are the one flash, as decided. +- **9P is cut and comes back.** What the connections had to send is let + go first (`Listener.flushOut`, the 200 ms the quit's path already + had), then the exec closes the listener and its lock, and the new + process claims the name again. ponytail: a dial during the exec and the + load is refused (one round of a client's retry); passing the listener + through as well would keep it queued. +- **Replicas are told in words.** Each connection a session serves gets + a `refused` frame ("the session is upgrading to another build of + pardes: connect again with that build") and a shutdown before the exec. + Their hello's semantics hash would not match the new build anyway, and + the session registers its new hash on the same link (a second `hello`, + which the daemon takes as an update). The reconnect, and a local + client of the same path re-exec'ing itself, are phase 5B's: there is no + replica client yet. +- **The daemon re-execs the same way**: its own start path, asked + `--version` first, the listener and lock (fds 3 and 4), the replicas' + listener, every session's link and every held master kept, its + registry on its command line (`--daemon --upgraded=` with a `c` line a + connection, its slot, kind, pid, state, semantics and name, and an `h` + line a master and its owner's slot); a replica whose hello is still on + its way is let go and dials again. Sessions never see it: their links + are the same sockets. +- **Failure never loses a shell.** + 1. The binary does not answer `--version`: nothing changes, said. + 2. The exec fails: the fds close-on-exec again, the readers and the + terminals' keys start again, the kitty flags are pushed back, the + dump deleted, said. + 3. The new build cannot load the dump: it execs the old binary again, + from `/proc/self/exe` opened before the exec (`old`, Linux), with + `refused` set; that one loads its own dump and says "Upgrade: the new + build could not load this session (); it goes on as ". + 4. No old binary to go back to (FreeBSD has no /proc by default), or it + refuses too: a fresh session adopts every kept master as a terminal + pane (the Adopt path) and says "Upgrade: no build could load the + session (); its shells are here, and Restore brings the + rest back". The dump stays. + 5. A crash after the exec is the crash Phase 3 covers: the daemon's + dups keep the shells for an Adopt, and the dump is in the DumpDir. + `PARDES_TEST_UPGRADE_REFUSE=1` makes a start refuse the load as such a + build would (the first start only), `=all` the fallback's start too: + test/daemon.py installs a script that sets it and execs the new build. + +Found on the way: +- A fresh session's first pane has no directory until its own shell's + spawn is performed, so the `Tty` word that adopts a shell refused ("Tty: + : no such directory") and the shell was hung up with "no room": the + kept shells are taken after the startup effects, and those they emit + performed then. +- A test reconnecting after the word could reach the old process, still + asking `--version` (and serving 9P meanwhile): the test waits for the + exec (the session's exe, or on FreeBSD its new semantics in the + daemon's registry) before it reconnects. +- An `Adopt` (or a `Detach`) served in a session's first pump, before its + first frame, is refused as if no daemon kept it: the reader that takes + the daemon's `ok` starts after that frame, and `kept()` waits 2 s for + it. The test's next session waits for its first frame. Pre-existing, + left as it is. + +Gates (Debug, `A` this line unstamped, `B` the same with `-Dstamp-commit`, +a trivial visible change: `pardes 0.24 ()` and another semantics +hash; both installed in turn at a private prefix by a rename, as +`zig build` and install(1) do): +- `python3 -B test/daemon.py --upgrade A B`, under a private runtime dir. + A tty session with an unsaved edit, a shell running `sleep 1000; echo + done-after-upgrade` and a second terminal (a second view), and a + replica of A's build connected: + - B installed, a focus-in says the newer build; `Upgrade` written to + `/ctl`: the session's 9P answers again 123 ms later (81 to 83 ms on + the box). The session's pid is the same, `/proc//exe` is B (not + deleted), and so is the daemon's, the same pid. + - The edit is there, the pane dirty, and `Undo` takes it back. + - The shell's pid is the same, still the session's child, its `sleep` + running; killed, the shell goes on and `done-after-upgrade` shows. + - Both terminals are drawn again (each its own view); the stubs wait on. + - The replica of A's build is told `the session is upgrading to + another build...`; a hello of B's build is welcomed by the daemon, + one of A's refused (the daemon lists the session as B). + - The fds level: the session's as before but the replica's, the + daemon's the same; the session's prompt files as many as before, the + new image's. + - A refusing build (a script that sets `PARDES_TEST_UPGRADE_REFUSE=1` + and execs B) installed and asked: the session goes back to the binary + it ran, its exe that inode again, the shell and text as they were. + - One that refuses everywhere (`=all`): a fresh session, the shell a + pane of it (`echo $$` there is its pid), the dump left in the DumpDir. + - The session killed outright: the daemon, upgraded three times, still + holds the shell's master, and a new session adopts it. + - Nothing left: no process, no socket. Passed 3 runs of 3 (and every + run since). +- `--upgrade --gui A-gui B-gui`: a window session in test mode (hidden + windows, captured) with a second window from `pardes-gui --attach`, the + same edit and job: after `Upgrade` the pid is the same and the exe B's, + the daemon B, the edit and undo there, the shell and its job the + session's, both windows' captures written again, B's hello welcomed; + the session ends with nothing left. Passed 5 runs of 5. +- FreeBSD 15.1 (the box, `pardes-ua` and `pardes-ub` cross-built from + this line): the same tty test but the fallback to a kept copy (there is + none). The new build is told by its semantics in the daemon's registry, + and the daemon's re-exec by `--upgraded=` on its command line; the fds + level by procstat. 81 to 83 ms from the word to 9P. Passed 9 runs of 10 + (the last two on this line's final builds); the one failure was the + test's first-frame race above, fixed, and it passed after. +- The existing gates: `test/daemon.py` (20 cycles) and `--gui` (10) + pass; `test/fs.py` passes (49 s); `zig build snap` 113 of 113; `zig + build replay` passes (the 113 scripts, 20 seeds at one view and 20 at + three, their dumps loaded to their digests); unit-test tty 1479 of + 1499, 20 skipped; unit-test gui 1550 of 1561, 9 skipped, and the two + known failures (`tagline pointer mapping...`, `a tag caret past its + row's end...`). core-test's new tests: the record read back as written + (and refused when cut), and an upgraded dump keeping its views, a + running terminal's alternate screen and no new shell, and settings the + startup config would change. + +Measured. The upgrade itself, Debug, a 300 MB binary: from the word +written to `/ctl` to the session's 9P answering again, 123 to 130 ms, +one run beside builds 299 ms (the `--version` ask, the drain, a dump of +a small session, the exec and the load); on the box, 81 to 83 ms with +its 125 MB binary. + +No local change (ReleaseFast tty, main kuovmsxw against this line, both +live at once, their samples alternating, through the perf lock; lat.py's +startup is a spawn with no daemon running to its first frame's end): + +| | A/B | A/A | +|---|---|---| +| startup, 100 a build: p50 | 0.944 | 1.010 | +| the same: mean | 0.919 | 1.061 | +| the same: median of the fast mode (and its share) | 0.964 (54 to 52) | 0.996 (69 to 73) | +| startup, 40 a build, two runs: p50 | 1.001, 1.266 | 0.998, 1.010 | +| key in a file, to its frame, p50, two runs | 0.997, 1.002 | 0.958, 0.990 | +| key echoed by bash, to its frame, p50, two runs | 0.997, 0.991 | 1.004, 0.999 | + +The startup times have two modes on this machine, about 4.5 and about 12 +ms, and the share in each moves between runs and between the builds of +an A/A alike, so a p50 that falls between them jumps (the 1.266). Within +a mode the builds are the same. A first A/B, made while the path was +still read at the start (a readlink and a stat before the first frame, +some microseconds), read p50 1.04 to 1.13 against an A/A of 0.98 to 1.00 +on runs that overlapped builds; the read was moved after the first frame +anyway, so a start does nothing it did not. + +Not done, and why: +- Remote replicas reconnect on their own, and a local client of the same + path re-execs itself first: phase 5B's (no replica client exists yet). + What they get today is the `refused` frame above, then the hangup. +- The 9P listener is not kept across the exec: a dial in that window (a + hundred milliseconds or so) is refused. Clients that dial afresh (the + hosts' Client) retry; a mounted session sees its connection end. +- On FreeBSD a build that cannot load the dump starts afresh (shells + adopted, dump kept): there is no `/proc/self/exe` to go back to by + default. Keeping an fd of the binary open from the start would do it, + at an fd a session. +- A crash of the new build after it starts is a crash: the shells wait in + the daemon for an Adopt and the dump is in the DumpDir, but nothing + restarts the session on its own (the stub, which runs the old binary, + could). +- An escape sequence a program had half written when the dump was made is + lost (the snapshot is at ground state, Phase 2 notes). +- Language servers and `|` pipes running at the upgrade are let go (their + pipes close at the exec; they end), and the shells of panes closed just + before it, still waiting to be reaped, are reaped only when the session + ends. A daemon note (a terminal attaching, adopted shells) read by the + old process's thread but not yet taken by its loop is lost; the stub of + such a terminal sees its pipe end. +- A detached session's upgrade (no terminal) was checked by hand + (scratch script), not in test/daemon.py. +- GUI windows come back at the size and place the compositor gives them. +- A daemon protocol bump across an upgrade is not handled: the sessions + of the older build keep talking theirs. +- A session name with a newline would break the daemon's registry line. +- An upgraded process records no tape (a recording ends at the upgrade). -- cgit v1.3 From bce88593014a4078dc9259855944f9ddbc19fe54 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Thu, 29 Oct 2026 23:14:10 -0300 Subject: views and the rollback copy: a live view's slot keeps no pointer into what the live core owns; a register's texts and a file's location rows are pinned, not copied views.zig: when a view goes live (to, close, drawOthers putting the live one back, only), its parked share is emptied (`emptied`): it was left holding copies of the pointers it handed the live fields (a column tag's text, a chord's argument, the presentation's), stale until it parked again. The rollback copy walks every slot: a stale pointer whose memory the live view had freed and another taken saved that other's bytes and put them back. A pane's share holds no pointer but its selections, which `unpark` already hands over. The test fails without it: a tag's text freed under view 0's stale slot, its memory taken by an allocation the copy has no business with, written under a held copy, and the copy put back. Registers.zig, locations.zig, rollback.zig: a register's values and joined text, and a file's location rows (the dropped ones, a query's saved), are replaced whole and never written, so the copy pins them as it does a file's text, freed raw (a safe build's free would write over one a copy still holds); rows pin whole with what each owns (`pinnedDeep`). In the harness at 30 ms the big copies were a file's location rows, 5-6 MB after the flood, not the register. ReleaseFast, 20 seeds, three clients, p99: copy 5.1 -> 0.22 ms at 30 ms (0.19-0.20 at 80 and 207); rollback 14.9 -> 9.5 ms at 30 ms (1.3-1.4 at 80 and 207), what is left being the replay's digests (half the harness's CPU is Wyhash over the whole core per replayed unit). Tests: a 2 MB register and 500 location rows pinned, not saved, and whole after a speculation drops them and the copy is put back. docs/next-daemon.md: the follow-ups. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 44 ++++++++++++++++--- src/Registers.zig | 13 ++++-- src/locations.zig | 9 ++-- src/rollback.zig | 123 +++++++++++++++++++++++++++++++++++++++++++++++++--- src/views.zig | 16 +++++++ 5 files changed, 184 insertions(+), 21 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 00f66cf8..55921027 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1743,7 +1743,7 @@ cores (ReleaseFast). the copy's p99 is 7.8 ms and a rollback's 17 ms: a register holding a terminal's whole grid (`%` then `y` in a shell's text, 2 MB after the flood) is copied whole by every speculation. Registers could be pinned - as file texts are; left as found. + as file texts are. (Done since: the follow-ups below.) - **No local regression.** `pardes-perf-live --live`, A and the new tty interleaved A B A ... A (4 runs of the new one, 5 of A) on the perf cores; the median of the runs' p50s, new/A, present and core: @@ -1782,12 +1782,8 @@ Not done, and why: - Text reads outside `buildRows` (a pointer selection's copy in tty mode, 9P's screen reads) do not ask for the rows; 9P's are the authority's anyway. -- **A hazard for part A, not fixed here.** The rollback copy follows a - view's stale slot (above). If a stale pointer lands on a live allocation - that a speculation then writes in place, the rollback puts the old bytes - back over that allocation's new owner. Emptying a slot's pointers when - its view goes live (views.zig `unpark`) would end it; the copy skipping - the live slot instead diverged, so part of it is still read. +- **A hazard for part A** (fixed since, below): the rollback copy followed + a view's stale slot. - Echo prediction (phase 6): a key into a terminal waits a round trip. - A replica steps every output step of the authority's (bytes-less), about 2000 a second under a flood, each a core step. @@ -1796,6 +1792,40 @@ Not done, and why: - The other agent wires it in: the tty and GUI hosts, `Screens` per attached peer, and a replica frontend drawing after `screen`. +Follow-ups (2026-10-29): +- **A live view's slot keeps no pointer.** When a view goes live (`to`, + `close`, `only`, and `drawOthers` putting the live one back), its parked + share is emptied (views.zig `emptied`): it held the pointers it handed + the live fields (a column tag's text, a chord's argument, the + presentation's) until it parked again, and the copy, walking every + slot, saved and put back whatever memory had since taken their place. + A pane's share holds no pointer but its selections, which `unpark` + already hands over. A test frees a tag's text under the stale slot, + hands its memory to another allocation, writes it under a held copy and + puts the copy back: it fails without the fix. +- **Registers and location rows are pinned, not copied.** A register's + values and joined text, and a file's location rows (with the ones an + edit dropped and a query's saved), are replaced whole and never written, + so the copy pins them as it does a file's text, freed raw. Rows pin + whole with what each owns (its path, its colours: `pinnedDeep`). In the + harness at 30 ms the big copies were not the register but a file's + location rows, 5 to 6 MB after the flood. ReleaseFast, 20 seeds, three + clients, p99: copy 5.1 ms to 0.22 ms at 30 ms (0.19 to 0.20 ms at 80 and + 207); rollback 14.9 to 9.5 ms at 30 ms (1.3 to 1.4 ms at 80 and 207). + What is left of a rollback's tail is the replay's digests: half the + harness's CPU is Wyhash over the whole core at every replayed unit, the + pinned blobs included (part A's digest). Hashing a pinned blob once, + while it lives, would take most of it. +- **The live perf harness ends what it starts.** Every process a live run + starts inherits its private runtime directory: after each cell, and in + the cleanup every path takes, the harness sends TERM, then KILL, to each + whose environment names it (`liveSweep`). With a wrapper that leaves a + daemon-kept detached session (as the remote agent's client-live.sh + does), the old harness left three sessions and their daemon a run; this + one leaves none. The gate's baselines are recorded again from base (main + and the new perf.zig), and .scratch/daemon/base/pardes-perf-live rebuilt + from it. + ## Phase 6: upgrade notes Built 2026-10-29 (`src/upgrade.zig`, the tty and GUI hosts, `daemon.zig`, diff --git a/src/Registers.zig b/src/Registers.zig index 5e23da5b..760aff9a 100644 --- a/src/Registers.zig +++ b/src/Registers.zig @@ -5,12 +5,17 @@ //! computed from the editor rather than stored (`_`, `#`, `.`, `%`) and the //! clipboard's paste side are read in edit.zig, which has the pane at hand. const std = @import("std"); +const freeRaw = @import("memory.zig").freeRaw; const Registers = @This(); list: std.ArrayList(Reg) = .empty, last_search: u21 = '/', +/// A value and the joined text are replaced whole, never written in place, +/// so a replica's rollback copy pins them as it does a file's text +/// (src/rollback.zig `pinned`), and they are freed raw: a safe build's +/// free would write over one the copy still holds. pub const Reg = struct { name: u21, /// gpa-owned, one per range, in document order @@ -30,9 +35,9 @@ pub fn deinit(r: *Registers, gpa: std.mem.Allocator) void { } fn freeReg(gpa: std.mem.Allocator, reg: *Reg) void { - for (reg.values) |v| gpa.free(v); + for (reg.values) |v| freeRaw(gpa, v); gpa.free(reg.values); - if (reg.joined) |j| gpa.free(j); + if (reg.joined) |j| freeRaw(gpa, j); } pub fn get(r: *Registers, name: u21) ?*Reg { @@ -69,9 +74,9 @@ pub fn put(r: *Registers, gpa: std.mem.Allocator, name: u21, bytes: []const u8, freeReg(gpa, reg); reg.* = .{ .name = name, .values = values }; } - gpa.free(reg.values[i]); + freeRaw(gpa, reg.values[i]); reg.values[i] = copy; - if (reg.joined) |j| gpa.free(j); + if (reg.joined) |j| freeRaw(gpa, j); reg.joined = null; } diff --git a/src/locations.zig b/src/locations.zig index cdb228a6..40f22f85 100644 --- a/src/locations.zig +++ b/src/locations.zig @@ -1,4 +1,5 @@ const std = @import("std"); +const freeRaw = @import("memory.zig").freeRaw; const pardes = @import("pardes.zig"); const look = @import("look.zig"); const syntax = @import("syntax.zig"); @@ -25,12 +26,14 @@ pub const Row = struct { colors: []u8 = &.{}, }; +/// Raw (no safe build's overwrite): a replica's rollback copy pins a row +/// blob and what its rows own (src/rollback.zig `pinnedDeep`). pub fn freeRows(gpa: std.mem.Allocator, rows: []Row) void { for (rows) |row| { - gpa.free(row.path); - gpa.free(row.colors); + freeRaw(gpa, row.path); + freeRaw(gpa, row.colors); } - gpa.free(rows); + freeRaw(gpa, rows); } /// A list of rows being built, given up: each row's own bytes, then the list. diff --git a/src/rollback.zig b/src/rollback.zig index 79b8202e..a7027da6 100644 --- a/src/rollback.zig +++ b/src/rollback.zig @@ -11,10 +11,11 @@ //! - `Copy`, the walker's walk of the core's state (src/walk.zig: what a //! digest covers, so views, buffers, registers, counters and terminal //! metadata), saving the bytes of every allocation it reaches, the core -//! struct's own included. A file's text, its line index and its undo -//! snapshots are pinned instead: an edit puts a new buffer in their -//! place (File.setContentSpan) and never writes one, so the copy keeps -//! a reference and the put-off free keeps it alive. +//! struct's own included. A file's text, its line index, its undo +//! snapshots and a register's texts are pinned instead: an edit puts a +//! new buffer in their place (File.setContentSpan, Registers.put) and +//! never writes one, so the copy keeps a reference and the put-off free +//! keeps it alive. //! //! Putting the copy back writes each saved allocation's bytes over it, so //! every pointer in them points where it did, at what was there; then the @@ -197,6 +198,8 @@ pub const Copy = struct { /// of the blobs: a file's text or an undo snapshot's. where: []const u8 = "", pinnable: bool = false, + /// Whether it is a blob of rows that own allocations of their own. + deep: bool = false, const Region = struct { addr: usize, off: usize, len: usize }; const Pin = struct { addr: usize, len: usize, hash: u64, where: []const u8 }; @@ -342,6 +345,7 @@ pub const Copy = struct { if (comptime opaqueTo(F)) continue; c.where = @typeName(T) ++ "." ++ name; c.pinnable = comptime pinned(T, name); + c.deep = comptime pinnedDeep(T, name); if (comptime walk.lengthOf(T, name) != null and @typeInfo(F) == .array) { const len = comptime walk.lengthOf(T, name).?; if (comptime heapIn(@typeInfo(F).array.child, 0)) for (@field(v, name)[0..walk.lengthIn(v, len)]) |*e| try c.visit(@typeInfo(F).array.child, e); @@ -394,11 +398,14 @@ pub const Copy = struct { if (s.len == 0) return; const addr = @intFromPtr(s.ptr); if (try c.reached([]const E, addr)) return; - if (comptime !heapIn(E, 0)) if (c.pinnable and s.len * @sizeOf(E) >= blob_min and c.epoch.?.owns(addr) != null) { + const flat = comptime !heapIn(E, 0); + if ((if (flat) c.pinnable else c.deep) and s.len * @sizeOf(E) >= blob_min and c.epoch.?.owns(addr) != null) { + // A blob of rows with allocations of their own pinned whole: + // they are as immutable as it, and kept alive with it. const bytes = std.mem.sliceAsBytes(s); try c.pins.append(c.meta, .{ .addr = addr, .len = bytes.len, .hash = if (checked) std.hash.Wyhash.hash(0, bytes) else 0, .where = c.where }); return; - }; + } _ = try c.allocation(addr); if (comptime heapIn(E, 0)) for (s) |*e| try c.visit(E, e); } @@ -409,7 +416,21 @@ pub const Copy = struct { fn pinned(comptime T: type, comptime name: []const u8) bool { const File = pardes.panes.File; return (T == File.State and (std.mem.eql(u8, name, "content") or std.mem.eql(u8, name, "line_starts"))) or - (T == File.Snapshot and std.mem.eql(u8, name, "content")); + (T == File.Snapshot and std.mem.eql(u8, name, "content")) or + // A register's values and joined text: a yank of a terminal's whole + // grid is megabytes, which every speculation would otherwise copy. + (T == pardes.Registers.Reg and (std.mem.eql(u8, name, "values") or std.mem.eql(u8, name, "joined"))); +} + +/// Blobs of rows that own allocations of their own (a location's path, its +/// colours), the array and they replaced whole and never written, freed raw +/// (locations.freeRows): a file's location rows (5 MB after a flood of +/// search results), the ones an edit dropped, a query's saved. +fn pinnedDeep(comptime T: type, comptime name: []const u8) bool { + const File = pardes.panes.File; + return (T == File.State and std.mem.eql(u8, name, "location_rows")) or + (T == File.DroppedMarks and std.mem.eql(u8, name, "rows")) or + (T == pardes.panes.Query.Saved and std.mem.eql(u8, name, "rows")); } fn annotatedStruct(comptime T: type) bool { @@ -497,3 +518,91 @@ test "a copy put back is the core as it was; one let go keeps what was done" { p.deinit(); try std.testing.expectEqual(@as(usize, 0), e.live.count()); } + +test "a view's slot while it is live keeps no pointer a rollback could put old bytes back through" { + // The C heap: a freed chunk comes back to the next allocation its size. + var e: Epoch = .{}; + defer e.deinit(); + const gpa = e.allocator(); + const p = try Pardes.init(gpa, .{ .cols = 80, .rows = 24, .file = "mise.toml", .image_allocator = gpa, .pdf_allocator = gpa, .tree_sitter_allocator = gpa, .frame_allocator = gpa }); + defer p.deinit(); + p.pump(.{}) catch {}; + _ = p.openView() orelse return error.NoView; + // View 0's column tag has a text of its own; out to view 1 and back. + const text = "the column tag of view 0, its own"; + p.col_tags[0].own = try gpa.dupe(u8, text); + p.toView(1); + p.toView(0); + // The live view drops it, and its memory is another's: something the + // copy has no business saving. + const was = p.col_tags[0].own.?; + gpa.free(was); + p.col_tags[0].own = null; + const other = try gpa.alloc(u8, text.len); + defer gpa.free(other); + if (other.ptr != was.ptr) return error.SkipZigTest; // the heap handed out another + @memset(other, 'a'); + var c: Copy = .{}; + defer c.deinit(); + try c.take(&e, p); + // Written in place while the copy is held, then the copy put back. + @memset(other, 'b'); + c.restore(); + try std.testing.expect(std.mem.allEqual(u8, other, 'b')); +} + +test "a register's texts are pinned, not copied, and come back whole" { + var e: Epoch = .{ .child = std.testing.allocator }; + defer e.deinit(); + const gpa = e.allocator(); + const p = try Pardes.init(gpa, .{ .cols = 80, .rows = 24, .file = "mise.toml", .image_allocator = gpa, .pdf_allocator = gpa, .tree_sitter_allocator = gpa, .frame_allocator = gpa }); + p.pump(.{}) catch {}; + // A terminal's whole grid yanked: megabytes, and joined once read. + const grid = try std.testing.allocator.alloc(u8, 2 << 20); + defer std.testing.allocator.free(grid); + @memset(grid, 'y'); + const reg = pardes.Registers.default; + p.registers.put(gpa, reg, grid, 0, 1, true); + _ = p.registers.text(gpa, reg) orelse return error.NoText; + var c: Copy = .{}; + defer c.deinit(); + try c.take(&e, p); + try std.testing.expect(c.savedBytes() < 1 << 20); + try std.testing.expect(c.pinnedBytes() >= 4 << 20); + // A speculation yanks over it, and is undone. + p.registers.put(gpa, reg, "a word", 0, 1, true); + c.restore(); + const back = p.registers.value(reg, 0) orelse return error.Lost; + try std.testing.expectEqual(grid.len, back.len); + try std.testing.expect(std.mem.allEqual(u8, back, 'y')); + p.deinit(); + try std.testing.expectEqual(@as(usize, 0), e.live.count()); +} + +test "a file's location rows are pinned whole, with what they own, and come back" { + const locations = @import("locations.zig"); + var e: Epoch = .{ .child = std.testing.allocator }; + defer e.deinit(); + const gpa = e.allocator(); + const p = try Pardes.init(gpa, .{ .cols = 80, .rows = 24, .file = "mise.toml", .image_allocator = gpa, .pdf_allocator = gpa, .tree_sitter_allocator = gpa, .frame_allocator = gpa }); + p.pump(.{}) catch {}; + const f = for (p.panes) |slot| { + if (slot) |pane| if (pane.file) |*file| break file; + } else return error.NoFile; + // A search's rows: each its own path. + const rows = try gpa.alloc(locations.Row, 500); + for (rows) |*row| row.* = .{ .kind = .match, .path = try gpa.dupe(u8, "src/rollback.zig"), .at = .{}, .location_end = 0, .code_start = 0 }; + f.location_rows = rows; + var c: Copy = .{}; + defer c.deinit(); + try c.take(&e, p); + try std.testing.expect(c.pinnedBytes() >= rows.len * @sizeOf(locations.Row)); + // A speculation's new search drops them, and is undone. + locations.freeRows(gpa, f.location_rows); + f.location_rows = &.{}; + c.restore(); + try std.testing.expectEqual(@as(usize, 500), f.location_rows.len); + try std.testing.expectEqualStrings("src/rollback.zig", f.location_rows[499].path); + p.deinit(); + try std.testing.expectEqual(@as(usize, 0), e.live.count()); +} diff --git a/src/views.zig b/src/views.zig index bf9b9147..b9760143 100644 --- a/src/views.zig +++ b/src/views.zig @@ -215,6 +215,7 @@ pub fn undumped(p: *Pardes, others: []const Dumped) !void { /// for one client (a Restore, `-l`) is one view. pub fn only(p: *Pardes) void { for (0..max) |k| if (k != p.view) close(p, @intCast(k)); + emptied(p, p.view); const k = p.view; if (k == 0) return; std.mem.swap(?*Parked, &p.parked[0], &p.parked[k]); @@ -390,6 +391,19 @@ pub fn to(p: *Pardes, k: u8) void { if (k == p.view) return; park(p); unpark(p, k); + emptied(p, k); +} + +/// View `k` is live: its slot, stale until it parks again (written whole +/// then), keeps no pointer to what it handed the live fields. A rollback +/// copy walks every slot (src/rollback.zig): one left there, freed by the +/// live view and the memory another's since, would be saved and its old +/// bytes put back over the other's. (A pane's share holds no pointer but +/// its selections, which `unpark` hands over.) +fn emptied(p: *Pardes, k: u8) void { + const slot = p.parked[k] orelse return; + slot.view = .{ .screen_w = p.screen_w, .screen_h = p.screen_h }; + slot.view.col_tags = @splat(.{ .what = .column, .gpa = p.gpa }); } /// A new view, as the live one is laid out and focused, with its panes @@ -431,6 +445,7 @@ pub fn close(p: *Pardes, k: u8) void { } else unreachable; park(p); unpark(p, other); + emptied(p, other); } const v = &p.parked[k].?.view; for (&v.col_tags) |*t| t.deinit(p.gpa); @@ -478,6 +493,7 @@ pub fn drawOthers(p: *Pardes, h: pardes.Host) !void { park(p); defer { copy(&fields, p, &p.parked[live].?.view); + emptied(p, live); p.view = live; for (p.panes) |s| if (s) |pane| { const pv = &pane.pv[live]; -- cgit v1.3 From f264dbc4a646da695331fe7401fa013cbb720b6a Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 00:25:19 -0300 Subject: the walker's digest: a blob's hash made once and folded in as a word (a 100 MB file was 5 ms a digest, two a unit, on the authority); blob fields annotated `.blob`, which the rollback copy's pins read too walk.zig: a field named in a type's `replica.blob` (a file's text, line index and location rows, an undo snapshot's text, a register's values and joined text, the rows an edit dropped, a query's saved rows) holds immutable blobs. A digest hashes one of 4 KiB or more once, keeps the word by its address and length, and folds the length and word in; `forget` drops the word, from memory.freeRaw and freeUnfilled, through which every such blob is freed raw. Only a Debug build (the tests, the harness, replay) hashes the bytes at every digest all the same and panics where the kept word is not their hash: a blob written in place, or freed past its word. A release build, ReleaseSafe the user's own as well as ReleaseFast, takes the word and never panics on a stale one: at worst a divergence, which a replica records and then stops at (nothing resets or rejoins it yet; the notes say so). File.zig: an edit's free of the old text went to `rawFree` itself and so kept its word; the next text of the same length at the same address folded in the old one (the ReleaseFast harness diverged in 162 of 180 replica runs, in `file.content`), and it goes through freeUnfilled now. Registers.zig: `put` makes a new values array rather than writing the old one. rollback.zig: the copy pins what `.blob` names (`pinned` and `pinnedDeep` gone). Measured, ReleaseFast on the perf cores: with a 100 MB file open (PARDES_DIGEST_BENCH=100, replica.zig's bench), a unit of a key into it went from p50 10.45 ms, p99 41.6 ms to p50 24 us, p99 6.1 ms (the first, which makes the word), a digest from 5.18 ms to 10 us. The harness (20 seeds x 120 inputs x 3 clients): rollback p99 9.5 -> 2.6 ms at 30 ms (1.5 and 1.4 ms at 80 and 207), copy p99 0.27 ms; none diverged, none unequal, no delta wrong. Tests: a blob hashed once, its kept word no longer its hash once written in place, its word gone when freed. docs/next-daemon.md: the follow-up. Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 44 +++++++++++++++++ src/File.zig | 12 +++-- src/Query.zig | 1 + src/Registers.zig | 38 +++++++------- src/memory.zig | 10 +++- src/replica.zig | 51 +++++++++++++++++++ src/rollback.zig | 36 ++------------ src/walk.zig | 139 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 8 files changed, 277 insertions(+), 54 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 55921027..681a2ed4 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1825,6 +1825,50 @@ Follow-ups (2026-10-29): one leaves none. The gate's baselines are recorded again from base (main and the new perf.zig), and .scratch/daemon/base/pardes-perf-live rebuilt from it. +- **A blob's digest is a word, made once.** The walker's digest, taken at + every unit a replica is sent and every unit it replays, hashed every blob + whole each time: a 100 MB file was 5 ms a digest, two digests a unit, on + the user's own session. A blob (a field annotated `.blob`: a file's text + and line index, an undo snapshot, a register's values and joined text, a + file's location rows, the ones an edit dropped, a query's saved) is now + hashed once, its word kept by its address (walk.zig), and the digest + folds that word in. The rollback copy pins what the same annotation + names (`pinned` and `pinnedDeep` are gone). + - **Exact while every free drops the word:** every such blob is freed + raw, through `memory.freeRaw` or `freeUnfilled` (the pins needed raw + frees), and those drop its word, so an address used again is hashed + again. The first ReleaseFast harness run found one that did not: an + edit freed the old text with `rawFree` itself, the next text of the + same length came back at the same address, and the old word was folded + in (162 of 180 replica runs diverged, in `file.content`, named with + PARDES_REPLICA_TRACE); it goes through `freeUnfilled` now. A replica's + blobs born under a copy are never digested (a speculation records + none), so the epoch's own frees leave no word behind. A register's + values array was the one written in place; `put` now makes a new one. + ponytail: the word is kept by address, so a new free path that skips + those two misses it; a word carried in the struct beside its blob if + that bites again. + - **Written in place, or freed past its word:** a Debug build hashes a + blob's bytes at every digest and folds those in, and panics where the + word kept at that address and length is not their hash: a blob written + in place (which the pins forbid too) or freed without dropping its + word. The tests, the replicas harness and replay run Debug and go + through it (and pass); walk.zig's test writes a blob in place and sees + its kept word is no longer its hash. A release build (ReleaseSafe, the + user's own, and ReleaseFast) takes the word and never panics on a stale + one: at worst it shows as a digest divergence. Not recovered yet: a + replica that diverges records why (`Replica.diverged`) and takes no + more units; nothing resets it or joins it again. That is the remote + client's to add (a fresh join from the authority's dump), not a crash. + - Measured (ReleaseFast, alone on the perf cores; PARDES_DIGEST_BENCH=100 + zig build core-test -Dtest-filter="big file open"): with a 100 MB file + open, a unit of a key into it (two digests) went from p50 10.45 ms, + p99 41.6 ms to p50 24 us, p99 6.1 ms (the first, which makes the + file's word), and a digest from 5.18 ms to 10 us. The harness (20 seeds + x 120 inputs x 3 clients, ReleaseFast): rollback p99 at 30 ms went from + 9.5 to 2.6 ms (1.5 and 1.4 ms at 80 and 207 ms, about as before), copy + p99 0.27 ms (0.22 before, within the run's noise); none diverged, none + unequal, no delta wrong. ## Phase 6: upgrade notes diff --git a/src/File.zig b/src/File.zig index 48a996a4..7acf8dd6 100644 --- a/src/File.zig +++ b/src/File.zig @@ -30,6 +30,7 @@ const long_paint: usize = 16 << 10; /// Content and primary selection at one file edit boundary. Keeping only the /// primary avoids putting Text.max_selections ranges in every history entry. pub const Snapshot = struct { + pub const replica = .{ .blob = .{"content"} }; content: []u8, cur_row: i32, cur_col: i32, @@ -139,7 +140,11 @@ pub const History = struct { pub const State = struct { /// What the replay digest leaves out (src/walk.zig): what is made again /// from the text whenever it is wanted. - pub const replica = .{ .derived = .{ "line_starts", "highlights", "highlight_start", "syntax_dirty", "node_tree", "node_tree_revision", "node_history", "tables" } }; + pub const replica = .{ + .derived = .{ "line_starts", "highlights", "highlight_start", "syntax_dirty", "node_tree", "node_tree_revision", "node_history", "tables" }, + // Replaced whole at every edit (setContentSpan), never written. + .blob = .{ "content", "line_starts", "location_rows" }, + }; path: []u8, content: []u8, @@ -840,6 +845,7 @@ fn reportEdit(p: *Pardes, f: *State, new: []const u8) void { } pub const DroppedMarks = struct { + pub const replica = .{ .blob = .{"rows"} }; len: usize, hash: u64, rows: []locations.Row, @@ -908,8 +914,8 @@ pub fn setContentSpan(p: *Pardes, f: *State, new: []u8, span_in: ?Span) void { f.saved_len = f.content.len; } // Raw: `free` overwrites a safe build's freed memory, a whole file's - // worth per keystroke. - if (f.content.len > 0) p.gpa.rawFree(f.content, .of(u8), @returnAddress()); + // worth per keystroke; and its digest word goes (src/walk.zig). + pardes.memory.freeUnfilled(p.gpa, f.content); f.content = new; f.revision +%= 1; f.splice = .{ .revision = f.revision, .head = known.head, .tail = known.tail, .old_len = old_len }; diff --git a/src/Query.zig b/src/Query.zig index 8296aa80..49f58a0e 100644 --- a/src/Query.zig +++ b/src/Query.zig @@ -173,6 +173,7 @@ capped: bool = false, saved: ?Saved = null, pub const Saved = struct { + pub const replica = .{ .blob = .{"rows"} }; content: []u8, rows: []locations.Row, asked: []u8, diff --git a/src/Registers.zig b/src/Registers.zig index 760aff9a..6df7401d 100644 --- a/src/Registers.zig +++ b/src/Registers.zig @@ -12,11 +12,12 @@ const Registers = @This(); list: std.ArrayList(Reg) = .empty, last_search: u21 = '/', -/// A value and the joined text are replaced whole, never written in place, -/// so a replica's rollback copy pins them as it does a file's text -/// (src/rollback.zig `pinned`), and they are freed raw: a safe build's -/// free would write over one the copy still holds. +/// The values (their array too) and the joined text are replaced whole, +/// never written in place (`put` makes a new array), freed raw: blobs +/// (src/walk.zig), which a digest hashes once and a replica's rollback copy +/// pins as it does a file's text. pub const Reg = struct { + pub const replica = .{ .blob = .{ "values", "joined" } }; name: u21, /// gpa-owned, one per range, in document order values: [][]u8, @@ -36,7 +37,7 @@ pub fn deinit(r: *Registers, gpa: std.mem.Allocator) void { fn freeReg(gpa: std.mem.Allocator, reg: *Reg) void { for (reg.values) |v| freeRaw(gpa, v); - gpa.free(reg.values); + freeRaw(gpa, reg.values); if (reg.joined) |j| freeRaw(gpa, j); } @@ -65,19 +66,20 @@ pub fn put(r: *Registers, gpa: std.mem.Allocator, name: u21, bytes: []const u8, }; break :blk &r.list.items[r.list.items.len - 1]; }; - if (fresh or reg.values.len != n) { - const values = gpa.alloc([]u8, n) catch { - gpa.free(copy); - return; - }; - for (values) |*v| v.* = &.{}; - freeReg(gpa, reg); - reg.* = .{ .name = name, .values = values }; - } - freeRaw(gpa, reg.values[i]); - reg.values[i] = copy; - if (reg.joined) |j| freeRaw(gpa, j); - reg.joined = null; + // A new array, the old one's values moved into it but value `i`. + const values = gpa.alloc([]u8, n) catch { + gpa.free(copy); + return; + }; + const keep = !fresh and reg.values.len == n; + for (values, 0..) |*v, j| v.* = if (keep and j != i) reg.values[j] else &.{}; + values[i] = copy; + if (keep) { + freeRaw(gpa, reg.values[i]); + freeRaw(gpa, reg.values); + if (reg.joined) |j| freeRaw(gpa, j); + } else freeReg(gpa, reg); + reg.* = .{ .name = name, .values = values }; } /// The whole register as one text: its values joined by newlines, no diff --git a/src/memory.zig b/src/memory.zig index 1d082964..7adcf914 100644 --- a/src/memory.zig +++ b/src/memory.zig @@ -1,5 +1,6 @@ const std = @import("std"); const builtin = @import("builtin"); +const walk = @import("walk.zig"); const config = @import("pardes_config"); const board = config.platform == .esp32p4; @@ -418,7 +419,10 @@ pub fn dupeUnfilled(allocator: Allocator, bytes: []const u8) Allocator.Error![]u /// Frees without the safe-build overwrite `free` does, for text-sized buffers /// freed on every edit. pub fn freeUnfilled(allocator: Allocator, bytes: []const u8) void { - if (bytes.len > 0) allocator.rawFree(@constCast(bytes), .of(u8), @returnAddress()); + if (bytes.len == 0) return; + // A blob's digest word goes with it (walk.zig). + walk.forget(@intFromPtr(bytes.ptr)); + allocator.rawFree(@constCast(bytes), .of(u8), @returnAddress()); } /// `freeUnfilled` for a slice of any element: a text's line index, made @@ -427,7 +431,9 @@ pub fn freeUnfilled(allocator: Allocator, bytes: []const u8) void { pub fn freeRaw(allocator: Allocator, items: anytype) void { const E = @typeInfo(@TypeOf(items)).pointer.child; const bytes = std.mem.sliceAsBytes(items); - if (bytes.len > 0) allocator.rawFree(@constCast(bytes), .of(E), @returnAddress()); + if (bytes.len == 0) return; + walk.forget(@intFromPtr(bytes.ptr)); + allocator.rawFree(@constCast(bytes), .of(E), @returnAddress()); } /// Gives free memory back to the system, the arenas' and libc's, inside its diff --git a/src/replica.zig b/src/replica.zig index a96a056a..0b55a5bf 100644 --- a/src/replica.zig +++ b/src/replica.zig @@ -1011,3 +1011,54 @@ test "a delta applied under a speculation outlives its rollback: the terminal st const shown = a.core.panes[shell].?.rows; for (rows_a[rows_a.len - shown ..], rows_r[rows_r.len - shown ..]) |x, y| try std.testing.expectEqualStrings(x, y); } + +/// p50 and p99 of `samples`, sorted in place. +fn spread(samples: []u64) [2]u64 { + std.mem.sort(u64, samples, {}, std.sort.asc(u64)); + return .{ samples[samples.len / 2], samples[@min(samples.len - 1, samples.len * 99 / 100)] }; +} + +test "the authority's units with a big file open: its blobs hashed once (PARDES_DIGEST_BENCH=)" { + const arg = std.c.getenv("PARDES_DIGEST_BENCH") orelse return error.SkipZigTest; + const mb = std.fmt.parseInt(usize, std.mem.span(arg), 10) catch 100; + const gpa = std.heap.c_allocator; + const tmp = std.mem.span(std.c.getenv("TMPDIR") orelse "/tmp"); + var path_buf: [256]u8 = undefined; + const path = try std.fmt.bufPrintSentinel(&path_buf, "{s}/pardes-digest-bench.txt", .{tmp}, 0); + { + const fd = std.c.open(path, .{ .ACCMODE = .WRONLY, .CREAT = true, .TRUNC = true }, @as(std.c.mode_t, 0o644)); + if (fd < 0) return error.OpenFailed; + defer _ = std.c.close(fd); + var line: [80]u8 = undefined; + var n: usize = 0; + var i: usize = 0; + while (n < mb << 20) : (i += 1) { + const text = try std.fmt.bufPrint(&line, "line {d:0>9} of a big file, as a log or a dump is\n", .{i}); + if (std.c.write(fd, text.ptr, text.len) != @as(isize, @intCast(text.len))) return error.WriteFailed; + n += text.len; + } + } + defer _ = std.c.unlink(path); + const a = try Authority.init(gpa, .{ .cols = 120, .rows = 40, .file = path }, TestHost.host(), true); + defer a.deinit(); + _ = try a.unit(0, 0); + var unit_ns: [40]u64 = undefined; + var digest_ns: [40]u64 = undefined; + for ([_]bool{ false, true }) |on| { + walk.blob_words_on = on; + defer walk.blob_words_on = true; + walk.forgetAll(); + for (&unit_ns, &digest_ns, 0..) |*u, *d, k| { + // A key into the file: a unit as a replica is sent it. + const t0 = nowNs(); + _ = try a.local(.{ .key = .{ .cp = if (k % 2 == 0) 'j' else 'k', .text = if (k % 2 == 0) "j" else "k" } }); + u.* = nowNs() - t0; + const t1 = nowNs(); + _ = digestOf(a); + d.* = nowNs() - t1; + } + const us = spread(&unit_ns); + const ds = spread(&digest_ns); + std.debug.print("{d} MB file, blobs {s}: a unit p50 {d} us p99 {d} us; a digest p50 {d} us p99 {d} us\n", .{ mb, if (on) "hashed once" else "hashed whole", us[0] / 1000, us[1] / 1000, ds[0] / 1000, ds[1] / 1000 }); + } +} diff --git a/src/rollback.zig b/src/rollback.zig index a7027da6..37fc481d 100644 --- a/src/rollback.zig +++ b/src/rollback.zig @@ -35,8 +35,8 @@ const Allocator = std.mem.Allocator; const Alignment = std.mem.Alignment; const Pardes = pardes.Pardes; -/// Blobs this long or longer (`pinned`) are pinned, not copied. Checked -/// unchanged in a safe build when the copy goes. +/// Blobs (walk.blob: fields annotated `.blob`) this long or longer are +/// pinned, not copied. Checked unchanged in a safe build when the copy goes. const blob_min = 4096; const checked = builtin.mode == .debug or builtin.mode == .safe; @@ -198,8 +198,6 @@ pub const Copy = struct { /// of the blobs: a file's text or an undo snapshot's. where: []const u8 = "", pinnable: bool = false, - /// Whether it is a blob of rows that own allocations of their own. - deep: bool = false, const Region = struct { addr: usize, off: usize, len: usize }; const Pin = struct { addr: usize, len: usize, hash: u64, where: []const u8 }; @@ -344,8 +342,7 @@ pub const Copy = struct { if (attr.@"comptime" or comptime (annotated(T, "authority", name) or ((annotated(T, "local", name) or annotated(T, "derived", name)) and unsaved(name)))) continue; if (comptime opaqueTo(F)) continue; c.where = @typeName(T) ++ "." ++ name; - c.pinnable = comptime pinned(T, name); - c.deep = comptime pinnedDeep(T, name); + c.pinnable = comptime walk.blob(T, name); if (comptime walk.lengthOf(T, name) != null and @typeInfo(F) == .array) { const len = comptime walk.lengthOf(T, name).?; if (comptime heapIn(@typeInfo(F).array.child, 0)) for (@field(v, name)[0..walk.lengthIn(v, len)]) |*e| try c.visit(@typeInfo(F).array.child, e); @@ -398,9 +395,8 @@ pub const Copy = struct { if (s.len == 0) return; const addr = @intFromPtr(s.ptr); if (try c.reached([]const E, addr)) return; - const flat = comptime !heapIn(E, 0); - if ((if (flat) c.pinnable else c.deep) and s.len * @sizeOf(E) >= blob_min and c.epoch.?.owns(addr) != null) { - // A blob of rows with allocations of their own pinned whole: + if (c.pinnable and s.len * @sizeOf(E) >= blob_min and c.epoch.?.owns(addr) != null) { + // A blob of rows with allocations of their own is pinned whole: // they are as immutable as it, and kept alive with it. const bytes = std.mem.sliceAsBytes(s); try c.pins.append(c.meta, .{ .addr = addr, .len = bytes.len, .hash = if (checked) std.hash.Wyhash.hash(0, bytes) else 0, .where = c.where }); @@ -411,28 +407,6 @@ pub const Copy = struct { } }; -/// The blobs: what an edit puts in place of its predecessor, never writes -/// (File.setContentSpan), and frees raw (memory.freeUnfilled, freeRaw). -fn pinned(comptime T: type, comptime name: []const u8) bool { - const File = pardes.panes.File; - return (T == File.State and (std.mem.eql(u8, name, "content") or std.mem.eql(u8, name, "line_starts"))) or - (T == File.Snapshot and std.mem.eql(u8, name, "content")) or - // A register's values and joined text: a yank of a terminal's whole - // grid is megabytes, which every speculation would otherwise copy. - (T == pardes.Registers.Reg and (std.mem.eql(u8, name, "values") or std.mem.eql(u8, name, "joined"))); -} - -/// Blobs of rows that own allocations of their own (a location's path, its -/// colours), the array and they replaced whole and never written, freed raw -/// (locations.freeRows): a file's location rows (5 MB after a flood of -/// search results), the ones an edit dropped, a query's saved. -fn pinnedDeep(comptime T: type, comptime name: []const u8) bool { - const File = pardes.panes.File; - return (T == File.State and std.mem.eql(u8, name, "location_rows")) or - (T == File.DroppedMarks and std.mem.eql(u8, name, "rows")) or - (T == pardes.panes.Query.Saved and std.mem.eql(u8, name, "rows")); -} - fn annotatedStruct(comptime T: type) bool { return @typeInfo(T) == .@"struct" and @hasDecl(T, "replica"); } diff --git a/src/walk.zig b/src/walk.zig index 5060de3d..33247d42 100644 --- a/src/walk.zig +++ b/src/walk.zig @@ -23,6 +23,10 @@ //! .len = .{ .{ "jumps", "njumps" } }, //! // What a field holds is made with another allocator on a load. //! .alloc = .{ .{ "image", "image" } }, +//! // Immutable blobs: replaced whole, never written, freed raw +//! // (memory.freeRaw, freeUnfilled). A digest hashes each once and +//! // folds that word in; a replica's rollback copy pins them. +//! .blob = .{ "content" }, //! }; //! //! None of the three kinds is in a digest or a dump. A type with a `pub fn @@ -31,6 +35,100 @@ const std = @import("std"); pub const Hasher = std.hash.Wyhash; +const builtin = @import("builtin"); +/// Only a Debug build hashes a blob's bytes at every digest and checks its +/// kept word; a release one, ReleaseSafe the user's own, takes the word. +const rehash = builtin.mode == .debug; + +/// Whether `T.name` holds immutable blobs (`.blob`): a file's text, an undo +/// snapshot, a register's text, a file's location rows. +pub fn blob(comptime T: type, comptime name: []const u8) bool { + @setEvalBranchQuota(100_000); + if (!@hasDecl(T, "replica") or !@hasField(@TypeOf(T.replica), "blob")) return false; + inline for (T.replica.blob) |n| if (comptime std.mem.eql(u8, n, name)) return true; + return false; +} + +// ---- a blob's hash, made once ---- +// +// A digest walks the whole core at every unit a replica is sent (and every +// unit a replica replays): a 100 MB file was hashed in full each time. A +// blob of `blob_min` bytes or more is hashed once instead, its word kept by +// its address, and the digest folds the word in. The word goes when the +// blob is freed (`forget`, from memory.freeRaw and freeUnfilled: every +// such blob is freed raw, for the rollback copy's pins), so an address +// used again is hashed again. A Debug build hashes the bytes all the same, +// folds those in, and stops where a kept word is not their hash: a blob +// written in place (which the pins forbid too), or freed past `forget`. A +// release build never stops there: a stale word shows as a divergence. + +const blob_min = 4096; +var blob_words: std.AutoHashMapUnmanaged(usize, struct { len: usize, word: u64 }) = .empty; +var blob_lock: std.atomic.Value(bool) = .init(false); +var blob_count: std.atomic.Value(usize) = .init(0); +threadlocal var in_blob: bool = false; +/// Off: every blob hashed whole at every digest, as before (the bench's). +pub var blob_words_on = true; + +fn lockBlobs() void { + while (blob_lock.cmpxchgWeak(false, true, .acquire, .monotonic) != null) std.atomic.spinLoopHint(); +} + +fn unlockBlobs() void { + blob_lock.store(false, .release); +} + +/// The blob at `addr` is freed: its word, if it has one, goes with it. +pub fn forget(addr: usize) void { + if (blob_count.load(.acquire) == 0) return; + lockBlobs(); + defer unlockBlobs(); + if (blob_words.remove(addr)) blob_count.store(blob_words.count(), .release); +} + +/// The word kept for the blob at `addr` (a test's). +pub fn wordOf(addr: usize) ?u64 { + lockBlobs(); + defer unlockBlobs(); + return if (blob_words.get(addr)) |b| b.word else null; +} + +/// Every word forgotten (the bench's cold digests). +pub fn forgetAll() void { + lockBlobs(); + defer unlockBlobs(); + blob_words.clearRetainingCapacity(); + blob_count.store(0, .release); +} + +/// A blob as a digest takes it: its length and its word. +fn hashBlob(comptime E: type, items: []const E, h: *Hasher) void { + const addr = @intFromPtr(items.ptr); + const word: u64 = word: { + const kept: ?u64 = if (blob_words_on) kept: { + lockBlobs(); + defer unlockBlobs(); + const b = blob_words.get(addr) orelse break :kept null; + break :kept if (b.len == items.len) b.word else null; + } else null; + if (!rehash) if (kept) |w| break :word w; + in_blob = false; + defer in_blob = true; + var one = Hasher.init(0); + hashSlice(E, items, &one); + const w = one.final(); + if (kept) |k| if (k != w) std.debug.panic("walk: the {d}-byte blob at 0x{x} is not what its kept word was: written in place, or freed without memory.freeRaw", .{ items.len, addr }); + if (blob_words_on and kept == null) { + lockBlobs(); + defer unlockBlobs(); + blob_words.put(std.heap.page_allocator, addr, .{ .len = items.len, .word = w }) catch {}; + blob_count.store(blob_words.count(), .release); + } + break :word w; + }; + h.update(std.mem.asBytes(&items.len)); + h.update(std.mem.asBytes(&word)); +} pub fn digest(comptime T: type, v: *const T) u64 { var h = Hasher.init(0); @@ -46,6 +144,9 @@ pub fn fieldDigests(comptime T: type, v: *const T, out: []u64) []u64 { inline for (s.field_names, s.field_types, s.field_attrs) |name, F, attr| { if (attr.@"comptime" or comptime excluded(T, name)) continue; var h = Hasher.init(0); + const was = in_blob; + in_blob = comptime blob(T, name); + defer in_blob = was; if (comptime lengthOf(T, name)) |len| { hashSlice(Element(F), @field(v, name)[0..lengthIn(v, len)], &h); } else hash(F, &@field(v, name), &h); @@ -180,6 +281,9 @@ pub fn hash(comptime T: type, v: *const T, h: *Hasher) void { inline for (s.field_names, s.field_types, s.field_attrs) |name, F, attr| { if (attr.@"comptime" or comptime excluded(T, name)) continue; if (comptime opaqueTo(F)) @compileError("walk: " ++ @typeName(T) ++ "." ++ name ++ " is " ++ @typeName(F) ++ ", which no walk sees into: say what it is (`replica`)"); + const was = in_blob; + in_blob = comptime blob(T, name); + defer in_blob = was; if (comptime lengthOf(T, name)) |len| { hashSlice(Element(F), @field(v, name)[0..lengthIn(v, len)], h); } else hash(F, &@field(v, name), h); @@ -214,6 +318,7 @@ fn hashInt(comptime I: type, n: I, h: *Hasher) void { } fn hashSlice(comptime E: type, items: []const E, h: *Hasher) void { + if (in_blob and items.len * @sizeOf(E) >= blob_min) return hashBlob(E, items, h); h.update(std.mem.asBytes(&items.len)); if (comptime flat(E)) return h.update(std.mem.sliceAsBytes(items)); for (items) |*e| hash(E, e, h); @@ -1584,3 +1689,37 @@ test "a dump that is not one, or is cut short, is refused" { // What it made before the end is freed again. try std.testing.expectError(error.EndOfDump, load(S, gpa, bytes[0 .. bytes.len - 2], null)); } + +test "a blob is hashed once and folded in as a word; one written in place is not its word, which a Debug build checks; a freed one's word goes" { + const memory = @import("memory.zig"); + const digestOfBlob = struct { + fn of(items: []const u8) u64 { + var h = Hasher.init(0); + hashSlice(u8, items, &h); + return h.final(); + } + }.of; + const gpa = std.testing.allocator; + const File = struct { + pub const replica = .{ .blob = .{"content"} }; + content: []u8, + cursor: u32 = 0, + }; + const text = try gpa.alloc(u8, 64 << 10); + @memset(text, 'a'); + var f: File = .{ .content = text }; + const before = digest(File, &f); + try std.testing.expect(wordOf(@intFromPtr(text.ptr)) != null); + // The same word again: nothing it holds moved. + try std.testing.expectEqual(before, digest(File, &f)); + // Written in place, which a blob never is: its kept word is no longer + // its hash, which a Debug build's digest checks (and panics at) where a + // release build's would fold the old word in. + text[1000] = 'b'; + try std.testing.expect(wordOf(@intFromPtr(text.ptr)).? != digestOfBlob(text)); + text[1000] = 'a'; + try std.testing.expectEqual(digestOfBlob(text), wordOf(@intFromPtr(text.ptr)).?); + // Freed, its word goes: an address used again is hashed again. + memory.freeUnfilled(gpa, text); + try std.testing.expect(wordOf(@intFromPtr(text.ptr)) == null); +} -- cgit v1.3 From 965a78eafacbffb7cd25104181c81fe5871102c1 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 00:01:29 -0300 Subject: docs/next-daemon.md: phase 6's 9P round trips: the decisions at its forks (the server's order kept, what a park lets go ahead and how the client meets it, the flight, the end of a read, read ahead, kept connections, no stat cache, /n/ as ssh -L and Mount), messages, round trips and times before and after at 30/80/207 ms and on the FreeBSD box (plain and ssh -C), what the box's link costs beyond the round trips, the local A/B, the gates, and what is not done Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 183 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 681a2ed4..0fe72b76 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2129,3 +2129,186 @@ Not done, and why: of the older build keep talking theirs. - A session name with a newline would break the daemon's registry line. - An upgraded process records no tape (a recording ends at the upgrade). + +## Phase 6: 9P round trips notes + +Built 2026-10-30 (`src/9p_io.zig` Client, `src/fs.zig`, the Mount word in +`src/builtins.zig`, a paragraph of `docs/typ/reference.typ`). pardes's own +9P client is what mounts, the Mount word, the forwarding CLI, `--wait` +and Handover use; scripting clients (9p(1), 9ns, test/ninep.py) are not +changed here. Decisions at the forks, with the ponytail lens: + +- **The server already takes a connection's requests in order; nothing + there changes.** cloud9's engine has one job slot: it starts the next + frame only once the request before it is answered. What parks leaves + that slot, and the frames after it go ahead: a read with nothing yet, + and, while the editor is out in a syscall, a write or a truncating + open (`tree.needsQuiet`; an OTRUNC open is a setattr first). So a walk + and the open of its fid can be sent together (a walk never parks), and + so can an open and its first read (an OREAD open never parks). Two + things cannot: + - a Twrite behind a truncating Topen that parks is refused `bad use of + fid` (not open yet); + - a Tclunk behind a parked Twrite interrupts it. + + The client therefore sends a write or read that came back `bad use of + fid` again after its open's answer: that refusal is the engine's, made + before the backend, so nothing was done. It sends its Tclunk only after + the last answer, and does not wait for the Rclunk. The park test ("a + change waits while the editor is out mid-step") takes the resend path. +- **One flight per operation.** The flight is `[Tversion Tattach] Twalk + [Twalk] Tstat Topen Tread` for a read, or `... Topen Twrite` for a + write, each request with its own tag (`Flight`, `Session.land`). Past 16 + names, the second walk goes from the fid to itself, so no fid is in + between. +- **Version and attach pipelined by a poke.** cloud9's client submits + nothing while a Tversion is out. `greet` sets its `versioning` and + `msize` around the submits, taking the asked msize until the Rversion + lands. Only a pardes is dialled here, and every pardes offers + `ninep.msize`. A write in that first flight is kept under + `min_msize - 23`. The fix belongs in cloud9's client (taking requests + while it versions). It is not made here, since this round pushes + nothing (`// ponytail:` at `greet`). +- **The end of a read.** 9P has no end of file but an empty read. + read(5) says a reply "may be less than the requested amount", and a + directory's next offset must be the last plus what it returned, so + neither the end nor a directory's next read can be sent ahead. The + Tstat in the flight gives a file's length: a read that comes back short + at that length is the end. pardes reports real lengths for bodies, + `/os` files, ctl, tag and the like. A file of no stated length, or a + directory, costs one round trip more: index, layout, screen and log are + generated whole by each read and say 0 by design (tree.zig `topSize`). + An end flag would be a protocol extension, which is not worth one round + trip on generated files. +- **Read ahead.** With the length known, the Treads for the rest go at + once, 14 at a time (the 16-tag window, less a clunk still out and a + spare). A 1.3 MB file takes 3 round trips where it took 22. A file of + no stated length is not read ahead (`// ponytail:`). +- **Reads are timed by their progress.** Each Rread gives the rest the + 2 s budget again, as each Twrite already did. Found on the way: on main, + every read of one operation shared one 2 s budget, so a remote file of + 1.3 MB failed to open (`dial failed: timed out`) at 80 and 207 ms. On + the box without `-C`, the 77 KB `src/fs.zig` failed too. +- **A mount keeps its connection** (`kept`), as Plan 9's mount keeps its + channel: an operation through it is the flight alone, with no dial, + version or attach. + - There are eight slots, keyed by dial. A session is taken out of the + table while it is in use, so two users of one dial each get their + own. + - Unmount forgets it (`Client.forget`). + - Before reuse, a poll takes what arrived while it waited (a Rclunk + owed); an EOF, or anything else, means the peer let go, and a new + connection is dialled. + - A read on a kept connection that hangs up all the same is asked once + more on a new one. A write is not, since it may have been done. + - Only `readMount`, `writeMount` and `probeMount` (the Mount word) + keep one, since each costs the peer one of its 16 connections while + kept. The CLI, `--wait` and Handover dial afresh as before, with + their version, attach and walk pipelined. +- **No qid or stat cache.** The trace shows that a Look through a mount + is one operation with one walk. With walk, stat, open and read in one + flight, a cache would save bytes but no round trip, and a cached stat + is wrong once the peer changes the file. +- **msize and iounit stay.** They are already 64 KiB, with iounit = + msize - 24, so a body of up to 65,525 bytes is one Tread. A larger + msize would cost each of a session's 16 connections five buffers of + it. test/ninep.py asks 8192 by its own choice, so its body reads of + more than that cost it more messages. +- **`/n/`.** The daemon serves no 9P yet (phase 3), so there is no + remote daemon to mount. A remote session mounts today with two + commands, now in reference.typ: `ssh -C -N -L /tmp/box.sock: host`, then `Mount box /tmp/box.sock`. A relay in the daemon + that lists a host's sessions under one mount waits for the daemon to + serve 9P. `-C` is there because the box's link is slow in that + direction (below). + +Measured. The harness is `.scratch/daemon/ninep/rt.py`. It runs a peer +session and a local one that Mounts it through a relay. The relay holds +each chunk a fixed delay after it arrives, as test/delay_relay.py does, +and it also splits the stream into 9P frames, counting the T-messages +and the rounds (runs of T-messages with no answer delivered between +them). `box.sh` runs the same harness against a detached session on the +box through `ssh -L`. Builds are ReleaseFast (main ymwmwrvx against this +line), 3 reps each, medians. + +The RTTs column counts the round trips the client waited on. pardes's +own client sends its last Tclunk without waiting, before and after; the +python client waits on every message. The box columns are `ssh -L` at +207 ms, plain and with `ssh -C`; "fails" means `dial failed: timed out`. + +| operation (pardes's client) | msgs | RTTs | 30 ms | 80 ms | 207 ms | box | box `-C` | +|---|---|---|---|---|---|---|---| +| connect: `Mount` (version, attach, root read) | 5 → 7 | 4 → 1 | 124 → 33 | 324 → 81 | 831 → 208 | 1032 → 426 | 1024 → 410 | +| file read: Look `/n/peer/index` (no length) | 7 → 6 | 6 → 2 | 185 → 61 | 486 → 161 | 1247 → 415 | 1441 → 426 | 1434 → 409 | +| directory listing: Look `/n/peer/pane/` | 7 → 6 | 6 → 2 | 185 → 62 | 483 → 161 | 1247 → 415 | 1437 → 427 | 1430 → 410 | +| ctl write: Save to `/n/peer/pane/1/ctl` | 6 → 4 | 5 → 1 | 155 → 32 | 402 → 81 | 1039 → 208 | 1237 → 214 | 1233 → 206 | +| pane body read: Look `/n/peer/pane/1/body` (30 KB) | 7 → 5 | 6 → 1 | 186 → 33 | 483 → 81 | 1247 → 208 | 1843 → 641 | 1432 → 206 | +| a remote file open: Look `/n/peer/os/.../fs.zig` (77 KB) | 8 → 6 | 7 → 2 | 213 → 64 | 563 → 162 | 1454 → 416 | fails → 1300 | 1844 → 617 | +| a remote file open, 1.3 MB | 27 → 25 | 26 → 3 | 807 → 103 | fails → 244 | fails → 625 | fails → 5334 | fails → 1273 | + +| the python client (unchanged, for scale) | msgs, RTTs | 207 ms | box | +|---|---|---|---| +| connect | 2 | 415 | 614 to 635 (with ssh's channel open) | +| read `/index`, list `/pane` | 5 | 1037 | 1023 to 1067 | +| ctl write | 4 | 829 | 818 to 852 | +| body read, 30 KB | 5 | 1037 | 1024 (`-C`) to 1283 (plain) | + +- **Each operation is one round trip.** A directory, or a file of no + stated length, takes two. The first operation through a new `ssh -L` + forward pays one more, the channel opening (the Mount, 426 ms on the + box). +- **On the box, bytes cost more than the 9P rounds.** The link from the + box to here carries 170 to 400 KB/s through ssh. A 1 MiB `dd` takes + about 6 s and 4 MiB about 10 s, where 4 MiB up takes 2.4 s. A 64 KiB + answer arrives 1 to 2 RTTs after the request, so the 30 KB body reads + in 641 ms plain and 206 ms with `-C`. The 1.3 MB file's Rreads come in + at about one a round trip (the timeline is in `rt.py`'s `TIMELINE=1`). + `blast.py` measured all 21 of its Treads in flight at once at 1.9 to + 3.0 s, against 5.3 s for this client's 14 and then 7. A sliding window + instead of batches would buy that, but only for large files on this + link (not done). Compression helps every size: `-C` is in the + documented recipe, and the replica stream's zstd (phase 6) will face + the same link. + +No local regression. Local 9P has no delay. lab.py runs the base and +the new build at once with their samples alternating, 1000 each, under +the perf lock, and gives the p50 ratio new/base. The server is not +changed, so its rows are level. The mount rows are faster because they +dial nothing. + +| local, p50 | base | new | A/B | A/A | +|---|---|---|---|---| +| python client: read a 9 KB body (server path) | 88 µs | 89 µs | 1.004 | 1.000 | +| python client: ctl write `clean` (server path) | 80 µs | 81 µs | 1.011 | 0.992 | +| Look `/n/peer/pane/1/body` through a local mount | 329 µs | 255 µs | 0.776 | 1.009 | +| Save to `/n/peer/pane/1/ctl` through a local mount | 214 µs | 141 µs | 0.660 | 1.000 | + +The gates: +- test/fs.py, `zig build 9p-io-test`, unit-test tty, the snapshots and + test/daemon.py (20 cycles) pass. +- New: `a mount's connection is kept between operations, and one the + peer hung up is dialled again` (a write longer than a message on a + kept connection, a miss, a hangup, a redial and a forget). It also + passes on the box (FreeBSD test build). + +Not done, and why: +- **cloud9's client should take requests while it versions;** the poke + in `greet` stands in for that until cloud9 is pushed and re-pinned. + 9ns (cloud9's) still waits one round trip per message. +- **A write longer than one message** sends its chunks after the first + one at a time: a Save of a 1 MB file through a mount is 16 round + trips. Pipelined Twrites could run in the wrong order once one parks; + that does not matter for `/os` (pwrite at an offset) but would for + some pane files. +- **A file of no stated length** is not read ahead, and its end (like a + directory's) costs a round trip more. +- **The CLI's `ask`** (`pardes file` from a pane, `pardes -`) and + `--wait` pipeline only version, attach and walk. Their open, writes + and reads go one at a time, on a connection of their own. +- **A peer's Restore cuts its clients up to 200 ms after it starts.** A + request on a kept connection in that window is refused `Input/output + error`, where a new dial would already have been the replacement's. +- **Scripting from here to a remote session** through test/ninep.py or + 9p(1) is still one round trip per message (1 s for a read at 207 ms). + A python helper for remote scripting could pipeline the same way; + nobody scripts that way yet. -- cgit v1.3 From 3703c2526fb59708b00e12816bd88ffd8a15fc69 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 01:34:23 -0300 Subject: echo: a replica draws a key typed into a terminal at once, mosh's way (a printable character at the cursor, Backspace, Left and Right, in tty mode on the primary screen), from an overlay of its own that no digest, dump or rollback copy sees, painted over the terminal's body at the end of renderBody (the tty's and the GUI's); judged at each delta once the authority has the key and 50 ms have passed, every prediction taken back on a mismatch; drawn at once at a prompt pardes's marks vouch for, elsewhere only once an echo was confirmed, never after a drawn misprediction until one is, so a password prompt or a program that ignores input shows nothing typed; the keys typed behind one not predicted (Tab, Enter) predicted over the screen that shows its doing; zig build replicas -- --echo types a corpus into a played shell at 30/80/207 ms: a key to its character p50 239 -> 8.9 ms and p90 248 -> 15.8 ms at 207, no wrong character drawn; docs/next-daemon.md: the decisions, the hook for the remote client, the numbers, the gates and what is not done Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 232 ++++++++++++++++ src/body_layer.zig | 3 + src/echo.zig | 781 ++++++++++++++++++++++++++++++++++++++++++++++++++++ src/pardes.zig | 7 + src/replica.zig | 16 ++ src/rollback.zig | 5 +- src/terminal.zig | 2 + test/replicas.zig | 766 +++++++++++++++++++++++++++++++++++++++++++++++++-- 8 files changed, 1781 insertions(+), 31 deletions(-) create mode 100644 src/echo.zig (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 0fe72b76..5039c81e 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2312,3 +2312,235 @@ Not done, and why: 9p(1) is still one round trip per message (1 s for a read at 207 ms). A python helper for remote scripting could pipeline the same way; nobody scripts that way yet. + +## Phase 6: predicted echo notes + +Built 2026-10-30 (`src/echo.zig`; hooks in `src/replica.zig`; a field on +the core and one call at the end of `body_layer.renderBody`; `zig build +replicas -- --echo`). Transport-free like part B: proven in part A's +in-process harness, carried by the remote client. A key typed into a +terminal pane goes to the authority, and its echo comes back as a delta a +round trip later: 207 ms a character in a shell on the FreeBSD box. The +replica now draws it at once, mosh's way, and the authority's screen +confirms or takes it back. Decisions at the forks, with the ponytail lens: + +- **Presentation, the replica's own.** The overlay is `Replica.echo`; the + core points at it (`Pardes.echo`, `.local`, and in rollback.zig's + `unsaved`, so the copy neither saves nor puts it back). No digest, dump + or tape sees it, and the core's emulator stays the authority's. Its + only write into what is drawn is `Echo.paint`, at the end of + `renderBody` for a terminal pane: the cells predicted and the cursor + after them, over the body as drawn. A terminal's body is painted + straight onto the grid (a body layer is a file's with context rows), + and the tty and the GUI both draw the grid's cells and cursor, so both + get it. A session's own core has `echo` null: one null test a terminal + pane a frame, and no overlay. +- **What is predicted:** a printable ASCII character at the cursor, + Backspace, Left and Right, typed into the view's terminal in tty mode + (keys going to its program), on the primary screen, at the live bottom, + with the cursor shown. Never for a chord, Enter, Tab, Esc, Up and Down, + a paste, or anything on the alternate screen (vim never sees one). More + narrowly, so as never to draw what the line editor would not: + - a character only over a blank cell or the same character (typing + along a fish suggestion); a line edited in its middle, or a + suggestion left, is the shell's to redraw; + - not in the last column (wrap is the program's); + - Backspace only at the end of the line, over a prompt's input or a + predicted character; Left over the same; Right over typed text, a + prompt's input in the colour of the cell before it (past the text's + end fish's Right takes its suggestion). + Each prediction is a cell (an absolute row, the authority's numbering + that the delta's header carries) and the cursor after it; a newer key's + cell replaces an older one's, as mosh's overlay does. +- **Judged once the authority has the key and its echo has had time.** + Each unit of the replica's own says which of its keys the authority + stepped (`Echo.confirmed`). A stream that loses nothing keeps the order + the authority sent in, and it sends what it owes of the terminals + before it steps an input, so a delta that arrives after that unit was + made after the key was written. 50 ms after the unit (`grace_ns`, + mosh's ECHO_TIMEOUT: the shell's echo, then the authority's next frame) + each applied delta judges it: a cell the screen shows goes (confirmed), + one it does not is a misprediction and every prediction goes. A + predicted space (Backspace's too) is judged neither way, as mosh's + blanks are; the cursor after the newest key is judged with it. With no + delta at all (nothing changed: no echo), a frame judges it 250 ms + later (`stale_ns`). +- **A key that is not predicted** (Enter, Tab, a chord, or one the rules + above refuse) is the next to wait for: once the authority has it, every + prediction made before it goes, judged neither way (the deltas from + then show that key's doing, which nothing predicted). The keys typed + behind it wait (`deferred`) until its grace has passed; then those the + authority does not have yet are predicted, in order, over the screen + that shows its doing (`catchUp`). One the authority has whose grace is + not over puts that off a little. So the characters typed straight on + after Tab are drawn where the completion left the cursor once it shows + (a round trip and 50 ms after Tab), the rest at once, and never where + the cursor was before it. + Found by the harness: the first version predicted nothing until the + wait was over, and every key typed in that time became a new wait, so a + word typed after Tab or Enter at 207 ms was all late (p90 242 ms). An + input that is not a key into this terminal (a click, a command) has no + deferral: where the keys after it go is not known here until the + authority has it. +- **Shown where the echo is vouched for.** At a shell's prompt, pardes's + OSC 133 marks (phase input: a B and no C since) and the cursor's own + semantic content say its line editor echoes: drawn at once. Elsewhere + (`cat`, a shell without pardes's marks) only once an echo was confirmed + there since the last Enter, Tab or chord (`trusted`), so the first + character of each line waits a round trip. A key before the last one + not predicted vouches for nothing after it: the harness found the + characters of `sudo ls`, confirmed after its Enter, vouching for the + password prompt that Enter brought up. After a misprediction that was + drawn, nothing is drawn until an echo is confirmed again (`doubt`). A + password prompt or a program that ignores what is typed confirms none, + so nothing typed there is ever drawn. +- **Underlined off a prompt only.** At a prompt the marks vouch for the + echo, and an underline trailing every character for a round trip at + 207 ms is noise; off one the trust is mosh's inference, and the + underline says so. After a misprediction nothing is drawn at all, which + mosh's adaptive underline approximates. +- **No protocol change.** No echo acknowledgement crosses (mosh's + `echo_ack`): the replica's own units stand for it, and the grace for its + 50 ms. No termios flag crosses either: a program's echo is inferred from + confirmations, as mosh does. + +The hook for the remote client (its `Replica` and stream as they are): +- `Replica.init(..., screens = true)` points its core at its echo + (`core.echo = &r.echo`). A replica made another way (the remote line's + `initDump`) does that itself once its core exists. +- Per key: nothing more. `r.input(ev)` predicts after its settle, in + `r.view`. +- Per unit: nothing more. `r.confirm(unit)` tells it which of its keys the + authority has. +- Per delta: nothing more. `r.screen(msg)` judges after each delta it + applies. +- Per frame, before drawing: `if (r.echoFrame()) dirty = true;` judges + what is stale, predicts the keys deferred whose wait is over, and says + whether the overlay changed since the last call; `r.echo.changed` says + the same between frames (after `r.input` and `r.screen`), for a client + that draws only when its state moved (the remote line's `moved`). +- Draw its own view (`core.view == r.view`): the overlay is drawn in no + other, and over a terminal in tty mode only. +- The clock is the monotonic one (`FrameStats.nowNs`) unless `r.echo.now` + is set (the harness's virtual clock). + +Measured. + +- **The harness** (`zig build replicas -Dplatform=tty -- --echo`): one + authority and one replica over part A's delayed, jittered link, the + authority's pty a shell played in the harness (`FakeShell`): a line + editor at a prompt with pardes's marks and Tab completion, and the + programs the corpus runs in it. Each key's echo comes 2 ms after the + authority writes it. The client types a corpus of 222 keys at a + typist's pace (60 to 180 ms a key) and in bursts (12 to 60 ms): lines at + a prompt, Backspace, Left and Right, Tab typed straight on, `sudo` and + its password, vim, a program that ignores what is typed, `cat` (the + kernel's echo, no marks), and a line typed before the last one's prompt. + Each client frame (60 Hz, its own phase) is what its screen shows: its + core rendered, the overlay where it is drawn. Ground truth is the + authority's screen right after each key's echo. +- **Key to its character on the client's screen** (the simulation's clock; + 20 seeds, 3060 keys the shell echoed at each RTT, ms, without → with): + + | | 30 ms | 80 ms | 207 ms | + |---|---|---|---| + | p50 | 49.8 → 8.7 | 102.3 → 8.7 | 239.0 → 8.9 | + | p90 | 59.0 → 15.2 | 111.8 → 15.4 | 247.9 → 15.8 | + | max | 66.9 → 65.3 | 121.5 → 114.5 | 261.7 → 251.6 | + + With it, a key waits for the client's next frame, 0 to 16.7 ms. The tty + host draws an input's frame at once, so a real client's is the frame's + cost. The tail is by design: at 207 ms the p90 of the characters typed + straight on after Tab is 38 ms, of a line typed ahead of its prompt + 48 ms, and of `cat`'s 179 ms (its first characters of a line wait for an + echo to be confirmed); every other part's is 14 to 15 ms. +- **Accuracy** (the same runs, 4440 keys at each RTT): 3718 to 3720 + predicted (84%); 714 to 722 not, by the rules (each Enter, Tab, Ctrl-C, + Ctrl-D and Esc, everything typed in vim, and the characters behind a + key not predicted that the screen showed by the time it was over). Of + those predicted, 3238 to 3240 were what the authority's screen showed + right after the echo; the other 480, at each RTT, are exactly the + password's and the ignoring program's 24 a seed, and none was drawn. + Wrong characters drawn: none, at every RTT (0 keys, 0 frames). Frames a + password's or the ignoring program's key was drawn: 0. vim's keys + predicted: 0. The overlay judged 97 to 132 cells against the screen as + mispredictions, all of them undrawn ones (the password's, the ignoring + program's), and no cursor. Every frame drawn with a prediction shown + (13,083 to 21,009 a cell) had each predicted cell where it was + predicted in the rendered surface. +- **Costs** (ReleaseFast): `paint` p50 50 to 63 ns, p99 86 to 155 ns; the + replica's render with a prediction shown p50 54 µs; a key's `typed` is + a few array scans. +- **No divergence**: the replicas harness (ReleaseFast, 20 seeds x 120 + inputs x 3 clients): none diverged, none unequal, 23,547 deltas checked + and none wrong; its deltas a second (76.3, 75.7, 71.7) are part B's to + the decimal. +- **No local regression.** A session's own core draws no overlay + (`echo` is null: a unit test says the authority's core has none). + `pardes-perf-live --live` (base's driver), main (vtzpkxwu) and this + line interleaved A B A ... A on the perf cores (3 runs of the new one, 4 + of main), ReleaseFast; the median of the runs' p50s, new/main: + + | | new/main | A/A (main) | + |---|---|---| + | key-down, present / core | 0.99 / 0.99 | 1.00 / 0.99 | + | key-right | 0.99 / 1.00 | 1.00 / 1.01 | + | edit-char | 0.97 / 0.98 | 0.99 / 0.98 | + | page-down | 0.85 / 0.92 | 0.82 / 0.85 | + | wheel | 0.97 / 0.99 | 0.98 / 0.98 | + | `yes \| head -c 32M` wall | 1.11 | 1.04 | + | a 1M-line open | 0.93 | 1.01 | + + Every cell's runs overlap main's. The `yes` cell is the kernel's (Tty + throughput notes): main's own runs read 4.5 to 7.1 s in the A/B and 5.8 + to 7.6 s in the A/A, this line's 5.0 to 7.2 s. What the local path + gained: a null test a terminal pane a frame, and 8 bytes of core. + +The gates: +- Unit tests (`zig build core-test -Dtest-filter=echo`): keys typed at a + prompt drawn at once in the rendered surface, unflagged, the cursor past + them, then confirmed by the echo, Backspace too, and the replica's digest + the authority's; a password prompt draws nothing typed and lets it go + when stale, and vim's alternate screen predicts nothing; a misprediction + takes every prediction back and hides the next until an echo is + confirmed, and after Tab the keys behind it are drawn where the + completion left the cursor, or not at all if the screen already shows + them. +- `zig build replicas -- --echo` fails on any divergence, unequal end, + wrong delta, drawn password or ignored key, predicted vim key, or cell + rendered otherwise than predicted; it passes at 20 seeds, Debug and + ReleaseFast. `zig build replicas` passes at 20 seeds, ReleaseFast (and 4 + in Debug). +- Unit tests tty (`zig build unit-test -Dplatform=tty`, its perf gate + too), the full `core-test`, and the 113 snapshot scripts pass; the GUI + build's echo tests pass (`core-test -Dplatform=gui`). + +Not done, and why: +- ASCII only: a width table would let the rest of Unicode in (`kindOf`). +- A line edited in its middle (insert, Backspace), Delete, Home, End, the + readline chords, word moves and a paste are not predicted: the line + editor redraws what is right of the cursor, which a prediction would + have to guess. Each waits a round trip, and the keys typed straight on + behind it are drawn when it is over. +- A key typed after an input that is not a key into this terminal (a + click, a command) and that the replica could not apply itself waits for + the authority, and so do the keys after it until a pause: where they go + is not known here until the authority has it. +- The echo is inferred, not told. No termios flag and no echo + acknowledgement cross. A password prompt is safe because nothing typed + there is ever confirmed. A shell at its prompt that does not echo a key + (zsh's vi command mode) has it drawn until its grace passes, about a + round trip and 50 ms, and then nothing more until an echo is confirmed; + so does a program that turns echo off mid-line in `cat`'s plain echo. +- An echo later than about 30 ms on the authority's host (a loaded box) + makes a drawn character go when judged and come back with the echo: a + flicker, never a wrong character. The grace could follow the measured + echo delay. +- A shell without pardes's marks (fish's own marks, a shell started + without the integration) is trusted only by confirmation, so each line's + first character waits a round trip, as in `cat`. +- The GUI was not run: it draws the grid's cells and cursor that the + harness checks in the rendered surface, and its build compiles and + passes the echo tests. +- The remote client (the other line) wires it: `initDump`'s core, the + frame hook, its `moved`. diff --git a/src/body_layer.zig b/src/body_layer.zig index 5a8d51e5..7a4103c3 100644 --- a/src/body_layer.zig +++ b/src/body_layer.zig @@ -481,6 +481,9 @@ pub fn renderBody(p: *Pardes, s: *Surface, arena: std.mem.Allocator, pane: *Pane } } + // A replica's predicted echo, over what its authority's screen shows. + if (p.echo) |e| if (pane.isTerminal()) e.paint(p, s, pane, id, tx, tw, body_y, body_h, active); + if (rail) |bar| if (bar.rect.h > 0) { // The rail covers the notice bands too: nothing else paints those // two gutter columns, so a gap there keeps the frame-wide border diff --git a/src/echo.zig b/src/echo.zig new file mode 100644 index 00000000..4acd3d11 --- /dev/null +++ b/src/echo.zig @@ -0,0 +1,781 @@ +//! Predicted echo for a replica's terminal panes, mosh's way +//! (docs/next-daemon.md, "Phase 6: predicted echo notes"). A key typed into +//! a terminal goes to the authority, and its echo comes back as screen state +//! a round trip later: this shows it at once. +//! +//! Presentation only. The overlay is the replica's own (`Pardes.echo`, a +//! `.local` field the rollback copy skips), painted over a terminal's body +//! after it is drawn (`paint`, from body_layer.renderBody, which the tty and +//! the GUI share). The core's emulator stays the authority's, and no digest +//! sees any of this. +//! +//! What is predicted: a printable character at the cursor, backspace, and +//! left and right, in tty mode on the primary screen at the live bottom. Any +//! other input this replica could not apply itself (Enter, Tab, a chord, a +//! paste, a click) stops prediction until the authority has confirmed it and +//! its echo has had `grace` to come back: predictions start again from the +//! screen's own cursor. So a key typed after Tab waits a round trip; it is +//! never drawn where the completion was not. +//! +//! Judged as mosh does, once the authority confirmed the key and `grace` +//! passed: at a delta, a cell the screen shows goes, and one it does not is +//! a misprediction, and every prediction goes. With no delta (nothing +//! changed: no echo), a frame judges it `stale` later. Shown at once at a +//! shell's prompt (pardes's OSC 133 marks say its line editor echoes); +//! elsewhere only once an echo was confirmed there since the last key that +//! was not predicted (`cat`, a shell without marks), and underlined; never +//! after a shown misprediction until an echo is confirmed again. A password +//! prompt or a program that ignores input confirms none, so nothing typed +//! there is ever shown. +const std = @import("std"); +const pardes = @import("pardes.zig"); +const terminal = pardes.panes.terminal; +const Pardes = pardes.Pardes; +const Pane = pardes.Pane; +const Surface = pardes.Surface; +const Key = pardes.Key; + +const ms = std.time.ns_per_ms; +/// After the authority confirmed a key, how long its echo has to come back +/// before a delta that does not show it makes a misprediction (mosh's +/// ECHO_TIMEOUT): the shell's echo, then the authority's next frame. +// ponytail: fixed; a loaded host's later echo makes a drawn character go and +// come back. The echo delay measured per terminal would be the bound. +pub const grace_ns = 50 * ms; +/// With no delta at all, how much longer a frame waits to judge a key. +pub const stale_ns = 250 * ms; +const never = std.math.maxInt(u64); +/// Cells predicted at once: a line typed a round trip ahead. +const max = 64; + +/// A cell predicted: `cp` at column `x` of absolute row `row` (the +/// authority's numbering, screen_state's), a space for a cell backspace +/// cleared; `was` what the screen showed there before; `seq` the key's. +pub const Cell = struct { x: u16, row: u64, cp: u21, was: u21, seq: u32, due: u64 = never }; +pub const Cursor = struct { x: u16, row: u64, seq: u32, due: u64 = never }; + +pub const Stats = struct { + /// Keys into a terminal predicted, and not. + predicted: u32 = 0, + unpredicted: u32 = 0, + /// Cells the screen showed when judged; cells and cursors it did not. + confirmed: u32 = 0, + mispredicted: u32 = 0, + cursor_missed: u32 = 0, +}; + +pub const Echo = struct { + /// Off: nothing predicted (the harness's baseline). + on: bool = true, + /// The harness's clock (ns); the monotonic one when null. + now: ?u64 = null, + /// Its client's view, and the terminal it predicts into. + view: u8 = 0, + pane: u8 = 0, + serial: u32 = 0, + /// Each terminal's last delta: its active area's first row, absolute. + tops: [pardes.MAX_PANES]u64 = @splat(0), + cells: [max]Cell = undefined, + n: u8 = 0, + cursor: ?Cursor = null, + /// The newest of its client's inputs the authority confirmed. + acked: u32 = 0, + /// The newest input not predicted, and when its echo is due: nothing is + /// predicted until then. `wait_key`: it was a key forwarded to this + /// terminal (no other pane's), so the keys typed into it meanwhile wait + /// in `deferred`, to be predicted over the screen that shows its doing. + wait: u32 = 0, + wait_due: u64 = 0, + wait_key: bool = false, + deferred: [max]Deferred = undefined, + nd: u8 = 0, + /// An echo confirmed since the last key not predicted: shown off a + /// prompt too. A shown misprediction since the last confirmed echo: + /// shown nowhere. + trusted: bool = false, + doubt: bool = false, + /// What it draws changed: the client draws a frame. + changed: bool = false, + stats: Stats = .{}, + /// Each key's prediction as it is made, when set (the harness's; on + /// the C heap). + log: ?*std.ArrayListUnmanaged(Predicted) = null, + + pub const Predicted = struct { seq: u32, cell: ?Cell, cursor: Cursor }; + + fn time(e: *const Echo) u64 { + return e.now orelse pardes.FrameStats.nowNs(); + } + + const Deferred = struct { seq: u32, kind: ?Kind, due: u64 = never }; + + fn drop(e: *Echo) void { + if (e.n > 0 or e.cursor != null) e.changed = true; + e.n = 0; + e.cursor = null; + } + + fn waiting(e: *const Echo) bool { + return e.acked < e.wait or e.time() < e.wait_due; + } + + fn waitFor(e: *Echo, seq: u32, key: bool) void { + e.wait = seq; + e.wait_due = never; + e.wait_key = key; + e.nd = 0; + } + + /// Its client's input `seq`, after the replica took it: `applied`, + /// it was applied here at once (so it was no key into a terminal). + pub fn typed(e: *Echo, c: *Pardes, view: u8, ev: pardes.Event, seq: u32, applied: bool) void { + e.view = view; + if (applied or comptime !terminal.enabled) return; + switch (ev) { + // ponytail: a motion or focus report a program asked for could + // move its cursor; the judgement catches that. + .mouse => |m| if (m.kind == .motion) return, + .pointer_leave, .focus => return, + else => {}, + } + const id = keyboardPane(c, view) orelse { + // Another pane's, or not a key: where the keys after it go is + // not known here until the authority has it. ponytail: so they + // wait for it too, each, until a pause. + e.waitFor(seq, false); + return; + }; + if (id != e.pane or c.panes[id].?.serial != e.serial) { + e.drop(); + e.nd = 0; + e.pane = @intCast(id); + e.serial = c.panes[id].?.serial; + } + const kind = kindOf(ev); + const key = forwarded(c, ev); + // A key that is not one of those predicted says nothing of the + // program the keys after it reach (Enter: a password prompt's). + if (kind == null) e.trusted = false; + if (!e.on) { + e.stats.unpredicted += 1; + return e.waitFor(seq, key); + } + // Behind keys still deferred: after them, in order. + e.catchUp(c); + if (e.waiting() or e.nd > 0) { + if (e.wait_key and key and e.nd < max) { + e.deferred[e.nd] = .{ .seq = seq, .kind = kind }; + e.nd += 1; + return; + } + e.stats.unpredicted += 1; + return e.waitFor(seq, false); + } + if (kind) |k| if (e.predict(c.panes[id].?, k, seq)) { + e.stats.predicted += 1; + e.changed = true; + return; + }; + e.stats.unpredicted += 1; + e.waitFor(seq, key); + } + + /// The wait over (its key's doing on the screen), the keys deferred + /// behind it that the screen does not show yet are predicted over it, + /// in order; those it shows are not. One the authority has whose echo + /// may be on its way puts it off until its grace has passed. + fn catchUp(e: *Echo, c: *Pardes) void { + if (e.nd == 0 or e.waiting()) return; + const now = e.time(); + var i: usize = 0; + while (i < e.nd and e.deferred[i].due <= now) i += 1; + if (i < e.nd and e.deferred[i].due != never) return; + e.stats.unpredicted += @intCast(i); + const pane = c.panes[e.pane] orelse return e.waitFor(e.wait, false); + if (pane.serial != e.serial) return e.waitFor(e.wait, false); + while (i < e.nd) : (i += 1) { + const d = e.deferred[i]; + if (d.kind) |k| if (e.predict(pane, k, d.seq)) { + e.stats.predicted += 1; + e.changed = true; + continue; + }; + // The next to wait for, the rest behind it. + e.stats.unpredicted += 1; + const rest = e.nd - i - 1; + std.mem.copyForwards(Deferred, e.deferred[0..rest], e.deferred[i + 1 .. e.nd]); + e.wait = d.seq; + e.wait_due = never; + e.wait_key = true; + e.nd = @intCast(rest); + return; + } + e.nd = 0; + } + + /// Whether a key `kind` into terminal `pane` was predicted. + fn predict(e: *Echo, pane: *Pane, kind: Kind, seq: u32) bool { + const state = pane.terminal orelse return false; + const vt = &state.vt; + if (vt.screens.active_key != .primary or !terminal.atBottom(pane) or !vt.modes.get(.cursor_visible)) return false; + const top = e.tops[e.pane]; + const screen = vt.screens.active; + const cur: Cursor = e.cursor orelse .{ .x = @intCast(screen.cursor.x), .row = top + screen.cursor.y, .seq = 0 }; + if (cur.row < top or cur.row - top >= vt.rows or e.n == max) return false; + var next: Cursor = .{ .x = cur.x, .row = cur.row, .seq = seq }; + var cell: ?Cell = null; + switch (kind) { + .char => |cp| { + // The last column wraps as a program pleases; a cell that + // holds another character is a line edited in its middle, + // or a suggestion (fish's) being left: the shell redraws + // what is right of it. ponytail: mosh's shift of the rest + // would cover bash's middle, and mispredict fish's. + if (cur.x + 1 >= vt.cols) return false; + const under = e.cpAt(state, cur.x, cur.row); + if (!blank(under) and under != cp) return false; + cell = .{ .x = cur.x, .row = cur.row, .cp = cp, .was = under, .seq = seq }; + next.x += 1; + }, + // At the end of a line only: in its middle a line editor moves + // the rest left. + .backspace => { + if (cur.x == 0 or !e.isInput(state, cur.x - 1, cur.row) or !blank(e.cpAt(state, cur.x, cur.row))) return false; + cell = .{ .x = cur.x - 1, .row = cur.row, .cp = ' ', .was = e.cpAt(state, cur.x - 1, cur.row), .seq = seq }; + next.x -= 1; + }, + .left => { + if (cur.x == 0 or !e.isInput(state, cur.x - 1, cur.row)) return false; + next.x -= 1; + }, + // Over typed text only: past its end, fish takes the right + // arrow for its suggestion (another colour). + .right => { + if (cur.x + 1 >= vt.cols or !e.isText(state, cur.x, cur.row)) return false; + next.x += 1; + }, + } + if (cell) |want| e.put(want); + e.cursor = next; + if (e.log) |l| l.append(std.heap.c_allocator, .{ .seq = seq, .cell = cell, .cursor = next }) catch {}; + return true; + } + + fn put(e: *Echo, want: Cell) void { + for (e.cells[0..e.n]) |*have| if (have.x == want.x and have.row == want.row) { + const was = have.was; + have.* = want; + have.was = was; + return; + }; + e.cells[e.n] = want; + e.n += 1; + } + + fn mine(e: *const Echo, x: u16, row: u64) ?*const Cell { + for (e.cells[0..e.n]) |*have| if (have.x == x and have.row == row) return have; + return null; + } + + /// What the screen shows at a cell as predicted so far. + fn cpAt(e: *const Echo, state: *terminal.State, x: u16, row: u64) u21 { + if (e.mine(x, row)) |have| return have.cp; + return e.screenCp(state, x, row); + } + + /// What the screen itself shows there; off its active area, a + /// codepoint no key types. + fn screenCp(e: *const Echo, state: *terminal.State, x: u16, row: u64) u21 { + const ci = e.screenCell(state, x, row) orelse return 0x10ffff; + return ci.cell.codepoint(); + } + + fn screenCell(e: *const Echo, state: *terminal.State, x: u16, row: u64) ?terminal.PageCell { + if (comptime !terminal.enabled) return null; + const top = e.tops[e.pane]; + if (row < top or row - top >= state.vt.rows) return null; + return state.vt.screens.active.pages.getCell(.{ .active = .{ .x = x, .y = @intCast(row - top) } }); + } + + /// A prompt's input there, or a character predicted. + fn isInput(e: *const Echo, state: *terminal.State, x: u16, row: u64) bool { + if (e.mine(x, row)) |have| return !blank(have.cp); + const ci = e.screenCell(state, x, row) orelse return false; + return ci.cell.semantic_content == .input; + } + + /// Typed text there: a character predicted, or a prompt's input in the + /// colour of the cell before it. + fn isText(e: *const Echo, state: *terminal.State, x: u16, row: u64) bool { + if (e.mine(x, row)) |have| return !blank(have.cp); + const ci = e.screenCell(state, x, row) orelse return false; + if (ci.cell.semantic_content != .input or blank(ci.cell.codepoint())) return false; + if (x == 0) return true; + const before = e.screenCell(state, x - 1, row) orelse return false; + return before.cell.style_id == ci.cell.style_id; + } + + /// The authority confirmed its client's input `seq`. + pub fn confirmed(e: *Echo, c: *Pardes, seq: u32) void { + e.acked = seq; + const due = e.time() + grace_ns; + for (e.deferred[0..e.nd]) |*d| if (d.seq <= seq and d.due == never) { + d.due = due; + }; + for (e.cells[0..e.n]) |*cell| if (cell.seq <= seq and cell.due == never) { + cell.due = due; + }; + if (e.cursor) |*cur| if (cur.seq <= seq and cur.due == never) { + cur.due = due; + }; + if (e.wait <= seq and e.wait_due == never) { + // The authority stepped a key this did not predict, after every + // one it did: its deltas from here show that key's doing, which + // no prediction knew of. They go, judged neither way. + e.wait_due = due; + e.drop(); + } + e.catchUp(c); + } + + /// Terminal `id`'s delta was applied, its active area's first row now + /// `top`. + pub fn delta(e: *Echo, c: *Pardes, id: u8, top: u64) void { + if (id == e.pane and top != e.tops[id] and (e.n > 0 or e.cursor != null)) { + // Scrolled: the rows predicted are others now. + e.tops[id] = top; + return e.drop(); + } + e.tops[id] = top; + if (id == e.pane) e.judge(c, 0); + e.catchUp(c); + } + + /// Once a frame: what is stale judged. Whether to draw. + pub fn frame(e: *Echo, c: *Pardes) bool { + if (e.n > 0 or e.cursor != null) e.judge(c, stale_ns); + e.catchUp(c); + defer e.changed = false; + return e.changed; + } + + fn judge(e: *Echo, c: *Pardes, slack: u64) void { + if (comptime !terminal.enabled) return; + if (e.n == 0 and e.cursor == null) return; + const now = e.time(); + const pane = c.panes[e.pane] orelse return e.drop(); + if (pane.serial != e.serial) return e.drop(); + const state = pane.terminal orelse return e.drop(); + if (state.vt.screens.active_key != .primary) return e.drop(); + const shown = e.visible(c); + var i: usize = 0; + while (i < e.n) { + const cell = e.cells[i]; + if (now < cell.due) { + i += 1; + continue; + } + if (!blank(cell.cp) and e.screenCp(state, cell.x, cell.row) != cell.cp) { + if (now < cell.due + slack) { + i += 1; + continue; + } + e.stats.mispredicted += 1; + if (shown) e.doubt = true; + e.trusted = false; + return e.drop(); + } + // Shown, or a space (mosh: too easy to show by chance to vouch). + // A key before the last one not predicted vouches for nothing + // after it (a line's characters for the password prompt its + // Enter brought up). + if (!blank(cell.cp)) { + e.stats.confirmed += 1; + if (cell.was != cell.cp and cell.seq > e.wait) { + e.trusted = true; + e.doubt = false; + } + } + e.cells[i] = e.cells[e.n - 1]; + e.n -= 1; + e.changed = true; + } + const cur = e.cursor orelse return; + if (now < cur.due) return; + const screen = state.vt.screens.active; + if (screen.cursor.x == cur.x and e.tops[e.pane] + screen.cursor.y == cur.row) { + e.cursor = null; + e.changed = true; + } else if (now >= cur.due + slack) { + e.stats.cursor_missed += 1; + e.drop(); + } + } + + /// Whether its predictions are drawn now. + pub fn visible(e: *const Echo, c: *const Pardes) bool { + if (e.doubt) return false; + if (e.trusted) return true; + const pane = c.panes[e.pane] orelse return false; + return atPrompt(pane); + } + + /// Over terminal pane `id`'s body, as renderBody drew it: the + /// characters predicted and the cursor after them. + pub fn paint(e: *const Echo, p: *const Pardes, s: *Surface, pane: *const Pane, id: usize, tx: u16, tw: u16, body_y: u16, body_h: u16, active: bool) void { + if (comptime !terminal.enabled) return; + if (e.n == 0 and e.cursor == null) return; + if (id != e.pane or pane.serial != e.serial or p.view != e.view or pane.body.mode != .tty) return; + if (!e.visible(p)) return; + const state = pane.terminal orelse return; + const sb = state.vt.screens.active.pages.scrollbar(); + const top = e.tops[id]; + // Off a prompt, the trust is inferred: underlined, as mosh does. + const flag = !atPrompt(pane); + const at = struct { + fn f(sb_: @TypeOf(sb), pan: u16, h: u16, top_: u64, row: u64) ?u16 { + if (row < top_ or row - top_ >= sb_.len) return null; + const screen_row = sb_.total - sb_.len + (row - top_); + if (screen_row < sb_.offset + pan) return null; + const y = screen_row - sb_.offset - pan; + return if (y < h) @intCast(y) else null; + } + }.f; + for (e.cells[0..e.n]) |cell| { + const y = at(sb, pane.pan, body_h, top, cell.row) orelse continue; + if (cell.x >= tw) continue; + const dst = s.at(tx + cell.x, body_y + y); + // Typed text continues the text before it (mosh's heuristic). + var style = dst.style; + if (cell.x > 0) { + const before = s.at(tx + cell.x - 1, body_y + y); + if (!before.default and !blank(before.text[0])) style = before.style; + } + if (flag and !blank(cell.cp)) style.ul = .single; + dst.default = false; + dst.text[0] = @intCast(cell.cp); + dst.len = 1; + dst.style = style; + } + if (!active or pane.focus != .body) return; + const cur = e.cursor orelse return; + const y = at(sb, pane.pan, body_h, top, cur.row) orelse return; + if (cur.x < tw) s.cursor = .{ .x = tx + cur.x, .y = body_y + y }; + } +}; + +const Kind = union(enum) { char: u21, backspace, left, right }; + +fn kindOf(ev: pardes.Event) ?Kind { + const k = switch (ev) { + .key => |k| k, + else => return null, + }; + if (k.ctrl or k.alt) return null; + switch (k.cp) { + Key.backspace => return .backspace, + Key.left => return .left, + Key.right => return .right, + else => {}, + } + // As forwardKey sends it. ponytail: ASCII only; a width table would + // let the rest in. + const cp: u21 = if (k.text.len == 1) + k.text[0] + else if (k.text.len == 0 and k.cp >= 'a' and k.cp <= 'z' and k.shift) + k.cp - 'a' + 'A' + else if (k.text.len == 0) + k.cp + else + return null; + return if (cp >= 0x20 and cp < 0x7f) .{ .char = cp } else null; +} + +/// Whether key `ev` goes to the terminal's program as it is (forwardKey's) +/// and not to pardes: Esc backs out of the pane, the tty chord leaves tty +/// mode, Ctrl-V pastes the register. +fn forwarded(c: *const Pardes, ev: pardes.Event) bool { + const k = switch (ev) { + .key => |k| k, + else => return false, + }; + if (k.cp == Key.escape) return false; + return !(k.ctrl and (k.cp == c.opts.tty_toggle or k.cp == 'v' or k.cp == 'V')); +} + +fn blank(cp: u21) bool { + return cp == 0 or cp == ' '; +} + +/// At a shell's prompt, reading its input: pardes's marks (a B and no C +/// since) and the cursor's own, on the primary screen. A shell without +/// pardes's marks is trusted only once its echo is confirmed. +fn atPrompt(pane: *const Pane) bool { + if (comptime !terminal.enabled) return false; + const state = pane.terminal orelse return false; + return state.vt.screens.active_key == .primary and state.stream.handler.phase == .input and terminal.promptInputReady(pane); +} + +/// The terminal view `view`'s keys go to, in tty mode: none when they go +/// elsewhere (a prompt, a header, window mode, another pane). +fn keyboardPane(c: *const Pardes, view: u8) ?usize { + const live = c.view == view; + const part = if (live) null else &(c.parked[view] orelse return null).view; + const id = if (part) |v| v.active else c.active; + if (if (part) |v| v.window_pick != null or v.header_focus else c.window_pick != null or c.header_focus) return null; + const pane = c.panes[id] orelse return null; + if (!pane.isTerminal() or pane.terminal == null or pane.prompt != .none) return null; + const mode = if (live) pane.body.mode else pane.pv[view].body.mode; + const focus = if (live) pane.focus else pane.pv[view].focus; + return if (mode == .tty and focus == .body) id else null; +} + +// ---- tests: one authority and one replica, its shell played by hand ---- + +const testing = std.testing; +const replica = @import("replica.zig"); + +const Rig = struct { + a: *replica.Authority, + r: *replica.Replica, + screens: replica.Screens, + peer: u16, + shell: u8, + /// What the authority wrote its shell's pty. + written: std.ArrayListUnmanaged(u8) = .empty, + now: u64 = 1_000 * ms, + + fn ptyWrite(ctx: ?*anyopaque, _: u8, bytes: []const u8) void { + const g: *Rig = @ptrCast(@alignCast(ctx.?)); + g.written.appendSlice(testing.allocator, bytes) catch {}; + } + fn watched(_: ?*anyopaque) bool { + return false; + } + const vtable: pardes.Host.VTable = .{ .watched = watched, .pty_write = ptyWrite }; + + fn init(g: *Rig) !void { + const gpa = testing.allocator; + g.written = .empty; + g.now = 1_000 * ms; + g.a = try replica.Authority.init(gpa, .{ .cols = 60, .rows = 16, .boot = .tty_shell }, .{ .ctx = g, .vtable = &vtable }, true); + g.screens = .init(gpa); + g.peer = try g.screens.addPeer(); + g.r = try replica.Replica.init(gpa, 1, try g.a.unit(0, 0), true); + g.shell = for (g.a.core.panes, 0..) |slot, id| { + if (slot) |pane| if (pane.isTerminal()) break @intCast(id); + } else return error.NoShell; + g.r.echo.now = g.now; + } + + fn deinit(g: *Rig) void { + g.r.deinit(); + g.screens.deinit(g.a.core); + g.a.deinit(); + g.written.deinit(testing.allocator); + } + + fn later(g: *Rig, dt: u64) void { + g.now += dt; + g.r.echo.now = g.now; + } + + /// The shell prints `bytes`; the replica gets the unit and the delta. + fn prints(g: *Rig, bytes: []const u8) !void { + g.a.output(.{ .output = .{ .pane = g.shell, .bytes = bytes } }); + try g.sync(); + } + + fn sync(g: *Rig) !void { + const gpa = testing.allocator; + if (try g.a.pending()) |u| g.r.confirm(u); + g.r.settle(); + var msgs: std.ArrayListUnmanaged([]u8) = .empty; + defer { + for (msgs.items) |m| gpa.free(m); + msgs.deinit(gpa); + } + try g.screens.flush(g.a.core, g.peer, &msgs); + for (msgs.items) |m| if (g.r.screen(m)) |reply| { + _ = try g.screens.take(g.a.core, g.peer, reply); + }; + } + + /// The replica's client types `k`; the message for the authority. + fn key(g: *Rig, k: Key) ![]u8 { + return testing.allocator.dupe(u8, try g.r.input(.{ .key = k })); + } + + /// The authority takes it, and the replica its unit. + fn take(g: *Rig, msg: []u8) !void { + defer testing.allocator.free(msg); + if (try g.a.pending()) |u| g.r.confirm(u); + g.r.confirm(try g.a.take(msg)); + g.r.settle(); + } + + /// What the replica's screen shows at the shell's cursor row, drawn, + /// and whether the cell at `x` is underlined. + fn drawn(g: *Rig, arena: *std.heap.ArenaAllocator, x: u16) !struct { cp: u8, ul: bool, cursor: ?u16 } { + _ = arena.reset(.retain_capacity); + const c = g.r.core(); + c.needs_frame = true; + const s = try c.render(arena.allocator()); + const pane = c.panes[g.shell].?; + const body = s.region(.body, g.shell).?.rect; + const y = body.y + terminal.gridCursor(pane).y; + const cell = s.at(body.x + x, y); + return .{ .cp = cell.text[0], .ul = cell.style.ul != .off, .cursor = if (s.cursor) |cur| if (cur.y == y) cur.x - body.x else null else null }; + } +}; + +const prompt = "\x1b]133;A;cl=line;aid=pardes\x07$ \x1b]133;B;aid=pardes\x07"; + +test "keys typed at a shell's prompt are drawn at once, and the authority's echo takes their place" { + if (comptime !terminal.enabled) return error.SkipZigTest; + var g: Rig = undefined; + try g.init(); + defer g.deinit(); + try g.prints(prompt); + var arena: std.heap.ArenaAllocator = .init(testing.allocator); + defer arena.deinit(); + const before = try g.drawn(&arena, 2); + try testing.expectEqual(@as(?u16, 2), before.cursor); + // Typed: shown at once, the cursor past it, and nothing underlined at + // a prompt. + const l = try g.key(.{ .cp = 'l', .text = "l" }); + const s = try g.key(.{ .cp = 's', .text = "s" }); + try testing.expectEqual(@as(u32, 2), g.r.echo.stats.predicted); + var shown = try g.drawn(&arena, 2); + try testing.expectEqual(@as(u8, 'l'), shown.cp); + try testing.expect(!shown.ul); + try testing.expectEqual(@as(?u16, 4), shown.cursor); + try testing.expectEqual(@as(u8, 's'), (try g.drawn(&arena, 3)).cp); + // The authority's core draws no overlay of its own. + try testing.expect(g.a.core.echo == null); + // The authority takes them, its shell echoes, the delta lands after the + // grace: confirmed, and the screen's own cells and cursor shown. + try g.take(l); + try g.take(s); + try testing.expectEqualStrings("ls", g.written.items); + g.later(grace_ns); + try g.prints("ls"); + try testing.expectEqual(@as(u8, 0), g.r.echo.n); + try testing.expect(g.r.echo.cursor == null); + try testing.expectEqual(@as(u32, 2), g.r.echo.stats.confirmed); + shown = try g.drawn(&arena, 3); + try testing.expectEqual(@as(u8, 's'), shown.cp); + try testing.expectEqual(@as(?u16, 4), shown.cursor); + // Backspace at the end of the line, shown at once. + const bs = try g.key(.{ .cp = Key.backspace }); + shown = try g.drawn(&arena, 3); + try testing.expectEqual(@as(u8, ' '), shown.cp); + try testing.expectEqual(@as(?u16, 3), shown.cursor); + try g.take(bs); + g.later(grace_ns); + try g.prints("\x08\x1b[K"); + try testing.expectEqual(@as(u32, 0), g.r.echo.stats.mispredicted + g.r.echo.stats.cursor_missed); + try testing.expect(g.r.quiet()); + try testing.expectEqual(replica.digestOf(g.a), g.r.digest()); +} + +test "a password prompt shows nothing typed, and a full-screen program is not predicted" { + if (comptime !terminal.enabled) return error.SkipZigTest; + var g: Rig = undefined; + try g.init(); + defer g.deinit(); + try g.prints(prompt ++ "sudo ls\r\n\x1b]133;C;aid=pardes\x07[sudo] password: "); + var arena: std.heap.ArenaAllocator = .init(testing.allocator); + defer arena.deinit(); + const x = terminal.gridCursor(g.r.core().panes[g.shell].?).x; + const k = try g.key(.{ .cp = 'h', .text = "h" }); + // Predicted, and not drawn: nothing has confirmed an echo here. + try testing.expectEqual(@as(u32, 1), g.r.echo.stats.predicted); + try testing.expectEqual(@as(u8, ' '), (try g.drawn(&arena, x)).cp); + try g.take(k); + // No echo, so no delta: a frame lets it go once it is stale. + g.later(grace_ns + stale_ns); + _ = g.r.echoFrame(); + try testing.expectEqual(@as(u8, 0), g.r.echo.n); + try testing.expectEqual(@as(u32, 1), g.r.echo.stats.mispredicted); + // The next key still shows nothing. + const k2 = try g.key(.{ .cp = 'i', .text = "i" }); + try testing.expectEqual(@as(u8, ' '), (try g.drawn(&arena, x)).cp); + try g.take(k2); + // vim: the alternate screen, never predicted. + try g.prints("\r\n\x1b]133;D;1;aid=pardes\x07" ++ prompt ++ "vim\r\n\x1b]133;C;aid=pardes\x07\x1b[?1049h\x1b[H~"); + g.later(grace_ns + stale_ns); + _ = g.r.echoFrame(); + const before = g.r.echo.stats.predicted; + try g.take(try g.key(.{ .cp = 'i', .text = "i" })); + try testing.expectEqual(before, g.r.echo.stats.predicted); +} + +test "a misprediction takes every prediction back and shows none until an echo is confirmed; after Tab nothing is predicted until its echo" { + if (comptime !terminal.enabled) return error.SkipZigTest; + var g: Rig = undefined; + try g.init(); + defer g.deinit(); + try g.prints(prompt); + var arena: std.heap.ArenaAllocator = .init(testing.allocator); + defer arena.deinit(); + // A line editor that echoes something else (vi's command mode): shown, + // then taken back. + const a = try g.key(.{ .cp = 'a', .text = "a" }); + try testing.expectEqual(@as(u8, 'a'), (try g.drawn(&arena, 2)).cp); + try g.take(a); + g.later(grace_ns); + try g.prints("A"); + try testing.expectEqual(@as(u32, 1), g.r.echo.stats.mispredicted); + try testing.expect(g.r.echo.doubt); + try testing.expectEqual(@as(u8, 'A'), (try g.drawn(&arena, 2)).cp); + // Doubted: predicted, not drawn, until one is confirmed. + const b = try g.key(.{ .cp = 'b', .text = "b" }); + try testing.expectEqual(@as(u8, ' '), (try g.drawn(&arena, 3)).cp); + try g.take(b); + g.later(grace_ns); + try g.prints("b"); + try testing.expect(!g.r.echo.doubt); + const c = try g.key(.{ .cp = 'c', .text = "c" }); + try testing.expectEqual(@as(u8, 'c'), (try g.drawn(&arena, 4)).cp); + // Tab: not predicted, and neither is what follows until the authority + // has it and its echo has had its grace. + const tab = try g.key(.{ .cp = Key.tab }); + const d = try g.key(.{ .cp = 'd', .text = "d" }); + try testing.expectEqual(@as(u32, 1), g.r.echo.stats.unpredicted); + try g.take(c); + try testing.expectEqual(@as(u8, 'c'), (try g.drawn(&arena, 4)).cp); + // The authority has Tab (and the key after it): what was predicted + // before it goes, the screen's own shown. + try g.take(tab); + try g.take(d); + try testing.expectEqual(@as(u8, 0), g.r.echo.n); + g.later(grace_ns); + try g.prints("c" ++ "ompleted d"); + // `d`, typed behind Tab, the screen showed by then: not predicted. + try testing.expectEqual(@as(u32, 2), g.r.echo.stats.unpredicted); + const e = try g.key(.{ .cp = 'e', .text = "e" }); + try testing.expectEqual(@as(u8, 'e'), (try g.drawn(&arena, 15)).cp); + try g.take(e); + g.later(grace_ns); + try g.prints("e"); + // A key typed behind Tab that the authority does not have yet is + // predicted once Tab's doing is on the screen, where it now goes. + const tab2 = try g.key(.{ .cp = Key.tab }); + const f = try g.key(.{ .cp = 'f', .text = "f" }); + try g.take(tab2); + try g.prints("ff"); + try testing.expectEqual(@as(u8, 0), g.r.echo.n); + g.later(grace_ns); + _ = g.r.echoFrame(); + try testing.expectEqual(@as(u8, 1), g.r.echo.n); + try testing.expectEqual(@as(u8, 'f'), (try g.drawn(&arena, 18)).cp); + try g.take(f); + g.later(grace_ns); + try g.prints("f"); + try testing.expectEqual(@as(u8, 0), g.r.echo.n); + try testing.expectEqual(@as(u32, 1), g.r.echo.stats.mispredicted); + try testing.expectEqual(@as(u32, 0), g.r.echo.stats.cursor_missed); + try testing.expect(g.r.quiet()); + try testing.expectEqual(replica.digestOf(g.a), g.r.digest()); +} diff --git a/src/pardes.zig b/src/pardes.zig index 081861ac..02983c84 100644 --- a/src/pardes.zig +++ b/src/pardes.zig @@ -67,6 +67,7 @@ pub const screen_state = if (terminal_panes) @import("screen_state.zig") else st pub const views = @import("views.zig"); pub const rollback = @import("rollback.zig"); pub const replica = @import("replica.zig"); +pub const echo = @import("echo.zig"); pub const image = @import("image.zig"); pub const dump = @import("dump.zig"); pub const lsp = @import("lsp/lsp.zig"); @@ -478,6 +479,7 @@ pub const Loc = struct { test { _ = @import("rollback.zig"); _ = @import("replica.zig"); + _ = @import("echo.zig"); _ = @import("modal.zig"); _ = @import("tagline.zig"); _ = @import("hunks.zig"); @@ -4698,6 +4700,10 @@ pub const Pardes = struct { restore_buf: [1024]u8 = @splat(0), surface: Surface = .{}, + /// A replica's predicted echo (src/echo.zig), drawn over its terminals: + /// presentation, its own, in no digest and no rollback copy. Null on a + /// session's own core. + echo: ?*echo.Echo = null, /// The view the fields above are of: whose layout, focus, pointer and /// panes' cursors and scroll (views.zig). `views_open` says which views /// there are, one a window or terminal; each not live is in `parked`. @@ -4727,6 +4733,7 @@ pub const Pardes = struct { "surface", "presentation", "frames", "chrome_animation", "lifts", "lift_serials", "cursor_glide", "cursor_view", "cursor_crossing", "cursor_held", "frame_arena", "scratch", "needs_frame", "present_skipped", "shell_continuous", "native_image_patches", + "echo", // Drained by every pump before its digest; a replica's queues // are its own. "effects", "effects_head", "effects_len", "in_q", "in_head", "in_len", diff --git a/src/replica.zig b/src/replica.zig index 0b55a5bf..032e50af 100644 --- a/src/replica.zig +++ b/src/replica.zig @@ -467,6 +467,9 @@ pub const Replica = struct { /// Its terminals' screen state, as the authority's deltas made it. screens: [pardes.MAX_PANES]?ScreenOf = @splat(null), screen_out: std.ArrayListUnmanaged(u8) = .empty, + /// Its predicted echo (src/echo.zig), drawn over its core's terminals: + /// fed by its keys, its units and its deltas here, and by `echoFrame`. + echo: pardes.echo.Echo = .{}, const ScreenOf = struct { serial: u32, @@ -490,6 +493,7 @@ pub const Replica = struct { r.deinit(); return error.NoCore; } + if (screens) r.core().echo = &r.echo; return r; } @@ -538,6 +542,7 @@ pub const Replica = struct { try r.pending.append(r.gpa, .{ .seq = seq, .msg = msg }); r.dirty = true; r.settle(); + if (r.t.screen_state and r.diverged == null) r.echo.typed(r.core(), r.view, ev, seq, r.applied == r.pending.items.len); r.out.clearRetainingCapacity(); try r.out.appendSlice(r.gpa, msg); return r.out.items; @@ -548,6 +553,7 @@ pub const Replica = struct { if (r.diverged != null) return; const u = parseUnit(msg) orelse return r.fail("a unit too short to read"); r.stats.units += 1; + if (u.origin == r.id) if (r.t.core) |c| r.echo.confirmed(c, u.seq); const was = Stepping.begin(&r.t, &r.epoch); defer was.end(); if (u.origin == r.id and r.pending.items.len > 0 and r.pending.items[0].seq == u.seq and r.applied > 0 and @@ -716,6 +722,15 @@ pub const Replica = struct { return true; } + /// Its client's, once a frame before it draws: a predicted echo the + /// screen has not shown in time goes, and keys deferred behind one not + /// predicted are predicted once its doing is on the screen. Whether what + /// the overlay draws changed since the last call (by a key, a unit, a + /// delta or this: `echo.changed`). + pub fn echoFrame(r: *Replica) bool { + return r.echo.frame(r.core()); + } + /// Nothing pending and nothing applied: the replica is the authority's /// confirmed state, which `digest` says. pub fn quiet(r: *const Replica) bool { @@ -754,6 +769,7 @@ pub const Replica = struct { r.stats.apply_ns.append(r.gpa, nowNs() - t0) catch {}; sc.refused = false; terminal.takeMarks(state, applied.extra); + r.echo.delta(c, m.pane, sc.rep.top); // Its rows as the core keeps them read are another grid's. c.shell_rows.markStale(pane); var seq: [4]u8 = undefined; diff --git a/src/rollback.zig b/src/rollback.zig index 37fc481d..6496a69f 100644 --- a/src/rollback.zig +++ b/src/rollback.zig @@ -415,10 +415,11 @@ fn annotatedStruct(comptime T: type) bool { /// is taken: what is in their slots is nothing), the frame statistics /// (numbers, and a static name), and the rows a terminal was last read as /// (a grid's worth of text, emptied when a copy is taken and when it is -/// put back: src/replica.zig). +/// put back: src/replica.zig), and a replica's predicted echo (src/echo.zig: +/// the replica's, never its core's). fn unsaved(comptime name: []const u8) bool { return std.mem.eql(u8, name, "effects") or std.mem.eql(u8, name, "in_q") or std.mem.eql(u8, name, "frames") or - std.mem.eql(u8, name, "shell_rows"); + std.mem.eql(u8, name, "shell_rows") or std.mem.eql(u8, name, "echo"); } fn annotated(comptime T: type, comptime kind: []const u8, comptime name: []const u8) bool { diff --git a/src/terminal.zig b/src/terminal.zig index aedc007f..3a589659 100644 --- a/src/terminal.zig +++ b/src/terminal.zig @@ -20,6 +20,8 @@ const ghostty_vt = if (enabled) @import("ghostty-vt") else struct {}; pub const history_max = if (enabled) 64 else 8; pub const VtSlot = if (enabled) ghostty_vt.Terminal else void; +/// A cell of an emulator's pages, as `getCell` reads it (src/echo.zig). +pub const PageCell = if (enabled) ghostty_vt.PageList.Cell else void; /// Kitty graphics are drawn where pixel geometry is (pdf_enabled, as for /// image panes' native placements). pub const images_enabled = enabled and pardes.pdf_enabled; diff --git a/test/replicas.zig b/test/replicas.zig index 9b5c3c3b..baaa669d 100644 --- a/test/replicas.zig +++ b/test/replicas.zig @@ -13,6 +13,11 @@ //! asking for the rows it scrolls back into and does not hold. //! //! zig build replicas -Dplatform=tty -- [--seeds N] [--steps M] [--clients C] [--rtt 30,80,207] [--first K] [--flood MB] [--shared] +//! zig build replicas -Dplatform=tty -- --echo [--seeds N] [--rtt 30,80,207] [--first K] +//! +//! `--echo` types a corpus into a shell instead, with and without the +//! predicted echo (src/echo.zig), and gates what it draws: below, at +//! "predicted echo". //! //! Gates, per seed and RTT: no replica diverges (every confirmed unit's //! digests agree as it is replayed), and at quiescence every replica's @@ -29,6 +34,7 @@ const libc = std.c; const pardes = @import("pardes"); const replica = pardes.replica; const screen_state = pardes.screen_state; +const echo = pardes.echo; const Key = pardes.Key; const gpa = std.heap.c_allocator; @@ -58,7 +64,7 @@ const Host = struct { } }; -const Kind = enum { input, to_authority, to_replica, echo, frame, output, screen_down, screen_up }; +const Kind = enum { input, to_authority, to_replica, echo, frame, output, screen_down, screen_up, client_frame }; const Msg = struct { at: u64, @@ -66,6 +72,8 @@ const Msg = struct { kind: Kind, who: u16, bytes: []u8, + /// An echo's: the client's input that made it (--echo). + seq: u32 = 0, fn order(_: void, a: Msg, b: Msg) std.math.Order { if (a.at != b.at) return std.math.order(a.at, b.at); @@ -288,14 +296,6 @@ fn run(r: Run, dir: []const u8, totals: *Totals) !void { var down: [8]Link = @splat(.{}); var left: [8]usize = @splat(r.steps); var typing: usize = r.clients; - const push = struct { - fn f(q: *std.PriorityQueue(Msg, void, Msg.order), nn: *u64, m: Msg) !void { - var mm = m; - mm.n = nn.*; - nn.* += 1; - try q.push(gpa, mm); - } - }.f; // Each client's first input, soon; the programs; the first frame. for (0..r.clients) |c| try push(&queue, &n, .{ .at = random.uintLessThan(u64, 200_000), .n = 0, .kind = .input, .who = @intCast(c + 1), .bytes = &.{} }); try push(&queue, &n, .{ .at = tui_at, .n = 0, .kind = .output, .who = 0, .bytes = &.{} }); @@ -309,26 +309,6 @@ fn run(r: Run, dir: []const u8, totals: *Totals) !void { // authority steps an input, and once a frame. var owed: std.ArrayListUnmanaged([]u8) = .empty; defer owed.deinit(gpa); - const flushAll = struct { - fn f(sc: *replica.Screens, core: *pardes.Pardes, ps: []const u16, list: *std.ArrayListUnmanaged([]u8), q: *std.PriorityQueue(Msg, void, Msg.order), nn: *u64, now: u64, dl: []Link, rnd: std.Random, ow: u64, jt: u64, vs: *std.AutoHashMapUnmanaged(u64, u64), tot: *Totals) !void { - for (ps, 0..) |peer, c| { - list.clearRetainingCapacity(); - try sc.flush(core, peer, list); - for (list.items) |bytes| { - tot.deltas += 1; - tot.delta_bytes += bytes.len; - try push(q, nn, .{ .at = dl[c].arrive(rnd, now, ow, jt), .n = 0, .kind = .screen_down, .who = @intCast(c + 1), .bytes = bytes }); - } - } - // Each version as the authority's screen showed it. - for (&sc.tracks, 0..) |*slot, id| if (slot.*) |*tr| { - const key = @as(u64, tr.serial) << 32 | tr.track.version; - if (tr.track.version == 0 or vs.contains(key)) continue; - _ = core.panes[id] orelse continue; - try vs.put(gpa, key, screenHash(&tr.state.vt)); - }; - } - }.f; while (queue.pop()) |m| { defer if (m.bytes.len > 0) gpa.free(m.bytes); @@ -432,6 +412,7 @@ fn run(r: Run, dir: []const u8, totals: *Totals) !void { } if (rp.screenRequest()) |req| try push(&queue, &n, .{ .at = up[c].arrive(random, m.at, one_way, jitter), .n = 0, .kind = .screen_up, .who = m.who, .bytes = try gpa.dupe(u8, req) }); }, + .client_frame => {}, .screen_up => { const c = m.who - 1; if (m.bytes[5] == 2) totals.fetched += std.mem.readInt(u64, m.bytes[14..22], .little) - std.mem.readInt(u64, m.bytes[6..14], .little); @@ -467,6 +448,35 @@ fn run(r: Run, dir: []const u8, totals: *Totals) !void { } } +fn push(q: *std.PriorityQueue(Msg, void, Msg.order), nn: *u64, m: Msg) !void { + var mm = m; + mm.n = nn.*; + nn.* += 1; + try q.push(gpa, mm); +} + +/// What each peer is owed of the terminals, onto its link (before the +/// authority steps an input, and once a frame); each version as the +/// authority's screen showed it. +fn flushAll(sc: *replica.Screens, core: *pardes.Pardes, ps: []const u16, list: *std.ArrayListUnmanaged([]u8), q: *std.PriorityQueue(Msg, void, Msg.order), nn: *u64, now: u64, dl: []Link, rnd: std.Random, ow: u64, jt: u64, vs: *std.AutoHashMapUnmanaged(u64, u64), tot: *Totals) !void { + for (ps, 0..) |peer, c| { + list.clearRetainingCapacity(); + try sc.flush(core, peer, list); + for (list.items) |bytes| { + tot.deltas += 1; + tot.delta_bytes += bytes.len; + try push(q, nn, .{ .at = dl[c].arrive(rnd, now, ow, jt), .n = 0, .kind = .screen_down, .who = @intCast(c + 1), .bytes = bytes }); + } + } + // Each version as the authority's screen showed it. + for (&sc.tracks, 0..) |*slot, id| if (slot.*) |*tr| { + const key = @as(u64, tr.serial) << 32 | tr.track.version; + if (tr.track.version == 0 or vs.contains(key)) continue; + _ = core.panes[id] orelse continue; + try vs.put(gpa, key, screenHash(&tr.state.vt)); + }; +} + /// A unit of the authority's to every replica, down its link: first the /// rows of its terminals its step read that a replica holds as holes. fn sendUnit(q: *std.PriorityQueue(Msg, void, Msg.order), n: *u64, unit: []const u8, now: u64, down: []Link, random: std.Random, one_way: u64, jitter: u64, totals: *Totals, flooding: bool, screens: *replica.Screens, core: *pardes.Pardes, peers: []const u16) !void { @@ -490,6 +500,695 @@ fn sendUnit(q: *std.PriorityQueue(Msg, void, Msg.order), n: *u64, unit: []const } } +// ---- predicted echo (--echo): one client typing a corpus into a shell ---- +// +// The authority's pty is a shell played here (`FakeShell`): a line editor at +// a prompt with pardes's OSC 133 marks and Tab completion, and the programs +// the corpus runs in it: a password prompt (no echo), vim (the alternate +// screen), a program that ignores what is typed, and `cat` (the kernel's +// plain echo, no marks). Each key's echo comes back 2 ms after the authority +// writes it, as a shell's. The client types at a typist's pace and in bursts; +// each of its frames (60 Hz, a phase of its own) is what its screen shows: +// the predicted echo where it is drawn (src/echo.zig), the screen state +// under it. Per key the shell echoed at a prompt or in cat: the time from +// the key to its character on that screen, with prediction and without. Per +// key predicted: whether the authority's screen, right after its echo, +// shows what was predicted where it was. Per key drawn wrong: its frames. + +const ascii: [128][1]u8 = blk: { + var t: [128][1]u8 = undefined; + for (&t, 0..) |*c, i| c.* = .{@intCast(i)}; + break :blk t; +}; + +const Step = struct { gap: u64, key: Key, section: u8 }; + +/// The corpus's parts, each a case of the gates. +const sections = [_][]const u8{ "typing", "a burst", "backspace", "left, right", "tab", "password", "vim", "ignores input", "cat", "typed ahead" }; + +const Corpus = struct { + steps: std.ArrayListUnmanaged(Step) = .empty, + random: std.Random, + extra: u64 = 0, + section: u8 = 0, + + fn add(c: *Corpus, gap: u64, key: Key) !void { + try c.steps.append(gpa, .{ .gap = gap + c.extra, .key = key, .section = c.section }); + c.extra = 0; + } + + fn typing(c: *Corpus, text: []const u8, fast: bool) !void { + for (text) |ch| try c.add(if (fast) c.random.intRangeAtMost(u64, 12_000, 60_000) else c.random.intRangeAtMost(u64, 60_000, 180_000), .{ .cp = ch, .text = &ascii[ch] }); + } + + fn press(c: *Corpus, key: Key) !void { + try c.add(c.random.intRangeAtMost(u64, 80_000, 250_000), key); + } + + fn pause(c: *Corpus) void { + c.extra += c.random.intRangeAtMost(u64, 500_000, 1_200_000); + } + + fn build(c: *Corpus) !void { + const enter: Key = .{ .cp = Key.enter }; + const bs: Key = .{ .cp = Key.backspace }; + c.pause(); + // At a prompt, at a typist's pace, then in a burst. + c.section = 0; + try c.typing("echo hello world", false); + try c.press(enter); + c.pause(); + c.section = 1; + try c.typing("echo the quick brown fox jumps over", true); + try c.press(enter); + c.pause(); + // Backspace. + c.section = 2; + try c.typing("ecoh", false); + try c.press(bs); + try c.press(bs); + try c.typing("ho fixed tpyo", false); + for (0..3) |_| try c.press(bs); + try c.typing("ypo", false); + try c.press(enter); + c.pause(); + // Left and right over what was typed. + c.section = 3; + try c.typing("echo abc", false); + for ([_]u21{ Key.left, Key.left, Key.right, Key.right }) |k| try c.press(.{ .cp = k }); + try c.typing("def", false); + try c.press(enter); + c.pause(); + // Tab completion, typed straight on. + c.section = 4; + try c.typing("ec", false); + try c.press(.{ .cp = Key.tab }); + try c.typing("tab completed", false); + try c.press(enter); + c.pause(); + // A password prompt: nothing typed there may show. + c.section = 5; + try c.typing("sudo ls", false); + try c.press(enter); + c.pause(); + try c.typing("hunter2", false); + try c.press(enter); + c.pause(); + // vim: the alternate screen, never predicted. + c.section = 6; + try c.typing("vim", false); + try c.press(enter); + c.pause(); + try c.typing("ihello from vim", false); + try c.press(.{ .cp = Key.escape, .shift = true }); + try c.typing(":q", false); + try c.press(enter); + c.pause(); + // A program that ignores what is typed. + c.section = 7; + try c.typing("sink", false); + try c.press(enter); + c.pause(); + try c.typing("this goes nowhere", false); + try c.press(.{ .cp = 'c', .ctrl = true }); + c.pause(); + // cat: the kernel's plain echo. + c.section = 8; + try c.typing("cat", false); + try c.press(enter); + c.pause(); + try c.typing("plain echo", false); + try c.press(enter); + try c.typing("a second line", false); + try c.press(enter); + try c.press(.{ .cp = 'd', .ctrl = true }); + c.pause(); + // Typed on before the next prompt. + c.section = 9; + try c.typing("echo one", false); + try c.press(enter); + try c.typing("echo two", false); + try c.press(enter); + c.pause(); + } +}; + +const prompt_bytes = "\x1b]133;A;cl=line;aid=pardes\x07$ \x1b]133;B;aid=pardes\x07"; + +/// The shell on the authority's pty, a key a write (as forwardKey writes). +const FakeShell = struct { + mode: Mode = .prompt, + line: [256]u8 = undefined, + len: usize = 0, + cur: usize = 0, + vim_cmd: [8]u8 = undefined, + vim_n: usize = 0, + out: std.ArrayListUnmanaged(u8) = .empty, + + const Mode = enum { prompt, password, vim, vim_insert, sink, cat }; + + fn put(sh: *FakeShell, bytes: []const u8) void { + sh.out.appendSlice(gpa, bytes) catch {}; + } + + fn print(sh: *FakeShell, comptime fmt: []const u8, args: anytype) void { + sh.out.print(gpa, fmt, args) catch {}; + } + + fn done(sh: *FakeShell, status: u8) void { + sh.print("\x1b]133;D;{d};aid=pardes\x07" ++ prompt_bytes, .{status}); + sh.mode = .prompt; + sh.len = 0; + sh.cur = 0; + } + + fn insert(sh: *FakeShell, c: u8) void { + if (sh.len + 1 >= sh.line.len) return; + sh.put(&.{c}); + if (sh.cur < sh.len) { + sh.put(sh.line[sh.cur..sh.len]); + sh.print("\x1b[{d}D", .{sh.len - sh.cur}); + } + std.mem.copyBackwards(u8, sh.line[sh.cur + 1 .. sh.len + 1], sh.line[sh.cur..sh.len]); + sh.line[sh.cur] = c; + sh.len += 1; + sh.cur += 1; + } + + fn key(sh: *FakeShell, b: []const u8) void { + const printable = b.len == 1 and b[0] >= 0x20 and b[0] < 0x7f; + const eql = std.mem.eql; + switch (sh.mode) { + .prompt => if (printable) sh.insert(b[0]) else if (eql(u8, b, "\x7f")) { + if (sh.cur == 0) return; + if (sh.cur == sh.len) sh.put("\x08\x1b[K") else { + sh.put("\x08"); + sh.put(sh.line[sh.cur..sh.len]); + sh.print(" \x1b[{d}D", .{sh.len - sh.cur + 1}); + } + std.mem.copyForwards(u8, sh.line[sh.cur - 1 .. sh.len - 1], sh.line[sh.cur..sh.len]); + sh.len -= 1; + sh.cur -= 1; + } else if (eql(u8, b, "\x1b[D")) { + if (sh.cur == 0) return; + sh.put(b); + sh.cur -= 1; + } else if (eql(u8, b, "\x1b[C")) { + if (sh.cur == sh.len) return; + sh.put(b); + sh.cur += 1; + } else if (eql(u8, b, "\t")) { + if (sh.cur != sh.len or std.mem.indexOfScalar(u8, sh.line[0..sh.len], ' ') != null) return; + for ([_][]const u8{ "echo", "sudo", "sink", "cat", "vim" }) |cmd| { + if (sh.len == 0 or sh.len >= cmd.len or !std.mem.startsWith(u8, cmd, sh.line[0..sh.len])) continue; + for (cmd[sh.len..]) |ch| sh.insert(ch); + sh.insert(' '); + return; + } + } else if (eql(u8, b, "\r")) { + var cmd_buf: [256]u8 = undefined; + const cmd = cmd_buf[0..sh.len]; + @memcpy(cmd, sh.line[0..sh.len]); + sh.len = 0; + sh.cur = 0; + sh.put("\r\n\x1b]133;C;aid=pardes\x07"); + if (cmd.len == 0) return sh.done(0); + if (std.mem.startsWith(u8, cmd, "echo")) { + sh.print("{s}\r\n", .{std.mem.trimStart(u8, cmd[4..], " ")}); + sh.done(0); + } else if (std.mem.startsWith(u8, cmd, "sudo")) { + sh.put("[sudo] password for you: "); + sh.mode = .password; + } else if (eql(u8, cmd, "vim")) { + sh.put("\x1b[?1049h\x1b[2J\x1b[H~\r\n~\r\n~\x1b[H"); + sh.mode = .vim; + } else if (eql(u8, cmd, "sink")) { + sh.mode = .sink; + } else if (eql(u8, cmd, "cat")) { + sh.mode = .cat; + } else { + sh.print("fake: unknown command: {s}\r\n", .{cmd}); + sh.done(127); + } + } else if (eql(u8, b, "\x03")) { + sh.put("^C\r\n"); + sh.done(130); + }, + .password => if (eql(u8, b, "\r")) { + sh.put("\r\nsudo: 1 incorrect password attempt\r\n"); + sh.done(1); + } else if (eql(u8, b, "\x03")) { + sh.put("^C\r\n"); + sh.done(130); + }, + .vim => if (sh.vim_n > 0) { + if (eql(u8, b, "\r")) { + const quit = eql(u8, sh.vim_cmd[0..sh.vim_n], ":q"); + sh.vim_n = 0; + if (quit) { + sh.put("\x1b[?1049l"); + sh.done(0); + } + } else if (printable and sh.vim_n < sh.vim_cmd.len) { + sh.vim_cmd[sh.vim_n] = b[0]; + sh.vim_n += 1; + } + } else if (eql(u8, b, "i")) { + sh.mode = .vim_insert; + } else if (eql(u8, b, ":")) { + sh.vim_cmd[0] = ':'; + sh.vim_n = 1; + }, + .vim_insert => if (eql(u8, b, "\x1b")) { + sh.mode = .vim; + } else if (printable) sh.put(b), + .sink => if (eql(u8, b, "\x03")) { + sh.put("^C\r\n"); + sh.done(130); + }, + .cat => if (printable) { + sh.put(b); + if (sh.len < sh.line.len) { + sh.line[sh.len] = b[0]; + sh.len += 1; + } + } else if (eql(u8, b, "\x7f")) { + if (sh.len == 0) return; + sh.put("\x08 \x08"); + sh.len -= 1; + } else if (eql(u8, b, "\r")) { + sh.print("\r\n{s}\r\n", .{sh.line[0..sh.len]}); + sh.len = 0; + } else if (eql(u8, b, "\x04") and sh.len == 0) { + sh.done(0); + } else if (eql(u8, b, "\x03")) { + sh.put("^C\r\n"); + sh.done(130); + }, + } + } +}; + +/// The authority's host: each pty write tagged with the client input that +/// made it. +const EchoHost = struct { + writes: std.ArrayListUnmanaged(struct { seq: u32, bytes: []u8 }) = .empty, + current: u32 = 0, + + fn watched(_: ?*anyopaque) bool { + return false; + } + + fn ptyWrite(ctx: ?*anyopaque, _: u8, bytes: []const u8) void { + const h: *EchoHost = @ptrCast(@alignCast(ctx.?)); + h.writes.append(gpa, .{ .seq = h.current, .bytes = gpa.dupe(u8, bytes) catch return }) catch {}; + } + + fn ignore(_: ?*anyopaque, _: []const u8) void {} + + const vtable: pardes.Host.VTable = .{ .watched = watched, .pty_write = ptyWrite, .set_clipboard = ignore }; + + fn host(h: *EchoHost) pardes.Host { + return .{ .ctx = h, .vtable = &vtable }; + } +}; + +const Typed = struct { + seq: u32, + section: u8, + at: u64, + printable: bool, + /// What was predicted: the cell (its row in the active area) and the + /// cursor after it. + pred: ?struct { x: u16, y: u16, cp: u21 } = null, + pcur: ?struct { x: u16, y: u16 } = null, + /// The shell's mode as it took the key; whether it printed anything for + /// it; the authority's version of the screen before that. + mode: ?FakeShell.Mode = null, + echoed: bool = false, + version_before: u32 = 0, + /// The authority's screen right after its echo: its cursor and the + /// cursor's row; whether that is what was predicted. + after: ?struct { x: u16, y: u16, row: [128]u21 } = null, + truth: ?bool = null, + /// Its prediction drawn: first, and in how many frames. + first_visible: ?u64 = null, + visible_frames: u32 = 0, + /// The screen state showing its echo, first. + seen: ?u64 = null, +}; + +const EchoTotals = struct { + keys: u64 = 0, + echo_keys: u64 = 0, + latency: std.ArrayListUnmanaged(u64) = .empty, + predicted: u64 = 0, + unpredicted: u64 = 0, + confirmed: u64 = 0, + mispredicted: u64 = 0, + cursor_missed: u64 = 0, + right: u64 = 0, + wrong: u64 = 0, + wrong_shown: u64 = 0, + wrong_frames: u64 = 0, + wrong_max: u64 = 0, + secret_frames: u64 = 0, + alt_predicted: u64 = 0, + divergences: u64 = 0, + unequal: u64 = 0, + screens_wrong: u64 = 0, + /// The replica's frame drawn while it shows a prediction (its core's + /// render, the tty's and the GUI's), and the overlay's part of it (ns); + /// cells drawn otherwise than predicted. + render_ns: std.ArrayListUnmanaged(u64) = .empty, + paint_ns: std.ArrayListUnmanaged(u64) = .empty, + drawn_frames: u64 = 0, + paint_wrong: u64 = 0, + sec: [sections.len]Section = @splat(.{}), + + const Section = struct { keys: u64 = 0, predicted: u64 = 0, wrong_frames: u64 = 0, latency: std.ArrayListUnmanaged(u64) = .empty }; + + fn deinit(et: *EchoTotals) void { + for (&et.sec) |*x| x.latency.deinit(gpa); + et.latency.deinit(gpa); + et.render_ns.deinit(gpa); + et.paint_ns.deinit(gpa); + } +}; + +fn cellCp(state: *pardes.panes.terminal.State, x: u16, y: u16) u21 { + const ci = state.vt.screens.active.pages.getCell(.{ .active = .{ .x = x, .y = y } }) orelse return 0x10ffff; + const cp = ci.cell.codepoint(); + return if (cp == 0) ' ' else cp; +} + +/// A key's prediction against the authority's screen right after its echo, +/// once both are known (a key typed behind one not predicted is predicted +/// after the authority has it). +fn judgeTruth(k: *Typed) void { + if (k.truth != null) return; + const pc = k.pcur orelse return; + const after = k.after orelse return; + var ok = after.x == pc.x and after.y == pc.y; + if (k.pred) |cell| ok = ok and cell.y == after.y and cell.x < after.row.len and after.row[cell.x] == cell.cp; + k.truth = ok; + if (trace and !ok) std.debug.print("key {d} in {t} predicted cursor {d},{d} cell {any}; the authority's cursor {d},{d}\n", .{ k.seq, k.mode.?, pc.x, pc.y, k.pred, after.x, after.y }); +} + +fn runEcho(seed: u64, rtt_ms: u64, on: bool, et: *EchoTotals) !void { + var prng = std.Random.DefaultPrng.init(seed *% 0x9e3779b97f4a7c15 +% rtt_ms); + const random = prng.random(); + var host: EchoHost = .{}; + defer { + for (host.writes.items) |w| gpa.free(w.bytes); + host.writes.deinit(gpa); + } + var sh: FakeShell = .{}; + defer sh.out.deinit(gpa); + const a = try replica.Authority.init(gpa, .{ .cols = 100, .rows = 30, .boot = .tty_shell }, host.host(), true); + defer a.deinit(); + var screens: replica.Screens = .init(gpa); + defer screens.deinit(a.core); + const peer = try screens.addPeer(); + const peers = [_]u16{peer}; + const rp = try replica.Replica.init(gpa, 1, try a.unit(0, 0), true); + defer rp.deinit(); + rp.echo.on = on; + const shell: u8 = for (a.core.panes, 0..) |slot, id| { + if (slot) |pane| if (pane.isTerminal()) break @intCast(id); + } else return error.NoShell; + var corpus: Corpus = .{ .random = random }; + defer corpus.steps.deinit(gpa); + try corpus.build(); + var typed: std.ArrayListUnmanaged(Typed) = .empty; + defer typed.deinit(gpa); + var t: Totals = .{}; + defer { + t.spec_ns.deinit(gpa); + t.copy_ns.deinit(gpa); + t.rollback_ns.deinit(gpa); + t.apply_ns.deinit(gpa); + t.frame_ns.deinit(gpa); + } + var versions: std.AutoHashMapUnmanaged(u64, u64) = .empty; + defer versions.deinit(gpa); + var queue: std.PriorityQueue(Msg, void, Msg.order) = .empty; + defer { + while (queue.pop()) |m| if (m.bytes.len > 0) gpa.free(m.bytes); + queue.deinit(gpa); + } + var n: u64 = 0; + const one_way = rtt_ms * 1000 / 2; + const jitter = one_way / 10; + var up: [1]Link = @splat(.{}); + var down: [1]Link = @splat(.{}); + var owed: std.ArrayListUnmanaged([]u8) = .empty; + defer owed.deinit(gpa); + + // The shell's first prompt. + sh.put(prompt_bytes); + a.output(.{ .output = .{ .pane = shell, .bytes = sh.out.items } }); + sh.out.clearRetainingCapacity(); + var next_step: usize = 0; + try push(&queue, &n, .{ .at = corpus.steps.items[0].gap, .n = 0, .kind = .input, .who = 1, .bytes = &.{} }); + try push(&queue, &n, .{ .at = frame_us, .n = 0, .kind = .frame, .who = 0, .bytes = &.{} }); + try push(&queue, &n, .{ .at = random.uintLessThan(u64, frame_us), .n = 0, .kind = .client_frame, .who = 1, .bytes = &.{} }); + // Two clocks tick while anything else moves, and a second after. + var periodic: usize = 2; + var left: [2]u32 = .{ 60, 60 }; + + var frame_arena: std.heap.ArenaAllocator = .init(gpa); + defer frame_arena.deinit(); + var made: std.ArrayListUnmanaged(echo.Echo.Predicted) = .empty; + defer made.deinit(gpa); + rp.echo.log = &made; + while (queue.pop()) |m| { + defer if (m.bytes.len > 0) gpa.free(m.bytes); + rp.echo.now = m.at * 1000; + // What it predicted in the last event's calls (a key typed, or the + // keys behind one it did not, at a unit, a delta or a frame). + defer { + const top = rp.echo.tops[shell]; + for (made.items) |pr| { + const k = &typed.items[pr.seq - 1]; + if (trace) { + const sc = rp.core().panes[shell].?.terminal.?.vt.screens.active.cursor; + std.debug.print("{d} ms ({t}): key {d} predicted {any} cursor {d}; acked {d} wait {d} nd {d}; the screen's cursor {d},{d}\n", .{ m.at / 1000, m.kind, pr.seq, pr.cell, pr.cursor.x, rp.echo.acked, rp.echo.wait, rp.echo.nd, sc.x, sc.y }); + } + if (pr.cell) |cell| k.pred = .{ .x = cell.x, .y = @intCast(cell.row - top), .cp = cell.cp }; + k.pcur = .{ .x = pr.cursor.x, .y = @intCast(pr.cursor.row - top) }; + judgeTruth(k); + } + made.clearRetainingCapacity(); + } + switch (m.kind) { + .input => { + const step = corpus.steps.items[next_step]; + next_step += 1; + const msg = try gpa.dupe(u8, try rp.input(.{ .key = step.key })); + const seq = rp.next_seq - 1; + try typed.append(gpa, .{ .seq = seq, .section = step.section, .at = m.at, .printable = step.key.text.len == 1 }); + if (trace) std.debug.print("{d} ms: key {d} typed: '{s}' cp {x}\n", .{ m.at / 1000, seq, step.key.text, step.key.cp }); + try push(&queue, &n, .{ .at = up[0].arrive(random, m.at, one_way, jitter), .n = 0, .kind = .to_authority, .who = 1, .bytes = msg }); + if (next_step < corpus.steps.items.len) try push(&queue, &n, .{ .at = m.at + corpus.steps.items[next_step].gap, .n = 0, .kind = .input, .who = 1, .bytes = &.{} }); + }, + .to_authority => { + if (try a.pending()) |batch| try sendUnit(&queue, &n, batch, m.at, &down, random, one_way, jitter, &t, false, &screens, a.core, &peers); + try flushAll(&screens, a.core, &peers, &owed, &queue, &n, m.at, &down, random, one_way, jitter, &versions, &t); + host.current = (replica.parseInput(m.bytes) orelse return error.BadInput).seq; + const unit = try a.take(m.bytes); + host.current = 0; + try sendUnit(&queue, &n, unit, m.at, &down, random, one_way, jitter, &t, false, &screens, a.core, &peers); + for (host.writes.items) |w| try push(&queue, &n, .{ .at = m.at + 2_000, .n = 0, .kind = .echo, .who = shell, .bytes = w.bytes, .seq = w.seq }); + host.writes.clearRetainingCapacity(); + }, + .echo => { + const ty: ?*Typed = if (m.seq > 0) &typed.items[m.seq - 1] else null; + const state = a.core.panes[shell].?.terminal.?; + if (ty) |k| { + k.mode = sh.mode; + k.version_before = if (screens.tracks[shell]) |*tr| tr.track.version else 0; + } + sh.key(m.bytes); + if (sh.out.items.len > 0) { + a.output(.{ .output = .{ .pane = shell, .bytes = sh.out.items } }); + sh.out.clearRetainingCapacity(); + if (ty) |k| k.echoed = true; + } + if (ty) |k| { + const cur = state.vt.screens.active.cursor; + k.after = .{ .x = @intCast(cur.x), .y = @intCast(cur.y), .row = undefined }; + for (&k.after.?.row, 0..) |*cp, x| cp.* = cellCp(state, @intCast(x), @intCast(cur.y)); + judgeTruth(k); + } + }, + .frame => { + if (try a.pending()) |batch| try sendUnit(&queue, &n, batch, m.at, &down, random, one_way, jitter, &t, false, &screens, a.core, &peers); + try flushAll(&screens, a.core, &peers, &owed, &queue, &n, m.at, &down, random, one_way, jitter, &versions, &t); + const busy = next_step < corpus.steps.items.len or queue.count() + 1 > periodic; + left[0] = if (busy) 60 else left[0] -| 1; + if (left[0] > 0) try push(&queue, &n, .{ .at = m.at + frame_us, .n = 0, .kind = .frame, .who = 0, .bytes = &.{} }) else periodic -= 1; + }, + .client_frame => { + _ = rp.echoFrame(); + const core = rp.core(); + const drawn = rp.echo.visible(core) and core.panes[shell].?.body.mode == .tty; + if (drawn and rp.echo.n > 0) { + // Drawn as the client draws it: each cell where it was + // predicted. + _ = frame_arena.reset(.retain_capacity); + core.needs_frame = true; + const t0 = pardes.FrameStats.nowNs(); + const surf = try core.render(frame_arena.allocator()); + try et.render_ns.append(gpa, pardes.FrameStats.nowNs() - t0); + const body = surf.region(.body, shell).?.rect; + const top = rp.echo.tops[shell]; + for (rp.echo.cells[0..rp.echo.n]) |cell| { + if (cell.row < top or surf.at(body.x + cell.x, body.y + @as(u16, @intCast(cell.row - top))).text[0] != cell.cp) et.paint_wrong += 1; + } + const t1 = pardes.FrameStats.nowNs(); + rp.echo.paint(core, surf, core.panes[shell].?, shell, body.x, body.w, body.y, body.h, true); + try et.paint_ns.append(gpa, pardes.FrameStats.nowNs() - t1); + et.drawn_frames += 1; + } + if (drawn) for (rp.echo.cells[0..rp.echo.n]) |cell| { + const k = &typed.items[cell.seq - 1]; + k.visible_frames += 1; + if (k.first_visible == null) { + k.first_visible = m.at; + if (trace) { + const st = core.panes[shell].?.terminal.?; + std.debug.print("{d} ms: key {d} ('{u}') drawn: trusted {} doubt {} phase {t} sem {t}\n", .{ m.at / 1000, cell.seq, cell.cp, rp.echo.trusted, rp.echo.doubt, st.stream.handler.phase, st.vt.screens.active.cursor.semantic_content }); + } + } + }; + const applied = if (rp.screens[shell]) |sc| sc.rep.seq else 0; + for (typed.items) |*k| if (k.echoed and k.seen == null and applied > k.version_before) { + k.seen = m.at; + }; + const busy = next_step < corpus.steps.items.len or queue.count() + 1 > periodic; + left[1] = if (busy) 60 else left[1] -| 1; + if (left[1] > 0) try push(&queue, &n, .{ .at = m.at + frame_us, .n = 0, .kind = .client_frame, .who = 1, .bytes = &.{} }) else periodic -= 1; + }, + .to_replica => { + rp.confirm(m.bytes); + rp.settle(); + }, + .screen_down => { + if (rp.screen(m.bytes)) |reply| { + const pane = m.bytes[0]; + if (reply[5] == 1) { + const sc = rp.screens[pane].?; + if (versions.get(@as(u64, sc.serial) << 32 | sc.rep.seq)) |want| { + if (want != screenHash(&rp.core().panes[pane].?.terminal.?.vt)) t.screens_wrong += 1; + } else t.screens_wrong += 1; + } + try push(&queue, &n, .{ .at = up[0].arrive(random, m.at, one_way, jitter), .n = 0, .kind = .screen_up, .who = 1, .bytes = try gpa.dupe(u8, reply) }); + } + if (rp.screenRequest()) |req| try push(&queue, &n, .{ .at = up[0].arrive(random, m.at, one_way, jitter), .n = 0, .kind = .screen_up, .who = 1, .bytes = try gpa.dupe(u8, req) }); + }, + .screen_up => { + if (try screens.take(a.core, peer, m.bytes)) |answer| try push(&queue, &n, .{ .at = down[0].arrive(random, m.at, one_way, jitter), .n = 0, .kind = .screen_down, .who = 1, .bytes = answer }); + }, + .output => {}, + } + } + const s = rp.echo.stats; + et.predicted += s.predicted; + et.unpredicted += s.unpredicted; + et.confirmed += s.confirmed; + et.mispredicted += s.mispredicted; + et.cursor_missed += s.cursor_missed; + et.screens_wrong += t.screens_wrong; + for (typed.items) |k| { + et.keys += 1; + const sec = &et.sec[k.section]; + sec.keys += 1; + if (k.pcur != null) sec.predicted += 1; + if (k.visible_frames > 0 and k.truth != true) sec.wrong_frames += k.visible_frames; + if (k.truth) |ok| { + if (ok) et.right += 1 else et.wrong += 1; + } + const md = k.mode orelse .prompt; + if (md == .password or md == .sink) et.secret_frames += k.visible_frames; + if ((md == .vim or md == .vim_insert) and k.pcur != null) et.alt_predicted += 1; + if (k.visible_frames > 0 and k.truth != true) { + if (trace) std.debug.print("seed {d}: key {d} at {d} ms in {t} drawn {d} frames, truth {?}, predicted {any} cursor {any}, the authority after it {any}\n", .{ seed, k.seq, k.at / 1000, md, k.visible_frames, k.truth, k.pred, k.pcur, if (k.after) |af| .{ af.x, af.y } else null }); + et.wrong_shown += 1; + et.wrong_frames += k.visible_frames; + et.wrong_max = @max(et.wrong_max, k.visible_frames); + } + if (k.printable and k.echoed and (md == .prompt or md == .cat)) { + const seen = k.seen orelse std.math.maxInt(u64); + const drawn = if (k.first_visible != null and k.truth == true) @min(k.first_visible.?, seen) else seen; + if (drawn == std.math.maxInt(u64)) { + std.debug.print("seed {d} rtt {d} ms: a key's echo was never drawn\n", .{ seed, rtt_ms }); + et.unequal += 1; + continue; + } + et.echo_keys += 1; + try et.latency.append(gpa, drawn - k.at); + try sec.latency.append(gpa, drawn - k.at); + if (trace and drawn - k.at > 2 * frame_us) std.debug.print("key {d} at {d} ms in {t}: drawn after {d} ms, {s}\n", .{ k.seq, k.at / 1000, md, (drawn - k.at) / 1000, if (k.pcur == null) "not predicted" else if (k.first_visible == null) "predicted, not drawn" else if (k.truth != true) "predicted wrong" else "predicted, drawn late" }); + } + } + if (rp.diverged) |why| { + et.divergences += 1; + std.debug.print("seed {d} rtt {d} ms: the replica diverged: {s}\n", .{ seed, rtt_ms, why }); + } else if (!rp.quiet() or rp.digest() != replica.digestOf(a)) { + et.unequal += 1; + std.debug.print("seed {d} rtt {d} ms: the replica at quiescence: {s}\n", .{ seed, rtt_ms, if (rp.quiet()) "another digest" else "inputs still pending" }); + } + if (screenHash(&a.core.panes[shell].?.terminal.?.vt) != screenHash(&rp.core().panes[shell].?.terminal.?.vt)) { + et.unequal += 1; + std.debug.print("seed {d} rtt {d} ms: the shell at quiescence is not the authority's\n", .{ seed, rtt_ms }); + } +} + +fn echoMain(seeds: u64, first: u64, rtts: []const u64) !bool { + var failed = false; + var draw_lines: std.ArrayList(u8) = .empty; + defer draw_lines.deinit(gpa); + std.debug.print("predicted echo: one client typing the corpus into a shell, {d} seeds a cell; from a key to its character on the client's screen (ms, frames at 60 Hz)\n", .{seeds}); + std.debug.print(" rtt echo keys echoed p50 p90 max predicted unpredicted right/wrong confirmed mispredicted cursor-missed drawn-wrong keys/frames/max secret-frames vim-predicted diverged unequal screens-wrong\n", .{}); + var by_section: std.ArrayList(u8) = .empty; + defer by_section.deinit(gpa); + var off: [sections.len][2]u64 = undefined; + for (rtts) |rtt| for ([_]bool{ false, true }) |on| { + var et: EchoTotals = .{}; + defer et.deinit(); + for (first..first + seeds) |seed| try runEcho(seed, rtt, on, &et); + const ms = struct { + fn f(us: u64) f64 { + return @as(f64, @floatFromInt(us)) / 1000; + } + }.f; + std.debug.print("{d:>4} {s:>4} {d:>4} {d:>6} {d:>5.1} {d:>5.1} {d:>5.1} {d:>9} {d:>11} {d:>5}/{d:<5} {d:>9} {d:>12} {d:>13} {d:>5}/{d:>5}/{d:<5} {d:>13} {d:>13} {d:>8} {d:>7} {d:>13}\n", .{ + rtt, if (on) "on" else "off", et.keys, et.echo_keys, + ms(pct(et.latency.items, 50)), ms(pct(et.latency.items, 90)), ms(pct(et.latency.items, 100)), et.predicted, + et.unpredicted, et.right, et.wrong, et.confirmed, + et.mispredicted, et.cursor_missed, et.wrong_shown, et.wrong_frames, + et.wrong_max, et.secret_frames, et.alt_predicted, et.divergences, + et.unequal, et.screens_wrong, + }); + for (&et.sec, &off, sections) |*x, *o, name| { + const p50 = pct(x.latency.items, 50) / 1000; + const p90 = pct(x.latency.items, 90) / 1000; + if (!on) { + o.* = .{ p50, p90 }; + continue; + } + try by_section.print(gpa, "{d:>4} {s:<14} {d:>5} {d:>9} {d:>7} {d:>4} -> {d:<4} {d:>4} -> {d:<4} {d:>12}\n", .{ rtt, name, x.keys, x.predicted, x.latency.items.len, o[0], p50, o[1], p90, x.wrong_frames }); + } + if (on) try draw_lines.print(gpa, "{d:>4} {d:>12} {d:>6}/{d:<6} {d:>5}/{d:<5} {d:>10}\n", .{ rtt, et.drawn_frames, pct(et.render_ns.items, 50) / 1000, pct(et.render_ns.items, 99) / 1000, pct(et.paint_ns.items, 50), pct(et.paint_ns.items, 99), et.paint_wrong }); + if (et.divergences > 0 or et.unequal > 0 or et.screens_wrong > 0 or et.secret_frames > 0 or et.alt_predicted > 0 or et.paint_wrong > 0) failed = true; + }; + std.debug.print("by part of the corpus (all seeds): keys typed, predicted, echoed (the latency sample); key to its character on the screen, ms, without -> with; frames a wrong prediction was drawn\n rtt part keys predicted echoed p50 off/on p90 off/on drawn-wrong\n{s}", .{by_section.items}); + std.debug.print("the client's frames drawn with a prediction shown: its render (us) and the overlay's part (ns); cells drawn otherwise than predicted\n rtt frames-drawn render p50/p99 paint p50/p99 drawn-wrong\n{s}", .{draw_lines.items}); + return !failed; +} + + fn pct(samples: []u64, p: usize) u64 { if (samples.len == 0) return 0; std.mem.sort(u64, samples, {}, std.sort.asc(u64)); @@ -504,6 +1203,7 @@ pub fn main(init: std.process.Init) !void { var clients: u16 = 3; var rtts: []const u64 = &.{ 30, 80, 207 }; var shared = false; + var echo_mode = false; var flood_mb: u64 = 8; var rtt_buf: [8]u64 = undefined; var i: usize = 1; @@ -513,6 +1213,10 @@ pub fn main(init: std.process.Init) !void { shared = true; continue; } + if (std.mem.eql(u8, name, "--echo")) { + echo_mode = true; + continue; + } if (i + 1 >= args.len) return error.MissingValue; i += 1; const value = args[i]; @@ -530,6 +1234,10 @@ pub fn main(init: std.process.Init) !void { const dir = try std.fmt.bufPrint(&dir_buf, "{s}/replicas-{d}", .{ tmp, @as(u32, @intCast(libc.getpid())) }); try std.Io.Dir.createDirAbsolute(init.io, dir, .default_dir); defer std.Io.Dir.cwd().deleteTree(init.io, dir) catch {}; + if (echo_mode) { + if (!try echoMain(seeds, first, rtts)) std.process.exit(1); + return; + } var failed = false; std.debug.print("{d} seeds x {d} inputs from each of {d} clients, {s}, a TUI then {d} MB of yes in the shell; costs in us (frame {d})\n", .{ seeds, steps, clients, if (shared) "sharing one view" else "a view each", flood_mb, frame_us }); std.debug.print(" rtt inputs at-once stopped view units fast rollback(mis+foreign) rate rebase spec p50/p99 copy p50/p99 (KB) rollback p50/p99 diverged unequal\n", .{}); -- cgit v1.3 From 1f5ef5bc77f12b721864c2785d9daedc82947e57 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 03:32:02 -0300 Subject: docs/next-daemon.md: phase 5 part B: the decisions at its forks (the stream, terminals as screen state in rterm's order and the wait's split, the taped prompt, predicted echo wired, the join's state in the step and its blobs streamed pinned, the texts hashed first, the rejoin, an Upgrade's words, ssh -C), the main test on the FreeBSD box, keystroke to present at 0/30/80/207 ms and through the box, echo with and without prediction, the visible rollback rate, the join's cost before and after, the join through the box plain and compressed, the step while a client lags, the local A/B against base and main, the gates, what was found and what is not done (a GUI client first) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 373 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 372 insertions(+), 1 deletion(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 5039c81e..f6400770 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1870,6 +1870,376 @@ Follow-ups (2026-10-29): p99 0.27 ms (0.22 before, within the run's noise); none diverged, none unequal, no delta wrong. +## Phase 5 part B notes + +Built 2026-10-29 and 30: the stream over the replica socket (src/replica.zig +`Peers`, and its client half, `dial` and `readStream`; detached/wire.zig +version 15), the client process (`pardes --attach=:` and +`--attach=:`, tty.zig `remote`), the session's side of it in +the tty host, rterm's screen-state terminals and the echo agent's +predicted echo carried by it, a client's rejoin, and test/daemon.py's +replica clients. Decisions taken at the forks, with the ponytail lens: + +- **A client's input is a unit of its own, settled by `step`.** On the + authority, at the step boundary after each pump (`Peers.boundary`), each + input that came is `toView(its view)`, `update(it)` and `step()`: the + middle of a pump (queued inputs and effects), with no wait, no clock + read and no frame. The client's speculation is the same three calls, so + the two tapes read alike and the fast path takes the unit as it came. + With the pump in its place (part A's harness) every speculation stopped + at the tty host's `wait_input` and read the clock in ns. +- **The stream.** Framed as wire.zig frames everything, a tag and a u32le + length: + - the session's: 0x01 welcome (pid, name, the client's id and view, the + dump's length, the session's offset from UTC, its digest as dumped + and its build's semantics), 0x02 refused (words), 0x04 a piece of the + dump (1 MiB), 0x05 a unit (origin, seq, the authority's realtime and + monotonic clocks when it was made, its tape), 0x06 a terminal's + message (rterm's: a delta or rows), 0x07 a check's answer; + - the client's: its inputs as the existing event tags (their seq is the + n-th on the connection), 0x04 a terminal's message (rterm's: an + acknowledgement, rows asked, a reset), 0x05 check. + + A unit carries one digest, at its end (`unit_digests`); under screen + state it is one word (`whole_digest`, PARDES_REPLICA_TRACE for one a + field). Blobs are hashed once and folded in as a word (main's + walk.zig), so a digest a unit is cheap. The draft's lengths-only digest + went with that. +- **Terminals cross as screen state, in rterm's order.** The session's + Peers keep a `Screens`, a peer each. At each boundary: the rows the + pump's steps read that a client holds as holes (`fill`, then `filled`), + the pump's unit, then the deltas (`flush`). Before each client input's + step its deltas again, after it the rows it read, then its unit. A + joiner's terminals start afresh after its dump. The client applies a + delta or rows in stream order, acknowledges, and asks for the history + rows a frame shows that it lacks. Units carry no pty bytes. + - **A wait takes one step of a person's input, and none after a pty's + output** (tty.zig `split`), while clients replicate: a step reads the + terminals as the deltas after each unit bring them, and in a pump that + took a pty's output and then a key, the key would read on the session + what the client has only after the unit. The next is held for the + next wait. Only while a client is attached. + - **A step that reads a terminal's prompt after its output is taped** + (tape.zig `.prompt`): the first prompt's greeting (`ls`, after a pty + resize) and a command waiting for the prompt. Found at once by a + client opening a shell of its own: the session's layout after the + shell's first output saw the prompt, and the client's replay, its + emulator a delta behind, did not. + - A dump-loaded client's emulators are on the C heap, as + `terminal.create` makes a screen replica's (walk.zig `.alloc` of a + pane's `terminal`, dump.zig `terminal_gpa`). +- **Predicted echo is the echo agent's** (src/echo.zig), wired as its + notes say: the client's core points at its replica's `echo`, a frame + judges it before drawing, and a wake comes a frame later while + predictions are out. PARDES_NO_ECHO turns it off. +- **Join, at a step boundary, the step paying for the mutable state + alone.** The view is opened and the session dumped without its blobs + (walk.zig `Dump.deferred`: a `.blob`'s texts of 4 KiB and up, noted + where they go) and without its terminals' emulators (their size: the + first delta brings each). The client's writer thread writes the + welcome, then the dump with the blobs put back where they go: what the + client reads is `dumpBytes`' dump byte for byte but for the terminals + (a test splices one of a session without any and compares), and the + client is unchanged. Each blob is pinned until it is + written (memory.zig `pin`: its free is put off; a blob is never written + in place and is freed raw). The texts the session never digested (it + records nothing until a client comes) would be hashed by the join's + first digest, 250 MB for a 100 MB file and its undo: they are hashed + first on a thread of their own (walk.zig `unhashed`, `makeWords`), + pinned meanwhile, and the join waits a boundary for them; a second + round joins whatever is left. +- **A joiner checks its load.** The welcome carries the session's digest + as dumped; a client whose load differs says so and ends. The first unit + after a join says the host and its writes again (`tape.resync`), which a + core made from a dump has not. Found on the way: the dump's reader took + the first unit with the dump's last piece and dropped it (the client + diverged at its first digest about one join in three); what it read past + the dump is the stream's now (`Framer`). +- **A diverged client joins again**, the same join as the first: its link + ended, the session dialled again, its dump loaded, its view's size sent, + in the same terminal, and a line over its frame until its next key says + why ("joined again: this client diverged from the session (...)"). It + gets a new view: the old one went with its link. PARDES_TEST_DIVERGE= + puts a client's core off the session's (a digested counter) once quiet + after its n-th unit. A divergence shows only at a unit the client + replays (another's input, or its own it did not apply): the fast path + compares records, not digests. +- **An Upgrade lets its clients go in words** (`Peers.refuseAll`, before + the dump): each is sent `refused` ("the session is upgrading to another + build of pardes") after what it was sent, for up to 200 ms, its view + closed. The client says it in its line and dials again, every 100 ms + for 20 s (the exec, the load, the daemon's own exec). A daemon's + refusal for another build, or a welcome whose semantics are not its + own (the daemon judged the hello by the hash it last heard, which an + upgrade under way may not have said yet), ends it with "session + upgraded to a newer pardes: upgrade this client too". "A newer pardes + is installed" is now a Host call the core makes in its focus step + (`newer_build`), so a session's clients take the line from the tape. +- **The session's clock, guessed.** A unit carries the authority's clocks + when it was made; a speculation's clock reads answer the last unit's + reading plus the time since it came plus the client's round trip + (`tape.Guess`): a file's mtime (realtime seconds), a message's time of + day (in the session's zone, its offset in the welcome) and a message's + monotonic ms. Never the client's own wall clock. +- **Mostly no rollback for the session's own pumps.** Every client input + woke the session's pump before its own unit, and that pump advanced the + clock: a foreign unit, a rollback a key. Two changes: a pump woken only + by a replica's post reads no new time (`mail_only`: the tty's `now` + answers the core's own), and a replica keeps what it applied over a + unit of another's that ends on the digest the last one ended on + (`neutral`; such a unit is replayed when the copy is next put back). A + unit that says what the host wrote, or which host it is, is never + neutral. +- **The copy is taken while idle.** The client takes its rollback copy + after drawing, when nothing is queued (`Replica.ready`), and lets it go + when a unit comes first: a key no longer pays a megabyte's copy (memcpy + was a third of the client's profile). The epoch's and the copy's maps + hash an address by a multiply, not Wyhash (the other sixth). +- **Each client a view, drawn by its own process.** `Pardes.undrawn` marks + a replica client's view: the session's pump and `drawOthers` skip it. + The client draws its own view between steps (`views.drawOne`), presenting + through the tty Shell's own `present`; what that writes in the core (a + frame skipped, one owed) is put back. A drawn parked view gives its + selections back to its slot (`drawParked`). +- **The client is the tty host with its inputs fed elsewhere.** Shell + gains `remote`; `feed` sends a terminal's input to the replica engine + and the session instead of to the core; quit, hangup and the size are + the link's. Its replica core's allocator is the rollback epoch; images' + and PDFs' are the C heap's, whose workers allocate on threads of their + own. ponytail: a picture's pixels freed while a copy is held are not + put off. +- **Every kind of input is speculated** but one into a terminal (rterm's + rule: its step reads the emulator), a view's own (a resize, the + pointer) too. +- **`unattended` is a watched field**, and a session with a client is + attended: its messages go by keys, not the clock. +- **Detach in a client ends that client**, a client's death too (its + link's end); the session and the others carry on. A Restore lets every + client go (they attach again). +- **A lagging client never holds the session up.** Each client has a + writer thread and a queue the session appends to (units coalesce there + into one write); one whose queue passes 64 MiB is cut off. rterm's + deltas skip states within that (32 in flight). +- **ssh: a terminal asked for, not -N.** `--attach=:` runs + `$PARDES_SSH -tt -q -L : 'exec sleep ...'`: only an + interactive connection gets TCP_NODELAY from ssh and sshd. With -N, + Nagle held each key of a burst for the last one's ACK, and a shell's + echo through the box took two round trips (413 ms) where it takes one. + `$PARDES_REMOTE_RUNTIME` names the remote runtime directory when it is + not this one's path. +- **`ssh -C` by default** when `$PARDES_SSH` is not set (it is then the + whole command). Through the box a 3.9 MB join took 7.8 to 7.9 s plain + and 5.4 to 5.8 s compressed. Of a join's time ssh's own forward is 2.0 s + (connect, key exchange, authentication, a pty) and opening a channel + 380 ms; a ControlMaster in the user's ssh config would skip the first. + A rejoin keeps its forward. + +Measured. ReleaseFast; timings alone on the perf cores (lock.sh), the +box's through its own link (207 ms, 170 to 400 KB/s from it). The harness +scripts are in `.scratch/daemon/remote/t2/`. + +- **The main test** (`e2e_box2.py`): a session on the FreeBSD box with a + 400-line file, two replica clients here through `ssh -L` each, and a + terminal on the box itself attached as a view (`pardes --attach` there + over `ssh -tt`). Each types into the file; A and B move and scroll; A + opens a shell, runs a command in it and exits it; B is hung up; A and + the box's terminal detach. A's and B's digests were the session's at + every quiet point (every check agreed), the session's text had + all three's words, the shell's answer was on A's screen 214 to 224 ms + after Enter, and nothing was left on the box once its idle daemon had + gone (a few seconds). The clients' first frames + came 3.4 to 4.0 s after they started (ssh's forward, below). +- **Keystroke to present on a client, local-only actions** (base's + `pardes-perf-live --live`, its keys on a 50k-line file; the client of a + session detached on that file, through test/delay_relay.py at each RTT + and through the box's link; two rounds interleaved; p50 / p99, µs): + + | an idle machine, p50 (p99), µs | key-down | key-right | edit-char | page-down | wheel | + |---|---|---|---|---|---| + | a local session, this line | 2070 / 2067 (2926) | 1507 / 1502 (1895) | 4744 / 4708 (7141) | 1997 / 2033 (5486) | 2098 / 2126 (2885) | + | a local session, base | 2059 / 2068 (2678) | 1517 / 1536 (2391) | 4755 / 4695 (7464) | 1916 / 1927 (5441) | 2100 / 2108 (2642) | + | client, direct | 1903 / 1917 (6813) | 1644 / 1650 (2136) | 4477 / 4413 (7912) | 1994 / 2244 (7405) | 1934 / 2063 (6793) | + | client, relay 30 ms | 1849 / 1838 (6910) | 1576 / 1557 (2695) | 4465 / 4300 (15560) | 1755 / 1712 (7682) | 1860 / 1757 (7163) | + | client, relay 80 ms | 1799 / 1753 (7281) | 1532 / 1501 (2034) | 3085 / 2654 (13705) | 1778 / 1743 (6386) | 2106 / 1870 (7692) | + | client, relay 207 ms | 1778 / 1814 (6721) | 1453 / 1478 (1887) | 4309 / 4333 (18092) | 4640 / 1745 (7562) | 1963 / 1978 (7315) | + | client, the box, ssh -C | 1773 (6947) | 1471 (2138) | 4441 (14018) | 3219 (7503) | 2163 (7692) | + + The same harness while builds ran on the other cores (the line before + its last two changes, which do not touch a client's keys; one round): + + | builds on the other cores, p50 (p99), µs | key-down | key-right | edit-char | page-down | wheel | + |---|---|---|---|---|---| + | a local session, this line | 464 (574) | 343 (417) | 1190 (1714) | 370 (1251) | 471 (638) | + | a local session, base | 463 (629) | 349 (499) | 1188 (1663) | 399 (2078) | 2096 (2591) | + | client, direct | 433 (1266) | 389 (441) | 1381 (2071) | 1183 (1400) | 451 (1360) | + | client, relay 30 ms | 413 (1243) | 364 (537) | 1332 (4083) | 1163 (1254) | 421 (1293) | + | client, relay 80 ms | 421 (1379) | 357 (500) | 1104 (1815) | 1143 (1359) | 1112 (1487) | + | client, relay 207 ms | 398 (1228) | 339 (444) | 1145 (6882) | 1151 (1263) | 615 (1469) | + + - **A client's local-only key is as fast as a local session's at any + RTT, the box's link included**: its p50s are the local session's, + whatever the delay. Its p99 is 1 to 3x the local one's: the rollbacks + (another's unit before its pending inputs; p99 1.6 to 17 ms in the + client's own count) land between some keys and their frames. + - **The machine's state moves every cell together** by up to 4.5x: on + an idle machine the cores sleep deeply between keys (the `powersave` + governor), and with builds on the other cores they do not; in a run + made while one build ran, half the cells were in each state. Within + one run, base and this line's local session read alike, and the + clients alike with them. + - Page-down on a client costs 1.15 ms where a local one costs 0.37 when + the machine is awake: a client draws outside the pump, so the render's + paint-ahead (`Stream.askAhead`, asked of the host after a pump's + present) is never asked, and the rows a page-down brings are + highlighted in its frame. + +- **Terminal echo** (`echo2.py`: a key written to the client's terminal, + to the first output holding its glyph; 30 keys; p50 / p90, ms): + + | | file | shell, predicted | shell, not predicted (PARDES_NO_ECHO) | + |---|---|---|---| + | a local session, base | 0.5 / 0.5 | | 0.8 / 0.8 | + | a local session, this line | 0.5 / 0.5 | | 0.8 / 0.8 | + | client, relay 30 ms | 0.6 / 0.6 | 0.8 / 0.9 | 31.4 / 31.4 | + | client, relay 80 ms | 0.5 / 0.5 | 0.5 / 0.8 | 81.4 / 81.6 | + | client, relay 207 ms | 0.5 / 0.5 | 0.5 / 0.5 | 208.4 / 208.6 | + | client, the box, ssh | 0.5 / 0.6 | 0.5 / 0.6 | 206.9 / 207.4 | + | client, the box, ssh -C | 0.5 / 0.5 | 0.5 / 0.6 | 215.8 / 216.9 | + + Predicted, a key into a shell at a prompt shows as fast as a local one + (it is drawn by the client's own frame); not predicted, it is one round + trip and a millisecond and a half. Before this round (the draft's) the + echo through the box was 211 ms p50. +- **The visible rollback rate.** A frame drawn after one of the client's + own inputs was recorded otherwise is counted as a rollback that may have + been seen (an upper bound: the frame shows the inputs after it too). + In the keys harness, of 669 inputs a run (two runs each): none at 0 + and 30 ms; 2 and 3 (0.4%) at 80 ms; 11 and 13 (1.6 to 1.9%) at 207 ms, + of 25 and 29 mispredicted (a key the session recorded otherwise than + the client applied it). Every other rollback (another's unit before the + client's pending inputs) is put back and replayed before the next frame. +- **Join cost on the session** (`join2.py`: three clients join a session + one after another, through the Unix socket; the session's worst + boundary, which holds the join, and the client's first frame): + + | session | its dump | of it, made in the step | worst step, before | worst step, after | first frame | + |---|---|---|---|---|---| + | typical: 4.1 MB of files, 1 MB of scrollback, 6 panes | 8.1 MB | 113 KB | 12.4 ms | 1.6 ms | 22 to 46 ms | + | big: 38.6 MB of files, 8 MB of scrollback, 14 panes | 75.6 MB | 122 KB | 51.3 ms | 1.5 ms | 87 to 117 ms | + | a 100 MB file beside typical's (130 MB of files) | 254 MB | 114 KB | 172.5 ms | 1.6 ms | 247 to 288 ms | + + Each join itself took 0.3 to 1.4 ms of a step. The dump is about twice + the files: their undo snapshots are blobs too. +- **Join through the box** (`boxjoin.py`: a session there with five of the + tree's source files open, 3.5 MB, and a shell with 1 MB of scrollback; + a 3.9 MB dump; two joins each): the client's first frame 7.8 and 7.9 s + through plain ssh, 5.4 and 5.8 s through `ssh -C` (the default now); the + session's worst step 0.9 ms. A 16 KB session's first frame takes 3.4 to + 4.0 s of which ssh's forward is 2.0 s and its channel's opening 380 ms. +- **The session's step while a client lags** (`lag2.py`: a shell printing + a line a millisecond for 12 s, client A typing a key every 30 ms, client + B stopped with SIGSTOP and cut off at a 256 KB lag; µs, p50 / p99 / + max): A's inputs 47 / 150 / 213 and the boundaries 66 / 208 / 681 with B + stopped (cut off: yes), against 60 / 134 / 309 and 69 / 276 / 4072 with + B running. +- **Digests at quiescence**: every client of every test above ended on + the session's digest (PARDES_REPLICA_CHECK): the main test's, the + clients test's four (one rejoined after a forced divergence), and the + upgrade test's client that joined again. + +- **No local regression.** With no client attached the recorder costs an + atomic load a pump (`Peers.boundary`), a flag test an event (`split`) + and one a raw free (`memory.pinned`); a session's own core has no echo. + ReleaseFast, interleaved base new base new base on the perf cores, the + median of each side's runs, new/old, beside an A/A of the old: + - keystroke to present (base's `pardes-perf-live --live`, against base, + present / core): + + | | new/base | A/A | + |---|---|---| + | key-down | 0.98 / 0.98 | 0.99 / 1.01 | + | key-right | 0.86 / 0.84 | 1.00 / 0.99 | + | edit-char | 0.76 / 0.86 | 1.01 / 1.00 | + | page-down | 0.81 / 0.81 | 0.91 / 0.93 | + | wheel | 0.89 / 0.88 | 0.96 / 1.00 | + | `yes \| head -c 32M` wall | 0.98 | 1.06 | + | a 1M-line open | 1.03 | 1.32 | + + - the core in process (`pardes-perf --reps 3`, each tree's own build of + it), against main (ykrsossp): the keys, page-down and wheel 0.92 to + 1.08 over the small, medium and large files; the edits 0.91 to 1.05; + the opens 0.93 to 1.02; body-2m 1.02 to 1.26 (small's 1.26 alone); the + scrollback cells 0.93 to 1.05. The A/A read 0.79 to 1.46 (most within + 0.9 to 1.1). Against base the keys read 1.11 to 1.20 and the + scrollback renders 1.68 to 2.18: main's since base (this line is level + with main), and base's harness is an older perf.zig, so that pair + measures the harness too. + +The gates, once at the end (Debug unless said): +- `zig build unit-test -Dplatform=tty` passes, its perf gate too; + `-Dplatform=gui` 1563 of 1575, 10 skipped, and the two that fail on main + already (`tagline pointer mapping...`, `a tag caret past its row's + end...`). +- `zig build snap`: 113 of 113. `zig build replay`: the 113 scripts and 20 + monkey seeds, none panicked or diverged. `zig build replicas`: 4 seeds + at each RTT, none diverged, none unequal, no delta wrong. +- `test/daemon.py` (20 cycles, its replica clients' test included: four + clients, one through the relay, one rejoined after a forced divergence, + one a shell's, a terminal attached while a client's view was live), + `test/daemon.py --upgrade` (an unstamped build to a stamped one, its two + replica clients included) and `test/fs.py` (49 s) pass. +- core-test's new tests: a join's dump with its blobs spliced back is the + session's dump byte for byte; a pinned blob's free is put off until its + last pin goes; and the replica tests (`-Dtest-filter=replica`). + +Found on the way: +- **A terminal attached while a client's view is live drew nothing.** + The tty's attach took the live view's screen, and a client's view has + none in the session: it returned without a word, its stub waiting (the + main test's terminal on the box). It takes the session's own now; the + clients test attaches one while a client's view is live. +- **The session's layout read a terminal's prompt after its output** + (above, `.prompt`): a client opening a shell diverged at its first + prompt. +- **A divergence shows only at a unit the client replays.** The fast path + compares a speculation's records with the unit's, not digests, so a + client off the session's state that only types on stays off until a + unit it did not apply comes (another's input, or its own it waited + for). The forced-divergence test has another client type. +- **Harness traps** (the scripts, not pardes): a terminal on the box + through `ssh -tt` from a pty must have its size before ssh starts, else + the remote pty is 0 by 0 and the session draws nothing in it; an Esc + and the next key written within the input parser's lone-Esc window read + as an Alt chord; a ReleaseFast build stamped with another commit has + another semantics hash than the box's of the same tree (`-Dstamp-commit + =false` for both); keys a test types into a client whose live pane is a + shell in tty mode go to the shell (end it with a hangup). + +Not done, and why: +- **A GUI client**, and a window session serving replicas (refused in + words). The GUI host feeds its core from 52 places, each to route + through the replica as the tty's `feed` does; its frame loop is the + pump's (the client draws between steps, outside it); and its own + presentation (the window's surface, its animations) would have to come + out of the core's step as the tty's present does. A window client of a + tty session is the main test's other half; it is the next piece. +- Paint-ahead on a client (above: page-down 1.15 ms where a local one is + 0.37): its draw would ask for it as a pump does, the worker's rows taken + in as the client's own events, outside the session's tape. +- A rejoin gets a new view: the old one's layout and focus go with its + link. A rejoin that kept the view would take the view's number in the + hello. +- A notice of the session's own while clients replicate goes through the + tape (the upgrade's line); a host that writes core state outside a step + (the GUI's notice) would diverge them, and the GUI serves none. +- The join waits for the session's texts to be hashed once a session (its + first join, or after new texts): the joiner pays some milliseconds + more, never the session. +- A held paste (`split`) is dropped if the session ends with it held. +- A `--attach=:` client ended by a hangup leaves its forward's + socket file in $TMPDIR (its ssh goes with the terminal; the unlink is on + the normal way out only). + ## Phase 6: upgrade notes Built 2026-10-29 (`src/upgrade.zig`, the tty and GUI hosts, `daemon.zig`, @@ -2543,4 +2913,5 @@ Not done, and why: harness checks in the rendered surface, and its build compiles and passes the echo tests. - The remote client (the other line) wires it: `initDump`'s core, the - frame hook, its `moved`. + frame hook, its `moved`. (Done since: Phase 5 part B notes, with the + echo through the FreeBSD box's link.) -- cgit v1.3 From 5cea19dfded7027090234bbe41fdaac3d379e012 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 03:48:41 -0300 Subject: terminal: a shell's cells painted from the row's own pin (cellAt), not getCell's walk from the viewport's top to the row for every cell: a frame of a 120x40 terminal over 64 KB to 8 MB of scrollback 107-127 -> 94-102 us against base, ReleaseFast; the 'regression' phase 5B read against base was a Debug-core perf binary (core_optimize debug, terminal safe) beside a ReleaseFast one, file gestures level once both are ReleaseFast Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 10 +++++++--- src/terminal.zig | 18 ++++++++++++++---- 2 files changed, 21 insertions(+), 7 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index f6400770..ce828b89 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2171,9 +2171,13 @@ scripts are in `.scratch/daemon/remote/t2/`. the opens 0.93 to 1.02; body-2m 1.02 to 1.26 (small's 1.26 alone); the scrollback cells 0.93 to 1.05. The A/A read 0.79 to 1.46 (most within 0.9 to 1.1). Against base the keys read 1.11 to 1.20 and the - scrollback renders 1.68 to 2.18: main's since base (this line is level - with main), and base's harness is an older perf.zig, so that pair - measures the harness too. + scrollback renders 1.68 to 2.18, but that pair was wrong: both of + this line's core benchmarks were built at the default `-Doptimize` + (their `--json` identity says `core_optimize` debug and + `terminal_optimize` safe; the driver alone is ReleaseFast) and base's + is ReleaseFast throughout. Built ReleaseFast, main's file gestures are + level with base, and a terminal's frame is faster than base's since + `terminal.cellAt` (the final gate's notes, at the end). The gates, once at the end (Debug unless said): - `zig build unit-test -Dplatform=tty` passes, its perf gate too; diff --git a/src/terminal.zig b/src/terminal.zig index 42268f0a..acff68f4 100644 --- a/src/terminal.zig +++ b/src/terminal.zig @@ -2313,7 +2313,7 @@ pub fn recolorAnsi(p: *Pardes, s: *Surface, pane: *Pane, r: pardes.Rect, tx: u16 crossed = true; const upto = anchor.shell_len - @min(anchor.suffix, anchor.shell_len); while (sb < upto) { - const ci = pages.getCell(.{ .viewport = .{ .x = gc, .y = @intCast(vi) } }) orelse break; + const ci = cellAt(pages, row_pin, gc) orelse break; sb += dumpedBytes(row_pin, ci.cell); gc = std.math.add(u16, gc, 1) catch break; } @@ -2324,7 +2324,7 @@ pub fn recolorAnsi(p: *Pardes, s: *Surface, pane: *Pane, r: pardes.Rect, tx: u16 var covered: usize = 0; var style: ?pardes.CellStyle = null; while (covered < want) { - const ci = pages.getCell(.{ .viewport = .{ .x = gc, .y = @intCast(vi) } }) orelse break; + const ci = cellAt(pages, row_pin, gc) orelse break; if (style == null and ci.cell.wide != .spacer_tail and ci.cell.wide != .spacer_head) style = cellStyle(p, ci, filtered, scnm); covered += dumpedBytes(row_pin, ci.cell); @@ -2333,7 +2333,7 @@ pub fn recolorAnsi(p: *Pardes, s: *Surface, pane: *Pane, r: pardes.Rect, tx: u16 // spin on a row that ends in one. if (covered == 0 and gc >= pane.cols) break; } - while (pages.getCell(.{ .viewport = .{ .x = gc, .y = @intCast(vi) } })) |t| { + while (cellAt(pages, row_pin, gc)) |t| { if (t.cell.wide != .spacer_tail) break; gc = std.math.add(u16, gc, 1) catch break; } @@ -2350,7 +2350,7 @@ pub fn recolorAnsi(p: *Pardes, s: *Surface, pane: *Pane, r: pardes.Rect, tx: u16 var tail: ?pardes.CellStyle = null; if (anchor.whole() or anchor.suffix > 0) { while (sc < tw) { - const ci = pages.getCell(.{ .viewport = .{ .x = gc, .y = @intCast(vi) } }) orelse break; + const ci = cellAt(pages, row_pin, gc) orelse break; gc = std.math.add(u16, gc, 1) catch break; if (ci.cell.wide == .spacer_tail) continue; const cell = s.at(tx + sc, body_y + vr); @@ -2371,6 +2371,16 @@ pub fn recolorAnsi(p: *Pardes, s: *Surface, pane: *Pane, r: pardes.Rect, tx: u16 } } +/// The cell at column `x` of the row `row` pins, as `pages.getCell` reads +/// it, without walking from the viewport's top down to the row again: that +/// walk, once a cell, was most of a terminal's paint (once getCell was no +/// longer inlined, a frame of a 120x40 shell took 75 µs more). +fn cellAt(pages: *const ghostty_vt.PageList, row: ghostty_vt.Pin, x: u16) ?ghostty_vt.PageList.Cell { + if (x >= pages.cols or x >= row.node.cols()) return null; + const rac = row.node.page().getRowAndCell(x, row.y); + return .{ .node = row.node, .row = rac.row, .cell = rac.cell, .row_idx = row.y, .col_idx = x }; +} + fn cellStyle(p: *Pardes, ci: ghostty_vt.PageList.Cell, filtered: ?*FilteredColors, scnm: bool) pardes.CellStyle { const style = ci.style(); var cs: pardes.CellStyle = .{ -- cgit v1.3 From 16f4a216e45fe8425d594d04803287eb3557bdf9 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 05:28:34 -0300 Subject: docs/next-daemon.md: the final gate's notes: the three found on the way (a terminal's paint walking the page list a cell, glibc's tcgetattr past Zig's termios, an optional slice loaded undefined in ReleaseSafe), the gates in Debug and against the installed ReleaseSafe binaries, the FreeBSD main test, the A/B against base, what was installed Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index ce828b89..6e47bd06 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2919,3 +2919,79 @@ Not done, and why: - The remote client (the other line) wires it: `initDump`'s core, the frame hook, its `moved`. (Done since: Phase 5 part B notes, with the echo through the FreeBSD box's link.) + +## The final gate notes + +Run 2026-10-30 on main with every phase merged (rqzuvput, then the three +fixes below), in the default workspace. Scripts and logs are in +`.scratch/daemon/final/`. + +Found and fixed on the way: +- **A terminal's paint walked the page list once a cell** (uvprzmxw). + `recolorAnsi` read each cell through `pages.getCell(viewport x, y)`, + which walks from the viewport's top down to the row. The main build no + longer inlined it. `terminal.cellAt` reads the cell from the row's own + pin instead. Phase 5B's "regression against base" was something else: + its core benchmark was a Debug core beside base's ReleaseFast one (the + `--json` identity says which). +- **glibc's `tcgetattr` overran Zig's termios** (qoprvkln). glibc's struct + is 60 bytes and Zig 0.17's `std.c.termios`/`std.posix.termios` is the + kernel's 44, so every call wrote 16 bytes past its caller's struct + (vaxis's `makeRaw` too). The ReleaseSafe tty trapped at its first + terminal (`BADF => unreachable`); Debug and ReleaseFast survived on the + stack's luck. On linux-gnu, main.zig exports `tcgetattr`/`tcsetattr` + as the kernel's TCGETS/TCSETS* ioctls, which touch only the 36 bytes + both layouts share. FreeBSD's libc struct is 44 bytes, so nothing + overruns there (checked on the box). +- **Restore of a session with a command pane failed in ReleaseSafe** + (xquktpqo): "end of dump". The walker loaded an optional by setting it + to an undefined payload and reading into `.?`. A `?[]u8`'s null is its + address, so the optional itself was undefined, and ReleaseSafe's + optimizer took it as null. `Pane.command` read one byte long and every + field after it fell out of step. Pointer-like payloads now load through + a local. Found by tracing (type.field, offset) from the loader in a + Debug and a ReleaseSafe build and diffing for the first divergence. + +The last two were invisible to every Debug suite. The gate therefore runs +the installed (ReleaseSafe) binaries through the suites that start a real +pardes. + +Gates, Debug unless said: +- `core-test`, `unit-test -Dplatform=tty` (its perf gate too), `snap` + (113/113), `replay`, `replicas` (no divergence, no wrong delta at + 30/80/207 ms), `9p-io-test`, `fs-test`, `test/daemon.py` (20 cycles), + `test/daemon.py --upgrade`, `test/fs.py`: all pass. +- `unit-test -Dplatform=gui`: 1563 of 1575, 10 skipped, and the two that + failed on main already. +- ReleaseSafe, the binaries installed: a tty in a pty with the daemon on + and off, `test/daemon.py` 20 cycles, `test/fs.py`, `snap -Doptimize= + ReleaseSafe` (113/113), `--upgrade` Debug to ReleaseSafe and back, the + GUI 5 s offscreen: all pass. +- The FreeBSD main test (ReleaseSafe on both sides, `e2e_box.py`): a + session on the box, two replica clients here and a terminal there, all + three typing. Every digest agreed (72 checks), first frames at 3.8 and + 3.9 s, a shell's answer 223 ms after Enter, 1 misprediction and none + visible. + +No local regression, ReleaseFast, interleaved base/new on the perf cores, +new/base of the medians: +- The core in process, on an idle machine (`--only medium|small --reps + 25`, three rounds): every cell 0.91 to 1.02. Run beside builds, the + big-buffer cells (`body-2m`, `body-appends`, `open`) spread 1.6x run to + run on base alone, so that run's 1.10 to 1.16 there is noise. The idle + rerun is the measure. +- A terminal's frame (the scrollback cells, beside builds): render 0.86 + to 0.90, output 0.89 to 0.91, edit-char 0.87 to 0.91. +- Startup: boot 0.93 to 1.00, first frame 0.94 to 0.96, peak memory the + same. +- Live tty keystroke-to-present: level within each CPU state. With the + machine busy, key-down is 459/469 µs against 452/465 and page-down + 370/378 against 371. Idle, it is 1553/2060 against 2033 and 1939/2102 + against 1921 (the `powersave` governor). The medians across both states + read 0.33 to 1.70 and mean nothing. +- Live GUI: present 32 to 38 ms on both (the offscreen software present), + core 220 to 560 µs on both. + +Installed 2026-10-30 05:27 into ~/.local/bin: pardes and pardes-gui +0.24 (bbcebf80c45e), ReleaseSafe. The previous ones (0.24 caffd9c1f874, +2026-10-29 08:57) are `*.bak-pre-daemon`, and pardes-v9fs is unchanged. -- cgit v1.3 From 771156f92661e1def2d4deec1465ebca9d4c2a9d Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 10:43:35 -0300 Subject: docs/next-daemon.md: phase 6's content cache and zstd notes: the decisions at its forks (far or near by the client's word, the session lists and the client asks, BLAKE3 hashed again on load, the core it leaves before the disk, the cache directory and its bound, zstd on the whole stream after the welcome, libzstd trimmed), zstd's numbers (gzip -6 against zstd's levels on four join streams, and their transfer through the box's link: level with -C without undo history, half its time with it), the first frames through the box and here for a first join, a second client, a rejoin and a client started again, the session's step during joins, the gates in Debug and ReleaseSafe, the A/B against base, what was found and what is not done Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 287 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 287 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 6e47bd06..e1d9e039 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2995,3 +2995,290 @@ new/base of the medians: Installed 2026-10-30 05:27 into ~/.local/bin: pardes and pardes-gui 0.24 (bbcebf80c45e), ReleaseSafe. The previous ones (0.24 caffd9c1f874, 2026-10-29 08:57) are `*.bak-pre-daemon`, and pardes-v9fs is unchanged. + +## Phase 6: content cache and zstd notes + +Built 2026-10-30 (`src/replica.zig`: `Peer.farJoin`, the `Framer`'s +envelopes, `Have` and `Cache`; detached/wire.zig version 16; `walk.blobs`; +tty.zig `remote`; main.zig `remoteAttach`; libzstd in build.zig). A far +client, one through ssh, is sent a join's blobs by hash and only those it +holds nowhere, and everything the session sends it after its welcome goes +in zstd. Decisions at the forks, with the ponytail lens: + +- **Far or near, the client says.** A client dialling through ssh + (`--attach=:`) is far; one on this machine's own socket is + near and joins as before, byte for byte: there a hash and a compression + cost more than the bytes they save (a 100 MB file's first join takes + 1.5 s far against 0.25 s near, the table below). + `PARDES_REPLICA_FAR=1` makes a socket client far (the tests, the local + measurements), `=0` an ssh one near. The daemon reads no further than the + hello, so the word is the client's first message after it (`far`), in + the same write: the session's writer reads it before anything after the + welcome, with no round trip. +- **The protocol: the session lists, the client asks.** After the welcome + (which now says how many blobs the dump holds) a far client's writer + sends the list (`blobs`: each blob's place in the state, its length and + its BLAKE3 hash, 48 bytes), then the state with the blobs cut out (its + pieces), then flushes and waits for the client's `want` (a bit a blob), + then sends the wanted blobs, in order. The client resolves the list as + soon as it is whole and answers while the state is still on its way + (a first join pays that round trip: not done, below). A blob listed + twice (an undo step that changed nothing) is asked for once and copied. + The client says what it lacks, not what it holds: its cache can hold + thousands of blobs, the list of a join a few hundred. The step never + waits for any of it: the list is hashed on the session's writer thread + for that client, from the blobs part B pins for it. ponytail: hashed + again at each far join (BLAKE3, 3.6 GB/s); a session's later joins could + keep the hashes beside its words. +- **BLAKE3 from Zig's std, 256 bits.** A collision or a planted file must + never become wrong content, so the client hashes what it reads from disk + again before it uses it, and a file of another length or content (cut + short, corrupted, planted, longer) is a miss: deleted, and the blob is + asked for. What comes over the link is not hashed (the session said its + hash); the cache file it is written to is checked when it is read. +- **Where a far client holds blobs.** In order: the core it is leaving + (a rejoin after a divergence or an Upgrade: the old core is kept until + the new dump is whole, its blobs looked at by length and each hashed + once when one of its length is asked for), then the cache on disk. + ponytail: a scan of the old core's blobs a lookup, fine to some thousands + of blobs. +- **The cache: `$XDG_CACHE_HOME/pardes/blobs` (else `~/.cache/...`), a + file a blob named by its hash in hex, the directory 0700.** Not the + runtime directory: the blobs are worth keeping across a logout and a + reboot (a client started tomorrow) and safe to lose, and they can be + hundreds of megabytes, which the runtime directory (a tmpfs, often + capped at a tenth of memory, emptied at logout) is not for. A file is + written aside and renamed into place, so a second client never reads half + of one. A far client writes what it was sent on a thread after its load + (the first frame waits for none of it), and at its end, the blobs of its + core the cache lacks: what it made since it joined (an edit's text, its + undo snapshots), so a client started again is not sent them. A hangup + now ends a remote client the normal way, for that (and its ssh forward's + socket is unlinked: part B's last note). +- **Bounded: 1 GiB, `PARDES_BLOB_CACHE_MB` to change it; the least recently + used go first.** A hit makes its file's time now; before a store, the + directory is scanned and the oldest files deleted until what comes fits. + A join bigger than the cap stores what fits. ponytail: the scan and + sort at each store, fine to tens of thousands of files, where an index + would take over. +- **zstd, decided by its numbers (below), on the whole stream after the + welcome, not the join alone.** The session's writer puts every byte it + sends a far client through one compressor and writes envelopes + (`zstd`), each the flush of at most 1 MiB of input, so a reader unpacks + one whole into a bounded buffer; the client's `Framer` unpacks them and + hands their messages on as if they had come bare, so the dial and the + stream's reader are unchanged. The join is one frame at level 6 over an + 8 MiB window: an undo snapshot is a near copy of its text, a file's + length before or after it, and zlib's 32 KiB window never sees the copy. + What follows (units, terminal deltas) is a frame at level 3 over 1 MiB. + The client's direction is not compressed. With all of that, `ssh -C` + only spends both ends' time, and is no longer in the default command. + ponytail: a big paste through a slow link goes as it is; it would be an + envelope of its own. +- **libzstd 1.5.7, vendored as the upstream tarball** (one copy; nothing + else in the build bundles one; Zig's std has a decompressor but no + compressor). Its compressor and decompressor only, single-threaded, + without its x86 assembly, the block compressors of levels 3 and 6 alone + (`ZSTD_EXCLUDE_*`), its unused functions dropped at link; every context + made by `ZSTD_create*_advanced` with a `ZSTD_customMem` over c_heap, so + a test's allocations are its testing allocator's. + +The zstd decision, measured. A join's stream as a client reads it, taped +by a tee relay (`.scratch/daemon/amort/h/capture.py`): `real` is part B's +box session made here (pardes.zig, host_io.zig, terminal.zig, look.zig and +3 MB of the tree's sources joined, a shell with 1 MB of scrollback, 4.0 MB +of stream), `edited` the same after three edits to each text (each text's +undo snapshots: 15.8 MB), `typical` and `big` join2.py's synthetic files +(their lines differ by a counter, so everything compresses far better than +source does). Compressed whole with the CLIs, here (one core); the box's +CPU takes 0.03 s, 0.11 s and 1.04 s for real at -3, -9 and -19: + +| stream | raw | gzip -6 (ssh -C's) | zstd -3 | zstd -3, 8 MiB window | zstd -6, 8 MiB | zstd -9 | zstd -19 | +|---|---|---|---|---|---|---|---| +| real | 4.04 MB | 1.00 MB, 91 ms | 1.02 MB, 19 ms | 0.95 MB, 23 ms | 0.85 MB, 49 ms | 0.82 MB, 58 ms | 0.72 MB, 1005 ms | +| edited | 15.82 MB | 3.97 MB, 336 ms | 3.36 MB, 63 ms | 0.96 MB, 30 ms | 0.86 MB, 61 ms | 0.85 MB, 87 ms | 0.73 MB, 967 ms | +| typical | 8.33 MB | 1.22 MB, 98 ms | 0.72 MB, 19 ms | | | 0.35 MB, 39 ms | 0.20 MB, 723 ms | +| big | 77.4 MB | 11.17 MB, 768 ms | 6.94 MB, 136 ms | 4.10 MB, 131 ms | 3.71 MB, 336 ms | 4.80 MB, 467 ms | 1.82 MB, 22.4 s | + +The box's link (each payload `cat` from the box through ssh, the time +from its first byte here to its last, the median of three interleaved): + +| payload | through `ssh -C` | zstd -3, plain ssh | zstd -9, plain ssh | zstd -19, plain ssh | +|---|---|---|---|---| +| real (4.04 MB raw, 10.0 s through plain ssh) | 4.32 s | 4.82 s | 4.06 s | 4.14 s | +| edited (15.82 MB raw) | 8.90 s | 7.03 s | 4.51 s | 3.87 s | + +- **Without undo history zstd buys nothing** that the link shows: 0.82 MB + against zlib's 1.00 MB, 4.06 s against 4.32 s, within the runs' spread + (3.9 to 5.0 s). The link is not the bottleneck its 170 to 400 KB/s + suggests for a payload this size: a megabyte or less takes about 4 s + whatever it is, TCP's window growing over a 207 ms round trip. +- **With it, zstd halves the transfer** (8.90 s to 4.51 s): an undo + snapshot is a near copy of its text, which zstd's window of megabytes + matches and zlib's 32 KiB does not, and a session one works in for an + hour holds up to 256 of them a file. The window decides it, not the + level: at -3 with 8 MiB, edited is 0.96 MB where -3's own 2 MiB leaves + 3.36 MB (the 3 MB file's snapshots are 3 MB apart). +- **So zstd, at level 6 over 8 MiB**: within 4% of -9's size at about + two thirds of its time; -19 saves a little more and costs a second of the session's CPU + every 4 MB (22 s for big). The cache does not replace it: a first join + has nothing cached, and an edit made while no client was attached is new. + +The benefit, measured (`h/joincache.py` here, `h/boxcache.py` through the +box, both on `h/cachejoin.py`; ReleaseFast here and ReleaseSafe for the +box's pair, timings alone on the perf cores). Four joins of one session, +each client with its own terminal and check file, one cache between them: +the first (the cache empty); a second client while the first is attached; +that second one put off the session's state (PARDES_TEST_DIVERGE), which +joins again from the core it leaves; and one started after both ended. +The first frame is from the client's start, ssh's forward included (2.0 s +of it on the box, part B); a rejoin keeps its forward, and its time is its +dial and load (PARDES_REPLICA_STATS `join_ms`), its frame drawn after. + +- **Through the box** (ms; `real` 3.9 MB of files, a 3.9 MB dump; `edited` + the same after three edits a text, a 15.8 MB dump; main is the final + gate's ReleaseSafe pair, bbcebf80, through its default `ssh -C`): + + | session | build and ssh | first join | second client | rejoin, dial and load | started again | + |---|---|---|---|---|---| + | real | main, `ssh -C` | 6535 | 7166 | a whole join | 6952 | + | real | this line, `ssh` (the default) | 7069 | 2484 | 434 | 2619 | + | real | this line, `ssh -C` | 7623 | 2449 | 430 | 2409 | + | edited | main, `ssh -C` | 11931 | 11257 | a whole join | 10921 | + | edited | this line, `ssh` (the default) | 7079 | 2442 | 452 | 2482 | + | edited | this line, `ssh -C` | 6824 | 2412 | 460 | 2437 | + + - A client that holds the blobs is drawn 2.4 to 2.6 s after it starts + where main took 7.0 to 11.9 s: past ssh's 2.0 s, 0.4 to 0.5 s, + whatever the session's size. Its link carries the state (113 KB + before zstd) and the list. + - A first join of a session with undo history: 7.1 s where main took + 11.9 s (zstd's window). Without one, level with main (7.1 s against + 6.5 s, within the link's spread): zstd's 15% fewer bytes are worth + about the round trip the list's answer costs. + - `ssh -C` over zstd: 7.6 and 6.8 s against 7.1 and 7.1 for the first + join, the same for the others; noise either way, and dropped. + - The session's step while each joined (its stats, real): 0.3 to 1.2 ms + on the box, main's 0.3 to 0.9 ms. + +- **Here, the socket** (ms; join2.py's shapes: typical 4.1 MB of files + and 1 MB of scrollback, big 38.6 MB and 8 MB, huge a 100 MB file beside + typical's; three edits a text; near as before, far with + PARDES_REPLICA_FAR=1): + + | session | client | first join | second client | rejoin, dial and load | started again | the session's worst step during the joins | + |---|---|---|---|---|---|---| + | typical | near | 22 | 22 | 9 | 22 | 0.6 | + | typical | far | 42 | 32 | 17 | 32 | 0.5 | + | big | near | 104 | 103 | 66 | 83 | 1.2 | + | big | far | 517 | 153 | 127 | 153 | 0.8 | + | huge | near | 254 | 244 | 170 | 285 | 0.7 | + | huge | far | 1514 | 517 | 450 | 506 | 2.3 | + + - Near is part B's join unchanged (22 to 46, 87 to 117 and 247 to 288 + ms then), and the default on this machine's socket: far there costs + more than the bytes it saves (the session's hashing and compression on + a first join, the client's read and hash of what it holds after). + - A join costs the session's step at most 2.3 ms with a 100 MB file + open, far or near (the gate: under 16.7 ms). + +Gates (Debug unless said; scripts and logs in `.scratch/daemon/amort/`): +- **Correctness.** `test/daemon.py --far` (in its main run too): far + clients of a detached session with two files (one 1.3 MB) and edits; + the first sent every blob (1104 KB), a second client on this machine + from disk (3 of 4, the fourth the text the first had just made), its + rejoin after a forced divergence from the core it left (4 of 4), one + started after both ended from disk (6 of 6: what they made, kept at their + end), one after every cache file was corrupted or cut in half by hand + sent them all again (5470 KB), and two with a 1 MiB cap held by older + files: those went first and the cache stayed under its cap. Every + client's digest was the session's at every quiet point (11 to 17 checks + a run, none differed), Debug and ReleaseSafe. Through the box, every + check of the four joins above agreed (28 to 41 a run). `zig build + replicas`: 4 seeds at 30, 80 and 207 ms, none diverged, none unequal, no + delta wrong. `test/daemon.py`'s replica tests pass, near and far. +- **New tests** (core-test, `-Dtest-filter=blob`): a cached blob is hashed + again as it is read, and one cut short, corrupted, planted or longer is + a miss and goes; the cache stays under its cap, the least recently used + going first (a read makes a file the newest; a file not a blob's counts + and goes by its time); a far join over a socket pair, the session's + writer and reader threads against the client's dial, is sent each blob + once: again with them on disk, or in the core it leaves, the state + alone, and a cached file gone bad sends that blob again; a near join's + dump is the session's byte for byte. + +- **Suites**, once at the end: `core-test` (its run beside the box + measurements had the 9P failure above; again, all pass), `unit-test + -Dplatform=tty` (its perf gate, and shared-libs-check, which now refuses + a second zstd too), `snap` 113 of 113, `replay` (113 scripts, 20 monkey + seeds, none panicked or diverged), `replicas`, `9p-io-test`, `fs-test`, + `test/daemon.py` 20 cycles (its far clients included), `--upgrade` + (unstamped Debug to stamped), `test/fs.py` (49 s): pass. + `unit-test -Dplatform=gui`: 1568 of 1580, 10 skipped, and the two that + fail on main (`tagline pointer mapping...`, `a tag caret past its row's + end...`). +- **ReleaseSafe** (`zig build -Doptimize=ReleaseSafe -Dhome-install=false`, + the binaries the box measurements ran): a tty in a pty with the daemon on + and off, `test/daemon.py` 20 cycles, `test/daemon.py --far`, `test/fs.py` + (39 s), `snap -Doptimize=ReleaseSafe` (113 of 113) and the cache tests + (`core-test -Doptimize=ReleaseSafe -Dtest-filter=blob`): pass. +- **No local regression.** ReleaseFast, interleaved base new base new base + new base on the perf cores (`core_optimize` fast on both sides), the + median of each side's runs, new/base, beside base's own A/A; builds ran + on the other cores throughout, so every cell is in that state: + - the core in process (`pardes-perf --reps 15`): the opens 0.90 to 0.99, + render 0.94 to 0.99, the keys, page-down and wheel 0.80 to 1.03, the + edits 0.90 to 1.05, body-2m 0.99 to 1.04, body-appends 0.92 to 1.07 + (its A/A 0.99 to 1.30), deep-sel 0.89 to 0.97 (and longline's 364 ms + on base, 0.1 ms on main already); a terminal's cells + (sb-64k to sb-8m) 0.87 to 1.01. The A/A elsewhere 0.94 to 1.05. + - live tty keystroke to present (base's `pardes-perf-live --live`, + present p50 / core p50): key-down 1.01 / 0.99, key-right 1.01 / 0.95, + edit-char 0.97 / 0.98, page-down 0.99 / 0.92, wheel 1.08 / 1.04 (A/A + 1.05 to 1.16); a 1M-line open 1.06 (A/A 0.92); `yes | head -c 32M` + 1.22 (A/A 1.13), its runs 4.1 to 7.3 s on this line and 5.4 to 6.7 on + base, as main's 7.0 to 7.3 beside base's 6.0 to 7.3 were in the final + gate. A session's own path is untouched by this change: a session with + no far client hashes and compresses nothing, and a near one is sent + its dump as before. + +Found on the way: +- **PARDES_REPLICA_CHECK keeps a session's units coming.** With two clients + checking, an idle session sends a unit about every 300 ms on main too + (`h/idle.py`: 27 checks in 8 s, near or far, main's build alike): a + check wakes the session's pump, whose unit makes the client's state + "changed", and the client asks again once quiet. Only the test knob + does it; a harness that expects a divergence to wait for a key it sends + finds the client joined again before (the measurements above wait for + the first link's stats line instead). +- **A rejoin's "joined again" line can be gone before it is drawn**: it + lasts until the next key, and keys typed during a long rejoin (a 100 MB + file's) come first. As designed; the harnesses do not look for it. +- **Harness traps.** A terminal's frame request split across two reads + went unanswered (test/daemon.py's `Term` now keeps the last bytes of a + read); keys written right after a first frame raced the terminal's + answer to it, an answer behind an Esc in one read being an Alt chord + (the far test waits 0.3 s, and writes the Esc on its own); a `pkill -f` + whose pattern was in the shell's own argv killed the shell. +- The core-test run of the suites had one failure, `9p_io`'s "a change + waits while the editor is out mid-step", with the box measurements and + builds running; alone three times, and the whole core-test again, it + passed. + +Not done, and why: +- **A first join pays the list's round trip** (207 ms on the box): the + wanted blobs wait for the client's answer, and the state (113 KB before + zstd) no longer hides it. A client with nothing cached could say so in + its `far` word and be sent everything at once; it is a first join on a + new machine only. +- **A blob is hashed again at every far join** on the session's writer + thread (a 254 MB dump: 70 ms there). The hashes could be kept beside + the blob words. +- **What a client sends is not compressed** (its keys, and a paste): with + `-C` gone, a megabyte's paste crosses the box's link raw. +- **A client killed outright keeps on disk only what it was sent**: what + it made since (its edits' texts and undo) is written at its end, a + hangup's included, but not on a kill -9; the next join gets those again, + in zstd. +- **The cache is per user, unencrypted, 0700**: a blob of a file the user + could read elsewhere, kept until evicted or the cache is cleared. -- cgit v1.3 From d2ab76c2f590144100d93d7fe12aeaa83697434f Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 13:10:49 -0300 Subject: docs/next-daemon.md: phase 6's GUI replica notes: the decisions at its forks (the window client the tty's with a window for a terminal, replica.Link shared; the window's writes into its core one funnel, read over the client's own view; no pump in a client, its window's font, opacity and shader files its own; the wheel a row a turn there; a client's frontend said first; one desktop semantics, a view's words by its frontend; a window's session serving at its step boundary, a client's view no window there; its font and opacity answers taped; the far path amort's; a check once quiet), what was found (a run's output read off a replica's emulator, a cut-off client's silent end, a second's turn in the replica tests), the window client's numbers (keystroke to present at each RTT and through the box, echo predicted and not, its joins, the session's step while it lags), the gates in Debug, ReleaseSafe and on the box, the A/B against base (the core, live tty and live window, idle and busy cores), and what is not done Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 319 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 319 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index e1d9e039..ab772b51 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3282,3 +3282,322 @@ Not done, and why: in zstd. - **The cache is per user, unencrypted, 0700**: a blob of a file the user could read elsewhere, kept until evicted or the cache is cleared. + +## Phase 6: GUI replica notes + +Built 2026-10-30 (`src/gui/gui.zig` `remote` and the window session's +`Peers`, `src/replica.zig` `Link`, `src/views.zig` `within`, the +`builtins.zig` registry's `here`, `Event.frontend`, wire 16; +`test/daemon.py --window-clients`). A window is now a replica client, of a +terminal's session or a window's, and a window's session serves replicas of +either kind. Decisions at the forks, with the ponytail lens: + +- **The window client is the tty's, with a window for a terminal.** + `pardes-gui --attach=:` and `--attach=:` (ssh, + main.zig `remoteAttach`, now the frontend's own `remote`) make the window + as a session's is (`runNative`) and run `remoteSession` instead of + `localSession`: the dial, the dump's replica checked against the + session's digest, a reader thread, the rejoin after a divergence or an + Upgrade's `refused`, the end in words. What is frontend-free in the tty's + `Remote` moved to `replica.Link` and both use it (`input`, `take`, + `drawn`, `quietCheck`, `rejoins`, `redial`, the stats and checks); the + tty's keeps its draw. `pardes-gui --attach` and `--attach=` (no + colon) re-open a local session's window through the daemon as before. +- **The window's 52 writes into its core are one funnel, read over the + client's own view.** The window feeds its core through `Input.post` + (dispatch, the wheel, the touch and gamepad paths, the timedemo, the + test feed), and a handful of frame chores call `core.update` themselves + (its size, the pointer under a frame that moved, the smooth scroll). In + a client `Input.post` appends to the client's list (`Client.post`, texts copied) and nothing writes the + core. Mapping a pointer reads the core's view fields (its rects, its + presentation, its row metrics), and another client's input may have made + the replica's live view another's: so the window's work runs inside + `Replica.within` (`views.within`: the live view parked, the client's + unparked, put back after exactly as it was, as `drawOne` draws), and its + inputs are applied after, one `Link.input` each. With one client the + live view is its own and `within` is a call. +- **No pump in the client.** Its core moves only by its own inputs and the + session's units; it draws its view between steps (`Replica.draw` through + the window's own `present`), puts back what presenting writes in the + core (a skipped image, a frame owed) and records its frames as the tty + client does. The window's per-frame host chores that write the core stay + out of a client: opacity, font requests and their answers, a scene + effect the GPU refused, user shader files (their compiles' messages). + ponytail: a client's window keeps its own font, opacity and shader + files; the built-in scene effects follow the session's settings. +- **The wheel in a client moves a row a turn.** The window's glide and a + touchpad's fraction run on the core's clock, which in a client is the + session's and moves only with its units. ponytail: whole rows, no glide; + the local window's is unchanged. +- **A client says its frontend first** (`Event.frontend`: windowed, native + images and patches; wire 16, client tag 0x20): the view the session opens + for a joiner copies the live view's, and a window's view of a terminal's + session would have drawn pictures as cells and the window's words would + have been refused there. A terminal's client says it is no window, and + draws pictures as cells (ponytail: a kitty-graphics terminal client was + drawing none before, its shm never probed). +- **The terminal's build and the window's step alike.** The semantics hash + named the platform, so a window could not join a terminal's session (the + FreeBSD box has terminals only) nor a terminal a window's. Their cores + differ in the step only by the window's own words (Fonts, Pet, Font, + TaglineSize, WindowOpacity, Ligatures, Lift and its family, the scene + effects, their leader paths `SPC t f` and `SPC t r`) and the Settings + report. Three ways: refuse cross-build joins (the box and a window's + clients impossible); accept known divergences (Help's index, `SPC t f` + and Settings would each make the other build's clients join again); or + give both builds every word and let the *view's frontend* decide + (kakoune's way: each client its own). The last: a desktop build's + registry is the window's (`builtins.capabilities`), and + `builtins.registry.here` says whether the live view takes a word (a + window's own words in a window's view alone, `Pardes.windowed`, which a + window's build starts true and a client says). `runBuiltinFrom` refuses + the rest in the words the terminal said before ("Lift is GUI-only, + invalid here"), and the leader paths, Help's index, `/commands`, the root + ctl's settings and the Settings report list the view's own. The render + keeps the build's own frontend (`builtins.frontend`: the tty draws no + lift). Local behaviour is unchanged: the 113 snapshots, which hold the + tty's Help index and leader groups, pass as they were. The hash names a + family (`desktop`) for the two; the tty binary carries fonts.zig and the + window's builtins (a few KB). +- **A window's session takes replicas at its step boundary,** as the tty's + does: `Peers` in the window's Shell, its wake an SDL event, `boundary` + after each pump, a Restore and an Upgrade letting clients go + (`dropAll`, `refuseAll`), Detach in a client's view ending that client. + What the tty's session does with clients, the window's does too: one + person's input a wait (`split`: the next SDL input held for the next + wait, so a pump is one unit of it), a pump woken only by a client's post + reads no new time (`mail_only`), and its last window closed with clients + attached is a Detach, not the session's end. +- **A client's view is no window here.** The window's Shell kept the live + view's window in the Gui (`follow`); a client's view live (its input the + last) has none, so `follow` keeps the window it holds (`windowed`, a mask + of the views that are windows), and what the window does to its own view + makes that view live first (`windowView`: an SDL event of the window, + `Input.post`, the frame chores when one is due: its size, its scroll, a + font). Found by the first test with a window typing beside a client: the + window's keys went into the client's view. +- **What the window writes into the core outside a step is taped.** The + window's font (observed, requested, taken, acknowledged, refused), its + opacity's answer and a refused scene effect are core calls now + (`tape.Input`), so a window session's clients replay them; the rest of + its writes were taped already (sizes, pointers, panel acknowledgements, + messages). ponytail: an adopted shell's pane (`Adopt`) and a terminal + reader's failure still write the core directly; neither is taped. +- **The far path is the tty's** (amort's line, merged under this one): + through ssh the window client is far as the tty's is + (`PARDES_REPLICA_FAR` either way), its join's blobs found by hash in the + core it leaves or the cache on disk, its stream in zstd, what it made + kept at its end (`replica.Have`, the same calls in `remoteSession` as in + tty.zig's `remote`, through `Link.load` and `Link.redial`). One wire + version, 16, carries amort's far word (the first message after the + hello) and the frontend (a client's first input). +- **A check once a quiet while, not once a wake.** The tty's client asked + the session's digest on its first wake after a change; a window session + in test mode pumps a frame a tick, a unit each, so no wake ever came + first. A client now asks once it is quiet, 300 ms after its last input + and its last check (`quietCheck`); the answer is in stream order, so it + holds with units flowing. + +Found on the way: +- **A pty/run's answer read the replica's emulator** (vnolzlvq). A run's end + (`ninep/pty.zig finish`) took the command's output off the terminal in + the step; a screen-state replica's emulator is its deltas', which a + lagging client's are states behind, and replaying the end read pins into + pages it never had: ghostty's `leftWrap` overflowed (a window client + stopped while a shell flooded the session aborted 3 runs in 8). The + output is taped now (`.prompt`, the same site as a prompt read after + output); none in 6 since. +- **A client cut off while lagging could end in silence** (ussmwtkk): its + own write found the link shut before its reader did, which ended the + loop as a quit, exit 0, no words. A failed write is the session's hangup. +- **The rejoin's drain** (the coordinator's ymzywopz, on main): the window + client joins its reader before it drains, so it never had that race. +- **A second's turn is a misprediction** (ssruwvql): an edit stamps its + file with the clock's second, which a replica guesses (its last unit's + time, the time since and the link's round trip); a key the session takes + in another second than the guess goes the slow way, replayed. Right, and + rare live (the round trip's jitter against a second); but a test's + replica has no round trip yet and its authority takes the keys after all + are typed, and `another client's input first` counted exact fast + confirmations: under load in a Debug window build (a slower step) they + straddled a second in 11 runs of 40. The two tests that count them start + early in a second now: 0 of 30. +- **Harness traps:** a window's test feed is a pipe, whose lone ESC waits + for the next byte (the parser cannot tell it from a sequence's start), so + the tests send Escape as `CSI 27 u`; a terminal client's Esc is resolved + by the 50 ms wait for more input, which the test terminal's answers to + frames (`CSI 0 n`) can keep refilling while other clients' units arrive + (main's `normal_again`); a window session in test mode starts no daemon + unless PARDES_DAEMON says so. + +Measured. ReleaseFast, unstamped (one semantics here and on the box), the +timings alone on the perf cores (lock.sh), the windows offscreen (SDL's +offscreen driver, software present); the harness scripts are in +`.scratch/daemon/guirep/live/`. + +- **Keystroke to present on a window client, local-only actions** (base's + `pardes-perf-live --live-gui`, its timedemo's keys on a 50k-line file, + through `gui-live.sh`: a local window, or a window client of a terminal's + session detached on that file, through test/delay_relay.py at each RTT + and through the box's link; two rounds; p50 (p99), ms): + + | | key-down | key-right | edit-char | page-down | wheel | + |---|---|---|---|---|---| + | a local window, this line | 31.1 (40.3) / 31.5 (45.5) | 34.8 (44.2) / 35.3 (41.0) | 35.9 (44.4) / 36.5 (46.2) | 37.0 (50.0) / 37.8 (50.3) | 36.5 (47.3) / 36.9 (52.8) | + | a local window, base | 29.0 (35.9) / 31.5 (37.5) | 35.1 (45.9) / 35.3 (42.2) | 36.2 (42.2) / 36.5 (45.1) | 37.2 (48.1) / 37.5 (52.5) | 36.5 (50.0) / 36.8 (44.2) | + | client, direct | 31.3 (40.4) / 28.6 (36.2) | 35.1 (47.3) / 35.1 (54.4) | 36.7 (41.1) / 35.8 (44.0) | 36.0 (46.4) / 35.4 (40.8) | 35.8 (45.1) / 35.2 (45.8) | + | client, relay 30 ms | 29.9 (54.6) / 29.9 (42.7) | 34.9 (45.8) / 36.4 (53.6) | 35.7 (54.0) / 35.8 (45.8) | 34.9 (52.1) / 35.1 (47.3) | 35.1 (44.0) / 35.1 (42.6) | + | client, relay 80 ms | 32.7 (39.5) / 30.5 (36.6) | 38.1 (46.6) / 34.9 (44.0) | 39.1 (57.4) / 35.6 (46.1) | 37.7 (56.5) / 35.1 (43.9) | 38.0 (46.7) / 34.8 (44.5) | + | client, relay 207 ms | 31.1 (35.9) / 30.6 (36.2) | 35.6 (46.3) / 34.6 (59.9) | 35.9 (45.4) / 35.5 (55.3) | 27.9 (56.6) / 34.9 (47.8) | 36.9 (55.3) / 34.6 (50.8) | + | client, the box (far, zstd) | 33.7 (48.1) / 28.3 (43.9) | 31.1 (43.4) / 30.2 (46.4) | 30.9 (44.6) / 29.4 (39.3) | 33.8 (51.0) / 29.6 (48.4) | 28.7 (49.9) / 30.1 (43.2) | + + - **A window client's local-only action is as fast as a local window's + at every RTT, the box's included.** Both are the offscreen software + present's (30 to 38 ms, the final gate's GUI numbers): what the harness + can tell apart is the core's share, the client's render 0.4 to 0.5 ms + p50 (0.1 to 0.9 through the box) against a local window's update and + render 0.24 to 0.6 ms. The client's render runs on its rollback epoch's + allocator, which a local core's does not. + - **Visible rollbacks** (a frame drawn after an input the session + recorded otherwise, an upper bound; of 664 inputs a run): none direct + and at 30 ms, 10 and 0 at 80 ms, 0 and 9 at 207 ms, 15 to 24 through + the box (2.3 to 3.6%). Every other rollback (the session's own pumps' + units before the client's pending inputs: 536 to 610 a run here, 268 + to 295 through the box) is put back and replayed before the next + frame, p99 2.6 to 8.9 ms. +- **Terminal echo** (`echo-gui.py`: a key read off the window's test feed, + to the end of the GPU frame that shows it, PARDES_TEST_LATENCY's notes; + 40 keys; p50 / p90, ms): + + | | file (insert) | shell, predicted | shell, not predicted (PARDES_NO_ECHO) | + |---|---|---|---| + | a local window | 2.8 / 4.6 | | | + | client, direct | 2.5 / 4.5 | 4.3 / 9.7 | 25.5 / 29.7 | + | client, relay 30 ms | 2.6 / 4.7 | 5.1 / 11.0 | 60.4 / 64.8 | + | client, relay 80 ms | 3.1 / 4.6 | 4.7 / 4.9 | 101.9 / 121.9 | + | client, relay 207 ms | 2.7 / 8.2 | 4.4 / 10.0 | 236.8 / 248.9 | + | client, the box (far, zstd) | 4.2 / 5.2 | 4.2 / 4.8 | 242.5 / 243.3 | + + (Through the box, the harness's ssh had `-C` beside the stream's zstd; + a predicted key never waits for the link.) Every key of the shell's was + predicted and confirmed (40 of 40, at each RTT): a key at a prompt shows + in the window's next frame, as a key into a file does, the 0.5 ms of the + tty client's being a terminal's write where a window's is a frame (its + software present's 2.5 ms). Not predicted, it is a round trip and the + session's paced output. +- **Through the box, its join** (far, amort's path): the main test's 400 + lines, 4 blobs, 51 KB sent, dial and load 0.63 s (ReleaseSafe 0.99 s), + its first frame about 2.6 s after it started (ssh's forward is 2.0 s); + its rejoin after a forced divergence from the core it left, 4 blobs of + 4, nothing sent, 0.42 s. The live harness's 3.1 MB file: 3126 KB, 2.3 to + 2.4 s. +- **The session's step while a window client lags** (`lag-gui.py`: a shell + printing a line a millisecond for 12 s, a terminal's client typing a key + every 30 ms, the window stopped with SIGSTOP and cut off at a 256 KB lag; + µs, p50 / p99 / max; ReleaseSafe): the terminal client's inputs 30 / 75 / + 289, the boundaries 49 / 148 / 764, the window cut off (`cut 1`), and let + go on it said "let this client go" and ended. + +Gates (Debug unless said; scripts and logs in `.scratch/daemon/guirep/`): +- **Correctness.** `test/daemon.py --window-clients` (new): window clients + of a terminal's session at 0, 30, 80 and 207 ms beside a terminal's + client, every check of every client the session's digest (28 a run, + none differed); one put off the session's state joined again, and again + across an Upgrade (agreed 7 times), and detached; its far client's first + join 9 blobs, 57 KB, its rejoins 9 of 9 and 10 of 10 from the core it + left, nothing sent; a window's session served a terminal's client and a + window's (agreed 9 and 7 times); nothing left behind. Debug and + ReleaseSafe. `zig build replicas` at 30, 80 and 207 ms: none diverged, + 1545, 1590 and 1512 deltas checked, none wrong. Through the box + (`live/e2e_box_gui.py`: a session on the box, a window client and a + terminal's client from here, a terminal on the box attached beside them, + a file's edits, scrolls and moves, a shell's command, a second window + far, forced off the session's state and joined again): every check + agreed (the window 128, the terminal's 132; ReleaseSafe 131 and 134), + none mispredicted, the second window's first join 4 blobs 51 KB, its + rejoin 4 of 4 from the core it left, nothing sent; nothing left on the + box. +- **The window's own reopen** (`test/daemon.py --gui`, 5 detach and attach + cycles, a second window detached alone): pass, Debug and ReleaseSafe + (SDL's offscreen driver). +- **Suites**, once at the end (before the tests' fix and the rebase below): + `core-test -Dplatform=tty`, `unit-test -Dplatform=tty` (its perf gate), + `snap` 113 of 113, `replay` (113 scripts, 20 monkey seeds, none panicked + or diverged), `replicas`, `9p-io-test`, `fs-test`, `test/daemon.py` 20 + cycles, `--window-clients`, `--upgrade` (unstamped Debug to stamped), + `test/fs.py` (49 s): pass. + `unit-test -Dplatform=gui`: 1568 of 1580, 10 skipped, and the two that + fail on main (`tagline pointer mapping...`, `a tag caret past its row's + end...`); `core-test -Dplatform=gui`: the caret one, and in two of its + full runs a replica test's count of fast confirmations (below, ssruwvql; + after it, the core-test runs pass but for the caret one). + After the rebase onto main's `--ps` (prvoxvzk): `core-test` tty and gui, + `test/daemon.py` 20 cycles (its `manage` included), `--window-clients` + (its `--ps` checks: a terminal's session with window clients listed + alone as `(tty, detached)`, a window's serving clients as `(gui, + attached)`), `--gui` 5 cycles: pass, `core-test -Dplatform=gui` but + for the caret one. +- **ReleaseSafe** (`-Doptimize=ReleaseSafe -Dhome-install=false`, the + binaries the box's second run used): `--version`, a tty in a pty with the + daemon on and off, a window session offscreen for 5 s, `test/daemon.py` + 20 cycles, `--window-clients`, `--gui`, the lag test (`live/lag-gui.py`, + above), `test/fs.py` (38 s), `snap -Doptimize=ReleaseSafe` 113 of 113: + pass. Again after the rebase, built afresh: `test/daemon.py` 20 cycles, + `--window-clients` (its `--ps` checks), `--manage`, `--gui` 5 cycles, + `test/fs.py` (38 s): pass. +- **No local regression.** ReleaseFast, interleaved base new base new base + on the perf cores (`core_optimize` fast on both sides; base is + `.scratch/daemon/base`'s), the median of each side's runs, new/base, + beside base's own A/A; the new binaries are this line before its rebase + onto main's last two (a test's rest and `--ps`, neither on these paths): + - the core in process (`pardes-perf --reps 15`, idle cores, the second + of two runs): the opens 0.98 to 1.03, render 0.96 to 0.98, the keys, + page-down and wheel 0.95 to 1.02 (longline's 0.79 to 0.87, main's + already), the edits 0.90 to 1.03, body-2m 0.95 to 1.02, body-appends 0.95 to 1.02, + deep-sel 0.92 to 0.99 (longline's 360 ms on base, 0.1 ms on main + already), a terminal's cells (sb-64k to sb-8m) 0.88 to 1.02. The first + run's two outliers did not come again: body-appends on the board's + fixture 1.29 (A/A 0.96; then 1.02 alone, A/A 0.99, and 1.06 with the + other cores busy) and resize-cols at 1M lines 1.10 (then 1.02). + - live tty keystroke to present (base's `pardes-perf-live --live`, the + other cores kept busy by spinners at nice 19, as idle cores let the + powersave governor put the package to sleep between keys: base's runs + read 347 to 1651 µs; present p50 / core p50): key-down 1.03 / 1.03, + key-right 1.00 / 0.94, edit-char 1.00 / 0.99, page-down 1.01 / 0.95, + wheel 1.05 / 1.05 (A/A 1.00 to 1.02; base's runs 389 to 469 µs, this + line's 444 to 480), a 1M-line open 1.01, `yes | head -c 32M` 0.63 (A/A + 1.03). + - live window keystroke to present (`--live-gui`, offscreen; present / + core): idle 0.93 to 0.98 / 0.94 to 1.03 (A/A 0.95 to 1.05), busy 0.90 + to 0.97 / 0.98 to 1.03 (A/A 0.85 to 0.97 / 0.97 to 1.10). + +Not done, and why: +- **A window client paints nothing ahead.** A key whose frame the session + owes (an image's, a PDF's lane) waits for its unit; a local window's + paint-ahead is the core's, which a client's moves only by units. +- **No glide in a client, and a touchpad's fractions round to rows** (the + core's clock is the session's there, above). +- **The notice a client gets is in its title and its log, not its frame** + (the session's words: joined again, let go, refused). The tty's draws + them on its notice band, which the window's frame has no slot for + outside the core. +- **A client's window keeps its own font, opacity and shader files**: the + session's `Font`, `Fonts` and `WindowOpacity` move the session's own + window, and a client's follows them only through its core (the scene + effects do; the font does not). `Fonts` reads the machine's font + directories untaped, so its list is the session machine's. +- **An EffectCode's sources differ by backend** (Vulkan or GL): a client of + another build's window reads the session's. +- **A terminal's client draws a window session's pictures as cells**, never + through kitty graphics (no shm across machines). +- **Adopt and a terminal reader's failure write the core untaped** (above): + a client replaying them would diverge and join again. +- **A rejoin opens a new view**: the old one's place, selection and scroll + are the session's to keep, and it lets the old view go. +- **A hit test reads the live view**, which `within` makes the client's for + its own events; a frame chore outside one (a pointer's hover after a + unit moved the frame) reads it as the unit left it. +- **A window client has no SIGHUP to quit on**: its ssh's end is the link's, + read as a hangup. +- **One window a client**: a session's second window through one client + would be a second client. -- cgit v1.3 From 5b120f481a5e581d5252c41a4ddcc3e28c8a07f3 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 13:55:38 -0300 Subject: docs/next-daemon.md: the final gate on everything merged: Debug, the installed ReleaseSafe binaries, the FreeBSD main test, the A/B against base with the edit cells rechecked, and the one divergence a window client of the box showed once and that has not come back Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index ab772b51..e2101740 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3601,3 +3601,52 @@ Not done, and why: read as a hangup. - **One window a client**: a session's second window through one client would be a second client. + +## The final gate, again, on everything merged + +Run 2026-10-30 on main lprqtsom (the content cache and zstd, the GUI +replica client, `--ps`/`--kill`, the four flaky tests' fixes) with the +same scripts (`.scratch/daemon/final/final2.sh`). + +- **Debug:** + - `core-test` tty passes. `core-test` gui and `unit-test -Dplatform=gui` + fail only on the two tests main already failed, 1568 of 1580 with 10 + skipped. + - `unit-test` tty, `snap` (113/113), `replay`, `replicas`, `9p-io-test`, + `fs-test`, `test/daemon.py` (20 cycles, `--manage` and `--far` + included), `--upgrade`, `--window-clients`, `--gui` (5 cycles) and + `test/fs.py` pass. +- **ReleaseSafe, the binaries installed:** + - a tty started in a pty, daemon on and off; + - `test/daemon.py` 20 cycles, `--window-clients`, `--gui` 5, `--upgrade` + from Debug, `test/fs.py`, `snap -Doptimize=ReleaseSafe` (113/113). + + All pass. +- **The FreeBSD main test:** ReleaseSafe at both ends. Two replica clients + here and a terminal on the box. Every one of 130 checks agreed, with 0 + mispredicted. First frames came 2.5 and 2.6 s after start (3.8 s before + the content cache and zstd), and a shell answered 224 ms after Enter. +- **The A/B against base:** ReleaseFast, interleaved, idle machine, + `core_optimize` fast on both sides. + - The core: medium 0.89 to 1.11, small 0.93 to 1.08. The edit cells + were the high end. Rechecked on the medium file with 40 reps and four + rounds, edit-twin reads 0.99, edit-char 1.00, key-down 0.99 and render + 0.95: the noise this cell's A/A showed before (0.79 to 1.46). + - Live tty keystroke to present: each run within base's own spread + (key-down 1769 to 2071 µs against 1586 to 2007). + - Live window: present 32.7 to 46.9 ms against 32.3 to 48.6, core 227 + to 568 µs against 229 to 581. +- **Found on the way, not fixed:** once, a window client of a session on + the box diverged at every rejoin, each time at the first unit's digest + (tape byte 119), for the 25 s it ran. The session was a directory and two + shells. + - Not reproduced since: not by the same launch, not by four session + shapes, and not by the session run with per-field digests + (PARDES_REPLICA_TRACE, which never diverged). + - The single-word digest is `walk.digest` over the whole core. The + per-field parts cover the core's fields, its stream and each pane's + Pane, File.State and Places. So the word covers something the parts do + not. + - The next step is a client that, when a word differs, asks the session + for that unit's parts, so the field is named the first time. + - The client heals by rejoining, at the cost of a redraw each time. -- cgit v1.3 From 3f62f4e2a0f40e9d92f79665c2e87804c2d55357 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 15:57:57 -0300 Subject: docs/next-daemon.md: the remote bugs, fix3's: an error, a stat and a search's budget that meant what their build said, the guard in semantics, the cross-build replays, a window client of the box before and after, and a MessageLinger 0 replay leak found on the way Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index e2101740..28449487 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3650,3 +3650,82 @@ same scripts (`.scratch/daemon/final/final2.sh`). - The next step is a client that, when a word differs, asks the session for that unit's parts, so the field is named the first time. - The client heals by rejoining, at the cost of a redraw each time. + +## The remote bugs the user hit + +### fix3: what the replica stream carried in its build's own words + +A unit of the replica stream is its session's tape, written by +`walk.encode`, which said "same build only": a window client (pardes-gui) +of a terminal's session, or of the box's, read another build's bytes. +Found by the variant hunt (V2, E), with real ReleaseSafe executables. + +- **An error crossed as its number** (`@intFromError`, read back with + `@errorFromInt`). The numbers are a build's own: 451 of 505 shared names + are numbered otherwise in the Linux window's build than in the terminal's, + 244 between the Linux and the FreeBSD terminal's (FileNotFound is 62 in + both terminals and 60 in the window, whose 62 is NoDefaultValue). A + terminal's tape replayed by pardes-gui panicked at its first failed read + (`unexpected error code, found error.NonLocalPath`); a replica read the + wrong error, and diverged. + - Fix: an error is taped by its name. A set decodes it among its own + names; a name this build lacks is `error.NoSuchError`, a divergence, + never another error. `anyerror` keeps no list of names, so the tape's + five (a host's reports: `shell_failed`, `save_failed`, ...) read back + as one of `tape.Failure`, the union of the sets a host reports from. + ponytail: kept by hand; one outside it panics a Debug build as it is + taped, and is a release build's divergence. +- **A stat crossed as `std.Io.File.Stat`** (the `os_stat` site, /os over + 9P), whose `nlink` is 4 bytes on Linux and 8 on FreeBSD and `permissions` + 4 and 2: a Linux client read a FreeBSD session's stat out of step. Fix: + the site returns `fs.OsStat` (kind, size, mtime), the same everywhere. +- **The guard:** `tape.shape`, what every type the tape carries is laid + out as (each field, element and tag, an enum's names and numbers; an + error as an error), goes into `wire.semantics`. Builds whose tapes differ + in shape refuse each other's replicas in words at the hello, as builds of + other options do; a site whose result `tape.results` does not list is a + compile error. The sweep that set it (every type the tape reaches, from + the Linux terminal, the Linux window and the FreeBSD terminal builds): the + stat was the only one laid out otherwise. All three builds now hash to + one semantics; with the stat taped as before, the Linux and the FreeBSD + build hash to two. No wire version for it: every build's semantics + changes with it, so an older build is refused in words. +- **A search's answer was the build mode's (E).** regexp.zig's step budget + was 4M in Debug and 32M in a release build, and semantics names no mode: + `\s*(\w+)\s*=` over 200 letters was TooSlow in a Debug replica and no + match in its ReleaseSafe session. Fix: 32M in every mode (a Debug build's + worst search takes about two seconds). +- **Found by the cross-build replay, the same class:** a core's first view + was a window's by the build's default (`windowed = platform == .gui`), so + a window's build replaying a terminal's session started otherwise (its + init file's window settings applied); now an option (`Options.windowed`), + taped with the rest. And a window's font words (`observe_font` and the + three beside it) were no-ops in a terminal's build, so a terminal's + replica of a window's session diverged at the window's first font; every + hosted build takes them now. +- **Tests:** walk.zig's "an error is taped by its name" and regexp.zig's + "a search's answer is every build mode's" (from the variant hunt; it + fails in Debug before). `test/cross_replay.py ...`: + a session that meets the host's failures (a Look of a fifo and of a + directory, a Save into a file, a Restore of no dump, a stat and a read + under /os), recorded by one build and replayed by the others, and + `--record` and `--replay` alone for the box. Another frontend's tape must + agree in the replica that draws nothing, as a replica client draws none of + its session's frames: the two that draw meet the other build's drawing, + which owes frames for animations of its own (on 132 snapshot tapes, a + window's build agrees in all three on 13). `zig build + replay -Dreplay-with=` replays every snapshot script's tape + with the window's build as well. + - Before: pardes-gui panicked replaying a terminal's tape, and live, a + window client (main's ReleaseSafe) of a session on the box (main's) + panicked the moment the session looked at a fifo: `unexpected error + code, found error.NonLocalPath`. + - After: every direction agrees, tty to gui and back, the FreeBSD build's + tape replayed here by both Linux builds and theirs on the box; live, a + window and a terminal client here of a session on the box agree after + the same failures. +- **Not fixed, found on the way:** in a session whose init file says + `MessageLinger 0` (as test/daemon.py's do), now and then a tape does not + replay even by its own build, on main too: pane 0's message differs a + step after a failure is said (seven times in the first runs, none in 15 + since). cross_replay.py leaves the setting out. -- cgit v1.3 From 12d15ce9c00c96f5f7aa5bf30102d7ebebf0f9b7 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 16:04:16 -0300 Subject: docs/next-daemon.md: the remote bugs the user hit, fix2's: what a step read that only the authority holds (the store, Recent, the 9P server's readers and socket, the process's options) and the model for it (tape.authority), the rollback over the host's heap, units that ended with effects queued, the lowered copy's heap bytes, the live slot left aliasing; the tests, the gate, the soak twice and H17's own configuration, the copy before and after; H18 handed back with what was found (a resize leaves a replica's screen state stale for a round trip) and a fix to try Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 196 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 28449487..8ec76742 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3653,6 +3653,19 @@ same scripts (`.scratch/daemon/final/final2.sh`). ## The remote bugs the user hit +> due to the ping I know it won't be possible to have the same snappy +> feeling as the local version, but these kinds of errors have nothing to +> do with this limitation, you should do it much better, and measure it +> too since I don't want to catch those myself + +The user's window client of the box blinked, lost its panes and jumped its +cursor: each was a divergence, and the rejoin after it opened a fresh +view. An adversarial soak of replica clients (`test/soak.py`, a session +here or on the box, terminal and window clients through a jittered delay +relay, a seeded monkey of a person's actions) found the causes; the fixers +took them by root cause, each with a test that fails before its fix. Ids +are the hunt's (`.scratch/daemon/hunt/findings.md`). + ### fix3: what the replica stream carried in its build's own words A unit of the replica stream is its session's tape, written by @@ -3729,3 +3742,186 @@ Found by the variant hunt (V2, E), with real ReleaseSafe executables. replay even by its own build, on main too: pane 0's message differs a step after a failure is said (seven times in the first runs, none in 15 since). cross_replay.py leaves the setting out. + +### fix2: what a step read that only one side had, and units that ended undrained + +Each of these made every client of a session diverge at the same unit, so +the session's step and its replicas' replay disagreed. None was a lag. + +- **A step read what only the authority holds** (H4, H8, H11, H12, H13, and + the variant hunt's list). The store's closed panes and Recent are the + authority's (`.authority`: every window of the user writes the store's + file); a replica keeps its own, which has no file and only what it + replayed since it joined. Steps made digested state of them: + - the Panes picker's closed rows and Recent's rows (H4: every client + diverged at the key that opened them; H13: a client on another machine + read its own store); + - `trackJump`, at every step, dropped a closed file's jumps when Recent + did not know the file (H11: once a pane had closed, every client that + joined diverged at its first unit, forever, in `jumps njumps jcur`); + - `noteClosed` looked in Recent before it asked the host whether the file + is there (`recentKeeps`, a taped stat), so the session skipped a + syscall the replica made (H12: every Del of a file crashed the client); + - a closed pane brought back (`Store.reopen`: a Look at `@pN`, a Panes + row), a jump to a closed file and the +Jumps rows (`recentByHash`); + - and, the same class: a Look or exec handed to a script reading a pane's + events (`pane.fs.readers`, the 9P server's), Tty9p's socket check, and + the session's config directory and startup file (`opts`, `.local`: + ThemeFile, DumpThemes, Config, DumpConfig, an init file saved). + + **The model:** what only the authority holds stays the authority's, and a + step reads it through `tape.authority` (src/tape.zig): a `value` read, + its answer taped; a replay takes the answer, and a speculation that + reaches one waits for it, as at a host call (one RTT, never wrong). A + replica's own store and Recent are never read by a step. Rejected: the + store and Recent as session state (dumped, digested, copied), which + needs the store's file I/O on the tape call by call (it reads other + windows' records at every step that writes), for the same rows. + - `trackJump` no longer reads Recent: a closed file's jump stays, and the + jump to it drops that file's entries when Recent has forgotten it, as + when the file is gone. + - `noteClosed` asks the host first, on every side. + - The tty toggle, read on every key (which cannot wait a round trip), is + session state: `Pardes.tty_toggle`, from the options when the core is + made, dumped. +- **A rollback put back pointers to freed memory** (H8, the segfault in the + Panes picker). The copy saves a struct's bytes field by field, the + `.authority` ones included, but walks only what the digest walks. The + store and Recent live on the host's heap (`recentGpa`), whose frees the + epoch never puts off: a speculated Del grew the closed list, the + rollback put the old header back, and the next listing read a freed + name. `store` and `recent` are never saved now (rollback.zig `unsaved`); + a step never reads a replica's own. +- **A unit taped outside a step ended with effects queued** (H6, H10). A + leave is the session's `closeView`, whose sync sizes again a terminal the + closed view sized and queues its pty's resize; the session did it at its + next pump, in another unit. A replica that replayed the close kept the + effect queued, and its next idle copy asserted (H10: a client killed when + another left). A client joining at that boundary was dumped with it + queued, which a loaded core has none of, and diverged at the session's + next pump (H6: `the core called poll_frame where the tape has a host + (12)`). `closeView` and `openView` settle inside their own input now. +- **The lowered copy of a searched text held heap bytes** (H5's and H17's + `places.hits|hay`). A smart-case search lowers the text a piece at a time + into `Places.hay`, made with `memory.unfilled`; the digest and a join's + dump take the whole slice, so the pieces not searched yet were whatever + each process's heap held. Zeroed where it is made: 20 µs for 1 MB, + 0.4 ms for 10 MB, 10 ms for 100 MB in one frame (ReleaseFast, page + faults included; lowering the same text is four times that, over + frames), the order of the copy an edit of that text already makes. +- **A view drawn outside a step left the live slot aliasing** (H5's + hypothesis, in a path it did not name). `withParked` (a replica client + drawing or reading its own view while another is live) parked the live + view and copied it back without emptying its slot or its panes' shares + of selections, which the rollback copy walks. Not the cause of H5's + signatures, which were the jumps (Recent) and the places (the lowered + copy) above; the slot was saved and put back all along. + +Tests, each failing before its fix: src/rollback.zig ("a rollback leaves the +store and Recent as a speculation left them", "a view drawn outside a step +leaves the live view's slot holding nothing"), src/replica.zig ("a replica +joined after a pane closed", with the variant hunt's Recent written by +another window; "a replica's Del of a file its session's Recent knows"; "a +client leaves, and one joins"), src/Places.zig ("a lowered copy's pieces not +searched yet are the same in every process"). The hunt's repros +(`panes_closed.py`, `del_file.py`, `recent_store.py` with the client's own +XDG_STATE_HOME, `leave_resize.py`) agree on the fixed builds; on the +installed 0.24 `panes_closed.py` still rejoins 16 times. `zig build +replicas` now looks at files, closes panes, opens the Panes and Recent +pickers and jumps. + +The gate, on fix2's five changes (`.scratch/daemon/fix2/gate.sh`): + +- **Debug:** `core-test` tty; `core-test` gui and `unit-test -Dplatform=gui` + fail only on main's two (`tagline pointer mapping...`, `a tag caret past + its row's end...`); `unit-test` tty, `snap` (113/113), `replay`, + `replicas`, `test/daemon.py` 20, `--far`, `--manage`, `--upgrade`, + `--window-clients`, `--gui` 5 and `test/fs.py` pass. +- **ReleaseSafe:** `test/daemon.py` 20, `--far`, `--window-clients`, + `--gui` 5, `--upgrade` from Debug and `test/fs.py` pass. +- **Rebased onto fix3's line** (myulmrpt; its `tape.results` now lists + what `tape.authority` returns): `core-test` and `unit-test` tty, + `snap`, `replicas`, `test/daemon.py` 20, `--far`, `--window-clients`, + `--gui` 5 and `test/fs.py` pass again in Debug; `test/daemon.py` 20, + `--window-clients`, `--gui` 5 and `test/fs.py` in ReleaseSafe. +- **The copy** (`zig build rollback-bench`, ReleaseFast, main's + rollback.zig against fix2's): a typical session's 980.3 KB to 980.1 KB, + a big one's 1487.9 to 1487.8 (the Store's and Recent's own bytes; their + heap was never saved). Its take 0.045 and 0.063 ms, 0.075 and 0.077; + its restore 0.036 and 0.034 ms, 0.053 and 0.054: noise, one run each. + +The soak (`test/soak.py local --minutes 5 --rtts 0,30,80,207 --clients +tty,win --trace`, ReleaseSafe, the mouse on, Upgrades, resizes, detaches +and scripts among the moves), twice: + +| cell | client | min | inputs | links | visible | mispred | p50 / p99 ms | cut short by | +|---|---|---|---|---|---|---|---|---| +| 0 ms | tty | 1.4 | 330 | 2 | 0 | 0 | 1.5 / 1298 | H7 (session) | +| 0 ms | window | 1.4 | 225 | 3 | 6 | 2 | 4.5 / 12.4 | H7 | +| 30 ms | tty | 0.6 | 101 | 1 | 0 | 0 | 43.2 / 1283 | H7 | +| 30 ms | window | 0.6 | 56 | 1 | 0 | 0 | 4.8 / 25.4 | H7 | +| 80 ms | tty | 5.1 | 962 | 1 | 6 | 30 | 14.4 / 1570 | | +| 80 ms | window | 5.1 | 911 | 3 | 63 | 35 | 5.9 / 117 | | +| 80 ms | window, attached later | 0.8 | 35 | 0 | 0 | 0 | 2.0 / 3.6 | its replay panic (H9) | +| 207 ms | tty | 0.8 | 98 | 1 | 0 | 0 | 42.3 / 1879 | H7 | +| 207 ms | window | 0.8 | 84 | 1 | 0 | 0 | 6.3 / 18.7 | H7 | + +- Every divergence of that run was the pointer's: `pointer_tag_hit + look_hover_wait` (H3, fixer 1's: a tag hit made from the process's own + frame), which takes every client of the session at once, and a window + client's replay that met the `pointer` site where the tape had none (H9's + panic, the same class). None of this section's classes. +- H7 (unassigned: `integer overflow` in ghostty's PageList under + `Screens.flush`, after a unit with a `pty_resize`) ended the session in + three of four cells; the first run's cells, of the same seeds, did not + meet it. +- The first run (its table lost to the harness, which raised when a session + with unsaved text would not Exit; fix2's copy now ends it by pid and + counts a breach): 0 ms, three rejoins on H3 alone; 30 ms, one window + client's replay panic at the `pointer` site; 80 ms, three on H3 and three + of H18's (`jumps … pane N.body` at units with a `pty_resize`, every + client together). +- H17's own configuration (30 ms, seed 22, 4 minutes, where both clients + had died at `n` on different places): no divergence in `places`. Its + rejoins were H3's (five) and H18's (two units, `jumps … pane 0.body` + and `pane 0.body pane 0.ovl`, each with a `pty_resize`). + +Not done, and found on the way: + +- **H18, handed back: a terminal's resize leaves a replica's screen state + stale for a round trip.** Two signatures, both on every client at one + unit (`to_view`, a key, a `step` with a `pty_resize`): + - `pane 2.ovl` (the hunt's seed 5 at 80 ms, deterministic): an edit copy + extended over a three-row terminal read history rows that both + replicas held as blank holes where the session read text, and the + session sent none before the unit (`Screens.fill`). The resize in that + unit was another pane's. An instrumented build showed the step read + the rows through the core's cache (`shellRows`), which marks a + terminal read only when it builds the rows, so `fill` skips it; and + with that fixed the unit still diverged. + - `jumps … pane N.body` (this table's 80 ms cell, H5's own signature). + - The window: a resize renumbers a terminal's rows (`Rows.epoch`); the + session's next delta is against the old base, a replica refuses it + (`Renumbered`) and is started afresh a round trip later, while + `noteSent` took it as applied and expects the replica's holes where + they were. A step that reads the terminal meanwhile, or in the step + that resized it after the reflow, reads another grid on each side. + - The likely fix, untried: a delta across a renumbering is a reset + (`encode` with no base when the base's epoch is another), so the peer + is afresh at once and `fill` knows its holes; `shellRows` marks a read + on a kept hit too; and, with screen-state replicas, the session + reflows at the cut of the step's unit, not inside it, so the step + reads one grid on both sides. A test needs Screens with two replicas + of different sizes and a short terminal with a history. +- **Two edges of the same class, left:** a pipe or an Edit in flight when + a client joins (`Pipe.wait`, `.authority`, not dumped: the joiner drops + its answer), and a 9P batch held at a join (`fs.batch`). Both need a + join in the instant between a request and its answer. +- **A Debug session with `PARDES_REPLICA_TRACE` is no soak:** its per-field + digests rehash every blob at every unit (`walk.rehash`, Debug's check), + a whole undo history of a big file a unit, and a boundary took 62 s. + Soak ReleaseSafe, or Debug without `--trace`. +- **std's segfault handler panics while it symbolizes** (H8's S3: + `SelfInfo.Elf.findModule`, index out of bounds), so a crash record says + that and not the segfault. It is std's, not pardes's (`crash.zig` takes + panics only). -- cgit v1.3 From 5938a84d98815818598f5505646edec762089270 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 15:19:25 -0300 Subject: docs/next-daemon.md: the remote bugs the user hit, fix's: the session's abort after 15 far joins (c_heap), the window's rejoins at every tag hover and click (a hit resolved on this process's frame), a reflow's overflow of the screen state, a replay's panic at a site it lacked, a rejoin's lost layout, a client's end (its ssh, a gone session redialled 20 s, a dying session's links); each one's test and numbers, here and on the box, and what of the user's symptoms is left Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 8ec76742..fc66edb1 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3925,3 +3925,113 @@ Not done, and found on the way: `SelfInfo.Elf.findModule`, index out of bounds), so a crash record says that and not the segfault. It is std's, not pardes's (`crash.zig` takes panics only). + +### fix: the session's abort, the window's rejoins, a client's end + +The user's window client of the box's session (`pardes-gui +--attach=root@65.109.81.103:freebsd`, 2026-10-30): the cursor not keeping +up, the window blinking, panes closing just after they opened, then a +freeze. The first three were the client joining again, 15 times in its +25 s; the freeze was the session's abort at about its 16th far join, and +the client's 20 s of redials after it. An adversarial hunt (a soak harness +and a variant sweep) found the rest below. Each fix has a test that failed +before it. + +- **The session aborted after 15 far joins (H2).** `c_heap: more distinct + allocators than it can name`, from zstd's context on a peer's writer + thread. The core's table held 64 copies of one allocator, `ptr` 0 and + c_allocator's vtable. `zstd.gpa` is `std.heap.c_allocator`, whose `ptr` + is undefined, and nothing outside a test writes it: a release build made + it a constant and folded `known.ptr == current.ptr` to false, so every + malloc named a new allocator (about four a join) and the 65th panicked. A + client's decompressor (one a join) would have aborted after about 30 + rejoins; the web shell's heap over `wasm_allocator` trapped the same way. + - Fix: c_heap reads `current` through a volatile pointer, as its bytes + are, so no heap compares a value the compiler may take for anything. + - Tests: c_heap's own, which unit-test now runs built ReleaseSafe too (it + panicked at the 65th malloc); `test/daemon.py --far` makes 200 far joins + of one session, each read whole, and checks its fds, threads and RSS. + The installed 0.24 aborted with c_heap's panic within that run. Now, + ReleaseSafe: 200 joins in 1.7 s, fds and threads level (8, 8), RSS +112 + KB; on the box (FreeBSD ReleaseSafe, through ssh): 200 in 185 s, fds 13 + and threads 11 level, RSS 48964 to 49168 KB. +- **A window's pointer over a tag diverged it at once (H3, the user's + `tape byte 119`), and a click there aborted it (H9).** A step resolved a + frontend's pointer hit again on `p.surface`, this process's last frame, + which is `.local`: the session never draws a client's view, so the same + motion gave the client a tag hit and the session none (`pointer_tag_hit`, + `look_hover_wait`). Body hits did the same through `reprojectBodyHit` and + a sweep's clamp (the variant hunt's F). + - Fix: a hit carries the place it was made on (`Layer.Place`: the layer's + viewport, columns, rows and compact rows) and the step resolves it from + that, never from a frame; a sweep keeps its body's place to clamp onto. + The wire carries the place and a tag hit's line: version 17. + - Tests: `--window-clients` sweeps the pointer over the tags and the body + and clicks there, at 0 and 207 ms (before: joined again, `tape byte + 119`); mouse.zig's two tests from the variant hunt. The hunter's + repros: 15 rejoins in 3 s of hovering a tag before, none after; clicks, + drags and B3 in a window aborted it before, none after. +- **A replay that met a site its tape lacked panicked (H9, H12).** Fix: a + site or value the replay does not find is a divergence and returns what a + speculation gets (`poisonOf`): any later leak costs a rejoin, never the + process. Test: tape.zig's. +- **A view closing left a terminal's resize queued (H10, H6).** `closeView` + syncs, which sizes a terminal the closed view sized and queues its + `resize_pty`, with no step after it to drain it: a replica's next copy + asserted, and a speculation over the queue lost the effect when it rolled + back, so the next pump's replay found no `pty_resize` where the tape had + one. With the user's config (a font of their own sizes their view) every + rejoin diverged at its first keys. + - Fix: `closeView` settles in its own unit (fix2's, above, which took + the place of a drain here), and so does `leaveView` below; a copy is + refused (`error.Queued`), not asserted, over a queue, and the input + waits for the authority. + - Tests: replica.zig's in-process one (two clients at two sizes, one + leaves; the other copies, types and agrees). The user's config with a + random mix of keys and pointer: 18 rejoins in a minute before, none in + four minutes after. +- **A reflow overflowed the session's screen state (H7).** A resize to + fewer columns puts more rows above the last active top than ever scrolled + past it, so `top - history` underflowed where a peer starts afresh; a + window client's font made such resizes. Fix: `Rows.update` keeps `top` + at or past the history's length (a resize renumbers anyway). Test: + screen_state.zig's, 200 long lines narrowed. +- **A rejoin dropped the client's layout.** Its new view copied the live + one, the session's own as it was before the client came. Fix: a session's + last client, while nobody else watches the session, leaves it its layout + (`leaveView`: columns, focus, scrolls, cursors and size), which a rejoin + (an Upgrade's, a divergence's) copies back. Test: views.zig's. +- **A client's end.** One refused at a redial exited from inside itself and + left its ssh and `/tmp/pardes-ssh-.sock`; one rejoining after a + divergence redialled a gone session for 20 s; a session that panicked + kept its clients' links open for the seconds its trace took. + - Fixes: the ssh is ended at `atexit` and dies with its client + (PDEATHSIG); a redial waits only after an Upgrade's `refused` and else + says "could not join again"; the panic and segfault handlers shut every + client's link first. + - Tests: `--far` attaches through an ssh stand-in, refused and detached, + and finds both gone; `test/daemon.py` kills a session under a replica + client, which ends within 2 s, in words; replica.zig's redial test. +- **A far join's leave held the session's step** while the writer hashed + the rest of its blobs (the hunter's review, unconfirmed). Fix: the writer + stops at the next blob once it is let go. ponytail: one blob's hash (about + 30 ms for 100 MB) is the ceiling. +- **Kept as diagnostics:** the trace's unit descriptions give a key's and a + pointer's words, and a confirmed speculation's state is checked against + its unit's digest, which names the input that diverged rather than a later + unit replayed. +- **The user's symptoms, rechecked** on the box (ReleaseSafe at both ends, a + window here with the user's own config through ssh, a monkey of keys, + pointer, wheel, clicks and shell commands, 2 minutes), on this line over + fix2's and fix3's: + - Rejoins: none in 760 actions, 20 checks all agreed. (Before fix2's + line, one in 799, at a `.kind` site its tape lacked, which had aborted + the client.) Panes no longer close: a rejoin keeps the layout anyway. + - The freeze: the session no longer aborts, and a client of one that does + ends at once, saying so. + - The blinking and the cursor: 4 visible rollbacks in the 2 minutes (4 + inputs of 2375 mispredicted; 45 and 32 of 2369 before fix2's line). + Each is a clock guess: an edit's file time a second off across a + second's turn, or the millisecond clock the steps of Ctrl-W, Ctrl-O and + Ctrl-U read. Not fixed: a guess of the session's clock is the design's + ceiling there; reading a coarser clock in those steps is the next step. -- cgit v1.3 From 945490c9cbbad797c76f165be2718ac40d03941b Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 17:19:12 -0300 Subject: docs/next-daemon.md: fix3's second round: H19 (an enum digested by its number), the linger leak, H15 (Adopt), the web build, and a 9P Grep's untaped search found on the way Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index fc66edb1..9589c24f 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3737,11 +3737,42 @@ Found by the variant hunt (V2, E), with real ReleaseSafe executables. tape replayed here by both Linux builds and theirs on the box; live, a window and a terminal client here of a session on the box agree after the same failures. -- **Not fixed, found on the way:** in a session whose init file says - `MessageLinger 0` (as test/daemon.py's do), now and then a tape does not - replay even by its own build, on main too: pane 0's message differs a - step after a failure is said (seven times in the first runs, none in 15 - since). cross_replay.py leaves the setting out. +- **The second round (H19, the linger leak, H15, the web build):** + - **A Linux client could not join a FreeBSD session holding a + +LocationsConfig (H19):** `the session's dump loaded as another + session`. An enum was digested by its number, and `registry.Builtin`, + made at compile time, has Tty9p in a Linux build only, so the pane's + origin (`Output.State.from = .{ .cmd = .LocationsConfig }`) was another + number in each. Found by one dump loaded in both builds, its digests + printed field by field. Fix: an enum is digested by its name's word (a + fixed-width hash, looked up by number), as the dump already reads it by + name; one numbered by hand stays its number. Before, main's window + client of main's session on the box could not join; after, a terminal + and a window client here join a FreeBSD session holding a +Search, a + directory, a renamed file and a +LocationsConfig, and agree. + cross_replay.py's session holds the last two (a FreeBSD tape replayed + here diverged at `pane 2.file.output`). + - **A detached session's messages went in the host's wait** (tty + waitInput, gui waitDetached), untaped, so a replay, and every replica + of a session nobody attached, kept a message its session had let go; + with `MessageLinger 0` (test/daemon.py's sessions) every tape whose + session said a failure diverged at pane 0's message. They go in the + pump now, after its wait, the clock read taped. cross_replay.py's + session says `MessageLinger 0` again. + - **Adopt and a window's failed terminal reader wrote the core between + steps (H15):** the Tty word run and the greeting cleared straight on + the core, and a pane's mode set; every client of a session that took in + a dead session's shells joined again. Now inputs (`adoptShell`, + `readerFailed`); replica.zig's test fails with the old writes. + - **`zig build web` failed** on main at terminal.zig:601 (`c_allocator` + with no libc), and at walk.zig's blob lists behind it: both are a + hosted build's alone now. The web build passes `touch.mjs`; the web + driver's own CDP-deadline test fails as it did, untouched. + - **Found, not fixed:** a Grep or Find a 9P write runs searches on the + loop (Stream.runHere, so that the write's answer says whether it found + anything), its reads no tape site: a replay of it panics (`a syscall + that is no tape site`), by any build, and so would a replica of a + session a script greps in. ### fix2: what a step read that only one side had, and units that ended undrained -- cgit v1.3 From 2d269375f7b49cf430a2c2d62caed88aa5c10e37 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 18:27:55 -0300 Subject: docs/next-daemon.md: fix2 on H18 (the same mechanism as fixer 1's, which lands; its create must make resize_to null), the kept-rows read flag, and the round's gate Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 66 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 26 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 9589c24f..b7820b06 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3848,13 +3848,52 @@ the session's step and its replicas' replay disagreed. None was a lag. signatures, which were the jumps (Recent) and the places (the lowered copy) above; the slot was saved and put back all along. +- **A step that resized a terminal read rows its replicas could not be + sent** (H18). A client's first key into a terminal sizes it to its view + (`typedInto`, `sizes`); one that also read the shell's history in that + step (an edit copy over its text, `ovl`) read rows the replicas held as + holes. The session's `sync` reflowed its emulator inside the step, so + the rows `Screens.fill` sent before the unit were the reflowed grid's, + in the resize's new numbering (`Rows.epoch`); a replica replaying the + step on the grid its last delta made drops rows of another numbering + and read blanks. Every client diverged at that unit (`pane N.ovl`, and + `jumps … pane N.body` where the rest of the step read the reflowed + grid; `registers pane 2.body pane 2.ovl` in fixer 1's soak, a `d` in + a terminal's text whose step resized it). Seen with an instrumented + build on the hunt's seed 5 at 80 ms: the replica's `rows pane 2 epoch + 69/68`. + - The fix is fixer 1's (`terminal.resizeGrid` notes `resize_to` on an + authority with screen-state replicas, made after the unit in + `Peers.unit` and at `Peers.stop`), found on both lines at once; fix2's + was the same and is dropped. With it, `terminal.create` must make + `resize_to` null: it sets every field by hand, and the heap's bytes + there reflowed an emulator to 43690x43690 in a Debug run (0xAA). + - Found on the way: `shellRows` marked a terminal read only when it + built its rows, not when it handed out the kept ones, so a client that + came since (its history all holes) was sent none of what a step read. + - Not a stale round trip, as first supposed: a delta across a + renumbering is a reset already. The hunt's H7 fix (fixer 1's + `Rows.update`) is under this one. + - The gate on fix2's version of it (main mqkxnsqu, fixer 1's line): Debug `core-test` + and `unit-test` tty, `snap`, `replay`, `replicas` (none diverged or + unequal at 30, 80 and 207 ms), `test/daemon.py` 20, `--far`, + `--manage`, `--upgrade`, `--window-clients`, `--gui` 5, `test/fs.py`; + gui `core-test` and `unit-test` on main's two; ReleaseSafe + `test/daemon.py` 20, `--far`, `--window-clients`, `--gui` 5, + `--upgrade`, `test/fs.py`. Once, `test/daemon.py` 20 failed in its + far clients' end: a session asked to Exit twice closed its 9P link + before the second write's clunk was answered, an `EOFError` the test + takes for no `OSError`; twice again it passed. + Tests, each failing before its fix: src/rollback.zig ("a rollback leaves the store and Recent as a speculation left them", "a view drawn outside a step leaves the live view's slot holding nothing"), src/replica.zig ("a replica joined after a pane closed", with the variant hunt's Recent written by another window; "a replica's Del of a file its session's Recent knows"; "a client leaves, and one joins"), src/Places.zig ("a lowered copy's pieces not -searched yet are the same in every process"). The hunt's repros +searched yet are the same in every process"; src/replica.zig's "a step +that reads a terminal's rows as the core keeps them"). +The hunt's repros (`panes_closed.py`, `del_file.py`, `recent_store.py` with the client's own XDG_STATE_HOME, `leave_resize.py`) agree on the fixed builds; on the installed 0.24 `panes_closed.py` still rejoins 16 times. `zig build @@ -3919,31 +3958,6 @@ and scripts among the moves), twice: Not done, and found on the way: -- **H18, handed back: a terminal's resize leaves a replica's screen state - stale for a round trip.** Two signatures, both on every client at one - unit (`to_view`, a key, a `step` with a `pty_resize`): - - `pane 2.ovl` (the hunt's seed 5 at 80 ms, deterministic): an edit copy - extended over a three-row terminal read history rows that both - replicas held as blank holes where the session read text, and the - session sent none before the unit (`Screens.fill`). The resize in that - unit was another pane's. An instrumented build showed the step read - the rows through the core's cache (`shellRows`), which marks a - terminal read only when it builds the rows, so `fill` skips it; and - with that fixed the unit still diverged. - - `jumps … pane N.body` (this table's 80 ms cell, H5's own signature). - - The window: a resize renumbers a terminal's rows (`Rows.epoch`); the - session's next delta is against the old base, a replica refuses it - (`Renumbered`) and is started afresh a round trip later, while - `noteSent` took it as applied and expects the replica's holes where - they were. A step that reads the terminal meanwhile, or in the step - that resized it after the reflow, reads another grid on each side. - - The likely fix, untried: a delta across a renumbering is a reset - (`encode` with no base when the base's epoch is another), so the peer - is afresh at once and `fill` knows its holes; `shellRows` marks a read - on a kept hit too; and, with screen-state replicas, the session - reflows at the cut of the step's unit, not inside it, so the step - reads one grid on both sides. A test needs Screens with two replicas - of different sizes and a short terminal with a history. - **Two edges of the same class, left:** a pipe or an Edit in flight when a client joins (`Pipe.wait`, `.authority`, not dumped: the joiner drops its answer), and a 9P batch held at a join (`fs.batch`). Both need a -- cgit v1.3 From 118a07c0ec5e4421fe6d5187763f8d6f9e6cd300 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 18:27:20 -0300 Subject: docs/next-daemon.md: fix3: a 9P write's Grep or Find, its search a tape site Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index b7820b06..a71bb3d9 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3768,11 +3768,14 @@ Found by the variant hunt (V2, E), with real ReleaseSafe executables. with no libc), and at walk.zig's blob lists behind it: both are a hosted build's alone now. The web build passes `touch.mjs`; the web driver's own CDP-deadline test fails as it did, untouched. - - **Found, not fixed:** a Grep or Find a 9P write runs searches on the - loop (Stream.runHere, so that the write's answer says whether it found - anything), its reads no tape site: a replay of it panics (`a syscall - that is no tape site`), by any build, and so would a replica of a - session a script greps in. + - **A Grep or Find a 9P write runs** searches on the loop + (Stream.runHere, so that the write's answer says whether it found + anything), and its reads were no tape site: every replica of a session + a script greps in joined again (`a syscall that is no tape site, made + replaying`). The search is a site now (`search`), its batches what a + replica takes, as it takes a worker's (Event.search). cross_replay.py's + session greps and finds; a terminal client of a session a script greps + in agrees. ### fix2: what a step read that only one side had, and units that ended undrained -- cgit v1.3 From f1e52f7b7bfc85340a6ea62d6f50c17f54da48fd Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 17:39:26 -0300 Subject: docs/next-daemon.md: fix's gates on uxwrlzlw, and the ReleaseSafe far joins' RSS as measured there Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index a71bb3d9..820e5e17 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4000,7 +4000,7 @@ before it. panicked at the 65th malloc); `test/daemon.py --far` makes 200 far joins of one session, each read whole, and checks its fds, threads and RSS. The installed 0.24 aborted with c_heap's panic within that run. Now, - ReleaseSafe: 200 joins in 1.7 s, fds and threads level (8, 8), RSS +112 + ReleaseSafe: 200 joins in 1.7 s, fds and threads level (8, 8), RSS +108 KB; on the box (FreeBSD ReleaseSafe, through ssh): 200 in 185 s, fds 13 and threads 11 level, RSS 48964 to 49168 KB. - **A window's pointer over a tag diverged it at once (H3, the user's @@ -4083,3 +4083,11 @@ before it. second's turn, or the millisecond clock the steps of Ctrl-W, Ctrl-O and Ctrl-U read. Not fixed: a guess of the session's clock is the design's ceiling there; reading a coarser clock in those steps is the next step. +- **Gates, on this line over fix2's (uxwrlzlw):** Debug `core-test` tty + and gui, `unit-test` tty and gui (only the two GUI tests main already + failed: the tagline pointer mapping and the tag caret), `snap` 113/113, + `replay`, `replicas`, `test/daemon.py` 20 cycles, `--far`, + `--window-clients`, `--manage`, `--gui`, `--upgrade` and `test/fs.py`; + then `test/daemon.py` 20, `--far`, `--window-clients`, `--manage`, `--gui` + 5, `--upgrade` from Debug and `test/fs.py` against ReleaseSafe builds. All + pass. -- cgit v1.3 From 7d989e485a38b025d0d6639afdcb4890d572f7cd Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 18:26:04 -0300 Subject: docs/next-daemon.md: two leaks the soak showed after the first round, a resize made inside a step and a loaded search's regex, their fixes and tests, and a breach not confirmed Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 820e5e17..892d38a6 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4083,6 +4083,36 @@ before it. second's turn, or the millisecond clock the steps of Ctrl-W, Ctrl-O and Ctrl-U read. Not fixed: a guess of the session's clock is the design's ceiling there; reading a coarser clock in those steps is the next step. +- **Two leaks the soak showed after that (its local80 and local207 cells):** + - A client typing into a terminal at another size than its last typist + sized it in that key's update (sync), and the session's emulator + reflowed at once while its replicas' wait for the delta. The rows the + step read that a replica held as holes then went to it in the new + grid's numbering, which it cannot take, so its replay read them blank + (`jumps`, `pane 1.body` at a `b`). Fix: an authority with screen-state + replicas makes the size after the step's unit (`resize_to`, in + `Peers.unit`); the step reads the grid it had on both sides, and the + next flush brings the new one; `create` sets `resize_to` null, as it + sets every field by hand (fixer 2 found a Debug session's terminals + asked to be 43690 by 43690, the heap's 0xAA). Tests: replica.zig's, + which reads history in the resizing step in the session's order of + flush, fill and unit; terminal.zig's, a terminal made from 0xAA memory. + - A client that joined had its panes' places loaded without their regex + (`re` is derived: in no dump and no copy), took a search for a list's + and found nothing an edit made, so it diverged at the next edit and at + every rejoin after (`pane 3.places.hits`; `pane 17.cwd` too, which + inherits pane 2's and hashes it whole). Fix: the regex is made again + from the pattern when a set holds none. Test: Places.zig's, a dump + loaded and the same edit made in both. + - Not confirmed: "panes [5, 8] left its view while still open" in local0 + came at 104 s, before any client of that cell had left (the rejoins' + joins were at units 2028 and 2029, near its end), so neither + `leaveView` nor a closed view's settle moved them. The soak compares + pane ids (/index, /layout), and a slot is taken again by the next pane + made: two panes the terminal's view showed, closed, and the window's + Looks just before (Enter on a row) opening panes into those slots, in + the window's view only as a view's new pane goes, reads the same. A + serial a pane in /layout would tell the two apart. - **Gates, on this line over fix2's (uxwrlzlw):** Debug `core-test` tty and gui, `unit-test` tty and gui (only the two GUI tests main already failed: the tagline pointer mapping and the tag caret), `snap` 113/113, -- cgit v1.3 From 9f84ac9f0d715d7717dc1d732299bdc586422392 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 19:12:28 -0300 Subject: docs/next-daemon.md: fix's gates, run again over npzpposw and wxpkrpwo Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 892d38a6..05ccc0d5 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4113,7 +4113,10 @@ before it. Looks just before (Enter on a row) opening panes into those slots, in the window's view only as a view's new pane goes, reads the same. A serial a pane in /layout would tell the two apart. -- **Gates, on this line over fix2's (uxwrlzlw):** Debug `core-test` tty +- **Gates, on this line over main npzpposw (fix2's and fix3's), again; + then over wxpkrpwo, the builds, `core-test` tty, `replicas`, + `test/daemon.py` 20 and `--window-clients` in Debug and ReleaseSafe and + `test/fs.py` ReleaseSafe, all passing:** Debug `core-test` tty and gui, `unit-test` tty and gui (only the two GUI tests main already failed: the tagline pointer mapping and the tag caret), `snap` 113/113, `replay`, `replicas`, `test/daemon.py` 20 cycles, `--far`, -- cgit v1.3 From 97431acd8e77410dbaea9fb1093d29327999cd5b Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 19:39:41 -0300 Subject: docs/next-daemon.md: fix3: a +Search's rows recoloured in place (dropped_marks) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 05ccc0d5..74930734 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -3776,6 +3776,16 @@ Found by the variant hunt (V2, E), with real ReleaseSafe executables. replica takes, as it takes a worker's (Event.search). cross_replay.py's session greps and finds; a terminal client of a session a script greps in agrees. + - **A +Search's rows were recoloured in place** (`Query.recolor`, as a + paint lands): the rows are a blob, whose word a digest keeps by address + and a rollback copy pins, so a release session kept the word of their + colours before; an edit moved them into `dropped_marks` with it. A + replica that hashed them at another time diverged there (fixer 1's soak, + `pane 5.file.dropped_marks`), and a client loading the session's dump + hashed the colours it carried: "loaded as another session". They are + made again now (`locations.cloneRows`). `dropped_marks` stays in the + digest: an undo back to its text reads it. Stream's test fails in Debug + before (`written in place`). ### fix2: what a step read that only one side had, and units that ended undrained -- cgit v1.3 From e2d74d9160a491d270b271904d5c4338dd203cf9 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 18:21:15 -0300 Subject: docs/next-daemon.md: phase 6's instant interaction notes: what stopped each input, counted by name; pardes's own steps over a terminal speculated (H14), its view moved here and put back on a rollback, applied again over each delta; a pane's teardown no stop; the echo's lost predictions waiting for their keys; the authority's yes or no guessed; an input's time its client's; mosh's rules checked; before and after per action class at 0/30/80/207 ms and through the box, the soak's blink, the replicas harness; what is left Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 220 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 220 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 74930734..b1ee51d5 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4134,3 +4134,223 @@ before it. then `test/daemon.py` 20, `--far`, `--window-clients`, `--manage`, `--gui` 5, `--upgrade` from Debug and `test/fs.py` against ReleaseSafe builds. All pass. + +## Phase 6: instant interaction notes + +Built 2026-10-30 (`src/replica.zig`, `src/tape.zig`, `src/terminal.zig`, +`src/echo.zig`, `src/detached/wire.zig`'s `stamp`, one line in +`src/pardes.zig`; PARDES_TEST_LATENCY's `I` and `U` lines in +`src/gui/gui.zig`). The user: "make sure to use the mosh +tricks so that user interaction to type/scroll/move the cursor feels +instant specially when they don't require a request going to the daemon", +and "measure it too since I don't want to catch those myself". The +measure is test/soak.py's `instant` mode (the hunter's, per action class +against a local session's own frontend) and a bench of my own that says +what stopped each input (`.scratch/daemon/instant/harness/classes.py`). +Decisions at the forks, with the ponytail lens: + +- **What stopped an input is counted by name.** A speculation that + meets a host call, a site or a poison records which (`Tape.poisoned_by`), + and the client's PARDES_REPLICA_STATS line ends `stops name:n,...` + (PARDES_REPLICA_TRACE says it with the input). Every number below that + says why an input waited comes from it. +- **pardes's own steps over a terminal are speculated (H14).** Until now + every input of a client whose keyboard was in a terminal waited a round + trip (`a terminal has the keyboard`): normal-mode keys over its history, + the wheel, Ctrl-u. The blanket poison goes. What reaches the program + (a key in tty mode, a paste, a mouse report) is a pty write, a host + call, and stops as before; the predicted echo covers it. The rest reads + the emulator and may move its view: + - **Its view moves here.** A screen-state replica's speculation may + scroll a terminal (`terminal.viewOnly`: `scrollGrid`, `followOutput`); + nothing else of an emulator. The rollback copy holds no emulator, so + the replica keeps each terminal's view as the copy was taken + (`Replica.views`), a delta sets it the authority's, and a rollback + puts it back, between the units it replays in the order the deltas + came among them (a delta's view comes after the units it shows: a + neutral unit of another's that scrolled, replayed over it, scrolled + twice; the harness's per-delta check found it). + - **Applied again over each delta.** What it did depends on the + emulator, here the last delta's and there what the pty said since. + The stream sends a client its terminals before the authority steps + any input and the rows a step read before its unit, so a speculation + over a terminal (`Pending.terminal`: it read a terminal's rows, + moved a view, or a terminal had the keyboard) is put back and applied + again over each delta or rows that come while it waits + (`overTerminals`): when its unit comes it read the emulator the + authority's step read. + - **No digest in a speculation.** A first version kept a terminal + speculation's digest to check its unit's against; but a blob born + under the copy and digested keeps its word past the epoch's free of + it (src/walk.zig), and a later blob at that address read as it (the + harness's Debug blob check, seed 11 at 207 ms). Gone: it is confirmed + by its records, as any speculation is. + - **Never kept to replay later.** A unit confirmed on the fast path + with more applied on top is kept in `since`, to replay at the next + rollback, over the deltas that came after it. One over a terminal is + taken at once instead, over the emulator its step read, and those on + top applied again. + - Closing a terminal stops (`retirePane`: its pty's end is the + authority's); the blanket poison had covered it, and the in-process + harness's host has no `close_pty` to stop at. + - Found on the way, others' to fix and fixed in main since: with more + of a terminal's steps speculated, the replicas harness met H18 more + often (a client typing at another size than the last typist: the + session's emulator reflowed inside the step, the rows sent for the + replicas' holes came renumbered, their replay read blanks; seeds 10, + 11 and 17 at 207 ms, `pane 0.ovl`, `pane 0.sel_snap`). Main met it too + (seed 53 of 60 at 207 ms). Fixer 1's `resize_to` and fixer 2's `read` + through the rows cache (wwkwwlzt, tysmmuzx) fix it: on them, 20 seeds + at each RTT, none diverged. +- **A pane closes in a speculation.** `deinitEmulator` asked the authority + before it looked for an emulator, so the reap after any close (a file's + pane, a picker's, a preview's) stopped the input after it: 4 of the + finder's 6 stops. +- **The authority's yes or no is guessed (mosh's way).** Fixer 2's + `tape.authority` makes a step that reads what only the authority holds + wait for it: whether a script reads a pane's events, which every Look + and click's command asks, put every Look a round trip late. A bool + answer is now guessed in a speculation from the authority's last answer + to the same question (`Tape.answers`, filled as units replay), recorded + as the answer, so its unit confirms the guess or rolls it back. Lists + (the closed panes the Panes picker shows, Recent) change with what the + user does and are still asked: opening the Panes picker waits a round + trip (its typing does not). +- **An input's time is its client's.** A speculation guessed the + session's clocks (`Tape.guess`: the last unit's reading, the time since, + a round trip), and a step that read one recorded the guess; the + authority read its own, so a message's millisecond (Ctrl-W's, a jump's) + always and an edit's second across a turn sometimes recorded otherwise: + a rollback each, which a window counts as seen (fixer 1: 1 to 8 a + minute, window clients worst). Now the input carries the guess (wire + 18's `stamp` frame before its event; `Pending.guess`, the same at every + reapply), and the authority's step reads it as its clocks (`Tape.stamp`, + `stampOf`; within a second of its own, else its own), so the two record + the same time. A message's linger is then off by the guess's error, tens + of milliseconds. +- **The echo's lost predictions wait for their keys.** Found by measuring + at 207 ms: the shell's typing was predicted (24 of 26) and drawn almost + nowhere, key to output p50 65 ms. A drop with keys still in flight (a + delta that moved the screen's top, here the pty's resize at the first + key; or a misprediction) restarted the predictions at the screen's + cursor, where those keys' echo was about to land: every key after it was + judged wrong (5 of 6), and once wrong, nothing was drawn until an echo + was confirmed, which none could be. Now the keys typed after a loss wait + for the newest key in flight to show (`lose`, the `deferred` path keys + behind Tab take), then are predicted over it. +- **mosh's rules, checked** (its source is in ~/05-genizah/mosh now, + src/frontend/terminaloverlay.cc `new_user_byte`): a printable character + (inserted, the rest of the row shifted right), Backspace (the rest + shifted left), Left and Right; CR moves the predicted cursor to the next + line, tentative; everything else (Home, End, Delete, Up, Down, any other + CSI or control) makes it tentative, nothing predicted. pardes predicts + the same less the mid-line insert and delete (a line editor redraws what + is right of the cursor; fish's suggestions mispredict a shift) and CR. + Nothing mosh does is missing that would raise the measured share: at a + prompt every letter is predicted. + +Measured. Before is main at uxwrlzlw with the counting (`base2`), after +is this line on mqkxnsqu (`new2`; fixer 1's line between them changes +none of what is measured here, and the line has since moved onto +vplsqtyz); both ReleaseFast, unstamped, on the perf cores, the box's +session the same build cross-compiled. The hunter's `instant` mode +(test/soak.py, its xvmytsym copy pointed at my box directories): a +class's 25 inputs a frame or more apart in one client, against a local +session's own frontend measured in the same run. `at once` is the share +that neither stopped nor showed later than half a round trip (a window's +two frames). + +- **A terminal's history, by keys (Ctrl-b, k, Ctrl-u) and by the wheel** + (tty; at once, then key to output p99 ms; local p99 0.6 to 1.2 ms): + + | | 0 ms | 30 ms | 80 ms | 207 ms | the box | + |---|---|---|---|---|---| + | term_scroll_keys | 0% 15.2 → 100% 1.1 | 0% 54.3 → 100% 1.5 | 0% 101.2 → 100% 0.9 | 0% 232.2 → 100% 0.9 | 4% 428.6 → 100% 1.4 | + | term_scroll_wheel | 0% 1.5 → 100% 0.6 | 0% 48.6 → 100% 0.6 | 0% 99.1 → 100% 0.6 | 0% 226.3 → 100% 0.6 | 0% 207.7 → 100% 0.6 | + + The window client's, the same at once: 0% (4% on the box) → 100% at + every cell. (A window's p99 in this mode is the first frame after the + read, which a window draws whether or not the input waited: its at + once is the measure.) +- **Shell typing at a prompt** (tty, p99 ms; every letter predicted + before and after, 25 of 25): 0.8 → 1.4 at 0 ms, 1.5 → 1.1 at 30, 0.8 → + 0.9 at 80, 49.0 → 27.7 at 207, 210.6 → 18.0 on the box. My own bench's + shell class at 207 ms (lines of `echo` and three words, Backspace, + Enter): p50 65 ms (main, ReleaseFast) → 1.8 ms (this line, Debug), the + echo judged wrong 5 of 6 → 0. +- **Every file class stays at once** (type, move by keys and by click, + scroll by keys and the wheel, select, the picker's typing): 100% before + and after at every cell and on the box, tty p99 0.5 to 1.5 ms after + (a local session's 0.6 to 0.9; its wheel 48 ms, the glide). +- The gate: `instant: pass` at 0, 30, 80, 207 ms and through the box + (before: 4 to 6 breaches a cell, each a terminal class's, 28 in all). +- What stopped inputs, by name (my bench, 207 ms, before; `stops` in + PARDES_REPLICA_STATS): the shell's keys `pty_write` (their echo is + predicted), a terminal's history `scroll a terminal` 23 and `a terminal + has the keyboard` 3 of 27, the finder `close a terminal` 4 and + `preview` 2, a Look the authority's `readsEvents` 11 of 12 and + `resolve` (the first open of a file) 1, a Save `kind` (and then its + write), the Panes picker's opening the authority's closed panes; typing + in a file, moving, scrolling a file, selecting: none. After: the + shell's `pty_write`, the finder's `preview`, a Look's first `resolve`, + a Save's `kind`, the picker's opening. +- **The soak's blink** (test/soak.py `local --minutes 2.5`, its gate, + a tty and a window client a cell at 0, 30, 80 and 207 ms, far): + before (`base2`) visible rollbacks 0.0 to 1.0 a minute, the window at + 80 ms 4.2 (13 in 3.1 min, a breach), mispredicted inputs 1 to 13 a + client, and that line's crashes and rejoins (fixer 1's since). Fixer 1's + own build, with them fixed: windows 2.2 to 2.7 a minute, ttys 0 to 1.3, + clock guesses all. The final line (this on xymsrmso): 0 visible, 0 + mispredicted, 0 rejoins at every cell, tty and window; `soak: pass`. +- **The replicas harness** (ReleaseFast, 20 seeds x 120 inputs x 3 + clients): inputs that stopped 731/649/790 at 30/80/207 ms → 453/390/398; + confirmed on the fast path 6249/6246/5953 → 6695/6755/6753; mispredicted + 220/305/458 → 15/15/21; none diverged, none unequal (before, main at + uxwrlzlw: seed 9 at 207 ms unequal at quiescence, as main at vplsqtyz + still was). +- **The final line** (this on xymsrmso, fixer 1's line): the gate at + 207 ms passes again, every class at once in both clients (tty p99 1.7 + to 4.4 ms, a local session's 0.5 to 1.4 ms; shell typing 25 of 25 + predicted, p99 28 ms); and through the box, every class at once in + both clients (tty p99 0.5 to 2.4 ms, a local session's 1.6 to 3.6; + shell typing p99 1.7 ms, 210.6 before). + +The gates (on the final line): Debug `core-test` tty, `unit-test` tty, +`snap` 113/113, `replay` (20 seeds), `replicas` (4 seeds) and `--echo`, +`test/daemon.py` 20 cycles, `--far`, `--window-clients`, `--manage`, +`--gui`, `--upgrade`, `test/fs.py`: all pass; `core-test` gui and +`unit-test` gui fail only main's two (the tag caret, the tagline pointer +mapping). ReleaseSafe (`zig build -Doptimize=ReleaseSafe`): +`test/daemon.py` 20 cycles, `--far`, `--window-clients`, `--manage` and +`test/fs.py` pass. The new unit tests: a terminal's history +by keys at once with a delta while they wait (replica.zig); keys still on +their way when their predictions go (echo.zig); an edit's second across a +turn confirmed on the fast path (replica.zig, which fails without the +stamp). + +Not done, and why: +- Opening the Panes picker and Recent wait a round trip (their lists are + the authority's, and change with what the user does); the finder's + preview reads the file it previews (a host call); a Save waits for its + `kind` site and then its write; the first Look of a pane waits once (no + answer to guess from yet). Each is one round trip, never wrong. +- The first key typed into a shell scrolled back waits a round trip: + typing snaps the view to the bottom, but the key is a pty write and its + speculation, the snap included, is undone; the echo predicts only at the + live bottom. +- Under the in-flight bound (32 deltas) a step can read a terminal state + the client was not sent: replayed, its unit diverges (a rejoin), as + another client's always could; a speculation of one is confirmed by its + records, so it is caught at the next digest a replay checks (fixer 1's + check when nothing is left on top), not at once. +- A window counts every misprediction followed by a present as seen, + whatever it drew (a tty only one that wrote to its terminal): an upper + bound, now 0 in the soak. +- A terminal's history is not fetched ahead of its view: a scroll back + (at once now) into rows a flood skipped shows them blank for a round + trip. Asking for two screens around the view diverged a replica in the + harness (seed 10 at 30 ms) before H18's fix, which may have been H18 + met in the other order of events it made; dropped, not tried since. +- The window's present is its frame (a software present in the test, up + to a frame): a prediction is drawn in the same pass as its key, so it + cannot come sooner than the window's own frame. -- cgit v1.3 From e6a114eed043dd1ca50be1cab912662fa9150fda Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 21:04:02 -0300 Subject: dump: a closed terminal brought back on a screen-state replica gets its emulator on the C heap, as terminal.create makes one (terminal.emulatorGpa): loadPane named no `.terminal` allocator, so its emulator was the rollback epoch's, and its screen replica's blobs, made with State.gpa, were freed with the C heap's behind the epoch at a client's end and again by it (H21: `double free or corruption` in Epoch.deinit at a hangup, `free(): invalid pointer` at a rejoin), and a rollback freed what a delta under a held copy made of them Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 21 +++++++++++++++++++++ src/dump.zig | 2 +- src/replica.zig | 25 +++++++++++++++++++++++++ src/terminal.zig | 10 ++++++++-- 4 files changed, 55 insertions(+), 3 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index b1ee51d5..6dfa799c 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4354,3 +4354,24 @@ Not done, and why: - The window's present is its frame (a software present in the test, up to a frame): a prediction is drawn in the same pass as its key, so it cannot come sooner than the window's own frame. + +After the line, the hunt's H21 (a client's normal end aborted): +- **A closed terminal brought back put its emulator on the rollback + epoch.** Both clients still alive at a box soak's end died at their + hangup with `double free or corruption (!prev)` in `Epoch.deinit`; a + local soak at 207 ms did it in 2.5 min, `free(): invalid pointer` in a + terminal's screen replica's teardown at a rejoin. A terminal closed and + brought back (Panes, `Look @p`) is loaded by `dump.loadPane`, which + named no `.terminal` allocator, so on a screen-state replica its + emulator was made with the core's, the epoch's, against `State.gpa`'s + rule. Its screen replica's blobs (`apply`, with `State.gpa`) were the + epoch's then, freed with the C heap's at the end behind its back, and + freed again by the epoch; a delta under a held copy made them born in + it, and a rollback freed them, and the emulator's pages, from under it. + - Fix: `terminal.emulatorGpa`, `create`'s rule, is `loadPane`'s + `.terminal` too. + - Test: replica.zig's closed terminal brought back on a screen-state + replica, its State not the epoch's (failed before). The local soak at + 207 ms, ReleaseSafe (before, on lppumkvm, both its clients aborted): + 2.5 min, both end 0; 4 min at seed 32, four clients, none aborts (its + one breach, a window's panes gone from its view, is not this one). diff --git a/src/dump.zig b/src/dump.zig index c64f83c4..5f5f5870 100644 --- a/src/dump.zig +++ b/src/dump.zig @@ -539,7 +539,7 @@ pub fn savePane(gpa: std.mem.Allocator, pane: *pardes.panes.Pane, slim: bool) ![ pub fn loadPane(p: *Pardes, id: usize, bytes: []const u8) !*pardes.panes.Pane { var l = try walk.Load(&.{pardes.panes.Pane}).init(p.gpa, bytes); defer l.deinit(); - l.allocs = .{ .image = p.image_gpa, .pdf = p.pdf_gpa }; + l.allocs = .{ .image = p.image_gpa, .pdf = p.pdf_gpa, .terminal = pardes.panes.terminal.emulatorGpa(p.gpa) }; const pane = try p.gpa.create(pardes.panes.Pane); errdefer { l.rollback(); diff --git a/src/replica.zig b/src/replica.zig index 439f71a2..da0c3e64 100644 --- a/src/replica.zig +++ b/src/replica.zig @@ -3266,6 +3266,31 @@ test "a delta applied under a speculation outlives its rollback: the terminal st for (rows_a[rows_a.len - shown ..], rows_r[rows_r.len - shown ..]) |x, y| try std.testing.expectEqualStrings(x, y); } +test "a closed terminal brought back on a screen-state replica: its emulator on the C heap, never the rollback epoch's" { + if (comptime !terminal.enabled) return error.SkipZigTest; + const gpa = std.testing.allocator; + const a = try Authority.init(gpa, .{ .cols = 80, .rows = 24, .boot = .tty }, TestHost.host(), true); + defer a.deinit(); + const r = try Replica.init(gpa, 1, try a.unit(0, 0), true); + defer r.deinit(); + const shell: u8 = for (a.core.panes, 0..) |slot, id| { + if (slot) |pane| if (pane.isTerminal()) break @intCast(id); + } else return error.NoShell; + a.core.active = shell; + r.confirm(try a.local(.{ .command = "Del" })); + try std.testing.expectEqual(@as(usize, 1), a.core.store.closed.items.len); + var line: [32]u8 = undefined; + r.confirm(try a.local(.{ .command = try std.fmt.bufPrint(&line, "Look @p{d}", .{a.core.store.closed.items[0].serial}) })); + try std.testing.expect(r.diverged == null); + const back = for (r.core().panes) |slot| { + if (slot) |pane| if (pane.terminal) |state| break state; + } else return error.NotBack; + // Its deltas write it while a copy is held, and its screen replica's + // blobs are freed with the C heap's (`deinit`): the epoch's, a rollback + // freed what they held and the exit freed them behind its back (H21). + try std.testing.expect(r.epoch.owns(@intFromPtr(back)) == null); +} + test "a view that sized a terminal closing leaves nothing queued: the others' replicas copy, type and agree" { if (comptime !terminal.enabled) return error.SkipZigTest; const gpa = std.testing.allocator; diff --git a/src/terminal.zig b/src/terminal.zig index d4238c83..86eec32d 100644 --- a/src/terminal.zig +++ b/src/terminal.zig @@ -607,6 +607,13 @@ pub fn takeMarks(state: *State, bytes: []const u8) void { /// A screen-state replica's emulators' allocator (`State.gpa`). pub const emulator_gpa = std.heap.c_allocator; +/// What a terminal's emulator is made with, by `create` or a pane's load +/// (dump.loadPane, a closed terminal brought back): `gpa`, but on a screen- +/// state replica `emulator_gpa`. A hosted build's: the web has no C heap. +pub fn emulatorGpa(gpa: std.mem.Allocator) std.mem.Allocator { + return if (comptime pardes.hosted) (if (pardes.tape.screenReplica()) emulator_gpa else gpa) else gpa; +} + /// A replica's speculation leaves an emulator be: it is the authority's /// (docs/next-daemon.md, phase 5), so the input waits for its order. fn authorityOnly(comptime why: []const u8) bool { @@ -936,8 +943,7 @@ pub fn create(gpa: std.mem.Allocator, cols: u16, rows: u16) !*Pane { const pane = try createDoc(gpa, cols, rows); errdefer gpa.destroy(pane); if (comptime !enabled) return pane; - // A screen-state replica is a hosted build's: the web has no C heap. - const own = if (comptime pardes.hosted) (if (pardes.tape.screenReplica()) emulator_gpa else gpa) else gpa; + const own = emulatorGpa(gpa); const state = try own.create(State); errdefer own.destroy(state); state.gpa = own; -- cgit v1.3 From d00c40694dc239854263616f710b4a197d25d145 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 21:00:30 -0300 Subject: views: a client that joins again has its own view back, layout, focus and the panes opened in it: an Upgrade's clients and one whose bye says it joins again (a divergence's, wire 19's `bye.again`) leave their view kept for them (`views_away`, digested and dumped, so an Upgrade's new build has it; taped `awayView`), the client names it in its next hello (`Hello.view`, handed on by the daemon with the connection) and its join takes it back (taped `rejoinView`) rather than a copy of the session's; the live view is never a kept one, `views.count` counts the others, and a join with every view taken lets one go (every Upgrade made each client's own panes leave its view: the hunt's 80 ms cell lost `+Search` and `+New`) Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 48 ++++++++++++++++++ src/daemon.zig | 14 +++-- src/detached/wire.zig | 29 ++++++++--- src/gui/gui.zig | 13 ++--- src/pardes.zig | 31 ++++++++++++ src/replica.zig | 138 ++++++++++++++++++++++++++++++++++++-------------- src/tape.zig | 4 ++ src/tty/tty.zig | 25 +++++---- src/views.zig | 83 +++++++++++++++++++++++++++--- test/daemon.py | 90 +++++++++++++++++++++++--------- 10 files changed, 378 insertions(+), 97 deletions(-) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 6dfa799c..c7be4b5a 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4375,3 +4375,51 @@ After the line, the hunt's H21 (a client's normal end aborted): 207 ms, ReleaseSafe (before, on lppumkvm, both its clients aborted): 2.5 min, both end 0; 4 min at seed 32, four clients, none aborts (its one breach, a window's panes gone from its view, is not this one). + +**A rejoin's view is its own again** (2026-10-30; `src/views.zig`'s +`away`, `Pardes.awayView`/`rejoinView`, wire 19). Every rejoin gave the +client a new view, a copy of the session's: a divergence's, and above all +an Upgrade's, which every client makes after each install (the hunt's +80 ms cell: the terminal client lost `+Search` and `+New` from its view; +likely the user's "some panes just close randomly right after I open +them"). `leaveView` covered only a session's last client, while no one +else watched it. +- **A view kept for its client.** An Upgrade's clients + (`Peers.refuseAll`) and one whose bye says it joins again (a + divergence's: `bye.again`) leave their view open as it is, kept for them + (`views_away`, a session field: digested, and dumped, so the new build + has it). The client names the view it had in its next hello + (`Hello.view`; the daemon hands it on with the connection, + `replica `), and its join takes that view back (`rejoinView`, settled + as `openView` is) rather than a new one. Both are taped inputs. Any other + end closes the view as before. +- **The live view is never one.** One kept going live sends the session + into another, or into a copy of its own. `views.count` counts the views + someone looks at, so the last of those stays, a tty's Detach is "one of + several" only beside another, and `closeKeeping` never writes into a kept + one. Drawing skips them, and an Upgrade's new build opens no window for + one and closes none for want of a terminal (`resumeTerminals`, + `resumeWindows`). +- **Bounded by room.** A kept view goes when a join finds every view taken + (`openView`, the lowest first): a client that never comes back (one of + the old build, told to upgrade too) leaves one. ponytail: no timer; until + then it costs a parked slot, and a terminal it sized last keeps that size + until another view types into it. ponytail: a divergence's redial that + reached the session before its old link's end takes a new view. +- **Tests.** views.zig's: two clients' views through an Upgrade's dump, + each back in its own, and a join with no room lets one go. replica.zig's: + a divergence's rejoin, its pane in its view again, and the other client + agrees. `test/daemon.py --upgrade`: two replica clients each open a pane + in their own view, and after the Upgrade's rejoin each one's `/layout` + holds its own pane and not the other's; and the plain run's client D, + put off now at its 100th unit, opens a pane in its own view first, and + after its divergence's rejoin its view holds it and its keys go into it. + main's binaries fail both (`('its own view again', 4, 5, ...)`, `('client + D in its own view again', 2, ...)`). +- **Not this one:** the H21 soak's "panes [...] left its view while still + open, with no rejoin since the last quiet point" (local 207 ms, seed 32, + a window client with one link all through) is no rejoin's. +- **Gates** (on zponxort): Debug `core-test` tty, `replicas`, `replay`, + `test/daemon.py` 20 cycles, `--upgrade` (against a `-Dstamp-commit` + build), `--window-clients` and `--gui`; ReleaseSafe `--upgrade` (from + the Debug build) and `--window-clients`: all pass. diff --git a/src/daemon.zig b/src/daemon.zig index 467af8b8..b93f5d43 100644 --- a/src/daemon.zig +++ b/src/daemon.zig @@ -379,10 +379,13 @@ pub const Note = union(enum) { /// stub's pipe; for a window session, both -1: open a window. attach: struct { tty: c_int, stub: c_int }, adopted: struct { pids: [max_fds]libc.pid_t, fds: [max_fds]c_int, n: usize }, - /// A replica's connection, its hello taken (src/replica.zig `Peers`). - replica: c_int, + /// A replica's connection, its hello taken (src/replica.zig `Peers`), + /// and the view it joins again for, if it does. + replica: Joining, }; +pub const Joining = struct { fd: c_int, view: ?u8 = null }; + /// Hands every note the daemon sends on to `deliver`, on a thread of its /// own, until the link ends (`quit`). Called each loop: started once. pub fn watch(ctx: *anyopaque, deliver: *const fn (ctx: *anyopaque, note: Note) void) void { @@ -409,7 +412,8 @@ fn readNotes(fd: c_int, ctx: *anyopaque, deliver: *const fn (ctx: *anyopaque, no } }); for (got.fds[@min(got.nfds, 2)..got.nfds]) |f| _ = libc.close(f); } else if (std.mem.eql(u8, verb, "replica")) { - if (got.nfds >= 1) deliver(ctx, .{ .replica = got.fds[0] }); + const view = std.fmt.parseInt(u8, words.next() orelse "", 10) catch null; + if (got.nfds >= 1) deliver(ctx, .{ .replica = .{ .fd = got.fds[0], .view = view } }); for (got.fds[@min(got.nfds, 1)..got.nfds]) |f| _ = libc.close(f); } else if (std.mem.eql(u8, verb, "adopted")) { var note: Note = .{ .adopted = .{ .pids = undefined, .fds = undefined, .n = 0 } }; @@ -951,7 +955,9 @@ const Daemon = struct { } else return refuse(r, std.fmt.bufPrint(&why, "no session called '{s}'", .{h.session}) catch "no such session"); const t = &d.conns[target]; if (t.semantics != h.semantics) return refuse(r, std.fmt.bufPrint(&why, "session '{s}' runs another build of pardes (semantics {x:0>16}, this client's {x:0>16}): a replica needs the same build at both ends, as `pardes --version` shows", .{ h.session, t.semantics, h.semantics }) catch "another build"); - if (!send(t.fd, "replica", &.{r.fd})) { + var words: [16]u8 = undefined; + const note = if (h.view) |k| std.fmt.bufPrint(&words, "replica {d}", .{k}) catch unreachable else "replica"; + if (!send(t.fd, note, &.{r.fd})) { d.hangUp(@intCast(target)); return refuse(r, "that session does not answer"); } diff --git a/src/detached/wire.zig b/src/detached/wire.zig index 6090dedb..337211b7 100644 --- a/src/detached/wire.zig +++ b/src/detached/wire.zig @@ -9,6 +9,8 @@ const builtin = @import("builtin"); const pardes = @import("../pardes.zig"); const config = @import("pardes_config"); +/// 19: a client joining again names the view it had in its hello, and a +/// bye says whether it joins again for it (views.away); /// 18: a replica's input is stamped with its guess of the session's /// clocks, the time its step reads (phase 6's instant interaction notes); /// 17: a pointer's hit carries the place it was made on (Layer.Place) and @@ -22,7 +24,7 @@ const config = @import("pardes_config"); /// part B); 14: the hello names a session and /// carries the semantics hash, and a ping is answered (phase 5's /// groundwork); 13: the frames went (phase 3). -pub const version: u16 = 18; +pub const version: u16 = 19; /// The terminal's build and the window's step alike (`semantics`). const family = switch (config.platform) { @@ -134,12 +136,15 @@ pub const Hello = struct { semantics: u64 = semantics, /// Which session (its 9P name, as `--attach=` names it). session: []const u8, + /// The view it had, joining again for it (a divergence's rejoin, an + /// Upgrade's): the daemon hands it on with the connection. 0xff: none. + view: ?u8 = null, }; /// A session's name on the wire, as the daemon keeps it. pub const max_name = 128; /// The longest hello there is, header and all. -pub const hello_max = header_len + 2 + 8 + 2 + max_name; +pub const hello_max = header_len + 2 + 8 + 2 + max_name + 1; /// Daemon or session -> client. pub const ServerTag = enum(u8) { @@ -268,7 +273,8 @@ pub fn helloVersion(payload: []const u8) Error!u16 { /// one call and no longer. pub const ClientMsg = union(enum) { hello: Hello, - bye, + /// The client's end: `again`, it joins again for its view at once. + bye: struct { again: bool = false }, /// A round trip's probe, answered `pong` with the same value. ping: u64, screen_ack: []const u8, @@ -498,8 +504,9 @@ pub fn encodeClient(out: []u8, msg: ClientMsg) Error![]const u8 { try w.writeInt(u64, h.semantics, .little); if (h.session.len > max_name) return error.Overlong; try putSlice16(&w, h.session); + try w.writeByte(h.view orelse 0xff); }, - .bye => {}, + .bye => |b| try w.writeByte(@intFromBool(b.again)), .ping => |v| try w.writeInt(u64, v, .little), .screen_ack, .want => |bytes| try w.writeAll(bytes), .check => {}, @@ -573,8 +580,12 @@ pub fn decodeClient(tag: u8, payload: []const u8) Error!ClientMsg { if (n > max_name) return error.BadValue; break :blk try r.take(n); }, + .view = switch (try r.takeByte()) { + 0xff => null, + else => |k| k, + }, } }, - .bye => .bye, + .bye => .{ .bye = .{ .again = try getBool(&r) } }, .ping => .{ .ping = try r.takeInt(u64, .little) }, .screen_ack => return .{ .screen_ack = payload }, .check => .check, @@ -693,7 +704,8 @@ test "detached wire: every Event variant round-trips" { try testing.expectEqual(semantics, got.hello.semantics); try testing.expectEqualStrings("work", got.hello.session); } - try testing.expectEqual(ClientMsg.bye, try roundClient(&buf, .bye)); + try testing.expect((try roundClient(&buf, .{ .bye = .{ .again = true } })).bye.again); + try testing.expectEqual(@as(?u8, 2), (try roundClient(&buf, .{ .hello = .{ .session = "work", .view = 2 } })).hello.view); try testing.expectEqual(@as(u64, 0xfeed_beef_0000_0001), (try roundClient(&buf, .{ .ping = 0xfeed_beef_0000_0001 })).ping); { // The other way: framed the same, its tag the ServerTag's number. @@ -920,8 +932,9 @@ test "detached wire: values a field cannot mean are refused" { std.mem.writeInt(u16, hello[10..12], max_name + 1, .little); try testing.expectError(error.BadValue, decodeClient(@backingInt(ClientTag.hello), &hello)); std.mem.writeInt(u16, hello[10..12], max_name, .little); - try testing.expectEqual(@as(usize, max_name), (try decodeClient(@backingInt(ClientTag.hello), hello[0 .. hello.len - 1])).hello.session.len); - try testing.expect(header_len + hello.len - 1 == hello_max); + hello[hello.len - 1] = 0xff; + try testing.expectEqual(@as(usize, max_name), (try decodeClient(@backingInt(ClientTag.hello), &hello)).hello.session.len); + try testing.expect(header_len + hello.len == hello_max); } // A NaN scroll distance. `pinch` multiplies into a zoom the pane keeps. { diff --git a/src/gui/gui.zig b/src/gui/gui.zig index dd21680f..d3acf454 100644 --- a/src/gui/gui.zig +++ b/src/gui/gui.zig @@ -2752,7 +2752,7 @@ fn remoteSession(init: std.process.Init, g: *Gui, opts_in: pardes.Options, test_ defer have_store.deinit(); const have: ?*replica.Have = if (far) &have_store else null; var dial_t0 = std.Io.Clock.awake.now(io).nanoseconds; - var joined = replica.dial(gpa, to.path, to.session, &why, have) orelse Said.no(io, "{s}", .{std.mem.sliceTo(&why, 0)}); + var joined = replica.dial(gpa, to.path, to.session, &why, have, null) orelse Said.no(io, "{s}", .{std.mem.sliceTo(&why, 0)}); if (joined.semantics != wire.semantics) Said.no(io, "{s}", .{replica.Link.upgrade_words}); const allocs = pardes.memory.init(gpa); @@ -2878,7 +2878,7 @@ fn remoteSession(init: std.process.Init, g: *Gui, opts_in: pardes.Options, test_ if (r.diverged != null) link.quit = true; } if (init.environ_map.get("PARDES_REPLICA_STATS")) |file| link.writeStats(file); - if (!link.gone) replica.bye(joined.fd); + if (!link.gone) replica.bye(joined.fd, r.diverged != null); // The reader ends with the link: what it took meanwhile is let go. _ = libc.shutdown(joined.fd, 2); reader.join(); @@ -2901,7 +2901,7 @@ fn remoteSession(init: std.process.Init, g: *Gui, opts_in: pardes.Options, test_ // The session again, the core it leaves held meanwhile. if (have) |h| h.leaving(r.core()); dial_t0 = std.Io.Clock.awake.now(io).nanoseconds; - joined = replica.Link.redial(gpa, to.path, to.session, &why, have, r.diverged == null) orelse Said.no(io, "could not join again: {s}", .{std.mem.sliceTo(&why, 0)}); + joined = replica.Link.redial(gpa, to.path, to.session, &why, have, r.diverged == null, joined.view) orelse Said.no(io, "could not join again: {s}", .{std.mem.sliceTo(&why, 0)}); if (have) |h| h.left(); r.deinit(); r = replica.Link.load(gpa, &joined, o, &why, have) orelse Said.no(io, "{s}", .{std.mem.sliceTo(&why, 0)}); @@ -4553,13 +4553,14 @@ const Shell = struct { } /// An upgraded session's views, a window each again: the Gui's is the - /// live view's, the others open as `--attach` opens one. + /// live view's, the others open as `--attach` opens one; a replica + /// client's, kept for its rejoin (views.away), none. fn resumeWindows(s: *Shell) void { const live = s.core.view; s.gview = live; for (0..pardes.views.max) |i| { const k: u8 = @intCast(i); - if (k != live and pardes.views.isOpen(s.core, k)) s.openAnotherWindow(k); + if (k != live and pardes.views.isOpen(s.core, k) and !pardes.views.isAway(s.core, k)) s.openAnotherWindow(k); s.windowed |= @as(u8, 1) << @intCast(live); } s.core.toView(live); @@ -5362,7 +5363,7 @@ fn deliverNote(ctx: *anyopaque, note: daemon.Note) void { q.push(.{ .adopted = .{ .pid = 0, .fd = -1 } }); }, // Joined at the next step boundary (Peers, any thread). - .replica => |fd| s.peers.handed(fd), + .replica => |j| s.peers.handed(j.fd, j.view), } } diff --git a/src/pardes.zig b/src/pardes.zig index 22f446a5..edab4a58 100644 --- a/src/pardes.zig +++ b/src/pardes.zig @@ -4732,6 +4732,8 @@ pub const Pardes = struct { /// there are, one a window or terminal; each not live is in `parked`. view: u8 = 0, views_open: u8 = 1, + /// The open views kept for their clients' rejoin (views.away). + views_away: u8 = 0, parked: [views.max]?*views.Parked = @splat(null), /// The views no frontend here draws: a replica client's, which its own /// process draws (src/replica.zig). A pump skips them. @@ -5706,6 +5708,14 @@ pub const Pardes = struct { pub fn openView(p: *Pardes) ?u8 { const taped = tape.enter(p, .open_view); defer tape.leave(p, taped); + // Every view taken: one kept for a client that has not come back + // for it (views.away) goes first. ponytail: no timer lets one go; + // until its place is wanted it costs a parked slot, and a terminal + // it sized last keeps that size until another view types into it. + if (@popCount(p.views_open) == views.max and p.views_away != 0) { + views.close(p, @ctz(p.views_away)); + p.sync(); + } // What was queued since the last pump (a leaving client's view // closed, a 9P request's step) done first, as every client of the // tape does it here: a join dumps the session next, and a core @@ -5740,6 +5750,27 @@ pub const Pardes = struct { p.settle(); } + /// The client of view `k` gone, to join again for it (an Upgrade's + /// rejoin, a divergence's): its view kept as it is (views.away). + pub fn awayView(p: *Pardes, k: u8) void { + const taped = tape.enter(p, .{ .away_view = k }); + defer tape.leave(p, taped); + views.away(p, k); + } + + /// A client back for view `k`, kept for it (`awayView`): its own again, + /// as it left it; false when it was let go meanwhile. Settled as + /// `openView` is, for the join's dump next. + pub fn rejoinView(p: *Pardes, k: u8) bool { + const taped = tape.enter(p, .{ .rejoin_view = k }); + defer tape.leave(p, taped); + if (!views.isAway(p, k)) return false; + p.views_away &= ~(@as(u8, 1) << @intCast(k)); + p.settle(); + p.needs_frame = true; + return true; + } + /// Into view `k`, before the input its client sent. pub fn toView(p: *Pardes, k: u8) void { if (k == p.view) return; diff --git a/src/replica.zig b/src/replica.zig index da0c3e64..4f84712f 100644 --- a/src/replica.zig +++ b/src/replica.zig @@ -1414,6 +1414,9 @@ pub const Peer = struct { fd: c_int, id: u16, view: u8, + /// Its bye said it joins again for its view (a divergence's rejoin): + /// kept for it (views.away), not closed. Its reader's, read after its end. + again: bool = false, /// Its number in the session's `Screens`. sid: u16 = 0, /// Inputs read (each one's seq, as the client numbered them). @@ -1626,7 +1629,11 @@ pub const Peer = struct { fn take(peer: *Peer, tag: u8, payload: []const u8) bool { const t = std.enums.fromInt(wire.ClientTag, tag) orelse return false; switch (t) { - .hello, .bye => return false, + .hello => return false, + .bye => { + peer.again = (wire.decodeClient(tag, payload) catch return false).bye.again; + return false; + }, .ping => { var out: [16]u8 = undefined; const v = (wire.decodeClient(tag, payload) catch return false).ping; @@ -1747,12 +1754,14 @@ pub const Peers = struct { msgs: std.ArrayListUnmanaged([]u8) = .empty, /// Clients waiting to join while the session's texts are hashed /// (`joins`), the texts and the thread hashing them. - held: std.ArrayListUnmanaged(c_int) = .empty, + held: std.ArrayListUnmanaged(Joining) = .empty, texts: std.ArrayListUnmanaged([]const u8) = .empty, hasher: ?std.Thread = null, rounds: u8 = 0, - pub const Note = union(enum) { join: c_int, input: Msg, screen: Msg, gone: *Peer, check: *Peer, joined: *Peer, hashed }; + pub const Note = union(enum) { join: Joining, input: Msg, screen: Msg, gone: *Peer, check: *Peer, joined: *Peer, hashed }; + /// A connection to join, and the view it joins again for, if it does. + pub const Joining = struct { fd: c_int, view: ?u8 = null }; pub const Msg = struct { peer: *Peer, msg: []u8 }; pub fn count(ps: *const Peers) usize { @@ -1766,7 +1775,7 @@ pub const Peers = struct { ps.inbox.append(ps.gpa, note) catch { ps.lock.unlock(ps.io); switch (note) { - .join => |fd| _ = libc.close(fd), + .join => |j| _ = libc.close(j.fd), .input, .screen => |in| ps.gpa.free(in.msg), .gone, .check, .joined, .hashed => {}, } @@ -1778,9 +1787,10 @@ pub const Peers = struct { } /// A replica's connection the daemon handed over (its hello taken - /// there): joined at the next step boundary. Any thread. - pub fn handed(ps: *Peers, fd: c_int) void { - ps.post(.{ .join = fd }); + /// there), and the view its hello says it had: joined at the next step + /// boundary. Any thread. + pub fn handed(ps: *Peers, fd: c_int, view: ?u8) void { + ps.post(.{ .join = .{ .fd = fd, .view = view } }); } /// What was recorded since the last unit, to every client as one, its @@ -1824,9 +1834,9 @@ pub const Peers = struct { ps.mail.store(false, .release); ps.lock.unlock(ps.io); for (ps.taken.items) |note| switch (note) { - .join => |fd| { - ps.held.append(ps.gpa, fd) catch { - refuse(fd, "out of memory"); + .join => |j| { + ps.held.append(ps.gpa, j) catch { + refuse(j.fd, "out of memory"); continue; }; ps.joins(core); @@ -1922,7 +1932,7 @@ pub const Peers = struct { } } ps.rounds = 0; - for (ps.held.items) |fd| ps.join(core, fd); + for (ps.held.items) |j| ps.join(core, j); ps.held.clearRetainingCapacity(); } @@ -1939,7 +1949,8 @@ pub const Peers = struct { ps.texts.clearRetainingCapacity(); } - fn join(ps: *Peers, core: *Pardes, fd: c_int) void { + fn join(ps: *Peers, core: *Pardes, joining: Joining) void { + const fd = joining.fd; const t0 = nowNs(); defer { ps.join_ns = @max(ps.join_ns, nowNs() - t0); @@ -1964,7 +1975,13 @@ pub const Peers = struct { } // Someone looks at it now: its messages go by keys, not the clock. core.unattended = false; - const k = core.openView() orelse { + // Its own view again, kept for it since it left (views.away), or one + // new. ponytail: a divergence's redial that came before its old + // link's end (posted by that link's reader, microseconds after the + // bye; the redial is a dial and the daemon's hand-over later) finds + // its view not kept yet, and takes a new one. + const back = if (joining.view) |v| core.rejoinView(v) else false; + const k = (if (back) joining.view else core.openView()) orelse { core.unattended = ps.count() == 0 and ps.alone_unattended; if (ps.count() == 0) ps.stop(core); return refuse(fd, "that session has as many views as it takes"); @@ -2035,7 +2052,7 @@ pub const Peers = struct { log.info("client {d} left '{s}'", .{ peer.id, ps.name }); ps.screens.dropPeer(peer.sid); core.undrawn &= ~(@as(u8, 1) << @intCast(peer.view)); - ps.closeViewOf(core, peer); + ps.closeViewOf(core, peer, peer.again); if (ps.count() == 0) core.unattended = ps.alone_unattended; ps.seq += 1; ps.unit(0, ps.seq); @@ -2043,11 +2060,12 @@ pub const Peers = struct { if (ps.count() == 0) ps.stop(core); } - /// A gone client's view closed (its slot let go already): the last one - /// of a session no one else watches leaves the session its layout, for - /// its rejoin (an Upgrade's, a diverged one's) to find. - fn closeViewOf(ps: *Peers, core: *Pardes, peer: *Peer) void { - if (ps.count() == 0 and ps.alone_unattended) core.leaveView(peer.view) else core.closeView(peer.view); + /// A gone client's view closed (its slot let go already), or kept for + /// it (`again`: it joins again for it, views.away): the last one of a + /// session no one else watches leaves the session its layout, for a + /// client that comes after to find. + fn closeViewOf(ps: *Peers, core: *Pardes, peer: *Peer, again: bool) void { + if (again) core.awayView(peer.view) else if (ps.count() == 0 and ps.alone_unattended) core.leaveView(peer.view) else core.closeView(peer.view); } /// Its threads ended and joined, its link closed. @@ -2114,7 +2132,8 @@ pub const Peers = struct { const ts: libc.timespec = .{ .sec = 0, .nsec = 2 * std.time.ns_per_ms }; _ = libc.nanosleep(&ts, null); } - ps.dropAll(core); + // Each joins again for its view, kept in the dump (views.away). + ps.drop(core, true); } /// A join's dump and pins let go: written, or its client gone. @@ -2149,21 +2168,26 @@ pub const Peers = struct { /// Every client let go (the session's end, a Restore): each sees its /// link end. pub fn dropAll(ps: *Peers, core: *Pardes) void { + ps.drop(core, false); + } + + /// ...each view kept for its client if `again` (an Upgrade's). + fn drop(ps: *Peers, core: *Pardes, again: bool) void { ps.unhashedDone(); - for (ps.held.items) |fd| _ = libc.close(fd); + for (ps.held.items) |j| _ = libc.close(j.fd); ps.held.clearRetainingCapacity(); ps.rounds = 0; for (&ps.list) |*slot| if (slot.*) |peer| { slot.* = null; core.undrawn &= ~(@as(u8, 1) << @intCast(peer.view)); - ps.closeViewOf(core, peer); + ps.closeViewOf(core, peer, again); ps.screens.dropPeer(peer.sid); ps.letGo(peer); }; ps.stop(core); ps.lock.lockUncancelable(ps.io); for (ps.inbox.items) |note| switch (note) { - .join => |fd| _ = libc.close(fd), + .join => |j| _ = libc.close(j.fd), .input, .screen => |in| ps.gpa.free(in.msg), .gone, .check, .joined, .hashed => {}, }; @@ -2238,8 +2262,9 @@ pub const Got = struct { pos: usize, len: usize, hash: [32]u8 }; /// Connected to the replica socket at `path`, its hello said for /// `session`, welcomed, and the dump read whole: what a client joins with. /// A far client (`have`) is sent its dump's blobs by hash first, and asks -/// for those it holds nowhere. Errors are said in `why`. -pub fn dial(gpa: Allocator, path: []const u8, session: []const u8, why: []u8, have: ?*Have) ?Joined { +/// for those it holds nowhere. Errors are said in `why`. `view`: the one +/// it had, joining again for it. +pub fn dial(gpa: Allocator, path: []const u8, session: []const u8, why: []u8, have: ?*Have, view: ?u8) ?Joined { const fd = libc.socket(libc.AF.UNIX, libc.SOCK.STREAM | libc.SOCK.CLOEXEC, 0); if (fd < 0) return said(why, "no socket", .{}); var addr: libc.sockaddr.un = .{ .path = @splat(0) }; @@ -2252,14 +2277,14 @@ pub fn dial(gpa: Allocator, path: []const u8, session: []const u8, why: []u8, ha _ = libc.close(fd); return said(why, "nothing answers at {s}", .{path}); } - return joinOn(gpa, fd, session, why, have); + return joinOn(gpa, fd, session, why, have, view); } /// `dial`'s, on a connection made: the hello, then the join. The /// connection is closed if the join fails. -fn joinOn(gpa: Allocator, fd: c_int, session: []const u8, why: []u8, have: ?*Have) ?Joined { +fn joinOn(gpa: Allocator, fd: c_int, session: []const u8, why: []u8, have: ?*Have, view: ?u8) ?Joined { var out: [wire.hello_max + 16]u8 = undefined; - const hello = wire.encodeClient(&out, .{ .hello = .{ .session = session } }) catch { + const hello = wire.encodeClient(&out, .{ .hello = .{ .session = session, .view = view } }) catch { _ = libc.close(fd); return said(why, "that session's name is too long", .{}); }; @@ -2690,10 +2715,10 @@ pub fn askCheck(fd: c_int) bool { return writeAll(fd, wire.encodeClient(&out, .check) catch unreachable); } -/// The client's last word. -pub fn bye(fd: c_int) void { +/// The client's last word: `again`, it joins again for its view at once. +pub fn bye(fd: c_int, again: bool) void { var out: [16]u8 = undefined; - if (wire.encodeClient(&out, .bye)) |b| _ = writeAll(fd, b) else |_| {} + if (wire.encodeClient(&out, .{ .bye = .{ .again = again } })) |b| _ = writeAll(fd, b) else |_| {} } /// What follows the dump on the client's connection, each message handed @@ -2985,13 +3010,14 @@ pub const Link = struct { /// `patient`: after an Upgrade's `refused` alone, while the session is /// away a moment; after a divergence a session that will not answer is /// gone, and the client says so at once (it froze 20 s, 2026-10-30). - pub fn redial(gpa: Allocator, path: []const u8, session: []const u8, why: *[600]u8, have: ?*Have, patient: bool) ?Joined { + /// `view`: the one it had, which the session kept for it (views.away). + pub fn redial(gpa: Allocator, path: []const u8, session: []const u8, why: *[600]u8, have: ?*Have, patient: bool, view: u8) ?Joined { var tries: u32 = 0; while (true) : (tries += 1) { why.* = @splat(0); - if (dial(gpa, path, session, why, have)) |j| { + if (dial(gpa, path, session, why, have, view)) |j| { if (j.semantics == wire.semantics) return j; - bye(j.fd); + bye(j.fd, false); _ = libc.close(j.fd); var jj = j; jj.rest.deinit(gpa); @@ -3013,7 +3039,7 @@ pub const Link = struct { test "a client joining again after a divergence is told at once that its session is gone" { var why: [600]u8 = undefined; const t0 = nowNs(); - try std.testing.expect(Link.redial(std.testing.allocator, "/nonexistent/pardes-replica.sock", "gone", &why, null, false) == null); + try std.testing.expect(Link.redial(std.testing.allocator, "/nonexistent/pardes-replica.sock", "gone", &why, null, false, 1) == null); try std.testing.expect(nowNs() - t0 < 100 * std.time.ns_per_ms); } @@ -3777,7 +3803,7 @@ fn farJoinOnce(ps: *Peers, core: *Pardes, have: ?*Have) !Joined { peer.welcome_len = (wire.encodeServer(&peer.welcome, .{ .welcome = .{ .pid = 1, .session = "t", .dump_len = whole, .blobs = @intCast(blobs.items.len) } }) catch unreachable).len; const Serve = struct { fn run(p: *Peer) void { - var hello: [wire.header_len + 13]u8 = undefined; + var hello: [wire.header_len + 14]u8 = undefined; var got: usize = 0; while (got < hello.len) { const n = libc.read(p.fd, hello[got..].ptr, hello.len - got); @@ -3791,7 +3817,7 @@ fn farJoinOnce(ps: *Peers, core: *Pardes, have: ?*Have) !Joined { peer.reader = try std.Thread.spawn(.{}, Serve.run, .{peer}); defer ps.letGo(peer); var why: [300]u8 = @splat(0); - return joinOn(std.testing.allocator, fds[1], "t", &why, have) orelse { + return joinOn(std.testing.allocator, fds[1], "t", &why, have, null) orelse { std.debug.print("the join failed: {s}\n", .{std.mem.sliceTo(&why, 0)}); return error.JoinFailed; }; @@ -4059,6 +4085,44 @@ test "a client leaves, and one joins: what the leave queued (a terminal's resize try agrees(a, joiner); } +test "a client that diverged joins again for its view: kept for it meanwhile, its pane in it again; the other client agrees" { + const gpa = std.testing.allocator; + const a = try Authority.init(gpa, .{ .cols = 80, .rows = 24, .file = "mise.toml" }, TestHost.host(), false); + defer a.deinit(); + const first = try gpa.dupe(u8, try a.unit(0, 0)); + defer gpa.free(first); + const old = try Replica.init(gpa, 1, first, false); + defer old.deinit(); + const opened = try a.openView(2); + old.confirm(opened.unit); + var buf: [256]u8 = undefined; + old.confirm(try a.take(try inputOf(&buf, 2, 1, .{ .command = "New" }))); + const made = a.core.active; + // Its bye says it joins again: its view is kept, as every replica keeps it. + var was = Stepping.begin(&a.t, null); + a.core.awayView(opened.view); + was.end(); + old.confirm(try a.unit(0, 100)); + try agrees(a, old); + // Back for it: its own, its pane in it, in the dump it loads. + was = Stepping.begin(&a.t, null); + try std.testing.expect(a.core.rejoinView(opened.view)); + was.end(); + old.confirm(try a.unit(0, 101)); + const back = try joinedNow(a, 3, opened.view); + defer back.deinit(); + try std.testing.expectEqual(digestOf(a), back.digest()); + a.view_of[3] = opened.view; + const u = try gpa.dupe(u8, try a.take(try back.input(.{ .key = .{ .cp = 'l', .text = "l" } }))); + defer gpa.free(u); + old.confirm(u); + back.confirm(u); + try std.testing.expectEqual(made, a.core.active); + try std.testing.expect(pardes.layout.findPane(a.core, made) != null); + try agrees(a, old); + try agrees(a, back); +} + /// An input of a replica's in the session's order, as `Peers.input` takes /// it: the terminals flushed to every peer, the step, the rows it read that /// each peer holds as holes (`fill`), then its unit to every replica. diff --git a/src/tape.zig b/src/tape.zig index 4759d598..74576eed 100644 --- a/src/tape.zig +++ b/src/tape.zig @@ -67,6 +67,8 @@ pub const Input = union(enum) { open_view, close_view: u8, leave_view: u8, + away_view: u8, + rejoin_view: u8, to_view: u8, /// What a window says of its font and its opacity (gui.zig). take_font_request, @@ -630,6 +632,8 @@ pub const Tape = struct { .open_view => _ = p.openView(), .close_view => |k| p.closeView(k), .leave_view => |k| p.leaveView(k), + .away_view => |k| p.awayView(k), + .rejoin_view => |k| _ = p.rejoinView(k), .to_view => |k| p.toView(k), .take_font_request => _ = p.takeFontRequest(), .observe_font => |f| _ = p.observeFont(f.name, f.size, f.unit), diff --git a/src/tty/tty.zig b/src/tty/tty.zig index 430c020d..4c054021 100644 --- a/src/tty/tty.zig +++ b/src/tty/tty.zig @@ -153,7 +153,7 @@ pub const Command = struct { files_changed, fs_ready, /// A replica's connection, handed over by the daemon (Peers). - replica: posix.fd_t, + replica: daemon.Joining, /// A replica's reader posted: the step boundary takes it in. replica_mail, /// A replica client's: what the session streamed (replica.Link). @@ -1333,7 +1333,7 @@ pub fn remote(init: std.process.Init, opts: pardes.Options, path: []const u8, se defer have_store.deinit(); const have: ?*replica.Have = if (far) &have_store else null; var dial_t0 = std.Io.Clock.awake.now(io).nanoseconds; - var joined = replica.dial(gpa, path, session, &why, have) orelse Said.no(io, "{s}", .{std.mem.sliceTo(&why, 0)}); + var joined = replica.dial(gpa, path, session, &why, have, null) orelse Said.no(io, "{s}", .{std.mem.sliceTo(&why, 0)}); if (joined.semantics != wire.semantics) Said.no(io, "{s}", .{replica.Link.upgrade_words}); const allocs = pardes.memory.init(gpa); @@ -1445,7 +1445,7 @@ pub fn remote(init: std.process.Init, opts: pardes.Options, path: []const u8, se if (r.diverged != null) link.quit = true; } if (init.environ_map.get("PARDES_REPLICA_STATS")) |file| link.writeStats(file); - if (!link.gone) replica.bye(joined.fd); + if (!link.gone) replica.bye(joined.fd, r.diverged != null); // The link's reader ends with the link, posting as it goes: what it // and the terminal posted is let go until it has. Its end is read // before the drain, not after: its last post (`link_gone`) comes @@ -1482,7 +1482,7 @@ pub fn remote(init: std.process.Init, opts: pardes.Options, path: []const u8, se // are not sent again. if (have) |h| h.leaving(r.core()); dial_t0 = std.Io.Clock.awake.now(io).nanoseconds; - joined = replica.Link.redial(gpa, path, session, &why, have, r.diverged == null) orelse { + joined = replica.Link.redial(gpa, path, session, &why, have, r.diverged == null, joined.view) orelse { screen.leave(); Said.no(io, "could not join again: {s}", .{std.mem.sliceTo(&why, 0)}); }; @@ -1916,8 +1916,8 @@ fn deliverNote(ctx: *anyopaque, note: daemon.Note) void { }; loop.postEvent(.{ .adopted = .{ .pid = 0, .fd = -1 } }) catch {}; }, - .replica => |fd| loop.postEvent(.{ .replica = fd }) catch { - _ = libc.close(fd); + .replica => |j| loop.postEvent(.{ .replica = j }) catch { + _ = libc.close(j.fd); }, } } @@ -2338,7 +2338,7 @@ const Shell = struct { /// An upgraded session's terminals, each its view's again (`primary`, /// run's, entered already); in a fresh session each but the primary a /// view of its own, as `--attach` makes one. A view left with none - /// closes. + /// closes, but a replica client's, kept for its rejoin (views.away). fn resumeTerminals(s: *Shell, rec: *const upgrade.Record, primary: *Screen) void { if (comptime builtin.is_test) unreachable; const core = s.core; @@ -2348,8 +2348,11 @@ const Shell = struct { } s.terms = @splat(null); if (rec.nttys == 0) { - pardes.views.only(core); + // The live view alone (never one kept for a rejoin) is the + // session's own. + for (0..pardes.views.max) |k| if (k != core.view and pardes.views.isOpen(core, @intCast(k)) and !pardes.views.isAway(core, @intCast(k))) core.closeView(@intCast(k)); s.terms[core.view] = primary; + primary.feed.view = core.view; return; } for (rec.ttys[0..rec.nttys]) |t| { @@ -2386,7 +2389,7 @@ const Shell = struct { sc.feed.view = t.view; s.terms[t.view] = sc; } - for (0..pardes.views.max) |k| if (pardes.views.isOpen(core, @intCast(k)) and s.terms[k] == null) core.closeView(@intCast(k)); + for (0..pardes.views.max) |k| if (pardes.views.isOpen(core, @intCast(k)) and s.terms[k] == null and !pardes.views.isAway(core, @intCast(k))) core.closeView(@intCast(k)); if (s.terms[core.view] == null) s.terms[core.view] = primary; } @@ -2623,8 +2626,8 @@ const Shell = struct { }, .attach => |a| s.attachTerminal(a.tty, a.stub, null), .adopted => |a| s.adoptShell(a.pid, a.fd), - .replica => |fd| { - s.peers.handed(fd); + .replica => |j| { + s.peers.handed(j.fd, j.view); s.input_seen = true; }, .replica_mail => s.input_seen = true, diff --git a/src/views.zig b/src/views.zig index ac274621..58d7a1f7 100644 --- a/src/views.zig +++ b/src/views.zig @@ -173,6 +173,7 @@ pub fn undumped(p: *Pardes, others: []const Dumped) !void { }; // Nothing fails from here. p.views_open &= take | bit(p.view); + p.views_away &= p.views_open; // The live view's slot owning nothing, as `installed` makes it. for (p.panes) |s| if (s) |pane| { if (take != 0) pane.pv[p.view] = fresh(pane); @@ -214,6 +215,7 @@ pub fn undumped(p: *Pardes, others: []const Dumped) !void { /// The live view alone, as view 0: every other closed. A session loaded /// for one client (a Restore, `-l`) is one view. pub fn only(p: *Pardes) void { + p.views_away = 0; for (0..max) |k| if (k != p.view) close(p, @intCast(k)); emptied(p, p.view); const k = p.view; @@ -255,8 +257,20 @@ pub fn isOpen(p: *const Pardes, k: u8) bool { return k < max and p.views_open & (@as(u8, 1) << @intCast(k)) != 0; } +/// Kept for its client's rejoin (`away`). +pub fn isAway(p: *const Pardes, k: u8) bool { + return k < max and p.views_away & (@as(u8, 1) << @intCast(k)) != 0; +} + +/// The views someone looks at: open, none kept for a rejoin. pub fn count(p: *const Pardes) usize { - return @popCount(p.views_open); + return @popCount(p.views_open & ~p.views_away); +} + +/// An open view but `k` someone looks at (`count`), if there is one. +fn looked(p: *const Pardes, k: u8) ?u8 { + for (0..max) |i| if (i != k and isOpen(p, @intCast(i)) and !isAway(p, @intCast(i))) return @intCast(i); + return null; } /// More than one view: the one test the single client's path pays. @@ -448,9 +462,8 @@ const sized = .{ "screen_w", "screen_h", "cell_pixels", "row_metrics" }; pub fn closeKeeping(p: *Pardes, k: u8) void { if (!isOpen(p, k) or count(p) == 1) return; to(p, k); - const other: u8 = for (0..max) |i| { - if (i != k and isOpen(p, @intCast(i))) break @intCast(i); - } else unreachable; + // Never into one kept for a rejoin (`away`): that one is its client's. + const other = looked(p, k).?; const slot = p.parked[other].?; // Its size too: the terminals it sized keep theirs, no shell redrawn for // a size it is given back at the rejoin. @@ -469,11 +482,12 @@ pub fn closeKeeping(p: *Pardes, k: u8) void { } /// View `k` gone, and all it held; the live one moves to another first. -/// The last view stays: a session without a client keeps its own. +/// The last view someone looks at stays: a session without a client keeps +/// its own. One kept for a rejoin (`away`) goes whenever it is closed. pub fn close(p: *Pardes, k: u8) void { - if (!isOpen(p, k) or count(p) == 1) return; + if (!isOpen(p, k) or @popCount(p.views_open) == 1 or (!isAway(p, k) and count(p) == 1)) return; if (k == p.view) { - const other: u8 = for (0..max) |i| { + const other: u8 = looked(p, k) orelse for (0..max) |i| { if (i != k and isOpen(p, @intCast(i))) break @intCast(i); } else unreachable; park(p); @@ -493,6 +507,19 @@ pub fn close(p: *Pardes, k: u8) void { if (pane.prompt == .name and pane.prompt_view == k) pane.closePrompt(); }; p.views_open &= ~(@as(u8, 1) << @intCast(k)); + p.views_away &= ~(@as(u8, 1) << @intCast(k)); +} + +/// View `k`'s client gone a moment, to join again for it (an Upgrade's +/// rejoin, a divergence's): the view kept as it is, layout, focus and the +/// panes opened in it, until its client is back (Pardes.rejoinView) or a +/// join wants its place (Pardes.openView). Dumped with the rest, so an +/// Upgrade's new build has it too. The live view is never one: the session +/// goes into another someone looks at, or into a copy of this one of its own. +pub fn away(p: *Pardes, k: u8) void { + if (!isOpen(p, k)) return; + if (k == p.view) if (looked(p, k) orelse open(p)) |other| to(p, other); + p.views_away |= bit(k); } /// The session's end: what each view not live holds. @@ -541,7 +568,7 @@ pub fn drawOthers(p: *Pardes, h: pardes.Host) !void { } for (0..max) |i| { const k: u8 = @intCast(i); - if (k == live or !isOpen(p, k) or p.undrawn & (@as(u8, 1) << @intCast(k)) != 0) continue; + if (k == live or !isOpen(p, k) or (p.undrawn | p.views_away) & bit(k) != 0) continue; // No post_present: what a shell acknowledges there moves the // session's pointer and hover, which are the live view's to move; // a view's panels are acknowledged when it is live and drawn. @@ -836,6 +863,46 @@ test "the last client's view going leaves the session its layout, which a rejoin try testing.expectEqual(@as(i32, 60), pane.body.cur_row); } +test "a client's view kept for its rejoin: through an Upgrade's dump, each client back in its own; a join with no room lets one go" { + const gpa = testing.allocator; + const opts: pardes.Options = .{ .tty_only = true, .cols = 100, .rows = 30 }; + const p = try Pardes.init(gpa, opts); + defer p.deinit(); + _ = try p.setTestFile("one\ntwo\n"); + // Two clients, each a pane of its own in its own view. + const ks: [2]u8 = .{ p.openView().?, p.openView().? }; + var made: [2]usize = undefined; + for (ks, &made) |k, *m| { + p.toView(k); + p.update(.{ .command = "New" }); + m.* = p.active; + } + // Both let go to upgrade, the live view one of theirs: theirs are kept, + // and the session's own is live. + for (ks) |k| p.awayView(k); + try testing.expectEqual(@as(usize, 1), count(p)); + try testing.expect(!isAway(p, p.view)); + // The new build loads the dump; each client is back in its own view. + const bytes = try pardes.dump.dumpBytes(p); + defer gpa.free(bytes); + const q = try pardes.dump.load(gpa, opts, bytes, null); + defer q.deinit(); + for (ks, made, 0..) |k, m, i| { + try testing.expect(isAway(q, k)); + try testing.expect(q.rejoinView(k)); + q.toView(k); + try testing.expectEqual(m, q.active); + try testing.expect(layout.findPane(q, m) != null); + try testing.expect(layout.findPane(q, made[1 - i]) == null); + } + try testing.expect(!q.rejoinView(ks[0])); + // A client that never comes back: its view goes when a join has no room. + p.toView(0); + try testing.expectEqual(@as(?u8, 3), p.openView()); + try testing.expectEqual(ks[0], p.openView()); + try testing.expect(!isAway(p, ks[0]) and isAway(p, ks[1])); +} + test "drawing the other views leaves the live one as it was, to the digest" { const gpa = testing.allocator; const p = try Pardes.init(gpa, .{ .tty_only = true, .cols = 80, .rows = 24 }); diff --git a/test/daemon.py b/test/daemon.py index 8633238f..bf575fd7 100644 --- a/test/daemon.py +++ b/test/daemon.py @@ -208,7 +208,7 @@ class Replica: """A replica's connection (src/detached/wire.zig): length-prefixed messages, a tag byte and a little-endian u32 length. It is a near client: its first word after the hello says so.""" - WIRE = 18 + WIRE = 19 def __init__(self, path, timeout=10): self.s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) @@ -235,7 +235,7 @@ class Replica: """('welcome', pid, name) or ('refused', why); a welcome's dump is read, whole.""" name = session.encode() - self.send(0x01, struct.pack(' after +def made_live(term, check): + """A key in a replica client's terminal (`gg`), stepped by its session + (a check after it agreed): its view is the live one, the one /layout + reads and /exec runs in, until the next input. A client asks one check + at a time, and checks come on (each answer wakes its session): the + first after the key may have been asked before it, the second was not.""" + seen = len(checks(check)) + os.write(term.master, b'gg') + wait_for('the client\'s key stepped', lambda: agreed(check, seen + 1), timeout=20) + + +def view_panes(client): + """The panes the live view lays out (/layout's pane serials).""" + return {int(x) for line in client.read('/layout').decode().splitlines() + if line.split()[:1] != ['active'] for x in line.split()[6:]} + + def clients(binary): """Replica clients of a detached session: one on the replica socket, one through the delay relay; each one's keys in the session's text and @@ -388,23 +405,31 @@ def clients(binary): wait_for('the attached terminal\'s frame', lambda: b'line 3 of the notes' in tt.text(), timeout=20) stub.send_signal(signal.SIGHUP) stub.wait(timeout=10) - # One put off the session's state (PARDES_TEST_DIVERGE): the next - # unit's digest finds it, it says so in a line and joins again, and - # its keys after go in and it agrees. + # One put off the session's state (PARDES_TEST_DIVERGE, at its + # 100th unit): the next unit's digest finds it, it says so in a line + # and joins again for its view, a pane of its own in it, its keys + # after go into that pane and it agrees. td = Term(24, 90) terms.append(td) - d = td.spawn([binary, f'--attach={sock}:files'], dict(env, PARDES_REPLICA_CHECK=str(root / 'd.check'), PARDES_TEST_DIVERGE='1')) + d = td.spawn([binary, f'--attach={sock}:files'], dict(env, PARDES_REPLICA_CHECK=str(root / 'd.check'), PARDES_TEST_DIVERGE='100')) started.append(d.pid) wait_for('client D\'s first frame', lambda: b'line 3 of the notes' in td.text(), timeout=20) - os.write(td.master, b'jj') - time.sleep(.5) - # Another's unit, replayed on D's core: its digest finds it. - os.write(ta.master, b'jj') - wait_for('client D diverged and joined again', lambda: b'joined again' in td.text(), timeout=20) + made_live(td, root / 'd.check') + shown = view_panes(nine) + nine.write('/exec', b'New\n') + mine = wait_for('a pane in client D\'s view', lambda: view_panes(nine) - shown).pop() + # Others' units, replayed on D's core: one after its 100th finds it. + deadline = time.monotonic() + 60 + while b'joined again' not in td.text(): + assert time.monotonic() < deadline, 'client D diverged and joined again' + os.write(ta.master, b'jk') + time.sleep(.2) assert d.poll() is None + made_live(td, root / 'd.check') + assert mine in view_panes(nine), ('client D in its own view again', mine, nine.read('/layout')) seen = len(checks(root / 'd.check')) os.write(td.master, b'Oafter-rejoin-D\x1b') - wait_for('client D\'s keys after its rejoin', lambda: b'after-rejoin-D' in nine.read(f'/pane/{pane}/body')) + wait_for('client D\'s keys after its rejoin', lambda: b'after-rejoin-D' in nine.read(f'/pane/{mine}/body')) wait_for('client D out of insert mode', lambda: normal_again(td)) wait_for('client D agrees after its rejoin', lambda: agreed(root / 'd.check', seen)) os.write(td.master, b' sD') @@ -446,7 +471,7 @@ def clients(binary): wait_for('the session\'s fds level', lambda: fds(sess.pid) == alone, timeout=10) print(f'daemon: two replica clients (one through a 40 ms relay) typed into a detached session; ' f'digests agreed {len(checks(root / "a.check"))} and {len(checks(root / "b.check"))} times; ' - f'one put off the session\'s state joined again and agreed {len(checks(root / "d.check"))} times; ' + f'one put off the session\'s state joined again in its own view and agreed {len(checks(root / "d.check"))} times; ' f'one killed, the other typed on and detached; a third typed into a shell, agreed ' f'{len(checks(root / "c.check"))} times, and was hung up') exit_session(nine, pane) @@ -980,7 +1005,7 @@ def far_joins(sock, session, binary, n): def join(): r = Replica(sock, timeout=30) name = session.encode() - r.send(0x01, struct.pack(' Date: Fri, 30 Oct 2026 21:54:00 -0300 Subject: docs/next-daemon.md: the box soak's divergences and its clean quit: the session that quit at its 1093rd input was asked to (window mode left open took `imike quebec `, `SPC q` in a tag); a FreeBSD session's +Help one row short of its Linux clients' (Tty9p a Linux build's word alone), the hunt's `pane 2.file.scroll` gone with it, before and after at seeds 32 and 41 on the box; a Debug replica of a release session at every Grep (Names' query buffer past its length); the gates Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index c7be4b5a..434e1da4 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4423,3 +4423,47 @@ else watched it. `test/daemon.py` 20 cycles, `--upgrade` (against a `-Dstamp-commit` build), `--window-clients` and `--gui`; ReleaseSafe `--upgrade` (from the Debug build) and `--window-clients`: all pass. + +After the line, the box soak's divergences and its clean quit: +- **The session that quit at its 1093rd input was asked to.** Seed 32's + window typed `Ctrl-W h`, then `imike quebec `. Window mode stays open + until Enter, Esc, `:` or a label, so `i`, `m` and `i` went nowhere. `k` + at a column's top went to its tag, and there `e SPC q` ran Exit, with + nothing unsaved. The soak now leaves window mode with Enter. +- **A FreeBSD session's +Help had one row fewer than its Linux + clients'.** The box soak's rejoins after a divergence were + `pane 4.file.content`, and `.splice` over a +Help made again. Each was + at a `?` after SPC, which runs Help. Help's rows are the build's words, + and Tty9p was a Linux build's alone (`enabled` asked the OS), so a + Linux replica made a `SPC n 9 Tty9p` row its session never had. H19 + made the enum's digest the same in every build; the set of words was + still not. + - Fix: Tty9p is a word of every hosted build. Whether the host is + Linux is the authority's answer (`tape.authority`), so a replica of + a FreeBSD session refuses Tty9p as its session does. + - Test: test/cross_replay.py's session runs Help. Main's FreeBSD tape, + replayed by a Linux ReleaseSafe build, diverged at it (`pane + 5.file.content`); with the fix, 30 digests agree in Debug and in + ReleaseSafe. +- **A Debug replica of a release session diverged at every Grep's query + pane** (`pane 2.query`: cross_replay.py's ReleaseSafe tape replayed by + Debug). `Names.query_buf` was digested whole, and past `query_len` it + is undefined: 0xAA in Debug, something else in a release build. Fix: a + replica `len` on it. The same tape now agrees in Debug (30 digests). +- **The hunt's `pane 2.file.scroll` (seed 41) went with the Help fix.** + On main (zponxort) it came again at the same unit: tty207-1's `SPC ?` + (1/596, 1/597) made a +Help, a window client joined, and tty207-1's `%` + (1/713) diverged that window at `pane 2.file.scroll`. With the fix, the + same keys and units pass with no rejoin, on lppumkvm and on zznqlptu. + Not traced: which field the joined window got wrong, and why the window + that replayed the Help unit itself did not diverge there. +- **Box** (traced, ReleaseSafe at both ends): seed 32 (win,tty), 3 min. + On main: four rejoins in three clients. The traces put three of them at + the two Helps; the fourth, in the window that typed the first Help, + gave no reason. With the fix: no rejoin. Seed 41 (tty,win), 3 min, the same before and + after (one rejoin on main, none with the fix). +- **Gates** (on zznqlptu): Debug `core-test` tty, `replicas` (none + diverged or unequal), `test/daemon.py` 20 cycles and + `--window-clients`, cross_replay.py with tty and window recorders, and + the FreeBSD tape replayed by Debug tty, ReleaseSafe tty and Debug gui; + ReleaseSafe `test/daemon.py` 20 cycles: all pass. -- cgit v1.3 From 10cedf97c90d6a11a89d7e682e243b77fd6b6f1f Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Fri, 30 Oct 2026 23:54:48 -0300 Subject: docs/next-daemon.md: a window that joined late diverged at the next key that moved another view's pane (`pane 23.file.scroll`: a pane's `view` was derived, never dumped, read before the step's sync, the default in a core loaded from a dump; the tty's `SPC t w` at 243 s, its lines unwrapped in the session and wrapped in the joiner), the fix, the test, a 120-seed exploration before and after; not this one: the box run's query-pane divergence (`location_rows`) and `Query.land`'s panic; the gates Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 434e1da4..ef1b868d 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4467,3 +4467,45 @@ After the line, the box soak's divergences and its clean quit: `--window-clients`, cross_replay.py with tty and window recorders, and the FreeBSD tape replayed by Debug tty, ReleaseSafe tty and Debug gui; ReleaseSafe `test/daemon.py` 20 cycles: all pass. + +After the line, a window that joined late: +- **It diverged at the next key that moved a pane of another view** + (`pane 23.file.scroll`, the box soak's seed 7: win207-3 at its join + 466 s in; the same field in the two windows that joined at 16 and 17 + min of an earlier run, not kept). A pane's `view` (`Pane.SessionView`: Wrap, the context rows' + style, the body's and the tag's row heights, its rect's height) is + written by `sync`, and was derived: never dumped. A key's move reads it + before its step syncs (`halfPageMove`, `ensureCursorVisible`, through + `File.geometry`), and a core loaded from a dump had the default, Wrap + on and no row heights. The tty had typed `SPC t w` at 243 s, so the + session's lines no longer wrapped and the joiner's did; its first + stepped unit after the dump was the tty's Ctrl-D (1/2018, after + `open_view` and a pump), and it scrolled further. A window there all + along had made `view` in its own steps and agreed. A parked view's + share (`PaneView.view`) was read the same way at a `to_view`, before + the step syncs. Window row heights with TreeContext's context rows + diverged it too, with Wrap on. + - Fix: `view` is digested and dumped, in a pane and in a parked view's + share. A dump of an older build has none and takes the default, as + before. + - Test: replica.zig's "a client joined from a dump moves through another + view's pane as the session does, its lines not wrapped" (Wrap off, the + cursor near the view's bottom, a join, Ctrl-D). Main fails it at + `pane 0.file.scroll`. A throwaway exploration (120 seeds: two views, + window row heights, TreeContext, Wrap, keys from either view after a + join) diverged 20 times on main (`file.scroll`, `hscroll`), none with + the fix. + - ponytail: `notices` and `pan` stay derived. `unpark` makes `pan` + again; a pane's notices are read by a click on its bands. +- **Not this one:** the box run with the fix (seed 7, 21 min, seven + clients) showed no `scroll` divergence in the four windows that joined + late and no `view` difference anywhere, but win207-4 diverged once at its + own key in a query pane (`pane 21.file.location_rows`, unit 4/447: the + session's tape had the same host answers), and after its rejoin a + speculated Enter panicked in `Query.land` (`f.location_rows[c + 1]`, + len 0). Both left open. +- **Gates** (on vtoozmwn): Debug `core-test` tty, `replay`, `replicas` + (none diverged or unequal), `test/daemon.py` 20 cycles and + `--window-clients`; ReleaseSafe `test/daemon.py` 20 cycles and + `--window-clients`: all pass. The box soak, ReleaseSafe at both ends: + 0 rejoins in six of seven clients, the seventh's the query pane's above. -- cgit v1.3 From 4c761893e5b3fdae422622cfa02104ab36633045 Mon Sep 17 00:00:00 2001 From: Gabriel Schneider Date: Sat, 31 Oct 2026 01:07:45 -0300 Subject: docs/next-daemon.md: the query pane's two: a speculated Enter that panicked in Query.land (a current item outliving its row: a frozen pane's rows edited, an Esc's `saved` of them), and a window joined from a dump making a `/` query's rows plain where its session had coloured them (a row's colours painted from the derived highlights and locations cache); the fixes, the tests, the gates and the seed-7 box soak with no rejoin Co-Authored-By: Claude Opus 5.5 --- docs/next-daemon.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) (limited to 'docs/next-daemon.md') diff --git a/docs/next-daemon.md b/docs/next-daemon.md index ef1b868d..5537bf1a 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -4509,3 +4509,38 @@ After the line, a window that joined late: `--window-clients`; ReleaseSafe `test/daemon.py` 20 cycles and `--window-clients`: all pass. The box soak, ReleaseSafe at both ends: 0 rejoins in six of seven clients, the seventh's the query pane's above. + +After it, the query pane's two: +- **A speculated Enter panicked in `Query.land`** (`f.location_rows[c + + 1]`, `index 1, len 0`: win207-4 after its rejoin). An edit of a frozen + query pane's rows drops them with the text they were of + (`setContentSpan`'s dropped marks), while its items and its current row + stay, and a later search in the pane keeps them so in its `saved` for an + Esc. `land` indexed the rows by the current item all the same; every + other reader of them asks first. In the soak it was line 1 of a live pane + (another view's SPC b b and Esc, and the window's own keys, in another + order than the session took them), which the session never ran: the + input panicked before it was sent. A session's own keys reach it too: + SPC b b, Enter, then `jxd` on the pane's rows and Enter. + - Fix: a current item without its row is none: the keyboard to the + owner, as with no current row. + - Test: Query.zig's "Enter on a frozen query pane's rows after its text + was edited: no row to land on, the keyboard to its owner". Main panics + in it. +- **A window joined from a dump made a `/` query's rows plain where its + session had coloured them** (`pane 21.file.location_rows`, win207-4's + `/r`, unit 4/447: the session's tape had the same host answers). A hit + row's colours are the owner's highlights where a frame painted them, + else the locations cache's: both derived, made by what the process has + drawn and painted, and a joiner has neither. + - Fix: a row's colours are in no digest (Row's own `digest`); a dump + still keeps them. They are how a row looks, not where it points. + - Test: replica.zig's "a client joined from a dump makes a `/` query's + rows as the session does, though only the session has drawn its + owner". Main fails it at `pane 1.file.location_rows`. +- **Gates** (on nopmlypk): Debug `core-test` tty, `replay`, `replicas` + (none diverged or unequal), `test/daemon.py` 20 cycles and + `--window-clients`; ReleaseSafe `test/daemon.py` 20 cycles and + `--window-clients`; the box soak, seed 7, ReleaseSafe at both ends, 21 + min and six clients (late windows and a late terminal among them): no + rejoin, no differing check, no panic. All pass. -- cgit v1.3