diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/next-daemon.md | 260 |
1 files changed, 260 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md index c7cb719d..00f66cf8 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -1795,3 +1795,263 @@ Not done, and why: path is the tty's. - The other agent wires it in: the tty and GUI hosts, `Screens` per attached peer, and a replica frontend drawing after `screen`. + +## Phase 6: upgrade notes + +Built 2026-10-29 (`src/upgrade.zig`, the tty and GUI hosts, `daemon.zig`, +`main.zig`, `dump.resumeFromDump`; `test/daemon.py --upgrade`). A running +session, and the daemon, move onto the pardes installed since they +started, in place: text, undo, views and layout, terminals with their +running shells, and their terminals, windows and daemon link. Decisions +taken at the forks, with the ponytail lens: + +- **The trigger is the word, `Upgrade`** (`SPC s U`, any tag, `/ctl` and + `/pane/N/exec` over 9P). It moves the daemon too: the session sends it + `upgrade` just before its own exec, so one word moves both, and no word + of the daemon's own was needed. Nothing upgrades by itself. A focus-in + (one comes back to the terminal or window from rebuilding elsewhere) + stats the start path, and a file there that is not the one the process + started from is said once, in a line: `a newer pardes is installed: + Upgrade (SPC s U) moves this session onto it`. Rejected: a watch on the + binary's directory (an inotify entry and an event source for a line), + and upgrading on its own (a build half-written, or one the user means + to test first, would take every session). +- **Which binary: the path this process was started from**, read once + just after its first frame (`std.process.executablePath`, a readlink, + and a stat for the notice: off a startup's way, and no build is + installed within a frame of a start), exec'd again at the upgrade. + `install(1)` and `zig build` put a new file at the path, renamed over + the running one, which `/proc/self/exe` then names `(deleted)`. The + record carries the path + exec'd, so a process that fell back to the old binary (whose own exe is + that deleted file) upgrades from the same path next time. Not argv[0] + and PATH: a session's PATH is the one it started with, and the readlink + is what the daemon's spawn already used. +- **Asked first, `<path> --version`**: run as it is, it must exit 0 with a + `pardes ` line within 10 s. A file not there, not a pardes, or one that + dies at start is found before anything is let go, and said ("Upgrade: + <path> exited 3 at --version; this session stays as it is"). 9P is + served while it runs (the turn rested), so a client's clunk after its + `Upgrade` write, and anything else, is answered; after it, nothing is. +- **At a step boundary**: after a pump, the pty readers are canceled (a + canceled tty reader posts an end of file, passed over while upgrading), + the terminals' input readers stopped, the loop's queue applied, and the + core settled (`Pardes.settle`: queued inputs, then effects). In the GUI + the queue is drained, the readers canceled, and drained again (a reader + waiting for room gets it). Bytes read are never lost; keys typed during + the exec wait in the terminal's kernel buffer for the new process. +- **The dump goes to the DumpDir, `pardes-upgrade-<pid>.dump`**, not a + memfd and not the runtime directory: a refused load leaves it where a + relative `Restore` finds it, on a disk and not a tmpfs (a big file's + session dumps 100 MB and up). Never $PARDES_DUMP, which is a Dump's. A + successful load deletes it. +- **The record is one argument**, `--upgraded=<lines>`, ahead of the + arguments the session was started with, which configure it again + (`--9p=`, `--tty-toggle`, `--geometry`, `--nested`); its file, stdin, + directory, `--detach` check, forwarding and daemon fork are skipped. A + line a word: `link`, `stub`, `old` (fds), `window`, `pty <pane> <pid> + <fd>`, `tty <view> <fd> <stub> <primary> <termios hex>`, `exe`, + `dump`, and `refused` in a fallback; a word another build adds is + passed over. Every fd it names is close-on-exec but across the one exec + (`Record.closeOnExec`), and the new process sets them back at once, so + no shell it forks inherits a master or a terminal. +- **The pid stays**: the shells stay the session's children, the stub + keeps waiting on it, the 9P name of a session named by its pid is the + same, and PDEATHSIG (set again) still means the stub's death. +- **Terminals.** A kept terminal pane is not revived (`resumeFromDump` + with `kept`): no reset, no banner, no new shell; ghostty's state is the + dump's, the program's own, and its pty keeps its size (no 1x1 nudge: + the screen is already right). The host puts the master back in the + pane's slot and reads on (a command pane's exit watched again). A + terminal pane with no live master (its shell gone) is revived as a + Restore revives it. +- **The tty's terminals stay raw, their cooked termios in the record**: + put back to cooked between the two processes, a key would echo and a + ^C (forwarded by the stub) could kill the new process before its + handlers. What is a stack is unwound: the kitty keyboard flags are + popped (the new process pushes them once its query is answered), its + kitty images deleted and its shm names unlinked (the pid is the same, + and so would the names be). The alternate screen, mouse and paste modes + stay on; the new process sets them again and draws everything, one + full frame over the old one in place. The old image's prompt rc files + (the running shells have read them) are swept by the new one, which + makes its own: at its sweep, any of its pid are the old image's. They + stay for an exec that failed. +- **Views come back whole.** `resumeFromDump` keeps every view the dump + has (no `views.only`) and does not run the startup config again (the + session's settings are the dump's). Each terminal's view is in its + `tty` line: the primary one is `run`'s, the others are entered as + `--attach` enters one, with their stubs; a view that lost its terminal + closes. The GUI opens a window for each open view, the live one first + (`openAnotherWindow(k)`): its windows are the one flash, as decided. +- **9P is cut and comes back.** What the connections had to send is let + go first (`Listener.flushOut`, the 200 ms the quit's path already + had), then the exec closes the listener and its lock, and the new + process claims the name again. ponytail: a dial during the exec and the + load is refused (one round of a client's retry); passing the listener + through as well would keep it queued. +- **Replicas are told in words.** Each connection a session serves gets + a `refused` frame ("the session is upgrading to another build of + pardes: connect again with that build") and a shutdown before the exec. + Their hello's semantics hash would not match the new build anyway, and + the session registers its new hash on the same link (a second `hello`, + which the daemon takes as an update). The reconnect, and a local + client of the same path re-exec'ing itself, are phase 5B's: there is no + replica client yet. +- **The daemon re-execs the same way**: its own start path, asked + `--version` first, the listener and lock (fds 3 and 4), the replicas' + listener, every session's link and every held master kept, its + registry on its command line (`--daemon --upgraded=` with a `c` line a + connection, its slot, kind, pid, state, semantics and name, and an `h` + line a master and its owner's slot); a replica whose hello is still on + its way is let go and dials again. Sessions never see it: their links + are the same sockets. +- **Failure never loses a shell.** + 1. The binary does not answer `--version`: nothing changes, said. + 2. The exec fails: the fds close-on-exec again, the readers and the + terminals' keys start again, the kitty flags are pushed back, the + dump deleted, said. + 3. The new build cannot load the dump: it execs the old binary again, + from `/proc/self/exe` opened before the exec (`old`, Linux), with + `refused` set; that one loads its own dump and says "Upgrade: the new + build could not load this session (<why>); it goes on as <version>". + 4. No old binary to go back to (FreeBSD has no /proc by default), or it + refuses too: a fresh session adopts every kept master as a terminal + pane (the Adopt path) and says "Upgrade: no build could load the + session (<why>); its shells are here, and Restore <dump> brings the + rest back". The dump stays. + 5. A crash after the exec is the crash Phase 3 covers: the daemon's + dups keep the shells for an Adopt, and the dump is in the DumpDir. + `PARDES_TEST_UPGRADE_REFUSE=1` makes a start refuse the load as such a + build would (the first start only), `=all` the fallback's start too: + test/daemon.py installs a script that sets it and execs the new build. + +Found on the way: +- A fresh session's first pane has no directory until its own shell's + spawn is performed, so the `Tty` word that adopts a shell refused ("Tty: + : no such directory") and the shell was hung up with "no room": the + kept shells are taken after the startup effects, and those they emit + performed then. +- A test reconnecting after the word could reach the old process, still + asking `--version` (and serving 9P meanwhile): the test waits for the + exec (the session's exe, or on FreeBSD its new semantics in the + daemon's registry) before it reconnects. +- An `Adopt` (or a `Detach`) served in a session's first pump, before its + first frame, is refused as if no daemon kept it: the reader that takes + the daemon's `ok` starts after that frame, and `kept()` waits 2 s for + it. The test's next session waits for its first frame. Pre-existing, + left as it is. + +Gates (Debug, `A` this line unstamped, `B` the same with `-Dstamp-commit`, +a trivial visible change: `pardes 0.24 (<commit>)` and another semantics +hash; both installed in turn at a private prefix by a rename, as +`zig build` and install(1) do): +- `python3 -B test/daemon.py --upgrade A B`, under a private runtime dir. + A tty session with an unsaved edit, a shell running `sleep 1000; echo + done-after-upgrade` and a second terminal (a second view), and a + replica of A's build connected: + - B installed, a focus-in says the newer build; `Upgrade` written to + `/ctl`: the session's 9P answers again 123 ms later (81 to 83 ms on + the box). The session's pid is the same, `/proc/<pid>/exe` is B (not + deleted), and so is the daemon's, the same pid. + - The edit is there, the pane dirty, and `Undo` takes it back. + - The shell's pid is the same, still the session's child, its `sleep` + running; killed, the shell goes on and `done-after-upgrade` shows. + - Both terminals are drawn again (each its own view); the stubs wait on. + - The replica of A's build is told `the session is upgrading to + another build...`; a hello of B's build is welcomed by the daemon, + one of A's refused (the daemon lists the session as B). + - The fds level: the session's as before but the replica's, the + daemon's the same; the session's prompt files as many as before, the + new image's. + - A refusing build (a script that sets `PARDES_TEST_UPGRADE_REFUSE=1` + and execs B) installed and asked: the session goes back to the binary + it ran, its exe that inode again, the shell and text as they were. + - One that refuses everywhere (`=all`): a fresh session, the shell a + pane of it (`echo $$` there is its pid), the dump left in the DumpDir. + - The session killed outright: the daemon, upgraded three times, still + holds the shell's master, and a new session adopts it. + - Nothing left: no process, no socket. Passed 3 runs of 3 (and every + run since). +- `--upgrade --gui A-gui B-gui`: a window session in test mode (hidden + windows, captured) with a second window from `pardes-gui --attach`, the + same edit and job: after `Upgrade` the pid is the same and the exe B's, + the daemon B, the edit and undo there, the shell and its job the + session's, both windows' captures written again, B's hello welcomed; + the session ends with nothing left. Passed 5 runs of 5. +- FreeBSD 15.1 (the box, `pardes-ua` and `pardes-ub` cross-built from + this line): the same tty test but the fallback to a kept copy (there is + none). The new build is told by its semantics in the daemon's registry, + and the daemon's re-exec by `--upgraded=` on its command line; the fds + level by procstat. 81 to 83 ms from the word to 9P. Passed 9 runs of 10 + (the last two on this line's final builds); the one failure was the + test's first-frame race above, fixed, and it passed after. +- The existing gates: `test/daemon.py` (20 cycles) and `--gui` (10) + pass; `test/fs.py` passes (49 s); `zig build snap` 113 of 113; `zig + build replay` passes (the 113 scripts, 20 seeds at one view and 20 at + three, their dumps loaded to their digests); unit-test tty 1479 of + 1499, 20 skipped; unit-test gui 1550 of 1561, 9 skipped, and the two + known failures (`tagline pointer mapping...`, `a tag caret past its + row's end...`). core-test's new tests: the record read back as written + (and refused when cut), and an upgraded dump keeping its views, a + running terminal's alternate screen and no new shell, and settings the + startup config would change. + +Measured. The upgrade itself, Debug, a 300 MB binary: from the word +written to `/ctl` to the session's 9P answering again, 123 to 130 ms, +one run beside builds 299 ms (the `--version` ask, the drain, a dump of +a small session, the exec and the load); on the box, 81 to 83 ms with +its 125 MB binary. + +No local change (ReleaseFast tty, main kuovmsxw against this line, both +live at once, their samples alternating, through the perf lock; lat.py's +startup is a spawn with no daemon running to its first frame's end): + +| | A/B | A/A | +|---|---|---| +| startup, 100 a build: p50 | 0.944 | 1.010 | +| the same: mean | 0.919 | 1.061 | +| the same: median of the fast mode (and its share) | 0.964 (54 to 52) | 0.996 (69 to 73) | +| startup, 40 a build, two runs: p50 | 1.001, 1.266 | 0.998, 1.010 | +| key in a file, to its frame, p50, two runs | 0.997, 1.002 | 0.958, 0.990 | +| key echoed by bash, to its frame, p50, two runs | 0.997, 0.991 | 1.004, 0.999 | + +The startup times have two modes on this machine, about 4.5 and about 12 +ms, and the share in each moves between runs and between the builds of +an A/A alike, so a p50 that falls between them jumps (the 1.266). Within +a mode the builds are the same. A first A/B, made while the path was +still read at the start (a readlink and a stat before the first frame, +some microseconds), read p50 1.04 to 1.13 against an A/A of 0.98 to 1.00 +on runs that overlapped builds; the read was moved after the first frame +anyway, so a start does nothing it did not. + +Not done, and why: +- Remote replicas reconnect on their own, and a local client of the same + path re-execs itself first: phase 5B's (no replica client exists yet). + What they get today is the `refused` frame above, then the hangup. +- The 9P listener is not kept across the exec: a dial in that window (a + hundred milliseconds or so) is refused. Clients that dial afresh (the + hosts' Client) retry; a mounted session sees its connection end. +- On FreeBSD a build that cannot load the dump starts afresh (shells + adopted, dump kept): there is no `/proc/self/exe` to go back to by + default. Keeping an fd of the binary open from the start would do it, + at an fd a session. +- A crash of the new build after it starts is a crash: the shells wait in + the daemon for an Adopt and the dump is in the DumpDir, but nothing + restarts the session on its own (the stub, which runs the old binary, + could). +- An escape sequence a program had half written when the dump was made is + lost (the snapshot is at ground state, Phase 2 notes). +- Language servers and `|` pipes running at the upgrade are let go (their + pipes close at the exec; they end), and the shells of panes closed just + before it, still waiting to be reaped, are reaped only when the session + ends. A daemon note (a terminal attaching, adopted shells) read by the + old process's thread but not yet taken by its loop is lost; the stub of + such a terminal sees its pipe end. +- A detached session's upgrade (no terminal) was checked by hand + (scratch script), not in test/daemon.py. +- GUI windows come back at the size and place the compositor gives them. +- A daemon protocol bump across an upgrade is not handled: the sessions + of the older build keep talking theirs. +- A session name with a newline would break the daemon's registry line. +- An upgraded process records no tape (a recording ends at the upgrade). |
