diff options
| -rw-r--r-- | docs/next-daemon.md | 287 |
1 files changed, 287 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 6e47bd06..e1d9e039 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2995,3 +2995,290 @@ new/base of the medians: Installed 2026-10-30 05:27 into ~/.local/bin: pardes and pardes-gui 0.24 (bbcebf80c45e), ReleaseSafe. The previous ones (0.24 caffd9c1f874, 2026-10-29 08:57) are `*.bak-pre-daemon`, and pardes-v9fs is unchanged. + +## Phase 6: content cache and zstd notes + +Built 2026-10-30 (`src/replica.zig`: `Peer.farJoin`, the `Framer`'s +envelopes, `Have` and `Cache`; detached/wire.zig version 16; `walk.blobs`; +tty.zig `remote`; main.zig `remoteAttach`; libzstd in build.zig). A far +client, one through ssh, is sent a join's blobs by hash and only those it +holds nowhere, and everything the session sends it after its welcome goes +in zstd. Decisions at the forks, with the ponytail lens: + +- **Far or near, the client says.** A client dialling through ssh + (`--attach=<host>:<name>`) is far; one on this machine's own socket is + near and joins as before, byte for byte: there a hash and a compression + cost more than the bytes they save (a 100 MB file's first join takes + 1.5 s far against 0.25 s near, the table below). + `PARDES_REPLICA_FAR=1` makes a socket client far (the tests, the local + measurements), `=0` an ssh one near. The daemon reads no further than the + hello, so the word is the client's first message after it (`far`), in + the same write: the session's writer reads it before anything after the + welcome, with no round trip. +- **The protocol: the session lists, the client asks.** After the welcome + (which now says how many blobs the dump holds) a far client's writer + sends the list (`blobs`: each blob's place in the state, its length and + its BLAKE3 hash, 48 bytes), then the state with the blobs cut out (its + pieces), then flushes and waits for the client's `want` (a bit a blob), + then sends the wanted blobs, in order. The client resolves the list as + soon as it is whole and answers while the state is still on its way + (a first join pays that round trip: not done, below). A blob listed + twice (an undo step that changed nothing) is asked for once and copied. + The client says what it lacks, not what it holds: its cache can hold + thousands of blobs, the list of a join a few hundred. The step never + waits for any of it: the list is hashed on the session's writer thread + for that client, from the blobs part B pins for it. ponytail: hashed + again at each far join (BLAKE3, 3.6 GB/s); a session's later joins could + keep the hashes beside its words. +- **BLAKE3 from Zig's std, 256 bits.** A collision or a planted file must + never become wrong content, so the client hashes what it reads from disk + again before it uses it, and a file of another length or content (cut + short, corrupted, planted, longer) is a miss: deleted, and the blob is + asked for. What comes over the link is not hashed (the session said its + hash); the cache file it is written to is checked when it is read. +- **Where a far client holds blobs.** In order: the core it is leaving + (a rejoin after a divergence or an Upgrade: the old core is kept until + the new dump is whole, its blobs looked at by length and each hashed + once when one of its length is asked for), then the cache on disk. + ponytail: a scan of the old core's blobs a lookup, fine to some thousands + of blobs. +- **The cache: `$XDG_CACHE_HOME/pardes/blobs` (else `~/.cache/...`), a + file a blob named by its hash in hex, the directory 0700.** Not the + runtime directory: the blobs are worth keeping across a logout and a + reboot (a client started tomorrow) and safe to lose, and they can be + hundreds of megabytes, which the runtime directory (a tmpfs, often + capped at a tenth of memory, emptied at logout) is not for. A file is + written aside and renamed into place, so a second client never reads half + of one. A far client writes what it was sent on a thread after its load + (the first frame waits for none of it), and at its end, the blobs of its + core the cache lacks: what it made since it joined (an edit's text, its + undo snapshots), so a client started again is not sent them. A hangup + now ends a remote client the normal way, for that (and its ssh forward's + socket is unlinked: part B's last note). +- **Bounded: 1 GiB, `PARDES_BLOB_CACHE_MB` to change it; the least recently + used go first.** A hit makes its file's time now; before a store, the + directory is scanned and the oldest files deleted until what comes fits. + A join bigger than the cap stores what fits. ponytail: the scan and + sort at each store, fine to tens of thousands of files, where an index + would take over. +- **zstd, decided by its numbers (below), on the whole stream after the + welcome, not the join alone.** The session's writer puts every byte it + sends a far client through one compressor and writes envelopes + (`zstd`), each the flush of at most 1 MiB of input, so a reader unpacks + one whole into a bounded buffer; the client's `Framer` unpacks them and + hands their messages on as if they had come bare, so the dial and the + stream's reader are unchanged. The join is one frame at level 6 over an + 8 MiB window: an undo snapshot is a near copy of its text, a file's + length before or after it, and zlib's 32 KiB window never sees the copy. + What follows (units, terminal deltas) is a frame at level 3 over 1 MiB. + The client's direction is not compressed. With all of that, `ssh -C` + only spends both ends' time, and is no longer in the default command. + ponytail: a big paste through a slow link goes as it is; it would be an + envelope of its own. +- **libzstd 1.5.7, vendored as the upstream tarball** (one copy; nothing + else in the build bundles one; Zig's std has a decompressor but no + compressor). Its compressor and decompressor only, single-threaded, + without its x86 assembly, the block compressors of levels 3 and 6 alone + (`ZSTD_EXCLUDE_*`), its unused functions dropped at link; every context + made by `ZSTD_create*_advanced` with a `ZSTD_customMem` over c_heap, so + a test's allocations are its testing allocator's. + +The zstd decision, measured. A join's stream as a client reads it, taped +by a tee relay (`.scratch/daemon/amort/h/capture.py`): `real` is part B's +box session made here (pardes.zig, host_io.zig, terminal.zig, look.zig and +3 MB of the tree's sources joined, a shell with 1 MB of scrollback, 4.0 MB +of stream), `edited` the same after three edits to each text (each text's +undo snapshots: 15.8 MB), `typical` and `big` join2.py's synthetic files +(their lines differ by a counter, so everything compresses far better than +source does). Compressed whole with the CLIs, here (one core); the box's +CPU takes 0.03 s, 0.11 s and 1.04 s for real at -3, -9 and -19: + +| stream | raw | gzip -6 (ssh -C's) | zstd -3 | zstd -3, 8 MiB window | zstd -6, 8 MiB | zstd -9 | zstd -19 | +|---|---|---|---|---|---|---|---| +| real | 4.04 MB | 1.00 MB, 91 ms | 1.02 MB, 19 ms | 0.95 MB, 23 ms | 0.85 MB, 49 ms | 0.82 MB, 58 ms | 0.72 MB, 1005 ms | +| edited | 15.82 MB | 3.97 MB, 336 ms | 3.36 MB, 63 ms | 0.96 MB, 30 ms | 0.86 MB, 61 ms | 0.85 MB, 87 ms | 0.73 MB, 967 ms | +| typical | 8.33 MB | 1.22 MB, 98 ms | 0.72 MB, 19 ms | | | 0.35 MB, 39 ms | 0.20 MB, 723 ms | +| big | 77.4 MB | 11.17 MB, 768 ms | 6.94 MB, 136 ms | 4.10 MB, 131 ms | 3.71 MB, 336 ms | 4.80 MB, 467 ms | 1.82 MB, 22.4 s | + +The box's link (each payload `cat` from the box through ssh, the time +from its first byte here to its last, the median of three interleaved): + +| payload | through `ssh -C` | zstd -3, plain ssh | zstd -9, plain ssh | zstd -19, plain ssh | +|---|---|---|---|---| +| real (4.04 MB raw, 10.0 s through plain ssh) | 4.32 s | 4.82 s | 4.06 s | 4.14 s | +| edited (15.82 MB raw) | 8.90 s | 7.03 s | 4.51 s | 3.87 s | + +- **Without undo history zstd buys nothing** that the link shows: 0.82 MB + against zlib's 1.00 MB, 4.06 s against 4.32 s, within the runs' spread + (3.9 to 5.0 s). The link is not the bottleneck its 170 to 400 KB/s + suggests for a payload this size: a megabyte or less takes about 4 s + whatever it is, TCP's window growing over a 207 ms round trip. +- **With it, zstd halves the transfer** (8.90 s to 4.51 s): an undo + snapshot is a near copy of its text, which zstd's window of megabytes + matches and zlib's 32 KiB does not, and a session one works in for an + hour holds up to 256 of them a file. The window decides it, not the + level: at -3 with 8 MiB, edited is 0.96 MB where -3's own 2 MiB leaves + 3.36 MB (the 3 MB file's snapshots are 3 MB apart). +- **So zstd, at level 6 over 8 MiB**: within 4% of -9's size at about + two thirds of its time; -19 saves a little more and costs a second of the session's CPU + every 4 MB (22 s for big). The cache does not replace it: a first join + has nothing cached, and an edit made while no client was attached is new. + +The benefit, measured (`h/joincache.py` here, `h/boxcache.py` through the +box, both on `h/cachejoin.py`; ReleaseFast here and ReleaseSafe for the +box's pair, timings alone on the perf cores). Four joins of one session, +each client with its own terminal and check file, one cache between them: +the first (the cache empty); a second client while the first is attached; +that second one put off the session's state (PARDES_TEST_DIVERGE), which +joins again from the core it leaves; and one started after both ended. +The first frame is from the client's start, ssh's forward included (2.0 s +of it on the box, part B); a rejoin keeps its forward, and its time is its +dial and load (PARDES_REPLICA_STATS `join_ms`), its frame drawn after. + +- **Through the box** (ms; `real` 3.9 MB of files, a 3.9 MB dump; `edited` + the same after three edits a text, a 15.8 MB dump; main is the final + gate's ReleaseSafe pair, bbcebf80, through its default `ssh -C`): + + | session | build and ssh | first join | second client | rejoin, dial and load | started again | + |---|---|---|---|---|---| + | real | main, `ssh -C` | 6535 | 7166 | a whole join | 6952 | + | real | this line, `ssh` (the default) | 7069 | 2484 | 434 | 2619 | + | real | this line, `ssh -C` | 7623 | 2449 | 430 | 2409 | + | edited | main, `ssh -C` | 11931 | 11257 | a whole join | 10921 | + | edited | this line, `ssh` (the default) | 7079 | 2442 | 452 | 2482 | + | edited | this line, `ssh -C` | 6824 | 2412 | 460 | 2437 | + + - A client that holds the blobs is drawn 2.4 to 2.6 s after it starts + where main took 7.0 to 11.9 s: past ssh's 2.0 s, 0.4 to 0.5 s, + whatever the session's size. Its link carries the state (113 KB + before zstd) and the list. + - A first join of a session with undo history: 7.1 s where main took + 11.9 s (zstd's window). Without one, level with main (7.1 s against + 6.5 s, within the link's spread): zstd's 15% fewer bytes are worth + about the round trip the list's answer costs. + - `ssh -C` over zstd: 7.6 and 6.8 s against 7.1 and 7.1 for the first + join, the same for the others; noise either way, and dropped. + - The session's step while each joined (its stats, real): 0.3 to 1.2 ms + on the box, main's 0.3 to 0.9 ms. + +- **Here, the socket** (ms; join2.py's shapes: typical 4.1 MB of files + and 1 MB of scrollback, big 38.6 MB and 8 MB, huge a 100 MB file beside + typical's; three edits a text; near as before, far with + PARDES_REPLICA_FAR=1): + + | session | client | first join | second client | rejoin, dial and load | started again | the session's worst step during the joins | + |---|---|---|---|---|---|---| + | typical | near | 22 | 22 | 9 | 22 | 0.6 | + | typical | far | 42 | 32 | 17 | 32 | 0.5 | + | big | near | 104 | 103 | 66 | 83 | 1.2 | + | big | far | 517 | 153 | 127 | 153 | 0.8 | + | huge | near | 254 | 244 | 170 | 285 | 0.7 | + | huge | far | 1514 | 517 | 450 | 506 | 2.3 | + + - Near is part B's join unchanged (22 to 46, 87 to 117 and 247 to 288 + ms then), and the default on this machine's socket: far there costs + more than the bytes it saves (the session's hashing and compression on + a first join, the client's read and hash of what it holds after). + - A join costs the session's step at most 2.3 ms with a 100 MB file + open, far or near (the gate: under 16.7 ms). + +Gates (Debug unless said; scripts and logs in `.scratch/daemon/amort/`): +- **Correctness.** `test/daemon.py --far` (in its main run too): far + clients of a detached session with two files (one 1.3 MB) and edits; + the first sent every blob (1104 KB), a second client on this machine + from disk (3 of 4, the fourth the text the first had just made), its + rejoin after a forced divergence from the core it left (4 of 4), one + started after both ended from disk (6 of 6: what they made, kept at their + end), one after every cache file was corrupted or cut in half by hand + sent them all again (5470 KB), and two with a 1 MiB cap held by older + files: those went first and the cache stayed under its cap. Every + client's digest was the session's at every quiet point (11 to 17 checks + a run, none differed), Debug and ReleaseSafe. Through the box, every + check of the four joins above agreed (28 to 41 a run). `zig build + replicas`: 4 seeds at 30, 80 and 207 ms, none diverged, none unequal, no + delta wrong. `test/daemon.py`'s replica tests pass, near and far. +- **New tests** (core-test, `-Dtest-filter=blob`): a cached blob is hashed + again as it is read, and one cut short, corrupted, planted or longer is + a miss and goes; the cache stays under its cap, the least recently used + going first (a read makes a file the newest; a file not a blob's counts + and goes by its time); a far join over a socket pair, the session's + writer and reader threads against the client's dial, is sent each blob + once: again with them on disk, or in the core it leaves, the state + alone, and a cached file gone bad sends that blob again; a near join's + dump is the session's byte for byte. + +- **Suites**, once at the end: `core-test` (its run beside the box + measurements had the 9P failure above; again, all pass), `unit-test + -Dplatform=tty` (its perf gate, and shared-libs-check, which now refuses + a second zstd too), `snap` 113 of 113, `replay` (113 scripts, 20 monkey + seeds, none panicked or diverged), `replicas`, `9p-io-test`, `fs-test`, + `test/daemon.py` 20 cycles (its far clients included), `--upgrade` + (unstamped Debug to stamped), `test/fs.py` (49 s): pass. + `unit-test -Dplatform=gui`: 1568 of 1580, 10 skipped, and the two that + fail on main (`tagline pointer mapping...`, `a tag caret past its row's + end...`). +- **ReleaseSafe** (`zig build -Doptimize=ReleaseSafe -Dhome-install=false`, + the binaries the box measurements ran): a tty in a pty with the daemon on + and off, `test/daemon.py` 20 cycles, `test/daemon.py --far`, `test/fs.py` + (39 s), `snap -Doptimize=ReleaseSafe` (113 of 113) and the cache tests + (`core-test -Doptimize=ReleaseSafe -Dtest-filter=blob`): pass. +- **No local regression.** ReleaseFast, interleaved base new base new base + new base on the perf cores (`core_optimize` fast on both sides), the + median of each side's runs, new/base, beside base's own A/A; builds ran + on the other cores throughout, so every cell is in that state: + - the core in process (`pardes-perf --reps 15`): the opens 0.90 to 0.99, + render 0.94 to 0.99, the keys, page-down and wheel 0.80 to 1.03, the + edits 0.90 to 1.05, body-2m 0.99 to 1.04, body-appends 0.92 to 1.07 + (its A/A 0.99 to 1.30), deep-sel 0.89 to 0.97 (and longline's 364 ms + on base, 0.1 ms on main already); a terminal's cells + (sb-64k to sb-8m) 0.87 to 1.01. The A/A elsewhere 0.94 to 1.05. + - live tty keystroke to present (base's `pardes-perf-live --live`, + present p50 / core p50): key-down 1.01 / 0.99, key-right 1.01 / 0.95, + edit-char 0.97 / 0.98, page-down 0.99 / 0.92, wheel 1.08 / 1.04 (A/A + 1.05 to 1.16); a 1M-line open 1.06 (A/A 0.92); `yes | head -c 32M` + 1.22 (A/A 1.13), its runs 4.1 to 7.3 s on this line and 5.4 to 6.7 on + base, as main's 7.0 to 7.3 beside base's 6.0 to 7.3 were in the final + gate. A session's own path is untouched by this change: a session with + no far client hashes and compresses nothing, and a near one is sent + its dump as before. + +Found on the way: +- **PARDES_REPLICA_CHECK keeps a session's units coming.** With two clients + checking, an idle session sends a unit about every 300 ms on main too + (`h/idle.py`: 27 checks in 8 s, near or far, main's build alike): a + check wakes the session's pump, whose unit makes the client's state + "changed", and the client asks again once quiet. Only the test knob + does it; a harness that expects a divergence to wait for a key it sends + finds the client joined again before (the measurements above wait for + the first link's stats line instead). +- **A rejoin's "joined again" line can be gone before it is drawn**: it + lasts until the next key, and keys typed during a long rejoin (a 100 MB + file's) come first. As designed; the harnesses do not look for it. +- **Harness traps.** A terminal's frame request split across two reads + went unanswered (test/daemon.py's `Term` now keeps the last bytes of a + read); keys written right after a first frame raced the terminal's + answer to it, an answer behind an Esc in one read being an Alt chord + (the far test waits 0.3 s, and writes the Esc on its own); a `pkill -f` + whose pattern was in the shell's own argv killed the shell. +- The core-test run of the suites had one failure, `9p_io`'s "a change + waits while the editor is out mid-step", with the box measurements and + builds running; alone three times, and the whole core-test again, it + passed. + +Not done, and why: +- **A first join pays the list's round trip** (207 ms on the box): the + wanted blobs wait for the client's answer, and the state (113 KB before + zstd) no longer hides it. A client with nothing cached could say so in + its `far` word and be sent everything at once; it is a first join on a + new machine only. +- **A blob is hashed again at every far join** on the session's writer + thread (a 254 MB dump: 70 ms there). The hashes could be kept beside + the blob words. +- **What a client sends is not compressed** (its keys, and a paste): with + `-C` gone, a megabyte's paste crosses the box's link raw. +- **A client killed outright keeps on disk only what it was sent**: what + it made since (its edits' texts and undo) is written at its end, a + hangup's included, but not on a kill -9; the next join gets those again, + in zstd. +- **The cache is per user, unencrypted, 0700**: a blob of a file the user + could read elsewhere, kept until evicted or the cache is cleared. |
