diff options
| author | Gabriel Schneider <[email protected]> | 2026-10-30 10:43:35 -0300 |
|---|---|---|
| committer | Gabriel Schneider <[email protected]> | 2026-10-30 10:44:19 -0300 |
| commit | 771156f92661e1def2d4deec1465ebca9d4c2a9d (patch) | |
| tree | 58bdb458cd6ce1a27c2a8c630ddbe373862fffed /docs | |
| parent | ddb2554fff073eed0b73a43825b2c007c895bc4d (diff) | |
| download | pardes-771156f92661e1def2d4deec1465ebca9d4c2a9d.tar.gz pardes-771156f92661e1def2d4deec1465ebca9d4c2a9d.zip | |
docs/next-daemon.md: phase 6's content cache and zstd notes: the decisions at its forks (far or near by the client's word, the session lists and the client asks, BLAKE3 hashed again on load, the core it leaves before the disk, the cache directory and its bound, zstd on the whole stream after the welcome, libzstd trimmed), zstd's numbers (gzip -6 against zstd's levels on four join streams, and their transfer through the box's link: level with -C without undo history, half its time with it), the first frames through the box and here for a first join, a second client, a rejoin and a client started again, the session's step during joins, the gates in Debug and ReleaseSafe, the A/B against base, what was found and what is not done
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/next-daemon.md | 287 |
1 files changed, 287 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 6e47bd06..e1d9e039 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2995,3 +2995,290 @@ new/base of the medians: Installed 2026-10-30 05:27 into ~/.local/bin: pardes and pardes-gui 0.24 (bbcebf80c45e), ReleaseSafe. The previous ones (0.24 caffd9c1f874, 2026-10-29 08:57) are `*.bak-pre-daemon`, and pardes-v9fs is unchanged. + +## Phase 6: content cache and zstd notes + +Built 2026-10-30 (`src/replica.zig`: `Peer.farJoin`, the `Framer`'s +envelopes, `Have` and `Cache`; detached/wire.zig version 16; `walk.blobs`; +tty.zig `remote`; main.zig `remoteAttach`; libzstd in build.zig). A far +client, one through ssh, is sent a join's blobs by hash and only those it +holds nowhere, and everything the session sends it after its welcome goes +in zstd. Decisions at the forks, with the ponytail lens: + +- **Far or near, the client says.** A client dialling through ssh + (`--attach=<host>:<name>`) is far; one on this machine's own socket is + near and joins as before, byte for byte: there a hash and a compression + cost more than the bytes they save (a 100 MB file's first join takes + 1.5 s far against 0.25 s near, the table below). + `PARDES_REPLICA_FAR=1` makes a socket client far (the tests, the local + measurements), `=0` an ssh one near. The daemon reads no further than the + hello, so the word is the client's first message after it (`far`), in + the same write: the session's writer reads it before anything after the + welcome, with no round trip. +- **The protocol: the session lists, the client asks.** After the welcome + (which now says how many blobs the dump holds) a far client's writer + sends the list (`blobs`: each blob's place in the state, its length and + its BLAKE3 hash, 48 bytes), then the state with the blobs cut out (its + pieces), then flushes and waits for the client's `want` (a bit a blob), + then sends the wanted blobs, in order. The client resolves the list as + soon as it is whole and answers while the state is still on its way + (a first join pays that round trip: not done, below). A blob listed + twice (an undo step that changed nothing) is asked for once and copied. + The client says what it lacks, not what it holds: its cache can hold + thousands of blobs, the list of a join a few hundred. The step never + waits for any of it: the list is hashed on the session's writer thread + for that client, from the blobs part B pins for it. ponytail: hashed + again at each far join (BLAKE3, 3.6 GB/s); a session's later joins could + keep the hashes beside its words. +- **BLAKE3 from Zig's std, 256 bits.** A collision or a planted file must + never become wrong content, so the client hashes what it reads from disk + again before it uses it, and a file of another length or content (cut + short, corrupted, planted, longer) is a miss: deleted, and the blob is + asked for. What comes over the link is not hashed (the session said its + hash); the cache file it is written to is checked when it is read. +- **Where a far client holds blobs.** In order: the core it is leaving + (a rejoin after a divergence or an Upgrade: the old core is kept until + the new dump is whole, its blobs looked at by length and each hashed + once when one of its length is asked for), then the cache on disk. + ponytail: a scan of the old core's blobs a lookup, fine to some thousands + of blobs. +- **The cache: `$XDG_CACHE_HOME/pardes/blobs` (else `~/.cache/...`), a + file a blob named by its hash in hex, the directory 0700.** Not the + runtime directory: the blobs are worth keeping across a logout and a + reboot (a client started tomorrow) and safe to lose, and they can be + hundreds of megabytes, which the runtime directory (a tmpfs, often + capped at a tenth of memory, emptied at logout) is not for. A file is + written aside and renamed into place, so a second client never reads half + of one. A far client writes what it was sent on a thread after its load + (the first frame waits for none of it), and at its end, the blobs of its + core the cache lacks: what it made since it joined (an edit's text, its + undo snapshots), so a client started again is not sent them. A hangup + now ends a remote client the normal way, for that (and its ssh forward's + socket is unlinked: part B's last note). +- **Bounded: 1 GiB, `PARDES_BLOB_CACHE_MB` to change it; the least recently + used go first.** A hit makes its file's time now; before a store, the + directory is scanned and the oldest files deleted until what comes fits. + A join bigger than the cap stores what fits. ponytail: the scan and + sort at each store, fine to tens of thousands of files, where an index + would take over. +- **zstd, decided by its numbers (below), on the whole stream after the + welcome, not the join alone.** The session's writer puts every byte it + sends a far client through one compressor and writes envelopes + (`zstd`), each the flush of at most 1 MiB of input, so a reader unpacks + one whole into a bounded buffer; the client's `Framer` unpacks them and + hands their messages on as if they had come bare, so the dial and the + stream's reader are unchanged. The join is one frame at level 6 over an + 8 MiB window: an undo snapshot is a near copy of its text, a file's + length before or after it, and zlib's 32 KiB window never sees the copy. + What follows (units, terminal deltas) is a frame at level 3 over 1 MiB. + The client's direction is not compressed. With all of that, `ssh -C` + only spends both ends' time, and is no longer in the default command. + ponytail: a big paste through a slow link goes as it is; it would be an + envelope of its own. +- **libzstd 1.5.7, vendored as the upstream tarball** (one copy; nothing + else in the build bundles one; Zig's std has a decompressor but no + compressor). Its compressor and decompressor only, single-threaded, + without its x86 assembly, the block compressors of levels 3 and 6 alone + (`ZSTD_EXCLUDE_*`), its unused functions dropped at link; every context + made by `ZSTD_create*_advanced` with a `ZSTD_customMem` over c_heap, so + a test's allocations are its testing allocator's. + +The zstd decision, measured. A join's stream as a client reads it, taped +by a tee relay (`.scratch/daemon/amort/h/capture.py`): `real` is part B's +box session made here (pardes.zig, host_io.zig, terminal.zig, look.zig and +3 MB of the tree's sources joined, a shell with 1 MB of scrollback, 4.0 MB +of stream), `edited` the same after three edits to each text (each text's +undo snapshots: 15.8 MB), `typical` and `big` join2.py's synthetic files +(their lines differ by a counter, so everything compresses far better than +source does). Compressed whole with the CLIs, here (one core); the box's +CPU takes 0.03 s, 0.11 s and 1.04 s for real at -3, -9 and -19: + +| stream | raw | gzip -6 (ssh -C's) | zstd -3 | zstd -3, 8 MiB window | zstd -6, 8 MiB | zstd -9 | zstd -19 | +|---|---|---|---|---|---|---|---| +| real | 4.04 MB | 1.00 MB, 91 ms | 1.02 MB, 19 ms | 0.95 MB, 23 ms | 0.85 MB, 49 ms | 0.82 MB, 58 ms | 0.72 MB, 1005 ms | +| edited | 15.82 MB | 3.97 MB, 336 ms | 3.36 MB, 63 ms | 0.96 MB, 30 ms | 0.86 MB, 61 ms | 0.85 MB, 87 ms | 0.73 MB, 967 ms | +| typical | 8.33 MB | 1.22 MB, 98 ms | 0.72 MB, 19 ms | | | 0.35 MB, 39 ms | 0.20 MB, 723 ms | +| big | 77.4 MB | 11.17 MB, 768 ms | 6.94 MB, 136 ms | 4.10 MB, 131 ms | 3.71 MB, 336 ms | 4.80 MB, 467 ms | 1.82 MB, 22.4 s | + +The box's link (each payload `cat` from the box through ssh, the time +from its first byte here to its last, the median of three interleaved): + +| payload | through `ssh -C` | zstd -3, plain ssh | zstd -9, plain ssh | zstd -19, plain ssh | +|---|---|---|---|---| +| real (4.04 MB raw, 10.0 s through plain ssh) | 4.32 s | 4.82 s | 4.06 s | 4.14 s | +| edited (15.82 MB raw) | 8.90 s | 7.03 s | 4.51 s | 3.87 s | + +- **Without undo history zstd buys nothing** that the link shows: 0.82 MB + against zlib's 1.00 MB, 4.06 s against 4.32 s, within the runs' spread + (3.9 to 5.0 s). The link is not the bottleneck its 170 to 400 KB/s + suggests for a payload this size: a megabyte or less takes about 4 s + whatever it is, TCP's window growing over a 207 ms round trip. +- **With it, zstd halves the transfer** (8.90 s to 4.51 s): an undo + snapshot is a near copy of its text, which zstd's window of megabytes + matches and zlib's 32 KiB does not, and a session one works in for an + hour holds up to 256 of them a file. The window decides it, not the + level: at -3 with 8 MiB, edited is 0.96 MB where -3's own 2 MiB leaves + 3.36 MB (the 3 MB file's snapshots are 3 MB apart). +- **So zstd, at level 6 over 8 MiB**: within 4% of -9's size at about + two thirds of its time; -19 saves a little more and costs a second of the session's CPU + every 4 MB (22 s for big). The cache does not replace it: a first join + has nothing cached, and an edit made while no client was attached is new. + +The benefit, measured (`h/joincache.py` here, `h/boxcache.py` through the +box, both on `h/cachejoin.py`; ReleaseFast here and ReleaseSafe for the +box's pair, timings alone on the perf cores). Four joins of one session, +each client with its own terminal and check file, one cache between them: +the first (the cache empty); a second client while the first is attached; +that second one put off the session's state (PARDES_TEST_DIVERGE), which +joins again from the core it leaves; and one started after both ended. +The first frame is from the client's start, ssh's forward included (2.0 s +of it on the box, part B); a rejoin keeps its forward, and its time is its +dial and load (PARDES_REPLICA_STATS `join_ms`), its frame drawn after. + +- **Through the box** (ms; `real` 3.9 MB of files, a 3.9 MB dump; `edited` + the same after three edits a text, a 15.8 MB dump; main is the final + gate's ReleaseSafe pair, bbcebf80, through its default `ssh -C`): + + | session | build and ssh | first join | second client | rejoin, dial and load | started again | + |---|---|---|---|---|---| + | real | main, `ssh -C` | 6535 | 7166 | a whole join | 6952 | + | real | this line, `ssh` (the default) | 7069 | 2484 | 434 | 2619 | + | real | this line, `ssh -C` | 7623 | 2449 | 430 | 2409 | + | edited | main, `ssh -C` | 11931 | 11257 | a whole join | 10921 | + | edited | this line, `ssh` (the default) | 7079 | 2442 | 452 | 2482 | + | edited | this line, `ssh -C` | 6824 | 2412 | 460 | 2437 | + + - A client that holds the blobs is drawn 2.4 to 2.6 s after it starts + where main took 7.0 to 11.9 s: past ssh's 2.0 s, 0.4 to 0.5 s, + whatever the session's size. Its link carries the state (113 KB + before zstd) and the list. + - A first join of a session with undo history: 7.1 s where main took + 11.9 s (zstd's window). Without one, level with main (7.1 s against + 6.5 s, within the link's spread): zstd's 15% fewer bytes are worth + about the round trip the list's answer costs. + - `ssh -C` over zstd: 7.6 and 6.8 s against 7.1 and 7.1 for the first + join, the same for the others; noise either way, and dropped. + - The session's step while each joined (its stats, real): 0.3 to 1.2 ms + on the box, main's 0.3 to 0.9 ms. + +- **Here, the socket** (ms; join2.py's shapes: typical 4.1 MB of files + and 1 MB of scrollback, big 38.6 MB and 8 MB, huge a 100 MB file beside + typical's; three edits a text; near as before, far with + PARDES_REPLICA_FAR=1): + + | session | client | first join | second client | rejoin, dial and load | started again | the session's worst step during the joins | + |---|---|---|---|---|---|---| + | typical | near | 22 | 22 | 9 | 22 | 0.6 | + | typical | far | 42 | 32 | 17 | 32 | 0.5 | + | big | near | 104 | 103 | 66 | 83 | 1.2 | + | big | far | 517 | 153 | 127 | 153 | 0.8 | + | huge | near | 254 | 244 | 170 | 285 | 0.7 | + | huge | far | 1514 | 517 | 450 | 506 | 2.3 | + + - Near is part B's join unchanged (22 to 46, 87 to 117 and 247 to 288 + ms then), and the default on this machine's socket: far there costs + more than the bytes it saves (the session's hashing and compression on + a first join, the client's read and hash of what it holds after). + - A join costs the session's step at most 2.3 ms with a 100 MB file + open, far or near (the gate: under 16.7 ms). + +Gates (Debug unless said; scripts and logs in `.scratch/daemon/amort/`): +- **Correctness.** `test/daemon.py --far` (in its main run too): far + clients of a detached session with two files (one 1.3 MB) and edits; + the first sent every blob (1104 KB), a second client on this machine + from disk (3 of 4, the fourth the text the first had just made), its + rejoin after a forced divergence from the core it left (4 of 4), one + started after both ended from disk (6 of 6: what they made, kept at their + end), one after every cache file was corrupted or cut in half by hand + sent them all again (5470 KB), and two with a 1 MiB cap held by older + files: those went first and the cache stayed under its cap. Every + client's digest was the session's at every quiet point (11 to 17 checks + a run, none differed), Debug and ReleaseSafe. Through the box, every + check of the four joins above agreed (28 to 41 a run). `zig build + replicas`: 4 seeds at 30, 80 and 207 ms, none diverged, none unequal, no + delta wrong. `test/daemon.py`'s replica tests pass, near and far. +- **New tests** (core-test, `-Dtest-filter=blob`): a cached blob is hashed + again as it is read, and one cut short, corrupted, planted or longer is + a miss and goes; the cache stays under its cap, the least recently used + going first (a read makes a file the newest; a file not a blob's counts + and goes by its time); a far join over a socket pair, the session's + writer and reader threads against the client's dial, is sent each blob + once: again with them on disk, or in the core it leaves, the state + alone, and a cached file gone bad sends that blob again; a near join's + dump is the session's byte for byte. + +- **Suites**, once at the end: `core-test` (its run beside the box + measurements had the 9P failure above; again, all pass), `unit-test + -Dplatform=tty` (its perf gate, and shared-libs-check, which now refuses + a second zstd too), `snap` 113 of 113, `replay` (113 scripts, 20 monkey + seeds, none panicked or diverged), `replicas`, `9p-io-test`, `fs-test`, + `test/daemon.py` 20 cycles (its far clients included), `--upgrade` + (unstamped Debug to stamped), `test/fs.py` (49 s): pass. + `unit-test -Dplatform=gui`: 1568 of 1580, 10 skipped, and the two that + fail on main (`tagline pointer mapping...`, `a tag caret past its row's + end...`). +- **ReleaseSafe** (`zig build -Doptimize=ReleaseSafe -Dhome-install=false`, + the binaries the box measurements ran): a tty in a pty with the daemon on + and off, `test/daemon.py` 20 cycles, `test/daemon.py --far`, `test/fs.py` + (39 s), `snap -Doptimize=ReleaseSafe` (113 of 113) and the cache tests + (`core-test -Doptimize=ReleaseSafe -Dtest-filter=blob`): pass. +- **No local regression.** ReleaseFast, interleaved base new base new base + new base on the perf cores (`core_optimize` fast on both sides), the + median of each side's runs, new/base, beside base's own A/A; builds ran + on the other cores throughout, so every cell is in that state: + - the core in process (`pardes-perf --reps 15`): the opens 0.90 to 0.99, + render 0.94 to 0.99, the keys, page-down and wheel 0.80 to 1.03, the + edits 0.90 to 1.05, body-2m 0.99 to 1.04, body-appends 0.92 to 1.07 + (its A/A 0.99 to 1.30), deep-sel 0.89 to 0.97 (and longline's 364 ms + on base, 0.1 ms on main already); a terminal's cells + (sb-64k to sb-8m) 0.87 to 1.01. The A/A elsewhere 0.94 to 1.05. + - live tty keystroke to present (base's `pardes-perf-live --live`, + present p50 / core p50): key-down 1.01 / 0.99, key-right 1.01 / 0.95, + edit-char 0.97 / 0.98, page-down 0.99 / 0.92, wheel 1.08 / 1.04 (A/A + 1.05 to 1.16); a 1M-line open 1.06 (A/A 0.92); `yes | head -c 32M` + 1.22 (A/A 1.13), its runs 4.1 to 7.3 s on this line and 5.4 to 6.7 on + base, as main's 7.0 to 7.3 beside base's 6.0 to 7.3 were in the final + gate. A session's own path is untouched by this change: a session with + no far client hashes and compresses nothing, and a near one is sent + its dump as before. + +Found on the way: +- **PARDES_REPLICA_CHECK keeps a session's units coming.** With two clients + checking, an idle session sends a unit about every 300 ms on main too + (`h/idle.py`: 27 checks in 8 s, near or far, main's build alike): a + check wakes the session's pump, whose unit makes the client's state + "changed", and the client asks again once quiet. Only the test knob + does it; a harness that expects a divergence to wait for a key it sends + finds the client joined again before (the measurements above wait for + the first link's stats line instead). +- **A rejoin's "joined again" line can be gone before it is drawn**: it + lasts until the next key, and keys typed during a long rejoin (a 100 MB + file's) come first. As designed; the harnesses do not look for it. +- **Harness traps.** A terminal's frame request split across two reads + went unanswered (test/daemon.py's `Term` now keeps the last bytes of a + read); keys written right after a first frame raced the terminal's + answer to it, an answer behind an Esc in one read being an Alt chord + (the far test waits 0.3 s, and writes the Esc on its own); a `pkill -f` + whose pattern was in the shell's own argv killed the shell. +- The core-test run of the suites had one failure, `9p_io`'s "a change + waits while the editor is out mid-step", with the box measurements and + builds running; alone three times, and the whole core-test again, it + passed. + +Not done, and why: +- **A first join pays the list's round trip** (207 ms on the box): the + wanted blobs wait for the client's answer, and the state (113 KB before + zstd) no longer hides it. A client with nothing cached could say so in + its `far` word and be sent everything at once; it is a first join on a + new machine only. +- **A blob is hashed again at every far join** on the session's writer + thread (a 254 MB dump: 70 ms there). The hashes could be kept beside + the blob words. +- **What a client sends is not compressed** (its keys, and a paste): with + `-C` gone, a megabyte's paste crosses the box's link raw. +- **A client killed outright keeps on disk only what it was sent**: what + it made since (its edits' texts and undo) is written at its end, a + hangup's included, but not on a kill -9; the next join gets those again, + in zstd. +- **The cache is per user, unencrypted, 0700**: a blob of a file the user + could read elsewhere, kept until evicted or the cache is cleared. |
