summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/next-daemon.md287
1 files changed, 287 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md
index 6e47bd06..e1d9e039 100644
--- a/docs/next-daemon.md
+++ b/docs/next-daemon.md
@@ -2995,3 +2995,290 @@ new/base of the medians:
Installed 2026-10-30 05:27 into ~/.local/bin: pardes and pardes-gui
0.24 (bbcebf80c45e), ReleaseSafe. The previous ones (0.24 caffd9c1f874,
2026-10-29 08:57) are `*.bak-pre-daemon`, and pardes-v9fs is unchanged.
+
+## Phase 6: content cache and zstd notes
+
+Built 2026-10-30 (`src/replica.zig`: `Peer.farJoin`, the `Framer`'s
+envelopes, `Have` and `Cache`; detached/wire.zig version 16; `walk.blobs`;
+tty.zig `remote`; main.zig `remoteAttach`; libzstd in build.zig). A far
+client, one through ssh, is sent a join's blobs by hash and only those it
+holds nowhere, and everything the session sends it after its welcome goes
+in zstd. Decisions at the forks, with the ponytail lens:
+
+- **Far or near, the client says.** A client dialling through ssh
+ (`--attach=<host>:<name>`) is far; one on this machine's own socket is
+ near and joins as before, byte for byte: there a hash and a compression
+ cost more than the bytes they save (a 100 MB file's first join takes
+ 1.5 s far against 0.25 s near, the table below).
+ `PARDES_REPLICA_FAR=1` makes a socket client far (the tests, the local
+ measurements), `=0` an ssh one near. The daemon reads no further than the
+ hello, so the word is the client's first message after it (`far`), in
+ the same write: the session's writer reads it before anything after the
+ welcome, with no round trip.
+- **The protocol: the session lists, the client asks.** After the welcome
+ (which now says how many blobs the dump holds) a far client's writer
+ sends the list (`blobs`: each blob's place in the state, its length and
+ its BLAKE3 hash, 48 bytes), then the state with the blobs cut out (its
+ pieces), then flushes and waits for the client's `want` (a bit a blob),
+ then sends the wanted blobs, in order. The client resolves the list as
+ soon as it is whole and answers while the state is still on its way
+ (a first join pays that round trip: not done, below). A blob listed
+ twice (an undo step that changed nothing) is asked for once and copied.
+ The client says what it lacks, not what it holds: its cache can hold
+ thousands of blobs, the list of a join a few hundred. The step never
+ waits for any of it: the list is hashed on the session's writer thread
+ for that client, from the blobs part B pins for it. ponytail: hashed
+ again at each far join (BLAKE3, 3.6 GB/s); a session's later joins could
+ keep the hashes beside its words.
+- **BLAKE3 from Zig's std, 256 bits.** A collision or a planted file must
+ never become wrong content, so the client hashes what it reads from disk
+ again before it uses it, and a file of another length or content (cut
+ short, corrupted, planted, longer) is a miss: deleted, and the blob is
+ asked for. What comes over the link is not hashed (the session said its
+ hash); the cache file it is written to is checked when it is read.
+- **Where a far client holds blobs.** In order: the core it is leaving
+ (a rejoin after a divergence or an Upgrade: the old core is kept until
+ the new dump is whole, its blobs looked at by length and each hashed
+ once when one of its length is asked for), then the cache on disk.
+ ponytail: a scan of the old core's blobs a lookup, fine to some thousands
+ of blobs.
+- **The cache: `$XDG_CACHE_HOME/pardes/blobs` (else `~/.cache/...`), a
+ file a blob named by its hash in hex, the directory 0700.** Not the
+ runtime directory: the blobs are worth keeping across a logout and a
+ reboot (a client started tomorrow) and safe to lose, and they can be
+ hundreds of megabytes, which the runtime directory (a tmpfs, often
+ capped at a tenth of memory, emptied at logout) is not for. A file is
+ written aside and renamed into place, so a second client never reads half
+ of one. A far client writes what it was sent on a thread after its load
+ (the first frame waits for none of it), and at its end, the blobs of its
+ core the cache lacks: what it made since it joined (an edit's text, its
+ undo snapshots), so a client started again is not sent them. A hangup
+ now ends a remote client the normal way, for that (and its ssh forward's
+ socket is unlinked: part B's last note).
+- **Bounded: 1 GiB, `PARDES_BLOB_CACHE_MB` to change it; the least recently
+ used go first.** A hit makes its file's time now; before a store, the
+ directory is scanned and the oldest files deleted until what comes fits.
+ A join bigger than the cap stores what fits. ponytail: the scan and
+ sort at each store, fine to tens of thousands of files, where an index
+ would take over.
+- **zstd, decided by its numbers (below), on the whole stream after the
+ welcome, not the join alone.** The session's writer puts every byte it
+ sends a far client through one compressor and writes envelopes
+ (`zstd`), each the flush of at most 1 MiB of input, so a reader unpacks
+ one whole into a bounded buffer; the client's `Framer` unpacks them and
+ hands their messages on as if they had come bare, so the dial and the
+ stream's reader are unchanged. The join is one frame at level 6 over an
+ 8 MiB window: an undo snapshot is a near copy of its text, a file's
+ length before or after it, and zlib's 32 KiB window never sees the copy.
+ What follows (units, terminal deltas) is a frame at level 3 over 1 MiB.
+ The client's direction is not compressed. With all of that, `ssh -C`
+ only spends both ends' time, and is no longer in the default command.
+ ponytail: a big paste through a slow link goes as it is; it would be an
+ envelope of its own.
+- **libzstd 1.5.7, vendored as the upstream tarball** (one copy; nothing
+ else in the build bundles one; Zig's std has a decompressor but no
+ compressor). Its compressor and decompressor only, single-threaded,
+ without its x86 assembly, the block compressors of levels 3 and 6 alone
+ (`ZSTD_EXCLUDE_*`), its unused functions dropped at link; every context
+ made by `ZSTD_create*_advanced` with a `ZSTD_customMem` over c_heap, so
+ a test's allocations are its testing allocator's.
+
+The zstd decision, measured. A join's stream as a client reads it, taped
+by a tee relay (`.scratch/daemon/amort/h/capture.py`): `real` is part B's
+box session made here (pardes.zig, host_io.zig, terminal.zig, look.zig and
+3 MB of the tree's sources joined, a shell with 1 MB of scrollback, 4.0 MB
+of stream), `edited` the same after three edits to each text (each text's
+undo snapshots: 15.8 MB), `typical` and `big` join2.py's synthetic files
+(their lines differ by a counter, so everything compresses far better than
+source does). Compressed whole with the CLIs, here (one core); the box's
+CPU takes 0.03 s, 0.11 s and 1.04 s for real at -3, -9 and -19:
+
+| stream | raw | gzip -6 (ssh -C's) | zstd -3 | zstd -3, 8 MiB window | zstd -6, 8 MiB | zstd -9 | zstd -19 |
+|---|---|---|---|---|---|---|---|
+| real | 4.04 MB | 1.00 MB, 91 ms | 1.02 MB, 19 ms | 0.95 MB, 23 ms | 0.85 MB, 49 ms | 0.82 MB, 58 ms | 0.72 MB, 1005 ms |
+| edited | 15.82 MB | 3.97 MB, 336 ms | 3.36 MB, 63 ms | 0.96 MB, 30 ms | 0.86 MB, 61 ms | 0.85 MB, 87 ms | 0.73 MB, 967 ms |
+| typical | 8.33 MB | 1.22 MB, 98 ms | 0.72 MB, 19 ms | | | 0.35 MB, 39 ms | 0.20 MB, 723 ms |
+| big | 77.4 MB | 11.17 MB, 768 ms | 6.94 MB, 136 ms | 4.10 MB, 131 ms | 3.71 MB, 336 ms | 4.80 MB, 467 ms | 1.82 MB, 22.4 s |
+
+The box's link (each payload `cat` from the box through ssh, the time
+from its first byte here to its last, the median of three interleaved):
+
+| payload | through `ssh -C` | zstd -3, plain ssh | zstd -9, plain ssh | zstd -19, plain ssh |
+|---|---|---|---|---|
+| real (4.04 MB raw, 10.0 s through plain ssh) | 4.32 s | 4.82 s | 4.06 s | 4.14 s |
+| edited (15.82 MB raw) | 8.90 s | 7.03 s | 4.51 s | 3.87 s |
+
+- **Without undo history zstd buys nothing** that the link shows: 0.82 MB
+ against zlib's 1.00 MB, 4.06 s against 4.32 s, within the runs' spread
+ (3.9 to 5.0 s). The link is not the bottleneck its 170 to 400 KB/s
+ suggests for a payload this size: a megabyte or less takes about 4 s
+ whatever it is, TCP's window growing over a 207 ms round trip.
+- **With it, zstd halves the transfer** (8.90 s to 4.51 s): an undo
+ snapshot is a near copy of its text, which zstd's window of megabytes
+ matches and zlib's 32 KiB does not, and a session one works in for an
+ hour holds up to 256 of them a file. The window decides it, not the
+ level: at -3 with 8 MiB, edited is 0.96 MB where -3's own 2 MiB leaves
+ 3.36 MB (the 3 MB file's snapshots are 3 MB apart).
+- **So zstd, at level 6 over 8 MiB**: within 4% of -9's size at about
+ two thirds of its time; -19 saves a little more and costs a second of the session's CPU
+ every 4 MB (22 s for big). The cache does not replace it: a first join
+ has nothing cached, and an edit made while no client was attached is new.
+
+The benefit, measured (`h/joincache.py` here, `h/boxcache.py` through the
+box, both on `h/cachejoin.py`; ReleaseFast here and ReleaseSafe for the
+box's pair, timings alone on the perf cores). Four joins of one session,
+each client with its own terminal and check file, one cache between them:
+the first (the cache empty); a second client while the first is attached;
+that second one put off the session's state (PARDES_TEST_DIVERGE), which
+joins again from the core it leaves; and one started after both ended.
+The first frame is from the client's start, ssh's forward included (2.0 s
+of it on the box, part B); a rejoin keeps its forward, and its time is its
+dial and load (PARDES_REPLICA_STATS `join_ms`), its frame drawn after.
+
+- **Through the box** (ms; `real` 3.9 MB of files, a 3.9 MB dump; `edited`
+ the same after three edits a text, a 15.8 MB dump; main is the final
+ gate's ReleaseSafe pair, bbcebf80, through its default `ssh -C`):
+
+ | session | build and ssh | first join | second client | rejoin, dial and load | started again |
+ |---|---|---|---|---|---|
+ | real | main, `ssh -C` | 6535 | 7166 | a whole join | 6952 |
+ | real | this line, `ssh` (the default) | 7069 | 2484 | 434 | 2619 |
+ | real | this line, `ssh -C` | 7623 | 2449 | 430 | 2409 |
+ | edited | main, `ssh -C` | 11931 | 11257 | a whole join | 10921 |
+ | edited | this line, `ssh` (the default) | 7079 | 2442 | 452 | 2482 |
+ | edited | this line, `ssh -C` | 6824 | 2412 | 460 | 2437 |
+
+ - A client that holds the blobs is drawn 2.4 to 2.6 s after it starts
+ where main took 7.0 to 11.9 s: past ssh's 2.0 s, 0.4 to 0.5 s,
+ whatever the session's size. Its link carries the state (113 KB
+ before zstd) and the list.
+ - A first join of a session with undo history: 7.1 s where main took
+ 11.9 s (zstd's window). Without one, level with main (7.1 s against
+ 6.5 s, within the link's spread): zstd's 15% fewer bytes are worth
+ about the round trip the list's answer costs.
+ - `ssh -C` over zstd: 7.6 and 6.8 s against 7.1 and 7.1 for the first
+ join, the same for the others; noise either way, and dropped.
+ - The session's step while each joined (its stats, real): 0.3 to 1.2 ms
+ on the box, main's 0.3 to 0.9 ms.
+
+- **Here, the socket** (ms; join2.py's shapes: typical 4.1 MB of files
+ and 1 MB of scrollback, big 38.6 MB and 8 MB, huge a 100 MB file beside
+ typical's; three edits a text; near as before, far with
+ PARDES_REPLICA_FAR=1):
+
+ | session | client | first join | second client | rejoin, dial and load | started again | the session's worst step during the joins |
+ |---|---|---|---|---|---|---|
+ | typical | near | 22 | 22 | 9 | 22 | 0.6 |
+ | typical | far | 42 | 32 | 17 | 32 | 0.5 |
+ | big | near | 104 | 103 | 66 | 83 | 1.2 |
+ | big | far | 517 | 153 | 127 | 153 | 0.8 |
+ | huge | near | 254 | 244 | 170 | 285 | 0.7 |
+ | huge | far | 1514 | 517 | 450 | 506 | 2.3 |
+
+ - Near is part B's join unchanged (22 to 46, 87 to 117 and 247 to 288
+ ms then), and the default on this machine's socket: far there costs
+ more than the bytes it saves (the session's hashing and compression on
+ a first join, the client's read and hash of what it holds after).
+ - A join costs the session's step at most 2.3 ms with a 100 MB file
+ open, far or near (the gate: under 16.7 ms).
+
+Gates (Debug unless said; scripts and logs in `.scratch/daemon/amort/`):
+- **Correctness.** `test/daemon.py --far` (in its main run too): far
+ clients of a detached session with two files (one 1.3 MB) and edits;
+ the first sent every blob (1104 KB), a second client on this machine
+ from disk (3 of 4, the fourth the text the first had just made), its
+ rejoin after a forced divergence from the core it left (4 of 4), one
+ started after both ended from disk (6 of 6: what they made, kept at their
+ end), one after every cache file was corrupted or cut in half by hand
+ sent them all again (5470 KB), and two with a 1 MiB cap held by older
+ files: those went first and the cache stayed under its cap. Every
+ client's digest was the session's at every quiet point (11 to 17 checks
+ a run, none differed), Debug and ReleaseSafe. Through the box, every
+ check of the four joins above agreed (28 to 41 a run). `zig build
+ replicas`: 4 seeds at 30, 80 and 207 ms, none diverged, none unequal, no
+ delta wrong. `test/daemon.py`'s replica tests pass, near and far.
+- **New tests** (core-test, `-Dtest-filter=blob`): a cached blob is hashed
+ again as it is read, and one cut short, corrupted, planted or longer is
+ a miss and goes; the cache stays under its cap, the least recently used
+ going first (a read makes a file the newest; a file not a blob's counts
+ and goes by its time); a far join over a socket pair, the session's
+ writer and reader threads against the client's dial, is sent each blob
+ once: again with them on disk, or in the core it leaves, the state
+ alone, and a cached file gone bad sends that blob again; a near join's
+ dump is the session's byte for byte.
+
+- **Suites**, once at the end: `core-test` (its run beside the box
+ measurements had the 9P failure above; again, all pass), `unit-test
+ -Dplatform=tty` (its perf gate, and shared-libs-check, which now refuses
+ a second zstd too), `snap` 113 of 113, `replay` (113 scripts, 20 monkey
+ seeds, none panicked or diverged), `replicas`, `9p-io-test`, `fs-test`,
+ `test/daemon.py` 20 cycles (its far clients included), `--upgrade`
+ (unstamped Debug to stamped), `test/fs.py` (49 s): pass.
+ `unit-test -Dplatform=gui`: 1568 of 1580, 10 skipped, and the two that
+ fail on main (`tagline pointer mapping...`, `a tag caret past its row's
+ end...`).
+- **ReleaseSafe** (`zig build -Doptimize=ReleaseSafe -Dhome-install=false`,
+ the binaries the box measurements ran): a tty in a pty with the daemon on
+ and off, `test/daemon.py` 20 cycles, `test/daemon.py --far`, `test/fs.py`
+ (39 s), `snap -Doptimize=ReleaseSafe` (113 of 113) and the cache tests
+ (`core-test -Doptimize=ReleaseSafe -Dtest-filter=blob`): pass.
+- **No local regression.** ReleaseFast, interleaved base new base new base
+ new base on the perf cores (`core_optimize` fast on both sides), the
+ median of each side's runs, new/base, beside base's own A/A; builds ran
+ on the other cores throughout, so every cell is in that state:
+ - the core in process (`pardes-perf --reps 15`): the opens 0.90 to 0.99,
+ render 0.94 to 0.99, the keys, page-down and wheel 0.80 to 1.03, the
+ edits 0.90 to 1.05, body-2m 0.99 to 1.04, body-appends 0.92 to 1.07
+ (its A/A 0.99 to 1.30), deep-sel 0.89 to 0.97 (and longline's 364 ms
+ on base, 0.1 ms on main already); a terminal's cells
+ (sb-64k to sb-8m) 0.87 to 1.01. The A/A elsewhere 0.94 to 1.05.
+ - live tty keystroke to present (base's `pardes-perf-live --live`,
+ present p50 / core p50): key-down 1.01 / 0.99, key-right 1.01 / 0.95,
+ edit-char 0.97 / 0.98, page-down 0.99 / 0.92, wheel 1.08 / 1.04 (A/A
+ 1.05 to 1.16); a 1M-line open 1.06 (A/A 0.92); `yes | head -c 32M`
+ 1.22 (A/A 1.13), its runs 4.1 to 7.3 s on this line and 5.4 to 6.7 on
+ base, as main's 7.0 to 7.3 beside base's 6.0 to 7.3 were in the final
+ gate. A session's own path is untouched by this change: a session with
+ no far client hashes and compresses nothing, and a near one is sent
+ its dump as before.
+
+Found on the way:
+- **PARDES_REPLICA_CHECK keeps a session's units coming.** With two clients
+ checking, an idle session sends a unit about every 300 ms on main too
+ (`h/idle.py`: 27 checks in 8 s, near or far, main's build alike): a
+ check wakes the session's pump, whose unit makes the client's state
+ "changed", and the client asks again once quiet. Only the test knob
+ does it; a harness that expects a divergence to wait for a key it sends
+ finds the client joined again before (the measurements above wait for
+ the first link's stats line instead).
+- **A rejoin's "joined again" line can be gone before it is drawn**: it
+ lasts until the next key, and keys typed during a long rejoin (a 100 MB
+ file's) come first. As designed; the harnesses do not look for it.
+- **Harness traps.** A terminal's frame request split across two reads
+ went unanswered (test/daemon.py's `Term` now keeps the last bytes of a
+ read); keys written right after a first frame raced the terminal's
+ answer to it, an answer behind an Esc in one read being an Alt chord
+ (the far test waits 0.3 s, and writes the Esc on its own); a `pkill -f`
+ whose pattern was in the shell's own argv killed the shell.
+- The core-test run of the suites had one failure, `9p_io`'s "a change
+ waits while the editor is out mid-step", with the box measurements and
+ builds running; alone three times, and the whole core-test again, it
+ passed.
+
+Not done, and why:
+- **A first join pays the list's round trip** (207 ms on the box): the
+ wanted blobs wait for the client's answer, and the state (113 KB before
+ zstd) no longer hides it. A client with nothing cached could say so in
+ its `far` word and be sent everything at once; it is a first join on a
+ new machine only.
+- **A blob is hashed again at every far join** on the session's writer
+ thread (a 254 MB dump: 70 ms there). The hashes could be kept beside
+ the blob words.
+- **What a client sends is not compressed** (its keys, and a paste): with
+ `-C` gone, a megabyte's paste crosses the box's link raw.
+- **A client killed outright keeps on disk only what it was sent**: what
+ it made since (its edits' texts and undo) is written at its end, a
+ hangup's included, but not on a kill -9; the next join gets those again,
+ in zstd.
+- **The cache is per user, unencrypted, 0700**: a blob of a file the user
+ could read elsewhere, kept until evicted or the cache is cleared.