diff options
| author | Gabriel Schneider <[email protected]> | 2026-11-05 10:46:54 -0300 |
|---|---|---|
| committer | Gabriel Schneider <[email protected]> | 2026-11-05 12:41:12 -0300 |
| commit | c0b2e5d9194ce35b9fbb58d8d6e18de66cb209a8 (patch) | |
| tree | fa4656b6d5e4203b8c52516d6695b8c7677de41d /docs | |
| parent | 60a891ce8127c87b55aed1ff71e435cb7fbd28fc (diff) | |
| download | pardes-c0b2e5d9194ce35b9fbb58d8d6e18de66cb209a8.tar.gz pardes-c0b2e5d9194ce35b9fbb58d8d6e18de66cb209a8.zip | |
store: Recent is a file of 200 lines under flock, written whole in place at the end of a step that changed it and read back under a session's own changes when another session wrote it since, and the panes a session closes are kept in its memory, the newest 64 within 64 MiB; the append-only CRC log with its resync and compaction, and the closed panes' self-describing dumps in it (only the session that closed one ever listed it), go; Options.store_path is recent_path ($XDG_STATE_HOME/pardes/recent), and the old store is read once while it is the newer of the two; tests: the store's four rewritten through the same windows (the 200 newest, a foreign file, a closed pane's way back from memory), the old store read once, 64 closed panes kept, recent.zig's merge; the process cases read Recent's file; docs/next-store.md is the decision
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Y9MNAtZenFv4weoMD3cge
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/next-store.md | 276 | ||||
| -rw-r--r-- | docs/typ/reference.typ | 2 |
2 files changed, 51 insertions, 227 deletions
diff --git a/docs/next-store.md b/docs/next-store.md index 91fb340c..89997ee2 100644 --- a/docs/next-store.md +++ b/docs/next-store.md @@ -1,4 +1,4 @@ -# The store: bounded state shared by every pardes window +# The store: Recent's file, and this session's closed panes > We should have some persistent state that is bounded (a houndred or so > MiB at most) stored on the fs and shared among all pardes windows. It will @@ -6,237 +6,61 @@ > picker and this should include panes from the current session that were > deleted/closed. -## What was there +## The decision (2026-11-05) -`recent.zig` kept 200 files in `$XDG_STATE_HOME/pardes/recent`, one line a -file. Each process read it once at start and, at the end of any step that -changed it, wrote the whole of its own list back (`saveRecent`: truncate, -write). With two windows that is last-writer-wins: window A opens `a.zig`, -window B opens `b.zig`, whichever saves last erases the other's entry, and -neither ever sees the other's files until it restarts. A crash between the -truncate and the write leaves an empty or half file (the reader passed bad -lines over, so it never stopped pardes starting, but the list was gone). -Closed panes were nowhere: a closed file's dot was its Recent entry, a -closed scratch, terminal or +Errors was gone. +**Recent is a file of at most 200 lines**, `$XDG_STATE_HOME/pardes/recent` +(else `~/.local/state/pardes/recent`), a line a file, the newest first: +`<line> <col> <aline> <acol> <path>`. Every session reads it at start, and +again on use (Recent, `/recent`) when a stat of it says another session +wrote it since. A step that changed the list writes it at its end: `flock` +on the file (held by another session: the next step tries again, so a +frame never waits for one; at exit the lock is waited for), the file read +again with this session's changes put over it (`recent.List.merge`) when +another session wrote it since, and the whole list written over it in +place. A line that is no entry is passed over: a missing, unreadable or +foreign file is an empty list, a reader meeting a write half done reads it +again once its stat moves on, a line a crash tore is one entry fewer, and +pardes always starts. A rename of a new file over it (atomic) cost twice +the write in Debug for nothing the lenient read does not already give. -## The format (argued first; the code follows it) +**Closed panes are this session's, in memory**: the newest 64, within +64 MiB together, each a one-pane dump (`dump.savePane`, at most 8 MiB; a +terminal past it is kept without its history, any other pane is not kept) +and where it was. Panes lists them after the open panes, `(closed)`; +Enter, or a Look at `@pN`, brings one back beside the pane that took its +rows. Restore keeps them; the session's end drops them. -One file, `$XDG_STATE_HOME/pardes/store`, an append-only log of framed -records, plain enough to `cat`: +## What went, and why -``` -@<kind> <len> <crc32, 8 hex>\n<body: len bytes>\n -``` +The store this replaced (2026-10-24, Store.zig 1,135 lines) was one +append-only log of CRC-framed records (`@h`, `@r`, `@c`, `@x`) under +flock, with a resync past torn or garbage records, a compaction by rename, +a 100 MiB cap, and each closed pane a self-describing dump in the file, +read back at its offset. Closed panes were only ever listed by the session +that closed them, so only the file and its bound were shared: an +in-memory list is the same behaviour for far less code and no format (the +position the store's own note held least firmly). Recent needs no log: 200 +lines rewritten whole under the lock is a few kilobytes a step that opens +or closes a file. -The first record is `@h` with the body `pardes-store 1`; a file that does -not start with it is not a store (garbage, an older format) and reads as -empty. The kinds: +What a user loses: the closed panes of other sessions, which the log kept +on disk until a compaction and nothing ever listed. -| kind | body | meaning | -|---|---|---| -| `h` | `pardes-store 1` | the header, a version | -| `r` | `<line> <col> <aline> <acol> <path>` | a recent file and its place: the old `recent` line, unchanged | -| `c` | `<session> <pid> <serial> <column> <gainer> <below> <owner> <rows>\t<name>\n<dump>` | a pane closed in session `<session>`, with where it was, and the pane itself as a one-pane Dump | -| `x` | `<session> <serial>` | that closed pane was reopened: no longer listed | - -Folding the log in order gives the state: a path's latest `r` wins and its -order is the order of last writes (so every window folds the same list); a -`c` not followed by its `x` is a closed pane. - -**Why a closed pane is a Dump.** `dump.zig` already turns any pane into -data and back: a file's dot and scroll (its text only when unsaved), a -scratch's or an output pane's text with its origin and rows' metadata, a -terminal's scrollback (a shell comes back running under its old history, a -command pane finished, as Restore does), a PDF's page, an image. Reusing it -means Panes can bring back every kind of pane with the same fidelity as -Restore, through the same validation (`dump.readZon` checks a record before -anything is built from it), and nothing new to keep in step when a pane -grows a field. - -**Why on disk and not in memory**, since only this session reads its own -closed panes: their text is what makes them big (a terminal's history, a -long +Errors, a discarded unsaved buffer), and the store is where the bound -lives; in memory each window keeps only an index (serial, name, place, -offset). Restore keeps them (same process, same session). A session that -crashed leaves its records for the next compaction to drop. Position, not a -verdict: an in-memory list would be simpler, and I would accept that -pushback; the cap argument is what tipped it. - -**Bounds.** The file never passes `max_bytes` (100 MiB): an append that -would compacts first. A compaction keeps the newest live records within -half of that, so it is not redone at once. A record is at most `max_record` -(8 MiB): a terminal whose dump is bigger is kept without its history (it -comes back as a fresh shell in its directory), any other pane that big is -not kept. Recent keeps `recent.cap` (200) files; a session keeps its 64 -newest closed panes. - -## The sharing - -- **Writers** take `flock(LOCK_EX | LOCK_NB)` on `store.lock`, a file of - its own (the store's inode changes at each compaction, so locking it - would let two writers hold locks on two different files). Busy, the - records wait in memory for the next step: a window never blocks a frame - on another's compaction. At exit the lock is waited for. -- Under the lock a writer first reads what others appended since it last - read (so its list's order is the log's), then writes its records with one - positional write at the end of the file, then notes the new end. -- **Readers** take no lock. On use (Recent, Panes, `/recent`, before an - append) one `stat` of the path: the same inode and size as last read, - nothing to do; the same inode and longer, read the tail; another inode - (a compaction renamed a new file over it) or shorter, read it all again. -- **Torn and garbage records.** A record whose length runs past the end of - the file is being written (or its writer died): reading stops before it - and tries again next time. A complete record with a bad CRC or a bad - header is garbage: the reader resynchronises at the next `\n@` that - starts a valid record, and marks the file for compaction, which the next - writer does under the lock (rewriting only valid records). A missing, - unreadable or foreign file is an empty store: pardes always starts. -- **Compaction**, under the lock: fold the whole file, write the live - records (Recent, oldest first so the fold gives the same order; closed - panes of sessions whose pid is alive, copied byte for byte) to - `store.new`, `fsync`, rename it over `store`. Readers see the new inode. - Triggered when the file is past twice its live size plus 1 MiB, when it - holds garbage, or when an append would pass the cap. -- **Startup reads headers, not bodies**: a new session has no closed panes - of its own, so every `c` body is skipped by its length; only `r` bodies - (a line each) are read. - -### Rejected - -1. **Rewrite the whole file under a lock** (today's scheme, made safe with - read-merge-write and a rename): correct, but every open or close costs - the whole file, and with closed panes' text in it that is megabytes per - Del. -2. **SQLite**: one more C library to vendor and hook into Zig's allocators - (the shared-deps policy), WAL needs shared memory, for three record - kinds and a fold. -3. **A file per entry** (maildir): atomic by rename and lock-free, but the - bound means listing and statting a directory of thousands of files on - every use, two windows race deleting the same oldest, and "what changed" - is not one `stat`. -4. **A file per window, merged by readers**: no lock, but a merge needs - wall clocks across processes for order, every read opens N files, and - dead windows' files need a reaper that races the living. -5. **A fixed-size mmap'd ring**: always 100 MiB on disk, cross-process - writes into shared pages need atomics, a writer that dies mid-record - corrupts in place, and a format change is a migration. -6. **A watcher thread / inotify** (spec says no) or **a daemon** (none - exists; pardes must start without one): a `stat` on use is cheaper than - either and as fresh as anything that is looked at. - -The old `recent` file was read once into a store that did not exist yet, -so nobody's list was lost; with the store in use, that import went -(2026-11-04). - -## As built - -| piece | where | -|---|---| -| the log: framing, reader, resync, fold, flush, compaction | `src/Store.zig` (`Pardes.store`) | -| Recent as records: `Entry.changed`, `List.note` (a record read), `changedFirst` | `src/recent.zig` | -| a pane to and from a Dump, one at a time | `dump.dumpPane`, `dump.restorePane` (lifted out of `dumpState` / `initDump`) | -| a closed pane kept: before its teardown, its rows' gainer after | `Pardes.removePane`, `Pardes.removeColumn` (`Store.keep`, `Store.close`) | -| back where it was | `Store.reopen` → `layout.giveBack` (rows from the gainer alone), `layout.attachFrom` for an attachment whose owner is open, else `exec.placeNew` | -| Panes: open panes, then this session's closed ones, `@p7 notes/a.md (closed)`, dimmer (the gutter's colour between the match marks) | `Query.startSource` / `makeNameRows` / `land` | -| Recent and `/recent` read the store on use | `Query.startSource`, `ninep/tree.zig` | -| the path | `main.storePath`: `$XDG_STATE_HOME/pardes/store` (`Options.store_path`, was `recent_path`) | - -Which closes are kept: every `removePane` / `removeColumn` of a pane the -user had (announced: a pane made and refused in one step never was), not -a query pane nor a +Preview. Exit and Restore keep nothing (Restore's new -core takes the old session's list, `Store.clone`). Brought back, a pane is -a new pane (a fresh serial, as Restore gives); its row goes (`x` record). - -A Look at `@p7` naming a pane this session closed brings it back instead of -the "no pane @p7" miss: the row's text is its address, as everywhere. That -is the only change to Look, in its miss branch. - -**The turn.** Every host call is made out of the turn (`Store.host(f, -args)`, one call each), so a store under a mount of this session's own -tree cannot deadlock, and nothing in the store or Recent changes while a -step is out. `/recent` (a 9P read) reads the store only when no step is -out; otherwise it answers the list as it was. - -## Measured (Debug, the `zz store timings` test, STORE_TIMES=<dir>) - -A store as a running session leaves it: 200 files and 90 closed panes of -1 MiB (94 MB). - -| what | cost | -|---|---| -| a session starting on it (`load`: headers only, every `c` body skipped) | 3.4 ms | -| a refresh with nothing new (one `stat`) | ~1 µs | -| a flush of one record (lock, read the tail, one write) | 36 µs | -| a compaction of the 94 MB file (CRC of every record, copy the live, rename) | 278 ms, to 51 MB (the newest within half the cap) | +The first session of a build with this reads the old `store` once more, +when it is newer than `recent`: its `@r` records' bodies are this file's +lines. The old file is left in place for an older pardes still running. ## Tests -`store: two windows share Recent...` (two cores on one file, a lock held by -a third party delays a write and loses nothing, a fresh window folds the -same list), `store: the cap holds...` (3000 files and 30 closed 20 KB -scratches against a 48 KiB / 256 KiB cap: never past it, compactions -happen, the newest 200 files and the newest closed pane survive), -`store: a garbage or torn file...` (a foreign file, a torn header, junk, -then a whole record; a half-written record waited for; mended by the next -writer), `store: a closed pane comes back...` (from the file, not memory; -its rect, the gainer's and another column's as before the close; listed -`(closed)` in Panes, Enter brings it back, unlisted after a re-read; another -session lists none). `test/fs.py store_shared_between_windows`: two live -sessions over 9P. The related suites (dump, Restore, query, Find, Grep, -Del, Delcol, attach, collapse, jump, Edit, recent) pass. The tty build -compiles. - -Found, not mine: `fs.py recent_survives_restart` fails on the converge -baseline too, at its Recent step (the query pane's rows are relative -names now, the test looks for absolute ones). - -## Open questions, and my position on each - -1. **Closed panes on disk or in memory?** On disk (argued above). The - honest counter: only this session reads them, and a Del now costs a - dump and a write. Measured, a flush is 36 µs; the dump of a big - terminal is the cost (its history base64'd), capped at 8 MiB. -2. **Recent's cap stays 200**, now for every window together. Shared, it - fills faster; 1000 is cheap for the fuzzy scorer. I kept 200 to change - one thing at a time; I would raise it. -3. **"open" means open here.** A file open in another window reads - `(closed)` in this one's Recent (its place is the other window's last - close). Saying `open elsewhere` needs the store to know which windows - are alive and what they hold (a record per open, its pid): doable, - not done. -4. **Compaction is on the loop** (278 ms for 94 MB, Debug). It happens - when the file doubles past what was live, or reaches the cap, which a - window that closes many big terminals does. Off the loop it would need - the disk engine's worker and the lock held across frames; I would do - it only if it shows up. A store of Recent alone is ~20 KB to rewrite - (not timed separately). -5. **A crashed session's closed panes** are dropped at the next - compaction (its pid is gone). Keeping them a day for a "what was I - doing" word would be cheap: the records are there. -6. **A dirty file pane closed with the second Del** comes back with its - unsaved text (it is in its dump, by Restore's rules; not tested - here). That is a feature (undo a discard) - and a surprise (the discard was asked for twice). I kept it; say if a - discard should be final. - -## What I would push back on - -- "shared among all pardes windows" for the closed panes: they are this - session's by the spec, so the sharing is only of the file and its bound. - If the bound is the only reason, an in-memory list with a byte budget - is less code (no `c`/`x` records, no offsets); I chose the disk for the - memory and Restore, but this is the part I hold least firmly. -- The 100 MiB: nothing in Recent comes near it; only closed terminals' - history does. A smaller cap (16 MiB) would bound compaction to ~50 ms - and lose little. 100 MiB is what was asked, so it is the default. -- Reopening a closed pane makes a new serial: `@p7` in old text names a - pane that is now `@p12`. Restore does the same and logs `restored 7 12`; - the store could log the same, not done. - -## Rough - -- The dim tint of a closed Panes row is the gutter colour (`lineno`), not - checked by eye in every theme. -- A Delcol'd pane comes back where `placeNew` puts it (its column is gone). -- An attachment whose owner was closed with it comes back as a plain pane - next to wherever placement puts it (its owner's serial is new). +Store.zig: two windows share Recent (each reads the other's writes on use; +a lock held by a third party delays a write, never loses it); the file +holds the newest 200, newest first; a foreign file's Recent lines are read +and the next write mends it; the old store read once, its newest records +first; a session keeps its newest 64 closed panes; a closed pane comes back +where it was, its rows from the pane that took them, listed `(closed)` in +Panes and unlisted once back, and another session lists none. +recent.zig: a merge puts this session's changes over the file's list. +test/process/other_files.zig: a file opened and closed is in `/recent` +closed and still is in the next session; two sessions share Recent, and a +Del'd pane comes back from `@p` only where it was. test/behaviour/ +other_files.zig: `/recent` and Recent's rows. diff --git a/docs/typ/reference.typ b/docs/typ/reference.typ index a7db9e2d..de886c87 100644 --- a/docs/typ/reference.typ +++ b/docs/typ/reference.typ @@ -942,7 +942,7 @@ at 200, ending in `…`. Control characters become spaces. pane from the root). - #file("/recent"): up to 200 files, PDFs and images, most recent first, `open <path>` or `closed <path>` (open in this window or not), every - window's: kept in `$XDG_STATE_HOME/pardes/store`, shared by every pardes + window's: kept in `$XDG_STATE_HOME/pardes/recent`, shared by every pardes process, each reading what the others wrote when it is next read. #word("Recent") lists them in a query pane (line 1 `Recent`, the rest narrowing them by a fuzzy match), an open one at its dot now and a |
