summaryrefslogtreecommitdiff
path: root/docs/next-store.md
diff options
context:
space:
mode:
authorGabriel Schneider <[email protected]>2026-11-05 10:46:54 -0300
committerGabriel Schneider <[email protected]>2026-11-05 12:41:12 -0300
commitc0b2e5d9194ce35b9fbb58d8d6e18de66cb209a8 (patch)
treefa4656b6d5e4203b8c52516d6695b8c7677de41d /docs/next-store.md
parent60a891ce8127c87b55aed1ff71e435cb7fbd28fc (diff)
downloadpardes-c0b2e5d9194ce35b9fbb58d8d6e18de66cb209a8.tar.gz
pardes-c0b2e5d9194ce35b9fbb58d8d6e18de66cb209a8.zip
store: Recent is a file of 200 lines under flock, written whole in place at the end of a step that changed it and read back under a session's own changes when another session wrote it since, and the panes a session closes are kept in its memory, the newest 64 within 64 MiB; the append-only CRC log with its resync and compaction, and the closed panes' self-describing dumps in it (only the session that closed one ever listed it), go; Options.store_path is recent_path ($XDG_STATE_HOME/pardes/recent), and the old store is read once while it is the newer of the two; tests: the store's four rewritten through the same windows (the 200 newest, a foreign file, a closed pane's way back from memory), the old store read once, 64 closed panes kept, recent.zig's merge; the process cases read Recent's file; docs/next-store.md is the decision
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_016Y9MNAtZenFv4weoMD3cge
Diffstat (limited to 'docs/next-store.md')
-rw-r--r--docs/next-store.md276
1 files changed, 50 insertions, 226 deletions
diff --git a/docs/next-store.md b/docs/next-store.md
index 91fb340c..89997ee2 100644
--- a/docs/next-store.md
+++ b/docs/next-store.md
@@ -1,4 +1,4 @@
-# The store: bounded state shared by every pardes window
+# The store: Recent's file, and this session's closed panes
> We should have some persistent state that is bounded (a houndred or so
> MiB at most) stored on the fs and shared among all pardes windows. It will
@@ -6,237 +6,61 @@
> picker and this should include panes from the current session that were
> deleted/closed.
-## What was there
+## The decision (2026-11-05)
-`recent.zig` kept 200 files in `$XDG_STATE_HOME/pardes/recent`, one line a
-file. Each process read it once at start and, at the end of any step that
-changed it, wrote the whole of its own list back (`saveRecent`: truncate,
-write). With two windows that is last-writer-wins: window A opens `a.zig`,
-window B opens `b.zig`, whichever saves last erases the other's entry, and
-neither ever sees the other's files until it restarts. A crash between the
-truncate and the write leaves an empty or half file (the reader passed bad
-lines over, so it never stopped pardes starting, but the list was gone).
-Closed panes were nowhere: a closed file's dot was its Recent entry, a
-closed scratch, terminal or +Errors was gone.
+**Recent is a file of at most 200 lines**, `$XDG_STATE_HOME/pardes/recent`
+(else `~/.local/state/pardes/recent`), a line a file, the newest first:
+`<line> <col> <aline> <acol> <path>`. Every session reads it at start, and
+again on use (Recent, `/recent`) when a stat of it says another session
+wrote it since. A step that changed the list writes it at its end: `flock`
+on the file (held by another session: the next step tries again, so a
+frame never waits for one; at exit the lock is waited for), the file read
+again with this session's changes put over it (`recent.List.merge`) when
+another session wrote it since, and the whole list written over it in
+place. A line that is no entry is passed over: a missing, unreadable or
+foreign file is an empty list, a reader meeting a write half done reads it
+again once its stat moves on, a line a crash tore is one entry fewer, and
+pardes always starts. A rename of a new file over it (atomic) cost twice
+the write in Debug for nothing the lenient read does not already give.
-## The format (argued first; the code follows it)
+**Closed panes are this session's, in memory**: the newest 64, within
+64 MiB together, each a one-pane dump (`dump.savePane`, at most 8 MiB; a
+terminal past it is kept without its history, any other pane is not kept)
+and where it was. Panes lists them after the open panes, `(closed)`;
+Enter, or a Look at `@pN`, brings one back beside the pane that took its
+rows. Restore keeps them; the session's end drops them.
-One file, `$XDG_STATE_HOME/pardes/store`, an append-only log of framed
-records, plain enough to `cat`:
+## What went, and why
-```
-@<kind> <len> <crc32, 8 hex>\n<body: len bytes>\n
-```
+The store this replaced (2026-10-24, Store.zig 1,135 lines) was one
+append-only log of CRC-framed records (`@h`, `@r`, `@c`, `@x`) under
+flock, with a resync past torn or garbage records, a compaction by rename,
+a 100 MiB cap, and each closed pane a self-describing dump in the file,
+read back at its offset. Closed panes were only ever listed by the session
+that closed them, so only the file and its bound were shared: an
+in-memory list is the same behaviour for far less code and no format (the
+position the store's own note held least firmly). Recent needs no log: 200
+lines rewritten whole under the lock is a few kilobytes a step that opens
+or closes a file.
-The first record is `@h` with the body `pardes-store 1`; a file that does
-not start with it is not a store (garbage, an older format) and reads as
-empty. The kinds:
+What a user loses: the closed panes of other sessions, which the log kept
+on disk until a compaction and nothing ever listed.
-| kind | body | meaning |
-|---|---|---|
-| `h` | `pardes-store 1` | the header, a version |
-| `r` | `<line> <col> <aline> <acol> <path>` | a recent file and its place: the old `recent` line, unchanged |
-| `c` | `<session> <pid> <serial> <column> <gainer> <below> <owner> <rows>\t<name>\n<dump>` | a pane closed in session `<session>`, with where it was, and the pane itself as a one-pane Dump |
-| `x` | `<session> <serial>` | that closed pane was reopened: no longer listed |
-
-Folding the log in order gives the state: a path's latest `r` wins and its
-order is the order of last writes (so every window folds the same list); a
-`c` not followed by its `x` is a closed pane.
-
-**Why a closed pane is a Dump.** `dump.zig` already turns any pane into
-data and back: a file's dot and scroll (its text only when unsaved), a
-scratch's or an output pane's text with its origin and rows' metadata, a
-terminal's scrollback (a shell comes back running under its old history, a
-command pane finished, as Restore does), a PDF's page, an image. Reusing it
-means Panes can bring back every kind of pane with the same fidelity as
-Restore, through the same validation (`dump.readZon` checks a record before
-anything is built from it), and nothing new to keep in step when a pane
-grows a field.
-
-**Why on disk and not in memory**, since only this session reads its own
-closed panes: their text is what makes them big (a terminal's history, a
-long +Errors, a discarded unsaved buffer), and the store is where the bound
-lives; in memory each window keeps only an index (serial, name, place,
-offset). Restore keeps them (same process, same session). A session that
-crashed leaves its records for the next compaction to drop. Position, not a
-verdict: an in-memory list would be simpler, and I would accept that
-pushback; the cap argument is what tipped it.
-
-**Bounds.** The file never passes `max_bytes` (100 MiB): an append that
-would compacts first. A compaction keeps the newest live records within
-half of that, so it is not redone at once. A record is at most `max_record`
-(8 MiB): a terminal whose dump is bigger is kept without its history (it
-comes back as a fresh shell in its directory), any other pane that big is
-not kept. Recent keeps `recent.cap` (200) files; a session keeps its 64
-newest closed panes.
-
-## The sharing
-
-- **Writers** take `flock(LOCK_EX | LOCK_NB)` on `store.lock`, a file of
- its own (the store's inode changes at each compaction, so locking it
- would let two writers hold locks on two different files). Busy, the
- records wait in memory for the next step: a window never blocks a frame
- on another's compaction. At exit the lock is waited for.
-- Under the lock a writer first reads what others appended since it last
- read (so its list's order is the log's), then writes its records with one
- positional write at the end of the file, then notes the new end.
-- **Readers** take no lock. On use (Recent, Panes, `/recent`, before an
- append) one `stat` of the path: the same inode and size as last read,
- nothing to do; the same inode and longer, read the tail; another inode
- (a compaction renamed a new file over it) or shorter, read it all again.
-- **Torn and garbage records.** A record whose length runs past the end of
- the file is being written (or its writer died): reading stops before it
- and tries again next time. A complete record with a bad CRC or a bad
- header is garbage: the reader resynchronises at the next `\n@` that
- starts a valid record, and marks the file for compaction, which the next
- writer does under the lock (rewriting only valid records). A missing,
- unreadable or foreign file is an empty store: pardes always starts.
-- **Compaction**, under the lock: fold the whole file, write the live
- records (Recent, oldest first so the fold gives the same order; closed
- panes of sessions whose pid is alive, copied byte for byte) to
- `store.new`, `fsync`, rename it over `store`. Readers see the new inode.
- Triggered when the file is past twice its live size plus 1 MiB, when it
- holds garbage, or when an append would pass the cap.
-- **Startup reads headers, not bodies**: a new session has no closed panes
- of its own, so every `c` body is skipped by its length; only `r` bodies
- (a line each) are read.
-
-### Rejected
-
-1. **Rewrite the whole file under a lock** (today's scheme, made safe with
- read-merge-write and a rename): correct, but every open or close costs
- the whole file, and with closed panes' text in it that is megabytes per
- Del.
-2. **SQLite**: one more C library to vendor and hook into Zig's allocators
- (the shared-deps policy), WAL needs shared memory, for three record
- kinds and a fold.
-3. **A file per entry** (maildir): atomic by rename and lock-free, but the
- bound means listing and statting a directory of thousands of files on
- every use, two windows race deleting the same oldest, and "what changed"
- is not one `stat`.
-4. **A file per window, merged by readers**: no lock, but a merge needs
- wall clocks across processes for order, every read opens N files, and
- dead windows' files need a reaper that races the living.
-5. **A fixed-size mmap'd ring**: always 100 MiB on disk, cross-process
- writes into shared pages need atomics, a writer that dies mid-record
- corrupts in place, and a format change is a migration.
-6. **A watcher thread / inotify** (spec says no) or **a daemon** (none
- exists; pardes must start without one): a `stat` on use is cheaper than
- either and as fresh as anything that is looked at.
-
-The old `recent` file was read once into a store that did not exist yet,
-so nobody's list was lost; with the store in use, that import went
-(2026-11-04).
-
-## As built
-
-| piece | where |
-|---|---|
-| the log: framing, reader, resync, fold, flush, compaction | `src/Store.zig` (`Pardes.store`) |
-| Recent as records: `Entry.changed`, `List.note` (a record read), `changedFirst` | `src/recent.zig` |
-| a pane to and from a Dump, one at a time | `dump.dumpPane`, `dump.restorePane` (lifted out of `dumpState` / `initDump`) |
-| a closed pane kept: before its teardown, its rows' gainer after | `Pardes.removePane`, `Pardes.removeColumn` (`Store.keep`, `Store.close`) |
-| back where it was | `Store.reopen` → `layout.giveBack` (rows from the gainer alone), `layout.attachFrom` for an attachment whose owner is open, else `exec.placeNew` |
-| Panes: open panes, then this session's closed ones, `@p7 notes/a.md (closed)`, dimmer (the gutter's colour between the match marks) | `Query.startSource` / `makeNameRows` / `land` |
-| Recent and `/recent` read the store on use | `Query.startSource`, `ninep/tree.zig` |
-| the path | `main.storePath`: `$XDG_STATE_HOME/pardes/store` (`Options.store_path`, was `recent_path`) |
-
-Which closes are kept: every `removePane` / `removeColumn` of a pane the
-user had (announced: a pane made and refused in one step never was), not
-a query pane nor a +Preview. Exit and Restore keep nothing (Restore's new
-core takes the old session's list, `Store.clone`). Brought back, a pane is
-a new pane (a fresh serial, as Restore gives); its row goes (`x` record).
-
-A Look at `@p7` naming a pane this session closed brings it back instead of
-the "no pane @p7" miss: the row's text is its address, as everywhere. That
-is the only change to Look, in its miss branch.
-
-**The turn.** Every host call is made out of the turn (`Store.host(f,
-args)`, one call each), so a store under a mount of this session's own
-tree cannot deadlock, and nothing in the store or Recent changes while a
-step is out. `/recent` (a 9P read) reads the store only when no step is
-out; otherwise it answers the list as it was.
-
-## Measured (Debug, the `zz store timings` test, STORE_TIMES=<dir>)
-
-A store as a running session leaves it: 200 files and 90 closed panes of
-1 MiB (94 MB).
-
-| what | cost |
-|---|---|
-| a session starting on it (`load`: headers only, every `c` body skipped) | 3.4 ms |
-| a refresh with nothing new (one `stat`) | ~1 µs |
-| a flush of one record (lock, read the tail, one write) | 36 µs |
-| a compaction of the 94 MB file (CRC of every record, copy the live, rename) | 278 ms, to 51 MB (the newest within half the cap) |
+The first session of a build with this reads the old `store` once more,
+when it is newer than `recent`: its `@r` records' bodies are this file's
+lines. The old file is left in place for an older pardes still running.
## Tests
-`store: two windows share Recent...` (two cores on one file, a lock held by
-a third party delays a write and loses nothing, a fresh window folds the
-same list), `store: the cap holds...` (3000 files and 30 closed 20 KB
-scratches against a 48 KiB / 256 KiB cap: never past it, compactions
-happen, the newest 200 files and the newest closed pane survive),
-`store: a garbage or torn file...` (a foreign file, a torn header, junk,
-then a whole record; a half-written record waited for; mended by the next
-writer), `store: a closed pane comes back...` (from the file, not memory;
-its rect, the gainer's and another column's as before the close; listed
-`(closed)` in Panes, Enter brings it back, unlisted after a re-read; another
-session lists none). `test/fs.py store_shared_between_windows`: two live
-sessions over 9P. The related suites (dump, Restore, query, Find, Grep,
-Del, Delcol, attach, collapse, jump, Edit, recent) pass. The tty build
-compiles.
-
-Found, not mine: `fs.py recent_survives_restart` fails on the converge
-baseline too, at its Recent step (the query pane's rows are relative
-names now, the test looks for absolute ones).
-
-## Open questions, and my position on each
-
-1. **Closed panes on disk or in memory?** On disk (argued above). The
- honest counter: only this session reads them, and a Del now costs a
- dump and a write. Measured, a flush is 36 µs; the dump of a big
- terminal is the cost (its history base64'd), capped at 8 MiB.
-2. **Recent's cap stays 200**, now for every window together. Shared, it
- fills faster; 1000 is cheap for the fuzzy scorer. I kept 200 to change
- one thing at a time; I would raise it.
-3. **"open" means open here.** A file open in another window reads
- `(closed)` in this one's Recent (its place is the other window's last
- close). Saying `open elsewhere` needs the store to know which windows
- are alive and what they hold (a record per open, its pid): doable,
- not done.
-4. **Compaction is on the loop** (278 ms for 94 MB, Debug). It happens
- when the file doubles past what was live, or reaches the cap, which a
- window that closes many big terminals does. Off the loop it would need
- the disk engine's worker and the lock held across frames; I would do
- it only if it shows up. A store of Recent alone is ~20 KB to rewrite
- (not timed separately).
-5. **A crashed session's closed panes** are dropped at the next
- compaction (its pid is gone). Keeping them a day for a "what was I
- doing" word would be cheap: the records are there.
-6. **A dirty file pane closed with the second Del** comes back with its
- unsaved text (it is in its dump, by Restore's rules; not tested
- here). That is a feature (undo a discard)
- and a surprise (the discard was asked for twice). I kept it; say if a
- discard should be final.
-
-## What I would push back on
-
-- "shared among all pardes windows" for the closed panes: they are this
- session's by the spec, so the sharing is only of the file and its bound.
- If the bound is the only reason, an in-memory list with a byte budget
- is less code (no `c`/`x` records, no offsets); I chose the disk for the
- memory and Restore, but this is the part I hold least firmly.
-- The 100 MiB: nothing in Recent comes near it; only closed terminals'
- history does. A smaller cap (16 MiB) would bound compaction to ~50 ms
- and lose little. 100 MiB is what was asked, so it is the default.
-- Reopening a closed pane makes a new serial: `@p7` in old text names a
- pane that is now `@p12`. Restore does the same and logs `restored 7 12`;
- the store could log the same, not done.
-
-## Rough
-
-- The dim tint of a closed Panes row is the gutter colour (`lineno`), not
- checked by eye in every theme.
-- A Delcol'd pane comes back where `placeNew` puts it (its column is gone).
-- An attachment whose owner was closed with it comes back as a plain pane
- next to wherever placement puts it (its owner's serial is new).
+Store.zig: two windows share Recent (each reads the other's writes on use;
+a lock held by a third party delays a write, never loses it); the file
+holds the newest 200, newest first; a foreign file's Recent lines are read
+and the next write mends it; the old store read once, its newest records
+first; a session keeps its newest 64 closed panes; a closed pane comes back
+where it was, its rows from the pane that took them, listed `(closed)` in
+Panes and unlisted once back, and another session lists none.
+recent.zig: a merge puts this session's changes over the file's list.
+test/process/other_files.zig: a file opened and closed is in `/recent`
+closed and still is in the next session; two sessions share Recent, and a
+Del'd pane comes back from `@p` only where it was. test/behaviour/
+other_files.zig: `/recent` and Recent's rows.