diff options
Diffstat (limited to 'docs/next-store.md')
| -rw-r--r-- | docs/next-store.md | 128 |
1 files changed, 128 insertions, 0 deletions
diff --git a/docs/next-store.md b/docs/next-store.md new file mode 100644 index 00000000..894ea2f5 --- /dev/null +++ b/docs/next-store.md @@ -0,0 +1,128 @@ +# The store: bounded state shared by every pardes window + +> We should have some persistent state that is bounded (a houndred or so +> MiB at most) stored on the fs and shared among all pardes windows. It will +> store stuff for like the recent files picker. We should also have a panes +> picker and this should include panes from the current session that were +> deleted/closed. + +## What was there + +`recent.zig` kept 200 files in `$XDG_STATE_HOME/pardes/recent`, one line a +file. Each process read it once at start and, at the end of any step that +changed it, wrote the whole of its own list back (`saveRecent`: truncate, +write). With two windows that is last-writer-wins: window A opens `a.zig`, +window B opens `b.zig`, whichever saves last erases the other's entry, and +neither ever sees the other's files until it restarts. A crash between the +truncate and the write leaves an empty or half file (the reader passed bad +lines over, so it never stopped pardes starting, but the list was gone). +Closed panes were nowhere: a closed file's dot was its Recent entry, a +closed scratch, terminal or +Errors was gone. + +## The format (argued first; the code follows it) + +One file, `$XDG_STATE_HOME/pardes/store`, an append-only log of framed +records, plain enough to `cat`: + +``` +@<kind> <len> <crc32, 8 hex>\n<body: len bytes>\n +``` + +The first record is `@h` with the body `pardes-store 1`; a file that does +not start with it is not a store (garbage, an older format) and reads as +empty. The kinds: + +| kind | body | meaning | +|---|---|---| +| `h` | `pardes-store 1` | the header, a version | +| `r` | `<line> <col> <aline> <acol> <path>` | a recent file and its place: the old `recent` line, unchanged | +| `c` | `<session> <pid> <serial> <column> <gainer> <below> <owner> <rows>\t<name>\n<dump>` | a pane closed in session `<session>`, with where it was, and the pane itself as a one-pane Dump | +| `x` | `<session> <serial>` | that closed pane was reopened: no longer listed | + +Folding the log in order gives the state: a path's latest `r` wins and its +order is the order of last writes (so every window folds the same list); a +`c` not followed by its `x` is a closed pane. + +**Why a closed pane is a Dump.** `dump.zig` already turns any pane into +data and back: a file's dot and scroll (its text only when unsaved), a +scratch's or an output pane's text with its origin and rows' metadata, a +terminal's scrollback (a shell comes back running under its old history, a +command pane finished, as Restore does), a PDF's page, an image. Reusing it +means Panes can bring back every kind of pane with the same fidelity as +Restore, through the same validation (`dump.readZon` checks a record before +anything is built from it), and nothing new to keep in step when a pane +grows a field. + +**Why on disk and not in memory**, since only this session reads its own +closed panes: their text is what makes them big (a terminal's history, a +long +Errors, a discarded unsaved buffer), and the store is where the bound +lives; in memory each window keeps only an index (serial, name, place, +offset). Restore keeps them (same process, same session). A session that +crashed leaves its records for the next compaction to drop. Position, not a +verdict: an in-memory list would be simpler, and I would accept that +pushback; the cap argument is what tipped it. + +**Bounds.** The file never passes `max_bytes` (100 MiB): an append that +would compacts first. A compaction keeps the newest live records within +half of that, so it is not redone at once. A record is at most `max_record` +(8 MiB): a terminal whose dump is bigger is kept without its history (it +comes back as a fresh shell in its directory), any other pane that big is +not kept. Recent keeps `recent.cap` (200) files; a session keeps its 64 +newest closed panes. + +## The sharing + +- **Writers** take `flock(LOCK_EX | LOCK_NB)` on `store.lock`, a file of + its own (the store's inode changes at each compaction, so locking it + would let two writers hold locks on two different files). Busy, the + records wait in memory for the next step: a window never blocks a frame + on another's compaction. At exit the lock is waited for. +- Under the lock a writer first reads what others appended since it last + read (so its list's order is the log's), then writes its records with one + positional write at the end of the file, then notes the new end. +- **Readers** take no lock. On use (Recent, Panes, `/recent`, before an + append) one `stat` of the path: the same inode and size as last read, + nothing to do; the same inode and longer, read the tail; another inode + (a compaction renamed a new file over it) or shorter, read it all again. +- **Torn and garbage records.** A record whose length runs past the end of + the file is being written (or its writer died): reading stops before it + and tries again next time. A complete record with a bad CRC or a bad + header is garbage: the reader resynchronises at the next `\n@` that + starts a valid record, and marks the file for compaction, which the next + writer does under the lock (rewriting only valid records). A missing, + unreadable or foreign file is an empty store: pardes always starts. +- **Compaction**, under the lock: fold the whole file, write the live + records (Recent, oldest first so the fold gives the same order; closed + panes of sessions whose pid is alive, copied byte for byte) to + `store.new`, `fsync`, rename it over `store`. Readers see the new inode. + Triggered when the file is past twice its live size plus 1 MiB, when it + holds garbage, or when an append would pass the cap. +- **Startup reads headers, not bodies**: a new session has no closed panes + of its own, so every `c` body is skipped by its length; only `r` bodies + (a line each) are read. + +### Rejected + +1. **Rewrite the whole file under a lock** (today's scheme, made safe with + read-merge-write and a rename): correct, but every open or close costs + the whole file, and with closed panes' text in it that is megabytes per + Del. +2. **SQLite**: one more C library to vendor and hook into Zig's allocators + (the shared-deps policy), WAL needs shared memory, for three record + kinds and a fold. +3. **A file per entry** (maildir): atomic by rename and lock-free, but the + bound means listing and statting a directory of thousands of files on + every use, two windows race deleting the same oldest, and "what changed" + is not one `stat`. +4. **A file per window, merged by readers**: no lock, but a merge needs + wall clocks across processes for order, every read opens N files, and + dead windows' files need a reaper that races the living. +5. **A fixed-size mmap'd ring**: always 100 MiB on disk, cross-process + writes into shared pages need atomics, a writer that dies mid-record + corrupts in place, and a format change is a migration. +6. **A watcher thread / inotify** (spec says no) or **a daemon** (none + exists; pardes must start without one): a `stat` on use is cheaper than + either and as fresh as anything that is looked at. + +The old `recent` file is read once into a store that does not exist yet, +so nobody's list is lost; it is left where it is. |
