summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/next-store.md128
1 files changed, 128 insertions, 0 deletions
diff --git a/docs/next-store.md b/docs/next-store.md
new file mode 100644
index 00000000..894ea2f5
--- /dev/null
+++ b/docs/next-store.md
@@ -0,0 +1,128 @@
+# The store: bounded state shared by every pardes window
+
+> We should have some persistent state that is bounded (a houndred or so
+> MiB at most) stored on the fs and shared among all pardes windows. It will
+> store stuff for like the recent files picker. We should also have a panes
+> picker and this should include panes from the current session that were
+> deleted/closed.
+
+## What was there
+
+`recent.zig` kept 200 files in `$XDG_STATE_HOME/pardes/recent`, one line a
+file. Each process read it once at start and, at the end of any step that
+changed it, wrote the whole of its own list back (`saveRecent`: truncate,
+write). With two windows that is last-writer-wins: window A opens `a.zig`,
+window B opens `b.zig`, whichever saves last erases the other's entry, and
+neither ever sees the other's files until it restarts. A crash between the
+truncate and the write leaves an empty or half file (the reader passed bad
+lines over, so it never stopped pardes starting, but the list was gone).
+Closed panes were nowhere: a closed file's dot was its Recent entry, a
+closed scratch, terminal or +Errors was gone.
+
+## The format (argued first; the code follows it)
+
+One file, `$XDG_STATE_HOME/pardes/store`, an append-only log of framed
+records, plain enough to `cat`:
+
+```
+@<kind> <len> <crc32, 8 hex>\n<body: len bytes>\n
+```
+
+The first record is `@h` with the body `pardes-store 1`; a file that does
+not start with it is not a store (garbage, an older format) and reads as
+empty. The kinds:
+
+| kind | body | meaning |
+|---|---|---|
+| `h` | `pardes-store 1` | the header, a version |
+| `r` | `<line> <col> <aline> <acol> <path>` | a recent file and its place: the old `recent` line, unchanged |
+| `c` | `<session> <pid> <serial> <column> <gainer> <below> <owner> <rows>\t<name>\n<dump>` | a pane closed in session `<session>`, with where it was, and the pane itself as a one-pane Dump |
+| `x` | `<session> <serial>` | that closed pane was reopened: no longer listed |
+
+Folding the log in order gives the state: a path's latest `r` wins and its
+order is the order of last writes (so every window folds the same list); a
+`c` not followed by its `x` is a closed pane.
+
+**Why a closed pane is a Dump.** `dump.zig` already turns any pane into
+data and back: a file's dot and scroll (its text only when unsaved), a
+scratch's or an output pane's text with its origin and rows' metadata, a
+terminal's scrollback (a shell comes back running under its old history, a
+command pane finished, as Restore does), a PDF's page, an image. Reusing it
+means Panes can bring back every kind of pane with the same fidelity as
+Restore, through the same validation (`dump.readZon` checks a record before
+anything is built from it), and nothing new to keep in step when a pane
+grows a field.
+
+**Why on disk and not in memory**, since only this session reads its own
+closed panes: their text is what makes them big (a terminal's history, a
+long +Errors, a discarded unsaved buffer), and the store is where the bound
+lives; in memory each window keeps only an index (serial, name, place,
+offset). Restore keeps them (same process, same session). A session that
+crashed leaves its records for the next compaction to drop. Position, not a
+verdict: an in-memory list would be simpler, and I would accept that
+pushback; the cap argument is what tipped it.
+
+**Bounds.** The file never passes `max_bytes` (100 MiB): an append that
+would compacts first. A compaction keeps the newest live records within
+half of that, so it is not redone at once. A record is at most `max_record`
+(8 MiB): a terminal whose dump is bigger is kept without its history (it
+comes back as a fresh shell in its directory), any other pane that big is
+not kept. Recent keeps `recent.cap` (200) files; a session keeps its 64
+newest closed panes.
+
+## The sharing
+
+- **Writers** take `flock(LOCK_EX | LOCK_NB)` on `store.lock`, a file of
+ its own (the store's inode changes at each compaction, so locking it
+ would let two writers hold locks on two different files). Busy, the
+ records wait in memory for the next step: a window never blocks a frame
+ on another's compaction. At exit the lock is waited for.
+- Under the lock a writer first reads what others appended since it last
+ read (so its list's order is the log's), then writes its records with one
+ positional write at the end of the file, then notes the new end.
+- **Readers** take no lock. On use (Recent, Panes, `/recent`, before an
+ append) one `stat` of the path: the same inode and size as last read,
+ nothing to do; the same inode and longer, read the tail; another inode
+ (a compaction renamed a new file over it) or shorter, read it all again.
+- **Torn and garbage records.** A record whose length runs past the end of
+ the file is being written (or its writer died): reading stops before it
+ and tries again next time. A complete record with a bad CRC or a bad
+ header is garbage: the reader resynchronises at the next `\n@` that
+ starts a valid record, and marks the file for compaction, which the next
+ writer does under the lock (rewriting only valid records). A missing,
+ unreadable or foreign file is an empty store: pardes always starts.
+- **Compaction**, under the lock: fold the whole file, write the live
+ records (Recent, oldest first so the fold gives the same order; closed
+ panes of sessions whose pid is alive, copied byte for byte) to
+ `store.new`, `fsync`, rename it over `store`. Readers see the new inode.
+ Triggered when the file is past twice its live size plus 1 MiB, when it
+ holds garbage, or when an append would pass the cap.
+- **Startup reads headers, not bodies**: a new session has no closed panes
+ of its own, so every `c` body is skipped by its length; only `r` bodies
+ (a line each) are read.
+
+### Rejected
+
+1. **Rewrite the whole file under a lock** (today's scheme, made safe with
+ read-merge-write and a rename): correct, but every open or close costs
+ the whole file, and with closed panes' text in it that is megabytes per
+ Del.
+2. **SQLite**: one more C library to vendor and hook into Zig's allocators
+ (the shared-deps policy), WAL needs shared memory, for three record
+ kinds and a fold.
+3. **A file per entry** (maildir): atomic by rename and lock-free, but the
+ bound means listing and statting a directory of thousands of files on
+ every use, two windows race deleting the same oldest, and "what changed"
+ is not one `stat`.
+4. **A file per window, merged by readers**: no lock, but a merge needs
+ wall clocks across processes for order, every read opens N files, and
+ dead windows' files need a reaper that races the living.
+5. **A fixed-size mmap'd ring**: always 100 MiB on disk, cross-process
+ writes into shared pages need atomics, a writer that dies mid-record
+ corrupts in place, and a format change is a migration.
+6. **A watcher thread / inotify** (spec says no) or **a daemon** (none
+ exists; pardes must start without one): a `stat` on use is cheaper than
+ either and as fresh as anything that is looked at.
+
+The old `recent` file is read once into a store that does not exist yet,
+so nobody's list is lost; it is left where it is.