summaryrefslogtreecommitdiff
path: root/docs/next-store.md
blob: 894ea2f520cb86b21e90a61c485d274a5b9f3e03 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
# The store: bounded state shared by every pardes window

> We should have some persistent state that is bounded (a houndred or so
> MiB at most) stored on the fs and shared among all pardes windows. It will
> store stuff for like the recent files picker. We should also have a panes
> picker and this should include panes from the current session that were
> deleted/closed.

## What was there

`recent.zig` kept 200 files in `$XDG_STATE_HOME/pardes/recent`, one line a
file. Each process read it once at start and, at the end of any step that
changed it, wrote the whole of its own list back (`saveRecent`: truncate,
write). With two windows that is last-writer-wins: window A opens `a.zig`,
window B opens `b.zig`, whichever saves last erases the other's entry, and
neither ever sees the other's files until it restarts. A crash between the
truncate and the write leaves an empty or half file (the reader passed bad
lines over, so it never stopped pardes starting, but the list was gone).
Closed panes were nowhere: a closed file's dot was its Recent entry, a
closed scratch, terminal or +Errors was gone.

## The format (argued first; the code follows it)

One file, `$XDG_STATE_HOME/pardes/store`, an append-only log of framed
records, plain enough to `cat`:

```
@<kind> <len> <crc32, 8 hex>\n<body: len bytes>\n
```

The first record is `@h` with the body `pardes-store 1`; a file that does
not start with it is not a store (garbage, an older format) and reads as
empty. The kinds:

| kind | body | meaning |
|---|---|---|
| `h` | `pardes-store 1` | the header, a version |
| `r` | `<line> <col> <aline> <acol> <path>` | a recent file and its place: the old `recent` line, unchanged |
| `c` | `<session> <pid> <serial> <column> <gainer> <below> <owner> <rows>\t<name>\n<dump>` | a pane closed in session `<session>`, with where it was, and the pane itself as a one-pane Dump |
| `x` | `<session> <serial>` | that closed pane was reopened: no longer listed |

Folding the log in order gives the state: a path's latest `r` wins and its
order is the order of last writes (so every window folds the same list); a
`c` not followed by its `x` is a closed pane.

**Why a closed pane is a Dump.** `dump.zig` already turns any pane into
data and back: a file's dot and scroll (its text only when unsaved), a
scratch's or an output pane's text with its origin and rows' metadata, a
terminal's scrollback (a shell comes back running under its old history, a
command pane finished, as Restore does), a PDF's page, an image. Reusing it
means Panes can bring back every kind of pane with the same fidelity as
Restore, through the same validation (`dump.readZon` checks a record before
anything is built from it), and nothing new to keep in step when a pane
grows a field.

**Why on disk and not in memory**, since only this session reads its own
closed panes: their text is what makes them big (a terminal's history, a
long +Errors, a discarded unsaved buffer), and the store is where the bound
lives; in memory each window keeps only an index (serial, name, place,
offset). Restore keeps them (same process, same session). A session that
crashed leaves its records for the next compaction to drop. Position, not a
verdict: an in-memory list would be simpler, and I would accept that
pushback; the cap argument is what tipped it.

**Bounds.** The file never passes `max_bytes` (100 MiB): an append that
would compacts first. A compaction keeps the newest live records within
half of that, so it is not redone at once. A record is at most `max_record`
(8 MiB): a terminal whose dump is bigger is kept without its history (it
comes back as a fresh shell in its directory), any other pane that big is
not kept. Recent keeps `recent.cap` (200) files; a session keeps its 64
newest closed panes.

## The sharing

- **Writers** take `flock(LOCK_EX | LOCK_NB)` on `store.lock`, a file of
  its own (the store's inode changes at each compaction, so locking it
  would let two writers hold locks on two different files). Busy, the
  records wait in memory for the next step: a window never blocks a frame
  on another's compaction. At exit the lock is waited for.
- Under the lock a writer first reads what others appended since it last
  read (so its list's order is the log's), then writes its records with one
  positional write at the end of the file, then notes the new end.
- **Readers** take no lock. On use (Recent, Panes, `/recent`, before an
  append) one `stat` of the path: the same inode and size as last read,
  nothing to do; the same inode and longer, read the tail; another inode
  (a compaction renamed a new file over it) or shorter, read it all again.
- **Torn and garbage records.** A record whose length runs past the end of
  the file is being written (or its writer died): reading stops before it
  and tries again next time. A complete record with a bad CRC or a bad
  header is garbage: the reader resynchronises at the next `\n@` that
  starts a valid record, and marks the file for compaction, which the next
  writer does under the lock (rewriting only valid records). A missing,
  unreadable or foreign file is an empty store: pardes always starts.
- **Compaction**, under the lock: fold the whole file, write the live
  records (Recent, oldest first so the fold gives the same order; closed
  panes of sessions whose pid is alive, copied byte for byte) to
  `store.new`, `fsync`, rename it over `store`. Readers see the new inode.
  Triggered when the file is past twice its live size plus 1 MiB, when it
  holds garbage, or when an append would pass the cap.
- **Startup reads headers, not bodies**: a new session has no closed panes
  of its own, so every `c` body is skipped by its length; only `r` bodies
  (a line each) are read.

### Rejected

1. **Rewrite the whole file under a lock** (today's scheme, made safe with
   read-merge-write and a rename): correct, but every open or close costs
   the whole file, and with closed panes' text in it that is megabytes per
   Del.
2. **SQLite**: one more C library to vendor and hook into Zig's allocators
   (the shared-deps policy), WAL needs shared memory, for three record
   kinds and a fold.
3. **A file per entry** (maildir): atomic by rename and lock-free, but the
   bound means listing and statting a directory of thousands of files on
   every use, two windows race deleting the same oldest, and "what changed"
   is not one `stat`.
4. **A file per window, merged by readers**: no lock, but a merge needs
   wall clocks across processes for order, every read opens N files, and
   dead windows' files need a reaper that races the living.
5. **A fixed-size mmap'd ring**: always 100 MiB on disk, cross-process
   writes into shared pages need atomics, a writer that dies mid-record
   corrupts in place, and a format change is a migration.
6. **A watcher thread / inotify** (spec says no) or **a daemon** (none
   exists; pardes must start without one): a `stat` on use is cheaper than
   either and as fresh as anything that is looked at.

The old `recent` file is read once into a store that does not exist yet,
so nobody's list is lost; it is left where it is.