Building gathers the platforms, options, tests, release gates and how the core, threads, detached sessions and 9P fit, from the old design notes checked against the code

Co-Authored-By: Claude Opus 5.5 <[email protected]>

1 files changed, 293 insertions, 0 deletions

diff --git a/docs/typ/building.typ b/docs/typ/building.typ
new file mode 100644
index 00000000..85f2dfbb
--- /dev/null
+++ b/docs/typ/building.typ
@@ -0,0 +1,293 @@
+// Building pardes, for contributors: platforms, options, tests and the
+// release gates, then how the pieces fit: the core and its shells, the
+// threads, detached sessions, the 9P engine, the listeners and mounts.
+#import "style.typ": key, keys, btn, chord, word, tag, addr, file, cmd, doc, pairs
+
+= Platforms
+
+Zig *0.16.0* (`build.zig.zon` pins `minimum_zig_version`). Dependencies are
+fetched and pinned by the manifest; the terminal build needs no system
+package. The SDL shell builds SDL3 and FreeType from source. PDF support
+builds MuPDF and is on by default (`-Dmupdf=false` drops it). 9P over QUIC
+(`-Dquic=true`) uses system OpenSSL 3.6+ and pkg-config.
+
+#pairs(
+  [`zig build`], [the terminal shell and the SDL window together, *installed into `~/.local/bin`* (`pardes`, `pardes-gui`, and `pardes-v9fs`, the #word("Tty9p") helper)],
+  [`zig build -Dplatform=tty`], [the terminal shell alone],
+  [`zig build -Dplatform=gui`], [the SDL3 window alone],
+  [`zig build web -Dplatform=web -Dtarget=wasm32-freestanding -Ddump=<dump.zon>`], [a freestanding wasm core plus vanilla JavaScript (`docs/web.md`)],
+  [`zig build -Dplatform=macos`], [an AppKit and CoreText app over a static `libpardes.a` (`docs/macos.md`)],
+  [`zig build -Dplatform=esp32p4`], [a freestanding riscv32 editor object for the ESP32-P4, with a 384 KiB heap],
+)
+
+A bare `zig build` installs into `~/.local`; `--prefix <dir>` installs
+elsewhere, except `--prefix zig-out`, which counts as no prefix. With
+`-Dplatform` the default prefix is `zig-out`, so always pass `-Dplatform`
+while developing. Test, benchmark and run steps build what they need
+without installing. `pardes --version` prints `pardes <version>` (from
+`build.zig.zon`), plus the commit for release builds and the `~/.local`
+install; `pardes --help` lists every flag.
+
+The ESP32-P4 firmware is linked in the sibling `../05-zig-p4` toolchain,
+which needs ESP-IDF register headers: after building the editor object
+here, `zig build -Dpardes` there links `src/esp32p4/app.zig`. The separate
+GPIO 9P image is `zig build -Dapp=../02-pardes-code/src/esp32p4_9p.zig`
+there; its namespace is in `src/esp32p4_gpio.zig`.
+
+== Build options
+
+`zig build --help` lists the options for the selected platform.
+
+#pairs(
+  [`-Dplatform`], [`tty`, `gui`, `web`, `macos`, `esp32p4`; absent: the tty and SDL shells together, installed into `~/.local`],
+  [`-Dstatic`], [bool, `false`],
+  [`-Dquic`], [bool, `false`: 9P over QUIC with system OpenSSL 3.6+],
+  [`-Dmupdf`], [bool; on natively, off for web and esp32p4],
+  [`-Djpx`], [bool, `true`: JPEG 2000, and with it scanned PDFs],
+  [`-Dtree-sitter`], [`disabled`, `zig`, `minimal`, `full`; `full` natively, `zig` for web, `disabled` for esp32p4],
+  [`-Dembed-sources`], [bool, `false`: serve the sources under #file("/src")],
+  [`-Dstamp-commit`], [bool: the git commit in `--version` and crash records; on for release builds and the `~/.local` install],
+  [`-Dtheme-animation`], [bool; on except for esp32p4],
+  [`-Dworkspace-tag`], [bool: draw the workspace tag row; on except for macOS, whose menu bar carries it],
+  [`-Dprebuilt-shaders`], [bool: embed the committed SPIR-V; on for a bare `zig build`, off with `-Dplatform`; `zig build shaders` refreshes it with glslc],
+  [`-Dtracy`], [path to a Tracy checkout; off],
+  [`-Dmacos-identity`], [codesigning identity for `pardes.app`; `-` (ad hoc)],
+  [`-Ddump`], [a `dump.zon` to embed in the web shell],
+  [`-Dtest-filter`], [run only tests whose name contains it; a filter that matches nothing fails],
+  [`-Dtest-rebuild`], [bool: fresh Zig test compilation],
+  [`-Dhelix-harness`], [reference executable for live differential tests; `HX_HARNESS`, else `hx-harness` on `PATH`],
+  [`-Desp32p4-cols`, `-Desp32p4-rows`], [the board's grid, 56 by 14],
+)
+
+= Tests
+
+```
+zig build unit-test        module and shell unit tests (the perf gate runs with it)
+zig build test-build       compile the unit-test programs without running them
+zig build core-test        core tests without native shell tests
+zig build pane-test        pane, output, PDF and namespace integration tests
+zig build syntax-test      tree-sitter tests without building the editor
+zig build syntax           deterministic per-byte highlighting snapshots
+zig build fs-test          real sessions and mounts over 9P (with a short 9P monkey)
+zig build monkey-9p        random 9P operations checking the documented rules
+zig build agent-session-test  interactive session driver checks over 9P
+zig build 9p-test          freestanding protocol tests
+zig build 9p-io-test       native 9P transports and client
+zig build quic-test -Dquic=true  optional QUIC transport tests
+zig build v9fs-test        a real kernel mount (needs sudo -v; fails, not skips, without it)
+zig build snap             scripted input traces against frozen golden grids
+zig build monkey           random snapshot scripts hunting panics (not a gate)
+zig build hxdiff           differential suite against helix's own behaviour
+zig build hxparity         file-pane vs pty-pane editing parity
+zig build hxgolf           every helix-golf example, step by step, against helix
+zig build perf-gate        a gesture on the 50k-line file over 3x its baseline fails
+zig build mupdf-check      compile, link, render and search docs/design.pdf
+zig build web-snap         browser highlighting and touch interactions
+zig build web-e2e          Chrome-driven DOM end-to-end suite
+zig build cheatsheet       render the cheatsheet PDF (needs typst)
+```
+
+- `-Dtest-filter=<text>` applies to every unit-test binary. A failed test's
+  printed trace can be stale: run it alone with the filter.
+- `snap -- --record=DIR` writes snapshots without touching goldens,
+  `snap -- --update` replaces goldens (re-record one script by name, after
+  reading its diff, never all), `snap -- --no-retry` makes the first
+  failure decisive. Snapshot scripts can use `snap9p` to capture core cells
+  through 9P.
+- `zig build monkey -Dplatform=tty -- <seeds> [steps] [--from=N] [--out=DIR] [--keep]` writes one random script per seed and keeps any
+  that panics; a seed always makes the same script. Each crash found gets a
+  fix and a regression script in `test/snapshots/`.
+- `zig build hxdiff -- --strict cases.jsonl reference.jsonl [waivers.jsonl]` runs custom differential cases. `docs/helix-keys.md`
+  tracks helix parity key by key and names the reference helix build;
+  `docs/selections.md` is the selection model under normal mode.
+- Benchmarks (`perf`, `pdf-bench`, `pdf-scroll-bench`,
+  `pdf-sections-bench`, `lspbench`, `fs-bench`) take `-- --json`; measure
+  with `-Doptimize=ReleaseFast`. `perf -- --base old.json` refuses reports
+  whose build metadata differs.
+- `zig build history -- run 'ancestors(@, 2)' DIR -- zig build unit-test`
+  records a command's output and runtimes across revisions; `history -- compare a.json b.json 1.20` fails past that runtime ratio.
+- `python3 -B test/agent_session.py <pardes> --ready 'text' --min-rows N -- command args` drives an interactive command in a private shell and checks
+  it through 9P.
+
+== Release gates
+
+A release passes all of these first: `unit-test` with `-Dplatform=tty` and
+with `-Dplatform=gui`, `core-test`, `fs-test`, `snap`, the GUI goldens
+(`python3 -B test/gui_golden.py <a Debug pardes-gui>`, a hidden window),
+`web`, and the ReleaseSafe tty and ReleaseFast gui builds. The committed
+`docs/typ/cheatsheet-a4.pdf` is rendered again when the docs change.
+
+= The core and its shells
+
+One core, five shells (tty, gui, web, macos, esp32p4). The core owns
+editing, layout, rendering and the control filesystem; a shell turns native
+input into `pardes.Event`, presents `pardes.Surface`, and performs host
+effects such as spawning processes.
+
+- The core is a state machine: input arrives as an `Event` through
+  `update`; output leaves as a `Surface` from `render(arena)` and as a ring
+  of `Effect` values. Effects are fixed-size values with no lifetime ties
+  into the core; unbounded content is read off the core when an effect is
+  drained (`save_text` carries a path and the pane's serial, so a reused
+  slot writes nothing). `emit` refuses when the ring is full and never
+  evicts.
+- Pty bytes leave as 64-byte `.write` effects; overflow parks in a per-pane
+  buffer that `nextEffect` drains in order. `postEvent` is a 64-entry value
+  queue for events that borrow no slices.
+- `pump` runs: wait for input (the host owns the sleep), flush paused 9P
+  write batches, drain queued events, drain effects, settle the turn, then
+  render and present only when a frame is needed and someone is looking.
+- `host_io.Host` is a context pointer and a vtable of optional callbacks; a
+  null method is not an error, and `Host{}` is a complete in-process pardes
+  that the tests use. Comptime decides what a build has (`pardes.platform`,
+  `hosted`, `can_attach`, `terminal_panes`, `pdf_enabled`); the vtable
+  decides who serves it.
+- The root module is chosen by platform: `main.zig` (tty, gui),
+  `web.zig`, `macos.zig`, `esp32p4.zig`. The web and macOS shells are
+  libraries whose host owns `main()`.
+- `memory.limits` is the one home for capacities that differ on the board
+  (panes 16, columns 6, selections 64, the tag's 512 bytes). Each core
+  allocator is a thread-safe stack-fallback allocator, under a
+  DebugAllocator in Debug builds.
+
+Code map: `src/panes.zig` and the pane kinds it names (`src/File.zig`,
+`src/Terminal.zig`, ...), `src/layout.zig`, `src/fs.zig` (host access,
+mounts and resolution) and `src/pardes.zig` (input), with one file per
+thing beside them (`edit.zig`, `normal.zig`, `look.zig`, `exec.zig`,
+`mouse.zig`, ...). `src/ninep/` is the control tree, `src/detached/` the
+wire, the detached core and the frontend client, `src/lsp/` the
+language-server client; `test/` holds harnesses, goldens and helix cases,
+`build/` the snapshot suite's build step.
+
+== Threads
+
+The core is single-threaded under a turn mutex, `pardes.turn`. The editor
+thread holds the turn and lets go of it while it waits for input and while
+it is out in a host syscall mid-step; a 9P connection task takes the turn
+in those gaps to answer a request. While a step is out, a request that
+would change a pane parks (`Status.again`) and is retried when the editor
+rests. 9P requests are not events: they enter through `Pardes.serveFs`, and
+only one that changes a pane costs a frame. Language-server and
+selection-pipe workers post completions through a bounded mailbox; a
+`host_io.Lsp.Job` owns copies of the source, path and arguments, never
+reading the live core, and a request id, the pane's serial and (for an
+edit) the file's revision reject a stale reply. A process that never calls
+`turn.start` (the tests, the board, the browser) has no second thread.
+
+== Detached sessions
+
+One poll loop in `src/detached/server.zig` owns core mutation, frontend
+connections, pty I/O and file watches. The frontend socket is
+`pardes-detached-<name>.sock` beside the session's 9P socket; a second
+session cannot take a live name. Up to 32 frontends attach; frames go to
+every one, while clipboard reads, browser opens and #word("Detach") go to
+the frontend that asked (else the first attached). Frames are full grids
+or changes against what each frontend last received, never queued: a
+frontend with unsent bytes skips that frame, and one that stops draining
+past 1 MiB of control backlog is closed, its peers untouched. Frontends
+never spawn shells, write session files or watch files. #word("Attach")
+connects first and swaps second: only after the handshake does a shell
+give up its own core. Restore builds the replacement core before touching
+the current one.
+
+`src/detached/wire.zig` is the versioned protocol (version 10): fixed-width
+little-endian fields, a 5-byte header (tag, then a u32 length), payloads up
+to 16 MiB and grids up to 512 by 128. A comptime hash of `pardes.Chrome`
+forces a version bump when that struct changes.
+
+= 9P
+
+The wire format, the client and server connections, the file-server engine
+and the Unix, TCP and QUIC transports are the `cloud9` package
+(`git.sr.ht/~gbrls/cloud9`), pinned in `build.zig.zon` and fetched into
+`zig-pkg/`. Re-pin with
+`zig fetch --save=cloud9 git+https://git.sr.ht/~gbrls/cloud9#<commit>`, or
+use `.cloud9 = .{ .path = "../cloud9" }` while editing both. cloud9's own
+`zig build test`, `transport-test`, `quic-test -Dquic=true`, `fuzz` and
+`differential` cover the shared code.
+
+- The engine (`fs.Server`) owns fids, permissions, directory reads,
+  flushes and what a hangup releases; it allocates nothing and makes no OS
+  calls. The control tree in `src/ninep/` is its backend: `tree.zig`
+  (nodes, dispatch), `pane.zig`, `ctl.zig`, `cols.zig`, `addr.zig`,
+  `pty.zig`, `events.zig` (event and log), `screen.zig`, `sources.zig`.
+  `src/9p.zig` names the editor's and the board's engine settings: msize
+  65536, 256 fids, 128 held reads a connection, names up to 255 bytes; the
+  board has 32 fids.
+- A read, write, open, clunk, remove or truncating wstat can park; a walk,
+  attach, stat, create or rename cannot. Every Rread is clamped to the
+  count and the msize; Tflush answers the original request first.
+- Unix and TCP listeners run on cloud9's `serve.Runner`: an accept task per
+  listener, a reader and a writer task per connection, 16 connections. QUIC
+  (`src/9p_quic.zig`) still runs on the editor's poll loop in
+  `src/9p_io.zig`.
+- A body write takes only whole UTF-8 sequences and answers a short count
+  for the rest. Consecutive writes from one open at the end of the text are
+  held and applied as one edit, flushed by any other request, the open's
+  release, 64 MiB, or a 20 ms pause in the editor's step.
+
+== Writes through a mount <writes-through-a-mount>
+
+A mount cuts a big write into pieces of at most one message (msize 65536,
+less the header), and each command line runs once it is whole. A write
+that does not fill its message is whole, so its last line runs even
+without a newline (`printf Save > exec`), unless it is a multiple of 4096
+bytes: that is where a writer's buffer (stdio, a mount's page cache)
+filled and cut a line, so its tail waits for the next write or the close.
+A line held to the close (such a tail, or an `Edit` block never ended)
+runs there, and its failure is only in the log, as its `err` record: the
+close reports no error, and the write that sent it had already succeeded.
+So a script that needs a line's result ends the write with a newline.
+
+== Listeners <listeners>
+
+`--9p-tcp='tcp!127.0.0.1!5640'` adds TCP; `--9p-quic='quic!127.0.0.1!5641'`
+adds QUIC (built with `-Dquic=true`; ALPN `pardes-9p`, an ephemeral TLS
+identity, no peer verification). Addresses are numeric IPv4 or IPv6; port
+0 picks one; #file("/listeners") reads them back. Every connection has full
+session access, #file("/os") included, and TCP is unencrypted: use
+loopback. Unix and TCP share 16 connection slots; a 17th client's Tversion
+gets `too many connections` (and the log
+`err - 9p: too many connections (N turned away)`). QUIC has 16 of its own.
+plan9port and v9fs need a userspace bridge for QUIC.
+
+`$XDG_RUNTIME_DIR/9p` is the machine's `/srv`: servers post themselves
+there by name, and `9ns --mntgen` mounts the whole registry. A pardes whose
+socket is in the runtime directory posts
+`$XDG_RUNTIME_DIR/9p/pardes/<name>`, a symlink to its socket, and unposts
+it on a clean stop if it is still its own; one whose socket fell back to
+`~/.local/state/pardes` posts nothing. Posting first sweeps the group: a
+symlink whose socket refuses a connect is removed with its socket. pardes
+binds its own socket rather than going through `cloud9.post`, which takes
+only flat names. A reader of the registry must `stat` through the symlink.
+
+== Kernel mounts and Tty9p
+
+For Linux v9fs use `version=9p2000,cache=none,access=any`, `trans=unix`
+(or `trans=tcp` with `port=`), `uname`, `dfltuid` and `dfltgid` for the
+local user, and an empty `aname`. Linux follows `O_TRUNC` with a `Twstat`
+of zero length and an `mtime` hint; pardes takes the truncation and drops
+the hint.
+
+#word("Tty9p") starts the normal shell and queues a quoted helper command,
+which bash and fish run at their first prompt as a foreground job, so
+sudo has the terminal. The unprivileged launcher makes a private temporary
+mountpoint and runs `sudo -E`; the elevated `pardes-v9fs` helper makes a
+private mount namespace, mounts the session's socket with
+`trans=unix,version=9p2000,cache=none,access=any,nosuid,nodev,noexec`,
+drops every root id and capability, and runs the shell as the user (with
+the caller's `PATH` again). The namespace and the mount go with its last
+process. Nothing setuid and no passwordless sudo rule is installed; the
+helper takes explicit paths and a command and is no restricted broker, so
+never grant it passwordless sudo. The host finds the helper beside its own
+executable, or at `PARDES_V9FS_HELPER`. Code: `src/linux/v9fs.zig`;
+`zig build v9fs-terminal-test` and `v9fs-driver-test` need no privileges.
+
+= Other notes
+
+`docs/` keeps the platform and design notes beside this book:
+`web.md` and `macos.md` (the browser and macOS shells), `effects.md` and
+`render-pipeline.md` (visual effects), `helix-keys.md` and `selections.md` (helix parity),
+`lsp-evaluation.md`, `ui-review.md`, `divergences.md` (bookmarks off
+`main`) and `open-questions.md`. `next-steps.txt` is a wishlist and
+`transactions.txt` records one open structural gap against helix.