diff options
Diffstat (limited to 'docs/catalog.md')
| -rw-r--r-- | docs/catalog.md | 177 |
1 files changed, 177 insertions, 0 deletions
diff --git a/docs/catalog.md b/docs/catalog.md new file mode 100644 index 0000000..cd6eec9 --- /dev/null +++ b/docs/catalog.md @@ -0,0 +1,177 @@ +# 9web local catalogue (Linux) + +`9web --catalog FILE` runs a local website catalogue in the gateway process. +It serves existing generated websites immediately, supervises requested builds, +and inventories listening TCP sockets without making requests to them. No +separate site servers are needed. The existing `--upstream` gateway mode is +unchanged; catalogue mode and an explicit upstream are mutually exclusive. + +## Configuration and serving + +See [catalog.example.json](catalog.example.json). Every site has a unique `name`, +`title`, absolute `cwd` and `root`, and a distinct nonzero HTTP `port`. Optional +`build` and `watch` values are argument arrays, not shell command strings. +Use an explicit `sh -c` only when a trusted recipe needs shell syntax. + +``` +9web --listen '[::]:8090' \ + --catalog "$HOME/.config/9web/sites.json" \ + --catalog-socket "$XDG_RUNTIME_DIR/9web/catalog.sock" \ + --state-dir "$HOME/.local/state/9web" +``` + +Create the socket's parent directory first. Configuration is read at startup; +restart the service to change it. A failed site-port bind fails startup rather +than advertising a site that is not actually served. All site listeners use +the dashboard's bind address. `[::]` uses a dual-stack listener for all IPv6 and +IPv4 interfaces; `0.0.0.0` restricts this to IPv4. + +Each site occupies its own HTTP origin so root-relative `/assets/` and `/blog/` +URLs work without HTML rewriting. Directory URLs redirect to a trailing slash +and serve `index.html`; extensionless paths try `.html`. Static serving supports +MIME types, HEAD, single byte ranges, and streaming large files. Symlink traversal, +dotfiles, `..`, and encoded path separators are refused. Directory listings are +available through 9P, not through site HTTP roots. + +The HTTP site root is static: provider-specific `_redirects`, `_headers`, and +Worker APIs are not executed. For example the café's articles and assets work, +but its Cloudflare poke API is not reproduced locally. + +## Builds and watch + +The dashboard has Build, Watch, Cancel, and Log controls. Builds execute in the +configured working directory with the service's environment. One child job runs +per site. Repeated build requests during a build coalesce into one later build. +Build writes enqueue work immediately; serving does not wait for compilation. + +For managed watch, configure `inputs` as source paths relative to `cwd` (or absolute +paths for shared libraries outside the project). The +watcher checks source names, modification times and sizes once per second and +invokes the build recipe on changes. It skips hidden entries, symlinks and common +generated directories (`out`, `dist`, `build`, `node_modules`, `__pycache__`). +List a particular build script explicitly if it lives in one of those directories. +This is a metadata watcher, not a content hash or a Typst dependency graph. +Changes during a build are detected by the next scan. Browser refresh is manual. + +Alternatively, `watch` can specify an existing persistent watcher. Its stdout and +stderr go to the same log, and watch-off cancels its process group. A native watcher +may itself serve a preview port, but the catalogue never requires that preview to +serve the site's generated files. Manual builds are refused while a native watcher +is active. Cancel disables watching and clears queued work. Server shutdown under +systemd kills the whole service cgroup, including build descendants. + +Each new job replaces `<state-dir>/<name>.log`. The UI and 9P expose its last +256 KiB; the on-disk log is not rotated during a long-running native watcher. +Watch state and build status reset at service restart; generated output remains. +No build or watch is started automatically. + +Builders currently write to their configured output directories in place. +`9web` does **not** provide atomic publication or a last-good snapshot: a failed +build can leave partial output, and a file truncated during a response closes that +response. Builders needing atomic publication should generate a separate release +directory and arrange publication themselves. The served root can be a symlink; +symlinks below that root are refused. + +## 9P and HTTP controls + +The catalogue is itself a 9P backend, consumed by the ordinary gateway mux. +`--serve` and WebSocket clients see the same tree as direct socket clients: + +``` +/index.html +/sites/<name>/public/ complete generated output tree (read-only) +/sites/<name>/ctl build | watch on | watch off | cancel +/sites/<name>/status JSON state, PID, exit code, times, generation +/sites/<name>/log last 256 KiB of current/most recent job +/sites/<name>/events blocking stream of JSON state changes +/services/index.json recognized HTTP/HTTPS listeners +/listeners/index.json all observed TCP listeners, including unknown ones +``` + +``` +9ns --unix "$XDG_RUNTIME_DIR/9web/catalog.sock" --name web -- sh +echo build > /mnt/9p/web/sites/entomology/ctl +cat /mnt/9p/web/sites/entomology/status +cat /mnt/9p/web/sites/entomology/log +``` + +Event opens receive the current state, then park until its revision changes. +Each reader has its own cursor; slow readers receive the newest state, not a +durable history of every transition. 9P flush cancels parked reads. State events +also work through `/fs/sites/<name>/events?follow=1` as SSE. `status` is the finite +snapshot to use for ordinary polling. + +The dashboard reads `GET /_cloud9/catalog.json` and uses +`POST /_cloud9/sites/<name>/ctl` with a plain-text command. The existing +`PUT /fs/sites/<name>/ctl` also reaches the same control implementation. + +## Passive discovery + +Every five seconds, an embedded Python 3 inventory helper reads `/proc/net/tcp*`, +joins socket inodes to process descriptors, and recognizes executable/argument +patterns for miniserve, Python `http.server`, Typst watch, Vite and Wrangler dev. +9web advertises only its explicitly registered HTTP ports: its 9P listener is +never labelled HTTP just because the owning executable is 9web. + +No TCP connections or HTTP probes are performed. Unknown or inaccessible process +ownership stays unknown. Full command lines and environments are not published; +entries contain address, actual bound port, PID, executable, cwd and the evidence +for classification. Process races and permissions can make entries temporarily +unknown. Discovery is in the service's network namespace, not across containers. +The helper is a short-lived child of 9web, not another daemon. + +Wildcard addresses in dashboard links use the browser's current server hostname. +Specific addresses remain specific; loopback services are labelled local-only. +There is no automatic proxy to loopback-only services. Recognizers identify likely +HTTP listeners from process evidence; they cannot prove application health. + +## Service and access + +[9web.service](9web.service) is a user-systemd unit template. Install the binary +at `~/.local/libexec/9web`, the config at `~/.config/9web/sites.json`, and the unit +at `~/.config/systemd/user/9web.service`, then: + +``` +systemctl --user daemon-reload +systemctl --user enable --now 9web.service +systemctl --user status 9web.service +journalctl --user -u 9web.service +``` + +The unit supplies a private runtime directory and socket, restarts on failure, +and kills child jobs together with the service. Lingering must be enabled if the +user service should run without an interactive login. It needs `python3`, `setsid`, +and the configured project build tools in PATH. Miniserve remains independent. + +This is a trusted-network operator interface: it has **no login or TLS**. Anyone +who can reach the dashboard can read the catalogue and trigger configured jobs. +Browser control requests reject foreign Origins; numeric Host validation and +separate site origins prevent served pages from using the dashboard as a same-origin +control API. This is not authentication. Bind to loopback or place an authenticated +gateway in front of it when those network permissions are inappropriate. No +arbitrary command can be supplied through HTTP or 9P; commands come from the local +registry, which should remain writable only by its owner. + +## Verification + +``` +zig build catalog-test http-fs-test mux-test -Doptimize=ReleaseSafe +``` + +The catalogue integration test uses temporary real sites and processes. It checks +immediate static serving, pretty URLs, MIME, HEAD/ranges, traversal and symlink +rejection, Host/Origin checks, HTTP and raw 9P controls, watched rebuilds, failed +build status, process-group cancellation, nonblocking event readers, and actual +socket ownership discovery. Existing gateway HTTP/filesystem and mux suites cover +the original mode. + +## Installed workstation configuration + +The initial workstation setup uses dashboard port 8090 and site ports 8091–8095: +Entomology lab, 0x4200.cafe, Standards, Pardes website, and the older café hub archive. +The archive is read-only (no build recipe registered). Other sites have explicit +local build recipes and opt-in managed watches. Port 8081 remains miniserve's. +Runtime config is `~/.config/9web/sites.json`; the service uses a separately +installed binary so subsequent development builds do not change the running +service's executable on disk. Source and deployment changes are described in +the implementing git commit; no remote push is required for the local service. |
