diff options
| -rw-r--r-- | docs/next-daemon.md | 183 |
1 files changed, 183 insertions, 0 deletions
diff --git a/docs/next-daemon.md b/docs/next-daemon.md index 681a2ed4..0fe72b76 100644 --- a/docs/next-daemon.md +++ b/docs/next-daemon.md @@ -2129,3 +2129,186 @@ Not done, and why: of the older build keep talking theirs. - A session name with a newline would break the daemon's registry line. - An upgraded process records no tape (a recording ends at the upgrade). + +## Phase 6: 9P round trips notes + +Built 2026-10-30 (`src/9p_io.zig` Client, `src/fs.zig`, the Mount word in +`src/builtins.zig`, a paragraph of `docs/typ/reference.typ`). pardes's own +9P client is what mounts, the Mount word, the forwarding CLI, `--wait` +and Handover use; scripting clients (9p(1), 9ns, test/ninep.py) are not +changed here. Decisions at the forks, with the ponytail lens: + +- **The server already takes a connection's requests in order; nothing + there changes.** cloud9's engine has one job slot: it starts the next + frame only once the request before it is answered. What parks leaves + that slot, and the frames after it go ahead: a read with nothing yet, + and, while the editor is out in a syscall, a write or a truncating + open (`tree.needsQuiet`; an OTRUNC open is a setattr first). So a walk + and the open of its fid can be sent together (a walk never parks), and + so can an open and its first read (an OREAD open never parks). Two + things cannot: + - a Twrite behind a truncating Topen that parks is refused `bad use of + fid` (not open yet); + - a Tclunk behind a parked Twrite interrupts it. + + The client therefore sends a write or read that came back `bad use of + fid` again after its open's answer: that refusal is the engine's, made + before the backend, so nothing was done. It sends its Tclunk only after + the last answer, and does not wait for the Rclunk. The park test ("a + change waits while the editor is out mid-step") takes the resend path. +- **One flight per operation.** The flight is `[Tversion Tattach] Twalk + [Twalk] Tstat Topen Tread` for a read, or `... Topen Twrite` for a + write, each request with its own tag (`Flight`, `Session.land`). Past 16 + names, the second walk goes from the fid to itself, so no fid is in + between. +- **Version and attach pipelined by a poke.** cloud9's client submits + nothing while a Tversion is out. `greet` sets its `versioning` and + `msize` around the submits, taking the asked msize until the Rversion + lands. Only a pardes is dialled here, and every pardes offers + `ninep.msize`. A write in that first flight is kept under + `min_msize - 23`. The fix belongs in cloud9's client (taking requests + while it versions). It is not made here, since this round pushes + nothing (`// ponytail:` at `greet`). +- **The end of a read.** 9P has no end of file but an empty read. + read(5) says a reply "may be less than the requested amount", and a + directory's next offset must be the last plus what it returned, so + neither the end nor a directory's next read can be sent ahead. The + Tstat in the flight gives a file's length: a read that comes back short + at that length is the end. pardes reports real lengths for bodies, + `/os` files, ctl, tag and the like. A file of no stated length, or a + directory, costs one round trip more: index, layout, screen and log are + generated whole by each read and say 0 by design (tree.zig `topSize`). + An end flag would be a protocol extension, which is not worth one round + trip on generated files. +- **Read ahead.** With the length known, the Treads for the rest go at + once, 14 at a time (the 16-tag window, less a clunk still out and a + spare). A 1.3 MB file takes 3 round trips where it took 22. A file of + no stated length is not read ahead (`// ponytail:`). +- **Reads are timed by their progress.** Each Rread gives the rest the + 2 s budget again, as each Twrite already did. Found on the way: on main, + every read of one operation shared one 2 s budget, so a remote file of + 1.3 MB failed to open (`dial failed: timed out`) at 80 and 207 ms. On + the box without `-C`, the 77 KB `src/fs.zig` failed too. +- **A mount keeps its connection** (`kept`), as Plan 9's mount keeps its + channel: an operation through it is the flight alone, with no dial, + version or attach. + - There are eight slots, keyed by dial. A session is taken out of the + table while it is in use, so two users of one dial each get their + own. + - Unmount forgets it (`Client.forget`). + - Before reuse, a poll takes what arrived while it waited (a Rclunk + owed); an EOF, or anything else, means the peer let go, and a new + connection is dialled. + - A read on a kept connection that hangs up all the same is asked once + more on a new one. A write is not, since it may have been done. + - Only `readMount`, `writeMount` and `probeMount` (the Mount word) + keep one, since each costs the peer one of its 16 connections while + kept. The CLI, `--wait` and Handover dial afresh as before, with + their version, attach and walk pipelined. +- **No qid or stat cache.** The trace shows that a Look through a mount + is one operation with one walk. With walk, stat, open and read in one + flight, a cache would save bytes but no round trip, and a cached stat + is wrong once the peer changes the file. +- **msize and iounit stay.** They are already 64 KiB, with iounit = + msize - 24, so a body of up to 65,525 bytes is one Tread. A larger + msize would cost each of a session's 16 connections five buffers of + it. test/ninep.py asks 8192 by its own choice, so its body reads of + more than that cost it more messages. +- **`/n/<host>`.** The daemon serves no 9P yet (phase 3), so there is no + remote daemon to mount. A remote session mounts today with two + commands, now in reference.typ: `ssh -C -N -L /tmp/box.sock:<its + socket> host`, then `Mount box /tmp/box.sock`. A relay in the daemon + that lists a host's sessions under one mount waits for the daemon to + serve 9P. `-C` is there because the box's link is slow in that + direction (below). + +Measured. The harness is `.scratch/daemon/ninep/rt.py`. It runs a peer +session and a local one that Mounts it through a relay. The relay holds +each chunk a fixed delay after it arrives, as test/delay_relay.py does, +and it also splits the stream into 9P frames, counting the T-messages +and the rounds (runs of T-messages with no answer delivered between +them). `box.sh` runs the same harness against a detached session on the +box through `ssh -L`. Builds are ReleaseFast (main ymwmwrvx against this +line), 3 reps each, medians. + +The RTTs column counts the round trips the client waited on. pardes's +own client sends its last Tclunk without waiting, before and after; the +python client waits on every message. The box columns are `ssh -L` at +207 ms, plain and with `ssh -C`; "fails" means `dial failed: timed out`. + +| operation (pardes's client) | msgs | RTTs | 30 ms | 80 ms | 207 ms | box | box `-C` | +|---|---|---|---|---|---|---|---| +| connect: `Mount` (version, attach, root read) | 5 → 7 | 4 → 1 | 124 → 33 | 324 → 81 | 831 → 208 | 1032 → 426 | 1024 → 410 | +| file read: Look `/n/peer/index` (no length) | 7 → 6 | 6 → 2 | 185 → 61 | 486 → 161 | 1247 → 415 | 1441 → 426 | 1434 → 409 | +| directory listing: Look `/n/peer/pane/` | 7 → 6 | 6 → 2 | 185 → 62 | 483 → 161 | 1247 → 415 | 1437 → 427 | 1430 → 410 | +| ctl write: Save to `/n/peer/pane/1/ctl` | 6 → 4 | 5 → 1 | 155 → 32 | 402 → 81 | 1039 → 208 | 1237 → 214 | 1233 → 206 | +| pane body read: Look `/n/peer/pane/1/body` (30 KB) | 7 → 5 | 6 → 1 | 186 → 33 | 483 → 81 | 1247 → 208 | 1843 → 641 | 1432 → 206 | +| a remote file open: Look `/n/peer/os/.../fs.zig` (77 KB) | 8 → 6 | 7 → 2 | 213 → 64 | 563 → 162 | 1454 → 416 | fails → 1300 | 1844 → 617 | +| a remote file open, 1.3 MB | 27 → 25 | 26 → 3 | 807 → 103 | fails → 244 | fails → 625 | fails → 5334 | fails → 1273 | + +| the python client (unchanged, for scale) | msgs, RTTs | 207 ms | box | +|---|---|---|---| +| connect | 2 | 415 | 614 to 635 (with ssh's channel open) | +| read `/index`, list `/pane` | 5 | 1037 | 1023 to 1067 | +| ctl write | 4 | 829 | 818 to 852 | +| body read, 30 KB | 5 | 1037 | 1024 (`-C`) to 1283 (plain) | + +- **Each operation is one round trip.** A directory, or a file of no + stated length, takes two. The first operation through a new `ssh -L` + forward pays one more, the channel opening (the Mount, 426 ms on the + box). +- **On the box, bytes cost more than the 9P rounds.** The link from the + box to here carries 170 to 400 KB/s through ssh. A 1 MiB `dd` takes + about 6 s and 4 MiB about 10 s, where 4 MiB up takes 2.4 s. A 64 KiB + answer arrives 1 to 2 RTTs after the request, so the 30 KB body reads + in 641 ms plain and 206 ms with `-C`. The 1.3 MB file's Rreads come in + at about one a round trip (the timeline is in `rt.py`'s `TIMELINE=1`). + `blast.py` measured all 21 of its Treads in flight at once at 1.9 to + 3.0 s, against 5.3 s for this client's 14 and then 7. A sliding window + instead of batches would buy that, but only for large files on this + link (not done). Compression helps every size: `-C` is in the + documented recipe, and the replica stream's zstd (phase 6) will face + the same link. + +No local regression. Local 9P has no delay. lab.py runs the base and +the new build at once with their samples alternating, 1000 each, under +the perf lock, and gives the p50 ratio new/base. The server is not +changed, so its rows are level. The mount rows are faster because they +dial nothing. + +| local, p50 | base | new | A/B | A/A | +|---|---|---|---|---| +| python client: read a 9 KB body (server path) | 88 µs | 89 µs | 1.004 | 1.000 | +| python client: ctl write `clean` (server path) | 80 µs | 81 µs | 1.011 | 0.992 | +| Look `/n/peer/pane/1/body` through a local mount | 329 µs | 255 µs | 0.776 | 1.009 | +| Save to `/n/peer/pane/1/ctl` through a local mount | 214 µs | 141 µs | 0.660 | 1.000 | + +The gates: +- test/fs.py, `zig build 9p-io-test`, unit-test tty, the snapshots and + test/daemon.py (20 cycles) pass. +- New: `a mount's connection is kept between operations, and one the + peer hung up is dialled again` (a write longer than a message on a + kept connection, a miss, a hangup, a redial and a forget). It also + passes on the box (FreeBSD test build). + +Not done, and why: +- **cloud9's client should take requests while it versions;** the poke + in `greet` stands in for that until cloud9 is pushed and re-pinned. + 9ns (cloud9's) still waits one round trip per message. +- **A write longer than one message** sends its chunks after the first + one at a time: a Save of a 1 MB file through a mount is 16 round + trips. Pipelined Twrites could run in the wrong order once one parks; + that does not matter for `/os` (pwrite at an offset) but would for + some pane files. +- **A file of no stated length** is not read ahead, and its end (like a + directory's) costs a round trip more. +- **The CLI's `ask`** (`pardes file` from a pane, `pardes -`) and + `--wait` pipeline only version, attach and walk. Their open, writes + and reads go one at a time, on a connection of their own. +- **A peer's Restore cuts its clients up to 200 ms after it starts.** A + request on a kept connection in that window is refused `Input/output + error`, where a new dial would already have been the replacement's. +- **Scripting from here to a remote session** through test/ninep.py or + 9p(1) is still one round trip per message (1 s for a read at 207 ms). + A python helper for remote scripting could pipeline the same way; + nobody scripts that way yet. |
