9ns --mntgen: a server that never answers stalls only its own name

Opening a fish (self-wrapped in `9ns --mntgen`) and running an agent in it
would sometimes freeze the whole session: no input reached it and nothing
under /mnt/9p answered, until the shell was killed from outside. The cause
was one posted server that accepted a connection and then never spoke 9P —
pardes, answering its 9P from the same loop that was walking its own mount,
was the one on this machine, but any wedged or half-dead server does it.

Three things conspired, and each is fixed on its own:

* The dispatcher dialed. A LOOKUP of an undialed name ran connect, Tversion,
  Tattach and Tstat on the one thread that reads /dev/fuse, so while that
  server kept quiet no request for any name was read, and no FUSE_INTERRUPT
  either. Now the dispatcher makes a Mount without touching the network and
  queues the walk to the mount's worker, which dials while serving it. The
  dial is the request in flight, so an interrupt of the walk abandons it at
  once (`Session.abort_on_cancel`: nothing to flush before a session
  exists) and the walk answers EINTR; a failed dial leaves the mount
  undialed for the next walk to retry; a full listen backlog (the server
  stopped accepting) is retried for 5s and then EIO. Every later LOOKUP of
  the name goes through the same queue and is answered from the remembered
  root attr, so the dispatcher never holds a session at all.

* Once the dispatcher had read a request the process behind it was
  unkillable (FUSE waits out a request userspace has taken), and an
  INTERRUPT for a request still sitting in a mount's queue was dropped. The
  dispatcher now takes a queued request out and answers EINTR itself, and
  forwards only in-flight ones to the worker; queue and in-flight unique
  are read under the mount's mutex, where the worker moves a request from
  one to the other. A Tflush the server never answers is given 3s
  (`Session.flush_grace_ms`) and then the session is declared wedged: the
  request answers EINTR, the mount dies, the next walk makes a new one.

* The kernel serialized the directory. Without FUSE_PARALLEL_DIROPS in the
  INIT reply every LOOKUP and READDIR in a directory takes its inode lock,
  so one parked walk held up every other name under /mnt/9p however free
  the dispatcher was (`cat` sat in fuse_lock_inode). The flag is now
  negotiated when the kernel offers it.

What remains is the kernel's own serialization of lookups of one *name*: a
second walker into the parked name waits for the first walk to end, and
only then proceeds (and can be interrupted in its turn).

An adversarial review of the above found three more things, fixed here:
the single-connection bridge's one-slot stash stopped polling the FUSE fd
while a second request was parked, so an INTERRUPT could not arrive (and
parallel dirops make a second request routine) — the stash is now a queue
of copies and the fd is always watched; a dead or wedged mount kept its
socket open until exit, where a late-answering single-threaded server
could block on it — the session is closed when the mount dies; and
teardown after DESTROY or ENODEV (the child still alive, so stop_fd says
nothing) could join a worker parked on a mute server forever — the
sockets are shut down before the join. The flush grace is a deadline now,
not a timer restarted on every wakeup. A black-box run against the binary
(hostile servers: mute, garbage, close-after-accept, full backlog, 100
mute names, interrupt storms, 300 deaths of one server) found that a dead
mount kept its socket, its interrupt pipe and a megabyte of buffers until
exit — three descriptors per death — so `retire` now frees all of it and
keeps only the slot; descriptors, threads and RSS stay flat across 400
deaths. The 4096-slot cap per process remains and is documented.

Reproduced with a socket that accepts and never writes, posted beside
9agents in a scratch registry: before, `cat /mnt/9p/agents/pid` parked
behind `stat /mnt/9p/hang` and SIGINT did nothing; after, it answers at
once, the parked walker dies of its signal within milliseconds, and a
server that answers the handshake but ignores reads and Tflush releases
its reader after the grace. mntgen.sh and adv_bridge_interrupt.sh now
check exactly that; nine.zig gains unit tests for the grace and the
abort. Also in this change: the uncommitted ESTALE-on-death and
FUSE_NOTIFY_INVAL_ENTRY work from the working copy, which the dead-mount
path here builds on.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

2 files changed, 52 insertions, 23 deletions

diff --git a/9ns/test/adv_bridge_interrupt.sh b/9ns/test/adv_bridge_interrupt.sh
index e364143..ba76f78 100755
--- a/9ns/test/adv_bridge_interrupt.sh
+++ b/9ns/test/adv_bridge_interrupt.sh
@@ -118,18 +118,20 @@ expect_eq "never_flush/cache0: reader killed" "124" "$(field rc "$OUT")"
 fids_same "never_flush/cache0"
 bridge_fids_same "never_flush/cache0"
 
-echo "# (b) a server that ignores Tflush too: the reader stays blocked, SIGTERM to 9ns still ends the session"
-start_server never
-timeout -s TERM 3 "$NS" --unix "$SOCK" --mount "$M" -- sh -c "timeout -s INT 1 cat $M/f; echo unreachable-rc=\$?" >"$TMP/never.out" 2>"$TMP/never.err" &
-TPID=$!
-sleep 4
-if kill -0 "$TPID" 2>/dev/null; then
-    fail "never: SIGTERM did not end 9ns while the reader was stuck"; kill -9 "$TPID"
-else
-    pass "never: SIGTERM ends 9ns even though the server ignores the Tflush"
-fi
-wait "$TPID" 2>/dev/null
-expect_eq "never: the reader never came back (server ignores Tflush)" "" "$(grep unreachable "$TMP/never.out")"
+echo "# (b) a server that ignores Tflush too: the reader comes back after the flush grace, and the session is gone with it"
+# `never` stops reading once the Tread hangs, so the Tflush is never even
+# seen. The protocol says wait for the Rflush; a server that has not managed
+# one in 3s (nine.Session.flush_grace_ms) is not going to, and the reader
+# behind the interrupt is unkillable until we stop waiting. So the read fails
+# EINTR after the grace, the session is wedged, and 9ns ends the mount: the
+# next access answers ENOTCONN instead of parking another process forever.
+run never "$(timed "timeout -s INT 1 cat $M/f"); cat $M/d/g 2>&1; echo after-rc=\$?"
+no_crash "never/grace"
+expect_eq "never/grace: reader killed by the signal (124)" "124" "$(field rc "$OUT")"
+NEVER_MS=$(field ms "$OUT")
+if [ "$NEVER_MS" -ge 3500 ] && [ "$NEVER_MS" -lt 9000 ]; then pass "never/grace: released after the 3s flush grace (${NEVER_MS}ms)"; else fail "never/grace: release time out of range" "ms=$NEVER_MS (expected 3500..9000)"; fi
+expect_eq "never/grace: the mount is gone afterwards (not a hang)" "after-rc=1" "$(printf '%s\n' "$OUT" | grep '^after-rc=')"
+expect_contains "never/grace: 9ns reports the closed session" "connection closed" "$STDERR"
 
 echo
 echo "passed=$PASSED failed=$FAILED"
diff --git a/9ns/test/mntgen.sh b/9ns/test/mntgen.sh
index aade76c..7dada53 100755
--- a/9ns/test/mntgen.sh
+++ b/9ns/test/mntgen.sh
@@ -254,14 +254,18 @@ echo "# --debug and --no-direct-io reach the mntgen dispatcher"
 DEBUG_ERR=$(timeout 60 "$NS" --mntgen --debug -- sh -c "ls $M/alpha >/dev/null" 2>&1 >/dev/null)
 expect_contains "--debug traces the dispatcher" "lookup 'alpha'" "$DEBUG_ERR"
 
-echo "# a dial parked on a mute server unwedges when the program exits"
-# A server that accepts the connection but never answers Tversion parks the
-# dispatcher in the dial (pinned: no dial timeout, no concurrent dial). The
-# dial must watch stop_fd through the whole handshake: when the program's
-# main flow exits while a background walk is parked there, 9ns must follow it
-# out instead of wedging forever (pre-fix it survived SIGTERM).
+echo "# a mute server costs only the walks into its own name"
+# A server that accepts the connection but never answers Tversion. The dial
+# runs on that name's worker, inside the walk that asked, so: every other
+# name (and the root) keeps answering; a signal releases the parked walker
+# at once (the dial is abandoned, EINTR); a fresh walk parks again and is
+# just as interruptible; and when the program exits with a walk still
+# parked, 9ns follows it out. Background jobs of a non-interactive sh ignore
+# SIGINT, so the parked walker is sent SIGTERM; the foreground `timeout -s
+# INT` case covers Ctrl-C. Pre-fix the dial ran on the dispatcher thread and
+# parked the whole mount, unkillably, until the program exited.
 if command -v python3 > /dev/null; then
-    python3 - "$REG/mute" <<'PYEOF' &
+    python3 - "$REG/mute" <<'MUTEEOF' &
 import socket, sys, os
 path = sys.argv[1]
 try:
@@ -273,22 +277,45 @@ s.bind(path)
 s.listen(8)
 while True:
     conn, _ = s.accept()  # accept, then never say a word
-PYEOF
+MUTEEOF
     MUTEPID=$!
     PIDS+=($MUTEPID)
     sleep 0.3
+    MUTE_OUT=$(timeout 60 "$NS" --mntgen -- sh -c '
+        stat '"$M"'/mute >/dev/null 2>&1 & W=$!
+        sleep 0.3
+        echo "alpha=$(timeout 5 cat '"$M"'/alpha/build/zig_version)"
+        echo "root=$(timeout 5 ls '"$M"' | grep -c .)"
+        echo "readdir_alpha=$(timeout 5 ls '"$M"'/alpha | grep -c .)"
+        s=$(date +%s%N); kill -TERM $W; wait $W; echo "term_rc=$? term_ms=$(( ($(date +%s%N) - s) / 1000000 ))"
+        s=$(date +%s%N); timeout -s INT 2 stat '"$M"'/mute >/dev/null 2>&1; echo "int_rc=$? int_ms=$(( ($(date +%s%N) - s) / 1000000 ))"
+        echo "alpha_again=$(timeout 5 cat '"$M"'/alpha/build/zig_version)"
+    ' 2>"$TMP/mute.err")
+    expect_eq "another name is served while a walk into mute is parked" "alpha=$(zig version)" "$(printf '%s\n' "$MUTE_OUT" | grep '^alpha=')"
+    ROOT_N=$(printf '%s\n' "$MUTE_OUT" | sed -n 's/^root=//p')
+    [ -n "$ROOT_N" ] && [ "$ROOT_N" -ge 2 ] && pass "the root lists while a walk into mute is parked ($ROOT_N entries)" || fail "the root listing did not answer" "$MUTE_OUT"
+    READDIR_N=$(printf '%s\n' "$MUTE_OUT" | sed -n 's/^readdir_alpha=//p')
+    [ -n "$READDIR_N" ] && [ "$READDIR_N" -ge 1 ] && pass "a readdir on another name is served meanwhile" || fail "readdir on alpha did not answer" "$MUTE_OUT"
+    expect_eq "SIGTERM releases the parked walker (died of the signal)" "term_rc=143" "$(printf '%s\n' "$MUTE_OUT" | sed -n 's/^\(term_rc=[0-9]*\) .*/\1/p')"
+    TERM_MS=$(printf '%s\n' "$MUTE_OUT" | sed -n 's/.*term_ms=//p')
+    [ -n "$TERM_MS" ] && [ "$TERM_MS" -lt 1500 ] && pass "released promptly (${TERM_MS}ms)" || fail "release of the parked walker was slow or missing" "term_ms=$TERM_MS"
+    expect_eq "a fresh walk into mute is interruptible (Ctrl-C after 2s)" "int_rc=124" "$(printf '%s\n' "$MUTE_OUT" | sed -n 's/^\(int_rc=[0-9]*\) .*/\1/p')"
+    INT_MS=$(printf '%s\n' "$MUTE_OUT" | sed -n 's/.*int_ms=//p')
+    [ -n "$INT_MS" ] && [ "$INT_MS" -ge 1900 ] && [ "$INT_MS" -lt 4000 ] && pass "the interrupted walk came back on the signal (${INT_MS}ms)" || fail "interrupted walk timing off" "int_ms=$INT_MS"
+    expect_eq "alpha still served after all that" "alpha_again=$(zig version)" "$(printf '%s\n' "$MUTE_OUT" | grep '^alpha_again=')"
+
     HANG_START=$(date +%s)
     timeout 20 "$NS" --mntgen -- bash -c "(stat $M/mute >/dev/null 2>&1) & sleep 1" >/dev/null 2>&1
     HANG_RC=$?
     HANG_SECONDS=$(( $(date +%s) - HANG_START ))
     if [ "$HANG_RC" -eq 124 ] || [ "$HANG_SECONDS" -ge 15 ]; then
-        fail "9ns unwedges after the program exits a parked dial" "rc=$HANG_RC after ${HANG_SECONDS}s (wedged)"
+        fail "9ns exits with a walk still parked in a dial" "rc=$HANG_RC after ${HANG_SECONDS}s (wedged)"
     else
-        pass "9ns unwedges after the program exits a parked dial (rc=$HANG_RC after ${HANG_SECONDS}s)"
+        pass "9ns exits with a walk still parked in a dial (rc=$HANG_RC after ${HANG_SECONDS}s)"
     fi
     rm -f "$REG/mute"
 else
-    echo "SKIP: mute-server dial-hang check needs python3"
+    echo "SKIP: mute-server checks need python3"
 fi
 
 echo