# computeranim Two things that meet at a file: - **ztrace**, a small tracer/debugger: many ways to watch a program run, one event format, one recording. - **canim**, animations of what was recorded (registers, memory, syscalls, data structures, source), written as code, exported as frames for ffmpeg. ```sh zig build # zig-out/bin/ztrace zig build test # everything; test- for one zig build bench-store # what a recording costs a moment zig build bench-agent -Doptimize=ReleaseFast # what the agent costs the target ztrace record -o fib.ztrace -- ./fib 10 # under ptrace, an instruction a moment ztrace record --by line -w dist -- ./graph # ... or a source line, a global watched ztrace record --syscalls --strace -- ls / # ... or a syscall, printed as strace does ztrace dbg -- ./fib 10 # step, break, look, and go back ztrace run --preload libztrace-agent.so -- ./prog # in-process, no ptrace ztrace cc -- clang sort.c -o sort && ztrace run -- ./sort # compiled in: lines, calls, variables, stores ztrace perf --faults --syscalls --watch counter -- ./prog # the kernel's view, at full speed ztrace probe --dry-run --usdt provider:name -- ./prog # ... with BPF, for root ztrace dump fib.ztrace ``` ## The pieces | Where | What | | --- | --- | | `src/trace/event.zig` | `Event` (64 bytes: time, thread, kind, six words) plus a payload; `Kind`; `Regs`; `Sink` | | `src/trace/Ring.zig` | events from the target to the tracer through shared memory; a producer can be held at a kind (`stop_mask`) until `resumeAll` | | `src/trace/client.zig`, `host.zig` | the ring's two ends: inside the target, inside the tracer | | `src/trace/store.zig` | `Recorder` (a `Sink`) and `Trace`: the recording, in xitdb | | `src/trace/ptrace.zig` | the out-of-process backend: a process and its threads under ptrace | | `src/trace/Debugger.zig` | on top of it: continue, step, breakpoints, hooks, watches, every stop an event | | `src/trace/Symbols.zig`, `syscalls.zig` | what an address means (modules, ELF symbols, DWARF lines); syscalls by name and shape | | `src/agent/`, `include/ztrace.h` | the in-process backend: a library preloaded or linked into the target, and what the target can say through it | | `llvm/pass.cpp`, `src/instr/` | compiler instrumentation: the LLVM pass, the runtime it calls, the type descriptions | | `src/trace/perf.zig`, `bpf.zig`, `usdt.zig` | the kernel's view: perf events for any user, BPF programs for root, the static probes of `` | | `anim/` | canim, a package of its own (it needs sketchbook and pardes; ztrace does not): the animation library and its scenes | | `design/` | the visual system the animations follow, with mock frames | | `src/cli/` | the `ztrace` command, a file a subcommand | | `targets/` | programs to trace | | `build/` | each area's part of the build | Every backend produces `Event`s into a `Sink`. The recorder is one; the ring is another, with the recorder at its far end. ## A recording An xitdb file, kept small. A **moment** stores only its events, encoded compactly (registers as the ones that changed since the thread's last). The program's **state** is stored at a keyframe every 4096 moments, each sharing what did not change with the one before; the state at any moment is the keyframe before it with the moments since replayed over it. `Trace.moment(i)` is any of them, in any order: time travel is an index. | `Moment` | from the events | | --- | --- | | `regs(tid)` | any event carrying `Regs` as its payload | | `read(addr, buf, known)` | `mem_write`: every byte seen so far, and which are known | | `heap()`, `allocation(addr)` | `alloc`, `free`: the live allocations | | `stack(tid)` | `call`, `ret`: the calls a thread is inside | | `variables()` | `variable`: the live named variables; a local goes with its call | | `line(tid)` | `line` | | `mappings()`, `mapping(addr)` | `map`, `unmap` | | `string(id)` | `string` | | `events()` | everything, as it came | From a debugger's recording, `zt.telescope`: `instruction(m, addr, buf)` is an instruction's text, its source `line`, and for a direct call its callee's `params` (DWARF's); `frames(m, regs, out)` is the call chain with each frame's extent on the stack, `low` to `high` (a `stack()` frame's is up to `sp + 8`), walked by the call frame information the recording holds (below), else by frame pointer; `slots(m, regs, low, out)` is what each stack word from `low` is (`out[i]` the word at `low + 8 i`), a `Slot`: its `frame`, an index into `frames` (null in none), and its `role`: `.saved` (the register a function saved for its caller there: `"rbx"`, `"rbp"`, `"r12"` to `"r15"`), `.return_address`, `.local` (a variable or parameter: `name`, `size`, and `offset`, where the word is in it), or the process's first stack, `.argc`, `.argv` i, `.envp` i (i = argc, envc: the zero that ends them) and `.auxv` i (the auxiliary vector's i-th word, two an entry); `.none` where the recording does not say. With `--alloc`, every live allocation and the 16 bytes of allocator header before it are in `read` at each stop (its first 4 KiB; what changed, a stop at a time; 256 KiB of them at most, lowest addresses first: `Debugger.allocation_watch`). Recorded `--by insn`, the instructions that ran between two moments are the `rip` of each moment between them; `--by line` does not keep them. A byte in `read` is as it was when the debugger last sent it: the stack window, the watched regions and the live allocations at every stop, a `read`'s or `write`'s buffer at that syscall (`-s` bytes of it). Memory it looked at once and not since is known and may be stale. The recorder ends a moment at the kinds in `Recorder.cut` (by default a step, a breakpoint, a syscall's exit, a signal, a source line, a mark). Of a memory write it keeps only the bytes that differ from what it already holds, so a backend that reads a region again pays for what changed in it. It stops with `error.RecordingFull` at `max_bytes` (1 GiB): a recording is meant to be megabytes. What a moment costs (`zig build bench-store`; release, this machine): | each moment | bytes | ns | | --- | --- | --- | | a mark | 24 | 290 | | a step with registers | 51 | 750 | | ... and a word pushed on the stack | 65 | 450 | | a word written somewhere in 1 MiB of heap | 320 | 9500 | | 4 KiB written again, unchanged | 370 | 22000 | | 4 KiB of new bytes | 12078 | 545000 | An instruction-level recording of `fib(12)` under ptrace, 142 000 moments with registers and the stack, is 8 MB at 197 000 moments a second. Reading is 6 µs a moment in order and a millisecond to any moment (three at worst). Scattered writes and new bulk data are what cost: each chunk of 64 bytes that changed is written at the next keyframe with its path in the tree. A larger program, to scale by: Lua 5.4 compiled by `ztrace cc` with everything instrumented (lines, calls, variables, stores, in all of its 26 000 lines) and run on a ten-line script that takes it a millisecond. | what is instrumented | moments | events | recording | run | | --- | --- | --- | --- | --- | | everything | 3 248 000 | 7 353 000 | 156 MB | 1.7 s | | `--only luaV_execute` (the interpreter's loop) | 2 176 000 | 4 640 000 | 93 MB | 1.0 s | | `--instr calls` | 189 000 | 191 000 | 7 MB | 0.06 s | A moment a C line of an interpreter is a great many moments: choose what is instrumented (`--only`, `--mem`, `--instr`) before choosing to wait. The other backends on the same Lua, uninstrumented: the agent's allocations and syscalls (368 allocations, no ptrace), breakpoints on `luaH_resize` (48 hits), source lines from DWARF (53 000 a second), counters and page faults from perf (10.8 million instructions, 89 faults). And pardes drawing an image, through its own harness (`zig build image-harness` there, the command it runs given to `ztrace run --preload ... --syscalls --capture 2048`): the harness and the pardes it starts are one recording, 9 threads, and the kitty graphics protocol is there to read in the captured writes: pardes asks (`_Gi=1,a=q`), the terminal answers (`_Gi=1;OK`), pardes sends the pixels (`_Gf=32,s=2,v=2,i=1;DU3n/w...`). Two limits it shows: a recording's state is one address space, so a child's memory and mappings share its parent's maps (its events are told apart by thread); and pardes allocates through its own heaps, so `--alloc` sees only what its C libraries ask of libc. And V8, as the test of scale: a release `d8` (74 MB, 46 threads, a JIT), on a script it runs in about 50 ms. | backend | what it did | time | | --- | --- | --- | | `ztrace record --syscalls` (ptrace) | 2 951 moments | 0.13 s | | `ztrace record --by insn --steps 2000000` | 2 million instructions, 96 MB (47 bytes a moment) | 9.4 s | | `ztrace record --by break -b malloc` | 1 753 stops | 0.12 s | | `ztrace dbg`, a breakpoint in `Heap::CollectGarbage` | an 18-frame backtrace through JIT code to `Shell::Main` | | | `ztrace perf --faults --sample 4000 --threads` | 784 million instructions, 10 669 faults, 474 samples | 0.07 s | | `ztrace run --preload ... --syscalls` | 2 900 syscalls, in the process | 0.06 s | | `ztrace run --preload ... --hook malloc` | 1 990 calls and returns | 0.07 s | V8 found what the small programs could not. Its JIT guards code with memory protection keys, and a signal handler starts with none of them: a compiler thread cloned from inside the agent's syscall handler could not read its own code space, and d8 died. The handlers now act with the thread's keys and float modes (`src/agent/signal.zig`). `--alloc` sees nothing in d8, which binds its own `malloc`: hook it instead. C++ names are shown and matched as the source has them, without their parameters (`-b v8::internal::Heap::CollectGarbage`; the mangled name works too): the command links zig's libc++ for its demangler. ## SQLite Between Lua and V8 on the ladder: SQLite 3.50.4's amalgamation (one C file of 263 000 lines), its shell built by clang `-O2 -g -fno-omit-frame-pointer`, on a file database: two tables and an index, 5 100 rows inserted in a transaction, a count, a join with an aggregate, a lookup through the index. Alone it takes 5.5 ms, 51 million instructions and 321 syscalls. `sh bench/sqlite.sh PREFIX SRC` builds the shells and makes each row (a release ztrace installed at PREFIX; time is the tracer's own measure of the program's run, the best of three; seek is `ztrace show` of the middle moment: the file opened, the moment replayed from its keyframe, printed). | backend | moments | events | recording | a moment | time | × alone | moments/s | events/s | seek | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `record --by syscall` | 322 | 3 593 | 0.3 MB | 830 B | 21 ms | 3.9 | 15 100 | 169 000 | 1.4 ms | | `record --by line`, its first million lines | 1 000 000 | 2 843 000 | 108 MB | 107 B | 31 s | | 32 000 | 91 000 | 6.8 ms | | `record --by line --max-size 2G`, all of it | 16 890 000 | 46 462 000 | 1.7 GB | 100 B | 504 s | 92 000 | 33 500 | 92 000 | 13 ms | | `record --by break -b sqlite3BtreeInsert` | 15 539 | 82 401 | 3.5 MB | 223 B | 440 ms | 80 | 35 300 | 187 000 | 4.3 ms | | `run --preload ... --alloc` | 1 | 1 353 | 36 KB | | 6.4 ms | 1.2 | | 211 000 | 1.1 ms | | `run --preload ... --syscalls` | 282 | 601 | 45 KB | 161 B | 7.0 ms | 1.3 | 40 300 | 86 000 | 3.9 ms | | `run --preload ... --hook sqlite3BtreeInsert` | 30 434 | 30 473 | 2.7 MB | 88 B | 25 ms | 4.5 | 1 240 000 | 1 240 000 | 3.0 ms | | `perf --faults --syscalls` | 321 | 889 | 0.1 MB | 160 B | 19 ms | 3.5 | 16 800 | 46 500 | 1.3 ms | | `perf --break sqlite3BtreeInsert` | 2 | 15 258 | 2.0 MB | | 25 ms | 4.6 | | 603 000 | 2.5 ms | | `cc` (everything), `run` | 21 789 000 | 40 825 000 | 1.06 GB | 48 B | 16.3 s | 2 960 | 1 340 000 | 2 511 000 | 45 ms | | `cc --only sqlite3VdbeExec`, `run` | 5 873 000 | 9 401 000 | 221 MB | 37 B | 2.5 s | 457 | 2 335 000 | 3 737 000 | 6.5 ms | | `cc --instr calls`, `run` | 2 925 000 | 2 930 000 | 98 MB | 33 B | 1.05 s | 191 | 2 777 000 | 2 782 000 | 2.0 ms | | `cc --instr calls`, `record --by break -b sqlite3BtreeInsert --alloc --preload ...` | 15 626 | 3 167 770 | 53 MB | 3 379 B | 1.16 s | 211 | 13 400 | 2 726 000 | 4.8 ms | Built by `ztrace cc` and run with nobody listening, the shell takes 108 ms with everything instrumented, 23 with `--only sqlite3VdbeExec`, 11 with `--instr calls`. `ztrace dbg` with `b sqlite3VdbeExec`, `r`, `bt` gives 19 frames, from the interpreter through `sqlite3InitOne`, the parser's `yy_reduce` and `sqlite3_prepare_v2` to the shell's `process_input` and `_start`; `b sqlite3.c:80895` is `sqlite3BtreeInsert`. Under ptrace with the allocations told (the last row), each stop compares the 155 live allocations, 78 KB of them with their headers: 3 379 bytes a moment and 1.16 s, against 3 098 and 0.63 s before allocations were read. SQLite found two things. A file's name in DWARF may be absolute (the amalgamation compiled by its full path), and the debugger joined it to the compilation's directory anyway: every `line` named a file that was not there. And `ztrace run` noticed a target's exit only when its idle wait of 10 ms ran out, so a 5 ms program took 15; it waits a millisecond now. Recorded a line a moment, the whole run is past the gigabyte a recording is allowed by default. ## Together The backends are not modes to choose between: they feed one sink. A program compiled by `ztrace cc`, with the agent preloaded, run under the debugger, gives one recording, in order: the stops and their registers (ptrace), the allocations (the agent), the lines, calls, stores and named variables (the pass). ```sh ztrace cc -- clang -O0 targets/instr_list.c -o list ztrace record --by break -b push -b reverse --alloc --preload zig-out/lib/libztrace-agent.so -- ./list ztrace dbg --ring -- ./list # breakpoints and time travel over what the program says of itself ztrace show out.ztrace 40 # the thread's line and calls, the live heap, each variable's bytes ztrace perf --faults --syscalls --alloc --preload zig-out/lib/libztrace-agent.so -- ./list # the kernel's view too: each malloc, the brk it makes, the faults of its first touch zig build test-together ``` The cleanest machine-level recording with names in it is `ztrace cc --instr vars` under the debugger: the pass declares each variable (name, type, address) and instruments nothing else, the debugger watches every variable declared through the ring and records its bytes at each stop, and the lines, registers, disassembly and memory touches are the debugger's own, of code that is otherwise the program's. ```sh ztrace cc --instr vars -- clang -O0 targets/instr_sort.c -o sort ztrace record --by line --ring -- ./sort && ztrace show out.ztrace 140 unsorted: [8]i32 = 05 00 00 00 02 00 00 00 ... i: i32 = 02 00 00 00 ``` Static probes (USDT, DTrace's) need neither BPF nor root this way: `ztrace record --by break -b 'usdt:libc:*' -- ./prog` arms each probe as a breakpoint that records a `probe` event with its arguments and runs on. glibc ships 78 of them (`memory_sbrk_more`, `setjmp`, `lll_lock_wait`...); `` puts your own into a program. `--ring` (implied by `--preload` and `--alloc`) gives a target under ptrace the ring `ztrace run` gives it; the debugger drains it while the target runs and before it reports a stop. The agent's hooks and syscall tracing are signals, which a debugger takes first: under ptrace, use breakpoints and the debugger's own syscall stops. Strings never collide: the pass counts its from 1, a symbol's name has its address as its id, and every other producer uses `event.stringId`, a hash with the top bit set. ## Context A viewer of a recording should be able to show what pwndbg's context shows, so the debugger records what that takes while the memory is there to read. At every stop it follows each register and the top sixteen words of the stack as pointers, three deep (`--telescope N`; 0 for not at all): the memory they lead to is recorded, every code address met gets a name (the string whose id is the address: `fib+40`, `libc.so.6!write+18`), and the instructions round the pc are disassembled by capstone, eight above and eight below it (strings with id `event.codeId(addr)`; inside a function the ones above are known, the function decoded once from its start). `zt.telescope` reads it back: `chase` gives a value's chain, each link with the kind of memory it points into (code, stack, heap, data, read-only, writable-and-executable: what a pointer is coloured by), a name for code and a string where the chain ends on one. ``` $ ztrace record --by break -b fib -- ./fib 5 && ztrace show out.ztrace 34 rbp 7ffc63068910 —▸ 7ffc63068930 —▸ 7ffc63068950 —▸ 7ffc63068970 —▸ 7ffc63068990 r12 7ffc63068ae8 —▸ 7ffc63069f22 ◂— "/tmp/..." rip 1001794 (fib+4) ► 000001001794 push r14 000001001796 push rbx 000001001797 mov ebx, edi 00:0000│ rsp 7ffc63068910 —▸ 7ffc63068930 —▸ 7ffc63068950 —▸ ... 01:0008│ 7ffc63068918 —▸ 10017b8 (fib+40) 02:0010│ 7ffc63068920 ◂— 3 ``` A stop also says what its instruction is about to do to memory, worked out from capstone's operands and the registers: a `touch` event (the address, the size, read or write) with what is there, and after the step what a write left there. So an instruction-level recording has every memory write the stepped thread makes, with nothing watched and nothing compiled in. A call, a jump or a return at the pc is a `branch` event: where it goes, and for a conditional jump whether it will. **The call chain** is walked at every stop by the unwind tables (`.eh_frame`, `.debug_frame`, through `std.debug.Dwarf.Unwind`), as gdb walks it: right inside a function that keeps no frame (a libc wrapper at a syscall, malloc's fast path), in a prologue, in a PLT entry; by frame pointer where there are none (JIT code, a PLT Zig's linker made). What the walk reads of the stack is recorded, its return addresses named, and each function it goes through once, as a string (`event.frameId` of where it begins): its rows of call frame information (where its caller's stack pointer is, the CFA, and where it saved each register, a row for each stretch of its code) and, from DWARF, its frame base and the locals the debug info places by it (`DW_OP_fbreg`), each with its size and its block's range of code. `zt.telescope.frames` walks the same chain again from the recording, which needs no ELF file, and `slots` names the words: the saved registers and return addresses of every frame; the locals of -O0 and -Og code where the frame's pc is in their block (a leaf's in the red zone too); and argc, argv, envp and the auxiliary vector, from where the stack began (the string `event.frameId(0)`, `Debugger.Start`, read from `/proc/pid/stat`). What it does not say: a local in a register or a list of places (-O1 and up), or in a block whose code is in pieces (`DW_AT_ranges`); a local's name in a prologue or an epilogue where the frame base is rbp and rbp is not yet or no longer the frame's; one of two small locals sharing a word (the first keeps it); a frame whose CFA is an expression (a signal frame) or that has no unwind tables, which falls back on the frame pointer; inlined calls, which have no frame (gdb lists them). What it costs, on `fib(12)` by instruction (142 000 moments): 154 000 moments a second and 64 bytes a moment with the disassembly and the touches, 57 000 and 77 bytes with the telescope and the call chain. The call chain by unwind tables costs nothing a second over the frame pointer's (the walk reads the stack once and remembers each pc's row) and 1 to 4 bytes a moment: on the list of `anim/tools` by instruction (`--from main --stack 1024 --around 80 --disasm 48`, -O0) 229 bytes a moment against 225, 16 000 moments a second either way; SQLite by breakpoint on `sqlite3BtreeInsert`, 310 bytes and 33 000 a second, as before. Capstone is built from source for x86 (`build/capstone.zig`), into the command only: the library takes a disassembler as a function (`Debugger.disasm`) and links none. ## The debugger `ztrace dbg -- PROGRAM` starts a program stopped and reads commands (from a script when stdin is no terminal): `run`/`r`, `continue`/`c`, `step`/`s` (a source line), `stepi`/`si`, `break`/`b` (a function, `file:line`, `*address`), `syscalls on|off|NAME`, `watch`, `regs`, `x`, `bt`, `quit`. Every stop is a moment of a recording, and `back [N]`, `forward [N]` and `goto MOMENT` look into it: `regs`, `x` and `bt` then answer from the recording, while the program stays where it stopped. ``` $ printf 'b fib\nr\nbt\ns\nback 1\nx $rsp 16\n' | ztrace dbg -- zig-out/targets/fib 10 [0] step at 7f82d095e7e0 ld-linux-x86-64.so.2!217e0 (ztrace) b fib breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) r [1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) bt #0 563f798b1814 fib+4 (fib.c:4) #1 563f798b17ea main+42 (fib.c:10) #2 7f82d0627781 libc.so.6!27781 #3 7f82d06278b9 libc.so.6!__libc_start_main+137 #4 563f798b16e5 _start+37 (ztrace) s [2] step at 563f798b181c fib+12 (fib.c:5) (ztrace) back 1 [1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) x $rsp 16 7ffc89ca6570 90 65 ca 89 fc 7f 00 00 ea 17 8b 79 3f 56 00 00 .e.........y?V.. (ztrace) out.ztrace: 3 moments ``` `ztrace record` is the same debugger left to run: `--by insn|line|syscall|break` says where it stops, `-b` adds breakpoints, `-w` watches, `--strace` prints the syscalls, `--from main` runs through the loader and the C library's start before the first stop. How much of the context a stop keeps is asked for: `--stack BYTES` above the stack pointer (128), every word of it followed as a pointer; `--around BYTES` of memory each side of what a pointer leads to (0: the 32 bytes at it); `--disasm ROWS` of instructions above and below the pc (8). On the list of `anim/tools` by instruction, `--stack 512 --around 64 --disasm 24` is 184 bytes a moment against 159, and 27 000 moments a second against 45 000. In a program, it is `zt.Debugger`: ```zig var d = try zt.Debugger.spawn(gpa, io, path, argv, envp, recorder.sink()); defer d.deinit(); d.syscalls = .record; // or .stop, or a hook: d.on_syscall try d.watch(d.symbol("dist").?.addr, 8); // recorded at every stop, the changed words only _ = try d.breakAt(try d.resolve("list_done"), .{ .func = hook }); // a hook says .go or .stop while (true) switch (try d.step()) { // or stepi(), cont() .exited => break, else => {}, // d.regs, d.read, d.write, d.setRegs, d.snapshot }; ``` Of memory it records what is asked for and no more: the watched regions, a window above the stack pointer, and at most 256 bytes of what a syscall points at (`-s`), each as the 8-byte words that changed since the stop before. What it costs (`zig build bench-debugger`; release, this machine): | | a second | µs | bytes a moment | | --- | --- | --- | --- | | an instruction: ptrace step and wait | 273 000 | 3.7 | | | ... as `Debugger.stepi` with events, the source line looked up | 260 000 | 3.9 | | | ... and 128 bytes of stack watched | 212 000 | 4.7 | | | ... recorded (`ztrace record`) | 189 000 | 5.3 | 55 | | a source line (16 instructions here), recorded | 16 000 | 62 | 60 | | a breakpoint hit, recorded | 82 000 | 12 | 94 | | a syscall, entry and exit: native 0.1 µs; stopped at | 120 000 | 8.3 | | | ... recorded (`--by syscall`) | 100 000 | 10 | 208 | Memory is read with `process_vm_readv`: 64 bytes in 0.3 µs where `PTRACE_PEEKDATA` takes 2.2, a megabyte at 22 GiB/s against 28 MiB/s. It is written the same way, and through `/proc/pid/mem` where the page is read-only (a breakpoint in code: 0.5 µs). Limits: `bt` walks the unwind tables, and where there are none (JIT code) frame pointers, where a function without a frame hides its caller. Threads are followed, each stop is one thread's and the others run on; stepping by line in a threaded program reports the other threads' instructions too. A target that execs is not followed into the new program. ## Compiled in For a program built from source, `ztrace cc` runs clang with a pass plugin (`llvm/pass.cpp`, built against the system LLVM) and links a runtime (`src/instr/rt.zig`, `libztrace-rt.a`). The program then tells its own story at the level of its source, names and types included: no DWARF is read at view time. ```sh ztrace cc -- clang -O0 targets/instr_sort.c -o sort && ztrace run -- ./sort ztrace cc --only 'sqlite3Btree*,*/pager.c' --mem '*Insert*' -- clang -c sqlite3.c zig build test-llvm # fib, sorts, a list, a BFS: asserted from their recordings zig build bench-llvm -Doptimize=ReleaseFast # the tables these numbers come from ``` | `--instr` | event | bytes recorded | | --- | --- | --- | | `lines` | `line` (file, line, column) before each new line of a block: a moment each | 31 | | `calls` | `call` (name, site, sp, type) at entry, `ret` (name, value) at exit | 14 | | `vars` | `variable` (address, size, name, type) per local at entry, per global at start | 15 | | `stores` | `mem_write` with the bytes after each store; a memset or memcpy is one write | 14 | | `loads` | `mem_read` after each load | 12 | | `edges`, `cmps`, `cyg` | stock clang, no plugin and no names: `edge`, `cmp`, `call`/`ret` by address | 12, 13, 19 | The default is the first four: 44 to 63 bytes a source line executed, of which the stores are 9 to 14 and the moment a line is 22. Stores are what grows, so they are what to select: `--only` is where anything is instrumented (globs on function names, source paths, file names), `--mem` where stores and loads are, within that. The compiler's own stack temporaries are never recorded; a write records its first 64 KiB; a local is declared, not dumped, and a global is written once, at the start. - **Names.** Every file, function, variable and type is a `string` event before it is used. The runtime numbers them when the program starts, so an id is a byte in the recording. - **Types.** A variable's type is a description a viewer can draw from (`src/instr/types.zig` has the grammar and its parser): `[8]i32`, `` *`struct node` ``, and under `types.defId("struct node")` the string `` struct(16){value@0:i32;next@8:*`struct node`;} ``. - **Before the optimizer** by default: every local still has an address, every source store exists, lines come in source order, and what the calls made observable the optimizer must keep, so `-O2` records the same events as `-O0`. `--late` instruments what the optimizer left: the real stores and loads, few variables. - **Stepping.** A `line` is an event like any other, so the ring's stop mask holds the program at each (`host.Options.stop_mask`): source-level step with no ptrace, 1.4 million stops a second with a tracer that polls, 290 000 with one that sleeps in between. - **Cost.** An event is 60 to 70 ns into the ring, whatever its kind (nearly all of it the ring's two contended cache lines), and 1.5 ns when nobody listens. A line of these targets is a few ns of work, so with the default selection they run 7 to 12 times slower untraced and 250 to 400 times slower traced at `-O0`. Recording at a moment a line takes 1.9 million events a second. ## The agent `libztrace-agent.so` is preloaded into a program (`ztrace run --preload`); `libztrace-agent.a` is linked into one, which can then say things about itself (`include/ztrace.h`; a Zig program imports the `ztrace-agent` module instead). Inside the target it has no libc and makes raw syscalls only, so the same library serves glibc and musl, dynamic and static. With no tracer it does nothing: a preloaded malloc costs 2 ns more. ```sh ztrace run --preload zig-out/lib/libztrace-agent.so --alloc -- ./agent_list 100 ztrace run --preload ... --syscalls -- ls / ztrace run --preload ... --hook fib -- ./agent_fib 10 # ztrace show FILE 6: the call stack then ztrace run --preload ... --hook swap --watch numbers -- ./agent_sort ztrace run --preload ... --hook swap --stop-at ret --read numbers -- ./agent_sort ztrace run --preload ... --hook fib --stop-at call -i -- ./agent_fib 5 ztrace run -- ./agent_api # the agent linked in, annotating itself zig build test-agent zig build bench-agent -Doptimize=ReleaseFast ``` | | events | how | | --- | --- | --- | | startup | `map` a mapping, `thread_start` a thread | /proc/self/maps; pthread_create and fork interposed | | `--alloc` | `alloc`, `free`, with the caller's pc | malloc and its family interposed (`dlsym(RTLD_NEXT)`; `--wrap` when linked statically) | | `--syscalls` | `syscall_enter` (path, bytes written: `--capture`, 256), `syscall_exit` | Syscall User Dispatch: SIGSYS, handled in the target | | `--hook SYM` | `call` with the registers, `ret` with the value | int3 at the entry, the return address swapped | | `--watch SYM` | `mem_write`: the bytes that changed, before each event that ends a moment | compared in place against a shadow copy | | `ztrace_mark`, `ztrace_watch`, `ztrace_snapshot` | `mark`; `variable` and the watch; `mem_write` | the target's own calls | | `--stop-at KIND` | the target held at each, while the tracer reads (`--read`) or asks (`-i`) | the ring's `stop_mask`, a futex | Why these. **Hooks** are breakpoints because a breakpoint needs to know nothing about the code it lands on: an inline jump needs an x86-64 length decoder to move the prologue, and patching the PLT misses calls inside a module. The price is a signal a call; the return costs none. **Watches** compare memory in the target's own address space because that is exact at any size, needs no syscall, and does not disturb the target: page protection makes the kernel refuse syscalls that write there, and soft-dirty bits still need the compare, a page at a time. **Syscalls** go through SIGSYS because the kernel then needs no second process: the cost is two signal frames, on a stack of the agent's so that it works on threads with almost none. A handler starts with the kernel's defaults, not the thread's rights to memory behind protection keys (PKRU) nor its floating-point modes; the agent's handlers take the thread's back first, so that a syscall made for it, a thread cloned for it and a watched page read for it are as the thread would have them (V8 keeps its code space behind a key). Measured (`zig build bench-agent -Doptimize=ReleaseFast`, an i7-11700, each loop 10^5 to 5 10^6 times; bytes are the recording's): | | native | traced | recorded | | --- | --- | --- | --- | | a malloc and a free, `--alloc` | 4 ns | 168 ns (11.7 M events/s) | 14 bytes an event | | ... each a moment (`--cut alloc,free`) | | 4.9 µs, the recorder's | 65 bytes a moment | | a syscall, `--syscalls` | 73 ns | 950 ns; 830 ns is the two signal frames | 87 bytes a syscall, a moment | | a syscall under ptrace (`ztrace record --syscalls`) | | 9000 ns | 129 bytes | | a hooked call | 1 ns | 990 ns | 77 bytes: two moments, with registers | | ... watching 1 KiB, 64 KiB, 1 MiB | | +43 ns, +1.5 µs, +17.5 µs a moment | what changed | | ... held at its return (`--stop-at ret`) | | +1.3 µs | | | ... the tracer reading 1 KiB, 64 KiB, 1 MiB from outside | | +2.5 µs, +6.5 µs, +66 µs a stop | what changed | | events that end no moment, through the ring | | 11 M a second | 8 bytes an event | | marks, each a moment | | 2.5 M a second | 32 bytes a moment | `ztrace run` ends a moment at the recorder's kinds and at `call` and `ret` (hooks are asked for by name, and a call and its return are the steps of such a recording); `alloc` and `free` end none unless `--cut` says so. A recording that reaches `--max-size` (1G) ends the run. Limits: a hooked function called with SIGTRAP blocked dies (not under `--syscalls`, which keeps the target's masks clear of it); C++ exceptions do not unwind through a hooked call; a pointer the kernel would answer with EFAULT faults in the handler; syscalls made from the vDSO never enter the kernel and are not seen, as under ptrace; a target that needs SIGSYS or its own alternate signal stack is told it has them. `--alloc` sees calls that reach libc's allocator by name: a program that brings its own `malloc` and binds it inside itself (d8 does) shows none, and `--hook malloc` is the way to see those. A target that dies under the agent dumps core like any other, which for a V8 takes half a minute; `ulimit -c 0` if that is not wanted. ## The kernel's view Two tiers, by what a machine lets its user do. Both hold the program at its first instruction until they are attached, then let it run at full speed: no stops, records out of a kernel ring buffer, stamped with the clock every other backend uses. **`ztrace perf`** needs no privilege: `perf_event_open` as `kernel.perf_event_paranoid=2` allows it, which is user space only, on one's own processes. `perf.Session.open(gpa, pid, options, sink)` is the same for a tracer that has its own target (stopped, or it misses the start). | option | events | costs the program | bytes an event | | --- | --- | --- | --- | | `-e`, `--interval MS` | `sample`: a counter's total | nothing measurable | 27 | | `--sample HZ`, `--callchain` | `sample`: pc, callers | 3% at 40 kHz with callers | 24, 48 | | `--faults` | `fault` and the registers | 0.3 us a fault (the fault itself: 0.9) | 40 | | `--switches` | `sched` | nothing measurable | 10 | | `--syscalls` | `syscall_enter` with the path or bytes, `syscall_exit` | 0.4 us a syscall (ptrace: 10), and a second at the end | 23 | | `--watch SYMBOL` | `mem_write` or `mem_read`, and the pc | 0.9 us a hit | 17 | | `--break SYMBOL` | `probe`, five arguments, the registers | 1.3 us a hit | 73 | | always | `map`, `thread_start`, `thread_exit`, `mark` (comm), `exit` | | | (Release build, an i7-11700; a ptrace stop is 5 us. It decodes and records one to four million events a second; what the kernel could not put in its ring is counted as lost: `--ring`.) `--threads` follows threads and children, a ring per CPU merged by time. Moments end where the recorder ends them anyway, a syscall's exit or a megabyte of events, unless `--cut` asks for one at every fault and hit. What to know: - Only user space is seen. A page the kernel touches first (`read()` into fresh memory) faults unseen; context switches and migrations cannot be counted, only reported as events. - A huge page is one fault for 2 MiB. An anonymous mapping the kernel merges with its neighbour is reported as the merged whole. - Syscalls come from the 736 per-syscall tracepoints, the only ones a user may read: found by trying every id, closed at a millisecond each. - There are four debug registers; a watchpoint covers 1, 2, 4 or 8 aligned bytes; the record comes after the access, so the value is read when it is decoded, later (or not at all, once the program is gone). **`ztrace probe`** is for root (CAP_BPF and CAP_PERFMON) and says so otherwise; `--dry-run` builds everything and lists it. BPF programs, built here as instruction arrays for one process, on the raw tracepoints `sys_enter`, `sys_exit` and `sched_switch` (both threads of a switch, through the kernel's BTF), on kprobes, on uprobes, and on USDT probes, whose arguments are read in the kernel at the probe (the program's own semaphores are raised by writing them: the kernel's counter misses when lld has mapped that page of the file twice). They write `Event`s into one BPF ring buffer and count what did not fit. A syscall costs 0.15 us, a probe 0.5 to 0.8 (in a VM), 22 to 73 bytes an event. **USDT**, the static probes of DTrace: `usdt.probes(gpa, elf, bias)` reads the notes `` leaves (provider, name, pc, semaphore, arguments) and `usdt.Arg.parse(spec).fetch(regs, memory)` gets an argument at a stopped thread. No BPF in it: a ptrace breakpoint on the nop does as well. `zig build test-kernel` runs the first tier and USDT on `targets/kernel_*`, and the BPF programs in a small BPF machine. `zig build test-kernel-vm` runs them for real: this machine's kernel in QEMU, where we are root. ## Depends on - [sketchbook](../sketchbook): window, drawing, shaders, hot reload, the manim-style `anim` module, frame export. - [pardes](../02-pardes-code): tree-sitter parsing, highlighting, themes. - [xitdb](https://github.com/xit-vcs/xitdb): the recording. - [capstone](https://github.com/capstone-engine/capstone): disassembly, in the command. x86-64 Linux, Zig 0.17.