# computeranim Two things that meet at a file: - **ztrace**, a small tracer/debugger: many ways to watch a program run, one event format, one recording. - **canim**, animations of what was recorded (registers, memory, syscalls, data structures, source), written as code, exported as frames for ffmpeg. ```sh zig build # zig-out/bin/ztrace zig build test # everything; test- for one zig build bench-store # what a recording costs a moment zig build bench-agent -Doptimize=ReleaseFast # what the agent costs the target ztrace record -o fib.ztrace -- ./fib 10 # under ptrace, an instruction a moment ztrace record --by line -w dist -- ./graph # ... or a source line, a global watched ztrace record --syscalls --strace -- ls / # ... or a syscall, printed as strace does ztrace dbg -- ./fib 10 # step, break, look, and go back ztrace run --preload libztrace-agent.so -- ./prog # in-process, no ptrace ztrace cc -- clang sort.c -o sort && ztrace run -- ./sort # compiled in: lines, calls, variables, stores ztrace dump fib.ztrace ``` ## The pieces | Where | What | | --- | --- | | `src/trace/event.zig` | `Event` (64 bytes: time, thread, kind, six words) plus a payload; `Kind`; `Regs`; `Sink` | | `src/trace/Ring.zig` | events from the target to the tracer through shared memory; a producer can be held at a kind (`stop_mask`) until `resumeAll` | | `src/trace/client.zig`, `host.zig` | the ring's two ends: inside the target, inside the tracer | | `src/trace/store.zig` | `Recorder` (a `Sink`) and `Trace`: the recording, in xitdb | | `src/trace/ptrace.zig` | the out-of-process backend: a process and its threads under ptrace | | `src/trace/Debugger.zig` | on top of it: continue, step, breakpoints, hooks, watches, every stop an event | | `src/trace/Symbols.zig`, `syscalls.zig` | what an address means (modules, ELF symbols, DWARF lines); syscalls by name and shape | | `src/agent/`, `include/ztrace.h` | the in-process backend: a library preloaded or linked into the target, and what the target can say through it | | `llvm/pass.cpp`, `src/instr/` | compiler instrumentation: the LLVM pass, the runtime it calls, the type descriptions | | `src/trace/kernel*` | perf events and BPF | | `anim/` | canim, a package of its own (it needs sketchbook and pardes; ztrace does not): the animation library and its scenes | | `design/` | the visual system the animations follow, with mock frames | | `src/cli/` | the `ztrace` command, a file a subcommand | | `targets/` | programs to trace | | `build/` | each area's part of the build | Every backend produces `Event`s into a `Sink`. The recorder is one; the ring is another, with the recorder at its far end. ## A recording An xitdb file, kept small. A **moment** stores only its events, encoded compactly (registers as the ones that changed since the thread's last). The program's **state** is stored at a keyframe every 256 moments, each sharing what did not change with the one before; the state at any moment is the keyframe before it with the moments since replayed over it. `Trace.moment(i)` is any of them, in any order: time travel is an index. | `Moment` | from the events | | --- | --- | | `regs(tid)` | any event carrying `Regs` as its payload | | `read(addr, buf, known)` | `mem_write`: every byte seen so far, and which are known | | `heap()`, `allocation(addr)` | `alloc`, `free`: the live allocations | | `stack(tid)` | `call`, `ret`: the calls a thread is inside | | `variables()` | `variable`: the live named variables; a local goes with its call | | `line(tid)` | `line` | | `mappings()`, `mapping(addr)` | `map`, `unmap` | | `string(id)` | `string` | | `events()` | everything, as it came | The recorder ends a moment at the kinds in `Recorder.cut` (by default a step, a breakpoint, a syscall's exit, a signal, a source line, a mark). Of a memory write it keeps only the bytes that differ from what it already holds, so a backend that reads a region again pays for what changed in it. It stops with `error.RecordingFull` at `max_bytes` (1 GiB): a recording is meant to be megabytes. What a moment costs (`zig build bench-store`; release, this machine): | each moment | bytes | ns | | --- | --- | --- | | a mark | 29 | 560 | | a step with registers | 57 | 1100 | | ... and a word pushed on the stack | 75 | 960 | | a word written somewhere in 1 MiB of heap | 717 | 14700 | | 4 KiB written again, unchanged | 376 | 24000 | | 4 KiB of new bytes | 12078 | 584000 | An instruction-level recording of `fib(12)` under ptrace, 142 000 moments with registers and the stack, is 8 MB at 197 000 moments a second; reading it back is 178 000 moments a second in order, a millisecond to any one. Scattered writes and new bulk data are what cost: each chunk of 64 bytes that changed is written at the next keyframe with its path in the tree. ## The debugger `ztrace dbg -- PROGRAM` starts a program stopped and reads commands (from a script when stdin is no terminal): `run`/`r`, `continue`/`c`, `step`/`s` (a source line), `stepi`/`si`, `break`/`b` (a function, `file:line`, `*address`), `syscalls on|off|NAME`, `watch`, `regs`, `x`, `bt`, `quit`. Every stop is a moment of a recording, and `back [N]`, `forward [N]` and `goto MOMENT` look into it: `regs`, `x` and `bt` then answer from the recording, while the program stays where it stopped. ``` $ printf 'b fib\nr\nbt\ns\nback 1\nx $rsp 16\n' | ztrace dbg -- zig-out/targets/fib 10 [0] step at 7f82d095e7e0 ld-linux-x86-64.so.2!217e0 (ztrace) b fib breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) r [1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) bt #0 563f798b1814 fib+4 (fib.c:4) #1 563f798b17ea main+42 (fib.c:10) #2 7f82d0627781 libc.so.6!27781 #3 7f82d06278b9 libc.so.6!__libc_start_main+137 #4 563f798b16e5 _start+37 (ztrace) s [2] step at 563f798b181c fib+12 (fib.c:5) (ztrace) back 1 [1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) x $rsp 16 7ffc89ca6570 90 65 ca 89 fc 7f 00 00 ea 17 8b 79 3f 56 00 00 .e.........y?V.. (ztrace) out.ztrace: 3 moments ``` `ztrace record` is the same debugger left to run: `--by insn|line|syscall|break` says where it stops, `-b` adds breakpoints, `-w` watches, `--strace` prints the syscalls. In a program, it is `zt.Debugger`: ```zig var d = try zt.Debugger.spawn(gpa, io, path, argv, envp, recorder.sink()); defer d.deinit(); d.syscalls = .record; // or .stop, or a hook: d.on_syscall try d.watch(d.symbol("dist").?.addr, 8); // recorded at every stop, the changed words only _ = try d.breakAt(try d.resolve("list_done"), .{ .func = hook }); // a hook says .go or .stop while (true) switch (try d.step()) { // or stepi(), cont() .exited => break, else => {}, // d.regs, d.read, d.write, d.setRegs, d.snapshot }; ``` Of memory it records what is asked for and no more: the watched regions, a window above the stack pointer, and at most 256 bytes of what a syscall points at (`-s`), each as the 8-byte words that changed since the stop before. What it costs (`zig build bench-debugger`; release, this machine): | | a second | µs | bytes a moment | | --- | --- | --- | --- | | an instruction: ptrace step and wait | 273 000 | 3.7 | | | ... as `Debugger.stepi` with events, the source line looked up | 260 000 | 3.9 | | | ... and 128 bytes of stack watched | 212 000 | 4.7 | | | ... recorded (`ztrace record`) | 189 000 | 5.3 | 55 | | a source line (16 instructions here), recorded | 16 000 | 62 | 60 | | a breakpoint hit, recorded | 82 000 | 12 | 94 | | a syscall, entry and exit: native 0.1 µs; stopped at | 120 000 | 8.3 | | | ... recorded (`--by syscall`) | 100 000 | 10 | 208 | Memory is read with `process_vm_readv`: 64 bytes in 0.3 µs where `PTRACE_PEEKDATA` takes 2.2, a megabyte at 22 GiB/s against 28 MiB/s. It is written the same way, and through `/proc/pid/mem` where the page is read-only (a breakpoint in code: 0.5 µs). Limits: `bt` walks frame pointers, so a function without a frame hides its caller. Threads are followed, each stop is one thread's and the others run on; stepping by line in a threaded program reports the other threads' instructions too. A target that execs is not followed into the new program. ## Compiled in For a program built from source, `ztrace cc` runs clang with a pass plugin (`llvm/pass.cpp`, built against the system LLVM) and links a runtime (`src/instr/rt.zig`, `libztrace-rt.a`). The program then tells its own story at the level of its source, names and types included: no DWARF is read at view time. ```sh ztrace cc -- clang -O0 targets/instr_sort.c -o sort && ztrace run -- ./sort ztrace cc --only 'sqlite3Btree*,*/pager.c' --mem '*Insert*' -- clang -c sqlite3.c zig build test-llvm # fib, sorts, a list, a BFS: asserted from their recordings zig build bench-llvm -Doptimize=ReleaseFast # the tables these numbers come from ``` | `--instr` | event | bytes recorded | | --- | --- | --- | | `lines` | `line` (file, line, column) before each new line of a block: a moment each | 31 | | `calls` | `call` (name, site, sp, type) at entry, `ret` (name, value) at exit | 14 | | `vars` | `variable` (address, size, name, type) per local at entry, per global at start | 15 | | `stores` | `mem_write` with the bytes after each store; a memset or memcpy is one write | 14 | | `loads` | `mem_read` after each load | 12 | | `edges`, `cmps`, `cyg` | stock clang, no plugin and no names: `edge`, `cmp`, `call`/`ret` by address | 12, 13, 19 | The default is the first four: 44 to 63 bytes a source line executed, of which the stores are 9 to 14 and the moment a line is 22. Stores are what grows, so they are what to select: `--only` is where anything is instrumented (globs on function names, source paths, file names), `--mem` where stores and loads are, within that. The compiler's own stack temporaries are never recorded; a write records its first 64 KiB; a local is declared, not dumped, and a global is written once, at the start. - **Names.** Every file, function, variable and type is a `string` event before it is used. The runtime numbers them when the program starts, so an id is a byte in the recording. - **Types.** A variable's type is a description a viewer can draw from (`src/instr/types.zig` has the grammar and its parser): `[8]i32`, `` *`struct node` ``, and under `types.defId("struct node")` the string `` struct(16){value@0:i32;next@8:*`struct node`;} ``. - **Before the optimizer** by default: every local still has an address, every source store exists, lines come in source order, and what the calls made observable the optimizer must keep, so `-O2` records the same events as `-O0`. `--late` instruments what the optimizer left: the real stores and loads, few variables. - **Stepping.** A `line` is an event like any other, so the ring's stop mask holds the program at each (`host.Options.stop_mask`): source-level step with no ptrace, 1.4 million stops a second with a tracer that polls, 290 000 with one that sleeps in between. - **Cost.** An event is 60 to 70 ns into the ring, whatever its kind (nearly all of it the ring's two contended cache lines), and 1.5 ns when nobody listens. A line of these targets is a few ns of work, so with the default selection they run 7 to 12 times slower untraced and 250 to 400 times slower traced at `-O0`. Recording at a moment a line takes 1.9 million events a second. ## The agent `libztrace-agent.so` is preloaded into a program (`ztrace run --preload`); `libztrace-agent.a` is linked into one, which can then say things about itself (`include/ztrace.h`; a Zig program imports the `ztrace-agent` module instead). Inside the target it has no libc and makes raw syscalls only, so the same library serves glibc and musl, dynamic and static. With no tracer it does nothing: a preloaded malloc costs 2 ns more. ```sh ztrace run --preload zig-out/lib/libztrace-agent.so --alloc -- ./agent_list 100 ztrace run --preload ... --syscalls -- ls / ztrace run --preload ... --hook fib -- ./agent_fib 10 # ztrace show FILE 6: the call stack then ztrace run --preload ... --hook swap --watch numbers -- ./agent_sort ztrace run --preload ... --hook swap --stop-at ret --read numbers -- ./agent_sort ztrace run --preload ... --hook fib --stop-at call -i -- ./agent_fib 5 ztrace run -- ./agent_api # the agent linked in, annotating itself zig build test-agent zig build bench-agent -Doptimize=ReleaseFast ``` | | events | how | | --- | --- | --- | | startup | `map` a mapping, `thread_start` a thread | /proc/self/maps; pthread_create and fork interposed | | `--alloc` | `alloc`, `free`, with the caller's pc | malloc and its family interposed (`dlsym(RTLD_NEXT)`; `--wrap` when linked statically) | | `--syscalls` | `syscall_enter` (path, bytes written: `--capture`, 256), `syscall_exit` | Syscall User Dispatch: SIGSYS, handled in the target | | `--hook SYM` | `call` with the registers, `ret` with the value | int3 at the entry, the return address swapped | | `--watch SYM` | `mem_write`: the bytes that changed, before each event that ends a moment | compared in place against a shadow copy | | `ztrace_mark`, `ztrace_watch`, `ztrace_snapshot` | `mark`; `variable` and the watch; `mem_write` | the target's own calls | | `--stop-at KIND` | the target held at each, while the tracer reads (`--read`) or asks (`-i`) | the ring's `stop_mask`, a futex | Why these. **Hooks** are breakpoints because a breakpoint needs to know nothing about the code it lands on: an inline jump needs an x86-64 length decoder to move the prologue, and patching the PLT misses calls inside a module. The price is a signal a call; the return costs none. **Watches** compare memory in the target's own address space because that is exact at any size, needs no syscall, and does not disturb the target: page protection makes the kernel refuse syscalls that write there, and soft-dirty bits still need the compare, a page at a time. **Syscalls** go through SIGSYS because the kernel then needs no second process: the cost is two signal frames, on a stack of the agent's so that it works on threads with almost none. Measured (`zig build bench-agent -Doptimize=ReleaseFast`, an i7-11700, each loop 10^5 to 5 10^6 times; bytes are the recording's): | | native | traced | recorded | | --- | --- | --- | --- | | a malloc and a free, `--alloc` | 4 ns | 168 ns (11.7 M events/s) | 14 bytes an event | | ... each a moment (`--cut alloc,free`) | | 4.9 µs, the recorder's | 65 bytes a moment | | a syscall, `--syscalls` | 73 ns | 950 ns; 830 ns is the two signal frames | 87 bytes a syscall, a moment | | a syscall under ptrace (`ztrace record --syscalls`) | | 9000 ns | 129 bytes | | a hooked call | 1 ns | 990 ns | 77 bytes: two moments, with registers | | ... watching 1 KiB, 64 KiB, 1 MiB | | +43 ns, +1.5 µs, +17.5 µs a moment | what changed | | ... held at its return (`--stop-at ret`) | | +1.3 µs | | | ... the tracer reading 1 KiB, 64 KiB, 1 MiB from outside | | +2.5 µs, +6.5 µs, +66 µs a stop | what changed | | events that end no moment, through the ring | | 11 M a second | 8 bytes an event | | marks, each a moment | | 2.5 M a second | 32 bytes a moment | `ztrace run` ends a moment at the recorder's kinds and at `call` and `ret` (hooks are asked for by name, and a call and its return are the steps of such a recording); `alloc` and `free` end none unless `--cut` says so. A recording that reaches `--max-size` (1G) ends the run. Limits: a hooked function called with SIGTRAP blocked dies (not under `--syscalls`, which keeps the target's masks clear of it); C++ exceptions do not unwind through a hooked call; a pointer the kernel would answer with EFAULT faults in the handler; syscalls made from the vDSO never enter the kernel and are not seen, as under ptrace; a target that needs SIGSYS or its own alternate signal stack is told it has them. ## Depends on - [sketchbook](../sketchbook): window, drawing, shaders, hot reload, the manim-style `anim` module, frame export. - [pardes](../02-pardes-code): tree-sitter parsing, highlighting, themes. - [xitdb](https://github.com/xit-vcs/xitdb): the recording. x86-64 Linux, Zig 0.17.