# computeranim Two things that meet at a file: - **ztrace**, a small tracer/debugger: many ways to watch a program run, one event format, one recording. - **canim**, animations of what was recorded (registers, memory, syscalls, data structures, source), written as code, exported as frames for ffmpeg. ```sh zig build # zig-out/bin/ztrace zig build test # everything; test- for one zig build bench-store # what a recording costs a moment ztrace record -o fib.ztrace -- ./fib 10 # under ptrace, an instruction a moment ztrace record --by line -w dist -- ./graph # ... or a source line, a global watched ztrace record --syscalls --strace -- ls / # ... or a syscall, printed as strace does ztrace dbg -- ./fib 10 # step, break, look, and go back ztrace run --preload libztrace-agent.so -- ./prog # in-process, no ptrace ztrace cc -- clang sort.c -o sort && ztrace run -- ./sort # compiled in: lines, calls, variables, stores ztrace dump fib.ztrace ``` ## The pieces | Where | What | | --- | --- | | `src/trace/event.zig` | `Event` (64 bytes: time, thread, kind, six words) plus a payload; `Kind`; `Regs`; `Sink` | | `src/trace/Ring.zig` | events from the target to the tracer through shared memory; a producer can be held at a kind (`stop_mask`) until `resumeAll` | | `src/trace/client.zig`, `host.zig` | the ring's two ends: inside the target, inside the tracer | | `src/trace/store.zig` | `Recorder` (a `Sink`) and `Trace`: the recording, in xitdb | | `src/trace/ptrace.zig` | the out-of-process backend: a process and its threads under ptrace | | `src/trace/Debugger.zig` | on top of it: continue, step, breakpoints, hooks, watches, every stop an event | | `src/trace/Symbols.zig`, `syscalls.zig` | what an address means (modules, ELF symbols, DWARF lines); syscalls by name and shape | | `src/agent/` | the in-process backend: a library preloaded or linked into the target | | `llvm/pass.cpp`, `src/instr/` | compiler instrumentation: the LLVM pass, the runtime it calls, the type descriptions | | `src/trace/kernel*` | perf events and BPF | | `anim/` | canim, a package of its own (it needs sketchbook and pardes; ztrace does not): the animation library and its scenes | | `design/` | the visual system the animations follow, with mock frames | | `src/cli/` | the `ztrace` command, a file a subcommand | | `targets/` | programs to trace | | `build/` | each area's part of the build | Every backend produces `Event`s into a `Sink`. The recorder is one; the ring is another, with the recorder at its far end. ## A recording An xitdb file, kept small. A **moment** stores only its events, encoded compactly (registers as the ones that changed since the thread's last). The program's **state** is stored at a keyframe every 256 moments, each sharing what did not change with the one before; the state at any moment is the keyframe before it with the moments since replayed over it. `Trace.moment(i)` is any of them, in any order: time travel is an index. | `Moment` | from the events | | --- | --- | | `regs(tid)` | any event carrying `Regs` as its payload | | `read(addr, buf, known)` | `mem_write`: every byte seen so far, and which are known | | `heap()`, `allocation(addr)` | `alloc`, `free`: the live allocations | | `stack(tid)` | `call`, `ret`: the calls a thread is inside | | `variables()` | `variable`: the live named variables; a local goes with its call | | `line(tid)` | `line` | | `mappings()`, `mapping(addr)` | `map`, `unmap` | | `string(id)` | `string` | | `events()` | everything, as it came | The recorder ends a moment at the kinds in `Recorder.cut` (by default a step, a breakpoint, a syscall's exit, a signal, a source line, a mark). Of a memory write it keeps only the bytes that differ from what it already holds, so a backend that reads a region again pays for what changed in it. It stops with `error.RecordingFull` at `max_bytes` (1 GiB): a recording is meant to be megabytes. What a moment costs (`zig build bench-store`; release, this machine): | each moment | bytes | ns | | --- | --- | --- | | a mark | 29 | 560 | | a step with registers | 57 | 1100 | | ... and a word pushed on the stack | 75 | 960 | | a word written somewhere in 1 MiB of heap | 717 | 14700 | | 4 KiB written again, unchanged | 376 | 24000 | | 4 KiB of new bytes | 12078 | 584000 | An instruction-level recording of `fib(12)` under ptrace, 142 000 moments with registers and the stack, is 8 MB at 197 000 moments a second; reading it back is 178 000 moments a second in order, a millisecond to any one. Scattered writes and new bulk data are what cost: each chunk of 64 bytes that changed is written at the next keyframe with its path in the tree. ## The debugger `ztrace dbg -- PROGRAM` starts a program stopped and reads commands (from a script when stdin is no terminal): `run`/`r`, `continue`/`c`, `step`/`s` (a source line), `stepi`/`si`, `break`/`b` (a function, `file:line`, `*address`), `syscalls on|off|NAME`, `watch`, `regs`, `x`, `bt`, `quit`. Every stop is a moment of a recording, and `back [N]`, `forward [N]` and `goto MOMENT` look into it: `regs`, `x` and `bt` then answer from the recording, while the program stays where it stopped. ``` $ printf 'b fib\nr\nbt\ns\nback 1\nx $rsp 16\n' | ztrace dbg -- zig-out/targets/fib 10 [0] step at 7f82d095e7e0 ld-linux-x86-64.so.2!217e0 (ztrace) b fib breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) r [1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) bt #0 563f798b1814 fib+4 (fib.c:4) #1 563f798b17ea main+42 (fib.c:10) #2 7f82d0627781 libc.so.6!27781 #3 7f82d06278b9 libc.so.6!__libc_start_main+137 #4 563f798b16e5 _start+37 (ztrace) s [2] step at 563f798b181c fib+12 (fib.c:5) (ztrace) back 1 [1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4) (ztrace) x $rsp 16 7ffc89ca6570 90 65 ca 89 fc 7f 00 00 ea 17 8b 79 3f 56 00 00 .e.........y?V.. (ztrace) out.ztrace: 3 moments ``` `ztrace record` is the same debugger left to run: `--by insn|line|syscall|break` says where it stops, `-b` adds breakpoints, `-w` watches, `--strace` prints the syscalls. In a program, it is `zt.Debugger`: ```zig var d = try zt.Debugger.spawn(gpa, io, path, argv, envp, recorder.sink()); defer d.deinit(); d.syscalls = .record; // or .stop, or a hook: d.on_syscall try d.watch(d.symbol("dist").?.addr, 8); // recorded at every stop, the changed words only _ = try d.breakAt(try d.resolve("list_done"), .{ .func = hook }); // a hook says .go or .stop while (true) switch (try d.step()) { // or stepi(), cont() .exited => break, else => {}, // d.regs, d.read, d.write, d.setRegs, d.snapshot }; ``` Of memory it records what is asked for and no more: the watched regions, a window above the stack pointer, and at most 256 bytes of what a syscall points at (`-s`), each as the 8-byte words that changed since the stop before. What it costs (`zig build bench-debugger`; release, this machine): | | a second | µs | bytes a moment | | --- | --- | --- | --- | | an instruction: ptrace step and wait | 273 000 | 3.7 | | | ... as `Debugger.stepi` with events, the source line looked up | 260 000 | 3.9 | | | ... and 128 bytes of stack watched | 212 000 | 4.7 | | | ... recorded (`ztrace record`) | 189 000 | 5.3 | 55 | | a source line (16 instructions here), recorded | 16 000 | 62 | 60 | | a breakpoint hit, recorded | 82 000 | 12 | 94 | | a syscall, entry and exit: native 0.1 µs; stopped at | 120 000 | 8.3 | | | ... recorded (`--by syscall`) | 100 000 | 10 | 208 | Memory is read with `process_vm_readv`: 64 bytes in 0.3 µs where `PTRACE_PEEKDATA` takes 2.2, a megabyte at 22 GiB/s against 28 MiB/s. It is written the same way, and through `/proc/pid/mem` where the page is read-only (a breakpoint in code: 0.5 µs). Limits: `bt` walks frame pointers, so a function without a frame hides its caller. Threads are followed, each stop is one thread's and the others run on; stepping by line in a threaded program reports the other threads' instructions too. A target that execs is not followed into the new program. ## Compiled in For a program built from source, `ztrace cc` runs clang with a pass plugin (`llvm/pass.cpp`, built against the system LLVM) and links a runtime (`src/instr/rt.zig`, `libztrace-rt.a`). The program then tells its own story at the level of its source, names and types included: no DWARF is read at view time. ```sh ztrace cc -- clang -O0 targets/instr_sort.c -o sort && ztrace run -- ./sort ztrace cc --only 'sqlite3Btree*,*/pager.c' --mem '*Insert*' -- clang -c sqlite3.c zig build test-llvm # fib, sorts, a list, a BFS: asserted from their recordings zig build bench-llvm -Doptimize=ReleaseFast # the tables these numbers come from ``` | `--instr` | event | bytes recorded | | --- | --- | --- | | `lines` | `line` (file, line, column) before each new line of a block: a moment each | 31 | | `calls` | `call` (name, site, sp, type) at entry, `ret` (name, value) at exit | 14 | | `vars` | `variable` (address, size, name, type) per local at entry, per global at start | 15 | | `stores` | `mem_write` with the bytes after each store; a memset or memcpy is one write | 14 | | `loads` | `mem_read` after each load | 12 | | `edges`, `cmps`, `cyg` | stock clang, no plugin and no names: `edge`, `cmp`, `call`/`ret` by address | 12, 13, 19 | The default is the first four: 44 to 63 bytes a source line executed, of which the stores are 9 to 14 and the moment a line is 22. Stores are what grows, so they are what to select: `--only` is where anything is instrumented (globs on function names, source paths, file names), `--mem` where stores and loads are, within that. The compiler's own stack temporaries are never recorded; a write records its first 64 KiB; a local is declared, not dumped, and a global is written once, at the start. - **Names.** Every file, function, variable and type is a `string` event before it is used. The runtime numbers them when the program starts, so an id is a byte in the recording. - **Types.** A variable's type is a description a viewer can draw from (`src/instr/types.zig` has the grammar and its parser): `[8]i32`, `` *`struct node` ``, and under `types.defId("struct node")` the string `` struct(16){value@0:i32;next@8:*`struct node`;} ``. - **Before the optimizer** by default: every local still has an address, every source store exists, lines come in source order, and what the calls made observable the optimizer must keep, so `-O2` records the same events as `-O0`. `--late` instruments what the optimizer left: the real stores and loads, few variables. - **Stepping.** A `line` is an event like any other, so the ring's stop mask holds the program at each (`host.Options.stop_mask`): source-level step with no ptrace, 1.4 million stops a second with a tracer that polls, 290 000 with one that sleeps in between. - **Cost.** An event is 60 to 70 ns into the ring, whatever its kind (nearly all of it the ring's two contended cache lines), and 1.5 ns when nobody listens. A line of these targets is a few ns of work, so with the default selection they run 7 to 12 times slower untraced and 250 to 400 times slower traced at `-O0`. Recording at a moment a line takes 1.9 million events a second. ## Depends on - [sketchbook](../sketchbook): window, drawing, shaders, hot reload, the manim-style `anim` module, frame export. - [pardes](../02-pardes-code): tree-sitter parsing, highlighting, themes. - [xitdb](https://github.com/xit-vcs/xitdb): the recording. x86-64 Linux, Zig 0.17.