1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
|
# computeranim
Two things that meet at a file:
- **ztrace**, a small tracer/debugger: many ways to watch a program run, one
event format, one recording.
- **canim**, animations of what was recorded (registers, memory, syscalls,
data structures, source), written as code, exported as frames for ffmpeg.
```sh
zig build # zig-out/bin/ztrace
zig build test # everything; test-<area> for one
zig build bench-store # what a recording costs a moment
ztrace record -o fib.ztrace -- ./fib 10 # under ptrace, an instruction a moment
ztrace record --by line -w dist -- ./graph # ... or a source line, a global watched
ztrace record --syscalls --strace -- ls / # ... or a syscall, printed as strace does
ztrace dbg -- ./fib 10 # step, break, look, and go back
ztrace run --preload libztrace-agent.so -- ./prog # in-process, no ptrace
ztrace dump fib.ztrace
```
## The pieces
| Where | What |
| --- | --- |
| `src/trace/event.zig` | `Event` (64 bytes: time, thread, kind, six words) plus a payload; `Kind`; `Regs`; `Sink` |
| `src/trace/Ring.zig` | events from the target to the tracer through shared memory; a producer can be held at a kind (`stop_mask`) until `resumeAll` |
| `src/trace/client.zig`, `host.zig` | the ring's two ends: inside the target, inside the tracer |
| `src/trace/store.zig` | `Recorder` (a `Sink`) and `Trace`: the recording, in xitdb |
| `src/trace/ptrace.zig` | the out-of-process backend: a process and its threads under ptrace |
| `src/trace/Debugger.zig` | on top of it: continue, step, breakpoints, hooks, watches, every stop an event |
| `src/trace/Symbols.zig`, `syscalls.zig` | what an address means (modules, ELF symbols, DWARF lines); syscalls by name and shape |
| `src/agent/` | the in-process backend: a library preloaded or linked into the target |
| `llvm/` | compiler instrumentation and its runtime |
| `src/trace/kernel*` | perf events and BPF |
| `anim/` | canim, a package of its own (it needs sketchbook and pardes; ztrace does not): the animation library and its scenes |
| `design/` | the visual system the animations follow, with mock frames |
| `src/cli/` | the `ztrace` command, a file a subcommand |
| `targets/` | programs to trace |
| `build/` | each area's part of the build |
Every backend produces `Event`s into a `Sink`. The recorder is one; the ring
is another, with the recorder at its far end.
## A recording
An xitdb file, kept small. A **moment** stores only its events, encoded
compactly (registers as the ones that changed since the thread's last). The
program's **state** is stored at a keyframe every 256 moments, each sharing
what did not change with the one before; the state at any moment is the
keyframe before it with the moments since replayed over it. `Trace.moment(i)`
is any of them, in any order: time travel is an index.
| `Moment` | from the events |
| --- | --- |
| `regs(tid)` | any event carrying `Regs` as its payload |
| `read(addr, buf, known)` | `mem_write`: every byte seen so far, and which are known |
| `heap()`, `allocation(addr)` | `alloc`, `free`: the live allocations |
| `stack(tid)` | `call`, `ret`: the calls a thread is inside |
| `variables()` | `variable`: the live named variables; a local goes with its call |
| `line(tid)` | `line` |
| `mappings()`, `mapping(addr)` | `map`, `unmap` |
| `string(id)` | `string` |
| `events()` | everything, as it came |
The recorder ends a moment at the kinds in `Recorder.cut` (by default a
step, a breakpoint, a syscall's exit, a signal, a source line, a mark). Of a
memory write it keeps only the bytes that differ from what it already
holds, so a backend that reads a region again pays for what changed in it.
It stops with `error.RecordingFull` at `max_bytes` (1 GiB): a recording is
meant to be megabytes.
What a moment costs (`zig build bench-store`; release, this machine):
| each moment | bytes | ns |
| --- | --- | --- |
| a mark | 29 | 560 |
| a step with registers | 57 | 1100 |
| ... and a word pushed on the stack | 75 | 960 |
| a word written somewhere in 1 MiB of heap | 717 | 14700 |
| 4 KiB written again, unchanged | 376 | 24000 |
| 4 KiB of new bytes | 12078 | 584000 |
An instruction-level recording of `fib(12)` under ptrace, 142 000 moments
with registers and the stack, is 8 MB at 197 000 moments a second; reading
it back is 178 000 moments a second in order, a millisecond to any one.
Scattered writes and new bulk data are what cost: each chunk of 64 bytes
that changed is written at the next keyframe with its path in the tree.
## The debugger
`ztrace dbg -- PROGRAM` starts a program stopped and reads commands (from a
script when stdin is no terminal): `run`/`r`, `continue`/`c`, `step`/`s` (a
source line), `stepi`/`si`, `break`/`b` (a function, `file:line`,
`*address`), `syscalls on|off|NAME`, `watch`, `regs`, `x`, `bt`, `quit`.
Every stop is a moment of a recording, and `back [N]`, `forward [N]` and
`goto MOMENT` look into it: `regs`, `x` and `bt` then answer from the
recording, while the program stays where it stopped.
```
$ printf 'b fib\nr\nbt\ns\nback 1\nx $rsp 16\n' | ztrace dbg -- zig-out/targets/fib 10
[0] step at 7f82d095e7e0 ld-linux-x86-64.so.2!217e0
(ztrace) b fib
breakpoint 1 at 563f798b1814 fib+4 (fib.c:4)
(ztrace) r
[1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4)
(ztrace) bt
#0 563f798b1814 fib+4 (fib.c:4)
#1 563f798b17ea main+42 (fib.c:10)
#2 7f82d0627781 libc.so.6!27781
#3 7f82d06278b9 libc.so.6!__libc_start_main+137
#4 563f798b16e5 _start+37
(ztrace) s
[2] step at 563f798b181c fib+12 (fib.c:5)
(ztrace) back 1
[1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4)
(ztrace) x $rsp 16
7ffc89ca6570 90 65 ca 89 fc 7f 00 00 ea 17 8b 79 3f 56 00 00 .e.........y?V..
(ztrace) out.ztrace: 3 moments
```
`ztrace record` is the same debugger left to run: `--by insn|line|syscall|break`
says where it stops, `-b` adds breakpoints, `-w` watches, `--strace` prints
the syscalls. In a program, it is `zt.Debugger`:
```zig
var d = try zt.Debugger.spawn(gpa, io, path, argv, envp, recorder.sink());
defer d.deinit();
d.syscalls = .record; // or .stop, or a hook: d.on_syscall
try d.watch(d.symbol("dist").?.addr, 8); // recorded at every stop, the changed words only
_ = try d.breakAt(try d.resolve("list_done"), .{ .func = hook }); // a hook says .go or .stop
while (true) switch (try d.step()) { // or stepi(), cont()
.exited => break,
else => {}, // d.regs, d.read, d.write, d.setRegs, d.snapshot
};
```
Of memory it records what is asked for and no more: the watched regions,
a window above the stack pointer, and at most 256 bytes of what a syscall
points at (`-s`), each as the 8-byte words that changed since the stop
before. What it costs (`zig build bench-debugger`; release, this machine):
| | a second | µs | bytes a moment |
| --- | --- | --- | --- |
| an instruction: ptrace step and wait | 273 000 | 3.7 | |
| ... as `Debugger.stepi` with events, the source line looked up | 260 000 | 3.9 | |
| ... and 128 bytes of stack watched | 212 000 | 4.7 | |
| ... recorded (`ztrace record`) | 189 000 | 5.3 | 55 |
| a source line (16 instructions here), recorded | 16 000 | 62 | 60 |
| a breakpoint hit, recorded | 82 000 | 12 | 94 |
| a syscall, entry and exit: native 0.1 µs; stopped at | 120 000 | 8.3 | |
| ... recorded (`--by syscall`) | 100 000 | 10 | 208 |
Memory is read with `process_vm_readv`: 64 bytes in 0.3 µs where
`PTRACE_PEEKDATA` takes 2.2, a megabyte at 22 GiB/s against 28 MiB/s. It is
written the same way, and through `/proc/pid/mem` where the page is
read-only (a breakpoint in code: 0.5 µs).
Limits: `bt` walks frame pointers, so a function without a frame hides its
caller. Threads are followed, each stop is one thread's and the others run
on; stepping by line in a threaded program reports the other threads'
instructions too. A target that execs is not followed into the new program.
## Depends on
- [sketchbook](../sketchbook): window, drawing, shaders, hot reload, the
manim-style `anim` module, frame export.
- [pardes](../02-pardes-code): tree-sitter parsing, highlighting, themes.
- [xitdb](https://github.com/xit-vcs/xitdb): the recording.
x86-64 Linux, Zig 0.17.
|