summaryrefslogtreecommitdiff
path: root/README.md
blob: 71142c36c46132665f17a8f37fbc8b576b063bd4 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
# computeranim

Two things that meet at a file:

- **ztrace**, a small tracer/debugger: many ways to watch a program run, one
  event format, one recording.
- **canim**, animations of what was recorded (registers, memory, syscalls,
  data structures, source), written as code, exported as frames for ffmpeg.

```sh
zig build                                   # zig-out/bin/ztrace
zig build test                              # everything; test-<area> for one
zig build bench-store                       # what a recording costs a moment
zig build bench-agent -Doptimize=ReleaseFast    # what the agent costs the target
ztrace record -o fib.ztrace -- ./fib 10     # under ptrace, an instruction a moment
ztrace record --by line -w dist -- ./graph  # ... or a source line, a global watched
ztrace record --syscalls --strace -- ls /   # ... or a syscall, printed as strace does
ztrace dbg -- ./fib 10                      # step, break, look, and go back
ztrace run --preload libztrace-agent.so -- ./prog   # in-process, no ptrace
ztrace cc -- clang sort.c -o sort && ztrace run -- ./sort   # compiled in: lines, calls, variables, stores
ztrace perf --faults --syscalls --watch counter -- ./prog   # the kernel's view, at full speed
ztrace probe --dry-run --usdt provider:name -- ./prog       # ... with BPF, for root
ztrace dump fib.ztrace
```

## The pieces

| Where | What |
| --- | --- |
| `src/trace/event.zig` | `Event` (64 bytes: time, thread, kind, six words) plus a payload; `Kind`; `Regs`; `Sink` |
| `src/trace/Ring.zig` | events from the target to the tracer through shared memory; a producer can be held at a kind (`stop_mask`) until `resumeAll` |
| `src/trace/client.zig`, `host.zig` | the ring's two ends: inside the target, inside the tracer |
| `src/trace/store.zig` | `Recorder` (a `Sink`) and `Trace`: the recording, in xitdb |
| `src/trace/ptrace.zig` | the out-of-process backend: a process and its threads under ptrace |
| `src/trace/Debugger.zig` | on top of it: continue, step, breakpoints, hooks, watches, every stop an event |
| `src/trace/Symbols.zig`, `syscalls.zig` | what an address means (modules, ELF symbols, DWARF lines); syscalls by name and shape |
| `src/agent/`, `include/ztrace.h` | the in-process backend: a library preloaded or linked into the target, and what the target can say through it |
| `llvm/pass.cpp`, `src/instr/` | compiler instrumentation: the LLVM pass, the runtime it calls, the type descriptions |
| `src/trace/perf.zig`, `bpf.zig`, `usdt.zig` | the kernel's view: perf events for any user, BPF programs for root, the static probes of `<sys/sdt.h>` |
| `anim/` | canim, a package of its own (it needs sketchbook and pardes; ztrace does not): the animation library and its scenes |
| `design/` | the visual system the animations follow, with mock frames |
| `src/cli/` | the `ztrace` command, a file a subcommand |
| `targets/` | programs to trace |
| `build/` | each area's part of the build |

Every backend produces `Event`s into a `Sink`. The recorder is one; the ring
is another, with the recorder at its far end.

## A recording

An xitdb file, kept small. A **moment** stores only its events, encoded
compactly (registers as the ones that changed since the thread's last). The
program's **state** is stored at a keyframe every 4096 moments, each sharing
what did not change with the one before; the state at any moment is the
keyframe before it with the moments since replayed over it. `Trace.moment(i)`
is any of them, in any order: time travel is an index.

| `Moment` | from the events |
| --- | --- |
| `regs(tid)` | any event carrying `Regs` as its payload |
| `read(addr, buf, known)` | `mem_write`: every byte seen so far, and which are known |
| `heap()`, `allocation(addr)` | `alloc`, `free`: the live allocations |
| `stack(tid)` | `call`, `ret`: the calls a thread is inside |
| `variables()` | `variable`: the live named variables; a local goes with its call |
| `line(tid)` | `line` |
| `mappings()`, `mapping(addr)` | `map`, `unmap` |
| `string(id)` | `string` |
| `events()` | everything, as it came |

From a debugger's recording, `zt.telescope`: `instruction(m, addr, buf)` is an instruction's text, its source `line`, and for a direct call its callee's `params` (DWARF's); `frames(m, regs, out)` is the call chain with each frame's extent on the stack, `low` to `high` (a `stack()` frame's is up to `sp + 8`), walked by the call frame information the recording holds (below), else by frame pointer; `slots(m, regs, low, out)` is what each stack word from `low` is (`out[i]` the word at `low + 8 i`), a `Slot`: its `frame`, an index into `frames` (null in none), and its `role`: `.saved` (the register a function saved for its caller there: `"rbx"`, `"rbp"`, `"r12"` to `"r15"`; or `"canary"`, its copy of the stack protector's value, below), `.return_address`, `.local` (a variable or parameter, a `Local`: `name`, `size`, and `offset`, where the word is in it: 0 at its first byte, 8 a word on, -4 its high half), or the process's first stack, `.argc`, `.argv` i, `.envp` i (i = argc, envc: the zero that ends them) and `.auxv` i (the auxiliary vector's i-th word, two an entry); `.none` where the recording does not say. Where small locals share a word, `role` is the one that begins first in it and `also` the others in the order they begin (two ints: `i` at offset 0 and `n` at -4, so `i, n`; a viewer writes `n, high half` from `n`'s -4 and size 4). At an instruction that saves a register (`push rbp`, the frame being made) the word it is about to write, under rsp, has the register's role already, in frame 0: the call frame information says where it goes from that instruction on (the row that begins after it); `function(m, pc)` is the function that holds a pc as the recorded walk went through it, its `start` and `len` by its call frame information (a PLT stub's 16 bytes), for every frame of every stop; `got(m, slot)` is what a GOT slot holds and whether that is still `lazy` (its name has `@plt`: the way back into the PLT, so the next call runs the loader's resolver) or bound.
For a call followed down, each frame has its module (`mapping`), whether it has source (its instructions' `line`), its function's bounds (`function`), and, from a caller, three instructions round its call: the return address, the call `back` from it, and the row before the call `back` from that.
A function as a graph (IDA's, Binary Ninja's): `blocks(m, pc, out)` gives the basic blocks of the function that holds `pc`, a `Graph`: its bounds, its blocks in address order, the one that holds the pc (`here`), and whether every instruction's text is recorded (`text`; then `instruction` reads each block's rows). A `Block` has its bounds, how it ends (`Debugger.End`: `fall` into the next, an unconditional `jump`, a conditional `branch`, `ret`, an `indirect` jump whose targets are not known, a `tail` call out of the function, a call that does not return, `noreturn`: the function's last instruction, or `abort`, `exit`, `__stack_chk_fail` and their kin) and its edges: `to`, where a jump, a taken branch, a tail call or a call that does not return goes, and `falls`, the block after it for `fall` and a branch not taken. A call that returns ends no block. The debugger records them once a function, at the first stop in it, decoded from its own instructions (its bounds from its symbol, else its unwind information: a stripped library's function has them too), with every instruction's text and the names of what its calls and its jumps out of it reach, up to `Debugger.function_text` instructions (2048); past that the blocks without the text (`sqlite3VdbeExec`, 11 077 instructions: 2 510 blocks). Code with no bounds (the C runtime's start, a JIT's) has none. Which edges a run took: by instruction, a stop whose instruction is a jump or a return has a `branch` event (a2 its pc, a0 where it goes, a1 1 when it will go there); its pc is the last instruction of the block that holds it, and the next stop is at `to` when a1 is 1, else at `falls` (a `ret`'s, at its caller). By line or by breakpoint there are no stops between, so a reader knows the blocks the stops are in and, at a stop on a jump, the edge it is about to take; the path between two stops it does not know. What it costs: the list of `anim/tools` by instruction, 243 bytes a moment against 224 (every function stopped in, recorded whole once) and 25 600 moments a second against 26 300; the first stop in `sqlite3BtreeInsert` (1 045 instructions) 118 KB and 2 to 4 ms, in `sqlite3VdbeExec` 80 KB (its blocks alone) and 6 ms; SQLite by breakpoint on `sqlite3BtreeInsert` 317 bytes a moment against 310.
Code is named at every pc, a function's first instruction too, and a PLT stub and a GOT slot for the function they lead to: `open@plt+0`, `open@got+0` (from the relocations and the dynamic symbols; an IFUNC's in libc by the function its resolver is, `libc.so.6!strlen@plt+0`); of the names libc gives one address (`open`, `open64`, `__open64`) the one the program's relocations use. A slot is named when an instruction jumps or calls through it.
Data is named as code is, by the global or static object of the symbol tables that holds it, `+` how far into it in decimal: `head+0`, `big+4096`, `libc.so.6!main_arena+96`, `libc.so.6!_IO_2_1_stdout_+0` (libc's statics from its separate debug file). A reader gets it as it gets a code name, the string whose id is the address (`Moment.string`, `m.trace.strings.get`), for every address an instruction touches and every pointer the telescope follows into a module's data, its .bss past the file's last page included. An address no object holds (a string literal, the stack, the heap) has none; the debugger looks each new address up once.
With `--alloc`, every live allocation and the 16 bytes of allocator header before it are in `read` at each stop (its first 4 KiB; what changed, a stop at a time; 256 KiB of them at most, lowest addresses first: `Debugger.allocation_watch`).
Recorded `--by insn`, the instructions that ran between two moments are the `rip` of each moment between them; `--by line` does not keep them.
A byte in `read` is as it was when the debugger last sent it: the stack window, the watched regions and the live allocations at every stop, a `read`'s or `write`'s buffer at that syscall (`-s` bytes of it). Memory it looked at once and not since is known and may be stale.

The recorder ends a moment at the kinds in `Recorder.cut` (by default a
step, a breakpoint, a syscall's exit, a signal, a source line, a mark). Of a
memory write it keeps only the bytes that differ from what it already
holds, so a backend that reads a region again pays for what changed in it.
It stops with `error.RecordingFull` at `max_bytes` (1 GiB): a recording is
meant to be megabytes.

What a moment costs (`zig build bench-store`; release, this machine):

| each moment | bytes | ns |
| --- | --- | --- |
| a mark | 24 | 290 |
| a step with registers | 51 | 750 |
| ... and a word pushed on the stack | 65 | 450 |
| a word written somewhere in 1 MiB of heap | 320 | 9500 |
| 4 KiB written again, unchanged | 370 | 22000 |
| 4 KiB of new bytes | 12078 | 545000 |

An instruction-level recording of `fib(12)` under ptrace, 142 000 moments
with registers and the stack, is 8 MB at 197 000 moments a second. Reading
is 6 µs a moment in order and a millisecond to any moment (three at worst).
Scattered writes and new bulk data are what cost: each chunk of 64 bytes
that changed is written at the next keyframe with its path in the tree.

A larger program, to scale by: Lua 5.4 compiled by `ztrace cc` with
everything instrumented (lines, calls, variables, stores, in all of its
26 000 lines) and run on a ten-line script that takes it a millisecond.

| what is instrumented | moments | events | recording | run |
| --- | --- | --- | --- | --- |
| everything | 3 248 000 | 7 353 000 | 156 MB | 1.7 s |
| `--only luaV_execute` (the interpreter's loop) | 2 176 000 | 4 640 000 | 93 MB | 1.0 s |
| `--instr calls` | 189 000 | 191 000 | 7 MB | 0.06 s |

A moment a C line of an interpreter is a great many moments: choose what
is instrumented (`--only`, `--mem`, `--instr`) before choosing to wait.

The other backends on the same Lua, uninstrumented: the agent's
allocations and syscalls (368 allocations, no ptrace), breakpoints on
`luaH_resize` (48 hits), source lines from DWARF (53 000 a second),
counters and page faults from perf (10.8 million instructions, 89 faults).

And pardes drawing an image, through its own harness (`zig build
image-harness` there, the command it runs given to `ztrace run --preload
... --syscalls --capture 2048`): the harness and the pardes it starts are
one recording, 9 threads, and the kitty graphics protocol is there to read
in the captured writes: pardes asks (`_Gi=1,a=q`), the terminal answers
(`_Gi=1;OK`), pardes sends the pixels (`_Gf=32,s=2,v=2,i=1;DU3n/w...`). Two
limits it shows: a recording's state is one address space, so a child's
memory and mappings share its parent's maps (its events are told apart by
thread); and pardes allocates through its own heaps, so `--alloc` sees only
what its C libraries ask of libc.

And V8, as the test of scale: a release `d8` (74 MB, 46 threads, a JIT),
on a script it runs in about 50 ms.

| backend | what it did | time |
| --- | --- | --- |
| `ztrace record --syscalls` (ptrace) | 2 951 moments | 0.13 s |
| `ztrace record --by insn --steps 2000000` | 2 million instructions, 96 MB (47 bytes a moment) | 9.4 s |
| `ztrace record --by break -b malloc` | 1 753 stops | 0.12 s |
| `ztrace dbg`, a breakpoint in `Heap::CollectGarbage` | an 18-frame backtrace through JIT code to `Shell::Main` | |
| `ztrace perf --faults --sample 4000 --threads` | 784 million instructions, 10 669 faults, 474 samples | 0.07 s |
| `ztrace run --preload ... --syscalls` | 2 900 syscalls, in the process | 0.06 s |
| `ztrace run --preload ... --hook malloc` | 1 990 calls and returns | 0.07 s |

V8 found what the small programs could not. Its JIT guards code with
memory protection keys, and a signal handler starts with none of them: a
compiler thread cloned from inside the agent's syscall handler could not
read its own code space, and d8 died. The handlers now act with the
thread's keys and float modes (`src/agent/signal.zig`). `--alloc` sees
nothing in d8, which binds its own `malloc`: hook it instead. C++ names
are shown and matched as the source has them, without their parameters
(`-b v8::internal::Heap::CollectGarbage`; the mangled name works too):
the command links zig's libc++ for its demangler.

## SQLite

Between Lua and V8 on the ladder: SQLite 3.50.4's amalgamation (one C
file of 263 000 lines), its shell built by clang `-O2 -g
-fno-omit-frame-pointer`, on a file database: two tables and an index,
5 100 rows inserted in a transaction, a count, a join with an aggregate,
a lookup through the index. Alone it takes 5.5 ms, 51 million
instructions and 321 syscalls. `sh bench/sqlite.sh PREFIX SRC` builds the
shells and makes each row (a release ztrace installed at PREFIX; time is
the tracer's own measure of the program's run, the best of three; seek is
`ztrace show` of the middle moment: the file opened, the moment replayed
from its keyframe, printed).

| backend | moments | events | recording | a moment | time | × alone | moments/s | events/s | seek |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `record --by syscall` | 322 | 3 593 | 0.3 MB | 830 B | 21 ms | 3.9 | 15 100 | 169 000 | 1.4 ms |
| `record --by line`, its first million lines | 1 000 000 | 2 843 000 | 108 MB | 107 B | 31 s | | 32 000 | 91 000 | 6.8 ms |
| `record --by line --max-size 2G`, all of it | 16 890 000 | 46 462 000 | 1.7 GB | 100 B | 504 s | 92 000 | 33 500 | 92 000 | 13 ms |
| `record --by break -b sqlite3BtreeInsert` | 15 539 | 82 401 | 3.5 MB | 223 B | 440 ms | 80 | 35 300 | 187 000 | 4.3 ms |
| `run --preload ... --alloc` | 1 | 1 353 | 36 KB | | 6.4 ms | 1.2 | | 211 000 | 1.1 ms |
| `run --preload ... --syscalls` | 282 | 601 | 45 KB | 161 B | 7.0 ms | 1.3 | 40 300 | 86 000 | 3.9 ms |
| `run --preload ... --hook sqlite3BtreeInsert` | 30 434 | 30 473 | 2.7 MB | 88 B | 25 ms | 4.5 | 1 240 000 | 1 240 000 | 3.0 ms |
| `perf --faults --syscalls` | 321 | 889 | 0.1 MB | 160 B | 19 ms | 3.5 | 16 800 | 46 500 | 1.3 ms |
| `perf --break sqlite3BtreeInsert` | 2 | 15 258 | 2.0 MB | | 25 ms | 4.6 | | 603 000 | 2.5 ms |
| `cc` (everything), `run` | 21 789 000 | 40 825 000 | 1.06 GB | 48 B | 16.3 s | 2 960 | 1 340 000 | 2 511 000 | 45 ms |
| `cc --only sqlite3VdbeExec`, `run` | 5 873 000 | 9 401 000 | 221 MB | 37 B | 2.5 s | 457 | 2 335 000 | 3 737 000 | 6.5 ms |
| `cc --instr calls`, `run` | 2 925 000 | 2 930 000 | 98 MB | 33 B | 1.05 s | 191 | 2 777 000 | 2 782 000 | 2.0 ms |
| `cc --instr calls`, `record --by break -b sqlite3BtreeInsert --alloc --preload ...` | 15 626 | 3 167 770 | 53 MB | 3 379 B | 1.16 s | 211 | 13 400 | 2 726 000 | 4.8 ms |

Built by `ztrace cc` and run with nobody listening, the shell takes 108 ms
with everything instrumented, 23 with `--only sqlite3VdbeExec`, 11 with
`--instr calls`. `ztrace dbg` with `b sqlite3VdbeExec`, `r`, `bt` gives 19
frames, from the interpreter through `sqlite3InitOne`, the parser's
`yy_reduce` and `sqlite3_prepare_v2` to the shell's `process_input` and
`_start`; `b sqlite3.c:80895` is `sqlite3BtreeInsert`. Under ptrace with
the allocations told (the last row), each stop compares the 155 live
allocations, 78 KB of them with their headers: 3 379 bytes a moment and
1.16 s, against 3 098 and 0.63 s before allocations were read.

SQLite found two things. A file's name in DWARF may be absolute (the
amalgamation compiled by its full path), and the debugger joined it to the
compilation's directory anyway: every `line` named a file that was not
there. And `ztrace run` noticed a target's exit only when its idle wait of
10 ms ran out, so a 5 ms program took 15; it waits a millisecond now.
Recorded a line a moment, the whole run is past the gigabyte a recording
is allowed by default.

## Together

The backends are not modes to choose between: they feed one sink. A
program compiled by `ztrace cc`, with the agent preloaded, run under the
debugger, gives one recording, in order: the stops and their registers
(ptrace), the allocations (the agent), the lines, calls, stores and named
variables (the pass).

```sh
ztrace cc -- clang -O0 targets/instr_list.c -o list
ztrace record --by break -b push -b reverse --alloc --preload zig-out/lib/libztrace-agent.so -- ./list
ztrace dbg --ring -- ./list        # breakpoints and time travel over what the program says of itself
ztrace show out.ztrace 40          # the thread's line and calls, the live heap, each variable's bytes
ztrace perf --faults --syscalls --alloc --preload zig-out/lib/libztrace-agent.so -- ./list
                                   # the kernel's view too: each malloc, the brk it makes, the faults of its first touch
zig build test-together
```

The cleanest machine-level recording with names in it is `ztrace cc
--instr vars` under the debugger: the pass declares each variable (name,
type, address) and instruments nothing else, the debugger watches every
variable declared through the ring and records its bytes at each stop, and
the lines, registers, disassembly and memory touches are the debugger's
own, of code that is otherwise the program's.

```sh
ztrace cc --instr vars -- clang -O0 targets/instr_sort.c -o sort
ztrace record --by line --ring -- ./sort && ztrace show out.ztrace 140
  unsorted: [8]i32 = 05 00 00 00 02 00 00 00 ...    i: i32 = 02 00 00 00
```

Static probes (USDT, DTrace's) need neither BPF nor root this way: `ztrace
record --by break -b 'usdt:libc:*' -- ./prog` arms each probe as a
breakpoint that records a `probe` event with its arguments and runs on.
glibc ships 78 of them (`memory_sbrk_more`, `setjmp`, `lll_lock_wait`...);
`<sys/sdt.h>` puts your own into a program.

`--ring` (implied by `--preload` and `--alloc`) gives a target under
ptrace the ring `ztrace run` gives it; the debugger drains it while the
target runs and before it reports a stop. The agent's hooks and syscall
tracing are signals, which a debugger takes first: under ptrace, use
breakpoints and the debugger's own syscall stops. Strings never collide:
the pass counts its from 1, a symbol's name has its address as its id, and
every other producer uses `event.stringId`, a hash with the top bit set.

## Context

A viewer of a recording should be able to show what pwndbg's context
shows, so the debugger records what that takes while the memory is there
to read. At every stop it follows each register and the top sixteen words
of the stack as pointers, three deep (`--telescope N`; 0 for not at all):
the memory they lead to is recorded, every code address met gets a name
(the string whose id is the address: `fib+40`, `libc.so.6!write+18`), and
the instructions round the pc are disassembled by capstone, eight above
and eight below it (strings with id `event.codeId(addr)`; inside a function
the ones above are known, the function decoded once from its start). `zt.telescope` reads it back: `chase` gives a
value's chain, each link with the kind of memory it points into (code,
stack, heap, data, read-only, writable-and-executable: what a pointer is
coloured by), a name for code and a string where the chain ends on one.

```
$ ztrace record --by break -b fib -- ./fib 5 && ztrace show out.ztrace 34
  rbp    7ffc63068910 —▸ 7ffc63068930 —▸ 7ffc63068950 —▸ 7ffc63068970 —▸ 7ffc63068990
  r12    7ffc63068ae8 —▸ 7ffc63069f22 ◂— "/tmp/..."
  rip    1001794 (fib+4)

 ► 000001001794  push r14
   000001001796  push rbx
   000001001797  mov ebx, edi

  00:0000│ rsp 7ffc63068910 —▸ 7ffc63068930 —▸ 7ffc63068950 —▸ ...
  01:0008│     7ffc63068918 —▸ 10017b8 (fib+40)
  02:0010│     7ffc63068920 ◂— 3
```

**Separate debug info.** A stripped library (a distribution's libc, its
loader) is named from its separate debug file where one is on disk: by
build id in the debuginfod client's cache (`$DEBUGINFOD_CACHE_PATH`, else
`$XDG_CACHE_HOME/debuginfod_client`, else `~/.cache/debuginfod_client`)
and in `/usr/lib/debug/.build-id`, or by `.gnu_debuglink` beside the file
(through `std.debug.ElfFile`). It is used only when its own build id is the
mapped library's, or, for a debuglink without one, its CRC matches:
anything else is ignored. Its symbol table names static functions
(`_int_malloc`, `__libc_start_call_main`, the loader's
`_dl_runtime_resolve_xsavec`), versions dropped (`__libc_start_main`, not
`@@GLIBC_2.34`); its DWARF gives libc's frames their lines, by the paths of
the distribution's build (`/usr/src/debug/glibc/glibc/...`, recorded as
they are), and its functions' bounds and parameters; the heap's
`main_arena` is found by its symbol. The tracer never fetches: on Arch,
`debuginfod-find debuginfo /usr/lib/libc.so.6` (or gdb, once) fills the
cache; with nothing on disk it is as before. Each module's build id, and
whether its names come from its own debug info, separate or in the file,
is recorded once (`event.moduleId` of its path; `zt.telescope.module(m,
path)`), and `ztrace show` prints it beside its mapping: a recording says
which libc it is of. A library run through another loader (pwndbg's glibc
2.40) has its own build id and gets nothing from the system's file.
Nothing of it is read until an address in the library wants a name: its
symbols when it is mapped (1 ms), its DWARF at the first stop that wants a
line there (30 ms and 25 MB in a release build; 1.6 s in a debug build,
`std.debug.Dwarf.open` reading every function). On the list of
`anim/tools` by instruction, libc's stops then have lines too: 325 bytes a
moment against 249, 12 000 moments a second against 16 000.

A stop also says what its instruction is about to do to memory, worked
out from capstone's operands and the registers: a `touch` event (the
address, the size, read or write) with what is there (and `--around`
bytes each side), and after the step what a write left there. So an
instruction-level recording has every memory write the stepped thread
makes, with nothing watched and nothing compiled in. The stack's own
instructions touch memory no operand names, and have their touch after
their operands' (a2 the pc, as every touch): `push`, `pushfq` and `call`
write the 8 bytes at rsp - 8 (the pushed register, the flags, the return
address), `enter` the saved rbp there; `pop`, `popfq` and `ret` read the 8
at rsp, `leave` the saved rbp at rbp (2 bytes for `pushf`, `push ax`: the
operand-size prefix). The string instructions (`movs`, `stos`, `lods`,
`cmps`, `scas`) name theirs, at rdi and rsi; repeated (`rep`), the touch is
the range the count covers, rcx elements up from there, or down to there
with the direction flag set, none when rcx is 0. Stepped by instruction a
`rep` stops after each element (rcx one less at each stop, the range one
element shorter, the pc the same); of a range only its first 4 KiB, from
where it begins, is recorded at a stop. A call, a jump or a return at the
pc is a `branch` event: where it goes, and for a conditional jump whether
it will. What the stack's touches and the names of data cost, on the list
of `anim/tools` by instruction (`--from main --stack 1024 --around 80
--disasm 48`, -O0, 2 464 moments): 205 more touches and 135 names, 362
bytes a moment against 356, and 11 800 moments a second against 12 100
(the median of eight runs each).

How far an instruction moves rsp or rbp (`push` -8, `sub rsp, 0x18`
-0x18, `leave` up to the caller's frame, `ret` +8) is no event: it is the
next stop's register less this one's, both `Moment.regs` of the same
thread. By instruction the next stop is the next moment, or the one after
the syscall's when the instruction is a `syscall`; by line or breakpoint
it is the sum of everything between.

**The call chain** is walked at every stop by the unwind tables
(`.eh_frame`, `.debug_frame`, through `std.debug.Dwarf.Unwind`), as gdb
walks it: right inside a function that keeps no frame (a libc wrapper at
a syscall, malloc's fast path), in a prologue, in a PLT entry; by frame
pointer where there are none (JIT code, a PLT Zig's linker made). What the walk reads of
the stack is recorded, its return addresses named, and each function it
goes through once, as a string (`event.frameId` of where it begins): its
rows of call frame information (where its caller's stack pointer is, the
CFA, and where it saved each register, a row for each stretch of its
code) and, from DWARF, its frame base and the locals the debug info
places by it (`DW_OP_fbreg`), each with its size and its block's range of
code. `zt.telescope.frames` walks the same chain again from the recording,
which needs no ELF file, and `slots` names the words: the saved registers
and return addresses of every frame; the locals of -O0 and -Og code where
the frame's pc is in their block (a leaf's in the red zone too), every one
that shares a word; the stack protector's copy; and argc, argv, envp and
the auxiliary vector, from where the stack began (the string
`event.frameId(0)`, `Debugger.Start`, read from `/proc/pid/stat`).
`ztrace show FILE MOMENT` prints each stack word's role after its chain:
`saved rbp`, `saved canary`, `return address`, `buf`, `buf+8` (a word into
it), `i[0..4], n[4..8]` (the bytes of the word each fills), `argv[1]`.

**The canary.** A function built with the stack protector copies the
value at `fs:[0x28]` into its frame and checks it before it returns: that
word's role is `.saved = "canary"`, from the store on. It is found by
seeing the store, not by position: when the debugger records a function's
frame it decodes its first 64 instructions, once, for a `mov reg,
fs:[0x28]` and the next store of that register to the stack (`mov [rbp -
8], rax` at -O0 by gcc and clang, `mov [rsp + 0x18], rax` in libc), and
records the word as a local with no name, placed from the frame base (from
the CFA in a function without debug info). By position it would be a guess:
the canary is under the saved registers, but with padding between, and a
function with no array has none at all. It cannot be known where the store
is past the first 64 instructions (a prologue shrink-wrapped behind an
early return), where its register is neither the frame base nor the one
the CFA is kept in at the store (`mov [rsp + 8], rax` in a function that
keeps rbp, at -O2), and in a recording made without a disassembler.

What it does not say: a local in a register or a list of places (-O1 and
up), or in a block whose code is in pieces (`DW_AT_ranges`); a local's
name in a prologue or an epilogue where the frame base is rbp and rbp is
not yet or no longer the frame's; a frame whose CFA is an expression (a
signal frame) or that has no unwind tables, which falls back on the frame
pointer; inlined calls, which have no frame (gdb lists them). In
`targets/stack.c`'s loop (gcc and clang, -O0) and in libc's `sbrk`
between its `push rbp` and its `mov rbp, rsp` (the list of `anim/tools`),
the roles are where gdb's `info frame` and `info locals` put the saved
registers and the locals (`&i`, `&n`, `&buf`); gdb names no canary, whose
word holds what `fs:[0x28]` does.

What it costs, on `fib(12)` by instruction (142 000 moments): 154 000
moments a second and 64 bytes a moment with the disassembly and the
touches, 57 000 and 77 bytes with the telescope and the call chain. The
call chain by unwind tables costs nothing a second over the frame
pointer's (the walk reads the stack once and remembers each pc's row) and
1 to 4 bytes a moment: on the list of `anim/tools` by instruction
(`--from main --stack 1024 --around 80 --disasm 48`, -O0) 229 bytes a
moment against 225, 16 000 moments a second either way; SQLite by
breakpoint on `sqlite3BtreeInsert`, 310 bytes and 33 000 a second, as
before. Capstone is built from source for x86
(`build/capstone.zig`), into the command only: the library takes a
disassembler as a function (`Debugger.disasm`) and links none.

## The heap

`ztrace record --heap` records glibc's malloc at every stop as pwndbg's
`heap` and `bins` show it, and `zt.heap.glibc(arena, moment, tid)` reads it
back: the main heap's chunks in address order (size, the P, M, A bits, free
or in use, the list that holds it), the thread's tcache (each bin's count
and chain), the fastbins, the unsorted, small and large bins, the top, the
arena's fields. Each link is given as stored and as the pointer it holds
(safe-linking undone), each with the address of the word it was read from;
a list ends where the recording has no memory, and says so. `ztrace show`
prints it. The debugger runs the reader's own walk on the live target,
every word it reads recorded as changed words are (`Debugger.Recorded`):
the arena and the tcache whole, each chunk's size word, the two words of
each chunk on a list. At most 4096 chunks and 256 entries a list a stop.

Where to begin is found once: the version from the banner in libc's file,
the main arena by its symbol where there is one (a static binary, an
unstripped libc) and else in libc's writable data as the struct whose
`next` is itself, the tcache as the word below the thread pointer that
points at a chunk of the tcache's size. Known: glibc 2.44 (this machine's,
stripped: no fastbins, 76 tcache bins counted down, 16 a bin, the tcache
made at the first free: until then the thread's pointer is on a dummy, zeros
in libc's read-only data, which the reader says as `tcache_dummy`; a bin's
count is the slots it has left, and `Tcache.capacity`, an empty bin's
count, what a bin holds when full) and 2.40 (pwndbg's own, the classic layout);
another version is refused by name. A static glibc has no version in it:
its layout is told by the size of `main_arena`. Not read: a stripped static
glibc, musl, the arenas of other threads, mmapped chunks (`--alloc` has
them). `zig build test-heap` records `targets/heap.c` under both and checks
each stop against the live process.

```
$ ztrace record --by break -b heap_step --heap -- zig-out/targets/heap && ztrace show out.ztrace -1
glibc 2.44 heap: main_arena 7f59cd219ac0, top 18fcab60, last remainder 0, system mem 21000, flags 0
  000018fca000     30 P-- x16 free (tcache 30)
  000018fca300    540 P-- free (large bin 69)
  000018fca840     20 --- in use
  000018fca860    300 P-- in use
  000018fcab60  204a0 P-- top
tcache 18fca870
  tcache 30 [16, 0 left]: 18fca2d0 (18fd2d7a) —▸ 18fca2a0 (18fd2d4a) —▸ 18fca270 (18fd2d9a) —▸ 18fca240 (18fd2dea) —▸ ... 12 more 0
  large bin 69: 18fca300 —▸ the bin
```

By instruction from `main` it costs a quarter of the moments a second
and 2 bytes a moment: `targets/heap.c` 55 000 to 41 000 and 112 to 114
bytes, the list of `anim/tools` 48 000 to 41 000 and 170 to 173.

## AddressSanitizer

`ztrace record --asan` records ASan's shadow beside every region it
records (the stack window, watched regions, live allocations, what
instructions touch and `--around` round it, what pointers lead to): a
byte for 8, at `(addr >> 3) + offset`, the offset the runtime exports
(`__asan_shadow_memory_dynamic_address`: 0x7fff8000 here, for gcc's
`libasan.so` and for clang's runtime linked in), found at the first stop
after the runtime set it and recorded once as the string
`Debugger.asan_id`. `zt.telescope.shadow(m, addr)` is the shadow byte's
address and what it holds: 00 addressable, 1 to 7 that many first bytes,
f1 f2 f3 a frame's left, middle and right redzones, f5 a frame returned
from, fa a heap block's redzone, fd freed. A program not built with
`-fsanitize=address` is refused. It costs a second read of each region: 4
bytes a moment and a quarter of the moments a second (by instruction on
`targets/overflow.c`, 243 to 248 bytes and 42 000 to 31 000).

`targets/overflow.c` writes 40 bytes into a 16-byte buffer. Built with
ASan by gcc or clang (`zig build test-debugger` builds both), at the stop
on `__asan_report_store1`'s first instruction its argument, the first byte
past the buffer, has `f3` for shadow, the buffer `00 00` and the 32 bytes
before it `f1`: the frame comes from ASan's fake stack
(`__asan_stack_malloc_0`); the branch into the report is a `branch` event
(gcc's through `__asan_report_store1@plt`), and the chain goes back from
inside the report through `fill` to `main`. The report itself is more
than a million instructions (`ListOfModules::init` clearing its buffers
a byte at a time): record by instruction up to it (`--steps`), and the
rest `--by syscall`, where the report is the `write`s to fd 2 with their
text (`-s 4096`: the line of `=`, then `ERROR: AddressSanitizer:
stack-buffer-overflow on address ...`) and the run ends in
`exit_group(1)`. ASan's leak check needs `ASAN_OPTIONS=detect_leaks=0`
under a tracer. Built plain at -O0 the same loop has its own `i` and `n`
above the buffer and overwrites them, and writes on to the end of the
stack: a SIGSEGV at the loop's store, its fault address the stack's end
(built at -O1 the loop is a `memset` and `fill` tail-calls `printf`,
whose `ret` pops 0x4141414141414141: a SIGSEGV at that `ret`, fault
address 0).

## The debugger

`ztrace dbg -- PROGRAM` starts a program stopped and reads commands (from a
script when stdin is no terminal): `run`/`r`, `continue`/`c`, `step`/`s` (a
source line), `stepi`/`si`, `break`/`b` (a function, `file:line`,
`*address`), `syscalls on|off|NAME`, `watch`, `regs`, `x`, `bt`, `quit`.
Every stop is a moment of a recording, and `back [N]`, `forward [N]` and
`goto MOMENT` look into it: `regs`, `x` and `bt` then answer from the
recording, while the program stays where it stopped.

```
$ printf 'b fib\nr\nbt\ns\nback 1\nx $rsp 16\n' | ztrace dbg -- zig-out/targets/fib 10
[0] step at 7f82d095e7e0 ld-linux-x86-64.so.2!217e0
(ztrace) b fib
breakpoint 1 at 563f798b1814 fib+4 (fib.c:4)
(ztrace) r
[1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4)
(ztrace) bt
#0 563f798b1814 fib+4 (fib.c:4)
#1 563f798b17ea main+42 (fib.c:10)
#2 7f82d0627781 libc.so.6!27781
#3 7f82d06278b9 libc.so.6!__libc_start_main+137
#4 563f798b16e5 _start+37
(ztrace) s
[2] step at 563f798b181c fib+12 (fib.c:5)
(ztrace) back 1
[1] breakpoint 1 at 563f798b1814 fib+4 (fib.c:4)
(ztrace) x $rsp 16
7ffc89ca6570  90 65 ca 89 fc 7f 00 00 ea 17 8b 79 3f 56 00 00  .e.........y?V..
(ztrace) out.ztrace: 3 moments
```

`ztrace record` is the same debugger left to run: `--by insn|line|syscall|break`
says where it stops, `-b` adds breakpoints, `-w` watches, `--strace` prints
the syscalls, `--from main` runs through the loader and the C library's
start before the first stop. How much of the context a stop keeps is
asked for: `--stack BYTES` above the stack pointer (128), every word of
it followed as a pointer; `--around BYTES` of memory each side of what a
pointer leads to (0: the 32 bytes at it); `--disasm ROWS` of instructions
above and below the pc (8). On the list of `anim/tools` by instruction,
`--stack 512 --around 64 --disasm 24` is 184 bytes a moment against 159,
and 27 000 moments a second against 45 000.

By instruction, a few kinds of code are not what one steps through:

- **The vDSO** (`clock_gettime`, `gettimeofday`: the kernel's code mapped
  into the process) is run through: the instruction that calls or jumps
  into it is a moment, the next is where it returns to, with rax the
  result, and none is inside it. Its loops retry while the kernel's time
  data changes under them, which a stop at each instruction made them do
  for as long as stops were slow: three clock reads were three million
  moments. The call is a `branch` into `[vdso]`; what it wrote is in
  memory where the stack window or a watch has it.
- **Lazy binding**: the first call through a PLT stub runs the loader's
  resolver, hundreds of instructions in ld.so (named by offset: Arch's
  ld.so has no symbols), the call chain under it intact. A target linked
  with `-Wl,-z,now` is bound at load and goes straight to the function.
- **A system call**: the moment whose pc is the `syscall` instruction is
  followed by one that holds the call, `syscall_enter` (aux the number,
  `zt.syscalls.name` its name; the six arguments; the path or bytes it
  points at, read by its signature in `zt.syscalls`) and `syscall_exit`
  (a0 the result), whose registers are those after it: rax the result,
  rcx the instruction after the `syscall`, r11 the flags (the trap flag of
  the step among them), the two the instruction clobbers. The moment after
  that is the step to the next instruction, at the same pc.

In a program, it is `zt.Debugger`:

```zig
var d = try zt.Debugger.spawn(gpa, io, path, argv, envp, recorder.sink());
defer d.deinit();
d.syscalls = .record;                       // or .stop, or a hook: d.on_syscall
try d.watch(d.symbol("dist").?.addr, 8);    // recorded at every stop, the changed words only
_ = try d.breakAt(try d.resolve("list_done"), .{ .func = hook });  // a hook says .go or .stop
while (true) switch (try d.step()) {        // or stepi(), cont()
    .exited => break,
    else => {},                             // d.regs, d.read, d.write, d.setRegs, d.snapshot
};
```

Of memory it records what is asked for and no more: the watched regions,
a window above the stack pointer, and at most 256 bytes of what a syscall
points at (`-s`), each as the 8-byte words that changed since the stop
before. What it costs (`zig build bench-debugger`; release, this machine):

| | a second | µs | bytes a moment |
| --- | --- | --- | --- |
| an instruction: ptrace step and wait | 273 000 | 3.7 | |
| ... as `Debugger.stepi` with events, the source line looked up | 260 000 | 3.9 | |
| ... and 128 bytes of stack watched | 212 000 | 4.7 | |
| ... recorded (`ztrace record`) | 189 000 | 5.3 | 55 |
| a source line (16 instructions here), recorded | 16 000 | 62 | 60 |
| a breakpoint hit, recorded | 82 000 | 12 | 94 |
| a syscall, entry and exit: native 0.1 µs; stopped at | 120 000 | 8.3 | |
| ... recorded (`--by syscall`) | 100 000 | 10 | 208 |

Memory is read with `process_vm_readv`: 64 bytes in 0.3 µs where
`PTRACE_PEEKDATA` takes 2.2, a megabyte at 22 GiB/s against 28 MiB/s. It is
written the same way, and through `/proc/pid/mem` where the page is
read-only (a breakpoint in code: 0.5 µs).

Limits: `bt` walks the unwind tables, and where there are none (JIT code)
frame pointers, where a function without a frame hides its caller. Threads are followed, each stop is one thread's and the others run
on; stepping by line in a threaded program reports the other threads'
instructions too. A target that execs is not followed into the new program.

## Compiled in

For a program built from source, `ztrace cc` runs clang with a pass plugin
(`llvm/pass.cpp`, built against the system LLVM) and links a runtime
(`src/instr/rt.zig`, `libztrace-rt.a`). The program then tells its own
story at the level of its source, names and types included: no DWARF is
read at view time.

```sh
ztrace cc -- clang -O0 targets/instr_sort.c -o sort && ztrace run -- ./sort
ztrace cc --only 'sqlite3Btree*,*/pager.c' --mem '*Insert*' -- clang -c sqlite3.c
zig build test-llvm                          # fib, sorts, a list, a BFS: asserted from their recordings
zig build bench-llvm -Doptimize=ReleaseFast  # the tables these numbers come from
```

| `--instr` | event | bytes recorded |
| --- | --- | --- |
| `lines` | `line` (file, line, column) before each new line of a block: a moment each | 31 |
| `calls` | `call` (name, site, sp, type) at entry, `ret` (name, value) at exit | 14 |
| `vars` | `variable` (address, size, name, type) per local at entry, per global at start | 15 |
| `stores` | `mem_write` with the bytes after each store; a memset or memcpy is one write | 14 |
| `loads` | `mem_read` after each load | 12 |
| `edges`, `cmps`, `cyg` | stock clang, no plugin and no names: `edge`, `cmp`, `call`/`ret` by address | 12, 13, 19 |

The default is the first four: 44 to 63 bytes a source line executed, of
which the stores are 9 to 14 and the moment a line is 22. Stores are what
grows, so they are what to select: `--only` is where anything is
instrumented (globs on function names, source paths, file names), `--mem`
where stores and loads are, within that. The compiler's own stack
temporaries are never recorded; a write records its first 64 KiB; a local
is declared, not dumped, and a global is written once, at the start.

- **Names.** Every file, function, variable and type is a `string` event
  before it is used. The runtime numbers them when the program starts, so
  an id is a byte in the recording.
- **Types.** A variable's type is a description a viewer can draw from
  (`src/instr/types.zig` has the grammar and its parser): `[8]i32`,
  `` *`struct node` ``, and under `types.defId("struct node")` the string
  `` struct(16){value@0:i32;next@8:*`struct node`;} ``.
- **Before the optimizer** by default: every local still has an address,
  every source store exists, lines come in source order, and what the
  calls made observable the optimizer must keep, so `-O2` records the same
  events as `-O0`. `--late` instruments what the optimizer left: the real
  stores and loads, few variables.
- **Stepping.** A `line` is an event like any other, so the ring's stop
  mask holds the program at each (`host.Options.stop_mask`): source-level
  step with no ptrace, 1.4 million stops a second with a tracer that
  polls, 290 000 with one that sleeps in between.
- **Cost.** An event is 60 to 70 ns into the ring, whatever its kind
  (nearly all of it the ring's two contended cache lines), and 1.5 ns when
  nobody listens. A line of these targets is a few ns of work, so with
  the default selection they run 7 to 12 times slower untraced and 250 to
  400 times slower traced at `-O0`. Recording at a moment a line takes
  1.9 million events a second.

## The agent

`libztrace-agent.so` is preloaded into a program (`ztrace run --preload`);
`libztrace-agent.a` is linked into one, which can then say things about
itself (`include/ztrace.h`; a Zig program imports the `ztrace-agent` module
instead). Inside the target it has no libc and makes raw syscalls only, so
the same library serves glibc and musl, dynamic and static. With no tracer
it does nothing: a preloaded malloc costs 2 ns more.

```sh
ztrace run --preload zig-out/lib/libztrace-agent.so --alloc -- ./agent_list 100
ztrace run --preload ... --syscalls -- ls /
ztrace run --preload ... --hook fib -- ./agent_fib 10        # ztrace show FILE 6: the call stack then
ztrace run --preload ... --hook swap --watch numbers -- ./agent_sort
ztrace run --preload ... --hook swap --stop-at ret --read numbers -- ./agent_sort
ztrace run --preload ... --hook fib --stop-at call -i -- ./agent_fib 5
ztrace run -- ./agent_api                                    # the agent linked in, annotating itself
zig build test-agent
zig build bench-agent -Doptimize=ReleaseFast
```

| | events | how |
| --- | --- | --- |
| startup | `map` a mapping, `thread_start` a thread | /proc/self/maps; pthread_create and fork interposed |
| `--alloc` | `alloc`, `free`, with the caller's pc | malloc and its family interposed (`dlsym(RTLD_NEXT)`; `--wrap` when linked statically) |
| `--syscalls` | `syscall_enter` (path, bytes written: `--capture`, 256), `syscall_exit` | Syscall User Dispatch: SIGSYS, handled in the target |
| `--hook SYM` | `call` with the registers, `ret` with the value | int3 at the entry, the return address swapped |
| `--watch SYM` | `mem_write`: the bytes that changed, before each event that ends a moment | compared in place against a shadow copy |
| `ztrace_mark`, `ztrace_watch`, `ztrace_snapshot` | `mark`; `variable` and the watch; `mem_write` | the target's own calls |
| `--stop-at KIND` | the target held at each, while the tracer reads (`--read`) or asks (`-i`) | the ring's `stop_mask`, a futex |

Why these. **Hooks** are breakpoints because a breakpoint needs to know
nothing about the code it lands on: an inline jump needs an x86-64 length
decoder to move the prologue, and patching the PLT misses calls inside a
module. The price is a signal a call; the return costs none. **Watches**
compare memory in the target's own address space because that is exact at
any size, needs no syscall, and does not disturb the target: page
protection makes the kernel refuse syscalls that write there, and soft-dirty
bits still need the compare, a page at a time. **Syscalls** go through
SIGSYS because the kernel then needs no second process: the cost is two
signal frames, on a stack of the agent's so that it works on threads with
almost none. A handler starts with the kernel's defaults, not the thread's
rights to memory behind protection keys (PKRU) nor its floating-point
modes; the agent's handlers take the thread's back first, so that a syscall
made for it, a thread cloned for it and a watched page read for it are as
the thread would have them (V8 keeps its code space behind a key).

Measured (`zig build bench-agent -Doptimize=ReleaseFast`, an i7-11700, each
loop 10^5 to 5 10^6 times; bytes are the recording's):

| | native | traced | recorded |
| --- | --- | --- | --- |
| a malloc and a free, `--alloc` | 4 ns | 168 ns (11.7 M events/s) | 14 bytes an event |
| ... each a moment (`--cut alloc,free`) | | 4.9 µs, the recorder's | 65 bytes a moment |
| a syscall, `--syscalls` | 73 ns | 950 ns; 830 ns is the two signal frames | 87 bytes a syscall, a moment |
| a syscall under ptrace (`ztrace record --syscalls`) | | 9000 ns | 129 bytes |
| a hooked call | 1 ns | 990 ns | 77 bytes: two moments, with registers |
| ... watching 1 KiB, 64 KiB, 1 MiB | | +43 ns, +1.5 µs, +17.5 µs a moment | what changed |
| ... held at its return (`--stop-at ret`) | | +1.3 µs | |
| ... the tracer reading 1 KiB, 64 KiB, 1 MiB from outside | | +2.5 µs, +6.5 µs, +66 µs a stop | what changed |
| events that end no moment, through the ring | | 11 M a second | 8 bytes an event |
| marks, each a moment | | 2.5 M a second | 32 bytes a moment |

`ztrace run` ends a moment at the recorder's kinds and at `call` and `ret`
(hooks are asked for by name, and a call and its return are the steps of
such a recording); `alloc` and `free` end none unless `--cut` says so. A
recording that reaches `--max-size` (1G) ends the run.

Limits: a hooked function called with SIGTRAP blocked dies (not under
`--syscalls`, which keeps the target's masks clear of it); C++ exceptions
do not unwind through a hooked call; a pointer the kernel would answer with
EFAULT faults in the handler; syscalls made from the vDSO never enter the
kernel and are not seen, as under ptrace; a target that needs SIGSYS or its
own alternate signal stack is told it has them. `--alloc` sees calls that
reach libc's allocator by name: a program that brings its own `malloc` and
binds it inside itself (d8 does) shows none, and `--hook malloc` is the way
to see those. A target that dies under the agent dumps core like any other,
which for a V8 takes half a minute; `ulimit -c 0` if that is not wanted.

## The kernel's view

Two tiers, by what a machine lets its user do. Both hold the program at its
first instruction until they are attached, then let it run at full speed:
no stops, records out of a kernel ring buffer, stamped with the clock every
other backend uses.

**`ztrace perf`** needs no privilege: `perf_event_open` as
`kernel.perf_event_paranoid=2` allows it, which is user space only, on one's
own processes. `perf.Session.open(gpa, pid, options, sink)` is the same for
a tracer that has its own target (stopped, or it misses the start).

| option | events | costs the program | bytes an event |
| --- | --- | --- | --- |
| `-e`, `--interval MS` | `sample`: a counter's total | nothing measurable | 27 |
| `--sample HZ`, `--callchain` | `sample`: pc, callers | 3% at 40 kHz with callers | 24, 48 |
| `--faults` | `fault` and the registers | 0.3 us a fault (the fault itself: 0.9) | 40 |
| `--switches` | `sched` | nothing measurable | 10 |
| `--syscalls` | `syscall_enter` with the path or bytes, `syscall_exit` | 0.4 us a syscall (ptrace: 10), and a second at the end | 23 |
| `--watch SYMBOL` | `mem_write` or `mem_read`, and the pc | 0.9 us a hit | 17 |
| `--break SYMBOL` | `probe`, five arguments, the registers | 1.3 us a hit | 73 |
| always | `map`, `thread_start`, `thread_exit`, `mark` (comm), `exit` | | |

(Release build, an i7-11700; a ptrace stop is 5 us. It decodes and records
one to four million events a second; what the kernel could not put in its
ring is counted as lost: `--ring`.) `--threads` follows threads and
children, a ring per CPU merged by time. Moments end where the recorder
ends them anyway, a syscall's exit or a megabyte of events, unless `--cut`
asks for one at every fault and hit. What to know:

- Only user space is seen. A page the kernel touches first (`read()` into
  fresh memory) faults unseen; context switches and migrations cannot be
  counted, only reported as events.
- A huge page is one fault for 2 MiB. An anonymous mapping the kernel
  merges with its neighbour is reported as the merged whole.
- Syscalls come from the 736 per-syscall tracepoints, the only ones a user
  may read: found by trying every id, closed at a millisecond each.
- There are four debug registers; a watchpoint covers 1, 2, 4 or 8 aligned
  bytes; the record comes after the access, so the value is read when it
  is decoded, later (or not at all, once the program is gone).

**`ztrace probe`** is for root (CAP_BPF and CAP_PERFMON) and says so
otherwise; `--dry-run` builds everything and lists it. BPF programs, built
here as instruction arrays for one process, on the raw tracepoints
`sys_enter`, `sys_exit` and `sched_switch` (both threads of a switch,
through the kernel's BTF), on kprobes, on uprobes, and on USDT probes,
whose arguments are read in the kernel at the probe (the program's own
semaphores are raised by writing them: the kernel's counter misses when
lld has mapped that page of the file twice). They write `Event`s
into one BPF ring buffer and count what did not fit. A syscall costs 0.15
us, a probe 0.5 to 0.8 (in a VM), 22 to 73 bytes an event.

**USDT**, the static probes of DTrace: `usdt.probes(gpa, elf, bias)` reads
the notes `<sys/sdt.h>` leaves (provider, name, pc, semaphore, arguments)
and `usdt.Arg.parse(spec).fetch(regs, memory)` gets an argument at a
stopped thread. No BPF in it: a ptrace breakpoint on the nop does as well.

`zig build test-kernel` runs the first tier and USDT on `targets/kernel_*`,
and the BPF programs in a small BPF machine. `zig build test-kernel-vm`
runs them for real: this machine's kernel in QEMU, where we are root.

## Depends on

- [sketchbook](../sketchbook): window, drawing, shaders, hot reload, the
  manim-style `anim` module, frame export.
- [pardes](../02-pardes-code): tree-sitter parsing, highlighting, themes.
- [xitdb](https://github.com/xit-vcs/xitdb): the recording.
- [capstone](https://github.com/capstone-engine/capstone): disassembly, in the command.

x86-64 Linux, Zig 0.17.