1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
# micscope
Records the microphone or opens sound files, plays them forwards or
backwards, and shows each as a waveform, its spectrogram on the same time
axis, and the spectrum at the playhead. A [sketchbook](../..) sketch, Linux only for now.
The window shows a vertical strip of these panels, laid out in a world
and seen through a camera, as in a game. Takes stack downwards in the
order they were made, so each keeps its place; at the bottom is now: what
the microphone hears, its last 6 s scrolling by, with the spectrum of the
newest samples. Recording turns that panel red and fills it with the new
take; stopping leaves the take there and puts the live view under it. The
camera follows the bottom until you pan away, and follows again once you
pan back down. Zoomed out, it overviews the whole strip. Every take keeps
its own time zoom and playhead.
A tuner reads the live sound (or the recording, or the selected take at
its playhead) and shows it on that panel's spectrum: a line at the pitch,
ticks at its harmonics, the two semitones either side shaded, and that
shaded span magnified across the top: the nearest notes of the 12-tone
chromatic scale, the pitch as a needle among them (green within 5 cents,
yellow within 15), and the note, cents and Hz. It is not a guitar tuner
in particular: it finds 30 Hz to 4.2 kHz, a five-string bass's low B to a
piccolo's top, against any A4 (`[` and `]`, 440 Hz by default).
Panels are rendered at world resolution and the camera only scales them,
so moving or zooming it recomputes nothing.
The waveform, spectrogram and spectrum shaders are sketchbook's `scope`
module's (`src/scope/shaders/`): the live view is a `scope.Scope`, and the
take panels run `scope.kernels` and `scope.pipelines` over the tape. The
tuner's are micscope's own.
| Shader | Does |
| --- | --- |
| `spectrum.comp` | Blackman-Harris window around the playhead, zero-padded 2×, radix-2 Stockham FFT (8192 points, one workgroup), dBFS per bin, then one value per log-spaced pixel column: a Catmull-Rom spline through the bins where columns are narrower than a bin, the loudest bin where they span several. Smoothed in time (fast attack, slow release) with falling peaks. |
| `waveform.comp` | Per pixel column: min, max and RMS of its samples, plus the interpolated signal at the column's edges, so zoomed past one sample per pixel the columns still join into a line. Recomputed only when the view or tape changes. |
| `spectrogram.comp` | One workgroup per waveform column, so it lines up with the waveform in time. A 2048-sample Blackman-Harris window zero-padded to 4096, transformed as 2048 complex points in shared memory (the real-input trick) and split into 2049 bins. Below 500 Hz a second analysis runs on the signal low-passed and decimated 4×: the same 2048 points span 170 ms and resolve ~6 Hz instead of 23, three windows a quarter apart averaged (Welch) so noise does not speckle; blended with the first between 250 and 500 Hz. Zoomed out, each column averages up to four windows spread across its samples, so it neither skips nor flickers. Rows are log frequency, interpolated like the spectrum. |
| `tuner_difference.comp`, `tuner.comp` | Pitch by YIN rather than by the strongest FFT peak, which is often a harmonic (low strings, voices and brass put more into the 2nd or 3rd) or missing altogether (a missing fundamental). The difference function d(τ) = Σ(x[j] − x[j−τ])² over 2048 samples for lags up to 1700, spread over 107 workgroups; then in one workgroup, normalised by its running mean (a parallel scan), the first dip under 0.12 (the shortest period that explains the signal, so no octave drops) followed to its bottom, and a parabola through the raw dip for the period between samples. Read back to the CPU (`f.download`), which follows the median of the last five clear estimates, restarts on a jump of more than a semitone, and only changes the note shown 60 cents away from it. |
| `waveform.frag`, `spectrogram.frag`, `spectrum.frag` | Full-screen triangles that draw those buffers: the waveform and spectrum anti-aliased with grids, the spectrogram through an inferno heat gradient. |
The takes share one tape buffer on the GPU (each analysis reads from an
offset), uploaded once. A take's waveform and spectrogram are recomputed
only when its view changes, and its panel redrawn only when something on
it moved; only the selected take's spectrum runs every frame.
The live view (the scope module's) uploads the microphone's last 1.4 s each frame and analyses
only the columns that became complete, into ring buffers the fragment
shaders read from the oldest column. A column waits until everything its
spectrogram windows reach has been heard (6160 samples past its centre),
so it is analysed once and final: the live spectrogram runs about 130 ms
behind, the live spectrum does not.
## Keys
| Key | |
| --- | --- |
| R | record a new take / stop |
| Space | play / pause the selected take, resuming in the last direction |
| Right / Left | play forwards / backwards |
| Home / End | seek to the start / end |
| Up / Down | select the older / newer take |
| Delete | delete the selected take |
| O | open a file (WAV, FLAC, Ogg Vorbis) as a take, through the system dialog |
| click, drag | select a take; on its timeline, seek |
| wheel, middle or right drag, Page Up / Down | pan the camera |
| Ctrl + wheel, or a pinch | zoom what is under the pointer: a take's time on its timeline (Z unzooms), the camera elsewhere |
| 0 | the whole strip, or back to full size |
| - / = | waveform gain |
| [ / ] | the tuner's A4, 1 Hz down / up |
| \` | the sketchbook console: `open <path>` opens a file too |
The buttons (REC, PLAY, REV, OPEN, DEL, ALL, A4−, A4+) do what the keys
do, for touch screens: there, one finger clicks and drags, two fingers
pan, and a pinch zooms like ctrl+wheel.
The canvas is the window, and the layout follows it. Wide, each panel has
the spectrum beside the waveform and spectrogram, and the buttons sit in
the header; narrower than 760 px (a phone held upright), the spectrum goes
under them at full width, and the buttons along the bottom, where thumbs
are.
## Rendering reference
`test/reference.flac` puts a mastered song, a city street and a sweep on one
screen (sources and licences in `test/README.md`); `zig build test` plays it
through `test/render.script` and leaves screenshots in `.zig-cache/micscope/`.
What it settled for the spectrogram's low end, checked against a plain
numpy STFT of the same columns:
- One long window resolves the street's two engine hums but shows noise as
speckle full of holes: a single window's spectrum of noise scatters as
much as its level.
- Multitaper averaging (sine tapers, DPSS with NW 2 to 4) calms the noise but
widens the hums into one slab and leaks into silence, which a 96 dB scale
shows. Wrong trade for lines.
- Averaging three windows a quarter apart in time keeps the hums sharp and
the kicks distinct, and settles the noise. That is what it does.
`test/tones.wav` has five notes where the strongest peak is not the note
(a bass E1 and a guitar E2 louder in their 2nd harmonic, a D4 without its
fundamental), an A4 12 cents sharp and a piccolo C7; `test/tuner.script`
screenshots the tuner on each. All read within 0.1 cent.
## GPU time
`zig build profile` zooms the reference one step per frame, recomputing the
waveform and spectrogram every frame, and prints GPU time per command
(`--profile-gpu`: each command timed alone on an idle GPU). On an RTX 3050:
| Case | GPU per frame |
| --- | --- |
| playing, views cached | ~0.45 ms |
| zooming the 32 s reference | ~1.1 ms on average, spectrogram 1.5 ms on the frames it recomputes |
| zooming a 10 minute tape | ~1.6 ms on average, spectrogram 2.1 ms (worst 4.6 ms) |
| zooming the reference, in the strip with the live view and both tuners | ~1.15 ms on average; the live view adds a spectrum (~0.15 ms) and a few columns, each tuner ~0.1 ms |
## Building
Zig 0.17 and `glslc` (shaderc) on PATH; for the web build also `naga`
(`cargo install naga-cli`), which translates the shaders to WGSL, and to
serve it `miniserve` and `openssl`.
zig build run # with hot reload: `zig build lib --watch` alongside
zig build test # unit tests and the headless scripts in test/
zig build profile # GPU time per command while zooming
zig-out/bin/micscope # the standalone build
zig build serve # the web build at https://<this machine>:8042/
# (`zig build web --watch` alongside rebuilds it)
## On the web
The same sketch, compiled to wasm32-freestanding, on WebGPU: every shader
runs as on native, translated to WGSL, and the tuner's readback is a
WebGPU buffer mapped a frame or two later. Sound and the microphone go
through Web Audio (the page asks for the microphone, and sound starts at
the first click or key, as browsers require); OPEN decodes a file with the
browser's own decoders. Takes last as long as the page.
It is an installable app (a PWA): browsers offer to add it to the home
screen, and once opened it works offline, from a copy its service worker
keeps of the page, the code and the wasm. The icon is `web/icon.svg`,
rendered to the PNGs beside it with `resvg -w N -h N icon.svg icon-N.png`. It needs a browser
with WebGPU (Chrome, Edge, Safari 26; Firefox on Linux with
`dom.webgpu.enabled` set in about:config), and a secure context, which
`zig build serve`'s https gives it on any device on the network once the
browser has accepted its self-signed certificate (miniserve prints the
addresses as QR codes, for phones).
Each take is a file of its samples (mono 32-bit floats at the host's
rate), `micscope.take-<id>.f32` beside the save (in `.sketchbook/`
under `zig build run`), written as it records (`Host.appendFile`). State
lists the takes, so they are still there after a restart. A take starts
its file empty (`Host.writeFile`). A deleted take's file stays: a seek
back, or any mark from before the delete, finds it again. Takes stop at ten minutes; the
strip holds 64. A `micscope.state.store` from before is not read: its takes
stay in it.
|