summaryrefslogtreecommitdiff
path: root/examples/micscope/README.md
diff options
context:
space:
mode:
authorGabriel Schneider <[email protected]>2026-11-17 00:15:15 -0300
committerGabriel Schneider <[email protected]>2026-11-17 12:15:01 -0300
commit9bd77e54943d954291b5faef287ccca5dcebd7a4 (patch)
tree3f2d628781441568620b8108fab920713f26f649 /examples/micscope/README.md
parente997fc3e6b312b30f5af90012fc22365bab701d4 (diff)
downloadsketchbook-9bd77e54943d954291b5faef287ccca5dcebd7a4.tar.gz
sketchbook-9bd77e54943d954291b5faef287ccca5dcebd7a4.zip
examples: micscope and spheres come home, as they were, building
Both lived outside any repository, beside sketchbook, with manifests that pointed at ../01-drat-meout/lib/sketchbook, which no longer exists. They are copied into examples/micscope and examples/spheres (their sources, shaders, tests, test material and web icons; not their caches, builds, packages, state files or .gitignore, which the root's covers), their manifests depend on "../.." as the other examples do, and their READMEs link sketchbook there. Nothing else changed: both build against today's library and pass their own zig build test (micscope 62/62 steps, 11/11 tests, its transport, record, render and tuner scripts; spheres 75/75 steps, 3/3 tests, its play script). The originals are untouched. micscope's test/reference.flac (3.7 MiB, CC BY-SA 3.0, credited in its test/README.md) is the rendering reference its render script opens. The README lists both examples. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_016Y9MNBECMWcJD5pdS4SR6T
Diffstat (limited to 'examples/micscope/README.md')
-rw-r--r--examples/micscope/README.md148
1 files changed, 148 insertions, 0 deletions
diff --git a/examples/micscope/README.md b/examples/micscope/README.md
new file mode 100644
index 0000000..759c27c
--- /dev/null
+++ b/examples/micscope/README.md
@@ -0,0 +1,148 @@
+# micscope
+
+Records the microphone or opens sound files, plays them forwards or
+backwards, and shows each as a waveform, its spectrogram on the same time
+axis, and the spectrum at the playhead. A [sketchbook](../..) sketch, Linux only for now.
+
+The window shows a vertical strip of these panels, laid out in a world
+and seen through a camera, as in a game. Takes stack downwards in the
+order they were made, so each keeps its place; at the bottom is now: what
+the microphone hears, its last 6 s scrolling by, with the spectrum of the
+newest samples. Recording turns that panel red and fills it with the new
+take; stopping leaves the take there and puts the live view under it. The
+camera follows the bottom until you pan away, and follows again once you
+pan back down. Zoomed out, it overviews the whole strip. Every take keeps
+its own time zoom and playhead.
+
+A tuner reads the live sound (or the recording, or the selected take at
+its playhead) and shows it on that panel's spectrum: a line at the pitch,
+ticks at its harmonics, the two semitones either side shaded, and that
+shaded span magnified across the top: the nearest notes of the 12-tone
+chromatic scale, the pitch as a needle among them (green within 5 cents,
+yellow within 15), and the note, cents and Hz. It is not a guitar tuner
+in particular: it finds 30 Hz to 4.2 kHz, a five-string bass's low B to a
+piccolo's top, against any A4 (`[` and `]`, 440 Hz by default).
+
+Panels are rendered at world resolution and the camera only scales them,
+so moving or zooming it recomputes nothing.
+
+| Shader | Does |
+| --- | --- |
+| `spectrum.comp` | Blackman-Harris window around the playhead, zero-padded 2×, radix-2 Stockham FFT (8192 points, one workgroup), dBFS per bin, then one value per log-spaced pixel column: a Catmull-Rom spline through the bins where columns are narrower than a bin, the loudest bin where they span several. Smoothed in time (fast attack, slow release) with falling peaks. |
+| `waveform.comp` | Per pixel column: min, max and RMS of its samples, plus the interpolated signal at the column's edges, so zoomed past one sample per pixel the columns still join into a line. Recomputed only when the view or tape changes. |
+| `spectrogram.comp` | One workgroup per waveform column, so it lines up with the waveform in time. A 2048-sample Blackman-Harris window zero-padded to 4096, transformed as 2048 complex points in shared memory (the real-input trick) and split into 2049 bins. Below 500 Hz a second analysis runs on the signal low-passed and decimated 4×: the same 2048 points span 170 ms and resolve ~6 Hz instead of 23, three windows a quarter apart averaged (Welch) so noise does not speckle; blended with the first between 250 and 500 Hz. Zoomed out, each column averages up to four windows spread across its samples, so it neither skips nor flickers. Rows are log frequency, interpolated like the spectrum. |
+| `tuner_difference.comp`, `tuner.comp` | Pitch by YIN rather than by the strongest FFT peak, which is often a harmonic (low strings, voices and brass put more into the 2nd or 3rd) or missing altogether (a missing fundamental). The difference function d(τ) = Σ(x[j] − x[j−τ])² over 2048 samples for lags up to 1700, spread over 107 workgroups; then in one workgroup, normalised by its running mean (a parallel scan), the first dip under 0.12 (the shortest period that explains the signal, so no octave drops) followed to its bottom, and a parabola through the raw dip for the period between samples. Read back to the CPU (`f.download`), which follows the median of the last five clear estimates, restarts on a jump of more than a semitone, and only changes the note shown 60 cents away from it. |
+| `waveform.frag`, `spectrogram.frag`, `spectrum.frag` | Full-screen triangles that draw those buffers: the waveform and spectrum anti-aliased with grids, the spectrogram through an inferno heat gradient. |
+
+The takes share one tape buffer on the GPU (each analysis reads from an
+offset), uploaded once. A take's waveform and spectrogram are recomputed
+only when its view changes, and its panel redrawn only when something on
+it moved; only the selected take's spectrum runs every frame.
+
+The live view uploads the microphone's last 1.4 s each frame and analyses
+only the columns that became complete, into ring buffers the fragment
+shaders read from the oldest column. A column waits until everything its
+spectrogram windows reach has been heard (6160 samples past its centre),
+so it is analysed once and final: the live spectrogram runs about 130 ms
+behind, the live spectrum does not.
+
+## Keys
+
+| Key | |
+| --- | --- |
+| R | record a new take / stop |
+| Space | play / pause the selected take, resuming in the last direction |
+| Right / Left | play forwards / backwards |
+| Home / End | seek to the start / end |
+| Up / Down | select the older / newer take |
+| Delete | delete the selected take |
+| O | open a file (WAV, FLAC, Ogg Vorbis) as a take, through the system dialog |
+| click, drag | select a take; on its timeline, seek |
+| wheel, middle or right drag, Page Up / Down | pan the camera |
+| Ctrl + wheel, or a pinch | zoom what is under the pointer: a take's time on its timeline (Z unzooms), the camera elsewhere |
+| 0 | the whole strip, or back to full size |
+| - / = | waveform gain |
+| [ / ] | the tuner's A4, 1 Hz down / up |
+| \` | the sketchbook console: `open <path>` opens a file too |
+
+The buttons (REC, PLAY, REV, OPEN, DEL, ALL, A4−, A4+) do what the keys
+do, for touch screens: there, one finger clicks and drags, two fingers
+pan, and a pinch zooms like ctrl+wheel.
+
+The canvas is the window, and the layout follows it. Wide, each panel has
+the spectrum beside the waveform and spectrogram, and the buttons sit in
+the header; narrower than 760 px (a phone held upright), the spectrum goes
+under them at full width, and the buttons along the bottom, where thumbs
+are.
+
+## Rendering reference
+
+`test/reference.flac` puts a mastered song, a city street and a sweep on one
+screen (sources and licences in `test/README.md`); `zig build test` plays it
+through `test/render.script` and leaves screenshots in `.zig-cache/micscope/`.
+
+What it settled for the spectrogram's low end, checked against a plain
+numpy STFT of the same columns:
+
+- One long window resolves the street's two engine hums but shows noise as
+ speckle full of holes: a single window's spectrum of noise scatters as
+ much as its level.
+- Multitaper averaging (sine tapers, DPSS with NW 2 to 4) calms the noise but
+ widens the hums into one slab and leaks into silence, which a 96 dB scale
+ shows. Wrong trade for lines.
+- Averaging three windows a quarter apart in time keeps the hums sharp and
+ the kicks distinct, and settles the noise. That is what it does.
+
+`test/tones.wav` has five notes where the strongest peak is not the note
+(a bass E1 and a guitar E2 louder in their 2nd harmonic, a D4 without its
+fundamental), an A4 12 cents sharp and a piccolo C7; `test/tuner.script`
+screenshots the tuner on each. All read within 0.1 cent.
+
+## GPU time
+
+`zig build profile` zooms the reference one step per frame, recomputing the
+waveform and spectrogram every frame, and prints GPU time per command
+(`--profile-gpu`: each command timed alone on an idle GPU). On an RTX 3050:
+
+| Case | GPU per frame |
+| --- | --- |
+| playing, views cached | ~0.45 ms |
+| zooming the 32 s reference | ~1.1 ms on average, spectrogram 1.5 ms on the frames it recomputes |
+| zooming a 10 minute tape | ~1.6 ms on average, spectrogram 2.1 ms (worst 4.6 ms) |
+| zooming the reference, in the strip with the live view and both tuners | ~1.15 ms on average; the live view adds a spectrum (~0.15 ms) and a few columns, each tuner ~0.1 ms |
+
+## Building
+
+Zig 0.17 and `glslc` (shaderc) on PATH; for the web build also `naga`
+(`cargo install naga-cli`), which translates the shaders to WGSL, and to
+serve it `miniserve` and `openssl`.
+
+ zig build run # with hot reload: `zig build lib --watch` alongside
+ zig build test # unit tests and the headless scripts in test/
+ zig build profile # GPU time per command while zooming
+ zig-out/bin/micscope # the standalone build
+ zig build serve # the web build at https://<this machine>:8042/
+ # (`zig build web --watch` alongside rebuilds it)
+
+## On the web
+
+The same sketch, compiled to wasm32-freestanding, on WebGPU: every shader
+runs as on native, translated to WGSL, and the tuner's readback is a
+WebGPU buffer mapped a frame or two later. Sound and the microphone go
+through Web Audio (the page asks for the microphone, and sound starts at
+the first click or key, as browsers require); OPEN decodes a file with the
+browser's own decoders. Takes last as long as the page.
+
+It is an installable app (a PWA): browsers offer to add it to the home
+screen, and once opened it works offline, from a copy its service worker
+keeps of the page, the code and the wasm. The icon is `web/icon.svg`,
+rendered to the PNGs beside it with `resvg -w N -h N icon.svg icon-N.png`. It needs a browser
+with WebGPU (Chrome, Edge, Safari 26; Firefox on Linux with
+`dom.webgpu.enabled` set in about:config), and a secure context, which
+`zig build serve`'s https gives it on any device on the network once the
+browser has accepted its self-signed certificate (miniserve prints the
+addresses as QR codes, for phones).
+
+The takes are the sketch's `Store`: they live in `micscope.state.store` and
+are still there after a restart. Takes stop at ten minutes; the strip holds
+64. A store from before the strip has its one tape as the first take.