diff options
| author | Gabriel Schneider <[email protected]> | 2026-11-17 00:15:15 -0300 |
|---|---|---|
| committer | Gabriel Schneider <[email protected]> | 2026-11-17 12:15:01 -0300 |
| commit | 9bd77e54943d954291b5faef287ccca5dcebd7a4 (patch) | |
| tree | 3f2d628781441568620b8108fab920713f26f649 /examples/micscope | |
| parent | e997fc3e6b312b30f5af90012fc22365bab701d4 (diff) | |
| download | sketchbook-9bd77e54943d954291b5faef287ccca5dcebd7a4.tar.gz sketchbook-9bd77e54943d954291b5faef287ccca5dcebd7a4.zip | |
examples: micscope and spheres come home, as they were, building
Both lived outside any repository, beside sketchbook, with manifests that
pointed at ../01-drat-meout/lib/sketchbook, which no longer exists. They
are copied into examples/micscope and examples/spheres (their sources,
shaders, tests, test material and web icons; not their caches, builds,
packages, state files or .gitignore, which the root's covers), their
manifests depend on "../.." as the other examples do, and their
READMEs link sketchbook there. Nothing else changed: both build against
today's library and pass their own zig build test (micscope 62/62 steps,
11/11 tests, its transport, record, render and tuner scripts; spheres
75/75 steps, 3/3 tests, its play script). The originals are untouched.
micscope's test/reference.flac (3.7 MiB, CC BY-SA 3.0, credited in its
test/README.md) is the rendering reference its render script opens. The
README lists both examples.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Y9MNBECMWcJD5pdS4SR6T
Diffstat (limited to 'examples/micscope')
27 files changed, 3143 insertions, 0 deletions
diff --git a/examples/micscope/README.md b/examples/micscope/README.md new file mode 100644 index 0000000..759c27c --- /dev/null +++ b/examples/micscope/README.md @@ -0,0 +1,148 @@ +# micscope + +Records the microphone or opens sound files, plays them forwards or +backwards, and shows each as a waveform, its spectrogram on the same time +axis, and the spectrum at the playhead. A [sketchbook](../..) sketch, Linux only for now. + +The window shows a vertical strip of these panels, laid out in a world +and seen through a camera, as in a game. Takes stack downwards in the +order they were made, so each keeps its place; at the bottom is now: what +the microphone hears, its last 6 s scrolling by, with the spectrum of the +newest samples. Recording turns that panel red and fills it with the new +take; stopping leaves the take there and puts the live view under it. The +camera follows the bottom until you pan away, and follows again once you +pan back down. Zoomed out, it overviews the whole strip. Every take keeps +its own time zoom and playhead. + +A tuner reads the live sound (or the recording, or the selected take at +its playhead) and shows it on that panel's spectrum: a line at the pitch, +ticks at its harmonics, the two semitones either side shaded, and that +shaded span magnified across the top: the nearest notes of the 12-tone +chromatic scale, the pitch as a needle among them (green within 5 cents, +yellow within 15), and the note, cents and Hz. It is not a guitar tuner +in particular: it finds 30 Hz to 4.2 kHz, a five-string bass's low B to a +piccolo's top, against any A4 (`[` and `]`, 440 Hz by default). + +Panels are rendered at world resolution and the camera only scales them, +so moving or zooming it recomputes nothing. + +| Shader | Does | +| --- | --- | +| `spectrum.comp` | Blackman-Harris window around the playhead, zero-padded 2×, radix-2 Stockham FFT (8192 points, one workgroup), dBFS per bin, then one value per log-spaced pixel column: a Catmull-Rom spline through the bins where columns are narrower than a bin, the loudest bin where they span several. Smoothed in time (fast attack, slow release) with falling peaks. | +| `waveform.comp` | Per pixel column: min, max and RMS of its samples, plus the interpolated signal at the column's edges, so zoomed past one sample per pixel the columns still join into a line. Recomputed only when the view or tape changes. | +| `spectrogram.comp` | One workgroup per waveform column, so it lines up with the waveform in time. A 2048-sample Blackman-Harris window zero-padded to 4096, transformed as 2048 complex points in shared memory (the real-input trick) and split into 2049 bins. Below 500 Hz a second analysis runs on the signal low-passed and decimated 4×: the same 2048 points span 170 ms and resolve ~6 Hz instead of 23, three windows a quarter apart averaged (Welch) so noise does not speckle; blended with the first between 250 and 500 Hz. Zoomed out, each column averages up to four windows spread across its samples, so it neither skips nor flickers. Rows are log frequency, interpolated like the spectrum. | +| `tuner_difference.comp`, `tuner.comp` | Pitch by YIN rather than by the strongest FFT peak, which is often a harmonic (low strings, voices and brass put more into the 2nd or 3rd) or missing altogether (a missing fundamental). The difference function d(τ) = Σ(x[j] − x[j−τ])² over 2048 samples for lags up to 1700, spread over 107 workgroups; then in one workgroup, normalised by its running mean (a parallel scan), the first dip under 0.12 (the shortest period that explains the signal, so no octave drops) followed to its bottom, and a parabola through the raw dip for the period between samples. Read back to the CPU (`f.download`), which follows the median of the last five clear estimates, restarts on a jump of more than a semitone, and only changes the note shown 60 cents away from it. | +| `waveform.frag`, `spectrogram.frag`, `spectrum.frag` | Full-screen triangles that draw those buffers: the waveform and spectrum anti-aliased with grids, the spectrogram through an inferno heat gradient. | + +The takes share one tape buffer on the GPU (each analysis reads from an +offset), uploaded once. A take's waveform and spectrogram are recomputed +only when its view changes, and its panel redrawn only when something on +it moved; only the selected take's spectrum runs every frame. + +The live view uploads the microphone's last 1.4 s each frame and analyses +only the columns that became complete, into ring buffers the fragment +shaders read from the oldest column. A column waits until everything its +spectrogram windows reach has been heard (6160 samples past its centre), +so it is analysed once and final: the live spectrogram runs about 130 ms +behind, the live spectrum does not. + +## Keys + +| Key | | +| --- | --- | +| R | record a new take / stop | +| Space | play / pause the selected take, resuming in the last direction | +| Right / Left | play forwards / backwards | +| Home / End | seek to the start / end | +| Up / Down | select the older / newer take | +| Delete | delete the selected take | +| O | open a file (WAV, FLAC, Ogg Vorbis) as a take, through the system dialog | +| click, drag | select a take; on its timeline, seek | +| wheel, middle or right drag, Page Up / Down | pan the camera | +| Ctrl + wheel, or a pinch | zoom what is under the pointer: a take's time on its timeline (Z unzooms), the camera elsewhere | +| 0 | the whole strip, or back to full size | +| - / = | waveform gain | +| [ / ] | the tuner's A4, 1 Hz down / up | +| \` | the sketchbook console: `open <path>` opens a file too | + +The buttons (REC, PLAY, REV, OPEN, DEL, ALL, A4−, A4+) do what the keys +do, for touch screens: there, one finger clicks and drags, two fingers +pan, and a pinch zooms like ctrl+wheel. + +The canvas is the window, and the layout follows it. Wide, each panel has +the spectrum beside the waveform and spectrogram, and the buttons sit in +the header; narrower than 760 px (a phone held upright), the spectrum goes +under them at full width, and the buttons along the bottom, where thumbs +are. + +## Rendering reference + +`test/reference.flac` puts a mastered song, a city street and a sweep on one +screen (sources and licences in `test/README.md`); `zig build test` plays it +through `test/render.script` and leaves screenshots in `.zig-cache/micscope/`. + +What it settled for the spectrogram's low end, checked against a plain +numpy STFT of the same columns: + +- One long window resolves the street's two engine hums but shows noise as + speckle full of holes: a single window's spectrum of noise scatters as + much as its level. +- Multitaper averaging (sine tapers, DPSS with NW 2 to 4) calms the noise but + widens the hums into one slab and leaks into silence, which a 96 dB scale + shows. Wrong trade for lines. +- Averaging three windows a quarter apart in time keeps the hums sharp and + the kicks distinct, and settles the noise. That is what it does. + +`test/tones.wav` has five notes where the strongest peak is not the note +(a bass E1 and a guitar E2 louder in their 2nd harmonic, a D4 without its +fundamental), an A4 12 cents sharp and a piccolo C7; `test/tuner.script` +screenshots the tuner on each. All read within 0.1 cent. + +## GPU time + +`zig build profile` zooms the reference one step per frame, recomputing the +waveform and spectrogram every frame, and prints GPU time per command +(`--profile-gpu`: each command timed alone on an idle GPU). On an RTX 3050: + +| Case | GPU per frame | +| --- | --- | +| playing, views cached | ~0.45 ms | +| zooming the 32 s reference | ~1.1 ms on average, spectrogram 1.5 ms on the frames it recomputes | +| zooming a 10 minute tape | ~1.6 ms on average, spectrogram 2.1 ms (worst 4.6 ms) | +| zooming the reference, in the strip with the live view and both tuners | ~1.15 ms on average; the live view adds a spectrum (~0.15 ms) and a few columns, each tuner ~0.1 ms | + +## Building + +Zig 0.17 and `glslc` (shaderc) on PATH; for the web build also `naga` +(`cargo install naga-cli`), which translates the shaders to WGSL, and to +serve it `miniserve` and `openssl`. + + zig build run # with hot reload: `zig build lib --watch` alongside + zig build test # unit tests and the headless scripts in test/ + zig build profile # GPU time per command while zooming + zig-out/bin/micscope # the standalone build + zig build serve # the web build at https://<this machine>:8042/ + # (`zig build web --watch` alongside rebuilds it) + +## On the web + +The same sketch, compiled to wasm32-freestanding, on WebGPU: every shader +runs as on native, translated to WGSL, and the tuner's readback is a +WebGPU buffer mapped a frame or two later. Sound and the microphone go +through Web Audio (the page asks for the microphone, and sound starts at +the first click or key, as browsers require); OPEN decodes a file with the +browser's own decoders. Takes last as long as the page. + +It is an installable app (a PWA): browsers offer to add it to the home +screen, and once opened it works offline, from a copy its service worker +keeps of the page, the code and the wasm. The icon is `web/icon.svg`, +rendered to the PNGs beside it with `resvg -w N -h N icon.svg icon-N.png`. It needs a browser +with WebGPU (Chrome, Edge, Safari 26; Firefox on Linux with +`dom.webgpu.enabled` set in about:config), and a secure context, which +`zig build serve`'s https gives it on any device on the network once the +browser has accepted its self-signed certificate (miniserve prints the +addresses as QR codes, for phones). + +The takes are the sketch's `Store`: they live in `micscope.state.store` and +are still there after a restart. Takes stop at ten minutes; the strip holds +64. A store from before the strip has its one tape as the first take. diff --git a/examples/micscope/build.zig b/examples/micscope/build.zig new file mode 100644 index 0000000..0853cdc --- /dev/null +++ b/examples/micscope/build.zig @@ -0,0 +1,59 @@ +const std = @import("std"); +const sketchbook = @import("sketchbook"); + +pub fn build(b: *std.Build) void { + const target = b.standardTargetOptions(.{}); + const optimize = b.standardOptimizeOption(.{}); + const sb = sketchbook.init(b, .{ .target = target, .optimize = optimize }); + + // The shaders, as SPIR-V for the native host and WGSL for the web; their + // uniform blocks come from the Zig structs in src/layouts.zig. + const layouts = b.createModule(.{ .root_source_file = b.path("src/layouts.zig") }); + const shaders = sb.addShaders(.{ + .layouts = .{ .module = layouts, .types = &.{ "SpectrumParams", "WaveParams", "WaveView", "SpectrumView", "SpectrogramParams", "SpectrogramView", "TunerParams" } }, + .sources = &.{ + b.path("shaders/view.vert"), + b.path("shaders/waveform.comp"), + b.path("shaders/waveform.frag"), + b.path("shaders/spectrogram.comp"), + b.path("shaders/spectrogram.frag"), + b.path("shaders/spectrum.comp"), + b.path("shaders/spectrum.frag"), + b.path("shaders/tuner_difference.comp"), + b.path("shaders/tuner.comp"), + }, + }); + + const app = sb.app(.{ + .name = "micscope", + .root_source_file = b.path("src/main.zig"), + .imports = &.{ + .{ .name = "layouts", .module = layouts }, + .{ .name = "shaders", .module = shaders.module }, + }, + }); + const steps = app.addSteps(.{ + // Installable, and it opens offline (icons from web/icon.svg). + .web = .{ .pwa = .{ + .description = "Records the microphone, shows its waveform, spectrogram and spectrum, and tunes any instrument.", + .background = "#0b0c0f", + .theme = "#0b0c0f", + .icons = &.{ + .{ .source = b.path("web/icon-192.png"), .size = 192 }, + .{ .source = b.path("web/icon-512.png"), .size = 512 }, + }, + } }, + .scripts = &.{ b.path("test/transport.script"), b.path("test/record.script"), b.path("test/render.script"), b.path("test/tuner.script") }, + }); + + // GPU time per command while zooming, which recomputes every frame. + const profile = b.addRunArtifact(steps.exe); + profile.addArgs(&.{ "--headless", "--fresh", "--profile-gpu", "--script" }); + profile.addFileArg(b.path("test/profile.script")); + profile.addArg("--state"); + _ = profile.addOutputFileArg("profile.state"); + profile.setCwd(b.path(".")); + profile.has_side_effects = true; + profile.stdio = .inherit; + b.step("profile", "Print GPU time per command while zooming the reference").dependOn(&profile.step); +} diff --git a/examples/micscope/build.zig.zon b/examples/micscope/build.zig.zon new file mode 100644 index 0000000..9c595cf --- /dev/null +++ b/examples/micscope/build.zig.zon @@ -0,0 +1,10 @@ +.{ + .name = .micscope, + .version = "0.0.1", + .dependencies = .{ + .sketchbook = .{ .path = "../.." }, + }, + .minimum_zig_version = "0.17.0", + .paths = .{""}, + .fingerprint = 0x780fbc0f374f0ce0, +} diff --git a/examples/micscope/shaders/spectrogram.comp b/examples/micscope/shaders/spectrogram.comp new file mode 100644 index 0000000..b32f4c5 --- /dev/null +++ b/examples/micscope/shaders/spectrogram.comp @@ -0,0 +1,208 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrogram, one workgroup per column, the same columns as the +// waveform (waveform.comp), so the two line up in time. +// +// Two resolutions per column, because one window cannot serve both ends of +// a log axis: 2048 samples resolve 23 Hz, which is sharp in time and fine +// in the kilohertz, but leaves 20 to 100 Hz (a quarter of the height) to +// four bins. So the low end is analysed again from the signal low-passed +// and decimated to a quarter of the rate: the same 2048 points then span +// 170 ms and resolve 6 Hz. Rows take the long analysis below 250 Hz, the +// short one above 500 Hz, and blend in between. +// +// A single window's spectrum of noise scatters as much as its level (each +// bin is chi-squared with 2 degrees of freedom), which in dB shows as +// speckle full of black holes, and finer bins only make more of them. So +// the long analysis averages three windows a quarter window apart in time +// (Welch): steady lines stay as sharp, noise settles. Multitaper averaging +// was tried and measured against a reference (see test/README.md): sine +// tapers and DPSS both widen narrow lines into slabs and leak, which a +// 96 dB scale shows. +// +// Each analysis is a Blackman-Harris window zero-padded to 4096 points, +// transformed as 2048 complex points (even and odd samples as real and +// imaginary parts) by an in-place radix-2 FFT in shared memory, then split +// into the real signal's bins. Zoomed out, a column averages the power of +// up to four windows spread across its samples, so it neither skips sound +// nor flickers as the view moves; zoomed in, neighbouring windows overlap. +// Rows are log-spaced: a Catmull-Rom spline through the bins' levels where +// rows are finer than bins, the loudest bin where they are coarser. +layout(local_size_x = 256) in; + +#include "SpectrogramParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// MAX_ROWS rows of MAX_COLUMNS levels, dBFS; row 0 is the lowest frequency. +layout(set = 1, binding = 0) buffer Levels { float levels[]; }; + +// Match layouts.zig. +const uint MAX_COLUMNS = 2048; +const uint W = 2048; +const uint N = 4096; +const uint HALF = N / 2; +const uint LOG2_HALF = 11; +const uint THREADS = 256; +// The long analysis: every DECIMATE-th sample of the low-passed signal. +const uint DECIMATE = 4; +// Its bins needed below the blend: 600 Hz at 12 kHz / 4096 is bin 205. +const uint LONG_BINS = 256; +// Windows the long analysis averages per column, a quarter window apart. +const uint WELCH = 3; +const float BLEND_LO = 250.0; +const float BLEND_HI = 500.0; +const float PI = 3.14159265358979; +const float SILENCE = -160.0; +// Columns past the end of the tape. +const float NO_TAPE = 1000.0; +const float A0 = 0.35875; +const float A1 = 0.48829; +const float A2 = 0.14128; +const float A3 = 0.01168; + +shared vec2 z[HALF]; +// Power per bin, summed over windows; then dBFS. +shared float short_bins[HALF + 1]; +shared float long_bins[LONG_BINS]; + +float sample_at(int i) { + return i >= 0 && i < int(u.samples) ? tape[u.offset + uint(i)] : 0.0; +} + +// The signal low-passed for decimation: a triangle over 7 samples (two +// boxes of 4), whose zeros sit on 12 and 24 kHz, exactly where anything +// that would fold onto the low bins comes from. +float low_passed(int i) { + return (sample_at(i - 3) + 2.0 * sample_at(i - 2) + 3.0 * sample_at(i - 1) + 4.0 * sample_at(i) + 3.0 * sample_at(i + 1) + 2.0 * sample_at(i + 2) + sample_at(i + 3)) / 16.0; +} + +float blackman_harris(uint i) { + float a = 2.0 * PI * float(i) / float(W - 1); + return A0 - A1 * cos(a) + A2 * cos(2.0 * a) - A3 * cos(3.0 * a); +} + +vec2 mul(vec2 a, vec2 b) { + return vec2(a.x * b.x - a.y * b.y, a.x * b.y + a.y * b.x); +} + +float windowed(int centre, uint i, bool long_window) { + if (long_window) return low_passed(centre + (int(i) - int(W / 2)) * int(DECIMATE)) * blackman_harris(i); + return sample_at(centre + int(i) - int(W / 2)) * blackman_harris(i); +} + +// Adds the power spectrum of the window centred on sample `centre`. +void analyse(int centre, bool long_window) { + uint t = gl_LocalInvocationID.x; + // Even and odd samples packed as one complex sequence, in bit-reversed + // order for the in-place transform. Past W it is all padding. + for (uint i = t; i < HALF; i += THREADS) { + vec2 v = vec2(0.0); + if (2u * i < W) v = vec2(windowed(centre, 2u * i, long_window), windowed(centre, 2u * i + 1u, long_window)); + z[bitfieldReverse(i) >> (32u - LOG2_HALF)] = v; + } + barrier(); + for (uint s = 1u; s <= LOG2_HALF; s++) { + uint half_size = 1u << (s - 1u); + for (uint j = t; j < HALF / 2u; j += THREADS) { + uint k = j & (half_size - 1u); + uint i0 = ((j >> (s - 1u)) << s) + k; + uint i1 = i0 + half_size; + float angle = -PI * float(k) / float(half_size); + vec2 b = mul(vec2(cos(angle), sin(angle)), z[i1]); + vec2 a = z[i0]; + z[i0] = a + b; + z[i1] = a - b; + } + barrier(); + } + // The real signal's bins from the packed transform: + // X[k] = E[k] + e^(-2 pi i k / N) O[k], with E and O the even and odd + // samples' spectra, recovered from Z[k] and conj(Z[HALF - k]). + uint count = long_window ? LONG_BINS : HALF + 1u; + for (uint k = t; k < count; k += THREADS) { + vec2 zk = z[k % HALF]; + vec2 zc = z[(HALF - k) % HALF]; + zc.y = -zc.y; + vec2 even = 0.5 * (zk + zc); + vec2 d = 0.5 * (zk - zc); + vec2 odd = vec2(d.y, -d.x); + float angle = -2.0 * PI * float(k) / float(N); + vec2 x = even + mul(vec2(cos(angle), sin(angle)), odd); + if (long_window) long_bins[k] += dot(x, x); + else short_bins[k] += dot(x, x); + } + barrier(); +} + +float bin_level(uint k, bool long_window) { + return long_window ? long_bins[min(k, LONG_BINS - 1u)] : short_bins[min(k, HALF)]; +} + +// The level of the row spanning bins `lo` to `hi` of one analysis. +float row_level(float lo, float hi, bool long_window) { + if (hi - lo < 1.0) { + float mid = sqrt(lo * hi); + uint i = uint(floor(mid)); + float f = mid - floor(mid); + float p0 = bin_level(max(i, 1u) - 1u, long_window); + float p1 = bin_level(i, long_window); + float p2 = bin_level(i + 1u, long_window); + float p3 = bin_level(i + 2u, long_window); + float v = p1 + 0.5 * f * (p2 - p0 + f * (2.0 * p0 - 5.0 * p1 + 4.0 * p2 - p3 + f * (3.0 * (p1 - p2) + p3 - p0))); + return clamp(v, SILENCE, max(max(p1, p2), max(p0, p3))); + } + float v = SILENCE; + for (uint i = uint(floor(lo)); i <= uint(ceil(hi)); i++) v = max(v, bin_level(i, long_window)); + return v; +} + +// Average power over `count` windows to dBFS: a full-scale sine at a bin's +// centre gives (W * A0 / 2) squared. +float to_db(float power, uint count) { + float ratio = power / float(count) / (0.25 * float(W) * A0 * float(W) * A0); + return ratio > 0.0 ? max(10.0 * log(ratio) / log(10.0), SILENCE) : SILENCE; +} + +void main() { + uint c = gl_WorkGroupID.x; + uint t = gl_LocalInvocationID.x; + for (uint k = t; k <= HALF; k += THREADS) short_bins[k] = 0.0; + for (uint k = t; k < LONG_BINS; k += THREADS) long_bins[k] = 0.0; + barrier(); + + float start = float(c) * u.spp; + uint short_windows = uint(clamp(ceil(u.spp / float(W / 2u)), 1.0, 4.0)); + for (uint w = 0u; w < short_windows; w++) { + analyse(int(float(u.first) + start + u.spp * (float(w) + 0.5) / float(short_windows)), false); + } + // Zoomed out, the windows spread across the column average already; + // zoomed in, WELCH windows a quarter window apart around its centre. + uint long_windows = uint(clamp(ceil(u.spp / float(W * DECIMATE / 2u)), 1.0, 4.0)); + if (long_windows == 1u) { + int centre = int(float(u.first) + start + 0.5 * u.spp); + for (uint w = 0u; w < WELCH; w++) analyse(centre + (int(w) - int(WELCH / 2u)) * int(W * DECIMATE / 4u), true); + long_windows = WELCH; + } else { + for (uint w = 0u; w < long_windows; w++) { + analyse(int(float(u.first) + start + u.spp * (float(w) + 0.5) / float(long_windows)), true); + } + } + + for (uint k = t; k <= HALF; k += THREADS) short_bins[k] = to_db(short_bins[k], short_windows); + for (uint k = t; k < LONG_BINS; k += THREADS) long_bins[k] = to_db(long_bins[k], long_windows); + barrier(); + + bool inside = float(u.first) + start < float(u.samples); + float short_hz = u.rate / float(N); + float long_hz = short_hz / float(DECIMATE); + float ratio = log(u.max_freq / u.min_freq); + for (uint r = t; r < u.rows; r += THREADS) { + float lo = u.min_freq * exp(ratio * float(r) / float(u.rows)); + float hi = u.min_freq * exp(ratio * float(r + 1u) / float(u.rows)); + float blend = smoothstep(BLEND_LO, BLEND_HI, sqrt(lo * hi)); + float v = 0.0; + if (blend < 1.0) v += (1.0 - blend) * row_level(lo / long_hz, hi / long_hz, true); + if (blend > 0.0) v += blend * row_level(lo / short_hz, hi / short_hz, false); + levels[r * MAX_COLUMNS + (u.slot + c) % u.ring] = inside ? v : NO_TAPE; + } +} diff --git a/examples/micscope/shaders/spectrogram.frag b/examples/micscope/shaders/spectrogram.frag new file mode 100644 index 0000000..9cb5e3d --- /dev/null +++ b/examples/micscope/shaders/spectrogram.frag @@ -0,0 +1,45 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrogram (spectrogram.comp): time across, as the waveform above +// it, log frequency up, each level through a heat gradient (inferno: +// black, violet, red, orange, pale yellow), with the playhead. +layout(location = 0) out vec4 out_color; + +#include "SpectrogramView.glsl" + +layout(set = 2, binding = 0) readonly buffer Levels { float levels[]; }; + +// Match layouts.zig and spectrogram.comp. +const uint MAX_COLUMNS = 2048; +const float NO_TAPE = 1000.0; + +// Matplotlib's inferno, as a polynomial fit (Matt Zucker's, CC0). +vec3 inferno(float t) { + const vec3 c0 = vec3(0.0002189403691192265, 0.001651004631001012, -0.01948089843709184); + const vec3 c1 = vec3(0.1065134194856116, 0.5639564367884091, 3.932712388889277); + const vec3 c2 = vec3(11.60249308247187, -3.972853965665698, -15.9423941062914); + const vec3 c3 = vec3(-41.70399613139459, 17.43639888205313, 44.35414519872813); + const vec3 c4 = vec3(77.162935699427, -33.40235894210092, -81.80730925738993); + const vec3 c5 = vec3(-71.31942824499214, 32.62606426397723, 73.20951985803202); + const vec3 c6 = vec3(25.13112622477341, -12.24266895238567, -23.07032500287172); + return clamp(c0 + t * (c1 + t * (c2 + t * (c3 + t * (c4 + t * (c5 + t * c6))))), 0.0, 1.0); +} + +void main() { + vec2 p = gl_FragCoord.xy; + uint row = uint(u.size.y - p.y); + uint x = uint(p.x); + float db = levels[row * MAX_COLUMNS + (x + u.scroll) % uint(u.size.x)]; + if (x + u.filled < uint(u.size.x)) db = NO_TAPE; + vec3 color = vec3(0.035, 0.03, 0.05); + if (db < NO_TAPE) { + float t = clamp((db - u.floor_db) / (u.ceil_db - u.floor_db), 0.0, 1.0); + // A touch of contrast: quiet stays dark, loud blooms. + color = inferno(pow(t, 1.3)); + } + if (u.playhead >= 0.0) { + float d = abs(p.x - u.playhead); + color = mix(color, vec3(1.0, 0.85, 0.3), clamp(1.5 - d, 0.0, 1.0) * 0.8); + } + out_color = vec4(color, 1.0); +} diff --git a/examples/micscope/shaders/spectrum.comp b/examples/micscope/shaders/spectrum.comp new file mode 100644 index 0000000..e576e30 --- /dev/null +++ b/examples/micscope/shaders/spectrum.comp @@ -0,0 +1,128 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrum at the playhead, entirely on the GPU, in one workgroup: +// +// 1. a Blackman-Harris window of the tape ending at `u.end`, zero-padded to +// N. Its sidelobes sit below -92 dB, so the finely sampled spectrum shows +// the signal rather than the window's ripples; +// 2. a radix-2 Stockham FFT, ping-ponging between the scratch halves; +// 3. levels in dBFS per bin; +// 4. one value per log-spaced column. Where columns are narrower than a bin +// (the low end), a Catmull-Rom spline through the bins' levels; where a +// column spans several bins (the high end), their loudest. The zero +// padding already interpolates between the window's own bins, so the +// curve is continuous at every zoom; +// 5. smoothing in time (fast attack, slow release) and falling peaks, kept +// in the column buffer between frames. +layout(local_size_x = 256) in; + +#include "SpectrumParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// Two halves of N complex values, which the workgroup's threads read from +// each other: coherent, for Vulkan; WGSL's storageBarrier covers it. +#ifdef SKETCHBOOK_WEB +layout(set = 1, binding = 0) buffer Scratch { vec2 scratch[]; }; +#else +layout(set = 1, binding = 0) coherent buffer Scratch { vec2 scratch[]; }; +#endif +// x: smoothed level, y: peak, both dBFS. +layout(set = 1, binding = 1) buffer Columns { vec2 columns[]; }; + +// Match layouts.zig. +const uint N = 8192; +const uint LOG2_N = 13; +const uint W = 4096; +const uint THREADS = 256; +const float PI = 3.14159265358979; +const float SILENCE = -160.0; +// Blackman-Harris, 4 terms. A0 is also its coherent gain. +const float A0 = 0.35875; +const float A1 = 0.48829; +const float A2 = 0.14128; +const float A3 = 0.01168; + +void sync() { + memoryBarrierBuffer(); + barrier(); +} + +void main() { + uint t = gl_LocalInvocationID.x; + + // 1. Window. + int first = int(u.end) - int(W); + for (uint i = t; i < N; i += THREADS) { + float x = 0.0; + int at = first + int(i); + if (i < W && at >= 0 && at < int(u.samples)) { + float a = 2.0 * PI * float(i) / float(W - 1); + x = tape[u.offset + uint(at)] * (A0 - A1 * cos(a) + A2 * cos(2.0 * a) - A3 * cos(3.0 * a)); + } + scratch[i] = vec2(x, 0.0); + } + sync(); + + // 2. FFT: after stage s, sub-transforms of 2p = 2^(s+1) points, in order. + for (uint s = 0; s < LOG2_N; s++) { + uint p = 1u << s; + uint from = (s & 1u) * N; + uint to = N - from; + for (uint i = t; i < N / 2; i += THREADS) { + uint k = i & (p - 1u); + vec2 a = scratch[from + i]; + vec2 b = scratch[from + i + N / 2]; + float angle = -PI * float(k) / float(p); + vec2 w = vec2(cos(angle), sin(angle)); + b = vec2(b.x * w.x - b.y * w.y, b.x * w.y + b.y * w.x); + uint j = (i << 1u) - k; + scratch[to + j] = a + b; + scratch[to + j + p] = a - b; + } + sync(); + } + uint result = (LOG2_N & 1u) * N; + + // 3. Levels into the other half. A full-scale sine peaks at W * A0 / 2: + // the window sums to W * A0, and a real sine splits between ±f. + uint levels = N - result; + for (uint i = t; i <= N / 2; i += THREADS) { + float amplitude = length(scratch[result + i]) * 2.0 / (float(W) * A0); + scratch[levels + i].x = amplitude > 0.0 ? max(20.0 * log(amplitude) / log(10.0), SILENCE) : SILENCE; + } + sync(); + + // 4 and 5. Columns. + float bin_hz = u.rate / float(N); + float ratio = log(u.max_freq / u.min_freq); + for (uint c = t; c < u.columns; c += THREADS) { + float lo = u.min_freq * exp(ratio * float(c) / float(u.columns)) / bin_hz; + float hi = u.min_freq * exp(ratio * float(c + 1u) / float(u.columns)) / bin_hz; + float mid = sqrt(lo * hi); + float v; + if (hi - lo < 1.0) { + uint i = uint(floor(mid)); + float f = mid - floor(mid); + float p0 = scratch[levels + min(max(i, 1u) - 1u, N / 2)].x; + float p1 = scratch[levels + min(i, N / 2)].x; + float p2 = scratch[levels + min(i + 1u, N / 2)].x; + float p3 = scratch[levels + min(i + 2u, N / 2)].x; + v = p1 + 0.5 * f * (p2 - p0 + f * (2.0 * p0 - 5.0 * p1 + 4.0 * p2 - p3 + f * (3.0 * (p1 - p2) + p3 - p0))); + v = clamp(v, SILENCE, max(max(p1, p2), max(p0, p3))); + } else { + v = SILENCE; + uint end = min(uint(ceil(hi)), N / 2); + for (uint i = uint(floor(lo)); i <= end; i++) v = max(v, scratch[levels + i].x); + } + vec2 old = columns[c]; + vec2 now; + if (u.reset != 0u) { + now = vec2(v, v); + } else { + float rate = v > old.x ? 40.0 : 6.0; + now.x = mix(old.x, v, 1.0 - exp(-rate * u.dt)); + now.y = max(now.x, old.y - 24.0 * u.dt); + } + columns[c] = now; + } +} diff --git a/examples/micscope/shaders/spectrum.frag b/examples/micscope/shaders/spectrum.frag new file mode 100644 index 0000000..e88b091 --- /dev/null +++ b/examples/micscope/shaders/spectrum.frag @@ -0,0 +1,51 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrum from its columns (spectrum.comp): one column per pixel on a +// log frequency axis. The curve is drawn by its distance along the normal, +// from the neighbours' slope, so steep parts stay as thick as flat ones. +layout(location = 0) out vec4 out_color; + +#include "SpectrumView.glsl" + +layout(set = 2, binding = 0) readonly buffer Columns { vec2 columns[]; }; + +float y_of(float db) { + return u.size.y * (1.0 - clamp((db - u.floor_db) / (u.ceil_db - u.floor_db), 0.0, 1.0)); +} + +float x_of(float hz) { + return u.size.x * log(hz / u.min_freq) / log(u.max_freq / u.min_freq); +} + +// How close `p` is to a grid line, 0..1. +float grid(vec2 p) { + const float lines[10] = float[](20.0, 50.0, 100.0, 200.0, 500.0, 1000.0, 2000.0, 5000.0, 10000.0, 20000.0); + float near = 1e9; + for (int i = 0; i < 10; i++) near = min(near, abs(p.x - x_of(lines[i]))); + float step = u.size.y * 12.0 / (u.ceil_db - u.floor_db); + float db = mod(p.y, step); + near = min(near, min(db, step - db)); + return clamp(1.0 - near, 0.0, 1.0); +} + +void main() { + vec2 p = gl_FragCoord.xy; + int n = int(u.size.x); + int x = int(p.x); + float y = y_of(columns[x].x); + float slope = (y_of(columns[min(x + 1, n - 1)].x) - y_of(columns[max(x - 1, 0)].x)) * 0.5; + float distance = abs(p.y - y) / sqrt(1.0 + slope * slope); + + vec3 color = vec3(0.055, 0.06, 0.07); + color = mix(color, vec3(0.16, 0.17, 0.2), grid(p)); + + // Fill under the curve, by frequency, fading downwards. + float along = p.x / u.size.x; + vec3 hue = mix(vec3(0.2, 0.75, 0.9), vec3(0.85, 0.35, 0.9), along); + if (p.y > y) color = mix(color, hue, 0.35 * (1.0 - (p.y - y) / max(u.size.y - y, 1.0)) + 0.08); + // Peaks, then the curve. + float peak = abs(p.y - y_of(columns[x].y)); + color = mix(color, vec3(0.9, 0.9, 0.95), clamp(1.0 - peak, 0.0, 1.0) * 0.45); + color = mix(color, mix(hue, vec3(1.0), 0.5), clamp(1.6 - distance, 0.0, 1.0)); + out_color = vec4(color, 1.0); +} diff --git a/examples/micscope/shaders/tuner.comp b/examples/micscope/shaders/tuner.comp new file mode 100644 index 0000000..1907f7f --- /dev/null +++ b/examples/micscope/shaders/tuner.comp @@ -0,0 +1,134 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The pitch of the newest (or the playhead's) sound, by YIN (de Cheveigné +// and Kawahara, 2002), in one workgroup. Picking the strongest FFT peak +// finds the loudest harmonic, which is often not the note: low strings, +// voices and brass put more energy in the 2nd or 3rd harmonic, and a +// missing fundamental has no peak at all. YIN looks for the period instead: +// +// 1. the difference function d(tau) = sum over the window of +// (x[j] - x[j + tau])^2, which dips to 0 at every multiple of the +// period, whatever the harmonics' balance; +// 2. normalised by its running mean, d'(tau) = d(tau) * tau / sum d(1..tau), +// which starts at 1 and removes the dip-at-zero bias, so the dips are +// comparable; +// 3. the first dip below an absolute threshold, then the bottom of it: the +// shortest period that explains the signal, which keeps it from +// dropping an octave onto a multiple of the period; +// 4. a parabola through that dip of d and its neighbours, for a period +// between samples: the semitone is 6 %, the cent 0.06 %, and whole samples +// alone would be 30 cents apart at 1 kHz. +layout(local_size_x = 256) in; + +#include "TunerParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// d(tau), from tuner_difference.comp, which does the bulk of the work. +layout(set = 0, binding = 1) readonly buffer Difference { float difference[]; }; +// freq, clarity, level, period: layouts.Pitch. +layout(set = 1, binding = 0) buffer Result { vec4 result; }; + +// Match layouts.zig. +const uint W = 2048; +const uint LAGS = 1700; +const uint THREADS = 256; + +// Each thread owns CHUNK consecutive lags. +const uint CHUNK = (LAGS + THREADS - 1u) / THREADS; +shared float d[THREADS * CHUNK]; +// d before normalising: the parabola goes through these, since the +// normalisation's factor tau tilts a dip by more than a cent at high pitch. +shared float raw[THREADS * CHUNK]; +shared float squares[THREADS]; +shared float sums[THREADS]; +shared uint first_dip; +shared float deepest[THREADS]; +shared uint deepest_at[THREADS]; + +void main() { + uint t = gl_LocalInvocationID.x; + uint tau_max = min(LAGS - 2u, uint(ceil(u.rate / u.min_freq))); + uint tau_min = max(2u, uint(floor(u.rate / u.max_freq))); + + // 1. The difference function (tuner_difference.comp), and the window's + // energy. + int first = int(u.end) - int(W); + float sq = 0.0; + for (uint j = t; j < W; j += THREADS) { + int at = first + int(j); + float v = at >= 0 && at < int(u.samples) ? tape[u.offset + uint(at)] : 0.0; + sq += v * v; + } + squares[t] = sq; + float own = 0.0; + for (uint i = 0u; i < CHUNK; i++) { + uint tau = t * CHUNK + i; + float v = tau >= 1u && tau <= tau_max + 1u ? difference[tau] : 0.0; + raw[tau] = v; + own += v; + } + sums[t] = own; + if (t == 0u) first_dip = 0xffffffffu; + barrier(); + + // 2. Normalised by the running sum: a scan of the threads' chunk sums + // (Hillis-Steele), then each chunk in order. + for (uint stride = 1u; stride < THREADS; stride <<= 1u) { + float add = t >= stride ? sums[t - stride] : 0.0; + barrier(); + sums[t] += add; + barrier(); + } + float running = t > 0u ? sums[t - 1u] : 0.0; + float low = 2.0; + uint low_at = 0u; + for (uint i = 0u; i < CHUNK; i++) { + uint tau = t * CHUNK + i; + running += raw[tau]; + float v = tau == 0u ? 1.0 : (running > 0.0 ? raw[tau] * float(tau) / running : 1.0); + d[tau] = v; + if (tau >= tau_min && tau <= tau_max) { + // 3. The first dip under the threshold, and the deepest. + if (v < u.threshold) atomicMin(first_dip, tau); + if (v < low) { + low = v; + low_at = tau; + } + } + } + deepest[t] = low; + deepest_at[t] = low_at; + barrier(); + for (uint stride = THREADS / 2u; stride > 0u; stride >>= 1u) { + if (t < stride && deepest[t + stride] < deepest[t]) { + deepest[t] = deepest[t + stride]; + deepest_at[t] = deepest_at[t + stride]; + } + barrier(); + } + if (t != 0u) return; + + float energy = 0.0; + for (uint i = 0u; i < THREADS; i++) energy += squares[i]; + float level = energy > 0.0 ? 10.0 * log(energy / float(W)) / log(10.0) : -160.0; + + // The first dip followed to its bottom; failing one, the deepest, which + // the CPU may still reject as unclear. + uint best = first_dip; + if (best == 0xffffffffu) { + best = max(deepest_at[0], tau_min); + } else { + while (best + 1u <= tau_max && d[best + 1u] < d[best]) best++; + } + + // 4. The parabola through the dip. + float a = raw[best - 1u]; + float b = raw[best]; + float c = raw[best + 1u]; + float bend = a - 2.0 * b + c; + float shift = bend > 0.0 ? clamp(0.5 * (a - c) / bend, -0.5, 0.5) : 0.0; + float period = float(best) + shift; + float depth = d[best]; + bool silent = energy <= 0.0; + result = vec4(silent ? 0.0 : u.rate / period, silent ? 0.0 : clamp(1.0 - depth, 0.0, 1.0), level, period); +} diff --git a/examples/micscope/shaders/tuner_difference.comp b/examples/micscope/shaders/tuner_difference.comp new file mode 100644 index 0000000..c7d1b6e --- /dev/null +++ b/examples/micscope/shaders/tuner_difference.comp @@ -0,0 +1,49 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// YIN's difference function for tuner.comp, spread over many workgroups: +// d(tau) = sum over the newest W samples of (x[j] - x[j - tau])^2. Each +// workgroup takes LAGS_PER_GROUP lags, with THREADS / LAGS_PER_GROUP +// threads summing every so-many term of each, then adding up. +layout(local_size_x = 256) in; + +#include "TunerParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +layout(set = 1, binding = 0) buffer Difference { float d[]; }; + +// Match layouts.zig and tuner.comp. +const uint W = 2048; +const uint LAGS = 1700; +const uint THREADS = 256; +const uint LAGS_PER_GROUP = 16; +const uint PER_LAG = THREADS / LAGS_PER_GROUP; + +shared float x[W + LAGS]; +shared float partial[THREADS]; + +void main() { + uint t = gl_LocalInvocationID.x; + int first = int(u.end) - int(W + LAGS); + for (uint i = t; i < W + LAGS; i += THREADS) { + int at = first + int(i); + x[i] = at >= 0 && at < int(u.samples) ? tape[u.offset + uint(at)] : 0.0; + } + barrier(); + + uint tau = gl_WorkGroupID.x * LAGS_PER_GROUP + t / PER_LAG; + uint lane = t % PER_LAG; + float sum = 0.0; + if (tau < LAGS) { + for (uint j = lane; j < W; j += PER_LAG) { + float diff = x[LAGS + j] - x[LAGS + j - tau]; + sum += diff * diff; + } + } + partial[t] = sum; + barrier(); + for (uint stride = PER_LAG / 2u; stride > 0u; stride >>= 1u) { + if (lane < stride) partial[t] += partial[t + stride]; + barrier(); + } + if (lane == 0u && tau < LAGS) d[tau] = partial[t]; +} diff --git a/examples/micscope/shaders/view.vert b/examples/micscope/shaders/view.vert new file mode 100644 index 0000000..d9461e9 --- /dev/null +++ b/examples/micscope/shaders/view.vert @@ -0,0 +1,7 @@ +#version 450 +// One triangle over the whole target, from the vertex index alone: the +// fragment shaders work in pixels (gl_FragCoord). +void main() { + vec2 p = vec2((gl_VertexIndex << 1) & 2, gl_VertexIndex & 2); + gl_Position = vec4(p * 2.0 - 1.0, 0.0, 1.0); +} diff --git a/examples/micscope/shaders/waveform.comp b/examples/micscope/shaders/waveform.comp new file mode 100644 index 0000000..d1d64e5 --- /dev/null +++ b/examples/micscope/shaders/waveform.comp @@ -0,0 +1,72 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// One workgroup per waveform column: the lowest and highest the signal goes +// in it, and its RMS. Each column also includes the linearly interpolated +// signal at its own edges, which it shares with its neighbours, so that +// zoomed in past one sample per column the columns still join into one +// continuous line. +layout(local_size_x = 64) in; + +#include "WaveParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// x: min, y: max, z: RMS, w: 1 where the column has tape under it. +layout(set = 1, binding = 0) buffer Columns { vec4 columns[]; }; + +const uint THREADS = 64; +shared float lows[THREADS]; +shared float highs[THREADS]; +shared float squares[THREADS]; + +float sample_at(int i) { + return i >= 0 && i < int(u.samples) ? tape[u.offset + uint(i)] : 0.0; +} + +// The signal at `t` samples after `u.first`, between samples linearly. +float signal(float t) { + float whole = floor(t); + int i = int(u.first) + int(whole); + return mix(sample_at(i), sample_at(i + 1), t - whole); +} + +void main() { + uint c = gl_WorkGroupID.x; + uint t = gl_LocalInvocationID.x; + float start = float(c) * u.spp; + float end = start + u.spp; + int first = int(ceil(start)); + int last = min(int(floor(end)), int(u.samples) - int(u.first) - 1); + + float lo = 1e9; + float hi = -1e9; + float sq = 0.0; + for (int i = first + int(t); i <= last; i += int(THREADS)) { + float v = tape[u.offset + u.first + uint(i)]; + lo = min(lo, v); + hi = max(hi, v); + sq += v * v; + } + if (t == 0u) { + float a = signal(start); + float b = signal(min(end, float(u.samples - u.first - 1u))); + lo = min(lo, min(a, b)); + hi = max(hi, max(a, b)); + } + lows[t] = lo; + highs[t] = hi; + squares[t] = sq; + barrier(); + for (uint stride = THREADS / 2u; stride > 0u; stride >>= 1u) { + if (t < stride) { + lows[t] = min(lows[t], lows[t + stride]); + highs[t] = max(highs[t], highs[t + stride]); + squares[t] += squares[t + stride]; + } + barrier(); + } + if (t == 0u) { + float count = float(max(last - first + 1, 1)); + float inside = float(u.first) + start < float(u.samples) ? 1.0 : 0.0; + columns[(u.slot + c) % u.ring] = vec4(lows[0], highs[0], sqrt(squares[0] / count), inside); + } +} diff --git a/examples/micscope/shaders/waveform.frag b/examples/micscope/shaders/waveform.frag new file mode 100644 index 0000000..37be7ce --- /dev/null +++ b/examples/micscope/shaders/waveform.frag @@ -0,0 +1,56 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The waveform from its columns (waveform.comp): one column per pixel, the +// band between its low and high filled, anti-aliased, at least a pixel +// thick; the RMS inside it brighter; played and unplayed in two colors, +// red while recording, green for the live microphone. +layout(location = 0) out vec4 out_color; + +#include "WaveView.glsl" + +layout(set = 2, binding = 0) readonly buffer Columns { vec4 columns[]; }; + +// Match layouts.Tint. +const uint TINT_RECORDING = 1u; +const uint TINT_LIVE = 2u; + +void main() { + vec2 p = gl_FragCoord.xy; + uint x = uint(p.x); + vec4 col = columns[(x + u.scroll) % uint(u.size.x)]; + if (x + u.filled < uint(u.size.x)) col.w = 0.0; + float mid = u.size.y * 0.5; + float scale = u.size.y * 0.46 * u.gain; + + vec3 color = vec3(0.055, 0.06, 0.07); + if (col.w > 0.0) color = vec3(0.075, 0.08, 0.095); + // Center line. + color = mix(color, vec3(0.2, 0.22, 0.26), clamp(1.0 - abs(p.y - mid), 0.0, 1.0) * 0.6); + + if (col.w > 0.0) { + float top = mid - col.y * scale - 0.5; + float bottom = mid - col.x * scale + 0.5; + float outside = max(top - p.y, p.y - bottom); + float cover = clamp(0.5 - outside, 0.0, 1.0); + float rms = col.z * scale; + float inner = clamp(rms - abs(p.y - mid) + 0.5, 0.0, 1.0); + bool played = u.playhead >= 0.0 && p.x < u.playhead; + vec3 band = played ? vec3(0.35, 0.8, 0.95) : vec3(0.45, 0.5, 0.58); + vec3 core = played ? vec3(0.7, 0.95, 1.0) : vec3(0.65, 0.7, 0.78); + if (u.tint == TINT_RECORDING) { + band = vec3(0.95, 0.35, 0.35); + core = vec3(1.0, 0.6, 0.55); + } else if (u.tint == TINT_LIVE) { + band = vec3(0.35, 0.75, 0.5); + core = vec3(0.65, 1.0, 0.75); + } + color = mix(color, mix(band, core, inner), cover); + } + + // Playhead. + if (u.playhead >= 0.0) { + float d = abs(p.x - u.playhead); + color = mix(color, vec3(1.0, 0.85, 0.3), clamp(1.5 - d, 0.0, 1.0)); + } + out_color = vec4(color, 1.0); +} diff --git a/examples/micscope/src/layouts.zig b/examples/micscope/src/layouts.zig new file mode 100644 index 0000000..cd5be65 --- /dev/null +++ b/examples/micscope/src/layouts.zig @@ -0,0 +1,154 @@ +//! What the shaders read, as Zig structs: the only definition. The shaders +//! include GLSL generated from these (see sketchbook's `glsl.zig`). No +//! imports, so the shader build can compile this alone. + +/// FFT length: the analysis window, zero-padded to twice its size so the +/// spectrum is sampled twice as finely as the window resolves. +pub const fft_size = 8192; +pub const window = 4096; +/// The most columns a view can have; the column buffers hold this many. +pub const max_columns = 2048; +/// The tallest spectrogram; its buffer holds this many rows of +/// `max_columns`. +pub const max_rows = 1024; +// All three analyses read samples `offset + i` of the tape buffer for +// `0 <= i < samples`: the takes share one buffer, and a take's analysis +// never reads its neighbours. +/// The spectrogram's window, zero-padded to `spectrogram_fft` points. +pub const spectrogram_window = 2048; +pub const spectrogram_fft = 4096; + +/// spectrum.comp: the window ending at `end`, and the columns to fill. +pub const SpectrumParams = extern struct { + offset: u32, + /// Sample index one past the window's last sample. + end: u32, + /// Samples on the tape. + samples: u32, + columns: u32, + /// Nonzero: forget the smoothed levels and peaks. + reset: u32, + rate: f32, + min_freq: f32, + max_freq: f32, + /// Seconds since the previous analysis, for smoothing. + dt: f32, + pad0: u32 = 0, + pad1: u32 = 0, + pad2: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// waveform.comp: column `c` covers samples `first + c * spp` up to the next +/// column's start, and is written to slot `(slot + c) % ring`: the live +/// view writes its newest columns into a ring, the others from slot 0. +pub const WaveParams = extern struct { + offset: u32, + first: u32, + samples: u32, + columns: u32, + spp: f32, + slot: u32, + ring: u32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +pub const Tint = enum(u32) { take, recording, live }; + +/// waveform.frag. Pixel column `x` shows slot `(x + scroll) % width`; only +/// the last `filled` columns have anything in them. +pub const WaveView = extern struct { + size: [2]f32, + /// Pixels; negative hides it. + playhead: f32, + gain: f32, + /// A `Tint`. + tint: u32, + scroll: u32, + filled: u32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; + +/// spectrum.frag. +pub const SpectrumView = extern struct { + size: [2]f32, + floor_db: f32, + ceil_db: f32, + min_freq: f32, + max_freq: f32, + pad: [2]f32 = .{ 0, 0 }, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; + +/// spectrogram.comp: the same columns as `WaveParams`, each analysed into +/// `rows` log-spaced frequency rows, row 0 the lowest. +/// Columns land in slots as `WaveParams` says. +pub const SpectrogramParams = extern struct { + offset: u32, + first: u32, + samples: u32, + columns: u32, + rows: u32, + spp: f32, + rate: f32, + min_freq: f32, + max_freq: f32, + slot: u32, + ring: u32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// spectrogram.frag; `scroll` and `filled` as in `WaveView`. +pub const SpectrogramView = extern struct { + size: [2]f32, + floor_db: f32, + ceil_db: f32, + /// Pixels; negative hides it. + playhead: f32, + scroll: u32, + filled: u32, + pad: f32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; + +/// tuner.comp: the YIN window is the `tuner_window + tuner_lags` samples +/// ending at `end`, read like the other analyses (`offset`, `samples`). +pub const tuner_window = 2048; +pub const tuner_lags = 1700; + +pub const TunerParams = extern struct { + offset: u32, + end: u32, + samples: u32, + rate: f32, + /// The pitch range searched. + min_freq: f32, + max_freq: f32, + /// YIN's absolute threshold on the normalised difference. + threshold: f32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// What tuner.comp writes (std430, read back by the CPU). +pub const Pitch = extern struct { + /// Hz; 0 when nothing periodic was found. + freq: f32, + /// 1 minus YIN's normalised difference at the period: 1 is perfectly + /// periodic. + clarity: f32, + /// RMS of the window, dBFS. + level: f32, + /// The period in samples, interpolated. + period: f32, +}; diff --git a/examples/micscope/src/main.zig b/examples/micscope/src/main.zig new file mode 100644 index 0000000..facc169 --- /dev/null +++ b/examples/micscope/src/main.zig @@ -0,0 +1,1498 @@ +//! micscope: records the microphone or opens sound files, plays them +//! forwards or backwards, and shows each as a waveform, its spectrogram on +//! the same time axis, and the spectrum at its playhead. The window is a +//! vertical strip: on top, what the microphone hears right now (or the take +//! being recorded); under it every take, newest first. The signal +//! processing (windowed FFTs, log-frequency interpolation, smoothing, +//! waveform reduction) and all the drawing run on the GPU; the CPU only +//! moves samples: microphone to tape, tape to the GPU once, tape to the +//! speakers. +//! +//! Memory by lifetime: State is the transport, the selection and the views, +//! changed by `update` from keys and the mouse, and replayable. Store holds +//! the takes, on disk: they are still there after a restart. Session is +//! this process's playback cursor, the microphone's recent past and GPU +//! objects. Transport changes, seeks and deletions are `sk.Request`s, which +//! `draw` carries out once each. +const std = @import("std"); +const sk = @import("sketch"); +const gpu = sk.abi.gpu; +const layouts = @import("layouts"); +const shaders = @import("shaders"); +const tuner = @import("tuner.zig"); + +pub const name = "micscope"; +pub const width = 1280; +pub const height = 900; +pub const commands = .{ "open", "takes" }; + +/// The longest take: ten minutes at the host's rate. +const max_seconds = 600; +const max_takes = 64; +/// The playback stream's key. +const stream = 1; +/// How far ahead playback keeps the stream filled. +const lead_seconds = 0.06; +const min_freq = 20.0; +const floor_db = -96.0; +/// The live view's span. +const live_seconds = 6; +/// The microphone's recent past, kept for the live view: a frame's new +/// columns and everything their analysis windows reach. +const live_history = 1 << 16; +/// How far spectrogram.comp reads past a column's centre: the long +/// analysis, 2048 points decimated 4×, three windows a quarter apart, plus +/// its low-pass. The live view analyses a column once that much has been +/// heard, so its columns are final; it runs this far behind. +const live_reach = 6160; + +pub const Transport = enum { stopped, recording, forward, backward }; + +/// The part of a take its timeline shows, as fractions of it. +pub const View = struct { + lo: f32 = 0, + hi: f32 = 1, +}; + +pub const State = struct { + transport: sk.Request(Transport) = .{}, + /// The direction space resumes in. + backward: bool = false, + /// To a fraction of the selected take. + seek: sk.Request(f32) = .{}, + scrubbing: bool = false, + dialog: sk.Request(void) = .{}, + /// Deletes the take at this index. + remove: sk.Request(u32) = .{}, + /// How many takes the Store holds. `update` keeps count as it records + /// and deletes; `draw` corrects it with the `takes` command if not. + takes: u32 = 0, + /// The take the transport and the keys work on, by index (oldest 0). + selected: u32 = 0, + views: [max_takes]View = @splat(.{}), + /// Maps the world (the panels, stacked) to the screen. + camera: sk.Camera = .{}, + /// The camera keeps the bottom, where now is, in view. + follow: bool = true, + panning: bool = false, + /// The pointer's height when panning last moved. + grab: f32 = 0, + gain: f32 = 1, + /// The tuner's reference pitch: A4, in Hz. + a4: f32 = 440, +}; + +pub const Take = struct { + /// Its samples: `Store.tape[start..][0..len]`. + start: usize = 0, + len: usize = 0, + /// Stays the same when earlier takes are deleted: GPU caches go by it. + id: u32 = 0, + /// Where the listener is, in samples into the take. + heard: f64 = 0, + /// A file's name, or empty for the microphone. + source: [64]u8 = @splat(0), + source_len: u32 = 0, +}; + +pub const Store = struct { + /// Every take's mono samples at `Input.sample_rate`, one after another, + /// oldest first. A recording grows the last one. + tape: std.ArrayList(f32) = .empty, + takes: std.ArrayList(Take) = .empty, + next_id: u32 = 1, +}; + +pub const Session = struct { + transport: sk.Handled = .{}, + seek: sk.Handled = .{}, + dialog: sk.Handled = .{}, + remove: sk.Handled = .{}, + /// The take playback follows, by id, and the next of its samples to + /// queue. + playing: u32 = 0, + cursor: usize = 0, + /// The take being recorded, by id. + recording: u32 = 0, + live: Live = .{}, + /// Pitch, by tuner.comp: of the microphone (live, or recording), and + /// of the selected take at its playhead. + tuners: [2]tuner.Tracker = @splat(.{}), + /// The take `tuners[1]` follows, by id. + tuned: u32 = 0, + gpu: Gpu = .{}, +}; + +/// Which tracker a readback belongs to: its index, as the download key. +const Tuned = enum(u32) { input, take }; + +/// The microphone's last `live_history` samples, and how far the live +/// view's columns have got. Column `k` covers samples `k * spp` up to the +/// next, counted from the first sample heard, and lands in slot +/// `k % columns`: a ring the views read from the oldest. +const Live = struct { + samples: [live_history]f32 = @splat(0), + /// Samples heard since the session started. + heard: u64 = 0, + /// The next column to analyse. + next: u64 = 0, + /// Columns analysed since the view was (re)made, up to `columns`. + filled: u32 = 0, + columns: u32 = 0, + spp: u32 = 0, + + fn push(live: *Live, heard: []const f32) void { + const h = &live.samples; + if (heard.len >= h.len) { + @memcpy(h, heard[heard.len - h.len ..]); + } else { + std.mem.copyForwards(f32, h[0 .. h.len - heard.len], h[heard.len..]); + @memcpy(h[h.len - heard.len ..], heard); + } + live.heard += heard.len; + } +}; + +/// GPU objects, made on first use. +const Gpu = struct { + ready: bool = false, + /// `Store.tape`, and how many of its samples are in it. + tape: sk.Buffer = .none, + capacity: usize = 0, + uploaded: usize = 0, + /// `Live.samples`. + live_tape: sk.Buffer = .none, + /// spectrum.comp's FFT, shared by every panel. + scratch: sk.Buffer = .none, + /// tuner.comp's results, a `layouts.Pitch` each, by `Tuned`, and the + /// difference functions it searches. + pitches: [2]sk.Buffer = @splat(.none), + differences: [2]sk.Buffer = @splat(.none), + live: Panel = .{}, + /// For the takes on screen, by take id; the least recently used is + /// reused. + panels: [16]Panel = @splat(.{}), +}; + +/// One panel's buffers and images, and the inputs they were last made from: +/// unchanged, they are not made again. +const Panel = struct { + id: u32 = 0, + used: u64 = 0, + wave_columns: sk.Buffer = .none, + spectrum_columns: sk.Buffer = .none, + levels: sk.Buffer = .none, + wave: sk.Image = .none, + wave_size: [2]u32 = .{ 0, 0 }, + spectrogram: sk.Image = .none, + spectrogram_size: [2]u32 = .{ 0, 0 }, + spectrum: sk.Image = .none, + spectrum_size: [2]u32 = .{ 0, 0 }, + wave_key: u64 = 0, + spectrogram_key: u64 = 0, + spectrum_key: u64 = 0, + wave_drawn: u64 = 0, + spectrogram_drawn: u64 = 0, + + fn forget(p: *Panel) void { + p.wave_key = 0; + p.spectrogram_key = 0; + p.spectrum_key = 0; + p.wave_drawn = 0; + p.spectrogram_drawn = 0; + } +}; + +// --------------------------------------------------------------------------- +// The world: panels stacked down in the order they were made, take 0 on top +// and the live microphone at the bottom, where now is. A take keeps its +// place as others are added. The camera maps the world to the screen: it +// follows the bottom unless moved, and zooms out to overview the strip. +// Panels are rendered at world resolution and scaled by the camera, so +// moving it never recomputes anything. + +const spectrogram_height = 132; +const min_zoom = 0.3; +/// The canvas follows the window, but no smaller than this. +const min_width = 320; +const min_height = 300; + +/// Sizes, from the canvas. Wide, a panel is a waveform over its spectrogram +/// with the spectrum beside them, and the buttons sit in the header. Narrow +/// (a phone), the spectrum goes under them, full width, and the buttons +/// along the bottom, where thumbs are, big enough to hit. +const Layout = struct { + w: f32, + h: f32, + narrow: bool, + /// World units are canvas pixels at zoom 1. + world_width: f32, + panel_height: f32, + /// Where the world shows, between the header and the footer or buttons. + view: sk.Rect, + + const header = 36; + const footer = 22; + const toolbar = 56; + const spectrum_width = 320; + + fn of(w: f32, h: f32) Layout { + const narrow = w < 760; + const margin: f32 = if (narrow) 8 else 16; + const bottom: f32 = if (narrow) toolbar else footer; + return .{ + .w = w, + .h = h, + .narrow = narrow, + .world_width = w - 2 * margin, + .panel_height = if (narrow) 412 else 236, + .view = .{ .x = 0, .y = header, .w = w, .h = h - header - bottom }, + }; + } + + /// The panel in world slot `k`. + fn box(l: Layout, k: u32) Box { + const y = @as(f32, @floatFromInt(k)) * l.panel_height; + const timeline_w = if (l.narrow) l.world_width else l.world_width - spectrum_width - 16; + const wave: sk.Rect = .{ .x = 0, .y = y + 18, .w = timeline_w, .h = 64 }; + const spectrogram: sk.Rect = .{ .x = 0, .y = wave.y + wave.h + 2, .w = timeline_w, .h = spectrogram_height }; + const timeline: sk.Rect = .{ .x = 0, .y = wave.y, .w = timeline_w, .h = spectrogram.y + spectrogram.h - wave.y }; + // Under the timeline and its time labels, or beside it; room for + // the frequency labels under it either way. + const levels: sk.Rect = if (l.narrow) + .{ .x = 0, .y = spectrogram.y + spectrogram.h + 22, .w = l.world_width, .h = 150 } + else + .{ .x = l.world_width - spectrum_width, .y = wave.y, .w = spectrum_width, .h = timeline.h - 14 }; + return .{ + .whole = .{ .x = 0, .y = y, .w = l.world_width, .h = l.panel_height }, + .wave = wave, + .spectrogram = spectrogram, + .timeline = timeline, + .spectrum = levels, + }; + } + + /// Button `i` of `buttons`: right to left in the header, or left to + /// right across the bottom. + fn button(l: Layout, i: usize) sk.Rect { + const n: f32 = @floatFromInt(buttons.len); + const at: f32 = @floatFromInt(i); + if (l.narrow) { + const w = (l.w - 8 - (n - 1) * 4) / n; + return .{ .x = 4 + (n - 1 - at) * (w + 4), .y = l.h - toolbar + 6, .w = w, .h = toolbar - 10 }; + } + return .{ .x = l.w - 16 - (at + 1) * (button_width + 6) + 6, .y = 5, .w = button_width, .h = 26 }; + } + + fn buttonAt(l: Layout, x: f32, y: f32) ?Action { + for (buttons, 0..) |a, i| if (l.button(i).contains(x, y)) return a; + return null; + } +}; + +const Box = struct { + whole: sk.Rect, + wave: sk.Rect, + spectrogram: sk.Rect, + /// Waveform and spectrogram: one time axis, where clicks seek and + /// ctrl+wheel (or a pinch) zooms. + timeline: sk.Rect, + spectrum: sk.Rect, + + fn toScreen(b: Box, cam: sk.Camera) Box { + return .{ + .whole = cam.toScreen(b.whole), + .wave = cam.toScreen(b.wave), + .spectrogram = cam.toScreen(b.spectrogram), + .timeline = cam.toScreen(b.timeline), + .spectrum = cam.toScreen(b.spectrum), + }; + } +}; + +const World = struct { + /// Slots `0..takes` are takes; slot `takes` is the live view, unless a + /// recording (the last take) stands in for it. + takes: u32, + live: bool, + + fn of(s: *const State) World { + return .{ .takes = s.takes, .live = s.transport.value != .recording }; + } + fn slots(w: World) u32 { + return w.takes + @intFromBool(w.live); + } + fn height(w: World, l: Layout) f32 { + return @as(f32, @floatFromInt(w.slots())) * l.panel_height; + } + /// The take at world point `p`. + fn takeAt(w: World, l: Layout, p: sk.shape.Point) ?u32 { + if (p.y < 0 or p.x < 0 or p.x > l.world_width) return null; + const k: u32 = @intFromFloat(@floor(p.y / l.panel_height)); + return if (k < w.takes) k else null; + } +}; + +/// The camera's vertical range: from the world's top at the viewport's top +/// to its bottom at the viewport's bottom. +fn cameraRange(world: World, scale: f32, l: Layout) [2]f32 { + const highest = l.view.y + 4; + const lowest = l.view.y + l.view.h - 4 - world.height(l) * scale; + return .{ @min(lowest, highest), highest }; +} + +/// The camera as drawn: centred across, inside the world, and on the +/// bottom while following. +fn cameraOf(s: *const State, l: Layout) sk.Camera { + var cam = s.camera; + cam.zoom = std.math.clamp(cam.zoom, min_zoom, 1); + cam.x = @round((l.w - l.world_width * cam.zoom) / 2); + const range = cameraRange(World.of(s), cam.zoom, l); + cam.y = @round(if (s.follow) range[0] else std.math.clamp(cam.y, range[0], range[1])); + return cam; +} + +/// Moves the camera, which follows again once it reaches the bottom. +fn moveCamera(s: *State, cam: sk.Camera, in: *const sk.Input) void { + const l = Layout.of(in.width, in.height); + s.camera = cam; + s.follow = false; + s.camera = cameraOf(s, l); + const range = cameraRange(World.of(s), s.camera.zoom, l); + s.follow = s.camera.y <= range[0] + 0.5; +} + +// --------------------------------------------------------------------------- +// Update: transport, selection, views and camera, from input alone. + +pub fn update(s: *State, in: *const sk.Input) void { + if (in.started) { + // Nothing resumes on its own after a restart; the takes are kept. + s.transport.value = .stopped; + s.scrubbing = false; + s.panning = false; + } + const recording = s.transport.value == .recording; + for (keys) |k| if (in.pressed(k[0])) act(s, k[1], in); + if (in.pressed(sk.Key.equals)) s.gain = @min(s.gain * 1.25, 64); + if (in.pressed(sk.Key.minus)) s.gain = @max(s.gain / 1.25, 0.25); + if (in.pressed(sk.Key.z) and s.selected < max_takes) s.views[s.selected] = .{}; + if (!recording and s.takes > 0) { + // Up the strip is older. + if (in.pressed(sk.Key.up)) select(s, s.selected -| 1, in); + if (in.pressed(sk.Key.down)) select(s, @min(s.selected + 1, s.takes - 1), in); + } + const l = Layout.of(in.width, in.height); + const page = l.view.h - l.panel_height / 2; + if (in.pressed(sk.Key.page_down)) panBy(s, -page, in); + if (in.pressed(sk.Key.page_up)) panBy(s, page, in); + + const view = l.view; + for (in.events) |e| { + const cam = cameraOf(s, l); + const world = World.of(s); + const p = cam.toWorld(sk.shape.Point{ .x = e.x, .y = e.y }); + switch (e.kind) { + .mouse_down => if (e.code == 1 and l.buttonAt(e.x, e.y) != null) { + act(s, l.buttonAt(e.x, e.y).?, in); + } else if (e.code == 2 or e.code == 3) { + // Middle or right drag pans. + s.panning = true; + s.grab = e.y; + } else if (e.code == 1 and !recording and view.contains(e.x, e.y)) { + const index = world.takeAt(l, p) orelse continue; + select(s, index, in); + const b = l.box(index); + if (b.timeline.contains(p.x, p.y)) { + s.scrubbing = true; + seekTo(s, fractionAt(s.views[index], b.timeline, p.x)); + } + }, + .mouse_move => if (s.panning) { + panBy(s, e.y - s.grab, in); + s.grab = e.y; + } else if (s.scrubbing and s.selected < world.takes) { + seekTo(s, fractionAt(s.views[s.selected], l.box(s.selected).timeline, p.x)); + }, + .mouse_up, .cancel => { + s.scrubbing = false; + s.panning = false; + }, + // Ctrl+wheel, or a pinch (which browsers and the web host send + // as ctrl+wheel), zooms what is under it: a take's time on its + // timeline, the view anywhere else. + .wheel => if (view.contains(e.x, e.y)) { + const ctrl = e.modifiers() & sk.Key.Mod.ctrl != 0; + const over = world.takeAt(l, p); + if (ctrl and over != null and l.box(over.?).timeline.contains(p.x, p.y)) { + const index = over.?; + zoom(&s.views[index], fractionAt(s.views[index], l.box(index).timeline, p.x), std.math.pow(f32, 0.8, e.wheel())); + } else if (ctrl) { + var moved = cam; + moved.zoomAt(sk.shape.Point{ .x = e.x, .y = e.y }, e.wheel(), .{ .min = min_zoom, .max = 1 }); + moveCamera(s, moved, in); + } else panBy(s, e.wheel() * 60, in); + }, + else => {}, + } + } +} + +/// What keys and the buttons on top do. +const Action = enum { record, play, forward, backward, start, end, open, delete, overview, a4_down, a4_up }; + +const keys = [_]struct { sk.Key, Action }{ + .{ .r, .record }, .{ .space, .play }, .{ .right, .forward }, .{ .left, .backward }, + .{ .home, .start }, .{ .end, .end }, .{ .o, .open }, .{ .delete, .delete }, + .{ .backspace, .delete }, .{ .@"0", .overview }, .{ .left_bracket, .a4_down }, .{ .right_bracket, .a4_up }, +}; + +/// Everything a touch screen cannot reach by key, as buttons (see +/// `Layout.button`). Labels change with what they would do. +const buttons = [_]Action{ .a4_up, .a4_down, .overview, .delete, .open, .backward, .play, .record }; +const button_width = 52; + +fn act(s: *State, a: Action, in: *const sk.Input) void { + const now = s.transport.value; + switch (a) { + .record => if (now == .recording) change(s, .stopped) else record(s), + .play => change(s, switch (now) { + .forward, .backward => .stopped, + .stopped, .recording => if (s.backward) .backward else .forward, + }), + .forward => change(s, .forward), + .backward => change(s, .backward), + .start => seekTo(s, 0), + .end => seekTo(s, 1), + .open => s.dialog.send({}), + .delete => if (now != .recording and s.takes > 0) remove(s, in), + .overview => { + // The whole strip, or back to full size. + const l = Layout.of(in.width, in.height); + var cam = cameraOf(s, l); + cam.zoom = if (cam.zoom < 1) 1 else l.view.h / @max(World.of(s).height(l), 1); + moveCamera(s, cam, in); + s.follow = true; + }, + .a4_down => s.a4 = @max(s.a4 - 1, 400), + .a4_up => s.a4 = @min(s.a4 + 1, 480), + } +} + +/// Starts a new take, unless the strip is full. It takes the live view's +/// place, which the camera follows. +fn record(s: *State) void { + if (s.takes >= max_takes) return; + s.selected = s.takes; + s.takes += 1; + s.views[s.selected] = .{}; + s.follow = true; + change(s, .recording); +} + +fn change(s: *State, to: Transport) void { + if (to == .forward or to == .backward) s.backward = to == .backward; + s.transport.send(to); +} + +fn seekTo(s: *State, fraction: f32) void { + if (s.transport.value == .recording) return; + s.seek.send(std.math.clamp(fraction, 0, 1)); +} + +/// Selects a take and brings it into view. +fn select(s: *State, index: u32, in: *const sk.Input) void { + s.selected = index; + const l = Layout.of(in.width, in.height); + var cam = cameraOf(s, l); + const b = cam.toScreen(l.box(index).whole); + const view = l.view; + if (b.y < view.y) cam.y += view.y - b.y + 4; + if (b.y + b.h > view.y + view.h) cam.y -= b.y + b.h - (view.y + view.h) + 4; + moveCamera(s, cam, in); +} + +/// Moves the world down by `pixels` on screen. +fn panBy(s: *State, pixels: f32, in: *const sk.Input) void { + var cam = cameraOf(s, Layout.of(in.width, in.height)); + cam.y += pixels; + moveCamera(s, cam, in); +} + +/// Deletes the selected take and selects the one above it. +fn remove(s: *State, in: *const sk.Input) void { + const index = @min(s.selected, s.takes - 1); + s.remove.send(index); + std.mem.copyForwards(View, s.views[index .. s.takes - 1], s.views[index + 1 .. s.takes]); + s.takes -= 1; + s.selected = index -| 1; + moveCamera(s, cameraOf(s, Layout.of(in.width, in.height)), in); +} + +fn fractionAt(v: View, timeline: sk.Rect, x: f32) f32 { + return v.lo + (x - timeline.x) / timeline.w * (v.hi - v.lo); +} + +/// Scales the view by `factor` around `at`, keeping it inside the take. +fn zoom(v: *View, at: f32, factor: f32) void { + const span = std.math.clamp((v.hi - v.lo) * factor, 1e-6, 1); + var lo = at - (at - v.lo) / (v.hi - v.lo) * span; + lo = std.math.clamp(lo, 0, 1 - span); + v.lo = lo; + v.hi = lo + span; +} + +/// `open PATH` adds a file as a take; `open` alone shows the dialog. +/// `takes N` tells State how many takes the Store holds (`draw` sends it). +pub fn command(s: *State, store: *Store, c: *sk.Console, line: []const u8) !bool { + var words = std.mem.tokenizeAny(u8, line, " \t"); + const verb = words.next() orelse return false; + if (std.mem.eql(u8, verb, "takes")) { + const n = try std.fmt.parseInt(u32, words.next() orelse return error.ExpectedCount, 10); + s.takes = @min(n, max_takes); + s.selected = @min(s.selected, s.takes -| 1); + return true; + } + if (!std.mem.eql(u8, verb, "open")) return false; + const path = std.mem.trim(u8, words.rest(), " \t"); + if (path.len == 0) { + s.dialog.send({}); + return true; + } + if (store.takes.items.len >= max_takes) return error.TooManyTakes; + const samples = c.host.samples(c.host.loadSound(path)); + if (samples.len == 0) return error.CannotOpen; + // Recording grows the last take: stop before this one goes after it. + if (s.transport.value == .recording) change(s, .stopped); + const kept = samples[0..@min(samples.len, max_seconds * c.in.sample_rate)]; + var take: Take = .{ .start = store.tape.items.len, .len = kept.len, .id = store.next_id }; + const base = std.fs.path.basename(path); + take.source_len = @intCast(@min(base.len, take.source.len)); + @memcpy(take.source[0..take.source_len], base[0..take.source_len]); + try store.takes.ensureUnusedCapacity(c.store, 1); + try store.tape.appendSlice(c.store, kept); + store.takes.appendAssumeCapacity(take); + store.next_id += 1; + s.takes = @intCast(store.takes.items.len); + s.selected = s.takes - 1; + s.views[s.selected] = .{}; + s.follow = true; + change(s, .stopped); + c.print("opened {s}: {d:.1} s", .{ path, @as(f64, @floatFromInt(kept.len)) / @as(f64, @floatFromInt(c.in.sample_rate)) }); + return true; +} + +// --------------------------------------------------------------------------- +// Draw: the takes, the microphone, the speakers and the GPU. + +pub fn draw(s: *const State, ss: *Session, store: *Store, f: *sk.Frame) void { + const in = f.in; + const rate: f64 = @floatFromInt(in.sample_rate); + const dt: f64 = @min(@as(f64, @floatFromInt(in.dt)) / std.time.ns_per_s, 0.1); + // As big as the window: the layout follows (see `Layout`). + f.canvas(@intFromFloat(@max(in.window_width, min_width)), @intFromFloat(@max(in.window_height, min_height))); + f.clear(0x0b0c0f); + + adoptTape(store, f.store); + if (ss.dialog.take(s.dialog) != null) f.openFile("open", .{ .pattern = "wav;flac;ogg" }); + if (ss.remove.take(s.remove)) |index| removeTake(ss, store, index); + + // Requests from State. Any of them, or another take selected, flushes + // what is queued, so the cursor restarts from what was heard. + const transport = ss.transport.take(s.transport); + const seek = ss.seek.take(s.seek); + // Recording goes by the transport's value rather than the request, so a + // fresh session that only caught up with the request still records. + if (s.transport.value != .recording) { + ss.recording = 0; + } else if (store.takes.items.len == 0 or store.takes.items[store.takes.items.len - 1].id != ss.recording) { + startTake(ss, store, f.store); + } + const takes = store.takes.items; + if (s.takes != takes.len) f.run(f.format("takes {d}", .{takes.len})); + const selected: ?*Take = if (takes.len > 0) &takes[@min(s.selected, takes.len - 1)] else null; + var queued: usize = in.stream(stream).queued; + if (selected) |t| { + if (transport != null or seek != null or t.id != ss.playing) { + f.audio(stream, &.{}, .{ .clear = true }); + queued = 0; + ss.playing = t.id; + ss.cursor = @intFromFloat(std.math.clamp(@round(t.heard), 0, @as(f64, @floatFromInt(t.len)))); + } + if (transport) |to| { + switch (to) { + .forward => if (ss.cursor >= t.len) { + ss.cursor = 0; + }, + .backward => if (ss.cursor == 0) { + ss.cursor = t.len; + }, + .stopped, .recording => {}, + } + t.heard = @floatFromInt(ss.cursor); + } + if (seek) |fraction| if (s.transport.value != .recording) { + ss.cursor = @intFromFloat(@round(fraction * @as(f32, @floatFromInt(t.len)))); + t.heard = @floatFromInt(ss.cursor); + }; + } + + // The microphone is always open: the live view shows it, and a + // recording keeps it. + f.microphone(); + ss.live.push(in.microphone); + switch (s.transport.value) { + .recording => if (takes.len > 0 and takes[takes.len - 1].id == ss.recording) { + const t = &takes[takes.len - 1]; + const room = max_seconds * in.sample_rate -| t.len; + const heard = in.microphone[0..@min(in.microphone.len, room)]; + store.tape.appendSlice(f.store, heard) catch f.host.log("the store is full: recording stopped"); + t.len = store.tape.items.len - t.start; + t.heard = @floatFromInt(t.len); + }, + .forward, .backward => if (selected) |t| play(ss, t, store.tape.items[t.start..][0..t.len], f, s.transport.value == .forward, queued, rate, dt), + .stopped => {}, + } + + // The tuner's readbacks, a frame or two old. + if (selected) |t| if (t.id != ss.tuned) { + ss.tuned = t.id; + ss.tuners[@intFromEnum(Tuned.take)] = .{}; + }; + for (in.downloads) |d| { + if (d.key >= ss.tuners.len or d.bytes.len < @sizeOf(layouts.Pitch)) continue; + ss.tuners[d.key].feed(std.mem.bytesToValue(layouts.Pitch, d.bytes[0..@sizeOf(layouts.Pitch)]), s.a4, @floatCast(dt)); + } + + if (!f.host.gpu()) { + f.text(16, Layout.header, 2, "micscope needs a GPU: the host has none.", 0xff8080); + return; + } + render(s, ss, store, f, rate, dt); +} + +/// Stores from before the strip held one tape: it becomes the first take. +fn adoptTape(store: *Store, a: std.mem.Allocator) void { + if (store.takes.items.len > 0 or store.tape.items.len == 0) return; + store.takes.append(a, .{ .len = store.tape.items.len, .id = store.next_id }) catch return; + store.next_id += 1; +} + +fn startTake(ss: *Session, store: *Store, a: std.mem.Allocator) void { + store.takes.append(a, .{ .start = store.tape.items.len, .id = store.next_id }) catch { + return; + }; + ss.recording = store.next_id; + store.next_id += 1; +} + +fn removeTake(ss: *Session, store: *Store, index: u32) void { + if (index >= store.takes.items.len) return; + const t = store.takes.orderedRemove(index); + const tape = store.tape.items; + std.mem.copyForwards(f32, tape[t.start .. tape.len - t.len], tape[t.start + t.len ..]); + store.tape.shrinkRetainingCapacity(tape.len - t.len); + for (store.takes.items[index..]) |*later| later.start -= t.len; + ss.gpu.uploaded = @min(ss.gpu.uploaded, t.start); + for (&ss.gpu.panels) |*p| if (p.id == t.id) { + p.id = 0; + p.forget(); + }; +} + +/// Keeps the stream `lead_seconds` ahead, reading the take either way, and +/// follows the listener. +fn play(ss: *Session, t: *Take, tape: []const f32, f: *sk.Frame, forward: bool, queued: usize, rate: f64, dt: f64) void { + const lead: usize = @intFromFloat(rate * lead_seconds); + const want = lead -| queued; + // What is heard now: the cursor, less what is queued in front of it. + const measured: f64 = @floatFromInt(if (forward) ss.cursor -| queued else @min(ss.cursor + queued, tape.len)); + if (forward) { + const n = @min(want, tape.len -| ss.cursor); + f.audio(stream, tape[@min(ss.cursor, tape.len)..][0..n], .{}); + ss.cursor += n; + } else { + const n = @min(want, ss.cursor); + const reversed = f.scratch.alloc(f32, n) catch return; + for (reversed, 0..) |*sample, i| sample.* = tape[ss.cursor - 1 - i]; + f.audio(stream, reversed, .{}); + ss.cursor -= n; + } + // The queue drains in device-sized blocks: advance by the clock and + // pull gently towards the measurement, snapping when far off. + const direction: f64 = if (forward) 1 else -1; + t.heard += direction * dt * rate; + const off = measured - t.heard; + t.heard = if (@abs(off) > rate * 0.25) measured else t.heard + off * 0.15; + t.heard = std.math.clamp(t.heard, 0, @as(f64, @floatFromInt(tape.len))); +} + +// --------------------------------------------------------------------------- +// GPU: buffers, compute, passes. + +const view_pipeline = struct { + fn make(fragment: []const u8) gpu.Pipeline { + return .{ + .vertex = .{ .code = shaders.view_vert }, + .fragment = .{ .code = fragment, .uniform_buffers = 1, .storage_buffers = 1 }, + .stride = 0, + .attributes = &.{}, + .blend = .none, + }; + } +}; +const wave_pipelines = [_]gpu.Pipeline{view_pipeline.make(shaders.waveform_frag)}; +const spectrum_pipelines = [_]gpu.Pipeline{view_pipeline.make(shaders.spectrum_frag)}; +const spectrogram_pipelines = [_]gpu.Pipeline{view_pipeline.make(shaders.spectrogram_frag)}; + +const spectrum_kernel: gpu.Kernel = .{ + .code = shaders.spectrum_comp, + .readonly_buffers = 1, + .readwrite_buffers = 2, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const spectrogram_kernel: gpu.Kernel = .{ + .code = shaders.spectrogram_comp, + .readonly_buffers = 1, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const tuner_difference_kernel: gpu.Kernel = .{ + .code = shaders.tuner_difference_comp, + .readonly_buffers = 1, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const tuner_kernel: gpu.Kernel = .{ + .code = shaders.tuner_comp, + .readonly_buffers = 2, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const waveform_kernel: gpu.Kernel = .{ + .code = shaders.waveform_comp, + .readonly_buffers = 1, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 64, 1, 1 }, +}; + +fn setup(g: *Gpu, f: *sk.Frame) void { + if (g.ready) return; + g.ready = true; + g.scratch = f.host.newBuffer(layouts.fft_size * 2 * @sizeOf([2]f32)); + g.live_tape = f.host.newBuffer(live_history * @sizeOf(f32)); + for (&g.pitches) |*b| b.* = f.host.newBuffer(@sizeOf(layouts.Pitch)); + for (&g.differences) |*b| b.* = f.host.newBuffer(layouts.tuner_lags * @sizeOf(f32)); +} + +fn makeBuffers(p: *Panel, f: *sk.Frame) void { + if (p.wave_columns != .none) return; + p.wave_columns = f.host.newBuffer(layouts.max_columns * @sizeOf([4]f32)); + p.spectrum_columns = f.host.newBuffer(layouts.max_columns * @sizeOf([2]f32)); + p.levels = f.host.newBuffer(layouts.max_columns * spectrogram_height * @sizeOf(f32)); +} + +/// The panel for take `id`: the one it had, or the least recently used. +fn panelFor(g: *Gpu, f: *sk.Frame, id: u32) *Panel { + const p = for (&g.panels) |*p| { + if (p.id == id) break p; + } else reuse: { + var oldest = &g.panels[0]; + for (&g.panels) |*p| if (p.used < oldest.used) { + oldest = p; + }; + oldest.id = id; + oldest.forget(); + break :reuse oldest; + }; + p.used = f.in.frame; + makeBuffers(p, f); + return p; +} + +/// An image of `size`, (re)made when the size changes. +fn target(f: *sk.Frame, image: *sk.Image, size: *[2]u32, r: sk.Rect) [2]u32 { + const want: [2]u32 = .{ @intFromFloat(@min(r.w, layouts.max_columns)), @intFromFloat(@min(r.h, spectrogram_height)) }; + if (image.* == .none or size[0] != want[0] or size[1] != want[1]) { + if (image.* != .none) f.host.freeImage(image.*); + image.* = f.host.newImage(want[0], want[1]); + size.* = want; + } + return want; +} + +/// Puts the tape on the GPU: new samples only, unless the buffer had to grow. +fn sync(g: *Gpu, f: *sk.Frame, tape: []const f32) void { + if (tape.len > g.capacity or g.tape == .none) { + if (g.tape != .none) f.host.freeBuffer(g.tape); + g.capacity = @max(tape.len, g.capacity * 2, 60 * f.in.sample_rate); + g.tape = f.host.newBuffer(@intCast(g.capacity * @sizeOf(f32))); + g.uploaded = 0; + } + g.uploaded = @min(g.uploaded, tape.len); + if (g.uploaded < tape.len) { + f.upload(g.tape, @intCast(g.uploaded * @sizeOf(f32)), std.mem.sliceAsBytes(tape[g.uploaded..])); + g.uploaded = tape.len; + } +} + +fn render(s: *const State, ss: *Session, store: *Store, f: *sk.Frame, rate: f64, dt: f64) void { + const g = &ss.gpu; + setup(g, f); + sync(g, f, store.tape.items); + // New code may mean new shaders: make everything again. + if (f.in.reloaded) { + g.live.forget(); + for (&g.panels) |*p| p.forget(); + } + const takes = store.takes.items; + const selected = if (takes.len > 0) @min(s.selected, takes.len - 1) else 0; + const recording = s.transport.value == .recording; + const world: World = .{ .takes = @intCast(takes.len), .live = !recording }; + const l = Layout.of(f.in.width, f.in.height); + const cam = cameraOf(s, l); + const view = l.view; + // Images are made at world size; the camera only scales them. + const size = l.box(0); + + // The live view keeps up even while it is hidden. + analyseLive(ss, f, size, rate); + // The tuner: on the microphone's newest sound, and on the selected take + // around its playhead. + const reach = layouts.tuner_window + layouts.tuner_lags; + if (recording and takes.len > 0) { + const t = takes[takes.len - 1]; + tune(g, f, .input, g.tape, t.start, t.len, t.len, rate); + } else tune(g, f, .input, g.live_tape, 0, live_history, live_history, rate); + if (!recording and takes.len > 0) { + const t = takes[selected]; + const end: usize = @intFromFloat(@max(t.heard, 0)); + tune(g, f, .take, g.tape, t.start, t.len, end + reach / 2, rate); + } + f.clip(view); + for (0..world.slots()) |slot| { + const k: u32 = @intCast(slot); + const b = l.box(k).toScreen(cam); + if (b.whole.y + b.whole.h <= view.y or b.whole.y >= view.y + view.h) continue; + if (k == world.takes) { + drawLive(s, ss, f, size, b, rate, dt); + } else { + const tint: layouts.Tint = if (recording and k + 1 == world.takes) .recording else .take; + drawTake(s, g, f, size, b, &takes[k], s.views[k], k == selected, tint, rate, dt); + } + } + f.clip(null); + labels(s, ss, store, f, world, cam, rate); +} + +/// Finds the pitch of the `tuner_window` samples before `end` (of the +/// `samples` from `offset`), and reads it back. +fn tune(g: *Gpu, f: *sk.Frame, which: Tuned, tape: sk.Buffer, offset: usize, samples: usize, end: usize, rate: f64) void { + const result = g.pitches[@intFromEnum(which)]; + const difference = g.differences[@intFromEnum(which)]; + const params = gpu.uniform(f.scratch, layouts.TunerParams{ + .offset = @intCast(offset), + .end = @intCast(end), + .samples = @intCast(samples), + .rate = @floatCast(rate), + .min_freq = tuner_low, + .max_freq = tuner_high, + .threshold = 0.12, + }); + f.compute(.{ + .label = "tuner_difference.comp", + .kernel = tuner_difference_kernel, + .readonly = &.{tape}, + .readwrite = &.{difference}, + .uniforms = params, + // Match tuner_difference.comp's LAGS_PER_GROUP. + .groups = .{ (layouts.tuner_lags + 15) / 16, 1, 1 }, + }); + f.compute(.{ + .label = "tuner.comp", + .kernel = tuner_kernel, + .readonly = &.{ tape, difference }, + .readwrite = &.{result}, + .uniforms = params, + }); + f.download(result, .{ .size = @sizeOf(layouts.Pitch), .key = @intFromEnum(which) }); +} + +/// The pitches the tuner looks for: a five-string bass's low B to a +/// piccolo's top. +const tuner_low = 30.0; +const tuner_high = 4200.0; + +fn wavePass(f: *sk.Frame, p: *Panel, size: [2]u32, view: layouts.WaveView) void { + f.gpu(.{ + .label = "waveform.frag", + .image = p.wave, + .width = size[0], + .height = size[1], + .clear = null, + .pipelines = &wave_pipelines, + .draws = &.{.{ .pipeline = 0, .vertex_count = 3, .fragment_uniforms = gpu.uniform(f.scratch, view), .fragment_storage = &.{p.wave_columns} }}, + }); +} + +fn spectrogramPass(f: *sk.Frame, p: *Panel, size: [2]u32, view: layouts.SpectrogramView) void { + f.gpu(.{ + .label = "spectrogram.frag", + .image = p.spectrogram, + .width = size[0], + .height = size[1], + .clear = null, + .pipelines = &spectrogram_pipelines, + .draws = &.{.{ .pipeline = 0, .vertex_count = 3, .fragment_uniforms = gpu.uniform(f.scratch, view), .fragment_storage = &.{p.levels} }}, + }); +} + +/// The spectrum's analysis of `tape`, then its pass. +fn spectrum(g: *Gpu, f: *sk.Frame, tape: sk.Buffer, p: *Panel, size: [2]u32, params: layouts.SpectrumParams, rate: f64) void { + f.compute(.{ + .label = "spectrum.comp", + .kernel = spectrum_kernel, + .readonly = &.{tape}, + .readwrite = &.{ g.scratch, p.spectrum_columns }, + .uniforms = gpu.uniform(f.scratch, params), + }); + f.gpu(.{ + .label = "spectrum.frag", + .image = p.spectrum, + .width = size[0], + .height = size[1], + .clear = null, + .pipelines = &spectrum_pipelines, + .draws = &.{.{ + .pipeline = 0, + .vertex_count = 3, + .fragment_uniforms = gpu.uniform(f.scratch, layouts.SpectrumView{ + .size = .{ @floatFromInt(size[0]), @floatFromInt(size[1]) }, + .floor_db = floor_db, + .ceil_db = 0, + .min_freq = min_freq, + .max_freq = maxFreq(rate), + }), + .fragment_storage = &.{p.spectrum_columns}, + }}, + }); +} + +fn maxFreq(rate: f64) f32 { + return @floatCast(@min(20_000, rate / 2)); +} + +fn drawTake(s: *const State, g: *Gpu, f: *sk.Frame, size: Box, b: Box, t: *const Take, view: View, selected: bool, tint: layouts.Tint, rate: f64, dt: f64) void { + const p = panelFor(g, f, t.id); + + // Waveform columns, when what they show changed. + const wave = target(f, &p.wave, &p.wave_size, size.wave); + const span = visible(view, t.len, wave[0]); + const first: u32 = @intFromFloat(@floor(span[0])); + const spp: f32 = @floatCast((span[1] - span[0]) / @as(f64, @floatFromInt(wave[0]))); + const params: layouts.WaveParams = .{ + .offset = @intCast(t.start), + .first = first, + .samples = @intCast(t.len), + .columns = wave[0], + .spp = spp, + .slot = 0, + .ring = layouts.max_columns, + }; + const key = std.hash.Wyhash.hash(0, std.mem.asBytes(¶ms)); + if (key != p.wave_key) { + p.wave_key = key; + f.compute(.{ + .label = "waveform.comp", + .kernel = waveform_kernel, + .readonly = &.{g.tape}, + .readwrite = &.{p.wave_columns}, + .uniforms = gpu.uniform(f.scratch, params), + .groups = .{ wave[0], 1, 1 }, + }); + } + + // The spectrogram over the same columns. + const spectrogram = target(f, &p.spectrogram, &p.spectrogram_size, size.spectrogram); + const analysis: layouts.SpectrogramParams = .{ + .offset = @intCast(t.start), + .first = first, + .samples = @intCast(t.len), + .columns = spectrogram[0], + .rows = spectrogram[1], + .spp = spp, + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .slot = 0, + .ring = layouts.max_columns, + }; + const analysis_key = std.hash.Wyhash.hash(0, std.mem.asBytes(&analysis)); + if (analysis_key != p.spectrogram_key) { + p.spectrogram_key = analysis_key; + f.compute(.{ + .label = "spectrogram.comp", + .kernel = spectrogram_kernel, + .readonly = &.{g.tape}, + .readwrite = &.{p.levels}, + .uniforms = gpu.uniform(f.scratch, analysis), + .groups = .{ spectrogram[0], 1, 1 }, + }); + } + + // The spectrum around the playhead, or of the latest while recording: + // every frame for the selected take, which plays and glides; the others + // only when it would change, and without smoothing. + const half: f64 = layouts.window / 2; + const len: f64 = @floatFromInt(t.len); + const end: f64 = if (tint == .recording) len else @min(t.heard + half, len + half); + const shown = target(f, &p.spectrum, &p.spectrum_size, size.spectrum); + var sparams: layouts.SpectrumParams = .{ + .offset = @intCast(t.start), + .end = @intFromFloat(@max(end, 0)), + .samples = @intCast(t.len), + .columns = shown[0], + .reset = 0, + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .dt = 0, + }; + const spectrum_key = std.hash.Wyhash.hash(1, std.mem.asBytes(&sparams)); + if (selected or spectrum_key != p.spectrum_key) { + sparams.reset = @intFromBool(!selected or p.spectrum_key == 0); + sparams.dt = @floatCast(dt); + p.spectrum_key = spectrum_key; + spectrum(g, f, g.tape, p, shown, sparams, rate); + } + + const at: f32 = @floatCast((t.heard - span[0]) / (span[1] - span[0]) * @as(f64, @floatFromInt(wave[0]))); + const playhead: f32 = if (!selected or t.len == 0 or at < 0 or at > @as(f32, @floatFromInt(wave[0]))) -1 else at; + const wave_view: layouts.WaveView = .{ + .size = .{ @floatFromInt(wave[0]), @floatFromInt(wave[1]) }, + .playhead = playhead, + .gain = s.gain, + .tint = @intFromEnum(tint), + .scroll = 0, + .filled = wave[0], + }; + const wave_drawn = std.hash.Wyhash.hash(p.wave_key, std.mem.asBytes(&wave_view)); + if (wave_drawn != p.wave_drawn) { + p.wave_drawn = wave_drawn; + wavePass(f, p, wave, wave_view); + } + const spectrogram_view: layouts.SpectrogramView = .{ + .size = .{ @floatFromInt(spectrogram[0]), @floatFromInt(spectrogram[1]) }, + .floor_db = floor_db, + .ceil_db = -6, + .playhead = playhead, + .scroll = 0, + .filled = spectrogram[0], + }; + const spectrogram_drawn = std.hash.Wyhash.hash(p.spectrogram_key, std.mem.asBytes(&spectrogram_view)); + if (spectrogram_drawn != p.spectrogram_drawn) { + p.spectrogram_drawn = spectrogram_drawn; + spectrogramPass(f, p, spectrogram, spectrogram_view); + } + f.image(p.wave, b.wave, .{}); + f.image(p.spectrogram, b.spectrogram, .{}); + f.image(p.spectrum, b.spectrum, .{}); +} + +/// Analyses the live view's new columns into its rings: only those whose +/// windows have been heard in full, so each is analysed once. +fn analyseLive(ss: *Session, f: *sk.Frame, b: Box, rate: f64) void { + const live = &ss.live; + const g = &ss.gpu; + const p = &g.live; + makeBuffers(p, f); + const columns: u32 = @intFromFloat(@min(b.wave.w, layouts.max_columns)); + const spp: u32 = @max(1, @as(u32, @intFromFloat(@round(live_seconds * rate / @as(f64, @floatFromInt(columns)))))); + if (columns != live.columns or spp != live.spp or f.in.reloaded) { + live.columns = columns; + live.spp = spp; + live.filled = 0; + live.next = 0; + } + if (f.in.microphone.len > 0) f.upload(g.live_tape, 0, std.mem.sliceAsBytes(&live.samples)); + + // History index of heard sample `i`: `i - base`. A column is ready once + // all it reads is heard, and analysable while all it reads is kept. + const base: i64 = @as(i64, @intCast(live.heard)) - live_history; + const ready = (live.heard -| live_reach) / spp; + const kept: u64 = @intCast(@max(0, @divFloor(base + live_reach + spp - 1, spp))); + live.next = @max(live.next, kept, ready -| columns); + if (ready <= live.next) return; + const count: u32 = @intCast(ready - live.next); + const first: u32 = @intCast(@as(i64, @intCast(live.next * spp)) - base); + const slot: u32 = @intCast(live.next % columns); + f.compute(.{ + .label = "waveform.comp live", + .kernel = waveform_kernel, + .readonly = &.{g.live_tape}, + .readwrite = &.{p.wave_columns}, + .uniforms = gpu.uniform(f.scratch, layouts.WaveParams{ + .offset = 0, + .first = first, + .samples = live_history, + .columns = count, + .spp = @floatFromInt(spp), + .slot = slot, + .ring = columns, + }), + .groups = .{ count, 1, 1 }, + }); + f.compute(.{ + .label = "spectrogram.comp live", + .kernel = spectrogram_kernel, + .readonly = &.{g.live_tape}, + .readwrite = &.{p.levels}, + .uniforms = gpu.uniform(f.scratch, layouts.SpectrogramParams{ + .offset = 0, + .first = first, + .samples = live_history, + .columns = count, + .rows = @intFromFloat(@min(b.spectrogram.h, spectrogram_height)), + .spp = @floatFromInt(spp), + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .slot = slot, + .ring = columns, + }), + .groups = .{ count, 1, 1 }, + }); + live.filled = @intCast(@min(columns, live.filled + count)); + live.next = ready; +} + +fn drawLive(s: *const State, ss: *Session, f: *sk.Frame, size: Box, b: Box, rate: f64, dt: f64) void { + const live = &ss.live; + const g = &ss.gpu; + const p = &g.live; + const wave = target(f, &p.wave, &p.wave_size, size.wave); + const spectrogram = target(f, &p.spectrogram, &p.spectrogram_size, size.spectrogram); + const scroll: u32 = @intCast(live.next % @max(live.columns, 1)); + const wave_view: layouts.WaveView = .{ + .size = .{ @floatFromInt(wave[0]), @floatFromInt(wave[1]) }, + .playhead = -1, + .gain = s.gain, + .tint = @intFromEnum(layouts.Tint.live), + .scroll = scroll, + .filled = live.filled, + }; + const wave_drawn = std.hash.Wyhash.hash(0, std.mem.asBytes(&wave_view)); + if (wave_drawn != p.wave_drawn) { + p.wave_drawn = wave_drawn; + wavePass(f, p, wave, wave_view); + } + const spectrogram_view: layouts.SpectrogramView = .{ + .size = .{ @floatFromInt(spectrogram[0]), @floatFromInt(spectrogram[1]) }, + .floor_db = floor_db, + .ceil_db = -6, + .playhead = -1, + .scroll = scroll, + .filled = live.filled, + }; + const spectrogram_drawn = std.hash.Wyhash.hash(0, std.mem.asBytes(&spectrogram_view)); + if (spectrogram_drawn != p.spectrogram_drawn) { + p.spectrogram_drawn = spectrogram_drawn; + spectrogramPass(f, p, spectrogram, spectrogram_view); + } + // The spectrum of the newest samples, smoothed. + const shown = target(f, &p.spectrum, &p.spectrum_size, size.spectrum); + spectrum(g, f, g.live_tape, p, shown, .{ + .offset = 0, + .end = live_history, + .samples = live_history, + .columns = shown[0], + .reset = @intFromBool(p.spectrum_key == 0), + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .dt = @floatCast(dt), + }, rate); + p.spectrum_key = 1; + f.image(p.wave, b.wave, .{}); + f.image(p.spectrogram, b.spectrogram, .{}); + f.image(p.spectrum, b.spectrum, .{}); +} + +/// The samples a take's timeline spans, at least 1/32 of a sample per +/// column. +fn visible(v: View, len: usize, columns: u32) [2]f64 { + const total: f64 = @floatFromInt(len); + var lo = v.lo * total; + var hi = v.hi * total; + const least = @as(f64, @floatFromInt(columns)) / 32; + if (hi - lo < least) { + const mid = (lo + hi) / 2; + lo = @max(mid - least / 2, 0); + hi = lo + least; + } + return .{ lo, hi }; +} + +// --------------------------------------------------------------------------- +// Text. + +const dim = 0x7a8090; +const bright = 0xe6e9ef; +const red = 0xff5a5a; +const green = 0x6fdc8c; +const yellow = 0xffd84d; + +fn labels(s: *const State, ss: *const Session, store: *const Store, f: *sk.Frame, world: World, cam: sk.Camera, rate: f64) void { + const takes = store.takes.items; + const recording = s.transport.value == .recording; + const selected = if (takes.len > 0) @min(s.selected, takes.len - 1) else 0; + + // What the transport is doing, to which take. + const status: struct { []const u8, u24 } = switch (s.transport.value) { + .recording => .{ "REC", red }, + .forward => .{ "PLAY >", 0x7fd7ff }, + .backward => .{ "PLAY <", 0x7fd7ff }, + .stopped => .{ "PAUSED", dim }, + }; + const l = Layout.of(f.in.width, f.in.height); + f.text(16, 10, 2, status[0], status[1]); + if (takes.len > 0) { + const t = takes[selected]; + const times = .{ Time{ .seconds = t.heard / rate }, Time{ .seconds = @as(f64, @floatFromInt(t.len)) / rate } }; + // As much as fits beside the buttons (or, narrow, the corner). + const room = if (l.narrow) l.w - 16 * 7 else l.button(buttons.len - 1).x - 16; + if (room >= 480) { + f.print(140, 10, 2, bright, "{f} / {f}", times); + if (room >= 740) f.print(500, 14, 1, dim, "take {d} of {d}: {s}", .{ selected + 1, takes.len, source(&t) }); + } else f.print(16 + 16 * 7, 14, 1, bright, "{f} / {f}", times); + } + for (buttons, 0..) |a, i| button(s, f, a, l.button(i)); + const corner = f.format("{s}{s}", .{ + if (s.gain != 1) f.format("gain x{d:.2} ", .{s.gain}) else "", + if (cam.zoom < 1) f.format("view x{d:.2}", .{cam.zoom}) else "", + }); + f.text(f.in.width - 16 - @as(f32, @floatFromInt(corner.len * 8)), if (l.narrow) 14 else f.in.height - 16, 1, corner, dim); + + const view = l.view; + f.clip(view); + for (0..world.slots()) |slot| { + const k: u32 = @intCast(slot); + const b = l.box(k).toScreen(cam); + if (b.whole.y + b.whole.h <= view.y or b.whole.y >= view.y + view.h) continue; + // Centred in the band above the waveform, whatever the zoom. + const title_y = @round(b.whole.y + (b.wave.y - b.whole.y - 8) / 2); + if (k == world.takes) { + f.print(b.whole.x, title_y, 1, green, "LIVE the microphone, the last {d} s", .{live_seconds}); + if (ss.live.heard == 0) f.text(b.wave.x + 8, b.wave.y + 8, 1, "nothing from the microphone yet", dim); + if (cam.zoom >= 0.8) axes(f, b, rate, null); + tunerOverlay(f, b.spectrum, ss.tuners[@intFromEnum(Tuned.input)].reading, s.a4, rate, cam.zoom >= 0.8); + continue; + } + const t = &takes[k]; + const seconds = Time{ .seconds = @as(f64, @floatFromInt(t.len)) / rate }; + if (recording and k + 1 == world.takes) { + f.print(b.whole.x, title_y, 1, red, "RECORDING take {d} {f}", .{ k + 1, seconds }); + } else { + const chosen = k == selected; + if (chosen) f.fill(.{ .x = b.whole.x - if (l.narrow) @as(f32, 6) else 12, .y = b.wave.y, .w = 3, .h = b.timeline.h }, yellow); + f.print(b.whole.x, title_y, 1, @as(u24, if (chosen) bright else dim), "#{d} {s} {f}", .{ k + 1, source(t), seconds }); + } + if (cam.zoom >= 0.8) { + const span = visible(s.views[k], t.len, @intFromFloat(@min(l.box(0).wave.w, layouts.max_columns))); + axes(f, b, rate, .{ span[0] / rate, span[1] / rate }); + } + const which: ?Tuned = if (recording) (if (k + 1 == world.takes) .input else null) else if (k == selected) .take else null; + if (which) |w| tunerOverlay(f, b.spectrum, ss.tuners[@intFromEnum(w)].reading, s.a4, rate, cam.zoom >= 0.8); + } + f.clip(null); + if (world.takes == 0) f.text(16, view.y + view.h - 12, 1, "No takes yet: REC records one, OPEN opens a file.", dim); + // Keys, where there is a keyboard to press them. + if (!l.narrow) f.text(16, f.in.height - 16, 1, "R SPACE LEFT RIGHT HOME END O DEL 0 [ ] UP/DOWN take wheel or 2 fingers pan CTRL+wheel or pinch zoom", dim); +} + +/// The tuner on a spectrum: the pitch and its harmonics where they fall on +/// it, the span of two semitones either side shaded, and that span +/// magnified across the top: the nearest notes of the chromatic scale, +/// the pitch as a needle between them, and the note in cents and Hz. +fn tunerOverlay(f: *sk.Frame, r: sk.Rect, reading: ?tuner.Reading, a4: f32, rate: f64, detailed: bool) void { + const at = reading orelse return; + const max_freq = maxFreq(rate); + const fade: u8 = @intFromFloat(255 * std.math.clamp(1 - at.quiet / tuner.hold_seconds, 0.2, 1)); + const cents = at.cents(); + const color = sk.Color.hex(if (@abs(cents) <= 5) green else if (@abs(cents) <= 15) yellow else 0xff8a4d).alpha(fade); + const x = struct { + fn of(rect: sk.Rect, hz: f32, ceiling: f32) f32 { + return rect.x + @log(hz / min_freq) / @log(ceiling / min_freq) * rect.w; + } + }.of; + + const freq = tuner.frequency(at.pitch, a4); + const lo = tuner.frequency(@as(f32, @floatFromInt(at.note)) - 2.5, a4); + const hi = tuner.frequency(@as(f32, @floatFromInt(at.note)) + 2.5, a4); + f.fill(.{ .x = x(r, lo, max_freq), .y = r.y, .w = @max(x(r, hi, max_freq) - x(r, lo, max_freq), 1), .h = r.h }, sk.Color.white.alpha(fade / 10)); + f.fill(.{ .x = @round(x(r, freq, max_freq)), .y = r.y, .w = 1, .h = r.h }, color); + // The harmonics the pitch accounts for. + for (2..9) |h| { + const hz = freq * @as(f32, @floatFromInt(h)); + if (hz > max_freq) break; + f.fill(.{ .x = @round(x(r, hz, max_freq)), .y = r.y + r.h - 10, .w = 1, .h = 10 }, color.alpha(fade / 2)); + } + if (!detailed) return; + + // The magnified scale: semitones `note - 2.5` to `note + 2.5`. + const strip: sk.Rect = .{ .x = r.x + 6, .y = r.y + 4, .w = r.w - 12, .h = 30 }; + const semitone = strip.w / 5; + const centre = strip.x + strip.w / 2; + f.fill(strip, sk.Color.hex(0x0b0c0f).alpha(220)); + var c: f32 = -250; + while (c <= 250) : (c += 10) { + const whole = @mod(c, 100) == 0; + const h: f32 = if (whole) 10 else if (@mod(c, 50) == 0) 6 else 3; + f.fill(.{ .x = @round(centre + c / 100 * semitone), .y = strip.y + strip.h - h, .w = 1, .h = h }, sk.Color.hex(if (c == 0) bright else dim).alpha(if (whole) 220 else 120)); + } + for (0..5) |i| { + const note = at.note - 2 + @as(i32, @intCast(i)); + const label = f.format("{s}{d}", .{ tuner.name(note), tuner.octave(note) }); + const lx = centre + (@as(f32, @floatFromInt(i)) - 2) * semitone - @as(f32, @floatFromInt(label.len * 4)); + f.text(lx, strip.y + 4, 1, label, if (note == at.note) color else sk.Color.hex(dim).alpha(fade)); + } + const needle = std.math.clamp(centre + cents / 100 * semitone, strip.x, strip.x + strip.w - 2); + f.fill(.{ .x = @round(needle) - 1, .y = strip.y + 14, .w = 3, .h = strip.h - 14 }, color); + + // The reading. + const big = f.format("{s}{d}", .{ tuner.name(at.note), tuner.octave(at.note) }); + const panel: sk.Rect = .{ .x = r.x + r.w - 6 - 124, .y = strip.y + strip.h + 4, .w = 124, .h = 50 }; + f.fill(panel, sk.Color.hex(0x0b0c0f).alpha(200)); + f.text(panel.x + 6, panel.y + 4, 3, big, color); + const whole = @round(cents); + f.print(panel.x + 6, panel.y + 30, 1, sk.Color.hex(bright).alpha(fade), "{s}{d:.0} cents", .{ if (whole > 0) "+" else "", if (whole == 0) 0 else whole }); + f.print(panel.x + 6, panel.y + 40, 1, sk.Color.hex(dim).alpha(fade), "{d:.2} Hz", .{freq}); + f.print(panel.x + 70, panel.y + 4, 1, sk.Color.hex(dim).alpha(fade), "A4 {d:.0}", .{a4}); +} + +fn button(s: *const State, f: *sk.Frame, a: Action, r: sk.Rect) void { + const now = s.transport.value; + const label: []const u8, const lit: ?u24 = switch (a) { + .record => if (now == .recording) .{ "STOP", red } else .{ "REC", null }, + .play => if (now == .forward or now == .backward) .{ "PAUSE", 0x7fd7ff } else .{ "PLAY", null }, + .backward => .{ "REV", if (now == .backward) 0x7fd7ff else null }, + .open => .{ "OPEN", null }, + .delete => .{ "DEL", null }, + .overview => .{ "ALL", null }, + .a4_down => .{ "A4-", null }, + .a4_up => .{ "A4+", null }, + .forward, .start, .end => .{ "", null }, + }; + f.fill(r, sk.Color.hex(if (lit) |c| c else 0x1c2029).alpha(if (lit != null) 70 else 255)); + f.fill(.{ .x = r.x, .y = r.y + r.h - 1, .w = r.w, .h = 1 }, sk.Color.hex(lit orelse 0x2e3440)); + f.text(r.x + (r.w - @as(f32, @floatFromInt(label.len * 8))) / 2, r.y + 9, 1, label, sk.Color.hex(lit orelse bright)); +} + +fn source(t: *const Take) []const u8 { + return if (t.source_len > 0) t.source[0..t.source_len] else "microphone"; +} + +/// A panel's scales: its time span (seconds; null for the live view, which +/// ends now) under the spectrogram, frequencies up the spectrogram, and +/// frequency and level on the spectrum. +fn axes(f: *sk.Frame, b: Box, rate: f64, span: ?[2]f64) void { + const below = b.spectrogram.y + b.spectrogram.h + 4; + const left = if (span) |sp| f.format("{f}", .{Time{ .seconds = sp[0] }}) else f.format("-{d} s", .{live_seconds}); + const right = if (span) |sp| f.format("{f}", .{Time{ .seconds = sp[1] }}) else "now"; + f.text(b.wave.x, below, 1, left, dim); + f.text(b.wave.x + b.wave.w - @as(f32, @floatFromInt(right.len * 8)), below, 1, right, dim); + + const max_freq = maxFreq(rate); + for ([_]f64{ 100, 1000, 10000 }) |hz| { + const y = b.spectrogram.y + b.spectrogram.h * (1 - @as(f32, @floatCast(@log(hz / min_freq) / @log(max_freq / min_freq)))); + f.fill(.{ .x = b.spectrogram.x, .y = y, .w = 6, .h = 1 }, sk.Color.white.alpha(140)); + f.text(b.spectrogram.x + 9, y - 4, 1, hzLabel(f, hz), sk.Color.white.alpha(170)); + } + + for ([_]f64{ 50, 200, 1000, 5000, 20000 }) |hz| { + if (hz > max_freq) continue; + const x = b.spectrum.x + @as(f32, @floatCast(@log(hz / min_freq) / @log(max_freq / min_freq))) * b.spectrum.w; + f.text(@min(x + 3, b.spectrum.x + b.spectrum.w - 24), b.spectrum.y + b.spectrum.h + 4, 1, hzLabel(f, hz), dim); + } + var db: f32 = -24; + while (db > floor_db) : (db -= 24) { + const y = b.spectrum.y + b.spectrum.h * (db / floor_db); + f.print(b.spectrum.x + 4, y + 3, 1, dim, "{d}", .{db}); + } +} + +fn hzLabel(f: *sk.Frame, hz: f64) []const u8 { + return if (hz >= 1000) f.format("{d}k", .{hz / 1000}) else f.format("{d}", .{hz}); +} + +const Time = struct { + seconds: f64, + + pub fn format(t: Time, w: *std.Io.Writer) std.Io.Writer.Error!void { + const ms: u64 = @intFromFloat(@max(t.seconds, 0) * 1000); + try w.print("{d}:{d:0>2}.{d:0>3}", .{ ms / 60_000, ms / 1000 % 60, ms % 1000 }); + } +}; + +test "zoom keeps the view inside the take and around the pointer" { + var v: View = .{}; + zoom(&v, 0.5, 0.5); + try std.testing.expectApproxEqAbs(@as(f32, 0.25), v.lo, 1e-6); + try std.testing.expectApproxEqAbs(@as(f32, 0.75), v.hi, 1e-6); + zoom(&v, 0.0, 4); + try std.testing.expectEqual(@as(f32, 0), v.lo); + try std.testing.expectEqual(@as(f32, 1), v.hi); +} + +test "the world stacks takes in order with the live view last" { + const world: World = .{ .takes = 3, .live = true }; + try std.testing.expectEqual(@as(u32, 4), world.slots()); + const l = Layout.of(width, height); + try std.testing.expectEqual(@as(?u32, 0), world.takeAt(l, .{ .x = 10, .y = 1 })); + try std.testing.expectEqual(@as(?u32, 2), world.takeAt(l, .{ .x = 10, .y = 2 * l.panel_height + 1 })); + try std.testing.expectEqual(@as(?u32, null), world.takeAt(l, .{ .x = 10, .y = 3 * l.panel_height + 1 })); +} + +test "the camera follows the bottom and stays inside the world, wide or narrow" { + for ([_]Layout{ .of(width, height), .of(390, 844) }) |l| { + var s: State = .{ .takes = 5 }; + const cam = cameraOf(&s, l); + // Following: the live view's bottom sits at the viewport's bottom. + try std.testing.expectApproxEqAbs(l.view.y + l.view.h - 4, cam.toScreen(l.box(5).whole).y + l.panel_height, 1); + s.follow = false; + s.camera.y = 1e6; + try std.testing.expectApproxEqAbs(l.view.y + 4, cameraOf(&s, l).toScreen(l.box(0).whole).y, 1); + } +} + +test "a narrow layout stacks the spectrum under the timeline, inside the panel" { + const l: Layout = .of(390, 844); + try std.testing.expect(l.narrow); + const b = l.box(0); + try std.testing.expect(b.spectrum.y > b.timeline.y + b.timeline.h); + try std.testing.expect(b.spectrum.y + b.spectrum.h + 16 <= b.whole.y + b.whole.h); + try std.testing.expectEqual(l.world_width, b.timeline.w); + // Buttons fill the bottom, inside the canvas, without overlapping. + for (0..buttons.len) |i| { + const r = l.button(i); + try std.testing.expect(r.x >= 0 and r.x + r.w <= l.w and r.y + r.h <= l.h); + try std.testing.expect(r.w >= 40); + } +} + +test { + _ = tuner; +} + +test "the live history keeps the newest samples in order" { + var live: Live = .{}; + live.push(&.{ 1, 2, 3 }); + live.push(&.{4}); + try std.testing.expectEqualSlices(f32, &.{ 1, 2, 3, 4 }, live.samples[live_history - 4 ..]); + try std.testing.expectEqual(@as(u64, 4), live.heard); +} diff --git a/examples/micscope/src/tuner.zig b/examples/micscope/src/tuner.zig new file mode 100644 index 0000000..92a8482 --- /dev/null +++ b/examples/micscope/src/tuner.zig @@ -0,0 +1,147 @@ +//! From pitch estimates (tuner.comp's YIN, one per frame) to a steady note on +//! the 12-tone equal-tempered chromatic scale, for any instrument: notes are +//! MIDI numbers (69 is A4, at `a4` Hz), so the range is whatever the +//! detector finds, from a bass's low B to a piccolo's top. +//! +//! Raw estimates jitter and, at a note's attack or release, misfire. So: +//! unclear or quiet estimates are ignored; the median of the last few +//! clear ones is followed, smoothed; a jump of over a semitone restarts the +//! median, so a new note shows at once; and the note shown only changes +//! once the pitch is 60 cents from it, so a note held a quarter tone sharp +//! does not flicker between two names. +const std = @import("std"); +const layouts = @import("layouts"); + +pub const names = [12][]const u8{ "C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B" }; + +/// Estimates clearer than this are a note (YIN's threshold is 0.12, so +/// clarity 0.88 at least when its first dip was found). +pub const min_clarity = 0.85; +/// Quieter than this is silence, dBFS. +pub const min_level = -60.0; +/// How long the last note stays (shown as fading) after the sound stops. +pub const hold_seconds = 1.0; + +/// Fractional MIDI number of `freq`. +pub fn midi(freq: f32, a4: f32) f32 { + return 69 + 12 * std.math.log2(freq / a4); +} + +pub fn frequency(m: f32, a4: f32) f32 { + return a4 * std.math.pow(f32, 2, (m - 69) / 12); +} + +pub fn name(note: i32) []const u8 { + return names[@intCast(@mod(note, 12))]; +} + +pub fn octave(note: i32) i32 { + return @divFloor(note, 12) - 1; +} + +pub const Reading = struct { + /// The note it is nearest, as a MIDI number. + note: i32, + /// The pitch, as a fractional MIDI number. + pitch: f32, + /// Seconds since a clear estimate: 0 while the note sounds. + quiet: f32, + + pub fn cents(r: Reading) f32 { + return (r.pitch - @as(f32, @floatFromInt(r.note))) * 100; + } +}; + +pub const Tracker = struct { + recent: [5]f32 = @splat(0), + count: u32 = 0, + reading: ?Reading = null, + + pub fn feed(t: *Tracker, p: layouts.Pitch, a4: f32, dt: f32) void { + const clear = p.freq > 0 and p.clarity >= min_clarity and p.level >= min_level; + if (!clear) { + t.count = 0; + if (t.reading) |*r| { + r.quiet += dt; + if (r.quiet > hold_seconds) t.reading = null; + } + return; + } + const m = midi(p.freq, a4); + if (t.count > 0 and @abs(m - t.median()) > 1) t.count = 0; + if (t.count == t.recent.len) { + std.mem.copyForwards(f32, t.recent[0 .. t.recent.len - 1], t.recent[1..]); + t.count -= 1; + } + t.recent[t.count] = m; + t.count += 1; + const target = t.median(); + + const fresh = t.count == 1 or t.reading == null or t.reading.?.quiet > 0; + var r: Reading = if (fresh) .{ .note = @intFromFloat(@round(target)), .pitch = target, .quiet = 0 } else t.reading.?; + r.pitch += (target - r.pitch) * 0.4; + r.quiet = 0; + if (@abs(r.pitch - @as(f32, @floatFromInt(r.note))) > 0.6) r.note = @intFromFloat(@round(r.pitch)); + t.reading = r; + } + + fn median(t: *const Tracker) f32 { + var sorted = t.recent; + std.mem.sort(f32, sorted[0..t.count], {}, std.sort.asc(f32)); + return sorted[t.count / 2]; + } +}; + +fn pitchOf(freq: f32) layouts.Pitch { + return .{ .freq = freq, .clarity = 0.97, .level = -20, .period = 48_000 / freq }; +} + +test "notes and octaves on the chromatic scale" { + try std.testing.expectApproxEqAbs(@as(f32, 69), midi(440, 440), 1e-4); + try std.testing.expectApproxEqAbs(@as(f32, 40), midi(82.4069, 440), 1e-3); + try std.testing.expectEqualStrings("E", name(40)); + try std.testing.expectEqual(@as(i32, 2), octave(40)); + try std.testing.expectEqualStrings("B", name(23)); + try std.testing.expectEqual(@as(i32, 0), octave(23)); + try std.testing.expectApproxEqAbs(@as(f32, 261.626), frequency(60, 440), 1e-2); + // Another reference pitch moves every note with it. + try std.testing.expectApproxEqAbs(@as(f32, 69), midi(442, 442), 1e-4); +} + +test "a held note settles, in cents from it" { + var t: Tracker = .{}; + // A4, 12 cents sharp. + const sharp = frequency(69.12, 440); + for (0..20) |_| t.feed(pitchOf(sharp), 440, 1.0 / 60.0); + const r = t.reading.?; + try std.testing.expectEqual(@as(i32, 69), r.note); + try std.testing.expectApproxEqAbs(@as(f32, 12), r.cents(), 0.5); +} + +test "one wild estimate does not move the note" { + var t: Tracker = .{}; + for (0..10) |_| t.feed(pitchOf(110), 440, 1.0 / 60.0); + t.feed(pitchOf(220), 440, 1.0 / 60.0); + t.feed(pitchOf(110), 440, 1.0 / 60.0); + try std.testing.expectEqual(@as(i32, 45), t.reading.?.note); +} + +test "a new note shows at once, a quarter tone does not flicker" { + var t: Tracker = .{}; + for (0..10) |_| t.feed(pitchOf(110), 440, 1.0 / 60.0); + for (0..3) |_| t.feed(pitchOf(146.83), 440, 1.0 / 60.0); + try std.testing.expectEqual(@as(i32, 50), t.reading.?.note); + // 55 cents sharp of D3 stays D3. + for (0..30) |_| t.feed(pitchOf(frequency(50.55, 440)), 440, 1.0 / 60.0); + try std.testing.expectEqual(@as(i32, 50), t.reading.?.note); +} + +test "silence holds the note a while, then lets go" { + var t: Tracker = .{}; + for (0..10) |_| t.feed(pitchOf(330), 440, 1.0 / 60.0); + const quiet: layouts.Pitch = .{ .freq = 0, .clarity = 0, .level = -120, .period = 0 }; + for (0..30) |_| t.feed(quiet, 440, 1.0 / 60.0); + try std.testing.expect(t.reading != null); + for (0..60) |_| t.feed(quiet, 440, 1.0 / 60.0); + try std.testing.expect(t.reading == null); +} diff --git a/examples/micscope/test/README.md b/examples/micscope/test/README.md new file mode 100644 index 0000000..6f4b833 --- /dev/null +++ b/examples/micscope/test/README.md @@ -0,0 +1,22 @@ +# Test material + +- `sweep.wav`: a 2 s log sweep (40 Hz to 16 kHz) over a quiet 440 Hz tone, + 48 kHz mono, generated for these tests. +- `tones.wav`: five 1 s notes for the tuner, generated for these tests + (harmonics with a little noise, 48 kHz mono): E1 with its 2nd harmonic + loudest, E2 with a weak fundamental, A4 12 cents sharp, D4 with no + fundamental at all, and C7. +- `reference.flac`: the rendering reference, all on one screen. 15 s of a + loudly mastered electronic track (sub-bass kicks, hats, a full mix), then + 15 s of a Berlin street (traffic, a close vehicle pass, broadband noise), + then `sweep.wav`; 48 kHz stereo, joined with short crossfades. + + - Square Punch, "Thundercat" (Plasma Pumper EP, 2011), from 3:36: + [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Square_Punch_-_Thundercat.flac), + CC BY-SA 3.0. + - LG, "traffic karl-liebknechtstrasse 111223", from 1:18: + [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:141977_lg_traffic-karl-liebknechtstrasse-111223.wav), + from [Freesound](http://freesound.org/people/LG/sounds/141977/), + CC BY-SA 3.0. + + The excerpt is licensed CC BY-SA 3.0 like its sources. diff --git a/examples/micscope/test/profile.script b/examples/micscope/test/profile.script new file mode 100644 index 0000000..aa56b3e --- /dev/null +++ b/examples/micscope/test/profile.script @@ -0,0 +1,167 @@ +# Zooms in and out one ctrl+wheel step per frame on the reference take, so +# waveform and spectrogram are recomputed every frame: run by +# `zig build profile` with --profile-gpu, which prints GPU time per command. +console open test/reference.flac +frames 3 +key right +hold left_ctrl +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 diff --git a/examples/micscope/test/record.script b/examples/micscope/test/record.script new file mode 100644 index 0000000..2750a18 --- /dev/null +++ b/examples/micscope/test/record.script @@ -0,0 +1,23 @@ +# Records (from whatever microphone the host gives a headless run) into a +# new take, stops, and plays it back. +expect takes 0 +key r +expect transport.value recording +expect takes 1 +frames 60 +screenshot .zig-cache/micscope/recording.png +key r +expect transport.value stopped +key space +expect transport.value forward +frames 10 +# Each recording is another take; opening a file while recording ends it. +key r +expect takes 2 +expect selected 1 +console open test/sweep.wav +expect transport.value stopped +expect takes 3 +frames 2 +expect takes 3 +screenshot .zig-cache/micscope/strip.png diff --git a/examples/micscope/test/reference.flac b/examples/micscope/test/reference.flac Binary files differnew file mode 100644 index 0000000..7ad1c63 --- /dev/null +++ b/examples/micscope/test/reference.flac diff --git a/examples/micscope/test/render.script b/examples/micscope/test/render.script new file mode 100644 index 0000000..71b50c7 --- /dev/null +++ b/examples/micscope/test/render.script @@ -0,0 +1,18 @@ +# The rendering reference (see README.md): song, street and sweep in one +# take. Screenshots land in .zig-cache/micscope/ for review. +console open test/reference.flac +expect transport.value stopped +frames 3 +screenshot .zig-cache/micscope/reference.png +# Playing: the playhead across waveform and spectrogram, the spectrum. +key right +frames 90 +expect-audio 1 +screenshot .zig-cache/micscope/reference-playing.png +# Zoomed into the street: the low end's hum lines and the noise floor. +key space +hold left_ctrl +wheel 660 150 4 +release left_ctrl +frames 3 +screenshot .zig-cache/micscope/reference-street.png diff --git a/examples/micscope/test/sweep.wav b/examples/micscope/test/sweep.wav Binary files differnew file mode 100644 index 0000000..1f131e3 --- /dev/null +++ b/examples/micscope/test/sweep.wav diff --git a/examples/micscope/test/tones.wav b/examples/micscope/test/tones.wav Binary files differnew file mode 100644 index 0000000..b2cf3ea --- /dev/null +++ b/examples/micscope/test/tones.wav diff --git a/examples/micscope/test/transport.script b/examples/micscope/test/transport.script new file mode 100644 index 0000000..f821a03 --- /dev/null +++ b/examples/micscope/test/transport.script @@ -0,0 +1,90 @@ +# Opens a file as a take and plays it both ways: a 2 s log sweep (40 Hz to +# 16 kHz) over a quiet 440 Hz tone. +console open test/sweep.wav +frames 2 +expect takes 1 +expect selected 0 +expect transport.value stopped +key right +expect transport.value forward +frames 30 +expect-audio 1 +screenshot .zig-cache/micscope/forward.png +key left +expect transport.value backward +expect backward true +frames 20 +key space +expect transport.value stopped +# Space resumes the way it last went. +key space +expect transport.value backward +key space +# Seeking: keys, and a click on the take's timeline. +key end +expect seek.value 1 +click 400 150 +expect seek.serial 2 +# Ctrl+wheel (or a pinch) on a take's timeline zooms its time around the +# pointer; Z zooms back out. +hold left_ctrl +wheel 400 150 3 +release left_ctrl +key z +key right +frames 30 +screenshot .zig-cache/micscope/played.png +# A second take goes under the first and is selected; up is older. +console open test/sweep.wav +expect takes 2 +expect selected 1 +key up +expect selected 0 +key down +expect selected 1 +# A click on the first take selects it. +click 400 150 +expect selected 0 +frames 2 +screenshot .zig-cache/micscope/two-takes.png +# Deleting it leaves the second. +key delete +expect takes 1 +expect selected 0 +# State that miscounts the takes is told by draw. +console set takes 0 +frames 2 +expect takes 1 +# More takes than fit: the camera follows the bottom until the wheel pans +# it, then 0 shows the whole strip. +console open test/sweep.wav +console open test/sweep.wav +console open test/sweep.wav +expect follow true +wheel 400 400 2 +expect follow false +frames 2 +screenshot .zig-cache/micscope/panned.png +key 0 +expect follow true +frames 2 +screenshot .zig-cache/micscope/overview.png +# Ctrl+wheel off the timelines zooms the view. +hold left_ctrl +wheel 1150 400 1 +release left_ctrl +frames 2 +screenshot .zig-cache/micscope/zoomed-view.png +# The buttons on top do what the keys do, for touch screens. +click 832 18 +expect transport.value recording +click 832 18 +expect transport.value stopped +click 890 18 +expect transport.value forward +click 890 18 +expect transport.value stopped +click 1238 18 +expect a4 441 +frames 2 +screenshot .zig-cache/micscope/buttons.png diff --git a/examples/micscope/test/tuner.script b/examples/micscope/test/tuner.script new file mode 100644 index 0000000..4e0e480 --- /dev/null +++ b/examples/micscope/test/tuner.script @@ -0,0 +1,31 @@ +# The tuner on a take, at its playhead: five notes where the strongest FFT +# peak is not (or not only) the note. Expected, by screenshot: +# E1 0 cents (41.2 Hz) bass, 2nd harmonic loudest +# E2 0 cents (82.4 Hz) guitar, weak fundamental +# A4 +12 cents (443.1 Hz) sharp +# D4 0 cents (293.7 Hz) fundamental missing +# C7 0 cents (2093 Hz) piccolo +console open test/tones.wav +click 107 150 +frames 20 +screenshot .zig-cache/micscope/tuner-e1.png +click 290 150 +frames 20 +screenshot .zig-cache/micscope/tuner-e2.png +click 472 150 +frames 20 +screenshot .zig-cache/micscope/tuner-a4.png +click 654 150 +frames 20 +screenshot .zig-cache/micscope/tuner-d4.png +click 837 150 +frames 20 +screenshot .zig-cache/micscope/tuner-c7.png +# Another reference pitch: A4 = 443 makes the sharp A4 in tune. +click 472 150 +key right_bracket +key right_bracket +key right_bracket +expect a4 443 +frames 20 +screenshot .zig-cache/micscope/tuner-a443.png diff --git a/examples/micscope/web/icon-192.png b/examples/micscope/web/icon-192.png Binary files differnew file mode 100644 index 0000000..0b26078 --- /dev/null +++ b/examples/micscope/web/icon-192.png diff --git a/examples/micscope/web/icon-512.png b/examples/micscope/web/icon-512.png Binary files differnew file mode 100644 index 0000000..0b5538c --- /dev/null +++ b/examples/micscope/web/icon-512.png diff --git a/examples/micscope/web/icon.svg b/examples/micscope/web/icon.svg new file mode 100644 index 0000000..1220011 --- /dev/null +++ b/examples/micscope/web/icon.svg @@ -0,0 +1,26 @@ +<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"> + <!-- micscope's icon: a waveform over its spectrogram's harmonics, in the + app's colours. Full-bleed, so it also works masked. Rendered to the + PNGs beside it with: resvg -w N -h N icon.svg icon-N.png --> + <defs> + <linearGradient id="heat" gradientUnits="userSpaceOnUse" x1="96" y1="0" x2="416" y2="0"> + <stop offset="0" stop-color="#2b0a4a"/> + <stop offset="0.25" stop-color="#b8325a"/> + <stop offset="0.6" stop-color="#f98c0a"/> + <stop offset="1" stop-color="#fcf3a6"/> + </linearGradient> + </defs> + <rect width="512" height="512" fill="#0b0c0f"/> + <!-- harmonics: a fundamental and its overtones, fading --> + <g stroke="url(#heat)" stroke-linecap="round"> + <line x1="96" y1="392" x2="416" y2="392" stroke-width="22"/> + <line x1="96" y1="340" x2="416" y2="340" stroke-width="16" opacity="0.85"/> + <line x1="96" y1="300" x2="416" y2="300" stroke-width="11" opacity="0.65"/> + <line x1="96" y1="270" x2="416" y2="270" stroke-width="7" opacity="0.45"/> + </g> + <!-- the waveform --> + <path d="M96 176 C120 96 152 96 176 176 S232 256 256 176 S312 96 336 176 S392 256 416 176" + fill="none" stroke="#6fdc8c" stroke-width="18" stroke-linecap="round"/> + <!-- the tuner's needle --> + <rect x="246" y="76" width="20" height="356" rx="10" fill="#ffd84d" opacity="0.9"/> +</svg> |
