diff options
Diffstat (limited to 'examples')
52 files changed, 5666 insertions, 0 deletions
diff --git a/examples/micscope/README.md b/examples/micscope/README.md new file mode 100644 index 0000000..759c27c --- /dev/null +++ b/examples/micscope/README.md @@ -0,0 +1,148 @@ +# micscope + +Records the microphone or opens sound files, plays them forwards or +backwards, and shows each as a waveform, its spectrogram on the same time +axis, and the spectrum at the playhead. A [sketchbook](../..) sketch, Linux only for now. + +The window shows a vertical strip of these panels, laid out in a world +and seen through a camera, as in a game. Takes stack downwards in the +order they were made, so each keeps its place; at the bottom is now: what +the microphone hears, its last 6 s scrolling by, with the spectrum of the +newest samples. Recording turns that panel red and fills it with the new +take; stopping leaves the take there and puts the live view under it. The +camera follows the bottom until you pan away, and follows again once you +pan back down. Zoomed out, it overviews the whole strip. Every take keeps +its own time zoom and playhead. + +A tuner reads the live sound (or the recording, or the selected take at +its playhead) and shows it on that panel's spectrum: a line at the pitch, +ticks at its harmonics, the two semitones either side shaded, and that +shaded span magnified across the top: the nearest notes of the 12-tone +chromatic scale, the pitch as a needle among them (green within 5 cents, +yellow within 15), and the note, cents and Hz. It is not a guitar tuner +in particular: it finds 30 Hz to 4.2 kHz, a five-string bass's low B to a +piccolo's top, against any A4 (`[` and `]`, 440 Hz by default). + +Panels are rendered at world resolution and the camera only scales them, +so moving or zooming it recomputes nothing. + +| Shader | Does | +| --- | --- | +| `spectrum.comp` | Blackman-Harris window around the playhead, zero-padded 2×, radix-2 Stockham FFT (8192 points, one workgroup), dBFS per bin, then one value per log-spaced pixel column: a Catmull-Rom spline through the bins where columns are narrower than a bin, the loudest bin where they span several. Smoothed in time (fast attack, slow release) with falling peaks. | +| `waveform.comp` | Per pixel column: min, max and RMS of its samples, plus the interpolated signal at the column's edges, so zoomed past one sample per pixel the columns still join into a line. Recomputed only when the view or tape changes. | +| `spectrogram.comp` | One workgroup per waveform column, so it lines up with the waveform in time. A 2048-sample Blackman-Harris window zero-padded to 4096, transformed as 2048 complex points in shared memory (the real-input trick) and split into 2049 bins. Below 500 Hz a second analysis runs on the signal low-passed and decimated 4×: the same 2048 points span 170 ms and resolve ~6 Hz instead of 23, three windows a quarter apart averaged (Welch) so noise does not speckle; blended with the first between 250 and 500 Hz. Zoomed out, each column averages up to four windows spread across its samples, so it neither skips nor flickers. Rows are log frequency, interpolated like the spectrum. | +| `tuner_difference.comp`, `tuner.comp` | Pitch by YIN rather than by the strongest FFT peak, which is often a harmonic (low strings, voices and brass put more into the 2nd or 3rd) or missing altogether (a missing fundamental). The difference function d(τ) = Σ(x[j] − x[j−τ])² over 2048 samples for lags up to 1700, spread over 107 workgroups; then in one workgroup, normalised by its running mean (a parallel scan), the first dip under 0.12 (the shortest period that explains the signal, so no octave drops) followed to its bottom, and a parabola through the raw dip for the period between samples. Read back to the CPU (`f.download`), which follows the median of the last five clear estimates, restarts on a jump of more than a semitone, and only changes the note shown 60 cents away from it. | +| `waveform.frag`, `spectrogram.frag`, `spectrum.frag` | Full-screen triangles that draw those buffers: the waveform and spectrum anti-aliased with grids, the spectrogram through an inferno heat gradient. | + +The takes share one tape buffer on the GPU (each analysis reads from an +offset), uploaded once. A take's waveform and spectrogram are recomputed +only when its view changes, and its panel redrawn only when something on +it moved; only the selected take's spectrum runs every frame. + +The live view uploads the microphone's last 1.4 s each frame and analyses +only the columns that became complete, into ring buffers the fragment +shaders read from the oldest column. A column waits until everything its +spectrogram windows reach has been heard (6160 samples past its centre), +so it is analysed once and final: the live spectrogram runs about 130 ms +behind, the live spectrum does not. + +## Keys + +| Key | | +| --- | --- | +| R | record a new take / stop | +| Space | play / pause the selected take, resuming in the last direction | +| Right / Left | play forwards / backwards | +| Home / End | seek to the start / end | +| Up / Down | select the older / newer take | +| Delete | delete the selected take | +| O | open a file (WAV, FLAC, Ogg Vorbis) as a take, through the system dialog | +| click, drag | select a take; on its timeline, seek | +| wheel, middle or right drag, Page Up / Down | pan the camera | +| Ctrl + wheel, or a pinch | zoom what is under the pointer: a take's time on its timeline (Z unzooms), the camera elsewhere | +| 0 | the whole strip, or back to full size | +| - / = | waveform gain | +| [ / ] | the tuner's A4, 1 Hz down / up | +| \` | the sketchbook console: `open <path>` opens a file too | + +The buttons (REC, PLAY, REV, OPEN, DEL, ALL, A4−, A4+) do what the keys +do, for touch screens: there, one finger clicks and drags, two fingers +pan, and a pinch zooms like ctrl+wheel. + +The canvas is the window, and the layout follows it. Wide, each panel has +the spectrum beside the waveform and spectrogram, and the buttons sit in +the header; narrower than 760 px (a phone held upright), the spectrum goes +under them at full width, and the buttons along the bottom, where thumbs +are. + +## Rendering reference + +`test/reference.flac` puts a mastered song, a city street and a sweep on one +screen (sources and licences in `test/README.md`); `zig build test` plays it +through `test/render.script` and leaves screenshots in `.zig-cache/micscope/`. + +What it settled for the spectrogram's low end, checked against a plain +numpy STFT of the same columns: + +- One long window resolves the street's two engine hums but shows noise as + speckle full of holes: a single window's spectrum of noise scatters as + much as its level. +- Multitaper averaging (sine tapers, DPSS with NW 2 to 4) calms the noise but + widens the hums into one slab and leaks into silence, which a 96 dB scale + shows. Wrong trade for lines. +- Averaging three windows a quarter apart in time keeps the hums sharp and + the kicks distinct, and settles the noise. That is what it does. + +`test/tones.wav` has five notes where the strongest peak is not the note +(a bass E1 and a guitar E2 louder in their 2nd harmonic, a D4 without its +fundamental), an A4 12 cents sharp and a piccolo C7; `test/tuner.script` +screenshots the tuner on each. All read within 0.1 cent. + +## GPU time + +`zig build profile` zooms the reference one step per frame, recomputing the +waveform and spectrogram every frame, and prints GPU time per command +(`--profile-gpu`: each command timed alone on an idle GPU). On an RTX 3050: + +| Case | GPU per frame | +| --- | --- | +| playing, views cached | ~0.45 ms | +| zooming the 32 s reference | ~1.1 ms on average, spectrogram 1.5 ms on the frames it recomputes | +| zooming a 10 minute tape | ~1.6 ms on average, spectrogram 2.1 ms (worst 4.6 ms) | +| zooming the reference, in the strip with the live view and both tuners | ~1.15 ms on average; the live view adds a spectrum (~0.15 ms) and a few columns, each tuner ~0.1 ms | + +## Building + +Zig 0.17 and `glslc` (shaderc) on PATH; for the web build also `naga` +(`cargo install naga-cli`), which translates the shaders to WGSL, and to +serve it `miniserve` and `openssl`. + + zig build run # with hot reload: `zig build lib --watch` alongside + zig build test # unit tests and the headless scripts in test/ + zig build profile # GPU time per command while zooming + zig-out/bin/micscope # the standalone build + zig build serve # the web build at https://<this machine>:8042/ + # (`zig build web --watch` alongside rebuilds it) + +## On the web + +The same sketch, compiled to wasm32-freestanding, on WebGPU: every shader +runs as on native, translated to WGSL, and the tuner's readback is a +WebGPU buffer mapped a frame or two later. Sound and the microphone go +through Web Audio (the page asks for the microphone, and sound starts at +the first click or key, as browsers require); OPEN decodes a file with the +browser's own decoders. Takes last as long as the page. + +It is an installable app (a PWA): browsers offer to add it to the home +screen, and once opened it works offline, from a copy its service worker +keeps of the page, the code and the wasm. The icon is `web/icon.svg`, +rendered to the PNGs beside it with `resvg -w N -h N icon.svg icon-N.png`. It needs a browser +with WebGPU (Chrome, Edge, Safari 26; Firefox on Linux with +`dom.webgpu.enabled` set in about:config), and a secure context, which +`zig build serve`'s https gives it on any device on the network once the +browser has accepted its self-signed certificate (miniserve prints the +addresses as QR codes, for phones). + +The takes are the sketch's `Store`: they live in `micscope.state.store` and +are still there after a restart. Takes stop at ten minutes; the strip holds +64. A store from before the strip has its one tape as the first take. diff --git a/examples/micscope/build.zig b/examples/micscope/build.zig new file mode 100644 index 0000000..0853cdc --- /dev/null +++ b/examples/micscope/build.zig @@ -0,0 +1,59 @@ +const std = @import("std"); +const sketchbook = @import("sketchbook"); + +pub fn build(b: *std.Build) void { + const target = b.standardTargetOptions(.{}); + const optimize = b.standardOptimizeOption(.{}); + const sb = sketchbook.init(b, .{ .target = target, .optimize = optimize }); + + // The shaders, as SPIR-V for the native host and WGSL for the web; their + // uniform blocks come from the Zig structs in src/layouts.zig. + const layouts = b.createModule(.{ .root_source_file = b.path("src/layouts.zig") }); + const shaders = sb.addShaders(.{ + .layouts = .{ .module = layouts, .types = &.{ "SpectrumParams", "WaveParams", "WaveView", "SpectrumView", "SpectrogramParams", "SpectrogramView", "TunerParams" } }, + .sources = &.{ + b.path("shaders/view.vert"), + b.path("shaders/waveform.comp"), + b.path("shaders/waveform.frag"), + b.path("shaders/spectrogram.comp"), + b.path("shaders/spectrogram.frag"), + b.path("shaders/spectrum.comp"), + b.path("shaders/spectrum.frag"), + b.path("shaders/tuner_difference.comp"), + b.path("shaders/tuner.comp"), + }, + }); + + const app = sb.app(.{ + .name = "micscope", + .root_source_file = b.path("src/main.zig"), + .imports = &.{ + .{ .name = "layouts", .module = layouts }, + .{ .name = "shaders", .module = shaders.module }, + }, + }); + const steps = app.addSteps(.{ + // Installable, and it opens offline (icons from web/icon.svg). + .web = .{ .pwa = .{ + .description = "Records the microphone, shows its waveform, spectrogram and spectrum, and tunes any instrument.", + .background = "#0b0c0f", + .theme = "#0b0c0f", + .icons = &.{ + .{ .source = b.path("web/icon-192.png"), .size = 192 }, + .{ .source = b.path("web/icon-512.png"), .size = 512 }, + }, + } }, + .scripts = &.{ b.path("test/transport.script"), b.path("test/record.script"), b.path("test/render.script"), b.path("test/tuner.script") }, + }); + + // GPU time per command while zooming, which recomputes every frame. + const profile = b.addRunArtifact(steps.exe); + profile.addArgs(&.{ "--headless", "--fresh", "--profile-gpu", "--script" }); + profile.addFileArg(b.path("test/profile.script")); + profile.addArg("--state"); + _ = profile.addOutputFileArg("profile.state"); + profile.setCwd(b.path(".")); + profile.has_side_effects = true; + profile.stdio = .inherit; + b.step("profile", "Print GPU time per command while zooming the reference").dependOn(&profile.step); +} diff --git a/examples/micscope/build.zig.zon b/examples/micscope/build.zig.zon new file mode 100644 index 0000000..9c595cf --- /dev/null +++ b/examples/micscope/build.zig.zon @@ -0,0 +1,10 @@ +.{ + .name = .micscope, + .version = "0.0.1", + .dependencies = .{ + .sketchbook = .{ .path = "../.." }, + }, + .minimum_zig_version = "0.17.0", + .paths = .{""}, + .fingerprint = 0x780fbc0f374f0ce0, +} diff --git a/examples/micscope/shaders/spectrogram.comp b/examples/micscope/shaders/spectrogram.comp new file mode 100644 index 0000000..b32f4c5 --- /dev/null +++ b/examples/micscope/shaders/spectrogram.comp @@ -0,0 +1,208 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrogram, one workgroup per column, the same columns as the +// waveform (waveform.comp), so the two line up in time. +// +// Two resolutions per column, because one window cannot serve both ends of +// a log axis: 2048 samples resolve 23 Hz, which is sharp in time and fine +// in the kilohertz, but leaves 20 to 100 Hz (a quarter of the height) to +// four bins. So the low end is analysed again from the signal low-passed +// and decimated to a quarter of the rate: the same 2048 points then span +// 170 ms and resolve 6 Hz. Rows take the long analysis below 250 Hz, the +// short one above 500 Hz, and blend in between. +// +// A single window's spectrum of noise scatters as much as its level (each +// bin is chi-squared with 2 degrees of freedom), which in dB shows as +// speckle full of black holes, and finer bins only make more of them. So +// the long analysis averages three windows a quarter window apart in time +// (Welch): steady lines stay as sharp, noise settles. Multitaper averaging +// was tried and measured against a reference (see test/README.md): sine +// tapers and DPSS both widen narrow lines into slabs and leak, which a +// 96 dB scale shows. +// +// Each analysis is a Blackman-Harris window zero-padded to 4096 points, +// transformed as 2048 complex points (even and odd samples as real and +// imaginary parts) by an in-place radix-2 FFT in shared memory, then split +// into the real signal's bins. Zoomed out, a column averages the power of +// up to four windows spread across its samples, so it neither skips sound +// nor flickers as the view moves; zoomed in, neighbouring windows overlap. +// Rows are log-spaced: a Catmull-Rom spline through the bins' levels where +// rows are finer than bins, the loudest bin where they are coarser. +layout(local_size_x = 256) in; + +#include "SpectrogramParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// MAX_ROWS rows of MAX_COLUMNS levels, dBFS; row 0 is the lowest frequency. +layout(set = 1, binding = 0) buffer Levels { float levels[]; }; + +// Match layouts.zig. +const uint MAX_COLUMNS = 2048; +const uint W = 2048; +const uint N = 4096; +const uint HALF = N / 2; +const uint LOG2_HALF = 11; +const uint THREADS = 256; +// The long analysis: every DECIMATE-th sample of the low-passed signal. +const uint DECIMATE = 4; +// Its bins needed below the blend: 600 Hz at 12 kHz / 4096 is bin 205. +const uint LONG_BINS = 256; +// Windows the long analysis averages per column, a quarter window apart. +const uint WELCH = 3; +const float BLEND_LO = 250.0; +const float BLEND_HI = 500.0; +const float PI = 3.14159265358979; +const float SILENCE = -160.0; +// Columns past the end of the tape. +const float NO_TAPE = 1000.0; +const float A0 = 0.35875; +const float A1 = 0.48829; +const float A2 = 0.14128; +const float A3 = 0.01168; + +shared vec2 z[HALF]; +// Power per bin, summed over windows; then dBFS. +shared float short_bins[HALF + 1]; +shared float long_bins[LONG_BINS]; + +float sample_at(int i) { + return i >= 0 && i < int(u.samples) ? tape[u.offset + uint(i)] : 0.0; +} + +// The signal low-passed for decimation: a triangle over 7 samples (two +// boxes of 4), whose zeros sit on 12 and 24 kHz, exactly where anything +// that would fold onto the low bins comes from. +float low_passed(int i) { + return (sample_at(i - 3) + 2.0 * sample_at(i - 2) + 3.0 * sample_at(i - 1) + 4.0 * sample_at(i) + 3.0 * sample_at(i + 1) + 2.0 * sample_at(i + 2) + sample_at(i + 3)) / 16.0; +} + +float blackman_harris(uint i) { + float a = 2.0 * PI * float(i) / float(W - 1); + return A0 - A1 * cos(a) + A2 * cos(2.0 * a) - A3 * cos(3.0 * a); +} + +vec2 mul(vec2 a, vec2 b) { + return vec2(a.x * b.x - a.y * b.y, a.x * b.y + a.y * b.x); +} + +float windowed(int centre, uint i, bool long_window) { + if (long_window) return low_passed(centre + (int(i) - int(W / 2)) * int(DECIMATE)) * blackman_harris(i); + return sample_at(centre + int(i) - int(W / 2)) * blackman_harris(i); +} + +// Adds the power spectrum of the window centred on sample `centre`. +void analyse(int centre, bool long_window) { + uint t = gl_LocalInvocationID.x; + // Even and odd samples packed as one complex sequence, in bit-reversed + // order for the in-place transform. Past W it is all padding. + for (uint i = t; i < HALF; i += THREADS) { + vec2 v = vec2(0.0); + if (2u * i < W) v = vec2(windowed(centre, 2u * i, long_window), windowed(centre, 2u * i + 1u, long_window)); + z[bitfieldReverse(i) >> (32u - LOG2_HALF)] = v; + } + barrier(); + for (uint s = 1u; s <= LOG2_HALF; s++) { + uint half_size = 1u << (s - 1u); + for (uint j = t; j < HALF / 2u; j += THREADS) { + uint k = j & (half_size - 1u); + uint i0 = ((j >> (s - 1u)) << s) + k; + uint i1 = i0 + half_size; + float angle = -PI * float(k) / float(half_size); + vec2 b = mul(vec2(cos(angle), sin(angle)), z[i1]); + vec2 a = z[i0]; + z[i0] = a + b; + z[i1] = a - b; + } + barrier(); + } + // The real signal's bins from the packed transform: + // X[k] = E[k] + e^(-2 pi i k / N) O[k], with E and O the even and odd + // samples' spectra, recovered from Z[k] and conj(Z[HALF - k]). + uint count = long_window ? LONG_BINS : HALF + 1u; + for (uint k = t; k < count; k += THREADS) { + vec2 zk = z[k % HALF]; + vec2 zc = z[(HALF - k) % HALF]; + zc.y = -zc.y; + vec2 even = 0.5 * (zk + zc); + vec2 d = 0.5 * (zk - zc); + vec2 odd = vec2(d.y, -d.x); + float angle = -2.0 * PI * float(k) / float(N); + vec2 x = even + mul(vec2(cos(angle), sin(angle)), odd); + if (long_window) long_bins[k] += dot(x, x); + else short_bins[k] += dot(x, x); + } + barrier(); +} + +float bin_level(uint k, bool long_window) { + return long_window ? long_bins[min(k, LONG_BINS - 1u)] : short_bins[min(k, HALF)]; +} + +// The level of the row spanning bins `lo` to `hi` of one analysis. +float row_level(float lo, float hi, bool long_window) { + if (hi - lo < 1.0) { + float mid = sqrt(lo * hi); + uint i = uint(floor(mid)); + float f = mid - floor(mid); + float p0 = bin_level(max(i, 1u) - 1u, long_window); + float p1 = bin_level(i, long_window); + float p2 = bin_level(i + 1u, long_window); + float p3 = bin_level(i + 2u, long_window); + float v = p1 + 0.5 * f * (p2 - p0 + f * (2.0 * p0 - 5.0 * p1 + 4.0 * p2 - p3 + f * (3.0 * (p1 - p2) + p3 - p0))); + return clamp(v, SILENCE, max(max(p1, p2), max(p0, p3))); + } + float v = SILENCE; + for (uint i = uint(floor(lo)); i <= uint(ceil(hi)); i++) v = max(v, bin_level(i, long_window)); + return v; +} + +// Average power over `count` windows to dBFS: a full-scale sine at a bin's +// centre gives (W * A0 / 2) squared. +float to_db(float power, uint count) { + float ratio = power / float(count) / (0.25 * float(W) * A0 * float(W) * A0); + return ratio > 0.0 ? max(10.0 * log(ratio) / log(10.0), SILENCE) : SILENCE; +} + +void main() { + uint c = gl_WorkGroupID.x; + uint t = gl_LocalInvocationID.x; + for (uint k = t; k <= HALF; k += THREADS) short_bins[k] = 0.0; + for (uint k = t; k < LONG_BINS; k += THREADS) long_bins[k] = 0.0; + barrier(); + + float start = float(c) * u.spp; + uint short_windows = uint(clamp(ceil(u.spp / float(W / 2u)), 1.0, 4.0)); + for (uint w = 0u; w < short_windows; w++) { + analyse(int(float(u.first) + start + u.spp * (float(w) + 0.5) / float(short_windows)), false); + } + // Zoomed out, the windows spread across the column average already; + // zoomed in, WELCH windows a quarter window apart around its centre. + uint long_windows = uint(clamp(ceil(u.spp / float(W * DECIMATE / 2u)), 1.0, 4.0)); + if (long_windows == 1u) { + int centre = int(float(u.first) + start + 0.5 * u.spp); + for (uint w = 0u; w < WELCH; w++) analyse(centre + (int(w) - int(WELCH / 2u)) * int(W * DECIMATE / 4u), true); + long_windows = WELCH; + } else { + for (uint w = 0u; w < long_windows; w++) { + analyse(int(float(u.first) + start + u.spp * (float(w) + 0.5) / float(long_windows)), true); + } + } + + for (uint k = t; k <= HALF; k += THREADS) short_bins[k] = to_db(short_bins[k], short_windows); + for (uint k = t; k < LONG_BINS; k += THREADS) long_bins[k] = to_db(long_bins[k], long_windows); + barrier(); + + bool inside = float(u.first) + start < float(u.samples); + float short_hz = u.rate / float(N); + float long_hz = short_hz / float(DECIMATE); + float ratio = log(u.max_freq / u.min_freq); + for (uint r = t; r < u.rows; r += THREADS) { + float lo = u.min_freq * exp(ratio * float(r) / float(u.rows)); + float hi = u.min_freq * exp(ratio * float(r + 1u) / float(u.rows)); + float blend = smoothstep(BLEND_LO, BLEND_HI, sqrt(lo * hi)); + float v = 0.0; + if (blend < 1.0) v += (1.0 - blend) * row_level(lo / long_hz, hi / long_hz, true); + if (blend > 0.0) v += blend * row_level(lo / short_hz, hi / short_hz, false); + levels[r * MAX_COLUMNS + (u.slot + c) % u.ring] = inside ? v : NO_TAPE; + } +} diff --git a/examples/micscope/shaders/spectrogram.frag b/examples/micscope/shaders/spectrogram.frag new file mode 100644 index 0000000..9cb5e3d --- /dev/null +++ b/examples/micscope/shaders/spectrogram.frag @@ -0,0 +1,45 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrogram (spectrogram.comp): time across, as the waveform above +// it, log frequency up, each level through a heat gradient (inferno: +// black, violet, red, orange, pale yellow), with the playhead. +layout(location = 0) out vec4 out_color; + +#include "SpectrogramView.glsl" + +layout(set = 2, binding = 0) readonly buffer Levels { float levels[]; }; + +// Match layouts.zig and spectrogram.comp. +const uint MAX_COLUMNS = 2048; +const float NO_TAPE = 1000.0; + +// Matplotlib's inferno, as a polynomial fit (Matt Zucker's, CC0). +vec3 inferno(float t) { + const vec3 c0 = vec3(0.0002189403691192265, 0.001651004631001012, -0.01948089843709184); + const vec3 c1 = vec3(0.1065134194856116, 0.5639564367884091, 3.932712388889277); + const vec3 c2 = vec3(11.60249308247187, -3.972853965665698, -15.9423941062914); + const vec3 c3 = vec3(-41.70399613139459, 17.43639888205313, 44.35414519872813); + const vec3 c4 = vec3(77.162935699427, -33.40235894210092, -81.80730925738993); + const vec3 c5 = vec3(-71.31942824499214, 32.62606426397723, 73.20951985803202); + const vec3 c6 = vec3(25.13112622477341, -12.24266895238567, -23.07032500287172); + return clamp(c0 + t * (c1 + t * (c2 + t * (c3 + t * (c4 + t * (c5 + t * c6))))), 0.0, 1.0); +} + +void main() { + vec2 p = gl_FragCoord.xy; + uint row = uint(u.size.y - p.y); + uint x = uint(p.x); + float db = levels[row * MAX_COLUMNS + (x + u.scroll) % uint(u.size.x)]; + if (x + u.filled < uint(u.size.x)) db = NO_TAPE; + vec3 color = vec3(0.035, 0.03, 0.05); + if (db < NO_TAPE) { + float t = clamp((db - u.floor_db) / (u.ceil_db - u.floor_db), 0.0, 1.0); + // A touch of contrast: quiet stays dark, loud blooms. + color = inferno(pow(t, 1.3)); + } + if (u.playhead >= 0.0) { + float d = abs(p.x - u.playhead); + color = mix(color, vec3(1.0, 0.85, 0.3), clamp(1.5 - d, 0.0, 1.0) * 0.8); + } + out_color = vec4(color, 1.0); +} diff --git a/examples/micscope/shaders/spectrum.comp b/examples/micscope/shaders/spectrum.comp new file mode 100644 index 0000000..e576e30 --- /dev/null +++ b/examples/micscope/shaders/spectrum.comp @@ -0,0 +1,128 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrum at the playhead, entirely on the GPU, in one workgroup: +// +// 1. a Blackman-Harris window of the tape ending at `u.end`, zero-padded to +// N. Its sidelobes sit below -92 dB, so the finely sampled spectrum shows +// the signal rather than the window's ripples; +// 2. a radix-2 Stockham FFT, ping-ponging between the scratch halves; +// 3. levels in dBFS per bin; +// 4. one value per log-spaced column. Where columns are narrower than a bin +// (the low end), a Catmull-Rom spline through the bins' levels; where a +// column spans several bins (the high end), their loudest. The zero +// padding already interpolates between the window's own bins, so the +// curve is continuous at every zoom; +// 5. smoothing in time (fast attack, slow release) and falling peaks, kept +// in the column buffer between frames. +layout(local_size_x = 256) in; + +#include "SpectrumParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// Two halves of N complex values, which the workgroup's threads read from +// each other: coherent, for Vulkan; WGSL's storageBarrier covers it. +#ifdef SKETCHBOOK_WEB +layout(set = 1, binding = 0) buffer Scratch { vec2 scratch[]; }; +#else +layout(set = 1, binding = 0) coherent buffer Scratch { vec2 scratch[]; }; +#endif +// x: smoothed level, y: peak, both dBFS. +layout(set = 1, binding = 1) buffer Columns { vec2 columns[]; }; + +// Match layouts.zig. +const uint N = 8192; +const uint LOG2_N = 13; +const uint W = 4096; +const uint THREADS = 256; +const float PI = 3.14159265358979; +const float SILENCE = -160.0; +// Blackman-Harris, 4 terms. A0 is also its coherent gain. +const float A0 = 0.35875; +const float A1 = 0.48829; +const float A2 = 0.14128; +const float A3 = 0.01168; + +void sync() { + memoryBarrierBuffer(); + barrier(); +} + +void main() { + uint t = gl_LocalInvocationID.x; + + // 1. Window. + int first = int(u.end) - int(W); + for (uint i = t; i < N; i += THREADS) { + float x = 0.0; + int at = first + int(i); + if (i < W && at >= 0 && at < int(u.samples)) { + float a = 2.0 * PI * float(i) / float(W - 1); + x = tape[u.offset + uint(at)] * (A0 - A1 * cos(a) + A2 * cos(2.0 * a) - A3 * cos(3.0 * a)); + } + scratch[i] = vec2(x, 0.0); + } + sync(); + + // 2. FFT: after stage s, sub-transforms of 2p = 2^(s+1) points, in order. + for (uint s = 0; s < LOG2_N; s++) { + uint p = 1u << s; + uint from = (s & 1u) * N; + uint to = N - from; + for (uint i = t; i < N / 2; i += THREADS) { + uint k = i & (p - 1u); + vec2 a = scratch[from + i]; + vec2 b = scratch[from + i + N / 2]; + float angle = -PI * float(k) / float(p); + vec2 w = vec2(cos(angle), sin(angle)); + b = vec2(b.x * w.x - b.y * w.y, b.x * w.y + b.y * w.x); + uint j = (i << 1u) - k; + scratch[to + j] = a + b; + scratch[to + j + p] = a - b; + } + sync(); + } + uint result = (LOG2_N & 1u) * N; + + // 3. Levels into the other half. A full-scale sine peaks at W * A0 / 2: + // the window sums to W * A0, and a real sine splits between ±f. + uint levels = N - result; + for (uint i = t; i <= N / 2; i += THREADS) { + float amplitude = length(scratch[result + i]) * 2.0 / (float(W) * A0); + scratch[levels + i].x = amplitude > 0.0 ? max(20.0 * log(amplitude) / log(10.0), SILENCE) : SILENCE; + } + sync(); + + // 4 and 5. Columns. + float bin_hz = u.rate / float(N); + float ratio = log(u.max_freq / u.min_freq); + for (uint c = t; c < u.columns; c += THREADS) { + float lo = u.min_freq * exp(ratio * float(c) / float(u.columns)) / bin_hz; + float hi = u.min_freq * exp(ratio * float(c + 1u) / float(u.columns)) / bin_hz; + float mid = sqrt(lo * hi); + float v; + if (hi - lo < 1.0) { + uint i = uint(floor(mid)); + float f = mid - floor(mid); + float p0 = scratch[levels + min(max(i, 1u) - 1u, N / 2)].x; + float p1 = scratch[levels + min(i, N / 2)].x; + float p2 = scratch[levels + min(i + 1u, N / 2)].x; + float p3 = scratch[levels + min(i + 2u, N / 2)].x; + v = p1 + 0.5 * f * (p2 - p0 + f * (2.0 * p0 - 5.0 * p1 + 4.0 * p2 - p3 + f * (3.0 * (p1 - p2) + p3 - p0))); + v = clamp(v, SILENCE, max(max(p1, p2), max(p0, p3))); + } else { + v = SILENCE; + uint end = min(uint(ceil(hi)), N / 2); + for (uint i = uint(floor(lo)); i <= end; i++) v = max(v, scratch[levels + i].x); + } + vec2 old = columns[c]; + vec2 now; + if (u.reset != 0u) { + now = vec2(v, v); + } else { + float rate = v > old.x ? 40.0 : 6.0; + now.x = mix(old.x, v, 1.0 - exp(-rate * u.dt)); + now.y = max(now.x, old.y - 24.0 * u.dt); + } + columns[c] = now; + } +} diff --git a/examples/micscope/shaders/spectrum.frag b/examples/micscope/shaders/spectrum.frag new file mode 100644 index 0000000..e88b091 --- /dev/null +++ b/examples/micscope/shaders/spectrum.frag @@ -0,0 +1,51 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The spectrum from its columns (spectrum.comp): one column per pixel on a +// log frequency axis. The curve is drawn by its distance along the normal, +// from the neighbours' slope, so steep parts stay as thick as flat ones. +layout(location = 0) out vec4 out_color; + +#include "SpectrumView.glsl" + +layout(set = 2, binding = 0) readonly buffer Columns { vec2 columns[]; }; + +float y_of(float db) { + return u.size.y * (1.0 - clamp((db - u.floor_db) / (u.ceil_db - u.floor_db), 0.0, 1.0)); +} + +float x_of(float hz) { + return u.size.x * log(hz / u.min_freq) / log(u.max_freq / u.min_freq); +} + +// How close `p` is to a grid line, 0..1. +float grid(vec2 p) { + const float lines[10] = float[](20.0, 50.0, 100.0, 200.0, 500.0, 1000.0, 2000.0, 5000.0, 10000.0, 20000.0); + float near = 1e9; + for (int i = 0; i < 10; i++) near = min(near, abs(p.x - x_of(lines[i]))); + float step = u.size.y * 12.0 / (u.ceil_db - u.floor_db); + float db = mod(p.y, step); + near = min(near, min(db, step - db)); + return clamp(1.0 - near, 0.0, 1.0); +} + +void main() { + vec2 p = gl_FragCoord.xy; + int n = int(u.size.x); + int x = int(p.x); + float y = y_of(columns[x].x); + float slope = (y_of(columns[min(x + 1, n - 1)].x) - y_of(columns[max(x - 1, 0)].x)) * 0.5; + float distance = abs(p.y - y) / sqrt(1.0 + slope * slope); + + vec3 color = vec3(0.055, 0.06, 0.07); + color = mix(color, vec3(0.16, 0.17, 0.2), grid(p)); + + // Fill under the curve, by frequency, fading downwards. + float along = p.x / u.size.x; + vec3 hue = mix(vec3(0.2, 0.75, 0.9), vec3(0.85, 0.35, 0.9), along); + if (p.y > y) color = mix(color, hue, 0.35 * (1.0 - (p.y - y) / max(u.size.y - y, 1.0)) + 0.08); + // Peaks, then the curve. + float peak = abs(p.y - y_of(columns[x].y)); + color = mix(color, vec3(0.9, 0.9, 0.95), clamp(1.0 - peak, 0.0, 1.0) * 0.45); + color = mix(color, mix(hue, vec3(1.0), 0.5), clamp(1.6 - distance, 0.0, 1.0)); + out_color = vec4(color, 1.0); +} diff --git a/examples/micscope/shaders/tuner.comp b/examples/micscope/shaders/tuner.comp new file mode 100644 index 0000000..1907f7f --- /dev/null +++ b/examples/micscope/shaders/tuner.comp @@ -0,0 +1,134 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The pitch of the newest (or the playhead's) sound, by YIN (de Cheveigné +// and Kawahara, 2002), in one workgroup. Picking the strongest FFT peak +// finds the loudest harmonic, which is often not the note: low strings, +// voices and brass put more energy in the 2nd or 3rd harmonic, and a +// missing fundamental has no peak at all. YIN looks for the period instead: +// +// 1. the difference function d(tau) = sum over the window of +// (x[j] - x[j + tau])^2, which dips to 0 at every multiple of the +// period, whatever the harmonics' balance; +// 2. normalised by its running mean, d'(tau) = d(tau) * tau / sum d(1..tau), +// which starts at 1 and removes the dip-at-zero bias, so the dips are +// comparable; +// 3. the first dip below an absolute threshold, then the bottom of it: the +// shortest period that explains the signal, which keeps it from +// dropping an octave onto a multiple of the period; +// 4. a parabola through that dip of d and its neighbours, for a period +// between samples: the semitone is 6 %, the cent 0.06 %, and whole samples +// alone would be 30 cents apart at 1 kHz. +layout(local_size_x = 256) in; + +#include "TunerParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// d(tau), from tuner_difference.comp, which does the bulk of the work. +layout(set = 0, binding = 1) readonly buffer Difference { float difference[]; }; +// freq, clarity, level, period: layouts.Pitch. +layout(set = 1, binding = 0) buffer Result { vec4 result; }; + +// Match layouts.zig. +const uint W = 2048; +const uint LAGS = 1700; +const uint THREADS = 256; + +// Each thread owns CHUNK consecutive lags. +const uint CHUNK = (LAGS + THREADS - 1u) / THREADS; +shared float d[THREADS * CHUNK]; +// d before normalising: the parabola goes through these, since the +// normalisation's factor tau tilts a dip by more than a cent at high pitch. +shared float raw[THREADS * CHUNK]; +shared float squares[THREADS]; +shared float sums[THREADS]; +shared uint first_dip; +shared float deepest[THREADS]; +shared uint deepest_at[THREADS]; + +void main() { + uint t = gl_LocalInvocationID.x; + uint tau_max = min(LAGS - 2u, uint(ceil(u.rate / u.min_freq))); + uint tau_min = max(2u, uint(floor(u.rate / u.max_freq))); + + // 1. The difference function (tuner_difference.comp), and the window's + // energy. + int first = int(u.end) - int(W); + float sq = 0.0; + for (uint j = t; j < W; j += THREADS) { + int at = first + int(j); + float v = at >= 0 && at < int(u.samples) ? tape[u.offset + uint(at)] : 0.0; + sq += v * v; + } + squares[t] = sq; + float own = 0.0; + for (uint i = 0u; i < CHUNK; i++) { + uint tau = t * CHUNK + i; + float v = tau >= 1u && tau <= tau_max + 1u ? difference[tau] : 0.0; + raw[tau] = v; + own += v; + } + sums[t] = own; + if (t == 0u) first_dip = 0xffffffffu; + barrier(); + + // 2. Normalised by the running sum: a scan of the threads' chunk sums + // (Hillis-Steele), then each chunk in order. + for (uint stride = 1u; stride < THREADS; stride <<= 1u) { + float add = t >= stride ? sums[t - stride] : 0.0; + barrier(); + sums[t] += add; + barrier(); + } + float running = t > 0u ? sums[t - 1u] : 0.0; + float low = 2.0; + uint low_at = 0u; + for (uint i = 0u; i < CHUNK; i++) { + uint tau = t * CHUNK + i; + running += raw[tau]; + float v = tau == 0u ? 1.0 : (running > 0.0 ? raw[tau] * float(tau) / running : 1.0); + d[tau] = v; + if (tau >= tau_min && tau <= tau_max) { + // 3. The first dip under the threshold, and the deepest. + if (v < u.threshold) atomicMin(first_dip, tau); + if (v < low) { + low = v; + low_at = tau; + } + } + } + deepest[t] = low; + deepest_at[t] = low_at; + barrier(); + for (uint stride = THREADS / 2u; stride > 0u; stride >>= 1u) { + if (t < stride && deepest[t + stride] < deepest[t]) { + deepest[t] = deepest[t + stride]; + deepest_at[t] = deepest_at[t + stride]; + } + barrier(); + } + if (t != 0u) return; + + float energy = 0.0; + for (uint i = 0u; i < THREADS; i++) energy += squares[i]; + float level = energy > 0.0 ? 10.0 * log(energy / float(W)) / log(10.0) : -160.0; + + // The first dip followed to its bottom; failing one, the deepest, which + // the CPU may still reject as unclear. + uint best = first_dip; + if (best == 0xffffffffu) { + best = max(deepest_at[0], tau_min); + } else { + while (best + 1u <= tau_max && d[best + 1u] < d[best]) best++; + } + + // 4. The parabola through the dip. + float a = raw[best - 1u]; + float b = raw[best]; + float c = raw[best + 1u]; + float bend = a - 2.0 * b + c; + float shift = bend > 0.0 ? clamp(0.5 * (a - c) / bend, -0.5, 0.5) : 0.0; + float period = float(best) + shift; + float depth = d[best]; + bool silent = energy <= 0.0; + result = vec4(silent ? 0.0 : u.rate / period, silent ? 0.0 : clamp(1.0 - depth, 0.0, 1.0), level, period); +} diff --git a/examples/micscope/shaders/tuner_difference.comp b/examples/micscope/shaders/tuner_difference.comp new file mode 100644 index 0000000..c7d1b6e --- /dev/null +++ b/examples/micscope/shaders/tuner_difference.comp @@ -0,0 +1,49 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// YIN's difference function for tuner.comp, spread over many workgroups: +// d(tau) = sum over the newest W samples of (x[j] - x[j - tau])^2. Each +// workgroup takes LAGS_PER_GROUP lags, with THREADS / LAGS_PER_GROUP +// threads summing every so-many term of each, then adding up. +layout(local_size_x = 256) in; + +#include "TunerParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +layout(set = 1, binding = 0) buffer Difference { float d[]; }; + +// Match layouts.zig and tuner.comp. +const uint W = 2048; +const uint LAGS = 1700; +const uint THREADS = 256; +const uint LAGS_PER_GROUP = 16; +const uint PER_LAG = THREADS / LAGS_PER_GROUP; + +shared float x[W + LAGS]; +shared float partial[THREADS]; + +void main() { + uint t = gl_LocalInvocationID.x; + int first = int(u.end) - int(W + LAGS); + for (uint i = t; i < W + LAGS; i += THREADS) { + int at = first + int(i); + x[i] = at >= 0 && at < int(u.samples) ? tape[u.offset + uint(at)] : 0.0; + } + barrier(); + + uint tau = gl_WorkGroupID.x * LAGS_PER_GROUP + t / PER_LAG; + uint lane = t % PER_LAG; + float sum = 0.0; + if (tau < LAGS) { + for (uint j = lane; j < W; j += PER_LAG) { + float diff = x[LAGS + j] - x[LAGS + j - tau]; + sum += diff * diff; + } + } + partial[t] = sum; + barrier(); + for (uint stride = PER_LAG / 2u; stride > 0u; stride >>= 1u) { + if (lane < stride) partial[t] += partial[t + stride]; + barrier(); + } + if (lane == 0u && tau < LAGS) d[tau] = partial[t]; +} diff --git a/examples/micscope/shaders/view.vert b/examples/micscope/shaders/view.vert new file mode 100644 index 0000000..d9461e9 --- /dev/null +++ b/examples/micscope/shaders/view.vert @@ -0,0 +1,7 @@ +#version 450 +// One triangle over the whole target, from the vertex index alone: the +// fragment shaders work in pixels (gl_FragCoord). +void main() { + vec2 p = vec2((gl_VertexIndex << 1) & 2, gl_VertexIndex & 2); + gl_Position = vec4(p * 2.0 - 1.0, 0.0, 1.0); +} diff --git a/examples/micscope/shaders/waveform.comp b/examples/micscope/shaders/waveform.comp new file mode 100644 index 0000000..d1d64e5 --- /dev/null +++ b/examples/micscope/shaders/waveform.comp @@ -0,0 +1,72 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// One workgroup per waveform column: the lowest and highest the signal goes +// in it, and its RMS. Each column also includes the linearly interpolated +// signal at its own edges, which it shares with its neighbours, so that +// zoomed in past one sample per column the columns still join into one +// continuous line. +layout(local_size_x = 64) in; + +#include "WaveParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Tape { float tape[]; }; +// x: min, y: max, z: RMS, w: 1 where the column has tape under it. +layout(set = 1, binding = 0) buffer Columns { vec4 columns[]; }; + +const uint THREADS = 64; +shared float lows[THREADS]; +shared float highs[THREADS]; +shared float squares[THREADS]; + +float sample_at(int i) { + return i >= 0 && i < int(u.samples) ? tape[u.offset + uint(i)] : 0.0; +} + +// The signal at `t` samples after `u.first`, between samples linearly. +float signal(float t) { + float whole = floor(t); + int i = int(u.first) + int(whole); + return mix(sample_at(i), sample_at(i + 1), t - whole); +} + +void main() { + uint c = gl_WorkGroupID.x; + uint t = gl_LocalInvocationID.x; + float start = float(c) * u.spp; + float end = start + u.spp; + int first = int(ceil(start)); + int last = min(int(floor(end)), int(u.samples) - int(u.first) - 1); + + float lo = 1e9; + float hi = -1e9; + float sq = 0.0; + for (int i = first + int(t); i <= last; i += int(THREADS)) { + float v = tape[u.offset + u.first + uint(i)]; + lo = min(lo, v); + hi = max(hi, v); + sq += v * v; + } + if (t == 0u) { + float a = signal(start); + float b = signal(min(end, float(u.samples - u.first - 1u))); + lo = min(lo, min(a, b)); + hi = max(hi, max(a, b)); + } + lows[t] = lo; + highs[t] = hi; + squares[t] = sq; + barrier(); + for (uint stride = THREADS / 2u; stride > 0u; stride >>= 1u) { + if (t < stride) { + lows[t] = min(lows[t], lows[t + stride]); + highs[t] = max(highs[t], highs[t + stride]); + squares[t] += squares[t + stride]; + } + barrier(); + } + if (t == 0u) { + float count = float(max(last - first + 1, 1)); + float inside = float(u.first) + start < float(u.samples) ? 1.0 : 0.0; + columns[(u.slot + c) % u.ring] = vec4(lows[0], highs[0], sqrt(squares[0] / count), inside); + } +} diff --git a/examples/micscope/shaders/waveform.frag b/examples/micscope/shaders/waveform.frag new file mode 100644 index 0000000..37be7ce --- /dev/null +++ b/examples/micscope/shaders/waveform.frag @@ -0,0 +1,56 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The waveform from its columns (waveform.comp): one column per pixel, the +// band between its low and high filled, anti-aliased, at least a pixel +// thick; the RMS inside it brighter; played and unplayed in two colors, +// red while recording, green for the live microphone. +layout(location = 0) out vec4 out_color; + +#include "WaveView.glsl" + +layout(set = 2, binding = 0) readonly buffer Columns { vec4 columns[]; }; + +// Match layouts.Tint. +const uint TINT_RECORDING = 1u; +const uint TINT_LIVE = 2u; + +void main() { + vec2 p = gl_FragCoord.xy; + uint x = uint(p.x); + vec4 col = columns[(x + u.scroll) % uint(u.size.x)]; + if (x + u.filled < uint(u.size.x)) col.w = 0.0; + float mid = u.size.y * 0.5; + float scale = u.size.y * 0.46 * u.gain; + + vec3 color = vec3(0.055, 0.06, 0.07); + if (col.w > 0.0) color = vec3(0.075, 0.08, 0.095); + // Center line. + color = mix(color, vec3(0.2, 0.22, 0.26), clamp(1.0 - abs(p.y - mid), 0.0, 1.0) * 0.6); + + if (col.w > 0.0) { + float top = mid - col.y * scale - 0.5; + float bottom = mid - col.x * scale + 0.5; + float outside = max(top - p.y, p.y - bottom); + float cover = clamp(0.5 - outside, 0.0, 1.0); + float rms = col.z * scale; + float inner = clamp(rms - abs(p.y - mid) + 0.5, 0.0, 1.0); + bool played = u.playhead >= 0.0 && p.x < u.playhead; + vec3 band = played ? vec3(0.35, 0.8, 0.95) : vec3(0.45, 0.5, 0.58); + vec3 core = played ? vec3(0.7, 0.95, 1.0) : vec3(0.65, 0.7, 0.78); + if (u.tint == TINT_RECORDING) { + band = vec3(0.95, 0.35, 0.35); + core = vec3(1.0, 0.6, 0.55); + } else if (u.tint == TINT_LIVE) { + band = vec3(0.35, 0.75, 0.5); + core = vec3(0.65, 1.0, 0.75); + } + color = mix(color, mix(band, core, inner), cover); + } + + // Playhead. + if (u.playhead >= 0.0) { + float d = abs(p.x - u.playhead); + color = mix(color, vec3(1.0, 0.85, 0.3), clamp(1.5 - d, 0.0, 1.0)); + } + out_color = vec4(color, 1.0); +} diff --git a/examples/micscope/src/layouts.zig b/examples/micscope/src/layouts.zig new file mode 100644 index 0000000..cd5be65 --- /dev/null +++ b/examples/micscope/src/layouts.zig @@ -0,0 +1,154 @@ +//! What the shaders read, as Zig structs: the only definition. The shaders +//! include GLSL generated from these (see sketchbook's `glsl.zig`). No +//! imports, so the shader build can compile this alone. + +/// FFT length: the analysis window, zero-padded to twice its size so the +/// spectrum is sampled twice as finely as the window resolves. +pub const fft_size = 8192; +pub const window = 4096; +/// The most columns a view can have; the column buffers hold this many. +pub const max_columns = 2048; +/// The tallest spectrogram; its buffer holds this many rows of +/// `max_columns`. +pub const max_rows = 1024; +// All three analyses read samples `offset + i` of the tape buffer for +// `0 <= i < samples`: the takes share one buffer, and a take's analysis +// never reads its neighbours. +/// The spectrogram's window, zero-padded to `spectrogram_fft` points. +pub const spectrogram_window = 2048; +pub const spectrogram_fft = 4096; + +/// spectrum.comp: the window ending at `end`, and the columns to fill. +pub const SpectrumParams = extern struct { + offset: u32, + /// Sample index one past the window's last sample. + end: u32, + /// Samples on the tape. + samples: u32, + columns: u32, + /// Nonzero: forget the smoothed levels and peaks. + reset: u32, + rate: f32, + min_freq: f32, + max_freq: f32, + /// Seconds since the previous analysis, for smoothing. + dt: f32, + pad0: u32 = 0, + pad1: u32 = 0, + pad2: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// waveform.comp: column `c` covers samples `first + c * spp` up to the next +/// column's start, and is written to slot `(slot + c) % ring`: the live +/// view writes its newest columns into a ring, the others from slot 0. +pub const WaveParams = extern struct { + offset: u32, + first: u32, + samples: u32, + columns: u32, + spp: f32, + slot: u32, + ring: u32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +pub const Tint = enum(u32) { take, recording, live }; + +/// waveform.frag. Pixel column `x` shows slot `(x + scroll) % width`; only +/// the last `filled` columns have anything in them. +pub const WaveView = extern struct { + size: [2]f32, + /// Pixels; negative hides it. + playhead: f32, + gain: f32, + /// A `Tint`. + tint: u32, + scroll: u32, + filled: u32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; + +/// spectrum.frag. +pub const SpectrumView = extern struct { + size: [2]f32, + floor_db: f32, + ceil_db: f32, + min_freq: f32, + max_freq: f32, + pad: [2]f32 = .{ 0, 0 }, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; + +/// spectrogram.comp: the same columns as `WaveParams`, each analysed into +/// `rows` log-spaced frequency rows, row 0 the lowest. +/// Columns land in slots as `WaveParams` says. +pub const SpectrogramParams = extern struct { + offset: u32, + first: u32, + samples: u32, + columns: u32, + rows: u32, + spp: f32, + rate: f32, + min_freq: f32, + max_freq: f32, + slot: u32, + ring: u32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// spectrogram.frag; `scroll` and `filled` as in `WaveView`. +pub const SpectrogramView = extern struct { + size: [2]f32, + floor_db: f32, + ceil_db: f32, + /// Pixels; negative hides it. + playhead: f32, + scroll: u32, + filled: u32, + pad: f32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; + +/// tuner.comp: the YIN window is the `tuner_window + tuner_lags` samples +/// ending at `end`, read like the other analyses (`offset`, `samples`). +pub const tuner_window = 2048; +pub const tuner_lags = 1700; + +pub const TunerParams = extern struct { + offset: u32, + end: u32, + samples: u32, + rate: f32, + /// The pitch range searched. + min_freq: f32, + max_freq: f32, + /// YIN's absolute threshold on the normalised difference. + threshold: f32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// What tuner.comp writes (std430, read back by the CPU). +pub const Pitch = extern struct { + /// Hz; 0 when nothing periodic was found. + freq: f32, + /// 1 minus YIN's normalised difference at the period: 1 is perfectly + /// periodic. + clarity: f32, + /// RMS of the window, dBFS. + level: f32, + /// The period in samples, interpolated. + period: f32, +}; diff --git a/examples/micscope/src/main.zig b/examples/micscope/src/main.zig new file mode 100644 index 0000000..facc169 --- /dev/null +++ b/examples/micscope/src/main.zig @@ -0,0 +1,1498 @@ +//! micscope: records the microphone or opens sound files, plays them +//! forwards or backwards, and shows each as a waveform, its spectrogram on +//! the same time axis, and the spectrum at its playhead. The window is a +//! vertical strip: on top, what the microphone hears right now (or the take +//! being recorded); under it every take, newest first. The signal +//! processing (windowed FFTs, log-frequency interpolation, smoothing, +//! waveform reduction) and all the drawing run on the GPU; the CPU only +//! moves samples: microphone to tape, tape to the GPU once, tape to the +//! speakers. +//! +//! Memory by lifetime: State is the transport, the selection and the views, +//! changed by `update` from keys and the mouse, and replayable. Store holds +//! the takes, on disk: they are still there after a restart. Session is +//! this process's playback cursor, the microphone's recent past and GPU +//! objects. Transport changes, seeks and deletions are `sk.Request`s, which +//! `draw` carries out once each. +const std = @import("std"); +const sk = @import("sketch"); +const gpu = sk.abi.gpu; +const layouts = @import("layouts"); +const shaders = @import("shaders"); +const tuner = @import("tuner.zig"); + +pub const name = "micscope"; +pub const width = 1280; +pub const height = 900; +pub const commands = .{ "open", "takes" }; + +/// The longest take: ten minutes at the host's rate. +const max_seconds = 600; +const max_takes = 64; +/// The playback stream's key. +const stream = 1; +/// How far ahead playback keeps the stream filled. +const lead_seconds = 0.06; +const min_freq = 20.0; +const floor_db = -96.0; +/// The live view's span. +const live_seconds = 6; +/// The microphone's recent past, kept for the live view: a frame's new +/// columns and everything their analysis windows reach. +const live_history = 1 << 16; +/// How far spectrogram.comp reads past a column's centre: the long +/// analysis, 2048 points decimated 4×, three windows a quarter apart, plus +/// its low-pass. The live view analyses a column once that much has been +/// heard, so its columns are final; it runs this far behind. +const live_reach = 6160; + +pub const Transport = enum { stopped, recording, forward, backward }; + +/// The part of a take its timeline shows, as fractions of it. +pub const View = struct { + lo: f32 = 0, + hi: f32 = 1, +}; + +pub const State = struct { + transport: sk.Request(Transport) = .{}, + /// The direction space resumes in. + backward: bool = false, + /// To a fraction of the selected take. + seek: sk.Request(f32) = .{}, + scrubbing: bool = false, + dialog: sk.Request(void) = .{}, + /// Deletes the take at this index. + remove: sk.Request(u32) = .{}, + /// How many takes the Store holds. `update` keeps count as it records + /// and deletes; `draw` corrects it with the `takes` command if not. + takes: u32 = 0, + /// The take the transport and the keys work on, by index (oldest 0). + selected: u32 = 0, + views: [max_takes]View = @splat(.{}), + /// Maps the world (the panels, stacked) to the screen. + camera: sk.Camera = .{}, + /// The camera keeps the bottom, where now is, in view. + follow: bool = true, + panning: bool = false, + /// The pointer's height when panning last moved. + grab: f32 = 0, + gain: f32 = 1, + /// The tuner's reference pitch: A4, in Hz. + a4: f32 = 440, +}; + +pub const Take = struct { + /// Its samples: `Store.tape[start..][0..len]`. + start: usize = 0, + len: usize = 0, + /// Stays the same when earlier takes are deleted: GPU caches go by it. + id: u32 = 0, + /// Where the listener is, in samples into the take. + heard: f64 = 0, + /// A file's name, or empty for the microphone. + source: [64]u8 = @splat(0), + source_len: u32 = 0, +}; + +pub const Store = struct { + /// Every take's mono samples at `Input.sample_rate`, one after another, + /// oldest first. A recording grows the last one. + tape: std.ArrayList(f32) = .empty, + takes: std.ArrayList(Take) = .empty, + next_id: u32 = 1, +}; + +pub const Session = struct { + transport: sk.Handled = .{}, + seek: sk.Handled = .{}, + dialog: sk.Handled = .{}, + remove: sk.Handled = .{}, + /// The take playback follows, by id, and the next of its samples to + /// queue. + playing: u32 = 0, + cursor: usize = 0, + /// The take being recorded, by id. + recording: u32 = 0, + live: Live = .{}, + /// Pitch, by tuner.comp: of the microphone (live, or recording), and + /// of the selected take at its playhead. + tuners: [2]tuner.Tracker = @splat(.{}), + /// The take `tuners[1]` follows, by id. + tuned: u32 = 0, + gpu: Gpu = .{}, +}; + +/// Which tracker a readback belongs to: its index, as the download key. +const Tuned = enum(u32) { input, take }; + +/// The microphone's last `live_history` samples, and how far the live +/// view's columns have got. Column `k` covers samples `k * spp` up to the +/// next, counted from the first sample heard, and lands in slot +/// `k % columns`: a ring the views read from the oldest. +const Live = struct { + samples: [live_history]f32 = @splat(0), + /// Samples heard since the session started. + heard: u64 = 0, + /// The next column to analyse. + next: u64 = 0, + /// Columns analysed since the view was (re)made, up to `columns`. + filled: u32 = 0, + columns: u32 = 0, + spp: u32 = 0, + + fn push(live: *Live, heard: []const f32) void { + const h = &live.samples; + if (heard.len >= h.len) { + @memcpy(h, heard[heard.len - h.len ..]); + } else { + std.mem.copyForwards(f32, h[0 .. h.len - heard.len], h[heard.len..]); + @memcpy(h[h.len - heard.len ..], heard); + } + live.heard += heard.len; + } +}; + +/// GPU objects, made on first use. +const Gpu = struct { + ready: bool = false, + /// `Store.tape`, and how many of its samples are in it. + tape: sk.Buffer = .none, + capacity: usize = 0, + uploaded: usize = 0, + /// `Live.samples`. + live_tape: sk.Buffer = .none, + /// spectrum.comp's FFT, shared by every panel. + scratch: sk.Buffer = .none, + /// tuner.comp's results, a `layouts.Pitch` each, by `Tuned`, and the + /// difference functions it searches. + pitches: [2]sk.Buffer = @splat(.none), + differences: [2]sk.Buffer = @splat(.none), + live: Panel = .{}, + /// For the takes on screen, by take id; the least recently used is + /// reused. + panels: [16]Panel = @splat(.{}), +}; + +/// One panel's buffers and images, and the inputs they were last made from: +/// unchanged, they are not made again. +const Panel = struct { + id: u32 = 0, + used: u64 = 0, + wave_columns: sk.Buffer = .none, + spectrum_columns: sk.Buffer = .none, + levels: sk.Buffer = .none, + wave: sk.Image = .none, + wave_size: [2]u32 = .{ 0, 0 }, + spectrogram: sk.Image = .none, + spectrogram_size: [2]u32 = .{ 0, 0 }, + spectrum: sk.Image = .none, + spectrum_size: [2]u32 = .{ 0, 0 }, + wave_key: u64 = 0, + spectrogram_key: u64 = 0, + spectrum_key: u64 = 0, + wave_drawn: u64 = 0, + spectrogram_drawn: u64 = 0, + + fn forget(p: *Panel) void { + p.wave_key = 0; + p.spectrogram_key = 0; + p.spectrum_key = 0; + p.wave_drawn = 0; + p.spectrogram_drawn = 0; + } +}; + +// --------------------------------------------------------------------------- +// The world: panels stacked down in the order they were made, take 0 on top +// and the live microphone at the bottom, where now is. A take keeps its +// place as others are added. The camera maps the world to the screen: it +// follows the bottom unless moved, and zooms out to overview the strip. +// Panels are rendered at world resolution and scaled by the camera, so +// moving it never recomputes anything. + +const spectrogram_height = 132; +const min_zoom = 0.3; +/// The canvas follows the window, but no smaller than this. +const min_width = 320; +const min_height = 300; + +/// Sizes, from the canvas. Wide, a panel is a waveform over its spectrogram +/// with the spectrum beside them, and the buttons sit in the header. Narrow +/// (a phone), the spectrum goes under them, full width, and the buttons +/// along the bottom, where thumbs are, big enough to hit. +const Layout = struct { + w: f32, + h: f32, + narrow: bool, + /// World units are canvas pixels at zoom 1. + world_width: f32, + panel_height: f32, + /// Where the world shows, between the header and the footer or buttons. + view: sk.Rect, + + const header = 36; + const footer = 22; + const toolbar = 56; + const spectrum_width = 320; + + fn of(w: f32, h: f32) Layout { + const narrow = w < 760; + const margin: f32 = if (narrow) 8 else 16; + const bottom: f32 = if (narrow) toolbar else footer; + return .{ + .w = w, + .h = h, + .narrow = narrow, + .world_width = w - 2 * margin, + .panel_height = if (narrow) 412 else 236, + .view = .{ .x = 0, .y = header, .w = w, .h = h - header - bottom }, + }; + } + + /// The panel in world slot `k`. + fn box(l: Layout, k: u32) Box { + const y = @as(f32, @floatFromInt(k)) * l.panel_height; + const timeline_w = if (l.narrow) l.world_width else l.world_width - spectrum_width - 16; + const wave: sk.Rect = .{ .x = 0, .y = y + 18, .w = timeline_w, .h = 64 }; + const spectrogram: sk.Rect = .{ .x = 0, .y = wave.y + wave.h + 2, .w = timeline_w, .h = spectrogram_height }; + const timeline: sk.Rect = .{ .x = 0, .y = wave.y, .w = timeline_w, .h = spectrogram.y + spectrogram.h - wave.y }; + // Under the timeline and its time labels, or beside it; room for + // the frequency labels under it either way. + const levels: sk.Rect = if (l.narrow) + .{ .x = 0, .y = spectrogram.y + spectrogram.h + 22, .w = l.world_width, .h = 150 } + else + .{ .x = l.world_width - spectrum_width, .y = wave.y, .w = spectrum_width, .h = timeline.h - 14 }; + return .{ + .whole = .{ .x = 0, .y = y, .w = l.world_width, .h = l.panel_height }, + .wave = wave, + .spectrogram = spectrogram, + .timeline = timeline, + .spectrum = levels, + }; + } + + /// Button `i` of `buttons`: right to left in the header, or left to + /// right across the bottom. + fn button(l: Layout, i: usize) sk.Rect { + const n: f32 = @floatFromInt(buttons.len); + const at: f32 = @floatFromInt(i); + if (l.narrow) { + const w = (l.w - 8 - (n - 1) * 4) / n; + return .{ .x = 4 + (n - 1 - at) * (w + 4), .y = l.h - toolbar + 6, .w = w, .h = toolbar - 10 }; + } + return .{ .x = l.w - 16 - (at + 1) * (button_width + 6) + 6, .y = 5, .w = button_width, .h = 26 }; + } + + fn buttonAt(l: Layout, x: f32, y: f32) ?Action { + for (buttons, 0..) |a, i| if (l.button(i).contains(x, y)) return a; + return null; + } +}; + +const Box = struct { + whole: sk.Rect, + wave: sk.Rect, + spectrogram: sk.Rect, + /// Waveform and spectrogram: one time axis, where clicks seek and + /// ctrl+wheel (or a pinch) zooms. + timeline: sk.Rect, + spectrum: sk.Rect, + + fn toScreen(b: Box, cam: sk.Camera) Box { + return .{ + .whole = cam.toScreen(b.whole), + .wave = cam.toScreen(b.wave), + .spectrogram = cam.toScreen(b.spectrogram), + .timeline = cam.toScreen(b.timeline), + .spectrum = cam.toScreen(b.spectrum), + }; + } +}; + +const World = struct { + /// Slots `0..takes` are takes; slot `takes` is the live view, unless a + /// recording (the last take) stands in for it. + takes: u32, + live: bool, + + fn of(s: *const State) World { + return .{ .takes = s.takes, .live = s.transport.value != .recording }; + } + fn slots(w: World) u32 { + return w.takes + @intFromBool(w.live); + } + fn height(w: World, l: Layout) f32 { + return @as(f32, @floatFromInt(w.slots())) * l.panel_height; + } + /// The take at world point `p`. + fn takeAt(w: World, l: Layout, p: sk.shape.Point) ?u32 { + if (p.y < 0 or p.x < 0 or p.x > l.world_width) return null; + const k: u32 = @intFromFloat(@floor(p.y / l.panel_height)); + return if (k < w.takes) k else null; + } +}; + +/// The camera's vertical range: from the world's top at the viewport's top +/// to its bottom at the viewport's bottom. +fn cameraRange(world: World, scale: f32, l: Layout) [2]f32 { + const highest = l.view.y + 4; + const lowest = l.view.y + l.view.h - 4 - world.height(l) * scale; + return .{ @min(lowest, highest), highest }; +} + +/// The camera as drawn: centred across, inside the world, and on the +/// bottom while following. +fn cameraOf(s: *const State, l: Layout) sk.Camera { + var cam = s.camera; + cam.zoom = std.math.clamp(cam.zoom, min_zoom, 1); + cam.x = @round((l.w - l.world_width * cam.zoom) / 2); + const range = cameraRange(World.of(s), cam.zoom, l); + cam.y = @round(if (s.follow) range[0] else std.math.clamp(cam.y, range[0], range[1])); + return cam; +} + +/// Moves the camera, which follows again once it reaches the bottom. +fn moveCamera(s: *State, cam: sk.Camera, in: *const sk.Input) void { + const l = Layout.of(in.width, in.height); + s.camera = cam; + s.follow = false; + s.camera = cameraOf(s, l); + const range = cameraRange(World.of(s), s.camera.zoom, l); + s.follow = s.camera.y <= range[0] + 0.5; +} + +// --------------------------------------------------------------------------- +// Update: transport, selection, views and camera, from input alone. + +pub fn update(s: *State, in: *const sk.Input) void { + if (in.started) { + // Nothing resumes on its own after a restart; the takes are kept. + s.transport.value = .stopped; + s.scrubbing = false; + s.panning = false; + } + const recording = s.transport.value == .recording; + for (keys) |k| if (in.pressed(k[0])) act(s, k[1], in); + if (in.pressed(sk.Key.equals)) s.gain = @min(s.gain * 1.25, 64); + if (in.pressed(sk.Key.minus)) s.gain = @max(s.gain / 1.25, 0.25); + if (in.pressed(sk.Key.z) and s.selected < max_takes) s.views[s.selected] = .{}; + if (!recording and s.takes > 0) { + // Up the strip is older. + if (in.pressed(sk.Key.up)) select(s, s.selected -| 1, in); + if (in.pressed(sk.Key.down)) select(s, @min(s.selected + 1, s.takes - 1), in); + } + const l = Layout.of(in.width, in.height); + const page = l.view.h - l.panel_height / 2; + if (in.pressed(sk.Key.page_down)) panBy(s, -page, in); + if (in.pressed(sk.Key.page_up)) panBy(s, page, in); + + const view = l.view; + for (in.events) |e| { + const cam = cameraOf(s, l); + const world = World.of(s); + const p = cam.toWorld(sk.shape.Point{ .x = e.x, .y = e.y }); + switch (e.kind) { + .mouse_down => if (e.code == 1 and l.buttonAt(e.x, e.y) != null) { + act(s, l.buttonAt(e.x, e.y).?, in); + } else if (e.code == 2 or e.code == 3) { + // Middle or right drag pans. + s.panning = true; + s.grab = e.y; + } else if (e.code == 1 and !recording and view.contains(e.x, e.y)) { + const index = world.takeAt(l, p) orelse continue; + select(s, index, in); + const b = l.box(index); + if (b.timeline.contains(p.x, p.y)) { + s.scrubbing = true; + seekTo(s, fractionAt(s.views[index], b.timeline, p.x)); + } + }, + .mouse_move => if (s.panning) { + panBy(s, e.y - s.grab, in); + s.grab = e.y; + } else if (s.scrubbing and s.selected < world.takes) { + seekTo(s, fractionAt(s.views[s.selected], l.box(s.selected).timeline, p.x)); + }, + .mouse_up, .cancel => { + s.scrubbing = false; + s.panning = false; + }, + // Ctrl+wheel, or a pinch (which browsers and the web host send + // as ctrl+wheel), zooms what is under it: a take's time on its + // timeline, the view anywhere else. + .wheel => if (view.contains(e.x, e.y)) { + const ctrl = e.modifiers() & sk.Key.Mod.ctrl != 0; + const over = world.takeAt(l, p); + if (ctrl and over != null and l.box(over.?).timeline.contains(p.x, p.y)) { + const index = over.?; + zoom(&s.views[index], fractionAt(s.views[index], l.box(index).timeline, p.x), std.math.pow(f32, 0.8, e.wheel())); + } else if (ctrl) { + var moved = cam; + moved.zoomAt(sk.shape.Point{ .x = e.x, .y = e.y }, e.wheel(), .{ .min = min_zoom, .max = 1 }); + moveCamera(s, moved, in); + } else panBy(s, e.wheel() * 60, in); + }, + else => {}, + } + } +} + +/// What keys and the buttons on top do. +const Action = enum { record, play, forward, backward, start, end, open, delete, overview, a4_down, a4_up }; + +const keys = [_]struct { sk.Key, Action }{ + .{ .r, .record }, .{ .space, .play }, .{ .right, .forward }, .{ .left, .backward }, + .{ .home, .start }, .{ .end, .end }, .{ .o, .open }, .{ .delete, .delete }, + .{ .backspace, .delete }, .{ .@"0", .overview }, .{ .left_bracket, .a4_down }, .{ .right_bracket, .a4_up }, +}; + +/// Everything a touch screen cannot reach by key, as buttons (see +/// `Layout.button`). Labels change with what they would do. +const buttons = [_]Action{ .a4_up, .a4_down, .overview, .delete, .open, .backward, .play, .record }; +const button_width = 52; + +fn act(s: *State, a: Action, in: *const sk.Input) void { + const now = s.transport.value; + switch (a) { + .record => if (now == .recording) change(s, .stopped) else record(s), + .play => change(s, switch (now) { + .forward, .backward => .stopped, + .stopped, .recording => if (s.backward) .backward else .forward, + }), + .forward => change(s, .forward), + .backward => change(s, .backward), + .start => seekTo(s, 0), + .end => seekTo(s, 1), + .open => s.dialog.send({}), + .delete => if (now != .recording and s.takes > 0) remove(s, in), + .overview => { + // The whole strip, or back to full size. + const l = Layout.of(in.width, in.height); + var cam = cameraOf(s, l); + cam.zoom = if (cam.zoom < 1) 1 else l.view.h / @max(World.of(s).height(l), 1); + moveCamera(s, cam, in); + s.follow = true; + }, + .a4_down => s.a4 = @max(s.a4 - 1, 400), + .a4_up => s.a4 = @min(s.a4 + 1, 480), + } +} + +/// Starts a new take, unless the strip is full. It takes the live view's +/// place, which the camera follows. +fn record(s: *State) void { + if (s.takes >= max_takes) return; + s.selected = s.takes; + s.takes += 1; + s.views[s.selected] = .{}; + s.follow = true; + change(s, .recording); +} + +fn change(s: *State, to: Transport) void { + if (to == .forward or to == .backward) s.backward = to == .backward; + s.transport.send(to); +} + +fn seekTo(s: *State, fraction: f32) void { + if (s.transport.value == .recording) return; + s.seek.send(std.math.clamp(fraction, 0, 1)); +} + +/// Selects a take and brings it into view. +fn select(s: *State, index: u32, in: *const sk.Input) void { + s.selected = index; + const l = Layout.of(in.width, in.height); + var cam = cameraOf(s, l); + const b = cam.toScreen(l.box(index).whole); + const view = l.view; + if (b.y < view.y) cam.y += view.y - b.y + 4; + if (b.y + b.h > view.y + view.h) cam.y -= b.y + b.h - (view.y + view.h) + 4; + moveCamera(s, cam, in); +} + +/// Moves the world down by `pixels` on screen. +fn panBy(s: *State, pixels: f32, in: *const sk.Input) void { + var cam = cameraOf(s, Layout.of(in.width, in.height)); + cam.y += pixels; + moveCamera(s, cam, in); +} + +/// Deletes the selected take and selects the one above it. +fn remove(s: *State, in: *const sk.Input) void { + const index = @min(s.selected, s.takes - 1); + s.remove.send(index); + std.mem.copyForwards(View, s.views[index .. s.takes - 1], s.views[index + 1 .. s.takes]); + s.takes -= 1; + s.selected = index -| 1; + moveCamera(s, cameraOf(s, Layout.of(in.width, in.height)), in); +} + +fn fractionAt(v: View, timeline: sk.Rect, x: f32) f32 { + return v.lo + (x - timeline.x) / timeline.w * (v.hi - v.lo); +} + +/// Scales the view by `factor` around `at`, keeping it inside the take. +fn zoom(v: *View, at: f32, factor: f32) void { + const span = std.math.clamp((v.hi - v.lo) * factor, 1e-6, 1); + var lo = at - (at - v.lo) / (v.hi - v.lo) * span; + lo = std.math.clamp(lo, 0, 1 - span); + v.lo = lo; + v.hi = lo + span; +} + +/// `open PATH` adds a file as a take; `open` alone shows the dialog. +/// `takes N` tells State how many takes the Store holds (`draw` sends it). +pub fn command(s: *State, store: *Store, c: *sk.Console, line: []const u8) !bool { + var words = std.mem.tokenizeAny(u8, line, " \t"); + const verb = words.next() orelse return false; + if (std.mem.eql(u8, verb, "takes")) { + const n = try std.fmt.parseInt(u32, words.next() orelse return error.ExpectedCount, 10); + s.takes = @min(n, max_takes); + s.selected = @min(s.selected, s.takes -| 1); + return true; + } + if (!std.mem.eql(u8, verb, "open")) return false; + const path = std.mem.trim(u8, words.rest(), " \t"); + if (path.len == 0) { + s.dialog.send({}); + return true; + } + if (store.takes.items.len >= max_takes) return error.TooManyTakes; + const samples = c.host.samples(c.host.loadSound(path)); + if (samples.len == 0) return error.CannotOpen; + // Recording grows the last take: stop before this one goes after it. + if (s.transport.value == .recording) change(s, .stopped); + const kept = samples[0..@min(samples.len, max_seconds * c.in.sample_rate)]; + var take: Take = .{ .start = store.tape.items.len, .len = kept.len, .id = store.next_id }; + const base = std.fs.path.basename(path); + take.source_len = @intCast(@min(base.len, take.source.len)); + @memcpy(take.source[0..take.source_len], base[0..take.source_len]); + try store.takes.ensureUnusedCapacity(c.store, 1); + try store.tape.appendSlice(c.store, kept); + store.takes.appendAssumeCapacity(take); + store.next_id += 1; + s.takes = @intCast(store.takes.items.len); + s.selected = s.takes - 1; + s.views[s.selected] = .{}; + s.follow = true; + change(s, .stopped); + c.print("opened {s}: {d:.1} s", .{ path, @as(f64, @floatFromInt(kept.len)) / @as(f64, @floatFromInt(c.in.sample_rate)) }); + return true; +} + +// --------------------------------------------------------------------------- +// Draw: the takes, the microphone, the speakers and the GPU. + +pub fn draw(s: *const State, ss: *Session, store: *Store, f: *sk.Frame) void { + const in = f.in; + const rate: f64 = @floatFromInt(in.sample_rate); + const dt: f64 = @min(@as(f64, @floatFromInt(in.dt)) / std.time.ns_per_s, 0.1); + // As big as the window: the layout follows (see `Layout`). + f.canvas(@intFromFloat(@max(in.window_width, min_width)), @intFromFloat(@max(in.window_height, min_height))); + f.clear(0x0b0c0f); + + adoptTape(store, f.store); + if (ss.dialog.take(s.dialog) != null) f.openFile("open", .{ .pattern = "wav;flac;ogg" }); + if (ss.remove.take(s.remove)) |index| removeTake(ss, store, index); + + // Requests from State. Any of them, or another take selected, flushes + // what is queued, so the cursor restarts from what was heard. + const transport = ss.transport.take(s.transport); + const seek = ss.seek.take(s.seek); + // Recording goes by the transport's value rather than the request, so a + // fresh session that only caught up with the request still records. + if (s.transport.value != .recording) { + ss.recording = 0; + } else if (store.takes.items.len == 0 or store.takes.items[store.takes.items.len - 1].id != ss.recording) { + startTake(ss, store, f.store); + } + const takes = store.takes.items; + if (s.takes != takes.len) f.run(f.format("takes {d}", .{takes.len})); + const selected: ?*Take = if (takes.len > 0) &takes[@min(s.selected, takes.len - 1)] else null; + var queued: usize = in.stream(stream).queued; + if (selected) |t| { + if (transport != null or seek != null or t.id != ss.playing) { + f.audio(stream, &.{}, .{ .clear = true }); + queued = 0; + ss.playing = t.id; + ss.cursor = @intFromFloat(std.math.clamp(@round(t.heard), 0, @as(f64, @floatFromInt(t.len)))); + } + if (transport) |to| { + switch (to) { + .forward => if (ss.cursor >= t.len) { + ss.cursor = 0; + }, + .backward => if (ss.cursor == 0) { + ss.cursor = t.len; + }, + .stopped, .recording => {}, + } + t.heard = @floatFromInt(ss.cursor); + } + if (seek) |fraction| if (s.transport.value != .recording) { + ss.cursor = @intFromFloat(@round(fraction * @as(f32, @floatFromInt(t.len)))); + t.heard = @floatFromInt(ss.cursor); + }; + } + + // The microphone is always open: the live view shows it, and a + // recording keeps it. + f.microphone(); + ss.live.push(in.microphone); + switch (s.transport.value) { + .recording => if (takes.len > 0 and takes[takes.len - 1].id == ss.recording) { + const t = &takes[takes.len - 1]; + const room = max_seconds * in.sample_rate -| t.len; + const heard = in.microphone[0..@min(in.microphone.len, room)]; + store.tape.appendSlice(f.store, heard) catch f.host.log("the store is full: recording stopped"); + t.len = store.tape.items.len - t.start; + t.heard = @floatFromInt(t.len); + }, + .forward, .backward => if (selected) |t| play(ss, t, store.tape.items[t.start..][0..t.len], f, s.transport.value == .forward, queued, rate, dt), + .stopped => {}, + } + + // The tuner's readbacks, a frame or two old. + if (selected) |t| if (t.id != ss.tuned) { + ss.tuned = t.id; + ss.tuners[@intFromEnum(Tuned.take)] = .{}; + }; + for (in.downloads) |d| { + if (d.key >= ss.tuners.len or d.bytes.len < @sizeOf(layouts.Pitch)) continue; + ss.tuners[d.key].feed(std.mem.bytesToValue(layouts.Pitch, d.bytes[0..@sizeOf(layouts.Pitch)]), s.a4, @floatCast(dt)); + } + + if (!f.host.gpu()) { + f.text(16, Layout.header, 2, "micscope needs a GPU: the host has none.", 0xff8080); + return; + } + render(s, ss, store, f, rate, dt); +} + +/// Stores from before the strip held one tape: it becomes the first take. +fn adoptTape(store: *Store, a: std.mem.Allocator) void { + if (store.takes.items.len > 0 or store.tape.items.len == 0) return; + store.takes.append(a, .{ .len = store.tape.items.len, .id = store.next_id }) catch return; + store.next_id += 1; +} + +fn startTake(ss: *Session, store: *Store, a: std.mem.Allocator) void { + store.takes.append(a, .{ .start = store.tape.items.len, .id = store.next_id }) catch { + return; + }; + ss.recording = store.next_id; + store.next_id += 1; +} + +fn removeTake(ss: *Session, store: *Store, index: u32) void { + if (index >= store.takes.items.len) return; + const t = store.takes.orderedRemove(index); + const tape = store.tape.items; + std.mem.copyForwards(f32, tape[t.start .. tape.len - t.len], tape[t.start + t.len ..]); + store.tape.shrinkRetainingCapacity(tape.len - t.len); + for (store.takes.items[index..]) |*later| later.start -= t.len; + ss.gpu.uploaded = @min(ss.gpu.uploaded, t.start); + for (&ss.gpu.panels) |*p| if (p.id == t.id) { + p.id = 0; + p.forget(); + }; +} + +/// Keeps the stream `lead_seconds` ahead, reading the take either way, and +/// follows the listener. +fn play(ss: *Session, t: *Take, tape: []const f32, f: *sk.Frame, forward: bool, queued: usize, rate: f64, dt: f64) void { + const lead: usize = @intFromFloat(rate * lead_seconds); + const want = lead -| queued; + // What is heard now: the cursor, less what is queued in front of it. + const measured: f64 = @floatFromInt(if (forward) ss.cursor -| queued else @min(ss.cursor + queued, tape.len)); + if (forward) { + const n = @min(want, tape.len -| ss.cursor); + f.audio(stream, tape[@min(ss.cursor, tape.len)..][0..n], .{}); + ss.cursor += n; + } else { + const n = @min(want, ss.cursor); + const reversed = f.scratch.alloc(f32, n) catch return; + for (reversed, 0..) |*sample, i| sample.* = tape[ss.cursor - 1 - i]; + f.audio(stream, reversed, .{}); + ss.cursor -= n; + } + // The queue drains in device-sized blocks: advance by the clock and + // pull gently towards the measurement, snapping when far off. + const direction: f64 = if (forward) 1 else -1; + t.heard += direction * dt * rate; + const off = measured - t.heard; + t.heard = if (@abs(off) > rate * 0.25) measured else t.heard + off * 0.15; + t.heard = std.math.clamp(t.heard, 0, @as(f64, @floatFromInt(tape.len))); +} + +// --------------------------------------------------------------------------- +// GPU: buffers, compute, passes. + +const view_pipeline = struct { + fn make(fragment: []const u8) gpu.Pipeline { + return .{ + .vertex = .{ .code = shaders.view_vert }, + .fragment = .{ .code = fragment, .uniform_buffers = 1, .storage_buffers = 1 }, + .stride = 0, + .attributes = &.{}, + .blend = .none, + }; + } +}; +const wave_pipelines = [_]gpu.Pipeline{view_pipeline.make(shaders.waveform_frag)}; +const spectrum_pipelines = [_]gpu.Pipeline{view_pipeline.make(shaders.spectrum_frag)}; +const spectrogram_pipelines = [_]gpu.Pipeline{view_pipeline.make(shaders.spectrogram_frag)}; + +const spectrum_kernel: gpu.Kernel = .{ + .code = shaders.spectrum_comp, + .readonly_buffers = 1, + .readwrite_buffers = 2, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const spectrogram_kernel: gpu.Kernel = .{ + .code = shaders.spectrogram_comp, + .readonly_buffers = 1, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const tuner_difference_kernel: gpu.Kernel = .{ + .code = shaders.tuner_difference_comp, + .readonly_buffers = 1, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const tuner_kernel: gpu.Kernel = .{ + .code = shaders.tuner_comp, + .readonly_buffers = 2, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 256, 1, 1 }, +}; +const waveform_kernel: gpu.Kernel = .{ + .code = shaders.waveform_comp, + .readonly_buffers = 1, + .readwrite_buffers = 1, + .uniform_buffers = 1, + .threads = .{ 64, 1, 1 }, +}; + +fn setup(g: *Gpu, f: *sk.Frame) void { + if (g.ready) return; + g.ready = true; + g.scratch = f.host.newBuffer(layouts.fft_size * 2 * @sizeOf([2]f32)); + g.live_tape = f.host.newBuffer(live_history * @sizeOf(f32)); + for (&g.pitches) |*b| b.* = f.host.newBuffer(@sizeOf(layouts.Pitch)); + for (&g.differences) |*b| b.* = f.host.newBuffer(layouts.tuner_lags * @sizeOf(f32)); +} + +fn makeBuffers(p: *Panel, f: *sk.Frame) void { + if (p.wave_columns != .none) return; + p.wave_columns = f.host.newBuffer(layouts.max_columns * @sizeOf([4]f32)); + p.spectrum_columns = f.host.newBuffer(layouts.max_columns * @sizeOf([2]f32)); + p.levels = f.host.newBuffer(layouts.max_columns * spectrogram_height * @sizeOf(f32)); +} + +/// The panel for take `id`: the one it had, or the least recently used. +fn panelFor(g: *Gpu, f: *sk.Frame, id: u32) *Panel { + const p = for (&g.panels) |*p| { + if (p.id == id) break p; + } else reuse: { + var oldest = &g.panels[0]; + for (&g.panels) |*p| if (p.used < oldest.used) { + oldest = p; + }; + oldest.id = id; + oldest.forget(); + break :reuse oldest; + }; + p.used = f.in.frame; + makeBuffers(p, f); + return p; +} + +/// An image of `size`, (re)made when the size changes. +fn target(f: *sk.Frame, image: *sk.Image, size: *[2]u32, r: sk.Rect) [2]u32 { + const want: [2]u32 = .{ @intFromFloat(@min(r.w, layouts.max_columns)), @intFromFloat(@min(r.h, spectrogram_height)) }; + if (image.* == .none or size[0] != want[0] or size[1] != want[1]) { + if (image.* != .none) f.host.freeImage(image.*); + image.* = f.host.newImage(want[0], want[1]); + size.* = want; + } + return want; +} + +/// Puts the tape on the GPU: new samples only, unless the buffer had to grow. +fn sync(g: *Gpu, f: *sk.Frame, tape: []const f32) void { + if (tape.len > g.capacity or g.tape == .none) { + if (g.tape != .none) f.host.freeBuffer(g.tape); + g.capacity = @max(tape.len, g.capacity * 2, 60 * f.in.sample_rate); + g.tape = f.host.newBuffer(@intCast(g.capacity * @sizeOf(f32))); + g.uploaded = 0; + } + g.uploaded = @min(g.uploaded, tape.len); + if (g.uploaded < tape.len) { + f.upload(g.tape, @intCast(g.uploaded * @sizeOf(f32)), std.mem.sliceAsBytes(tape[g.uploaded..])); + g.uploaded = tape.len; + } +} + +fn render(s: *const State, ss: *Session, store: *Store, f: *sk.Frame, rate: f64, dt: f64) void { + const g = &ss.gpu; + setup(g, f); + sync(g, f, store.tape.items); + // New code may mean new shaders: make everything again. + if (f.in.reloaded) { + g.live.forget(); + for (&g.panels) |*p| p.forget(); + } + const takes = store.takes.items; + const selected = if (takes.len > 0) @min(s.selected, takes.len - 1) else 0; + const recording = s.transport.value == .recording; + const world: World = .{ .takes = @intCast(takes.len), .live = !recording }; + const l = Layout.of(f.in.width, f.in.height); + const cam = cameraOf(s, l); + const view = l.view; + // Images are made at world size; the camera only scales them. + const size = l.box(0); + + // The live view keeps up even while it is hidden. + analyseLive(ss, f, size, rate); + // The tuner: on the microphone's newest sound, and on the selected take + // around its playhead. + const reach = layouts.tuner_window + layouts.tuner_lags; + if (recording and takes.len > 0) { + const t = takes[takes.len - 1]; + tune(g, f, .input, g.tape, t.start, t.len, t.len, rate); + } else tune(g, f, .input, g.live_tape, 0, live_history, live_history, rate); + if (!recording and takes.len > 0) { + const t = takes[selected]; + const end: usize = @intFromFloat(@max(t.heard, 0)); + tune(g, f, .take, g.tape, t.start, t.len, end + reach / 2, rate); + } + f.clip(view); + for (0..world.slots()) |slot| { + const k: u32 = @intCast(slot); + const b = l.box(k).toScreen(cam); + if (b.whole.y + b.whole.h <= view.y or b.whole.y >= view.y + view.h) continue; + if (k == world.takes) { + drawLive(s, ss, f, size, b, rate, dt); + } else { + const tint: layouts.Tint = if (recording and k + 1 == world.takes) .recording else .take; + drawTake(s, g, f, size, b, &takes[k], s.views[k], k == selected, tint, rate, dt); + } + } + f.clip(null); + labels(s, ss, store, f, world, cam, rate); +} + +/// Finds the pitch of the `tuner_window` samples before `end` (of the +/// `samples` from `offset`), and reads it back. +fn tune(g: *Gpu, f: *sk.Frame, which: Tuned, tape: sk.Buffer, offset: usize, samples: usize, end: usize, rate: f64) void { + const result = g.pitches[@intFromEnum(which)]; + const difference = g.differences[@intFromEnum(which)]; + const params = gpu.uniform(f.scratch, layouts.TunerParams{ + .offset = @intCast(offset), + .end = @intCast(end), + .samples = @intCast(samples), + .rate = @floatCast(rate), + .min_freq = tuner_low, + .max_freq = tuner_high, + .threshold = 0.12, + }); + f.compute(.{ + .label = "tuner_difference.comp", + .kernel = tuner_difference_kernel, + .readonly = &.{tape}, + .readwrite = &.{difference}, + .uniforms = params, + // Match tuner_difference.comp's LAGS_PER_GROUP. + .groups = .{ (layouts.tuner_lags + 15) / 16, 1, 1 }, + }); + f.compute(.{ + .label = "tuner.comp", + .kernel = tuner_kernel, + .readonly = &.{ tape, difference }, + .readwrite = &.{result}, + .uniforms = params, + }); + f.download(result, .{ .size = @sizeOf(layouts.Pitch), .key = @intFromEnum(which) }); +} + +/// The pitches the tuner looks for: a five-string bass's low B to a +/// piccolo's top. +const tuner_low = 30.0; +const tuner_high = 4200.0; + +fn wavePass(f: *sk.Frame, p: *Panel, size: [2]u32, view: layouts.WaveView) void { + f.gpu(.{ + .label = "waveform.frag", + .image = p.wave, + .width = size[0], + .height = size[1], + .clear = null, + .pipelines = &wave_pipelines, + .draws = &.{.{ .pipeline = 0, .vertex_count = 3, .fragment_uniforms = gpu.uniform(f.scratch, view), .fragment_storage = &.{p.wave_columns} }}, + }); +} + +fn spectrogramPass(f: *sk.Frame, p: *Panel, size: [2]u32, view: layouts.SpectrogramView) void { + f.gpu(.{ + .label = "spectrogram.frag", + .image = p.spectrogram, + .width = size[0], + .height = size[1], + .clear = null, + .pipelines = &spectrogram_pipelines, + .draws = &.{.{ .pipeline = 0, .vertex_count = 3, .fragment_uniforms = gpu.uniform(f.scratch, view), .fragment_storage = &.{p.levels} }}, + }); +} + +/// The spectrum's analysis of `tape`, then its pass. +fn spectrum(g: *Gpu, f: *sk.Frame, tape: sk.Buffer, p: *Panel, size: [2]u32, params: layouts.SpectrumParams, rate: f64) void { + f.compute(.{ + .label = "spectrum.comp", + .kernel = spectrum_kernel, + .readonly = &.{tape}, + .readwrite = &.{ g.scratch, p.spectrum_columns }, + .uniforms = gpu.uniform(f.scratch, params), + }); + f.gpu(.{ + .label = "spectrum.frag", + .image = p.spectrum, + .width = size[0], + .height = size[1], + .clear = null, + .pipelines = &spectrum_pipelines, + .draws = &.{.{ + .pipeline = 0, + .vertex_count = 3, + .fragment_uniforms = gpu.uniform(f.scratch, layouts.SpectrumView{ + .size = .{ @floatFromInt(size[0]), @floatFromInt(size[1]) }, + .floor_db = floor_db, + .ceil_db = 0, + .min_freq = min_freq, + .max_freq = maxFreq(rate), + }), + .fragment_storage = &.{p.spectrum_columns}, + }}, + }); +} + +fn maxFreq(rate: f64) f32 { + return @floatCast(@min(20_000, rate / 2)); +} + +fn drawTake(s: *const State, g: *Gpu, f: *sk.Frame, size: Box, b: Box, t: *const Take, view: View, selected: bool, tint: layouts.Tint, rate: f64, dt: f64) void { + const p = panelFor(g, f, t.id); + + // Waveform columns, when what they show changed. + const wave = target(f, &p.wave, &p.wave_size, size.wave); + const span = visible(view, t.len, wave[0]); + const first: u32 = @intFromFloat(@floor(span[0])); + const spp: f32 = @floatCast((span[1] - span[0]) / @as(f64, @floatFromInt(wave[0]))); + const params: layouts.WaveParams = .{ + .offset = @intCast(t.start), + .first = first, + .samples = @intCast(t.len), + .columns = wave[0], + .spp = spp, + .slot = 0, + .ring = layouts.max_columns, + }; + const key = std.hash.Wyhash.hash(0, std.mem.asBytes(¶ms)); + if (key != p.wave_key) { + p.wave_key = key; + f.compute(.{ + .label = "waveform.comp", + .kernel = waveform_kernel, + .readonly = &.{g.tape}, + .readwrite = &.{p.wave_columns}, + .uniforms = gpu.uniform(f.scratch, params), + .groups = .{ wave[0], 1, 1 }, + }); + } + + // The spectrogram over the same columns. + const spectrogram = target(f, &p.spectrogram, &p.spectrogram_size, size.spectrogram); + const analysis: layouts.SpectrogramParams = .{ + .offset = @intCast(t.start), + .first = first, + .samples = @intCast(t.len), + .columns = spectrogram[0], + .rows = spectrogram[1], + .spp = spp, + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .slot = 0, + .ring = layouts.max_columns, + }; + const analysis_key = std.hash.Wyhash.hash(0, std.mem.asBytes(&analysis)); + if (analysis_key != p.spectrogram_key) { + p.spectrogram_key = analysis_key; + f.compute(.{ + .label = "spectrogram.comp", + .kernel = spectrogram_kernel, + .readonly = &.{g.tape}, + .readwrite = &.{p.levels}, + .uniforms = gpu.uniform(f.scratch, analysis), + .groups = .{ spectrogram[0], 1, 1 }, + }); + } + + // The spectrum around the playhead, or of the latest while recording: + // every frame for the selected take, which plays and glides; the others + // only when it would change, and without smoothing. + const half: f64 = layouts.window / 2; + const len: f64 = @floatFromInt(t.len); + const end: f64 = if (tint == .recording) len else @min(t.heard + half, len + half); + const shown = target(f, &p.spectrum, &p.spectrum_size, size.spectrum); + var sparams: layouts.SpectrumParams = .{ + .offset = @intCast(t.start), + .end = @intFromFloat(@max(end, 0)), + .samples = @intCast(t.len), + .columns = shown[0], + .reset = 0, + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .dt = 0, + }; + const spectrum_key = std.hash.Wyhash.hash(1, std.mem.asBytes(&sparams)); + if (selected or spectrum_key != p.spectrum_key) { + sparams.reset = @intFromBool(!selected or p.spectrum_key == 0); + sparams.dt = @floatCast(dt); + p.spectrum_key = spectrum_key; + spectrum(g, f, g.tape, p, shown, sparams, rate); + } + + const at: f32 = @floatCast((t.heard - span[0]) / (span[1] - span[0]) * @as(f64, @floatFromInt(wave[0]))); + const playhead: f32 = if (!selected or t.len == 0 or at < 0 or at > @as(f32, @floatFromInt(wave[0]))) -1 else at; + const wave_view: layouts.WaveView = .{ + .size = .{ @floatFromInt(wave[0]), @floatFromInt(wave[1]) }, + .playhead = playhead, + .gain = s.gain, + .tint = @intFromEnum(tint), + .scroll = 0, + .filled = wave[0], + }; + const wave_drawn = std.hash.Wyhash.hash(p.wave_key, std.mem.asBytes(&wave_view)); + if (wave_drawn != p.wave_drawn) { + p.wave_drawn = wave_drawn; + wavePass(f, p, wave, wave_view); + } + const spectrogram_view: layouts.SpectrogramView = .{ + .size = .{ @floatFromInt(spectrogram[0]), @floatFromInt(spectrogram[1]) }, + .floor_db = floor_db, + .ceil_db = -6, + .playhead = playhead, + .scroll = 0, + .filled = spectrogram[0], + }; + const spectrogram_drawn = std.hash.Wyhash.hash(p.spectrogram_key, std.mem.asBytes(&spectrogram_view)); + if (spectrogram_drawn != p.spectrogram_drawn) { + p.spectrogram_drawn = spectrogram_drawn; + spectrogramPass(f, p, spectrogram, spectrogram_view); + } + f.image(p.wave, b.wave, .{}); + f.image(p.spectrogram, b.spectrogram, .{}); + f.image(p.spectrum, b.spectrum, .{}); +} + +/// Analyses the live view's new columns into its rings: only those whose +/// windows have been heard in full, so each is analysed once. +fn analyseLive(ss: *Session, f: *sk.Frame, b: Box, rate: f64) void { + const live = &ss.live; + const g = &ss.gpu; + const p = &g.live; + makeBuffers(p, f); + const columns: u32 = @intFromFloat(@min(b.wave.w, layouts.max_columns)); + const spp: u32 = @max(1, @as(u32, @intFromFloat(@round(live_seconds * rate / @as(f64, @floatFromInt(columns)))))); + if (columns != live.columns or spp != live.spp or f.in.reloaded) { + live.columns = columns; + live.spp = spp; + live.filled = 0; + live.next = 0; + } + if (f.in.microphone.len > 0) f.upload(g.live_tape, 0, std.mem.sliceAsBytes(&live.samples)); + + // History index of heard sample `i`: `i - base`. A column is ready once + // all it reads is heard, and analysable while all it reads is kept. + const base: i64 = @as(i64, @intCast(live.heard)) - live_history; + const ready = (live.heard -| live_reach) / spp; + const kept: u64 = @intCast(@max(0, @divFloor(base + live_reach + spp - 1, spp))); + live.next = @max(live.next, kept, ready -| columns); + if (ready <= live.next) return; + const count: u32 = @intCast(ready - live.next); + const first: u32 = @intCast(@as(i64, @intCast(live.next * spp)) - base); + const slot: u32 = @intCast(live.next % columns); + f.compute(.{ + .label = "waveform.comp live", + .kernel = waveform_kernel, + .readonly = &.{g.live_tape}, + .readwrite = &.{p.wave_columns}, + .uniforms = gpu.uniform(f.scratch, layouts.WaveParams{ + .offset = 0, + .first = first, + .samples = live_history, + .columns = count, + .spp = @floatFromInt(spp), + .slot = slot, + .ring = columns, + }), + .groups = .{ count, 1, 1 }, + }); + f.compute(.{ + .label = "spectrogram.comp live", + .kernel = spectrogram_kernel, + .readonly = &.{g.live_tape}, + .readwrite = &.{p.levels}, + .uniforms = gpu.uniform(f.scratch, layouts.SpectrogramParams{ + .offset = 0, + .first = first, + .samples = live_history, + .columns = count, + .rows = @intFromFloat(@min(b.spectrogram.h, spectrogram_height)), + .spp = @floatFromInt(spp), + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .slot = slot, + .ring = columns, + }), + .groups = .{ count, 1, 1 }, + }); + live.filled = @intCast(@min(columns, live.filled + count)); + live.next = ready; +} + +fn drawLive(s: *const State, ss: *Session, f: *sk.Frame, size: Box, b: Box, rate: f64, dt: f64) void { + const live = &ss.live; + const g = &ss.gpu; + const p = &g.live; + const wave = target(f, &p.wave, &p.wave_size, size.wave); + const spectrogram = target(f, &p.spectrogram, &p.spectrogram_size, size.spectrogram); + const scroll: u32 = @intCast(live.next % @max(live.columns, 1)); + const wave_view: layouts.WaveView = .{ + .size = .{ @floatFromInt(wave[0]), @floatFromInt(wave[1]) }, + .playhead = -1, + .gain = s.gain, + .tint = @intFromEnum(layouts.Tint.live), + .scroll = scroll, + .filled = live.filled, + }; + const wave_drawn = std.hash.Wyhash.hash(0, std.mem.asBytes(&wave_view)); + if (wave_drawn != p.wave_drawn) { + p.wave_drawn = wave_drawn; + wavePass(f, p, wave, wave_view); + } + const spectrogram_view: layouts.SpectrogramView = .{ + .size = .{ @floatFromInt(spectrogram[0]), @floatFromInt(spectrogram[1]) }, + .floor_db = floor_db, + .ceil_db = -6, + .playhead = -1, + .scroll = scroll, + .filled = live.filled, + }; + const spectrogram_drawn = std.hash.Wyhash.hash(0, std.mem.asBytes(&spectrogram_view)); + if (spectrogram_drawn != p.spectrogram_drawn) { + p.spectrogram_drawn = spectrogram_drawn; + spectrogramPass(f, p, spectrogram, spectrogram_view); + } + // The spectrum of the newest samples, smoothed. + const shown = target(f, &p.spectrum, &p.spectrum_size, size.spectrum); + spectrum(g, f, g.live_tape, p, shown, .{ + .offset = 0, + .end = live_history, + .samples = live_history, + .columns = shown[0], + .reset = @intFromBool(p.spectrum_key == 0), + .rate = @floatCast(rate), + .min_freq = min_freq, + .max_freq = maxFreq(rate), + .dt = @floatCast(dt), + }, rate); + p.spectrum_key = 1; + f.image(p.wave, b.wave, .{}); + f.image(p.spectrogram, b.spectrogram, .{}); + f.image(p.spectrum, b.spectrum, .{}); +} + +/// The samples a take's timeline spans, at least 1/32 of a sample per +/// column. +fn visible(v: View, len: usize, columns: u32) [2]f64 { + const total: f64 = @floatFromInt(len); + var lo = v.lo * total; + var hi = v.hi * total; + const least = @as(f64, @floatFromInt(columns)) / 32; + if (hi - lo < least) { + const mid = (lo + hi) / 2; + lo = @max(mid - least / 2, 0); + hi = lo + least; + } + return .{ lo, hi }; +} + +// --------------------------------------------------------------------------- +// Text. + +const dim = 0x7a8090; +const bright = 0xe6e9ef; +const red = 0xff5a5a; +const green = 0x6fdc8c; +const yellow = 0xffd84d; + +fn labels(s: *const State, ss: *const Session, store: *const Store, f: *sk.Frame, world: World, cam: sk.Camera, rate: f64) void { + const takes = store.takes.items; + const recording = s.transport.value == .recording; + const selected = if (takes.len > 0) @min(s.selected, takes.len - 1) else 0; + + // What the transport is doing, to which take. + const status: struct { []const u8, u24 } = switch (s.transport.value) { + .recording => .{ "REC", red }, + .forward => .{ "PLAY >", 0x7fd7ff }, + .backward => .{ "PLAY <", 0x7fd7ff }, + .stopped => .{ "PAUSED", dim }, + }; + const l = Layout.of(f.in.width, f.in.height); + f.text(16, 10, 2, status[0], status[1]); + if (takes.len > 0) { + const t = takes[selected]; + const times = .{ Time{ .seconds = t.heard / rate }, Time{ .seconds = @as(f64, @floatFromInt(t.len)) / rate } }; + // As much as fits beside the buttons (or, narrow, the corner). + const room = if (l.narrow) l.w - 16 * 7 else l.button(buttons.len - 1).x - 16; + if (room >= 480) { + f.print(140, 10, 2, bright, "{f} / {f}", times); + if (room >= 740) f.print(500, 14, 1, dim, "take {d} of {d}: {s}", .{ selected + 1, takes.len, source(&t) }); + } else f.print(16 + 16 * 7, 14, 1, bright, "{f} / {f}", times); + } + for (buttons, 0..) |a, i| button(s, f, a, l.button(i)); + const corner = f.format("{s}{s}", .{ + if (s.gain != 1) f.format("gain x{d:.2} ", .{s.gain}) else "", + if (cam.zoom < 1) f.format("view x{d:.2}", .{cam.zoom}) else "", + }); + f.text(f.in.width - 16 - @as(f32, @floatFromInt(corner.len * 8)), if (l.narrow) 14 else f.in.height - 16, 1, corner, dim); + + const view = l.view; + f.clip(view); + for (0..world.slots()) |slot| { + const k: u32 = @intCast(slot); + const b = l.box(k).toScreen(cam); + if (b.whole.y + b.whole.h <= view.y or b.whole.y >= view.y + view.h) continue; + // Centred in the band above the waveform, whatever the zoom. + const title_y = @round(b.whole.y + (b.wave.y - b.whole.y - 8) / 2); + if (k == world.takes) { + f.print(b.whole.x, title_y, 1, green, "LIVE the microphone, the last {d} s", .{live_seconds}); + if (ss.live.heard == 0) f.text(b.wave.x + 8, b.wave.y + 8, 1, "nothing from the microphone yet", dim); + if (cam.zoom >= 0.8) axes(f, b, rate, null); + tunerOverlay(f, b.spectrum, ss.tuners[@intFromEnum(Tuned.input)].reading, s.a4, rate, cam.zoom >= 0.8); + continue; + } + const t = &takes[k]; + const seconds = Time{ .seconds = @as(f64, @floatFromInt(t.len)) / rate }; + if (recording and k + 1 == world.takes) { + f.print(b.whole.x, title_y, 1, red, "RECORDING take {d} {f}", .{ k + 1, seconds }); + } else { + const chosen = k == selected; + if (chosen) f.fill(.{ .x = b.whole.x - if (l.narrow) @as(f32, 6) else 12, .y = b.wave.y, .w = 3, .h = b.timeline.h }, yellow); + f.print(b.whole.x, title_y, 1, @as(u24, if (chosen) bright else dim), "#{d} {s} {f}", .{ k + 1, source(t), seconds }); + } + if (cam.zoom >= 0.8) { + const span = visible(s.views[k], t.len, @intFromFloat(@min(l.box(0).wave.w, layouts.max_columns))); + axes(f, b, rate, .{ span[0] / rate, span[1] / rate }); + } + const which: ?Tuned = if (recording) (if (k + 1 == world.takes) .input else null) else if (k == selected) .take else null; + if (which) |w| tunerOverlay(f, b.spectrum, ss.tuners[@intFromEnum(w)].reading, s.a4, rate, cam.zoom >= 0.8); + } + f.clip(null); + if (world.takes == 0) f.text(16, view.y + view.h - 12, 1, "No takes yet: REC records one, OPEN opens a file.", dim); + // Keys, where there is a keyboard to press them. + if (!l.narrow) f.text(16, f.in.height - 16, 1, "R SPACE LEFT RIGHT HOME END O DEL 0 [ ] UP/DOWN take wheel or 2 fingers pan CTRL+wheel or pinch zoom", dim); +} + +/// The tuner on a spectrum: the pitch and its harmonics where they fall on +/// it, the span of two semitones either side shaded, and that span +/// magnified across the top: the nearest notes of the chromatic scale, +/// the pitch as a needle between them, and the note in cents and Hz. +fn tunerOverlay(f: *sk.Frame, r: sk.Rect, reading: ?tuner.Reading, a4: f32, rate: f64, detailed: bool) void { + const at = reading orelse return; + const max_freq = maxFreq(rate); + const fade: u8 = @intFromFloat(255 * std.math.clamp(1 - at.quiet / tuner.hold_seconds, 0.2, 1)); + const cents = at.cents(); + const color = sk.Color.hex(if (@abs(cents) <= 5) green else if (@abs(cents) <= 15) yellow else 0xff8a4d).alpha(fade); + const x = struct { + fn of(rect: sk.Rect, hz: f32, ceiling: f32) f32 { + return rect.x + @log(hz / min_freq) / @log(ceiling / min_freq) * rect.w; + } + }.of; + + const freq = tuner.frequency(at.pitch, a4); + const lo = tuner.frequency(@as(f32, @floatFromInt(at.note)) - 2.5, a4); + const hi = tuner.frequency(@as(f32, @floatFromInt(at.note)) + 2.5, a4); + f.fill(.{ .x = x(r, lo, max_freq), .y = r.y, .w = @max(x(r, hi, max_freq) - x(r, lo, max_freq), 1), .h = r.h }, sk.Color.white.alpha(fade / 10)); + f.fill(.{ .x = @round(x(r, freq, max_freq)), .y = r.y, .w = 1, .h = r.h }, color); + // The harmonics the pitch accounts for. + for (2..9) |h| { + const hz = freq * @as(f32, @floatFromInt(h)); + if (hz > max_freq) break; + f.fill(.{ .x = @round(x(r, hz, max_freq)), .y = r.y + r.h - 10, .w = 1, .h = 10 }, color.alpha(fade / 2)); + } + if (!detailed) return; + + // The magnified scale: semitones `note - 2.5` to `note + 2.5`. + const strip: sk.Rect = .{ .x = r.x + 6, .y = r.y + 4, .w = r.w - 12, .h = 30 }; + const semitone = strip.w / 5; + const centre = strip.x + strip.w / 2; + f.fill(strip, sk.Color.hex(0x0b0c0f).alpha(220)); + var c: f32 = -250; + while (c <= 250) : (c += 10) { + const whole = @mod(c, 100) == 0; + const h: f32 = if (whole) 10 else if (@mod(c, 50) == 0) 6 else 3; + f.fill(.{ .x = @round(centre + c / 100 * semitone), .y = strip.y + strip.h - h, .w = 1, .h = h }, sk.Color.hex(if (c == 0) bright else dim).alpha(if (whole) 220 else 120)); + } + for (0..5) |i| { + const note = at.note - 2 + @as(i32, @intCast(i)); + const label = f.format("{s}{d}", .{ tuner.name(note), tuner.octave(note) }); + const lx = centre + (@as(f32, @floatFromInt(i)) - 2) * semitone - @as(f32, @floatFromInt(label.len * 4)); + f.text(lx, strip.y + 4, 1, label, if (note == at.note) color else sk.Color.hex(dim).alpha(fade)); + } + const needle = std.math.clamp(centre + cents / 100 * semitone, strip.x, strip.x + strip.w - 2); + f.fill(.{ .x = @round(needle) - 1, .y = strip.y + 14, .w = 3, .h = strip.h - 14 }, color); + + // The reading. + const big = f.format("{s}{d}", .{ tuner.name(at.note), tuner.octave(at.note) }); + const panel: sk.Rect = .{ .x = r.x + r.w - 6 - 124, .y = strip.y + strip.h + 4, .w = 124, .h = 50 }; + f.fill(panel, sk.Color.hex(0x0b0c0f).alpha(200)); + f.text(panel.x + 6, panel.y + 4, 3, big, color); + const whole = @round(cents); + f.print(panel.x + 6, panel.y + 30, 1, sk.Color.hex(bright).alpha(fade), "{s}{d:.0} cents", .{ if (whole > 0) "+" else "", if (whole == 0) 0 else whole }); + f.print(panel.x + 6, panel.y + 40, 1, sk.Color.hex(dim).alpha(fade), "{d:.2} Hz", .{freq}); + f.print(panel.x + 70, panel.y + 4, 1, sk.Color.hex(dim).alpha(fade), "A4 {d:.0}", .{a4}); +} + +fn button(s: *const State, f: *sk.Frame, a: Action, r: sk.Rect) void { + const now = s.transport.value; + const label: []const u8, const lit: ?u24 = switch (a) { + .record => if (now == .recording) .{ "STOP", red } else .{ "REC", null }, + .play => if (now == .forward or now == .backward) .{ "PAUSE", 0x7fd7ff } else .{ "PLAY", null }, + .backward => .{ "REV", if (now == .backward) 0x7fd7ff else null }, + .open => .{ "OPEN", null }, + .delete => .{ "DEL", null }, + .overview => .{ "ALL", null }, + .a4_down => .{ "A4-", null }, + .a4_up => .{ "A4+", null }, + .forward, .start, .end => .{ "", null }, + }; + f.fill(r, sk.Color.hex(if (lit) |c| c else 0x1c2029).alpha(if (lit != null) 70 else 255)); + f.fill(.{ .x = r.x, .y = r.y + r.h - 1, .w = r.w, .h = 1 }, sk.Color.hex(lit orelse 0x2e3440)); + f.text(r.x + (r.w - @as(f32, @floatFromInt(label.len * 8))) / 2, r.y + 9, 1, label, sk.Color.hex(lit orelse bright)); +} + +fn source(t: *const Take) []const u8 { + return if (t.source_len > 0) t.source[0..t.source_len] else "microphone"; +} + +/// A panel's scales: its time span (seconds; null for the live view, which +/// ends now) under the spectrogram, frequencies up the spectrogram, and +/// frequency and level on the spectrum. +fn axes(f: *sk.Frame, b: Box, rate: f64, span: ?[2]f64) void { + const below = b.spectrogram.y + b.spectrogram.h + 4; + const left = if (span) |sp| f.format("{f}", .{Time{ .seconds = sp[0] }}) else f.format("-{d} s", .{live_seconds}); + const right = if (span) |sp| f.format("{f}", .{Time{ .seconds = sp[1] }}) else "now"; + f.text(b.wave.x, below, 1, left, dim); + f.text(b.wave.x + b.wave.w - @as(f32, @floatFromInt(right.len * 8)), below, 1, right, dim); + + const max_freq = maxFreq(rate); + for ([_]f64{ 100, 1000, 10000 }) |hz| { + const y = b.spectrogram.y + b.spectrogram.h * (1 - @as(f32, @floatCast(@log(hz / min_freq) / @log(max_freq / min_freq)))); + f.fill(.{ .x = b.spectrogram.x, .y = y, .w = 6, .h = 1 }, sk.Color.white.alpha(140)); + f.text(b.spectrogram.x + 9, y - 4, 1, hzLabel(f, hz), sk.Color.white.alpha(170)); + } + + for ([_]f64{ 50, 200, 1000, 5000, 20000 }) |hz| { + if (hz > max_freq) continue; + const x = b.spectrum.x + @as(f32, @floatCast(@log(hz / min_freq) / @log(max_freq / min_freq))) * b.spectrum.w; + f.text(@min(x + 3, b.spectrum.x + b.spectrum.w - 24), b.spectrum.y + b.spectrum.h + 4, 1, hzLabel(f, hz), dim); + } + var db: f32 = -24; + while (db > floor_db) : (db -= 24) { + const y = b.spectrum.y + b.spectrum.h * (db / floor_db); + f.print(b.spectrum.x + 4, y + 3, 1, dim, "{d}", .{db}); + } +} + +fn hzLabel(f: *sk.Frame, hz: f64) []const u8 { + return if (hz >= 1000) f.format("{d}k", .{hz / 1000}) else f.format("{d}", .{hz}); +} + +const Time = struct { + seconds: f64, + + pub fn format(t: Time, w: *std.Io.Writer) std.Io.Writer.Error!void { + const ms: u64 = @intFromFloat(@max(t.seconds, 0) * 1000); + try w.print("{d}:{d:0>2}.{d:0>3}", .{ ms / 60_000, ms / 1000 % 60, ms % 1000 }); + } +}; + +test "zoom keeps the view inside the take and around the pointer" { + var v: View = .{}; + zoom(&v, 0.5, 0.5); + try std.testing.expectApproxEqAbs(@as(f32, 0.25), v.lo, 1e-6); + try std.testing.expectApproxEqAbs(@as(f32, 0.75), v.hi, 1e-6); + zoom(&v, 0.0, 4); + try std.testing.expectEqual(@as(f32, 0), v.lo); + try std.testing.expectEqual(@as(f32, 1), v.hi); +} + +test "the world stacks takes in order with the live view last" { + const world: World = .{ .takes = 3, .live = true }; + try std.testing.expectEqual(@as(u32, 4), world.slots()); + const l = Layout.of(width, height); + try std.testing.expectEqual(@as(?u32, 0), world.takeAt(l, .{ .x = 10, .y = 1 })); + try std.testing.expectEqual(@as(?u32, 2), world.takeAt(l, .{ .x = 10, .y = 2 * l.panel_height + 1 })); + try std.testing.expectEqual(@as(?u32, null), world.takeAt(l, .{ .x = 10, .y = 3 * l.panel_height + 1 })); +} + +test "the camera follows the bottom and stays inside the world, wide or narrow" { + for ([_]Layout{ .of(width, height), .of(390, 844) }) |l| { + var s: State = .{ .takes = 5 }; + const cam = cameraOf(&s, l); + // Following: the live view's bottom sits at the viewport's bottom. + try std.testing.expectApproxEqAbs(l.view.y + l.view.h - 4, cam.toScreen(l.box(5).whole).y + l.panel_height, 1); + s.follow = false; + s.camera.y = 1e6; + try std.testing.expectApproxEqAbs(l.view.y + 4, cameraOf(&s, l).toScreen(l.box(0).whole).y, 1); + } +} + +test "a narrow layout stacks the spectrum under the timeline, inside the panel" { + const l: Layout = .of(390, 844); + try std.testing.expect(l.narrow); + const b = l.box(0); + try std.testing.expect(b.spectrum.y > b.timeline.y + b.timeline.h); + try std.testing.expect(b.spectrum.y + b.spectrum.h + 16 <= b.whole.y + b.whole.h); + try std.testing.expectEqual(l.world_width, b.timeline.w); + // Buttons fill the bottom, inside the canvas, without overlapping. + for (0..buttons.len) |i| { + const r = l.button(i); + try std.testing.expect(r.x >= 0 and r.x + r.w <= l.w and r.y + r.h <= l.h); + try std.testing.expect(r.w >= 40); + } +} + +test { + _ = tuner; +} + +test "the live history keeps the newest samples in order" { + var live: Live = .{}; + live.push(&.{ 1, 2, 3 }); + live.push(&.{4}); + try std.testing.expectEqualSlices(f32, &.{ 1, 2, 3, 4 }, live.samples[live_history - 4 ..]); + try std.testing.expectEqual(@as(u64, 4), live.heard); +} diff --git a/examples/micscope/src/tuner.zig b/examples/micscope/src/tuner.zig new file mode 100644 index 0000000..92a8482 --- /dev/null +++ b/examples/micscope/src/tuner.zig @@ -0,0 +1,147 @@ +//! From pitch estimates (tuner.comp's YIN, one per frame) to a steady note on +//! the 12-tone equal-tempered chromatic scale, for any instrument: notes are +//! MIDI numbers (69 is A4, at `a4` Hz), so the range is whatever the +//! detector finds, from a bass's low B to a piccolo's top. +//! +//! Raw estimates jitter and, at a note's attack or release, misfire. So: +//! unclear or quiet estimates are ignored; the median of the last few +//! clear ones is followed, smoothed; a jump of over a semitone restarts the +//! median, so a new note shows at once; and the note shown only changes +//! once the pitch is 60 cents from it, so a note held a quarter tone sharp +//! does not flicker between two names. +const std = @import("std"); +const layouts = @import("layouts"); + +pub const names = [12][]const u8{ "C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B" }; + +/// Estimates clearer than this are a note (YIN's threshold is 0.12, so +/// clarity 0.88 at least when its first dip was found). +pub const min_clarity = 0.85; +/// Quieter than this is silence, dBFS. +pub const min_level = -60.0; +/// How long the last note stays (shown as fading) after the sound stops. +pub const hold_seconds = 1.0; + +/// Fractional MIDI number of `freq`. +pub fn midi(freq: f32, a4: f32) f32 { + return 69 + 12 * std.math.log2(freq / a4); +} + +pub fn frequency(m: f32, a4: f32) f32 { + return a4 * std.math.pow(f32, 2, (m - 69) / 12); +} + +pub fn name(note: i32) []const u8 { + return names[@intCast(@mod(note, 12))]; +} + +pub fn octave(note: i32) i32 { + return @divFloor(note, 12) - 1; +} + +pub const Reading = struct { + /// The note it is nearest, as a MIDI number. + note: i32, + /// The pitch, as a fractional MIDI number. + pitch: f32, + /// Seconds since a clear estimate: 0 while the note sounds. + quiet: f32, + + pub fn cents(r: Reading) f32 { + return (r.pitch - @as(f32, @floatFromInt(r.note))) * 100; + } +}; + +pub const Tracker = struct { + recent: [5]f32 = @splat(0), + count: u32 = 0, + reading: ?Reading = null, + + pub fn feed(t: *Tracker, p: layouts.Pitch, a4: f32, dt: f32) void { + const clear = p.freq > 0 and p.clarity >= min_clarity and p.level >= min_level; + if (!clear) { + t.count = 0; + if (t.reading) |*r| { + r.quiet += dt; + if (r.quiet > hold_seconds) t.reading = null; + } + return; + } + const m = midi(p.freq, a4); + if (t.count > 0 and @abs(m - t.median()) > 1) t.count = 0; + if (t.count == t.recent.len) { + std.mem.copyForwards(f32, t.recent[0 .. t.recent.len - 1], t.recent[1..]); + t.count -= 1; + } + t.recent[t.count] = m; + t.count += 1; + const target = t.median(); + + const fresh = t.count == 1 or t.reading == null or t.reading.?.quiet > 0; + var r: Reading = if (fresh) .{ .note = @intFromFloat(@round(target)), .pitch = target, .quiet = 0 } else t.reading.?; + r.pitch += (target - r.pitch) * 0.4; + r.quiet = 0; + if (@abs(r.pitch - @as(f32, @floatFromInt(r.note))) > 0.6) r.note = @intFromFloat(@round(r.pitch)); + t.reading = r; + } + + fn median(t: *const Tracker) f32 { + var sorted = t.recent; + std.mem.sort(f32, sorted[0..t.count], {}, std.sort.asc(f32)); + return sorted[t.count / 2]; + } +}; + +fn pitchOf(freq: f32) layouts.Pitch { + return .{ .freq = freq, .clarity = 0.97, .level = -20, .period = 48_000 / freq }; +} + +test "notes and octaves on the chromatic scale" { + try std.testing.expectApproxEqAbs(@as(f32, 69), midi(440, 440), 1e-4); + try std.testing.expectApproxEqAbs(@as(f32, 40), midi(82.4069, 440), 1e-3); + try std.testing.expectEqualStrings("E", name(40)); + try std.testing.expectEqual(@as(i32, 2), octave(40)); + try std.testing.expectEqualStrings("B", name(23)); + try std.testing.expectEqual(@as(i32, 0), octave(23)); + try std.testing.expectApproxEqAbs(@as(f32, 261.626), frequency(60, 440), 1e-2); + // Another reference pitch moves every note with it. + try std.testing.expectApproxEqAbs(@as(f32, 69), midi(442, 442), 1e-4); +} + +test "a held note settles, in cents from it" { + var t: Tracker = .{}; + // A4, 12 cents sharp. + const sharp = frequency(69.12, 440); + for (0..20) |_| t.feed(pitchOf(sharp), 440, 1.0 / 60.0); + const r = t.reading.?; + try std.testing.expectEqual(@as(i32, 69), r.note); + try std.testing.expectApproxEqAbs(@as(f32, 12), r.cents(), 0.5); +} + +test "one wild estimate does not move the note" { + var t: Tracker = .{}; + for (0..10) |_| t.feed(pitchOf(110), 440, 1.0 / 60.0); + t.feed(pitchOf(220), 440, 1.0 / 60.0); + t.feed(pitchOf(110), 440, 1.0 / 60.0); + try std.testing.expectEqual(@as(i32, 45), t.reading.?.note); +} + +test "a new note shows at once, a quarter tone does not flicker" { + var t: Tracker = .{}; + for (0..10) |_| t.feed(pitchOf(110), 440, 1.0 / 60.0); + for (0..3) |_| t.feed(pitchOf(146.83), 440, 1.0 / 60.0); + try std.testing.expectEqual(@as(i32, 50), t.reading.?.note); + // 55 cents sharp of D3 stays D3. + for (0..30) |_| t.feed(pitchOf(frequency(50.55, 440)), 440, 1.0 / 60.0); + try std.testing.expectEqual(@as(i32, 50), t.reading.?.note); +} + +test "silence holds the note a while, then lets go" { + var t: Tracker = .{}; + for (0..10) |_| t.feed(pitchOf(330), 440, 1.0 / 60.0); + const quiet: layouts.Pitch = .{ .freq = 0, .clarity = 0, .level = -120, .period = 0 }; + for (0..30) |_| t.feed(quiet, 440, 1.0 / 60.0); + try std.testing.expect(t.reading != null); + for (0..60) |_| t.feed(quiet, 440, 1.0 / 60.0); + try std.testing.expect(t.reading == null); +} diff --git a/examples/micscope/test/README.md b/examples/micscope/test/README.md new file mode 100644 index 0000000..6f4b833 --- /dev/null +++ b/examples/micscope/test/README.md @@ -0,0 +1,22 @@ +# Test material + +- `sweep.wav`: a 2 s log sweep (40 Hz to 16 kHz) over a quiet 440 Hz tone, + 48 kHz mono, generated for these tests. +- `tones.wav`: five 1 s notes for the tuner, generated for these tests + (harmonics with a little noise, 48 kHz mono): E1 with its 2nd harmonic + loudest, E2 with a weak fundamental, A4 12 cents sharp, D4 with no + fundamental at all, and C7. +- `reference.flac`: the rendering reference, all on one screen. 15 s of a + loudly mastered electronic track (sub-bass kicks, hats, a full mix), then + 15 s of a Berlin street (traffic, a close vehicle pass, broadband noise), + then `sweep.wav`; 48 kHz stereo, joined with short crossfades. + + - Square Punch, "Thundercat" (Plasma Pumper EP, 2011), from 3:36: + [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Square_Punch_-_Thundercat.flac), + CC BY-SA 3.0. + - LG, "traffic karl-liebknechtstrasse 111223", from 1:18: + [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:141977_lg_traffic-karl-liebknechtstrasse-111223.wav), + from [Freesound](http://freesound.org/people/LG/sounds/141977/), + CC BY-SA 3.0. + + The excerpt is licensed CC BY-SA 3.0 like its sources. diff --git a/examples/micscope/test/profile.script b/examples/micscope/test/profile.script new file mode 100644 index 0000000..aa56b3e --- /dev/null +++ b/examples/micscope/test/profile.script @@ -0,0 +1,167 @@ +# Zooms in and out one ctrl+wheel step per frame on the reference take, so +# waveform and spectrogram are recomputed every frame: run by +# `zig build profile` with --profile-gpu, which prints GPU time per command. +console open test/reference.flac +frames 3 +key right +hold left_ctrl +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 +wheel 470 150 -0.15 +frames 1 diff --git a/examples/micscope/test/record.script b/examples/micscope/test/record.script new file mode 100644 index 0000000..2750a18 --- /dev/null +++ b/examples/micscope/test/record.script @@ -0,0 +1,23 @@ +# Records (from whatever microphone the host gives a headless run) into a +# new take, stops, and plays it back. +expect takes 0 +key r +expect transport.value recording +expect takes 1 +frames 60 +screenshot .zig-cache/micscope/recording.png +key r +expect transport.value stopped +key space +expect transport.value forward +frames 10 +# Each recording is another take; opening a file while recording ends it. +key r +expect takes 2 +expect selected 1 +console open test/sweep.wav +expect transport.value stopped +expect takes 3 +frames 2 +expect takes 3 +screenshot .zig-cache/micscope/strip.png diff --git a/examples/micscope/test/reference.flac b/examples/micscope/test/reference.flac Binary files differnew file mode 100644 index 0000000..7ad1c63 --- /dev/null +++ b/examples/micscope/test/reference.flac diff --git a/examples/micscope/test/render.script b/examples/micscope/test/render.script new file mode 100644 index 0000000..71b50c7 --- /dev/null +++ b/examples/micscope/test/render.script @@ -0,0 +1,18 @@ +# The rendering reference (see README.md): song, street and sweep in one +# take. Screenshots land in .zig-cache/micscope/ for review. +console open test/reference.flac +expect transport.value stopped +frames 3 +screenshot .zig-cache/micscope/reference.png +# Playing: the playhead across waveform and spectrogram, the spectrum. +key right +frames 90 +expect-audio 1 +screenshot .zig-cache/micscope/reference-playing.png +# Zoomed into the street: the low end's hum lines and the noise floor. +key space +hold left_ctrl +wheel 660 150 4 +release left_ctrl +frames 3 +screenshot .zig-cache/micscope/reference-street.png diff --git a/examples/micscope/test/sweep.wav b/examples/micscope/test/sweep.wav Binary files differnew file mode 100644 index 0000000..1f131e3 --- /dev/null +++ b/examples/micscope/test/sweep.wav diff --git a/examples/micscope/test/tones.wav b/examples/micscope/test/tones.wav Binary files differnew file mode 100644 index 0000000..b2cf3ea --- /dev/null +++ b/examples/micscope/test/tones.wav diff --git a/examples/micscope/test/transport.script b/examples/micscope/test/transport.script new file mode 100644 index 0000000..f821a03 --- /dev/null +++ b/examples/micscope/test/transport.script @@ -0,0 +1,90 @@ +# Opens a file as a take and plays it both ways: a 2 s log sweep (40 Hz to +# 16 kHz) over a quiet 440 Hz tone. +console open test/sweep.wav +frames 2 +expect takes 1 +expect selected 0 +expect transport.value stopped +key right +expect transport.value forward +frames 30 +expect-audio 1 +screenshot .zig-cache/micscope/forward.png +key left +expect transport.value backward +expect backward true +frames 20 +key space +expect transport.value stopped +# Space resumes the way it last went. +key space +expect transport.value backward +key space +# Seeking: keys, and a click on the take's timeline. +key end +expect seek.value 1 +click 400 150 +expect seek.serial 2 +# Ctrl+wheel (or a pinch) on a take's timeline zooms its time around the +# pointer; Z zooms back out. +hold left_ctrl +wheel 400 150 3 +release left_ctrl +key z +key right +frames 30 +screenshot .zig-cache/micscope/played.png +# A second take goes under the first and is selected; up is older. +console open test/sweep.wav +expect takes 2 +expect selected 1 +key up +expect selected 0 +key down +expect selected 1 +# A click on the first take selects it. +click 400 150 +expect selected 0 +frames 2 +screenshot .zig-cache/micscope/two-takes.png +# Deleting it leaves the second. +key delete +expect takes 1 +expect selected 0 +# State that miscounts the takes is told by draw. +console set takes 0 +frames 2 +expect takes 1 +# More takes than fit: the camera follows the bottom until the wheel pans +# it, then 0 shows the whole strip. +console open test/sweep.wav +console open test/sweep.wav +console open test/sweep.wav +expect follow true +wheel 400 400 2 +expect follow false +frames 2 +screenshot .zig-cache/micscope/panned.png +key 0 +expect follow true +frames 2 +screenshot .zig-cache/micscope/overview.png +# Ctrl+wheel off the timelines zooms the view. +hold left_ctrl +wheel 1150 400 1 +release left_ctrl +frames 2 +screenshot .zig-cache/micscope/zoomed-view.png +# The buttons on top do what the keys do, for touch screens. +click 832 18 +expect transport.value recording +click 832 18 +expect transport.value stopped +click 890 18 +expect transport.value forward +click 890 18 +expect transport.value stopped +click 1238 18 +expect a4 441 +frames 2 +screenshot .zig-cache/micscope/buttons.png diff --git a/examples/micscope/test/tuner.script b/examples/micscope/test/tuner.script new file mode 100644 index 0000000..4e0e480 --- /dev/null +++ b/examples/micscope/test/tuner.script @@ -0,0 +1,31 @@ +# The tuner on a take, at its playhead: five notes where the strongest FFT +# peak is not (or not only) the note. Expected, by screenshot: +# E1 0 cents (41.2 Hz) bass, 2nd harmonic loudest +# E2 0 cents (82.4 Hz) guitar, weak fundamental +# A4 +12 cents (443.1 Hz) sharp +# D4 0 cents (293.7 Hz) fundamental missing +# C7 0 cents (2093 Hz) piccolo +console open test/tones.wav +click 107 150 +frames 20 +screenshot .zig-cache/micscope/tuner-e1.png +click 290 150 +frames 20 +screenshot .zig-cache/micscope/tuner-e2.png +click 472 150 +frames 20 +screenshot .zig-cache/micscope/tuner-a4.png +click 654 150 +frames 20 +screenshot .zig-cache/micscope/tuner-d4.png +click 837 150 +frames 20 +screenshot .zig-cache/micscope/tuner-c7.png +# Another reference pitch: A4 = 443 makes the sharp A4 in tune. +click 472 150 +key right_bracket +key right_bracket +key right_bracket +expect a4 443 +frames 20 +screenshot .zig-cache/micscope/tuner-a443.png diff --git a/examples/micscope/web/icon-192.png b/examples/micscope/web/icon-192.png Binary files differnew file mode 100644 index 0000000..0b26078 --- /dev/null +++ b/examples/micscope/web/icon-192.png diff --git a/examples/micscope/web/icon-512.png b/examples/micscope/web/icon-512.png Binary files differnew file mode 100644 index 0000000..0b5538c --- /dev/null +++ b/examples/micscope/web/icon-512.png diff --git a/examples/micscope/web/icon.svg b/examples/micscope/web/icon.svg new file mode 100644 index 0000000..1220011 --- /dev/null +++ b/examples/micscope/web/icon.svg @@ -0,0 +1,26 @@ +<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"> + <!-- micscope's icon: a waveform over its spectrogram's harmonics, in the + app's colours. Full-bleed, so it also works masked. Rendered to the + PNGs beside it with: resvg -w N -h N icon.svg icon-N.png --> + <defs> + <linearGradient id="heat" gradientUnits="userSpaceOnUse" x1="96" y1="0" x2="416" y2="0"> + <stop offset="0" stop-color="#2b0a4a"/> + <stop offset="0.25" stop-color="#b8325a"/> + <stop offset="0.6" stop-color="#f98c0a"/> + <stop offset="1" stop-color="#fcf3a6"/> + </linearGradient> + </defs> + <rect width="512" height="512" fill="#0b0c0f"/> + <!-- harmonics: a fundamental and its overtones, fading --> + <g stroke="url(#heat)" stroke-linecap="round"> + <line x1="96" y1="392" x2="416" y2="392" stroke-width="22"/> + <line x1="96" y1="340" x2="416" y2="340" stroke-width="16" opacity="0.85"/> + <line x1="96" y1="300" x2="416" y2="300" stroke-width="11" opacity="0.65"/> + <line x1="96" y1="270" x2="416" y2="270" stroke-width="7" opacity="0.45"/> + </g> + <!-- the waveform --> + <path d="M96 176 C120 96 152 96 176 176 S232 256 256 176 S312 96 336 176 S392 256 416 176" + fill="none" stroke="#6fdc8c" stroke-width="18" stroke-linecap="round"/> + <!-- the tuner's needle --> + <rect x="246" y="76" width="20" height="356" rx="10" fill="#ffd84d" opacity="0.9"/> +</svg> diff --git a/examples/spheres/README.md b/examples/spheres/README.md new file mode 100644 index 0000000..3c65666 --- /dev/null +++ b/examples/spheres/README.md @@ -0,0 +1,153 @@ +# spheres + +The three-body problem, heard. Three bodies pull on each other; each is +also a voice, synthesised from its motion. A [sketchbook](../..) +sketch: native on Linux, and in browsers with WebGPU. + +Everything moves on the GPU: + +| Shader | Does | +| --- | --- | +| `sim.comp` | 16384 copies of the system at once, one per thread: system 0 as given, the others moved by up to 2×10⁻⁴. The copies drawn as a cloud show chaos as it happens: the Pythagorean problem's tears apart, the figure eight's stays a line. System 0 also leaves its trail and sings: from its own motion, or from a look-ahead snapshot while a chord is held. | +| `physics.glsl`, `voices.glsl` | Included by the simulation and the look-ahead, so the live system and its futures move and sing alike. Leapfrog (kick, drift, kick), symplectic and time-reversible, with close passes split into steps of 2 % of the local dynamical time; and the mapping from motion to voices below. | +| `lookahead.comp` | 32 futures of system 0, one per thread, from the preset's start: alternative 0 as system 0 began, the rest nudged by growing amounts so chaotic ones fan out. 8 snapshots a frame each (0.4 s of simulated time, against the live system's 0.7 s a second), every snapshot kept on the GPU with the chord sung there. | +| `rechord.comp` | The look-ahead's chords again when SCALE changes. | +| `synth.comp` | Each audio block, one thread per sample, from the voices: phases in closed form for frequencies gliding across the block, so blocks join seamlessly and every sample is computed alone. | +| `advance.comp` | Where the block's phases ended, for the next. | +| `sprite.*`, `trail.*`, `fade.frag` | The sky: the cloud as layered dust on a long exposure, system 0's bodies as glows, and their trails as ribbons tapering with age. | +| `link.*` | The coupling: a ribbon from modulator to carrier, as bright as the coupling, with pulses flowing along it. | + +The sound reaches the speakers by a readback (`f.download`) about two +frames later; the CPU only queues it, and hands it to sketchbook's `scope` +module (the views micscope uses) for the oscilloscope, spectrum and +spectrogram. + +## From motion to sound + +Each body is a voice: + +- **pitch** follows its speed (220 Hz at speed 0.6, two octaves down to two + and a half up), snapped to a scale (diatonic by default; pentatonic, chromatic or none); +- **modulator ratio** follows how lopsided its distances to the other two + are: near 1 or 2 (harmonic) when balanced, in between (bell-like) when + not; +- **modulation depth** follows how near its nearest neighbour is: a close + pass bursts into sidebands. + +And the **closest pair is one FM voice**: the faster of the two falls +silent and becomes the modulator, a sine at its own frequency modulating +the slower one's carrier (two operators, both moved by the bodies), with an +index that grows as they close in, up to 4. So two voices sound: the pair, +and the third body on its own. When the closest pair changes, the old +modulator fades back in as a voice while the new one fades out. In additive +mode every harmonic gets the same push, so the timbre bends without +turning harsh. The sky draws the pair: pulses running from the modulator to +its carrier; the voices' lines say who is `modulating` and who is +`FM by A 2.10`, and the spectrum marks the pair's sidebands, |J_k(index)| +at multiples of the modulator's frequency. + +The synthesis, with carrier phase c, modulator phase m and depth d: + +| Mode | Signal | +| --- | --- | +| SINE | sin c | +| ADD (the default) | harmonics k with weights r^(k−1), r = d/(d+2): brighter with depth | +| AM | sin c · (1 + a sin m)/(1 + a), a = min(d/4, 1): sidebands at c ± m | +| RING | sin c · sin m: only c ± m | +| FM | the carrier's frequency swung by d times itself, by cos m: the index (deviation over modulator frequency) is d/ratio, so it changes as the ratio does | +| PM | sin(c + d sin m): the index is d, whatever the modulator | + +FM and PM sound alike for a steady modulator and part when it moves. The +spectrum marks where the theory puts each voice's partials (Bessel +functions |J_k(index)| at carrier + k·modulator for FM and PM): the +focused voice's in full, the others' as ticks. + +## As an instrument + +CHORDS (or C) turns spheres into a chord instrument. The look-ahead runs +ahead of the sky, 120 s of simulated time filled in about five seconds, and +the chords it finds light up on a grid laid out as a sanfona's (Stradella) +bass side: columns round the circle of fifths (G♭ D♭ A♭ E♭ B♭ F C G D A E +B; a phone shows B♭ to B), so neighbours are related chords, and rows by +quality (major, minor, seventh, sus, power), each half a button over from +the one above, so a progression is the same hand shape in any key. A +button is dim until some future sings its chord. The computer keyboard's +rows are staggered the same way: the number row plays majors, QWERTY +minors, ASDF sevenths, ZXCV sus chords, column by column; Escape goes back +to running free. + +A chord is the pitch classes of the three bodies' notes, the silent +modulator's included (it sets the carrier's sidebands), and it only counts +in a playable voicing: every note from E2 to A5, an octave and a fifth at +most from lowest to highest, and the FM pair's index at most 3. Otherwise +the same pitch classes are another sound, shrill, sprawling or a cloud of +sidebands. How many there are +depends on the scale (SCALE): pentatonic gives few, diatonic (the default) +the white-key chords, chromatic every one, and unsnapped pitches round to +the nearest semitone. + +A button finds the best time its chord plays in any of the 32 futures, +weighing how near it is against how it sounds: half a second's travel more +for another future than the system's own, 0.6 s for a root not in the bass, +and up to 3 s for a wide, high or hard-modulated voicing. Then: + +- the sound jumps there at once: queued sound is dropped, and system 0 + sings from that snapshot on the GPU, holding the chord; +- the sky travels: the whole simulation runs forwards or backwards to that + time, about a second's ride, its trails drawn smoothly all the way (points + by simulated time, many a frame) or, going back, rewound, and system 0 steps into the snapshot, + into that future if it was another's, the jump leaving a break in the + trails rather than a line; +- the sky holds still on that moment, the configuration that sings the + chord, until the next. + +The strip over the grid shows the system's own future's chords along time, +coloured by row, now as a white line and the travel's target in yellow. +Costs: the look-ahead takes about 0.7 ms of GPU a frame (2.4 ms at worst, +through close passes) while it fills, and nothing after; the whole frame is +about 1.4 ms on an RTX 3050. + +## Presets + +1. **Figure eight** (Chenciner and Montgomery): three equal masses on one + figure of eight. Stable. +2. **Butterfly I** (Šuvakov and Dmitrašinović): periodic, unstable. +3. **Pythagorean** (Burrau): masses 3, 4, 5 at rest on a 3-4-5 triangle; + close passes, chaos, and in the end an escape. +4. **Lagrange triangle**: equal masses turning rigidly; unstable, so the + cloud shows it coming apart. +5. **Random**: three masses and velocities from a seed (R draws again). + +## Keys + +| Key | | +| --- | --- | +| C | the instrument (the chord grid); Escape back to running free | +| keyboard rows | as an instrument: number row majors, QWERTY minors, ASDF sevenths, ZXCV sus chords, by fifths | +| Space | pause / run | +| Left / Right | run time backwards / forwards (leapfrog retraces its steps, until chaos amplifies rounding) | +| Up / Down | faster / slower | +| P, E, X | the next preset, the figure eight, a random system | +| S | the next synthesis mode | +| Q | the scale: diatonic, chromatic, unsnapped, pentatonic | +| M | mute | +| Tab | the voice the spectrum and oscilloscope follow | +| R | the preset again, copies spread anew | + +The buttons do the same on touch screens. The header shows the simulated +time and the total energy: the figure eight's holds to the last digit +shown; through the Pythagorean problem's close passes it drifts a few +percent, where the adaptive steps cost leapfrog its exact symplecticity. + +On the web it is an installable app (a PWA) that opens offline. Its icon +is `web/icon.svg`, rendered to the PNGs beside it with +`resvg -w N -h N icon.svg icon-N.png`. + +## Building + +Zig 0.17 and `glslc`; for the web build `naga` (`cargo install naga-cli`), +and to serve it `miniserve` and `openssl`. + + zig build run # native, with hot reload (`zig build lib --watch` alongside) + zig build test # unit tests and a headless run of every preset and mode + zig build serve # the web build over https on port 8042 diff --git a/examples/spheres/build.zig b/examples/spheres/build.zig new file mode 100644 index 0000000..8cc8f60 --- /dev/null +++ b/examples/spheres/build.zig @@ -0,0 +1,55 @@ +const std = @import("std"); +const sketchbook = @import("sketchbook"); + +pub fn build(b: *std.Build) void { + const target = b.standardTargetOptions(.{}); + const optimize = b.standardOptimizeOption(.{}); + const sb = sketchbook.init(b, .{ .target = target, .optimize = optimize }); + + // The simulation, the synthesiser and the sky, as SPIR-V for the native + // host and WGSL for the web; their uniforms are src/layouts.zig's structs. + const layouts = b.createModule(.{ .root_source_file = b.path("src/layouts.zig") }); + const shaders = sb.addShaders(.{ + .layouts = .{ .module = layouts, .types = &.{ "SimParams", "LookParams", "SynthParams", "SkyView", "Fade" } }, + .sources = &.{ + b.path("shaders/sim.comp"), + b.path("shaders/lookahead.comp"), + b.path("shaders/rechord.comp"), + b.path("shaders/synth.comp"), + b.path("shaders/advance.comp"), + b.path("shaders/fullscreen.vert"), + b.path("shaders/fade.frag"), + b.path("shaders/sprite.vert"), + b.path("shaders/sprite.frag"), + b.path("shaders/trail.vert"), + b.path("shaders/trail.frag"), + b.path("shaders/link.vert"), + b.path("shaders/link.frag"), + }, + }); + + const app = sb.app(.{ + .name = "spheres", + .root_source_file = b.path("src/main.zig"), + .imports = &.{ + .{ .name = "layouts", .module = layouts }, + .{ .name = "shaders", .module = shaders.module }, + .{ .name = "scope", .module = sb.modules.scope }, + }, + }); + _ = app.addSteps(.{ + // Installable, and it opens offline (icons from web/icon.svg). + // ReleaseSmall: a third the size of ReleaseSafe's wasm, and Zig + // 0.17's LLVM spins for many minutes optimising this sketch for speed. + .web = .{ .optimize = .ReleaseSmall, .pwa = .{ + .description = "The three-body problem on the GPU, heard: each body a voice in FM, PM, AM, ring or additive synthesis.", + .background = "#05070d", + .theme = "#05070d", + .icons = &.{ + .{ .source = b.path("web/icon-192.png"), .size = 192 }, + .{ .source = b.path("web/icon-512.png"), .size = 512 }, + }, + } }, + .scripts = &.{b.path("test/play.script")}, + }); +} diff --git a/examples/spheres/build.zig.zon b/examples/spheres/build.zig.zon new file mode 100644 index 0000000..b982c30 --- /dev/null +++ b/examples/spheres/build.zig.zon @@ -0,0 +1,10 @@ +.{ + .name = .spheres, + .version = "0.0.1", + .dependencies = .{ + .sketchbook = .{ .path = "../.." }, + }, + .minimum_zig_version = "0.17.0", + .paths = .{""}, + .fingerprint = 0x688720ee43c1135a, +} diff --git a/examples/spheres/shaders/advance.comp b/examples/spheres/shaders/advance.comp new file mode 100644 index 0000000..e1e8789 --- /dev/null +++ b/examples/spheres/shaders/advance.comp @@ -0,0 +1,32 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// After synth.comp: where the block's phases ended, wrapped, for the next +// block to start from; and the voices and coupling it ended on, to glide +// from. (The coupling pushes phases without moving them on, so the phases +// need nothing from it.) +layout(local_size_x = 1) in; + +#include "SynthParams.glsl" + +layout(set = 1, binding = 0) buffer Voices { vec4 voices[]; }; +layout(set = 1, binding = 1) buffer Phases { vec4 phases[]; }; + +const float TAU = 6.28318530718; + +void main() { + float count = float(u.count); + for (int i = 0; i < 3; i++) { + vec4 from = voices[i]; + vec4 to = voices[3 + i]; + vec4 phase = phases[i]; + float wc0 = TAU * from.x / u.rate; + float wc1 = TAU * to.x / u.rate; + // As synth.comp's glide at n = count. + float c = phase.x + 0.5 * (wc0 + wc1) * count; + float m = phase.y + 0.5 * (wc0 * from.y + wc1 * to.y) * count; + float fm = phase.z + (c - phase.x) + to.z / max(to.y, 1e-3) * (sin(m) - sin(phase.y)); + phases[i] = vec4(mod(c, TAU), mod(m, TAU), mod(fm, TAU), 0.0); + voices[i] = to; + voices[6 + i] = voices[9 + i]; + } +} diff --git a/examples/spheres/shaders/fade.frag b/examples/spheres/shaders/fade.frag new file mode 100644 index 0000000..71b02f2 --- /dev/null +++ b/examples/spheres/shaders/fade.frag @@ -0,0 +1,11 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// Blended over the sky (premultiplied), it moves every pixel `amount` of +// the way to the background: what was drawn before fades, a long exposure. +layout(location = 0) out vec4 out_color; + +#include "Fade.glsl" + +void main() { + out_color = vec4(u.background * u.amount, u.amount); +} diff --git a/examples/spheres/shaders/fullscreen.vert b/examples/spheres/shaders/fullscreen.vert new file mode 100644 index 0000000..f45ee09 --- /dev/null +++ b/examples/spheres/shaders/fullscreen.vert @@ -0,0 +1,6 @@ +#version 450 +// One triangle covering the target. +void main() { + vec2 p = vec2(float((gl_VertexIndex << 1) & 2), float(gl_VertexIndex & 2)); + gl_Position = vec4(p * 2.0 - 1.0, 0.0, 1.0); +} diff --git a/examples/spheres/shaders/link.frag b/examples/spheres/shaders/link.frag new file mode 100644 index 0000000..53da3cf --- /dev/null +++ b/examples/spheres/shaders/link.frag @@ -0,0 +1,12 @@ +#version 450 +// Pulses running from the modulator to the carrier, soft at the edges. +layout(location = 0) in vec4 color; +layout(location = 1) in vec2 along; +layout(location = 2) flat in float time; +layout(location = 0) out vec4 out_color; + +void main() { + float pulse = 0.5 + 0.5 * sin(along.x * 0.25 - time * 9.0); + float edge = 1.0 - along.y * along.y; + out_color = color * edge * (0.3 + 0.7 * pulse * pulse); +} diff --git a/examples/spheres/shaders/link.vert b/examples/spheres/shaders/link.vert new file mode 100644 index 0000000..d4582e4 --- /dev/null +++ b/examples/spheres/shaders/link.vert @@ -0,0 +1,47 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The FM pair, drawn: from the modulating body to its carrier, a ribbon +// as wide and bright as the index is high, shading from the +// modulator's colour to the carrier's. Six vertices for each of the six +// ordered pairs; uncoupled pairs collapse to nothing. +layout(location = 0) out vec4 color; +layout(location = 1) out vec2 along; +layout(location = 2) flat out float time; + +#include "SkyView.glsl" + +layout(set = 0, binding = 0) readonly buffer Bodies { vec4 bodies[]; }; +// layouts.voices_*: coupling rows now at 9..11. +layout(set = 0, binding = 1) readonly buffer Voices { vec4 voices[]; }; + +// Match main.zig's palette. +const vec3 palette[3] = vec3[](vec3(0.40, 0.85, 1.0), vec3(1.0, 0.45, 0.82), vec3(1.0, 0.80, 0.35)); +// Pairs (carrier, modulator). +const ivec2 pairs[6] = ivec2[](ivec2(0, 1), ivec2(0, 2), ivec2(1, 0), ivec2(1, 2), ivec2(2, 0), ivec2(2, 1)); + +void main() { + ivec2 pair = pairs[uint(gl_VertexIndex) / 6u]; + uint corner = uint(gl_VertexIndex) % 6u; + float amount = voices[9 + pair.x][pair.y]; + time = u.time; + if (amount < 0.02) { + gl_Position = vec4(0.0, 0.0, 2.0, 1.0); + color = vec4(0.0); + along = vec2(0.0); + return; + } + // Corners 0, 3, 5 at the modulator, 1, 2, 4 at the carrier; 0, 1, 3 on + // one side. + bool far_end = corner == 1u || corner == 2u || corner == 4u; + bool side = corner == 2u || corner == 4u || corner == 5u; + vec2 from = (bodies[pair.y].xy - u.center) / u.per_pixel; + vec2 to = (bodies[pair.x].xy - u.center) / u.per_pixel; + vec2 d = to - from; + float len = max(length(d), 1e-3); + vec2 normal = vec2(-d.y, d.x) / len; + float width = u.radius * (0.4 + min(amount, 1.6)); + vec2 at = (far_end ? to : from) + normal * (side ? width : -width); + gl_Position = vec4(at * 2.0 / u.size, 0.0, 1.0); + along = vec2(far_end ? len : 0.0, side ? 1.0 : -1.0); + color = vec4(mix(palette[pair.y], palette[pair.x], far_end ? 1.0 : 0.0), 1.0) * u.alpha * clamp(amount, 0.0, 1.0); +} diff --git a/examples/spheres/shaders/lookahead.comp b/examples/spheres/shaders/lookahead.comp new file mode 100644 index 0000000..75902d7 --- /dev/null +++ b/examples/spheres/shaders/lookahead.comp @@ -0,0 +1,52 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// Futures of system 0, one per thread, for spheres as an instrument: from +// the preset's start, alternative 0 exactly as system 0 began and the +// others nudged by growing amounts, so where the system is chaotic their +// futures fan out. Each is integrated ahead `count` samples a frame (with +// physics.glsl, as the live system is), leaving a snapshot every +// `sample_dt` and the chord its bodies would sing there (voices.glsl). A +// chord chosen then finds the nearest time it plays, in any of them. +layout(local_size_x = 32) in; + +#include "LookParams.glsl" +#include "physics.glsl" +#include "voices.glsl" + +// Where each alternative has got to: 3 vec4s each (position xy, velocity zw). +layout(set = 1, binding = 0) buffer State { vec4 state[]; }; +// Snapshots, sample-major: sample t of alternative a at 3 (t * K + a). +layout(set = 1, binding = 1) buffer Snapshots { vec4 snapshots[]; }; +// Chords, sample-major: t * K + a. +layout(set = 1, binding = 2) buffer Chords { uint chords[]; }; + +// Match main.zig's leapfrog step. +const float BASE_STEP = 0.004; + +void main() { + uint alt = gl_GlobalInvocationID.x; + if (alt >= u.alternatives) return; + vec3 m = u.masses.xyz; + vec2 p[3]; + vec2 v[3]; + for (int i = 0; i < 3; i++) { + vec4 b = state[3u * alt + uint(i)]; + p[i] = b.xy; + v[i] = b.zw; + } + vec2 a[3]; + accelerations(p, m, u.softening, a); + Mapping map = Mapping(u.base, u.speed_ref, u.depth_scale, u.couple_scale, u.quantize); + uint steps = uint(ceil(u.sample_dt / BASE_STEP)); + float dt = u.sample_dt / float(steps); + for (uint t = u.first; t < min(u.first + u.count, u.horizon); t++) { + uint at = t * u.alternatives + alt; + for (int i = 0; i < 3; i++) snapshots[3u * at + uint(i)] = vec4(p[i], v[i]); + vec4 sung[3]; + vec4 coupled[3]; + sing(p, v, map, sung, coupled); + chords[at] = chordOf(sung) | voicing(sung, coupled); + for (uint k = 0u; k < steps; k++) leapfrog(p, v, a, dt, m, u.softening, 256.0); + } + for (int i = 0; i < 3; i++) state[3u * alt + uint(i)] = vec4(p[i], v[i]); +} diff --git a/examples/spheres/shaders/physics.glsl b/examples/spheres/shaders/physics.glsl new file mode 100644 index 0000000..5d74bee --- /dev/null +++ b/examples/spheres/shaders/physics.glsl @@ -0,0 +1,36 @@ +// The three-body physics, shared by sim.comp and lookahead.comp so the live +// system and its futures move alike: gravity with softening, and leapfrog +// (kick, drift, kick), symplectic and time-reversible, split finer through +// close passes, about 2 % of the local dynamical time a part: too coarse +// and a close pass gains energy and flings a body out. + +void accelerations(vec2 p[3], vec3 m, float softening, out vec2 a[3]) { + float eps2 = softening * softening; + a[0] = vec2(0.0); + a[1] = vec2(0.0); + a[2] = vec2(0.0); + for (int i = 0; i < 3; i++) { + for (int j = i + 1; j < 3; j++) { + vec2 d = p[j] - p[i]; + float inv = inversesqrt(dot(d, d) + eps2); + vec2 f = d * (inv * inv * inv); + a[i] += m[j] * f; + a[j] -= m[i] * f; + } + } +} + +// One step of `dt`, in up to `most` parts; `a` holds the accelerations at +// `p`, before and after. +void leapfrog(inout vec2 p[3], inout vec2 v[3], inout vec2 a[3], float dt, vec3 m, float softening, float most) { + float near = min(min(distance(p[0], p[1]), distance(p[0], p[2])), distance(p[1], p[2])) + softening; + float local = sqrt(near * near * near / (m.x + m.y + m.z)); + uint k = uint(clamp(ceil(abs(dt) / (0.02 * local)), 1.0, most)); + float h = dt / float(k); + for (uint j = 0u; j < k; j++) { + for (int i = 0; i < 3; i++) v[i] += 0.5 * h * a[i]; + for (int i = 0; i < 3; i++) p[i] += h * v[i]; + accelerations(p, m, softening, a); + for (int i = 0; i < 3; i++) v[i] += 0.5 * h * a[i]; + } +} diff --git a/examples/spheres/shaders/rechord.comp b/examples/spheres/shaders/rechord.comp new file mode 100644 index 0000000..6119120 --- /dev/null +++ b/examples/spheres/shaders/rechord.comp @@ -0,0 +1,27 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The look-ahead's chords again, for another scale setting: one thread per +// snapshot, the first `count` of them. +layout(local_size_x = 64) in; + +#include "LookParams.glsl" +#include "voices.glsl" + +layout(set = 0, binding = 0) readonly buffer Snapshots { vec4 snapshots[]; }; +layout(set = 1, binding = 0) buffer Chords { uint chords[]; }; + +void main() { + uint at = gl_GlobalInvocationID.x; + if (at >= u.count) return; + vec2 p[3]; + vec2 v[3]; + for (int i = 0; i < 3; i++) { + vec4 b = snapshots[3u * at + uint(i)]; + p[i] = b.xy; + v[i] = b.zw; + } + vec4 sung[3]; + vec4 coupled[3]; + sing(p, v, Mapping(u.base, u.speed_ref, u.depth_scale, u.couple_scale, u.quantize), sung, coupled); + chords[at] = chordOf(sung) | voicing(sung, coupled); +} diff --git a/examples/spheres/shaders/sim.comp b/examples/spheres/shaders/sim.comp new file mode 100644 index 0000000..4a79ce1 --- /dev/null +++ b/examples/spheres/shaders/sim.comp @@ -0,0 +1,86 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// The three-body problem, one system per thread: every copy of the system +// at once, each a hair apart from system 0, so the cloud they make shows +// how far apart chaos takes them (a stable orbit keeps it a thin line). +// The physics and the voices are physics.glsl's and voices.glsl's, which +// the look-ahead shares. +// +// System 0 also leaves its trail, reports its state, and sings: from its +// own motion, or, while a chord is held, from the look-ahead's snapshot +// that sings it. It can also adopt a snapshot, stepping into that future. +layout(local_size_x = 64) in; + +#include "SimParams.glsl" +#include "physics.glsl" +#include "voices.glsl" + +// The look-ahead's snapshots: 3 vec4s each (position xy, velocity zw). +layout(set = 0, binding = 0) readonly buffer Ahead { vec4 ahead[]; }; +// Position xy, velocity zw: system s's bodies are 3s, 3s + 1, 3s + 2. +layout(set = 1, binding = 0) buffer Bodies { vec4 bodies[]; }; +// System 0's positions: body b's ring at b * TRAIL. +layout(set = 1, binding = 1) buffer Trail { vec2 trail[]; }; +// layouts.voices_*: voices 0..2 as the synthesiser last used them, 3..5 +// now; coupling rows 6..8 last used, 9..11 now. +layout(set = 1, binding = 2) buffer Voices { vec4 voices[]; }; +// layouts.Lead: bodies 0..2, voices 3..5, coupling 6..8, (energy, time) 9. +layout(set = 1, binding = 3) buffer Lead { vec4 lead[]; }; + +// Match layouts.zig. +const uint TRAIL = 1024; + +void main() { + uint s = gl_GlobalInvocationID.x; + if (s >= u.systems) return; + vec3 m = u.masses.xyz; + vec2 p[3]; + vec2 v[3]; + for (int i = 0; i < 3; i++) { + vec4 b = s == 0u && u.adopt != 0u ? ahead[3u * (u.adopt - 1u) + uint(i)] : bodies[3u * s + uint(i)]; + p[i] = b.xy; + v[i] = b.zw; + } + vec2 a[3]; + accelerations(p, m, u.softening, a); + // System 0 leaves trail points as it goes, by simulated time, so a fast + // travel draws as smooth a path as running does. + uint written = 0u; + for (uint step = 0u; step < u.steps; step++) { + leapfrog(p, v, a, u.dt, m, u.softening, 512.0); + if (s == 0u && written < u.trail_points && step + 1u >= u.trail_first && (step + 1u - u.trail_first) % u.trail_every == 0u) { + for (int i = 0; i < 3; i++) trail[uint(i) * TRAIL + (u.trail_head + written) % TRAIL] = p[i]; + written++; + } + } + for (int i = 0; i < 3; i++) bodies[3u * s + uint(i)] = vec4(p[i], v[i]); + if (s != 0u) return; + + float kinetic = 0.0; + float potential = 0.0; + for (int i = 0; i < 3; i++) { + kinetic += 0.5 * m[i] * dot(v[i], v[i]); + for (int j = i + 1; j < 3; j++) potential -= m[i] * m[j] * inversesqrt(dot(p[j] - p[i], p[j] - p[i]) + u.softening * u.softening); + lead[i] = vec4(p[i], v[i]); + } + + vec2 sp[3]; + vec2 sv[3]; + for (int i = 0; i < 3; i++) { + vec4 b = u.voice_source != 0u ? ahead[3u * (u.voice_source - 1u) + uint(i)] : vec4(p[i], v[i]); + sp[i] = b.xy; + sv[i] = b.zw; + } + vec4 sung[3]; + vec4 coupled[3]; + sing(sp, sv, Mapping(u.base, u.speed_ref, u.depth_scale, u.couple_scale, u.quantize), sung, coupled); + for (int i = 0; i < 3; i++) { + voices[3 + i] = sung[i]; + lead[3 + i] = sung[i]; + // A quarter of the way each frame: pairs hand over smoothly. + vec4 row = mix(voices[9 + i], coupled[i], 0.25); + voices[9 + i] = row; + lead[6 + i] = row; + } + lead[9] = vec4(kinetic + potential, lead[9].y + float(u.steps) * u.dt, 0.0, 0.0); +} diff --git a/examples/spheres/shaders/sprite.frag b/examples/spheres/shaders/sprite.frag new file mode 100644 index 0000000..d39e7e4 --- /dev/null +++ b/examples/spheres/shaders/sprite.frag @@ -0,0 +1,11 @@ +#version 450 +// A Gaussian glow, premultiplied, for additive blending. +layout(location = 0) in vec2 corner; +layout(location = 1) in vec4 color; +layout(location = 0) out vec4 out_color; + +void main() { + float r2 = dot(corner, corner); + float glow = exp(-5.0 * r2) * (1.0 - smoothstep(0.8, 1.0, r2)); + out_color = color * glow; +} diff --git a/examples/spheres/shaders/sprite.vert b/examples/spheres/shaders/sprite.vert new file mode 100644 index 0000000..13e1c73 --- /dev/null +++ b/examples/spheres/shaders/sprite.vert @@ -0,0 +1,23 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// A soft disc per body: of every system from `first` for `count` systems, +// at `radius` pixels, coloured by body. Six vertices make each quad. +layout(location = 0) out vec2 corner; +layout(location = 1) out vec4 color; + +#include "SkyView.glsl" + +layout(set = 0, binding = 0) readonly buffer Bodies { vec4 bodies[]; }; + +const vec2 corners[6] = vec2[](vec2(-1, -1), vec2(1, -1), vec2(1, 1), vec2(-1, -1), vec2(1, 1), vec2(-1, 1)); +// Match main.zig's palette. +const vec3 palette[3] = vec3[](vec3(0.40, 0.85, 1.0), vec3(1.0, 0.45, 0.82), vec3(1.0, 0.80, 0.35)); + +void main() { + uint sprite = uint(gl_VertexIndex) / 6u; + uint index = 3u * u.first + sprite; + vec2 at = (bodies[index].xy - u.center) / u.per_pixel + corners[uint(gl_VertexIndex) % 6u] * u.radius; + gl_Position = vec4(at * 2.0 / u.size, 0.0, 1.0); + corner = corners[uint(gl_VertexIndex) % 6u]; + color = vec4(palette[index % 3u], 1.0) * u.alpha; +} diff --git a/examples/spheres/shaders/synth.comp b/examples/spheres/shaders/synth.comp new file mode 100644 index 0000000..3a6c40c --- /dev/null +++ b/examples/spheres/shaders/synth.comp @@ -0,0 +1,112 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// One audio block, one thread per sample: three voices from the bodies' +// parameters, gliding from what the last block ended on to what sim.comp +// set now, so blocks join without clicks. +// +// Each phase has a closed form, so every sample is computed alone: a +// frequency gliding linearly from w0 to w1 (radians per sample) over N +// samples has phase p0 + w0 n + (w1 - w0) n^2 / 2N after n of them. +// +// The modes, with carrier phase c, modulator phase m (at ratio times the +// carrier's frequency) and depth d: +// sine sin c +// additive sum over harmonics k of r^(k-1) sin kc, r = d / (d + 2), +// those under Nyquist, normalised +// AM sin c (1 + a sin m) / (1 + a), a = min(d / 4, 1) +// ring sin c sin m +// FM sin of the integral of the carrier's frequency, pushed +// d times the carrier's frequency either way by cos m: so the +// index, deviation over modulator frequency, is d / ratio and +// changes as the modulator does +// PM sin(c + d sin m): the index is d, whatever the modulator +// FM and PM sound alike for a steady modulator, and part when it moves. +// +// The closest pair (sim.comp) is one FM voice: the modulating body is +// silent, and a sine at its frequency, times the index, pushes the other +// body's carrier phase. Two operators, both moved by the bodies. The push +// moves every harmonic of the additive voice alike, so its timbre bends +// without turning harsh. +layout(local_size_x = 64) in; + +#include "SynthParams.glsl" + +layout(set = 0, binding = 0) readonly buffer Voices { vec4 voices[]; }; +// Per voice: carrier phase, modulator phase, FM's phase. +layout(set = 0, binding = 1) readonly buffer Phases { vec4 phases[]; }; +layout(set = 1, binding = 0) buffer Block { float block[]; }; + +const float TAU = 6.28318530718; +// Match layouts.Mode. +const uint SINE = 0u; +const uint ADDITIVE = 1u; +const uint AM = 2u; +const uint RING = 3u; +const uint FM = 4u; + +float glide(float phase, float w0, float w1, float n, float count) { + return phase + w0 * n + (w1 - w0) * n * n / (2.0 * count); +} + +// Voice i's signal, its carrier phase pushed by `push` radians. +float voice(int i, float n, float count, float push) { + vec4 from = voices[i]; + vec4 to = voices[3 + i]; + vec4 phase = phases[i]; + float t = n / count; + float wc0 = TAU * from.x / u.rate; + float wc1 = TAU * to.x / u.rate; + float c = glide(phase.x, wc0, wc1, n, count); + float m = glide(phase.y, wc0 * from.y, wc1 * to.y, n, count); + float depth = mix(from.z, to.z, t); + float ratio = mix(from.y, to.y, t); + float y; + if (u.mode == SINE) { + y = sin(c + push); + } else if (u.mode == ADDITIVE) { + float r = depth / (depth + 2.0); + float carrier = mix(from.x, to.x, t); + float sum = 0.0; + float norm = 0.0; + float weight = 1.0; + for (int k = 1; k <= 16; k++) { + if (float(k) * carrier >= 0.5 * u.rate) break; + sum += weight * sin(float(k) * c + push); + norm += weight; + weight *= r; + } + y = sum / max(norm, 1e-6); + } else if (u.mode == AM) { + float a = min(depth / 4.0, 1.0); + y = sin(c + push) * (1.0 + a * sin(m)) / (1.0 + a); + } else if (u.mode == RING) { + y = sin(c + push) * sin(m); + } else if (u.mode == FM) { + y = sin(phase.z + push + (c - phase.x) + depth / max(ratio, 1e-3) * (sin(m) - sin(phase.y))); + } else { + y = sin(c + push + depth * sin(m)); + } + return y; +} + +void main() { + uint n = gl_GlobalInvocationID.x; + if (n >= u.count) return; + float count = float(u.count); + float t = float(n) / count; + // Each body as a modulator: a sine at its carrier frequency. + vec3 sines; + for (int i = 0; i < 3; i++) { + float w0 = TAU * voices[i].x / u.rate; + float w1 = TAU * voices[3 + i].x / u.rate; + sines[i] = sin(glide(phases[i].x, w0, w1, float(n), count)); + } + float sum = 0.0; + for (int i = 0; i < 3; i++) { + vec4 row = mix(voices[6 + i], voices[9 + i], t); + float audible = (1.0 - row.w) * mix(voices[i].w, voices[3 + i].w, t); + if (audible < 1e-4) continue; + sum += audible * voice(i, float(n), count, dot(row.xyz, sines)); + } + block[n] = tanh(u.gain * sum); +} diff --git a/examples/spheres/shaders/trail.frag b/examples/spheres/shaders/trail.frag new file mode 100644 index 0000000..8a82628 --- /dev/null +++ b/examples/spheres/shaders/trail.frag @@ -0,0 +1,7 @@ +#version 450 +layout(location = 0) in vec4 color; +layout(location = 0) out vec4 out_color; + +void main() { + out_color = color; +} diff --git a/examples/spheres/shaders/trail.vert b/examples/spheres/shaders/trail.vert new file mode 100644 index 0000000..7f9f35c --- /dev/null +++ b/examples/spheres/shaders/trail.vert @@ -0,0 +1,48 @@ +#version 450 +#extension GL_GOOGLE_include_directive : require +// System 0's trails: per body, a ribbon through its last positions, +// `radius` pixels wide at the newest and tapering to nothing at the oldest, +// fading as it goes. Six vertices per segment; segments past what has been +// filled, or across a jump, collapse to nothing. +layout(location = 0) out vec4 color; + +#include "SkyView.glsl" + +layout(set = 0, binding = 0) readonly buffer Trail { vec2 trail[]; }; + +// Match layouts.zig and main.zig's palette. +const uint TRAIL = 1024u; +const vec3 palette[3] = vec3[](vec3(0.40, 0.85, 1.0), vec3(1.0, 0.45, 0.82), vec3(1.0, 0.80, 0.35)); + +void main() { + uint segment = uint(gl_VertexIndex) / 6u; + uint corner = uint(gl_VertexIndex) % 6u; + uint body = segment / (TRAIL - 1u); + uint age = segment % (TRAIL - 1u); + if (age + 1u >= u.filled) { + gl_Position = vec4(0.0, 0.0, 2.0, 1.0); + color = vec4(0.0); + return; + } + // Corners 0, 3, 5 at the newer end, 1, 2, 4 at the older; 0, 1, 3 on + // one side, 2, 4, 5 on the other. + bool older = corner == 1u || corner == 2u || corner == 4u; + bool side = corner == 2u || corner == 4u || corner == 5u; + uint a = (u.head + TRAIL - age) % TRAIL; + uint b = (u.head + TRAIL - age - 1u) % TRAIL; + // A jump (system 0 stepping into a snapshot) breaks the trail. + if (distance(trail[body * TRAIL + a], trail[body * TRAIL + b]) > u.gap) { + gl_Position = vec4(0.0, 0.0, 2.0, 1.0); + color = vec4(0.0); + return; + } + vec2 pa = (trail[body * TRAIL + a] - u.center) / u.per_pixel; + vec2 pb = (trail[body * TRAIL + b] - u.center) / u.per_pixel; + vec2 along = pb - pa; + float len = length(along); + vec2 normal = len > 1e-4 ? vec2(-along.y, along.x) / len : vec2(0.0, 1.0); + float fade = 1.0 - float(older ? age + 1u : age) / float(u.filled); + vec2 at = (older ? pb : pa) + normal * (side ? 1.0 : -1.0) * u.radius * fade; + gl_Position = vec4(at * 2.0 / u.size, 0.0, 1.0); + color = vec4(palette[body], 1.0) * u.alpha * fade * fade; +} diff --git a/examples/spheres/shaders/voices.glsl b/examples/spheres/shaders/voices.glsl new file mode 100644 index 0000000..c2fd56c --- /dev/null +++ b/examples/spheres/shaders/voices.glsl @@ -0,0 +1,101 @@ +// From motion to sound, shared by sim.comp (what sounds) and lookahead.comp +// (what would): each body a voice whose pitch follows its speed, whose +// modulator ratio follows how lopsided its two distances are, and whose +// depth follows how near its nearest neighbour is; and the closest pair one +// FM voice, the faster body silent, modulating the other. + +struct Mapping { + float base; + float speed_ref; + float depth_scale; + float couple_scale; + // Match layouts.Scale: free, pentatonic, diatonic, chromatic. + uint scale; +}; + +// The nearest note of the scale, as a MIDI number: A minor pentatonic (A C +// D E G), A minor / C major (the white keys), every semitone, or none. +float snap(float midi, uint scale) { + if (scale == 0u) return midi; + if (scale == 3u) return round(midi); + float m = midi - 69.0; + float octave = floor(m / 12.0); + float degree = m - 12.0 * octave; + const float pentatonic[6] = float[](0.0, 3.0, 5.0, 7.0, 10.0, 12.0); + const float diatonic[8] = float[](0.0, 2.0, 3.0, 5.0, 7.0, 8.0, 10.0, 12.0); + float best = 0.0; + if (scale == 1u) { + for (int i = 1; i < 6; i++) if (abs(pentatonic[i] - degree) < abs(best - degree)) best = pentatonic[i]; + } else { + for (int i = 1; i < 8; i++) if (abs(diatonic[i] - degree) < abs(best - degree)) best = diatonic[i]; + } + return 69.0 + 12.0 * octave + best; +} + +// Voices: (carrier Hz, ratio, depth, amplitude). Coupling rows: xyz the FM +// index each voice gets from each other, w whether it is the silent +// modulator. +void sing(vec2 p[3], vec2 v[3], Mapping map, out vec4 voices[3], out vec4 coupling[3]) { + for (int i = 0; i < 3; i++) { + float speed = length(v[i]); + float octaves = clamp(log2(max(speed, 1e-4) / map.speed_ref), -2.0, 2.5); + float midi = 12.0 * log2(map.base / 440.0) + 69.0 + 12.0 * octaves; + midi = snap(midi, map.scale); + float d1 = distance(p[i], p[(i + 1) % 3]); + float d2 = distance(p[i], p[(i + 2) % 3]); + float near = min(d1, d2); + float far = max(d1, d2); + voices[i] = vec4( + 440.0 * exp2((midi - 69.0) / 12.0), + 0.5 + 4.0 * near / max(near + far, 1e-6), + clamp(map.depth_scale / max(near, 1e-3), 0.0, 12.0), + 0.2 + 0.15 * clamp(speed / (2.0 * map.speed_ref), 0.0, 1.0)); + coupling[i] = vec4(0.0); + } + int first = 0; + int second = 1; + float closest = distance(p[0], p[1]); + if (distance(p[0], p[2]) < closest) { first = 0; second = 2; closest = distance(p[0], p[2]); } + if (distance(p[1], p[2]) < closest) { first = 1; second = 2; closest = distance(p[1], p[2]); } + int modulator = length(v[first]) > length(v[second]) ? first : second; + int carrier = modulator == first ? second : first; + coupling[carrier][modulator] = clamp(map.couple_scale / max(closest, 1e-3), 0.0, 4.0); + coupling[modulator].w = 1.0; +} + +// The chord with its voicing, packed: bits 0-11 the pitch classes (the +// chord), bit 12 set if it is playable, bits 16-19 the lowest note's pitch +// class (for root position), bits 24-31 how bad it sounds, 0 to 255. +// Playable: every note from E2 to A5, an octave and a fifth at most from +// lowest to highest, and the FM pair's index at most 3; past those, the +// same pitch classes are another sound, shrill, sprawling or a cloud of +// sidebands. Within them, wider, higher and harder modulated cost more. +uint voicing(vec4 voices[3], vec4 coupling[3]) { + int low = 1000; + int high = -1000; + int lowest_class = 0; + for (int i = 0; i < 3; i++) { + int midi = int(round(69.0 + 12.0 * log2(voices[i].x / 440.0))); + if (midi < low) { + low = midi; + lowest_class = ((midi % 12) + 12) % 12; + } + high = max(high, midi); + } + float index = 0.0; + for (int i = 0; i < 3; i++) index = max(index, max(coupling[i].x, max(coupling[i].y, coupling[i].z))); + int spread = high - low; + bool playable = low >= 40 && high <= 81 && spread <= 19 && index <= 3.0; + float bad = 6.0 * float(max(spread - 12, 0)) + 10.0 * float(max(high - 76, 0)) + 15.0 * index; + return (playable ? 1u << 12 : 0u) | (uint(lowest_class) << 16) | (uint(clamp(bad, 0.0, 255.0)) << 24); +} + +// The pitch classes the three voices sing, as bits (C is bit 0): the chord. +uint chordOf(vec4 voices[3]) { + uint mask = 0u; + for (int i = 0; i < 3; i++) { + int midi = int(round(69.0 + 12.0 * log2(voices[i].x / 440.0))); + mask |= 1u << uint(((midi % 12) + 12) % 12); + } + return mask; +} diff --git a/examples/spheres/src/chords.zig b/examples/spheres/src/chords.zig new file mode 100644 index 0000000..54a4413 --- /dev/null +++ b/examples/spheres/src/chords.zig @@ -0,0 +1,311 @@ +//! spheres as an instrument: the CPU's side of the look-ahead. The GPU +//! (lookahead.comp) integrates `alternatives` futures of system 0 from the +//! preset's start, a few samples a frame, keeping a snapshot every +//! `sample_dt` and the chord sung there: the pitch classes of the three +//! voices (the silent modulator's counts: it sets the carrier's sidebands). +//! The chords come back here, a frame or two later; the ones that fit the +//! grid of buttons (a sanfona's bass side, see `Grid`) light up as found, +//! and choosing one finds the nearest time it plays, in whichever future +//! plays it soonest. +const std = @import("std"); +const layouts = @import("layouts"); + +/// The mapping from motion to voices (voices.glsl), which the shaders get +/// as uniforms; and leapfrog's base step, as physics.glsl's callers use it. +pub const base: f32 = 220; +pub const speed_ref: f32 = 0.6; +pub const depth_scale: f32 = 1.2; +pub const couple_scale: f32 = 0.9; +pub const base_step = 0.004; + +/// Futures integrated side by side: 0 is system 0's own, the rest nudged. +pub const alternatives = layouts.alternatives; +pub const horizon = layouts.horizon; +pub const sample_dt = 0.05; +/// Samples the look-ahead advances each frame: 0.4 s of simulated time, +/// against the live system's 0.7 s a second, so it pulls well ahead. +pub const per_frame = 8; +/// What a pick costs besides its travel time, in seconds of travel: being +/// in another future than the system's own; not having the root in the +/// bass; and per unit of voicing badness (voices.glsl's 0 to 255: a +/// sprawling, high or hard-modulated voicing costs up to 3 s more). +const elsewhere = 0.5; +const inverted = 0.6; +const per_bad = 3.0 / 255.0; + +/// A snapshot's chord and voicing, as voices.glsl's `voicing` packs them. +pub const Sung = packed struct(u32) { + chord: u12, + playable: bool, + _: u3, + lowest: u4, + _2: u4, + bad: u8, +}; + +/// The pitch classes three voices sing, as bits (C is bit 0): as +/// voices.glsl's chordOf, for what is sounding. +pub fn chordOf(voices: [3]layouts.Voice) u16 { + var mask: u16 = 0; + for (voices) |v| { + if (v.carrier <= 0) continue; + const midi: i32 = @intFromFloat(@round(69 + 12 * std.math.log2(v.carrier / 440))); + mask |= @as(u16, 1) << @intCast(@mod(midi, 12)); + } + return mask; +} + +const names = [12][]const u8{ "C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B" }; + +fn bits(intervals: []const u4) u16 { + var m: u16 = 0; + for (intervals) |i| m |= @as(u16, 1) << i; + return m; +} + +/// Chord shapes, most wanted first, as intervals over the root. +const shapes = [_]struct { u16, []const u8 }{ + .{ bits(&.{ 0, 4, 7 }), "" }, + .{ bits(&.{ 0, 3, 7 }), "m" }, + .{ bits(&.{ 0, 2, 7 }), "sus2" }, + .{ bits(&.{ 0, 5, 7 }), "sus4" }, + .{ bits(&.{ 0, 4, 10 }), "7" }, + .{ bits(&.{ 0, 3, 10 }), "m7" }, + .{ bits(&.{ 0, 4, 9 }), "6" }, + .{ bits(&.{ 0, 3, 9 }), "m6" }, + .{ bits(&.{ 0, 3, 6 }), "dim" }, + .{ bits(&.{ 0, 4, 8 }), "aug" }, + .{ bits(&.{ 0, 2, 4 }), "add9" }, + .{ bits(&.{ 0, 5, 10 }), "7sus4" }, + .{ bits(&.{ 0, 7, 10 }), "7no3" }, + .{ bits(&.{ 0, 7 }), "5" }, +}; + +/// A name for a set of pitch classes: a chord where one fits ("Am", +/// "Csus2"), else the notes ("A+C+D"). +pub fn name(buf: []u8, mask: u16) []const u8 { + if (mask == 0) return ""; + if (@popCount(mask) == 1) return names[@ctz(mask)]; + for (shapes) |shape| for (0..12) |root| { + if (mask & (@as(u16, 1) << @intCast(root)) == 0) continue; + const r: u4 = @intCast(root); + const rotated = ((mask >> r) | (mask << @intCast(12 - @as(u5, r)))) & 0xfff; + if (rotated == shape[0]) return std.fmt.bufPrint(buf, "{s}{s}", .{ names[root], shape[1] }) catch "?"; + }; + var w: std.Io.Writer = .fixed(buf); + var first = true; + for (0..12) |pc| if (mask & (@as(u16, 1) << @intCast(pc)) != 0) { + w.print("{s}{s}", .{ if (first) "" else "+", names[pc] }) catch break; + first = false; + }; + return w.buffered(); +} + +/// A snapshot: sample `sample` of future `alternative`, at `index` in the +/// GPU's snapshot buffer. +pub const Pick = struct { + sample: u32, + alternative: u32, + + pub fn index(p: Pick) u32 { + return p.sample * alternatives + p.alternative; + } + pub fn time(p: Pick) f64 { + return @as(f64, @floatFromInt(p.sample)) * sample_dt; + } +}; + +/// The buttons, laid out as a sanfona's (Stradella) bass side: columns +/// round the circle of fifths, so neighbours are related chords, and rows +/// by quality, each half a button over from the one above, so a progression +/// is the same shape of hand in any key. +pub const Grid = struct { + pub const Quality = enum { major, minor, seventh, sus, power }; + pub const rows = @typeInfo(Quality).@"enum".field_names.len; + pub const columns = 12; + /// Roots by column, by fifths from G flat to B, as pitch classes. + pub const roots = [columns]u4{ 6, 1, 8, 3, 10, 5, 0, 7, 2, 9, 4, 11 }; + pub const root_names = [columns][]const u8{ "Gb", "Db", "Ab", "Eb", "Bb", "F", "C", "G", "D", "A", "E", "B" }; + + /// Three voices make triads at most: the seventh without its fifth. + pub fn shape(q: Quality) u16 { + return switch (q) { + .major => bits(&.{ 0, 4, 7 }), + .minor => bits(&.{ 0, 3, 7 }), + .seventh => bits(&.{ 0, 4, 10 }), + .sus => bits(&.{ 0, 5, 7 }), + .power => bits(&.{ 0, 7 }), + }; + } + + pub fn suffix(q: Quality) []const u8 { + return switch (q) { + .major => "", + .minor => "m", + .seventh => "7", + .sus => "sus", + .power => "5", + }; + } + + /// The chord of the button at `row`, `column`. + pub fn chord(row: usize, column: usize) u16 { + const s = shape(@enumFromInt(row)); + const r: u4 = roots[column]; + return ((s << r) | (s >> @intCast(12 - @as(u5, r)))) & 0xfff; + } + + /// The button playing `mask`, if one does. + pub fn find(mask: u16) ?struct { row: usize, column: usize } { + for (0..rows) |row| for (0..columns) |column| { + if (chord(row, column) == mask) return .{ .row = row, .column = column }; + }; + return null; + } +}; + +pub const Ahead = struct { + /// The chords back from the GPU, with their voicings, sample-major as + /// it keeps them. + sung: [horizon][alternatives]Sung = undefined, + /// Samples asked of the GPU, and how many have come back. + asked: u32 = 0, + known: u32 = 0, + /// Every chord found in a playable voicing, by mask, and how many. + found: [1 << 12]bool = @splat(false), + distinct: u32 = 0, + + pub fn reset(a: *Ahead) void { + a.asked = 0; + a.known = 0; + a.forget(); + } + + /// The chords again from nothing (another scale). + pub fn forget(a: *Ahead) void { + a.known = 0; + a.found = @splat(false); + a.distinct = 0; + } + + /// Chords back from the GPU: samples from `first`, all alternatives. + pub fn arrived(a: *Ahead, first: u32, words: []const u32) void { + const samples: u32 = @intCast(words.len / alternatives); + for (0..samples) |s| { + const t = first + @as(u32, @intCast(s)); + if (t >= horizon) break; + for (0..alternatives) |alt| { + const sung: Sung = @bitCast(words[s * alternatives + alt]); + a.sung[t][alt] = sung; + if (sung.playable) a.pad(sung.chord); + } + a.known = @max(a.known, t + 1); + } + } + + fn pad(a: *Ahead, chord: u16) void { + if (chord == 0 or a.found[chord]) return; + a.found[chord] = true; + a.distinct += 1; + } + + /// The best snapshot where `chord` plays in a playable voicing: near + /// `time`, in the system's own future, with `root` in the bass, and + /// sounding well, weighed against each other (see the costs above). + pub fn nearest(a: *const Ahead, chord: u16, root: u4, time: f64) ?Pick { + var best: ?Pick = null; + var cost: f64 = std.math.inf(f64); + for (a.sung[0..a.known], 0..) |row, t| for (row, 0..) |sung, alt| { + if (sung.chord != chord or !sung.playable) continue; + const pick: Pick = .{ .sample = @intCast(t), .alternative = @intCast(alt) }; + const c = @abs(pick.time() - time) + + @as(f64, if (alt == 0) 0 else elsewhere) + + @as(f64, if (sung.lowest == root) 0 else inverted) + + per_bad * @as(f64, @floatFromInt(sung.bad)); + if (c < cost) { + cost = c; + best = pick; + } + }; + return best; + } +}; + +test "chords by name" { + var buf: [32]u8 = undefined; + const set = struct { + fn of(pcs: []const u5) u16 { + var m: u16 = 0; + for (pcs) |p| m |= @as(u16, 1) << @intCast(p); + return m; + } + }.of; + try std.testing.expectEqualStrings("Am", name(&buf, set(&.{ 9, 0, 4 }))); + try std.testing.expectEqualStrings("C", name(&buf, set(&.{ 0, 4, 7 }))); + try std.testing.expectEqualStrings("Csus2", name(&buf, set(&.{ 0, 2, 7 }))); + try std.testing.expectEqualStrings("A5", name(&buf, set(&.{ 9, 4 }))); + try std.testing.expectEqualStrings("E", name(&buf, set(&.{4}))); + try std.testing.expectEqualStrings("C#+D", name(&buf, set(&.{ 1, 2 }))); +} + +fn word(chord: u12, playable: bool, lowest: u4, bad: u8) u32 { + return @bitCast(Sung{ .chord = chord, .playable = playable, ._ = 0, .lowest = lowest, ._2 = 0, .bad = bad }); +} + +test "a pick prefers the system's own future, unless another plays the chord much sooner" { + var a: Ahead = .{}; + var words: [3 * alternatives]u32 = @splat(word(1, true, 0, 0)); + // Chord 2 at sample 2 of future 0, and at sample 0 of future 5. + words[2 * alternatives + 0] = word(2, true, 1, 0); + words[0 * alternatives + 5] = word(2, true, 1, 0); + a.arrived(0, &words); + try std.testing.expectEqual(@as(u32, 3), a.known); + try std.testing.expect(a.found[1] and a.found[2]); + try std.testing.expectEqual(@as(u32, 2), a.distinct); + // From time 0.1: future 0 at 0.1 away beats future 5 at 0.1 + 0.5. + try std.testing.expectEqual(Pick{ .sample = 2, .alternative = 0 }, a.nearest(2, 1, 0.1).?); + // Ask for a chord no future plays: nothing. + try std.testing.expectEqual(@as(?Pick, null), a.nearest(4, 2, 0)); +} + +test "voicings: unplayable ones do not count, good ones win over near ones" { + var a: Ahead = .{}; + var words: [3 * alternatives]u32 = @splat(word(1, true, 0, 0)); + // Chord 2 now, but shrill and sprawling (unplayable); later, playable + // but inverted and harsh; later still, in root position and clean. + words[0] = word(2, false, 1, 0); + words[1 * alternatives] = word(2, true, 3, 200); + words[2 * alternatives] = word(2, true, 1, 0); + a.arrived(0, &words); + try std.testing.expectEqual(Pick{ .sample = 2, .alternative = 0 }, a.nearest(2, 1, 0).?); + // A chord heard only unplayably is not found. + var b: Ahead = .{}; + var only: [alternatives]u32 = @splat(word(4, false, 2, 0)); + b.arrived(0, &only); + try std.testing.expect(!b.found[4]); +} + +test "the grid: fifths across, qualities down" { + var buf: [16]u8 = undefined; + // Column 6 is C, column 9 is A. + try std.testing.expectEqualStrings("C", name(&buf, Grid.chord(0, 6))); + try std.testing.expectEqualStrings("Am", name(&buf, Grid.chord(1, 9))); + try std.testing.expectEqualStrings("G5", name(&buf, Grid.chord(4, 7))); + const at = Grid.find(Grid.chord(2, 3)).?; + try std.testing.expectEqual(@as(usize, 2), at.row); + try std.testing.expectEqual(@as(usize, 3), at.column); + // Every button its own chord. + for (0..Grid.rows) |r| for (0..Grid.columns) |c| { + try std.testing.expectEqual(r, Grid.find(Grid.chord(r, c)).?.row); + }; +} + +test "voices to chords" { + const v = [3]layouts.Voice{ + .{ .carrier = 220, .ratio = 1, .depth = 0, .amp = 1 }, + .{ .carrier = 261.63, .ratio = 1, .depth = 0, .amp = 1 }, + .{ .carrier = 329.63, .ratio = 1, .depth = 0, .amp = 1 }, + }; + var buf: [16]u8 = undefined; + try std.testing.expectEqualStrings("Am", name(&buf, chordOf(v))); +} diff --git a/examples/spheres/src/layouts.zig b/examples/spheres/src/layouts.zig new file mode 100644 index 0000000..c282e16 --- /dev/null +++ b/examples/spheres/src/layouts.zig @@ -0,0 +1,163 @@ +//! What spheres' shaders read and write, as Zig structs: the only +//! definition. The shaders include GLSL generated from the uniform blocks; +//! storage layouts (std430) are mirrored by hand where noted. No imports. + +/// Copies of the system integrated side by side: system 0 is the one heard +/// and drawn in full, the rest start a hair apart and show where it could +/// have gone. +pub const max_systems = 16384; +pub const bodies = 3; +/// Positions kept per body for the trail. +pub const trail = 1024; +/// The most samples synthesised in one frame. +pub const max_block = 8192; +/// The look-ahead (lookahead.comp): futures of system 0 integrated side by +/// side, and snapshots kept of each, one every `chords.sample_dt`. +pub const alternatives = 32; +pub const horizon = 2400; + +/// One body of one system (std430: vec4). +pub const Body = extern struct { + pos: [2]f32, + vel: [2]f32, +}; + +/// One voice's synthesis parameters (std430: vec4). +pub const Voice = extern struct { + /// Hz. + carrier: f32, + /// Modulator over carrier frequency. + ratio: f32, + /// The modulation depth: PM's index; FM's deviation over the carrier; + /// AM's and the additive voice's brightness. + depth: f32, + amp: f32, +}; + +/// The voices buffer, in vec4s: what the synthesiser last used and what +/// sim.comp set now, for the voices and the coupling between them. +pub const voices_previous = 0; +pub const voices_now = 3; +/// Row i of the coupling: how much each voice's signal (x, y, z) pushes +/// voice i's carrier phase. Only the closest pair couples. +pub const coupling_previous = 6; +pub const coupling_now = 9; +pub const voices_len = 12; + +/// What sim.comp reports of system 0, for the CPU (std430). +pub const Lead = extern struct { + bodies: [3]Body, + voices: [3]Voice, + /// Rows of the coupling, as in the voices buffer (w unused). + coupling: [3][4]f32, + /// Kinetic plus potential. + energy: f32, + /// Simulated time. + time: f32, + pad: [2]f32 = .{ 0, 0 }, +}; + +pub const Mode = enum(u32) { sine, additive, am, ring, fm, pm }; + +/// Where pitches snap; voices.glsl's `snap`. The uniforms carry it as a u32. +pub const Scale = enum(u32) { free, pentatonic, diatonic, chromatic }; + +/// sim.comp: one thread per system. +pub const SimParams = extern struct { + systems: u32, + /// Substeps this frame, each `dt` (negative runs time backwards), + /// subdivided further where bodies pass close. + steps: u32, + dt: f32, + softening: f32, + masses: [4]f32, + /// Where system 0's positions go in the trail ring; past `trail`, none. + trail_head: u32, + quantize: u32, + /// The pitch at `speed_ref`, Hz. + base: f32, + speed_ref: f32, + /// Modulation depth at unit distance. + depth_scale: f32, + /// How hard the closest pair couples at unit distance. + couple_scale: f32, + /// A look-ahead snapshot, as its index plus one, or 0: system 0 sings + /// from it (a held chord) rather than from its own motion. + voice_source: u32 = 0, + /// Likewise: system 0 takes the snapshot's state before moving on. + adopt: u32 = 0, + /// Trail points this frame, by simulated time rather than by frame, so + /// a fast travel leaves as smooth a path as running does: after step + /// `trail_first` (1-based), then every `trail_every` steps, `trail_points` + /// of them, from slot `trail_head` on. + trail_points: u32 = 0, + trail_first: u32 = 1, + trail_every: u32 = 1, + pad0: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// lookahead.comp: alternatives `count` samples on from `first`; and +/// rechord.comp: the chords of the first `count` snapshots again. +pub const LookParams = extern struct { + alternatives: u32, + first: u32, + count: u32, + horizon: u32, + masses: [4]f32, + softening: f32, + quantize: u32, + base: f32, + speed_ref: f32, + depth_scale: f32, + couple_scale: f32, + sample_dt: f32, + pad: u32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// synth.comp: one thread per sample. +pub const SynthParams = extern struct { + count: u32, + /// A `Mode`. + mode: u32, + rate: f32, + gain: f32, + + pub const glsl = .{ .uniform = .{ .set = 2, .instance = "u" } }; +}; + +/// The sky's camera and sprites (vertex uniforms). +pub const SkyView = extern struct { + /// World point at the image's centre, and world units per pixel. + center: [2]f32, + size: [2]f32, + per_pixel: f32, + radius: f32, + alpha: f32, + first: u32, + count: u32, + /// Trail: where the newest point is, and how many are filled. + head: u32, + filled: u32, + /// Seconds, for what flows. + time: f32, + /// Trail segments longer than this (world units) are jumps, not motion + /// (system 0 stepping into a snapshot): not drawn. + gap: f32 = 1e9, + pad0: f32 = 0, + pad1: f32 = 0, + pad2: f32 = 0, + + pub const glsl = .{ .uniform = .{ .set = 1, .instance = "u" } }; +}; + +/// fade.frag: moves what is there `amount` of the way to `background`. +pub const Fade = extern struct { + background: [3]f32, + amount: f32, + + pub const glsl = .{ .uniform = .{ .set = 3, .instance = "u" } }; +}; diff --git a/examples/spheres/src/main.zig b/examples/spheres/src/main.zig new file mode 100644 index 0000000..60b498f --- /dev/null +++ b/examples/spheres/src/main.zig @@ -0,0 +1,1134 @@ +//! spheres: the three-body problem, heard. Three bodies attract each other; +//! the GPU integrates thousands of copies of them at once, each a hair +//! apart, and draws them as a cloud around the one it follows in full, so +//! chaos shows as the cloud spreading (and a stable orbit as it staying a +//! thin line). Each body is also a voice, synthesised on the GPU from its +//! motion: speed is pitch, the balance of its two distances is the +//! modulator's ratio, and the nearness of its closest neighbour is the +//! modulation depth, as sine, additive, AM, ring, FM or PM synthesis. The +//! scope shows the result: an oscilloscope, the spectrum with each voice's +//! predicted partials marked, and a spectrogram. +//! +//! Memory by lifetime: State is the preset, the transport and the +//! synthesis settings, changed by `update` and replayable. Session holds the +//! GPU objects, the sound on its way to the speakers and the scope. No +//! Store: a run is not worth keeping. +const std = @import("std"); +const sk = @import("sketch"); +const gpu = sk.abi.gpu; +const scope = @import("scope"); +const layouts = @import("layouts"); +const shaders = @import("shaders"); +const chords = @import("chords.zig"); + +pub const name = "spheres"; +pub const width = 1280; +pub const height = 800; + +/// The audio stream's key. +const stream = 1; +/// How much sound is kept queued ahead: synthesised, read back, queued. +const lead_seconds = 0.1; +/// Systems integrated: system 0 and its cloud. +const systems = layouts.max_systems; +/// How far apart the copies start. +const spread = 2e-4; +const background = 0x05070d; +/// Body colours; match the shaders' palette. +const palette = [3]u24{ 0x66d9ff, 0xff73d1, 0xffcc59 }; + +pub const Preset = enum { + figure_eight, + butterfly, + pythagorean, + lagrange, + random, + + fn label(p: Preset) []const u8 { + return switch (p) { + .figure_eight => "figure eight", + .butterfly => "butterfly", + .pythagorean => "Pythagorean", + .lagrange => "Lagrange triangle", + .random => "random", + }; + } +}; +pub const Mode = layouts.Mode; +pub const Scale = layouts.Scale; + +pub const State = struct { + preset: Preset = .figure_eight, + /// Starts the preset over, its copies spread with this seed. + reset: sk.Request(u32) = .{}, + running: bool = true, + backward: bool = false, + /// Simulated time per second: a little under real time, so the + /// look-ahead's share of each frame keeps it well ahead. + speed: f32 = 0.7, + mode: Mode = .additive, + /// Where pitches snap: the more notes, the more chords. + scale: Scale = .diatonic, + muted: bool = false, + /// The voice whose partials the spectrum marks in full. + focus: u8 = 0, + /// Played as an instrument: chord pads, each sending the sky to the + /// nearest time its chord plays and holding the chord. Off, it runs on + /// by itself. + instrument: bool = false, + /// A chord to play, as its pitch classes (a grid button's). + chord: sk.Request(u16) = .{}, +}; + +/// On the way to a pad's chord: where to, and how fast (simulated seconds +/// a second). +const Travel = struct { to: chords.Pick, speed: f64 }; + +pub const Session = struct { + reset: sk.Handled = .{}, + started: bool = false, + /// Synthesised samples not back from the GPU yet. + in_flight: usize = 0, + /// System 0 as last read back, and whether it has been. + lead: layouts.Lead = std.mem.zeroes(layouts.Lead), + have_lead: bool = false, + masses: [3]f32 = .{ 1, 1, 1 }, + softening: f32 = 0.01, + /// The trail's newest slot, how many are filled, and the steps since its + /// last point. + trail_head: u32 = 0, + trail_filled: u32 = 0, + trail_carry: u32 = 0, + /// The camera: world point at the sky's centre, and the half-extent it + /// shows, smoothed. + center: [2]f32 = .{ 0, 0 }, + extent: f32 = 1.5, + scope: scope.Scope = .{}, + /// Simulated time since the preset started. + time: f64 = 0, + chord: sk.Handled = .{}, + ahead: chords.Ahead = .{}, + /// The scale the look-ahead's chords were found with. + scale: Scale = .diatonic, + travel: ?Travel = null, + /// The snapshot whose chord is held. + held: ?chords.Pick = null, + /// Samples still to come back from before a chord was chosen, dropped. + discard: usize = 0, + gpu: Gpu = .{}, +}; + +const Gpu = struct { + ahead_state: sk.Buffer = .none, + snapshots: sk.Buffer = .none, + ahead_chords: sk.Buffer = .none, + ready: bool = false, + bodies: sk.Buffer = .none, + trail: sk.Buffer = .none, + voices: sk.Buffer = .none, + phases: sk.Buffer = .none, + lead: sk.Buffer = .none, + block: sk.Buffer = .none, + sky: sk.Image = .none, + overlay: sk.Image = .none, + size: [2]u32 = .{ 0, 0 }, + clear_sky: bool = true, +}; + +/// Readback keys; from `chords_from`, look-ahead chords from that sample. +const Readback = enum(u32) { audio = 1, lead = 2, _ }; +const chords_from = 0x10000; + +// --------------------------------------------------------------------------- +// Layout. + +const Layout = struct { + w: f32, + h: f32, + narrow: bool, + sky: sk.Rect, + voices: sk.Rect, + oscilloscope: sk.Rect, + spectrum: sk.Rect, + spectrogram: sk.Rect, + /// As an instrument: the look-ahead's chords along time, and the grid. + strip: sk.Rect = .{}, + grid: sk.Rect = .{}, + + const header = 36; + const Grid = chords.Grid; + /// The grid's columns on a phone: B flat to B, round C. + const narrow_columns = 8; + const narrow_first = 4; + /// Two rows of buttons, on a phone. + const toolbar = 104; + const per_row = buttons.len / 2; + + /// Wide: the sky on the left, the sound on the right, buttons on top. + /// Narrow (a phone): the sky, then the sound, buttons along the bottom. + fn of(w: f32, h: f32, instrument: bool) Layout { + const narrow = w < 900; + if (narrow) { + const sky_h = @round(@min(w, h * if (instrument) @as(f32, 0.27) else 0.42)); + const sky: sk.Rect = .{ .x = 0, .y = header, .w = w, .h = sky_h }; + var y = sky.y + sky.h; + var strip: sk.Rect = .{}; + var grid: sk.Rect = .{}; + if (instrument) { + strip = .{ .x = 8, .y = y + 6, .w = w - 16, .h = 10 }; + grid = .{ .x = 8, .y = strip.y + strip.h + 6, .w = w - 16, .h = Grid.rows * 36 - 4 }; + y = grid.y + grid.h; + } + const voices: sk.Rect = .{ .x = 8, .y = y + 6, .w = w - 16, .h = 54 }; + // As an instrument the grid takes the oscilloscope's room. + const oscilloscope: sk.Rect = if (instrument) .{ .x = 8, .y = voices.y + voices.h, .w = 0, .h = 0 } else .{ .x = 8, .y = voices.y + voices.h + 4, .w = w - 16, .h = 64 }; + const spectrum: sk.Rect = .{ .x = 8, .y = oscilloscope.y + oscilloscope.h + 6, .w = w - 16, .h = if (instrument) 90 else 120 }; + const rest = h - toolbar - (spectrum.y + spectrum.h + 22); + return .{ .w = w, .h = h, .narrow = true, .sky = sky, .voices = voices, .oscilloscope = oscilloscope, .spectrum = spectrum, .spectrogram = .{ .x = 8, .y = spectrum.y + spectrum.h + 18, .w = w - 16, .h = @max(rest, 40) }, .strip = strip, .grid = grid }; + } + const sky_w = @round((w - 48) * 0.58); + const full: sk.Rect = .{ .x = 16, .y = header + 8, .w = sky_w, .h = h - header - 8 - 28 }; + var sky = full; + var strip: sk.Rect = .{}; + var grid: sk.Rect = .{}; + if (instrument) { + const grid_h: f32 = Grid.rows * 30 - 4; + sky.h -= grid_h + 34; + strip = .{ .x = sky.x, .y = sky.y + sky.h + 8, .w = sky.w, .h = 10 }; + grid = .{ .x = sky.x, .y = strip.y + strip.h + 8, .w = sky.w, .h = grid_h }; + } + const x = sky.x + sky.w + 16; + const cw = w - 16 - x; + const voices: sk.Rect = .{ .x = x, .y = sky.y, .w = cw, .h = 66 }; + const oscilloscope: sk.Rect = .{ .x = x, .y = voices.y + voices.h + 6, .w = cw, .h = 110 }; + const spectrum: sk.Rect = .{ .x = x, .y = oscilloscope.y + oscilloscope.h + 10, .w = cw, .h = 210 }; + const spectrogram_y = spectrum.y + spectrum.h + 20; + return .{ .w = w, .h = h, .narrow = false, .sky = sky, .voices = voices, .oscilloscope = oscilloscope, .spectrum = spectrum, .spectrogram = .{ .x = x, .y = spectrogram_y, .w = cw, .h = full.y + full.h - spectrogram_y }, .strip = strip, .grid = grid }; + } + + /// The grid's columns on screen: all twelve, or a phone's eight. + fn columns(l: Layout) struct { first: usize, count: usize } { + return if (l.narrow) .{ .first = narrow_first, .count = narrow_columns } else .{ .first = 0, .count = Grid.columns }; + } + + /// The button at `row` and grid `column`, if on screen: rows staggered + /// half a button right each, as on a sanfona. + fn key(l: Layout, row: usize, column: usize) ?sk.Rect { + const cols = l.columns(); + if (column < cols.first or column >= cols.first + cols.count) return null; + const gap = 4; + const n: f32 = @floatFromInt(cols.count); + const w = (l.grid.w - (n - 1) * gap) / (n + @as(f32, Grid.rows - 1) * 0.5); + const h = (l.grid.h - (Grid.rows - 1) * gap) / Grid.rows; + const r: f32 = @floatFromInt(row); + const j: f32 = @floatFromInt(column - cols.first); + return .{ .x = l.grid.x + r * w * 0.5 + j * (w + gap), .y = l.grid.y + r * (h + gap), .w = w, .h = h }; + } + + fn keyAt(l: Layout, x: f32, y: f32) ?u16 { + for (0..Grid.rows) |row| for (0..Grid.columns) |column| { + if (l.key(row, column)) |r| if (r.contains(x, y)) return Grid.chord(row, column); + }; + return null; + } + + fn button(l: Layout, i: usize) sk.Rect { + const n: f32 = @floatFromInt(buttons.len); + const at: f32 = @floatFromInt(i); + if (l.narrow) { + const per: f32 = per_row; + const w = (l.w - 8 - (per - 1) * 4) / per; + const column = @mod(at, per); + const line = @floor(at / per); + return .{ .x = 4 + column * (w + 4), .y = l.h - toolbar + 6 + line * 48, .w = w, .h = 44 }; + } + const w = 66; + return .{ .x = l.w - 16 - (n - at) * (w + 6) + 6, .y = 5, .w = w, .h = 26 }; + } + + fn buttonAt(l: Layout, x: f32, y: f32) ?Action { + for (buttons, 0..) |a, i| if (l.button(i).contains(x, y)) return a; + return null; + } +}; + +// --------------------------------------------------------------------------- +// Update. + +const Action = enum { play, reverse, mode, preset, scale, mute, faster, slower, reset, focus, eight, random, instrument }; + +const keys = [_]struct { sk.Key, Action }{ + .{ .space, .play }, .{ .left, .reverse }, .{ .right, .reverse }, .{ .s, .mode }, .{ .p, .preset }, + .{ .q, .scale }, .{ .m, .mute }, .{ .up, .faster }, .{ .down, .slower }, .{ .r, .reset }, + .{ .tab, .focus }, .{ .e, .eight }, .{ .x, .random }, .{ .c, .instrument }, +}; + +/// What touch screens reach by button; on a phone, in two rows of four. +const buttons = [_]Action{ .play, .reverse, .mode, .mute, .eight, .random, .instrument, .scale }; + +/// The computer keyboard as the grid: its rows are staggered like a +/// sanfona's, so the number row plays majors, QWERTY minors, ASDF sevenths +/// and ZXCV sus chords, column by column round the fifths. +const grid_keys = [_][]const sk.Key{ + &.{ .@"1", .@"2", .@"3", .@"4", .@"5", .@"6", .@"7", .@"8", .@"9", .@"0", .minus, .equals }, + &.{ .q, .w, .e, .r, .t, .y, .u, .i, .o, .p, .left_bracket, .right_bracket }, + &.{ .a, .s, .d, .f, .g, .h, .j, .k, .l, .semicolon, .apostrophe }, + &.{ .z, .x, .c, .v, .b, .n, .m, .comma, .period, .slash }, +}; + +fn isLetterOrDigit(k: sk.Key) bool { + const v = @intFromEnum(k); + return v >= @intFromEnum(sk.Key.a) and v <= @intFromEnum(sk.Key.@"0"); +} + +pub fn update(s: *State, in: *const sk.Input) void { + // As an instrument, letters and digits play chords; Escape leaves. + if (s.instrument) { + for (grid_keys, 0..) |row, r| for (row, 0..) |k, column| { + if (in.pressed(k)) s.chord.send(chords.Grid.chord(r, column)); + }; + if (in.pressed(sk.Key.escape)) s.instrument = false; + } + for (keys) |k| if (in.pressed(k[0]) and !(s.instrument and isLetterOrDigit(k[0]))) { + // Left and right choose the way rather than flip it. + if (k[0] == .left) s.backward = true else if (k[0] == .right) s.backward = false else act(s, k[1]); + if (k[0] == .left or k[0] == .right) s.running = true; + }; + const l = Layout.of(in.width, in.height, s.instrument); + for (in.events) |e| if (e.kind == .mouse_down and e.code == 1) { + if (l.buttonAt(e.x, e.y)) |a| act(s, a); + if (s.instrument) if (l.keyAt(e.x, e.y)) |chord| s.chord.send(chord); + }; +} + +fn act(s: *State, a: Action) void { + switch (a) { + .play => s.running = !s.running, + .reverse => { + s.backward = !s.backward; + s.running = true; + }, + .mode => s.mode = @enumFromInt((@intFromEnum(s.mode) + 1) % @typeInfo(Mode).@"enum".field_names.len), + .preset => choose(s, @enumFromInt((@intFromEnum(s.preset) + 1) % @typeInfo(Preset).@"enum".field_names.len)), + .scale => s.scale = @enumFromInt((@intFromEnum(s.scale) + 1) % @typeInfo(Scale).@"enum".field_names.len), + .mute => s.muted = !s.muted, + .faster => s.speed = @min(s.speed * 1.25, 8), + .slower => s.speed = @max(s.speed / 1.25, 0.05), + .reset => s.reset.send(s.reset.value +% 1), + .focus => s.focus = (s.focus + 1) % 3, + .eight => choose(s, .figure_eight), + // A new seed each time: new bodies. + .random => choose(s, .random), + .instrument => s.instrument = !s.instrument, + } +} + +fn choose(s: *State, p: Preset) void { + s.preset = p; + s.backward = false; + s.running = true; + s.reset.send(s.reset.value +% 1); +} + +// --------------------------------------------------------------------------- +// The presets. + +const Start = struct { masses: [3]f32, bodies: [3]layouts.Body, softening: f32 = 0.01 }; + +fn start(p: Preset, seed: u32) Start { + return switch (p) { + // Chenciner and Montgomery's: three equal masses chasing each + // other round one figure of eight. Stable: the cloud stays thin. + .figure_eight => .{ .masses = .{ 1, 1, 1 }, .bodies = .{ + .{ .pos = .{ 0.97000436, -0.24308753 }, .vel = .{ 0.46620368, 0.43236573 } }, + .{ .pos = .{ -0.97000436, 0.24308753 }, .vel = .{ 0.46620368, 0.43236573 } }, + .{ .pos = .{ 0, 0 }, .vel = .{ -0.93240737, -0.86473146 } }, + } }, + // Šuvakov and Dmitrašinović's butterfly I: periodic, and not stable. + .butterfly => .{ .masses = .{ 1, 1, 1 }, .bodies = .{ + .{ .pos = .{ -1, 0 }, .vel = .{ 0.30689, 0.12551 } }, + .{ .pos = .{ 1, 0 }, .vel = .{ 0.30689, 0.12551 } }, + .{ .pos = .{ 0, 0 }, .vel = .{ -0.61378, -0.25102 } }, + } }, + // Burrau's: masses 3, 4 and 5 at rest on a 3-4-5 triangle. Close + // passes, chaos, and in the end two bodies leave as a pair. + .pythagorean => .{ .masses = .{ 3, 4, 5 }, .bodies = centred(.{ 3, 4, 5 }, .{ + .{ .pos = .{ 1, 3 }, .vel = .{ 0, 0 } }, + .{ .pos = .{ -2, -1 }, .vel = .{ 0, 0 } }, + .{ .pos = .{ 1, -1 }, .vel = .{ 0, 0 } }, + }) }, + // Lagrange's equilateral solution, turning rigidly. For equal + // masses it is unstable: the cloud shows it coming apart. + .lagrange => blk: { + const speed: f32 = @sqrt(1.0 / @sqrt(3.0)); + var bodies: [3]layouts.Body = undefined; + for (&bodies, 0..) |*b, i| { + const a = @as(f32, @floatFromInt(i)) * std.math.tau / 3; + b.* = .{ .pos = .{ @cos(a), @sin(a) }, .vel = .{ -@sin(a) * speed, @cos(a) * speed } }; + } + break :blk .{ .masses = .{ 1, 1, 1 }, .bodies = bodies }; + }, + .random => blk: { + var rng: std.Random.DefaultPrng = .init(seed); + const r = rng.random(); + var masses: [3]f32 = undefined; + var bodies: [3]layouts.Body = undefined; + for (&masses, &bodies) |*m, *b| { + m.* = 0.6 + 0.8 * r.float(f32); + b.* = .{ .pos = .{ 2 * r.float(f32) - 1, 2 * r.float(f32) - 1 }, .vel = .{ 0.6 * (r.float(f32) - 0.5), 0.6 * (r.float(f32) - 0.5) } }; + } + break :blk .{ .masses = masses, .bodies = centred(masses, bodies) }; + }, + }; +} + +/// The bodies moved and pushed so their centre of mass sits still at 0. +fn centred(masses: [3]f32, bodies: [3]layouts.Body) [3]layouts.Body { + var total: f32 = 0; + var p: [2]f32 = .{ 0, 0 }; + var v: [2]f32 = .{ 0, 0 }; + for (masses, bodies) |m, b| { + total += m; + for (0..2) |k| { + p[k] += m * b.pos[k]; + v[k] += m * b.vel[k]; + } + } + var out = bodies; + for (&out) |*b| for (0..2) |k| { + b.pos[k] -= p[k] / total; + b.vel[k] -= v[k] / total; + }; + return out; +} + +// --------------------------------------------------------------------------- +// Draw. + +pub fn draw(s: *const State, ss: *Session, f: *sk.Frame) void { + const in = f.in; + f.canvas(@intFromFloat(@max(in.window_width, 320)), @intFromFloat(@max(in.window_height, 400))); + f.clear(background); + if (!f.host.gpu()) { + f.text(16, 40, 2, "spheres needs a GPU: the host has none.", 0xff8080); + return; + } + const rate: f64 = @floatFromInt(in.sample_rate); + const dt: f32 = @floatCast(@min(@as(f64, @floatFromInt(in.dt)) / std.time.ns_per_s, 0.05)); + const g = &ss.gpu; + setup(g, f); + + // What came back: sound for the speakers and the scope, and system 0. + for (in.downloads) |d| switch (@as(Readback, @enumFromInt(d.key))) { + .audio => { + const samples = std.mem.bytesAsSlice(f32, d.bytes[0 .. d.bytes.len / 4 * 4]); + const aligned = f.scratch.alloc(f32, samples.len) catch continue; + for (aligned, samples) |*to, from| to.* = from; + ss.in_flight -|= aligned.len; + // Sound made before a chord was chosen is not played. + const dropped = @min(ss.discard, aligned.len); + ss.discard -= dropped; + f.audio(stream, aligned[dropped..], .{}); + ss.scope.push(aligned[dropped..]); + }, + .lead => if (d.bytes.len >= @sizeOf(layouts.Lead)) { + ss.lead = std.mem.bytesToValue(layouts.Lead, d.bytes[0..@sizeOf(layouts.Lead)]); + ss.have_lead = true; + }, + _ => if (d.key >= chords_from) { + const words = f.scratch.alloc(u32, d.bytes.len / 4) catch continue; + @memcpy(std.mem.sliceAsBytes(words), d.bytes[0 .. words.len * 4]); + ss.ahead.arrived(d.key - chords_from, words); + }, + }; + + if (!ss.started or ss.reset.take(s.reset) != null) restart(s, ss, f); + if (!s.instrument) { + ss.travel = null; + ss.held = null; + } + lookAhead(s, ss, f); + + // A chord: the sound jumps to it now, the sky travels there. + if (ss.chord.take(s.chord)) |chord| if (s.instrument and ss.ahead.found[chord]) { + const root: u4 = if (chords.Grid.find(chord)) |at| chords.Grid.roots[at.column] else 0; + if (ss.ahead.nearest(chord, root, ss.time)) |pick| { + ss.held = pick; + ss.travel = .{ .to = pick, .speed = @max(@abs(pick.time() - ss.time) / 1.2, 2) }; + f.audio(stream, &.{}, .{ .clear = true }); + ss.discard = ss.in_flight; + } + }; + + // The simulation: as much simulated time as `speed` asks, either way; + // or, travelling to a chord, as fast as it takes to arrive in a second + // or so, where system 0 takes the snapshot's state. It runs every frame, + // still or not, so system 0 always sings: a held chord, or its motion. + var span: f64 = 0; + var adopt: u32 = 0; + if (ss.travel) |travel| { + const remaining = travel.to.time() - ss.time; + const step = travel.speed * dt; + if (@abs(remaining) <= step) { + adopt = travel.to.index() + 1; + ss.time = travel.to.time(); + ss.travel = null; + } else span = std.math.sign(remaining) * step; + } else if (s.running and !(s.instrument and ss.held != null)) { + // Arrived at a chord, the sky holds still on the moment that sings it. + span = s.speed * dt * @as(f64, if (s.backward) -1 else 1); + } + const steps: u32 = @intFromFloat(@min(@ceil(@abs(span) / chords.base_step), 256)); + var trail = trailPoints(ss, steps, span); + // Travelling back retraces system 0's own path: its trail rewinds rather + // than drawing over itself, as far as it reaches. + if (ss.travel != null and span < 0 and trail.points < ss.trail_filled) { + ss.trail_head = (ss.trail_head + layouts.trail - trail.points) % layouts.trail; + ss.trail_filled -= trail.points; + trail.points = 0; + } + f.compute(.{ + .label = "sim.comp", + .kernel = sim_kernel, + .readonly = &.{g.snapshots}, + .readwrite = &.{ g.bodies, g.trail, g.voices, g.lead }, + .uniforms = gpu.uniform(f.scratch, layouts.SimParams{ + .systems = systems, + .steps = steps, + .dt = if (steps > 0) @floatCast(span / @as(f64, @floatFromInt(steps))) else 0, + .softening = ss.softening, + .masses = .{ ss.masses[0], ss.masses[1], ss.masses[2], 0 }, + .trail_head = (ss.trail_head + 1) % layouts.trail, + .trail_points = trail.points, + .trail_first = trail.first, + .trail_every = trail.every, + .quantize = @intFromEnum(s.scale), + .base = chords.base, + .speed_ref = chords.speed_ref, + .depth_scale = chords.depth_scale, + .couple_scale = chords.couple_scale, + .voice_source = if (ss.held) |h| h.index() + 1 else 0, + .adopt = adopt, + }), + .groups = .{ (systems + 63) / 64, 1, 1 }, + }); + ss.time += span; + ss.trail_head = (ss.trail_head + trail.points) % layouts.trail; + ss.trail_filled = @min(ss.trail_filled + trail.points, layouts.trail); + f.download(g.lead, .{ .size = @sizeOf(layouts.Lead), .key = @intFromEnum(Readback.lead) }); + + // The sound: enough to keep `lead_seconds` queued, counting what is + // still on its way back from the GPU. + const queued = in.stream(stream).queued + ss.in_flight; + const want: usize = @min(@as(usize, @intFromFloat(lead_seconds * rate)) -| queued, layouts.max_block); + if (want > 0) { + const params = gpu.uniform(f.scratch, layouts.SynthParams{ + .count = @intCast(want), + .mode = @intFromEnum(s.mode), + .rate = @floatCast(rate), + // Loud enough for a phone's speaker; tanh rounds the peaks off. + .gain = if (s.muted) 0 else 1.4, + }); + f.compute(.{ + .label = "synth.comp", + .kernel = synth_kernel, + .readonly = &.{ g.voices, g.phases }, + .readwrite = &.{g.block}, + .uniforms = params, + .groups = .{ @intCast((want + 63) / 64), 1, 1 }, + }); + f.compute(.{ .label = "advance.comp", .kernel = advance_kernel, .readwrite = &.{ g.voices, g.phases }, .uniforms = params }); + f.download(g.block, .{ .size = @intCast(want * @sizeOf(f32)), .key = @intFromEnum(Readback.audio) }); + ss.in_flight += want; + } + + const l = Layout.of(in.width, in.height, s.instrument); + follow(ss, dt); + drawSky(s, ss, f, l); + sound(s, ss, f, l, rate); + labels(s, ss, f, l, rate); +} + +/// Simulated time between trail points: about a frame's at the default +/// speed, and many points a frame when travelling. +const trail_dt = 0.012; + +/// Which of this frame's `steps` leave trail points: every so many steps, +/// carrying the remainder to the next frame, 64 at most. +fn trailPoints(ss: *Session, steps: u32, span: f64) struct { points: u32, first: u32, every: u32 } { + if (steps == 0) return .{ .points = 0, .first = 1, .every = 1 }; + const step_dt = @abs(span) / @as(f64, @floatFromInt(steps)); + var every: u32 = @intFromFloat(@max(1, @round(trail_dt / step_dt))); + var first = every -| @min(ss.trail_carry, every - 1); + var points: u32 = if (first > steps) 0 else 1 + (steps - first) / every; + if (points > 64) { + every = (steps + 63) / 64; + first = every; + points = steps / every; + } + ss.trail_carry = if (points == 0) ss.trail_carry + steps else steps - (first + (points - 1) * every); + return .{ .points = points, .first = first, .every = every }; +} + +fn setup(g: *Gpu, f: *sk.Frame) void { + if (g.ready) return; + g.ready = true; + g.bodies = f.host.newBuffer(systems * layouts.bodies * @sizeOf(layouts.Body)); + g.trail = f.host.newBuffer(layouts.bodies * layouts.trail * @sizeOf([2]f32)); + g.voices = f.host.newBuffer(layouts.voices_len * @sizeOf([4]f32)); + g.phases = f.host.newBuffer(3 * @sizeOf([4]f32)); + g.lead = f.host.newBuffer(@sizeOf(layouts.Lead)); + g.block = f.host.newBuffer(layouts.max_block * @sizeOf(f32)); + g.ahead_state = f.host.newBuffer(layouts.alternatives * 3 * @sizeOf(layouts.Body)); + g.snapshots = f.host.newBuffer(layouts.horizon * layouts.alternatives * 3 * @sizeOf(layouts.Body)); + g.ahead_chords = f.host.newBuffer(layouts.horizon * layouts.alternatives * @sizeOf(u32)); +} + +fn lookParams(s: *const State, ss: *const Session, first: u32, count: u32) layouts.LookParams { + return .{ + .alternatives = layouts.alternatives, + .first = first, + .count = count, + .horizon = layouts.horizon, + .masses = .{ ss.masses[0], ss.masses[1], ss.masses[2], 0 }, + .softening = ss.softening, + .quantize = @intFromEnum(s.scale), + .base = chords.base, + .speed_ref = chords.speed_ref, + .depth_scale = chords.depth_scale, + .couple_scale = chords.couple_scale, + .sample_dt = chords.sample_dt, + }; +} + +/// The look-ahead: its futures a few samples further, their chords on +/// their way back; and all the chords again when the scale changes. +fn lookAhead(s: *const State, ss: *Session, f: *sk.Frame) void { + const g = &ss.gpu; + if (ss.scale != s.scale) { + ss.scale = s.scale; + const count = ss.ahead.asked * layouts.alternatives; + ss.ahead.forget(); + if (count > 0) { + f.compute(.{ + .label = "rechord.comp", + .kernel = rechord_kernel, + .readonly = &.{g.snapshots}, + .readwrite = &.{g.ahead_chords}, + .uniforms = gpu.uniform(f.scratch, lookParams(s, ss, 0, count)), + .groups = .{ (count + 63) / 64, 1, 1 }, + }); + f.download(g.ahead_chords, .{ .size = count * @sizeOf(u32), .key = chords_from }); + } + } + if (ss.ahead.asked >= layouts.horizon) return; + const first = ss.ahead.asked; + const count: u32 = @min(chords.per_frame, layouts.horizon - first); + f.compute(.{ + .label = "lookahead.comp", + .kernel = lookahead_kernel, + .readwrite = &.{ g.ahead_state, g.snapshots, g.ahead_chords }, + .uniforms = gpu.uniform(f.scratch, lookParams(s, ss, first, count)), + }); + f.download(g.ahead_chords, .{ .offset = first * layouts.alternatives * @sizeOf(u32), .size = count * layouts.alternatives * @sizeOf(u32), .key = chords_from + first }); + ss.ahead.asked += count; +} + +/// Puts the preset on the GPU: system 0 as given, the others each body +/// moved by up to `spread`. +fn restart(s: *const State, ss: *Session, f: *sk.Frame) void { + ss.started = true; + const p = start(s.preset, s.reset.value); + ss.masses = p.masses; + ss.softening = p.softening; + const all = f.scratch.alloc(layouts.Body, systems * layouts.bodies) catch return; + var rng: std.Random.DefaultPrng = .init(s.reset.value *% 0x9e3779b9 +% 1); + const r = rng.random(); + for (0..systems) |i| for (0..3) |b| { + var body = p.bodies[b]; + if (i > 0) for (&body.pos) |*x| { + x.* += spread * (2 * r.float(f32) - 1); + }; + all[3 * i + b] = body; + }; + f.upload(ss.gpu.bodies, 0, std.mem.sliceAsBytes(all)); + // The voices glide from a quiet start, uncoupled; the trail starts empty. + const quiet = f.scratch.alloc([4]f32, layouts.voices_len) catch return; + for (quiet, 0..) |*v, i| v.* = if (i < layouts.coupling_previous) .{ 220, 1, 0, 0 } else .{ 0, 0, 0, 0 }; + f.upload(ss.gpu.voices, 0, std.mem.sliceAsBytes(quiet)); + const lead = f.scratch.create(layouts.Lead) catch return; + lead.* = std.mem.zeroes(layouts.Lead); + lead.bodies = p.bodies; + f.upload(ss.gpu.lead, 0, std.mem.asBytes(lead)); + ss.lead = lead.*; + ss.have_lead = true; + ss.trail_filled = 0; + ss.trail_carry = 0; + // The look-ahead's futures: 0 as given, the rest nudged by growing + // amounts, so in a chaotic system they fan out. + const futures = f.scratch.alloc(layouts.Body, layouts.alternatives * 3) catch return; + for (0..layouts.alternatives) |a| for (0..3) |b| { + var body = p.bodies[b]; + // Small enough that every future is recognisably this system, + // apart where chaos amplifies the difference. + const nudge = 2e-4 * @as(f32, @floatFromInt(a)); + for (&body.pos) |*x| { + x.* += nudge * (2 * r.float(f32) - 1); + } + futures[3 * a + b] = body; + }; + f.upload(ss.gpu.ahead_state, 0, std.mem.sliceAsBytes(futures)); + ss.ahead.reset(); + ss.scale = s.scale; + ss.time = 0; + ss.travel = null; + ss.held = null; + ss.gpu.clear_sky = true; + ss.extent = 1.5; +} + +/// The camera keeps system 0's bodies in view: quick to widen, slow to +/// close in. +fn follow(ss: *Session, dt: f32) void { + var c: [2]f32 = .{ 0, 0 }; + var total: f32 = 0; + for (ss.lead.bodies, ss.masses) |b, m| { + c[0] += m * b.pos[0]; + c[1] += m * b.pos[1]; + total += m; + } + c = .{ c[0] / total, c[1] / total }; + var reach: f32 = 0; + for (ss.lead.bodies) |b| reach = @max(reach, @max(@abs(b.pos[0] - c[0]), @abs(b.pos[1] - c[1]))); + const want = std.math.clamp(reach * 1.25, 1.2, 40); + const rate: f32 = if (want > ss.extent) 4 else 0.6; + ss.extent += (want - ss.extent) * (1 - @exp(-rate * dt)); + for (&ss.center, c) |*x, to| x.* += (to - x.*) * (1 - @exp(-2 * dt)); +} + +const sim_kernel: gpu.Kernel = .{ .code = shaders.sim_comp, .readonly_buffers = 1, .readwrite_buffers = 4, .uniform_buffers = 1, .threads = .{ 64, 1, 1 } }; +const synth_kernel: gpu.Kernel = .{ .code = shaders.synth_comp, .readonly_buffers = 2, .readwrite_buffers = 1, .uniform_buffers = 1, .threads = .{ 64, 1, 1 } }; +const advance_kernel: gpu.Kernel = .{ .code = shaders.advance_comp, .readwrite_buffers = 2, .uniform_buffers = 1, .threads = .{ 1, 1, 1 } }; +const lookahead_kernel: gpu.Kernel = .{ .code = shaders.lookahead_comp, .readwrite_buffers = 3, .uniform_buffers = 1, .threads = .{ 32, 1, 1 } }; +const rechord_kernel: gpu.Kernel = .{ .code = shaders.rechord_comp, .readonly_buffers = 1, .readwrite_buffers = 1, .uniform_buffers = 1, .threads = .{ 64, 1, 1 } }; + +const sky_pipelines = [_]gpu.Pipeline{ + // 0: the fade, blended over what is there. + .{ .vertex = .{ .code = shaders.fullscreen_vert }, .fragment = .{ .code = shaders.fade_frag, .uniform_buffers = 1 }, .stride = 0, .attributes = &.{}, .blend = .premultiplied }, + // 1: glows, added. + .{ .vertex = .{ .code = shaders.sprite_vert, .uniform_buffers = 1, .storage_buffers = 1 }, .fragment = .{ .code = shaders.sprite_frag }, .stride = 0, .attributes = &.{}, .blend = .additive }, + // 2: trails, added. + .{ .vertex = .{ .code = shaders.trail_vert, .uniform_buffers = 1, .storage_buffers = 1 }, .fragment = .{ .code = shaders.trail_frag }, .stride = 0, .attributes = &.{}, .blend = .additive }, + // 3: the cloud, layered: where copies crowd it takes their body's + // colour rather than adding up to white. + .{ .vertex = .{ .code = shaders.sprite_vert, .uniform_buffers = 1, .storage_buffers = 1 }, .fragment = .{ .code = shaders.sprite_frag }, .stride = 0, .attributes = &.{}, .blend = .premultiplied }, + // 4: the coupling between the closest pair, added. + .{ .vertex = .{ .code = shaders.link_vert, .uniform_buffers = 1, .storage_buffers = 2 }, .fragment = .{ .code = shaders.link_frag }, .stride = 0, .attributes = &.{}, .blend = .additive }, +}; + +/// The sky: the cloud of copies on a long exposure, and over it system 0's +/// trails and bodies, drawn fresh each frame. +fn drawSky(s: *const State, ss: *Session, f: *sk.Frame, l: Layout) void { + const g = &ss.gpu; + const size: [2]u32 = .{ @intFromFloat(@max(l.sky.w, 1)), @intFromFloat(@max(l.sky.h, 1)) }; + if (g.sky == .none or g.size[0] != size[0] or g.size[1] != size[1]) { + if (g.sky != .none) f.host.freeImage(g.sky); + if (g.overlay != .none) f.host.freeImage(g.overlay); + g.sky = f.host.newImage(size[0], size[1]); + g.overlay = f.host.newImage(size[0], size[1]); + g.size = size; + g.clear_sky = true; + } + const bg = sk.Color.hex(background); + const bgf: [3]f32 = .{ @as(f32, @floatFromInt(bg.r)) / 255, @as(f32, @floatFromInt(bg.g)) / 255, @as(f32, @floatFromInt(bg.b)) / 255 }; + const per_pixel = 2 * ss.extent / @as(f32, @floatFromInt(@min(size[0], size[1]))); + const base: layouts.SkyView = .{ + .center = ss.center, + .size = .{ @floatFromInt(size[0]), @floatFromInt(size[1]) }, + .per_pixel = per_pixel, + .radius = 1.6, + .alpha = 0.06, + .first = 1, + .count = systems - 1, + .head = ss.trail_head, + .filled = ss.trail_filled, + .time = @floatCast(@mod(@as(f64, @floatFromInt(f.in.now)) / std.time.ns_per_s, 1000)), + }; + var cloud = base; + cloud.radius = 2.2; + cloud.alpha = 0.18; + f.gpu(.{ + .label = "sky cloud", + .image = g.sky, + .width = size[0], + .height = size[1], + .clear = if (g.clear_sky) .{ bgf[0], bgf[1], bgf[2], 1 } else null, + .pipelines = &sky_pipelines, + .draws = &.{ + .{ .pipeline = 0, .vertex_count = 3, .fragment_uniforms = gpu.uniform(f.scratch, layouts.Fade{ .background = bgf, .amount = if (s.running) 0.08 else 0.02 }) }, + .{ .pipeline = 3, .vertex_count = (systems - 1) * 3 * 6, .vertex_uniforms = gpu.uniform(f.scratch, cloud), .vertex_storage = &.{g.bodies} }, + }, + }); + g.clear_sky = false; + + var trails = base; + trails.radius = 2.2; + trails.alpha = 0.9; + // Points are 0.012 apart in time; anything this far is a jump. + trails.gap = @max(0.5, ss.extent * 0.25); + var link = base; + link.radius = 3; + link.alpha = 0.9; + var halo = base; + halo.first = 0; + halo.count = 1; + halo.radius = 34; + halo.alpha = 0.35; + var core = halo; + core.radius = 9; + core.alpha = 1; + f.gpu(.{ + .label = "sky bodies", + .image = g.overlay, + .width = size[0], + .height = size[1], + .clear = .{ 0, 0, 0, 0 }, + .pipelines = &sky_pipelines, + .draws = &.{ + .{ .pipeline = 2, .vertex_count = 3 * (layouts.trail - 1) * 6, .vertex_uniforms = gpu.uniform(f.scratch, trails), .vertex_storage = &.{g.trail} }, + .{ .pipeline = 4, .vertex_count = 6 * 6, .vertex_uniforms = gpu.uniform(f.scratch, link), .vertex_storage = &.{ g.bodies, g.voices } }, + .{ .pipeline = 1, .vertex_count = 3 * 6, .vertex_uniforms = gpu.uniform(f.scratch, halo), .vertex_storage = &.{g.bodies} }, + .{ .pipeline = 1, .vertex_count = 3 * 6, .vertex_uniforms = gpu.uniform(f.scratch, core), .vertex_storage = &.{g.bodies} }, + }, + }); + f.image(g.sky, l.sky, .{}); + f.image(g.overlay, l.sky, .{}); +} + +/// The scope: an oscilloscope steadied on the focused voice, the spectrum +/// and the spectrogram. +fn sound(s: *const State, ss: *Session, f: *sk.Frame, l: Layout, rate: f64) void { + const v = ss.lead.voices[s.focus]; + const period: u32 = if (v.carrier > 1) @intFromFloat(@round(rate / v.carrier)) else 256; + ss.scope.draw(f, .{ + .seconds = 8, + .spectrogram = l.spectrogram, + .spectrum = l.spectrum, + .oscilloscope = .{ + .rect = l.oscilloscope, + .samples = std.math.clamp(4 * period, 256, 4096), + .end = @intCast(ss.scope.heard % @max(period, 1)), + }, + }); +} + +// --------------------------------------------------------------------------- +// Text and marks. + +const dim = 0x7a8090; +const bright = 0xe6e9ef; + +fn labels(s: *const State, ss: *const Session, f: *sk.Frame, l: Layout, rate: f64) void { + f.text(16, 10, 2, "spheres", 0xc9b6ff); + const room = if (l.narrow) l.w else l.button(0).x; + const lead = ss.lead; + const how = if (!s.running) " paused" else if (s.backward) " backwards" else ""; + if (room > 420) { + f.print(140, 14, 1, dim, "{s} t {d:.1} energy {d:.4}{s}", .{ s.preset.label(), ss.time, lead.energy, how }); + } else f.print(140, 14, 1, dim, "{s} t {d:.1}{s}", .{ s.preset.label(), ss.time, how }); + for (buttons, 0..) |a, i| button(s, f, a, l.button(i)); + + // The voices, in their bodies' colours. + const v_rect = l.voices; + f.print(v_rect.x, v_rect.y, 1, bright, "{s} {s}", .{ modeName(s.mode), modeFormula(s.mode) }); + for (lead.voices, 0..) |v, i| { + const y = v_rect.y + 16 + @as(f32, @floatFromInt(i)) * 14; + const mark = if (i == s.focus) ">" else " "; + const letter: u8 = 'A' + @as(u8, @intCast(i)); + // Its role in the FM pair: modulating (silent), or modulated and by + // whom, at what index. + const row = lead.coupling[i]; + var by: usize = 0; + for (0..3) |j| if (row[j] > row[by]) { + by = j; + }; + const pushed = if (row[3] > 0.5) + f.format(" modulating", .{}) + else if (row[by] > 0.05) + f.format(" FM by {c} {d:.2}", .{ @as(u8, 'A' + @as(u8, @intCast(by))), row[by] }) + else + ""; + if (l.narrow) { + f.print(v_rect.x, y, 1, palette[i], "{s}{c} {d:7.1} Hz r {d:4.2} d {d:5.2}{s}", .{ mark, letter, v.carrier, v.ratio, v.depth, pushed }); + } else if (s.mode == .fm or s.mode == .pm) { + f.print(v_rect.x, y, 1, palette[i], "{s}{c} {d:7.1} Hz ratio {d:4.2} depth {d:5.2} index {d:5.2}{s}", .{ mark, letter, v.carrier, v.ratio, v.depth, index(s.mode, v), pushed }); + } else f.print(v_rect.x, y, 1, palette[i], "{s}{c} {d:7.1} Hz ratio {d:4.2} depth {d:5.2}{s}", .{ mark, letter, v.carrier, v.ratio, v.depth, pushed }); + } + + // The spectrum's scale and each voice's partials where the theory puts + // them: the focused voice's all of them, the others' as small ticks. + spectrumMarks(s, ss, f, l.spectrum, rate); + instrumentView(s, ss, f, l); + f.text(l.oscilloscope.x + 4, l.oscilloscope.y + 4, 1, "oscilloscope", dim); + f.text(l.spectrogram.x + 4, l.spectrogram.y + 4, 1, "spectrogram, 8 s", sk.Color.white.alpha(150)); + if (!l.narrow) f.text(16, l.h - 18, 1, if (s.instrument) "keys play the grid: 1 2 3 majors, Q W E minors, A S D sevenths, Z X C sus, by fifths ESC free SPACE pause" else "SPACE pause LEFT/RIGHT time's way UP/DOWN speed P/E/X preset C chords S synthesis Q scale M mute TAB voice R again", dim); +} + +/// Grid rows' colours: majors gold, minors blue, sevenths orange, sus +/// green, power chords violet. +const row_colors = [chords.Grid.rows]u24{ 0xffcc59, 0x66b3ff, 0xff7a59, 0x69db7c, 0xb197fc }; + +/// The chord sounding, over the sky; and as an instrument, the look-ahead's +/// chords along time (system 0's own future) and the grid. +fn instrumentView(s: *const State, ss: *const Session, f: *sk.Frame, l: Layout) void { + const Grid = chords.Grid; + var buf: [32]u8 = undefined; + const now = chords.chordOf(ss.lead.voices); + f.text(l.sky.x + 12, l.sky.y + 10, 2, f.format("{s}", .{chords.name(&buf, now)}), sk.Color.white.alpha(200)); + if (!s.instrument) return; + f.print(l.sky.x + 12, l.sky.y + l.sky.h - 18, 1, sk.Color.white.alpha(120), "{d} chords found in {d} futures", .{ ss.ahead.distinct, layouts.alternatives }); + + const strip = l.strip; + const span = @as(f64, chords.horizon) * chords.sample_dt; + f.fill(strip, sk.Color.hex(0x10131c)); + var t: u32 = 0; + while (t < ss.ahead.known) { + const sung = ss.ahead.sung[t][0]; + const chord: u16 = if (sung.playable) sung.chord else 0; + var end = t + 1; + while (end < ss.ahead.known and @as(u16, if (ss.ahead.sung[end][0].playable) ss.ahead.sung[end][0].chord else 0) == chord) end += 1; + const x0 = strip.x + strip.w * @as(f32, @floatFromInt(t)) / chords.horizon; + const x1 = strip.x + strip.w * @as(f32, @floatFromInt(end)) / chords.horizon; + const color: sk.Color = if (Grid.find(chord)) |at| sk.Color.hex(row_colors[at.row]).alpha(170) else sk.Color.hex(0x2a2f3a); + f.fill(.{ .x = x0, .y = strip.y, .w = @max(x1 - x0, 1), .h = strip.h }, color); + t = end; + } + const at = struct { + fn x(r: sk.Rect, time: f64, total: f64) f32 { + return r.x + r.w * @as(f32, @floatCast(std.math.clamp(time / total, 0, 1))); + } + }.x; + if (ss.travel) |travel| f.fill(.{ .x = at(strip, travel.to.time(), span) - 1, .y = strip.y - 4, .w = 3, .h = strip.h + 8 }, sk.Color.hex(0xffd84d)); + f.fill(.{ .x = at(strip, ss.time, span) - 1, .y = strip.y - 3, .w = 2, .h = strip.h + 6 }, sk.Color.white); + + const held: ?u16 = if (ss.held) |h| ss.ahead.sung[h.sample][h.alternative].chord else null; + for (0..Grid.rows) |row| for (0..Grid.columns) |column| { + const r = l.key(row, column) orelse continue; + const chord = Grid.chord(row, column); + const found = ss.ahead.found[chord]; + const lit = chord == now or chord == held; + const color = sk.Color.hex(row_colors[row]); + f.fill(r, if (lit) color.alpha(230) else if (found) color.alpha(70) else sk.Color.hex(0x12151d)); + if (found) f.fill(.{ .x = r.x, .y = r.y + r.h - 2, .w = r.w, .h = 2 }, color); + const label = f.format("{s}{s}", .{ Grid.root_names[column], Grid.suffix(@enumFromInt(row)) }); + const ink = if (lit) sk.Color.hex(0x05070d) else if (found) sk.Color.white else sk.Color.white.alpha(60); + f.text(r.x + (r.w - @as(f32, @floatFromInt(label.len * 8))) / 2, r.y + r.h / 2 - 4, 1, label, ink); + }; +} + +fn button(s: *const State, f: *sk.Frame, a: Action, r: sk.Rect) void { + const label: []const u8 = switch (a) { + .play => if (s.running) "PAUSE" else "PLAY", + .reverse => if (s.backward) "FWD" else "REV", + .mode => modeName(s.mode), + .preset => "NEXT", + .eight => "EIGHT", + .random => "RANDOM", + .instrument => if (s.instrument) "FREE" else "CHORDS", + .scale => switch (s.scale) { + .free => "UNSNAP", + .pentatonic => "PENTA", + .diatonic => "DIATON", + .chromatic => "CHROMA", + }, + .mute => if (s.muted) "UNMUTE" else "MUTE", + else => "", + }; + f.fill(r, sk.Color.hex(0x161a24)); + f.fill(.{ .x = r.x, .y = r.y + r.h - 1, .w = r.w, .h = 1 }, sk.Color.hex(0x2e3440)); + f.text(r.x + (r.w - @as(f32, @floatFromInt(label.len * 8))) / 2, r.y + r.h / 2 - 4, 1, label, bright); +} + +fn modeName(m: Mode) []const u8 { + return switch (m) { + .sine => "SINE", + .additive => "ADD", + .am => "AM", + .ring => "RING", + .fm => "FM", + .pm => "PM", + }; +} + +fn modeFormula(m: Mode) []const u8 { + return switch (m) { + .sine => "sin c", + .additive => "sum r^(k-1) sin kc, r = d/(d+2)", + .am => "sin c (1 + a sin m)/(1 + a), a = min(d/4, 1)", + .ring => "sin c sin m", + .fm => "carrier swung d times its frequency by m: index d/ratio", + .pm => "sin(c + d sin m): index d", + }; +} + +/// The modulation index the theory works with: what sets the sidebands. +fn index(m: Mode, v: layouts.Voice) f32 { + return switch (m) { + .fm => v.depth / @max(v.ratio, 1e-3), + .pm => v.depth, + else => 0, + }; +} + +/// A partial: frequency and amplitude relative to the voice's. +const Partial = struct { hz: f64, amp: f64 }; + +/// Where a voice's energy goes, by the textbook: FM and PM put +/// |J_k(index)| at carrier + k modulator; AM puts a/2 at carrier ± +/// modulator; ring modulation halves between the two; additive steps +/// through harmonics. +fn partials(m: Mode, v: layouts.Voice, out: []Partial) []Partial { + const fc: f64 = v.carrier; + const fm: f64 = fc * v.ratio; + var n: usize = 0; + switch (m) { + .sine => { + out[0] = .{ .hz = fc, .amp = 1 }; + n = 1; + }, + .additive => { + const r = v.depth / (v.depth + 2); + var norm: f64 = 0; + var w: f64 = 1; + for (0..16) |_| { + norm += w; + w *= r; + } + w = 1; + for (0..16) |k| { + out[n] = .{ .hz = fc * @as(f64, @floatFromInt(k + 1)), .amp = w / norm }; + n += 1; + w *= r; + } + }, + .am => { + const a = @min(v.depth / 4, 1); + out[0] = .{ .hz = fc, .amp = 1 / (1 + a) }; + out[1] = .{ .hz = fc + fm, .amp = a / 2 / (1 + a) }; + out[2] = .{ .hz = @abs(fc - fm), .amp = a / 2 / (1 + a) }; + n = 3; + }, + .ring => { + out[0] = .{ .hz = fc + fm, .amp = 0.5 }; + out[1] = .{ .hz = @abs(fc - fm), .amp = 0.5 }; + n = 2; + }, + .fm, .pm => { + const beta: f64 = index(m, v); + var k: i32 = -12; + while (k <= 12) : (k += 1) { + out[n] = .{ .hz = @abs(fc + @as(f64, @floatFromInt(k)) * fm), .amp = @abs(bessel(@intCast(@abs(k)), beta)) }; + n += 1; + } + }, + } + return out[0..n]; +} + +/// A voice's partials with the FM pair's modulation on top: each of its own +/// partials spread into sidebands |J_k(index)| at k times the modulating +/// body's frequency (exact for sine and additive, whose harmonics all get +/// the same push; a guide for the others). +fn fmPair(m: Mode, v: layouts.Voice, voices: [3]layouts.Voice, row: [4]f32, own: []Partial, out: []Partial) []Partial { + const base = partials(m, v, own); + var by: usize = 0; + for (0..3) |j| if (row[j] > row[by]) { + by = j; + }; + const beta: f64 = row[by]; + if (beta < 0.05) return base; + const fm: f64 = voices[by].carrier; + var n: usize = 0; + for (base) |p| { + var k: i32 = -6; + while (k <= 6) : (k += 1) { + if (n == out.len) return out; + out[n] = .{ .hz = @abs(p.hz + @as(f64, @floatFromInt(k)) * fm), .amp = p.amp * @abs(bessel(@intCast(@abs(k)), beta)) }; + n += 1; + } + } + return out[0..n]; +} + +/// J_n(x), the Bessel function of the first kind, by its series: enough +/// terms for the indices here (x up to about 24). +fn bessel(n: u32, x: f64) f64 { + const half = x / 2; + var term = std.math.pow(f64, half, @floatFromInt(n)); + for (1..n + 1) |k| term /= @floatFromInt(k); + var sum = term; + for (1..80) |m| { + const mf: f64 = @floatFromInt(m); + term *= -half * half / (mf * (mf + @as(f64, @floatFromInt(n)))); + sum += term; + if (@abs(term) < 1e-14 * @abs(sum) and m > 2 * @as(usize, @intFromFloat(@abs(x)))) break; + } + return sum; +} + +fn spectrumMarks(s: *const State, ss: *const Session, f: *sk.Frame, r: sk.Rect, rate: f64) void { + for ([_]f64{ 50, 200, 1000, 5000, 20000 }) |hz| { + if (hz > scope.maxFreq(rate)) continue; + const x = r.x + scope.place(hz, rate) * r.w; + const label = if (hz >= 1000) f.format("{d}k", .{hz / 1000}) else f.format("{d}", .{hz}); + f.text(@min(x + 3, r.x + r.w - 24), r.y + r.h + 4, 1, label, dim); + } + var own: [32]Partial = undefined; + var buf: [32 * 13]Partial = undefined; + for (ss.lead.voices, 0..) |v, i| { + const row = ss.lead.coupling[i]; + if (row[3] > 0.5) continue; + const focused = i == s.focus; + const color = sk.Color.hex(palette[i]); + for (fmPair(s.mode, v, ss.lead.voices, row, &own, &buf)) |p| { + if (p.hz < scope.min_freq or p.hz > scope.maxFreq(rate) or p.amp < 1e-4) continue; + const x = @round(r.x + scope.place(p.hz, rate) * r.w); + // Level in dB of full scale, as the spectrum shows it. + const db = 20 * std.math.log10(@max(p.amp * v.amp, 1e-9)); + const y = r.y + r.h * @as(f32, @floatCast(std.math.clamp(db / scope.floor_db, 0, 1))); + if (focused) { + f.fill(.{ .x = x, .y = y, .w = 1, .h = r.y + r.h - y }, color.alpha(70)); + f.fill(.{ .x = x - 3, .y = y - 1, .w = 7, .h = 3 }, color.alpha(230)); + } else f.fill(.{ .x = x - 2, .y = y, .w = 5, .h = 2 }, color.alpha(150)); + } + } +} + +test "the presets start with the centre of mass at rest" { + inline for (@typeInfo(Preset).@"enum".field_names) |field| { + const p = start(@field(Preset, field), 1); + var momentum: [2]f32 = .{ 0, 0 }; + for (p.masses, p.bodies) |m, b| { + momentum[0] += m * b.vel[0]; + momentum[1] += m * b.vel[1]; + } + try std.testing.expectApproxEqAbs(@as(f32, 0), momentum[0], 1e-5); + try std.testing.expectApproxEqAbs(@as(f32, 0), momentum[1], 1e-5); + } +} + +test "Bessel functions, against known values" { + try std.testing.expectApproxEqAbs(@as(f64, 1), bessel(0, 0), 1e-12); + try std.testing.expectApproxEqAbs(@as(f64, 0.7651976866), bessel(0, 1), 1e-9); + try std.testing.expectApproxEqAbs(@as(f64, 0.5767248078), bessel(1, 2), 1e-9); + // The first zero of J0: FM's carrier vanishes at index 2.405. + try std.testing.expectApproxEqAbs(@as(f64, 0), bessel(0, 2.404825558), 1e-8); + try std.testing.expectApproxEqAbs(@as(f64, -0.2459357645), bessel(0, 10), 1e-8); +} + +test "PM's partials hold all the energy" { + var buf: [32]Partial = undefined; + var energy: f64 = 0; + for (partials(.pm, .{ .carrier = 220, .ratio = 1.5, .depth = 3, .amp = 1 }, &buf)) |p| energy += p.amp * p.amp; + try std.testing.expectApproxEqAbs(@as(f64, 1), energy, 1e-6); +} diff --git a/examples/spheres/test/play.script b/examples/spheres/test/play.script new file mode 100644 index 0000000..1b06372 --- /dev/null +++ b/examples/spheres/test/play.script @@ -0,0 +1,58 @@ +# Plays each preset and synthesis mode headless; screenshots land in +# .zig-cache/spheres/ for review. +frames 120 +expect running true +expect preset figure_eight +screenshot .zig-cache/spheres/figure-eight.png +expect mode additive +key s +key s +key s +expect mode fm +frames 30 +key p +key p +expect preset pythagorean +frames 240 +screenshot .zig-cache/spheres/pythagorean.png +# Time backwards, then paused. +key left +expect backward true +frames 30 +key space +expect running false +frames 10 +key space +key right +expect backward false +key tab +expect focus 1 +key p +frames 200 +screenshot .zig-cache/spheres/lagrange.png +# The buttons do what the keys do: pause, a random system, the figure eight. +click 727 18 +expect running false +click 1087 18 +expect preset random +expect running true +frames 60 +screenshot .zig-cache/spheres/random.png +click 1015 18 +expect preset figure_eight +# As an instrument: the look-ahead finds chords, a grid button sends the sky +# to the nearest time its chord plays, and the sky holds still there. +key c +expect instrument true +frames 200 +# 7 is the number row's C column: C major (bits 0, 4, 7). +key 7 +expect chord.value 145 +frames 90 +screenshot .zig-cache/spheres/chords.png +# U is QWERTY's C column, a row down: C minor (bits 0, 3, 7). +key u +expect chord.value 137 +frames 60 +key escape +expect instrument false diff --git a/examples/spheres/web/icon-192.png b/examples/spheres/web/icon-192.png Binary files differnew file mode 100644 index 0000000..2a47f3d --- /dev/null +++ b/examples/spheres/web/icon-192.png diff --git a/examples/spheres/web/icon-512.png b/examples/spheres/web/icon-512.png Binary files differnew file mode 100644 index 0000000..fb096b8 --- /dev/null +++ b/examples/spheres/web/icon-512.png diff --git a/examples/spheres/web/icon.svg b/examples/spheres/web/icon.svg new file mode 100644 index 0000000..7ed8bf8 --- /dev/null +++ b/examples/spheres/web/icon.svg @@ -0,0 +1,28 @@ +<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"> + <!-- spheres' icon: the figure-eight orbit, its three bodies glowing in + their colours. Full-bleed, so it also works masked. Rendered to the + PNGs beside it with: resvg -w N -h N icon.svg icon-N.png --> + <defs> + <filter id="glow" x="-1" y="-1" width="3" height="3"> + <feGaussianBlur stdDeviation="14"/> + </filter> + <linearGradient id="orbit" gradientUnits="userSpaceOnUse" x1="76" y1="0" x2="436" y2="0"> + <stop offset="0" stop-color="#ffcc59"/> + <stop offset="0.5" stop-color="#ff73d1"/> + <stop offset="1" stop-color="#66d9ff"/> + </linearGradient> + </defs> + <rect width="512" height="512" fill="#05070d"/> + <!-- the figure eight --> + <path d="M256 256 C 300 196 436 176 436 256 C 436 336 300 316 256 256 C 212 196 76 176 76 256 C 76 336 212 316 256 256 Z" + fill="none" stroke="url(#orbit)" stroke-width="10" opacity="0.8"/> + <!-- the bodies: glow, then core --> + <g filter="url(#glow)"> + <circle cx="398" cy="205" r="30" fill="#66d9ff"/> + <circle cx="140" cy="307" r="30" fill="#ff73d1"/> + <circle cx="256" cy="256" r="30" fill="#ffcc59"/> + </g> + <circle cx="398" cy="205" r="15" fill="#e6f8ff"/> + <circle cx="140" cy="307" r="15" fill="#ffe6f6"/> + <circle cx="256" cy="256" r="15" fill="#fff4dc"/> +</svg> |
