00 · why i built it
why i built it.
I’ve always loved music. When I’m on my own, something is always playing: on aux in the car, through my Sony WH-CH720Ns, or through the Edifier MR3s on my desk. And it’s always at full volume (my family and friends will vouch for that), because I want to catch every small detail in a song, exactly the way the artist made it. I don’t play an instrument, I can’t really sing, and I have zero musical training. I just love listening.
In April 2026 I was at my desk with music on, as usual, and noticed my second screen sitting there doing nothing. My laptop was plugged into my monitor, all the work happened on the big one, and the small one barely got used. What if it showed visuals for whatever I was listening to?
I didn’t even know what that was called. ChatGPT gave me the words: a live, audio-reactive music visualizer. So I went looking for a free one that looked good and just worked with whatever was playing. After days of searching, I came up empty. So I looked up how hard it would be to build one myself. That’s where it started.
Seven months later, it’s called sonora: a free Windows app that listens to whatever’s playing on your PC (Spotify, YouTube, a DJ set, anything) and turns it into live visuals that move with every beat.
01 · the problem
raw fft looks dead.
The obvious music visualizer takes an FFT of the audio and draws the magnitudes: bars, rings, a waveform. It works on a four-on-the-floor house track for about ten seconds. Then you notice that the picture only ever does one thing, and only when the kick hits.
The reasons are measurable. On a loud modern master the band energies sit near their ceiling all the time, so “bigger when louder” barely moves. The high bands carry about a hundredth of the bass band’s energy, so anything driven by hats or vocals never fires. And a magnitude has no idea where it is in the song: a breakdown and an intro look the same, and a drop is just a louder frame.
The fix is not better drawing. It is a layer between the FFT and the visuals that turns sound into a few signals a designer can actually use.
02 · analysis
from samples to features, sixty times a second.
The analysis is a platform-free Rust crate, sonora-dsp. Mono samples go into a 4096-point FFT with a hop of one sixtieth of a second of audio (the hop is derived from the sample rate, so 44.1 and 48 kHz both give exactly 60 frames per second of audio time). Each frame produces seven bands from sub-bass to air, spectral shape, onsets, and a beat.
- Per-band adaptive normalisation. Each band is divided by its own slowly decaying running peak (30 s half-life) with an absolute floor. Hats and vocals finally register next to the bass, and a quiet song moves as much as a loud one.
- Beat tracking. Autocorrelation of a smoothed onset function with a log-Gaussian tempo prior around 120 BPM, and a phase that runs freely and is pulled toward onset peaks like a phase-locked loop instead of being reset. It locks the 124 BPM test track exactly; a 140 BPM half-time track reads as 70 in its drop, which is a fair reading of half time.
03 · the mapping layer
four signals on three timescales.
Styles never read the FFT. They read the mapping layer: a handful of signals, each with a job and a timescale, so that slow things stay slow and accents stay sharp.
| signal | timescale | what it does |
|---|---|---|
| fade in / out | macro · rises in ~3.3 s, falls in ~6.7 s | The scene’s silence gate. A stopped track fades the picture instead of cutting it. |
| intensity | macro · 4 s attack, 6 s release | How much is going on: loudness against the track’s own recent peak, motion density, active bands. Sets size, density, speed. |
| presence × 7 | meso · 200 ms attack, 1.5 s release | Sustained activity per band: a held bass line, a dense mid. Shapes ongoing motion without pulsing on every transient. |
| hits × 7 | micro · 80 ms decay | Transients per band: the kick, the snare, the hat. The frame-accurate accents. |
One rule came out of a flicker bug: responsiveness drives motion, never brightness. At “Punchy”, the fast follower passed frame-to-frame noise from real music into anything that set brightness, and Ink strobed. Now the Responsiveness setting only scales motion (kick strength, pushes, punches), while brightness and density read a fixed, slower follower (120 ms attack, 700 ms release), and hit brightening has fixed headroom per style.
04 · events
drops, builds and track changes.
On top of the signals, an event layer names what is happening: build, drop, breakdown, big hit and track change. It runs once per analysis frame (not per rendered frame), so for the same audio it gives the same events on any machine and in any browser. Every state uses evidence integrators with separate enter and exit levels, so nothing flickers on and off.
- A drop is a sharp jump in loudness and low end within 0.9 s, but only when something armed it: a build, a short gap inside the music, or a long stretch without kick. Then it can’t fire again for 10 s.
- A build is rising tension held for two seconds: the mid level climbing over four seconds with the highs, or highs without bass.
- A track change is at least 1.5 s of silence followed by sound, or a real one from the Windows media session when a player reports it.
05 · one crate, two targets
the same frames on the desktop and in a browser.
The desktop app feeds the crate from WASAPI loopback (through cpal), so it hears whatever Windows plays without a virtual cable. The same crate compiles to WebAssembly with wasm-bindgen (about 316 KB) and runs in an AudioWorklet pipeline in the browser. Frames cross the boundary as JSON produced by the same serde_json as the desktop’s event payload, so the shape is identical by construction.
A parity check runs the native build and the WebAssembly build over the same track and compares every field of every frame. The only differences are single rounding quanta of 0.001, from MSVC’s and WebAssembly’s maths libraries disagreeing in the last digit. That is what makes the browser a faithful test bench: the verification harness, the media pipeline, the live page and this site all see what the app sees.
The desktop side also had to survive real hardware: plugging speakers into a jack could re-route the endpoint without changing its name, and WASAPI loopback delivers no packets during silence. The capture now reopens itself after three seconds without samples, and analyses silence at the normal rate after 120 ms, so the picture decays instead of freezing.
06 · rendering
raw webgl 2, one technique per style.
The renderer is TypeScript and raw WebGL 2, no engine. Every style gets the mapping signals, this frame’s events, a world camera, the resolved palette and a quality level, and draws into a float scene target. A shared post pass adds threshold mip bloom, a hue-preserving tonemap (the brightest channel is tonemapped and the colour scaled with it, so saturated colours above 1.0 stay coloured instead of going white), slow auto exposure, grain and vignette.
ink
Ink started as textbook stable fluids: semi-Lagrangian advection and a pressure projection on a periodic grid. It looked good and was badly out of time. Measured, only 9% of kicks produced a visible change: the analysis caught 95–100% of them within ~30 ms, but an incompressible solver takes hundreds of milliseconds to turn a splat into motion, and the pressure projection deletes a radial splat outright (it is pure divergence).
The Ink that shipped keeps the flow and drops the physics. Velocity is a curl-noise field; ink is advected softly in fixed 1/60 s steps, and between steps the display samples the ink along the flow, so motion is smooth at any refresh. Dye stores palette slots rather than colours, so a palette change recolours ink already on screen. Each hit lands as a seed on the frame of the kick and spreads to full size over 50–100 ms, carried by a CPU copy of the same curl field so it enters the flow instead of popping in.
silk, wisp, warp
One GPU particle engine, its state in RGBA32F textures written with multiple render targets, drives three styles. Silk pulls up to millions of points into threads that accumulate additively into density; its Depth view lays the simulation on the camera’s ground plane and gives each particle a height. Wisp is three flame agents shedding smoke rings on the bass, over a lit ground or down an endless tunnel with its own chase camera. Warp streaks stars past at a speed that follows the song’s intensity, slows to 10% in breakdowns and jumps on drops.
voyage
A raymarched Mandelbox (scale 2.9, chosen after measuring the free space inside candidate scales). A JavaScript copy of the distance estimator lays waypoints ahead of the camera so every point of the path keeps at least a quarter of the preferred clearance, and the camera rides a Catmull-Rom spline through them. The march runs at half resolution with a Halton jitter; a resolve pass reprojects the previous frame through the hit depth and clamps it to the current neighbourhood. Brightening is accepted immediately, so bass lands on the hit frame. The world never moves on beats; the camera does.
| style · look | technique | frame cost |
|---|---|---|
| ink · glow · depth | curl-noise flow, fixed-step advection, 3D height field | 1.50 ms |
| silk · flat | GPU particles, additive density | 2.92 ms |
| wisp · flat / ground / tunnel | flame agents, smoke particles | 1.35 / 1.47 / 1.24 ms |
| warp | GPU particle streaks | 1.43 ms |
| voyage · void / city / ocean | raymarched Mandelbox, temporal upscale | 2.98 / 3.13 / 3.03 ms |
Synced frame cost (GPU and CPU), 1080p, Medium quality, median of three interleaved rounds at the drop of the house test track, on a laptop AMD Radeon RX 7700S.
Every deliberate flash (Silk’s inversion cut, drop flares) asks a WCAG 2.3.1 flash guard first: at most three per second, otherwise the effect is skipped. Independently, a GPU luminance probe counts large swings in what actually reached the screen, from any source, and damps the frame when more than three happen in a second.
07 · the harness
nothing is done until it’s measured.
A verification harness renders any style in a real browser on the real GPU, in audio time with a fixed step, against procedurally generated test tracks whose kicks are known to the sample. Because time is simulated, every render is deterministic and independent of the machine’s frame rate. It gates:
| gate | what it measures | bar |
|---|---|---|
| sync | per known kick, the largest luminance change within 133 ms, whole screen and per region | ≥ 90% of kicks seen |
| flicker | large brightness swings either way on a loud, dense track at Punchy | ≤ 2.5 swings/s, one-frame step ≤ 0.12, ≤ 6% blown out |
| motion | share of frame-to-frame change in fine detail vs large shapes; background stillness | fine share ≤ 45%, background step p99 ≤ 0.02 |
| pop score | new ink not explained by the flow or a brightness change, per 1/60 s | compared before / after |
| dt sweep | the same seeded passage at 60, 120 and 165 fps must agree | sync, flicker, motion, pop within tolerance |
| pacing | real-time frame intervals on the actual display | no held frames |
Today Ink Glow sees 100% of kicks on both the house track and a real song, Marble 100 / 96%, Voyage 100%, Silk 91–96%. The same harness renders every thumbnail and loop in the app.
08 · the ink investigation
four bugs that looked like one.
After the 1.0 release, Ink felt wrong on the developer’s 165 Hz display in a way still frames couldn’t show: blobs “teleporting” in, lines that jagged and broke in fast flow, and a general stutter. The evidence was two 240 fps slow-motion phone recordings of the screen.
Taking it apart one factor at a time found four separate causes:
- Frame pacing. The frame cap defaulted to 60 on a 165 Hz display. No offered cap divided 165, so frames were held for 3-3-3-2 refreshes: judder. A real-time pacing check confirmed it (cap 60: alternating 3 and 2 refreshes per frame; match display: 0 held frames in 1319). The default is now “match display”, with caps that divide the measured refresh evenly.
- Frame-rate dependence. Five places assumed 60 fps. A hit landed over three frames (18 ms at 165 fps instead of 50); a one-frame push was 2.75× weaker; and the ink was resampled, and so blurred, once per rendered frame, roughly twice as much per second at 165 fps (Glow’s median kick brightening fell from 47% to 39%). Everything timed now runs in seconds, and advection runs in fixed steps.
- Spawn pops. Besides the frame-counted landing, a hit’s second and third shares landed at its original spot after the first had already flowed on. Pop score at 165 fps (p99): Glow 7.29 → 1.63, Marble 1.33 → 0.09, Sumi 4.79 → 1.92.
- Jagged lines. Of ten candidate factors (dye resolution, sharpening, three display filters, smoother sampling of the blurred field, advection scheme, grain, dither, veins), only Marble’s veins moved the numbers: a sine of the ink ratio packs several lines into one pixel where the flow stretches, and they alias into crawling saw-teeth. Veins thinner than a pixel now fade to their average tone (Marble Depth fine-detail change 11.2% → 5.6% with them off).
09 · colour
nine palettes, tuned in oklch.
The 1.0 palettes were HSL colours, and a colour matrix (every palette × 19 style / look / view combinations × two tracks, measured per frame) showed what the eye suspected: Voyage’s stone was a fixed grey mixed with a tint, so every palette came out at 0.06–0.11 chroma there; palettes whose two core colours sit far apart on the wheel averaged to grey in RGB (Aurora was 57% grey in Voyage); dark yellows read olive; and HSL lightness made some palettes far brighter than others.
The 1.0.1 set is defined in OKLCH and checked by gates calibrated on the palettes people liked: content chroma, grey share, off-palette and olive share, exposure against Ocean, and colour separation. Voyage’s stone now blends the core colours without losing chroma (Ocean: chroma 0.055 → 0.094, grey 20 → 4%). Palette changes crossfade in OKLCH along the shorter hue arc: the old RGB crossfade dipped to chroma 0.008 between Ocean and Solar; the new one never goes below 0.103. A custom hue skips the two bands sRGB can’t make vivid (dark yellow becomes gold or lime, mid cyan becomes teal or azure).
10 · this website
the real engine, without a microphone.
The live sections of the home page run the actual renderer from the app’s source, not a video. The demo track was generated for the site, analysed by the WebAssembly build of the same crate at build time, and stored as a quantised frame stream: 23 signals, 8 or 16 bits each, about 153 KB for 73 seconds. The track is analysed twice in a row and the second pass kept, so the analyser’s adaptive state at the loop point is exactly what looping playback hears. The events found on the quantised frames match the unquantised ones.
One engine serves the whole page: a single fixed canvas that each section hands a scene as it scrolls into view, rendering only while a live section is on screen. Compiling Voyage’s raymarcher the first time blocks for about three seconds, so every style is first compiled in a Web Worker on an OffscreenCanvas while nothing live is showing; the GPU process caches the programs, and switching sections then takes milliseconds (Voyage: ~3 s → ~40 ms). Phones, low-power devices, browsers without WebGL 2 and reduced-motion settings get the pre-rendered loops or stills instead.
11 · limitations
what it doesn’t do.
- The analysis is heuristic. Hits are band transients, not separated instruments; a low-band hit usually is a kick but isn’t proven to be one. Builds and drops are rules with hysteresis tuned on electronic music. The dense test track’s first drop, which has no gap before it, isn’t flagged as a drop.
- Capture is Windows-only. System audio comes from WASAPI loopback. In a browser the only ways in are a file, the microphone, or sharing a tab’s sound (Chrome and Edge).
- The installer isn’t code-signed yet, so Windows SmartScreen asks before the first run.
- Known gate misses. Wisp Flat’s fine-detail share is 48–50% against the 45% bar. Silk’s drop inversion cut is a deliberate, flash-guarded one-frame change of 0.64–0.77.
The source, the decisions log and the harness are on GitHub.