Audio: detect a pre-existing render stream's true sample rate (fix pitch)

Hooked audio mirroring played back pitch-shifted on games we inject into
that render at a non-device sample rate (e.g. Godot/Brotato render 44100 Hz
on a 48000 Hz endpoint via WASAPI AUTOCONVERTPCM). We attach to an
already-running game, so the render-hook never saw its IAudioClient::
Initialize and assumed the device mix format -- right channels/bits, wrong
rate -- so 44100 audio was rendered as 48000 (+~1.5 semitones).

Fix: treat a pre-existing client's format as a guess and measure its true
sample rate from the render cadence (frames/sec over a steady-state window,
snapped to the nearest standard rate) before publishing it, deferring
capture until verified. Discard the first measurement window so the
buffer-fill burst at attach time doesn't over-count. Streams created after
we inject still carry their exact Initialize format.

Channels/bit-depth genuinely can't be recovered for a pre-existing client:
AUTOCONVERTPCM hands GetBuffer a fixed staging buffer (no buffer stride to
measure -- confirmed empirically) and WASAPI exposes no API for the format.
They stay the device-mix guess, which is correct for the common case
(engines render stereo float, matching the endpoint). To keep a wrong guess
safe, a VirtualQuery clamp stops the capture copy from ever over-reading the
source buffer when the guessed bytes/frame is too large.

Surface all of this: a per-stream AudioFormatState (known / measuring /
measured rate (ch/bits assumed)) in HookStatus, shown in the Audio panel for
the hooked path and as "device endpoint (known)" for loopback; clear hook
logs; and enriched mirror status strings. Documented in README (Limitations
+ Lessons learned). The loopback fallback was always correct (post-mix at
the device format).

Tests: extract a shared, configurable ToneSource (used by coop_tone and the
hook self-test); coop_tone takes rate/channels/bits/format args. Rewrite
audio_hook_test to a format matrix x both code paths -- see-init (exact) and
guess (rate measured) -- plus a byte-incompatible guess that asserts the
clamp keeps capture safe. The matrix caught the attach-burst over-count.
audio_loopback_test now spawns coop_tone at several source formats to
confirm loopback is format-agnostic. 11/11 x64 + 3/3 x86 pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 23:41:57 +02:00
parent f15f5cdb36
commit 7264cb2ef4
11 changed files with 970 additions and 445 deletions

View File

@@ -64,6 +64,21 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
back to process-loopback capture, which does *not* mute the game — so the local
machine hears the audio twice (guests hear it once). The Audio panel shows which
path is active.
- **Hooked audio can only recover a pre-existing stream's *sample rate*, not its
channels/bit-depth.** The tool injects into an already-running game, so the audio
render-hook usually never saw the game's `IAudioClient::Initialize`. It recovers the
true **sample rate** by measuring the render cadence (so playback pitch is correct,
e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), but **channels and bit-depth
can't be detected** — with `AUTOCONVERTPCM` `GetBuffer` returns a fixed staging
buffer (no buffer stride to measure) and WASAPI exposes no API for a pre-existing
client's format — so they're *assumed* to match the device mix format. That's correct
for the common case (engines render stereo float, matching the endpoint, differing
only in rate). A game rendering a *different* channel count or bit depth than the
device would be mirrored with the wrong layout (garbled audio) on the hooked path —
but never an over-read/crash (a `VirtualQuery` clamp guards the copy), and the
loopback fallback is always format-correct. The Audio panel shows each stream's
format provenance (*known* / *measuring* / *measured rate (ch/bits assumed)*) so the
assumption is visible. Streams created *after* injection are captured exactly.
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
for RPT.
@@ -150,11 +165,15 @@ ctest --test-dir build -C Debug --output-on-failure
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
soft-clip / encode for float32 + int16). No device needed.
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs
the hooks, renders a tone through WASAPI in the same process, and asserts the
COM vtables were discovered, the frames reached the ring (non-silent), the
primary stream was silenced, and the render stream was counted. Skips cleanly if
the machine has no audio endpoint.
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook's **format
detection**, the part that gets pitch right. Using a shared configurable
`ToneSource` (the same render helper `coop_tone` uses), it renders tones at a matrix
of common formats (44100/48000/96000 Hz, mono/stereo/5.1, 16-bit PCM / 32-bit float)
and asserts the hook reports the right rate/channels/bits + provenance for **both**
code paths: **see-init** (hooks installed first → exact `Initialize` format) and
**guess** (render client pre-exists → device-mix guess whose true rate is measured
from the cadence, the Brotato/Godot case). Also checks the frames reached the ring
non-silent. Skips cleanly with no audio endpoint.
- **`srgb_format_test`** — unit test of the `srgb_to_unorm` mapping the hooked
video path uses so `*_SRGB`-backbuffer games aren't darkened. No device.
- **`opengl_hook_test`** — in-process self-test of the OpenGL capture path:
@@ -173,10 +192,12 @@ ctest --test-dir build -C Debug --output-on-failure
the backbuffer reached the shared keyed-mutex texture, and a second device can
open it by name and read the exact pixels back. Skips cleanly if the machine has
no D3D11 device.
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the
shipping process-loopback capture (the fallback path) receives its audio by
PID. Skips cleanly if the machine has no audio endpoint.
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone configurable WASAPI
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) at several source
formats (device default, 44100/48000/96000 Hz) and verifies the shipping
process-loopback capture (the fallback backend) receives non-silent audio by PID for
each — confirming loopback is format-agnostic (it captures post-mix at the device
endpoint format). Skips cleanly if the machine has no audio endpoint.
### Debugging the hooks against a real game
@@ -300,6 +321,27 @@ Non-obvious things that cost time and constrain the design:
the producer-side `AcquireSync` non-blocking (`timeout 0`) so a busy mutex drops a
*mirror* frame instead of stalling the game; the Video panel's "Frames lost" line
surfaces both capture- and display-stage drops.
- **A render client that predates our injection has no knowable format — measure it.**
We inject into already-running games, so we usually never see the game's
`IAudioClient::Initialize`; the render-hook then assumes the device mix format for that
stream. That's wrong for games that render at a non-device rate via WASAPI
`AUTOCONVERTPCM` (e.g. Godot / Brotato render 44100 Hz while the endpoint mixes at
48000), so the captured audio plays back **pitch-shifted up**. Fix: treat such a format
as a *guess* and measure the stream's true sample rate from its render cadence
(frames/sec over a short active window, snapped to the nearest standard rate) before
publishing it, deferring capture until verified. Discard the first measurement window:
the moment we attach, the stream's already-queued buffers arrive in a burst that
over-counts (the `audio_hook_test` matrix caught this), so measure the next,
steady-state window. **Only the rate is recoverable, though** — channels/bit-depth can't
be measured (`AUTOCONVERTPCM` hands `GetBuffer` a *fixed* staging buffer, so there's no
buffer stride; confirmed empirically) and WASAPI has no API for a pre-existing client's
format, so they stay the device-mix guess. That's right for the common case (engines
render stereo float = the endpoint), and a `VirtualQuery` clamp on the capture copy
keeps a too-large guessed block from ever over-reading the source buffer. Streams we *do*
watch get created carry their exact `Initialize` format, and the loopback fallback
captures post-mix at the device format (always correct). The Audio panel shows each
stream's provenance (known / measuring / measured rate (ch/bits assumed)) so what the
mirror is using is always visible — see Limitations.
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
the backbuffer inside the game's `Present`, which the game issues at its true render
rate regardless of how DWM composites that *window*. So an unfocused game window can