Audio: detect a pre-existing render stream's true sample rate (fix pitch)
Hooked audio mirroring played back pitch-shifted on games we inject into that render at a non-device sample rate (e.g. Godot/Brotato render 44100 Hz on a 48000 Hz endpoint via WASAPI AUTOCONVERTPCM). We attach to an already-running game, so the render-hook never saw its IAudioClient:: Initialize and assumed the device mix format -- right channels/bits, wrong rate -- so 44100 audio was rendered as 48000 (+~1.5 semitones). Fix: treat a pre-existing client's format as a guess and measure its true sample rate from the render cadence (frames/sec over a steady-state window, snapped to the nearest standard rate) before publishing it, deferring capture until verified. Discard the first measurement window so the buffer-fill burst at attach time doesn't over-count. Streams created after we inject still carry their exact Initialize format. Channels/bit-depth genuinely can't be recovered for a pre-existing client: AUTOCONVERTPCM hands GetBuffer a fixed staging buffer (no buffer stride to measure -- confirmed empirically) and WASAPI exposes no API for the format. They stay the device-mix guess, which is correct for the common case (engines render stereo float, matching the endpoint). To keep a wrong guess safe, a VirtualQuery clamp stops the capture copy from ever over-reading the source buffer when the guessed bytes/frame is too large. Surface all of this: a per-stream AudioFormatState (known / measuring / measured rate (ch/bits assumed)) in HookStatus, shown in the Audio panel for the hooked path and as "device endpoint (known)" for loopback; clear hook logs; and enriched mirror status strings. Documented in README (Limitations + Lessons learned). The loopback fallback was always correct (post-mix at the device format). Tests: extract a shared, configurable ToneSource (used by coop_tone and the hook self-test); coop_tone takes rate/channels/bits/format args. Rewrite audio_hook_test to a format matrix x both code paths -- see-init (exact) and guess (rate measured) -- plus a byte-incompatible guess that asserts the clamp keeps capture safe. The matrix caught the attach-burst over-count. audio_loopback_test now spawns coop_tone at several source formats to confirm loopback is format-agnostic. 11/11 x64 + 3/3 x86 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
60
README.md
60
README.md
@@ -64,6 +64,21 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
|
||||
back to process-loopback capture, which does *not* mute the game — so the local
|
||||
machine hears the audio twice (guests hear it once). The Audio panel shows which
|
||||
path is active.
|
||||
- **Hooked audio can only recover a pre-existing stream's *sample rate*, not its
|
||||
channels/bit-depth.** The tool injects into an already-running game, so the audio
|
||||
render-hook usually never saw the game's `IAudioClient::Initialize`. It recovers the
|
||||
true **sample rate** by measuring the render cadence (so playback pitch is correct,
|
||||
e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), but **channels and bit-depth
|
||||
can't be detected** — with `AUTOCONVERTPCM` `GetBuffer` returns a fixed staging
|
||||
buffer (no buffer stride to measure) and WASAPI exposes no API for a pre-existing
|
||||
client's format — so they're *assumed* to match the device mix format. That's correct
|
||||
for the common case (engines render stereo float, matching the endpoint, differing
|
||||
only in rate). A game rendering a *different* channel count or bit depth than the
|
||||
device would be mirrored with the wrong layout (garbled audio) on the hooked path —
|
||||
but never an over-read/crash (a `VirtualQuery` clamp guards the copy), and the
|
||||
loopback fallback is always format-correct. The Audio panel shows each stream's
|
||||
format provenance (*known* / *measuring* / *measured rate (ch/bits assumed)*) so the
|
||||
assumption is visible. Streams created *after* injection are captured exactly.
|
||||
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
|
||||
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
|
||||
for RPT.
|
||||
@@ -150,11 +165,15 @@ ctest --test-dir build -C Debug --output-on-failure
|
||||
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
|
||||
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
|
||||
soft-clip / encode for float32 + int16). No device needed.
|
||||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs
|
||||
the hooks, renders a tone through WASAPI in the same process, and asserts the
|
||||
COM vtables were discovered, the frames reached the ring (non-silent), the
|
||||
primary stream was silenced, and the render stream was counted. Skips cleanly if
|
||||
the machine has no audio endpoint.
|
||||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook's **format
|
||||
detection**, the part that gets pitch right. Using a shared configurable
|
||||
`ToneSource` (the same render helper `coop_tone` uses), it renders tones at a matrix
|
||||
of common formats (44100/48000/96000 Hz, mono/stereo/5.1, 16-bit PCM / 32-bit float)
|
||||
and asserts the hook reports the right rate/channels/bits + provenance for **both**
|
||||
code paths: **see-init** (hooks installed first → exact `Initialize` format) and
|
||||
**guess** (render client pre-exists → device-mix guess whose true rate is measured
|
||||
from the cadence, the Brotato/Godot case). Also checks the frames reached the ring
|
||||
non-silent. Skips cleanly with no audio endpoint.
|
||||
- **`srgb_format_test`** — unit test of the `srgb_to_unorm` mapping the hooked
|
||||
video path uses so `*_SRGB`-backbuffer games aren't darkened. No device.
|
||||
- **`opengl_hook_test`** — in-process self-test of the OpenGL capture path:
|
||||
@@ -173,10 +192,12 @@ ctest --test-dir build -C Debug --output-on-failure
|
||||
the backbuffer reached the shared keyed-mutex texture, and a second device can
|
||||
open it by name and read the exact pixels back. Skips cleanly if the machine has
|
||||
no D3D11 device.
|
||||
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI
|
||||
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the
|
||||
shipping process-loopback capture (the fallback path) receives its audio by
|
||||
PID. Skips cleanly if the machine has no audio endpoint.
|
||||
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone configurable WASAPI
|
||||
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) at several source
|
||||
formats (device default, 44100/48000/96000 Hz) and verifies the shipping
|
||||
process-loopback capture (the fallback backend) receives non-silent audio by PID for
|
||||
each — confirming loopback is format-agnostic (it captures post-mix at the device
|
||||
endpoint format). Skips cleanly if the machine has no audio endpoint.
|
||||
|
||||
### Debugging the hooks against a real game
|
||||
|
||||
@@ -300,6 +321,27 @@ Non-obvious things that cost time and constrain the design:
|
||||
the producer-side `AcquireSync` non-blocking (`timeout 0`) so a busy mutex drops a
|
||||
*mirror* frame instead of stalling the game; the Video panel's "Frames lost" line
|
||||
surfaces both capture- and display-stage drops.
|
||||
- **A render client that predates our injection has no knowable format — measure it.**
|
||||
We inject into already-running games, so we usually never see the game's
|
||||
`IAudioClient::Initialize`; the render-hook then assumes the device mix format for that
|
||||
stream. That's wrong for games that render at a non-device rate via WASAPI
|
||||
`AUTOCONVERTPCM` (e.g. Godot / Brotato render 44100 Hz while the endpoint mixes at
|
||||
48000), so the captured audio plays back **pitch-shifted up**. Fix: treat such a format
|
||||
as a *guess* and measure the stream's true sample rate from its render cadence
|
||||
(frames/sec over a short active window, snapped to the nearest standard rate) before
|
||||
publishing it, deferring capture until verified. Discard the first measurement window:
|
||||
the moment we attach, the stream's already-queued buffers arrive in a burst that
|
||||
over-counts (the `audio_hook_test` matrix caught this), so measure the next,
|
||||
steady-state window. **Only the rate is recoverable, though** — channels/bit-depth can't
|
||||
be measured (`AUTOCONVERTPCM` hands `GetBuffer` a *fixed* staging buffer, so there's no
|
||||
buffer stride; confirmed empirically) and WASAPI has no API for a pre-existing client's
|
||||
format, so they stay the device-mix guess. That's right for the common case (engines
|
||||
render stereo float = the endpoint), and a `VirtualQuery` clamp on the capture copy
|
||||
keeps a too-large guessed block from ever over-reading the source buffer. Streams we *do*
|
||||
watch get created carry their exact `Initialize` format, and the loopback fallback
|
||||
captures post-mix at the device format (always correct). The Audio panel shows each
|
||||
stream's provenance (known / measuring / measured rate (ch/bits assumed)) so what the
|
||||
mirror is using is always visible — see Limitations.
|
||||
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
|
||||
the backbuffer inside the game's `Present`, which the game issues at its true render
|
||||
rate regardless of how DWM composites that *window*. So an unfocused game window can
|
||||
|
||||
Reference in New Issue
Block a user