Recover a guessed stream's channels + bit depth by correlation too (step b)

Extends the two-path correlation from rate-only to the full layout, removing the
"channels/bit-depth assumed = device" limitation. correlate_format tries each
candidate de-interleaving (float32 / int16; mono..7.1) of the hook capture, runs
the rate correlation per layout, and keeps whichever aligns with the loopback;
a wrong de-interleaving is noise and won't.

The catch: the hook can't know a guessed stream's real frame size, so its verify
tap pads each render buffer to the device block -- which over-reads stale staging
bytes for a stream with fewer channels/bits, scrambling the audio. So the tap is
now self-describing: it prefixes each buffer with its frame count
([count][count*device_block bytes]), and the host strips the padding per candidate
layout (take the real count*real_block of each chunk) before de-interleaving.

- audio_correlate.hpp: ChunkedCapture + chunk-aware correlate_format + candidate
  layouts; absolute-margin confidence gate (the true layout scores ~1.0, a truly
  ambiguous alternative within ~0.001 -- 2ch@R == 1ch@2R for identical channels --
  is correctly left unconfident).
- audio_hook.cpp: chunked verify tap (free-space-checked so framing can't tear).
- audio_format_verifier: parse chunks; recover_layout path. AudioMirror now corrects
  the full format.
- audio_correlation_test: layout recovery from padded chunks (stereo float, 16-bit
  PCM, 5.1, mono). audio_verify_test gains scenario (b): 2ch on a multichannel
  endpoint with distinct per-channel content (new env-gated ToneSource mode) ->
  recovers ch=2/32-bit float end-to-end.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-23 02:50:59 +02:00
parent 00244bcfd7
commit 7ada550930
9 changed files with 554 additions and 138 deletions

View File

@@ -77,29 +77,30 @@ default** and covers anything the hooked path doesn't.
back to process-loopback capture, which does *not* mute the game — so the local
machine hears the audio twice (guests hear it once). The Audio panel shows which
path is active.
- **Hooked audio can only recover a pre-existing stream's *sample rate*, not its
channels/bit-depth.** The tool injects into an already-running game, so the audio
render-hook usually never saw the game's `IAudioClient::Initialize`. It recovers the
true **sample rate** by measuring the render cadence (so playback pitch is correct,
e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), but **channels and bit-depth
can't be detected** — with `AUTOCONVERTPCM` `GetBuffer` returns a fixed staging
buffer (no buffer stride to measure) and WASAPI exposes no API for a pre-existing
client's format — so they're *assumed* to match the device mix format. That's correct
for the common case (engines render stereo float, matching the endpoint, differing
only in rate). A game rendering a *different* channel count or bit depth than the
device is mirrored with the wrong layout (garbled audio) on the hooked path, but never
an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`),
and the local **mute** is done with `AUDCLNT_BUFFERFLAGS_SILENT` (which makes WASAPI
ignore the buffer's contents), so it never *writes* the wrongly-sized buffer either. So
every captured stream is silenced locally — guessed or exact — and there is no echo on
the hooked path. The loopback fallback is always format-correct. The Audio panel shows
each stream's
format provenance (*known* / *measuring* / *measured rate* / *low-confidence* /
*override*) so the assumption is visible, and (under Debug details) lets the operator
**re-measure** the rate or **override** the format when the guess is wrong. Overrides
are **remembered per game** (and a format caught exactly at `Initialize` is auto-saved
as that game's override), so a known-bad game is corrected automatically next launch.
Streams created *after* injection are captured exactly.
- **A pre-existing stream's format is *recovered*, not known — by measurement and
cross-correlation.** The tool injects into an already-running game, so the audio
render-hook usually never saw the game's `IAudioClient::Initialize`, and `AUTOCONVERTPCM`
hides the buffer stride (WASAPI exposes no API for a pre-existing client's format). The
hook first recovers the **sample rate** by measuring the render cadence (so playback pitch
is correct, e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), assuming the device's
channels/bit-depth. When the game is still audible (the measurement window), the host then
**cross-correlates the two capture paths** — the hook (pre-mix) against a parallel
process-loopback (post-mix, the known device format) — to verify/correct the rate and to
**recover the channels + bit depth** by trying candidate de-interleavings and keeping the one
that aligns (`audio_format_verifier` → `coop/audio_correlate.hpp`). The recovered format feeds
the existing override channel. The one case it can't resolve is a *genuinely ambiguous* layout
(a stream whose channels carry identical content looks the same as one channel at double the
rate); there it stays the device assumption and the correlation reports low confidence rather
than guess. A wrong/assumed layout is mirrored with the wrong de-interleaving (garbled) but
never an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`),
and the local **mute** uses `AUDCLNT_BUFFERFLAGS_SILENT` (WASAPI ignores the buffer contents),
so it never *writes* the wrongly-sized buffer either. Every captured stream is silenced locally
— guessed or exact — so there's no echo on the hooked path, and the loopback fallback is always
format-correct. The Audio panel shows each stream's format provenance (*known* / *measuring* /
*measured rate* / *low-confidence* / *override*), and (under Debug details) lets the operator
**re-measure** or **override** the format when needed. Overrides are **remembered per game** (and
a format caught exactly at `Initialize` is auto-saved), so a known-bad game is corrected
automatically next launch. Streams created *after* injection are captured exactly.
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
for RPT; F2 frees the operator cursor; **F10 saves a PNG screenshot** (back buffer,
@@ -109,26 +110,6 @@ default** and covers anything the hooked path doesn't.
### Current Tasks
- **Determine a pre-existing stream's audio format by *correlating* the two capture
paths, instead of guessing.** When we attach to an already-running game we never saw its
`IAudioClient::Initialize`, so the render-hook assumes the device mix format and measures
only the sample rate from the render cadence — which can be wrong on a jittery game
(intermittent pitch shift) and can't recover channels/bit-depth at all. But during the
measurement window the game is still audible, so we already have *both* signals of the
same audio: the **process-loopback** capture (post-mix, at the **known** device format)
and the **render-hook** capture (pre-mix, at the unknown format). Cross-correlating them
pins the real format from ground truth rather than a guess. Two tasks:
- **(a) Rate verification/correction.** Resample the hook stream by each candidate standard
rate and cross-correlate against the loopback; the rate that holds alignment with no drift
over the window is the truth. Robust where cadence measurement is noisy — directly hardens
the intermittent pitch-shift symptom. Lands as a verify-and-correct step feeding the
existing rate path (the operator override stays as the manual escape hatch).
- **(b) Channels + bit-depth recovery.** Extend the correlation to the layout the cadence
method *can't* recover: interpret the hook bytes under candidate layouts (float32 vs
int16; mono/stereo/…) and keep whichever de-interleaving correlates with the loopback (a
wrong interpretation is noise and won't). Removes the "channels/bit-depth assumed = device"
limitation, so the garbled-layout case stops being undetectable.
- **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB
subsystem forwards via window messages (`PostMessage`) plus synthesized
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop
@@ -235,15 +216,20 @@ ctest --test-dir build -C Debug --output-on-failure
skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the
true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000
case the cadence method can misread), and that unrelated signals are *not* confidently matched.
Pure header logic, no device.
Also covers **layout recovery** (`correlate_format`): from a self-describing *chunked* capture
(each render buffer padded to the device block, as the hook's verify tap produces it) it strips
the padding per candidate de-interleaving and recovers the true channels + bit depth + rate
(stereo float, 16-bit PCM, 5.1, mono), and leaves a genuinely-ambiguous identical-channel layout
unconfident. Pure header logic, no device.
- **`audio_verify_test`** — integration test of the host's two-path verifier
([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches
`coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so
this rate test isn't perturbed by a channel mismatch — that's the next task), injects the hook
late (a guessed stream), and runs the real `verify_stream_format()`: it co-captures the hook
(pre-mix, via the ring's `verify_capture` tap) and a parallel process-loopback (post-mix) of the
same audio and correlates them. Asserts it recovers the game's true rate, not the device guess.
Skips cleanly without an audio endpoint.
this rate test isn't perturbed by a channel mismatch), injects the hook late (a guessed stream),
and runs the real `verify_stream_format()`: it co-captures the hook (pre-mix, via the ring's
`verify_capture` tap) and a parallel process-loopback (post-mix) of the same audio and correlates
them. Scenario (a) asserts it recovers the true rate; scenario (b) renders a *different* channel
count than the device with distinct per-channel content and asserts it recovers the full layout
(channels + bit depth + rate). Skips cleanly without an audio endpoint.
- **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy
(`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts
the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill
@@ -551,7 +537,15 @@ Non-obvious things that cost time and constrain the design:
the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step
assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a
*different* channel count / bit depth is the layout step, which tries candidate de-interleavings and
keeps whichever correlates.
keeps whichever correlates. **The layout step needs a self-describing tap**: the hook can't know a
guessed stream's real frame size, so it pads each render buffer to the *device* block — which
over-reads stale staging bytes for a stream with fewer channels/bits. The raw padded bytes are
un-decodable (the stale tail scrambles the audio), so the tap prefixes each buffer with its frame
*count* (`[count][count*device_block bytes]`); the host strips the padding per candidate layout
(take the real `count*real_block` of each chunk) before de-interleaving. **The genuinely-ambiguous
case stays unresolved**: a stream whose channels carry identical content is indistinguishable from
one channel at double the rate (`2ch@R` == `1ch@2R` byte-for-byte), so the correlator reports low
confidence and the format stays the device assumption rather than guessing wrong.
- **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The
mirror re-renders the captured ring to the output device. The original feed loop re-primed
(withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free