Recover a guessed stream's channels + bit depth by correlation too (step b)
Extends the two-path correlation from rate-only to the full layout, removing the "channels/bit-depth assumed = device" limitation. correlate_format tries each candidate de-interleaving (float32 / int16; mono..7.1) of the hook capture, runs the rate correlation per layout, and keeps whichever aligns with the loopback; a wrong de-interleaving is noise and won't. The catch: the hook can't know a guessed stream's real frame size, so its verify tap pads each render buffer to the device block -- which over-reads stale staging bytes for a stream with fewer channels/bits, scrambling the audio. So the tap is now self-describing: it prefixes each buffer with its frame count ([count][count*device_block bytes]), and the host strips the padding per candidate layout (take the real count*real_block of each chunk) before de-interleaving. - audio_correlate.hpp: ChunkedCapture + chunk-aware correlate_format + candidate layouts; absolute-margin confidence gate (the true layout scores ~1.0, a truly ambiguous alternative within ~0.001 -- 2ch@R == 1ch@2R for identical channels -- is correctly left unconfident). - audio_hook.cpp: chunked verify tap (free-space-checked so framing can't tear). - audio_format_verifier: parse chunks; recover_layout path. AudioMirror now corrects the full format. - audio_correlation_test: layout recovery from padded chunks (stereo float, 16-bit PCM, 5.1, mono). audio_verify_test gains scenario (b): 2ch on a multichannel endpoint with distinct per-channel content (new env-gated ToneSource mode) -> recovers ch=2/32-bit float end-to-end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
94
README.md
94
README.md
@@ -77,29 +77,30 @@ default** and covers anything the hooked path doesn't.
|
||||
back to process-loopback capture, which does *not* mute the game — so the local
|
||||
machine hears the audio twice (guests hear it once). The Audio panel shows which
|
||||
path is active.
|
||||
- **Hooked audio can only recover a pre-existing stream's *sample rate*, not its
|
||||
channels/bit-depth.** The tool injects into an already-running game, so the audio
|
||||
render-hook usually never saw the game's `IAudioClient::Initialize`. It recovers the
|
||||
true **sample rate** by measuring the render cadence (so playback pitch is correct,
|
||||
e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), but **channels and bit-depth
|
||||
can't be detected** — with `AUTOCONVERTPCM` `GetBuffer` returns a fixed staging
|
||||
buffer (no buffer stride to measure) and WASAPI exposes no API for a pre-existing
|
||||
client's format — so they're *assumed* to match the device mix format. That's correct
|
||||
for the common case (engines render stereo float, matching the endpoint, differing
|
||||
only in rate). A game rendering a *different* channel count or bit depth than the
|
||||
device is mirrored with the wrong layout (garbled audio) on the hooked path, but never
|
||||
an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`),
|
||||
and the local **mute** is done with `AUDCLNT_BUFFERFLAGS_SILENT` (which makes WASAPI
|
||||
ignore the buffer's contents), so it never *writes* the wrongly-sized buffer either. So
|
||||
every captured stream is silenced locally — guessed or exact — and there is no echo on
|
||||
the hooked path. The loopback fallback is always format-correct. The Audio panel shows
|
||||
each stream's
|
||||
format provenance (*known* / *measuring* / *measured rate* / *low-confidence* /
|
||||
*override*) so the assumption is visible, and (under Debug details) lets the operator
|
||||
**re-measure** the rate or **override** the format when the guess is wrong. Overrides
|
||||
are **remembered per game** (and a format caught exactly at `Initialize` is auto-saved
|
||||
as that game's override), so a known-bad game is corrected automatically next launch.
|
||||
Streams created *after* injection are captured exactly.
|
||||
- **A pre-existing stream's format is *recovered*, not known — by measurement and
|
||||
cross-correlation.** The tool injects into an already-running game, so the audio
|
||||
render-hook usually never saw the game's `IAudioClient::Initialize`, and `AUTOCONVERTPCM`
|
||||
hides the buffer stride (WASAPI exposes no API for a pre-existing client's format). The
|
||||
hook first recovers the **sample rate** by measuring the render cadence (so playback pitch
|
||||
is correct, e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), assuming the device's
|
||||
channels/bit-depth. When the game is still audible (the measurement window), the host then
|
||||
**cross-correlates the two capture paths** — the hook (pre-mix) against a parallel
|
||||
process-loopback (post-mix, the known device format) — to verify/correct the rate and to
|
||||
**recover the channels + bit depth** by trying candidate de-interleavings and keeping the one
|
||||
that aligns (`audio_format_verifier` → `coop/audio_correlate.hpp`). The recovered format feeds
|
||||
the existing override channel. The one case it can't resolve is a *genuinely ambiguous* layout
|
||||
(a stream whose channels carry identical content looks the same as one channel at double the
|
||||
rate); there it stays the device assumption and the correlation reports low confidence rather
|
||||
than guess. A wrong/assumed layout is mirrored with the wrong de-interleaving (garbled) but
|
||||
never an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`),
|
||||
and the local **mute** uses `AUDCLNT_BUFFERFLAGS_SILENT` (WASAPI ignores the buffer contents),
|
||||
so it never *writes* the wrongly-sized buffer either. Every captured stream is silenced locally
|
||||
— guessed or exact — so there's no echo on the hooked path, and the loopback fallback is always
|
||||
format-correct. The Audio panel shows each stream's format provenance (*known* / *measuring* /
|
||||
*measured rate* / *low-confidence* / *override*), and (under Debug details) lets the operator
|
||||
**re-measure** or **override** the format when needed. Overrides are **remembered per game** (and
|
||||
a format caught exactly at `Initialize` is auto-saved), so a known-bad game is corrected
|
||||
automatically next launch. Streams created *after* injection are captured exactly.
|
||||
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
|
||||
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
|
||||
for RPT; F2 frees the operator cursor; **F10 saves a PNG screenshot** (back buffer,
|
||||
@@ -109,26 +110,6 @@ default** and covers anything the hooked path doesn't.
|
||||
|
||||
### Current Tasks
|
||||
|
||||
- **Determine a pre-existing stream's audio format by *correlating* the two capture
|
||||
paths, instead of guessing.** When we attach to an already-running game we never saw its
|
||||
`IAudioClient::Initialize`, so the render-hook assumes the device mix format and measures
|
||||
only the sample rate from the render cadence — which can be wrong on a jittery game
|
||||
(intermittent pitch shift) and can't recover channels/bit-depth at all. But during the
|
||||
measurement window the game is still audible, so we already have *both* signals of the
|
||||
same audio: the **process-loopback** capture (post-mix, at the **known** device format)
|
||||
and the **render-hook** capture (pre-mix, at the unknown format). Cross-correlating them
|
||||
pins the real format from ground truth rather than a guess. Two tasks:
|
||||
- **(a) Rate verification/correction.** Resample the hook stream by each candidate standard
|
||||
rate and cross-correlate against the loopback; the rate that holds alignment with no drift
|
||||
over the window is the truth. Robust where cadence measurement is noisy — directly hardens
|
||||
the intermittent pitch-shift symptom. Lands as a verify-and-correct step feeding the
|
||||
existing rate path (the operator override stays as the manual escape hatch).
|
||||
- **(b) Channels + bit-depth recovery.** Extend the correlation to the layout the cadence
|
||||
method *can't* recover: interpret the hook bytes under candidate layouts (float32 vs
|
||||
int16; mono/stereo/…) and keep whichever de-interleaving correlates with the loopback (a
|
||||
wrong interpretation is noise and won't). Removes the "channels/bit-depth assumed = device"
|
||||
limitation, so the garbled-layout case stops being undetectable.
|
||||
|
||||
- **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB
|
||||
subsystem forwards via window messages (`PostMessage`) plus synthesized
|
||||
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop
|
||||
@@ -235,15 +216,20 @@ ctest --test-dir build -C Debug --output-on-failure
|
||||
skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the
|
||||
true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000
|
||||
case the cadence method can misread), and that unrelated signals are *not* confidently matched.
|
||||
Pure header logic, no device.
|
||||
Also covers **layout recovery** (`correlate_format`): from a self-describing *chunked* capture
|
||||
(each render buffer padded to the device block, as the hook's verify tap produces it) it strips
|
||||
the padding per candidate de-interleaving and recovers the true channels + bit depth + rate
|
||||
(stereo float, 16-bit PCM, 5.1, mono), and leaves a genuinely-ambiguous identical-channel layout
|
||||
unconfident. Pure header logic, no device.
|
||||
- **`audio_verify_test`** — integration test of the host's two-path verifier
|
||||
([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches
|
||||
`coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so
|
||||
this rate test isn't perturbed by a channel mismatch — that's the next task), injects the hook
|
||||
late (a guessed stream), and runs the real `verify_stream_format()`: it co-captures the hook
|
||||
(pre-mix, via the ring's `verify_capture` tap) and a parallel process-loopback (post-mix) of the
|
||||
same audio and correlates them. Asserts it recovers the game's true rate, not the device guess.
|
||||
Skips cleanly without an audio endpoint.
|
||||
this rate test isn't perturbed by a channel mismatch), injects the hook late (a guessed stream),
|
||||
and runs the real `verify_stream_format()`: it co-captures the hook (pre-mix, via the ring's
|
||||
`verify_capture` tap) and a parallel process-loopback (post-mix) of the same audio and correlates
|
||||
them. Scenario (a) asserts it recovers the true rate; scenario (b) renders a *different* channel
|
||||
count than the device with distinct per-channel content and asserts it recovers the full layout
|
||||
(channels + bit depth + rate). Skips cleanly without an audio endpoint.
|
||||
- **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy
|
||||
(`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts
|
||||
the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill
|
||||
@@ -551,7 +537,15 @@ Non-obvious things that cost time and constrain the design:
|
||||
the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step
|
||||
assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a
|
||||
*different* channel count / bit depth is the layout step, which tries candidate de-interleavings and
|
||||
keeps whichever correlates.
|
||||
keeps whichever correlates. **The layout step needs a self-describing tap**: the hook can't know a
|
||||
guessed stream's real frame size, so it pads each render buffer to the *device* block — which
|
||||
over-reads stale staging bytes for a stream with fewer channels/bits. The raw padded bytes are
|
||||
un-decodable (the stale tail scrambles the audio), so the tap prefixes each buffer with its frame
|
||||
*count* (`[count][count*device_block bytes]`); the host strips the padding per candidate layout
|
||||
(take the real `count*real_block` of each chunk) before de-interleaving. **The genuinely-ambiguous
|
||||
case stays unresolved**: a stream whose channels carry identical content is indistinguishable from
|
||||
one channel at double the rate (`2ch@R` == `1ch@2R` byte-for-byte), so the correlator reports low
|
||||
confidence and the format stays the device assumption rather than guessing wrong.
|
||||
- **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The
|
||||
mirror re-renders the captured ring to the output device. The original feed loop re-primed
|
||||
(withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free
|
||||
|
||||
Reference in New Issue
Block a user