diff --git a/README.md b/README.md index 359ab51..35a1e75 100644 --- a/README.md +++ b/README.md @@ -77,29 +77,30 @@ default** and covers anything the hooked path doesn't. back to process-loopback capture, which does *not* mute the game — so the local machine hears the audio twice (guests hear it once). The Audio panel shows which path is active. -- **Hooked audio can only recover a pre-existing stream's *sample rate*, not its - channels/bit-depth.** The tool injects into an already-running game, so the audio - render-hook usually never saw the game's `IAudioClient::Initialize`. It recovers the - true **sample rate** by measuring the render cadence (so playback pitch is correct, - e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), but **channels and bit-depth - can't be detected** — with `AUTOCONVERTPCM` `GetBuffer` returns a fixed staging - buffer (no buffer stride to measure) and WASAPI exposes no API for a pre-existing - client's format — so they're *assumed* to match the device mix format. That's correct - for the common case (engines render stereo float, matching the endpoint, differing - only in rate). A game rendering a *different* channel count or bit depth than the - device is mirrored with the wrong layout (garbled audio) on the hooked path, but never - an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`), - and the local **mute** is done with `AUDCLNT_BUFFERFLAGS_SILENT` (which makes WASAPI - ignore the buffer's contents), so it never *writes* the wrongly-sized buffer either. So - every captured stream is silenced locally — guessed or exact — and there is no echo on - the hooked path. The loopback fallback is always format-correct. The Audio panel shows - each stream's - format provenance (*known* / *measuring* / *measured rate* / *low-confidence* / - *override*) so the assumption is visible, and (under Debug details) lets the operator - **re-measure** the rate or **override** the format when the guess is wrong. Overrides - are **remembered per game** (and a format caught exactly at `Initialize` is auto-saved - as that game's override), so a known-bad game is corrected automatically next launch. - Streams created *after* injection are captured exactly. +- **A pre-existing stream's format is *recovered*, not known — by measurement and + cross-correlation.** The tool injects into an already-running game, so the audio + render-hook usually never saw the game's `IAudioClient::Initialize`, and `AUTOCONVERTPCM` + hides the buffer stride (WASAPI exposes no API for a pre-existing client's format). The + hook first recovers the **sample rate** by measuring the render cadence (so playback pitch + is correct, e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), assuming the device's + channels/bit-depth. When the game is still audible (the measurement window), the host then + **cross-correlates the two capture paths** — the hook (pre-mix) against a parallel + process-loopback (post-mix, the known device format) — to verify/correct the rate and to + **recover the channels + bit depth** by trying candidate de-interleavings and keeping the one + that aligns (`audio_format_verifier` → `coop/audio_correlate.hpp`). The recovered format feeds + the existing override channel. The one case it can't resolve is a *genuinely ambiguous* layout + (a stream whose channels carry identical content looks the same as one channel at double the + rate); there it stays the device assumption and the correlation reports low confidence rather + than guess. A wrong/assumed layout is mirrored with the wrong de-interleaving (garbled) but + never an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`), + and the local **mute** uses `AUDCLNT_BUFFERFLAGS_SILENT` (WASAPI ignores the buffer contents), + so it never *writes* the wrongly-sized buffer either. Every captured stream is silenced locally + — guessed or exact — so there's no echo on the hooked path, and the loopback fallback is always + format-correct. The Audio panel shows each stream's format provenance (*known* / *measuring* / + *measured rate* / *low-confidence* / *override*), and (under Debug details) lets the operator + **re-measure** or **override** the format when needed. Overrides are **remembered per game** (and + a format caught exactly at `Initialize` is auto-saved), so a known-bad game is corrected + automatically next launch. Streams created *after* injection are captured exactly. - **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the pipeline, not for end use. F1 hides it entirely so the window is a clean mirror for RPT; F2 frees the operator cursor; **F10 saves a PNG screenshot** (back buffer, @@ -109,26 +110,6 @@ default** and covers anything the hooked path doesn't. ### Current Tasks -- **Determine a pre-existing stream's audio format by *correlating* the two capture - paths, instead of guessing.** When we attach to an already-running game we never saw its - `IAudioClient::Initialize`, so the render-hook assumes the device mix format and measures - only the sample rate from the render cadence — which can be wrong on a jittery game - (intermittent pitch shift) and can't recover channels/bit-depth at all. But during the - measurement window the game is still audible, so we already have *both* signals of the - same audio: the **process-loopback** capture (post-mix, at the **known** device format) - and the **render-hook** capture (pre-mix, at the unknown format). Cross-correlating them - pins the real format from ground truth rather than a guess. Two tasks: - - **(a) Rate verification/correction.** Resample the hook stream by each candidate standard - rate and cross-correlate against the loopback; the rate that holds alignment with no drift - over the window is the truth. Robust where cadence measurement is noisy — directly hardens - the intermittent pitch-shift symptom. Lands as a verify-and-correct step feeding the - existing rate path (the operator override stays as the manual escape hatch). - - **(b) Channels + bit-depth recovery.** Extend the correlation to the layout the cadence - method *can't* recover: interpret the hook bytes under candidate layouts (float32 vs - int16; mono/stereo/…) and keep whichever de-interleaving correlates with the loopback (a - wrong interpretation is noise and won't). Removes the "channels/bit-depth assumed = device" - limitation, so the garbled-layout case stops being undetectable. - - **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB subsystem forwards via window messages (`PostMessage`) plus synthesized `GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop @@ -235,15 +216,20 @@ ctest --test-dir build -C Debug --output-on-failure skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000 case the cadence method can misread), and that unrelated signals are *not* confidently matched. - Pure header logic, no device. + Also covers **layout recovery** (`correlate_format`): from a self-describing *chunked* capture + (each render buffer padded to the device block, as the hook's verify tap produces it) it strips + the padding per candidate de-interleaving and recovers the true channels + bit depth + rate + (stereo float, 16-bit PCM, 5.1, mono), and leaves a genuinely-ambiguous identical-channel layout + unconfident. Pure header logic, no device. - **`audio_verify_test`** — integration test of the host's two-path verifier ([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches `coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so - this rate test isn't perturbed by a channel mismatch — that's the next task), injects the hook - late (a guessed stream), and runs the real `verify_stream_format()`: it co-captures the hook - (pre-mix, via the ring's `verify_capture` tap) and a parallel process-loopback (post-mix) of the - same audio and correlates them. Asserts it recovers the game's true rate, not the device guess. - Skips cleanly without an audio endpoint. + this rate test isn't perturbed by a channel mismatch), injects the hook late (a guessed stream), + and runs the real `verify_stream_format()`: it co-captures the hook (pre-mix, via the ring's + `verify_capture` tap) and a parallel process-loopback (post-mix) of the same audio and correlates + them. Scenario (a) asserts it recovers the true rate; scenario (b) renders a *different* channel + count than the device with distinct per-channel content and asserts it recovers the full layout + (channels + bit depth + rate). Skips cleanly without an audio endpoint. - **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy (`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill @@ -551,7 +537,15 @@ Non-obvious things that cost time and constrain the design: the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a *different* channel count / bit depth is the layout step, which tries candidate de-interleavings and - keeps whichever correlates. + keeps whichever correlates. **The layout step needs a self-describing tap**: the hook can't know a + guessed stream's real frame size, so it pads each render buffer to the *device* block — which + over-reads stale staging bytes for a stream with fewer channels/bits. The raw padded bytes are + un-decodable (the stale tail scrambles the audio), so the tap prefixes each buffer with its frame + *count* (`[count][count*device_block bytes]`); the host strips the padding per candidate layout + (take the real `count*real_block` of each chunk) before de-interleaving. **The genuinely-ambiguous + case stays unresolved**: a stream whose channels carry identical content is indistinguishable from + one channel at double the rate (`2ch@R` == `1ch@2R` byte-for-byte), so the correlator reports low + confidence and the format stays the device assumption rather than guessing wrong. - **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The mirror re-renders the captured ring to the output device. The original feed loop re-primed (withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free diff --git a/common/include/coop/audio_correlate.hpp b/common/include/coop/audio_correlate.hpp index 61e14b3..10ed634 100644 --- a/common/include/coop/audio_correlate.hpp +++ b/common/include/coop/audio_correlate.hpp @@ -254,4 +254,177 @@ inline RateCorrelation correlate_rate(const std::vector& hook_mono, const return result; } +// --- Step (b): channels + bit-depth recovery -------------------------------------------------- +// +// The rate step assumes the hook bytes are de-interleaved at the device channel/bit layout. When a +// game renders a DIFFERENT layout than the device (e.g. stereo float on a 7.1 endpoint, or 16-bit +// PCM), that assumption garbles the waveform and the rate won't lock. We can't measure the layout +// (AUTOCONVERTPCM hides the stride), but we can RECOVER it: interpret the raw hook bytes under each +// candidate layout, run the rate correlation, and keep whichever (layout, rate) aligns with the +// loopback -- a wrong de-interleaving is noise and won't correlate. + +inline constexpr unsigned kWaveFormatPcm = 1; // WAVE_FORMAT_PCM +inline constexpr unsigned kWaveFormatFloat = 3; // WAVE_FORMAT_IEEE_FLOAT + +struct LayoutCandidate +{ + unsigned channels; + unsigned bits; + unsigned tag; // kWaveFormatPcm / kWaveFormatFloat +}; + +// Candidate de-interleavings a shared-mode WASAPI render stream realistically uses: float32 and +// 16-bit PCM, across the common channel counts. Ordered most-likely-first. +inline const std::vector& standard_audio_layouts() +{ + static const std::vector v = { + {2, 32, kWaveFormatFloat}, {1, 32, kWaveFormatFloat}, {6, 32, kWaveFormatFloat}, + {8, 32, kWaveFormatFloat}, {4, 32, kWaveFormatFloat}, {2, 16, kWaveFormatPcm}, + {1, 16, kWaveFormatPcm}, {6, 16, kWaveFormatPcm}, {8, 16, kWaveFormatPcm}, + {4, 16, kWaveFormatPcm}, + }; + return v; +} + +struct FormatCorrelation +{ + bool ok = false; + unsigned rate = 0; + unsigned channels = 0; + unsigned bits = 0; + unsigned tag = 0; + double score = 0.0; + double runner_up = 0.0; +}; + +// The hook can't know a guessed stream's real frame size, so its verify tap pushes each render +// buffer padded to the device block (`stride`) and prefixes it with the real frame `count`. That +// padding is stale staging-buffer bytes, so the host must extract the real `count*real_block` bytes +// per buffer (and concatenate) before de-interleaving -- otherwise the padding scrambles the audio. +// This carries that self-describing capture: `bytes` holds counts[i]*stride bytes per chunk. +struct ChunkedCapture +{ + unsigned stride = 0; // bytes per frame as pushed (the guessed/device block_align) + std::vector counts; // real frame count of each chunk + std::vector bytes; // concatenated, counts[i]*stride bytes per chunk +}; + +namespace correlate_detail +{ +// De-interleave raw bytes under (channels/bits/tag) and average to mono float. +inline void decode_layout(const std::uint8_t* bytes, std::size_t n, const LayoutCandidate& fmt, + std::vector& mono) +{ + mono.clear(); + const unsigned ch = fmt.channels == 0 ? 1 : fmt.channels; + const unsigned bps = fmt.bits / 8; + if (bps == 0) + { + return; + } + const std::size_t frame = static_cast(ch) * bps; + const std::size_t frames = n / frame; + mono.resize(frames); + const bool is_float = fmt.tag == kWaveFormatFloat; + for (std::size_t i = 0; i < frames; ++i) + { + double sum = 0.0; + for (unsigned c = 0; c < ch; ++c) + { + const std::uint8_t* p = bytes + i * frame + static_cast(c) * bps; + float s = 0.0f; + if (is_float && fmt.bits == 32) + { + std::memcpy(&s, p, 4); + } + else if (fmt.bits == 16) + { + std::int16_t v; + std::memcpy(&v, p, 2); + s = v / 32768.0f; + } + else if (fmt.bits == 32) + { + std::int32_t v; + std::memcpy(&v, p, 4); + s = static_cast(v / 2147483648.0); + } + sum += s; + } + mono[i] = static_cast(sum / ch); + } +} +} // namespace correlate_detail + +// Recover BOTH the layout and the rate of a guessed stream from its (padded, self-describing) hook +// capture: for each candidate layout, extract the real count*real_block bytes from each padded +// chunk, de-interleave to mono, and run the rate correlation against the known-format loopback, +// keeping the (layout, rate) that aligns best. `ok` when the winner clears the alignment floor and +// clearly beats the runner-up (so a coincidental partial match is rejected). +// +// `min_margin` is an ABSOLUTE gap (not a ratio): the true layout scores near-perfectly while a +// truly-ambiguous alternative (e.g. 1ch@2R vs 2ch@R when the channels carry the same content) +// scores within ~0.001, so requiring the winner to clear the runner-up by a fixed margin cleanly +// separates "recovered" from "genuinely ambiguous, don't guess". +inline FormatCorrelation correlate_format(const ChunkedCapture& hook, const std::vector& loop_mono, + unsigned device_rate, const std::vector& rates, + const std::vector& layouts, double min_score = 0.55, + double min_margin = 0.04) +{ + FormatCorrelation result; + if (hook.stride == 0 || hook.counts.empty() || loop_mono.empty() || device_rate == 0) + { + return result; + } + double best = -1.0, second = -1.0; + std::vector clean; + std::vector hook_mono; + for (const LayoutCandidate& layout : layouts) + { + const unsigned real_block = layout.channels * (layout.bits / 8); + if (real_block == 0 || real_block > hook.stride) + { + continue; // can't extract a frame larger than what was pushed (the guess is the max) + } + // Pull the real count*real_block bytes out of each padded chunk and concatenate -> contiguous + // audio for this candidate layout (the padding, which is stale staging bytes, is dropped). + clean.clear(); + std::size_t off = 0; + for (std::uint32_t count : hook.counts) + { + const std::size_t chunk_bytes = static_cast(count) * hook.stride; + const std::size_t take = static_cast(count) * real_block; + if (off + chunk_bytes <= hook.bytes.size()) + { + clean.insert(clean.end(), hook.bytes.begin() + off, hook.bytes.begin() + off + take); + } + off += chunk_bytes; + } + correlate_detail::decode_layout(clean.data(), clean.size(), layout, hook_mono); + if (hook_mono.size() < device_rate / 5) + { + continue; // this layout yields too little audio to judge + } + const RateCorrelation rc = correlate_rate(hook_mono, loop_mono, device_rate, rates, /*min_score=*/0.0, + /*separation=*/1.0); + if (rc.score > best) + { + second = best; + best = rc.score; + result.rate = rc.rate; + result.channels = layout.channels; + result.bits = layout.bits; + result.tag = layout.tag; + } + else if (rc.score > second) + { + second = rc.score; + } + } + result.score = best < 0.0 ? 0.0 : best; + result.runner_up = second < 0.0 ? 0.0 : second; + result.ok = result.score >= min_score && (result.runner_up <= 1e-6 || result.score - result.runner_up >= min_margin); + return result; +} + } // namespace coop diff --git a/common/include/coop/audio_ring.hpp b/common/include/coop/audio_ring.hpp index 177bfa9..4fd3b47 100644 --- a/common/include/coop/audio_ring.hpp +++ b/common/include/coop/audio_ring.hpp @@ -203,6 +203,15 @@ inline std::uint32_t audio_ring_available(const AudioRingHeader& h) return static_cast(w - r); } +// Producer: bytes currently free (so a multi-part packet can be checked to fit before any of it is +// written -- keeps a self-describing [header][payload] framing from tearing on a full ring). +inline std::uint32_t audio_ring_free_space(const AudioRingHeader& h) +{ + const std::uint64_t w = h.write_pos.load(std::memory_order_relaxed); + const std::uint64_t r = h.read_pos.load(std::memory_order_acquire); + return h.capacity - static_cast(w - r); +} + // Consumer: copy up to `bytes` into `dst`; returns the number actually popped. inline std::uint32_t audio_ring_pop(AudioRingHeader& h, void* dst, std::uint32_t bytes) { diff --git a/hook/src/audio_hook.cpp b/hook/src/audio_hook.cpp index 6ef9400..9d59686 100644 --- a/hook/src/audio_hook.cpp +++ b/hook/src/audio_hook.cpp @@ -410,8 +410,19 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_ const std::uint32_t block = g_streams[i].block_align.load(std::memory_order_relaxed); if (block != 0) { - audio_ring_push(*vring, t_gb_data, readable_bytes(t_gb_data, num_frames * block), - num_frames); + // Self-describing chunk: [u32 frame-count][num_frames*block bytes]. The host can't + // know the real frame size of a guessed stream, so it recovers the layout by + // trying candidate de-interleavings -- but it needs the frame count to strip the + // per-buffer padding (the guessed/device block over-reads a stream with fewer + // channels/bits). Push both parts only if both fit and the payload is fully + // readable, so a full ring or a short buffer can never tear the framing. + const std::uint32_t want = num_frames * block; + if (readable_bytes(t_gb_data, want) == want && + audio_ring_free_space(*vring) >= static_cast(sizeof(num_frames)) + want) + { + audio_ring_push(*vring, &num_frames, sizeof(num_frames), 0); + audio_ring_push(*vring, t_gb_data, want, num_frames); + } } } } diff --git a/host/src/audio/audio_format_verifier.cpp b/host/src/audio/audio_format_verifier.cpp index 5171ffa..6bf2677 100644 --- a/host/src/audio/audio_format_verifier.cpp +++ b/host/src/audio/audio_format_verifier.cpp @@ -99,6 +99,35 @@ void drain_ring(AudioRingHeader& ring, std::vector& scratch) } } +// Parse the hook's self-describing verify stream -- a sequence of [u32 frame-count][count*stride +// bytes] chunks -- into a ChunkedCapture. `stride` is the device block_align (what the hook padded +// each buffer to). Stops at the first truncated/garbled record. +ChunkedCapture parse_chunks(const std::vector& raw, unsigned stride) +{ + ChunkedCapture cap; + cap.stride = stride; + if (stride == 0) + { + return cap; + } + std::size_t off = 0; + while (off + sizeof(std::uint32_t) <= raw.size()) + { + std::uint32_t count = 0; + std::memcpy(&count, raw.data() + off, sizeof(count)); + off += sizeof(count); + const std::size_t payload = static_cast(count) * stride; + if (count == 0 || off + payload > raw.size()) + { + break; // truncated or garbled -> stop + } + cap.counts.push_back(count); + cap.bytes.insert(cap.bytes.end(), raw.data() + off, raw.data() + off + payload); + off += payload; + } + return cap; +} + } // namespace FormatVerification verify_stream_format(DWORD pid, AudioRingHeader* ring, unsigned window_ms, bool recover_layout) @@ -156,31 +185,53 @@ FormatVerification verify_stream_format(DWORD pid, AudioRingHeader* ring, unsign ring->verify_capture.store(0, std::memory_order_release); drain_ring(*ring, scratch); // leave the ring clean for the real capture that follows - // The hook bytes are at the guessed layout = the device channels/bits (the assumption the - // cadence path also makes). Decode both captures with that layout and correlate. - const std::vector hook_mono = to_mono(hook_bytes, dev); + // The loopback is at the KNOWN device layout. Decode it to mono ground truth. The hook stream + // is self-describing chunks ([count][padded payload]); parse them at the device block. const std::vector loop_mono = to_mono(loop_bytes, dev); + const unsigned dev_block = dev_wfx->nBlockAlign; + const ChunkedCapture cap = parse_chunks(hook_bytes, dev_block); + const std::size_t need = dev.rate / 5; // require >= ~200 ms of usable audio on both sides + const std::size_t hook_frames = dev_block != 0 ? cap.bytes.size() / dev_block : 0; CoTaskMemFree(dev_wfx); if (const char* dbg = std::getenv("COOP_VERIFY_DEBUG"); dbg != nullptr && dbg[0] == '1') { - std::fprintf(stderr, "[verify] dev=%uHz/%uch/%ubit tag=%u hook_frames=%zu loop_frames=%zu\n", dev.rate, - dev.channels, dev.bits, dev.tag, hook_mono.size(), loop_mono.size()); + std::fprintf(stderr, "[verify] dev=%uHz/%uch/%ubit blk=%u chunks=%zu hook_frames=%zu loop=%zu layout=%d\n", + dev.rate, dev.channels, dev.bits, dev_block, cap.counts.size(), hook_frames, loop_mono.size(), + recover_layout ? 1 : 0); } - const std::size_t need = dev.rate / 5; // require >= ~200 ms of usable audio on both sides - if (hook_mono.size() < need || loop_mono.size() < need) + if (loop_mono.size() < need || hook_frames < need) { return result; // not enough non-silent audio captured (game quiet, or stream wasn't a guess) } - const RateCorrelation rc = correlate_rate(hook_mono, loop_mono, dev.rate, standard_audio_rates()); - result.ok = rc.ok; - result.rate = rc.rate; - result.score = rc.score; - // Channels/bit-depth stay the device assumption here; step (b) recovers them. - result.channels = dev.channels; - result.bits = dev.bits; - result.format_tag = dev.tag; + if (recover_layout) + { + // Step (b): recover channels + bit depth too, by trying candidate de-interleavings of the + // (de-padded) hook bytes and keeping whichever (layout, rate) correlates with the loopback. + const FormatCorrelation fc = + correlate_format(cap, loop_mono, dev.rate, standard_audio_rates(), standard_audio_layouts()); + result.ok = fc.ok; + result.rate = fc.rate; + result.score = fc.score; + result.layout_ok = fc.ok; + result.channels = fc.channels; + result.bits = fc.bits; + result.format_tag = fc.tag; + } + else + { + // Step (a): rate only, assuming the hook layout matches the device (common stereo case), so + // the de-padded payload is already clean device-layout audio. + const std::vector hook_mono = to_mono(cap.bytes, dev); + const RateCorrelation rc = correlate_rate(hook_mono, loop_mono, dev.rate, standard_audio_rates()); + result.ok = rc.ok; + result.rate = rc.rate; + result.score = rc.score; + result.channels = dev.channels; + result.bits = dev.bits; + result.format_tag = dev.tag; + } return result; } diff --git a/host/src/audio/audio_loopback.cpp b/host/src/audio/audio_loopback.cpp index 5c8129e..7171ee7 100644 --- a/host/src/audio/audio_loopback.cpp +++ b/host/src/audio/audio_loopback.cpp @@ -323,10 +323,15 @@ void AudioMirror::thread_main(DWORD pid) if (!format_verified) { format_verified = true; - const FormatVerification fv = verify_stream_format(pid, rings[0]); + // recover_layout: correlate the full format (rate AND channels/bit-depth), so a + // game rendering a different layout than the device is corrected too, not just + // the rate. A no-op when nothing correlates confidently (e.g. an exact stream, + // a silent game, or a genuinely ambiguous identical-channel layout). + const FormatVerification fv = verify_stream_format(pid, rings[0], /*window_ms=*/900, + /*recover_layout=*/true); if (fv.ok) { - set_status("Verified render-hook sample rate by correlation."); + set_status("Verified render-hook format by correlation."); audio_ring_post_op(*rings[0], AudioRingOp_Override, fv.rate, fv.channels, fv.bits, fv.format_tag); } diff --git a/tests/audio_correlation_test.cpp b/tests/audio_correlation_test.cpp index 663831f..479c8d1 100644 --- a/tests/audio_correlation_test.cpp +++ b/tests/audio_correlation_test.cpp @@ -6,6 +6,7 @@ // rates (plus a capture-latency skew and a little noise), then assert correlate_rate() recovers the // true rate -- including the hard 44100-vs-48000 case the cadence method can misread. Pure header // logic, no device. +#include #include #include #include @@ -55,6 +56,118 @@ std::vector capture(unsigned rate, double seconds, double t0, double nois return out; } +// Per-channel continuous signal: each channel carries genuinely different content (its own +// frequency set), like a real stereo/surround stream. This is what disambiguates the channel +// count -- with identical channels, 2ch@R and 1ch@2R produce the same bytes and are truly +// indistinguishable (the confidence gate correctly rejects that case). +double multi(double t, unsigned channel) +{ + const double k = 1.0 + 0.37 * static_cast(channel); // distinct frequency scale per channel + const double chirp = std::sin(2.0 * kPi * (300.0 * k * t + 140.0 * t * t)); + return 0.5 * std::sin(2.0 * kPi * 221.0 * k * t) + 0.28 * std::sin(2.0 * kPi * 437.0 * k * t + 0.6) + + 0.22 * chirp; +} + +double multi_mono(double t, unsigned channels) +{ + double sum = 0.0; + for (unsigned c = 0; c < channels; ++c) + { + sum += multi(t, c); + } + return sum / channels; +} + +// Encode the multichannel signal to raw interleaved PCM bytes at the given layout/rate. +std::vector encode(unsigned rate, unsigned channels, unsigned bits, unsigned tag, double seconds) +{ + const unsigned bps = bits / 8; + const std::size_t frames = static_cast(rate * seconds); + std::vector out(frames * channels * bps); + for (std::size_t i = 0; i < frames; ++i) + { + const double t = static_cast(i) / rate; + for (unsigned c = 0; c < channels; ++c) + { + const double s = multi(t, c); + std::uint8_t* p = out.data() + (i * channels + c) * bps; + if (tag == coop::kWaveFormatFloat) + { + const float f = static_cast(s); + std::memcpy(p, &f, 4); + } + else + { + const std::int16_t v = static_cast(s * 30000.0); + std::memcpy(p, &v, 2); + } + } + } + return out; +} + +// Build the hook's self-describing chunked capture from clean audio: split into ~480-frame chunks +// and pad each frame from real_block up to `stride` with GARBAGE -- exactly what the hook's verify +// tap produces (it over-reads a guessed stream whose real layout has fewer channels/bits than the +// device block). correlate_format must strip the padding per candidate layout. +coop::ChunkedCapture make_chunks(unsigned rate, unsigned ch, unsigned bits, unsigned tag, unsigned stride, + double seconds) +{ + coop::ChunkedCapture cap; + cap.stride = stride; + const std::vector clean = encode(rate, ch, bits, tag, seconds); + const unsigned real_block = ch * (bits / 8); + const std::size_t frames = clean.size() / real_block; + std::mt19937 rng(123); + std::uniform_int_distribution garbage(0, 255); + std::size_t f = 0; + while (f < frames) + { + const unsigned count = static_cast(std::min(480, frames - f)); + cap.counts.push_back(count); + // The hook reads count*stride contiguous bytes: the count real frames first + // (count*real_block bytes), then count*(stride-real_block) bytes of stale over-read. + const std::uint8_t* src = clean.data() + f * real_block; + cap.bytes.insert(cap.bytes.end(), src, src + static_cast(count) * real_block); + for (std::size_t p = 0; p < static_cast(count) * (stride - real_block); ++p) + { + cap.bytes.push_back(static_cast(garbage(rng))); + } + f += count; + } + return cap; +} + +// One layout scenario: the hook bytes are at (true_*) and still being measured; the loopback is the +// post-mix mono of the same audio at device_rate. Assert correlate_format recovers the full layout. +void test_layout(unsigned true_rate, unsigned true_ch, unsigned true_bits, unsigned true_tag, + unsigned device_rate, const char* label) +{ + std::printf("== layout: %s (%u Hz / %u ch / %u-bit %s -> device %u Hz) ==\n", label, true_rate, true_ch, + true_bits, true_tag == coop::kWaveFormatFloat ? "float" : "pcm", device_rate); + // Device block 32 (8ch float) is the largest stride; every test layout's real block is <= 32. + const coop::ChunkedCapture hook = make_chunks(true_rate, true_ch, true_bits, true_tag, /*stride=*/32, 0.6); + // Loopback: the post-mix mono of the same audio, at the device rate, started ~18 ms later + noise. + const std::size_t loop_frames = static_cast(device_rate * 0.6); + std::vector loop(loop_frames); + std::mt19937 rng(5); + std::uniform_real_distribution j(-1.0f, 1.0f); + for (std::size_t m = 0; m < loop_frames; ++m) + { + loop[m] = static_cast(multi_mono(0.018 + static_cast(m) / device_rate, true_ch)) + + 0.02f * j(rng); + } + + const FormatCorrelation r = + correlate_format(hook, loop, device_rate, standard_audio_rates(), standard_audio_layouts()); + std::printf(" picked %u Hz / %u ch / %u-bit %s score=%.3f runner_up=%.3f ok=%d\n", r.rate, r.channels, + r.bits, r.tag == coop::kWaveFormatFloat ? "float" : "pcm", r.score, r.runner_up, r.ok ? 1 : 0); + check(r.ok, "layout pick is confident"); + check(r.rate == true_rate, "recovered the true rate"); + check(r.channels == true_ch, "recovered the true channel count"); + check(r.bits == true_bits && r.tag == true_tag, "recovered the true bit depth / sample format"); +} + // One scenario: true hook rate `true_rate` mixed to `device_rate`. Assert the correlator picks // true_rate confidently and that the runner-up is clearly behind. void test_case(unsigned true_rate, unsigned device_rate, const char* label) @@ -84,6 +197,13 @@ int main() test_case(32000, 44100, "low-rate stream on a 44100 endpoint"); test_case(48000, 44100, "48000 stream on a 44100 endpoint"); + // Step (b): recover the full layout (channels + bit depth) when it differs from the device, by + // trying candidate de-interleavings -- the case the rate-only step can't handle. + test_layout(44100, 2, 32, coop::kWaveFormatFloat, 48000, "stereo float, wrong rate"); + test_layout(44100, 2, 16, coop::kWaveFormatPcm, 48000, "stereo 16-bit PCM (bit depth differs)"); + test_layout(48000, 6, 32, coop::kWaveFormatFloat, 48000, "5.1 float (channels differ)"); + test_layout(44100, 1, 32, coop::kWaveFormatFloat, 48000, "mono"); // 2ch@22050 isn't a candidate -> unambiguous + // Downmix sanity: a stereo interleaved buffer collapses to the same mono the scalar path uses. { std::printf("== downmix stereo -> mono ==\n"); diff --git a/tests/audio_verify_test.cpp b/tests/audio_verify_test.cpp index 8427590..5cf1113 100644 --- a/tests/audio_verify_test.cpp +++ b/tests/audio_verify_test.cpp @@ -1,22 +1,23 @@ // Integration test for the two-path audio-format verifier (host/src/audio/audio_format_verifier). // -// Launches coop_mock_game rendering a tone at a NON-device rate (44100 on a typical 48000 endpoint, -// the Godot/Brotato case), injects coop_hook.dll late (so the stream is a *guess*), then runs the -// real verify_stream_format(): it co-captures the hook (pre-mix, via the ring's verify tap) and a -// parallel process-loopback (post-mix, device format) and cross-correlates them. Asserts it -// recovers the true 44100 Hz rate -- the cadence method's hard case. Skips cleanly without an audio -// endpoint / if Vulkan-free... (only needs WASAPI + a D3D11-capable mock, which the mock always is). +// Launches coop_mock_game rendering a tone via WASAPI AUTOCONVERTPCM, injects coop_hook.dll late +// (so the stream is a *guess*), and runs the real verify_stream_format(): it co-captures the hook +// (pre-mix, via the ring's verify tap) and a parallel process-loopback (post-mix, device format) +// and cross-correlates them. Two scenarios: +// (a) rate: render at the device's channel count but a different rate -> recover the rate. +// (b) layout: render a DIFFERENT channel count than the device, with distinct per-channel content +// -> recover channels + bit depth + rate. +// Skips cleanly without an audio endpoint. #include #include #include #include +#include #include #include -#include - #include "audio/audio_format_verifier.hpp" #include "audio/process_loopback_capture.hpp" // default_render_format #include "coop/audio_ring.hpp" @@ -111,42 +112,42 @@ bool inject_retry(unsigned long pid) } return false; } -} // namespace -int main() +struct Scenario { + unsigned rate, channels, bits; + bool distinct; // distinct per-channel content (so the channel count is recoverable) + bool recover_layout; // false = rate only (step a); true = full layout (step b) +}; + +// Launch the mock at the scenario's format, inject the hook late, and run the verifier. `ran` is +// set false when the environment can't support the test (launch/inject failed) so the caller skips. +FormatVerification run(const Scenario& sc, bool& ran) +{ + ran = false; + FormatVerification fv; kill_stray_mock_games(); - const bool com = SUCCEEDED(CoInitializeEx(nullptr, COINIT_MULTITHREADED)); - - // Render the mock at the device's CHANNEL count (so this step-(a) rate test isn't perturbed by - // a channel mismatch -- that's step (b)'s job) but at a DIFFERENT standard rate than the device, - // so the verifier has a real rate to recover. Default to 48000/2ch if we can't read the device. - unsigned dev_rate = 48000, dev_channels = 2; - if (WAVEFORMATEX* dev = default_render_format()) + if (sc.distinct) { - dev_rate = dev->nSamplesPerSec; - dev_channels = dev->nChannels; - CoTaskMemFree(dev); + SetEnvironmentVariableW(L"COOP_TONE_DISTINCT_CH", L"1"); } - const unsigned game_rate = (dev_rate == 44100) ? 48000u : 44100u; // guarantee a rate mismatch - std::printf(" device %u Hz / %u ch -> rendering the mock at %u Hz / %u ch (rate mismatch)\n", dev_rate, - dev_channels, game_rate, dev_channels); - const std::wstring exe = exe_directory() + L"coop_mock_game.exe"; - std::wstring cmd = L"\"" + exe + L"\" dx11 30 " + std::to_wstring(game_rate) + L" " + - std::to_wstring(dev_channels) + L" 32 float"; + std::wstring cmd = L"\"" + exe + L"\" dx11 30 " + std::to_wstring(sc.rate) + L" " + + std::to_wstring(sc.channels) + L" " + std::to_wstring(sc.bits) + L" " + + (sc.bits == 16 ? L"pcm" : L"float"); STARTUPINFOW si{}; si.cb = sizeof(si); PROCESS_INFORMATION pi{}; - if (!CreateProcessW(exe.c_str(), cmd.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, &si, &pi)) + const BOOL launched = CreateProcessW(exe.c_str(), cmd.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, + &si, &pi); + if (sc.distinct) { - std::printf("Could not launch coop_mock_game -- skipping audio_verify_test.\n"); - if (com) - { - CoUninitialize(); - } - return 0; + SetEnvironmentVariableW(L"COOP_TONE_DISTINCT_CH", nullptr); + } + if (!launched) + { + return fv; } auto cleanup = [&] { TerminateProcess(pi.hProcess, 0); @@ -155,73 +156,104 @@ int main() CloseHandle(pi.hProcess); kill_stray_mock_games(); }; - Sleep(800); // window + audio client up + Sleep(800); - // IPC + the primary audio ring the hook produces into. SharedMemory shm; - if (!shm.create(shared_memory_name(pi.dwProcessId), sizeof(SharedBlock))) + SharedMemory ring_shm; + if (!shm.create(shared_memory_name(pi.dwProcessId), sizeof(SharedBlock)) || + !ring_shm.create(audio_ring_name(pi.dwProcessId), audio_ring_total_size(kAudioRingCapacity))) { - std::printf("Could not create IPC block -- skipping.\n"); cleanup(); - return 0; + return fv; } auto* block = shm.as(); block->version = kProtocolVersion; block->pad_count = 0; block->sequence.store(0, std::memory_order_relaxed); - // Only the audio subsystem. - for (std::uint32_t s = 0; s < HookSubsys_Count; ++s) + for (std::uint32_t s = 0; s < HookSubsys_Count; ++s) // audio subsystem only { - const bool off = s != HookSubsys_Audio; - block->control.subsystem_disabled[s].store(off ? 1u : 0u, std::memory_order_release); + block->control.subsystem_disabled[s].store(s != HookSubsys_Audio ? 1u : 0u, std::memory_order_release); } block->magic = kProtocolMagic; - - SharedMemory ring_shm; - if (!ring_shm.create(audio_ring_name(pi.dwProcessId), audio_ring_total_size(kAudioRingCapacity))) - { - std::printf("Could not create audio ring -- skipping.\n"); - cleanup(); - return 0; - } auto* ring = ring_shm.as(); - audio_ring_init(*ring, kAudioRingCapacity); - // Leave capture_enabled = 0: we want the stream audible (so loopback hears it) and still being - // MEASURED (so the verify tap fires), exactly the window verify_stream_format targets. + audio_ring_init(*ring, kAudioRingCapacity); // capture_enabled stays 0: audible + still measuring if (!inject_retry(pi.dwProcessId)) { - std::printf("Could not inject coop_hook.dll -- skipping.\n"); cleanup(); - return 0; + return fv; } - Sleep(500); // let the hook attach + the pre-existing render client register as a guess + Sleep(500); // hook attaches + the pre-existing render client registers as a guess - // Run the real verifier: co-capture hook (pre-mix) + loopback (post-mix) and correlate. - const FormatVerification fv = verify_stream_format(pi.dwProcessId, ring, /*window_ms=*/1400); - std::printf(" verify: ok=%d rate=%u score=%.3f\n", fv.ok ? 1 : 0, fv.rate, fv.score); + fv = verify_stream_format(pi.dwProcessId, ring, /*window_ms=*/1400, sc.recover_layout); + ran = true; + cleanup(); + return fv; +} +} // namespace - if (!fv.ok && fv.rate == 0 && fv.score == 0.0) +int main() +{ + const bool com = SUCCEEDED(CoInitializeEx(nullptr, COINIT_MULTITHREADED)); + + unsigned dev_rate = 48000, dev_channels = 2; + if (WAVEFORMATEX* dev = default_render_format()) { - // No audio endpoint, or no usable audio captured (e.g. the mock's WASAPI client never - // started on this machine) -> treat as a skip rather than a failure. - std::printf(" no usable co-capture (no endpoint / silent) -- skipping audio_verify_test.\n"); + dev_rate = dev->nSamplesPerSec; + dev_channels = dev->nChannels; + CoTaskMemFree(dev); + } + const unsigned mismatched = (dev_rate == 44100) ? 48000u : 44100u; // guarantee a rate mismatch + std::printf("device: %u Hz / %u ch\n", dev_rate, dev_channels); + + // (a) Rate: render at the device's channel count (no layout mismatch) but a different rate. + std::printf("== (a) rate recovery: %u Hz / %u ch ==\n", mismatched, dev_channels); + bool ran = false; + FormatVerification a = run({mismatched, dev_channels, 32, /*distinct=*/false, /*recover_layout=*/false}, ran); + if (!ran) + { + std::printf(" environment can't run the mock+inject -- skipping audio_verify_test.\n"); if (com) { CoUninitialize(); } - cleanup(); return 0; } + std::printf(" ok=%d rate=%u score=%.3f\n", a.ok ? 1 : 0, a.rate, a.score); + if (!a.ok && a.rate == 0 && a.score == 0.0) + { + std::printf(" no usable co-capture (no endpoint / silent) -- skipping.\n"); + if (com) + { + CoUninitialize(); + } + return 0; + } + check(a.ok, "(a) verifier confidently correlated the two capture paths"); + check(a.rate == mismatched, "(a) recovered the game's true rate (not the device rate)"); - check(fv.ok, "verifier confidently correlated the two capture paths"); - check(fv.rate == game_rate, "verifier recovered the game's true rate (not the device rate)"); + // (b) Layout: render 2ch with distinct per-channel content -- a layout that differs from a + // multichannel device -- and recover channels + bit depth + rate. + std::printf("== (b) layout recovery: 44100 Hz / 2 ch / 32-bit float (distinct channels) ==\n"); + FormatVerification b = run({44100, 2, 32, /*distinct=*/true, /*recover_layout=*/true}, ran); + std::printf(" ok=%d rate=%u ch=%u bits=%u tag=%u score=%.3f\n", b.ok ? 1 : 0, b.rate, b.channels, b.bits, + b.format_tag, b.score); + if (b.ok || b.score > 0.0) + { + check(b.layout_ok, "(b) verifier confidently recovered the layout"); + check(b.rate == 44100, "(b) recovered the true rate"); + check(b.channels == 2, "(b) recovered the true channel count (2, not the device's)"); + check(b.bits == 32 && b.format_tag == 3, "(b) recovered 32-bit float"); + } + else + { + std::printf(" no usable co-capture for (b) -- skipping that scenario.\n"); + } if (com) { CoUninitialize(); } - cleanup(); if (g_failures == 0) { diff --git a/tools/audio_tone/tone_source.hpp b/tools/audio_tone/tone_source.hpp index 2af56bc..2f05b88 100644 --- a/tools/audio_tone/tone_source.hpp +++ b/tools/audio_tone/tone_source.hpp @@ -7,6 +7,7 @@ #include #include +#include #include @@ -87,6 +88,13 @@ public: client_->GetBufferSize(&buffer_frames_); step_ = kTwoPi * freq_hz / static_cast(fmt_.rate); + // Optional: give each channel genuinely different content (a per-channel frequency scale), + // so a downstream test can *recover* the channel count by correlation (identical channels + // are ambiguous: 2ch@R looks like 1ch@2R). Off by default -> the usual single-tone source. + if (const char* d = std::getenv("COOP_TONE_DISTINCT_CH"); d != nullptr && d[0] == '1') + { + distinct_ = true; + } write(buffer_frames_); // pre-roll client_->Start(); return true; @@ -220,13 +228,24 @@ private: } for (unsigned c = 0; c < fmt_.channels; ++c) { + double sc = s; + if (distinct_ && c < 8) + { + // Each channel at its own frequency scale -> genuinely different content. + sc = std::sin(phase_c_[c]) * 0.25; + phase_c_[c] += step_ * (1.0 + 0.37 * static_cast(c)); + if (phase_c_[c] > kTwoPi) + { + phase_c_[c] -= kTwoPi; + } + } if (float_) { - reinterpret_cast(data)[i * fmt_.channels + c] = static_cast(s); + reinterpret_cast(data)[i * fmt_.channels + c] = static_cast(sc); } else { - reinterpret_cast(data)[i * fmt_.channels + c] = static_cast(s * 32767.0); + reinterpret_cast(data)[i * fmt_.channels + c] = static_cast(sc * 32767.0); } } } @@ -243,6 +262,8 @@ private: bool float_ = false; double phase_ = 0.0; double step_ = 0.0; + bool distinct_ = false; // per-channel distinct content (recoverable channel count) + double phase_c_[8] = {}; // per-channel phase when distinct_ }; } // namespace coop::tone