diff --git a/README.md b/README.md index 3f59cf2..359ab51 100644 --- a/README.md +++ b/README.md @@ -228,6 +228,22 @@ ctest --test-dir build -C Debug --output-on-failure reader/writer. Synthesizes a clean tone, a wrong-rate (pitch-shifted) tone, a tone with injected clicks, and one with silence gaps, and asserts each metric matches what was injected (e.g. 44100 played as 48000 → +147 cents). Pure header logic, no device. +- **`audio_correlation_test`** — unit test of the two-path audio-format correlator + ([`common/include/coop/audio_correlate.hpp`](common/include/coop/audio_correlate.hpp)), which + recovers a guessed stream's true sample rate from *ground truth* instead of cadence. Synthesizes + one continuous signal sampled at two rates (the hook's true rate + the device rate, with capture + skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the + true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000 + case the cadence method can misread), and that unrelated signals are *not* confidently matched. + Pure header logic, no device. +- **`audio_verify_test`** — integration test of the host's two-path verifier + ([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches + `coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so + this rate test isn't perturbed by a channel mismatch — that's the next task), injects the hook + late (a guessed stream), and runs the real `verify_stream_format()`: it co-captures the hook + (pre-mix, via the ring's `verify_capture` tap) and a parallel process-loopback (post-mix) of the + same audio and correlates them. Asserts it recovers the game's true rate, not the device guess. + Skips cleanly without an audio endpoint. - **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy (`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill @@ -518,6 +534,24 @@ Non-obvious things that cost time and constrain the design: estimate as explicitly *low-confidence* (shown red). The operator can also re-measure or override the format via a per-stream `AudioRingHeader` op channel; the host rebuilds its render client when `format_generation` bumps, so it takes effect live. +- **Two capture paths beat one guess: correlate the hook against the loopback.** Cadence + measurement *rejects* a bad rate reading but is still a guess from one signal, and it can't recover + channels/bit-depth at all. But during the measurement window the game is still audible, so we have + the *same* audio twice: the **hook** (pre-mix, unknown format) and a **process-loopback** (post-mix, + the *known* device format — it's the hook signal resampled by WASAPI's AUTOCONVERTPCM). Resampling + the hook by each candidate rate and cross-correlating against the loopback pins the true rate from + ground truth: the right rate holds alignment across the whole window (score ≈1.0); a wrong rate + time-warps the hook so a single alignment can't hold and the correlation collapses (≈0). The catch + that makes this need a measurement *tap*: the no-echo path **silences** the game, so a loopback of a + silenced game is silent — the co-capture must happen while the stream is still being measured (not + yet published, so not yet silenced). A host-set `verify_capture` ring flag makes the hook push the + guessed stream's raw pre-mix bytes (no silence) during that window; the host + (`audio_format_verifier`) co-captures both, correlates (`coop/audio_correlate.hpp`), and feeds a + correction into the existing override channel. **Rate vs layout are coupled, though**: correlating + the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step + assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a + *different* channel count / bit depth is the layout step, which tries candidate de-interleavings and + keeps whichever correlates. - **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The mirror re-renders the captured ring to the output device. The original feed loop re-primed (withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free diff --git a/common/include/coop/audio_correlate.hpp b/common/include/coop/audio_correlate.hpp new file mode 100644 index 0000000..61e14b3 --- /dev/null +++ b/common/include/coop/audio_correlate.hpp @@ -0,0 +1,257 @@ +// Recover a pre-existing render stream's true audio format by *correlating* the two capture +// paths, instead of guessing. +// +// When we attach to an already-running game we never saw its IAudioClient::Initialize, so the +// render-hook assumes the device mix format and measures only the sample rate from the render +// cadence -- which can be wrong on a jittery game (intermittent pitch shift). But during the +// measurement window the game is still audible, so we have BOTH signals of the same audio: +// * the render-hook capture -- pre-mix, at the *unknown* format, +// * the process-loopback capture -- post-mix, at the *known* device format. +// The loopback is just the hook signal resampled by WASAPI's AUTOCONVERTPCM from the stream's +// true rate to the device rate. So if we resample the hook stream by a candidate rate up to the +// device rate and it lines up with the loopback over the whole window (no drift), that candidate +// is the truth. A wrong rate time-warps the hook stream, so a single alignment can't hold across +// the window and the correlation collapses. +// +// This header is the pure, headless-testable core (no devices, no WASAPI). The host downmixes the +// two captures to mono float, calls correlate_rate(), and feeds the result into the existing rate +// path (publish / override). audio_correlate_layout.hpp (step b) reuses these helpers to also +// recover channels + bit depth by trying candidate de-interleavings. +#pragma once + +#include +#include +#include +#include + +namespace coop +{ + +// The standard sample rates a shared-mode WASAPI stream realistically uses. Candidates are this +// set; a non-standard true rate is out of scope (and would show as low-confidence either way). +inline const std::vector& standard_audio_rates() +{ + static const std::vector rates = {32000, 44100, 48000, 88200, 96000}; + return rates; +} + +struct RateCorrelation +{ + bool ok = false; // a confident pick was made (winner clears the threshold AND beats the runner-up) + unsigned rate = 0; // best candidate rate (Hz) + double score = 0.0; // alignment score of the winner, in [0,1] (1 = perfect) + double runner_up = 0.0; // score of the second-best candidate (for separation) +}; + +namespace correlate_detail +{ + +// Average interleaved float frames down to a single mono channel. +inline void downmix(const float* interleaved, std::size_t frames, unsigned channels, std::vector& out) +{ + out.resize(frames); + if (channels == 0) + { + channels = 1; + } + for (std::size_t i = 0; i < frames; ++i) + { + float sum = 0.0f; + for (unsigned c = 0; c < channels; ++c) + { + sum += interleaved[i * channels + c]; + } + out[i] = sum / static_cast(channels); + } +} + +// Linear-resample a mono signal from src_rate to dst_rate. +inline void resample_linear(const std::vector& in, unsigned src_rate, unsigned dst_rate, + std::vector& out) +{ + if (src_rate == 0 || dst_rate == 0 || in.empty()) + { + out.clear(); + return; + } + if (src_rate == dst_rate) + { + out = in; + return; + } + const double step = static_cast(src_rate) / static_cast(dst_rate); + const std::size_t out_n = static_cast(static_cast(in.size()) / step); + out.resize(out_n); + for (std::size_t i = 0; i < out_n; ++i) + { + const double pos = static_cast(i) * step; + const std::size_t j = static_cast(pos); + const double frac = pos - static_cast(j); + const float a = in[j]; + const float b = (j + 1 < in.size()) ? in[j + 1] : a; + out[i] = a + static_cast(frac) * (b - a); + } +} + +// Box-decimate a mono signal from `rate` down to ~corr_rate for a cheap, content-preserving +// alignment search (the envelope/content alignment doesn't need full bandwidth). +inline void decimate(const std::vector& in, unsigned rate, unsigned corr_rate, std::vector& out) +{ + if (rate <= corr_rate || in.empty()) + { + out = in; + return; + } + const double factor = static_cast(rate) / static_cast(corr_rate); + const std::size_t out_n = static_cast(static_cast(in.size()) / factor); + out.resize(out_n); + for (std::size_t i = 0; i < out_n; ++i) + { + const std::size_t lo = static_cast(static_cast(i) * factor); + std::size_t hi = static_cast(static_cast(i + 1) * factor); + if (hi <= lo) + { + hi = lo + 1; + } + if (hi > in.size()) + { + hi = in.size(); + } + float sum = 0.0f; + for (std::size_t k = lo; k < hi; ++k) + { + sum += in[k]; + } + out[i] = sum / static_cast(hi - lo); + } +} + +// Zero-mean, unit-norm cross-correlation of a vs b over [start, start+len), with b shifted by lag. +// Returns a value in [-1, 1]; out-of-range samples are skipped. ~0 when the two don't align. +inline double ncc(const std::vector& a, const std::vector& b, long lag, std::size_t start, + std::size_t len) +{ + double sa = 0.0, sb = 0.0; + std::size_t n = 0; + for (std::size_t i = start; i < start + len && i < a.size(); ++i) + { + const long bi = static_cast(i) + lag; + if (bi < 0 || static_cast(bi) >= b.size()) + { + continue; + } + sa += a[i]; + sb += b[bi]; + ++n; + } + if (n < 8) + { + return 0.0; + } + const double ma = sa / static_cast(n); + const double mb = sb / static_cast(n); + double num = 0.0, da = 0.0, db = 0.0; + for (std::size_t i = start; i < start + len && i < a.size(); ++i) + { + const long bi = static_cast(i) + lag; + if (bi < 0 || static_cast(bi) >= b.size()) + { + continue; + } + const double xa = a[i] - ma; + const double xb = b[bi] - mb; + num += xa * xb; + da += xa * xa; + db += xb * xb; + } + if (da < 1e-9 || db < 1e-9) + { + return 0.0; + } + return num / std::sqrt(da * db); +} + +// Alignment score of two same-rate mono signals: find the single best lag over the whole window, +// then require that lag to hold in BOTH an early and a late segment (drift detection). The score +// is the weaker of the two segment correlations, so a rate that only lines up at the start (a +// wrong rate, which time-warps and drifts) scores low while the true rate scores high. +inline double aligned_score(const std::vector& a, const std::vector& b, unsigned rate) +{ + const std::size_t n = a.size() < b.size() ? a.size() : b.size(); + if (n < rate / 10) // need at least ~100 ms of overlap to judge + { + return 0.0; + } + const long max_lag = static_cast(rate / 8); // search +/-125 ms of capture-path latency skew + // Coarse global lag from the middle half of the window. + const std::size_t mid_start = n / 4; + const std::size_t mid_len = n / 2; + double best = -2.0; + long best_lag = 0; + for (long lag = -max_lag; lag <= max_lag; ++lag) + { + const double c = ncc(a, b, lag, mid_start, mid_len); + if (c > best) + { + best = c; + best_lag = lag; + } + } + // Re-evaluate that lag in an early and a late third: the true rate holds; a drifting (wrong) + // rate does not. + const std::size_t third = n / 3; + const double early = ncc(a, b, best_lag, 0, third); + const double late = ncc(a, b, best_lag, 2 * third, third); + const double weaker = early < late ? early : late; + return weaker < 0.0 ? 0.0 : weaker; +} + +} // namespace correlate_detail + +// Determine the hook stream's true sample rate by resampling it by each candidate rate up to the +// known device (loopback) rate and scoring how well it aligns with the loopback across the window. +// hook_mono / loop_mono are mono float (caller downmixes). `min_score` is the absolute alignment +// floor and `separation` the ratio by which the winner must beat the runner-up to be `ok`. +inline RateCorrelation correlate_rate(const std::vector& hook_mono, const std::vector& loop_mono, + unsigned device_rate, const std::vector& candidates, + double min_score = 0.55, double separation = 1.2) +{ + using namespace correlate_detail; + RateCorrelation result; + if (hook_mono.empty() || loop_mono.empty() || device_rate == 0) + { + return result; + } + constexpr unsigned kCorrRate = 8000; // alignment search rate (Nyquist 4 kHz -- plenty for content) + + std::vector loop_ds; + decimate(loop_mono, device_rate, kCorrRate, loop_ds); + + double best = -1.0, second = -1.0; + unsigned best_rate = 0; + std::vector resampled, hook_ds; + for (unsigned cand : candidates) + { + resample_linear(hook_mono, cand, device_rate, resampled); // treat hook as sampled at `cand` + decimate(resampled, device_rate, kCorrRate, hook_ds); + const double s = aligned_score(hook_ds, loop_ds, kCorrRate); + if (s > best) + { + second = best; + best = s; + best_rate = cand; + } + else if (s > second) + { + second = s; + } + } + + result.rate = best_rate; + result.score = best < 0.0 ? 0.0 : best; + result.runner_up = second < 0.0 ? 0.0 : second; + result.ok = result.score >= min_score && (result.runner_up <= 1e-6 || result.score >= result.runner_up * separation); + return result; +} + +} // namespace coop diff --git a/common/include/coop/audio_ring.hpp b/common/include/coop/audio_ring.hpp index 6337f02..177bfa9 100644 --- a/common/include/coop/audio_ring.hpp +++ b/common/include/coop/audio_ring.hpp @@ -87,7 +87,16 @@ struct AudioRingHeader std::uint32_t op_bits; // override: bits per sample std::uint32_t op_format_tag; // override: WAVE_FORMAT_PCM / _IEEE_FLOAT - std::uint8_t reserved[40]; + // Host -> hook: format-verification co-capture. While 1, the hook pushes a still-being-measured + // (guessed) stream's raw pre-mix bytes into the ring WITHOUT silencing the game, so the host can + // capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same audio + // and cross-correlate them to recover the true sample rate (and, in step b, channels/bit-depth) + // from ground truth instead of guessing. Inert (0) by default -- normal capture is unaffected, + // so it never changes the shipping no-echo path. Repurposed from `reserved`, so the layout and + // size are unchanged (old builds saw it as a zero reserved byte). + std::atomic verify_capture; + + std::uint8_t reserved[36]; // std::uint8_t data[capacity] follows immediately in the mapping. }; @@ -131,6 +140,7 @@ inline void audio_ring_init(AudioRingHeader& h, std::uint32_t capacity) h.op_channels = 0; h.op_bits = 0; h.op_format_tag = 0; + h.verify_capture.store(0, std::memory_order_relaxed); std::memset(h.reserved, 0, sizeof(h.reserved)); } diff --git a/hook/src/audio_hook.cpp b/hook/src/audio_hook.cpp index aa2b5c7..6ef9400 100644 --- a/hook/src/audio_hook.cpp +++ b/hook/src/audio_hook.cpp @@ -391,6 +391,30 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_ } } } + // Format-verification co-capture (host-driven, step 2a). While the host has set + // verify_capture and this guessed stream's rate is still being MEASURED (format not yet + // published), push the raw pre-mix bytes to the ring WITHOUT silencing, so the host can + // capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same + // audio and cross-correlate them to recover the true format from ground truth. The game + // stays audible (the host runs loopback during the measurement window anyway), and the + // shipping no-echo capture/silence path above is left completely untouched. + if (num_frames > 0 && (flags & AUDCLNT_BUFFERFLAGS_SILENT) == 0) + { + AudioRingHeader* vring = g_rings[i].load(std::memory_order_acquire); + if (vring != nullptr && vring->verify_capture.load(std::memory_order_relaxed) != 0 && + !audio_ring_format_ready(*vring) && + g_streams[i].assumed_format.load(std::memory_order_relaxed) != 0 && t_gb_client == self && + t_gb_data != nullptr && t_gb_frames == num_frames && + t_gb_epoch == g_hook_epoch.load(std::memory_order_acquire)) + { + const std::uint32_t block = g_streams[i].block_align.load(std::memory_order_relaxed); + if (block != 0) + { + audio_ring_push(*vring, t_gb_data, readable_bytes(t_gb_data, num_frames * block), + num_frames); + } + } + } break; } return g_vh_releasebuffer.original()(self, num_frames, flags); diff --git a/host/CMakeLists.txt b/host/CMakeLists.txt index dc1f126..d26a542 100644 --- a/host/CMakeLists.txt +++ b/host/CMakeLists.txt @@ -19,6 +19,7 @@ add_executable(coop_host WIN32 src/capture/window_capture.cpp src/capture/shared_texture.cpp src/audio/audio_loopback.cpp + src/audio/audio_format_verifier.cpp src/audio/audio_overrides.cpp src/audio/process_loopback_capture.cpp src/vk_layer_setup.cpp diff --git a/host/src/audio/audio_format_verifier.cpp b/host/src/audio/audio_format_verifier.cpp new file mode 100644 index 0000000..5171ffa --- /dev/null +++ b/host/src/audio/audio_format_verifier.cpp @@ -0,0 +1,187 @@ +#include "audio/audio_format_verifier.hpp" + +#include +#include +#include +#include + +#include +#include + +#include "audio/process_loopback_capture.hpp" +#include "coop/audio_correlate.hpp" + +namespace coop +{ +namespace +{ + +// Resolve a (possibly EXTENSIBLE) WAVEFORMATEX to scalar channels / bits / tag. +struct ScalarFormat +{ + unsigned rate = 0; + unsigned channels = 0; + unsigned bits = 0; + unsigned tag = 0; // WAVE_FORMAT_PCM / _IEEE_FLOAT +}; + +ScalarFormat resolve(const WAVEFORMATEX* wfx) +{ + ScalarFormat f; + f.rate = wfx->nSamplesPerSec; + f.channels = wfx->nChannels; + f.bits = wfx->wBitsPerSample; + f.tag = wfx->wFormatTag; + if (wfx->wFormatTag == WAVE_FORMAT_EXTENSIBLE && wfx->cbSize >= 22) + { + const auto* ext = reinterpret_cast(wfx); + if (ext->SubFormat == KSDATAFORMAT_SUBTYPE_IEEE_FLOAT) + { + f.tag = WAVE_FORMAT_IEEE_FLOAT; + } + else if (ext->SubFormat == KSDATAFORMAT_SUBTYPE_PCM) + { + f.tag = WAVE_FORMAT_PCM; + } + } + return f; +} + +// Decode interleaved PCM (`channels`/`bits`/`tag`) into per-channel float samples, then average to +// mono. Handles float32 and 16/32-bit PCM (the formats WASAPI shared-mode streams use). +std::vector to_mono(const std::vector& bytes, const ScalarFormat& fmt) +{ + std::vector mono; + const unsigned ch = fmt.channels == 0 ? 1 : fmt.channels; + const unsigned bps = fmt.bits / 8; + if (bps == 0) + { + return mono; + } + const std::size_t frame = static_cast(ch) * bps; + const std::size_t frames = bytes.size() / frame; + mono.resize(frames); + const bool is_float = fmt.tag == WAVE_FORMAT_IEEE_FLOAT; + for (std::size_t i = 0; i < frames; ++i) + { + double sum = 0.0; + for (unsigned c = 0; c < ch; ++c) + { + const BYTE* p = bytes.data() + i * frame + static_cast(c) * bps; + float s = 0.0f; + if (is_float && fmt.bits == 32) + { + std::memcpy(&s, p, 4); + } + else if (fmt.bits == 16) + { + std::int16_t v; + std::memcpy(&v, p, 2); + s = v / 32768.0f; + } + else if (fmt.bits == 32) + { + std::int32_t v; + std::memcpy(&v, p, 4); + s = static_cast(v / 2147483648.0); + } + sum += s; + } + mono[i] = static_cast(sum / ch); + } + return mono; +} + +void drain_ring(AudioRingHeader& ring, std::vector& scratch) +{ + while (audio_ring_pop(ring, scratch.data(), static_cast(scratch.size())) > 0) + { + } +} + +} // namespace + +FormatVerification verify_stream_format(DWORD pid, AudioRingHeader* ring, unsigned window_ms, bool recover_layout) +{ + (void)recover_layout; // step (b) extends this; step (a) recovers the rate only + FormatVerification result; + if (ring == nullptr) + { + return result; + } + + WAVEFORMATEX* dev_wfx = default_render_format(); + if (dev_wfx == nullptr) + { + return result; + } + const ScalarFormat dev = resolve(dev_wfx); + + // Ask the hook to push the guessed stream's pre-mix bytes (no silence) while it measures, and + // clear any stale ring contents so we only collect this window. + std::vector scratch(64 * 1024); + drain_ring(*ring, scratch); + ring->verify_capture.store(1, std::memory_order_release); + + // Capture the post-mix loopback in parallel (this is the known-format ground truth). + std::vector loop_bytes; + ProcessLoopbackCapture loop; + const std::uint32_t loop_block = dev_wfx->nBlockAlign; + loop.start(pid, dev_wfx, [&](const BYTE* data, std::uint32_t frames, bool silent) { + if (!silent && data != nullptr) + { + loop_bytes.insert(loop_bytes.end(), data, data + static_cast(frames) * loop_block); + } + }); + + // Pull the hook's pre-mix bytes out of the ring across the window. + std::vector hook_bytes; + const DWORD end = GetTickCount() + window_ms; + while (GetTickCount() < end) + { + std::uint32_t n = 0; + while ((n = audio_ring_pop(*ring, scratch.data(), static_cast(scratch.size()))) > 0) + { + hook_bytes.insert(hook_bytes.end(), scratch.data(), scratch.data() + n); + } + Sleep(10); + } + std::uint32_t n = 0; + while ((n = audio_ring_pop(*ring, scratch.data(), static_cast(scratch.size()))) > 0) + { + hook_bytes.insert(hook_bytes.end(), scratch.data(), scratch.data() + n); + } + + loop.stop(); + ring->verify_capture.store(0, std::memory_order_release); + drain_ring(*ring, scratch); // leave the ring clean for the real capture that follows + + // The hook bytes are at the guessed layout = the device channels/bits (the assumption the + // cadence path also makes). Decode both captures with that layout and correlate. + const std::vector hook_mono = to_mono(hook_bytes, dev); + const std::vector loop_mono = to_mono(loop_bytes, dev); + CoTaskMemFree(dev_wfx); + + if (const char* dbg = std::getenv("COOP_VERIFY_DEBUG"); dbg != nullptr && dbg[0] == '1') + { + std::fprintf(stderr, "[verify] dev=%uHz/%uch/%ubit tag=%u hook_frames=%zu loop_frames=%zu\n", dev.rate, + dev.channels, dev.bits, dev.tag, hook_mono.size(), loop_mono.size()); + } + const std::size_t need = dev.rate / 5; // require >= ~200 ms of usable audio on both sides + if (hook_mono.size() < need || loop_mono.size() < need) + { + return result; // not enough non-silent audio captured (game quiet, or stream wasn't a guess) + } + + const RateCorrelation rc = correlate_rate(hook_mono, loop_mono, dev.rate, standard_audio_rates()); + result.ok = rc.ok; + result.rate = rc.rate; + result.score = rc.score; + // Channels/bit-depth stay the device assumption here; step (b) recovers them. + result.channels = dev.channels; + result.bits = dev.bits; + result.format_tag = dev.tag; + return result; +} + +} // namespace coop diff --git a/host/src/audio/audio_format_verifier.hpp b/host/src/audio/audio_format_verifier.hpp new file mode 100644 index 0000000..d38a709 --- /dev/null +++ b/host/src/audio/audio_format_verifier.hpp @@ -0,0 +1,48 @@ +// Two-path audio-format verification: recover a still-being-measured (guessed) render stream's +// true format by *correlating* the two capture paths instead of guessing. +// +// When the host attaches to an already-running game the render-hook never saw the stream's +// Initialize, so it assumes the device mix format and measures only the sample rate from the +// render cadence -- which a jittery game can make wrong (intermittent pitch shift). During the +// measurement window the game is still audible, so we can capture BOTH signals of the same audio: +// * the hook (pre-mix, at the unknown format) via the ring's verify_capture tap, and +// * a WASAPI process-loopback (post-mix, at the KNOWN device format). +// Cross-correlating them (common/include/coop/audio_correlate.hpp) pins the true rate from ground +// truth. The result feeds the existing rate path as an operator-style override. +// +// Step (a) here recovers the sample rate. Step (b) (audio_format_verifier.cpp) extends the same +// co-capture to recover channels + bit depth by trying candidate de-interleavings. +#pragma once + +#include + +#include + +#include "coop/audio_ring.hpp" + +namespace coop +{ + +struct FormatVerification +{ + bool ok = false; // a confident rate correlation was found + unsigned rate = 0; // recovered true sample rate (Hz) + double score = 0.0; // correlation score of the winning rate, [0,1] + + bool layout_ok = false; // a confident channels/bit-depth correlation was found (step b) + unsigned channels = 0; // recovered channel count + unsigned bits = 0; // recovered bits per sample + unsigned format_tag = 0; // recovered WAVE_FORMAT_PCM / _IEEE_FLOAT +}; + +// One-shot: co-capture the hook (pre-mix, via the ring's verify_capture tap) and a parallel +// process-loopback (post-mix, at the device format) of `pid`'s audio for ~window_ms, then +// cross-correlate to recover the stream's true sample rate (and, with recover_layout, its channels +// + bit depth). Returns ok=false (a no-op for the caller) when the stream isn't a guess, the game +// is silent, or the two captures don't correlate. Requires a COM-initialized thread. Toggles +// ring->verify_capture for the duration and drains the ring afterwards so normal capture starts +// clean. +FormatVerification verify_stream_format(DWORD pid, AudioRingHeader* ring, unsigned window_ms = 800, + bool recover_layout = false); + +} // namespace coop diff --git a/host/src/audio/audio_loopback.cpp b/host/src/audio/audio_loopback.cpp index e96b7e4..5c8129e 100644 --- a/host/src/audio/audio_loopback.cpp +++ b/host/src/audio/audio_loopback.cpp @@ -9,6 +9,7 @@ #include #include +#include "audio/audio_format_verifier.hpp" #include "audio/audio_mix.hpp" #include "audio/process_loopback_capture.hpp" #include "audio/render_pacer.hpp" @@ -278,6 +279,7 @@ void AudioMirror::thread_main(DWORD pid) // The guessed-rate path takes a few seconds to reach consensus; loopback covers // that gap and the promote hands off seamlessly. constexpr DWORD kHookWaitMs = 1200; + bool format_verified = false; // run the two-path correlation verify/correct at most once for (;;) { if (stop_requested()) @@ -312,6 +314,23 @@ void AudioMirror::thread_main(DWORD pid) set_fallback_reason( "Render-hook hasn't published a format yet; using loopback (echo) -- will switch to " "hooked automatically once it does."); + // No format after the wait -> a guessed late-attach stream is being *measured*. Once, + // while it measures and the game is still audible, run the two-path correlation (hook + // pre-mix vs a parallel loopback post-mix of the same audio) to recover the true rate + // and correct it through the existing override channel -- this hardens the cadence + // method's intermittent pitch-shift. It's hidden inside the measurement gap loopback + // already covers, so exact streams (format published immediately) never pay for it. + if (!format_verified) + { + format_verified = true; + const FormatVerification fv = verify_stream_format(pid, rings[0]); + if (fv.ok) + { + set_status("Verified render-hook sample rate by correlation."); + audio_ring_post_op(*rings[0], AudioRingOp_Override, fv.rate, fv.channels, fv.bits, + fv.format_tag); + } + } } enable_capture(rings, false); // game audible locally so loopback can capture it diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 8841bfd..df992c4 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -44,6 +44,14 @@ add_executable(tone_analysis_test tone_analysis_test.cpp) target_link_libraries(tone_analysis_test PRIVATE coop_common) add_test(NAME tone_analysis_test COMMAND tone_analysis_test) +# Unit test for the two-path audio-format correlator (common/include/coop/audio_correlate.hpp). +# Synthesizes one continuous signal sampled at two rates (the hook's true rate + the device rate, +# with capture skew + noise) and asserts correlate_rate() recovers the true rate -- including the +# 44100-vs-48000 case the cadence method can misread. Pure header logic, no device. +add_executable(audio_correlation_test audio_correlation_test.cpp) +target_link_libraries(audio_correlation_test PRIVATE coop_common) +add_test(NAME audio_correlation_test COMMAND audio_correlation_test) + # Unit test for the audio-mirror render pacing policy (host/src/audio/render_pacer.hpp). # Simulates a producer/consumer device timeline and asserts the shipping RenderPacer rides # through producer jitter that makes the old re-prime-on-partial-fill policy glitch @@ -170,6 +178,20 @@ target_link_libraries(mock_game_test PRIVATE coop_common d3d11 dxgi) add_dependencies(mock_game_test coop_mock_game coop_hook) add_test(NAME mock_game_test COMMAND mock_game_test) +# Integration test for the two-path audio-format verifier: launches coop_mock_game rendering a +# NON-device rate (44100 on a 48000 endpoint), injects coop_hook.dll late (guessed stream), and +# runs the real verify_stream_format() -- co-capturing the hook (pre-mix) + a parallel loopback +# (post-mix) and correlating to recover the true rate. Skips cleanly without an audio endpoint. +add_executable(audio_verify_test + audio_verify_test.cpp + ${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp + ${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp) +target_include_directories(audio_verify_test PRIVATE ${CMAKE_SOURCE_DIR}/host/src) +target_compile_definitions(audio_verify_test PRIVATE NTDDI_VERSION=0x0A00000B) +target_link_libraries(audio_verify_test PRIVATE coop_common ole32 mmdevapi) +add_dependencies(audio_verify_test coop_mock_game coop_hook) +add_test(NAME audio_verify_test COMMAND audio_verify_test) + # In-process self-test for the OpenGL capture path. Reuses the shipping # opengl_hook.cpp and drives a real OpenGL context in the same process, so it # exercises the SwapBuffers hook, the glReadPixels readback, and the upload into @@ -201,6 +223,7 @@ add_executable(ui_fit_test ${CMAKE_SOURCE_DIR}/host/src/audio_panel.cpp ${CMAKE_SOURCE_DIR}/host/src/controllers_panel.cpp ${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp + ${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp ${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp ${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp ${CMAKE_SOURCE_DIR}/host/src/ui/app_chrome.cpp) @@ -224,6 +247,7 @@ coop_output_subdir(tests mkb_map_test audio_mix_test tone_analysis_test + audio_correlation_test render_pacer_test rate_estimator_test audio_overrides_test @@ -234,4 +258,5 @@ coop_output_subdir(tests dx12_present_hook_test opengl_hook_test mock_game_test + audio_verify_test ui_fit_test) diff --git a/tests/audio_correlation_test.cpp b/tests/audio_correlation_test.cpp new file mode 100644 index 0000000..663831f --- /dev/null +++ b/tests/audio_correlation_test.cpp @@ -0,0 +1,121 @@ +// Unit test for the two-path audio-format correlator (common/include/coop/audio_correlate.hpp). +// +// Models the real situation: the same game audio is captured twice -- by the render-hook at the +// stream's true (unknown) rate, and by process-loopback at the known device rate (the hook signal +// resampled by WASAPI's AUTOCONVERTPCM). We synthesize one continuous signal and sample it at both +// rates (plus a capture-latency skew and a little noise), then assert correlate_rate() recovers the +// true rate -- including the hard 44100-vs-48000 case the cadence method can misread. Pure header +// logic, no device. +#include +#include +#include +#include +#include + +#include "coop/audio_correlate.hpp" + +using namespace coop; + +namespace +{ +int g_failures = 0; +void check(bool ok, const char* what) +{ + std::printf("%s %s\n", ok ? " ok:" : "FAIL:", what); + if (!ok) + { + ++g_failures; + } +} + +constexpr double kPi = 3.14159265358979323846; + +// A non-periodic, correlation-friendly continuous signal s(t): a couple of incommensurate tones +// plus a slow chirp, so cross-correlation has a single sharp peak (unlike a pure sine). +double source(double t) +{ + const double chirp = std::sin(2.0 * kPi * (300.0 * t + 140.0 * t * t)); + return 0.5 * std::sin(2.0 * kPi * 221.0 * t) + 0.28 * std::sin(2.0 * kPi * 437.0 * t + 0.6) + + 0.22 * chirp; +} + +// Sample s(t) at `rate` for `seconds`, starting at t0 (capture-latency skew), optionally adding +// white noise of amplitude `noise` (the post-mix path is not a bit-identical copy). +std::vector capture(unsigned rate, double seconds, double t0, double noise, std::uint32_t seed) +{ + const std::size_t n = static_cast(rate * seconds); + std::vector out(n); + std::mt19937 rng(seed); + std::uniform_real_distribution jitter(-1.0f, 1.0f); + for (std::size_t i = 0; i < n; ++i) + { + const double t = t0 + static_cast(i) / static_cast(rate); + out[i] = static_cast(source(t)) + static_cast(noise) * jitter(rng); + } + return out; +} + +// One scenario: true hook rate `true_rate` mixed to `device_rate`. Assert the correlator picks +// true_rate confidently and that the runner-up is clearly behind. +void test_case(unsigned true_rate, unsigned device_rate, const char* label) +{ + std::printf("== %s (true %u Hz -> device %u Hz) ==\n", label, true_rate, device_rate); + // The hook captures at the true rate; the loopback captures the same signal at the device + // rate, started ~22 ms later (capture skew) with a little measurement noise. + const std::vector hook = capture(true_rate, 0.55, 0.0, 0.0, 1); + const std::vector loop = capture(device_rate, 0.55, 0.022, 0.02, 7); + + const RateCorrelation r = correlate_rate(hook, loop, device_rate, standard_audio_rates()); + std::printf(" picked %u Hz score=%.3f runner_up=%.3f ok=%d\n", r.rate, r.score, r.runner_up, + r.ok ? 1 : 0); + check(r.rate == true_rate, "correlator picked the true rate"); + check(r.ok, "pick is confident (clears threshold + beats runner-up)"); + check(r.score > r.runner_up, "winner scores above the runner-up"); +} +} // namespace + +int main() +{ + // The headline case: Godot/Brotato render 44100 while the endpoint mixes 48000 -- the cadence + // method can misread this, the correlator must not. + test_case(44100, 48000, "godot/brotato case"); + test_case(48000, 48000, "rate matches device"); + test_case(96000, 48000, "high-rate stream"); + test_case(32000, 44100, "low-rate stream on a 44100 endpoint"); + test_case(48000, 44100, "48000 stream on a 44100 endpoint"); + + // Downmix sanity: a stereo interleaved buffer collapses to the same mono the scalar path uses. + { + std::printf("== downmix stereo -> mono ==\n"); + std::vector stereo = {1.0f, 3.0f, 2.0f, 4.0f, -1.0f, 1.0f}; + std::vector mono; + correlate_detail::downmix(stereo.data(), 3, 2, mono); + check(mono.size() == 3 && std::fabs(mono[0] - 2.0f) < 1e-6 && std::fabs(mono[1] - 3.0f) < 1e-6 && + std::fabs(mono[2] - 0.0f) < 1e-6, + "stereo frames average to mono"); + } + + // A pure guess with no shared signal must NOT be reported confident (loopback is unrelated noise). + { + std::printf("== unrelated signals are not confidently matched ==\n"); + const std::vector hook = capture(44100, 0.5, 0.0, 0.0, 1); + std::vector noise(static_cast(48000 * 0.5)); + std::mt19937 rng(99); + std::uniform_real_distribution d(-1.0f, 1.0f); + for (float& x : noise) + { + x = d(rng); + } + const RateCorrelation r = correlate_rate(hook, noise, 48000, standard_audio_rates()); + std::printf(" picked %u Hz score=%.3f ok=%d\n", r.rate, r.score, r.ok ? 1 : 0); + check(!r.ok, "unrelated loopback is not a confident match"); + } + + if (g_failures == 0) + { + std::printf("PASS audio_correlation_test\n"); + return 0; + } + std::printf("FAILED audio_correlation_test (%d)\n", g_failures); + return 1; +} diff --git a/tests/audio_verify_test.cpp b/tests/audio_verify_test.cpp new file mode 100644 index 0000000..8427590 --- /dev/null +++ b/tests/audio_verify_test.cpp @@ -0,0 +1,233 @@ +// Integration test for the two-path audio-format verifier (host/src/audio/audio_format_verifier). +// +// Launches coop_mock_game rendering a tone at a NON-device rate (44100 on a typical 48000 endpoint, +// the Godot/Brotato case), injects coop_hook.dll late (so the stream is a *guess*), then runs the +// real verify_stream_format(): it co-captures the hook (pre-mix, via the ring's verify tap) and a +// parallel process-loopback (post-mix, device format) and cross-correlates them. Asserts it +// recovers the true 44100 Hz rate -- the cadence method's hard case. Skips cleanly without an audio +// endpoint / if Vulkan-free... (only needs WASAPI + a D3D11-capable mock, which the mock always is). +#include +#include +#include + +#include + +#include +#include + +#include + +#include "audio/audio_format_verifier.hpp" +#include "audio/process_loopback_capture.hpp" // default_render_format +#include "coop/audio_ring.hpp" +#include "coop/protocol.hpp" +#include "coop/shared_memory.hpp" +#include "coop/tool_paths.hpp" + +using namespace coop; + +namespace +{ +int g_failures = 0; +void check(bool ok, const char* what) +{ + std::printf("%s %s\n", ok ? " ok:" : "FAIL:", what); + if (!ok) + { + ++g_failures; + } +} + +void kill_stray_mock_games() +{ + HANDLE snap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0); + if (snap == INVALID_HANDLE_VALUE) + { + return; + } + PROCESSENTRY32W pe{}; + pe.dwSize = sizeof(pe); + for (BOOL ok = Process32FirstW(snap, &pe); ok; ok = Process32NextW(snap, &pe)) + { + if (_wcsicmp(pe.szExeFile, L"coop_mock_game.exe") == 0) + { + if (HANDLE h = OpenProcess(PROCESS_TERMINATE, FALSE, pe.th32ProcessID)) + { + TerminateProcess(h, 0); + CloseHandle(h); + } + } + } + CloseHandle(snap); +} + +bool inject(unsigned long pid) +{ + const std::wstring dll = deployed_artifact_path(L"coop_hook.dll"); + if (GetFileAttributesW(dll.c_str()) == INVALID_FILE_ATTRIBUTES) + { + return false; + } + const DWORD access = PROCESS_CREATE_THREAD | PROCESS_QUERY_INFORMATION | PROCESS_VM_OPERATION | + PROCESS_VM_WRITE | PROCESS_VM_READ; + HANDLE process = OpenProcess(access, FALSE, pid); + if (process == nullptr) + { + return false; + } + const SIZE_T bytes = (dll.size() + 1) * sizeof(wchar_t); + void* remote = VirtualAllocEx(process, nullptr, bytes, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE); + bool ok = false; + if (remote != nullptr && WriteProcessMemory(process, remote, dll.c_str(), bytes, nullptr)) + { + auto load = reinterpret_cast( + GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "LoadLibraryW")); + if (HANDLE th = CreateRemoteThread(process, nullptr, 0, load, remote, 0, nullptr)) + { + WaitForSingleObject(th, INFINITE); + DWORD code = 0; + GetExitCodeThread(th, &code); + CloseHandle(th); + ok = code != 0; + } + } + if (remote != nullptr) + { + VirtualFreeEx(process, remote, 0, MEM_RELEASE); + } + CloseHandle(process); + return ok; +} + +bool inject_retry(unsigned long pid) +{ + for (int i = 0; i < 4; ++i) + { + if (inject(pid)) + { + return true; + } + Sleep(300); + } + return false; +} +} // namespace + +int main() +{ + kill_stray_mock_games(); + + const bool com = SUCCEEDED(CoInitializeEx(nullptr, COINIT_MULTITHREADED)); + + // Render the mock at the device's CHANNEL count (so this step-(a) rate test isn't perturbed by + // a channel mismatch -- that's step (b)'s job) but at a DIFFERENT standard rate than the device, + // so the verifier has a real rate to recover. Default to 48000/2ch if we can't read the device. + unsigned dev_rate = 48000, dev_channels = 2; + if (WAVEFORMATEX* dev = default_render_format()) + { + dev_rate = dev->nSamplesPerSec; + dev_channels = dev->nChannels; + CoTaskMemFree(dev); + } + const unsigned game_rate = (dev_rate == 44100) ? 48000u : 44100u; // guarantee a rate mismatch + std::printf(" device %u Hz / %u ch -> rendering the mock at %u Hz / %u ch (rate mismatch)\n", dev_rate, + dev_channels, game_rate, dev_channels); + + const std::wstring exe = exe_directory() + L"coop_mock_game.exe"; + std::wstring cmd = L"\"" + exe + L"\" dx11 30 " + std::to_wstring(game_rate) + L" " + + std::to_wstring(dev_channels) + L" 32 float"; + STARTUPINFOW si{}; + si.cb = sizeof(si); + PROCESS_INFORMATION pi{}; + if (!CreateProcessW(exe.c_str(), cmd.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, &si, &pi)) + { + std::printf("Could not launch coop_mock_game -- skipping audio_verify_test.\n"); + if (com) + { + CoUninitialize(); + } + return 0; + } + auto cleanup = [&] { + TerminateProcess(pi.hProcess, 0); + WaitForSingleObject(pi.hProcess, 2000); + CloseHandle(pi.hThread); + CloseHandle(pi.hProcess); + kill_stray_mock_games(); + }; + Sleep(800); // window + audio client up + + // IPC + the primary audio ring the hook produces into. + SharedMemory shm; + if (!shm.create(shared_memory_name(pi.dwProcessId), sizeof(SharedBlock))) + { + std::printf("Could not create IPC block -- skipping.\n"); + cleanup(); + return 0; + } + auto* block = shm.as(); + block->version = kProtocolVersion; + block->pad_count = 0; + block->sequence.store(0, std::memory_order_relaxed); + // Only the audio subsystem. + for (std::uint32_t s = 0; s < HookSubsys_Count; ++s) + { + const bool off = s != HookSubsys_Audio; + block->control.subsystem_disabled[s].store(off ? 1u : 0u, std::memory_order_release); + } + block->magic = kProtocolMagic; + + SharedMemory ring_shm; + if (!ring_shm.create(audio_ring_name(pi.dwProcessId), audio_ring_total_size(kAudioRingCapacity))) + { + std::printf("Could not create audio ring -- skipping.\n"); + cleanup(); + return 0; + } + auto* ring = ring_shm.as(); + audio_ring_init(*ring, kAudioRingCapacity); + // Leave capture_enabled = 0: we want the stream audible (so loopback hears it) and still being + // MEASURED (so the verify tap fires), exactly the window verify_stream_format targets. + + if (!inject_retry(pi.dwProcessId)) + { + std::printf("Could not inject coop_hook.dll -- skipping.\n"); + cleanup(); + return 0; + } + Sleep(500); // let the hook attach + the pre-existing render client register as a guess + + // Run the real verifier: co-capture hook (pre-mix) + loopback (post-mix) and correlate. + const FormatVerification fv = verify_stream_format(pi.dwProcessId, ring, /*window_ms=*/1400); + std::printf(" verify: ok=%d rate=%u score=%.3f\n", fv.ok ? 1 : 0, fv.rate, fv.score); + + if (!fv.ok && fv.rate == 0 && fv.score == 0.0) + { + // No audio endpoint, or no usable audio captured (e.g. the mock's WASAPI client never + // started on this machine) -> treat as a skip rather than a failure. + std::printf(" no usable co-capture (no endpoint / silent) -- skipping audio_verify_test.\n"); + if (com) + { + CoUninitialize(); + } + cleanup(); + return 0; + } + + check(fv.ok, "verifier confidently correlated the two capture paths"); + check(fv.rate == game_rate, "verifier recovered the game's true rate (not the device rate)"); + + if (com) + { + CoUninitialize(); + } + cleanup(); + + if (g_failures == 0) + { + std::printf("PASS audio_verify_test\n"); + return 0; + } + std::printf("FAILED audio_verify_test (%d)\n", g_failures); + return 1; +} diff --git a/tools/audio_validate/CMakeLists.txt b/tools/audio_validate/CMakeLists.txt index cb6733a..423ba89 100644 --- a/tools/audio_validate/CMakeLists.txt +++ b/tools/audio_validate/CMakeLists.txt @@ -8,6 +8,7 @@ add_executable(coop_audio_validate main.cpp ${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp + ${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp ${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp ${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp)