Recover a guessed audio stream's rate by correlating hook vs loopback (step a)
When the host attaches to an already-running game it never saw the stream's Initialize, so the render-hook assumes the device mix format and measures only the sample rate from the render cadence -- which a jittery game can make wrong (intermittent pitch shift). But during the measurement window the game is still audible, so we have the same audio twice: the hook (pre-mix, unknown format) and a process-loopback (post-mix, the known device format). Cross-correlating them pins the true rate from ground truth. - common/include/coop/audio_correlate.hpp: the pure correlator. Resample the hook by each candidate standard rate up to the device rate and score how well it aligns with the loopback across the window (drift-detecting). audio_correlation_test recovers every rate (score ~1.0 vs ~0.01 for wrong ones), incl. 44100-vs-48000, and rejects unrelated signals. - Hook measurement tap: a host-set verify_capture ring flag makes the hook push a still-being-measured (guessed) stream's raw pre-mix bytes WITHOUT silencing, so the host can co-capture both signals (a silenced game's loopback is silent). Inert by default -- the shipping no-echo path is untouched. - host/src/audio/audio_format_verifier: co-captures hook + loopback and correlates, feeding a correction into the existing override channel. Wired into AudioMirror's measurement window (hidden in the gap loopback already covers, so exact streams pay nothing). audio_verify_test drives it end-to-end against coop_mock_game. Rate vs layout are coupled (correlating the waveform needs the right channel de-interleaving), so this step assumes the hook layout matches the device (the common stereo-on-stereo case); recovering a different channel count / bit depth is step b. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
34
README.md
34
README.md
@@ -228,6 +228,22 @@ ctest --test-dir build -C Debug --output-on-failure
|
|||||||
reader/writer. Synthesizes a clean tone, a wrong-rate (pitch-shifted) tone, a tone with
|
reader/writer. Synthesizes a clean tone, a wrong-rate (pitch-shifted) tone, a tone with
|
||||||
injected clicks, and one with silence gaps, and asserts each metric matches what was injected
|
injected clicks, and one with silence gaps, and asserts each metric matches what was injected
|
||||||
(e.g. 44100 played as 48000 → +147 cents). Pure header logic, no device.
|
(e.g. 44100 played as 48000 → +147 cents). Pure header logic, no device.
|
||||||
|
- **`audio_correlation_test`** — unit test of the two-path audio-format correlator
|
||||||
|
([`common/include/coop/audio_correlate.hpp`](common/include/coop/audio_correlate.hpp)), which
|
||||||
|
recovers a guessed stream's true sample rate from *ground truth* instead of cadence. Synthesizes
|
||||||
|
one continuous signal sampled at two rates (the hook's true rate + the device rate, with capture
|
||||||
|
skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the
|
||||||
|
true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000
|
||||||
|
case the cadence method can misread), and that unrelated signals are *not* confidently matched.
|
||||||
|
Pure header logic, no device.
|
||||||
|
- **`audio_verify_test`** — integration test of the host's two-path verifier
|
||||||
|
([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches
|
||||||
|
`coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so
|
||||||
|
this rate test isn't perturbed by a channel mismatch — that's the next task), injects the hook
|
||||||
|
late (a guessed stream), and runs the real `verify_stream_format()`: it co-captures the hook
|
||||||
|
(pre-mix, via the ring's `verify_capture` tap) and a parallel process-loopback (post-mix) of the
|
||||||
|
same audio and correlates them. Asserts it recovers the game's true rate, not the device guess.
|
||||||
|
Skips cleanly without an audio endpoint.
|
||||||
- **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy
|
- **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy
|
||||||
(`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts
|
(`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts
|
||||||
the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill
|
the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill
|
||||||
@@ -518,6 +534,24 @@ Non-obvious things that cost time and constrain the design:
|
|||||||
estimate as explicitly *low-confidence* (shown red). The operator can also re-measure
|
estimate as explicitly *low-confidence* (shown red). The operator can also re-measure
|
||||||
or override the format via a per-stream `AudioRingHeader` op channel; the host rebuilds
|
or override the format via a per-stream `AudioRingHeader` op channel; the host rebuilds
|
||||||
its render client when `format_generation` bumps, so it takes effect live.
|
its render client when `format_generation` bumps, so it takes effect live.
|
||||||
|
- **Two capture paths beat one guess: correlate the hook against the loopback.** Cadence
|
||||||
|
measurement *rejects* a bad rate reading but is still a guess from one signal, and it can't recover
|
||||||
|
channels/bit-depth at all. But during the measurement window the game is still audible, so we have
|
||||||
|
the *same* audio twice: the **hook** (pre-mix, unknown format) and a **process-loopback** (post-mix,
|
||||||
|
the *known* device format — it's the hook signal resampled by WASAPI's AUTOCONVERTPCM). Resampling
|
||||||
|
the hook by each candidate rate and cross-correlating against the loopback pins the true rate from
|
||||||
|
ground truth: the right rate holds alignment across the whole window (score ≈1.0); a wrong rate
|
||||||
|
time-warps the hook so a single alignment can't hold and the correlation collapses (≈0). The catch
|
||||||
|
that makes this need a measurement *tap*: the no-echo path **silences** the game, so a loopback of a
|
||||||
|
silenced game is silent — the co-capture must happen while the stream is still being measured (not
|
||||||
|
yet published, so not yet silenced). A host-set `verify_capture` ring flag makes the hook push the
|
||||||
|
guessed stream's raw pre-mix bytes (no silence) during that window; the host
|
||||||
|
(`audio_format_verifier`) co-captures both, correlates (`coop/audio_correlate.hpp`), and feeds a
|
||||||
|
correction into the existing override channel. **Rate vs layout are coupled, though**: correlating
|
||||||
|
the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step
|
||||||
|
assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a
|
||||||
|
*different* channel count / bit depth is the layout step, which tries candidate de-interleavings and
|
||||||
|
keeps whichever correlates.
|
||||||
- **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The
|
- **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The
|
||||||
mirror re-renders the captured ring to the output device. The original feed loop re-primed
|
mirror re-renders the captured ring to the output device. The original feed loop re-primed
|
||||||
(withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free
|
(withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free
|
||||||
|
|||||||
257
common/include/coop/audio_correlate.hpp
Normal file
257
common/include/coop/audio_correlate.hpp
Normal file
@@ -0,0 +1,257 @@
|
|||||||
|
// Recover a pre-existing render stream's true audio format by *correlating* the two capture
|
||||||
|
// paths, instead of guessing.
|
||||||
|
//
|
||||||
|
// When we attach to an already-running game we never saw its IAudioClient::Initialize, so the
|
||||||
|
// render-hook assumes the device mix format and measures only the sample rate from the render
|
||||||
|
// cadence -- which can be wrong on a jittery game (intermittent pitch shift). But during the
|
||||||
|
// measurement window the game is still audible, so we have BOTH signals of the same audio:
|
||||||
|
// * the render-hook capture -- pre-mix, at the *unknown* format,
|
||||||
|
// * the process-loopback capture -- post-mix, at the *known* device format.
|
||||||
|
// The loopback is just the hook signal resampled by WASAPI's AUTOCONVERTPCM from the stream's
|
||||||
|
// true rate to the device rate. So if we resample the hook stream by a candidate rate up to the
|
||||||
|
// device rate and it lines up with the loopback over the whole window (no drift), that candidate
|
||||||
|
// is the truth. A wrong rate time-warps the hook stream, so a single alignment can't hold across
|
||||||
|
// the window and the correlation collapses.
|
||||||
|
//
|
||||||
|
// This header is the pure, headless-testable core (no devices, no WASAPI). The host downmixes the
|
||||||
|
// two captures to mono float, calls correlate_rate(), and feeds the result into the existing rate
|
||||||
|
// path (publish / override). audio_correlate_layout.hpp (step b) reuses these helpers to also
|
||||||
|
// recover channels + bit depth by trying candidate de-interleavings.
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <cmath>
|
||||||
|
#include <cstddef>
|
||||||
|
#include <cstdint>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
namespace coop
|
||||||
|
{
|
||||||
|
|
||||||
|
// The standard sample rates a shared-mode WASAPI stream realistically uses. Candidates are this
|
||||||
|
// set; a non-standard true rate is out of scope (and would show as low-confidence either way).
|
||||||
|
inline const std::vector<unsigned>& standard_audio_rates()
|
||||||
|
{
|
||||||
|
static const std::vector<unsigned> rates = {32000, 44100, 48000, 88200, 96000};
|
||||||
|
return rates;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RateCorrelation
|
||||||
|
{
|
||||||
|
bool ok = false; // a confident pick was made (winner clears the threshold AND beats the runner-up)
|
||||||
|
unsigned rate = 0; // best candidate rate (Hz)
|
||||||
|
double score = 0.0; // alignment score of the winner, in [0,1] (1 = perfect)
|
||||||
|
double runner_up = 0.0; // score of the second-best candidate (for separation)
|
||||||
|
};
|
||||||
|
|
||||||
|
namespace correlate_detail
|
||||||
|
{
|
||||||
|
|
||||||
|
// Average interleaved float frames down to a single mono channel.
|
||||||
|
inline void downmix(const float* interleaved, std::size_t frames, unsigned channels, std::vector<float>& out)
|
||||||
|
{
|
||||||
|
out.resize(frames);
|
||||||
|
if (channels == 0)
|
||||||
|
{
|
||||||
|
channels = 1;
|
||||||
|
}
|
||||||
|
for (std::size_t i = 0; i < frames; ++i)
|
||||||
|
{
|
||||||
|
float sum = 0.0f;
|
||||||
|
for (unsigned c = 0; c < channels; ++c)
|
||||||
|
{
|
||||||
|
sum += interleaved[i * channels + c];
|
||||||
|
}
|
||||||
|
out[i] = sum / static_cast<float>(channels);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Linear-resample a mono signal from src_rate to dst_rate.
|
||||||
|
inline void resample_linear(const std::vector<float>& in, unsigned src_rate, unsigned dst_rate,
|
||||||
|
std::vector<float>& out)
|
||||||
|
{
|
||||||
|
if (src_rate == 0 || dst_rate == 0 || in.empty())
|
||||||
|
{
|
||||||
|
out.clear();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (src_rate == dst_rate)
|
||||||
|
{
|
||||||
|
out = in;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const double step = static_cast<double>(src_rate) / static_cast<double>(dst_rate);
|
||||||
|
const std::size_t out_n = static_cast<std::size_t>(static_cast<double>(in.size()) / step);
|
||||||
|
out.resize(out_n);
|
||||||
|
for (std::size_t i = 0; i < out_n; ++i)
|
||||||
|
{
|
||||||
|
const double pos = static_cast<double>(i) * step;
|
||||||
|
const std::size_t j = static_cast<std::size_t>(pos);
|
||||||
|
const double frac = pos - static_cast<double>(j);
|
||||||
|
const float a = in[j];
|
||||||
|
const float b = (j + 1 < in.size()) ? in[j + 1] : a;
|
||||||
|
out[i] = a + static_cast<float>(frac) * (b - a);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Box-decimate a mono signal from `rate` down to ~corr_rate for a cheap, content-preserving
|
||||||
|
// alignment search (the envelope/content alignment doesn't need full bandwidth).
|
||||||
|
inline void decimate(const std::vector<float>& in, unsigned rate, unsigned corr_rate, std::vector<float>& out)
|
||||||
|
{
|
||||||
|
if (rate <= corr_rate || in.empty())
|
||||||
|
{
|
||||||
|
out = in;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const double factor = static_cast<double>(rate) / static_cast<double>(corr_rate);
|
||||||
|
const std::size_t out_n = static_cast<std::size_t>(static_cast<double>(in.size()) / factor);
|
||||||
|
out.resize(out_n);
|
||||||
|
for (std::size_t i = 0; i < out_n; ++i)
|
||||||
|
{
|
||||||
|
const std::size_t lo = static_cast<std::size_t>(static_cast<double>(i) * factor);
|
||||||
|
std::size_t hi = static_cast<std::size_t>(static_cast<double>(i + 1) * factor);
|
||||||
|
if (hi <= lo)
|
||||||
|
{
|
||||||
|
hi = lo + 1;
|
||||||
|
}
|
||||||
|
if (hi > in.size())
|
||||||
|
{
|
||||||
|
hi = in.size();
|
||||||
|
}
|
||||||
|
float sum = 0.0f;
|
||||||
|
for (std::size_t k = lo; k < hi; ++k)
|
||||||
|
{
|
||||||
|
sum += in[k];
|
||||||
|
}
|
||||||
|
out[i] = sum / static_cast<float>(hi - lo);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Zero-mean, unit-norm cross-correlation of a vs b over [start, start+len), with b shifted by lag.
|
||||||
|
// Returns a value in [-1, 1]; out-of-range samples are skipped. ~0 when the two don't align.
|
||||||
|
inline double ncc(const std::vector<float>& a, const std::vector<float>& b, long lag, std::size_t start,
|
||||||
|
std::size_t len)
|
||||||
|
{
|
||||||
|
double sa = 0.0, sb = 0.0;
|
||||||
|
std::size_t n = 0;
|
||||||
|
for (std::size_t i = start; i < start + len && i < a.size(); ++i)
|
||||||
|
{
|
||||||
|
const long bi = static_cast<long>(i) + lag;
|
||||||
|
if (bi < 0 || static_cast<std::size_t>(bi) >= b.size())
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
sa += a[i];
|
||||||
|
sb += b[bi];
|
||||||
|
++n;
|
||||||
|
}
|
||||||
|
if (n < 8)
|
||||||
|
{
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
const double ma = sa / static_cast<double>(n);
|
||||||
|
const double mb = sb / static_cast<double>(n);
|
||||||
|
double num = 0.0, da = 0.0, db = 0.0;
|
||||||
|
for (std::size_t i = start; i < start + len && i < a.size(); ++i)
|
||||||
|
{
|
||||||
|
const long bi = static_cast<long>(i) + lag;
|
||||||
|
if (bi < 0 || static_cast<std::size_t>(bi) >= b.size())
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const double xa = a[i] - ma;
|
||||||
|
const double xb = b[bi] - mb;
|
||||||
|
num += xa * xb;
|
||||||
|
da += xa * xa;
|
||||||
|
db += xb * xb;
|
||||||
|
}
|
||||||
|
if (da < 1e-9 || db < 1e-9)
|
||||||
|
{
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
return num / std::sqrt(da * db);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Alignment score of two same-rate mono signals: find the single best lag over the whole window,
|
||||||
|
// then require that lag to hold in BOTH an early and a late segment (drift detection). The score
|
||||||
|
// is the weaker of the two segment correlations, so a rate that only lines up at the start (a
|
||||||
|
// wrong rate, which time-warps and drifts) scores low while the true rate scores high.
|
||||||
|
inline double aligned_score(const std::vector<float>& a, const std::vector<float>& b, unsigned rate)
|
||||||
|
{
|
||||||
|
const std::size_t n = a.size() < b.size() ? a.size() : b.size();
|
||||||
|
if (n < rate / 10) // need at least ~100 ms of overlap to judge
|
||||||
|
{
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
const long max_lag = static_cast<long>(rate / 8); // search +/-125 ms of capture-path latency skew
|
||||||
|
// Coarse global lag from the middle half of the window.
|
||||||
|
const std::size_t mid_start = n / 4;
|
||||||
|
const std::size_t mid_len = n / 2;
|
||||||
|
double best = -2.0;
|
||||||
|
long best_lag = 0;
|
||||||
|
for (long lag = -max_lag; lag <= max_lag; ++lag)
|
||||||
|
{
|
||||||
|
const double c = ncc(a, b, lag, mid_start, mid_len);
|
||||||
|
if (c > best)
|
||||||
|
{
|
||||||
|
best = c;
|
||||||
|
best_lag = lag;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Re-evaluate that lag in an early and a late third: the true rate holds; a drifting (wrong)
|
||||||
|
// rate does not.
|
||||||
|
const std::size_t third = n / 3;
|
||||||
|
const double early = ncc(a, b, best_lag, 0, third);
|
||||||
|
const double late = ncc(a, b, best_lag, 2 * third, third);
|
||||||
|
const double weaker = early < late ? early : late;
|
||||||
|
return weaker < 0.0 ? 0.0 : weaker;
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace correlate_detail
|
||||||
|
|
||||||
|
// Determine the hook stream's true sample rate by resampling it by each candidate rate up to the
|
||||||
|
// known device (loopback) rate and scoring how well it aligns with the loopback across the window.
|
||||||
|
// hook_mono / loop_mono are mono float (caller downmixes). `min_score` is the absolute alignment
|
||||||
|
// floor and `separation` the ratio by which the winner must beat the runner-up to be `ok`.
|
||||||
|
inline RateCorrelation correlate_rate(const std::vector<float>& hook_mono, const std::vector<float>& loop_mono,
|
||||||
|
unsigned device_rate, const std::vector<unsigned>& candidates,
|
||||||
|
double min_score = 0.55, double separation = 1.2)
|
||||||
|
{
|
||||||
|
using namespace correlate_detail;
|
||||||
|
RateCorrelation result;
|
||||||
|
if (hook_mono.empty() || loop_mono.empty() || device_rate == 0)
|
||||||
|
{
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
constexpr unsigned kCorrRate = 8000; // alignment search rate (Nyquist 4 kHz -- plenty for content)
|
||||||
|
|
||||||
|
std::vector<float> loop_ds;
|
||||||
|
decimate(loop_mono, device_rate, kCorrRate, loop_ds);
|
||||||
|
|
||||||
|
double best = -1.0, second = -1.0;
|
||||||
|
unsigned best_rate = 0;
|
||||||
|
std::vector<float> resampled, hook_ds;
|
||||||
|
for (unsigned cand : candidates)
|
||||||
|
{
|
||||||
|
resample_linear(hook_mono, cand, device_rate, resampled); // treat hook as sampled at `cand`
|
||||||
|
decimate(resampled, device_rate, kCorrRate, hook_ds);
|
||||||
|
const double s = aligned_score(hook_ds, loop_ds, kCorrRate);
|
||||||
|
if (s > best)
|
||||||
|
{
|
||||||
|
second = best;
|
||||||
|
best = s;
|
||||||
|
best_rate = cand;
|
||||||
|
}
|
||||||
|
else if (s > second)
|
||||||
|
{
|
||||||
|
second = s;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
result.rate = best_rate;
|
||||||
|
result.score = best < 0.0 ? 0.0 : best;
|
||||||
|
result.runner_up = second < 0.0 ? 0.0 : second;
|
||||||
|
result.ok = result.score >= min_score && (result.runner_up <= 1e-6 || result.score >= result.runner_up * separation);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace coop
|
||||||
@@ -87,7 +87,16 @@ struct AudioRingHeader
|
|||||||
std::uint32_t op_bits; // override: bits per sample
|
std::uint32_t op_bits; // override: bits per sample
|
||||||
std::uint32_t op_format_tag; // override: WAVE_FORMAT_PCM / _IEEE_FLOAT
|
std::uint32_t op_format_tag; // override: WAVE_FORMAT_PCM / _IEEE_FLOAT
|
||||||
|
|
||||||
std::uint8_t reserved[40];
|
// Host -> hook: format-verification co-capture. While 1, the hook pushes a still-being-measured
|
||||||
|
// (guessed) stream's raw pre-mix bytes into the ring WITHOUT silencing the game, so the host can
|
||||||
|
// capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same audio
|
||||||
|
// and cross-correlate them to recover the true sample rate (and, in step b, channels/bit-depth)
|
||||||
|
// from ground truth instead of guessing. Inert (0) by default -- normal capture is unaffected,
|
||||||
|
// so it never changes the shipping no-echo path. Repurposed from `reserved`, so the layout and
|
||||||
|
// size are unchanged (old builds saw it as a zero reserved byte).
|
||||||
|
std::atomic<std::uint32_t> verify_capture;
|
||||||
|
|
||||||
|
std::uint8_t reserved[36];
|
||||||
|
|
||||||
// std::uint8_t data[capacity] follows immediately in the mapping.
|
// std::uint8_t data[capacity] follows immediately in the mapping.
|
||||||
};
|
};
|
||||||
@@ -131,6 +140,7 @@ inline void audio_ring_init(AudioRingHeader& h, std::uint32_t capacity)
|
|||||||
h.op_channels = 0;
|
h.op_channels = 0;
|
||||||
h.op_bits = 0;
|
h.op_bits = 0;
|
||||||
h.op_format_tag = 0;
|
h.op_format_tag = 0;
|
||||||
|
h.verify_capture.store(0, std::memory_order_relaxed);
|
||||||
std::memset(h.reserved, 0, sizeof(h.reserved));
|
std::memset(h.reserved, 0, sizeof(h.reserved));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -391,6 +391,30 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Format-verification co-capture (host-driven, step 2a). While the host has set
|
||||||
|
// verify_capture and this guessed stream's rate is still being MEASURED (format not yet
|
||||||
|
// published), push the raw pre-mix bytes to the ring WITHOUT silencing, so the host can
|
||||||
|
// capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same
|
||||||
|
// audio and cross-correlate them to recover the true format from ground truth. The game
|
||||||
|
// stays audible (the host runs loopback during the measurement window anyway), and the
|
||||||
|
// shipping no-echo capture/silence path above is left completely untouched.
|
||||||
|
if (num_frames > 0 && (flags & AUDCLNT_BUFFERFLAGS_SILENT) == 0)
|
||||||
|
{
|
||||||
|
AudioRingHeader* vring = g_rings[i].load(std::memory_order_acquire);
|
||||||
|
if (vring != nullptr && vring->verify_capture.load(std::memory_order_relaxed) != 0 &&
|
||||||
|
!audio_ring_format_ready(*vring) &&
|
||||||
|
g_streams[i].assumed_format.load(std::memory_order_relaxed) != 0 && t_gb_client == self &&
|
||||||
|
t_gb_data != nullptr && t_gb_frames == num_frames &&
|
||||||
|
t_gb_epoch == g_hook_epoch.load(std::memory_order_acquire))
|
||||||
|
{
|
||||||
|
const std::uint32_t block = g_streams[i].block_align.load(std::memory_order_relaxed);
|
||||||
|
if (block != 0)
|
||||||
|
{
|
||||||
|
audio_ring_push(*vring, t_gb_data, readable_bytes(t_gb_data, num_frames * block),
|
||||||
|
num_frames);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
return g_vh_releasebuffer.original<ReleaseBufferFn>()(self, num_frames, flags);
|
return g_vh_releasebuffer.original<ReleaseBufferFn>()(self, num_frames, flags);
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ add_executable(coop_host WIN32
|
|||||||
src/capture/window_capture.cpp
|
src/capture/window_capture.cpp
|
||||||
src/capture/shared_texture.cpp
|
src/capture/shared_texture.cpp
|
||||||
src/audio/audio_loopback.cpp
|
src/audio/audio_loopback.cpp
|
||||||
|
src/audio/audio_format_verifier.cpp
|
||||||
src/audio/audio_overrides.cpp
|
src/audio/audio_overrides.cpp
|
||||||
src/audio/process_loopback_capture.cpp
|
src/audio/process_loopback_capture.cpp
|
||||||
src/vk_layer_setup.cpp
|
src/vk_layer_setup.cpp
|
||||||
|
|||||||
187
host/src/audio/audio_format_verifier.cpp
Normal file
187
host/src/audio/audio_format_verifier.cpp
Normal file
@@ -0,0 +1,187 @@
|
|||||||
|
#include "audio/audio_format_verifier.hpp"
|
||||||
|
|
||||||
|
#include <cstdio>
|
||||||
|
#include <cstdlib>
|
||||||
|
#include <cstring>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include <audioclient.h>
|
||||||
|
#include <mmreg.h>
|
||||||
|
|
||||||
|
#include "audio/process_loopback_capture.hpp"
|
||||||
|
#include "coop/audio_correlate.hpp"
|
||||||
|
|
||||||
|
namespace coop
|
||||||
|
{
|
||||||
|
namespace
|
||||||
|
{
|
||||||
|
|
||||||
|
// Resolve a (possibly EXTENSIBLE) WAVEFORMATEX to scalar channels / bits / tag.
|
||||||
|
struct ScalarFormat
|
||||||
|
{
|
||||||
|
unsigned rate = 0;
|
||||||
|
unsigned channels = 0;
|
||||||
|
unsigned bits = 0;
|
||||||
|
unsigned tag = 0; // WAVE_FORMAT_PCM / _IEEE_FLOAT
|
||||||
|
};
|
||||||
|
|
||||||
|
ScalarFormat resolve(const WAVEFORMATEX* wfx)
|
||||||
|
{
|
||||||
|
ScalarFormat f;
|
||||||
|
f.rate = wfx->nSamplesPerSec;
|
||||||
|
f.channels = wfx->nChannels;
|
||||||
|
f.bits = wfx->wBitsPerSample;
|
||||||
|
f.tag = wfx->wFormatTag;
|
||||||
|
if (wfx->wFormatTag == WAVE_FORMAT_EXTENSIBLE && wfx->cbSize >= 22)
|
||||||
|
{
|
||||||
|
const auto* ext = reinterpret_cast<const WAVEFORMATEXTENSIBLE*>(wfx);
|
||||||
|
if (ext->SubFormat == KSDATAFORMAT_SUBTYPE_IEEE_FLOAT)
|
||||||
|
{
|
||||||
|
f.tag = WAVE_FORMAT_IEEE_FLOAT;
|
||||||
|
}
|
||||||
|
else if (ext->SubFormat == KSDATAFORMAT_SUBTYPE_PCM)
|
||||||
|
{
|
||||||
|
f.tag = WAVE_FORMAT_PCM;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return f;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode interleaved PCM (`channels`/`bits`/`tag`) into per-channel float samples, then average to
|
||||||
|
// mono. Handles float32 and 16/32-bit PCM (the formats WASAPI shared-mode streams use).
|
||||||
|
std::vector<float> to_mono(const std::vector<BYTE>& bytes, const ScalarFormat& fmt)
|
||||||
|
{
|
||||||
|
std::vector<float> mono;
|
||||||
|
const unsigned ch = fmt.channels == 0 ? 1 : fmt.channels;
|
||||||
|
const unsigned bps = fmt.bits / 8;
|
||||||
|
if (bps == 0)
|
||||||
|
{
|
||||||
|
return mono;
|
||||||
|
}
|
||||||
|
const std::size_t frame = static_cast<std::size_t>(ch) * bps;
|
||||||
|
const std::size_t frames = bytes.size() / frame;
|
||||||
|
mono.resize(frames);
|
||||||
|
const bool is_float = fmt.tag == WAVE_FORMAT_IEEE_FLOAT;
|
||||||
|
for (std::size_t i = 0; i < frames; ++i)
|
||||||
|
{
|
||||||
|
double sum = 0.0;
|
||||||
|
for (unsigned c = 0; c < ch; ++c)
|
||||||
|
{
|
||||||
|
const BYTE* p = bytes.data() + i * frame + static_cast<std::size_t>(c) * bps;
|
||||||
|
float s = 0.0f;
|
||||||
|
if (is_float && fmt.bits == 32)
|
||||||
|
{
|
||||||
|
std::memcpy(&s, p, 4);
|
||||||
|
}
|
||||||
|
else if (fmt.bits == 16)
|
||||||
|
{
|
||||||
|
std::int16_t v;
|
||||||
|
std::memcpy(&v, p, 2);
|
||||||
|
s = v / 32768.0f;
|
||||||
|
}
|
||||||
|
else if (fmt.bits == 32)
|
||||||
|
{
|
||||||
|
std::int32_t v;
|
||||||
|
std::memcpy(&v, p, 4);
|
||||||
|
s = static_cast<float>(v / 2147483648.0);
|
||||||
|
}
|
||||||
|
sum += s;
|
||||||
|
}
|
||||||
|
mono[i] = static_cast<float>(sum / ch);
|
||||||
|
}
|
||||||
|
return mono;
|
||||||
|
}
|
||||||
|
|
||||||
|
void drain_ring(AudioRingHeader& ring, std::vector<BYTE>& scratch)
|
||||||
|
{
|
||||||
|
while (audio_ring_pop(ring, scratch.data(), static_cast<std::uint32_t>(scratch.size())) > 0)
|
||||||
|
{
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
FormatVerification verify_stream_format(DWORD pid, AudioRingHeader* ring, unsigned window_ms, bool recover_layout)
|
||||||
|
{
|
||||||
|
(void)recover_layout; // step (b) extends this; step (a) recovers the rate only
|
||||||
|
FormatVerification result;
|
||||||
|
if (ring == nullptr)
|
||||||
|
{
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
WAVEFORMATEX* dev_wfx = default_render_format();
|
||||||
|
if (dev_wfx == nullptr)
|
||||||
|
{
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
const ScalarFormat dev = resolve(dev_wfx);
|
||||||
|
|
||||||
|
// Ask the hook to push the guessed stream's pre-mix bytes (no silence) while it measures, and
|
||||||
|
// clear any stale ring contents so we only collect this window.
|
||||||
|
std::vector<BYTE> scratch(64 * 1024);
|
||||||
|
drain_ring(*ring, scratch);
|
||||||
|
ring->verify_capture.store(1, std::memory_order_release);
|
||||||
|
|
||||||
|
// Capture the post-mix loopback in parallel (this is the known-format ground truth).
|
||||||
|
std::vector<BYTE> loop_bytes;
|
||||||
|
ProcessLoopbackCapture loop;
|
||||||
|
const std::uint32_t loop_block = dev_wfx->nBlockAlign;
|
||||||
|
loop.start(pid, dev_wfx, [&](const BYTE* data, std::uint32_t frames, bool silent) {
|
||||||
|
if (!silent && data != nullptr)
|
||||||
|
{
|
||||||
|
loop_bytes.insert(loop_bytes.end(), data, data + static_cast<std::size_t>(frames) * loop_block);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Pull the hook's pre-mix bytes out of the ring across the window.
|
||||||
|
std::vector<BYTE> hook_bytes;
|
||||||
|
const DWORD end = GetTickCount() + window_ms;
|
||||||
|
while (GetTickCount() < end)
|
||||||
|
{
|
||||||
|
std::uint32_t n = 0;
|
||||||
|
while ((n = audio_ring_pop(*ring, scratch.data(), static_cast<std::uint32_t>(scratch.size()))) > 0)
|
||||||
|
{
|
||||||
|
hook_bytes.insert(hook_bytes.end(), scratch.data(), scratch.data() + n);
|
||||||
|
}
|
||||||
|
Sleep(10);
|
||||||
|
}
|
||||||
|
std::uint32_t n = 0;
|
||||||
|
while ((n = audio_ring_pop(*ring, scratch.data(), static_cast<std::uint32_t>(scratch.size()))) > 0)
|
||||||
|
{
|
||||||
|
hook_bytes.insert(hook_bytes.end(), scratch.data(), scratch.data() + n);
|
||||||
|
}
|
||||||
|
|
||||||
|
loop.stop();
|
||||||
|
ring->verify_capture.store(0, std::memory_order_release);
|
||||||
|
drain_ring(*ring, scratch); // leave the ring clean for the real capture that follows
|
||||||
|
|
||||||
|
// The hook bytes are at the guessed layout = the device channels/bits (the assumption the
|
||||||
|
// cadence path also makes). Decode both captures with that layout and correlate.
|
||||||
|
const std::vector<float> hook_mono = to_mono(hook_bytes, dev);
|
||||||
|
const std::vector<float> loop_mono = to_mono(loop_bytes, dev);
|
||||||
|
CoTaskMemFree(dev_wfx);
|
||||||
|
|
||||||
|
if (const char* dbg = std::getenv("COOP_VERIFY_DEBUG"); dbg != nullptr && dbg[0] == '1')
|
||||||
|
{
|
||||||
|
std::fprintf(stderr, "[verify] dev=%uHz/%uch/%ubit tag=%u hook_frames=%zu loop_frames=%zu\n", dev.rate,
|
||||||
|
dev.channels, dev.bits, dev.tag, hook_mono.size(), loop_mono.size());
|
||||||
|
}
|
||||||
|
const std::size_t need = dev.rate / 5; // require >= ~200 ms of usable audio on both sides
|
||||||
|
if (hook_mono.size() < need || loop_mono.size() < need)
|
||||||
|
{
|
||||||
|
return result; // not enough non-silent audio captured (game quiet, or stream wasn't a guess)
|
||||||
|
}
|
||||||
|
|
||||||
|
const RateCorrelation rc = correlate_rate(hook_mono, loop_mono, dev.rate, standard_audio_rates());
|
||||||
|
result.ok = rc.ok;
|
||||||
|
result.rate = rc.rate;
|
||||||
|
result.score = rc.score;
|
||||||
|
// Channels/bit-depth stay the device assumption here; step (b) recovers them.
|
||||||
|
result.channels = dev.channels;
|
||||||
|
result.bits = dev.bits;
|
||||||
|
result.format_tag = dev.tag;
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace coop
|
||||||
48
host/src/audio/audio_format_verifier.hpp
Normal file
48
host/src/audio/audio_format_verifier.hpp
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
// Two-path audio-format verification: recover a still-being-measured (guessed) render stream's
|
||||||
|
// true format by *correlating* the two capture paths instead of guessing.
|
||||||
|
//
|
||||||
|
// When the host attaches to an already-running game the render-hook never saw the stream's
|
||||||
|
// Initialize, so it assumes the device mix format and measures only the sample rate from the
|
||||||
|
// render cadence -- which a jittery game can make wrong (intermittent pitch shift). During the
|
||||||
|
// measurement window the game is still audible, so we can capture BOTH signals of the same audio:
|
||||||
|
// * the hook (pre-mix, at the unknown format) via the ring's verify_capture tap, and
|
||||||
|
// * a WASAPI process-loopback (post-mix, at the KNOWN device format).
|
||||||
|
// Cross-correlating them (common/include/coop/audio_correlate.hpp) pins the true rate from ground
|
||||||
|
// truth. The result feeds the existing rate path as an operator-style override.
|
||||||
|
//
|
||||||
|
// Step (a) here recovers the sample rate. Step (b) (audio_format_verifier.cpp) extends the same
|
||||||
|
// co-capture to recover channels + bit depth by trying candidate de-interleavings.
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <cstdint>
|
||||||
|
|
||||||
|
#include <windows.h>
|
||||||
|
|
||||||
|
#include "coop/audio_ring.hpp"
|
||||||
|
|
||||||
|
namespace coop
|
||||||
|
{
|
||||||
|
|
||||||
|
struct FormatVerification
|
||||||
|
{
|
||||||
|
bool ok = false; // a confident rate correlation was found
|
||||||
|
unsigned rate = 0; // recovered true sample rate (Hz)
|
||||||
|
double score = 0.0; // correlation score of the winning rate, [0,1]
|
||||||
|
|
||||||
|
bool layout_ok = false; // a confident channels/bit-depth correlation was found (step b)
|
||||||
|
unsigned channels = 0; // recovered channel count
|
||||||
|
unsigned bits = 0; // recovered bits per sample
|
||||||
|
unsigned format_tag = 0; // recovered WAVE_FORMAT_PCM / _IEEE_FLOAT
|
||||||
|
};
|
||||||
|
|
||||||
|
// One-shot: co-capture the hook (pre-mix, via the ring's verify_capture tap) and a parallel
|
||||||
|
// process-loopback (post-mix, at the device format) of `pid`'s audio for ~window_ms, then
|
||||||
|
// cross-correlate to recover the stream's true sample rate (and, with recover_layout, its channels
|
||||||
|
// + bit depth). Returns ok=false (a no-op for the caller) when the stream isn't a guess, the game
|
||||||
|
// is silent, or the two captures don't correlate. Requires a COM-initialized thread. Toggles
|
||||||
|
// ring->verify_capture for the duration and drains the ring afterwards so normal capture starts
|
||||||
|
// clean.
|
||||||
|
FormatVerification verify_stream_format(DWORD pid, AudioRingHeader* ring, unsigned window_ms = 800,
|
||||||
|
bool recover_layout = false);
|
||||||
|
|
||||||
|
} // namespace coop
|
||||||
@@ -9,6 +9,7 @@
|
|||||||
#include <mmdeviceapi.h>
|
#include <mmdeviceapi.h>
|
||||||
#include <mmreg.h>
|
#include <mmreg.h>
|
||||||
|
|
||||||
|
#include "audio/audio_format_verifier.hpp"
|
||||||
#include "audio/audio_mix.hpp"
|
#include "audio/audio_mix.hpp"
|
||||||
#include "audio/process_loopback_capture.hpp"
|
#include "audio/process_loopback_capture.hpp"
|
||||||
#include "audio/render_pacer.hpp"
|
#include "audio/render_pacer.hpp"
|
||||||
@@ -278,6 +279,7 @@ void AudioMirror::thread_main(DWORD pid)
|
|||||||
// The guessed-rate path takes a few seconds to reach consensus; loopback covers
|
// The guessed-rate path takes a few seconds to reach consensus; loopback covers
|
||||||
// that gap and the promote hands off seamlessly.
|
// that gap and the promote hands off seamlessly.
|
||||||
constexpr DWORD kHookWaitMs = 1200;
|
constexpr DWORD kHookWaitMs = 1200;
|
||||||
|
bool format_verified = false; // run the two-path correlation verify/correct at most once
|
||||||
for (;;)
|
for (;;)
|
||||||
{
|
{
|
||||||
if (stop_requested())
|
if (stop_requested())
|
||||||
@@ -312,6 +314,23 @@ void AudioMirror::thread_main(DWORD pid)
|
|||||||
set_fallback_reason(
|
set_fallback_reason(
|
||||||
"Render-hook hasn't published a format yet; using loopback (echo) -- will switch to "
|
"Render-hook hasn't published a format yet; using loopback (echo) -- will switch to "
|
||||||
"hooked automatically once it does.");
|
"hooked automatically once it does.");
|
||||||
|
// No format after the wait -> a guessed late-attach stream is being *measured*. Once,
|
||||||
|
// while it measures and the game is still audible, run the two-path correlation (hook
|
||||||
|
// pre-mix vs a parallel loopback post-mix of the same audio) to recover the true rate
|
||||||
|
// and correct it through the existing override channel -- this hardens the cadence
|
||||||
|
// method's intermittent pitch-shift. It's hidden inside the measurement gap loopback
|
||||||
|
// already covers, so exact streams (format published immediately) never pay for it.
|
||||||
|
if (!format_verified)
|
||||||
|
{
|
||||||
|
format_verified = true;
|
||||||
|
const FormatVerification fv = verify_stream_format(pid, rings[0]);
|
||||||
|
if (fv.ok)
|
||||||
|
{
|
||||||
|
set_status("Verified render-hook sample rate by correlation.");
|
||||||
|
audio_ring_post_op(*rings[0], AudioRingOp_Override, fv.rate, fv.channels, fv.bits,
|
||||||
|
fv.format_tag);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
enable_capture(rings, false); // game audible locally so loopback can capture it
|
enable_capture(rings, false); // game audible locally so loopback can capture it
|
||||||
|
|||||||
@@ -44,6 +44,14 @@ add_executable(tone_analysis_test tone_analysis_test.cpp)
|
|||||||
target_link_libraries(tone_analysis_test PRIVATE coop_common)
|
target_link_libraries(tone_analysis_test PRIVATE coop_common)
|
||||||
add_test(NAME tone_analysis_test COMMAND tone_analysis_test)
|
add_test(NAME tone_analysis_test COMMAND tone_analysis_test)
|
||||||
|
|
||||||
|
# Unit test for the two-path audio-format correlator (common/include/coop/audio_correlate.hpp).
|
||||||
|
# Synthesizes one continuous signal sampled at two rates (the hook's true rate + the device rate,
|
||||||
|
# with capture skew + noise) and asserts correlate_rate() recovers the true rate -- including the
|
||||||
|
# 44100-vs-48000 case the cadence method can misread. Pure header logic, no device.
|
||||||
|
add_executable(audio_correlation_test audio_correlation_test.cpp)
|
||||||
|
target_link_libraries(audio_correlation_test PRIVATE coop_common)
|
||||||
|
add_test(NAME audio_correlation_test COMMAND audio_correlation_test)
|
||||||
|
|
||||||
# Unit test for the audio-mirror render pacing policy (host/src/audio/render_pacer.hpp).
|
# Unit test for the audio-mirror render pacing policy (host/src/audio/render_pacer.hpp).
|
||||||
# Simulates a producer/consumer device timeline and asserts the shipping RenderPacer rides
|
# Simulates a producer/consumer device timeline and asserts the shipping RenderPacer rides
|
||||||
# through producer jitter that makes the old re-prime-on-partial-fill policy glitch
|
# through producer jitter that makes the old re-prime-on-partial-fill policy glitch
|
||||||
@@ -170,6 +178,20 @@ target_link_libraries(mock_game_test PRIVATE coop_common d3d11 dxgi)
|
|||||||
add_dependencies(mock_game_test coop_mock_game coop_hook)
|
add_dependencies(mock_game_test coop_mock_game coop_hook)
|
||||||
add_test(NAME mock_game_test COMMAND mock_game_test)
|
add_test(NAME mock_game_test COMMAND mock_game_test)
|
||||||
|
|
||||||
|
# Integration test for the two-path audio-format verifier: launches coop_mock_game rendering a
|
||||||
|
# NON-device rate (44100 on a 48000 endpoint), injects coop_hook.dll late (guessed stream), and
|
||||||
|
# runs the real verify_stream_format() -- co-capturing the hook (pre-mix) + a parallel loopback
|
||||||
|
# (post-mix) and correlating to recover the true rate. Skips cleanly without an audio endpoint.
|
||||||
|
add_executable(audio_verify_test
|
||||||
|
audio_verify_test.cpp
|
||||||
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp
|
||||||
|
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp)
|
||||||
|
target_include_directories(audio_verify_test PRIVATE ${CMAKE_SOURCE_DIR}/host/src)
|
||||||
|
target_compile_definitions(audio_verify_test PRIVATE NTDDI_VERSION=0x0A00000B)
|
||||||
|
target_link_libraries(audio_verify_test PRIVATE coop_common ole32 mmdevapi)
|
||||||
|
add_dependencies(audio_verify_test coop_mock_game coop_hook)
|
||||||
|
add_test(NAME audio_verify_test COMMAND audio_verify_test)
|
||||||
|
|
||||||
# In-process self-test for the OpenGL capture path. Reuses the shipping
|
# In-process self-test for the OpenGL capture path. Reuses the shipping
|
||||||
# opengl_hook.cpp and drives a real OpenGL context in the same process, so it
|
# opengl_hook.cpp and drives a real OpenGL context in the same process, so it
|
||||||
# exercises the SwapBuffers hook, the glReadPixels readback, and the upload into
|
# exercises the SwapBuffers hook, the glReadPixels readback, and the upload into
|
||||||
@@ -201,6 +223,7 @@ add_executable(ui_fit_test
|
|||||||
${CMAKE_SOURCE_DIR}/host/src/audio_panel.cpp
|
${CMAKE_SOURCE_DIR}/host/src/audio_panel.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/controllers_panel.cpp
|
${CMAKE_SOURCE_DIR}/host/src/controllers_panel.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp
|
||||||
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp
|
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/ui/app_chrome.cpp)
|
${CMAKE_SOURCE_DIR}/host/src/ui/app_chrome.cpp)
|
||||||
@@ -224,6 +247,7 @@ coop_output_subdir(tests
|
|||||||
mkb_map_test
|
mkb_map_test
|
||||||
audio_mix_test
|
audio_mix_test
|
||||||
tone_analysis_test
|
tone_analysis_test
|
||||||
|
audio_correlation_test
|
||||||
render_pacer_test
|
render_pacer_test
|
||||||
rate_estimator_test
|
rate_estimator_test
|
||||||
audio_overrides_test
|
audio_overrides_test
|
||||||
@@ -234,4 +258,5 @@ coop_output_subdir(tests
|
|||||||
dx12_present_hook_test
|
dx12_present_hook_test
|
||||||
opengl_hook_test
|
opengl_hook_test
|
||||||
mock_game_test
|
mock_game_test
|
||||||
|
audio_verify_test
|
||||||
ui_fit_test)
|
ui_fit_test)
|
||||||
|
|||||||
121
tests/audio_correlation_test.cpp
Normal file
121
tests/audio_correlation_test.cpp
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
// Unit test for the two-path audio-format correlator (common/include/coop/audio_correlate.hpp).
|
||||||
|
//
|
||||||
|
// Models the real situation: the same game audio is captured twice -- by the render-hook at the
|
||||||
|
// stream's true (unknown) rate, and by process-loopback at the known device rate (the hook signal
|
||||||
|
// resampled by WASAPI's AUTOCONVERTPCM). We synthesize one continuous signal and sample it at both
|
||||||
|
// rates (plus a capture-latency skew and a little noise), then assert correlate_rate() recovers the
|
||||||
|
// true rate -- including the hard 44100-vs-48000 case the cadence method can misread. Pure header
|
||||||
|
// logic, no device.
|
||||||
|
#include <cmath>
|
||||||
|
#include <cstdint>
|
||||||
|
#include <cstdio>
|
||||||
|
#include <random>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include "coop/audio_correlate.hpp"
|
||||||
|
|
||||||
|
using namespace coop;
|
||||||
|
|
||||||
|
namespace
|
||||||
|
{
|
||||||
|
int g_failures = 0;
|
||||||
|
void check(bool ok, const char* what)
|
||||||
|
{
|
||||||
|
std::printf("%s %s\n", ok ? " ok:" : "FAIL:", what);
|
||||||
|
if (!ok)
|
||||||
|
{
|
||||||
|
++g_failures;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
constexpr double kPi = 3.14159265358979323846;
|
||||||
|
|
||||||
|
// A non-periodic, correlation-friendly continuous signal s(t): a couple of incommensurate tones
|
||||||
|
// plus a slow chirp, so cross-correlation has a single sharp peak (unlike a pure sine).
|
||||||
|
double source(double t)
|
||||||
|
{
|
||||||
|
const double chirp = std::sin(2.0 * kPi * (300.0 * t + 140.0 * t * t));
|
||||||
|
return 0.5 * std::sin(2.0 * kPi * 221.0 * t) + 0.28 * std::sin(2.0 * kPi * 437.0 * t + 0.6) +
|
||||||
|
0.22 * chirp;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sample s(t) at `rate` for `seconds`, starting at t0 (capture-latency skew), optionally adding
|
||||||
|
// white noise of amplitude `noise` (the post-mix path is not a bit-identical copy).
|
||||||
|
std::vector<float> capture(unsigned rate, double seconds, double t0, double noise, std::uint32_t seed)
|
||||||
|
{
|
||||||
|
const std::size_t n = static_cast<std::size_t>(rate * seconds);
|
||||||
|
std::vector<float> out(n);
|
||||||
|
std::mt19937 rng(seed);
|
||||||
|
std::uniform_real_distribution<float> jitter(-1.0f, 1.0f);
|
||||||
|
for (std::size_t i = 0; i < n; ++i)
|
||||||
|
{
|
||||||
|
const double t = t0 + static_cast<double>(i) / static_cast<double>(rate);
|
||||||
|
out[i] = static_cast<float>(source(t)) + static_cast<float>(noise) * jitter(rng);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// One scenario: true hook rate `true_rate` mixed to `device_rate`. Assert the correlator picks
|
||||||
|
// true_rate confidently and that the runner-up is clearly behind.
|
||||||
|
void test_case(unsigned true_rate, unsigned device_rate, const char* label)
|
||||||
|
{
|
||||||
|
std::printf("== %s (true %u Hz -> device %u Hz) ==\n", label, true_rate, device_rate);
|
||||||
|
// The hook captures at the true rate; the loopback captures the same signal at the device
|
||||||
|
// rate, started ~22 ms later (capture skew) with a little measurement noise.
|
||||||
|
const std::vector<float> hook = capture(true_rate, 0.55, 0.0, 0.0, 1);
|
||||||
|
const std::vector<float> loop = capture(device_rate, 0.55, 0.022, 0.02, 7);
|
||||||
|
|
||||||
|
const RateCorrelation r = correlate_rate(hook, loop, device_rate, standard_audio_rates());
|
||||||
|
std::printf(" picked %u Hz score=%.3f runner_up=%.3f ok=%d\n", r.rate, r.score, r.runner_up,
|
||||||
|
r.ok ? 1 : 0);
|
||||||
|
check(r.rate == true_rate, "correlator picked the true rate");
|
||||||
|
check(r.ok, "pick is confident (clears threshold + beats runner-up)");
|
||||||
|
check(r.score > r.runner_up, "winner scores above the runner-up");
|
||||||
|
}
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
int main()
|
||||||
|
{
|
||||||
|
// The headline case: Godot/Brotato render 44100 while the endpoint mixes 48000 -- the cadence
|
||||||
|
// method can misread this, the correlator must not.
|
||||||
|
test_case(44100, 48000, "godot/brotato case");
|
||||||
|
test_case(48000, 48000, "rate matches device");
|
||||||
|
test_case(96000, 48000, "high-rate stream");
|
||||||
|
test_case(32000, 44100, "low-rate stream on a 44100 endpoint");
|
||||||
|
test_case(48000, 44100, "48000 stream on a 44100 endpoint");
|
||||||
|
|
||||||
|
// Downmix sanity: a stereo interleaved buffer collapses to the same mono the scalar path uses.
|
||||||
|
{
|
||||||
|
std::printf("== downmix stereo -> mono ==\n");
|
||||||
|
std::vector<float> stereo = {1.0f, 3.0f, 2.0f, 4.0f, -1.0f, 1.0f};
|
||||||
|
std::vector<float> mono;
|
||||||
|
correlate_detail::downmix(stereo.data(), 3, 2, mono);
|
||||||
|
check(mono.size() == 3 && std::fabs(mono[0] - 2.0f) < 1e-6 && std::fabs(mono[1] - 3.0f) < 1e-6 &&
|
||||||
|
std::fabs(mono[2] - 0.0f) < 1e-6,
|
||||||
|
"stereo frames average to mono");
|
||||||
|
}
|
||||||
|
|
||||||
|
// A pure guess with no shared signal must NOT be reported confident (loopback is unrelated noise).
|
||||||
|
{
|
||||||
|
std::printf("== unrelated signals are not confidently matched ==\n");
|
||||||
|
const std::vector<float> hook = capture(44100, 0.5, 0.0, 0.0, 1);
|
||||||
|
std::vector<float> noise(static_cast<std::size_t>(48000 * 0.5));
|
||||||
|
std::mt19937 rng(99);
|
||||||
|
std::uniform_real_distribution<float> d(-1.0f, 1.0f);
|
||||||
|
for (float& x : noise)
|
||||||
|
{
|
||||||
|
x = d(rng);
|
||||||
|
}
|
||||||
|
const RateCorrelation r = correlate_rate(hook, noise, 48000, standard_audio_rates());
|
||||||
|
std::printf(" picked %u Hz score=%.3f ok=%d\n", r.rate, r.score, r.ok ? 1 : 0);
|
||||||
|
check(!r.ok, "unrelated loopback is not a confident match");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (g_failures == 0)
|
||||||
|
{
|
||||||
|
std::printf("PASS audio_correlation_test\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
std::printf("FAILED audio_correlation_test (%d)\n", g_failures);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
233
tests/audio_verify_test.cpp
Normal file
233
tests/audio_verify_test.cpp
Normal file
@@ -0,0 +1,233 @@
|
|||||||
|
// Integration test for the two-path audio-format verifier (host/src/audio/audio_format_verifier).
|
||||||
|
//
|
||||||
|
// Launches coop_mock_game rendering a tone at a NON-device rate (44100 on a typical 48000 endpoint,
|
||||||
|
// the Godot/Brotato case), injects coop_hook.dll late (so the stream is a *guess*), then runs the
|
||||||
|
// real verify_stream_format(): it co-captures the hook (pre-mix, via the ring's verify tap) and a
|
||||||
|
// parallel process-loopback (post-mix, device format) and cross-correlates them. Asserts it
|
||||||
|
// recovers the true 44100 Hz rate -- the cadence method's hard case. Skips cleanly without an audio
|
||||||
|
// endpoint / if Vulkan-free... (only needs WASAPI + a D3D11-capable mock, which the mock always is).
|
||||||
|
#include <cstdint>
|
||||||
|
#include <cstdio>
|
||||||
|
#include <string>
|
||||||
|
|
||||||
|
#include <windows.h>
|
||||||
|
|
||||||
|
#include <objbase.h>
|
||||||
|
#include <tlhelp32.h>
|
||||||
|
|
||||||
|
#include <mmreg.h>
|
||||||
|
|
||||||
|
#include "audio/audio_format_verifier.hpp"
|
||||||
|
#include "audio/process_loopback_capture.hpp" // default_render_format
|
||||||
|
#include "coop/audio_ring.hpp"
|
||||||
|
#include "coop/protocol.hpp"
|
||||||
|
#include "coop/shared_memory.hpp"
|
||||||
|
#include "coop/tool_paths.hpp"
|
||||||
|
|
||||||
|
using namespace coop;
|
||||||
|
|
||||||
|
namespace
|
||||||
|
{
|
||||||
|
int g_failures = 0;
|
||||||
|
void check(bool ok, const char* what)
|
||||||
|
{
|
||||||
|
std::printf("%s %s\n", ok ? " ok:" : "FAIL:", what);
|
||||||
|
if (!ok)
|
||||||
|
{
|
||||||
|
++g_failures;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void kill_stray_mock_games()
|
||||||
|
{
|
||||||
|
HANDLE snap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
|
||||||
|
if (snap == INVALID_HANDLE_VALUE)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
PROCESSENTRY32W pe{};
|
||||||
|
pe.dwSize = sizeof(pe);
|
||||||
|
for (BOOL ok = Process32FirstW(snap, &pe); ok; ok = Process32NextW(snap, &pe))
|
||||||
|
{
|
||||||
|
if (_wcsicmp(pe.szExeFile, L"coop_mock_game.exe") == 0)
|
||||||
|
{
|
||||||
|
if (HANDLE h = OpenProcess(PROCESS_TERMINATE, FALSE, pe.th32ProcessID))
|
||||||
|
{
|
||||||
|
TerminateProcess(h, 0);
|
||||||
|
CloseHandle(h);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
CloseHandle(snap);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool inject(unsigned long pid)
|
||||||
|
{
|
||||||
|
const std::wstring dll = deployed_artifact_path(L"coop_hook.dll");
|
||||||
|
if (GetFileAttributesW(dll.c_str()) == INVALID_FILE_ATTRIBUTES)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const DWORD access = PROCESS_CREATE_THREAD | PROCESS_QUERY_INFORMATION | PROCESS_VM_OPERATION |
|
||||||
|
PROCESS_VM_WRITE | PROCESS_VM_READ;
|
||||||
|
HANDLE process = OpenProcess(access, FALSE, pid);
|
||||||
|
if (process == nullptr)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const SIZE_T bytes = (dll.size() + 1) * sizeof(wchar_t);
|
||||||
|
void* remote = VirtualAllocEx(process, nullptr, bytes, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
|
||||||
|
bool ok = false;
|
||||||
|
if (remote != nullptr && WriteProcessMemory(process, remote, dll.c_str(), bytes, nullptr))
|
||||||
|
{
|
||||||
|
auto load = reinterpret_cast<LPTHREAD_START_ROUTINE>(
|
||||||
|
GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "LoadLibraryW"));
|
||||||
|
if (HANDLE th = CreateRemoteThread(process, nullptr, 0, load, remote, 0, nullptr))
|
||||||
|
{
|
||||||
|
WaitForSingleObject(th, INFINITE);
|
||||||
|
DWORD code = 0;
|
||||||
|
GetExitCodeThread(th, &code);
|
||||||
|
CloseHandle(th);
|
||||||
|
ok = code != 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (remote != nullptr)
|
||||||
|
{
|
||||||
|
VirtualFreeEx(process, remote, 0, MEM_RELEASE);
|
||||||
|
}
|
||||||
|
CloseHandle(process);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool inject_retry(unsigned long pid)
|
||||||
|
{
|
||||||
|
for (int i = 0; i < 4; ++i)
|
||||||
|
{
|
||||||
|
if (inject(pid))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
Sleep(300);
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
int main()
|
||||||
|
{
|
||||||
|
kill_stray_mock_games();
|
||||||
|
|
||||||
|
const bool com = SUCCEEDED(CoInitializeEx(nullptr, COINIT_MULTITHREADED));
|
||||||
|
|
||||||
|
// Render the mock at the device's CHANNEL count (so this step-(a) rate test isn't perturbed by
|
||||||
|
// a channel mismatch -- that's step (b)'s job) but at a DIFFERENT standard rate than the device,
|
||||||
|
// so the verifier has a real rate to recover. Default to 48000/2ch if we can't read the device.
|
||||||
|
unsigned dev_rate = 48000, dev_channels = 2;
|
||||||
|
if (WAVEFORMATEX* dev = default_render_format())
|
||||||
|
{
|
||||||
|
dev_rate = dev->nSamplesPerSec;
|
||||||
|
dev_channels = dev->nChannels;
|
||||||
|
CoTaskMemFree(dev);
|
||||||
|
}
|
||||||
|
const unsigned game_rate = (dev_rate == 44100) ? 48000u : 44100u; // guarantee a rate mismatch
|
||||||
|
std::printf(" device %u Hz / %u ch -> rendering the mock at %u Hz / %u ch (rate mismatch)\n", dev_rate,
|
||||||
|
dev_channels, game_rate, dev_channels);
|
||||||
|
|
||||||
|
const std::wstring exe = exe_directory() + L"coop_mock_game.exe";
|
||||||
|
std::wstring cmd = L"\"" + exe + L"\" dx11 30 " + std::to_wstring(game_rate) + L" " +
|
||||||
|
std::to_wstring(dev_channels) + L" 32 float";
|
||||||
|
STARTUPINFOW si{};
|
||||||
|
si.cb = sizeof(si);
|
||||||
|
PROCESS_INFORMATION pi{};
|
||||||
|
if (!CreateProcessW(exe.c_str(), cmd.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, &si, &pi))
|
||||||
|
{
|
||||||
|
std::printf("Could not launch coop_mock_game -- skipping audio_verify_test.\n");
|
||||||
|
if (com)
|
||||||
|
{
|
||||||
|
CoUninitialize();
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
auto cleanup = [&] {
|
||||||
|
TerminateProcess(pi.hProcess, 0);
|
||||||
|
WaitForSingleObject(pi.hProcess, 2000);
|
||||||
|
CloseHandle(pi.hThread);
|
||||||
|
CloseHandle(pi.hProcess);
|
||||||
|
kill_stray_mock_games();
|
||||||
|
};
|
||||||
|
Sleep(800); // window + audio client up
|
||||||
|
|
||||||
|
// IPC + the primary audio ring the hook produces into.
|
||||||
|
SharedMemory shm;
|
||||||
|
if (!shm.create(shared_memory_name(pi.dwProcessId), sizeof(SharedBlock)))
|
||||||
|
{
|
||||||
|
std::printf("Could not create IPC block -- skipping.\n");
|
||||||
|
cleanup();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
auto* block = shm.as<SharedBlock>();
|
||||||
|
block->version = kProtocolVersion;
|
||||||
|
block->pad_count = 0;
|
||||||
|
block->sequence.store(0, std::memory_order_relaxed);
|
||||||
|
// Only the audio subsystem.
|
||||||
|
for (std::uint32_t s = 0; s < HookSubsys_Count; ++s)
|
||||||
|
{
|
||||||
|
const bool off = s != HookSubsys_Audio;
|
||||||
|
block->control.subsystem_disabled[s].store(off ? 1u : 0u, std::memory_order_release);
|
||||||
|
}
|
||||||
|
block->magic = kProtocolMagic;
|
||||||
|
|
||||||
|
SharedMemory ring_shm;
|
||||||
|
if (!ring_shm.create(audio_ring_name(pi.dwProcessId), audio_ring_total_size(kAudioRingCapacity)))
|
||||||
|
{
|
||||||
|
std::printf("Could not create audio ring -- skipping.\n");
|
||||||
|
cleanup();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
auto* ring = ring_shm.as<AudioRingHeader>();
|
||||||
|
audio_ring_init(*ring, kAudioRingCapacity);
|
||||||
|
// Leave capture_enabled = 0: we want the stream audible (so loopback hears it) and still being
|
||||||
|
// MEASURED (so the verify tap fires), exactly the window verify_stream_format targets.
|
||||||
|
|
||||||
|
if (!inject_retry(pi.dwProcessId))
|
||||||
|
{
|
||||||
|
std::printf("Could not inject coop_hook.dll -- skipping.\n");
|
||||||
|
cleanup();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
Sleep(500); // let the hook attach + the pre-existing render client register as a guess
|
||||||
|
|
||||||
|
// Run the real verifier: co-capture hook (pre-mix) + loopback (post-mix) and correlate.
|
||||||
|
const FormatVerification fv = verify_stream_format(pi.dwProcessId, ring, /*window_ms=*/1400);
|
||||||
|
std::printf(" verify: ok=%d rate=%u score=%.3f\n", fv.ok ? 1 : 0, fv.rate, fv.score);
|
||||||
|
|
||||||
|
if (!fv.ok && fv.rate == 0 && fv.score == 0.0)
|
||||||
|
{
|
||||||
|
// No audio endpoint, or no usable audio captured (e.g. the mock's WASAPI client never
|
||||||
|
// started on this machine) -> treat as a skip rather than a failure.
|
||||||
|
std::printf(" no usable co-capture (no endpoint / silent) -- skipping audio_verify_test.\n");
|
||||||
|
if (com)
|
||||||
|
{
|
||||||
|
CoUninitialize();
|
||||||
|
}
|
||||||
|
cleanup();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
check(fv.ok, "verifier confidently correlated the two capture paths");
|
||||||
|
check(fv.rate == game_rate, "verifier recovered the game's true rate (not the device rate)");
|
||||||
|
|
||||||
|
if (com)
|
||||||
|
{
|
||||||
|
CoUninitialize();
|
||||||
|
}
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
if (g_failures == 0)
|
||||||
|
{
|
||||||
|
std::printf("PASS audio_verify_test\n");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
std::printf("FAILED audio_verify_test (%d)\n", g_failures);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
@@ -8,6 +8,7 @@
|
|||||||
add_executable(coop_audio_validate
|
add_executable(coop_audio_validate
|
||||||
main.cpp
|
main.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp
|
||||||
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp
|
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp
|
||||||
${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp)
|
${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user