BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Recover a guessed audio stream's rate by correlating hook vs loopback (step a)

Browse Source 
When the host attaches to an already-running game it never saw the stream's
Initialize, so the render-hook assumes the device mix format and measures only
the sample rate from the render cadence -- which a jittery game can make wrong
(intermittent pitch shift). But during the measurement window the game is still
audible, so we have the same audio twice: the hook (pre-mix, unknown format) and
a process-loopback (post-mix, the known device format). Cross-correlating them
pins the true rate from ground truth.

- common/include/coop/audio_correlate.hpp: the pure correlator. Resample the hook
  by each candidate standard rate up to the device rate and score how well it
  aligns with the loopback across the window (drift-detecting). audio_correlation_test
  recovers every rate (score ~1.0 vs ~0.01 for wrong ones), incl. 44100-vs-48000,
  and rejects unrelated signals.
- Hook measurement tap: a host-set verify_capture ring flag makes the hook push a
  still-being-measured (guessed) stream's raw pre-mix bytes WITHOUT silencing, so
  the host can co-capture both signals (a silenced game's loopback is silent).
  Inert by default -- the shipping no-echo path is untouched.
- host/src/audio/audio_format_verifier: co-captures hook + loopback and correlates,
  feeding a correction into the existing override channel. Wired into AudioMirror's
  measurement window (hidden in the gap loopback already covers, so exact streams
  pay nothing). audio_verify_test drives it end-to-end against coop_mock_game.

Rate vs layout are coupled (correlating the waveform needs the right channel
de-interleaving), so this step assumes the hook layout matches the device (the
common stereo-on-stereo case); recovering a different channel count / bit depth
is step b.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-23 02:26:06 +02:00
parent 21c15b162b
commit 00244bcfd7
12 changed files with 961 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
tests/CMakeLists.txt
View File

@@ -44,6 +44,14 @@ add_executable(tone_analysis_test tone_analysis_test.cpp)
target_link_libraries(tone_analysis_test PRIVATE coop_common)
add_test(NAME tone_analysis_test COMMAND tone_analysis_test)
# Unit test for the two-path audio-format correlator (common/include/coop/audio_correlate.hpp).
# Synthesizes one continuous signal sampled at two rates (the hook's true rate + the device rate,
# with capture skew + noise) and asserts correlate_rate() recovers the true rate -- including the
# 44100-vs-48000 case the cadence method can misread. Pure header logic, no device.
add_executable(audio_correlation_test audio_correlation_test.cpp)
target_link_libraries(audio_correlation_test PRIVATE coop_common)
add_test(NAME audio_correlation_test COMMAND audio_correlation_test)
# Unit test for the audio-mirror render pacing policy (host/src/audio/render_pacer.hpp).
# Simulates a producer/consumer device timeline and asserts the shipping RenderPacer rides
# through producer jitter that makes the old re-prime-on-partial-fill policy glitch
@@ -170,6 +178,20 @@ target_link_libraries(mock_game_test PRIVATE coop_common d3d11 dxgi)
add_dependencies(mock_game_test coop_mock_game coop_hook)
add_test(NAME mock_game_test COMMAND mock_game_test)
# Integration test for the two-path audio-format verifier: launches coop_mock_game rendering a
# NON-device rate (44100 on a 48000 endpoint), injects coop_hook.dll late (guessed stream), and
# runs the real verify_stream_format() -- co-capturing the hook (pre-mix) + a parallel loopback
# (post-mix) and correlating to recover the true rate. Skips cleanly without an audio endpoint.
add_executable(audio_verify_test
audio_verify_test.cpp
${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp)
target_include_directories(audio_verify_test PRIVATE ${CMAKE_SOURCE_DIR}/host/src)
target_compile_definitions(audio_verify_test PRIVATE NTDDI_VERSION=0x0A00000B)
target_link_libraries(audio_verify_test PRIVATE coop_common ole32 mmdevapi)
add_dependencies(audio_verify_test coop_mock_game coop_hook)
add_test(NAME audio_verify_test COMMAND audio_verify_test)
# In-process self-test for the OpenGL capture path. Reuses the shipping
# opengl_hook.cpp and drives a real OpenGL context in the same process, so it
# exercises the SwapBuffers hook, the glReadPixels readback, and the upload into
@@ -201,6 +223,7 @@ add_executable(ui_fit_test
${CMAKE_SOURCE_DIR}/host/src/audio_panel.cpp
${CMAKE_SOURCE_DIR}/host/src/controllers_panel.cpp
${CMAKE_SOURCE_DIR}/host/src/audio/audio_loopback.cpp
${CMAKE_SOURCE_DIR}/host/src/audio/audio_format_verifier.cpp
${CMAKE_SOURCE_DIR}/host/src/audio/process_loopback_capture.cpp
${CMAKE_SOURCE_DIR}/host/src/audio/audio_overrides.cpp
${CMAKE_SOURCE_DIR}/host/src/ui/app_chrome.cpp)
@@ -224,6 +247,7 @@ coop_output_subdir(tests
mkb_map_test
audio_mix_test
tone_analysis_test
audio_correlation_test
render_pacer_test
rate_estimator_test
audio_overrides_test
@@ -234,4 +258,5 @@ coop_output_subdir(tests
dx12_present_hook_test
opengl_hook_test
mock_game_test
audio_verify_test
ui_fit_test)

121
tests/audio_correlation_test.cpp Normal file
View File

@@ -0,0 +1,121 @@
// Unit test for the two-path audio-format correlator (common/include/coop/audio_correlate.hpp).
//
// Models the real situation: the same game audio is captured twice -- by the render-hook at the
// stream's true (unknown) rate, and by process-loopback at the known device rate (the hook signal
// resampled by WASAPI's AUTOCONVERTPCM). We synthesize one continuous signal and sample it at both
// rates (plus a capture-latency skew and a little noise), then assert correlate_rate() recovers the
// true rate -- including the hard 44100-vs-48000 case the cadence method can misread. Pure header
// logic, no device.
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <random>
#include <vector>
#include "coop/audio_correlate.hpp"
using namespace coop;
namespace
{
int g_failures = 0;
void check(bool ok, const char* what)
{
std::printf("%s %s\n", ok ? " ok:" : "FAIL:", what);
if (!ok)
{
++g_failures;
}
}
constexpr double kPi = 3.14159265358979323846;
// A non-periodic, correlation-friendly continuous signal s(t): a couple of incommensurate tones
// plus a slow chirp, so cross-correlation has a single sharp peak (unlike a pure sine).
double source(double t)
{
const double chirp = std::sin(2.0 * kPi * (300.0 * t + 140.0 * t * t));
return 0.5 * std::sin(2.0 * kPi * 221.0 * t) + 0.28 * std::sin(2.0 * kPi * 437.0 * t + 0.6) +
0.22 * chirp;
}
// Sample s(t) at `rate` for `seconds`, starting at t0 (capture-latency skew), optionally adding
// white noise of amplitude `noise` (the post-mix path is not a bit-identical copy).
std::vector<float> capture(unsigned rate, double seconds, double t0, double noise, std::uint32_t seed)
{
const std::size_t n = static_cast<std::size_t>(rate * seconds);
std::vector<float> out(n);
std::mt19937 rng(seed);
std::uniform_real_distribution<float> jitter(-1.0f, 1.0f);
for (std::size_t i = 0; i < n; ++i)
{
const double t = t0 + static_cast<double>(i) / static_cast<double>(rate);
out[i] = static_cast<float>(source(t)) + static_cast<float>(noise) * jitter(rng);
}
return out;
}
// One scenario: true hook rate `true_rate` mixed to `device_rate`. Assert the correlator picks
// true_rate confidently and that the runner-up is clearly behind.
void test_case(unsigned true_rate, unsigned device_rate, const char* label)
{
std::printf("== %s (true %u Hz -> device %u Hz) ==\n", label, true_rate, device_rate);
// The hook captures at the true rate; the loopback captures the same signal at the device
// rate, started ~22 ms later (capture skew) with a little measurement noise.
const std::vector<float> hook = capture(true_rate, 0.55, 0.0, 0.0, 1);
const std::vector<float> loop = capture(device_rate, 0.55, 0.022, 0.02, 7);
const RateCorrelation r = correlate_rate(hook, loop, device_rate, standard_audio_rates());
std::printf(" picked %u Hz score=%.3f runner_up=%.3f ok=%d\n", r.rate, r.score, r.runner_up,
r.ok ? 1 : 0);
check(r.rate == true_rate, "correlator picked the true rate");
check(r.ok, "pick is confident (clears threshold + beats runner-up)");
check(r.score > r.runner_up, "winner scores above the runner-up");
}
} // namespace
int main()
{
// The headline case: Godot/Brotato render 44100 while the endpoint mixes 48000 -- the cadence
// method can misread this, the correlator must not.
test_case(44100, 48000, "godot/brotato case");
test_case(48000, 48000, "rate matches device");
test_case(96000, 48000, "high-rate stream");
test_case(32000, 44100, "low-rate stream on a 44100 endpoint");
test_case(48000, 44100, "48000 stream on a 44100 endpoint");
// Downmix sanity: a stereo interleaved buffer collapses to the same mono the scalar path uses.
{
std::printf("== downmix stereo -> mono ==\n");
std::vector<float> stereo = {1.0f, 3.0f, 2.0f, 4.0f, -1.0f, 1.0f};
std::vector<float> mono;
correlate_detail::downmix(stereo.data(), 3, 2, mono);
check(mono.size() == 3 && std::fabs(mono[0] - 2.0f) < 1e-6 && std::fabs(mono[1] - 3.0f) < 1e-6 &&
std::fabs(mono[2] - 0.0f) < 1e-6,
"stereo frames average to mono");
}
// A pure guess with no shared signal must NOT be reported confident (loopback is unrelated noise).
{
std::printf("== unrelated signals are not confidently matched ==\n");
const std::vector<float> hook = capture(44100, 0.5, 0.0, 0.0, 1);
std::vector<float> noise(static_cast<std::size_t>(48000 * 0.5));
std::mt19937 rng(99);
std::uniform_real_distribution<float> d(-1.0f, 1.0f);
for (float& x : noise)
{
x = d(rng);
}
const RateCorrelation r = correlate_rate(hook, noise, 48000, standard_audio_rates());
std::printf(" picked %u Hz score=%.3f ok=%d\n", r.rate, r.score, r.ok ? 1 : 0);
check(!r.ok, "unrelated loopback is not a confident match");
}
if (g_failures == 0)
{
std::printf("PASS audio_correlation_test\n");
return 0;
}
std::printf("FAILED audio_correlation_test (%d)\n", g_failures);
return 1;
}

233
tests/audio_verify_test.cpp Normal file
View File

@@ -0,0 +1,233 @@
// Integration test for the two-path audio-format verifier (host/src/audio/audio_format_verifier).
//
// Launches coop_mock_game rendering a tone at a NON-device rate (44100 on a typical 48000 endpoint,
// the Godot/Brotato case), injects coop_hook.dll late (so the stream is a *guess*), then runs the
// real verify_stream_format(): it co-captures the hook (pre-mix, via the ring's verify tap) and a
// parallel process-loopback (post-mix, device format) and cross-correlates them. Asserts it
// recovers the true 44100 Hz rate -- the cadence method's hard case. Skips cleanly without an audio
// endpoint / if Vulkan-free... (only needs WASAPI + a D3D11-capable mock, which the mock always is).
#include <cstdint>
#include <cstdio>
#include <string>
#include <windows.h>
#include <objbase.h>
#include <tlhelp32.h>
#include <mmreg.h>
#include "audio/audio_format_verifier.hpp"
#include "audio/process_loopback_capture.hpp" // default_render_format
#include "coop/audio_ring.hpp"
#include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "coop/tool_paths.hpp"
using namespace coop;
namespace
{
int g_failures = 0;
void check(bool ok, const char* what)
{
std::printf("%s %s\n", ok ? " ok:" : "FAIL:", what);
if (!ok)
{
++g_failures;
}
}
void kill_stray_mock_games()
{
HANDLE snap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
if (snap == INVALID_HANDLE_VALUE)
{
return;
}
PROCESSENTRY32W pe{};
pe.dwSize = sizeof(pe);
for (BOOL ok = Process32FirstW(snap, &pe); ok; ok = Process32NextW(snap, &pe))
{
if (_wcsicmp(pe.szExeFile, L"coop_mock_game.exe") == 0)
{
if (HANDLE h = OpenProcess(PROCESS_TERMINATE, FALSE, pe.th32ProcessID))
{
TerminateProcess(h, 0);
CloseHandle(h);
}
}
}
CloseHandle(snap);
}
bool inject(unsigned long pid)
{
const std::wstring dll = deployed_artifact_path(L"coop_hook.dll");
if (GetFileAttributesW(dll.c_str()) == INVALID_FILE_ATTRIBUTES)
{
return false;
}
const DWORD access = PROCESS_CREATE_THREAD | PROCESS_QUERY_INFORMATION | PROCESS_VM_OPERATION |
PROCESS_VM_WRITE | PROCESS_VM_READ;
HANDLE process = OpenProcess(access, FALSE, pid);
if (process == nullptr)
{
return false;
}
const SIZE_T bytes = (dll.size() + 1) * sizeof(wchar_t);
void* remote = VirtualAllocEx(process, nullptr, bytes, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
bool ok = false;
if (remote != nullptr && WriteProcessMemory(process, remote, dll.c_str(), bytes, nullptr))
{
auto load = reinterpret_cast<LPTHREAD_START_ROUTINE>(
GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "LoadLibraryW"));
if (HANDLE th = CreateRemoteThread(process, nullptr, 0, load, remote, 0, nullptr))
{
WaitForSingleObject(th, INFINITE);
DWORD code = 0;
GetExitCodeThread(th, &code);
CloseHandle(th);
ok = code != 0;
}
}
if (remote != nullptr)
{
VirtualFreeEx(process, remote, 0, MEM_RELEASE);
}
CloseHandle(process);
return ok;
}
bool inject_retry(unsigned long pid)
{
for (int i = 0; i < 4; ++i)
{
if (inject(pid))
{
return true;
}
Sleep(300);
}
return false;
}
} // namespace
int main()
{
kill_stray_mock_games();
const bool com = SUCCEEDED(CoInitializeEx(nullptr, COINIT_MULTITHREADED));
// Render the mock at the device's CHANNEL count (so this step-(a) rate test isn't perturbed by
// a channel mismatch -- that's step (b)'s job) but at a DIFFERENT standard rate than the device,
// so the verifier has a real rate to recover. Default to 48000/2ch if we can't read the device.
unsigned dev_rate = 48000, dev_channels = 2;
if (WAVEFORMATEX* dev = default_render_format())
{
dev_rate = dev->nSamplesPerSec;
dev_channels = dev->nChannels;
CoTaskMemFree(dev);
}
const unsigned game_rate = (dev_rate == 44100) ? 48000u : 44100u; // guarantee a rate mismatch
std::printf(" device %u Hz / %u ch -> rendering the mock at %u Hz / %u ch (rate mismatch)\n", dev_rate,
dev_channels, game_rate, dev_channels);
const std::wstring exe = exe_directory() + L"coop_mock_game.exe";
std::wstring cmd = L"\"" + exe + L"\" dx11 30 " + std::to_wstring(game_rate) + L" " +
std::to_wstring(dev_channels) + L" 32 float";
STARTUPINFOW si{};
si.cb = sizeof(si);
PROCESS_INFORMATION pi{};
if (!CreateProcessW(exe.c_str(), cmd.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, &si, &pi))
{
std::printf("Could not launch coop_mock_game -- skipping audio_verify_test.\n");
if (com)
{
CoUninitialize();
}
return 0;
}
auto cleanup = [&] {
TerminateProcess(pi.hProcess, 0);
WaitForSingleObject(pi.hProcess, 2000);
CloseHandle(pi.hThread);
CloseHandle(pi.hProcess);
kill_stray_mock_games();
};
Sleep(800); // window + audio client up
// IPC + the primary audio ring the hook produces into.
SharedMemory shm;
if (!shm.create(shared_memory_name(pi.dwProcessId), sizeof(SharedBlock)))
{
std::printf("Could not create IPC block -- skipping.\n");
cleanup();
return 0;
}
auto* block = shm.as<SharedBlock>();
block->version = kProtocolVersion;
block->pad_count = 0;
block->sequence.store(0, std::memory_order_relaxed);
// Only the audio subsystem.
for (std::uint32_t s = 0; s < HookSubsys_Count; ++s)
{
const bool off = s != HookSubsys_Audio;
block->control.subsystem_disabled[s].store(off ? 1u : 0u, std::memory_order_release);
}
block->magic = kProtocolMagic;
SharedMemory ring_shm;
if (!ring_shm.create(audio_ring_name(pi.dwProcessId), audio_ring_total_size(kAudioRingCapacity)))
{
std::printf("Could not create audio ring -- skipping.\n");
cleanup();
return 0;
}
auto* ring = ring_shm.as<AudioRingHeader>();
audio_ring_init(*ring, kAudioRingCapacity);
// Leave capture_enabled = 0: we want the stream audible (so loopback hears it) and still being
// MEASURED (so the verify tap fires), exactly the window verify_stream_format targets.
if (!inject_retry(pi.dwProcessId))
{
std::printf("Could not inject coop_hook.dll -- skipping.\n");
cleanup();
return 0;
}
Sleep(500); // let the hook attach + the pre-existing render client register as a guess
// Run the real verifier: co-capture hook (pre-mix) + loopback (post-mix) and correlate.
const FormatVerification fv = verify_stream_format(pi.dwProcessId, ring, /*window_ms=*/1400);
std::printf(" verify: ok=%d rate=%u score=%.3f\n", fv.ok ? 1 : 0, fv.rate, fv.score);
if (!fv.ok && fv.rate == 0 && fv.score == 0.0)
{
// No audio endpoint, or no usable audio captured (e.g. the mock's WASAPI client never
// started on this machine) -> treat as a skip rather than a failure.
std::printf(" no usable co-capture (no endpoint / silent) -- skipping audio_verify_test.\n");
if (com)
{
CoUninitialize();
}
cleanup();
return 0;
}
check(fv.ok, "verifier confidently correlated the two capture paths");
check(fv.rate == game_rate, "verifier recovered the game's true rate (not the device rate)");
if (com)
{
CoUninitialize();
}
cleanup();
if (g_failures == 0)
{
std::printf("PASS audio_verify_test\n");
return 0;
}
std::printf("FAILED audio_verify_test (%d)\n", g_failures);
return 1;
}