Audio: operator re-measure + format override (host<->hook op channel)
Add a per-stream op channel in AudioRingHeader (op_seq + op_* fields, version 2): the host posts re-measure / override commands, the hook applies them and re-publishes (bumping format_generation), and the host rebuilds its render client live on the change. The Audio panel (under Debug details) gains a "Re-measure rate" button and a rate/channels/bit-depth/format override -- for when detection is wrong or the channels/bit-depth were unrecoverable. Also add a debug-only IPC test harness (-DCOOP_TEST_HARNESS, off by default, absent from the shipped host): a file-based command channel that drives the host's real UI code paths (inject / audio / re-measure / override / screenshot / status) for scripted validation, instead of unreliable synthetic mouse input. Used to validate live: late-attach to coop_tone@44100 -> measured 44100, promoted to hooked; override -> 2ch state, re-measure -> reconverge. Trim the README roadmap to what's left; document the harness + rate_estimator_test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
105
README.md
105
README.md
@@ -77,8 +77,10 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
|
||||
device would be mirrored with the wrong layout (garbled audio) on the hooked path —
|
||||
but never an over-read/crash (a `VirtualQuery` clamp guards the copy), and the
|
||||
loopback fallback is always format-correct. The Audio panel shows each stream's
|
||||
format provenance (*known* / *measuring* / *measured rate (ch/bits assumed)*) so the
|
||||
assumption is visible. Streams created *after* injection are captured exactly.
|
||||
format provenance (*known* / *measuring* / *measured rate* / *low-confidence* /
|
||||
*override*) so the assumption is visible, and (under Debug details) lets the operator
|
||||
**re-measure** the rate or **override** the format when the guess is wrong. Streams
|
||||
created *after* injection are captured exactly.
|
||||
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
|
||||
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
|
||||
for RPT; F2 frees the operator cursor; **F10 saves a PNG screenshot** (back buffer,
|
||||
@@ -88,71 +90,30 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
|
||||
|
||||
### Planned (next up)
|
||||
|
||||
**Audio format reliability.** The hooked audio path can mis-detect a pre-existing
|
||||
stream's sample rate, or fall back to loopback with no explanation. Two root causes:
|
||||
the rate measurement is fragile, and late injection forces the guess path in the
|
||||
first place.
|
||||
|
||||
- *Why the rate is sometimes wrong.* The measurement window is only ~200 ms. WASAPI
|
||||
delivers audio in quantized ~10 ms buffers, so one extra buffer at a window edge is
|
||||
a ~5% error (e.g. 44100 → ~46205 Hz). The snap-to-standard tolerance is ±2%, so a
|
||||
~5%-off value snaps to *nothing* and is published verbatim instead of rejected, and
|
||||
it's single-shot (no averaging). Fix: **measure over a longer window (~1–1.5 s),
|
||||
sample the (frames, QPC) endpoints atomically, require consensus across a few
|
||||
windows, and refuse to publish a rate that doesn't land near a standard rate** —
|
||||
keep measuring (or flag low-confidence) rather than committing a bogus value.
|
||||
- *Why it falls back to loopback silently.* The host waits a fixed 1 s for the hook
|
||||
to publish a format; the guess+measure path needs ~400–700 ms of *continuous* audio
|
||||
after the ring attaches, which a momentarily-quiet game can miss. Once on loopback
|
||||
it never retries, and the UI shows no reason. Fix: **keep the rings live and
|
||||
auto-promote to the hooked path whenever the hook later publishes a format, surface
|
||||
the concrete fallback reason** (no format in time / not renderable / hook off) in the
|
||||
Audio panel + log, and **flag a low-confidence rate (non-standard, didn't snap) in
|
||||
red** so a bad measurement is obvious at a glance.
|
||||
- **Re-measure button.** A debug action (Audio panel) to re-run a stream's rate
|
||||
measurement on demand. Needs a small host→hook control surface: add operator-control
|
||||
fields to the reserved area of `AudioRingHeader` (per-stream, already shared both
|
||||
ways) — a bumped `op_seq` plus a "re-measure" request. The hook resets that stream's
|
||||
measurement window, returns it to *Measuring*, clears `format_valid`, and republishes
|
||||
once it reconverges; the host re-inits its render client when `format_generation`
|
||||
bumps (so the corrected rate takes effect live).
|
||||
- **Per-game format overrides (persisted), incl. the unmeasurable channels/bit-depth.**
|
||||
A dropdown/inputs in the Audio panel — **shown only under the Debug details flag**,
|
||||
per stream — to set sample rate, channels, bit-depth, and PCM/float when detection is
|
||||
wrong or unrecoverable, **keyed by game image name and persisted** to a small store
|
||||
next to the exe, so a known-bad game is auto-corrected on its next launch. A
|
||||
**rate-only** override is host-side only (re-init
|
||||
the render loop at the operator's rate — the bytes are already framed correctly); a
|
||||
**channels/bits** override goes through the same `AudioRingHeader` op channel so the
|
||||
*hook* re-frames its capture copy (`block_align`) and drops the over-read clamp, then
|
||||
the host re-inits on the `format_generation` bump. **A format caught exactly at
|
||||
`IAudioClient::Initialize` is itself saved as that game's override** (ground truth),
|
||||
so a later late-attach to the same game is corrected automatically; if that overwrites
|
||||
a previously-stored override that *differs*, log a warning.
|
||||
- **Per-game persisted audio overrides + auto-learn.** The Audio panel already lets
|
||||
the operator re-measure or override a stream's format (Debug details → *Fix the
|
||||
primary stream*), but the override is session-only. Persist overrides **keyed by game
|
||||
image name** to a small store next to the exe so a known-bad game is auto-corrected on
|
||||
its next launch, and **auto-save a format caught exactly at `IAudioClient::Initialize`
|
||||
as that game's override** (ground truth) so a later late-attach is corrected
|
||||
automatically. If a caught format overwrites a stored override that *differs*, log a
|
||||
warning.
|
||||
- **Auto re-attach the same game on relaunch (session-only).** A checkbox on the
|
||||
attached (or just-terminated) target, default off, *not* persisted, and **always
|
||||
visible — not gated behind Debug details** (it's the recommended recovery path, not a
|
||||
diagnostic). While it's on and the target has terminated, the host watches the process
|
||||
list for a process of the same image name and injects automatically the moment it
|
||||
reappears — built on the existing re-attach-by-image-name path, so there's no target
|
||||
to type (it's the same game that was just attached). The workflow this enables: if
|
||||
audio came out wrong (late attach forced the guess), tick the box, kill and relaunch
|
||||
the game, and the host re-attaches *early* — early enough to catch
|
||||
`IAudioClient::Initialize` and read the exact format, no guessing. This is a
|
||||
**dependable fallback, not the intended primary workflow**: hardened first-launch
|
||||
detection (above) should get it right without a relaunch; auto-re-attach is the
|
||||
reliable recovery when it doesn't, and never asks the operator to guess a format.
|
||||
- **Color-coded log levels.** The Log window should color warnings and errors (amber /
|
||||
red) so they stand out from routine lines. `LogRecord` already carries a
|
||||
(currently-unused) `level` field; add severity variants to the hook's `logf` and have
|
||||
the host color by level. Used by the override-overwrite warning and the fallback
|
||||
reasons above.
|
||||
|
||||
Ordering: measurement hardening + visible/red fallback reason + log levels first
|
||||
(fixes the bug outright for most cases, no protocol change) → the `AudioRingHeader` op
|
||||
channel (re-measure + channels/bits override; bumps `kAudioRingVersion`) → per-game
|
||||
override persistence + auto-saving caught formats → session auto-re-attach (host-only,
|
||||
extends the existing re-attach path).
|
||||
attached (or just-terminated) target, default off, *not* persisted, **always visible**
|
||||
(it's the recommended recovery path, not a diagnostic). While on and the target has
|
||||
terminated, the host watches the process list for the same image name and injects the
|
||||
moment it reappears — built on the existing re-attach-by-image-name path, so there's no
|
||||
target to type. Workflow: if audio came out wrong (late attach forced the guess), tick
|
||||
the box, kill and relaunch the game, and the host re-attaches *early* — early enough to
|
||||
catch `IAudioClient::Initialize` and read the exact format. A dependable fallback, not
|
||||
the intended primary path (hardened first-launch detection should get it right anyway).
|
||||
- **Mock game + stress-test suite.** A bare-bones test game that renders an animated
|
||||
(non-static) pattern — so dropped/duplicated/torn frames are obvious — with a
|
||||
selectable graphics backend (DX11/DX12 now; structured to add OpenGL/Vulkan) and a
|
||||
configurable audio output (rate/channels/bit-depth/format, like `coop_tone`). Then a
|
||||
comprehensive suite that drives it through the real capture + audio + hook paths:
|
||||
verify both capture backends, every audio variant, hook/unhook cycles, and that
|
||||
nothing races or crashes under stress.
|
||||
|
||||
### Future work
|
||||
|
||||
@@ -225,6 +186,11 @@ ctest --test-dir build -C Debug --output-on-failure
|
||||
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
|
||||
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
|
||||
soft-clip / encode for float32 + int16). No device needed.
|
||||
- **`rate_estimator_test`** — unit test of the robust sample-rate estimator (the fix for
|
||||
the wrong-rate bug). Feeds synthetic, adversarial render cadences and asserts it
|
||||
converges to the right standard rate, rejects burst windows (never commits to a wrong
|
||||
neighbour, incl. the real 46205 misread), flags a genuinely non-standard rate
|
||||
low-confidence instead of spinning, and ignores idle windows. Pure logic, no device.
|
||||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook's **format
|
||||
detection**, the part that gets pitch right. Using a shared configurable
|
||||
`ToneSource` (the same render helper `coop_tone` uses), it renders tones at a matrix
|
||||
@@ -283,6 +249,13 @@ artifacts; tests build into `bin/<config>/tests/`) and resolve `coop_hook.dll` f
|
||||
the root one level up, so run them from there. **Kill the game between runs** — the
|
||||
loaded DLL locks `coop_hook.dll` against the next rebuild.
|
||||
|
||||
A debug-only **test harness** drives the host overlay's own code paths (inject / enable
|
||||
audio / re-measure / override / screenshot / read state) without simulating mouse input,
|
||||
for scripted UI validation. Build it with `-DCOOP_TEST_HARNESS=ON` (off by default, so
|
||||
the shipped host never contains it); the host then reads one command line from
|
||||
`%TEMP%\coop_test_cmd.txt` and replies in `%TEMP%\coop_test_resp.txt`. See
|
||||
[`host/src/test_harness.hpp`](host/src/test_harness.hpp).
|
||||
|
||||
## Running the tool (manual, end-to-end)
|
||||
|
||||
This needs Steam, a donor game that supports Remote Play Together, and a second
|
||||
|
||||
Reference in New Issue
Block a user