Capture every audio stream into its own ring and mix them on the host

Games with several concurrent WASAPI render streams (e.g. Spider-Man: Miles
Morales) only had their first ("primary") stream mirrored; the rest kept playing
locally and never reached the guest. Now the render-hook captures + silences EVERY
tracked stream into its own ring (coop_audio_<pid>[_<index>]), each published with
that stream's own detected format (Initialize when caught, else GetMixFormat -- the
per-stream format detection, now actually used per ring rather than only for the
primary). The host creates a ring per stream and mixes the same-format streams with
a soft clip (host/src/audio/audio_mix.hpp); streams whose format differs from the
primary are still silenced (no echo) but skipped from the mix (would need
resampling).

The single-stream case is byte-for-byte unchanged: when only one stream is active
the host passes it through without the mixer, so the common path has no overhead or
fidelity change.

Verified: new audio_mix_test covers the decode/sum/soft-clip/encode math (float32 +
int16); audio_hook_test (x64 + x86) still passes, guarding the primary
capture+silence path against regression; full build x64 + x86 clean; ctest x64
11/11, x86 3/3. Multi-stream mixing against a real multi-stream game needs a live
session to fully confirm.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 05:45:47 +02:00
parent dfd2be8c17
commit 784c31a9b5
10 changed files with 412 additions and 147 deletions

View File

@@ -29,7 +29,7 @@ and forwards guest controllers back into it.
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` |
| Mirror video (default) | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` |
| Mirror video (hooked) | Injected Present / OpenGL hook copies the backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` |
| Mirror audio | Injected render-hook copies the game's WASAPI frames into a shared ring and silences the game locally (no echo); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` |
| Mirror audio | Injected render-hook copies each of the game's WASAPI render streams into its own shared ring and silences the game locally (no echo); the host mixes the streams (soft-clipped); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` |
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel, video/audio/log shares) | `common/` |
The hooked video path has two producers: **Direct3D (DXGI)** hooks
@@ -72,23 +72,12 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
### Planned (next up)
Worked top-to-bottom: each item is a milestone with its own tests and commit, and
is removed from this list once done — so the top item is always next. The
self-verifiable tooling / UI / input items come first; the game-pipeline items that
need a real game (and Remote Play) to fully validate come last.
- **Multi-stream audio capture + mixing, with per-stream format detection.** Games
with several concurrent WASAPI render streams (e.g. Miles Morales) only get their
first ("primary") stream mirrored today; the rest keep playing locally and never
reach the guest. Capture every tracked render stream into its own shared ring,
silence each, and add a host-side mixer that resamples each ring to the render
format and sums them (with soft-clip). **Fold in real per-stream format
detection** here, since it touches the same hook + ring plumbing: a stream that
already existed when we injected is never seen at `Initialize`, so the hook
currently assumes the device **mix format** and a shared-mode stream opened at a
different format comes out wrong-pitched. Resolve each stream's true format (the
stream's own `Initialize` when caught, else the original `GetMixFormat`) so every
mixed ring is pitched correctly.
Nothing queued — the previous backlog (bin restructure, terminated/hung detection,
re-attach, window-based target picker, overlay auto-layout, Audio "live" column,
moving the synthetic-input toggle, mouse + keyboard forwarding, rumble forwarding,
per-backend input debug view, cursor release, capture metrics + latency, DX12 hooked
capture, multi-stream audio + per-stream formats) is all shipped. See Future work
for what's left.
### Future work
@@ -152,6 +141,8 @@ ctest --test-dir build -C Debug --output-on-failure
no controller needed).
- **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
soft-clip / encode for float32 + int16). No device needed.
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs
the hooks, renders a tone through WASAPI in the same process, and asserts the
COM vtables were discovered, the frames reached the ring (non-silent), the