Audio render-hook: document the echo fix (M1-M4 done, M5 manual)

- README: audio mirror is now the injected render-hook with process-loopback as
  automatic fallback; update Architecture/Limitations/Status/Roadmap and the
  manual run step (Source indicator + render-stream table). Add tests
  (audio_ring_test, audio_hook_test) and a lessons-learned note on COM vtable
  indices (IAudioClient::GetService is 14, not 13).
- plan doc: mark M1-M4 implemented/committed, M5 (real-game end-to-end) pending,
  and record the GetService index correction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-19 14:21:38 +02:00
parent 31390d8c15
commit 2f1c036320
2 changed files with 50 additions and 14 deletions

View File

@@ -23,7 +23,7 @@ XInput game becomes Remote-Play-Together-able.
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done | | Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done |
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done | | Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done |
| Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done | | Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done |
| Mirror audio | WASAPI process-loopback capture of the game, re-rendered on the host | `coop_host.exe` | done | | Mirror audio | Injected render-hook copies the game's WASAPI frames into a shared ring and silences the game locally (no echo); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` | done |
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done | | Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done |
## Limitations ## Limitations
@@ -38,9 +38,13 @@ XInput game becomes Remote-Play-Together-able.
DirectInput-only / RawInput-only games are not handled. DirectInput-only / RawInput-only games are not handled.
- **x64 only:** the host and hook DLL must match the game's bitness, and only x64 - **x64 only:** the host and hook DLL must match the game's bitness, and only x64
is built today. 32-bit games need the x86 hook + injector (see Roadmap). is built today. 32-bit games need the x86 hook + injector (see Roadmap).
- **Local audio echo:** process-loopback capture does not mute the game, so the - **Local audio echo (fixed via the hook; falls back otherwise):** when the
game's audio plays locally *and* the host re-renders it — the local machine render-hook is active it silences the game's local playback while mirroring it,
hears it twice. Guests hear it once. Fixing this is on the Roadmap. so there is no echo. If the hook can't attach or the game uses an
unhooked/exotic render path, the host automatically falls back to
process-loopback capture, which does *not* mute the game — so on the fallback
path the local machine still hears the audio twice (guests hear it once). The
Audio panel shows which path is active.
- **Debug-oriented UI:** the ImGui overlay is always visible and laid out for - **Debug-oriented UI:** the ImGui overlay is always visible and laid out for
diagnosing the pipeline, not for end use. It can't yet be toggled off. diagnosing the pipeline, not for end use. It can't yet be toggled off.
@@ -62,6 +66,15 @@ All phases below are implemented and verified.
- **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via - **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via
WASAPI process loopback and re-renders it on the default endpoint, so RPT WASAPI process loopback and re-renders it on the default endpoint, so RPT
carries game audio to guests. carries game audio to guests.
- **Audio render-hook (echo fix). ✅ (pending manual end-to-end)** The injected
hook intercepts the game's WASAPI render path (`IAudioRenderClient`), copies
the frames into a shared audio ring for the host to re-render, and releases the
game's buffer silenced — so the operator no longer hears the audio twice. The
host owns an enable flag and re-renders the game's format via `AUTOCONVERTPCM`;
if the hook doesn't publish a format in time it reverts to process loopback.
The Audio panel shows the active source and a render-stream-count debug table.
Validated in-process by `audio_hook_test`; the real-game pass is the remaining
manual step.
## Roadmap ## Roadmap
@@ -72,8 +85,6 @@ Future work, roughly in priority order:
- **Generalize the UI:** rework the panels from bug-specific debug readouts into - **Generalize the UI:** rework the panels from bug-specific debug readouts into
general-purpose status, and add broader debug info (latency, frame timing, general-purpose status, and add broader debug info (latency, frame timing,
per-channel stats). per-channel stats).
- **Fix the local audio echo:** mute the game's local render (or otherwise avoid
the double playback) while still capturing it for the mirror.
- **Present-hook video path:** capture the game's frames by hooking - **Present-hook video path:** capture the game's frames by hooking
`IDXGISwapChain::Present` in the injected DLL and sharing the backbuffer via a `IDXGISwapChain::Present` in the injected DLL and sharing the backbuffer via a
shared D3D11 texture, as a lower-latency / more stable alternative to WGC. shared D3D11 texture, as a lower-latency / more stable alternative to WGC.
@@ -119,10 +130,17 @@ ctest --test-dir build -C Debug --output-on-failure
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game, - **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
no controller needed). no controller needed).
- **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs
the hooks, renders a tone through WASAPI in the same process, and asserts the
COM vtables were discovered, the frames reached the ring (non-silent), the
primary stream was silenced, and exactly one render stream was counted. Skips
cleanly if the machine has no audio endpoint.
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI - **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the
shipping process-loopback capture receives its audio by PID. Skips cleanly if shipping process-loopback capture (the fallback path) receives its audio by
the machine has no audio endpoint. PID. Skips cleanly if the machine has no audio endpoint.
## Running the tool (manual, end-to-end) ## Running the tool (manual, end-to-end)
@@ -149,8 +167,11 @@ person/account to receive the stream.
the host window now shows a live, letterboxed copy of the game. the host window now shows a live, letterboxed copy of the game.
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**. 4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
(You'll hear the game twice locally; that's the known echo — guests hear it With the hook injected, **Source** shows **Hooked (no echo)** and the game's
once.) local playback goes silent while guests still hear it. If it shows **Loopback
(echo)** the hook's render path wasn't caught and you'll hear the game twice
locally (guests still hear it once). The **Render streams** table shows how
many WASAPI streams the game emits (v1 mirrors the first/primary).
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest 5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
sees the mirrored video, hears the audio, and that their controller drives the sees the mirrored video, hears the audio, and that their controller drives the
@@ -183,5 +204,15 @@ Non-obvious things that cost time and constrain the design:
also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`. also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`.
- **WGC captures occluded windows but not minimized ones.** The game may sit - **WGC captures occluded windows but not minimized ones.** The game may sit
behind the host window, but must not be minimized. behind the host window, but must not be minimized.
- **Process-loopback capture doesn't mute the source**, hence the local audio - **Process-loopback capture doesn't mute the source** — capturing a process's
echo — capturing a process's render does not stop it reaching the speakers. render does not stop it reaching the speakers. That's why the echo fix instead
injects a WASAPI render-hook that silences the game's own buffer
(`AUDCLNT_BUFFERFLAGS_SILENT`) after copying it for the mirror; loopback stays
as the fallback.
- **COM has no exports, so the render-hook walks vtables — and the indices are
easy to miscount.** All instances of a COM coclass share one vtable, so hooking
one object's method (resolved by frozen-ABI vtable index) catches every
instance. But the indices must be exact: `IAudioClient::GetService` is **14**,
not 13 — `SetEventHandle` (13) sits between `Reset` and `GetService`. Count the
full interface (including every inherited `IUnknown`/base method) when adding a
new COM hook.

View File

@@ -1,7 +1,12 @@
# Plan: fix the local audio echo via an injection render-hook (Option B) # Plan: fix the local audio echo via an injection render-hook (Option B)
Status: **scoped, not started.** This document is the green-lit design; implement Status: **implemented (M1–M4), pending manual end-to-end (M5).** Milestones 1–4
against it. below are built, committed, and covered by `audio_ring_test` + `audio_hook_test`
(in-process). The remaining step is M5: confirm in a real game with a guest that
there's no local echo, the guest still hears audio, and the stream count reads
correctly. This document is the green-lit design; the only correction applied
during implementation was the `IAudioClient::GetService` vtable index (14, not
13 — `SetEventHandle` is 13).
## Problem ## Problem