Audio render-hook: document the echo fix (M1-M4 done, M5 manual)
- README: audio mirror is now the injected render-hook with process-loopback as automatic fallback; update Architecture/Limitations/Status/Roadmap and the manual run step (Source indicator + render-stream table). Add tests (audio_ring_test, audio_hook_test) and a lessons-learned note on COM vtable indices (IAudioClient::GetService is 14, not 13). - plan doc: mark M1-M4 implemented/committed, M5 (real-game end-to-end) pending, and record the GetService index correction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
55
README.md
55
README.md
@@ -23,7 +23,7 @@ XInput game becomes Remote-Play-Together-able.
|
||||
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done |
|
||||
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done |
|
||||
| Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done |
|
||||
| Mirror audio | WASAPI process-loopback capture of the game, re-rendered on the host | `coop_host.exe` | done |
|
||||
| Mirror audio | Injected render-hook copies the game's WASAPI frames into a shared ring and silences the game locally (no echo); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` | done |
|
||||
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done |
|
||||
|
||||
## Limitations
|
||||
@@ -38,9 +38,13 @@ XInput game becomes Remote-Play-Together-able.
|
||||
DirectInput-only / RawInput-only games are not handled.
|
||||
- **x64 only:** the host and hook DLL must match the game's bitness, and only x64
|
||||
is built today. 32-bit games need the x86 hook + injector (see Roadmap).
|
||||
- **Local audio echo:** process-loopback capture does not mute the game, so the
|
||||
game's audio plays locally *and* the host re-renders it — the local machine
|
||||
hears it twice. Guests hear it once. Fixing this is on the Roadmap.
|
||||
- **Local audio echo (fixed via the hook; falls back otherwise):** when the
|
||||
render-hook is active it silences the game's local playback while mirroring it,
|
||||
so there is no echo. If the hook can't attach or the game uses an
|
||||
unhooked/exotic render path, the host automatically falls back to
|
||||
process-loopback capture, which does *not* mute the game — so on the fallback
|
||||
path the local machine still hears the audio twice (guests hear it once). The
|
||||
Audio panel shows which path is active.
|
||||
- **Debug-oriented UI:** the ImGui overlay is always visible and laid out for
|
||||
diagnosing the pipeline, not for end use. It can't yet be toggled off.
|
||||
|
||||
@@ -62,6 +66,15 @@ All phases below are implemented and verified.
|
||||
- **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via
|
||||
WASAPI process loopback and re-renders it on the default endpoint, so RPT
|
||||
carries game audio to guests.
|
||||
- **Audio render-hook (echo fix). ✅ (pending manual end-to-end)** The injected
|
||||
hook intercepts the game's WASAPI render path (`IAudioRenderClient`), copies
|
||||
the frames into a shared audio ring for the host to re-render, and releases the
|
||||
game's buffer silenced — so the operator no longer hears the audio twice. The
|
||||
host owns an enable flag and re-renders the game's format via `AUTOCONVERTPCM`;
|
||||
if the hook doesn't publish a format in time it reverts to process loopback.
|
||||
The Audio panel shows the active source and a render-stream-count debug table.
|
||||
Validated in-process by `audio_hook_test`; the real-game pass is the remaining
|
||||
manual step.
|
||||
|
||||
## Roadmap
|
||||
|
||||
@@ -72,8 +85,6 @@ Future work, roughly in priority order:
|
||||
- **Generalize the UI:** rework the panels from bug-specific debug readouts into
|
||||
general-purpose status, and add broader debug info (latency, frame timing,
|
||||
per-channel stats).
|
||||
- **Fix the local audio echo:** mute the game's local render (or otherwise avoid
|
||||
the double playback) while still capturing it for the mirror.
|
||||
- **Present-hook video path:** capture the game's frames by hooking
|
||||
`IDXGISwapChain::Present` in the injected DLL and sharing the backbuffer via a
|
||||
shared D3D11 texture, as a lower-latency / more stable alternative to WGC.
|
||||
@@ -119,10 +130,17 @@ ctest --test-dir build -C Debug --output-on-failure
|
||||
|
||||
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
|
||||
no controller needed).
|
||||
- **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC
|
||||
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
|
||||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs
|
||||
the hooks, renders a tone through WASAPI in the same process, and asserts the
|
||||
COM vtables were discovered, the frames reached the ring (non-silent), the
|
||||
primary stream was silenced, and exactly one render stream was counted. Skips
|
||||
cleanly if the machine has no audio endpoint.
|
||||
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI
|
||||
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the
|
||||
shipping process-loopback capture receives its audio by PID. Skips cleanly if
|
||||
the machine has no audio endpoint.
|
||||
shipping process-loopback capture (the fallback path) receives its audio by
|
||||
PID. Skips cleanly if the machine has no audio endpoint.
|
||||
|
||||
## Running the tool (manual, end-to-end)
|
||||
|
||||
@@ -149,8 +167,11 @@ person/account to receive the stream.
|
||||
the host window now shows a live, letterboxed copy of the game.
|
||||
|
||||
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
|
||||
(You'll hear the game twice locally; that's the known echo — guests hear it
|
||||
once.)
|
||||
With the hook injected, **Source** shows **Hooked (no echo)** and the game's
|
||||
local playback goes silent while guests still hear it. If it shows **Loopback
|
||||
(echo)** the hook's render path wasn't caught and you'll hear the game twice
|
||||
locally (guests still hear it once). The **Render streams** table shows how
|
||||
many WASAPI streams the game emits (v1 mirrors the first/primary).
|
||||
|
||||
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
|
||||
sees the mirrored video, hears the audio, and that their controller drives the
|
||||
@@ -183,5 +204,15 @@ Non-obvious things that cost time and constrain the design:
|
||||
also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`.
|
||||
- **WGC captures occluded windows but not minimized ones.** The game may sit
|
||||
behind the host window, but must not be minimized.
|
||||
- **Process-loopback capture doesn't mute the source**, hence the local audio
|
||||
echo — capturing a process's render does not stop it reaching the speakers.
|
||||
- **Process-loopback capture doesn't mute the source** — capturing a process's
|
||||
render does not stop it reaching the speakers. That's why the echo fix instead
|
||||
injects a WASAPI render-hook that silences the game's own buffer
|
||||
(`AUDCLNT_BUFFERFLAGS_SILENT`) after copying it for the mirror; loopback stays
|
||||
as the fallback.
|
||||
- **COM has no exports, so the render-hook walks vtables — and the indices are
|
||||
easy to miscount.** All instances of a COM coclass share one vtable, so hooking
|
||||
one object's method (resolved by frozen-ABI vtable index) catches every
|
||||
instance. But the indices must be exact: `IAudioClient::GetService` is **14**,
|
||||
not 13 — `SetEventHandle` (13) sits between `Reset` and `GetService`. Count the
|
||||
full interface (including every inherited `IUnknown`/base method) when adding a
|
||||
new COM hook.
|
||||
|
||||
Reference in New Issue
Block a user