diff --git a/README.md b/README.md index 76323bf..208917d 100644 --- a/README.md +++ b/README.md @@ -273,60 +273,44 @@ source), and click away from the game to confirm focus spoofing keeps it running Non-obvious things that cost time and constrain the design: -- **RPT only streams the *focused* window.** The game therefore can't hold focus - itself; the hook spoofs focus (`GetForegroundWindow` / `GetActiveWindow` / - `GetFocus`, plus swallowing window-deactivation messages) so the game keeps - polling and rendering while the host owns the real OS focus. -- **Run target games windowed or borderless, never exclusive fullscreen.** - Exclusive fullscreen minimizes on focus loss (defeating the focus spoof) and - can't be window-captured. While unfocused the game gets no OS keyboard/mouse — - only the forwarded controller. -- **`ActivateAudioInterfaceAsync` requires an *agile* completion handler.** If the +- **RPT only streams the *focused* window.** The game can't hold focus itself, so + the hook spoofs it (`GetForegroundWindow` / `GetActiveWindow` / `GetFocus` + + swallowing deactivation messages) to keep the game polling and rendering while + the host owns real OS focus. +- **Run target games windowed or borderless, never exclusive fullscreen** — + exclusive fullscreen minimizes on focus loss (defeating the spoof) and can't be + window-captured. While unfocused the game gets no OS keyboard/mouse, only the + forwarded pad. +- **WGC captures occluded windows but not minimized ones.** +- **Process-loopback capture doesn't mute the source.** Capturing a process's + render doesn't stop it reaching the speakers, so the no-echo path instead injects + a WASAPI render-hook that copies each buffer then releases it with + `AUDCLNT_BUFFERFLAGS_SILENT`; loopback stays as the (echoing) fallback. +- **`ActivateAudioInterfaceAsync` needs an *agile* completion handler.** If the handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected **synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of - COM apartment, MFStartup, device path, or activation params. (WRL/wil-based - samples hide this because they make the handler agile for you.) Process loopback - also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`. -- **WGC captures occluded windows but not minimized ones.** The game may sit - behind the host window, but must not be minimized. -- **Process-loopback capture doesn't mute the source** — capturing a process's - render does not stop it reaching the speakers. That's why the echo fix instead - injects a WASAPI render-hook that silences the game's own buffer - (`AUDCLNT_BUFFERFLAGS_SILENT`) after copying it for the mirror; loopback stays - as the fallback. -- **COM has no exports, so the render-hook walks vtables — and the indices are - easy to miscount.** All instances of a COM coclass share one vtable, so hooking - one object's method (resolved by frozen-ABI vtable index) catches every - instance. But the indices must be exact: `IAudioClient::GetService` is **14**, - not 13 — `SetEventHandle` (13) sits between `Reset` and `GetService`. Count the - full interface (including every inherited `IUnknown`/base method) when adding a - new COM hook. -- **SafetyHook's `call()` is `__cdecl` on x86 — use `stdcall()` for `__stdcall` - targets.** `InlineHook::call()` invokes the trampoline through a pointer with the - compiler's default convention, which is `__cdecl` on 32-bit. Most things we hook - are `__stdcall` (COM methods like `IDXGISwapChain::Present` and the WASAPI render - interfaces, plus `WINAPI` `SwapBuffers`). On x64 every convention collapses to one, - so `call()` is fine; on x86 it double-cleans the stack → ESP imbalance → an - instant crash (Debug builds surface it as **Run-Time Check Failure #0**). This - froze 32-bit games (Slaps and Beans) the moment the Present hook ran. Always call - trampolines with the matching convention — `stdcall()` for these — which is a - no-op on x64. The XInput/focus hooks dodged it only because they never call the - trampoline (they return synthesized data). -- **Hook COM methods by swapping the vtable entry, not by inline-patching the - function — on x86.** Inline hooking relocates the target's overwritten prologue - into a trampoline. Some x86 prologues defeat that: MMDevApi/AudioSes methods open - with `push ebp; mov ebp,esp; and esp,-8` (dynamic stack alignment) and read their - arguments **EBP-relative**. SafetyHook's relocated copy leaves EBP wrong, so the - original ran with garbage arguments and faulted — this crashed 32-bit FMOD games - (Slaps and Beans) the instant audio init flowed through the hook, *after* the - `stdcall()` fix above. The robust fix is vtable-entry hooking: `VirtualProtect` the - shared vtable slot, overwrite the function pointer, call the saved original - directly. No code patching, no trampoline, pristine stack regardless of prologue. - The audio hooks use this; inline hooking is fine for `Present`/`SwapBuffers`, whose - prologues relocate cleanly. (One swap covers every instance — a coclass shares one - vtable.) Guarded by `audio_hook_test_x86`. -- **Steam Input init suppresses XInput.** Initializing the Steam Input API turns on - Steam's in-process XInput interception, which hides controllers from - `XInputGetState` unless they're bound to the running appid's action set — - defaulting to it silently broke input forwarding. XInput is the primary path; - Steam Input is opt-in. + apartment, device path, or activation params. (WRL/wil samples make the handler + agile for you.) Process loopback also needs the Win10 20H1 headers + (`NTDDI_VERSION ≥ 0x0A00000B`). +- **COM methods have no exports, so hooks walk vtables by frozen-ABI index — count + exactly.** All instances of a coclass share one vtable, so hooking one object's + slot catches every instance; but `IAudioClient::GetService` is **14**, not 13 + (`SetEventHandle` sits at 13 between `Reset` and `GetService`). Count every + inherited `IUnknown`/base method when adding a hook. +- **SafetyHook on x86 has two traps that froze 32-bit Slaps and Beans.** (1) + `InlineHook::call()` invokes the trampoline as `__cdecl`, but most targets are + `__stdcall` (COM methods like `IDXGISwapChain::Present`, the WASAPI interfaces, + `WINAPI` `SwapBuffers`); on 32-bit that double-cleans the stack → ESP imbalance → + crash (Debug: **Run-Time Check Failure #0**). Use **`stdcall()`** (a no-op on + x64). (2) Don't *inline-hook* COM methods on x86 at all: MMDevApi/AudioSes + prologues do `push ebp; mov ebp,esp; and esp,-8` and read args **EBP-relative**, + which SafetyHook's trampoline relocation breaks (the original then runs with + garbage args and faults). Hook COM methods by **swapping the vtable entry** + instead (`VirtualProtect` the slot, overwrite the pointer, call the saved + original) — no code patching, pristine stack regardless of prologue. Inline + hooking stays fine for `Present`/`SwapBuffers` (clean prologues). Guarded by the + x86 hook tests. +- **Steam Input init suppresses XInput.** Initializing Steam Input turns on Steam's + in-process XInput interception, which hides controllers from `XInputGetState` + unless they're bound to the running appid's action set — defaulting to it + silently broke forwarding. XInput is primary; Steam Input is opt-in. diff --git a/docs/audio-render-hook-plan.md b/docs/audio-render-hook-plan.md deleted file mode 100644 index 2f6f75a..0000000 --- a/docs/audio-render-hook-plan.md +++ /dev/null @@ -1,281 +0,0 @@ -# Plan: fix the local audio echo via an injection render-hook (Option B) - -Status: **implemented and validated against a real already-playing game.** M1–M4 -are built and covered by `audio_ring_test` + `audio_hook_test`. The render-hook -now correctly captures a game's audio when injected into an already-running, -already-playing process (verified with Phantom Brave, `coop_audio_probe`: the -pre-existing 48 kHz/2ch/float render client is detected, real non-silent audio -reaches the ring, zero overruns when consumed). The remaining step is the human -end-to-end with a guest over RPT: no local echo, guest hears audio. - -Corrections applied during/after implementation: the `IAudioClient::GetService` -vtable index is **14**, not 13 (`SetEventHandle` is 13); the inner hooks must be -installed **proactively** (see below), not only reactively — the original -reactive-only design captured nothing on late injection, which is the normal case; -and the captured format must be **(re)published when the ring attaches**, not only -at stream registration, or the full app falls back to the echo (see below). - -## Lesson: reactive-only hooking fails on late injection (the bug that broke it) - -The first cut installed the IAudioClient / IAudioRenderClient hooks only when the -*game* called `IMMDevice::Activate` → `GetService`. But the tool injects into a -game that is already running and already playing — its render client was created -before we attached, so those calls never fire again. Result: no stream is ever -registered, nothing is captured, and the host always falls back to process -loopback (the echo). Every game tested fell back. - -Fix (commit `c7be4ee`): at anchor time, build our **own** probe `IAudioClient` + -`IAudioRenderClient` with raw calls and hook `GetBuffer`/`ReleaseBuffer` (plus -`Initialize`/`GetService`) on *their* vtables. Because every instance of a COM -coclass shares one vtable, this patches the shared vtables and intercepts the -game's pre-existing render client too. The first render client seen actively -releasing buffers is adopted as the primary on the audio thread (try-lock, -one-time) using the device **mix format** as its assumed format — we never saw its -`Initialize`, and shared-mode clients overwhelmingly use the mix format. Streams -created *after* injection still register via the reactive path with their real -format. - -Sub-lesson — ordering / self-deadlock: create the probe objects *before* -installing the `Activate` hook. If `Activate` is hooked first, the probe's own -`device->Activate` re-enters `hk_Activate` → `install_audioclient_hooks`, which -blocks on the setup mutex the installer already holds — freezing the worker thread -(and any game thread that later calls `Activate`, which crashed the game). - -## Lesson: publish the format when the ring attaches, not only at registration - -A second late-binding bug only showed up in the full app, never in the probe. -The host creates the audio ring **when the operator toggles audio mirroring on**, -which is *after* injection — so the hook registers the game's primary stream while -`g_ring` is still null, and `register_render_client_locked` skips publishing the -format (there's no ring to publish to). When the ring later attaches via -`set_audio_ring`, the already-registered stream's format was never re-published, -so `format_valid` stayed 0, the host's `wait_for_format` timed out, and it fell -back to loopback (the echo) on every game. The in-process probe created the ring -*before* injecting, so it never reproduced this — `coop_audio_probe` now defaults -to creating the ring ~1.5 s **after** injecting to match the app. - -Fix (commit follows): the hook stores the primary stream's format and -`republish_audio_format()` (re)publishes it whenever a ring is attached but has no -format yet — called from `set_audio_ring` and once per worker tick (the latter -also covers the host re-initializing the ring on a mirror re-toggle, which clears -`format_valid`). - -## Problem - -`coop_host.exe` mirrors the game's audio so Steam Remote Play Together (which -streams only the host process's own audio session) carries game sound to guests. -Today the host captures the game via **WASAPI process loopback** and re-renders it -on the default endpoint. The game *also* still plays locally, so the operator's -default endpoint carries two copies of the audio ("double audio" / echo). The -guest hears one copy (the host re-render); only the local operator hears it twice. - -The fix must suppress the game's *direct* local playback **without** killing the -signal that feeds the host re-render. - -## Why the cheaper options are out - -- **Option A — per-session mute (`ISimpleAudioVolume` / the Volume Mixer).** - Tested manually: muting or mixing down the game's session also mutes/mixes down - the mirror. The Windows Volume Mixer *is* the `ISimpleAudioVolume` per-session - API (same calls), so there is no API-vs-mixer difference to exploit. This proves - the process-loopback tap sits **downstream** of the session volume gate. - **Conclusively out.** -- **Option C — redirect the game to a separate sink (`IAudioPolicyConfig`).** The - per-app endpoint redirect is real and usable (it backs Windows' "App volume and - device preferences"), but it only *routes* — it needs a destination endpoint - that is silent to the operator yet capturable by us, i.e. a virtual sink. - **Windows has no public API to instantiate a virtual audio endpoint at runtime - without a driver** (endpoints are driver-backed; every "virtual cable" ships an - installed signed kernel driver). The desired *ad-hoc, driverless, auto-removed* - virtual device does not exist. **Out.** - -The only place to both grab the audio *and* stop it reaching the shared endpoint -is **before it leaves the game process** — i.e. injection, which we already do for -input. That is Option B. - -## Approach (Option B) - -In `coop_hook.dll`, hook the game's WASAPI render path: - -``` -IMMDevice::Activate(IID_IAudioClient) → IAudioClient -IAudioClient::Initialize(format) ← capture WAVEFORMATEX here -IAudioClient::GetService(IID_IAudioRenderClient)→ IAudioRenderClient (= a "stream") - loop: GetBuffer(n,&p) → game writes PCM → ReleaseBuffer(n,flags) -``` - -On `ReleaseBuffer`, copy the just-written frames into a shared audio ring (the host -re-renders them for RPT), then release with `AUDCLNT_BUFFERFLAGS_SILENT` so WASAPI -emits silence locally. Result: operator hears one copy (the host re-render), guest -hears one copy, no virtual device, no echo. - -### Reaching the vtables - -COM methods aren't exports, so we resolve them via vtable indices (frozen COM ABI) -and inline-hook the resolved addresses with SafetyHook (same engine as the XInput -hooks): - -- **Anchor (only shared-vtable assumption):** the worker thread `CoCreateInstance`s - its *own* `IMMDeviceEnumerator`, gets the default render `IMMDevice`, and hooks - `IMMDevice::Activate` (vtable idx 3). All `IMMDevice` instances in the process - share that vtable (single coclass), so the game's `Activate` calls are caught. -- **Everything else is hooked off the live pointers the game received** (no further - assumptions), install-once guarded: - - `Activate` hook → if IID is `IAudioClient`/`2`/`3`, hook `Initialize` (idx 3) - and `GetService` (idx 14) on that object. - - `Initialize` hook → capture the `WAVEFORMATEX` (rate / channels / bits / tag). - Fallback for games using `IAudioClient3::InitializeSharedAudioStream`: read the - format via the original `GetMixFormat` in the `GetService` hook. - - `GetService` hook → if IID is `IAudioRenderClient`, register the stream and hook - its `GetBuffer` (idx 3) and `ReleaseBuffer` (idx 4). - -Vtable indices: `IMMDevice::Activate`=3; `IAudioClient::Initialize`=3, -`GetService`=14 (after `SetEventHandle`=13); `IAudioRenderClient::GetBuffer`=3, -`ReleaseBuffer`=4. - -The worker thread `CoInitializeEx(MTA)` for the lifetime of the DLL (needed for the -enumerator instance); installs are retried on the existing 250 ms worker tick, like -the XInput / focus installs. - -### Capture + silence - -- **GetBuffer hook:** call original; stash `pData` + `numFrames` thread-local (the - pair is always called on one thread, never nested). -- **ReleaseBuffer hook:** act only when `this == primary render client`. If not - already silent: `memcpy` `numFrames × nBlockAlign` into the ring, then call the - original `ReleaseBuffer(numFrames, flags | AUDCLNT_BUFFERFLAGS_SILENT)`. Copy - happens before the original call (buffer is valid until release). memset-to-zero - is kept as a fallback if any driver mishandles the SILENT flag. -- No allocation and no locks on the game's audio thread — only a lock-free ring - write. - -### Hooks always install; only copy+silence is gated - -The Activate/Initialize/GetService/GetBuffer/ReleaseBuffer hooks install whenever -the DLL is injected, so **stream counting works even when audio mirroring is off**. -A `capture_enabled` flag (host-owned) gates *only* the copy+silence behavior in -`ReleaseBuffer`. Flag off → audio passes through untouched (game audible locally, -no mirror, but streams are still counted for the debug view). - -## Audio IPC ring (new shared mapping) - -The 20-byte `SharedBlock` can't hold PCM, so a **separate named mapping** -`Local\coop_audio_`, ~1 MB (>1 s at typical formats): - -``` -AudioRing header: magic, version, capture_enabled, - format_valid, format_generation, - sample_rate, channels, bits, format_tag, block_align, - capacity, atomic write_pos, read_pos, - frames_produced, overruns; // + reserved tail -data[capacity]: PCM ring -``` - -- **Lock-free SPSC** (hook produces, host consumes): free-running 64-bit - `write_pos` / `read_pos`, release on publish / acquire on read — same - cross-process atomic model as the existing input seqlock. On full → drop the - packet and bump `overruns` (host should always keep up). -- **Format handshake:** hook sets the format fields + `format_valid` once; the host - spins on `format_valid` before creating its render client. `format_generation` - is in the layout now so a future mid-session device re-init is forward-compatible - (v1 handles format-set-once). -- **Ownership / toggle:** the **host creates the mapping and owns `capture_enabled`**; - the hook holds it open and copies+silences only while the flag is set. Toggling - the Audio panel checkbox just flips the flag — no injection churn, no teardown - races. Closing the host's handle while the hook holds it is safe (the section is - refcounted by the OS). - -## Stream-count debug visualization (required) - -Goal: easily see when a game emits more than one render stream, since v1 captures -only the first. These diagnostics live in the **always-present** `HookStatus` -back-channel in `SharedBlock` (not the audio ring), so the count is visible even -before/without enabling audio mirror. This bumps `kProtocolVersion` (3 → 4); new -members go at the **end** of `HookStatus` so existing offsets never shift. - -Add to `HookStatus`: - -``` -uint32 audio_streams_seen; // distinct render clients ever created -AudioStreamInfo audio_streams[kMaxAudioStreams]; // kMaxAudioStreams = 4 - -struct AudioStreamInfo { - uint32 is_primary; // 1 = the stream we capture - uint32 sample_rate; - uint16 channels; - uint16 bits; - uint32 format_tag; - uint64 frames_rendered; // cumulative; host derives "live vs idle" from deltas -}; -``` - -- The hook assigns each distinct `IAudioRenderClient` a slot, marks the first as - `is_primary`, and bumps `frames_rendered` on each `ReleaseBuffer`. -- **Audio panel UI:** show `Render streams: N`, and a small table — one row per - stream with format + a frames counter, the primary row tagged, and rows whose - `frames_rendered` is advancing highlighted as live. This makes a multi-stream - game obvious at a glance. -- If `audio_streams_seen > kMaxAudioStreams` (more streams than slots), still show - the total count and note the overflow. - -## Host changes - -`AudioMirror` keeps its event-driven render client and prime/underrun logic almost -verbatim; only the **source** changes from the `ProcessLoopbackCapture` callback to -popping the shared audio ring. The host initializes its render client with the -*game's* published format and lets shared-mode WASAPI convert game-format → -endpoint. `audio_panel` gains a source indicator (Hooked vs Loopback) plus the -stream table above. - -## Fallback — never regress - -If the render hooks don't install, or `format_valid` never appears within ~1 s -(unusual COM setup, an uncaught backend, anti-cheat, etc.), the host -**automatically falls back to the existing process-loopback path** (works, but with -the echo) and surfaces that in the panel. Worst case = today's behavior. -`host/src/audio/process_loopback_capture.*` stays in the tree as the fallback. - -## Known limitations (v1) - -- **Primary stream only.** With multiple simultaneous render streams we capture + - silence only the first; secondaries stay local and unmirrored (graceful degrade, - no corruption). The debug view exists precisely to detect this. Per-stream rings - + host-side mix is a follow-up if it ever matters. -- **`ActivateAudioInterfaceAsync` activation path not hooked in v1** (most games use - `IMMDevice::Activate`; the rest hit the loopback fallback). -- Exclusive-mode and DirectSound/XAudio2 backends still bottom out in an - `IAudioClient`, so they're covered; truly exotic backends fall back. - -## Work breakdown - -| File | Change | Size | -| --- | --- | --- | -| `common/include/coop/audio_ring.hpp` | NEW — ring layout + lock-free push/pop + format/flag | ~120 ln | -| `common/include/coop/protocol.hpp` | add `AudioStreamInfo` + audio diag fields to `HookStatus`; bump version 3→4 | small | -| `hook/src/audio_hook.{hpp,cpp}` | NEW — vtable discovery, install/remove, the 4 hooks, ring producer, stream counting | ~350 ln (the risk) | -| `hook/src/dllmain.cpp` | open `coop_audio_`, install audio hooks in worker loop | small | -| `hook/CMakeLists.txt` | link `ole32 mmdevapi`; COM-init the worker thread | small | -| `host/src/audio/audio_loopback.{cpp,hpp}` | "hooked ring" source mode + create mapping + auto-fallback | medium | -| `host/src/audio_panel.{cpp,hpp}` | source indicator + stream-count debug table | small | -| `host/CMakeLists.txt` | new sources | small | -| `tests/audio_hook_test.cpp` | NEW — in-process: render a tone, install hooks, assert ring gets non-silent frames AND local output went silent; assert stream count == 1 | ~150 ln | - -The in-process self-test is the key de-risker: it exercises vtable discovery + -GetBuffer/ReleaseBuffer interception with no game and no second Steam account, the -same way `hook_selftest` covers the XInput core. - -**Estimate: ~1.5–2 days.** `audio_hook.cpp` is the only real risk; the rest mirrors -patterns already in the repo. - -## Milestones (build order) - -1. `audio_ring.hpp` + `HookStatus` audio fields + host ring consumer + ring unit - test (no hook yet). -2. `audio_hook.cpp` + the in-process tone → ring → silenced self-test, including the - stream count. ← *proves the concept* -3. Wire into `dllmain` + host "hooked" mode + automatic loopback fallback. -4. Audio panel UI: source indicator + stream-count debug table. -5. Manual end-to-end in a real game: confirm no local echo, guest still hears audio, - and the stream count reads correctly. - -If milestone 2 passes, the rest is plumbing.