CoopAllTheThings/docs/audio-render-hook-plan.md

Plan: fix the local audio echo via an injection render-hook (Option B)

Status: scoped, not started. This document is the green-lit design; implement against it.

Problem

coop_host.exe mirrors the game's audio so Steam Remote Play Together (which streams only the host process's own audio session) carries game sound to guests. Today the host captures the game via WASAPI process loopback and re-renders it on the default endpoint. The game also still plays locally, so the operator's default endpoint carries two copies of the audio ("double audio" / echo). The guest hears one copy (the host re-render); only the local operator hears it twice.

The fix must suppress the game's direct local playback without killing the signal that feeds the host re-render.

Why the cheaper options are out

• Option A — per-session mute (ISimpleAudioVolume / the Volume Mixer). Tested manually: muting or mixing down the game's session also mutes/mixes down the mirror. The Windows Volume Mixer is the ISimpleAudioVolume per-session API (same calls), so there is no API-vs-mixer difference to exploit. This proves the process-loopback tap sits downstream of the session volume gate. Conclusively out.
• Option C — redirect the game to a separate sink (IAudioPolicyConfig). The per-app endpoint redirect is real and usable (it backs Windows' "App volume and device preferences"), but it only routes — it needs a destination endpoint that is silent to the operator yet capturable by us, i.e. a virtual sink. Windows has no public API to instantiate a virtual audio endpoint at runtime without a driver (endpoints are driver-backed; every "virtual cable" ships an installed signed kernel driver). The desired ad-hoc, driverless, auto-removed virtual device does not exist. Out.

The only place to both grab the audio and stop it reaching the shared endpoint is before it leaves the game process — i.e. injection, which we already do for input. That is Option B.

Approach (Option B)

In coop_hook.dll, hook the game's WASAPI render path:

IMMDevice::Activate(IID_IAudioClient)          → IAudioClient
IAudioClient::Initialize(format)                ← capture WAVEFORMATEX here
IAudioClient::GetService(IID_IAudioRenderClient)→ IAudioRenderClient  (= a "stream")
  loop:  GetBuffer(n,&p) → game writes PCM → ReleaseBuffer(n,flags)

On ReleaseBuffer, copy the just-written frames into a shared audio ring (the host re-renders them for RPT), then release with AUDCLNT_BUFFERFLAGS_SILENT so WASAPI emits silence locally. Result: operator hears one copy (the host re-render), guest hears one copy, no virtual device, no echo.

Reaching the vtables

COM methods aren't exports, so we resolve them via vtable indices (frozen COM ABI) and inline-hook the resolved addresses with SafetyHook (same engine as the XInput hooks):

• Anchor (only shared-vtable assumption): the worker thread CoCreateInstances its own IMMDeviceEnumerator, gets the default render IMMDevice, and hooks IMMDevice::Activate (vtable idx 3). All IMMDevice instances in the process share that vtable (single coclass), so the game's Activate calls are caught.
• Everything else is hooked off the live pointers the game received (no further assumptions), install-once guarded:
  • Activate hook → if IID is IAudioClient/2/3, hook Initialize (idx 3) and GetService (idx 14) on that object.
  • Initialize hook → capture the WAVEFORMATEX (rate / channels / bits / tag). Fallback for games using IAudioClient3::InitializeSharedAudioStream: read the format via the original GetMixFormat in the GetService hook.
  • GetService hook → if IID is IAudioRenderClient, register the stream and hook its GetBuffer (idx 3) and ReleaseBuffer (idx 4).

Vtable indices: IMMDevice::Activate=3; IAudioClient::Initialize=3, GetService=14 (after SetEventHandle=13); IAudioRenderClient::GetBuffer=3, ReleaseBuffer=4.

The worker thread CoInitializeEx(MTA) for the lifetime of the DLL (needed for the enumerator instance); installs are retried on the existing 250 ms worker tick, like the XInput / focus installs.

Capture + silence

• GetBuffer hook: call original; stash pData + numFrames thread-local (the pair is always called on one thread, never nested).
• ReleaseBuffer hook: act only when this == primary render client. If not already silent: memcpy numFrames × nBlockAlign into the ring, then call the original ReleaseBuffer(numFrames, flags | AUDCLNT_BUFFERFLAGS_SILENT). Copy happens before the original call (buffer is valid until release). memset-to-zero is kept as a fallback if any driver mishandles the SILENT flag.
• No allocation and no locks on the game's audio thread — only a lock-free ring write.

Hooks always install; only copy+silence is gated

The Activate/Initialize/GetService/GetBuffer/ReleaseBuffer hooks install whenever the DLL is injected, so stream counting works even when audio mirroring is off. A capture_enabled flag (host-owned) gates only the copy+silence behavior in ReleaseBuffer. Flag off → audio passes through untouched (game audible locally, no mirror, but streams are still counted for the debug view).

Audio IPC ring (new shared mapping)

The 20-byte SharedBlock can't hold PCM, so a separate named mapping Local\coop_audio_<pid>, ~1 MB (>1 s at typical formats):

AudioRing header:  magic, version, capture_enabled,
                   format_valid, format_generation,
                   sample_rate, channels, bits, format_tag, block_align,
                   capacity, atomic<u64> write_pos, read_pos,
                   frames_produced, overruns;     // + reserved tail
data[capacity]:    PCM ring

• Lock-free SPSC (hook produces, host consumes): free-running 64-bit write_pos / read_pos, release on publish / acquire on read — same cross-process atomic model as the existing input seqlock. On full → drop the packet and bump overruns (host should always keep up).
• Format handshake: hook sets the format fields + format_valid once; the host spins on format_valid before creating its render client. format_generation is in the layout now so a future mid-session device re-init is forward-compatible (v1 handles format-set-once).
• Ownership / toggle: the host creates the mapping and owns capture_enabled; the hook holds it open and copies+silences only while the flag is set. Toggling the Audio panel checkbox just flips the flag — no injection churn, no teardown races. Closing the host's handle while the hook holds it is safe (the section is refcounted by the OS).

Stream-count debug visualization (required)

Goal: easily see when a game emits more than one render stream, since v1 captures only the first. These diagnostics live in the always-present HookStatus back-channel in SharedBlock (not the audio ring), so the count is visible even before/without enabling audio mirror. This bumps kProtocolVersion (3 → 4); new members go at the end of HookStatus so existing offsets never shift.

Add to HookStatus:

uint32 audio_streams_seen;                       // distinct render clients ever created
AudioStreamInfo audio_streams[kMaxAudioStreams]; // kMaxAudioStreams = 4

struct AudioStreamInfo {
  uint32 is_primary;       // 1 = the stream we capture
  uint32 sample_rate;
  uint16 channels;
  uint16 bits;
  uint32 format_tag;
  uint64 frames_rendered;  // cumulative; host derives "live vs idle" from deltas
};

• The hook assigns each distinct IAudioRenderClient a slot, marks the first as is_primary, and bumps frames_rendered on each ReleaseBuffer.
• Audio panel UI: show Render streams: N, and a small table — one row per stream with format + a frames counter, the primary row tagged, and rows whose frames_rendered is advancing highlighted as live. This makes a multi-stream game obvious at a glance.
• If audio_streams_seen > kMaxAudioStreams (more streams than slots), still show the total count and note the overflow.

Host changes

AudioMirror keeps its event-driven render client and prime/underrun logic almost verbatim; only the source changes from the ProcessLoopbackCapture callback to popping the shared audio ring. The host initializes its render client with the game's published format and lets shared-mode WASAPI convert game-format → endpoint. audio_panel gains a source indicator (Hooked vs Loopback) plus the stream table above.

Fallback — never regress

If the render hooks don't install, or format_valid never appears within ~1 s (unusual COM setup, an uncaught backend, anti-cheat, etc.), the host automatically falls back to the existing process-loopback path (works, but with the echo) and surfaces that in the panel. Worst case = today's behavior. host/src/audio/process_loopback_capture.* stays in the tree as the fallback.

Known limitations (v1)

• Primary stream only. With multiple simultaneous render streams we capture + silence only the first; secondaries stay local and unmirrored (graceful degrade, no corruption). The debug view exists precisely to detect this. Per-stream rings
  • host-side mix is a follow-up if it ever matters.
• ActivateAudioInterfaceAsync activation path not hooked in v1 (most games use IMMDevice::Activate; the rest hit the loopback fallback).
• Exclusive-mode and DirectSound/XAudio2 backends still bottom out in an IAudioClient, so they're covered; truly exotic backends fall back.

Work breakdown

File	Change	Size
common/include/coop/audio_ring.hpp	NEW — ring layout + lock-free push/pop + format/flag	~120 ln
common/include/coop/protocol.hpp	add AudioStreamInfo + audio diag fields to HookStatus; bump version 3→4	small
hook/src/audio_hook.{hpp,cpp}	NEW — vtable discovery, install/remove, the 4 hooks, ring producer, stream counting	~350 ln (the risk)
hook/src/dllmain.cpp	open coop_audio_<pid>, install audio hooks in worker loop	small
hook/CMakeLists.txt	link ole32 mmdevapi; COM-init the worker thread	small
host/src/audio/audio_loopback.{cpp,hpp}	"hooked ring" source mode + create mapping + auto-fallback	medium
host/src/audio_panel.{cpp,hpp}	source indicator + stream-count debug table	small
host/CMakeLists.txt	new sources	small
tests/audio_hook_test.cpp	NEW — in-process: render a tone, install hooks, assert ring gets non-silent frames AND local output went silent; assert stream count == 1	~150 ln

The in-process self-test is the key de-risker: it exercises vtable discovery + GetBuffer/ReleaseBuffer interception with no game and no second Steam account, the same way hook_selftest covers the XInput core.

Estimate: ~1.5–2 days. audio_hook.cpp is the only real risk; the rest mirrors patterns already in the repo.

Milestones (build order)

1. audio_ring.hpp + HookStatus audio fields + host ring consumer + ring unit test (no hook yet).
2. audio_hook.cpp + the in-process tone → ring → silenced self-test, including the stream count. ← proves the concept
3. Wire into dllmain + host "hooked" mode + automatic loopback fallback.
4. Audio panel UI: source indicator + stream-count debug table.
5. Manual end-to-end in a real game: confirm no local echo, guest still hears audio, and the stream count reads correctly.

If milestone 2 passes, the rest is plumbing.