From 8059f0e488a5df9776e8ab99d24ff8709af0a933 Mon Sep 17 00:00:00 2001 From: BlackMark Date: Fri, 19 Jun 2026 10:27:57 +0200 Subject: [PATCH] docs: update README for Phases 0-2 done, add roadmap + lessons - Reflect audio mirror (Phase 2) and video mirror (Phase 1b) as implemented; drop stale "(current)"/"still ahead" framing and the broken docs/ link. - Add a Roadmap section for the planned work: toggle/generalize the debug UI, fix the local audio echo, Present-hook video path, Steam Input, x86 support. - Add Lessons learned (RPT focus requirement, no exclusive fullscreen, the ActivateAudioInterfaceAsync agile-handler gotcha, WGC occlusion, audio echo). - Document clangd/compile_commands setup and the CTest suite. Co-Authored-By: Claude Opus 4.8 --- README.md | 220 +++++++++++++++++++++++++++++------------------------- 1 file changed, 117 insertions(+), 103 deletions(-) diff --git a/README.md b/README.md index a66dfff..fb5ffab 100644 --- a/README.md +++ b/README.md @@ -15,17 +15,16 @@ window that is a live copy of the game's video + audio, and forwards the guests' input back into the real game. Steam's RPT captures the mirror window — so any XInput game becomes Remote-Play-Together-able. -See [`docs`](docs) and the in-repo plan for the full design. +## Architecture -## Architecture (target) - -| Concern | Mechanism | Where | -| --- | --- | --- | -| Receive guest input | Steam Input / XInput (RPT delivers guests to the focused window) | `coop_host.exe` | -| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | -| Mirror video | Windows Graphics Capture first; `IDXGISwapChain::Present` hook as the low-latency upgrade | host (+ hook) | -| Mirror audio | WASAPI process-loopback capture of the game, re-rendered | `coop_host.exe` | -| Host ↔ hook IPC | Named shared memory (seqlock for input, shared D3D11 texture for video) | `common/` | +| Concern | Mechanism | Component | Status | +| --- | --- | --- | --- | +| Receive guest input | XInput (RPT delivers guest pads to the focused window) | `coop_host.exe` | done | +| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done | +| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done | +| Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done | +| Mirror audio | WASAPI process-loopback capture of the game, re-rendered on the host | `coop_host.exe` | done | +| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done | ## Limitations @@ -37,32 +36,53 @@ See [`docs`](docs) and the in-repo plan for the full design. co-op/local-multiplayer titles without active anti-cheat. - **XInput only:** the game must read controllers via XInput (the common case). DirectInput-only / RawInput-only games are not handled. -- **Architecture match:** the host and hook DLL must match the game's bitness. - x64 is supported first; x86 support is a later phase (see the plan). +- **x64 only:** the host and hook DLL must match the game's bitness, and only x64 + is built today. 32-bit games need the x86 hook + injector (see Roadmap). +- **Local audio echo:** process-loopback capture does not mute the game, so the + game's audio plays locally *and* the host re-renders it — the local machine + hears it twice. Guests hear it once. Fixing this is on the Roadmap. +- **Debug-oriented UI:** the ImGui overlay is always visible and laid out for + diagnosing the pipeline, not for end use. It can't yet be toggled off. ## Status -**Phase 0 — donor spike. ✅ Validated.** `coop_host.exe` is a borderless D3D11 -window with an ImGui overlay listing every controller it sees. Confirmed -end-to-end: Steam RPT streams the window under a donor appid, and guest gamepads -arrive (with correct slot assignment) as XInput. +All phases below are implemented and verified. -**Phase 1a — input forwarding + focus spoofing (current).** The host injects -`coop_hook.dll`; the DLL hooks XInput (via SafetyHook) so the game reads the -forwarded controller state and *only* that state. The DLL also **spoofs focus** -(hooks `GetForegroundWindow`/`GetActiveWindow`/`GetFocus` and subclasses the game -window to swallow deactivation messages) so the game keeps running and polling -while the tool holds the real OS focus — required because Steam RPT only captures -the focused window. A hook→host status back-channel shows whether the hook is -attached and how fast the game is polling it. The in-process `hook_selftest` -validates the IPC + hook core without needing a game. +- **Phase 0 — donor spike. ✅** Confirmed Steam RPT streams an arbitrary + borderless window under a donor appid and routes guest gamepads into it as + XInput (correct slot assignment). This is the premise the whole tool rests on. +- **Phase 1a — input forwarding + focus spoofing. ✅** The host injects + `coop_hook.dll`; the DLL hooks XInput (SafetyHook) so the game reads the + forwarded pad state and *only* that state, and spoofs focus so the game keeps + running while the host holds the real OS focus. A hook→host status + back-channel reports attach state and poll rate. +- **Phase 1b — video mirror (WGC). ✅** The host captures the injected game's + window with Windows Graphics Capture and draws it letterboxed as its + background, so RPT streams a live mirror. +- **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via + WASAPI process loopback and re-renders it on the default endpoint, so RPT + carries game audio to guests. -**Phase 1b — video mirror (current).** The host captures the injected game's -window with Windows Graphics Capture and draws it letterboxed as its background, -so RPT streams a live mirror of the game. See the test steps below. +## Roadmap -Still ahead (scoped in the plan): a `Present`-hook capture path if WGC latency -disappoints, audio (WASAPI process loopback), and x86 support. +Future work, roughly in priority order: + +- **Make the tool usable, not just demonstrable:** toggle the debug overlay + on/off so the mirror window can be used clean. +- **Generalize the UI:** rework the panels from bug-specific debug readouts into + general-purpose status, and add broader debug info (latency, frame timing, + per-channel stats). +- **Fix the local audio echo:** mute the game's local render (or otherwise avoid + the double playback) while still capturing it for the mirror. +- **Present-hook video path:** capture the game's frames by hooking + `IDXGISwapChain::Present` in the injected DLL and sharing the backbuffer via a + shared D3D11 texture, as a lower-latency / more stable alternative to WGC. +- **Steam Input:** consume guest input through the Steam Input API directly + rather than XInput. +- **x86 support:** add an x86 build of `coop_hook.dll` plus an x86 injector + helper the x64 host spawns, so 32-bit games work (a 64-bit process can't + cleanly inject a 32-bit one). Detect target bitness with `IsWow64Process2`. + The shared-memory IPC layout is already fixed-width / bitness-stable. ## Building @@ -75,99 +95,93 @@ git submodule update --init --recursive cmake -S . -B build -G "Visual Studio 17 2022" -A x64 cmake --build build --config Debug -# output: bin/Debug/coop_host.exe +# output: bin/Debug/coop_host.exe (+ coop_hook.dll, test exes) ``` Third-party dependencies (Dear ImGui, SafetyHook) are git submodules under `third_party/`; the Steamworks SDK is vendored manually there when wired. No vcpkg / package manager is used. -## Phase 0: validating the donor-launch assumption +### clangd / IDE setup -This is a manual test — it needs Steam, a second person (or second account), and -a donor game that supports Remote Play Together. +The Visual Studio CMake generator does **not** emit `compile_commands.json`, so +clangd has no include paths and reports false errors. Run +[`gen-compile-commands.bat`](gen-compile-commands.bat) once (and after adding +sources or include dirs); it configures a parallel Ninja build in `build-clangd/` +that produces the database, which [`.clangd`](.clangd) points clangd at. clangd's +clang-cl driver resolves the MSVC / Windows SDK system includes on its own. -1. **Pick a donor game** you own that supports Remote Play Together (check the - "Remote Play Together" tag on its store page). The donor only needs RPT - support; it is never actually played. +## Tests -2. **Launch the host under the donor's appid.** Find the donor's appid (the - number in its store URL), then run: +```sh +ctest --test-dir build -C Debug --output-on-failure +``` + +- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game, + no controller needed). +- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI + sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the + shipping process-loopback capture receives its audio by PID. Skips cleanly if + the machine has no audio endpoint. + +## Running the tool (manual, end-to-end) + +This needs Steam, a donor game that supports Remote Play Together, and a second +person/account to receive the stream. + +1. **Launch the host under a donor appid.** Find the donor's appid (the number in + its store URL); the donor only needs RPT support and is never actually played: ```text "C:\Program Files (x86)\Steam\steam.exe" -applaunch "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe" ``` - The borderless overlay window should appear and Steam should consider the - donor "running" (green status / "Stop" button in the library). + The borderless window appears and Steam marks the donor "running". If the donor + ignores the trailing path, set the host as the donor's **Launch Options** + (`"D:\...\coop_host.exe" %command%`) or use a launcher like RemotePlayDetached. - > If the donor ignores the trailing path, set the host as the donor's - > **Launch Options** instead (`"D:\...\coop_host.exe" %command%` variants), - > or use a launcher such as RemotePlayDetached. Recording which method makes - > Steam attribute our window to the donor is the main deliverable of Phase 0. +2. **Start the real game** windowed or borderless (not exclusive fullscreen — see + Lessons learned). In the host's **Injection** panel, filter for the game's + `.exe`, select it, and click **Inject & Connect**. Watch **Hook status** for + **Attached**, a non-zero **XInput polled: N/s**, and **Focus spoof: active**. -3. **Start Remote Play Together** from the Steam friends list / overlay and invite - a friend (or a second machine/account). +3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** — + the host window now shows a live, letterboxed copy of the game. -4. **Verify on the guest side:** - - The guest sees the borderless overlay window streamed (not a black screen). - - The guest presses buttons on their controller and the corresponding slot in - the overlay lights up. This proves RPT routes guest input into our window - as XInput — the foundation the whole tool relies on. +4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**. + (You'll hear the game twice locally; that's the known echo — guests hear it + once.) -5. Press **Esc** in the host window to quit. +5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest + sees the mirrored video, hears the audio, and that their controller drives the + real game. -**If steps 2 and 4 both work, the core premise holds** and we proceed to Phase 1 -(window capture + input injection). If not, we revisit the donor-attribution -approach before building further. +Useful checks while developing without RPT: tick **Forward synthetic test input** +in the Injection panel to make the game move on its own (proving forwarding is the +source), and click away from the game to confirm focus spoofing keeps it running. -## Phase 1a: testing input forwarding locally +> Injection access error → run the host as administrator. "Target is 32-bit" → +> that game needs the x86 hook (see Roadmap). -This needs no RPT, donor, or second account — just the host, the hook, a -controller, and a target game. `coop_host.exe` and `coop_hook.dll` must sit in -the same folder (the build places both in `bin//`). +## Lessons learned -**Requirement:** run the target game **windowed or borderless**, not exclusive -fullscreen. Exclusive fullscreen minimizes on focus loss (defeating the focus -spoof) and can't be window-captured later. Only **controller** input is -forwarded — while the game is unfocused it won't receive OS keyboard/mouse. +Non-obvious things that cost time and constrain the design: -1. Start a DRM-free, **non-anti-cheat**, XInput game in windowed/borderless mode - and get to a screen that reads the pad. -2. Run `bin\Debug\coop_host.exe`. In the **Injection** panel, filter for the - game's `.exe`, select it, and click **Inject & Connect**. -3. Watch the **Hook status** section. Once it shows **Attached** and a non-zero - **"XInput polled: N/s"**, the game is provably reading our hook — injection - works. **Focus spoof: active** confirms the window was found and subclassed. -4. **Prove forwarding is the source:** tick **Forward synthetic test input**. - The game should now move on its own — left stick sweeping a circle, A pressed - every other second — independent of your physical controller. Untick it to - return control to your pad. -5. Sanity-check the focus spoof: click into another window so the game loses real - focus. It should keep running/animating (not pause), and the poll rate should - stay non-zero. -6. Click **Stop forwarding** (or quit the host) to tear down the channel. - -> If injection fails with an access error, run the host as administrator. If it -> reports "target is 32-bit", that game needs the x86 hook (a later phase). - -For a quick sanity check of the forwarding core without a game, run -`bin\Debug\hook_selftest.exe` — it should print `SELFTEST PASS`. - -## Phase 1b: video mirror (Windows Graphics Capture) - -The **Video mirror** panel captures the injected game's window (via Windows -Graphics Capture — no injection needed for video) and draws it, letterboxed, as -the host window's background. This is what Steam RPT streams to guests. - -1. Inject into a game as in Phase 1a (the hook reports the game's window, which - the mirror uses as its source). -2. In the **Video mirror** panel, tick **Mirror game window**. The host window - should now show a live copy of the game; the panel reports the capture - resolution and render FPS. -3. The game can be behind the host window — WGC still captures occluded (but not - minimized) windows, and focus spoofing keeps it rendering. - -Requires Windows 10 1903+ for WGC; hiding the capture border needs build 20348+. -This is the place to judge capture latency/stutter; if it's not good enough, the -fallback is a `Present`-hook capture path in the same injected DLL. +- **RPT only streams the *focused* window.** The game therefore can't hold focus + itself; the hook spoofs focus (`GetForegroundWindow` / `GetActiveWindow` / + `GetFocus`, plus swallowing window-deactivation messages) so the game keeps + polling and rendering while the host owns the real OS focus. +- **Run target games windowed or borderless, never exclusive fullscreen.** + Exclusive fullscreen minimizes on focus loss (defeating the focus spoof) and + can't be window-captured. While unfocused the game gets no OS keyboard/mouse — + only the forwarded controller. +- **`ActivateAudioInterfaceAsync` requires an *agile* completion handler.** If the + handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected + **synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of + COM apartment, MFStartup, device path, or activation params. (WRL/wil-based + samples hide this because they make the handler agile for you.) Process loopback + also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`. +- **WGC captures occluded windows but not minimized ones.** The game may sit + behind the host window, but must not be minimized. +- **Process-loopback capture doesn't mute the source**, hence the local audio + echo — capturing a process's render does not stop it reaching the speakers.