BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README for Phases 0-2 done, add roadmap + lessons

Browse Source 
- Reflect audio mirror (Phase 2) and video mirror (Phase 1b) as implemented;
  drop stale "(current)"/"still ahead" framing and the broken docs/ link.
- Add a Roadmap section for the planned work: toggle/generalize the debug UI,
  fix the local audio echo, Present-hook video path, Steam Input, x86 support.
- Add Lessons learned (RPT focus requirement, no exclusive fullscreen, the
  ActivateAudioInterfaceAsync agile-handler gotcha, WGC occlusion, audio echo).
- Document clangd/compile_commands setup and the CTest suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-19 10:27:57 +02:00
parent 663d86e6ec
commit 8059f0e488
1 changed files with 117 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
README.md
View File

@@ -15,17 +15,16 @@ window that is a live copy of the game's video + audio, and forwards the guests'
input back into the real game. Steam's RPT captures the mirror window — so any input back into the real game. Steam's RPT captures the mirror window — so any
XInput game becomes Remote-Play-Together-able. XInput game becomes Remote-Play-Together-able.
See [`docs`](docs) and the in-repo plan for the full design. ## Architecture
## Architecture (target) | Concern | Mechanism | Component | Status |
| --- | --- | --- | --- |
| Concern | Mechanism | Where | | Receive guest input | XInput (RPT delivers guest pads to the focused window) | `coop_host.exe` | done |
| --- | --- | --- | | Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done |
| Receive guest input | Steam Input / XInput (RPT delivers guests to the focused window) | `coop_host.exe` | | Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done |
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | | Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done |
| Mirror video | Windows Graphics Capture first; `IDXGISwapChain::Present` hook as the low-latency upgrade | host (+ hook) | | Mirror audio | WASAPI process-loopback capture of the game, re-rendered on the host | `coop_host.exe` | done |
| Mirror audio | WASAPI process-loopback capture of the game, re-rendered | `coop_host.exe` | | Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done |
| Host ↔ hook IPC | Named shared memory (seqlock for input, shared D3D11 texture for video) | `common/` |
## Limitations ## Limitations
@@ -37,32 +36,53 @@ See [`docs`](docs) and the in-repo plan for the full design.
co-op/local-multiplayer titles without active anti-cheat. co-op/local-multiplayer titles without active anti-cheat.
- **XInput only:** the game must read controllers via XInput (the common case). - **XInput only:** the game must read controllers via XInput (the common case).
DirectInput-only / RawInput-only games are not handled. DirectInput-only / RawInput-only games are not handled.
- **Architecture match:** the host and hook DLL must match the game's bitness. - **x64 only:** the host and hook DLL must match the game's bitness, and only x64
x64 is supported first; x86 support is a later phase (see the plan). is built today. 32-bit games need the x86 hook + injector (see Roadmap).
- **Local audio echo:** process-loopback capture does not mute the game, so the
game's audio plays locally *and* the host re-renders it — the local machine
hears it twice. Guests hear it once. Fixing this is on the Roadmap.
- **Debug-oriented UI:** the ImGui overlay is always visible and laid out for
diagnosing the pipeline, not for end use. It can't yet be toggled off.
## Status ## Status
**Phase 0 — donor spike. ✅ Validated.** `coop_host.exe` is a borderless D3D11 All phases below are implemented and verified.
window with an ImGui overlay listing every controller it sees. Confirmed
end-to-end: Steam RPT streams the window under a donor appid, and guest gamepads
arrive (with correct slot assignment) as XInput.
**Phase 1a — input forwarding + focus spoofing (current).** The host injects - **Phase 0 — donor spike. ✅** Confirmed Steam RPT streams an arbitrary
`coop_hook.dll`; the DLL hooks XInput (via SafetyHook) so the game reads the borderless window under a donor appid and routes guest gamepads into it as
forwarded controller state and *only* that state. The DLL also **spoofs focus** XInput (correct slot assignment). This is the premise the whole tool rests on.
(hooks `GetForegroundWindow`/`GetActiveWindow`/`GetFocus` and subclasses the game - **Phase 1a — input forwarding + focus spoofing. ✅** The host injects
window to swallow deactivation messages) so the game keeps running and polling `coop_hook.dll`; the DLL hooks XInput (SafetyHook) so the game reads the
while the tool holds the real OS focus — required because Steam RPT only captures forwarded pad state and *only* that state, and spoofs focus so the game keeps
the focused window. A hook→host status back-channel shows whether the hook is running while the host holds the real OS focus. A hook→host status
attached and how fast the game is polling it. The in-process `hook_selftest` back-channel reports attach state and poll rate.
validates the IPC + hook core without needing a game. - **Phase 1b — video mirror (WGC). ✅** The host captures the injected game's
window with Windows Graphics Capture and draws it letterboxed as its
background, so RPT streams a live mirror.
- **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via
WASAPI process loopback and re-renders it on the default endpoint, so RPT
carries game audio to guests.
**Phase 1b — video mirror (current).** The host captures the injected game's ## Roadmap
window with Windows Graphics Capture and draws it letterboxed as its background,
so RPT streams a live mirror of the game. See the test steps below.
Still ahead (scoped in the plan): a `Present`-hook capture path if WGC latency Future work, roughly in priority order:
disappoints, audio (WASAPI process loopback), and x86 support.
- **Make the tool usable, not just demonstrable:** toggle the debug overlay
on/off so the mirror window can be used clean.
- **Generalize the UI:** rework the panels from bug-specific debug readouts into
general-purpose status, and add broader debug info (latency, frame timing,
per-channel stats).
- **Fix the local audio echo:** mute the game's local render (or otherwise avoid
the double playback) while still capturing it for the mirror.
- **Present-hook video path:** capture the game's frames by hooking
`IDXGISwapChain::Present` in the injected DLL and sharing the backbuffer via a
shared D3D11 texture, as a lower-latency / more stable alternative to WGC.
- **Steam Input:** consume guest input through the Steam Input API directly
rather than XInput.
- **x86 support:** add an x86 build of `coop_hook.dll` plus an x86 injector
helper the x64 host spawns, so 32-bit games work (a 64-bit process can't
cleanly inject a 32-bit one). Detect target bitness with `IsWow64Process2`.
The shared-memory IPC layout is already fixed-width / bitness-stable.
## Building ## Building
@@ -75,99 +95,93 @@ git submodule update --init --recursive
cmake -S . -B build -G "Visual Studio 17 2022" -A x64 cmake -S . -B build -G "Visual Studio 17 2022" -A x64
cmake --build build --config Debug cmake --build build --config Debug
# output: bin/Debug/coop_host.exe # output: bin/Debug/coop_host.exe (+ coop_hook.dll, test exes)
``` ```
Third-party dependencies (Dear ImGui, SafetyHook) are git submodules under Third-party dependencies (Dear ImGui, SafetyHook) are git submodules under
`third_party/`; the Steamworks SDK is vendored manually there when wired. No `third_party/`; the Steamworks SDK is vendored manually there when wired. No
vcpkg / package manager is used. vcpkg / package manager is used.
## Phase 0: validating the donor-launch assumption ### clangd / IDE setup
This is a manual test — it needs Steam, a second person (or second account), and The Visual Studio CMake generator does **not** emit `compile_commands.json`, so
a donor game that supports Remote Play Together. clangd has no include paths and reports false errors. Run
[`gen-compile-commands.bat`](gen-compile-commands.bat) once (and after adding
sources or include dirs); it configures a parallel Ninja build in `build-clangd/`
that produces the database, which [`.clangd`](.clangd) points clangd at. clangd's
clang-cl driver resolves the MSVC / Windows SDK system includes on its own.
1. **Pick a donor game** you own that supports Remote Play Together (check the ## Tests
"Remote Play Together" tag on its store page). The donor only needs RPT
support; it is never actually played.
2. **Launch the host under the donor's appid.** Find the donor's appid (the ```sh
number in its store URL), then run: ctest --test-dir build -C Debug --output-on-failure
```
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
no controller needed).
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the
shipping process-loopback capture receives its audio by PID. Skips cleanly if
the machine has no audio endpoint.
## Running the tool (manual, end-to-end)
This needs Steam, a donor game that supports Remote Play Together, and a second
person/account to receive the stream.
1. **Launch the host under a donor appid.** Find the donor's appid (the number in
its store URL); the donor only needs RPT support and is never actually played:
```text ```text
"C:\Program Files (x86)\Steam\steam.exe" -applaunch <donorAppId> "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe" "C:\Program Files (x86)\Steam\steam.exe" -applaunch <donorAppId> "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe"
``` ```
The borderless overlay window should appear and Steam should consider the The borderless window appears and Steam marks the donor "running". If the donor
donor "running" (green status / "Stop" button in the library). ignores the trailing path, set the host as the donor's **Launch Options**
(`"D:\...\coop_host.exe" %command%`) or use a launcher like RemotePlayDetached.
> If the donor ignores the trailing path, set the host as the donor's 2. **Start the real game** windowed or borderless (not exclusive fullscreen — see
> **Launch Options** instead (`"D:\...\coop_host.exe" %command%` variants), Lessons learned). In the host's **Injection** panel, filter for the game's
> or use a launcher such as RemotePlayDetached. Recording which method makes `.exe`, select it, and click **Inject & Connect**. Watch **Hook status** for
> Steam attribute our window to the donor is the main deliverable of Phase 0. **Attached**, a non-zero **XInput polled: N/s**, and **Focus spoof: active**.
3. **Start Remote Play Together** from the Steam friends list / overlay and invite 3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** —
a friend (or a second machine/account). the host window now shows a live, letterboxed copy of the game.
4. **Verify on the guest side:** 4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
- The guest sees the borderless overlay window streamed (not a black screen). (You'll hear the game twice locally; that's the known echo — guests hear it
- The guest presses buttons on their controller and the corresponding slot in once.)
the overlay lights up. This proves RPT routes guest input into our window
as XInput — the foundation the whole tool relies on.
5. Press **Esc** in the host window to quit. 5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
sees the mirrored video, hears the audio, and that their controller drives the
real game.
**If steps 2 and 4 both work, the core premise holds** and we proceed to Phase 1 Useful checks while developing without RPT: tick **Forward synthetic test input**
(window capture + input injection). If not, we revisit the donor-attribution in the Injection panel to make the game move on its own (proving forwarding is the
approach before building further. source), and click away from the game to confirm focus spoofing keeps it running.
## Phase 1a: testing input forwarding locally > Injection access error → run the host as administrator. "Target is 32-bit" →
> that game needs the x86 hook (see Roadmap).
This needs no RPT, donor, or second account — just the host, the hook, a ## Lessons learned
controller, and a target game. `coop_host.exe` and `coop_hook.dll` must sit in
the same folder (the build places both in `bin/<Config>/`).
**Requirement:** run the target game **windowed or borderless**, not exclusive Non-obvious things that cost time and constrain the design:
fullscreen. Exclusive fullscreen minimizes on focus loss (defeating the focus
spoof) and can't be window-captured later. Only **controller** input is
forwarded — while the game is unfocused it won't receive OS keyboard/mouse.
1. Start a DRM-free, **non-anti-cheat**, XInput game in windowed/borderless mode - **RPT only streams the *focused* window.** The game therefore can't hold focus
and get to a screen that reads the pad. itself; the hook spoofs focus (`GetForegroundWindow` / `GetActiveWindow` /
2. Run `bin\Debug\coop_host.exe`. In the **Injection** panel, filter for the `GetFocus`, plus swallowing window-deactivation messages) so the game keeps
game's `.exe`, select it, and click **Inject & Connect**. polling and rendering while the host owns the real OS focus.
3. Watch the **Hook status** section. Once it shows **Attached** and a non-zero - **Run target games windowed or borderless, never exclusive fullscreen.**
**"XInput polled: N/s"**, the game is provably reading our hook — injection Exclusive fullscreen minimizes on focus loss (defeating the focus spoof) and
works. **Focus spoof: active** confirms the window was found and subclassed. can't be window-captured. While unfocused the game gets no OS keyboard/mouse —
4. **Prove forwarding is the source:** tick **Forward synthetic test input**. only the forwarded controller.
The game should now move on its own — left stick sweeping a circle, A pressed - **`ActivateAudioInterfaceAsync` requires an *agile* completion handler.** If the
every other second — independent of your physical controller. Untick it to handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected
return control to your pad. **synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of
5. Sanity-check the focus spoof: click into another window so the game loses real COM apartment, MFStartup, device path, or activation params. (WRL/wil-based
focus. It should keep running/animating (not pause), and the poll rate should samples hide this because they make the handler agile for you.) Process loopback
stay non-zero. also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`.
6. Click **Stop forwarding** (or quit the host) to tear down the channel. - **WGC captures occluded windows but not minimized ones.** The game may sit
behind the host window, but must not be minimized.
> If injection fails with an access error, run the host as administrator. If it - **Process-loopback capture doesn't mute the source**, hence the local audio
> reports "target is 32-bit", that game needs the x86 hook (a later phase). echo — capturing a process's render does not stop it reaching the speakers.
For a quick sanity check of the forwarding core without a game, run
`bin\Debug\hook_selftest.exe` — it should print `SELFTEST PASS`.
## Phase 1b: video mirror (Windows Graphics Capture)
The **Video mirror** panel captures the injected game's window (via Windows
Graphics Capture — no injection needed for video) and draws it, letterboxed, as
the host window's background. This is what Steam RPT streams to guests.
1. Inject into a game as in Phase 1a (the hook reports the game's window, which
the mirror uses as its source).
2. In the **Video mirror** panel, tick **Mirror game window**. The host window
should now show a live copy of the game; the panel reports the capture
resolution and render FPS.
3. The game can be behind the host window — WGC still captures occluded (but not
minimized) windows, and focus spoofing keeps it rendering.
Requires Windows 10 1903+ for WGC; hiding the capture border needs build 20348+.
This is the place to judge capture latency/stutter; if it's not good enough, the
fallback is a `Present`-hook capture path in the same injected DLL.