BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README for Phases 0-2 done, add roadmap + lessons

Browse Source 
- Reflect audio mirror (Phase 2) and video mirror (Phase 1b) as implemented;
  drop stale "(current)"/"still ahead" framing and the broken docs/ link.
- Add a Roadmap section for the planned work: toggle/generalize the debug UI,
  fix the local audio echo, Present-hook video path, Steam Input, x86 support.
- Add Lessons learned (RPT focus requirement, no exclusive fullscreen, the
  ActivateAudioInterfaceAsync agile-handler gotcha, WGC occlusion, audio echo).
- Document clangd/compile_commands setup and the CTest suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-19 10:27:57 +02:00
parent 663d86e6ec
commit 8059f0e488
1 changed files with 117 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
README.md
View File

@@ -15,17 +15,16 @@ window that is a live copy of the game's video + audio, and forwards the guests'
input back into the real game. Steam's RPT captures the mirror window — so any
XInput game becomes Remote-Play-Together-able.
See [`docs`](docs) and the in-repo plan for the full design.
## Architecture
## Architecture (target)
| Concern | Mechanism | Where |
| --- | --- | --- |
| Receive guest input | Steam Input / XInput (RPT delivers guests to the focused window) | `coop_host.exe` |
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` |
| Mirror video | Windows Graphics Capture first; `IDXGISwapChain::Present` hook as the low-latency upgrade | host (+ hook) |
| Mirror audio | WASAPI process-loopback capture of the game, re-rendered | `coop_host.exe` |
| Host ↔ hook IPC | Named shared memory (seqlock for input, shared D3D11 texture for video) | `common/` |
| Concern | Mechanism | Component | Status |
| --- | --- | --- | --- |
| Receive guest input | XInput (RPT delivers guest pads to the focused window) | `coop_host.exe` | done |
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done |
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done |
| Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done |
| Mirror audio | WASAPI process-loopback capture of the game, re-rendered on the host | `coop_host.exe` | done |
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done |
## Limitations
@@ -37,32 +36,53 @@ See [`docs`](docs) and the in-repo plan for the full design.
co-op/local-multiplayer titles without active anti-cheat.
- **XInput only:** the game must read controllers via XInput (the common case).
DirectInput-only / RawInput-only games are not handled.
- **Architecture match:** the host and hook DLL must match the game's bitness.
x64 is supported first; x86 support is a later phase (see the plan).
- **x64 only:** the host and hook DLL must match the game's bitness, and only x64
is built today. 32-bit games need the x86 hook + injector (see Roadmap).
- **Local audio echo:** process-loopback capture does not mute the game, so the
game's audio plays locally *and* the host re-renders it — the local machine
hears it twice. Guests hear it once. Fixing this is on the Roadmap.
- **Debug-oriented UI:** the ImGui overlay is always visible and laid out for
diagnosing the pipeline, not for end use. It can't yet be toggled off.
## Status
**Phase 0 — donor spike. ✅ Validated.** `coop_host.exe` is a borderless D3D11
window with an ImGui overlay listing every controller it sees. Confirmed
end-to-end: Steam RPT streams the window under a donor appid, and guest gamepads
arrive (with correct slot assignment) as XInput.
All phases below are implemented and verified.
**Phase 1a — input forwarding + focus spoofing (current).** The host injects
`coop_hook.dll`; the DLL hooks XInput (via SafetyHook) so the game reads the
forwarded controller state and *only* that state. The DLL also **spoofs focus**
(hooks `GetForegroundWindow`/`GetActiveWindow`/`GetFocus` and subclasses the game
window to swallow deactivation messages) so the game keeps running and polling
while the tool holds the real OS focus — required because Steam RPT only captures
the focused window. A hook→host status back-channel shows whether the hook is
attached and how fast the game is polling it. The in-process `hook_selftest`
validates the IPC + hook core without needing a game.
- **Phase 0 — donor spike. ✅** Confirmed Steam RPT streams an arbitrary
borderless window under a donor appid and routes guest gamepads into it as
XInput (correct slot assignment). This is the premise the whole tool rests on.
- **Phase 1a — input forwarding + focus spoofing. ✅** The host injects
`coop_hook.dll`; the DLL hooks XInput (SafetyHook) so the game reads the
forwarded pad state and *only* that state, and spoofs focus so the game keeps
running while the host holds the real OS focus. A hook→host status
back-channel reports attach state and poll rate.
- **Phase 1b — video mirror (WGC). ✅** The host captures the injected game's
window with Windows Graphics Capture and draws it letterboxed as its
background, so RPT streams a live mirror.
- **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via
WASAPI process loopback and re-renders it on the default endpoint, so RPT
carries game audio to guests.
**Phase 1b — video mirror (current).** The host captures the injected game's
window with Windows Graphics Capture and draws it letterboxed as its background,
so RPT streams a live mirror of the game. See the test steps below.
## Roadmap
Still ahead (scoped in the plan): a `Present`-hook capture path if WGC latency
disappoints, audio (WASAPI process loopback), and x86 support.
Future work, roughly in priority order:
- **Make the tool usable, not just demonstrable:** toggle the debug overlay
on/off so the mirror window can be used clean.
- **Generalize the UI:** rework the panels from bug-specific debug readouts into
general-purpose status, and add broader debug info (latency, frame timing,
per-channel stats).
- **Fix the local audio echo:** mute the game's local render (or otherwise avoid
the double playback) while still capturing it for the mirror.
- **Present-hook video path:** capture the game's frames by hooking
`IDXGISwapChain::Present` in the injected DLL and sharing the backbuffer via a
shared D3D11 texture, as a lower-latency / more stable alternative to WGC.
- **Steam Input:** consume guest input through the Steam Input API directly
rather than XInput.
- **x86 support:** add an x86 build of `coop_hook.dll` plus an x86 injector
helper the x64 host spawns, so 32-bit games work (a 64-bit process can't
cleanly inject a 32-bit one). Detect target bitness with `IsWow64Process2`.
The shared-memory IPC layout is already fixed-width / bitness-stable.
## Building
@@ -75,99 +95,93 @@ git submodule update --init --recursive
cmake -S . -B build -G "Visual Studio 17 2022" -A x64
cmake --build build --config Debug
# output: bin/Debug/coop_host.exe
# output: bin/Debug/coop_host.exe (+ coop_hook.dll, test exes)
```
Third-party dependencies (Dear ImGui, SafetyHook) are git submodules under
`third_party/`; the Steamworks SDK is vendored manually there when wired. No
vcpkg / package manager is used.
## Phase 0: validating the donor-launch assumption
### clangd / IDE setup
This is a manual test — it needs Steam, a second person (or second account), and
a donor game that supports Remote Play Together.
The Visual Studio CMake generator does **not** emit `compile_commands.json`, so
clangd has no include paths and reports false errors. Run
[`gen-compile-commands.bat`](gen-compile-commands.bat) once (and after adding
sources or include dirs); it configures a parallel Ninja build in `build-clangd/`
that produces the database, which [`.clangd`](.clangd) points clangd at. clangd's
clang-cl driver resolves the MSVC / Windows SDK system includes on its own.
1. **Pick a donor game** you own that supports Remote Play Together (check the
"Remote Play Together" tag on its store page). The donor only needs RPT
support; it is never actually played.
## Tests
2. **Launch the host under the donor's appid.** Find the donor's appid (the
number in its store URL), then run:
```sh
ctest --test-dir build -C Debug --output-on-failure
```
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
no controller needed).
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone WASAPI
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) and verifies the
shipping process-loopback capture receives its audio by PID. Skips cleanly if
the machine has no audio endpoint.
## Running the tool (manual, end-to-end)
This needs Steam, a donor game that supports Remote Play Together, and a second
person/account to receive the stream.
1. **Launch the host under a donor appid.** Find the donor's appid (the number in
its store URL); the donor only needs RPT support and is never actually played:
```text
"C:\Program Files (x86)\Steam\steam.exe" -applaunch <donorAppId> "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe"
```
The borderless overlay window should appear and Steam should consider the
donor "running" (green status / "Stop" button in the library).
The borderless window appears and Steam marks the donor "running". If the donor
ignores the trailing path, set the host as the donor's **Launch Options**
(`"D:\...\coop_host.exe" %command%`) or use a launcher like RemotePlayDetached.
> If the donor ignores the trailing path, set the host as the donor's
> **Launch Options** instead (`"D:\...\coop_host.exe" %command%` variants),
> or use a launcher such as RemotePlayDetached. Recording which method makes
> Steam attribute our window to the donor is the main deliverable of Phase 0.
2. **Start the real game** windowed or borderless (not exclusive fullscreen — see
Lessons learned). In the host's **Injection** panel, filter for the game's
`.exe`, select it, and click **Inject & Connect**. Watch **Hook status** for
**Attached**, a non-zero **XInput polled: N/s**, and **Focus spoof: active**.
3. **Start Remote Play Together** from the Steam friends list / overlay and invite
a friend (or a second machine/account).
3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** —
the host window now shows a live, letterboxed copy of the game.
4. **Verify on the guest side:**
- The guest sees the borderless overlay window streamed (not a black screen).
- The guest presses buttons on their controller and the corresponding slot in
the overlay lights up. This proves RPT routes guest input into our window
as XInput — the foundation the whole tool relies on.
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
(You'll hear the game twice locally; that's the known echo — guests hear it
once.)
5. Press **Esc** in the host window to quit.
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
sees the mirrored video, hears the audio, and that their controller drives the
real game.
**If steps 2 and 4 both work, the core premise holds** and we proceed to Phase 1
(window capture + input injection). If not, we revisit the donor-attribution
approach before building further.
Useful checks while developing without RPT: tick **Forward synthetic test input**
in the Injection panel to make the game move on its own (proving forwarding is the
source), and click away from the game to confirm focus spoofing keeps it running.
## Phase 1a: testing input forwarding locally
> Injection access error → run the host as administrator. "Target is 32-bit" →
> that game needs the x86 hook (see Roadmap).
This needs no RPT, donor, or second account — just the host, the hook, a
controller, and a target game. `coop_host.exe` and `coop_hook.dll` must sit in
the same folder (the build places both in `bin/<Config>/`).
## Lessons learned
**Requirement:** run the target game **windowed or borderless**, not exclusive
fullscreen. Exclusive fullscreen minimizes on focus loss (defeating the focus
spoof) and can't be window-captured later. Only **controller** input is
forwarded — while the game is unfocused it won't receive OS keyboard/mouse.
Non-obvious things that cost time and constrain the design:
1. Start a DRM-free, **non-anti-cheat**, XInput game in windowed/borderless mode
and get to a screen that reads the pad.
2. Run `bin\Debug\coop_host.exe`. In the **Injection** panel, filter for the
game's `.exe`, select it, and click **Inject & Connect**.
3. Watch the **Hook status** section. Once it shows **Attached** and a non-zero
**"XInput polled: N/s"**, the game is provably reading our hook — injection
works. **Focus spoof: active** confirms the window was found and subclassed.
4. **Prove forwarding is the source:** tick **Forward synthetic test input**.
The game should now move on its own — left stick sweeping a circle, A pressed
every other second — independent of your physical controller. Untick it to
return control to your pad.
5. Sanity-check the focus spoof: click into another window so the game loses real
focus. It should keep running/animating (not pause), and the poll rate should
stay non-zero.
6. Click **Stop forwarding** (or quit the host) to tear down the channel.
> If injection fails with an access error, run the host as administrator. If it
> reports "target is 32-bit", that game needs the x86 hook (a later phase).
For a quick sanity check of the forwarding core without a game, run
`bin\Debug\hook_selftest.exe` — it should print `SELFTEST PASS`.
## Phase 1b: video mirror (Windows Graphics Capture)
The **Video mirror** panel captures the injected game's window (via Windows
Graphics Capture — no injection needed for video) and draws it, letterboxed, as
the host window's background. This is what Steam RPT streams to guests.
1. Inject into a game as in Phase 1a (the hook reports the game's window, which
the mirror uses as its source).
2. In the **Video mirror** panel, tick **Mirror game window**. The host window
should now show a live copy of the game; the panel reports the capture
resolution and render FPS.
3. The game can be behind the host window — WGC still captures occluded (but not
minimized) windows, and focus spoofing keeps it rendering.
Requires Windows 10 1903+ for WGC; hiding the capture border needs build 20348+.
This is the place to judge capture latency/stutter; if it's not good enough, the
fallback is a `Present`-hook capture path in the same injected DLL.
- **RPT only streams the *focused* window.** The game therefore can't hold focus
itself; the hook spoofs focus (`GetForegroundWindow` / `GetActiveWindow` /
`GetFocus`, plus swallowing window-deactivation messages) so the game keeps
polling and rendering while the host owns the real OS focus.
- **Run target games windowed or borderless, never exclusive fullscreen.**
Exclusive fullscreen minimizes on focus loss (defeating the focus spoof) and
can't be window-captured. While unfocused the game gets no OS keyboard/mouse —
only the forwarded controller.
- **`ActivateAudioInterfaceAsync` requires an *agile* completion handler.** If the
handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected
**synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of
COM apartment, MFStartup, device path, or activation params. (WRL/wil-based
samples hide this because they make the handler agile for you.) Process loopback
also needs the Windows 10 20H1 headers — build with `NTDDI_VERSION ≥ 0x0A00000B`.
- **WGC captures occluded windows but not minimized ones.** The game may sit
behind the host window, but must not be minimized.
- **Process-loopback capture doesn't mute the source**, hence the local audio
echo — capturing a process's render does not stop it reaching the speakers.