The earlier validation concluded suspended-inject was "not applicable -- the title requires launching through Steam." That was wrong; it was two bugs: 1. coop_vk_validate's inject mode launched the exe with CreateProcessW and a null working directory, so the game couldn't load steam_api64.dll / resources/ (loaded relative to cwd) and never rendered -> no presents. Launch with the game's own folder as cwd and it runs fine directly, no Steam needed. 2. The game resolves vkQueuePresentKHR / vkCreateSwapchainKHR via vkGetInstanceProcAddr (volk's volkLoadInstance does this), but vk_hook only substituted our detours when they were resolved via vkGetDeviceProcAddr -- so the present bypassed the hook. Intercept those names in hk_vkGetInstanceProcAddr too (our detours already gate on g_capture_enabled/g_device, so handing them out before the device exists is safe). With both fixed, inject mode captures Sphere Spectacle correctly: 1920x1080, correct colors/orientation (screenshot), ~480 fps present while mirroring at the ~150 Hz throttle -- no present-thread impact (the VkCapture fix is shared). Also makes the validator ASSERT a present-rate floor while capturing (it used to report the rate and rationalize it, which is exactly what hid the 144->3 FPS stall), and reports the true mirror rate from video.generation. Division of labor is about who launches the game: layer for Steam-launched (can't suspend), inject when we control the launch. README lessons-learned corrected accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
702 lines
54 KiB
Markdown
702 lines
54 KiB
Markdown
# CoopAllTheThings
|
||
|
||
Steam **Remote Play Together (RPT)** for any XInput game — without breaking DRM,
|
||
achievements, or playtime.
|
||
|
||
Existing "donor game" tools (e.g. RemotePlayWhatever) copy a target game's files
|
||
into a donor game's folder and rename the executable so Steam streams the target
|
||
under the donor's appid. That breaks DRM-protected games, breaks achievements,
|
||
and credits playtime to the donor.
|
||
|
||
CoopAllTheThings takes a different approach: the **real game runs normally under
|
||
its own appid** (so DRM, achievements, and playtime all work), while a lightweight
|
||
**mirror app runs under the donor appid**. The mirror presents a borderless
|
||
window that is a live copy of the game's video + audio, and forwards the guests'
|
||
input back into the real game. Steam's RPT captures the mirror window — so any
|
||
XInput game becomes Remote-Play-Together-able.
|
||
|
||
The end-to-end path is working: launched under a donor appid, the host streams a
|
||
live video + audio mirror of a separately-running game over Remote Play Together
|
||
and forwards guest controllers back into it.
|
||
|
||
## Architecture
|
||
|
||
| Concern | Mechanism | Component |
|
||
| --- | --- | --- |
|
||
| Receive guest input | XInput (RPT delivers guest pads to the focused window); optional, opt-in Steam Input when built with the Steamworks SDK | `coop_host.exe` |
|
||
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` |
|
||
| Forward mouse + keyboard | Opt-in MKB subsystem: host streams its window's clicks/keys; the hook posts the matching window messages, synthesizes `GetAsyncKeyState`/`GetKeyboardState`/`GetCursorPos` for polling games, augments `IDirectInputDevice8::GetDeviceState` for **DirectInput** games, and synthesizes `WM_INPUT` + `GetRawInputData` for **Raw Input** games | `coop_hook.dll` + `coop_host.exe` |
|
||
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` |
|
||
| Mirror video (default) | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` |
|
||
| Mirror video (hooked) | Injected Present / OpenGL hook copies the backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` |
|
||
| Mirror audio | Injected render-hook copies each of the game's WASAPI render streams into its own shared ring and silences the game locally (no echo); the host mixes the streams (soft-clipped); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` |
|
||
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel, video/audio/log shares) | `common/` |
|
||
|
||
The hooked video path has three producers: **Direct3D (DXGI)** hooks
|
||
`IDXGISwapChain::Present` / `Present1` and copies the backbuffer — directly for
|
||
D3D11 games (the backbuffer is an `ID3D11Texture2D`), via a **D3D11On12
|
||
bridge** for D3D12 games (wrap the `ID3D12Resource` backbuffer, `CopyResource` into
|
||
the shared texture), and via a **D3D10 read-back** for D3D10 games (their backbuffer's
|
||
D3D11 view is empty and a feature-level-10 device can't host the shared texture, so
|
||
read it through the game's own D3D10 device and upload it via a hook-owned D3D11
|
||
device); **Direct3D 9** inline-hooks `IDirect3DDevice9::Present` (vtable index 17) and
|
||
read-backs the backbuffer with `GetRenderTargetData` (a D3D9 surface isn't D3D11-shareable),
|
||
swizzling BGRA→RGBA and uploading via a hook-owned D3D11 device — one path covers both plain
|
||
D3D9 and D3D9Ex; **OpenGL** hooks `SwapBuffers` / `wglSwapBuffers` and reads the
|
||
backbuffer with `glReadPixels` (for games that never touch DXGI, e.g. Phantom
|
||
Brave); and **Vulkan** inline-hooks the `vulkan-1.dll` `vkGetInstanceProcAddr` export to
|
||
intercept the resolution chain (`vkCreateInstance` / `vkCreateDevice` /
|
||
`vkCreateSwapchainKHR` / `vkQueuePresentKHR`) and reads the presented image back with
|
||
`vkCmdCopyImageToBuffer` — but only when the hook is present **before** the game initializes
|
||
Vulkan (it caches its present pointer at init), so the Vulkan path needs early presence via
|
||
Auto-attach or the opt-in implicit **capture layer** (`coop_vk_layer`, registered per-user by the
|
||
Injection panel's "Set up Vulkan layer" checkbox and scoped to the target game); a too-late attach
|
||
shows a red relaunch banner and falls back to WGC. The host samples the copy as plain UNORM
|
||
(`srgb_to_unorm`) so `*_SRGB`-backbuffer games mirror at correct brightness. **WGC remains the
|
||
default** and covers anything the hooked path doesn't.
|
||
|
||
## Limitations
|
||
|
||
- **Anti-cheat:** the input path injects `coop_hook.dll` into the target game.
|
||
Games protected by kernel-level anti-cheat (Easy Anti-Cheat, BattlEye,
|
||
Vanguard, etc.) will detect the injected module and may **kick the player or
|
||
issue a ban**. Such games are explicitly **out of scope and unsupported** — do
|
||
not use CoopAllTheThings with them. The tool targets single-player and
|
||
co-op/local-multiplayer titles without active anti-cheat.
|
||
- **XInput only:** the game must read controllers via XInput (the common case).
|
||
DirectInput-only / RawInput-only games are not handled.
|
||
- **32-bit games supported via a helper:** the host is x64, but the build also
|
||
produces an x86 hook DLL (`coop_hook_x86.dll`) and a 32-bit injector helper
|
||
(`coop_inject_x86.exe`). When the target is a 32-bit (WOW64) process the host
|
||
detects it (`IsWow64Process2`) and shells out to the helper to load the x86 DLL
|
||
(a 64-bit process can't cleanly inject a 32-bit one). The shared-memory IPC is
|
||
fixed-width / bitness-stable, so the x64 host and x86 hook interoperate.
|
||
- **Local audio echo on the fallback path:** when the render-hook is active it
|
||
silences the game's local playback while mirroring it, so there is no echo. If
|
||
the hook can't attach or the game uses an unhooked render path, the host falls
|
||
back to process-loopback capture, which does *not* mute the game — so the local
|
||
machine hears the audio twice (guests hear it once). The Audio panel shows which
|
||
path is active.
|
||
- **A pre-existing stream's format is *recovered*, not known — by measurement and
|
||
cross-correlation.** The tool injects into an already-running game, so the audio
|
||
render-hook usually never saw the game's `IAudioClient::Initialize`, and `AUTOCONVERTPCM`
|
||
hides the buffer stride (WASAPI exposes no API for a pre-existing client's format). The
|
||
hook first recovers the **sample rate** by measuring the render cadence (so playback pitch
|
||
is correct, e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), assuming the device's
|
||
channels/bit-depth. When the game is still audible (the measurement window), the host then
|
||
**cross-correlates the two capture paths** — the hook (pre-mix) against a parallel
|
||
process-loopback (post-mix, the known device format) — to verify/correct the rate and to
|
||
**recover the channels + bit depth** by trying candidate de-interleavings and keeping the one
|
||
that aligns (`audio_format_verifier` → `coop/audio_correlate.hpp`). The recovered format feeds
|
||
the existing override channel. The one case it can't resolve is a *genuinely ambiguous* layout
|
||
(a stream whose channels carry identical content looks the same as one channel at double the
|
||
rate); there it stays the device assumption and the correlation reports low confidence rather
|
||
than guess. A wrong/assumed layout is mirrored with the wrong de-interleaving (garbled) but
|
||
never an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`),
|
||
and the local **mute** uses `AUDCLNT_BUFFERFLAGS_SILENT` (WASAPI ignores the buffer contents),
|
||
so it never *writes* the wrongly-sized buffer either. Every captured stream is silenced locally
|
||
— guessed or exact — so there's no echo on the hooked path, and the loopback fallback is always
|
||
format-correct. The Audio panel shows each stream's format provenance (*known* / *measuring* /
|
||
*measured rate* / *low-confidence* / *override*), and (under Debug details) lets the operator
|
||
**re-measure** or **override** the format when needed. Overrides are **remembered per game** (and
|
||
a format caught exactly at `Initialize` is auto-saved), so a known-bad game is corrected
|
||
automatically next launch. Streams created *after* injection are captured exactly.
|
||
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
|
||
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
|
||
for RPT; F2 frees the operator cursor; **F10 saves a PNG screenshot** (back buffer,
|
||
written next to the exe) regardless of window focus or occlusion.
|
||
|
||
## Roadmap
|
||
|
||
The near-term tracked tasks are complete: injection hardening (the cross-backend safe-unhook drain),
|
||
two-path audio-format correlation (rate + channels/bit-depth recovery), mouse + keyboard forwarding
|
||
for DirectInput and Raw Input games, real-game Vulkan validation (`coop_vk_validate` against Sphere
|
||
Spectacle), the **Vulkan capture performance fix** — the read-back that dropped a 144 FPS game to ~3
|
||
FPS now runs off the present thread (shared `coop::hook::VkCapture`), so the game keeps its frame
|
||
rate while mirroring (guarded by `vk_capture_perf_test`; every GPU backend's hook test also asserts a
|
||
present-thread overhead bound) — and the **inline-hook (suspended-inject) Vulkan path**, which now
|
||
captures Sphere Spectacle correctly once the harness launches the exe with the right working
|
||
directory and `vk_hook` intercepts present/swapchain resolved via `vkGetInstanceProcAddr` (not just
|
||
`vkGetDeviceProcAddr`). Both Vulkan paths are verified on the real game. See **Lessons learned** and
|
||
the test suite for each. Open directions: per-game profiles, multi-guest virtual-pad mapping, and
|
||
continuous raw-mouse *movement* forwarding (the MKB event stream is position-based today).
|
||
|
||
## Building
|
||
|
||
Requirements: Windows 10/11, Visual Studio 2022 (MSVC + C++ workload), CMake ≥ 3.21.
|
||
|
||
```sh
|
||
git clone --recurse-submodules <repo-url>
|
||
# or, if already cloned:
|
||
git submodule update --init --recursive
|
||
|
||
cmake -S . -B build -G "Visual Studio 17 2022" -A x64
|
||
cmake --build build --config Debug
|
||
# output: bin/Debug/coop_host.exe (+ coop_hook.dll, coop_vk_layer.dll + manifest, test exes)
|
||
```
|
||
|
||
The x64 build also drives a nested Win32 sub-build (CMake `ExternalProject`,
|
||
configured into `build/x86/`) that produces `coop_hook_x86.dll` and
|
||
`coop_inject_x86.exe` for 32-bit games, staged next to the x64 binaries. Disable
|
||
it with `-DCOOP_BUILD_X86_HELPER=OFF` if you don't need 32-bit support.
|
||
|
||
Third-party dependencies (Dear ImGui, SafetyHook, the official Khronos Vulkan-Headers, and the
|
||
`zeux/volk` Vulkan meta-loader) are git submodules under `third_party/`. No vcpkg / package
|
||
manager is used; CMake fails with a clear `git submodule update --init --recursive` hint if one
|
||
is missing (`coop_require_submodule`).
|
||
|
||
**Steam Input is optional.** It's enabled automatically when the Steamworks SDK is
|
||
vendored at `third_party/steamworks_sdk/` (extract the `steamworks_sdk_*.zip`
|
||
there). The SDK isn't redistributable, so it's gitignored and never committed; if
|
||
it's absent the host builds XInput-only (no other features depend on it). When
|
||
present, the build links `steam_api64.lib`, stages `steam_api64.dll` and the
|
||
action manifest next to the host, and also builds `coop_steam_input_probe`.
|
||
|
||
Steam Input is **off by default and XInput is the primary path**: merely
|
||
initializing Steam Input activates Steam's in-process XInput interception, which
|
||
hides controllers from XInput unless they're bound to our action set for the
|
||
running appid. Enable it (Controllers panel → **Use Steam Input**) only once a
|
||
controller is bound to Steam Input for the donor appid.
|
||
|
||
### clangd / IDE setup
|
||
|
||
The Visual Studio CMake generator does **not** emit `compile_commands.json`, so
|
||
clangd has no include paths and reports false errors. Run
|
||
[`gen-compile-commands.bat`](gen-compile-commands.bat) once (and after adding
|
||
sources or include dirs); it configures a parallel Ninja build in `build-clangd/`
|
||
that produces the database, which [`.clangd`](.clangd) points clangd at. clangd's
|
||
clang-cl driver resolves the MSVC / Windows SDK system includes on its own.
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
ctest --test-dir build -C Debug --output-on-failure
|
||
```
|
||
|
||
> **Fix bugs test-first.** Every bug fix must begin with a test that *reproduces* the bug
|
||
> and **fails** on the unfixed code — run it, watch it fail, and confirm it fails for the
|
||
> right reason. Only then write the fix, and confirm the same test now passes. A fix without
|
||
> a first-failing test is not done: the test is what proves the bug existed, that the change
|
||
> addresses it, and that it can't silently come back. (The hooked-audio double-play bug is the
|
||
> worked example: `audio_hook_test`'s guessed-path mute assertion was added and seen to fail
|
||
> before the one-line mute fix landed.)
|
||
|
||
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
|
||
no controller needed).
|
||
- **`dinput_hook_test`** — in-process self-test of the **DirectInput** forwarding path. Reuses
|
||
`mkb_hook.cpp`, installs the MKB hooks (which vtable-swap `IDirectInputDevice8::GetDeviceState`
|
||
via a kept-alive probe device), forwards a synthetic key + mouse button through the MKB ring,
|
||
pumps it, then creates a *real* DirectInput keyboard + mouse device and asserts `GetDeviceState`
|
||
returns the forwarded input (the key at its DIK scan-code, the left mouse button). Skips cleanly
|
||
if DirectInput can't acquire a device. (The Raw Input path is operator-validated against a real
|
||
game — an in-process WM_INPUT round-trip is too brittle to assert reliably.)
|
||
- **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC
|
||
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
|
||
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
|
||
soft-clip / encode for float32 + int16). No device needed.
|
||
- **`rate_estimator_test`** — unit test of the robust sample-rate estimator (the fix for
|
||
the wrong-rate bug). Feeds synthetic, adversarial render cadences and asserts it
|
||
converges to the right standard rate, rejects burst windows (never commits to a wrong
|
||
neighbour, incl. the real 46205 misread), flags a genuinely non-standard rate
|
||
low-confidence instead of spinning, and ignores idle windows. Pure logic, no device.
|
||
- **`audio_overrides_test`** — unit test of the per-game audio override store
|
||
(persist/reload, case-insensitive lookup by image name, and the differing-overwrite
|
||
detection that drives the warning). No device.
|
||
- **`tone_analysis_test`** — unit test of the audio fidelity analyzer used by
|
||
`coop_audio_validate` (pitch error in cents, SNR/THD, click + dropout detection) and the WAV
|
||
reader/writer. Synthesizes a clean tone, a wrong-rate (pitch-shifted) tone, a tone with
|
||
injected clicks, and one with silence gaps, and asserts each metric matches what was injected
|
||
(e.g. 44100 played as 48000 → +147 cents). Pure header logic, no device.
|
||
- **`audio_correlation_test`** — unit test of the two-path audio-format correlator
|
||
([`common/include/coop/audio_correlate.hpp`](common/include/coop/audio_correlate.hpp)), which
|
||
recovers a guessed stream's true sample rate from *ground truth* instead of cadence. Synthesizes
|
||
one continuous signal sampled at two rates (the hook's true rate + the device rate, with capture
|
||
skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the
|
||
true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000
|
||
case the cadence method can misread), and that unrelated signals are *not* confidently matched.
|
||
Also covers **layout recovery** (`correlate_format`): from a self-describing *chunked* capture
|
||
(each render buffer padded to the device block, as the hook's verify tap produces it) it strips
|
||
the padding per candidate de-interleaving and recovers the true channels + bit depth + rate
|
||
(stereo float, 16-bit PCM, 5.1, mono), and leaves a genuinely-ambiguous identical-channel layout
|
||
unconfident. Pure header logic, no device.
|
||
- **`audio_verify_test`** — integration test of the host's two-path verifier
|
||
([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches
|
||
`coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so
|
||
this rate test isn't perturbed by a channel mismatch), injects the hook late (a guessed stream),
|
||
and runs the real `verify_stream_format()`: it co-captures the hook (pre-mix, via the ring's
|
||
`verify_capture` tap) and a parallel process-loopback (post-mix) of the same audio and correlates
|
||
them. Scenario (a) asserts it recovers the true rate; scenario (b) renders a *different* channel
|
||
count than the device with distinct per-channel content and asserts it recovers the full layout
|
||
(channels + bit depth + rate). Skips cleanly without an audio endpoint.
|
||
- **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy
|
||
(`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts
|
||
the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill
|
||
policy withhold available data ~168× and drain the buffer to the brink of silence (the
|
||
under-run / "metallic" bug `coop_audio_validate` found). No device.
|
||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook's **format
|
||
detection**, the part that gets pitch right. Using a shared configurable
|
||
`ToneSource` (the same render helper `coop_tone` uses), it renders tones at a matrix
|
||
of common formats (44100/48000/96000 Hz, mono/stereo/5.1, 16-bit PCM / 32-bit float)
|
||
and asserts the hook reports the right rate/channels/bits + provenance for **both**
|
||
code paths: **see-init** (hooks installed first → exact `Initialize` format) and
|
||
**guess** (render client pre-exists → device-mix guess whose true rate is measured
|
||
from the cadence, the Brotato/Godot case). Also checks the frames reached the ring
|
||
non-silent **and that the hook muted the game's local playback** (the no-echo guarantee)
|
||
for *both* paths — the guessed-path mute assertion is the regression guard for the
|
||
double-audio bug. Skips cleanly with no audio endpoint.
|
||
- **`srgb_format_test`** — unit test of the `srgb_to_unorm` mapping the hooked
|
||
video path uses so `*_SRGB`-backbuffer games aren't darkened. No device.
|
||
- **`opengl_hook_test`** — in-process self-test of the OpenGL capture path:
|
||
installs the swap hooks, drives a real OpenGL context (clears the backbuffer to a
|
||
known color, calls `SwapBuffers`), and asserts the detour fired, the frame was
|
||
`glReadPixels`'d into the shared texture, and a second device reads the exact
|
||
pixels back by name. Skips cleanly without an OpenGL / D3D11 device.
|
||
- **`dx12_present_hook_test`** — in-process self-test of the Present hook's **D3D12**
|
||
path: drives a real D3D12 swapchain through the (shared) `IDXGISwapChain::Present`
|
||
vtable and asserts the D3D11On12 bridge wraps the `ID3D12Resource` backbuffer and
|
||
copies it into the shared texture, then reads the exact rendered color back by
|
||
name. Skips cleanly without a D3D12 device.
|
||
- **`present_hook_test`** — in-process self-test of the Present-hook video path:
|
||
installs the hook, drives a real D3D11 swapchain in the same process (clears the
|
||
backbuffer to a known color and calls `Present`), and asserts the detour fired,
|
||
the backbuffer reached the shared keyed-mutex texture, and a second device can
|
||
open it by name and read the exact pixels back. Skips cleanly if the machine has
|
||
no D3D11 device.
|
||
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone configurable WASAPI
|
||
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) at several source
|
||
formats (device default, 44100/48000/96000 Hz) and verifies the shipping
|
||
process-loopback capture (the fallback backend) receives non-silent audio by PID for
|
||
each — confirming loopback is format-agnostic (it captures post-mix at the device
|
||
endpoint format). Skips cleanly if the machine has no audio endpoint.
|
||
- **`mock_game_test`** — comprehensive capture/audio/hook stress test against
|
||
**`coop_mock_game`** (an animated, frame-numbered A/V test game under
|
||
[`tools/mock_game`](tools/mock_game) with selectable **DX9 / DX9Ex / DX10 / DX11 / DX12 /
|
||
OpenGL / Vulkan** backends and a configurable WASAPI tone). It launches the game, injects
|
||
`coop_hook.dll`, opens the hook's shared video texture, and **decodes the frame number out of
|
||
the captured pixels** to assert the mirror sees a *monotonic, advancing* sequence for each
|
||
backend (the bar for no dropped / stale / out-of-order frames — what the DX12 rotating-backbuffer
|
||
bug broke). **Vulkan** is special: since its present pointer is cached at init, the test launches
|
||
the mock **suspended**, injects, then resumes (the mock loads Vulkan and waits so the hook arms
|
||
first) to decode frames; it also exercises the **implicit capture layer** (registered via
|
||
`VK_LAYER_PATH` / `VK_INSTANCE_LAYERS`, decoding frames through the loader-inserted layer) and
|
||
late-injects a Vulkan game to assert the **too-late** flag trips (which drives the host's
|
||
relaunch banner). It launches the game at several **audio formats**
|
||
(44100/48000/96000, PCM + float) and asserts the hook measures each one's rate through the full
|
||
inject path, then injects with audio + video, checks both stream, cycles the audio subsystem
|
||
off/on (hook/unhook stress), and confirms the game never crashes and capture resumes. It then
|
||
runs an **aggressive hook/unhook storm** — a separate thread thrashes *every* subsystem on/off as
|
||
fast as the worker reconciles while the game is actively presenting, across **all** backends — to
|
||
catch an install/remove race that frees a hook's shared D3D / Vulkan state under a live capture
|
||
detour (the use-after-free that crashed a real game when its "Mirror video" button was spammed).
|
||
This suite drove out five real audio races plus the cross-backend unhook race (see Lessons
|
||
learned). Skips cleanly without a D3D11 / Vulkan device.
|
||
|
||
### Debugging the hooks against a real game
|
||
|
||
[`tools/audio_probe`](tools/audio_probe) (`coop_audio_probe.exe <pid> [seconds]`)
|
||
brings up the audio render-hook without Steam / RPT / the host UI: it creates the
|
||
IPC block + audio ring the hook expects, injects `coop_hook.dll` into the target
|
||
game, then drains the ring and prints per-stream format, captured-frame counts,
|
||
peak amplitude (proves the audio is real, not silence), and overruns. It enables
|
||
the hook's file trace (`%TEMP%\coop_hook.log`) for the run.
|
||
|
||
[`tools/audio_validate`](tools/audio_validate) (`coop_audio_validate.exe`) quantifies
|
||
audio-capture **fidelity** — it turns "the audio sounds slightly off" into numbers. With no
|
||
args it plays a known sine (`coop_tone` at 44100 Hz on a 48000 Hz endpoint, the Godot/Brotato
|
||
case), injects the hook exactly as the host does (late attach), captures the ring, and runs the
|
||
analyzer ([`common/include/coop/tone_analysis.hpp`](common/include/coop/tone_analysis.hpp)):
|
||
**pitch error in cents** (detects a mis-measured rate), **SNR/THD**, and **click / dropout**
|
||
counts. It also dumps a `.wav` so the capture can be *listened* to. Modes: `--render` drives the
|
||
real `AudioMirror` and measures its *rendered* output (surfaces the under-run / re-prime gaps the
|
||
capture side can't show — see Lessons learned); `--baseline` / `--selfcheck` establish the
|
||
measurement floor; `--listen <pid>` passively records a live process's output (point it at a
|
||
running `coop_host` to hear/quantify exactly what a guest gets on a real game); `--wav <file>`
|
||
analyzes any recording.
|
||
|
||
[`tools/input_probe`](tools/input_probe)
|
||
(`coop_input_probe.exe <pid> [seconds] [disable_mask]`) does the same for input: it
|
||
injects, reports one connected pad, and toggles a button each second so the game's
|
||
input layer sees a real state change. `disable_mask` (hex bits `0x1`=input
|
||
`0x2`=focus `0x4`=audio `0x8`=video) skips installing a subsystem, so you can
|
||
**bisect which injected subsystem affects a game** — this is how the 32-bit
|
||
Present-hook crash was isolated.
|
||
|
||
[`tools/vk_validate`](tools/vk_validate) (`coop_vk_validate.exe <layer|inject> [seconds] [exe]`)
|
||
validates the **Vulkan capture backend against a real game** (defaults to Sphere Spectacle). It
|
||
drives both early-presence methods — the implicit **layer** (registers `coop_vk_layer` scoped to the
|
||
game, launches via Steam, also late-injects `coop_hook.dll` for focus-spoofing so the game renders
|
||
unfocused) and **inject** (suspended-launch the exe *with the game's own folder as the working
|
||
directory* + early-inject before `vkCreateInstance`) — and asserts frames reach the shared texture
|
||
and advance, the captured resolution/colors are sane, saves a BMP screenshot for visual
|
||
confirmation, and **gates on the game keeping a healthy present rate while capturing** (so a
|
||
present-thread stall fails the tool, not just gets reported). Confirmed against Sphere Spectacle:
|
||
**both** paths mirror it correctly (1920×1080, right colors, no swizzle/darkening) at the game's full
|
||
present rate. The suspended-inject path needs the correct working directory (the game loads
|
||
`steam_api64.dll` / `resources/` relative to cwd) and only applies to titles that actually run when
|
||
launched directly; a title that refuses to run outside Steam produces no presents, and the tool
|
||
SKIPs inject for it (use the layer).
|
||
|
||
Both auto-detect a 32-bit (WOW64) target and inject via `coop_inject_x86.exe` +
|
||
`coop_hook_x86.dll`, exactly like the host. The probes build into
|
||
`bin/<config>/tools/` (the deployable `bin/<config>/` root holds only shipping
|
||
artifacts; tests build into `bin/<config>/tests/`) and resolve `coop_hook.dll` from
|
||
the root one level up, so run them from there. **Kill the game between runs** — the
|
||
loaded DLL locks `coop_hook.dll` against the next rebuild.
|
||
|
||
A debug-only **test harness** drives the host overlay's own code paths (inject / enable
|
||
audio / re-measure / override / screenshot / read state) without simulating mouse input,
|
||
for scripted UI validation. Build it with `-DCOOP_TEST_HARNESS=ON` (off by default, so
|
||
the shipped host never contains it); the host then reads one command line from
|
||
`%TEMP%\coop_test_cmd.txt` and replies in `%TEMP%\coop_test_resp.txt`. See
|
||
[`host/src/test_harness.hpp`](host/src/test_harness.hpp).
|
||
|
||
## Running the tool (manual, end-to-end)
|
||
|
||
This needs Steam, a donor game that supports Remote Play Together, and a second
|
||
person/account to receive the stream.
|
||
|
||
1. **Launch the host under a donor appid.** Find the donor's appid (the number in
|
||
its store URL); the donor only needs RPT support and is never actually played:
|
||
|
||
```text
|
||
"C:\Program Files (x86)\Steam\steam.exe" -applaunch <donorAppId> "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe"
|
||
```
|
||
|
||
The borderless window appears and Steam marks the donor "running". If the donor
|
||
ignores the trailing path, set the host as the donor's **Launch Options**
|
||
(`"D:\...\coop_host.exe" %command%`) or use a launcher like RemotePlayDetached.
|
||
|
||
2. **Start the real game** windowed or borderless (not exclusive fullscreen — see
|
||
Lessons learned). In the host's **Injection** panel, filter for the game's
|
||
`.exe`, select it, and click **Inject & Connect**. Watch **Hook status** for
|
||
**Attached**, a non-zero **XInput polled: N/s**, and **Focus spoof: active**.
|
||
|
||
3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** —
|
||
the host window now shows a live, letterboxed copy of the game. **Source**
|
||
picks how the frames are grabbed: **WGC** (default, Windows Graphics Capture —
|
||
works for any window) or **Hooked (Present)** (the injected hook's shared
|
||
texture — lower latency and no capture border, for DXGI / D3D11 and OpenGL
|
||
games; selecting it installs the video subsystem in the game).
|
||
|
||
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
|
||
With the hook injected, **Source** shows **Hooked (no echo)** and the game's
|
||
local playback goes silent while guests still hear it. If it shows **Loopback
|
||
(echo)** the hook's render path wasn't caught and you'll hear the game twice
|
||
locally (guests still hear it once). The **Render streams** table shows how
|
||
many WASAPI streams the game emits.
|
||
|
||
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
|
||
sees the mirrored video, hears the audio, and that their controller drives the
|
||
real game.
|
||
|
||
Useful checks while developing without RPT: tick **Forward synthetic test input**
|
||
in the Injection panel to make the game move on its own (proving forwarding is the
|
||
source), and click away from the game to confirm focus spoofing keeps it running.
|
||
|
||
> Injection access error → run the host as administrator. A 32-bit (WOW64) target
|
||
> is injected automatically via `coop_inject_x86.exe` + `coop_hook_x86.dll`; if
|
||
> those aren't next to the host, rebuild (the x86 sub-build stages them there).
|
||
|
||
## Lessons learned
|
||
|
||
Non-obvious things that cost time and constrain the design:
|
||
|
||
- **RPT only streams the *focused* window.** The game can't hold focus itself, so
|
||
the hook spoofs it (`GetForegroundWindow` / `GetActiveWindow` / `GetFocus` +
|
||
swallowing deactivation messages) to keep the game polling and rendering while
|
||
the host owns real OS focus.
|
||
- **Run target games windowed or borderless, never exclusive fullscreen** —
|
||
exclusive fullscreen minimizes on focus loss (defeating the spoof) and can't be
|
||
window-captured. While unfocused the game gets no OS keyboard/mouse, only the
|
||
forwarded pad.
|
||
- **WGC captures occluded windows but not minimized ones.**
|
||
- **Process-loopback capture doesn't mute the source.** Capturing a process's
|
||
render doesn't stop it reaching the speakers, so the no-echo path instead injects
|
||
a WASAPI render-hook that copies each buffer then releases it with
|
||
`AUDCLNT_BUFFERFLAGS_SILENT`; loopback stays as the (echoing) fallback.
|
||
- **`ActivateAudioInterfaceAsync` needs an *agile* completion handler.** If the
|
||
handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected
|
||
**synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of
|
||
apartment, device path, or activation params. (WRL/wil samples make the handler
|
||
agile for you.) Process loopback also needs the Win10 20H1 headers
|
||
(`NTDDI_VERSION ≥ 0x0A00000B`).
|
||
- **COM methods have no exports, so hooks walk vtables by frozen-ABI index — count
|
||
exactly.** All instances of a coclass share one vtable, so hooking one object's
|
||
slot catches every instance; but `IAudioClient::GetService` is **14**, not 13
|
||
(`SetEventHandle` sits at 13 between `Reset` and `GetService`). Count every
|
||
inherited `IUnknown`/base method when adding a hook.
|
||
- **D3D12 capture copies the *rotating* back buffer, not `GetBuffer(0)`.** D3D11
|
||
flip-model keeps `GetBuffer(0)` pointing at the live back buffer, but D3D12 rotates
|
||
buffers explicitly — the game renders into the buffer at
|
||
`IDXGISwapChain3::GetCurrentBackBufferIndex()`, which advances each `Present`.
|
||
Grabbing buffer 0 copies a stale buffer on N-1 of every N frames, so the mirror
|
||
silently runs at refresh/N — yet the Present counter, published FPS, generation, and
|
||
latency all read full rate (they count Presents, not unique content), so the metrics
|
||
look perfect while the eye sees missing frames. Query the current index right before
|
||
the trampoline `Present` (that's the just-rendered buffer) and copy that one.
|
||
- **Keep the D3D12 capture copy off the game's present queue, but ordered after its
|
||
frame.** The D3D11 path copies on the game's immediate context, so it's naturally
|
||
ordered after the frame and on the game's own timeline. For D3D12 the D3D11On12
|
||
bridge needs a queue: running the copy on the *game's* present queue orders it
|
||
correctly but stalls the game's own presents (GPU back-pressure, plus the shared
|
||
keyed-mutex `AcquireSync` is a **CPU-blocking** call on the render thread). Running
|
||
it on an independent queue avoids the stall but races the game's render → stale
|
||
frames. The fix is both: run the copy on **our own** queue, and order it with a
|
||
**fence** the game's present queue signals after its frame (a near-free op) and our
|
||
queue waits on. The present queue is recovered for late injection by hooking
|
||
`ID3D12CommandQueue::ExecuteCommandLists` (the per-frame method, not creation). Make
|
||
the producer-side `AcquireSync` non-blocking (`timeout 0`) so a busy mutex drops a
|
||
*mirror* frame instead of stalling the game; the Video panel's "Frames lost" line
|
||
surfaces both capture- and display-stage drops.
|
||
- **A D3D10 game's backbuffer lies about being D3D11.** A pure-D3D10 swapchain's backbuffer
|
||
QIs to `ID3D11Texture2D` *successfully*, but that D3D11 view reads back **empty** — the
|
||
rendered content only exists on the game's own D3D10 device. (And a D3D11 backbuffer QIs to
|
||
`ID3D10Texture2D` too, so `GetBuffer` alone can't tell them apart.) The reliable signal is
|
||
that a feature-level-10 device **rejects** `CreateTexture2D` with the NT-handle keyed-mutex
|
||
share flags (`E_INVALIDARG`): so try the D3D11 fast path, and on that failure switch (sticky)
|
||
to reading the backbuffer through the game's **D3D10** device into a staging texture and
|
||
`UpdateSubresource`-ing it into the shared texture on a **hook-owned D3D11 device** (the game
|
||
has no usable D3D11 device of its own). The staging `Map` blocks until the GPU copy completes,
|
||
so there's no cross-device race.
|
||
- **D3D9 capture: one read-back path covers plain D3D9 *and* D3D9Ex.** D3D9 has no DXGI
|
||
swapchain, so inline-hook `IDirect3DDevice9::Present` (vtable index **17**, found from a
|
||
throwaway device — clean prologue, so inline is safe; `stdcall()` on x86). A D3D9 surface
|
||
isn't D3D11-shareable, so read the backbuffer back with `GetRenderTargetData` into a
|
||
`D3DPOOL_SYSTEMMEM` surface (`LockRect` blocks until the copy → no race), **swizzle BGRA→RGBA**
|
||
(`X8R8G8B8`/`A8R8G8B8` store as little-endian `0xAARRGGBB` → bytes B,G,R,A) and upload into the
|
||
shared texture on a hook-owned D3D11 device. Both plain D3D9 and D3D9Ex call this same Present,
|
||
so **one path serves both** — the GPU shared-surface fast path the plan sketched for D3D9Ex
|
||
wasn't worth it (D3D9 games are light enough that the read-back cost is fine, and it dodges the
|
||
cross-device keyed-mutex-less sync of a legacy shared surface).
|
||
- **The OpenGL mock needs no GL loader.** Drawing the animated pattern with scissored clears
|
||
(`glClear` + `glScissor`) only touches **GL 1.1**, which `opengl32` exports directly — so the
|
||
planned **glad** submodule wasn't needed (a legacy `wglCreateContext` + `<GL/gl.h>` suffices).
|
||
GL's framebuffer is **bottom-left** origin and the capture flips it top-down, so the mock draws
|
||
the frame-counter block at the GL *top* (`y = h - block`) to land at the captured image's
|
||
top-left. The GL `SwapBuffers` hook now also bumps the shared present counter (it's the GL
|
||
present), so the Video panel's present rate works for GL games too.
|
||
- **Vulkan can't be late-hooked, and capturing its present needs semaphore surgery.** Vulkan
|
||
games cache `vkQueuePresentKHR` at init (often via volk, which bypasses the loader trampoline),
|
||
so late injection misses it — the hook must be in **before `vkCreateInstance`**. Catch the
|
||
resolution chain instead: inline-hook the `vulkan-1.dll` `vkGetInstanceProcAddr` export and hand
|
||
back wrappers for `vkCreateInstance` / `vkCreateDevice` / `vkCreateSwapchainKHR` /
|
||
`vkQueuePresentKHR` (so a volk app resolves *ours*), tracking the device / queue / images /
|
||
format; read the presented image back with `vkCmdCopyImageToBuffer` and upload it to the shared
|
||
texture on a hook-owned D3D11 device. The trap: the read-back submit must **wait on the present's
|
||
wait semaphores** (consume them) and **signal a fresh semaphore the real present then waits on**
|
||
— both your copy and the present can't wait the same binary semaphore. "Injected too late" is
|
||
detected as *vulkan-1.dll loaded + hook in for >4 s + we never saw `vkCreateDevice`* (the app set
|
||
everything up before us) → red relaunch banner, WGC meanwhile. Testing needs an **early-load**
|
||
path (suspended launch + inject + resume; the mock loads Vulkan and waits), since the normal
|
||
late-inject flow can't catch a Vulkan present.
|
||
- **The reliable Vulkan path is a real implicit layer — and the loader tags its link structs with
|
||
small `sType`s.** Games that init Vulkan instantly resolve their present pointer before any
|
||
inject can land, so the only robust early presence is an implicit layer (`coop_vk_layer`) the
|
||
loader inserts at `vkCreateInstance`. It must do **chain dispatch** (pull
|
||
`pfnNextGetInstanceProcAddr` / `pfnNextGetDeviceProcAddr` out of the `VkLayer*CreateInfo` link in
|
||
`pNext`, advance the link, call down) rather than inline-hook. The trap that cost time: those
|
||
link structs use **loader-internal `sType` values 47 (instance) / 48 (device)** — *not* the
|
||
`1000000000`-range I assumed (and `vk_layer.h` isn't in Vulkan-Headers, so the structs are
|
||
hand-declared) — matching the wrong value silently failed the device-chain walk
|
||
(`vkCreateDevice: Failed to create device chain`). An implicit layer loads into *every* Vulkan
|
||
app, so it self-scopes: capture only when the process image matches the host-written target file,
|
||
else pure pass-through. Register it per-user (HKCU `…\Vulkan\ImplicitLayers`, no admin) and
|
||
unregister on untick / host exit.
|
||
- **Both Vulkan early-presence paths work on a real game — the "only the layer works" belief was two
|
||
bugs in disguise.** Against Sphere Spectacle (`coop_vk_validate`), the inline **inject** path
|
||
(suspended-launch + early-inject before `vkCreateInstance`) initially captured nothing, which was
|
||
wrongly written off as "the title requires launching through Steam." It does not. Two real causes:
|
||
(1) the harness launched the exe with **our** working directory, so the game couldn't load
|
||
`steam_api64.dll` / `resources/` and never rendered — launch it with the game's own folder as cwd
|
||
and it runs fine directly; and (2) the game (volk-style) resolves `vkQueuePresentKHR` /
|
||
`vkCreateSwapchainKHR` via **`vkGetInstanceProcAddr`**, but `vk_hook` only substituted those when
|
||
resolved via `vkGetDeviceProcAddr` — so the present bypassed us. Intercepting them in the
|
||
instance-level resolver too fixed it. The division of labor is about *who launches the game*, not
|
||
capability: for a game launched **through Steam** we can't suspend the launch, so the implicit
|
||
**layer** (always in the loader chain) is the path; when **we** launch the exe, suspended-inject
|
||
works (and is how you'd debug). When the layer runs, the game still needs `coop_hook.dll`
|
||
co-injected for **focus-spoofing** so an unfocused window doesn't throttle itself.
|
||
- **A Vulkan present-rate *is* a valid no-FPS-impact gate — once the read-back is off the present
|
||
thread.** The earlier validator reported the present rate and rationalized it ("the game's own
|
||
cadence; a hard FPS gate is meaningless"), which hid a real **144→3 FPS** stall: the read-back ran
|
||
on the present thread and spent ~370 ms/frame doing a CPU read of write-combined staging memory.
|
||
After moving the read-back to a reaper thread (`coop::hook::VkCapture`), `present_calls` (counted
|
||
every present, independent of the throttled mirror) reflects the game's true rate, so the tool now
|
||
**asserts** it stays healthy while capturing. The mirror is deliberately throttled to ~150 Hz, so
|
||
`copied < present_calls` is expected, not a drop. Lesson: a perf check must assert a bound — if you
|
||
find yourself explaining why a number is fine, make the test prove it.
|
||
- **A render client that predates our injection has no knowable format — measure it.**
|
||
We inject into already-running games, so we usually never see the game's
|
||
`IAudioClient::Initialize`; the render-hook then assumes the device mix format for that
|
||
stream. That's wrong for games that render at a non-device rate via WASAPI
|
||
`AUTOCONVERTPCM` (e.g. Godot / Brotato render 44100 Hz while the endpoint mixes at
|
||
48000), so the captured audio plays back **pitch-shifted up**. Fix: treat such a format
|
||
as a *guess* and measure the stream's true sample rate from its render cadence
|
||
(frames/sec over a short active window, snapped to the nearest standard rate) before
|
||
publishing it, deferring capture until verified. Discard the first measurement window:
|
||
the moment we attach, the stream's already-queued buffers arrive in a burst that
|
||
over-counts (the `audio_hook_test` matrix caught this), so measure the next,
|
||
steady-state window. **Only the rate is recoverable, though** — channels/bit-depth can't
|
||
be measured (`AUTOCONVERTPCM` hands `GetBuffer` a *fixed* staging buffer, so there's no
|
||
buffer stride; confirmed empirically) and WASAPI has no API for a pre-existing client's
|
||
format, so they stay the device-mix guess. That's right for the common case (engines
|
||
render stereo float = the endpoint), and a `VirtualQuery` clamp on the capture copy
|
||
keeps a too-large guessed block from ever over-reading the source buffer. Streams we *do*
|
||
watch get created carry their exact `Initialize` format, and the loopback fallback
|
||
captures post-mix at the device format (always correct). The Audio panel shows each
|
||
stream's provenance (known / measuring / measured rate (ch/bits assumed)) so what the
|
||
mirror is using is always visible — see Limitations.
|
||
- **A short measurement window *rejects* a bad reading, it can't *average* it away.**
|
||
Measuring the rate over ~200 ms made one extra ~10 ms WASAPI buffer at a window edge a
|
||
~5% error, which lands *between* standard rates (they're >8% apart) — so 44100 read as
|
||
~46205 and got published verbatim. The robust fix isn't just a longer window: it's to
|
||
**refuse any window that doesn't snap to a standard rate** and require **consensus**
|
||
across windows, since a quantization/burst error big enough to miss the right rate
|
||
lands in no-man's-land rather than on a wrong neighbour. Only commit a non-standard
|
||
estimate as explicitly *low-confidence* (shown red). The operator can also re-measure
|
||
or override the format via a per-stream `AudioRingHeader` op channel; the host rebuilds
|
||
its render client when `format_generation` bumps, so it takes effect live.
|
||
- **Two capture paths beat one guess: correlate the hook against the loopback.** Cadence
|
||
measurement *rejects* a bad rate reading but is still a guess from one signal, and it can't recover
|
||
channels/bit-depth at all. But during the measurement window the game is still audible, so we have
|
||
the *same* audio twice: the **hook** (pre-mix, unknown format) and a **process-loopback** (post-mix,
|
||
the *known* device format — it's the hook signal resampled by WASAPI's AUTOCONVERTPCM). Resampling
|
||
the hook by each candidate rate and cross-correlating against the loopback pins the true rate from
|
||
ground truth: the right rate holds alignment across the whole window (score ≈1.0); a wrong rate
|
||
time-warps the hook so a single alignment can't hold and the correlation collapses (≈0). The catch
|
||
that makes this need a measurement *tap*: the no-echo path **silences** the game, so a loopback of a
|
||
silenced game is silent — the co-capture must happen while the stream is still being measured (not
|
||
yet published, so not yet silenced). A host-set `verify_capture` ring flag makes the hook push the
|
||
guessed stream's raw pre-mix bytes (no silence) during that window; the host
|
||
(`audio_format_verifier`) co-captures both, correlates (`coop/audio_correlate.hpp`), and feeds a
|
||
correction into the existing override channel. **Rate vs layout are coupled, though**: correlating
|
||
the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step
|
||
assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a
|
||
*different* channel count / bit depth is the layout step, which tries candidate de-interleavings and
|
||
keeps whichever correlates. **The layout step needs a self-describing tap**: the hook can't know a
|
||
guessed stream's real frame size, so it pads each render buffer to the *device* block — which
|
||
over-reads stale staging bytes for a stream with fewer channels/bits. The raw padded bytes are
|
||
un-decodable (the stale tail scrambles the audio), so the tap prefixes each buffer with its frame
|
||
*count* (`[count][count*device_block bytes]`); the host strips the padding per candidate layout
|
||
(take the real `count*real_block` of each chunk) before de-interleaving. **The genuinely-ambiguous
|
||
case stays unresolved**: a stream whose channels carry identical content is indistinguishable from
|
||
one channel at double the rate (`2ch@R` == `1ch@2R` byte-for-byte), so the correlator reports low
|
||
confidence and the format stays the device assumption rather than guessing wrong.
|
||
- **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The
|
||
mirror re-renders the captured ring to the output device. The original feed loop re-primed
|
||
(withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free
|
||
buffer that tick (`to_write < avail`). But a partial fill is *normal* producer jitter — and
|
||
withholding the feed drains the device, so a one-frame ring dip became a full ~30 ms drop-out.
|
||
On a jittery game it fired constantly → choppy, "metallic" mirror audio, intermittent by
|
||
buffer phase. Fix: feed whatever is available every tick; re-prime **only** on a genuine
|
||
starvation (device buffer empty *and* ring empty). The policy is factored into a pure
|
||
`RenderPacer` reused by the loop and unit-tested headlessly (`render_pacer_test`: on a jittery
|
||
schedule the old policy withholds available data ~168× and drains the cushion to one period
|
||
from silence; the new one never withholds). **Diagnosing it needed device-side capture, not a
|
||
write-side tap** — the artifact is silence the device *plays* during an under-run, not bytes
|
||
the loop writes, so a tap on the write would look clean. `coop_audio_validate` quantifies it
|
||
by loopback-capturing the *rendered* output (pitch error in cents, SNR, click/dropout counts)
|
||
and contrasts the capture ring (pristine), the measurement floor, and the render path.
|
||
- **Mute the game with the SILENT flag, not a `memset` — they are not the same thing.** The
|
||
hooked path's whole point is "no echo": capture the game's frames into the ring AND silence
|
||
its local playback, so the only audio is the host's re-render. The mute was implemented by
|
||
*zeroing the buffer* and releasing it with `AUDCLNT_BUFFERFLAGS_SILENT`. But zeroing
|
||
`num_frames * block` is only safe when `block` is the real frame size — for a **guessed**
|
||
format (late attach, the Brotato case) the guessed block can exceed the real buffer, so the
|
||
zero would over-write adjacent memory. The original code's conservative answer was to skip the
|
||
whole mute for guessed streams — which left the game **audible**: it played locally *and* the
|
||
mirror re-rendered the same audio a few ms later = a metallic, slightly-out-of-sync double
|
||
(exactly the reported symptom). The fix: `AUDCLNT_BUFFERFLAGS_SILENT` already makes WASAPI
|
||
*ignore the buffer contents* and play silence — it mutes with **no write at all**, so it's safe
|
||
for any format. Decouple the two: always mute via the flag; keep the `memset` only for an exact
|
||
format (belt-and-suspenders). `audio_hook_test` now asserts the mute engages for both the
|
||
exact and the guessed path (the guessed assertion fails on the old code — write the failing
|
||
test first), and the byte-incompatible cases prove the flag-mute never over-writes.
|
||
- **Panels must *fit* their assigned size at max info -- measure it, don't eyeball it.** The
|
||
overlay opens panels at fixed sizes that scale with the monitor, so with Debug details on a
|
||
dense panel can overflow and scroll content out of view. `ui_fit_test` drives the real
|
||
Controllers + Audio panels headlessly (null-backend ImGui: build the atlas with
|
||
`GetTexDataAsRGBA32`, set a non-null `TexID`, `Render()` needs no GPU) at reference
|
||
resolutions and asserts each window's `ScrollMax` is 0 -- read it on the *2nd+* frame, since
|
||
ImGui computes `ScrollMax` in `Begin()` from the *previous* frame's content size. Two fixes
|
||
fell out: the center column was too narrow (horizontal overflow -> widen it, merge the
|
||
Controllers poll/round-trip tables), and a *static* height split can't serve both modes, so
|
||
it's now **Debug-details-aware** (debug on -> Controllers/Audio get the height for their
|
||
tables; off -> Video's perf graphs are the tall content, since Video's height is
|
||
mirroring-driven, not debug-driven). The host's `uisize`/`uifit` harness commands check the
|
||
same thing against the live overlay.
|
||
- **Drive the ImGui overlay for tests through an IPC harness, not synthetic input.**
|
||
`PostMessage`-d mouse clicks don't reliably reach ImGui widgets, and key/coordinate
|
||
simulation is brittle. A tiny debug-only command channel (`-DCOOP_TEST_HARNESS`, file-
|
||
based) that calls the *same* code the buttons do — and replies with state — makes UI
|
||
validation deterministic and scriptable, and is compiled out of the shipped product.
|
||
- **A capture-style stress test against a frame-numbered mock game is worth a lot.** An
|
||
animated game that encodes its frame number in the pixels lets a test assert the mirror
|
||
shows a *monotonic, advancing* sequence (the bar for "no dropped / stale / out-of-order
|
||
frames"), and toggling subsystems while it renders flushes out concurrency bugs. This
|
||
one caught **five** real audio races: a `memset` over-*write* past a guessed stream's
|
||
real buffer (zeroing 8ch into a 2ch buffer corrupts adjacent audio memory → crash; the
|
||
fix: capture but don't silence a guessed stream), a null call through a vtable hook's
|
||
`original` after unhook (keep it valid), a non-atomic `g_ipc` TOCTOU, a stale
|
||
GetBuffer/ReleaseBuffer pairing across a hook toggle (epoch-stamp it), and COM-object
|
||
churn from re-creating the probe each toggle (build it once, keep it, only swap vtable
|
||
slots). **Silently silencing/zeroing a buffer whose true size you only guessed is an
|
||
over-write, not just an over-read** — clamp the read, but don't write what you can't size.
|
||
- **Every removable hook needs the same safe-unhook drain, not just audio.** The audio hooks
|
||
learned to restore the vtable slot first and *drain in-flight detours* before tearing down the
|
||
shared state they read; the video (Present / D3D9 / D3D10 / OpenGL / Vulkan) and the
|
||
XInput / focus / MKB hooks did not — `remove_*` freed the hook's shared D3D device / keyed-mutex
|
||
texture (or the Vulkan read-back resources, or the IPC pointer) *immediately*. So spamming a
|
||
subsystem toggle (the "Mirror video" button) freed that state while a capture detour was still
|
||
mid-flight on the game's render thread → use-after-free → the game crashed (Brotato, on its
|
||
OpenGL path; reproduced across every backend by the `mock_game_test` storm). The generalised fix
|
||
(`hook/src/hook_guard.hpp`, `DetourGate`): every detour wraps its body in an RAII active-count
|
||
`Guard`; `remove_*` (1) restores the hook so **no new detour can start** — reset the inline hook
|
||
(SafetyHook's mutex-guarded call wrappers make any in-flight trampoline call safe), or, for the
|
||
focus WNDPROC subclass, restore the window proc — then (2) `drain()`s the active count to zero,
|
||
and only **then** (3) frees the shared state. **Vulkan is the exception**: the game caches our
|
||
`hk_vkQueuePresentKHR` pointer at resolution time and keeps calling it even after the GPA hook is
|
||
reset, so a reset can't stop new detours — instead removal closes an atomic **capture gate**
|
||
first (the detour then passes straight through to the real present without touching the read-back
|
||
state), drains, and only then frees. The drain is bounded (~400 ms) so a wedged game thread can't
|
||
hang the worker; detours are micro- to milliseconds, so it returns almost immediately.
|
||
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
|
||
the backbuffer inside the game's `Present`, which the game issues at its true render
|
||
rate regardless of how DWM composites that *window*. So an unfocused game window can
|
||
judder (DWM under-composites background windows; only the focused window gets VRR /
|
||
independent flip) while the mirror — which receives every `Present` — stays smooth.
|
||
This is why the game window not being focused doesn't matter: it isn't the surface
|
||
anyone sees. The same focus rule explains why an unfocused *tool* window can render
|
||
below the game's rate (it loses VRR), so in use the mirror is the focused window.
|
||
- **SafetyHook on x86 has two traps that froze 32-bit Slaps and Beans.** (1)
|
||
`InlineHook::call()` invokes the trampoline as `__cdecl`, but most targets are
|
||
`__stdcall` (COM methods like `IDXGISwapChain::Present`, the WASAPI interfaces,
|
||
`WINAPI` `SwapBuffers`); on 32-bit that double-cleans the stack → ESP imbalance →
|
||
crash (Debug: **Run-Time Check Failure #0**). Use **`stdcall()`** (a no-op on
|
||
x64). (2) Don't *inline-hook* COM methods on x86 at all: MMDevApi/AudioSes
|
||
prologues do `push ebp; mov ebp,esp; and esp,-8` and read args **EBP-relative**,
|
||
which SafetyHook's trampoline relocation breaks (the original then runs with
|
||
garbage args and faults). Hook COM methods by **swapping the vtable entry**
|
||
instead (`VirtualProtect` the slot, overwrite the pointer, call the saved
|
||
original) — no code patching, pristine stack regardless of prologue. Inline
|
||
hooking stays fine for `Present`/`SwapBuffers` (clean prologues). Guarded by the
|
||
x86 hook tests.
|
||
- **Steam Input init suppresses XInput.** Initializing Steam Input turns on Steam's
|
||
in-process XInput interception, which hides controllers from `XInputGetState`
|
||
unless they're bound to the running appid's action set — defaulting to it
|
||
silently broke forwarding. XInput is primary; Steam Input is opt-in.
|
||
- **Forwarding keyboard/mouse means covering three read paths, each differently.** Games read
|
||
keyboard/mouse three ways and a forwarder has to satisfy all of them from one synthesized state.
|
||
(1) **Message-loop** games get `PostMessage`d `WM_KEYDOWN`/`WM_*BUTTON*`. (2) **Polling** games
|
||
get synthesized `GetAsyncKeyState`/`GetKeyboardState`/`GetCursorPos`. (3) **DirectInput** games
|
||
call `IDirectInputDevice8::GetDeviceState` — a COM method, so it's hooked by **vtable swap, not
|
||
inline** (the x86 COM-prologue trap), reading the vtable from a kept-alive *probe* device we
|
||
create (the game made its devices before we injected, but all DI devices share one vtable);
|
||
dispatch on `cbData` (256 = keyboard `BYTE[256]` indexed by **DIK scan-code**, not VK — map with
|
||
`MapVirtualKey(VK_TO_VSC)`; `sizeof(DIMOUSESTATE)` = mouse). (4) **Raw Input** games read
|
||
`WM_INPUT` → `GetRawInputData`, but get **no `WM_INPUT` while unfocused** — so we *synthesize* it:
|
||
post `WM_INPUT` with `lParam` = the address of one of our `RAWINPUT` slots, and the hooked
|
||
`GetRawInputData` serves that slot's data back (`RID_HEADER` and `RID_INPUT`). Relative raw mouse
|
||
*movement* isn't in the position-based MKB event stream, so raw-input forwarding covers keys +
|
||
buttons, not free-look — a known limitation.
|