BlackMark
3f1ce0c78a
From an in-depth review pass: confirmed bugs, correctness/robustness items to verify-then-fix, the D3D9/OpenGL present-thread readback perf fixes, cross-process ABI hardening, test-coverage gaps, the static-CRT + Vulkan-layer-cleanup features, three UX fixes, and stale-comment/VtableHook cleanup. Each lands test-first as its own commit and is removed from this list when done. Also drop the inaccurate completed-work-lives-in-Lessons-learned line -- done tasks live in git history; Lessons learned is only for genuinely important lessons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
821 lines
62 KiB
Markdown
821 lines
62 KiB
Markdown
# CoopAllTheThings
|
||
|
||
Steam **Remote Play Together (RPT)** for any XInput game — without breaking DRM,
|
||
achievements, or playtime.
|
||
|
||
Existing "donor game" tools (e.g. RemotePlayWhatever) copy a target game's files
|
||
into a donor game's folder and rename the executable so Steam streams the target
|
||
under the donor's appid. That breaks DRM-protected games, breaks achievements,
|
||
and credits playtime to the donor.
|
||
|
||
CoopAllTheThings takes a different approach: the **real game runs normally under
|
||
its own appid** (so DRM, achievements, and playtime all work), while a lightweight
|
||
**mirror app runs under the donor appid**. The mirror presents a borderless
|
||
window that is a live copy of the game's video + audio, and forwards the guests'
|
||
input back into the real game. Steam's RPT captures the mirror window — so any
|
||
XInput game becomes Remote-Play-Together-able.
|
||
|
||
The end-to-end path is working: launched under a donor appid, the host streams a
|
||
live video + audio mirror of a separately-running game over Remote Play Together
|
||
and forwards guest controllers back into it.
|
||
|
||
## Architecture
|
||
|
||
| Concern | Mechanism | Component |
|
||
| --- | --- | --- |
|
||
| Receive guest input | XInput (RPT delivers guest pads to the focused window); optional, opt-in Steam Input when built with the Steamworks SDK | `coop_host.exe` |
|
||
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` |
|
||
| Forward mouse + keyboard | Opt-in MKB subsystem: host streams its window's clicks/keys; the hook posts the matching window messages, synthesizes `GetAsyncKeyState`/`GetKeyboardState`/`GetCursorPos` for polling games, augments `IDirectInputDevice8::GetDeviceState` for **DirectInput** games, and synthesizes `WM_INPUT` + `GetRawInputData` for **Raw Input** games | `coop_hook.dll` + `coop_host.exe` |
|
||
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` |
|
||
| Mirror video (default) | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` |
|
||
| Mirror video (hooked) | Injected Present / OpenGL hook copies the backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` |
|
||
| Mirror audio | Injected render-hook copies each of the game's WASAPI render streams into its own shared ring and silences the game locally (no echo); the host mixes the streams (soft-clipped); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` |
|
||
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel, video/audio/log shares) | `common/` |
|
||
|
||
The hooked video path has three producers: **Direct3D (DXGI)** hooks
|
||
`IDXGISwapChain::Present` / `Present1` and copies the backbuffer — directly for
|
||
D3D11 games (the backbuffer is an `ID3D11Texture2D`), via a **D3D11On12
|
||
bridge** for D3D12 games (wrap the `ID3D12Resource` backbuffer, `CopyResource` into
|
||
the shared texture), and via a **D3D10 read-back** for D3D10 games (their backbuffer's
|
||
D3D11 view is empty and a feature-level-10 device can't host the shared texture, so
|
||
read it through the game's own D3D10 device and upload it via a hook-owned D3D11
|
||
device); **Direct3D 9** inline-hooks `IDirect3DDevice9::Present` (vtable index 17) and
|
||
read-backs the backbuffer with `GetRenderTargetData` (a D3D9 surface isn't D3D11-shareable),
|
||
swizzling BGRA→RGBA and uploading via a hook-owned D3D11 device — one path covers both plain
|
||
D3D9 and D3D9Ex; **OpenGL** hooks `SwapBuffers` / `wglSwapBuffers` and reads the
|
||
backbuffer with `glReadPixels` (for games that never touch DXGI, e.g. Phantom
|
||
Brave); and **Vulkan** inline-hooks the `vulkan-1.dll` `vkGetInstanceProcAddr` export to
|
||
intercept the resolution chain (`vkCreateInstance` / `vkCreateDevice` /
|
||
`vkCreateSwapchainKHR` / `vkQueuePresentKHR`) and reads the presented image back with
|
||
`vkCmdCopyImageToBuffer` — but only when the hook is present **before** the game initializes
|
||
Vulkan (it caches its present pointer at init), so the Vulkan path needs early presence via
|
||
Auto-attach or the opt-in implicit **capture layer** (`coop_vk_layer`, registered per-user by the
|
||
Injection panel's "Set up Vulkan layer" checkbox and scoped to the target game); a too-late attach
|
||
shows a red relaunch banner and falls back to WGC. The host samples the copy as plain UNORM
|
||
(`srgb_to_unorm`) so `*_SRGB`-backbuffer games mirror at correct brightness. **WGC remains the
|
||
default** and covers anything the hooked path doesn't.
|
||
|
||
## Limitations
|
||
|
||
- **Anti-cheat:** the input path injects `coop_hook.dll` into the target game.
|
||
Games protected by kernel-level anti-cheat (Easy Anti-Cheat, BattlEye,
|
||
Vanguard, etc.) will detect the injected module and may **kick the player or
|
||
issue a ban**. Such games are explicitly **out of scope and unsupported** — do
|
||
not use CoopAllTheThings with them. The tool targets single-player and
|
||
co-op/local-multiplayer titles without active anti-cheat.
|
||
- **XInput only:** the game must read controllers via XInput (the common case).
|
||
DirectInput-only / RawInput-only games are not handled.
|
||
- **32-bit games supported via a helper:** the host is x64, but the build also
|
||
produces an x86 hook DLL (`coop_hook_x86.dll`) and a 32-bit injector helper
|
||
(`coop_inject_x86.exe`). When the target is a 32-bit (WOW64) process the host
|
||
detects it (`IsWow64Process2`) and shells out to the helper to load the x86 DLL
|
||
(a 64-bit process can't cleanly inject a 32-bit one). The shared-memory IPC is
|
||
fixed-width / bitness-stable, so the x64 host and x86 hook interoperate.
|
||
- **Local audio echo on the fallback path:** when the render-hook is active it
|
||
silences the game's local playback while mirroring it, so there is no echo. If
|
||
the hook can't attach or the game uses an unhooked render path, the host falls
|
||
back to process-loopback capture, which does *not* mute the game — so the local
|
||
machine hears the audio twice (guests hear it once). The Audio panel shows which
|
||
path is active.
|
||
- **A pre-existing stream's format is *recovered*, not known — by measurement and
|
||
cross-correlation.** The tool injects into an already-running game, so the audio
|
||
render-hook usually never saw the game's `IAudioClient::Initialize`, and `AUTOCONVERTPCM`
|
||
hides the buffer stride (WASAPI exposes no API for a pre-existing client's format). The
|
||
hook first recovers the **sample rate** by measuring the render cadence (so playback pitch
|
||
is correct, e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), assuming the device's
|
||
channels/bit-depth. When the game is still audible (the measurement window), the host then
|
||
**cross-correlates the two capture paths** — the hook (pre-mix) against a parallel
|
||
process-loopback (post-mix, the known device format) — to verify/correct the rate and to
|
||
**recover the channels + bit depth** by trying candidate de-interleavings and keeping the one
|
||
that aligns (`audio_format_verifier` → `coop/audio_correlate.hpp`). The recovered format feeds
|
||
the existing override channel. The one case it can't resolve is a *genuinely ambiguous* layout
|
||
(a stream whose channels carry identical content looks the same as one channel at double the
|
||
rate); there it stays the device assumption and the correlation reports low confidence rather
|
||
than guess. A wrong/assumed layout is mirrored with the wrong de-interleaving (garbled) but
|
||
never an over-read/crash: the capture copy is clamped to the readable region (`VirtualQuery`),
|
||
and the local **mute** uses `AUDCLNT_BUFFERFLAGS_SILENT` (WASAPI ignores the buffer contents),
|
||
so it never *writes* the wrongly-sized buffer either. Every captured stream is silenced locally
|
||
— guessed or exact — so there's no echo on the hooked path, and the loopback fallback is always
|
||
format-correct. The Audio panel shows each stream's format provenance (*known* / *measuring* /
|
||
*measured rate* / *low-confidence* / *override*), and (under Debug details) lets the operator
|
||
**re-measure** or **override** the format when needed. Overrides are **remembered per game** (and
|
||
a format caught exactly at `Initialize` is auto-saved), so a known-bad game is corrected
|
||
automatically next launch. Streams created *after* injection are captured exactly.
|
||
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
|
||
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
|
||
for RPT; F2 frees the operator cursor; **F10 saves a PNG screenshot** (back buffer,
|
||
written next to the exe) regardless of window focus or occlusion.
|
||
|
||
## Roadmap
|
||
|
||
### Current work
|
||
|
||
From an in-depth review pass. Each item is fixed test-first (a failing test, then the fix) and lands
|
||
as its own commit; "verify" items are confirmed real before any change, and dropped if not.
|
||
|
||
Confirmed bugs:
|
||
- **Shutdown use-after-free of `ui`** — the ImGui settings handler holds `&ui`, but `~ImGuiLayer`'s
|
||
`DestroyContext` saves settings *after* `ui` (declared later in `run()`) is destroyed. Reorder /
|
||
remove the handler before teardown.
|
||
- **Host device loss unhandled** — `Present`/`ResizeBuffers`/`CreateRenderTargetView` HRESULTs are
|
||
ignored and a removed/reset device spins silently. Detect `DEVICE_REMOVED/RESET`, log
|
||
`GetDeviceRemovedReason`, surface it, and halt the render loop cleanly.
|
||
- **Keyed-mutex `WAIT_ABANDONED` not handled** — `shared_texture` treats it as "skip", so a mirror
|
||
never recovers after a host crash + reconnect. Treat it as acquired (copy + release).
|
||
|
||
Correctness (verify, then fix if real):
|
||
- **`vk_hook` / `vk_layer` `g_swaps`** — no synchronization on push/iterate and never pruned on
|
||
swapchain destroy (unbounded growth + stale-handle match). Add a guard + a `vkDestroySwapchainKHR`
|
||
prune.
|
||
- **`mkb_hook` raw-input slot reuse** — a 64-slot ring can overwrite an event before the game reads
|
||
the `WM_INPUT`. Verify; fix if real.
|
||
- **Non-atomic cross-process diagnostic counters** — `present_calls`, `frames_dropped`,
|
||
`frames_rendered` are plain `+=`/stores read by the host. Make them atomic.
|
||
- **`publish()` synthetic-input timing** — confirm it doesn't read ImGui/UI state from the input
|
||
thread; fix if so.
|
||
|
||
Performance (failing perf-regression test first):
|
||
- **D3D9 capture readback off the present thread** — move the swizzle/flip to an off-thread reaper
|
||
(the inline-readback pattern that caused the Vulkan 144→3 FPS stall), add a present-thread overhead
|
||
guard + dedicated test.
|
||
- **OpenGL capture readback off the present thread** — async PBO readback + off-thread swizzle/flip,
|
||
with an overhead guard.
|
||
|
||
Robustness (verify, then fix if real):
|
||
- **`mkb_forward` sticky mouse-up** — a held button isn't released when mirroring toggles off / ImGui
|
||
grabs the mouse. Release held buttons.
|
||
- **Swallowed audio loopback `start()` failures** — surface a process-loopback activation failure
|
||
distinctly from "no correlation" in the verifier + loopback.
|
||
- **Permissive injector bitness gate** — `IsWow64Process2` failure is treated as 64-bit; fall back to
|
||
`IsWow64Process` instead of mis-injecting.
|
||
- **Ignored HRESULT/BOOL returns** — `CreateShaderResourceView`, `GetClientRect`/`ClientToScreen`,
|
||
`XInputSetState`, `GetExitCodeProcess`: check and surface/handle.
|
||
|
||
Cross-process / ABI:
|
||
- **Tie `kProtocolVersion` to the layout** — `static_assert` `sizeof(SharedBlock)` + the full offset
|
||
set, and make `hook_selftest`'s `dump_layout` assert instead of only printing.
|
||
- **`log_ring` torn-text window** — verify the MPSC overwrite race; fix or bound it.
|
||
- **`narrow`/`widen` not inverse** — make the audio-override name round-trip lossless (or document).
|
||
|
||
Test coverage:
|
||
- **`log_ring`** — threaded push/drain + wrap-skip generation test (currently zero coverage).
|
||
- **Seqlock reader paths** — torn-read retry, odd-sequence skip, attempt-exhaustion → false.
|
||
- **Version/magic mismatch rejection** — negative test that the hook refuses a bad version.
|
||
- **Dedicated hook tests** — `focus_spoof`, `vk_hook` (present), `d3d9_hook`.
|
||
- **Misc units** — `SharedMemory` RAII/move, `wav` malformed input, `tool_paths` resolution, injector
|
||
bitness check; strengthen the `audio_ring` overrun-at-seam case.
|
||
|
||
Features:
|
||
- **Static CRT (`/MT`) for `coop_hook.dll`** (x64 + x86) so it loads in games without the VC++ redist.
|
||
- **Stale Vulkan-layer registration cleanup** — remove a leftover `HKCU` implicit-layer entry from a
|
||
crashed prior run on host startup.
|
||
|
||
UX:
|
||
- **Auto re-attach control** — keep it visible/controllable when not connected (today it can be
|
||
enabled-but-invisible).
|
||
- **Tooltips on disabled controls** — explain *why* Inject/subsystem controls are disabled.
|
||
- **Case-insensitive log filter.**
|
||
|
||
Code quality:
|
||
- **Stale comments** — `hook_guard.hpp` top block (still the old destroy-on-remove model),
|
||
`input_source.hpp` ("future" Steam source), `audio_ring.hpp` (v1/reserved), `shared_memory.hpp` /
|
||
`audio_ring.hpp` ("20-byte pads").
|
||
- **Consolidate `VtableHook`** — remove the duplicate copy in `audio_hook.cpp`.
|
||
|
||
### Future work
|
||
|
||
- **Per-game profiles** — persist each game's subsystem / capture-mode / audio choices and re-apply
|
||
them on attach.
|
||
- **Multi-guest virtual-pad mapping** — map multiple Remote Play guests onto distinct synthesized
|
||
pads in the XInput hook.
|
||
- **Continuous raw-mouse *movement* forwarding** — the MKB event stream is position-based today; also
|
||
forward relative motion for games that read raw deltas.
|
||
- **Native-D3D12 capture path** — DX12's higher capture cost is the inherent D3D11On12 bridge
|
||
(documented in **Lessons learned**); a native D3D12 copy would avoid it.
|
||
|
||
## Building
|
||
|
||
Requirements: Windows 10/11, Visual Studio 2022 (MSVC + C++ workload), CMake ≥ 3.21.
|
||
|
||
```sh
|
||
git clone --recurse-submodules <repo-url>
|
||
# or, if already cloned:
|
||
git submodule update --init --recursive
|
||
|
||
cmake -S . -B build -G "Visual Studio 17 2022" -A x64
|
||
cmake --build build --config Debug
|
||
# output: bin/Debug/coop_host.exe (+ coop_hook.dll, coop_vk_layer.dll + manifest, test exes)
|
||
```
|
||
|
||
The x64 build also drives a nested Win32 sub-build (CMake `ExternalProject`,
|
||
configured into `build/x86/`) that produces `coop_hook_x86.dll` and
|
||
`coop_inject_x86.exe` for 32-bit games, staged next to the x64 binaries. Disable
|
||
it with `-DCOOP_BUILD_X86_HELPER=OFF` if you don't need 32-bit support.
|
||
|
||
Third-party dependencies (Dear ImGui, SafetyHook, the official Khronos Vulkan-Headers, and the
|
||
`zeux/volk` Vulkan meta-loader) are git submodules under `third_party/`. No vcpkg / package
|
||
manager is used; CMake fails with a clear `git submodule update --init --recursive` hint if one
|
||
is missing (`coop_require_submodule`).
|
||
|
||
**Steam Input is optional.** It's enabled automatically when the Steamworks SDK is
|
||
vendored at `third_party/steamworks_sdk/` (extract the `steamworks_sdk_*.zip`
|
||
there). The SDK isn't redistributable, so it's gitignored and never committed; if
|
||
it's absent the host builds XInput-only (no other features depend on it). When
|
||
present, the build links `steam_api64.lib`, stages `steam_api64.dll` and the
|
||
action manifest next to the host, and also builds `coop_steam_input_probe`.
|
||
|
||
Steam Input is **off by default and XInput is the primary path**: merely
|
||
initializing Steam Input activates Steam's in-process XInput interception, which
|
||
hides controllers from XInput unless they're bound to our action set for the
|
||
running appid. Enable it (Controllers panel → **Use Steam Input**) only once a
|
||
controller is bound to Steam Input for the donor appid.
|
||
|
||
### clangd / IDE setup
|
||
|
||
The Visual Studio CMake generator does **not** emit `compile_commands.json`, so
|
||
clangd has no include paths and reports false errors. Run
|
||
[`gen-compile-commands.bat`](gen-compile-commands.bat) once (and after adding
|
||
sources or include dirs); it configures a parallel Ninja build in `build-clangd/`
|
||
that produces the database, which [`.clangd`](.clangd) points clangd at. clangd's
|
||
clang-cl driver resolves the MSVC / Windows SDK system includes on its own.
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
ctest --test-dir build -C Debug --output-on-failure
|
||
```
|
||
|
||
> **Fix bugs test-first.** Every bug fix must begin with a test that *reproduces* the bug
|
||
> and **fails** on the unfixed code — run it, watch it fail, and confirm it fails for the
|
||
> right reason. Only then write the fix, and confirm the same test now passes. A fix without
|
||
> a first-failing test is not done: the test is what proves the bug existed, that the change
|
||
> addresses it, and that it can't silently come back. (The hooked-audio double-play bug is the
|
||
> worked example: `audio_hook_test`'s guessed-path mute assertion was added and seen to fail
|
||
> before the one-line mute fix landed.)
|
||
|
||
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
|
||
no controller needed).
|
||
- **`dinput_hook_test`** — in-process self-test of the **DirectInput** forwarding path. Reuses
|
||
`mkb_hook.cpp`, installs the MKB hooks (which vtable-swap `IDirectInputDevice8::GetDeviceState`
|
||
via a kept-alive probe device), forwards a synthetic key + mouse button through the MKB ring,
|
||
pumps it, then creates a *real* DirectInput keyboard + mouse device and asserts `GetDeviceState`
|
||
returns the forwarded input (the key at its DIK scan-code, the left mouse button). Skips cleanly
|
||
if DirectInput can't acquire a device. (The Raw Input path is operator-validated against a real
|
||
game — an in-process WM_INPUT round-trip is too brittle to assert reliably.)
|
||
- **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC
|
||
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
|
||
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
|
||
soft-clip / encode for float32 + int16). No device needed.
|
||
- **`rate_estimator_test`** — unit test of the robust sample-rate estimator (the fix for
|
||
the wrong-rate bug). Feeds synthetic, adversarial render cadences and asserts it
|
||
converges to the right standard rate, rejects burst windows (never commits to a wrong
|
||
neighbour, incl. the real 46205 misread), flags a genuinely non-standard rate
|
||
low-confidence instead of spinning, and ignores idle windows. Pure logic, no device.
|
||
- **`audio_overrides_test`** — unit test of the per-game audio override store
|
||
(persist/reload, case-insensitive lookup by image name, and the differing-overwrite
|
||
detection that drives the warning). No device.
|
||
- **`tone_analysis_test`** — unit test of the audio fidelity analyzer used by
|
||
`coop_audio_validate` (pitch error in cents, SNR/THD, click + dropout detection) and the WAV
|
||
reader/writer. Synthesizes a clean tone, a wrong-rate (pitch-shifted) tone, a tone with
|
||
injected clicks, and one with silence gaps, and asserts each metric matches what was injected
|
||
(e.g. 44100 played as 48000 → +147 cents). Pure header logic, no device.
|
||
- **`audio_correlation_test`** — unit test of the two-path audio-format correlator
|
||
([`common/include/coop/audio_correlate.hpp`](common/include/coop/audio_correlate.hpp)), which
|
||
recovers a guessed stream's true sample rate from *ground truth* instead of cadence. Synthesizes
|
||
one continuous signal sampled at two rates (the hook's true rate + the device rate, with capture
|
||
skew + noise — exactly the hook-vs-loopback situation) and asserts `correlate_rate()` recovers the
|
||
true rate, scoring the right candidate ≈1.0 and the wrong ones ≈0 (incl. the hard 44100-vs-48000
|
||
case the cadence method can misread), and that unrelated signals are *not* confidently matched.
|
||
Also covers **layout recovery** (`correlate_format`): from a self-describing *chunked* capture
|
||
(each render buffer padded to the device block, as the hook's verify tap produces it) it strips
|
||
the padding per candidate de-interleaving and recovers the true channels + bit depth + rate
|
||
(stereo float, 16-bit PCM, 5.1, mono), and leaves a genuinely-ambiguous identical-channel layout
|
||
unconfident. Pure header logic, no device.
|
||
- **`audio_verify_test`** — integration test of the host's two-path verifier
|
||
([`host/src/audio/audio_format_verifier.cpp`](host/src/audio/audio_format_verifier.cpp)). Launches
|
||
`coop_mock_game` rendering a tone at a non-device rate (matching the device's *channel* count, so
|
||
this rate test isn't perturbed by a channel mismatch), injects the hook late (a guessed stream),
|
||
and runs the real `verify_stream_format()`: it co-captures the hook (pre-mix, via the ring's
|
||
`verify_capture` tap) and a parallel process-loopback (post-mix) of the same audio and correlates
|
||
them. Scenario (a) asserts it recovers the true rate; scenario (b) renders a *different* channel
|
||
count than the device with distinct per-channel content and asserts it recovers the full layout
|
||
(channels + bit depth + rate). Skips cleanly without an audio endpoint.
|
||
- **`render_pacer_test`** — unit test of the mirror's render-feed pacing policy
|
||
(`host/src/audio/render_pacer.hpp`). Simulates a producer/consumer device timeline and asserts
|
||
the shipping `RenderPacer` rides producer jitter that makes the old re-prime-on-partial-fill
|
||
policy withhold available data ~168× and drain the buffer to the brink of silence (the
|
||
under-run / "metallic" bug `coop_audio_validate` found). No device.
|
||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook's **format
|
||
detection**, the part that gets pitch right. Using a shared configurable
|
||
`ToneSource` (the same render helper `coop_tone` uses), it renders tones at a matrix
|
||
of common formats (44100/48000/96000 Hz, mono/stereo/5.1, 16-bit PCM / 32-bit float)
|
||
and asserts the hook reports the right rate/channels/bits + provenance for **both**
|
||
code paths: **see-init** (hooks installed first → exact `Initialize` format) and
|
||
**guess** (render client pre-exists → device-mix guess whose true rate is measured
|
||
from the cadence, the Brotato/Godot case). Also checks the frames reached the ring
|
||
non-silent **and that the hook muted the game's local playback** (the no-echo guarantee)
|
||
for *both* paths — the guessed-path mute assertion is the regression guard for the
|
||
double-audio bug. Skips cleanly with no audio endpoint.
|
||
- **`srgb_format_test`** — unit test of the `srgb_to_unorm` mapping the hooked
|
||
video path uses so `*_SRGB`-backbuffer games aren't darkened. No device.
|
||
- **`opengl_hook_test`** — in-process self-test of the OpenGL capture path:
|
||
installs the swap hooks, drives a real OpenGL context (clears the backbuffer to a
|
||
known color, calls `SwapBuffers`), and asserts the detour fired, the frame was
|
||
`glReadPixels`'d into the shared texture, and a second device reads the exact
|
||
pixels back by name. Skips cleanly without an OpenGL / D3D11 device.
|
||
- **`dx12_present_hook_test`** — in-process self-test of the Present hook's **D3D12**
|
||
path: drives a real D3D12 swapchain through the (shared) `IDXGISwapChain::Present`
|
||
vtable and asserts the D3D11On12 bridge wraps the `ID3D12Resource` backbuffer and
|
||
copies it into the shared texture, then reads the exact rendered color back by
|
||
name. Skips cleanly without a D3D12 device.
|
||
- **`present_hook_test`** — in-process self-test of the Present-hook video path:
|
||
installs the hook, drives a real D3D11 swapchain in the same process (clears the
|
||
backbuffer to a known color and calls `Present`), and asserts the detour fired,
|
||
the backbuffer reached the shared keyed-mutex texture, and a second device can
|
||
open it by name and read the exact pixels back. Skips cleanly if the machine has
|
||
no D3D11 device.
|
||
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone configurable WASAPI
|
||
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) at several source
|
||
formats (device default, 44100/48000/96000 Hz) and verifies the shipping
|
||
process-loopback capture (the fallback backend) receives non-silent audio by PID for
|
||
each — confirming loopback is format-agnostic (it captures post-mix at the device
|
||
endpoint format). Skips cleanly if the machine has no audio endpoint.
|
||
- **`mock_game_test`** — comprehensive capture/audio/hook stress test against
|
||
**`coop_mock_game`** (an animated, frame-numbered A/V test game under
|
||
[`tools/mock_game`](tools/mock_game) with selectable **DX9 / DX9Ex / DX10 / DX11 / DX12 /
|
||
OpenGL / Vulkan** backends and a configurable WASAPI tone). It launches the game, injects
|
||
`coop_hook.dll`, opens the hook's shared video texture, and **decodes the frame number out of
|
||
the captured pixels** to assert the mirror sees a *monotonic, advancing* sequence for each
|
||
backend (the bar for no dropped / stale / out-of-order frames — what the DX12 rotating-backbuffer
|
||
bug broke). **Vulkan** is special: since its present pointer is cached at init, the test launches
|
||
the mock **suspended**, injects, then resumes (the mock loads Vulkan and waits so the hook arms
|
||
first) to decode frames; it also exercises the **implicit capture layer** (registered via
|
||
`VK_LAYER_PATH` / `VK_INSTANCE_LAYERS`, decoding frames through the loader-inserted layer) and
|
||
late-injects a Vulkan game to assert the **too-late** flag trips (which drives the host's
|
||
relaunch banner). It launches the game at several **audio formats**
|
||
(44100/48000/96000, PCM + float) and asserts the hook measures each one's rate through the full
|
||
inject path, then injects with audio + video, checks both stream, cycles the audio subsystem
|
||
off/on (hook/unhook stress), and confirms the game never crashes and capture resumes. It then
|
||
runs an **aggressive hook/unhook storm** — a separate thread thrashes *every* subsystem on/off as
|
||
fast as the worker reconciles while the game is actively presenting, across **all** backends — to
|
||
catch an install/remove race that frees a hook's shared D3D / Vulkan state under a live capture
|
||
detour (the use-after-free that crashed a real game when its "Mirror video" button was spammed).
|
||
This suite drove out five real audio races plus the cross-backend unhook race (see Lessons
|
||
learned). Skips cleanly without a D3D11 / Vulkan device.
|
||
|
||
### Debugging the hooks against a real game
|
||
|
||
[`tools/audio_probe`](tools/audio_probe) (`coop_audio_probe.exe <pid> [seconds]`)
|
||
brings up the audio render-hook without Steam / RPT / the host UI: it creates the
|
||
IPC block + audio ring the hook expects, injects `coop_hook.dll` into the target
|
||
game, then drains the ring and prints per-stream format, captured-frame counts,
|
||
peak amplitude (proves the audio is real, not silence), and overruns. It enables
|
||
the hook's file trace (`%TEMP%\coop_hook.log`) for the run.
|
||
|
||
[`tools/audio_validate`](tools/audio_validate) (`coop_audio_validate.exe`) quantifies
|
||
audio-capture **fidelity** — it turns "the audio sounds slightly off" into numbers. With no
|
||
args it plays a known sine (`coop_tone` at 44100 Hz on a 48000 Hz endpoint, the Godot/Brotato
|
||
case), injects the hook exactly as the host does (late attach), captures the ring, and runs the
|
||
analyzer ([`common/include/coop/tone_analysis.hpp`](common/include/coop/tone_analysis.hpp)):
|
||
**pitch error in cents** (detects a mis-measured rate), **SNR/THD**, and **click / dropout**
|
||
counts. It also dumps a `.wav` so the capture can be *listened* to. Modes: `--render` drives the
|
||
real `AudioMirror` and measures its *rendered* output (surfaces the under-run / re-prime gaps the
|
||
capture side can't show — see Lessons learned); `--baseline` / `--selfcheck` establish the
|
||
measurement floor; `--listen <pid>` passively records a live process's output (point it at a
|
||
running `coop_host` to hear/quantify exactly what a guest gets on a real game); `--wav <file>`
|
||
analyzes any recording.
|
||
|
||
[`tools/input_probe`](tools/input_probe)
|
||
(`coop_input_probe.exe <pid> [seconds] [disable_mask]`) does the same for input: it
|
||
injects, reports one connected pad, and toggles a button each second so the game's
|
||
input layer sees a real state change. `disable_mask` (hex bits `0x1`=input
|
||
`0x2`=focus `0x4`=audio `0x8`=video) skips installing a subsystem, so you can
|
||
**bisect which injected subsystem affects a game** — this is how the 32-bit
|
||
Present-hook crash was isolated.
|
||
|
||
[`tools/vk_validate`](tools/vk_validate) (`coop_vk_validate.exe <layer|inject> [seconds] [exe]`)
|
||
validates the **Vulkan capture backend against a real game** (defaults to Sphere Spectacle). It
|
||
drives both early-presence methods — the implicit **layer** (registers `coop_vk_layer` scoped to the
|
||
game, launches via Steam, also late-injects `coop_hook.dll` for focus-spoofing so the game renders
|
||
unfocused) and **inject** (suspended-launch the exe *with the game's own folder as the working
|
||
directory* + early-inject before `vkCreateInstance`) — and asserts frames reach the shared texture
|
||
and advance, the captured resolution/colors are sane, saves a BMP screenshot for visual
|
||
confirmation, and **gates on the game keeping a healthy present rate while capturing** (so a
|
||
present-thread stall fails the tool, not just gets reported). Confirmed against Sphere Spectacle:
|
||
**both** paths mirror it correctly (1920×1080, right colors, no swizzle/darkening) at the game's full
|
||
present rate. The suspended-inject path needs the correct working directory (the game loads
|
||
`steam_api64.dll` / `resources/` relative to cwd) and only applies to titles that actually run when
|
||
launched directly; a title that refuses to run outside Steam produces no presents, and the tool
|
||
SKIPs inject for it (use the layer).
|
||
|
||
Both auto-detect a 32-bit (WOW64) target and inject via `coop_inject_x86.exe` +
|
||
`coop_hook_x86.dll`, exactly like the host. The probes build into
|
||
`bin/<config>/tools/` (the deployable `bin/<config>/` root holds only shipping
|
||
artifacts; tests build into `bin/<config>/tests/`) and resolve `coop_hook.dll` from
|
||
the root one level up, so run them from there. **Kill the game between runs** — the
|
||
loaded DLL locks `coop_hook.dll` against the next rebuild.
|
||
|
||
A debug-only **test harness** drives the host overlay's own code paths (inject / enable
|
||
audio / re-measure / override / screenshot / read state) without simulating mouse input,
|
||
for scripted UI validation. Build it with `-DCOOP_TEST_HARNESS=ON` (off by default, so
|
||
the shipped host never contains it); the host then reads one command line from
|
||
`%TEMP%\coop_test_cmd.txt` and replies in `%TEMP%\coop_test_resp.txt`. See
|
||
[`host/src/test_harness.hpp`](host/src/test_harness.hpp).
|
||
|
||
## Running the tool (manual, end-to-end)
|
||
|
||
This needs Steam, a donor game that supports Remote Play Together, and a second
|
||
person/account to receive the stream.
|
||
|
||
1. **Launch the host under a donor appid.** Find the donor's appid (the number in
|
||
its store URL); the donor only needs RPT support and is never actually played:
|
||
|
||
```text
|
||
"C:\Program Files (x86)\Steam\steam.exe" -applaunch <donorAppId> "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe"
|
||
```
|
||
|
||
The borderless window appears and Steam marks the donor "running". If the donor
|
||
ignores the trailing path, set the host as the donor's **Launch Options**
|
||
(`"D:\...\coop_host.exe" %command%`) or use a launcher like RemotePlayDetached.
|
||
|
||
2. **Start the real game** windowed or borderless (not exclusive fullscreen — see
|
||
Lessons learned). In the host's **Injection** panel, filter for the game's
|
||
`.exe`, select it, and click **Inject & Connect**. Watch **Hook status** for
|
||
**Attached**, a non-zero **XInput polled: N/s**, and **Focus spoof: active**.
|
||
|
||
3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** —
|
||
the host window now shows a live, letterboxed copy of the game. **Source**
|
||
picks how the frames are grabbed: **WGC** (default, Windows Graphics Capture —
|
||
works for any window) or **Hooked (Present)** (the injected hook's shared
|
||
texture — lower latency and no capture border, for DXGI / D3D11 and OpenGL
|
||
games; selecting it installs the video subsystem in the game).
|
||
|
||
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
|
||
With the hook injected, **Source** shows **Hooked (no echo)** and the game's
|
||
local playback goes silent while guests still hear it. If it shows **Loopback
|
||
(echo)** the hook's render path wasn't caught and you'll hear the game twice
|
||
locally (guests still hear it once). The **Render streams** table shows how
|
||
many WASAPI streams the game emits.
|
||
|
||
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
|
||
sees the mirrored video, hears the audio, and that their controller drives the
|
||
real game.
|
||
|
||
**Disconnect / reconnect.** **Disconnect** asks the injected DLL to remove every hook so the game
|
||
behaves exactly as if it was never touched, then drops the channel — but leaves the DLL injected
|
||
(dormant). Clicking **Inject & Connect** on a game that still has a live DLL (left dormant, or
|
||
surviving a tool restart/crash — a connected DLL keeps its shared section alive) **reconnects** to it
|
||
and resumes, without injecting again. Closing the host also unhooks the game on the way out. So a
|
||
clean cycle is: connect → play → disconnect (game back to normal, DLL parked) → reconnect later. The
|
||
DLL is never force-unloaded; it goes away when the game exits.
|
||
|
||
Useful checks while developing without RPT: tick **Forward synthetic test input**
|
||
in the Injection panel to make the game move on its own (proving forwarding is the
|
||
source), and click away from the game to confirm focus spoofing keeps it running.
|
||
|
||
> Injection access error → run the host as administrator. A 32-bit (WOW64) target
|
||
> is injected automatically via `coop_inject_x86.exe` + `coop_hook_x86.dll`; if
|
||
> those aren't next to the host, rebuild (the x86 sub-build stages them there).
|
||
|
||
## Lessons learned
|
||
|
||
Non-obvious things that cost time and constrain the design:
|
||
|
||
- **RPT only streams the *focused* window.** The game can't hold focus itself, so
|
||
the hook spoofs it (`GetForegroundWindow` / `GetActiveWindow` / `GetFocus` +
|
||
swallowing deactivation messages) to keep the game polling and rendering while
|
||
the host owns real OS focus.
|
||
- **Run target games windowed or borderless, never exclusive fullscreen** —
|
||
exclusive fullscreen minimizes on focus loss (defeating the spoof) and can't be
|
||
window-captured. While unfocused the game gets no OS keyboard/mouse, only the
|
||
forwarded pad.
|
||
- **WGC captures occluded windows but not minimized ones.**
|
||
- **Process-loopback capture doesn't mute the source.** Capturing a process's
|
||
render doesn't stop it reaching the speakers, so the no-echo path instead injects
|
||
a WASAPI render-hook that copies each buffer then releases it with
|
||
`AUDCLNT_BUFFERFLAGS_SILENT`; loopback stays as the (echoing) fallback.
|
||
- **`ActivateAudioInterfaceAsync` needs an *agile* completion handler.** If the
|
||
handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected
|
||
**synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of
|
||
apartment, device path, or activation params. (WRL/wil samples make the handler
|
||
agile for you.) Process loopback also needs the Win10 20H1 headers
|
||
(`NTDDI_VERSION ≥ 0x0A00000B`).
|
||
- **COM methods have no exports, so hooks walk vtables by frozen-ABI index — count
|
||
exactly.** All instances of a coclass share one vtable, so hooking one object's
|
||
slot catches every instance; but `IAudioClient::GetService` is **14**, not 13
|
||
(`SetEventHandle` sits at 13 between `Reset` and `GetService`). Count every
|
||
inherited `IUnknown`/base method when adding a hook.
|
||
- **D3D12 capture copies the *rotating* back buffer, not `GetBuffer(0)`.** D3D11
|
||
flip-model keeps `GetBuffer(0)` pointing at the live back buffer, but D3D12 rotates
|
||
buffers explicitly — the game renders into the buffer at
|
||
`IDXGISwapChain3::GetCurrentBackBufferIndex()`, which advances each `Present`.
|
||
Grabbing buffer 0 copies a stale buffer on N-1 of every N frames, so the mirror
|
||
silently runs at refresh/N — yet the Present counter, published FPS, generation, and
|
||
latency all read full rate (they count Presents, not unique content), so the metrics
|
||
look perfect while the eye sees missing frames. Query the current index right before
|
||
the trampoline `Present` (that's the just-rendered buffer) and copy that one.
|
||
- **Keep the D3D12 capture copy off the game's present queue, but ordered after its
|
||
frame.** The D3D11 path copies on the game's immediate context, so it's naturally
|
||
ordered after the frame and on the game's own timeline. For D3D12 the D3D11On12
|
||
bridge needs a queue: running the copy on the *game's* present queue orders it
|
||
correctly but stalls the game's own presents (GPU back-pressure, plus the shared
|
||
keyed-mutex `AcquireSync` is a **CPU-blocking** call on the render thread). Running
|
||
it on an independent queue avoids the stall but races the game's render → stale
|
||
frames. The fix is both: run the copy on **our own** queue, and order it with a
|
||
**fence** the game's present queue signals after its frame (a near-free op) and our
|
||
queue waits on. The present queue is recovered for late injection by hooking
|
||
`ID3D12CommandQueue::ExecuteCommandLists` (the per-frame method, not creation). Make
|
||
the producer-side `AcquireSync` non-blocking (`timeout 0`) so a busy mutex drops a
|
||
*mirror* frame instead of stalling the game; the Video panel's "Frames lost" line
|
||
surfaces both capture- and display-stage drops.
|
||
- **A D3D10 game's backbuffer lies about being D3D11.** A pure-D3D10 swapchain's backbuffer
|
||
QIs to `ID3D11Texture2D` *successfully*, but that D3D11 view reads back **empty** — the
|
||
rendered content only exists on the game's own D3D10 device. (And a D3D11 backbuffer QIs to
|
||
`ID3D10Texture2D` too, so `GetBuffer` alone can't tell them apart.) The reliable signal is
|
||
that a feature-level-10 device **rejects** `CreateTexture2D` with the NT-handle keyed-mutex
|
||
share flags (`E_INVALIDARG`): so try the D3D11 fast path, and on that failure switch (sticky)
|
||
to reading the backbuffer through the game's **D3D10** device into a staging texture and
|
||
`UpdateSubresource`-ing it into the shared texture on a **hook-owned D3D11 device** (the game
|
||
has no usable D3D11 device of its own). The staging `Map` blocks until the GPU copy completes,
|
||
so there's no cross-device race.
|
||
- **D3D9 capture: one read-back path covers plain D3D9 *and* D3D9Ex.** D3D9 has no DXGI
|
||
swapchain, so inline-hook `IDirect3DDevice9::Present` (vtable index **17**, found from a
|
||
throwaway device — clean prologue, so inline is safe; `stdcall()` on x86). A D3D9 surface
|
||
isn't D3D11-shareable, so read the backbuffer back with `GetRenderTargetData` into a
|
||
`D3DPOOL_SYSTEMMEM` surface (`LockRect` blocks until the copy → no race), **swizzle BGRA→RGBA**
|
||
(`X8R8G8B8`/`A8R8G8B8` store as little-endian `0xAARRGGBB` → bytes B,G,R,A) and upload into the
|
||
shared texture on a hook-owned D3D11 device. Both plain D3D9 and D3D9Ex call this same Present,
|
||
so **one path serves both** — the GPU shared-surface fast path the plan sketched for D3D9Ex
|
||
wasn't worth it (D3D9 games are light enough that the read-back cost is fine, and it dodges the
|
||
cross-device keyed-mutex-less sync of a legacy shared surface).
|
||
- **The OpenGL mock needs no GL loader.** Drawing the animated pattern with scissored clears
|
||
(`glClear` + `glScissor`) only touches **GL 1.1**, which `opengl32` exports directly — so the
|
||
planned **glad** submodule wasn't needed (a legacy `wglCreateContext` + `<GL/gl.h>` suffices).
|
||
GL's framebuffer is **bottom-left** origin and the capture flips it top-down, so the mock draws
|
||
the frame-counter block at the GL *top* (`y = h - block`) to land at the captured image's
|
||
top-left. The GL `SwapBuffers` hook now also bumps the shared present counter (it's the GL
|
||
present), so the Video panel's present rate works for GL games too.
|
||
- **Vulkan can't be late-hooked, and capturing its present needs semaphore surgery.** Vulkan
|
||
games cache `vkQueuePresentKHR` at init (often via volk, which bypasses the loader trampoline),
|
||
so late injection misses it — the hook must be in **before `vkCreateInstance`**. Catch the
|
||
resolution chain instead: inline-hook the `vulkan-1.dll` `vkGetInstanceProcAddr` export and hand
|
||
back wrappers for `vkCreateInstance` / `vkCreateDevice` / `vkCreateSwapchainKHR` /
|
||
`vkQueuePresentKHR` (so a volk app resolves *ours*), tracking the device / queue / images /
|
||
format; read the presented image back with `vkCmdCopyImageToBuffer` and upload it to the shared
|
||
texture on a hook-owned D3D11 device. The trap: the read-back submit must **wait on the present's
|
||
wait semaphores** (consume them) and **signal a fresh semaphore the real present then waits on**
|
||
— both your copy and the present can't wait the same binary semaphore. "Injected too late" is
|
||
detected as *vulkan-1.dll loaded + hook in for >4 s + we never saw `vkCreateDevice`* (the app set
|
||
everything up before us) → red relaunch banner, WGC meanwhile. Testing needs an **early-load**
|
||
path (suspended launch + inject + resume; the mock loads Vulkan and waits), since the normal
|
||
late-inject flow can't catch a Vulkan present.
|
||
- **The reliable Vulkan path is a real implicit layer — and the loader tags its link structs with
|
||
small `sType`s.** Games that init Vulkan instantly resolve their present pointer before any
|
||
inject can land, so the only robust early presence is an implicit layer (`coop_vk_layer`) the
|
||
loader inserts at `vkCreateInstance`. It must do **chain dispatch** (pull
|
||
`pfnNextGetInstanceProcAddr` / `pfnNextGetDeviceProcAddr` out of the `VkLayer*CreateInfo` link in
|
||
`pNext`, advance the link, call down) rather than inline-hook. The trap that cost time: those
|
||
link structs use **loader-internal `sType` values 47 (instance) / 48 (device)** — *not* the
|
||
`1000000000`-range I assumed (and `vk_layer.h` isn't in Vulkan-Headers, so the structs are
|
||
hand-declared) — matching the wrong value silently failed the device-chain walk
|
||
(`vkCreateDevice: Failed to create device chain`). An implicit layer loads into *every* Vulkan
|
||
app, so it self-scopes: capture only when the process image matches the host-written target file,
|
||
else pure pass-through. Register it per-user (HKCU `…\Vulkan\ImplicitLayers`, no admin) and
|
||
unregister on untick / host exit.
|
||
- **Both Vulkan early-presence paths work on a real game — the "only the layer works" belief was two
|
||
bugs in disguise.** Against Sphere Spectacle (`coop_vk_validate`), the inline **inject** path
|
||
(suspended-launch + early-inject before `vkCreateInstance`) initially captured nothing, which was
|
||
wrongly written off as "the title requires launching through Steam." It does not. Two real causes:
|
||
(1) the harness launched the exe with **our** working directory, so the game couldn't load
|
||
`steam_api64.dll` / `resources/` and never rendered — launch it with the game's own folder as cwd
|
||
and it runs fine directly; and (2) the game (volk-style) resolves `vkQueuePresentKHR` /
|
||
`vkCreateSwapchainKHR` via **`vkGetInstanceProcAddr`**, but `vk_hook` only substituted those when
|
||
resolved via `vkGetDeviceProcAddr` — so the present bypassed us. Intercepting them in the
|
||
instance-level resolver too fixed it. The division of labor is about *who launches the game*, not
|
||
capability: for a game launched **through Steam** we can't suspend the launch, so the implicit
|
||
**layer** (always in the loader chain) is the path; when **we** launch the exe, suspended-inject
|
||
works (and is how you'd debug). When the layer runs, the game still needs `coop_hook.dll`
|
||
co-injected for **focus-spoofing** so an unfocused window doesn't throttle itself.
|
||
- **A Vulkan present-rate *is* a valid no-FPS-impact gate — once the read-back is off the present
|
||
thread.** The earlier validator reported the present rate and rationalized it ("the game's own
|
||
cadence; a hard FPS gate is meaningless"), which hid a real **144→3 FPS** stall: the read-back ran
|
||
on the present thread and spent ~370 ms/frame doing a CPU read of write-combined staging memory.
|
||
After moving the read-back to a reaper thread (`coop::hook::VkCapture`), `present_calls` (counted
|
||
every present) reflects the game's true rate, so the tool now **asserts** it stays healthy while
|
||
capturing. Lesson: a perf check must assert a bound — if you find yourself explaining why a number
|
||
is fine, make the test prove it.
|
||
- **The capture must not touch the game's sync mode — and must not throttle itself.** The swapchain's
|
||
**present mode** *is* the sync mode (`FIFO` = vsync, `IMMEDIATE`/`MAILBOX` = off); `VkCapture` and
|
||
the layer pass `VkSwapchainCreateInfoKHR` straight through, so whatever the game asked for is what
|
||
it gets — proven by logging `ci->presentMode` (Sphere Spectacle = `FIFO`, and it holds a steady
|
||
144 Hz with the layer attached). An earlier ~150 Hz *capture* throttle was wrong and is removed:
|
||
capture follows the present rate, which **vsync** already paces (a 144 Hz FIFO game presents 144×/s,
|
||
so we mirror 144×/s). The only limiter left is ring backpressure — if the reaper can't keep up we
|
||
skip a frame rather than stall the game — which is correctness, not a frame cap. (A direct-launch
|
||
window that isn't composited/foreground can present uncapped because DWM doesn't throttle a
|
||
non-foreground windowed FIFO app — that's the OS, not us, and not the case under Steam.)
|
||
- **A render client that predates our injection has no knowable format — measure it.**
|
||
We inject into already-running games, so we usually never see the game's
|
||
`IAudioClient::Initialize`; the render-hook then assumes the device mix format for that
|
||
stream. That's wrong for games that render at a non-device rate via WASAPI
|
||
`AUTOCONVERTPCM` (e.g. Godot / Brotato render 44100 Hz while the endpoint mixes at
|
||
48000), so the captured audio plays back **pitch-shifted up**. Fix: treat such a format
|
||
as a *guess* and measure the stream's true sample rate from its render cadence
|
||
(frames/sec over a short active window, snapped to the nearest standard rate) before
|
||
publishing it, deferring capture until verified. Discard the first measurement window:
|
||
the moment we attach, the stream's already-queued buffers arrive in a burst that
|
||
over-counts (the `audio_hook_test` matrix caught this), so measure the next,
|
||
steady-state window. **Only the rate is recoverable, though** — channels/bit-depth can't
|
||
be measured (`AUTOCONVERTPCM` hands `GetBuffer` a *fixed* staging buffer, so there's no
|
||
buffer stride; confirmed empirically) and WASAPI has no API for a pre-existing client's
|
||
format, so they stay the device-mix guess. That's right for the common case (engines
|
||
render stereo float = the endpoint), and a `VirtualQuery` clamp on the capture copy
|
||
keeps a too-large guessed block from ever over-reading the source buffer. Streams we *do*
|
||
watch get created carry their exact `Initialize` format, and the loopback fallback
|
||
captures post-mix at the device format (always correct). The Audio panel shows each
|
||
stream's provenance (known / measuring / measured rate (ch/bits assumed)) so what the
|
||
mirror is using is always visible — see Limitations.
|
||
- **A short measurement window *rejects* a bad reading, it can't *average* it away.**
|
||
Measuring the rate over ~200 ms made one extra ~10 ms WASAPI buffer at a window edge a
|
||
~5% error, which lands *between* standard rates (they're >8% apart) — so 44100 read as
|
||
~46205 and got published verbatim. The robust fix isn't just a longer window: it's to
|
||
**refuse any window that doesn't snap to a standard rate** and require **consensus**
|
||
across windows, since a quantization/burst error big enough to miss the right rate
|
||
lands in no-man's-land rather than on a wrong neighbour. Only commit a non-standard
|
||
estimate as explicitly *low-confidence* (shown red). The operator can also re-measure
|
||
or override the format via a per-stream `AudioRingHeader` op channel; the host rebuilds
|
||
its render client when `format_generation` bumps, so it takes effect live.
|
||
- **Two capture paths beat one guess: correlate the hook against the loopback.** Cadence
|
||
measurement *rejects* a bad rate reading but is still a guess from one signal, and it can't recover
|
||
channels/bit-depth at all. But during the measurement window the game is still audible, so we have
|
||
the *same* audio twice: the **hook** (pre-mix, unknown format) and a **process-loopback** (post-mix,
|
||
the *known* device format — it's the hook signal resampled by WASAPI's AUTOCONVERTPCM). Resampling
|
||
the hook by each candidate rate and cross-correlating against the loopback pins the true rate from
|
||
ground truth: the right rate holds alignment across the whole window (score ≈1.0); a wrong rate
|
||
time-warps the hook so a single alignment can't hold and the correlation collapses (≈0). The catch
|
||
that makes this need a measurement *tap*: the no-echo path **silences** the game, so a loopback of a
|
||
silenced game is silent — the co-capture must happen while the stream is still being measured (not
|
||
yet published, so not yet silenced). A host-set `verify_capture` ring flag makes the hook push the
|
||
guessed stream's raw pre-mix bytes (no silence) during that window; the host
|
||
(`audio_format_verifier`) co-captures both, correlates (`coop/audio_correlate.hpp`), and feeds a
|
||
correction into the existing override channel. **Rate vs layout are coupled, though**: correlating
|
||
the *waveform* needs the hook bytes de-interleaved at the right channel count, so the rate step
|
||
assumes the hook layout matches the device (true for the common stereo-on-stereo case); recovering a
|
||
*different* channel count / bit depth is the layout step, which tries candidate de-interleavings and
|
||
keeps whichever correlates. **The layout step needs a self-describing tap**: the hook can't know a
|
||
guessed stream's real frame size, so it pads each render buffer to the *device* block — which
|
||
over-reads stale staging bytes for a stream with fewer channels/bits. The raw padded bytes are
|
||
un-decodable (the stale tail scrambles the audio), so the tap prefixes each buffer with its frame
|
||
*count* (`[count][count*device_block bytes]`); the host strips the padding per candidate layout
|
||
(take the real `count*real_block` of each chunk) before de-interleaving. **The genuinely-ambiguous
|
||
case stays unresolved**: a stream whose channels carry identical content is indistinguishable from
|
||
one channel at double the rate (`2ch@R` == `1ch@2R` byte-for-byte), so the correlator reports low
|
||
confidence and the format stays the device assumption rather than guessing wrong.
|
||
- **Re-priming the render feed on a *partial* fill manufactures the gap it's avoiding.** The
|
||
mirror re-renders the captured ring to the output device. The original feed loop re-primed
|
||
(withheld the feed until ~30 ms had rebuffered) whenever it couldn't completely fill the free
|
||
buffer that tick (`to_write < avail`). But a partial fill is *normal* producer jitter — and
|
||
withholding the feed drains the device, so a one-frame ring dip became a full ~30 ms drop-out.
|
||
On a jittery game it fired constantly → choppy, "metallic" mirror audio, intermittent by
|
||
buffer phase. Fix: feed whatever is available every tick; re-prime **only** on a genuine
|
||
starvation (device buffer empty *and* ring empty). The policy is factored into a pure
|
||
`RenderPacer` reused by the loop and unit-tested headlessly (`render_pacer_test`: on a jittery
|
||
schedule the old policy withholds available data ~168× and drains the cushion to one period
|
||
from silence; the new one never withholds). **Diagnosing it needed device-side capture, not a
|
||
write-side tap** — the artifact is silence the device *plays* during an under-run, not bytes
|
||
the loop writes, so a tap on the write would look clean. `coop_audio_validate` quantifies it
|
||
by loopback-capturing the *rendered* output (pitch error in cents, SNR, click/dropout counts)
|
||
and contrasts the capture ring (pristine), the measurement floor, and the render path.
|
||
- **Mute the game with the SILENT flag, not a `memset` — they are not the same thing.** The
|
||
hooked path's whole point is "no echo": capture the game's frames into the ring AND silence
|
||
its local playback, so the only audio is the host's re-render. The mute was implemented by
|
||
*zeroing the buffer* and releasing it with `AUDCLNT_BUFFERFLAGS_SILENT`. But zeroing
|
||
`num_frames * block` is only safe when `block` is the real frame size — for a **guessed**
|
||
format (late attach, the Brotato case) the guessed block can exceed the real buffer, so the
|
||
zero would over-write adjacent memory. The original code's conservative answer was to skip the
|
||
whole mute for guessed streams — which left the game **audible**: it played locally *and* the
|
||
mirror re-rendered the same audio a few ms later = a metallic, slightly-out-of-sync double
|
||
(exactly the reported symptom). The fix: `AUDCLNT_BUFFERFLAGS_SILENT` already makes WASAPI
|
||
*ignore the buffer contents* and play silence — it mutes with **no write at all**, so it's safe
|
||
for any format. Decouple the two: always mute via the flag; keep the `memset` only for an exact
|
||
format (belt-and-suspenders). `audio_hook_test` now asserts the mute engages for both the
|
||
exact and the guessed path (the guessed assertion fails on the old code — write the failing
|
||
test first), and the byte-incompatible cases prove the flag-mute never over-writes.
|
||
- **Panels must *fit* their assigned size at max info -- measure it, don't eyeball it.** The
|
||
overlay opens panels at fixed sizes that scale with the monitor, so with Debug details on a
|
||
dense panel can overflow and scroll content out of view. `ui_fit_test` drives the real
|
||
Controllers + Audio panels headlessly (null-backend ImGui: build the atlas with
|
||
`GetTexDataAsRGBA32`, set a non-null `TexID`, `Render()` needs no GPU) at reference
|
||
resolutions and asserts each window's `ScrollMax` is 0 -- read it on the *2nd+* frame, since
|
||
ImGui computes `ScrollMax` in `Begin()` from the *previous* frame's content size. Two fixes
|
||
fell out: the center column was too narrow (horizontal overflow -> widen it, merge the
|
||
Controllers poll/round-trip tables), and a *static* height split can't serve both modes, so
|
||
it's now **Debug-details-aware** (debug on -> Controllers/Audio get the height for their
|
||
tables; off -> Video's perf graphs are the tall content, since Video's height is
|
||
mirroring-driven, not debug-driven). The host's `uisize`/`uifit` harness commands check the
|
||
same thing against the live overlay.
|
||
- **Drive the ImGui overlay for tests through an IPC harness, not synthetic input.**
|
||
`PostMessage`-d mouse clicks don't reliably reach ImGui widgets, and key/coordinate
|
||
simulation is brittle. A tiny debug-only command channel (`-DCOOP_TEST_HARNESS`, file-
|
||
based) that calls the *same* code the buttons do — and replies with state — makes UI
|
||
validation deterministic and scriptable, and is compiled out of the shipped product.
|
||
- **A capture-style stress test against a frame-numbered mock game is worth a lot.** An
|
||
animated game that encodes its frame number in the pixels lets a test assert the mirror
|
||
shows a *monotonic, advancing* sequence (the bar for "no dropped / stale / out-of-order
|
||
frames"), and toggling subsystems while it renders flushes out concurrency bugs. This
|
||
one caught **five** real audio races: a `memset` over-*write* past a guessed stream's
|
||
real buffer (zeroing 8ch into a 2ch buffer corrupts adjacent audio memory → crash; the
|
||
fix: capture but don't silence a guessed stream), a null call through a vtable hook's
|
||
`original` after unhook (keep it valid), a non-atomic `g_ipc` TOCTOU, a stale
|
||
GetBuffer/ReleaseBuffer pairing across a hook toggle (epoch-stamp it), and COM-object
|
||
churn from re-creating the probe each toggle (build it once, keep it, only swap vtable
|
||
slots). **Silently silencing/zeroing a buffer whose true size you only guessed is an
|
||
over-write, not just an over-read** — clamp the read, but don't write what you can't size.
|
||
- **Spamming a subsystem on/off must always be safe — install *and* remove.** Toggling a subsystem
|
||
install/removes its hooks while the game keeps calling the hooked API; the `mock_game_test` storm
|
||
(uncapped backends presenting at thousands/s, the mock now polling XInput / GetAsyncKeyState /
|
||
GetKeyboardState / GetForegroundWindow so the input/focus/MKB hooks are exercised too) drove out a
|
||
whole family of races. The fixes:
|
||
- **Removal — drain in-flight detours, never free the trampoline.** Each detour wraps its body in an
|
||
RAII `DetourGate::Guard` (active-count). `remove_*` `disable()`s the SafetyHook inline hook
|
||
(restores the original bytes under thread suspension; **not** `= {}`, which frees the trampoline a
|
||
detour may be about to call), then `drain()`s the count to zero, then frees the shared state. Two
|
||
drain subtleties: it `Sleep(1)`s **before** each zero-check, because a thread can be *inside* the
|
||
detour but not yet past its `Guard` constructor (the prologue is unguarded); and even after the
|
||
drain we **keep the hooks alive (disabled), never destroying them during the session** — they're
|
||
*persistent*, re-enabled on re-install (`hook/src/hook_install.hpp`) — so a stale detour can never
|
||
jump through a freed trampoline (it runs the original instead). The trampoline is allocated once
|
||
and reused, so no churn and no leak.
|
||
- **Install — populate the global before the detour can fire.** `create_inline()` enables the hook
|
||
(patches the bytes) *before* the result is move-assigned into the global the detour reads to reach
|
||
the trampoline; a call landing in the detour during that assign reads a torn hook → AV.
|
||
`install_inline()` instead creates `StartDisabled`, lets the global be assigned, and only then
|
||
`enable()`s it (and reuses the existing hook on re-install — the persistent model).
|
||
- **Focus is special twice.** The WNDPROC subclass publishes `g_orig_proc` *before*
|
||
`SetWindowLongPtr` activates it (and `subclass_proc` falls back to `DefWindowProc` if it's still
|
||
null), and `GetForegroundWindow`/`GetActiveWindow` share user32 code — so the focus query hooks
|
||
are **disabled in reverse install order** (GetForegroundWindow stays hooked until GetActiveWindow
|
||
is unhooked), keeping the invariant "GetActiveWindow hooked ⇒ GetForegroundWindow hooked" so a
|
||
call never lands in a half-patched shared region.
|
||
- **Vulkan is the exception to disable-stops-new-detours.** The game caches our
|
||
`hk_vkQueuePresentKHR` pointer at resolution time and keeps calling it after the GPA hook is
|
||
disabled, so removal closes an atomic **capture gate** first (the detour then passes straight
|
||
through to the saved real present without touching the read-back state), drains, then frees.
|
||
The drain is bounded (~400 ms) so a wedged game thread can't hang the worker. Guarded by
|
||
`mock_game_test`'s storm (every backend) and the deterministic `detour_gate_test`. (This started as
|
||
the audio hooks' restore-then-drain idea; the video / XInput / focus / MKB hooks needed all of the
|
||
above to be spam-safe — the original "Mirror video" spam crashed Brotato's OpenGL path.)
|
||
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
|
||
the backbuffer inside the game's `Present`, which the game issues at its true render
|
||
rate regardless of how DWM composites that *window*. So an unfocused game window can
|
||
judder (DWM under-composites background windows; only the focused window gets VRR /
|
||
independent flip) while the mirror — which receives every `Present` — stays smooth.
|
||
This is why the game window not being focused doesn't matter: it isn't the surface
|
||
anyone sees. The same focus rule explains why an unfocused *tool* window can render
|
||
below the game's rate (it loses VRR), so in use the mirror is the focused window.
|
||
- **SafetyHook on x86 has two traps that froze 32-bit Slaps and Beans.** (1)
|
||
`InlineHook::call()` invokes the trampoline as `__cdecl`, but most targets are
|
||
`__stdcall` (COM methods like `IDXGISwapChain::Present`, the WASAPI interfaces,
|
||
`WINAPI` `SwapBuffers`); on 32-bit that double-cleans the stack → ESP imbalance →
|
||
crash (Debug: **Run-Time Check Failure #0**). Use **`stdcall()`** (a no-op on
|
||
x64). (2) Don't *inline-hook* COM methods on x86 at all: MMDevApi/AudioSes
|
||
prologues do `push ebp; mov ebp,esp; and esp,-8` and read args **EBP-relative**,
|
||
which SafetyHook's trampoline relocation breaks (the original then runs with
|
||
garbage args and faults). Hook COM methods by **swapping the vtable entry**
|
||
instead (`VirtualProtect` the slot, overwrite the pointer, call the saved
|
||
original) — no code patching, pristine stack regardless of prologue. Inline
|
||
hooking stays fine for `Present`/`SwapBuffers` (clean prologues). Guarded by the
|
||
x86 hook tests.
|
||
- **SafetyHook's `enable()`/`disable()` aren't safe to call in a *tight loop* concurrent with calls to
|
||
the hooked function.** They re-patch the prologue in place under a VEH "trap" that reliably relocates
|
||
a thread *parked on the prologue*, but a thread executing the function **body** faults when the whole
|
||
page is briefly made non-executable during the patch and relies on a retry — and under rapid toggling
|
||
that retry races a half-rewritten prologue → AV in the caller. A minimal reproducer (one thread
|
||
tight-looping enable/disable, two hammering the target; hook created once, so no install race / UAF)
|
||
faults ~1/3 of runs in Debug while the no-toggle control is clean. This is a SafetyHook limitation,
|
||
not our code: we reconcile install/remove from a **single, tick-bounded worker thread**, never a
|
||
tight loop, so the `mock_game_test` storm (a single enable/disable concurrent with thousands of
|
||
calls/s — SafetyHook's designed-safe case) is reliably crash-free. The finding is preserved as a
|
||
committed, non-CI reproducer (`tools/sh_concurrency_repro/`) and written up for upstream in
|
||
[docs/safetyhook-concurrency.md](docs/safetyhook-concurrency.md); an earlier flaky tight-loop CTest
|
||
that hit this was replaced by the deterministic `hook_install_test` + `detour_gate_test` (our
|
||
contract) plus the storm (realistic concurrency).
|
||
- **Steam Input init suppresses XInput.** Initializing Steam Input turns on Steam's
|
||
in-process XInput interception, which hides controllers from `XInputGetState`
|
||
unless they're bound to the running appid's action set — defaulting to it
|
||
silently broke forwarding. XInput is primary; Steam Input is opt-in.
|
||
- **Forwarding keyboard/mouse means covering three read paths, each differently.** Games read
|
||
keyboard/mouse three ways and a forwarder has to satisfy all of them from one synthesized state.
|
||
(1) **Message-loop** games get `PostMessage`d `WM_KEYDOWN`/`WM_*BUTTON*`. (2) **Polling** games
|
||
get synthesized `GetAsyncKeyState`/`GetKeyboardState`/`GetCursorPos`. (3) **DirectInput** games
|
||
call `IDirectInputDevice8::GetDeviceState` — a COM method, so it's hooked by **vtable swap, not
|
||
inline** (the x86 COM-prologue trap), reading the vtable from a kept-alive *probe* device we
|
||
create (the game made its devices before we injected, but all DI devices share one vtable);
|
||
dispatch on `cbData` (256 = keyboard `BYTE[256]` indexed by **DIK scan-code**, not VK — map with
|
||
`MapVirtualKey(VK_TO_VSC)`; `sizeof(DIMOUSESTATE)` = mouse). (4) **Raw Input** games read
|
||
`WM_INPUT` → `GetRawInputData`, but get **no `WM_INPUT` while unfocused** — so we *synthesize* it:
|
||
post `WM_INPUT` with `lParam` = the address of one of our `RAWINPUT` slots, and the hooked
|
||
`GetRawInputData` serves that slot's data back (`RID_HEADER` and `RID_INPUT`). Relative raw mouse
|
||
*movement* isn't in the position-based MKB event stream, so raw-input forwarding covers keys +
|
||
buttons, not free-look — a known limitation.
|