Harden every hook against the install/remove use-after-free

Spamming a subsystem toggle (the "Mirror video" button) could crash the game:
remove_*_hooks freed a hook's shared D3D / Vulkan / IPC state immediately, while a
capture detour was still mid-flight on the game's render thread -> use-after-free.
Only the audio hooks had the safe-unhook drain; the video (Present/D3D9/D3D10/GL/
Vulkan) and XInput/focus/MKB hooks did not.

Test-first: mock_game_test now runs an aggressive hook/unhook storm -- a separate
thread thrashes every subsystem on/off while the game presents, across all backends.
It crashed gl + vk (0xC0000005) and failed dx9 capture-resume before the fix.

Fix (hook/src/hook_guard.hpp, DetourGate): each detour wraps its body in an RAII
active-count Guard; remove_* restores the hook first (so no new detour starts),
drains the in-flight detours to zero, and only then frees the shared state. Vulkan
is special-cased -- the game caches hk_vkQueuePresentKHR, so removal closes an
atomic capture gate (detours then pass through to the real present), drains, then
frees the read-back resources. Storm now passes on every backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-23 01:56:36 +02:00
parent 79399e88f1
commit 21c15b162b
10 changed files with 365 additions and 22 deletions

View File

@@ -109,20 +109,6 @@ default** and covers anything the hooked path doesn't.
### Current Tasks
- **Injection hardening — provoke, then fix, the hook install/remove races.** Rapidly
toggling the **"Mirror video"** button has crashed a real game (Brotato): flipping the
Present-hook video subsystem on/off in quick succession races hook install/remove against
the game's Present thread and the host's capture path. Per the test-first rule, *first make
the crash reproducible in a test* — make `mock_game_test`'s hook/unhook stress against
`coop_mock_game` far more aggressive: tight, repeated subsystem toggles (especially video)
while the game is actively presenting, driven from a separate thread so the toggles interleave
with the Present detour, sustained long enough that an unguarded race is near-certain to
fault, and across all backends (DX11/12/9/10/GL/Vulkan). Then fix what it surfaces — the
install/remove path is the same safe-unhook problem the **audio** hooks already solved (bump
an epoch on each toggle, restore the vtable/inline first so no new detour starts, then drain
in-flight detours before tearing down shared state); apply the equivalent guard to the
Present / D3D9 / D3D10 / OpenGL / Vulkan hooks.
- **Determine a pre-existing stream's audio format by *correlating* the two capture
paths, instead of guessing.** When we attach to an already-running game we never saw its
`IAudioClient::Initialize`, so the render-hook assumes the device mix format and measures
@@ -297,9 +283,13 @@ ctest --test-dir build -C Debug --output-on-failure
relaunch banner). It launches the game at several **audio formats**
(44100/48000/96000, PCM + float) and asserts the hook measures each one's rate through the full
inject path, then injects with audio + video, checks both stream, cycles the audio subsystem
off/on (hook/unhook stress), and confirms the game never crashes and capture resumes. This suite
drove out five real audio races (see Lessons learned). Skips cleanly without a D3D11 / Vulkan
device.
off/on (hook/unhook stress), and confirms the game never crashes and capture resumes. It then
runs an **aggressive hook/unhook storm** — a separate thread thrashes *every* subsystem on/off as
fast as the worker reconciles while the game is actively presenting, across **all** backends — to
catch an install/remove race that frees a hook's shared D3D / Vulkan state under a live capture
detour (the use-after-free that crashed a real game when its "Mirror video" button was spammed).
This suite drove out five real audio races plus the cross-backend unhook race (see Lessons
learned). Skips cleanly without a D3D11 / Vulkan device.
### Debugging the hooks against a real game
@@ -588,6 +578,24 @@ Non-obvious things that cost time and constrain the design:
churn from re-creating the probe each toggle (build it once, keep it, only swap vtable
slots). **Silently silencing/zeroing a buffer whose true size you only guessed is an
over-write, not just an over-read** — clamp the read, but don't write what you can't size.
- **Every removable hook needs the same safe-unhook drain, not just audio.** The audio hooks
learned to restore the vtable slot first and *drain in-flight detours* before tearing down the
shared state they read; the video (Present / D3D9 / D3D10 / OpenGL / Vulkan) and the
XInput / focus / MKB hooks did not — `remove_*` freed the hook's shared D3D device / keyed-mutex
texture (or the Vulkan read-back resources, or the IPC pointer) *immediately*. So spamming a
subsystem toggle (the "Mirror video" button) freed that state while a capture detour was still
mid-flight on the game's render thread → use-after-free → the game crashed (Brotato, on its
OpenGL path; reproduced across every backend by the `mock_game_test` storm). The generalised fix
(`hook/src/hook_guard.hpp`, `DetourGate`): every detour wraps its body in an RAII active-count
`Guard`; `remove_*` (1) restores the hook so **no new detour can start** — reset the inline hook
(SafetyHook's mutex-guarded call wrappers make any in-flight trampoline call safe), or, for the
focus WNDPROC subclass, restore the window proc — then (2) `drain()`s the active count to zero,
and only **then** (3) frees the shared state. **Vulkan is the exception**: the game caches our
`hk_vkQueuePresentKHR` pointer at resolution time and keeps calling it even after the GPA hook is
reset, so a reset can't stop new detours — instead removal closes an atomic **capture gate**
first (the detour then passes straight through to the real present without touching the read-back
state), drains, and only then frees. The drain is bounded (~400 ms) so a wedged game thread can't
hang the worker; detours are micro- to milliseconds, so it returns almost immediately.
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
the backbuffer inside the game's `Present`, which the game issues at its true render
rate regardless of how DWM composites that *window*. So an unfocused game window can