Harden every hook against the install/remove use-after-free
Spamming a subsystem toggle (the "Mirror video" button) could crash the game: remove_*_hooks freed a hook's shared D3D / Vulkan / IPC state immediately, while a capture detour was still mid-flight on the game's render thread -> use-after-free. Only the audio hooks had the safe-unhook drain; the video (Present/D3D9/D3D10/GL/ Vulkan) and XInput/focus/MKB hooks did not. Test-first: mock_game_test now runs an aggressive hook/unhook storm -- a separate thread thrashes every subsystem on/off while the game presents, across all backends. It crashed gl + vk (0xC0000005) and failed dx9 capture-resume before the fix. Fix (hook/src/hook_guard.hpp, DetourGate): each detour wraps its body in an RAII active-count Guard; remove_* restores the hook first (so no new detour starts), drains the in-flight detours to zero, and only then frees the shared state. Vulkan is special-cased -- the game caches hk_vkQueuePresentKHR, so removal closes an atomic capture gate (detours then pass through to the real present), drains, then frees the read-back resources. Storm now passes on every backend. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
42
README.md
42
README.md
@@ -109,20 +109,6 @@ default** and covers anything the hooked path doesn't.
|
||||
|
||||
### Current Tasks
|
||||
|
||||
- **Injection hardening — provoke, then fix, the hook install/remove races.** Rapidly
|
||||
toggling the **"Mirror video"** button has crashed a real game (Brotato): flipping the
|
||||
Present-hook video subsystem on/off in quick succession races hook install/remove against
|
||||
the game's Present thread and the host's capture path. Per the test-first rule, *first make
|
||||
the crash reproducible in a test* — make `mock_game_test`'s hook/unhook stress against
|
||||
`coop_mock_game` far more aggressive: tight, repeated subsystem toggles (especially video)
|
||||
while the game is actively presenting, driven from a separate thread so the toggles interleave
|
||||
with the Present detour, sustained long enough that an unguarded race is near-certain to
|
||||
fault, and across all backends (DX11/12/9/10/GL/Vulkan). Then fix what it surfaces — the
|
||||
install/remove path is the same safe-unhook problem the **audio** hooks already solved (bump
|
||||
an epoch on each toggle, restore the vtable/inline first so no new detour starts, then drain
|
||||
in-flight detours before tearing down shared state); apply the equivalent guard to the
|
||||
Present / D3D9 / D3D10 / OpenGL / Vulkan hooks.
|
||||
|
||||
- **Determine a pre-existing stream's audio format by *correlating* the two capture
|
||||
paths, instead of guessing.** When we attach to an already-running game we never saw its
|
||||
`IAudioClient::Initialize`, so the render-hook assumes the device mix format and measures
|
||||
@@ -297,9 +283,13 @@ ctest --test-dir build -C Debug --output-on-failure
|
||||
relaunch banner). It launches the game at several **audio formats**
|
||||
(44100/48000/96000, PCM + float) and asserts the hook measures each one's rate through the full
|
||||
inject path, then injects with audio + video, checks both stream, cycles the audio subsystem
|
||||
off/on (hook/unhook stress), and confirms the game never crashes and capture resumes. This suite
|
||||
drove out five real audio races (see Lessons learned). Skips cleanly without a D3D11 / Vulkan
|
||||
device.
|
||||
off/on (hook/unhook stress), and confirms the game never crashes and capture resumes. It then
|
||||
runs an **aggressive hook/unhook storm** — a separate thread thrashes *every* subsystem on/off as
|
||||
fast as the worker reconciles while the game is actively presenting, across **all** backends — to
|
||||
catch an install/remove race that frees a hook's shared D3D / Vulkan state under a live capture
|
||||
detour (the use-after-free that crashed a real game when its "Mirror video" button was spammed).
|
||||
This suite drove out five real audio races plus the cross-backend unhook race (see Lessons
|
||||
learned). Skips cleanly without a D3D11 / Vulkan device.
|
||||
|
||||
### Debugging the hooks against a real game
|
||||
|
||||
@@ -588,6 +578,24 @@ Non-obvious things that cost time and constrain the design:
|
||||
churn from re-creating the probe each toggle (build it once, keep it, only swap vtable
|
||||
slots). **Silently silencing/zeroing a buffer whose true size you only guessed is an
|
||||
over-write, not just an over-read** — clamp the read, but don't write what you can't size.
|
||||
- **Every removable hook needs the same safe-unhook drain, not just audio.** The audio hooks
|
||||
learned to restore the vtable slot first and *drain in-flight detours* before tearing down the
|
||||
shared state they read; the video (Present / D3D9 / D3D10 / OpenGL / Vulkan) and the
|
||||
XInput / focus / MKB hooks did not — `remove_*` freed the hook's shared D3D device / keyed-mutex
|
||||
texture (or the Vulkan read-back resources, or the IPC pointer) *immediately*. So spamming a
|
||||
subsystem toggle (the "Mirror video" button) freed that state while a capture detour was still
|
||||
mid-flight on the game's render thread → use-after-free → the game crashed (Brotato, on its
|
||||
OpenGL path; reproduced across every backend by the `mock_game_test` storm). The generalised fix
|
||||
(`hook/src/hook_guard.hpp`, `DetourGate`): every detour wraps its body in an RAII active-count
|
||||
`Guard`; `remove_*` (1) restores the hook so **no new detour can start** — reset the inline hook
|
||||
(SafetyHook's mutex-guarded call wrappers make any in-flight trampoline call safe), or, for the
|
||||
focus WNDPROC subclass, restore the window proc — then (2) `drain()`s the active count to zero,
|
||||
and only **then** (3) frees the shared state. **Vulkan is the exception**: the game caches our
|
||||
`hk_vkQueuePresentKHR` pointer at resolution time and keeps calling it even after the GPA hook is
|
||||
reset, so a reset can't stop new detours — instead removal closes an atomic **capture gate**
|
||||
first (the detour then passes straight through to the real present without touching the read-back
|
||||
state), drains, and only then frees. The drain is bounded (~400 ms) so a wedged game thread can't
|
||||
hang the worker; detours are micro- to milliseconds, so it returns almost immediately.
|
||||
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
|
||||
the backbuffer inside the game's `Present`, which the game issues at its true render
|
||||
rate regardless of how DWM composites that *window*. So an unfocused game window can
|
||||
|
||||
Reference in New Issue
Block a user