BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden every hook against the install/remove use-after-free

Browse Source 
Spamming a subsystem toggle (the "Mirror video" button) could crash the game:
remove_*_hooks freed a hook's shared D3D / Vulkan / IPC state immediately, while a
capture detour was still mid-flight on the game's render thread -> use-after-free.
Only the audio hooks had the safe-unhook drain; the video (Present/D3D9/D3D10/GL/
Vulkan) and XInput/focus/MKB hooks did not.

Test-first: mock_game_test now runs an aggressive hook/unhook storm -- a separate
thread thrashes every subsystem on/off while the game presents, across all backends.
It crashed gl + vk (0xC0000005) and failed dx9 capture-resume before the fix.

Fix (hook/src/hook_guard.hpp, DetourGate): each detour wraps its body in an RAII
active-count Guard; remove_* restores the hook first (so no new detour starts),
drains the in-flight detours to zero, and only then frees the shared state. Vulkan
is special-cased -- the game caches hk_vkQueuePresentKHR, so removal closes an
atomic capture gate (detours then pass through to the real present), drains, then
frees the read-back resources. Storm now passes on every backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-23 01:56:36 +02:00
parent 79399e88f1
commit 21c15b162b
10 changed files with 365 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
hook/src/d3d9_hook.cpp
View File

@@ -14,6 +14,7 @@
#include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_registry.hpp"
namespace coop::hook
@@ -22,6 +23,8 @@ namespace coop::hook
namespace
{
DetourGate g_gate; // drains in-flight Present detours before remove frees the shared D3D state
// IDirect3DDevice9 vtable: IUnknown 0-2, then the device methods. Present is index 17
// (TestCooperativeLevel 3, GetAvailableTextureMem 4, EvictManagedResources 5, GetDirect3D 6,
// GetDeviceCaps 7, GetDisplayMode 8, GetCreationParameters 9, SetCursorProperties 10,
@@ -260,6 +263,7 @@ void capture_d3d9(IDirect3DDevice9* dev)
HRESULT STDMETHODCALLTYPE hk_Present9(IDirect3DDevice9* dev, const RECT* src, const RECT* dst, HWND wnd,
const RGNDATA* dirty)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
hook_note_call(g_id_present9);
g_presents.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
@@ -362,8 +366,11 @@ bool install_d3d9_hooks(IpcClient& ipc)
void remove_d3d9_hooks()
{
// Reset the hook first (restores Present's bytes under thread suspension -> no new detour),
// then drain any Present detour still in-flight before freeing the D3D state it reads (UAF).
g_hk_present9 = {};
hook_set_installed(g_id_present9, false);
g_gate.drain();
release_shared();
release_sysmem();
if (g_ctx != nullptr)