BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden every hook against the install/remove use-after-free

Browse Source 
Spamming a subsystem toggle (the "Mirror video" button) could crash the game:
remove_*_hooks freed a hook's shared D3D / Vulkan / IPC state immediately, while a
capture detour was still mid-flight on the game's render thread -> use-after-free.
Only the audio hooks had the safe-unhook drain; the video (Present/D3D9/D3D10/GL/
Vulkan) and XInput/focus/MKB hooks did not.

Test-first: mock_game_test now runs an aggressive hook/unhook storm -- a separate
thread thrashes every subsystem on/off while the game presents, across all backends.
It crashed gl + vk (0xC0000005) and failed dx9 capture-resume before the fix.

Fix (hook/src/hook_guard.hpp, DetourGate): each detour wraps its body in an RAII
active-count Guard; remove_* restores the hook first (so no new detour starts),
drains the in-flight detours to zero, and only then frees the shared state. Vulkan
is special-cased -- the game caches hk_vkQueuePresentKHR, so removal closes an
atomic capture gate (detours then pass through to the real present), drains, then
frees the read-back resources. Storm now passes on every backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-23 01:56:36 +02:00
parent 79399e88f1
commit 21c15b162b
10 changed files with 365 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
hook/src/present_hook.cpp
View File

@@ -16,6 +16,7 @@
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_registry.hpp"
namespace coop::hook
@@ -24,6 +25,8 @@ namespace coop::hook
namespace
{
DetourGate g_gate; // drains in-flight Present/ECL detours before remove frees the shared state
// IDXGISwapChain vtable layout (frozen ABI). IUnknown 0..2, IDXGIObject 3..6,
// IDXGIDeviceSubObject 7, then IDXGISwapChain: Present = 8, GetBuffer = 9.
// IDXGISwapChain1 adds methods after IDXGISwapChain (18 methods, 0..17), so
@@ -755,6 +758,7 @@ void capture_backbuffer(IDXGISwapChain* sc)
void STDMETHODCALLTYPE hk_ExecuteCommandLists(ID3D12CommandQueue* queue, UINT num_lists,
ID3D12CommandList* const* lists)
{
DetourGate::Guard guard(g_gate); // keep g_present_queue/g_hk_ecl alive for this detour
// Record the graphics queue; compute/copy queues never present, so skip them and
// keep the last DIRECT one (the present queue on single-graphics-queue engines).
if (queue != nullptr && queue->GetDesc().Type == D3D12_COMMAND_LIST_TYPE_DIRECT)
@@ -806,6 +810,7 @@ void* grab_execute_command_lists_address()
HRESULT STDMETHODCALLTYPE hk_Present(IDXGISwapChain* sc, UINT sync_interval, UINT flags)
{
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
hook_note_call(g_id_present);
g_present_calls.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
@@ -836,6 +841,7 @@ HRESULT STDMETHODCALLTYPE hk_Present(IDXGISwapChain* sc, UINT sync_interval, UIN
HRESULT STDMETHODCALLTYPE hk_Present1(IDXGISwapChain1* sc, UINT sync_interval, UINT flags,
const DXGI_PRESENT_PARAMETERS* params)
{
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
hook_note_call(g_id_present1);
g_present_calls.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
@@ -978,12 +984,16 @@ bool install_present_hooks(IpcClient& ipc)
void remove_present_hooks()
{
// Reset the hooks first (restores Present/Present1/ECL bytes under thread suspension, so no
// NEW detour starts), then drain any detour still in-flight on the render thread BEFORE
// freeing the shared texture / On12 bridge / present queue it reads -- otherwise UAF.
g_hk_present = {};
g_hk_present1 = {};
g_hk_ecl = {};
hook_set_installed(g_id_present, false);
hook_set_installed(g_id_present1, false);
hook_set_installed(g_id_ecl, false);
g_gate.drain();
g_present_queue.store(nullptr, std::memory_order_relaxed);
g_logged_presents_n = 0; // let a fresh injection re-log the present pattern
g_logged_swapchains_n = 0;