BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden every hook against the install/remove use-after-free

Browse Source 
Spamming a subsystem toggle (the "Mirror video" button) could crash the game:
remove_*_hooks freed a hook's shared D3D / Vulkan / IPC state immediately, while a
capture detour was still mid-flight on the game's render thread -> use-after-free.
Only the audio hooks had the safe-unhook drain; the video (Present/D3D9/D3D10/GL/
Vulkan) and XInput/focus/MKB hooks did not.

Test-first: mock_game_test now runs an aggressive hook/unhook storm -- a separate
thread thrashes every subsystem on/off while the game presents, across all backends.
It crashed gl + vk (0xC0000005) and failed dx9 capture-resume before the fix.

Fix (hook/src/hook_guard.hpp, DetourGate): each detour wraps its body in an RAII
active-count Guard; remove_* restores the hook first (so no new detour starts),
drains the in-flight detours to zero, and only then frees the shared state. Vulkan
is special-cased -- the game caches hk_vkQueuePresentKHR, so removal closes an
atomic capture gate (detours then pass through to the real present), drains, then
frees the read-back resources. Storm now passes on every backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-23 01:56:36 +02:00
parent 79399e88f1
commit 21c15b162b
10 changed files with 365 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
hook/src/opengl_hook.cpp
View File

@@ -13,6 +13,7 @@
#include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_registry.hpp"
namespace coop::hook
@@ -21,6 +22,8 @@ namespace coop::hook
namespace
{
DetourGate g_gate; // drains in-flight swap detours before remove frees the shared D3D state
// OpenGL enums (avoid pulling in <GL/gl.h> / linking opengl32 at load time).
constexpr unsigned GL_RGBA = 0x1908;
constexpr unsigned GL_UNSIGNED_BYTE = 0x1401;
@@ -234,6 +237,7 @@ void capture_gl(HDC hdc)
BOOL WINAPI hk_SwapBuffers(HDC hdc)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
hook_note_call(g_id_swapbuffers);
g_swaps.fetch_add(1, std::memory_order_relaxed);
const bool outer = !t_in_swap;
@@ -256,6 +260,7 @@ BOOL WINAPI hk_SwapBuffers(HDC hdc)
BOOL WINAPI hk_wglSwapBuffers(HDC hdc)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
hook_note_call(g_id_wglswap);
g_swaps.fetch_add(1, std::memory_order_relaxed);
const bool outer = !t_in_swap;
@@ -317,10 +322,14 @@ bool install_opengl_hooks(IpcClient& ipc)
void remove_opengl_hooks()
{
// Reset the hooks first (restores the original SwapBuffers bytes under thread suspension, so
// no NEW detour starts), then drain any swap detour still in-flight on the render thread
// BEFORE freeing the D3D state it reads -- otherwise the detour uses freed memory (UAF).
g_hk_swapbuffers = {};
g_hk_wglswap = {};
hook_set_installed(g_id_swapbuffers, false);
hook_set_installed(g_id_wglswap, false);
g_gate.drain();
release_shared();
if (g_ctx != nullptr)
{