Deduplicate the hook DLL and scrub history from its comments

Consolidate four copies of the keyed-mutex shared-texture setup
(present/opengl/d3d9/vk_capture) into one RAII SharedVideoTexture,
two copies of find_main_window into find_window.hpp, audio_hook's
hand-rolled detour guard into the shared DetourGate, the duplicated
vtable_method into vtable_hook.hpp, and the near-identical
Present/Present1 and SwapBuffers/wglSwapBuffers detour pairs into one
shared body each. The vk_layer and vk_capture_perf_test targets now
compile debug_log.cpp since the shared texture code logs.

Comments no longer narrate the past: drop stress-test/game anecdotes,
"used to"/"the old model" phrasing, plan-step labels, and pointers to
docs that do not exist; fix present_hook.hpp/opengl_hook.hpp claims
that predate the D3D12/D3D9/Vulkan backends. Net -266 lines, no
behavior change (full x64 + x86 suites pass, including the mock-game
hook/unhook storm).
This commit is contained in:
2026-07-12 08:53:58 +02:00
parent 05039ab104
commit 635ef51283
22 changed files with 358 additions and 624 deletions

View File

@@ -4,9 +4,9 @@
// The hazard: remove_*_hooks restores the hook and then frees the shared state the detour
// touches (D3D device/context, the keyed-mutex texture, Vulkan read-back resources, the IPC
// pointer). A capture detour mid-flight on the game's render thread then uses freed memory ->
// use-after-free -> the game crashes (the "spamming Mirror video crashed Brotato" bug).
// use-after-free -> the game crashes.
//
// The fix mirrors the audio hooks' epoch+drain pattern, generalised for inline hooks:
// The removal sequence that prevents it:
// 1. Disable the hook FIRST so no NEW detour can start. For a SafetyHook inline hook that's
// `disable_for_removal(hook)` (disable, NOT `= {}` destroy): it restores the original bytes but
// keeps the trampoline alive -- hooks are PERSISTENT, never destroyed mid-session, so an in-flight
@@ -60,8 +60,8 @@ public:
// instruction prologue is unguarded), so m_active reads 0 even though a detour is about to run.
// Returning then would free the state out from under it. With the hook disabled no NEW detour can
// start, so any such thread reaches its Guard within nanoseconds; a 1 ms settle before concluding
// "zero" lets it register. Without this, a backend presenting at thousands/s (the uncapped
// mock-game storm) reliably crashed on remove (0xC0000005); with it, the count is accurate.
// "zero" lets it register. A backend presenting at thousands of frames/s hits this window
// reliably, so checking before the first sleep is not safe.
void drain()
{
for (int spins = 0; spins < 400; ++spins)