BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Deduplicate the hook DLL and scrub history from its comments

Browse Source 
Consolidate four copies of the keyed-mutex shared-texture setup
(present/opengl/d3d9/vk_capture) into one RAII SharedVideoTexture,
two copies of find_main_window into find_window.hpp, audio_hook's
hand-rolled detour guard into the shared DetourGate, the duplicated
vtable_method into vtable_hook.hpp, and the near-identical
Present/Present1 and SwapBuffers/wglSwapBuffers detour pairs into one
shared body each. The vk_layer and vk_capture_perf_test targets now
compile debug_log.cpp since the shared texture code logs.

Comments no longer narrate the past: drop stress-test/game anecdotes,
"used to"/"the old model" phrasing, plan-step labels, and pointers to
docs that do not exist; fix present_hook.hpp/opengl_hook.hpp claims
that predate the D3D12/D3D9/Vulkan backends. Net -266 lines, no
behavior change (full x64 + x86 suites pass, including the mock-game
hook/unhook storm).
This commit is contained in:
BlackMark
2026-07-12 08:53:58 +02:00
parent 05039ab104
commit 635ef51283
22 changed files with 358 additions and 624 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
hook/src/audio_hook.cpp
View File

@@ -12,6 +12,7 @@
#include <mmreg.h>
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_registry.hpp"
#include "rate_estimator.hpp"
#include "vtable_hook.hpp"
@@ -45,8 +46,8 @@ using ReleaseBufferFn = HRESULT(STDMETHODCALLTYPE*)(IAudioRenderClient*, UINT32,
// The WASAPI COM methods are hooked by SWAPPING their vtable slot -- coop::hook::VtableHook from the
// shared vtable_hook.hpp -- rather than inline-hooking: on x86 the MMDevApi/AudioSes prologues do
// dynamic stack alignment (`and esp,-8`) with EBP-relative args that SafetyHook's trampoline
// relocation mishandles (it froze 32-bit FMOD games the instant audio init ran through the hook).
// Swapping the slot leaves the original code untouched. See the project's stdcall-x86 note.
// relocation mishandles (a 32-bit game freezes the instant audio init runs through such a hook).
// Swapping the slot leaves the original code untouched.
// The scalar audio format we forward; resolved from the game's WAVEFORMATEX.
struct CapturedFormat
@@ -61,9 +62,9 @@ struct CapturedFormat
// --- Global hook state -----------------------------------------------------
// Atomic: the audio hot path (hk_ReleaseBuffer) reads it without the setup lock, while
// remove_audio_hooks nulls it under the lock during an unhook. A plain pointer was a
// TOCTOU null-deref (check non-null, then it's nulled, then the call) -- a rapid
// hook/unhook crash the mock-game stress test caught. Load it once and use that.
// remove_audio_hooks nulls it under the lock during an unhook. A plain pointer would be a
// TOCTOU null-deref under a rapid hook/unhook (check non-null, then it's nulled, then the
// call). Load it once and use that.
std::atomic<IpcClient*> g_ipc{nullptr};
// One ring per tracked stream (index = the stream's debug slot). Stream 0 is the
// primary; the host creates a ring per stream and mixes them.
@@ -80,26 +81,12 @@ bool g_audioclient_hooked = false;
// Bumped on every detour install/remove. hk_GetBuffer stamps the current epoch into a
// thread-local; hk_ReleaseBuffer captures only if the epoch still matches -- so a
// GetBuffer/ReleaseBuffer pair that straddles a hook toggle (the stashed buffer pointer is
// then stale) is skipped instead of memset-ing a freed buffer (a rapid hook/unhook crash
// the mock-game stress test caught).
// then stale) is skipped instead of memset-ing a freed buffer.
std::atomic<std::uint32_t> g_hook_epoch{0};
// Number of detours (hk_GetBuffer/hk_ReleaseBuffer) currently executing on the audio
// thread. remove_audio_hooks restores the vtable slots (so no NEW detour starts) and then
// waits for this to drain to 0 before tearing down shared state -- guaranteeing no detour
// is mid-flight when state is cleared. The classic safe-unhook race; the alternative was an
// intermittent access violation in the game during a hook/unhook (the stress test caught it).
std::atomic<int> g_detours_active{0};
struct DetourGuard
{
DetourGuard()
{
g_detours_active.fetch_add(1, std::memory_order_acq_rel);
}
~DetourGuard()
{
g_detours_active.fetch_sub(1, std::memory_order_acq_rel);
}
};
// Drains in-flight GetBuffer/ReleaseBuffer detours: remove_audio_hooks restores the vtable
// slots (so no NEW detour starts) and then drains before tearing down the shared state a
// mid-flight detour reads (see hook_guard.hpp).
DetourGate g_gate;
// Registry ids for the hook list.
int g_id_activate = -1;
@@ -118,8 +105,8 @@ IAudioClient* g_self_client = nullptr;
std::atomic<IAudioRenderClient*> g_self_render{nullptr};
// The probe objects above are built ONCE and kept for the DLL's lifetime; an audio
// enable/disable toggle then only swaps vtable slots, never creates/destroys COM objects.
// Rapid create/destroy raced AudioSes and crashed the game (the mock-game stress test
// caught this). Released only on detach (shutdown_audio_hooks).
// Rapid create/destroy races AudioSes internals and crashes the game. Released only on
// detach (shutdown_audio_hooks).
// Device mix format, captured from our probe client. Used as the assumed format
// for a stream we discover on the hot path (we never saw its Initialize, so we
@@ -172,9 +159,8 @@ RateEstimator g_rate_estimator[kMaxAudioStreams] = {};
// too large and the capture copy would over-read the game's buffer. We can't detect the true
// layout (AUTOCONVERTPCM hands back a fixed staging buffer, so there's no buffer-stride to
// measure, and WASAPI exposes no API for a pre-existing client's format), so we instead clamp
// every guessed-stream copy to the source buffer's committed region (copy_bound_locked /
// readable_bytes) -- the audio may be misinterpreted, but it can never read past the
// allocation. See README "Lessons learned".
// every guessed-stream copy to the source buffer's committed region (readable_bytes) -- the
// audio may be misinterpreted, but it can never read past the allocation.
// Per-stream AudioFormatState (how its format was determined), mirrored to the host UI.
std::uint32_t g_stream_format_state[kMaxAudioStreams] = {};
@@ -244,7 +230,7 @@ std::uint32_t readable_bytes(const void* ptr, std::uint32_t want)
HRESULT STDMETHODCALLTYPE hk_GetBuffer(IAudioRenderClient* self, UINT32 num_frames, BYTE** data)
{
DetourGuard guard; // counts this detour as in-flight (drained before an unhook tears down)
DetourGate::Guard guard(g_gate); // in-flight until return (drained before an unhook tears down)
hook_note_call(g_id_getbuffer);
const HRESULT hr = g_vh_getbuffer.original<GetBufferFn>()(self, num_frames, data);
if (SUCCEEDED(hr) && data != nullptr)
@@ -259,7 +245,7 @@ HRESULT STDMETHODCALLTYPE hk_GetBuffer(IAudioRenderClient* self, UINT32 num_fram
HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_frames, DWORD flags)
{
DetourGuard guard; // counts this detour as in-flight (drained before an unhook tears down)
DetourGate::Guard guard(g_gate); // in-flight until return (drained before an unhook tears down)
hook_note_call(g_id_releasebuffer);
// A render client we've never seen actively rendering is almost certainly one
// the game created before we injected; adopt it now (the first becomes the
@@ -314,16 +300,13 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_
g_frames_captured.fetch_add(num_frames, std::memory_order_relaxed);
// Mute the game's local playback so the only audio is the host's re-render.
// Otherwise the game plays locally AND the mirror re-renders the same audio a
// few ms later = a metallic double (the Brotato symptom). AUDCLNT_BUFFERFLAGS_SILENT
// tells WASAPI to treat the buffer as silence and IGNORE its contents, so it
// mutes WITHOUT writing the buffer -- safe even for a guessed-format stream whose
// true frame size we don't know. (Muting used to be tied to the memset below,
// which is unsafe for a guessed block, so guessed streams -- the late-attach /
// Brotato case -- were captured but left audible. The flag is the actual mute;
// the memset is not needed for it.) For an exact/override format we additionally
// zero the buffer (belt-and-suspenders; `block` is the real frame size there, so
// it stays in-bounds). Only mutes once the frames made the ring (above) -- a
// stalled host degrades to echo, never to dead silence.
// few ms later = a metallic double. AUDCLNT_BUFFERFLAGS_SILENT tells WASAPI to
// treat the buffer as silence and IGNORE its contents, so it mutes WITHOUT
// writing the buffer -- safe even for a guessed-format stream whose true frame
// size we don't know; the flag is the mute, not the memset. For an exact/override
// format we additionally zero the buffer (belt-and-suspenders; `block` is the
// real frame size there, so it stays in-bounds). Only mutes once the frames made
// the ring (above) -- a stalled host degrades to echo, never to dead silence.
if (!guessed)
{
std::memset(t_gb_data, 0, bytes);
@@ -334,7 +317,7 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_
}
}
}
// Format-verification co-capture (host-driven, step 2a). While the host has set
// Format-verification co-capture (host-driven). While the host has set
// verify_capture and this guessed stream's rate is still being MEASURED (format not yet
// published), push the raw pre-mix bytes to the ring WITHOUT silencing, so the host can
// capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same
@@ -691,7 +674,7 @@ namespace
// Build the probe COM objects (enumerator -> device -> client -> render) and capture the
// device mix format. Created ONCE and kept for the DLL's lifetime: every instance of a
// coclass shares one vtable, so a toggle then only re-swaps vtable slots on these kept
// objects -- no COM create/destroy churn (which raced AudioSes). Caller holds g_setup_mutex.
// objects -- no COM create/destroy churn (which races AudioSes). Caller holds g_setup_mutex.
bool build_probe_locked()
{
if (g_self_device != nullptr)
@@ -868,17 +851,8 @@ void remove_audio_hooks()
hook_set_installed(g_id_releasebuffer, false);
// The slots are restored above, so no NEW detour will start. Drain any detour still
// in-flight on the audio thread before clearing the shared state it reads (an initial
// sleep covers a caller that read the old slot but hasn't entered the guard yet). Bounded
// so a wedged audio thread can't hang us. Detours are microseconds, so this is ~1-2 ms.
for (int spins = 0; spins < 200; ++spins)
{
Sleep(1);
if (g_detours_active.load(std::memory_order_acquire) == 0)
{
break;
}
}
// in-flight on the audio thread before clearing the shared state it reads.
g_gate.drain();
// Clear per-stream tracking (the game's streams re-register lazily on a re-enable).
g_registered = 0;

2
hook/src/audio_hook.hpp
View File

@@ -1,7 +1,7 @@
// Injected WASAPI render-hook: captures the game's audio render frames into the
// shared audio ring and silences the game's local playback, so the host can
// re-render the audio for Steam Remote Play Together without the operator
// hearing it twice (the "local audio echo"). See docs/audio-render-hook-plan.md.
// hearing it twice (the "local audio echo").
//
// The hooks always install (so render streams are counted for the debug view
// even with mirroring off); the copy+silence behavior is gated by the ring's

98
hook/src/d3d9_hook.cpp
View File

@@ -12,11 +12,12 @@
#include <safetyhook.hpp>
#include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_install.hpp"
#include "hook_registry.hpp"
#include "shared_video_texture.hpp"
#include "vtable_hook.hpp"
namespace coop::hook
{
@@ -48,11 +49,7 @@ bool g_unsupported_logged = false;
// Our own D3D11 device hosting the shared texture (the D3D9 game has no D3D11 device).
ID3D11Device* g_device = nullptr;
ID3D11DeviceContext* g_ctx = nullptr;
ID3D11Texture2D* g_shared_tex = nullptr;
IDXGIKeyedMutex* g_shared_mutex = nullptr;
HANDLE g_shared_handle = nullptr;
UINT g_share_w = 0;
UINT g_share_h = 0;
SharedVideoTexture g_shared;
// System-memory read-back surface on the game's D3D9 device (GetRenderTargetData target).
IDirect3DSurface9* g_sysmem = nullptr;
@@ -66,11 +63,6 @@ std::vector<unsigned char> g_rgba; // swizzled RGBA, uploaded to D3D11
// Present may be issued re-entrantly by some engines; capture only on the outermost call.
thread_local bool t_in_present = false;
void* vtable_method(void* obj, unsigned index)
{
return (*reinterpret_cast<void***>(obj))[index];
}
bool ensure_device()
{
if (g_device != nullptr)
@@ -87,26 +79,6 @@ bool ensure_device()
return true;
}
void release_shared()
{
if (g_shared_mutex != nullptr)
{
g_shared_mutex->Release();
g_shared_mutex = nullptr;
}
if (g_shared_tex != nullptr)
{
g_shared_tex->Release();
g_shared_tex = nullptr;
}
if (g_shared_handle != nullptr)
{
CloseHandle(g_shared_handle);
g_shared_handle = nullptr;
}
g_share_w = g_share_h = 0;
}
void release_sysmem()
{
if (g_sysmem != nullptr)
@@ -123,56 +95,6 @@ void release_sysmem()
g_sysmem_fmt = D3DFMT_UNKNOWN;
}
bool ensure_shared_texture(UINT w, UINT h)
{
if (g_shared_tex != nullptr && g_share_w == w && g_share_h == h)
{
return true;
}
release_shared();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM; // we swizzle the D3D9 BGRA backbuffer to RGBA
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
if (FAILED(g_device->CreateTexture2D(&desc, nullptr, &g_shared_tex)) || g_shared_tex == nullptr)
{
return false;
}
IDXGIResource1* res = nullptr;
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res))) ||
res == nullptr)
{
release_shared();
return false;
}
const std::wstring name = video_share_name(g_pid);
const HRESULT hr = res->CreateSharedHandle(
nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, name.c_str(), &g_shared_handle);
res->Release();
if (FAILED(hr) || g_shared_handle == nullptr)
{
release_shared();
return false;
}
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&g_shared_mutex))))
{
release_shared();
return false;
}
g_share_w = w;
g_share_h = h;
logf("d3d9: shared texture ready %ux%u name=%ls", w, h, name.c_str());
return true;
}
// Read the game's D3D9 backbuffer back to system memory, swizzle BGRA->RGBA, and upload it.
void capture_d3d9(IDirect3DDevice9* dev)
{
@@ -239,12 +161,14 @@ void capture_d3d9(IDirect3DDevice9* dev)
}
g_sysmem->UnlockRect();
if (ensure_device() && ensure_shared_texture(w, h) && g_shared_mutex != nullptr &&
g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK)
// DXGI_FORMAT_R8G8B8A8_UNORM: we swizzle the D3D9 BGRA backbuffer to RGBA above.
if (ensure_device() &&
g_shared.ensure(g_device, w, h, DXGI_FORMAT_R8G8B8A8_UNORM, g_pid, "d3d9") &&
g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{
g_ctx->UpdateSubresource(g_shared_tex, 0, nullptr, g_rgba.data(), static_cast<UINT>(dst_row), 0);
g_ctx->UpdateSubresource(g_shared.texture(), 0, nullptr, g_rgba.data(), static_cast<UINT>(dst_row), 0);
g_ctx->Flush();
g_shared_mutex->ReleaseSync(kVideoMutexKey);
g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true;
}
}
@@ -375,9 +299,7 @@ void remove_d3d9_hooks()
disable_for_removal(g_hk_present9);
hook_set_installed(g_id_present9, false);
g_gate.drain();
// Persistent hook: keep g_hk_present9 ALIVE (disabled) so a stale detour's trampoline call is
// never freed -- re-install re-enables it (see hook_install.hpp).
release_shared();
g_shared.release();
release_sysmem();
if (g_ctx != nullptr)
{

4
hook/src/dllmain.cpp
View File

@@ -230,8 +230,8 @@ DWORD WINAPI worker_thread(LPVOID)
g_ipc.set_vk_too_late(coop::hook::vk_injected_too_late()); // Vulkan attached-too-late banner
g_ipc.heartbeat();
// Reconcile ~4x/s, but drain MKB events far more often (input must be
// responsive). 50 slices x 5 ms ~= the old 250 ms reconcile period.
// Reconcile ~4x/s (50 slices x 5 ms), but drain MKB events every slice --
// input must stay responsive at a far higher rate than the reconcile.
for (int slice = 0; slice < 50 && g_running.load(std::memory_order_relaxed); ++slice)
{
if (mkb_installed)

46
hook/src/find_window.hpp Normal file
View File

@@ -0,0 +1,46 @@
// Locate the game's main window from inside the game process: the largest
// visible, unowned top-level window the pid owns. Shared by the focus spoof
// (subclass target) and the MKB forwarder (PostMessage target).
#pragma once
#include <windows.h>
namespace coop::hook
{
inline HWND find_main_window(DWORD pid)
{
struct Ctx
{
DWORD pid;
HWND best;
long best_area;
} ctx{pid, nullptr, 0};
EnumWindows(
[](HWND hwnd, LPARAM lparam) -> BOOL {
auto* c = reinterpret_cast<Ctx*>(lparam);
DWORD pid = 0;
GetWindowThreadProcessId(hwnd, &pid);
if (pid != c->pid || !IsWindowVisible(hwnd) || GetWindow(hwnd, GW_OWNER) != nullptr)
{
return TRUE; // not ours, hidden, or an owned dialog -- keep looking
}
RECT rect = {};
if (!GetWindowRect(hwnd, &rect))
{
return TRUE;
}
const long area = (rect.right - rect.left) * (rect.bottom - rect.top);
if (area > c->best_area)
{
c->best_area = area;
c->best = hwnd;
}
return TRUE;
},
reinterpret_cast<LPARAM>(&ctx));
return ctx.best;
}
} // namespace coop::hook

43
hook/src/focus_spoof.cpp
View File

@@ -6,6 +6,7 @@
#include <safetyhook.hpp>
#include "find_window.hpp"
#include "hook_guard.hpp"
#include "hook_install.hpp"
#include "hook_registry.hpp"
@@ -35,46 +36,6 @@ int g_id_setcursorpos = -1;
safetyhook::InlineHook g_hk_clipcursor;
safetyhook::InlineHook g_hk_setcursorpos;
struct EnumContext
{
DWORD pid;
HWND best;
long best_area;
};
BOOL CALLBACK enum_proc(HWND hwnd, LPARAM lparam)
{
auto* ctx = reinterpret_cast<EnumContext*>(lparam);
DWORD pid = 0;
GetWindowThreadProcessId(hwnd, &pid);
if (pid != ctx->pid || !IsWindowVisible(hwnd) || GetWindow(hwnd, GW_OWNER) != nullptr)
{
return TRUE; // not ours, hidden, or an owned dialog -- keep looking
}
RECT rect = {};
if (!GetWindowRect(hwnd, &rect))
{
return TRUE;
}
const long area = (rect.right - rect.left) * (rect.bottom - rect.top);
if (area > ctx->best_area)
{
ctx->best_area = area;
ctx->best = hwnd;
}
return TRUE;
}
// The game's main window = the largest visible, unowned top-level window it owns.
HWND find_main_window(DWORD pid)
{
EnumContext ctx{pid, nullptr, 0};
EnumWindows(&enum_proc, reinterpret_cast<LPARAM>(&ctx));
return ctx.best;
}
// Replacement window procedure: convince the game it is never deactivated.
LRESULT CALLBACK subclass_proc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam)
{
@@ -313,7 +274,7 @@ void remove_focus_spoof()
// GetForegroundWindow is still hooked -- its detour returns g_game_hwnd and never reaches the
// patched bytes. The reverse of the enable order (GFW first) keeps the invariant "GetActiveWindow
// hooked => GetForegroundWindow hooked" across the whole install/remove cycle, so a call never
// lands in a half-patched shared region (the intermittent storm crash: GetActiveWindow+0x8).
// lands in a half-patched shared region.
for (auto it = g_focus_hooks.rbegin(); it != g_focus_hooks.rend(); ++it)
{
disable_for_removal(*it);

8
hook/src/hook_guard.hpp
View File

@@ -4,9 +4,9 @@
// The hazard: remove_*_hooks restores the hook and then frees the shared state the detour
// touches (D3D device/context, the keyed-mutex texture, Vulkan read-back resources, the IPC
// pointer). A capture detour mid-flight on the game's render thread then uses freed memory ->
// use-after-free -> the game crashes (the "spamming Mirror video crashed Brotato" bug).
// use-after-free -> the game crashes.
//
// The fix mirrors the audio hooks' epoch+drain pattern, generalised for inline hooks:
// The removal sequence that prevents it:
// 1. Disable the hook FIRST so no NEW detour can start. For a SafetyHook inline hook that's
// `disable_for_removal(hook)` (disable, NOT `= {}` destroy): it restores the original bytes but
// keeps the trampoline alive -- hooks are PERSISTENT, never destroyed mid-session, so an in-flight
@@ -60,8 +60,8 @@ public:
// instruction prologue is unguarded), so m_active reads 0 even though a detour is about to run.
// Returning then would free the state out from under it. With the hook disabled no NEW detour can
// start, so any such thread reaches its Guard within nanoseconds; a 1 ms settle before concluding
// "zero" lets it register. Without this, a backend presenting at thousands/s (the uncapped
// mock-game storm) reliably crashed on remove (0xC0000005); with it, the count is accurate.
// "zero" lets it register. A backend presenting at thousands of frames/s hits this window
// reliably, so checking before the first sleep is not safe.
void drain()
{
for (int spins = 0; spins < 400; ++spins)

12
hook/src/hook_install.hpp
View File

@@ -1,17 +1,17 @@
// Safe INSTALL of a SafetyHook inline hook -- the symmetric partner of hook_guard.hpp's safe removal.
//
// Two problems this avoids, both surfaced by mock_game_test's hook/unhook storm (a game polling a
// hooked API at thousands/s while the subsystem is toggled on/off every 60 ms):
// Two problems this avoids, both hit when a game polls a hooked API at thousands of calls/s while
// the subsystem is toggled on/off (mock_game_test's hook/unhook storm exercises exactly that):
//
// 1. Enable-before-assign. create_inline() builds the hook AND enables it (patches the target's bytes
// to jump to the detour), THEN the result is move-assigned into the global the detour reads to
// reach the trampoline. A call landing in the detour during that assign reads a torn global -> AV.
// So we create StartDisabled, let dst be populated, and only THEN enable().
//
// 2. Trampoline use-after-free on re-install. The old model recreated the hook on every install and
// DESTROYED it on every remove (= {}), which frees the trampoline. A detour that has entered but
// not yet reached its DetourGate::Guard (the unguarded prologue) can then call a freed trampoline
// -> AV at a garbage address. drain() narrows that window but can't fully close it under heavy
// 2. Trampoline use-after-free on re-install. Recreating the hook on every install and destroying
// it on every remove (= {}) frees the trampoline. A detour that has entered but not yet reached
// its DetourGate::Guard (the unguarded prologue) can then call a freed trampoline -> AV at a
// garbage address. drain() narrows that window but can't fully close it under heavy
// preemption. So instead we treat hooks as PERSISTENT: create each once and thereafter only
// enable()/disable() it across install/remove cycles. The trampoline is allocated once and never
// freed during the session, so a stale detour always calls a live trampoline (which, when

48
hook/src/mkb_hook.cpp
View File

@@ -9,6 +9,7 @@
#include <safetyhook.hpp>
#include "find_window.hpp"
#include "hook_guard.hpp"
#include "hook_install.hpp"
#include "hook_registry.hpp"
@@ -278,40 +279,6 @@ void post_raw_mouse(HWND hwnd, USHORT button_flags)
// --- Target window + message synthesis --------------------------------------
struct FindCtx
{
DWORD pid;
HWND best;
long area;
};
BOOL CALLBACK find_main_proc(HWND hwnd, LPARAM lparam)
{
auto* c = reinterpret_cast<FindCtx*>(lparam);
DWORD pid = 0;
GetWindowThreadProcessId(hwnd, &pid);
if (pid != c->pid || !IsWindowVisible(hwnd) || GetWindow(hwnd, GW_OWNER) != nullptr)
{
return TRUE;
}
RECT r{};
GetWindowRect(hwnd, &r);
const long area = (r.right - r.left) * (r.bottom - r.top);
if (area > c->area)
{
c->area = area;
c->best = hwnd;
}
return TRUE;
}
HWND find_main_window()
{
FindCtx c{GetCurrentProcessId(), nullptr, 0};
EnumWindows(&find_main_proc, reinterpret_cast<LPARAM>(&c));
return c.best;
}
LPARAM key_lparam(UINT vk, bool key_up)
{
const UINT scan = MapVirtualKeyW(vk, MAPVK_VK_TO_VSC);
@@ -524,19 +491,16 @@ void remove_mkb_hooks()
}
g_active.store(false, std::memory_order_release);
// Disable (restore original bytes) the inline hooks so no new detour starts, KEEPING the
// trampolines alive for any in-flight detour calling its trampoline (persistent model -- never
// destroyed during the session; re-install re-enables, see hook_install.hpp). Destroying would
// free a trampoline under a detour about to call it -- the present-storm UAF class. (The DI hook
// is a vtable swap: remove() restores the slot and keeps m_original valid, no trampoline either.)
// trampolines alive for any in-flight detour calling its trampoline -- these detours DO call
// the trampoline, so destroying one would free it under a detour about to call it (persistent
// model, re-enabled on re-install; see hook_install.hpp). The DI hook is a vtable swap:
// remove() restores the slot and keeps m_original valid, no trampoline involved.
disable_for_removal(g_hk_async);
disable_for_removal(g_hk_kbstate);
disable_for_removal(g_hk_cursor);
disable_for_removal(g_hk_getrawinputdata);
g_vh_di_getstate.remove(); // restore the DI GetDeviceState slot (probe kept alive for re-enable)
g_gate.drain(); // wait for any in-flight polling / DI / raw detour before clearing state
// Persistent hooks: keep g_hk_async/kbstate/cursor/getrawinputdata ALIVE (disabled), so a stale
// detour's trampoline call (these detours DO call the trampoline) is never freed -- re-install
// re-enables them (see hook_install.hpp).
hook_set_installed(g_id_di_getstate, false);
hook_set_installed(g_id_rawinput, false);
for (int vk = 0; vk < 256; ++vk)
@@ -566,7 +530,7 @@ void mkb_pump(IpcClient& ipc)
HWND hwnd = static_cast<HWND>(g_target.load(std::memory_order_relaxed));
if (hwnd == nullptr || !IsWindow(hwnd))
{
hwnd = find_main_window();
hwnd = find_main_window(GetCurrentProcessId());
g_target.store(hwnd, std::memory_order_relaxed);
}

127
hook/src/opengl_hook.cpp
View File

@@ -11,11 +11,11 @@
#include <safetyhook.hpp>
#include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_install.hpp"
#include "hook_registry.hpp"
#include "shared_video_texture.hpp"
namespace coop::hook
{
@@ -54,11 +54,7 @@ bool g_unsupported_logged = false;
// Our own D3D11 device hosting the shared texture (the GL path has no D3D device).
ID3D11Device* g_device = nullptr;
ID3D11DeviceContext* g_ctx = nullptr;
ID3D11Texture2D* g_shared_tex = nullptr;
IDXGIKeyedMutex* g_shared_mutex = nullptr;
HANDLE g_shared_handle = nullptr;
UINT g_share_w = 0;
UINT g_share_h = 0;
SharedVideoTexture g_shared;
std::vector<unsigned char> g_read_buf; // glReadPixels target (bottom-up)
std::vector<unsigned char> g_flip_buf; // vertically flipped, uploaded to D3D
@@ -100,76 +96,6 @@ bool ensure_device()
return true;
}
void release_shared()
{
if (g_shared_mutex != nullptr)
{
g_shared_mutex->Release();
g_shared_mutex = nullptr;
}
if (g_shared_tex != nullptr)
{
g_shared_tex->Release();
g_shared_tex = nullptr;
}
if (g_shared_handle != nullptr)
{
CloseHandle(g_shared_handle);
g_shared_handle = nullptr;
}
g_share_w = g_share_h = 0;
}
bool ensure_shared_texture(UINT w, UINT h)
{
if (g_shared_tex != nullptr && g_share_w == w && g_share_h == h)
{
return true;
}
release_shared();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM; // glReadPixels(GL_RGBA) byte order
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
if (FAILED(g_device->CreateTexture2D(&desc, nullptr, &g_shared_tex)) || g_shared_tex == nullptr)
{
return false;
}
IDXGIResource1* res = nullptr;
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res))) ||
res == nullptr)
{
release_shared();
return false;
}
const std::wstring name = video_share_name(g_pid);
const HRESULT hr = res->CreateSharedHandle(
nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, name.c_str(), &g_shared_handle);
res->Release();
if (FAILED(hr) || g_shared_handle == nullptr)
{
release_shared();
return false;
}
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&g_shared_mutex))))
{
release_shared();
return false;
}
g_share_w = w;
g_share_h = h;
logf("opengl: shared texture ready %ux%u name=%ls", w, h, name.c_str());
return true;
}
// Read the current GL backbuffer for the window behind `hdc` and upload it.
void capture_gl(HDC hdc)
{
@@ -197,7 +123,8 @@ void capture_gl(HDC hdc)
return;
}
if (!ensure_device() || !ensure_shared_texture(w, h))
// DXGI_FORMAT_R8G8B8A8_UNORM matches glReadPixels(GL_RGBA) byte order.
if (!ensure_device() || !g_shared.ensure(g_device, w, h, DXGI_FORMAT_R8G8B8A8_UNORM, g_pid, "opengl"))
{
return;
}
@@ -223,11 +150,11 @@ void capture_gl(HDC hdc)
memcpy(g_flip_buf.data() + y * row, g_read_buf.data() + (h - 1 - y) * row, row);
}
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK)
if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{
g_ctx->UpdateSubresource(g_shared_tex, 0, nullptr, g_flip_buf.data(), static_cast<UINT>(row), 0);
g_ctx->UpdateSubresource(g_shared.texture(), 0, nullptr, g_flip_buf.data(), static_cast<UINT>(row), 0);
g_ctx->Flush();
g_shared_mutex->ReleaseSync(kVideoMutexKey);
g_shared.mutex()->ReleaseSync(kVideoMutexKey);
g_frames_shared.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
{
@@ -236,10 +163,11 @@ void capture_gl(HDC hdc)
}
}
BOOL WINAPI hk_SwapBuffers(HDC hdc)
// Shared body of both swap detours: count the swap (SwapBuffers/wglSwapBuffers is the GL
// present), capture on the outermost call only, then forward through `hook`'s trampoline.
BOOL swap_detour(safetyhook::InlineHook& hook, int hook_id, HDC hdc)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
hook_note_call(g_id_swapbuffers);
hook_note_call(hook_id);
g_swaps.fetch_add(1, std::memory_order_relaxed);
const bool outer = !t_in_swap;
if (outer)
@@ -247,11 +175,11 @@ BOOL WINAPI hk_SwapBuffers(HDC hdc)
t_in_swap = true;
if (g_ipc != nullptr)
{
g_ipc->note_present(); // SwapBuffers is the GL present (counts toward video.present_calls)
g_ipc->note_present();
}
capture_gl(hdc);
}
const BOOL r = g_hk_swapbuffers.stdcall<BOOL>(hdc); // __stdcall: call() is __cdecl on x86 -> crash
const BOOL r = hook.stdcall<BOOL>(hdc); // __stdcall: call() is __cdecl on x86 -> crash
if (outer)
{
t_in_swap = false;
@@ -259,27 +187,16 @@ BOOL WINAPI hk_SwapBuffers(HDC hdc)
return r;
}
BOOL WINAPI hk_SwapBuffers(HDC hdc)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
return swap_detour(g_hk_swapbuffers, g_id_swapbuffers, hdc);
}
BOOL WINAPI hk_wglSwapBuffers(HDC hdc)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
hook_note_call(g_id_wglswap);
g_swaps.fetch_add(1, std::memory_order_relaxed);
const bool outer = !t_in_swap;
if (outer)
{
t_in_swap = true;
if (g_ipc != nullptr)
{
g_ipc->note_present(); // wglSwapBuffers is the GL present
}
capture_gl(hdc);
}
const BOOL r = g_hk_wglswap.stdcall<BOOL>(hdc); // __stdcall: call() is __cdecl on x86 -> crash
if (outer)
{
t_in_swap = false;
}
return r;
return swap_detour(g_hk_wglswap, g_id_wglswap, hdc);
}
} // namespace
@@ -333,9 +250,7 @@ void remove_opengl_hooks()
hook_set_installed(g_id_swapbuffers, false);
hook_set_installed(g_id_wglswap, false);
g_gate.drain();
// Persistent hooks: keep them ALIVE (disabled) so a stale detour's trampoline call is never freed
// -- re-install re-enables them (see hook_install.hpp).
release_shared();
g_shared.release();
if (g_ctx != nullptr)
{
g_ctx->Release();

9
hook/src/opengl_hook.hpp
View File

@@ -1,13 +1,12 @@
// Injected OpenGL capture path: hooks SwapBuffers / wglSwapBuffers in OpenGL games
// (which never call IDXGISwapChain::Present, so the DXGI present hook can't see
// them -- e.g. Phantom Brave), reads the GL backbuffer with glReadPixels, and
// uploads it into the same shared keyed-mutex texture (coop_video_<pid>) the host
// samples. Part of the video subsystem, alongside present_hook.
// them), reads the GL backbuffer with glReadPixels, and uploads it into the same
// shared keyed-mutex texture (coop_video_<pid>) the host samples. Part of the video
// subsystem, alongside present_hook, d3d9_hook, and vk_hook.
//
// glReadPixels forces a GPU->CPU readback each frame, so this is heavier than the
// D3D shared-texture copy; fine for the 2D / lower-framerate games that tend to be
// OpenGL. Vulkan games present via vkQueuePresentKHR and aren't covered here (see
// the README) -- use WGC for those.
// OpenGL.
#pragma once
#include "ipc_client.hpp"

209
hook/src/present_hook.cpp
View File

@@ -14,11 +14,13 @@
#include <safetyhook.hpp>
#include "coop/shared_memory.hpp"
#include "coop/protocol.hpp"
#include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_install.hpp"
#include "hook_registry.hpp"
#include "shared_video_texture.hpp"
#include "vtable_hook.hpp"
namespace coop::hook
{
@@ -50,13 +52,11 @@ std::atomic<std::uint64_t> g_frames_shared{0};
// Present once we can see the game's device + backbuffer format. Guarded by
// g_tex_mutex (touched only on the render thread, but install/remove may race).
std::mutex g_tex_mutex;
ID3D11Texture2D* g_shared_tex = nullptr;
IDXGIKeyedMutex* g_shared_mutex = nullptr;
HANDLE g_shared_handle = nullptr;
UINT g_share_w = 0;
UINT g_share_h = 0;
DXGI_FORMAT g_share_fmt = DXGI_FORMAT_UNKNOWN;
SharedVideoTexture g_shared;
bool g_unsupported_logged = false;
// The Present hook's shared texture also carries RENDER_TARGET so the On12-wrapped D3D12
// backbuffer can be copied into it.
constexpr UINT kShareBind = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
// D3D11On12 bridge for D3D12 games: we build our own D3D11 device on top of the
// game's D3D12 device (with a queue we create on it) so we can wrap the D3D12
@@ -162,103 +162,6 @@ bool first_capture_from(void* swapchain)
return true;
}
void* vtable_method(void* obj, unsigned index)
{
return (*reinterpret_cast<void***>(obj))[index];
}
// Drop the shared texture/mutex/handle. Caller holds g_tex_mutex.
void release_shared_locked()
{
if (g_shared_mutex != nullptr)
{
g_shared_mutex->Release();
g_shared_mutex = nullptr;
}
if (g_shared_tex != nullptr)
{
g_shared_tex->Release();
g_shared_tex = nullptr;
}
if (g_shared_handle != nullptr)
{
CloseHandle(g_shared_handle);
g_shared_handle = nullptr;
}
g_share_w = g_share_h = 0;
g_share_fmt = DXGI_FORMAT_UNKNOWN;
}
// (Re)create the shared keyed-mutex texture for a w x h `fmt` backbuffer on the
// game's `device`. Caller holds g_tex_mutex. Returns true if it's ready.
bool ensure_shared_texture_locked(ID3D11Device* device, UINT w, UINT h, DXGI_FORMAT fmt)
{
if (g_shared_tex != nullptr && g_share_w == w && g_share_h == h && g_share_fmt == fmt)
{
return true; // already matches the current backbuffer
}
release_shared_locked();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = fmt;
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
ID3D11Texture2D* tex = nullptr;
HRESULT hr = device->CreateTexture2D(&desc, nullptr, &tex);
if (FAILED(hr) || tex == nullptr)
{
logf("present: CreateTexture2D(shared) failed hr=0x%08lX (%ux%u fmt=%d)",
static_cast<unsigned long>(hr), w, h, static_cast<int>(fmt));
return false;
}
IDXGIResource1* res = nullptr;
hr = tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res));
if (FAILED(hr) || res == nullptr)
{
logf("present: QI IDXGIResource1 failed hr=0x%08lX", static_cast<unsigned long>(hr));
tex->Release();
return false;
}
const std::wstring name = video_share_name(g_pid);
HANDLE handle = nullptr;
hr = res->CreateSharedHandle(nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE,
name.c_str(), &handle);
res->Release();
if (FAILED(hr) || handle == nullptr)
{
logf("present: CreateSharedHandle failed hr=0x%08lX", static_cast<unsigned long>(hr));
tex->Release();
return false;
}
IDXGIKeyedMutex* mutex = nullptr;
hr = tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&mutex));
if (FAILED(hr) || mutex == nullptr)
{
logf("present: QI IDXGIKeyedMutex failed hr=0x%08lX", static_cast<unsigned long>(hr));
CloseHandle(handle);
tex->Release();
return false;
}
g_shared_tex = tex;
g_shared_mutex = mutex;
g_shared_handle = handle;
g_share_w = w;
g_share_h = h;
g_share_fmt = fmt;
logf("present: shared texture ready %ux%u fmt=%d name=%ls", w, h, static_cast<int>(fmt), name.c_str());
return true;
}
// Drop the D3D11On12 bridge. Caller holds g_tex_mutex.
void release_on12_locked()
{
@@ -425,15 +328,13 @@ void capture_backbuffer_d3d12(IDXGISwapChain* sc)
std::scoped_lock lock(g_tex_mutex);
if (ensure_on12_locked(dev))
{
// DX12 capture is pricier than DX11/OpenGL (measured ~0.38 ms vs ~0.05/0.09 ms of
// present-thread overhead) because it goes through the D3D11On12 bridge. The cost is NOT
// CreateWrappedResource (measured ~0.012 ms) -- it's the CopyResource issued on the 11On12
// immediate context (~0.13 ms) plus the mandatory Flush to make the shared copy visible to
// the host (~0.06 ms), neither of which the native-D3D11 path pays. Eliminating it needs a
// native-D3D12 copy-queue path into a D3D12-shared texture, but the host consumes the
// shared surface via an IDXGIKeyedMutex (a D3D11 concept), so that also means switching the
// DX12 producer<->host sync to a shared ID3D12Fence -- a cross-API rewrite. Deferred: the
// overhead is ~5% of a 144 Hz frame and the capture is correct; the bridge stays for now.
// DX12 capture costs more present-thread overhead than DX11/OpenGL (~0.38 ms vs
// ~0.05/0.09 ms, measured) because it goes through the D3D11On12 bridge: the
// CopyResource on the 11On12 immediate context (~0.13 ms) plus the mandatory Flush
// (~0.06 ms) that makes the shared copy visible to the host -- CreateWrappedResource
// itself is ~0.012 ms. Avoiding the bridge needs a native-D3D12 copy-queue path into a
// D3D12-shared texture AND a shared-fence host sync (the keyed mutex is a D3D11
// concept) -- a cross-API rewrite that isn't worth ~5% of a 144 Hz frame.
//
// Order our copy after the game's frame without burdening the game's queue: the
// game queue signals the fence (cheap), our copy queue waits on it. Skipped if the
@@ -464,13 +365,13 @@ void capture_backbuffer_d3d12(IDXGISwapChain* sc)
w = d.Width;
h = d.Height;
fmt = d.Format;
if (d.SampleDesc.Count == 1 && ensure_shared_texture_locked(g_on12_d3d11, w, h, fmt) &&
g_shared_mutex != nullptr)
if (d.SampleDesc.Count == 1 &&
g_shared.ensure(g_on12_d3d11, w, h, fmt, g_pid, "present", kShareBind))
{
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 0) == S_OK)
if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 0) == S_OK)
{
g_on12_ctx->CopyResource(g_shared_tex, wtex);
g_shared_mutex->ReleaseSync(kVideoMutexKey);
g_on12_ctx->CopyResource(g_shared.texture(), wtex);
g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true;
}
else
@@ -623,16 +524,16 @@ void capture_backbuffer_d3d10(IDXGISwapChain* sc, ID3D10Texture2D* backbuf)
}
if (g_d3d10_staging != nullptr && ensure_aux_d3d11_locked() &&
ensure_shared_texture_locked(g_aux_d3d11, bd.Width, bd.Height, bd.Format) && g_shared_mutex != nullptr)
g_shared.ensure(g_aux_d3d11, bd.Width, bd.Height, bd.Format, g_pid, "present", kShareBind))
{
gdev->CopyResource(g_d3d10_staging, backbuf);
D3D10_MAPPED_TEXTURE2D m{};
if (SUCCEEDED(g_d3d10_staging->Map(0, D3D10_MAP_READ, 0, &m)) && m.pData != nullptr)
{
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK)
if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{
g_aux_ctx->UpdateSubresource(g_shared_tex, 0, nullptr, m.pData, m.RowPitch, 0);
g_shared_mutex->ReleaseSync(kVideoMutexKey);
g_aux_ctx->UpdateSubresource(g_shared.texture(), 0, nullptr, m.pData, m.RowPitch, 0);
g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true;
}
else
@@ -698,7 +599,7 @@ void capture_backbuffer(IDXGISwapChain* sc)
if (device != nullptr && ctx != nullptr)
{
std::scoped_lock lock(g_tex_mutex);
if (ensure_shared_texture_locked(device, bd.Width, bd.Height, bd.Format) && g_shared_mutex != nullptr)
if (g_shared.ensure(device, bd.Width, bd.Height, bd.Format, g_pid, "present", kShareBind))
{
if (first_capture_from(sc))
{
@@ -708,10 +609,10 @@ void capture_backbuffer(IDXGISwapChain* sc)
// Key 0 on both sides: a plain cross-process mutex on the texture (created
// released at key 0). Bounded wait so a stalled host consumer can never hang
// the game's render thread.
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK)
if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{
ctx->CopyResource(g_shared_tex, backbuf);
g_shared_mutex->ReleaseSync(kVideoMutexKey);
ctx->CopyResource(g_shared.texture(), backbuf);
g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true;
}
else
@@ -722,7 +623,6 @@ void capture_backbuffer(IDXGISwapChain* sc)
else
{
cant_host = true; // device can't host the shared texture -> try the D3D10 path
release_shared_locked();
}
}
if (ctx != nullptr)
@@ -819,33 +719,39 @@ void* grab_execute_command_lists_address()
return addr;
}
HRESULT STDMETHODCALLTYPE hk_Present(IDXGISwapChain* sc, UINT sync_interval, UINT flags)
// Shared body of the Present/Present1 detours: count the call, log each distinct
// (swapchain, flags) once, and capture the backbuffer. A DXGI_PRESENT_TEST present draws
// nothing (it only probes occlusion), so it inflates the Present count without producing a
// frame -- which is why some games show more presents than captured frames -- and is not
// worth copying.
void on_present(IDXGISwapChain* sc, UINT flags, int hook_id, const char* method)
{
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
hook_note_call(g_id_present);
hook_note_call(hook_id);
g_present_calls.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
{
g_ipc->note_present();
}
// Log each distinct (swapchain, flags) once. A DXGI_PRESENT_TEST present draws nothing
// (it only probes occlusion), so those inflate the Present count without producing a
// frame -- which is why some games show more presents than captured frames.
if (first_present_with_flags(sc, flags))
{
logf("present: swapchain=%p Present flags=0x%08X%s", sc, flags,
logf("present: swapchain=%p %s flags=0x%08X%s", sc, method, flags,
(flags & DXGI_PRESENT_TEST) ? " (DXGI_PRESENT_TEST: occlusion probe, no frame drawn)" : "");
}
// DXGI_PRESENT_TEST presents nothing; don't bother copying for it.
if ((flags & DXGI_PRESENT_TEST) == 0)
{
capture_backbuffer(sc);
}
}
HRESULT STDMETHODCALLTYPE hk_Present(IDXGISwapChain* sc, UINT sync_interval, UINT flags)
{
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
on_present(sc, flags, g_id_present, "Present");
// stdcall(), NOT call(): IDXGISwapChain::Present is __stdcall, but SafetyHook's
// call() invokes the trampoline through a __cdecl pointer (the default on x86).
// On 32-bit that double-cleans the stack -> ESP imbalance -> Run-Time Check
// Failure #0 and an instant crash. On x64 the conventions collapse, so it only
// bit 32-bit games (e.g. Slaps and Beans froze the moment it presented).
// bites 32-bit games.
return g_hk_present.stdcall<HRESULT>(sc, sync_interval, flags);
}
@@ -853,21 +759,7 @@ HRESULT STDMETHODCALLTYPE hk_Present1(IDXGISwapChain1* sc, UINT sync_interval, U
const DXGI_PRESENT_PARAMETERS* params)
{
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
hook_note_call(g_id_present1);
g_present_calls.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
{
g_ipc->note_present();
}
if (first_present_with_flags(sc, flags)) // see hk_Present
{
logf("present: swapchain=%p Present1 flags=0x%08X%s", sc, flags,
(flags & DXGI_PRESENT_TEST) ? " (DXGI_PRESENT_TEST: occlusion probe, no frame drawn)" : "");
}
if ((flags & DXGI_PRESENT_TEST) == 0)
{
capture_backbuffer(sc); // IDXGISwapChain1 derives from IDXGISwapChain
}
on_present(sc, flags, g_id_present1, "Present1"); // IDXGISwapChain1 derives from IDXGISwapChain
return g_hk_present1.stdcall<HRESULT>(sc, sync_interval, flags, params); // __stdcall, see hk_Present
}
@@ -995,12 +887,11 @@ bool install_present_hooks(IpcClient& ipc)
void remove_present_hooks()
{
// DISABLE (persistent model -- never destroy during the session): this restores the
// Present/Present1/ECL bytes under thread suspension so no NEW detour starts, while KEEPING the
// trampolines alive -- an in-flight detour about to call g_hk_present.stdcall() (the trampoline)
// must never have it freed under it (destroying = {} would, an AV the thousands/s storm hits
// reliably). So: disable -> drain (in-flight detours finish on the live trampoline) -> leave the
// hooks alive (re-install re-enables them; see hook_install.hpp).
// DISABLE, never destroy (persistent-hook model, see hook_install.hpp): this restores the
// Present/Present1/ECL bytes so no NEW detour starts, while KEEPING the trampolines alive -- an
// in-flight detour about to call g_hk_present.stdcall() (the trampoline) must never have it
// freed under it. So: disable -> drain (in-flight detours finish on the live trampoline) ->
// leave the hooks alive for a re-install to re-enable.
disable_for_removal(g_hk_present);
disable_for_removal(g_hk_present1);
disable_for_removal(g_hk_ecl);
@@ -1008,14 +899,12 @@ void remove_present_hooks()
hook_set_installed(g_id_present1, false);
hook_set_installed(g_id_ecl, false);
g_gate.drain();
// Persistent hooks: keep g_hk_present/present1/ecl ALIVE (disabled), so the trampoline a stale
// detour may still call is never freed -- re-install re-enables them (see hook_install.hpp).
g_present_queue.store(nullptr, std::memory_order_relaxed);
g_logged_presents_n = 0; // let a fresh injection re-log the present pattern
g_logged_swapchains_n = 0;
{
std::scoped_lock lock(g_tex_mutex);
release_shared_locked();
g_shared.release();
release_on12_locked();
release_aux_locked();
}

7
hook/src/present_hook.hpp
View File

@@ -5,9 +5,10 @@
// stable alternative to Windows Graphics Capture (which composites off the
// desktop and can stutter / show a capture border).
//
// Only DXGI swapchains (D3D10/11/12-backed games) are caught; the backbuffer
// must be an ID3D11Texture2D (the common D3D11 case). Games on D3D9 or a pure
// D3D12 resource path won't engage this hook -- WGC stays as the fallback.
// Only DXGI swapchains (D3D10/11/12-backed games) are caught: D3D11 backbuffers
// copy directly, D3D12 goes through a D3D11On12 bridge, and D3D10 reads back
// through the game's own device. D3D9 / OpenGL / Vulkan games have their own
// hooks (d3d9_hook, opengl_hook, vk_hook); WGC stays as the capture fallback.
#pragma once
#include "ipc_client.hpp"

10
hook/src/rate_estimator.hpp
View File

@@ -2,13 +2,13 @@
//
// When we attach to an already-running game we never saw its IAudioClient::Initialize,
// so we assume the device mix format and recover the *true* sample rate by timing how
// fast the game renders frames. The naive version (one short ~200 ms window, snap to the
// nearest standard rate, accept whatever came out) is fragile: WASAPI delivers audio in
// fast the game renders frames. A naive estimator (one short ~200 ms window, snap to the
// nearest standard rate, accept whatever comes out) is fragile: WASAPI delivers audio in
// quantized ~10 ms buffers, so one extra buffer at a window edge is a ~5% error over
// 200 ms, which lands *between* standard rates (they're >8% apart) and used to be
// published verbatim -- e.g. 44100 measured as ~46205.
// 200 ms, which lands *between* standard rates (they're >8% apart) -- e.g. 44100
// measuring as ~46205.
//
// This estimator fixes that with three rules:
// Three rules make it robust:
// 1. Longer windows (~0.5 s) -> the per-buffer quantization error drops to ~2%.
// 2. Reject a window that doesn't snap to a standard rate. Standard rates are far
// enough apart that any error big enough to miss the right one lands in no-man's-

135
hook/src/shared_video_texture.hpp Normal file
View File

@@ -0,0 +1,135 @@
// The keyed-mutex D3D11 texture a video backend publishes captured frames into. Created on
// whichever D3D11 device the backend has (the game's, an On12 bridge, or a hook-owned aux
// device), named coop_video_<pid> (video_share_name) so the host opens it by name, and
// synchronized with an IDXGIKeyedMutex at key 0 on both sides (kVideoMutexKey). Shared by the
// DXGI/D3D10/D3D12 Present hook, the OpenGL and D3D9 hooks, and the Vulkan capture.
#pragma once
#include <windows.h>
#include <d3d11.h>
#include <dxgi1_2.h>
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
namespace coop::hook
{
class SharedVideoTexture
{
public:
SharedVideoTexture() = default;
~SharedVideoTexture()
{
release();
}
SharedVideoTexture(const SharedVideoTexture&) = delete;
SharedVideoTexture& operator=(const SharedVideoTexture&) = delete;
// (Re)create the shared texture for a w x h `fmt` frame on `device`; cheap no-op when the
// current texture already matches. `tag` prefixes the log lines with the owning backend.
// `bind` lets a backend request bind flags beyond SHADER_RESOURCE. Returns true when
// texture() and mutex() are ready; false (released) when the device can't host it.
bool ensure(ID3D11Device* device, UINT w, UINT h, DXGI_FORMAT fmt, unsigned long pid, const char* tag,
UINT bind = D3D11_BIND_SHADER_RESOURCE)
{
if (m_tex != nullptr && m_w == w && m_h == h && m_fmt == fmt)
{
return true;
}
release();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = fmt;
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = bind;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
HRESULT hr = device->CreateTexture2D(&desc, nullptr, &m_tex);
if (FAILED(hr) || m_tex == nullptr)
{
logf("%s: CreateTexture2D(shared) failed hr=0x%08lX (%ux%u fmt=%d)", tag,
static_cast<unsigned long>(hr), w, h, static_cast<int>(fmt));
release();
return false;
}
IDXGIResource1* res = nullptr;
hr = m_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res));
if (FAILED(hr) || res == nullptr)
{
logf("%s: QI IDXGIResource1 failed hr=0x%08lX", tag, static_cast<unsigned long>(hr));
release();
return false;
}
const std::wstring name = video_share_name(pid);
hr = res->CreateSharedHandle(nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE,
name.c_str(), &m_handle);
res->Release();
if (FAILED(hr) || m_handle == nullptr)
{
logf("%s: CreateSharedHandle failed hr=0x%08lX", tag, static_cast<unsigned long>(hr));
release();
return false;
}
hr = m_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&m_mutex));
if (FAILED(hr) || m_mutex == nullptr)
{
logf("%s: QI IDXGIKeyedMutex failed hr=0x%08lX", tag, static_cast<unsigned long>(hr));
release();
return false;
}
m_w = w;
m_h = h;
m_fmt = fmt;
logf("%s: shared texture ready %ux%u fmt=%d name=%ls", tag, w, h, static_cast<int>(fmt), name.c_str());
return true;
}
void release()
{
if (m_mutex != nullptr)
{
m_mutex->Release();
m_mutex = nullptr;
}
if (m_tex != nullptr)
{
m_tex->Release();
m_tex = nullptr;
}
if (m_handle != nullptr)
{
CloseHandle(m_handle);
m_handle = nullptr;
}
m_w = m_h = 0;
m_fmt = DXGI_FORMAT_UNKNOWN;
}
[[nodiscard]] ID3D11Texture2D* texture() const
{
return m_tex;
}
[[nodiscard]] IDXGIKeyedMutex* mutex() const
{
return m_mutex;
}
private:
ID3D11Texture2D* m_tex = nullptr;
IDXGIKeyedMutex* m_mutex = nullptr;
HANDLE m_handle = nullptr; // named NT handle backing the share; closed on release
UINT m_w = 0;
UINT m_h = 0;
DXGI_FORMAT m_fmt = DXGI_FORMAT_UNKNOWN;
};
} // namespace coop::hook

80
hook/src/vk_capture.cpp
View File

@@ -3,7 +3,6 @@
#include <cstring>
#include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
namespace coop::hook
{
@@ -15,7 +14,7 @@ VkCapture::~VkCapture()
// Prefer HOST_CACHED memory: the reaper reads every byte of this buffer back on the CPU, and an
// uncached / write-combined mapping (what a plain HOST_VISIBLE|HOST_COHERENT type usually is on a
// discrete GPU) makes that read run at PCIe latency -- the original ~370 ms/frame stall. Order of
// discrete GPU) makes that read run at PCIe latency -- hundreds of ms per frame. Order of
// preference: cached+coherent (fast read, no invalidate) > cached (fast read, needs invalidate) >
// coherent-only (the slow fallback, only if nothing cached is host-visible).
bool VkCapture::find_readback_memory(std::uint32_t type_bits, std::uint32_t& out_index, bool& out_coherent)
@@ -220,29 +219,9 @@ bool VkCapture::ensure_d3d()
m_d3d != nullptr;
}
void VkCapture::release_shared()
{
if (m_shared_mutex != nullptr)
{
m_shared_mutex->Release();
m_shared_mutex = nullptr;
}
if (m_shared_tex != nullptr)
{
m_shared_tex->Release();
m_shared_tex = nullptr;
}
if (m_shared_handle != nullptr)
{
CloseHandle(m_shared_handle);
m_shared_handle = nullptr;
}
m_share_w = m_share_h = 0;
}
void VkCapture::release_d3d()
{
release_shared();
m_shared.release();
if (m_d3d_ctx != nullptr)
{
m_d3d_ctx->Release();
@@ -255,49 +234,6 @@ void VkCapture::release_d3d()
}
}
bool VkCapture::ensure_shared_texture(UINT w, UINT h)
{
if (m_shared_tex != nullptr && m_share_w == w && m_share_h == h)
{
return true;
}
release_shared();
D3D11_TEXTURE2D_DESC d{};
d.Width = w;
d.Height = h;
d.MipLevels = 1;
d.ArraySize = 1;
d.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
d.SampleDesc.Count = 1;
d.Usage = D3D11_USAGE_DEFAULT;
d.BindFlags = D3D11_BIND_SHADER_RESOURCE;
d.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
if (FAILED(m_d3d->CreateTexture2D(&d, nullptr, &m_shared_tex)) || m_shared_tex == nullptr)
{
return false;
}
IDXGIResource1* res = nullptr;
if (FAILED(m_shared_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res))) ||
res == nullptr)
{
release_shared();
return false;
}
const std::wstring name = video_share_name(m_pid);
const HRESULT hr = res->CreateSharedHandle(
nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, name.c_str(), &m_shared_handle);
res->Release();
if (FAILED(hr) || m_shared_handle == nullptr ||
FAILED(m_shared_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&m_shared_mutex))))
{
release_shared();
return false;
}
m_share_w = w;
m_share_h = h;
return true;
}
// --- init / present / reaper -------------------------------------------------
void VkCapture::init(VkPhysicalDevice phys, VkDevice device, std::uint32_t queue_family, const Fns& fns,
unsigned long pid, std::function<void(std::uint32_t, std::uint32_t)> on_frame)
@@ -449,12 +385,12 @@ void VkCapture::reap_slot(Slot& s)
}
bool published = false;
if (ensure_d3d() && ensure_shared_texture(s.w, s.h) &&
m_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK)
if (ensure_d3d() && m_shared.ensure(m_d3d, s.w, s.h, DXGI_FORMAT_R8G8B8A8_UNORM, m_pid, "vk") &&
m_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{
m_d3d_ctx->UpdateSubresource(m_shared_tex, 0, nullptr, m_rgba.data(), static_cast<UINT>(row), 0);
m_d3d_ctx->UpdateSubresource(m_shared.texture(), 0, nullptr, m_rgba.data(), static_cast<UINT>(row), 0);
m_d3d_ctx->Flush();
m_shared_mutex->ReleaseSync(kVideoMutexKey);
m_shared.mutex()->ReleaseSync(kVideoMutexKey);
published = true;
}
@@ -507,8 +443,8 @@ void VkCapture::shutdown()
m_q_cv.notify_all();
m_reaper.join();
}
// The reaper is gone (no more submits/reads); drain any GPU work still referencing our resources,
// then free. DeviceWaitIdle here mirrors what the inline hook / layer did before this component.
// The reaper is gone (no more submits/reads); drain any GPU work still referencing our
// resources, then free.
if (m_device != VK_NULL_HANDLE)
{
if (m_fns.DeviceWaitIdle != nullptr)

22
hook/src/vk_capture.hpp
View File

@@ -3,12 +3,12 @@
// how they get into the dispatch chain; the capture itself -- copy the presented image into a shared
// keyed-mutex D3D11 texture for the host to sample -- is identical, so it lives here once.
//
// Performance contract (the whole reason this is a separate component): the read-back must NOT stall
// the game's present thread. Against a real 144 FPS game the original inline version dropped it to
// ~3 FPS because it did the GPU copy + a synchronous CPU read of the mapped staging buffer + the
// swizzle + the D3D upload all on the present thread -- and the staging buffer was HOST_COHERENT (on
// a discrete GPU that's write-combined/uncached, where a scattered CPU read runs at PCIe latency:
// ~370 ms for one 1080p frame). This component fixes both halves:
// Performance contract (the whole reason this is a separate component): the read-back must NOT
// stall the game's present thread. Doing the GPU copy + a synchronous CPU read of the mapped
// staging buffer + the swizzle + the D3D upload on the present thread drops a 144 FPS game to a
// few FPS -- especially when the staging buffer is plain HOST_COHERENT (on a discrete GPU that's
// write-combined/uncached, where a scattered CPU read runs at PCIe latency: hundreds of ms for
// one 1080p frame). This component avoids both halves:
// * the present thread only records + submits the copy (sub-millisecond) and returns immediately;
// * a dedicated reaper thread waits the copy's fence, reads the staging buffer, swizzles and
// uploads -- off the critical path; and
@@ -34,6 +34,8 @@
#define VK_NO_PROTOTYPES
#include <vulkan/vulkan.h>
#include "shared_video_texture.hpp"
namespace coop::hook
{
@@ -131,8 +133,6 @@ private:
bool ensure_d3d();
void release_d3d();
bool ensure_shared_texture(UINT w, UINT h);
void release_shared();
void reaper_main();
void reap_slot(Slot& s);
@@ -150,11 +150,7 @@ private:
// --- D3D11 shared texture (reaper thread only) ---
ID3D11Device* m_d3d = nullptr;
ID3D11DeviceContext* m_d3d_ctx = nullptr;
ID3D11Texture2D* m_shared_tex = nullptr;
IDXGIKeyedMutex* m_shared_mutex = nullptr;
HANDLE m_shared_handle = nullptr;
UINT m_share_w = 0;
UINT m_share_h = 0;
SharedVideoTexture m_shared;
std::vector<unsigned char> m_rgba; // reaper scratch (swizzled frame)
unsigned long m_pid = 0;
std::function<void(std::uint32_t, std::uint32_t)> m_on_frame;

5
hook/src/vk_hook.cpp
View File

@@ -319,9 +319,8 @@ VKAPI_ATTR PFN_vkVoidFunction VKAPI_CALL hk_vkGetInstanceProcAddr(VkInstance ins
// vkGetInstanceProcAddr can also resolve device-level functions (the loader returns a
// dispatch trampoline). A game -- or volk's volkLoadInstance -- that resolves the present /
// swapchain entry points this way (rather than via vkGetDeviceProcAddr) would otherwise get
// the real loader pointer and bypass our capture. Sphere Spectacle does exactly this, so
// intercept them here too. (Our detours gate on g_capture_enabled / g_device, so handing them
// out before the device exists is safe.)
// the real loader pointer and bypass our capture, so intercept them here too. (Our detours
// gate on g_capture_enabled / g_device, so handing them out before the device exists is safe.)
if (std::strcmp(name, "vkQueuePresentKHR") == 0)
{
return reinterpret_cast<PFN_vkVoidFunction>(&hk_vkQueuePresentKHR);

11
hook/src/vtable_hook.hpp
View File

@@ -5,8 +5,8 @@
// mishandles -> the original runs with garbage args and faults. Swapping the vtable entry leaves
// the original code untouched, so it runs with a pristine stack regardless of prologue shape.
// Every instance of a COM coclass shares one vtable, so a single swap intercepts all of them.
// See the project's stdcall-x86 note. Shared by the audio render-hook (audio_hook.cpp) and the
// input-side DirectInput hook (mkb_hook.cpp).
// Shared by the audio render-hook (audio_hook.cpp) and the input-side DirectInput hook
// (mkb_hook.cpp).
#pragma once
#include <windows.h>
@@ -14,6 +14,13 @@
namespace coop::hook
{
// Read a COM object's vtable slot (e.g. to grab a method's address off a probe object for an
// inline hook).
inline void* vtable_method(void* obj, unsigned index)
{
return (*reinterpret_cast<void***>(obj))[index];
}
class VtableHook
{
public:

18
hook/src/xinput_hook.cpp
View File

@@ -167,6 +167,7 @@ DWORD WINAPI hk_XInputSetState(DWORD user_index, XINPUT_VIBRATION* vibration)
return ERROR_SUCCESS;
}
// `name` is a GetProcAddress LPCSTR: an export name, or MAKEINTRESOURCEA(ordinal).
void hook_export(HMODULE module, const char* name, void* detour, int registry_id)
{
if (module == nullptr)
@@ -181,20 +182,6 @@ void hook_export(HMODULE module, const char* name, void* detour, int registry_id
}
}
void hook_ordinal(HMODULE module, WORD ordinal, void* detour, int registry_id)
{
if (module == nullptr)
{
return;
}
if (void* target = reinterpret_cast<void*>(GetProcAddress(module, MAKEINTRESOURCEA(ordinal))))
{
g_hooks.emplace_back();
install_inline(g_hooks.back(), target, detour); // assign-then-enable (no install race)
hook_set_installed(registry_id, true);
}
}
} // namespace
bool install_xinput_hooks(IpcClient& ipc)
@@ -222,7 +209,8 @@ bool install_xinput_hooks(IpcClient& ipc)
continue;
}
hook_export(module, "XInputGetState", reinterpret_cast<void*>(&hk_XInputGetState), g_id_getstate);
hook_ordinal(module, 100, reinterpret_cast<void*>(&hk_XInputGetStateEx), g_id_getstateex);
hook_export(module, MAKEINTRESOURCEA(100), reinterpret_cast<void*>(&hk_XInputGetStateEx),
g_id_getstateex); // XInputGetStateEx is exported by ordinal only
hook_export(module, "XInputGetCapabilities", reinterpret_cast<void*>(&hk_XInputGetCapabilities),
g_id_getcaps);
hook_export(module, "XInputSetState", reinterpret_cast<void*>(&hk_XInputSetState), g_id_setstate);