BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Deduplicate the hook DLL and scrub history from its comments

Browse Source 
Consolidate four copies of the keyed-mutex shared-texture setup
(present/opengl/d3d9/vk_capture) into one RAII SharedVideoTexture,
two copies of find_main_window into find_window.hpp, audio_hook's
hand-rolled detour guard into the shared DetourGate, the duplicated
vtable_method into vtable_hook.hpp, and the near-identical
Present/Present1 and SwapBuffers/wglSwapBuffers detour pairs into one
shared body each. The vk_layer and vk_capture_perf_test targets now
compile debug_log.cpp since the shared texture code logs.

Comments no longer narrate the past: drop stress-test/game anecdotes,
"used to"/"the old model" phrasing, plan-step labels, and pointers to
docs that do not exist; fix present_hook.hpp/opengl_hook.hpp claims
that predate the D3D12/D3D9/Vulkan backends. Net -266 lines, no
behavior change (full x64 + x86 suites pass, including the mock-game
hook/unhook storm).
This commit is contained in:
BlackMark
2026-07-12 08:53:58 +02:00
parent 05039ab104
commit 635ef51283
22 changed files with 358 additions and 624 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
hook/src/audio_hook.cpp
View File

@@ -12,6 +12,7 @@
#include <mmreg.h> #include <mmreg.h>
#include "debug_log.hpp" #include "debug_log.hpp"
#include "hook_guard.hpp"
#include "hook_registry.hpp" #include "hook_registry.hpp"
#include "rate_estimator.hpp" #include "rate_estimator.hpp"
#include "vtable_hook.hpp" #include "vtable_hook.hpp"
@@ -45,8 +46,8 @@ using ReleaseBufferFn = HRESULT(STDMETHODCALLTYPE*)(IAudioRenderClient*, UINT32,
// The WASAPI COM methods are hooked by SWAPPING their vtable slot -- coop::hook::VtableHook from the // The WASAPI COM methods are hooked by SWAPPING their vtable slot -- coop::hook::VtableHook from the
// shared vtable_hook.hpp -- rather than inline-hooking: on x86 the MMDevApi/AudioSes prologues do // shared vtable_hook.hpp -- rather than inline-hooking: on x86 the MMDevApi/AudioSes prologues do
// dynamic stack alignment (`and esp,-8`) with EBP-relative args that SafetyHook's trampoline // dynamic stack alignment (`and esp,-8`) with EBP-relative args that SafetyHook's trampoline
// relocation mishandles (it froze 32-bit FMOD games the instant audio init ran through the hook). // relocation mishandles (a 32-bit game freezes the instant audio init runs through such a hook).
// Swapping the slot leaves the original code untouched. See the project's stdcall-x86 note. // Swapping the slot leaves the original code untouched.
// The scalar audio format we forward; resolved from the game's WAVEFORMATEX. // The scalar audio format we forward; resolved from the game's WAVEFORMATEX.
struct CapturedFormat struct CapturedFormat
@@ -61,9 +62,9 @@ struct CapturedFormat
// --- Global hook state ----------------------------------------------------- // --- Global hook state -----------------------------------------------------
// Atomic: the audio hot path (hk_ReleaseBuffer) reads it without the setup lock, while // Atomic: the audio hot path (hk_ReleaseBuffer) reads it without the setup lock, while
// remove_audio_hooks nulls it under the lock during an unhook. A plain pointer was a // remove_audio_hooks nulls it under the lock during an unhook. A plain pointer would be a
// TOCTOU null-deref (check non-null, then it's nulled, then the call) -- a rapid // TOCTOU null-deref under a rapid hook/unhook (check non-null, then it's nulled, then the
// hook/unhook crash the mock-game stress test caught. Load it once and use that. // call). Load it once and use that.
std::atomic<IpcClient*> g_ipc{nullptr}; std::atomic<IpcClient*> g_ipc{nullptr};
// One ring per tracked stream (index = the stream's debug slot). Stream 0 is the // One ring per tracked stream (index = the stream's debug slot). Stream 0 is the
// primary; the host creates a ring per stream and mixes them. // primary; the host creates a ring per stream and mixes them.
@@ -80,26 +81,12 @@ bool g_audioclient_hooked = false;
// Bumped on every detour install/remove. hk_GetBuffer stamps the current epoch into a // Bumped on every detour install/remove. hk_GetBuffer stamps the current epoch into a
// thread-local; hk_ReleaseBuffer captures only if the epoch still matches -- so a // thread-local; hk_ReleaseBuffer captures only if the epoch still matches -- so a
// GetBuffer/ReleaseBuffer pair that straddles a hook toggle (the stashed buffer pointer is // GetBuffer/ReleaseBuffer pair that straddles a hook toggle (the stashed buffer pointer is
// then stale) is skipped instead of memset-ing a freed buffer (a rapid hook/unhook crash // then stale) is skipped instead of memset-ing a freed buffer.
// the mock-game stress test caught).
std::atomic<std::uint32_t> g_hook_epoch{0}; std::atomic<std::uint32_t> g_hook_epoch{0};
// Number of detours (hk_GetBuffer/hk_ReleaseBuffer) currently executing on the audio // Drains in-flight GetBuffer/ReleaseBuffer detours: remove_audio_hooks restores the vtable
// thread. remove_audio_hooks restores the vtable slots (so no NEW detour starts) and then // slots (so no NEW detour starts) and then drains before tearing down the shared state a
// waits for this to drain to 0 before tearing down shared state -- guaranteeing no detour // mid-flight detour reads (see hook_guard.hpp).
// is mid-flight when state is cleared. The classic safe-unhook race; the alternative was an DetourGate g_gate;
// intermittent access violation in the game during a hook/unhook (the stress test caught it).
std::atomic<int> g_detours_active{0};
struct DetourGuard
{
DetourGuard()
{
g_detours_active.fetch_add(1, std::memory_order_acq_rel);
}
~DetourGuard()
{
g_detours_active.fetch_sub(1, std::memory_order_acq_rel);
}
};
// Registry ids for the hook list. // Registry ids for the hook list.
int g_id_activate = -1; int g_id_activate = -1;
@@ -118,8 +105,8 @@ IAudioClient* g_self_client = nullptr;
std::atomic<IAudioRenderClient*> g_self_render{nullptr}; std::atomic<IAudioRenderClient*> g_self_render{nullptr};
// The probe objects above are built ONCE and kept for the DLL's lifetime; an audio // The probe objects above are built ONCE and kept for the DLL's lifetime; an audio
// enable/disable toggle then only swaps vtable slots, never creates/destroys COM objects. // enable/disable toggle then only swaps vtable slots, never creates/destroys COM objects.
// Rapid create/destroy raced AudioSes and crashed the game (the mock-game stress test // Rapid create/destroy races AudioSes internals and crashes the game. Released only on
// caught this). Released only on detach (shutdown_audio_hooks). // detach (shutdown_audio_hooks).
// Device mix format, captured from our probe client. Used as the assumed format // Device mix format, captured from our probe client. Used as the assumed format
// for a stream we discover on the hot path (we never saw its Initialize, so we // for a stream we discover on the hot path (we never saw its Initialize, so we
@@ -172,9 +159,8 @@ RateEstimator g_rate_estimator[kMaxAudioStreams] = {};
// too large and the capture copy would over-read the game's buffer. We can't detect the true // too large and the capture copy would over-read the game's buffer. We can't detect the true
// layout (AUTOCONVERTPCM hands back a fixed staging buffer, so there's no buffer-stride to // layout (AUTOCONVERTPCM hands back a fixed staging buffer, so there's no buffer-stride to
// measure, and WASAPI exposes no API for a pre-existing client's format), so we instead clamp // measure, and WASAPI exposes no API for a pre-existing client's format), so we instead clamp
// every guessed-stream copy to the source buffer's committed region (copy_bound_locked / // every guessed-stream copy to the source buffer's committed region (readable_bytes) -- the
// readable_bytes) -- the audio may be misinterpreted, but it can never read past the // audio may be misinterpreted, but it can never read past the allocation.
// allocation. See README "Lessons learned".
// Per-stream AudioFormatState (how its format was determined), mirrored to the host UI. // Per-stream AudioFormatState (how its format was determined), mirrored to the host UI.
std::uint32_t g_stream_format_state[kMaxAudioStreams] = {}; std::uint32_t g_stream_format_state[kMaxAudioStreams] = {};
@@ -244,7 +230,7 @@ std::uint32_t readable_bytes(const void* ptr, std::uint32_t want)
HRESULT STDMETHODCALLTYPE hk_GetBuffer(IAudioRenderClient* self, UINT32 num_frames, BYTE** data) HRESULT STDMETHODCALLTYPE hk_GetBuffer(IAudioRenderClient* self, UINT32 num_frames, BYTE** data)
{ {
DetourGuard guard; // counts this detour as in-flight (drained before an unhook tears down) DetourGate::Guard guard(g_gate); // in-flight until return (drained before an unhook tears down)
hook_note_call(g_id_getbuffer); hook_note_call(g_id_getbuffer);
const HRESULT hr = g_vh_getbuffer.original<GetBufferFn>()(self, num_frames, data); const HRESULT hr = g_vh_getbuffer.original<GetBufferFn>()(self, num_frames, data);
if (SUCCEEDED(hr) && data != nullptr) if (SUCCEEDED(hr) && data != nullptr)
@@ -259,7 +245,7 @@ HRESULT STDMETHODCALLTYPE hk_GetBuffer(IAudioRenderClient* self, UINT32 num_fram
HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_frames, DWORD flags) HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_frames, DWORD flags)
{ {
DetourGuard guard; // counts this detour as in-flight (drained before an unhook tears down) DetourGate::Guard guard(g_gate); // in-flight until return (drained before an unhook tears down)
hook_note_call(g_id_releasebuffer); hook_note_call(g_id_releasebuffer);
// A render client we've never seen actively rendering is almost certainly one // A render client we've never seen actively rendering is almost certainly one
// the game created before we injected; adopt it now (the first becomes the // the game created before we injected; adopt it now (the first becomes the
@@ -314,16 +300,13 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_
g_frames_captured.fetch_add(num_frames, std::memory_order_relaxed); g_frames_captured.fetch_add(num_frames, std::memory_order_relaxed);
// Mute the game's local playback so the only audio is the host's re-render. // Mute the game's local playback so the only audio is the host's re-render.
// Otherwise the game plays locally AND the mirror re-renders the same audio a // Otherwise the game plays locally AND the mirror re-renders the same audio a
// few ms later = a metallic double (the Brotato symptom). AUDCLNT_BUFFERFLAGS_SILENT // few ms later = a metallic double. AUDCLNT_BUFFERFLAGS_SILENT tells WASAPI to
// tells WASAPI to treat the buffer as silence and IGNORE its contents, so it // treat the buffer as silence and IGNORE its contents, so it mutes WITHOUT
// mutes WITHOUT writing the buffer -- safe even for a guessed-format stream whose // writing the buffer -- safe even for a guessed-format stream whose true frame
// true frame size we don't know. (Muting used to be tied to the memset below, // size we don't know; the flag is the mute, not the memset. For an exact/override
// which is unsafe for a guessed block, so guessed streams -- the late-attach / // format we additionally zero the buffer (belt-and-suspenders; `block` is the
// Brotato case -- were captured but left audible. The flag is the actual mute; // real frame size there, so it stays in-bounds). Only mutes once the frames made
// the memset is not needed for it.) For an exact/override format we additionally // the ring (above) -- a stalled host degrades to echo, never to dead silence.
// zero the buffer (belt-and-suspenders; `block` is the real frame size there, so
// it stays in-bounds). Only mutes once the frames made the ring (above) -- a
// stalled host degrades to echo, never to dead silence.
if (!guessed) if (!guessed)
{ {
std::memset(t_gb_data, 0, bytes); std::memset(t_gb_data, 0, bytes);
@@ -334,7 +317,7 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_
} }
} }
} }
// Format-verification co-capture (host-driven, step 2a). While the host has set // Format-verification co-capture (host-driven). While the host has set
// verify_capture and this guessed stream's rate is still being MEASURED (format not yet // verify_capture and this guessed stream's rate is still being MEASURED (format not yet
// published), push the raw pre-mix bytes to the ring WITHOUT silencing, so the host can // published), push the raw pre-mix bytes to the ring WITHOUT silencing, so the host can
// capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same // capture both the hook (pre-mix) and a parallel process-loopback (post-mix) of the same
@@ -691,7 +674,7 @@ namespace
// Build the probe COM objects (enumerator -> device -> client -> render) and capture the // Build the probe COM objects (enumerator -> device -> client -> render) and capture the
// device mix format. Created ONCE and kept for the DLL's lifetime: every instance of a // device mix format. Created ONCE and kept for the DLL's lifetime: every instance of a
// coclass shares one vtable, so a toggle then only re-swaps vtable slots on these kept // coclass shares one vtable, so a toggle then only re-swaps vtable slots on these kept
// objects -- no COM create/destroy churn (which raced AudioSes). Caller holds g_setup_mutex. // objects -- no COM create/destroy churn (which races AudioSes). Caller holds g_setup_mutex.
bool build_probe_locked() bool build_probe_locked()
{ {
if (g_self_device != nullptr) if (g_self_device != nullptr)
@@ -868,17 +851,8 @@ void remove_audio_hooks()
hook_set_installed(g_id_releasebuffer, false); hook_set_installed(g_id_releasebuffer, false);
// The slots are restored above, so no NEW detour will start. Drain any detour still // The slots are restored above, so no NEW detour will start. Drain any detour still
// in-flight on the audio thread before clearing the shared state it reads (an initial // in-flight on the audio thread before clearing the shared state it reads.
// sleep covers a caller that read the old slot but hasn't entered the guard yet). Bounded g_gate.drain();
// so a wedged audio thread can't hang us. Detours are microseconds, so this is ~1-2 ms.
for (int spins = 0; spins < 200; ++spins)
{
Sleep(1);
if (g_detours_active.load(std::memory_order_acquire) == 0)
{
break;
}
}
// Clear per-stream tracking (the game's streams re-register lazily on a re-enable). // Clear per-stream tracking (the game's streams re-register lazily on a re-enable).
g_registered = 0; g_registered = 0;

2
hook/src/audio_hook.hpp
View File

@@ -1,7 +1,7 @@
// Injected WASAPI render-hook: captures the game's audio render frames into the // Injected WASAPI render-hook: captures the game's audio render frames into the
// shared audio ring and silences the game's local playback, so the host can // shared audio ring and silences the game's local playback, so the host can
// re-render the audio for Steam Remote Play Together without the operator // re-render the audio for Steam Remote Play Together without the operator
// hearing it twice (the "local audio echo"). See docs/audio-render-hook-plan.md. // hearing it twice (the "local audio echo").
// //
// The hooks always install (so render streams are counted for the debug view // The hooks always install (so render streams are counted for the debug view
// even with mirroring off); the copy+silence behavior is gated by the ring's // even with mirroring off); the copy+silence behavior is gated by the ring's

98
hook/src/d3d9_hook.cpp
View File

@@ -12,11 +12,12 @@
#include <safetyhook.hpp> #include <safetyhook.hpp>
#include "coop/protocol.hpp" #include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "debug_log.hpp" #include "debug_log.hpp"
#include "hook_guard.hpp" #include "hook_guard.hpp"
#include "hook_install.hpp" #include "hook_install.hpp"
#include "hook_registry.hpp" #include "hook_registry.hpp"
#include "shared_video_texture.hpp"
#include "vtable_hook.hpp"
namespace coop::hook namespace coop::hook
{ {
@@ -48,11 +49,7 @@ bool g_unsupported_logged = false;
// Our own D3D11 device hosting the shared texture (the D3D9 game has no D3D11 device). // Our own D3D11 device hosting the shared texture (the D3D9 game has no D3D11 device).
ID3D11Device* g_device = nullptr; ID3D11Device* g_device = nullptr;
ID3D11DeviceContext* g_ctx = nullptr; ID3D11DeviceContext* g_ctx = nullptr;
ID3D11Texture2D* g_shared_tex = nullptr; SharedVideoTexture g_shared;
IDXGIKeyedMutex* g_shared_mutex = nullptr;
HANDLE g_shared_handle = nullptr;
UINT g_share_w = 0;
UINT g_share_h = 0;
// System-memory read-back surface on the game's D3D9 device (GetRenderTargetData target). // System-memory read-back surface on the game's D3D9 device (GetRenderTargetData target).
IDirect3DSurface9* g_sysmem = nullptr; IDirect3DSurface9* g_sysmem = nullptr;
@@ -66,11 +63,6 @@ std::vector<unsigned char> g_rgba; // swizzled RGBA, uploaded to D3D11
// Present may be issued re-entrantly by some engines; capture only on the outermost call. // Present may be issued re-entrantly by some engines; capture only on the outermost call.
thread_local bool t_in_present = false; thread_local bool t_in_present = false;
void* vtable_method(void* obj, unsigned index)
{
return (*reinterpret_cast<void***>(obj))[index];
}
bool ensure_device() bool ensure_device()
{ {
if (g_device != nullptr) if (g_device != nullptr)
@@ -87,26 +79,6 @@ bool ensure_device()
return true; return true;
} }
void release_shared()
{
if (g_shared_mutex != nullptr)
{
g_shared_mutex->Release();
g_shared_mutex = nullptr;
}
if (g_shared_tex != nullptr)
{
g_shared_tex->Release();
g_shared_tex = nullptr;
}
if (g_shared_handle != nullptr)
{
CloseHandle(g_shared_handle);
g_shared_handle = nullptr;
}
g_share_w = g_share_h = 0;
}
void release_sysmem() void release_sysmem()
{ {
if (g_sysmem != nullptr) if (g_sysmem != nullptr)
@@ -123,56 +95,6 @@ void release_sysmem()
g_sysmem_fmt = D3DFMT_UNKNOWN; g_sysmem_fmt = D3DFMT_UNKNOWN;
} }
bool ensure_shared_texture(UINT w, UINT h)
{
if (g_shared_tex != nullptr && g_share_w == w && g_share_h == h)
{
return true;
}
release_shared();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM; // we swizzle the D3D9 BGRA backbuffer to RGBA
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
if (FAILED(g_device->CreateTexture2D(&desc, nullptr, &g_shared_tex)) || g_shared_tex == nullptr)
{
return false;
}
IDXGIResource1* res = nullptr;
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res))) ||
res == nullptr)
{
release_shared();
return false;
}
const std::wstring name = video_share_name(g_pid);
const HRESULT hr = res->CreateSharedHandle(
nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, name.c_str(), &g_shared_handle);
res->Release();
if (FAILED(hr) || g_shared_handle == nullptr)
{
release_shared();
return false;
}
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&g_shared_mutex))))
{
release_shared();
return false;
}
g_share_w = w;
g_share_h = h;
logf("d3d9: shared texture ready %ux%u name=%ls", w, h, name.c_str());
return true;
}
// Read the game's D3D9 backbuffer back to system memory, swizzle BGRA->RGBA, and upload it. // Read the game's D3D9 backbuffer back to system memory, swizzle BGRA->RGBA, and upload it.
void capture_d3d9(IDirect3DDevice9* dev) void capture_d3d9(IDirect3DDevice9* dev)
{ {
@@ -239,12 +161,14 @@ void capture_d3d9(IDirect3DDevice9* dev)
} }
g_sysmem->UnlockRect(); g_sysmem->UnlockRect();
if (ensure_device() && ensure_shared_texture(w, h) && g_shared_mutex != nullptr && // DXGI_FORMAT_R8G8B8A8_UNORM: we swizzle the D3D9 BGRA backbuffer to RGBA above.
g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK) if (ensure_device() &&
g_shared.ensure(g_device, w, h, DXGI_FORMAT_R8G8B8A8_UNORM, g_pid, "d3d9") &&
g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{ {
g_ctx->UpdateSubresource(g_shared_tex, 0, nullptr, g_rgba.data(), static_cast<UINT>(dst_row), 0); g_ctx->UpdateSubresource(g_shared.texture(), 0, nullptr, g_rgba.data(), static_cast<UINT>(dst_row), 0);
g_ctx->Flush(); g_ctx->Flush();
g_shared_mutex->ReleaseSync(kVideoMutexKey); g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true; shared = true;
} }
} }
@@ -375,9 +299,7 @@ void remove_d3d9_hooks()
disable_for_removal(g_hk_present9); disable_for_removal(g_hk_present9);
hook_set_installed(g_id_present9, false); hook_set_installed(g_id_present9, false);
g_gate.drain(); g_gate.drain();
// Persistent hook: keep g_hk_present9 ALIVE (disabled) so a stale detour's trampoline call is g_shared.release();
// never freed -- re-install re-enables it (see hook_install.hpp).
release_shared();
release_sysmem(); release_sysmem();
if (g_ctx != nullptr) if (g_ctx != nullptr)
{ {

4
hook/src/dllmain.cpp
View File

@@ -230,8 +230,8 @@ DWORD WINAPI worker_thread(LPVOID)
g_ipc.set_vk_too_late(coop::hook::vk_injected_too_late()); // Vulkan attached-too-late banner g_ipc.set_vk_too_late(coop::hook::vk_injected_too_late()); // Vulkan attached-too-late banner
g_ipc.heartbeat(); g_ipc.heartbeat();
// Reconcile ~4x/s, but drain MKB events far more often (input must be // Reconcile ~4x/s (50 slices x 5 ms), but drain MKB events every slice --
// responsive). 50 slices x 5 ms ~= the old 250 ms reconcile period. // input must stay responsive at a far higher rate than the reconcile.
for (int slice = 0; slice < 50 && g_running.load(std::memory_order_relaxed); ++slice) for (int slice = 0; slice < 50 && g_running.load(std::memory_order_relaxed); ++slice)
{ {
if (mkb_installed) if (mkb_installed)

46
hook/src/find_window.hpp Normal file
View File

@@ -0,0 +1,46 @@
// Locate the game's main window from inside the game process: the largest
// visible, unowned top-level window the pid owns. Shared by the focus spoof
// (subclass target) and the MKB forwarder (PostMessage target).
#pragma once
#include <windows.h>
namespace coop::hook
{
inline HWND find_main_window(DWORD pid)
{
struct Ctx
{
DWORD pid;
HWND best;
long best_area;
} ctx{pid, nullptr, 0};
EnumWindows(
[](HWND hwnd, LPARAM lparam) -> BOOL {
auto* c = reinterpret_cast<Ctx*>(lparam);
DWORD pid = 0;
GetWindowThreadProcessId(hwnd, &pid);
if (pid != c->pid || !IsWindowVisible(hwnd) || GetWindow(hwnd, GW_OWNER) != nullptr)
{
return TRUE; // not ours, hidden, or an owned dialog -- keep looking
}
RECT rect = {};
if (!GetWindowRect(hwnd, &rect))
{
return TRUE;
}
const long area = (rect.right - rect.left) * (rect.bottom - rect.top);
if (area > c->best_area)
{
c->best_area = area;
c->best = hwnd;
}
return TRUE;
},
reinterpret_cast<LPARAM>(&ctx));
return ctx.best;
}
} // namespace coop::hook

43
hook/src/focus_spoof.cpp
View File

@@ -6,6 +6,7 @@
#include <safetyhook.hpp> #include <safetyhook.hpp>
#include "find_window.hpp"
#include "hook_guard.hpp" #include "hook_guard.hpp"
#include "hook_install.hpp" #include "hook_install.hpp"
#include "hook_registry.hpp" #include "hook_registry.hpp"
@@ -35,46 +36,6 @@ int g_id_setcursorpos = -1;
safetyhook::InlineHook g_hk_clipcursor; safetyhook::InlineHook g_hk_clipcursor;
safetyhook::InlineHook g_hk_setcursorpos; safetyhook::InlineHook g_hk_setcursorpos;
struct EnumContext
{
DWORD pid;
HWND best;
long best_area;
};
BOOL CALLBACK enum_proc(HWND hwnd, LPARAM lparam)
{
auto* ctx = reinterpret_cast<EnumContext*>(lparam);
DWORD pid = 0;
GetWindowThreadProcessId(hwnd, &pid);
if (pid != ctx->pid || !IsWindowVisible(hwnd) || GetWindow(hwnd, GW_OWNER) != nullptr)
{
return TRUE; // not ours, hidden, or an owned dialog -- keep looking
}
RECT rect = {};
if (!GetWindowRect(hwnd, &rect))
{
return TRUE;
}
const long area = (rect.right - rect.left) * (rect.bottom - rect.top);
if (area > ctx->best_area)
{
ctx->best_area = area;
ctx->best = hwnd;
}
return TRUE;
}
// The game's main window = the largest visible, unowned top-level window it owns.
HWND find_main_window(DWORD pid)
{
EnumContext ctx{pid, nullptr, 0};
EnumWindows(&enum_proc, reinterpret_cast<LPARAM>(&ctx));
return ctx.best;
}
// Replacement window procedure: convince the game it is never deactivated. // Replacement window procedure: convince the game it is never deactivated.
LRESULT CALLBACK subclass_proc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam) LRESULT CALLBACK subclass_proc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam)
{ {
@@ -313,7 +274,7 @@ void remove_focus_spoof()
// GetForegroundWindow is still hooked -- its detour returns g_game_hwnd and never reaches the // GetForegroundWindow is still hooked -- its detour returns g_game_hwnd and never reaches the
// patched bytes. The reverse of the enable order (GFW first) keeps the invariant "GetActiveWindow // patched bytes. The reverse of the enable order (GFW first) keeps the invariant "GetActiveWindow
// hooked => GetForegroundWindow hooked" across the whole install/remove cycle, so a call never // hooked => GetForegroundWindow hooked" across the whole install/remove cycle, so a call never
// lands in a half-patched shared region (the intermittent storm crash: GetActiveWindow+0x8). // lands in a half-patched shared region.
for (auto it = g_focus_hooks.rbegin(); it != g_focus_hooks.rend(); ++it) for (auto it = g_focus_hooks.rbegin(); it != g_focus_hooks.rend(); ++it)
{ {
disable_for_removal(*it); disable_for_removal(*it);

8
hook/src/hook_guard.hpp
View File

@@ -4,9 +4,9 @@
// The hazard: remove_*_hooks restores the hook and then frees the shared state the detour // The hazard: remove_*_hooks restores the hook and then frees the shared state the detour
// touches (D3D device/context, the keyed-mutex texture, Vulkan read-back resources, the IPC // touches (D3D device/context, the keyed-mutex texture, Vulkan read-back resources, the IPC
// pointer). A capture detour mid-flight on the game's render thread then uses freed memory -> // pointer). A capture detour mid-flight on the game's render thread then uses freed memory ->
// use-after-free -> the game crashes (the "spamming Mirror video crashed Brotato" bug). // use-after-free -> the game crashes.
// //
// The fix mirrors the audio hooks' epoch+drain pattern, generalised for inline hooks: // The removal sequence that prevents it:
// 1. Disable the hook FIRST so no NEW detour can start. For a SafetyHook inline hook that's // 1. Disable the hook FIRST so no NEW detour can start. For a SafetyHook inline hook that's
// `disable_for_removal(hook)` (disable, NOT `= {}` destroy): it restores the original bytes but // `disable_for_removal(hook)` (disable, NOT `= {}` destroy): it restores the original bytes but
// keeps the trampoline alive -- hooks are PERSISTENT, never destroyed mid-session, so an in-flight // keeps the trampoline alive -- hooks are PERSISTENT, never destroyed mid-session, so an in-flight
@@ -60,8 +60,8 @@ public:
// instruction prologue is unguarded), so m_active reads 0 even though a detour is about to run. // instruction prologue is unguarded), so m_active reads 0 even though a detour is about to run.
// Returning then would free the state out from under it. With the hook disabled no NEW detour can // Returning then would free the state out from under it. With the hook disabled no NEW detour can
// start, so any such thread reaches its Guard within nanoseconds; a 1 ms settle before concluding // start, so any such thread reaches its Guard within nanoseconds; a 1 ms settle before concluding
// "zero" lets it register. Without this, a backend presenting at thousands/s (the uncapped // "zero" lets it register. A backend presenting at thousands of frames/s hits this window
// mock-game storm) reliably crashed on remove (0xC0000005); with it, the count is accurate. // reliably, so checking before the first sleep is not safe.
void drain() void drain()
{ {
for (int spins = 0; spins < 400; ++spins) for (int spins = 0; spins < 400; ++spins)

12
hook/src/hook_install.hpp
View File

@@ -1,17 +1,17 @@
// Safe INSTALL of a SafetyHook inline hook -- the symmetric partner of hook_guard.hpp's safe removal. // Safe INSTALL of a SafetyHook inline hook -- the symmetric partner of hook_guard.hpp's safe removal.
// //
// Two problems this avoids, both surfaced by mock_game_test's hook/unhook storm (a game polling a // Two problems this avoids, both hit when a game polls a hooked API at thousands of calls/s while
// hooked API at thousands/s while the subsystem is toggled on/off every 60 ms): // the subsystem is toggled on/off (mock_game_test's hook/unhook storm exercises exactly that):
// //
// 1. Enable-before-assign. create_inline() builds the hook AND enables it (patches the target's bytes // 1. Enable-before-assign. create_inline() builds the hook AND enables it (patches the target's bytes
// to jump to the detour), THEN the result is move-assigned into the global the detour reads to // to jump to the detour), THEN the result is move-assigned into the global the detour reads to
// reach the trampoline. A call landing in the detour during that assign reads a torn global -> AV. // reach the trampoline. A call landing in the detour during that assign reads a torn global -> AV.
// So we create StartDisabled, let dst be populated, and only THEN enable(). // So we create StartDisabled, let dst be populated, and only THEN enable().
// //
// 2. Trampoline use-after-free on re-install. The old model recreated the hook on every install and // 2. Trampoline use-after-free on re-install. Recreating the hook on every install and destroying
// DESTROYED it on every remove (= {}), which frees the trampoline. A detour that has entered but // it on every remove (= {}) frees the trampoline. A detour that has entered but not yet reached
// not yet reached its DetourGate::Guard (the unguarded prologue) can then call a freed trampoline // its DetourGate::Guard (the unguarded prologue) can then call a freed trampoline -> AV at a
// -> AV at a garbage address. drain() narrows that window but can't fully close it under heavy // garbage address. drain() narrows that window but can't fully close it under heavy
// preemption. So instead we treat hooks as PERSISTENT: create each once and thereafter only // preemption. So instead we treat hooks as PERSISTENT: create each once and thereafter only
// enable()/disable() it across install/remove cycles. The trampoline is allocated once and never // enable()/disable() it across install/remove cycles. The trampoline is allocated once and never
// freed during the session, so a stale detour always calls a live trampoline (which, when // freed during the session, so a stale detour always calls a live trampoline (which, when

48
hook/src/mkb_hook.cpp
View File

@@ -9,6 +9,7 @@
#include <safetyhook.hpp> #include <safetyhook.hpp>
#include "find_window.hpp"
#include "hook_guard.hpp" #include "hook_guard.hpp"
#include "hook_install.hpp" #include "hook_install.hpp"
#include "hook_registry.hpp" #include "hook_registry.hpp"
@@ -278,40 +279,6 @@ void post_raw_mouse(HWND hwnd, USHORT button_flags)
// --- Target window + message synthesis -------------------------------------- // --- Target window + message synthesis --------------------------------------
struct FindCtx
{
DWORD pid;
HWND best;
long area;
};
BOOL CALLBACK find_main_proc(HWND hwnd, LPARAM lparam)
{
auto* c = reinterpret_cast<FindCtx*>(lparam);
DWORD pid = 0;
GetWindowThreadProcessId(hwnd, &pid);
if (pid != c->pid || !IsWindowVisible(hwnd) || GetWindow(hwnd, GW_OWNER) != nullptr)
{
return TRUE;
}
RECT r{};
GetWindowRect(hwnd, &r);
const long area = (r.right - r.left) * (r.bottom - r.top);
if (area > c->area)
{
c->area = area;
c->best = hwnd;
}
return TRUE;
}
HWND find_main_window()
{
FindCtx c{GetCurrentProcessId(), nullptr, 0};
EnumWindows(&find_main_proc, reinterpret_cast<LPARAM>(&c));
return c.best;
}
LPARAM key_lparam(UINT vk, bool key_up) LPARAM key_lparam(UINT vk, bool key_up)
{ {
const UINT scan = MapVirtualKeyW(vk, MAPVK_VK_TO_VSC); const UINT scan = MapVirtualKeyW(vk, MAPVK_VK_TO_VSC);
@@ -524,19 +491,16 @@ void remove_mkb_hooks()
} }
g_active.store(false, std::memory_order_release); g_active.store(false, std::memory_order_release);
// Disable (restore original bytes) the inline hooks so no new detour starts, KEEPING the // Disable (restore original bytes) the inline hooks so no new detour starts, KEEPING the
// trampolines alive for any in-flight detour calling its trampoline (persistent model -- never // trampolines alive for any in-flight detour calling its trampoline -- these detours DO call
// destroyed during the session; re-install re-enables, see hook_install.hpp). Destroying would // the trampoline, so destroying one would free it under a detour about to call it (persistent
// free a trampoline under a detour about to call it -- the present-storm UAF class. (The DI hook // model, re-enabled on re-install; see hook_install.hpp). The DI hook is a vtable swap:
// is a vtable swap: remove() restores the slot and keeps m_original valid, no trampoline either.) // remove() restores the slot and keeps m_original valid, no trampoline involved.
disable_for_removal(g_hk_async); disable_for_removal(g_hk_async);
disable_for_removal(g_hk_kbstate); disable_for_removal(g_hk_kbstate);
disable_for_removal(g_hk_cursor); disable_for_removal(g_hk_cursor);
disable_for_removal(g_hk_getrawinputdata); disable_for_removal(g_hk_getrawinputdata);
g_vh_di_getstate.remove(); // restore the DI GetDeviceState slot (probe kept alive for re-enable) g_vh_di_getstate.remove(); // restore the DI GetDeviceState slot (probe kept alive for re-enable)
g_gate.drain(); // wait for any in-flight polling / DI / raw detour before clearing state g_gate.drain(); // wait for any in-flight polling / DI / raw detour before clearing state
// Persistent hooks: keep g_hk_async/kbstate/cursor/getrawinputdata ALIVE (disabled), so a stale
// detour's trampoline call (these detours DO call the trampoline) is never freed -- re-install
// re-enables them (see hook_install.hpp).
hook_set_installed(g_id_di_getstate, false); hook_set_installed(g_id_di_getstate, false);
hook_set_installed(g_id_rawinput, false); hook_set_installed(g_id_rawinput, false);
for (int vk = 0; vk < 256; ++vk) for (int vk = 0; vk < 256; ++vk)
@@ -566,7 +530,7 @@ void mkb_pump(IpcClient& ipc)
HWND hwnd = static_cast<HWND>(g_target.load(std::memory_order_relaxed)); HWND hwnd = static_cast<HWND>(g_target.load(std::memory_order_relaxed));
if (hwnd == nullptr || !IsWindow(hwnd)) if (hwnd == nullptr || !IsWindow(hwnd))
{ {
hwnd = find_main_window(); hwnd = find_main_window(GetCurrentProcessId());
g_target.store(hwnd, std::memory_order_relaxed); g_target.store(hwnd, std::memory_order_relaxed);
} }

127
hook/src/opengl_hook.cpp
View File

@@ -11,11 +11,11 @@
#include <safetyhook.hpp> #include <safetyhook.hpp>
#include "coop/protocol.hpp" #include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
#include "debug_log.hpp" #include "debug_log.hpp"
#include "hook_guard.hpp" #include "hook_guard.hpp"
#include "hook_install.hpp" #include "hook_install.hpp"
#include "hook_registry.hpp" #include "hook_registry.hpp"
#include "shared_video_texture.hpp"
namespace coop::hook namespace coop::hook
{ {
@@ -54,11 +54,7 @@ bool g_unsupported_logged = false;
// Our own D3D11 device hosting the shared texture (the GL path has no D3D device). // Our own D3D11 device hosting the shared texture (the GL path has no D3D device).
ID3D11Device* g_device = nullptr; ID3D11Device* g_device = nullptr;
ID3D11DeviceContext* g_ctx = nullptr; ID3D11DeviceContext* g_ctx = nullptr;
ID3D11Texture2D* g_shared_tex = nullptr; SharedVideoTexture g_shared;
IDXGIKeyedMutex* g_shared_mutex = nullptr;
HANDLE g_shared_handle = nullptr;
UINT g_share_w = 0;
UINT g_share_h = 0;
std::vector<unsigned char> g_read_buf; // glReadPixels target (bottom-up) std::vector<unsigned char> g_read_buf; // glReadPixels target (bottom-up)
std::vector<unsigned char> g_flip_buf; // vertically flipped, uploaded to D3D std::vector<unsigned char> g_flip_buf; // vertically flipped, uploaded to D3D
@@ -100,76 +96,6 @@ bool ensure_device()
return true; return true;
} }
void release_shared()
{
if (g_shared_mutex != nullptr)
{
g_shared_mutex->Release();
g_shared_mutex = nullptr;
}
if (g_shared_tex != nullptr)
{
g_shared_tex->Release();
g_shared_tex = nullptr;
}
if (g_shared_handle != nullptr)
{
CloseHandle(g_shared_handle);
g_shared_handle = nullptr;
}
g_share_w = g_share_h = 0;
}
bool ensure_shared_texture(UINT w, UINT h)
{
if (g_shared_tex != nullptr && g_share_w == w && g_share_h == h)
{
return true;
}
release_shared();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM; // glReadPixels(GL_RGBA) byte order
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
if (FAILED(g_device->CreateTexture2D(&desc, nullptr, &g_shared_tex)) || g_shared_tex == nullptr)
{
return false;
}
IDXGIResource1* res = nullptr;
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res))) ||
res == nullptr)
{
release_shared();
return false;
}
const std::wstring name = video_share_name(g_pid);
const HRESULT hr = res->CreateSharedHandle(
nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, name.c_str(), &g_shared_handle);
res->Release();
if (FAILED(hr) || g_shared_handle == nullptr)
{
release_shared();
return false;
}
if (FAILED(g_shared_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&g_shared_mutex))))
{
release_shared();
return false;
}
g_share_w = w;
g_share_h = h;
logf("opengl: shared texture ready %ux%u name=%ls", w, h, name.c_str());
return true;
}
// Read the current GL backbuffer for the window behind `hdc` and upload it. // Read the current GL backbuffer for the window behind `hdc` and upload it.
void capture_gl(HDC hdc) void capture_gl(HDC hdc)
{ {
@@ -197,7 +123,8 @@ void capture_gl(HDC hdc)
return; return;
} }
if (!ensure_device() || !ensure_shared_texture(w, h)) // DXGI_FORMAT_R8G8B8A8_UNORM matches glReadPixels(GL_RGBA) byte order.
if (!ensure_device() || !g_shared.ensure(g_device, w, h, DXGI_FORMAT_R8G8B8A8_UNORM, g_pid, "opengl"))
{ {
return; return;
} }
@@ -223,11 +150,11 @@ void capture_gl(HDC hdc)
memcpy(g_flip_buf.data() + y * row, g_read_buf.data() + (h - 1 - y) * row, row); memcpy(g_flip_buf.data() + y * row, g_read_buf.data() + (h - 1 - y) * row, row);
} }
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK) if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{ {
g_ctx->UpdateSubresource(g_shared_tex, 0, nullptr, g_flip_buf.data(), static_cast<UINT>(row), 0); g_ctx->UpdateSubresource(g_shared.texture(), 0, nullptr, g_flip_buf.data(), static_cast<UINT>(row), 0);
g_ctx->Flush(); g_ctx->Flush();
g_shared_mutex->ReleaseSync(kVideoMutexKey); g_shared.mutex()->ReleaseSync(kVideoMutexKey);
g_frames_shared.fetch_add(1, std::memory_order_relaxed); g_frames_shared.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr) if (g_ipc != nullptr)
{ {
@@ -236,10 +163,11 @@ void capture_gl(HDC hdc)
} }
} }
BOOL WINAPI hk_SwapBuffers(HDC hdc) // Shared body of both swap detours: count the swap (SwapBuffers/wglSwapBuffers is the GL
// present), capture on the outermost call only, then forward through `hook`'s trampoline.
BOOL swap_detour(safetyhook::InlineHook& hook, int hook_id, HDC hdc)
{ {
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour hook_note_call(hook_id);
hook_note_call(g_id_swapbuffers);
g_swaps.fetch_add(1, std::memory_order_relaxed); g_swaps.fetch_add(1, std::memory_order_relaxed);
const bool outer = !t_in_swap; const bool outer = !t_in_swap;
if (outer) if (outer)
@@ -247,11 +175,11 @@ BOOL WINAPI hk_SwapBuffers(HDC hdc)
t_in_swap = true; t_in_swap = true;
if (g_ipc != nullptr) if (g_ipc != nullptr)
{ {
g_ipc->note_present(); // SwapBuffers is the GL present (counts toward video.present_calls) g_ipc->note_present();
} }
capture_gl(hdc); capture_gl(hdc);
} }
const BOOL r = g_hk_swapbuffers.stdcall<BOOL>(hdc); // __stdcall: call() is __cdecl on x86 -> crash const BOOL r = hook.stdcall<BOOL>(hdc); // __stdcall: call() is __cdecl on x86 -> crash
if (outer) if (outer)
{ {
t_in_swap = false; t_in_swap = false;
@@ -259,27 +187,16 @@ BOOL WINAPI hk_SwapBuffers(HDC hdc)
return r; return r;
} }
BOOL WINAPI hk_SwapBuffers(HDC hdc)
{
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
return swap_detour(g_hk_swapbuffers, g_id_swapbuffers, hdc);
}
BOOL WINAPI hk_wglSwapBuffers(HDC hdc) BOOL WINAPI hk_wglSwapBuffers(HDC hdc)
{ {
DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour DetourGate::Guard guard(g_gate); // keep the shared D3D state alive for this whole detour
hook_note_call(g_id_wglswap); return swap_detour(g_hk_wglswap, g_id_wglswap, hdc);
g_swaps.fetch_add(1, std::memory_order_relaxed);
const bool outer = !t_in_swap;
if (outer)
{
t_in_swap = true;
if (g_ipc != nullptr)
{
g_ipc->note_present(); // wglSwapBuffers is the GL present
}
capture_gl(hdc);
}
const BOOL r = g_hk_wglswap.stdcall<BOOL>(hdc); // __stdcall: call() is __cdecl on x86 -> crash
if (outer)
{
t_in_swap = false;
}
return r;
} }
} // namespace } // namespace
@@ -333,9 +250,7 @@ void remove_opengl_hooks()
hook_set_installed(g_id_swapbuffers, false); hook_set_installed(g_id_swapbuffers, false);
hook_set_installed(g_id_wglswap, false); hook_set_installed(g_id_wglswap, false);
g_gate.drain(); g_gate.drain();
// Persistent hooks: keep them ALIVE (disabled) so a stale detour's trampoline call is never freed g_shared.release();
// -- re-install re-enables them (see hook_install.hpp).
release_shared();
if (g_ctx != nullptr) if (g_ctx != nullptr)
{ {
g_ctx->Release(); g_ctx->Release();

9
hook/src/opengl_hook.hpp
View File

@@ -1,13 +1,12 @@
// Injected OpenGL capture path: hooks SwapBuffers / wglSwapBuffers in OpenGL games // Injected OpenGL capture path: hooks SwapBuffers / wglSwapBuffers in OpenGL games
// (which never call IDXGISwapChain::Present, so the DXGI present hook can't see // (which never call IDXGISwapChain::Present, so the DXGI present hook can't see
// them -- e.g. Phantom Brave), reads the GL backbuffer with glReadPixels, and // them), reads the GL backbuffer with glReadPixels, and uploads it into the same
// uploads it into the same shared keyed-mutex texture (coop_video_<pid>) the host // shared keyed-mutex texture (coop_video_<pid>) the host samples. Part of the video
// samples. Part of the video subsystem, alongside present_hook. // subsystem, alongside present_hook, d3d9_hook, and vk_hook.
// //
// glReadPixels forces a GPU->CPU readback each frame, so this is heavier than the // glReadPixels forces a GPU->CPU readback each frame, so this is heavier than the
// D3D shared-texture copy; fine for the 2D / lower-framerate games that tend to be // D3D shared-texture copy; fine for the 2D / lower-framerate games that tend to be
// OpenGL. Vulkan games present via vkQueuePresentKHR and aren't covered here (see // OpenGL.
// the README) -- use WGC for those.
#pragma once #pragma once
#include "ipc_client.hpp" #include "ipc_client.hpp"

209
hook/src/present_hook.cpp
View File

@@ -14,11 +14,13 @@
#include <safetyhook.hpp> #include <safetyhook.hpp>
#include "coop/shared_memory.hpp" #include "coop/protocol.hpp"
#include "debug_log.hpp" #include "debug_log.hpp"
#include "hook_guard.hpp" #include "hook_guard.hpp"
#include "hook_install.hpp" #include "hook_install.hpp"
#include "hook_registry.hpp" #include "hook_registry.hpp"
#include "shared_video_texture.hpp"
#include "vtable_hook.hpp"
namespace coop::hook namespace coop::hook
{ {
@@ -50,13 +52,11 @@ std::atomic<std::uint64_t> g_frames_shared{0};
// Present once we can see the game's device + backbuffer format. Guarded by // Present once we can see the game's device + backbuffer format. Guarded by
// g_tex_mutex (touched only on the render thread, but install/remove may race). // g_tex_mutex (touched only on the render thread, but install/remove may race).
std::mutex g_tex_mutex; std::mutex g_tex_mutex;
ID3D11Texture2D* g_shared_tex = nullptr; SharedVideoTexture g_shared;
IDXGIKeyedMutex* g_shared_mutex = nullptr;
HANDLE g_shared_handle = nullptr;
UINT g_share_w = 0;
UINT g_share_h = 0;
DXGI_FORMAT g_share_fmt = DXGI_FORMAT_UNKNOWN;
bool g_unsupported_logged = false; bool g_unsupported_logged = false;
// The Present hook's shared texture also carries RENDER_TARGET so the On12-wrapped D3D12
// backbuffer can be copied into it.
constexpr UINT kShareBind = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
// D3D11On12 bridge for D3D12 games: we build our own D3D11 device on top of the // D3D11On12 bridge for D3D12 games: we build our own D3D11 device on top of the
// game's D3D12 device (with a queue we create on it) so we can wrap the D3D12 // game's D3D12 device (with a queue we create on it) so we can wrap the D3D12
@@ -162,103 +162,6 @@ bool first_capture_from(void* swapchain)
return true; return true;
} }
void* vtable_method(void* obj, unsigned index)
{
return (*reinterpret_cast<void***>(obj))[index];
}
// Drop the shared texture/mutex/handle. Caller holds g_tex_mutex.
void release_shared_locked()
{
if (g_shared_mutex != nullptr)
{
g_shared_mutex->Release();
g_shared_mutex = nullptr;
}
if (g_shared_tex != nullptr)
{
g_shared_tex->Release();
g_shared_tex = nullptr;
}
if (g_shared_handle != nullptr)
{
CloseHandle(g_shared_handle);
g_shared_handle = nullptr;
}
g_share_w = g_share_h = 0;
g_share_fmt = DXGI_FORMAT_UNKNOWN;
}
// (Re)create the shared keyed-mutex texture for a w x h `fmt` backbuffer on the
// game's `device`. Caller holds g_tex_mutex. Returns true if it's ready.
bool ensure_shared_texture_locked(ID3D11Device* device, UINT w, UINT h, DXGI_FORMAT fmt)
{
if (g_shared_tex != nullptr && g_share_w == w && g_share_h == h && g_share_fmt == fmt)
{
return true; // already matches the current backbuffer
}
release_shared_locked();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = fmt;
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
ID3D11Texture2D* tex = nullptr;
HRESULT hr = device->CreateTexture2D(&desc, nullptr, &tex);
if (FAILED(hr) || tex == nullptr)
{
logf("present: CreateTexture2D(shared) failed hr=0x%08lX (%ux%u fmt=%d)",
static_cast<unsigned long>(hr), w, h, static_cast<int>(fmt));
return false;
}
IDXGIResource1* res = nullptr;
hr = tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res));
if (FAILED(hr) || res == nullptr)
{
logf("present: QI IDXGIResource1 failed hr=0x%08lX", static_cast<unsigned long>(hr));
tex->Release();
return false;
}
const std::wstring name = video_share_name(g_pid);
HANDLE handle = nullptr;
hr = res->CreateSharedHandle(nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE,
name.c_str(), &handle);
res->Release();
if (FAILED(hr) || handle == nullptr)
{
logf("present: CreateSharedHandle failed hr=0x%08lX", static_cast<unsigned long>(hr));
tex->Release();
return false;
}
IDXGIKeyedMutex* mutex = nullptr;
hr = tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&mutex));
if (FAILED(hr) || mutex == nullptr)
{
logf("present: QI IDXGIKeyedMutex failed hr=0x%08lX", static_cast<unsigned long>(hr));
CloseHandle(handle);
tex->Release();
return false;
}
g_shared_tex = tex;
g_shared_mutex = mutex;
g_shared_handle = handle;
g_share_w = w;
g_share_h = h;
g_share_fmt = fmt;
logf("present: shared texture ready %ux%u fmt=%d name=%ls", w, h, static_cast<int>(fmt), name.c_str());
return true;
}
// Drop the D3D11On12 bridge. Caller holds g_tex_mutex. // Drop the D3D11On12 bridge. Caller holds g_tex_mutex.
void release_on12_locked() void release_on12_locked()
{ {
@@ -425,15 +328,13 @@ void capture_backbuffer_d3d12(IDXGISwapChain* sc)
std::scoped_lock lock(g_tex_mutex); std::scoped_lock lock(g_tex_mutex);
if (ensure_on12_locked(dev)) if (ensure_on12_locked(dev))
{ {
// DX12 capture is pricier than DX11/OpenGL (measured ~0.38 ms vs ~0.05/0.09 ms of // DX12 capture costs more present-thread overhead than DX11/OpenGL (~0.38 ms vs
// present-thread overhead) because it goes through the D3D11On12 bridge. The cost is NOT // ~0.05/0.09 ms, measured) because it goes through the D3D11On12 bridge: the
// CreateWrappedResource (measured ~0.012 ms) -- it's the CopyResource issued on the 11On12 // CopyResource on the 11On12 immediate context (~0.13 ms) plus the mandatory Flush
// immediate context (~0.13 ms) plus the mandatory Flush to make the shared copy visible to // (~0.06 ms) that makes the shared copy visible to the host -- CreateWrappedResource
// the host (~0.06 ms), neither of which the native-D3D11 path pays. Eliminating it needs a // itself is ~0.012 ms. Avoiding the bridge needs a native-D3D12 copy-queue path into a
// native-D3D12 copy-queue path into a D3D12-shared texture, but the host consumes the // D3D12-shared texture AND a shared-fence host sync (the keyed mutex is a D3D11
// shared surface via an IDXGIKeyedMutex (a D3D11 concept), so that also means switching the // concept) -- a cross-API rewrite that isn't worth ~5% of a 144 Hz frame.
// DX12 producer<->host sync to a shared ID3D12Fence -- a cross-API rewrite. Deferred: the
// overhead is ~5% of a 144 Hz frame and the capture is correct; the bridge stays for now.
// //
// Order our copy after the game's frame without burdening the game's queue: the // Order our copy after the game's frame without burdening the game's queue: the
// game queue signals the fence (cheap), our copy queue waits on it. Skipped if the // game queue signals the fence (cheap), our copy queue waits on it. Skipped if the
@@ -464,13 +365,13 @@ void capture_backbuffer_d3d12(IDXGISwapChain* sc)
w = d.Width; w = d.Width;
h = d.Height; h = d.Height;
fmt = d.Format; fmt = d.Format;
if (d.SampleDesc.Count == 1 && ensure_shared_texture_locked(g_on12_d3d11, w, h, fmt) && if (d.SampleDesc.Count == 1 &&
g_shared_mutex != nullptr) g_shared.ensure(g_on12_d3d11, w, h, fmt, g_pid, "present", kShareBind))
{ {
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 0) == S_OK) if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 0) == S_OK)
{ {
g_on12_ctx->CopyResource(g_shared_tex, wtex); g_on12_ctx->CopyResource(g_shared.texture(), wtex);
g_shared_mutex->ReleaseSync(kVideoMutexKey); g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true; shared = true;
} }
else else
@@ -623,16 +524,16 @@ void capture_backbuffer_d3d10(IDXGISwapChain* sc, ID3D10Texture2D* backbuf)
} }
if (g_d3d10_staging != nullptr && ensure_aux_d3d11_locked() && if (g_d3d10_staging != nullptr && ensure_aux_d3d11_locked() &&
ensure_shared_texture_locked(g_aux_d3d11, bd.Width, bd.Height, bd.Format) && g_shared_mutex != nullptr) g_shared.ensure(g_aux_d3d11, bd.Width, bd.Height, bd.Format, g_pid, "present", kShareBind))
{ {
gdev->CopyResource(g_d3d10_staging, backbuf); gdev->CopyResource(g_d3d10_staging, backbuf);
D3D10_MAPPED_TEXTURE2D m{}; D3D10_MAPPED_TEXTURE2D m{};
if (SUCCEEDED(g_d3d10_staging->Map(0, D3D10_MAP_READ, 0, &m)) && m.pData != nullptr) if (SUCCEEDED(g_d3d10_staging->Map(0, D3D10_MAP_READ, 0, &m)) && m.pData != nullptr)
{ {
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK) if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{ {
g_aux_ctx->UpdateSubresource(g_shared_tex, 0, nullptr, m.pData, m.RowPitch, 0); g_aux_ctx->UpdateSubresource(g_shared.texture(), 0, nullptr, m.pData, m.RowPitch, 0);
g_shared_mutex->ReleaseSync(kVideoMutexKey); g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true; shared = true;
} }
else else
@@ -698,7 +599,7 @@ void capture_backbuffer(IDXGISwapChain* sc)
if (device != nullptr && ctx != nullptr) if (device != nullptr && ctx != nullptr)
{ {
std::scoped_lock lock(g_tex_mutex); std::scoped_lock lock(g_tex_mutex);
if (ensure_shared_texture_locked(device, bd.Width, bd.Height, bd.Format) && g_shared_mutex != nullptr) if (g_shared.ensure(device, bd.Width, bd.Height, bd.Format, g_pid, "present", kShareBind))
{ {
if (first_capture_from(sc)) if (first_capture_from(sc))
{ {
@@ -708,10 +609,10 @@ void capture_backbuffer(IDXGISwapChain* sc)
// Key 0 on both sides: a plain cross-process mutex on the texture (created // Key 0 on both sides: a plain cross-process mutex on the texture (created
// released at key 0). Bounded wait so a stalled host consumer can never hang // released at key 0). Bounded wait so a stalled host consumer can never hang
// the game's render thread. // the game's render thread.
if (g_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK) if (g_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{ {
ctx->CopyResource(g_shared_tex, backbuf); ctx->CopyResource(g_shared.texture(), backbuf);
g_shared_mutex->ReleaseSync(kVideoMutexKey); g_shared.mutex()->ReleaseSync(kVideoMutexKey);
shared = true; shared = true;
} }
else else
@@ -722,7 +623,6 @@ void capture_backbuffer(IDXGISwapChain* sc)
else else
{ {
cant_host = true; // device can't host the shared texture -> try the D3D10 path cant_host = true; // device can't host the shared texture -> try the D3D10 path
release_shared_locked();
} }
} }
if (ctx != nullptr) if (ctx != nullptr)
@@ -819,33 +719,39 @@ void* grab_execute_command_lists_address()
return addr; return addr;
} }
HRESULT STDMETHODCALLTYPE hk_Present(IDXGISwapChain* sc, UINT sync_interval, UINT flags) // Shared body of the Present/Present1 detours: count the call, log each distinct
// (swapchain, flags) once, and capture the backbuffer. A DXGI_PRESENT_TEST present draws
// nothing (it only probes occlusion), so it inflates the Present count without producing a
// frame -- which is why some games show more presents than captured frames -- and is not
// worth copying.
void on_present(IDXGISwapChain* sc, UINT flags, int hook_id, const char* method)
{ {
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour hook_note_call(hook_id);
hook_note_call(g_id_present);
g_present_calls.fetch_add(1, std::memory_order_relaxed); g_present_calls.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr) if (g_ipc != nullptr)
{ {
g_ipc->note_present(); g_ipc->note_present();
} }
// Log each distinct (swapchain, flags) once. A DXGI_PRESENT_TEST present draws nothing
// (it only probes occlusion), so those inflate the Present count without producing a
// frame -- which is why some games show more presents than captured frames.
if (first_present_with_flags(sc, flags)) if (first_present_with_flags(sc, flags))
{ {
logf("present: swapchain=%p Present flags=0x%08X%s", sc, flags, logf("present: swapchain=%p %s flags=0x%08X%s", sc, method, flags,
(flags & DXGI_PRESENT_TEST) ? " (DXGI_PRESENT_TEST: occlusion probe, no frame drawn)" : ""); (flags & DXGI_PRESENT_TEST) ? " (DXGI_PRESENT_TEST: occlusion probe, no frame drawn)" : "");
} }
// DXGI_PRESENT_TEST presents nothing; don't bother copying for it.
if ((flags & DXGI_PRESENT_TEST) == 0) if ((flags & DXGI_PRESENT_TEST) == 0)
{ {
capture_backbuffer(sc); capture_backbuffer(sc);
} }
}
HRESULT STDMETHODCALLTYPE hk_Present(IDXGISwapChain* sc, UINT sync_interval, UINT flags)
{
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
on_present(sc, flags, g_id_present, "Present");
// stdcall(), NOT call(): IDXGISwapChain::Present is __stdcall, but SafetyHook's // stdcall(), NOT call(): IDXGISwapChain::Present is __stdcall, but SafetyHook's
// call() invokes the trampoline through a __cdecl pointer (the default on x86). // call() invokes the trampoline through a __cdecl pointer (the default on x86).
// On 32-bit that double-cleans the stack -> ESP imbalance -> Run-Time Check // On 32-bit that double-cleans the stack -> ESP imbalance -> Run-Time Check
// Failure #0 and an instant crash. On x64 the conventions collapse, so it only // Failure #0 and an instant crash. On x64 the conventions collapse, so it only
// bit 32-bit games (e.g. Slaps and Beans froze the moment it presented). // bites 32-bit games.
return g_hk_present.stdcall<HRESULT>(sc, sync_interval, flags); return g_hk_present.stdcall<HRESULT>(sc, sync_interval, flags);
} }
@@ -853,21 +759,7 @@ HRESULT STDMETHODCALLTYPE hk_Present1(IDXGISwapChain1* sc, UINT sync_interval, U
const DXGI_PRESENT_PARAMETERS* params) const DXGI_PRESENT_PARAMETERS* params)
{ {
DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour DetourGate::Guard guard(g_gate); // keep the shared texture / On12 bridge alive for this detour
hook_note_call(g_id_present1); on_present(sc, flags, g_id_present1, "Present1"); // IDXGISwapChain1 derives from IDXGISwapChain
g_present_calls.fetch_add(1, std::memory_order_relaxed);
if (g_ipc != nullptr)
{
g_ipc->note_present();
}
if (first_present_with_flags(sc, flags)) // see hk_Present
{
logf("present: swapchain=%p Present1 flags=0x%08X%s", sc, flags,
(flags & DXGI_PRESENT_TEST) ? " (DXGI_PRESENT_TEST: occlusion probe, no frame drawn)" : "");
}
if ((flags & DXGI_PRESENT_TEST) == 0)
{
capture_backbuffer(sc); // IDXGISwapChain1 derives from IDXGISwapChain
}
return g_hk_present1.stdcall<HRESULT>(sc, sync_interval, flags, params); // __stdcall, see hk_Present return g_hk_present1.stdcall<HRESULT>(sc, sync_interval, flags, params); // __stdcall, see hk_Present
} }
@@ -995,12 +887,11 @@ bool install_present_hooks(IpcClient& ipc)
void remove_present_hooks() void remove_present_hooks()
{ {
// DISABLE (persistent model -- never destroy during the session): this restores the // DISABLE, never destroy (persistent-hook model, see hook_install.hpp): this restores the
// Present/Present1/ECL bytes under thread suspension so no NEW detour starts, while KEEPING the // Present/Present1/ECL bytes so no NEW detour starts, while KEEPING the trampolines alive -- an
// trampolines alive -- an in-flight detour about to call g_hk_present.stdcall() (the trampoline) // in-flight detour about to call g_hk_present.stdcall() (the trampoline) must never have it
// must never have it freed under it (destroying = {} would, an AV the thousands/s storm hits // freed under it. So: disable -> drain (in-flight detours finish on the live trampoline) ->
// reliably). So: disable -> drain (in-flight detours finish on the live trampoline) -> leave the // leave the hooks alive for a re-install to re-enable.
// hooks alive (re-install re-enables them; see hook_install.hpp).
disable_for_removal(g_hk_present); disable_for_removal(g_hk_present);
disable_for_removal(g_hk_present1); disable_for_removal(g_hk_present1);
disable_for_removal(g_hk_ecl); disable_for_removal(g_hk_ecl);
@@ -1008,14 +899,12 @@ void remove_present_hooks()
hook_set_installed(g_id_present1, false); hook_set_installed(g_id_present1, false);
hook_set_installed(g_id_ecl, false); hook_set_installed(g_id_ecl, false);
g_gate.drain(); g_gate.drain();
// Persistent hooks: keep g_hk_present/present1/ecl ALIVE (disabled), so the trampoline a stale
// detour may still call is never freed -- re-install re-enables them (see hook_install.hpp).
g_present_queue.store(nullptr, std::memory_order_relaxed); g_present_queue.store(nullptr, std::memory_order_relaxed);
g_logged_presents_n = 0; // let a fresh injection re-log the present pattern g_logged_presents_n = 0; // let a fresh injection re-log the present pattern
g_logged_swapchains_n = 0; g_logged_swapchains_n = 0;
{ {
std::scoped_lock lock(g_tex_mutex); std::scoped_lock lock(g_tex_mutex);
release_shared_locked(); g_shared.release();
release_on12_locked(); release_on12_locked();
release_aux_locked(); release_aux_locked();
} }

7
hook/src/present_hook.hpp
View File

@@ -5,9 +5,10 @@
// stable alternative to Windows Graphics Capture (which composites off the // stable alternative to Windows Graphics Capture (which composites off the
// desktop and can stutter / show a capture border). // desktop and can stutter / show a capture border).
// //
// Only DXGI swapchains (D3D10/11/12-backed games) are caught; the backbuffer // Only DXGI swapchains (D3D10/11/12-backed games) are caught: D3D11 backbuffers
// must be an ID3D11Texture2D (the common D3D11 case). Games on D3D9 or a pure // copy directly, D3D12 goes through a D3D11On12 bridge, and D3D10 reads back
// D3D12 resource path won't engage this hook -- WGC stays as the fallback. // through the game's own device. D3D9 / OpenGL / Vulkan games have their own
// hooks (d3d9_hook, opengl_hook, vk_hook); WGC stays as the capture fallback.
#pragma once #pragma once
#include "ipc_client.hpp" #include "ipc_client.hpp"

10
hook/src/rate_estimator.hpp
View File

@@ -2,13 +2,13 @@
// //
// When we attach to an already-running game we never saw its IAudioClient::Initialize, // When we attach to an already-running game we never saw its IAudioClient::Initialize,
// so we assume the device mix format and recover the *true* sample rate by timing how // so we assume the device mix format and recover the *true* sample rate by timing how
// fast the game renders frames. The naive version (one short ~200 ms window, snap to the // fast the game renders frames. A naive estimator (one short ~200 ms window, snap to the
// nearest standard rate, accept whatever came out) is fragile: WASAPI delivers audio in // nearest standard rate, accept whatever comes out) is fragile: WASAPI delivers audio in
// quantized ~10 ms buffers, so one extra buffer at a window edge is a ~5% error over // quantized ~10 ms buffers, so one extra buffer at a window edge is a ~5% error over
// 200 ms, which lands *between* standard rates (they're >8% apart) and used to be // 200 ms, which lands *between* standard rates (they're >8% apart) -- e.g. 44100
// published verbatim -- e.g. 44100 measured as ~46205. // measuring as ~46205.
// //
// This estimator fixes that with three rules: // Three rules make it robust:
// 1. Longer windows (~0.5 s) -> the per-buffer quantization error drops to ~2%. // 1. Longer windows (~0.5 s) -> the per-buffer quantization error drops to ~2%.
// 2. Reject a window that doesn't snap to a standard rate. Standard rates are far // 2. Reject a window that doesn't snap to a standard rate. Standard rates are far
// enough apart that any error big enough to miss the right one lands in no-man's- // enough apart that any error big enough to miss the right one lands in no-man's-

135
hook/src/shared_video_texture.hpp Normal file
View File

@@ -0,0 +1,135 @@
// The keyed-mutex D3D11 texture a video backend publishes captured frames into. Created on
// whichever D3D11 device the backend has (the game's, an On12 bridge, or a hook-owned aux
// device), named coop_video_<pid> (video_share_name) so the host opens it by name, and
// synchronized with an IDXGIKeyedMutex at key 0 on both sides (kVideoMutexKey). Shared by the
// DXGI/D3D10/D3D12 Present hook, the OpenGL and D3D9 hooks, and the Vulkan capture.
#pragma once
#include <windows.h>
#include <d3d11.h>
#include <dxgi1_2.h>
#include "coop/shared_memory.hpp"
#include "debug_log.hpp"
namespace coop::hook
{
class SharedVideoTexture
{
public:
SharedVideoTexture() = default;
~SharedVideoTexture()
{
release();
}
SharedVideoTexture(const SharedVideoTexture&) = delete;
SharedVideoTexture& operator=(const SharedVideoTexture&) = delete;
// (Re)create the shared texture for a w x h `fmt` frame on `device`; cheap no-op when the
// current texture already matches. `tag` prefixes the log lines with the owning backend.
// `bind` lets a backend request bind flags beyond SHADER_RESOURCE. Returns true when
// texture() and mutex() are ready; false (released) when the device can't host it.
bool ensure(ID3D11Device* device, UINT w, UINT h, DXGI_FORMAT fmt, unsigned long pid, const char* tag,
UINT bind = D3D11_BIND_SHADER_RESOURCE)
{
if (m_tex != nullptr && m_w == w && m_h == h && m_fmt == fmt)
{
return true;
}
release();
D3D11_TEXTURE2D_DESC desc{};
desc.Width = w;
desc.Height = h;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = fmt;
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = bind;
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
HRESULT hr = device->CreateTexture2D(&desc, nullptr, &m_tex);
if (FAILED(hr) || m_tex == nullptr)
{
logf("%s: CreateTexture2D(shared) failed hr=0x%08lX (%ux%u fmt=%d)", tag,
static_cast<unsigned long>(hr), w, h, static_cast<int>(fmt));
release();
return false;
}
IDXGIResource1* res = nullptr;
hr = m_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res));
if (FAILED(hr) || res == nullptr)
{
logf("%s: QI IDXGIResource1 failed hr=0x%08lX", tag, static_cast<unsigned long>(hr));
release();
return false;
}
const std::wstring name = video_share_name(pid);
hr = res->CreateSharedHandle(nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE,
name.c_str(), &m_handle);
res->Release();
if (FAILED(hr) || m_handle == nullptr)
{
logf("%s: CreateSharedHandle failed hr=0x%08lX", tag, static_cast<unsigned long>(hr));
release();
return false;
}
hr = m_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&m_mutex));
if (FAILED(hr) || m_mutex == nullptr)
{
logf("%s: QI IDXGIKeyedMutex failed hr=0x%08lX", tag, static_cast<unsigned long>(hr));
release();
return false;
}
m_w = w;
m_h = h;
m_fmt = fmt;
logf("%s: shared texture ready %ux%u fmt=%d name=%ls", tag, w, h, static_cast<int>(fmt), name.c_str());
return true;
}
void release()
{
if (m_mutex != nullptr)
{
m_mutex->Release();
m_mutex = nullptr;
}
if (m_tex != nullptr)
{
m_tex->Release();
m_tex = nullptr;
}
if (m_handle != nullptr)
{
CloseHandle(m_handle);
m_handle = nullptr;
}
m_w = m_h = 0;
m_fmt = DXGI_FORMAT_UNKNOWN;
}
[[nodiscard]] ID3D11Texture2D* texture() const
{
return m_tex;
}
[[nodiscard]] IDXGIKeyedMutex* mutex() const
{
return m_mutex;
}
private:
ID3D11Texture2D* m_tex = nullptr;
IDXGIKeyedMutex* m_mutex = nullptr;
HANDLE m_handle = nullptr; // named NT handle backing the share; closed on release
UINT m_w = 0;
UINT m_h = 0;
DXGI_FORMAT m_fmt = DXGI_FORMAT_UNKNOWN;
};
} // namespace coop::hook

80
hook/src/vk_capture.cpp
View File

@@ -3,7 +3,6 @@
#include <cstring> #include <cstring>
#include "coop/protocol.hpp" #include "coop/protocol.hpp"
#include "coop/shared_memory.hpp"
namespace coop::hook namespace coop::hook
{ {
@@ -15,7 +14,7 @@ VkCapture::~VkCapture()
// Prefer HOST_CACHED memory: the reaper reads every byte of this buffer back on the CPU, and an // Prefer HOST_CACHED memory: the reaper reads every byte of this buffer back on the CPU, and an
// uncached / write-combined mapping (what a plain HOST_VISIBLE|HOST_COHERENT type usually is on a // uncached / write-combined mapping (what a plain HOST_VISIBLE|HOST_COHERENT type usually is on a
// discrete GPU) makes that read run at PCIe latency -- the original ~370 ms/frame stall. Order of // discrete GPU) makes that read run at PCIe latency -- hundreds of ms per frame. Order of
// preference: cached+coherent (fast read, no invalidate) > cached (fast read, needs invalidate) > // preference: cached+coherent (fast read, no invalidate) > cached (fast read, needs invalidate) >
// coherent-only (the slow fallback, only if nothing cached is host-visible). // coherent-only (the slow fallback, only if nothing cached is host-visible).
bool VkCapture::find_readback_memory(std::uint32_t type_bits, std::uint32_t& out_index, bool& out_coherent) bool VkCapture::find_readback_memory(std::uint32_t type_bits, std::uint32_t& out_index, bool& out_coherent)
@@ -220,29 +219,9 @@ bool VkCapture::ensure_d3d()
m_d3d != nullptr; m_d3d != nullptr;
} }
void VkCapture::release_shared()
{
if (m_shared_mutex != nullptr)
{
m_shared_mutex->Release();
m_shared_mutex = nullptr;
}
if (m_shared_tex != nullptr)
{
m_shared_tex->Release();
m_shared_tex = nullptr;
}
if (m_shared_handle != nullptr)
{
CloseHandle(m_shared_handle);
m_shared_handle = nullptr;
}
m_share_w = m_share_h = 0;
}
void VkCapture::release_d3d() void VkCapture::release_d3d()
{ {
release_shared(); m_shared.release();
if (m_d3d_ctx != nullptr) if (m_d3d_ctx != nullptr)
{ {
m_d3d_ctx->Release(); m_d3d_ctx->Release();
@@ -255,49 +234,6 @@ void VkCapture::release_d3d()
} }
} }
bool VkCapture::ensure_shared_texture(UINT w, UINT h)
{
if (m_shared_tex != nullptr && m_share_w == w && m_share_h == h)
{
return true;
}
release_shared();
D3D11_TEXTURE2D_DESC d{};
d.Width = w;
d.Height = h;
d.MipLevels = 1;
d.ArraySize = 1;
d.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
d.SampleDesc.Count = 1;
d.Usage = D3D11_USAGE_DEFAULT;
d.BindFlags = D3D11_BIND_SHADER_RESOURCE;
d.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
if (FAILED(m_d3d->CreateTexture2D(&d, nullptr, &m_shared_tex)) || m_shared_tex == nullptr)
{
return false;
}
IDXGIResource1* res = nullptr;
if (FAILED(m_shared_tex->QueryInterface(__uuidof(IDXGIResource1), reinterpret_cast<void**>(&res))) ||
res == nullptr)
{
release_shared();
return false;
}
const std::wstring name = video_share_name(m_pid);
const HRESULT hr = res->CreateSharedHandle(
nullptr, DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE, name.c_str(), &m_shared_handle);
res->Release();
if (FAILED(hr) || m_shared_handle == nullptr ||
FAILED(m_shared_tex->QueryInterface(__uuidof(IDXGIKeyedMutex), reinterpret_cast<void**>(&m_shared_mutex))))
{
release_shared();
return false;
}
m_share_w = w;
m_share_h = h;
return true;
}
// --- init / present / reaper ------------------------------------------------- // --- init / present / reaper -------------------------------------------------
void VkCapture::init(VkPhysicalDevice phys, VkDevice device, std::uint32_t queue_family, const Fns& fns, void VkCapture::init(VkPhysicalDevice phys, VkDevice device, std::uint32_t queue_family, const Fns& fns,
unsigned long pid, std::function<void(std::uint32_t, std::uint32_t)> on_frame) unsigned long pid, std::function<void(std::uint32_t, std::uint32_t)> on_frame)
@@ -449,12 +385,12 @@ void VkCapture::reap_slot(Slot& s)
} }
bool published = false; bool published = false;
if (ensure_d3d() && ensure_shared_texture(s.w, s.h) && if (ensure_d3d() && m_shared.ensure(m_d3d, s.w, s.h, DXGI_FORMAT_R8G8B8A8_UNORM, m_pid, "vk") &&
m_shared_mutex->AcquireSync(kVideoMutexKey, 8) == S_OK) m_shared.mutex()->AcquireSync(kVideoMutexKey, 8) == S_OK)
{ {
m_d3d_ctx->UpdateSubresource(m_shared_tex, 0, nullptr, m_rgba.data(), static_cast<UINT>(row), 0); m_d3d_ctx->UpdateSubresource(m_shared.texture(), 0, nullptr, m_rgba.data(), static_cast<UINT>(row), 0);
m_d3d_ctx->Flush(); m_d3d_ctx->Flush();
m_shared_mutex->ReleaseSync(kVideoMutexKey); m_shared.mutex()->ReleaseSync(kVideoMutexKey);
published = true; published = true;
} }
@@ -507,8 +443,8 @@ void VkCapture::shutdown()
m_q_cv.notify_all(); m_q_cv.notify_all();
m_reaper.join(); m_reaper.join();
} }
// The reaper is gone (no more submits/reads); drain any GPU work still referencing our resources, // The reaper is gone (no more submits/reads); drain any GPU work still referencing our
// then free. DeviceWaitIdle here mirrors what the inline hook / layer did before this component. // resources, then free.
if (m_device != VK_NULL_HANDLE) if (m_device != VK_NULL_HANDLE)
{ {
if (m_fns.DeviceWaitIdle != nullptr) if (m_fns.DeviceWaitIdle != nullptr)

22
hook/src/vk_capture.hpp
View File

@@ -3,12 +3,12 @@
// how they get into the dispatch chain; the capture itself -- copy the presented image into a shared // how they get into the dispatch chain; the capture itself -- copy the presented image into a shared
// keyed-mutex D3D11 texture for the host to sample -- is identical, so it lives here once. // keyed-mutex D3D11 texture for the host to sample -- is identical, so it lives here once.
// //
// Performance contract (the whole reason this is a separate component): the read-back must NOT stall // Performance contract (the whole reason this is a separate component): the read-back must NOT
// the game's present thread. Against a real 144 FPS game the original inline version dropped it to // stall the game's present thread. Doing the GPU copy + a synchronous CPU read of the mapped
// ~3 FPS because it did the GPU copy + a synchronous CPU read of the mapped staging buffer + the // staging buffer + the swizzle + the D3D upload on the present thread drops a 144 FPS game to a
// swizzle + the D3D upload all on the present thread -- and the staging buffer was HOST_COHERENT (on // few FPS -- especially when the staging buffer is plain HOST_COHERENT (on a discrete GPU that's
// a discrete GPU that's write-combined/uncached, where a scattered CPU read runs at PCIe latency: // write-combined/uncached, where a scattered CPU read runs at PCIe latency: hundreds of ms for
// ~370 ms for one 1080p frame). This component fixes both halves: // one 1080p frame). This component avoids both halves:
// * the present thread only records + submits the copy (sub-millisecond) and returns immediately; // * the present thread only records + submits the copy (sub-millisecond) and returns immediately;
// * a dedicated reaper thread waits the copy's fence, reads the staging buffer, swizzles and // * a dedicated reaper thread waits the copy's fence, reads the staging buffer, swizzles and
// uploads -- off the critical path; and // uploads -- off the critical path; and
@@ -34,6 +34,8 @@
#define VK_NO_PROTOTYPES #define VK_NO_PROTOTYPES
#include <vulkan/vulkan.h> #include <vulkan/vulkan.h>
#include "shared_video_texture.hpp"
namespace coop::hook namespace coop::hook
{ {
@@ -131,8 +133,6 @@ private:
bool ensure_d3d(); bool ensure_d3d();
void release_d3d(); void release_d3d();
bool ensure_shared_texture(UINT w, UINT h);
void release_shared();
void reaper_main(); void reaper_main();
void reap_slot(Slot& s); void reap_slot(Slot& s);
@@ -150,11 +150,7 @@ private:
// --- D3D11 shared texture (reaper thread only) --- // --- D3D11 shared texture (reaper thread only) ---
ID3D11Device* m_d3d = nullptr; ID3D11Device* m_d3d = nullptr;
ID3D11DeviceContext* m_d3d_ctx = nullptr; ID3D11DeviceContext* m_d3d_ctx = nullptr;
ID3D11Texture2D* m_shared_tex = nullptr; SharedVideoTexture m_shared;
IDXGIKeyedMutex* m_shared_mutex = nullptr;
HANDLE m_shared_handle = nullptr;
UINT m_share_w = 0;
UINT m_share_h = 0;
std::vector<unsigned char> m_rgba; // reaper scratch (swizzled frame) std::vector<unsigned char> m_rgba; // reaper scratch (swizzled frame)
unsigned long m_pid = 0; unsigned long m_pid = 0;
std::function<void(std::uint32_t, std::uint32_t)> m_on_frame; std::function<void(std::uint32_t, std::uint32_t)> m_on_frame;

5
hook/src/vk_hook.cpp
View File

@@ -319,9 +319,8 @@ VKAPI_ATTR PFN_vkVoidFunction VKAPI_CALL hk_vkGetInstanceProcAddr(VkInstance ins
// vkGetInstanceProcAddr can also resolve device-level functions (the loader returns a // vkGetInstanceProcAddr can also resolve device-level functions (the loader returns a
// dispatch trampoline). A game -- or volk's volkLoadInstance -- that resolves the present / // dispatch trampoline). A game -- or volk's volkLoadInstance -- that resolves the present /
// swapchain entry points this way (rather than via vkGetDeviceProcAddr) would otherwise get // swapchain entry points this way (rather than via vkGetDeviceProcAddr) would otherwise get
// the real loader pointer and bypass our capture. Sphere Spectacle does exactly this, so // the real loader pointer and bypass our capture, so intercept them here too. (Our detours
// intercept them here too. (Our detours gate on g_capture_enabled / g_device, so handing them // gate on g_capture_enabled / g_device, so handing them out before the device exists is safe.)
// out before the device exists is safe.)
if (std::strcmp(name, "vkQueuePresentKHR") == 0) if (std::strcmp(name, "vkQueuePresentKHR") == 0)
{ {
return reinterpret_cast<PFN_vkVoidFunction>(&hk_vkQueuePresentKHR); return reinterpret_cast<PFN_vkVoidFunction>(&hk_vkQueuePresentKHR);

11
hook/src/vtable_hook.hpp
View File

@@ -5,8 +5,8 @@
// mishandles -> the original runs with garbage args and faults. Swapping the vtable entry leaves // mishandles -> the original runs with garbage args and faults. Swapping the vtable entry leaves
// the original code untouched, so it runs with a pristine stack regardless of prologue shape. // the original code untouched, so it runs with a pristine stack regardless of prologue shape.
// Every instance of a COM coclass shares one vtable, so a single swap intercepts all of them. // Every instance of a COM coclass shares one vtable, so a single swap intercepts all of them.
// See the project's stdcall-x86 note. Shared by the audio render-hook (audio_hook.cpp) and the // Shared by the audio render-hook (audio_hook.cpp) and the input-side DirectInput hook
// input-side DirectInput hook (mkb_hook.cpp). // (mkb_hook.cpp).
#pragma once #pragma once
#include <windows.h> #include <windows.h>
@@ -14,6 +14,13 @@
namespace coop::hook namespace coop::hook
{ {
// Read a COM object's vtable slot (e.g. to grab a method's address off a probe object for an
// inline hook).
inline void* vtable_method(void* obj, unsigned index)
{
return (*reinterpret_cast<void***>(obj))[index];
}
class VtableHook class VtableHook
{ {
public: public:

18
hook/src/xinput_hook.cpp
View File

@@ -167,6 +167,7 @@ DWORD WINAPI hk_XInputSetState(DWORD user_index, XINPUT_VIBRATION* vibration)
return ERROR_SUCCESS; return ERROR_SUCCESS;
} }
// `name` is a GetProcAddress LPCSTR: an export name, or MAKEINTRESOURCEA(ordinal).
void hook_export(HMODULE module, const char* name, void* detour, int registry_id) void hook_export(HMODULE module, const char* name, void* detour, int registry_id)
{ {
if (module == nullptr) if (module == nullptr)
@@ -181,20 +182,6 @@ void hook_export(HMODULE module, const char* name, void* detour, int registry_id
} }
} }
void hook_ordinal(HMODULE module, WORD ordinal, void* detour, int registry_id)
{
if (module == nullptr)
{
return;
}
if (void* target = reinterpret_cast<void*>(GetProcAddress(module, MAKEINTRESOURCEA(ordinal))))
{
g_hooks.emplace_back();
install_inline(g_hooks.back(), target, detour); // assign-then-enable (no install race)
hook_set_installed(registry_id, true);
}
}
} // namespace } // namespace
bool install_xinput_hooks(IpcClient& ipc) bool install_xinput_hooks(IpcClient& ipc)
@@ -222,7 +209,8 @@ bool install_xinput_hooks(IpcClient& ipc)
continue; continue;
} }
hook_export(module, "XInputGetState", reinterpret_cast<void*>(&hk_XInputGetState), g_id_getstate); hook_export(module, "XInputGetState", reinterpret_cast<void*>(&hk_XInputGetState), g_id_getstate);
hook_ordinal(module, 100, reinterpret_cast<void*>(&hk_XInputGetStateEx), g_id_getstateex); hook_export(module, MAKEINTRESOURCEA(100), reinterpret_cast<void*>(&hk_XInputGetStateEx),
g_id_getstateex); // XInputGetStateEx is exported by ordinal only
hook_export(module, "XInputGetCapabilities", reinterpret_cast<void*>(&hk_XInputGetCapabilities), hook_export(module, "XInputGetCapabilities", reinterpret_cast<void*>(&hk_XInputGetCapabilities),
g_id_getcaps); g_id_getcaps);
hook_export(module, "XInputSetState", reinterpret_cast<void*>(&hk_XInputSetState), g_id_setstate); hook_export(module, "XInputSetState", reinterpret_cast<void*>(&hk_XInputSetState), g_id_setstate);

3
tests/CMakeLists.txt
View File

@@ -346,7 +346,8 @@ add_test(NAME opengl_hook_test COMMAND opengl_hook_test)
# regression (the assertion fails on the synchronous design). Skips without a Vulkan ICD. # regression (the assertion fails on the synchronous design). Skips without a Vulkan ICD.
add_executable(vk_capture_perf_test add_executable(vk_capture_perf_test
vk_capture_perf_test.cpp vk_capture_perf_test.cpp
${CMAKE_SOURCE_DIR}/hook/src/vk_capture.cpp) ${CMAKE_SOURCE_DIR}/hook/src/vk_capture.cpp
${CMAKE_SOURCE_DIR}/hook/src/debug_log.cpp)
target_include_directories(vk_capture_perf_test PRIVATE target_include_directories(vk_capture_perf_test PRIVATE
${CMAKE_SOURCE_DIR}/hook/src ${CMAKE_SOURCE_DIR}/hook/src
${CMAKE_SOURCE_DIR}/third_party/Vulkan-Headers/include) ${CMAKE_SOURCE_DIR}/third_party/Vulkan-Headers/include)

3
vk_layer/CMakeLists.txt
View File

@@ -7,7 +7,8 @@ coop_require_submodule("Vulkan-Headers" "third_party/Vulkan-Headers/include/vulk
add_library(coop_vk_layer SHARED add_library(coop_vk_layer SHARED
coop_vk_layer.cpp coop_vk_layer.cpp
${CMAKE_SOURCE_DIR}/hook/src/vk_capture.cpp) # shared off-present-thread read-back (also used by vk_hook) ${CMAKE_SOURCE_DIR}/hook/src/vk_capture.cpp # shared off-present-thread read-back (also used by vk_hook)
${CMAKE_SOURCE_DIR}/hook/src/debug_log.cpp) # logging used by the shared capture code
target_include_directories(coop_vk_layer PRIVATE target_include_directories(coop_vk_layer PRIVATE
${CMAKE_SOURCE_DIR}/hook/src # ipc_client.hpp (reused IPC client) ${CMAKE_SOURCE_DIR}/hook/src # ipc_client.hpp (reused IPC client)