BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix 32-bit game crash: call hooked __stdcall functions with stdcall()

Browse Source 
SafetyHook's InlineHook::call() invokes the trampoline through a __cdecl
pointer (the compiler default on x86). The functions we hook are __stdcall
(IDXGISwapChain::Present/Present1, the WASAPI render interfaces, and the
WINAPI SwapBuffers/wglSwapBuffers), so on 32-bit both sides cleaned the
stack -> ESP imbalance -> Run-Time Check Failure #0 and an instant crash.
On x64 every convention collapses to one, so it only bit 32-bit games:
Slaps and Beans (Unity/Rewired, 32-bit D3D11) froze the moment the Present
hook ran. The user's "crashes as soon as a button is pressed" was the
Present, not the button.

Switch every __stdcall trampoline call to SafetyHook's stdcall() (a no-op
on x64). The XInput/focus hooks were unaffected because they never call
the trampoline -- they return synthesized data.

Reproduction + regression coverage:
- tools/input_probe (coop_input_probe): injects, reports a connected pad,
  toggles a button, and takes a disable_mask to bisect which subsystem
  affects a game. Isolated the freeze to the video subsystem live.
- hook_selftest_x86 + present_hook_test_x86: the x86 sub-build now builds
  and runs these (the x64 present_hook_test can't see a one-convention
  bug). present_hook_test_x86 drives a real swapchain through the
  trampoline -- it would hit RTC #0 before this fix.
- hook_selftest strengthened to exercise every loaded xinput DLL's full
  export set (GetState, ordinal-100 GetStateEx, GetCapabilities, rumble
  SetState) and to dump the SharedBlock layout.
- protocol.hpp: static_asserts lock the cross-bitness front-of-block
  offsets (verified byte-identical on x86 and x64).

README roadmap trimmed (this milestone done) and a lessons-learned note
added on the call()/stdcall() convention trap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-20 20:07:17 +02:00
parent 9977ad5d5a
commit 435ab9d30f
9 changed files with 528 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
tests/hook_selftest.cpp
View File

@@ -2,6 +2,7 @@
// SafetyHook XInput interception. No injection or physical controller needed --
// this process plays both host and game. Exits 0 on pass, 1 on failure.
#include <cstddef>
#include <cstdio>
#include <windows.h>
@@ -33,8 +34,75 @@ void check(bool ok, const char* what)
} // namespace
// Exercises every export the game touches on one loaded xinput DLL, so the inline
// hook + trampoline over *that DLL's* real prologue is actually called -- not just
// installed. Older games load older variants (e.g. xinput1_3.dll) whose export
// prologues differ, which is where a 32-bit trampoline-relocation fault hides.
void exercise_dll(const wchar_t* dll_name)
{
HMODULE m = GetModuleHandleW(dll_name);
if (m == nullptr)
{
return; // not loaded on this machine; nothing to exercise
}
char tag[96];
using GetState_t = DWORD(WINAPI*)(DWORD, XINPUT_STATE*);
using SetState_t = DWORD(WINAPI*)(DWORD, XINPUT_VIBRATION*);
using GetCaps_t = DWORD(WINAPI*)(DWORD, DWORD, XINPUT_CAPABILITIES*);
auto get_state = reinterpret_cast<GetState_t>(GetProcAddress(m, "XInputGetState"));
auto get_state_ex = reinterpret_cast<GetState_t>(GetProcAddress(m, MAKEINTRESOURCEA(100)));
auto get_caps = reinterpret_cast<GetCaps_t>(GetProcAddress(m, "XInputGetCapabilities"));
auto set_state = reinterpret_cast<SetState_t>(GetProcAddress(m, "XInputSetState"));
if (get_state != nullptr)
{
XINPUT_STATE s = {};
std::snprintf(tag, sizeof(tag), "%ls XInputGetState forwards state", dll_name);
check(get_state(0, &s) == ERROR_SUCCESS && s.dwPacketNumber == 7, tag);
}
if (get_state_ex != nullptr)
{
XINPUT_STATE s = {};
std::snprintf(tag, sizeof(tag), "%ls XInputGetStateEx (ord 100) forwards state", dll_name);
check(get_state_ex(0, &s) == ERROR_SUCCESS && s.dwPacketNumber == 7, tag);
}
if (get_caps != nullptr)
{
XINPUT_CAPABILITIES c = {};
std::snprintf(tag, sizeof(tag), "%ls XInputGetCapabilities reports gamepad", dll_name);
check(get_caps(0, 0, &c) == ERROR_SUCCESS && c.Type == XINPUT_DEVTYPE_GAMEPAD, tag);
}
if (set_state != nullptr)
{
// A game commonly rumbles in response to a button press; this is the call
// path "crashes as soon as a button is pressed" pointed at.
XINPUT_VIBRATION v = {};
v.wLeftMotorSpeed = 0x8000;
v.wRightMotorSpeed = 0x4000;
std::snprintf(tag, sizeof(tag), "%ls XInputSetState (rumble) accepted, no crash", dll_name);
check(set_state(0, &v) == ERROR_SUCCESS, tag);
}
}
void dump_layout()
{
std::printf("LAYOUT sizeof(SharedBlock)=%zu CoopPadState=%zu\n", sizeof(SharedBlock), sizeof(CoopPadState));
std::printf("LAYOUT off pads=%zu sequence=%zu status=%zu control=%zu video=%zu\n",
offsetof(SharedBlock, pads), offsetof(SharedBlock, sequence), offsetof(SharedBlock, status),
offsetof(SharedBlock, control), offsetof(SharedBlock, video));
std::printf("LAYOUT HookStatus sizeof=%zu get_state_calls=%zu attached=%zu audio_streams=%zu hook_entries=%zu\n",
sizeof(HookStatus), offsetof(HookStatus, get_state_calls), offsetof(HookStatus, attached),
offsetof(HookStatus, audio_streams), offsetof(HookStatus, hook_entries));
std::printf("LAYOUT VideoShare sizeof=%zu present_calls=%zu HookControl sizeof=%zu\n", sizeof(VideoShare),
offsetof(VideoShare, present_calls), sizeof(HookControl));
}
int main()
{
dump_layout();
// --- Host side: create the section (named by our pid) and publish a pad. ---
SharedMemory shm;
if (!shm.create(shared_memory_name(GetCurrentProcessId()), sizeof(SharedBlock)))
@@ -56,6 +124,17 @@ int main()
pads[0].thumb_ry = -4321;
publish_pads(*block, pads, kMaxPads);
// Load every xinput variant *before* installing hooks, so install_xinput_hooks
// (which only hooks already-loaded modules) covers all of them and the matrix
// below exercises each DLL's real export prologue under SafetyHook. A real game
// loads exactly one, but which one varies by game age -- and the 32-bit crash
// only reproduces over the specific DLL the game uses.
const wchar_t* xinput_modules[] = {L"xinput1_4.dll", L"xinput1_3.dll", L"xinput9_1_0.dll", L"xinputuap.dll"};
for (const wchar_t* name : xinput_modules)
{
LoadLibraryW(name); // best-effort; absent variants stay unloaded
}
// --- Hook side: connect and install over this process's own xinput. ---
hook::IpcClient ipc;
check(ipc.connect(10, 5), "IPC client connect");
@@ -77,6 +156,14 @@ int main()
check(XInputGetCapabilities(0, 0, &caps) == ERROR_SUCCESS, "slot 0 capabilities reported");
check(caps.Type == XINPUT_DEVTYPE_GAMEPAD, "capability device type");
// Now drive every loaded variant's full export set (GetState, ordinal-100
// GetStateEx, GetCapabilities, and the rumble SetState a game calls on a button
// press) so each DLL's hooked prologue/trampoline is actually run.
for (const wchar_t* name : xinput_modules)
{
exercise_dll(name);
}
// Status back-channel: the host relies on these to prove the hook is live.
check(block->status.attached == 1, "status reports attached");
check(block->status.get_state_calls[0].load(std::memory_order_relaxed) >= 1, "status counts slot 0 GetState");