Fix 32-bit game crash: call hooked __stdcall functions with stdcall()
SafetyHook's InlineHook::call() invokes the trampoline through a __cdecl pointer (the compiler default on x86). The functions we hook are __stdcall (IDXGISwapChain::Present/Present1, the WASAPI render interfaces, and the WINAPI SwapBuffers/wglSwapBuffers), so on 32-bit both sides cleaned the stack -> ESP imbalance -> Run-Time Check Failure #0 and an instant crash. On x64 every convention collapses to one, so it only bit 32-bit games: Slaps and Beans (Unity/Rewired, 32-bit D3D11) froze the moment the Present hook ran. The user's "crashes as soon as a button is pressed" was the Present, not the button. Switch every __stdcall trampoline call to SafetyHook's stdcall() (a no-op on x64). The XInput/focus hooks were unaffected because they never call the trampoline -- they return synthesized data. Reproduction + regression coverage: - tools/input_probe (coop_input_probe): injects, reports a connected pad, toggles a button, and takes a disable_mask to bisect which subsystem affects a game. Isolated the freeze to the video subsystem live. - hook_selftest_x86 + present_hook_test_x86: the x86 sub-build now builds and runs these (the x64 present_hook_test can't see a one-convention bug). present_hook_test_x86 drives a real swapchain through the trampoline -- it would hit RTC #0 before this fix. - hook_selftest strengthened to exercise every loaded xinput DLL's full export set (GetState, ordinal-100 GetStateEx, GetCapabilities, rumble SetState) and to dump the SharedBlock layout. - protocol.hpp: static_asserts lock the cross-bitness front-of-block offsets (verified byte-identical on x86 and x64). README roadmap trimmed (this milestone done) and a lessons-learned note added on the call()/stdcall() convention trap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -43,6 +43,32 @@ if(COOP_X86_HELPER_BUILD)
|
||||
add_subdirectory(third_party/safetyhook)
|
||||
add_subdirectory(hook)
|
||||
add_subdirectory(tools/inject_helper)
|
||||
|
||||
# x86 reproduction of the XInput self-test. The x64 hook_selftest passes, so an
|
||||
# x86 build of the same logic is how we reproduce (and then guard against) the
|
||||
# 32-bit input-forwarding crash on real games like Slaps and Beans.
|
||||
enable_testing()
|
||||
add_executable(hook_selftest_x86
|
||||
tests/hook_selftest.cpp
|
||||
hook/src/xinput_hook.cpp
|
||||
hook/src/hook_registry.cpp)
|
||||
target_include_directories(hook_selftest_x86 PRIVATE hook/src)
|
||||
target_link_libraries(hook_selftest_x86 PRIVATE coop_common safetyhook::safetyhook xinput)
|
||||
add_test(NAME hook_selftest_x86 COMMAND hook_selftest_x86)
|
||||
|
||||
# x86 build of the Present-hook test. It drives a real swapchain through the
|
||||
# SafetyHook trampoline, so it catches the x86-only calling-convention bug that
|
||||
# froze 32-bit games (SafetyHook's call() is __cdecl; Present is __stdcall ->
|
||||
# ESP imbalance / Run-Time Check Failure #0). The x64 present_hook_test can't see
|
||||
# it (one calling convention), so this 32-bit build is the regression guard.
|
||||
add_executable(present_hook_test_x86
|
||||
tests/present_hook_test.cpp
|
||||
hook/src/present_hook.cpp
|
||||
hook/src/debug_log.cpp
|
||||
hook/src/hook_registry.cpp)
|
||||
target_include_directories(present_hook_test_x86 PRIVATE hook/src)
|
||||
target_link_libraries(present_hook_test_x86 PRIVATE coop_common safetyhook::safetyhook d3d11 dxgi)
|
||||
add_test(NAME present_hook_test_x86 COMMAND present_hook_test_x86)
|
||||
return()
|
||||
endif()
|
||||
|
||||
@@ -63,6 +89,7 @@ if(COOP_BUILD_HOOK)
|
||||
enable_testing()
|
||||
add_subdirectory(tools/audio_tone) # coop_tone: audio source for the loopback test
|
||||
add_subdirectory(tools/audio_probe) # coop_audio_probe: inject + diagnose the render-hook
|
||||
add_subdirectory(tools/input_probe) # coop_input_probe: inject + forward synthetic input
|
||||
add_subdirectory(tests)
|
||||
endif()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user