BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix 32-bit FMOD audio crash: hook WASAPI COM methods via vtable swap

Browse Source 
The stdcall() fix stopped the Present-hook crash but 32-bit games (Slaps
and Beans, FMOD) still crashed the instant audio init ran through the
hook. Root cause: SafetyHook's inline hook relocates the target's
overwritten prologue into a trampoline, but MMDevApi/AudioSes COM methods
on x86 open with `push ebp; mov ebp,esp; and esp,-8` (dynamic stack
alignment) and read arguments EBP-relative. The relocated copy leaves EBP
wrong, so the original runs with garbage arguments and faults (AV writing
*ppInterface inside CEndpointDevice::Activate+0x3d).

Switch all five WASAPI COM hooks (IMMDevice::Activate, IAudioClient::
Initialize/GetService, IAudioRenderClient::GetBuffer/ReleaseBuffer) from
safetyhook::create_inline to a small VtableHook helper: VirtualProtect the
shared vtable slot, overwrite the function pointer, call the saved original
directly. No code patching, no trampoline, pristine stack regardless of
prologue. One swap covers every instance (a coclass shares one vtable), so
the existing shared-vtable strategy is preserved. Inline hooking stays for
Present/SwapBuffers, whose prologues relocate cleanly.

Reproduced in-process with a new x86 build of the audio render-hook test
(audio_hook_test_x86): it installs the hooks, then drives a fresh
IAudioClient through them and renders -- segfaulted before, passes now.
The x64 audio_hook_test passes regardless of the bug, so the 32-bit build
is the regression guard.

ctest: x64 7/7, x86 3/3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-20 20:39:29 +02:00
parent 435ab9d30f
commit 4985239222
3 changed files with 151 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
CMakeLists.txt
View File

@@ -69,6 +69,23 @@ if(COOP_X86_HELPER_BUILD)
target_include_directories(present_hook_test_x86 PRIVATE hook/src)
target_link_libraries(present_hook_test_x86 PRIVATE coop_common safetyhook::safetyhook d3d11 dxgi)
add_test(NAME present_hook_test_x86 COMMAND present_hook_test_x86)
# x86 build of the audio render-hook test. It installs the WASAPI hooks and then
# creates a *fresh* IAudioClient/IAudioRenderClient that calls Initialize /
# GetService / GetBuffer / ReleaseBuffer back through the SafetyHook trampolines
# -- the exact ordering that crashed 32-bit Slaps & Beans (FMOD) when audio hooks
# were live as the game initialized its output. The x64 audio_hook_test passes,
# so this 32-bit build is the regression guard for any x86-only fault in the
# audio setup detours / trampoline relocation of AudioSes.dll prologues.
add_executable(audio_hook_test_x86
tests/audio_hook_test.cpp
hook/src/audio_hook.cpp
hook/src/debug_log.cpp
hook/src/hook_registry.cpp)
target_include_directories(audio_hook_test_x86 PRIVATE hook/src)
target_compile_definitions(audio_hook_test_x86 PRIVATE NTDDI_VERSION=0x0A00000B)
target_link_libraries(audio_hook_test_x86 PRIVATE coop_common safetyhook::safetyhook ole32 mmdevapi)
add_test(NAME audio_hook_test_x86 COMMAND audio_hook_test_x86)
return()
endif()