BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix 32-bit FMOD audio crash: hook WASAPI COM methods via vtable swap

Browse Source 
The stdcall() fix stopped the Present-hook crash but 32-bit games (Slaps
and Beans, FMOD) still crashed the instant audio init ran through the
hook. Root cause: SafetyHook's inline hook relocates the target's
overwritten prologue into a trampoline, but MMDevApi/AudioSes COM methods
on x86 open with `push ebp; mov ebp,esp; and esp,-8` (dynamic stack
alignment) and read arguments EBP-relative. The relocated copy leaves EBP
wrong, so the original runs with garbage arguments and faults (AV writing
*ppInterface inside CEndpointDevice::Activate+0x3d).

Switch all five WASAPI COM hooks (IMMDevice::Activate, IAudioClient::
Initialize/GetService, IAudioRenderClient::GetBuffer/ReleaseBuffer) from
safetyhook::create_inline to a small VtableHook helper: VirtualProtect the
shared vtable slot, overwrite the function pointer, call the saved original
directly. No code patching, no trampoline, pristine stack regardless of
prologue. One swap covers every instance (a coclass shares one vtable), so
the existing shared-vtable strategy is preserved. Inline hooking stays for
Present/SwapBuffers, whose prologues relocate cleanly.

Reproduced in-process with a new x86 build of the audio render-hook test
(audio_hook_test_x86): it installs the hooks, then drives a fresh
IAudioClient through them and renders -- segfaulted before, passes now.
The x64 audio_hook_test passes regardless of the bug, so the 32-bit build
is the regression guard.

ctest: x64 7/7, x86 3/3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-20 20:39:29 +02:00
parent 435ab9d30f
commit 4985239222
3 changed files with 151 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
README.md
View File

@@ -312,6 +312,19 @@ Non-obvious things that cost time and constrain the design:
trampolines with the matching convention — `stdcall()` for these — which is a
no-op on x64. The XInput/focus hooks dodged it only because they never call the
trampoline (they return synthesized data).
- **Hook COM methods by swapping the vtable entry, not by inline-patching the
function — on x86.** Inline hooking relocates the target's overwritten prologue
into a trampoline. Some x86 prologues defeat that: MMDevApi/AudioSes methods open
with `push ebp; mov ebp,esp; and esp,-8` (dynamic stack alignment) and read their
arguments **EBP-relative**. SafetyHook's relocated copy leaves EBP wrong, so the
original ran with garbage arguments and faulted — this crashed 32-bit FMOD games
(Slaps and Beans) the instant audio init flowed through the hook, *after* the
`stdcall()` fix above. The robust fix is vtable-entry hooking: `VirtualProtect` the
shared vtable slot, overwrite the function pointer, call the saved original
directly. No code patching, no trampoline, pristine stack regardless of prologue.
The audio hooks use this; inline hooking is fine for `Present`/`SwapBuffers`, whose
prologues relocate cleanly. (One swap covers every instance — a coclass shares one
vtable.) Guarded by `audio_hook_test_x86`.
- **Steam Input init suppresses XInput.** Initializing the Steam Input API turns on
Steam's in-process XInput interception, which hides controllers from
`XInputGetState` unless they're bound to the running appid's action set —