BlackMark/CoopAllTheThings
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix 32-bit game crash: call hooked __stdcall functions with stdcall()

Browse Source 
SafetyHook's InlineHook::call() invokes the trampoline through a __cdecl
pointer (the compiler default on x86). The functions we hook are __stdcall
(IDXGISwapChain::Present/Present1, the WASAPI render interfaces, and the
WINAPI SwapBuffers/wglSwapBuffers), so on 32-bit both sides cleaned the
stack -> ESP imbalance -> Run-Time Check Failure #0 and an instant crash.
On x64 every convention collapses to one, so it only bit 32-bit games:
Slaps and Beans (Unity/Rewired, 32-bit D3D11) froze the moment the Present
hook ran. The user's "crashes as soon as a button is pressed" was the
Present, not the button.

Switch every __stdcall trampoline call to SafetyHook's stdcall() (a no-op
on x64). The XInput/focus hooks were unaffected because they never call
the trampoline -- they return synthesized data.

Reproduction + regression coverage:
- tools/input_probe (coop_input_probe): injects, reports a connected pad,
  toggles a button, and takes a disable_mask to bisect which subsystem
  affects a game. Isolated the freeze to the video subsystem live.
- hook_selftest_x86 + present_hook_test_x86: the x86 sub-build now builds
  and runs these (the x64 present_hook_test can't see a one-convention
  bug). present_hook_test_x86 drives a real swapchain through the
  trampoline -- it would hit RTC #0 before this fix.
- hook_selftest strengthened to exercise every loaded xinput DLL's full
  export set (GetState, ordinal-100 GetStateEx, GetCapabilities, rumble
  SetState) and to dump the SharedBlock layout.
- protocol.hpp: static_asserts lock the cross-bitness front-of-block
  offsets (verified byte-identical on x86 and x64).

README roadmap trimmed (this milestone done) and a lessons-learned note
added on the call()/stdcall() convention trap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
BlackMark
2026-06-20 20:07:17 +02:00
parent 9977ad5d5a
commit 435ab9d30f
9 changed files with 528 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files

276
README.md
View File

@@ -15,17 +15,30 @@ window that is a live copy of the game's video + audio, and forwards the guests'
input back into the real game. Steam's RPT captures the mirror window — so any
XInput game becomes Remote-Play-Together-able.
The end-to-end path is working: launched under a donor appid, the host streams a
live video + audio mirror of a separately-running game over Remote Play Together
and forwards guest controllers back into it.
## Architecture
| Concern | Mechanism | Component | Status |
| --- | --- | --- | --- |
| Receive guest input | XInput (RPT delivers guest pads to the focused window); optional, opt-in Steam Input when built with the Steamworks SDK | `coop_host.exe` | done |
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` | done |
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | done |
| Mirror video | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | done |
| Mirror video (alt) | Injected Present-hook copies the DXGI backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` | done |
| Mirror audio | Injected render-hook copies the game's WASAPI frames into a shared ring and silences the game locally (no echo); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` | done |
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel) | `common/` | done |
| Concern | Mechanism | Component |
| --- | --- | --- |
| Receive guest input | XInput (RPT delivers guest pads to the focused window); optional, opt-in Steam Input when built with the Steamworks SDK | `coop_host.exe` |
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` |
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` |
| Mirror video (default) | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` |
| Mirror video (hooked) | Injected Present / OpenGL hook copies the backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` |
| Mirror audio | Injected render-hook copies the game's WASAPI frames into a shared ring and silences the game locally (no echo); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` |
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel, video/audio/log shares) | `common/` |
The hooked video path has two producers: **Direct3D (DXGI)** hooks
`IDXGISwapChain::Present` / `Present1` and copies the backbuffer (D3D10/11 games
whose backbuffer is an `ID3D11Texture2D`); **OpenGL** hooks
`SwapBuffers` / `wglSwapBuffers` and reads the backbuffer with `glReadPixels` (for
games that never touch DXGI, e.g. Phantom Brave). The host samples the copy as
plain UNORM (`srgb_to_unorm`) so `*_SRGB`-backbuffer games mirror at correct
brightness. **WGC remains the default** and covers anything the hooked path
doesn't (Vulkan, D3D9, DX12 — see Roadmap).
## Limitations
@@ -43,150 +56,72 @@ XInput game becomes Remote-Play-Together-able.
detects it (`IsWow64Process2`) and shells out to the helper to load the x86 DLL
(a 64-bit process can't cleanly inject a 32-bit one). The shared-memory IPC is
fixed-width / bitness-stable, so the x64 host and x86 hook interoperate.
- **Local audio echo (fixed via the hook; falls back otherwise):** when the
render-hook is active it silences the game's local playback while mirroring it,
so there is no echo. If the hook can't attach or the game uses an
unhooked/exotic render path, the host automatically falls back to
process-loopback capture, which does *not* mute the game — so on the fallback
path the local machine still hears the audio twice (guests hear it once). The
Audio panel shows which path is active.
- **Debug-oriented UI:** the ImGui overlay is always visible and laid out for
diagnosing the pipeline, not for end use. It can't yet be toggled off.
## Status
All phases below are implemented and verified.
- **Phase 0 — donor spike. ✅** Confirmed Steam RPT streams an arbitrary
borderless window under a donor appid and routes guest gamepads into it as
XInput (correct slot assignment). This is the premise the whole tool rests on.
- **Phase 1a — input forwarding + focus spoofing. ✅** The host injects
`coop_hook.dll`; the DLL hooks XInput (SafetyHook) so the game reads the
forwarded pad state and *only* that state, and spoofs focus so the game keeps
running while the host holds the real OS focus. A hook→host status
back-channel reports attach state and poll rate.
- **Phase 1b — video mirror (WGC). ✅** The host captures the injected game's
window with Windows Graphics Capture and draws it letterboxed as its
background, so RPT streams a live mirror.
- **Phase 2 — audio mirror. ✅** The host captures the game's audio by PID via
WASAPI process loopback and re-renders it on the default endpoint, so RPT
carries game audio to guests.
- **Audio render-hook (echo fix). ✅ (capture proven on a real game; full RPT
pass pending)** The injected hook intercepts the game's WASAPI render path
(`IAudioRenderClient`), copies the frames into a shared audio ring for the host
to re-render, and releases the game's buffer silenced — so the operator no
longer hears the audio twice. It hooks the render vtables *proactively* so a
game that's already playing when injected is still captured. The host owns an
enable flag and re-renders the game's format via `AUTOCONVERTPCM`; if the hook
doesn't publish a format in time it reverts to process loopback. The Audio panel
shows the active source and a render-stream-count debug table. Validated
in-process by `audio_hook_test` and against a real already-playing game
(Phantom Brave) with `coop_audio_probe`: the pre-existing render client is
detected and real, non-silent audio reaches the ring with zero overruns.
### Remaining verification
The full-app capture path now works: the hook publishes the captured format to the
ring whenever the host attaches it, so toggling **Mirror game audio** switches to
**Source: Hooked (no echo)** instead of falling back to loopback (verified with
`coop_audio_probe` reproducing the app's inject-then-create-ring ordering against
Phantom Brave). These still need a human / full setup to confirm:
- **Hooked audio over RPT, end-to-end.** Run the host, inject, tick **Mirror game
audio**, and confirm **Source: Hooked (no echo)**, the game goes locally silent,
and a guest on Remote Play Together still hears it.
- **Format guess for non-mix-format games.** For a stream that already exists at
injection time the hook can't see the game's `Initialize`, so it assumes the
device **mix format**. Phantom Brave matched it exactly (48 kHz/2ch/float). A
game that initialized shared mode with a different format would come out
wrong-pitched/garbled and needs per-stream format detection — not yet handled.
- **Multi-stream games.** v1 captures only the first ("primary") render stream;
confirm the Audio panel's render-stream table makes a multi-stream game obvious
(secondary streams stay local until per-stream mixing is added).
- **Local audio echo on the fallback path:** when the render-hook is active it
silences the game's local playback while mirroring it, so there is no echo. If
the hook can't attach or the game uses an unhooked render path, the host falls
back to process-loopback capture, which does *not* mute the game — so the local
machine hears the audio twice (guests hear it once). The Audio panel shows which
path is active.
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
for RPT.
## Roadmap
Done:
### Planned (next up)
- **Usable overlay. ✅** F1 hides the whole overlay so the window is a clean
mirror for Remote Play Together; a fading hint shows the way back. The
pipelines keep running while hidden.
- **Generalized UI. ✅** A top menu bar carries a consolidated FPS / frame-time
(with jitter) readout and a **View** menu that toggles each panel and a global
**Debug details** switch. Panels default to general-purpose status (attached,
forwarding, controller polling rate, mirror resolution, audio source + buffered
ms) and reveal the verbose diagnostics (per-slot poll table, focus-API counts,
input-path detection, per-render-stream table) only when Debug details is on.
- **Installed-hooks list. ✅** The DLL keeps a registry of every individual hook
it installs (XInput, focus, audio), with a running call counter per hook,
reported over IPC. The Injection panel shows it grouped by subsystem
(input / focus / audio) so you can see exactly what's hooked and how busy each
hook is. `coop_audio_probe` prints the same table headless.
- **Per-subsystem hook control. ✅** The three injectable subsystems — input
forwarding (XInput), focus spoof, and the audio render-hook — are independently
controllable. The Injection panel has a checkbox per subsystem that
installs/removes its hooks at runtime over a host→hook control channel; the hook
reconciles each tick. Dependent features are guarded: the synthetic-input
control is disabled when input forwarding is off, and the Audio panel notes when
the render-hook is off (mirroring then uses loopback). Defaults to all-on so
behavior is unchanged unless you toggle something.
- **In-app Log window. ✅** The injected DLL streams its log lines to the host
over a shared log ring (`coop_log_<pid>`, a lossy multi-producer ring), and the
host shows them in a **Log** window with auto-scroll, a text filter, and clear.
Replaces tailing `%TEMP%\coop_hook.log` (which stays as an opt-in file mirror).
`coop_audio_probe` drains and prints the same stream headless.
The current focus is making specific games work end-to-end. Each item is a
milestone with its own tests and commit.
- **Hooked video path (Present + OpenGL). ✅** The injected DLL captures the
game's frames into a shared keyed-mutex texture (`coop_video_<pid>`) that the
host opens by name and samples — a lower-latency, border-free alternative to WGC.
It's an opt-in subsystem (the **Video Present-hook** checkbox in the Injection
panel, or just pick **Source: Hooked (Present)** in the Video mirror panel, which
installs it). Two producers cover the common graphics APIs:
- **Direct3D (DXGI):** hooks `IDXGISwapChain::Present` / `Present1` and copies
the backbuffer. Catches D3D10/11/12 games whose backbuffer is an
`ID3D11Texture2D` (the common D3D11 case). Validated by `present_hook_test`
and against Slaps and Beans (32-bit D3D11) and Life is Strange: Before the
Storm. The host samples the copy as plain UNORM (see `srgb_to_unorm`) so games
with an `*_SRGB` backbuffer (Life is Strange is `R8G8B8A8_UNORM_SRGB`) mirror
with correct brightness instead of being darkened by an sRGB→linear decode.
- **OpenGL:** hooks `SwapBuffers` / `wglSwapBuffers`, reads the backbuffer with
`glReadPixels`, and uploads it into the shared texture. Catches OpenGL games
that never touch DXGI (e.g. **Phantom Brave**, which is OpenGL — that's why
the Present hook alone showed no image). Validated by `opengl_hook_test` and
against Phantom Brave. `glReadPixels` forces a per-frame GPU→CPU readback, so
it's heavier than the D3D copy, but fine for the typically-2D OpenGL titles.
1. **Release the mouse cursor for cursor-clipping games.** Games that confine the
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` /
per-frame `SetCursorPos` re-centering) trap the operator's mouse permanently,
because the focus spoof makes the game believe it's always focused — so the
operator can't reach the ImGui overlay. Add a cursor-release capability to the
Focus subsystem: hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the
game's clip) and the re-centering `SetCursorPos`, gated by a new host→hook flag
driven by a host toggle + hotkey. Defaults to released (the guest plays via the
pad, so the game's own cursor clip is operator-only), with the option to
re-enable clipping per game.
**Vulkan** games (present via `vkQueuePresentKHR`) aren't hooked yet — that needs
a Vulkan layer / device-dispatch hook plus a `vkCmdCopyImage` to a readable
image, a larger effort. **Use WGC** (the default Video source) for Vulkan, D3D9,
or anything the hooked path doesn't capture — WGC works for any window.
2. **Real capture metrics + latency stats.** The current FPS readout only measures
how fast the host renders its own window, which hides capture stutter. Add a
three-line frametime/FPS graph — **game present rate** (from `VideoShare`
present deltas), **capture rate** (generation deltas / WGC arrivals), and
**tool render rate** — plus a **capture→display latency** stat: stamp each
published frame with a `QueryPerformanceCounter` value in `VideoShare`, and the
host reports `host-present QPC − game-present QPC` (min/avg/max ms) for the
matched frame. QPC is system-wide, so the two processes' timestamps compare
directly.
- **x86 (32-bit) game support. ✅** A nested Win32 sub-build (CMake
`ExternalProject`, driven from the normal x64 build) produces `coop_hook_x86.dll`
and a 32-bit `coop_inject_x86.exe`, staged next to the x64 binaries. The host
detects a WOW64 target with `IsWow64Process2` and spawns the helper to inject
the x86 DLL. Validated end-to-end against Slaps and Beans (a 32-bit D3D11 game):
all subsystems hooked, and the IPC channels (status, audio ring, video share,
log) all flow across the x64↔x86 boundary.
3. **DX12 hooked capture.** Marvel's Spider-Man is D3D12, so the Present hook fires
but `GetBuffer(0)` as `ID3D11Texture2D` fails (the backbuffer is an
`ID3D12Resource`) and the hook idles; WGC works but stutters. Add a D3D12 path
via a **D3D11On12 bridge**: capture the game's D3D12 command queue (hook
`ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
- **Steam Input (opt-in). ✅** When the host is built with the Steamworks SDK
(auto-detected under `third_party/steamworks_sdk/`), a **Use Steam Input**
checkbox in the Controllers panel switches the input backend to the Steam Input
API (action-based) at runtime; it loads a bundled action manifest
(`steam_input_actions.vdf`) via `SetInputActionManifestFilePath`, so it needs no
partner-backend config. **It's off by default and XInput is the primary path**:
merely initializing Steam Input activates Steam's in-process XInput interception,
which *hides* controllers from XInput unless they're bound to our action set for
the running appid — so defaulting to it silently broke input forwarding. Enabling
it is only useful once a controller is bound to Steam Input for the donor appid;
otherwise leave it off and the proven XInput path carries the guest input.
Verified to initialize and enumerate controllers against the live Steam client
(`coop_steam_input_probe`), and that toggling it off restores XInput.
4. **Multi-stream audio capture + mixing.** Games with several concurrent WASAPI
render streams (e.g. Spider-Man) only get their first ("primary") stream
mirrored today; the rest keep playing locally and never reach the guest. Capture
every tracked render stream into its own shared ring, silence each, and add a
host-side mixer that resamples each ring to the render format and sums them
(with soft-clip).
All planned phases are now implemented. Possible later work: per-stream audio
mixing for multi-stream games, per-stream format detection for the audio hook, and
hooking the remaining present paths for the video hook (Vulkan `vkQueuePresentKHR`,
D3D9, pure-D3D12) — WGC already covers those today.
### Future work
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
them needs a Vulkan layer / device-dispatch hook plus a `vkCmdCopyImage` to a
readable image. Use WGC in the meantime.
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
hook would be the lower-latency upgrade.
- **Per-stream audio format detection.** A render stream that already exists when
we inject is never seen at `Initialize`, so the hook assumes the device **mix
format**. A stream initialized in shared mode at a different format would come
out wrong-pitched. Detecting the real per-stream format would remove that guess.
- **Rumble / haptics forwarding.** `XInputSetState` is currently swallowed; routing
it back to the guest is a later phase.
## Building
@@ -217,6 +152,12 @@ it's absent the host builds XInput-only (no other features depend on it). When
present, the build links `steam_api64.lib`, stages `steam_api64.dll` and the
action manifest next to the host, and also builds `coop_steam_input_probe`.
Steam Input is **off by default and XInput is the primary path**: merely
initializing Steam Input activates Steam's in-process XInput interception, which
hides controllers from XInput unless they're bound to our action set for the
running appid. Enable it (Controllers panel → **Use Steam Input**) only once a
controller is bound to Steam Input for the donor appid.
### clangd / IDE setup
The Visual Studio CMake generator does **not** emit `compile_commands.json`, so
@@ -239,8 +180,8 @@ ctest --test-dir build -C Debug --output-on-failure
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs
the hooks, renders a tone through WASAPI in the same process, and asserts the
COM vtables were discovered, the frames reached the ring (non-silent), the
primary stream was silenced, and exactly one render stream was counted. Skips
cleanly if the machine has no audio endpoint.
primary stream was silenced, and the render stream was counted. Skips cleanly if
the machine has no audio endpoint.
- **`srgb_format_test`** — unit test of the `srgb_to_unorm` mapping the hooked
video path uses so `*_SRGB`-backbuffer games aren't darkened. No device.
- **`opengl_hook_test`** — in-process self-test of the OpenGL capture path:
@@ -259,16 +200,27 @@ ctest --test-dir build -C Debug --output-on-failure
shipping process-loopback capture (the fallback path) receives its audio by
PID. Skips cleanly if the machine has no audio endpoint.
### Debugging the render-hook against a real game
### Debugging the hooks against a real game
[`tools/audio_probe`](tools/audio_probe) (`coop_audio_probe.exe <pid> [seconds]`)
brings up the audio render-hook without Steam / RPT / the host UI: it creates the
IPC block + audio ring the hook expects, injects `coop_hook.dll` into the target
game, then drains the ring and prints per-stream format, captured-frame counts,
peak amplitude (proves the audio is real, not silence), and overruns. It enables
the hook's file trace (`%TEMP%\coop_hook.log`) for the run. Run it from
`bin/<config>/`. **Kill the game between runs** — the loaded DLL locks
`coop_hook.dll` against the next rebuild.
the hook's file trace (`%TEMP%\coop_hook.log`) for the run.
[`tools/input_probe`](tools/input_probe)
(`coop_input_probe.exe <pid> [seconds] [disable_mask]`) does the same for input: it
injects, reports one connected pad, and toggles a button each second so the game's
input layer sees a real state change. `disable_mask` (hex bits `0x1`=input
`0x2`=focus `0x4`=audio `0x8`=video) skips installing a subsystem, so you can
**bisect which injected subsystem affects a game** — this is how the 32-bit
Present-hook crash was isolated.
Both auto-detect a 32-bit (WOW64) target and inject via `coop_inject_x86.exe` +
`coop_hook_x86.dll`, exactly like the host. Run them from `bin/<config>/`.
**Kill the game between runs** — the loaded DLL locks `coop_hook.dll` against the
next rebuild.
## Running the tool (manual, end-to-end)
@@ -294,16 +246,16 @@ person/account to receive the stream.
3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** —
the host window now shows a live, letterboxed copy of the game. **Source**
picks how the frames are grabbed: **WGC** (default, Windows Graphics Capture —
works for any window) or **Hooked (Present)** (the injected Present-hook's
shared texture — lower latency and no capture border, but only for DXGI /
D3D11 games; selecting it installs the video subsystem in the game).
works for any window) or **Hooked (Present)** (the injected hook's shared
texture — lower latency and no capture border, for DXGI / D3D11 and OpenGL
games; selecting it installs the video subsystem in the game).
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
With the hook injected, **Source** shows **Hooked (no echo)** and the game's
local playback goes silent while guests still hear it. If it shows **Loopback
(echo)** the hook's render path wasn't caught and you'll hear the game twice
locally (guests still hear it once). The **Render streams** table shows how
many WASAPI streams the game emits (v1 mirrors the first/primary).
many WASAPI streams the game emits.
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
sees the mirrored video, hears the audio, and that their controller drives the
@@ -349,3 +301,19 @@ Non-obvious things that cost time and constrain the design:
not 13 — `SetEventHandle` (13) sits between `Reset` and `GetService`. Count the
full interface (including every inherited `IUnknown`/base method) when adding a
new COM hook.
- **SafetyHook's `call()` is `__cdecl` on x86 — use `stdcall()` for `__stdcall`
targets.** `InlineHook::call()` invokes the trampoline through a pointer with the
compiler's default convention, which is `__cdecl` on 32-bit. Most things we hook
are `__stdcall` (COM methods like `IDXGISwapChain::Present` and the WASAPI render
interfaces, plus `WINAPI` `SwapBuffers`). On x64 every convention collapses to one,
so `call()` is fine; on x86 it double-cleans the stack → ESP imbalance → an
instant crash (Debug builds surface it as **Run-Time Check Failure #0**). This
froze 32-bit games (Slaps and Beans) the moment the Present hook ran. Always call
trampolines with the matching convention — `stdcall()` for these — which is a
no-op on x64. The XInput/focus hooks dodged it only because they never call the
trampoline (they return synthesized data).
- **Steam Input init suppresses XInput.** Initializing the Steam Input API turns on
Steam's in-process XInput interception, which hides controllers from
`XInputGetState` unless they're bound to the running appid's action set —
defaulting to it silently broke input forwarding. XInput is the primary path;
Steam Input is opt-in.