Document the diagnosis and plan for the unreliable hooked-audio format detection: robust rate measurement (longer window, atomic endpoints, consensus, reject non-standard rates), visible + red-flagged loopback fallback with auto-promote, a re-measure button and per-game persisted format overrides via an AudioRingHeader op channel, session-only auto-re-attach on relaunch, and color-coded log levels. Drop the stale "previous backlog is all shipped" note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
429 lines
28 KiB
Markdown
429 lines
28 KiB
Markdown
# CoopAllTheThings
|
||
|
||
Steam **Remote Play Together (RPT)** for any XInput game — without breaking DRM,
|
||
achievements, or playtime.
|
||
|
||
Existing "donor game" tools (e.g. RemotePlayWhatever) copy a target game's files
|
||
into a donor game's folder and rename the executable so Steam streams the target
|
||
under the donor's appid. That breaks DRM-protected games, breaks achievements,
|
||
and credits playtime to the donor.
|
||
|
||
CoopAllTheThings takes a different approach: the **real game runs normally under
|
||
its own appid** (so DRM, achievements, and playtime all work), while a lightweight
|
||
**mirror app runs under the donor appid**. The mirror presents a borderless
|
||
window that is a live copy of the game's video + audio, and forwards the guests'
|
||
input back into the real game. Steam's RPT captures the mirror window — so any
|
||
XInput game becomes Remote-Play-Together-able.
|
||
|
||
The end-to-end path is working: launched under a donor appid, the host streams a
|
||
live video + audio mirror of a separately-running game over Remote Play Together
|
||
and forwards guest controllers back into it.
|
||
|
||
## Architecture
|
||
|
||
| Concern | Mechanism | Component |
|
||
| --- | --- | --- |
|
||
| Receive guest input | XInput (RPT delivers guest pads to the focused window); optional, opt-in Steam Input when built with the Steamworks SDK | `coop_host.exe` |
|
||
| Forward input to game | DLL injection + XInput hook (SafetyHook) — game sees *only* our pad | `coop_hook.dll` |
|
||
| Forward mouse + keyboard | Opt-in MKB subsystem: host streams its window's clicks/keys, the hook posts the matching window messages and synthesizes `GetAsyncKeyState`/`GetKeyboardState`/`GetCursorPos` for polling games | `coop_hook.dll` + `coop_host.exe` |
|
||
| Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` |
|
||
| Mirror video (default) | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` |
|
||
| Mirror video (hooked) | Injected Present / OpenGL hook copies the backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` |
|
||
| Mirror audio | Injected render-hook copies each of the game's WASAPI render streams into its own shared ring and silences the game locally (no echo); the host mixes the streams (soft-clipped); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` |
|
||
| Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel, video/audio/log shares) | `common/` |
|
||
|
||
The hooked video path has two producers: **Direct3D (DXGI)** hooks
|
||
`IDXGISwapChain::Present` / `Present1` and copies the backbuffer — directly for
|
||
D3D10/11 games (the backbuffer is an `ID3D11Texture2D`), and via a **D3D11On12
|
||
bridge** for D3D12 games (wrap the `ID3D12Resource` backbuffer, `CopyResource` into
|
||
the shared texture); **OpenGL** hooks `SwapBuffers` / `wglSwapBuffers` and reads the
|
||
backbuffer with `glReadPixels` (for games that never touch DXGI, e.g. Phantom
|
||
Brave). The host samples the copy as plain UNORM (`srgb_to_unorm`) so
|
||
`*_SRGB`-backbuffer games mirror at correct brightness. **WGC remains the default**
|
||
and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
|
||
|
||
## Limitations
|
||
|
||
- **Anti-cheat:** the input path injects `coop_hook.dll` into the target game.
|
||
Games protected by kernel-level anti-cheat (Easy Anti-Cheat, BattlEye,
|
||
Vanguard, etc.) will detect the injected module and may **kick the player or
|
||
issue a ban**. Such games are explicitly **out of scope and unsupported** — do
|
||
not use CoopAllTheThings with them. The tool targets single-player and
|
||
co-op/local-multiplayer titles without active anti-cheat.
|
||
- **XInput only:** the game must read controllers via XInput (the common case).
|
||
DirectInput-only / RawInput-only games are not handled.
|
||
- **32-bit games supported via a helper:** the host is x64, but the build also
|
||
produces an x86 hook DLL (`coop_hook_x86.dll`) and a 32-bit injector helper
|
||
(`coop_inject_x86.exe`). When the target is a 32-bit (WOW64) process the host
|
||
detects it (`IsWow64Process2`) and shells out to the helper to load the x86 DLL
|
||
(a 64-bit process can't cleanly inject a 32-bit one). The shared-memory IPC is
|
||
fixed-width / bitness-stable, so the x64 host and x86 hook interoperate.
|
||
- **Local audio echo on the fallback path:** when the render-hook is active it
|
||
silences the game's local playback while mirroring it, so there is no echo. If
|
||
the hook can't attach or the game uses an unhooked render path, the host falls
|
||
back to process-loopback capture, which does *not* mute the game — so the local
|
||
machine hears the audio twice (guests hear it once). The Audio panel shows which
|
||
path is active.
|
||
- **Hooked audio can only recover a pre-existing stream's *sample rate*, not its
|
||
channels/bit-depth.** The tool injects into an already-running game, so the audio
|
||
render-hook usually never saw the game's `IAudioClient::Initialize`. It recovers the
|
||
true **sample rate** by measuring the render cadence (so playback pitch is correct,
|
||
e.g. Godot/Brotato's 44100 Hz on a 48000 Hz endpoint), but **channels and bit-depth
|
||
can't be detected** — with `AUTOCONVERTPCM` `GetBuffer` returns a fixed staging
|
||
buffer (no buffer stride to measure) and WASAPI exposes no API for a pre-existing
|
||
client's format — so they're *assumed* to match the device mix format. That's correct
|
||
for the common case (engines render stereo float, matching the endpoint, differing
|
||
only in rate). A game rendering a *different* channel count or bit depth than the
|
||
device would be mirrored with the wrong layout (garbled audio) on the hooked path —
|
||
but never an over-read/crash (a `VirtualQuery` clamp guards the copy), and the
|
||
loopback fallback is always format-correct. The Audio panel shows each stream's
|
||
format provenance (*known* / *measuring* / *measured rate (ch/bits assumed)*) so the
|
||
assumption is visible. Streams created *after* injection are captured exactly.
|
||
- **Debug-oriented UI:** the ImGui overlay is laid out for diagnosing the
|
||
pipeline, not for end use. F1 hides it entirely so the window is a clean mirror
|
||
for RPT.
|
||
|
||
## Roadmap
|
||
|
||
### Planned (next up)
|
||
|
||
**Audio format reliability.** The hooked audio path can mis-detect a pre-existing
|
||
stream's sample rate, or fall back to loopback with no explanation. Two root causes:
|
||
the rate measurement is fragile, and late injection forces the guess path in the
|
||
first place.
|
||
|
||
- *Why the rate is sometimes wrong.* The measurement window is only ~200 ms. WASAPI
|
||
delivers audio in quantized ~10 ms buffers, so one extra buffer at a window edge is
|
||
a ~5% error (e.g. 44100 → ~46205 Hz). The snap-to-standard tolerance is ±2%, so a
|
||
~5%-off value snaps to *nothing* and is published verbatim instead of rejected, and
|
||
it's single-shot (no averaging). Fix: **measure over a longer window (~1–1.5 s),
|
||
sample the (frames, QPC) endpoints atomically, require consensus across a few
|
||
windows, and refuse to publish a rate that doesn't land near a standard rate** —
|
||
keep measuring (or flag low-confidence) rather than committing a bogus value.
|
||
- *Why it falls back to loopback silently.* The host waits a fixed 1 s for the hook
|
||
to publish a format; the guess+measure path needs ~400–700 ms of *continuous* audio
|
||
after the ring attaches, which a momentarily-quiet game can miss. Once on loopback
|
||
it never retries, and the UI shows no reason. Fix: **keep the rings live and
|
||
auto-promote to the hooked path whenever the hook later publishes a format, surface
|
||
the concrete fallback reason** (no format in time / not renderable / hook off) in the
|
||
Audio panel + log, and **flag a low-confidence rate (non-standard, didn't snap) in
|
||
red** so a bad measurement is obvious at a glance.
|
||
- **Re-measure button.** A debug action (Audio panel) to re-run a stream's rate
|
||
measurement on demand. Needs a small host→hook control surface: add operator-control
|
||
fields to the reserved area of `AudioRingHeader` (per-stream, already shared both
|
||
ways) — a bumped `op_seq` plus a "re-measure" request. The hook resets that stream's
|
||
measurement window, returns it to *Measuring*, clears `format_valid`, and republishes
|
||
once it reconverges; the host re-inits its render client when `format_generation`
|
||
bumps (so the corrected rate takes effect live).
|
||
- **Per-game format overrides (persisted), incl. the unmeasurable channels/bit-depth.**
|
||
A dropdown/inputs in the Audio panel — **shown only under the Debug details flag**,
|
||
per stream — to set sample rate, channels, bit-depth, and PCM/float when detection is
|
||
wrong or unrecoverable, **keyed by game image name and persisted** to a small store
|
||
next to the exe, so a known-bad game is auto-corrected on its next launch. A
|
||
**rate-only** override is host-side only (re-init
|
||
the render loop at the operator's rate — the bytes are already framed correctly); a
|
||
**channels/bits** override goes through the same `AudioRingHeader` op channel so the
|
||
*hook* re-frames its capture copy (`block_align`) and drops the over-read clamp, then
|
||
the host re-inits on the `format_generation` bump. **A format caught exactly at
|
||
`IAudioClient::Initialize` is itself saved as that game's override** (ground truth),
|
||
so a later late-attach to the same game is corrected automatically; if that overwrites
|
||
a previously-stored override that *differs*, log a warning.
|
||
- **Auto re-attach the same game on relaunch (session-only).** A checkbox on the
|
||
attached (or just-terminated) target, default off, *not* persisted, and **always
|
||
visible — not gated behind Debug details** (it's the recommended recovery path, not a
|
||
diagnostic). While it's on and the target has terminated, the host watches the process
|
||
list for a process of the same image name and injects automatically the moment it
|
||
reappears — built on the existing re-attach-by-image-name path, so there's no target
|
||
to type (it's the same game that was just attached). The workflow this enables: if
|
||
audio came out wrong (late attach forced the guess), tick the box, kill and relaunch
|
||
the game, and the host re-attaches *early* — early enough to catch
|
||
`IAudioClient::Initialize` and read the exact format, no guessing. This is a
|
||
**dependable fallback, not the intended primary workflow**: hardened first-launch
|
||
detection (above) should get it right without a relaunch; auto-re-attach is the
|
||
reliable recovery when it doesn't, and never asks the operator to guess a format.
|
||
- **Color-coded log levels.** The Log window should color warnings and errors (amber /
|
||
red) so they stand out from routine lines. `LogRecord` already carries a
|
||
(currently-unused) `level` field; add severity variants to the hook's `logf` and have
|
||
the host color by level. Used by the override-overwrite warning and the fallback
|
||
reasons above.
|
||
|
||
Ordering: measurement hardening + visible/red fallback reason + log levels first
|
||
(fixes the bug outright for most cases, no protocol change) → the `AudioRingHeader` op
|
||
channel (re-measure + channels/bits override; bumps `kAudioRingVersion`) → per-game
|
||
override persistence + auto-saving caught formats → session auto-re-attach (host-only,
|
||
extends the existing re-attach path).
|
||
|
||
### Future work
|
||
|
||
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
|
||
them needs a Vulkan layer / device-dispatch hook plus a `vkCmdCopyImage` to a
|
||
readable image. Use WGC in the meantime.
|
||
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
|
||
hook would be the lower-latency upgrade.
|
||
- **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB
|
||
subsystem forwards via window messages (`PostMessage`) plus synthesized
|
||
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop
|
||
and polling games. Games that read keyboard/mouse via **Raw Input** (`WM_INPUT` /
|
||
`GetRawInputData`, e.g. Trails through Daybreak) or **DirectInput**
|
||
(`IDirectInputDevice8::GetDeviceState/GetDeviceData`) don't see it. Add hooks for
|
||
those paths to synthesize the forwarded input there too.
|
||
|
||
## Building
|
||
|
||
Requirements: Windows 10/11, Visual Studio 2022 (MSVC + C++ workload), CMake ≥ 3.21.
|
||
|
||
```sh
|
||
git clone --recurse-submodules <repo-url>
|
||
# or, if already cloned:
|
||
git submodule update --init --recursive
|
||
|
||
cmake -S . -B build -G "Visual Studio 17 2022" -A x64
|
||
cmake --build build --config Debug
|
||
# output: bin/Debug/coop_host.exe (+ coop_hook.dll, test exes)
|
||
```
|
||
|
||
The x64 build also drives a nested Win32 sub-build (CMake `ExternalProject`,
|
||
configured into `build/x86/`) that produces `coop_hook_x86.dll` and
|
||
`coop_inject_x86.exe` for 32-bit games, staged next to the x64 binaries. Disable
|
||
it with `-DCOOP_BUILD_X86_HELPER=OFF` if you don't need 32-bit support.
|
||
|
||
Third-party dependencies (Dear ImGui, SafetyHook) are git submodules under
|
||
`third_party/`. No vcpkg / package manager is used.
|
||
|
||
**Steam Input is optional.** It's enabled automatically when the Steamworks SDK is
|
||
vendored at `third_party/steamworks_sdk/` (extract the `steamworks_sdk_*.zip`
|
||
there). The SDK isn't redistributable, so it's gitignored and never committed; if
|
||
it's absent the host builds XInput-only (no other features depend on it). When
|
||
present, the build links `steam_api64.lib`, stages `steam_api64.dll` and the
|
||
action manifest next to the host, and also builds `coop_steam_input_probe`.
|
||
|
||
Steam Input is **off by default and XInput is the primary path**: merely
|
||
initializing Steam Input activates Steam's in-process XInput interception, which
|
||
hides controllers from XInput unless they're bound to our action set for the
|
||
running appid. Enable it (Controllers panel → **Use Steam Input**) only once a
|
||
controller is bound to Steam Input for the donor appid.
|
||
|
||
### clangd / IDE setup
|
||
|
||
The Visual Studio CMake generator does **not** emit `compile_commands.json`, so
|
||
clangd has no include paths and reports false errors. Run
|
||
[`gen-compile-commands.bat`](gen-compile-commands.bat) once (and after adding
|
||
sources or include dirs); it configures a parallel Ninja build in `build-clangd/`
|
||
that produces the database, which [`.clangd`](.clangd) points clangd at. clangd's
|
||
clang-cl driver resolves the MSVC / Windows SDK system includes on its own.
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
ctest --test-dir build -C Debug --output-on-failure
|
||
```
|
||
|
||
- **`hook_selftest`** — in-process check of the IPC + XInput hook core (no game,
|
||
no controller needed).
|
||
- **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC
|
||
push/pop, wrap-around, format handshake, overrun/drop). No device needed.
|
||
- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum /
|
||
soft-clip / encode for float32 + int16). No device needed.
|
||
- **`audio_hook_test`** — in-process self-test of the WASAPI render-hook's **format
|
||
detection**, the part that gets pitch right. Using a shared configurable
|
||
`ToneSource` (the same render helper `coop_tone` uses), it renders tones at a matrix
|
||
of common formats (44100/48000/96000 Hz, mono/stereo/5.1, 16-bit PCM / 32-bit float)
|
||
and asserts the hook reports the right rate/channels/bits + provenance for **both**
|
||
code paths: **see-init** (hooks installed first → exact `Initialize` format) and
|
||
**guess** (render client pre-exists → device-mix guess whose true rate is measured
|
||
from the cadence, the Brotato/Godot case). Also checks the frames reached the ring
|
||
non-silent. Skips cleanly with no audio endpoint.
|
||
- **`srgb_format_test`** — unit test of the `srgb_to_unorm` mapping the hooked
|
||
video path uses so `*_SRGB`-backbuffer games aren't darkened. No device.
|
||
- **`opengl_hook_test`** — in-process self-test of the OpenGL capture path:
|
||
installs the swap hooks, drives a real OpenGL context (clears the backbuffer to a
|
||
known color, calls `SwapBuffers`), and asserts the detour fired, the frame was
|
||
`glReadPixels`'d into the shared texture, and a second device reads the exact
|
||
pixels back by name. Skips cleanly without an OpenGL / D3D11 device.
|
||
- **`dx12_present_hook_test`** — in-process self-test of the Present hook's **D3D12**
|
||
path: drives a real D3D12 swapchain through the (shared) `IDXGISwapChain::Present`
|
||
vtable and asserts the D3D11On12 bridge wraps the `ID3D12Resource` backbuffer and
|
||
copies it into the shared texture, then reads the exact rendered color back by
|
||
name. Skips cleanly without a D3D12 device.
|
||
- **`present_hook_test`** — in-process self-test of the Present-hook video path:
|
||
installs the hook, drives a real D3D11 swapchain in the same process (clears the
|
||
backbuffer to a known color and calls `Present`), and asserts the detour fired,
|
||
the backbuffer reached the shared keyed-mutex texture, and a second device can
|
||
open it by name and read the exact pixels back. Skips cleanly if the machine has
|
||
no D3D11 device.
|
||
- **`audio_loopback_test`** — spawns `coop_tone.exe` (a standalone configurable WASAPI
|
||
sine-wave source under [`tools/audio_tone`](tools/audio_tone)) at several source
|
||
formats (device default, 44100/48000/96000 Hz) and verifies the shipping
|
||
process-loopback capture (the fallback backend) receives non-silent audio by PID for
|
||
each — confirming loopback is format-agnostic (it captures post-mix at the device
|
||
endpoint format). Skips cleanly if the machine has no audio endpoint.
|
||
|
||
### Debugging the hooks against a real game
|
||
|
||
[`tools/audio_probe`](tools/audio_probe) (`coop_audio_probe.exe <pid> [seconds]`)
|
||
brings up the audio render-hook without Steam / RPT / the host UI: it creates the
|
||
IPC block + audio ring the hook expects, injects `coop_hook.dll` into the target
|
||
game, then drains the ring and prints per-stream format, captured-frame counts,
|
||
peak amplitude (proves the audio is real, not silence), and overruns. It enables
|
||
the hook's file trace (`%TEMP%\coop_hook.log`) for the run.
|
||
|
||
[`tools/input_probe`](tools/input_probe)
|
||
(`coop_input_probe.exe <pid> [seconds] [disable_mask]`) does the same for input: it
|
||
injects, reports one connected pad, and toggles a button each second so the game's
|
||
input layer sees a real state change. `disable_mask` (hex bits `0x1`=input
|
||
`0x2`=focus `0x4`=audio `0x8`=video) skips installing a subsystem, so you can
|
||
**bisect which injected subsystem affects a game** — this is how the 32-bit
|
||
Present-hook crash was isolated.
|
||
|
||
Both auto-detect a 32-bit (WOW64) target and inject via `coop_inject_x86.exe` +
|
||
`coop_hook_x86.dll`, exactly like the host. The probes build into
|
||
`bin/<config>/tools/` (the deployable `bin/<config>/` root holds only shipping
|
||
artifacts; tests build into `bin/<config>/tests/`) and resolve `coop_hook.dll` from
|
||
the root one level up, so run them from there. **Kill the game between runs** — the
|
||
loaded DLL locks `coop_hook.dll` against the next rebuild.
|
||
|
||
## Running the tool (manual, end-to-end)
|
||
|
||
This needs Steam, a donor game that supports Remote Play Together, and a second
|
||
person/account to receive the stream.
|
||
|
||
1. **Launch the host under a donor appid.** Find the donor's appid (the number in
|
||
its store URL); the donor only needs RPT support and is never actually played:
|
||
|
||
```text
|
||
"C:\Program Files (x86)\Steam\steam.exe" -applaunch <donorAppId> "D:\dev\CoopAllTheThings\bin\Debug\coop_host.exe"
|
||
```
|
||
|
||
The borderless window appears and Steam marks the donor "running". If the donor
|
||
ignores the trailing path, set the host as the donor's **Launch Options**
|
||
(`"D:\...\coop_host.exe" %command%`) or use a launcher like RemotePlayDetached.
|
||
|
||
2. **Start the real game** windowed or borderless (not exclusive fullscreen — see
|
||
Lessons learned). In the host's **Injection** panel, filter for the game's
|
||
`.exe`, select it, and click **Inject & Connect**. Watch **Hook status** for
|
||
**Attached**, a non-zero **XInput polled: N/s**, and **Focus spoof: active**.
|
||
|
||
3. **Mirror video:** in the **Video mirror** panel, tick **Mirror game window** —
|
||
the host window now shows a live, letterboxed copy of the game. **Source**
|
||
picks how the frames are grabbed: **WGC** (default, Windows Graphics Capture —
|
||
works for any window) or **Hooked (Present)** (the injected hook's shared
|
||
texture — lower latency and no capture border, for DXGI / D3D11 and OpenGL
|
||
games; selecting it installs the video subsystem in the game).
|
||
|
||
4. **Mirror audio:** in the **Audio mirror** panel, tick **Mirror game audio**.
|
||
With the hook injected, **Source** shows **Hooked (no echo)** and the game's
|
||
local playback goes silent while guests still hear it. If it shows **Loopback
|
||
(echo)** the hook's render path wasn't caught and you'll hear the game twice
|
||
locally (guests still hear it once). The **Render streams** table shows how
|
||
many WASAPI streams the game emits.
|
||
|
||
5. **Start Remote Play Together** from Steam and invite a guest. Verify the guest
|
||
sees the mirrored video, hears the audio, and that their controller drives the
|
||
real game.
|
||
|
||
Useful checks while developing without RPT: tick **Forward synthetic test input**
|
||
in the Injection panel to make the game move on its own (proving forwarding is the
|
||
source), and click away from the game to confirm focus spoofing keeps it running.
|
||
|
||
> Injection access error → run the host as administrator. A 32-bit (WOW64) target
|
||
> is injected automatically via `coop_inject_x86.exe` + `coop_hook_x86.dll`; if
|
||
> those aren't next to the host, rebuild (the x86 sub-build stages them there).
|
||
|
||
## Lessons learned
|
||
|
||
Non-obvious things that cost time and constrain the design:
|
||
|
||
- **RPT only streams the *focused* window.** The game can't hold focus itself, so
|
||
the hook spoofs it (`GetForegroundWindow` / `GetActiveWindow` / `GetFocus` +
|
||
swallowing deactivation messages) to keep the game polling and rendering while
|
||
the host owns real OS focus.
|
||
- **Run target games windowed or borderless, never exclusive fullscreen** —
|
||
exclusive fullscreen minimizes on focus loss (defeating the spoof) and can't be
|
||
window-captured. While unfocused the game gets no OS keyboard/mouse, only the
|
||
forwarded pad.
|
||
- **WGC captures occluded windows but not minimized ones.**
|
||
- **Process-loopback capture doesn't mute the source.** Capturing a process's
|
||
render doesn't stop it reaching the speakers, so the no-echo path instead injects
|
||
a WASAPI render-hook that copies each buffer then releases it with
|
||
`AUDCLNT_BUFFERFLAGS_SILENT`; loopback stays as the (echoing) fallback.
|
||
- **`ActivateAudioInterfaceAsync` needs an *agile* completion handler.** If the
|
||
handler doesn't answer `QueryInterface` for `IAgileObject`, the call is rejected
|
||
**synchronously** with `E_ILLEGAL_METHOD_CALL` (`0x8000000E`) — regardless of
|
||
apartment, device path, or activation params. (WRL/wil samples make the handler
|
||
agile for you.) Process loopback also needs the Win10 20H1 headers
|
||
(`NTDDI_VERSION ≥ 0x0A00000B`).
|
||
- **COM methods have no exports, so hooks walk vtables by frozen-ABI index — count
|
||
exactly.** All instances of a coclass share one vtable, so hooking one object's
|
||
slot catches every instance; but `IAudioClient::GetService` is **14**, not 13
|
||
(`SetEventHandle` sits at 13 between `Reset` and `GetService`). Count every
|
||
inherited `IUnknown`/base method when adding a hook.
|
||
- **D3D12 capture copies the *rotating* back buffer, not `GetBuffer(0)`.** D3D11
|
||
flip-model keeps `GetBuffer(0)` pointing at the live back buffer, but D3D12 rotates
|
||
buffers explicitly — the game renders into the buffer at
|
||
`IDXGISwapChain3::GetCurrentBackBufferIndex()`, which advances each `Present`.
|
||
Grabbing buffer 0 copies a stale buffer on N-1 of every N frames, so the mirror
|
||
silently runs at refresh/N — yet the Present counter, published FPS, generation, and
|
||
latency all read full rate (they count Presents, not unique content), so the metrics
|
||
look perfect while the eye sees missing frames. Query the current index right before
|
||
the trampoline `Present` (that's the just-rendered buffer) and copy that one.
|
||
- **Keep the D3D12 capture copy off the game's present queue, but ordered after its
|
||
frame.** The D3D11 path copies on the game's immediate context, so it's naturally
|
||
ordered after the frame and on the game's own timeline. For D3D12 the D3D11On12
|
||
bridge needs a queue: running the copy on the *game's* present queue orders it
|
||
correctly but stalls the game's own presents (GPU back-pressure, plus the shared
|
||
keyed-mutex `AcquireSync` is a **CPU-blocking** call on the render thread). Running
|
||
it on an independent queue avoids the stall but races the game's render → stale
|
||
frames. The fix is both: run the copy on **our own** queue, and order it with a
|
||
**fence** the game's present queue signals after its frame (a near-free op) and our
|
||
queue waits on. The present queue is recovered for late injection by hooking
|
||
`ID3D12CommandQueue::ExecuteCommandLists` (the per-frame method, not creation). Make
|
||
the producer-side `AcquireSync` non-blocking (`timeout 0`) so a busy mutex drops a
|
||
*mirror* frame instead of stalling the game; the Video panel's "Frames lost" line
|
||
surfaces both capture- and display-stage drops.
|
||
- **A render client that predates our injection has no knowable format — measure it.**
|
||
We inject into already-running games, so we usually never see the game's
|
||
`IAudioClient::Initialize`; the render-hook then assumes the device mix format for that
|
||
stream. That's wrong for games that render at a non-device rate via WASAPI
|
||
`AUTOCONVERTPCM` (e.g. Godot / Brotato render 44100 Hz while the endpoint mixes at
|
||
48000), so the captured audio plays back **pitch-shifted up**. Fix: treat such a format
|
||
as a *guess* and measure the stream's true sample rate from its render cadence
|
||
(frames/sec over a short active window, snapped to the nearest standard rate) before
|
||
publishing it, deferring capture until verified. Discard the first measurement window:
|
||
the moment we attach, the stream's already-queued buffers arrive in a burst that
|
||
over-counts (the `audio_hook_test` matrix caught this), so measure the next,
|
||
steady-state window. **Only the rate is recoverable, though** — channels/bit-depth can't
|
||
be measured (`AUTOCONVERTPCM` hands `GetBuffer` a *fixed* staging buffer, so there's no
|
||
buffer stride; confirmed empirically) and WASAPI has no API for a pre-existing client's
|
||
format, so they stay the device-mix guess. That's right for the common case (engines
|
||
render stereo float = the endpoint), and a `VirtualQuery` clamp on the capture copy
|
||
keeps a too-large guessed block from ever over-reading the source buffer. Streams we *do*
|
||
watch get created carry their exact `Initialize` format, and the loopback fallback
|
||
captures post-mix at the device format (always correct). The Audio panel shows each
|
||
stream's provenance (known / measuring / measured rate (ch/bits assumed)) so what the
|
||
mirror is using is always visible — see Limitations.
|
||
- **Capturing at `Present` decouples the mirror from DWM composition.** The hook copies
|
||
the backbuffer inside the game's `Present`, which the game issues at its true render
|
||
rate regardless of how DWM composites that *window*. So an unfocused game window can
|
||
judder (DWM under-composites background windows; only the focused window gets VRR /
|
||
independent flip) while the mirror — which receives every `Present` — stays smooth.
|
||
This is why the game window not being focused doesn't matter: it isn't the surface
|
||
anyone sees. The same focus rule explains why an unfocused *tool* window can render
|
||
below the game's rate (it loses VRR), so in use the mirror is the focused window.
|
||
- **SafetyHook on x86 has two traps that froze 32-bit Slaps and Beans.** (1)
|
||
`InlineHook::call()` invokes the trampoline as `__cdecl`, but most targets are
|
||
`__stdcall` (COM methods like `IDXGISwapChain::Present`, the WASAPI interfaces,
|
||
`WINAPI` `SwapBuffers`); on 32-bit that double-cleans the stack → ESP imbalance →
|
||
crash (Debug: **Run-Time Check Failure #0**). Use **`stdcall()`** (a no-op on
|
||
x64). (2) Don't *inline-hook* COM methods on x86 at all: MMDevApi/AudioSes
|
||
prologues do `push ebp; mov ebp,esp; and esp,-8` and read args **EBP-relative**,
|
||
which SafetyHook's trampoline relocation breaks (the original then runs with
|
||
garbage args and faults). Hook COM methods by **swapping the vtable entry**
|
||
instead (`VirtualProtect` the slot, overwrite the pointer, call the saved
|
||
original) — no code patching, pristine stack regardless of prologue. Inline
|
||
hooking stays fine for `Present`/`SwapBuffers` (clean prologues). Guarded by the
|
||
x86 hook tests.
|
||
- **Steam Input init suppresses XInput.** Initializing Steam Input turns on Steam's
|
||
in-process XInput interception, which hides controllers from `XInputGetState`
|
||
unless they're bound to the running appid's action set — defaulting to it
|
||
silently broke forwarding. XInput is primary; Steam Input is opt-in.
|