A real chain-aware Vulkan implicit layer the loader inserts at vkCreateInstance
-- the reliable early-presence path for games that init Vulkan immediately,
which the inline-hook vk_hook can't catch. It intercepts vkCreateInstance /
Device / CreateSwapchainKHR / QueuePresentKHR via proper layer-chain dispatch
and does the same read-back capture (vkCmdCopyImageToBuffer -> swizzle ->
hook-owned D3D11 shared texture, with present-semaphore re-chaining) as vk_hook.
The loader/layer link structs (VkLayer*CreateInfo, VkNegotiateLayerInterface)
aren't in Vulkan-Headers, so they're hand-declared to interface version 2. Key
gotcha found via tracing: the loader tags those link structs with small internal
sType values (LOADER_INSTANCE_CREATE_INFO=47, _DEVICE=48), not the 1000000000
range -- matching the wrong value made the device-chain walk fail.
Scoping: an implicit layer loads into every Vulkan app, so it only *captures*
when COOP_VK_LAYER_FORCE is set (tests) or this process's image matches
%TEMP%\coop_vk_target.txt (the host writes it); otherwise pure pass-through.
mock_game_test registers it via VK_LAYER_PATH/VK_INSTANCE_LAYERS and decodes
frames through it. Ships at the bin root with its JSON manifest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The opt-in implicit-layer piece requires the loader/layer interface header
vk_layer.h, which Vulkan-Headers doesn't ship (it's in Vulkan-Loader / the SDK)
-- so it needs a new submodule or hand-declared link structs, confirming it's a
separate, higher-risk component rather than a quick add.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Architecture now lists Vulkan as a hooked producer (GPA interception + read-back,
early-presence required). Adds a lessons-learned bullet (can't late-hook Vulkan;
the present-semaphore re-chaining trap; the too-late heuristic; early-load
testing). mock_game_test docs note the suspended-launch capture + too-late check.
The Vulkan milestone is trimmed to its one remaining piece -- the opt-in implicit
layer for immediate-init games -- with the capture/banner/best-effort marked done.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hook reports vk_too_late when vulkan-1.dll is loaded and the GPA hook has
been in for >4s but it never caught the app creating its device -- i.e. the app
resolved its Vulkan functions before we hooked (injected too late). Surfaced
through a new HookStatus.vk_too_late field (protocol 15->16) and shown by the
host as a red banner on the mirror window: "Detected a Vulkan game, but the
mirror hook attached too late... enable Auto re-attach (and Set up Vulkan layer
if it persists) and relaunch." WGC keeps mirroring meanwhile.
mock_game_test gains a too-late detection check (late-inject a Vulkan game ->
vk_too_late trips). Verified live: the banner shows over the running tool when a
Vulkan game is injected late with the video subsystem on (added a debug-harness
`video on/off` command to drive it). 15/15 ctest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New vk_hook.cpp inline-hooks the vulkan-1.dll vkGetInstanceProcAddr export and
hands back our wrappers for vkCreateInstance / vkCreateDevice / vkGetDeviceProcAddr
/ vkCreateSwapchainKHR / vkQueuePresentKHR, so a volk-using (loader-bypass) app
resolves our hooks. On present it reads the swap-chain image back with
vkCmdCopyImageToBuffer into a host-visible buffer (same read-back model as
D3D10/D3D9/OpenGL), swizzles BGRA->RGBA, and uploads it into the shared
keyed-mutex texture on a hook-owned D3D11 device. The read-back submit re-chains
the present's wait semaphores (consume the originals, signal our own that the
real present waits on) so capture orders after rendering without double-waiting.
Wired into the video subsystem with a lazy retry (vulkan-1.dll loads late). Hook
links the official Vulkan-Headers (headers only, VK_NO_PROTOTYPES) via the
include dir so the x86 sub-build builds too.
Because Vulkan caches its present pointer at init, late injection can't hook it:
mock_game_test launches the mock **suspended**, injects, resumes, and under
COOP_MOCK_VK_EARLY the mock loads Vulkan and waits so the hook arms first -- then
decodes frames through the hook like the other backends. 15/15 ctest (x64 +
x86); skips cleanly without a Vulkan driver.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reflect that the Vulkan mock backend + submodules are done, and scope the
remaining Vulkan capture honestly: hook vkGetInstance/DeviceProcAddr +
intercept device/swapchain creation, and read the presented image back with
vkCmdCopyImageToBuffer (the same read-back pattern as D3D10/D3D9/OpenGL --
simpler/robuster than a VK_KHR_external_memory keyed-mutex blit). Note testing
needs the early-load path (late-inject can't catch a Vulkan present), and Vulkan
games mirror via WGC meanwhile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The headless ui_fit_test proves panels *fit* their windows but not that the
overlay *looks* right. Document that closing M1 requires running the actual
coop_host.exe (-DCOOP_TEST_HARNESS=ON), driving it to maximum info, taking an
F10 screenshot, and eyeballing the real overlay -- a green unit test is not a
substitute for looking at the product. Verified live: every panel fits and reads
correctly at the monitor resolution, and the live `uifit` reports "fit".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add render_vk.cpp (selectable as `vk`): brings up a real Vulkan
instance/device/swap chain via volk (which dlopens vulkan-1.dll -- the
loader-bypass case the capture hook must handle) and clears the swap-chain image
to the frame-counter colour each frame with vkCmdClearColorImage (no pipeline,
no shaders, no SPIR-V) and presents. The whole image encodes the frame number,
so it animates and stale frames are detectable.
Adds the official Khronos Vulkan-Headers + zeux/volk submodules and a
coop_require_submodule() CMake helper that fails with a clear "git submodule
update --init --recursive" message rather than auto-cloning. volk is pinned to
the project's dynamic CRT (no LNK4098).
mock_game_test gets a vk liveness check (its present pointer is cached at init,
so late injection can't hook it -- the capture path needs the early-load path,
to come). 16/16 ctest, skips cleanly without a Vulkan driver.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add render_gl.cpp (selectable as `gl`): renders the animated pattern with
scissored clears (glClear + glScissor -- GL 1.1, exported straight from
opengl32) and presents with SwapBuffers. No glad/submodule needed: a legacy
wglCreateContext + <GL/gl.h> suffices, so the planned loader dependency was
dropped. GL is bottom-left origin and the capture flips top-down, so the
frame-counter block is drawn at the GL top to land top-left in the captured
image. Window class gains CS_OWNDC for a stable GL DC; best-effort vsync via a
runtime wglSwapIntervalEXT lookup.
The GL SwapBuffers/wglSwapBuffers hook now bumps the shared present counter
(it's the GL present), so present_calls works for GL games too. mock_game_test
decodes GL frames through the existing glReadPixels capture path; 15/15 ctest.
Roadmap: OpenGL milestone done and removed (Vulkan renumbered to M2);
architecture/lessons/test docs updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New d3d9_hook.cpp (own file like opengl_hook): inline-hooks
IDirect3DDevice9::Present (vtable index 17, discovered from a throwaway device;
clean prologue so inline is safe, stdcall() on x86) and read-backs the backbuffer
with GetRenderTargetData into a D3DPOOL_SYSTEMMEM surface, swizzles BGRA->RGBA,
and uploads it into the shared keyed-mutex texture on a hook-owned D3D11 device
(a D3D9 surface isn't D3D11-shareable). Both plain D3D9 and D3D9Ex call this same
Present, so one read-back path serves both -- the GPU shared-surface fast path
the plan sketched for D3D9Ex wasn't worth it. Wired into the video subsystem
(install/remove in dllmain); hook links d3d9.
mock_game_test decodes DX9 + DX9Ex frames through the hook; 15/15 ctest (incl.
the x86 sub-build). Roadmap: DX9 milestone done and removed (remaining
renumbered); architecture + lessons updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add render_dx09.cpp (selectable as `dx9ex` / `dx9`): renders the animated
pattern with Clear + ColorFill (D3D9's built-in rect fill -- no shaders) and
presents through a real D3D9 / D3D9Ex device. Two modes so the two D3D9 capture
paths each have a matching game. Smoke-verified: both present frames and exit
cleanly. The capture hook + frame-decode test land next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A pure-D3D10 game's backbuffer QIs to ID3D11Texture2D but that view reads back
empty (content lives on the game's D3D10 device), and a D3D11 backbuffer also
QIs to ID3D10Texture2D -- so GetBuffer can't discriminate. The reliable signal
is that a feature-level-10 device rejects CreateTexture2D with the NT-handle
keyed-mutex share flags (E_INVALIDARG). The present hook now tries the D3D11
fast path and, on that failure, switches (sticky) to reading the backbuffer
through the game's own D3D10 device into a staging texture and uploading it into
the shared texture on a hook-owned D3D11 device (Map blocks until the GPU copy
completes -> no cross-device race). The host reader is unchanged.
mock_game_test now decodes DX10 frames through the hook (monotonic/advancing)
alongside DX11/DX12; 15/15 ctest. Roadmap: DX10 milestone done and removed
(remaining renumbered); architecture + lessons updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add render_dx10.cpp (selectable as `dx10`): composes the animated pattern
(background + moving bar + top-left frame-counter block) on the CPU each frame
and CopyResource's it into a real D3D10 DXGI swap-chain back buffer (DX10 has no
rect-clear; no shaders). Smoke-verified: `coop_mock_game dx10 2` presents frames
and exits cleanly. The frame-accurate decode-through-the-hook assertion lands
with the DX10 capture commit next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add panel overflow instrumentation (record_panel_fit reading ImGui ScrollMax),
a forced layout-reference + debug-aware center split, and host harness commands
(uisize/uifit). New headless ui_fit_test drives the real Controllers + Audio
panels at reference resolutions with Debug details on and asserts no panel
overflows its assigned size; the Audio panel gains a demo mode so its richest
content renders without a live mirror.
Fixes from the measured overflow: widen the center column (was too narrow ->
horizontal overflow), merge the Controllers poll/round-trip tables and fold the
trigger line into the slot line, and make the center height split
Debug-details-aware (Video's height is mirroring-driven, not debug-driven, so a
static split can't serve both modes). Controllers + Audio now fit at 1920x1080
with max info. Video/Injection/Log fit is finalized at the end via the live
uifit harness (M1 stays open until then).
15/15 ctest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restructure the roadmap into one "Current work" section where each rendering
API is a single milestone that builds its mock-game backend and the injected
capture for that same API side by side, so each API reaches verified
end-to-end before the next. Adds an M1 end-user UI fit pass (drive the overlay
to maximum info via the harness, screenshot, assert every panel fits its
window; resize/rearrange otherwise) kept green by every later milestone.
Folds in DX10, DX9 (D3D9Ex + plain non-Ex), OpenGL, and Vulkan (best-effort
inject + opt-in implicit layer, too-late red prompt). Drops the old Vulkan/
D3D9 future-work bullets.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mock_game_test now launches coop_mock_game at 44100/48000/96000 (PCM + float)
and asserts the hook measures each variant's rate through the full inject path
(+ non-silent capture) -- the "all audio variants work" part of the suite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Since guessed streams are now captured-but-not-silenced (the over-write fix),
the hooked path is only no-echo for an exact/override format. The panel
inferred "Hooked (no echo)" unconditionally, which was misleading. Show
"Hooked (echo -- guessed format)" (amber) for a guessed primary stream and
"Hooked (no echo)" (green) only for exact/override; drop the unconditional
"no echo" from source_name() and the mirror status string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mock_game_test launches coop_mock_game (DX11/DX12, frame-numbered A/V), injects
coop_hook.dll, opens the hook's shared video texture, and decodes the frame
number from the captured pixels to assert a monotonic, advancing mirror for
both backends (catches dropped/stale/out-of-order frames). Then it runs an
A/V + hook/unhook stress pass and confirms no crash + capture resumes. Adds a
read_pixel() readback to the shipping SharedTextureSource for the decode.
Robust to repeated/CI runs (kills stray games, retries inject). Trim the
roadmap (the mock-game item is done) and document the test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found by the mock-game capture/audio/hook stress test (toggling the audio
subsystem while a game renders):
1. Guessed-stream silence over-WRITE: hk_ReleaseBuffer zeroed num_frames *
guessed_block bytes, but a guess can be larger than the real per-frame size
(e.g. an 8ch device guess for a 2ch game), so the memset wrote past the real
buffer into adjacent audio memory -> intermittent access violation in the
game. Fix: capture but do NOT silence a guessed stream (it stays audible --
echo); only an exact/override format, whose frame size is known, gets the
no-echo silence.
2. VtableHook::remove nulled m_original, racing an in-flight detour into a null
call -> keep it valid (the original function stays mapped).
3. g_ipc was a non-atomic pointer read on the hot path while unhook nulled it
(TOCTOU) -> make it atomic, load once.
4. Stale GetBuffer/ReleaseBuffer pairing across a toggle -> epoch-stamp the
GetBuffer and only capture in the same hooked epoch.
5. COM-object churn: re-creating the probe client every enable raced AudioSes ->
build the probe once, keep it across toggles (only swap vtable slots);
release on detach (shutdown_audio_hooks). Plus drain in-flight detours before
tearing down state.
Stress test: 0 crashes in many repeated runs (was ~50%). Guessed streams now
echo (the no-echo path is reached via an exact/auto-attach format or override).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A tiny test "game" for exercising the capture/audio/hook paths. Opens a normal
window and renders an animated pattern -- moving bar + per-frame background so
motion (and dropped frames) are obvious -- with a top-left block whose RGB
encodes the exact frame number, so a capture test can decode it and detect
dropped / duplicated / stale frames. Selectable backend (dx11 / dx12 today),
behind a RenderBackend interface so OpenGL/Vulkan can be added. With audio args
it also plays a configurable WASAPI tone (shared ToneSource), so it's a full A/V
source with a window (unlike coop_tone).
coop_mock_game.exe [dx11|dx12] [seconds] [rate] [channels] [bits] [pcm|float]
Milestone 1 of the mock-game roadmap item (the stress-test suite follows).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an "Auto re-attach this game on relaunch" checkbox (default off, never
persisted, shown while attached or terminated). While on and the target has
terminated, the host polls the process list for the same image name and
re-injects the moment it reappears -- built on the existing
re-attach-by-image-name path, so there's no target to pick. The kill+relaunch
recovery for a wrong audio format: re-attaching early catches
IAudioClient::Initialize and reads the exact format.
Validated live (harness): inject coop_tone -> kill -> relaunch -> auto
re-injected into the new pid, and early enough that the format came back as
Exact. Trim README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist audio overrides keyed by game image name (coop_audio_overrides.ini
next to the exe) so a known-bad game is auto-corrected on its next launch:
on attach to a guessed stream the host applies any saved override, and a
manual Override now saves too. A format the hook catches exactly at
IAudioClient::Initialize is auto-saved as that game's override (ground truth);
if it overwrites a differing stored value, a warning is logged. Host-originated
log lines now reach the Log window via IpcServer::host_log (color-coded).
Validated live (harness): a pre-seeded override for coop_tone is auto-applied
over the guess (state -> manual override, 48000/2/16). audio_overrides_test
covers persist/reload/case-insensitive lookup/differing-overwrite. Trim README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a per-stream op channel in AudioRingHeader (op_seq + op_* fields, version
2): the host posts re-measure / override commands, the hook applies them and
re-publishes (bumping format_generation), and the host rebuilds its render
client live on the change. The Audio panel (under Debug details) gains a
"Re-measure rate" button and a rate/channels/bit-depth/format override -- for
when detection is wrong or the channels/bit-depth were unrecoverable.
Also add a debug-only IPC test harness (-DCOOP_TEST_HARNESS, off by default,
absent from the shipped host): a file-based command channel that drives the
host's real UI code paths (inject / audio / re-measure / override / screenshot
/ status) for scripted validation, instead of unreliable synthetic mouse input.
Used to validate live: late-attach to coop_tone@44100 -> measured 44100,
promoted to hooked; override -> 2ch state, re-measure -> reconverge.
Trim the README roadmap to what's left; document the harness + rate_estimator_test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The host used to wait a fixed 1 s for the hook to publish a format and, on
timeout, fall to loopback permanently with no explanation -- which the now
slower (consensus) measurement made common. Restructure the audio thread to
keep the rings live the whole session and alternate: prefer hooked, and while
it isn't ready run loopback (echo) so guests still hear audio, watching the
ring to promote to the no-echo hooked path the instant the hook publishes a
format. Surface the concrete reason loopback is active (no format yet / not
renderable / no ring) in the Audio panel, in amber.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Harden the guessed-stream sample-rate measurement that produced wrong rates
(e.g. 44100 read as ~46205). New rate_estimator.hpp measures over longer
~0.5 s windows, rejects any window that doesn't snap to a standard rate
(standard rates are >8% apart, so a quantization/burst error big enough to
miss one lands in no-man's-land, never on a wrong neighbour), and requires
consensus across windows before committing. If consensus isn't reached it
commits a low-confidence estimate (new AudioFormat_LowConfidence, shown red)
rather than spinning or publishing garbage. Pure logic, unit-tested with
adversarial cadences (rate_estimator_test) incl. the real 46205 bug value.
Add log severity levels: hook logw/loge set LogRecord.level; the host Log
window colors warnings amber and errors red. The low-confidence rate logs a
warning. Protocol -> v15 (new format states); also reserves AudioFormat_Override.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Capture the rendered back buffer to a timestamped PNG next to the exe via
WIC, triggered by F10 (delivered even when unfocused). The capture runs in
render_frame just before Present so it includes the ImGui overlay, and reads
off the GPU so it works regardless of window focus, z-order, or occlusion. A
brief toast confirms the save (drawn the next frame, so it's never in the shot).
F10 chosen to avoid Steam's F12; its WM_SYSKEYDOWN is swallowed so Windows
doesn't enter menu mode. Documented in the Help menu + README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the diagnosis and plan for the unreliable hooked-audio format
detection: robust rate measurement (longer window, atomic endpoints,
consensus, reject non-standard rates), visible + red-flagged loopback
fallback with auto-promote, a re-measure button and per-game persisted
format overrides via an AudioRingHeader op channel, session-only
auto-re-attach on relaunch, and color-coded log levels. Drop the stale
"previous backlog is all shipped" note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The inline hint still claimed Esc quits (no longer true) and duplicated the
F1 shortcut + RPT-capture note that the Help menu already lists. Remove the
redundant block; the Help menu is the single source for those.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Escape was a leftover spike convenience that quit the tool. Esc is a common
in-game key, so dropping it from a borderless mirror was too easy to do by
accident. Remove the Escape-quit handler and add a File -> Exit menu item
(shown with its Alt+F4 shortcut). Alt+F4 already worked via DefWindowProc ->
WM_CLOSE -> WM_DESTROY; the menu item sets a one-shot flag the main loop reads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The View-menu verbosity switch reset to off on every launch. Register a
custom ImGui settings handler (a [CoopUI][State] section in coop_layout.ini,
alongside the window layout) so the operator's choice survives restarts.
Toggling it marks settings dirty so ImGui's auto-save writes it back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hooked audio mirroring played back pitch-shifted on games we inject into
that render at a non-device sample rate (e.g. Godot/Brotato render 44100 Hz
on a 48000 Hz endpoint via WASAPI AUTOCONVERTPCM). We attach to an
already-running game, so the render-hook never saw its IAudioClient::
Initialize and assumed the device mix format -- right channels/bits, wrong
rate -- so 44100 audio was rendered as 48000 (+~1.5 semitones).
Fix: treat a pre-existing client's format as a guess and measure its true
sample rate from the render cadence (frames/sec over a steady-state window,
snapped to the nearest standard rate) before publishing it, deferring
capture until verified. Discard the first measurement window so the
buffer-fill burst at attach time doesn't over-count. Streams created after
we inject still carry their exact Initialize format.
Channels/bit-depth genuinely can't be recovered for a pre-existing client:
AUTOCONVERTPCM hands GetBuffer a fixed staging buffer (no buffer stride to
measure -- confirmed empirically) and WASAPI exposes no API for the format.
They stay the device-mix guess, which is correct for the common case
(engines render stereo float, matching the endpoint). To keep a wrong guess
safe, a VirtualQuery clamp stops the capture copy from ever over-reading the
source buffer when the guessed bytes/frame is too large.
Surface all of this: a per-stream AudioFormatState (known / measuring /
measured rate (ch/bits assumed)) in HookStatus, shown in the Audio panel for
the hooked path and as "device endpoint (known)" for loopback; clear hook
logs; and enriched mirror status strings. Documented in README (Limitations
+ Lessons learned). The loopback fallback was always correct (post-mix at
the device format).
Tests: extract a shared, configurable ToneSource (used by coop_tone and the
hook self-test); coop_tone takes rate/channels/bits/format args. Rewrite
audio_hook_test to a format matrix x both code paths -- see-init (exact) and
guess (rate measured) -- plus a byte-incompatible guess that asserts the
clamp keeps capture safe. The matrix caught the attach-burst over-count.
audio_loopback_test now spawns coop_tone at several source formats to
confirm loopback is format-agnostic. 11/11 x64 + 3/3 x86 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A game that used the Hooked source with "Sync flip to game frames" left
the overlay sluggish (~5 FPS) after it terminated or was detached: with no
live game the generation never bumps, so wait_for_hooked_frame waited out
its full timeout every iteration.
Gate frame_sync_active() on InjectionPanel::target_state() == Alive, so a
terminated/hung/detached target falls back to normal vsync. Alive means the
process is running and the hook heartbeat is advancing, so this also covers
a frozen game (which would stall the generation the same way). The checkbox
preference is preserved -- frame-sync auto-resumes when a new game is
injected rather than silently unchecking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves the DX12 mirror stutter and makes dropped frames observable.
Decouple the D3D11On12 copy from the game's present queue. Submitting the
copy on the game's own present queue (the prior approach) ordered it
correctly but stalled the game's presents: GPU back-pressure, plus the
shared keyed-mutex AcquireSync is a CPU-blocking call on the render
thread. Running it on an independent queue avoids the stall but races the
game's render -> stale frames. Do both: run the copy on our own queue and
order it after the frame with an ID3D12Fence the game's present queue
signals (near-free) and our queue waits on. The present queue is still
recovered for late injection via the ExecuteCommandLists hook (now used to
signal the fence, not host the copy). Producer AcquireSync stays
non-blocking (timeout 0) so a busy mutex drops a mirror frame instead of
stalling the game.
Add drop detection (protocol v12 -> v13). The hook counts captures skipped
because the keyed mutex was busy (VideoShare.frames_dropped); the host
counts published frames it never displayed (generation gaps). The Video
panel shows "Frames lost: N/s capture N/s display", red when nonzero.
This confirmed the game-window-vs-mirror behavior is a display-path
artifact (unfocused windows lose VRR/independent flip), not a capture loss.
Add a one-shot present-pattern log: per distinct swapchain (size/format/
buffer index) and per distinct present-flags value, with DXGI_PRESENT_TEST
spelled out as an occlusion probe that draws nothing -- which is why
Miles Morales shows ~2 presents per captured frame (the test present is
counted but produces no frame).
Docs: add the DX12 capture lessons to the README (rotating back buffer,
fence/own-queue, capture-at-Present decoupling from DWM) and drop the now
-moot DX12 overhead future-work item.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Even copying the correct back buffer, the DX12 path occasionally showed a
several-frames-old frame under GPU load (visible when the game window has
true focus and the camera is whipped around with the mouse). Cause: the
On12 bridge submitted CopyResource on a command queue of our own, which
knows nothing about the game's queue. With frames in flight, our copy
could race ahead of the game's render of that buffer and capture its
previous (rotated) contents. The DX11 path never had this because it
copies on the game's immediate context, ordered after the frame.
Fix: submit the copy on the game's actual present queue so it's ordered
after the frame's rendering, matching the DX11 path. Recover the queue by
inline-hooking ID3D12CommandQueue::ExecuteCommandLists (the per-frame
method, not swapchain/queue creation) so it works for late injection --
the queue already exists when we attach. Record the last DIRECT queue
seen, preferring the one on the render thread (Present and its queue's
ExecuteCommandLists share that thread); the atomic is a cross-thread
fallback. Thread it into the On12 bridge, rebuilding if the captured
queue changes, and fall back to our own queue until it's captured.
Resolves D3D12CreateDevice dynamically from an already-loaded d3d12.dll,
so it adds no link dependency and no-ops for D3D11 / x86 games.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The D3D12 capture path grabbed GetBuffer(0) every Present. Unlike D3D11
flip-model -- where DXGI keeps GetBuffer(0) pointing at the live back
buffer -- D3D12 rotates buffers explicitly: the game renders into the
buffer at GetCurrentBackBufferIndex(), which advances each Present. So
buffer 0 only holds fresh content every Nth frame; the rest copied a
stale buffer, and the mirror silently ran at refresh/N with duplicate
frames in between.
Every metric read full rate (Present counter, published FPS, generation
bump, capture->display latency) because they count Presents, not unique
content -- which is why it looked fine but felt like missing frames,
especially on high-refresh DX12 games (DMC5/Myst at 144, Miles Morales).
Query IDXGISwapChain3::GetCurrentBackBufferIndex() before the trampoline
Present (so it's the just-rendered buffer) and copy that one; fall back
to 0 only if the interface is unavailable. The DX11 path is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Input thread: controller polling, pad publishing, and rumble forwarding were
driven by the render loop, so a low/synced frame rate throttled how often guest
input reached the game. New InputWorker owns the InputSource and runs poll +
IPC publish + rumble on a dedicated ~1 kHz thread, independent of rendering. The
UI thread reads a copy-safe InputSnapshot for the Controllers panel and relays the
Steam-Input request/active/failed state to/from the worker (Steam init/shutdown now
happen on the worker thread). IpcServer gained a mutex so the worker's publish() /
hook_status() can't race the UI thread starting/stopping the shared-memory channel
(use-after-unmap); InjectionPanel::test_input_ is now atomic. ControllersPanel::draw
takes an InputSnapshot instead of the live InputSource.
Fixed-width numbers: fast-changing readouts (menu-bar FPS/ms, Video pipeline rates +
latency + graph legend, controller poll rates + round-trip sticks, audio buffered ms
+ frames/s) printed with %.0f etc., so they shifted/blurred as values crossed digit
thresholds (99 -> 100) each frame. Padded them to fixed field widths so they stay put.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Frame sync: new "Sync flip to game frames" toggle in the Video mirror panel
(Hooked source only -- WGC frames are delivered by the compositor at monitor
refresh and don't carry the game's true present cadence, so it's disabled there).
When on, the main loop waits for the hook's next published frame (its generation
bump) before rendering and presents with sync interval 0, so the tool flips in
lockstep with the game instead of vsync. The wait pumps messages to stay
responsive and times out after 200 ms so a paused/stalled game can't hang the
overlay. timeBeginPeriod(1) keeps the wait's Sleep(1) granular; links winmm.
render_frame() gained a sync_interval parameter (default 1 = vsync).
Perf graphs: the old graphs drew the tool's frametime and FPS as single same-color
lines. Replaced with a custom multi-series plotter (ImDrawList polylines) that
overlays Tool (blue), Game present (green), and Hook publish (orange) -- or Tool +
WGC capture in WGC mode -- in distinct colors with a colored legend, for both an
FPS (0-144) and a frametime (0-33 ms) view. Game/hook rates come from an EdgeRate
tracker that measures the instantaneous rate the moment each counter advances, so
the lines have real per-frame resolution rather than 0.5 s stair-steps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Layout persistence: re-enable io.IniFilename (was nullptr "for the spike"),
anchored to a coop_layout.ini next to the exe so window positions/sizes survive
restarts even when Steam launches us under the donor appid (CWD is unreliable).
Path is UTF-8 for ImGui's file IO. When a saved layout is restored at startup,
suppress the computed-default force so it does not clobber the user's positions;
Reset layout (and a fresh install with no .ini) still applies the default.
Log spam: the worker thread re-attaches every audio ring every tick (idempotent),
and set_audio_ring logged unconditionally, flooding the log. Only log when the
ring pointer actually changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Center stack heights set to Controllers 45% / Video 32% / Audio 23% (measured from
the operator's preferred layout): Controllers has the most content, Audio the least.
- Future work: MKB forwarding for Raw Input / DirectInput games (the message +
polling-state path doesn't reach them -- e.g. Trails through Daybreak uses Raw
Input), and reducing the D3D11On12 (DX12) per-frame capture overhead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feedback from real use: the center column was far too wide and Injection/Log too
narrow; Audio was too tall while Controllers/Video were too short; and the layout
overlapped on launch until "Reset layout" was pressed.
- Columns are now proportional: Injection (left) and Log (right) take 36% each of the
usable width, the center control column 28% -- so the wide panels are wide and the
control panels narrow.
- The center stack heights are evened out (Controllers 30% / Video 38% / Audio 32% of
the column), so Audio no longer hogs it.
- The default layout is now forced (ImGuiCond_Always) for the first few frames after
launch, because the viewport WorkSize isn't trustworthy on frame 0 -- FirstUseEver
was locking in those wrong (overlapping) positions until a manual Reset layout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Games with several concurrent WASAPI render streams (e.g. Spider-Man: Miles
Morales) only had their first ("primary") stream mirrored; the rest kept playing
locally and never reached the guest. Now the render-hook captures + silences EVERY
tracked stream into its own ring (coop_audio_<pid>[_<index>]), each published with
that stream's own detected format (Initialize when caught, else GetMixFormat -- the
per-stream format detection, now actually used per ring rather than only for the
primary). The host creates a ring per stream and mixes the same-format streams with
a soft clip (host/src/audio/audio_mix.hpp); streams whose format differs from the
primary are still silenced (no echo) but skipped from the mix (would need
resampling).
The single-stream case is byte-for-byte unchanged: when only one stream is active
the host passes it through without the mixer, so the common path has no overhead or
fidelity change.
Verified: new audio_mix_test covers the decode/sum/soft-clip/encode math (float32 +
int16); audio_hook_test (x64 + x86) still passes, guarding the primary
capture+silence path against regression; full build x64 + x86 clean; ctest x64
11/11, x86 3/3. Multi-stream mixing against a real multi-stream game needs a live
session to fully confirm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
D3D12 games (e.g. Spider-Man: Miles Morales) fire the Present hook -- the DXGI
swapchain's Present is the same vtable function for D3D11 and D3D12 -- but
GetBuffer(0) as ID3D11Texture2D fails, so the hook used to idle. Now, when the D3D11
GetBuffer fails, present_hook bridges via D3D11On12: it gets the game's ID3D12Device
from the backbuffer, creates its own DIRECT command queue on it (no need to hook the
game's ExecuteCommandLists), builds an ID3D11On12Device, CreateWrappedResource's the
D3D12 backbuffer, and CopyResource's it into the existing shared keyed-mutex texture
-- so the host side is unchanged. The bridge is created lazily and torn down with the
hook.
Verified with a new in-process dx12_present_hook_test: it drives a real D3D12
swapchain (clears a backbuffer, Presents) and asserts present fired, the backbuffer
was bridged into the shared texture, and a second device reads the exact color back
by name -- {51,102,153,255}. Full build x64 + x86 clean (the x86 hook compiles the
D3D12 path too); ctest x64 10/10, x86 3/3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FPS readout only measured the host's own render rate, hiding capture stutter.
The Video panel now shows a "Pipeline rates" section:
- Tool render (host FPS / frametime, as before);
- hooked source: Game present (/s, from VideoShare.present_calls deltas), Hook
publish (/s, generation deltas), and capture->display latency avg/min/max ms;
- WGC source: WGC capture (/s) from a new WindowCapture frame-arrival counter
(game present + latency are n/a, since WGC frames aren't game-timestamped).
Latency uses a system-wide clock: protocol v11->v12 adds VideoShare.present_qpc,
stamped by the hook at publish (publish_video_frame); the host measures
now_qpc - present_qpc per newly published frame, windowed to min/avg/max each second.
Verified: present_hook_test (x64 + x86) now asserts present_qpc is stamped; full
build x64 + x86 clean; ctest x64 9/9, x86 3/3. The live rate/latency numbers need a
real game mirroring to read meaningfully; wiring validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Games that ClipCursor / re-center via SetCursorPos while focused trap the operator's
mouse (the focus spoof makes them think they're always focused), so the operator
can't reach the overlay. The Focus subsystem now inline-hooks ClipCursor and
SetCursorPos (stdcall trampolines): while "release" is requested it forces
ClipCursor(NULL) and swallows the re-centering SetCursorPos; otherwise it passes them
through. It frees any existing clip at install and re-frees each worker tick (covers
one-time clippers and a runtime clip->release toggle).
Host: protocol v10->v11 adds HookControl::allow_cursor_clip (0 = release, the
default). The Injection panel gets a "Release operator cursor" checkbox and an F2
hotkey (InjectionPanel::toggle_cursor_release); default released, since the guest
plays via the pad so the game's clip is operator-only.
Verified: full build x64 + x86 clean; ctest x64 9/9, x86 3/3. The cursor behavior
against a real clipping game (Trails through Daybreak) needs a live injected session
to confirm; logic reviewed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes it possible to isolate an input->tool problem from a tool->game one in the
Controllers panel (under Debug details):
- which backend fed each slot is already shown via the per-pad source tag
("XInput #0" / "Steam Input") in the Incoming section;
- a new "Round-trip" table shows, per slot, the state we forwarded (the active
backend's pad) next to what the game actually read back, highlighting matches.
For the round-trip, the XInput hook now echoes the state it returns to the game into
the status (protocol v9->v10: per-slot read_state in HookStatus).
Verified: hook_selftest (x64 + x86) asserts the hook records the game-read state;
full build x64 + x86 clean; ctest x64 9/9, x86 3/3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The XInput hook used to swallow XInputSetState; now it records the requested
left/right motor speeds into the status back-channel (protocol v8->v9: per-slot
rumble_left/right in HookStatus). Each frame the host reads them and, only on
change, drives the guest's actuator via the active backend:
- XInput: XInputSetState on the guest's slot (the open question is whether Steam's
RPT virtual pad accepts vibration and routes it to the guest -- needs live RPT);
- Steam Input: SteamInput TriggerVibration on the slot's controller handle, with the
XInput fallback for slots Steam isn't driving.
InputSource gains a set_rumble(slot,left,right) hook (default no-op) implemented by
both backends; SteamInputSource now tracks per-slot controller handles + which slots
it drives.
Verified: hook_selftest (x64 + x86) now asserts the hook records the rumble from
XInputSetState into the status; full build x64 + x86 clean; ctest x64 9/9, x86 3/3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forward the host window's clicks and keystrokes into the injected game so guests
can drive menus / "Press Start" / text entry that a pad can't.
Protocol (v7->v8): new HookSubsys_Mkb and an SPSC MkbRing of MkbEvents in
SharedBlock (host produces, hook consumes); push/pop helpers.
Hook (hook/src/mkb_hook.cpp, new subsystem): a worker-loop pump drains the ring at
~5 ms and PostMessageW's the matching window messages (WM_KEY*/WM_CHAR, mouse
buttons, WM_MOUSEWHEEL) to the game's main window; it also inline-hooks user32
GetAsyncKeyState / GetKeyboardState / GetCursorPos (stdcall trampolines per the x86
rule) to report a synthesized state so polling games react too. Removing the
subsystem clears all synthesized keys (no stuck input).
Host: the Injection panel gets a "Mouse + keyboard forwarding" subsystem toggle
(opt-in, default off -- the toggle is the hook). host/src/inject/mkb_forward.cpp
reads ImGui IO each frame and forwards only when the host window is focused and
ImGui isn't capturing the event; keyboard always, mouse only while mirroring (clicks
+ wheel, not movement). Mouse coords are mapped through the letterbox to game-client
space (host/src/inject/mkb_map.hpp), accounting for WGC-of-decorated-window vs
hooked/borderless. RawInput/DirectInput games are out of scope for this version.
Verified: new mkb_ring_test + mkb_map_test pass; full build x64 + x86 clean; ctest
x64 9/9 and x86 3/3 green (no regression from the protocol bump). The subsystem is
opt-in, so it can't affect existing behavior unless enabled; the end-to-end
click-into-game path needs live Remote Play + a real game to confirm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Forward synthetic test input" is a controller-debug aid, so it now lives in the
Controllers panel (gated behind Debug details, disabled until the XInput hook is
attached) instead of the Injection panel. ControllersPanel owns the flag and exposes
test_input(); main feeds it into InjectionPanel::set_test_input each frame, so the
existing synthetic-pad substitution in publish() is unchanged.
Verified: x64 build green; review (default view no longer shows it in Injection;
appears in Controllers under Debug details).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The live indicator compared frames_rendered to the previous UI frame's value, but
audio buffers release in bursts so most frames saw no change -- the cell flickered
between a green dot and grey "idle". Now each stream remembers when it last advanced
and reads "live" for a short window (0.4 s) afterwards, with a ~2 Hz frames/s
estimate next to it; otherwise "idle". Steady and readable for multi-stream games.
Verified: x64 build green. Full visual confirmation needs an injected, audio-playing
game with the per-stream table open (Debug details); logic reviewed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
