Finalize roadmap into a single execution-ordered Planned list

Merge the Tooling/UI/input section into Planned (next up) and order it for an
unattended top-to-bottom run: the self-verifiable tooling/UI/input items (1-10)
first, the game-pipeline items that need a real game + Remote Play to validate
(11-14) last, so the top item is always the next task. Fold "per-stream audio
format detection" into the multi-stream audio item (same hook/ring plumbing).
Resolve the re-attach name-collision to use the picker, add hung/stalled-heartbeat
detection to the terminated-state item, and retarget the DX12/multi-audio items at
Spider-Man: Miles Morales (the installed, launcher-free build). Future work is now
just the Vulkan and D3D9 hooked paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 00:38:59 +02:00
parent dcde2a66ac
commit e9d1253b8f

100
README.md
View File

@@ -70,49 +70,10 @@ doesn't (Vulkan, D3D9, DX12 — see Roadmap).
### Planned (next up) ### Planned (next up)
The current focus is making specific games work end-to-end. Each item is a Worked top-to-bottom: each item is a milestone with its own tests and commit, and
milestone with its own tests and commit. is removed from this list once done — so the top item is always next. The
self-verifiable tooling / UI / input items come first; the game-pipeline items that
1. **Release the mouse cursor for cursor-clipping games.** Games that confine the need a real game (and Remote Play) to fully validate come last.
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` /
per-frame `SetCursorPos` re-centering) trap the operator's mouse permanently,
because the focus spoof makes the game believe it's always focused — so the
operator can't reach the ImGui overlay. Add a cursor-release capability to the
Focus subsystem: hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the
game's clip) and the re-centering `SetCursorPos`, gated by a new host→hook flag
driven by a host toggle + hotkey. Defaults to released (the guest plays via the
pad, so the game's own cursor clip is operator-only), with the option to
re-enable clipping per game.
2. **Real capture metrics + latency stats.** The current FPS readout only measures
how fast the host renders its own window, which hides capture stutter. Add a
three-line frametime/FPS graph — **game present rate** (from `VideoShare`
present deltas), **capture rate** (generation deltas / WGC arrivals), and
**tool render rate** — plus a **capture→display latency** stat: stamp each
published frame with a `QueryPerformanceCounter` value in `VideoShare`, and the
host reports `host-present QPC − game-present QPC` (min/avg/max ms) for the
matched frame. QPC is system-wide, so the two processes' timestamps compare
directly.
3. **DX12 hooked capture.** Marvel's Spider-Man is D3D12, so the Present hook fires
but `GetBuffer(0)` as `ID3D11Texture2D` fails (the backbuffer is an
`ID3D12Resource`) and the hook idles; WGC works but stutters. Add a D3D12 path
via a **D3D11On12 bridge**: capture the game's D3D12 command queue (hook
`ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
4. **Multi-stream audio capture + mixing.** Games with several concurrent WASAPI
render streams (e.g. Spider-Man) only get their first ("primary") stream
mirrored today; the rest keep playing locally and never reach the guest. Capture
every tracked render stream into its own shared ring, silence each, and add a
host-side mixer that resamples each ring to the render format and sums them
(with soft-clip).
### Tooling, UI & input
Operability, UI, and input-debugging improvements, mostly independent of the
game-specific milestones above. Ordered roughly as requested, not by priority.
1. **Stage only deployable artifacts directly under `bin/`.** Today every target — 1. **Stage only deployable artifacts directly under `bin/`.** Today every target —
host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in
@@ -140,15 +101,18 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
`WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to `WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to
Terminated, gray out / disable the per-subsystem controls and mirror toggles, and Terminated, gray out / disable the per-subsystem controls and mirror toggles, and
show a clear banner; the Video and Audio panels should drop to idle (their hook show a clear banner; the Video and Audio panels should drop to idle (their hook
channels are stale) rather than freezing on the last live frame/state. channels are stale) rather than freezing on the last live frame/state. Note a live
process isn't proof it's running — also flag a **stalled heartbeat** (no advance
for ~2 s while the process still exists) as a distinct "hung / not responding"
state, since games here can freeze without exiting.
3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid, 3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid,
but the UI still holds the stale one. In the Terminated state (task 2), remember but the UI still holds the stale one. In the Terminated state (task 2), remember
the target's image name (the panel already keeps the selected exe name) and offer a the target's image name (the panel already keeps the selected exe name) and offer a
**Re-attach** button that injects only if a live process with that *same name* **Re-attach** button that injects only if a live process with that *same name*
exists, rebinding the IPC server to the new pid via the existing `inject_dll` path. exists, rebinding the IPC server to the new pid via the existing `inject_dll` path.
*Open decision (will confirm at build time):* if several processes share the name, If several live processes share that name, don't guess — surface the matches in the
default to the most-recently-started one with a note, or fall back to the picker. picker (task 4) for a manual choice.
4. **Select targets by window, not just process.** A flat process list is fine as an 4. **Select targets by window, not just process.** A flat process list is fine as an
advanced/debug view, but the default should be a **window list** — there are far advanced/debug view, but the default should be a **window list** — there are far
@@ -225,6 +189,46 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
reports per-slot poll counts; extend it to echo the last state the game read so (c) reports per-slot poll counts; extend it to echo the last state the game read so (c)
is a true round-trip. is a true round-trip.
11. **Release the mouse cursor for cursor-clipping games.** Games that confine the
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` / per-frame
`SetCursorPos` re-centering) trap the operator's mouse permanently, because the
focus spoof makes the game believe it's always focused — so the operator can't
reach the ImGui overlay. Add a cursor-release capability to the Focus subsystem:
hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the game's clip) and the
re-centering `SetCursorPos`, gated by a new host→hook flag driven by a host
toggle + hotkey. Defaults to released (the guest plays via the pad, so the game's
own cursor clip is operator-only), with the option to re-enable clipping per game.
12. **Real capture metrics + latency stats.** The current FPS readout only measures
how fast the host renders its own window, which hides capture stutter. Add a
three-line frametime/FPS graph — **game present rate** (from `VideoShare` present
deltas), **capture rate** (generation deltas / WGC arrivals), and **tool render
rate** — plus a **capture→display latency** stat: stamp each published frame with
a `QueryPerformanceCounter` value in `VideoShare`, and the host reports
`host-present QPC − game-present QPC` (min/avg/max ms) for the matched frame. QPC
is system-wide, so the two processes' timestamps compare directly.
13. **DX12 hooked capture (Spider-Man: Miles Morales).** Miles Morales is D3D12, so
the Present hook fires but `GetBuffer(0)` as `ID3D11Texture2D` fails (the
backbuffer is an `ID3D12Resource`) and the hook idles; WGC works but stutters. Add
a D3D12 path via a **D3D11On12 bridge**: capture the game's D3D12 command queue
(hook `ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
14. **Multi-stream audio capture + mixing, with per-stream format detection.** Games
with several concurrent WASAPI render streams (e.g. Miles Morales) only get their
first ("primary") stream mirrored today; the rest keep playing locally and never
reach the guest. Capture every tracked render stream into its own shared ring,
silence each, and add a host-side mixer that resamples each ring to the render
format and sums them (with soft-clip). **Fold in real per-stream format
detection** here, since it touches the same hook + ring plumbing: a stream that
already existed when we injected is never seen at `Initialize`, so the hook
currently assumes the device **mix format** and a shared-mode stream opened at a
different format comes out wrong-pitched. Resolve each stream's true format (the
stream's own `Initialize` when caught, else the original `GetMixFormat`) so every
mixed ring is pitched correctly.
### Future work ### Future work
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking - **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
@@ -232,10 +236,6 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
readable image. Use WGC in the meantime. readable image. Use WGC in the meantime.
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present` - **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
hook would be the lower-latency upgrade. hook would be the lower-latency upgrade.
- **Per-stream audio format detection.** A render stream that already exists when
we inject is never seen at `Initialize`, so the hook assumes the device **mix
format**. A stream initialized in shared mode at a different format would come
out wrong-pitched. Detecting the real per-stream format would remove that guess.
## Building ## Building