Finalize roadmap into a single execution-ordered Planned list
Merge the Tooling/UI/input section into Planned (next up) and order it for an unattended top-to-bottom run: the self-verifiable tooling/UI/input items (1-10) first, the game-pipeline items that need a real game + Remote Play to validate (11-14) last, so the top item is always the next task. Fold "per-stream audio format detection" into the multi-stream audio item (same hook/ring plumbing). Resolve the re-attach name-collision to use the picker, add hung/stalled-heartbeat detection to the terminated-state item, and retarget the DX12/multi-audio items at Spider-Man: Miles Morales (the installed, launcher-free build). Future work is now just the Vulkan and D3D9 hooked paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
100
README.md
100
README.md
@@ -70,49 +70,10 @@ doesn't (Vulkan, D3D9, DX12 — see Roadmap).
|
||||
|
||||
### Planned (next up)
|
||||
|
||||
The current focus is making specific games work end-to-end. Each item is a
|
||||
milestone with its own tests and commit.
|
||||
|
||||
1. **Release the mouse cursor for cursor-clipping games.** Games that confine the
|
||||
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` /
|
||||
per-frame `SetCursorPos` re-centering) trap the operator's mouse permanently,
|
||||
because the focus spoof makes the game believe it's always focused — so the
|
||||
operator can't reach the ImGui overlay. Add a cursor-release capability to the
|
||||
Focus subsystem: hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the
|
||||
game's clip) and the re-centering `SetCursorPos`, gated by a new host→hook flag
|
||||
driven by a host toggle + hotkey. Defaults to released (the guest plays via the
|
||||
pad, so the game's own cursor clip is operator-only), with the option to
|
||||
re-enable clipping per game.
|
||||
|
||||
2. **Real capture metrics + latency stats.** The current FPS readout only measures
|
||||
how fast the host renders its own window, which hides capture stutter. Add a
|
||||
three-line frametime/FPS graph — **game present rate** (from `VideoShare`
|
||||
present deltas), **capture rate** (generation deltas / WGC arrivals), and
|
||||
**tool render rate** — plus a **capture→display latency** stat: stamp each
|
||||
published frame with a `QueryPerformanceCounter` value in `VideoShare`, and the
|
||||
host reports `host-present QPC − game-present QPC` (min/avg/max ms) for the
|
||||
matched frame. QPC is system-wide, so the two processes' timestamps compare
|
||||
directly.
|
||||
|
||||
3. **DX12 hooked capture.** Marvel's Spider-Man is D3D12, so the Present hook fires
|
||||
but `GetBuffer(0)` as `ID3D11Texture2D` fails (the backbuffer is an
|
||||
`ID3D12Resource`) and the hook idles; WGC works but stutters. Add a D3D12 path
|
||||
via a **D3D11On12 bridge**: capture the game's D3D12 command queue (hook
|
||||
`ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
|
||||
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
|
||||
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
|
||||
|
||||
4. **Multi-stream audio capture + mixing.** Games with several concurrent WASAPI
|
||||
render streams (e.g. Spider-Man) only get their first ("primary") stream
|
||||
mirrored today; the rest keep playing locally and never reach the guest. Capture
|
||||
every tracked render stream into its own shared ring, silence each, and add a
|
||||
host-side mixer that resamples each ring to the render format and sums them
|
||||
(with soft-clip).
|
||||
|
||||
### Tooling, UI & input
|
||||
|
||||
Operability, UI, and input-debugging improvements, mostly independent of the
|
||||
game-specific milestones above. Ordered roughly as requested, not by priority.
|
||||
Worked top-to-bottom: each item is a milestone with its own tests and commit, and
|
||||
is removed from this list once done — so the top item is always next. The
|
||||
self-verifiable tooling / UI / input items come first; the game-pipeline items that
|
||||
need a real game (and Remote Play) to fully validate come last.
|
||||
|
||||
1. **Stage only deployable artifacts directly under `bin/`.** Today every target —
|
||||
host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in
|
||||
@@ -140,15 +101,18 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
|
||||
`WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to
|
||||
Terminated, gray out / disable the per-subsystem controls and mirror toggles, and
|
||||
show a clear banner; the Video and Audio panels should drop to idle (their hook
|
||||
channels are stale) rather than freezing on the last live frame/state.
|
||||
channels are stale) rather than freezing on the last live frame/state. Note a live
|
||||
process isn't proof it's running — also flag a **stalled heartbeat** (no advance
|
||||
for ~2 s while the process still exists) as a distinct "hung / not responding"
|
||||
state, since games here can freeze without exiting.
|
||||
|
||||
3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid,
|
||||
but the UI still holds the stale one. In the Terminated state (task 2), remember
|
||||
the target's image name (the panel already keeps the selected exe name) and offer a
|
||||
**Re-attach** button that injects only if a live process with that *same name*
|
||||
exists, rebinding the IPC server to the new pid via the existing `inject_dll` path.
|
||||
*Open decision (will confirm at build time):* if several processes share the name,
|
||||
default to the most-recently-started one with a note, or fall back to the picker.
|
||||
If several live processes share that name, don't guess — surface the matches in the
|
||||
picker (task 4) for a manual choice.
|
||||
|
||||
4. **Select targets by window, not just process.** A flat process list is fine as an
|
||||
advanced/debug view, but the default should be a **window list** — there are far
|
||||
@@ -225,6 +189,46 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
|
||||
reports per-slot poll counts; extend it to echo the last state the game read so (c)
|
||||
is a true round-trip.
|
||||
|
||||
11. **Release the mouse cursor for cursor-clipping games.** Games that confine the
|
||||
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` / per-frame
|
||||
`SetCursorPos` re-centering) trap the operator's mouse permanently, because the
|
||||
focus spoof makes the game believe it's always focused — so the operator can't
|
||||
reach the ImGui overlay. Add a cursor-release capability to the Focus subsystem:
|
||||
hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the game's clip) and the
|
||||
re-centering `SetCursorPos`, gated by a new host→hook flag driven by a host
|
||||
toggle + hotkey. Defaults to released (the guest plays via the pad, so the game's
|
||||
own cursor clip is operator-only), with the option to re-enable clipping per game.
|
||||
|
||||
12. **Real capture metrics + latency stats.** The current FPS readout only measures
|
||||
how fast the host renders its own window, which hides capture stutter. Add a
|
||||
three-line frametime/FPS graph — **game present rate** (from `VideoShare` present
|
||||
deltas), **capture rate** (generation deltas / WGC arrivals), and **tool render
|
||||
rate** — plus a **capture→display latency** stat: stamp each published frame with
|
||||
a `QueryPerformanceCounter` value in `VideoShare`, and the host reports
|
||||
`host-present QPC − game-present QPC` (min/avg/max ms) for the matched frame. QPC
|
||||
is system-wide, so the two processes' timestamps compare directly.
|
||||
|
||||
13. **DX12 hooked capture (Spider-Man: Miles Morales).** Miles Morales is D3D12, so
|
||||
the Present hook fires but `GetBuffer(0)` as `ID3D11Texture2D` fails (the
|
||||
backbuffer is an `ID3D12Resource`) and the hook idles; WGC works but stutters. Add
|
||||
a D3D12 path via a **D3D11On12 bridge**: capture the game's D3D12 command queue
|
||||
(hook `ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
|
||||
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
|
||||
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
|
||||
|
||||
14. **Multi-stream audio capture + mixing, with per-stream format detection.** Games
|
||||
with several concurrent WASAPI render streams (e.g. Miles Morales) only get their
|
||||
first ("primary") stream mirrored today; the rest keep playing locally and never
|
||||
reach the guest. Capture every tracked render stream into its own shared ring,
|
||||
silence each, and add a host-side mixer that resamples each ring to the render
|
||||
format and sums them (with soft-clip). **Fold in real per-stream format
|
||||
detection** here, since it touches the same hook + ring plumbing: a stream that
|
||||
already existed when we injected is never seen at `Initialize`, so the hook
|
||||
currently assumes the device **mix format** and a shared-mode stream opened at a
|
||||
different format comes out wrong-pitched. Resolve each stream's true format (the
|
||||
stream's own `Initialize` when caught, else the original `GetMixFormat`) so every
|
||||
mixed ring is pitched correctly.
|
||||
|
||||
### Future work
|
||||
|
||||
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
|
||||
@@ -232,10 +236,6 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
|
||||
readable image. Use WGC in the meantime.
|
||||
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
|
||||
hook would be the lower-latency upgrade.
|
||||
- **Per-stream audio format detection.** A render stream that already exists when
|
||||
we inject is never seen at `Initialize`, so the hook assumes the device **mix
|
||||
format**. A stream initialized in shared mode at a different format would come
|
||||
out wrong-pitched. Detecting the real per-stream format would remove that guess.
|
||||
|
||||
## Building
|
||||
|
||||
|
||||
Reference in New Issue
Block a user