Finalize roadmap into a single execution-ordered Planned list
Merge the Tooling/UI/input section into Planned (next up) and order it for an unattended top-to-bottom run: the self-verifiable tooling/UI/input items (1-10) first, the game-pipeline items that need a real game + Remote Play to validate (11-14) last, so the top item is always the next task. Fold "per-stream audio format detection" into the multi-stream audio item (same hook/ring plumbing). Resolve the re-attach name-collision to use the picker, add hung/stalled-heartbeat detection to the terminated-state item, and retarget the DX12/multi-audio items at Spider-Man: Miles Morales (the installed, launcher-free build). Future work is now just the Vulkan and D3D9 hooked paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
100
README.md
100
README.md
@@ -70,49 +70,10 @@ doesn't (Vulkan, D3D9, DX12 — see Roadmap).
|
|||||||
|
|
||||||
### Planned (next up)
|
### Planned (next up)
|
||||||
|
|
||||||
The current focus is making specific games work end-to-end. Each item is a
|
Worked top-to-bottom: each item is a milestone with its own tests and commit, and
|
||||||
milestone with its own tests and commit.
|
is removed from this list once done — so the top item is always next. The
|
||||||
|
self-verifiable tooling / UI / input items come first; the game-pipeline items that
|
||||||
1. **Release the mouse cursor for cursor-clipping games.** Games that confine the
|
need a real game (and Remote Play) to fully validate come last.
|
||||||
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` /
|
|
||||||
per-frame `SetCursorPos` re-centering) trap the operator's mouse permanently,
|
|
||||||
because the focus spoof makes the game believe it's always focused — so the
|
|
||||||
operator can't reach the ImGui overlay. Add a cursor-release capability to the
|
|
||||||
Focus subsystem: hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the
|
|
||||||
game's clip) and the re-centering `SetCursorPos`, gated by a new host→hook flag
|
|
||||||
driven by a host toggle + hotkey. Defaults to released (the guest plays via the
|
|
||||||
pad, so the game's own cursor clip is operator-only), with the option to
|
|
||||||
re-enable clipping per game.
|
|
||||||
|
|
||||||
2. **Real capture metrics + latency stats.** The current FPS readout only measures
|
|
||||||
how fast the host renders its own window, which hides capture stutter. Add a
|
|
||||||
three-line frametime/FPS graph — **game present rate** (from `VideoShare`
|
|
||||||
present deltas), **capture rate** (generation deltas / WGC arrivals), and
|
|
||||||
**tool render rate** — plus a **capture→display latency** stat: stamp each
|
|
||||||
published frame with a `QueryPerformanceCounter` value in `VideoShare`, and the
|
|
||||||
host reports `host-present QPC − game-present QPC` (min/avg/max ms) for the
|
|
||||||
matched frame. QPC is system-wide, so the two processes' timestamps compare
|
|
||||||
directly.
|
|
||||||
|
|
||||||
3. **DX12 hooked capture.** Marvel's Spider-Man is D3D12, so the Present hook fires
|
|
||||||
but `GetBuffer(0)` as `ID3D11Texture2D` fails (the backbuffer is an
|
|
||||||
`ID3D12Resource`) and the hook idles; WGC works but stutters. Add a D3D12 path
|
|
||||||
via a **D3D11On12 bridge**: capture the game's D3D12 command queue (hook
|
|
||||||
`ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
|
|
||||||
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
|
|
||||||
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
|
|
||||||
|
|
||||||
4. **Multi-stream audio capture + mixing.** Games with several concurrent WASAPI
|
|
||||||
render streams (e.g. Spider-Man) only get their first ("primary") stream
|
|
||||||
mirrored today; the rest keep playing locally and never reach the guest. Capture
|
|
||||||
every tracked render stream into its own shared ring, silence each, and add a
|
|
||||||
host-side mixer that resamples each ring to the render format and sums them
|
|
||||||
(with soft-clip).
|
|
||||||
|
|
||||||
### Tooling, UI & input
|
|
||||||
|
|
||||||
Operability, UI, and input-debugging improvements, mostly independent of the
|
|
||||||
game-specific milestones above. Ordered roughly as requested, not by priority.
|
|
||||||
|
|
||||||
1. **Stage only deployable artifacts directly under `bin/`.** Today every target —
|
1. **Stage only deployable artifacts directly under `bin/`.** Today every target —
|
||||||
host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in
|
host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in
|
||||||
@@ -140,15 +101,18 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
|
|||||||
`WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to
|
`WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to
|
||||||
Terminated, gray out / disable the per-subsystem controls and mirror toggles, and
|
Terminated, gray out / disable the per-subsystem controls and mirror toggles, and
|
||||||
show a clear banner; the Video and Audio panels should drop to idle (their hook
|
show a clear banner; the Video and Audio panels should drop to idle (their hook
|
||||||
channels are stale) rather than freezing on the last live frame/state.
|
channels are stale) rather than freezing on the last live frame/state. Note a live
|
||||||
|
process isn't proof it's running — also flag a **stalled heartbeat** (no advance
|
||||||
|
for ~2 s while the process still exists) as a distinct "hung / not responding"
|
||||||
|
state, since games here can freeze without exiting.
|
||||||
|
|
||||||
3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid,
|
3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid,
|
||||||
but the UI still holds the stale one. In the Terminated state (task 2), remember
|
but the UI still holds the stale one. In the Terminated state (task 2), remember
|
||||||
the target's image name (the panel already keeps the selected exe name) and offer a
|
the target's image name (the panel already keeps the selected exe name) and offer a
|
||||||
**Re-attach** button that injects only if a live process with that *same name*
|
**Re-attach** button that injects only if a live process with that *same name*
|
||||||
exists, rebinding the IPC server to the new pid via the existing `inject_dll` path.
|
exists, rebinding the IPC server to the new pid via the existing `inject_dll` path.
|
||||||
*Open decision (will confirm at build time):* if several processes share the name,
|
If several live processes share that name, don't guess — surface the matches in the
|
||||||
default to the most-recently-started one with a note, or fall back to the picker.
|
picker (task 4) for a manual choice.
|
||||||
|
|
||||||
4. **Select targets by window, not just process.** A flat process list is fine as an
|
4. **Select targets by window, not just process.** A flat process list is fine as an
|
||||||
advanced/debug view, but the default should be a **window list** — there are far
|
advanced/debug view, but the default should be a **window list** — there are far
|
||||||
@@ -225,6 +189,46 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
|
|||||||
reports per-slot poll counts; extend it to echo the last state the game read so (c)
|
reports per-slot poll counts; extend it to echo the last state the game read so (c)
|
||||||
is a true round-trip.
|
is a true round-trip.
|
||||||
|
|
||||||
|
11. **Release the mouse cursor for cursor-clipping games.** Games that confine the
|
||||||
|
cursor while focused (e.g. Trails through Daybreak via `ClipCursor` / per-frame
|
||||||
|
`SetCursorPos` re-centering) trap the operator's mouse permanently, because the
|
||||||
|
focus spoof makes the game believe it's always focused — so the operator can't
|
||||||
|
reach the ImGui overlay. Add a cursor-release capability to the Focus subsystem:
|
||||||
|
hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the game's clip) and the
|
||||||
|
re-centering `SetCursorPos`, gated by a new host→hook flag driven by a host
|
||||||
|
toggle + hotkey. Defaults to released (the guest plays via the pad, so the game's
|
||||||
|
own cursor clip is operator-only), with the option to re-enable clipping per game.
|
||||||
|
|
||||||
|
12. **Real capture metrics + latency stats.** The current FPS readout only measures
|
||||||
|
how fast the host renders its own window, which hides capture stutter. Add a
|
||||||
|
three-line frametime/FPS graph — **game present rate** (from `VideoShare` present
|
||||||
|
deltas), **capture rate** (generation deltas / WGC arrivals), and **tool render
|
||||||
|
rate** — plus a **capture→display latency** stat: stamp each published frame with
|
||||||
|
a `QueryPerformanceCounter` value in `VideoShare`, and the host reports
|
||||||
|
`host-present QPC − game-present QPC` (min/avg/max ms) for the matched frame. QPC
|
||||||
|
is system-wide, so the two processes' timestamps compare directly.
|
||||||
|
|
||||||
|
13. **DX12 hooked capture (Spider-Man: Miles Morales).** Miles Morales is D3D12, so
|
||||||
|
the Present hook fires but `GetBuffer(0)` as `ID3D11Texture2D` fails (the
|
||||||
|
backbuffer is an `ID3D12Resource`) and the hook idles; WGC works but stutters. Add
|
||||||
|
a D3D12 path via a **D3D11On12 bridge**: capture the game's D3D12 command queue
|
||||||
|
(hook `ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`,
|
||||||
|
`CreateWrappedResource` around the backbuffer, and `CopyResource` into the
|
||||||
|
*existing* D3D11 shared keyed-mutex texture — so the host side is unchanged.
|
||||||
|
|
||||||
|
14. **Multi-stream audio capture + mixing, with per-stream format detection.** Games
|
||||||
|
with several concurrent WASAPI render streams (e.g. Miles Morales) only get their
|
||||||
|
first ("primary") stream mirrored today; the rest keep playing locally and never
|
||||||
|
reach the guest. Capture every tracked render stream into its own shared ring,
|
||||||
|
silence each, and add a host-side mixer that resamples each ring to the render
|
||||||
|
format and sums them (with soft-clip). **Fold in real per-stream format
|
||||||
|
detection** here, since it touches the same hook + ring plumbing: a stream that
|
||||||
|
already existed when we injected is never seen at `Initialize`, so the hook
|
||||||
|
currently assumes the device **mix format** and a shared-mode stream opened at a
|
||||||
|
different format comes out wrong-pitched. Resolve each stream's true format (the
|
||||||
|
stream's own `Initialize` when caught, else the original `GetMixFormat`) so every
|
||||||
|
mixed ring is pitched correctly.
|
||||||
|
|
||||||
### Future work
|
### Future work
|
||||||
|
|
||||||
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
|
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
|
||||||
@@ -232,10 +236,6 @@ game-specific milestones above. Ordered roughly as requested, not by priority.
|
|||||||
readable image. Use WGC in the meantime.
|
readable image. Use WGC in the meantime.
|
||||||
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
|
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
|
||||||
hook would be the lower-latency upgrade.
|
hook would be the lower-latency upgrade.
|
||||||
- **Per-stream audio format detection.** A render stream that already exists when
|
|
||||||
we inject is never seen at `Initialize`, so the hook assumes the device **mix
|
|
||||||
format**. A stream initialized in shared mode at a different format would come
|
|
||||||
out wrong-pitched. Detecting the real per-stream format would remove that guess.
|
|
||||||
|
|
||||||
## Building
|
## Building
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user