diff --git a/README.md b/README.md index e835b85..fa9d36f 100644 --- a/README.md +++ b/README.md @@ -70,49 +70,10 @@ doesn't (Vulkan, D3D9, DX12 — see Roadmap). ### Planned (next up) -The current focus is making specific games work end-to-end. Each item is a -milestone with its own tests and commit. - -1. **Release the mouse cursor for cursor-clipping games.** Games that confine the - cursor while focused (e.g. Trails through Daybreak via `ClipCursor` / - per-frame `SetCursorPos` re-centering) trap the operator's mouse permanently, - because the focus spoof makes the game believe it's always focused — so the - operator can't reach the ImGui overlay. Add a cursor-release capability to the - Focus subsystem: hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the - game's clip) and the re-centering `SetCursorPos`, gated by a new host→hook flag - driven by a host toggle + hotkey. Defaults to released (the guest plays via the - pad, so the game's own cursor clip is operator-only), with the option to - re-enable clipping per game. - -2. **Real capture metrics + latency stats.** The current FPS readout only measures - how fast the host renders its own window, which hides capture stutter. Add a - three-line frametime/FPS graph — **game present rate** (from `VideoShare` - present deltas), **capture rate** (generation deltas / WGC arrivals), and - **tool render rate** — plus a **capture→display latency** stat: stamp each - published frame with a `QueryPerformanceCounter` value in `VideoShare`, and the - host reports `host-present QPC − game-present QPC` (min/avg/max ms) for the - matched frame. QPC is system-wide, so the two processes' timestamps compare - directly. - -3. **DX12 hooked capture.** Marvel's Spider-Man is D3D12, so the Present hook fires - but `GetBuffer(0)` as `ID3D11Texture2D` fails (the backbuffer is an - `ID3D12Resource`) and the hook idles; WGC works but stutters. Add a D3D12 path - via a **D3D11On12 bridge**: capture the game's D3D12 command queue (hook - `ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`, - `CreateWrappedResource` around the backbuffer, and `CopyResource` into the - *existing* D3D11 shared keyed-mutex texture — so the host side is unchanged. - -4. **Multi-stream audio capture + mixing.** Games with several concurrent WASAPI - render streams (e.g. Spider-Man) only get their first ("primary") stream - mirrored today; the rest keep playing locally and never reach the guest. Capture - every tracked render stream into its own shared ring, silence each, and add a - host-side mixer that resamples each ring to the render format and sums them - (with soft-clip). - -### Tooling, UI & input - -Operability, UI, and input-debugging improvements, mostly independent of the -game-specific milestones above. Ordered roughly as requested, not by priority. +Worked top-to-bottom: each item is a milestone with its own tests and commit, and +is removed from this list once done — so the top item is always next. The +self-verifiable tooling / UI / input items come first; the game-pipeline items that +need a real game (and Remote Play) to fully validate come last. 1. **Stage only deployable artifacts directly under `bin/`.** Today every target — host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in @@ -140,15 +101,18 @@ game-specific milestones above. Ordered roughly as requested, not by priority. `WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to Terminated, gray out / disable the per-subsystem controls and mirror toggles, and show a clear banner; the Video and Audio panels should drop to idle (their hook - channels are stale) rather than freezing on the last live frame/state. + channels are stale) rather than freezing on the last live frame/state. Note a live + process isn't proof it's running — also flag a **stalled heartbeat** (no advance + for ~2 s while the process still exists) as a distinct "hung / not responding" + state, since games here can freeze without exiting. 3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid, but the UI still holds the stale one. In the Terminated state (task 2), remember the target's image name (the panel already keeps the selected exe name) and offer a **Re-attach** button that injects only if a live process with that *same name* exists, rebinding the IPC server to the new pid via the existing `inject_dll` path. - *Open decision (will confirm at build time):* if several processes share the name, - default to the most-recently-started one with a note, or fall back to the picker. + If several live processes share that name, don't guess — surface the matches in the + picker (task 4) for a manual choice. 4. **Select targets by window, not just process.** A flat process list is fine as an advanced/debug view, but the default should be a **window list** — there are far @@ -225,6 +189,46 @@ game-specific milestones above. Ordered roughly as requested, not by priority. reports per-slot poll counts; extend it to echo the last state the game read so (c) is a true round-trip. +11. **Release the mouse cursor for cursor-clipping games.** Games that confine the + cursor while focused (e.g. Trails through Daybreak via `ClipCursor` / per-frame + `SetCursorPos` re-centering) trap the operator's mouse permanently, because the + focus spoof makes the game believe it's always focused — so the operator can't + reach the ImGui overlay. Add a cursor-release capability to the Focus subsystem: + hook `ClipCursor` (force `ClipCursor(NULL)` and swallow the game's clip) and the + re-centering `SetCursorPos`, gated by a new host→hook flag driven by a host + toggle + hotkey. Defaults to released (the guest plays via the pad, so the game's + own cursor clip is operator-only), with the option to re-enable clipping per game. + +12. **Real capture metrics + latency stats.** The current FPS readout only measures + how fast the host renders its own window, which hides capture stutter. Add a + three-line frametime/FPS graph — **game present rate** (from `VideoShare` present + deltas), **capture rate** (generation deltas / WGC arrivals), and **tool render + rate** — plus a **capture→display latency** stat: stamp each published frame with + a `QueryPerformanceCounter` value in `VideoShare`, and the host reports + `host-present QPC − game-present QPC` (min/avg/max ms) for the matched frame. QPC + is system-wide, so the two processes' timestamps compare directly. + +13. **DX12 hooked capture (Spider-Man: Miles Morales).** Miles Morales is D3D12, so + the Present hook fires but `GetBuffer(0)` as `ID3D11Texture2D` fails (the + backbuffer is an `ID3D12Resource`) and the hook idles; WGC works but stutters. Add + a D3D12 path via a **D3D11On12 bridge**: capture the game's D3D12 command queue + (hook `ID3D12CommandQueue::ExecuteCommandLists`), create an `ID3D11On12Device`, + `CreateWrappedResource` around the backbuffer, and `CopyResource` into the + *existing* D3D11 shared keyed-mutex texture — so the host side is unchanged. + +14. **Multi-stream audio capture + mixing, with per-stream format detection.** Games + with several concurrent WASAPI render streams (e.g. Miles Morales) only get their + first ("primary") stream mirrored today; the rest keep playing locally and never + reach the guest. Capture every tracked render stream into its own shared ring, + silence each, and add a host-side mixer that resamples each ring to the render + format and sums them (with soft-clip). **Fold in real per-stream format + detection** here, since it touches the same hook + ring plumbing: a stream that + already existed when we injected is never seen at `Initialize`, so the hook + currently assumes the device **mix format** and a shared-mode stream opened at a + different format comes out wrong-pitched. Resolve each stream's true format (the + stream's own `Initialize` when caught, else the original `GetMixFormat`) so every + mixed ring is pitched correctly. + ### Future work - **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking @@ -232,10 +236,6 @@ game-specific milestones above. Ordered roughly as requested, not by priority. readable image. Use WGC in the meantime. - **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present` hook would be the lower-latency upgrade. -- **Per-stream audio format detection.** A render stream that already exists when - we inject is never seen at `Initialize`, so the hook assumes the device **mix - format**. A stream initialized in shared mode at a different format would come - out wrong-pitched. Detecting the real per-stream format would remove that guess. ## Building