Roadmap: sequence backend work per-API (mock + capture together) + UI-fit pass

Restructure the roadmap into one "Current work" section where each rendering
API is a single milestone that builds its mock-game backend and the injected
capture for that same API side by side, so each API reaches verified
end-to-end before the next. Adds an M1 end-user UI fit pass (drive the overlay
to maximum info via the harness, screenshot, assert every panel fits its
window; resize/rearrange otherwise) kept green by every later milestone.
Folds in DX10, DX9 (D3D9Ex + plain non-Ex), OpenGL, and Vulkan (best-effort
inject + opt-in implicit layer, too-late red prompt). Drops the old Vulkan/
D3D9 future-work bullets.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-22 10:47:53 +02:00
parent 2d65961bf2
commit fc43355db2

127
README.md
View File

@@ -95,13 +95,130 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
## Roadmap ## Roadmap
### Current work — per-API capture, end to end
Each rendering API is built as **one milestone**: first its `coop_mock_game` backend (an
animated, frame-numbered A/V source), then the injected **capture** for that same API
directly after — so every API reaches *verified end-to-end* (the mirror decodes the frame
counter and asserts a *monotonic, advancing* sequence) before the next one starts, rather
than building all the mock backends first and all the capture later. Order is
easiest-to-hardest, dependencies last.
Conventions for every milestone below:
- Each mock backend implements the existing
[`RenderBackend`](tools/mock_game/render_backend.hpp) interface (animated background +
moving bar + frame-counter block), is selectable on the command line, and renders
**clear/fill-only** — no geometry, no shaders, so **no shader-compiler dependency**.
- Sub-steps are **separate commits**; the README, CMake, and (where needed) `.gitmodules`
move with each. A mock-backend commit lands with a *liveness* smoke check (launches,
presents N frames, exits clean); the frame-accurate decode-through-the-hook assertion
lands with that API's capture commit right after.
- The host samples the **standard shared keyed-mutex texture** unchanged regardless of
source API: every backend publishes into it — native D3D11 on the game's device, D3D12 via
D3D11On12, and D3D9 / D3D10 / OpenGL / Vulkan via a **hook-owned D3D11 device**.
- New submodules are **not** auto-cloned — CMake checks each is populated and stops with a
`FATAL_ERROR` naming `git submodule update --init --recursive` (a reusable
`coop_require_submodule()` helper).
- **M1 — End-user UI fit pass.** Verify the shipping ImGui overlay (the *end-user* view, not
just the dev layout) with the debug-driving harness + F10 screenshot, and guarantee **every
panel fits its assigned window size even with Debug details enabled** — the
maximum-information case. Drive each panel to its fullest state via the harness (inject,
enable audio + video, expand Debug details, show the override controls, multiple audio
streams, the longest status / fallback strings), screenshot, and check nothing is clipped
or scrolled out of view. Where the densest case overflows, **resize and/or rearrange** the
panel so it fits (content can be moved between columns / rows — the most detailed case
should still fit). Land an automated harness check (drive-to-max → screenshot → assert no
overflow) so later milestones that add UI keep it green. *Independent of the backend work;
every milestone below must preserve this test* (M5's red banner and Vulkan-layer checkbox
in particular).
- **M2 — DX10 (mock → capture).**
1. **Mock backend** (`render_dx10.cpp`). `ID3D10Device` + DXGI swap chain; Windows SDK only
(`d3d10`, `dxgi`), **no new deps**. Sub-region fills (bar, counter block) via
`CopySubresourceRegion` of small solid-colour textures (DX10 has no clear-rect). Top-left
origin, so the counter block maps straight to the capture's sample point.
2. **Capture.** The DXGI `Present`/`Present1` hook already catches D3D10 swap chains, but the
copy QIs the backbuffer to `ID3D11Texture2D`, which a *pure* D3D10 device fails. Share the
D3D10 backbuffer into the hook's **own D3D11 device** (legacy shared handle) and copy it
into the standard keyed-mutex texture; add a `dx10_present_hook_test` that decodes the
mock's frames. Validates the "D3D10/11" claim the Architecture section currently makes
untested.
- **M3 — DX9 (mock → D3D9Ex capture → plain-D3D9 capture).**
1. **Mock backend** (`render_dx09.cpp`). `IDirect3DDevice9` / `IDirect3DDevice9Ex` + present;
Windows SDK only (`d3d9`), **no new deps**. `Clear` for the background and `ColorFill` for
the bar + block (D3D9's built-in rect fill — exactly the primitive DX10/11 lack). Top-left
origin. Runs in **two selectable modes** — `dx9ex` (default) and plain `dx9`
(`Direct3DCreate9`) — so both capture paths have a matching game.
2. **D3D9Ex capture (clean GPU path).** Hook `IDirect3DDevice9::Present` by vtable — discover
the slot from a throwaway `Direct3DCreate9Ex` device (`stdcall()` on x86 per the SafetyHook
trap); the vtable is shared across all devices of the class, so the game's existing device
is caught → **late-attach works**. On Present, `GetBackBuffer(0)` + `StretchRect` (GPU-side)
into a `CreateRenderTarget(..., pSharedHandle=&h)` surface (`D3DPOOL_DEFAULT`); share that
into the hook's own D3D11 device and copy into the standard keyed-mutex texture (host
unchanged, proper keyed-mutex sync). `A8R8G8B8` is **BGRA** → record the swizzle. Re-create
the surface on `Reset` / resize. `d3d9_present_hook_test` decodes the mock's frames (Ex mode).
3. **Plain (non-Ex) D3D9 capture (slow path).** Plain `Direct3DCreate9` devices **cannot**
produce a D3D11-shareable surface (WGC remains their default mirror until this lands).
Detect via a failed `QueryInterface(IID_IDirect3DDevice9Ex)` (else take the D3D9Ex path). On
Present, `GetRenderTargetData` the backbuffer into a
`CreateOffscreenPlainSurface(..., D3DPOOL_SYSTEMMEM)` surface, `LockRect`, copy out
(respect `Pitch`; BGRA→RGBA), and upload into the standard keyed-mutex texture via the
hook's own D3D11 device (`Map` / `UpdateSubresource`) — so only *how pixels reach the
texture* differs (CPU copy, not GPU). `GetRenderTargetData` is a GPU→sysmem stall, so
**drop / throttle** mirror frames rather than back-pressure the game. Same test, plain mode.
- **M4 — OpenGL (mock → capture coverage).**
1. **Mock backend** (`render_gl.cpp`). Raw WGL context (`wglCreateContextAttribsARB`) with the
**glad** loader (new submodule, `Dav1dde/glad`). Background via `glClearColor`/`glClear`;
bar + block via `glScissor` + clear (shader-free GL 1.x). GL's framebuffer is
**bottom-left** origin, so the counter block is placed flipped so the captured (top-left)
pixel still decodes — matching the existing `glReadPixels` capture flip. Adds the
`.gitmodules` entry + the `coop_require_submodule()` check.
2. **Capture coverage.** The GL `SwapBuffers`/`wglSwapBuffers` + `glReadPixels` path already
ships; add a mock-backed regression that decodes the mock's frames through it (upgrading the
synthetic `opengl_hook_test` to a real animated game). Small.
- **M5 — Vulkan (mock → capture).** The largest.
1. **Mock backend** (`render_vk.cpp`). New submodules **Vulkan-Headers**
(`KhronosGroup/Vulkan-Headers`, official) + **volk** (`zeux/volk`); raw
`vkCreateWin32SurfaceKHR`, swap chain, per-frame acquire → clear → present. Background via
`vkCmdClearColorImage`, bar + block via `vkCmdClearAttachments` clear-rects (no
pipeline/shaders → **no SPIR-V toolchain**). Loads Vulkan via **volk** on purpose — that's
the loader-bypass case the capture hook must handle. Validation layers used when a Vulkan
SDK is present, skipped otherwise.
2. **Capture.** Hook `vkQueuePresentKHR`; import a D3D11 keyed-mutex shared texture into Vulkan
(`VK_KHR_external_memory_win32` + `VK_KHR_win32_keyed_mutex`) and `vkCmdBlitImage` the
swap-chain image into it each present; the host samples it as usual. Because Vulkan caches
its present pointer at init, the hook **cannot be placed by late injection** — it must be
present before `vkCreateInstance`:
- **Attach via auto-attach — best-effort by default, opt-in layer for reliability.** By
default, arming Auto-attach for a detected Vulkan game injects as early as possible on
relaunch (the existing poll-and-inject) — enough for games that initialize Vulkan a little
into startup. For games that init Vulkan immediately, an **opt-in** Injection-panel
checkbox ("Set up Vulkan layer") registers a per-user (HKCU, no admin) implicit Vulkan
layer (thin `coop_vk_layer.dll` + JSON manifest) scoped to that game's image name; on
relaunch the loader loads it at `vkCreateInstance` — guaranteed before init — and it wires
up present capture + IPC. The layer is removed when the option is unticked or the host
exits, and self-deactivates for any non-target app, so a stale registration (e.g. after a
host crash) is harmless.
- **Too-late detection + red prompt.** A late-injected hook that finds `vulkan-1.dll` loaded
but no working hooked-present reports a *"Vulkan, injected too late"* status; the host
overlays a **red** banner on the mirror window: "Detected a Vulkan game — the hook attached
too late to mirror it with low latency. Enable Auto-attach (and, if it persists, 'Set up
Vulkan layer') and relaunch the game." **WGC keeps mirroring meanwhile** so the session
stays usable; the banner clears once the hooked path goes live. (New end-user UI → keep
M1's fit test green.)
- Sub-steps (separate commits): present-hook placement (best-effort inject + the implicit
layer), external-memory image import + keyed-mutex sync, the too-late status code, the
host red-banner UI, and the Auto-attach / Vulkan-layer Injection-panel controls. Test the
early (layer) path against the mock (decode frames) and the too-late path (assert the
status + banner fire).
### Future work ### Future work
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
them needs a Vulkan layer / device-dispatch hook plus a `vkCmdCopyImage` to a
readable image. Use WGC in the meantime.
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
hook would be the lower-latency upgrade.
- **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB - **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB
subsystem forwards via window messages (`PostMessage`) plus synthesized subsystem forwards via window messages (`PostMessage`) plus synthesized
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop `GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop