Roadmap: sequence backend work per-API (mock + capture together) + UI-fit pass
Restructure the roadmap into one "Current work" section where each rendering API is a single milestone that builds its mock-game backend and the injected capture for that same API side by side, so each API reaches verified end-to-end before the next. Adds an M1 end-user UI fit pass (drive the overlay to maximum info via the harness, screenshot, assert every panel fits its window; resize/rearrange otherwise) kept green by every later milestone. Folds in DX10, DX9 (D3D9Ex + plain non-Ex), OpenGL, and Vulkan (best-effort inject + opt-in implicit layer, too-late red prompt). Drops the old Vulkan/ D3D9 future-work bullets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
127
README.md
127
README.md
@@ -95,13 +95,130 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap).
|
|||||||
|
|
||||||
## Roadmap
|
## Roadmap
|
||||||
|
|
||||||
|
### Current work — per-API capture, end to end
|
||||||
|
|
||||||
|
Each rendering API is built as **one milestone**: first its `coop_mock_game` backend (an
|
||||||
|
animated, frame-numbered A/V source), then the injected **capture** for that same API
|
||||||
|
directly after — so every API reaches *verified end-to-end* (the mirror decodes the frame
|
||||||
|
counter and asserts a *monotonic, advancing* sequence) before the next one starts, rather
|
||||||
|
than building all the mock backends first and all the capture later. Order is
|
||||||
|
easiest-to-hardest, dependencies last.
|
||||||
|
|
||||||
|
Conventions for every milestone below:
|
||||||
|
|
||||||
|
- Each mock backend implements the existing
|
||||||
|
[`RenderBackend`](tools/mock_game/render_backend.hpp) interface (animated background +
|
||||||
|
moving bar + frame-counter block), is selectable on the command line, and renders
|
||||||
|
**clear/fill-only** — no geometry, no shaders, so **no shader-compiler dependency**.
|
||||||
|
- Sub-steps are **separate commits**; the README, CMake, and (where needed) `.gitmodules`
|
||||||
|
move with each. A mock-backend commit lands with a *liveness* smoke check (launches,
|
||||||
|
presents N frames, exits clean); the frame-accurate decode-through-the-hook assertion
|
||||||
|
lands with that API's capture commit right after.
|
||||||
|
- The host samples the **standard shared keyed-mutex texture** unchanged regardless of
|
||||||
|
source API: every backend publishes into it — native D3D11 on the game's device, D3D12 via
|
||||||
|
D3D11On12, and D3D9 / D3D10 / OpenGL / Vulkan via a **hook-owned D3D11 device**.
|
||||||
|
- New submodules are **not** auto-cloned — CMake checks each is populated and stops with a
|
||||||
|
`FATAL_ERROR` naming `git submodule update --init --recursive` (a reusable
|
||||||
|
`coop_require_submodule()` helper).
|
||||||
|
|
||||||
|
- **M1 — End-user UI fit pass.** Verify the shipping ImGui overlay (the *end-user* view, not
|
||||||
|
just the dev layout) with the debug-driving harness + F10 screenshot, and guarantee **every
|
||||||
|
panel fits its assigned window size even with Debug details enabled** — the
|
||||||
|
maximum-information case. Drive each panel to its fullest state via the harness (inject,
|
||||||
|
enable audio + video, expand Debug details, show the override controls, multiple audio
|
||||||
|
streams, the longest status / fallback strings), screenshot, and check nothing is clipped
|
||||||
|
or scrolled out of view. Where the densest case overflows, **resize and/or rearrange** the
|
||||||
|
panel so it fits (content can be moved between columns / rows — the most detailed case
|
||||||
|
should still fit). Land an automated harness check (drive-to-max → screenshot → assert no
|
||||||
|
overflow) so later milestones that add UI keep it green. *Independent of the backend work;
|
||||||
|
every milestone below must preserve this test* (M5's red banner and Vulkan-layer checkbox
|
||||||
|
in particular).
|
||||||
|
|
||||||
|
- **M2 — DX10 (mock → capture).**
|
||||||
|
1. **Mock backend** (`render_dx10.cpp`). `ID3D10Device` + DXGI swap chain; Windows SDK only
|
||||||
|
(`d3d10`, `dxgi`), **no new deps**. Sub-region fills (bar, counter block) via
|
||||||
|
`CopySubresourceRegion` of small solid-colour textures (DX10 has no clear-rect). Top-left
|
||||||
|
origin, so the counter block maps straight to the capture's sample point.
|
||||||
|
2. **Capture.** The DXGI `Present`/`Present1` hook already catches D3D10 swap chains, but the
|
||||||
|
copy QIs the backbuffer to `ID3D11Texture2D`, which a *pure* D3D10 device fails. Share the
|
||||||
|
D3D10 backbuffer into the hook's **own D3D11 device** (legacy shared handle) and copy it
|
||||||
|
into the standard keyed-mutex texture; add a `dx10_present_hook_test` that decodes the
|
||||||
|
mock's frames. Validates the "D3D10/11" claim the Architecture section currently makes
|
||||||
|
untested.
|
||||||
|
|
||||||
|
- **M3 — DX9 (mock → D3D9Ex capture → plain-D3D9 capture).**
|
||||||
|
1. **Mock backend** (`render_dx09.cpp`). `IDirect3DDevice9` / `IDirect3DDevice9Ex` + present;
|
||||||
|
Windows SDK only (`d3d9`), **no new deps**. `Clear` for the background and `ColorFill` for
|
||||||
|
the bar + block (D3D9's built-in rect fill — exactly the primitive DX10/11 lack). Top-left
|
||||||
|
origin. Runs in **two selectable modes** — `dx9ex` (default) and plain `dx9`
|
||||||
|
(`Direct3DCreate9`) — so both capture paths have a matching game.
|
||||||
|
2. **D3D9Ex capture (clean GPU path).** Hook `IDirect3DDevice9::Present` by vtable — discover
|
||||||
|
the slot from a throwaway `Direct3DCreate9Ex` device (`stdcall()` on x86 per the SafetyHook
|
||||||
|
trap); the vtable is shared across all devices of the class, so the game's existing device
|
||||||
|
is caught → **late-attach works**. On Present, `GetBackBuffer(0)` + `StretchRect` (GPU-side)
|
||||||
|
into a `CreateRenderTarget(..., pSharedHandle=&h)` surface (`D3DPOOL_DEFAULT`); share that
|
||||||
|
into the hook's own D3D11 device and copy into the standard keyed-mutex texture (host
|
||||||
|
unchanged, proper keyed-mutex sync). `A8R8G8B8` is **BGRA** → record the swizzle. Re-create
|
||||||
|
the surface on `Reset` / resize. `d3d9_present_hook_test` decodes the mock's frames (Ex mode).
|
||||||
|
3. **Plain (non-Ex) D3D9 capture (slow path).** Plain `Direct3DCreate9` devices **cannot**
|
||||||
|
produce a D3D11-shareable surface (WGC remains their default mirror until this lands).
|
||||||
|
Detect via a failed `QueryInterface(IID_IDirect3DDevice9Ex)` (else take the D3D9Ex path). On
|
||||||
|
Present, `GetRenderTargetData` the backbuffer into a
|
||||||
|
`CreateOffscreenPlainSurface(..., D3DPOOL_SYSTEMMEM)` surface, `LockRect`, copy out
|
||||||
|
(respect `Pitch`; BGRA→RGBA), and upload into the standard keyed-mutex texture via the
|
||||||
|
hook's own D3D11 device (`Map` / `UpdateSubresource`) — so only *how pixels reach the
|
||||||
|
texture* differs (CPU copy, not GPU). `GetRenderTargetData` is a GPU→sysmem stall, so
|
||||||
|
**drop / throttle** mirror frames rather than back-pressure the game. Same test, plain mode.
|
||||||
|
|
||||||
|
- **M4 — OpenGL (mock → capture coverage).**
|
||||||
|
1. **Mock backend** (`render_gl.cpp`). Raw WGL context (`wglCreateContextAttribsARB`) with the
|
||||||
|
**glad** loader (new submodule, `Dav1dde/glad`). Background via `glClearColor`/`glClear`;
|
||||||
|
bar + block via `glScissor` + clear (shader-free GL 1.x). GL's framebuffer is
|
||||||
|
**bottom-left** origin, so the counter block is placed flipped so the captured (top-left)
|
||||||
|
pixel still decodes — matching the existing `glReadPixels` capture flip. Adds the
|
||||||
|
`.gitmodules` entry + the `coop_require_submodule()` check.
|
||||||
|
2. **Capture coverage.** The GL `SwapBuffers`/`wglSwapBuffers` + `glReadPixels` path already
|
||||||
|
ships; add a mock-backed regression that decodes the mock's frames through it (upgrading the
|
||||||
|
synthetic `opengl_hook_test` to a real animated game). Small.
|
||||||
|
|
||||||
|
- **M5 — Vulkan (mock → capture).** The largest.
|
||||||
|
1. **Mock backend** (`render_vk.cpp`). New submodules **Vulkan-Headers**
|
||||||
|
(`KhronosGroup/Vulkan-Headers`, official) + **volk** (`zeux/volk`); raw
|
||||||
|
`vkCreateWin32SurfaceKHR`, swap chain, per-frame acquire → clear → present. Background via
|
||||||
|
`vkCmdClearColorImage`, bar + block via `vkCmdClearAttachments` clear-rects (no
|
||||||
|
pipeline/shaders → **no SPIR-V toolchain**). Loads Vulkan via **volk** on purpose — that's
|
||||||
|
the loader-bypass case the capture hook must handle. Validation layers used when a Vulkan
|
||||||
|
SDK is present, skipped otherwise.
|
||||||
|
2. **Capture.** Hook `vkQueuePresentKHR`; import a D3D11 keyed-mutex shared texture into Vulkan
|
||||||
|
(`VK_KHR_external_memory_win32` + `VK_KHR_win32_keyed_mutex`) and `vkCmdBlitImage` the
|
||||||
|
swap-chain image into it each present; the host samples it as usual. Because Vulkan caches
|
||||||
|
its present pointer at init, the hook **cannot be placed by late injection** — it must be
|
||||||
|
present before `vkCreateInstance`:
|
||||||
|
- **Attach via auto-attach — best-effort by default, opt-in layer for reliability.** By
|
||||||
|
default, arming Auto-attach for a detected Vulkan game injects as early as possible on
|
||||||
|
relaunch (the existing poll-and-inject) — enough for games that initialize Vulkan a little
|
||||||
|
into startup. For games that init Vulkan immediately, an **opt-in** Injection-panel
|
||||||
|
checkbox ("Set up Vulkan layer") registers a per-user (HKCU, no admin) implicit Vulkan
|
||||||
|
layer (thin `coop_vk_layer.dll` + JSON manifest) scoped to that game's image name; on
|
||||||
|
relaunch the loader loads it at `vkCreateInstance` — guaranteed before init — and it wires
|
||||||
|
up present capture + IPC. The layer is removed when the option is unticked or the host
|
||||||
|
exits, and self-deactivates for any non-target app, so a stale registration (e.g. after a
|
||||||
|
host crash) is harmless.
|
||||||
|
- **Too-late detection + red prompt.** A late-injected hook that finds `vulkan-1.dll` loaded
|
||||||
|
but no working hooked-present reports a *"Vulkan, injected too late"* status; the host
|
||||||
|
overlays a **red** banner on the mirror window: "Detected a Vulkan game — the hook attached
|
||||||
|
too late to mirror it with low latency. Enable Auto-attach (and, if it persists, 'Set up
|
||||||
|
Vulkan layer') and relaunch the game." **WGC keeps mirroring meanwhile** so the session
|
||||||
|
stays usable; the banner clears once the hooked path goes live. (New end-user UI → keep
|
||||||
|
M1's fit test green.)
|
||||||
|
- Sub-steps (separate commits): present-hook placement (best-effort inject + the implicit
|
||||||
|
layer), external-memory image import + keyed-mutex sync, the too-late status code, the
|
||||||
|
host red-banner UI, and the Auto-attach / Vulkan-layer Injection-panel controls. Test the
|
||||||
|
early (layer) path against the mock (decode frames) and the too-late path (assert the
|
||||||
|
status + banner fire).
|
||||||
|
|
||||||
### Future work
|
### Future work
|
||||||
|
|
||||||
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
|
|
||||||
them needs a Vulkan layer / device-dispatch hook plus a `vkCmdCopyImage` to a
|
|
||||||
readable image. Use WGC in the meantime.
|
|
||||||
- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present`
|
|
||||||
hook would be the lower-latency upgrade.
|
|
||||||
- **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB
|
- **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB
|
||||||
subsystem forwards via window messages (`PostMessage`) plus synthesized
|
subsystem forwards via window messages (`PostMessage`) plus synthesized
|
||||||
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop
|
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop
|
||||||
|
|||||||
Reference in New Issue
Block a user