From fc43355db2ae12fcd5aec896de40b24cdd3b94bd Mon Sep 17 00:00:00 2001 From: BlackMark Date: Mon, 22 Jun 2026 10:47:53 +0200 Subject: [PATCH] Roadmap: sequence backend work per-API (mock + capture together) + UI-fit pass Restructure the roadmap into one "Current work" section where each rendering API is a single milestone that builds its mock-game backend and the injected capture for that same API side by side, so each API reaches verified end-to-end before the next. Adds an M1 end-user UI fit pass (drive the overlay to maximum info via the harness, screenshot, assert every panel fits its window; resize/rearrange otherwise) kept green by every later milestone. Folds in DX10, DX9 (D3D9Ex + plain non-Ex), OpenGL, and Vulkan (best-effort inject + opt-in implicit layer, too-late red prompt). Drops the old Vulkan/ D3D9 future-work bullets. Co-Authored-By: Claude Opus 4.8 --- README.md | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 122 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 8840551..ac95022 100644 --- a/README.md +++ b/README.md @@ -95,13 +95,130 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap). ## Roadmap +### Current work — per-API capture, end to end + +Each rendering API is built as **one milestone**: first its `coop_mock_game` backend (an +animated, frame-numbered A/V source), then the injected **capture** for that same API +directly after — so every API reaches *verified end-to-end* (the mirror decodes the frame +counter and asserts a *monotonic, advancing* sequence) before the next one starts, rather +than building all the mock backends first and all the capture later. Order is +easiest-to-hardest, dependencies last. + +Conventions for every milestone below: + +- Each mock backend implements the existing + [`RenderBackend`](tools/mock_game/render_backend.hpp) interface (animated background + + moving bar + frame-counter block), is selectable on the command line, and renders + **clear/fill-only** — no geometry, no shaders, so **no shader-compiler dependency**. +- Sub-steps are **separate commits**; the README, CMake, and (where needed) `.gitmodules` + move with each. A mock-backend commit lands with a *liveness* smoke check (launches, + presents N frames, exits clean); the frame-accurate decode-through-the-hook assertion + lands with that API's capture commit right after. +- The host samples the **standard shared keyed-mutex texture** unchanged regardless of + source API: every backend publishes into it — native D3D11 on the game's device, D3D12 via + D3D11On12, and D3D9 / D3D10 / OpenGL / Vulkan via a **hook-owned D3D11 device**. +- New submodules are **not** auto-cloned — CMake checks each is populated and stops with a + `FATAL_ERROR` naming `git submodule update --init --recursive` (a reusable + `coop_require_submodule()` helper). + +- **M1 — End-user UI fit pass.** Verify the shipping ImGui overlay (the *end-user* view, not + just the dev layout) with the debug-driving harness + F10 screenshot, and guarantee **every + panel fits its assigned window size even with Debug details enabled** — the + maximum-information case. Drive each panel to its fullest state via the harness (inject, + enable audio + video, expand Debug details, show the override controls, multiple audio + streams, the longest status / fallback strings), screenshot, and check nothing is clipped + or scrolled out of view. Where the densest case overflows, **resize and/or rearrange** the + panel so it fits (content can be moved between columns / rows — the most detailed case + should still fit). Land an automated harness check (drive-to-max → screenshot → assert no + overflow) so later milestones that add UI keep it green. *Independent of the backend work; + every milestone below must preserve this test* (M5's red banner and Vulkan-layer checkbox + in particular). + +- **M2 — DX10 (mock → capture).** + 1. **Mock backend** (`render_dx10.cpp`). `ID3D10Device` + DXGI swap chain; Windows SDK only + (`d3d10`, `dxgi`), **no new deps**. Sub-region fills (bar, counter block) via + `CopySubresourceRegion` of small solid-colour textures (DX10 has no clear-rect). Top-left + origin, so the counter block maps straight to the capture's sample point. + 2. **Capture.** The DXGI `Present`/`Present1` hook already catches D3D10 swap chains, but the + copy QIs the backbuffer to `ID3D11Texture2D`, which a *pure* D3D10 device fails. Share the + D3D10 backbuffer into the hook's **own D3D11 device** (legacy shared handle) and copy it + into the standard keyed-mutex texture; add a `dx10_present_hook_test` that decodes the + mock's frames. Validates the "D3D10/11" claim the Architecture section currently makes + untested. + +- **M3 — DX9 (mock → D3D9Ex capture → plain-D3D9 capture).** + 1. **Mock backend** (`render_dx09.cpp`). `IDirect3DDevice9` / `IDirect3DDevice9Ex` + present; + Windows SDK only (`d3d9`), **no new deps**. `Clear` for the background and `ColorFill` for + the bar + block (D3D9's built-in rect fill — exactly the primitive DX10/11 lack). Top-left + origin. Runs in **two selectable modes** — `dx9ex` (default) and plain `dx9` + (`Direct3DCreate9`) — so both capture paths have a matching game. + 2. **D3D9Ex capture (clean GPU path).** Hook `IDirect3DDevice9::Present` by vtable — discover + the slot from a throwaway `Direct3DCreate9Ex` device (`stdcall()` on x86 per the SafetyHook + trap); the vtable is shared across all devices of the class, so the game's existing device + is caught → **late-attach works**. On Present, `GetBackBuffer(0)` + `StretchRect` (GPU-side) + into a `CreateRenderTarget(..., pSharedHandle=&h)` surface (`D3DPOOL_DEFAULT`); share that + into the hook's own D3D11 device and copy into the standard keyed-mutex texture (host + unchanged, proper keyed-mutex sync). `A8R8G8B8` is **BGRA** → record the swizzle. Re-create + the surface on `Reset` / resize. `d3d9_present_hook_test` decodes the mock's frames (Ex mode). + 3. **Plain (non-Ex) D3D9 capture (slow path).** Plain `Direct3DCreate9` devices **cannot** + produce a D3D11-shareable surface (WGC remains their default mirror until this lands). + Detect via a failed `QueryInterface(IID_IDirect3DDevice9Ex)` (else take the D3D9Ex path). On + Present, `GetRenderTargetData` the backbuffer into a + `CreateOffscreenPlainSurface(..., D3DPOOL_SYSTEMMEM)` surface, `LockRect`, copy out + (respect `Pitch`; BGRA→RGBA), and upload into the standard keyed-mutex texture via the + hook's own D3D11 device (`Map` / `UpdateSubresource`) — so only *how pixels reach the + texture* differs (CPU copy, not GPU). `GetRenderTargetData` is a GPU→sysmem stall, so + **drop / throttle** mirror frames rather than back-pressure the game. Same test, plain mode. + +- **M4 — OpenGL (mock → capture coverage).** + 1. **Mock backend** (`render_gl.cpp`). Raw WGL context (`wglCreateContextAttribsARB`) with the + **glad** loader (new submodule, `Dav1dde/glad`). Background via `glClearColor`/`glClear`; + bar + block via `glScissor` + clear (shader-free GL 1.x). GL's framebuffer is + **bottom-left** origin, so the counter block is placed flipped so the captured (top-left) + pixel still decodes — matching the existing `glReadPixels` capture flip. Adds the + `.gitmodules` entry + the `coop_require_submodule()` check. + 2. **Capture coverage.** The GL `SwapBuffers`/`wglSwapBuffers` + `glReadPixels` path already + ships; add a mock-backed regression that decodes the mock's frames through it (upgrading the + synthetic `opengl_hook_test` to a real animated game). Small. + +- **M5 — Vulkan (mock → capture).** The largest. + 1. **Mock backend** (`render_vk.cpp`). New submodules **Vulkan-Headers** + (`KhronosGroup/Vulkan-Headers`, official) + **volk** (`zeux/volk`); raw + `vkCreateWin32SurfaceKHR`, swap chain, per-frame acquire → clear → present. Background via + `vkCmdClearColorImage`, bar + block via `vkCmdClearAttachments` clear-rects (no + pipeline/shaders → **no SPIR-V toolchain**). Loads Vulkan via **volk** on purpose — that's + the loader-bypass case the capture hook must handle. Validation layers used when a Vulkan + SDK is present, skipped otherwise. + 2. **Capture.** Hook `vkQueuePresentKHR`; import a D3D11 keyed-mutex shared texture into Vulkan + (`VK_KHR_external_memory_win32` + `VK_KHR_win32_keyed_mutex`) and `vkCmdBlitImage` the + swap-chain image into it each present; the host samples it as usual. Because Vulkan caches + its present pointer at init, the hook **cannot be placed by late injection** — it must be + present before `vkCreateInstance`: + - **Attach via auto-attach — best-effort by default, opt-in layer for reliability.** By + default, arming Auto-attach for a detected Vulkan game injects as early as possible on + relaunch (the existing poll-and-inject) — enough for games that initialize Vulkan a little + into startup. For games that init Vulkan immediately, an **opt-in** Injection-panel + checkbox ("Set up Vulkan layer") registers a per-user (HKCU, no admin) implicit Vulkan + layer (thin `coop_vk_layer.dll` + JSON manifest) scoped to that game's image name; on + relaunch the loader loads it at `vkCreateInstance` — guaranteed before init — and it wires + up present capture + IPC. The layer is removed when the option is unticked or the host + exits, and self-deactivates for any non-target app, so a stale registration (e.g. after a + host crash) is harmless. + - **Too-late detection + red prompt.** A late-injected hook that finds `vulkan-1.dll` loaded + but no working hooked-present reports a *"Vulkan, injected too late"* status; the host + overlays a **red** banner on the mirror window: "Detected a Vulkan game — the hook attached + too late to mirror it with low latency. Enable Auto-attach (and, if it persists, 'Set up + Vulkan layer') and relaunch the game." **WGC keeps mirroring meanwhile** so the session + stays usable; the banner clears once the hooked path goes live. (New end-user UI → keep + M1's fit test green.) + - Sub-steps (separate commits): present-hook placement (best-effort inject + the implicit + layer), external-memory image import + keyed-mutex sync, the too-late status code, the + host red-banner UI, and the Auto-attach / Vulkan-layer Injection-panel controls. Test the + early (layer) path against the mock (decode frames) and the too-late path (assert the + status + banner fire). + ### Future work -- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking - them needs a Vulkan layer / device-dispatch hook plus a `vkCmdCopyImage` to a - readable image. Use WGC in the meantime. -- **D3D9 hooked path.** Covered by WGC today; a dedicated `IDirect3DDevice9::Present` - hook would be the lower-latency upgrade. - **Mouse + keyboard forwarding for Raw Input / DirectInput games.** The MKB subsystem forwards via window messages (`PostMessage`) plus synthesized `GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos`, which covers message-loop