Add Tooling, UI & input backlog to the roadmap
Append ten requested future-work items with how-to detail: bin restructure (deployable set at root, tests/tools in subdirs), terminated-target detection, re-attach by image name, window-based target selection, auto-layout of the overlay, the Audio "live" column fix, moving the synthetic-input toggle under Controllers/Debug, mouse+keyboard forwarding (messages + polling-state hooks), rumble forwarding on both backends, and per-backend input debug visualization. Promote the terse rumble bullet from Future work into the fleshed-out task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
118
README.md
118
README.md
@@ -109,6 +109,122 @@ milestone with its own tests and commit.
|
||||
host-side mixer that resamples each ring to the render format and sums them
|
||||
(with soft-clip).
|
||||
|
||||
### Tooling, UI & input
|
||||
|
||||
Operability, UI, and input-debugging improvements, mostly independent of the
|
||||
game-specific milestones above. Ordered roughly as requested, not by priority.
|
||||
|
||||
1. **Stage only deployable artifacts directly under `bin/`.** Today every target —
|
||||
host, both hook DLLs, the injector helper, tests, probes, `coop_tone` — lands in
|
||||
`bin/<config>/`, so deploying to a donor folder means hand-picking files. Keep
|
||||
the **deployable set** at the `bin/<config>/` root (`coop_host.exe`,
|
||||
`coop_hook.dll`, `coop_hook_x86.dll`, `coop_inject_x86.exe`, and — when Steam is
|
||||
built — `steam_api64.dll` + `steam_input_actions.vdf`) and push everything else
|
||||
into subfolders: tests → `bin/<config>/tests/`, debug/probe tools
|
||||
(`coop_audio_probe`, `coop_input_probe`, `coop_tone`, `coop_steam_input_probe`) →
|
||||
`bin/<config>/tools/`. *How:* give those targets a per-target
|
||||
`RUNTIME_OUTPUT_DIRECTORY[_<CONFIG>]` (the global `CMAKE_RUNTIME_OUTPUT_DIRECTORY`
|
||||
stays the deployable root; a small helper or `set_target_properties` overrides the
|
||||
non-deployable ones). The x86 sub-build must still stage `coop_hook_x86.dll` +
|
||||
`coop_inject_x86.exe` into the deployable root while its x86 *test* exes go to
|
||||
`tests/`. Watch the cross-target paths: `audio_loopback_test` spawns
|
||||
`coop_tone.exe`, and the host's post-build copy of `steam_api64.dll` / the `.vdf`
|
||||
must follow the host. End result: copying `bin/<config>/` non-recursively yields a
|
||||
clean deployable bundle.
|
||||
|
||||
2. **Detect a terminated target and reflect it in the UI.** The Injection panel keeps
|
||||
showing "Attached" after the game exits. Add a **Terminated** state: the host
|
||||
already knows the target pid and tracks a DLL heartbeat (`InjectionPanel`); on top
|
||||
of that, hold the `OpenProcess` handle from injection (or re-open with
|
||||
`PROCESS_QUERY_LIMITED_INFORMATION`) and poll `GetExitCodeProcess` /
|
||||
`WaitForSingleObject(h, 0)` each tick. When the process is gone, switch to
|
||||
Terminated, gray out / disable the per-subsystem controls and mirror toggles, and
|
||||
show a clear banner; the Video and Audio panels should drop to idle (their hook
|
||||
channels are stale) rather than freezing on the last live frame/state.
|
||||
|
||||
3. **Re-attach to a relaunched target.** A killed-and-relaunched game gets a new pid,
|
||||
but the UI still holds the stale one. In the Terminated state (task 2), remember
|
||||
the target's image name (the panel already keeps the selected exe name) and offer a
|
||||
**Re-attach** button that injects only if a live process with that *same name*
|
||||
exists, rebinding the IPC server to the new pid via the existing `inject_dll` path.
|
||||
*Open decision (will confirm at build time):* if several processes share the name,
|
||||
default to the most-recently-started one with a note, or fall back to the picker.
|
||||
|
||||
4. **Select targets by window, not just process.** A flat process list is fine as an
|
||||
advanced/debug view, but the default should be a **window list** — there are far
|
||||
fewer top-level windows than processes, and a window directly yields the HWND the
|
||||
WGC capturer and focus spoof already want. *How:* add a window enumerator
|
||||
(`EnumWindows`, keeping visible, titled, non-tool top-level windows —
|
||||
`IsWindowVisible`, `GetWindowTextLength > 0`, exclude `WS_EX_TOOLWINDOW` and our own
|
||||
HWND, resolve to the root owner) and map each via `GetWindowThreadProcessId` → pid →
|
||||
image name. Show **title + process name + pid** with a filter box like the process
|
||||
list; injecting by window injects into its pid and hands the HWND straight to
|
||||
capture. Keep the process list behind "Debug details" as the advanced path.
|
||||
|
||||
5. **Auto-size and lay out the overlay windows so none need manual resizing.** Panels
|
||||
currently `Begin` at default cascade positions, so they overlap and clip. Give each
|
||||
`ImGuiWindowFlags_AlwaysAutoResize` and an initial position computed from
|
||||
`ImGui::GetMainViewport()->WorkPos/WorkSize`, applied with `ImGuiCond_FirstUseEver`
|
||||
(still movable), plus a **View → Reset layout** menu item that re-applies it. Target
|
||||
layout: **Injection** left/top (room to grow downward for hook diagnostics);
|
||||
**Controllers** top-center; **Video mirror** center, below Controllers; **Audio
|
||||
mirror** below Video; **Log** right edge, full height (most room for the log
|
||||
stream). Auto-resize fits these because they're all control/debug panels — the live
|
||||
mirror image is drawn to the whole host window *behind* the overlay, not inside a
|
||||
panel.
|
||||
|
||||
6. **Fix the Audio panel "live" column.** It overlays a green dot and grey "idle"
|
||||
because liveness is recomputed each frame from the per-stream `frames_rendered`
|
||||
delta, which is zero on most frames (buffers release in bursts), so it flickers.
|
||||
Replace it with a **debounced activity indicator**: keep a per-stream "last
|
||||
advanced" timestamp (the panel already stores the previous frame counts) and show
|
||||
**live** if frames advanced within the last ~300–500 ms, else **idle** — optionally
|
||||
a small frames/s or activity bar so multi-stream games read clearly.
|
||||
|
||||
7. **Move the synthetic-input toggle to the Controllers panel, under Debug details.**
|
||||
The "Forward synthetic test input" checkbox is a controller-debugging aid, so move
|
||||
it out of the Injection panel into `ControllersPanel` and gate it behind
|
||||
`debug_details`. The publish path is unchanged (the Injection panel already
|
||||
substitutes the synthetic pattern in `publish`); the flag just moves with it (or is
|
||||
passed from Controllers into the publish call).
|
||||
|
||||
8. **Mouse & keyboard forwarding (messages + polling-state hooks).** Forward guest
|
||||
clicks and keystrokes into the unfocused game via a new **MKB hook subsystem** in
|
||||
`coop_hook.dll` — the toggle *is* the hook (not installed → no forwarding).
|
||||
*Delivery:* `PostMessage` window-message input (`WM_KEYDOWN`/`WM_KEYUP`/`WM_CHAR`,
|
||||
`WM_*BUTTONDOWN`/`UP`, `WM_MOUSEWHEEL`) to the game HWND, **plus** hook
|
||||
`GetAsyncKeyState` / `GetKeyboardState` / `GetCursorPos` in the DLL so polling games
|
||||
see the synthesized keyboard/cursor state (RawInput and DirectInput games are out of
|
||||
scope for this version). *Keyboard* is always forwarded; *mouse* only while video is
|
||||
mirrored (otherwise the operator can't see where they click), and only **clicks +
|
||||
wheel, not movement** (one cursor can't be in two places). *Coordinate mapping* (the
|
||||
part that must be exact): WGC + decorated windowed → translate by the window
|
||||
decoration / client-area offset; hooked capture → relative to the mirrored viewport
|
||||
only (decorations aren't mirrored); borderless → the same under both backends.
|
||||
*Critical gating:* forward only when the host's main window is focused **and** ImGui
|
||||
doesn't want the event (`ImGuiIO::WantCaptureMouse` / `WantCaptureKeyboard`), so
|
||||
interacting with the overlay's own windows never leaks input into the game. The host
|
||||
sends MKB events to the hook over a new (or extended) IPC region.
|
||||
|
||||
9. **Rumble / haptics forwarding (both backends).** Currently unsupported — the XInput
|
||||
hook swallows `XInputSetState`. Add a reverse path: the hook captures the game's
|
||||
`XInputSetState` (left/right motor) and publishes it over a hook→host channel (the
|
||||
back-channel already exists), and the host drives the guest's actuators per backend —
|
||||
**XInput:** call `XInputSetState` on the guest's slot (the viability unknown is
|
||||
whether Steam's RPT virtual pad accepts vibration and routes it to the guest);
|
||||
**Steam Input:** `SteamInput()->TriggerVibration` / `Legacy_TriggerHapticPulse`.
|
||||
Map each guest slot to the right actuator.
|
||||
|
||||
10. **Per-backend input debug visualization.** To separate "wrong input *into* the
|
||||
tool" from "wrong input *out to* the game", show three distinct views in the
|
||||
Controllers panel (under Debug details): (a) **received via XInput** (raw
|
||||
`XInputSource` state), (b) **received via Steam Input** (raw `SteamInputSource`
|
||||
action values) — so it's obvious which backend delivered what — and (c)
|
||||
**forwarded to the game** (the `PadInfo` we write to shared memory, alongside what
|
||||
the game actually read back via the hook's per-slot channel). The hook already
|
||||
reports per-slot poll counts; extend it to echo the last state the game read so (c)
|
||||
is a true round-trip.
|
||||
|
||||
### Future work
|
||||
|
||||
- **Vulkan video hook.** Vulkan games present via `vkQueuePresentKHR`; hooking
|
||||
@@ -120,8 +236,6 @@ milestone with its own tests and commit.
|
||||
we inject is never seen at `Initialize`, so the hook assumes the device **mix
|
||||
format**. A stream initialized in shared mode at a different format would come
|
||||
out wrong-pitched. Detecting the real per-stream format would remove that guess.
|
||||
- **Rumble / haptics forwarding.** `XInputSetState` is currently swallowed; routing
|
||||
it back to the guest is a later phase.
|
||||
|
||||
## Building
|
||||
|
||||
|
||||
Reference in New Issue
Block a user