M2(Vulkan): implicit capture layer (coop_vk_layer) + chain dispatch + env test

A real chain-aware Vulkan implicit layer the loader inserts at vkCreateInstance
-- the reliable early-presence path for games that init Vulkan immediately,
which the inline-hook vk_hook can't catch. It intercepts vkCreateInstance /
Device / CreateSwapchainKHR / QueuePresentKHR via proper layer-chain dispatch
and does the same read-back capture (vkCmdCopyImageToBuffer -> swizzle ->
hook-owned D3D11 shared texture, with present-semaphore re-chaining) as vk_hook.

The loader/layer link structs (VkLayer*CreateInfo, VkNegotiateLayerInterface)
aren't in Vulkan-Headers, so they're hand-declared to interface version 2. Key
gotcha found via tracing: the loader tags those link structs with small internal
sType values (LOADER_INSTANCE_CREATE_INFO=47, _DEVICE=48), not the 1000000000
range -- matching the wrong value made the device-chain walk fail.

Scoping: an implicit layer loads into every Vulkan app, so it only *captures*
when COOP_VK_LAYER_FORCE is set (tests) or this process's image matches
%TEMP%\coop_vk_target.txt (the host writes it); otherwise pure pass-through.
mock_game_test registers it via VK_LAYER_PATH/VK_INSTANCE_LAYERS and decodes
frames through it. Ships at the bin root with its JSON manifest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-22 13:24:37 +02:00
parent 945d7fbf78
commit 2532ffed56
5 changed files with 924 additions and 0 deletions

19
vk_layer/CMakeLists.txt Normal file
View File

@@ -0,0 +1,19 @@
# CoopAllTheThings Vulkan capture layer: a real implicit Vulkan layer the loader inserts at
# vkCreateInstance, so it's in the chain before a game (that caches its present pointer at init)
# can resolve vkQueuePresentKHR -- the reliable early-presence path the inline-hook vk_hook can't
# get for immediate-init games. Ships at the bin root (registered for real games), with its JSON
# manifest copied alongside (the manifest's library_path is relative).
coop_require_submodule("Vulkan-Headers" "third_party/Vulkan-Headers/include/vulkan/vulkan.h")
add_library(coop_vk_layer SHARED coop_vk_layer.cpp)
target_include_directories(coop_vk_layer PRIVATE
${CMAKE_SOURCE_DIR}/hook/src # ipc_client.hpp (reused IPC client)
${CMAKE_SOURCE_DIR}/third_party/Vulkan-Headers/include)
target_link_libraries(coop_vk_layer PRIVATE coop_common d3d11 dxgi)
# Drop the manifest next to the built DLL so VK_LAYER_PATH / the registry can find it.
add_custom_command(TARGET coop_vk_layer POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different
${CMAKE_CURRENT_SOURCE_DIR}/coop_vk_layer.json $<TARGET_FILE_DIR:coop_vk_layer>/coop_vk_layer.json)