From 784c31a9b5c63945429d34302f52ecea28442262 Mon Sep 17 00:00:00 2001 From: BlackMark Date: Sun, 21 Jun 2026 05:45:47 +0200 Subject: [PATCH] Capture every audio stream into its own ring and mix them on the host Games with several concurrent WASAPI render streams (e.g. Spider-Man: Miles Morales) only had their first ("primary") stream mirrored; the rest kept playing locally and never reached the guest. Now the render-hook captures + silences EVERY tracked stream into its own ring (coop_audio_[_]), each published with that stream's own detected format (Initialize when caught, else GetMixFormat -- the per-stream format detection, now actually used per ring rather than only for the primary). The host creates a ring per stream and mixes the same-format streams with a soft clip (host/src/audio/audio_mix.hpp); streams whose format differs from the primary are still silenced (no echo) but skipped from the mix (would need resampling). The single-stream case is byte-for-byte unchanged: when only one stream is active the host passes it through without the mixer, so the common path has no overhead or fidelity change. Verified: new audio_mix_test covers the decode/sum/soft-clip/encode math (float32 + int16); audio_hook_test (x64 + x86) still passes, guarding the primary capture+silence path against regression; full build x64 + x86 clean; ctest x64 11/11, x86 3/3. Multi-stream mixing against a real multi-stream game needs a live session to fully confirm. Co-Authored-By: Claude Opus 4.8 --- README.md | 27 ++--- common/include/coop/audio_ring.hpp | 14 ++- hook/src/audio_hook.cpp | 164 +++++++++++++++-------------- hook/src/audio_hook.hpp | 14 +-- hook/src/dllmain.cpp | 47 +++++---- host/src/audio/audio_loopback.cpp | 113 ++++++++++++++++---- host/src/audio/audio_loopback.hpp | 8 +- host/src/audio/audio_mix.hpp | 77 ++++++++++++++ tests/CMakeLists.txt | 6 ++ tests/audio_mix_test.cpp | 89 ++++++++++++++++ 10 files changed, 412 insertions(+), 147 deletions(-) create mode 100644 host/src/audio/audio_mix.hpp create mode 100644 tests/audio_mix_test.cpp diff --git a/README.md b/README.md index e3aba6b..2cdd3b5 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ and forwards guest controllers back into it. | Keep game running unfocused | Hook spoofs focus so the game polls while the host holds OS focus | `coop_hook.dll` | | Mirror video (default) | Windows Graphics Capture of the game window, letterboxed into the host window | `coop_host.exe` | | Mirror video (hooked) | Injected Present / OpenGL hook copies the backbuffer into a shared keyed-mutex texture the host samples (lower latency, no capture border) | `coop_hook.dll` + `coop_host.exe` | -| Mirror audio | Injected render-hook copies the game's WASAPI frames into a shared ring and silences the game locally (no echo); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` | +| Mirror audio | Injected render-hook copies each of the game's WASAPI render streams into its own shared ring and silences the game locally (no echo); the host mixes the streams (soft-clipped); WASAPI process loopback is the automatic fallback | `coop_hook.dll` + `coop_host.exe` | | Host ↔ hook IPC | Named shared memory (seqlock for input, status back-channel, video/audio/log shares) | `common/` | The hooked video path has two producers: **Direct3D (DXGI)** hooks @@ -72,23 +72,12 @@ and covers anything the hooked path doesn't (Vulkan, D3D9 — see Roadmap). ### Planned (next up) -Worked top-to-bottom: each item is a milestone with its own tests and commit, and -is removed from this list once done — so the top item is always next. The -self-verifiable tooling / UI / input items come first; the game-pipeline items that -need a real game (and Remote Play) to fully validate come last. - -- **Multi-stream audio capture + mixing, with per-stream format detection.** Games - with several concurrent WASAPI render streams (e.g. Miles Morales) only get their - first ("primary") stream mirrored today; the rest keep playing locally and never - reach the guest. Capture every tracked render stream into its own shared ring, - silence each, and add a host-side mixer that resamples each ring to the render - format and sums them (with soft-clip). **Fold in real per-stream format - detection** here, since it touches the same hook + ring plumbing: a stream that - already existed when we injected is never seen at `Initialize`, so the hook - currently assumes the device **mix format** and a shared-mode stream opened at a - different format comes out wrong-pitched. Resolve each stream's true format (the - stream's own `Initialize` when caught, else the original `GetMixFormat`) so every - mixed ring is pitched correctly. +Nothing queued — the previous backlog (bin restructure, terminated/hung detection, +re-attach, window-based target picker, overlay auto-layout, Audio "live" column, +moving the synthetic-input toggle, mouse + keyboard forwarding, rumble forwarding, +per-backend input debug view, cursor release, capture metrics + latency, DX12 hooked +capture, multi-stream audio + per-stream formats) is all shipped. See Future work +for what's left. ### Future work @@ -152,6 +141,8 @@ ctest --test-dir build -C Debug --output-on-failure no controller needed). - **`audio_ring_test`** — unit test of the shared audio ring (lock-free SPSC push/pop, wrap-around, format handshake, overrun/drop). No device needed. +- **`audio_mix_test`** — unit test of the multi-stream mixer math (decode / sum / + soft-clip / encode for float32 + int16). No device needed. - **`audio_hook_test`** — in-process self-test of the WASAPI render-hook: installs the hooks, renders a tone through WASAPI in the same process, and asserts the COM vtables were discovered, the frames reached the ring (non-silent), the diff --git a/common/include/coop/audio_ring.hpp b/common/include/coop/audio_ring.hpp index bce7abd..bbc8a60 100644 --- a/common/include/coop/audio_ring.hpp +++ b/common/include/coop/audio_ring.hpp @@ -185,10 +185,18 @@ inline std::uint32_t audio_ring_pop(AudioRingHeader& h, void* dst, std::uint32_t return bytes; } -// Build the per-pid audio ring name both sides agree on. -inline std::wstring audio_ring_name(unsigned long target_pid) +// Build the per-pid audio ring name both sides agree on. Stream 0 keeps the bare +// coop_audio_ name (backward compatible / the single-stream case); additional +// streams append _ (coop_audio__1, _2, ...). The host captures every +// render stream into its own ring and mixes them. +inline std::wstring audio_ring_name(unsigned long target_pid, unsigned index = 0) { - return std::wstring(kAudioRingPrefix) + std::to_wstring(target_pid); + std::wstring name = std::wstring(kAudioRingPrefix) + std::to_wstring(target_pid); + if (index != 0) + { + name += L"_" + std::to_wstring(index); + } + return name; } } // namespace coop diff --git a/hook/src/audio_hook.cpp b/hook/src/audio_hook.cpp index 640a818..286d530 100644 --- a/hook/src/audio_hook.cpp +++ b/hook/src/audio_hook.cpp @@ -113,7 +113,9 @@ struct CapturedFormat // --- Global hook state ----------------------------------------------------- IpcClient* g_ipc = nullptr; -std::atomic g_ring{nullptr}; +// One ring per tracked stream (index = the stream's debug slot). Stream 0 is the +// primary; the host creates a ring per stream and mixes them. +std::atomic g_rings[kMaxAudioStreams]{}; std::mutex g_setup_mutex; // guards installs + the format map + stream registration @@ -146,32 +148,28 @@ std::atomic g_self_render{nullptr}; CapturedFormat g_mix_format; std::atomic g_have_mix_format{0}; -// The primary stream's actual format, captured when it's registered. The host -// creates the ring only when audio mirroring is toggled on — typically *after* -// the primary stream was already registered — so the format must be (re)published -// to the ring whenever it attaches. (The in-process probe creates the ring before -// injecting, so it never exercises this ordering; the full app always does.) -// Guarded by g_setup_mutex. -CapturedFormat g_primary_format; +// Each tracked stream's actual format, captured when it's registered. The host +// creates the rings only when audio mirroring is toggled on — typically *after* a +// stream was already registered — so the format must be (re)published to a ring +// whenever it attaches. Guarded by g_setup_mutex. +CapturedFormat g_stream_formats[kMaxAudioStreams]; // Per IAudioClient, the format captured at Initialize, looked up when its render // client is created. Setup-path only (never touched on the audio thread). std::unordered_map g_client_formats; -// Streams we track for the debug view (frame counting). Index 0 is primary. +// Streams we track (frame counting + per-stream capture). Index 0 is primary. struct TrackedStream { std::atomic client{nullptr}; std::atomic frames{0}; + std::atomic block_align{0}; // hot-path frame size for this stream }; TrackedStream g_streams[kMaxAudioStreams]; std::uint32_t g_registered = 0; // slots filled (<= kMaxAudioStreams), under mutex std::atomic g_streams_seen{0}; // total distinct clients ever seen -// Hot-path primary identity + frame size (avoids touching the map/mutex). -std::atomic g_primary{nullptr}; -std::atomic g_primary_block_align{0}; -std::atomic g_frames_captured{0}; +std::atomic g_frames_captured{0}; // total frames captured across streams // GetBuffer/ReleaseBuffer are paired on one thread, never nested: stash the // pointer the game just got so ReleaseBuffer can copy it before releasing. @@ -235,42 +233,43 @@ HRESULT STDMETHODCALLTYPE hk_ReleaseBuffer(IAudioRenderClient* self, UINT32 num_ try_register_lazy(self); } - // Frame counting for any tracked stream (drives the live/idle debug view). + // Per tracked stream: count frames (debug view) and, into the stream's own ring, + // capture + silence its buffer while capture is enabled. Every stream is captured + // into its own ring; the host mixes them. for (std::uint32_t i = 0; i < kMaxAudioStreams; ++i) { - if (g_streams[i].client.load(std::memory_order_acquire) == self) + if (g_streams[i].client.load(std::memory_order_acquire) != self) { - const std::uint64_t total = - g_streams[i].frames.fetch_add(num_frames, std::memory_order_relaxed) + num_frames; - if (g_ipc != nullptr) - { - g_ipc->note_audio_frames(i, total); - } - break; + continue; + } + const std::uint64_t total = + g_streams[i].frames.fetch_add(num_frames, std::memory_order_relaxed) + num_frames; + if (g_ipc != nullptr) + { + g_ipc->note_audio_frames(i, total); } - } - // Capture + silence only the primary stream, only while enabled. - if (self == g_primary.load(std::memory_order_acquire) && num_frames > 0 && - (flags & AUDCLNT_BUFFERFLAGS_SILENT) == 0) - { - AudioRingHeader* ring = g_ring.load(std::memory_order_acquire); - if (ring != nullptr && ring->capture_enabled.load(std::memory_order_relaxed) != 0 && - t_gb_client == self && t_gb_data != nullptr && t_gb_frames == num_frames) + if (num_frames > 0 && (flags & AUDCLNT_BUFFERFLAGS_SILENT) == 0) { - const std::uint32_t block = g_primary_block_align.load(std::memory_order_relaxed); - const std::uint32_t bytes = num_frames * block; - // Only silence if the frames made it into the ring; if the host has - // stalled (ring full) keep playing locally rather than going dead - // silent — degrades to today's echo, never to silence. - if (audio_ring_push(*ring, t_gb_data, bytes, num_frames)) + AudioRingHeader* ring = g_rings[i].load(std::memory_order_acquire); + if (ring != nullptr && ring->capture_enabled.load(std::memory_order_relaxed) != 0 && + t_gb_client == self && t_gb_data != nullptr && t_gb_frames == num_frames) { - std::memset(t_gb_data, 0, bytes); // belt-and-suspenders vs a driver ignoring SILENT - g_frames_captured.fetch_add(num_frames, std::memory_order_relaxed); - return g_vh_releasebuffer.original()( - self, num_frames, flags | AUDCLNT_BUFFERFLAGS_SILENT); + const std::uint32_t block = g_streams[i].block_align.load(std::memory_order_relaxed); + const std::uint32_t bytes = num_frames * block; + // Only silence if the frames made it into the ring; if the host has + // stalled (ring full) keep playing locally rather than going dead + // silent — degrades to today's echo, never to silence. + if (block != 0 && audio_ring_push(*ring, t_gb_data, bytes, num_frames)) + { + std::memset(t_gb_data, 0, bytes); // belt-and-suspenders vs a driver ignoring SILENT + g_frames_captured.fetch_add(num_frames, std::memory_order_relaxed); + return g_vh_releasebuffer.original()( + self, num_frames, flags | AUDCLNT_BUFFERFLAGS_SILENT); + } } } + break; } return g_vh_releasebuffer.original()(self, num_frames, flags); } @@ -303,7 +302,9 @@ void register_render_client_locked(IAudioRenderClient* rc, const CapturedFormat& } g_registered = slot + 1; + g_stream_formats[slot] = cf; g_streams[slot].frames.store(0, std::memory_order_relaxed); + g_streams[slot].block_align.store(cf.block_align, std::memory_order_relaxed); // before client (hot path) g_streams[slot].client.store(rc, std::memory_order_release); AudioStreamInfo info{}; @@ -318,18 +319,13 @@ void register_render_client_locked(IAudioRenderClient* rc, const CapturedFormat& g_ipc->publish_audio_stream(slot, info); } - if (slot == 0) + // Publish this stream's format to its own ring if the host has attached one yet. + AudioRingHeader* ring = g_rings[slot].load(std::memory_order_acquire); + logf("stream %u set: rc=%p ring=%p fmt=%uHz/%uch/%ubit (%s)", slot, rc, ring, cf.rate, cf.channels, cf.bits, + ring ? "published" : "no ring yet"); + if (ring != nullptr) { - g_primary_format = cf; - g_primary_block_align.store(cf.block_align, std::memory_order_relaxed); - g_primary.store(rc, std::memory_order_release); - AudioRingHeader* ring = g_ring.load(std::memory_order_acquire); - logf("primary stream set: rc=%p ring=%p (format %s)", rc, ring, - ring ? "published" : "no ring yet"); - if (ring) - { - audio_ring_set_format(*ring, cf.rate, cf.channels, cf.bits, cf.tag, cf.block_align); - } + audio_ring_set_format(*ring, cf.rate, cf.channels, cf.bits, cf.tag, cf.block_align); } // GetBuffer/ReleaseBuffer are hooked proactively at install time (the shared // vtable covers every render client), so nothing to install per-stream here. @@ -464,7 +460,7 @@ bool install_audio_hooks(IpcClient& ipc, AudioRingHeader* ring) { std::scoped_lock lock(g_setup_mutex); g_ipc = &ipc; - g_ring.store(ring, std::memory_order_release); + g_rings[0].store(ring, std::memory_order_release); if (g_vh_activate) { return true; // anchor already installed @@ -568,33 +564,48 @@ bool install_audio_hooks(IpcClient& ipc, AudioRingHeader* ring) void republish_audio_format() { - AudioRingHeader* ring = g_ring.load(std::memory_order_acquire); - if (ring == nullptr || audio_ring_format_ready(*ring)) + // Nothing to do if no ring needs a format yet (cheap pre-check, no lock). + bool any_pending = false; + for (std::uint32_t i = 0; i < kMaxAudioStreams; ++i) { - return; // no ring yet, or the format is already published + AudioRingHeader* ring = g_rings[i].load(std::memory_order_acquire); + if (ring != nullptr && !audio_ring_format_ready(*ring)) + { + any_pending = true; + break; + } + } + if (!any_pending) + { + return; } std::scoped_lock lock(g_setup_mutex); - if (audio_ring_format_ready(*ring)) + for (std::uint32_t i = 0; i < kMaxAudioStreams; ++i) { - return; // raced with another publisher - } - if (g_primary.load(std::memory_order_acquire) != nullptr && g_primary_format.rate != 0) - { - audio_ring_set_format(*ring, g_primary_format.rate, g_primary_format.channels, g_primary_format.bits, - g_primary_format.tag, g_primary_format.block_align); - logf("republish_audio_format: published %uHz/%uch/%ubit to ring %p", g_primary_format.rate, - g_primary_format.channels, g_primary_format.bits, ring); + AudioRingHeader* ring = g_rings[i].load(std::memory_order_acquire); + if (ring == nullptr || audio_ring_format_ready(*ring) || g_stream_formats[i].rate == 0) + { + continue; // no ring, already published, or this slot has no stream yet + } + const CapturedFormat& cf = g_stream_formats[i]; + audio_ring_set_format(*ring, cf.rate, cf.channels, cf.bits, cf.tag, cf.block_align); + logf("republish_audio_format: stream %u -> %uHz/%uch/%ubit ring %p", i, cf.rate, cf.channels, cf.bits, + ring); } } -void set_audio_ring(AudioRingHeader* ring) +void set_audio_ring(unsigned index, AudioRingHeader* ring) { - g_ring.store(ring, std::memory_order_release); - logf("set_audio_ring: ring=%p capture_enabled=%u", ring, + if (index >= kMaxAudioStreams) + { + return; + } + g_rings[index].store(ring, std::memory_order_release); + logf("set_audio_ring: index=%u ring=%p capture_enabled=%u", index, ring, ring ? ring->capture_enabled.load(std::memory_order_relaxed) : 0u); - // The primary may already be registered (game was playing before we injected - // and before the host created the ring); publish its format so the host stops - // waiting and consumes the ring instead of falling back to loopback. + // The stream may already be registered (game was playing before we injected and + // before the host created the ring); publish its format so the host stops waiting + // and consumes the ring instead of falling back to loopback. republish_audio_format(); } @@ -624,20 +635,19 @@ void remove_audio_hooks() g_self_client = nullptr; } g_have_mix_format.store(0, std::memory_order_relaxed); - g_primary_format = CapturedFormat{}; g_registered = 0; g_streams_seen.store(0, std::memory_order_relaxed); - g_primary.store(nullptr, std::memory_order_release); - g_primary_block_align.store(0, std::memory_order_relaxed); g_frames_captured.store(0, std::memory_order_relaxed); - for (auto& s : g_streams) + for (std::uint32_t i = 0; i < kMaxAudioStreams; ++i) { - s.client.store(nullptr, std::memory_order_relaxed); - s.frames.store(0, std::memory_order_relaxed); + g_streams[i].client.store(nullptr, std::memory_order_relaxed); + g_streams[i].frames.store(0, std::memory_order_relaxed); + g_streams[i].block_align.store(0, std::memory_order_relaxed); + g_stream_formats[i] = CapturedFormat{}; + g_rings[i].store(nullptr, std::memory_order_release); } g_client_formats.clear(); - g_ring.store(nullptr, std::memory_order_release); g_ipc = nullptr; } diff --git a/hook/src/audio_hook.hpp b/hook/src/audio_hook.hpp index fcb1cf2..d0bf0fa 100644 --- a/hook/src/audio_hook.hpp +++ b/hook/src/audio_hook.hpp @@ -23,13 +23,15 @@ namespace coop::hook // anchor hook is in place; safe to call repeatedly (install-once internally). bool install_audio_hooks(IpcClient& ipc, AudioRingHeader* ring); -// Attach/replace the producer ring after install (e.g. host created it late). -void set_audio_ring(AudioRingHeader* ring); +// Attach/replace the producer ring for stream `index` after install (e.g. host +// created it late). Index 0 is the primary stream; the host creates one ring per +// tracked stream (coop_audio_[_]) and mixes them. +void set_audio_ring(unsigned index, AudioRingHeader* ring); -// Publish the registered primary stream's format to the attached ring if it -// isn't published yet. Idempotent; call periodically so a ring the host attaches -// (or re-initializes on a mirror re-toggle) gets the format even though the -// stream was registered earlier. No-op if there's no ring / no primary yet. +// Publish each registered stream's format to its attached ring if not published +// yet. Idempotent; call periodically so a ring the host attaches (or re-initializes +// on a mirror re-toggle) gets the format even though the stream was registered +// earlier. No-op for rings that have no stream yet. void republish_audio_format(); // Removes all installed render hooks (best effort; used on DLL detach). diff --git a/hook/src/dllmain.cpp b/hook/src/dllmain.cpp index c658083..3b2f7cb 100644 --- a/hook/src/dllmain.cpp +++ b/hook/src/dllmain.cpp @@ -30,8 +30,8 @@ namespace coop::hook::IpcClient g_ipc; std::atomic g_running{true}; -coop::SharedMemory g_audio_shm; // the host's audio ring, opened when present -coop::SharedMemory g_log_shm; // the host's log ring, opened when present +coop::SharedMemory g_audio_shm[coop::kMaxAudioStreams]; // per-stream audio rings, opened when present +coop::SharedMemory g_log_shm; // the host's log ring, opened when present DWORD WINAPI worker_thread(LPVOID) { @@ -168,31 +168,40 @@ DWORD WINAPI worker_thread(LPVOID) coop::hook::logf("worker_thread: MKB hooks removed (host request)"); } - if (audio_installed && !audio_ring_open) + // Attach a ring per stream. The host creates up to kMaxAudioStreams rings + // (coop_audio_[_]); we open each as it appears and (re)attach it so + // every stream is captured + silenced into its own ring for the host to mix. + if (audio_installed) { - const std::wstring name = coop::audio_ring_name(GetCurrentProcessId()); - if (!g_audio_shm.valid()) + for (unsigned i = 0; i < coop::kMaxAudioStreams; ++i) { - g_audio_shm.open(name, coop::audio_ring_total_size(coop::kAudioRingCapacity)); - } - if (g_audio_shm.valid()) - { - auto* ring = g_audio_shm.as(); - if (coop::audio_ring_valid(*ring)) + if (!g_audio_shm[i].valid()) { - coop::hook::set_audio_ring(ring); - audio_ring_open = true; - coop::hook::logf("worker_thread: audio ring opened"); + g_audio_shm[i].open(coop::audio_ring_name(GetCurrentProcessId(), i), + coop::audio_ring_total_size(coop::kAudioRingCapacity)); } - else + if (g_audio_shm[i].valid()) { - g_audio_shm.reset(); // present but not our contract; retry + auto* ring = g_audio_shm[i].as(); + if (coop::audio_ring_valid(*ring)) + { + coop::hook::set_audio_ring(i, ring); // idempotent re-attach + if (i == 0 && !audio_ring_open) + { + audio_ring_open = true; + coop::hook::logf("worker_thread: audio ring 0 opened"); + } + } + else + { + g_audio_shm[i].reset(); // present but not our contract; retry + } } } } - // The primary stream is often registered before the ring is attached (or the - // host re-inits the ring on a mirror re-toggle, clearing its format); keep - // the format published so the host consumes the ring instead of falling back. + // A stream is often registered before its ring is attached (or the host re-inits + // a ring on a mirror re-toggle, clearing its format); keep formats published so + // the host consumes the rings instead of falling back to loopback. if (audio_ring_open) { coop::hook::republish_audio_format(); diff --git a/host/src/audio/audio_loopback.cpp b/host/src/audio/audio_loopback.cpp index 709b19e..010c4e4 100644 --- a/host/src/audio/audio_loopback.cpp +++ b/host/src/audio/audio_loopback.cpp @@ -9,6 +9,7 @@ #include #include +#include "audio/audio_mix.hpp" #include "audio/process_loopback_capture.hpp" namespace coop @@ -151,7 +152,10 @@ void AudioMirror::stop() CloseHandle(stop_event_); stop_event_ = nullptr; } - audio_ring_shm_.reset(); + for (auto& shm : audio_ring_shm_) + { + shm.reset(); + } running_.store(false, std::memory_order_release); source_.store(Source::None, std::memory_order_relaxed); buffered_ms_.store(0, std::memory_order_relaxed); @@ -191,15 +195,24 @@ void AudioMirror::thread_main(DWORD pid) // capture. If the hook is present it publishes a format within ~1 s and we // consume the ring (no echo); otherwise we fall back to process loopback. bool handled = false; - if (audio_ring_shm_.create(audio_ring_name(pid), audio_ring_total_size(kAudioRingCapacity))) + AudioRingHeader* rings[kMaxAudioStreams] = {}; + bool created_primary = false; + for (unsigned i = 0; i < kMaxAudioStreams; ++i) { - auto* ring = audio_ring_shm_.as(); - audio_ring_init(*ring, kAudioRingCapacity); - ring->capture_enabled.store(1, std::memory_order_release); - set_status("Waiting for render-hook…"); - if (wait_for_format(ring, 1000)) + if (audio_ring_shm_[i].create(audio_ring_name(pid, i), audio_ring_total_size(kAudioRingCapacity))) { - handled = run_hooked(ring); + rings[i] = audio_ring_shm_[i].as(); + audio_ring_init(*rings[i], kAudioRingCapacity); + rings[i]->capture_enabled.store(1, std::memory_order_release); + created_primary = created_primary || (i == 0); + } + } + if (created_primary) + { + set_status("Waiting for render-hook…"); + if (wait_for_format(rings[0], 1000)) + { + handled = run_hooked(rings); } } @@ -208,7 +221,10 @@ void AudioMirror::thread_main(DWORD pid) run_loopback(pid); } - audio_ring_shm_.reset(); + for (auto& shm : audio_ring_shm_) + { + shm.reset(); + } if (running_.load(std::memory_order_acquire)) { @@ -226,17 +242,28 @@ void AudioMirror::thread_main(DWORD pid) // Consume the render-hook's shared ring and re-render the game's frames. The // game is silenced locally by the hook, so the operator hears no echo. Returns // true if it ran to a clean stop; false on setup failure (caller falls back). -bool AudioMirror::run_hooked(AudioRingHeader* ring) +bool AudioMirror::run_hooked(AudioRingHeader* const* rings) { + AudioRingHeader* primary = rings[0]; + auto disable_all = [&] { + for (unsigned i = 0; i < kMaxAudioStreams; ++i) + { + if (rings[i] != nullptr) + { + rings[i]->capture_enabled.store(0, std::memory_order_release); + } + } + }; auto fail_to_loopback = [&] { - ring->capture_enabled.store(0, std::memory_order_release); // let the game play locally again + disable_all(); // let the game play locally again return false; }; - const unsigned rate = ring->sample_rate; - const unsigned channels = ring->channels; - const unsigned bits = ring->bits; - const unsigned block_align = ring->block_align ? ring->block_align : channels * (bits / 8); + const unsigned rate = primary->sample_rate; + const unsigned channels = primary->channels; + const unsigned bits = primary->bits; + const unsigned tag = primary->format_tag; + const unsigned block_align = primary->block_align ? primary->block_align : channels * (bits / 8); if (rate == 0 || channels == 0 || block_align == 0) { return fail_to_loopback(); @@ -267,12 +294,12 @@ bool AudioMirror::run_hooked(AudioRingHeader* ring) wfx.dwChannelMask = (channels >= 32) ? 0xFFFFFFFFu : ((1u << channels) - 1u); break; } - wfx.SubFormat = (ring->format_tag == WAVE_FORMAT_IEEE_FLOAT) ? KSDATAFORMAT_SUBTYPE_IEEE_FLOAT - : KSDATAFORMAT_SUBTYPE_PCM; + wfx.SubFormat = + (tag == WAVE_FORMAT_IEEE_FLOAT) ? KSDATAFORMAT_SUBTYPE_IEEE_FLOAT : KSDATAFORMAT_SUBTYPE_PCM; } else { - wfx.Format.wFormatTag = static_cast(ring->format_tag ? ring->format_tag : WAVE_FORMAT_PCM); + wfx.Format.wFormatTag = static_cast(tag ? tag : WAVE_FORMAT_PCM); wfx.Format.cbSize = 0; } auto* fmt = reinterpret_cast(&wfx); @@ -355,6 +382,12 @@ bool AudioMirror::run_hooked(AudioRingHeader* ring) const size_t prime_bytes = frame_bytes * (rate * 30 / 1000); // ~30 ms before feeding bool primed = false; + // Mixing scratch (only used when >1 same-format stream is active): a per-stream + // temp buffer and a float accumulator sized to the render buffer. + const bool mixer_ok = mix_format_supported(tag, bits); + std::vector temp(static_cast(render_frames) * frame_bytes); + std::vector acc(static_cast(render_frames) * channels); + if (FAILED(hr = render_client->Start())) { fail("Render Start", hr); @@ -380,7 +413,8 @@ bool AudioMirror::run_hooked(AudioRingHeader* ring) continue; } const UINT32 avail = render_frames - padding; - const std::uint32_t ring_bytes = audio_ring_available(*ring); + // The primary ring is the master clock for priming + how much to write. + const std::uint32_t ring_bytes = audio_ring_available(*primary); buffered_ms_.store(static_cast(ring_bytes / frame_bytes * 1000 / rate), std::memory_order_relaxed); if (!primed && ring_bytes >= prime_bytes) @@ -393,10 +427,47 @@ bool AudioMirror::run_hooked(AudioRingHeader* ring) const UINT32 to_write = std::min(avail, have); if (to_write > 0) { + // Active streams = same format as primary (so they can be summed). + // Streams with a different format are still silenced by the hook (no + // echo) but can't be mixed here without resampling -> skipped. + unsigned active[kMaxAudioStreams]; + unsigned n_active = 0; + for (unsigned i = 0; i < kMaxAudioStreams; ++i) + { + AudioRingHeader* r = rings[i]; + if (r == nullptr) + { + continue; + } + if (i == 0 || (audio_ring_format_ready(*r) && r->sample_rate == rate && + r->channels == channels && r->bits == bits && r->format_tag == tag)) + { + active[n_active++] = i; + } + } + BYTE* dst = nullptr; if (SUCCEEDED(render->GetBuffer(to_write, &dst))) { - audio_ring_pop(*ring, dst, to_write * static_cast(frame_bytes)); + const std::uint32_t want_bytes = to_write * static_cast(frame_bytes); + if (n_active <= 1 || !mixer_ok) + { + // Single stream (the common case) or an unmixable format: + // passthrough the primary, byte-for-byte (no mixer overhead). + audio_ring_pop(*primary, dst, want_bytes); + } + else + { + const std::uint32_t samples = to_write * channels; + std::fill(acc.begin(), acc.begin() + samples, 0.0f); + for (unsigned k = 0; k < n_active; ++k) + { + std::memset(temp.data(), 0, want_bytes); // zero-fill short reads + audio_ring_pop(*rings[active[k]], temp.data(), want_bytes); + mix_add(acc.data(), temp.data(), samples, tag, bits); + } + mix_store(dst, acc.data(), samples, tag, bits); + } render->ReleaseBuffer(to_write, 0); } } @@ -410,7 +481,7 @@ bool AudioMirror::run_hooked(AudioRingHeader* ring) render_client->Stop(); } while (false); - ring->capture_enabled.store(0, std::memory_order_release); // game audible again on stop + disable_all(); // game audible again on stop if (render) { diff --git a/host/src/audio/audio_loopback.hpp b/host/src/audio/audio_loopback.hpp index dfcad8e..c65ad9a 100644 --- a/host/src/audio/audio_loopback.hpp +++ b/host/src/audio/audio_loopback.hpp @@ -18,6 +18,7 @@ #include #include "coop/audio_ring.hpp" +#include "coop/protocol.hpp" // kMaxAudioStreams #include "coop/shared_memory.hpp" namespace coop @@ -97,8 +98,9 @@ public: private: void thread_main(DWORD pid); // Returns true if it owned the session to a clean stop; false if setup failed - // and the caller should fall back to the loopback path. - bool run_hooked(AudioRingHeader* ring); + // and the caller should fall back to the loopback path. `rings[0]` is the primary + // stream; additional non-null rings are mixed in. + bool run_hooked(AudioRingHeader* const* rings); void run_loopback(DWORD pid); bool wait_for_format(AudioRingHeader* ring, DWORD timeout_ms); bool stop_requested() const; @@ -108,7 +110,7 @@ private: HANDLE stop_event_ = nullptr; DWORD pid_ = 0; - SharedMemory audio_ring_shm_; // host-created shared ring (named coop_audio_) + SharedMemory audio_ring_shm_[kMaxAudioStreams]; // per-stream rings (coop_audio_[_]) std::atomic running_{false}; std::atomic source_{Source::None}; diff --git a/host/src/audio/audio_mix.hpp b/host/src/audio/audio_mix.hpp new file mode 100644 index 0000000..ef59a37 --- /dev/null +++ b/host/src/audio/audio_mix.hpp @@ -0,0 +1,77 @@ +// Pure helpers for mixing several same-format audio streams (float32 or int16) +// into one render buffer, with a soft clip on the sum. Header-only so the host +// mixer and a unit test share the exact same math. +#pragma once + +#include +#include + +namespace coop +{ + +// WAVE_FORMAT_* values used here (kept local to avoid an mmreg.h dependency). +inline constexpr std::uint32_t kWaveFormatPcm = 1; +inline constexpr std::uint32_t kWaveFormatFloat = 3; + +// The mixer decodes/encodes these formats; anything else falls back to a plain +// single-stream passthrough on the host (no mixing). +inline bool mix_format_supported(std::uint32_t format_tag, std::uint32_t bits) +{ + return (format_tag == kWaveFormatFloat && bits == 32) || (format_tag == kWaveFormatPcm && bits == 16); +} + +// Smoothly limit the summed signal to [-1, 1]; ~identity for small inputs. +inline float soft_clip(float x) +{ + return std::tanh(x); +} + +// Decode `samples` interleaved samples from `src` (format_tag/bits) and add them +// into the float accumulator `acc`. No-op for unsupported formats. +inline void mix_add(float* acc, const std::uint8_t* src, std::uint32_t samples, std::uint32_t format_tag, + std::uint32_t bits) +{ + if (format_tag == kWaveFormatFloat && bits == 32) + { + const auto* f = reinterpret_cast(src); + for (std::uint32_t i = 0; i < samples; ++i) + { + acc[i] += f[i]; + } + } + else if (format_tag == kWaveFormatPcm && bits == 16) + { + const auto* s = reinterpret_cast(src); + for (std::uint32_t i = 0; i < samples; ++i) + { + acc[i] += static_cast(s[i]) / 32768.0f; + } + } +} + +// Soft-clip `acc` and encode `samples` interleaved samples into `dst` in the given +// format. No-op for unsupported formats. +inline void mix_store(std::uint8_t* dst, const float* acc, std::uint32_t samples, std::uint32_t format_tag, + std::uint32_t bits) +{ + if (format_tag == kWaveFormatFloat && bits == 32) + { + auto* f = reinterpret_cast(dst); + for (std::uint32_t i = 0; i < samples; ++i) + { + f[i] = soft_clip(acc[i]); + } + } + else if (format_tag == kWaveFormatPcm && bits == 16) + { + auto* s = reinterpret_cast(dst); + for (std::uint32_t i = 0; i < samples; ++i) + { + int v = static_cast(soft_clip(acc[i]) * 32767.0f); + v = v > 32767 ? 32767 : (v < -32768 ? -32768 : v); + s[i] = static_cast(v); + } + } +} + +} // namespace coop diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 99a4ba4..c5fbcae 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -25,6 +25,11 @@ add_executable(mkb_ring_test mkb_ring_test.cpp) target_link_libraries(mkb_ring_test PRIVATE coop_common) add_test(NAME mkb_ring_test COMMAND mkb_ring_test) +# Unit test for the audio mixer math (decode/sum/soft-clip/encode). Header-only. +add_executable(audio_mix_test audio_mix_test.cpp) +target_include_directories(audio_mix_test PRIVATE ${CMAKE_SOURCE_DIR}/host/src) +add_test(NAME audio_mix_test COMMAND audio_mix_test) + # Unit test for the host->game mouse coordinate mapping (letterbox inverse + # decorated-window client offset). Header-only, no device. add_executable(mkb_map_test mkb_map_test.cpp) @@ -148,6 +153,7 @@ coop_output_subdir(tests audio_ring_test mkb_ring_test mkb_map_test + audio_mix_test audio_loopback_test audio_hook_test srgb_format_test diff --git a/tests/audio_mix_test.cpp b/tests/audio_mix_test.cpp new file mode 100644 index 0000000..04744d5 --- /dev/null +++ b/tests/audio_mix_test.cpp @@ -0,0 +1,89 @@ +// Unit test for the audio mixer math (decode/sum/soft-clip/encode, float32 + int16). +#include +#include +#include +#include +#include + +#include "audio/audio_mix.hpp" + +using namespace coop; + +namespace +{ +int g_failures = 0; +void check(bool ok, const char* what) +{ + if (!ok) + { + std::printf("FAIL: %s\n", what); + ++g_failures; + } +} +bool near_f(float a, float b) +{ + return std::fabs(a - b) < 1e-4f; +} +} // namespace + +int main() +{ + check(mix_format_supported(kWaveFormatFloat, 32), "float32 supported"); + check(mix_format_supported(kWaveFormatPcm, 16), "int16 supported"); + check(!mix_format_supported(kWaveFormatPcm, 24), "24-bit not supported"); + + // soft_clip is ~identity for small inputs and bounded for large ones. + check(near_f(soft_clip(0.0f), 0.0f), "soft_clip(0)=0"); + check(soft_clip(10.0f) <= 1.0f && soft_clip(10.0f) > 0.99f, "soft_clip bounds large +"); + check(soft_clip(-10.0f) >= -1.0f && soft_clip(-10.0f) < -0.99f, "soft_clip bounds large -"); + + // --- float32: two streams sum, small values pass ~unchanged --- + { + const float a[4] = {0.1f, -0.2f, 0.3f, -0.05f}; + const float b[4] = {0.2f, 0.1f, -0.1f, 0.05f}; + float acc[4] = {}; + mix_add(acc, reinterpret_cast(a), 4, kWaveFormatFloat, 32); + mix_add(acc, reinterpret_cast(b), 4, kWaveFormatFloat, 32); + float out[4] = {}; + mix_store(reinterpret_cast(out), acc, 4, kWaveFormatFloat, 32); + // Sum then tanh; small sums are ~unchanged. + for (int i = 0; i < 4; ++i) + { + check(near_f(out[i], std::tanh(a[i] + b[i])), "float32 mix == tanh(sum)"); + } + } + + // --- float32: summing many loud streams stays within [-1, 1] (soft clip) --- + { + float acc[2] = {}; + const float loud[2] = {0.9f, -0.9f}; + for (int s = 0; s < 5; ++s) + { + mix_add(acc, reinterpret_cast(loud), 2, kWaveFormatFloat, 32); + } + float out[2] = {}; + mix_store(reinterpret_cast(out), acc, 2, kWaveFormatFloat, 32); + check(out[0] <= 1.0f && out[0] > 0.99f, "loud sum soft-clipped near +1"); + check(out[1] >= -1.0f && out[1] < -0.99f, "loud sum soft-clipped near -1"); + } + + // --- int16: decode/encode round-trip of a single quiet stream --- + { + const std::int16_t a[2] = {1000, -2000}; + float acc[2] = {}; + mix_add(acc, reinterpret_cast(a), 2, kWaveFormatPcm, 16); + check(near_f(acc[0], 1000.0f / 32768.0f) && near_f(acc[1], -2000.0f / 32768.0f), "int16 decode"); + std::int16_t out[2] = {}; + mix_store(reinterpret_cast(out), acc, 2, kWaveFormatPcm, 16); + // tanh of a tiny value ~ the value, so re-encoding is within a couple of LSB. + check(std::abs(out[0] - 1000) <= 3 && std::abs(out[1] - (-2000)) <= 3, "int16 round-trip"); + } + + if (g_failures == 0) + { + std::printf("PASS: audio_mix_test\n"); + return 0; + } + std::printf("FAIL: %d checks\n", g_failures); + return 1; +}