From 2b4f60867bc4f2f750606d7555192c55570d7586 Mon Sep 17 00:00:00 2001 From: TUNA_BIN <65464657+TUNA-NOPE@users.noreply.github.com> Date: Tue, 10 Nov 2026 07:55:39 +0000 Subject: [PATCH] Production I: 117 fps at 3x, three simulators at once; Coherent on by default on Arrow Lake macbox build now makes the build that has run day to day since 2026-11-09 ("production I"): - patches/metal2vulkan: 92 patches on 0db5825a (was 34), rebuilding m2v 7fee47f5 exactly: grouped completion, inline_lazy, inline_checks, the surface fixes the lanes and the census found (make_once, reader_first, own_back, open_ids, max_texels), lab switches (all off) and diagnostics (off unless set). - patches/reims-vgpu: 33 patches on 7198fe6 (was 25), Reims 8dea1e59 without its local build wiring: staged page-ups, REIMS_VGPU_GROUPED, FIFO exec pipelining, interrupt moderation. - patches/qemu: 13 patches on bd95509 (was 9), QEMU 3999026c68: KVM irqfd, ioeventfd and halt-poll properties for the GPU device (off unless set). Each series was applied as container/build-reims.sh applies it, on fresh checkouts of the pinned commits, and compared with the production sources: no difference. macbox: MACBOX_COHERENT defaults to auto, on with an Arrow Lake iGPU (where it runs every day), and turns on the switches measured with Coherent: the live.conf lines, REIMS_VGPU_GROUPED and the file-named Reims switches in $MACBOX_HOME/run/switches. macbox rewrites only its own live.conf lines, so a mode change applies and the user's lines stay. README, docs, website and llms.txt: a clearer pitch up front, the new numbers (3x simulator 17.2 upstream -> 117 fps; 2 at once 51-54 each, 3 at once 31-32 each; desktop drag 240 ticks/s alone, 126 with a simulator), a 3x frame broken into real rendering and what translation adds, what was tried without gain since, and the open GPU hang under repeated 3-simulator boots. Co-authored-by: Claude Opus 5.5 --- README.md | 73 +- docs/architecture.md | 3 +- docs/images/fps-dark.svg | 72 +- docs/images/fps-light.svg | 72 +- docs/images/make-chart.py | 24 +- docs/performance.md | 132 ++- docs/setup.md | 2 +- llms.txt | 4 +- macbox | 66 +- patches/README.md | 31 +- ...-live-switch-groups-up-to-16-batches.patch | 124 +++ ...ile-Coherent-s-wait_settled-waits-Ho.patch | 127 +++ ...e-group_batches-up-to-64-hold-delay-.patch | 114 ++ ...-host-wait-sends-held-groups-HostWai.patch | 37 + ...ine_us-a-group-held-past-it-goes-at-.patch | 29 + ...-stays-SIMD16-and-counts-a-1-D-workg.patch | 483 +++++++++ ...letion-and-cross-import-coherence-ch.patch | 160 +++ ...al-leaves-a-render-end-in-flight-on-.patch | 70 ++ ...t-queues-render-calls-on-a-bounded-s.patch | 953 +++++++++++++++++ ...wering-lanes-on-first-use-so-idle-qu.patch | 127 +++ ...ied-textures-images-that-the-host-wr.patch | 298 ++++++ ...py-initialization-of-recycled-images.patch | 97 ++ ...raw-units-live-pass_draws-and-a-pipe.patch | 410 ++++++++ ...a-copy-unit-s-images-in-the-pass-tim.patch | 137 +++ ...1714-behind-live-group_keep-0-group_.patch | 385 +++++++ ...y_max_texels-keeps-large-surfaces-th.patch | 31 + ...n_reader_first-a-lazy-surface-s-read.patch | 134 +++ ...tic-compute-census-in-the-pass-timer.patch | 118 +++ ...IEW_SHADOW_LOG-names-the-views-that-.patch | 50 + ...rite_risk-by-site-in-the-coherent-ce.patch | 110 ++ ...WIN_LOG-says-why-a-texture-over-a-la.patch | 81 ++ ...ESIDE_LOG-names-the-commands-writing.patch | 44 + ...WIN_LOG-also-names-views-apart-from-.patch | 48 + ...ge_shadow_image-a-lazy-surface-witho.patch | 35 + ...erent_own_back-a-trapping-surface-s-.patch | 60 ++ ..._own_back-tags-own-copies-back-inste.patch | 153 +++ ...y_open_ids-a-standing-s-command-buff.patch | 122 +++ ...its_off-lab-ceiling-of-the-unused-bi.patch | 30 + ...shadows-copy-back-lab-diagnostics-pr.patch | 140 +++ ...k-dispatches-by-site-lab-diagnostics.patch | 109 ++ ...r-refusal-counters-and-coherent-stat.patch | 94 ++ ...-second-over-trapped-pages-lab-diagn.patch | 53 + ...a-large-lazy-surface-s-shadow-lazy-i.patch | 102 ++ ...rent_writer_wins-live-switch-default.patch | 161 +++ ..._make_once-live-switch-default-off-t.patch | 87 ++ ..._make_once-waits-per-surface-not-glo.patch | 87 ++ ...T-diagnostic-begins-resets-of-comman.patch | 515 +++++++++ ...ads-each-semaphore-through-the-devic.patch | 76 ++ ...ads-no-fence-and-only-long-lived-tim.patch | 72 ++ ...vice-index-no-longer-shadowed-by-the.patch | 189 ++++ ...rgets-destroyed-devices-and-a-hit-li.patch | 109 ++ ...E-TWxTH-a-flat-grid-s-32-bit-index-w.patch | 125 +++ ...ccs-1-a-GPU-written-Shared-Managed-t.patch | 33 + ...ips-semaphores-that-read-0-binary-se.patch | 26 + ...udit-skips-semaphores-that-read-0-bi.patch | 27 + ...e-host-transfer-color-images-mutable.patch | 33 + ...ight-no-end-barrier-per-batch-in-exa.patch | 50 + ...IEW_LOG-names-views-kept-apart-from-.patch | 36 + ..._direct-an-A8Unorm-view-of-an-R8Unor.patch | 79 ++ ...ct-also-for-reinterpreted-views-a-vi.patch | 67 ++ ...-a-lazy-surface-s-passes-record-thei.patch | 49 + ...BARRIER_LOG-and-live-precise_barrier.patch | 351 +++++++ ...er-log-counts-image-to-image-shadow-.patch | 98 ++ ...uler-log-M2V_SCHED_LOG-commit-call-i.patch | 207 ++++ ...uler-log-held-reasons-per-logical-qu.patch | 144 +++ ...t-1-a-left-submission-check-runs-bef.patch | 44 + ...uler-log-a-checked-head-s-landing-an.patch | 89 ++ ...ks-1-run-a-batch-s-submission-checks.patch | 55 + ...I-through-a-KVM-irqfd-x-irqfd-live-s.patch | 333 ++++++ ...halt-poll-ns-a-live-per-VM-KVM_CAP_H.patch | 134 +++ ...keFifo-doorbells-through-KVM-ioevent.patch | 229 ++++ ...ioeventfd-KVM-always-has-eventfds-he.patch | 24 + ...direct3-on-a-FIFO-prototype-off-unle.patch | 991 ++++++++++++++++++ ...ED-1-boot-switch-for-grouped-complet.patch | 34 + ...tion-off-unless-REIMS_VGPU_IRQ_MODER.patch | 297 ++++++ ...d-render-lowering-and-multi-queue-re.patch | 382 +++++++ ..._BUFFER_BY_PAGES-1-finds-kept-paged-.patch | 70 ++ ...a-new-texture-s-first-page-up-copies.patch | 698 ++++++++++++ ...-ups-a-texture-s-first-prepare-over-.patch | 508 +++++++++ ...read-guest-pages-with-one-memcpy-per.patch | 64 ++ .../gpu-accelerated-macos-vm-on-linux.html | 13 +- site/pages/guides/ios-simulator-on-linux.html | 4 +- site/pages/guides/run-macos-on-linux.html | 2 +- site/pages/index.html | 22 +- versions.env | 2 +- 85 files changed, 11977 insertions(+), 183 deletions(-) create mode 100644 patches/metal2vulkan/0035-group_batches-live-switch-groups-up-to-16-batches.patch create mode 100644 patches/metal2vulkan/0036-no-group-kept-while-Coherent-s-wait_settled-waits-Ho.patch create mode 100644 patches/metal2vulkan/0037-group-cap-64-live-group_batches-up-to-64-hold-delay-.patch create mode 100644 patches/metal2vulkan/0038-a-shared-event-s-host-wait-sends-held-groups-HostWai.patch create mode 100644 patches/metal2vulkan/0039-live-group_deadline_us-a-group-held-past-it-goes-at-.patch create mode 100644 patches/metal2vulkan/0040-the-flat-variant-stays-SIMD16-and-counts-a-1-D-workg.patch create mode 100644 patches/metal2vulkan/0041-Add-grouped-completion-and-cross-import-coherence-ch.patch create mode 100644 patches/metal2vulkan/0042-a-scheduled-signal-leaves-a-render-end-in-flight-on-.patch create mode 100644 patches/metal2vulkan/0043-lower-independent-queues-render-calls-on-a-bounded-s.patch create mode 100644 patches/metal2vulkan/0044-Assign-render-lowering-lanes-on-first-use-so-idle-qu.patch create mode 100644 patches/metal2vulkan/0045-Recycle-host-copied-textures-images-that-the-host-wr.patch create mode 100644 patches/metal2vulkan/0046-Clarify-host-copy-initialization-of-recycled-images.patch create mode 100644 patches/metal2vulkan/0047-diagnostic-per-draw-units-live-pass_draws-and-a-pipe.patch create mode 100644 patches/metal2vulkan/0048-diagnostic-name-a-copy-unit-s-images-in-the-pass-tim.patch create mode 100644 patches/metal2vulkan/0049-port-strong-d2da1714-behind-live-group_keep-0-group_.patch create mode 100644 patches/metal2vulkan/0050-live-surface_lazy_max_texels-keeps-large-surfaces-th.patch create mode 100644 patches/metal2vulkan/0051-live-surface_twin_reader_first-a-lazy-surface-s-read.patch create mode 100644 patches/metal2vulkan/0052-diagnostic-compute-census-in-the-pass-timer.patch create mode 100644 patches/metal2vulkan/0053-diagnostic-M2V_VIEW_SHADOW_LOG-names-the-views-that-.patch create mode 100644 patches/metal2vulkan/0054-diagnostic-overwrite_risk-by-site-in-the-coherent-ce.patch create mode 100644 patches/metal2vulkan/0055-diagnostic-M2V_TWIN_LOG-says-why-a-texture-over-a-la.patch create mode 100644 patches/metal2vulkan/0056-diagnostic-M2V_BESIDE_LOG-names-the-commands-writing.patch create mode 100644 patches/metal2vulkan/0057-diagnostic-M2V_TWIN_LOG-also-names-views-apart-from-.patch create mode 100644 patches/metal2vulkan/0058-live-surface_usage_shadow_image-a-lazy-surface-witho.patch create mode 100644 patches/metal2vulkan/0059-live-surface_coherent_own_back-a-trapping-surface-s-.patch create mode 100644 patches/metal2vulkan/0060-surface_coherent_own_back-tags-own-copies-back-inste.patch create mode 100644 patches/metal2vulkan/0061-live-surface_lazy_open_ids-a-standing-s-command-buff.patch create mode 100644 patches/metal2vulkan/0062-live-xr_unused_bits_off-lab-ceiling-of-the-unused-bi.patch create mode 100644 patches/metal2vulkan/0063-count-why-large-shadows-copy-back-lab-diagnostics-pr.patch create mode 100644 patches/metal2vulkan/0064-count-copy-back-dispatches-by-site-lab-diagnostics.patch create mode 100644 patches/metal2vulkan/0065-per-surface-defer-refusal-counters-and-coherent-stat.patch create mode 100644 patches/metal2vulkan/0066-print-who-tracks-second-over-trapped-pages-lab-diagn.patch create mode 100644 patches/metal2vulkan/0067-print-who-makes-a-large-lazy-surface-s-shadow-lazy-i.patch create mode 100644 patches/metal2vulkan/0068-lab-surface_coherent_writer_wins-live-switch-default.patch create mode 100644 patches/metal2vulkan/0069-lab-surface_twin_make_once-live-switch-default-off-t.patch create mode 100644 patches/metal2vulkan/0070-lab-surface_twin_make_once-waits-per-surface-not-glo.patch create mode 100644 patches/metal2vulkan/0071-M2V_PENDING_AUDIT-diagnostic-begins-resets-of-comman.patch create mode 100644 patches/metal2vulkan/0072-pending-audit-reads-each-semaphore-through-the-devic.patch create mode 100644 patches/metal2vulkan/0073-pending-audit-reads-no-fence-and-only-long-lived-tim.patch create mode 100644 patches/metal2vulkan/0074-pending-audit-device-index-no-longer-shadowed-by-the.patch create mode 100644 patches/metal2vulkan/0075-pending-audit-forgets-destroyed-devices-and-a-hit-li.patch create mode 100644 patches/metal2vulkan/0076-lab-M2V_FLAT_TILE-TWxTH-a-flat-grid-s-32-bit-index-w.patch create mode 100644 patches/metal2vulkan/0077-lab-live-shared_ccs-1-a-GPU-written-Shared-Managed-t.patch create mode 100644 patches/metal2vulkan/0078-pending-audit-skips-semaphores-that-read-0-binary-se.patch create mode 100644 patches/metal2vulkan/0079-Revert-pending-audit-skips-semaphores-that-read-0-bi.patch create mode 100644 patches/metal2vulkan/0080-live-host_mutable-host-transfer-color-images-mutable.patch create mode 100644 patches/metal2vulkan/0081-live-batch_end_light-no-end-barrier-per-batch-in-exa.patch create mode 100644 patches/metal2vulkan/0082-diagnostic-M2V_VIEW_LOG-names-views-kept-apart-from-.patch create mode 100644 patches/metal2vulkan/0083-live-view_render_direct-an-A8Unorm-view-of-an-R8Unor.patch create mode 100644 patches/metal2vulkan/0084-view_render_direct-also-for-reinterpreted-views-a-vi.patch create mode 100644 patches/metal2vulkan/0085-live-inline_lazy-a-lazy-surface-s-passes-record-thei.patch create mode 100644 patches/metal2vulkan/0086-barrier-log-M2V_BARRIER_LOG-and-live-precise_barrier.patch create mode 100644 patches/metal2vulkan/0087-diagnostic-barrier-log-counts-image-to-image-shadow-.patch create mode 100644 patches/metal2vulkan/0088-diagnostic-scheduler-log-M2V_SCHED_LOG-commit-call-i.patch create mode 100644 patches/metal2vulkan/0089-diagnostic-scheduler-log-held-reasons-per-logical-qu.patch create mode 100644 patches/metal2vulkan/0090-live-checks_first-1-a-left-submission-check-runs-bef.patch create mode 100644 patches/metal2vulkan/0091-diagnostic-scheduler-log-a-checked-head-s-landing-an.patch create mode 100644 patches/metal2vulkan/0092-live-inline_checks-1-run-a-batch-s-submission-checks.patch create mode 100644 patches/qemu/0010-apple-gfx-pci-MSI-through-a-KVM-irqfd-x-irqfd-live-s.patch create mode 100644 patches/qemu/0011-apple-gfx-pci-x-halt-poll-ns-a-live-per-VM-KVM_CAP_H.patch create mode 100644 patches/qemu/0012-apple-gfx-pci-WakeFifo-doorbells-through-KVM-ioevent.patch create mode 100644 patches/qemu/0013-apple-gfx-pci-x-ioeventfd-KVM-always-has-eventfds-he.patch create mode 100644 patches/reims-vgpu/0026-pipelined-ExecIndirect3-on-a-FIFO-prototype-off-unle.patch create mode 100644 patches/reims-vgpu/0027-REIMS_VGPU_GROUPED-1-boot-switch-for-grouped-complet.patch create mode 100644 patches/reims-vgpu/0028-interrupt-moderation-off-unless-REIMS_VGPU_IRQ_MODER.patch create mode 100644 patches/reims-vgpu/0029-Integrate-guarded-render-lowering-and-multi-queue-re.patch create mode 100644 patches/reims-vgpu/0030-REIMS_VGPU_PAGED_BUFFER_BY_PAGES-1-finds-kept-paged-.patch create mode 100644 patches/reims-vgpu/0031-staged-page-ups-a-new-texture-s-first-page-up-copies.patch create mode 100644 patches/reims-vgpu/0032-staged-zero-page-ups-a-texture-s-first-prepare-over-.patch create mode 100644 patches/reims-vgpu/0033-staged-page-ups-read-guest-pages-with-one-memcpy-per.patch diff --git a/README.md b/README.md index 1bce720..2dbaab0 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,20 @@ -# macbox: macOS, Xcode and the iOS Simulator on Linux, GPU-accelerated, for AI coding agents +# macbox -**Run the real iOS Simulator on a Linux machine, without a Mac and without a cloud service.** -macbox runs a full macOS VM in Docker with real GPU acceleration (Metal, through Reims vGPU on -your Vulkan GPU), with Xcode, a pool of iOS Simulators, T3 Code's Device panel, and the tools AI -coding agents (Claude Code, Codex, T3 Code) need to build, run, drive and check iOS apps on their -own. One command, `macbox`, sets it up and runs it. +### The real iOS Simulator and Xcode on your Linux machine. GPU-accelerated. No Mac. + +macbox runs a full macOS VM in Docker that draws with **Metal on your Linux GPU**, so Xcode and +the iOS Simulator run on hardware you already have. An iPhone 17 Pro simulator at full 3x +resolution, in a heavy Liquid Glass scene, runs at **117 fps on an Intel integrated GPU**. Before +macbox's patches the same GPU managed 17. It's built for AI coding agents: Claude Code, Codex and +T3 Code each get their own simulator to build, run, tap, screenshot and record iOS apps on their own. +One command, `macbox`, builds it and runs it. + +| | | +|---|---| +| **117 fps** | iPhone 17 Pro simulator at 3x (1206×2622), Liquid Glass, on an Arrow Lake iGPU; 17.2 before | +| **3 at once** | three 3x simulators side by side, ~32 fps each | +| **No Mac, no cloud** | a Linux host with KVM and Docker; nothing billed per minute | +| **Agent-ready** | `mac-sim` gives every checkout its own simulator; `sim-fps` checks 60 fps; T3 Code's Device panel streams it live | **Website and guides:** [tuna-nope.github.io/macbox](https://tuna-nope.github.io/macbox/): macOS, Xcode and the iOS Simulator on Linux, Expo apps, the GPU path, and AI coding agents with T3 Code. @@ -27,8 +37,8 @@ Linux host (KVM + a Vulkan GPU + Docker) - **macOS draws on your GPU.** [Reims vGPU](https://github.com/steelbrain/reims-vgpu) gives QEMU Apple's paravirtual GPU, and macOS's own driver talks to it, so Metal works with no guest driver. [metal2vulkan](https://github.com/steelbrain/metal2vulkan) runs Metal's work on Vulkan. - macbox builds both, with patches that took a 3x iOS Simulator from 17 to 50 fps on the GPU it - was measured on, and to 70 with the experimental Coherent switch (see [Performance](#performance)). + macbox builds both, with patches that took a 3x iOS Simulator from 17 to 117 fps on the GPU it + was measured on (see [Performance](#performance)). - **The screen** opens in a window on your Linux desktop, or in a browser tab (Selkies: H.264 on the GPU, its bitrate following the link). Agents can see it and click and type into it (`macbox screenshot`, `click`, `type`), so an agent can drive the macOS installer too. @@ -90,34 +100,37 @@ floating glass toolbar, counted at the simulator's own screen (frames per second - iOS Simulator at 3x, Liquid Glass benchmark: 17.2 fps as upstream shipped it, 39.0 with the October 28 macbox build, 50.6 with the cohwl build, 70.0 with Coherent on + iOS Simulator at 3x, Liquid Glass benchmark: 17.2 fps as upstream shipped it, 39.0 with the October 28 macbox build, 50.6 on November 5, 70.0 with Coherent, 117 with the build macbox makes now | iOS Simulator at 3x (1206x2622, an iPhone's pixels), glass | fps | |---|---| | Reims 2ce3209, as upstream shipped it on 2026-10-26 | 17.2 | | macbox, 2026-10-28: rendering guest-RAM textures in place | 38.5 / 39.5 | -| **macbox build now** (cohwl, 2026-11-05) | **50.6** (p95 21 ms) | -| macbox build with `MACBOX_COHERENT=on` (experimental) | **70.0** (p95 18 ms) | - -| | Reims 2ce3209 | macbox, 2026-10-28 | macbox now, Coherent on | -|---|---|---|---| -| iOS Simulator 1x (402x874), glass | 53.5 | 95.8 / 92.5 | not measured yet | -| iOS Simulator 2x (804x1748), glass | 33.5 | 67.8 / 60.5 | not measured yet | -| macOS desktop, dragging a window (presents/s) | | 105 (Coherent off) | 130 | -| macOS desktop, Safari scrolling (presents/s) | | 75–79 (Coherent off) | 110 | -| macOS desktop, Genie minimize at 3440x1440 (frames per animation) | ~12 | 19.5 | 29.5 / 30.5 | - -Coherent keeps the GPU's image of the simulator's and the desktop's surfaces ahead of guest RAM, -and syncs a page only when the guest's CPU touches it: the copies that were left are gone, and the -GPU drops from 97% to 70% busy at 3x. It is off by default for now: it hung the GPU once on its -first day of daily use (the next boot came up without it, as designed), and the cause isn't -settled. [docs/performance.md](docs/performance.md#coherent-experimental-off-by-default) has the details. - -The GPU is the limit: without Coherent a 3x simulator keeps this iGPU 96 to 98% busy. On a faster -GPU every number moves up. [docs/performance.md](docs/performance.md) has what the patches do, the method, what -didn't help, and the known limits (an idle macOS display gets no vblank from its driver, so an -animation starts about 0.1 s late: [reims-vgpu#141](https://github.com/steelbrain/reims-vgpu/issues/141)). +| macbox, 2026-11-05: flat32 | 50.6 | +| macbox, 2026-11-05: + Coherent, zero-copy surfaces | 70.0 | +| **macbox now** (2026-11-09): + grouped completion, staged page-ups, inline checks | **117** (114–121; the screen's limit is 120) | + +| The build macbox makes now | | +|---|---| +| two 3x simulators at once | 51–54 fps each | +| three 3x simulators at once | 31–32 fps each | +| macOS desktop (3440x1440), dragging a window | 240 display-link ticks a second alone; 126 with a simulator running | +| copying frames back to guest RAM | 0.2% of what eager copies would cost: about 5 µs a frame | + +Coherent keeps the GPU's image of the simulator's and the desktop's surfaces ahead of guest RAM and +syncs a page only when the guest's CPU touches it, so the copies between them are gone. The rest +removed per-page mappings, per-batch completions and a scheduling round per batch. It's on by +default with an Arrow Lake iGPU, where it runs every day, and a crash guard falls back to the plain +path after a bad run. One rare GPU hang under repeated boots of three simulators is still open. +[docs/performance.md](docs/performance.md) has the details. + +The GPU is the limit: one 3x simulator takes about 7.7 ms of this iGPU's time per frame, and several +simulators share about 105 frames a second. On a faster GPU every number moves up. +[docs/performance.md](docs/performance.md) breaks a frame down (what is real rendering and what the +translation adds), lists what didn't help, and the known limits (an idle macOS display gets no +vblank from its driver, so an animation starts about 0.1 s late: +[reims-vgpu#141](https://github.com/steelbrain/reims-vgpu/issues/141)). ## Questions people ask diff --git a/docs/architecture.md b/docs/architecture.md index d4730ac..4ed355e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -61,7 +61,8 @@ and the QEMU command line; `reims-ctl command mac` (inside the container) prints - Before each start: `WAYLAND_DISPLAY` (sway's, or the host's), and the GPU tuning: `M2V_LIVE=$MACBOX_HOME/run/live.conf` with `host_rt_in_place=1`, `REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1`, `M2V_ARL_FACTS` on an Arrow Lake GPU, and with - `MACBOX_COHERENT=on` the census and Coherent switches ([performance.md](performance.md)). + Coherent (`MACBOX_COHERENT`, on by default with an Arrow Lake GPU) the census, Coherent and the + switches measured with it, in `live.conf` and Reims's environment ([performance.md](performance.md)). The disk's options in `extra_args` are written again, so a changed `MACBOX_DISK_CACHE` applies. The system disk starts on SATA, because macOS's installer may not find a disk on NVMe, and moves to diff --git a/docs/images/fps-dark.svg b/docs/images/fps-dark.svg index db56b45..69d4660 100644 --- a/docs/images/fps-dark.svg +++ b/docs/images/fps-dark.svg @@ -1,34 +1,44 @@ - - + + iOS Simulator at 3x (1206×2622), Liquid Glass benchmark Frames per second on an Intel Arrow Lake iGPU (Core Ultra 7 265). Higher is better. - -0 - -20 - -40 - -60 - -80 -frames per second - -60 fps -Upstream -Reims 2ce3209, 2026-10-26 - -17.2 -macbox, Oct 28 -in place rendering - -39.0 -macbox now -cohwl, the default - -50.6 -+ Coherent -experimental, off by default - -70.0 + +0 + +20 + +40 + +60 + +80 + +100 + +120 +frames per second + +60 fps + +120 Hz screen +Upstream +Reims 2ce3209, 2026-10-26 + +17.2 +macbox, Oct 28 +in place rendering + +39.0 +macbox, Nov 5 +cohwl: flat32 + +50.6 ++ Coherent +zero-copy surfaces, Nov 5 + +70.0 +macbox now +Coherent + grouped completion + +117.0 diff --git a/docs/images/fps-light.svg b/docs/images/fps-light.svg index 6eeefda..debc91c 100644 --- a/docs/images/fps-light.svg +++ b/docs/images/fps-light.svg @@ -1,34 +1,44 @@ - - + + iOS Simulator at 3x (1206×2622), Liquid Glass benchmark Frames per second on an Intel Arrow Lake iGPU (Core Ultra 7 265). Higher is better. - -0 - -20 - -40 - -60 - -80 -frames per second - -60 fps -Upstream -Reims 2ce3209, 2026-10-26 - -17.2 -macbox, Oct 28 -in place rendering - -39.0 -macbox now -cohwl, the default - -50.6 -+ Coherent -experimental, off by default - -70.0 + +0 + +20 + +40 + +60 + +80 + +100 + +120 +frames per second + +60 fps + +120 Hz screen +Upstream +Reims 2ce3209, 2026-10-26 + +17.2 +macbox, Oct 28 +in place rendering + +39.0 +macbox, Nov 5 +cohwl: flat32 + +50.6 ++ Coherent +zero-copy surfaces, Nov 5 + +70.0 +macbox now +Coherent + grouped completion + +117.0 diff --git a/docs/images/make-chart.py b/docs/images/make-chart.py index 51ea13a..12ab1e2 100644 --- a/docs/images/make-chart.py +++ b/docs/images/make-chart.py @@ -1,21 +1,22 @@ #!/usr/bin/env python3 """Writes fps-light.svg and fps-dark.svg, the README's frame-rate chart (docs/performance.md's numbers). -Colors: the validated categorical slots 1 and 2 (blue, orange) for each mode's surface; the -experimental bar is the macbox blue at reduced opacity, and its label says so.""" +Colors: the validated categorical slots 1 and 2 (blue, orange) for each mode's surface: orange for +upstream, blue for macbox's builds, the build macbox makes now at full strength and the earlier ones lighter.""" import os ROWS = [ # label, sublabel, fps (3x, mean of the valid rounds), kind ("Upstream", "Reims 2ce3209, 2026-10-26", 17.2, "before"), ("macbox, Oct 28", "in place rendering", (38.5 + 39.5) / 2, "after"), - ("macbox now", "cohwl, the default", 50.6, "after"), - ("+ Coherent", "experimental, off by default", 70.0, "exp"), + ("macbox, Nov 5", "cohwl: flat32", 50.6, "after"), + ("+ Coherent", "zero-copy surfaces, Nov 5", 70.0, "after"), + ("macbox now", "Coherent + grouped completion", 117.0, "now"), ] MODES = { "light": dict(surface="#fcfcfb", text="#0b0b0b", text2="#52514e", grid="#e4e3df", before="#eb6834", after="#2a78d6"), "dark": dict(surface="#1a1a19", text="#f0efec", text2="#c3c2b7", grid="#3a3a37", before="#d95926", after="#3987e5"), } -W, H = 900, 320 -X0, X1, MAX = 230, 840, 80 # bar area and its scale (fps) +W, H = 900, 380 +X0, X1, MAX = 250, 830, 130 # bar area and its scale (fps) BAR, GAP = 30, 16 # bar thickness, space between bars TOP = 96 FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif" @@ -34,19 +35,20 @@ def svg(m): f'iOS Simulator at 3x (1206×2622), Liquid Glass benchmark', f'Frames per second on an Intel Arrow Lake iGPU (Core Ultra 7 265). Higher is better.'] bottom = TOP + len(ROWS) * BAR + (len(ROWS) - 1) * GAP - for v in (0, 20, 40, 60, 80): + for v in (0, 20, 40, 60, 80, 100, 120): o.append(f'') o.append(f'{v}') o.append(f'frames per second') - # the 60 fps target - o.append(f'') - o.append(f'60 fps') + # 60 fps, and the iPhone 17 Pro's 120 Hz screen: the most its simulator can show + for v, t in ((60, "60 fps"), (120, "120 Hz screen")): + o.append(f'') + o.append(f'{t}') y = TOP for label, sub, v, kind in ROWS: col = c["before"] if kind == "before" else c["after"] o.append(f'{label}') o.append(f'{sub}') - o.append(bar(y, v, col, 0.55 if kind == "exp" else 1.0)) + o.append(bar(y, v, col, 1.0 if kind in ("before", "now") else 0.55)) o.append(f'{v:.1f}') y += BAR + GAP o.append("") diff --git a/docs/performance.md b/docs/performance.md index 73b902c..866b0c3 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -2,21 +2,35 @@ ## Where the time goes -On every GPU measured so far, the GPU is the limit. A 3x iOS Simulator keeps an Arrow Lake iGPU -96 to 98% busy at its full clock, while QEMU's CPU threads have room to spare. So the frame rate -follows two things: how many pixels are drawn (1x, 2x, 3x; the desktop's resolution), and how much -work the translation adds per frame (Metal → Vulkan). +On every GPU measured so far, the GPU is the limit. With the build macbox makes now, one 3x iOS +Simulator reaches its screen's 120 Hz on an Arrow Lake iGPU at about 7.7 ms of GPU time per frame, +and with two or three simulators the GPU runs 80 to 93% busy at its top clock (1950 MHz) while +QEMU's CPU threads have room to spare. So the frame rate follows two things: how many pixels are +drawn (1x, 2x, 3x; the desktop's resolution), and how much work the translation adds per frame +(Metal → Vulkan). Before macbox's patches, most of that added work was copies: macOS's framebuffer and the simulator's display surface lie in guest RAM, and metal2vulkan copied each into a GPU image and back around every render pass. Those shadow copies were 51% of all GPU time with the simulator and 67% with the desktop. Patch 0004 renders those textures in place, and Coherent (patches 0009 onwards, with QEMU's census) removes most of the copies left ([patches/README.md](../patches/README.md)). +What is left of a 3x simulator frame, measured with the pass timer on 2026-11-09: + +| GPU time per 3x frame (~7.7–8.2 ms) | ms | | +|---|---|---| +| the simulator's own render passes on its screen | ~3.2 | the real rendering | +| iOS's own full-screen compute pass (a masked copy of the screen) | ~1.5 | the real rendering, near the memory bandwidth | +| copies for draws that read the pass they draw into (Liquid Glass backdrops) | ~0.76 | translation: Vulkan can't read an attachment where Metal can | +| between command buffers: barriers, pipeline drains, ~200 command buffers | ~2.4 | translation, and the open work | + +Copying a frame back to guest RAM, the cost Coherent was built to remove, is down to bookkeeping: in +a 90-minute census of the desktop at 3440x1440, 0.2% of what eager copies back would have cost +(about 5 µs a frame), and 2% for the simulators' screens. ## Measurements The reference host: Intel Core Ultra 7 265 (Arrow Lake iGPU), 128 GB DDR5 at 4400 MT/s, Mesa 26.2.4, -a NVMe disk; the VM had 8 vCPUs and 12 GB, macOS 26.7.1, iOS 26.5, one simulator booted at a time. +a NVMe disk; the VM had 8 vCPUs and 12 GB, macOS 26.7.1, iOS 26.5. ### iOS Simulator at 3x, step by step @@ -27,15 +41,39 @@ Pro at 3x (1206x2622), frames per second counted at the simulator's own screen: |---|---|---| | Reims 2ce3209 as upstream shipped it (2026-10-26) | 17.2 | | | macbox, 2026-10-28 (in place rendering, 0004) | 38.5 / 39.5 | 25.0 ms/frame | -| macbox cohwl, Coherent off (flat32) | **50.6** (p95 21.4 ms, p99 25.8 ms) | 97% busy | -| macbox cohwl, `MACBOX_COHERENT=on` | **70.0** (p95 17.7 ms, p99 22.0 ms); 63–66 in a later run | 70% busy | +| macbox cohwl, Coherent off (flat32), 2026-11-05 | 50.6 (p95 21.4 ms, p99 25.8 ms) | 97% busy | +| macbox cohwl, Coherent on, 2026-11-05 | 70.0 (p95 17.7 ms, p99 22.0 ms) | 70% busy | +| **macbox now** (Coherent and the switches below, 2026-11-09) | **117** (114–121 over runs) | 7.7 ms/frame | + +The iPhone 17 Pro's screen runs at up to 120 Hz, so 117 is close to the most a single simulator can +show; GlassBench's own content animates at 60 Hz, and the simulator redraws on top of it. The +cohwl rows are one A/B in one window; the last row comes from the soak and A/B runs of 2026-11-09 +and -10 (fresh headless boots, Simulator.app closed). flat32 also has an A/B of its own against the +build before it: 41.6 → 48–50 fps. + +### Several simulators at once, and the desktop under load + +The build macbox makes now, 3x iPhone 17 Pro simulators all running GlassBench, the macOS desktop at +3440x1440 (2026-11-09, soak with simulators booted and shut down in cycles, 0 GPU hangs): -The last two rows are one A/B in one window on 2026-11-05: fresh headless boots, a quiet gate -before each round (guest background CPU and `mediaanalysisd` low), means of the valid rounds. The -first two are earlier days, on the same host but not in the same window. With Coherent, QEMU's CPU -use rises by about 1.6 cores (333% to 491%): the GPU waits less, the CPU traps more. +| Load | Each simulator, fps | GPU | +|---|---|---| +| 1 simulator | 114–121 | 7.7 ms/frame | +| 1 simulator while a window is dragged on the desktop | 99–102 | | +| 2 simulators | 51–54 each (~105 frames/s in total) | 80–85% busy, ~8 ms/frame | +| 3 simulators | 31–32 each (~95 in total) | 93% busy | + +| Desktop: dragging a window (display-link ticks per second, p99 interval) | | +|---|---| +| alone | 240/s, p99 10 ms | +| with 1 simulator running | 126–128/s, p99 22–26 ms | +| with 2 simulators running | 96–102/s, p99 32–38 ms | -flat32 also has an A/B of its own against the build before it: 41.6 → 48–50 fps at 3x. +Two or more simulators share the GPU: about 105 simulator frames a second in total on this iGPU, +whatever their number. All guest work reaches the GPU in one order, so the desktop's frames wait +behind the simulators' earlier work; that, not the desktop's own cost, is its latency under load. +These runs placed the VM's vCPUs on the CPU's performance cores and QEMU's other threads on all +cores (`MACBOX_CPUSET` pins all of QEMU instead). ### Earlier measurements, the 2026-10-28 build @@ -46,17 +84,17 @@ flat32 also has an A/B of its own against the build before it: 41.6 → 48–50 | 3x (1206x2622) | 17.2 | 19.2 | 21.9 | 38.5 / 39.5 (25.0 ms) | | Safari at 3x, blurred floating bar | n/a | 39.5 | 44.5 | 66.4 / 64.2 | -1x and 2x haven't been measured again on cohwl. +1x and 2x haven't been measured again since. -### The macOS desktop +### The macOS desktop, 2026-11-05 -| 3440x1440 desktop | Coherent off | Coherent on | +| 3440x1440 desktop (cohwl) | Coherent off | Coherent on | |---|---|---| | dragging a 600x400 window (presents per second) | 105–107 | 129–130 | | Safari scrolling a local page (presents per second) | 75–79 | 110 | | Genie minimize of a 2400x1100 window (frames per animation; ~12 upstream) | 19.5 (the 10-28 build) | 29.5 / 30.5 | -The first two rows are the same A/B as above. The Genie number with Coherent comes from a smoke +The first two rows are the cohwl A/B above. The Genie number with Coherent comes from a smoke check, not a matched A/B. A 0.5 s animation at 60 Hz would be 30 frames. ### Disk @@ -99,6 +137,7 @@ With the macbox build (`MACBOX_GPU_TUNING=auto`, the default), `macbox up` sets: | `M2V_LIVE=$MACBOX_HOME/run/live.conf` | metal2vulkan's live switches (patch 0003). The file holds `host_rt_in_place=1` | | `M2V_ARL_FACTS=<11 facts>` | only on an Arrow Lake GPU (PCI ids 7d67, 7dd1, 7d41, 7d51, 7d45) | | `REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1` | skips a new Private texture's first page-up (Reims patch 0013) | +| Coherent and the switches below | with `MACBOX_COHERENT=auto` (the default) on an Arrow Lake GPU | `host_rt_in_place` is read when a texture is made: change it in `live.conf`, then remake the surfaces (change the resolution, or restart the VM). `MACBOX_GPU_TUNING=off` turns all of it off. @@ -107,25 +146,42 @@ On another GPU, in place rendering still runs its device check per memory kind a the check fails; it has only been measured on Arrow Lake. Measure your own GPU before and after: the facts list is specific to Arrow Lake and is never set elsewhere. -### Coherent (experimental, off by default) +### Coherent and the switches measured with it (on by default on Arrow Lake) -`MACBOX_COHERENT=on` in the config, then `macbox down && macbox up`. macbox then also sets -`REIMS_VGPU_CENSUS=$MACBOX_HOME/run/coherent`, `REIMS_VGPU_COHERENT=1` and a report a minute, and -adds `surface_lazy=1`, `surface_lazy_scoped=1` and `surface_coherent=1` to `live.conf`. -`MACBOX_COHERENT=off` removes them all again. +`MACBOX_COHERENT=auto` (the default) turns this on with an Arrow Lake iGPU, the GPU it runs on day +to day; `on` turns it on with any GPU, `off` turns it off. A change applies at the next +`macbox down && macbox up`. With it, macbox sets: -It is off by default because it isn't settled yet: -- **One GPU hang with it on.** On 2026-11-05, the day it went into daily use, i915 - reset the GPU's render engine and QEMU stopped. The guard did its job: the next boot came up without - coherent pages. A later hang with Coherent off, on a newer development build, has the same error - code, so the cause is open, not pinned on Coherent. -- **A write-loss risk the counters flagged once,** during a test that killed Safari mid-frame - (`overwrite_risk` 1 in the census report), and that hasn't been reproduced or ruled out since. - -If a coherent run crashes, `$MACBOX_HOME/run/coherent/coherent.running` stays behind and every later -start boots without coherent pages. `macbox coherent status` says so; `macbox coherent reset` -(VM stopped) allows the next try. It needs a kernel with `UFFD_USER_MODE_ONLY` (Linux 5.11 or -later), which works with `vm.unprivileged_userfaultfd=0`. +| | | +|---|---| +| `REIMS_VGPU_CENSUS=$MACBOX_HOME/run/coherent`, `REIMS_VGPU_COHERENT=1` | QEMU's census traps the guest CPU's touches of the surfaces metal2vulkan keeps on the GPU (a report a minute) | +| `surface_lazy`, `surface_lazy_scoped`, `surface_coherent` | the simulator's and the desktop's surfaces stay on the GPU; a page syncs only when the guest's CPU touches it | +| `surface_lazy_max_texels=6000000`, `surface_twin_reader_first`, `surface_coherent_own_back`, `surface_lazy_open_ids`, `surface_twin_make_once` | fixes the lanes and the census found on top of it: which surfaces qualify, a race between two textures over one surface (`make_once`: 91 → 100 fps over 12 boots), and exact answers about committed command buffers | +| `inline_lazy`, `inline_checks` | a lazy surface's first pass instance is recorded inline; a batch's submission checks run in the scheduler's pass, so no chained batch waits a round for them | +| `REIMS_VGPU_GROUPED=1`, `group_batches=64`, `group_deadline_us=500`, `group_keep=0`, `group_events=0` | grouped completion: up to 64 batches complete together; keep and events off, which ended a class of fence timeouts under several simulators | +| `REIMS_VGPU_STAGED_PAGE_UP`, `REIMS_VGPU_STAGED_ZERO` | a new texture's first page-up copies guest pages into a staging buffer instead of mapping a view per page run (55k mappings a second and a storm of TLB shootdowns before: simulator 89 → 101 fps), and pages of zeros write nothing | +| `REIMS_VGPU_PIPELINE_EXEC`, `REIMS_VGPU_IRQ_MODERATE` | a FIFO's execs pipelined (8 in flight), and one interrupt outstanding at a time | +| `facts_on=SubmissionsWithoutSignalsMerge` | Arrow Lake only: submissions without signals merge | + +The four `REIMS_VGPU_*` switches that name a file are on while the file exists +(`$MACBOX_HOME/run/switches/`), so each can be turned off without a restart. `live.conf` lines are +re-read every second; macbox rewrites its own lines at every start and leaves yours alone. + +What to know before relying on it: +- **The crash guard.** If a coherent run crashes, `$MACBOX_HOME/run/coherent/coherent.running` + stays behind and every later start boots without coherent pages. `macbox coherent status` says + so; `macbox coherent reset` (VM stopped) allows the next try. A fault costs one crash, not a crash + loop. +- **It needs** a kernel with `UFFD_USER_MODE_ONLY` (Linux 5.11 or later), which works with + `vm.unprivileged_userfaultfd=0`. In the cohwl A/B QEMU used about 1.6 more CPU cores with it: + the GPU waits less, the CPU traps more. +- **A rare GPU hang is still open.** Under repeated boots and shutdowns of three simulators, a + command buffer is occasionally begun again while the GPU still runs it, and i915 resets the GPU. + It is traced to command-buffer reuse in metal2vulkan's submission path, not to Coherent's pages, + and is being hunted with an exact detector in Mesa. +- **Write-loss check.** The census counts `overwrite_risk`, a guest write a held GPU copy could + overwrite. It was flagged once in an early test that killed Safari mid-frame; across 64 + production boots since 2026-11-08 (16,850 reports) it stayed at 0. ## Tried, no gain @@ -147,6 +203,13 @@ later), which works with `vm.unprivileged_userfaultfd=0`. | a reaper for deleted textures | Safari 5–8 fps slower, and Genie breaks | | executing the FIFO from its own thread (fifoexec) | a GPU hang on its first day; rolled back | | merging adjacent page runs of a mapping | negligible: 49 merges in 629,632 pages of a real trace | +| chaining a submission's primary command buffers in ANV (one preamble, one end) | 13 → 2 µs per command buffer in a micro benchmark, 0–1% on a real frame: the barrier before already drains the GPU | +| precise barriers (no L3 flush between images) | within boot-to-boot noise: real frames leave little dirty | +| lossless compression (CCS) on the simulator's surfaces, in three forms | no GPU time saved; a lone simulator lost a third of its frames to host copies | +| several Vulkan queues (`ANV_QUEUE_OVERRIDE=gc=18`) | about 0% (+4% with three simulators): the GPU is already busy; one i915 context per queue (`gc=8`) was unstable | +| keeping groups of batches open (`group_keep=1`) | +6–7% with two simulators, but held groups stalled frames up to 200 ms | +| lowering a simulator's draws on a second thread | 90 → 72 fps: the first thread waits for it | +| a deeper FIFO pipeline (32 execs instead of 8) | slower: 109 → 97 fps, desktop p99 27 → 37 ms | ## Known limits @@ -154,7 +217,8 @@ later), which works with `vm.unprivileged_userfaultfd=0`. sends no vblank while nothing is presenting, so the first frame of an animation waits a median 114 ms for display timing (8 ms when something already animates). [steelbrain/reims-vgpu#141](https://github.com/steelbrain/reims-vgpu/issues/141). -- **3x is GPU-bound** at about 50 fps on this iGPU in heavy Liquid Glass scenes (70 with Coherent). +- **The GPU is shared.** One 3x simulator reaches 120 Hz; two or three share about 105 frames a + second on this iGPU, and the desktop's frames wait behind theirs. - **x86-64 macOS ends with macOS 26.** iOS 27 simulators are arm64 only. - **Recording** uses `simctl io recordVideo`, H.264 in software in the VM: about 50 fps with one simulator animating, 18 each with three. `sim-fps` counts what the simulator wrote, and it writes diff --git a/docs/setup.md b/docs/setup.md index 5949bf0..df86c43 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -38,7 +38,7 @@ node), makes an SSH key `~/.ssh/macbox_ed25519`, and adds a `Host macbox` block | `MACBOX_CPUSET` | empty | pin QEMU to some cores, e.g. a hybrid CPU's performance cores (`0-7`) | | `MACBOX_BUILD_CPUSET` | empty | cores for `macbox build`, e.g. the efficiency cores | | `MACBOX_DISK_CACHE` | none | `writeback` panicked macOS's NVMe driver under heavy swap; `unsafe` is faster, but a crash of the host itself can corrupt the VM's disk. Applies at the next `macbox up` | -| `MACBOX_COHERENT` | off | `on`: Coherent, zero-copy surfaces, about 40% faster at 3x and experimental ([performance.md](performance.md#coherent-experimental-off-by-default)) | +| `MACBOX_COHERENT` | auto | `auto`: on with an Arrow Lake iGPU. Coherent, zero-copy surfaces, and the switches measured with it: a 3x simulator 50 → 117 fps there ([performance.md](performance.md#coherent-and-the-switches-measured-with-it-on-by-default-on-arrow-lake)); `on` with any GPU, `off` | | `MACBOX_HOME` | ~/.local/share/macbox | everything macbox keeps | Then: diff --git a/llms.txt b/llms.txt index e80aabe..b73681e 100644 --- a/llms.txt +++ b/llms.txt @@ -1,6 +1,6 @@ # macbox -> Run macOS, Xcode and the real iOS Simulator on a Linux host, in Docker, with GPU acceleration (Metal through Reims vGPU and metal2vulkan on the host's Vulkan GPU), for AI coding agents such as Claude Code, Codex and T3 Code. No Mac and no cloud service: one CLI, `macbox`, builds and runs it, and `mac-sim` gives each agent checkout its own iOS Simulator to build, run, tap, screenshot, record and measure at 60 fps. +> Run macOS, Xcode and the real iOS Simulator on a Linux host, in Docker, with GPU acceleration (Metal through Reims vGPU and metal2vulkan on the host's Vulkan GPU), for AI coding agents such as Claude Code, Codex and T3 Code. An iPhone 17 Pro simulator at 3x runs a Liquid Glass benchmark at 117 fps on an Intel Arrow Lake iGPU (17 before macbox's patches). No Mac and no cloud service: one CLI, `macbox`, builds and runs it, and `mac-sim` gives each agent checkout its own iOS Simulator to build, run, tap, screenshot, record and measure at 60 fps. License: PolyForm Noncommercial 1.0.0 for macbox's own code (noncommercial use; commercial use needs permission); the patches stay GPL-2.0-or-later OR LGPL-3.0-or-later (GPL-2.0-or-later for the QEMU ones). @@ -33,7 +33,7 @@ publishing on their behalf. ## How it works - [Architecture](https://github.com/TUNA-NOPE/macbox/blob/main/docs/architecture.md): the container, QEMU and Reims vGPU, the display stream, the network, the guest, the simulator pool -- [Performance](https://github.com/TUNA-NOPE/macbox/blob/main/docs/performance.md): the patches (in place rendering, flat32, experimental Coherent), measurements, tuning, known limits +- [Performance](https://github.com/TUNA-NOPE/macbox/blob/main/docs/performance.md): the patches (in place rendering, flat32, Coherent zero-copy surfaces, grouped completion, staged page-ups), measurements for one to three simulators and the desktop, a frame broken down, tuning, known limits - [Patches](https://github.com/TUNA-NOPE/macbox/blob/main/patches/README.md) - [Troubleshooting](https://github.com/TUNA-NOPE/macbox/blob/main/docs/troubleshooting.md) - [Credits](https://github.com/TUNA-NOPE/macbox/blob/main/CREDITS.md) diff --git a/macbox b/macbox index 1f5bfc0..4b4c056 100755 --- a/macbox +++ b/macbox @@ -15,7 +15,7 @@ eval "$_env" : "${MACBOX_GUEST_USER:=dev}" "${MACBOX_CPUS:=8}" "${MACBOX_MEMORY:=12G}" "${MACBOX_DISK_SIZE:=160G}" : "${MACBOX_RESOLUTION:=1920x1080}" "${MACBOX_REFRESH:=60}" "${MACBOX_CPUSET:=}" : "${MACBOX_RENDER_NODE:=/dev/dri/renderD128}" "${MACBOX_DISK_CACHE:=none}" "${MACBOX_GPU_TUNING:=auto}" -: "${MACBOX_COHERENT:=off}" +: "${MACBOX_COHERENT:=auto}" : "${MACBOX_SIM_PREFIX:=macbox}" "${MACBOX_POOL:=full slim-1 slim-2 sharp small}" : "${MACBOX_T3_HOST_ID:=$MACBOX_SSH_HOST}" "${MACBOX_T3_SETTINGS:=$HOME/.t3/userdata/settings.json}" DOCKER="${MACBOX_DOCKER:-docker}" @@ -95,28 +95,65 @@ tuning_on() { *) [[ "$(build_id)" == *-macbox ]] ;; esac } +# Coherent with the switches measured on top of it, the build that runs day to day on the Arrow +# Lake iGPU ("production I", docs/performance.md): metal2vulkan's live lines, then Reims's switches +# (each of the file-named ones is on while its file exists, so it can be flipped without a restart). +COHERENT_LIVE="surface_lazy surface_lazy_scoped surface_lazy_max_texels surface_coherent surface_twin_reader_first surface_coherent_own_back surface_lazy_open_ids surface_twin_make_once inline_lazy inline_checks group_batches group_deadline_us group_keep group_events facts_on" +COHERENT_ENV="REIMS_VGPU_CENSUS REIMS_VGPU_COHERENT REIMS_VGPU_CENSUS_FLUSH_MS REIMS_VGPU_GROUPED REIMS_VGPU_STAGED_PAGE_UP REIMS_VGPU_STAGED_ZERO REIMS_VGPU_PIPELINE_EXEC REIMS_VGPU_IRQ_MODERATE" +coherent_on() { # auto: on with the Arrow Lake iGPU it was measured on + case "$MACBOX_COHERENT" in + on) return 0 ;; off) return 1 ;; + *) gpu_is_arl ;; + esac +} +drop_managed() { # drop_managed : the lines macbox writes, so a change of mode applies + [ -f "$1" ] || return 0 + local k + for k in $COHERENT_LIVE; do sed -i "/^$k=/d" "$1"; done +} apply_tuning() { # before each start - local live="$RUN/live.conf" + local live="$RUN/live.conf" k if tuning_on; then [ -s "$live" ] || printf '# metal2vulkan live switches (patches/metal2vulkan), re-read every second\nhost_rt_in_place=1\n' >"$live" - sed -i '/^surface_\(lazy\|lazy_scoped\|coherent\)=/d' "$live" + drop_managed "$live" local env=(--env "M2V_LIVE=$live" --env REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1) if gpu_is_arl; then env+=(--env "M2V_ARL_FACTS=$ARL_FACTS"); else env+=(--env "M2V_ARL_FACTS="); fi - if [ "$MACBOX_COHERENT" = on ]; then + if coherent_on; then # QEMU's census traps the guest CPU's touches of the surfaces metal2vulkan renders in place. # A run that doesn't end well leaves coherent/coherent.running, and every later start then # boots without Coherent until that file is removed (macbox coherent reset). - mkdir -p "$RUN/coherent" - printf 'surface_lazy=1\nsurface_lazy_scoped=1\nsurface_coherent=1\n' >>"$live" - env+=(--env "REIMS_VGPU_CENSUS=$RUN/coherent" --env REIMS_VGPU_COHERENT=1 --env REIMS_VGPU_CENSUS_FLUSH_MS=60000) + mkdir -p "$RUN/coherent" "$RUN/switches" + cat >>"$live" <<'LIVE' +surface_lazy=1 +surface_lazy_scoped=1 +surface_lazy_max_texels=6000000 +surface_coherent=1 +surface_twin_reader_first=1 +surface_coherent_own_back=1 +surface_lazy_open_ids=1 +surface_twin_make_once=1 +inline_lazy=1 +inline_checks=1 +group_batches=64 +group_deadline_us=500 +group_keep=0 +group_events=0 +LIVE + gpu_is_arl && echo 'facts_on=SubmissionsWithoutSignalsMerge' >>"$live" + for k in staged-page-up staged-zero pipeline-exec irq-moderate; do : >"$RUN/switches/$k"; done + env+=(--env "REIMS_VGPU_CENSUS=$RUN/coherent" --env REIMS_VGPU_COHERENT=1 --env REIMS_VGPU_CENSUS_FLUSH_MS=60000 + --env REIMS_VGPU_GROUPED=1 --env "REIMS_VGPU_STAGED_PAGE_UP=$RUN/switches/staged-page-up" + --env "REIMS_VGPU_STAGED_ZERO=$RUN/switches/staged-zero" --env "REIMS_VGPU_PIPELINE_EXEC=$RUN/switches/pipeline-exec" + --env "REIMS_VGPU_IRQ_MODERATE=$RUN/switches/irq-moderate") else - env+=(--env REIMS_VGPU_CENSUS= --env REIMS_VGPU_COHERENT= --env REIMS_VGPU_CENSUS_FLUSH_MS=) + for k in $COHERENT_ENV; do env+=(--env "$k="); done fi ctl set "$MACBOX_VM" "${env[@]}" >/dev/null else - [ -f "$live" ] && sed -i '/^surface_\(lazy\|lazy_scoped\|coherent\)=/d' "$live" - ctl set "$MACBOX_VM" --env "M2V_LIVE=" --env "M2V_ARL_FACTS=" --env REIMS_VGPU_PRIVATE_SKIP_PAGE_UP= \ - --env REIMS_VGPU_CENSUS= --env REIMS_VGPU_COHERENT= --env REIMS_VGPU_CENSUS_FLUSH_MS= >/dev/null + drop_managed "$live" + local env=(--env "M2V_LIVE=" --env "M2V_ARL_FACTS=" --env REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=) + for k in $COHERENT_ENV; do env+=(--env "$k="); done + ctl set "$MACBOX_VM" "${env[@]}" >/dev/null fi } @@ -169,7 +206,8 @@ MACBOX_RENDER_NODE=${node:-$MACBOX_RENDER_NODE} MACBOX_DISK_CACHE=$MACBOX_DISK_CACHE # auto: macbox's metal2vulkan tuning with the macbox build; on; off MACBOX_GPU_TUNING=$MACBOX_GPU_TUNING -# on: Coherent, zero-copy IOSurfaces (docs/performance.md). Experimental: one GPU hang with it on +# auto (on with an Arrow Lake iGPU, where it runs day to day), on, off: Coherent, zero-copy +# IOSurfaces, and the switches measured with it (docs/performance.md) MACBOX_COHERENT=$MACBOX_COHERENT # iOS Simulator pool: names are " ( (), command_buffer $(, $a)*); + return logged; + } ++ // LOCAL (Atlas) diagnostic: the pending audit's hook (`pending_audit.rs`). ++ crate::pending_audit::hook!($f, command_buffer $(, $a)*); + let real = REAL.$f[S].load(Ordering::Acquire); + if real.is_null() { + missing($n); +diff --git a/crates/metal2vulkan-runtime/src/spares.rs b/crates/metal2vulkan-runtime/src/spares.rs +index 5a4ad4e2..cd4a108d 100644 +--- a/crates/metal2vulkan-runtime/src/spares.rs ++++ b/crates/metal2vulkan-runtime/src/spares.rs +@@ -313,6 +313,12 @@ impl Spares { + return Ok(None); + }; + if pool.recording { ++ // LOCAL (Atlas) diagnostic (`pending_audit.rs`). ++ if crate::pending_audit::on() { ++ let all: Vec = ++ pool.made.iter().flatten().copied().chain(pool.begun).collect(); ++ crate::pending_audit::reset_pool(&all, "spare pool reset"); ++ } + // SAFETY: the pool is the caller's alone now, and nothing of it is pending: the + // command buffer that gave it back ended its work or was never submitted. + let reset = unsafe { diff --git a/patches/metal2vulkan/0072-pending-audit-reads-each-semaphore-through-the-devic.patch b/patches/metal2vulkan/0072-pending-audit-reads-each-semaphore-through-the-devic.patch new file mode 100644 index 0000000..185c770 --- /dev/null +++ b/patches/metal2vulkan/0072-pending-audit-reads-each-semaphore-through-the-devic.patch @@ -0,0 +1,76 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 16:56:45 +0000 +Subject: [PATCH 72/92] pending audit reads each semaphore through the device + that submitted it + +--- + .../metal2vulkan-runtime/src/pending_audit.rs | 24 +++++++++++++++---- + 1 file changed, 19 insertions(+), 5 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/pending_audit.rs b/crates/metal2vulkan-runtime/src/pending_audit.rs +index 6bc05d27..cb98b9ef 100644 +--- a/crates/metal2vulkan-runtime/src/pending_audit.rs ++++ b/crates/metal2vulkan-runtime/src/pending_audit.rs +@@ -15,7 +15,7 @@ + use ash::vk; + use std::collections::HashMap; + use std::sync::atomic::{AtomicU64, Ordering}; +-use std::sync::{LazyLock, Mutex, OnceLock}; ++use std::sync::{LazyLock, Mutex}; + use std::time::Instant; + + /// The most lines written. +@@ -32,11 +32,14 @@ pub(crate) fn on() -> bool { + *ON + } + +-/// The device whose semaphores and fences are read (the first that submits). +-static DEVICE: OnceLock = OnceLock::new(); ++/// The devices that submitted, each read through its own (a semaphore read through another ++/// device reads 0). ++static DEVICES: LazyLock>> = LazyLock::new(|| Mutex::new(Vec::new())); + + /// A primary's last submission. + struct Sub { ++ /// The index of its device in [`DEVICES`]. ++ device: usize, + site: &'static str, + at: Instant, + /// The timeline signals of its info or the first later info of the same call with any. +@@ -70,7 +73,8 @@ fn line(text: String) { + + /// Why `sub` is still pending, or `None` where it completed (or nothing tells). + fn pending(sub: &Sub) -> Option { +- let device = DEVICE.get()?; ++ let devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ let device = devices.get(sub.device)?; + for &(semaphore, value) in &sub.signals { + // SAFETY: a semaphore of the device a submission signalled; read only. + if let Ok(now) = unsafe { device.get_semaphore_counter_value(semaphore) } { +@@ -214,7 +218,16 @@ pub(crate) fn submits( + if !*ON { + return; + } +- let _ = DEVICE.set(device.clone()); ++ let index = { ++ let mut devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ match devices.iter().position(|d| d.handle() == device.handle()) { ++ Some(at) => at, ++ None => { ++ devices.push(device.clone()); ++ devices.len() - 1 ++ } ++ } ++ }; + // Each info's timeline signals, nonzero. + let signals: Vec> = infos + .iter() +@@ -283,6 +296,7 @@ pub(crate) fn submits( + state.subs.insert( + p, + Sub { ++ device: index, + site, + at, + signals: mine.clone(), diff --git a/patches/metal2vulkan/0073-pending-audit-reads-no-fence-and-only-long-lived-tim.patch b/patches/metal2vulkan/0073-pending-audit-reads-no-fence-and-only-long-lived-tim.patch new file mode 100644 index 0000000..a6e4bea --- /dev/null +++ b/patches/metal2vulkan/0073-pending-audit-reads-no-fence-and-only-long-lived-tim.patch @@ -0,0 +1,72 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 17:10:48 +0000 +Subject: [PATCH 73/92] pending audit reads no fence and only long-lived + timelines (a destroyed fence's read crashed ANV) + +--- + .../metal2vulkan-runtime/src/pending_audit.rs | 30 +++++++++++++++---- + 1 file changed, 24 insertions(+), 6 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/pending_audit.rs b/crates/metal2vulkan-runtime/src/pending_audit.rs +index cb98b9ef..8f0549fc 100644 +--- a/crates/metal2vulkan-runtime/src/pending_audit.rs ++++ b/crates/metal2vulkan-runtime/src/pending_audit.rs +@@ -18,6 +18,9 @@ use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::{LazyLock, Mutex}; + use std::time::Instant; + ++/// Submissions a semaphore must have signalled before it is read. ++const LONG_LIVED: u64 = 64; ++ + /// The most lines written. + const MOST_LINES: u64 = 300; + +@@ -49,6 +52,9 @@ struct Sub { + + #[derive(Default)] + struct State { ++ /// How many submissions signalled each semaphore: only one seen in [`LONG_LIVED`] is read ++ /// (a queue's timeline, which lives with its device). ++ seen: HashMap, + /// Each command buffer's begins so far. + generation: HashMap, + /// A primary's executed secondaries since its last begin, with each one's generation then. +@@ -86,12 +92,9 @@ fn pending(sub: &Sub) -> Option { + } + } + } +- if sub.signals.is_empty() && sub.fence != vk::Fence::null() { +- // SAFETY: as above (a fence the runtime may have reset or reused since: a hint). +- if let Ok(false) = unsafe { device.get_fence_status(sub.fence) } { +- return Some(format!("fence {:#x} unsignalled", vk::Handle::as_raw(sub.fence))); +- } +- } ++ // No fence is read: the runtime destroys its fences, and a read of a destroyed one ++ // crashed the driver (PAUD 2026-11-08). ++ let _ = sub.fence; + None + } + +@@ -255,6 +258,21 @@ pub(crate) fn submits( + .collect(); + let mut state = STATE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let at = Instant::now(); ++ for list in &signals { ++ for &(semaphore, _) in list { ++ *state.seen.entry(vk::Handle::as_raw(semaphore)).or_insert(0) += 1; ++ } ++ } ++ let signals: Vec> = signals ++ .into_iter() ++ .map(|list| { ++ list.into_iter() ++ .filter(|&(s, _)| { ++ state.seen.get(&vk::Handle::as_raw(s)).copied().unwrap_or(0) >= LONG_LIVED ++ }) ++ .collect() ++ }) ++ .collect(); + for (index, info) in infos.iter().enumerate() { + if info.p_command_buffers.is_null() || info.command_buffer_count == 0 { + continue; diff --git a/patches/metal2vulkan/0074-pending-audit-device-index-no-longer-shadowed-by-the.patch b/patches/metal2vulkan/0074-pending-audit-device-index-no-longer-shadowed-by-the.patch new file mode 100644 index 0000000..9fe4ef4 --- /dev/null +++ b/patches/metal2vulkan/0074-pending-audit-device-index-no-longer-shadowed-by-the.patch @@ -0,0 +1,189 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 17:21:20 +0000 +Subject: [PATCH 74/92] pending audit: device index no longer shadowed by the + info index; tracks pools' command buffers (a pool destroyed while one is + pending is reported) and stops reading destroyed semaphores + +--- + crates/metal2vulkan-runtime/src/device/mod.rs | 3 + + .../metal2vulkan-runtime/src/device/prime.rs | 3 + + .../metal2vulkan-runtime/src/pending_audit.rs | 58 +++++++++++++++++-- + .../src/recorder/replays.rs | 1 + + crates/metal2vulkan-runtime/src/release.rs | 2 + + 5 files changed, 61 insertions(+), 6 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/device/mod.rs b/crates/metal2vulkan-runtime/src/device/mod.rs +index 4814fae2..920f2e14 100644 +--- a/crates/metal2vulkan-runtime/src/device/mod.rs ++++ b/crates/metal2vulkan-runtime/src/device/mod.rs +@@ -612,6 +612,9 @@ impl DeviceShared { + self.vk().allocate_command_buffers(info) + }; + exit_guard::called(Call::AllocateCommandBuffers); ++ if let Ok(buffers) = &allocated { ++ crate::pending_audit::allocated(info.command_pool, buffers); ++ } + allocated.map_err(super::error::vulkan) + } + +diff --git a/crates/metal2vulkan-runtime/src/device/prime.rs b/crates/metal2vulkan-runtime/src/device/prime.rs +index b0b2de77..a0ecd0d1 100644 +--- a/crates/metal2vulkan-runtime/src/device/prime.rs ++++ b/crates/metal2vulkan-runtime/src/device/prime.rs +@@ -663,6 +663,9 @@ impl Primer<'_> { + self.loader_word.restore(); + self.device.allocate_command_buffers(info) + }; ++ if let Ok(made) = &buffers { ++ crate::pending_audit::allocated(info.command_pool, made); ++ } + Ok(buffers.map_err(vulkan)?[0]) + } + +diff --git a/crates/metal2vulkan-runtime/src/pending_audit.rs b/crates/metal2vulkan-runtime/src/pending_audit.rs +index 8f0549fc..73f11b7c 100644 +--- a/crates/metal2vulkan-runtime/src/pending_audit.rs ++++ b/crates/metal2vulkan-runtime/src/pending_audit.rs +@@ -63,6 +63,8 @@ struct State { + executed_by: HashMap>, + /// A primary's last submission since its last begin. + subs: HashMap, ++ /// Each command pool's command buffers, as allocated. ++ pools: HashMap>, + } + + static STATE: LazyLock> = LazyLock::new(|| Mutex::new(State::default())); +@@ -78,10 +80,15 @@ fn line(text: String) { + } + + /// Why `sub` is still pending, or `None` where it completed (or nothing tells). +-fn pending(sub: &Sub) -> Option { ++fn pending(seen: &HashMap, sub: &Sub) -> Option { + let devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let device = devices.get(sub.device)?; + for &(semaphore, value) in &sub.signals { ++ // Only a semaphore still live and long-lived: a destroyed one is out of `seen` ++ // ([`released`]), and one made again at its handle starts its count over. ++ if seen.get(&vk::Handle::as_raw(semaphore)).copied().unwrap_or(0) < LONG_LIVED { ++ continue; ++ } + // SAFETY: a semaphore of the device a submission signalled; read only. + if let Ok(now) = unsafe { device.get_semaphore_counter_value(semaphore) } { + if now < value { +@@ -102,7 +109,7 @@ fn pending(sub: &Sub) -> Option { + fn check_and_forget(state: &mut State, buffer: u64, site: &'static str) { + CHECKED.fetch_add(1, Ordering::Relaxed); + if let Some(sub) = state.subs.get(&buffer) { +- if let Some(why) = pending(sub) { ++ if let Some(why) = pending(&state.seen, sub) { + let n = FOUND.fetch_add(1, Ordering::Relaxed) + 1; + line(format!( + "#{n} {site}: command buffer {buffer:#x} itself, submitted at {} {:.2} ms ago, is pending ({why})", +@@ -115,7 +122,7 @@ fn check_and_forget(state: &mut State, buffer: u64, site: &'static str) { + for primary in primaries { + match state.subs.get(primary) { + Some(sub) => { +- if let Some(why) = pending(sub) { ++ if let Some(why) = pending(&state.seen, sub) { + let n = FOUND.fetch_add(1, Ordering::Relaxed) + 1; + line(format!( + "#{n} {site}: secondary {buffer:#x} executed by primary {primary:#x}, submitted at {} {:.2} ms ago, is pending ({why})", +@@ -221,7 +228,7 @@ pub(crate) fn submits( + if !*ON { + return; + } +- let index = { ++ let device_index = { + let mut devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + match devices.iter().position(|d| d.handle() == device.handle()) { + Some(at) => at, +@@ -302,7 +309,7 @@ pub(crate) fn submits( + } + } + if let Some(previous) = state.subs.get(&p) { +- if let Some(why) = pending(previous) { ++ if let Some(why) = pending(&state.seen, previous) { + let n = FOUND.fetch_add(1, Ordering::Relaxed) + 1; + line(format!( + "#{n} {site}: primary {p:#x} submitted again while its submission at {} {:.2} ms ago is pending ({why})", +@@ -314,7 +321,7 @@ pub(crate) fn submits( + state.subs.insert( + p, + Sub { +- device: index, ++ device: device_index, + site, + at, + signals: mine.clone(), +@@ -332,6 +339,45 @@ pub(crate) fn submits( + } + } + ++/// A `vkAllocateCommandBuffers` from `pool` that made `buffers`. ++pub(crate) fn allocated(pool: vk::CommandPool, buffers: &[vk::CommandBuffer]) { ++ if !*ON { ++ return; ++ } ++ let mut state = STATE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ let list = state.pools.entry(vk::Handle::as_raw(pool)).or_default(); ++ list.extend(buffers.iter().map(|&b| key(b))); ++} ++ ++/// A release (`release.rs`) of the object `raw` by `call`, before it: a command pool's command ++/// buffers are checked as by a reset and forgotten (their handles may be made again); a ++/// semaphore is no longer read. ++pub(crate) fn released(call: crate::exit_guard::Call, raw: u64) { ++ if !*ON { ++ return; ++ } ++ use crate::exit_guard::Call; ++ let mut state = STATE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ match call { ++ Call::DestroyCommandPool => { ++ let Some(buffers) = state.pools.remove(&raw) else { ++ return; ++ }; ++ for buffer in buffers { ++ check_and_forget(&mut state, buffer, "pool destroy"); ++ state.generation.remove(&buffer); ++ state.executed_by.remove(&buffer); ++ state.executes.remove(&buffer); ++ state.subs.remove(&buffer); ++ } ++ } ++ Call::DestroySemaphore => { ++ state.seen.remove(&raw); ++ } ++ _ => {} ++ } ++} ++ + /// The routed commands' hook (`recorder/thunks.rs`), for a command that reaches the driver. + macro_rules! hook { + (begin_command_buffer, $cb:ident, $info:ident) => { +diff --git a/crates/metal2vulkan-runtime/src/recorder/replays.rs b/crates/metal2vulkan-runtime/src/recorder/replays.rs +index 7e673d29..6fec2282 100644 +--- a/crates/metal2vulkan-runtime/src/recorder/replays.rs ++++ b/crates/metal2vulkan-runtime/src/recorder/replays.rs +@@ -553,6 +553,7 @@ impl Replays { + raw.allocate_command_buffers(&info) + }?[0]; + exit_guard::called(Call::AllocateCommandBuffers); ++ crate::pending_audit::allocated(info.command_pool, &[buffer]); + pool.made(level, buffer); + Ok((buffer, false)) + } +diff --git a/crates/metal2vulkan-runtime/src/release.rs b/crates/metal2vulkan-runtime/src/release.rs +index 684d84ec..dad738c7 100644 +--- a/crates/metal2vulkan-runtime/src/release.rs ++++ b/crates/metal2vulkan-runtime/src/release.rs +@@ -144,6 +144,8 @@ macro_rules! released { + unsafe fn destroy(self, device: &ash::Device, _: Door) { + // LOCAL (Atlas) diagnostic: the image census and registry (`image_log.rs`). + crate::image_log::released(Call::$call, vk::Handle::as_raw(self)); ++ // LOCAL (Atlas) diagnostic: the pending audit (`pending_audit.rs`). ++ crate::pending_audit::released(Call::$call, vk::Handle::as_raw(self)); + // SAFETY: the caller's. + unsafe { device.$method(self, None) } + } diff --git a/patches/metal2vulkan/0075-pending-audit-forgets-destroyed-devices-and-a-hit-li.patch b/patches/metal2vulkan/0075-pending-audit-forgets-destroyed-devices-and-a-hit-li.patch new file mode 100644 index 0000000..4ccfc35 --- /dev/null +++ b/patches/metal2vulkan/0075-pending-audit-forgets-destroyed-devices-and-a-hit-li.patch @@ -0,0 +1,109 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 17:58:56 +0000 +Subject: [PATCH 75/92] pending audit: forgets destroyed devices, and a hit + line gives each live device's read of the timeline + +--- + .../metal2vulkan-runtime/src/pending_audit.rs | 52 +++++++++++++++++-- + crates/metal2vulkan-runtime/src/release.rs | 2 + + 2 files changed, 51 insertions(+), 3 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/pending_audit.rs b/crates/metal2vulkan-runtime/src/pending_audit.rs +index 73f11b7c..76361024 100644 +--- a/crates/metal2vulkan-runtime/src/pending_audit.rs ++++ b/crates/metal2vulkan-runtime/src/pending_audit.rs +@@ -82,6 +82,10 @@ fn line(text: String) { + /// Why `sub` is still pending, or `None` where it completed (or nothing tells). + fn pending(seen: &HashMap, sub: &Sub) -> Option { + let devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ let dead = DEAD.lock().unwrap_or_else(std::sync::PoisonError::into_inner).clone(); ++ if dead.contains(&sub.device) { ++ return None; ++ } + let device = devices.get(sub.device)?; + for &(semaphore, value) in &sub.signals { + // Only a semaphore still live and long-lived: a destroyed one is out of `seen` +@@ -92,9 +96,25 @@ fn pending(seen: &HashMap, sub: &Sub) -> Option { + // SAFETY: a semaphore of the device a submission signalled; read only. + if let Ok(now) = unsafe { device.get_semaphore_counter_value(semaphore) } { + if now < value { ++ // Each device's read of it, to tell a wrong device (another fd's syncobj) apart. ++ let others: Vec = devices ++ .iter() ++ .enumerate() ++ .filter(|(i, _)| !dead.contains(i)) ++ .map(|(i, d)| { ++ // SAFETY: as above; a read through another device only reads. ++ match unsafe { d.get_semaphore_counter_value(semaphore) } { ++ Ok(v) => format!("{i}:{v}"), ++ Err(e) => format!("{i}:{e:?}"), ++ } ++ }) ++ .collect(); + return Some(format!( +- "timeline {:#x} at {now} < {value}", +- vk::Handle::as_raw(semaphore) ++ "timeline {:#x} at {now} < {value} (device {} of {}; reads {})", ++ vk::Handle::as_raw(semaphore), ++ sub.device, ++ devices.len(), ++ others.join(",") + )); + } + } +@@ -230,7 +250,10 @@ pub(crate) fn submits( + } + let device_index = { + let mut devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); +- match devices.iter().position(|d| d.handle() == device.handle()) { ++ let dead = DEAD.lock().unwrap_or_else(std::sync::PoisonError::into_inner).clone(); ++ match (devices.iter().enumerate()) ++ .position(|(i, d)| !dead.contains(&i) && d.handle() == device.handle()) ++ { + Some(at) => at, + None => { + devices.push(device.clone()); +@@ -339,6 +362,29 @@ pub(crate) fn submits( + } + } + ++/// A `vkDestroyDevice` of `device`, before it: it is no longer read through, and what it ++/// submitted is forgotten. ++pub(crate) fn device_destroyed(device: &ash::Device) { ++ if !*ON { ++ return; ++ } ++ let mut state = STATE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ let mut devices = DEVICES.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ let mut dead = DEAD.lock().unwrap_or_else(std::sync::PoisonError::into_inner); ++ let found = (devices.iter().enumerate()) ++ .position(|(i, d)| !dead.contains(&i) && d.handle() == device.handle()); ++ if let Some(at) = found { ++ // Keep the indices of the others: a destroyed device's slot reads nothing. ++ state.subs.retain(|_, sub| sub.device != at); ++ dead.push(at); ++ } ++ drop(dead); ++ drop(devices); ++} ++ ++/// The indices in [`DEVICES`] of devices destroyed. ++static DEAD: LazyLock>> = LazyLock::new(|| Mutex::new(Vec::new())); ++ + /// A `vkAllocateCommandBuffers` from `pool` that made `buffers`. + pub(crate) fn allocated(pool: vk::CommandPool, buffers: &[vk::CommandBuffer]) { + if !*ON { +diff --git a/crates/metal2vulkan-runtime/src/release.rs b/crates/metal2vulkan-runtime/src/release.rs +index dad738c7..3dc7c08f 100644 +--- a/crates/metal2vulkan-runtime/src/release.rs ++++ b/crates/metal2vulkan-runtime/src/release.rs +@@ -180,6 +180,8 @@ impl Released for WholeDevice { + const CALL: Call = Call::DestroyDevice; + + unsafe fn destroy(self, device: &ash::Device, _: Door) { ++ // LOCAL (Atlas) diagnostic: the pending audit (`pending_audit.rs`). ++ crate::pending_audit::device_destroyed(device); + // SAFETY: the caller's: every object made from the device is released. + unsafe { device.destroy_device(None) } + } diff --git a/patches/metal2vulkan/0076-lab-M2V_FLAT_TILE-TWxTH-a-flat-grid-s-32-bit-index-w.patch b/patches/metal2vulkan/0076-lab-M2V_FLAT_TILE-TWxTH-a-flat-grid-s-32-bit-index-w.patch new file mode 100644 index 0000000..983c891 --- /dev/null +++ b/patches/metal2vulkan/0076-lab-M2V_FLAT_TILE-TWxTH-a-flat-grid-s-32-bit-index-w.patch @@ -0,0 +1,125 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 18:02:33 +0000 +Subject: [PATCH 76/92] lab: M2V_FLAT_TILE=TWxTH: a flat grid's 32-bit index + walks each plane in TW x TH tiles (a bijection onto the same positions), for + 2-D locality + +--- + crates/metal2vulkan/src/lower/dispatch.rs | 85 +++++++++++++++++++---- + 1 file changed, 73 insertions(+), 12 deletions(-) + +diff --git a/crates/metal2vulkan/src/lower/dispatch.rs b/crates/metal2vulkan/src/lower/dispatch.rs +index e49b0bc2..9a5d83e0 100644 +--- a/crates/metal2vulkan/src/lower/dispatch.rs ++++ b/crates/metal2vulkan/src/lower/dispatch.rs +@@ -1348,12 +1348,7 @@ impl KernelDecl { + let first: ArmFn<'_, ()> = Box::new(move |fb, _| { + let first = fb.convert(ConvOp::UConvert, start, u32_)?; + let index = fb.binary(BinOp::IAdd, first, lin)?; +- let x = fb.binary(BinOp::UMod, index, gx)?; +- let row = fb.binary(BinOp::UDiv, index, gx)?; +- let y = fb.binary(BinOp::UMod, row, gy)?; +- let z = fb.binary(BinOp::UDiv, row, gy)?; +- let inside = fb.binary(BinOp::ULessThan, z, gz)?; +- Ok(ArmEnd::FallThrough(vec![x, y, z, inside])) ++ Ok(ArmEnd::FallThrough(Self::position32(fb, index, [gx, gy, gz])?)) + }); + let narrow: ArmFn<'_, ()> = Box::new(move |fb, _| { + let group = fb.binary(BinOp::IMul, wz, ny)?; +@@ -1362,12 +1357,7 @@ impl KernelDecl { + let group = fb.binary(BinOp::IAdd, group, wx)?; + let first = fb.binary(BinOp::IMul, group, n)?; + let index = fb.binary(BinOp::IAdd, first, lin)?; +- let x = fb.binary(BinOp::UMod, index, gx)?; +- let row = fb.binary(BinOp::UDiv, index, gx)?; +- let y = fb.binary(BinOp::UMod, row, gy)?; +- let z = fb.binary(BinOp::UDiv, row, gy)?; +- let inside = fb.binary(BinOp::ULessThan, z, gz)?; +- Ok(ArmEnd::FallThrough(vec![x, y, z, inside])) ++ Ok(ArmEnd::FallThrough(Self::position32(fb, index, [gx, gy, gz])?)) + }); + let wide: ArmFn<'_, ()> = Box::new(move |fb, _| { + let w = |fb: &mut FunctionBuilder<'_>, v: SValue| fb.convert(ConvOp::UConvert, v, u64_); +@@ -1419,6 +1409,65 @@ impl KernelDecl { + Ok((pos, inside)) + } + ++ /// A flat grid's thread position from its 32-bit `index` over the grid `[gx, gy, gz]`, and ++ /// whether it lies inside (`z < gz`): `x` fastest, then `y`, then `z` ([`emit_entry`]'s ++ /// order), or, under LOCAL (Atlas) `M2V_FLAT_TILE=TWxTH` ([`flat_tile`]), a bijection of the ++ /// same indices onto the same positions that walks each plane in bands of `TH` rows and each ++ /// band in tiles of `TW` columns (the last band and the last tile of a band narrower), so a ++ /// threadgroup's consecutive indices cover a 2-D tile rather than a run of one row: the ++ /// partition-oblivious kernel reads and writes the same texels, with 2-D locality in tiled ++ /// images. The tiled form is taken only where `gx` and `gy` are below 2^16 (so a plane and a ++ /// band count in 32 bits); elsewhere the plain one. ++ fn position32( ++ fb: &mut FunctionBuilder<'_>, ++ index: SValue, ++ [gx, gy, gz]: [SValue; 3], ++ ) -> BResult> { ++ let x = fb.binary(BinOp::UMod, index, gx)?; ++ let row = fb.binary(BinOp::UDiv, index, gx)?; ++ let y = fb.binary(BinOp::UMod, row, gy)?; ++ let z = fb.binary(BinOp::UDiv, row, gy)?; ++ let inside = fb.binary(BinOp::ULessThan, z, gz)?; ++ let Some((tw, th)) = flat_tile() else { ++ return Ok(vec![x, y, z, inside]); ++ }; ++ let min = |fb: &mut FunctionBuilder<'_>, a: SValue, b: SValue| -> BResult { ++ let less = fb.binary(BinOp::ULessThan, a, b)?; ++ fb.select(less, a, b) ++ }; ++ let (tw, th) = (Self::k(fb, tw)?, Self::k(fb, th)?); ++ let plane = fb.binary(BinOp::IMul, gx, gy)?; ++ let tz = fb.binary(BinOp::UDiv, index, plane)?; ++ let r = fb.binary(BinOp::UMod, index, plane)?; ++ let band = fb.binary(BinOp::IMul, th, gx)?; ++ let b = fb.binary(BinOp::UDiv, r, band)?; ++ let j = fb.binary(BinOp::UMod, r, band)?; ++ let top = fb.binary(BinOp::IMul, b, th)?; ++ let left_rows = fb.binary(BinOp::ISub, gy, top)?; ++ let rows = min(fb, th, left_rows)?; ++ let tile = fb.binary(BinOp::IMul, tw, rows)?; ++ let c = fb.binary(BinOp::UDiv, j, tile)?; ++ let k = fb.binary(BinOp::UMod, j, tile)?; ++ let left = fb.binary(BinOp::IMul, c, tw)?; ++ let left_cols = fb.binary(BinOp::ISub, gx, left)?; ++ let w = min(fb, tw, left_cols)?; ++ let kx = fb.binary(BinOp::UMod, k, w)?; ++ let ky = fb.binary(BinOp::UDiv, k, w)?; ++ let tx = fb.binary(BinOp::IAdd, left, kx)?; ++ let ty = fb.binary(BinOp::IAdd, top, ky)?; ++ let tinside = fb.binary(BinOp::ULessThan, tz, gz)?; ++ let bound = Self::k(fb, 1 << 16)?; ++ let small_x = fb.binary(BinOp::ULessThan, gx, bound)?; ++ let small_y = fb.binary(BinOp::ULessThan, gy, bound)?; ++ let small = fb.binary(BinOp::LogicalAnd, small_x, small_y)?; ++ Ok(vec![ ++ fb.select(small, tx, x)?, ++ fb.select(small, ty, y)?, ++ fb.select(small, tz, z)?, ++ fb.select(small, tinside, inside)?, ++ ]) ++ } ++ + /// The quotient (low word, high word) and remainder of the `u64` `n` (`[low, high]`) by the + /// `u32` `d` (not 0), in 32-bit operations: `hi / d` and `r = hi % d` (so `r < d`), then 32 + /// restoring steps over the low word's bits, highest first, each `r = 2r + bit` and, when +@@ -1727,3 +1776,15 @@ impl KernelDecl { + + #[cfg(test)] + mod tests; ++ ++/// LOCAL (Atlas, lab): the flat grid's tile, `M2V_FLAT_TILE=TWxTH` (each 1 to 1024), read once ++/// per process; unset or malformed, none ([`KernelDecl::position32`]'s plain order). ++fn flat_tile() -> Option<(u32, u32)> { ++ static TILE: std::sync::OnceLock> = std::sync::OnceLock::new(); ++ *TILE.get_or_init(|| { ++ let v = std::env::var("M2V_FLAT_TILE").ok()?; ++ let (a, b) = v.split_once('x')?; ++ let (a, b) = (a.parse::().ok()?, b.parse::().ok()?); ++ ((1..=1024).contains(&a) && (1..=1024).contains(&b)).then_some((a, b)) ++ }) ++} diff --git a/patches/metal2vulkan/0077-lab-live-shared_ccs-1-a-GPU-written-Shared-Managed-t.patch b/patches/metal2vulkan/0077-lab-live-shared_ccs-1-a-GPU-written-Shared-Managed-t.patch new file mode 100644 index 0000000..7d1f9f5 --- /dev/null +++ b/patches/metal2vulkan/0077-lab-live-shared_ccs-1-a-GPU-written-Shared-Managed-t.patch @@ -0,0 +1,33 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 18:03:51 +0000 +Subject: [PATCH 77/92] lab: live shared_ccs=1: a GPU-written Shared/Managed + texture that does not trap takes no host-transfer usage (keeps CCS on ANV); + its CPU copies take the staging path + +--- + crates/metal2vulkan-runtime/src/texture.rs | 11 ++++++++++- + 1 file changed, 10 insertions(+), 1 deletion(-) + +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 8486ee10..21aca83e 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -3223,7 +3223,16 @@ impl Device { + let one_sample = layout.samples == 1; + let staged = + !traps && !held && !private && one_sample && self.stages(&plan, layout.pixel_format); +- let refused = (held || private) && !traps; ++ // LOCAL (Atlas, lab): live `shared_ccs=1`: a `Shared` or `Managed` texture the GPU writes ++ // (storage or color attachment), whose pages do not trap, takes no host-transfer usage, ++ // which on ANV turns its compression off: its CPU copies go through the staging queue (or ++ // the device's other copies) instead, and the GPU's passes and kernels over it keep CCS. ++ let gpu_written = (plan.usage) ++ .intersects(ImageUsageFlags::STORAGE | ImageUsageFlags::COLOR_ATTACHMENT); ++ let compressed = !traps ++ && gpu_written ++ && crate::live::get("shared_ccs").as_deref() == Some("1"); ++ let refused = ((held || private) && !traps) || compressed; + let hosted = match refused || !one_sample || is_pvrtc(descriptor.pixel_format) { + true => None, + false => self.host_placed_image(&plan, layout.texture_type, class, staged, traps), diff --git a/patches/metal2vulkan/0078-pending-audit-skips-semaphores-that-read-0-binary-se.patch b/patches/metal2vulkan/0078-pending-audit-skips-semaphores-that-read-0-binary-se.patch new file mode 100644 index 0000000..d887817 --- /dev/null +++ b/patches/metal2vulkan/0078-pending-audit-skips-semaphores-that-read-0-binary-se.patch @@ -0,0 +1,26 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 20:02:49 +0000 +Subject: [PATCH 78/92] pending audit skips semaphores that read 0 (binary + semaphores in a signal list) + +--- + crates/metal2vulkan-runtime/src/pending_audit.rs | 5 +++++ + 1 file changed, 5 insertions(+) + +diff --git a/crates/metal2vulkan-runtime/src/pending_audit.rs b/crates/metal2vulkan-runtime/src/pending_audit.rs +index 76361024..a3899011 100644 +--- a/crates/metal2vulkan-runtime/src/pending_audit.rs ++++ b/crates/metal2vulkan-runtime/src/pending_audit.rs +@@ -95,6 +95,11 @@ fn pending(seen: &HashMap, sub: &Sub) -> Option { + } + // SAFETY: a semaphore of the device a submission signalled; read only. + if let Ok(now) = unsafe { device.get_semaphore_counter_value(semaphore) } { ++ // A read of 0 under a signal value seen 64 times or more is a binary semaphore in the ++ // signal list (its values are ignored, and its counter reads 0): not a timeline. ++ if now == 0 { ++ continue; ++ } + if now < value { + // Each device's read of it, to tell a wrong device (another fd's syncobj) apart. + let others: Vec = devices diff --git a/patches/metal2vulkan/0079-Revert-pending-audit-skips-semaphores-that-read-0-bi.patch b/patches/metal2vulkan/0079-Revert-pending-audit-skips-semaphores-that-read-0-bi.patch new file mode 100644 index 0000000..192a463 --- /dev/null +++ b/patches/metal2vulkan/0079-Revert-pending-audit-skips-semaphores-that-read-0-bi.patch @@ -0,0 +1,27 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 20:03:08 +0000 +Subject: [PATCH 79/92] Revert "pending audit skips semaphores that read 0 + (binary semaphores in a signal list)" + +This reverts commit 87a9669b4a730e5b3153cebcfb72bd53091392e1. +--- + crates/metal2vulkan-runtime/src/pending_audit.rs | 5 ----- + 1 file changed, 5 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/pending_audit.rs b/crates/metal2vulkan-runtime/src/pending_audit.rs +index a3899011..76361024 100644 +--- a/crates/metal2vulkan-runtime/src/pending_audit.rs ++++ b/crates/metal2vulkan-runtime/src/pending_audit.rs +@@ -95,11 +95,6 @@ fn pending(seen: &HashMap, sub: &Sub) -> Option { + } + // SAFETY: a semaphore of the device a submission signalled; read only. + if let Ok(now) = unsafe { device.get_semaphore_counter_value(semaphore) } { +- // A read of 0 under a signal value seen 64 times or more is a binary semaphore in the +- // signal list (its values are ignored, and its counter reads 0): not a timeline. +- if now == 0 { +- continue; +- } + if now < value { + // Each device's read of it, to tell a wrong device (another fd's syncobj) apart. + let others: Vec = devices diff --git a/patches/metal2vulkan/0080-live-host_mutable-host-transfer-color-images-mutable.patch b/patches/metal2vulkan/0080-live-host_mutable-host-transfer-color-images-mutable.patch new file mode 100644 index 0000000..dcb72db --- /dev/null +++ b/patches/metal2vulkan/0080-live-host_mutable-host-transfer-color-images-mutable.patch @@ -0,0 +1,33 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 20:25:55 +0000 +Subject: [PATCH 80/92] live host_mutable: host-transfer color images mutable + to any compatible format (no view shadows; ANV keeps no aux there) + +--- + crates/metal2vulkan-runtime/src/texture.rs | 12 ++++++++++++ + 1 file changed, 12 insertions(+) + +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 21aca83e..43c8b18f 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -3115,6 +3115,18 @@ impl Device { + } + let mut hosted = plan.clone(); + hosted.usage |= ImageUsageFlags::HOST_TRANSFER; ++ // LOCAL (Atlas), live `host_mutable=1`: ANV keeps no aux surface for a host-transfer ++ // image (Mesa 26.2.4 anv_image.c, `HOST_TRANSFER` disables compression), so making it ++ // mutable to any compatible format costs nothing there, and every later view of another ++ // format of its class is a Vulkan view of it, not a shadow image copied in and back ++ // (`texture/view_shadow.rs`, plan (b)). Color images only. ++ if crate::live::on("host_mutable", false) ++ && !hosted.usage.contains(ImageUsageFlags::DEPTH_STENCIL_ATTACHMENT) ++ && !texture_aspects(&hosted).intersects(vk::ImageAspectFlags::DEPTH | vk::ImageAspectFlags::STENCIL) ++ { ++ hosted.flags |= metal2vulkan::vk::ImageCreateFlags::MUTABLE_FORMAT; ++ hosted.view_formats.clear(); ++ } + // An image the host writes too is a parked one of its making where one is parked + // (`recycle.rs`), its host-transfer usage in its plan: on a miss, `placed_image`'s, as + // before. One the host only reads (a trapping surface's) may lie in memory the host maps diff --git a/patches/metal2vulkan/0081-live-batch_end_light-no-end-barrier-per-batch-in-exa.patch b/patches/metal2vulkan/0081-live-batch_end_light-no-end-barrier-per-batch-in-exa.patch new file mode 100644 index 0000000..4aecf01 --- /dev/null +++ b/patches/metal2vulkan/0081-live-batch_end_light-no-end-barrier-per-batch-in-exa.patch @@ -0,0 +1,50 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 20:50:21 +0000 +Subject: [PATCH 81/92] live batch_end_light: no end barrier per batch in exact + order (i915 flushes at each request end) + +--- + crates/metal2vulkan-runtime/src/batch_order.rs | 8 ++++++++ + crates/metal2vulkan-runtime/src/tracker.rs | 9 +++++++++ + 2 files changed, 17 insertions(+) + +diff --git a/crates/metal2vulkan-runtime/src/batch_order.rs b/crates/metal2vulkan-runtime/src/batch_order.rs +index 1fef7309..0d5a352d 100644 +--- a/crates/metal2vulkan-runtime/src/batch_order.rs ++++ b/crates/metal2vulkan-runtime/src/batch_order.rs +@@ -58,6 +58,14 @@ fn ends_order(driver: Option) -> bool { + pub(crate) fn end(by_ends: bool) -> Barrier { + match by_ends { + true => Barrier::to_all(), ++ // LOCAL (Atlas), live `batch_end_light=1`, exact order only: no memory barrier at a ++ // batch's end. The host reads a batch's writes only after its queue's timeline signal, ++ // which a group gives only at its last batch; i915 ends every kernel submission (request) ++ // with a pipe control that flushes the render, depth, data and tile caches with a CS stall ++ // before the seqno write (gen12 fini breadcrumb), so the writes are in memory when the ++ // signal lands. Inside a group the batches' GPU order is the exact form's start barriers. ++ // cbbench (ARL, ANV 26.2.4): an ALL -> HOST end costs 27 us a Sim-sized pass (58.9 vs 31.8). ++ false if crate::live::on("batch_end_light", false) => Barrier::nothing(), + false => Barrier::to_host(), + } + } +diff --git a/crates/metal2vulkan-runtime/src/tracker.rs b/crates/metal2vulkan-runtime/src/tracker.rs +index 5905bd90..d32cb2ca 100644 +--- a/crates/metal2vulkan-runtime/src/tracker.rs ++++ b/crates/metal2vulkan-runtime/src/tracker.rs +@@ -354,6 +354,15 @@ impl Barrier { + + /// The end of every batch: every earlier write before the host's reads and writes, so the CPU + /// sees the GPU's writes once the batch completes, and before the batch's signals. ++ /// LOCAL (Atlas): a barrier that orders nothing (top of pipe to bottom of pipe, no access): ++ /// drivers record no flush or stall for it (`batch_order::end`, `batch_end_light`). ++ pub(crate) fn nothing() -> Barrier { ++ Barrier { ++ src: Access::new(vk::PipelineStageFlags2::TOP_OF_PIPE, vk::AccessFlags2::NONE), ++ dst: Access::new(vk::PipelineStageFlags2::BOTTOM_OF_PIPE, vk::AccessFlags2::NONE), ++ } ++ } ++ + pub(super) fn to_host() -> Barrier { + Barrier { + src: Access::new( diff --git a/patches/metal2vulkan/0082-diagnostic-M2V_VIEW_LOG-names-views-kept-apart-from-.patch b/patches/metal2vulkan/0082-diagnostic-M2V_VIEW_LOG-names-views-kept-apart-from-.patch new file mode 100644 index 0000000..ecb5797 --- /dev/null +++ b/patches/metal2vulkan/0082-diagnostic-M2V_VIEW_LOG-names-views-kept-apart-from-.patch @@ -0,0 +1,36 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 21:04:44 +0000 +Subject: [PATCH 82/92] diagnostic: M2V_VIEW_LOG names views kept apart from + their image + +--- + crates/metal2vulkan-runtime/src/texture.rs | 15 ++++++++++++++- + 1 file changed, 14 insertions(+), 1 deletion(-) + +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 43c8b18f..797a3dbf 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -5466,7 +5466,20 @@ impl Texture { + Storage::Image(image) + | Storage::Alias { image, .. } + | Storage::Shadow { image, .. } => { +- !self.device().image_takes_view(image, record.pixel_format) ++ let apart = !self.device().image_takes_view(image, record.pixel_format); ++ // LOCAL (Atlas) diagnostic, `M2V_VIEW_LOG`: why a view is not a Vulkan view of its ++ // texture's image (the first 40). ++ if apart && std::env::var_os("M2V_VIEW_LOG").is_some() { ++ static N: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); ++ if N.fetch_add(1, std::sync::atomic::Ordering::Relaxed) < 40 { ++ let p = image.plan(); ++ eprintln!( ++ "m2v-view-log: view {:?} usage {:?} of image {:?} {:?} flags {:?} usage {:?} views {:?}", ++ record.pixel_format, record.usage, p.pixel_format, p.format, p.flags, p.usage, p.view_formats ++ ); ++ } ++ } ++ apart + } + }; + let mut record = record; diff --git a/patches/metal2vulkan/0083-live-view_render_direct-an-A8Unorm-view-of-an-R8Unor.patch b/patches/metal2vulkan/0083-live-view_render_direct-an-A8Unorm-view-of-an-R8Unor.patch new file mode 100644 index 0000000..4fe3971 --- /dev/null +++ b/patches/metal2vulkan/0083-live-view_render_direct-an-A8Unorm-view-of-an-R8Unor.patch @@ -0,0 +1,79 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 21:26:40 +0000 +Subject: [PATCH 83/92] live view_render_direct: an A8Unorm view of an R8Unorm + image renders into the image (no render shadow copies) + +--- + .../src/texture/rotation.rs | 5 +++ + .../src/texture/view_shadow.rs | 41 +++++++++++++++++++ + 2 files changed, 46 insertions(+) + +diff --git a/crates/metal2vulkan-runtime/src/texture/rotation.rs b/crates/metal2vulkan-runtime/src/texture/rotation.rs +index e0570caf..c5f36459 100644 +--- a/crates/metal2vulkan-runtime/src/texture/rotation.rs ++++ b/crates/metal2vulkan-runtime/src/texture/rotation.rs +@@ -123,6 +123,11 @@ impl Texture { + // A view its bindings present in its image's own format renders into its render shadow + // (`texture/view_shadow.rs`). + if record.read_as.is_some() { ++ if crate::live::on("view_render_direct", false) { ++ if let Some(direct) = self.view_render_direct() { ++ return Ok(direct); ++ } ++ } + return self.view_render_shadow("render_command_encoder"); + } + let rotation = record.presented.rotation; +diff --git a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +index d2b5ef59..2dee1c90 100644 +--- a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs ++++ b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +@@ -253,6 +253,47 @@ impl Texture { + Ok(texture) + } + ++ /// LOCAL (Atlas), live `view_render_direct=1`: a view its bindings read through its image's ++ /// own format (`read_as`) renders into that image itself where the view's own color ++ /// attachment format is the image's: an `A8Unorm` view of an `R8Unorm` texture on a device ++ /// without `A8_UNORM`, whose attachment is `R8_UNORM` with Metal's alpha in red ++ /// (`FormatEmulation::AlphaInRed`, lowered by the pipeline's attachment format). The render ++ /// shadow there is an `R8_UNORM` image copied in from and back to the same bytes' layout ++ /// (Liquid Glass masks over the Sim screen: ~2400 copies a second, 0.8 ms a frame). `None` ++ /// elsewhere, where the shadow stays. ++ pub(crate) fn view_render_direct(&self) -> Option { ++ let mut held = lock(&self.0.render_shadow); ++ if let Some(texture) = held.as_ref() { ++ return Some(texture.clone()); ++ } ++ let Storage::Image(image) = &self.0.storage else { ++ return None; ++ }; ++ let base = truth(image); ++ let device = self.device(); ++ let profile = device.compile_device().profile(); ++ let usage = [metal2vulkan::device::FormatUsage::ColorAttachment].into(); ++ let mapping = ++ metal2vulkan::metal::format::vk_format(self.0.record.pixel_format, &usage, profile).ok()?; ++ let plan = base.plan(); ++ if mapping.format != plan.format || !plan.usage.contains(ImageUsageFlags::COLOR_ATTACHMENT) { ++ return None; ++ } ++ let record = ViewRecord { ++ read_as: None, ++ ..self.0.record ++ }; ++ let texture = Texture::made( ++ device, ++ Storage::Image(Arc::clone(base)), ++ record, ++ Some(self.clone()), ++ IdUse::Runtime, ++ ); ++ *held = Some(texture.clone()); ++ Some(texture) ++ } ++ + /// The texture views are made from, at the root of `parentTexture`. + pub(super) fn root(&self) -> &Texture { + let mut root = self; diff --git a/patches/metal2vulkan/0084-view_render_direct-also-for-reinterpreted-views-a-vi.patch b/patches/metal2vulkan/0084-view_render_direct-also-for-reinterpreted-views-a-vi.patch new file mode 100644 index 0000000..7f18c57 --- /dev/null +++ b/patches/metal2vulkan/0084-view_render_direct-also-for-reinterpreted-views-a-vi.patch @@ -0,0 +1,67 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 21:46:20 +0000 +Subject: [PATCH 84/92] view_render_direct also for reinterpreted views: a view + whose every use is held in the image's own format is the image (A8 over R8) + +--- + .../src/texture/view_shadow.rs | 39 +++++++++++++++++++ + 1 file changed, 39 insertions(+) + +diff --git a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +index 2dee1c90..a9aeb65e 100644 +--- a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs ++++ b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +@@ -130,6 +130,11 @@ impl Texture { + if self.device().image_takes_view(base, record.pixel_format) { + return Ok(Storage::Image(Arc::clone(base))); + } ++ // LOCAL (Atlas), live `view_render_direct=1`: a view whose every use the device ++ // holds in the image's own format is the image itself (`same_bits`). ++ if crate::live::on("view_render_direct", false) && self.device().same_bits(base, record) { ++ return Ok(Storage::Image(Arc::clone(base))); ++ } + let shadow = self.device().new_view_shadow( + call, + base, +@@ -305,6 +310,40 @@ impl Texture { + } + + impl Device { ++ /// LOCAL (Atlas): whether every use of a view presenting `record` is held in `image`'s own ++ /// Vulkan format, so the view's texture may be the image itself, as a texture of the view's ++ /// pixel format made over it would be: an `A8Unorm` view of an `R8Unorm` image on a device ++ /// without `A8_UNORM`, whose sampled, storage and attachment forms are all `R8_UNORM` with ++ /// Metal's alpha in red (`FormatEmulation::AlphaInRed`). The image has each use's usage. ++ /// Liquid Glass's masks over the Sim screen took a shadow of the same bits, copied in and back ++ /// about 2400 times a second. ++ pub(super) fn same_bits(&self, image: &Image, record: &ViewRecord) -> bool { ++ use metal2vulkan::device::FormatUsage as U; ++ let plan = image.plan(); ++ let profile = self.compile_device().profile(); ++ let mut uses = Vec::new(); ++ if record.usage.contains(TextureUsage::SHADER_READ) { ++ uses.push((U::SampledImage, ImageUsageFlags::SAMPLED)); ++ } ++ if record.usage.contains(TextureUsage::SHADER_WRITE) { ++ uses.push((U::StorageImage, ImageUsageFlags::STORAGE)); ++ } ++ if record.usage.contains(TextureUsage::RENDER_TARGET) { ++ uses.push((U::ColorAttachment, ImageUsageFlags::COLOR_ATTACHMENT)); ++ } ++ if uses.is_empty() || record.alpha_reads_one || record.presented.rotation != 0 { ++ return false; ++ } ++ if !matches!(image.owner, Owner::Runtime { .. } | Owner::Caller(_) | Owner::HeapUnaliased(_)) { ++ return false; ++ } ++ uses.iter().all(|&(usage, flag)| { ++ plan.usage.contains(flag) ++ && metal2vulkan::metal::format::vk_format(record.pixel_format, &[usage].into(), profile) ++ .is_ok_and(|m| m.format == plan.format) ++ }) ++ } ++ + /// Where a view presenting `record` of `image` (its texture's image, or the image a shadowed + /// view's texels are copied from) reads and writes the image in the image's own format (the + /// module docs): that format and the swizzle over its channels its bindings present the view diff --git a/patches/metal2vulkan/0085-live-inline_lazy-a-lazy-surface-s-passes-record-thei.patch b/patches/metal2vulkan/0085-live-inline_lazy-a-lazy-surface-s-passes-record-thei.patch new file mode 100644 index 0000000..13101e4 --- /dev/null +++ b/patches/metal2vulkan/0085-live-inline_lazy-a-lazy-surface-s-passes-record-thei.patch @@ -0,0 +1,49 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 22:31:17 +0000 +Subject: [PATCH 85/92] live inline_lazy: a lazy surface's passes record their + first instance inline (no secondary) + +--- + .../metal2vulkan-runtime/src/render/inline_first.rs | 4 +++- + crates/metal2vulkan-runtime/src/texture.rs | 12 ++++++++++++ + 2 files changed, 15 insertions(+), 1 deletion(-) + +diff --git a/crates/metal2vulkan-runtime/src/render/inline_first.rs b/crates/metal2vulkan-runtime/src/render/inline_first.rs +index e67b4acd..ffda157a 100644 +--- a/crates/metal2vulkan-runtime/src/render/inline_first.rs ++++ b/crates/metal2vulkan-runtime/src/render/inline_first.rs +@@ -259,7 +259,9 @@ pub(super) fn opens_inline( + return Err(NotInline::Actions(refused)); + } + } +- if texture.over_buffer() && !texture.image_over_buffer() { ++ // LOCAL (Atlas), live `inline_lazy=1`: a lazy surface's image renders inline too. ++ let lazy = crate::live::on("inline_lazy", false) && texture.lazy_surface_attached(); ++ if texture.over_buffer() && !texture.image_over_buffer() && !lazy { + return Err(NotInline::BufferTextures); + } + let own = color.stand_in.is_none() +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 797a3dbf..105e3937 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -4430,6 +4430,18 @@ impl Texture { + } + } + ++ /// LOCAL (Atlas), live `inline_lazy`: whether the texture is a lazy surface's image (a shadow ++ /// whose copies are lazy, `shadow_lazy.rs`) with its bytes attached, which a pass renders in ++ /// place as it renders an image over a buffer's bytes (`render/inline_first.rs`). ++ pub(crate) fn lazy_surface_attached(&self) -> bool { ++ match &self.0.storage { ++ Storage::Shadow { image, copies, .. } => { ++ copies.lazy.is_some() && lock(&image.detached).is_none() ++ } ++ _ => false, ++ } ++ } ++ + /// Whether the texture is a buffer's bytes: a texture buffer or a shadowed texture. + pub(super) fn over_buffer(&self) -> bool { + !matches!(self.0.storage, Storage::Image(_)) diff --git a/patches/metal2vulkan/0086-barrier-log-M2V_BARRIER_LOG-and-live-precise_barrier.patch b/patches/metal2vulkan/0086-barrier-log-M2V_BARRIER_LOG-and-live-precise_barrier.patch new file mode 100644 index 0000000..2aa324a --- /dev/null +++ b/patches/metal2vulkan/0086-barrier-log-M2V_BARRIER_LOG-and-live-precise_barrier.patch @@ -0,0 +1,351 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 08:32:47 +0000 +Subject: [PATCH 86/92] barrier log (M2V_BARRIER_LOG) and live + precise_barriers: shadow runs between images and reach clears without the L3 + flush + +--- + .../metal2vulkan-runtime/src/barrier_log.rs | 72 +++++++++++++++++++ + .../metal2vulkan-runtime/src/clear_reach.rs | 5 +- + .../src/command_buffer/mod.rs | 4 ++ + .../src/command_buffer/submit.rs | 10 +++ + crates/metal2vulkan-runtime/src/encoder.rs | 22 +++++- + crates/metal2vulkan-runtime/src/lib.rs | 1 + + crates/metal2vulkan-runtime/src/tracker.rs | 52 ++++++++++++++ + .../metal2vulkan-runtime/src/work_counts.rs | 26 +++++++ + 8 files changed, 188 insertions(+), 4 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/barrier_log.rs + +diff --git a/crates/metal2vulkan-runtime/src/barrier_log.rs b/crates/metal2vulkan-runtime/src/barrier_log.rs +new file mode 100644 +index 00000000..0bfc7a7a +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/barrier_log.rs +@@ -0,0 +1,72 @@ ++//! LOCAL (Atlas) diagnostic: the barrier log, `M2V_BARRIER_LOG=1`. Counts each global memory ++//! barrier the runtime records through [`Barrier::record`](crate::tracker::Barrier) by the place ++//! that recorded it and its masks, and each barrier a batch's barrier command buffer carries as the ++//! batch is submitted (kept or recorded: `command_buffer/submit.rs`), and logs every 10 s the rates ++//! of the 30 most frequent, with the device's counts of command buffers begun by kind, submissions ++//! and finales. Unset (or `0`), each hook is one check of a `OnceLock`. ++ ++use std::collections::HashMap; ++use std::panic::Location; ++use std::sync::atomic::{AtomicU64, Ordering}; ++use std::sync::{Mutex, OnceLock}; ++use std::time::Instant; ++ ++/// Whether the log is on: read once. ++pub(crate) fn on() -> bool { ++ static ON: OnceLock = OnceLock::new(); ++ *ON.get_or_init(|| std::env::var("M2V_BARRIER_LOG").is_ok_and(|v| !v.is_empty() && v != "0")) ++} ++ ++/// The place, a label, and the masks: src stages, dst stages, src access, dst access. ++type Key = (&'static str, u32, &'static str, u64, u64, u64, u64); ++ ++static TABLE: Mutex>> = Mutex::new(None); ++ ++fn start() -> Instant { ++ static START: OnceLock = OnceLock::new(); ++ *START.get_or_init(Instant::now) ++} ++ ++/// Counts one barrier recorded or submitted at `site` (`label` says which). ++pub(crate) fn note(site: &'static Location<'static>, label: &'static str, masks: [u64; 4]) { ++ let _ = start(); ++ let key = (site.file(), site.line(), label, masks[0], masks[1], masks[2], masks[3]); ++ if let Ok(mut table) = TABLE.lock() { ++ *table.get_or_insert_with(HashMap::new).entry(key).or_insert(0) += 1; ++ } ++} ++ ++/// Logs the table and `counts` once 10 s passed since the last log; called on the scheduler's ++/// thread as batches are submitted. ++pub(crate) fn tick(counts: &crate::work_counts::Counters) { ++ static LAST: AtomicU64 = AtomicU64::new(0); ++ let now = start().elapsed().as_millis() as u64; ++ let last = LAST.load(Ordering::Relaxed); ++ if now.saturating_sub(last) < 10_000 ++ || LAST.compare_exchange(last, now, Ordering::Relaxed, Ordering::Relaxed).is_err() ++ { ++ return; ++ } ++ let secs = (now - last).max(1) as f64 / 1000.0; ++ let rows: Vec<(Key, u64)> = match TABLE.lock() { ++ Ok(mut table) => table.take().map(|t| t.into_iter().collect()).unwrap_or_default(), ++ Err(_) => return, ++ }; ++ let mut rows = rows; ++ rows.sort_by(|a, b| b.1.cmp(&a.1)); ++ let total: u64 = rows.iter().map(|r| r.1).sum(); ++ let mut text = format!( ++ "m2v barrier log: {:.0}/s barriers over {:.1} s; {}\n", ++ total as f64 / secs, ++ secs, ++ counts.barrier_log_line() ++ ); ++ for ((file, line, label, ss, ds, sa, da), n) in rows.iter().take(30) { ++ let file = file.rsplit("src/").next().unwrap_or(file); ++ text += &format!( ++ "m2v barrier log: {:8.0}/s {label} {file}:{line} src {ss:#x}/{sa:#x} dst {ds:#x}/{da:#x}\n", ++ *n as f64 / secs ++ ); ++ } ++ eprint!("{text}"); ++} +diff --git a/crates/metal2vulkan-runtime/src/clear_reach.rs b/crates/metal2vulkan-runtime/src/clear_reach.rs +index c6118d1f..76cb956a 100644 +--- a/crates/metal2vulkan-runtime/src/clear_reach.rs ++++ b/crates/metal2vulkan-runtime/src/clear_reach.rs +@@ -752,7 +752,8 @@ impl ReachClears { + crate::work_counts::PoolCompiled::AttachmentClear, + clears.len() as u64, + ); +- core.barrier(Barrier::everything()); ++ // LOCAL (Atlas), live `precise_barriers`: the clears write attachments' images alone. ++ core.barrier(Barrier::everything_on_images()); + core.command( + render_target::ATTACHMENT_STAGES, + &uses, +@@ -796,7 +797,7 @@ impl ReachClears { + } + }, + ); +- core.barrier(Barrier::everything()); ++ core.barrier(Barrier::everything_on_images()); + core.retain(clears); + } + } +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 3a0b2881..13e06af0 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -3781,6 +3781,10 @@ impl CommandBufferShared { + // and ended in its own recordings), and only this call records it, under the state + // lock; the slot's queries were reset by the batch's start, before it in submission + // order (`submit.rs`). ++ // LOCAL (Atlas) diagnostic: the barrier log's tick (`barrier_log.rs`). ++ if crate::barrier_log::on() { ++ crate::barrier_log::tick(&self.device.shared().counts); ++ } + unsafe { + crate::batch_order::end(self.device.shared().by_ends).record(device, raw); + if let Some((queries, first)) = stamp { +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index c6fc03ce..4d59cc9b 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -1523,6 +1523,16 @@ impl CommandBufferShared { + stamps: Stamps, + ) -> Result { + let device = &self.device; ++ // LOCAL (Atlas) diagnostic: the barrier log (`barrier_log.rs`). ++ if crate::barrier_log::on() { ++ for barrier in barriers { ++ crate::barrier_log::note(std::panic::Location::caller(), "batch-cb", barrier.log_masks()); ++ } ++ if barriers.is_empty() { ++ crate::barrier_log::note(std::panic::Location::caller(), "batch-cb-none", [0; 4]); ++ } ++ crate::barrier_log::tick(&device.shared().counts); ++ } + if transitions.is_empty() && !super::fences::ends_at_fence(device, at.queue) { + let taken = kept + .take(device, at.queue, barriers, stamps.kept(), at.value) +diff --git a/crates/metal2vulkan-runtime/src/encoder.rs b/crates/metal2vulkan-runtime/src/encoder.rs +index 25504a87..08ba2e43 100644 +--- a/crates/metal2vulkan-runtime/src/encoder.rs ++++ b/crates/metal2vulkan-runtime/src/encoder.rs +@@ -731,6 +731,7 @@ impl EncoderCore { + } + + /// Record a barrier the program asked for (a compute encoder's memory barriers). ++ #[track_caller] // LOCAL (Atlas) diagnostic: the barrier log names the caller (`barrier_log.rs`). + pub(super) fn barrier(&mut self, barrier: Barrier) { + let Some(raw) = self.recording() else { + return; +@@ -1617,6 +1618,11 @@ impl EncoderCore { + pub(super) fn shadow_copies(&mut self, copies: &[ShadowCopy]) { + let device = self.device().clone(); + let mut run = ShadowRun::default(); ++ // LOCAL (Atlas), live `precise_barriers`: a run whose copies are all between images takes ++ // `Barrier::everything_on_images` (no buffer's bytes, so no indirect arguments, are written). ++ run.images = copies ++ .iter() ++ .all(|copy| !matches!(copy.copies.source, crate::shadow::ShadowSource::Bytes { .. })); + // The keeps a dispatch at their copy in took, which record nothing. + let mut taken = Vec::new(); + for (i, copy) in copies.iter().enumerate() { +@@ -1675,7 +1681,7 @@ impl EncoderCore { + } + } + if run.recorded() { +- self.barrier(Barrier::everything()); ++ self.barrier(run.barrier()); + } + } + +@@ -1700,7 +1706,7 @@ impl EncoderCore { + let aliased = self.aliased(uses); + let tracked = aliased.as_deref().unwrap_or(uses); + if run.before(tracked, writes_truth) { +- self.barrier(Barrier::everything()); ++ self.barrier(run.barrier()); + } + let Some(raw) = self.recording() else { + return; +@@ -2453,6 +2459,8 @@ struct ShadowRun { + fenced: bool, + /// Whether a copy was recorded in the run. + recorded: bool, ++ /// LOCAL (Atlas): whether every copy of the run is between images (`ShadowSource::Image`). ++ images: bool, + } + + impl ShadowRun { +@@ -2462,6 +2470,16 @@ impl ShadowRun { + since: Some(Vec::new()), + fenced: false, + recorded: false, ++ images: false, ++ } ++ } ++ ++ /// LOCAL (Atlas): the run's barrier from everything before or to everything after: ++ /// `Barrier::everything_on_images` where every copy is between images, else `everything`. ++ fn barrier(&self) -> Barrier { ++ match self.images { ++ true => Barrier::everything_on_images(), ++ false => Barrier::everything(), + } + } + +diff --git a/crates/metal2vulkan-runtime/src/lib.rs b/crates/metal2vulkan-runtime/src/lib.rs +index 145f310a..4036c1c7 100644 +--- a/crates/metal2vulkan-runtime/src/lib.rs ++++ b/crates/metal2vulkan-runtime/src/lib.rs +@@ -165,6 +165,7 @@ mod noted_apart; + mod owned_kernel; + // LOCAL (Atlas) diagnostic: the pass timer (M2V_PASS_TIMES). + mod pass_times; ++mod barrier_log; // LOCAL (Atlas) diagnostic: M2V_BARRIER_LOG + mod pending_dispatch; + mod pipeline; + mod pixel_format_info; +diff --git a/crates/metal2vulkan-runtime/src/tracker.rs b/crates/metal2vulkan-runtime/src/tracker.rs +index d32cb2ca..49da39c2 100644 +--- a/crates/metal2vulkan-runtime/src/tracker.rs ++++ b/crates/metal2vulkan-runtime/src/tracker.rs +@@ -411,6 +411,43 @@ impl Barrier { + } + } + ++ /// LOCAL (Atlas), live `precise_barriers=1` (default off): [`everything`](Barrier::everything) ++ /// in the access types device commands perform, for a run of commands that write images alone ++ /// (a shadow's copies between images, `encoder.rs`; a pass's reach clears, `clear_reach.rs`): ++ /// every write a device command makes (shader, attachment, transfer) before every access of ++ /// every later command but a read of indirect arguments, which lie in a buffer; no `MEMORY_*` ++ /// mask, and so none of the host's, whose accesses a queue submission and the timeline's ++ /// signal order. Where the switch is off, `everything` itself. On ANV before Gfx20 a ++ /// `MEMORY_WRITE` on either side or an `INDIRECT_COMMAND_READ` is a flush of the whole L3 ++ /// (`anv_pipe_flush_bits_for_access_flags`, Mesa 26.2.4); cbbench, ARL, ANV 26.0.8, a ++ /// 1206x2622 pass a unit: no barrier 26.8 us, this one 28.2, `everything` 57.7. ++ pub(super) fn everything_on_images() -> Barrier { ++ if !crate::live::on("precise_barriers", false) { ++ return Barrier::everything(); ++ } ++ const WRITES: vk::AccessFlags2 = vk::AccessFlags2::from_raw( ++ vk::AccessFlags2::SHADER_WRITE.as_raw() ++ | vk::AccessFlags2::COLOR_ATTACHMENT_WRITE.as_raw() ++ | vk::AccessFlags2::DEPTH_STENCIL_ATTACHMENT_WRITE.as_raw() ++ | vk::AccessFlags2::TRANSFER_WRITE.as_raw(), ++ ); ++ const ACCESSES: vk::AccessFlags2 = vk::AccessFlags2::from_raw( ++ WRITES.as_raw() ++ | vk::AccessFlags2::INDEX_READ.as_raw() ++ | vk::AccessFlags2::VERTEX_ATTRIBUTE_READ.as_raw() ++ | vk::AccessFlags2::UNIFORM_READ.as_raw() ++ | vk::AccessFlags2::INPUT_ATTACHMENT_READ.as_raw() ++ | vk::AccessFlags2::SHADER_READ.as_raw() ++ | vk::AccessFlags2::COLOR_ATTACHMENT_READ.as_raw() ++ | vk::AccessFlags2::DEPTH_STENCIL_ATTACHMENT_READ.as_raw() ++ | vk::AccessFlags2::TRANSFER_READ.as_raw(), ++ ); ++ Barrier { ++ src: Access::new(vk::PipelineStageFlags2::ALL_COMMANDS, WRITES), ++ dst: Access::new(vk::PipelineStageFlags2::ALL_COMMANDS, ACCESSES), ++ } ++ } ++ + /// LOCAL (Atlas): from a tile dispatch's copies into its scratch to the dispatch, which reads + /// and writes the scratch (`render/tile_writes.rs`). + pub(super) fn copied_to_dispatch() -> Barrier { +@@ -489,7 +526,12 @@ impl Barrier { + /// + /// `command_buffer` is a command buffer of `device` in the recording state, and no other + /// thread records it. ++ #[track_caller] + pub(super) unsafe fn record(self, device: &Vk, command_buffer: vk::CommandBuffer) { ++ // LOCAL (Atlas) diagnostic: the barrier log (`barrier_log.rs`). ++ if crate::barrier_log::on() { ++ crate::barrier_log::note(std::panic::Location::caller(), "rec", self.log_masks()); ++ } + let masks = self.legacy(); + let barrier = vk::MemoryBarrier::default() + .src_access_mask(masks.src_access) +@@ -508,6 +550,16 @@ impl Barrier { + } + } + ++ /// LOCAL (Atlas) diagnostic: the masks the barrier log keys on (`barrier_log.rs`). ++ pub(crate) fn log_masks(self) -> [u64; 4] { ++ [ ++ self.src.stages.as_raw(), ++ self.dst.stages.as_raw(), ++ self.src.access.as_raw(), ++ self.dst.access.as_raw(), ++ ] ++ } ++ + fn add(&mut self, src: Access, dst: Access) { + self.src = self.src.union(src); + self.dst = self.dst.union(dst); +diff --git a/crates/metal2vulkan-runtime/src/work_counts.rs b/crates/metal2vulkan-runtime/src/work_counts.rs +index 873d951e..f305652c 100644 +--- a/crates/metal2vulkan-runtime/src/work_counts.rs ++++ b/crates/metal2vulkan-runtime/src/work_counts.rs +@@ -1236,6 +1236,32 @@ impl Counters { + self.instance_splits_by[cause as usize].fetch_add(1, Ordering::Relaxed); + } + ++ /// LOCAL (Atlas) diagnostic: the barrier log's line of counts (`barrier_log.rs`), totals so far. ++ pub(crate) fn barrier_log_line(&self) -> String { ++ let l = |a: &AtomicU64| a.load(Ordering::Relaxed); ++ let by: Vec = self.command_buffers_begun_by.iter().map(l).collect(); ++ let ends: Vec = self.group_ends.iter().map(l).collect(); ++ format!( ++ "totals: begun {} by [enc seg cont sec work bar kept int] {:?} submitted {} queue_submits {} infos {} \ ++ grouped {} batches_grouped {} group_ends {:?} kept_bar {} finales folded {} apart {} \ ++ bare_starts {} prefixed_starts {} command_barriers {}", ++ l(&self.command_buffers_begun), ++ by, ++ l(&self.command_buffers_submitted), ++ l(&self.queue_submissions), ++ l(&self.submit_infos), ++ l(&self.grouped_submissions), ++ l(&self.batches_grouped), ++ ends, ++ l(&self.barrier_buffers_kept), ++ l(&self.finales_folded), ++ l(&self.finales_apart), ++ l(&self.bare_starts), ++ l(&self.prefixed_starts), ++ l(&self.command_barriers), ++ ) ++ } ++ + /// A `VkCommandBuffer` was begun (`vkBeginCommandBuffer`) for `what`. + pub(crate) fn command_buffer_begun(&self, what: Begun) { + self.command_buffers_begun.fetch_add(1, Ordering::Relaxed); diff --git a/patches/metal2vulkan/0087-diagnostic-barrier-log-counts-image-to-image-shadow-.patch b/patches/metal2vulkan/0087-diagnostic-barrier-log-counts-image-to-image-shadow-.patch new file mode 100644 index 0000000..abaa138 --- /dev/null +++ b/patches/metal2vulkan/0087-diagnostic-barrier-log-counts-image-to-image-shadow-.patch @@ -0,0 +1,98 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 09:26:03 +0000 +Subject: [PATCH 87/92] diagnostic: barrier log counts image-to-image shadow + copies by size and cause, and shadow counters + +--- + crates/metal2vulkan-runtime/src/shadow.rs | 24 +++++++++++++++++++ + .../metal2vulkan-runtime/src/work_counts.rs | 20 +++++++++++++++- + 2 files changed, 43 insertions(+), 1 deletion(-) + +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index 496cab01..fb6f4f1c 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -529,6 +529,11 @@ impl ShadowStaging { + } + + impl ShadowCopies { ++ /// LOCAL (Atlas) diagnostic: whether the source is an image shadow over a lazy surface's. ++ pub(crate) fn source_is_over(&self) -> bool { ++ matches!(&self.source, ShadowSource::Image { over: Some(_), .. }) ++ } ++ + /// Whether the shadow is a large IOSurface plane's, from which its copies in count apart + /// (`WorkCounts::shadow_texels_first_copied_in_large_surfaces` and kin): a texture over an + /// IOSurface's plane of at least [`LARGE_SURFACE_TEXELS`] texels. +@@ -1412,6 +1417,7 @@ impl ShadowCopy { + /// after what precedes it; the buffer and the shadow are live until it completes, and an + /// image is in the `GENERAL` layout; `back` is this copy's `staged_back`, over the buffer's + /// `VkBuffer` as it is now. ++ #[track_caller] + pub(super) unsafe fn record( + &self, + device: &Vk, +@@ -1537,6 +1543,24 @@ impl ShadowCopy { + }) + .collect() + }; ++ // LOCAL (Atlas) diagnostic: the shadow copies image to image, by size and cause. ++ if crate::barrier_log::on() { ++ let e = regions.first().map(|r| r.extent).unwrap_or_default(); ++ let texels: u64 = regions ++ .iter() ++ .map(|r| r.extent.width as u64 * r.extent.height as u64) ++ .sum(); ++ crate::barrier_log::note( ++ std::panic::Location::caller(), ++ match (into_shadow, c.source_is_over()) { ++ (true, true) => "img-in-over", ++ (true, false) => "img-in", ++ (false, true) => "img-back-over", ++ (false, false) => "img-back", ++ }, ++ [e.width as u64, e.height as u64, regions.len() as u64, texels], ++ ); ++ } + // SAFETY: the caller's contract; the regions lie in both images, which have + // the same shape and formats of one copy unit (planned when the view was made). + unsafe { +diff --git a/crates/metal2vulkan-runtime/src/work_counts.rs b/crates/metal2vulkan-runtime/src/work_counts.rs +index f305652c..3812d519 100644 +--- a/crates/metal2vulkan-runtime/src/work_counts.rs ++++ b/crates/metal2vulkan-runtime/src/work_counts.rs +@@ -1244,7 +1244,9 @@ impl Counters { + format!( + "totals: begun {} by [enc seg cont sec work bar kept int] {:?} submitted {} queue_submits {} infos {} \ + grouped {} batches_grouped {} group_ends {:?} kept_bar {} finales folded {} apart {} \ +- bare_starts {} prefixed_starts {} command_barriers {}", ++ bare_starts {} prefixed_starts {} command_barriers {} | shadow copies {} att_reads {} att_splits {} \ ++ snapshotted {} snap_splits {} own_pixel {} bounded {} via_swizzle {} view_shadows {} att_passes {} \ ++ texels_in {} texels_in_images {} bounded_texels {} texels_back {} back_dispatched {} kept {}", + l(&self.command_buffers_begun), + by, + l(&self.command_buffers_submitted), +@@ -1259,6 +1261,22 @@ impl Counters { + l(&self.bare_starts), + l(&self.prefixed_starts), + l(&self.command_barriers), ++ l(&self.shadow_copies), ++ l(&self.attachment_reads_through_copies), ++ l(&self.attachment_read_splits), ++ l(&self.reading_draws_snapshotted), ++ l(&self.snapshot_inline_splits), ++ l(&self.own_pixel_reads), ++ l(&self.bounded_shadow_copies), ++ l(&self.views_through_swizzle), ++ l(&self.view_shadows), ++ l(&self.shadow_attachment_passes), ++ l(&self.shadow_texels_copied_in), ++ l(&self.shadow_texels_copied_in_images), ++ l(&self.shadow_texels_bounded_copied_in), ++ l(&self.shadow_texels_copied_back), ++ l(&self.shadow_copies_back_dispatched), ++ l(&self.shadow_texels_kept), + ) + } + diff --git a/patches/metal2vulkan/0088-diagnostic-scheduler-log-M2V_SCHED_LOG-commit-call-i.patch b/patches/metal2vulkan/0088-diagnostic-scheduler-log-M2V_SCHED_LOG-commit-call-i.patch new file mode 100644 index 0000000..e1f8843 --- /dev/null +++ b/patches/metal2vulkan/0088-diagnostic-scheduler-log-M2V_SCHED_LOG-commit-call-i.patch @@ -0,0 +1,207 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 10:01:30 +0000 +Subject: [PATCH 88/92] diagnostic: scheduler log (M2V_SCHED_LOG): commit call + -> inbox post -> take -> submit start, per logical queue + +--- + .../src/command_buffer/mod.rs | 9 ++ + .../src/command_buffer/submit.rs | 2 + + crates/metal2vulkan-runtime/src/gpu_time.rs | 5 + + crates/metal2vulkan-runtime/src/lib.rs | 1 + + crates/metal2vulkan-runtime/src/sched_log.rs | 97 +++++++++++++++++++ + .../metal2vulkan-runtime/src/scheduler/mod.rs | 2 + + 6 files changed, 116 insertions(+) + create mode 100644 crates/metal2vulkan-runtime/src/sched_log.rs + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 13e06af0..4181fbb0 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2481,6 +2481,8 @@ impl CommandBuffer { + /// [`commit`](Self::commit) for `call`, or, with `failure`, + /// [`commit_failed`](Self::commit_failed). + fn commit_as(&self, call: &'static str, failure: Option) -> Result<(), Error> { ++ // LOCAL (Atlas) diagnostic: the commit call's start (`sched_log.rs`). ++ crate::sched_log::mark(&self.0.kernel.called); + // Left to wait for the last render encoder's end, where its call left it to the device's + // lowering thread and the thread has not lowered it, or for the commits left so. + if failure.is_none() && self.queue_commit(call)? { +@@ -3426,6 +3428,13 @@ impl CommandBuffer { + } + } + ++impl CommandBufferShared { ++ /// LOCAL (Atlas) diagnostic (`sched_log.rs`): its kernel times, for the scheduler's marks. ++ pub(crate) fn kernel_times(&self) -> &KernelTimes { ++ &self.kernel ++ } ++} ++ + impl CommandBufferShared { + /// The device. + pub(super) fn device(&self) -> &Device { +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index 4d59cc9b..a4ba9028 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -506,6 +506,8 @@ impl CommandBufferShared { + // One a hang's thread made `Scheduled` early began then (`hang.rs`). + if index == 0 && self.kernel.start() == 0 { + self.kernel.begin(); ++ // LOCAL (Atlas) diagnostic: where the commit waited (`sched_log.rs`). ++ crate::sched_log::note(self.queue.id(), queue, &self.kernel); + } + let slot = state + .times +diff --git a/crates/metal2vulkan-runtime/src/gpu_time.rs b/crates/metal2vulkan-runtime/src/gpu_time.rs +index fbe7cee4..14705afd 100644 +--- a/crates/metal2vulkan-runtime/src/gpu_time.rs ++++ b/crates/metal2vulkan-runtime/src/gpu_time.rs +@@ -940,6 +940,11 @@ pub(crate) struct KernelTimes { + start: AtomicU64, + /// The scheduler finished submitting its last batch. + end: AtomicU64, ++ /// LOCAL (Atlas) diagnostic (`sched_log.rs`): the commit call began, its batches were posted ++ /// to the scheduler's inbox, and a pass took them into the order; 0 unless `M2V_SCHED_LOG`. ++ pub(crate) called: AtomicU64, ++ pub(crate) posted: AtomicU64, ++ pub(crate) taken: AtomicU64, + } + + impl KernelTimes { +diff --git a/crates/metal2vulkan-runtime/src/lib.rs b/crates/metal2vulkan-runtime/src/lib.rs +index 4036c1c7..0a0d08df 100644 +--- a/crates/metal2vulkan-runtime/src/lib.rs ++++ b/crates/metal2vulkan-runtime/src/lib.rs +@@ -166,6 +166,7 @@ mod owned_kernel; + // LOCAL (Atlas) diagnostic: the pass timer (M2V_PASS_TIMES). + mod pass_times; + mod barrier_log; // LOCAL (Atlas) diagnostic: M2V_BARRIER_LOG ++mod sched_log; // LOCAL (Atlas) diagnostic: M2V_SCHED_LOG + mod pending_dispatch; + mod pipeline; + mod pixel_format_info; +diff --git a/crates/metal2vulkan-runtime/src/sched_log.rs b/crates/metal2vulkan-runtime/src/sched_log.rs +new file mode 100644 +index 00000000..6be9bec1 +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/sched_log.rs +@@ -0,0 +1,97 @@ ++//! LOCAL (Atlas) diagnostic: the scheduler log, `M2V_SCHED_LOG=1`. Where a command buffer waits ++//! between its commit call and the scheduler's first submission of its work, per logical queue: ++//! the call to the inbox post (`commit` settling ends, queued commits), the post to a pass taking ++//! it into the order (the scheduler's thread busy or asleep), and the take to the submission's ++//! start (event and binary waits, the queue's earlier batches, gates, checks, the pass's cap). ++//! Logged every 10 s for each queue with 20 or more command buffers in the window. Unset (or ++//! `0`), each hook is one check of a `OnceLock`. ++ ++use crate::gpu_time::{host_nanos, KernelTimes}; ++use std::collections::HashMap; ++use std::sync::atomic::{AtomicU64, Ordering}; ++use std::sync::{Mutex, OnceLock}; ++ ++/// Whether the log is on: read once. ++pub(crate) fn on() -> bool { ++ static ON: OnceLock = OnceLock::new(); ++ *ON.get_or_init(|| std::env::var("M2V_SCHED_LOG").is_ok_and(|v| !v.is_empty() && v != "0")) ++} ++ ++/// Stores the host time into `at` where the log is on. ++pub(crate) fn mark(at: &AtomicU64) { ++ if on() { ++ at.store(host_nanos(), Ordering::Release); ++ } ++} ++ ++/// Per logical queue: its `VkQueue` and the waits in microseconds, call->post, post->take, ++/// take->begin. ++type Table = HashMap; 3])>; ++ ++static TABLE: Mutex> = Mutex::new(None); ++static LAST: AtomicU64 = AtomicU64::new(0); ++ ++/// The scheduler began submitting the first batch of a command buffer of logical queue `queue` ++/// on `VkQueue` `vk_queue`, whose marks `kernel` holds. ++pub(crate) fn note(queue: u64, vk_queue: usize, kernel: &KernelTimes) { ++ if !on() { ++ return; ++ } ++ let now = host_nanos(); ++ let called = kernel.called.load(Ordering::Acquire); ++ let posted = kernel.posted.load(Ordering::Acquire); ++ let taken = kernel.taken.load(Ordering::Acquire); ++ let us = |a: u64, b: u64| -> u32 { ++ if a == 0 || b == 0 { ++ u32::MAX ++ } else { ++ (b.saturating_sub(a) / 1000).min(u64::from(u32::MAX - 1)) as u32 ++ } ++ }; ++ let row = [us(called, posted), us(posted, taken), us(taken, now)]; ++ let Ok(mut table) = TABLE.lock() else { return }; ++ let entry = table ++ .get_or_insert_with(HashMap::new) ++ .entry(queue) ++ .or_insert_with(|| (vk_queue, [Vec::new(), Vec::new(), Vec::new()])); ++ for (list, value) in entry.1.iter_mut().zip(row) { ++ if value != u32::MAX { ++ list.push(value); ++ } ++ } ++ let last = LAST.load(Ordering::Relaxed); ++ if last == 0 { ++ LAST.store(now, Ordering::Relaxed); ++ return; ++ } ++ if now.saturating_sub(last) < 10_000_000_000 { ++ return; ++ } ++ LAST.store(now, Ordering::Relaxed); ++ let secs = now.saturating_sub(last) as f64 / 1e9; ++ let Some(rows) = table.take() else { return }; ++ drop(table); ++ let mut rows: Vec<(u64, (usize, [Vec; 3]))> = rows.into_iter().collect(); ++ rows.sort_by_key(|(_, (_, lists))| std::cmp::Reverse(lists[2].len())); ++ let mut text = format!("m2v sched log: {secs:.1} s window\n"); ++ for (queue, (vk_queue, mut lists)) in rows { ++ let n = lists[2].len(); ++ if n < 20 { ++ continue; ++ } ++ text += &format!("m2v sched log: queue {queue} vk {vk_queue} n {n} ({:.0}/s)", n as f64 / secs); ++ for (name, list) in ["call>post", "post>take", "take>begin"].iter().zip(lists.iter_mut()) { ++ list.sort_unstable(); ++ let at = |p: usize| list.get((list.len() * p / 100).min(list.len().saturating_sub(1))).copied().unwrap_or(0) as f64 / 1000.0; ++ text += &format!( ++ " | {name} p50 {:.2} p90 {:.2} p99 {:.2} max {:.2}", ++ at(50), ++ at(90), ++ at(99), ++ list.last().copied().unwrap_or(0) as f64 / 1000.0 ++ ); ++ } ++ text += " ms\n"; ++ } ++ eprint!("{text}"); ++} +diff --git a/crates/metal2vulkan-runtime/src/scheduler/mod.rs b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +index 64de47fe..f3975540 100644 +--- a/crates/metal2vulkan-runtime/src/scheduler/mod.rs ++++ b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +@@ -656,6 +656,7 @@ impl Scheduler { + batches: Vec, + ) { + let shared = &*self.shared; ++ crate::sched_log::mark(&buffer.kernel_times().posted); // LOCAL (Atlas) diagnostic + let committed = Committed { + buffer: buffer.hold(), + queue, +@@ -1082,6 +1083,7 @@ fn take_in(state: &mut State, commits: &mut Vec, spent: &mut Spent) { + } in commits.drain(..) + { + let count = batches.len(); ++ crate::sched_log::mark(&buffer.kernel_times().taken); // LOCAL (Atlas) diagnostic + let State { order, binary, .. } = &mut *state; + let held = batches.into_iter().enumerate().map(|(index, batch)| Held { + buffer: buffer.hold(), diff --git a/patches/metal2vulkan/0089-diagnostic-scheduler-log-held-reasons-per-logical-qu.patch b/patches/metal2vulkan/0089-diagnostic-scheduler-log-held-reasons-per-logical-qu.patch new file mode 100644 index 0000000..b8efca9 --- /dev/null +++ b/patches/metal2vulkan/0089-diagnostic-scheduler-log-held-reasons-per-logical-qu.patch @@ -0,0 +1,144 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 10:20:47 +0000 +Subject: [PATCH 89/92] diagnostic: scheduler log: held reasons per logical + queue (host-held event waits apart) + +--- + .../src/command_buffer/mod.rs | 5 +++ + crates/metal2vulkan-runtime/src/sched_log.rs | 39 +++++++++++++++++++ + .../metal2vulkan-runtime/src/scheduler/mod.rs | 13 +++++++ + 3 files changed, 57 insertions(+) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 4181fbb0..69b0cc92 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -3433,6 +3433,11 @@ impl CommandBufferShared { + pub(crate) fn kernel_times(&self) -> &KernelTimes { + &self.kernel + } ++ ++ /// LOCAL (Atlas) diagnostic (`sched_log.rs`): its logical queue's identity. ++ pub(crate) fn queue_id(&self) -> u64 { ++ self.queue.id() ++ } + } + + impl CommandBufferShared { +diff --git a/crates/metal2vulkan-runtime/src/sched_log.rs b/crates/metal2vulkan-runtime/src/sched_log.rs +index 6be9bec1..8cc8fc35 100644 +--- a/crates/metal2vulkan-runtime/src/sched_log.rs ++++ b/crates/metal2vulkan-runtime/src/sched_log.rs +@@ -29,6 +29,31 @@ pub(crate) fn mark(at: &AtomicU64) { + type Table = HashMap; 3])>; + + static TABLE: Mutex> = Mutex::new(None); ++ ++/// Per logical queue, per reason bit (`scheduler::HeldFor` as bits, bit 7 a host-held event ++/// wait): batches held, their held nanoseconds, the most. ++type Holds = HashMap; ++ ++static HOLDS: Mutex> = Mutex::new(None); ++ ++/// A batch of logical queue `queue` held for `reasons` (bits) went after `nanos`. ++pub(crate) fn held(queue: u64, reasons: u8, nanos: u64) { ++ if !on() { ++ return; ++ } ++ let Ok(mut holds) = HOLDS.lock() else { return }; ++ let row = holds.get_or_insert_with(HashMap::new).entry(queue).or_insert([(0, 0, 0); 8]); ++ for (bit, cell) in row.iter_mut().enumerate() { ++ if reasons & (1 << bit) != 0 { ++ cell.0 += 1; ++ cell.1 += nanos; ++ cell.2 = cell.2.max(nanos); ++ } ++ } ++} ++ ++const REASONS: [&str; 8] = ++ ["binary", "reencode", "decode", "hoststep", "check", "event", "earlier", "event-host"]; + static LAST: AtomicU64 = AtomicU64::new(0); + + /// The scheduler began submitting the first batch of a command buffer of logical queue `queue` +@@ -73,6 +98,7 @@ pub(crate) fn note(queue: u64, vk_queue: usize, kernel: &KernelTimes) { + drop(table); + let mut rows: Vec<(u64, (usize, [Vec; 3]))> = rows.into_iter().collect(); + rows.sort_by_key(|(_, (_, lists))| std::cmp::Reverse(lists[2].len())); ++ let holds = HOLDS.lock().ok().and_then(|mut h| h.take()); + let mut text = format!("m2v sched log: {secs:.1} s window\n"); + for (queue, (vk_queue, mut lists)) in rows { + let n = lists[2].len(); +@@ -92,6 +118,19 @@ pub(crate) fn note(queue: u64, vk_queue: usize, kernel: &KernelTimes) { + ); + } + text += " ms\n"; ++ if let Some(row) = holds.as_ref().and_then(|h| h.get(&queue)) { ++ text += &format!("m2v sched log: held for (n, total ms/s, max ms) on queue {queue}:"); ++ for (name, (n, total, most)) in REASONS.iter().zip(row) { ++ if *n > 0 { ++ text += &format!( ++ " {name} {n} {:.1} {:.2};", ++ *total as f64 / 1e6 / secs, ++ *most as f64 / 1e6 ++ ); ++ } ++ } ++ text += "\n"; ++ } + } + eprint!("{text}"); + } +diff --git a/crates/metal2vulkan-runtime/src/scheduler/mod.rs b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +index f3975540..93cd9190 100644 +--- a/crates/metal2vulkan-runtime/src/scheduler/mod.rs ++++ b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +@@ -529,6 +529,9 @@ struct Held { + /// The host nanoseconds at the first look of the scheduler's that found it held + /// ([`held_for`](Self::held_for)), until it goes ([`hold_ended`](Self::hold_ended)). + held_since: Option, ++ /// LOCAL (Atlas) diagnostic (`sched_log.rs`): each `HeldFor` a look found it held for, as a ++ /// bit; bit 7, an event wait held on the host for a submitted GPU signal to land. ++ reasons: u8, + } + + impl Held { +@@ -536,6 +539,7 @@ impl Held { + /// noted at the first look that found it held at all, whatever the reason then. Nothing is + /// read from the host's clock while nothing waits. + fn held_for(&mut self, reason: HeldFor) { ++ self.reasons |= 1 << (reason as u8); // LOCAL (Atlas) diagnostic + (self.buffer.device().shared().counts).head_held_for(reason); + if self.held_since.is_none() { + self.held_since = Some(gpu_time::host_nanos()); +@@ -548,6 +552,8 @@ impl Held { + if let Some(since) = self.held_since.take() { + let nanos = gpu_time::host_nanos().saturating_sub(since); + (self.buffer.device().shared().counts).batch_hold_ended(nanos); ++ // LOCAL (Atlas) diagnostic: per queue, by reason (`sched_log.rs`). ++ crate::sched_log::held(self.buffer.queue_id(), self.reasons, nanos); + } + } + } +@@ -1098,6 +1104,7 @@ fn take_in(state: &mut State, commits: &mut Vec, spent: &mut Spent) { + hastened: false, + counted_held: false, + held_since: None, ++ reasons: 0, + }); + order.push(queue, held); + spent.push(buffer); +@@ -1232,6 +1239,12 @@ fn submit_ready( + let waits = event_waits(held); + if waits.is_none() { + held.held_for(HeldFor::EventValue); ++ // LOCAL (Atlas) diagnostic: a wait held on the host for a GPU signal to land. ++ if crate::sched_log::on() ++ && (held.waits.iter()).any(|(event, value)| event.held_for_the_host(*value)) ++ { ++ held.reasons |= 1 << 7; ++ } + } + waits + }); diff --git a/patches/metal2vulkan/0090-live-checks_first-1-a-left-submission-check-runs-bef.patch b/patches/metal2vulkan/0090-live-checks_first-1-a-left-submission-check-runs-bef.patch new file mode 100644 index 0000000..d776256 --- /dev/null +++ b/patches/metal2vulkan/0090-live-checks_first-1-a-left-submission-check-runs-bef.patch @@ -0,0 +1,44 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 10:52:42 +0000 +Subject: [PATCH 90/92] live checks_first=1: a left submission check runs + before the thread drains commits that keep coming (desktop batches starved + behind a Simulator's commits) + +--- + crates/metal2vulkan-runtime/src/scheduler/left.rs | 15 ++++++++++++++- + 1 file changed, 14 insertions(+), 1 deletion(-) + +diff --git a/crates/metal2vulkan-runtime/src/scheduler/left.rs b/crates/metal2vulkan-runtime/src/scheduler/left.rs +index 366e43e0..801fc455 100644 +--- a/crates/metal2vulkan-runtime/src/scheduler/left.rs ++++ b/crates/metal2vulkan-runtime/src/scheduler/left.rs +@@ -64,6 +64,11 @@ impl Left { + self.ordered = self.work.len(); + } + ++ /// LOCAL (Atlas): the oldest piece's kind. ++ fn front_kind(&self) -> Option { ++ self.work.front().map(|(kind, _)| *kind) ++ } ++ + /// The oldest piece, taken out. + fn pop(&mut self) -> Option<(Kind, W)> { + let work = self.work.pop_front()?; +@@ -97,7 +102,15 @@ pub(crate) fn work_through( + while !left.is_empty() { + // A commit left to this thread meanwhile goes before the next piece of work, as does one + // left to it during the round that took that one in. +- while inbox.holds_commits() { ++ // LOCAL (Atlas), live `checks_first=1`: not before a submission check. Under a program ++ // that commits without pause (a Simulator, ~9k command buffers a second), the commits ++ // kept coming during every round, so a checked batch of another queue (the Mac desktop's, ++ // every one of whose command buffers Reims checks) waited 1 to 5 ms for a round without ++ // a commit (`sched_log.rs`, 2026-11-09). A check is a short call of the program's; the ++ // commits it delays wait for it alone, and the next round takes them in with the batch. ++ let checks_first = ++ left.front_kind() == Some(Kind::Check) && crate::live::on("checks_first", false); ++ while !checks_first && inbox.holds_commits() { + if !round(left) { + return None; + } diff --git a/patches/metal2vulkan/0091-diagnostic-scheduler-log-a-checked-head-s-landing-an.patch b/patches/metal2vulkan/0091-diagnostic-scheduler-log-a-checked-head-s-landing-an.patch new file mode 100644 index 0000000..e1565a4 --- /dev/null +++ b/patches/metal2vulkan/0091-diagnostic-scheduler-log-a-checked-head-s-landing-an.patch @@ -0,0 +1,89 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 11:26:16 +0000 +Subject: [PATCH 91/92] diagnostic: scheduler log: a checked head's landing and + unknown event waits apart + +--- + crates/metal2vulkan-runtime/src/sched_log.rs | 12 +++++++----- + crates/metal2vulkan-runtime/src/scheduler/mod.rs | 10 +++++++--- + 2 files changed, 14 insertions(+), 8 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/sched_log.rs b/crates/metal2vulkan-runtime/src/sched_log.rs +index 8cc8fc35..0cd4bf92 100644 +--- a/crates/metal2vulkan-runtime/src/sched_log.rs ++++ b/crates/metal2vulkan-runtime/src/sched_log.rs +@@ -32,17 +32,17 @@ static TABLE: Mutex> = Mutex::new(None); + + /// Per logical queue, per reason bit (`scheduler::HeldFor` as bits, bit 7 a host-held event + /// wait): batches held, their held nanoseconds, the most. +-type Holds = HashMap; ++type Holds = HashMap; + + static HOLDS: Mutex> = Mutex::new(None); + + /// A batch of logical queue `queue` held for `reasons` (bits) went after `nanos`. +-pub(crate) fn held(queue: u64, reasons: u8, nanos: u64) { ++pub(crate) fn held(queue: u64, reasons: u16, nanos: u64) { + if !on() { + return; + } + let Ok(mut holds) = HOLDS.lock() else { return }; +- let row = holds.get_or_insert_with(HashMap::new).entry(queue).or_insert([(0, 0, 0); 8]); ++ let row = holds.get_or_insert_with(HashMap::new).entry(queue).or_insert([(0, 0, 0); 10]); + for (bit, cell) in row.iter_mut().enumerate() { + if reasons & (1 << bit) != 0 { + cell.0 += 1; +@@ -52,8 +52,10 @@ pub(crate) fn held(queue: u64, reasons: u8, nanos: u64) { + } + } + +-const REASONS: [&str; 8] = +- ["binary", "reencode", "decode", "hoststep", "check", "event", "earlier", "event-host"]; ++const REASONS: [&str; 10] = [ ++ "binary", "reencode", "decode", "hoststep", "check", "event", "earlier", "event-host", ++ "check-landing", "check-unknown", ++]; + static LAST: AtomicU64 = AtomicU64::new(0); + + /// The scheduler began submitting the first batch of a command buffer of logical queue `queue` +diff --git a/crates/metal2vulkan-runtime/src/scheduler/mod.rs b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +index 93cd9190..b5e50b3d 100644 +--- a/crates/metal2vulkan-runtime/src/scheduler/mod.rs ++++ b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +@@ -531,7 +531,7 @@ struct Held { + held_since: Option, + /// LOCAL (Atlas) diagnostic (`sched_log.rs`): each `HeldFor` a look found it held for, as a + /// bit; bit 7, an event wait held on the host for a submitted GPU signal to land. +- reasons: u8, ++ reasons: u16, + } + + impl Held { +@@ -539,7 +539,7 @@ impl Held { + /// noted at the first look that found it held at all, whatever the reason then. Nothing is + /// read from the host's clock while nothing waits. + fn held_for(&mut self, reason: HeldFor) { +- self.reasons |= 1 << (reason as u8); // LOCAL (Atlas) diagnostic ++ self.reasons |= 1 << (reason as u16); // LOCAL (Atlas) diagnostic + (self.buffer.device().shared().counts).head_held_for(reason); + if self.held_since.is_none() { + self.held_since = Some(gpu_time::host_nanos()); +@@ -1740,12 +1740,16 @@ fn check_heads(state: &mut State, behind: &[u64], left: &mut Left, awaits: + return !due && goes(held); + } + match landing_waits(held) { +- None => return false, ++ None => { ++ held.reasons |= 1 << 9; // LOCAL (Atlas) diagnostic: a value no signal reaches yet ++ return false; ++ } + Some(waits) if !waits.is_empty() && !held.buffer.device().is_lost() => { + awaits.events(&held.waits); + fence(Ordering::SeqCst); + if landing_waits(held).is_some_and(|waits| !waits.is_empty()) { + (held.buffer.device().shared().counts).head_awaiting_check_signals(); ++ held.reasons |= 1 << 8; // LOCAL (Atlas) diagnostic: a GPU signal to land + landing = true; + return false; + } diff --git a/patches/metal2vulkan/0092-live-inline_checks-1-run-a-batch-s-submission-checks.patch b/patches/metal2vulkan/0092-live-inline_checks-1-run-a-batch-s-submission-checks.patch new file mode 100644 index 0000000..0aa1fb2 --- /dev/null +++ b/patches/metal2vulkan/0092-live-inline_checks-1-run-a-batch-s-submission-checks.patch @@ -0,0 +1,55 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 11:52:29 +0000 +Subject: [PATCH 92/92] live inline_checks=1: run a batch's submission checks + in the pass, under the scheduler's lock (Reims's checks), so a chained + command buffer goes in the pass that finds it ready + +--- + .../metal2vulkan-runtime/src/scheduler/mod.rs | 21 +++++++++++++++++-- + 1 file changed, 19 insertions(+), 2 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/scheduler/mod.rs b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +index b5e50b3d..8fb340aa 100644 +--- a/crates/metal2vulkan-runtime/src/scheduler/mod.rs ++++ b/crates/metal2vulkan-runtime/src/scheduler/mod.rs +@@ -1731,7 +1731,7 @@ fn check_heads(state: &mut State, behind: &[u64], left: &mut Left, awaits: + return false; + } + if due { +- left.push(Kind::Check, (held.buffer.hold(), held.index)); ++ check_now_or_leave(held, left); + } + return true; + } +@@ -1756,12 +1756,29 @@ fn check_heads(state: &mut State, behind: &[u64], left: &mut Left, awaits: + } + Some(_) => {} + } +- left.push(Kind::Check, (held.buffer.hold(), held.index)); ++ check_now_or_leave(held, left); + goes(held) + }); + landing + } + ++/// LOCAL (Atlas), live `inline_checks=1`: run a batch's submission checks here, under the ++/// scheduler's lock, inside the pass's section, so the batch goes in this pass, not in the next ++/// round after the check ran off the lock. Each command buffer's events chain on the one before ++/// (Reims's FIFO order, `encodeSignalEventScheduled`), so each checked link of the chain cost a ++/// round, 1-4 ms under a Simulator's commits (`sched_log.rs`, 2026-11-09). Only for a program ++/// whose check takes no lock of the runtime's but a command buffer's, which the lock order ++/// allows after the scheduler's (Reims's: its own registry and command-buffer locks, then ++/// `Submission::uses`), and holds no command buffer's last handle (the scheduler holds the ++/// batch's). Off: left to the thread, as upstream. ++fn check_now_or_leave(held: &Held, left: &mut Left) { ++ if crate::live::on("inline_checks", false) { ++ held.buffer.run_checks(held.index); ++ } else { ++ left.push(Kind::Check, (held.buffer.hold(), held.index)); ++ } ++} ++ + /// The waits of `held`, a line's head, for a shared event's value that a submitted GPU signal + /// reaches: each chained on the `VkQueue` before and is held on the host now, so each cost moved + /// (`WorkCounts::shared_event_waits_held`, `event_timeline.rs`). Counted once per batch, at the diff --git a/patches/qemu/0010-apple-gfx-pci-MSI-through-a-KVM-irqfd-x-irqfd-live-s.patch b/patches/qemu/0010-apple-gfx-pci-MSI-through-a-KVM-irqfd-x-irqfd-live-s.patch new file mode 100644 index 0000000..0fa6670 --- /dev/null +++ b/patches/qemu/0010-apple-gfx-pci-MSI-through-a-KVM-irqfd-x-irqfd-live-s.patch @@ -0,0 +1,333 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 15:15:21 +0000 +Subject: [PATCH 10/13] apple-gfx-pci: MSI through a KVM irqfd (x-irqfd, live + switch) + +LOCAL (Atlas), DEEP-VIRT experiment. A raise scheduled a main-loop BH that took +the BQL and called msi_notify: per interrupt a main-loop wakeup, its run-queue +wait, the BQL and a KVM_SIGNAL_MSI ioctl. With x-irqfd the library thread writes +an eventfd that KVM routes to the vector (posted interrupt), as vfio-pci and +ivshmem do. Default off (APPLE_GFX_IRQFD=1 turns it on at start; qom-set +toggles it live). While disarmed the eventfd is watched by the main loop so a +raise racing a disarm still reaches msi_notify. x-irq-stats counts both paths. + +Co-Authored-By: Claude Opus 5.5 +--- + hw/display/apple-gfx-pci.c | 208 +++++++++++++++++++++++++++++++++++++ + 1 file changed, 208 insertions(+) + +diff --git a/hw/display/apple-gfx-pci.c b/hw/display/apple-gfx-pci.c +index 34228552f3..66eedde9b8 100644 +--- a/hw/display/apple-gfx-pci.c ++++ b/hw/display/apple-gfx-pci.c +@@ -29,15 +29,155 @@ + #include "hw/core/boards.h" + #include "hw/pci/pci_device.h" + #include "hw/pci/msi.h" ++#include "qemu/atomic.h" ++#include "qemu/event_notifier.h" ++#include "qemu/error-report.h" ++#include "system/accel-irq.h" ++#include "system/kvm.h" + #include "apple-gfx.h" + #include "trace.h" + ++/* ++ * LOCAL (Atlas): the MSI through a KVM irqfd. ++ * ++ * A raise used to schedule a BH in the main loop, which took the BQL and ++ * called msi_notify: per interrupt an aio_notify wakeup of the main loop ++ * thread, its run-queue wait, the BQL and a KVM_SIGNAL_MSI ioctl, before the ++ * vCPU saw it. With the irqfd armed, the library's thread writes an eventfd ++ * that KVM routes to the vector itself (a posted interrupt with APICv), as ++ * vfio-pci and ivshmem do: no main loop, no BQL, no allocation. ++ * ++ * - x-irqfd (bool, default APPLE_GFX_IRQFD=1 in the environment, else off) can ++ * be set live with qom-set; off is the BH path exactly as before. ++ * - It arms while MSI is enabled (each config write re-syncs, and an armed ++ * route follows the guest's address and data) and only vector 0 (the ++ * devices have one). ++ * - While disarmed, the eventfd stays watched by the main loop, which turns ++ * a stray write (a raise that read "armed" just before a disarm) into an ++ * msi_notify: no interrupt is lost at a switch. ++ * - x-irq-stats: "irqfd= bh= armed=<0|1>". ++ */ ++typedef struct AppleGFXPCIIrq { ++ PCIDevice *dev; ++ bool wanted; /* x-irqfd */ ++ bool have_notifier; ++ bool attached; /* BQL: the irqfd is in KVM */ ++ int armed; /* atomic: raisers write the eventfd */ ++ int virq; ++ EventNotifier notifier; ++ uint64_t via_irqfd; /* atomic */ ++ uint64_t via_bh; /* atomic */ ++} AppleGFXPCIIrq; ++ ++static void apple_gfx_pci_irq_stray(EventNotifier *n) ++{ ++ AppleGFXPCIIrq *irq = container_of(n, AppleGFXPCIIrq, notifier); ++ ++ if (event_notifier_test_and_clear(n) && msi_enabled(irq->dev)) { ++ msi_notify(irq->dev, 0); ++ } ++} ++ ++/* Under the BQL: arm, follow the route, or disarm, as wanted and MSI say. */ ++static void apple_gfx_pci_irq_sync(AppleGFXPCIIrq *irq) ++{ ++ bool want = irq->wanted && irq->have_notifier && kvm_enabled() && ++ accel_msi_via_irqfd_enabled() && msi_enabled(irq->dev); ++ AccelRouteChange c; ++ int virq; ++ ++ if (want && !irq->attached) { ++ c = accel_irqchip_begin_route_changes(); ++ virq = accel_irqchip_add_msi_route(&c, 0, irq->dev); ++ if (virq < 0) { ++ warn_report_once("apple-gfx: no MSI route for the irqfd (%d); " ++ "interrupts stay on the main loop", virq); ++ return; ++ } ++ accel_irqchip_commit_route_changes(&c); ++ event_notifier_set_handler(&irq->notifier, NULL); ++ if (accel_irqchip_add_irqfd_notifier_gsi(&irq->notifier, NULL, ++ virq) < 0) { ++ warn_report_once("apple-gfx: irqfd refused; interrupts stay on " ++ "the main loop"); ++ accel_irqchip_release_virq(virq); ++ event_notifier_set_handler(&irq->notifier, ++ apple_gfx_pci_irq_stray); ++ return; ++ } ++ irq->virq = virq; ++ irq->attached = true; ++ qatomic_store_release(&irq->armed, 1); ++ info_report("apple-gfx: %s: MSI through irqfd (virq %d)", ++ object_get_typename(OBJECT(irq->dev)), virq); ++ } else if (want) { ++ accel_irqchip_update_msi_route(irq->virq, ++ msi_get_message(irq->dev, 0), irq->dev); ++ accel_irqchip_commit_routes(); ++ } else if (irq->attached) { ++ qatomic_set(&irq->armed, 0); ++ smp_mb(); ++ accel_irqchip_remove_irqfd_notifier_gsi(&irq->notifier, irq->virq); ++ accel_irqchip_release_virq(irq->virq); ++ irq->virq = -1; ++ irq->attached = false; ++ event_notifier_set_handler(&irq->notifier, apple_gfx_pci_irq_stray); ++ apple_gfx_pci_irq_stray(&irq->notifier); ++ info_report("apple-gfx: %s: MSI back on the main loop", ++ object_get_typename(OBJECT(irq->dev))); ++ } ++} ++ ++/* On any thread: true when the raise went to the irqfd. */ ++static bool apple_gfx_pci_irq_fast(AppleGFXPCIIrq *irq, uint32_t vector) ++{ ++ if (vector == 0 && qatomic_load_acquire(&irq->armed) && ++ msi_enabled(irq->dev)) { ++ event_notifier_set(&irq->notifier); ++ qatomic_inc(&irq->via_irqfd); ++ return true; ++ } ++ qatomic_inc(&irq->via_bh); ++ return false; ++} ++ ++static bool apple_gfx_pci_irq_get_wanted(Object *obj, Error **errp); ++static void apple_gfx_pci_irq_set_wanted(Object *obj, bool value, ++ Error **errp); ++static char *apple_gfx_pci_irq_get_stats(Object *obj, Error **errp); ++ ++static void apple_gfx_pci_irq_init(Object *obj, AppleGFXPCIIrq *irq) ++{ ++ const char *env = getenv("APPLE_GFX_IRQFD"); ++ ++ irq->dev = PCI_DEVICE(obj); ++ irq->virq = -1; ++ irq->wanted = env && strcmp(env, "1") == 0; ++ object_property_add_bool(obj, "x-irqfd", apple_gfx_pci_irq_get_wanted, ++ apple_gfx_pci_irq_set_wanted); ++ object_property_add_str(obj, "x-irq-stats", apple_gfx_pci_irq_get_stats, ++ NULL); ++} ++ ++/* In realize, after msi_init. */ ++static void apple_gfx_pci_irq_realize(AppleGFXPCIIrq *irq) ++{ ++ if (event_notifier_init(&irq->notifier, 0) < 0) { ++ warn_report("apple-gfx: no eventfd for the irqfd"); ++ return; ++ } ++ irq->have_notifier = true; ++ event_notifier_set_handler(&irq->notifier, apple_gfx_pci_irq_stray); ++ apple_gfx_pci_irq_sync(irq); ++} ++ + OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXPCIState, APPLE_GFX_PCI) + + struct AppleGFXPCIState { + PCIDevice parent_obj; + + AppleGFXState common; ++ AppleGFXPCIIrq irq; + bool efi_fb_bar; + MemoryRegion efi_fb_bar_mr; /* BAR 2 */ + MemoryRegion efi_fb; /* in BAR 2: an alias of the machine's RAM */ +@@ -56,6 +196,7 @@ static void apple_gfx_pci_init(Object *obj) + AppleGFXPCIState *s = APPLE_GFX_PCI(obj); + + apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI); ++ apple_gfx_pci_irq_init(obj, &s->irq); + } + + static void apple_gfx_pci_finalize(Object *obj) +@@ -91,6 +232,10 @@ static void apple_gfx_pci_raise_interrupt_cb(void *ctx, uint32_t vector) + AppleGFXPCIInterruptJob *job; + + trace_apple_gfx_raise_irq(vector); ++ if (apple_gfx_pci_irq_fast(&container_of(dev, AppleGFXPCIState, ++ parent_obj)->irq, vector)) { ++ return; ++ } + job = g_new0(AppleGFXPCIInterruptJob, 1); + job->device = dev; + job->vector = vector; +@@ -190,6 +335,7 @@ static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp) + if (ret != 0) { + return; + } ++ apple_gfx_pci_irq_realize(&s->irq); + + apple_gfx_common_realize(&s->common, DEVICE(dev), &config, errp); + } +@@ -202,6 +348,15 @@ static void apple_gfx_pci_reset(Object *obj, ResetType type) + if (s->common.mmio_written) { + memory_region_set_enabled(&s->efi_fb, false); + } ++ apple_gfx_pci_irq_sync(&s->irq); ++} ++ ++/* LOCAL (Atlas): MSI's enable, address and data reach the irqfd. */ ++static void apple_gfx_pci_write_config(PCIDevice *dev, uint32_t addr, ++ uint32_t val, int len) ++{ ++ pci_default_write_config(dev, addr, val, len); ++ apple_gfx_pci_irq_sync(&APPLE_GFX_PCI(dev)->irq); + } + + static const Property apple_gfx_pci_properties[] = { +@@ -237,6 +392,7 @@ static void apple_gfx_pci_class_init(ObjectClass *klass, const void *data) + pci->class_id = PCI_CLASS_DISPLAY_OTHER; + pci->romfile = "ReimsVgpu.rom"; + pci->realize = apple_gfx_pci_realize; ++ pci->config_write = apple_gfx_pci_write_config; + + device_class_set_props(dc, apple_gfx_pci_properties); + } +@@ -262,8 +418,44 @@ struct AppleGFXIOSurfacePCIState { + AppleGFXPCIState *gpu; + MemoryRegion iomem; /* BAR 0 */ + RvgIOSurfaceDevice *pgiosfc; ++ AppleGFXPCIIrq irq; + }; + ++static AppleGFXPCIIrq *apple_gfx_pci_irq_of(Object *obj) ++{ ++ if (object_dynamic_cast(obj, TYPE_APPLE_GFX_PCI)) { ++ return &APPLE_GFX_PCI(obj)->irq; ++ } ++ return &APPLE_GFX_IOSURFACE_PCI(obj)->irq; ++} ++ ++static bool apple_gfx_pci_irq_get_wanted(Object *obj, Error **errp) ++{ ++ return apple_gfx_pci_irq_of(obj)->wanted; ++} ++ ++/* qom-set runs under the BQL. */ ++static void apple_gfx_pci_irq_set_wanted(Object *obj, bool value, ++ Error **errp) ++{ ++ AppleGFXPCIIrq *irq = apple_gfx_pci_irq_of(obj); ++ ++ irq->wanted = value; ++ if (DEVICE(obj)->realized) { ++ apple_gfx_pci_irq_sync(irq); ++ } ++} ++ ++static char *apple_gfx_pci_irq_get_stats(Object *obj, Error **errp) ++{ ++ AppleGFXPCIIrq *irq = apple_gfx_pci_irq_of(obj); ++ ++ return g_strdup_printf("irqfd=%" PRIu64 " bh=%" PRIu64 " armed=%d", ++ qatomic_read(&irq->via_irqfd), ++ qatomic_read(&irq->via_bh), ++ qatomic_read(&irq->armed)); ++} ++ + /* + * The registers are 64 bits wide, and an 8-byte access passes whole: the + * ring's and the descriptor table's bases are guest physical addresses, which +@@ -312,6 +504,10 @@ static void apple_gfx_iosurface_pci_raise_interrupt_cb(void *ctx, + AppleGFXPCIInterruptJob *job; + + trace_apple_gfx_iosfc_raise_irq(vector); ++ if (apple_gfx_pci_irq_fast(&container_of(ctx, AppleGFXIOSurfacePCIState, ++ parent_obj)->irq, vector)) { ++ return; ++ } + job = g_new0(AppleGFXPCIInterruptJob, 1); + job->device = ctx; + job->vector = vector; +@@ -380,6 +576,7 @@ static void apple_gfx_iosurface_pci_realize(PCIDevice *dev, Error **errp) + if (ret != 0) { + return; + } ++ apple_gfx_pci_irq_realize(&s->irq); + + /* + * The guest's RAM, as the gpu's device has it. On Linux the library maps +@@ -419,6 +616,7 @@ static void apple_gfx_iosurface_pci_reset(Object *obj, ResetType type) + 0x1018, /* the write counter */ + }; + ++ apple_gfx_pci_irq_sync(&s->irq); + if (!s->pgiosfc) { + return; + } +@@ -427,6 +625,14 @@ static void apple_gfx_iosurface_pci_reset(Object *obj, ResetType type) + } + } + ++static void apple_gfx_iosurface_pci_write_config(PCIDevice *dev, ++ uint32_t addr, uint32_t val, ++ int len) ++{ ++ pci_default_write_config(dev, addr, val, len); ++ apple_gfx_pci_irq_sync(&APPLE_GFX_IOSURFACE_PCI(dev)->irq); ++} ++ + static void apple_gfx_iosurface_pci_init(Object *obj) + { + AppleGFXIOSurfacePCIState *s = APPLE_GFX_IOSURFACE_PCI(obj); +@@ -434,6 +640,7 @@ static void apple_gfx_iosurface_pci_init(Object *obj) + memory_region_init_io(&s->iomem, obj, &apple_gfx_iosurface_pci_ops, s, + TYPE_APPLE_GFX_IOSURFACE_PCI, + APPLE_GFX_IOSURFACE_PCI_MMIO_SIZE); ++ apple_gfx_pci_irq_init(obj, &s->irq); + } + + static const Property apple_gfx_iosurface_pci_properties[] = { +@@ -463,6 +670,7 @@ static void apple_gfx_iosurface_pci_class_init(ObjectClass *klass, + pci->device_id = APPLE_GFX_IOSURFACE_PCI_DEVICE_ID; + pci->class_id = PCI_CLASS_SYSTEM_OTHER; + pci->realize = apple_gfx_iosurface_pci_realize; ++ pci->config_write = apple_gfx_iosurface_pci_write_config; + + device_class_set_props(dc, apple_gfx_iosurface_pci_properties); + } diff --git a/patches/qemu/0011-apple-gfx-pci-x-halt-poll-ns-a-live-per-VM-KVM_CAP_H.patch b/patches/qemu/0011-apple-gfx-pci-x-halt-poll-ns-a-live-per-VM-KVM_CAP_H.patch new file mode 100644 index 0000000..a6bf479 --- /dev/null +++ b/patches/qemu/0011-apple-gfx-pci-x-halt-poll-ns-a-live-per-VM-KVM_CAP_H.patch @@ -0,0 +1,134 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 15:17:12 +0000 +Subject: [PATCH 11/13] apple-gfx-pci: x-halt-poll-ns, a live per-VM + KVM_CAP_HALT_POLL knob + +LOCAL (Atlas), DEEP-VIRT experiment (on top of the irqfd branch). qom-set on +the gfx device sets this VM's halt-poll ceiling, so a halt-poll A/B needs no +change of the host's kvm.halt_poll_ns. Unset, nothing changes. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/kvm-all.c | 8 ++++++++ + accel/stubs/kvm-stub.c | 5 +++++ + hw/display/apple-gfx-pci.c | 42 ++++++++++++++++++++++++++++++++++++++ + include/system/kvm.h | 7 +++++++ + 4 files changed, 62 insertions(+) + +diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c +index 7a84a3e33b..1777860eaf 100644 +--- a/accel/kvm/kvm-all.c ++++ b/accel/kvm/kvm-all.c +@@ -1510,6 +1510,14 @@ int kvm_check_extension(KVMState *s, unsigned int extension) + return ret; + } + ++int kvm_set_halt_poll_ns(uint64_t ns) ++{ ++ if (!kvm_state || !kvm_vm_check_extension(kvm_state, KVM_CAP_HALT_POLL)) { ++ return -ENOSYS; ++ } ++ return kvm_vm_enable_cap(kvm_state, KVM_CAP_HALT_POLL, 0, ns); ++} ++ + int kvm_vm_check_extension(KVMState *s, unsigned int extension) + { + int ret; +diff --git a/accel/stubs/kvm-stub.c b/accel/stubs/kvm-stub.c +index 32b4b07403..15dfff140e 100644 +--- a/accel/stubs/kvm-stub.c ++++ b/accel/stubs/kvm-stub.c +@@ -143,3 +143,8 @@ int kvm_create_guest_memfd(uint64_t size, uint64_t flags, Error **errp) + { + return -ENOSYS; + } ++ ++int kvm_set_halt_poll_ns(uint64_t ns) ++{ ++ return -ENOSYS; ++} +diff --git a/hw/display/apple-gfx-pci.c b/hw/display/apple-gfx-pci.c +index 66eedde9b8..e2f1da1faa 100644 +--- a/hw/display/apple-gfx-pci.c ++++ b/hw/display/apple-gfx-pci.c +@@ -32,6 +32,7 @@ + #include "qemu/atomic.h" + #include "qemu/event_notifier.h" + #include "qemu/error-report.h" ++#include "qapi/visitor.h" + #include "system/accel-irq.h" + #include "system/kvm.h" + #include "apple-gfx.h" +@@ -191,12 +192,53 @@ struct AppleGFXPCIState { + #define APPLE_GFX_PCI_PAGE_SHIFT 14 + #endif + ++/* ++ * LOCAL (Atlas), DEEP-VIRT experiment: x-halt-poll-ns sets KVM_CAP_HALT_POLL ++ * for this VM (qom-set, live; -1 reads until set: the kvm module's ++ * halt_poll_ns applies), so a halt-poll A/B needs no host-wide change. The ++ * device carries it because the accelerator is not in the QOM tree. ++ */ ++static int64_t apple_gfx_halt_poll_ns = -1; ++ ++static void apple_gfx_pci_get_halt_poll(Object *obj, Visitor *v, ++ const char *name, void *opaque, ++ Error **errp) ++{ ++ visit_type_int64(v, name, &apple_gfx_halt_poll_ns, errp); ++} ++ ++static void apple_gfx_pci_set_halt_poll(Object *obj, Visitor *v, ++ const char *name, void *opaque, ++ Error **errp) ++{ ++ int64_t ns; ++ int ret; ++ ++ if (!visit_type_int64(v, name, &ns, errp)) { ++ return; ++ } ++ if (ns < 0 || ns > 10 * 1000 * 1000) { ++ error_setg(errp, "x-halt-poll-ns: 0 to 10000000"); ++ return; ++ } ++ ret = kvm_set_halt_poll_ns(ns); ++ if (ret < 0) { ++ error_setg_errno(errp, -ret, "KVM_CAP_HALT_POLL"); ++ return; ++ } ++ apple_gfx_halt_poll_ns = ns; ++ info_report("apple-gfx: KVM halt polling at most %" PRId64 " ns", ns); ++} ++ + static void apple_gfx_pci_init(Object *obj) + { + AppleGFXPCIState *s = APPLE_GFX_PCI(obj); + + apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI); + apple_gfx_pci_irq_init(obj, &s->irq); ++ object_property_add(obj, "x-halt-poll-ns", "int", ++ apple_gfx_pci_get_halt_poll, ++ apple_gfx_pci_set_halt_poll, NULL, NULL); + } + + static void apple_gfx_pci_finalize(Object *obj) +diff --git a/include/system/kvm.h b/include/system/kvm.h +index 714b8c7b01..d38ec31d61 100644 +--- a/include/system/kvm.h ++++ b/include/system/kvm.h +@@ -214,6 +214,13 @@ int kvm_check_extension(KVMState *s, unsigned int extension); + + int kvm_vm_ioctl(KVMState *s, unsigned long type, ...); + ++/* ++ * LOCAL (Atlas): KVM_CAP_HALT_POLL for this VM, the most a halted vCPU ++ * polls before it sleeps, in ns (the kvm module's halt_poll_ns otherwise). ++ * 0 on success, -errno (-ENOSYS without KVM or the capability). ++ */ ++int kvm_set_halt_poll_ns(uint64_t ns); ++ + void kvm_flush_coalesced_mmio_buffer(void); + + void kvm_irqchip_add_change_notifier(Notifier *n); diff --git a/patches/qemu/0012-apple-gfx-pci-WakeFifo-doorbells-through-KVM-ioevent.patch b/patches/qemu/0012-apple-gfx-pci-WakeFifo-doorbells-through-KVM-ioevent.patch new file mode 100644 index 0000000..15d7c28 --- /dev/null +++ b/patches/qemu/0012-apple-gfx-pci-WakeFifo-doorbells-through-KVM-ioevent.patch @@ -0,0 +1,229 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 17:07:54 +0000 +Subject: [PATCH 12/13] apple-gfx-pci: WakeFifo doorbells through KVM + ioeventfds (x-ioeventfd, live switch) + +LOCAL (Atlas), ARCH experiment. A write of 0x1020 with FIFO id 1..8 is completed by KVM +and signals that id's eventfd; a doorbell thread makes the same library call off the vCPU. +Default off (APPLE_GFX_IOEVENTFD=1 turns it on at start); x-db-stats counts kicks/folds. + +Co-Authored-By: Claude Opus 5.5 +--- + hw/display/apple-gfx-pci.c | 164 +++++++++++++++++++++++++++++++++++++ + 1 file changed, 164 insertions(+) + +diff --git a/hw/display/apple-gfx-pci.c b/hw/display/apple-gfx-pci.c +index e2f1da1faa..ed0c00e2d0 100644 +--- a/hw/display/apple-gfx-pci.c ++++ b/hw/display/apple-gfx-pci.c +@@ -35,6 +35,8 @@ + #include "qapi/visitor.h" + #include "system/accel-irq.h" + #include "system/kvm.h" ++#include "qemu/thread.h" ++#include + #include "apple-gfx.h" + #include "trace.h" + +@@ -174,17 +176,167 @@ static void apple_gfx_pci_irq_realize(AppleGFXPCIIrq *irq) + + OBJECT_DECLARE_SIMPLE_TYPE(AppleGFXPCIState, APPLE_GFX_PCI) + ++/* ++ * LOCAL (Atlas), ARCH experiment: the WakeFifo doorbell (0x1020) through KVM ++ * ioeventfds. ++ * ++ * The guest's kext writes 0x1020 with the child FIFO's id after every command ++ * it queues (~100 per Sim frame). Each write was a full exit to QEMU's vCPU ++ * thread and a call into the library, which only queues a poke of that FIFO. ++ * With x-ioeventfd, KVM completes a write of id 1..APPLE_GFX_DB_IDS itself and ++ * signals that id's eventfd; this device's doorbell thread reads the eventfds ++ * and makes the same library call (rvg_device_mmio_write(0x1020, id)), off ++ * the vCPU. Doorbells that come while a poke is pending fold into one (the ++ * eventfd's counter): a poke makes the FIFO read its ring up to "written", ++ * so a folded doorbell loses nothing. Any other value still exits to QEMU. ++ * ++ * - x-ioeventfd (bool, default APPLE_GFX_IOEVENTFD=1 in the environment, else ++ * off) can be set live with qom-set; off is the exit path exactly as before. ++ * A doorbell signalled just before a switch off is still delivered: the ++ * thread keeps polling the eventfds. ++ * - x-db-stats: "kicks= folded= attached=<0|1>" (kicks: library calls ++ * from the thread; folded: doorbells that landed on a pending one). ++ */ ++#define APPLE_GFX_DB_IDS 8 ++#define APPLE_GFX_DB_OFFSET 0x1020 ++ ++typedef struct AppleGFXPCIDoorbell { ++ AppleGFXState *common; ++ MemoryRegion *mr; ++ bool wanted; /* x-ioeventfd */ ++ bool have_notifiers; ++ bool attached; /* BQL: the eventfds are in KVM */ ++ bool thread_started; ++ EventNotifier n[APPLE_GFX_DB_IDS]; ++ QemuThread thread; ++ uint64_t kicks; /* atomic */ ++ uint64_t folded; /* atomic */ ++} AppleGFXPCIDoorbell; ++ ++static void *apple_gfx_pci_db_thread(void *opaque) ++{ ++ AppleGFXPCIDoorbell *db = opaque; ++ struct pollfd pfd[APPLE_GFX_DB_IDS]; ++ int i; ++ ++ for (i = 0; i < APPLE_GFX_DB_IDS; i++) { ++ pfd[i].fd = event_notifier_get_fd(&db->n[i]); ++ pfd[i].events = POLLIN; ++ } ++ for (;;) { ++ if (poll(pfd, APPLE_GFX_DB_IDS, -1) < 0) { ++ if (errno == EINTR) { ++ continue; ++ } ++ error_report("apple-gfx: doorbell poll: %s", strerror(errno)); ++ return NULL; ++ } ++ for (i = 0; i < APPLE_GFX_DB_IDS; i++) { ++ uint64_t count; ++ ssize_t r; ++ ++ if (!(pfd[i].revents & POLLIN)) { ++ continue; ++ } ++ r = read(pfd[i].fd, &count, sizeof(count)); ++ if (r != sizeof(count) || count == 0) { ++ continue; ++ } ++ qatomic_add(&db->folded, count - 1); ++ qatomic_inc(&db->kicks); ++ rvg_device_mmio_write(db->common->pgdev, APPLE_GFX_DB_OFFSET, ++ (uint64_t)(i + 1)); ++ } ++ } ++ return NULL; ++} ++ ++/* Under the BQL: attach or detach the eventfds as wanted. */ ++static void apple_gfx_pci_db_sync(AppleGFXPCIDoorbell *db) ++{ ++ bool want = db->wanted && db->have_notifiers && db->mr && ++ kvm_enabled() && kvm_eventfds_enabled(); ++ int i; ++ ++ if (want == db->attached) { ++ return; ++ } ++ if (want && !db->thread_started) { ++ qemu_thread_create(&db->thread, "apple-gfx-db", ++ apple_gfx_pci_db_thread, db, QEMU_THREAD_DETACHED); ++ db->thread_started = true; ++ } ++ memory_region_transaction_begin(); ++ for (i = 0; i < APPLE_GFX_DB_IDS; i++) { ++ if (want) { ++ memory_region_add_eventfd(db->mr, APPLE_GFX_DB_OFFSET, 4, true, ++ i + 1, &db->n[i]); ++ } else { ++ memory_region_del_eventfd(db->mr, APPLE_GFX_DB_OFFSET, 4, true, ++ i + 1, &db->n[i]); ++ } ++ } ++ memory_region_transaction_commit(); ++ db->attached = want; ++ info_report("apple-gfx: WakeFifo doorbells %s", want ? ++ "through KVM ioeventfds" : "back on vCPU exits"); ++} ++ ++static void apple_gfx_pci_db_realize(AppleGFXPCIDoorbell *db, ++ AppleGFXState *common) ++{ ++ int i; ++ ++ db->common = common; ++ db->mr = &common->iomem_gfx; ++ for (i = 0; i < APPLE_GFX_DB_IDS; i++) { ++ if (event_notifier_init(&db->n[i], 0) < 0) { ++ warn_report("apple-gfx: no eventfd for the doorbells"); ++ while (--i >= 0) { ++ event_notifier_cleanup(&db->n[i]); ++ } ++ return; ++ } ++ } ++ db->have_notifiers = true; ++ apple_gfx_pci_db_sync(db); ++} ++ + struct AppleGFXPCIState { + PCIDevice parent_obj; + + AppleGFXState common; + AppleGFXPCIIrq irq; ++ AppleGFXPCIDoorbell db; + bool efi_fb_bar; + MemoryRegion efi_fb_bar_mr; /* BAR 2 */ + MemoryRegion efi_fb; /* in BAR 2: an alias of the machine's RAM */ + }; + + #define APPLE_GFX_PCI_EFI_FB_BAR_SIZE (64 << 20) ++ ++static bool apple_gfx_pci_db_get(Object *obj, Error **errp) ++{ ++ return APPLE_GFX_PCI(obj)->db.wanted; ++} ++ ++static void apple_gfx_pci_db_set(Object *obj, bool value, Error **errp) ++{ ++ AppleGFXPCIDoorbell *db = &APPLE_GFX_PCI(obj)->db; ++ ++ db->wanted = value; ++ apple_gfx_pci_db_sync(db); ++} ++ ++static char *apple_gfx_pci_db_stats(Object *obj, Error **errp) ++{ ++ AppleGFXPCIDoorbell *db = &APPLE_GFX_PCI(obj)->db; ++ ++ return g_strdup_printf("kicks=%" PRIu64 " folded=%" PRIu64 ++ " attached=%d", qatomic_read(&db->kicks), ++ qatomic_read(&db->folded), db->attached); ++} ++ + /* The library's page for the EFI registers: APV_PAGE_SHIFT in reims-vgpu */ + #ifdef HOST_X86_64 + #define APPLE_GFX_PCI_PAGE_SHIFT 12 +@@ -236,6 +388,14 @@ static void apple_gfx_pci_init(Object *obj) + + apple_gfx_common_init(obj, &s->common, TYPE_APPLE_GFX_PCI); + apple_gfx_pci_irq_init(obj, &s->irq); ++ { ++ const char *env = getenv("APPLE_GFX_IOEVENTFD"); ++ ++ s->db.wanted = env && strcmp(env, "1") == 0; ++ } ++ object_property_add_bool(obj, "x-ioeventfd", apple_gfx_pci_db_get, ++ apple_gfx_pci_db_set); ++ object_property_add_str(obj, "x-db-stats", apple_gfx_pci_db_stats, NULL); + object_property_add(obj, "x-halt-poll-ns", "int", + apple_gfx_pci_get_halt_poll, + apple_gfx_pci_set_halt_poll, NULL, NULL); +@@ -323,6 +483,7 @@ static void apple_gfx_pci_mmio_written(AppleGFXState *common, hwaddr offset, + + static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp) + { ++ ERRP_GUARD(); + AppleGFXPCIState *s = APPLE_GFX_PCI(dev); + RvgHostCallbacks callbacks = { + .struct_size = sizeof(callbacks), +@@ -380,6 +541,9 @@ static void apple_gfx_pci_realize(PCIDevice *dev, Error **errp) + apple_gfx_pci_irq_realize(&s->irq); + + apple_gfx_common_realize(&s->common, DEVICE(dev), &config, errp); ++ if (!*errp && s->common.pgdev) { ++ apple_gfx_pci_db_realize(&s->db, &s->common); ++ } + } + + static void apple_gfx_pci_reset(Object *obj, ResetType type) diff --git a/patches/qemu/0013-apple-gfx-pci-x-ioeventfd-KVM-always-has-eventfds-he.patch b/patches/qemu/0013-apple-gfx-pci-x-ioeventfd-KVM-always-has-eventfds-he.patch new file mode 100644 index 0000000..a03da3f --- /dev/null +++ b/patches/qemu/0013-apple-gfx-pci-x-ioeventfd-KVM-always-has-eventfds-he.patch @@ -0,0 +1,24 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 17:38:39 +0000 +Subject: [PATCH 13/13] apple-gfx-pci: x-ioeventfd: KVM always has eventfds + here (kvm_eventfds_enabled is gone) + +Co-Authored-By: Claude Opus 5.5 +--- + hw/display/apple-gfx-pci.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/hw/display/apple-gfx-pci.c b/hw/display/apple-gfx-pci.c +index ed0c00e2d0..cdc53eb7b3 100644 +--- a/hw/display/apple-gfx-pci.c ++++ b/hw/display/apple-gfx-pci.c +@@ -255,7 +255,7 @@ static void *apple_gfx_pci_db_thread(void *opaque) + static void apple_gfx_pci_db_sync(AppleGFXPCIDoorbell *db) + { + bool want = db->wanted && db->have_notifiers && db->mr && +- kvm_enabled() && kvm_eventfds_enabled(); ++ kvm_enabled(); + int i; + + if (want == db->attached) { diff --git a/patches/reims-vgpu/0026-pipelined-ExecIndirect3-on-a-FIFO-prototype-off-unle.patch b/patches/reims-vgpu/0026-pipelined-ExecIndirect3-on-a-FIFO-prototype-off-unle.patch new file mode 100644 index 0000000..cdb77d0 --- /dev/null +++ b/patches/reims-vgpu/0026-pipelined-ExecIndirect3-on-a-FIFO-prototype-off-unle.patch @@ -0,0 +1,991 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 16:12:13 +0000 +Subject: [PATCH 26/33] pipelined ExecIndirect3 on a FIFO (prototype, off + unless REIMS_VGPU_PIPELINE_EXEC names an existing file) + +A pipelining FIFO sends a remote task's exec and goes on; the reply's follow-up (backing +invalidation, failure completion) settles later on the FIFO thread in send order. The FIFO +event value is reserved at send; a refused/faulted exec commits a fill command buffer that +keeps the cross-task event chain moving, and the host signals after a crash. +Not for adoption. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-core/src/fifo.rs | 8 + + crates/reims-vgpu-core/src/fifo/commands.rs | 3 + + .../reims-vgpu-core/src/fifo/commands/exec.rs | 24 ++ + crates/reims-vgpu-core/src/fifo/pipeline.rs | 238 ++++++++++++++ + crates/reims-vgpu-core/src/fifo/ring.rs | 14 +- + crates/reims-vgpu-core/src/fifo/thread.rs | 7 + + crates/reims-vgpu-core/src/gpu_task/exec.rs | 106 ++++++ + .../reims-vgpu-core/src/gpu_task/service.rs | 2 + + crates/reims-vgpu-core/src/tasks/remote.rs | 304 ++++++++++++++++++ + .../src/tasks/remote/completions.rs | 5 + + crates/reims-vgpu-core/src/tasks/wire.rs | 4 + + 11 files changed, 714 insertions(+), 1 deletion(-) + create mode 100644 crates/reims-vgpu-core/src/fifo/pipeline.rs + +diff --git a/crates/reims-vgpu-core/src/fifo.rs b/crates/reims-vgpu-core/src/fifo.rs +index 0cd8e87a..3d58517d 100644 +--- a/crates/reims-vgpu-core/src/fifo.rs ++++ b/crates/reims-vgpu-core/src/fifo.rs +@@ -28,6 +28,7 @@ + //! `tests::a_define_whose_event_raises_dies_under_its_handler`). + + pub mod commands; ++pub mod pipeline; + pub mod ring; + pub mod stamps; + pub mod thread; +@@ -241,6 +242,11 @@ pub struct Fifo { + /// A child's ring mapping, from a start to its thread's exit. + mapping: Mutex>>>, + platform: Arc

, ++ /// LOCAL (Atlas, VM-B): the remote execs sent and not yet settled, oldest first ++ /// ([`pipeline`]). Touched by the FIFO's thread only. ++ pending: Mutex>>, ++ /// Whether this FIFO pipelines its execs now; it changes only while nothing is pending. ++ pipelining: AtomicBool, + } + + impl Drop for Fifo

{ +@@ -292,6 +298,8 @@ impl Fifo

{ + event_value: Arc::new(AtomicU64::new(0)), + mapping: Mutex::new(None), + platform: Arc::clone(platform), ++ pending: Mutex::new(std::collections::VecDeque::new()), ++ pipelining: AtomicBool::new(false), + }) + } + +diff --git a/crates/reims-vgpu-core/src/fifo/commands.rs b/crates/reims-vgpu-core/src/fifo/commands.rs +index e21b8595..ed1ec9bb 100644 +--- a/crates/reims-vgpu-core/src/fifo/commands.rs ++++ b/crates/reims-vgpu-core/src/fifo/commands.rs +@@ -50,6 +50,9 @@ use crate::tasks::{Completion, ExecStatus, Task}; + const DUMP_MAX: usize = 64; + + /// The text every feature gate raises with. ++/// LOCAL (Atlas, VM-B): ExecIndirect3's command id, the one opcode a pipelining FIFO overlaps. ++pub(crate) const EXEC_INDIRECT3_ID: u16 = 67; ++ + pub(crate) const UNSUPPORTED: &str = "the command is not supported at this binary version"; + /// The text of commands 4, 6, 7, 8 and 54 naming another task. + pub(crate) const KERNEL_ONLY: &str = "the command must name a kernel task"; +diff --git a/crates/reims-vgpu-core/src/fifo/commands/exec.rs b/crates/reims-vgpu-core/src/fifo/commands/exec.rs +index cce756c8..87ce3cb6 100644 +--- a/crates/reims-vgpu-core/src/fifo/commands/exec.rs ++++ b/crates/reims-vgpu-core/src/fifo/commands/exec.rs +@@ -141,11 +141,35 @@ pub(super) fn exec_indirect2(c: &mut Command<'_, P>) { + /// 67. + pub(super) fn exec_indirect3(c: &mut Command<'_, P>) { + let hold = c.hold; ++ // LOCAL (Atlas, VM-B): a pipelining FIFO settles what it can before the exec ++ // (`fifo::pipeline`), sends a remote task's exec without waiting for its reply, and runs a local ++ // task's on the base path once everything pending is settled. ++ let pipelining = c.fifo.pipelining(); ++ if pipelining { ++ c.fifo.settle_before_exec(); ++ } + let Some((task, rc, cbc)) = c.f_class(|c, _| prologue(c, Kind::Exec3)) else { + return; + }; + let (resources, cmd_buffers) = records::(c, rc, cbc, RESOURCE3); + let done = c.completion(); ++ if pipelining { ++ if let Task::Remote(t) = &task { ++ let pending = t.exec_indirect3_pipelined( ++ ExecArgs { ++ cmd_buffers, ++ resources, ++ stamp_index: c.fifo.stamp_index(), ++ fifo_event: c.fifo.event.as_ref(), ++ fifo_event_value: &c.fifo.event_value, ++ }, ++ done, ++ ); ++ c.fifo.keep_pending(pending); ++ return; ++ } ++ c.fifo.settle_all(); ++ } + // Outside any handler. + if let Err(r) = task.exec_indirect3( + hold, +diff --git a/crates/reims-vgpu-core/src/fifo/pipeline.rs b/crates/reims-vgpu-core/src/fifo/pipeline.rs +new file mode 100644 +index 00000000..9734e771 +--- /dev/null ++++ b/crates/reims-vgpu-core/src/fifo/pipeline.rs +@@ -0,0 +1,238 @@ ++//! LOCAL (Atlas, VM-B, branch atlas/deep-multisim-pipeline-exec): pipelined remote execs. ++//! ++//! The base FIFO thread blocks in each remote exec until the task's service replied ++//! (`tasks::remote::RemoteTask::exec`), so every task's execs on one FIFO run one after ++//! another. A pipelining FIFO sends a remote exec and goes on; the part of the exec that followed ++//! the reply (the kernel task's shared-texture invalidation, a failure's completion) runs later on ++//! the FIFO's thread, in send order, at a settlement: ++//! - before each pipelined exec, every pending exec whose reply came (no wait), and the oldest ++//! while [`WINDOW`] are pending (a wait); ++//! - before every other command, all of them (waits): every non-exec command sees what it sees on ++//! the base path; ++//! - before the thread waits for ring bytes, the replied ones, and the wait is then at most ++//! [`IDLE_POLL`] while any is pending; at the loop's end, all of them. ++//! ++//! Per task, the task's serial connection and service queues keep its execs in order. Across ++//! tasks the FIFO's shared-event chain keeps them in order on the GPU (`gpu_task::exec`): the FIFO's ++//! event value is reserved at the send, and a refused or faulted exec still moves the chain on ++//! (the service's fill, or the host's signal at settlement after a crash). ++//! ++//! The switch: `REIMS_VGPU_PIPELINE_EXEC=`; pipelining is on while that file exists, looked ++//! at most once a second and only while nothing is pending. Unset: the base path, unchanged. ++ ++use std::sync::OnceLock; ++use std::sync::atomic::{AtomicU64, Ordering}; ++use std::time::{Duration, Instant}; ++ ++use parking_lot::Mutex; ++ ++use super::Fifo; ++use crate::platform::Platform; ++use crate::rt::Hold; ++use crate::rt::held::Held; ++use crate::rt::pool::Pool; ++ ++/// The most remote execs a FIFO keeps sent and unsettled. ++pub const WINDOW: usize = 8; ++ ++/// While execs are pending, the longest wait for ring bytes before the replied ones settle. ++pub const IDLE_POLL: Duration = Duration::from_micros(500); ++ ++/// One remote exec sent and not settled. ++pub trait PendingExec: Send + Sync { ++ /// Whether its reply came (its settlement will not wait). ++ fn replied(&self) -> bool; ++ /// What followed the reply on the base path, after waiting for it. ++ fn settle(self: Box, hold: Hold<'_>); ++} ++ ++/// The switch file, and its last look. ++struct Switch { ++ path: Option, ++ look: Mutex<(Option, bool)>, ++} ++ ++fn switch() -> &'static Switch { ++ static S: OnceLock = OnceLock::new(); ++ S.get_or_init(|| Switch { ++ path: std::env::var_os("REIMS_VGPU_PIPELINE_EXEC") ++ .filter(|p| !p.is_empty()) ++ .map(std::path::PathBuf::from), ++ look: Mutex::new((None, false)), ++ }) ++} ++ ++/// Counts for the log, all FIFOs together: execs sent pipelined, settlements that waited for a ++/// reply (and of those, the ones the full window forced), and the settlements of pending execs a ++/// non-exec command forced, by opcode. ++struct Stats { ++ sent: AtomicU64, ++ waited: AtomicU64, ++ full: AtomicU64, ++ forced: [AtomicU64; 128], ++ last: Mutex>, ++} ++ ++static STATS: Stats = Stats { ++ sent: AtomicU64::new(0), ++ waited: AtomicU64::new(0), ++ full: AtomicU64::new(0), ++ forced: [const { AtomicU64::new(0) }; 128], ++ last: Mutex::new(None), ++}; ++ ++/// Logs the counts (and resets them) at most every 10 s. ++fn log_stats() { ++ let now = Instant::now(); ++ { ++ let mut last = STATS.last.lock(); ++ match *last { ++ None => { ++ *last = Some(now); ++ return; ++ } ++ Some(t) if now.duration_since(t) < Duration::from_secs(10) => return, ++ Some(_) => *last = Some(now), ++ } ++ } ++ let sent = STATS.sent.swap(0, Ordering::Relaxed); ++ let waited = STATS.waited.swap(0, Ordering::Relaxed); ++ let full = STATS.full.swap(0, Ordering::Relaxed); ++ let mut forced = String::new(); ++ for (op, n) in STATS.forced.iter().enumerate() { ++ let n = n.swap(0, Ordering::Relaxed); ++ if n != 0 { ++ forced.push_str(&format!(" {op}:{n}")); ++ } ++ } ++ if sent != 0 { ++ log::info!("FIFO exec pipelining: {sent} sent, {waited} waited ({full} window full), forced by opcode:{forced}"); ++ } ++} ++ ++/// Whether pipelining is wanted now (the file, looked at most once a second). ++fn wanted() -> bool { ++ let s = switch(); ++ let Some(path) = &s.path else { ++ return false; ++ }; ++ let mut look = s.look.lock(); ++ let now = Instant::now(); ++ if look.0.is_none_or(|t| now.duration_since(t) >= Duration::from_secs(1)) { ++ log_stats(); ++ let on = path.exists(); ++ if on != look.1 { ++ log::info!("FIFO exec pipelining {}", if on { "on" } else { "off" }); ++ } ++ *look = (Some(now), on); ++ } ++ look.1 ++} ++ ++impl Fifo

{ ++ /// Whether this FIFO pipelines its next remote exec: it follows the switch only while nothing ++ /// is pending. ++ pub(crate) fn pipelining(&self) -> bool { ++ if self.pending.lock().is_empty() { ++ let on = wanted(); ++ self.pipelining.store(on, Ordering::Relaxed); ++ on ++ } else { ++ self.pipelining.load(Ordering::Relaxed) ++ } ++ } ++ ++ /// Settles `p` under a pool and a holder of its own, as a command's. ++ fn settle_one(&self, p: Box) { ++ if !p.replied() { ++ STATS.waited.fetch_add(1, Ordering::Relaxed); ++ } ++ let pool = Pool::push(&*self.platform); ++ let held = Held::new(); ++ p.settle(Hold::of(&held)); ++ drop(held); ++ drop(pool); ++ } ++ ++ /// Before a pipelined exec: settles the replied execs at the front, then the oldest while the ++ /// window is full. ++ pub(crate) fn settle_before_exec(&self) { ++ loop { ++ let next = { ++ let mut q = self.pending.lock(); ++ match q.front() { ++ Some(f) if f.replied() => q.pop_front(), ++ Some(_) if q.len() >= WINDOW => { ++ STATS.full.fetch_add(1, Ordering::Relaxed); ++ q.pop_front() ++ } ++ _ => None, ++ } ++ }; ++ match next { ++ Some(p) => self.settle_one(p), ++ None => return, ++ } ++ } ++ } ++ ++ /// Settles the replied execs at the front (no wait). ++ pub(crate) fn settle_replied(&self) { ++ loop { ++ let next = { ++ let mut q = self.pending.lock(); ++ match q.front() { ++ Some(f) if f.replied() => q.pop_front(), ++ _ => None, ++ } ++ }; ++ match next { ++ Some(p) => self.settle_one(p), ++ None => return, ++ } ++ } ++ } ++ ++ /// Whether an exec is pending. ++ pub(crate) fn has_pending(&self) -> bool { ++ !self.pending.lock().is_empty() ++ } ++ ++ /// Keeps a sent exec for its settlement. ++ pub(crate) fn keep_pending(&self, p: Box) { ++ STATS.sent.fetch_add(1, Ordering::Relaxed); ++ self.pending.lock().push_back(p); ++ } ++ ++ /// Before a command other than ExecIndirect3: settles every pending exec, counted under the ++ /// command's opcode. ++ pub(crate) fn settle_for_command(&self, command_id: u16) { ++ if self.settle_all() { ++ let i = usize::from(command_id).min(STATS.forced.len() - 1); ++ if let Some(n) = STATS.forced.get(i) { ++ n.fetch_add(1, Ordering::Relaxed); ++ } ++ } ++ } ++ ++ /// Settles every pending exec, oldest first; whether there was one. ++ pub(crate) fn settle_all(&self) -> bool { ++ let mut any = false; ++ loop { ++ let next = self.pending.lock().pop_front(); ++ match next { ++ Some(p) => { ++ any = true; ++ self.settle_one(p); ++ } ++ None => return any, ++ } ++ } ++ } ++ ++ /// Tests: how many execs are pending. ++ #[cfg(test)] ++ pub(crate) fn pending_count(&self) -> usize { ++ self.pending.lock().len() ++ } ++} +diff --git a/crates/reims-vgpu-core/src/fifo/ring.rs b/crates/reims-vgpu-core/src/fifo/ring.rs +index 5331dc88..4a8cd0c3 100644 +--- a/crates/reims-vgpu-core/src/fifo/ring.rs ++++ b/crates/reims-vgpu-core/src/fifo/ring.rs +@@ -213,7 +213,13 @@ impl Fifo

{ + fn wait_for_bytes(&self, g: &mut MutexGuard<'_, ()>) { + #[cfg(test)] + self.waiting_for_bytes.fetch_add(1, Ordering::SeqCst); +- self.bytes.wait(g); ++ // LOCAL (Atlas, VM-B): while pipelined execs are pending, a short wait, so their ++ // settlement follows their replies (`fifo::pipeline`). ++ if self.has_pending() { ++ let _ = self.bytes.wait_for(g, crate::fifo::pipeline::IDLE_POLL); ++ } else { ++ self.bytes.wait(g); ++ } + #[cfg(test)] + self.waiting_for_bytes.fetch_sub(1, Ordering::SeqCst); + } +@@ -249,6 +255,9 @@ impl Fifo

{ + .load(Ordering::SeqCst) + .wrapping_sub(r.read.load(Ordering::SeqCst)); + if available == 0 { ++ // LOCAL (Atlas, VM-B): the replied pending execs settle before the wait for bytes; ++ // with execs still pending the wait is short (`fifo::pipeline`). ++ self.settle_replied(); + let mut g = self.lock.lock(); + if !self.quiesced.load(Ordering::SeqCst) + && r.written.load(Ordering::SeqCst) == r.read.load(Ordering::SeqCst) +@@ -307,6 +316,9 @@ impl Fifo

{ + // The cache is reloaded only when too small; a lowered `written` wraps. + let count = available(dst.len()); + if count == 0 { ++ // LOCAL (Atlas, VM-B): the replied pending execs settle before the wait for bytes; ++ // with execs still pending the wait is short (`fifo::pipeline`). ++ self.settle_replied(); + let mut g = self.lock.lock(); + if !self.quiesced.load(Ordering::SeqCst) && available(dst.len()) == 0 { + self.wait_for_bytes(&mut g); +diff --git a/crates/reims-vgpu-core/src/fifo/thread.rs b/crates/reims-vgpu-core/src/fifo/thread.rs +index 8d6a6f8a..cb381144 100644 +--- a/crates/reims-vgpu-core/src/fifo/thread.rs ++++ b/crates/reims-vgpu-core/src/fifo/thread.rs +@@ -280,6 +280,11 @@ impl Fifo

{ + } + self.push_stamp(command.stamp_value, command.command_id); + let body = payload.get(..payload_len).unwrap_or(&[]); ++ // LOCAL (Atlas, VM-B): every command but a pipelined exec sees the pending execs ++ // settled, as on the base path (`fifo::pipeline`). ++ if command.command_id != crate::fifo::commands::EXEC_INDIRECT3_ID { ++ self.settle_for_command(command.command_id); ++ } + self.dispatch(&command, body); + // ran: FIFO, opcode, stamp value: its handler returned + crate::trace::event( +@@ -292,6 +297,8 @@ impl Fifo

{ + ); + } + drop(payload); ++ // LOCAL (Atlas, VM-B): nothing stays pending past the loop. ++ self.settle_all(); + // Outside the lock + self.rewind(); + let _g = self.lock.lock(); +diff --git a/crates/reims-vgpu-core/src/gpu_task/exec.rs b/crates/reims-vgpu-core/src/gpu_task/exec.rs +index f038b3b8..9ac17a94 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/exec.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/exec.rs +@@ -131,6 +131,7 @@ impl GpuTask

{ + stamp_index: u32, + fifo_event_value_in: u64, + gpu_completion: u64, ++ fill_on_failure: Option>, + reply: Reply<(u32, Option>, u32)>, + ) { + self.job(move |this, hold| { +@@ -142,11 +143,55 @@ impl GpuTask

{ + stamp_index, + fifo_event_value_in, + gpu_completion, ++ fill_on_failure.as_deref(), + &reply, + ) + }); + } + ++ /// LOCAL (Atlas, VM-B): an exec of a pipelining FIFO that was refused or faulted before its ++ /// commit still moves the FIFO's event chain on, as its success would have: one command buffer ++ /// that waits as the exec's would (`fifo_event_value_in`, unless 0 or this task's own last ++ /// value), stores and signals `fifo_event_value_in + 1` when scheduled, and commits. A later ++ /// exec of another task, sent with that value before this reply, waits for it. `false` when ++ /// nothing was committed (no FIFO event, no command queue, or a raise, landed): the host's ++ /// settlement then signals it (`tasks::remote::PendingRemoteExec`). ++ fn fill_chain(&self, hold: Hold<'_>, stamp_index: u32, fifo_event_value_in: u64) -> bool { ++ let Some((event, signaled)) = self.fifo(stamp_index) else { ++ return false; ++ }; ++ if event.is_none() { ++ return false; ++ } ++ let filled = (|| -> RaiseResult { ++ let command_queue = self.command_queue.lock().as_ref().map(Handle::clone_ref); ++ let command_buffer = match &command_queue { ++ Some(q) => q.command_buffer(OnFault::Return)?, ++ None => None, ++ }; ++ let Some(b) = &command_buffer else { ++ return Ok(false); ++ }; ++ let signaled_value = signaled.load(Ordering::SeqCst); ++ if fifo_event_value_in != 0 && fifo_event_value_in != signaled_value { ++ b.encode_wait(event.as_ref().map(EventRef::Shared), fifo_event_value_in)?; ++ } ++ let value = fifo_event_value_in.wrapping_add(1); ++ signaled.store(value, Ordering::SeqCst); ++ b.encode_signal_when_scheduled(event.as_ref(), value)?; ++ b.commit(OnFault::Return)?; ++ Ok(true) ++ })(); ++ match filled { ++ Ok(f) => f, ++ Err(t) => { ++ log::warn!("task {}: the FIFO chain's fill faulted: {}", self.id(), exception(&t)); ++ hold.land(t); ++ false ++ } ++ } ++ } ++ + /// The exec, under the shared task lock: the object lists mapped (2 when they cannot be), + /// the payload's size and the FIFO checked (2), then, under a handler that replies 2: the + /// resources paged in, one command buffer that waits on the paging event (when a paging +@@ -169,11 +214,24 @@ impl GpuTask

{ + stamp_index: u32, + fifo_event_value_in: u64, + gpu_completion: u64, ++ fill_on_failure: Option<&std::sync::atomic::AtomicBool>, + reply: &Reply<(u32, Option>, u32)>, + ) { + let _lock = self.task_lock(Access::Shared); ++ // LOCAL (Atlas, VM-B): a refusal or a fault of a pipelining FIFO's exec keeps its place in ++ // the FIFO's event chain before the reply: its own buffer was committed, or `fill_chain`. ++ let fill = |committed: bool| { ++ // The flag tells the host the chain's value is the service's to signal: the exec's own ++ // committed buffer carries it, or the fill does. ++ if let Some(filled) = fill_on_failure ++ && (committed || self.fill_chain(hold, stamp_index, fifo_event_value_in)) ++ { ++ filled.store(true, Ordering::SeqCst); ++ } ++ }; + if let Err(t) = self.ensure_object_list_mapped(hold) { + log::warn!("task {}: exec refused: {}", self.id(), exception(&t)); ++ fill(false); + reply.call((2, None, 0)); + hold.land(t); + return; +@@ -185,6 +243,7 @@ impl GpuTask

{ + self.id(), + data.len() + ); ++ fill(false); + reply.call((2, None, 0)); + return; + } +@@ -200,6 +259,7 @@ impl GpuTask

{ + let wait_value = fifo_event_value_in; + let signaled_value = signaled.load(Ordering::SeqCst); + let mut shared_texture_backing_ids: Vec = Vec::new(); ++ let committed = std::cell::Cell::new(false); + let executed = (|| -> RaiseResult<_> { + let prepare_event = if wait_value == 0 || wait_value == signaled_value { + None +@@ -255,6 +315,7 @@ impl GpuTask

{ + } + if let Some(b) = &command_buffer { + b.commit(OnFault::Return)?; ++ committed.set(true); + } + let complete = CompleteSync { + needs_complete: sync.needs_complete, +@@ -273,6 +334,7 @@ impl GpuTask

{ + })(); + if let Err(t) = executed { + log::warn!("task {}: exec faulted: {}", self.id(), exception(&t)); ++ fill(committed.get()); + reply.call((2, None, 0)); + hold.land(t); + return; +@@ -693,6 +755,7 @@ mod tests { + stamp_index, + fifo_event_value_in, + gpu_completion, ++ fill_on_failure: None, + reply, + } + } +@@ -721,6 +784,7 @@ mod tests { + stamp_index: index, + fifo_event_value_in: 0, + gpu_completion: 0x77, ++ fill_on_failure: None, + reply, + }); + let v: Vec = got.lock().iter().map(|r| r.0).collect(); +@@ -762,6 +826,48 @@ mod tests { + assert_eq!(*tx.host.sent.lock(), vec!["complete 119 1".to_owned()]); + } + ++ /// LOCAL (Atlas, VM-B): a refused exec of a pipelining FIFO commits the fill before its ++ /// reply: it waits for the value it was sent with, stores and signals the next one; a refusal ++ /// with no FIFO event leaves the flag down (the host signals). Without the flag nothing is ++ /// committed, as on the base path. ++ #[test] ++ fn a_refused_pipelined_exec_fills_the_chain_before_its_reply() { ++ let tx = ready(&rw_pages(4)); ++ with_fifo(&tx, 1); ++ let refused = |index: u32, flag: Option>| { ++ let seen = Arc::new(Mutex::new(Vec::new())); ++ let (s2, f2) = (seen.clone(), flag.clone()); ++ tx.send(ToService::ExecIndirect3 { ++ data: vec![0; 8], ++ cmd_buffer_count: 1, ++ resource_count: 0, ++ stamp_index: index, ++ fifo_event_value_in: 5, ++ gpu_completion: 0x77, ++ fill_on_failure: flag, ++ reply: Reply::new(move |r: (u32, Option>, u32)| { ++ s2.lock() ++ .push((r.0, f2.as_ref().map(|f| f.load(Ordering::SeqCst)))); ++ }), ++ }); ++ let v = seen.lock().clone(); ++ v ++ }; ++ let mark = tx.p.call_log().mark(); ++ assert_eq!(refused(1, None), vec![(2, None)]); ++ assert!(!tx.p.call_log().sels_since(mark).contains(&"commit")); ++ let mark = tx.p.call_log().mark(); ++ let flag = Arc::new(AtomicBool::new(false)); ++ assert_eq!(refused(1, Some(flag.clone())), vec![(2, Some(true))]); ++ let sels = tx.p.call_log().sels_since(mark); ++ let pos = |s: &str| sels.iter().position(|x| *x == s).unwrap(); ++ assert!(pos("encodeWaitForEvent:value:") < pos("encodeSignalEventScheduled:value:")); ++ assert!(pos("encodeSignalEventScheduled:value:") < pos("commit")); ++ assert_eq!(tx.task.fifos.lock()[&1].signaled.load(Ordering::SeqCst), 6); ++ let flag = Arc::new(AtomicBool::new(false)); ++ assert_eq!(refused(3, Some(flag.clone())), vec![(2, Some(false))]); ++ } ++ + /// A wait value other than the last signalled one makes the exec wait on the FIFO + #[test] + fn exec_waits_for_the_fifo() { +diff --git a/crates/reims-vgpu-core/src/gpu_task/service.rs b/crates/reims-vgpu-core/src/gpu_task/service.rs +index 861d9b35..afdcf70d 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/service.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/service.rs +@@ -622,6 +622,7 @@ impl ServiceSide

for GpuTask

{ + stamp_index, + fifo_event_value_in, + gpu_completion, ++ fill_on_failure, + reply, + } => self.exec( + data, +@@ -630,6 +631,7 @@ impl ServiceSide

for GpuTask

{ + stamp_index, + fifo_event_value_in, + gpu_completion, ++ fill_on_failure, + reply, + ), + ToService::InfoIndirect { +diff --git a/crates/reims-vgpu-core/src/tasks/remote.rs b/crates/reims-vgpu-core/src/tasks/remote.rs +index a1d20c18..c2abd64d 100644 +--- a/crates/reims-vgpu-core/src/tasks/remote.rs ++++ b/crates/reims-vgpu-core/src/tasks/remote.rs +@@ -414,6 +414,7 @@ impl RemoteTask

{ + // Read at the send. + fifo_event_value_in: fifo_event_value.load(Ordering::Relaxed), + gpu_completion: token.id(), ++ fill_on_failure: None, + reply, + }); + let status = gpu_token.wait(); +@@ -440,6 +441,127 @@ impl RemoteTask

{ + } + } + ++/// LOCAL (Atlas, VM-B): a remote exec a pipelining FIFO sent and has not settled ++/// (`fifo::pipeline`): what `RemoteTask::exec` does after its reply, run at the settlement. ++pub(crate) struct PendingRemoteExec { ++ /// The reply's wait: its handler stores nothing (the value was reserved at the send). ++ gpu_token: Arc, ++ /// The completion token, completed here on a failure as on the base path. ++ token: Arc, ++ backings: Arc>, ++ endpoint: Arc>, ++ /// The FIFO's event and the value this exec waits for: on a failure the service did not ++ /// carry, the host signals `value_in + 1` once `value_in` is reached. ++ event: Option, ++ value_in: u64, ++ /// Set by the service when the chain's value is its to signal (`GpuTask::fill_chain`). ++ carried: Arc, ++} ++ ++/// How long a failure's settlement waits for the chain to reach its value before it signals the ++/// next one anyway. ++const CHAIN_WAIT_MS: u64 = 5000; ++ ++impl crate::fifo::pipeline::PendingExec for PendingRemoteExec

{ ++ fn replied(&self) -> bool { ++ self.gpu_token.is_complete() ++ } ++ ++ fn settle(self: Box, hold: Hold<'_>) { ++ let status = self.gpu_token.wait(); ++ let ids = backing_ids(&self.backings); ++ self.endpoint ++ .device_link() ++ .invalidate_shared_texture_backings(hold, &ids); ++ if status != ExecStatus::OK && status != ExecStatus::CRASHED { ++ self.endpoint.completions().complete(&self.token, status); ++ } ++ if status != ExecStatus::OK ++ && !self.carried.load(Ordering::SeqCst) ++ && let Some(event) = &self.event ++ { ++ // The chain waits for this exec's value: once the previous one is reached, and unless ++ // this one already is (a dead task's committed buffer), the host signals it. ++ let next = self.value_in.wrapping_add(1); ++ if !event.wait_until_signaled_value(self.value_in, CHAIN_WAIT_MS) { ++ log::warn!( ++ "FIFO chain: value {} not reached in {CHAIN_WAIT_MS} ms; signalling {next}", ++ self.value_in ++ ); ++ } ++ if !event.wait_until_signaled_value(next, 0) { ++ if let Err(e) = event.set_signaled_value(next) { ++ log::warn!("FIFO chain: signalling {next} failed: {}", e.exception()); ++ hold.land(e); ++ } ++ } ++ } ++ } ++} ++ ++/// The reply's count of ids, bounded by the ids it sent (the service always sends as many as it ++/// counts). ++fn backing_ids(backings: &Mutex) -> Vec { ++ let b = backings.lock(); ++ let all: Vec = b ++ .data ++ .as_chunks::<4>() ++ .0 ++ .iter() ++ .map(|w| u32::from_ne_bytes(*w)) ++ .collect(); ++ let n = b.count.to_usize().min(all.len()); ++ all.get(..n).map(<[u32]>::to_vec).unwrap_or_default() ++} ++ ++impl RemoteTask

{ ++ /// LOCAL (Atlas, VM-B): ExecIndirect3 on a pipelining FIFO: shares the FIFO's event, reserves ++ /// the FIFO's event value (the base path advances it on the reply of 1), sends the command with ++ /// the fill flag, and returns what follows the reply, for the FIFO's settlement. ++ pub(crate) fn exec_indirect3_pipelined( ++ &self, ++ args: ExecArgs<'_, P, CmdExecIndirectResource3>, ++ done: Completion, ++ ) -> Box { ++ let data = command_data(&args.resources, &args.cmd_buffers); ++ self.share_fifo_event(args.fifo_event, args.stamp_index); ++ let value_in = args.fifo_event_value.fetch_add(1, Ordering::SeqCst); ++ let gpu_token = self.endpoint.completions().issue(); ++ let backings = Arc::new(Mutex::new(Backings::default())); ++ let token = self.endpoint.completions().issue_with(completing(done)); ++ let carried = Arc::new(std::sync::atomic::AtomicBool::new(false)); ++ let reply = { ++ let reply = ExecReply { ++ gpu_token: gpu_token.clone(), ++ _token: token.clone(), ++ backings: backings.clone(), ++ endpoint: self.endpoint.clone(), ++ _task: self.me.upgrade(), ++ }; ++ Reply::new(move |v: (u32, Option>, u32)| reply.invoke(v)) ++ }; ++ self.link.send(ToService::ExecIndirect3 { ++ data, ++ cmd_buffer_count: args.cmd_buffers.count(), ++ resource_count: args.resources.count(), ++ stamp_index: args.stamp_index, ++ fifo_event_value_in: value_in, ++ gpu_completion: token.id(), ++ fill_on_failure: Some(carried.clone()), ++ reply, ++ }); ++ Box::new(PendingRemoteExec { ++ gpu_token, ++ token, ++ backings, ++ endpoint: self.endpoint.clone(), ++ event: args.fifo_event.map(Handle::clone_ref), ++ value_in, ++ carried, ++ }) ++ } ++} ++ + impl Drop for RemoteTask

{ + /// Drains, invalidates the link, then releases the FIFO events in index order. + fn drop(&mut self) { +@@ -2504,4 +2626,186 @@ mod tests { + assert_eq!(v.load(Ordering::SeqCst), 0); + } + } ++ ++ /// LOCAL (Atlas, VM-B): the pipelined exec (`fifo::pipeline`). ++ mod pipelined { ++ use super::*; ++ use crate::fifo::pipeline::PendingExec; ++ use crate::tasks::test_support::Lines; ++ ++ const EXEC3: &str = "execIndirect3WithData:cmdBufferCount:resourceCount:stampIndex:fifoEventValueIn:gpuCompletion:reply:"; ++ ++ fn send( ++ t: &RemoteTask

, ++ event: Option<&

::SharedEvent>, ++ value: &Arc, ++ d: Completion, ++ ) -> Box { ++ let (bytes, split) = payload(&res3(), &cmd()); ++ let args = ExecArgs { ++ cmd_buffers: Records::new(&bytes[split..], 1).unwrap(), ++ resources: Records::new(&bytes[..split], 2).unwrap(), ++ stamp_index: 2, ++ fifo_event: event, ++ fifo_event_value: value, ++ }; ++ t.exec_indirect3_pipelined(args, d) ++ } ++ ++ /// The send returns before the reply, with the value reserved; the settlement follows the ++ /// reply, and the completion comes from the service as on the base path. ++ #[test] ++ fn sends_without_waiting_and_reserves_the_value() { ++ let gate = Gate::default(); ++ let g2 = gate.clone(); ++ let h = harness(move |ctx, m| match m { ++ ToService::ExecIndirect3 { ++ gpu_completion, ++ reply, ++ fill_on_failure, ++ .. ++ } => { ++ assert!(fill_on_failure.is_some()); ++ g2.wait(); ++ reply.call((1, None, 0)); ++ ctx.complete(gpu_completion, 1); ++ } ++ other => standard(ctx, other), ++ }); ++ let t = h.task(); ++ let ev = h.event(); ++ let v = Arc::new(AtomicU64::new(41)); ++ let (s, d) = done(); ++ let p = send(&t, Some(&ev), &v, d); ++ assert_eq!(v.load(Ordering::SeqCst), 42); ++ h.wait_for(3); ++ assert_eq!(h.msgs()[2].sel, EXEC3); ++ assert_eq!(h.msgs()[2].nums[3], 41); ++ assert!(!p.replied()); ++ gate.open(); ++ wait_until(|| p.replied()); ++ in_command(|hold| p.settle(hold)); ++ wait_until(|| *s.lock() == vec![1]); ++ // A success leaves the chain to the exec's own command buffer. ++ assert!(!ev.wait_until_signaled_value(42, 0)); ++ } ++ ++ /// A failure the service did not carry: the settlement completes it and signals the next ++ /// value once the exec's own is reached. ++ #[test] ++ fn an_uncarried_failure_signals_the_next_value() { ++ let h = harness(|ctx, m| match m { ++ ToService::ExecIndirect3 { reply, .. } => reply.call((2, None, 0)), ++ other => standard(ctx, other), ++ }); ++ let t = h.task(); ++ let ev = h.event(); ++ ev.set_signaled_value(41).unwrap(); ++ let v = Arc::new(AtomicU64::new(41)); ++ let (s, d) = done(); ++ let p = send(&t, Some(&ev), &v, d); ++ wait_until(|| p.replied()); ++ in_command(|hold| p.settle(hold)); ++ assert_eq!(*s.lock(), vec![2]); ++ assert!(ev.wait_until_signaled_value(42, 0)); ++ } ++ ++ /// A failure the service carried (its fill command buffer signals): the host does not. ++ #[test] ++ fn a_carried_failure_leaves_the_chain_to_the_service() { ++ let h = harness(|ctx, m| match m { ++ ToService::ExecIndirect3 { ++ reply, ++ fill_on_failure, ++ .. ++ } => { ++ fill_on_failure.unwrap().store(true, Ordering::SeqCst); ++ reply.call((2, None, 0)); ++ } ++ other => standard(ctx, other), ++ }); ++ let t = h.task(); ++ let ev = h.event(); ++ ev.set_signaled_value(41).unwrap(); ++ let v = Arc::new(AtomicU64::new(41)); ++ let (s, d) = done(); ++ let p = send(&t, Some(&ev), &v, d); ++ wait_until(|| p.replied()); ++ in_command(|hold| p.settle(hold)); ++ assert_eq!(*s.lock(), vec![2]); ++ assert!(!ev.wait_until_signaled_value(42, 0)); ++ } ++ ++ /// An uncarried failure waits for the chain to reach its value before it signals the next: ++ /// the value never goes ahead of an earlier exec. ++ #[test] ++ fn an_uncarried_failure_waits_for_its_value() { ++ let h = harness(|ctx, m| match m { ++ ToService::ExecIndirect3 { reply, .. } => reply.call((2, None, 0)), ++ other => standard(ctx, other), ++ }); ++ let t = h.task(); ++ let ev = h.event(); ++ ev.set_signaled_value(40).unwrap(); ++ let v = Arc::new(AtomicU64::new(41)); ++ let (_s, d) = done(); ++ let p = send(&t, Some(&ev), &v, d); ++ wait_until(|| p.replied()); ++ let settled = Arc::new(std::sync::atomic::AtomicBool::new(false)); ++ let s2 = settled.clone(); ++ let th = std::thread::spawn(move || { ++ in_command(|hold| p.settle(hold)); ++ s2.store(true, Ordering::SeqCst); ++ }); ++ std::thread::sleep(Duration::from_millis(30)); ++ assert!(!settled.load(Ordering::SeqCst)); ++ assert!(!ev.wait_until_signaled_value(42, 0)); ++ ev.set_signaled_value(41).unwrap(); ++ th.join().unwrap(); ++ assert!(ev.wait_until_signaled_value(42, 0)); ++ } ++ ++ /// A crash during the exec: 4 from the crash, and the host moves the chain on. ++ #[test] ++ fn a_crash_moves_the_chain_on() { ++ let h = harness(dies_on(EXEC3)); ++ let t = h.task(); ++ let ev = h.event(); ++ ev.set_signaled_value(41).unwrap(); ++ let v = Arc::new(AtomicU64::new(41)); ++ let (s, d) = done(); ++ let p = send(&t, Some(&ev), &v, d); ++ wait_until(|| p.replied()); ++ in_command(|hold| p.settle(hold)); ++ assert_eq!(*s.lock(), vec![4]); ++ assert!(ev.wait_until_signaled_value(42, 0)); ++ } ++ ++ /// The reply's backings go to the kernel task at the settlement, not before. ++ #[test] ++ fn the_backings_are_invalidated_at_the_settlement() { ++ let h = kernel_harness(|ctx, m| match m { ++ ToService::ExecIndirect3 { ++ gpu_completion, ++ reply, ++ .. ++ } => { ++ reply.call((1, Some([7u32, 8].as_bytes().to_vec()), 2)); ++ ctx.complete(gpu_completion, 1); ++ } ++ other => standard(ctx, other), ++ }); ++ let t = h.task(); ++ let lines = Lines::capture(); ++ let v = Arc::new(AtomicU64::new(0)); ++ let (s, d) = done(); ++ let p = send(&t, None, &v, d); ++ wait_until(|| p.replied()); ++ wait_until(|| *s.lock() == vec![1]); ++ let miss = |l: &Vec| l.iter().filter(|l| l.starts_with("shared texture backing ")).count(); ++ assert_eq!(miss(&lines.take()), 0); ++ in_command(|hold| p.settle(hold)); ++ assert_eq!(miss(&lines.take()), 2); ++ } ++ } + } +diff --git a/crates/reims-vgpu-core/src/tasks/remote/completions.rs b/crates/reims-vgpu-core/src/tasks/remote/completions.rs +index 27142c19..7d04da8f 100644 +--- a/crates/reims-vgpu-core/src/tasks/remote/completions.rs ++++ b/crates/reims-vgpu-core/src/tasks/remote/completions.rs +@@ -110,6 +110,11 @@ impl Token { + self.cond.notify_all(); + } + ++ /// LOCAL (Atlas, VM-B): whether the token is complete, without waiting. ++ pub fn is_complete(&self) -> bool { ++ *imp::lock(&self.status) != ExecStatus::NOT_COMPLETE ++ } ++ + /// Waits, with no timeout, until the token is complete, and returns its status. + pub fn wait(&self) -> ExecStatus { + let mut st = imp::lock(&self.status); +diff --git a/crates/reims-vgpu-core/src/tasks/wire.rs b/crates/reims-vgpu-core/src/tasks/wire.rs +index 5b9aa5e2..e92cdc13 100644 +--- a/crates/reims-vgpu-core/src/tasks/wire.rs ++++ b/crates/reims-vgpu-core/src/tasks/wire.rs +@@ -190,6 +190,10 @@ pub enum ToService { + fifo_event_value_in: u64, + /// `gpuCompletion`. + gpu_completion: u64, ++ /// LOCAL (Atlas, VM-B): the FIFO pipelines its execs (`Fifo::pipelining`). A refusal or a ++ /// fault before the commit then still moves the FIFO's event chain on, with a fill ++ /// command buffer (`GpuTask::fill_chain`), which sets the flag. `None` on the base path. ++ fill_on_failure: Option>, + /// `reply`. + reply: Reply<(u32, Option>, u32)>, + }, diff --git a/patches/reims-vgpu/0027-REIMS_VGPU_GROUPED-1-boot-switch-for-grouped-complet.patch b/patches/reims-vgpu/0027-REIMS_VGPU_GROUPED-1-boot-switch-for-grouped-complet.patch new file mode 100644 index 0000000..76a9d9a --- /dev/null +++ b/patches/reims-vgpu/0027-REIMS_VGPU_GROUPED-1-boot-switch-for-grouped-complet.patch @@ -0,0 +1,34 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 15:51:31 +0000 +Subject: [PATCH 27/33] REIMS_VGPU_GROUPED=1 boot switch for grouped completion + (DEEP-GPU A/B) + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-vulkan/src/platform.rs | 12 ++++++++++++ + 1 file changed, 12 insertions(+) + +diff --git a/crates/reims-vgpu-vulkan/src/platform.rs b/crates/reims-vgpu-vulkan/src/platform.rs +index 8c7771db..f8b22b88 100644 +--- a/crates/reims-vgpu-vulkan/src/platform.rs ++++ b/crates/reims-vgpu-vulkan/src/platform.rs +@@ -273,6 +273,18 @@ pub(crate) fn device_options() -> &'static rt::DeviceOptions { + crate::os::add_window_extensions(&mut options); + options.gpu_times_from_host = true; + options.policy = exact_policy(std::env::var("REIMS_VGPU_EXACT").ok().as_deref()); ++ // LOCAL (Atlas, DEEP-GPU): `REIMS_VGPU_GROUPED=1`, read once per process, takes the ++ // runtime's grouped completion (`CompletionPolicy::Grouped`) for an A/B of kernel ++ // submissions per frame. It groups only where the driver fact ++ // `SubmissionsWithoutSignalsMerge` holds (the live switch `facts_on=` gives it), and the ++ // runtime announces `GroupedCompletion` once, relayed outside strict mode. ++ if std::env::var_os("REIMS_VGPU_GROUPED").is_some_and(|v| v == "1") { ++ options.policy.completion = metal2vulkan::device::CompletionPolicy::Grouped; ++ let _ = std::io::Write::write_all( ++ &mut std::io::stderr(), ++ b"reims-vgpu: REIMS_VGPU_GROUPED=1: grouped completion\n", ++ ); ++ } + let sink = rt::MessageSink::new(crate::answer::notice); + options.messages = Some(sink.with_least_severity(crate::answer::least_severity())); + options.announce_each = diff --git a/patches/reims-vgpu/0028-interrupt-moderation-off-unless-REIMS_VGPU_IRQ_MODER.patch b/patches/reims-vgpu/0028-interrupt-moderation-off-unless-REIMS_VGPU_IRQ_MODER.patch new file mode 100644 index 0000000..edffc85 --- /dev/null +++ b/patches/reims-vgpu/0028-interrupt-moderation-off-unless-REIMS_VGPU_IRQ_MODER.patch @@ -0,0 +1,297 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 17:07:53 +0000 +Subject: [PATCH 28/33] interrupt moderation (off unless + REIMS_VGPU_IRQ_MODERATE names an existing file) + +A raise goes out only while none is outstanding; a read of 0x1018/0x1014 ends the +outstanding one before the bits are taken, so no bit is left without a raise. A raise +held past 2 ms goes out anyway. Experiment, not for adoption. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-core/src/device.rs | 4 + + crates/reims-vgpu-core/src/device/fifos.rs | 4 +- + .../src/device/irq_moderation.rs | 181 ++++++++++++++++++ + .../reims-vgpu-core/src/device/lifecycle.rs | 1 + + crates/reims-vgpu-core/src/device/mmio.rs | 10 +- + crates/reims-vgpu-core/src/device/root.rs | 1 + + 6 files changed, 198 insertions(+), 3 deletions(-) + create mode 100644 crates/reims-vgpu-core/src/device/irq_moderation.rs + +diff --git a/crates/reims-vgpu-core/src/device.rs b/crates/reims-vgpu-core/src/device.rs +index 3f21d031..9a370356 100644 +--- a/crates/reims-vgpu-core/src/device.rs ++++ b/crates/reims-vgpu-core/src/device.rs +@@ -37,6 +37,7 @@ + pub mod config; + pub mod displays; + pub mod fifos; ++pub mod irq_moderation; + pub mod lifecycle; + pub mod mmio; + pub mod root; +@@ -953,6 +954,8 @@ pub struct DeviceCore { + display_port_count: AtomicU32, + display_port_connection_types: AtomicU32, + interrupt_display: AtomicU32, ++ /// LOCAL (Atlas, ARCH): interrupt moderation (off unless switched on). ++ irq_moderation: irq_moderation::Moderation, + displays: StagedMutex>, + interrupt_stamp: AtomicU32, + stamp_memory: Mutex>, +@@ -1257,6 +1260,7 @@ impl DeviceCore

{ + display_port_count: AtomicU32::new(display_port_count), + display_port_connection_types: AtomicU32::new(display_port_connection_types), + interrupt_display: AtomicU32::new(0), ++ irq_moderation: irq_moderation::Moderation::default(), + displays: StagedMutex::new(DisplaysLocked { + display_port_busy_mask: 0, + guest_display: Default::default(), +diff --git a/crates/reims-vgpu-core/src/device/fifos.rs b/crates/reims-vgpu-core/src/device/fifos.rs +index 717bacb3..6cbf844e 100644 +--- a/crates/reims-vgpu-core/src/device/fifos.rs ++++ b/crates/reims-vgpu-core/src/device/fifos.rs +@@ -428,7 +428,9 @@ impl DeviceCore

{ + }; + // irq: the guest interrupt raised + crate::trace::event("irq", &[]); +- raise(0); ++ if self.irq_moderation.should_raise() { ++ raise(0); ++ } + } + } + +diff --git a/crates/reims-vgpu-core/src/device/irq_moderation.rs b/crates/reims-vgpu-core/src/device/irq_moderation.rs +new file mode 100644 +index 00000000..987dfbb0 +--- /dev/null ++++ b/crates/reims-vgpu-core/src/device/irq_moderation.rs +@@ -0,0 +1,181 @@ ++//! LOCAL (Atlas, ARCH experiment, branch atlas/arch-irqmod): interrupt moderation. ++//! ++//! Every stamp signal and display poke raised the guest interrupt (~100 raises per Sim frame, ++//! ~1.9 per read of the status under load). The status the guest's ISR reads is read-to-clear ++//! (0x1018 stamps, 0x1014 display) and its completion is level-based on stamp memory ++//! (`notifyStamp` reads the stamp values): a raise while an earlier one is not yet read adds ++//! nothing the guest's next read would not see. With the switch on, a raise goes out only when ++//! none is outstanding; a read of 0x1018 or 0x1014 ends the outstanding one *before* the bits ++//! are taken, so a bit set after the read always raises again (no lost interrupt): ++//! - writer: set bit, then `pending.swap(true)`; raise when it was false; ++//! - reader: `pending.store(false)`, then take the bits. ++//! ++//! A bit set before the read is either taken by it, or its writer's swap came after the ++//! store and raised. As a safety net a raise held more than [`RESCUE`] after the outstanding ++//! one goes out anyway (a raise the guest never read: MSI off then, a lost message). ++//! A reset or a restore ends the outstanding raise. ++//! ++//! The switch: `REIMS_VGPU_IRQ_MODERATE=`; moderation is on while that file exists, ++//! looked at most once a second. Unset: every raise goes out, as before. ++ ++use std::sync::OnceLock; ++use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; ++use std::time::{Duration, Instant}; ++ ++/// The longest a raise is held behind an outstanding one. ++pub const RESCUE: Duration = Duration::from_millis(2); ++ ++const LOOK_EVERY_NS: u64 = 1_000_000_000; ++const LOG_EVERY_NS: u64 = 10_000_000_000; ++ ++fn base() -> Instant { ++ static B: OnceLock = OnceLock::new(); ++ *B.get_or_init(Instant::now) ++} ++ ++fn now_ns() -> u64 { ++ u64::try_from(base().elapsed().as_nanos()).unwrap_or(u64::MAX) ++} ++ ++struct Switch { ++ path: Option, ++ on: AtomicBool, ++ /// When the file was last looked at (ns after `base`, `u64::MAX` before the first look). ++ looked: AtomicU64, ++} ++ ++fn switch() -> &'static Switch { ++ static S: OnceLock = OnceLock::new(); ++ S.get_or_init(|| Switch { ++ path: std::env::var_os("REIMS_VGPU_IRQ_MODERATE") ++ .filter(|p| !p.is_empty()) ++ .map(std::path::PathBuf::from), ++ on: AtomicBool::new(false), ++ looked: AtomicU64::new(u64::MAX), ++ }) ++} ++ ++fn switched_on(now: u64) -> bool { ++ let s = switch(); ++ let Some(path) = &s.path else { ++ return false; ++ }; ++ let looked = s.looked.load(Ordering::Relaxed); ++ if (looked == u64::MAX || now.saturating_sub(looked) >= LOOK_EVERY_NS) ++ && s ++ .looked ++ .compare_exchange(looked, now, Ordering::Relaxed, Ordering::Relaxed) ++ .is_ok() ++ { ++ let on = path.exists(); ++ if s.on.swap(on, Ordering::SeqCst) != on { ++ log::info!("irq moderation {}", if on { "on" } else { "off" }); ++ } ++ } ++ s.on.load(Ordering::SeqCst) ++} ++ ++/// A device's moderation state. ++#[derive(Default)] ++pub struct Moderation { ++ pending: AtomicBool, ++ since: AtomicU64, ++ raised: AtomicU64, ++ held: AtomicU64, ++ rescued: AtomicU64, ++ logged: AtomicU64, ++} ++ ++impl Moderation { ++ /// Whether this raise goes out (always, with the switch off). ++ pub fn should_raise(&self) -> bool { ++ let now = now_ns(); ++ if !switched_on(now) { ++ return true; ++ } ++ self.log(now); ++ self.decide(now) ++ } ++ ++ /// The moderation itself, with the switch on. ++ fn decide(&self, now: u64) -> bool { ++ if !self.pending.swap(true, Ordering::SeqCst) { ++ self.since.store(now, Ordering::SeqCst); ++ self.raised.fetch_add(1, Ordering::Relaxed); ++ return true; ++ } ++ let since = self.since.load(Ordering::SeqCst); ++ let rescue = u64::try_from(RESCUE.as_nanos()).unwrap_or(u64::MAX); ++ if now.saturating_sub(since) > rescue ++ && self ++ .since ++ .compare_exchange(since, now, Ordering::SeqCst, Ordering::SeqCst) ++ .is_ok() ++ { ++ self.rescued.fetch_add(1, Ordering::Relaxed); ++ return true; ++ } ++ self.held.fetch_add(1, Ordering::Relaxed); ++ false ++ } ++ ++ /// The guest read its status (or the device reset or restored): the outstanding raise ends. ++ /// Before the bits are taken. ++ pub fn acked(&self) { ++ self.pending.store(false, Ordering::SeqCst); ++ } ++ ++ fn log(&self, now: u64) { ++ let last = self.logged.load(Ordering::Relaxed); ++ if now.saturating_sub(last) >= LOG_EVERY_NS ++ && self ++ .logged ++ .compare_exchange(last, now, Ordering::Relaxed, Ordering::Relaxed) ++ .is_ok() ++ { ++ log::info!( ++ "irq moderation: raised {} held {} rescued {}", ++ self.raised.load(Ordering::Relaxed), ++ self.held.load(Ordering::Relaxed), ++ self.rescued.load(Ordering::Relaxed) ++ ); ++ } ++ } ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ ++ const MS: u64 = 1_000_000; ++ ++ #[test] ++ fn holds_behind_an_outstanding_raise_until_the_read() { ++ let m = Moderation::default(); ++ assert!(m.decide(10 * MS)); // first raise goes out ++ assert!(!m.decide(10 * MS + 1)); // held: the guest has not read ++ assert!(!m.decide(11 * MS)); ++ m.acked(); // the ISR's read of 0x1018, before it takes the bits ++ assert!(m.decide(11 * MS + 1)); // a bit after the read raises again ++ assert_eq!(m.held.load(Ordering::Relaxed), 2); ++ } ++ ++ #[test] ++ fn rescues_a_raise_the_guest_never_read() { ++ let m = Moderation::default(); ++ assert!(m.decide(0)); ++ assert!(!m.decide(MS)); ++ assert!(m.decide(3 * MS)); // past RESCUE (2 ms) ++ assert!(!m.decide(3 * MS + 1)); // the rescue restarts the clock ++ assert_eq!(m.rescued.load(Ordering::Relaxed), 1); ++ } ++ ++ #[test] ++ fn off_without_the_switch() { ++ let m = Moderation::default(); ++ for _ in 0..4 { ++ assert!(m.should_raise()); ++ } ++ assert!(!m.pending.load(Ordering::SeqCst)); ++ } ++} +diff --git a/crates/reims-vgpu-core/src/device/lifecycle.rs b/crates/reims-vgpu-core/src/device/lifecycle.rs +index bd00794d..76b663a2 100644 +--- a/crates/reims-vgpu-core/src/device/lifecycle.rs ++++ b/crates/reims-vgpu-core/src/device/lifecycle.rs +@@ -421,6 +421,7 @@ impl DeviceCore

{ + .store(device_state.interrupt_display, Ordering::SeqCst); + self.interrupt_stamp + .store(device_state.interrupt_stamp, Ordering::SeqCst); ++ self.irq_moderation.acked(); + self.interrupt_fault_mask.store(0, Ordering::SeqCst); + { + let e = self.displays.atomics(); +diff --git a/crates/reims-vgpu-core/src/device/mmio.rs b/crates/reims-vgpu-core/src/device/mmio.rs +index 8f9d2de5..9571f0d9 100644 +--- a/crates/reims-vgpu-core/src/device/mmio.rs ++++ b/crates/reims-vgpu-core/src/device/mmio.rs +@@ -156,8 +156,14 @@ impl DeviceCore

{ + let fifo = self.root_fifo.atomics().root_fifo(); + fifo.and_then(|f| f.root_read()).unwrap_or(0) + } +- Register::InterruptDisplay => self.interrupt_display.swap(0, Ordering::SeqCst), +- Register::InterruptStamp => self.interrupt_stamp.swap(0, Ordering::SeqCst), ++ Register::InterruptDisplay => { ++ self.irq_moderation.acked(); ++ self.interrupt_display.swap(0, Ordering::SeqCst) ++ } ++ Register::InterruptStamp => { ++ self.irq_moderation.acked(); ++ self.interrupt_stamp.swap(0, Ordering::SeqCst) ++ } + Register::RootPage => self + .root_task + .atomics() +diff --git a/crates/reims-vgpu-core/src/device/root.rs b/crates/reims-vgpu-core/src/device/root.rs +index ad965a9f..d14647ce 100644 +--- a/crates/reims-vgpu-core/src/device/root.rs ++++ b/crates/reims-vgpu-core/src/device/root.rs +@@ -163,6 +163,7 @@ impl DeviceCore

{ + self.interrupt_fault_mask.store(0, Ordering::SeqCst); + self.interrupt_stamp.store(0, Ordering::SeqCst); + self.interrupt_display.store(0, Ordering::SeqCst); ++ self.irq_moderation.acked(); + { + let _rt = self.root_task.lock(); + let info = self.root_task.atomics().task_info(); diff --git a/patches/reims-vgpu/0029-Integrate-guarded-render-lowering-and-multi-queue-re.patch b/patches/reims-vgpu/0029-Integrate-guarded-render-lowering-and-multi-queue-re.patch new file mode 100644 index 0000000..66ef613 --- /dev/null +++ b/patches/reims-vgpu/0029-Integrate-guarded-render-lowering-and-multi-queue-re.patch @@ -0,0 +1,382 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Fri, 6 Nov 2026 11:44:38 +0000 +Subject: [PATCH 29/33] Integrate guarded render lowering and multi-queue + regression coverage + +Integrate CPU lane 0f1cb0f and 3a75482 while retaining the integration tree m2v patch paths. Worker remains opt-in and is refused under strict, census and EXACT=all. Isolated lavapipe evidence is recorded in DEEP-CPU-PARALLEL-20261106.md; the combined tree still needs its gate. +--- + .../src/gpu/encode_render.rs | 220 ++++++++++++++++++ + crates/reims-vgpu-vulkan/src/platform.rs | 114 +++++++++ + 2 files changed, 334 insertions(+) + +diff --git a/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs b/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs +index 82e1a207..67a2e34b 100644 +--- a/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs ++++ b/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs +@@ -4022,6 +4022,226 @@ pub(crate) mod tests { + assert_eq!(event.signaled_value(), 5); + } + ++ /// LOCAL (Atlas, DEEP-CPU): a guest task's stream of command buffers on one queue, each a ++ /// fill, then a cleared pass of 1 to 40 draws, a completed handler and a scheduled signal ++ /// after the pass's end (in the order `gpu_task/exec.rs` encodes an exec's tail), runs on a device that lowers render calls on the runtime's ++ /// lowering thread (`REIMS_VGPU_RENDER_LOWERING=worker`, [`crate::platform::render_lowering`]) ++ /// as on one that lowers them at the call: the same texels and bytes, the event at the last ++ /// value, every command buffer completed, in commit order. The worker device's counts show its ++ /// ends left to the lowering thread, the at-call device's none ++ #[test] ++ fn render_calls_lowered_on_the_worker_run_as_at_the_call() { ++ use metal2vulkan::device::RenderLoweringPolicy; ++ const N: usize = 24; ++ let run = |d: &crate::gpu::device::VulkanDevice| exec_shaped_stream(d, N); ++ let at_call = dev(); ++ let mut policy = crate::platform::device_options().policy; ++ assert_eq!(policy.render_lowering, RenderLoweringPolicy::AtCall); ++ policy.render_lowering = RenderLoweringPolicy::Worker; ++ let worker = crate::gpu::device::tests::device_with_policy(policy); ++ let before = (at_call.raw.work_counts(), worker.raw.work_counts()); ++ let (a, w) = (run(&at_call), run(&worker)); ++ let after = (at_call.raw.work_counts(), worker.raw.work_counts()); ++ let expected: Vec<_> = (0..N).map(|i| (i, 4)).collect(); ++ assert_eq!(a.0, expected, "at the call: completed in order"); ++ assert_eq!(w.0, expected, "worker: completed in order"); ++ assert_eq!(w.1, a.1, "texels"); ++ assert!( ++ a.1.iter().any(|t| t.iter().any(|&b| b != 0)), ++ "the draws drew" ++ ); ++ let filled: Vec = (0..N) ++ .flat_map(|i| [u8::try_from(i + 1).unwrap(); 4]) ++ .collect(); ++ assert_eq!((a.2.clone(), w.2), (filled.clone(), filled)); ++ assert_eq!((a.3, w.3), (N as u64, N as u64)); ++ eprintln!( ++ "lowering counts: at_call {:?} worker {:?}", ++ lowering_counts(&before.0, &after.0), ++ lowering_counts(&before.1, &after.1) ++ ); ++ let ends = |c: &metal2vulkan_runtime::WorkCounts| c.render_log_ends_by_thread; ++ assert_eq!( ++ ends(&after.0), ++ ends(&before.0), ++ "at the call: nothing left to the thread" ++ ); ++ assert!( ++ ends(&after.1) > ends(&before.1), ++ "worker: ends left to the lowering thread" ++ ); ++ assert!( ++ after.1.render_log_batches_by_thread + after.1.render_log_batches_by_program ++ > before.1.render_log_batches_by_thread + before.1.render_log_batches_by_program, ++ "worker: the 40-draw passes logged batches" ++ ); ++ } ++ ++ /// LOCAL (Atlas, DEEP-CPU): two Sims' exec streams at once, each on its own thread and queue, ++ /// lower on the device's lanes (m2v `render/lowering_worker.rs`, one lane per queue in turn) ++ /// and give what they give lowered at the call: per queue, every command buffer completed in ++ /// its commit order with status 4, the same texels and bytes, the event at the last value. ++ /// The worker's ends ran off the program's threads. The lanes' CPU time against the wall ++ /// time is printed (diagnostic only: lavapipe, debug, not a frame-rate figure). ++ #[test] ++ fn two_sims_lowered_on_the_lanes_run_as_at_the_call() { ++ use metal2vulkan::device::RenderLoweringPolicy; ++ const N: usize = 24; ++ let run = |d: &crate::gpu::device::VulkanDevice| { ++ std::thread::scope(|scope| { ++ let sims: Vec<_> = (0..2) ++ .map(|_| scope.spawn(|| exec_shaped_stream(d, N))) ++ .collect(); ++ sims.into_iter() ++ .map(|sim| sim.join().unwrap()) ++ .collect::>() ++ }) ++ }; ++ let at_call = dev(); ++ let mut policy = crate::platform::device_options().policy; ++ policy.render_lowering = RenderLoweringPolicy::Worker; ++ let worker = crate::gpu::device::tests::device_with_policy(policy); ++ let a = run(&at_call); ++ let (before, lanes_before) = (worker.raw.work_counts(), lane_cpu_nanos()); ++ let wall = std::time::Instant::now(); ++ let w = run(&worker); ++ let wall = wall.elapsed().as_nanos() as u64; ++ let lanes_after = lane_cpu_nanos(); ++ let after = worker.raw.work_counts(); ++ let expected: Vec<_> = (0..N).map(|i| (i, 4)).collect(); ++ let filled: Vec = (0..N) ++ .flat_map(|i| [u8::try_from(i + 1).unwrap(); 4]) ++ .collect(); ++ for (sim, (a, w)) in a.iter().zip(&w).enumerate() { ++ assert_eq!(a.0, expected, "at the call, sim {sim}: completed in order"); ++ assert_eq!(w.0, expected, "worker, sim {sim}: completed in order"); ++ assert_eq!(w.1, a.1, "sim {sim}: texels"); ++ assert!( ++ a.1.iter().any(|t| t.iter().any(|&b| b != 0)), ++ "sim {sim}: drew" ++ ); ++ assert_eq!((&a.2, &w.2), (&filled, &filled), "sim {sim}: filled"); ++ assert_eq!((a.3, w.3), (N as u64, N as u64), "sim {sim}: event"); ++ } ++ let counts = lowering_counts(&before, &after); ++ let lane_nanos: Vec = (lanes_after.iter()) ++ .map(|(tid, n)| { ++ n - lanes_before ++ .iter() ++ .find(|(t, _)| t == tid) ++ .map_or(0, |b| b.1) ++ }) ++ .filter(|&n| n > 0) ++ .collect(); ++ eprintln!( ++ "two sims, worker, {} lanes: counts {counts:?}; lowering threads busy {lane_nanos:?} ns \ ++ of {wall} ns wall (sum/wall {:.2})", ++ lane_nanos.len(), ++ lane_nanos.iter().sum::() as f64 / wall as f64 ++ ); ++ assert!(counts[2] > 0, "worker: ends left to the lowering threads"); ++ } ++ ++ /// The device's lowering threads' (`metal2vulkan-lowering`) CPU time in ns, by thread id ++ /// (`/proc/self/task/*/schedstat`). ++ fn lane_cpu_nanos() -> Vec<(u64, u64)> { ++ let Ok(tasks) = std::fs::read_dir("/proc/self/task") else { ++ return Vec::new(); ++ }; ++ (tasks.flatten()) ++ .filter_map(|task| { ++ let comm = std::fs::read_to_string(task.path().join("comm")).ok()?; ++ if !comm.starts_with("metal2vulkan-lo") { ++ return None; ++ } ++ let stat = std::fs::read_to_string(task.path().join("schedstat")).ok()?; ++ let tid = task.file_name().to_str()?.parse().ok()?; ++ Some((tid, stat.split_whitespace().next()?.parse().ok()?)) ++ }) ++ .collect() ++ } ++ ++ /// What [`exec_shaped_stream`] reads: the completions in order (index, status), the three ++ /// targets' texels, the filled bytes and the event's value. ++ type ExecStream = (Vec<(usize, u64)>, Vec>, Vec, u64); ++ ++ /// `n` command buffers on a new queue of `d`, in the order Reims's exec makes them: a fill, ++ /// a cleared pass of 1-40 draws over 3 rotating targets, end, completed handler, scheduled ++ /// signal on one shared event, commit. What it read once all completed. ++ fn exec_shaped_stream(d: &crate::gpu::device::VulkanDevice, n: usize) -> ExecStream { ++ use reims_vgpu_core::platform::types::NsRange; ++ let p = platform(); ++ let state = fixture_state::(d, "vert:tri_vert", "frag:tri_frag"); ++ let targets: Vec<_> = (0..3).map(|_| pass_of(&p, d, (4, 4))).collect(); ++ let bytes = buffer::(d, (n * 4) as u64, 0, &vec![0; n * 4]); ++ let event = d.new_shared_event().unwrap(); ++ let q = d.new_command_queue().unwrap(); ++ let (done, completed) = std::sync::mpsc::channel(); ++ for i in 0..n { ++ let cb = q.command_buffer(OnFault::Die).unwrap().unwrap(); ++ let blit = cb.blit_encoder(OnFault::Die).unwrap().unwrap(); ++ let fill = BlitCmd::FillBuffer { ++ buffer: &bytes, ++ range: NsRange::new((i * 4) as u64, 4), ++ value: u8::try_from(i + 1).unwrap(), ++ }; ++ blit.encode(OnFault::Die, fill).unwrap(); ++ blit.encode(OnFault::Die, BlitCmd::EndEncoding).unwrap(); ++ let e = cb ++ .render_encoder(OnFault::Die, &targets[i % 3].1) ++ .unwrap() ++ .unwrap(); ++ let mut cmds = vec![RenderCmd::SetRenderPipelineState { ++ pipeline_state: Some(&state), ++ }]; ++ let draws = if i % 4 == 0 { 40 } else { i % 4 }; ++ cmds.extend((0..draws).map(|_| three(3, None))); ++ cmds.push(RenderCmd::EndEncoding); ++ send_all(&e, cmds); ++ let done = done.clone(); ++ cb.add_completed_handler( ++ OnFault::Die, ++ Box::new(move |info| done.send((i, info.status.0)).unwrap()), ++ ) ++ .unwrap(); ++ cb.encode_signal_when_scheduled(Some(&event), (i + 1) as u64) ++ .unwrap(); ++ cb.commit(OnFault::Die).unwrap(); ++ } ++ let order: Vec<_> = (0..n) ++ .map(|_| { ++ completed ++ .recv_timeout(std::time::Duration::from_secs(30)) ++ .unwrap() ++ }) ++ .collect(); ++ let texels: Vec<_> = targets ++ .iter() ++ .map(|(t, _)| crate::gpu::resource::tests::read_back(t)) ++ .collect(); ++ // SAFETY: a Shared buffer's bytes, read after every command buffer completed. ++ let filled = ++ unsafe { std::slice::from_raw_parts(bytes.contents().as_ptr::(), n * 4) }.to_vec(); ++ (order, texels, filled, event.signaled_value()) ++ } ++ ++ /// The lowering counts from `b` to `a`: batches by thread and program, ends by thread and ++ /// program, commits run by thread and program, waits. ++ fn lowering_counts( ++ b: &metal2vulkan_runtime::WorkCounts, ++ a: &metal2vulkan_runtime::WorkCounts, ++ ) -> [u64; 7] { ++ [ ++ a.render_log_batches_by_thread - b.render_log_batches_by_thread, ++ a.render_log_batches_by_program - b.render_log_batches_by_program, ++ a.render_log_ends_by_thread - b.render_log_ends_by_thread, ++ a.render_log_ends_by_program - b.render_log_ends_by_program, ++ a.render_log_commits_by_thread - b.render_log_commits_by_thread, ++ a.render_log_commits_by_program - b.render_log_commits_by_program, ++ a.render_log_waits - b.render_log_waits, ++ ] ++ } ++ + /// What a command buffer of a new queue shows on `p`'s device `d` whose render pass, over a + /// 4 by 4 target filled with 0x11, binds a memoryless 4 by 4 `RGBA8Unorm` texture and a + /// sampler at fragment slot 0 and draws a triangle over the target with the fixtures' +diff --git a/crates/reims-vgpu-vulkan/src/platform.rs b/crates/reims-vgpu-vulkan/src/platform.rs +index f8b22b88..b957171b 100644 +--- a/crates/reims-vgpu-vulkan/src/platform.rs ++++ b/crates/reims-vgpu-vulkan/src/platform.rs +@@ -285,6 +285,18 @@ pub(crate) fn device_options() -> &'static rt::DeviceOptions { + b"reims-vgpu: REIMS_VGPU_GROUPED=1: grouped completion\n", + ); + } ++ // LOCAL (Atlas, DEEP-CPU): `REIMS_VGPU_RENDER_LOWERING=worker`, read once per process, ++ // takes the runtime's lowering thread for render encoders ++ // (`RenderLoweringPolicy::Worker`): see [`render_lowering`]. ++ let (lowering, line) = render_lowering( ++ std::env::var_os(RENDER_LOWERING_VAR).as_deref(), ++ crate::answer::strict_process() || crate::answer::census::on(), ++ std::env::var("REIMS_VGPU_EXACT").ok().as_deref(), ++ ); ++ options.policy.render_lowering = lowering; ++ if let Some(line) = line { ++ let _ = std::io::Write::write_all(&mut std::io::stderr(), line.as_bytes()); ++ } + let sink = rt::MessageSink::new(crate::answer::notice); + options.messages = Some(sink.with_least_severity(crate::answer::least_severity())); + options.announce_each = +@@ -378,6 +390,76 @@ fn exact_policy(exact: Option<&str>) -> metal2vulkan::device::DevicePolicy { + policy + } + ++/// LOCAL (Atlas, DEEP-CPU): the variable that picks where render encoders' calls are lowered. ++pub const RENDER_LOWERING_VAR: &str = "REIMS_VGPU_RENDER_LOWERING"; ++ ++/// Where the device lowers its render encoders' calls (`DevicePolicy::render_lowering`), from ++/// `REIMS_VGPU_RENDER_LOWERING` (`value`), and the stderr line to write, if any. ++/// ++/// Unset, empty or `at_call`: inside each call, the runtime's default ++/// (`RenderLoweringPolicy::AtCall`), production's form. `worker`: the runtime logs each render ++/// call after its checks at the call and lowers it, the encoder's end and the commit's own work ++/// on the device's lowering thread, in order (metal2vulkan docs/API.md section 1, ++/// `render/deferral.rs`). The guest task's service thread then decodes the next commands while ++/// that thread lowers the earlier ones: two stages of one task's serial stream, its order ++/// unchanged. Commit order across the device's queues stays the program's ++/// (`render/queued_commits.rs`), so every FIFO's event chain, scheduled signal and completion ++/// keeps its order. ++/// ++/// What changes under `worker`, and why it is kept out of audits: ++/// - A refusal only the lowering finds (a runtime gap of the draw's variant, out of memory, a ++/// Vulkan error) fails the command buffer (`CallRefused`, completion status 3) instead of ++/// raising at the call (the exec's reply 2). Refusals the call's checks find still return ++/// from the call. ++/// - A degradation announced while the lowering thread lowers a call is announced outside every ++/// call of this platform: `answer::hearing` does not hear it (the `HalfOutputPrecision` relay ++/// of `binds`), census attributes it to no selector, and strict mode dies on that thread rather ++/// than at the call. So `worker` is refused, with a line, in a strict or census process ++/// (`audited`) and under `REIMS_VGPU_EXACT=all`, whose exact answers take `AtCall`. ++/// ++/// Another value changes nothing, and a line says so. ++pub(crate) fn render_lowering( ++ value: Option<&std::ffi::OsStr>, ++ audited: bool, ++ exact: Option<&str>, ++) -> (metal2vulkan::device::RenderLoweringPolicy, Option) { ++ use metal2vulkan::device::RenderLoweringPolicy::{AtCall, Worker}; ++ let Some(value) = value else { ++ return (AtCall, None); ++ }; ++ let value = value.to_string_lossy(); ++ match value.trim() { ++ "" | "at_call" => (AtCall, None), ++ "worker" => { ++ let exact_all = exact ++ .unwrap_or_default() ++ .split(',') ++ .any(|n| n.trim() == EXACT_ALL); ++ if audited || exact_all { ++ let why = if audited { "strict or census mode" } else { "REIMS_VGPU_EXACT=all" }; ++ let line = format!( ++ "reims-vgpu: {RENDER_LOWERING_VAR}=worker refused under {why}: render calls \ ++ lowered at the call\n" ++ ); ++ (AtCall, Some(line)) ++ } else { ++ let line = format!( ++ "reims-vgpu: {RENDER_LOWERING_VAR}=worker: render calls lowered on the \ ++ runtime's lowering thread\n" ++ ); ++ (Worker, Some(line)) ++ } ++ } ++ other => ( ++ AtCall, ++ Some(format!( ++ "reims-vgpu: {RENDER_LOWERING_VAR} names {other:?}, none of at_call, worker: \ ++ ignored\n" ++ )), ++ ), ++ } ++} ++ + /// The shader cache's directory under `HOME`, which every process of the library shares. It + /// holds the runtime's translations of the guest's functions and each pipeline's driver record + /// (MoltenVK's included: about 1.7 MB for a macOS 26 guest's `Xgc` pipeline +@@ -594,6 +676,38 @@ pub(crate) mod tests { + use super::*; + use crate::gpu::device::tests::dev; + ++ /// LOCAL (Atlas, DEEP-CPU): `REIMS_VGPU_RENDER_LOWERING` is off by default and off for ++ /// anything but `worker`; `worker` takes the lowering thread unless the process audits ++ /// (strict or census mode) or `REIMS_VGPU_EXACT` names `all` ++ #[test] ++ fn render_lowering_takes_the_worker_only_when_asked_and_unaudited() { ++ use metal2vulkan::device::RenderLoweringPolicy::{AtCall, Worker}; ++ use std::ffi::OsStr; ++ assert_eq!(render_lowering(None, false, None), (AtCall, None)); ++ for quiet in ["", " ", "at_call", " at_call "] { ++ assert_eq!(render_lowering(Some(OsStr::new(quiet)), false, None), (AtCall, None)); ++ } ++ for other in ["Worker", "1", "on", "worker,", "atcall"] { ++ let (p, line) = render_lowering(Some(OsStr::new(other)), false, None); ++ assert_eq!(p, AtCall, "{other}"); ++ assert!(line.unwrap().contains("ignored"), "{other}"); ++ } ++ let worker = Some(OsStr::new("worker")); ++ let (p, line) = render_lowering(worker, false, None); ++ assert_eq!(p, Worker); ++ assert!(line.unwrap().contains("lowering thread")); ++ assert_eq!( ++ render_lowering(worker, false, Some("batch_order, host_page_textures")).0, ++ Worker ++ ); ++ for (audited, exact) in [(true, None), (false, Some("all")), (false, Some("batch_order, all"))] ++ { ++ let (p, line) = render_lowering(worker, audited, exact); ++ assert_eq!(p, AtCall, "{audited} {exact:?}"); ++ assert!(line.unwrap().contains("refused")); ++ } ++ } ++ + /// The kinds a call hears and those that die by default are announced at every occurrence in + /// every mode, and every other kind of `DIES` only in strict and census mode + #[test] diff --git a/patches/reims-vgpu/0030-REIMS_VGPU_PAGED_BUFFER_BY_PAGES-1-finds-kept-paged-.patch b/patches/reims-vgpu/0030-REIMS_VGPU_PAGED_BUFFER_BY_PAGES-1-finds-kept-paged-.patch new file mode 100644 index 0000000..703d417 --- /dev/null +++ b/patches/reims-vgpu/0030-REIMS_VGPU_PAGED_BUFFER_BY_PAGES-1-finds-kept-paged-.patch @@ -0,0 +1,70 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 8 Nov 2026 21:47:42 +0000 +Subject: [PATCH 30/33] REIMS_VGPU_PAGED_BUFFER_BY_PAGES=1 finds kept paged + buffers by the guest pages they show + +--- + .../src/gpu_task/paged_buffers.rs | 35 ++++++++++++++----- + 1 file changed, 27 insertions(+), 8 deletions(-) + +diff --git a/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs b/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs +index 79384ec9..38d10c01 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs +@@ -65,6 +65,12 @@ pub(crate) fn entries_by_env() -> usize { + }) + } + ++/// LOCAL (Atlas): whether entries are found by the guest pages they show (read once). ++fn by_pages() -> bool { ++ static ON: OnceLock = OnceLock::new(); ++ *ON.get_or_init(|| std::env::var("REIMS_VGPU_PAGED_BUFFER_BY_PAGES").as_deref() == Ok("1")) ++} ++ + /// The mode [`SKIP_ENV_VAR`] asks for (read once): 0 without it, or for any other value. + pub(crate) fn skip_mode_by_env() -> u8 { + static MODE: OnceLock = OnceLock::new(); +@@ -335,10 +341,18 @@ impl PagedBuffers { + now: MonoTime, + ) -> (Option, Gone) { + let mut entries = self.entries.lock(); +- let Some(at) = entries +- .iter() +- .position(|e| e.page == page && e.length == length) +- else { ++ // LOCAL (Atlas): `REIMS_VGPU_PAGED_BUFFER_BY_PAGES=1` finds an entry by the guest pages it ++ // shows, at any task page: a view shows those physical pages whichever task range maps ++ // them, and the compositor's recycled ranges come back over pages of a few surfaces. ++ let at = match by_pages() { ++ true => entries ++ .iter() ++ .position(|e| e.length == length && e.lease == *lease && e.runs.same_as(runs)), ++ false => entries ++ .iter() ++ .position(|e| e.page == page && e.length == length), ++ }; ++ let Some(at) = at else { + return (None, Gone(Vec::new())); + }; + let Some(mut entry) = entries.remove(at) else { +@@ -370,10 +384,15 @@ impl PagedBuffers { + ) -> Gone { + let mut gone = Vec::new(); + let mut entries = self.entries.lock(); +- if let Some(at) = entries +- .iter() +- .position(|e| e.page == page && e.length == length) +- { ++ let at = match by_pages() { ++ true => entries ++ .iter() ++ .position(|e| e.length == length && e.lease == lease && e.runs.same_as(&runs)), ++ false => entries ++ .iter() ++ .position(|e| e.page == page && e.length == length), ++ }; ++ if let Some(at) = at { + gone.extend(entries.remove(at)); + } + while entries.len() >= self.capacity() { diff --git a/patches/reims-vgpu/0031-staged-page-ups-a-new-texture-s-first-page-up-copies.patch b/patches/reims-vgpu/0031-staged-page-ups-a-new-texture-s-first-page-up-copies.patch new file mode 100644 index 0000000..5acf549 --- /dev/null +++ b/patches/reims-vgpu/0031-staged-page-ups-a-new-texture-s-first-page-up-copies.patch @@ -0,0 +1,698 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 05:28:43 +0000 +Subject: [PATCH 31/33] staged page-ups: a new texture's first page-up copies + its guest pages into a runtime buffer when no live view covers them + +A new paged texture's first page-up went through a view: one mmap per run of guest pages +(~600 for a 2.4 MB texture of the Sim's compositor), a Vulkan import and, at its release, a +munmap that flushes the TLB once per run on every CPU the process ran on, the vCPUs too. +Measured with one Sim: ~520 views/s, all first page-ups of new Shared render targets, 15% +of the Sim task thread, ~57k TLB shootdowns/s. + +memory::page_refs counts the live views over each guest page (new_range_mapping adds, +Mapping's last release removes). The GPU reaches guest RAM only through such views, each +alive until the runtime frees its last buffer, after the GPU's last use; so pages no live +view covers have no GPU access pending, and a CPU copy at the exec reads the bytes the +view's blit would read. Covered pages take the view as ever. + +Switch: REIMS_VGPU_STAGED_PAGE_UP= counts from the start and stages while the file +exists (looked at once a second). Unset: nothing counted, nothing changed. +--- + crates/reims-vgpu-core/src/gpu_task/memory.rs | 72 +++++ + .../reims-vgpu-core/src/gpu_task/service.rs | 9 + + crates/reims-vgpu-core/src/memory.rs | 1 + + crates/reims-vgpu-core/src/memory/map.rs | 41 ++- + crates/reims-vgpu-core/src/memory/mapping.rs | 36 ++- + .../reims-vgpu-core/src/memory/page_refs.rs | 262 ++++++++++++++++++ + crates/reims-vgpu-core/src/resources/host.rs | 13 + + .../reims-vgpu-core/src/resources/texture.rs | 13 + + .../src/resources/texture/blit.rs | 37 ++- + 9 files changed, 474 insertions(+), 10 deletions(-) + create mode 100644 crates/reims-vgpu-core/src/memory/page_refs.rs + +diff --git a/crates/reims-vgpu-core/src/gpu_task/memory.rs b/crates/reims-vgpu-core/src/gpu_task/memory.rs +index f6fbc6a0..936ccb15 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/memory.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/memory.rs +@@ -29,6 +29,8 @@ use crate::memory::mapping::Mapping; + use crate::num::{ToU64, ToUsize}; + use crate::platform::Platform; + use crate::platform::device::GpuDevice; ++use crate::platform::resource::Buffer as _; ++use crate::platform::values::BufferDesc; + use crate::platform::os::clock::{Clock, MonoTime}; + use crate::platform::types::ResourceAddressRange; + use crate::raise::{Kept, keep_forever}; +@@ -565,6 +567,76 @@ impl GpuTask

{ + Ok(buffer) + } + ++ /// LOCAL (Atlas): a new buffer holding a copy of the task's pages `page..page + length`, for ++ /// a texture's first page-up, when page-ups are staged and no live view covers any of the ++ /// pages (`memory::page_refs` says why that copy is the bytes a view's blit would read); ++ /// `None` takes the view as ever. Raises as the view's walk would. ++ pub(super) fn staged_page_buffer( ++ &self, ++ hold: Hold<'_>, ++ page: u32, ++ length: u64, ++ ) -> RaiseResult> { ++ if !crate::memory::page_refs::staging() || length == 0 || length & PAGE_MASK != 0 { ++ return Ok(None); ++ } ++ let offset = u64::from(page).wrapping_shl(APV_PAGE_SHIFT); ++ Self::check_task_range(offset, length, self.task_length())?; ++ let (Some(map), Some(device)) = (self.map(), self.device_handle()) else { ++ return Ok(None); ++ }; ++ let Some(refs) = map.page_refs().cloned() else { ++ return Ok(None); ++ }; ++ let started = self.mono(); ++ let runs = self.task_page_runs(hold, offset, length)?; ++ let total = runs ++ .ranges ++ .iter() ++ .try_fold(0_u64, |sum, r| sum.checked_add(r.physical_length)); ++ if total != Some(length) { ++ return Ok(None); ++ } ++ if refs.covered(&runs.ranges) { ++ crate::memory::page_refs::note_covered(); ++ return Ok(None); ++ } ++ let Some(buffer) = device.new_buffer(BufferDesc { ++ length, ++ options: Vec::new(), ++ pinned_gpu_address: None, ++ contents: None, ++ no_copy: false, ++ deallocator: None, ++ })? ++ else { ++ return Ok(None); ++ }; ++ let contents = buffer.contents(); ++ if contents.is_null() { ++ return Ok(None); ++ } ++ #[allow(unsafe_code)] // architecture 17.3: the new buffer's own bytes, written once ++ // SAFETY: `contents` is the CPU address of the `length` bytes of a `Shared` buffer made ++ // just now, which nothing else holds yet, and `buffer` keeps them for this borrow. ++ let bytes = unsafe { std::slice::from_raw_parts_mut(contents.as_ptr::(), length.to_usize()) }; ++ let mut at: usize = 0; ++ for r in &runs.ranges { ++ let end = at.wrapping_add(r.physical_length.to_usize()); ++ let Some(dst) = bytes.get_mut(at..end) else { ++ return Ok(None); ++ }; ++ if !map.read(r.physical_address, dst) { ++ return Ok(None); ++ } ++ at = end; ++ } ++ let ns = u64::try_from(self.mono().saturating_duration_since(started).as_nanos()) ++ .unwrap_or(u64::MAX); ++ crate::memory::page_refs::note_staged(length, ns); ++ Ok(Some(buffer)) ++ } ++ + /// LOCAL (Atlas): the buffer a paged texture pages through: a kept one over exactly the pages + /// the task maps at `page` now, else a new one, kept (`paged_buffers`). Every call walks the + /// page table, so a stale entry is never used. +diff --git a/crates/reims-vgpu-core/src/gpu_task/service.rs b/crates/reims-vgpu-core/src/gpu_task/service.rs +index afdcf70d..dee53297 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/service.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/service.rs +@@ -716,6 +716,15 @@ impl ResourceHost

for GpuTask

{ + GpuTask::paged_texture_buffer(self, hold, page, length) + } + ++ fn staged_page_buffer( ++ &self, ++ hold: Hold<'_>, ++ page: u32, ++ length: u64, ++ ) -> RaiseResult> { ++ GpuTask::staged_page_buffer(self, hold, page, length) ++ } ++ + fn fresh_page_up_mode(&self, page: u32) -> u8 { + self.fresh_page_ups + .mode_at(page, super::paged_buffers::resumed()) +diff --git a/crates/reims-vgpu-core/src/memory.rs b/crates/reims-vgpu-core/src/memory.rs +index 2ee0182c..3ec4e69b 100644 +--- a/crates/reims-vgpu-core/src/memory.rs ++++ b/crates/reims-vgpu-core/src/memory.rs +@@ -50,6 +50,7 @@ pub mod cursor; + pub mod descriptor; + pub mod map; + pub mod mapping; ++pub mod page_refs; + pub mod page_table; + pub mod tracker; + +diff --git a/crates/reims-vgpu-core/src/memory/map.rs b/crates/reims-vgpu-core/src/memory/map.rs +index b9221d50..14ef2fdd 100644 +--- a/crates/reims-vgpu-core/src/memory/map.rs ++++ b/crates/reims-vgpu-core/src/memory/map.rs +@@ -9,6 +9,7 @@ use crate::host::{HostPtr, PhysRange}; + use crate::memory::DeathKind; + use crate::memory::descriptor::MemoryMapDescriptor; + use crate::memory::mapping::{Mapping, Teardown}; ++use crate::memory::page_refs::PageRefs; + use crate::num::{ToU64, ToUsize}; + use crate::platform::Platform; + use crate::platform::os::vm::VmProt; +@@ -33,6 +34,9 @@ pub struct MemoryMapRange { + pub struct MemoryMap { + platform: Arc

, + ranges: Vec, ++ /// LOCAL (Atlas): the live views over each page, shared by copies; `None` unless counting ++ /// (`page_refs`). ++ refs: Option>, + } + + impl std::fmt::Debug for MemoryMap

{ +@@ -130,7 +134,23 @@ impl MemoryMap

{ + return None; + } + +- Some(Arc::new(MemoryMap { platform, ranges })) ++ let refs = crate::memory::page_refs::counting() ++ .then(|| { ++ let start = ranges.iter().map(|r| r.physical_address).min()?; ++ let end = ranges.iter().map(|r| r.physical_end).max()?; ++ PageRefs::new(start, end).map(Arc::new) ++ }) ++ .flatten(); ++ Some(Arc::new(MemoryMap { ++ platform, ++ ranges, ++ refs, ++ })) ++ } ++ ++ /// LOCAL (Atlas): the live views over each page, when counting (`page_refs`). ++ pub(crate) fn page_refs(&self) -> Option<&Arc> { ++ self.refs.as_ref() + } + + /// A plain init (no validation): the same ranges. +@@ -138,6 +158,7 @@ impl MemoryMap

{ + Arc::new(MemoryMap { + platform: self.platform.clone(), + ranges: self.ranges.clone(), ++ refs: self.refs.clone(), + }) + } + +@@ -361,13 +382,25 @@ impl MemoryMap

{ + free_all(); + } + // Writable unless read-only; making the mapping never fails. ++ let teardown = Teardown::Vm { ++ platform: self.platform.clone(), ++ }; ++ // LOCAL (Atlas): counted while it lives (`page_refs`). ++ if let Some(refs) = &self.refs { ++ refs.add(ranges); ++ return Some(Mapping::new_counted( ++ address, ++ total_length, ++ read_only == 0, ++ teardown, ++ (Arc::clone(refs), ranges.into()), ++ )); ++ } + Some(Mapping::new( + address, + total_length, + read_only == 0, +- Teardown::Vm { +- platform: self.platform.clone(), +- }, ++ teardown, + )) + } + +diff --git a/crates/reims-vgpu-core/src/memory/mapping.rs b/crates/reims-vgpu-core/src/memory/mapping.rs +index d7cff8c6..b25503df 100644 +--- a/crates/reims-vgpu-core/src/memory/mapping.rs ++++ b/crates/reims-vgpu-core/src/memory/mapping.rs +@@ -3,7 +3,8 @@ + use std::sync::Arc; + use std::sync::atomic::AtomicU64; + +-use crate::host::{HostPtr, HostTask}; ++use crate::host::{HostPtr, HostTask, PhysRange}; ++use crate::memory::page_refs::PageRefs; + use crate::memory::DeathKind; + use crate::platform::Platform; + use crate::seams::links::MappingDeviceLink; +@@ -48,6 +49,8 @@ pub struct Mapping { + length: u64, + writable: bool, + teardown: Teardown

, ++ /// LOCAL (Atlas): the page counts this view is in, and its guest ranges (`page_refs`). ++ counted: Option<(Arc, Box<[PhysRange]>)>, + } + + impl std::fmt::Debug for Mapping

{ +@@ -89,9 +92,28 @@ impl Mapping

{ + length, + writable, + teardown, ++ counted: None, + }) + } + ++ /// LOCAL (Atlas): a map's view, as [`Mapping::new`], counted in `counted`'s pages until its ++ /// last release (the caller added it there). ++ pub(crate) fn new_counted( ++ virtual_address: HostPtr, ++ length: u64, ++ writable: bool, ++ teardown: Teardown

, ++ counted: (Arc, Box<[PhysRange]>), ++ ) -> Arc { ++ let mut mapping = Mapping::new(virtual_address, length, writable, teardown); ++ match Arc::get_mut(&mut mapping) { ++ Some(m) => m.counted = Some(counted), ++ // Unreachable: the mapping was just made. Uncounted, it would stay counted for good. ++ None => std::mem::forget(counted), ++ } ++ mapping ++ } ++ + /// Where the range is mapped. + pub fn virtual_address(&self) -> HostPtr { + self.virtual_address +@@ -110,6 +132,18 @@ impl Mapping

{ + + impl Drop for Mapping

{ + fn drop(&mut self) { ++ // LOCAL (Atlas): uncounted after the teardown below. ++ let counted = self.counted.take(); ++ self.tear_down(); ++ if let Some((refs, ranges)) = counted { ++ refs.remove(&ranges); ++ } ++ } ++} ++ ++impl Mapping

{ ++ /// What the last release does. ++ fn tear_down(&mut self) { + match &self.teardown { + Teardown::None => {} + Teardown::Vm { platform } => { +diff --git a/crates/reims-vgpu-core/src/memory/page_refs.rs b/crates/reims-vgpu-core/src/memory/page_refs.rs +new file mode 100644 +index 00000000..5ebfa14e +--- /dev/null ++++ b/crates/reims-vgpu-core/src/memory/page_refs.rs +@@ -0,0 +1,262 @@ ++//! LOCAL (Atlas): the live views over each guest page, and the staged page-up they allow. ++//! ++//! **What it is for.** A new paged texture's first page-up copies its guest pages into the host ++//! texture through a view: one `mmap` per run of guest pages (a 2.4 MB texture of the Sim's ++//! compositor is about 600 of them), a Vulkan import, and at the view's release a `munmap` whose ++//! dirty shared PTEs flush the TLB once per run on every CPU the process ran on, the vCPUs among ++//! them. Measured 2026-11-09 with one Sim: ~520 such views a second, all first page-ups of new ++//! `Shared` render targets, 15% of the Sim task thread, and ~57k TLB shootdowns a second. A staged ++//! page-up instead copies the pages on the CPU into a new buffer of the runtime's own and blits ++//! from it: no view, no import, no `munmap`. ++//! ++//! **Why the copy is the same bytes.** The view's blit reads the pages on the GPU, after every ++//! earlier exec's GPU work (the paging command buffer waits on the FIFO's chain). A copy at the ++//! exec reads them earlier. The two can differ only by a GPU write still pending at the exec, and ++//! the GPU reaches guest RAM only through views a [`MemoryMap`](super::map::MemoryMap) makes ++//! (`new_range_mapping`), each alive until the runtime frees the last buffer over it, after the ++//! GPU's last use. So a page no live view covers has nothing pending, and the copy is taken only ++//! when no page of the texture is covered; else the view is made as ever. The guest's CPU writing ++//! the pages of a committed command buffer's resource is outside Metal's rules either way. ++//! ++//! **The switch.** `REIMS_VGPU_STAGED_PAGE_UP=`: views are counted from the start, and ++//! page-ups are staged while that file exists (looked at once a second). Unset: nothing is ++//! counted and nothing changes. Counts stay at `u16::MAX` once they reach it (the page then ++//! always takes the view). ++ ++use std::sync::OnceLock; ++use std::sync::atomic::{AtomicU16, AtomicU64, Ordering}; ++use std::time::{Duration, Instant}; ++ ++use crate::guest::records::{APV_PAGE_MASK, APV_PAGE_SHIFT}; ++use crate::host::PhysRange; ++use crate::num::ToUsize; ++ ++/// The variable naming the switch file. ++pub const ENV_VAR: &str = "REIMS_VGPU_STAGED_PAGE_UP"; ++ ++/// The switch file and its last look. ++struct Switch { ++ path: Option, ++ look: crate::sync::Mutex<(Option, bool)>, ++} ++ ++fn switch() -> &'static Switch { ++ static S: OnceLock = OnceLock::new(); ++ S.get_or_init(|| Switch { ++ path: std::env::var_os(ENV_VAR) ++ .filter(|p| !p.is_empty()) ++ .map(std::path::PathBuf::from), ++ look: crate::sync::Mutex::new((None, false)), ++ }) ++} ++ ++/// Whether views are counted: the variable is set (read once). ++pub(crate) fn counting() -> bool { ++ !cfg!(test) && switch().path.is_some() ++} ++ ++/// Whether page-ups are staged now: counting, and the file exists at the last look. ++pub(crate) fn staging() -> bool { ++ let s = switch(); ++ let Some(path) = &s.path else { ++ return false; ++ }; ++ if cfg!(test) { ++ return false; ++ } ++ let mut look = s.look.lock(); ++ let now = Instant::now(); ++ if look.0.is_none_or(|t| now.duration_since(t) >= Duration::from_secs(1)) { ++ *look = (Some(now), path.exists()); ++ } ++ look.1 ++} ++ ++/// The staged page-ups' counters, all tasks together. ++struct Stats { ++ staged: AtomicU64, ++ staged_bytes: AtomicU64, ++ covered: AtomicU64, ++ copy_ns: AtomicU64, ++ last: crate::sync::Mutex>, ++} ++ ++static STATS: Stats = Stats { ++ staged: AtomicU64::new(0), ++ staged_bytes: AtomicU64::new(0), ++ covered: AtomicU64::new(0), ++ copy_ns: AtomicU64::new(0), ++ last: crate::sync::Mutex::new(None), ++}; ++ ++/// Counts a staged page-up of `bytes`, whose buffer and copy took `ns`. ++pub(crate) fn note_staged(bytes: u64, ns: u64) { ++ STATS.staged.fetch_add(1, Ordering::Relaxed); ++ STATS.staged_bytes.fetch_add(bytes, Ordering::Relaxed); ++ STATS.copy_ns.fetch_add(ns, Ordering::Relaxed); ++ log_stats(); ++} ++ ++/// Counts a page-up that took the view because a live view covered one of its pages. ++pub(crate) fn note_covered() { ++ STATS.covered.fetch_add(1, Ordering::Relaxed); ++ log_stats(); ++} ++ ++/// Logs the counts (and resets them) at most every 10 s. ++fn log_stats() { ++ let now = Instant::now(); ++ { ++ let mut last = STATS.last.lock(); ++ match *last { ++ None => { ++ *last = Some(now); ++ return; ++ } ++ Some(t) if now.duration_since(t) < Duration::from_secs(10) => return, ++ Some(_) => *last = Some(now), ++ } ++ } ++ let staged = STATS.staged.swap(0, Ordering::Relaxed); ++ let bytes = STATS.staged_bytes.swap(0, Ordering::Relaxed); ++ let covered = STATS.covered.swap(0, Ordering::Relaxed); ++ let ns = STATS.copy_ns.swap(0, Ordering::Relaxed); ++ log::info!( ++ "staged page-ups: {staged} staged ({} KiB, {} us making and copying), {covered} took \ ++ the view (a live view covered a page)", ++ bytes >> 10, ++ ns.wrapping_div(1000) ++ ); ++} ++ ++/// The live views over each guest page of `first..first + counts.len()` pages. ++pub struct PageRefs { ++ first: u64, ++ counts: Box<[AtomicU16]>, ++} ++ ++impl std::fmt::Debug for PageRefs { ++ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { ++ f.debug_struct("PageRefs") ++ .field("first", &self.first) ++ .field("pages", &self.counts.len()) ++ .finish() ++ } ++} ++ ++impl PageRefs { ++ /// Counts for the guest physical range `start..end`, all zero; `None` for an empty or ++ /// overflowing range. ++ pub(crate) fn new(start: u64, end: u64) -> Option { ++ let first = start >> APV_PAGE_SHIFT; ++ let last = end.checked_add(APV_PAGE_MASK)? >> APV_PAGE_SHIFT; ++ let pages = last.checked_sub(first).filter(|&n| n != 0)?.to_usize(); ++ let counts = std::iter::repeat_with(|| AtomicU16::new(0)) ++ .take(pages) ++ .collect(); ++ Some(PageRefs { first, counts }) ++ } ++ ++ /// The pages of `range`, as indices: `None` when one lies outside. ++ fn indices(&self, range: &PhysRange) -> Option> { ++ let start = range.physical_address >> APV_PAGE_SHIFT; ++ let end = range ++ .physical_address ++ .checked_add(range.physical_length)? ++ .checked_add(APV_PAGE_MASK)? ++ >> APV_PAGE_SHIFT; ++ let from = start.checked_sub(self.first)?.to_usize(); ++ let to = end.checked_sub(self.first)?.to_usize(); ++ (to <= self.counts.len()).then_some(from..to) ++ } ++ ++ /// A view over `ranges` was made. ++ pub(crate) fn add(&self, ranges: &[PhysRange]) { ++ for r in ranges { ++ if let Some(pages) = self.indices(r) { ++ for c in &self.counts[pages] { ++ // Saturates, and then stays. ++ let _ = c.fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { ++ n.checked_add(1) ++ }); ++ } ++ } ++ } ++ } ++ ++ /// A view over `ranges` is gone. ++ pub(crate) fn remove(&self, ranges: &[PhysRange]) { ++ for r in ranges { ++ if let Some(pages) = self.indices(r) { ++ for c in &self.counts[pages] { ++ // A saturated count stays; 0 stays (never reached by a counted view). ++ let _ = c.fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { ++ (n != 0 && n != u16::MAX).then(|| n.wrapping_sub(1)) ++ }); ++ } ++ } ++ } ++ } ++ ++ /// Whether a live view covers a page of `ranges`, or a page lies outside the counts. ++ pub(crate) fn covered(&self, ranges: &[PhysRange]) -> bool { ++ ranges.iter().any(|r| match self.indices(r) { ++ Some(pages) => self.counts[pages] ++ .iter() ++ .any(|c| c.load(Ordering::Acquire) != 0), ++ None => true, ++ }) ++ } ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ ++ const PAGE: u64 = 1 << APV_PAGE_SHIFT; ++ ++ fn pr(at: u64, len: u64) -> PhysRange { ++ PhysRange { ++ physical_address: at, ++ physical_length: len, ++ } ++ } ++ ++ #[test] ++ fn a_page_is_covered_while_a_view_over_it_lives() { ++ let refs = PageRefs::new(PAGE, 9 * PAGE).unwrap(); ++ let view = [pr(2 * PAGE, PAGE), pr(5 * PAGE, 2 * PAGE)]; ++ assert!(!refs.covered(&[pr(PAGE, 8 * PAGE)])); ++ refs.add(&view); ++ refs.add(&view[..1]); ++ assert!(refs.covered(&[pr(6 * PAGE, PAGE)])); ++ assert!(!refs.covered(&[pr(3 * PAGE, 2 * PAGE), pr(7 * PAGE, PAGE)])); ++ refs.remove(&view); ++ assert!(!refs.covered(&[pr(5 * PAGE, 2 * PAGE)])); ++ assert!(refs.covered(&[pr(2 * PAGE, PAGE)])); ++ refs.remove(&view[..1]); ++ assert!(!refs.covered(&[pr(PAGE, 8 * PAGE)])); ++ } ++ ++ #[test] ++ fn pages_outside_the_counts_count_as_covered() { ++ let refs = PageRefs::new(PAGE, 3 * PAGE).unwrap(); ++ assert!(refs.covered(&[pr(0, PAGE)])); ++ assert!(refs.covered(&[pr(2 * PAGE, 2 * PAGE)])); ++ assert!(refs.covered(&[pr(u64::MAX - 1, 2)])); ++ refs.add(&[pr(3 * PAGE, PAGE)]); ++ assert!(!refs.covered(&[pr(PAGE, 2 * PAGE)])); ++ } ++ ++ #[test] ++ fn a_saturated_count_stays() { ++ let refs = PageRefs::new(PAGE, 2 * PAGE).unwrap(); ++ for _ in 0..u32::from(u16::MAX) + 2 { ++ refs.add(&[pr(PAGE, PAGE)]); ++ } ++ for _ in 0..u32::from(u16::MAX) + 2 { ++ refs.remove(&[pr(PAGE, PAGE)]); ++ } ++ assert!(refs.covered(&[pr(PAGE, PAGE)])); ++ } ++} +diff --git a/crates/reims-vgpu-core/src/resources/host.rs b/crates/reims-vgpu-core/src/resources/host.rs +index d65a4c52..c440733f 100644 +--- a/crates/reims-vgpu-core/src/resources/host.rs ++++ b/crates/reims-vgpu-core/src/resources/host.rs +@@ -84,6 +84,19 @@ pub trait ResourceHost: Send + Sync { + self.new_buffer_for_virtual_page(hold, page, length) + } + ++ /// LOCAL (Atlas): a new buffer holding a copy of the guest's pages, for a texture's first ++ /// page-up, when no GPU access to them can be pending (`memory::page_refs`); `None` pages ++ /// through the transfer buffer as ever. ++ fn staged_page_buffer( ++ &self, ++ hold: Hold<'_>, ++ page: u32, ++ length: u64, ++ ) -> RaiseResult> { ++ let _ = (hold, page, length); ++ Ok(None) ++ } ++ + /// LOCAL (Atlas): whether a Private paged texture at task page `page` may skip its first + /// page-up (`PagedTexture::skip_first_page_up`): 0 no, else the mode. Only a task of a guest + /// process says yes, never the kernel's. +diff --git a/crates/reims-vgpu-core/src/resources/texture.rs b/crates/reims-vgpu-core/src/resources/texture.rs +index b5b88bb0..878bc401 100644 +--- a/crates/reims-vgpu-core/src/resources/texture.rs ++++ b/crates/reims-vgpu-core/src/resources/texture.rs +@@ -420,6 +420,19 @@ impl PagedTexture

{ + Ok(()) + } + ++ /// LOCAL (Atlas): for a page-up of a texture with no transfer buffer, a staged copy of its ++ /// pages to blit from instead (`memory::page_refs`), or `None`. The texture keeps no buffer: ++ /// a later page-down makes its view then. ++ pub(crate) fn staged(&self, hold: Hold<'_>) -> RaiseResult> { ++ if self.transfer_buffer.lock().is_some() { ++ return Ok(None); ++ } ++ let Some(task) = self.task.upgrade() else { ++ return Ok(None); ++ }; ++ task.staged_page_buffer(hold, self.page, self.length) ++ } ++ + /// Drops the guest pages and makes the texture volatile; host-valid is kept. + pub(crate) fn discard(&self, state: &ResourceState) -> RaiseResult<()> { + drop(self.transfer_buffer.lock().take()); +diff --git a/crates/reims-vgpu-core/src/resources/texture/blit.rs b/crates/reims-vgpu-core/src/resources/texture/blit.rs +index e4f5b2d8..55e4875c 100644 +--- a/crates/reims-vgpu-core/src/resources/texture/blit.rs ++++ b/crates/reims-vgpu-core/src/resources/texture/blit.rs +@@ -57,7 +57,20 @@ pub(crate) fn page( + is_download: bool, + planes: &[(&TextureDimensionRecord, u64)], + ) -> RaiseResult<()> { +- let transfer = paged.transfer_buffer.lock(); ++ page_through(paged, encoder, is_download, planes, None) ++} ++ ++/// [`page`] through `staged` instead of the transfer buffer when given (LOCAL (Atlas): a ++/// staged page-up, `PagedTexture::staged`). ++pub(crate) fn page_through( ++ paged: &PagedTexture

, ++ encoder: Option<&P::BlitEncoder>, ++ is_download: bool, ++ planes: &[(&TextureDimensionRecord, u64)], ++ staged: Option<&P::Buffer>, ++) -> RaiseResult<()> { ++ let held = paged.transfer_buffer.lock(); ++ let transfer = staged.or(held.as_ref()); + let Some(&(first, _)) = planes.first() else { + return Ok(()); + }; +@@ -106,7 +119,7 @@ pub(crate) fn page( + source_level: level, + source_origin: MtlOrigin::new(0, 0, 0), + source_size: size, +- destination_buffer: transfer.as_ref(), ++ destination_buffer: transfer, + destination_offset: offset, + destination_bytes_per_row: bytes_per_row, + destination_bytes_per_image: bytes_per_image, +@@ -125,7 +138,7 @@ pub(crate) fn page( + encode( + OnFault::Return, + BlitCmd::CopyBufferToTextureOptions { +- source_buffer: transfer.as_ref(), ++ source_buffer: transfer, + source_offset: offset, + source_bytes_per_row: bytes_per_row, + source_bytes_per_image: bytes_per_image, +@@ -242,11 +255,25 @@ impl SinglePlane { + state: &ResourceState, + encoder: Option<&P::BlitEncoder>, + ) -> RaiseResult<()> { ++ // LOCAL (Atlas): a first page-up may blit from a staged copy (`PagedTexture::staged`). ++ let staged = std::cell::OnceCell::new(); + paged.prepare( + state, +- || paged.ensure_physical(hold), ++ || match paged.staged(hold)? { ++ Some(buffer) => { ++ let _ = staged.set(buffer); ++ Ok(()) ++ } ++ None => paged.ensure_physical(hold), ++ }, + || { +- self.page(paged, encoder, false)?; ++ page_through( ++ paged, ++ encoder, ++ false, ++ &[(&self.dimension, self.blit_option)], ++ staged.get(), ++ )?; + if self.generate_mipmaps { + nil_send!( + encoder, diff --git a/patches/reims-vgpu/0032-staged-zero-page-ups-a-texture-s-first-prepare-over-.patch b/patches/reims-vgpu/0032-staged-zero-page-ups-a-texture-s-first-prepare-over-.patch new file mode 100644 index 0000000..6e226cf --- /dev/null +++ b/patches/reims-vgpu/0032-staged-zero-page-ups-a-texture-s-first-prepare-over-.patch @@ -0,0 +1,508 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 05:57:17 +0000 +Subject: [PATCH 32/33] staged zero page-ups: a texture's first prepare over + guest pages of zeros writes nothing + +A new texture's host texture reads zero before any command (the runtime's FirstMove), and +before its first prepare no GPU work reached it. So when that first prepare's guest pages all +read zero, the page-up (buffer, copy, blit) changes nothing and is skipped. The pages are +scanned a page at a time and the scan stops at the first non-zero page; covered pages take +the view as before. PagedTexture::prepared_once is set by every prepare, synchronize and +skipped Private page-up. + +Switch: REIMS_VGPU_STAGED_ZERO=, live while the file exists and page-ups are staged. +The staged log line now splits walk, scan, buffer and copy time and counts all-zero copies. +--- + crates/reims-vgpu-core/src/gpu_task/memory.rs | 66 +++++++-- + .../reims-vgpu-core/src/gpu_task/service.rs | 5 +- + .../reims-vgpu-core/src/memory/page_refs.rs | 131 ++++++++++++++---- + crates/reims-vgpu-core/src/resources/host.rs | 6 +- + .../reims-vgpu-core/src/resources/texture.rs | 27 +++- + .../src/resources/texture/blit.rs | 21 ++- + 6 files changed, 205 insertions(+), 51 deletions(-) + +diff --git a/crates/reims-vgpu-core/src/gpu_task/memory.rs b/crates/reims-vgpu-core/src/gpu_task/memory.rs +index 936ccb15..ccefff6a 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/memory.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/memory.rs +@@ -31,6 +31,8 @@ use crate::platform::Platform; + use crate::platform::device::GpuDevice; + use crate::platform::resource::Buffer as _; + use crate::platform::values::BufferDesc; ++use crate::memory::map::MemoryMap; ++use crate::resources::texture::Staged; + use crate::platform::os::clock::{Clock, MonoTime}; + use crate::platform::types::ResourceAddressRange; + use crate::raise::{Kept, keep_forever}; +@@ -570,14 +572,18 @@ impl GpuTask

{ + /// LOCAL (Atlas): a new buffer holding a copy of the task's pages `page..page + length`, for + /// a texture's first page-up, when page-ups are staged and no live view covers any of the + /// pages (`memory::page_refs` says why that copy is the bytes a view's blit would read); +- /// `None` takes the view as ever. Raises as the view's walk would. ++ /// `None` takes the view as ever. On the texture's first prepare (`first`), pages of zeros ++ /// answer `Staged::Zero` while the zero switch is on: its host texture reads zero already. ++ /// Raises as the view's walk would. + pub(super) fn staged_page_buffer( + &self, + hold: Hold<'_>, + page: u32, + length: u64, +- ) -> RaiseResult> { +- if !crate::memory::page_refs::staging() || length == 0 || length & PAGE_MASK != 0 { ++ first: bool, ++ ) -> RaiseResult>> { ++ use crate::memory::page_refs::{self, StagedTimes}; ++ if !page_refs::staging() || length == 0 || length & PAGE_MASK != 0 { + return Ok(None); + } + let offset = u64::from(page).wrapping_shl(APV_PAGE_SHIFT); +@@ -588,6 +594,10 @@ impl GpuTask

{ + let Some(refs) = map.page_refs().cloned() else { + return Ok(None); + }; ++ let ns = |from: MonoTime, to: MonoTime| { ++ u64::try_from(to.saturating_duration_since(from).as_nanos()).unwrap_or(u64::MAX) ++ }; ++ let mut times = StagedTimes::default(); + let started = self.mono(); + let runs = self.task_page_runs(hold, offset, length)?; + let total = runs +@@ -598,9 +608,21 @@ impl GpuTask

{ + return Ok(None); + } + if refs.covered(&runs.ranges) { +- crate::memory::page_refs::note_covered(); ++ page_refs::note_covered(); + return Ok(None); + } ++ let walked = self.mono(); ++ times.walk = ns(started, walked); ++ if first && page_refs::zero_skipping() { ++ let zeros = Self::runs_read_zero(&map, &runs.ranges); ++ let scanned = self.mono(); ++ times.scan = ns(walked, scanned); ++ if zeros == Some(true) { ++ page_refs::note_zero_skipped(length, times); ++ return Ok(Some(Staged::Zero)); ++ } ++ } ++ let making = self.mono(); + let Some(buffer) = device.new_buffer(BufferDesc { + length, + options: Vec::new(), +@@ -616,10 +638,13 @@ impl GpuTask

{ + if contents.is_null() { + return Ok(None); + } ++ let copying = self.mono(); ++ times.make = ns(making, copying); + #[allow(unsafe_code)] // architecture 17.3: the new buffer's own bytes, written once + // SAFETY: `contents` is the CPU address of the `length` bytes of a `Shared` buffer made + // just now, which nothing else holds yet, and `buffer` keeps them for this borrow. +- let bytes = unsafe { std::slice::from_raw_parts_mut(contents.as_ptr::(), length.to_usize()) }; ++ let bytes = ++ unsafe { std::slice::from_raw_parts_mut(contents.as_ptr::(), length.to_usize()) }; + let mut at: usize = 0; + for r in &runs.ranges { + let end = at.wrapping_add(r.physical_length.to_usize()); +@@ -631,10 +656,33 @@ impl GpuTask

{ + } + at = end; + } +- let ns = u64::try_from(self.mono().saturating_duration_since(started).as_nanos()) +- .unwrap_or(u64::MAX); +- crate::memory::page_refs::note_staged(length, ns); +- Ok(Some(buffer)) ++ // Counted on first prepares only: what the zero switch would have skipped. ++ let zeros = first && bytes.iter().fold(0_u8, |any, &b| any | b) == 0; ++ times.copy = ns(copying, self.mono()); ++ page_refs::note_staged(length, zeros, times); ++ Ok(Some(Staged::Buffer(buffer))) ++ } ++ ++ /// Whether every byte of the guest's `ranges` reads zero, read a page at a time and stopping ++ /// at the first page that does not; `None` when a range is outside the map. ++ fn runs_read_zero(map: &MemoryMap

, ranges: &[PhysRange]) -> Option { ++ let mut chunk = [0_u8; PAGE as usize]; ++ for r in ranges { ++ let mut at = r.physical_address; ++ let end = r.physical_address.checked_add(r.physical_length)?; ++ while at < end { ++ let n = end.wrapping_sub(at).min(PAGE).to_usize(); ++ let dst = chunk.get_mut(..n)?; ++ if !map.read(at, dst) { ++ return None; ++ } ++ if dst.iter().fold(0_u8, |any, &b| any | b) != 0 { ++ return Some(false); ++ } ++ at = at.wrapping_add(PAGE); ++ } ++ } ++ Some(true) + } + + /// LOCAL (Atlas): the buffer a paged texture pages through: a kept one over exactly the pages +diff --git a/crates/reims-vgpu-core/src/gpu_task/service.rs b/crates/reims-vgpu-core/src/gpu_task/service.rs +index dee53297..8ab1b8a4 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/service.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/service.rs +@@ -721,8 +721,9 @@ impl ResourceHost

for GpuTask

{ + hold: Hold<'_>, + page: u32, + length: u64, +- ) -> RaiseResult> { +- GpuTask::staged_page_buffer(self, hold, page, length) ++ first: bool, ++ ) -> RaiseResult>> { ++ GpuTask::staged_page_buffer(self, hold, page, length, first) + } + + fn fresh_page_up_mode(&self, page: u32) -> u8 { +diff --git a/crates/reims-vgpu-core/src/memory/page_refs.rs b/crates/reims-vgpu-core/src/memory/page_refs.rs +index 5ebfa14e..f12cfe05 100644 +--- a/crates/reims-vgpu-core/src/memory/page_refs.rs ++++ b/crates/reims-vgpu-core/src/memory/page_refs.rs +@@ -34,20 +34,52 @@ use crate::num::ToUsize; + /// The variable naming the switch file. + pub const ENV_VAR: &str = "REIMS_VGPU_STAGED_PAGE_UP"; + +-/// The switch file and its last look. ++/// The variable naming the zero switch's file: while it exists (and page-ups are staged), a ++/// texture's first prepare over pages of zeros writes nothing (`Staged::Zero`). ++pub const ZERO_ENV_VAR: &str = "REIMS_VGPU_STAGED_ZERO"; ++ ++/// A switch file and its last look. + struct Switch { + path: Option, + look: crate::sync::Mutex<(Option, bool)>, + } + ++impl Switch { ++ fn of(var: &str) -> Switch { ++ Switch { ++ path: std::env::var_os(var) ++ .filter(|p| !p.is_empty()) ++ .map(std::path::PathBuf::from), ++ look: crate::sync::Mutex::new((None, false)), ++ } ++ } ++ ++ /// Whether the file existed at the last look (once a second); false unset. ++ fn on(&self) -> bool { ++ let Some(path) = &self.path else { ++ return false; ++ }; ++ if cfg!(test) { ++ return false; ++ } ++ let mut look = self.look.lock(); ++ let now = Instant::now(); ++ if look.0.is_none_or(|t| now.duration_since(t) >= Duration::from_secs(1)) { ++ *look = (Some(now), path.exists()); ++ } ++ look.1 ++ } ++} ++ + fn switch() -> &'static Switch { + static S: OnceLock = OnceLock::new(); +- S.get_or_init(|| Switch { +- path: std::env::var_os(ENV_VAR) +- .filter(|p| !p.is_empty()) +- .map(std::path::PathBuf::from), +- look: crate::sync::Mutex::new((None, false)), +- }) ++ S.get_or_init(|| Switch::of(ENV_VAR)) ++} ++ ++/// Whether a first prepare over pages of zeros writes nothing now. ++pub(crate) fn zero_skipping() -> bool { ++ static S: OnceLock = OnceLock::new(); ++ S.get_or_init(|| Switch::of(ZERO_ENV_VAR)).on() + } + + /// Whether views are counted: the variable is set (read once). +@@ -57,19 +89,7 @@ pub(crate) fn counting() -> bool { + + /// Whether page-ups are staged now: counting, and the file exists at the last look. + pub(crate) fn staging() -> bool { +- let s = switch(); +- let Some(path) = &s.path else { +- return false; +- }; +- if cfg!(test) { +- return false; +- } +- let mut look = s.look.lock(); +- let now = Instant::now(); +- if look.0.is_none_or(|t| now.duration_since(t) >= Duration::from_secs(1)) { +- *look = (Some(now), path.exists()); +- } +- look.1 ++ switch().on() + } + + /// The staged page-ups' counters, all tasks together. +@@ -77,7 +97,13 @@ struct Stats { + staged: AtomicU64, + staged_bytes: AtomicU64, + covered: AtomicU64, ++ walk_ns: AtomicU64, ++ make_ns: AtomicU64, + copy_ns: AtomicU64, ++ zeros_copied: AtomicU64, ++ zeros_skipped: AtomicU64, ++ zeros_skipped_bytes: AtomicU64, ++ scan_ns: AtomicU64, + last: crate::sync::Mutex>, + } + +@@ -85,15 +111,51 @@ static STATS: Stats = Stats { + staged: AtomicU64::new(0), + staged_bytes: AtomicU64::new(0), + covered: AtomicU64::new(0), ++ walk_ns: AtomicU64::new(0), ++ make_ns: AtomicU64::new(0), + copy_ns: AtomicU64::new(0), ++ zeros_copied: AtomicU64::new(0), ++ zeros_skipped: AtomicU64::new(0), ++ zeros_skipped_bytes: AtomicU64::new(0), ++ scan_ns: AtomicU64::new(0), + last: crate::sync::Mutex::new(None), + }; + +-/// Counts a staged page-up of `bytes`, whose buffer and copy took `ns`. +-pub(crate) fn note_staged(bytes: u64, ns: u64) { ++/// The times of one staged page-up, in nanoseconds: the walk, the zero scan, the buffer's ++/// making and the copy. ++#[derive(Debug, Clone, Copy, Default)] ++pub(crate) struct StagedTimes { ++ pub(crate) walk: u64, ++ pub(crate) scan: u64, ++ pub(crate) make: u64, ++ pub(crate) copy: u64, ++} ++ ++impl StagedTimes { ++ fn add(self) { ++ STATS.walk_ns.fetch_add(self.walk, Ordering::Relaxed); ++ STATS.scan_ns.fetch_add(self.scan, Ordering::Relaxed); ++ STATS.make_ns.fetch_add(self.make, Ordering::Relaxed); ++ STATS.copy_ns.fetch_add(self.copy, Ordering::Relaxed); ++ } ++} ++ ++/// Counts a staged page-up of `bytes` copied into a buffer; `zeros` when every byte was zero. ++pub(crate) fn note_staged(bytes: u64, zeros: bool, times: StagedTimes) { + STATS.staged.fetch_add(1, Ordering::Relaxed); + STATS.staged_bytes.fetch_add(bytes, Ordering::Relaxed); +- STATS.copy_ns.fetch_add(ns, Ordering::Relaxed); ++ if zeros { ++ STATS.zeros_copied.fetch_add(1, Ordering::Relaxed); ++ } ++ times.add(); ++ log_stats(); ++} ++ ++/// Counts a first page-up of `bytes` of zeros that wrote nothing. ++pub(crate) fn note_zero_skipped(bytes: u64, times: StagedTimes) { ++ STATS.zeros_skipped.fetch_add(1, Ordering::Relaxed); ++ STATS.zeros_skipped_bytes.fetch_add(bytes, Ordering::Relaxed); ++ times.add(); + log_stats(); + } + +@@ -117,15 +179,24 @@ fn log_stats() { + Some(_) => *last = Some(now), + } + } +- let staged = STATS.staged.swap(0, Ordering::Relaxed); +- let bytes = STATS.staged_bytes.swap(0, Ordering::Relaxed); +- let covered = STATS.covered.swap(0, Ordering::Relaxed); +- let ns = STATS.copy_ns.swap(0, Ordering::Relaxed); ++ let take = |c: &AtomicU64| c.swap(0, Ordering::Relaxed); ++ let us = |c: &AtomicU64| take(c).wrapping_div(1000); ++ let staged = take(&STATS.staged); ++ let bytes = take(&STATS.staged_bytes); ++ let covered = take(&STATS.covered); ++ let zeros_copied = take(&STATS.zeros_copied); ++ let skipped = take(&STATS.zeros_skipped); ++ let skipped_bytes = take(&STATS.zeros_skipped_bytes); + log::info!( +- "staged page-ups: {staged} staged ({} KiB, {} us making and copying), {covered} took \ +- the view (a live view covered a page)", ++ "staged page-ups: {staged} staged ({} KiB, {zeros_copied} all zero), {skipped} zero \ ++ skipped ({} KiB), {covered} took the view (a live view covered a page); us: walk {}, \ ++ scan {}, make {}, copy {}", + bytes >> 10, +- ns.wrapping_div(1000) ++ skipped_bytes >> 10, ++ us(&STATS.walk_ns), ++ us(&STATS.scan_ns), ++ us(&STATS.make_ns), ++ us(&STATS.copy_ns), + ); + } + +diff --git a/crates/reims-vgpu-core/src/resources/host.rs b/crates/reims-vgpu-core/src/resources/host.rs +index c440733f..19391023 100644 +--- a/crates/reims-vgpu-core/src/resources/host.rs ++++ b/crates/reims-vgpu-core/src/resources/host.rs +@@ -87,13 +87,15 @@ pub trait ResourceHost: Send + Sync { + /// LOCAL (Atlas): a new buffer holding a copy of the guest's pages, for a texture's first + /// page-up, when no GPU access to them can be pending (`memory::page_refs`); `None` pages + /// through the transfer buffer as ever. ++ /// `first`: the texture's first prepare, which may answer `Staged::Zero` for pages of zeros. + fn staged_page_buffer( + &self, + hold: Hold<'_>, + page: u32, + length: u64, +- ) -> RaiseResult> { +- let _ = (hold, page, length); ++ first: bool, ++ ) -> RaiseResult>> { ++ let _ = (hold, page, length, first); + Ok(None) + } + +diff --git a/crates/reims-vgpu-core/src/resources/texture.rs b/crates/reims-vgpu-core/src/resources/texture.rs +index 878bc401..7577d432 100644 +--- a/crates/reims-vgpu-core/src/resources/texture.rs ++++ b/crates/reims-vgpu-core/src/resources/texture.rs +@@ -222,6 +222,17 @@ pub struct PagedTexture { + pub(crate) texture: P::Texture, + /// The guest pages, bound at the first paging of a life and dropped by a discard. + pub(crate) transfer_buffer: Mutex>, ++ /// LOCAL (Atlas): set at the first prepare, synchronize or skipped page-up: before it, no ++ /// GPU work reached the host texture, which reads zero (the runtime's `FirstMove`). ++ pub(crate) prepared_once: std::sync::atomic::AtomicBool, ++} ++ ++/// LOCAL (Atlas): what a staged page-up found (`ResourceHost::staged_page_buffer`). ++pub enum Staged { ++ /// A buffer holding a copy of the pages, to blit from. ++ Buffer(B), ++ /// Every byte of the pages is zero: a first page-up has nothing to write. ++ Zero, + } + + /// The five layouts of a paged texture. +@@ -366,6 +377,7 @@ impl PagedTexture

{ + is_pvrtc: false, + texture: texture.clone_ref(), + transfer_buffer: Mutex::new(None), ++ prepared_once: std::sync::atomic::AtomicBool::new(false), + } + } + +@@ -423,14 +435,21 @@ impl PagedTexture

{ + /// LOCAL (Atlas): for a page-up of a texture with no transfer buffer, a staged copy of its + /// pages to blit from instead (`memory::page_refs`), or `None`. The texture keeps no buffer: + /// a later page-down makes its view then. +- pub(crate) fn staged(&self, hold: Hold<'_>) -> RaiseResult> { ++ /// ++ /// `Staged::Zero` only for the texture's first prepare (`first`), whose host texture no GPU ++ /// work reached and so reads zero, as the pages do. ++ pub(crate) fn staged( ++ &self, ++ hold: Hold<'_>, ++ first: bool, ++ ) -> RaiseResult>> { + if self.transfer_buffer.lock().is_some() { + return Ok(None); + } + let Some(task) = self.task.upgrade() else { + return Ok(None); + }; +- task.staged_page_buffer(hold, self.page, self.length) ++ task.staged_page_buffer(hold, self.page, self.length, first) + } + + /// Drops the guest pages and makes the texture volatile; host-valid is kept. +@@ -474,6 +493,7 @@ impl PagedTexture

{ + return false; + } + state.set_host_valid(true); ++ self.prepared_once.store(true, Ordering::Release); + STATS.page_ups_skipped.fetch_add(1, Ordering::Relaxed); + true + } +@@ -488,6 +508,7 @@ impl PagedTexture

{ + ) -> RaiseResult<()> { + self.devolatilize(state)?; + if state.host_valid() { ++ self.prepared_once.store(true, std::sync::atomic::Ordering::Release); + return Ok(()); + } + if !state.guest_valid() { +@@ -498,6 +519,7 @@ impl PagedTexture

{ + page()?; + } + state.set_host_valid(true); ++ self.prepared_once.store(true, std::sync::atomic::Ordering::Release); + Ok(()) + } + +@@ -509,6 +531,7 @@ impl PagedTexture

{ + ensure: impl FnOnce() -> RaiseResult<()>, + page: impl FnOnce() -> RaiseResult<()>, + ) -> RaiseResult<()> { ++ self.prepared_once.store(true, std::sync::atomic::Ordering::Release); + if state.guest_valid() { + return Ok(()); + } +diff --git a/crates/reims-vgpu-core/src/resources/texture/blit.rs b/crates/reims-vgpu-core/src/resources/texture/blit.rs +index 55e4875c..017d4f37 100644 +--- a/crates/reims-vgpu-core/src/resources/texture/blit.rs ++++ b/crates/reims-vgpu-core/src/resources/texture/blit.rs +@@ -17,7 +17,7 @@ use crate::resources::texture::dimension::{ + TextureDimensionRecord, TexturePagingInfo, validate_texture_dimension, + }; + use crate::resources::texture::{ +- DIMENSION_SIZE, DeathKind, MIP_DIMENSION_SIZE, PagedTexture, RaiseKind, TextureLayout, ++ DIMENSION_SIZE, DeathKind, MIP_DIMENSION_SIZE, PagedTexture, RaiseKind, Staged, TextureLayout, + into_resource, mip_at, raise, record_at, + }; + use crate::rt::Hold; +@@ -255,24 +255,33 @@ impl SinglePlane { + state: &ResourceState, + encoder: Option<&P::BlitEncoder>, + ) -> RaiseResult<()> { +- // LOCAL (Atlas): a first page-up may blit from a staged copy (`PagedTexture::staged`). ++ // LOCAL (Atlas): a first page-up may blit from a staged copy, or write nothing over ++ // pages of zeros on the texture's first prepare (`PagedTexture::staged`). ++ let first = !paged ++ .prepared_once ++ .load(std::sync::atomic::Ordering::Acquire); + let staged = std::cell::OnceCell::new(); + paged.prepare( + state, +- || match paged.staged(hold)? { +- Some(buffer) => { +- let _ = staged.set(buffer); ++ || match paged.staged(hold, first)? { ++ Some(found) => { ++ let _ = staged.set(found); + Ok(()) + } + None => paged.ensure_physical(hold), + }, + || { ++ let source = match staged.get() { ++ Some(Staged::Zero) => return Ok(()), ++ Some(Staged::Buffer(buffer)) => Some(buffer), ++ None => None, ++ }; + page_through( + paged, + encoder, + false, + &[(&self.dimension, self.blit_option)], +- staged.get(), ++ source, + )?; + if self.generate_mipmaps { + nil_send!( diff --git a/patches/reims-vgpu/0033-staged-page-ups-read-guest-pages-with-one-memcpy-per.patch b/patches/reims-vgpu/0033-staged-page-ups-read-guest-pages-with-one-memcpy-per.patch new file mode 100644 index 0000000..268c60e --- /dev/null +++ b/patches/reims-vgpu/0033-staged-page-ups-read-guest-pages-with-one-memcpy-per.patch @@ -0,0 +1,64 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Mon, 9 Nov 2026 06:19:27 +0000 +Subject: [PATCH 33/33] staged page-ups read guest pages with one memcpy per + run (the volatile 8-byte loads ran at ~4 GB/s) and never read the runtime + buffer back (write-combined) + +--- + crates/reims-vgpu-core/src/gpu_task/memory.rs | 27 +++++++++++++++---- + 1 file changed, 22 insertions(+), 5 deletions(-) + +diff --git a/crates/reims-vgpu-core/src/gpu_task/memory.rs b/crates/reims-vgpu-core/src/gpu_task/memory.rs +index ccefff6a..93ebf586 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/memory.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/memory.rs +@@ -651,18 +651,35 @@ impl GpuTask

{ + let Some(dst) = bytes.get_mut(at..end) else { + return Ok(None); + }; +- if !map.read(r.physical_address, dst) { ++ if !Self::read_guest(&map, r.physical_address, dst) { + return Ok(None); + } + at = end; + } +- // Counted on first prepares only: what the zero switch would have skipped. +- let zeros = first && bytes.iter().fold(0_u8, |any, &b| any | b) == 0; ++ // The buffer's bytes are not read back: its memory may be write-combined. + times.copy = ns(copying, self.mono()); +- page_refs::note_staged(length, zeros, times); ++ page_refs::note_staged(length, false, times); + Ok(Some(Staged::Buffer(buffer))) + } + ++ /// Copies guest physical `physical..+dst.len()` into `dst` with one `memcpy`; false when ++ /// no range of the map holds it all. For a staged page-up's pages only: no GPU access to ++ /// them is pending (`memory::page_refs`) and the guest's CPU may not write a committed ++ /// command buffer's resource, so nothing writes them while they are read. ++ fn read_guest(map: &MemoryMap

, physical: u64, dst: &mut [u8]) -> bool { ++ let source = map.virtual_address_for_physical(physical, dst.len().to_u64()); ++ if source.is_null() { ++ return false; ++ } ++ #[allow(unsafe_code)] // architecture 17.3: a bulk read of mapped guest RAM ++ // SAFETY: `source..source + dst.len()` lies in one range of the map, which the map's ++ // owner keeps mapped while the map lives (`map` holds it here), and `dst` is ours. ++ unsafe { ++ std::ptr::copy_nonoverlapping(source.as_ptr::(), dst.as_mut_ptr(), dst.len()); ++ } ++ true ++ } ++ + /// Whether every byte of the guest's `ranges` reads zero, read a page at a time and stopping + /// at the first page that does not; `None` when a range is outside the map. + fn runs_read_zero(map: &MemoryMap

, ranges: &[PhysRange]) -> Option { +@@ -673,7 +690,7 @@ impl GpuTask

{ + while at < end { + let n = end.wrapping_sub(at).min(PAGE).to_usize(); + let dst = chunk.get_mut(..n)?; +- if !map.read(at, dst) { ++ if !Self::read_guest(map, at, dst) { + return None; + } + if dst.iter().fold(0_u8, |any, &b| any | b) != 0 { diff --git a/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html b/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html index e74855f..43b4887 100644 --- a/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html +++ b/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html @@ -13,7 +13,7 @@

In short

  • No VFIO, no second GPU, no guest driver: macOS's built-in driver talks to the paravirtual device.
  • Works with a Mesa Vulkan driver: Intel (measured on Arrow Lake) or AMD (supported by Reims, untested with macbox).
  • -
  • macbox's patches raised the iOS Simulator's frame rate 1.8 to 2.3 times on the measured GPU.
  • +
  • macbox's patches took a 3x iOS Simulator from 17 to 117 fps on the measured GPU.
@@ -33,20 +33,21 @@

The path macbox uses

Reims vGPU, metal2vulkan, reims-util and their QEMU fork are the work of Anees Iqbal (steelbrain). macbox builds them from pinned commits, applies its patches, and runs them in a container.

What macbox's patches change

-

On every GPU measured so far the GPU is the bottleneck, so what matters is how much extra work translation adds per frame. Before the patches, most of it was copies: macOS's framebuffer and the simulator's display surface live in guest RAM, and metal2vulkan copied each into a GPU image and back around every render pass. Those shadow copies were 51% of all GPU time with the simulator and 67% with the desktop. The patch series renders those textures in place, counts flat compute grids in 32 bits, adds Arrow Lake-specific device facts and a file of live switches for experiments, and, behind an experimental switch, Coherent: QEMU traps the guest CPU's first touch of a surface's page, so the GPU's image can stay ahead of guest RAM and only the touched pages are synced. Details: patches/README.md.

+

On every GPU measured so far the GPU is the bottleneck, so what matters is how much extra work translation adds per frame. Before the patches, most of it was copies: macOS's framebuffer and the simulator's display surface live in guest RAM, and metal2vulkan copied each into a GPU image and back around every render pass. Those shadow copies were 51% of all GPU time with the simulator and 67% with the desktop. The patch series renders those textures in place, counts flat compute grids in 32 bits, adds Arrow Lake-specific device facts and a file of live switches for experiments, and Coherent: QEMU traps the guest CPU's first touch of a surface's page, so the GPU's image can stay ahead of guest RAM and only the touched pages are synced. On top of it, batches complete in groups, a new texture's pages are staged with one copy instead of a mapping per page run, and submission checks run in the scheduler's own pass. Details: patches/README.md.

Results

Intel Core Ultra 7 265 (Arrow Lake iGPU), Mesa 26.2.4, macOS 26.7.1, iOS 26.5, 8 vCPUs and 12 GB. GlassBench is a table scrolling under Liquid Glass bars and a floating glass toolbar; frames counted at the simulator's screen, two runs each.

- + - + +
WorkloadReims 2ce3209Reims 7198fe6macbox, Oct 28macbox nownow, Coherent on
WorkloadReims 2ce3209Reims 7198fe6macbox, Oct 28macbox, Nov 5, Coherent onmacbox now
Simulator 1x (402×874)53.557.095.8 / 92.5n/mn/m
Simulator 2x (804×1748)33.537.667.8 / 60.5n/mn/m
Simulator 3x (1206×2622)17.219.238.5 / 39.550.670.0
Simulator 3x (1206×2622)17.219.238.5 / 39.570.0117
Three 3x simulators at once, eachn/m31–32
Safari at 3x, blurred floating barn/a39.566.4 / 64.2n/mn/m
-

Frames per second; n/m: not measured on the new build yet. "Now" and "Coherent on" are one A/B in one window (2026-11-05). With Coherent the GPU is 70% busy at 3x instead of 97%, and QEMU uses about 1.6 more CPU cores. A Genie minimize on a 3440×1440 desktop went from about 12 to 19.5 frames per animation on the Oct 28 build, and 30 with Coherent. Coherent is off by default (MACBOX_COHERENT=on): it hung the GPU once on its first day of daily use, and the cause isn't settled. Measure your own GPU with the tools in bench/.

+

Frames per second; n/m: not measured on that build. The Nov 5 column is the cohwl A/B (2026-11-05); "now" is the build macbox makes since 2026-11-09, from its soak and A/B runs. A 3x frame now takes about 7.7 ms of GPU time: ~4.7 ms is the simulator's own rendering, the rest is what translation still adds (copies for draws that read their own pass, barriers between command buffers). A Genie minimize on a 3440×1440 desktop went from about 12 to 19.5 frames per animation on the Oct 28 build, and 30 with Coherent. Coherent is on by default with an Arrow Lake iGPU (MACBOX_COHERENT=off turns it off), behind a crash guard; one rare GPU hang under repeated boots of three simulators is still open. Measure your own GPU with the tools in bench/.

Which GPUs

@@ -70,7 +71,7 @@

Settings that matter

Known limits

  • Animations start about 0.1 s late on an idle display: macOS's driver for this device sends no vblank while nothing presents (reims-vgpu#141).
  • -
  • 3x is GPU-bound at about 50 fps in heavy Liquid Glass scenes on this iGPU (70 with Coherent).
  • +
  • The GPU is shared: one 3x simulator reaches about 117 fps on this iGPU; two or three share about 105 frames a second.
  • Tried without gain: answering as an Intel or AMD Mac GPU, compressed heap textures, newer Mesa, hugepages (impossible: Reims maps guest RAM in 16 KiB views). The full list is in docs/performance.md.

To set it up, follow How to run macOS on Linux.

diff --git a/site/pages/guides/ios-simulator-on-linux.html b/site/pages/guides/ios-simulator-on-linux.html index a7cec24..3e6388d 100644 --- a/site/pages/guides/ios-simulator-on-linux.html +++ b/site/pages/guides/ios-simulator-on-linux.html @@ -63,10 +63,10 @@

How fast it is

- +
1x (402×874)53.5 fps95.8 / 92.5 fps
2x (804×1748)33.5 fps67.8 / 60.5 fps
3x (1206×2622)17.2 fps50.6 fps (70.0 with Coherent, experimental)
3x (1206×2622)17.2 fps117 fps (31–32 each with three at once)
-

1x and 2x are from the October 28 build; 3x is the current one. The GPU sets the ceiling: a 3x simulator keeps that iGPU 96 to 98% busy (70% with Coherent). Simulators animating at the same time share it. Recording uses the VM's software H.264 encoder, about 50 fps with one simulator animating. Full method: docs/performance.md.

+

1x and 2x are from the October 28 build; 3x is the current one. The GPU sets the ceiling: a 3x simulator takes about 7.7 ms of that iGPU per frame and comes close to the iPhone 17 Pro's 120 Hz screen; simulators animating at the same time share about 105 frames a second. Recording uses the VM's software H.264 encoder, about 50 fps with one simulator animating. Full method: docs/performance.md.

Limits

    diff --git a/site/pages/guides/run-macos-on-linux.html b/site/pages/guides/run-macos-on-linux.html index 56af118..d448804 100644 --- a/site/pages/guides/run-macos-on-linux.html +++ b/site/pages/guides/run-macos-on-linux.html @@ -24,7 +24,7 @@

    How it works

  • metal2vulkan runs the Metal work that device receives on Vulkan, through Mesa on your GPU.
  • The host keeps its GPU. Nothing is passed through, and an integrated GPU is enough.
-

macbox builds both from pinned commits with its own patches, which made the iOS Simulator 1.8 to 2.3 times faster on the GPU they were measured on. The GPU guide explains the path in detail.

+

macbox builds both from pinned commits with its own patches, which took a 3x iOS Simulator from 17 to 117 fps on the GPU they were measured on. The GPU guide explains the path in detail.

Requirements

diff --git a/site/pages/index.html b/site/pages/index.html index 044a9fd..24969a0 100644 --- a/site/pages/index.html +++ b/site/pages/index.html @@ -5,7 +5,7 @@

macbox

Give your agents a Mac.

-

macOS, Xcode and real iPhone simulators, on your Linux machine. Your coding agent builds the app, taps through it and checks it for itself.

+

macOS, Xcode and real iPhone simulators at up to 120 fps, on your Linux machine and its own GPU. Your coding agent builds the app, taps through it and checks it for itself.

Get started View on GitHub @@ -92,27 +92,27 @@

Works with the agents you already use.

Performance

-

17 → 50 fps

-

Smooth enough to judge an animation.

-

An iPhone simulator at 3x, an iPhone's real pixels, running a heavy Liquid Glass screen on an Intel integrated GPU. That's three times what the GPU code macbox builds on manages, and the experimental Coherent switch takes it to 70.

+

17 → 117 fps

+

As smooth as the iPhone's 120 Hz screen.

+

An iPhone 17 Pro simulator at 3x, an iPhone's real pixels, running a heavy Liquid Glass screen on an Intel integrated GPU: almost seven times what the GPU code macbox builds on manages. Three simulators at once still run at about 32 fps each.

- iOS Simulator at 3x in a Liquid Glass benchmark: 17.2 fps as upstream shipped it, 39.0 with the October 28 macbox build, 50.6 with the current build, 70.0 with Coherent on. + iOS Simulator at 3x in a Liquid Glass benchmark: 17.2 fps as upstream shipped it, 39.0 with the October 28 macbox build, 50.6 on November 5, 70.0 with Coherent, 117 with the build macbox makes now.
The exact numbers
- + - + + + - - - +
WorkloadReims 2ce3209, upstreammacbox, Oct 28macbox nownow, Coherent on
WorkloadReims 2ce3209, upstreammacbox, Oct 28macbox, Nov 5, Coherent onmacbox now
iOS Simulator 3x (1206×2622, an iPhone's pixels)17.2 fps38.5 / 39.5 fps50.6 fps70.0 fps
iOS Simulator 3x (1206×2622, an iPhone's pixels)17.2 fps38.5 / 39.5 fps70.0 fps117 fps
Two 3x simulators at once51–54 fps each
Three 3x simulators at once31–32 fps each
iOS Simulator 1x (402×874), Liquid Glass53.5 fps95.8 / 92.5 fpsn/mn/m
iOS Simulator 2x (804×1748), Liquid Glass33.5 fps67.8 / 60.5 fpsn/mn/m
macOS desktop, dragging a window (presents/s)105130
macOS desktop, Safari scrolling (presents/s)75–79110
macOS desktop, Genie minimize at 3440×1440~12 frames19.5 frames29.5 / 30.5 frames
macOS desktop, Genie minimize at 3440×1440~12 frames19.5 frames29.5 / 30.5 framesn/m
-

Intel Core Ultra 7 265 (Arrow Lake iGPU), Mesa 26.2.4, macOS 26.7.1, iOS 26.5, a VM with 8 vCPUs and 12 GB. Frames counted at the simulator's own screen, after a warm-up, one simulator animating; n/m: not measured on the new build yet. "Now" and "Coherent on" are one A/B in one window. Coherent is off by default: it hung the GPU once on its first day of daily use, and the cause isn't settled. The GPU is the limit, so a faster GPU moves every number up; other GPUs haven't been measured yet. Method and patches: docs/performance.md.

+

Intel Core Ultra 7 265 (Arrow Lake iGPU), Mesa 26.2.4, macOS 26.7.1, iOS 26.5, a VM with 8 vCPUs and 12 GB. Frames counted at the simulator's own screen, after a warm-up, headless; n/m: not measured on that build. The iPhone 17 Pro's screen runs at up to 120 Hz, so one simulator is close to its limit; several share the GPU, about 105 frames a second in total. Coherent is on by default with an Arrow Lake iGPU, with a crash guard; one rare GPU hang under repeated boots of three simulators is still open. The GPU is the limit, so a faster GPU moves every number up; other GPUs haven't been measured yet. Method and patches: docs/performance.md.

diff --git a/versions.env b/versions.env index e13bd0c..06988b2 100644 --- a/versions.env +++ b/versions.env @@ -1,5 +1,5 @@ # Pinned sources for `macbox build`. With the patches in patches/ they make the build that has run day -# to day since 2026-11-05 on an Intel Arrow Lake iGPU (Core Ultra 200S, Mesa 26.2.4 ANV), "cohwl": +# to day since 2026-11-09 on an Intel Arrow Lake iGPU (Core Ultra 200S, Mesa 26.2.4 ANV), "production I": # Reims vGPU 7198fe6 + patches/reims-vgpu, its QEMU at bd95509 + patches/qemu, and metal2vulkan's # development branch at 0db5825a + patches/metal2vulkan. `macbox build --upstream` takes Reims's # master instead, with no patches.