diff --git a/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx b/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx index 408b8ce1b1e..5f8fbdc6ad5 100644 --- a/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx +++ b/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx @@ -172,6 +172,19 @@ device_working_mem_gb: 4 Once decoding completes, the model manager "reclaims" the extra VRAM allocated as working memory for future model loading operations. ::: +#### Calibrating a model's working memory + +Invoke's decode estimates are measured on a few reference GPUs. Your GPU, driver and settings can need more or less. A calibration measures what a model's VAE decode really takes on your GPU, and Invoke uses those measurements from then on. + +- **What it runs**: a few short text-to-image generations of the model at three sizes, up to the largest your GPU can decode in one piece with the VAE fully loaded in the memory free on it when the calibration is planned, in each VAE precision the model can decode in. Before each measured operation, Invoke moves the other models off the GPU, so the measured model is fully loaded. Each size runs once to warm up and twice more. The generations go through the normal queue, so they wait behind queued work and appear there like any other generation. +- **Who and where**: administrators only. It needs exactly one GPU set as the generation device; with several, it is not available. +- **What it changes**: decodes up to the largest size measured use the measured values, plus the same 5% margin the built-in values carry. Measurements that a fragmented GPU memory pool inflated, or taken while the VAE was only partly on the GPU, are left out, and the Model Manager shows those sizes as not counted. Larger decodes use whichever of the measured and the built-in estimate is larger. Other models that decode the same way (for example, SD1.5 and SDXL models when you calibrate an SDXL model) use the measurements too. Because the estimate decides when a large decode is split into tiles up front, a calibration can change which large images are decoded in tiles. +- **Where it applies**: on this kind of GPU only (identical cards share it), and only while the software that affects memory stays the same: the operating system, the PyTorch and GPU runtime versions, the memory-related environment variables and Invoke's memory settings. After such a change Invoke goes back to the built-in values until you calibrate again. A driver update keeps the measurements but marks them as possibly out of date until you calibrate again. +- **What it does not cover**: image-to-image and inpainting encodes keep the built-in values. +- **Reset**: remove a model's calibration from the Model Manager, or every calibration of this GPU; the change applies right away. Resetting one model removes only its own measurements: where other models' measurements cover the same decode, those still apply, and the Model Manager shows which models measured it. Otherwise the built-in values apply again. + +The measurements are stored next to the database in `working_memory/.json`. + ### Keeping a RAM weight copy Invoke has the option of keeping a RAM copy of all model weights, even when they are loaded onto the GPU. This optimization is _on_ by default, and enables faster model switching and LoRA patching. Disabling this feature will reduce the average RAM load while running Invoke (peak RAM likely won't change), at the cost of slower model switching and LoRA patching. If you have limited RAM, you can disable this optimization: diff --git a/docs/src/content/docs/contributing/new-model-integration.mdx b/docs/src/content/docs/contributing/new-model-integration.mdx index c6325083423..2a4ce44cc00 100644 --- a/docs/src/content/docs/contributing/new-model-integration.mdx +++ b/docs/src/content/docs/contributing/new-model-integration.mdx @@ -601,7 +601,7 @@ Packages are discovered automatically. A directory of node modules without an `_ ... ``` - Report progress with `context.util.sd_step_callback(state, BaseModelType.NewModel)`; it renders previews through the architecture's `LatentSpaceFacet`. The denoise node is also where these are checked: interrupts (`context.util.is_canceled()`), inpainting, img2img through `denoising_start`, and a working-memory estimate passed to `model_on_device(working_mem_bytes=)`, or `working_memory=` for an estimator that declares an `EstimatorSpec`. + Report progress with `context.util.sd_step_callback(state, BaseModelType.NewModel)`; it renders previews through the architecture's `LatentSpaceFacet`. The denoise node is also where these are checked: interrupts (`context.util.is_canceled()`), inpainting, img2img through `denoising_start`, and a working-memory estimate passed to `model_on_device()`: `working_memory=` from an estimator that declares an `EstimatorSpec`, so that calibration runs measure the forward (the Z-Image denoise's is in `invokeai/backend/z_image/denoise_working_memory.py`), or `working_mem_bytes=` for a plain reservation. A model locked while the transformer is, such as a ControlNet adapter, reserves the estimate's `reservation_bytes` with `working_mem_bytes=`. 4. **VAE encode and decode** (`vae/new_model_image_to_latents.py`, `vae/new_model_latents_to_image.py`) diff --git a/docs/src/content/docs/development/Architecture/model-manager.mdx b/docs/src/content/docs/development/Architecture/model-manager.mdx index dd3a5680b95..2f99742daf9 100644 --- a/docs/src/content/docs/development/Architecture/model-manager.mdx +++ b/docs/src/content/docs/development/Architecture/model-manager.mdx @@ -1012,13 +1012,123 @@ with vae_info.model_on_device(working_memory=estimate) as (_, vae): An estimator that declares an `EstimatorSpec` returns a `WorkingMemoryEstimate` (`invokeai/backend/working_memory/estimate.py`): the reservation, the terms it adds up to, the features it was evaluated from, and the source of its coefficients, which `model_on_device()` logs at debug level. A plain byte count goes in as `working_mem_bytes=` instead; never both. The cache never reserves less than `device_working_mem_gb`. -To write such an estimator (`invokeai/backend/util/vae_working_memory.py` has five): +To write such an estimator (`invokeai/backend/util/vae_working_memory.py` has five, `invokeai/backend/z_image/denoise_working_memory.py` the Z-Image denoise's, `invokeai/backend/flux2/denoise_working_memory.py` the FLUX.2 denoise's): - Register an `EstimatorSpec` at module import, and list the module in `ESTIMATOR_MODULES` (`working_memory/spec.py`); the data test fails on a module that declares a spec without being listed. -- Split the estimator in two. The impure part reads the model and the device and records every runtime decision (tile, element size, score-matrix bytes) as a feature. The pure `terms(features, values)` turns features and coefficient values into terms, building fitted terms with `WorkingMemoryTerm.fitted(...)`. The public function returns `SPEC.estimate(features, device)`. +- Split the estimator in two. The impure part reads the model and the device and records every runtime decision (tile, the dtype the operation runs in, score-matrix bytes, a quantized model's dequantization transient) as a feature, and refuses a dtype the spec has no line for with a `ValueError` naming the estimator, dtype and platform. The pure `terms(features, values)` turns features and coefficient values into terms, building fitted terms with `WorkingMemoryTerm.fitted(...)`. The public function returns `SPEC.estimate(features, device)`. +- Record the dtype the operation will run in, not the one the cached model holds. The SD1/SDXL nodes cast the shared VAE with `sd_vae_run_dtype`: fp16 for every GPU run that does not ask for fp32 (bf16 never runs), fp32 when asked and for a CPU decode. Their estimate prices the same function's answer, plus what the in-place cast adds to the resident weights. Uncast VAEs compute in their first floating-point parameter wider than a byte, so fp8 or packed SDNQ storage does not count. +- Declare the line the fitted terms price on with `line=` (a function returning a `fit.Line`): the suffix that names the line's coefficients, the operation's size on its x-axis, whether a measurement of these features fits the line, and how point ids label them. Every estimator declares it and nothing defaults it, so a new estimator cannot inherit another's line, and with it which of its measurements fit. The VAE estimators declare the AutoencoderKL convolution line in output pixels (`fit.convolution_line`), the Z-Image and FLUX.2 denoise ones in attended tokens. The fit subtracts nothing from a point's demand, so a line admits only measurements whose demand is the line's alone: an operation with a term of its own on top of the line (a tile, an in-place upcast, a ControlNet's hints, regional prompting's bias, LoRAs, a dequantization or fp8 cast transient, a materialized score matrix the estimate adds rather than takes the larger of) bounds its estimate but fits nothing. +- Opt in with `user_calibrated=True` where a user's calibration may move the estimator: the five VAE estimators do. An estimator that does not is shipped-fit only; a calibration run still records its windows. The calibration summary and plan are built on pixel lines (they key sizes by `width` and `height`), so an estimator on another line, such as the Z-Image denoise's in tokens, must not opt in until they key sizes by `Line.size`. - Bump `version` whenever `terms` changes the formula. -- Ship one coefficient entry per measured platform value in `invokeai/backend/working_memory/data/`, either with its own `values` or with `values_from` another estimator's entry. Unmeasured platform values (MPS, XPU, CPU) are priced as `UNMEASURED_PRICED_AS` in `working_memory/platform.py` says. -- Ship the measured points next to them: peak reserved bytes (`reserved_peak`), one fresh process per point. `tests/backend/working_memory/` holds each estimate to at least its point, and where a fitted term decides it, to at most 1.3 times the point, or the point's tighter `max_ratio`. The `device_working_mem_gb` floor counts only where an analytic term decides. Every value an entry carries must be decided by at least one point. +- Ship one coefficient entry per measured platform in `invokeai/backend/working_memory/data/`, either with its own `values` or with `values_from` another estimator's entry, which may carry more values than the borrower reads. + - A platform is the convolution backend (`cudnn`, `miopen`, `mps`, `xpu`, `cpu`), torch's caching allocator and the OS. The allocator is torch's backend first (`cudaMallocAsync` ignores expandable segments), then `expandable` if the allocator environment variables enable expandable segments on the native allocator, else `native`; anything but a CUDA/HIP GPU is `native`. `platform_for(device)` reads them from this process; `platform_for_environment(backend, hard)` reads the same facts from a measurement's recorded environment, through the same function. + - `MEASURED` in `working_memory/platform.py` lists the measured backend, allocator and OS combinations (an OS of `None` for every OS). `priced_as` prices every other pair as one of them, keeping the OS: a backend nothing is measured on as cuDNN with the native allocator, an allocator not measured with its backend as that backend's native allocator. There the entry that carries the values raises them to its `unmeasured_floor`: the pre-dtype slopes for the VAE lines, the legacy constants for the denoise lines. A fit describes the card it was measured on, not the platforms it stands in for. + - An entry applies to the OS only when its platform names one, and then takes precedence there over the one that does not. Only MIOpen names one. With the native allocator, Windows demand is concave and up to 1.6x Linux's, so Windows has its own fit and every other OS keeps the Linux one. Expandable segments are measured on Windows only, where they are Invoke's ROCm default, so `MEASURED` lists them for Windows and every other OS prices them as the native allocator there. cuDNN is priced the same on every OS. +- Ship the measured points next to them. A point's `method` says how it was read: one fresh process per point for `reserved_peak` (the allocator's peak reserved bytes), `driver_peak` (the driver-level device usage), and `demand_peak` (the largest of driver-level, reserved and allocated peak, over two runs, which catches the memory HIP maps beyond the allocator on a process's first operation); or `probe_demand`, the probe's `demand_bytes`, the largest over the valid, fit-eligible windows of calibration runs in a running server. `scripts/working_memory/build_shipped.py` builds `probe_demand` points from calibration store exports, takes fresh-process points from the point files a calibration script writes (`--points`), and refits the values (see the Family E2E guide). A point names the OS only where it measured an OS-specific entry. `working_memory/bounds.py` holds the rule the data test (`tests/backend/working_memory/`) and the builder apply: each estimate is at least its point, and where a fitted term decides an in-fit point, at most 1.3 times the point, its ratio in `bounds.UPPER_BOUND_EXCEPTIONS` (points no line can hold within 1.3x), or the point's tighter `max_ratio`. A `probe_demand` point is a lower bound only (`MeasuredPoint.holds_upper_bound`) and takes no `max_ratio`: a warm window in a running server reads below the cold first run after a server start, the worst case a reservation must cover. It may raise a line; only the fresh-process points cap one. The `device_working_mem_gb` floor counts only where an analytic term decides. +- Fit the values, never hand-tune them. Fitted terms price a line per operation and dtype: `k_` bytes per unit of the line's size plus a base of `b_` bytes. The AutoencoderKL estimators price their convolutions on lines in output pixels (`k__`), the Z-Image denoise its forward on lines in attended tokens (`k_denoise_`), the FLUX.2 denoise its forward on one such line per dtype and transformer width (`k_denoise__h`). + - `working_memory/fit.py` fits each line on the in-fit points its estimators' `line` admits, of every estimator that resolves to the entry: for the convolution lines the untiled single-sample points, leaving out those whose VAE an in-place cast grew (`upcast_weight_bytes`, a term of its own). The line is the edge of the points' upper convex hull (with the origin, so the base is never negative) that over-reserves least, rated on the highest reading per size of the points that hold an upper bound, the steeper on a tie and never a falling one. A `probe_demand` point is covered by the line and can raise it, but rates no edge. It needs at least two sizes twice apart. The shipped values add a 5% margin and round up: the slope to a whole byte per unit of size, the base to a whole MiB. + - A point measured in a regime its platform does not run sets `in_fit: false`: it fits nothing and gets only the lower bound (CUDA with the math attention kernel forced). The test lists every such point. + - A line no in-fit point measures is a declared stand-in, listed in the entry's `stand_ins` and never below the pre-dtype slope. MIOpen with the native allocator off Windows prices fp16 and fp32 at the pre-dtype figure, and CogView4's lines there at its expandable-segments ones. + - Every value an entry carries is a stand-in, a `legacy` constant, or decided by an in-fit point. The data test fails when a fitted value is not the fit of its points, naming the value to ship, and when an in-fit point measures a stand-in's line. + - An entry marked `legacy` carries hand-set constants rather than a fit: no point decides them and `fit.py` does not reproduce them, though its estimator's points still bound its estimates. `LEGACY_ENTRIES` in the data test lists the entries, by estimator and platform, allowed to be legacy, so a fit has to remove its entry there too. The builder replaces an entry's legacy values with the fit of its points once they fit every line it carries and each line has a point that holds the upper bound, a fresh-process one: probe windows alone never replace hand-set values. An entry whose platform they do not measure stays legacy. +- The Z-Image denoise (`denoise.z_image`, `data/denoise.json`) prices its activations as `k_denoise_` bytes per attended token plus a base of `b_denoise_`. The lines are fitted on fresh-process points of `scripts/calibrate_z_image_working_memory.py` (a random-init transformer of the real geometry with 2 main blocks, bf16, fp16 and fp32 from 512 to 2048 px): an RTX 4090 with the native allocator and with `cudaMallocAsync`, and an RX 9060 XT under ROCm 10 with AOTriton's fused kernels, with expandable segments and with the native allocator on Windows (an entry of its own). In a running server the real 30-block Z-Image Turbo measured below its estimate on the RTX 4090 under the native allocator and `cudaMallocAsync` (bf16) and on the RX 9060 XT with expandable segments (an int8 build). ROCm with the native allocator off Windows has no points and keeps the legacy constants, 0.25 MiB per token and a 1 GiB base for every dtype; they cover bf16 and fp16 but not fp32 from 1440 px up, whose measured lines are steeper. Every other unmeasured platform (XPU, MPS, CPU, cuDNN with expandable segments, MIOpen under `hipMallocAsync`) is priced as a measured one, and the entries it borrows floor every value at the same legacy constants there (`unmeasured_floor`), so no unmeasured platform reserves less than the legacy formula. + - The line admits only the plain forward, measured with a fused attention kernel: every analytic term zero and no attention mask, read off the estimator's terms, so a term added later takes the forwards it prices out of the fit. A point whose score matrix is materialized fits nothing, because the score term is an upper bound, and fitting the demand less the term would leave the line low by the term's slack, short for a build of the same platform keys that runs a fused kernel. A ControlNet's hints (n + 4 hint-sized tensors) and a forced math kernel's score matrix are checked against measurements of their own in `tests/app/invocations/test_z_image_denoise_working_memory.py`. + - An fp8-stored transformer reserves the per-forward cast of its weights as a term of its own (`util/fp8.py`, `peak_fp8_cast_transient_bytes`): the largest layer's float8 weights at the compute dtype, twice for a scaled checkpoint weight. The legacy base covers that cast too, so wherever the legacy constants price, such a model reserves it twice. + - Denoise lines are shipped-fit only: `denoise.z_image` is not `user_calibrated`. +- The FLUX.2 denoise (`denoise.flux2`, `data/denoise.json`) prices its activations as `k_denoise_bf16_h` bytes per attended token and sample plus a base of `b_denoise_bf16_h`, a line per transformer width (Klein 4B 3072, Klein 9B 4096, [dev] 6144; an unrecognized variant prices as the widest). The attended tokens are the caption's, the image's and every reference image's, and the line's size is those times the batch. The node runs bf16 only. + - The lines are fitted on fresh-process points of `scripts/calibrate_flux2_denoise_working_memory.py` (per width 512 to 2048 px with a 512-token caption, 1024 and 1440 px with a 77-token one, at 1024 px one and three 1024 px references and a batch of two, and long sequences up to 2048 px with one 2024 px reference (33025 tokens), on a random-init transformer of the loader's geometry with 2 blocks per stream): an RTX 4090 with the native allocator and with `cudaMallocAsync`, and an RX 9060 XT under ROCm 10 with AOTriton's fused kernels, with expandable segments and with the native allocator on Windows. On the RX 9060 XT [dev] does not fit 16 GB above 1440 px or with three references, so its [dev] lines there come from 512 to 1440 px, one reference included; every other line is measured up to 33025 tokens where the forward fits the card. The node caps neither the image's side nor the number of references, so a longer sequence prices on its line extrapolated linearly: fused attention keeps the activations linear in the sequence, and from 16896 to 33025 tokens the line reserves a flat 1.05 to 1.13 times what each measured forward demands under the native allocator and expandable segments; under `cudaMallocAsync` the 33025-token forwards decide the line (1.05). Several references are resized to 1024 px each, as the node resizes them. 4 blocks per stream measured the same as 2 on the RTX 4090 and 5 to 7% more on the RX 9060 XT. In a running server the real models demanded less than the 2-block point at the same tokens: an fp8 Klein 4B and 9B 0.96 to 0.99 times it on the RTX 4090 (native allocator), a bf16 Klein 4B 0.81 to 0.95 and an fp8 Klein 9B 0.71 to 0.77 (after its fp8 cast term) on the RX 9060 XT (expandable segments). The Klein 4B line holds its 1440 px forward with a 77-token caption at 1.42x and 1.40x under the native allocator, listed in `bounds.UPPER_BOUND_EXCEPTIONS`. ROCm with the native allocator off Windows has no points and keeps the legacy constants the node reserved by hand, 0.42 MiB per token at width 4096 scaled linearly by the width and truncated per width (330300, 440401 and 660601 bytes) and a 1 GiB base, which `tests/app/invocations/test_flux2_denoise_working_memory.py` holds byte for byte to the formula they replace; they lie above every measured line. The entries unmeasured platforms borrow (cuDNN with the native allocator, MIOpen with the native allocator on Windows) floor every value at those constants, as the Z-Image entries do. A forced math kernel measured 8.74 bytes per score element over the default kernel on the RTX 4090, against the 17 the score term charges. + - The line admits the plain forward, reference images and batches included: a batch adds sequences, which the line prices, and its base is paid once per forward. Every forward with an analytic term (regional prompting's bias, a materialized score matrix, LoRAs, a dequantization or fp8 cast transient) or a mask bounds its estimate but fits nothing. An fp8-stored transformer reserves its cast transient on top, which the legacy base covers too. + - `denoise.flux2` is not `user_calibrated`. + +#### Measuring estimates: the probe + +`invokeai/backend/working_memory/probe.py` measures what an estimated operation really takes. It is active only inside a `calibration_scope(...)`, which the session runner opens per node of a calibration queue item; without one, `model_on_device()` behaves as above and reads no counter. + +Inside a scope, every `model_on_device(working_memory=...)` on a GPU is a window. The lock reserves the estimate plus calibration headroom (half the estimate, at least 1 GiB), so a low estimate cannot spill. The window runs from the settled lock to the unlock and collects a `MeasuredWindow` in the scope's `windows`; nothing is written while it is open. + +- Counters (`working_memory/counters.py`): torch's reserved and allocated peaks, and this process's usage as the driver counts it. That is D3DKMT on Windows for CUDA and ROCm (the only readers verified on hardware), NVML on Linux CUDA, DRM fdinfo on Linux ROCm, and the MPS driver figure. XPU's Sysman figure covers the whole device, so it only detects another process's load. MPS has no peaks; a sampler thread keeps them while a window is open. +- Demand: per counter, the largest segment peak less the baseline and the bytes co-locks moved onto the device; `demand_bytes` is the largest of those deltas. Each counter misses something: the driver misses HIP's retained pool, and reserved misses cuBLAS and cuDNN workspaces and, under `cudaMallocAsync`, part of the allocated peak. +- Segments: the window opens after a release, so cached blocks and tensors from before the lock form the baseline. Another model locked on the same thread inside the window is a co-lock: its weights go into the offset, and its lock interval is kept out of the peak. An estimated window inside another measures alone while the outer one is frozen. A `TorchDevice.empty_cache()` ends a segment. A direct `torch.cuda.empty_cache()` call (from a library, or the one cuDNN and MIOpen run after an algorithm search) and an offload by the cache's background thread do not: the driver figure can miss what they released, while the reserved and allocated peaks still bound the demand. +- Invalid (`valid=False`, with `reasons`): another thread locked a model on the same device's cache during the window, held one when it opened, or was still locking one then (`foreign_lock`); the allocator failed or retried; Windows still holds more than 256 MiB of the process in shared memory after a short settle; the operation raised or was canceled, or its node failed or was canceled after the window closed; the Krea-2 SDPA benchmark is active; XPU's device-wide usage grew beyond this process's own (`foreign_device_load`); a probe step failed (`probe_error`: a reading, a release, a peak reset or a synchronization, which also voids the scope's later windows when it hit the lock bookkeeping; a window that cannot open is recorded invalid, and its operation runs unmeasured); or no counter could be read. +- Not fit-eligible: a model that was not fully resident when the window opened. The measurement stays valid, but streamed weights are working memory the estimate does not price. + +`working_memory/fingerprint.py` keys measurements. `device_class_for()` gives the same id to identical cards. `environment_for()` splits hard facts, which change an operation's memory and make up its id (the operating system as `windows`, `linux` or `other`, torch, runtime and convolution-library versions, allocator backend and configuration, the ROCm SDPA guard, the memory settings passed in), from soft facts kept for diagnosis (full versions, driver, OS build). + +#### Calibration runs + +A calibration run measures one model's operations on this machine's GPU; estimates read the result as user fits (below). The service is `invokeai/app/services/working_memory_calibration/`, the routes are in `invokeai/app/api/routers/working_memory.py`, and every route is admin-only (403 otherwise). + +1. `POST /api/v2/models/i/{key}/working_memory/calibrate` takes a client-built graph, batch data, an optional `idempotency_key` and an optional `attempt_id`. The server turns off `use_cache` on every node, sets `runs=1`, sets the graph's id to the run id, and sets the origin to `invokeai:wm-calibration:` and the destination to `invokeai:wm-calibration`. It records the run, enqueues the batch as the requesting admin, then records the run's items. + - An `attempt_id` (1–64 of letters, digits, `_ - . :`) groups the runs a client enqueues for one calibration attempt; it is stored on the run (and left out of the file when absent, so older readers keep reading such runs) and returned in the run summaries of the status and the model summary. + - The queue takes all of the batch's items or none (`enqueue_batch(..., all_or_none=True)`, checked under its admission lock): a run with only some of its cases would measure only some of them. Without room for all of them nothing is enqueued, the key stays unused, and the run is removed if this request recorded it. + - With a key, the run id is derived from the owner, the key, the request's fingerprint (the model key, the graph without its id, and the batch data) and the `attempt_id` if given. A client that sends no graph id would otherwise get a fresh one each time and could never repeat a request. A repeated request finds its run and gets the same response; nothing new is enqueued, even after the queue collected the key's receipt. A key therefore names one run. Once a reset removed the run, a repeat is refused (404) whether or not the queue still keeps the receipt: the store keeps the ids of removed runs, so the run is never started again and its queued items keep failing at admission. Another request that reuses the key (another model, graph, batch data or `attempt_id`) gets another run id and so another origin. While the key's receipt exists, the queue refuses that request (409) and the first run stays. Once the queue has collected the acknowledged receipt, the request starts a run of its own. Clients make each key unique per run of an attempt, for example from the attempt id and the step. Acknowledge the key through `POST /api/v1/queue/{queue_id}/enqueue_batch/acknowledge`. + - Once its items are queued, the request answers with them, also when the enqueue raised after its commit (the queue announces a batch after committing it). When the queue cannot tell whether it holds them, the run is kept and the enqueue's own error answers. + - If the store fails for a passing reason (an unwritable root, a lock timeout) to record the queued items, a request without a key still succeeds: its items run, and the status finds them by their origin. With a key that is a 503, and a repeat records them: from the key's receipt while the queue keeps it, otherwise from the queue items with the run's origin and the items the runner saw. It never enqueues a second batch for the run. Recording them from the receipt checks neither the model nor the device, so a model removed meanwhile does not fail the repeat; a request that has to enqueue needs the model. A run removed meanwhile (404) or an unusable store file (409) fails the request with or without a key; its queued items then fail at admission. + - Per run id, one creation runs at a time, and it runs to its end even when its request is canceled: the enqueue runs in a worker thread and can commit after the cancellation. Only the server's shutdown cancels the creation with it; the run then stays as far as it got, and with a key a repeat of the request completes it. The lock belongs to the process, so two processes serving one root and its SQLite database can interleave the creations of one run. A run is removed only when this request recorded it and nothing of it was queued. + - The destination only stops clients from routing the results to a gallery or canvas view. Images the graph's nodes save are stored like those of any other session: on the board and with the intermediate flag each node sets. +2. `enqueue_batch` refuses any origin that starts with that prefix, in any case, for every user (400). Only the calibration route creates calibration items. Retries and workflow-call children inherit origin and owner from their source item. +3. `DefaultSessionRunner.run()` checks each item's origin once. For a calibration item it calls `admit()` before any node runs, also for one left pending before a restart: the invoker starts the session processor after every other service. (The queue's start cancels the items that were in progress.) Retries and children are checked here too. Otherwise the item fails with the reason. The checks: + - the owner is an active admin (single-user mode counts as admin); + - the machine has exactly one GPU generation device; + - the run exists for that GPU's device class. +4. Each admitted node runs in `probe.calibration_scope(...)`, keyed by the node's source id. Its windows are persisted after the node completes, fails or is canceled. Unless the node completed, every one of them is persisted invalid (`exception` when it failed, `canceled` otherwise), including a window whose own operation finished before the node raised: `run_node` swallows the failure or cancellation, so the runner takes the outcome from its node hooks. Windows from a device of another class, or of a run that was removed, are dropped with a warning. So are windows of a model reset after the node started, also by another process sharing the root: the runner reads the device class's reset epochs before the node's scope opens, and the write drops every point attributed to a model whose epoch has moved since, as the reset attributes them (`measured_models`: a submodel's `:vae` window counts for its model). Without that, a reset while the node runs (of a VAE inside a transformer's run, say) would be undone by the node's later write. If they cannot be saved, the item fails and the session stops. Every other item gets no scope and runs as before. +5. After the item, the runner records the item's outcome in the run, even when the session raised. An item the queue no longer has (Clear Queue) is recorded as canceled. An item that was still running when the runner raised is recorded as failed with the error. + +The store has one JSON file per device class at `/working_memory/.json` (`CalibrationStoreFile`, `schema_version` 1). It is machine data, so it lives next to the database rather than inside it. It holds: + +- the device class, with its id; +- the runs: model key, model hash as provenance, owner, environment and its soft facts at creation, queue items, and the last status the runner saw for each item; the soft facts and the `attempt_id` are left out when absent, so runs stored without them still load and older readers still read runs that lack them; +- the points: `MeasuredWindow` plus `env_id`; +- the environments those reference; +- the ids of the runs with a retry key that a reset removed (`removed_run_ids`), so that a repeat of the request that started one cannot start it again. They are kept for good, about 40 bytes per removed run. A run without a key gets a fresh id and is never repeated, so its id is not kept; +- per model key, its reset epoch (`model_reset_epochs`): how often the model was reset. The reset raises it in the same file change, so it is only written when the reset is. + +A file without either of these two fields reads them as empty. + +Writes and reads: + +- Every change takes a process lock and an OS lock on the file's `.lock` sibling, re-reads the file, applies the change, and replaces the file atomically. Two processes sharing a root therefore never lose each other's writes. +- On Windows a process that ignores the lock (an antivirus scanner, a search indexer, an editor) can hold the file open and refuse the replace. The write then retries a few times with backoff, about 1.5 s in all (`REPLACE_ATTEMPTS`, `REPLACE_RETRY_S`). +- Readers take the lock as well, because Windows cannot replace a file another process has open. +- Each change rewrites the whole file, which is fine at calibration volumes. + +A file is unusable when it has another schema version, invalid content, an id that does not match its fields or its file name, or a field this version does not know. Every store model forbids unknown fields: ignoring one would drop it at the next rewrite. An unusable file is reported in the status, makes calibration unavailable for its device class, and is never written, moved or deleted. + +A run's status is derived, never stored: + +- The live queue items with the run's origin come first. For items the queue has pruned, the runner's recorded statuses are used. An item the queue no longer has and that the runner never saw finish (the process died mid-item) counts as canceled. +- A retry replaces the item it retried. The queue's own record of the retry links them while it has the retry, also before the runner started it; afterwards, what the runner saw does. A retry the runner never started and the queue no longer has leaves no trace: the run then reads as the outcome of the item it retried, failed for instance. +- The run is `running` while any item is pending or running, `failed` when any failed, and `completed` when all completed. Otherwise it is `partial` if valid points were recorded, or `canceled` if not: an invalid point measured nothing usable. + +Routes, besides the 403 for non-admins and a 503 when the store cannot be read or written (an unwritable root, a lock timeout): + +| Route | Does | Other errors | +| --- | --- | --- | +| `POST .../i/{key}/working_memory/calibrate` | starts a run (above) | 404 unknown model, the run was removed while it was being created, or a reset removed the run the retry key started; 409 no GPU or several generation devices, unusable store file, no room in the queue for all of the run's items (nothing is enqueued; the run is removed if this request recorded it), or a retry key used for another request; the 404 and 409 for the run's store, and with a key the 503, can come after the items were queued (above); 422 a `call_saved_workflow` node, batch data setting `use_cache`, an invalid batch, or fields the server sets; 429 too many unacknowledged retry keys | +| `GET .../working_memory/status` | every device class with its runs, point counts per estimator, version, validity and fit eligibility, and environments with hard and soft facts compared to the current environment (`soft_matches_current` compares the soft facts first recorded with the environment, not a later run's); `available` says whether a run can start now (one GPU generation device and a usable store file); per device class with a generation device here, `measurement_method` (the driver reader its windows use, `counters.driver_method`) and `measurement_verified` (`counters.VERIFIED_METHODS`), null for other classes | – | +| `DELETE .../working_memory/runs/{run_id}` | removes a run and its points; its items that have not started yet fail at admission, also after a repeat of the request that started it | 404 no usable file holds the run | +| `DELETE .../i/{key}/working_memory` | removes the points the summary attributes to the model (`measured_models`): its runs with all their points, and every point of the model or one of its submodels (`:vae`) taken in other runs (a VAE measured inside a transformer's run, say); other models' points on the same lines stay | 409 no current device class, or an unusable file | +| `DELETE .../working_memory` | removes every run, point and environment of a device class | 409 as above | +| `GET .../working_memory/export` | the file byte for byte: a `CalibrationStoreFile` when this version can read it, otherwise whatever the file holds | 404 no file; 409 no current device class | +| `GET .../i/{key}/working_memory/plan` | the cases that calibrate the model here (`WorkingMemoryCalibrationPlan`, below), or `supported: false` with the reason | 404 unknown model | +| `GET .../i/{key}/working_memory/summary` | per line and environment: the lines the model's windows measured, and in the current environment each line the model's decode prices on (its plan's estimator and dtypes) that other models' windows measured. Per line, every window that feeds it (all models' counted windows of that environment that join the same user line): sizes, largest demand, shipped and user estimate; a size whose valid windows all are excluded is listed too, with `counted: false` and `excluded_reason` (`fragmented` or `partial_residency`), its largest such window and the user estimate there; the user line's measured range and state (`in_use`, `stale`, `unusable`, `insufficient`); `own_windows`, the model's share (0 for a line only others measured); `measured_by`, each contributing model with its name (null once uninstalled) and window count. Also the model's windows that fit no line, and its runs | 404 unknown model; 409 no current device class | + +The reset, export and summary routes act on the current device class unless `device_class_id` is given. + +#### User fits + +`backend/working_memory/user_fit.py` owns the rule; the calibration service owns the store and the cache. + +- Only the estimators that opt in (`EstimatorSpec.user_calibrated`, the five VAE estimators) take part. The windows of any other, a Z-Image denoise's say, stay in the store and count among the windows that fit no line, but no user line is fitted on them, none moves their estimates, the summary lists none of their lines, and they size no plan: denoise lines are shipped-fit only. +- A user line is fitted per line of `fit.py`: the shipped entry that carries the values (one hop of `values_from`) and `_`. A window joins where its estimator's `line` admits it (for the convolution lines: untiled, one sample, no in-place upcast, an AutoencoderKL: the SD estimator's AutoencoderTiny windows join no line), it is valid and fit-eligible, it is not fragmented, its demand is positive, and its estimator version has shipped values on the platform. Every user point is a `probe_demand` window, so all of them rate the hull's edges. `fit.user_pair` adds `USER_MARGIN`, the shipped 5%, to the envelope and rounds up as the shipped values do. A line needs two sizes `MIN_AREA_RATIO` apart. +- Fragmented windows measured the allocator's history, not the operation. One rule, `fragmentation.fragmented`, judges them for the user fit and for `scripts/working_memory/build_shipped.py`, which flags shipped points by it. It judges a measurement: the fit-eligible windows of one attempt (`attempt_id`, else the run; for the builder also one server lifetime) at one environment, estimator, version and features, in the order they ran. A measurement drops out whole when reserved grew by 10% from its first window to its largest while allocated stayed within 2%, or when its largest window's reserved-to-allocated ratio is 1.2 times the median of the same line's smaller measurements that count, from any attempt. Each attempt is judged on its own, because its windows ran back to back on one allocator state: a recalibration can bring back a size an earlier attempt measured fragmented, and a later fragmented attempt drops no clean one. The ratio rule catches a size whose windows all ran fragmented, or that kept only one fit-eligible window: on an RX 9060 XT the fp32 SDXL decode at 1216 px measured 1.47 against 1.10 at 576 and 832 px, with one fully resident window beside two partly resident ones. Windows of a running attempt are judged but set no reference, as they join no user line yet. Partly resident windows (`partial_residency`) never count and enter neither rule. A size counts when any attempt's windows at it count, and its `windows` and `measured_bytes` cover those windows only. +- The windows are those of the current device class's file taken under an environment whose hard facts equal the current ones (same `env_id`). A soft-fact difference keeps the fit. The summary reports a line as `stale`, with the facts that changed, when the soft facts of the newest finished run that fed it differ from the current ones; a run stored without soft facts stands for the environment's, those first recorded with it. So recalibrating after a driver update clears `stale`. Older windows of the same hard environment keep feeding the line: the hard facts are what decide whether a window still measures this machine. A hard mismatch leaves the shipped values. +- `EstimatorSpec.estimate` asks the registered provider (`user_fit.set_user_fit_provider`; the service registers it in `start()` and clears it in `stop()`) for the user fit of the spec, platform and device. Without one, the estimate is the shipped one, bit for bit. With one, `user_fit.resolve` reads the operation's line and its size there (output pixels) as `EstimatorSpec.line` places it. At or below the largest measured size the user line replaces the shipped one; above it, the larger of the two estimates wins. A tiled operation is placed by its image, priced on its tile. The `source` starts with `user::` and names the line where the user line decides. A line the user did not measure stays shipped. Because estimates feed `reservation_bytes`, a user line also moves `should_pretile_vae_decode`. +- Only finished runs and attempts feed a fit. The windows of a run whose derived status is `running` (an item pending or running; a run records its items as soon as the calibrate call has queued them, so a run whose items all wait counts, and an item it could not record counts once the runner admitted it) join no user line, whoever refreshes the fits meanwhile. Nor do the windows of any run that shares its `attempt_id`: the client posts each step of an attempt as its own run, so an attempt's windows feed no fit until every run of it recorded so far has finished. A step not yet posted holds back nothing, so an ordinary item or a plan in that gap can see the finished steps' windows; a calibration item refreshes only once its own run exists. They join at the first refresh after that, whether the runs completed, failed, were canceled or ended `partial`. So an attempt's later, larger cases are priced without its own earlier windows. A run without an `attempt_id` is judged on its own; another attempt that finished counts as usual. The summary shows a running run's windows in the sizes, window counts and `measured_by`, while its user line, range, user estimates and state are those of the fit estimates read; a line only a running run measured is `insufficient` and says so. +- The provider answers from memory. `refresh_user_fits()` rebuilds that memory per GPU generation device. `DefaultSessionRunner.run()` calls it once when it starts any queue item, calibration items included; `get_plan` calls it first, so the case sizes price with the store as it is, and every reset calls it after its write, so a reset applies at once. It stats each device class's file and re-reads it only when the mtime, size or inode or the environment changed. To tell which runs are running it asks the queue, by id, only about the items the file does not record as finished; items found finished are remembered, since a finished item never runs again. When every run finished, a refresh reads neither the file nor the queue. A retry's item is not in the file until admission records it, and the runner refreshes before admitting, so a retry of an attempt's last open run is priced with that attempt's finished windows. Nothing on the estimate path reads a file. Several generation devices each use their own class's fit; calibrating them is not supported. +- Shared lines: on cuDNN the SD, SD3, FLUX.1 and FLUX.2 estimators borrow one entry, so an SDXL fp16 decode calibrates every estimator's fp16 decode there. The plan lists the installed models this affects. + +The plan (`working_memory_calibration_plan.py`) covers the decode of SD1, SD2, SDXL, SD3, FLUX.1, FLUX.2, Z-Image and CogView4 main models, and standalone VAEs of those bases through an installed main model that accepts them (`VaeFacet`): through Generate's `vae` slot where the base has one (FLUX.1, FLUX.2, SD3, Z-Image), otherwise with the main model's own VAE, which prices on the same lines. Each case is a two-step `txt2img` graph-export case per size and decode dtype; the SD bases set `vaePrecision` to fp16 and to fp32, the others run the dtype their loader picks (SD3 the `precision` setting). The sizes are about 0.25, 0.5 and 1.0 of the largest square, in steps of 64 and at most 4096 px, whose calibration reservation (the estimate plus `max(1 GiB, 0.5 × estimate)`, as the lock applies it) leaves the VAE fully resident (`decode_budget`): it must fit both 90% of the card (`VAE_PRETILE_VRAM_FRACTION`) and what the model cache can give on the device, each less 1 GiB for the VAE's weights. Nothing else stays beside it: inside a calibration scope, `model_on_device` first moves every other unlocked model on the device to RAM whole (`ModelCache.offload_unlocked_models_from_vram`, under the global load lock, inside the probe's lock interval; locked models, a window's co-locks among them, stay), and only then locks. The reason: a lock offloads only the shortfall it predicts and splits the last model it reaches, and a split can free a little less than predicted. On an RX 9060 XT, 9 of the 12 calibration decodes that had to split the SDXL UNet came up 35 to 55 MB short and ran the VAE partly resident, so their windows fit no line. Outside a calibration scope, locks are unchanged. That capacity is read when the plan is built, as `ModelCache._get_vram_available` counts a lock: the driver's free memory plus what torch has allocated, within `max_cache_vram_gb`, less `reserve_vram_gb`. Memory other processes hold is thereby left out. What this process holds outside torch's allocator counts at the larger of what it holds now and the most a valid stored window of the current environment recorded at its baseline (the driver's figure for the process less torch's reserve, `recorded_outside_torch_bytes`), over the windows of the estimators a user calibration moves only, because it grows with the first generations: on a 16 GB RX 9060 XT under ROCm, HIP's retained pool and MIOpen's workspaces took about 0.3 GB in a fresh server and 2.6 GiB once it had decoded, and sized for the fresh reading its largest cases ran partly resident. A first calibration has no stored window and sizes for the reading alone, so on such a card its largest cases may run partly resident; those windows do not count, and the next plan sizes for what the first one recorded. The recorded figure comes from the store `refresh_user_fits` just read; `estimate_untiled_vae_decode` prices them with the decode nodes' own features. `DECODE_LINES` restates which estimator each base's decode node prices with; `test_working_memory_calibration_plan.py` holds it to the nodes in `invocations/vae/`. Each case runs once to warm up and twice more; `estimated_seconds` is a rough order of magnitude, not a measurement. Encodes are not planned: text-to-image runs none. ### Emitting model loading events diff --git a/docs/src/content/docs/development/Architecture/overview.mdx b/docs/src/content/docs/development/Architecture/overview.mdx index 8cbe18efad5..3d6272db7ff 100644 --- a/docs/src/content/docs/development/Architecture/overview.mdx +++ b/docs/src/content/docs/development/Architecture/overview.mdx @@ -85,6 +85,8 @@ The invoker (`/invokeai/app/services/invoker.py`) is the primary interface throu - **invocation services**, which are used by invocations to interact with core functionality. - **invoker services**, which are used by the invoker to manage sessions and manage the invocation queue. +Creating the invoker starts every service, the session processor last: its workers dequeue at once, and an item left pending before the restart may use any other service. (The queue's start cancels the items that were in progress.) `stop()` stops the session processor first: its workers stop taking items and cancel their sessions, and the other services stop only once the workers have exited. The processor waits up to 30 seconds for all its workers together, and after a second still waiting it logs which worker and queue item it waits for. A worker still in a node at that deadline (one without a cancel point) is given up on with an error in the log, so that shutdown finishes; a later `stop()` does not wait for it again. It keeps its GPU, so until it exits no session processor in the same process starts and none replaces the generation device pool: `ApiDependencies.initialize` refuses a restart before it opens the database. + ### Sessions Invocations and links between them form a graph, which is maintained in a session. Sessions can be queued for invocation, which will execute their graph (either the next ready invocation, or all invocations). Sessions also maintain execution history for the graph (including storage of any outputs). An invocation may be added to a session at any time, and there is capability to add and entire graph at once, as well as to automatically link new invocations to previous invocations. Invocations can not be deleted or modified once added. diff --git a/docs/src/content/docs/development/Guides/family-e2e.mdx b/docs/src/content/docs/development/Guides/family-e2e.mdx new file mode 100644 index 00000000000..f6a8ca56de3 --- /dev/null +++ b/docs/src/content/docs/development/Guides/family-e2e.mdx @@ -0,0 +1,151 @@ +--- +title: Family E2E Runs +lastUpdated: 2026-10-17 +--- + +`scripts/family_e2e` runs model families end to end through the HTTP API of a server it launches itself. Per case and size it records the image, the run time, the time per denoise step and the VRAM the server process used, measured at driver level. The report compares them with an explicitly accepted reference and with the previous run of the same profile. + +It is a developer tool. Memory sampling uses WDDM, so it runs on Windows only for now, with NVIDIA and AMD (ROCm) GPUs. + +## How a run works + +1. **Scratch root.** The template root's `invokeai.yaml` and a snapshot of its SQLite database go to `/scratch-roots/-default/`. The template's database files (with a WAL left by an unclean stop) are copied, and the copy is backed up through InvokeAI's database layer, so the WAL's contents are included and the template's files are only read. + - The tool refuses while the database file is open in any process, or while anything listens on the template's configured port. Either means its server is still running. + - It also refuses while a model install is unfinished (see [Shared paths](#shared-paths)). + - Stores whose files the database owns start empty in the scratch root: outputs, uploaded fonts, style preset images, workflow thumbnails and profiles. The scratch server reconciles them with its own database; at the template it would, for example, delete uploaded fonts its database does not list. + - Every other path setting points back at the template root as an absolute path, so models, custom nodes and memory settings are the template's own. + - The tool overrides a few settings: a free port on `127.0.0.1`, a plain-format log file in the run directory, the image index off, and the queue cleared at startup. SSL and `base_url` are dropped. + - `INVOKEAI_*` variables of the calling environment are not passed on. Put settings the server needs into the manifest's `server.env`. +2. **Server.** The tool checks that `server.python` imports `invokeai` from `server.code_dir`. It then starts ` /scripts/invokeai-web.py --root ` and reads the actual port from the server log. Multiuser mode is refused. + - On Windows the server runs in a job object of the runner, so it dies with the runner however the runner ends. +3. **Graphs.** The real webv2 graph builders produce each case's batch through `pnpm graphs:export`, run in the tool checkout. Run `pnpm install` in `invokeai/frontend/webv2` there first. + - The runner disables the invocation cache on every node and saves the batch as `graphs/.json`. + - It hashes the graph and its batch data. Uploaded image names are replaced by the input file's hash, so the graph hash changes only when the graph or an input changes. + - Init and control images enter the canvas compiler as finished layers. One whose size differs from the case size is resized (LANCZOS) into `runs//inputs/` before upload. Reference images are uploaded as they are. +4. **Steps.** Per family the model cache is emptied first. Each case size then gets its warm-ups, then its measured runs. + - Per run, the tool waits up to 10 minutes for the queue to drain, samples the server's GPU memory, enqueues the batch and follows its socket events. + - When the run finishes, it saves the output image and the memory trace, reads the run's slice of the server log and appends a record to `records.jsonl`. + - If the server dies during a step, that step gets an error record and a new server starts on a new scratch root. A case whose steps crashed the server twice is given up; its remaining steps are not run again, not even by `resume`. +5. **Report.** The server is stopped, the scratch root removed, and `report.html` and `summary.md` are written. The exit code is not 0 when a case fails or errors. + +## Data directory + +``` +/inputs/ input images the manifest refers to +/references/// image.png, reference.json (written by `accept` only) +/runs// run.json, records.jsonl, heartbeat.json, runner.log, server-default.log, + graphs/, images/, traces/.csv.gz, report.html, summary.md, + calibration/ (calibration runs only) +/scratch-roots/-default/ +``` + +`run_id` is `YYYYMMDD-HHMMSS-`, and `case_key` is `@x`. A trace holds the raw samples as `series,t,bytes`: series `dedicated` or `shared`, and `t` in seconds from the enqueue. + +One runner per GPU: a lock file per adapter lives in `%LOCALAPPDATA%\invokeai-family-e2e\locks\`, shared by every data directory on the machine. + +## Manifest + +A YAML file per machine and GPU, kept with the data rather than in the repository. Start from `scripts/family_e2e/manifest.example.yaml`. Relative paths are relative to the manifest file. + +| Field | Meaning | +| --- | --- | +| `profile` | Names runs and references. | +| `data_dir` | The data directory above. | +| `gpu` | Substring of the adapter name that matches exactly one adapter, e.g. `RTX 4090` or `9060`. `Radeon` alone also matches an AMD integrated GPU. | +| `server.template_root` | The root whose settings and database are copied. The scratch server shares its other paths; see [Shared paths](#shared-paths). | +| `server.python` | The interpreter that runs the server. | +| `server.code_dir` | The checkout the server runs. Defaults to the tool's checkout. | +| `server.env` | Extra server environment, e.g. `HIP_VISIBLE_DEVICES`. | +| `defaults` | `warmup` (at least 1), `repeats`, `seed`, `prompt`, `negative_prompt`, `psnr_min_db`, `time_tolerance` (relative), `peak_tolerance_mb`, `run_timeout_s`. | +| `families[].cases[]` | `id`, `model` (key), `mode` (`txt2img`, `img2img` or `controlnet`), `sizes` (a list of `[width, height]`), `steps`, optional `seed` and `prompt`, `components` (component slot to model key), `loras`, `reference_images`, `init_image` (`path`, `strength`), `control` (`model`, `image`, `weight`, `begin`, `end`), `settings` (generate settings), and per-case thresholds. | + +Input paths are relative to `/inputs` and must exist. `init_image` belongs to `img2img` cases only and `control` to `controlnet` cases only. The exporter rejects an unknown `settings` key, component slot or model key; that case then gets error records and the run goes on. Several `sizes` per case give a size sweep for working-memory calibration. + +## Commands + +Run the tool from the repository root with the checkout's environment, for example `uv run --no-sync python -m scripts.family_e2e …`. + +| Command | What it does | +| --- | --- | +| `start --manifest M [--family F]… [--case ID]… [--calibrate]` | Checks the GPU lock, the template's server, unfinished model installs and foreign GPU load. It then creates the run and starts its runner in the background, and prints the run id and paths. `--calibrate` makes it a [calibration run](#working-memory-calibration). | +| `status [RUN]` | Shows progress, the current step, the heartbeat age and whether the runner is alive, and for a calibration run the number of saved store exports. | +| `stop [RUN]` | Kills the runner and its server, saves a calibration run's store from its scratch roots, removes them and marks the run stopped. | +| `resume [RUN]` | Starts the runner of a stopped or crashed run again. | +| `report [RUN]` | Writes `report.html` and `summary.md` again. | +| `accept [RUN] [--case KEY]…` | Makes the run's measured results the references of its profile. Nothing else writes references. | +| `run --run-dir D` | The runner in the foreground: what `start` and `resume` launch. | + +`RUN` is a run directory, or a run id together with `--manifest M`. Without it, the profile's latest run is used. + +On Windows the background runner breaks away from the caller's job object, so a terminal or an agent harness that kills its job's processes does not end a long run. Where breakaway is refused, the runner is created through WMI. It then starts with the user's default environment instead of the caller's. + +A resumed run skips every step that already has a final record and keeps the measures of the earlier sessions. Its new server starts with a cold model cache, so each case size with measures left first repeats its last warm-up, once. The manifest, the graphs, and the server's commit and dirty flag must not have changed since the run started; further edits to an already dirty checkout cannot be detected. + +## Working-memory calibration + +`start --calibrate` measures what each model operation of the cases needs on this GPU. Like the rest of the tool it runs on Windows only. Every step compiles its graph as a normal run does, then submits it to `POST /api/v2/models/i/{model}/working_memory/calibrate` with the case's `model` key instead of `enqueue_batch`. The server runs it with its working-memory probe, which records a window per estimated model operation in its calibration store (see "Calibration runs" in the [model manager architecture](/development/architecture/model-manager/#calibration-runs)). Warm-ups and measures alike are submitted, and every window is kept. + +- **Before the steps.** The runner reads `GET /api/v2/models/working_memory/status`. With `available: false` (no GPU, several generation devices, an unusable store file) the run fails with the server's reason. The scratch server runs single-user, so the tool is an administrator. +- **Refusals.** A 404 or 422 answer to a calibrate request (an unknown model, a graph the run cannot measure) gives that step an error record, and the run goes on. Any other error response ends the run with the server's detail, after its store was saved; fix the cause and `resume`. +- **Records.** Each accepted step's record carries `calibration`: the calibration run id and the device class. The probe synchronizes and empties the allocator cache around every model operation, so time and VRAM are invalid with that reason; `status` shows it as a calibration marker rather than counting it, and images are still compared. A step foreign load spoiled is not repeated: the server keeps every window. `accept` refuses calibration runs, and reports of later runs do not compare against them. +- **Exports.** Every server lifetime has its own scratch database and store, so the tool submits each attempt once and sends no retry key. When a lifetime's steps are done, the runner saves `calibration/--.json` (`GET /api/v2/models/working_memory/export`, byte for byte) and `calibration/--status.json`; `n` counts the run's lifetimes. After a server crash, a runner failure or `stop`, the store file is copied from the scratch root instead, without a status. `run.json` lists them all under `calibration_exports`. + +On a machine the tool does not run on, an administrator takes the export of a server's own calibration runs from `GET /api/v2/models/working_memory/export?device_class_id=`; `GET /api/v2/models/working_memory/status` lists the device class ids. + +### Building shipped data + +``` +uv run --no-sync python -m scripts.working_memory.build_shipped /runs//calibration [more exports or directories] [--points FILE ...] [--write] +``` + +`scripts/working_memory/build_shipped.py` reads the exports, and the point files `--points` names, and prints what it would change in `invokeai/backend/working_memory/data/`; `--write` writes it. A directory stands for the exports in it, in lifetime order. Then run `uv run --no-sync pytest tests/backend/working_memory`. + +- **Point files.** A calibration script such as `scripts/calibrate_z_image_working_memory.py` writes one JSON object per line: a `MeasuredPoint`'s fields (`estimator`, `version`, `platform`, `features`, `measured_bytes`, `method`, `device`, `provenance`) and whatever else it keeps, such as raw readings, which the builder ignores. These are fresh-process measurements, so they cap a line as well as raise it; a `probe_demand` record is refused. A check's record (a forced math kernel, a ControlNet, more blocks) says `in_fit: false`: it tests an analytic term or an assumption rather than the line, so it ships as an out-of-fit point that bounds its estimate from below only, and the data test's `OUT_OF_FIT` has to list it. It stays beside a fit point with the same features rather than replacing it, since being in the fit is part of what makes two points one measurement. A record's platform may name the OS anywhere and keeps it only where an entry names it. A measurement given twice keeps its larger reading. +- **Points.** Every valid, fit-eligible window with a positive demand becomes a `probe_demand` point. Per card, estimator, version, platform and features it takes the largest `demand_bytes` of all windows given: repetitions, lifetimes, and several runs or machines with the same card. A window given twice (the same store in two files) counts once. The provenance names the card, the torch and runtime versions, the calibration run, item and node of the largest window, and its export file. Each point line shows its windows' demands in the order they ran and the largest over the first (`[3 windows, MiB 1960, 1820, 1820; max/first 1.00]`). +- **Fragmentation.** A point whose reserved bytes grew by 10% or more from its first window to its largest while the allocated bytes stayed within 2% gets a `warning: fragmentation?` line with both series. That is the allocator's cache growing around the same tensors in a long-running server, not the operation needing more. +- **Platforms.** A point's platform is the one `platform_for` gives on the machine that measured it, derived from the export's device class and its environment's hard facts (`platform_for_environment`), whatever machine the builder runs on. It names the OS only where an entry for that platform names it. A window whose environment lacks a fact the platform is read from is left out and counted. +- **Bounds.** `probe_demand` points bound their estimates from below only: the estimate must cover them, and they may raise a line, but they take no part in choosing it (`fit.upper_envelope`), and the 1.3x upper bound and `max_ratio` stay with the fresh-process points, because the cold first run after a server start is the worst case. +- **Merge and fit.** A point joins the data file that declares the estimator's coefficients; where a point of the same measurement (card, platform, method, features and whether it is in the fit) is there, the larger of the two stays, and a remeasurement below the shipped point prints `kept shipped over remeasured `. The carried values are then refitted with `fit.py`, and a stand-in line that an in-fit point now measures is fitted from then on. A refitted entry's provenance gets a note. Every moved line gets a `line` row: old and new slope and base, whether it was raised, and its new points. +- **Legacy entries.** A `legacy` entry (hand-set values, such as the Z-Image denoise's) is replaced by its fit once the points fit every line it carries and each line has a fresh-process point, which holds the upper bound: its flag cleared, its provenance replaced by one that says it is the fit of the platform's points, its `unmeasured_floor` kept. Probe windows alone never replace hand-set values, since a warm window can read below the cold first run. The data test's `LEGACY_ENTRIES`, which lists the legacy entries by estimator and platform, then needs that entry removed. An entry the points do not fully measure keeps its values, with a `note:` line saying which lines lack a fit or a fresh-process point; its new points still have to pass their bounds against the hand-set values, which a fresh-process point well below them does not. +- **Refusals.** An export or point of another estimator version than the registered one, features the estimator cannot evaluate, a platform `MEASURED` does not list, and a line whose points span less than twice its sizes: exit code 2. Exit code 1, with nothing written: + - any point of the result breaks its bounds (`working_memory/bounds.py`, the rule the data test applies); the error lists them. A probe point that raises a line above what the fresh-process points of that line allow is refused this way; + - with `--write`, a line is raised on a point flagged as fragmentation, unless `--accept-fragmentation` is given. + +## Validity + +A run's record can be complete and still unfit for some comparisons. Each invalid flag comes with its reason in `invalid_reasons`. + +- **Residency.** The model cache logs `Loaded model '…' … VRAM: … (P%)` whenever a lock moves weights or leaves a model short of fully resident. A warm run usually logs nothing, because its models stay resident. The tool therefore keeps the latest line per model since the last cache flush. Time and memory are invalid when: + - a line in the run's log slice is below 100 %, + - a `VRAM budget … is short` warning appears, or + - no full load of the graph's models was seen since the flush ("residency unknown"). +- **fp8.** With fp8 weights in play, an incompletely resident run does not reproduce its image, so PSNR is invalid too. fp8 is in play when a graph model sets `fp8_storage` in its default settings, or when the server's `fp8_compute` is on. Model configs do not record which checkpoints are scaled fp8, so then every graph counts. +- **Shared memory.** If the server's shared GPU memory grows by more than 256 MiB during a run, it paged to system memory: time and memory are invalid. +- **Foreign load.** Another process above 1 GiB of dedicated GPU memory at either end of the run, or grown by more than 256 MiB, invalidates time and memory. So does a queue that is still busy after the 10-minute wait. The run is repeated, up to three attempts. `start` refuses outright when another process holds more than 1 GiB. + +## Shared paths + +The scratch server uses the template root's models directory, model caches, legacy model configs, custom fonts and custom nodes. Its startup touches them as starting the template's own server would: + +- `models/README.txt` is written. +- Install claims (`*.invokeai_install_active`) of processes that no longer run are removed. +- Missing model files are reported in the log. +- `configs/` receives the legacy model configs of the server's code when they differ. +- `fonts/README.txt` is written when missing. +- Models a node fetches on first use are cached in `models/.download_cache`. + +The server would also finish an incomplete install, but the model would then be registered in the scratch database only. So `start` and `resume` refuse while the models directory holds a `tmpinstall_*` entry: finish or cancel that install in the dev server first. + +Database migrations of a newer `code_dir` run against the scratch database only. A migration that also moves or rewrites files acts on the shared models directory and caches; keep that in mind before pointing a run at a checkout ahead of the template's own server. + +## Limits + +- Memory is sampled every 3 ms with `D3DKMTQueryVideoMemoryInfo`: shorter peaks are missed. It counts resident memory (`CurrentUsage`), not commit, including the CUDA or HIP context and every resident model. That is why records keep a baseline and a peak per node. +- With PyTorch's native allocator, reserved memory stays allocated between nodes, so a node's peak can stem from an earlier node. Compare peaks only between runs with the same allocator configuration; the run records the runtime config. +- On Windows with ROCm, HIP keeps freed blocks at driver level. Per-node peaks after the first large node are upper bounds there (observed on an RX 9060 XT). +- Time per step comes from the preview frames, which the server throttles to one per 0.1 s; a step's number is read from its progress percentage. + - For txt2img and controlnet the case's `steps` fixes the step count. + - img2img starts inside the schedule at a family-specific step, so the count is inferred. When steps are faster than 0.2 s that inference can be off by a factor, so the time per step is left empty. +- Ideogram 4 ignores `steps`; its step count follows `settings.ideogram4Steps` or `settings.ideogram4SamplerPreset`. `img2img` and `controlnet` are not available for ERNIE-Image and Ideogram 4. Dynamic prompts, canvas rasterization, video and upscale graphs are not covered. +- Linux memory sampling and A/B runs of two checkouts are not available yet. +- A calibration run measures the windows of estimated model operations only, inside a running server: they are not a fresh process's first operation, as the `demand_peak` points are. diff --git a/invokeai/app/api/dependencies.py b/invokeai/app/api/dependencies.py index f11b206427e..ee776633b79 100644 --- a/invokeai/app/api/dependencies.py +++ b/invokeai/app/api/dependencies.py @@ -55,6 +55,7 @@ from invokeai.app.services.session_processor.session_processor_default import ( DefaultSessionProcessor, DefaultSessionRunner, + refuse_while_an_abandoned_worker_runs, ) from invokeai.app.services.session_queue.session_queue_default import SessionQueue from invokeai.app.services.shared.database.database import Database @@ -70,6 +71,10 @@ from invokeai.app.services.wildcard_records.wildcard_records_default import WildcardRecordsStorage from invokeai.app.services.workflow_records.workflow_records_default import WorkflowRecordsStorage from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_disk import WorkflowThumbnailFileStorageDisk +from invokeai.app.services.working_memory_calibration.working_memory_calibration_default import ( + WorkingMemoryCalibrationService, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_store import CalibrationStore from invokeai.backend.architectures import conditioning_safe_globals from invokeai.backend.architectures import validate as validate_architectures from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData @@ -117,6 +122,10 @@ def initialize( # lazily inside this function for the same reason, one step earlier: it is what fills the # registry, and importing this module is all `scripts/generate_openapi_schema.py` does. validate_architectures() + # A worker the previous invoker's stop gave up on still holds its GPU, and the session processor would refuse + # to start beside it. Refused here, before the database opens (and migrates), so a refused restart leaves + # nothing open behind it. + refuse_while_an_abandoned_worker_runs() logger.info(f"InvokeAI version {__version__}") logger.info(f"Root directory = {str(config.root_path)}") @@ -216,6 +225,10 @@ def initialize( image_index_records = ImageIndexRecords(database) image_index = ImageIndexService() intermediates = IntermediatesService(records=IntermediatesRecords(database), logger=logger) + # Machine data next to the database rather than in it: see working_memory_calibration_store. + working_memory_calibration = WorkingMemoryCalibrationService( + CalibrationStore(configuration.db_path.parent / "working_memory") + ) fonts = FontService( database, fonts_dir=configuration.fonts_path, @@ -273,8 +286,13 @@ def initialize( fonts=fonts, intermediates=intermediates, gallery_maintenance=gallery_maintenance, + working_memory_calibration=working_memory_calibration, ) + # Before the Invoker starts any service: work outside the session queue, such as the image index's backfill, + # borrows idle GPUs from the generation device pool as soon as its service starts. + session_processor.register_generation_devices(configuration) + # Constructing the Invoker starts every service, including the session # processor thread (which may immediately resume queue items that were # pending at shutdown). Trigger torch attention's lazy, non-thread-safe diff --git a/invokeai/app/api/routers/session_queue.py b/invokeai/app/api/routers/session_queue.py index 2d3c0221abc..25a25c4d8ef 100644 --- a/invokeai/app/api/routers/session_queue.py +++ b/invokeai/app/api/routers/session_queue.py @@ -41,6 +41,10 @@ from invokeai.app.services.shared.graph import Graph, GraphExecutionState from invokeai.app.services.shared.pagination import SQLiteDirection from invokeai.app.services.video_records.video_records_common import VideoRecordNotFoundException +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CALIBRATION_ORIGIN_PREFIX, + is_reserved_origin, +) session_queue_router = APIRouter(prefix="/v1/queue", tags=["queue"]) @@ -256,6 +260,10 @@ def get_queue_item_for_mutation(queue_id: str, item_id: int, current_user: Curre operation_id="enqueue_batch", responses={ 201: {"model": EnqueueBatchResult}, + 400: { + "description": f"The batch's origin starts with '{CALIBRATION_ORIGIN_PREFIX}' (in any case), which is " + "reserved for working-memory calibration runs" + }, }, ) async def enqueue_batch( @@ -265,6 +273,14 @@ async def enqueue_batch( prepend: bool = Body(default=False, description="Whether or not to prepend this batch in the queue"), ) -> EnqueueBatchResult: """Processes a batch and enqueues the output graphs for execution for the current user.""" + # Calibration items are measured and admitted as such by the session runner; only the calibration route may + # create them, for administrators. + if is_reserved_origin(batch.origin): + raise HTTPException( + status_code=400, + detail=f"Origins starting with '{CALIBRATION_ORIGIN_PREFIX}' are reserved for working-memory calibration " + "runs, which start through POST /api/v2/models/i/{key}/working_memory/calibrate.", + ) await asyncio.to_thread(assert_image_move_maintenance_inactive) try: diff --git a/invokeai/app/api/routers/working_memory.py b/invokeai/app/api/routers/working_memory.py new file mode 100644 index 00000000000..7fd6a1ffa04 --- /dev/null +++ b/invokeai/app/api/routers/working_memory.py @@ -0,0 +1,286 @@ +"""Working-memory calibration runs: administrators measure what a model's operations really take on this machine's GPU. + +Mounted under the model manager's prefix: a run calibrates one model (`/i/{key}/working_memory/...`), and its +measurements belong to the machine's device class (`/working_memory/...`). Every route is admin-only: runs spend GPU +time on the instance's behalf, and the store is machine data that names the accounts that created runs. +""" + +import asyncio +from typing import Final, Optional + +from fastapi import Body, HTTPException, Path, Query, Response +from fastapi.routing import APIRouter + +from invokeai.app.api.auth_dependencies import AdminUserOrDefault +from invokeai.app.api.dependencies import ApiDependencies +from invokeai.app.api.routers.image_move_maintenance import assert_image_move_maintenance_inactive +from invokeai.app.services.model_records import UnknownModelException +from invokeai.app.services.session_queue.session_queue_common import ( + EnqueueIdempotencyConflictError, + EnqueueReceiptLimitError, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_base import ( + WorkingMemoryCalibrationServiceBase, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CalibrationGraphError, + CalibrationQueueFullError, + CalibrationRunNotFoundError, + CalibrationStoreError, + CalibrationStoreUnusableError, + CalibrationUnavailableError, + WorkingMemoryCalibrationModelSummary, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationRequest, + WorkingMemoryCalibrationResetResult, + WorkingMemoryCalibrationRunCreated, + WorkingMemoryCalibrationStatus, +) + +working_memory_router = APIRouter(prefix="/v2/models", tags=["model_manager"]) + +_DEVICE_CLASS_ID = Query( + default=None, + pattern=r"^[0-9a-f]{16}$", + description="The device class to act on; the current generation device's by default", +) + + +# One description per error, shared by the routes that return it. +_STORE_UNREADABLE: Final = { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" +} +_NO_DEVICE_CLASS: Final = { + "description": "No device_class_id was given and there is no current one (no GPU, or several generation " + "devices), or the device class's store file is unusable and kept as it is" +} + + +def _service() -> WorkingMemoryCalibrationServiceBase: + service = ApiDependencies.invoker.services.working_memory_calibration + if service is None: + raise HTTPException(status_code=503, detail="Working-memory calibration is not available") + return service + + +def _http_error(error: Exception) -> HTTPException: + if isinstance(error, (CalibrationRunNotFoundError, UnknownModelException)): + return HTTPException(status_code=404, detail=str(error)) + if isinstance(error, CalibrationGraphError): + return HTTPException(status_code=422, detail=str(error)) + if isinstance( + error, + ( + CalibrationUnavailableError, + CalibrationStoreUnusableError, + CalibrationQueueFullError, + EnqueueIdempotencyConflictError, + ), + ): + return HTTPException(status_code=409, detail=str(error)) + if isinstance(error, EnqueueReceiptLimitError): + return HTTPException(status_code=429, detail=str(error)) + # CalibrationStoreError: the root cannot be read or written (read-only, lock timeout). Also the 503 of `_service`. + return HTTPException(status_code=503, detail=str(error)) + + +@working_memory_router.post( + "/i/{key}/working_memory/calibrate", + operation_id="calibrate_model_working_memory", + status_code=201, + response_model=WorkingMemoryCalibrationRunCreated, + responses={ + 404: { + "description": "The model could not be found; the run was removed while it was being created, whose " + "items are queued then and fail when they start; or a reset removed the run the idempotency key started" + }, + 409: { + "description": "This machine cannot calibrate now (no GPU, or several generation devices), the device " + "class's store file is unusable and kept as it is (possibly after the items were queued, which then fail " + "when they start), the queue has no room for all of the run's items (it then takes none), or the " + "idempotency key was used for another request" + }, + 422: {"description": "The graph cannot be measured as a calibration run"}, + 429: {"description": "Too many unacknowledged idempotency keys; acknowledge or wait, then retry"}, + 503: { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout); with an " + "idempotency key, possibly after the items were queued" + }, + }, +) +async def calibrate_model_working_memory( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model the run calibrates"), + request: WorkingMemoryCalibrationRequest = Body(description="The batch to measure"), +) -> WorkingMemoryCalibrationRunCreated: + """Start a working-memory calibration run: enqueue the batch with every node uncached and measure each node's + model operations on the generation device. Needs exactly one generation device, and it must be a GPU. The queue + takes all of the batch's items or none. + + Once the items are queued, a request without an `idempotency_key` succeeds even when the store fails for a passing + reason (an unwritable root, a lock timeout) to record them: they run all the same, and the status finds them by + their origin. With a key, that failure is a 503 although the items are queued. A run removed meanwhile (404) or an + unusable store file (409) fails the request with or without a key; its queued items then fail when they start. + + With a key, repeating the request (also after a lost response) records the queued items and returns the run; it + enqueues nothing new. The items come from the queue's enqueue receipt for the key while it exists, otherwise from + the queue items that carry the run's origin and those the runner saw. Once a reset removed the run, a repeat is + refused (404). Acknowledge the key through the queue's enqueue_batch/acknowledge route once the response + arrived.""" + await asyncio.to_thread(assert_image_move_maintenance_inactive) + try: + return await _service().create_run(key, request, current_admin.user_id) + except ( + UnknownModelException, + CalibrationRunNotFoundError, + CalibrationGraphError, + CalibrationUnavailableError, + CalibrationStoreError, + CalibrationQueueFullError, + EnqueueIdempotencyConflictError, + EnqueueReceiptLimitError, + ) as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/working_memory/status", + operation_id="get_working_memory_calibration_status", + response_model=WorkingMemoryCalibrationStatus, + responses={503: _STORE_UNREADABLE}, +) +def get_working_memory_calibration_status(current_admin: AdminUserOrDefault) -> WorkingMemoryCalibrationStatus: + """Every device class's calibration runs, with their derived status, point counts and environments.""" + try: + return _service().get_status() + except CalibrationStoreError as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/i/{key}/working_memory/plan", + operation_id="get_model_working_memory_calibration_plan", + response_model=WorkingMemoryCalibrationPlan, + responses={404: {"description": "The model could not be found"}}, +) +def get_model_working_memory_calibration_plan( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model to calibrate"), +) -> WorkingMemoryCalibrationPlan: + """The text-to-image cases that calibrate the model's decode on this machine's GPU, how often each runs, which + other installed models the calibration also moves, or why the model cannot be calibrated here (`supported`).""" + try: + return _service().get_plan(key) + except UnknownModelException as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/i/{key}/working_memory/summary", + operation_id="get_model_working_memory_calibration_summary", + response_model=WorkingMemoryCalibrationModelSummary, + responses={ + 404: {"description": "The model could not be found"}, + 409: {"description": "No device_class_id was given and there is no current one"}, + 503: _STORE_UNREADABLE, + }, +) +def get_model_working_memory_calibration_summary( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model whose measurements to summarise"), + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> WorkingMemoryCalibrationModelSummary: + """What the calibration windows of the model's lines measured on one device class, per estimator line and + environment: the sizes, the largest demand, the shipped and the user estimate, whether the user line is in use, + stale or unusable, and which models measured it, also for a line the model's decode shares with other models' + measurements; and the model's runs.""" + try: + return _service().get_model_summary(key, device_class_id) + except (UnknownModelException, CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.delete( + "/working_memory/runs/{run_id}", + operation_id="delete_working_memory_calibration_run", + response_model=WorkingMemoryCalibrationResetResult, + responses={ + 404: {"description": "No usable store file holds the run"}, + 409: {"description": "The run's store file became unusable after the run was found; it is kept as it is"}, + 503: _STORE_UNREADABLE, + }, +) +def delete_working_memory_calibration_run( + current_admin: AdminUserOrDefault, + run_id: str = Path(description="The calibration run to remove"), +) -> WorkingMemoryCalibrationResetResult: + """Remove a calibration run and its measurements. Its queue items that have not run yet fail when they start.""" + try: + return _service().reset_run(run_id) + except (CalibrationRunNotFoundError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.delete( + "/i/{key}/working_memory", + operation_id="reset_model_working_memory_calibration", + response_model=WorkingMemoryCalibrationResetResult, + responses={409: _NO_DEVICE_CLASS, 503: _STORE_UNREADABLE}, +) +def reset_model_working_memory_calibration( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model whose runs and measurements to remove"), + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> WorkingMemoryCalibrationResetResult: + """Remove a model's calibration runs with all their measurements, and every measurement of the model taken in + other runs (a VAE measured inside a transformer's run, say), from one device class.""" + try: + return _service().reset_model(key, device_class_id) + except (CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.delete( + "/working_memory", + operation_id="reset_working_memory_calibration", + response_model=WorkingMemoryCalibrationResetResult, + responses={409: _NO_DEVICE_CLASS, 503: _STORE_UNREADABLE}, +) +def reset_working_memory_calibration( + current_admin: AdminUserOrDefault, + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> WorkingMemoryCalibrationResetResult: + """Remove every calibration run and measurement of one device class.""" + try: + return _service().reset_all(device_class_id) + except (CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/working_memory/export", + operation_id="export_working_memory_calibration", + response_class=Response, + responses={ + 200: { + "description": "The device class's store file byte for byte: a CalibrationStoreFile (schema_version 1) " + "when this version can read it, otherwise whatever the file holds", + "content": {"application/json": {}}, + }, + 404: {"description": "The device class has no store file"}, + 409: {"description": "No device_class_id was given and there is no current one"}, + 503: _STORE_UNREADABLE, + }, +) +def export_working_memory_calibration( + current_admin: AdminUserOrDefault, + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> Response: + """The device class's calibration store file as it is on disk, readable by this version or not.""" + try: + raw = _service().export(device_class_id) + except (CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + if raw is None: + raise HTTPException(status_code=404, detail="This device class has no calibration store") + return Response(content=raw, media_type="application/json") diff --git a/invokeai/app/api_app.py b/invokeai/app/api_app.py index 4802132c2b6..7b359a9e02e 100644 --- a/invokeai/app/api_app.py +++ b/invokeai/app/api_app.py @@ -52,6 +52,7 @@ virtual_boards, wildcards, workflows, + working_memory, ) from invokeai.app.api.sockets import SocketIO from invokeai.app.services.config.config_default import get_config @@ -786,6 +787,7 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: app.include_router(utilities.utilities_router, prefix="/api") app.include_router(fonts.fonts_router, prefix="/api") app.include_router(model_manager.model_manager_router, prefix="/api") +app.include_router(working_memory.working_memory_router, prefix="/api") app.include_router(image_moves.image_moves_router, prefix="/api") app.include_router(images.images_router, prefix="/api") app.include_router(image_map.image_map_router, prefix="/api") diff --git a/invokeai/app/invocations/flux2/flux2_denoise.py b/invokeai/app/invocations/flux2/flux2_denoise.py index 928d2e1d117..af8ac8e47a8 100644 --- a/invokeai/app/invocations/flux2/flux2_denoise.py +++ b/invokeai/app/invocations/flux2/flux2_denoise.py @@ -28,6 +28,7 @@ from invokeai.backend.flux.sampling_utils import clip_timestep_schedule_fractional from invokeai.backend.flux.schedulers import FLUX_SCHEDULER_LABELS, FLUX_SCHEDULER_MAP, FLUX_SCHEDULER_NAME_VALUES from invokeai.backend.flux2.denoise import denoise +from invokeai.backend.flux2.denoise_working_memory import estimate_flux2_denoise_working_memory from invokeai.backend.flux2.extensions.regional_prompting_extension import Flux2RegionalPromptingExtension from invokeai.backend.flux2.ref_image_extension import Flux2RefImageExtension from invokeai.backend.flux2.sampling_utils import ( @@ -50,31 +51,13 @@ ) from invokeai.backend.patches.lora_conversions.flux_lora_constants import FLUX_LORA_TRANSFORMER_PREFIX from invokeai.backend.patches.model_patch_raw import ModelPatchRaw -from invokeai.backend.quantization.dequantizing_linear import ( - peak_dequant_transient_bytes, - requires_sidecar_patching, -) +from invokeai.backend.quantization.dequantizing_linear import requires_sidecar_patching from invokeai.backend.rectified_flow.rectified_flow_inpaint_extension import RectifiedFlowInpaintExtension from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState from invokeai.backend.stable_diffusion.diffusion.conditioning_data import FLUXConditioningInfo -from invokeai.backend.util.attention import sdpa_score_matrix_bytes from invokeai.backend.util.devices import TorchDevice from invokeai.backend.util.sage_attention import sage_attention_scope -# FLUX.2 attention geometry. The head dim is 128 across every variant and the head count follows -# the hidden size (Klein 4B: 3072/24, Klein 9B: 4096/32, [dev] 6144/48), so the width is the single -# number that describes both. Only the head dim decides which SDPA kernel is eligible; the head -# count scales the `math` fallback's score matrix. -FLUX2_ATTENTION_HEAD_DIM = 128 -# The width the per-token constant below was measured on. Estimates scale off this. -FLUX2_REFERENCE_HIDDEN_SIZE = 4096 -# Peak reserved activation bytes per attended token at the reference width. Measured slope between -# 4608 and 9216 tokens: 0.3859 MB/tok on CUDA (RTX 4090, torch 2.7.1) and 0.4067 on ROCm (RX 9070 -# XT, gfx1201, torch 2.10). 0.42 is an upper bound on both -- it was 0.4, which the ROCm point -# exceeds. The margin is deliberately small because this is the dominant term at every resolution. -FLUX2_BYTES_PER_TOKEN_AT_REFERENCE_WIDTH = int(0.42 * 1024**2) -# The widest variant, used when the config does not tell us which one this is -- over-reserving on -# an unknown model beats under-reserving on the largest one. FLUX2_MAX_HIDDEN_SIZE = 6144 @@ -532,54 +515,41 @@ def _run_diffusion(self, context: InvocationContext) -> torch.Tensor: bn_std=bn_std, ) - # Estimate the peak activation memory the transformer forward will need and ask the model cache - # to keep that much VRAM free. Without this hint the cache reserves only the small default - # working memory and fills the rest of the card with the model, so anything beyond a plain - # low-resolution generation OOMs. Reference images are the dominant term: their latents are - # concatenated onto the image stream, so three 1024x1024 references quadruple the sequence - # (and with it the activation footprint) of a 1024x1024 generation. - ref_image_seq_len = ref_image_extension.ref_image_latents.shape[1] if ref_image_extension is not None else 0 - # The additive bias is skipped entirely when reference images are present (see below), so the - # mask only costs anything -- storage, and possibly a materialized score matrix -- without them. - regional_attn_mask = regional_extension.restricted_attn_mask if ref_image_seq_len == 0 else None - estimated_working_memory = self._estimate_working_memory( - image_seq_len=packed_h * packed_w, - ref_image_seq_len=ref_image_seq_len, - text_seq_len=max(txt.shape[1], neg_txt.shape[1] if neg_txt is not None else 0), - num_loras=len(self.transformer.loras), - # Taken from `x`, not from `b`. `b` is the *noise* tensor's batch, which this node builds - # at 1 from width/height/seed even when the init latents carry more; the img2img preblend - # above then broadcasts the two, so `x` is the only thing that knows how many samples - # actually go through the transformer. Reference latents are repeated to match it - # (`ensure_batch_size` below), so they scale with it too. - batch_size=x.shape[0], - # Activation cost per token scales with the transformer's width, and [dev] is 1.5x - # Klein 9B. Fall back to the widest variant when the config cannot tell us. - hidden_size=flux2_hidden_size(getattr(transformer_config, "variant", None)) or FLUX2_MAX_HIDDEN_SIZE, - # The mask itself is already allocated; only the additive bias built per forward is new. - regional_attention_bias_bytes=( - regional_attn_mask.numel() * torch.empty((), dtype=inference_dtype).element_size() - if regional_attn_mask is not None - else 0 - ), - has_regional_attention_mask=regional_attn_mask is not None, - device=device, - dtype=inference_dtype, - ) - with ExitStack() as exit_stack: transformer_info = context.models.load(self.transformer.transformer) - # An `int8_tensorwise` or GGUF build materializes each linear's dequantized weight per forward. - # That transient is not part of the model's resident size, so it is added to the - # activation estimate -- the two are alive at the same time -- rather than taking its - # chances against whatever slack the estimate happens to have. Read from the unlocked - # model, before the VRAM lock the reservation applies to; zero for every other build. - dequant_bytes = peak_dequant_transient_bytes(transformer_info.model, inference_dtype) + # Ask the model cache to keep free what the transformer forward needs. Without it the cache keeps only + # the default working memory and fills the rest of the card with the model, so anything beyond a plain + # low-resolution generation OOMs. Reference images are the dominant term: their latents are concatenated + # onto the image stream, so three 1024x1024 references quadruple the attended sequence of a 1024x1024 + # generation. Priced on the unlocked model, whose quantized or fp8-stored layers add a transient per + # forward, before the VRAM lock the reservation applies to. + ref_image_seq_len = ref_image_extension.ref_image_latents.shape[1] if ref_image_extension is not None else 0 + working_memory = estimate_flux2_denoise_working_memory( + transformer_info.model, + # Activation cost per token follows the transformer's width, and [dev] is 1.5x Klein 9B. Priced as + # the widest variant when the config cannot tell: over-reserving on an unknown model beats + # under-reserving on the largest one. + hidden_size=flux2_hidden_size(getattr(transformer_config, "variant", None)) or FLUX2_MAX_HIDDEN_SIZE, + image_seq_len=packed_h * packed_w, + ref_image_seq_len=ref_image_seq_len, + text_seq_len=max(txt.shape[1], neg_txt.shape[1] if neg_txt is not None else 0), + # Taken from `x`, not from `b`. `b` is the *noise* tensor's batch, which this node builds at 1 from + # width/height/seed even when the init latents carry more; the img2img preblend above then + # broadcasts the two, so `x` is the only thing that knows how many samples actually go through the + # transformer. Reference latents are repeated to match it (`ensure_batch_size` below). + batch_size=x.shape[0], + lora_count=len(self.transformer.loras), + # The additive bias is skipped entirely when reference images are present (see below), so the mask + # only costs anything without them. + regional_attn_mask=regional_extension.restricted_attn_mask if ref_image_seq_len == 0 else None, + device=device, + dtype=inference_dtype, + ) # Load the transformer model (cached_weights, transformer) = exit_stack.enter_context( - transformer_info.model_on_device(working_mem_bytes=estimated_working_memory + dequant_bytes) + transformer_info.model_on_device(working_memory=working_memory) ) config = transformer_config @@ -694,95 +664,6 @@ def _prep_inpaint_mask(self, context: InvocationContext, latents: torch.Tensor) mask = mask.to(device=latents.device, dtype=latents.dtype) return mask.expand_as(latents) - def _estimate_working_memory( - self, - image_seq_len: int, - ref_image_seq_len: int, - text_seq_len: int, - num_loras: int, - batch_size: int = 1, - hidden_size: int = FLUX2_REFERENCE_HIDDEN_SIZE, - regional_attention_bias_bytes: int = 0, - has_regional_attention_mask: bool = False, - device: torch.device | None = None, - dtype: torch.dtype = torch.bfloat16, - ) -> int: - """Estimate peak transformer activation memory (bytes) so the model cache reserves enough headroom. - - FLUX.2 attention runs through SDPA without materializing the O(seq^2) score matrix, so the - activation footprint scales *linearly* with the total attended sequence -- text tokens, image - tokens, and reference-image tokens alike. Measured on the Klein 9B geometry in bf16 as peak - reserved memory, that slope holds from 1.5k to 28k tokens and is independent of the block - count (a no-grad forward frees each block's intermediates). It is *not* independent of the - build: 0.3859 MB/token on CUDA, 0.4067 on ROCm/gfx1201, so the constant is 0.42. - - It is *not* independent of the transformer's width, which is why ``hidden_size`` is a - parameter rather than a constant. Measured slope between 4608 and 9216 tokens, block count - and everything else held fixed: - - CUDA 3072 (Klein 4B) 0.291 MB/tok 4096 (Klein 9B) 0.386 6144 ([dev]) 0.555 - ROCm/gfx1201 3072 0.315 4096 0.407 6144 0.589 - - which is 0.755 / 1.000 / 1.438 on CUDA and 0.774 / 1.000 / 1.448 on ROCm, against width - ratios of 0.75 / 1.00 / 1.50 -- linear in width on both, - and slightly sub-linear at the top so scaling by width stays an upper bound. Calibrating on - Klein 9B alone would have under-reserved [dev] by a third: 1024x1024 with three 1024x1024 - references is 16896 tokens, ~7.6GB reserved against ~10GB needed on both platforms. The head - count follows the same width, so the score-matrix term gets the real one instead of the - widest. - - The reference-image term is what makes this estimate necessary rather than merely nice to - have: a 1024x1024 generation is 4096 image tokens (~1.7GB), but attaching three 1024x1024 - references adds 12288 more for ~6.5GB, and a 1328px tile with three 1328px references reaches - ~10.9GB -- against a default ``device_working_mem_gb`` of 3. - - A fixed base covers resolution-independent overhead (transient fp8 -> bf16 weight casts during - the forward, and allocator slack across many steps); the dequantization copy of an int8 or GGUF - build is added by the caller. LoRA sidecar patches add an extra - activation branch per patched layer, so we add a per-LoRA margin. - - Batch multiplies the token count and nothing else. A batch of B is B independent sequences, - so it enters the linear term exactly as extra sequence does -- measured on the Klein geometry - with a reduced block count: 4608 tokens at B=1 peaks at 2570MB, the same 4608 at B=2 (9216 - tokens) at 5126MB, and 9728 tokens at B=1 at 5584MB. Batch and sequence are interchangeable - to within the noise. Reference latents are repeated per sample (`ensure_batch_size`), so they - scale with it too, and the score matrix is shaped (batch, heads, S, S). The fixed base does - not scale -- it is about weights, not activations -- and neither does the regional bias, which - is built as (1, 1, S, S) and broadcast across the batch. - - The linear model holds only while attention runs on a fused kernel. Regional prompting is - where that stops being a given: it hands the transformer a dense additive ``S x S`` bias, - which flash attention never accepts, leaving whatever else the build has -- and if that is - the ``math`` fallback, a materialized ``heads x S x S`` score matrix. Whether a mask forces - that is build-specific and not worth predicting: CUDA's memory-efficient kernel takes it, and - so does ROCm's on gfx1100. The device decides too (MPS has no fused SDPA kernel at all), and - so does the diffusers attention backend this build dispatches through. - ``sdpa_score_matrix_bytes`` asks all three and adds the score matrix only where it is really - built: on CUDA with the stock backend the term is zero (verified: peak stays linear with the - bias attached). - """ - GB = 1024**3 - per_token_bytes = int(FLUX2_BYTES_PER_TOKEN_AT_REFERENCE_WIDTH * hidden_size / FLUX2_REFERENCE_HIDDEN_SIZE) - total_seq_len = image_seq_len + ref_image_seq_len + text_seq_len - estimated = total_seq_len * batch_size * per_token_bytes - estimated += int(1.0 * GB) - estimated += regional_attention_bias_bytes - estimated += sdpa_score_matrix_bytes( - device=device if device is not None else TorchDevice.choose_torch_device(), - dtype=dtype, - num_heads=(hidden_size // FLUX2_ATTENTION_HEAD_DIM) * batch_size, - head_dim=FLUX2_ATTENTION_HEAD_DIM, - seq_len=total_seq_len, - has_attn_mask=has_regional_attention_mask, - # The FLUX.2 transformer's attention goes through diffusers' `dispatch_attention_fn`, - # which can route around torch's SDPA entirely -- including to a forced `math` backend. - via_diffusers_dispatch=True, - ) - if num_loras > 0: - # A sidecar branch is an activation, so it scales with the batch like the rest of them. - estimated += int(0.5 * num_loras * batch_size * GB) - return estimated - def _load_text_conditioning( self, context: InvocationContext, diff --git a/invokeai/app/invocations/vae/image_to_latents.py b/invokeai/app/invocations/vae/image_to_latents.py index d599fadd862..1be14f136b2 100644 --- a/invokeai/app/invocations/vae/image_to_latents.py +++ b/invokeai/app/invocations/vae/image_to_latents.py @@ -27,7 +27,7 @@ from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor from invokeai.backend.stable_diffusion.vae_tiling import patch_vae_tiling_params from invokeai.backend.util.devices import TorchDevice -from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl +from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl, sd_vae_run_dtype """ SDXL VAE color compensation values determined experimentally to reduce color drift. @@ -86,9 +86,10 @@ def vae_encode( with vae_info.model_on_device(working_memory=estimated_working_memory) as (_, vae): assert isinstance(vae, (AutoencoderKL, AutoencoderTiny)), "VAE must be of type SD-1.5 or SDXL" orig_dtype = vae.dtype - if upcast: - vae.to(dtype=torch.float32) - + # The dtype the estimate above priced: fp32 when upcasting, fp16 otherwise, on every device. + run_dtype = sd_vae_run_dtype("encode", upcast, vae_info.compute_device) + vae.to(dtype=run_dtype) + if run_dtype == torch.float32: use_torch_2_0_or_xformers = hasattr(vae.decoder, "mid_block") and isinstance( vae.decoder.mid_block.attentions[0].processor, ( @@ -107,10 +108,6 @@ def vae_encode( # else: # latents = latents.float() - else: - vae.to(dtype=torch.float16) - # latents = latents.half() - if tiled: vae.enable_tiling() else: diff --git a/invokeai/app/invocations/vae/latents_to_image.py b/invokeai/app/invocations/vae/latents_to_image.py index 29c6a1b9db8..bb2aafd0bfb 100644 --- a/invokeai/app/invocations/vae/latents_to_image.py +++ b/invokeai/app/invocations/vae/latents_to_image.py @@ -20,7 +20,7 @@ from invokeai.backend.stable_diffusion.extensions.seamless import SeamlessExt from invokeai.backend.stable_diffusion.vae_tiling import patch_vae_tiling_params from invokeai.backend.util.devices import TorchDevice -from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl +from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl, sd_vae_run_dtype @invocation( @@ -74,16 +74,12 @@ def invoke(self, context: InvocationContext) -> ImageOutput: # wrongly place the latents (and thus the whole decode) on the CPU (see #9373). device = vae_info.compute_device latents = latents.to(device) - # Force fp32 when running on CPU (e.g. when the VAE is configured cpu_only). fp16 conv does run on CPU with + # fp32 when asked, and on the CPU (e.g. when the VAE is configured cpu_only): fp16 conv does run on CPU with # the pinned torch, but it's much slower than fp32 there, and SD/SDXL VAE has known fp16 overflow issues - # that produce black images — fp32 avoids both. - if self.fp32 or device.type == "cpu": - # FP32 mode: convert everything to float32 for maximum precision - vae.to(dtype=torch.float32) - latents = latents.float() - else: - vae.to(dtype=torch.float16) - latents = latents.half() + # that produce black images — fp32 avoids both. fp16 otherwise. The estimate above priced the same rule. + run_dtype = sd_vae_run_dtype("decode", self.fp32, device) + vae.to(dtype=run_dtype) + latents = latents.to(dtype=run_dtype) if use_tiling: vae.enable_tiling() diff --git a/invokeai/app/invocations/z_image/z_image_denoise.py b/invokeai/app/invocations/z_image/z_image_denoise.py index a96d0962b40..9a783b0c64e 100644 --- a/invokeai/app/invocations/z_image/z_image_denoise.py +++ b/invokeai/app/invocations/z_image/z_image_denoise.py @@ -32,15 +32,15 @@ from invokeai.backend.patches.layer_patcher import LayerPatcher, PatchSpec from invokeai.backend.patches.lora_conversions.z_image_lora_constants import Z_IMAGE_LORA_TRANSFORMER_PREFIX from invokeai.backend.patches.model_patch_raw import ModelPatchRaw -from invokeai.backend.quantization.dequantizing_linear import peak_dequant_transient_bytes, requires_sidecar_patching +from invokeai.backend.quantization.dequantizing_linear import requires_sidecar_patching from invokeai.backend.rectified_flow.rectified_flow_inpaint_extension import RectifiedFlowInpaintExtension from invokeai.backend.rectified_flow.schedule_shift import ShiftFit, time_shift from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ZImageConditioningInfo -from invokeai.backend.util.attention import sdpa_score_matrix_bytes from invokeai.backend.util.devices import TorchDevice from invokeai.backend.util.fp8 import get_model_compute_dtype from invokeai.backend.util.sage_attention import sage_attention_scope +from invokeai.backend.z_image.denoise_working_memory import estimate_z_image_denoise_working_memory from invokeai.backend.z_image.extensions.regional_prompting_extension import ZImageRegionalPromptingExtension from invokeai.backend.z_image.text_conditioning import ZImageTextConditioning from invokeai.backend.z_image.z_image_control_adapter import ZImageControlAdapter @@ -48,21 +48,8 @@ ZImageControlNetExtension, z_image_forward_with_control, ) -from invokeai.backend.z_image.z_image_patchify_utils import SEQ_MULTI_OF from invokeai.backend.z_image.z_image_transformer_patch import patch_transformer_for_regional_prompting -# The transformer's width and head count, which its ControlNet adapters share: 30 heads of 128. -Z_IMAGE_HIDDEN_SIZE = 3840 -Z_IMAGE_ATTENTION_HEADS = 30 -# Peak reserved activation bytes per attended token on a fused SDPA kernel; see `_estimate_working_memory`. -Z_IMAGE_BYTES_PER_TOKEN = int(0.25 * 1024**2) - - -def _padded(tokens: int) -> int: - """The transformer pads the image and the caption token runs to a multiple of `SEQ_MULTI_OF` each.""" - return tokens + (-tokens) % SEQ_MULTI_OF - - # The shift fit of diffusers' ZImagePipeline. 4096 image tokens is a 1024x1024 frame -- one megapixel, and where the # reference fit ends. Extrapolated past it the shift runs away fast -- 25 at 2048px, 810 at 3072px and 103_777 at # 4096px -- and `_get_sigmas` flattens with it. Because `denoising_start` is clipped by *index* here, the step count @@ -264,76 +251,6 @@ def _get_sigmas(self, shift: float, num_steps: int) -> list[float]: sigmas.append(1.0 if t >= 1 else 0.0 if t <= 0 else time_shift(shift, t)) return sigmas - @staticmethod - def _estimate_working_memory( - image_seq_len: int, - text_seq_len: int, - num_loras: int, - num_control_blocks: int = 0, - regional_attention_bias_bytes: int = 0, - has_attention_mask: bool = False, - device: torch.device | None = None, - dtype: torch.dtype = torch.bfloat16, - ) -> int: - """Estimate peak transformer working memory (bytes) so the model cache keeps that much VRAM free. - - The transformer attends over one unified sequence of image and caption tokens, each run padded to a - multiple of ``SEQ_MULTI_OF``. On a fused SDPA kernel its activations scale linearly with that sequence. - Measured as peak reserved memory in bf16 on CUDA with a 512-token caption, by - `scripts/calibrate_z_image_working_memory.py`: 0.99 GiB at 1024px (4608 tokens), 1.96 GiB at 1440px (8640) - and 3.85 GiB at 2048px (16896) -- 0.22 to 0.23 MB per token, unchanged between 2 and 4 blocks, because a - no-grad forward frees each block's intermediates. 0.25 MB per token bounds every point on its own, with - room for the ~5% more ROCm needed on FLUX.2. The fixed base covers what does not scale with the sequence - -- fp8 weights cast per forward, and allocator slack across steps; the caller adds the dequantization copy - of an int8, nvfp4 or GGUF build. Conditional and unconditional - passes run one after the other, so only the longer caption counts. - - Where SDPA has no fused kernel for these shapes it materializes the score matrices, and those dominate: - on ROCm under Windows, whose fused kernels are disabled, a 1024px generation spilled 7.7 GB into shared - system memory while this node reserved only the default working memory. ``sdpa_score_matrix_bytes`` adds - the term only where it is really built. An attention mask narrows which fused kernels a build can use, so - the call is told whether one is passed: regional prompting's additive bias, or the padding mask the - ControlNet forward always builds. The plain forward passes none at batch size 1. - - A ControlNet adapter first runs its own blocks over a sequence of the same length and keeps one hint per - block, alive through the main layers. Measured over the plain forward at 1024px: 3.3, 5.9 and 18.9 - hint-sized tensors (sequence x width x element size) for 1, 6 and 15 blocks, so n + 4 are charged. - """ - GB = 1024**3 - seq_len = _padded(image_seq_len) + _padded(text_seq_len) - estimated = seq_len * Z_IMAGE_BYTES_PER_TOKEN + int(1.0 * GB) - estimated += regional_attention_bias_bytes - estimated += sdpa_score_matrix_bytes( - device=device if device is not None else TorchDevice.choose_torch_device(), - dtype=dtype, - num_heads=Z_IMAGE_ATTENTION_HEADS, - head_dim=Z_IMAGE_HIDDEN_SIZE // Z_IMAGE_ATTENTION_HEADS, - seq_len=seq_len, - has_attn_mask=has_attention_mask, - # The attention goes through diffusers' `dispatch_attention_fn`, which can route around torch's SDPA. - via_diffusers_dispatch=True, - ) - if num_control_blocks: - estimated += (num_control_blocks + 4) * seq_len * Z_IMAGE_HIDDEN_SIZE * dtype.itemsize - estimated += int(0.5 * num_loras * GB) - return estimated - - @staticmethod - def _regional_attention_bias_bytes( - regional_attn_mask: torch.Tensor | None, image_seq_len: int, dtype: torch.dtype - ) -> int: - """What `regional_forward` allocates per positive pass for regional prompting: a boolean copy of the mask, - the additive bias over the padded unified sequence, and the image-to-image block `torch.where` fills it - from. The mask itself is already on the device when the estimate is taken. Measured at 1024px on CUDA: 68 MiB - over the plain forward, against 93 MiB charged.""" - if regional_attn_mask is None: - return 0 - mask_side = regional_attn_mask.shape[0] - unified_seq_len = _padded(image_seq_len) + _padded(mask_side - image_seq_len) - return ( - mask_side * mask_side + (unified_seq_len * unified_seq_len + image_seq_len * image_seq_len) * dtype.itemsize - ) - def _run_diffusion(self, context: InvocationContext) -> torch.Tensor: device = TorchDevice.choose_torch_device() inference_dtype = TorchDevice.choose_bfloat16_safe_dtype(device) @@ -514,28 +431,23 @@ def _run_diffusion(self, context: InvocationContext) -> torch.Tensor: # transformer as that takes. Without it the cache reserves only `device_working_mem_gb`, which a # build that materializes the score matrices (ROCm on Windows, MPS) overruns several times at 1024px. regional_attn_mask = regional_extension.regional_attn_mask - working_mem_bytes = self._estimate_working_memory( + working_memory = estimate_z_image_denoise_working_memory( + transformer_info.model, image_seq_len=img_seq_len, text_seq_len=max( pos_prompt_embeds.shape[0], neg_prompt_embeds.shape[0] if neg_prompt_embeds is not None else 0 ), - num_loras=len(self.transformer.loras), - num_control_blocks=num_control_blocks, - regional_attention_bias_bytes=self._regional_attention_bias_bytes( - regional_attn_mask, img_seq_len, inference_dtype - ), + lora_count=len(self.transformer.loras), + control_blocks=num_control_blocks, + regional_attn_mask=regional_attn_mask, has_attention_mask=regional_attn_mask is not None or control_model_info is not None, device=device, dtype=inference_dtype, ) - # An int8_tensorwise, nvfp4 or GGUF build materializes each quantized linear's weight per forward, alive - # alongside the activations above and invisible to them. Read from the unlocked model; zero for - # every other build. - working_mem_bytes += peak_dequant_transient_bytes(transformer_info.model, inference_dtype) # Load transformer - always use base transformer, control is handled via extension (cached_weights, transformer) = exit_stack.enter_context( - transformer_info.model_on_device(working_mem_bytes=working_mem_bytes) + transformer_info.model_on_device(working_memory=working_memory) ) # Whether LoRA has to go on as a sidecar. Asked of the loaded module tree, not the @@ -551,9 +463,11 @@ def _run_diffusion(self, context: InvocationContext) -> torch.Tensor: if self.control is not None: assert control_model_info is not None # Locked with the same working memory: the transformer can no longer be offloaded, so an - # adapter placed with only the default reservation would fill the room kept for the forward. + # adapter placed with only the default reservation would fill the room kept for the forward. A + # plain reservation, not a second estimate: inside the transformer's calibration window this lock + # is a co-lock, whose weights go into the window's offset rather than its demand. (_, control_adapter) = exit_stack.enter_context( - control_model_info.model_on_device(working_mem_bytes=working_mem_bytes) + control_model_info.model_on_device(working_mem_bytes=working_memory.reservation_bytes) ) assert isinstance(control_adapter, ZImageControlAdapter) diff --git a/invokeai/app/run_app.py b/invokeai/app/run_app.py index d12fced8b35..dc39ef9aae3 100644 --- a/invokeai/app/run_app.py +++ b/invokeai/app/run_app.py @@ -1,5 +1,9 @@ +import logging import os -from typing import Any +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from invokeai.app.services.config.config_default import InvokeAIAppConfig # Suppress the HuggingFace tokenizers fork-after-parallelism warning. The Rust # ``tokenizers`` library warms a thread pool the first time a tokenizer is used @@ -17,20 +21,82 @@ # keeps their value. os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") -# ROCm: run MIOpen's convolution-kernel search in FAST mode. The default mode benchmarks -# every candidate kernel the first time a convolution shape is seen, which on a VAE-shaped -# workload (many layers x spatial tiles x temporal chunks, separately per dtype) costs minutes -# of "warm-up" per new shape set, and torch's MIOpen ``chooseAlgorithm`` falls back to a -# *global* ``emptyCache`` whenever a candidate's workspace allocation fails - on multi-GPU -# installs that convoys the peer device mid-step (see ``install_peer_aware_empty_cache``). -# FAST looks the shape up in MIOpen's find-db and takes the first viable kernel otherwise; -# with a warmed find-db the difference in kernel quality is negligible. MIOpen reads the -# variable when it is first used, so, like the tokenizer flag, it is set at module level - -# before torch is even imported - rather than inside ``run_app()``, so no early import can run -# a convolution ahead of it. On -# CUDA/CPU/MPS builds MIOpen is absent and the variable is inert. ``setdefault`` keeps an -# explicitly exported value (e.g. ``MIOPEN_FIND_MODE=NORMAL`` to re-tune the find-db). -os.environ.setdefault("MIOPEN_FIND_MODE", "FAST") + +def _default_miopen_find_mode() -> None: + """ROCm: run MIOpen's convolution-kernel search in FAST mode. + + The default mode benchmarks every candidate kernel the first time a convolution shape is seen, which on a VAE-shaped + workload (many layers x spatial tiles x temporal chunks, separately per dtype) costs minutes of "warm-up" per new + shape set, and torch's MIOpen ``chooseAlgorithm`` falls back to a *global* ``emptyCache`` whenever a candidate's + workspace allocation fails - on multi-GPU installs that convoys the peer device mid-step (see + ``install_peer_aware_empty_cache``). FAST looks the shape up in MIOpen's find-db and takes the first viable kernel + otherwise; with a warmed find-db the difference in kernel quality is negligible. On CUDA/CPU/MPS builds MIOpen is + absent and the variable is inert. ``setdefault`` keeps an explicitly exported value (e.g. + ``MIOPEN_FIND_MODE=NORMAL`` to re-tune the find-db). + """ + os.environ.setdefault("MIOPEN_FIND_MODE", "FAST") + + +# MIOpen reads the variable when it is first used, so, like the tokenizer flag, it is set at module level - before torch +# is even imported - rather than inside ``run_app()``, so no early import can run a convolution ahead of it. +# `apply_server_runtime` repeats it as its first step, so that function states the whole runtime; ``setdefault`` makes +# the repeat a no-op. +_default_miopen_find_mode() + + +def _configure_torch_environment(config: "InvokeAIAppConfig", logger: logging.Logger) -> None: + """What torch, HIP and MIOpen read once, when they initialize, as `config` sets it: the first part of the server's + runtime (`apply_server_runtime`). Must run before torch is imported, which its last step does.""" + from invokeai.app.util.rocm_integrated_gpu import hide_integrated_gpus_on_rocm_windows + from invokeai.app.util.torch_cuda_allocator import ( + apply_rocm_windows_allocator_default, + configure_torch_cuda_allocator, + ) + + _default_miopen_find_mode() + + # Before torch is imported anywhere: HIP reads its device list once, when torch initializes it, and an explicit + # allocator configuration below imports torch. Runs a short child process on Windows ROCm. + hide_integrated_gpus_on_rocm_windows(logger) + + # Configure the torch CUDA memory allocator. + # NOTE: It is important that this happens before torch is imported. + if config.pytorch_cuda_alloc_conf: + configure_torch_cuda_allocator(config.pytorch_cuda_alloc_conf, logger) + else: + apply_rocm_windows_allocator_default(logger) + + # Decide on AOTriton's experimental attention kernels before anything can run attention: torch reads the switch + # once, at its first fused-kernel check, and the server already runs attention while starting up. This imports + # torch, so it comes after the allocator configuration. + from invokeai.app.util.rocm_aotriton import apply_rocm_aotriton_setting + + apply_rocm_aotriton_setting(config.rocm_aotriton_experimental, config.generation_devices, logger) + + +def apply_server_runtime(config: "InvokeAIAppConfig", logger: logging.Logger) -> None: + """Give this process the server's kernels and allocation, for a process that measures what the server runs (the + working-memory calibration scripts). Must run before torch is imported. + + The steps, in the server's order (`run_app`), with `config`'s settings: + + 1. ``MIOPEN_FIND_MODE`` defaults to FAST. + 2. On ROCm under Windows, integrated GPUs are hidden from HIP. + 3. The allocator: ``pytorch_cuda_alloc_conf``, or without one the platform's default (expandable segments for ROCm + on Windows). + 4. AOTriton's experimental attention kernels, as ``rocm_aotriton_experimental`` decides. + 5. The monkeypatches, the ROCm SDPA guard among them, with ``attention_backend`` (`apply_monkeypatches`). + + Between 4 and 5 the server imports its invocations, logs its device and finds its port. The invocations' modules bind + library functions there, before the monkeypatches: one that binds a patched function by name at import keeps the + unpatched one in the server (PiD's ``scaled_dot_product_attention``), where a module imported after this function + binds the patched one. + """ + _configure_torch_environment(config, logger) + + from invokeai.app.util.startup_utils import apply_monkeypatches + + apply_monkeypatches(config.attention_backend) def get_app(): @@ -58,10 +124,6 @@ def run_app() -> None: import uvicorn from invokeai.app.services.config.config_default import get_config - from invokeai.app.util.torch_cuda_allocator import ( - apply_rocm_windows_allocator_default, - configure_torch_cuda_allocator, - ) from invokeai.backend.util.logging import InvokeAILogger # Load config. @@ -69,27 +131,11 @@ def run_app() -> None: logger = InvokeAILogger.get_logger(config=app_config) - # Before torch is imported anywhere: HIP reads its device list once, when torch initializes it, and an explicit - # allocator configuration below imports torch. Runs a short child process on Windows ROCm. - from invokeai.app.util.rocm_integrated_gpu import hide_integrated_gpus_on_rocm_windows - - hide_integrated_gpus_on_rocm_windows(logger) - - # Configure the torch CUDA memory allocator. - # NOTE: It is important that this happens before torch is imported. - if app_config.pytorch_cuda_alloc_conf: - configure_torch_cuda_allocator(app_config.pytorch_cuda_alloc_conf, logger) - else: - apply_rocm_windows_allocator_default(logger) + # The server's runtime (`apply_server_runtime`) in its two parts, around the startup steps below that need torch: + # first what torch, HIP and MIOpen read when they initialize, before torch is imported anywhere. + _configure_torch_environment(app_config, logger) - # Decide on AOTriton's experimental attention kernels before anything can run attention: torch reads the switch - # once, at its first fused-kernel check, and the server already runs attention while starting up. This imports - # torch, so it comes after the allocator configuration. - from invokeai.app.util.rocm_aotriton import apply_rocm_aotriton_setting - - apply_rocm_aotriton_setting(app_config.rocm_aotriton_experimental, app_config.generation_devices, logger) - - # This import must happen after configure_torch_cuda_allocator() is called, because the module imports torch. + # This import must happen after the allocator is configured, because the module imports torch. from invokeai.app.invocations.baseinvocation import InvocationRegistry from invokeai.app.invocations.load_custom_nodes import load_custom_nodes from invokeai.backend.util.devices import TorchDevice @@ -97,7 +143,7 @@ def run_app() -> None: torch_device_name = TorchDevice.get_generation_devices_summary(app_config.generation_devices) logger.info(f"Using torch device: {torch_device_name}") - # Import from startup_utils here to avoid importing torch before configure_torch_cuda_allocator() is called. + # Import from startup_utils here to avoid importing torch before the allocator is configured. from invokeai.app.util.startup_utils import ( apply_monkeypatches, check_cuda_build_compatibility, @@ -118,8 +164,10 @@ def run_app() -> None: app_config.port = first_open_port logger.warning(f"Port {orig_config_port} is already in use. Using port {app_config.port}.") - # Miscellaneous startup tasks. + # The runtime's second part: the monkeypatches, once the invocations are imported. apply_monkeypatches(app_config.attention_backend) + + # Miscellaneous startup tasks. register_mime_types() check_cudnn(logger) check_cuda_build_compatibility(logger) diff --git a/invokeai/app/services/invocation_services.py b/invokeai/app/services/invocation_services.py index f5416b02bb6..618de627229 100644 --- a/invokeai/app/services/invocation_services.py +++ b/invokeai/app/services/invocation_services.py @@ -56,6 +56,9 @@ from invokeai.app.services.videos.videos_base import VideoServiceABC from invokeai.app.services.workflow_records.workflow_records_base import WorkflowRecordsStorageBase from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_base import WorkflowThumbnailServiceBase + from invokeai.app.services.working_memory_calibration.working_memory_calibration_base import ( + WorkingMemoryCalibrationServiceBase, + ) from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData @@ -117,6 +120,7 @@ def __init__( progress_previews: "ProgressPreviewsBase | None" = None, fonts: "FontService | None" = None, gallery_maintenance: "GalleryMaintenanceService | None" = None, + working_memory_calibration: "WorkingMemoryCalibrationServiceBase | None" = None, board_access: "BoardAccessService | None" = None, ): self.board_images = board_images @@ -170,6 +174,7 @@ def __init__( self.fonts = fonts self.intermediates = intermediates self.gallery_maintenance = gallery_maintenance + self.working_memory_calibration = working_memory_calibration # Stateless over the database, so a real one is the default: no caller should get looser rules by leaving it # out. self.board_access = board_access or BoardAccessService(database) diff --git a/invokeai/app/services/invoker.py b/invokeai/app/services/invoker.py index b504f511ba7..8c01125a05b 100644 --- a/invokeai/app/services/invoker.py +++ b/invokeai/app/services/invoker.py @@ -22,13 +22,25 @@ def __stop_service(self, service) -> None: if callable(stop_op): stop_op(self) + def _services_but_the_session_processor(self) -> list[object]: + return [getattr(self.services, name) for name in vars(self.services) if name != "session_processor"] + def _start(self) -> None: - """Starts the invoker. This is called automatically when the invoker is created.""" - for service in vars(self.services): - self.__start_service(getattr(self.services, service)) + """Starts the invoker. This is called automatically when the invoker is created. + + The session processor starts after every other service: its workers dequeue at once, and an item left pending + before the restart (the queue's start cancels those that were in progress) may reach any service, which must + have been started by then. + """ + for service in self._services_but_the_session_processor(): + self.__start_service(service) + self.__start_service(self.services.session_processor) def stop(self) -> None: """Stops the invoker. A new invoker will have to be created to execute further.""" - # First stop all services - for service in vars(self.services): - self.__stop_service(getattr(self.services, service)) + # The session processor stops first: its workers stop taking queue items and cancel the sessions they run, and + # its stop() waits for them to exit (bounded, see DefaultSessionProcessor.stop) before the services those + # sessions use stop. + self.__stop_service(self.services.session_processor) + for service in self._services_but_the_session_processor(): + self.__stop_service(service) diff --git a/invokeai/app/services/session_processor/session_processor_default.py b/invokeai/app/services/session_processor/session_processor_default.py index 4308e7a4f35..5d41b46eefc 100644 --- a/invokeai/app/services/session_processor/session_processor_default.py +++ b/invokeai/app/services/session_processor/session_processor_default.py @@ -4,13 +4,14 @@ from contextlib import contextmanager, nullcontext, suppress from threading import BoundedSemaphore, Thread from threading import Event as ThreadEvent -from typing import Iterator, Optional +from typing import Iterator, Literal, Optional import torch from starlette.concurrency import run_in_threadpool from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput from invokeai.app.invocations.call_saved_workflow import CallSavedWorkflowInvocation +from invokeai.app.services.config.config_default import InvokeAIAppConfig from invokeai.app.services.events.events_common import ( BatchEnqueuedEvent, FastAPIEvent, @@ -45,6 +46,7 @@ ) from invokeai.app.services.shared.graph import CollectInvocation, IterateInvocation, NodeInputError from invokeai.app.services.shared.invocation_context import InvocationContextData, build_invocation_context +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import calibration_run_id from invokeai.app.util.profiler import Profiler from invokeai.backend.util.device_pool import ( GENERATION_DEVICE_POOL, @@ -52,6 +54,7 @@ ) from invokeai.backend.util.devices import TorchDevice, disable_conv_benchmark_empty_cache from invokeai.backend.util.wddm import paged_bytes +from invokeai.backend.working_memory import probe # Paged-out VRAM worth a warning. Weight uploads page a few hundred MB for an instant (0.25-0.36 GiB measured on an # RX 9060 XT), so the threshold sits above that; a session that ends with more than this paged has lost real VRAM. @@ -64,6 +67,27 @@ OWNER_LOOKUP_ATTEMPTS = 3 OWNER_LOOKUP_RETRY_SECONDS = 0.25 +# How long `DefaultSessionProcessor.stop` waits, in all, for its workers to leave their sessions. A node that checks for +# cancellation stops within one step; the bound also covers most nodes without such a check (a large VAE decode), and +# keeps a node that never returns from holding up shutdown for good. +WORKER_STOP_TIMEOUT_SECONDS = 30.0 +# After this long, `stop` says which worker it is waiting for: the wait runs on the event-loop thread at shutdown. +WORKER_STOP_NOTICE_SECONDS = 1.0 + +# Workers a stop gave up on. Process-wide, like the device pool whose locks they may still hold: no processor of this +# process starts, and none replaces the pool, while one of them runs. +_abandoned_workers: list[Thread] = [] + + +def refuse_while_an_abandoned_worker_runs() -> None: + """Raise RuntimeError while a worker an earlier `DefaultSessionProcessor.stop` gave up on still runs.""" + _abandoned_workers[:] = [thread for thread in _abandoned_workers if thread.is_alive()] + if _abandoned_workers: + raise RuntimeError( + f"Session worker {', '.join(thread.name for thread in _abandoned_workers)} of the previous session " + "processor is still running a node; it would share its GPU with the new workers. Restart the process." + ) + def queue_owner_is_active( services: InvocationServices, queue_item: SessionQueueItem, *, unreadable_is_active: bool = False @@ -165,6 +189,11 @@ def __init__( self.workflow_call_queue_lifecycle = WorkflowCallQueueLifecycle(self) # Per runner, hence per worker and device: each worker remembers the architecture it last ran. self._architecture_switch = ArchitectureSwitchOffload() + # The calibration run of the item this runner is executing, None for every other item. + self._calibration_run_id: Optional[str] = None + # What the outcome hooks last reported: `run_node` swallows a node's failure or cancellation, and a calibration + # node must not keep valid measurements then. + self._node_outcome: Optional[Literal["completed", "failed"]] = None def start(self, services: InvocationServices, cancel_event: ThreadEvent, profiler: Optional[Profiler] = None): self._services = services @@ -203,7 +232,10 @@ def _run_session_loop(self, queue_item: SessionQueueItem) -> None: if self._cancel_if_owner_revoked(queue_item): break - self.run_node(invocation, queue_item) + if self._calibration_run_id is None: + self.run_node(invocation, queue_item) + elif not self._run_calibration_node(invocation, queue_item, self._calibration_run_id): + break # The session is complete if all invocations have been run or there is an error on the session. # At this time, the queue item may be canceled, but the object itself here won't be updated yet. We must @@ -246,10 +278,117 @@ def _cancel_if_owner_revoked(self, queue_item: SessionQueueItem, *, unreadable_i def run(self, queue_item: SessionQueueItem): # Exceptions raised outside `run_node` are handled by the processor. There is no need to catch them here. + self._refresh_user_fits() + # getattr, like the workflow-call lifecycle's reads: runners also execute stand-in items without an origin. + run_id = calibration_run_id(getattr(queue_item, "origin", None)) + if run_id is not None: + self._run_calibration_item(queue_item, run_id) + return self._on_before_run_session(queue_item=queue_item) self._run_session_loop(queue_item) self._on_after_run_session(queue_item=queue_item) + def _refresh_user_fits(self) -> None: + """Let this item's estimates read the calibration store as it is now: once per item, never per estimate.""" + # getattr: stand-in service containers that runners execute with need not carry the calibration service. + service = getattr(self._services, "working_memory_calibration", None) + if service is None: + return + try: + service.refresh_user_fits() + except Exception: + # The fits read so far stay in force; a store that cannot be read must not fail a generation. + self._services.logger.warning("Could not refresh the working-memory user fits", exc_info=True) + + def _run_calibration_item(self, queue_item: SessionQueueItem, run_id: str) -> None: + """Run a working-memory calibration item, measuring each node (see `working_memory.probe`). + + The item is admitted again here, whatever admitted it to the queue -- a retry and a workflow-call child carry + the origin and owner of the item they came from: its owner must still be an administrator, the machine must + still calibrate on one GPU, and its run must still exist. Otherwise it fails before any node runs. + """ + service = self._services.working_memory_calibration + raised: Optional[BaseException] = None + try: + try: + refusal = ( + "Working-memory calibration is not available in this process." + if service is None + else service.admit(queue_item, run_id) + ) + except Exception as error: + self._services.logger.warning(f"Could not admit calibration item {queue_item.item_id}", exc_info=True) + refusal = f"Could not admit the calibration item: {error}" + if refusal is not None: + self._services.logger.warning(f"Failing calibration item {queue_item.item_id}: {refusal}") + with suppress(SessionQueueItemNotFoundError): + self._services.session_queue.fail_queue_item( + queue_item.item_id, "WorkingMemoryCalibrationError", refusal, refusal + ) + return + self._calibration_run_id = run_id + try: + self._on_before_run_session(queue_item=queue_item) + self._run_session_loop(queue_item) + self._on_after_run_session(queue_item=queue_item) + finally: + self._calibration_run_id = None + except BaseException as error: + raised = error + raise + finally: + # Also when the session raised: `admit` recorded the item as running, and only this replaces that. + if service is not None: + try: + service.observe(queue_item, run_id, raised) + except Exception: + self._services.logger.warning( + f"Could not record the outcome of calibration item {queue_item.item_id}", exc_info=True + ) + + def _run_calibration_node(self, invocation: BaseInvocation, queue_item: SessionQueueItem, run_id: str) -> bool: + """Run one node of a calibration item inside a probe scope and persist its windows, whether the node completed, + failed or was canceled; unless it completed, they are persisted invalid. Returns False when they could not be + persisted: the item has failed then.""" + service = self._services.working_memory_calibration + try: + assert service is not None + # Read before the node runs: a model reset while it runs removes points these windows must not bring back. + model_reset_epochs = service.model_reset_epochs(run_id) + except Exception as error: + return self._fail_calibration_node(invocation, queue_item, error) + with probe.calibration_scope( + run_id=run_id, + queue_item_id=queue_item.item_id, + # The source id: what the client that built the graph knows the node by. + node_id=queue_item.session.prepared_source_mapping.get(invocation.id, invocation.id), + node_type=invocation.get_type(), + ) as scope: + self._node_outcome = None + self.run_node(invocation, queue_item) + if self._node_outcome != "completed": + # A node that neither completed nor failed was canceled: `run_node` passes over a CanceledException. + scope.invalidate("exception" if self._node_outcome == "failed" else "canceled") + if not scope.windows: + return True + try: + service.record_windows(run_id, scope.windows, model_reset_epochs) + except Exception as error: + return self._fail_calibration_node(invocation, queue_item, error) + return True + + def _fail_calibration_node( + self, invocation: BaseInvocation, queue_item: SessionQueueItem, error: Exception + ) -> bool: + """Fail the item whose node's measurements cannot be saved. Call from the `except` block (for the traceback).""" + message = f"Could not save the working-memory measurements of node {invocation.id}: {error}" + self._services.logger.error(message, exc_info=True) + with suppress(SessionQueueItemNotFoundError): + self._services.session_queue.fail_queue_item( + queue_item.item_id, "WorkingMemoryCalibrationError", message, traceback.format_exc() + ) + return False + def run_node(self, invocation: BaseInvocation, queue_item: SessionQueueItem): try: # Any unhandled exception in this scope is an invocation error & will fail the graph @@ -573,6 +712,7 @@ def _on_after_run_node( - Emits an invocation complete event. - Run any callbacks registered for this event. + - Records that the node completed (see `_run_calibration_node`). """ self._services.logger.debug( @@ -587,6 +727,7 @@ def _on_after_run_node( for callback in self._on_after_run_node_callbacks: callback(invocation=invocation, queue_item=queue_item, output=output) + self._node_outcome = "completed" def _on_node_error( self, @@ -599,12 +740,14 @@ def _on_node_error( ): """Called when a node errors. Node errors may occur when running or preparing the node.. + - Records that the node failed, ahead of anything that can raise (see `_run_calibration_node`). - Set the node error on the session object. - Log the error. - Fail the queue item. - Emits an invocation error event. - Run any callbacks registered for this event. """ + self._node_outcome = "failed" self._services.progress_previews.clear_node(queue_item.item_id, invocation.id) self._services.logger.debug( @@ -708,20 +851,30 @@ def __init__( self._thread_limit = thread_limit self._polling_interval = polling_interval self._workers: list[_SessionWorker] = [] + # One per worker, once `register_generation_devices` resolved them. + self._devices: Optional[list[Optional[torch.device]]] = None - def _resolve_devices(self) -> list[Optional[torch.device]]: - """Determine the per-worker devices from config. + def register_generation_devices(self, configuration: InvokeAIAppConfig) -> list[Optional[torch.device]]: + """Resolve the workers' devices and register them in `GENERATION_DEVICE_POOL`, so the model loader can find + idle GPUs to borrow. Returns the devices, one per worker. - Resolves `generation_devices` (which defaults to `"auto"` — every available GPU) into one - normalized device per worker. Returns a single `None` (legacy single-worker, device chosen by - the global config) only if the resolution is empty (e.g. `generation_devices` set to an empty - list). + Work outside the session queue borrows from the pool from the moment its service starts (the image index's + backfill, say), so the composition root calls this before the invoker starts any service + (`ApiDependencies.initialize`). `start` calls it only when nothing did: registering again would drop a borrow in + flight. Registering creates no CUDA context: the pool holds device objects, and a worker pins its GPU in CUDA + only once it claims its first queue item (#9413). + + `generation_devices` (by default `"auto"`, every available GPU) resolves to one normalized device per worker, + or to a single `None` (one worker on the device the global config chooses) when the resolution is empty, as + with an empty list. + + Raises RuntimeError while a worker an earlier `stop` gave up on still runs (see `stop`). """ - generation_devices = self._invoker.services.configuration.generation_devices - devices = TorchDevice.get_generation_devices(generation_devices) - if not devices: - return [None] - return list(devices) + refuse_while_an_abandoned_worker_runs() + devices = TorchDevice.get_generation_devices(configuration.generation_devices) + self._devices = list(devices) if devices else [None] + GENERATION_DEVICE_POOL.set_generation_devices([d for d in self._devices if d is not None]) + return self._devices def _clone_session_runner(self, template: SessionRunnerBase) -> SessionRunnerBase: """Create an independent runner for an additional worker. @@ -749,6 +902,9 @@ def _clone_session_runner(self, template: SessionRunnerBase) -> SessionRunnerBas ) def start(self, invoker: Invoker) -> None: + """Start one worker per generation device. Raises RuntimeError while a worker an earlier `stop` gave up on + still runs (see `stop`).""" + refuse_while_an_abandoned_worker_runs() self._invoker: Invoker = invoker self._resume_event = ThreadEvent() @@ -760,11 +916,11 @@ def start(self, invoker: Invoker) -> None: register_events(QueueItemStatusChangedEvent, self._on_queue_item_status_changed) register_events(UserAccessChangedEvent, self._on_user_access_changed) - devices = self._resolve_devices() - - # Register the generation devices so the model loader can discover idle GPUs to host text - # encoders on (see offload_text_encoders_to_idle_gpus). None means legacy single-device mode. - GENERATION_DEVICE_POOL.set_generation_devices([d for d in devices if d is not None]) + devices = ( + self._devices + if self._devices is not None + else self.register_generation_devices(self._invoker.services.configuration) + ) # A worker that stood aside for a lent GPU re-polls as soon as that borrow ends. GENERATION_DEVICE_POOL.set_release_listener(self._poll_now_event.set) @@ -827,6 +983,15 @@ def start(self, invoker: Invoker) -> None: worker.thread.start() def stop(self, *args, **kwargs) -> None: + """Stop the workers and wait for them to exit, up to `WORKER_STOP_TIMEOUT_SECONDS` for all of them together. + + The invoker stops every other service once this returns, so a worker that exits in time never runs beside a + stopped service. A wait longer than `WORKER_STOP_NOTICE_SECONDS` names the worker and its queue item in the + log. A worker still in a node at the deadline is given up on, with an error in the log, so that shutdown + finishes: a node without a cancel point may never return. A later stop does not wait for it again. It keeps + its GPU, so until it exits no session processor of this process starts and none replaces the generation device + pool. The services it uses stop under it; once the database is disposed, its writes are refused. + """ self._stop_event.set() # Cancel any in-progress generation so that long-running nodes (e.g. denoising) stop at # the next step boundary instead of running to completion. Without this, a generation @@ -838,6 +1003,32 @@ def stop(self, *args, **kwargs) -> None: self._poll_now_event.set() self._resume_event.set() + deadline = time.monotonic() + WORKER_STOP_TIMEOUT_SECONDS + for worker in self._workers: + # A worker given up on by an earlier stop gets no second wait. + if worker.thread is None or worker.thread in _abandoned_workers: + continue + worker.thread.join(timeout=max(0.0, min(deadline - time.monotonic(), WORKER_STOP_NOTICE_SECONDS))) + remaining = deadline - time.monotonic() + if worker.thread.is_alive() and remaining > 0: + self._invoker.services.logger.info( + f"Waiting up to {remaining:.0f}s for the {self._describe_running(worker)} to stop" + ) + worker.thread.join(timeout=remaining) + if worker.thread.is_alive(): + _abandoned_workers.append(worker.thread) + self._invoker.services.logger.error( + f"The {self._describe_running(worker)} did not stop within {WORKER_STOP_TIMEOUT_SECONDS:.0f}s; " + "shutting down without it. Restart the process before starting Invoke again." + ) + + @staticmethod + def _describe_running(worker: _SessionWorker) -> str: + assert worker.thread is not None + queue_item = worker.queue_item + item = f" (queue item {queue_item.item_id})" if queue_item is not None else "" + return f"session worker {worker.thread.name} on {worker.label}{item}" + def _poll_now(self) -> None: self._poll_now_event.set() diff --git a/invokeai/app/services/session_queue/session_queue_base.py b/invokeai/app/services/session_queue/session_queue_base.py index 3e7bc4841fb..edf6dc85836 100644 --- a/invokeai/app/services/session_queue/session_queue_base.py +++ b/invokeai/app/services/session_queue/session_queue_base.py @@ -21,6 +21,7 @@ NodeFieldValue, PruneResult, QueueItemDeliveryResult, + QueueItemLineage, QueueItemOwner, RetryItemsResult, SessionQueueCountsByDestination, @@ -54,9 +55,11 @@ def dequeue(self, device: Optional[str] = None) -> Optional[SessionQueueItem]: @abstractmethod def enqueue_batch( - self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system" + self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system", *, all_or_none: bool = False ) -> Coroutine[Any, Any, EnqueueBatchResult]: - """Enqueues all permutations of a batch for execution for a specific user.""" + """Enqueues all permutations of a batch for execution for a specific user. A queue without room for all of + them takes as many as fit, or with `all_or_none` none of them. A repeat with the batch's idempotency key + returns the key's settled receipt as it is: `all_or_none` is not part of the payload the receipt matches.""" pass @abstractmethod @@ -290,6 +293,12 @@ def get_queue_item_ids( matching items only when no limit is given.""" pass + @abstractmethod + def get_queue_item_lineage(self, queue_id: str, origin_prefix: str) -> list[QueueItemLineage]: + """The status, origin and retry source of every item whose origin starts with `origin_prefix` (ignoring + case), in no order.""" + pass + @abstractmethod def get_queue_item_summaries_by_ids(self, queue_id: str, item_ids: list[int]) -> list[SessionQueueItemSummary]: """Gets lightweight queue item summaries in the requested item ID order.""" diff --git a/invokeai/app/services/session_queue/session_queue_common.py b/invokeai/app/services/session_queue/session_queue_common.py index 31fc0ada27c..458f043af4d 100644 --- a/invokeai/app/services/session_queue/session_queue_common.py +++ b/invokeai/app/services/session_queue/session_queue_common.py @@ -216,6 +216,15 @@ def validate_graph(cls, v: Graph): QUEUE_ITEM_STATUS = Literal["pending", "in_progress", "waiting", "completed", "failed", "canceled"] +class QueueItemLineage(NamedTuple): + """An item's status and origin, and the item it retried: a retry of a retry names the first item as well.""" + + item_id: int + status: QUEUE_ITEM_STATUS + origin: Optional[str] + retried_from_item_id: Optional[int] + + class ItemIdsResult(BaseModel): """Response containing ordered item ids with metadata for optimistic updates.""" diff --git a/invokeai/app/services/session_queue/session_queue_default.py b/invokeai/app/services/session_queue/session_queue_default.py index 01df20c0052..33f8968838b 100644 --- a/invokeai/app/services/session_queue/session_queue_default.py +++ b/invokeai/app/services/session_queue/session_queue_default.py @@ -3,7 +3,7 @@ import json import math from datetime import datetime, timedelta, timezone -from typing import Any, Literal, Optional +from typing import Any, Literal, Optional, cast from pydantic_core import to_jsonable_python @@ -32,6 +32,7 @@ NodeFieldValue, PruneResult, QueueItemDeliveryResult, + QueueItemLineage, QueueItemOwner, RetriedQueueItem, RetryItemsResult, @@ -144,11 +145,11 @@ def _prune_terminal_to_limit(self, queue_id: str, keep: int) -> int: return self._queries.session_queue.prune(queue_id, user_id=None, keep=keep) async def enqueue_batch( - self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system" + self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system", *, all_or_none: bool = False ) -> EnqueueBatchResult: # The route awaits this method, but every operation below is synchronous database/CPU work. Keep the # complete enqueue in one worker operation, off the event loop. - return await asyncio.to_thread(self._enqueue_batch, queue_id, batch, prepend, user_id) + return await asyncio.to_thread(self._enqueue_batch, queue_id, batch, prepend, user_id, all_or_none) def acknowledge_enqueue(self, queue_id: str, idempotency_key: str, user_id: str = "system") -> None: self._queries.session_queue.acknowledge_receipt(queue_id, user_id, idempotency_key) @@ -159,7 +160,9 @@ def get_enqueue_receipt( receipt = self._queries.session_queue.enqueue_receipt(queue_id, user_id, idempotency_key) return _decode_enqueue_receipt(*receipt) if receipt is not None else None - def _enqueue_batch(self, queue_id: str, batch: Batch, prepend: bool, user_id: str) -> EnqueueBatchResult: + def _enqueue_batch( + self, queue_id: str, batch: Batch, prepend: bool, user_id: str, all_or_none: bool = False + ) -> EnqueueBatchResult: requested_count = calc_session_count(batch=batch) payload_hash = ( hashlib.sha256( @@ -208,11 +211,12 @@ def survey(q: Queries) -> SettledReceipt | int: # Preparing the values is CPU work, done outside a transaction; admission below takes what still fits. max_queue_size = self.__invoker.services.configuration.max_queue_size + room = max(0, max_queue_size - surveyed) prepared_values = prepare_values_to_insert( queue_id=queue_id, batch=batch, priority=0, - max_new_queue_items=max(0, max_queue_size - surveyed), + max_new_queue_items=0 if all_or_none and requested_count > room else room, user_id=user_id, ) @@ -225,6 +229,9 @@ def admit(q: Queries) -> SettledReceipt | EnqueueBatchResult: require_project(q) max_new_queue_items = max(0, max_queue_size - q.session_queue.pending_count(queue_id)) + # Checked under the admission lock: the room the survey saw may have been taken meanwhile. + if all_or_none and min(len(prepared_values), max_new_queue_items) < requested_count: + max_new_queue_items = 0 priority = q.session_queue.top_pending_priority(queue_id) + 1 if prepend else 0 values_to_insert = prepared_values[:max_new_queue_items] if priority != 0: @@ -1037,6 +1044,14 @@ def get_queue_item_ids( def has_active_queue_work(self) -> bool: return self._queries.session_queue.has_active_work() + def get_queue_item_lineage(self, queue_id: str, origin_prefix: str) -> list[QueueItemLineage]: + return [ + QueueItemLineage(item_id, cast(QUEUE_ITEM_STATUS, status), origin, retried_from_item_id) + for item_id, status, origin, retried_from_item_id in self._queries.session_queue.lineage( + queue_id, origin_prefix + ) + ] + def get_queue_item_summaries_by_ids(self, queue_id: str, item_ids: list[int]) -> list[SessionQueueItemSummary]: if not item_ids: return [] diff --git a/invokeai/app/services/shared/database/queries/session_queue.py b/invokeai/app/services/shared/database/queries/session_queue.py index 120ee0c38af..0e8056d09aa 100644 --- a/invokeai/app/services/shared/database/queries/session_queue.py +++ b/invokeai/app/services/shared/database/queries/session_queue.py @@ -114,6 +114,10 @@ def _origin_pattern(origin_prefix: Optional[str]) -> Optional[str]: return like_prefix(origin_prefix) if origin_prefix is not None else None +# The origin is tested on the listing index; only the matching items' rows are read. +_LINEAGE = select(_Q.item_id, _Q.status, _Q.origin, _Q.retried_from_item_id).where(_IN_QUEUE, _origin_matches()) + + @functools.cache def _item_by_status(status: str, by_origin: bool, by_owner: bool) -> Select[Any]: """The pending item to run next, or an item in progress (the earliest enqueued of several).""" @@ -249,6 +253,10 @@ def _summaries(rows: Sequence[Sequence[Any]]) -> list[dict[str, Any]]: return [dict(zip(_SUMMARY_NAMES, row, strict=True)) for row in rows] +def _lineage(rows: Sequence[Sequence[Any]]) -> list[tuple[int, str, Optional[str], Optional[int]]]: + return [(row[0], row[1], row[2], row[3]) for row in rows] + + def _counts(rows: Sequence[Sequence[Any]]) -> dict[str, int]: return {str(row[0]): int(row[1]) for row in rows} @@ -841,6 +849,12 @@ def item_ids( } return list(conn.execute(statement, parameters).scalars().all()) + @mapped(_lineage) + @read + def lineage(self, conn: Connection, queue_id: str, origin_prefix: str) -> Sequence[Row[Any]]: + """Id, status, origin and retry source of the queue's items with the origin, in no order.""" + return conn.execute(_LINEAGE, {"queue_id": queue_id, "origin_pattern": like_prefix(origin_prefix)}).all() + @mapped(_summaries) @read def summaries(self, conn: Connection, queue_id: str, item_ids: Sequence[int]) -> Sequence[Row[Any]]: diff --git a/invokeai/app/services/working_memory_calibration/__init__.py b/invokeai/app/services/working_memory_calibration/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_base.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_base.py new file mode 100644 index 00000000000..28f909883b0 --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_base.py @@ -0,0 +1,93 @@ +from abc import ABC, abstractmethod +from collections.abc import Mapping +from typing import Optional + +from invokeai.app.services.session_queue.session_queue_common import SessionQueueItem +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + WorkingMemoryCalibrationModelSummary, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationRequest, + WorkingMemoryCalibrationResetResult, + WorkingMemoryCalibrationRunCreated, + WorkingMemoryCalibrationStatus, +) +from invokeai.backend.working_memory.probe import MeasuredWindow + + +class WorkingMemoryCalibrationServiceBase(ABC): + """Calibration runs: enqueues them, admits their queue items, and keeps their measurements per device class.""" + + @abstractmethod + async def create_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str + ) -> WorkingMemoryCalibrationRunCreated: + """Validate the request, record the run and enqueue its batch as `user_id`, who must be an administrator. + + The queue takes all of the batch's items or none (`CalibrationQueueFullError`). With a retry key, a repeat of + the request returns the run it created and enqueues nothing; once a reset removed the run, it raises + `CalibrationRunNotFoundError`.""" + + @abstractmethod + def admit(self, queue_item: SessionQueueItem, run_id: str) -> Optional[str]: + """Called by the session runner before a calibration item runs: None to run it, or why it must fail.""" + + @abstractmethod + def model_reset_epochs(self, run_id: str) -> dict[str, int]: + """The reset epoch of every model reset on the run's device class; empty when the run does not exist. Read + when a node starts and passed to `record_windows`.""" + + @abstractmethod + def record_windows( + self, run_id: str, windows: list[MeasuredWindow], model_reset_epochs: Optional[Mapping[str, int]] = None + ) -> None: + """Persist one node's measured windows. Windows of a run that no longer exists are dropped, and so are those + of a model reset since `model_reset_epochs` was read (by default, at this call): the reset removed the + model's points, these among them. A reset by another process sharing the root counts as well.""" + + @abstractmethod + def observe(self, queue_item: SessionQueueItem, run_id: str, error: Optional[BaseException] = None) -> None: + """Record the item's queue status in its run, after the session runner is done with it. `error` is what the + runner raised, which fails an item the queue still has as running; a removed item counts as canceled.""" + + @abstractmethod + def get_status(self) -> WorkingMemoryCalibrationStatus: + """Every device class's runs, with derived status, point counts and environments.""" + + @abstractmethod + def reset_run(self, run_id: str) -> WorkingMemoryCalibrationResetResult: + """Remove a run and its points. Every reset keeps the ids of the runs it removed: a repeat of the request that + started one cannot start it again.""" + + @abstractmethod + def reset_model(self, model_key: str, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + """Remove a model's runs and points from one device class, the current one by default.""" + + @abstractmethod + def reset_all(self, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + """Remove every run and point of one device class, the current one by default.""" + + @abstractmethod + def export(self, device_class_id: Optional[str] = None) -> Optional[bytes]: + """The device class's store file as it is on disk, the current one by default; None when it has none.""" + + @abstractmethod + def refresh_user_fits(self) -> None: + """Bring the user fits estimates read (`backend/working_memory/user_fit.py`) up to date with the store: per + GPU generation device, the fit of its device class's windows in the current environment, leaving out the + windows of calibration runs still running. Re-reads a file only when its signature or the environment changed, + so an unchanged store whose runs all finished costs one stat per device; while a run is unfinished, also one + queue read of its unfinished items by id. The session runner calls this when it starts any queue item; + building a plan and a reset call it too. Estimates read the fits from memory.""" + + @abstractmethod + def get_plan(self, model_key: str) -> WorkingMemoryCalibrationPlan: + """The cases that calibrate the model on this machine's GPU, or why it cannot be calibrated here. Raises + UnknownModelException for an unknown key.""" + + @abstractmethod + def get_model_summary( + self, model_key: str, device_class_id: Optional[str] = None + ) -> WorkingMemoryCalibrationModelSummary: + """What the windows that feed the model's lines measured on one device class, the current one by default: per + line and environment, the shipped and the user estimate at each size, whether the user line applies, and which + models measured it; and the model's runs. Raises UnknownModelException for an unknown key.""" diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_common.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_common.py new file mode 100644 index 00000000000..637da8c2ecf --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_common.py @@ -0,0 +1,496 @@ +"""Working-memory calibration runs: what is stored per device class, and what the API returns about it. + +A calibration run is a batch an administrator enqueues to measure one model's working memory. Its queue items carry the +reserved origin `CALIBRATION_ORIGIN_PREFIX` + run id; the session runner measures every node of such an item with the +working-memory probe and hands the windows to the calibration service, which keeps them in one JSON file per device +class (`CalibrationStoreFile`). The service fits the windows of the current environment into user lines that estimates +read (`backend/working_memory/user_fit.py`), plans the cases a client runs to calibrate a model, and summarises what a +model's runs measured. +""" + +import datetime +from typing import Final, Literal, Optional + +from pydantic import BaseModel, ConfigDict, Field, SerializerFunctionWrapHandler, model_serializer, model_validator + +from invokeai.app.services.session_queue.session_queue_common import QUEUE_ITEM_STATUS, BatchDataCollection +from invokeai.app.services.shared.graph import Graph +from invokeai.backend.working_memory.counters import Backend +from invokeai.backend.working_memory.fingerprint import DeviceClass, Environment +from invokeai.backend.working_memory.probe import MeasuredWindow + +CALIBRATION_ORIGIN_PREFIX: Final = "invokeai:wm-calibration:" +"""Queue items whose origin starts with this are calibration items. Only the calibration service may enqueue them.""" +CALIBRATION_DESTINATION: Final = "invokeai:wm-calibration" +"""The destination of calibration items: not a gallery or canvas destination, so no client routes their results.""" +STORE_SCHEMA_VERSION: Final = 1 + + +def calibration_origin(run_id: str) -> str: + return CALIBRATION_ORIGIN_PREFIX + run_id + + +def calibration_run_id(origin: Optional[str]) -> Optional[str]: + """The run id a queue item's origin names, or None for an item that is not a calibration item.""" + if origin is None or not origin.startswith(CALIBRATION_ORIGIN_PREFIX): + return None + return origin[len(CALIBRATION_ORIGIN_PREFIX) :] + + +def is_reserved_origin(origin: Optional[str]) -> bool: + """Whether a client-supplied origin claims the calibration prefix. Case-insensitive, because the queue's origin + filters are: a look-alike origin would otherwise be counted among a run's items.""" + return origin is not None and origin.casefold().startswith(CALIBRATION_ORIGIN_PREFIX) + + +# region Store +# +# One policy for every store model: unknown fields make the file unusable (`extra="forbid"`). Ignoring them instead +# would drop them on the next rewrite, destroying data a newer writer put there. + + +class _StoreModel(BaseModel): + model_config = ConfigDict(extra="forbid") + + +class CalibrationDeviceClass(_StoreModel): + """A `fingerprint.DeviceClass` with its id as a plain field, as it is stored and returned.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + id: str = Field(description="A short stable hash of the other fields") + vendor: str + name: str + total_mib: int + backend: Backend + arch: Optional[str] + + @classmethod + def of(cls, device_class: DeviceClass) -> "CalibrationDeviceClass": + return cls(id=device_class.id, **device_class.model_dump(exclude={"id"})) + + @model_validator(mode="after") + def _id_matches(self) -> "CalibrationDeviceClass": + if DeviceClass(**self.model_dump(exclude={"id"})).id != self.id: + raise ValueError(f"Device class id {self.id} does not match its fields") + return self + + +class CalibrationEnvironment(_StoreModel): + """A `fingerprint.Environment` with its id as a plain field.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + id: str = Field(description="A short stable hash of the hard facts") + hard: dict[str, str] + soft: dict[str, Optional[str]] + + @classmethod + def of(cls, env: Environment) -> "CalibrationEnvironment": + return cls(id=env.id, hard=env.hard, soft=env.soft) + + @model_validator(mode="after") + def _id_matches(self) -> "CalibrationEnvironment": + if Environment(hard=self.hard, soft=self.soft).id != self.id: + raise ValueError(f"Environment id {self.id} does not match its hard facts") + return self + + +class CalibrationPoint(MeasuredWindow): + """One measured window, keyed to the environment it ran under. Forbids unknown fields, as `MeasuredWindow` does.""" + + env_id: str + + +class ObservedItem(_StoreModel): + """What the session runner saw of one of the run's queue items. Kept so that a run's status survives the queue + pruning its items.""" + + status: QUEUE_ITEM_STATUS + error_message: Optional[str] = None + retried_from_item_id: Optional[int] = None + observed_at: datetime.datetime + + +class CalibrationRun(_StoreModel): + run_id: str + model_key: str + model_hash: Optional[str] = Field(description="The model's hash when the run was created; provenance only") + user_id: str + created_at: datetime.datetime + env_id: str = Field(description="The environment the run was created under") + queue_item_ids: list[int] = Field(default_factory=list, description="The items the run was enqueued with") + observed: dict[int, ObservedItem] = Field( + default_factory=dict, description="The last status the session runner saw per item, retries included" + ) + attempt_id: Optional[str] = Field(default=None, description="The client's attempt the run belongs to, if any") + soft: Optional[dict[str, Optional[str]]] = Field( + default=None, + description="The soft facts of the environment the run was created under; absent in runs stored before they " + "were recorded, which fall back to the environment's", + ) + + @model_serializer(mode="wrap") + def _omit_absent_fields(self, handler: SerializerFunctionWrapHandler) -> dict: + # Written only when set: a run without them stays readable by a version that forbids the unknown fields. + data = handler(self) + for name in ("attempt_id", "soft"): + if data.get(name) is None: + data.pop(name, None) + return data + + +class CalibrationStoreFile(_StoreModel): + """The content of `/working_memory/.json`, which the export returns as it is.""" + + schema_version: Literal[1] + device_class: CalibrationDeviceClass + runs: list[CalibrationRun] = Field(default_factory=list) + points: list[CalibrationPoint] = Field(default_factory=list) + envs: list[CalibrationEnvironment] = Field(default_factory=list) + removed_run_ids: list[str] = Field( + default_factory=list, + description="The runs with a retry key that a reset removed, kept for good. A repeat of the request that " + "started one cannot start it again, so its queue items keep failing at admission", + ) + model_reset_epochs: dict[str, int] = Field( + default_factory=dict, + description="Per model key, how often the model was reset. A node's points of a model reset after the node " + "started are dropped", + ) + + +# endregion + +# region API + +ATTEMPT_ID_DESCRIPTION: Final = ( + "A client-generated id of the calibration attempt the run belongs to (1-64 letters, digits, '_', '-', '.', ':'), " + "so a client that enqueues one run per step can group them; no window of an attempt feeds an estimate until every " + "run of the attempt has finished. The run id derives from the idempotency_key, the request and the attempt, so an " + "idempotency_key names one run: repeating the request under the same attempt returns that run, and reusing the key " + "under another attempt is refused with 409 while the queue keeps the key's receipt. Make each key unique per run " + "of an attempt." +) +MEASUREMENT_METHOD_DESCRIPTION: Final = ( + "Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this " + "platform has no driver reader; null when no generation device of this class is present" +) +MEASUREMENT_VERIFIED_DESCRIPTION: Final = ( + "Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when " + "no generation device of this class is present" +) + + +class WorkingMemoryCalibrationRequest(BaseModel): + """The batch a calibration run enqueues, built by the client. The server sets origin, destination, runs and the + graph's id, and turns the node cache off.""" + + model_config = ConfigDict(extra="forbid") + + graph: Graph = Field(description="The graph to measure") + data: Optional[BatchDataCollection] = Field(default=None, description="The batch data collection") + idempotency_key: Optional[str] = Field( + default=None, + min_length=1, + max_length=255, + description="A caller-owned retry key. Repeating the request with it returns the run it created and enqueues " + "nothing new; once a reset removed that run, the repeat is refused. Reusing the key for another request " + "(another model, graph or batch data) is rejected while the queue keeps the key's enqueue receipt.", + ) + attempt_id: Optional[str] = Field( + default=None, + min_length=1, + max_length=64, + pattern=r"^[A-Za-z0-9_.:-]+$", + description=ATTEMPT_ID_DESCRIPTION, + ) + + +class WorkingMemoryCalibrationRunCreated(BaseModel): + run_id: str + device_class_id: str + item_ids: list[int] = Field( + description="The queue items the run enqueued. A repeat of the request that finds them unrecorded takes over " + "every queue item carrying the run's origin, retries included" + ) + + +CalibrationRunStatus = Literal["running", "completed", "failed", "partial", "canceled"] + + +class WorkingMemoryCalibrationRunSummary(BaseModel): + run_id: str + model_key: str + model_hash: Optional[str] + user_id: str + created_at: datetime.datetime + env_id: str + status: CalibrationRunStatus = Field( + description="Derived from the run's queue items, a retry standing in for the item it retried: running while " + "any is pending or running; failed when any failed; completed when all completed; otherwise partial when " + "valid points were recorded, else canceled" + ) + error: Optional[str] = Field(description="The error of a failed item, when the run failed") + item_ids: list[int] = Field(description="Every queue item of the run, retries included") + point_count: int + attempt_id: Optional[str] = Field(description="The client's attempt the run belongs to; null when none was sent") + + +class WorkingMemoryCalibrationPointCount(BaseModel): + estimator: str + version: int + valid: bool + fit_eligible: bool + count: int + + +class WorkingMemoryCalibrationEnvironment(BaseModel): + id: str + hard: dict[str, str] + soft: dict[str, Optional[str]] + hard_matches_current: Optional[bool] = Field( + description="Whether the current environment has the same hard facts; null off the current device class" + ) + soft_matches_current: Optional[bool] = Field( + description="Whether the current environment has the soft facts first recorded with this environment (`soft`); " + "a later run's own soft facts are not compared here. Null off the current device class" + ) + point_count: int + + +class WorkingMemoryCalibrationDeviceClass(BaseModel): + device_class_id: str + device_class: Optional[CalibrationDeviceClass] = Field(description="Null when the file is unusable") + usable: bool + problem: Optional[str] = Field(description="Why the file is unusable. It is kept as it is and never written") + is_current: bool + measurement_method: Optional[str] = Field(description=MEASUREMENT_METHOD_DESCRIPTION) + measurement_verified: Optional[bool] = Field(description=MEASUREMENT_VERIFIED_DESCRIPTION) + runs: list[WorkingMemoryCalibrationRunSummary] + point_counts: list[WorkingMemoryCalibrationPointCount] + envs: list[WorkingMemoryCalibrationEnvironment] + + +class WorkingMemoryCalibrationStatus(BaseModel): + available: bool = Field( + description="Whether a calibration run can start now: one GPU generation device whose store file is usable" + ) + unavailable_reason: Optional[str] + current_device_class_id: Optional[str] + current_env_id: Optional[str] + device_classes: list[WorkingMemoryCalibrationDeviceClass] + + +class WorkingMemoryCalibrationResetResult(BaseModel): + runs_removed: int + points_removed: int + + +class WorkingMemoryCalibrationDeviceClassRef(BaseModel): + id: str + name: str + + +class WorkingMemoryCalibrationCase(BaseModel): + """One generation a calibration plan runs, in the fields of the client's graph-export case (mode txt2img).""" + + id: str = Field(description="Unique within the plan; becomes the graph id") + mode: Literal["txt2img"] = "txt2img" + model_key: str = Field(description="The main model the case generates with") + components: dict[str, str] = Field( + description="Component picks by Generate slot key; `vae` when a standalone VAE is calibrated or a GPU VAE " + "must be picked over one that runs on the CPU" + ) + width: int + height: int + steps: int = Field(description="The fewest steps that still run the whole pipeline") + seed: int + positive_prompt: str + negative_prompt: str + settings: dict[str, str] = Field( + description="Generate settings overrides: `vaePrecision` (fp16 or fp32) where the decode node casts the VAE" + ) + estimator: str = Field(description="The working-memory estimator of the operation the case measures") + line: str = Field(description="The estimator line the case measures, `_`") + + +class WorkingMemoryCalibrationRepetitions(BaseModel): + warm_up: int = Field(description="Runs of each case before the measured ones; their windows count too") + measured: int + + +class WorkingMemoryCalibrationSharedModel(BaseModel): + key: str + name: str + base: str + type: str + lines: list[str] = Field(description="The lines it shares with the planned model, `_`") + + +class WorkingMemoryCalibrationPlan(BaseModel): + """What calibrating one model on this machine's GPU runs, or why it cannot.""" + + model_key: str + supported: bool + reason: Optional[str] = Field(description="Why the model cannot be calibrated here, when it cannot") + device_class: Optional[WorkingMemoryCalibrationDeviceClassRef] = Field( + description="The class of the GPU the cases run on; null without exactly one GPU generation device" + ) + calibrated_through: Optional[str] = Field( + description="For a standalone VAE, the installed main model the cases generate with" + ) + estimators: list[str] = Field(description="The working-memory estimators of the model's covered operations") + cases: list[WorkingMemoryCalibrationCase] + repetitions: WorkingMemoryCalibrationRepetitions + estimated_seconds: Optional[int] = Field( + description="A rough order of magnitude for all cases and repetitions, model load included; not a measurement" + ) + shared_with: list[WorkingMemoryCalibrationSharedModel] = Field( + description="Other installed models whose operations price with the same lines on this GPU, so a calibration " + "changes their estimates too" + ) + not_covered: list[str] = Field(description="Operations of the model a calibration does not measure, and why") + + +CalibrationLineState = Literal["in_use", "stale", "unusable", "insufficient"] + +CalibrationSizeExclusion = Literal["fragmented", "partial_residency"] + + +class WorkingMemoryCalibrationLineSize(BaseModel): + width: int + height: int + pixels: int + windows: int = Field( + description="Counted: the windows at this size that feed the line, valid, fit-eligible and not fragmented, of " + "every model in the environment; those of a calibration run still running are counted, and join the fit once " + "it finished. Not counted: the valid windows measured at this size" + ) + measured_bytes: int = Field(description="The largest demand of those windows") + counted: bool = Field(description="Whether windows of this size feed the user line") + excluded_reason: Optional[CalibrationSizeExclusion] = Field( + description="Why a size is not counted: fragmented, its windows measured the allocator's fragmentation " + "(`backend/working_memory/fragmentation.py`); partial_residency, the model was not fully on the device. Null " + "for a counted size" + ) + shipped_bytes: Optional[int] = Field( + description="The shipped estimate at this size, priced with the line's estimator where one of its windows " + "measured the size, otherwise with that of the largest window; null when the platform of the environment is " + "unknown" + ) + user_bytes: Optional[int] = Field( + description="The estimate with the user line applied, as estimates read it (without the windows of runs still " + "running); null without a user line" + ) + + +class WorkingMemoryCalibrationPixelRange(BaseModel): + min_pixels: int + max_pixels: int + + +class WorkingMemoryCalibrationLineContributor(BaseModel): + """A model whose windows feed a line: the model a run calibrated, or the model a window measured.""" + + model_key: str + name: Optional[str] = Field(description="The model's name; null when it is no longer installed") + windows: int = Field(description="Its counted windows on the line: valid, fit-eligible and not fragmented") + + +class WorkingMemoryCalibrationModelLine(BaseModel): + """One estimator line in one environment: one the model's windows measured, or, in the current environment, one + the model's decode prices on that other models' windows measured.""" + + estimator: str + version: int + line: str = Field(description="`_`") + carrier: Optional[str] = Field(description="The estimator whose shipped entry carries the line on this platform") + env_id: str + state: CalibrationLineState = Field( + description="in_use: the user line applies; stale: it applies, but the soft facts of the newest finished run " + "that contributed a window differ from the current ones; unusable: hard facts " + "differ from the current environment (or no GPU of this class generates here), so shipped values apply; " + "insufficient: the windows of finished runs cannot fit a line yet" + ) + changed_facts: list[str] = Field( + description="The soft facts that changed (stale) or the hard facts that differ (unusable)" + ) + problem: Optional[str] = Field(description="Why no line could be fitted, when it could not") + measured_range: Optional[WorkingMemoryCalibrationPixelRange] = Field( + description="The sizes the user line was fitted on, over every model's windows of the line from finished runs" + ) + slope_bytes_per_pixel: Optional[int] + base_bytes: Optional[int] + sizes: list[WorkingMemoryCalibrationLineSize] + own_windows: int = Field(description="The model's own windows that feed the line; 0 when only other models' do") + measured_by: list[WorkingMemoryCalibrationLineContributor] = Field( + description="Every model whose windows feed the line, this one included when it has any; by name, then key" + ) + + +class WorkingMemoryCalibrationModelRun(BaseModel): + run_id: str + status: CalibrationRunStatus + error: Optional[str] + created_at: datetime.datetime + point_count: int + attempt_id: Optional[str] = Field(description="The client's attempt the run belongs to; null when none was sent") + + +class WorkingMemoryCalibrationModelSummary(BaseModel): + """What one model's calibration runs measured on one device class, and what estimates make of it.""" + + model_key: str + device_class_id: str + usable: bool + problem: Optional[str] = Field(description="Why the store file is unusable") + current_env_id: Optional[str] = Field(description="Null when no generation device of this class is present") + measurement_method: Optional[str] = Field(description=MEASUREMENT_METHOD_DESCRIPTION) + measurement_verified: Optional[bool] = Field(description=MEASUREMENT_VERIFIED_DESCRIPTION) + lines: list[WorkingMemoryCalibrationModelLine] + excluded_windows: int = Field( + description="The model's windows that feed no line: invalid, not fit-eligible, fragmented, tiled, batched, " + "upcast, a VAE outside the AutoencoderKL family, no demand, or of an estimator a user calibration does not " + "move, such as the Z-Image denoise. Of these, only the not fit-eligible and the fragmented windows show among " + "a line's sizes, as a size that is not counted where no counted window measured it" + ) + runs: list[WorkingMemoryCalibrationModelRun] + + +# endregion + +# region Errors + + +class CalibrationUnavailableError(Exception): + """This machine cannot calibrate now: no GPU, or more than one generation device.""" + + +class CalibrationGraphError(ValueError): + """The graph cannot be measured as a calibration run.""" + + +class CalibrationRunNotFoundError(KeyError): + def __init__(self, run_id: str, reason: str = "does not exist") -> None: + super().__init__(run_id) + self.run_id = run_id + self.reason = reason + + def __str__(self) -> str: + return f"Calibration run {self.run_id} {self.reason}" + + +class CalibrationQueueFullError(Exception): + """The queue had no room for all of the run's items, so it took none.""" + + +class CalibrationStoreError(Exception): + """The store could not be read or written.""" + + +class CalibrationStoreUnusableError(CalibrationStoreError): + """The device class's file has an unknown schema or is invalid; it is kept untouched and not written.""" + + +# endregion diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_default.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_default.py new file mode 100644 index 00000000000..d0916f6a1cb --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_default.py @@ -0,0 +1,1267 @@ +import asyncio +import datetime +import hashlib +import json +import os +import threading +import uuid +import weakref +from collections.abc import Callable, Mapping +from dataclasses import dataclass, replace +from functools import partial +from itertools import chain +from typing import TYPE_CHECKING, Final, Optional + +import torch + +from invokeai.app.invocations.call_saved_workflow import CallSavedWorkflowInvocation +from invokeai.app.services.model_records import UnknownModelException +from invokeai.app.services.session_queue.session_queue_common import ( + DEFAULT_QUEUE_ID, + QUEUE_ITEM_STATUS, + Batch, + QueueItemLineage, + SessionQueueItem, + SessionQueueItemNotFoundError, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_base import ( + WorkingMemoryCalibrationServiceBase, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CALIBRATION_DESTINATION, + CALIBRATION_ORIGIN_PREFIX, + STORE_SCHEMA_VERSION, + CalibrationDeviceClass, + CalibrationEnvironment, + CalibrationGraphError, + CalibrationPoint, + CalibrationQueueFullError, + CalibrationRun, + CalibrationRunNotFoundError, + CalibrationRunStatus, + CalibrationStoreError, + CalibrationStoreFile, + CalibrationStoreUnusableError, + CalibrationUnavailableError, + ObservedItem, + WorkingMemoryCalibrationDeviceClass, + WorkingMemoryCalibrationEnvironment, + WorkingMemoryCalibrationModelRun, + WorkingMemoryCalibrationModelSummary, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationPointCount, + WorkingMemoryCalibrationRequest, + WorkingMemoryCalibrationResetResult, + WorkingMemoryCalibrationRunCreated, + WorkingMemoryCalibrationRunSummary, + WorkingMemoryCalibrationStatus, + calibration_origin, + calibration_run_id, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_fits import ( + measured_models, + model_lines, + recorded_outside_torch_bytes, + run_models, + user_fit_of, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_plan import ( + build_plan, + decode_lines_of, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_store import ( + CalibrationStore, + StoreRead, +) +from invokeai.app.util.misc import uuid_string +from invokeai.backend.util.devices import TorchDevice +from invokeai.backend.util.vae_working_memory import estimate_untiled_vae_decode +from invokeai.backend.working_memory import fingerprint, user_fit +from invokeai.backend.working_memory.counters import Backend, backend_for, driver_method, method_verified, read_counters +from invokeai.backend.working_memory.fingerprint import DeviceClass, Environment +from invokeai.backend.working_memory.platform import Platform, platform_for, platform_for_environment +from invokeai.backend.working_memory.probe import MeasuredWindow +from invokeai.backend.working_memory.spec import EstimatorSpec, import_estimator_modules +from invokeai.backend.working_memory.user_fit import UserFit + +if TYPE_CHECKING: + from invokeai.app.services.invocation_services import InvocationServices + from invokeai.app.services.invoker import Invoker + +MEMORY_SETTINGS: Final = ( + "pytorch_cuda_alloc_conf", + "attention_backend", + "attention_type", + "attention_slice_size", + "sequential_guidance", + "pid_memory_optimization", + "wan_memory_optimization", + "rocm_aotriton_experimental", +) +"""The InvokeAI settings that change how much working memory an operation takes; hard facts of a measurement's +environment.""" + +_SUMMARY_CHUNK: Final = 500 +"""Queue items read per summary query when `_live_runs` asks about the unfinished items of the runs.""" + +_RUN_ID_NAMESPACE: Final = uuid.UUID("3c1f6a52-8d0e-4b6f-9a57-2f0d5c8e41b3") +"""Derives a run id from an owner, a retry key and the request's fingerprint.""" + +_RUNNING: Final[frozenset[QUEUE_ITEM_STATUS]] = frozenset({"pending", "in_progress", "waiting"}) + +_RESET: Final = "was reset; a repeat of the request that started it cannot start it again" + +SettingValue = str | int | float | bool | None + + +def _is_keyed(run_id: str) -> bool: + """Whether a retry key named the run: its id is then derived (a UUID 5, see `_identify`), otherwise fresh + (`uuid_string`, a UUID 4) and never repeated.""" + try: + return uuid.UUID(run_id).version == 5 + except ValueError: + return False + + +def _now() -> datetime.datetime: + return datetime.datetime.now(datetime.timezone.utc) + + +def _untiled_decode_reservation(estimator: str, height: int, width: int, dtype: str, device: torch.device) -> int: + return estimate_untiled_vae_decode(estimator, height, width, dtype, device).reservation_bytes + + +@dataclass(frozen=True) +class DeviceMemory: + """A calibration device's memory as the plan sizes for it, read now.""" + + process_bytes: int + """What this process can hold, models included, as `ModelCache._get_vram_available` counts it before its + reserves: what the driver has free plus what torch has allocated, without the cache's credit for the allocator's + reclaimable blocks.""" + outside_torch_bytes: Optional[int] + """What this process holds outside torch's allocator: the driver's figure for it less torch's reserve. None where + the driver has no figure for this process.""" + + +def _device_memory(device: torch.device) -> Optional[DeviceMemory]: + """`DeviceMemory` for `device`; None for a device that shares system memory (MPS).""" + if device.type == "cuda": + free, _ = TorchDevice.cuda_mem_get_info(device) + process_bytes = free + torch.cuda.memory_allocated(device) + elif device.type == "xpu": + free, _ = TorchDevice.xpu_mem_get_info(device) + process_bytes = free + torch.xpu.memory_allocated(device) + else: + return None + reading = read_counters(device) + outside = None if reading.driver is None or reading.reserved is None else reading.driver - reading.reserved + return DeviceMemory(process_bytes=process_bytes, outside_torch_bytes=outside) + + +def _measurement(backend: Optional[Backend]) -> tuple[Optional[str], Optional[bool]]: + """The driver reader a window of a `backend` GPU here uses, and whether it is verified; both None for a device + class no generation device here belongs to.""" + if backend is None: + return None, None + return driver_method(backend), method_verified(backend) + + +def _device_key(device: torch.device) -> str: + """One name per device, whether or not it carries its index: the cache's devices may come without one.""" + if device.index is not None or device.type == "cpu": + return str(device) + index = torch.cuda.current_device() if device.type == "cuda" and torch.cuda.is_available() else 0 + return f"{device.type}:{index}" + + +FileSignature = Optional[tuple[int, int, int]] +"""A store file's mtime, size and inode (file index on Windows), None when it has none. Every write replaces the file, +so a change shows in at least one of them.""" + + +@dataclass(frozen=True) +class _DeviceFit: + """What `refresh_user_fits` found for one generation device, and what it found it from.""" + + signature: FileSignature + env_id: str + platform: Platform + fit: Optional[UserFit] + file: Optional[CalibrationStoreFile] + """The file the fit was made from, kept to re-fit it when only the runs still running changed.""" + live_runs: frozenset[str] + """The runs whose windows the fit left out, because they were still running.""" + settled_items: frozenset[int] + """Calibration queue items found finished in the queue, which `_live_runs` asks about no more.""" + + +def _add_env(file: CalibrationStoreFile, env: Environment) -> None: + if all(known.id != env.id for known in file.envs): + file.envs.append(CalibrationEnvironment.of(env)) + + +def _prune_envs(file: CalibrationStoreFile) -> None: + used = {run.env_id for run in file.runs} | {point.env_id for point in file.points} + file.envs = [env for env in file.envs if env.id in used] + + +def _add_points( + file: CalibrationStoreFile, + *, + run_id: str, + env: Environment, + points: list[CalibrationPoint], + epochs: Mapping[str, int], +) -> tuple[bool, int]: + """Attach `points` to their run, except those attributed (`measured_models`, as a model reset removes them) to a + model whose reset epoch moved past `epochs`: a reset removed the model's points after the points were measured. + How many it attached; none when the file lacks the run.""" + if all(run.run_id != run_id for run in file.runs): + return False, 0 + current = file.model_reset_epochs + models = run_models(file) + kept = [ + point + for point in points + if all(current.get(model, 0) <= epochs.get(model, 0) for model in measured_models(point, models)) + ] + if not kept: + return False, 0 + _add_env(file, env) + file.points.extend(kept) + return True, len(kept) + + +def derive_run_status( + run: CalibrationRun, live: Mapping[int, QueueItemLineage], valid_point_count: int +) -> tuple[CalibrationRunStatus, Optional[str], list[int]]: + """A run's status, error and items, from its items: `live` as the queue has them now, the run's own observations + for items the queue no longer has. An item the queue no longer has counts as canceled unless the runner saw it + finish. A retry stands in for the item it retried, also before the runner started it. A run that ended without + completing is partial only with valid points: an invalid one (a canceled or failed operation, a spill) measured + nothing usable.""" + item_ids = sorted(set(run.queue_item_ids) | set(run.observed) | set(live)) + + def status_of(item_id: int) -> QUEUE_ITEM_STATUS: + if item_id in live: + return live[item_id].status + observed = run.observed.get(item_id) + # Gone from the queue while the runner last saw it running (a crash, a cleared queue): it will not finish. + if observed is None or observed.status in _RUNNING: + return "canceled" + return observed.status + + def retried_from(item_id: int) -> Optional[int]: + if item_id in live: + return live[item_id].retried_from_item_id + observed = run.observed.get(item_id) + return observed.retried_from_item_id if observed is not None else None + + latest: dict[int, int] = {} + for item_id in item_ids: + source = retried_from(item_id) + lineage = item_id if source is None else source + latest[lineage] = max(latest.get(lineage, item_id), item_id) + effective = sorted(latest.values()) + statuses = {item_id: status_of(item_id) for item_id in effective} + + if not effective: + return "canceled", None, item_ids + if any(status in _RUNNING for status in statuses.values()): + return "running", None, item_ids + failed = [item_id for item_id, status in statuses.items() if status == "failed"] + if failed: + observed = run.observed.get(failed[0]) + error = observed.error_message if observed is not None and observed.error_message else None + return "failed", error or f"Queue item {failed[0]} failed", item_ids + if all(status == "completed" for status in statuses.values()): + return "completed", None, item_ids + return ("partial" if valid_point_count > 0 else "canceled"), None, item_ids + + +class WorkingMemoryCalibrationService(WorkingMemoryCalibrationServiceBase): + """Calibration runs and their store. + + `generation_devices`, `device_class_for`, `environment_for`, `untiled_decode_reservation` and `device_memory` + default to the machine's own; tests pass their own because a device class and its memory can only be read from + real hardware. + """ + + def __init__( + self, + store: CalibrationStore, + *, + generation_devices: Optional[Callable[[], list[torch.device]]] = None, + device_class_for: Callable[[torch.device], DeviceClass] = fingerprint.device_class_for, + environment_for: Callable[ + [torch.device, Mapping[str, SettingValue]], Environment + ] = fingerprint.environment_for, + untiled_decode_reservation: Callable[[str, int, int, str, torch.device], int] = _untiled_decode_reservation, + device_memory: Callable[[torch.device], Optional[DeviceMemory]] = _device_memory, + ) -> None: + self._store = store + self._generation_devices = generation_devices + self._device_class_for = device_class_for + self._environment_for = environment_for + self._untiled_decode_reservation = untiled_decode_reservation + self._device_memory = device_memory + # A card's class cannot change while the process runs; reading it touches the driver. + self._device_classes: dict[str, DeviceClass] = {} + # Which file holds a run, so admitting an item and recording its windows do not scan every file. + self._run_files: dict[str, str] = {} + # Creations in flight, referenced here so that one whose request was canceled still runs to its end. + self._creations: set[asyncio.Task[WorkingMemoryCalibrationRunCreated]] = set() + # Per run id while a creation uses it: a repeat of a request waits for the attempt still in flight. + self._run_locks: weakref.WeakValueDictionary[str, asyncio.Lock] = weakref.WeakValueDictionary() + self._invoker: Optional["Invoker"] = None + # The user fit per generation device: replaced whole by `refresh_user_fits`, read by every estimate. + self._fits: Mapping[str, _DeviceFit] = {} + self._fits_lock = threading.Lock() + # One object, so that `stop` unregisters exactly what `start` registered. + self._provider: user_fit.UserFitProvider = self._user_fit_for + + def start(self, invoker: "Invoker") -> None: + self._invoker = invoker + user_fit.set_user_fit_provider(self._provider) + + def stop(self, invoker: "Invoker") -> None: + user_fit.clear_user_fit_provider(self._provider) + + @property + def _services(self) -> "InvocationServices": + assert self._invoker is not None, "The calibration service has not been started" + return self._invoker.services + + # region Devices and environments + + def _devices(self) -> list[torch.device]: + if self._generation_devices is not None: + return list(self._generation_devices()) + configured = TorchDevice.get_generation_devices(self._services.configuration.generation_devices) + return configured or [TorchDevice.choose_torch_device()] + + def _calibration_device(self) -> torch.device: + devices = self._devices() + if len(devices) > 1: + raise CalibrationUnavailableError( + f"Calibration runs on a single generation device, and this machine generates on {len(devices)} " + f"({', '.join(str(device) for device in devices)}). Set generation_devices to one GPU to calibrate it." + ) + if not devices or backend_for(devices[0]) is None: + raise CalibrationUnavailableError( + f"Calibration measures GPU memory, and this machine generates on {devices[0] if devices else 'no device'}." + ) + return devices[0] + + def _device_class(self, device: torch.device) -> DeviceClass: + key = str(device) + if key not in self._device_classes: + self._device_classes[key] = self._device_class_for(device) + return self._device_classes[key] + + def _environment(self, device: torch.device) -> Environment: + config = self._services.configuration + settings: dict[str, SettingValue] = {name: getattr(config, name) for name in MEMORY_SETTINGS} + return self._environment_for(device, settings) + + def _current_device_class_id(self, device_class_id: Optional[str]) -> str: + if device_class_id is not None: + return device_class_id + return self._device_class(self._calibration_device()).id + + # endregion + + # region Runs + + def _find_run(self, run_id: str) -> Optional[tuple[str, CalibrationRun]]: + cached = self._run_files.get(run_id) + candidates = [cached] if cached is not None else [] + candidates += [ + device_class_id for device_class_id in self._store.device_class_ids() if device_class_id != cached + ] + for device_class_id in candidates: + read = self._store.read(device_class_id) + if read is None or read.file is None: + continue + for run in read.file.runs: + if run.run_id == run_id: + self._run_files[run_id] = device_class_id + return device_class_id, run + self._run_files.pop(run_id, None) + return None + + @staticmethod + def _identify(model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str) -> str: + """Check the request's graph and turn its node cache off, then return the run's id, which also becomes the + graph's id. + + Without a retry key the id is fresh. With one it is derived from the owner, the key, the request's + fingerprint (the model and the batch the server builds: graph and batch data) and the client's attempt, if it + names one. A repeat of the request finds its run by that id, and another request reusing the key, another + attempt included, gets another id. The id is part of the batch's origin, so the queue's receipt for the key + refuses that other request. The client's graph id is not part of the request: a graph sent without one gets a + fresh id each time, so a repeat could never match. + """ + graph = request.graph + for node in graph.nodes.values(): + if isinstance(node, CallSavedWorkflowInvocation): + raise CalibrationGraphError( + f"Node {node.id} calls a saved workflow. A calibration graph cannot: the called workflow would " + "run as separate queue items, outside the measured node." + ) + for datum in chain.from_iterable(request.data or []): + if datum.field_name == "use_cache": + raise CalibrationGraphError( + f"Batch data sets use_cache on node {datum.node_path}; calibration runs every node uncached." + ) + # A cached output would skip the very operation the run measures. + for node in graph.nodes.values(): + node.use_cache = False + if request.idempotency_key is None: + run_id = uuid_string() + else: + fingerprint = { + "model_key": model_key, + "graph": graph.model_dump(mode="json", exclude={"id"}), + "data": None + if request.data is None + else [[datum.model_dump(mode="json") for datum in group] for group in request.data], + } + digest = hashlib.sha256( + json.dumps(fingerprint, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8") + ).hexdigest() + # Appended only with an attempt: a request without one keeps the id it had before attempts existed. + attempt = f"\x00{request.attempt_id}" if request.attempt_id is not None else "" + run_id = str(uuid.uuid5(_RUN_ID_NAMESPACE, f"{user_id}\x00{request.idempotency_key}\x00{digest}{attempt}")) + # The queue's receipt hashes the batch, graph id included: a repeat must build the same one. + graph.id = run_id + return run_id + + def _run_lock(self, run_id: str) -> asyncio.Lock: + lock = self._run_locks.get(run_id) + if lock is None: + lock = self._run_locks[run_id] = asyncio.Lock() + return lock + + async def create_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str + ) -> WorkingMemoryCalibrationRunCreated: + # The enqueue runs in a worker thread and can commit after the request was canceled. So the creation runs to + # its end whatever happens to the request: the run keeps the items that were queued, or is removed. Only the + # server's shutdown cancels the creation as well; the run stays as far as it got, for a keyed repeat to finish. + creation = asyncio.create_task(self._create_run(model_key, request, user_id)) + self._creations.add(creation) + creation.add_done_callback(self._creations.discard) + try: + return await asyncio.shield(creation) + except asyncio.CancelledError: + creation.add_done_callback(self._log_unawaited_failure) + raise + + def _log_unawaited_failure(self, creation: "asyncio.Task[WorkingMemoryCalibrationRunCreated]") -> None: + """Nobody awaits the creation of a canceled request: its failure would go unseen.""" + if not creation.cancelled() and (error := creation.exception()) is not None: + self._services.logger.warning( + "The creation of a calibration run whose request was canceled failed", exc_info=error + ) + + async def _create_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str + ) -> WorkingMemoryCalibrationRunCreated: + run_id = await asyncio.to_thread(self._identify, model_key, request, user_id) + # Recording the run, enqueuing its batch and recording its items or removing it: one step for each run id. The + # lock is this process's; two processes serving one root and its SQLite database are not kept apart. + async with self._run_lock(run_id): + prepared = await asyncio.to_thread(self._prepare_run, model_key, request, user_id, run_id) + if isinstance(prepared, WorkingMemoryCalibrationRunCreated): + return prepared + batch, device_class_id, created = prepared + try: + result = await self._services.session_queue.enqueue_batch( + queue_id=DEFAULT_QUEUE_ID, batch=batch, prepend=False, user_id=user_id, all_or_none=True + ) + except Exception: + # The queue announces a batch after committing it, so an enqueue can raise once its items are queued. + # Items of an earlier attempt that a reset left behind carry the run's origin too, and answer a repeat + # of the request as the key's receipt would. + try: + queued: Optional[list[int]] = await asyncio.to_thread(self._queued_item_ids, run_id) + except Exception: + self._services.logger.warning( + f"Could not tell whether calibration run {run_id} has queue items; it is kept", exc_info=True + ) + queued = None + if queued is None: + # The enqueue's own error answers the request. + raise + if not queued: + if created: + await asyncio.to_thread(self._discard, run_id) + raise + self._services.logger.warning( + f"Enqueuing calibration run {run_id} raised, and the queue holds its items {queued}", + exc_info=True, + ) + item_ids = queued + else: + if result.enqueued == 0: + if created: + await asyncio.to_thread(self._discard, run_id) + # A partial run would measure some of the batch's cases, so the queue takes all of them or none. + raise CalibrationQueueFullError( + f"The queue is full: it has no room for all {result.requested} items of the calibration run, " + "so the run was not started." + ) + item_ids = list(result.item_ids) + try: + await asyncio.to_thread(self._record_items, device_class_id, run_id, item_ids) + except CalibrationStoreError as error: + # An unusable file fails the run's items at admission, as a removed run (not caught here) does: an error + # with a key or without. With a key, a repeat records the items, from the key's receipt or, once the + # queue has collected that, from their origin. Without one the request cannot be repeated, only sent + # again, which would queue the batch twice: a passing store failure answers with the items. + if isinstance(error, CalibrationStoreUnusableError) or request.idempotency_key is not None: + raise + self._services.logger.warning( + f"Could not record the queue items {item_ids} of calibration run {run_id}; they run all the same, " + "and the status finds them by their origin", + exc_info=True, + ) + return WorkingMemoryCalibrationRunCreated(run_id=run_id, device_class_id=device_class_id, item_ids=item_ids) + + def _prepare_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str, run_id: str + ) -> WorkingMemoryCalibrationRunCreated | tuple[Batch, str, bool]: + """The run a repeated request already enqueued; or the batch to enqueue, the run's device class and whether + this call recorded the run.""" + key = request.idempotency_key + found: Optional[tuple[str, CalibrationRun]] = None + if key is not None and (found := self._find_run(run_id)) is not None: + device_class_id, run = found + if run.queue_item_ids: + return WorkingMemoryCalibrationRunCreated( + run_id=run_id, device_class_id=device_class_id, item_ids=list(run.queue_item_ids) + ) + # An earlier attempt may have queued the batch without recording its items. While the queue keeps the + # key's receipt, enqueuing the batch again answers with the items it names, all of them. That needs + # neither the model, which may have been removed since, nor the device. Without the receipt, enqueuing + # again would queue the batch a second time: the items carrying the run's origin, and those the runner + # saw, are taken over instead. + if self._services.session_queue.get_enqueue_receipt(DEFAULT_QUEUE_ID, key, user_id) is not None: + return self._batch(request, run_id), device_class_id, False + item_ids = sorted(set(self._queued_item_ids(run_id)) | set(run.observed)) + if item_ids: + self._record_items(device_class_id, run_id, item_ids) + return WorkingMemoryCalibrationRunCreated( + run_id=run_id, device_class_id=device_class_id, item_ids=item_ids + ) + elif key is not None and self._was_reset(run_id): + raise CalibrationRunNotFoundError(run_id, _RESET) + device = self._calibration_device() + # Raises UnknownModelException for an unknown key. + config = self._services.model_manager.store.get_model(model_key) + batch = self._batch(request, run_id) + if found is not None: + # The run is recorded without items, and nothing of it is queued: enqueue its batch. + return batch, found[0], False + device_class = self._device_class(device) + env = self._environment(device) + run = CalibrationRun( + run_id=run_id, + model_key=model_key, + model_hash=getattr(config, "hash", None), + user_id=user_id, + created_at=_now(), + env_id=env.id, + attempt_id=request.attempt_id, + soft=env.soft, + ) + + def add(file: CalibrationStoreFile) -> tuple[bool, bool]: + if run_id in file.removed_run_ids: + raise CalibrationRunNotFoundError(run_id, _RESET) + if any(known.run_id == run_id for known in file.runs): + return False, False + file.runs.append(run) + _add_env(file, env) + return True, True + + created = self._store.change(device_class.id, add, create=CalibrationDeviceClass.of(device_class)) + self._run_files[run_id] = device_class.id + return batch, device_class.id, bool(created) + + @staticmethod + def _batch(request: WorkingMemoryCalibrationRequest, run_id: str) -> Batch: + try: + return Batch( + graph=request.graph, + data=request.data, + runs=1, + origin=calibration_origin(run_id), + destination=CALIBRATION_DESTINATION, + idempotency_key=request.idempotency_key, + ) + except ValueError as error: + raise CalibrationGraphError(str(error)) from error + + def _was_reset(self, run_id: str) -> bool: + for device_class_id in self._store.device_class_ids(): + read = self._store.read(device_class_id) + if read is not None and read.file is not None and run_id in read.file.removed_run_ids: + return True + return False + + def _queued_item_ids(self, run_id: str) -> list[int]: + """The queue items carrying the run's origin: its batch's, with their retries and workflow-call children.""" + queue = self._services.session_queue + return sorted(queue.get_queue_item_ids(DEFAULT_QUEUE_ID, origin_prefix=calibration_origin(run_id)).item_ids) + + def _discard(self, run_id: str) -> None: + """Remove a run this request recorded and queued nothing for. Its key stays unused: a repeat may start it.""" + try: + self._remove_run(run_id, remember=False) + except Exception: + self._services.logger.warning(f"Could not discard calibration run {run_id}", exc_info=True) + + def _record_items(self, device_class_id: str, run_id: str, item_ids: list[int]) -> None: + def remember(file: CalibrationStoreFile) -> tuple[bool, bool]: + for run in file.runs: + if run.run_id == run_id: + changed = run.queue_item_ids != item_ids + run.queue_item_ids = list(item_ids) + return changed, True + return False, False + + if not self._store.change(device_class_id, remember): + # Reset while it was being created: its items fail at admission. + raise CalibrationRunNotFoundError(run_id) + + def admit(self, queue_item: SessionQueueItem, run_id: str) -> Optional[str]: + if self._services.configuration.multiuser: + owner = self._services.users.get(queue_item.user_id) + if owner is None or not owner.is_active or not owner.is_admin: + return ( + f"Calibration runs are for administrators only, and the owner of queue item {queue_item.item_id} " + "is not an active administrator." + ) + try: + device = self._calibration_device() + except CalibrationUnavailableError as error: + return str(error) + found = self._find_run(run_id) + if found is None: + return f"Calibration run {run_id} does not exist; it was reset or never created." + device_class_id, _ = found + if device_class_id != self._device_class(device).id: + return ( + f"Calibration run {run_id} was created for device class {device_class_id}, and this machine's GPU " + f"is of class {self._device_class(device).id}." + ) + self._observe( + run_id, + queue_item.item_id, + ObservedItem( + status="in_progress", retried_from_item_id=queue_item.retried_from_item_id, observed_at=_now() + ), + ) + return None + + def _observe(self, run_id: str, item_id: int, observation: ObservedItem) -> None: + found = self._find_run(run_id) + if found is None: + return + + def record(file: CalibrationStoreFile) -> tuple[bool, None]: + for run in file.runs: + if run.run_id == run_id: + previous = run.observed.get(item_id) + if observation.retried_from_item_id is None and previous is not None: + observation_ = observation.model_copy( + update={"retried_from_item_id": previous.retried_from_item_id} + ) + else: + observation_ = observation + run.observed[item_id] = observation_ + return True, None + return False, None + + self._store.change(found[0], record) + + def observe(self, queue_item: SessionQueueItem, run_id: str, error: Optional[BaseException] = None) -> None: + try: + current = self._services.session_queue.get_queue_item(queue_item.item_id) + except SessionQueueItemNotFoundError: + # Removed while it ran (Clear Queue, a deleted item): it will not finish. + self._observe( + run_id, + queue_item.item_id, + ObservedItem( + status="canceled", + error_message="The queue item was removed from the queue.", + retried_from_item_id=queue_item.retried_from_item_id, + observed_at=_now(), + ), + ) + return + status, error_message = current.status, current.error_message + if error is not None and status in _RUNNING: + # The processor fails the item after the runner raised; recorded here, a later prune cannot lose it. + status, error_message = "failed", f"{type(error).__name__}: {error}" + self._observe( + run_id, + current.item_id, + ObservedItem( + status=status, + error_message=error_message, + retried_from_item_id=current.retried_from_item_id, + observed_at=_now(), + ), + ) + + def model_reset_epochs(self, run_id: str) -> dict[str, int]: + found = self._find_run(run_id) + if found is None: + return {} + read = self._store.read(found[0]) + return dict(read.file.model_reset_epochs) if read is not None and read.file is not None else {} + + def record_windows( + self, run_id: str, windows: list[MeasuredWindow], model_reset_epochs: Optional[Mapping[str, int]] = None + ) -> None: + if model_reset_epochs is None: + model_reset_epochs = self.model_reset_epochs(run_id) + found = self._find_run(run_id) + if found is None: + self._services.logger.warning( + f"Dropped {len(windows)} working-memory measurements of calibration run {run_id}: the run was reset." + ) + return + run_class_id = found[0] + by_device: dict[str, list[MeasuredWindow]] = {} + for window in windows: + by_device.setdefault(window.device, []).append(window) + for device_name, group in by_device.items(): + device = torch.device(device_name) + device_class = self._device_class(device) + if device_class.id != run_class_id: + # Admission made the run's device class the generation device's; a window elsewhere is a model the + # node ran on another device, which this run does not calibrate. + self._services.logger.warning( + f"Dropped {len(group)} working-memory measurements of calibration run {run_id}: they ran on " + f"{device_name} (device class {device_class.id}), not on the run's device class {run_class_id}." + ) + continue + env = self._environment(device) + points = [CalibrationPoint(**window.model_dump(), env_id=env.id) for window in group] + add = partial(_add_points, run_id=run_id, env=env, points=points, epochs=model_reset_epochs) + dropped = len(points) - (self._store.change(run_class_id, add) or 0) + if dropped: + self._services.logger.warning( + f"Dropped {dropped} working-memory measurements of calibration run {run_id}: the run, or the " + "model they measured, was reset." + ) + + # endregion + + # region Status + + def _live_items(self) -> dict[str, dict[int, QueueItemLineage]]: + """The queue's calibration items, per run, as the queue has them now.""" + live: dict[str, dict[int, QueueItemLineage]] = {} + for item in self._services.session_queue.get_queue_item_lineage(DEFAULT_QUEUE_ID, CALIBRATION_ORIGIN_PREFIX): + # The queue's origin filter ignores case; only the exact prefix is the server's. + run_id = calibration_run_id(item.origin) + if run_id is not None: + live.setdefault(run_id, {})[item.item_id] = item + return live + + def get_status(self) -> WorkingMemoryCalibrationStatus: + current_class: Optional[DeviceClass] = None + current_env: Optional[Environment] = None + unavailable: Optional[str] = None + try: + device = self._calibration_device() + current_class = self._device_class(device) + current_env = self._environment(device) + except CalibrationUnavailableError as error: + unavailable = str(error) + + live = self._live_items() + present = self._present_backends() + device_class_ids = set(self._store.device_class_ids()) + if current_class is not None: + device_class_ids.add(current_class.id) + device_classes = [] + for device_class_id in sorted(device_class_ids): + is_current = current_class is not None and device_class_id == current_class.id + read = self._store.read(device_class_id) + if read is None: + if current_class is None or not is_current: + continue + empty = CalibrationStoreFile( + schema_version=STORE_SCHEMA_VERSION, device_class=CalibrationDeviceClass.of(current_class) + ) + read = StoreRead(device_class_id, empty, None) + if is_current and read.file is None: + # Creating a run would have to write this file, which is kept as it is. + unavailable = f"The calibration store of this GPU's device class cannot be written: {read.problem}" + device_classes.append( + self._device_class_status( + read, is_current, current_env if is_current else None, live, present.get(device_class_id) + ) + ) + return WorkingMemoryCalibrationStatus( + available=unavailable is None, + unavailable_reason=unavailable, + current_device_class_id=current_class.id if current_class is not None else None, + current_env_id=current_env.id if current_env is not None else None, + device_classes=device_classes, + ) + + @staticmethod + def _device_class_status( + read: StoreRead, + is_current: bool, + current_env: Optional[Environment], + live: Mapping[str, Mapping[int, QueueItemLineage]], + backend: Optional[Backend], + ) -> WorkingMemoryCalibrationDeviceClass: + method, verified = _measurement(backend) + file = read.file + if file is None: + return WorkingMemoryCalibrationDeviceClass( + device_class_id=read.device_class_id, + device_class=None, + usable=False, + problem=read.problem, + is_current=is_current, + measurement_method=method, + measurement_verified=verified, + runs=[], + point_counts=[], + envs=[], + ) + points_per_run: dict[str, int] = {} + valid_points_per_run: dict[str, int] = {} + points_per_env: dict[str, int] = {} + counts: dict[tuple[str, int, bool, bool], int] = {} + for point in file.points: + points_per_run[point.run_id] = points_per_run.get(point.run_id, 0) + 1 + if point.valid: + valid_points_per_run[point.run_id] = valid_points_per_run.get(point.run_id, 0) + 1 + points_per_env[point.env_id] = points_per_env.get(point.env_id, 0) + 1 + key = (point.estimator, point.version, point.valid, point.fit_eligible) + counts[key] = counts.get(key, 0) + 1 + runs = [] + for run in file.runs: + point_count = points_per_run.get(run.run_id, 0) + status, error, item_ids = derive_run_status( + run, live.get(run.run_id, {}), valid_points_per_run.get(run.run_id, 0) + ) + runs.append( + WorkingMemoryCalibrationRunSummary( + run_id=run.run_id, + model_key=run.model_key, + model_hash=run.model_hash, + user_id=run.user_id, + created_at=run.created_at, + env_id=run.env_id, + status=status, + error=error, + item_ids=item_ids, + point_count=point_count, + attempt_id=run.attempt_id, + ) + ) + envs = [ + WorkingMemoryCalibrationEnvironment( + id=env.id, + hard=env.hard, + soft=env.soft, + hard_matches_current=None if current_env is None else env.id == current_env.id, + soft_matches_current=None if current_env is None else env.soft == current_env.soft, + point_count=points_per_env.get(env.id, 0), + ) + for env in file.envs + ] + return WorkingMemoryCalibrationDeviceClass( + device_class_id=read.device_class_id, + device_class=file.device_class, + usable=True, + problem=None, + is_current=is_current, + measurement_method=method, + measurement_verified=verified, + runs=runs, + point_counts=[ + WorkingMemoryCalibrationPointCount( + estimator=estimator, version=version, valid=valid, fit_eligible=fit_eligible, count=count + ) + for (estimator, version, valid, fit_eligible), count in sorted(counts.items()) + ], + envs=envs, + ) + + # endregion + + # region User fits + + def _user_fit_for(self, spec: EstimatorSpec, platform: Platform, device: torch.device) -> Optional[UserFit]: + """What estimates ask (`user_fit.user_fit_for`): a lookup in memory, no file or device access.""" + found = self._fits.get(_device_key(device)) + if found is None or found.platform != platform: + return None + return found.fit + + @staticmethod + def _signature(path: str) -> FileSignature: + try: + stat = os.stat(path) + except FileNotFoundError: + return None + return stat.st_mtime_ns, stat.st_size, stat.st_ino + + def refresh_user_fits(self) -> None: + with self._fits_lock: + fits: dict[str, _DeviceFit] = {} + for device in self._devices(): + if backend_for(device) is None: + continue + device_class = self._device_class(device) + env = self._environment(device) + platform = platform_for(device) + signature = self._signature(str(self._store.path_for(device_class.id))) + key = _device_key(device) + known = self._fits.get(key) + settled = known.settled_items if known is not None else frozenset() + file: Optional[CalibrationStoreFile] = None + if known is not None and (known.signature, known.env_id, known.platform) == ( + signature, + env.id, + platform, + ): + if known.file is None: + fits[key] = known + continue + # The same file: only a run that finished since (in the queue, unwritten) can change the fit. + live_runs, settled = self._live_runs(known.file, settled) + if live_runs == known.live_runs: + fits[key] = replace(known, settled_items=settled) + continue + file = known.file + elif signature is not None: + read = self._store.read(device_class.id) + if read is not None and read.file is None: + self._services.logger.warning( + f"Working-memory estimates on {device} keep the shipped values: {read.problem}" + ) + file = read.file if read is not None else None + live_runs, settled = self._live_runs(file, settled) if file is not None else (frozenset(), settled) + fit = ( + user_fit_of(file, env.id, platform, import_estimator_modules(), exclude_runs=live_runs)[0] + if file is not None + else None + ) + fits[key] = _DeviceFit( + signature=signature, + env_id=env.id, + platform=platform, + fit=fit, + file=file, + live_runs=live_runs, + settled_items=settled, + ) + self._fits = fits + + def _live_runs( + self, file: CalibrationStoreFile, settled: frozenset[int] = frozenset() + ) -> tuple[frozenset[str], frozenset[int]]: + """The runs whose windows no fit uses yet: those still running (`derive_run_status`; a run records its items as + soon as they are queued, so one whose items all wait counts, and an item it could not record counts once the + runner admitted it), and every run of an attempt (`attempt_id`) one of whose runs still runs. And `settled` + with the items found finished. A finished queue item never runs again (a retry is a new item), so a caller + that keeps `settled` asks the queue about each item until it finished, and no more. + + Reads the queue, by id, only for the items the file does not record as finished and `settled` does not hold: + those pending or running, and those that finished without the runner (canceled while pending, cleared). Once + every run's items finished, it reads nothing. A retry the runner has not started yet is not in the file, so + it holds back nothing until it starts.""" + open_items: dict[str, list[int]] = {} + for run in file.runs: + unfinished = [ + item_id + for item_id in set(run.queue_item_ids) | set(run.observed) + if item_id not in settled + and ((observed := run.observed.get(item_id)) is None or observed.status in _RUNNING) + ] + if unfinished: + open_items[run.run_id] = unfinished + if not open_items: + return frozenset(), settled + item_ids = sorted({item_id for unfinished in open_items.values() for item_id in unfinished}) + statuses: dict[int, QUEUE_ITEM_STATUS] = {} + queue = self._services.session_queue + for start in range(0, len(item_ids), _SUMMARY_CHUNK): + chunk = item_ids[start : start + _SUMMARY_CHUNK] + for summary in queue.get_queue_item_summaries_by_ids(DEFAULT_QUEUE_ID, chunk): + statuses[summary.item_id] = summary.status + live = set() + for run in file.runs: + unfinished = open_items.get(run.run_id) + # Only "running" matters: an item left out reads as finished, and so does a settled one. + if unfinished is not None: + # The summaries carry no retry source; the runner recorded it when it admitted the item. + queued = { + item_id: QueueItemLineage( + item_id=item_id, + status=statuses[item_id], + origin=None, + retried_from_item_id=observed.retried_from_item_id + if (observed := run.observed.get(item_id)) is not None + else None, + ) + for item_id in unfinished + if item_id in statuses + } + if derive_run_status(run, queued, valid_point_count=0)[0] == "running": + live.add(run.run_id) + # A client posts each step of an attempt as its own run, all of them up front: the finished earlier steps + # must not price the later, larger ones either. + attempts = {run.attempt_id for run in file.runs if run.run_id in live and run.attempt_id is not None} + live |= {run.run_id for run in file.runs if run.attempt_id is not None and run.attempt_id in attempts} + # Gone from the queue counts as finished too: it will not run. + finished = {item_id for item_id in item_ids if statuses.get(item_id) not in _RUNNING} + return frozenset(live), settled | finished + + def _try_refresh_user_fits(self, when: str) -> None: + """Refresh the fits `when` (for the log). A store that cannot be read now must not fail what triggered it, a + reset that already happened or a plan; the fits read so far then stay until the next refresh.""" + try: + self.refresh_user_fits() + except Exception: + self._services.logger.warning(f"Could not refresh the working-memory user fits {when}", exc_info=True) + + # endregion + + # region Plan and summary + + def get_plan(self, model_key: str) -> WorkingMemoryCalibrationPlan: + store = self._services.model_manager.store + # Raises UnknownModelException for an unknown key. + target = store.get_model(model_key) + # The case sizes price the decode with the fits estimates read: the store as it is now, not as the last + # queue item found it. + self._try_refresh_user_fits("before building a calibration plan") + device: Optional[torch.device] = None + unavailable: Optional[str] = None + try: + device = self._calibration_device() + except CalibrationUnavailableError as error: + unavailable = str(error) + + def reservation(estimator: str, height: int, width: int, dtype: str) -> int: + assert device is not None + return self._untiled_decode_reservation(estimator, height, width, dtype, device) + + specs = import_estimator_modules() + return build_plan( + target, + store.all_models(), + device_class=self._device_class(device) if device is not None else None, + unavailable=unavailable, + platform=platform_for(device) if device is not None else None, + precision=self._services.configuration.precision, + reservation=reservation, + specs=specs, + capacity_bytes=self._decode_capacity(device, specs) if device is not None else None, + ) + + def _decode_capacity(self, device: torch.device, specs: Mapping[str, EstimatorSpec]) -> Optional[int]: + """What the model cache can give one decode and its VAE on `device` once it offloaded every other model: this + process's share of the device now, less `reserve_vram_gb`, and within `max_cache_vram_gb` when that is set, + as `ModelCache._get_vram_available` budgets a lock. + + This process's memory outside torch's allocator counts at the larger of what it is now and the most a stored + window of this environment found it at (`recorded_outside_torch_bytes`, over the windows of the estimators a + user calibration moves): it grows with the first generations (on ROCm under Windows, HIP's retained pool and + MIOpen's workspaces, from about 0.3 GB in a fresh server to 2.6 GB on an RX 9060 XT), so a fresh server's + reading alone overstates what a decode will find. Without a stored window, a first calibration sizes for the + reading alone. Where the driver gives no figure now, the recorded one counts whole. The store is the one + `refresh_user_fits` read last; nothing here reads it again.""" + memory = self._device_memory(device) + if memory is None: + return None + held = memory.process_bytes + known = self._fits.get(_device_key(device)) + if known is not None and known.file is not None: + recorded = recorded_outside_torch_bytes(known.file, known.env_id, specs) + if recorded is not None: + held -= max(0, recorded - (memory.outside_torch_bytes or 0)) + config = self._services.configuration + if config.max_cache_vram_gb is not None: + held = min(held, int(config.max_cache_vram_gb * 2**30)) + return held - int(config.reserve_vram_gb * 2**30) + + def _present_backends(self) -> dict[str, Backend]: + """The backend of each device class a GPU generation device here belongs to, by device class id.""" + present: dict[str, Backend] = {} + for device in self._devices(): + backend = backend_for(device) + if backend is not None: + present[self._device_class(device).id] = backend + return present + + def _present(self, device_class_id: str) -> Optional[tuple[Environment, Platform]]: + """The environment and platform of a GPU generation device of this class; None when no such device is one.""" + for device in self._devices(): + if backend_for(device) is not None and self._device_class(device).id == device_class_id: + return self._environment(device), platform_for(device) + return None + + def get_model_summary( + self, model_key: str, device_class_id: Optional[str] = None + ) -> WorkingMemoryCalibrationModelSummary: + store = self._services.model_manager.store + # Raises UnknownModelException for an unknown key. + target = store.get_model(model_key) + class_id = self._current_device_class_id(device_class_id) + read = self._store.read(class_id) + present = self._present(class_id) + method, verified = _measurement(self._present_backends().get(class_id)) + summary = WorkingMemoryCalibrationModelSummary( + model_key=model_key, + device_class_id=class_id, + usable=read is None or read.file is not None, + problem=read.problem if read is not None else None, + current_env_id=present[0].id if present is not None else None, + measurement_method=method, + measurement_verified=verified, + lines=[], + excluded_windows=0, + runs=[], + ) + if read is None or read.file is None: + return summary + file = read.file + runs = [run for run in file.runs if run.model_key == model_key] + run_ids = {run.run_id for run in runs} + + def name_of(key: str) -> Optional[str]: + try: + return getattr(store.get_model(key), "name", None) + except UnknownModelException: + return None + + def platform_of(env: CalibrationEnvironment) -> Optional[Platform]: + if present is not None and env.id == present[0].id: + return present[1] + try: + return platform_for_environment(file.device_class.backend, env.hard) + except ValueError: + return None + + lines, excluded = model_lines( + file, + model_key, + present[0] if present is not None else None, + platform_of, + import_estimator_modules(), + priced_on=decode_lines_of(target, self._services.configuration.precision), + name_of=name_of, + live_runs=self._live_runs(file)[0], + ) + points_per_run: dict[str, int] = {} + valid_points_per_run: dict[str, int] = {} + for point in file.points: + if point.run_id in run_ids: + points_per_run[point.run_id] = points_per_run.get(point.run_id, 0) + 1 + if point.valid: + valid_points_per_run[point.run_id] = valid_points_per_run.get(point.run_id, 0) + 1 + live = self._live_items() if runs else {} + model_runs = [] + for run in sorted(runs, key=lambda run: run.created_at, reverse=True): + point_count = points_per_run.get(run.run_id, 0) + status, error, _ = derive_run_status(run, live.get(run.run_id, {}), valid_points_per_run.get(run.run_id, 0)) + model_runs.append( + WorkingMemoryCalibrationModelRun( + run_id=run.run_id, + status=status, + error=error, + created_at=run.created_at, + point_count=point_count, + attempt_id=run.attempt_id, + ) + ) + return summary.model_copy(update={"lines": lines, "excluded_windows": excluded, "runs": model_runs}) + + # endregion + + # region Reset and export + + @staticmethod + def _remove( + file: CalibrationStoreFile, run_ids: set[str], model_key: Optional[str] = None, *, remember: bool = True + ) -> tuple[bool, WorkingMemoryCalibrationResetResult]: + """Remove the runs `run_ids` with their points, and with `model_key` every point the summary attributes to + that model (`measured_models`). `remember` keeps the removed runs' ids, so that a repeat of the request that + started one cannot start it again.""" + models = run_models(file) + if remember: + file.removed_run_ids.extend( + run.run_id for run in file.runs if run.run_id in run_ids and _is_keyed(run.run_id) + ) + runs = [run for run in file.runs if run.run_id not in run_ids] + points = [ + point + for point in file.points + if point.run_id not in run_ids and (model_key is None or model_key not in measured_models(point, models)) + ] + result = WorkingMemoryCalibrationResetResult( + runs_removed=len(file.runs) - len(runs), points_removed=len(file.points) - len(points) + ) + file.runs, file.points = runs, points + _prune_envs(file) + return result.runs_removed > 0 or result.points_removed > 0, result + + def reset_run(self, run_id: str) -> WorkingMemoryCalibrationResetResult: + return self._remove_run(run_id, remember=True) + + def _remove_run(self, run_id: str, *, remember: bool) -> WorkingMemoryCalibrationResetResult: + found = self._find_run(run_id) + if found is None: + raise CalibrationRunNotFoundError(run_id) + result = self._store.change(found[0], lambda file: self._remove(file, {run_id}, remember=remember)) + self._run_files.pop(run_id, None) + self._try_refresh_user_fits("after a reset") + return result or WorkingMemoryCalibrationResetResult(runs_removed=0, points_removed=0) + + def reset_model(self, model_key: str, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + class_id = self._current_device_class_id(device_class_id) + + def remove(file: CalibrationStoreFile) -> tuple[bool, WorkingMemoryCalibrationResetResult]: + # Written with the removal or not at all: a node that read the older epoch drops its points of the model. + file.model_reset_epochs[model_key] = file.model_reset_epochs.get(model_key, 0) + 1 + run_ids = {run.run_id for run in file.runs if run.model_key == model_key} + return True, self._remove(file, run_ids, model_key)[1] + + result = self._store.change(class_id, remove) + self._try_refresh_user_fits("after a reset") + return result or WorkingMemoryCalibrationResetResult(runs_removed=0, points_removed=0) + + def reset_all(self, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + def remove(file: CalibrationStoreFile) -> tuple[bool, WorkingMemoryCalibrationResetResult]: + result = WorkingMemoryCalibrationResetResult(runs_removed=len(file.runs), points_removed=len(file.points)) + changed = bool(file.runs or file.points or file.envs) + file.removed_run_ids.extend(run.run_id for run in file.runs if _is_keyed(run.run_id)) + file.runs, file.points, file.envs = [], [], [] + return changed, result + + result = self._store.change(self._current_device_class_id(device_class_id), remove) + self._try_refresh_user_fits("after a reset") + return result or WorkingMemoryCalibrationResetResult(runs_removed=0, points_removed=0) + + def export(self, device_class_id: Optional[str] = None) -> Optional[bytes]: + return self._store.read_raw(self._current_device_class_id(device_class_id)) + + # endregion diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_fits.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_fits.py new file mode 100644 index 00000000000..8d30903d25c --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_fits.py @@ -0,0 +1,449 @@ +"""User lines from a calibration store file, which models its windows count for, and what the windows of one model's +lines measured. + +The fitting and resolution rules are the backend's (`backend/working_memory/user_fit.py`); this module picks the +windows a fit uses: those of one environment whose hard facts are the current ones, valid, fit-eligible and not +fragmented (`exclusions`), of the estimators a user calibration moves (`EstimatorSpec.user_calibrated`). The windows of +any other estimator, a Z-Image denoise's say, are kept in the store but measure no line here and size no plan +(`recorded_outside_torch_bytes`). +""" + +import json +from collections import Counter +from collections.abc import Callable, Iterable, Mapping +from collections.abc import Set as AbstractSet +from dataclasses import dataclass +from typing import Final, Optional + +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CalibrationEnvironment, + CalibrationLineState, + CalibrationPoint, + CalibrationRun, + CalibrationSizeExclusion, + CalibrationStoreFile, + WorkingMemoryCalibrationLineContributor, + WorkingMemoryCalibrationLineSize, + WorkingMemoryCalibrationModelLine, + WorkingMemoryCalibrationPixelRange, +) +from invokeai.backend.model_manager.taxonomy import SubModelType +from invokeai.backend.working_memory import fragmentation, user_fit +from invokeai.backend.working_memory.coefficients import shipped_coefficients +from invokeai.backend.working_memory.fingerprint import Environment +from invokeai.backend.working_memory.fit import EntryKey, Line +from invokeai.backend.working_memory.platform import Platform +from invokeai.backend.working_memory.spec import EstimatorSpec +from invokeai.backend.working_memory.user_fit import UserFit, UserLine + + +def fit_label(device_class_id: str, env_id: str) -> str: + return f"user:{device_class_id}:{env_id}" + + +def _measured_line(point: CalibrationPoint, specs: Mapping[str, EstimatorSpec]) -> Optional[Line]: + """The line a window measured: valid, a positive demand, and an estimator of `specs` that is `user_calibrated` + and whose line admits its features. None for every other window.""" + if not point.valid or point.demand_bytes is None or point.demand_bytes <= 0: + return None + spec = specs.get(point.estimator) + if spec is None or not spec.user_calibrated: + return None + found = spec.line(point.features) + return found if found is not None and found.fits else None + + +def _attempts(file: CalibrationStoreFile) -> dict[str, str]: + """Per run id, the attempt its windows are judged with: its `attempt_id`, or the run itself without one.""" + return {run.run_id: run.attempt_id or run.run_id for run in file.runs} + + +def exclusions( + file: CalibrationStoreFile, specs: Mapping[str, EstimatorSpec], live_runs: AbstractSet[str] = frozenset() +) -> dict[int, CalibrationSizeExclusion]: + """Why each window that measured a line (`specs` registers the estimators) does not count toward its fit, by + position in `file.points`; a window that measured a line and is absent counts. + + - partial_residency: valid but not fit-eligible, its model not fully on the device. + - fragmented: the fit-eligible windows of one measurement -- one attempt (`attempt_id`, else the run) at one + environment, estimator, version and features, in the order they ran -- are what `fragmentation.fragmented` + flags: they grew around a flat allocation, or their largest window is a ratio outlier against the smaller + sizes of the same line in the environment that count, from any attempt. The measurement drops out whole: its + windows ran back to back on one allocator state. Another attempt's windows of the same size ran on another and + are judged on their own, so a recalibration can bring a size back and a later fragmented attempt drops no clean + one. Windows of `live_runs` (runs still running) are judged, but set no reference for larger sizes, as they + join no user line yet. + """ + points = file.points + attempt_of = _attempts(file) + excluded: dict[int, CalibrationSizeExclusion] = {} + groups: dict[tuple[str, str, int, str, str], list[int]] = {} + lines: dict[int, Line] = {} + for index, point in enumerate(points): + line = _measured_line(point, specs) + if line is None: + continue + if not point.fit_eligible: + excluded[index] = "partial_residency" + continue + lines[index] = line + features = json.dumps(point.features, sort_keys=True, separators=(",", ":")) + attempt = attempt_of.get(point.run_id, point.run_id) + groups.setdefault((point.env_id, point.estimator, point.version, features, attempt), []).append(index) + + members = list(groups.values()) + measurements = [] + for (env_id, estimator, version, _, _), indices in groups.items(): + windows = [points[index] for index in indices] + # The windows of a measurement share their features, so they place it on one line at one size. + line = lines[indices[0]] + measurements.append( + fragmentation.Measurement( + line=(env_id, estimator, version, line.suffix), + size=line.size, + demand=[window.demand_bytes or 0 for window in windows], + reserved=[window.reserved.delta if window.reserved else None for window in windows], + allocated=[window.allocated.delta if window.allocated else None for window in windows], + reference=not any(window.run_id in live_runs for window in windows), + ) + ) + for indices, flagged in zip(members, fragmentation.fragmented(measurements), strict=True): + if flagged: + excluded.update(dict.fromkeys(indices, "fragmented")) + return excluded + + +def counted_windows( + file: CalibrationStoreFile, specs: Mapping[str, EstimatorSpec], live_runs: AbstractSet[str] = frozenset() +) -> list[CalibrationPoint]: + """The windows that count toward a fit: they measured a line, and `exclusions` names no reason against them.""" + excluded = exclusions(file, specs, live_runs) + return [ + point + for index, point in enumerate(file.points) + if index not in excluded and _measured_line(point, specs) is not None + ] + + +def recorded_outside_torch_bytes( + file: CalibrationStoreFile, env_id: str, specs: Mapping[str, EstimatorSpec] +) -> Optional[int]: + """The most the measuring process held outside torch's allocator when a valid window of environment `env_id` + opened, of an estimator a user calibration moves (`EstimatorSpec.user_calibrated`, as `specs` registers it): the + driver's figure for the process less torch's reserve, at the window's baseline. None without such a window, or + where the driver had no figure for the process (`torch-only`, and XPU's device-wide reading). The windows of any + other estimator, a Z-Image denoise's say, size no calibration plan.""" + calibrated = {estimator for estimator, spec in specs.items() if spec.user_calibrated} + found = [ + point.driver.baseline - point.reserved.baseline + for point in file.points + if point.env_id == env_id + and point.valid + and point.estimator in calibrated + and point.driver is not None + and point.reserved is not None + ] + return max(found, default=None) + + +def user_fit_of( + file: CalibrationStoreFile, + env_id: str, + platform: Platform, + specs: Mapping[str, EstimatorSpec], + *, + exclude_runs: AbstractSet[str] = frozenset(), +) -> tuple[Optional[UserFit], Mapping[user_fit.LineKey, str]]: + """The user fit of the file's counted windows (`counted_windows`) taken under environment `env_id`, measured on + `platform`, or None when they fit no line; and why the lines they measure but cannot fit cannot be. The windows of + `exclude_runs` (runs still running) take no part: they join once their run finished.""" + points = [ + user_fit.UserPoint(point.estimator, point.version, point.features, point.demand_bytes or 0) + for point in counted_windows(file, specs, exclude_runs) + if point.env_id == env_id and point.run_id not in exclude_runs + ] + lines, problems = user_fit.fit_user_lines(points, platform, specs) + if not lines: + return None, problems + return UserFit(label=fit_label(file.device_class.id, env_id), platform=platform, lines=lines), problems + + +def _changed(old: Mapping[str, Optional[str]], new: Mapping[str, Optional[str]]) -> list[str]: + return sorted(key for key in set(old) | set(new) if old.get(key) != new.get(key)) + + +_SUBMODEL_TYPES: Final = frozenset(submodel.value for submodel in SubModelType) + + +def measured_models(point: CalibrationPoint, run_models: Mapping[str, str]) -> frozenset[str]: + """The models a window is attributed to: the model its run calibrated (`run_models`, by run id), and the model it + measured, where a submodel's window (`:vae`) counts for its model. The summary shows a model's windows by + this, and resetting the model removes exactly these.""" + head, separator, tail = point.model_key.rpartition(":") + measured = head if separator and tail in _SUBMODEL_TYPES else point.model_key + run_model = run_models.get(point.run_id) + return frozenset({measured} if run_model is None else {measured, run_model}) + + +def run_models(file: CalibrationStoreFile) -> dict[str, str]: + return {run.run_id: run.model_key for run in file.runs} + + +@dataclass(frozen=True) +class _EnvFit: + env: Optional[CalibrationEnvironment] + platform: Optional[Platform] + fit: Optional[UserFit] + problems: Mapping[user_fit.LineKey, str] + + +LineId = tuple[str, str, int, str] +"""A summary line: environment id, estimator, version and ``_`` (`fit.Line.suffix`).""" + + +def model_lines( + file: CalibrationStoreFile, + model_key: str, + current_env: Optional[Environment], + platform_of: Callable[[CalibrationEnvironment], Optional[Platform]], + specs: Mapping[str, EstimatorSpec], + priced_on: Iterable[tuple[str, str]], + name_of: Callable[[str], Optional[str]], + live_runs: AbstractSet[str] = frozenset(), +) -> tuple[list[WorkingMemoryCalibrationModelLine], int]: + """Per environment and line, what the windows that feed it measured, with the shipped and the user estimate at + each size; and how many of the model's own windows fit no line. + + The lines are those the model's own windows (`measured_models`) measured, and, in `current_env`, each line of + `priced_on` (estimator, ``_``: what the model's operations price on) that other models' windows + measured, of the estimators of `specs` a user calibration moves (`EstimatorSpec.user_calibrated`). A line is fed + by every model's counted windows (`exclusions`) of its environment that join the same user line, as estimates read + it: those whose estimator resolves to the same carrying entry on the environment's platform. Its sizes also show, + as not counted, the sizes only excluded windows measured. A window of any other estimator measured no line here, and + counts among the model's windows that fit none. + + The sizes, window counts and contributors show every such window, those of `live_runs` (runs still running) + included, so a running calibration shows what it measured so far. The user line, its range, the user estimates and + the state are what estimates use: the fit without the windows of `live_runs`. + + The state compares the environment with `current_env` (None: no GPU of the file's class generates here): the same + hard facts use the line, stale when the soft facts of the newest finished run that fed it differ from the current + ones. A run stored without soft facts stands for the environment's, those first recorded with it. Older windows of + the same hard environment keep feeding the line whatever their soft facts: soft facts do not change what a window + measured enough to drop it. + """ + envs = {env.id: env for env in file.envs} + runs = {run.run_id: run for run in file.runs} + models = run_models(file) + reasons = exclusions(file, specs, live_runs) + # By identity: the windows below are the file's own objects. + reason_of: dict[int, CalibrationSizeExclusion] = { + id(file.points[index]): reason for index, reason in reasons.items() + } + excluded = 0 + own: dict[LineId, list[CalibrationPoint]] = {} + # Per environment, each window that measured a line, with that line's suffix. + measured_by_env: dict[str, list[tuple[CalibrationPoint, str]]] = {} + for point in file.points: + measured = _measured_line(point, specs) + if measured is not None: + measured_by_env.setdefault(point.env_id, []).append((point, measured.suffix)) + if model_key not in measured_models(point, models): + continue + if measured is None or id(point) in reason_of: + excluded += 1 + if measured is not None: + own.setdefault((point.env_id, point.estimator, point.version, measured.suffix), []).append(point) + + def counts(point: CalibrationPoint) -> bool: + return id(point) not in reason_of + + env_fits: dict[str, _EnvFit] = {} + + def env_fit(env_id: str) -> _EnvFit: + if env_id not in env_fits: + env = envs.get(env_id) + platform = platform_of(env) if env is not None else None + fit, problems = ( + user_fit_of(file, env_id, platform, specs, exclude_runs=live_runs) + if platform is not None + else (None, {}) + ) + env_fits[env_id] = _EnvFit(env, platform, fit, problems) + return env_fits[env_id] + + def carrier_of(estimator: str, version: int, platform: Optional[Platform]) -> Optional[EntryKey]: + """The entry whose user line a window of this estimator version joins; None where it is not priced here.""" + spec = specs.get(estimator) + if spec is None or spec.version != version or platform is None: + return None + return user_fit.carrier_key(estimator, version, platform) + + def feeding(line_id: LineId) -> list[CalibrationPoint]: + """The windows of the line's environment that measured its user line, counted or not; without a carrying + entry, those of the same estimator version.""" + env_id, estimator, version, suffix = line_id + platform = env_fit(env_id).platform + carrier = carrier_of(estimator, version, platform) + found = [] + for point, measured in measured_by_env.get(env_id, []): + if measured != suffix: + continue + if carrier is None: + same = (point.estimator, point.version) == (estimator, version) + else: + same = carrier_of(point.estimator, point.version, platform) == carrier + if same: + found.append(point) + return found + + line_ids = set(own) + if current_env is not None: + for estimator, suffix in priced_on: + spec = specs.get(estimator) + if spec is not None and spec.user_calibrated: + line_ids.add((current_env.id, estimator, spec.version, suffix)) + + names: dict[str, Optional[str]] = {} + lines: list[WorkingMemoryCalibrationModelLine] = [] + for line_id in sorted(line_ids): + env_id, estimator, version, suffix = line_id + measured = feeding(line_id) + if not measured: + # A line the model prices on that no window measured in this environment. + continue + points = [point for point in measured if counts(point)] + own_points = [point for point in own.get(line_id, []) if counts(point)] + found_fit = env_fit(env_id) + env, platform, fit = found_fit.env, found_fit.platform, found_fit.fit + spec = specs.get(estimator) + priced = spec is not None and spec.version == version + carrier = carrier_of(estimator, version, platform) + line: Optional[UserLine] = None + if fit is not None and carrier is not None: + line = fit.lines.get((carrier, suffix)) + + changed: list[str] = [] + problem: Optional[str] = None + state: CalibrationLineState + if env is None or current_env is None: + state, problem = "unusable", "No generation device of this class is present." + elif env.id != current_env.id: + state, changed = "unusable", _changed(env.hard, current_env.hard) + elif not priced: + state = "unusable" + problem = f"Measured with {estimator} version {version}, which this version does not price with." + elif line is None: + state = "insufficient" + known = found_fit.problems.get((carrier, suffix)) if carrier is not None else None + if not points: + problem = "None of this line's windows count: see each size's excluded_reason." + elif any(point.run_id in live_runs for point in points): + problem = "A calibration run is still measuring this line; its windows count once it finished." + if known: + problem += f" So far: {known}" + else: + problem = known or "The shipped data has no such line for this estimator on this platform." + else: + finished = [point for point in points if point.run_id not in live_runs] + changed = _changed(_newest_soft(finished, runs, env), current_env.soft) + state = "stale" if changed else "in_use" + + windows: Counter[str] = Counter() + for point in points: + windows.update(measured_models(point, models)) + for key in windows: + if key not in names: + names[key] = name_of(key) + measured_by = [ + WorkingMemoryCalibrationLineContributor(model_key=key, name=names[key], windows=count) + for key, count in windows.items() + ] + measured_by.sort( + key=lambda contributor: (contributor.name is None, contributor.name or "", contributor.model_key) + ) + + lines.append( + WorkingMemoryCalibrationModelLine( + estimator=estimator, + version=version, + line=suffix, + carrier=carrier[0] if carrier else None, + env_id=env_id, + state=state, + changed_facts=changed, + problem=problem, + measured_range=( + WorkingMemoryCalibrationPixelRange(min_pixels=line.x_min, max_pixels=line.x_max) if line else None + ), + slope_bytes_per_pixel=line.k if line else None, + base_bytes=line.b if line else None, + sizes=_sizes(measured, reason_of, estimator, specs if priced else {}, platform, fit if line else None), + own_windows=len(own_points), + measured_by=measured_by, + ) + ) + return lines, excluded + + +def _newest_soft( + points: Iterable[CalibrationPoint], runs: Mapping[str, CalibrationRun], env: CalibrationEnvironment +) -> Mapping[str, Optional[str]]: + """The soft facts of the newest run among those that measured `points`; the environment's when it recorded none. + `runs` is in the file's order, which is the order they were created in: it decides a tie of the clock.""" + order = {run_id: index for index, run_id in enumerate(runs)} + fed = {point.run_id for point in points if point.run_id in runs} + newest = max((runs[run_id] for run_id in fed), key=lambda run: (run.created_at, order[run.run_id]), default=None) + return newest.soft if newest is not None and newest.soft is not None else env.soft + + +def _sizes( + points: Iterable[CalibrationPoint], + reason_of: Mapping[int, CalibrationSizeExclusion], + estimator: str, + specs: Mapping[str, EstimatorSpec], + platform: Optional[Platform], + fit: Optional[UserFit], +) -> list[WorkingMemoryCalibrationLineSize]: + """Per measured size, the windows and their largest demand, and the shipped and the user estimate there. + + A size with counted windows shows those; a size whose windows all are excluded (`reason_of`, by window identity) + shows them as not counted, fragmented where any of them is. The estimates price the largest window of the line's + own estimator among those shown, or, where only other estimators that share the line measured the size, the largest + window with its own estimator: their features are their own.""" + by_size: dict[tuple[int, int], list[CalibrationPoint]] = {} + for point in points: + by_size.setdefault((int(point.features["width"]), int(point.features["height"])), []).append(point) # type: ignore[arg-type] + sizes = [] + for (width, height), measured in sorted(by_size.items(), key=lambda item: (item[0][0] * item[0][1], item[0])): + group = [point for point in measured if id(point) not in reason_of] + counted = bool(group) + excluded_reason: Optional[CalibrationSizeExclusion] = None + if not counted: + group = measured + found = {reason_of[id(point)] for point in measured} + excluded_reason = "fragmented" if "fragmented" in found else "partial_residency" + largest = max(group, key=lambda point: point.demand_bytes or 0) + mine = [point for point in group if point.estimator == estimator] + priced = max(mine, key=lambda point: point.demand_bytes or 0) if mine else largest + spec = specs.get(priced.estimator) if estimator in specs else None + shipped_bytes = user_bytes = None + if spec is not None and spec.version == priced.version and platform is not None: + shipped = shipped_coefficients(spec, platform) + shipped_bytes = spec.evaluate(priced.features, shipped).reservation_bytes + if fit is not None: + user_bytes = user_fit.resolve(spec, priced.features, shipped, fit).reservation_bytes + sizes.append( + WorkingMemoryCalibrationLineSize( + width=width, + height=height, + pixels=width * height, + windows=len(group), + measured_bytes=largest.demand_bytes or 0, + counted=counted, + excluded_reason=excluded_reason, + shipped_bytes=shipped_bytes, + user_bytes=user_bytes, + ) + ) + return sizes diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_plan.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_plan.py new file mode 100644 index 00000000000..622607420f9 --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_plan.py @@ -0,0 +1,421 @@ +"""The cases a client runs to calibrate one model's working memory, and which other models that calibration moves. + +Covered: the decode of main models whose decode node prices with one of the five AutoencoderKL estimators of +`backend/util/vae_working_memory.py` (`DECODE_LINES`), and standalone VAEs of those bases, calibrated by generating +with a compatible installed main model. Each case is a two-step text-to-image generation, which runs the whole +pipeline and ends in an untiled decode; image-to-image would add an encode, which v1 does not plan. +""" + +import math +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from functools import partial +from typing import Any, Final, Optional + +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + WorkingMemoryCalibrationCase, + WorkingMemoryCalibrationDeviceClassRef, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationRepetitions, + WorkingMemoryCalibrationSharedModel, +) +from invokeai.backend.architectures import VaeFacet +from invokeai.backend.architectures import get as get_facet +from invokeai.backend.model_manager.taxonomy import ( + BaseModelType, + FluxVariantType, + ModelType, + ModelVariantType, +) +from invokeai.backend.util.vae_working_memory import VAE_PRETILE_VRAM_FRACTION +from invokeai.backend.working_memory import user_fit +from invokeai.backend.working_memory.fingerprint import DeviceClass +from invokeai.backend.working_memory.fit import MIN_AREA_RATIO, EntryKey +from invokeai.backend.working_memory.platform import Platform +from invokeai.backend.working_memory.probe import calibration_reservation +from invokeai.backend.working_memory.spec import EstimatorSpec + +ModelConfig = Any +"""An installed model's config. Read by attribute: `key`, `name`, `base`, `type`, and where present `variant` and +`cpu_only`.""" + +ReservationFunction = Callable[[str, int, int, str], int] +"""The reservation an untiled decode of (estimator, height, width, dtype) gets on the calibration device.""" + + +@dataclass(frozen=True) +class DecodeLines: + estimator: str + dtypes: tuple[str, ...] + """The dtypes the decode runs in by default. Where the node casts the VAE (`vaePrecision`), each is a case.""" + precision_setting: bool = False + + +_SD = DecodeLines("vae.sd", ("fp16", "fp32"), precision_setting=True) +_FLUX = DecodeLines("vae.flux", ("bf16",)) +_FLUX2 = DecodeLines("vae.flux2", ("bf16",)) + +DECODE_LINES: Final[Mapping[BaseModelType, DecodeLines]] = { + BaseModelType.StableDiffusion1: _SD, + BaseModelType.StableDiffusion2: _SD, + BaseModelType.StableDiffusionXL: _SD, + BaseModelType.StableDiffusionXLRefiner: _SD, + # The SD3 VAE runs in the configured precision (`_sd3_dtype`). + BaseModelType.StableDiffusion3: DecodeLines("vae.sd3", ()), + # The FLUX.1 loaders avoid fp16, and Z-Image decodes with a FLUX.1 VAE. + BaseModelType.Flux: _FLUX, + BaseModelType.ZImage: _FLUX, + BaseModelType.Flux2: _FLUX2, + # Decodes with the FLUX.2 VAE; shares its lines, but generates from structured prompts the plan does not write. + BaseModelType.Ideogram4: _FLUX2, + # The CogView4 loader forces bf16. + BaseModelType.CogView4: DecodeLines("vae.cogview4", ("bf16",)), +} +"""Per base, the estimator its decode node prices with and the dtypes it runs in.""" + +PLANNABLE_BASES: Final = frozenset( + { + BaseModelType.StableDiffusion1, + BaseModelType.StableDiffusion2, + BaseModelType.StableDiffusionXL, + BaseModelType.StableDiffusion3, + BaseModelType.Flux, + BaseModelType.Flux2, + BaseModelType.ZImage, + BaseModelType.CogView4, + } +) +"""Bases whose main models a text-to-image case can calibrate. The SDXL refiner generates from no text, and Ideogram 4 +from structured prompts only.""" + +VAE_SLOT_BASES: Final = frozenset( + {BaseModelType.Flux, BaseModelType.Flux2, BaseModelType.StableDiffusion3, BaseModelType.ZImage} +) +"""Bases whose Generate widget has a `vae` component slot. The SD bases pick a VAE through a setting a case cannot +override, and CogView4 has none: their standalone VAEs are calibrated through a main model's own VAE, which prices on +the same lines.""" + +NOT_TEXT_TO_IMAGE_VARIANTS: Final = frozenset( + {ModelVariantType.Inpaint.value, ModelVariantType.Depth.value, FluxVariantType.DevFill.value} +) + +CASE_STEPS: Final = 2 +CASE_SEED: Final = 1 +CASE_PROMPT: Final = "a mountain lake at dawn, detailed photograph" +REPETITIONS: Final = WorkingMemoryCalibrationRepetitions(warm_up=1, measured=2) + +AREA_FRACTIONS: Final = (0.25, 0.5, 1.0) +SIDE_MULTIPLE: Final = 64 +"""A multiple of every covered base's dimension grid (8, 16 or 32).""" +MAX_SIDE: Final = 4096 +"""The largest width or height Generate accepts; no covered architecture declares a smaller limit.""" +MIN_LARGEST_SIDE: Final = 512 +VAE_WEIGHTS_ALLOWANCE_BYTES: Final = 2**30 +"""What the VAE's own weights take on the device while it decodes: under 1 GiB for every covered VAE.""" + +SECONDS_MODEL_LOAD: Final = 30 +SECONDS_PER_RUN: Final = 3 +SECONDS_PER_MEGAPIXEL_STEP: Final = 1.0 +SECONDS_PER_MEGAPIXEL_DECODE: Final = 1.0 + +ENCODE_NOT_COVERED: Final = ( + "encode: text-to-image runs no encode, so image-to-image and inpainting encodes keep the shipped values" +) + + +def sd3_dtype(precision: str) -> str: + """The dtype the SD3 VAE runs in under the `precision` setting: `auto` is fp16 on a GPU.""" + return {"bfloat16": "bf16", "float32": "fp32"}.get(precision, "fp16") + + +def decode_dtypes(base: BaseModelType, precision: str) -> tuple[str, ...]: + lines = DECODE_LINES[base] + return lines.dtypes or (sd3_dtype(precision),) + + +def _value(field: object) -> str: + return str(getattr(field, "value", field)) + + +def _variant(config: ModelConfig) -> Optional[str]: + variant = getattr(config, "variant", None) + return None if variant is None else _value(variant) + + +def _runs_on_gpu(vae: ModelConfig) -> bool: + return getattr(vae, "cpu_only", None) is not True + + +def _accepts(main_base: BaseModelType, vae_base: BaseModelType) -> bool: + facet = get_facet(main_base, VaeFacet) + if facet is None: + return main_base == vae_base + return any(compatibility.base == vae_base for compatibility in facet.accepted) + + +def _is_main(config: ModelConfig) -> bool: + return bool(config.type == ModelType.Main) + + +def _is_vae(config: ModelConfig) -> bool: + return bool(config.type == ModelType.VAE) + + +def _plannable_main(config: ModelConfig) -> bool: + return _is_main(config) and config.base in PLANNABLE_BASES and _variant(config) not in NOT_TEXT_TO_IMAGE_VARIANTS + + +def decode_budget(device_class: DeviceClass, capacity_bytes: Optional[int]) -> int: + """The largest calibration reservation a case may take: within the share of the card a decode may claim before the + node tiles it up front (`VAE_PRETILE_VRAM_FRACTION`), and within `capacity_bytes`, what the model cache can give on + the device; each less the VAE's own weights, which must stay fully resident beside it. Nothing else does: a + calibration lock first moves every other unlocked model out (`LoadedModelWithoutConfig.model_on_device`). Past the + capacity the cache loads the VAE partly (`partial_residency`), and such a window fits no line. `capacity_bytes` + None: the device does not say (shared memory), and the share of the card alone bounds it.""" + ceiling = int(VAE_PRETILE_VRAM_FRACTION * device_class.total_mib * 2**20) + if capacity_bytes is not None: + ceiling = min(ceiling, capacity_bytes) + return ceiling - VAE_WEIGHTS_ALLOWANCE_BYTES + + +def case_sides(reservation: Callable[[int], int], budget_bytes: int) -> list[int]: + """The square sides of the three cases: about `AREA_FRACTIONS` of the largest area that decodes untiled here. + + The largest area is the largest square, in steps of `SIDE_MULTIPLE` and at most `MAX_SIDE`, whose calibration + reservation (the estimate plus calibration headroom, `probe.calibration_reservation`) fits `budget_bytes` + (`decode_budget`). A larger decode would be tiled or partly loaded, and neither fits a line. Each fraction of that + area is rounded down to a square side that is a multiple of `SIDE_MULTIPLE`; repeated sides are dropped. Returns + no sides when the largest is below `MIN_LARGEST_SIDE` or the sides do not span two sizes `MIN_AREA_RATIO` apart. + """ + largest = 0 + for side in range(SIDE_MULTIPLE, MAX_SIDE + 1, SIDE_MULTIPLE): + if calibration_reservation(reservation(side)) > budget_bytes: + break + largest = side + if largest < MIN_LARGEST_SIDE: + return [] + sides: list[int] = [] + for fraction in AREA_FRACTIONS: + side = max(SIDE_MULTIPLE, int(math.sqrt(fraction) * largest) // SIDE_MULTIPLE * SIDE_MULTIPLE) + if side not in sides: + sides.append(side) + if sides[-1] ** 2 < MIN_AREA_RATIO * sides[0] ** 2: + return [] + return sides + + +def _square(reservation: ReservationFunction, estimator: str, dtype: str, side: int) -> int: + return reservation(estimator, side, side, dtype) + + +def decode_lines_of(model: ModelConfig, precision: str) -> list[tuple[str, str]]: + """The (estimator, ``decode_``) lines a model's decode prices on: a main model or a GPU VAE of a base in + `DECODE_LINES`; none for any other model.""" + if getattr(model, "base", None) not in DECODE_LINES: + return [] + if not (_is_main(model) or (_is_vae(model) and _runs_on_gpu(model))): + return [] + estimator = DECODE_LINES[model.base].estimator + return [(estimator, f"decode_{dtype}") for dtype in decode_dtypes(model.base, precision)] + + +def estimated_seconds(cases: Sequence[WorkingMemoryCalibrationCase]) -> int: + """A rough order of magnitude: one model load, then per run a fixed cost and a cost per megapixel for each step + and for the decode. Not measured; real times vary several-fold with the GPU and the model.""" + runs = REPETITIONS.warm_up + REPETITIONS.measured + seconds: float = SECONDS_MODEL_LOAD + for case in cases: + megapixels = case.width * case.height / 1e6 + per_run = SECONDS_PER_RUN + megapixels * ( + case.steps * SECONDS_PER_MEGAPIXEL_STEP + SECONDS_PER_MEGAPIXEL_DECODE + ) + seconds += runs * per_run + return math.ceil(seconds) + + +def _lines( + base: BaseModelType, precision: str, platform: Platform, specs: Mapping[str, EstimatorSpec] +) -> set[tuple[EntryKey, str]]: + """The lines a base's decode prices with on `platform`.""" + estimator = DECODE_LINES[base].estimator + spec = specs.get(estimator) + carrier = user_fit.carrier_key(estimator, spec.version, platform) if spec is not None else None + if carrier is None: + return set() + return {(carrier, f"decode_{dtype}") for dtype in decode_dtypes(base, precision)} + + +def shared_with( + target: ModelConfig, + models: Sequence[ModelConfig], + precision: str, + platform: Platform, + specs: Mapping[str, EstimatorSpec], +) -> list[WorkingMemoryCalibrationSharedModel]: + """The other installed main models and GPU VAEs of a covered base whose decode prices with a line the target's + does, on this platform: a line is the shipped entry that carries the values (one estimator may borrow another's) + and the operation and dtype.""" + mine = _lines(target.base, precision, platform, specs) + shared = [] + for model in models: + if model.key == target.key or model.base not in DECODE_LINES: + continue + if not (_is_main(model) or (_is_vae(model) and _runs_on_gpu(model))): + continue + common = sorted(suffix for _, suffix in mine & _lines(model.base, precision, platform, specs)) + if common: + shared.append( + WorkingMemoryCalibrationSharedModel( + key=model.key, name=model.name, base=_value(model.base), type=_value(model.type), lines=common + ) + ) + return sorted(shared, key=lambda model: (model.name, model.key)) + + +def _gpu_vae_pick(main: ModelConfig, models: Sequence[ModelConfig]) -> dict[str, str]: + """A `vae` pick where Generate's own fill could land on a VAE that runs on the CPU: the first GPU VAE the main + model accepts. None needed where every accepted VAE runs on the GPU, or where the base has no VAE slot.""" + if main.base not in VAE_SLOT_BASES: + return {} + accepted = [model for model in models if _is_vae(model) and _accepts(main.base, model.base)] + if all(_runs_on_gpu(vae) for vae in accepted): + return {} + gpu = sorted((vae for vae in accepted if _runs_on_gpu(vae)), key=lambda vae: (vae.name, vae.key)) + return {"vae": gpu[0].key} if gpu else {} + + +def _unsupported( + target: ModelConfig, reason: str, device: Optional[DeviceClass], estimators: list[str] +) -> WorkingMemoryCalibrationPlan: + return WorkingMemoryCalibrationPlan( + model_key=target.key, + supported=False, + reason=reason, + device_class=WorkingMemoryCalibrationDeviceClassRef(id=device.id, name=device.name) if device else None, + calibrated_through=None, + estimators=estimators, + cases=[], + repetitions=REPETITIONS, + estimated_seconds=None, + shared_with=[], + not_covered=[ENCODE_NOT_COVERED] if estimators else [], + ) + + +def build_plan( + target: ModelConfig, + models: Sequence[ModelConfig], + *, + device_class: Optional[DeviceClass], + unavailable: Optional[str], + platform: Optional[Platform], + precision: str, + reservation: ReservationFunction, + specs: Mapping[str, EstimatorSpec], + capacity_bytes: Optional[int] = None, +) -> WorkingMemoryCalibrationPlan: + """The plan for calibrating `target`, among the installed `models`, on the GPU of `device_class`. + + `unavailable` says why this machine cannot calibrate (no GPU, several generation devices); `device_class` and + `platform` are then None. `reservation` prices an untiled decode on the calibration device; `precision` is the + `precision` setting; `capacity_bytes` is what the cache can give on the device (`decode_budget`). + """ + covered = target.base in DECODE_LINES and (_is_main(target) or _is_vae(target)) + estimators = [DECODE_LINES[target.base].estimator] if covered else [] + if not covered: + kind = f"{_value(target.base)} {_value(target.type)}" + return _unsupported( + target, + f"Calibration covers the decode of SD1, SD2, SDXL, SD3, FLUX.1, FLUX.2, Z-Image and CogView4 main models " + f"and their VAEs; this is a {kind} model.", + device_class, + estimators, + ) + + main = target + if _is_vae(target): + if not _runs_on_gpu(target): + return _unsupported( + target, + "This VAE runs on the CPU (cpu_only); calibration measures GPU memory.", + device_class, + estimators, + ) + candidates = sorted( + (model for model in models if _plannable_main(model) and _accepts(model.base, target.base)), + key=lambda model: (model.name, model.key), + ) + if not candidates: + return _unsupported( + target, + "A VAE is calibrated by generating with a main model that decodes with it, and no compatible main " + "model is installed.", + device_class, + estimators, + ) + main = candidates[0] + elif target.base not in PLANNABLE_BASES: + return _unsupported( + target, + "This model shares its decode with others but cannot generate from a plain text prompt; calibrate one of " + "the models listed as sharing it.", + device_class, + estimators, + ) + elif _variant(target) in NOT_TEXT_TO_IMAGE_VARIANTS: + return _unsupported( + target, f"The {_variant(target)} variant does not generate from text alone.", device_class, estimators + ) + + if unavailable is not None or device_class is None or platform is None: + return _unsupported(target, unavailable or "This machine has no GPU to calibrate.", None, estimators) + + if _is_vae(target) and main.base in VAE_SLOT_BASES: + components = {"vae": target.key} + else: + components = _gpu_vae_pick(main, models) + + lines = DECODE_LINES[target.base] + budget = decode_budget(device_class, capacity_bytes) + cases: list[WorkingMemoryCalibrationCase] = [] + for dtype in decode_dtypes(target.base, precision): + sides = case_sides(partial(_square, reservation, lines.estimator, dtype), budget) + for side in sides: + cases.append( + WorkingMemoryCalibrationCase( + id=f"{lines.estimator}-decode-{dtype}-{side}x{side}", + model_key=main.key, + components=components, + width=side, + height=side, + steps=CASE_STEPS, + seed=CASE_SEED, + positive_prompt=CASE_PROMPT, + negative_prompt="", + settings={"vaePrecision": dtype} if lines.precision_setting else {}, + estimator=lines.estimator, + line=f"decode_{dtype}", + ) + ) + if not cases: + return _unsupported( + target, + f"{device_class.name} cannot hold an untiled decode of {MIN_LARGEST_SIDE}x{MIN_LARGEST_SIDE} with " + "calibration headroom and the VAE fully loaded, in the memory free on it now.", + device_class, + estimators, + ) + + return WorkingMemoryCalibrationPlan( + model_key=target.key, + supported=True, + reason=None, + device_class=WorkingMemoryCalibrationDeviceClassRef(id=device_class.id, name=device_class.name), + calibrated_through=main.key if main is not target else None, + estimators=estimators, + cases=cases, + repetitions=REPETITIONS, + estimated_seconds=estimated_seconds(cases), + shared_with=shared_with(target, models, precision, platform, specs), + not_covered=[ENCODE_NOT_COVERED], + ) diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_store.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_store.py new file mode 100644 index 00000000000..c669385f78f --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_store.py @@ -0,0 +1,249 @@ +"""The calibration store: one JSON file per device class under `/working_memory/`. + +Machine data, not account data, so it lives beside the database rather than in it: a device class's measurements stay +valid whichever database the root points at. Every change re-reads the file and is applied to what is on disk under a +process lock and an OS file lock, so two processes sharing a root never lose each other's writes. Files are replaced +atomically. A file this version cannot read -- another schema version, invalid content, a field it does not know, or a device +class that does not match its name -- is reported and never written, moved or deleted. +""" + +import json +import os +import re +import tempfile +import threading +import time +from collections.abc import Callable, Iterator +from contextlib import contextmanager, suppress +from pathlib import Path +from typing import IO, Final, Optional, TypeVar + +from pydantic import ValidationError + +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + STORE_SCHEMA_VERSION, + CalibrationDeviceClass, + CalibrationStoreError, + CalibrationStoreFile, + CalibrationStoreUnusableError, +) + +LOCK_TIMEOUT_S: float = 30.0 +"""How long a change waits for another process's lock on the same file before it gives up.""" +LOCK_RETRY_S: float = 0.05 +REPLACE_ATTEMPTS: int = 6 +"""How often a write tries to replace the file. On Windows the replace fails with PermissionError while a process that +does not take the store's lock (an antivirus scanner, a search indexer, an editor) has the file open; such a handle is +usually gone within a fraction of a second.""" +REPLACE_RETRY_S: float = 0.05 +"""The first wait between those tries; it doubles each time (about 1.5 s in all).""" + +_FILE_NAME: Final = re.compile(r"^[0-9a-f]{16}\.json$") + +T = TypeVar("T") + + +class StoreRead: + """A device class's file as read: its content, or why it cannot be used.""" + + def __init__(self, device_class_id: str, file: Optional[CalibrationStoreFile], problem: Optional[str]) -> None: + self.device_class_id = device_class_id + self.file = file + self.problem = problem + + +def _parse(raw: bytes, device_class_id: str) -> tuple[Optional[CalibrationStoreFile], Optional[str]]: + try: + data = json.loads(raw) + except ValueError: + return None, "The file is not valid JSON." + if not isinstance(data, dict): + return None, "The file does not hold a calibration store." + version = data.get("schema_version") + if version != STORE_SCHEMA_VERSION: + return None, f"The file has schema version {version!r}; this version of Invoke reads {STORE_SCHEMA_VERSION}." + try: + file = CalibrationStoreFile.model_validate(data) + except ValidationError as error: + first = error.errors()[0] + location = ".".join(str(part) for part in first["loc"]) + return None, f"The file failed validation ({error.error_count()} errors; first at {location}: {first['msg']})." + if file.device_class.id != device_class_id: + return None, f"The file holds device class {file.device_class.id}, not the {device_class_id} its name says." + return file, None + + +@contextmanager +def _os_lock(handle: IO[bytes]) -> Iterator[None]: + if os.name == "nt": + import msvcrt + + deadline = time.monotonic() + LOCK_TIMEOUT_S + while True: + handle.seek(0) + try: + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + break + except OSError: + if time.monotonic() > deadline: + raise CalibrationStoreError(f"Timed out waiting for the lock on {handle.name}") from None + time.sleep(LOCK_RETRY_S) + try: + yield + finally: + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + deadline = time.monotonic() + LOCK_TIMEOUT_S + while True: + try: + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + break + except BlockingIOError: + if time.monotonic() > deadline: + raise CalibrationStoreError(f"Timed out waiting for the lock on {handle.name}") from None + time.sleep(LOCK_RETRY_S) + try: + yield + finally: + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + + +def _replace(staging: str, path: Path) -> None: + """`os.replace`, retried while another process's handle on the target refuses it (Windows only).""" + delay = REPLACE_RETRY_S + for attempt in range(REPLACE_ATTEMPTS): + try: + os.replace(staging, path) + return + except PermissionError: + if attempt + 1 == REPLACE_ATTEMPTS: + raise + time.sleep(delay) + delay *= 2 + + +class CalibrationStore: + def __init__(self, directory: Path) -> None: + self._dir = directory + self._lock = threading.Lock() + + @property + def directory(self) -> Path: + return self._dir + + def path_for(self, device_class_id: str) -> Path: + return self._dir / f"{device_class_id}.json" + + def device_class_ids(self) -> list[str]: + """The device classes that have a file, readable or not.""" + if not self._dir.is_dir(): + return [] + return sorted(path.stem for path in self._dir.iterdir() if _FILE_NAME.match(path.name) and path.is_file()) + + @contextmanager + def _locked(self, device_class_id: str, *, writing: bool) -> Iterator[None]: + """Hold this process's lock and the OS lock on the file's `.lock` sibling. + + Readers lock too: on Windows a file another process holds open cannot be replaced. A reader that cannot create + the lock file (a read-only root) reads without it, since nobody can write there either. + """ + with self._lock: + lock_path = self._dir / f"{device_class_id}.json.lock" + handle: Optional[IO[bytes]] = None + try: + if writing: + self._dir.mkdir(parents=True, exist_ok=True) + handle = open(lock_path, "a+b") + except OSError as error: + if writing: + raise CalibrationStoreError(f"Cannot open {lock_path}: {error}") from error + if handle is None: + yield + return + with handle, _os_lock(handle): + yield + + def _read_unlocked(self, device_class_id: str) -> Optional[StoreRead]: + path = self.path_for(device_class_id) + try: + raw = path.read_bytes() + except FileNotFoundError: + return None + except OSError as error: + raise CalibrationStoreError(f"Cannot read {path}: {error}") from error + file, problem = _parse(raw, device_class_id) + return StoreRead(device_class_id, file, problem) + + def read(self, device_class_id: str) -> Optional[StoreRead]: + """The device class's file, or None when it has none.""" + if not self.path_for(device_class_id).exists(): + return None + with self._locked(device_class_id, writing=False): + return self._read_unlocked(device_class_id) + + def read_raw(self, device_class_id: str) -> Optional[bytes]: + """The file's bytes as they are, readable or not; None when it has none.""" + path = self.path_for(device_class_id) + if not path.exists(): + return None + with self._locked(device_class_id, writing=False): + try: + return path.read_bytes() + except FileNotFoundError: + return None + except OSError as error: + raise CalibrationStoreError(f"Cannot read {path}: {error}") from error + + def change( + self, + device_class_id: str, + apply: Callable[[CalibrationStoreFile], tuple[bool, T]], + *, + create: Optional[CalibrationDeviceClass] = None, + ) -> Optional[T]: + """Apply `apply` to the file as it is on disk now, and write the file when `apply` says it changed it. + + `apply` returns whether it changed the file, and the result to return. Without a file, `create` names the + device class to start one for; without `create` nothing is written and the result is None. Raises + `CalibrationStoreUnusableError` for a file that cannot be read, which stays as it is. + """ + with self._locked(device_class_id, writing=True): + current = self._read_unlocked(device_class_id) + if current is None: + if create is None: + return None + assert create.id == device_class_id + file = CalibrationStoreFile(schema_version=STORE_SCHEMA_VERSION, device_class=create) + elif current.file is None: + raise CalibrationStoreUnusableError( + f"{self.path_for(device_class_id)} is kept as it is and not written: {current.problem}" + ) + else: + file = current.file + changed, result = apply(file) + if changed: + self._write(self.path_for(device_class_id), file) + return result + + @staticmethod + def _write(path: Path, file: CalibrationStoreFile) -> None: + payload = file.model_dump_json(indent=1).encode("utf-8") + try: + descriptor, staging = tempfile.mkstemp(dir=path.parent, prefix=f"{path.name}.", suffix=".tmp") + except OSError as error: + raise CalibrationStoreError(f"Cannot write {path}: {error}") from error + try: + with os.fdopen(descriptor, "wb") as handle: + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + _replace(staging, path) + except BaseException as error: + with suppress(OSError): + os.unlink(staging) + if isinstance(error, OSError): + raise CalibrationStoreError(f"Cannot write {path}: {error}") from error + raise diff --git a/invokeai/app/util/torch_cuda_allocator.py b/invokeai/app/util/torch_cuda_allocator.py index 7e94be57900..25c23478a77 100644 --- a/invokeai/app/util/torch_cuda_allocator.py +++ b/invokeai/app/util/torch_cuda_allocator.py @@ -4,15 +4,30 @@ import os import re import sys +from collections.abc import Mapping from pathlib import Path -# Every variable torch reads its caching-allocator configuration from. `ModelCache._expandable_segments_enabled` keeps -# its own copy of this list: the model cache imports torch, and this module must not. -_ALLOCATOR_CONF_ENV_VARS = ("PYTORCH_ALLOC_CONF", "PYTORCH_CUDA_ALLOC_CONF", "PYTORCH_HIP_ALLOC_CONF") +ALLOCATOR_CONF_ENV_VARS = ("PYTORCH_ALLOC_CONF", "PYTORCH_CUDA_ALLOC_CONF", "PYTORCH_HIP_ALLOC_CONF") +"""Every variable torch reads its caching-allocator configuration from.""" ROCM_WINDOWS_ALLOC_CONF = "expandable_segments:True" +def expandable_segments_enabled(environ: Mapping[str, str] = os.environ) -> bool: + """Whether the torch caching allocator runs in expandable-segments mode, by default in this process. + + The mode is configured through the allocator env vars before torch import (InvokeAI's own + `pytorch_cuda_alloc_conf` setting is plumbed into PYTORCH_CUDA_ALLOC_CONF the same way), so + parsing them is authoritative for the life of the process. It lives here, beside the variables, because this + module imports neither torch nor its callers: the model cache and the working-memory platform. `environ` is + another process's variables, as a working-memory measurement recorded them. + """ + for var in ALLOCATOR_CONF_ENV_VARS: + if "expandable_segments:true" in environ.get(var, "").replace(" ", "").lower(): + return True + return False + + def _installed_torch_version() -> str | None: """The installed torch distribution's version, read from its metadata so torch itself is not imported.""" try: @@ -61,7 +76,7 @@ def apply_rocm_windows_allocator_default(logger: logging.Logger) -> None: Must run before torch is imported. Any allocator variable already in the environment, like an explicit `pytorch_cuda_alloc_conf` (whose caller skips this default), wins. """ - if sys.platform != "win32" or any(os.environ.get(var) for var in _ALLOCATOR_CONF_ENV_VARS): + if sys.platform != "win32" or any(os.environ.get(var) for var in ALLOCATOR_CONF_ENV_VARS): return if not _installed_torch_is_rocm(): return diff --git a/invokeai/backend/flux2/denoise_working_memory.py b/invokeai/backend/flux2/denoise_working_memory.py new file mode 100644 index 00000000000..28344ba811e --- /dev/null +++ b/invokeai/backend/flux2/denoise_working_memory.py @@ -0,0 +1,204 @@ +"""The working memory of one FLUX.2 transformer forward, as the denoise node reserves it. + +The transformer attends over one joint sequence per sample: the caption's tokens, the image's and those of every +reference image, which `Flux2RefImageExtension` concatenates onto the image stream and repeats per sample. On a fused +SDPA kernel its activations scale linearly with that sequence and with the batch, which runs as that many sequences: +`k_denoise_bf16_h` bytes per attended token per sample plus a base of `b_denoise_bf16_h` bytes, shipped +per platform in `backend/working_memory/data/denoise.json`. The cost per token follows the transformer's width (Klein +4B 3072, Klein 9B 4096, [dev] 6144), so each width has a line of its own. The node runs the transformer in bf16 only. +Conditional and unconditional passes run one after the other, so only the longer caption counts. The platform keys +are those of the VAE estimators; for a transformer, `conv_backend` just names the CUDA (`cudnn`) or ROCm (`miopen`) +stack it runs on. + +That line is `denoise_bf16_h` over the attended tokens times the batch (`_line`), and only the plain forward +fits it: every analytic term below comes to zero, and no attention mask is passed. A batch fits too, because the line +prices exactly what a batch adds, its sequences, and a forward's base is paid once whatever the batch. `_line` reads +the fit off the terms themselves, so a term added here takes the forwards it prices out of the fit. The fit takes a +point's demand as measured, so a forward with an analytic term bounds its estimate but fits nothing: a materialized +score matrix's term is an upper bound, and a line fitted on the demand less that term would come out low by its slack. +The spec does not opt in to user calibration (`EstimatorSpec.user_calibrated`): the lines are the shipped fit's alone. + +The lines are fitted per platform and width on fresh-process points of +`scripts/calibrate_flux2_denoise_working_memory.py` (their provenance is in the data): an RTX 4090 with the native +allocator and with `cudaMallocAsync`, and an RX 9060 XT under ROCm 10 with AOTriton's fused kernels, with expandable +segments and with the native allocator on Windows. Per width they run 512 to 2048px with a 512-token caption, 1024 and +1440px with a 77-token one, at 1024px one and three 1024px references and a batch of two, and long sequences: 2048px +with one 2024px reference (33025 tokens), 1536px with one (25857) and 1328px with three 1328px references, resized to +1024px each (19689). They run on a random-init transformer of the loader's geometry with 2 double-stream and 2 +single-stream blocks, so the lines are measured up to 33025 tokens, except where a forward does not fit the card: [dev] +at 33025 tokens on the RTX 4090 under the native allocator, Klein 9B at 33025 on the RX 9060 XT under the native +allocator, and on the RX 9060 XT [dev] above 1440px and with three references, so its [dev] lines there come from 512 to +1440px, one reference included. Reference images and the batch lie on the line of the plain forwards. 4 blocks per +stream measured the same as 2 on the RTX 4090 and 5 to 7% more on the RX 9060 XT at 1024px (Klein 4B and 9B widths). The +real models in a running server demanded less than the 2-block point at the same tokens: 0.96 to 0.99 times it for an +fp8 Klein 4B and 0.96 to 0.97 for an fp8 Klein 9B on the RTX 4090 under the native allocator, 0.81 to 0.95 for a bf16 +Klein 4B and 0.71 to 0.77 for an fp8 Klein 9B (after its 576 MiB fp8 cast term) on the RX 9060 XT with expandable +segments. On ROCm a process's first forward maps ~245 MiB more than its second, which the base carries. + +The node caps neither the image's side nor the number of references (each of several is resized to 1024px, 4096 tokens), +so a longer sequence prices on its line extrapolated linearly. That is expected to hold: on fused attention kernels the +activations grow linearly with the sequence, and from 16896 to 33025 tokens the line reserves a flat 1.05 to 1.13 times +what each measured forward demands under the native allocator and expandable segments. Under `cudaMallocAsync` the +33025-token forwards decide the line (1.05) and the shorter ones sit up to 1.22 below it. + +ROCm with the native allocator on every OS but Windows is not measured, and keeps the `legacy` constants the node +reserved by hand: 0.42 MiB per attended token at the Klein 9B width, scaled linearly by the width and truncated to +whole bytes per width (330300, 440401 and 660601), and a 1 GiB base. They lie above every measured line. Every other +platform nothing measures is priced as a measured one (`priced_as`), and the two fitted entries those borrow, the RTX +4090's and the RX 9060 XT's on Windows with the native allocator, floor every value at the legacy constants there +(`unmeasured_floor`): no unmeasured platform reserves less than the node did by hand. The legacy base also covers the +fp8 weights cast per forward, which an fp8-stored model reserves as a term of its own as well, so it pays that cast +twice wherever they price. + +The analytic terms: + +- Regional prompting's additive attention bias, one ``S x S`` tensor in the compute dtype broadcast over the batch and + the heads (`Flux2RegionalPromptingExtension.get_joint_attention_kwargs`). The node passes no mask with reference + images, which the mask does not cover. +- Where SDPA has no fused kernel for these shapes it materializes the score matrices, shaped (batch, heads, S, S): + `sdpa_score_matrix_bytes` prices them only where they are really built. A mask narrows which fused kernels a build + can use, so the call is told whether one is passed. +- Half a GiB per LoRA and sample: a sidecar patch adds an activation branch per patched layer. +- The dequantization copy an int8_tensorwise, nvfp4 or GGUF build materializes per quantized linear and forward, + alive alongside the activations and invisible to them (`peak_dequant_transient_bytes`); zero for every other build. +- The compute-dtype copies of a layer's float8-stored weights that FP8 Storage, or a kept fp8 checkpoint, widens per + forward (`peak_fp8_cast_transient_bytes`); zero for every other build. +""" + +from collections.abc import Mapping +from typing import Final, Optional, cast + +import torch + +from invokeai.backend.quantization.dequantizing_linear import peak_dequant_transient_bytes +from invokeai.backend.util.attention import sdpa_score_matrix_bytes +from invokeai.backend.util.fp8 import peak_fp8_cast_transient_bytes +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm +from invokeai.backend.working_memory.fit import Line, off_line_reasons +from invokeai.backend.working_memory.platform import platform_for +from invokeai.backend.working_memory.spec import EstimatorSpec, register + +FLUX2_ATTENTION_HEAD_DIM: Final = 128 +"""Every variant's head width; the head count follows the hidden size (Klein 4B 24, Klein 9B 32, [dev] 48).""" +FLUX2_HIDDEN_SIZES: Final = (3072, 4096, 6144) +"""The transformer widths with a line: Klein 4B, Klein 9B and [dev].""" + +_LORA_PATCH_BYTES: Final = 2**29 +_DTYPE_NAMES: Final[Mapping[torch.dtype, str]] = {torch.bfloat16: "bf16"} +"""The dtypes the node runs the transformer in.""" +_LINES: Final = tuple(f"denoise_{dtype}_h{width}" for dtype in _DTYPE_NAMES.values() for width in FLUX2_HIDDEN_SIZES) +_COEFFICIENTS: Final = frozenset(f"{part}_{line}" for part in ("k", "b") for line in _LINES) + + +def _suffix(features: Mapping[str, FeatureValue]) -> Optional[str]: + """The line a forward's dtype and width price on, None for a pair that has none.""" + dtype, width = features.get("dtype"), features.get("hidden_size") + suffix = f"denoise_{dtype}_h{width}" + return suffix if suffix in _LINES else None + + +def _line(features: Mapping[str, FeatureValue]) -> Optional[Line]: + """The forward's line, `denoise__h` over the attended tokens of every sample. Only the plain forward + fits it: no attention mask, and every analytic term of `_terms` zero (see the module docstring). The label names + what puts a forward off the line: the mask, and each analytic term that is not zero, in bytes. A dtype or width + without a line names none.""" + suffix = _suffix(features) + tokens, batch = features.get("tokens"), features.get("batch_size") + if suffix is None or not isinstance(tokens, int) or not isinstance(batch, int): + return None + off_line = off_line_reasons(_terms, features, _COEFFICIENTS) + references = features.get("reference_tokens") + shape = "+".join( + ( + str(features.get("image_tokens")), + *([f"{references}ref"] if references else []), + f"{features.get('text_tokens')}tok", + ) + ) + batched = [f"b{batch}"] if batch != 1 else [] + label = "-".join(("denoise", str(features["dtype"]), f"h{features['hidden_size']}", shape, *batched, *off_line)) + return Line(suffix, tokens * batch, not off_line, label) + + +def _terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: + suffix = _suffix(features) + if suffix is None: + raise ValueError(f"no denoise line for {features.get('dtype')} at width {features.get('hidden_size')}") + batch = cast(int, features["batch_size"]) + return ( + WorkingMemoryTerm.fitted("activations", f"k_{suffix}", cast(int, features["tokens"]) * batch, values), + WorkingMemoryTerm.fitted("base", f"b_{suffix}", 1, values), + WorkingMemoryTerm("regional_attention_bias", cast(int, features["regional_bias_bytes"]), True), + WorkingMemoryTerm("attention_scores", cast(int, features["score_matrix_bytes"]), True), + WorkingMemoryTerm("lora_patches", cast(int, features["lora_count"]) * batch * _LORA_PATCH_BYTES, True), + WorkingMemoryTerm("dequant_transient", cast(int, features["dequant_transient_bytes"]), True), + WorkingMemoryTerm("fp8_cast_transient", cast(int, features["fp8_cast_transient_bytes"]), True), + ) + + +FLUX2_DENOISE_SPEC = register( + EstimatorSpec( + id="denoise.flux2", + version=1, + coefficients=_COEFFICIENTS, + platform_keys=("conv_backend", "allocator"), + terms=_terms, + line=_line, + ) +) + + +def estimate_flux2_denoise_working_memory( + transformer: torch.nn.Module, + *, + hidden_size: int, + image_seq_len: int, + ref_image_seq_len: int, + text_seq_len: int, + batch_size: int, + lora_count: int, + regional_attn_mask: Optional[torch.Tensor] = None, + device: torch.device, + dtype: torch.dtype, +) -> WorkingMemoryEstimate: + """The working memory one forward of `transformer` needs on `device` in `dtype`. + + `transformer` is the model about to be locked, read unlocked for its dequantization and fp8 cast transients, and + `hidden_size` its width. `image_seq_len`, `ref_image_seq_len` and `text_seq_len` are one sample's token counts, the + longer caption of a CFG pair, and `batch_size` the samples the forward runs. `regional_attn_mask` is regional + prompting's mask, which the forward passes as an additive bias. + """ + dtype_name = _DTYPE_NAMES.get(dtype) + if dtype_name is None or hidden_size not in FLUX2_HIDDEN_SIZES: + raise ValueError( + f"Working-memory estimator '{FLUX2_DENOISE_SPEC.id}' has no {dtype} line at width {hidden_size} on " + f"{platform_for(device)}: the transformer runs in a dtype or at a width it is not measured for." + ) + tokens = image_seq_len + ref_image_seq_len + text_seq_len + has_attention_mask = regional_attn_mask is not None + features: dict[str, FeatureValue] = { + "hidden_size": hidden_size, + "image_tokens": image_seq_len, + "reference_tokens": ref_image_seq_len, + "text_tokens": text_seq_len, + "tokens": tokens, + "batch_size": batch_size, + "dtype": dtype_name, + "has_attention_mask": has_attention_mask, + "regional_bias_bytes": regional_attn_mask.numel() * dtype.itemsize if regional_attn_mask is not None else 0, + "score_matrix_bytes": sdpa_score_matrix_bytes( + device=device, + dtype=dtype, + num_heads=(hidden_size // FLUX2_ATTENTION_HEAD_DIM) * batch_size, + head_dim=FLUX2_ATTENTION_HEAD_DIM, + seq_len=tokens, + has_attn_mask=has_attention_mask, + # The attention goes through diffusers' `dispatch_attention_fn`, which can route around torch's SDPA, + # including to a forced `math` backend. + via_diffusers_dispatch=True, + ), + "lora_count": lora_count, + "dequant_transient_bytes": peak_dequant_transient_bytes(transformer, dtype), + "fp8_cast_transient_bytes": peak_fp8_cast_transient_bytes(transformer, dtype), + } + return FLUX2_DENOISE_SPEC.estimate(features, device) diff --git a/invokeai/backend/model_manager/load/load_base.py b/invokeai/backend/model_manager/load/load_base.py index fddf9a368dc..cdddcbb3ce6 100644 --- a/invokeai/backend/model_manager/load/load_base.py +++ b/invokeai/backend/model_manager/load/load_base.py @@ -26,6 +26,7 @@ ) from invokeai.backend.model_manager.taxonomy import AnyModel, SubModelType from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.working_memory import probe from invokeai.backend.working_memory.estimate import WorkingMemoryEstimate @@ -181,18 +182,57 @@ def _lock_paced(self, working_mem_bytes: Optional[int]) -> None: # invariant turning this loop into a spin. assert settled, "uncapped continue_lock pass did not settle" - def __enter__(self) -> AnyModel: - self._lock_paced(None) - self._end_first_use_window() + def _lock_on_device(self, working_mem_bytes: Optional[int], *, offload_others: bool = False) -> None: + """Lock the model onto its compute device and repair its required tensors there. Nothing stays locked if this + raises. + + While a calibration scope is open anywhere (`probe.armed`), the working-memory probe hears of the lock from its + start until it has settled with the tensors repaired: inside a measured window on this thread that interval is + a co-lock, which moves weights and is kept out of the window's peak; on another thread it voids the window. + The lock is in `probe.locking` from before `armed` is read until it has settled or failed, so a window that + opens on this cache meanwhile is void even if this read missed the scope arming. + + `offload_others`: first move every other unlocked model on the device to RAM whole, within that interval. A + calibration lock asks for it (`model_on_device`): the lock itself offloads only the shortfall it predicts and + splits the last model it moves, and a split can free a little less than predicted, so the measured model would + run partly resident and its window fit no line. Locked models, a window's co-locks among them, stay. + """ + observed = locked = False + probe.locking.append(self._cache) try: + observed = probe.armed + if observed: + probe.lock_started(self._cache, self._cache_record) + if offload_others: + # The global load lock before the cache's own, as `_lock_paced` takes them: a move to RAM loads state + # dicts, which must not overlap a model construction. + with MODEL_LOAD_LOCK.read_lock(): + self._cache.offload_unlocked_models_from_vram(keep_cache_keys=(self._cache_record.key,)) + self._lock_paced(working_mem_bytes) + locked = True + self._end_first_use_window() self.repair_required_tensors_on_device() - return self.model - except Exception: - self._cache.unlock(self._cache_record) + except BaseException: + if locked: + self._cache.unlock(self._cache_record) + locked = False raise + finally: + probe.locking.remove(self._cache) + if observed: + probe.lock_settled(self._cache, self._cache_record, locked) - def __exit__(self, *args: Any, **kwargs: Any) -> None: + def _unlock(self) -> None: self._cache.unlock(self._cache_record) + if probe.armed: + probe.unlocked(self._cache, self._cache_record) + + def __enter__(self) -> AnyModel: + self._lock_on_device(None) + return self.model + + def __exit__(self, *args: Any, **kwargs: Any) -> None: + self._unlock() @contextmanager def model_on_device( @@ -203,27 +243,46 @@ def model_on_device( :param working_mem_bytes: The amount of working memory to keep available on the compute device when loading the model. :param working_memory: A working-memory estimate whose `reservation_bytes` is kept available instead; see - `invokeai.backend.working_memory`. Pass this or `working_mem_bytes`, not both. + `invokeai.backend.working_memory`. Pass this or `working_mem_bytes`, not both. Inside a calibration scope + (`working_memory.probe`) on a GPU, the lock first moves every other unlocked model on the device to RAM, + reserves calibration headroom on top, and the probe measures the operation from the settled lock until the + unlock. """ if isinstance(working_mem_bytes, WorkingMemoryEstimate): raise TypeError( "model_on_device() got a WorkingMemoryEstimate as working_mem_bytes; pass it as working_memory=." ) + calibration: Optional[probe.ProbeScope] = None if working_memory is not None: if working_mem_bytes is not None: raise ValueError("Pass either working_mem_bytes or working_memory to model_on_device(), not both.") working_mem_bytes = working_memory.reservation_bytes - InvokeAILogger.get_logger(__name__).debug( + message = ( f"Working memory for {self._cache_record.key}: {working_memory.estimator} reserves " f"{working_mem_bytes / 2**20:.0f} MB ({working_memory.source})" ) - self._lock_paced(working_mem_bytes) - self._end_first_use_window() + scope = probe.current_scope() + if scope is not None and self.compute_device.type != "cpu": + calibration = scope + working_mem_bytes = probe.calibration_reservation(working_mem_bytes) + message += f"; the calibration lock reserves {working_mem_bytes / 2**20:.0f} MB" + InvokeAILogger.get_logger(__name__).debug(message) + self._lock_on_device(working_mem_bytes, offload_others=calibration is not None) + window = None try: - self.repair_required_tensors_on_device() + if calibration is not None: + assert working_memory is not None and working_mem_bytes is not None + window = calibration.open_window(self._cache, self._cache_record, working_memory, working_mem_bytes) yield (self._cache_record.cached_model.get_cpu_state_dict(), self._cache_record.cached_model.model) + except BaseException as error: + if window is not None: + window.close(error) + window = None + raise finally: - self._cache.unlock(self._cache_record) + if window is not None: + window.close(None) + self._unlock() @property def model(self) -> AnyModel: @@ -234,11 +293,13 @@ def model(self) -> AnyModel: def model_in_ram(self) -> Generator[AnyModel, None, None]: """Pin the model's cache record in RAM without moving the model to its execution device.""" self._cache.lock_in_ram(self._cache_record) - self._end_first_use_window() + if probe.armed: + probe.locked_in_ram(self._cache, self._cache_record) try: + self._end_first_use_window() yield self.model finally: - self._cache.unlock(self._cache_record) + self._unlock() @property def compute_device(self) -> torch.device: diff --git a/invokeai/backend/model_manager/load/model_cache/cache_record.py b/invokeai/backend/model_manager/load/model_cache/cache_record.py index 14fb4879af4..5a30eb74712 100644 --- a/invokeai/backend/model_manager/load/model_cache/cache_record.py +++ b/invokeai/backend/model_manager/load/model_cache/cache_record.py @@ -114,9 +114,11 @@ def lock(self) -> None: self._locks += 1 def unlock(self) -> None: - """Unlock this record.""" + """Unlock this record. An unlock without a lock to release raises and changes nothing, so the count stays + right for the next lock and for the cache's `locks_held`.""" + if self._locks == 0: + raise RuntimeError(f"Model cache entry {self.key} was unlocked more often than it was locked") self._locks -= 1 - assert self._locks >= 0 @property def is_locked(self) -> bool: diff --git a/invokeai/backend/model_manager/load/model_cache/model_cache.py b/invokeai/backend/model_manager/load/model_cache/model_cache.py index 3691f374425..7932c6fdd64 100644 --- a/invokeai/backend/model_manager/load/model_cache/model_cache.py +++ b/invokeai/backend/model_manager/load/model_cache/model_cache.py @@ -1,6 +1,5 @@ import gc import logging -import os import queue import sys import threading @@ -15,6 +14,7 @@ import psutil import torch +from invokeai.app.util.torch_cuda_allocator import expandable_segments_enabled from invokeai.backend.model_manager.load.memory_snapshot import MemorySnapshot from invokeai.backend.model_manager.load.model_cache.cache_record import CacheRecord from invokeai.backend.model_manager.load.model_cache.cache_stats import CacheStats @@ -39,20 +39,6 @@ from invokeai.backend.util.logging import InvokeAILogger from invokeai.backend.util.prefix_logger_adapter import PrefixedLoggerAdapter - -def _expandable_segments_enabled() -> bool: - """Whether the torch caching allocator runs in expandable-segments mode. - - The mode is configured through the allocator env vars before torch import (InvokeAI's own - `pytorch_cuda_alloc_conf` setting is plumbed into PYTORCH_CUDA_ALLOC_CONF the same way), so - parsing them is authoritative for the life of the process. - """ - for var in ("PYTORCH_ALLOC_CONF", "PYTORCH_CUDA_ALLOC_CONF", "PYTORCH_HIP_ALLOC_CONF"): - if "expandable_segments:true" in os.environ.get(var, "").replace(" ", "").lower(): - return True - return False - - # Size of a GB in bytes. GB = 2**30 @@ -630,6 +616,8 @@ def __init__( self._cached_models: Dict[str, CacheRecord] = {} self._cache_stack: List[str] = [] + # Locks on this cache's records, detached ones included (see `locks_held`); changed only with `_lock` held. + self._locks_held = 0 self._ram_cache_size_bytes = self._calc_ram_available_to_model_cache() @@ -1888,6 +1876,28 @@ def _claim_first_use(self, cache_entry: CacheRecord) -> Optional[FirstUseClaim]: self.release_first_use_hold(cache_entry, hold_epoch) raise + @synchronized + def locks_held(self) -> int: + """How many locks this cache's records hold right now: VRAM and RAM pins of every thread together. + + The working-memory probe compares it with the locks its own thread took, so that a window opening while + another thread holds a model on this device knows it shares the device. Counted as locks are taken and + released, not over the cached records: a record dropped while locked (the VRAM-move error paths) still holds + its model on the device until its holder unlocks it. + """ + return self._locks_held + + def _pin(self, cache_entry: CacheRecord) -> None: + """Lock `cache_entry` and count the lock in `locks_held`. Caller holds the lock.""" + cache_entry.lock() + self._locks_held += 1 + + def _unpin(self, cache_entry: CacheRecord) -> None: + """Release a lock `_pin` took. Caller holds the lock. The record refuses an unlock it has no lock for before + either count changes, so a double unlock leaves `locks_held` right.""" + cache_entry.unlock() + self._locks_held -= 1 + @synchronized @record_activity def lock_in_ram(self, cache_entry: CacheRecord) -> None: @@ -1902,7 +1912,7 @@ def lock_in_ram(self, cache_entry: CacheRecord) -> None: "the RAM cache. This is a sign that the model loading order is non-optimal in the invocation code " "(See https://github.com/invoke-ai/InvokeAI/issues/7513)." ) - cache_entry.lock() + self._pin(cache_entry) cache_entry.awaiting_first_use = False @synchronized @@ -1930,7 +1940,7 @@ def lock( "(See https://github.com/invoke-ai/InvokeAI/issues/7513)." ) # cache_entry = self._cached_models[key] - cache_entry.lock() + self._pin(cache_entry) # End of the post-admission grace: from here the entry is pinned by its lock count, and # after the final unlock it is ordinary evictable cache content. cache_entry.awaiting_first_use = False @@ -1966,10 +1976,10 @@ def lock( ) except torch.OutOfMemoryError: self._logger.warning("Insufficient GPU memory to load model. Aborting") - cache_entry.unlock() + self._unpin(cache_entry) raise except Exception: - cache_entry.unlock() + self._unpin(cache_entry) raise self._log_cache_state() @@ -2014,10 +2024,10 @@ def continue_lock( ) except torch.OutOfMemoryError: self._logger.warning("Insufficient GPU memory to load model. Aborting") - cache_entry.unlock() + self._unpin(cache_entry) raise except Exception: - cache_entry.unlock() + self._unpin(cache_entry) raise if settled: @@ -2036,7 +2046,7 @@ def unlock(self, cache_entry: CacheRecord) -> None: "(See https://github.com/invoke-ai/InvokeAI/issues/7513)." ) # cache_entry = self._cached_models[key] - cache_entry.unlock() + self._unpin(cache_entry) self._logger.debug( f"Unlocked model {cache_entry.key} (Type: {cache_entry.cached_model.model.__class__.__name__})" ) @@ -2231,7 +2241,7 @@ def _release_allocator_blocks_for_reserve(self, working_mem_bytes: Optional[int] driver_free, _ = TorchDevice.cuda_mem_get_info(self._execution_device) if driver_free >= reserve: return - if _expandable_segments_enabled(): + if expandable_segments_enabled(): # Freed pages inside a segment are not counted as reclaimable there, but empty_cache() unmaps them. releasable = torch.cuda.memory_reserved(self._execution_device) - torch.cuda.memory_allocated( self._execution_device @@ -2403,7 +2413,7 @@ def _get_reclaimable_allocator_bytes(self) -> int: `_offload_unlocked_models` runs one after each offload in that mode; what it cannot do is make the credit trustworthy before it runs. """ - if _expandable_segments_enabled(): + if expandable_segments_enabled(): return 0 try: if self._execution_device.type == "cuda": @@ -2618,7 +2628,7 @@ def _offload_unlocked_models( # them -- so it is deliberately loop-local: it stops this loop offloading everything, and # the load that follows re-measures for itself (`lock`, `make_room_in_vram`) and streams # from RAM if the room did not materialize. - empty_cache_per_offload = _expandable_segments_enabled() + empty_cache_per_offload = expandable_segments_enabled() vram_bytes_freed_uncounted = 0 # TODO(ryand): Give more thought to the offloading policy used here. cache_entries_increasing_size = sorted(self._cached_models.values(), key=lambda x: x.cached_model.total_bytes()) @@ -3149,6 +3159,23 @@ def offload_models_from_vram_except(self, keep_model_keys: Collection[str]) -> i keep = set(keep_model_keys) return self._offload_entries_from_vram(lambda key: key.split(":", 1)[0] not in keep) + @synchronized + def offload_unlocked_models_from_vram(self, keep_cache_keys: Collection[str] = ()) -> int: + """Move every unlocked entry but those in `keep_cache_keys` from VRAM to RAM whole, keeping it cached. + + `keep_cache_keys` are exact cache keys: keeping a submodel (`:vae`) keeps none of its siblings, unlike + `offload_models_from_vram_except`. Each entry leaves whole, where `lock()` moves out only the shortfall it + predicts and so splits the last model it reaches. A working-memory calibration calls this before its measured + lock (`LoadedModelWithoutConfig.model_on_device`), so that lock has nothing to split. + + Returns the number of VRAM bytes freed. + """ + if not _has_dedicated_vram(self._execution_device): + # CPU, MPS, integrated GPUs: their "VRAM" is system RAM, so moving models out frees nothing. + return 0 + keep = set(keep_cache_keys) + return self._offload_entries_from_vram(lambda key: key not in keep) + def _offload_entries_from_vram(self, select: Callable[[str], bool]) -> int: bytes_freed = 0 for key, entry in list(self._cached_models.items()): diff --git a/invokeai/backend/util/device_pool.py b/invokeai/backend/util/device_pool.py index e9044160689..fbd68f7af16 100644 --- a/invokeai/backend/util/device_pool.py +++ b/invokeai/backend/util/device_pool.py @@ -75,7 +75,10 @@ def set_release_listener(self, listener: Optional[Callable[[], None]]) -> None: self._on_release = listener def set_generation_devices(self, devices: list[torch.device]) -> None: - """Register the full set of generation devices (called once at processor startup). + """Register the full set of generation devices, replacing those registered before (a borrow in flight is + dropped). The session processor's `register_generation_devices` calls it once: from the composition root + (`ApiDependencies.initialize`) before the invoker starts any service, or from the processor's `start` when + nothing did. Only GPU devices participate in idle-offload; others are ignored. """ diff --git a/invokeai/backend/util/devices.py b/invokeai/backend/util/devices.py index b648adc0c8f..8e1c31479a3 100644 --- a/invokeai/backend/util/devices.py +++ b/invokeai/backend/util/devices.py @@ -2,7 +2,7 @@ import threading from collections import Counter, defaultdict from functools import wraps -from typing import Dict, Literal, Optional, Union +from typing import Dict, Literal, Optional, Protocol, Union import torch from deprecated import deprecated @@ -25,6 +25,19 @@ _XPU_MEM_FALLBACK_WARNED: set[str] = set() +class EmptyCacheObserver(Protocol): + def before_empty_cache(self) -> None: ... + + def after_empty_cache(self) -> None: ... + + +EMPTY_CACHE_OBSERVERS: list[EmptyCacheObserver] = [] +"""Told about every `TorchDevice.empty_cache` that releases memory, just before and just after the release, in +registration order. The working-memory probe registers here while a calibration scope is open, so that a release inside +a measured window ends a segment there (`invokeai.backend.working_memory.probe`); otherwise the list is empty and a +release pays one truth test for it. Observers must not raise.""" + + def _xpu_is_available() -> bool: """Return True if a torch XPU (Intel GPU) device is available.""" return hasattr(torch, "xpu") and torch.xpu.is_available() @@ -447,14 +460,21 @@ def empty_cache(cls, force: bool = False) -> bool: # Clear before running: a skip that races in after this point re-sets the flag, so a # request is never lost, only (harmlessly) repeated. cls._empty_cache_deferred.clear() - if torch.backends.mps.is_available(): - torch.mps.empty_cache() - if torch.cuda.is_available(): - # A forced call must get past `install_peer_aware_empty_cache`'s wrapper, which would defer it again. - cuda_empty_cache = torch.cuda.empty_cache - (getattr(cuda_empty_cache, "__wrapped__", cuda_empty_cache) if force else cuda_empty_cache)() - if _xpu_is_available(): - torch.xpu.empty_cache() + observers = tuple(EMPTY_CACHE_OBSERVERS) if EMPTY_CACHE_OBSERVERS else () + for observer in observers: + observer.before_empty_cache() + try: + if torch.backends.mps.is_available(): + torch.mps.empty_cache() + if torch.cuda.is_available(): + # A forced call must get past `install_peer_aware_empty_cache`'s wrapper, which would defer it again. + cuda_empty_cache = torch.cuda.empty_cache + (getattr(cuda_empty_cache, "__wrapped__", cuda_empty_cache) if force else cuda_empty_cache)() + if _xpu_is_available(): + torch.xpu.empty_cache() + finally: + for observer in observers: + observer.after_empty_cache() return True @classmethod diff --git a/invokeai/backend/util/fp8.py b/invokeai/backend/util/fp8.py index ce4d5b7a753..17e8ffc9439 100644 --- a/invokeai/backend/util/fp8.py +++ b/invokeai/backend/util/fp8.py @@ -9,6 +9,9 @@ arithmetic on it (`"pow_cuda" not implemented for 'Float8_e4m3fn'`). So any code that reads a model's dtype in order to build or cast *tensors* (latents, noise, conditioning, control images, LoRA patch weights) must use `get_model_compute_dtype()` instead of `model.dtype`. + +The widened weights are memory one forward needs beyond the model's resident size, which a +working-memory estimate prices with `peak_fp8_cast_transient_bytes()`. """ import torch @@ -72,3 +75,42 @@ def get_model_compute_dtype(model: torch.nn.Module) -> torch.dtype: "expect a dtype mismatch during the forward pass." ) return fallback + + +def peak_fp8_cast_transient_bytes(model: torch.nn.Module, compute_dtype: torch.dtype) -> int: + """Peak bytes one forward transiently needs to run this model's float8-stored weights in `compute_dtype`. + + Two paths keep weights in float8 and widen them per forward, and both leave the stored tensors resident: + + - FP8 Storage (`ModelLoader._wrap_forward_with_fp8_cast`): a layer's hooks widen its params to the compute + dtype when its forward starts and put the stored ones back when it ends, so the widened copies die with the + layer's forward. + - float8 weights a checkpoint ships and the loader keeps packed (`fp8_scaled.should_keep_fp8_weights`, or FP8 + Storage on a scaled-fp8 checkpoint): `CustomLinear` dequantizes them per forward with `dequantize_weight`, + which holds the widened weight and its product with the `weight_scale` at once, two compute-dtype copies. + + The hooked layers are leaf modules and run one after another, each freeing its copies before the next widens, so + the peak is the largest single layer's: its float8 params at the compute dtype, twice for one with a + `weight_scale`. Zero for a model that holds no float8 parameter. A quantized payload in a tensor subclass (SDNQ, + GGUF) is its scheme's to dequantize, not one of these casts, and is not counted. + + Assumes the layers widen to `compute_dtype`, the dtype the forward runs in, as the loaders record it + (`get_model_compute_dtype`); a model widened to another dtype would be cast again per layer, which this does not + count. Counts every kept checkpoint weight, also where `torch._scaled_mm` multiplies it in float8 without widening + it: whether it does is decided per forward (the fp8 matmul setting, the device, alignment, a producer's + full-precision hint, the weight's device under partial loading), and the miss over-reserves one layer. Does not + count the host-to-device copy of a layer that partial loading left on the CPU, which every partially loaded weight + costs, whatever its dtype. + """ + peak = 0 + for module in model.modules(): + stored = sum( + param.numel() + for param in module.parameters(recurse=False) + if type(param.data) is torch.Tensor and param.dtype in FP8_STORAGE_DTYPES + ) + if not stored: + continue + copies = 2 if getattr(module, "weight_scale", None) is not None else 1 + peak = max(peak, copies * stored * compute_dtype.itemsize) + return peak diff --git a/invokeai/backend/util/nvml.py b/invokeai/backend/util/nvml.py new file mode 100644 index 00000000000..cffb36efd67 --- /dev/null +++ b/invokeai/backend/util/nvml.py @@ -0,0 +1,146 @@ +"""Minimal NVML queries through ctypes: this process's memory on an NVIDIA device, and the driver version. + +NVML ships with the NVIDIA driver (``libnvidia-ml.so.1`` on Linux, ``nvml.dll`` in System32 on Windows), so this needs +no package. Under WDDM, Windows reports no per-process memory through NVML; the working-memory probe reads it through +D3DKMT there (`wddm.process_local_usage`), and uses NVML on Windows for the driver version only. + +Everything here is best-effort: any failure yields ``None`` ("unknown") rather than raising. +""" + +import ctypes +import functools +import os +import sys +import threading +from typing import Optional + +import torch + +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.util.pci import pci_address + +_NVML_SUCCESS = 0 +_NVML_ERROR_INSUFFICIENT_SIZE = 7 +# `usedGpuMemory` when the driver cannot attribute memory to the process (always so under WDDM). +_NVML_VALUE_NOT_AVAILABLE = 0xFFFFFFFFFFFFFFFF +# NVML_SYSTEM_DRIVER_VERSION_BUFFER_SIZE. +_DRIVER_VERSION_BUFFER_SIZE = 80 +# Room for processes that start between the count and the fill call. +_SPARE_PROCESS_ENTRIES = 8 + +_LIBRARY_NAMES = ("nvml.dll",) if sys.platform == "win32" else ("libnvidia-ml.so.1", "libnvidia-ml.so") + + +class _ProcessInfo(ctypes.Structure): + """``nvmlProcessInfo_t``, the entry ``nvmlDeviceGetComputeRunningProcesses_v3`` fills.""" + + _fields_ = [ + ("pid", ctypes.c_uint), + ("usedGpuMemory", ctypes.c_ulonglong), + ("gpuInstanceId", ctypes.c_uint), + ("computeInstanceId", ctypes.c_uint), + ] + + +_lock = threading.Lock() +# Per torch device index: the NVML device handle, or None once the lookup has failed. +_handles: dict[int, Optional[ctypes.c_void_p]] = {} + + +def _configure_prototypes(lib: ctypes.CDLL) -> None: + """Declare every entry point used here, so ctypes passes handles at full pointer width.""" + prototypes = { + "nvmlInit_v2": [], + "nvmlDeviceGetHandleByPciBusId_v2": [ctypes.c_char_p, ctypes.POINTER(ctypes.c_void_p)], + "nvmlDeviceGetComputeRunningProcesses_v3": [ + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_uint), + ctypes.c_void_p, + ], + "nvmlSystemGetDriverVersion": [ctypes.c_char_p, ctypes.c_uint], + } + for name, argtypes in prototypes.items(): + fn = getattr(lib, name) + fn.argtypes = argtypes + fn.restype = ctypes.c_int + + +@functools.lru_cache(maxsize=1) +def _load() -> Optional[ctypes.CDLL]: + """Open and initialise NVML once per process; None where there is no NVIDIA driver.""" + for name in _LIBRARY_NAMES: + try: + lib = ctypes.CDLL(name) + except OSError: + continue + try: + _configure_prototypes(lib) + if lib.nvmlInit_v2() != _NVML_SUCCESS: + return None + except Exception as exc: + InvokeAILogger.get_logger(__name__).debug(f"NVML unusable: {exc}") + return None + return lib + return None + + +def _handle_for(lib: ctypes.CDLL, device: torch.device) -> Optional[ctypes.c_void_p]: + index = device.index if device.index is not None else torch.cuda.current_device() + with _lock: + if index not in _handles: + handle = ctypes.c_void_p() + found = lib.nvmlDeviceGetHandleByPciBusId_v2(pci_address(index).encode(), ctypes.byref(handle)) + _handles[index] = handle if found == _NVML_SUCCESS else None + return _handles[index] + + +def process_used_memory(device: torch.device) -> Optional[int]: + """Bytes NVML attributes to this process on `device`: everything the process holds there, the CUDA context + included. ``None`` off CUDA builds, without NVML, and wherever the driver does not attribute memory to processes.""" + if device.type != "cuda" or torch.version.hip is not None: + return None + try: + lib = _load() + if lib is None: + return None + handle = _handle_for(lib, device) + if handle is None: + return None + count = ctypes.c_uint(0) + status = lib.nvmlDeviceGetComputeRunningProcesses_v3(handle, ctypes.byref(count), None) + if status != _NVML_ERROR_INSUFFICIENT_SIZE: + # Success with an empty buffer means no process at all, so not this one either. + return None + count = ctypes.c_uint(count.value + _SPARE_PROCESS_ENTRIES) + infos = (_ProcessInfo * count.value)() + if lib.nvmlDeviceGetComputeRunningProcesses_v3(handle, ctypes.byref(count), infos) != _NVML_SUCCESS: + return None + pid = os.getpid() + for info in infos[: count.value]: + if info.pid == pid: + return None if info.usedGpuMemory == _NVML_VALUE_NOT_AVAILABLE else int(info.usedGpuMemory) + return None + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"NVML process memory lookup for {device} failed", exc_info=True) + return None + + +def driver_version() -> Optional[str]: + """The NVIDIA driver version, e.g. ``"581.57"``, or None without NVML.""" + try: + lib = _load() + if lib is None: + return None + buffer = ctypes.create_string_buffer(_DRIVER_VERSION_BUFFER_SIZE) + if lib.nvmlSystemGetDriverVersion(buffer, _DRIVER_VERSION_BUFFER_SIZE) != _NVML_SUCCESS: + return None + return buffer.value.decode("ascii", errors="replace") or None + except Exception: + return None + + +def reset_cache() -> None: + """Forget the loader and the device handles (tests only). NVML itself stays initialised.""" + with _lock: + _handles.clear() + _load.cache_clear() diff --git a/invokeai/backend/util/pci.py b/invokeai/backend/util/pci.py new file mode 100644 index 00000000000..765f9431d03 --- /dev/null +++ b/invokeai/backend/util/pci.py @@ -0,0 +1,17 @@ +"""Where a CUDA or HIP device sits on the PCI bus: what links a torch device to the driver interfaces that know it by +location (D3DKMT adapters, NVML, DRM fdinfo).""" + +import torch + + +def pci_location(index: int) -> tuple[int, int, int]: + """``(domain, bus, device)`` of torch's CUDA or HIP device `index`.""" + props = torch.cuda.get_device_properties(index) + return int(props.pci_domain_id), int(props.pci_bus_id), int(props.pci_device_id) + + +def pci_address(index: int) -> str: + """The device's address as ``domain:bus:device.function`` in lower-case hex, e.g. ``0000:03:00.0``: how DRM + fdinfo spells it, and one of the forms NVML parses.""" + domain, bus, device = pci_location(index) + return f"{domain:04x}:{bus:02x}:{device:02x}.0" diff --git a/invokeai/backend/util/vae_working_memory.py b/invokeai/backend/util/vae_working_memory.py index bc2f3123d64..2e068d01824 100644 --- a/invokeai/backend/util/vae_working_memory.py +++ b/invokeai/backend/util/vae_working_memory.py @@ -16,6 +16,8 @@ from invokeai.backend.util.logging import InvokeAILogger from invokeai.backend.util.vae_tiling_scope import resolve_tile_size from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm +from invokeai.backend.working_memory.fit import convolution_line +from invokeai.backend.working_memory.platform import platform_for from invokeai.backend.working_memory.spec import EstimatorSpec, register # Share of the device's memory an untiled decode may claim before the decode nodes tile it up front (see @@ -77,7 +79,7 @@ def should_pretile_vae_decode( return True -# The diffusers AutoencoderKL (SD1/SDXL, SD3, CogView4) and the FLUX.1 AutoEncoder run the same +# The diffusers AutoencoderKL (SD1/SDXL, SD3) and the FLUX.1 AutoEncoder run the same # mid-block self-attention as the FLUX.2 VAE: one head over the 512-channel width, on the # 8x-downsampled grid. Where that attention runs on the math kernel (see `sdpa_score_matrix_bytes`) # its score matrix is the dominant term for a large untiled image, exactly as it is for FLUX.2; @@ -114,25 +116,87 @@ def _vae_mid_block_score_matrix_bytes( ) -# The SD1/SDXL, SD3, CogView4, FLUX.1 and FLUX.2 VAEs run the same diffusers AutoencoderKL convolution stack, and pay -# the same for it: peak reserved bytes per output pixel per element byte, one fitted coefficient per operation -# (`k_decode`, `k_encode`) and convolution backend. The values ship in `backend/working_memory/data/vae.json`, with the -# points they were fitted on and the per-card table behind them; `scripts/calibrate_flux2_working_memory.py --only vae` -# reproduces that table on any build. +# The SD1/SDXL, SD3, FLUX.1 and FLUX.2 VAEs run the same diffusers AutoencoderKL convolution stack, and pay the same +# for it: a line in the output's pixel count per operation and the dtype it runs in, `k__` +# bytes per pixel plus `b__` bytes the process pays whatever the size, shipped per convolution +# backend and allocator (`platform_for`). The values ship in `backend/working_memory/data/vae.json` with the points +# they are fitted on (`backend/working_memory/fit.py`): calibration runs in a running server measure them on a card, +# and `scripts/working_memory/build_shipped.py` turns their exports into shipped points. Every spec here declares that +# line (`fit.convolution_line`) and opts in to user calibration (`user_calibrated`), so a calibration run's decode +# windows refit these lines per device. # -# MIOpen's convolution workspaces are simply larger than cuDNN's -- ~1.6x for a decode, ~2.4x for an encode. This is -# not the attention term: it shows up identically on the fused path, and the implied coefficients are flat across -# resolution on both backends, so the linear model itself holds. Nor is "encoding costs half of decoding" architectural: -# it is a cuDNN property (0.49 there, 0.78 on MIOpen), which is why each operation carries its own coefficient instead -# of a ratio. Shipping the MIOpen values everywhere would add ~60% to every cuDNN decode for nothing, so the -# coefficients follow the backend of the device the VAE runs on (`platform_for`). +# None of the three scales out of the line. The demand does not follow the element size: on MIOpen fp16 needs about a +# third of what bf16 does, and on both backends fp32 needs less per value byte than bf16. The allocator matters as +# much: `cudaMallocAsync` needs 25-35% less than the native allocator on cuDNN. And MIOpen's bf16 workspaces are +# larger than cuDNN's, ~1.5x for a decode and ~2.4x for an encode -- not the attention term, which shows up the same +# on the fused path. The base is why the per-pixel figure rises at small sizes; on HIP it includes the ~300 MiB the +# driver maps beyond the allocator's own figure on the first operation of a process. +# +# Each estimator declares the lines of the dtypes it can run, and the estimate prices the dtype the operation will run +# in, which is not always the dtype the cached VAE holds (see `sd_vae_run_dtype`). CogView4's VAE is an AutoencoderKL +# too, but a wider one (block widths up to 1024, three layers per block, no mid-block attention): it carries its own +# lines, about 1.9x the family's for a decode. # # Caveat for AMD users: `MIOPEN_FIND_MODE=2` (FAST) selects convolution algorithms heuristically instead of by # benchmark, and measured a uniform 1.28x more memory at every resolution when the MIOpen values were first fitted. -# Invoke has since made FAST its default (`run_app.py`); the RX 9060 XT measurement was taken under it and did not show -# the 1.28x. If you see decodes exceed their reservation on another card, raise `device_working_mem_gb`. -_AUTOENCODER_KL_COEFFICIENTS = frozenset({"k_decode", "k_encode"}) -_CONV_BACKEND_KEYS = ("conv_backend",) +# Invoke has since made FAST its default (`run_app.py`); the RX 9060 XT points were taken under it. If you see decodes +# exceed their reservation on another card, raise `device_working_mem_gb`. +_PLATFORM_KEYS = ("conv_backend", "allocator") +_DTYPE_NAMES: dict[torch.dtype, str] = {torch.float16: "fp16", torch.bfloat16: "bf16", torch.float32: "fp32"} +_ELEMENT_SIZES = {"fp16": 2, "bf16": 2, "fp32": 4} + + +def _convolution_coefficients(*dtypes: str) -> frozenset[str]: + """The coefficients of the convolution lines for both operations in these dtypes.""" + return frozenset( + f"{part}_{operation}_{dtype}" for part in ("k", "b") for operation in ("decode", "encode") for dtype in dtypes + ) + + +def _dtype_name(dtype: torch.dtype) -> str: + name = _DTYPE_NAMES.get(dtype) + if name is None: + raise ValueError(f"No VAE working-memory coefficients are measured for {dtype}.") + return name + + +def _compute_dtype(vae: torch.nn.Module) -> torch.dtype: + """The dtype a VAE no node casts computes in: that of its first floating-point parameter wider than a byte. + + Narrower or integer weights (fp8 storage, an SDNQ VAE's packed ones) are what a layer dequantizes from, not what + its activations are. + """ + for param in vae.parameters(): + if param.is_floating_point() and param.element_size() > 1: + return param.dtype + raise ValueError(f"{type(vae).__name__} has no floating-point parameters to compute in.") + + +def _estimate(spec: EstimatorSpec, features: Mapping[str, FeatureValue], device: torch.device) -> WorkingMemoryEstimate: + """`spec.estimate`, refusing a dtype the spec has no line for with the estimator, dtype and platform named, rather + than a bare `KeyError` from inside the terms.""" + line = f"k_{features['operation']}_{features['dtype']}" + if line not in spec.coefficients: + raise ValueError( + f"Working-memory estimator '{spec.id}' has no {features['dtype']} {features['operation']} line on " + f"{platform_for(device)}: the VAE computes in a dtype it is not measured for." + ) + return spec.estimate(features, device) + + +def _convolutions( + features: Mapping[str, FeatureValue], values: Mapping[str, float], pixels: float, samples: int = 1 +) -> tuple[WorkingMemoryTerm, ...]: + """The convolution line of this operation and dtype over `pixels` output pixels, for `samples` samples. + + The base scales with the samples too. That is an upper bound rather than a fit -- the base is paid once per process + -- and it keeps a batch at exactly that many single-sample estimates. + """ + line = f"{features['operation']}_{features['dtype']}" + return ( + WorkingMemoryTerm.fitted("convolutions", f"k_{line}", pixels * samples, values), + WorkingMemoryTerm.fitted("convolution_base", f"b_{line}", samples, values), + ) def _peak_of_two_phases( @@ -170,40 +234,62 @@ def _peak_of_two_phases( def _sd_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: - coefficient = f"k_{features['operation']}" - element_size = cast(int, features["element_size"]) tile = cast(int | None, features["tile"]) if tile is not None: - area_bytes = tile * tile * element_size # We add 25% to the working memory estimate when tiling is enabled to account for factors like tile overlap # and number of tiles. We could make this more precise in the future, but this should be good enough for # most use cases. - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * 1.25, values) + terms = _convolutions(features, values, tile * tile * 1.25) else: - area_bytes = cast(int, features["height"]) * cast(int, features["width"]) * element_size - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes, values) + terms = _convolutions(features, values, cast(int, features["height"]) * cast(int, features["width"])) - terms: tuple[WorkingMemoryTerm, ...] = (convolutions,) # The tiny autoencoder has no mid-block attention, so it is priced without a score matrix. if features["vae_class"] == "AutoencoderKL": terms = _peak_of_two_phases(terms, cast(int, features["score_matrix_bytes"])) - if features["fp32"]: - # If we are running in FP32, then we should account for the likely increase in model size (~250MB). - terms += (WorkingMemoryTerm("fp32_weights", 250 * 2**20, True),) + upcast_weight_bytes = cast(int, features["upcast_weight_bytes"]) + if upcast_weight_bytes: + terms += (WorkingMemoryTerm("upcast_weights", upcast_weight_bytes, True),) return terms VAE_SD_SPEC = register( EstimatorSpec( id="vae.sd", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("fp16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_sd_terms, + line=convolution_line, + user_calibrated=True, ) ) +def sd_vae_run_dtype(operation: Literal["encode", "decode"], fp32: bool, device: torch.device) -> torch.dtype: + """The dtype the SD1/SDXL nodes cast the shared VAE to before they run it: `l2i` and `i2l` cast with this, and + `estimate_vae_working_memory_sd15_sdxl` prices it, so the two cannot drift. + + fp32 when the node asks for it, and for a decode on the CPU; fp16 otherwise, on every GPU and for every + `precision`, so bf16 never runs. The cached VAE's own dtype says nothing about it: it is whatever the previous run + cast it to. + """ + if fp32 or (operation == "decode" and device.type == "cpu"): + return torch.float32 + return torch.float16 + + +def _upcast_weight_bytes(vae: torch.nn.Module, run_dtype: torch.dtype) -> int: + """What the node's in-place cast to `run_dtype` adds to the VAE's resident weights. + + The cast runs once the cache has placed the VAE, so the growth comes out of the reservation: 160 MiB for the 83.7M + parameters of the SD1/SDXL VAE going from fp16 to fp32, nothing for a VAE a previous run left in fp32. Measured + points exclude resident weights by construction (they record 0 here), so the fitted line does not cover it. An + upcast encode casts the decoder's mid-block and post-quant conv back, so for it this counts a little more than + grows: conservative. + """ + return sum(p.numel() * max(0, run_dtype.itemsize - p.element_size()) for p in vae.parameters()) + + def estimate_vae_working_memory_sd15_sdxl( operation: Literal["encode", "decode"], image_tensor: torch.Tensor, @@ -214,12 +300,13 @@ def estimate_vae_working_memory_sd15_sdxl( ) -> WorkingMemoryEstimate: """Estimate the working memory required to encode or decode the given tensor. - It was found experimentally that the peak working memory scales linearly with the number of pixels and the element - size (precision). This estimate is accurate for both SD1 and SDXL. `device` is where the VAE will run; it picks the - convolution backend's coefficients and defaults to the session device. + It was found experimentally that the peak working memory is a line in the number of pixels, per dtype. This + estimate is accurate for both SD1 and SDXL. It prices the dtype the node will cast the VAE to (`sd_vae_run_dtype`), + not the one the cached VAE holds. `device` is where the VAE will run; it picks the platform's coefficients and + defaults to the session device. """ - element_size = 4 if fp32 else 2 device = device if device is not None else TorchDevice.choose_torch_device() + run_dtype = sd_vae_run_dtype(operation, fp32, device) latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 height = latent_scale_factor_for_operation * image_tensor.shape[-2] width = latent_scale_factor_for_operation * image_tensor.shape[-1] @@ -233,26 +320,23 @@ def estimate_vae_working_memory_sd15_sdxl( "height": height, "width": width, "tile": tile_size, - "element_size": element_size, - "fp32": fp32, + "dtype": _dtype_name(run_dtype), "vae_class": "AutoencoderKL" if isinstance(vae, AutoencoderKL) else "AutoencoderTiny", + "upcast_weight_bytes": _upcast_weight_bytes(vae, run_dtype), } if isinstance(vae, AutoencoderKL): - score_dtype = torch.float32 if fp32 else next(vae.parameters()).dtype score_h, score_w = (tile_size, tile_size) if tile_size is not None else (height, width) - features["score_matrix_bytes"] = _vae_mid_block_score_matrix_bytes(score_h, score_w, score_dtype, device=device) + features["score_matrix_bytes"] = _vae_mid_block_score_matrix_bytes(score_h, score_w, run_dtype, device=device) - return VAE_SD_SPEC.estimate(features, device) + return _estimate(VAE_SD_SPEC, features, device) def _single_pass_terms( features: Mapping[str, FeatureValue], values: Mapping[str, float] ) -> tuple[WorkingMemoryTerm, ...]: """An untiled AutoencoderKL pass: the convolutions over the whole image, or the mid-block score matrix.""" - coefficient = f"k_{features['operation']}" - area_bytes = cast(int, features["height"]) * cast(int, features["width"]) * cast(int, features["element_size"]) - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes, values) - return _peak_of_two_phases((convolutions,), cast(int, features["score_matrix_bytes"])) + convolutions = _convolutions(features, values, cast(int, features["height"]) * cast(int, features["width"])) + return _peak_of_two_phases(convolutions, cast(int, features["score_matrix_bytes"])) def _single_pass_features( @@ -261,23 +345,32 @@ def _single_pass_features( latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 height = latent_scale_factor_for_operation * image_tensor.shape[-2] width = latent_scale_factor_for_operation * image_tensor.shape[-1] - param = next(vae.parameters()) + dtype = _compute_dtype(vae) return { "operation": operation, "height": height, "width": width, - "element_size": param.element_size(), - "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(height, width, param.dtype, device=device), + "dtype": _dtype_name(dtype), + "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(height, width, dtype, device=device), } +def _cogview4_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: + """An untiled CogView4 VAE pass: its convolutions over the whole image. It has no mid-block attention + (`mid_block_add_attention: false`), so there is no score matrix to price.""" + return _convolutions(features, values, cast(int, features["height"]) * cast(int, features["width"])) + + +# The CogView4 loader forces bf16: the VAE produces black images in fp16. VAE_COGVIEW4_SPEC = register( EstimatorSpec( id="vae.cogview4", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, - terms=_single_pass_terms, + version=2, + coefficients=_convolution_coefficients("bf16"), + platform_keys=_PLATFORM_KEYS, + terms=_cogview4_terms, + line=convolution_line, + user_calibrated=True, ) ) @@ -294,7 +387,14 @@ def estimate_vae_working_memory_cogview4( device. """ device = device if device is not None else TorchDevice.choose_torch_device() - return VAE_COGVIEW4_SPEC.estimate(_single_pass_features(operation, image_tensor, vae, device), device) + latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 + features: dict[str, FeatureValue] = { + "operation": operation, + "height": latent_scale_factor_for_operation * image_tensor.shape[-2], + "width": latent_scale_factor_for_operation * image_tensor.shape[-1], + "dtype": _dtype_name(_compute_dtype(vae)), + } + return _estimate(VAE_COGVIEW4_SPEC, features, device) # What a tiled decode does *not* bound: the assembled image, several times over, per output pixel. @@ -330,28 +430,23 @@ def estimate_vae_working_memory_cogview4( def _flux_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: operation = features["operation"] - coefficient = f"k_{operation}" out_h = cast(int, features["height"]) out_w = cast(int, features["width"]) - element_size = cast(int, features["element_size"]) + element_size = _ELEMENT_SIZES[cast(str, features["dtype"])] tile = cast(int | None, features["tile"]) if tile is None: - area_bytes = out_h * out_w * element_size - convolution_phase: tuple[WorkingMemoryTerm, ...] = ( - WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes, values), - ) + convolution_phase = _convolutions(features, values, out_h * out_w) return _peak_of_two_phases(convolution_phase, cast(int, features["score_matrix_bytes"])) # `calc_tiles_min_overlap` clamps the tile to the image *per axis*, so the largest tile the # decode builds is this, not `tile` squared. Without the clamp an 8192px tile on a # 1024x8200 image -- tiled, because one axis exceeds it -- prices 8192x8192. - area_bytes = min(tile, out_h) * min(tile, out_w) * element_size # A 25% margin for tile overlap and the number of tiles, mirroring the SD1/SDXL estimator. - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * 1.25, values) + convolutions = _convolutions(features, values, min(tile, out_h) * min(tile, out_w) * 1.25) if operation == "decode": convolution_phase = ( - convolutions, + *convolutions, WorkingMemoryTerm( "assembled_image", out_h * out_w * _IMAGE_CHANNELS * (_FLUX_VAE_TILED_IMAGE_COPIES * element_size + 1), @@ -377,20 +472,24 @@ def _flux_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float * element_size ) convolution_phase = ( - convolutions, + *convolutions, WorkingMemoryTerm("input_image", out_h * out_w * _IMAGE_CHANNELS * element_size, True), WorkingMemoryTerm("assembled_moments", moments * live_moment_copies, True), ) return _peak_of_two_phases(convolution_phase, cast(int, features["score_matrix_bytes"])) +# The FLUX.1 loaders avoid fp16, which this autoencoder is broken in, except one: an SDNQ VAE folder loads through the +# generic VAE loader and computes in the configured precision, which `auto` makes fp16 on CUDA. VAE_FLUX_SPEC = register( EstimatorSpec( id="vae.flux", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("fp16", "bf16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_flux_terms, + line=convolution_line, + user_calibrated=True, ) ) @@ -418,7 +517,7 @@ def estimate_vae_working_memory_flux( backend's coefficients. Defaults to the session device. """ latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 - param = next(vae.parameters()) + dtype = _compute_dtype(vae) device = device if device is not None else TorchDevice.choose_torch_device() out_h = latent_scale_factor_for_operation * image_tensor.shape[-2] @@ -443,9 +542,9 @@ def estimate_vae_working_memory_flux( "operation": operation, "height": out_h, "width": out_w, - "element_size": param.element_size(), + "dtype": _dtype_name(dtype), "tile": tile, - "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(score_h, score_w, param.dtype, device=device), + "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(score_h, score_w, dtype, device=device), } if tile is not None and operation != "decode": # Read from the VAE rather than hard-coded: an overlap factor of 0.5 would make the tiles @@ -454,7 +553,7 @@ def estimate_vae_working_memory_flux( features["latent_channels"] = getattr(getattr(vae, "config", None), "latent_channels", 16) features["tile_overlap_factor"] = getattr(vae, "tile_overlap_factor", 0.25) - return VAE_FLUX_SPEC.estimate(features, device) + return _estimate(VAE_FLUX_SPEC, features, device) # The FLUX.2 VAE runs one attention block at the bottom of the encoder and one at the top of the @@ -468,30 +567,29 @@ def estimate_vae_working_memory_flux( def _flux2_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: - coefficient = f"k_{features['operation']}" - element_size = cast(int, features["element_size"]) batch_size = cast(int, features["batch_size"]) tile = cast(int | None, features["tile"]) height, width = cast(int, features["height"]), cast(int, features["width"]) if tile is not None: - # The largest slice `AutoencoderKLFlux2` takes: the tile, clamped to the image per axis. - area_bytes = min(tile, height) * min(tile, width) * element_size - # Add 25% for tile overlap and the blending buffers, mirroring the SD1/SDXL estimate. - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * 1.25 * batch_size, values) + # The largest slice `AutoencoderKLFlux2` takes: the tile, clamped to the image per axis. Add 25% for tile + # overlap and the blending buffers, mirroring the SD1/SDXL estimate. + convolutions = _convolutions(features, values, min(tile, height) * min(tile, width) * 1.25, samples=batch_size) else: - area_bytes = height * width * element_size - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * batch_size, values) + convolutions = _convolutions(features, values, height * width, samples=batch_size) # Both terms are per sample; the larger is reserved (see `_peak_of_two_phases` for why, and the measurements). - return _peak_of_two_phases((convolutions,), cast(int, features["score_matrix_bytes"])) + return _peak_of_two_phases(convolutions, cast(int, features["score_matrix_bytes"])) +# The FLUX.2 loaders avoid fp16 (as FLUX.1's do), and none of them goes through the generic VAE loader. VAE_FLUX2_SPEC = register( EstimatorSpec( id="vae.flux2", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("bf16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_flux2_terms, + line=convolution_line, + user_calibrated=True, ) ) @@ -505,16 +603,16 @@ def estimate_vae_working_memory_flux2( ) -> WorkingMemoryEstimate: """Estimate the working memory required to encode or decode with the FLUX.2 (32-channel) VAE. - Peak memory scales linearly with pixel area and element size, as it does for the FLUX.1 VAE, and - the implied coefficient is flat across 512-1536px on every backend measured. What is *not* constant - is the coefficient itself: MIOpen's convolution workspaces cost ~1.6x cuDNN's for a decode and ~2.4x - for an encode, so it is shipped per backend -- see `_AUTOENCODER_KL_COEFFICIENTS` for the caveats - and `backend/working_memory/data/vae.json` for the fitted values and their points. Peak *reserved* - memory is what is measured throughout, the conservative quantity that includes allocator overhead. + Peak memory is a line in pixel area per dtype, as it is for the FLUX.1 VAE, from 512 to 2048px on + every backend measured. The line itself is not one: MIOpen's bf16 convolution workspaces cost ~1.5x + cuDNN's for a decode and ~2.4x for an encode, and the allocator and the dtype move it as much, so it + is shipped per platform and dtype -- see `_convolution_coefficients` for the caveats and + `backend/working_memory/data/vae.json` for the fitted values and their points. The points measure + peak *reserved* memory or more, the conservative quantity that includes allocator overhead. - For reference, decoding 1024x1024 peaks at ~4.3GB on cuDNN and ~6.6GB on MIOpen, and 1536x1536 at - ~9.6GB -- far above the default ``device_working_mem_gb``, which is why this estimate must be - passed to the model cache. + For reference, decoding 1024x1024 in bf16 peaks at ~4.3GB on cuDNN and ~6.6GB on MIOpen, and + 1536x1536 at ~9.6GB -- far above the default ``device_working_mem_gb``, which is why this estimate + must be passed to the model cache. That linear term holds only while ``AutoencoderKLFlux2``'s mid-block attention runs through a fused SDPA kernel, which is what CUDA does (verified: the memory-efficient kernel takes the @@ -546,7 +644,7 @@ def estimate_vae_working_memory_flux2( bound. The score matrix is shaped (batch, heads, S, S), so it scales the same way. The encode call sites all pass batch 1 today; the shared estimator does not assume it. """ - param = next(vae.parameters()) + dtype = _compute_dtype(vae) device = device if device is not None else TorchDevice.choose_torch_device() batch_size = image_tensor.shape[0] if image_tensor.dim() >= 4 else 1 latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 @@ -570,17 +668,17 @@ def estimate_vae_working_memory_flux2( "width": out_w, "tile": tile, "batch_size": batch_size, - "element_size": param.element_size(), + "dtype": _dtype_name(dtype), "score_matrix_bytes": sdpa_score_matrix_bytes( device=device, - dtype=param.dtype, + dtype=dtype, # The score matrix is (batch, heads, S, S); one head per sample prices the whole batch. num_heads=_FLUX2_VAE_MID_BLOCK_HEADS * batch_size, head_dim=_FLUX2_VAE_MID_BLOCK_HEAD_DIM, seq_len=mid_block_seq_len, ), } - return VAE_FLUX2_SPEC.estimate(features, device) + return _estimate(VAE_FLUX2_SPEC, features, device) def estimate_vae_working_memory_anima( @@ -1077,13 +1175,17 @@ def estimate_vae_working_memory_qwen_image( return int(working_memory) +# The SD3 VAE loads in the configured precision and runs uncast: `auto` gives fp16 on CUDA, and bf16 and fp32 are +# honoured. It carries the family's values in `data/vae.json`, as the one member that runs all three. VAE_SD3_SPEC = register( EstimatorSpec( id="vae.sd3", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("fp16", "bf16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_single_pass_terms, + line=convolution_line, + user_calibrated=True, ) ) @@ -1100,4 +1202,32 @@ def estimate_vae_working_memory_sd3( device. """ device = device if device is not None else TorchDevice.choose_torch_device() - return VAE_SD3_SPEC.estimate(_single_pass_features(operation, image_tensor, vae, device), device) + return _estimate(VAE_SD3_SPEC, _single_pass_features(operation, image_tensor, vae, device), device) + + +def estimate_untiled_vae_decode( + estimator: str, height: int, width: int, dtype: str, device: torch.device +) -> WorkingMemoryEstimate: + """The estimate an untiled single-image decode of `height` x `width` output pixels in `dtype` gets from the decode + node that prices with `estimator`, on `device`: the features each estimator builds from its VAE, without one. + + For planning sizes before anything is loaded (a calibration plan). The SD1/SDXL VAE is priced as one already in + `dtype`, so without the growth of an in-place cast, which a first fp32 decode after an fp16 one adds on top. + """ + torch_dtype = {name: candidate for candidate, name in _DTYPE_NAMES.items()}.get(dtype) + if torch_dtype is None: + raise ValueError(f"No VAE working-memory coefficients are measured for {dtype}.") + features: dict[str, FeatureValue] = {"operation": "decode", "height": height, "width": width, "dtype": dtype} + if estimator == VAE_COGVIEW4_SPEC.id: + return _estimate(VAE_COGVIEW4_SPEC, features, device) + features["score_matrix_bytes"] = _vae_mid_block_score_matrix_bytes(height, width, torch_dtype, device=device) + if estimator == VAE_SD3_SPEC.id: + return _estimate(VAE_SD3_SPEC, features, device) + if estimator == VAE_FLUX_SPEC.id: + return _estimate(VAE_FLUX_SPEC, features | {"tile": None}, device) + if estimator == VAE_FLUX2_SPEC.id: + return _estimate(VAE_FLUX2_SPEC, features | {"tile": None, "batch_size": 1}, device) + if estimator == VAE_SD_SPEC.id: + sd_features = features | {"tile": None, "vae_class": "AutoencoderKL", "upcast_weight_bytes": 0} + return _estimate(VAE_SD_SPEC, sd_features, device) + raise ValueError(f"No untiled decode is priced by working-memory estimator '{estimator}'.") diff --git a/invokeai/backend/util/wddm.py b/invokeai/backend/util/wddm.py index ec39f7b7389..445cbe1ffd4 100644 --- a/invokeai/backend/util/wddm.py +++ b/invokeai/backend/util/wddm.py @@ -21,6 +21,11 @@ Whether Windows has actually paged this process out is not visible through D3DKMT (``CurrentUsage`` counts paged bytes as local); it is visible in the ``GPU Process Memory`` performance counters Task Manager shows, read here through PDH. +The working-memory probe measures what an operation needs on CUDA and on ROCm alike, so `process_local_usage` (this +process's ``CurrentUsage``) and `process_paged_bytes` answer on either build under Windows; on an RTX 4090 with torch +2.13 the usage follows torch's reserved bytes to three decimals. The ROCm-only readers the model cache decides with keep +their gate. + Everything here is best-effort: any failure yields ``None`` ("unknown") rather than raising, and callers must keep their existing behaviour when the answer is unknown. """ @@ -36,6 +41,7 @@ import torch from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.util.pci import pci_location _KMTQAITYPE_ADAPTERADDRESS = 6 _MEMORY_SEGMENT_GROUP_LOCAL = 0 @@ -149,8 +155,12 @@ class _Adapter: _pdh_queries: dict[str, Optional[tuple[ctypes.c_void_p, ctypes.c_void_p]]] = {} +def _windows_gpu(device: torch.device) -> bool: + return sys.platform == "win32" and device.type == "cuda" + + def _supported(device: torch.device) -> bool: - return sys.platform == "win32" and torch.version.hip is not None and device.type == "cuda" + return _windows_gpu(device) and torch.version.hip is not None @functools.lru_cache(maxsize=1) @@ -211,11 +221,11 @@ def _close(lib: ctypes.CDLL, handle: int) -> None: def _resolve_adapter(lib: ctypes.CDLL, index: int) -> Optional[_Adapter]: """Open the WDDM adapter at the torch device's PCI location, closing every other handle the enumeration opened. - ROCm's device properties carry no LUID, so the PCI bus and device numbers are the link. Anything but exactly one - match is treated as unknown rather than guessed. + torch's device properties carry no LUID, on ROCm or CUDA, so the PCI bus and device numbers are the link. Anything + but exactly one match is treated as unknown rather than guessed. """ props = torch.cuda.get_device_properties(index) - location = (int(props.pci_bus_id), int(props.pci_device_id)) + _domain, bus, device_number = pci_location(index) enum = _EnumAdapters2(0, None) if lib.D3DKMTEnumAdapters2(ctypes.byref(enum)) != 0: @@ -241,9 +251,9 @@ def _resolve_adapter(lib: ctypes.CDLL, index: int) -> Optional[_Adapter]: ctypes.cast(ctypes.byref(address), ctypes.c_void_p), ctypes.sizeof(address), ) - if ( - lib.D3DKMTQueryAdapterInfo(ctypes.byref(query)) == 0 - and (address.BusNumber, address.DeviceNumber) == location + if lib.D3DKMTQueryAdapterInfo(ctypes.byref(query)) == 0 and (address.BusNumber, address.DeviceNumber) == ( + bus, + device_number, ): matches.append(info) else: @@ -293,25 +303,61 @@ def local_video_memory(device: torch.device) -> Optional[tuple[int, Optional[int if not _supported(device): return None try: - resolved = _adapter_for(device) - if resolved is None: - return None - lib, adapter = resolved - info = _QueryVideoMemoryInfo(None, adapter.handle, _MEMORY_SEGMENT_GROUP_LOCAL) - if lib.D3DKMTQueryVideoMemoryInfo(ctypes.byref(info)) != 0: + queried = _query_local_memory(device) + if queried is None: return None + info, adapter = queried if not 0 < info.Budget <= adapter.total_bytes: return None - # `CurrentUsage` counts bytes Windows has already paged out, so it exceeds the adapter's own size exactly when - # the process over-commits. It says nothing about residency then, so it is reported as unknown; the budget - # stands, since the model cache's free-VRAM cap is the caller that needs it most in that state. - usage = int(info.CurrentUsage) - return int(info.Budget), usage if 0 <= usage <= adapter.total_bytes else None + # The budget stands when the usage is unknown (see `_plausible_usage`): the model cache's free-VRAM cap is the + # caller that needs it most in exactly that state. + return int(info.Budget), _plausible_usage(info, adapter) except Exception: InvokeAILogger.get_logger(__name__).debug(f"WDDM memory lookup for {device} failed", exc_info=True) return None +def _query_local_memory(device: torch.device) -> Optional[tuple[_QueryVideoMemoryInfo, _Adapter]]: + """This process's local-segment memory info on the device's adapter, with the adapter; None if the driver cannot + answer.""" + resolved = _adapter_for(device) + if resolved is None: + return None + lib, adapter = resolved + info = _QueryVideoMemoryInfo(None, adapter.handle, _MEMORY_SEGMENT_GROUP_LOCAL) + if lib.D3DKMTQueryVideoMemoryInfo(ctypes.byref(info)) != 0: + return None + return info, adapter + + +def _plausible_usage(info: _QueryVideoMemoryInfo, adapter: _Adapter) -> Optional[int]: + """`CurrentUsage`, or None when it exceeds the adapter's own size. + + It counts bytes Windows has already paged out, so it exceeds the adapter exactly when the process over-commits, + and says nothing about residency then. + """ + usage = int(info.CurrentUsage) + return usage if 0 <= usage <= adapter.total_bytes else None + + +def process_local_usage(device: torch.device) -> Optional[int]: + """This process's ``CurrentUsage`` in the local segment of the device's adapter, in bytes, on a CUDA or a ROCm + build under Windows: everything the process has committed there, the allocator's cached blocks, the libraries' own + allocations and any bytes Windows paged out included. + + ``None`` off Windows, for a device that is not a GPU, when the driver cannot answer, and when the usage exceeds the + adapter. + """ + if not _windows_gpu(device): + return None + try: + queried = _query_local_memory(device) + return None if queried is None else _plausible_usage(*queried) + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"WDDM usage lookup for {device} failed", exc_info=True) + return None + + def video_memory_budget(device: torch.device) -> Optional[int]: """This process's WDDM budget on the device's adapter, in bytes: what Windows keeps resident before paging.""" local = local_video_memory(device) @@ -377,6 +423,13 @@ def paged_bytes(device: torch.device) -> Optional[int]: """ if not _supported(device): return None + return process_paged_bytes(device) + + +def process_paged_bytes(device: torch.device) -> Optional[int]: + """`paged_bytes` on a CUDA or a ROCm build under Windows; ``None`` elsewhere.""" + if not _windows_gpu(device): + return None resolved = _adapter_for(device) if resolved is None: return None diff --git a/invokeai/backend/working_memory/bounds.py b/invokeai/backend/working_memory/bounds.py new file mode 100644 index 00000000000..545f4d0afa9 --- /dev/null +++ b/invokeai/backend/working_memory/bounds.py @@ -0,0 +1,99 @@ +"""How a measured point bounds the estimate for its features. + +One rule, read by the data test (`tests/backend/working_memory/test_shipped_coefficients.py`), which holds the shipped +data to it, and by `scripts/working_memory/build_shipped.py`, which refuses to write data that breaks it: + +- Lower bound. The model cache consumes an estimate as ``free >= estimate``, so the estimate must be at least the point. + Where only analytic terms decide it (a materialized score matrix), the cache's `device_working_mem_gb` floor counts. +- Upper bound, for points that hold one (`MeasuredPoint.holds_upper_bound`): where a fitted term decides an in-fit + point, the estimate is at most `MAX_OVER_RESERVATION` times the point, its listed `UPPER_BOUND_EXCEPTIONS` ratio, or + its tighter `max_ratio`, which applies whichever term decides. A reservation the operation never uses is model + weights evicted for nothing. +""" + +import json +from collections.abc import Iterable, Mapping +from types import MappingProxyType +from typing import TYPE_CHECKING, Final, Optional + +from invokeai.app.services.config.config_default import InvokeAIAppConfig +from invokeai.backend.working_memory.coefficients import MeasuredPoint, WorkingMemoryData, resolve_coefficients +from invokeai.backend.working_memory.estimate import WorkingMemoryEstimate +from invokeai.backend.working_memory.platform import Platform + +if TYPE_CHECKING: + from invokeai.backend.working_memory.spec import EstimatorSpec + +MAX_OVER_RESERVATION: Final = 1.3 +"""How far above a measured point a reservation may sit where a fitted coefficient decides it, unless the point is +listed in `UPPER_BOUND_EXCEPTIONS` or carries a tighter `max_ratio`.""" + +WORKING_MEMORY_FLOOR: Final = int(InvokeAIAppConfig.model_fields["device_working_mem_gb"].default * 2**30) +"""What `ModelCache._working_mem_reserve` floors every reservation at. The field default, not a config instance an +environment variable could move.""" + +UPPER_BOUND_EXCEPTIONS: Final[Mapping[str, float]] = MappingProxyType( + { + "vae.sd-miopen,native,windows-decode-fp16-1280x1280-RX9060XT-demand_peak": 1.33, + "vae.sd-miopen,native,windows-encode-fp16-1024x1024-RX9060XT-demand_peak": 1.57, + "vae.sd-miopen,native,windows-encode-fp16-1280x1280-RX9060XT-demand_peak": 1.78, + "vae.flux-miopen,native,windows-decode-bf16-512x512-RX9060XT-demand_peak": 1.33, + "vae.flux2-miopen,native,windows-decode-bf16-512x512-RX9060XT-demand_peak": 1.33, + "denoise.flux2-cudnn,native-denoise-bf16-h3072-8100+77tok-RTX4090-demand_peak": 1.42, + "denoise.flux2-miopen,native,windows-denoise-bf16-h3072-8100+77tok-RX9060XT-demand_peak": 1.40, + } +) +"""In-fit points, by `point_id`, whose envelope sits more than 1.3x above them, with the ratio they are held to +instead. All are the native allocator, never expandable segments or `cudaMallocAsync`: + +- The VAE points are MIOpen on Windows, where the demand is concave in the size: first-run driver spikes at small sizes + and a second run reserving up to 1.6x the first at large ones, so a line that covers every point cannot stay within + 1.3x of all of them. Windows ROCm defaults to expandable segments; this is the override's price. +- The FLUX.2 denoise points are the Klein 4B width at 1440px with a 77-token caption, 8177 tokens, on both cards: the + native allocator served them from 1676 MiB (RTX 4090) and 1879 MiB (RX 9060 XT), while the 512-token caption at + 8612 tokens took 2382 and 2577 MiB. The line has to cover the latter. Under `cudaMallocAsync` and expandable + segments the two lie on one line.""" + + +def point_id(point: MeasuredPoint, spec: "EstimatorSpec") -> str: + """A readable id of a point: estimator, platform, its features as its estimator's line labels them (`fit.Line`: + operation, dtype, size, tile and batch of a VAE operation), card and method. Features that name no line show as + they are.""" + found = spec.line(point.features) + label = found.label if found is not None else json.dumps(point.features, sort_keys=True, separators=(",", ":")) + out_of_fit = "-outfit" if not point.in_fit else "" + platform = ",".join(point.platform.values()) + card = point.device.split(" (")[0].replace(" ", "") + return f"{point.estimator}-{platform}-{label}{out_of_fit}-{card}-{point.method}" + + +def platform_of(point: MeasuredPoint) -> Platform: + """The platform a point is evaluated on. One that names no OS measured what holds on every OS.""" + return Platform(**({"os": "other"} | point.platform)) # type: ignore[arg-type] + + +def evaluate_point( + point: MeasuredPoint, spec: "EstimatorSpec", data: Iterable[WorkingMemoryData] +) -> WorkingMemoryEstimate: + """The estimate for the point's features, priced with the coefficients `data` gives its platform.""" + return spec.evaluate(point.features, resolve_coefficients(data, spec, platform_of(point), origin="data")) + + +def fitted_decides(estimate: WorkingMemoryEstimate) -> bool: + return any(term.included and term.coefficient is not None for term in estimate.terms) + + +def bound_violation(point: MeasuredPoint, spec: "EstimatorSpec", estimate: WorkingMemoryEstimate) -> Optional[str]: + """Why `estimate`, the estimate `spec` gives `point`'s features, breaks a bound `point` sets for it, or None.""" + fitted = fitted_decides(estimate) + reserved = estimate.reservation_bytes if fitted else max(estimate.reservation_bytes, WORKING_MEMORY_FLOOR) + if reserved < point.measured_bytes: + return f"reserves {reserved / 2**30:.3f} GiB for a measured {point.measured_bytes / 2**30:.3f} GiB" + # An out-of-fit point checks another term, and a lower-bound-only point is no cold first run: neither caps. + if point.holds_upper_bound and (point.max_ratio is not None or (fitted and point.in_fit)): + default = UPPER_BOUND_EXCEPTIONS.get(point_id(point, spec), MAX_OVER_RESERVATION) + max_ratio = point.max_ratio if point.max_ratio is not None else default + ratio = estimate.reservation_bytes / point.measured_bytes + if ratio > max_ratio: + return f"reserves {ratio:.2f}x the measured peak, over {max_ratio}x" + return None diff --git a/invokeai/backend/working_memory/coefficients.py b/invokeai/backend/working_memory/coefficients.py index d8e07b13854..db9e3f8b029 100644 --- a/invokeai/backend/working_memory/coefficients.py +++ b/invokeai/backend/working_memory/coefficients.py @@ -1,24 +1,26 @@ """The coefficients estimators are evaluated with, and the measured points that bound them. Both ship as JSON under `data/`, read through `importlib.resources` so a wheel install finds them. A coefficient entry -belongs to one estimator version on one measured platform value, and either carries its values or borrows another -entry's (`values_from`); a point records the features of an operation that was measured and its measured peak. -`tests/backend/working_memory/test_shipped_coefficients.py` resolves every entry against its `EstimatorSpec` and holds -every point to the estimate the shipped coefficients give it. +belongs to one estimator version on one measured platform, and either carries its values or borrows another entry's +(`values_from`); a point records the features of an operation that was measured and its measured peak. A carried value +is the fit of the points it is resolved for (`fit.py`), a declared stand-in (`stand_ins`), or a hand-set constant of an +entry marked `legacy`. `tests/backend/working_memory/test_shipped_coefficients.py` resolves every entry against its +`EstimatorSpec`, holds every point to the estimate the shipped coefficients give it, and every fitted value to the fit +of its points. """ import functools from collections.abc import Iterable, Mapping -from dataclasses import dataclass +from dataclasses import asdict, dataclass from importlib.resources import files from importlib.resources.abc import Traversable from types import MappingProxyType -from typing import TYPE_CHECKING, Literal, Self +from typing import TYPE_CHECKING, Final, Literal, Self from pydantic import BaseModel, ConfigDict, ValidationError, model_validator from invokeai.backend.working_memory.estimate import FeatureValue -from invokeai.backend.working_memory.platform import UNMEASURED_PRICED_AS, Platform +from invokeai.backend.working_memory.platform import Platform, priced_as if TYPE_CHECKING: from invokeai.backend.working_memory.spec import EstimatorSpec @@ -29,7 +31,8 @@ class _DataModel(BaseModel): class ValuesFrom(_DataModel): - """The entry whose values another entry takes: same platform, this estimator id and version.""" + """The entry whose values another entry takes: same platform, this estimator id and version. It may carry more + values than the borrower reads -- a family member that runs fewer dtypes reads fewer of them -- never fewer.""" estimator: str version: int @@ -44,7 +47,21 @@ class CoefficientEntry(_DataModel): values: dict[str, float] | None = None values_from: ValuesFrom | None = None """Borrow the values of another estimator's entry on the same platform, so that its measured points bound this - estimator too. One hop: the entry borrowed from carries its own values.""" + estimator too, and this estimator's points join the fit of those values. One hop: the entry borrowed from carries + its own values.""" + stand_ins: list[str] = [] + """Carried values that are declared conservative stand-ins rather than fits: what this platform prices a line no + in-fit point measures at. `provenance` says what they stand for; every other carried value is a fit, unless the + entry is `legacy`.""" + unmeasured_floor: dict[str, float] = {} + """Declared lower bounds, not fits: what carried values are raised to where this entry prices a platform nothing + is measured on (see `priced_as`). A fit describes the platform its points were measured on, not the one it stands + in for. `provenance` says where the floors come from.""" + legacy: bool = False + """True for an entry whose carried values are hand-set constants, not a fit of measured points: no point decides + them and `fit.py` does not reproduce them, though the points of its estimator still bound its estimates. The data + test lists the entries that may ship such values; `scripts/working_memory/build_shipped.py` replaces them with the + fit of measured points once those fit every line and a fresh-process point measures each.""" provenance: str """Card, torch build, method and script the values come from, and which points they were fitted on.""" @@ -52,26 +69,62 @@ class CoefficientEntry(_DataModel): def _values_or_values_from(self) -> Self: if (self.values is None) == (self.values_from is None): raise ValueError("a coefficient entry carries exactly one of `values` and `values_from`") + for field, names in (("stand-ins", self.stand_ins), ("unmeasured floors", self.unmeasured_floor)): + unknown = sorted(set(names) - set(self.values or {})) + if unknown: + raise ValueError(f"{field} {unknown} are not values this entry carries") return self +LOWER_BOUND_ONLY_METHODS: Final = frozenset({"probe_demand"}) +"""Point methods that bound an estimate from below only (see `MeasuredPoint.holds_upper_bound`).""" + + class MeasuredPoint(_DataModel): """One measured operation. The estimate for its features must be at least `measured_bytes`, and where a fitted - term decides it, at most 1.3 times that (or `max_ratio`).""" + term decides an in-fit point that `holds_upper_bound`, at most 1.3 times that (or `max_ratio`). Measured with the + model's weights already in place, so `measured_bytes` is the operation's working memory alone.""" estimator: str version: int platform: dict[str, str] features: dict[str, FeatureValue] measured_bytes: int - method: Literal["reserved_peak"] - """``reserved_peak``: the caching allocator's peak reserved bytes over the operation, one fresh process per - point. Reserved rather than allocated, because reserved is what the model cache's reservation has to cover.""" + method: Literal["reserved_peak", "driver_peak", "demand_peak", "probe_demand"] + """How `measured_bytes` was read, each as a peak over its value just before the operation: + + - ``reserved_peak``: the caching allocator's reserved bytes, one fresh process per point. Reserved rather than + allocated, because reserved is what the model cache's reservation has to cover. + - ``driver_peak``: the process's device usage as the driver reports it (Windows ``D3DKMT`` ``CurrentUsage``), the + larger of two runs, one fresh process per point. + - ``demand_peak``: the largest of the driver-level, the reserved and the allocated peak, the larger of two runs, + one fresh process per point. The driver-level figure is what reserved misses: on HIP the first operation of a + process maps ~300 MiB beyond ``max_memory_reserved``. + - ``probe_demand``: the working-memory probe's ``demand_bytes`` (`probe.MeasuredWindow`), the largest over the + valid, fit-eligible windows of calibration runs in a running server, warm-ups included. + `scripts/working_memory/build_shipped.py` builds these points from calibration store exports. A lower bound + only: a warm window in a running server can read well below the cold first run after a server start, which is + the worst case the reservation must cover, so it may raise a line but never caps one.""" device: str provenance: str max_ratio: float | None = None """A tighter upper bound than the default 1.3, as a multiple of `measured_bytes`, for a point the estimate has to stay closer to. It applies whichever term decides the estimate.""" + in_fit: bool = True + """False for a point measured in a regime its platform does not run, kept to check another term: CUDA with the + math attention kernel forced, whose score matrix shares the forward with the convolutions. The fit leaves it out, + and it gets the lower bound only.""" + + @model_validator(mode="after") + def _max_ratio_needs_an_upper_bound(self) -> Self: + if self.max_ratio is not None and not self.holds_upper_bound: + raise ValueError(f"a {self.method} point bounds its estimate from below only; it takes no max_ratio") + return self + + @property + def holds_upper_bound(self) -> bool: + """Whether the point also bounds its estimate from above: a fresh-process peak, not a `probe_demand` window.""" + return self.method not in LOWER_BOUND_ONLY_METHODS class WorkingMemoryData(_DataModel): @@ -123,21 +176,30 @@ def _where(keys: Mapping[str, str]) -> str: return ",".join(f"{key}={value}" for key, value in keys.items()) -def _entry( - entries: list[CoefficientEntry], estimator: str, version: int, keys: Mapping[str, str], where: str +def find_entry( + entries: Iterable[CoefficientEntry], estimator: str, version: int, platform: Mapping[str, str], where: str ) -> CoefficientEntry: - candidates = [entry for entry in entries if entry.estimator == estimator and entry.platform == keys] + """The entry of `estimator` `version` that applies on `platform`: of those whose every platform key it matches, + the most specific one. An entry that also names the OS thereby takes precedence on that OS over the one that does + not (see `Platform`). Raises, naming `where`, if none applies or the most specific is declared twice.""" + candidates = [ + entry + for entry in entries + if entry.estimator == estimator and all(platform.get(key) == value for key, value in entry.platform.items()) + ] matches = [entry for entry in candidates if entry.version == version] - if len(matches) > 1: - raise LookupError( - f"Working-memory coefficients for estimator '{estimator}' version {version} on {where} are declared " - f"{len(matches)} times." - ) if not matches: found = f" (found only version {sorted({entry.version for entry in candidates})})" if candidates else "" raise LookupError( f"No working-memory coefficients for estimator '{estimator}' version {version} on {where}{found}." ) + most_specific = max(len(entry.platform) for entry in matches) + matches = [entry for entry in matches if len(entry.platform) == most_specific] + if len(matches) > 1: + raise LookupError( + f"Working-memory coefficients for estimator '{estimator}' version {version} on {where} are declared " + f"{len(matches)} times." + ) return matches[0] @@ -146,43 +208,51 @@ def resolve_coefficients( ) -> ResolvedCoefficients: """The values `spec` takes on `platform`, and a `source` that says how they were found. - A platform value nothing is measured for is priced as the measured value `UNMEASURED_PRICED_AS` names. The entry - for the spec's id, version and those platform values either carries the values or borrows them, one hop, from - another estimator's entry on the same platform. No other fallback: an estimate priced with coefficients fitted for - something else is the shortfall this data exists to prevent, so a missing, ambiguous, chained or mismatched entry - raises, naming what is wrong. `origin` prefixes the `source`. + A platform nothing is measured on is priced as the measured platform `priced_as` names. The entry for the spec's + id and version that applies there (`find_entry`) either carries the values or borrows them, one hop, from the + entry of another estimator that applies on the same platform. No other fallback: an estimate priced with + coefficients fitted for something else is the shortfall this data exists to prevent, so a missing, ambiguous, + chained or mismatched entry raises, naming what is wrong. `origin` prefixes the `source`. """ + measured = priced_as(platform) + full = {key: str(value) for key, value in asdict(measured).items()} + keys = {key: full[key] for key in spec.platform_keys} requested = {key: getattr(platform, key) for key in spec.platform_keys} - keys = {key: UNMEASURED_PRICED_AS.get(key, {}).get(value, value) for key, value in requested.items()} where = _where(keys) if keys == requested else f"{_where(keys)} (priced for {_where(requested)})" entries = [entry for file in data for entry in file.coefficients] - entry = _entry(entries, spec.id, spec.version, keys, where) - values, lender = entry.values, None + entry = find_entry(entries, spec.id, spec.version, full, where) + carrier, lender = entry, None if entry.values_from is not None: lender = entry.values_from.estimator - lent = _entry(entries, lender, entry.values_from.version, keys, where) - if lent.values is None: + carrier = find_entry(entries, lender, entry.values_from.version, full, where) + if carrier.values is None: raise LookupError( f"Working-memory coefficients for estimator '{spec.id}' on {where} borrow from '{lender}', whose " "entry borrows in turn; borrow from the entry that carries the values." ) - values = lent.values + values = carrier.values assert values is not None, "a validated entry carries values or values_from" - if set(values) != spec.coefficients: + # An entry's own values are exactly what its spec reads; a lender's may be more (see `ValuesFrom`). + matches = set(values) == spec.coefficients if lender is None else spec.coefficients <= set(values) + if not matches: raise LookupError( f"Working-memory coefficients for estimator '{spec.id}' on {where} provide {sorted(values)}" f"{f' (from {lender})' if lender else ''}; the estimator reads {sorted(spec.coefficients)}." ) - priced = ",".join(value for key, value in keys.items() if value != requested[key]) - if priced and lender: - note = f" ({priced} values of {lender})" - elif priced: - note = f" ({priced} values)" - else: - note = f" (values of {lender})" if lender else "" - return ResolvedCoefficients(values=MappingProxyType(dict(values)), source=f"{origin}:{_where(requested)}{note}") + # The source names the spec's keys, and a refining key (the OS) where an entry it read names one. + shown = [*spec.platform_keys, *(key for key in full if key in {*entry.platform, *carrier.platform})] + shown = list(dict.fromkeys(shown)) + read = {name: values[name] for name in sorted(spec.coefficients)} + notes = [f"values of {lender}"] if lender else [] + if measured != platform: + floors = {name: floor for name, floor in carrier.unmeasured_floor.items() if name in read} + read |= {name: max(read[name], floor) for name, floor in floors.items()} + notes.insert(0, f"priced as {_where({key: full[key] for key in shown})}{', floored' if floors else ''}") + note = f" ({', '.join(notes)})" if notes else "" + source = f"{origin}:{_where({key: str(getattr(platform, key)) for key in shown})}{note}" + return ResolvedCoefficients(values=MappingProxyType(read), source=source) @functools.cache diff --git a/invokeai/backend/working_memory/counters.py b/invokeai/backend/working_memory/counters.py new file mode 100644 index 00000000000..2ffeb278722 --- /dev/null +++ b/invokeai/backend/working_memory/counters.py @@ -0,0 +1,352 @@ +"""This process's memory on a compute device, as the working-memory probe reads it at the boundaries of a window. + +Two kinds of counter, and neither alone is an operation's demand: + +- torch's caching allocator: the bytes it reserved from the driver and the bytes live tensors occupy, with exact peaks + since the last reset, on CUDA, ROCm and XPU. MPS keeps current values only; there `PeakSampler` keeps the peaks. +- the driver: what this process holds on the device, including what torch's allocator does not see -- cuBLAS and + cuDNN workspaces, kernel modules, and on ROCm the pool HIP keeps after a free. + +Measured on an RTX 4090 (torch 2.13+cu130, Windows): the driver delta equals torch's reserved-peak delta to three +decimals over VAE decode and encode, matmul, SDPA with and without a mask, and convolutions; first use outside the +allocator adds to the driver alone (+16 MiB at the first matmul, +2 MiB at the first convolution). Under +``backend:cudaMallocAsync`` the reserved peak can sit below the allocated peak (128 vs 136 MiB). On an RX 9060 XT (torch +2.13+rocm10, Windows, expandable segments) the driver keeps HIP's retained pool after ``empty_cache()``, so its delta +reads below reserved (4.3 vs 6.4 GiB for a 1024px decode), except for the first MIOpen decode, which grows it beyond +(2.0 vs 1.72 GiB). So the probe takes the largest delta of the counters it has. + +The driver is read at boundaries only: torch's peaks are exact, so what a boundary read misses is memory outside torch's +allocator that came and went between two boundaries. + +Every reader is best-effort: a failure reads as None. A peak reset or a synchronization that fails raises instead: the +window would go on with peaks from before its boundary or with work still in flight, so the probe, which never lets an +exception reach a generation, marks the window invalid. +""" + +import dataclasses +import sys +import threading +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Final, Literal, Optional + +import torch + +from invokeai.backend.util import nvml, wddm +from invokeai.backend.util.level_zero import xpu_memory_info +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.util.pci import pci_address + +Backend = Literal["cuda", "hip", "mps", "xpu"] + +Method = Literal["windows-d3dkmt", "nvml", "fdinfo", "mps-driver", "xpu-sysman", "torch-only"] +"""Where a reading's driver figures come from. + +- ``windows-d3dkmt``: this process's ``CurrentUsage`` in the adapter's local segment (`wddm.process_local_usage`), on + CUDA and ROCm under Windows, with the paged bytes from PDH. +- ``nvml``: the ``usedGpuMemory`` NVML attributes to this process, on CUDA under Linux. +- ``fdinfo``: ``drm-memory-vram`` summed over this process's DRM clients on the device, on ROCm under Linux. +- ``mps-driver``: ``torch.mps.driver_allocated_memory()``. +- ``xpu-sysman``: the device's used memory from Level Zero Sysman. Device-wide, so it guards against another process + loading the device during a window and is never taken as demand. +- ``torch-only``: no driver figure; the allocator's counters alone. +""" + +VERIFIED_METHODS: Final[frozenset[tuple[Method, Backend]]] = frozenset( + {("windows-d3dkmt", "cuda"), ("windows-d3dkmt", "hip")} +) +"""The (method, backend) pairs whose readings were checked on real hardware: the RTX 4090 and the RX 9060 XT above. The +other readers follow their vendors' documentation, and a window measured with them says so (``verified=False``).""" + +PEAK_SAMPLER_INTERVAL_S: float = 0.003 +"""How often `PeakSampler` samples. A few milliseconds is far below an operation's allocation lifetimes and costs one +getter call per sample.""" + +_FDINFO_DIR = Path("/proc/self/fdinfo") +_FDINFO_UNITS: Final[Mapping[str, int]] = {"": 1, "KiB": 2**10, "MiB": 2**20, "GiB": 2**30} + +# torch.cuda.memory_stats / torch.xpu.memory_stats keys, per CounterReading field. +_STAT_KEYS: Final[Mapping[str, str]] = { + "reserved": "reserved_bytes.all.current", + "allocated": "allocated_bytes.all.current", + "peak_reserved": "reserved_bytes.all.peak", + "peak_allocated": "allocated_bytes.all.peak", + "num_ooms": "num_ooms", + "num_alloc_retries": "num_alloc_retries", +} + + +@dataclass(frozen=True) +class CounterReading: + """One reading of a device's counters for this process, in bytes; None where the platform has no such counter or + its reader failed.""" + + reserved: Optional[int] + allocated: Optional[int] + peak_reserved: Optional[int] + """Since the last `reset_peaks`, like the other peaks.""" + peak_allocated: Optional[int] + num_ooms: Optional[int] + """Allocations that failed, cumulative for the process.""" + num_alloc_retries: Optional[int] + """Allocations that only succeeded after the allocator released its cache and retried, cumulative. A retry + squeezes the peak below what the operation would have taken with room to spare.""" + driver: Optional[int] + """This process's memory on the device as the driver counts it.""" + peak_driver: Optional[int] + """Only where a sampler keeps it (MPS); elsewhere the driver is read at boundaries.""" + paged: Optional[int] + """This process's bytes Windows keeps in shared system memory for the device's adapter.""" + device_used: Optional[int] + """Used memory of the whole device, every process together (XPU).""" + method: Method + + +def backend_for(device: torch.device) -> Optional[Backend]: + """The GPU backend `device` runs on; None for a device that is not a GPU. A HIP build reports ``cuda`` devices.""" + if device.type == "cuda": + return "hip" if torch.version.hip is not None else "cuda" + if device.type in ("mps", "xpu"): + return device.type # type: ignore[return-value] + return None + + +def _allocator_stats(device: torch.device) -> Mapping[str, int]: + try: + if device.type == "cuda": + return torch.cuda.memory_stats(device) + if device.type == "xpu": + return torch.xpu.memory_stats(device) + if device.type == "mps": + return {_STAT_KEYS["allocated"]: torch.mps.current_allocated_memory()} + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"Allocator statistics for {device} unavailable", exc_info=True) + return {} + + +def _fdinfo_size(value: str) -> Optional[int]: + number, _, unit = value.strip().partition(" ") + scale = _FDINFO_UNITS.get(unit.strip()) + return int(number) * scale if number.isdigit() and scale is not None else None + + +def fdinfo_vram_bytes(pdev: str) -> Optional[int]: + """VRAM this process holds on the DRM device at PCI address `pdev`, from ``/proc/self/fdinfo``. + + Each open DRM file reports its client's ``drm-memory-vram``. Duplicated descriptors share a client and report the + same memory, so each ``drm-client-id`` counts once. None when no descriptor of this process is a client of that + device. + """ + try: + entries = list(_FDINFO_DIR.iterdir()) + except OSError: + return None + total: Optional[int] = None + clients: set[str] = set() + for entry in entries: + try: + text = entry.read_text(encoding="utf-8", errors="replace") + except OSError: + # Closed since the listing, or not readable. + continue + fields = {key.strip(): value.strip() for key, _, value in (line.partition(":") for line in text.splitlines())} + if fields.get("drm-pdev", "").lower() != pdev.lower(): + continue + client = fields.get("drm-client-id") + if client is not None: + if client in clients: + continue + clients.add(client) + vram = _fdinfo_size(fields.get("drm-memory-vram", "")) + if vram is not None: + total = vram if total is None else total + vram + return total + + +def driver_method(backend: Optional[Backend], platform_name: Optional[str] = None) -> Method: + """The reader a `backend` device's driver figure comes from on the OS `platform_name` (a ``sys.platform`` value, + this process's by default), ``torch-only`` where there is none. A reading that fails falls back to ``torch-only`` + too.""" + platform_name = sys.platform if platform_name is None else platform_name + if backend in ("cuda", "hip") and platform_name == "win32": + return "windows-d3dkmt" + if backend == "cuda" and platform_name.startswith("linux"): + return "nvml" + if backend == "hip" and platform_name.startswith("linux"): + return "fdinfo" + if backend == "mps": + return "mps-driver" + if backend == "xpu": + return "xpu-sysman" + return "torch-only" + + +def method_verified(backend: Backend, platform_name: Optional[str] = None) -> bool: + """Whether the windows of a `backend` device on this OS read a driver figure verified on real hardware.""" + return (driver_method(backend, platform_name), backend) in VERIFIED_METHODS + + +def _driver_reading(device: torch.device) -> tuple[Optional[int], Optional[int], Optional[int], Method]: + """``(driver, paged, device_used, method)`` for `device` on this platform.""" + backend = backend_for(device) + driver: Optional[int] = None + paged: Optional[int] = None + device_used: Optional[int] = None + method = driver_method(backend) + try: + if method == "windows-d3dkmt": + driver, paged = wddm.process_local_usage(device), wddm.process_paged_bytes(device) + elif method == "nvml": + driver = nvml.process_used_memory(device) + elif method == "fdinfo": + index = device.index if device.index is not None else torch.cuda.current_device() + driver = fdinfo_vram_bytes(pci_address(index)) + elif method == "mps-driver": + driver = int(torch.mps.driver_allocated_memory()) + elif method == "xpu-sysman": + info = xpu_memory_info(device) + device_used = None if info is None else info[1] - info[0] + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"Driver memory reading for {device} failed", exc_info=True) + return None, None, None, "torch-only" + if driver is None and device_used is None: + method = "torch-only" + return driver, paged, device_used, method + + +def read_counters(device: torch.device) -> CounterReading: + """This process's counters on `device` now. Peaks are torch's, since the last `reset_peaks`; MPS has none.""" + stats = _allocator_stats(device) + values = {field: (None if key not in stats else int(stats[key])) for field, key in _STAT_KEYS.items()} + driver, paged, device_used, method = _driver_reading(device) + return CounterReading( + **values, driver=driver, peak_driver=None, paged=paged, device_used=device_used, method=method + ) + + +def reset_peaks(device: torch.device) -> None: + """Start torch's peak counters over at the current values. Cumulative counters (OOMs, retries) are kept. Raises + what torch raises.""" + if device.type == "cuda": + torch.cuda.reset_peak_memory_stats(device) + elif device.type == "xpu": + torch.xpu.reset_peak_memory_stats(device) + + +def synchronize(device: torch.device) -> None: + """Wait for the work queued on `device`, so frees it implies have happened. Raises what torch raises.""" + if device.type == "cuda": + torch.cuda.synchronize(device) + elif device.type == "xpu": + torch.xpu.synchronize(device) + elif device.type == "mps": + torch.mps.synchronize() + + +def _larger(a: Optional[int], b: Optional[int]) -> Optional[int]: + if a is None: + return b + return a if b is None else max(a, b) + + +class PeakSampler: + """The largest values `read` returned since the last `reset`, sampled every `PEAK_SAMPLER_INTERVAL_S` on a daemon + thread between `start` and `stop`. + + For a device whose counters have no peaks: without it, a window would see only the values at its boundaries and + miss everything that came and went in between. `read` returns one value per counter, None where it has none; a + `read` that raises is a sample of Nones. + """ + + def __init__(self, read: Callable[[], tuple[Optional[int], ...]]) -> None: + self._read = read + self._lock = threading.Lock() + self._peaks: tuple[Optional[int], ...] = () + self._stop = threading.Event() + self._thread: Optional[threading.Thread] = None + + def _sample(self) -> tuple[Optional[int], ...]: + try: + return self._read() + except Exception: + return () + + def _fold(self, sample: tuple[Optional[int], ...]) -> tuple[Optional[int], ...]: + with self._lock: + width = max(len(self._peaks), len(sample)) + peaks = self._peaks + (None,) * (width - len(self._peaks)) + sample = sample + (None,) * (width - len(sample)) + self._peaks = tuple(_larger(peak, value) for peak, value in zip(peaks, sample, strict=True)) + return self._peaks + + def _run(self) -> None: + while not self._stop.wait(PEAK_SAMPLER_INTERVAL_S): + self._fold(self._sample()) + + def start(self) -> None: + self.reset() + self._stop.clear() + self._thread = threading.Thread(target=self._run, name="working-memory-peak-sampler", daemon=True) + self._thread.start() + + def stop(self) -> None: + self._stop.set() + if self._thread is not None: + self._thread.join() + self._thread = None + + @property + def running(self) -> bool: + return self._thread is not None and self._thread.is_alive() + + def reset(self) -> None: + """Start the peaks over at the values now.""" + sample = self._sample() + with self._lock: + self._peaks = sample + + def peaks(self) -> tuple[Optional[int], ...]: + """The peaks since the last reset, the values now included.""" + return self._fold(self._sample()) + + +def _mps_values() -> tuple[Optional[int], ...]: + return int(torch.mps.current_allocated_memory()), int(torch.mps.driver_allocated_memory()) + + +class DeviceCounters: + """One device's counters for the length of a window: readings, peak resets and synchronization. + + The peaks are torch's where it keeps them. On MPS, which keeps none, a `PeakSampler` keeps the allocated and the + driver peaks from construction until `close`. + """ + + def __init__(self, device: torch.device) -> None: + self.device = device + self._sampler = PeakSampler(_mps_values) if device.type == "mps" else None + if self._sampler is not None: + self._sampler.start() + + def read(self) -> CounterReading: + reading = read_counters(self.device) + if self._sampler is None: + return reading + peak_allocated, peak_driver = (self._sampler.peaks() + (None, None))[:2] + return dataclasses.replace( + reading, + peak_allocated=_larger(peak_allocated, reading.allocated), + peak_driver=_larger(peak_driver, reading.driver), + ) + + def reset_peaks(self) -> None: + reset_peaks(self.device) + if self._sampler is not None: + self._sampler.reset() + + def synchronize(self) -> None: + synchronize(self.device) + + def close(self) -> None: + if self._sampler is not None: + self._sampler.stop() diff --git a/invokeai/backend/working_memory/data/denoise.json b/invokeai/backend/working_memory/data/denoise.json new file mode 100644 index 00000000000..b8004fd0468 --- /dev/null +++ b/invokeai/backend/working_memory/data/denoise.json @@ -0,0 +1,6312 @@ +{ + "schema_version": 1, + "coefficients": [ + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values": { + "k_denoise_bf16": 266684, + "b_denoise_bf16": 0, + "k_denoise_fp16": 266684, + "b_denoise_fp16": 0, + "k_denoise_fp32": 459628, + "b_denoise_fp32": 0 + }, + "unmeasured_floor": { + "k_denoise_bf16": 262144, + "b_denoise_bf16": 1073741824, + "k_denoise_fp16": 262144, + "b_denoise_fp16": 1073741824, + "k_denoise_fp32": 262144, + "b_denoise_fp32": 1073741824 + }, + "provenance": "The Z-Image transformer forward's lines, bytes per attended token plus a base, per dtype. RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA. Measured by scripts/calibrate_z_image_working_memory.py on 2026-10-18: bf16, fp16 and fp32 at 512, 768, 1024, 1440 and 2048px with a 512-token caption, and at 1024 and 1440px with a 77-token one, 21 plain forwards on a random-init transformer of the real geometry (memory does not depend on weight values) with 2 main blocks and one refiner block each, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB, cudaMallocAsync). bf16 and fp16 measured identical; fp32 needs about 1.7 times as much per token. Every line runs through the origin: no point needs a base. Where this entry prices a platform nothing is measured on (MPS, XPU, the CPU, or cuDNN with expandable segments), each value is floored at the legacy constants the node reserved by hand, 0.25 MiB per attended token and a 1 GiB base for every dtype: the fit describes the card its points were measured on, not the platforms it stands in for." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "values": { + "k_denoise_bf16": 189236, + "b_denoise_bf16": 9437184, + "k_denoise_fp16": 189236, + "b_denoise_fp16": 9437184, + "k_denoise_fp32": 341719, + "b_denoise_fp32": 0 + }, + "provenance": "The Z-Image transformer forward's lines, bytes per attended token plus a base, per dtype. RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Measured by scripts/calibrate_z_image_working_memory.py on 2026-10-18: bf16, fp16 and fp32 at 512, 768, 1024, 1440 and 2048px with a 512-token caption, and at 1024 and 1440px with a 77-token one, 21 plain forwards on a random-init transformer of the real geometry (memory does not depend on weight values) with 2 main blocks and one refiner block each, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. 4 main blocks measured no more at 1024px bf16 (768 MiB against 800 MiB). The real 30-block Z-Image Turbo in a running server measured below this line: 2176 MiB of demand at 15424 tokens in bf16, which it prices at 2793 MiB (calibration probe windows, kept as evidence and not shipped)." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values": { + "k_denoise_bf16": 262144, + "b_denoise_bf16": 1073741824, + "k_denoise_fp16": 262144, + "b_denoise_fp16": 1073741824, + "k_denoise_fp32": 262144, + "b_denoise_fp32": 1073741824 + }, + "legacy": true, + "provenance": "Legacy, not a fit, for every OS but Windows, which has its own entry: Linux ROCm with the native allocator is not measured. The constants the Z-Image denoise node reserved by hand, 0.25 MiB per attended token and a 1 GiB base, the same for every dtype, derived from peak reserved memory in bf16 on an RTX 4090 with a 512-token caption (scripts/calibrate_z_image_working_memory.py): 0.99 GiB at 1024px, 1.96 GiB at 1440px and 3.85 GiB at 2048px, 0.22 to 0.23 MB per token. They cover bf16 and fp16 on every platform measured since, but not fp32 from 1440px up: every measured fp32 line is steeper (the RX 9060 XT with the native allocator on Windows measured 6.9 GiB at 2048px, which these constants price at 5.1 GiB)." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "values": { + "k_denoise_bf16": 267790, + "b_denoise_bf16": 233832448, + "k_denoise_fp16": 267790, + "b_denoise_fp16": 271581184, + "k_denoise_fp32": 495266, + "b_denoise_fp32": 42991616 + }, + "unmeasured_floor": { + "k_denoise_bf16": 262144, + "b_denoise_bf16": 1073741824, + "k_denoise_fp16": 262144, + "b_denoise_fp16": 1073741824, + "k_denoise_fp32": 262144, + "b_denoise_fp32": 1073741824 + }, + "provenance": "Windows only; every other OS keeps its own entry. The Z-Image transformer forward's lines, bytes per attended token plus a base, per dtype. RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Measured by scripts/calibrate_z_image_working_memory.py on 2026-10-18: bf16, fp16 and fp32 at 512, 768, 1024, 1440 and 2048px with a 512-token caption, and at 1024 and 1440px with a 77-token one, 21 plain forwards on a random-init transformer of the real geometry (memory does not depend on weight values) with 2 main blocks and one refiner block each, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, expandable segments). The base carries the ~250 MiB a process's first forward maps beyond its second (585 against 334 MiB at 512px bf16). Where this entry prices a platform nothing is measured on (MIOpen under hipMallocAsync on Windows), each value is floored at the legacy constants the node reserved by hand, 0.25 MiB per attended token and a 1 GiB base for every dtype: the fit describes the card its points were measured on, not the platform it stands in for." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "values": { + "k_denoise_bf16": 167383, + "b_denoise_bf16": 269484032, + "k_denoise_fp16": 167383, + "b_denoise_fp16": 306184192, + "k_denoise_fp32": 318260, + "b_denoise_fp32": 134217728 + }, + "provenance": "Windows only, where expandable segments are Invoke's ROCm default. The Z-Image transformer forward's lines, bytes per attended token plus a base, per dtype. RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Measured by scripts/calibrate_z_image_working_memory.py on 2026-10-18: bf16, fp16 and fp32 at 512, 768, 1024, 1440 and 2048px with a 512-token caption, and at 1024 and 1440px with a 77-token one, 21 plain forwards on a random-init transformer of the real geometry (memory does not depend on weight values) with 2 main blocks and one refiner block each, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. 4 main blocks measured the same at 1024px bf16 (937 MiB). The base carries the ~255 MiB a process's first forward maps beyond its second (477 against 222 MiB at 512px bf16). The real 30-block Z-Image Turbo in a running server measured below this line: 686 MiB of demand at 4128 tokens in bf16, which it prices at 1066 MiB (calibration probe windows, kept as evidence and not shipped)." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values": { + "k_denoise_bf16_h3072": 304528, + "b_denoise_bf16_h3072": 0, + "k_denoise_bf16_h4096": 406842, + "b_denoise_bf16_h4096": 0, + "k_denoise_bf16_h6144": 607199, + "b_denoise_bf16_h6144": 0 + }, + "unmeasured_floor": { + "k_denoise_bf16_h3072": 330300, + "b_denoise_bf16_h3072": 1073741824, + "k_denoise_bf16_h4096": 440401, + "b_denoise_bf16_h4096": 1073741824, + "k_denoise_bf16_h6144": 660601, + "b_denoise_bf16_h6144": 1073741824 + }, + "provenance": "The FLUX.2 transformer forward's lines, bytes per attended token and sample plus a base, one per width (3072, 4096, 6144), bf16. RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA. Measured by scripts/calibrate_flux2_denoise_working_memory.py on 2026-10-18 per width: 512, 768, 1024, 1440 and 2048px with a 512-token caption, 1024 and 1440px with a 77-token one, and at 1024px one and three 1024px reference images and a batch of two, and with --mode large long sequences: 2048px with one 2024px reference (33025 tokens), 1536px with one (25857) and 1328px with three 1328px references, which the node resizes to 1024px each (19689); plain forwards on a random-init transformer of the model loader's geometry (memory does not depend on weight values) with 2 double-stream and 2 single-stream blocks, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. Reference images and the batch lie on the line of the plain forwards. 4 blocks per stream measured the same as 2 at 1024px at every width on this card (native allocator). The real models in a running server (calibration probe windows, kept as evidence and not shipped) demanded 0.96 to 0.99 times the 2-block point at the same tokens for an fp8 Klein 4B at 1024 to 2048px, and 0.96 to 0.97 for an fp8 Klein 9B, some of its windows partially resident. The Klein 4B line holds its 1440px forward with a 77-token caption at 1.42x (`bounds.UPPER_BOUND_EXCEPTIONS`): the native allocator served it from 1676 MiB, the 512-token caption at 5% more tokens from 2382 MiB. Where this entry prices a platform nothing is measured on (MPS, XPU, the CPU, or cuDNN with expandable segments), each value is floored at the legacy constants the node reserved by hand, 0.42 MiB per token at width 4096 scaled by the width (330300, 440401 and 660601 bytes) and a 1 GiB base: the fit describes the card its points were measured on, not the platforms it stands in for." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "values": { + "k_denoise_bf16_h3072": 202699, + "b_denoise_bf16_h3072": 0, + "k_denoise_bf16_h4096": 267429, + "b_denoise_bf16_h4096": 11534336, + "k_denoise_bf16_h6144": 359718, + "b_denoise_bf16_h6144": 65011712 + }, + "provenance": "The FLUX.2 transformer forward's lines, bytes per attended token and sample plus a base, one per width (3072, 4096, 6144), bf16. RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Measured by scripts/calibrate_flux2_denoise_working_memory.py on 2026-10-18 per width: 512, 768, 1024, 1440 and 2048px with a 512-token caption, 1024 and 1440px with a 77-token one, and at 1024px one and three 1024px reference images and a batch of two, and with --mode large long sequences: 2048px with one 2024px reference (33025 tokens), 1536px with one (25857) and 1328px with three 1328px references, which the node resizes to 1024px each (19689); plain forwards on a random-init transformer of the model loader's geometry (memory does not depend on weight values) with 2 double-stream and 2 single-stream blocks, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. Reference images and the batch lie on the line of the plain forwards. The driver's pool demands 33 to 41% less per token than the native allocator on this card." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values": { + "k_denoise_bf16_h3072": 330300, + "b_denoise_bf16_h3072": 1073741824, + "k_denoise_bf16_h4096": 440401, + "b_denoise_bf16_h4096": 1073741824, + "k_denoise_bf16_h6144": 660601, + "b_denoise_bf16_h6144": 1073741824 + }, + "legacy": true, + "provenance": "Legacy, not a fit, for every OS but Windows, which has its own entry: no fresh-process point measures ROCm with the native allocator yet. The constants the FLUX.2 denoise node reserved by hand: 0.42 MiB per attended token and sample at the Klein 9B width (4096), scaled linearly by the transformer's width and truncated to whole bytes per width (330300 at 3072, 440401 at 4096, 660601 at 6144), and a 1 GiB base. The 0.42 bounded the slope of peak reserved memory between 4608 and 9216 tokens on the Klein 9B geometry in bf16, 0.386 MiB per token on an RTX 4090 (torch 2.7.1) and 0.407 on an RX 9070 XT (gfx1201, torch 2.10); per width 0.291, 0.386 and 0.555 MiB on CUDA and 0.315, 0.407 and 0.589 on ROCm, each point in a fresh process on a random-init transformer with reduced blocks." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "values": { + "k_denoise_bf16_h3072": 305267, + "b_denoise_bf16_h3072": 245366784, + "k_denoise_bf16_h4096": 404611, + "b_denoise_bf16_h4096": 238026752, + "k_denoise_bf16_h6144": 601721, + "b_denoise_bf16_h6144": 232783872 + }, + "unmeasured_floor": { + "k_denoise_bf16_h3072": 330300, + "b_denoise_bf16_h3072": 1073741824, + "k_denoise_bf16_h4096": 440401, + "b_denoise_bf16_h4096": 1073741824, + "k_denoise_bf16_h6144": 660601, + "b_denoise_bf16_h6144": 1073741824 + }, + "provenance": "Windows only; every other OS keeps its own entry. The FLUX.2 transformer forward's lines, bytes per attended token and sample plus a base, one per width (3072, 4096, 6144), bf16. RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Measured by scripts/calibrate_flux2_denoise_working_memory.py on 2026-10-18 per width: 512, 768, 1024, 1440 and 2048px with a 512-token caption, 1024 and 1440px with a 77-token one, and at 1024px one and three 1024px reference images and a batch of two, and with --mode large long sequences: 2048px with one 2024px reference (33025 tokens), 1536px with one (25857) and 1328px with three 1328px references, which the node resizes to 1024px each (19689); plain forwards on a random-init transformer of the model loader's geometry (memory does not depend on weight values) with 2 double-stream and 2 single-stream blocks, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. Reference images and the batch lie on the line of the plain forwards. 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB). [dev]'s line comes from 512 to 1440px, one 1024px reference included: at 2048px, with three references and in the large grid it does not fit 16 GB beside its weights; the Klein 4B and 9B lines are measured up to 33025 tokens where they fit. The base carries the ~245 MiB a process's first forward maps beyond its second (565 against 320 MiB at 512px, width 3072). The Klein 4B line holds its 1440px forward with a 77-token caption at 1.40x (`bounds.UPPER_BOUND_EXCEPTIONS`): the native allocator served it from 1879 MiB, the 512-token caption at 5% more tokens from 2577 MiB. Where this entry prices a platform nothing is measured on (MIOpen under hipMallocAsync on Windows), each value is floored at the legacy constants the node reserved by hand, 0.42 MiB per token at width 4096 scaled by the width (330300, 440401 and 660601 bytes) and a 1 GiB base: the fit describes the card its points were measured on, not the platforms it stands in for." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "values": { + "k_denoise_bf16_h3072": 181485, + "b_denoise_bf16_h3072": 292552704, + "k_denoise_bf16_h4096": 241980, + "b_denoise_bf16_h4096": 283115520, + "k_denoise_bf16_h6144": 362969, + "b_denoise_bf16_h6144": 284164096 + }, + "provenance": "Windows only, where expandable segments are Invoke's ROCm default. The FLUX.2 transformer forward's lines, bytes per attended token and sample plus a base, one per width (3072, 4096, 6144), bf16. RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Measured by scripts/calibrate_flux2_denoise_working_memory.py on 2026-10-18 per width: 512, 768, 1024, 1440 and 2048px with a 512-token caption, 1024 and 1440px with a 77-token one, and at 1024px one and three 1024px reference images and a batch of two, and with --mode large long sequences: 2048px with one 2024px reference (33025 tokens), 1536px with one (25857) and 1328px with three 1328px references, which the node resizes to 1024px each (19689); plain forwards on a random-init transformer of the model loader's geometry (memory does not depend on weight values) with 2 double-stream and 2 single-stream blocks, one fresh process per point with the server's runtime, demand_peak (the largest of the driver, reserved and allocated peak) of the larger of a cold and a warm forward. Fitted by scripts/working_memory/build_shipped.py: working_memory/fit.py's upper envelope of the points below, plus 5%. Reference images and the batch lie on the line of the plain forwards. 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB). [dev]'s line comes from 512 to 1440px, one 1024px reference included: at 2048px, with three references and in the large grid it does not fit 16 GB beside its weights; the Klein 4B and 9B lines are measured up to 33025 tokens where they fit. The base carries the ~245 MiB a process's first forward maps beyond its second (453 against 224 MiB at 512px, width 3072). The real models in a running server (calibration probe windows, kept as evidence and not shipped) demanded 0.81 to 0.95 times the 2-block point at the same tokens for a bf16 Klein 4B, and 0.71 to 0.77 for an fp8 Klein 9B after its 576 MiB fp8 cast term: the 2-block points bound the real models, the extra blocks' 5 to 7% included." + } + ], + "points": [ + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 367001600, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 645922816, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1077936128, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2118123520, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4148166656, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1050673152, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2088763392, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 367001600, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 645922816, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1077936128, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2118123520, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4148166656, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1050673152, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2088763392, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 624951296, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1101004800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1881145344, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3667918848, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7266631680, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1835008000, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3579838464, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 285212672, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 452984832, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 838860800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1426063360, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2969567232, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 754974720, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1409286144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 285212672, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 452984832, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 838860800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1426063360, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2785017856, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 754974720, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1459617792, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 446693376, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 916455424, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1352663040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2627731456, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5077204992, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1319108608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2550136832, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; MIOPEN_FIND_MODE, AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured no more at 1024px bf16 on this card (768 MiB against 800 MiB with 2, under cudaMallocAsync). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 500629504, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 689373184, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 982974464, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1633091584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2618753024, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 857145344, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1507262464, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 536346624, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 725090304, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1018691584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1668808704, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2654470144, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 892862464, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1542979584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 593362944, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 949878784, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1432223744, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2627600384, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5249040384, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1355415552, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2257190912, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 613875712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 892796928, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1324810240, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2341928960, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4501995520, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1291255808, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "bf16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2308374528, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 649592832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 928514048, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1360527360, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2377646080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4537712640, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1326972928, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp16", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2344091648, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 1024, + "text_tokens": 512, + "tokens": 1536, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 765329408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 2304, + "text_tokens": 512, + "tokens": 2816, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1258160128, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 512, + "tokens": 4608, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2036203520, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 512, + "tokens": 8640, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3802005504, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 16384, + "text_tokens": 512, + "tokens": 16896, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7383941120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 4096, + "text_tokens": 96, + "tokens": 4192, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2018115584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.z_image", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "image_tokens": 8128, + "text_tokens": 96, + "tokens": 8224, + "dtype": "fp32", + "control_blocks": 0, + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3721003008, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_z_image_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init Z-Image transformer with the real geometry (memory does not depend on weight values), 2 main blocks and one refiner block each; 4 main blocks measured the same at 1024px bf16 on this card (937 MiB, under expandable segments). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 373293056, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 799014912, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1317011456, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2497708032, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4884267008, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1207959552, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1757413376, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2506096640, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4884267008, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2606759936, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 494927872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1046478848, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1738539008, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3319791616, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 6505365504, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1616904192, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2791309312, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3338665984, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 6505365504, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3433037824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 742391808, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1564475392, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2585788416, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4940890112, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 9718202368, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2397044736, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4185915392, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4974444544, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 9718202368, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5140119552, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 285212672, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 520093696, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 822083584, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1476395008, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2919235584, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 805306368, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1392508928, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1493172224, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2919235584, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1627389952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 385875968, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 721420288, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1090519040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1946157056, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3707764736, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1073741824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1828716544, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1979711488, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3707764736, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2130706432, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 587202560, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1023410176, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1627389952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2919235584, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5486149632, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1442840576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2734686208, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2952790016, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5486149632, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3070230528, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 475398144, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 685113344, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 936771584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1649803264, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2991980544, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 999686144, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1691746304, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1670774784, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2991980544, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1649803264, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 580255744, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 831913984, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1188429824, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2132148224, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3914727424, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1230372864, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2153119744, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2153119744, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3914727424, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2090205184, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 727056384, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1104543744, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1649803264, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3033923584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1712717824, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3096838144, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3075866624, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3012952064, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 592838656, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1035337728, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1538654208, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2702573568, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5049286656, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1446379520, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1970667520, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2713059328, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5049286656, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2824208384, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 722862080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1289093120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1976958976, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3509977088, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 6636830720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1834352640, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3025534976, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3528851456, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 16896, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 6636830720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3671457792, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 1024, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 1536, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 974520320, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 2304, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 2816, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 1784020992, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2824208384, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 8612, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5156241408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 4173, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 2612396032, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 8100, + "reference_tokens": 0, + "text_tokens": 77, + "tokens": 8177, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4399169536, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 4096, + "text_tokens": 512, + "tokens": 8704, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5189795840, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 4096, + "reference_tokens": 0, + "text_tokens": 512, + "tokens": 4608, + "batch_size": 2, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5357568000, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 9412018176, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7381975040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5635047424, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 12549357568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 10009706496, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7453278208, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 14952693760, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 11358175232, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, no allocator configuration (the native allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 6375342080, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4966055936, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3422552064, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 8422162432, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5939134464, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4529848320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 11374952448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 8371830784, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "hidden_size": 6144, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 6425673728, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RTX 4090, torch 2.13.0+cu130, Windows, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (the cudaMallocAsync allocator), fused SDPA; AOTriton and the ROCm SDPA guard are ROCm's and do not apply. Random-init FLUX.2 transformer of width 6144 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured the same as 2 at 1024px at every width on this card (1256, 1658 and 2466 MiB, native allocator). demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5613420544, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4439015424, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 3453353984, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7354056704, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5823135744, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 4480958464, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (expandable segments), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 16384, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 33025, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 9660923904, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7605714944, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 3072, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 5957353472, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 3072 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 9216, + "reference_tokens": 16129, + "text_tokens": 512, + "tokens": 25857, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 10063577088, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + }, + { + "estimator": "denoise.flux2", + "version": 1, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "hidden_size": 4096, + "image_tokens": 6889, + "reference_tokens": 12288, + "text_tokens": 512, + "tokens": 19689, + "batch_size": 1, + "dtype": "bf16", + "has_attention_mask": false, + "regional_bias_bytes": 0, + "score_matrix_bytes": 0, + "lora_count": 0, + "dequant_transient_bytes": 0, + "fp8_cast_transient_bytes": 0 + }, + "measured_bytes": 7813332992, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "scripts/calibrate_flux2_denoise_working_memory.py, 2026-10-18: RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0 (HIP 7.15.26333), Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 (fused SDPA), ROCm SDPA guard installed. Random-init FLUX.2 transformer of width 4096 with the model loader's geometry (memory does not depend on weight values), 2 double-stream and 2 single-stream blocks; 4 blocks per stream measured 5 to 7% more than 2 at 1024px on this card (953 against 893 MiB at width 3072, 1193 against 1133 MiB at 4096, expandable segments; [dev] with 4 blocks does not fit 16 GB), which the 5% margin and the real models' lower demand cover. demand_peak: the largest of the driver, reserved and allocated peak over the baseline, the larger of a cold and a warm forward, one fresh process per point with the server's runtime; no failed allocation, allocator retry or spill." + } + ] +} diff --git a/invokeai/backend/working_memory/data/vae.json b/invokeai/backend/working_memory/data/vae.json index bae9081384f..5321cdbec12 100644 --- a/invokeai/backend/working_memory/data/vae.json +++ b/invokeai/backend/working_memory/data/vae.json @@ -3,131 +3,423 @@ "coefficients": [ { "estimator": "vae.sd", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { - "estimator": "vae.sd", - "version": 1, + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values": { + "k_decode_fp16": 4597, + "b_decode_fp16": 83886080, + "k_decode_bf16": 4500, + "b_decode_bf16": 68157440, + "k_decode_fp32": 7702, + "b_decode_fp32": 0, + "k_encode_fp16": 2226, + "b_encode_fp16": 22020096, + "k_encode_bf16": 2249, + "b_encode_bf16": 18874368, + "k_encode_fp32": 3236, + "b_encode_fp32": 41943040 + }, + "unmeasured_floor": { + "k_decode_fp16": 4400, + "k_decode_bf16": 4400, + "k_decode_fp32": 8800, + "k_encode_fp16": 2200, + "k_encode_bf16": 2200, + "k_encode_fp32": 4400 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on the RTX 4090 points (torch 2.13 and, for the older FLUX.2 rows, 2.7.1) of the SDXL VAE (fp16, fp32), the FLUX.1 VAE (bf16, fp32) and the FLUX.2 VAE (bf16, fp32). Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 2200 bytes per pixel per element byte for a decode, 1100 for an encode." + }, + { + "estimator": "vae.flux", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "cudnn", + "allocator": "native" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { - "estimator": "vae.sd3", - "version": 1, + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values_from": { + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.2 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values": { + "k_decode_bf16": 8696, + "b_decode_bf16": 66060288, + "k_encode_bf16": 2420, + "b_encode_bf16": 22020096 + }, + "unmeasured_floor": { + "k_decode_bf16": 4400, + "k_encode_bf16": 2200 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on its RTX 4090 points (torch 2.13, native allocator). Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 2200 bytes per pixel per element byte for a decode, 1100 for an encode." + }, + { + "estimator": "vae.sd", + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD3 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { "estimator": "vae.sd3", - "version": 1, + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "values": { + "k_decode_fp16": 3797, + "b_decode_fp16": 0, + "k_decode_bf16": 3614, + "b_decode_bf16": 0, + "k_decode_fp32": 5914, + "b_decode_fp32": 13631488, + "k_encode_fp16": 1882, + "b_encode_fp16": 17825792, + "k_encode_bf16": 1882, + "b_encode_bf16": 17825792, + "k_encode_fp32": 2722, + "b_encode_fp32": 3145728 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on the RTX 4090 points (torch 2.13, backend:cudaMallocAsync) of the SDXL VAE (fp16, fp32), the FLUX.1 VAE (bf16, fp32) and the FLUX.2 VAE (bf16, fp32)." + }, + { + "estimator": "vae.flux", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD3 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { - "estimator": "vae.cogview4", - "version": 1, + "estimator": "vae.flux2", + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the CogView4 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.2 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { "estimator": "vae.cogview4", - "version": 1, + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "values": { + "k_decode_bf16": 6765, + "b_decode_bf16": 0, + "k_encode_bf16": 1882, + "b_encode_bf16": 9437184 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on its RTX 4090 points (torch 2.13, backend:cudaMallocAsync)." + }, + { + "estimator": "vae.sd", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the CogView4 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "values": { + "k_decode_fp16": 7200, + "b_decode_fp16": 0, + "k_decode_bf16": 7143, + "b_decode_bf16": 68157440, + "k_decode_fp32": 14400, + "b_decode_fp32": 0, + "k_encode_fp16": 5500, + "b_encode_fp16": 0, + "k_encode_bf16": 5645, + "b_encode_bf16": 0, + "k_encode_fp32": 11000, + "b_encode_fp32": 0 + }, + "stand_ins": [ + "k_decode_fp16", + "b_decode_fp16", + "k_decode_fp32", + "b_decode_fp32", + "k_encode_fp16", + "b_encode_fp16", + "k_encode_fp32", + "b_encode_fp32" + ], + "unmeasured_floor": { + "k_decode_fp16": 7200, + "k_decode_bf16": 7200, + "k_decode_fp32": 14400, + "k_encode_fp16": 5500, + "k_encode_bf16": 5500, + "k_encode_fp32": 11000 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Every OS but Windows, which has its own entry. Fitted: bf16 on the FLUX.2 VAE points of Linux with the native allocator (RX 9070 XT gfx1201 and PRO W7900 gfx1100, torch 2.10). Stand-ins, no measurement here: fp16 and fp32 at 3600 decode / 2750 encode bytes per pixel per element byte, no base, the pre-dtype figure. Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." }, { "estimator": "vae.flux", - "version": 1, + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values_from": { + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values_from": { + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.2 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values": { + "k_decode_bf16": 13720, + "b_decode_bf16": 14680064, + "k_encode_bf16": 5500, + "b_encode_bf16": 161480704 + }, + "stand_ins": [ + "k_decode_bf16", + "b_decode_bf16", + "k_encode_bf16", + "b_encode_bf16" + ], + "unmeasured_floor": { + "k_decode_bf16": 7200, + "k_encode_bf16": 5500 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Every OS but Windows, which has its own entry. Not measured here: both lines are stand-ins at its fitted MIOpen expandable-segments lines, as the family's bf16 lines measure the same under the native allocator on Linux and under expandable segments on Windows (7208 vs 7117 MiB decode at 1024px), with each slope raised to the pre-dtype figure where that is larger (the encode). Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." + }, + { + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "values": { + "k_decode_fp16": 3710, + "b_decode_fp16": 274726912, + "k_decode_bf16": 11009, + "b_decode_bf16": 0, + "k_decode_fp32": 10975, + "b_decode_fp32": 0, + "k_encode_fp16": 3703, + "b_encode_fp16": 77594624, + "k_encode_bf16": 4974, + "b_encode_bf16": 1104150528, + "k_encode_fp32": 4114, + "b_encode_fp32": 531628032 + }, + "unmeasured_floor": { + "k_decode_fp16": 7200, + "k_decode_bf16": 7200, + "k_decode_fp32": 14400, + "k_encode_fp16": 5500, + "k_encode_bf16": 5500, + "k_encode_fp32": 11000 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows: fitted on the RX 9060 XT (gfx1200, torch 2.13+rocm10) points of the SDXL VAE (fp16, fp32), the FLUX.1 VAE (bf16, fp32) and the FLUX.2 VAE (bf16, fp32) with expandable segments switched off. The demand is concave in the size, so the envelope over-reserves some points by more than 1.3x; the data test lists them. Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "values": { + "k_decode_bf16": 17640, + "b_decode_bf16": 0, + "k_encode_bf16": 5060, + "b_encode_bf16": 1059061760 + }, + "unmeasured_floor": { + "k_decode_bf16": 7200, + "k_encode_bf16": 5500 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows: fitted on its RX 9060 XT (gfx1200, torch 2.13+rocm10) points with expandable segments switched off. Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." + }, + { + "estimator": "vae.sd", + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "values": { + "k_decode_fp16": 2016, + "b_decode_fp16": 506462208, + "k_decode_bf16": 6612, + "b_decode_bf16": 529530880, + "k_decode_fp32": 5988, + "b_decode_fp32": 80740352, + "k_encode_fp16": 1960, + "b_encode_fp16": 353370112, + "k_encode_bf16": 3255, + "b_encode_bf16": 483393536, + "k_encode_fp32": 3898, + "b_encode_fp32": 128974848 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. The FLUX.1 point below falls inside them." + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows only, where expandable segments are Invoke's ROCm default: measured on gfx1200 alone. Elsewhere they are a user override nothing is measured for, priced as MIOpen with the native allocator on that OS. Fitted on the RX 9060 XT (gfx1200, torch 2.13+rocm10) points of the SDXL VAE (fp16, fp32) and the FLUX.1 VAE (bf16, fp32). vae.flux2 carries its own lines here: its fp32 decode differs from the family's by more than the margin." }, { "estimator": "vae.flux", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. Confirmed on the FLUX.1 VAE: RX 9060 XT (gfx1200), torch 2.12, Windows, bf16, 768-1536px, Invoke's ROCm SDPA guard installed, MIOPEN_FIND_MODE=FAST: implied 3451 decode / 2688 encode." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values": { - "k_decode": 2200, - "k_encode": 1100 + "k_decode_bf16": 6612, + "b_decode_bf16": 529530880, + "k_decode_fp32": 5842, + "b_decode_fp32": 272629760, + "k_encode_bf16": 3255, + "b_encode_bf16": 483393536, + "k_encode_fp32": 3898, + "b_encode_fp32": 128974848 }, - "provenance": "Peak reserved bytes per output pixel per element byte of the diffusers AutoencoderKL convolution stack. Fitted on the FLUX.2 VAE (the vae.flux2 cudnn points) with scripts/calibrate_flux2_working_memory.py --only vae: RTX 4090, torch 2.7.1, bf16, untiled, fused SDPA; implied 2180/2185/2165 decode and 1072/1063/1061 encode at 512/768/1024px, flat across resolution. Rounded up to 2200 / 1100, the pair the FLUX.1 estimator already shipped." + "provenance": "The FLUX.2 VAE's own convolution lines under MIOpen with expandable segments. Windows only, where expandable segments are Invoke's ROCm default: measured on gfx1200 alone. Elsewhere they are a user override nothing is measured for, priced as MIOpen with the native allocator on that OS. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on its RX 9060 XT (gfx1200, torch 2.13+rocm10) points. Not borrowed from vae.sd3: its fp32 decode needs ~140 MiB more at 512px than the SDXL and FLUX.1 VAEs, which would lift the family's fp32 decode line 9% above theirs, past the 5% margin. Its bf16 lines and fp32 encode equal the family's." }, { - "estimator": "vae.flux2", - "version": 1, + "estimator": "vae.cogview4", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values": { - "k_decode": 3600, - "k_encode": 2750 + "k_decode_bf16": 13720, + "b_decode_bf16": 14680064, + "k_encode_bf16": 3764, + "b_encode_bf16": 161480704 }, - "provenance": "Peak reserved bytes per output pixel per element byte of the diffusers AutoencoderKL convolution stack on MIOpen. Fitted on the FLUX.2 VAE (the vae.flux2 miopen points) with scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled: RX 9070 XT (gfx1201) implied 3453/3369/3368 decode, PRO W7900 (gfx1100) 3525 at 512px; encode 2687/2688/2688 on both cards, to the byte. Two AMD generations agreeing that closely make this MIOpen rather than a per-card quirk, so the values are the larger card's plus ~2% headroom: 3600 / 2750. MIOpen's convolution workspaces are larger than cuDNN's (~1.6x decode, ~2.4x encode), identically with fused and math attention, and its encode/decode ratio is 0.78 against cuDNN's 0.49." + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows only, where expandable segments are Invoke's ROCm default: measured on gfx1200 alone. Elsewhere they are a user override nothing is measured for, priced as MIOpen with the native allocator on that OS. Fitted on its RX 9060 XT (gfx1200, torch 2.13+rocm10) points." } ], "points": [ { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -135,19 +427,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 1142461301, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -155,19 +448,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 2576980378, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -175,19 +469,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 4540854174, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -195,19 +490,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 561566974, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -215,19 +511,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 1254130450, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -235,19 +532,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 2224793059, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -255,19 +553,20 @@ "width": 1280, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 7075958620, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, bf16, untiled, fused SDPA: the fused column of the max-versus-sum measurement (#9519). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, bf16, untiled, fused SDPA: the fused column of the max-versus-sum measurement (#9519). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -275,19 +574,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 1138753536, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -295,19 +595,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 2531262464, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -315,19 +616,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 4466933760, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -335,19 +637,20 @@ "width": 1328, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 7493124096, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -355,19 +658,20 @@ "width": 1536, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 10043260928, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -375,19 +679,20 @@ "width": 1328, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 3168796672, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -395,19 +700,21 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 561566974, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -415,19 +722,21 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 1933809025, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -435,19 +744,21 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 4372276707, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -455,19 +766,21 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 3787087413, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -475,19 +788,21 @@ "width": 1280, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 11141120000 + "score_matrix_bytes": 11141120000, + "dtype": "bf16" }, "measured_bytes": 6360846565, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -495,19 +810,21 @@ "width": 1536, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 23102226432 + "score_matrix_bytes": 23102226432, + "dtype": "bf16" }, "measured_bytes": 12847320924, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -515,19 +832,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 1810328715, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -535,19 +853,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 3973918491, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -555,19 +874,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 7063073718, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -575,20 +895,21 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 1847909679, "method": "reserved_peak", "device": "PRO W7900 (gfx1100)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals.", + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it.", "max_ratio": 1.15 }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -596,20 +917,21 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 4059817837, "method": "reserved_peak", "device": "PRO W7900 (gfx1100)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals.", + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it.", "max_ratio": 1.15 }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -617,20 +939,21 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 7197291446, "method": "reserved_peak", "device": "PRO W7900 (gfx1100)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals.", + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it.", "max_ratio": 1.15 }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "encode", @@ -638,19 +961,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 1408749273, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201) and PRO W7900 (gfx1100), identical to the byte", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "encode", @@ -658,19 +982,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 3170759606, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201) and PRO W7900 (gfx1100), identical to the byte", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "encode", @@ -678,19 +1003,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 5637144576, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201) and PRO W7900 (gfx1100), identical to the byte", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -698,19 +1024,20 @@ "width": 1024, "tile": null, "batch_size": 2, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 8541616210, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above." + "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -718,19 +1045,20 @@ "width": 1024, "tile": null, "batch_size": 3, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 12765716546, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above." + "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -738,32 +1066,5247 @@ "width": 2024, "tile": 512, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 585105408, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "#9519: reference-image encode at the 2024px reference cap with the forced 512px tile, CUDA, bf16, fused SDPA; flat at ~0.55 GB from 1024px up. Recorded in MiB." + "provenance": "#9519: reference-image encode at the 2024px reference cap with the forced 512px tile, CUDA, bf16, fused SDPA; flat at ~0.55 GB from 1024px up. Recorded in MiB. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", "height": 3072, "width": 3072, - "element_size": 2, "tile": null, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 20229295964, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "Commit 326bdf1bd0: the FLUX.1 VAE as a diffusers AutoencoderKL, bf16, fused SDPA, untiled 3072px encode: 18.84 GiB. Re-measured 18.57 GiB on torch 2.13+cu130, 2026-10-17, with scripts/calibrate_flux_vae_working_memory.py." + "provenance": "Commit 326bdf1bd0: the FLUX.1 VAE as a diffusers AutoencoderKL, bf16, fused SDPA, untiled 3072px encode: 18.84 GiB. Re-measured 18.57 GiB on torch 2.13+cu130, 2026-10-17, with scripts/calibrate_flux_vae_working_memory.py. Standalone script with no allocator configuration: the native allocator." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1488977920, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3334471680, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5817499648, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 9105833984, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1744830464, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4326424576, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7572815872, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 11741954048, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 891289600, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2132803584, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3709861888, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5689573376, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1226833920, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2661285888, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4622123008, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7157579776, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1140850688, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4538236928, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10158604288, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 2048, + "width": 2048, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 18029215744, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 562036736, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2225078272, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4989124608, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 2048, + "width": 2048, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8864661504, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 3072, + "width": 3072, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 19937624064, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 985661440, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 1614807040, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2495610880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3271557120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 825229312, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 1433403392, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2293235712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3040870400, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1570766848, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3355443200, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5997854720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 9418309632, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1095761920, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2312110080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3900702720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2154823680, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3984588800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6983516160, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10821304320, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1197473792, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2287992832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3546284032, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5538578432, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1570766848, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3355443200, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5997854720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 9418309632, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1095761920, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2312110080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3900702720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2154823680, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3984588800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6983516160, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10821304320, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1197473792, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2287992832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3548381184, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5538578432, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1717567488, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3376414720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6018826240, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 9374269440, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1095761920, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2312110080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3900702720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6081740800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1186988032, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2344615936, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3212836864, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4796186624, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 998244352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2153775104, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2526019584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3462397952, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2686451712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6155141120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10959716352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16733175808, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1533018112, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2495610880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4433379328, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6924795904, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2176843776, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5135925248, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10993270784, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16787701760, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2293235712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3206545408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5639241728, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 8812232704, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2684354560, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6155141120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10959716352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16733175808, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1533018112, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2495610880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4433379328, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6922698752, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2178940928, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5135925248, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10993270784, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16789798912, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2293235712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3206545408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5641338880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 8810135552, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2684354560, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6155141120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10959716352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16689135616, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1533018112, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2493513728, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4431282176, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6922698752, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1157627904, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2592079872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4552916992, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10173284352, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 576716800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1268776960, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2241855488, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5022679040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1675624448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4257218560, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7503609856, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 16770924544, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 847249408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1843396608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3261071360, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7308574720, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1157627904, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2592079872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4555014144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10175381504, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 576716800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1270874112, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2239758336, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5005901824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1675624448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4259315712, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7501512704, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 16770924544, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 845152256, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1843396608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3261071360, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7306477568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1159725056, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2592079872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4555014144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10057940992, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 578813952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1270874112, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2241855488, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5005901824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1675624448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4257218560, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7501512704, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 16791896064, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 847249408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1841299456, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3258974208, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7306477568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 822083584, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2063597568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3640655872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8069840896, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 486539264, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1056964608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1895825408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4244635648, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1442840576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3265265664, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5748293632, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 12996050944, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 681574400, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1522532352, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2694840320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 788529152, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2030043136, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3607101440, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8036286464, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 486539264, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1056964608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1895825408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4244635648, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1442840576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3265265664, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5714739200, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 12962496512, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 681574400, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1520435200, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2694840320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 788529152, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2030043136, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3607101440, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8069840896, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 486539264, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1073741824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1895825408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4244635648, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1409286144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3265265664, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5781848064, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 12962496512, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 681574400, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1520435200, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2694840320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 6117392384, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 2233466880, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 4947181568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 8690597888, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 19386073088, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 624951296, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 1379926016, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 2436890624, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 5454692352, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 1644167168, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 3800039424, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 6685720576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 14738784256, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 478150656, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 1048576000, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 1887436800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 4236247040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 3439329280, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 7717519360, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 13715374080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 1092616192, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 2267021312, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 3755999232, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16" + }, + "measured_bytes": 5874122752, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 4399824896, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 9909043200, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 17616076800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 2271215616, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 3206545408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 5695864832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16" + }, + "measured_bytes": 8902410240, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." } ] } diff --git a/invokeai/backend/working_memory/fingerprint.py b/invokeai/backend/working_memory/fingerprint.py new file mode 100644 index 00000000000..ccdde0a04c1 --- /dev/null +++ b/invokeai/backend/working_memory/fingerprint.py @@ -0,0 +1,173 @@ +"""What a measurement is keyed by: the class of device it ran on, and the software environment that decides how much +memory an operation takes there. + +A device class names a kind of card, not one card: identical cards share it, whatever their index or PCI slot, so +measurements taken on one apply to the other. An environment splits into hard facts, any change of which can change an +operation's memory (the operating system, torch and runtime versions, the convolution library, the allocator and its +configuration, the attention path, InvokeAI's memory settings), and soft facts recorded for diagnosis only (full +versions, the driver, the OS build). Both are produced deterministically here; which change makes a measurement stale +is the caller's decision. +""" + +import ctypes +import ctypes.util +import hashlib +import json +import os +import platform +import sys +from collections.abc import Mapping +from typing import Final, Literal, Optional + +import psutil +import torch +from pydantic import BaseModel, ConfigDict, computed_field + +from invokeai.backend.util import nvml +from invokeai.backend.util.attention import rocm_sdpa_chunks_math +from invokeai.backend.working_memory.counters import Backend, backend_for +from invokeai.version import __version__ + +_VENDORS: Final[Mapping[Backend, str]] = {"cuda": "nvidia", "hip": "amd", "mps": "apple", "xpu": "intel"} + +HARD_ENVIRONMENT_VARIABLES: Final = ( + "PYTORCH_ALLOC_CONF", + "PYTORCH_CUDA_ALLOC_CONF", + "PYTORCH_HIP_ALLOC_CONF", + "MIOPEN_FIND_MODE", + "PYTORCH_TUNABLEOP_ENABLED", + # The head width above which the ROCm SDPA guard sends attention to the chunked math kernel (`util/attention.py`). + "INVOKE_ROCM_FUSED_SDPA_MAX_HEAD_DIM", + # Whether AOTriton's fused attention runs on GPUs PyTorch marks experimental; without it attention takes the math + # kernel. What `rocm_aotriton_experimental` resolved to (`app/util/rocm_aotriton.py`), or a value set beforehand. + "TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL", + # Native or decomposed causal 3D convolutions on ROCm (`backend/wan/rocm_causal_conv3d.py`). + "INVOKEAI_ROCM_CONV3D", +) +"""Environment variables that change how much memory an operation takes: the allocator's configuration, MIOpen's +algorithm search, TunableOp's GEMM selection, and the ROCm attention and convolution kernel choices. Recorded as +``env.``, empty when unset.""" + + +def _short_hash(fields: Mapping[str, object]) -> str: + canonical = json.dumps(fields, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16] + + +class DeviceClass(BaseModel): + """A kind of compute device. Identical cards share one, whatever their index or PCI location.""" + + model_config = ConfigDict(frozen=True) + + vendor: str + name: str + total_mib: int + backend: Backend + arch: Optional[str] + """``gcnArchName`` on HIP (e.g. ``gfx1200``), ``sm_`` on CUDA, the reported architecture on XPU.""" + + @computed_field # type: ignore[prop-decorator] + @property + def id(self) -> str: + """A short stable hash of the other fields. Derived, never given.""" + return _short_hash(self.model_dump(exclude={"id"})) + + +def _mps_name() -> str: + """The Mac's chip, e.g. ``Apple M2 Max``; MPS names no device.""" + try: + libc = ctypes.CDLL(ctypes.util.find_library("c")) + size = ctypes.c_size_t(0) + if libc.sysctlbyname(b"machdep.cpu.brand_string", None, ctypes.byref(size), None, 0) == 0 and size.value: + buffer = ctypes.create_string_buffer(size.value) + if libc.sysctlbyname(b"machdep.cpu.brand_string", buffer, ctypes.byref(size), None, 0) == 0: + return buffer.value.decode("utf-8", errors="replace") + except Exception: + pass + return "Apple GPU" + + +def device_class_for(device: torch.device) -> DeviceClass: + """The class of `device`. Raises ValueError for a device that is not a GPU.""" + backend = backend_for(device) + if backend is None: + raise ValueError(f"{device} is not a GPU; only GPU devices have a device class.") + if backend in ("cuda", "hip"): + props = torch.cuda.get_device_properties(device) + name, total = torch.cuda.get_device_name(device), int(props.total_memory) + arch = str(props.gcnArchName) if backend == "hip" else f"sm_{props.major}{props.minor}" + elif backend == "xpu": + props = torch.xpu.get_device_properties(device) + name, total = str(props.name), int(props.total_memory) + architecture = getattr(props, "architecture", None) + arch = None if architecture is None else str(architecture) + else: + # Unified memory: the device's share is the machine's RAM. + name, total, arch = _mps_name(), int(psutil.virtual_memory().total), None + return DeviceClass(vendor=_VENDORS[backend], name=name, total_mib=total // 2**20, backend=backend, arch=arch) + + +class Environment(BaseModel): + """The software a measurement ran under: hard facts that change an operation's memory, soft facts that do not.""" + + model_config = ConfigDict(frozen=True) + + hard: dict[str, str] + soft: dict[str, Optional[str]] + + @computed_field # type: ignore[prop-decorator] + @property + def id(self) -> str: + """A short stable hash of the hard facts. Soft facts do not enter it.""" + return _short_hash(self.hard) + + +def _major_minor(version: str) -> str: + return ".".join(version.split("+", 1)[0].split(".")[:2]) + + +def _conv_library_major() -> Optional[str]: + """The major version of cuDNN, or of MIOpen on a HIP build, as loaded at runtime.""" + try: + if torch.backends.cudnn.version() is None: + return None + # (major, minor, patch) on either build; the packed integer `version()` returns is encoded per library. + return str(torch._C._cudnn.getRuntimeVersion()[0]) + except Exception: + return None + + +def os_family(platform_name: str) -> Literal["windows", "linux", "other"]: + """The operating system family of a ``sys.platform`` value, as a hard fact: the driver model differs (WDDM pages + an allocation that does not fit into shared memory instead of failing it), so the same card and torch build can + take different memory on Windows and on Linux.""" + if platform_name == "win32": + return "windows" + return "linux" if platform_name.startswith("linux") else "other" + + +def environment_for(device: torch.device, settings: Mapping[str, str | int | float | bool | None]) -> Environment: + """The environment an operation on `device` runs under. Every key of `settings` is a hard fact, as + ``setting.``: the caller passes the InvokeAI settings that change memory.""" + backend = backend_for(device) + hard: dict[str, str] = {"os": os_family(sys.platform), "torch": _major_minor(torch.__version__)} + if backend in ("cuda", "hip"): + runtime = torch.version.hip if backend == "hip" else torch.version.cuda + hard[backend] = _major_minor(runtime or "") + hard["miopen" if backend == "hip" else "cudnn"] = _conv_library_major() or "" + hard["allocator_backend"] = torch.cuda.get_allocator_backend() + hard["rocm_sdpa_guard"] = str(rocm_sdpa_chunks_math(device.type)).lower() + elif backend == "xpu": + hard["xpu"] = str(getattr(torch.version, "xpu", None) or "") + for name in HARD_ENVIRONMENT_VARIABLES: + hard[f"env.{name}"] = os.environ.get(name, "") + for key, value in settings.items(): + hard[f"setting.{key}"] = str(value) + + soft: dict[str, Optional[str]] = { + "torch_build": torch.__version__, + "driver": nvml.driver_version() if backend == "cuda" else None, + "os_build": platform.platform(), + "invokeai": __version__, + } + return Environment(hard=hard, soft=soft) diff --git a/invokeai/backend/working_memory/fit.py b/invokeai/backend/working_memory/fit.py new file mode 100644 index 00000000000..996743243af --- /dev/null +++ b/invokeai/backend/working_memory/fit.py @@ -0,0 +1,286 @@ +"""The shipped fit: the values under `data/` reproduced from their measured points. + +An estimator's fitted terms price a line per operation and dtype: `k_` bytes per unit of the line's size plus +`b_` bytes, the base a process pays whatever the size. Its `EstimatorSpec.line` says which line an operation's +features price on, their size on it, and whether a measurement of them can join its fit (`Line`). The AutoencoderKL +estimators price their convolutions on a line in the output's pixels (`convolution_line`); the Z-Image and FLUX.2 denoise +price their forwards on ones in attended tokens. Each carried pair is the upper envelope of the points it is +resolved for (see `upper_envelope`), plus `SHIPPED_MARGIN`. A point joins the fit of the entry its estimator resolves +to on its platform -- the lender's, where the estimator borrows -- when it is `in_fit` (not measured out of regime) and +its line admits it. + +A line is fitted on its points' demand as measured; no term is subtracted from it. So a line admits only measurements +whose demand is the line's alone. Where an estimate adds a term of its own on top of the line, a measurement of it +fits nothing: a tiled VAE operation (the tile term reuses the untiled pair), a VAE an in-place cast grew +(`upcast_weight_bytes`), and every denoise forward with an analytic term. The VAE mid-block score matrix is not such a +term: an estimate reserves the larger of it and the convolutions, which peak apart, and on the builds that materialize +it the convolutions stay the larger over the measured sizes (`vae_working_memory._peak_of_two_phases`); the CUDA points +that force a math kernel are out of fit. A point that bounds its estimate from below only +(`MeasuredPoint.holds_upper_bound`, a `probe_demand` window) is covered by the line and can raise it, but takes no part +in choosing it: the fresh-process points cap a line. + +`tests/backend/working_memory/test_shipped_coefficients.py` holds every carried value that is not a declared stand-in +to `fitted_values` of the shipped points, so a new point that moves a fit fails there with the value to ship. A +`legacy` entry's hand-set values are neither fitted nor held to a fit. + +A user's own calibration fits the same lines from its windows (`user_pair`, read by `user_fit.py`), with +`USER_MARGIN`, the same margin, for the estimators that opt in (`EstimatorSpec.user_calibrated`). +""" + +import math +from collections.abc import Callable, Iterable, Mapping, Sequence +from fractions import Fraction +from typing import TYPE_CHECKING, Final, NamedTuple, Optional + +from invokeai.backend.working_memory.coefficients import CoefficientEntry, MeasuredPoint, WorkingMemoryData, find_entry +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryTerm + +if TYPE_CHECKING: + from invokeai.backend.working_memory.spec import EstimatorSpec + +SHIPPED_MARGIN: Final = Fraction(1, 20) +"""What the shipped slope and intercept add over the envelope of their points: 5%, for the runs and cards the points +do not cover.""" + +USER_MARGIN: Final = SHIPPED_MARGIN +"""What a user calibration's slope and intercept add over the envelope of its points: the shipped margin. A user line +measures the card it prices, and the windows that would inflate it (fragmented, partly resident) never join it.""" + +MIN_AREA_RATIO: Final = 2 +"""How far apart the sizes a line is fitted on must lie, as a ratio of its sizes (pixel or token counts): closer points +pin the intercept no better than their noise.""" + +_MIB: Final = 2**20 + +EntryKey = tuple[str, int, tuple[tuple[str, str], ...]] +"""An entry's estimator id, version and platform keys: what a point is resolved by.""" + + +class Line(NamedTuple): + """Where an operation's features place it on the line its estimator's fitted terms price it on.""" + + suffix: str + """``_``, naming the line's coefficients ``k_`` and ``b_``.""" + size: int + """The operation's size on the line, in the unit its slope is per: output pixels, attended tokens.""" + fits: bool + """Whether a measurement of these features can join the line's fit: its demand is the line's alone.""" + label: str + """How a point of these features reads in its id (`bounds.point_id`), e.g. ``decode-bf16-1024x1024-tile512``.""" + + +LineFunction = Callable[[Mapping[str, FeatureValue]], Optional[Line]] +"""What an `EstimatorSpec` declares about its fitted line: the `Line` an operation's features place it on, or None for +features that name no line.""" + + +def off_line_reasons( + terms: Callable[[Mapping[str, FeatureValue], Mapping[str, float]], Iterable[WorkingMemoryTerm]], + features: Mapping[str, FeatureValue], + coefficients: Iterable[str], +) -> list[str]: + """What puts an operation off the line its fitted terms price: an attention mask, which narrows the kernels SDPA can + pick (``has_attention_mask``), and each analytic term of `terms` that is not zero, as ``=``. Empty for + an operation whose demand is the line's alone. The terms are priced with every coefficient at zero, so only what + the operation adds on top of the line remains; a term added to an estimator takes the operations it prices out of + the fit by itself. The estimators label their points with the reasons in this order.""" + unpriced = dict.fromkeys(coefficients, 0.0) + reasons = ["attention_mask"] if features.get("has_attention_mask") else [] + return reasons + [ + f"{term.name}={int(term.bytes)}" + for term in terms(features, unpriced) + if term.coefficient is None and term.bytes + ] + + +def convolution_line(features: Mapping[str, FeatureValue]) -> Optional[Line]: + """The AutoencoderKL convolution line ``_``, in output pixels: every VAE estimator's. + + An untiled single-sample operation of an AutoencoderKL whose VAE no in-place cast grew fits it. Tiled operations + reuse the untiled line, the cast's growth is a term of its own, and the lines are the AutoencoderKL convolutions': + the SD estimator also prices the tiny autoencoder (`vae_class` AutoencoderTiny) on them, a different network whose + windows say nothing about those lines. Estimators that name no `vae_class` price an AutoencoderKL only. + """ + operation, dtype = features.get("operation"), features.get("dtype") + height, width = features.get("height"), features.get("width") + if not isinstance(operation, str) or not isinstance(dtype, str): + return None + if not isinstance(height, int) or not isinstance(width, int): + return None + tile, batch = features.get("tile"), features.get("batch_size", 1) + fits = ( + tile is None + and batch == 1 + and not features.get("upcast_weight_bytes", 0) + and features.get("vae_class", "AutoencoderKL") == "AutoencoderKL" + ) + tiled = f"-tile{tile}" if tile is not None else "" + batched = f"-b{batch}" if batch != 1 else "" + return Line(f"{operation}_{dtype}", height * width, fits, f"{operation}-{dtype}-{height}x{width}{tiled}{batched}") + + +def entry_key(entry: CoefficientEntry | MeasuredPoint) -> EntryKey: + return (entry.estimator, entry.version, tuple(sorted(entry.platform.items()))) + + +class FitPoint(NamedTuple): + size: int + bytes: int + caps: bool + """Whether the point takes part in choosing the line (`MeasuredPoint.holds_upper_bound`); every point is covered.""" + + +def upper_envelope( + points: Sequence[tuple[int, int]], capping: Optional[Sequence[tuple[int, int]]] = None +) -> tuple[Fraction, Fraction]: + """The line ``y = k * x + b`` on or above every ``(x, y)`` point, with ``b >= 0``, that over-reserves least. + + The candidates are the edges of the upper convex hull of the points and the origin: each such line lies on or + above every point, and the origin keeps its intercept non-negative. Of those, the one whose largest ratio to the + highest reading of a size is smallest wins, the steeper one on a tie, since it is the more conservative past the + largest point. Only the highest reading counts: a lower one at a measured size is covered whatever the line, and + rating against it would tilt the line though no point lies above it. With `capping`, a subset of the points, only + its readings rate the edges: the others are covered but cap nothing. Without any, all points rate them. Raises + unless the points span two sizes at least `MIN_AREA_RATIO` apart. + """ + if not points or any(x <= 0 or y <= 0 for x, y in points): + raise ValueError(f"an envelope needs points with positive sizes and bytes, got {list(points)}") + sizes = sorted({x for x, _ in points}) + if len(sizes) < 2 or sizes[-1] < MIN_AREA_RATIO * sizes[0]: + raise ValueError( + f"an envelope needs two sizes at least {MIN_AREA_RATIO}x apart, got {sizes}; measure more sizes" + ) + + highest = {x: max(y for px, y in points if px == x) for x in sizes} + hull: list[tuple[int, int]] = [(0, 0)] + for point in highest.items(): + # Drop the last vertex while it is not above the chord from the one before it to this point. + while len(hull) >= 2: + (x0, y0), (x1, y1) = hull[-2], hull[-1] + if (x1 - x0) * (point[1] - y0) - (y1 - y0) * (point[0] - x0) < 0: + break + hull.pop() + hull.append(point) + + rated = {x: max(y for px, y in capping if px == x) for x in sorted({x for x, _ in capping})} if capping else highest + best: tuple[Fraction, Fraction, Fraction] | None = None + for (x0, y0), (x1, y1) in zip(hull, hull[1:], strict=False): + k = Fraction(y1 - y0, x1 - x0) + b = y0 - k * x0 + if k < 0: + continue + worst = max((k * x + b) / y for x, y in rated.items()) + if best is None or (worst, -k) < (best[0], -best[1]): + best = (worst, k, b) + assert best is not None, "the edge from the origin always rises" + return best[1], best[2] + + +def shipped_pair( + points: Sequence[tuple[int, int]], capping: Optional[Sequence[tuple[int, int]]] = None +) -> tuple[int, int]: + """The slope and intercept to ship for these points: the envelope plus `SHIPPED_MARGIN`, the slope rounded up to + a whole byte per unit of size and the intercept to a whole MiB.""" + k, b = upper_envelope(points, capping) + margin = 1 + SHIPPED_MARGIN + return math.ceil(k * margin), math.ceil(b * margin / _MIB) * _MIB + + +def user_pair(points: Sequence[tuple[int, int]]) -> tuple[int, int]: + """The slope and intercept a user calibration's points give a line: the envelope plus `USER_MARGIN`, the slope + rounded up to a whole byte per unit of size and the intercept to a whole MiB. + + Every user point is a `probe_demand` window, so no point caps the line and all of them rate its edges (the + fallback of `upper_envelope`). Raises like `upper_envelope` unless the points span two sizes `MIN_AREA_RATIO` + apart.""" + k, b = upper_envelope(points) + margin = 1 + USER_MARGIN + return math.ceil(k * margin), math.ceil(b * margin / _MIB) * _MIB + + +def carried_lines(entry: CoefficientEntry) -> set[str]: + """The suffixes an entry carries both halves of a line for.""" + names = set(entry.values or {}) + return {name[2:] for name in names if name.startswith("k_") and f"b_{name[2:]}" in names} + + +def fit_points( + data: Iterable[WorkingMemoryData], specs: Mapping[str, "EstimatorSpec"] +) -> Mapping[tuple[EntryKey, str], list[FitPoint]]: + """Per carried entry and line, the points that line is fitted on. + + A point belongs to the entry its own estimator's entry resolves to on its platform, one `values_from` hop, and to + the line its estimator's `EstimatorSpec.line` places its features on, where that admits them. `specs` holds the + spec of every estimator a point names. Lines no point measures are absent. + """ + files = list(data) + entries = [entry for file in files for entry in file.coefficients] + groups: dict[tuple[EntryKey, str], list[FitPoint]] = {} + for point in (point for file in files for point in file.points): + if not point.in_fit: + continue + line = specs[point.estimator].line(point.features) + if line is None or not line.fits: + continue + owner = carrier_of(entries, point) + if line.suffix in carried_lines(owner): + fitted_on = groups.setdefault((entry_key(owner), line.suffix), []) + fitted_on.append(FitPoint(line.size, point.measured_bytes, point.holds_upper_bound)) + return groups + + +def carrier_of(entries: Sequence[CoefficientEntry], point: MeasuredPoint) -> CoefficientEntry: + """The entry that carries the values `point` is evaluated with: its estimator's entry on its platform, or the one + that entry borrows from. A point names the OS only where it measured an OS-specific entry's platform.""" + where = f"{point.estimator} point on {point.platform}" + entry = find_entry(entries, point.estimator, point.version, point.platform, where) + if entry.values_from is None: + return entry + return find_entry(entries, entry.values_from.estimator, entry.values_from.version, point.platform, where) + + +def entry_fit( + entry: CoefficientEntry, groups: Mapping[tuple[EntryKey, str], Sequence[FitPoint]] +) -> tuple[dict[str, int], dict[str, str]]: + """What `shipped_pair` gives each line `entry` carries and does not declare a stand-in, rated on the points that cap + it, from `groups` (`fit_points`); and, per such line its points cannot fit, why.""" + key = entry_key(entry) + values: dict[str, int] = {} + problems: dict[str, str] = {} + for suffix in sorted(carried_lines(entry)): + if {f"k_{suffix}", f"b_{suffix}"} <= set(entry.stand_ins): + continue + line = groups.get((key, suffix), []) + if not line: + problems[suffix] = "no in-fit point measures it" + continue + try: + k, b = shipped_pair([(p.size, p.bytes) for p in line], [(p.size, p.bytes) for p in line if p.caps]) + except ValueError as e: + problems[suffix] = str(e) + continue + values |= {f"k_{suffix}": k, f"b_{suffix}": b} + return values, problems + + +def fitted_values( + data: Iterable[WorkingMemoryData], specs: Mapping[str, "EstimatorSpec"] +) -> Mapping[EntryKey, Mapping[str, int]]: + """Per entry that carries values and is not `legacy`, what `shipped_pair` gives each line it does not declare a + stand-in (`entry_fit`). + + Raises, naming the entry and line, where the points cannot fit that line. + """ + files = list(data) + groups = fit_points(files, specs) + fitted: dict[EntryKey, dict[str, int]] = {} + for entry in (entry for file in files for entry in file.coefficients): + if entry.values is None or entry.legacy: + continue + values, problems = entry_fit(entry, groups) + if problems: + suffix, problem = next(iter(problems.items())) + raise ValueError(f"{entry.estimator} on {entry.platform}, {suffix}: {problem}") + if values: + fitted[entry_key(entry)] = values + return fitted diff --git a/invokeai/backend/working_memory/fragmentation.py b/invokeai/backend/working_memory/fragmentation.py new file mode 100644 index 00000000000..c45d4c5ec38 --- /dev/null +++ b/invokeai/backend/working_memory/fragmentation.py @@ -0,0 +1,107 @@ +"""When a measured window's demand is the allocator's fragmentation rather than the operation's. + +A long-running server's caching allocator grows its reserve around the same tensors: reserved bytes climb while the +allocated bytes, what the operation really holds, stay put. A window taken then measures the allocator's history, and a +line fitted on it over-reserves every later operation. One rule, `fragmented`, read by +`scripts/working_memory/build_shipped.py`, which flags such points for review, and by the user fit +(`app/services/working_memory_calibration/`), which leaves them out. It judges measurements: the fit-eligible windows +of one operation at one size from one attempt (a calibration attempt, or one run without one), in the order they ran. +Windows of one attempt at one size ran back to back on one allocator state; another attempt's ran on another, before +or after offloads and other work, so each attempt is judged on its own and a recalibration can bring a size back. + +- `grew_around_flat_allocation`: across a measurement's windows, reserved grew by `FRAGMENTATION_GROWTH` from the + first window to the largest while allocated stayed within `ALLOCATED_FLAT`. +- `ratio_outlier`: a measurement's largest window has a reserved-to-allocated ratio `RATIO_OUTLIER` above the median of + the same line's smaller measurements that are not flagged, from any attempt: the ratio is the operation's on the + card, which no attempt owns. This catches a size whose every window ran fragmented, so that it shows no growth of its + own; a single window, too. Measured on an RX 9060 XT (fp32 SDXL decode): 1216px measured 1.47 against 1.10 at 576px + and 832px, its one fit-eligible window 1.33 times their median, while the fp16 sizes of the same run stayed within + 1.06 of theirs. +""" + +from collections.abc import Hashable, Sequence +from dataclasses import dataclass +from statistics import median +from typing import Final, Optional + +FRAGMENTATION_GROWTH: Final = 1.10 +"""Reserved growth from a measurement's first window to its largest that, with flat allocated bytes, flags +fragmentation.""" + +ALLOCATED_FLAT: Final = 0.02 +"""How far allocated bytes may spread across a measurement's windows and still count as flat.""" + +RATIO_OUTLIER: Final = 1.2 +"""How far a size's reserved-to-allocated ratio may exceed the median of the same line's smaller sizes before it +counts as fragmentation.""" + + +@dataclass(frozen=True) +class Measurement: + """The fit-eligible windows of one operation at one size from one attempt, in the order they ran.""" + + line: Hashable + """What its sizes are compared within: one line (`fit.Line.suffix`) of one estimator version on one card and + platform or environment.""" + size: int + """The operation's size on that line (`fit.Line.size`).""" + demand: Sequence[int] + reserved: Sequence[Optional[int]] + """Each window's reserved delta.""" + allocated: Sequence[Optional[int]] + reference: bool = True + """Whether larger sizes may be compared with it; a measurement still being taken is judged, but sets no standard.""" + + +def fragmented(measurements: Sequence[Measurement]) -> list[bool]: + """Per measurement, whether its windows measured fragmentation: `grew_around_flat_allocation`, or a + `ratio_outlier` of its largest window against the smaller measurements of its line that are not flagged and may + be a `reference`.""" + flags = [ + grew_around_flat_allocation(measurement.demand, measurement.reserved, measurement.allocated) + for measurement in measurements + ] + lines: dict[Hashable, list[int]] = {} + for index, measurement in enumerate(measurements): + if not flags[index]: + lines.setdefault(measurement.line, []).append(index) + for members in lines.values(): + kept: list[tuple[int, float]] = [] + for index in sorted(members, key=lambda index: measurements[index].size): + measurement = measurements[index] + largest = list(measurement.demand).index(max(measurement.demand)) + ratio = reserved_ratio(measurement.reserved[largest], measurement.allocated[largest]) + if ratio_outlier(ratio, [kept_ratio for size, kept_ratio in kept if size < measurement.size]): + flags[index] = True + elif ratio is not None and measurement.reference: + kept.append((measurement.size, ratio)) + return flags + + +def grew_around_flat_allocation( + demand: Sequence[int], reserved: Sequence[Optional[int]], allocated: Sequence[Optional[int]] +) -> bool: + """Whether the windows of one measurement, in the order they ran, show reserved growing by `FRAGMENTATION_GROWTH` + from the first window to the one of the largest demand while allocated stayed flat. False when that largest + window is the first, or a counter is missing.""" + largest = list(demand).index(max(demand)) + first, peak = reserved[0], reserved[largest] + if largest == 0 or first is None or peak is None or None in allocated: + return False + flat = [value for value in allocated if value is not None] + return peak >= FRAGMENTATION_GROWTH * first and max(flat) <= (1 + ALLOCATED_FLAT) * min(flat) + + +def reserved_ratio(reserved: Optional[int], allocated: Optional[int]) -> Optional[float]: + """Reserved over allocated bytes of one window; None where either is missing or allocated is not positive.""" + if reserved is None or allocated is None or allocated <= 0: + return None + return reserved / allocated + + +def ratio_outlier(ratio: Optional[float], smaller: Sequence[float]) -> bool: + """Whether a size's `ratio` is `RATIO_OUTLIER` times the median ratio of the same line's `smaller` sizes; False + without a ratio or a smaller size to compare with.""" + if ratio is None or not smaller: + return False + return ratio >= RATIO_OUTLIER * median(smaller) diff --git a/invokeai/backend/working_memory/measurement.py b/invokeai/backend/working_memory/measurement.py new file mode 100644 index 00000000000..dd1379db45e --- /dev/null +++ b/invokeai/backend/working_memory/measurement.py @@ -0,0 +1,218 @@ +"""An operation's working memory, from this process's counter readings at the two ends of a window. + +A window spans one operation on a compute device with its weights and inputs in place. It opens with a synchronize, a +release of the allocator's cached blocks, the baseline reading and a reset of the peaks: with nothing cached, whatever +the operation takes shows as growth. It closes with a synchronize and a reading. Per counter (the driver's figure, +torch's reserved and allocated bytes; see `counters`) the window takes its peak's growth over the baseline, and the +operation's *demand* is the largest growth of the counters that could be read: no counter alone covers every platform +(`counters` has the measurements). + +A window is invalid when its readings show that + +- an allocation failed (``oom``), or only succeeded after the allocator released its cache and retried + (``alloc_retry``). A retry squeezes the peak: an untiled 3072px FLUX VAE decode on an RTX 4090 read 22.5 GiB at the + card's limit, ~1280 bytes per pixel instead of ~2150; +- Windows still keeps more than `SPILL_TOLERANCE_BYTES` of the process in shared memory `SPILL_SETTLE_S` after the close + (``spill``). Right after a decode the counter spikes transiently, 589 MiB, and is back at 9 MiB at the next read; +- a device-wide counter (XPU) grew by more than this process's own reserved bytes: another process's load + (``foreign_device_load``); +- no counter could be read (``counters_unavailable``). + +Two kinds of window take these definitions. The in-server probe's (`probe`) follow an operation through co-locks, nested +windows and releases, in segments, and have reasons of their own. A *fresh window* (`measure`) is one operation alone: +the calibration scripts measure their points that way, each in a fresh process with the server's runtime +(`invokeai.app.run_app.apply_server_runtime`). Such a point is a cold and a warm window of the same operation +(`measure_cold_and_warm`), and its bytes are the larger of the two (`measured_bytes_and_method`). +""" + +import time +from collections.abc import Callable, Iterable, Sequence +from dataclasses import dataclass +from typing import Final, Literal, Optional + +import torch + +from invokeai.backend.util.devices import TorchDevice +from invokeai.backend.working_memory import counters +from invokeai.backend.working_memory.counters import CounterReading, Method + +MIB = 2**20 + +SPILL_TOLERANCE_BYTES: Final = 256 * MIB +"""Shared-memory bytes above the window's baseline that count as spill once they persist.""" +SPILL_SETTLE_S: float = 0.5 +"""How long a close waits before re-reading a raised shared-memory counter. Module-level so tests can shorten it.""" +FOREIGN_DEVICE_TOLERANCE_BYTES: Final = 256 * MIB +"""Growth of a device-wide counter (XPU) beyond this process's own reserved growth that marks another process's load.""" + +CounterName = Literal["driver", "reserved", "allocated"] +COUNTERS: Final[tuple[CounterName, ...]] = ("driver", "reserved", "allocated") + +ReadingReason = Literal["oom", "alloc_retry", "spill", "foreign_device_load", "counters_unavailable"] +"""Why a window's readings void it. The probe has reasons of its own on top (`probe.Reason`).""" + + +def counter_value(reading: CounterReading, name: CounterName) -> Optional[int]: + """`name`'s value in `reading`, as a baseline takes it.""" + return reading.driver if name == "driver" else reading.reserved if name == "reserved" else reading.allocated + + +def counter_peak(reading: CounterReading, name: CounterName) -> Optional[int]: + """`name`'s peak since the last reset in `reading`. The driver keeps none outside a sampler (MPS): its value then.""" + if name == "driver": + return reading.peak_driver if reading.peak_driver is not None else reading.driver + return reading.peak_reserved if name == "reserved" else reading.peak_allocated + + +def demand(deltas: Iterable[Optional[int]]) -> Optional[int]: + """The demand of an operation whose counters grew by `deltas`: the largest that could be read, None if none could.""" + readable = [delta for delta in deltas if delta is not None] + return max(readable) if readable else None + + +def window_method(first: CounterReading, driver_readable: bool) -> Method: + """Where a window's driver figure came from: the method of its baseline `first`, unless a driver figure read there + dropped out later (`driver_readable` False), which leaves the window to torch's counters.""" + return "torch-only" if first.driver is not None and not driver_readable else first.method + + +def _rose(before: Optional[int], after: Optional[int]) -> bool: + return before is not None and after is not None and after > before + + +def reading_reasons(first: CounterReading, final: CounterReading) -> list[ReadingReason]: + """Why the readings at a window's open (`first`) and close (`final`) void it, spill and unreadable counters aside.""" + reasons: list[ReadingReason] = [] + if _rose(first.num_ooms, final.num_ooms): + reasons.append("oom") + if _rose(first.num_alloc_retries, final.num_alloc_retries): + reasons.append("alloc_retry") + if ( + first.device_used is not None + and final.device_used is not None + and first.reserved is not None + and final.reserved is not None + and (final.device_used - first.device_used) - (final.reserved - first.reserved) > FOREIGN_DEVICE_TOLERANCE_BYTES + ): + reasons.append("foreign_device_load") + return reasons + + +def spilled(first: CounterReading, final: CounterReading, read: Callable[[], CounterReading]) -> bool: + """Whether the process stays paged into shared memory after a window: `final` holds more than + `SPILL_TOLERANCE_BYTES` above `first`, and so does a `read` `SPILL_SETTLE_S` later.""" + baseline = first.paged + if baseline is None or final.paged is None or final.paged - baseline <= SPILL_TOLERANCE_BYTES: + return False + time.sleep(SPILL_SETTLE_S) + settled = read().paged + return settled is None or settled - baseline > SPILL_TOLERANCE_BYTES + + +@dataclass(frozen=True) +class FreshWindow: + """One operation measured as a window of its own: per counter its peak's growth over the baseline, None where it + could not be read.""" + + driver: Optional[int] + reserved: Optional[int] + allocated: Optional[int] + paged: Optional[int] + """How many more of the process's bytes Windows kept in shared memory at the close than at the baseline. A record: + `spilled` judges spill after the settle time.""" + demand: Optional[int] + counter_method: Method + """Where the driver figure came from (`window_method`).""" + reasons: tuple[ReadingReason, ...] + """Why the window is invalid; none for a valid one.""" + + +def fresh_window(first: CounterReading, final: CounterReading, *, oom: bool, spill: bool) -> FreshWindow: + """The fresh window of an operation between `first`, read after the release before it, and `final`, read after it. + + `oom` says the operation ran out of memory, `spill` that the process stayed paged (`spilled`). + """ + deltas: dict[CounterName, Optional[int]] = {} + for name in COUNTERS: + base, top = counter_value(first, name), counter_peak(final, name) + deltas[name] = None if base is None or top is None else top - base + reasons: list[ReadingReason] = ["oom"] if oom else [] + reasons += [reason for reason in reading_reasons(first, final) if reason not in reasons] + if spill: + reasons.append("spill") + operation_demand = demand(deltas.values()) + if operation_demand is None: + reasons.append("counters_unavailable") + return FreshWindow( + driver=deltas["driver"], + reserved=deltas["reserved"], + allocated=deltas["allocated"], + paged=None if first.paged is None or final.paged is None else final.paged - first.paged, + demand=operation_demand, + counter_method=window_method(first, driver_readable=deltas["driver"] is not None), + reasons=tuple(reasons), + ) + + +def measure(device: torch.device, operation: Callable[[], object]) -> FreshWindow: + """Run `operation` on `device` as a fresh window: synchronize, release the allocator's cached blocks, read the + baseline and reset the peaks; run it; synchronize and read. + + An operation that runs out of memory gives an invalid window (``oom``); its other errors propagate, and so does a + synchronization or a peak reset that fails (`counters`): no reading after it could be trusted, so there is no + window. Its output is released after the closing reading. + """ + device_counters = counters.DeviceCounters(device) + try: + device_counters.synchronize() + TorchDevice.empty_cache(force=True) + first = device_counters.read() + device_counters.reset_peaks() + output, oom = None, False + try: + output = operation() + except RuntimeError as e: # torch.OutOfMemoryError among them + if "out of memory" not in str(e).lower(): + raise + oom = True + device_counters.synchronize() + final = device_counters.read() + spill = spilled(first, final, device_counters.read) + del output + return fresh_window(first, final, oom=oom, spill=spill) + finally: + device_counters.close() + + +FreshPointMethod = Literal["demand_peak", "reserved_peak"] +"""The `MeasuredPoint.method` of a fresh-process point.""" + + +def measure_cold_and_warm(device: torch.device, operation: Callable[[], object]) -> tuple[FreshWindow, ...]: + """A fresh-process point's windows: `operation` measured twice (`measure`), cold and then warm. + + The first run after a load is a real peak in a server (kernel modules, library workspaces), and so is a later one, + which can demand more: a warm Z-Image forward under `cudaMallocAsync` read 800 MiB against its cold run's 720. A + cold run that ran out of memory ends the point, whose warm run would measure what the failure left behind. + """ + cold = measure(device, operation) + if "oom" in cold.reasons: + return (cold,) + return (cold, measure(device, operation)) + + +def point_reasons(windows: Sequence[FreshWindow]) -> list[ReadingReason]: + """Why a fresh-process point is invalid: its windows' reasons, each once, in order. Empty for a valid point.""" + return list(dict.fromkeys(reason for window in windows for reason in window.reasons)) + + +def measured_bytes_and_method(windows: Sequence[FreshWindow]) -> tuple[int, FreshPointMethod]: + """A valid fresh-process point's bytes and `MeasuredPoint.method`, from its windows (`measure_cold_and_warm`). + + `demand_peak` where every window read the driver's figure: the largest demand. Otherwise `reserved_peak`, the + largest growth of torch's counters alone, so that the method names what the bytes were read from. + """ + if all(window.driver is not None for window in windows): + return max(window.demand for window in windows if window.demand is not None), "demand_peak" + torch_counters = [delta for window in windows for delta in (window.reserved, window.allocated) if delta is not None] + return max(torch_counters), "reserved_peak" diff --git a/invokeai/backend/working_memory/platform.py b/invokeai/backend/working_memory/platform.py index 5dfc2a00997..6cab1793e02 100644 --- a/invokeai/backend/working_memory/platform.py +++ b/invokeai/backend/working_memory/platform.py @@ -1,45 +1,134 @@ +import os +import sys from collections.abc import Mapping -from dataclasses import dataclass -from types import MappingProxyType -from typing import Final, Literal +from dataclasses import astuple, dataclass, replace +from typing import Final, Literal, Optional, cast, get_args import torch +from invokeai.app.util.torch_cuda_allocator import ALLOCATOR_CONF_ENV_VARS, expandable_segments_enabled +from invokeai.backend.working_memory.counters import Backend, backend_for + ConvBackend = Literal["cudnn", "miopen", "mps", "xpu", "cpu"] +Allocator = Literal["native", "cudaMallocAsync", "expandable"] +"""torch's caching allocator: the native one, the native one in expandable-segments mode, or the driver's +``cudaMallocAsync`` pool (``hipMallocAsync`` on a HIP build, which torch reports under the same name).""" +OperatingSystem = Literal["windows", "linux", "other"] @dataclass(frozen=True) class Platform: - """The traits of the machine an operation runs on, as coefficients are keyed. Each field is a platform key an - `EstimatorSpec` can name, and holds the true value: what was measured, and what was not, is `UNMEASURED_PRICED_AS`'s - business.""" + """The traits of the machine an operation runs on, as coefficients are keyed. Each field holds the true value: + what was measured, and what was not, is `priced_as`'s business. + + `conv_backend` and `allocator` are the keys an `EstimatorSpec` names. `os` refines them only where an entry says + so: an entry whose platform names an `os` applies there and takes precedence over the entry that does not, so the + operating system enters the pricing of exactly the platforms measured on one OS or measured to differ by it, and + of no other. Both are MIOpen: its native allocator demands up to 1.6x more on Windows than on Linux, and expandable + segments, Invoke's Windows default, are measured only on Windows. + """ conv_backend: ConvBackend + allocator: Allocator + os: OperatingSystem -UNMEASURED_PRICED_AS: Final[Mapping[str, Mapping[str, str]]] = MappingProxyType( +MEASURED: Final = frozenset( { - # MPS, XPU and CPU convolutions are unmeasured and take the cuDNN coefficients. On MPS that pairs with a - # score-matrix term the SDPA probe always charges there, so the total is not thin. - "conv_backend": MappingProxyType({"mps": "cudnn", "xpu": "cudnn", "cpu": "cudnn"}), + ("cudnn", "native", None), + ("cudnn", "cudaMallocAsync", None), + ("miopen", "native", None), + ("miopen", "expandable", "windows"), } ) -"""Per platform key, the values no coefficients are measured for, and the measured value each is priced as. The one -place that borrowing is declared: the resolver applies it and names it in an estimate's `source`, and shipped data -carries entries for measured values only.""" +"""The (convolution backend, allocator, OS) combinations coefficients are measured on, None for every OS. Shipped data +carries entries for these only.""" + + +def is_measured(platform: Platform) -> bool: + return {(platform.conv_backend, platform.allocator, None), astuple(platform)} & MEASURED != set() + + +def priced_as(platform: Platform) -> Platform: + """The measured platform an operation on `platform` is priced as: `platform` itself where it is measured. The OS + stays: it only selects among the entries of the measured pair. + + The one place that borrowing is declared; the resolver applies it and names it in an estimate's `source`. + + - A convolution backend nothing is measured on (MPS, XPU, CPU) takes cuDNN under the native allocator. On MPS that + pairs with a score-matrix term the SDPA probe always charges there, so the total is not thin. + - An allocator not measured with its backend on this OS takes that backend's native allocator: cuDNN in + expandable-segments mode, MIOpen under ``hipMallocAsync``, and MIOpen with expandable segments off Windows (a + user override there; Linux defaults to native). On cuDNN, native is the larger of the two measured allocators + (``cudaMallocAsync`` needs 25-35% less); on MIOpen, native is the larger too, on Windows by up to 1.6x. + """ + if is_measured(platform): + return platform + if platform.conv_backend in ("mps", "xpu", "cpu"): + return replace(platform, conv_backend="cudnn", allocator="native") + return replace(platform, allocator="native") + + +_CONV_BACKENDS: Final[Mapping[Optional[Backend], ConvBackend]] = { + "cuda": "cudnn", + "hip": "miopen", + "mps": "mps", + "xpu": "xpu", + None: "cpu", +} +"""The convolution backend of each GPU backend. A HIP build reports its GPUs as ``cuda`` devices, so the torch build -- +not the device string -- separates MIOpen from cuDNN, and a `cpu_only` model on a ROCm build is not on MIOpen. Any +device type other than CUDA, MPS and XPU runs torch's CPU kernels, or none at all, and is priced as the CPU.""" + + +def _allocator(backend: Optional[Backend], allocator_backend: Optional[str], environ: Mapping[str, str]) -> Allocator: + """The caching allocator of a `backend` device under torch's allocator backend and these allocator variables: + `native` for anything but a CUDA/HIP GPU. The driver's pool ignores expandable segments, so torch's backend + decides first; expandable segments are a mode of the native allocator.""" + if backend not in ("cuda", "hip"): + return "native" + if allocator_backend == "cudaMallocAsync": + return "cudaMallocAsync" + return "expandable" if expandable_segments_enabled(environ) else "native" + + +def _allocator_for(device: torch.device) -> Allocator: + """The caching allocator an operation on `device` allocates through. + + The backend is fixed before torch initializes CUDA, so torch's answer holds for the process. The variables are read + from the environment the model cache reads them from. A ``cuda`` device on a build without one (a test pinning the + device) has no backend to ask. + """ + backend = backend_for(device) + allocator_backend = ( + torch.cuda.get_allocator_backend() if backend in ("cuda", "hip") and torch.cuda.is_available() else None + ) + return _allocator(backend, allocator_backend, os.environ) + + +def _host_os() -> OperatingSystem: + """The operating system this process runs on, as platforms name it.""" + if sys.platform == "win32": + return "windows" + return "linux" if sys.platform.startswith("linux") else "other" def platform_for(device: torch.device) -> Platform: - """The platform an operation on `device` runs on. + """The platform an operation on `device` runs on.""" + return Platform(_CONV_BACKENDS[backend_for(device)], _allocator_for(device), _host_os()) + - A HIP build reports ``device.type == "cuda"``, so the torch build -- not the device string -- is what separates - MIOpen from cuDNN, and a `cpu_only` model on a ROCm build is not on MIOpen. Any device type other than CUDA, MPS and - XPU runs torch's CPU kernels, or none at all, and is priced as the CPU. +def platform_for_environment(backend: Backend, hard: Mapping[str, str]) -> Platform: + """The platform `platform_for` gives a `backend` device in the environment whose hard facts are `hard` + (`fingerprint.environment_for`), as a calibration store records them: torch's allocator backend, the allocator + variables and the OS. Raises ValueError when they lack a fact the platform needs. """ - if device.type == "cuda": - return Platform(conv_backend="miopen" if torch.version.hip is not None else "cudnn") - if device.type == "mps": - return Platform(conv_backend="mps") - if device.type == "xpu": - return Platform(conv_backend="xpu") - return Platform(conv_backend="cpu") + os_name = hard.get("os") + if os_name not in get_args(OperatingSystem): + raise ValueError(f"The environment records no operating system the platform knows: os={os_name!r}.") + allocator_backend = hard.get("allocator_backend") + if backend in ("cuda", "hip") and allocator_backend is None: + raise ValueError(f"The environment of a {backend} device records no allocator_backend.") + environ = {var: hard.get(f"env.{var}", "") for var in ALLOCATOR_CONF_ENV_VARS} + allocator = _allocator(backend, allocator_backend, environ) + return Platform(_CONV_BACKENDS[backend], allocator, cast(OperatingSystem, os_name)) diff --git a/invokeai/backend/working_memory/probe.py b/invokeai/backend/working_memory/probe.py new file mode 100644 index 00000000000..e9ef6b72df8 --- /dev/null +++ b/invokeai/backend/working_memory/probe.py @@ -0,0 +1,728 @@ +"""The working-memory probe: what an operation really took on its device, recorded next to what was estimated for it. + +Active only inside a `calibration_scope`, which the session runner opens around each node of a calibration queue item. +There, every `LoadedModel.model_on_device(working_memory=...)` on a GPU is a *window*: the model is locked with +calibration headroom (`calibration_reservation`, so that a too-low estimate cannot spill and could never be corrected), +and from the moment the lock has settled until just before the unlock, the probe measures the device. The window's +`MeasuredWindow` lands in the scope's `windows`; nothing is written anywhere while a window is open. + +What a window measures, per counter (driver, torch reserved, torch allocated; see `counters`): + +1. Open: synchronize, release the allocator's cached blocks, read the baseline, reset the peaks. Tensors allocated + before the lock are part of the baseline: in use, as they are in the model cache's own budget. +2. A *co-lock* -- another model locked on the same thread and device while the window is open (a ControlNet, an audio + VAE, the vocoder) -- moves weights, not working memory. The segment ends before its lock (R0, read after a release) + and a new one starts after it settled (R2, after a release); R2 - R0 joins the window's offset, negative when the + lock moved more off the device than on. +3. A *nested window* (an estimated `model_on_device` inside an open one, such as a VAE encode inside a transformer's + window): its lock is a co-lock for the parent, the parent is frozen while it is open, and it measures alone. When + it closes, the parent releases the child's cached blocks, resets the peaks and goes on; the child's lasting outputs + stay in the parent's figures. +4. A `TorchDevice.empty_cache()` inside the window ends a segment: the driver is read before the release. A direct + `torch.cuda.empty_cache()` (a library's, or cuDNN's and MIOpen's after an algorithm search) and an offload on the + cache's background thread do not, so the driver figure can miss what they released; torch's peaks still bound it. +5. Close: synchronize and read the final segment. Per counter, the delta is the largest segment peak less the + baseline and that segment's offset; `demand_bytes` is the largest delta of the counters that could be read. + +The baseline, the demand, the counters' method and the reasons the readings give are `measurement`'s, which the +calibration scripts' fresh windows share. A window is invalid (`valid=False`, with its reasons) for any reason its +readings give (a failed or retried allocation, spill, another process's load on an XPU, no readable counter), and when +another thread locked a model on the same device's cache while it was open, held one when it opened or was still +locking one then (Expand Prompt and Image to Prompt run next to sessions; see `locking`), when the operation raised or +was canceled, or its node failed or was canceled after the window closed (`ProbeScope.invalidate`), when the Krea-2 +SDPA benchmark (which resets the peaks itself) is active, and when a probe step failed (`probe_error`: a reading, a +release, a peak reset or a synchronization, at the window's open as much as later; a failed lock hook also voids the +scope's later windows, since its lock bookkeeping is suspect). A model less than fully resident at open keeps the +window valid but not `fit_eligible`: streamed weights are working memory the estimate does not price. + +Without a scope, the probe costs a context-variable read per estimated `model_on_device`, an entry in `locking` and a +read of `armed` per lock, a read of `armed` per unlock, and a truth test of `devices.EMPTY_CACHE_OBSERVERS` per +`empty_cache`. +""" + +import contextvars +import os +import threading +import time +from collections.abc import Iterator +from contextlib import contextmanager +from typing import TYPE_CHECKING, Final, Literal, Optional + +from pydantic import BaseModel, ConfigDict + +from invokeai.app.services.session_processor.session_processor_common import CanceledException +from invokeai.backend.util import devices +from invokeai.backend.util.devices import TorchDevice +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.working_memory import counters +from invokeai.backend.working_memory.counters import ( + VERIFIED_METHODS, + CounterReading, + DeviceCounters, + Method, + backend_for, +) +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm +from invokeai.backend.working_memory.measurement import ( + COUNTERS, + CounterName, + counter_peak, + counter_value, + demand, + reading_reasons, + spilled, + window_method, +) + +if TYPE_CHECKING: + from invokeai.backend.model_manager.load.model_cache.cache_record import CacheRecord + +GIB = 2**30 +MIB = 2**20 + +CALIBRATION_HEADROOM_MIN_BYTES: Final = 1 * GIB +CALIBRATION_HEADROOM_FRACTION: Final = 0.5 + +Reason = Literal[ + "foreign_lock", + "foreign_device_load", + "oom", + "alloc_retry", + "spill", + "exception", + "canceled", + "krea2_sdpa_benchmark", + "counters_unavailable", + "probe_error", + "partial_residency", +] +"""Why a window is invalid, or -- ``partial_residency`` alone -- valid but not fit-eligible.""" + +_NOT_INVALIDATING: Final[frozenset[Reason]] = frozenset({"partial_residency"}) + + +def _validity(reasons: list[Reason]) -> tuple[bool, bool]: + """``(valid, fit_eligible)`` of a window with `reasons`.""" + valid = all(reason in _NOT_INVALIDATING for reason in reasons) + return valid, valid and "partial_residency" not in reasons + + +class _Record(BaseModel): + model_config = ConfigDict(frozen=True, extra="forbid") + + +class CounterDelta(_Record): + """One counter over a window, in bytes.""" + + baseline: int + peak: int + """The largest value any segment reached, offsets not applied.""" + offset: int + """What co-locks moved onto the device by the end of the window; negative when they moved more off.""" + delta: int + """The largest of each segment's peak less the baseline and that segment's offset: the operation's own share.""" + + +class MeasuredWindow(_Record): + """One measured `model_on_device` window: who ran it, what was estimated, and what the device saw.""" + + run_id: str + queue_item_id: int + node_id: str + node_type: str + model_key: str + device: str + + estimator: str + version: int + features: dict[str, FeatureValue] + terms: tuple[WorkingMemoryTerm, ...] + reservation_bytes: int + source: str + reservation_applied_bytes: int + """What the lock reserved: the estimate plus calibration headroom.""" + + driver: Optional[CounterDelta] + reserved: Optional[CounterDelta] + allocated: Optional[CounterDelta] + demand_bytes: Optional[int] + """The largest delta of the counters that could be read.""" + method: Method + """Where the driver figures came from; ``torch-only`` when there were none for the whole window.""" + verified: bool + """Whether that method was verified on this backend on real hardware (`counters.VERIFIED_METHODS`).""" + + resident_fraction: float + """The share of the window's model's weights on the device when the window opened.""" + co_locked_models: list[str] + """The other models this thread held on the device while the window was open.""" + segments: int + duration_s: float + + valid: bool + fit_eligible: bool + reasons: list[Reason] + + def invalidated(self, reason: Reason) -> "MeasuredWindow": + """This window with `reason` among its reasons, and valid and fit-eligible as they then are.""" + if reason in self.reasons: + return self + reasons = [*self.reasons, reason] + valid, fit_eligible = _validity(reasons) + return self.model_copy(update={"reasons": reasons, "valid": valid, "fit_eligible": fit_eligible}) + + +def calibration_reservation(reservation_bytes: int) -> int: + """What a calibration lock reserves for an estimate of `reservation_bytes`: half as much again, at least 1 GiB more.""" + return reservation_bytes + max( + CALIBRATION_HEADROOM_MIN_BYTES, int(CALIBRATION_HEADROOM_FRACTION * reservation_bytes) + ) + + +# All window state, and the registry of open windows, under one lock. Re-entrant because a window's own release runs +# `TorchDevice.empty_cache`, which notifies the probe on the same thread. Lock order: the model cache's lock before +# this one, never the other way round -- `TorchDevice.empty_cache` runs inside cache methods, so nothing here calls into +# a cache while holding it. +_state_lock = threading.RLock() +_open_windows: list["_Window"] = [] +_armed_scopes = 0 + +armed: bool = False +"""True while any thread has a calibration scope open. The model loader checks it before telling the probe about a lock +or an unlock, so without a scope that costs one attribute read.""" + +locking: list[object] = [] +"""The cache of every model lock in progress, once per lock, whichever thread takes it. The model loader adds it before +it reads `armed` and removes it once the lock has settled or failed. A lock that read `armed` before a scope armed never +reports to the probe, and one that reported its start before a window was registered reports again only once it has +settled, perhaps after the window closed. Both move weights while the window measures, so a window that opens while +another thread's lock is in progress on its cache is void. A list, so that the loader's side is an append and a remove, +each atomic under the GIL, and takes no lock.""" + +_boundary = threading.local() + + +def _release() -> None: + """Return the allocator's cached blocks to the driver as a boundary of this thread's windows, which handle it + themselves; other threads' windows see it like any other release.""" + _boundary.active = True + try: + TorchDevice.empty_cache(force=True) + finally: + _boundary.active = False + + +def _krea2_sdpa_benchmark_active() -> bool: + # Imported here: the Krea-2 attention module pulls in diffusers, which a lock must not. + from invokeai.backend.krea2.attention import KREA2_SDPA_BACKEND_ENV_VAR + + # Any non-empty value runs the benchmark (`_Krea2StepBenchmark.create`), or fails the node if it is invalid. + return bool(os.environ.get(KREA2_SDPA_BACKEND_ENV_VAR, "").strip()) + + +class _Window: + """An open window's measurement. Mutated under `_state_lock` only.""" + + def __init__( + self, + scope: "ProbeScope", + cache: object, + record: "CacheRecord", + estimate: WorkingMemoryEstimate, + reservation_applied_bytes: int, + ) -> None: + self.scope = scope + self.cache = cache + self.record = record + self.estimate = estimate + self.reservation_applied_bytes = reservation_applied_bytes + self.device = record.cached_model.compute_device + self.thread = threading.get_ident() + self.parent: Optional[_Window] = None + self.counters: Optional[DeviceCounters] = None + self.state: Literal["measuring", "co_lock", "suspended"] = "measuring" + self.first: Optional[CounterReading] = None + self.baseline: dict[CounterName, Optional[int]] = {} + self.offset: dict[CounterName, int] = dict.fromkeys(COUNTERS, 0) + self.peak: dict[CounterName, int] = {} + self.delta: dict[CounterName, int] = {} + self.unavailable: set[CounterName] = set() + self.reasons: list[Reason] = [] + self.segments = 0 + self.pending_reset = False + self.co_lock_record: Optional["CacheRecord"] = None + self.co_lock_start: Optional[CounterReading] = None + self.co_locked: list[str] = [] + self.resident_fraction = 1.0 + self.opened_at = 0.0 + self.recorded = False + + def add_reason(self, reason: Reason) -> None: + if reason not in self.reasons: + self.reasons.append(reason) + + def fail(self) -> None: + """A probe step on this window failed: its figures can no longer be trusted. Mark it, and leave it in a state + the rest of the window can run in.""" + self.add_reason("probe_error") + self.pending_reset = False + if self.state == "co_lock": + self.state = "measuring" + self.co_lock_record = self.co_lock_start = None + + def start(self, reading: CounterReading) -> None: + self.first = reading + self.opened_at = time.perf_counter() + for name in COUNTERS: + self.baseline[name] = counter_value(reading, name) + if self.baseline[name] is None: + self.unavailable.add(name) + + def fold(self, reading: CounterReading, ends_segment: bool = True) -> None: + """Take `reading` as the end of the current segment, or with `ends_segment=False` as a sample within it.""" + for name in COUNTERS: + if name in self.unavailable: + continue + value, base = counter_peak(reading, name), self.baseline[name] + if value is None or base is None: + self.unavailable.add(name) + continue + self.peak[name] = max(self.peak.get(name, value), value) + share = value - base - self.offset[name] + self.delta[name] = max(self.delta.get(name, share), share) + if ends_segment: + self.segments += 1 + + def begin_co_lock(self, record: "CacheRecord") -> None: + assert self.counters is not None + self.counters.synchronize() + self.fold(self.counters.read()) + _release() + self.co_lock_start = self.counters.read() + self.co_lock_record = record + self.state = "co_lock" + + def end_co_lock(self) -> None: + assert self.counters is not None and self.co_lock_start is not None and self.co_lock_record is not None + self.counters.synchronize() + _release() + settled = self.counters.read() + for name in COUNTERS: + before, after = counter_value(self.co_lock_start, name), counter_value(settled, name) + if before is None or after is None: + self.unavailable.add(name) + else: + self.offset[name] += after - before + if self.co_lock_record.key not in self.co_locked and self.co_lock_record.key != self.record.key: + self.co_locked.append(self.co_lock_record.key) + self.co_lock_record = self.co_lock_start = None + self.counters.reset_peaks() + self.state = "measuring" + + def counter(self, name: CounterName) -> Optional[CounterDelta]: + base = self.baseline.get(name) + if name in self.unavailable or base is None or name not in self.delta: + return None + return CounterDelta(baseline=base, peak=self.peak[name], offset=self.offset[name], delta=self.delta[name]) + + def method(self) -> Method: + if self.first is None: + return "torch-only" + return window_method(self.first, driver_readable="driver" not in self.unavailable) + + def measured(self, duration_s: float) -> MeasuredWindow: + counter_deltas = {name: self.counter(name) for name in COUNTERS} + demand_bytes = demand(None if delta is None else delta.delta for delta in counter_deltas.values()) + if demand_bytes is None: + self.add_reason("counters_unavailable") + valid, fit_eligible = _validity(self.reasons) + method = self.method() + backend = backend_for(self.device) + estimate, scope = self.estimate, self.scope + return MeasuredWindow( + run_id=scope.run_id, + queue_item_id=scope.queue_item_id, + node_id=scope.node_id, + node_type=scope.node_type, + model_key=self.record.key, + device=str(self.device), + estimator=estimate.estimator, + version=estimate.version, + features=dict(estimate.features), + terms=estimate.terms, + reservation_bytes=estimate.reservation_bytes, + source=estimate.source, + reservation_applied_bytes=self.reservation_applied_bytes, + driver=counter_deltas["driver"], + reserved=counter_deltas["reserved"], + allocated=counter_deltas["allocated"], + demand_bytes=demand_bytes, + method=method, + verified=backend is not None and (method, backend) in VERIFIED_METHODS, + resident_fraction=self.resident_fraction, + co_locked_models=list(self.co_locked), + segments=self.segments, + duration_s=duration_s, + valid=valid, + fit_eligible=fit_eligible, + reasons=list(self.reasons), + ) + + def close(self, error: Optional[BaseException]) -> None: + """End the window on its own thread, `error` being what the operation raised, and record it in its scope.""" + try: + self._close(error) + except Exception: + InvokeAILogger.get_logger(__name__).warning( + f"Working-memory probe failed to close the window of {self.record.key}", exc_info=True + ) + self.abort() + + def abort(self) -> None: + """End the window after a probe step failed as it opened or closed; record it as invalid. Never raises.""" + try: + with _state_lock: + self.add_reason("probe_error") + self._unregister() + parent = self.parent + if parent is not None and parent.state == "suspended": + # Folded and released for this window, and not reset after it: what it measures from here is + # suspect. + parent.fail() + parent.state = "measuring" + if not self.recorded: + duration_s = time.perf_counter() - self.opened_at if self.first is not None else 0.0 + self.scope.windows.append(self.measured(duration_s)) + self.recorded = True + except Exception: + InvokeAILogger.get_logger(__name__).debug("Working-memory probe could not record", exc_info=True) + + def _unregister(self) -> None: + if self in _open_windows: + _open_windows.remove(self) + if self in self.scope._stack: + self.scope._stack.remove(self) + if self.counters is not None: + self.counters.close() + + def _close(self, error: Optional[BaseException]) -> None: + if error is not None: + self.add_reason("canceled" if isinstance(error, CanceledException) else "exception") + final: Optional[CounterReading] = None + with _state_lock: + if self.counters is not None and self.first is not None: + self.counters.synchronize() + final = self.counters.read() + self.fold(final) + for reason in reading_reasons(self.first, final): + self.add_reason(reason) + if self in _open_windows: + _open_windows.remove(self) + if self in self.scope._stack: + self.scope._stack.remove(self) + duration_s = time.perf_counter() - self.opened_at + + # Outside the lock: it may wait out the settle time. + if final is not None and self.first is not None and self.counters is not None: + if spilled(self.first, final, self.counters.read): + self.add_reason("spill") + + with _state_lock: + parent = self.parent + if parent is not None and parent in _open_windows and parent.counters is not None: + _release() + parent.counters.reset_peaks() + parent.state = "measuring" + if self.counters is not None: + self.counters.close() + record = self.measured(duration_s) + self.scope.windows.append(record) + self.recorded = True + InvokeAILogger.get_logger(__name__).debug( + f"Measured {record.estimator} on {record.device} for {record.model_key}: " + f"demand {(record.demand_bytes or 0) / MIB:.0f} MB against {record.reservation_bytes / MIB:.0f} MB " + f"estimated ({record.method}{'' if record.valid else ', invalid: ' + ', '.join(record.reasons)})" + ) + + +class ProbeScope: + """One node of a calibration queue item, on the thread that runs it. `windows` collects that node's measurements + in the order they closed; the caller persists them after the node.""" + + def __init__(self, *, run_id: str, queue_item_id: int, node_id: str, node_type: str) -> None: + self.run_id = run_id + self.queue_item_id = queue_item_id + self.node_id = node_id + self.node_type = node_type + self.windows: list[MeasuredWindow] = [] + # This thread's open windows, innermost last. + self._stack: list[_Window] = [] + # The locks this thread holds, per (cache, model key). + self._held: dict[tuple[int, str], int] = {} + # Set when a probe hook failed on this thread: `_held` may be off, so no later window can trust its check. + self._tainted = False + + def _held_on(self, cache: object) -> int: + return sum(count for (cache_id, _key), count in self._held.items() if cache_id == id(cache)) + + def invalidate(self, reason: Literal["exception", "canceled"]) -> None: + """Void every window recorded so far: the node failed or was canceled. + + The session runner swallows a node's failure or cancellation, and calls this instead. It covers the windows that + closed before the node raised: their own operation finished, but the node did not, and no measurement of a node + that did not complete is trusted. + """ + self.windows = [window.invalidated(reason) for window in self.windows] + + def open_window( + self, cache: object, record: "CacheRecord", estimate: WorkingMemoryEstimate, reservation_applied_bytes: int + ) -> Optional[_Window]: + """Start measuring the operation `record`'s model is about to run, once its lock has settled. + + `cache` is the model cache the record is locked in. Never raises: a probe failure leaves the operation alone + and records the window as invalid. A window that cannot be opened is recorded at once, and the operation runs + unmeasured. + """ + window = _Window(self, cache, record, estimate, reservation_applied_bytes) + # Registered before the cache is asked: from here on, a lock another thread starts or settles on this cache + # voids the window (`lock_started`, `lock_settled`), and one in progress or held shows in `locking` and + # `locks_held()`. Its counters are not open yet, so releases pass it by. + with _state_lock: + _open_windows.append(window) + try: + if self._tainted: + window.add_reason("probe_error") + cached_model = record.cached_model + total = cached_model.total_bytes() + resident = cached_model.cur_vram_bytes() + window.resident_fraction = resident / total if total > 0 else 1.0 + if resident < total: + window.add_reason("partial_residency") + window.co_locked = sorted( + key + for (cache_id, key), count in self._held.items() + if cache_id == id(cache) and count > 0 and key != record.key + ) + # Asked outside the probe's lock (see `_state_lock`). A lock in progress is another thread's: this thread + # opens a window only once its own lock has settled. Every lock this thread took since the scope opened is + # in `_held`, so any lock held beyond those is another thread's too. `locking` is read first: a lock that + # leaves it before `locks_held()` is read still holds its record, or is over before the baseline. + in_progress = any(entry is cache for entry in tuple(locking)) + if in_progress or cache.locks_held() > self._held_on(cache): # type: ignore[attr-defined] + window.add_reason("foreign_lock") + if window.device.type == "cuda" and _krea2_sdpa_benchmark_active(): + window.add_reason("krea2_sdpa_benchmark") + except Exception: + InvokeAILogger.get_logger(__name__).debug("Working-memory probe could not inspect the lock", exc_info=True) + window.add_reason("probe_error") + + try: + window.counters = counters.DeviceCounters(window.device) + with _state_lock: + parent = self._stack[-1] if self._stack else None + window.parent = parent + window.counters.synchronize() + if parent is not None and parent.counters is not None: + if parent.state == "measuring": + parent.fold(parent.counters.read()) + parent.state = "suspended" + _release() + window.start(window.counters.read()) + window.counters.reset_peaks() + self._stack.append(window) + return window + except Exception: + InvokeAILogger.get_logger(__name__).warning( + f"Working-memory probe could not open a window for {record.key}", exc_info=True + ) + window.abort() + return None + + def _abandon(self) -> None: + """Drop the windows still open when the scope ends, so no sampler outlives it.""" + with _state_lock: + for window in list(self._stack): + window._unregister() + + +_current_scope: contextvars.ContextVar[Optional[ProbeScope]] = contextvars.ContextVar( + "working_memory_probe_scope", default=None +) + + +def current_scope() -> Optional[ProbeScope]: + """The calibration scope open on this thread, if any.""" + return _current_scope.get() + + +class _EmptyCacheObserver: + """Turns `TorchDevice.empty_cache` into segment boundaries of the open windows.""" + + def before_empty_cache(self) -> None: + try: + me = threading.get_ident() + own_boundary = getattr(_boundary, "active", False) + with _state_lock: + for window in _open_windows: + if window.counters is None or window.state != "measuring": + continue + if window.thread != me: + # Another thread's release (process-global on CUDA) takes this window's cached blocks too: + # read its driver before they go, but leave its peaks alone mid-operation. + window.fold(window.counters.read(), ends_segment=False) + elif not own_boundary: + window.fold(window.counters.read()) + window.pending_reset = True + except Exception: + _all_failed("follow a release") + + def after_empty_cache(self) -> None: + try: + with _state_lock: + for window in _open_windows: + if window.pending_reset and window.counters is not None: + window.counters.reset_peaks() + window.pending_reset = False + except Exception: + _all_failed("follow a release") + + +_observer = _EmptyCacheObserver() + + +def _arm() -> None: + global armed, _armed_scopes + with _state_lock: + _armed_scopes += 1 + if _armed_scopes == 1: + devices.EMPTY_CACHE_OBSERVERS.append(_observer) + armed = True + + +def _disarm() -> None: + global armed, _armed_scopes + with _state_lock: + _armed_scopes -= 1 + if _armed_scopes == 0: + armed = False + if _observer in devices.EMPTY_CACHE_OBSERVERS: + devices.EMPTY_CACHE_OBSERVERS.remove(_observer) + + +@contextmanager +def calibration_scope(*, run_id: str, queue_item_id: int, node_id: str, node_type: str) -> Iterator[ProbeScope]: + """Measure the windows of one node on this thread. Scopes do not nest.""" + if _current_scope.get() is not None: + raise RuntimeError("A calibration scope is already open on this thread; scopes do not nest.") + scope = ProbeScope(run_id=run_id, queue_item_id=queue_item_id, node_id=node_id, node_type=node_type) + token = _current_scope.set(scope) + _arm() + try: + yield scope + finally: + scope._abandon() + _disarm() + _current_scope.reset(token) + + +def _all_failed(what: str) -> None: + """A release boundary failed part-way: any open window may have been folded without its reset.""" + InvokeAILogger.get_logger(__name__).debug(f"Working-memory probe failed to {what}", exc_info=True) + try: + with _state_lock: + for window in _open_windows: + window.fail() + except Exception: + pass + + +def _hook_failed(scope: Optional[ProbeScope], cache: object, what: str) -> None: + """A lock hook failed part-way. On a scope's own thread every window of the scope is suspect, and so is its lock + bookkeeping; on another thread, the windows on that cache which the hook was there to void.""" + InvokeAILogger.get_logger(__name__).debug(f"Working-memory probe failed to {what}", exc_info=True) + try: + with _state_lock: + if scope is not None: + scope._tainted = True + affected = list(scope._stack) + else: + affected = [window for window in _open_windows if window.cache is cache] + for window in affected: + window.fail() + except Exception: + pass + + +def _invalidate_foreign(cache: object) -> None: + me = threading.get_ident() + for window in _open_windows: + if window.cache is cache and window.thread != me: + window.add_reason("foreign_lock") + + +def lock_started(cache: object, record: "CacheRecord") -> None: + """A lock of `record` on its compute device begins on this thread. Called by the model loader while `armed`.""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is None or not scope._stack: + return + window = scope._stack[-1] + if ( + window.cache is cache + and window.state == "measuring" + and record.cached_model.compute_device.type != "cpu" + ): + window.begin_co_lock(record) + except Exception: + _hook_failed(scope, cache, "follow a lock") + + +def lock_settled(cache: object, record: "CacheRecord", locked: bool) -> None: + """That lock has settled, its required tensors repaired, or it failed (`locked=False`, nothing stays locked).""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is None: + return + if locked: + key = (id(cache), record.key) + scope._held[key] = scope._held.get(key, 0) + 1 + if scope._stack: + window = scope._stack[-1] + if window.state == "co_lock" and window.co_lock_record is record: + window.end_co_lock() + except Exception: + _hook_failed(scope, cache, "follow a lock") + + +def locked_in_ram(cache: object, record: "CacheRecord") -> None: + """`record` was pinned in RAM on this thread: no device memory moved, but the lock counts for `locks_held`.""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is not None: + key = (id(cache), record.key) + scope._held[key] = scope._held.get(key, 0) + 1 + except Exception: + _hook_failed(scope, cache, "follow a lock") + + +def unlocked(cache: object, record: "CacheRecord") -> None: + """`record` was unlocked on this thread.""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is not None: + key = (id(cache), record.key) + remaining = scope._held.get(key, 0) - 1 + if remaining > 0: + scope._held[key] = remaining + else: + scope._held.pop(key, None) + except Exception: + _hook_failed(scope, cache, "follow an unlock") diff --git a/invokeai/backend/working_memory/spec.py b/invokeai/backend/working_memory/spec.py index 3f0c116a2eb..18ddb5483da 100644 --- a/invokeai/backend/working_memory/spec.py +++ b/invokeai/backend/working_memory/spec.py @@ -6,14 +6,20 @@ import torch +from invokeai.backend.working_memory import user_fit from invokeai.backend.working_memory.coefficients import ResolvedCoefficients, shipped_coefficients from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm +from invokeai.backend.working_memory.fit import LineFunction from invokeai.backend.working_memory.platform import platform_for TermsFunction = Callable[[Mapping[str, FeatureValue], Mapping[str, float]], tuple[WorkingMemoryTerm, ...]] """An estimator's pure part: its features and its coefficient values in, its terms out.""" -ESTIMATOR_MODULES: Final = ("invokeai.backend.util.vae_working_memory",) +ESTIMATOR_MODULES: Final = ( + "invokeai.backend.util.vae_working_memory", + "invokeai.backend.z_image.denoise_working_memory", + "invokeai.backend.flux2.denoise_working_memory", +) """Every module that registers an `EstimatorSpec`. The registry holds only the specs of modules imported so far, so a caller that needs all of them imports these (`import_estimator_modules`); `tests/backend/working_memory/` fails on a module that declares a spec without being listed here.""" @@ -21,12 +27,14 @@ @dataclass(frozen=True) class EstimatorSpec: - """What a working-memory estimator declares: its id and version, the coefficients its fitted terms read, and the - platform keys those coefficients are shipped per. + """What a working-memory estimator declares: its id and version, the coefficients its fitted terms read, the + platform keys those coefficients are shipped per, and how its features map to the line those terms price on. Registered at import of the module that defines the estimator. `terms` is that estimator's pure part, so an estimate can be evaluated again from recorded features alone -- which is how the measured points under `data/` are - checked. It is behaviour rather than declaration and stays out of equality and hashing. + checked. `line` places features on the estimator's fitted line for the shipped fit (`fit.py`), the ids of measured + points (`bounds.point_id`) and a user calibration. Both are behaviour rather than declaration and stay out of + equality and hashing. """ id: str @@ -35,10 +43,31 @@ class EstimatorSpec: coefficients: frozenset[str] platform_keys: tuple[str, ...] terms: TermsFunction = field(compare=False, repr=False) + line: LineFunction = field(compare=False, repr=False) + """The line an operation's features price on, their size on it, and whether a measurement of them fits it: the + AutoencoderKL convolution line in output pixels for every VAE estimator (`fit.convolution_line`), one in attended + tokens for each denoise estimator. Declared by every estimator, so a new one cannot inherit another's line, and with it + which of its measurements join a fit.""" + user_calibrated: bool = False + """Whether a user's calibration moves this estimator: its windows fit user lines, its estimates read them, the + calibration summary shows its lines, and what its windows found held outside torch sizes the calibration plan + (`recorded_outside_torch_bytes`). Without it, its lines are the shipped fit's alone: a calibration run still records + its windows, and the summary counts them among the windows that fit no line. + + The summary and the calibration plan are built on pixel lines: the summary keys a line's sizes by its windows' + `width` and `height`, and the plan sizes square decodes. A spec whose line is another, such as the Z-Image + denoise's in tokens, must not opt in until they key sizes by `fit.Line.size`.""" def estimate(self, features: Mapping[str, FeatureValue], device: torch.device) -> WorkingMemoryEstimate: - """The estimate for an operation on `device`, priced with the shipped coefficients of the platform it runs on.""" - return self.evaluate(features, shipped_coefficients(self, platform_for(device))) + """The estimate for an operation on `device`, priced with the shipped coefficients of the platform it runs on, + and, for a `user_calibrated` estimator, with the user's own line where a calibration of this device measured it + (`user_fit.resolve`).""" + platform = platform_for(device) + shipped = shipped_coefficients(self, platform) + fit = user_fit.user_fit_for(self, platform, device) + if fit is None: + return self.evaluate(features, shipped) + return user_fit.resolve(self, features, shipped, fit) def evaluate( self, features: Mapping[str, FeatureValue], coefficients: ResolvedCoefficients diff --git a/invokeai/backend/working_memory/user_fit.py b/invokeai/backend/working_memory/user_fit.py new file mode 100644 index 00000000000..0bd46adb432 --- /dev/null +++ b/invokeai/backend/working_memory/user_fit.py @@ -0,0 +1,218 @@ +"""A user's own calibration, fitted per line and read by estimates in place of the shipped values where it measured. + +The app keeps a device class's calibration windows (`app/services/working_memory_calibration/`); this module owns the +rule that turns them into lines and the rule by which an estimate uses them. Lines are those of `fit.py`: a window +joins the line its estimator resolves to on the platform it ran on -- the lender's, where the estimator borrows its +values -- so a calibration of one model's decode moves every estimator that prices with the same line. Only the +estimators that opt in (`EstimatorSpec.user_calibrated`) take part: the others' windows fit no user line, and no user +line moves their estimates. + +An estimate reads a user line through the provider seam: the app registers one (`set_user_fit_provider`) that answers +from memory, and `EstimatorSpec.estimate` asks it per estimate. Without a provider, or when it has no fit for the +device, the estimate is the shipped one, bit for bit. With one, `resolve` applies the user line where it was measured: + +- at or below the largest size the line measured (`UserLine.x_max`), the user line replaces the shipped one; +- above it, the larger of the user and the shipped estimate, since nothing measured says how the line continues. +""" + +import functools +from collections.abc import Callable, Iterable, Mapping +from dataclasses import asdict, dataclass, field, replace +from types import MappingProxyType +from typing import TYPE_CHECKING, Final, Optional + +import torch + +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.working_memory.coefficients import ResolvedCoefficients, find_entry, shipped_data +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate +from invokeai.backend.working_memory.fit import EntryKey, carried_lines, entry_key, user_pair +from invokeai.backend.working_memory.platform import Platform, priced_as + +if TYPE_CHECKING: + from invokeai.backend.working_memory.spec import EstimatorSpec + +LineKey = tuple[EntryKey, str] +"""A line: the entry that carries its values, and its suffix (`fit.Line.suffix`).""" + + +@dataclass(frozen=True) +class UserPoint: + """One calibration window as a fit reads it.""" + + estimator: str + version: int + features: Mapping[str, FeatureValue] + demand_bytes: int + + +@dataclass(frozen=True) +class UserLine: + """A line fitted on a user's windows (`fit.user_pair`), and the sizes it was fitted on.""" + + k: int + """Bytes per unit of the line's size (`fit.Line.size`).""" + b: int + """Bytes whatever the size.""" + x_min: int + """The smallest size measured.""" + x_max: int + """The largest size measured.""" + points: int + + +@dataclass(frozen=True) +class UserFit: + """A device class's user lines in one environment, on the platform they were measured on.""" + + label: str + """Names the fit in an estimate's `source`, e.g. ``user::``.""" + platform: Platform + lines: Mapping[LineKey, UserLine] + _by_spec: dict[tuple[str, int], Mapping[str, UserLine]] = field( + default_factory=dict, init=False, repr=False, compare=False + ) + + def lines_for(self, spec: "EstimatorSpec") -> Mapping[str, UserLine]: + """The user lines `spec` prices with on this fit's platform, by suffix. Resolved once per spec.""" + key = (spec.id, spec.version) + found = self._by_spec.get(key) + if found is None: + carrier = carrier_key(spec.id, spec.version, self.platform) + found = MappingProxyType({suffix: line for (entry, suffix), line in self.lines.items() if entry == carrier}) + self._by_spec[key] = found + return found + + +@functools.cache +def carrier_key(estimator: str, version: int, platform: Platform) -> Optional[EntryKey]: + """The shipped entry whose values `estimator` `version` prices with on `platform`: its own, or the one it borrows + from. None where the shipped data has no entry for it.""" + full = {key: str(value) for key, value in asdict(priced_as(platform)).items()} + entries = [entry for file in shipped_data().values() for entry in file.coefficients] + where = f"{estimator} on {full}" + try: + entry = find_entry(entries, estimator, version, full, where) + if entry.values_from is not None: + entry = find_entry(entries, entry.values_from.estimator, entry.values_from.version, full, where) + except LookupError: + return None + return entry_key(entry) + + +def _carried_lines(key: EntryKey) -> frozenset[str]: + entries = [entry for file in shipped_data().values() for entry in file.coefficients] + for entry in entries: + if entry_key(entry) == key and entry.values is not None: + return frozenset(carried_lines(entry)) + return frozenset() + + +def fit_user_lines( + points: Iterable[UserPoint], platform: Platform, specs: Mapping[str, "EstimatorSpec"] +) -> tuple[Mapping[LineKey, UserLine], Mapping[LineKey, str]]: + """The user lines `points`, measured on `platform`, fit; and, per line they measure but cannot fit, why. + + A point joins a line where its estimator, as `specs` registers it, is `user_calibrated` and its `line` admits the + point's features, its estimator version has shipped values on the platform, the entry carrying them has the line, + and it demanded more than nothing. Each line is `fit.user_pair` of its points: two sizes at least + `fit.MIN_AREA_RATIO` apart, or no line. + """ + groups: dict[LineKey, list[tuple[int, int]]] = {} + for point in points: + spec = specs.get(point.estimator) + if spec is None or not spec.user_calibrated or point.demand_bytes <= 0: + continue + found = spec.line(point.features) + if found is None or not found.fits: + continue + carrier = carrier_key(point.estimator, point.version, platform) + if carrier is None or found.suffix not in _carried_lines(carrier): + continue + groups.setdefault((carrier, found.suffix), []).append((found.size, point.demand_bytes)) + + lines: dict[LineKey, UserLine] = {} + problems: dict[LineKey, str] = {} + for key, measured in groups.items(): + try: + k, b = user_pair(measured) + except ValueError as e: + problems[key] = str(e) + continue + sizes = [size for size, _ in measured] + lines[key] = UserLine(k=k, b=b, x_min=min(sizes), x_max=max(sizes), points=len(measured)) + return MappingProxyType(lines), MappingProxyType(problems) + + +def resolve( + spec: "EstimatorSpec", + features: Mapping[str, FeatureValue], + shipped: ResolvedCoefficients, + fit: UserFit, +) -> WorkingMemoryEstimate: + """The estimate for `features` with `fit`'s line in place of the shipped one, where `fit` has the line and `spec` + is `user_calibrated`. + + The line and its size are read as `fit.py` reads a point's (`EstimatorSpec.line`): for the convolution lines the + output's pixels, also for a tiled operation, whose image is at least its tile, so a tiled operation past the + measured sizes takes the larger estimate. At or below `UserLine.x_max` the user line decides; above, the larger + estimate wins, the shipped one on a tie. `source` names the user fit and the line wherever the user line decides. + """ + found = spec.line(features) if spec.user_calibrated else None + line = fit.lines_for(spec).get(found.suffix) if found is not None else None + if found is None or line is None: + return spec.evaluate(features, shipped) + suffix = found.suffix + values = MappingProxyType({**shipped.values, f"k_{suffix}": line.k, f"b_{suffix}": line.b}) + within = found.size <= line.x_max + where = "within its measured sizes" if within else "above its measured sizes, over shipped" + user = spec.evaluate( + features, ResolvedCoefficients(values=values, source=f"{fit.label} ({suffix} {where}) + {shipped.source}") + ) + if within: + return user + shipped_estimate = spec.evaluate(features, shipped) + if user.reservation_bytes > shipped_estimate.reservation_bytes: + return user + return replace(shipped_estimate, source=f"{shipped.source} (over {fit.label} {suffix}, above its measured sizes)") + + +UserFitProvider = Callable[["EstimatorSpec", Platform, torch.device], Optional[UserFit]] +"""Answers, from memory, the user fit an estimate of `spec` on `device` reads, or None. Called per estimate, so it +reads no file and touches no device.""" + +_provider: Optional[UserFitProvider] = None +_logged_failure: bool = False + +_LOGGER: Final = InvokeAILogger.get_logger(__name__) + + +def set_user_fit_provider(provider: UserFitProvider) -> None: + """Make `provider` the one estimates ask. The app's calibration service registers it when it starts.""" + global _provider + _provider = provider + + +def clear_user_fit_provider(provider: UserFitProvider) -> None: + """Stop asking `provider`, if it is the registered one: a service that stops must not unregister its successor.""" + global _provider + if _provider is provider: + _provider = None + + +def user_fit_for(spec: "EstimatorSpec", platform: Platform, device: torch.device) -> Optional[UserFit]: + """The registered provider's answer, None without one. A provider that raises leaves the estimate shipped: an + estimate must not fail over a calibration.""" + global _logged_failure + provider = _provider + if provider is None: + return None + try: + return provider(spec, platform, device) + except Exception: + if not _logged_failure: + _logged_failure = True + _LOGGER.warning( + "The working-memory user fit could not be read; estimates use the shipped values.", exc_info=True + ) + return None diff --git a/invokeai/backend/z_image/denoise_working_memory.py b/invokeai/backend/z_image/denoise_working_memory.py new file mode 100644 index 00000000000..e279980adfa --- /dev/null +++ b/invokeai/backend/z_image/denoise_working_memory.py @@ -0,0 +1,204 @@ +"""The working memory of one Z-Image transformer forward, as the denoise node reserves it. + +The transformer attends over one unified sequence of image and caption tokens, each run padded to a multiple of +``SEQ_MULTI_OF``. On a fused SDPA kernel its activations scale linearly with that sequence: `k_denoise_` bytes +per attended token plus a base of `b_denoise_` bytes, shipped per platform in +`backend/working_memory/data/denoise.json`. Conditional and unconditional passes run one after the other, so only the +longer caption counts. The platform keys are those of the VAE estimators; for a transformer, `conv_backend` just names +the CUDA (`cudnn`) or ROCm (`miopen`) stack it runs on. + +That line is `denoise_` over the attended tokens (`_line`), and only the plain forward fits it: every analytic +term below comes to zero, and no attention mask is passed, which narrows the kernels SDPA can pick. `_line` reads that +off the terms themselves, so a term added here takes the forwards it prices out of the fit. The fit takes a point's +demand as measured, so a forward with an analytic term bounds its estimate but fits nothing. A materialized score +matrix is no exception, although its term could be subtracted from the demand instead: the term is an upper bound +(`SDPA_MATH_BYTES_PER_SCORE_ELEMENT` against the 9.6 to 16.4 bytes per score measured, and a whole chunk under the ROCm +guard), so the line would come out low by its slack, and as the platform keys do not tell a build that materializes +from one with a fused kernel, that low line would price the fused build short. The line is the shipped fit's alone: +the spec does not opt in to user calibration (`EstimatorSpec.user_calibrated`), so a calibration run records the +forward's windows but no user line moves the estimate. + +The lines are fitted per platform and dtype on fresh-process points of `scripts/calibrate_z_image_working_memory.py` +(their provenance is in the data): an RTX 4090 with the native allocator and with `cudaMallocAsync`, and an RX 9060 XT +under ROCm 10 with AOTriton's fused kernels, with expandable segments and with the native allocator on Windows. bf16, +fp16 and fp32 run at 512 to 2048px with a 512-token caption, and at 1024 and 1440px with a 77-token one, on a +random-init transformer of the real geometry with 2 main blocks: a no-grad forward frees each block's intermediates, so +4 blocks measured no more. bf16 and fp16 take the same per token, fp32 1.7 to 1.9 times that; on ROCm a process's first +forward maps ~250 MiB more than its second, which the base carries. In a running server, the real 30-block Z-Image +Turbo measured below its estimate on the RTX 4090 under the native allocator and under `cudaMallocAsync` in bf16, and +on the RX 9060 XT with expandable segments as an int8 build, whose dequantization transient sits on top. + +ROCm with the native allocator on every OS but Windows is not measured, and keeps the `legacy` constants the node +reserved by hand: 0.25 MiB per attended token and a 1 GiB base, the same for every dtype, from peak reserved memory in +bf16 on the RTX 4090. They cover bf16 and fp16 everywhere measured since, but not fp32 from 1440px up, whose measured +lines are steeper. Their base also covers the fp8 weights cast per forward, which an fp8-stored model reserves as a +term of its own as well, so it pays that cast twice wherever they price. + +Every other platform nothing measures is priced as a measured one (`priced_as`): XPU, MPS, the CPU and cuDNN with +expandable segments as cuDNN with the native allocator, MIOpen under `hipMallocAsync` (and with expandable segments off +Windows) as MIOpen with the native allocator on its OS. A fit describes the card it was measured on, so the two fitted +entries they borrow, the RTX 4090's and the RX 9060 XT's on Windows with the native allocator, floor every value at +the legacy constants there (`unmeasured_floor`): no unmeasured platform reserves less than the node did by hand. + +The analytic terms: + +- Where SDPA has no fused kernel for these shapes it materializes the score matrices, and those dominate: on ROCm + under Windows, whose fused kernels are disabled, a 1024px generation spilled 7.7 GB into shared system memory while + the node reserved only the default working memory. `sdpa_score_matrix_bytes` prices the term only where it is really + built. An attention mask narrows which fused kernels a build can use, so the call is told whether one is passed: + regional prompting's additive bias, or the padding mask the ControlNet forward always builds. The plain forward + passes none at batch size 1. +- Regional prompting's attention bias (`_regional_attention_bias_bytes`). +- A ControlNet adapter first runs its own blocks over a sequence of the same length and keeps one hint per block, + alive through the main layers. Measured over the plain forward at 1024px: 3.3, 5.9 and 18.9 hint-sized tensors + (sequence x width x element size) for 1, 6 and 15 blocks in the script's first runs, and at 15 blocks 19.5 on the + RTX 4090 and 18.4 on the RX 9060 XT since. n + 4 are charged; with the line's margin that covers each of them. +- Half a GiB per LoRA. +- The dequantization copy an int8_tensorwise, nvfp4 or GGUF build materializes per quantized linear and forward, + alive alongside the activations and invisible to them (`peak_dequant_transient_bytes`); zero for every other build. +- The compute-dtype copies of a layer's float8-stored weights that FP8 Storage, or a kept fp8 checkpoint, widens per + forward (`peak_fp8_cast_transient_bytes`); zero for every other build. +""" + +from collections.abc import Mapping +from typing import Final, Optional, cast + +import torch + +from invokeai.backend.quantization.dequantizing_linear import peak_dequant_transient_bytes +from invokeai.backend.util.attention import sdpa_score_matrix_bytes +from invokeai.backend.util.fp8 import peak_fp8_cast_transient_bytes +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm +from invokeai.backend.working_memory.fit import Line, off_line_reasons +from invokeai.backend.working_memory.platform import platform_for +from invokeai.backend.working_memory.spec import EstimatorSpec, register +from invokeai.backend.z_image.z_image_patchify_utils import SEQ_MULTI_OF + +Z_IMAGE_HIDDEN_SIZE: Final = 3840 +"""The transformer's width, which its ControlNet adapters share.""" +Z_IMAGE_ATTENTION_HEADS: Final = 30 +"""Heads of 128.""" + +_CONTROL_HINTS_BEYOND_BLOCKS: Final = 4 +_LORA_PATCH_BYTES: Final = 2**29 +_DTYPE_NAMES: Final[Mapping[torch.dtype, str]] = {torch.bfloat16: "bf16", torch.float16: "fp16", torch.float32: "fp32"} +"""The dtypes `TorchDevice.choose_bfloat16_safe_dtype` runs the transformer in.""" +_ELEMENT_SIZES: Final[Mapping[str, int]] = {"bf16": 2, "fp16": 2, "fp32": 4} +_COEFFICIENTS: Final = frozenset(f"{part}_denoise_{dtype}" for part in ("k", "b") for dtype in _ELEMENT_SIZES) + + +def padded_tokens(tokens: int) -> int: + """The transformer pads the image and the caption token runs to a multiple of `SEQ_MULTI_OF` each.""" + return tokens + (-tokens) % SEQ_MULTI_OF + + +def _regional_attention_bias_bytes( + regional_attn_mask: Optional[torch.Tensor], image_seq_len: int, dtype: torch.dtype +) -> int: + """What `regional_forward` allocates per positive pass for regional prompting: a boolean copy of the mask, the + additive bias over the padded unified sequence, and the image-to-image block `torch.where` fills it from. The mask + itself is already on the device when the estimate is taken. Measured at 1024px on CUDA: 68 MiB over the plain + forward, against 93 MiB charged.""" + if regional_attn_mask is None: + return 0 + mask_side = regional_attn_mask.shape[0] + unified_seq_len = padded_tokens(image_seq_len) + padded_tokens(mask_side - image_seq_len) + return mask_side * mask_side + (unified_seq_len * unified_seq_len + image_seq_len * image_seq_len) * dtype.itemsize + + +def _line(features: Mapping[str, FeatureValue]) -> Optional[Line]: + """The forward's line, `denoise_` over the attended tokens. Only the plain forward fits it: no attention + mask, and every analytic term of `_terms` zero (see the module docstring). The label names what puts a forward off + the line: the mask, and each analytic term that is not zero, in bytes. A dtype without a line names none.""" + dtype, tokens = features.get("dtype"), features.get("tokens") + if not isinstance(dtype, str) or dtype not in _ELEMENT_SIZES or not isinstance(tokens, int): + return None + off_line = off_line_reasons(_terms, features, _COEFFICIENTS) + shape = f"{features.get('image_tokens')}+{features.get('text_tokens')}tok" + return Line(f"denoise_{dtype}", tokens, not off_line, "-".join(("denoise", dtype, shape, *off_line))) + + +def _terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: + tokens = cast(int, features["tokens"]) + dtype = cast(str, features["dtype"]) + control_blocks = cast(int, features["control_blocks"]) + hint_bytes = 0 + if control_blocks: + hint_bytes = ( + (control_blocks + _CONTROL_HINTS_BEYOND_BLOCKS) * tokens * Z_IMAGE_HIDDEN_SIZE * _ELEMENT_SIZES[dtype] + ) + return ( + WorkingMemoryTerm.fitted("activations", f"k_denoise_{dtype}", tokens, values), + WorkingMemoryTerm.fitted("base", f"b_denoise_{dtype}", 1, values), + WorkingMemoryTerm("regional_attention_bias", cast(int, features["regional_bias_bytes"]), True), + WorkingMemoryTerm("attention_scores", cast(int, features["score_matrix_bytes"]), True), + WorkingMemoryTerm("control_hints", hint_bytes, True), + WorkingMemoryTerm("lora_patches", cast(int, features["lora_count"]) * _LORA_PATCH_BYTES, True), + WorkingMemoryTerm("dequant_transient", cast(int, features["dequant_transient_bytes"]), True), + WorkingMemoryTerm("fp8_cast_transient", cast(int, features["fp8_cast_transient_bytes"]), True), + ) + + +Z_IMAGE_DENOISE_SPEC = register( + EstimatorSpec( + id="denoise.z_image", + version=1, + coefficients=_COEFFICIENTS, + platform_keys=("conv_backend", "allocator"), + terms=_terms, + line=_line, + ) +) + + +def estimate_z_image_denoise_working_memory( + transformer: torch.nn.Module, + *, + image_seq_len: int, + text_seq_len: int, + lora_count: int, + control_blocks: int = 0, + regional_attn_mask: Optional[torch.Tensor] = None, + has_attention_mask: bool = False, + device: torch.device, + dtype: torch.dtype, +) -> WorkingMemoryEstimate: + """The working memory one forward of `transformer` needs on `device` in `dtype`. + + `transformer` is the model about to be locked, read unlocked for its dequantization and fp8 cast transients. + `image_seq_len` and `text_seq_len` are the unpadded token counts, the longer caption of a CFG pair. `control_blocks` + is the ControlNet adapter's block count, `regional_attn_mask` regional prompting's mask, and `has_attention_mask` + whether the forward passes an attention mask at all. + """ + dtype_name = _DTYPE_NAMES.get(dtype) + if dtype_name is None: + raise ValueError( + f"Working-memory estimator '{Z_IMAGE_DENOISE_SPEC.id}' has no {dtype} line on {platform_for(device)}: the " + "transformer runs in a dtype it is not measured for." + ) + image_tokens = padded_tokens(image_seq_len) + text_tokens = padded_tokens(text_seq_len) + tokens = image_tokens + text_tokens + features: dict[str, FeatureValue] = { + "image_tokens": image_tokens, + "text_tokens": text_tokens, + "tokens": tokens, + "dtype": dtype_name, + "control_blocks": control_blocks, + "has_attention_mask": has_attention_mask, + "regional_bias_bytes": _regional_attention_bias_bytes(regional_attn_mask, image_seq_len, dtype), + "score_matrix_bytes": sdpa_score_matrix_bytes( + device=device, + dtype=dtype, + num_heads=Z_IMAGE_ATTENTION_HEADS, + head_dim=Z_IMAGE_HIDDEN_SIZE // Z_IMAGE_ATTENTION_HEADS, + seq_len=tokens, + has_attn_mask=has_attention_mask, + # The attention goes through diffusers' `dispatch_attention_fn`, which can route around torch's SDPA. + via_diffusers_dispatch=True, + ), + "lora_count": lora_count, + "dequant_transient_bytes": peak_dequant_transient_bytes(transformer, dtype), + "fp8_cast_transient_bytes": peak_fp8_cast_transient_bytes(transformer, dtype), + } + return Z_IMAGE_DENOISE_SPEC.estimate(features, device) diff --git a/invokeai/frontend/api/openapi.json b/invokeai/frontend/api/openapi.json index 537d0836d51..a955ba82d9b 100644 --- a/invokeai/frontend/api/openapi.json +++ b/invokeai/frontend/api/openapi.json @@ -5660,6 +5660,478 @@ ] } }, + "/api/v2/models/i/{key}/working_memory/calibrate": { + "post": { + "tags": ["model_manager"], + "summary": "Calibrate Model Working Memory", + "description": "Start a working-memory calibration run: enqueue the batch with every node uncached and measure each node's\nmodel operations on the generation device. Needs exactly one generation device, and it must be a GPU. The queue\ntakes all of the batch's items or none.\n\nOnce the items are queued, a request without an `idempotency_key` succeeds even when the store fails for a passing\nreason (an unwritable root, a lock timeout) to record them: they run all the same, and the status finds them by\ntheir origin. With a key, that failure is a 503 although the items are queued. A run removed meanwhile (404) or an\nunusable store file (409) fails the request with or without a key; its queued items then fail when they start.\n\nWith a key, repeating the request (also after a lost response) records the queued items and returns the run; it\nenqueues nothing new. The items come from the queue's enqueue receipt for the key while it exists, otherwise from\nthe queue items that carry the run's origin and those the runner saw. Once a reset removed the run, a repeat is\nrefused (404). Acknowledge the key through the queue's enqueue_batch/acknowledge route once the response\narrived.", + "operationId": "calibrate_model_working_memory", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model the run calibrates", + "title": "Key" + }, + "description": "Key of the model the run calibrates" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRequest", + "description": "The batch to measure" + } + } + } + }, + "responses": { + "201": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRunCreated" + } + } + } + }, + "404": { + "description": "The model could not be found; the run was removed while it was being created, whose items are queued then and fail when they start; or a reset removed the run the idempotency key started" + }, + "409": { + "description": "This machine cannot calibrate now (no GPU, or several generation devices), the device class's store file is unusable and kept as it is (possibly after the items were queued, which then fail when they start), the queue has no room for all of the run's items (it then takes none), or the idempotency key was used for another request" + }, + "422": { + "description": "The graph cannot be measured as a calibration run" + }, + "429": { + "description": "Too many unacknowledged idempotency keys; acknowledge or wait, then retry" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout); with an idempotency key, possibly after the items were queued" + } + } + } + }, + "/api/v2/models/working_memory/status": { + "get": { + "tags": ["model_manager"], + "summary": "Get Working Memory Calibration Status", + "description": "Every device class's calibration runs, with their derived status, point counts and environments.", + "operationId": "get_working_memory_calibration_status", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationStatus" + } + } + } + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + } + }, + "security": [ + { + "HTTPBearer": [] + } + ] + } + }, + "/api/v2/models/i/{key}/working_memory/plan": { + "get": { + "tags": ["model_manager"], + "summary": "Get Model Working Memory Calibration Plan", + "description": "The text-to-image cases that calibrate the model's decode on this machine's GPU, how often each runs, which\nother installed models the calibration also moves, or why the model cannot be calibrated here (`supported`).", + "operationId": "get_model_working_memory_calibration_plan", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model to calibrate", + "title": "Key" + }, + "description": "Key of the model to calibrate" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationPlan" + } + } + } + }, + "404": { + "description": "The model could not be found" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/i/{key}/working_memory/summary": { + "get": { + "tags": ["model_manager"], + "summary": "Get Model Working Memory Calibration Summary", + "description": "What the calibration windows of the model's lines measured on one device class, per estimator line and\nenvironment: the sizes, the largest demand, the shipped and the user estimate, whether the user line is in use,\nstale or unusable, and which models measured it, also for a line the model's decode shares with other models'\nmeasurements; and the model's runs.", + "operationId": "get_model_working_memory_calibration_summary", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model whose measurements to summarise", + "title": "Key" + }, + "description": "Key of the model whose measurements to summarise" + }, + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationModelSummary" + } + } + } + }, + "404": { + "description": "The model could not be found" + }, + "409": { + "description": "No device_class_id was given and there is no current one" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/working_memory/runs/{run_id}": { + "delete": { + "tags": ["model_manager"], + "summary": "Delete Working Memory Calibration Run", + "description": "Remove a calibration run and its measurements. Its queue items that have not run yet fail when they start.", + "operationId": "delete_working_memory_calibration_run", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "run_id", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "The calibration run to remove", + "title": "Run Id" + }, + "description": "The calibration run to remove" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationResetResult" + } + } + } + }, + "404": { + "description": "No usable store file holds the run" + }, + "409": { + "description": "The run's store file became unusable after the run was found; it is kept as it is" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/i/{key}/working_memory": { + "delete": { + "tags": ["model_manager"], + "summary": "Reset Model Working Memory Calibration", + "description": "Remove a model's calibration runs with all their measurements, and every measurement of the model taken in\nother runs (a VAE measured inside a transformer's run, say), from one device class.", + "operationId": "reset_model_working_memory_calibration", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model whose runs and measurements to remove", + "title": "Key" + }, + "description": "Key of the model whose runs and measurements to remove" + }, + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationResetResult" + } + } + } + }, + "409": { + "description": "No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/working_memory": { + "delete": { + "tags": ["model_manager"], + "summary": "Reset Working Memory Calibration", + "description": "Remove every calibration run and measurement of one device class.", + "operationId": "reset_working_memory_calibration", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationResetResult" + } + } + } + }, + "409": { + "description": "No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/working_memory/export": { + "get": { + "tags": ["model_manager"], + "summary": "Export Working Memory Calibration", + "description": "The device class's calibration store file as it is on disk, readable by this version or not.", + "operationId": "export_working_memory_calibration", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "The device class's store file byte for byte: a CalibrationStoreFile (schema_version 1) when this version can read it, otherwise whatever the file holds", + "content": { + "application/json": {} + } + }, + "404": { + "description": "The device class has no store file" + }, + "409": { + "description": "No device_class_id was given and there is no current one" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/image_moves/start": { "post": { "tags": ["image_moves"], @@ -12557,6 +13029,9 @@ }, "description": "Created" }, + "400": { + "description": "The batch's origin starts with 'invokeai:wm-calibration:' (in any case), which is reserved for working-memory calibration runs" + }, "422": { "description": "Validation Error", "content": { @@ -24571,6 +25046,48 @@ "title": "CalculateImageTilesOutput", "type": "object" }, + "CalibrationDeviceClass": { + "properties": { + "id": { + "type": "string", + "title": "Id", + "description": "A short stable hash of the other fields" + }, + "vendor": { + "type": "string", + "title": "Vendor" + }, + "name": { + "type": "string", + "title": "Name" + }, + "total_mib": { + "type": "integer", + "title": "Total Mib" + }, + "backend": { + "type": "string", + "enum": ["cuda", "hip", "mps", "xpu"], + "title": "Backend" + }, + "arch": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Arch" + } + }, + "additionalProperties": false, + "type": "object", + "required": ["id", "vendor", "name", "total_mib", "backend", "arch"], + "title": "CalibrationDeviceClass", + "description": "A `fingerprint.DeviceClass` with its id as a plain field, as it is stored and returned." + }, "CallSavedWorkflowInvocation": { "category": "workflow", "class": "invocation", @@ -116446,6 +116963,1095 @@ ], "title": "WorkflowWithoutID" }, + "WorkingMemoryCalibrationCase": { + "properties": { + "id": { + "type": "string", + "title": "Id", + "description": "Unique within the plan; becomes the graph id" + }, + "mode": { + "type": "string", + "const": "txt2img", + "title": "Mode", + "default": "txt2img" + }, + "model_key": { + "type": "string", + "title": "Model Key", + "description": "The main model the case generates with" + }, + "components": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Components", + "description": "Component picks by Generate slot key; `vae` when a standalone VAE is calibrated or a GPU VAE must be picked over one that runs on the CPU" + }, + "width": { + "type": "integer", + "title": "Width" + }, + "height": { + "type": "integer", + "title": "Height" + }, + "steps": { + "type": "integer", + "title": "Steps", + "description": "The fewest steps that still run the whole pipeline" + }, + "seed": { + "type": "integer", + "title": "Seed" + }, + "positive_prompt": { + "type": "string", + "title": "Positive Prompt" + }, + "negative_prompt": { + "type": "string", + "title": "Negative Prompt" + }, + "settings": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Settings", + "description": "Generate settings overrides: `vaePrecision` (fp16 or fp32) where the decode node casts the VAE" + }, + "estimator": { + "type": "string", + "title": "Estimator", + "description": "The working-memory estimator of the operation the case measures" + }, + "line": { + "type": "string", + "title": "Line", + "description": "The estimator line the case measures, `_`" + } + }, + "type": "object", + "required": [ + "id", + "model_key", + "components", + "width", + "height", + "steps", + "seed", + "positive_prompt", + "negative_prompt", + "settings", + "estimator", + "line" + ], + "title": "WorkingMemoryCalibrationCase", + "description": "One generation a calibration plan runs, in the fields of the client's graph-export case (mode txt2img)." + }, + "WorkingMemoryCalibrationDeviceClass": { + "properties": { + "device_class_id": { + "type": "string", + "title": "Device Class Id" + }, + "device_class": { + "anyOf": [ + { + "$ref": "#/components/schemas/CalibrationDeviceClass" + }, + { + "type": "null" + } + ], + "description": "Null when the file is unusable" + }, + "usable": { + "type": "boolean", + "title": "Usable" + }, + "problem": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Problem", + "description": "Why the file is unusable. It is kept as it is and never written" + }, + "is_current": { + "type": "boolean", + "title": "Is Current" + }, + "measurement_method": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Measurement Method", + "description": "Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present" + }, + "measurement_verified": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Measurement Verified", + "description": "Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present" + }, + "runs": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRunSummary" + }, + "type": "array", + "title": "Runs" + }, + "point_counts": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationPointCount" + }, + "type": "array", + "title": "Point Counts" + }, + "envs": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationEnvironment" + }, + "type": "array", + "title": "Envs" + } + }, + "type": "object", + "required": [ + "device_class_id", + "device_class", + "usable", + "problem", + "is_current", + "measurement_method", + "measurement_verified", + "runs", + "point_counts", + "envs" + ], + "title": "WorkingMemoryCalibrationDeviceClass" + }, + "WorkingMemoryCalibrationDeviceClassRef": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "name": { + "type": "string", + "title": "Name" + } + }, + "type": "object", + "required": ["id", "name"], + "title": "WorkingMemoryCalibrationDeviceClassRef" + }, + "WorkingMemoryCalibrationEnvironment": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "hard": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Hard" + }, + "soft": { + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "type": "object", + "title": "Soft" + }, + "hard_matches_current": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Hard Matches Current", + "description": "Whether the current environment has the same hard facts; null off the current device class" + }, + "soft_matches_current": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Soft Matches Current", + "description": "Whether the current environment has the soft facts first recorded with this environment (`soft`); a later run's own soft facts are not compared here. Null off the current device class" + }, + "point_count": { + "type": "integer", + "title": "Point Count" + } + }, + "type": "object", + "required": ["id", "hard", "soft", "hard_matches_current", "soft_matches_current", "point_count"], + "title": "WorkingMemoryCalibrationEnvironment" + }, + "WorkingMemoryCalibrationLineContributor": { + "properties": { + "model_key": { + "type": "string", + "title": "Model Key" + }, + "name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Name", + "description": "The model's name; null when it is no longer installed" + }, + "windows": { + "type": "integer", + "title": "Windows", + "description": "Its counted windows on the line: valid, fit-eligible and not fragmented" + } + }, + "type": "object", + "required": ["model_key", "name", "windows"], + "title": "WorkingMemoryCalibrationLineContributor", + "description": "A model whose windows feed a line: the model a run calibrated, or the model a window measured." + }, + "WorkingMemoryCalibrationLineSize": { + "properties": { + "width": { + "type": "integer", + "title": "Width" + }, + "height": { + "type": "integer", + "title": "Height" + }, + "pixels": { + "type": "integer", + "title": "Pixels" + }, + "windows": { + "type": "integer", + "title": "Windows", + "description": "Counted: the windows at this size that feed the line, valid, fit-eligible and not fragmented, of every model in the environment; those of a calibration run still running are counted, and join the fit once it finished. Not counted: the valid windows measured at this size" + }, + "measured_bytes": { + "type": "integer", + "title": "Measured Bytes", + "description": "The largest demand of those windows" + }, + "counted": { + "type": "boolean", + "title": "Counted", + "description": "Whether windows of this size feed the user line" + }, + "excluded_reason": { + "anyOf": [ + { + "type": "string", + "enum": ["fragmented", "partial_residency"] + }, + { + "type": "null" + } + ], + "title": "Excluded Reason", + "description": "Why a size is not counted: fragmented, its windows measured the allocator's fragmentation (`backend/working_memory/fragmentation.py`); partial_residency, the model was not fully on the device. Null for a counted size" + }, + "shipped_bytes": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Shipped Bytes", + "description": "The shipped estimate at this size, priced with the line's estimator where one of its windows measured the size, otherwise with that of the largest window; null when the platform of the environment is unknown" + }, + "user_bytes": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "User Bytes", + "description": "The estimate with the user line applied, as estimates read it (without the windows of runs still running); null without a user line" + } + }, + "type": "object", + "required": [ + "width", + "height", + "pixels", + "windows", + "measured_bytes", + "counted", + "excluded_reason", + "shipped_bytes", + "user_bytes" + ], + "title": "WorkingMemoryCalibrationLineSize" + }, + "WorkingMemoryCalibrationModelLine": { + "properties": { + "estimator": { + "type": "string", + "title": "Estimator" + }, + "version": { + "type": "integer", + "title": "Version" + }, + "line": { + "type": "string", + "title": "Line", + "description": "`_`" + }, + "carrier": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Carrier", + "description": "The estimator whose shipped entry carries the line on this platform" + }, + "env_id": { + "type": "string", + "title": "Env Id" + }, + "state": { + "type": "string", + "enum": ["in_use", "stale", "unusable", "insufficient"], + "title": "State", + "description": "in_use: the user line applies; stale: it applies, but the soft facts of the newest finished run that contributed a window differ from the current ones; unusable: hard facts differ from the current environment (or no GPU of this class generates here), so shipped values apply; insufficient: the windows of finished runs cannot fit a line yet" + }, + "changed_facts": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Changed Facts", + "description": "The soft facts that changed (stale) or the hard facts that differ (unusable)" + }, + "problem": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Problem", + "description": "Why no line could be fitted, when it could not" + }, + "measured_range": { + "anyOf": [ + { + "$ref": "#/components/schemas/WorkingMemoryCalibrationPixelRange" + }, + { + "type": "null" + } + ], + "description": "The sizes the user line was fitted on, over every model's windows of the line from finished runs" + }, + "slope_bytes_per_pixel": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Slope Bytes Per Pixel" + }, + "base_bytes": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Base Bytes" + }, + "sizes": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationLineSize" + }, + "type": "array", + "title": "Sizes" + }, + "own_windows": { + "type": "integer", + "title": "Own Windows", + "description": "The model's own windows that feed the line; 0 when only other models' do" + }, + "measured_by": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationLineContributor" + }, + "type": "array", + "title": "Measured By", + "description": "Every model whose windows feed the line, this one included when it has any; by name, then key" + } + }, + "type": "object", + "required": [ + "estimator", + "version", + "line", + "carrier", + "env_id", + "state", + "changed_facts", + "problem", + "measured_range", + "slope_bytes_per_pixel", + "base_bytes", + "sizes", + "own_windows", + "measured_by" + ], + "title": "WorkingMemoryCalibrationModelLine", + "description": "One estimator line in one environment: one the model's windows measured, or, in the current environment, one\nthe model's decode prices on that other models' windows measured." + }, + "WorkingMemoryCalibrationModelRun": { + "properties": { + "run_id": { + "type": "string", + "title": "Run Id" + }, + "status": { + "type": "string", + "enum": ["running", "completed", "failed", "partial", "canceled"], + "title": "Status" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + }, + "created_at": { + "type": "string", + "format": "date-time", + "title": "Created At" + }, + "point_count": { + "type": "integer", + "title": "Point Count" + }, + "attempt_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Attempt Id", + "description": "The client's attempt the run belongs to; null when none was sent" + } + }, + "type": "object", + "required": ["run_id", "status", "error", "created_at", "point_count", "attempt_id"], + "title": "WorkingMemoryCalibrationModelRun" + }, + "WorkingMemoryCalibrationModelSummary": { + "properties": { + "model_key": { + "type": "string", + "title": "Model Key" + }, + "device_class_id": { + "type": "string", + "title": "Device Class Id" + }, + "usable": { + "type": "boolean", + "title": "Usable" + }, + "problem": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Problem", + "description": "Why the store file is unusable" + }, + "current_env_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Current Env Id", + "description": "Null when no generation device of this class is present" + }, + "measurement_method": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Measurement Method", + "description": "Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present" + }, + "measurement_verified": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Measurement Verified", + "description": "Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present" + }, + "lines": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationModelLine" + }, + "type": "array", + "title": "Lines" + }, + "excluded_windows": { + "type": "integer", + "title": "Excluded Windows", + "description": "The model's windows that feed no line: invalid, not fit-eligible, fragmented, tiled, batched, upcast, a VAE outside the AutoencoderKL family, no demand, or of an estimator a user calibration does not move, such as the Z-Image denoise. Of these, only the not fit-eligible and the fragmented windows show among a line's sizes, as a size that is not counted where no counted window measured it" + }, + "runs": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationModelRun" + }, + "type": "array", + "title": "Runs" + } + }, + "type": "object", + "required": [ + "model_key", + "device_class_id", + "usable", + "problem", + "current_env_id", + "measurement_method", + "measurement_verified", + "lines", + "excluded_windows", + "runs" + ], + "title": "WorkingMemoryCalibrationModelSummary", + "description": "What one model's calibration runs measured on one device class, and what estimates make of it." + }, + "WorkingMemoryCalibrationPixelRange": { + "properties": { + "min_pixels": { + "type": "integer", + "title": "Min Pixels" + }, + "max_pixels": { + "type": "integer", + "title": "Max Pixels" + } + }, + "type": "object", + "required": ["min_pixels", "max_pixels"], + "title": "WorkingMemoryCalibrationPixelRange" + }, + "WorkingMemoryCalibrationPlan": { + "properties": { + "model_key": { + "type": "string", + "title": "Model Key" + }, + "supported": { + "type": "boolean", + "title": "Supported" + }, + "reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Reason", + "description": "Why the model cannot be calibrated here, when it cannot" + }, + "device_class": { + "anyOf": [ + { + "$ref": "#/components/schemas/WorkingMemoryCalibrationDeviceClassRef" + }, + { + "type": "null" + } + ], + "description": "The class of the GPU the cases run on; null without exactly one GPU generation device" + }, + "calibrated_through": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Calibrated Through", + "description": "For a standalone VAE, the installed main model the cases generate with" + }, + "estimators": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Estimators", + "description": "The working-memory estimators of the model's covered operations" + }, + "cases": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationCase" + }, + "type": "array", + "title": "Cases" + }, + "repetitions": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRepetitions" + }, + "estimated_seconds": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Estimated Seconds", + "description": "A rough order of magnitude for all cases and repetitions, model load included; not a measurement" + }, + "shared_with": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationSharedModel" + }, + "type": "array", + "title": "Shared With", + "description": "Other installed models whose operations price with the same lines on this GPU, so a calibration changes their estimates too" + }, + "not_covered": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Not Covered", + "description": "Operations of the model a calibration does not measure, and why" + } + }, + "type": "object", + "required": [ + "model_key", + "supported", + "reason", + "device_class", + "calibrated_through", + "estimators", + "cases", + "repetitions", + "estimated_seconds", + "shared_with", + "not_covered" + ], + "title": "WorkingMemoryCalibrationPlan", + "description": "What calibrating one model on this machine's GPU runs, or why it cannot." + }, + "WorkingMemoryCalibrationPointCount": { + "properties": { + "estimator": { + "type": "string", + "title": "Estimator" + }, + "version": { + "type": "integer", + "title": "Version" + }, + "valid": { + "type": "boolean", + "title": "Valid" + }, + "fit_eligible": { + "type": "boolean", + "title": "Fit Eligible" + }, + "count": { + "type": "integer", + "title": "Count" + } + }, + "type": "object", + "required": ["estimator", "version", "valid", "fit_eligible", "count"], + "title": "WorkingMemoryCalibrationPointCount" + }, + "WorkingMemoryCalibrationRepetitions": { + "properties": { + "warm_up": { + "type": "integer", + "title": "Warm Up", + "description": "Runs of each case before the measured ones; their windows count too" + }, + "measured": { + "type": "integer", + "title": "Measured" + } + }, + "type": "object", + "required": ["warm_up", "measured"], + "title": "WorkingMemoryCalibrationRepetitions" + }, + "WorkingMemoryCalibrationRequest": { + "properties": { + "graph": { + "$ref": "#/components/schemas/Graph", + "description": "The graph to measure" + }, + "data": { + "anyOf": [ + { + "items": { + "items": { + "$ref": "#/components/schemas/BatchDatum" + }, + "type": "array" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "The batch data collection" + }, + "idempotency_key": { + "anyOf": [ + { + "type": "string", + "maxLength": 255, + "minLength": 1 + }, + { + "type": "null" + } + ], + "title": "Idempotency Key", + "description": "A caller-owned retry key. Repeating the request with it returns the run it created and enqueues nothing new; once a reset removed that run, the repeat is refused. Reusing the key for another request (another model, graph or batch data) is rejected while the queue keeps the key's enqueue receipt." + }, + "attempt_id": { + "anyOf": [ + { + "type": "string", + "maxLength": 64, + "minLength": 1, + "pattern": "^[A-Za-z0-9_.:-]+$" + }, + { + "type": "null" + } + ], + "title": "Attempt Id", + "description": "A client-generated id of the calibration attempt the run belongs to (1-64 letters, digits, '_', '-', '.', ':'), so a client that enqueues one run per step can group them; no window of an attempt feeds an estimate until every run of the attempt has finished. The run id derives from the idempotency_key, the request and the attempt, so an idempotency_key names one run: repeating the request under the same attempt returns that run, and reusing the key under another attempt is refused with 409 while the queue keeps the key's receipt. Make each key unique per run of an attempt." + } + }, + "additionalProperties": false, + "type": "object", + "required": ["graph"], + "title": "WorkingMemoryCalibrationRequest", + "description": "The batch a calibration run enqueues, built by the client. The server sets origin, destination, runs and the\ngraph's id, and turns the node cache off." + }, + "WorkingMemoryCalibrationResetResult": { + "properties": { + "runs_removed": { + "type": "integer", + "title": "Runs Removed" + }, + "points_removed": { + "type": "integer", + "title": "Points Removed" + } + }, + "type": "object", + "required": ["runs_removed", "points_removed"], + "title": "WorkingMemoryCalibrationResetResult" + }, + "WorkingMemoryCalibrationRunCreated": { + "properties": { + "run_id": { + "type": "string", + "title": "Run Id" + }, + "device_class_id": { + "type": "string", + "title": "Device Class Id" + }, + "item_ids": { + "items": { + "type": "integer" + }, + "type": "array", + "title": "Item Ids", + "description": "The queue items the run enqueued. A repeat of the request that finds them unrecorded takes over every queue item carrying the run's origin, retries included" + } + }, + "type": "object", + "required": ["run_id", "device_class_id", "item_ids"], + "title": "WorkingMemoryCalibrationRunCreated" + }, + "WorkingMemoryCalibrationRunSummary": { + "properties": { + "run_id": { + "type": "string", + "title": "Run Id" + }, + "model_key": { + "type": "string", + "title": "Model Key" + }, + "model_hash": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model Hash" + }, + "user_id": { + "type": "string", + "title": "User Id" + }, + "created_at": { + "type": "string", + "format": "date-time", + "title": "Created At" + }, + "env_id": { + "type": "string", + "title": "Env Id" + }, + "status": { + "type": "string", + "enum": ["running", "completed", "failed", "partial", "canceled"], + "title": "Status", + "description": "Derived from the run's queue items, a retry standing in for the item it retried: running while any is pending or running; failed when any failed; completed when all completed; otherwise partial when valid points were recorded, else canceled" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error", + "description": "The error of a failed item, when the run failed" + }, + "item_ids": { + "items": { + "type": "integer" + }, + "type": "array", + "title": "Item Ids", + "description": "Every queue item of the run, retries included" + }, + "point_count": { + "type": "integer", + "title": "Point Count" + }, + "attempt_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Attempt Id", + "description": "The client's attempt the run belongs to; null when none was sent" + } + }, + "type": "object", + "required": [ + "run_id", + "model_key", + "model_hash", + "user_id", + "created_at", + "env_id", + "status", + "error", + "item_ids", + "point_count", + "attempt_id" + ], + "title": "WorkingMemoryCalibrationRunSummary" + }, + "WorkingMemoryCalibrationSharedModel": { + "properties": { + "key": { + "type": "string", + "title": "Key" + }, + "name": { + "type": "string", + "title": "Name" + }, + "base": { + "type": "string", + "title": "Base" + }, + "type": { + "type": "string", + "title": "Type" + }, + "lines": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Lines", + "description": "The lines it shares with the planned model, `_`" + } + }, + "type": "object", + "required": ["key", "name", "base", "type", "lines"], + "title": "WorkingMemoryCalibrationSharedModel" + }, + "WorkingMemoryCalibrationStatus": { + "properties": { + "available": { + "type": "boolean", + "title": "Available", + "description": "Whether a calibration run can start now: one GPU generation device whose store file is usable" + }, + "unavailable_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Unavailable Reason" + }, + "current_device_class_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Current Device Class Id" + }, + "current_env_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Current Env Id" + }, + "device_classes": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationDeviceClass" + }, + "type": "array", + "title": "Device Classes" + } + }, + "type": "object", + "required": ["available", "unavailable_reason", "current_device_class_id", "current_env_id", "device_classes"], + "title": "WorkingMemoryCalibrationStatus" + }, "ZImageConditioningField": { "description": "A Z-Image conditioning tensor primitive value", "properties": { diff --git a/invokeai/frontend/api/schema.ts b/invokeai/frontend/api/schema.ts index a845bdf0a0f..1c1494973a3 100644 --- a/invokeai/frontend/api/schema.ts +++ b/invokeai/frontend/api/schema.ts @@ -1195,6 +1195,184 @@ export type paths = { patch?: never; trace?: never; }; + "/api/v2/models/i/{key}/working_memory/calibrate": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Calibrate Model Working Memory + * @description Start a working-memory calibration run: enqueue the batch with every node uncached and measure each node's + * model operations on the generation device. Needs exactly one generation device, and it must be a GPU. The queue + * takes all of the batch's items or none. + * + * Once the items are queued, a request without an `idempotency_key` succeeds even when the store fails for a passing + * reason (an unwritable root, a lock timeout) to record them: they run all the same, and the status finds them by + * their origin. With a key, that failure is a 503 although the items are queued. A run removed meanwhile (404) or an + * unusable store file (409) fails the request with or without a key; its queued items then fail when they start. + * + * With a key, repeating the request (also after a lost response) records the queued items and returns the run; it + * enqueues nothing new. The items come from the queue's enqueue receipt for the key while it exists, otherwise from + * the queue items that carry the run's origin and those the runner saw. Once a reset removed the run, a repeat is + * refused (404). Acknowledge the key through the queue's enqueue_batch/acknowledge route once the response + * arrived. + */ + post: operations["calibrate_model_working_memory"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory/status": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get Working Memory Calibration Status + * @description Every device class's calibration runs, with their derived status, point counts and environments. + */ + get: operations["get_working_memory_calibration_status"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/i/{key}/working_memory/plan": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get Model Working Memory Calibration Plan + * @description The text-to-image cases that calibrate the model's decode on this machine's GPU, how often each runs, which + * other installed models the calibration also moves, or why the model cannot be calibrated here (`supported`). + */ + get: operations["get_model_working_memory_calibration_plan"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/i/{key}/working_memory/summary": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get Model Working Memory Calibration Summary + * @description What the calibration windows of the model's lines measured on one device class, per estimator line and + * environment: the sizes, the largest demand, the shipped and the user estimate, whether the user line is in use, + * stale or unusable, and which models measured it, also for a line the model's decode shares with other models' + * measurements; and the model's runs. + */ + get: operations["get_model_working_memory_calibration_summary"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory/runs/{run_id}": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Delete Working Memory Calibration Run + * @description Remove a calibration run and its measurements. Its queue items that have not run yet fail when they start. + */ + delete: operations["delete_working_memory_calibration_run"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/i/{key}/working_memory": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Reset Model Working Memory Calibration + * @description Remove a model's calibration runs with all their measurements, and every measurement of the model taken in + * other runs (a VAE measured inside a transformer's run, say), from one device class. + */ + delete: operations["reset_model_working_memory_calibration"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Reset Working Memory Calibration + * @description Remove every calibration run and measurement of one device class. + */ + delete: operations["reset_working_memory_calibration"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory/export": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Export Working Memory Calibration + * @description The device class's calibration store file as it is on disk, readable by this version or not. + */ + get: operations["export_working_memory_calibration"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/image_moves/start": { parameters: { query?: never; @@ -7975,6 +8153,30 @@ export type components = { */ type: "calculate_image_tiles_output"; }; + /** + * CalibrationDeviceClass + * @description A `fingerprint.DeviceClass` with its id as a plain field, as it is stored and returned. + */ + CalibrationDeviceClass: { + /** + * Id + * @description A short stable hash of the other fields + */ + id: string; + /** Vendor */ + vendor: string; + /** Name */ + name: string; + /** Total Mib */ + total_mib: number; + /** + * Backend + * @enum {string} + */ + backend: "cuda" | "hip" | "mps" | "xpu"; + /** Arch */ + arch: string | null; + }; /** * Call Saved Workflow * @description Displays and later executes against a selected saved workflow. @@ -52635,6 +52837,501 @@ export type components = { [key: string]: components["schemas"]["JsonValue"]; } | null; }; + /** + * WorkingMemoryCalibrationCase + * @description One generation a calibration plan runs, in the fields of the client's graph-export case (mode txt2img). + */ + WorkingMemoryCalibrationCase: { + /** + * Id + * @description Unique within the plan; becomes the graph id + */ + id: string; + /** + * Mode + * @default txt2img + * @constant + */ + mode?: "txt2img"; + /** + * Model Key + * @description The main model the case generates with + */ + model_key: string; + /** + * Components + * @description Component picks by Generate slot key; `vae` when a standalone VAE is calibrated or a GPU VAE must be picked over one that runs on the CPU + */ + components: { + [key: string]: string; + }; + /** Width */ + width: number; + /** Height */ + height: number; + /** + * Steps + * @description The fewest steps that still run the whole pipeline + */ + steps: number; + /** Seed */ + seed: number; + /** Positive Prompt */ + positive_prompt: string; + /** Negative Prompt */ + negative_prompt: string; + /** + * Settings + * @description Generate settings overrides: `vaePrecision` (fp16 or fp32) where the decode node casts the VAE + */ + settings: { + [key: string]: string; + }; + /** + * Estimator + * @description The working-memory estimator of the operation the case measures + */ + estimator: string; + /** + * Line + * @description The estimator line the case measures, `_` + */ + line: string; + }; + /** WorkingMemoryCalibrationDeviceClass */ + WorkingMemoryCalibrationDeviceClass: { + /** Device Class Id */ + device_class_id: string; + /** @description Null when the file is unusable */ + device_class: components["schemas"]["CalibrationDeviceClass"] | null; + /** Usable */ + usable: boolean; + /** + * Problem + * @description Why the file is unusable. It is kept as it is and never written + */ + problem: string | null; + /** Is Current */ + is_current: boolean; + /** + * Measurement Method + * @description Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present + */ + measurement_method: string | null; + /** + * Measurement Verified + * @description Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present + */ + measurement_verified: boolean | null; + /** Runs */ + runs: components["schemas"]["WorkingMemoryCalibrationRunSummary"][]; + /** Point Counts */ + point_counts: components["schemas"]["WorkingMemoryCalibrationPointCount"][]; + /** Envs */ + envs: components["schemas"]["WorkingMemoryCalibrationEnvironment"][]; + }; + /** WorkingMemoryCalibrationDeviceClassRef */ + WorkingMemoryCalibrationDeviceClassRef: { + /** Id */ + id: string; + /** Name */ + name: string; + }; + /** WorkingMemoryCalibrationEnvironment */ + WorkingMemoryCalibrationEnvironment: { + /** Id */ + id: string; + /** Hard */ + hard: { + [key: string]: string; + }; + /** Soft */ + soft: { + [key: string]: string | null; + }; + /** + * Hard Matches Current + * @description Whether the current environment has the same hard facts; null off the current device class + */ + hard_matches_current: boolean | null; + /** + * Soft Matches Current + * @description Whether the current environment has the soft facts first recorded with this environment (`soft`); a later run's own soft facts are not compared here. Null off the current device class + */ + soft_matches_current: boolean | null; + /** Point Count */ + point_count: number; + }; + /** + * WorkingMemoryCalibrationLineContributor + * @description A model whose windows feed a line: the model a run calibrated, or the model a window measured. + */ + WorkingMemoryCalibrationLineContributor: { + /** Model Key */ + model_key: string; + /** + * Name + * @description The model's name; null when it is no longer installed + */ + name: string | null; + /** + * Windows + * @description Its counted windows on the line: valid, fit-eligible and not fragmented + */ + windows: number; + }; + /** WorkingMemoryCalibrationLineSize */ + WorkingMemoryCalibrationLineSize: { + /** Width */ + width: number; + /** Height */ + height: number; + /** Pixels */ + pixels: number; + /** + * Windows + * @description Counted: the windows at this size that feed the line, valid, fit-eligible and not fragmented, of every model in the environment; those of a calibration run still running are counted, and join the fit once it finished. Not counted: the valid windows measured at this size + */ + windows: number; + /** + * Measured Bytes + * @description The largest demand of those windows + */ + measured_bytes: number; + /** + * Counted + * @description Whether windows of this size feed the user line + */ + counted: boolean; + /** + * Excluded Reason + * @description Why a size is not counted: fragmented, its windows measured the allocator's fragmentation (`backend/working_memory/fragmentation.py`); partial_residency, the model was not fully on the device. Null for a counted size + */ + excluded_reason: ("fragmented" | "partial_residency") | null; + /** + * Shipped Bytes + * @description The shipped estimate at this size, priced with the line's estimator where one of its windows measured the size, otherwise with that of the largest window; null when the platform of the environment is unknown + */ + shipped_bytes: number | null; + /** + * User Bytes + * @description The estimate with the user line applied, as estimates read it (without the windows of runs still running); null without a user line + */ + user_bytes: number | null; + }; + /** + * WorkingMemoryCalibrationModelLine + * @description One estimator line in one environment: one the model's windows measured, or, in the current environment, one + * the model's decode prices on that other models' windows measured. + */ + WorkingMemoryCalibrationModelLine: { + /** Estimator */ + estimator: string; + /** Version */ + version: number; + /** + * Line + * @description `_` + */ + line: string; + /** + * Carrier + * @description The estimator whose shipped entry carries the line on this platform + */ + carrier: string | null; + /** Env Id */ + env_id: string; + /** + * State + * @description in_use: the user line applies; stale: it applies, but the soft facts of the newest finished run that contributed a window differ from the current ones; unusable: hard facts differ from the current environment (or no GPU of this class generates here), so shipped values apply; insufficient: the windows of finished runs cannot fit a line yet + * @enum {string} + */ + state: "in_use" | "stale" | "unusable" | "insufficient"; + /** + * Changed Facts + * @description The soft facts that changed (stale) or the hard facts that differ (unusable) + */ + changed_facts: string[]; + /** + * Problem + * @description Why no line could be fitted, when it could not + */ + problem: string | null; + /** @description The sizes the user line was fitted on, over every model's windows of the line from finished runs */ + measured_range: components["schemas"]["WorkingMemoryCalibrationPixelRange"] | null; + /** Slope Bytes Per Pixel */ + slope_bytes_per_pixel: number | null; + /** Base Bytes */ + base_bytes: number | null; + /** Sizes */ + sizes: components["schemas"]["WorkingMemoryCalibrationLineSize"][]; + /** + * Own Windows + * @description The model's own windows that feed the line; 0 when only other models' do + */ + own_windows: number; + /** + * Measured By + * @description Every model whose windows feed the line, this one included when it has any; by name, then key + */ + measured_by: components["schemas"]["WorkingMemoryCalibrationLineContributor"][]; + }; + /** WorkingMemoryCalibrationModelRun */ + WorkingMemoryCalibrationModelRun: { + /** Run Id */ + run_id: string; + /** + * Status + * @enum {string} + */ + status: "running" | "completed" | "failed" | "partial" | "canceled"; + /** Error */ + error: string | null; + /** + * Created At + * Format: date-time + */ + created_at: string; + /** Point Count */ + point_count: number; + /** + * Attempt Id + * @description The client's attempt the run belongs to; null when none was sent + */ + attempt_id: string | null; + }; + /** + * WorkingMemoryCalibrationModelSummary + * @description What one model's calibration runs measured on one device class, and what estimates make of it. + */ + WorkingMemoryCalibrationModelSummary: { + /** Model Key */ + model_key: string; + /** Device Class Id */ + device_class_id: string; + /** Usable */ + usable: boolean; + /** + * Problem + * @description Why the store file is unusable + */ + problem: string | null; + /** + * Current Env Id + * @description Null when no generation device of this class is present + */ + current_env_id: string | null; + /** + * Measurement Method + * @description Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present + */ + measurement_method: string | null; + /** + * Measurement Verified + * @description Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present + */ + measurement_verified: boolean | null; + /** Lines */ + lines: components["schemas"]["WorkingMemoryCalibrationModelLine"][]; + /** + * Excluded Windows + * @description The model's windows that feed no line: invalid, not fit-eligible, fragmented, tiled, batched, upcast, a VAE outside the AutoencoderKL family, no demand, or of an estimator a user calibration does not move, such as the Z-Image denoise. Of these, only the not fit-eligible and the fragmented windows show among a line's sizes, as a size that is not counted where no counted window measured it + */ + excluded_windows: number; + /** Runs */ + runs: components["schemas"]["WorkingMemoryCalibrationModelRun"][]; + }; + /** WorkingMemoryCalibrationPixelRange */ + WorkingMemoryCalibrationPixelRange: { + /** Min Pixels */ + min_pixels: number; + /** Max Pixels */ + max_pixels: number; + }; + /** + * WorkingMemoryCalibrationPlan + * @description What calibrating one model on this machine's GPU runs, or why it cannot. + */ + WorkingMemoryCalibrationPlan: { + /** Model Key */ + model_key: string; + /** Supported */ + supported: boolean; + /** + * Reason + * @description Why the model cannot be calibrated here, when it cannot + */ + reason: string | null; + /** @description The class of the GPU the cases run on; null without exactly one GPU generation device */ + device_class: components["schemas"]["WorkingMemoryCalibrationDeviceClassRef"] | null; + /** + * Calibrated Through + * @description For a standalone VAE, the installed main model the cases generate with + */ + calibrated_through: string | null; + /** + * Estimators + * @description The working-memory estimators of the model's covered operations + */ + estimators: string[]; + /** Cases */ + cases: components["schemas"]["WorkingMemoryCalibrationCase"][]; + repetitions: components["schemas"]["WorkingMemoryCalibrationRepetitions"]; + /** + * Estimated Seconds + * @description A rough order of magnitude for all cases and repetitions, model load included; not a measurement + */ + estimated_seconds: number | null; + /** + * Shared With + * @description Other installed models whose operations price with the same lines on this GPU, so a calibration changes their estimates too + */ + shared_with: components["schemas"]["WorkingMemoryCalibrationSharedModel"][]; + /** + * Not Covered + * @description Operations of the model a calibration does not measure, and why + */ + not_covered: string[]; + }; + /** WorkingMemoryCalibrationPointCount */ + WorkingMemoryCalibrationPointCount: { + /** Estimator */ + estimator: string; + /** Version */ + version: number; + /** Valid */ + valid: boolean; + /** Fit Eligible */ + fit_eligible: boolean; + /** Count */ + count: number; + }; + /** WorkingMemoryCalibrationRepetitions */ + WorkingMemoryCalibrationRepetitions: { + /** + * Warm Up + * @description Runs of each case before the measured ones; their windows count too + */ + warm_up: number; + /** Measured */ + measured: number; + }; + /** + * WorkingMemoryCalibrationRequest + * @description The batch a calibration run enqueues, built by the client. The server sets origin, destination, runs and the + * graph's id, and turns the node cache off. + */ + WorkingMemoryCalibrationRequest: { + /** @description The graph to measure */ + graph: components["schemas"]["Graph"]; + /** + * Data + * @description The batch data collection + */ + data?: components["schemas"]["BatchDatum"][][] | null; + /** + * Idempotency Key + * @description A caller-owned retry key. Repeating the request with it returns the run it created and enqueues nothing new; once a reset removed that run, the repeat is refused. Reusing the key for another request (another model, graph or batch data) is rejected while the queue keeps the key's enqueue receipt. + */ + idempotency_key?: string | null; + /** + * Attempt Id + * @description A client-generated id of the calibration attempt the run belongs to (1-64 letters, digits, '_', '-', '.', ':'), so a client that enqueues one run per step can group them; no window of an attempt feeds an estimate until every run of the attempt has finished. The run id derives from the idempotency_key, the request and the attempt, so an idempotency_key names one run: repeating the request under the same attempt returns that run, and reusing the key under another attempt is refused with 409 while the queue keeps the key's receipt. Make each key unique per run of an attempt. + */ + attempt_id?: string | null; + }; + /** WorkingMemoryCalibrationResetResult */ + WorkingMemoryCalibrationResetResult: { + /** Runs Removed */ + runs_removed: number; + /** Points Removed */ + points_removed: number; + }; + /** WorkingMemoryCalibrationRunCreated */ + WorkingMemoryCalibrationRunCreated: { + /** Run Id */ + run_id: string; + /** Device Class Id */ + device_class_id: string; + /** + * Item Ids + * @description The queue items the run enqueued. A repeat of the request that finds them unrecorded takes over every queue item carrying the run's origin, retries included + */ + item_ids: number[]; + }; + /** WorkingMemoryCalibrationRunSummary */ + WorkingMemoryCalibrationRunSummary: { + /** Run Id */ + run_id: string; + /** Model Key */ + model_key: string; + /** Model Hash */ + model_hash: string | null; + /** User Id */ + user_id: string; + /** + * Created At + * Format: date-time + */ + created_at: string; + /** Env Id */ + env_id: string; + /** + * Status + * @description Derived from the run's queue items, a retry standing in for the item it retried: running while any is pending or running; failed when any failed; completed when all completed; otherwise partial when valid points were recorded, else canceled + * @enum {string} + */ + status: "running" | "completed" | "failed" | "partial" | "canceled"; + /** + * Error + * @description The error of a failed item, when the run failed + */ + error: string | null; + /** + * Item Ids + * @description Every queue item of the run, retries included + */ + item_ids: number[]; + /** Point Count */ + point_count: number; + /** + * Attempt Id + * @description The client's attempt the run belongs to; null when none was sent + */ + attempt_id: string | null; + }; + /** WorkingMemoryCalibrationSharedModel */ + WorkingMemoryCalibrationSharedModel: { + /** Key */ + key: string; + /** Name */ + name: string; + /** Base */ + base: string; + /** Type */ + type: string; + /** + * Lines + * @description The lines it shares with the planned model, `_` + */ + lines: string[]; + }; + /** WorkingMemoryCalibrationStatus */ + WorkingMemoryCalibrationStatus: { + /** + * Available + * @description Whether a calibration run can start now: one GPU generation device whose store file is usable + */ + available: boolean; + /** Unavailable Reason */ + unavailable_reason: string | null; + /** Current Device Class Id */ + current_device_class_id: string | null; + /** Current Env Id */ + current_env_id: string | null; + /** Device Classes */ + device_classes: components["schemas"]["WorkingMemoryCalibrationDeviceClass"][]; + }; /** * ZImageConditioningField * @description A Z-Image conditioning tensor primitive value @@ -55795,6 +56492,391 @@ export interface operations { }; }; }; + calibrate_model_working_memory: { + parameters: { + query?: never; + header?: never; + path: { + /** @description Key of the model the run calibrates */ + key: string; + }; + cookie?: never; + }; + requestBody: { + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationRequest"]; + }; + }; + responses: { + /** @description Successful Response */ + 201: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationRunCreated"]; + }; + }; + /** @description The model could not be found; the run was removed while it was being created, whose items are queued then and fail when they start; or a reset removed the run the idempotency key started */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description This machine cannot calibrate now (no GPU, or several generation devices), the device class's store file is unusable and kept as it is (possibly after the items were queued, which then fail when they start), the queue has no room for all of the run's items (it then takes none), or the idempotency key was used for another request */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The graph cannot be measured as a calibration run */ + 422: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Too many unacknowledged idempotency keys; acknowledge or wait, then retry */ + 429: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout); with an idempotency key, possibly after the items were queued */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + get_working_memory_calibration_status: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationStatus"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + get_model_working_memory_calibration_plan: { + parameters: { + query?: never; + header?: never; + path: { + /** @description Key of the model to calibrate */ + key: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationPlan"]; + }; + }; + /** @description The model could not be found */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + get_model_working_memory_calibration_summary: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path: { + /** @description Key of the model whose measurements to summarise */ + key: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationModelSummary"]; + }; + }; + /** @description The model could not be found */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description No device_class_id was given and there is no current one */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + delete_working_memory_calibration_run: { + parameters: { + query?: never; + header?: never; + path: { + /** @description The calibration run to remove */ + run_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationResetResult"]; + }; + }; + /** @description No usable store file holds the run */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The run's store file became unusable after the run was found; it is kept as it is */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + reset_model_working_memory_calibration: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path: { + /** @description Key of the model whose runs and measurements to remove */ + key: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationResetResult"]; + }; + }; + /** @description No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + reset_working_memory_calibration: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationResetResult"]; + }; + }; + /** @description No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + export_working_memory_calibration: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description The device class's store file byte for byte: a CalibrationStoreFile (schema_version 1) when this version can read it, otherwise whatever the file holds */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description The device class has no store file */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description No device_class_id was given and there is no current one */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; start_image_move: { parameters: { query?: never; @@ -59489,6 +60571,13 @@ export interface operations { "application/json": components["schemas"]["EnqueueBatchResult"]; }; }; + /** @description The batch's origin starts with 'invokeai:wm-calibration:' (in any case), which is reserved for working-memory calibration runs */ + 400: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; /** @description Validation Error */ 422: { headers: { diff --git a/invokeai/frontend/webv2/ARCHITECTURE.md b/invokeai/frontend/webv2/ARCHITECTURE.md index d72f3f3a710..6caa7acb868 100644 --- a/invokeai/frontend/webv2/ARCHITECTURE.md +++ b/invokeai/frontend/webv2/ARCHITECTURE.md @@ -104,7 +104,7 @@ Feature interfaces stay ordinary module interfaces unless two adapters add lever | Nodes | Custom-node catalog/templates and execution lifecycle; Workflow editor and node manager | Platform; descriptors, catalog commands, execution snapshots and manager contribution | | Queue | Backend queue state, realtime interpretation, submission/reconciliation and Queue UI; App runtime, Workbench aggregate/shell, Generation/Gallery/Workflow | Platform and public envelopes/capabilities from Identity, Models, Generation, Workflow, Upscale; backend port, queue reads/commands and widget contribution | | Gallery | Image collection, selection, board policy and Gallery UI; App adapter, Generation, Workflow, Upscale, Models, Workbench image actions | Platform and Queue's public progress reads; gallery reads/commands and widget contribution | -| Generation | Generation settings, validation, graph compilation and Generate UI; Workbench invocation, Queue, Gallery, Workflow, Upscale | Platform and public Models/Gallery/Queue interfaces; settings/validation, compiled submission envelope and widget contribution | +| Generation | Generation settings, validation, graph compilation and Generate UI; Workbench invocation, Queue, Gallery, Workflow, Upscale | Platform and public Models/Gallery/Queue interfaces; settings/validation, compiled submission envelope, offline graph export and widget contribution | | Workflow | Workflow document, validation, compilation, preview/editor/library and explicit publication; App adapter, Workbench invocation, Nodes | Platform and public Nodes/Models/Gallery/Queue/Generation interfaces; document reads/commands, compiled submission envelope and widget contribution | | Upscale | Upscale settings, validation, compilation and UI; Workbench invocation and Queue | Platform and public Models/Gallery/Queue interfaces; value/validation reads, submission command and widget contribution | | Intermediates | Per-project intermediates read models, cleanup previews/operations and the Settings section that manages them; Workbench settings, project menus, account administration | Platform plus Identity's public session/capability reads; query keys, read models, the focus intent, the browser hold lease and the `settingsContribution` metadata whose loader owns the section body | diff --git a/invokeai/frontend/webv2/package.json b/invokeai/frontend/webv2/package.json index 8cc729ef6b6..069b1cc3701 100644 --- a/invokeai/frontend/webv2/package.json +++ b/invokeai/frontend/webv2/package.json @@ -17,6 +17,7 @@ "fix": "pnpm run lint:oxc:fix && pnpm run format:write", "format:check": "oxfmt --check .", "format:write": "oxfmt --write .", + "graphs:export": "node scripts/export-graphs.mjs", "i18n:report": "node scripts/i18n-report.mjs", "i18n:report:verbose": "node scripts/i18n-report.mjs --verbose", "lint": "pnpm run format:check && pnpm run lint:oxc && pnpm run lint:tsc && pnpm run architecture:check", @@ -32,7 +33,7 @@ "test:all": "pnpm run test && pnpm run test:browser && pnpm run test:fixtures", "test:browser": "vitest run --config vitest.browser.config.mts", "test:browser:update-goldens": "CANVAS_GOLDEN_UPDATE=1 vitest run --config vitest.browser.config.mts src/workbench/canvas-engine/render/golden", - "test:fixtures": "node --test scripts/mock-backend-fixtures.test.mjs scripts/preview-server.test.mjs scripts/run-project-file-journey.test.mjs scripts/locale-assets-plugin.test.mjs", + "test:fixtures": "node --test scripts/mock-backend-fixtures.test.mjs scripts/preview-server.test.mjs scripts/run-project-file-journey.test.mjs scripts/locale-assets-plugin.test.mjs scripts/export-graphs.test.mjs", "test:performance:architecture": "pnpm run test:performance:build && pnpm run test:performance:browser", "test:performance:browser": "node scripts/measure-architecture-performance.mjs", "test:performance:browser:update-baseline": "node scripts/measure-architecture-performance.mjs --update-baseline", diff --git a/invokeai/frontend/webv2/performance/browser-baseline.json b/invokeai/frontend/webv2/performance/browser-baseline.json index 002ee7bca76..88f75b73e52 100644 --- a/invokeai/frontend/webv2/performance/browser-baseline.json +++ b/invokeai/frontend/webv2/performance/browser-baseline.json @@ -30,7 +30,7 @@ "fontRawBytes": 23664, "imageRawBytes": 0, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 29, "scriptRawBytes": 1868300, "scriptRequestCount": 25, @@ -68,7 +68,7 @@ "fontRawBytes": 23664, "imageRawBytes": 0, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 29, "scriptRawBytes": 1868300, "scriptRequestCount": 25, @@ -106,7 +106,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, @@ -144,7 +144,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, @@ -182,7 +182,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 96, "scriptRawBytes": 4718489, "scriptRequestCount": 91, @@ -220,7 +220,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 96, "scriptRawBytes": 4718489, "scriptRequestCount": 91, @@ -258,7 +258,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 94, "scriptRawBytes": 4644510, "scriptRequestCount": 88, @@ -296,7 +296,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 94, "scriptRawBytes": 4644510, "scriptRequestCount": 88, @@ -334,7 +334,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, @@ -372,7 +372,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, diff --git a/invokeai/frontend/webv2/public/locales/en.json b/invokeai/frontend/webv2/public/locales/en.json index d09d2247f9d..154188e6349 100644 --- a/invokeai/frontend/webv2/public/locales/en.json +++ b/invokeai/frontend/webv2/public/locales/en.json @@ -1616,7 +1616,86 @@ "variant": "Variant", "variantHelp": "Architecture variant, e.g. inpaint or dev.", "viewModel": "View model", - "viewQueue": "View queue" + "viewQueue": "View queue", + "workingMemory": { + "title": "Working memory", + "description": "Measures this model's VAE decode on this GPU, so Invoke reserves what it really needs.", + "loadFailed": "Could not load working memory", + "stateInUse": "Calibrated", + "stateStale": "May be outdated", + "stateUnusable": "Not used", + "stateInsufficient": "Too few sizes", + "stateNone": "Not calibrated", + "operationDecode": "Decode", + "operationEncode": "Encode", + "pair": "{{label}} · {{detail}}", + "size": "{{width}} × {{height}}", + "measuredRange": "Fitted on {{min}}–{{max}} MP", + "tableLabel": "{{line}} measurements", + "columnSize": "Size", + "columnMeasured": "Measured", + "columnShipped": "Built-in", + "columnCalibrated": "Calibrated estimate", + "aboutColumn": "About {{column}}", + "hintMeasured": "The most VRAM the decode took at this size on this GPU, beyond the model weights.", + "hintShipped": "What Invoke reserves without calibration: values measured on reference GPUs, plus 5%.", + "hintCalibrated": "What Invoke reserves after calibration: a line fitted over your counted measurements, plus 5%. Beyond the largest measured size, the larger of this and Built-in applies.", + "notCounted": "Not counted", + "notCountedFragmented": "Not counted: allocator fragmentation", + "notCountedPartial": "Not counted: model not fully on GPU", + "helpInUse": "Estimates use these measurements.", + "helpStale": "Still used, but {{facts}} changed since.", + "helpUnusable": "Built-in estimates apply: {{facts}} differs.", + "helpUnusableNoFacts": "Built-in estimates apply: measured on a GPU not present here.", + "helpInsufficient": "Built-in estimates apply until more sizes are measured.", + "sharedWith_one": "Also changes {{count}} other model", + "sharedWith_other": "Also changes {{count}} other models", + "notVerified": "Measurement method not verified on this hardware", + "neverCalibrated": "Not calibrated on this GPU. Built-in estimates apply.", + "usesOthers": "Uses measurements from other models on this GPU.", + "measuredBy": "Measured on: {{models}}", + "removedModel": "Removed model", + "modelRemoved": "This model is no longer installed.", + "encodeNotCovered": "Image-to-image and inpainting encodes keep the built-in estimates.", + "floorNote": "Only matters above the working-memory floor. It can change which large decodes tile, and tiled output is not pixel-identical.", + "calibrate": "Calibrate", + "recalibrate": "Recalibrate", + "stop": "Stop", + "blockedUnsupported": "This model cannot be calibrated.", + "blockedUnavailable": "Not available on this machine.", + "blockedStoreUnusable": "Calibration file unusable: {{problem}}", + "blockedRunning": "A calibration is queued or running.", + "queuing": "Queuing: {{done}}/{{total}}", + "running": "Calibrating: {{done}}/{{total}} runs", + "progressLabel": "Calibration progress", + "canceled": "Calibration stopped. Measurements so far are kept.", + "failed": "Calibration failed", + "partial": "Calibration incomplete", + "partialQueued": "Only {{queued}} of {{total}} runs were queued.", + "partialCanceled": "Some runs were canceled.", + "stopTitle": "Stop calibration?", + "stopBody": "Queued runs are canceled. Measurements so far are kept.", + "stopFailed": "Could not stop calibration", + "resetTitle": "Reset calibration?", + "resetBody": "Removes this model's measurements on this GPU. Other models' measurements on the same lines still apply.", + "resetDone": "Calibration reset", + "resetFailed": "Could not reset calibration", + "dialogTitle": "Calibrate working memory", + "dialogDescription": "Runs short generations on {{device}}.", + "dialogDuration": "About {{minutes}} min.", + "dialogQueueNote": "They wait in the queue behind current work.", + "dialogThrough": "Generates with {{model}}.", + "dialogRuns": "Runs: {{count}}", + "dialogRepetitions": "Per size: {{warmUp}} warm-up, {{measured}} measured.", + "facts": { + "allocator": "memory allocator", + "driver": "GPU driver", + "invokeai": "Invoke version", + "osBuild": "OS build", + "sdpaGuard": "ROCm attention guard", + "torchBuild": "PyTorch build" + } + } }, "nodes": { "addNodes": "Add Nodes", diff --git a/invokeai/frontend/webv2/scripts/export-graphs.mjs b/invokeai/frontend/webv2/scripts/export-graphs.mjs new file mode 100644 index 00000000000..cc44c137198 --- /dev/null +++ b/invokeai/frontend/webv2/scripts/export-graphs.mjs @@ -0,0 +1,80 @@ +/** + * `pnpm graphs:export ` compiles offline export cases into the enqueue batches the UI + * would send (src/app/graphExport.ts). Vite's module runner loads the TypeScript sources with the app's aliases and + * opens no port. Exit 0 once the response is written, with per-case errors inside it; exit 2 for an unreadable or + * malformed request, with no response written. + */ + +import { readFile, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import process from 'node:process'; +import { fileURLToPath } from 'node:url'; +import { loadConfigFromFile, runnerImport } from 'vite'; + +const PACKAGE_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const EXIT_MALFORMED_REQUEST = 2; + +const rejectRequest = (message) => { + process.stderr.write(`graphs:export: ${message}\n`); + process.exitCode = EXIT_MALFORMED_REQUEST; +}; + +const readRequest = async (path) => { + try { + return JSON.parse(await readFile(path, 'utf8')); + } catch (error) { + rejectRequest(`cannot read ${path}: ${error instanceof Error ? error.message : String(error)}`); + return undefined; + } +}; + +/** Only the aliases and defines: plugins transform the browser bundle, not these pure modules. */ +const loadExporter = async () => { + const { config } = await loadConfigFromFile( + { command: 'serve', mode: 'development' }, + resolve(PACKAGE_ROOT, 'vite.config.mts'), + PACKAGE_ROOT, + 'error' + ); + const { module } = await runnerImport(resolve(PACKAGE_ROOT, 'src/app/graphExport.ts'), { + define: config.define, + logLevel: 'error', + resolve: { alias: config.resolve.alias }, + root: PACKAGE_ROOT, + }); + + return module; +}; + +const main = async () => { + const [requestPath, responsePath, ...extra] = process.argv.slice(2); + + if (!requestPath || !responsePath || extra.length > 0) { + rejectRequest('usage: pnpm graphs:export '); + return; + } + + const request = await readRequest(requestPath); + + if (request === undefined) { + return; + } + + const { exportGraphs, GraphExportRequestError } = await loadExporter(); + let response; + + try { + response = exportGraphs(request); + } catch (error) { + if (error instanceof GraphExportRequestError) { + rejectRequest(`malformed request: ${error.message}`); + return; + } + + throw error; + } + + await writeFile(responsePath, `${JSON.stringify(response, null, 2)}\n`); +}; + +await main(); diff --git a/invokeai/frontend/webv2/scripts/export-graphs.test.mjs b/invokeai/frontend/webv2/scripts/export-graphs.test.mjs new file mode 100644 index 00000000000..607daf3856b --- /dev/null +++ b/invokeai/frontend/webv2/scripts/export-graphs.test.mjs @@ -0,0 +1,74 @@ +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { access, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, resolve } from 'node:path'; +import test from 'node:test'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; + +const PACKAGE_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const CLI = resolve(PACKAGE_ROOT, 'scripts/export-graphs.mjs'); +const CAPABILITIES = resolve(PACKAGE_ROOT, 'src/features/generation/core/__fixtures__/architectureCapabilities.json'); + +const runCli = async (context, requestText) => { + const directory = await mkdtemp(resolve(tmpdir(), 'invokeai-graphs-export-')); + context.after(() => rm(directory, { force: true, recursive: true })); + const requestPath = resolve(directory, 'request.json'); + const responsePath = resolve(directory, 'response.json'); + await writeFile(requestPath, requestText); + + const { code, stderr } = await promisify(execFile)(process.execPath, [CLI, requestPath, responsePath], { + cwd: PACKAGE_ROOT, + }).then( + ({ stderr }) => ({ code: 0, stderr }), + (error) => ({ code: error.code, stderr: error.stderr }) + ); + const response = await access(responsePath).then( + async () => JSON.parse(await readFile(responsePath, 'utf8')), + () => null + ); + + return { code, response, stderr }; +}; + +test('writes per-case results through the real module graph and exits 0', async (context) => { + const capabilities = JSON.parse(await readFile(CAPABILITIES, 'utf8')); + const sdxl = { base: 'sdxl', format: 'diffusers', hash: 'blake3:sdxl', key: 'sdxl', name: 'SDXL', type: 'main' }; + const testCase = { height: 1024, mode: 'txt2img', positive_prompt: 'a fox', seed: 7, steps: 4, width: 1024 }; + const request = { + capabilities, + cases: [ + { ...testCase, id: 'ok', model_key: 'sdxl' }, + { ...testCase, id: 'missing', model_key: 'missing' }, + ], + models: [sdxl], + version: 1, + }; + + const { code, response } = await runCli(context, JSON.stringify(request)); + + assert.equal(code, 0); + assert.equal(response.version, 1); + assert.deepEqual( + response.cases.map((result) => [result.id, 'batch' in result]), + [ + ['ok', true], + ['missing', false], + ] + ); + assert.match(response.cases[1].error, /model_key "missing"/); +}); + +for (const [label, requestText] of [ + ['unparsable JSON', '{ "version": 1,'], + ['a malformed request', JSON.stringify({ cases: [], models: [], version: 1 })], +]) { + test(`rejects ${label} with exit 2 and writes no response`, async (context) => { + const { code, response, stderr } = await runCli(context, requestText); + + assert.equal(code, 2); + assert.equal(response, null); + assert.match(stderr, /^graphs:export: /); + }); +} diff --git a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs index 8f4d1fbcb50..561b0bc7c28 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs @@ -224,8 +224,8 @@ test('the HTTP reset contract selects profiles explicitly and restores the start } }); -const withRepresentativeBackend = async (run) => { - const backend = await startMockBackend(0, { profile: 'representative' }); +const withRepresentativeBackend = async (run, options = {}) => { + const backend = await startMockBackend(0, { ...options, profile: 'representative' }); try { await run(backend); @@ -1182,6 +1182,142 @@ test('workflow thumbnails round-trip their bytes and refresh the listed URL on e }); }); +test('working-memory calibration: a run per request, replayed by its key, advanced by summary reads', async () => { + let now = 0; + + await withRepresentativeBackend( + async (backend) => { + const base = '/api/v2/models/i/fixture-model-001/working_memory'; + const plan = await getJson(backend, `${base}/plan`); + const post = (key) => + getJson(backend, `${base}/calibrate`, { + body: JSON.stringify({ + attempt_id: 'attempt-1', + data: [], + graph: { edges: [], id: plan.cases[0].id, nodes: {} }, + idempotency_key: key, + }), + method: 'POST', + }); + + assert.equal(plan.supported, true); + assert.deepEqual((await getJson(backend, `${base}/summary`)).runs, []); + + const first = await post('attempt:0'); + assert.deepEqual(await post('attempt:0'), first); + const second = await post('attempt:1'); + assert.notEqual(second.run_id, first.run_id); + + // A read a second after the last step finishes the running item and starts the next one. + const statuses = async () => { + now += 1_050; + return (await getJson(backend, `${base}/summary`)).runs.map((run) => run.status); + }; + assert.deepEqual(await statuses(), ['running', 'running']); + const summary = await getJson(backend, `${base}/summary`); + assert.deepEqual( + summary.runs.map((run) => run.attempt_id), + ['attempt-1', 'attempt-1'] + ); + assert.equal(summary.measurement_verified, true); + const badAttempt = await fetch(`${backend.origin}${base}/calibrate`, { + body: JSON.stringify({ attempt_id: 'has space', graph: { edges: [], id: 'x', nodes: {} } }), + method: 'POST', + }); + assert.equal(badAttempt.status, 422); + assert.deepEqual(await statuses(), ['completed', 'running']); + + const origin = encodeURIComponent(`invokeai:wm-calibration:${second.run_id}`); + assert.deepEqual( + await getJson(backend, `/api/v1/queue/default/cancel_all?origin_prefix=${origin}`, { method: 'PUT' }), + { + canceled: 1, + } + ); + assert.deepEqual(await statuses(), ['completed', 'canceled']); + + assert.deepEqual(await getJson(backend, base, { method: 'DELETE' }), { points_removed: 1, runs_removed: 2 }); + assert.deepEqual((await getJson(backend, `${base}/summary`)).lines, []); + // Calibration items never join the fixture queue. + assert.equal((await getJson(backend, '/api/v1/queue/default/item_ids')).total_count, 500); + }, + { clock: () => now } + ); +}); + +test('working-memory calibration: seeded model states and device scenarios', async () => { + await withRepresentativeBackend(async (backend) => { + const summaryOf = (key) => getJson(backend, `/api/v2/models/i/${key}/working_memory/summary`); + const states = async (key) => (await summaryOf(key)).lines.map((line) => line.state); + + assert.deepEqual(await states('fixture-model-006'), ['in_use', 'in_use']); + assert.deepEqual(await states('fixture-model-011'), ['stale', 'stale']); + assert.deepEqual(await states('fixture-model-016'), ['unusable', 'unusable']); + assert.deepEqual(await states('fixture-model-036'), ['insufficient']); + assert.ok((await summaryOf('fixture-model-021')).runs.some((run) => run.status === 'running')); + assert.ok((await summaryOf('fixture-model-026')).runs.some((run) => run.status === 'failed')); + assert.equal((await getJson(backend, '/api/v2/models/i/fixture-model-031/working_memory/plan')).supported, false); + + // A model never calibrated itself shows the lines another model measured, which name both sides. + const shared = (await summaryOf('fixture-model-041')).lines; + assert.deepEqual( + shared.map((line) => [line.state, line.own_windows, line.measured_by.map((entry) => entry.model_key)]), + [ + ['in_use', 0, ['fixture-model-006']], + ['in_use', 0, ['fixture-model-006']], + ] + ); + const carrier = (await summaryOf('fixture-model-006')).lines[0]; + assert.equal(carrier.own_windows, 9); + assert.deepEqual(carrier.measured_by, [{ model_key: 'fixture-model-006', name: 'Fixture Model 006', windows: 9 }]); + assert.equal((await fetch(`${backend.origin}/api/v2/models/i/no-such-model/working_memory/summary`)).status, 404); + + // A warm 9060 XT: the largest fp32 size measured fragmentation and is left out of the fit and its range; an fp16 + // size with one clean window among partially resident ones still counts, on that window alone. + const [fp16, fp32] = (await summaryOf('fixture-model-046')).lines; + const sizesOf = (line) => line.sizes.map((size) => [size.width, size.counted, size.excluded_reason, size.windows]); + assert.deepEqual(sizesOf(fp32), [ + [576, true, null, 3], + [832, true, null, 3], + [1216, false, 'fragmented', 3], + ]); + assert.deepEqual(fp32.measured_range, { max_pixels: 832 * 832, min_pixels: 576 * 576 }); + // Above that range the server prices the larger of the user line and the shipped value; here the line is lower. + const [, within, above] = fp32.sizes; + assert.equal(above.user_bytes, above.shipped_bytes); + assert.ok(within.user_bytes < within.shipped_bytes); + assert.equal(fp32.own_windows, 6); + assert.equal(fp32.state, 'in_use'); + assert.deepEqual(sizesOf(fp16), [ + [576, true, null, 3], + [832, true, null, 1], + [1216, true, null, 3], + ]); + assert.equal(fp16.own_windows, 7); + + await getJson(backend, '/__faults', { body: JSON.stringify({ workingMemory: 'notVerified' }), method: 'POST' }); + assert.equal((await summaryOf('fixture-model-006')).measurement_verified, false); + + await getJson(backend, '/__faults', { body: JSON.stringify({ workingMemory: 'multiGpu' }), method: 'POST' }); + const status = await getJson(backend, '/api/v2/models/working_memory/status'); + assert.equal(status.available, false); + assert.match(status.unavailable_reason, /exactly one generation device/); + assert.equal( + (await fetch(`${backend.origin}/api/v2/models/i/fixture-model-001/working_memory/summary`)).status, + 409 + ); + + await getJson(backend, '/__faults', { body: JSON.stringify({ workingMemory: 'error' }), method: 'POST' }); + assert.equal((await fetch(`${backend.origin}/api/v2/models/i/fixture-model-001/working_memory/plan`)).status, 503); + + const unknown = await fetch(`${backend.origin}/__faults`, { + body: JSON.stringify({ workingMemory: 'sideways' }), + method: 'POST', + }); + assert.equal(unknown.status, 400); + }); +}); + test('shared boards answer as the backend role table does', async () => { const backend = await startMockBackend(0, { profile: 'representative' }); const call = (path, init) => diff --git a/invokeai/frontend/webv2/scripts/mock-backend.mjs b/invokeai/frontend/webv2/scripts/mock-backend.mjs index 7d101a6cbda..a23d0f36691 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend.mjs @@ -11,6 +11,7 @@ import { getMockBackendFixtureCounts, MOCK_BACKEND_FIXED_EPOCH, } from './mock-backend-fixtures.mjs'; +import { createWorkingMemoryMock, WORKING_MEMORY_SCENARIOS } from './mock-working-memory.mjs'; /** The canvas schema every mock project declares; the real server keeps the same floor per record. */ const DEFAULT_CANVAS_SCHEMA_VERSION = 3; @@ -77,11 +78,12 @@ const ARCHITECTURE_CAPABILITIES = JSON.parse( * multi-user auth, with a non-admin or an admin session: sign in with any email and password (or reload a page that * already holds the token). `sharedBoards: true` adds boards other accounts own (`SHARED_BOARD_FIXTURES`), which the * caller sees through general access, a membership, or admin rights. + * `workingMemory` selects a calibration device scenario (see mock-working-memory.mjs). */ const CAPABILITY_FAULTS = new Set(['ok', 'error', 'empty']); const INTERMEDIATES_CALLERS = new Set(['admin', 'user', 'multiuser-admin']); -const createFaults = () => ({ capabilities: 'ok', intermediatesCaller: 'admin' }); +const createFaults = () => ({ capabilities: 'ok', intermediatesCaller: 'admin', workingMemory: 'ok' }); const isAdminCaller = (faults) => faults.intermediatesCaller !== 'user'; const isMultiuserCaller = (faults) => faults.intermediatesCaller !== 'admin'; @@ -249,7 +251,8 @@ const STARTER_MODELS_RESPONSE = { const DELIVERY_CLAIM_TTL_MS = 120_000; -const createState = (profile) => { +/** `clock` is the working-memory mock's time source (see mock-working-memory.mjs). */ +const createState = (profile, clock) => { const fixture = assertMockBackendFixture(createMockBackendFixture(profile)); return { @@ -282,6 +285,7 @@ const createState = (profile) => { invocationCache: { enabled: true, hits: 0, max_size: 1000, misses: 0, size: 0 }, videos: new Map(fixture.videos.map((video) => [video.video_name, clone(video)])), workflows: new Map(fixture.workflows.map((workflow) => [workflow.workflow_id, clone(workflow)])), + workingMemory: createWorkingMemoryMock(fixture.models, { clock }), nextWorkflowNumber: fixture.workflows.length + 1, /** Custom workflow thumbnails by workflow id; the version stands in for the server's cache-busting query. */ workflowThumbnails: new Map(), @@ -929,9 +933,9 @@ const parseByteRange = (value, size) => { : null; }; -export const startMockBackend = async (port, { profile = 'empty' } = {}) => { +export const startMockBackend = async (port, { clock = Date.now, profile = 'empty' } = {}) => { const initialProfile = assertMockBackendProfileName(profile); - let state = createState(initialProfile); + let state = createState(initialProfile, clock); let faults = createFaults(); const server = createServer(async (request, response) => { @@ -950,7 +954,7 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { const requestedProfile = url.searchParams.get('profile') ?? body.profile ?? initialProfile; try { - state = createState(assertMockBackendProfileName(requestedProfile)); + state = createState(assertMockBackendProfileName(requestedProfile), clock); } catch (error) { return json(400, { detail: error instanceof Error ? error.message : String(error) }); } @@ -983,6 +987,7 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { const body = await readJsonBody(request); const requested = url.searchParams.get('capabilities') ?? body.capabilities; const caller = url.searchParams.get('intermediatesCaller') ?? body.intermediatesCaller; + const workingMemory = url.searchParams.get('workingMemory') ?? body.workingMemory; if (body.sharedBoards === true && !state.boards.has(SHARED_BOARD_FIXTURES[0].board_id)) { seedSharedBoards(state); @@ -1003,6 +1008,14 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { faults.intermediatesCaller = caller; } + + if (workingMemory !== undefined && workingMemory !== null) { + if (!WORKING_MEMORY_SCENARIOS.has(workingMemory)) { + return json(400, { detail: `Unknown working-memory scenario: ${String(workingMemory)}` }); + } + + faults.workingMemory = workingMemory; + } } return json(200, { ok: true, faults: { ...faults } }); @@ -1649,6 +1662,23 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { return json(200, { item_ids: returned, total_count: returned.length }); } + if (action === 'cancel_all' && method === 'PUT') { + const originPrefix = url.searchParams.get('origin_prefix'); + + // Calibration items live with the working-memory mock; see mock-working-memory.mjs. + if (originPrefix !== null && state.workingMemory.isCalibrationPrefix(originPrefix)) { + return json(200, { canceled: state.workingMemory.cancelByOrigin(originPrefix) }); + } + + let canceled = 0; + for (const item of scopedItems) { + if (item.status === 'pending' || item.status === 'in_progress') { + item.status = 'canceled'; + canceled += 1; + } + } + return json(200, { canceled }); + } if (action === 'clear' && method === 'PUT') { state.queueItems.clear(); return json(200, { deleted: true }); @@ -1732,6 +1762,15 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { return json(200, null); } + if (path.startsWith('/api/v2/models/') && path.includes('working_memory')) { + const body = method === 'POST' ? await readJsonBody(request) : null; + const answer = state.workingMemory.handle({ body, method, path, scenario: faults.workingMemory }); + + if (answer) { + return json(answer.status, answer.body); + } + } + if (method === 'GET' && path === '/api/v2/models/stats') { return json(200, null); } diff --git a/invokeai/frontend/webv2/scripts/mock-working-memory.mjs b/invokeai/frontend/webv2/scripts/mock-working-memory.mjs new file mode 100644 index 00000000000..5845f9ea18e --- /dev/null +++ b/invokeai/frontend/webv2/scripts/mock-working-memory.mjs @@ -0,0 +1,593 @@ +/** + * Working-memory calibration for the mock backend: plans, summaries, the device status, calibration runs and resets + * (`/api/v2/models/.../working_memory`), and cancelling a run's items by origin. The mock has no Socket.IO, so a run + * advances when a client reads the model's summary: a read at least a second after the last step finishes the running + * item and starts the next, which a section polling every two seconds turns into visible progress. + * + * Main models (`fixture-model-001`, `-006`, ...) each show one state, by key (`MODEL_SCENARIOS`); every other main model is never + * calibrated, and VAEs are unsupported. `SHARED_LINES` pairs models whose decode lines pool their measurements, so one + * of them shows lines only the other measured. `fixture-model-046` follows a warm RX 9060 XT: its largest fp32 size + * measured allocator fragmentation and is not counted, and two of three windows at one fp16 size ran partially resident, + * which leaves that size counted on the third. `POST /__faults` with `workingMemory` selects a device scenario instead. + * Calibration items live here rather than in the fixture queue, so profile counts stay fixed. Tests pass their own + * `clock` to advance runs without waiting. + */ + +export const WORKING_MEMORY_SCENARIOS = new Set(['ok', 'multiGpu', 'noGpu', 'storeUnusable', 'error', 'notVerified']); + +const ORIGIN_PREFIX = 'invokeai:wm-calibration:'; +const ATTEMPT_ID_PATTERN = /^[A-Za-z0-9_.:-]{1,64}$/; +const DEVICE_CLASS = { + arch: 'sm_89', + backend: 'cuda', + id: '0123456789abcdef', + name: 'NVIDIA GeForce RTX 4090', + total_mib: 24_564, + vendor: 'nvidia', +}; +const ENV_ID = 'fedcba9876543210'; +const OLD_ENV_ID = '00ff00ff00ff00ff'; +const MIB = 2 ** 20; +const SIDES = [768, 1088, 1536]; +const DTYPES = ['fp16', 'fp32']; +const REPETITIONS = { measured: 2, warm_up: 1 }; +const ENCODE_NOT_COVERED = + 'encode: text-to-image runs no encode, so image-to-image and inpainting encodes keep the shipped values'; +const SEEDED_AT = Date.parse('2025-12-01T10:00:00.000Z'); +/** A run's items advance at most this often, so a section that refetches on every queued step still shows progress. */ +const STEP_MS = 1_000; + +/** What each seeded main model shows, by key. */ +const MODEL_SCENARIOS = { + 'fixture-model-006': 'completed', + 'fixture-model-011': 'stale', + 'fixture-model-016': 'unusable', + 'fixture-model-021': 'running', + 'fixture-model-026': 'failed', + 'fixture-model-031': 'unsupported', + 'fixture-model-036': 'insufficient', + 'fixture-model-046': 'fragmented', +}; + +/** The sizes the `fragmented` scenario measured, as the 9060 XT plan picked them. */ +const FRAGMENTED_SIDES = [576, 832, 1216]; + +/** Why a seeded window of the `fragmented` scenario is excluded from the fit, by its step; undefined when it counts. */ +const excludedIn = (scenario, step, repetition) => { + if (scenario !== 'fragmented') { + return undefined; + } + + if (step.line === 'decode_fp32' && step.width === 1216) { + return 'fragmented'; + } + + return step.line === 'decode_fp16' && step.width === 832 && repetition < 2 ? 'partial_residency' : undefined; +}; + +/** Models whose lines pool both models' measurements; the first is never calibrated itself. */ +const SHARED_LINES = [['fixture-model-041', 'fixture-model-006']]; + +const partnersOf = (modelKey) => + SHARED_LINES.flatMap(([left, right]) => (left === modelKey ? [right] : right === modelKey ? [left] : [])); + +/** A plausible decode demand: the VAE's weights plus a per-pixel activation cost, twice that in fp32. */ +const demandOf = (pixels, dtype) => Math.round((dtype === 'fp32' ? 2 : 1) * (420 * MIB + pixels * 1_100)); + +const casesOf = (model) => + DTYPES.flatMap((dtype) => + (MODEL_SCENARIOS[model.key] === 'fragmented' ? FRAGMENTED_SIDES : SIDES).map((side) => ({ + components: {}, + estimator: 'sd_vae_decode', + height: side, + id: `sd_vae_decode-decode-${dtype}-${side}x${side}`, + line: `decode_${dtype}`, + mode: 'txt2img', + model_key: model.key, + negative_prompt: '', + positive_prompt: 'a mountain lake at dawn, detailed photograph', + seed: 1, + settings: { vaePrecision: dtype }, + steps: 2, + width: side, + })) + ); + +const stepsOf = (cases) => + DTYPES.flatMap((dtype) => + cases + .filter((item) => item.line === `decode_${dtype}`) + .flatMap((item) => Array.from({ length: REPETITIONS.warm_up + REPETITIONS.measured }, () => item)) + ); + +export const createWorkingMemoryMock = (models, { clock = Date.now } = {}) => { + const mainModels = models.filter((model) => model.type === 'main'); + const byKey = new Map(models.map((model) => [model.key, model])); + /** Runs by model key, oldest first: `{ run_id, created_at, items: [{ item_id, case, status, error }] }`. */ + const runs = new Map(); + /** Measured points by model key: `{ line, width, height, bytes, env_id }`. */ + const points = new Map(); + /** Run ids by idempotency key, so a resent request returns the run it created. */ + const receipts = new Map(); + let nextRunNumber = 1; + let nextItemId = 900_000; + let createdCount = 0; + let lastAdvanceAt = 0; + + const now = () => new Date(SEEDED_AT + 86_400_000 * 30 + createdCount++ * 1_000).toISOString(); + + /** `excluded` names why the fit leaves the window out: `fragmented` or `partial_residency`. */ + const addPoint = (modelKey, item, envId = ENV_ID, excluded = undefined) => { + const dtype = item.case.line.split('_').at(-1); + const list = points.get(modelKey) ?? []; + const demand = demandOf(item.case.width * item.case.height, dtype); + + list.push({ + // A fragmented allocator reserves well beyond what the decode needs. + bytes: excluded === 'fragmented' ? Math.round(demand * 1.4) : demand, + env_id: envId, + excluded, + height: item.case.height, + line: item.case.line, + width: item.case.width, + }); + points.set(modelKey, list); + }; + + const createRun = (modelKey, steps, createdAt, attemptId) => { + const run = { + attempt_id: attemptId, + created_at: createdAt, + items: steps.map((step) => ({ case: step, error: null, item_id: nextItemId++, status: 'pending' })), + run_id: `mock-wm-run-${String(nextRunNumber++).padStart(4, '0')}`, + }; + const list = runs.get(modelKey) ?? []; + + list.push(run); + runs.set(modelKey, list); + + return run; + }; + + /** Seeds one model's state: settled runs one per step, two seconds apart, a month before now. */ + const seed = (model, scenario) => { + const cases = casesOf(model); + const steps = scenario === 'insufficient' ? stepsOf(cases).slice(0, 3) : stepsOf(cases); + const settledRuns = steps.map((step, index) => + createRun(model.key, [step], new Date(SEEDED_AT + index * 2_000).toISOString(), `seed-${model.key}`) + ); + + settledRuns.forEach((run, index) => { + const [item] = run.items; + const isRunning = scenario === 'running' && index >= 4; + + if (scenario === 'failed' && index === 7) { + item.status = 'failed'; + item.error = 'OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB'; + } else if (scenario === 'failed' && index > 7) { + item.status = 'canceled'; + } else if (isRunning) { + item.status = index === 4 ? 'in_progress' : 'pending'; + } else { + const repetition = index % (REPETITIONS.warm_up + REPETITIONS.measured); + + item.status = 'completed'; + addPoint( + model.key, + item, + scenario === 'unusable' ? OLD_ENV_ID : ENV_ID, + excludedIn(scenario, item.case, repetition) + ); + } + }); + }; + + const reset = () => { + runs.clear(); + points.clear(); + receipts.clear(); + nextRunNumber = 1; + nextItemId = 900_000; + createdCount = 0; + lastAdvanceAt = 0; + + for (const model of mainModels) { + const scenario = MODEL_SCENARIOS[model.key]; + + if (scenario && scenario !== 'unsupported') { + seed(model, scenario); + } + } + }; + + reset(); + + const runStatus = (run, pointCount) => { + const statuses = run.items.map((item) => item.status); + + if (statuses.some((status) => status === 'pending' || status === 'in_progress')) { + return { error: null, status: 'running' }; + } + + const failed = run.items.find((item) => item.status === 'failed'); + + if (failed) { + return { error: failed.error, status: 'failed' }; + } + + if (statuses.every((status) => status === 'completed')) { + return { error: null, status: 'completed' }; + } + + return { error: null, status: pointCount > 0 ? 'partial' : 'canceled' }; + }; + + const pointCountOf = (run) => run.items.filter((item) => item.status === 'completed').length; + + const modelRuns = (modelKey) => + (runs.get(modelKey) ?? []).map((run) => { + const pointCount = pointCountOf(run); + + return { + attempt_id: run.attempt_id, + created_at: run.created_at, + point_count: pointCount, + run_id: run.run_id, + ...runStatus(run, pointCount), + }; + }); + + /** Finishes the running item of the model's oldest unsettled run and starts the next one. */ + const advance = (modelKey) => { + if (clock() - lastAdvanceAt < STEP_MS) { + return; + } + lastAdvanceAt = clock(); + + const items = (runs.get(modelKey) ?? []).flatMap((run) => run.items); + const running = items.find((item) => item.status === 'in_progress'); + + if (running) { + running.status = 'completed'; + addPoint(modelKey, running); + } + + const next = items.find((item) => item.status === 'pending'); + + if (next) { + next.status = 'in_progress'; + } + }; + + /** + * A model's lines: its own points, plus its partners' points in the current environment, which the server pools + * the same way. A line only partners measured appears with `own_windows` 0. + */ + const linesOf = (modelKey, scenario) => { + const own = (points.get(modelKey) ?? []).map((point) => ({ ...point, model_key: modelKey })); + const shared = partnersOf(modelKey).flatMap((key) => + (points.get(key) ?? []).filter((point) => point.env_id === ENV_ID).map((point) => ({ ...point, model_key: key })) + ); + + return DTYPES.flatMap((dtype) => { + const line = `decode_${dtype}`; + const ownPoints = own.filter((point) => point.line === line); + const envId = ownPoints.at(-1)?.env_id ?? ENV_ID; + const linePoints = [...ownPoints, ...(envId === ENV_ID ? shared.filter((point) => point.line === line) : [])]; + + if (linePoints.length === 0) { + return []; + } + + // Only counted windows feed the fit; a size counts while any of its windows does. + const countedPoints = linePoints.filter((point) => point.excluded === undefined); + const sides = [...new Set(linePoints.map((point) => point.width))].sort((left, right) => left - right); + const countedSides = new Set(countedPoints.map((point) => point.width)); + const fitted = countedSides.size >= 2; + const maxCounted = Math.max(...countedSides); + const sizes = sides.map((side) => { + const counted = countedSides.has(side); + const atSide = (counted ? countedPoints : linePoints).filter((point) => point.width === side); + const largest = atSide.reduce((left, right) => (right.bytes > left.bytes ? right : left)); + const pixels = side * side; + const demand = demandOf(pixels, dtype); + const shipped = Math.round(demand * 1.35); + const line = Math.round(demand * 1.05); + + return { + counted, + excluded_reason: counted ? null : largest.excluded, + height: side, + measured_bytes: largest.bytes, + pixels, + shipped_bytes: shipped, + // Like the server's resolve: above the measured range the larger of the user line and the shipped value. + user_bytes: fitted && envId === ENV_ID ? (side > maxCounted ? Math.max(line, shipped) : line) : null, + width: side, + windows: atSide.length, + }; + }); + const state = + envId !== ENV_ID ? 'unusable' : !fitted ? 'insufficient' : scenario === 'stale' ? 'stale' : 'in_use'; + const measuredBy = [...new Set(countedPoints.map((point) => point.model_key))] + .map((key) => ({ + model_key: key, + name: byKey.get(key)?.name ?? null, + windows: countedPoints.filter((point) => point.model_key === key).length, + })) + .sort((left, right) => left.name.localeCompare(right.name) || left.model_key.localeCompare(right.model_key)); + + return [ + { + base_bytes: fitted ? 420 * MIB : null, + carrier: 'sd_vae_decode', + changed_facts: state === 'stale' ? ['driver'] : state === 'unusable' ? ['torch', 'cuda'] : [], + env_id: envId, + estimator: 'sd_vae_decode', + line, + measured_by: measuredBy, + measured_range: fitted + ? { max_pixels: Math.max(...countedSides) ** 2, min_pixels: Math.min(...countedSides) ** 2 } + : null, + own_windows: countedPoints.filter((point) => point.model_key === modelKey).length, + problem: fitted ? null : 'A line needs two sizes at least 1.5 times apart.', + slope_bytes_per_pixel: fitted ? 1_100 : null, + sizes, + state, + version: 1, + }, + ]; + }); + }; + + const unavailableReason = (scenario) => + scenario === 'multiGpu' + ? 'Calibration needs exactly one generation device; this machine generates on cuda:0 and cuda:1.' + : scenario === 'noGpu' + ? 'This machine has no GPU to calibrate.' + : scenario === 'storeUnusable' + ? `working_memory/${DEVICE_CLASS.id}.json has schema_version 9; this version reads up to 1.` + : null; + + const plan = (model, scenario) => { + const base = { + calibrated_through: null, + cases: [], + device_class: null, + estimated_seconds: null, + estimators: model.type === 'main' ? ['sd_vae_decode'] : [], + model_key: model.key, + not_covered: [], + reason: null, + repetitions: REPETITIONS, + shared_with: [], + supported: false, + }; + + if (model.type !== 'main') { + return { + ...base, + reason: `Calibration covers the decode of SD1, SD2, SDXL, SD3, FLUX.1, FLUX.2, Z-Image and CogView4 main models and their VAEs; no installed main model generates with this ${model.base} ${model.type}.`, + }; + } + + if (MODEL_SCENARIOS[model.key] === 'unsupported') { + return { ...base, reason: 'The inpaint variant does not generate from text alone.' }; + } + + if (scenario === 'multiGpu' || scenario === 'noGpu') { + return { ...base, reason: unavailableReason(scenario) }; + } + + const cases = casesOf(model); + const shared = mainModels.filter((other) => other.key !== model.key).slice(0, 3); + + return { + ...base, + cases, + device_class: { id: DEVICE_CLASS.id, name: DEVICE_CLASS.name }, + estimated_seconds: 30 + stepsOf(cases).length * 6, + not_covered: [ENCODE_NOT_COVERED], + shared_with: shared.map((other) => ({ + base: other.base, + key: other.key, + lines: DTYPES.map((dtype) => `decode_${dtype}`), + name: other.name, + type: other.type, + })), + supported: true, + }; + }; + + const status = (scenario) => { + const reason = unavailableReason(scenario); + const hasClass = scenario !== 'multiGpu' && scenario !== 'noGpu'; + const allRuns = [...runs.entries()].flatMap(([modelKey, list]) => + list.map((run) => { + const pointCount = pointCountOf(run); + + return { + attempt_id: run.attempt_id, + created_at: run.created_at, + env_id: ENV_ID, + item_ids: run.items.map((item) => item.item_id), + model_hash: byKey.get(modelKey)?.hash ?? null, + model_key: modelKey, + point_count: pointCount, + run_id: run.run_id, + user_id: 'system', + ...runStatus(run, pointCount), + }; + }) + ); + + return { + available: reason === null, + current_device_class_id: hasClass ? DEVICE_CLASS.id : null, + current_env_id: hasClass ? ENV_ID : null, + device_classes: hasClass + ? [ + { + device_class: scenario === 'storeUnusable' ? null : DEVICE_CLASS, + device_class_id: DEVICE_CLASS.id, + envs: + scenario === 'storeUnusable' + ? [] + : [ + { + hard: { + cuda: '12.8', + os: scenario === 'notVerified' ? 'linux' : 'windows', + torch: '2.7', + }, + hard_matches_current: true, + id: ENV_ID, + point_count: [...points.values()].flat().filter((point) => point.env_id === ENV_ID).length, + soft: { driver: '576.02' }, + soft_matches_current: true, + }, + ], + is_current: true, + ...measurement(scenario), + point_counts: [], + problem: scenario === 'storeUnusable' ? reason : null, + runs: scenario === 'storeUnusable' ? [] : allRuns, + usable: scenario !== 'storeUnusable', + }, + ] + : [], + unavailable_reason: reason, + }; + }; + + /** The measurement method of the device class; `notVerified` stands for a backend nobody checked it on. */ + const measurement = (scenario) => ({ + measurement_method: 'allocator_peak', + measurement_verified: scenario !== 'notVerified', + }); + + const summary = (model, scenario) => { + advance(model.key); + + return { + current_env_id: ENV_ID, + device_class_id: DEVICE_CLASS.id, + excluded_windows: 0, + lines: scenario === 'storeUnusable' ? [] : linesOf(model.key, MODEL_SCENARIOS[model.key]), + ...measurement(scenario), + model_key: model.key, + problem: scenario === 'storeUnusable' ? unavailableReason(scenario) : null, + runs: scenario === 'storeUnusable' ? [] : modelRuns(model.key), + usable: scenario !== 'storeUnusable', + }; + }; + + const calibrate = (model, body, scenario) => { + if (unavailableReason(scenario) !== null) { + return { body: { detail: unavailableReason(scenario) }, status: 409 }; + } + + if (typeof body?.graph?.nodes !== 'object' || body.graph.nodes === null) { + return { body: { detail: 'The graph cannot be measured as a calibration run.' }, status: 422 }; + } + + const attemptId = body.attempt_id ?? null; + + if (attemptId !== null && !(typeof attemptId === 'string' && ATTEMPT_ID_PATTERN.test(attemptId))) { + return { body: { detail: 'attempt_id must be 1 to 64 of A-Z a-z 0-9 _ . : -' }, status: 422 }; + } + + const key = typeof body.idempotency_key === 'string' ? body.idempotency_key : null; + const known = key ? receipts.get(key) : undefined; + const existing = known && (runs.get(model.key) ?? []).find((run) => run.run_id === known); + const caseId = typeof body.graph.id === 'string' ? body.graph.id : null; + const step = casesOf(model).find((item) => item.id === caseId) ?? casesOf(model)[0]; + const run = existing ?? createRun(model.key, [step], now(), attemptId); + + if (key) { + receipts.set(key, run.run_id); + } + + return { + body: { device_class_id: DEVICE_CLASS.id, item_ids: run.items.map((item) => item.item_id), run_id: run.run_id }, + status: 201, + }; + }; + + const cancelByOrigin = (originPrefix) => { + let canceled = 0; + + for (const list of runs.values()) { + for (const run of list) { + if (!`${ORIGIN_PREFIX}${run.run_id}`.startsWith(originPrefix)) { + continue; + } + + for (const item of run.items) { + if (item.status === 'pending' || item.status === 'in_progress') { + item.status = 'canceled'; + canceled += 1; + } + } + } + } + + return canceled; + }; + + const resetModel = (modelKey) => { + const removed = runs.get(modelKey) ?? []; + const pointsRemoved = (points.get(modelKey) ?? []).length; + + runs.delete(modelKey); + points.delete(modelKey); + + return { points_removed: pointsRemoved, runs_removed: removed.length }; + }; + + /** Answers a working-memory route, or returns null for any other path. */ + const handle = ({ body, method, path, scenario }) => { + if (path === '/api/v2/models/working_memory/status' && method === 'GET') { + return scenario === 'error' + ? { body: { detail: 'The calibration store cannot be read (fixture outage).' }, status: 503 } + : { body: status(scenario), status: 200 }; + } + + const match = /^\/api\/v2\/models\/i\/([^/]+)\/working_memory(?:\/(plan|summary|calibrate))?$/.exec(path); + + if (!match) { + return null; + } + + const model = byKey.get(decodeURIComponent(match[1])); + + if (!model) { + return { body: { detail: 'The model could not be found' }, status: 404 }; + } + + if (scenario === 'error') { + return { body: { detail: 'The calibration store cannot be read (fixture outage).' }, status: 503 }; + } + + const noDeviceClass = scenario === 'multiGpu' || scenario === 'noGpu'; + + switch (`${method} ${match[2] ?? ''}`) { + case 'GET plan': + return { body: plan(model, scenario), status: 200 }; + case 'GET summary': + return noDeviceClass + ? { body: { detail: 'No device_class_id was given and there is no current one' }, status: 409 } + : { body: summary(model, scenario), status: 200 }; + case 'POST calibrate': + return calibrate(model, body, scenario); + case 'DELETE ': + return noDeviceClass + ? { body: { detail: 'No device_class_id was given and there is no current one' }, status: 409 } + : { body: resetModel(model.key), status: 200 }; + default: + return { body: { detail: 'Method Not Allowed' }, status: 405 }; + } + }; + + return { cancelByOrigin, handle, isCalibrationPrefix: (prefix) => prefix.startsWith(ORIGIN_PREFIX), reset }; +}; diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.browser.test.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.browser.test.ts new file mode 100644 index 00000000000..3158852172e --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.browser.test.ts @@ -0,0 +1,118 @@ +import type { ModelConfig, WorkingMemoryCase } from '@features/models'; + +import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; +import { setModelsSnapshotForTests } from '@features/models/data/modelsStore'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +/** Serves the capability table; module mocks cannot be used here, since they would instrument the worker's modules. */ +const fetchCapabilities = () => + vi + .spyOn(globalThis, 'fetch') + .mockImplementation(() => Promise.resolve(Response.json(architectureCapabilitiesFixture))); + +const SDXL = { + base: 'sdxl', + format: 'diffusers', + hash: 'blake3:sdxl', + key: 'sdxl', + name: 'SDXL', + type: 'main', + variant: 'normal', +} as ModelConfig; + +const planCase = (id: string, modelKey = 'sdxl'): WorkingMemoryCase => ({ + components: {}, + estimator: 'sdxl_vae_decode', + height: 768, + id, + line: 'decode_fp32', + mode: 'txt2img', + model_key: modelKey, + negative_prompt: '', + positive_prompt: 'a lake', + seed: 1, + settings: { vaePrecision: 'fp32' }, + steps: 2, + width: 768, +}); + +afterEach(() => { + setModelsSnapshotForTests({ models: [], status: 'idle' }); + vi.restoreAllMocks(); +}); + +/** Runs the real worker: its bundle must start in a browser and compile without the page's modules. */ +describe('compileCalibrationGraphs', () => { + it('compiles the cases in a worker over the tab’s models and the served capability table', async () => { + const fetch = fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { compileCalibrationGraphs } = await import('./calibrationGraphs'); + + const [compiled] = await compileCalibrationGraphs([planCase('fp32-768')], new AbortController().signal); + const nodes = Object.values((compiled!.graph as { nodes: Record> }).nodes); + + expect(String(fetch.mock.calls[0]?.[0])).toMatch(/\/api\/v2\/models\/capabilities$/); + expect(compiled!.caseId).toBe('fp32-768'); + expect(nodes.find((node) => node.type === 'l2i')).toMatchObject({ fp32: true }); + expect(nodes.find((node) => node.type === 'noise')).toMatchObject({ height: 768, width: 768 }); + }); + + it('rejects with the compiler’s message for a case that cannot compile', async () => { + fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { compileCalibrationGraphs } = await import('./calibrationGraphs'); + + await expect( + compileCalibrationGraphs([planCase('lost', 'removed-model')], new AbortController().signal) + ).rejects.toThrow(/^Case lost: model_key "removed-model" is not a selectable main model/); + }); + + /** A worker that never answers: its request is dropped. Resolves once the request would have been sent. */ + const silenceWorker = () => { + const terminate = vi.spyOn(Worker.prototype, 'terminate'); + let sent = () => undefined as void; + const posted = new Promise((resolve) => { + sent = resolve; + }); + vi.spyOn(Worker.prototype, 'postMessage').mockImplementation(() => sent()); + + return { posted, terminate }; + }; + + it('stops the worker and rejects when its owner aborts', async () => { + fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { compileCalibrationGraphs } = await import('./calibrationGraphs'); + const { posted, terminate } = silenceWorker(); + const owner = new AbortController(); + + const compiling = compileCalibrationGraphs([planCase('fp32-768')], owner.signal); + await posted; + owner.abort(new Error('Calibration stopped.')); + + await expect(compiling).rejects.toThrow('Calibration stopped.'); + expect(terminate).toHaveBeenCalledTimes(1); + }); + + it('stops a worker that does not answer in time and rejects', async () => { + fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { COMPILE_TIMEOUT_MS, compileCalibrationGraphs } = await import('./calibrationGraphs'); + const { posted, terminate } = silenceWorker(); + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + + try { + const compiling = compileCalibrationGraphs([planCase('fp32-768')], new AbortController().signal); + const outcome = expect(compiling).rejects.toThrow(/did not answer within 60 s/); + await posted; + vi.advanceTimersByTime(COMPILE_TIMEOUT_MS - 1); + expect(terminate).not.toHaveBeenCalled(); + vi.advanceTimersByTime(1); + + await outcome; + expect(terminate).toHaveBeenCalledTimes(1); + } finally { + vi.useRealTimers(); + } + }); +}); diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.test.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.test.ts new file mode 100644 index 00000000000..d8bbb86828e --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.test.ts @@ -0,0 +1,86 @@ +import type { WorkingMemoryCase } from '@features/models'; + +import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; +import { describe, expect, it } from 'vitest'; + +import { toCalibrationGraphs, toGraphExportCase } from './calibrationGraphs'; +import { exportGraphs } from './graphExport'; + +const MODELS = [ + { + base: 'sdxl', + format: 'diffusers', + hash: 'blake3:sdxl', + key: 'sdxl', + name: 'SDXL', + type: 'main', + variant: 'normal', + }, +]; + +/** A case as the server's plan sends it for an SDXL decode in fp32. */ +const planCase = (overrides: Partial = {}): WorkingMemoryCase => ({ + components: {}, + estimator: 'sdxl_vae_decode', + height: 1088, + id: 'sdxl_vae_decode-decode-fp32-1088x1088', + line: 'decode_fp32', + mode: 'txt2img', + model_key: 'sdxl', + negative_prompt: '', + positive_prompt: 'a mountain lake at dawn, detailed photograph', + seed: 1, + settings: { vaePrecision: 'fp32' }, + steps: 2, + width: 1088, + ...overrides, +}); + +const compile = (cases: WorkingMemoryCase[]) => + toCalibrationGraphs( + cases, + exportGraphs({ + capabilities: architectureCapabilitiesFixture, + cases: cases.map(toGraphExportCase), + models: MODELS, + version: 1, + }) + ); + +const nodesOf = (graph: unknown) => Object.values((graph as { nodes: Record> }).nodes); + +describe('calibration graphs', () => { + it('compiles a plan case as Generate would, at its size, steps and decode precision', () => { + const [compiled] = compile([planCase()]); + const nodes = nodesOf(compiled!.graph); + + expect(compiled!.caseId).toBe('sdxl_vae_decode-decode-fp32-1088x1088'); + expect((compiled!.graph as { id: string }).id).toBe('sdxl_vae_decode-decode-fp32-1088x1088'); + expect(nodes.find((node) => node.type === 'noise')).toMatchObject({ height: 1088, width: 1088 }); + expect(nodes.find((node) => node.type === 'denoise_latents')).toMatchObject({ steps: 2 }); + expect(nodes.find((node) => node.type === 'l2i')).toMatchObject({ fp32: true }); + // Seed and prompt travel as batch data, as Generate queues them. + expect(compiled!.data).toEqual([ + [ + { field_name: 'value', items: [1], node_path: 'seed' }, + { field_name: 'value', items: ['a mountain lake at dawn, detailed photograph'], node_path: 'positive_prompt' }, + { field_name: 'value', items: [''], node_path: 'negative_prompt' }, + ], + ]); + }); + + it('keeps fp16 cases on the fp16 decode', () => { + const [compiled] = compile([ + planCase({ id: 'fp16-case', line: 'decode_fp16', settings: { vaePrecision: 'fp16' } }), + ]); + const nodes = nodesOf(compiled!.graph); + + expect(nodes.find((node) => node.type === 'l2i')).toMatchObject({ fp32: false }); + }); + + it('rejects naming the case that cannot compile, so nothing is queued', () => { + expect(() => compile([planCase(), planCase({ id: 'missing-model', model_key: 'not-installed' })])).toThrow( + /^Case missing-model: model_key "not-installed" is not a selectable main model/ + ); + }); +}); diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.ts new file mode 100644 index 00000000000..5d619f13693 --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.ts @@ -0,0 +1,109 @@ +/** + * The graphs a working-memory calibration queues, compiled exactly as `pnpm graphs:export` compiles its cases, which + * is how Generate compiles them: over the models this tab holds and the capability table the server serves. The + * Model Manager loads this module only when a calibration starts (`compileCalibrationGraphs` in `router.tsx`); the + * compiler itself runs in `calibrationGraphs.worker.ts`. + */ + +import { ensureModelsLoaded, getModelsSnapshot, type CalibrationGraph, type WorkingMemoryCase } from '@features/models'; +import { apiFetchJson } from '@platform/transport/http'; + +import type { CalibrationWorkerReply } from './calibrationGraphs.worker'; +import type { GraphExportResponse } from './graphExport'; + +const GRAPH_EXPORT_VERSION = 1; + +/** + * A plan case in the export's snake_case case format. The plan adds what the case measures (`estimator`, `line`), + * which the export reader rejects as unknown fields. + */ +export const toGraphExportCase = ({ + estimator: _estimator, + line: _line, + ...record +}: WorkingMemoryCase): Omit => record; + +/** Each case's graph and batch data; rejects naming the first case that cannot compile, before anything is queued. */ +export const toCalibrationGraphs = ( + cases: readonly WorkingMemoryCase[], + response: GraphExportResponse +): CalibrationGraph[] => + cases.map((item) => { + const result = response.cases.find((candidate) => candidate.id === item.id); + + if (!result) { + throw new Error(`Case ${item.id} was not compiled.`); + } + + if ('error' in result) { + throw new Error(`Case ${item.id}: ${result.error}`); + } + + return { caseId: item.id, data: result.batch.data, graph: result.batch.graph }; + }); + +/** Compiling a large plan takes seconds; a worker that has not answered by then is stuck. */ +export const COMPILE_TIMEOUT_MS = 60_000; + +/** Settles exactly once: with the worker's answer, or rejecting on a failure, the timeout or `signal`. */ +const runWorker = (request: unknown, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + signal.throwIfAborted(); + + const worker = new Worker(new URL('./calibrationGraphs.worker.ts', import.meta.url), { + name: 'calibration-graphs', + type: 'module', + }); + const settle = (error: Error | null, response?: GraphExportResponse) => { + worker.terminate(); + clearTimeout(timeout); + signal.removeEventListener('abort', onAbort); + + if (error) { + reject(error); + } else { + resolve(response!); + } + }; + const onAbort = () => settle(signal.reason instanceof Error ? signal.reason : new Error('Compiling was stopped.')); + const timeout = setTimeout( + () => settle(new Error(`The graph compiler did not answer within ${String(COMPILE_TIMEOUT_MS / 1_000)} s.`)), + COMPILE_TIMEOUT_MS + ); + + signal.addEventListener('abort', onAbort, { once: true }); + worker.addEventListener('message', (event: MessageEvent) => { + if (event.data.ok) { + settle(null, event.data.response); + } else { + settle(new Error(event.data.error)); + } + }); + worker.addEventListener('messageerror', () => settle(new Error('The graph compiler’s answer could not be read.'))); + worker.addEventListener('error', (event) => { + settle(new Error(event.message || 'The graph compiler did not start.')); + }); + worker.postMessage(request); + }); + +export const compileCalibrationGraphs = async ( + cases: readonly WorkingMemoryCase[], + signal: AbortSignal +): Promise => { + const [capabilities] = await Promise.all([ + apiFetchJson('/api/v2/models/capabilities', { signal }), + ensureModelsLoaded(), + ]); + const { error, models } = getModelsSnapshot(); + + if (models.length === 0) { + throw new Error(error ?? 'Models are not loaded.'); + } + + const response = await runWorker( + { capabilities, cases: cases.map(toGraphExportCase), models, version: GRAPH_EXPORT_VERSION }, + signal + ); + + return toCalibrationGraphs(cases, response); +}; diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.worker.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.worker.ts new file mode 100644 index 00000000000..f1dd3a477b3 --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.worker.ts @@ -0,0 +1,26 @@ +/** + * Compiles working-memory calibration cases with the offline graph export (`graphExport.ts`), in a worker. Its own + * bundle keeps Generate's compiler out of both routes' chunk graphs, and its own capability table cannot replace the + * one an open editor holds. + */ + +import { exportGraphs } from './graphExport'; + +export type CalibrationWorkerReply = + | { ok: true; response: ReturnType } + | { ok: false; error: string }; + +interface WorkerScope { + addEventListener(type: 'message', listener: (event: MessageEvent) => void): void; + postMessage(message: CalibrationWorkerReply): void; +} + +const scope = globalThis as unknown as WorkerScope; + +scope.addEventListener('message', (event) => { + try { + scope.postMessage({ ok: true, response: exportGraphs(event.data) }); + } catch (error) { + scope.postMessage({ error: error instanceof Error ? error.message : String(error), ok: false }); + } +}); diff --git a/invokeai/frontend/webv2/src/app/graphExport.test.ts b/invokeai/frontend/webv2/src/app/graphExport.test.ts new file mode 100644 index 00000000000..5ea045a5dfc --- /dev/null +++ b/invokeai/frontend/webv2/src/app/graphExport.test.ts @@ -0,0 +1,232 @@ +import type { BackendGraphContract, BackendInvocationContract } from '@features/generation/contracts'; + +import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; +import { generateGraphCases, satisfiedSettingsFor } from '@features/generation/core/graphCoverage.testing'; +import { withArchitectureCapabilities } from '@features/generation/graphExport'; +import { describe, expect, it } from 'vitest'; + +import { exportGraphs, GraphExportRequestError, type GraphExportCaseResult } from './graphExport'; + +const model = (key: string, base: string, type: string, extra: Record = {}) => ({ + base, + hash: `blake3:${key}`, + key, + name: `${key} name`, + type, + ...extra, +}); + +const MODELS = [ + model('sdxl', 'sdxl', 'main', { format: 'diffusers', variant: 'normal' }), + model('sdxl-lora', 'sdxl', 'lora', { format: 'lycoris' }), + model('sdxl-canny', 'sdxl', 'controlnet', { format: 'diffusers' }), + model('sdxl-ip-adapter', 'sdxl', 'ip_adapter', { format: 'invokeai' }), + model('z-image-gguf', 'z-image', 'main', { format: 'gguf_quantized' }), + model('qwen3-06b', 'any', 'qwen3_encoder', { format: 'checkpoint', variant: 'qwen3_06b' }), + model('qwen3-4b', 'any', 'qwen3_encoder', { format: 'checkpoint', variant: 'qwen3_4b' }), + model('qwen3-8b', 'any', 'qwen3_encoder', { format: 'checkpoint', variant: 'qwen3_8b' }), + model('qwen3-4b-gguf', 'any', 'qwen3_encoder', { format: 'gguf_quantized', variant: 'qwen3_4b' }), + model('sdxl-vae', 'sdxl', 'vae', { format: 'diffusers' }), + model('flux-vae', 'flux', 'vae', { format: 'checkpoint' }), +]; + +const sdxlCase = (id: string, overrides: Record = {}) => ({ + height: 832, + id, + mode: 'txt2img', + model_key: 'sdxl', + negative_prompt: 'blurry', + positive_prompt: 'a lighthouse at dusk', + seed: 4242, + steps: 12, + width: 1216, + ...overrides, +}); + +const request = (cases: unknown[], models: unknown[] = MODELS) => ({ + capabilities: architectureCapabilitiesFixture, + cases, + models, + version: 1, +}); + +const exportOne = (testCase: Record, models?: unknown[]) => { + const [result] = exportGraphs(request([testCase], models)).cases; + + if (!result || !('batch' in result)) { + throw new Error(`expected a batch, got ${JSON.stringify(result)}`); + } + + return result; +}; + +const nodesOfType = (graph: BackendGraphContract, type: string): BackendInvocationContract[] => + Object.values(graph.nodes).filter((node) => node.type === type); + +const errorOf = (result: GraphExportCaseResult | undefined): string => + result && 'error' in result ? result.error : ''; + +describe('graph export', () => { + it('sends the case through Generate: seed and prompts as batch data, size and steps in the graph', () => { + const { batch, id } = exportOne( + sdxlCase('sdxl-t2i', { loras: [{ key: 'sdxl-lora', weight: 0.65 }], reference_images: ['reference.png'] }) + ); + const { graph } = batch; + const noise = nodesOfType(graph, 'noise')[0]; + const denoise = graph.nodes.denoise_latents; + + expect(id).toBe('sdxl-t2i'); + expect(batch.data).toEqual([ + [ + { field_name: 'value', items: [4242], node_path: 'seed' }, + { field_name: 'value', items: ['a lighthouse at dusk'], node_path: 'positive_prompt' }, + { field_name: 'value', items: ['blurry'], node_path: 'negative_prompt' }, + ], + ]); + expect(batch.data[0]?.map((datum) => graph.nodes[datum.node_path]?.type)).toEqual(['integer', 'string', 'string']); + expect(batch).toMatchObject({ destination: 'gallery', origin: 'graphs-export:sdxl-t2i' }); + expect(graph.id).toBe('sdxl-t2i'); + expect([noise?.width, noise?.height, denoise?.steps]).toEqual([1216, 832, 12]); + expect( + nodesOfType(graph, 'lora_selector').map((node) => [(node.lora as { key: string }).key, node.weight]) + ).toEqual([['sdxl-lora', 0.65]]); + expect(nodesOfType(graph, 'ip_adapter')).toMatchObject([ + { image: { image_name: 'reference.png' }, ip_adapter_model: { key: 'sdxl-ip-adapter' } }, + ]); + }); + + it('sends img2img through Canvas with the uploaded image and strength', () => { + const { graph } = exportOne( + sdxlCase('sdxl-i2i', { init_image: { image_name: 'init.png', strength: 0.3 }, mode: 'img2img' }) + ).batch; + + expect(nodesOfType(graph, 'i2l').map((node) => node.image)).toEqual([{ image_name: 'init.png' }]); + expect(graph.nodes.denoise_latents?.denoising_start).toBeCloseTo(0.7); + expect(nodesOfType(graph, 'core_metadata')[0]).toMatchObject({ generation_mode: 'sdxl_img2img', strength: 0.3 }); + }); + + it('adds each control entry as a Canvas control layer feeding the denoiser', () => { + const { graph } = exportOne( + sdxlCase('sdxl-control', { + control: [{ begin: 0.1, end: 0.6, image_name: 'canny.png', model_key: 'sdxl-canny', weight: 0.9 }], + mode: 'controlnet', + }) + ).batch; + const [controlNet] = nodesOfType(graph, 'controlnet'); + + expect(controlNet).toMatchObject({ + begin_step_percent: 0.1, + control_model: { key: 'sdxl-canny' }, + control_weight: 0.9, + end_step_percent: 0.6, + image: { image_name: 'canny.png' }, + }); + expect(graph.edges).toContainEqual({ + destination: { field: 'control', node_id: 'denoise_latents' }, + source: { field: 'collection', node_id: 'control_net_collector' }, + }); + }); + + it('fills required components from the installed models and keeps explicit picks', () => { + const zImage = { ...sdxlCase('z-image'), model_key: 'z-image-gguf' }; + + // The 0.6B encoder is listed first but Z-Image cannot use it. + expect(exportOne(zImage).components).toEqual({ qwen3EncoderModel: 'qwen3-4b', vae: 'flux-vae' }); + expect(exportOne({ ...zImage, components: { qwen3EncoderModel: 'qwen3-4b-gguf' } }).components).toEqual({ + qwen3EncoderModel: 'qwen3-4b-gguf', + vae: 'flux-vae', + }); + }); + + it('rejects an installed component the slot does not offer', () => { + const [result] = exportGraphs( + request([sdxlCase('z-image', { components: { qwen3EncoderModel: 'qwen3-8b' }, model_key: 'z-image-gguf' })]) + ).cases; + + expect(errorOf(result)).toBe( + 'components.qwen3EncoderModel: "qwen3-8b" cannot be the Qwen3 Encoder of z-image-gguf name. ' + + 'Choices: qwen3-4b, qwen3-4b-gguf.' + ); + }); + + it.each(generateGraphCases)('compiles $label from exactly the components it needs', ({ base, shape }) => { + const { + filled, + model: main, + settings, + } = withArchitectureCapabilities(architectureCapabilitiesFixture, () => satisfiedSettingsFor(base, shape)); + const components = Object.fromEntries(filled.map((key) => [key, (settings[key] as { key: string }).key])); + const testCase = sdxlCase(base, { height: settings.height, model_key: main.key, width: settings.width }); + + expect(exportOne(testCase, [main, ...filled.map((key) => settings[key])]).components).toEqual(components); + }); + + it('applies settings overrides and rejects values Generate would not keep', () => { + const { graph } = exportOne(sdxlCase('cfg', { settings: { cfgScale: 3.5, scheduler: 'dpmpp_2m' } })).batch; + const [badPid, badScheduler] = exportGraphs( + request([ + sdxlCase('pid', { settings: { pidMode: 'sideways' } }), + sdxlCase('scheduler', { settings: { scheduler: 'not-a-scheduler' } }), + ]) + ).cases; + + expect(graph.nodes.denoise_latents).toMatchObject({ cfg_scale: 3.5, scheduler: 'dpmpp_2m' }); + expect(errorOf(badPid)).toMatch(/^settings\.pidMode: Generate does not keep "sideways"/); + expect(errorOf(badScheduler)).toMatch(/^settings\.scheduler: .* Options: .*euler_a/); + }); + + it('reports unknown names per case, with the valid choices, and still exports the other cases', () => { + const response = exportGraphs( + request([ + sdxlCase('unknown-setting', { settings: { cfgscale: 3 } }), + sdxlCase('unknown-slot', { components: { t5EncoderModel: 'qwen3-4b' }, model_key: 'z-image-gguf' }), + sdxlCase('unknown-component', { components: { vae: 'missing-vae' }, model_key: 'z-image-gguf' }), + sdxlCase('unknown-model', { model_key: 'missing' }), + sdxlCase('unknown-lora', { loras: [{ key: 'missing-lora', weight: 1 }] }), + sdxlCase('unknown-control', { + control: [{ begin: 0, end: 1, image_name: 'canny.png', model_key: 'sdxl-lora', weight: 1 }], + mode: 'controlnet', + }), + sdxlCase('ok'), + ]) + ); + const errors = Object.fromEntries(response.cases.map((result) => [result.id, errorOf(result)])); + + expect(errors['unknown-setting']).toMatch(/^Unknown settings key "cfgscale"\. Valid keys: .*\bcfgScale\b/); + expect(errors['unknown-slot']).toMatch( + /^Unknown component slot "t5EncoderModel" .* Valid slots: .*qwen3EncoderModel, vae/ + ); + expect(errors['unknown-component']).toBe( + 'components.vae: model key "missing-vae" is not installed. Choices: flux-vae.' + ); + expect(errors['unknown-model']).toBe( + 'model_key "missing" is not a selectable main model. Choices: sdxl, z-image-gguf.' + ); + expect(errors['unknown-lora']).toMatch(/^loras\[0\]: "missing-lora" .* Choices: sdxl-lora\.$/); + expect(errors['unknown-control']).toMatch(/^control\[0\]: "sdxl-lora" cannot drive .* Choices: sdxl-canny\.$/); + expect(response.cases.find((result) => result.id === 'ok')).toHaveProperty('batch'); + }); + + it('exports identical requests identically, with stable node ids', () => { + const cases = [ + sdxlCase('t2i', { loras: [{ key: 'sdxl-lora', weight: 1 }], reference_images: ['reference.png'] }), + sdxlCase('i2i', { init_image: { image_name: 'init.png', strength: 0.5 }, mode: 'img2img' }), + { ...sdxlCase('z-image'), model_key: 'z-image-gguf' }, + ]; + const first = exportGraphs(request(cases)); + const nodeIds = first.cases.flatMap((result) => ('batch' in result ? Object.keys(result.batch.graph.nodes) : [])); + + expect(exportGraphs(request(cases))).toEqual(first); + expect(nodeIds.filter((id) => /_[a-z0-9]{7,9}_[a-z0-9]{4,6}$/.test(id))).toEqual([]); + }); + + it.each([ + ['a non-object request', []], + ['another version', { ...request([]), version: 2 }], + ['models that are not model configs', request([], [{ key: 'only-a-key' }])], + ['cases without ids', request([{ mode: 'txt2img' }])], + ['duplicate case ids', request([sdxlCase('same'), sdxlCase('same')])], + ])('rejects %s as a malformed request', (_label, malformed) => { + expect(() => exportGraphs(malformed)).toThrow(GraphExportRequestError); + }); +}); diff --git a/invokeai/frontend/webv2/src/app/graphExport.ts b/invokeai/frontend/webv2/src/app/graphExport.ts new file mode 100644 index 00000000000..951736f7525 --- /dev/null +++ b/invokeai/frontend/webv2/src/app/graphExport.ts @@ -0,0 +1,247 @@ +/** + * Composition root of `pnpm graphs:export` (scripts/export-graphs.mjs): installed model configs, the capability table + * and test cases become the enqueue batches the UI would send, offline, so end-to-end runs exercise the graphs users + * submit. Each batch is what Generate or Canvas puts into `enqueue_batch` minus `runs`, `project_id` and + * `idempotency_key`, with the case seed held fixed. Generation and Queue own every rule; this module maps the wire + * format and supplies the Workbench defaults the widgets would. + */ + +import type { GeneratePromptBatchPlan } from '@features/queue/contracts'; + +import { + compileGraphExportCase, + withArchitectureCapabilities, + type ArchitectureCapabilitiesRow, + type CompiledGraphExportCase, + type GraphExportCase, + type GraphExportEnvironment, + type GraphExportMode, +} from '@features/generation/graphExport'; +import { planGenerateSubmissionBatch } from '@features/queue/contracts'; +import { resolveRandDeviceMetadata } from '@features/queue/devices'; +import { SEED_MAX } from '@platform/core/seed'; +import { DEFAULT_CANVAS_COMPOSITING } from '@workbench/widgets/canvas/invoke/canvasCompositing'; +import { DEFAULT_CANVAS_SCALING } from '@workbench/widgets/canvas/invoke/canvasScaling'; +import { DEFAULT_CANVAS_DENOISING_STRENGTH } from '@workbench/widgets/canvas/invoke/canvasStrength'; + +const GRAPH_EXPORT_VERSION = 1; + +/** Durable outputs: Canvas staging would mark them intermediate. */ +const DESTINATION = 'gallery'; +/** Not a webv2 origin, so no open editor adopts the items as its own submissions. */ +const ORIGIN_PREFIX = 'graphs-export:'; +/** `DEFAULT_PROJECT_SETTINGS` (workbench/settings/store.ts); importing it would load the whole settings runtime. */ +const PROJECT_SETTINGS = { useCpuNoise: true, useNsfwChecker: false, useWatermark: false }; +/** ROCm builds of torch report their GPUs as CUDA devices as well. */ +const GPU_DEVICE_OPTIONS = [{ device: 'cuda:0', name: 'GPU' }]; +const MODES: readonly GraphExportMode[] = ['txt2img', 'img2img', 'controlnet']; +const CASE_FIELDS = [ + 'components', + 'control', + 'height', + 'id', + 'init_image', + 'loras', + 'mode', + 'model_key', + 'negative_prompt', + 'positive_prompt', + 'reference_images', + 'seed', + 'settings', + 'steps', + 'width', +]; + +/** The request itself is unusable; no response is written. */ +export class GraphExportRequestError extends Error { + override name = 'GraphExportRequestError'; +} + +export interface GraphExportBatch { + data: GeneratePromptBatchPlan['data']; + destination: typeof DESTINATION; + graph: CompiledGraphExportCase['graph']; + origin: string; +} + +export type GraphExportCaseResult = + | { id: string; batch: GraphExportBatch; components: Record } + | { id: string; error: string }; + +export interface GraphExportResponse { + version: typeof GRAPH_EXPORT_VERSION; + cases: GraphExportCaseResult[]; +} + +type JsonRecord = Record; + +const isRecord = (value: unknown): value is JsonRecord => + typeof value === 'object' && value !== null && !Array.isArray(value); + +const fail = (message: string): never => { + throw new Error(message); +}; + +const readString = (record: JsonRecord, field: string, label = field): string => + typeof record[field] === 'string' ? record[field] : fail(`${label} must be a string.`); + +const readNumber = (record: JsonRecord, field: string, label = field): number => + typeof record[field] === 'number' && Number.isFinite(record[field]) + ? record[field] + : fail(`${label} must be a number.`); + +const readInteger = (record: JsonRecord, field: string, min: number, max = Number.MAX_SAFE_INTEGER): number => { + const value = record[field]; + + return typeof value === 'number' && Number.isInteger(value) && value >= min && value <= max + ? value + : fail(`${field} must be an integer from ${min} to ${max}.`); +}; + +/** Absent and null both read as empty. */ +const readList = (record: JsonRecord, field: string, read: (item: unknown, label: string) => T): T[] => { + const value = record[field] ?? []; + + return Array.isArray(value) + ? value.map((item, index) => read(item, `${field}[${index}]`)) + : fail(`${field} must be a list.`); +}; + +const readObject = (value: unknown, label: string): JsonRecord => + isRecord(value) ? value : fail(`${label} must be an object.`); + +const readCase = (record: JsonRecord): GraphExportCase => { + const unknownField = Object.keys(record).find((field) => !CASE_FIELDS.includes(field)); + + if (unknownField !== undefined) { + fail(`Unknown case field "${unknownField}". Valid fields: ${CASE_FIELDS.join(', ')}.`); + } + + const mode = record.mode; + const components = readObject(record.components ?? {}, 'components'); + const initImage = + record.init_image === undefined || record.init_image === null ? null : readObject(record.init_image, 'init_image'); + + return { + components: Object.fromEntries( + Object.keys(components).map((slot) => [slot, readString(components, slot, `components.${slot}`)]) + ), + control: readList(record, 'control', (item, label) => { + const layer = readObject(item, label); + + return { + begin: readNumber(layer, 'begin', `${label}.begin`), + end: readNumber(layer, 'end', `${label}.end`), + imageName: readString(layer, 'image_name', `${label}.image_name`), + modelKey: readString(layer, 'model_key', `${label}.model_key`), + weight: readNumber(layer, 'weight', `${label}.weight`), + }; + }), + height: readInteger(record, 'height', 1), + id: readString(record, 'id'), + initImage: initImage && { + imageName: readString(initImage, 'image_name', 'init_image.image_name'), + strength: readNumber(initImage, 'strength', 'init_image.strength'), + }, + loras: readList(record, 'loras', (item, label) => { + const lora = readObject(item, label); + + return { key: readString(lora, 'key', `${label}.key`), weight: readNumber(lora, 'weight', `${label}.weight`) }; + }), + mode: MODES.includes(mode as GraphExportMode) + ? (mode as GraphExportMode) + : fail(`mode must be one of ${MODES.join(', ')}.`), + modelKey: readString(record, 'model_key'), + negativePrompt: + record.negative_prompt === null || record.negative_prompt === undefined + ? '' + : readString(record, 'negative_prompt'), + positivePrompt: readString(record, 'positive_prompt'), + referenceImages: readList(record, 'reference_images', (item, label) => + typeof item === 'string' ? item : fail(`${label} must be an image name.`) + ), + seed: readInteger(record, 'seed', 0, SEED_MAX), + settings: readObject(record.settings ?? {}, 'settings'), + steps: readInteger(record, 'steps', 1), + width: readInteger(record, 'width', 1), + }; +}; + +const exportCase = (record: JsonRecord, env: GraphExportEnvironment): GraphExportCaseResult => { + const id = record.id as string; + + try { + const { components, graph, submission } = compileGraphExportCase(readCase(record), env); + const batch: GraphExportBatch = { + data: planGenerateSubmissionBatch(submission).data, + destination: DESTINATION, + graph, + origin: `${ORIGIN_PREFIX}${id}`, + }; + + return { batch, components, id }; + } catch (error) { + return { error: error instanceof Error ? error.message : String(error), id }; + } +}; + +const isModelConfig = (value: unknown): value is GraphExportEnvironment['models'][number] => + isRecord(value) && ['key', 'name', 'base', 'type'].every((field) => typeof value[field] === 'string'); + +const isCapabilitiesRow = (value: unknown): value is ArchitectureCapabilitiesRow => + isRecord(value) && typeof value.base === 'string' && isRecord(value.features); + +const readRequest = (request: unknown) => { + if (!isRecord(request)) { + throw new GraphExportRequestError('The request must be a JSON object.'); + } + + if (request.version !== GRAPH_EXPORT_VERSION) { + throw new GraphExportRequestError(`Unsupported request version ${JSON.stringify(request.version)}; expected 1.`); + } + + const { capabilities, cases, models } = request; + + if (!Array.isArray(models) || !models.every(isModelConfig)) { + throw new GraphExportRequestError('models must be the `models` list of GET /api/v2/models/.'); + } + + if (!Array.isArray(capabilities) || !capabilities.every(isCapabilitiesRow)) { + throw new GraphExportRequestError('capabilities must be the body of GET /api/v2/models/capabilities.'); + } + + if (!Array.isArray(cases) || !cases.every((item) => isRecord(item) && typeof item.id === 'string' && item.id)) { + throw new GraphExportRequestError('cases must be a list of objects with a non-empty string id.'); + } + + const ids = cases.map((item) => item.id as string); + const duplicate = ids.find((id, index) => ids.indexOf(id) !== index); + + if (duplicate !== undefined) { + throw new GraphExportRequestError(`Case id "${duplicate}" is not unique.`); + } + + return { capabilities, cases: cases as JsonRecord[], models }; +}; + +/** Throws GraphExportRequestError for a malformed request; a case that cannot compile carries its error instead. */ +export const exportGraphs = (request: unknown): GraphExportResponse => { + const { capabilities, cases, models } = readRequest(request); + const env: GraphExportEnvironment = { + canvas: { + compositing: DEFAULT_CANVAS_COMPOSITING, + scaling: DEFAULT_CANVAS_SCALING, + strength: DEFAULT_CANVAS_DENOISING_STRENGTH, + }, + destination: DESTINATION, + models, + projectSettings: PROJECT_SETTINGS, + randDevice: resolveRandDeviceMetadata(PROJECT_SETTINGS.useCpuNoise, GPU_DEVICE_OPTIONS), + }; + + return withArchitectureCapabilities(capabilities, () => ({ + cases: cases.map((record) => exportCase(record, env)), + version: GRAPH_EXPORT_VERSION, + })); +}; diff --git a/invokeai/frontend/webv2/src/app/router.tsx b/invokeai/frontend/webv2/src/app/router.tsx index 2d2b4acd503..53860a20ccc 100644 --- a/invokeai/frontend/webv2/src/app/router.tsx +++ b/invokeai/frontend/webv2/src/app/router.tsx @@ -9,7 +9,7 @@ import { useAuthSession, type Capabilities, } from '@features/identity'; -import { ModelInstallRuntime } from '@features/models'; +import { CalibrationGraphCompilerProvider, ModelInstallRuntime, type CalibrationGraphCompiler } from '@features/models'; import { createLogger } from '@platform/logging/logger'; import { createHashHistory, @@ -118,8 +118,18 @@ const requireLaunchpadCapability = async (capability: keyof Capabilities): Promi } }; +/** Loads Generate's compiler only when the Model Manager starts a working-memory calibration. */ +const compileCalibrationGraphs: CalibrationGraphCompiler = (cases, signal) => + import('./calibrationGraphs').then((module) => module.compileCalibrationGraphs(cases, signal)); + +const LaunchpadRoute = () => ( + + + +); + const launchpadRouteOptions = { - component: Launchpad, + component: LaunchpadRoute, getParentRoute: () => authenticatedRoute, }; diff --git a/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts b/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts index 812832d1ce7..60e477c547b 100644 --- a/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts +++ b/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts @@ -235,6 +235,7 @@ describe('feature public-interface registry', () => { ]); expect(checkDependency('workbench/invocationSubmit.ts', '@features/generation/prompts')).toEqual([]); expect(checkDependency('app/GenerateWidgetSyncRuntime.tsx', '@features/generation/runtime')).toEqual([]); + expect(checkDependency('app/graphExport.ts', '@features/generation/graphExport')).toEqual([]); expect(checkDependency('workbench/widgets/canvas/invoke.ts', '@features/generation/canvasGraph')).toEqual([]); expect(checkDependency('workbench/WorkbenchContext.tsx', '@features/intermediates/holdLease')).toEqual([]); expect( diff --git a/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts b/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts index 9405bcfd449..c5fb3a9b00d 100644 --- a/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts +++ b/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts @@ -22,6 +22,7 @@ export const FEATURE_PUBLIC_INTERFACES: Readonly ({ +export const getComponentPolicyContext = ( + model: GenerateModelConfig, + settings: GenerateSettings +): ComponentPolicyContext => ({ model, settings, selectedComponents: { @@ -1235,6 +1238,9 @@ const COMPONENT_SETTING_LABELS: Record = { componentSourceModel: 'Component source', }; +/** Every settings key that holds a component model. */ +export const GENERATE_COMPONENT_VALUE_KEYS = Object.keys(COMPONENT_SETTING_LABELS) as GenerateComponentValueKey[]; + const addClearedLabel = (labels: string[], label: string) => { if (!labels.includes(label)) { labels.push(label); @@ -1509,7 +1515,7 @@ const getSettingsWithCompatibleModelSelections = ( const slotsByKey = new Map(policy.slots.map((slotPolicy) => [slotPolicy.key, slotPolicy])); const uiPolicy = getGenerationUiPolicy(model); - for (const key of Object.keys(COMPONENT_SETTING_LABELS) as GenerateComponentValueKey[]) { + for (const key of GENERATE_COMPONENT_VALUE_KEYS) { if (!nextSettings[key]) { continue; } @@ -1619,7 +1625,7 @@ export const getGenerationModelAvailabilityReasons = ( reasons.push(`Selected model "${model.name}" is no longer installed.`); } - for (const key of Object.keys(COMPONENT_SETTING_LABELS) as GenerateComponentValueKey[]) { + for (const key of GENERATE_COMPONENT_VALUE_KEYS) { const value = settings[key]; if (value && !hasModelKey(models, value.key, value.type)) { diff --git a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts index 38d2ee38047..f1a513880ed 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts @@ -6,9 +6,11 @@ import { hasDynamicPromptSyntax, matchesKnownWildcard, normalizeWildcardValues, + planGenerateSubmission, scanWildcardReferences, sanitizeDynamicPromptsConfig, sanitizeMaxPrompts, + type GenerateSubmissionSource, } from './dynamicPrompts'; describe('hasDynamicPromptSyntax', () => { @@ -239,3 +241,70 @@ describe('getWildcardValuesError', () => { expect(getWildcardValuesError(['x'.repeat(2_000)])).toBeNull(); }); }); + +const source = (overrides: Partial = {}): GenerateSubmissionSource => ({ + batchCount: 2, + negativePrompt: 'blurry', + negativePromptEnabled: true, + positivePrompt: 'a cat', + promptTemplate: null, + seed: 10, + seedMode: 'increment', + ...overrides, +}); + +describe('planGenerateSubmission', () => { + it('merges the template and suppresses its negative side with the negative prompt', () => { + const promptTemplate = { + id: 't', + name: 'Photo', + negativePrompt: '{prompt}, lowres', + positivePrompt: 'photo of {prompt}', + }; + + expect(planGenerateSubmission(source({ promptTemplate })).submission).toMatchObject({ + negativePrompt: 'blurry, lowres', + positivePrompt: 'photo of a cat', + }); + + const suppressed = planGenerateSubmission(source({ negativePromptEnabled: false, promptTemplate })); + + expect(suppressed.submission.negativePrompt).toBe(''); + expect(suppressed.effectivePrompts.negativePrompt).toBe('blurry, lowres'); + }); + + it('sends an expansion and its seed behaviour only for a prompt with dynamic syntax', () => { + const expansion = ['a red cat', 'a green cat']; + const expanded = planGenerateSubmission( + source({ dynamicPromptsSeedBehaviour: 'per-image', positivePrompt: 'a {red|green} cat' }), + expansion + ); + const plain = planGenerateSubmission(source({ dynamicPromptsSeedBehaviour: 'per-image' }), expansion); + + expect(expanded.submission).toEqual({ + batchCount: 2, + negativePrompt: 'blurry', + positivePrompt: 'a {red|green} cat', + positivePrompts: expansion, + seed: 10, + seedBehaviour: 'per-image', + seedStep: 1, + }); + // Two prompts per image across two runs consume four seeds. + expect(expanded.seedPlan.nextSeed).toBe(14); + expect(plain.submission).toEqual({ + batchCount: 2, + negativePrompt: 'blurry', + positivePrompt: 'a cat', + seed: 10, + seedStep: 1, + }); + expect(plain.seedPlan.nextSeed).toBe(12); + }); + + it('holds a fixed seed', () => { + const { seedPlan, submission } = planGenerateSubmission(source({ seedMode: 'fixed' })); + + expect([submission.seed, submission.seedStep, seedPlan.nextSeed]).toEqual([10, 0, null]); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts index 3512f3c45e7..52967295237 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts @@ -1,3 +1,9 @@ +import { planSeedSubmission, type SeedMode, type SeedStep, type SeedSubmissionPlan } from '@platform/core/seed'; + +import type { EffectivePrompts, PromptTemplateSnapshot } from './promptTemplates'; + +import { getEffectivePrompts } from './promptTemplates'; + const SEED_MAX = 4_294_967_295; export const DYNAMIC_PROMPTS_MIN_PROMPTS = 1; @@ -302,3 +308,75 @@ export const sanitizeSampleSeed = (value: unknown): number => typeof value === 'number' && Number.isFinite(value) ? Math.abs(Math.round(value)) % SEED_MAX : 0; export const createDynamicPromptsSampleSeed = (): number => Math.floor(Math.random() * SEED_MAX); + +/** The prompted, seeded values Generate, Canvas, Upscale and Video submit from. */ +export interface GenerateSubmissionSource { + batchCount: number; + negativePrompt: string; + negativePromptEnabled: boolean; + positivePrompt: string; + promptTemplate?: PromptTemplateSnapshot | null; + seed: number; + seedMode: SeedMode; + dynamicPromptsSeedBehaviour?: DynamicPromptsSeedBehaviour; +} + +/** The prompt and seed fields of a queued generate submission; the host adds the graph and its node ids. */ +export interface GenerateSubmissionPrompts { + batchCount: number; + negativePrompt: string; + positivePrompt: string; + /** Present when the positive prompt expanded; the expansion is sent instead of the prompt. */ + positivePrompts?: string[]; + seed: number; + seedBehaviour?: DynamicPromptsSeedBehaviour; + seedStep: SeedStep; +} + +export interface GenerateSubmissionPlan { + /** Template-merged prompts, before negative-prompt suppression. */ + effectivePrompts: EffectivePrompts; + seedPlan: SeedSubmissionPlan; + submission: GenerateSubmissionPrompts; +} + +/** + * The prompts and seeds a submission sends. `expandedPositivePrompts` are what the host expanded the merged positive + * prompt into; they count only when that prompt has dynamic syntax, so templates that introduce syntax count too. + */ +export const planGenerateSubmission = ( + source: GenerateSubmissionSource, + expandedPositivePrompts?: string[] +): GenerateSubmissionPlan => { + const effectivePrompts = getEffectivePrompts(source); + // Record expansions even when only one prompt results. + const positivePrompts = + expandedPositivePrompts && + expandedPositivePrompts.length > 0 && + hasDynamicPromptSyntax(effectivePrompts.positivePrompt) + ? expandedPositivePrompts + : undefined; + const seedBehaviour = positivePrompts ? source.dynamicPromptsSeedBehaviour : undefined; + const seedPlan = planSeedSubmission({ + batchCount: source.batchCount, + promptCount: positivePrompts?.length ?? 1, + seedBehaviour: seedBehaviour ?? 'per-iteration', + seedMode: source.seedMode, + startSeed: source.seed, + }); + + return { + effectivePrompts, + seedPlan, + submission: { + batchCount: source.batchCount, + // Disabling the negative prompt also suppresses the template's negative side. + negativePrompt: source.negativePromptEnabled ? effectivePrompts.negativePrompt : '', + positivePrompt: effectivePrompts.positivePrompt, + ...(positivePrompts ? { positivePrompts } : {}), + seed: source.seed, + ...(seedBehaviour ? { seedBehaviour } : {}), + seedStep: seedPlan.step, + }, + }; +}; diff --git a/invokeai/frontend/webv2/src/features/generation/core/graphExport.ts b/invokeai/frontend/webv2/src/features/generation/core/graphExport.ts new file mode 100644 index 00000000000..72ae821403a --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/core/graphExport.ts @@ -0,0 +1,492 @@ +/** + * Offline export cases compile to the graph Generate or Canvas would submit: the model's defaults, the case's values + * and component picks, required components filled from the installed models the way a user fills them, and the same + * reconciliation and compilers the widget runs. Callers own the wire format and the Workbench canvas defaults. + */ + +import type { GenerationModelCatalogItem as ModelConfig } from '@features/generation/core/contracts'; + +import { areJsonValuesStructurallyEqual } from '@platform/core/json'; + +import type { ArchitectureCapabilitiesRow } from './architectureCapabilities'; +import type { ComponentPolicyContext, ComponentSlotPolicy } from './baseGenerationPolicies'; +import type { ControlLayerGraphInput } from './canvas/addControlLayers'; +import type { CanvasCompositingSettings, CanvasScalingSettings } from './canvas/types'; +import type { BackendGraphContract, GenerationProjectSettings, ResultDestination } from './contracts'; +import type { ComponentModelConfig, GenerateModelConfig, GenerateWidgetValues } from './types'; + +import { resetArchitectureCapabilities, setArchitectureCapabilities } from './architectureCapabilities'; +import { + coerceSchedulerForGraph, + GENERATE_COMPONENT_VALUE_KEYS, + getComponentPolicyContext, + getComponentSectionPolicy, + getDefaultGenerateSettings, + getDefaultReferenceImageConfig, + getGenerateModelSelectionResult, + getSchedulerOptions, + isArchitectureDescribed, + isGenerateModelSelectable, + isLoraSupported, +} from './baseGenerationPolicies'; +import { compileCanvasGraph } from './canvas/compileCanvasGraph'; +import { + CONTROL_ADAPTER_KINDS, + isControlKindSupportedForBase, + isControlModelUsableForKind, +} from './canvas/controlValidation'; +import { hasDynamicPromptSyntax, planGenerateSubmission, type GenerateSubmissionPrompts } from './dynamicPrompts'; +import { compileGenerateGraph } from './graph'; +import { stabilizeBackendGraphNodeIds } from './previewGraph'; +import { resolveGenerateWidgetValues } from './resolveGenerateWidgetValues'; +import { + deriveAspectRatioId, + GENERATE_UI_STATE_KEYS, + getModelDefaultVae, + hasModelDefaultVae, + isLoraCompatibleWithModel, + isLoraModelConfig, + isMainModelConfig, + isModelIdentifierConfig, + isVaeModelConfig, +} from './settings'; + +/** `txt2img` compiles like Generate; `img2img` and `controlnet` compile like a full-frame Canvas invoke. */ +export type GraphExportMode = 'txt2img' | 'img2img' | 'controlnet'; + +export interface GraphExportCase { + /** Becomes the graph id. */ + id: string; + mode: GraphExportMode; + modelKey: string; + /** Picks keyed by component slot; required slots left out are filled from the installed models. */ + components: Readonly>; + width: number; + height: number; + steps: number; + seed: number; + positivePrompt: string; + negativePrompt: string; + loras: readonly { key: string; weight: number }[]; + /** Uploaded image names. */ + referenceImages: readonly string[]; + /** An uploaded image already sized width × height, as the Canvas composite always is. */ + initImage: { imageName: string; strength: number } | null; + /** Each image is an uploaded width × height control composite. */ + control: readonly { modelKey: string; imageName: string; weight: number; begin: number; end: number }[]; + /** Generate settings overrides. Keys the case sets through its own fields are rejected. */ + settings: Readonly>; +} + +export interface GraphExportEnvironment { + models: readonly ModelConfig[]; + destination: ResultDestination; + projectSettings: GenerationProjectSettings; + randDevice: string; + /** What the Canvas widget supplies from its persisted values. */ + canvas: { compositing: CanvasCompositingSettings; scaling: CanvasScalingSettings; strength: number }; +} + +export interface CompiledGraphExportCase { + /** Node ids are stabilized: identical cases compile to identical graphs. */ + graph: BackendGraphContract; + /** The key of every component model the case compiled with, by settings key. */ + components: Record; + /** What Generate queues for the graph, with the prompt and seed node ids in `graph`. */ + submission: GenerateSubmissionPrompts & { + negativePromptNodeId: string; + positivePromptNodeId: string; + seedNodeId: string; + }; +} + +/** Keys a case sets through its own fields, picks as components, or that need backend catalogs or expansion. */ +const NON_OVERRIDABLE_SETTINGS: ReadonlySet = new Set([ + 'aspectRatioId', + 'aspectRatioValue', + 'dynamicPromptsCombinatorial', + 'dynamicPromptsMaxPrompts', + 'dynamicPromptsResample', + 'dynamicPromptsSampleSeed', + 'dynamicPromptsSeedBehaviour', + 'height', + 'loras', + 'model', + 'modelKey', + 'negativePrompt', + 'positivePrompt', + 'promptTemplate', + 'referenceImages', + 'seed', + 'seedMode', + 'steps', + 'width', + ...GENERATE_COMPONENT_VALUE_KEYS, + ...Object.keys(GENERATE_UI_STATE_KEYS), +]); + +const listKeys = (keys: readonly string[]): string => (keys.length > 0 ? keys.join(', ') : 'none'); + +const listModelKeys = (models: readonly ModelConfig[]): string => listKeys(models.map((model) => model.key)); + +/** Hold `rows` as the capability table, which every Generation policy reads, while `run` compiles. */ +export const withArchitectureCapabilities = (rows: readonly ArchitectureCapabilitiesRow[], run: () => T): T => { + setArchitectureCapabilities(rows); + + try { + return run(); + } finally { + resetArchitectureCapabilities(); + } +}; + +const getSelectableModel = (key: string, models: readonly ModelConfig[]): GenerateModelConfig => { + const selectable = models.filter(isGenerateModelSelectable).filter(isArchitectureDescribed); + const model = selectable.find((candidate) => candidate.key === key); + + if (!model) { + throw new Error(`model_key "${key}" is not a selectable main model. Choices: ${listModelKeys(selectable)}.`); + } + + return model; +}; + +/** Selecting the model, then "Reset to model defaults", which also applies the model's default VAE. */ +const getModelDefaultValues = (model: GenerateModelConfig, models: readonly ModelConfig[]): GenerateWidgetValues => { + const { settings } = getGenerateModelSelectionResult({ + currentValues: getDefaultGenerateSettings(model), + model, + models, + }); + const vae = hasModelDefaultVae(model) ? getModelDefaultVae(model, models.filter(isVaeModelConfig)) : settings.vae; + + return { ...settings, model, vae }; +}; + +const withCaseValues = (values: GenerateWidgetValues, input: GraphExportCase): GenerateWidgetValues => ({ + ...values, + aspectRatioId: deriveAspectRatioId(input.width, input.height), + aspectRatioValue: input.width / input.height, + height: input.height, + negativePrompt: input.negativePrompt, + positivePrompt: input.positivePrompt, + seed: input.seed, + seedMode: 'fixed', + steps: input.steps, + width: input.width, +}); + +const withSettingsOverrides = ( + values: GenerateWidgetValues, + overrides: GraphExportCase['settings'] +): GenerateWidgetValues => { + const validKeys = Object.keys(values) + .filter((key) => !NON_OVERRIDABLE_SETTINGS.has(key)) + .sort(); + const unknownKey = Object.keys(overrides).find((key) => !validKeys.includes(key)); + + if (unknownKey !== undefined) { + throw new Error(`Unknown settings key "${unknownKey}". Valid keys: ${listKeys(validKeys)}.`); + } + + return { ...values, ...overrides }; +}; + +const fitsSlot = (slot: ComponentSlotPolicy, candidate: ModelConfig, ctx: ComponentPolicyContext): boolean => + (slot.modelTypes as readonly string[]).includes(candidate.type) && (!slot.filter || slot.filter(candidate, ctx)); + +/** The value the component picker commits for `model`, by the slot's value kind. */ +const toSlotValue = (slot: ComponentSlotPolicy, model: ModelConfig): ComponentModelConfig | null => { + switch (slot.valueKind) { + case 'main': + return isMainModelConfig(model) ? model : null; + case 'vae': + return isVaeModelConfig(model) ? model : null; + case 'component': + return isModelIdentifierConfig(model) ? model : null; + } +}; + +const withComponentPicks = ( + values: GenerateWidgetValues, + picks: GraphExportCase['components'], + models: readonly ModelConfig[] +): GenerateWidgetValues => { + const { slots } = getComponentSectionPolicy(values.model, values); + let next = values; + + for (const [slotKey, modelKey] of Object.entries(picks)) { + const slot = slots.find((candidate) => candidate.key === slotKey); + + if (!slot) { + const validSlots = listKeys(slots.map((candidate) => candidate.key)); + + throw new Error(`Unknown component slot "${slotKey}" for ${values.model.name}. Valid slots: ${validSlots}.`); + } + + const ctx = getComponentPolicyContext(next.model, next); + const fitting = models.filter((candidate) => fitsSlot(slot, candidate, ctx)); + const pick = fitting.find((candidate) => candidate.key === modelKey); + const value = pick ? toSlotValue(slot, pick) : null; + + if (!value) { + const problem = models.some((candidate) => candidate.key === modelKey) + ? `"${modelKey}" cannot be the ${slot.label} of ${values.model.name}` + : `model key "${modelKey}" is not installed`; + + throw new Error(`components.${slotKey}: ${problem}. Choices: ${listModelKeys(fitting)}.`); + } + + next = { ...next, [slot.key]: value }; + } + + return next; +}; + +/** Pick the first installed model for the first required slot that is unfilled; null when none can be filled. */ +const withFirstRequiredComponent = ( + values: GenerateWidgetValues, + models: readonly ModelConfig[] +): GenerateWidgetValues | null => { + const ctx = getComponentPolicyContext(values.model, values); + + for (const slot of getComponentSectionPolicy(values.model, values).slots) { + const current = ctx.selectedComponents[slot.key]; + + if (!slot.required?.(ctx) || (current && (!slot.filter || slot.filter(current as ModelConfig, ctx)))) { + continue; + } + + for (const candidate of models) { + const value = fitsSlot(slot, candidate, ctx) ? toSlotValue(slot, candidate) : null; + + if (value) { + return { ...values, [slot.key]: value }; + } + } + } + + return null; +}; + +const withLoras = ( + values: GenerateWidgetValues, + loras: GraphExportCase['loras'], + models: readonly ModelConfig[] +): GenerateWidgetValues => { + if (loras.length === 0) { + return values; + } + + const { model } = values; + + if (!isLoraSupported(model)) { + throw new Error(`${model.name} does not load LoRAs.`); + } + + const compatible = models.filter(isLoraModelConfig).filter((lora) => isLoraCompatibleWithModel(lora, model)); + + return { + ...values, + loras: loras.map(({ key, weight }, index) => { + const lora = compatible.find((candidate) => candidate.key === key); + + if (!lora) { + const choices = listModelKeys(compatible); + + throw new Error(`loras[${index}]: "${key}" is not an installed LoRA for ${model.name}. Choices: ${choices}.`); + } + + return { isEnabled: true, model: lora, weight }; + }), + }; +}; + +const withReferenceImages = ( + values: GenerateWidgetValues, + input: GraphExportCase, + models: readonly ModelConfig[] +): GenerateWidgetValues => ({ + ...values, + referenceImages: input.referenceImages.map((imageName, index) => ({ + // Cases name images only; the output size stands in for dimensions that reach metadata, not inference. + config: getDefaultReferenceImageConfig(values.model, models, { + original: { image: { height: input.height, image_name: imageName, width: input.width } }, + }), + id: `reference_image_${index + 1}`, + isEnabled: true, + })), +}); + +/** The widget reconciles after every commit; reconciliation can pick a FLUX.2 component source. */ +const reconcile = (values: GenerateWidgetValues, models: readonly ModelConfig[]): GenerateWidgetValues => { + const resolved = resolveGenerateWidgetValues({ models, storedValues: values }); + + if (!resolved) { + throw new Error('Generate cannot resolve settings without the capability table.'); + } + + return resolved.values; +}; + +/** Report a requested value the widget would not keep instead of compiling a graph nobody asked for. */ +const assertRequestedValuesKept = (values: GenerateWidgetValues, input: GraphExportCase): void => { + for (const [key, requested] of Object.entries(input.settings)) { + const kept = values[key as keyof GenerateWidgetValues]; + + if (!areJsonValuesStructurallyEqual(kept, requested)) { + throw new Error( + `settings.${key}: Generate does not keep ${JSON.stringify(requested)}; it holds ${JSON.stringify(kept)}.` + ); + } + } + + if ('scheduler' in input.settings && coerceSchedulerForGraph(values.model, values.scheduler) !== values.scheduler) { + const options = getSchedulerOptions(values.model).map((option) => option.value); + + throw new Error( + `settings.scheduler: ${values.model.name} does not use "${values.scheduler}". Options: ${listKeys(options)}.` + ); + } + + for (const [key, requested] of Object.entries(input.components)) { + const kept = values[key as keyof GenerateWidgetValues] as ComponentModelConfig | null; + + if (kept?.key !== requested) { + throw new Error( + `components.${key}: Generate replaces "${requested}" with ${kept ? `"${kept.key}"` : 'nothing'}.` + ); + } + } +}; + +const getControlLayers = ( + input: GraphExportCase, + model: GenerateModelConfig, + models: readonly ModelConfig[] +): ControlLayerGraphInput[] => { + // A layer's kind is the one its model can drive on this base, the kind a user switches the layer to. + const getKind = (candidate: ModelConfig) => + candidate.base === model.base + ? CONTROL_ADAPTER_KINDS.find( + (kind) => isControlKindSupportedForBase(model.base, kind) && isControlModelUsableForKind(candidate, kind) + ) + : undefined; + + return input.control.map((layer, index) => { + const adapter = models.find((candidate) => candidate.key === layer.modelKey); + const kind = adapter ? getKind(adapter) : undefined; + + if (!adapter || !kind) { + const choices = listModelKeys(models.filter((candidate) => getKind(candidate) !== undefined)); + const problem = `"${layer.modelKey}" cannot drive a control layer of ${model.name}`; + + throw new Error(`control[${index}]: ${problem}. Choices: ${choices}.`); + } + + return { + beginEndStepPct: [layer.begin, layer.end], + // A new ControlNet layer starts "balanced", which the compiler sends for null; other kinds carry no mode. + controlMode: null, + id: `control_${index + 1}`, + imageName: layer.imageName, + kind, + model: { + base: adapter.base, + key: adapter.key, + name: adapter.name, + type: adapter.type, + ...(typeof adapter.hash === 'string' ? { hash: adapter.hash } : {}), + }, + modelCondInChannels: typeof adapter.cond_in_channels === 'number' ? adapter.cond_in_channels : null, + weight: layer.weight, + }; + }); +}; + +const assertModeInputs = (input: GraphExportCase): void => { + if (input.mode === 'txt2img' && (input.initImage || input.control.length > 0)) { + throw new Error('txt2img cases take no init_image or control; use img2img or controlnet.'); + } + + if (input.mode === 'img2img' && !input.initImage) { + throw new Error('img2img cases need init_image.'); + } + + if (input.mode === 'controlnet' && input.control.length === 0) { + throw new Error('controlnet cases need at least one control entry.'); + } +}; + +/** Throws a message naming the offending field and its valid choices when the case cannot compile. */ +export const compileGraphExportCase = ( + input: GraphExportCase, + env: GraphExportEnvironment +): CompiledGraphExportCase => { + assertModeInputs(input); + + const { models } = env; + let values = withCaseValues(getModelDefaultValues(getSelectableModel(input.modelKey, models), models), input); + values = withSettingsOverrides(values, input.settings); + values = withComponentPicks(values, input.components, models); + values = withLoras(values, input.loras, models); + values = reconcile(withReferenceImages(values, input, models), models); + + // Filling one slot can satisfy or require another; the bound stops a reconciliation that keeps undoing a pick. + for (let pass = 0; pass < GENERATE_COMPONENT_VALUE_KEYS.length; pass += 1) { + const filled = withFirstRequiredComponent(values, models); + + if (!filled) { + break; + } + + values = reconcile(filled, models); + } + + assertRequestedValuesKept(values, input); + + // No expansion: the UI expands dynamic prompts through the backend before it queues. + const { effectivePrompts, submission } = planGenerateSubmission(values); + + if (hasDynamicPromptSyntax(effectivePrompts.positivePrompt)) { + throw new Error( + 'positive_prompt has dynamic prompt syntax, which only the backend expands; export each expansion.' + ); + } + + const compiled = + input.mode === 'txt2img' + ? compileGenerateGraph(values, values.model, env.destination, env.projectSettings, env.randDevice) + : compileCanvasGraph({ + bbox: { height: input.height, width: input.width, x: 0, y: 0 }, + compositeImageName: input.initImage?.imageName ?? null, + compositing: env.canvas.compositing, + controlLayers: getControlLayers(input, values.model, models), + destination: env.destination, + mode: input.initImage ? 'img2img' : 'txt2img', + model: values.model, + projectSettings: env.projectSettings, + randDevice: env.randDevice, + scaling: env.canvas.scaling, + settings: values, + strength: input.initImage?.strength ?? env.canvas.strength, + }); + + const stable = stabilizeBackendGraphNodeIds(compiled.backendGraph, { + negativePromptNodeId: compiled.negativePromptNodeId, + positivePromptNodeId: compiled.positivePromptNodeId, + seedNodeId: compiled.seedNodeId, + }); + + return { + components: Object.fromEntries( + GENERATE_COMPONENT_VALUE_KEYS.flatMap((key) => { + const component = values[key]; + + return component ? [[key, component.key]] : []; + }) + ), + graph: { ...stable.graph, id: input.id }, + submission: { ...submission, ...stable.nodeIds }, + }; +}; diff --git a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts index 8858477edb8..026722eab6d 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts @@ -10,7 +10,7 @@ import { setArchitectureCapabilities, } from './architectureCapabilities'; import { getDefaultGenerateSettings } from './baseGenerationPolicies'; -import { compileGeneratePreviewGraph, stabilizeBackendGraphIds } from './previewGraph'; +import { compileGeneratePreviewGraph, stabilizeBackendGraphIds, stabilizeBackendGraphNodeIds } from './previewGraph'; const sdxlModel: MainModelConfig = { base: 'sdxl', key: 'sdxl-model', name: 'SDXL', type: 'main' }; @@ -191,4 +191,21 @@ describe('stabilizeBackendGraphIds', () => { source: { field: 'value', node_id: 'seed' }, }); }); + + it('follows fixed node ids that lose their name to an earlier created id', () => { + const graph = { + edges: [], + id: 'graph', + nodes: { + seed_lqx1abc_a1b2c3: { id: 'seed_lqx1abc_a1b2c3', type: 'noise' }, + seed: { id: 'seed', type: 'integer' }, + positive_prompt: { id: 'positive_prompt', type: 'string' }, + }, + }; + + const result = stabilizeBackendGraphNodeIds(graph, { positive: 'positive_prompt', seed: 'seed' }); + + expect(result.nodeIds).toEqual({ positive: 'positive_prompt', seed: 'seed_2' }); + expect(result.graph.nodes[result.nodeIds.seed]?.type).toBe('integer'); + }); }); diff --git a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts index 12c3659af8e..9d829efdbf5 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts @@ -31,7 +31,11 @@ export type GeneratePreviewResult = const CREATED_ID_PATTERN = /^(.+)_[a-z0-9]{7,9}_[a-z0-9]{4,6}$/; -export const stabilizeBackendGraphIds = (graph: BackendGraphContract): BackendGraphContract => { +/** Stabilize ids and follow `nodeIds` through the renames: a fixed id loses its name to an earlier created one. */ +export const stabilizeBackendGraphNodeIds = >( + graph: BackendGraphContract, + nodeIds: T +): { graph: BackendGraphContract; nodeIds: T } => { const rename = new Map(); const used = new Set(); @@ -50,26 +54,27 @@ export const stabilizeBackendGraphIds = (graph: BackendGraphContract): BackendGr rename.set(id, candidate); } + const renamed = (id: string): string => rename.get(id) ?? id; + return { - edges: graph.edges.map((edge) => ({ - destination: { - field: edge.destination.field, - node_id: rename.get(edge.destination.node_id) ?? edge.destination.node_id, - }, - source: { field: edge.source.field, node_id: rename.get(edge.source.node_id) ?? edge.source.node_id }, - ...(edge.type ? { type: edge.type } : {}), - })), - id: 'generate-preview', - nodes: Object.fromEntries( - Object.entries(graph.nodes).map(([id, node]) => { - const nextId = rename.get(id) ?? id; - - return [nextId, { ...node, id: nextId }]; - }) - ), + graph: { + edges: graph.edges.map((edge) => ({ + destination: { field: edge.destination.field, node_id: renamed(edge.destination.node_id) }, + source: { field: edge.source.field, node_id: renamed(edge.source.node_id) }, + ...(edge.type ? { type: edge.type } : {}), + })), + id: 'generate-preview', + nodes: Object.fromEntries( + Object.entries(graph.nodes).map(([id, node]) => [renamed(id), { ...node, id: renamed(id) }]) + ), + }, + nodeIds: Object.fromEntries(Object.entries(nodeIds).map(([key, id]) => [key, renamed(id)])) as T, }; }; +export const stabilizeBackendGraphIds = (graph: BackendGraphContract): BackendGraphContract => + stabilizeBackendGraphNodeIds(graph, {}).graph; + const toPreviewContract = (backendGraph: BackendGraphContract, label: string): GraphContract => ({ backendGraph, edges: backendGraph.edges.map((edge, index) => ({ diff --git a/invokeai/frontend/webv2/src/features/generation/graphExport.ts b/invokeai/frontend/webv2/src/features/generation/graphExport.ts new file mode 100644 index 00000000000..2d1915f7a74 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/graphExport.ts @@ -0,0 +1,10 @@ +/** Offline graph export for tooling such as `pnpm graphs:export`; the browser bundle never loads it. */ +export { + compileGraphExportCase, + withArchitectureCapabilities, + type CompiledGraphExportCase, + type GraphExportCase, + type GraphExportEnvironment, + type GraphExportMode, +} from './core/graphExport'; +export type { ArchitectureCapabilitiesRow } from './core/architectureCapabilities'; diff --git a/invokeai/frontend/webv2/src/features/generation/settings.ts b/invokeai/frontend/webv2/src/features/generation/settings.ts index d4a29593997..06f3349e407 100644 --- a/invokeai/frontend/webv2/src/features/generation/settings.ts +++ b/invokeai/frontend/webv2/src/features/generation/settings.ts @@ -108,6 +108,7 @@ export { DYNAMIC_PROMPTS_MIN_PROMPTS, hasDynamicPromptSyntax, isDynamicPromptsSeedBehaviour, + planGenerateSubmission, sanitizeDynamicPromptsConfig, sanitizeMaxPrompts, sanitizeSampleSeed, diff --git a/invokeai/frontend/webv2/src/features/models/core/workingMemory.test.ts b/invokeai/frontend/webv2/src/features/models/core/workingMemory.test.ts new file mode 100644 index 00000000000..2ca4f19fe75 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/core/workingMemory.test.ts @@ -0,0 +1,440 @@ +import { describe, expect, it } from 'vitest'; + +import { + calibrationIdempotencyKey, + deriveCalibrationProgress, + describeCoverage, + describeFact, + formatAttemptId, + getCalibrationBlocker, + isMeasurementMethodVerified, + planCalibrationSteps, + selectAttemptRuns, + summarizeLineStates, + type LocalAttempt, + type WorkingMemoryCase, + type WorkingMemoryLine, + type WorkingMemoryPlan, + type WorkingMemoryRun, + type WorkingMemoryStatus, +} from './workingMemory'; + +const planCase = (dtype: string, side: number): WorkingMemoryCase => ({ + components: {}, + estimator: 'sd_vae_decode', + height: side, + id: `sd_vae_decode-decode-${dtype}-${String(side)}x${String(side)}`, + line: `decode_${dtype}`, + mode: 'txt2img', + model_key: 'main', + negative_prompt: '', + positive_prompt: 'a lake', + seed: 1, + settings: { vaePrecision: dtype }, + steps: 2, + width: side, +}); + +const plan = ( + cases: WorkingMemoryCase[], + warmUp = 1, + measured = 2 +): Pick => ({ + cases, + repetitions: { measured, warm_up: warmUp }, +}); + +const T0 = Date.parse('2026-10-01T12:00:00.000Z'); + +const run = ( + id: string, + offsetSeconds: number, + status: WorkingMemoryRun['status'], + error: string | null = null, + attemptId: string | null = 'attempt' +): WorkingMemoryRun => ({ + attempt_id: attemptId, + created_at: new Date(T0 + offsetSeconds * 1_000).toISOString(), + error, + point_count: status === 'completed' ? 1 : 0, + run_id: id, + status, +}); + +const describeSteps = (steps: ReturnType) => + steps.map((step) => `${step.case.line}@${String(step.case.width)}:${step.phase}${String(step.repetition)}`); + +describe('planCalibrationSteps', () => { + it('groups cases by precision and warms each case up right before measuring it', () => { + // The plan lists an fp32 case between two fp16 ones; grouping keeps one precision switch. + const steps = planCalibrationSteps(plan([planCase('fp16', 512), planCase('fp32', 512), planCase('fp16', 1024)])); + + expect(describeSteps(steps)).toEqual([ + 'decode_fp16@512:warm_up1', + 'decode_fp16@512:measured1', + 'decode_fp16@512:measured2', + 'decode_fp16@1024:warm_up1', + 'decode_fp16@1024:measured1', + 'decode_fp16@1024:measured2', + 'decode_fp32@512:warm_up1', + 'decode_fp32@512:measured1', + 'decode_fp32@512:measured2', + ]); + expect(steps.map((step) => step.index)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8]); + }); + + it('starts every precision with a warm-up, whatever the repetition counts', () => { + const steps = planCalibrationSteps( + plan([planCase('fp32', 768), planCase('fp16', 768), planCase('fp32', 1536)], 2, 3) + ); + const switches = steps.filter((step, index) => index === 0 || steps[index - 1]!.case.line !== step.case.line); + + expect(steps).toHaveLength(15); + expect(switches.map((step) => [step.case.line, step.phase])).toEqual([ + ['decode_fp32', 'warm_up'], + ['decode_fp16', 'warm_up'], + ]); + }); +}); + +describe('calibrationIdempotencyKey', () => { + const steps = planCalibrationSteps(plan([planCase('fp16', 512), planCase('fp32', 512)])); + + it('is the same for a resent step and differs across steps and attempts', () => { + const keys = steps.map((step) => calibrationIdempotencyKey('attempt-a', step)); + + expect(steps.map((step) => calibrationIdempotencyKey('attempt-a', step))).toEqual(keys); + expect(new Set(keys).size).toBe(steps.length); + expect(keys.some((key) => steps.some((step) => calibrationIdempotencyKey('attempt-b', step) === key))).toBe(false); + }); + + it('fits the server limit for plan ids of realistic length', () => { + const longStep = planCalibrationSteps( + plan([{ ...planCase('fp16', 4096), id: 'flux2_vae_decode-decode-bf16-4096x4096' }]) + )[2]!; + + expect(calibrationIdempotencyKey('0f8fad5b-d9cb-469f-a165-70867728950e', longStep).length).toBeLessThanOrEqual(255); + }); +}); + +describe('selectAttemptRuns', () => { + it("takes the newest run's attempt, oldest run first, however long its runs took", () => { + const runs = [ + run('new-2', 3_000, 'running', null, 'new'), + run('old-1', 0, 'failed', null, 'old'), + run('new-0', 300, 'completed', null, 'new'), + run('old-2', 1, 'completed', null, 'old'), + ]; + + expect(selectAttemptRuns(runs).map((entry) => entry.run_id)).toEqual(['new-0', 'new-2']); + expect(selectAttemptRuns(runs, 'old').map((entry) => entry.run_id)).toEqual(['old-1', 'old-2']); + expect(selectAttemptRuns([])).toEqual([]); + }); + + it('counts each run without an attempt id as an attempt of its own', () => { + const runs = [run('legacy-1', 0, 'completed', null, null), run('legacy-2', 1, 'failed', 'boom', null)]; + + expect(selectAttemptRuns(runs).map((entry) => entry.run_id)).toEqual(['legacy-2']); + }); +}); + +describe('deriveCalibrationProgress', () => { + const steps = planCalibrationSteps(plan([planCase('fp16', 512), planCase('fp32', 512)])); + const local = (overrides: Partial = {}): LocalAttempt => ({ + attemptId: 'mine', + newestRunAtStart: null, + stopped: false, + submissionError: null, + submitting: false, + total: steps.length, + ...overrides, + }); + + it('reports progress and the case the queue is on while any run is pending or running', () => { + const runs = [ + run('r0', 0, 'completed'), + run('r1', 1, 'completed'), + run('r2', 2, 'running'), + run('r3', 3, 'running'), + ]; + + expect(deriveCalibrationProgress({ local: null, runs, steps })).toEqual({ + activity: { currentCase: steps[2]!.case, done: 2, kind: 'running', queuing: false, total: 6 }, + outcome: { kind: 'none' }, + }); + }); + + it('counts an attempt that is still queuing as running before the server lists any run', () => { + const { activity } = deriveCalibrationProgress({ local: local({ submitting: true }), runs: [], steps }); + + expect(activity).toEqual({ currentCase: steps[0]!.case, done: 0, kind: 'running', queuing: true, total: 6 }); + }); + + it('reports the error of a failed run', () => { + const runs = [run('r0', 0, 'completed'), run('r1', 1, 'failed', 'CUDA out of memory'), run('r2', 2, 'canceled')]; + + expect(deriveCalibrationProgress({ local: null, runs, steps }).outcome).toEqual({ + error: 'CUDA out of memory', + kind: 'failed', + }); + }); + + it('tells completed, canceled and partial attempts apart', () => { + const completed = steps.map((_, index) => run(`r${String(index)}`, index, 'completed')); + const canceled = [run('r0', 0, 'canceled'), run('r1', 1, 'canceled')]; + const someCanceled = [...completed.slice(0, 3), run('r3', 3, 'canceled'), run('r4', 4, 'partial')]; + + expect(deriveCalibrationProgress({ local: null, runs: completed, steps }).outcome).toEqual({ kind: 'completed' }); + expect(deriveCalibrationProgress({ local: null, runs: canceled, steps }).outcome).toEqual({ kind: 'canceled' }); + expect(deriveCalibrationProgress({ local: null, runs: someCanceled, steps }).outcome).toEqual({ + canceled: true, + error: null, + kind: 'partial', + queued: 5, + total: 5, + }); + }); + + it('judges an attempt another tab started by its runs, not by a plan that may have changed since', () => { + const fewerThanPlanned = [run('r0', 0, 'completed'), run('r1', 1, 'completed')]; + + expect(deriveCalibrationProgress({ local: null, runs: fewerThanPlanned, steps }).outcome).toEqual({ + kind: 'completed', + }); + }); + + it('reports an attempt whose queuing stopped early as partial, or failed when nothing was queued', () => { + const runs = [run('r0', 0, 'completed', null, 'mine'), run('r1', 1, 'completed', null, 'mine')]; + + expect(deriveCalibrationProgress({ local: local({ submissionError: 'Queue full' }), runs, steps }).outcome).toEqual( + { + canceled: false, + error: 'Queue full', + kind: 'partial', + queued: 2, + total: 6, + } + ); + // Its graphs did not compile, so nothing of it was queued; an older attempt does not hide the error. + const older = [run('old', 0, 'completed', null, 'old')]; + const failedStart = local({ newestRunAtStart: 'old', submissionError: 'Case x: unknown model' }); + expect(deriveCalibrationProgress({ local: failedStart, runs: older, steps }).outcome).toEqual({ + error: 'Case x: unknown model', + kind: 'failed', + }); + }); + + it('lets an attempt that queued nothing give way to any attempt that ran after it', () => { + // This tab could not reach the server; another tab then calibrated to the end. + const failedStart = local({ newestRunAtStart: 'old', submissionError: 'Failed to fetch' }); + const theirs = steps.map((_, index) => run(`theirs-${String(index)}`, 100 + index, 'completed', null, 'theirs')); + + expect( + deriveCalibrationProgress({ + local: failedStart, + runs: [run('old', 0, 'completed', null, 'old'), ...theirs], + steps, + }).outcome + ).toEqual({ kind: 'completed' }); + expect( + deriveCalibrationProgress({ local: { ...failedStart, newestRunAtStart: null }, runs: theirs, steps }).outcome + ).toEqual({ kind: 'completed' }); + }); + + it('reports an attempt the user stopped as canceled, even where queuing ended before any run was canceled', () => { + const runs = [run('r0', 0, 'completed', null, 'mine'), run('r1', 1, 'completed', null, 'mine')]; + + expect(deriveCalibrationProgress({ local: local({ stopped: true }), runs, steps }).outcome).toEqual({ + canceled: true, + error: null, + kind: 'partial', + queued: 2, + total: 6, + }); + // Stopped while compiling: nothing was queued. + expect(deriveCalibrationProgress({ local: local({ stopped: true }), runs: [], steps }).outcome).toEqual({ + kind: 'canceled', + }); + }); + + it('tells an attempt whose queuing ended early from a complete one by the step count its id carries', () => { + const attemptId = formatAttemptId('0f8fad5b-d9cb-469f-a165-70867728950e', 6); + const truncated = [run('r0', 0, 'completed', null, attemptId), run('r1', 1, 'completed', null, attemptId)]; + const complete = steps.map((_, index) => run(`r${String(index)}`, index, 'completed', null, attemptId)); + + // After a reload this tab has no record of it. + expect(deriveCalibrationProgress({ local: null, runs: truncated, steps }).outcome).toEqual({ + canceled: false, + error: null, + kind: 'partial', + queued: 2, + total: 6, + }); + expect(deriveCalibrationProgress({ local: null, runs: complete, steps }).outcome).toEqual({ kind: 'completed' }); + }); + + it('follows the newer attempt another tab started, not this tab’s older one', () => { + const runs = [ + run('mine', 0, 'failed', 'old failure', 'mine'), + run('theirs-0', 100, 'completed', null, 'theirs'), + run('theirs-1', 101, 'running', null, 'theirs'), + ]; + const { activity } = deriveCalibrationProgress({ local: local(), runs, steps }); + + expect(activity).toMatchObject({ done: 1, kind: 'running' }); + }); +}); + +describe('summarizeLineStates', () => { + it('reports the most severe line, or null without lines', () => { + expect(summarizeLineStates([{ state: 'in_use' }, { state: 'stale' }])).toBe('stale'); + expect(summarizeLineStates([{ state: 'insufficient' }, { state: 'in_use' }])).toBe('insufficient'); + expect(summarizeLineStates([{ state: 'stale' }, { state: 'unusable' }])).toBe('unusable'); + expect(summarizeLineStates([{ state: 'in_use' }])).toBe('in_use'); + expect(summarizeLineStates([])).toBeNull(); + }); +}); + +describe('describeCoverage', () => { + type Line = Pick; + const line = ( + state: WorkingMemoryLine['state'], + { + envId = 'current', + lineName = 'decode_fp16', + ownWindows = 3, + version = 2, + }: { envId?: string; lineName?: string; ownWindows?: number; version?: number } = {} + ): Line => ({ + env_id: envId, + estimator: 'sd_vae_decode', + line: lineName, + own_windows: ownWindows, + state, + version, + }); + const coverage = (lines: Line[], runs: WorkingMemoryRun[] = []) => + describeCoverage({ current_env_id: 'current', lines, runs }); + + it("takes the headline from the current environment's lines, else from all lines", () => { + // An old environment's unusable line does not outrank what applies now. + expect(coverage([line('unusable', { envId: 'old' }), line('in_use')])).toMatchObject({ + hasCurrentLines: true, + state: 'in_use', + }); + expect(coverage([line('unusable', { envId: 'old' })])).toMatchObject({ hasCurrentLines: false, state: 'unusable' }); + expect(coverage([])).toMatchObject({ hasCurrentLines: false, hasOwnMeasurements: false, state: null }); + }); + + it('lets a line recalibrated under a new estimator version supersede the old version’s unusable line', () => { + const oldVersion = line('unusable', { version: 1 }); + + expect(coverage([oldVersion, line('in_use')]).state).toBe('in_use'); + expect(coverage([oldVersion, line('stale')]).state).toBe('stale'); + expect(coverage([oldVersion]).state).toBe('unusable'); + // Only the same line supersedes it. + expect(coverage([oldVersion, line('in_use', { lineName: 'decode_fp32' })]).state).toBe('unusable'); + expect(coverage([oldVersion, line('insufficient')]).state).toBe('unusable'); + }); + + it('tells lines only other models measured from the model’s own', () => { + expect(coverage([line('in_use', { ownWindows: 0 })])).toMatchObject({ + hasOwnMeasurements: false, + usesOthersOnly: true, + }); + expect(coverage([line('in_use', { ownWindows: 0 }), line('stale')])).toMatchObject({ + hasOwnMeasurements: true, + usesOthersOnly: false, + }); + // A shared line that does not apply is not used. + expect(coverage([line('insufficient', { ownWindows: 0 })]).usesOthersOnly).toBe(false); + // Runs that measured nothing still leave something to reset. + expect(coverage([], [run('r0', 0, 'canceled')]).hasOwnMeasurements).toBe(true); + }); +}); + +describe('getCalibrationBlocker', () => { + const supported = { reason: null, supported: true }; + const available = { available: true, unavailable_reason: null }; + const usable = { problem: null, usable: true }; + + it('names the reason a user can least act on first', () => { + expect( + getCalibrationBlocker({ + isRunning: true, + plan: { reason: 'Not a text-to-image model.', supported: false }, + status: { available: false, unavailable_reason: 'Two GPUs.' }, + summary: null, + }) + ).toEqual({ kind: 'unsupported', reason: 'Not a text-to-image model.' }); + expect( + getCalibrationBlocker({ + isRunning: true, + plan: supported, + status: { available: false, unavailable_reason: 'File unusable.' }, + summary: { problem: 'schema_version 9', usable: false }, + }) + ).toEqual({ kind: 'storeUnusable', reason: 'schema_version 9' }); + expect( + getCalibrationBlocker({ + isRunning: false, + plan: supported, + status: { available: false, unavailable_reason: 'Two GPUs.' }, + summary: null, + }) + ).toEqual({ kind: 'unavailable', reason: 'Two GPUs.' }); + expect(getCalibrationBlocker({ isRunning: true, plan: supported, status: available, summary: usable })).toEqual({ + kind: 'running', + }); + expect(getCalibrationBlocker({ isRunning: false, plan: supported, status: available, summary: usable })).toBeNull(); + }); +}); + +describe('isMeasurementMethodVerified', () => { + const status = (verified: boolean | null): WorkingMemoryStatus => ({ + available: true, + current_device_class_id: 'class', + current_env_id: 'env', + device_classes: [ + { + device_class: { arch: null, backend: 'cuda', id: 'class', name: 'GPU', total_mib: 8192, vendor: 'x' }, + device_class_id: 'class', + envs: [], + is_current: true, + measurement_method: 'allocator', + measurement_verified: verified, + point_counts: [], + problem: null, + runs: [], + usable: true, + }, + ], + unavailable_reason: null, + }); + + it("takes the model summary's answer, then the current device class's", () => { + expect(isMeasurementMethodVerified({ measurement_verified: false }, status(true))).toBe(false); + expect(isMeasurementMethodVerified({ measurement_verified: null }, status(false))).toBe(false); + expect(isMeasurementMethodVerified(null, status(true))).toBe(true); + expect(isMeasurementMethodVerified(null, status(null))).toBeNull(); + }); +}); + +describe('describeFact', () => { + it('names runtimes by product, translates the rest, and shows variables and settings by their own name', () => { + expect(['torch', 'hip', 'cudnn', 'os'].map(describeFact)).toEqual([ + { name: 'PyTorch' }, + { name: 'ROCm' }, + { name: 'cuDNN' }, + { name: 'OS' }, + ]); + expect(['driver', 'allocator_backend', 'rocm_sdpa_guard'].map(describeFact)).toEqual([ + { translated: 'driver' }, + { translated: 'allocator' }, + { translated: 'sdpaGuard' }, + ]); + expect(describeFact('setting.attention_type')).toEqual({ name: 'attention_type' }); + expect(describeFact('env.PYTORCH_CUDA_ALLOC_CONF')).toEqual({ name: 'PYTORCH_CUDA_ALLOC_CONF' }); + expect(describeFact('something_new')).toEqual({ name: 'something_new' }); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/models/core/workingMemory.ts b/invokeai/frontend/webv2/src/features/models/core/workingMemory.ts new file mode 100644 index 00000000000..fd85e04b217 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/core/workingMemory.ts @@ -0,0 +1,428 @@ +/** + * Working-memory calibration as the Model Manager sees it: the order a plan's cases run in, the retry keys that make a + * lost response safe to resend, which of a model's runs make up the latest attempt, and what the section shows for + * a plan, a summary and the device status. The server owns every measurement and every state; this module only + * orders, groups and reads them. + */ + +import type { components } from '@api/schema'; + +type Schemas = components['schemas']; + +export type WorkingMemoryPlan = Schemas['WorkingMemoryCalibrationPlan']; +export type WorkingMemoryCase = Schemas['WorkingMemoryCalibrationCase']; +export type WorkingMemorySummary = Schemas['WorkingMemoryCalibrationModelSummary']; +export type WorkingMemoryLine = Schemas['WorkingMemoryCalibrationModelLine']; +export type WorkingMemoryLineSize = Schemas['WorkingMemoryCalibrationLineSize']; +export type WorkingMemoryRun = Schemas['WorkingMemoryCalibrationModelRun']; +export type WorkingMemoryStatus = Schemas['WorkingMemoryCalibrationStatus']; +export type WorkingMemoryRunCreated = Schemas['WorkingMemoryCalibrationRunCreated']; +export type WorkingMemoryLineState = WorkingMemoryLine['state']; +export type WorkingMemoryLineContributor = Schemas['WorkingMemoryCalibrationLineContributor']; + +/** The server reserves this origin prefix for calibration queue items; the run id follows it. */ +export const CALIBRATION_ORIGIN_PREFIX = 'invokeai:wm-calibration:'; + +export const calibrationRunOrigin = (runId: string): string => `${CALIBRATION_ORIGIN_PREFIX}${runId}`; + +export const isCalibrationOrigin = (origin: unknown): boolean => + typeof origin === 'string' && origin.startsWith(CALIBRATION_ORIGIN_PREFIX); + +/** Models whose plan the section asks for: calibration covers main models and standalone VAEs. */ +export const supportsWorkingMemoryCalibration = (model: { type: string }): boolean => + model.type === 'main' || model.type === 'vae'; + +/** A case's enqueue payload as Generate would build it: the graph and its batch data. */ +export interface CalibrationGraph { + caseId: string; + graph: unknown; + data: unknown; +} + +/** + * Builds every case's graph, or rejects naming the case that cannot compile, and rejects once `signal` aborts. + * Generation owns graph compilation and depends on Models, so App supplies this; its implementation loads lazily on + * first call. + */ +export type CalibrationGraphCompiler = ( + cases: readonly WorkingMemoryCase[], + signal: AbortSignal +) => Promise; + +export type CalibrationPhase = 'warm_up' | 'measured'; + +/** One queue item of an attempt: a case at one repetition. Each step is its own POST and its own server run. */ +export interface CalibrationStep { + /** Position in the attempt, which is also the order the server creates the step's run in. */ + index: number; + case: WorkingMemoryCase; + phase: CalibrationPhase; + /** 1-based within the phase. */ + repetition: number; +} + +/** + * The plan's cases grouped by line (one decode dtype each), each case's warm-ups right before its measured runs. + * Grouping keeps one precision switch per dtype: the first decode after a switch recasts the VAE, and the server + * excludes those windows, so a switch must land on a warm-up. + */ +export const planCalibrationSteps = (plan: Pick): CalibrationStep[] => { + const groups = new Map(); + + for (const item of plan.cases) { + const group = groups.get(item.line); + + if (group) { + group.push(item); + } else { + groups.set(item.line, [item]); + } + } + + const steps: CalibrationStep[] = []; + const push = (item: WorkingMemoryCase, phase: CalibrationPhase, count: number) => { + for (let repetition = 1; repetition <= count; repetition += 1) { + steps.push({ case: item, index: steps.length, phase, repetition }); + } + }; + + for (const cases of groups.values()) { + for (const item of cases) { + push(item, 'warm_up', plan.repetitions.warm_up); + push(item, 'measured', plan.repetitions.measured); + } + } + + return steps; +}; + +/** + * The retry key of one step. Resending a step after a lost response with the same key returns the run the first + * request created. The key carries the attempt id: the server keys its retry receipts on the key alone, so a key + * reused by a new attempt would be refused. + */ +export const calibrationIdempotencyKey = (attemptId: string, step: CalibrationStep): string => + `webv2:wm-calibration:${attemptId}:${String(step.index)}:${step.case.id}:${step.phase}-${String(step.repetition)}`; + +const byCreation = (left: WorkingMemoryRun, right: WorkingMemoryRun): number => + Date.parse(left.created_at) - Date.parse(right.created_at); + +/** The id of the most recently created run, or null without runs. */ +export const newestRunId = (runs: readonly WorkingMemoryRun[]): string | null => + [...runs].sort(byCreation).at(-1)?.run_id ?? null; + +/** + * The runs of one attempt, oldest first: those carrying `attemptId`, or by default those of the newest run's attempt. + * A run without an attempt id (created before the server stored one) is an attempt of its own. + */ +export const selectAttemptRuns = (runs: readonly WorkingMemoryRun[], attemptId?: string): WorkingMemoryRun[] => { + const ordered = [...runs].sort(byCreation); + const newest = ordered.at(-1); + const target = attemptId ?? newest?.attempt_id ?? null; + + if (target === null) { + return newest ? [newest] : []; + } + + return ordered.filter((run) => run.attempt_id === target); +}; + +/** + * An attempt's id: a fresh id plus the number of steps it means to queue, so any tab can tell an attempt whose + * queuing ended early (a reload, a closed tab) from a complete one. The server treats the id as opaque. + */ +export const formatAttemptId = (id: string, total: number): string => `${id}.${String(total)}`; + +/** The step count an attempt id carries; null for an id without one. */ +export const plannedStepsOf = (attemptId: string | null | undefined): number | null => { + const match = /\.(\d+)$/.exec(attemptId ?? ''); + + return match ? Number(match[1]) : null; +}; + +/** What this tab knows of an attempt it started. */ +export interface LocalAttempt { + attemptId: string; + /** Steps the attempt meant to queue. */ + total: number; + /** Still compiling graphs or posting steps. */ + submitting: boolean; + /** Why compiling or posting stopped early, when it did. */ + submissionError: string | null; + /** The user stopped it. */ + stopped: boolean; + /** + * The newest of the model's runs this tab knew when the attempt started. While the attempt has no runs of its own + * (it failed before queuing anything), it is the latest attempt only until a newer run appears. + */ + newestRunAtStart: string | null; +} + +export type CalibrationActivity = + | { kind: 'idle' } + | { + kind: 'running'; + /** Steps settled so far. */ + done: number; + total: number; + /** The case the queue is running or about to run, when it can be told. */ + currentCase: WorkingMemoryCase | null; + /** Still compiling graphs or posting steps. */ + queuing: boolean; + }; + +export type CalibrationOutcome = + | { kind: 'none' } + | { kind: 'completed' } + | { kind: 'canceled' } + /** `canceled`: runs were canceled, so the attempt did not stop because queuing ended early. */ + | { kind: 'partial'; canceled: boolean; error: string | null; queued: number; total: number } + | { kind: 'failed'; error: string | null }; + +/** + * Progress and the latest attempt's outcome. A run counts as running while any of its items is pending or running + * (the server derives that from the queue), so a pending item blocks a second start across reloads and tabs. + */ +export const deriveCalibrationProgress = ({ + local, + runs, + steps, +}: { + local: LocalAttempt | null; + runs: readonly WorkingMemoryRun[]; + steps: readonly CalibrationStep[]; +}): { activity: CalibrationActivity; outcome: CalibrationOutcome } => { + // Another tab may have started a newer attempt since; this tab's record then describes an older one. A record whose + // attempt queued nothing (it failed to compile or to post its first step) stays current until a newer run exists. + const newest = [...runs].sort(byCreation).at(-1); + const current = + local !== null && + (local.submitting || + (newest !== undefined && newest.attempt_id === local.attemptId) || + (!runs.some((run) => run.attempt_id === local.attemptId) && + (newest?.run_id ?? null) === local.newestRunAtStart && + !runs.some((run) => run.status === 'running'))) + ? local + : null; + const attempt = selectAttemptRuns(runs, current?.attemptId); + // The attempt's own step count: this tab's record, else the one its id carries. A running attempt with neither is + // assumed to follow the current plan; a settled one is judged by its runs alone, since the plan may have changed. + const planned = current?.total ?? plannedStepsOf(attempt[0]?.attempt_id); + const total = Math.max(planned ?? steps.length, attempt.length); + const plannedTotal = planned ?? attempt.length; + const runningIndex = attempt.findIndex((run) => run.status === 'running'); + const queuing = current?.submitting ?? false; + + if (queuing || runningIndex !== -1) { + // Steps settle in order: the queue runs an attempt's items first in, first out. + const done = attempt.filter((run) => run.status !== 'running').length; + const currentIndex = runningIndex === -1 ? attempt.length : runningIndex; + + return { + activity: { currentCase: steps[currentIndex]?.case ?? null, done, kind: 'running', queuing, total }, + outcome: { kind: 'none' }, + }; + } + + const idle: CalibrationActivity = { kind: 'idle' }; + const canceled = + (current?.stopped ?? false) || attempt.some((run) => run.status === 'canceled' || run.status === 'partial'); + const partial = (error: string | null): CalibrationOutcome => ({ + canceled, + error, + kind: 'partial', + queued: attempt.length, + total: plannedTotal, + }); + + if (current?.submissionError) { + return { + activity: idle, + outcome: + attempt.length === 0 ? { error: current.submissionError, kind: 'failed' } : partial(current.submissionError), + }; + } + + if (attempt.length === 0) { + return { activity: idle, outcome: current?.stopped ? { kind: 'canceled' } : { kind: 'none' } }; + } + + const failed = attempt.find((run) => run.status === 'failed'); + + if (failed) { + return { activity: idle, outcome: { error: failed.error, kind: 'failed' } }; + } + + if (attempt.every((run) => run.status === 'canceled')) { + return { activity: idle, outcome: { kind: 'canceled' } }; + } + + if (attempt.every((run) => run.status === 'completed') && attempt.length >= plannedTotal) { + return { activity: idle, outcome: { kind: 'completed' } }; + } + + return { activity: idle, outcome: partial(null) }; +}; + +const LINE_STATE_SEVERITY: readonly WorkingMemoryLineState[] = ['unusable', 'stale', 'insufficient', 'in_use']; + +/** The most severe state among lines, or null without lines. */ +export const summarizeLineStates = ( + lines: readonly Pick[] +): WorkingMemoryLineState | null => + LINE_STATE_SEVERITY.find((state) => lines.some((line) => line.state === state)) ?? null; + +/** What a summary says about the model as a whole. */ +export interface CalibrationCoverage { + /** The headline state: of the current environment's lines, or of all lines when it has none. */ + state: WorkingMemoryLineState | null; + /** Some line of the current environment exists, measured by this model or another. */ + hasCurrentLines: boolean; + /** This model has windows on some line or has runs, so a reset removes something. */ + hasOwnMeasurements: boolean; + /** Estimates use lines that only other models measured. */ + usesOthersOnly: boolean; +} + +export const describeCoverage = ( + summary: { + current_env_id: string | null; + lines: readonly Pick[]; + runs: readonly unknown[]; + } | null +): CalibrationCoverage => { + const lines = summary?.lines ?? []; + const current = lines.filter((line) => line.env_id === summary?.current_env_id); + const hasOwnWindows = lines.some((line) => line.own_windows > 0); + const applies = (line: (typeof lines)[number]) => line.state === 'in_use' || line.state === 'stale'; + // An estimator update leaves the old version's windows as an unusable line; once the new version's line applies, + // the old one is superseded and does not head the section. + const headline = current.filter( + (line) => + line.state !== 'unusable' || + !current.some( + (other) => + other.estimator === line.estimator && + other.line === line.line && + other.version !== line.version && + applies(other) + ) + ); + + return { + hasCurrentLines: current.length > 0, + hasOwnMeasurements: hasOwnWindows || (summary?.runs.length ?? 0) > 0, + state: summarizeLineStates(current.length > 0 ? headline : lines), + usesOthersOnly: !hasOwnWindows && current.some(applies), + }; +}; + +/** A line's contributors with this model first, the rest in the server's order (by name). */ +export const orderContributors = ( + contributors: readonly WorkingMemoryLineContributor[], + modelKey: string +): WorkingMemoryLineContributor[] => [ + ...contributors.filter((entry) => entry.model_key === modelKey), + ...contributors.filter((entry) => entry.model_key !== modelKey), +]; + +export type CalibrationBlocker = + | { kind: 'unsupported'; reason: string | null } + | { kind: 'unavailable'; reason: string | null } + | { kind: 'storeUnusable'; reason: string | null } + | { kind: 'running' }; + +/** + * Why Calibrate is unavailable, in the order a user can act on: nothing helps an unsupported model; a missing or + * second GPU and an unusable store are machine problems; a running attempt only needs to finish. + */ +export const getCalibrationBlocker = ({ + isRunning, + plan, + status, + summary, +}: { + isRunning: boolean; + plan: Pick; + status: Pick; + /** Null when the server has no current device class to summarise. */ + summary: Pick | null; +}): CalibrationBlocker | null => { + if (!plan.supported) { + return { kind: 'unsupported', reason: plan.reason }; + } + + if (summary && !summary.usable) { + return { kind: 'storeUnusable', reason: summary.problem }; + } + + if (!status.available || summary === null) { + return { kind: 'unavailable', reason: status.unavailable_reason }; + } + + return isRunning ? { kind: 'running' } : null; +}; + +/** + * Whether the server's way of measuring GPU memory was checked against real hardware on this GPU's backend: the + * model summary's answer, else the current device class's; null while neither can tell. + */ +export const isMeasurementMethodVerified = ( + summary: Pick | null, + status: WorkingMemoryStatus +): boolean | null => + summary?.measurement_verified ?? + status.device_classes.find((entry) => entry.is_current)?.measurement_verified ?? + null; + +/** `decode_fp16` → operation `decode`, dtype `fp16`. A line without a dtype suffix keeps its whole name. */ +export const splitLine = (line: string): { dtype: string | null; operation: string } => { + const separator = line.lastIndexOf('_'); + + return separator <= 0 + ? { dtype: null, operation: line } + : { dtype: line.slice(separator + 1), operation: line.slice(0, separator) }; +}; + +/** Changed facts the section names with a translated label. */ +export type TranslatedFact = 'allocator' | 'driver' | 'invokeai' | 'osBuild' | 'sdpaGuard' | 'torchBuild'; + +/** Facts named by a product or runtime name, which needs no translation. */ +const FACT_NAMES: Readonly> = { + cuda: 'CUDA', + cudnn: 'cuDNN', + hip: 'ROCm', + miopen: 'MIOpen', + os: 'OS', + torch: 'PyTorch', + xpu: 'XPU runtime', +}; + +const TRANSLATED_FACTS: Readonly> = { + allocator_backend: 'allocator', + driver: 'driver', + invokeai: 'invokeai', + os_build: 'osBuild', + rocm_sdpa_guard: 'sdpaGuard', + torch_build: 'torchBuild', +}; + +/** + * How the section names a changed environment fact (the server's fingerprint keys): a product name, a translated + * label, or for environment variables and Invoke settings their own name. An unknown key shows as sent. + */ +export const describeFact = (key: string): { name: string } | { translated: TranslatedFact } => { + const translated = TRANSLATED_FACTS[key]; + + if (translated) { + return { translated }; + } + + for (const prefix of ['env.', 'setting.']) { + if (key.startsWith(prefix) && key.length > prefix.length) { + return { name: key.slice(prefix.length) }; + } + } + + return { name: FACT_NAMES[key] ?? key }; +}; diff --git a/invokeai/frontend/webv2/src/features/models/data/workingMemoryApi.ts b/invokeai/frontend/webv2/src/features/models/data/workingMemoryApi.ts new file mode 100644 index 00000000000..61dae8c1fdd --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/data/workingMemoryApi.ts @@ -0,0 +1,71 @@ +import type { + WorkingMemoryPlan, + WorkingMemoryRunCreated, + WorkingMemoryStatus, + WorkingMemorySummary, +} from '@features/models/core/workingMemory'; + +import { calibrationRunOrigin } from '@features/models/core/workingMemory'; + +import { browserModelsDataPort } from './transport'; + +/** REST client for working-memory calibration, admin-only on the server. */ + +const { request, requestJson } = browserModelsDataPort; + +const modelPath = (key: string): string => `/api/v2/models/i/${encodeURIComponent(key)}/working_memory`; + +/** Calibration items are ordinary queue items; the queue routes act on them by origin. */ +const QUEUE_BASE = '/api/v1/queue/default'; + +export const getWorkingMemoryPlan = (key: string, signal?: AbortSignal): Promise => + requestJson(`${modelPath(key)}/plan`, { signal }); + +/** 409 when there is no current device class (no GPU, or several generation devices). */ +export const getWorkingMemorySummary = (key: string, signal?: AbortSignal): Promise => + requestJson(`${modelPath(key)}/summary`, { signal }); + +export const getWorkingMemoryStatus = (signal?: AbortSignal): Promise => + requestJson('/api/v2/models/working_memory/status', { signal }); + +export interface CalibrateRequest { + /** Groups the runs of one Calibrate click; the server returns it on every run. */ + attemptId: string; + graph: unknown; + data: unknown; + idempotencyKey: string; +} + +/** Enqueues one queue item as one server run. Resending with the same key returns that run and enqueues nothing. */ +export const calibrateWorkingMemory = ( + key: string, + { attemptId, data, graph, idempotencyKey }: CalibrateRequest, + signal?: AbortSignal +): Promise => + requestJson(`${modelPath(key)}/calibrate`, { + body: JSON.stringify({ attempt_id: attemptId, data, graph, idempotency_key: idempotencyKey }), + method: 'POST', + signal, + }); + +/** Releases the server's receipt for a key whose response arrived; until then it counts toward a per-owner limit. */ +export const acknowledgeCalibrationEnqueue = async (idempotencyKey: string, signal?: AbortSignal): Promise => { + await request(`${QUEUE_BASE}/enqueue_batch/acknowledge`, { + body: JSON.stringify({ idempotency_key: idempotencyKey }), + headers: { 'Content-Type': 'application/json' }, + method: 'POST', + signal, + }); +}; + +/** Cancels a run's items, the running one included. */ +export const cancelCalibrationRun = async (runId: string, signal?: AbortSignal): Promise => { + const params = new URLSearchParams({ origin_prefix: calibrationRunOrigin(runId) }); + + await request(`${QUEUE_BASE}/cancel_all?${params.toString()}`, { method: 'PUT', signal }); +}; + +/** Removes the model's runs and measurements on the current device class. */ +export const resetWorkingMemory = async (key: string, signal?: AbortSignal): Promise => { + await request(modelPath(key), { method: 'DELETE', signal }); +}; diff --git a/invokeai/frontend/webv2/src/features/models/data/workingMemoryCalibration.ts b/invokeai/frontend/webv2/src/features/models/data/workingMemoryCalibration.ts new file mode 100644 index 00000000000..cd1378d0f3d --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/data/workingMemoryCalibration.ts @@ -0,0 +1,306 @@ +import type { QueryClient } from '@tanstack/react-query'; + +import { + calibrationIdempotencyKey, + formatAttemptId, + isCalibrationOrigin, + newestRunId, + planCalibrationSteps, + type CalibrationGraphCompiler, + type LocalAttempt, + type WorkingMemoryPlan, + type WorkingMemoryRunCreated, + type WorkingMemorySummary, +} from '@features/models/core/workingMemory'; +import { createUuid } from '@platform/browser/randomUuid'; +import { + assertAccountScopeCurrent, + captureAccountScope, + isAccountScopeCurrent, + registerAccountOwnedResource, + type AccountScope, +} from '@platform/state/accountLifecycle'; +import { createExternalStore } from '@platform/state/externalStore'; +import { ApiError, getApiErrorMessage, sleep } from '@platform/transport/http'; +import { socketHub } from '@platform/transport/socketHub'; + +import { + acknowledgeCalibrationEnqueue, + calibrateWorkingMemory, + cancelCalibrationRun, + resetWorkingMemory, + type CalibrateRequest, +} from './workingMemoryApi'; +import { workingMemoryKeys, workingMemorySummaryQueryOptions } from './workingMemoryQueries'; + +/** + * The calibration attempts this tab started, by model key: what only this tab knows (that it is still compiling or + * posting steps, how many it meant to post, why it stopped). Every step's run carries the attempt id, so the server's + * runs say which runs belong together, after a reload too. Kept outside the section so leaving a model's details + * while an attempt queues, and coming back, shows the same attempt. + */ +const attemptsStore = createExternalStore<{ byModel: ReadonlyMap }>({ byModel: new Map() }); + +interface Flight { + promise: Promise; + stopRequested: boolean; + /** Aborts compiling; a step being posted is not aborted, since its run exists once the response arrives. */ + stop: AbortController; +} + +/** The attempt queuing per model; at most one, so a double click cannot post a second attempt. */ +const flights = new Map(); + +registerAccountOwnedResource({ + clear: () => { + for (const flight of flights.values()) { + flight.stopRequested = true; + flight.stop.abort(); + } + flights.clear(); + attemptsStore.setSnapshot({ byModel: new Map() }); + }, + name: 'working-memory-calibration', +}); + +const setAttempt = (modelKey: string, attempt: LocalAttempt | null): void => { + const byModel = new Map(attemptsStore.getSnapshot().byModel); + + if (attempt) { + byModel.set(modelKey, attempt); + } else { + byModel.delete(modelKey); + } + + attemptsStore.setSnapshot({ byModel }); +}; + +const updateAttempt = (modelKey: string, attemptId: string, change: (attempt: LocalAttempt) => LocalAttempt) => { + const attempt = attemptsStore.getSnapshot().byModel.get(modelKey); + + // A reset or a newer attempt replaced it meanwhile. + if (attempt?.attemptId === attemptId) { + setAttempt(modelKey, change(attempt)); + } +}; + +export const useLocalAttempt = (modelKey: string): LocalAttempt | null => + attemptsStore.useSelector((snapshot) => snapshot.byModel.get(modelKey) ?? null, Object.is); + +/** A lost response is resent with the same key, which returns the run the first request created. */ +const POST_ATTEMPTS = 3; +const POST_RETRY_DELAY_MS = 1_000; + +const postStep = async ( + modelKey: string, + request: CalibrateRequest, + signal: AbortSignal +): Promise => { + for (let attempt = 1; ; attempt += 1) { + try { + return await calibrateWorkingMemory(modelKey, request, signal); + } catch (error) { + // The server answered: resending the same request gets the same answer. + if (error instanceof ApiError || signal.aborted || attempt >= POST_ATTEMPTS) { + throw error; + } + await sleep(POST_RETRY_DELAY_MS, signal); + } + } +}; + +/** + * Every model's summary moves: lines are shared across models, and a reset or new points change their fits. The + * changed model's plan prices on those fits too. + */ +const invalidateAfterChange = async (queryClient: QueryClient, owner: AccountScope, modelKey?: string) => { + await Promise.all([ + queryClient.invalidateQueries({ queryKey: workingMemoryKeys.summaries(owner) }), + queryClient.invalidateQueries({ queryKey: workingMemoryKeys.status(owner) }), + modelKey === undefined + ? undefined + : queryClient.invalidateQueries({ queryKey: workingMemoryKeys.plan(owner, modelKey) }), + ]); +}; + +/** + * Compiles the plan's graphs (loading the compiler on first use) and posts every step in order, each after the last + * one's response, so the queue runs them in step order. Stops at the first error the server answers with and keeps + * the runs already queued: they run, and the section reports the attempt as partial. + */ +export const startCalibration = ({ + compile, + modelKey, + plan, + queryClient, +}: { + compile: CalibrationGraphCompiler; + modelKey: string; + plan: WorkingMemoryPlan; + queryClient: QueryClient; +}): Promise => { + const running = flights.get(modelKey); + + if (running) { + return running.promise; + } + + const owner = captureAccountScope(); + const steps = planCalibrationSteps(plan); + const attemptId = formatAttemptId(createUuid(), steps.length); + const flight: Flight = { promise: Promise.resolve(), stop: new AbortController(), stopRequested: false }; + + const known = queryClient.getQueryData(workingMemoryKeys.summary(owner, modelKey)); + + setAttempt(modelKey, { + attemptId, + newestRunAtStart: newestRunId(known?.runs ?? []), + stopped: false, + submissionError: null, + submitting: true, + total: steps.length, + }); + + const queue = async () => { + const compiling = compile(plan.cases, AbortSignal.any([owner.signal, flight.stop.signal])); + const graphs = new Map((await compiling).map((compiled) => [compiled.caseId, compiled])); + + for (const step of steps) { + if (flight.stopRequested || !isAccountScopeCurrent(owner)) { + return; + } + + const compiled = graphs.get(step.case.id); + + if (!compiled) { + throw new Error(`No graph was built for case ${step.case.id}.`); + } + + const idempotencyKey = calibrationIdempotencyKey(attemptId, step); + await postStep(modelKey, { attemptId, data: compiled.data, graph: compiled.graph, idempotencyKey }, owner.signal); + + assertAccountScopeCurrent(owner); + // Best effort: an unacknowledged receipt only expires later. + void acknowledgeCalibrationEnqueue(idempotencyKey, owner.signal).catch(() => undefined); + void queryClient.invalidateQueries({ queryKey: workingMemoryKeys.summary(owner, modelKey) }); + } + }; + + flight.promise = queue() + .catch((error: unknown) => { + // A stop that aborted compiling is not an error. + if (isAccountScopeCurrent(owner) && !flight.stopRequested) { + updateAttempt(modelKey, attemptId, (attempt) => ({ + ...attempt, + submissionError: getApiErrorMessage(error, ''), + })); + } + }) + .finally(() => { + if (flights.get(modelKey) === flight) { + flights.delete(modelKey); + } + + if (isAccountScopeCurrent(owner)) { + updateAttempt(modelKey, attemptId, (attempt) => ({ ...attempt, submitting: false })); + void invalidateAfterChange(queryClient, owner, modelKey); + } + }); + flights.set(modelKey, flight); + + return flight.promise; +}; + +/** + * Stops queuing, then cancels every run of the model the queue still holds, newest first so no later item starts + * while an earlier one is cancelled. Waits for the step in flight: its run exists once the response arrives. + */ +export const stopCalibration = async ({ + modelKey, + queryClient, +}: { + modelKey: string; + queryClient: QueryClient; +}): Promise => { + const owner = captureAccountScope(); + const flight = flights.get(modelKey); + + const local = attemptsStore.getSnapshot().byModel.get(modelKey); + + if (local) { + updateAttempt(modelKey, local.attemptId, (attempt) => ({ ...attempt, stopped: true })); + } + + if (flight) { + flight.stopRequested = true; + flight.stop.abort(); + await flight.promise; + } + + const summary = await queryClient.fetchQuery({ + ...workingMemorySummaryQueryOptions(modelKey, { isConnected: true, isSubmitting: false }, owner), + staleTime: 0, + }); + const running = (summary?.runs ?? []) + .filter((run) => run.status === 'running') + .sort((left, right) => Date.parse(right.created_at) - Date.parse(left.created_at)); + + try { + for (const run of running) { + await cancelCalibrationRun(run.run_id, owner.signal); + } + } finally { + // Awaited, so a confirmation closing after this sees whether Calibrate is available again. + if (isAccountScopeCurrent(owner)) { + await invalidateAfterChange(queryClient, owner, modelKey); + } + } +}; + +/** Removes the model's runs and measurements; the shipped estimates apply again from the next generation on. */ +export const resetCalibration = async ({ + modelKey, + queryClient, +}: { + modelKey: string; + queryClient: QueryClient; +}): Promise => { + const owner = captureAccountScope(); + + await resetWorkingMemory(modelKey, owner.signal); + assertAccountScopeCurrent(owner); + setAttempt(modelKey, null); + await invalidateAfterChange(queryClient, owner, modelKey); +}; + +/** + * Keeps an open section current: a calibration item changing status refreshes the summaries, and a reconnect + * refreshes everything, since events during the gap were lost. Polling covers the time without a socket. + */ +export const attachWorkingMemoryRealtime = (queryClient: QueryClient): (() => void) => { + const owner = captureAccountScope(); + + const detachStatus = socketHub.on('queue_item_status_changed', (payload: never) => { + const origin = (payload as { origin?: unknown } | null)?.origin; + + if (isCalibrationOrigin(origin) && isAccountScopeCurrent(owner)) { + void invalidateAfterChange(queryClient, owner); + } + }); + + // The hub reports the current status on subscribe; only a later transition to connected is a reconnect. + let wasConnected: boolean | null = null; + const detachConnection = socketHub.onConnectionChange((status) => { + const isConnected = status === 'connected'; + + if (isConnected && wasConnected === false && isAccountScopeCurrent(owner)) { + void queryClient.invalidateQueries({ queryKey: workingMemoryKeys.forAccount(owner) }); + } + wasConnected = isConnected; + }); + + return () => { + detachStatus(); + detachConnection(); + }; +}; diff --git a/invokeai/frontend/webv2/src/features/models/data/workingMemoryQueries.ts b/invokeai/frontend/webv2/src/features/models/data/workingMemoryQueries.ts new file mode 100644 index 00000000000..bd9b4e73976 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/data/workingMemoryQueries.ts @@ -0,0 +1,76 @@ +import type { WorkingMemorySummary } from '@features/models/core/workingMemory'; + +import { assertAccountScopeCurrent, captureAccountScope, type AccountScope } from '@platform/state/accountLifecycle'; +import { ApiError } from '@platform/transport/http'; +import { queryOptions } from '@tanstack/react-query'; + +import { getWorkingMemoryPlan, getWorkingMemoryStatus, getWorkingMemorySummary } from './workingMemoryApi'; + +/** Polls while a run is pending or running and the socket is down, which includes a backend without one (the mock). */ +export const WORKING_MEMORY_POLL_MS = 2_000; +/** With a socket, its events drive updates; this only catches a missed one. */ +export const WORKING_MEMORY_FALLBACK_POLL_MS = 30_000; + +const accountKey = (owner: AccountScope) => ({ accountId: owner.accountId, epoch: owner.epoch }); + +/** Keys carry the account so a sign-out never serves one account's calibration to the next. */ +export const workingMemoryKeys = { + forAccount: (owner: AccountScope) => ['models', 'workingMemory', accountKey(owner)] as const, + plan: (owner: AccountScope, modelKey: string) => [...workingMemoryKeys.forAccount(owner), 'plan', modelKey] as const, + status: (owner: AccountScope) => [...workingMemoryKeys.forAccount(owner), 'status'] as const, + summaries: (owner: AccountScope) => [...workingMemoryKeys.forAccount(owner), 'summary'] as const, + summary: (owner: AccountScope, modelKey: string) => [...workingMemoryKeys.summaries(owner), modelKey] as const, +}; + +const fenced = (owner: AccountScope, request: (signal: AbortSignal) => Promise, signal: AbortSignal) => + request(AbortSignal.any([signal, owner.signal])).then((result) => { + assertAccountScopeCurrent(owner); + return result; + }); + +export const hasRunningRun = (summary: WorkingMemorySummary | null | undefined): boolean => + summary?.runs.some((run) => run.status === 'running') ?? false; + +/** The plan depends on installed models and the GPU, both of which change rarely. */ +export const workingMemoryPlanQueryOptions = (modelKey: string, owner = captureAccountScope()) => + queryOptions({ + queryFn: ({ signal }) => fenced(owner, (fencedSignal) => getWorkingMemoryPlan(modelKey, fencedSignal), signal), + queryKey: workingMemoryKeys.plan(owner, modelKey), + staleTime: 60_000, + }); + +/** Null when the server has no current device class to summarise; the status then says why. */ +export const workingMemorySummaryQueryOptions = ( + modelKey: string, + { isConnected, isSubmitting }: { isConnected: boolean; isSubmitting: boolean }, + owner = captureAccountScope() +) => + queryOptions({ + queryFn: ({ signal }) => + fenced( + owner, + (fencedSignal) => + getWorkingMemorySummary(modelKey, fencedSignal).catch((error: unknown) => { + if (error instanceof ApiError && error.status === 409) { + return null; + } + throw error; + }), + signal + ), + queryKey: workingMemoryKeys.summary(owner, modelKey), + refetchInterval: (query) => + isSubmitting || hasRunningRun(query.state.data) + ? isConnected + ? WORKING_MEMORY_FALLBACK_POLL_MS + : WORKING_MEMORY_POLL_MS + : false, + staleTime: 5_000, + }); + +export const workingMemoryStatusQueryOptions = (owner = captureAccountScope()) => + queryOptions({ + queryFn: ({ signal }) => fenced(owner, (fencedSignal) => getWorkingMemoryStatus(fencedSignal), signal), + queryKey: workingMemoryKeys.status(owner), + staleTime: 15_000, + }); diff --git a/invokeai/frontend/webv2/src/features/models/index.ts b/invokeai/frontend/webv2/src/features/models/index.ts index c0f818e38cc..e31ee7ff02d 100644 --- a/invokeai/frontend/webv2/src/features/models/index.ts +++ b/invokeai/frontend/webv2/src/features/models/index.ts @@ -30,7 +30,9 @@ export { useInstallActions } from './ui/add-models/useInstallActions'; export { getStarterModelInstallSources, type StarterInstallSource } from './ui/add-models/starterModelInstallSources'; export { ModelInstallRuntime } from './ui/ModelInstallRuntime'; export { ModelsPage } from './ui/ModelsPage'; +export type { CalibrationGraph, CalibrationGraphCompiler, WorkingMemoryCase } from './core/workingMemory'; export { + CalibrationGraphCompilerProvider, ModelsUiProvider, useOpenAddModelsSearch, useOpenModelInManager, diff --git a/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx b/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx index b6b1ed7cc43..fdfd797510e 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx +++ b/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx @@ -1,3 +1,4 @@ +import type { CalibrationGraphCompiler } from '@features/models/core/workingMemory'; import type { ReactNode } from 'react'; import { captureAccountScope, isAccountScopeCurrent } from '@platform/state/accountLifecycle'; @@ -28,6 +29,20 @@ export const ModelsUiProvider = ({ adapter, children }: { adapter: ModelsUiAdapt export const useModelsUi = (): ModelsUiAdapter => use(ModelsUiContext); +const CalibrationGraphCompilerContext = createContext(null); + +/** Supplies working-memory calibration with Generate's graph compiler, which App composes: Models cannot import it. */ +export const CalibrationGraphCompilerProvider = ({ + children, + compile, +}: { + children: ReactNode; + compile: CalibrationGraphCompiler; +}) => {children}; + +/** Null outside a provider; calibration is then unavailable. */ +export const useCalibrationGraphCompiler = (): CalibrationGraphCompiler | null => use(CalibrationGraphCompilerContext); + /** * Opens the Model Manager on a model's details, returning to the manager's project like its other entry points; null * when this session may not manage models. The manager's UI store loads on first use so editor boot does not carry it. diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.browser.test.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.browser.test.tsx new file mode 100644 index 00000000000..7298c8c9313 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.browser.test.tsx @@ -0,0 +1,78 @@ +import type { ModelConfig } from '@features/models/core/types'; + +import { ChakraProvider } from '@chakra-ui/react'; +import { setModelsSnapshotForTests } from '@features/models/data/modelsStore'; +import { accountLifecycle } from '@platform/state/accountLifecycle'; +import { system } from '@theme/system'; +import { act } from 'react'; +import { createRoot, type Root } from 'react-dom/client'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { ModelDetail } from './ModelDetail'; + +// The sections own their data; here they only mark where the page places them. +const { section } = vi.hoisted(() => ({ + section: (name: string) => () =>
, +})); + +vi.mock('@features/identity', async (importOriginal) => ({ + ...(await importOriginal()), + useCapabilities: () => ({ canManageModels: true }), +})); +vi.mock('./DefaultSettingsSection', () => ({ DefaultSettingsSection: section('default-settings') })); +vi.mock('./RelatedModelsSection', () => ({ RelatedModelsSection: section('related-models') })); +vi.mock('./TriggerPhrasesEditor', () => ({ MemoizedTriggerPhrasesEditor: section('trigger-phrases') })); +vi.mock('./WorkingMemorySection', () => ({ WorkingMemorySection: section('working-memory') })); +vi.mock('react-i18next', () => ({ useTranslation: () => ({ t: (key: string) => key }) })); + +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const MAIN = { + base: 'sdxl', + file_size: 1024, + format: 'checkpoint', + hash: 'hash-main', + key: 'main-1', + name: 'Main Model', + path: '/models/main.safetensors', + source: '/models/main.safetensors', + source_type: 'path', + trigger_phrases: [], + type: 'main', + variant: 'normal', +} as unknown as ModelConfig; + +const ignoreConfirm = () => undefined; + +describe('ModelDetail', () => { + let host: HTMLDivElement; + let root: Root; + + beforeEach(() => { + accountLifecycle.activate('model-detail-test', ':user:model-detail-test'); + setModelsSnapshotForTests({ models: [MAIN], status: 'loaded' }); + host = document.createElement('div'); + document.body.append(host); + root = createRoot(host); + }); + + afterEach(async () => { + await act(() => root.unmount()); + host.remove(); + setModelsSnapshotForTests({ models: [], status: 'idle' }); + accountLifecycle.invalidate(); + }); + + it('places working memory last, after the trigger phrases', async () => { + await act(() => + root.render( + + + + ) + ); + + const order = [...host.querySelectorAll('[data-section]')].map((element) => element.getAttribute('data-section')); + expect(order).toEqual(['default-settings', 'related-models', 'trigger-phrases', 'working-memory']); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx index a354aa5f32e..a3e4c9f1d40 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx @@ -3,10 +3,12 @@ import type { ModelConfig } from '@features/models/core/types'; import type { ReactNode } from 'react'; import { chakra, DataList, HStack, Icon, Menu, Portal, Separator, Stack, Text } from '@chakra-ui/react'; +import { useCapabilities } from '@features/identity'; import { isConvertibleToDiffusers } from '@features/models/core/baseIdentity'; import { isLinkableType } from '@features/models/core/relationships'; import { isAbsoluteModelPath, resolveModelAbsolutePath } from '@features/models/core/schemas'; import { getModelSourceHref } from '@features/models/core/taxonomy'; +import { supportsWorkingMemoryCalibration } from '@features/models/core/workingMemory'; import { useModelsSelector, type ModelsSnapshot } from '@features/models/data/modelsStore'; import { discardModelDraft, @@ -35,6 +37,7 @@ import { ModelSettingsMenuItems } from './ModelSettingsMenuItems'; import { RelatedModelsSection } from './RelatedModelsSection'; import { MemoizedTriggerPhrasesEditor } from './TriggerPhrasesEditor'; import { UpdatePathDialog } from './UpdatePathDialog'; +import { WorkingMemorySection } from './WorkingMemorySection'; const TRIGGER_PHRASE_TYPES = new Set(['main', 'lora', 'embedding']); const EMPTY_TRIGGER_PHRASES: readonly string[] = []; @@ -152,6 +155,7 @@ export const ModelDetail = ({ }) => { const { t } = useTranslation(); const model = useModelsSelector((snapshot) => selectModelShell(snapshot, modelKey)); + const { canManageModels } = useCapabilities(); if (!model) { return ( @@ -194,6 +198,14 @@ export const ModelDetail = ({ ) : null} + + {/* Last and collapsed: calibration is a rare, admin-only task. */} + {canManageModels && supportsWorkingMemoryCalibration(model) ? ( + <> + + + + ) : null} ); }; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryCalibrateDialog.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryCalibrateDialog.tsx new file mode 100644 index 00000000000..cc9707f2420 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryCalibrateDialog.tsx @@ -0,0 +1,130 @@ +/* eslint-disable react-perf/jsx-no-new-function-as-prop */ +import type { WorkingMemoryPlan } from '@features/models/core/workingMemory'; + +import { chakra, Portal, Stack, Text } from '@chakra-ui/react'; +import { planCalibrationSteps } from '@features/models/core/workingMemory'; +import { Button, CloseButton } from '@platform/ui/Button'; +import { Dialog } from '@platform/ui/Dialog'; +import { useCallback, useRef } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { SharedModelsList, useCaseLabel } from './WorkingMemoryParts'; + +const SECONDS_PER_MINUTE = 60; + +/** Confirms a calibration: what it runs, where, for how long, and which other models it moves. */ +export const WorkingMemoryCalibrateDialog = ({ + calibratedThroughName, + deviceName, + getFinalFocus, + isOpen, + onClose, + onConfirm, + onExitComplete, + plan, +}: { + /** The main model a standalone VAE's cases generate with. */ + calibratedThroughName: string | null; + deviceName: string | null; + /** Where focus goes once the dialog closes; starting a calibration can disable the opener. */ + getFinalFocus: () => HTMLElement | null; + isOpen: boolean; + onClose: () => void; + onConfirm: () => void; + onExitComplete: () => void; + plan: WorkingMemoryPlan; +}) => { + const { t } = useTranslation(); + const startRef = useRef(null); + const getInitialFocus = useCallback(() => startRef.current, []); + const runCount = planCalibrationSteps(plan).length; + const minutes = + plan.estimated_seconds === null ? null : Math.max(1, Math.round(plan.estimated_seconds / SECONDS_PER_MINUTE)); + const caseLabel = useCaseLabel(); + + return ( + { + if (!event.open) { + onClose(); + } + }} + > + + + + + + + {t('models.workingMemory.dialogTitle')} + {deviceName ? ( + + {t('models.workingMemory.dialogDescription', { device: deviceName })} + + ) : null} + + + + + + {minutes === null ? null : ( + {t('models.workingMemory.dialogDuration', { minutes })} + )} + {t('models.workingMemory.dialogQueueNote')} + {calibratedThroughName ? ( + + {t('models.workingMemory.dialogThrough', { model: calibratedThroughName })} + + ) : null} + + + + {t('models.workingMemory.dialogRuns', { count: runCount })} + + + {plan.cases.map((item) => ( + + {caseLabel(item)} + + ))} + + + {t('models.workingMemory.dialogRepetitions', { + measured: plan.repetitions.measured, + warmUp: plan.repetitions.warm_up, + })} + + + {plan.shared_with.length > 0 ? : null} + + + + + + + + + + + + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryParts.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryParts.tsx new file mode 100644 index 00000000000..ee51e17ef2d --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryParts.tsx @@ -0,0 +1,351 @@ +import type { SystemStyleObject } from '@chakra-ui/react'; +import type { + WorkingMemoryCase, + WorkingMemoryLine, + WorkingMemoryLineSize, + WorkingMemoryLineState, + WorkingMemoryPlan, +} from '@features/models/core/workingMemory'; + +import { Badge, chakra, HStack, Icon, Popover, Portal, Stack, Table, Text } from '@chakra-ui/react'; +import { getModelBaseLabel } from '@features/models/core/baseIdentity'; +import { describeFact, orderContributors, splitLine } from '@features/models/core/workingMemory'; +import { formatBytes, formatCount } from '@platform/i18n/languages'; +import { IconButton } from '@platform/ui/Button'; +import { PopoverContent } from '@platform/ui/Popover'; +import { Tooltip, useTooltipTriggerIds } from '@platform/ui/Tooltip'; +import { InfoIcon } from 'lucide-react'; +import { useTranslation } from 'react-i18next'; + +const STATE_PALETTES: Record = { + in_use: 'green', + insufficient: 'gray', + stale: 'orange', + unusable: 'red', +}; + +/** One state chip; the color repeats the text, never replaces it. */ +export const LineStateBadge = ({ state }: { state: WorkingMemoryLineState | null }) => { + const { t } = useTranslation(); + const label = + state === 'in_use' + ? t('models.workingMemory.stateInUse') + : state === 'stale' + ? t('models.workingMemory.stateStale') + : state === 'unusable' + ? t('models.workingMemory.stateUnusable') + : state === 'insufficient' + ? t('models.workingMemory.stateInsufficient') + : t('models.workingMemory.stateNone'); + + return ( + + {label} + + ); +}; + +/** A plan case as "768 × 768 · fp16". */ +export const useCaseLabel = () => { + const { t } = useTranslation(); + + return (item: Pick): string => + t('models.workingMemory.pair', { + detail: splitLine(item.line).dtype ?? item.line, + label: t('models.workingMemory.size', { height: item.height, width: item.width }), + }); +}; + +/** `decode_fp16` → "Decode · fp16". An operation this version has no name for shows as the server sends it. */ +export const useLineLabel = () => { + const { t } = useTranslation(); + + return (line: string): string => { + const { dtype, operation } = splitLine(line); + const operationLabel = + operation === 'decode' + ? t('models.workingMemory.operationDecode') + : operation === 'encode' + ? t('models.workingMemory.operationEncode') + : operation; + + return dtype ? t('models.workingMemory.pair', { detail: dtype, label: operationLabel }) : operationLabel; + }; +}; + +const MEGAPIXEL = 1_000_000; + +const formatMegapixels = (pixels: number): string => formatCount(Math.round((pixels / MEGAPIXEL) * 100) / 100); + +const TABLE_SX: SystemStyleObject = { + '& td, & th': { borderColor: 'border.subtle', fontSize: 'sm', px: '2', py: '1' }, + '& thead th': { + color: 'fg.muted', + fontSize: 'xs', + fontWeight: '700', + letterSpacing: 'wider', + textTransform: 'uppercase', + }, + '& tbody th': { fontWeight: '400', textAlign: 'start' }, + // Every row starts with its size as a `th`, so each `td` and every header after the first holds a number. + '& td, & thead th + th': { fontVariantNumeric: 'tabular-nums', textAlign: 'end' }, +}; + +/** A changed environment fact by a readable name; see `describeFact`. */ +const useFactLabel = () => { + const { t } = useTranslation(); + + return (key: string): string => { + const fact = describeFact(key); + + if ('name' in fact) { + return fact.name; + } + + switch (fact.translated) { + case 'allocator': + return t('models.workingMemory.facts.allocator'); + case 'driver': + return t('models.workingMemory.facts.driver'); + case 'invokeai': + return t('models.workingMemory.facts.invokeai'); + case 'osBuild': + return t('models.workingMemory.facts.osBuild'); + case 'sdpaGuard': + return t('models.workingMemory.facts.sdpaGuard'); + case 'torchBuild': + return t('models.workingMemory.facts.torchBuild'); + } + }; +}; + +/** The state help of a line: why it applies or not, naming the facts that changed or differ. */ +const LineStateHelp = ({ line }: { line: WorkingMemoryLine }) => { + const { t } = useTranslation(); + const factLabel = useFactLabel(); + const facts = line.changed_facts.map(factLabel).join(', '); + const text = + line.state === 'in_use' + ? t('models.workingMemory.helpInUse') + : line.state === 'stale' + ? t('models.workingMemory.helpStale', { facts }) + : line.state === 'unusable' + ? facts + ? t('models.workingMemory.helpUnusable', { facts }) + : t('models.workingMemory.helpUnusableNoFacts') + : (line.problem ?? t('models.workingMemory.helpInsufficient')); + + return ( + + {text} + + ); +}; + +const HINT_POSITIONING = { placement: 'top' } as const; + +/** A column header with an info button after its label. */ +const HintedColumnHeader = ({ hint, label }: { hint: string; label: string }) => { + const { t } = useTranslation(); + const name = t('models.workingMemory.aboutColumn', { column: label }); + // The button stays on the line of the label's last word when a narrow column wraps the label. + const split = label.lastIndexOf(' ') + 1; + + return ( + + {label.slice(0, split)} + + {label.slice(split)} + + + + ); +}; + +/** The explanation opens in a popover, by click or keyboard, so it does not depend on hover. */ +const HintButton = ({ hint, name }: { hint: string; name: string }) => { + const ids = useTooltipTriggerIds(); + + return ( + + + + + + + + + + + + {hint} + + + + + ); +}; + +/** Why a measured size does not feed the line, or null for a size that does. */ +const useNotCountedLabel = () => { + const { t } = useTranslation(); + + return (size: Pick): string | null => { + if (size.counted) { + return null; + } + + switch (size.excluded_reason) { + case 'fragmented': + return t('models.workingMemory.notCountedFragmented'); + case 'partial_residency': + return t('models.workingMemory.notCountedPartial'); + case null: + return t('models.workingMemory.notCounted'); + } + }; +}; + +/** + * One estimator line: its state, the models whose measurements it pools (named when another model is among them, + * this model first), the range the user line was fitted on, and every measured size. + */ +export const LineMeasurements = ({ line, modelKey }: { line: WorkingMemoryLine; modelKey: string }) => { + const { t } = useTranslation(); + const lineLabel = useLineLabel()(line.line); + const notCountedLabel = useNotCountedLabel(); + const isShared = line.measured_by.some((entry) => entry.model_key !== modelKey); + const contributors = orderContributors(line.measured_by, modelKey) + .map((entry) => entry.name ?? t('models.workingMemory.removedModel')) + .join(', '); + + return ( + + + + {lineLabel} + + + + + {isShared ? ( + + {t('models.workingMemory.measuredBy', { models: contributors })} + + ) : null} + {line.measured_range ? ( + + {t('models.workingMemory.measuredRange', { + max: formatMegapixels(line.measured_range.max_pixels), + min: formatMegapixels(line.measured_range.min_pixels), + })} + + ) : null} + {line.sizes.length > 0 ? ( + + + + + + + + + + + {t('models.workingMemory.columnSize')} + + + + + + + {line.sizes.map((size) => { + // A size the fit leaves out stays listed, muted, and names why in its row header. + const notCounted = notCountedLabel(size); + const color = notCounted ? 'fg.muted' : undefined; + + return ( + + + {t('models.workingMemory.size', { height: size.height, width: size.width })} + {notCounted ? ( + + {notCounted} + + ) : null} + + + {formatBytes(size.measured_bytes)} + + + {formatBytes(size.shipped_bytes)} + + + {formatBytes(size.user_bytes)} + + + ); + })} + + + + ) : null} + + ); +}; + +/** Other installed models whose estimates the same lines price, so a calibration moves them too. */ +export const SharedModelsList = ({ models }: { models: WorkingMemoryPlan['shared_with'] }) => { + const { t } = useTranslation(); + const lineLabel = useLineLabel(); + + return ( + + + {t('models.workingMemory.sharedWith', { count: models.length })} + + + {models.map((model) => ( + + {t('models.workingMemory.pair', { + detail: [getModelBaseLabel(model.base), ...model.lines.map(lineLabel)].join(', '), + label: model.name, + })} + + ))} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.browser.test.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.browser.test.tsx new file mode 100644 index 00000000000..c413c6bc595 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.browser.test.tsx @@ -0,0 +1,942 @@ +import type { + CalibrationGraphCompiler, + WorkingMemoryCase, + WorkingMemoryLine, + WorkingMemoryPlan, + WorkingMemoryRun, + WorkingMemoryStatus, + WorkingMemorySummary, +} from '@features/models/core/workingMemory'; + +import { ChakraProvider } from '@chakra-ui/react'; +import { CalibrationGraphCompilerProvider } from '@features/models/ui/ModelsUiContext'; +import { accountLifecycle } from '@platform/state/accountLifecycle'; +import { ApiError } from '@platform/transport/http'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import { system } from '@theme/system'; +import { act } from 'react'; +import { createRoot, type Root } from 'react-dom/client'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { userEvent } from 'vitest/browser'; + +const api = vi.hoisted(() => ({ + acknowledgeCalibrationEnqueue: vi.fn((_idempotencyKey: string) => Promise.resolve()), + calibrateWorkingMemory: vi.fn(), + cancelCalibrationRun: vi.fn((_runId: string) => Promise.resolve()), + getWorkingMemoryPlan: vi.fn(), + getWorkingMemoryStatus: vi.fn(), + getWorkingMemorySummary: vi.fn(), + resetWorkingMemory: vi.fn((_modelKey: string, _signal?: AbortSignal) => Promise.resolve()), +})); + +vi.mock('@features/models/data/workingMemoryApi', () => api); +vi.mock('react-i18next', () => ({ + useTranslation: () => ({ + t: (key: string, options?: Record) => + options && Object.keys(options).length > 0 + ? `${key}(${Object.entries(options) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([name, value]) => `${name}=${String(value)}`) + .join(',')})` + : key, + }), +})); + +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const MODEL_KEY = 'sdxl-main'; +const DEVICE = { id: '0123456789abcdef', name: 'Fixture GPU 24GB' }; +const T0 = Date.parse('2026-10-01T12:00:00.000Z'); + +const planCase = (dtype: string, side: number): WorkingMemoryCase => ({ + components: {}, + estimator: 'sdxl_vae_decode', + height: side, + id: `sdxl_vae_decode-decode-${dtype}-${String(side)}x${String(side)}`, + line: `decode_${dtype}`, + mode: 'txt2img', + model_key: MODEL_KEY, + negative_prompt: '', + positive_prompt: 'a lake', + seed: 1, + settings: { vaePrecision: dtype }, + steps: 2, + width: side, +}); + +const CASES = [planCase('fp16', 512), planCase('fp32', 512)]; + +const supportedPlan = (): WorkingMemoryPlan => ({ + calibrated_through: null, + cases: CASES, + device_class: DEVICE, + estimated_seconds: 150, + estimators: ['sdxl_vae_decode'], + model_key: MODEL_KEY, + not_covered: ['encode: not measured'], + reason: null, + repetitions: { measured: 2, warm_up: 1 }, + shared_with: [{ base: 'sd-1', key: 'sd1', lines: ['decode_fp16'], name: 'Shared SD1', type: 'main' }], + supported: true, +}); + +const status = (overrides: Partial = {}): WorkingMemoryStatus => ({ + available: true, + current_device_class_id: DEVICE.id, + current_env_id: 'env', + device_classes: [], + unavailable_reason: null, + ...overrides, +}); + +const run = ( + id: string, + offset: number, + runStatus: WorkingMemoryRun['status'], + error: string | null = null, + attemptId: string | null = 'earlier-attempt' +): WorkingMemoryRun => ({ + attempt_id: attemptId, + created_at: new Date(T0 + offset * 1_000).toISOString(), + error, + point_count: runStatus === 'completed' ? 1 : 0, + run_id: id, + status: runStatus, +}); + +const OWN = { model_key: MODEL_KEY, name: 'This SDXL', windows: 6 }; + +const line = ( + state: WorkingMemoryLine['state'], + changedFacts: string[] = [], + overrides: Partial = {} +): WorkingMemoryLine => ({ + base_bytes: 400_000_000, + carrier: 'sdxl_vae_decode', + changed_facts: changedFacts, + env_id: 'env', + estimator: 'sdxl_vae_decode', + line: 'decode_fp16', + measured_by: [OWN], + measured_range: { max_pixels: 1_048_576, min_pixels: 262_144 }, + problem: null, + sizes: [ + { + counted: true, + excluded_reason: null, + height: 512, + measured_bytes: 1_234_000_000, + pixels: 262_144, + shipped_bytes: null, + user_bytes: null, + width: 512, + windows: 3, + }, + { + counted: true, + excluded_reason: null, + height: 1024, + measured_bytes: 2_100_000_000, + pixels: 1_048_576, + shipped_bytes: 2_800_000_000, + user_bytes: 2_600_000_000, + width: 1024, + windows: 3, + }, + ], + own_windows: 6, + slope_bytes_per_pixel: 1_100, + state, + version: 1, + ...overrides, +}); + +const summary = (overrides: Partial = {}): WorkingMemorySummary => ({ + current_env_id: 'env', + device_class_id: DEVICE.id, + excluded_windows: 0, + lines: [], + measurement_method: 'allocator', + measurement_verified: true, + model_key: MODEL_KEY, + problem: null, + runs: [], + usable: true, + ...overrides, +}); + +let host: HTMLDivElement; +let root: Root; +let queryClient: QueryClient; +let compile: ReturnType>; +let currentSummary: WorkingMemorySummary | Error; + +/** Renders the section, opened as a user would unless `expanded` is false: it starts collapsed. */ +const renderSection = async ({ expanded = true }: { expanded?: boolean } = {}) => { + host = document.createElement('div'); + host.style.width = '720px'; + document.body.append(host); + root = createRoot(host); + queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + const { WorkingMemorySection } = await import('./WorkingMemorySection'); + + await act(() => + root.render( + + + + + + + + ) + ); + + if (expanded) { + await act(() => userEvent.click(sectionToggle())); + } +}; + +/** The disclosure button inside the section heading. */ +const sectionToggle = () => host.querySelector('h3 button')!; + +const buttonNamed = (name: string, within: ParentNode = document): HTMLButtonElement | undefined => + [...within.querySelectorAll('button')].find((button) => button.textContent.trim() === name); + +const openDialog = () => document.querySelector('[role="dialog"][data-state="open"]'); +const openAlertDialog = () => document.querySelector('[role="alertdialog"][data-state="open"]'); +const sectionText = () => host.textContent; + +/** The summary the section reads next; a poll or an invalidation picks it up. */ +const serveSummary = (next: WorkingMemorySummary | Error) => { + currentSummary = next; +}; + +const refreshSummary = () => act(() => queryClient.invalidateQueries()); + +beforeEach(() => { + document.documentElement.lang = 'en'; + accountLifecycle.activate('wm-section-test', ':user:wm-section-test'); + compile = vi.fn((cases) => + Promise.resolve(cases.map((item) => ({ caseId: item.id, data: [], graph: { edges: [], id: item.id, nodes: {} } }))) + ); + currentSummary = summary(); + api.getWorkingMemoryPlan.mockImplementation(() => Promise.resolve(supportedPlan())); + api.getWorkingMemoryStatus.mockImplementation(() => Promise.resolve(status())); + api.getWorkingMemorySummary.mockImplementation(() => + currentSummary instanceof Error ? Promise.reject(currentSummary) : Promise.resolve(currentSummary) + ); + let runNumber = 0; + api.calibrateWorkingMemory.mockImplementation(() => { + runNumber += 1; + return Promise.resolve({ device_class_id: DEVICE.id, item_ids: [runNumber], run_id: `run-${String(runNumber)}` }); + }); +}); + +afterEach(async () => { + await act(() => root.unmount()); + host.remove(); + accountLifecycle.invalidate(); + vi.clearAllMocks(); +}); + +describe('WorkingMemorySection', () => { + it('calibrates a model that was never calibrated: dialog, queued runs, progress, then the measurements', async () => { + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.neverCalibrated'); + + const section = host.querySelector('section')!; + expect(section.getAttribute('aria-labelledby')).toBe(host.querySelector('h3')!.id); + expect(host.querySelector('h3')!.textContent).toBe('models.workingMemory.title'); + expect(sectionText()).toContain(DEVICE.name); + expect(sectionText()).toContain('models.workingMemory.stateNone'); + // The compiler loads only once a calibration starts. + expect(compile).not.toHaveBeenCalled(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + const dialog = openDialog()!; + expect(dialog.textContent).toContain('models.workingMemory.dialogDuration(minutes=3)'); + expect(dialog.textContent).toContain('models.workingMemory.dialogQueueNote'); + expect(dialog.textContent).toContain( + 'models.workingMemory.pair(detail=fp16,label=models.workingMemory.size(height=512,width=512))' + ); + expect(dialog.textContent).toContain( + 'models.workingMemory.pair(detail=fp32,label=models.workingMemory.size(height=512,width=512))' + ); + expect(dialog.textContent).toContain('models.workingMemory.dialogRuns(count=6)'); + expect(dialog.textContent).toContain('Shared SD1'); + expect(compile).not.toHaveBeenCalled(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + + expect(compile).toHaveBeenCalledTimes(1); + expect(compile).toHaveBeenCalledWith(CASES, expect.any(AbortSignal)); + const requests = api.calibrateWorkingMemory.mock.calls.map( + ([key, request]) => [key, (request as { graph: { id: string } }).graph.id] as const + ); + // Each precision warms up first: fp16 warm-up, two measured, then the same in fp32. + expect(requests).toEqual([ + [MODEL_KEY, CASES[0]!.id], + [MODEL_KEY, CASES[0]!.id], + [MODEL_KEY, CASES[0]!.id], + [MODEL_KEY, CASES[1]!.id], + [MODEL_KEY, CASES[1]!.id], + [MODEL_KEY, CASES[1]!.id], + ]); + const keys = api.calibrateWorkingMemory.mock.calls.map( + ([, request]) => (request as { idempotencyKey: string }).idempotencyKey + ); + expect(new Set(keys).size).toBe(6); + // One attempt id for the click, on every step and inside every retry key. + const attemptIds = new Set( + api.calibrateWorkingMemory.mock.calls.map(([, request]) => (request as { attemptId: string }).attemptId) + ); + expect(attemptIds.size).toBe(1); + const [attemptId] = [...attemptIds]; + expect(keys.every((key) => key.includes(attemptId!))).toBe(true); + expect(api.acknowledgeCalibrationEnqueue.mock.calls.map(([key]) => key)).toEqual(keys); + + serveSummary( + summary({ + runs: [ + run('run-1', 0, 'completed', null, attemptId), + run('run-2', 1, 'completed', null, attemptId), + ...[3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'running', null, attemptId)), + ], + }) + ); + await refreshSummary(); + await expect.poll(sectionText).toContain('models.workingMemory.running(done=2,total=6)'); + expect(host.querySelector('[role="progressbar"]')?.getAttribute('aria-label')).toBe( + 'models.workingMemory.progressLabel' + ); + expect(sectionText()).toContain( + 'models.workingMemory.pair(detail=fp16,label=models.workingMemory.size(height=512,width=512))' + ); + expect(buttonNamed('models.workingMemory.recalibrate')?.disabled).toBe(true); + expect(sectionText()).toContain('models.workingMemory.blockedRunning'); + // Reset names the same reason it is disabled for. + const reset = buttonNamed('common.reset')!; + expect(reset.disabled).toBe(true); + expect(document.getElementById(reset.getAttribute('aria-describedby')!)?.textContent).toBe( + 'models.workingMemory.blockedRunning' + ); + + serveSummary( + summary({ + lines: [line('in_use')], + // An earlier attempt failed; the newest attempt is the one that counts. + runs: [ + run('old', -600, 'failed', 'old failure'), + ...[1, 2, 3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'completed', null, attemptId)), + ], + }) + ); + // Polling picks the end up without any other trigger. + await expect.poll(() => host.querySelector('table'), { timeout: 5_000 }).not.toBeNull(); + + const table = host.querySelector('table')!; + expect(table.getAttribute('aria-label')).toBe( + 'models.workingMemory.tableLabel(line=models.workingMemory.pair(detail=fp16,label=models.workingMemory.operationDecode))' + ); + const rows = [...table.querySelectorAll('tbody tr')].map((row) => + [...row.querySelectorAll('th, td')].map((cell) => cell.textContent) + ); + expect([...table.querySelectorAll('tbody th')].map((cell) => cell.getAttribute('scope'))).toEqual(['row', 'row']); + expect(table.querySelectorAll('thead th')[3]?.textContent).toBe('models.workingMemory.columnCalibrated'); + // The size labels its row on the left; every value sits right-aligned under its header. + const alignments = (selector: string) => + [...table.querySelectorAll(selector)].map((cell) => getComputedStyle(cell).textAlign); + expect(alignments('thead th')).toEqual(['start', 'end', 'end', 'end']); + expect(alignments('tbody tr:first-child > *')).toEqual(['start', 'end', 'end', 'end']); + // Missing estimates show a dash; sizes format like the rest of the app. + expect(rows).toEqual([ + ['models.workingMemory.size(height=512,width=512)', '1.2 GB', '—', '—'], + ['models.workingMemory.size(height=1024,width=1024)', '2.1 GB', '2.8 GB', '2.6 GB'], + ]); + expect(sectionText()).toContain('models.workingMemory.measuredRange(max=1.05,min=0.26)'); + expect(sectionText()).toContain('models.workingMemory.stateInUse'); + expect(host.querySelector('[role="progressbar"]')).toBeNull(); + }); + + it('operates the dialog by keyboard and returns focus to its opener', async () => { + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + const opener = buttonNamed('models.workingMemory.calibrate')!; + opener.focus(); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(openDialog).not.toBeNull(); + await expect + .poll(() => openDialog()?.contains(document.activeElement) && document.activeElement?.textContent) + .toBe('models.workingMemory.calibrate'); + expect(openDialog()!.querySelector('h2, [id$="title"]')?.textContent).toBe('models.workingMemory.dialogTitle'); + + await act(() => userEvent.keyboard('{Escape}')); + await expect.poll(openDialog).toBeNull(); + await expect.poll(() => document.activeElement).toBe(opener); + expect(api.calibrateWorkingMemory).not.toHaveBeenCalled(); + expect(compile).not.toHaveBeenCalled(); + + await act(() => userEvent.keyboard('{Enter}')); + await expect + .poll(() => openDialog()?.contains(document.activeElement) && document.activeElement?.textContent) + .toBe('models.workingMemory.calibrate'); + // Hold the compiler so the attempt stays in its running state while focus settles. + let releaseCompile = () => undefined as void; + const compiled = new Promise((resolve) => { + releaseCompile = resolve; + }); + const compileCases = compile.getMockImplementation()!; + compile.mockImplementationOnce(async (cases) => { + await compiled; + return compileCases(cases, new AbortController().signal); + }); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(openDialog).toBeNull(); + // Starting disables the opener; focus moves to the action that is available now instead of the page. + await expect.poll(() => document.activeElement?.textContent).toBe('models.workingMemory.stop'); + expect(host.contains(document.activeElement)).toBe(true); + + await act(() => releaseCompile()); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + }); + + it('stops a running calibration after confirmation, newest run first', async () => { + serveSummary( + summary({ runs: [run('run-1', 0, 'completed'), run('run-2', 1, 'running'), run('run-3', 2, 'running')] }) + ); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.stop')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + expect(openAlertDialog()!.textContent).toContain('models.workingMemory.stopBody'); + expect(api.cancelCalibrationRun).not.toHaveBeenCalled(); + + api.cancelCalibrationRun.mockImplementation(() => { + serveSummary( + summary({ runs: [run('run-1', 0, 'completed'), run('run-2', 1, 'partial'), run('run-3', 2, 'canceled')] }) + ); + return Promise.resolve(); + }); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop', openAlertDialog()!)!)); + await expect.poll(openAlertDialog).toBeNull(); + + expect(api.cancelCalibrationRun.mock.calls.map(([runId]) => runId)).toEqual(['run-3', 'run-2']); + await expect.poll(sectionText).toContain('models.workingMemory.partial'); + // The Stop button is gone; focus lands on the action that is available now. + await expect.poll(() => document.activeElement?.textContent).toBe('common.retry'); + }); + + it('reports an attempt stopped while queuing as canceled, not as queued short', async () => { + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + // The third step's response arrives only once the stop is under way. + let releasePost = () => undefined as void; + const held = new Promise((resolve) => { + releasePost = resolve; + }); + const post = api.calibrateWorkingMemory.getMockImplementation()!; + api.calibrateWorkingMemory.mockImplementation(async (...args: unknown[]) => { + if (api.calibrateWorkingMemory.mock.calls.length === 3) { + await held; + } + return post(...args); + }); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(3); + + const attemptId = (api.calibrateWorkingMemory.mock.calls[0]![1] as { attemptId: string }).attemptId; + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + serveSummary( + summary({ runs: [1, 2, 3].map((number) => run(`run-${String(number)}`, number, 'completed', null, attemptId)) }) + ); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop', openAlertDialog()!)!)); + await act(() => releasePost()); + await expect.poll(openAlertDialog).toBeNull(); + + expect(api.calibrateWorkingMemory).toHaveBeenCalledTimes(3); + await expect.poll(sectionText).toContain('models.workingMemory.partialCanceled'); + expect(sectionText()).not.toContain('models.workingMemory.partialQueued'); + }); + + it('stops an attempt that is still compiling: the compiler is aborted and nothing is queued', async () => { + compile.mockImplementationOnce( + (_cases, signal) => + new Promise((_resolve, reject) => { + signal.addEventListener('abort', () => reject(new Error('aborted')), { once: true }); + }) + ); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => buttonNamed('models.workingMemory.stop')).toBeDefined(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop', openAlertDialog()!)!)); + + // The attempt left queuing: it reads as stopped, not as a failure, and Calibrate is available again. + await expect.poll(sectionText).toContain('models.workingMemory.canceled'); + expect(sectionText()).not.toContain('models.workingMemory.failed'); + expect(buttonNamed('models.workingMemory.stop', host)).toBeUndefined(); + expect(buttonNamed('models.workingMemory.calibrate')?.disabled).toBe(false); + expect(api.calibrateWorkingMemory).not.toHaveBeenCalled(); + }); + + it('heads the section with the current environment’s state, not an old environment’s', async () => { + serveSummary( + summary({ + lines: [line('unusable', ['torch'], { env_id: 'old-env' }), line('in_use')], + runs: [run('run-1', 0, 'completed')], + }) + ); + await renderSection(); + await expect.poll(() => host.querySelectorAll('table').length).toBe(2); + + const header = [...host.querySelectorAll('p')].find( + (element) => element.textContent === DEVICE.name + )!.parentElement!; + expect(header.textContent).toBe(`${DEVICE.name}models.workingMemory.stateInUse`); + expect(sectionText()).not.toContain('models.workingMemory.neverCalibrated'); + + // With only an old environment's lines, they head it, and this GPU counts as not calibrated. + serveSummary(summary({ lines: [line('unusable', ['torch'], { env_id: 'old-env' })] })); + await refreshSummary(); + await expect.poll(() => header.textContent).toBe(`${DEVICE.name}models.workingMemory.stateUnusable`); + expect(sectionText()).toContain('models.workingMemory.neverCalibrated'); + }); + + it('shows lines other models measured as in use, naming them, with nothing of this model to reset', async () => { + const other = { model_key: 'other', name: 'Other SDXL', windows: 6 }; + serveSummary(summary({ lines: [line('in_use', [], { measured_by: [other], own_windows: 0 })] })); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.usesOthers'); + + expect(sectionText()).toContain('models.workingMemory.measuredBy(models=Other SDXL)'); + expect(sectionText()).not.toContain('models.workingMemory.neverCalibrated'); + expect(buttonNamed('common.reset')).toBeUndefined(); + expect(buttonNamed('models.workingMemory.calibrate')?.disabled).toBe(false); + + // Once this model measured too, it is listed first and has measurements to reset. + serveSummary( + summary({ + lines: [line('in_use', [], { measured_by: [other, OWN], own_windows: 6 })], + runs: [run('run-1', 0, 'completed')], + }) + ); + await refreshSummary(); + await expect.poll(sectionText).toContain('models.workingMemory.measuredBy(models=This SDXL, Other SDXL)'); + expect(sectionText()).not.toContain('models.workingMemory.usesOthers'); + expect(buttonNamed('common.reset')).toBeDefined(); + }); + + it('keeps a finished attempt completed through a lost connection to the server', async () => { + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + + const attemptId = (api.calibrateWorkingMemory.mock.calls[0]![1] as { attemptId: string }).attemptId; + const finished = summary({ + lines: [line('in_use')], + runs: [1, 2, 3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'completed', null, attemptId)), + }); + serveSummary(finished); + await refreshSummary(); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + + // The server goes away: every read fails as the browser reports an unreachable host. + const unreachable = () => Promise.reject(new TypeError('Failed to fetch')); + api.getWorkingMemoryPlan.mockImplementation(unreachable); + api.getWorkingMemoryStatus.mockImplementation(unreachable); + serveSummary(new TypeError('Failed to fetch')); + await refreshSummary(); + await expect.poll(sectionText).toContain('models.workingMemory.loadFailed'); + + // It comes back with the same runs. + api.getWorkingMemoryPlan.mockImplementation(() => Promise.resolve(supportedPlan())); + api.getWorkingMemoryStatus.mockImplementation(() => Promise.resolve(status())); + serveSummary(finished); + await refreshSummary(); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + expect(sectionText()).not.toContain('models.workingMemory.failed'); + expect(sectionText()).not.toContain('Failed to fetch'); + }); + + it('lets an attempt that could not reach the server give way to one that ran after it', async () => { + const earlier = run('old', 0, 'completed', null, 'old'); + serveSummary(summary({ lines: [line('in_use')], runs: [earlier] })); + compile.mockImplementationOnce(() => Promise.reject(new TypeError('Failed to fetch'))); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.recalibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(sectionText).toContain('models.workingMemory.failed'); + expect(sectionText()).toContain('Failed to fetch'); + + // Another tab, or a script, then calibrated to the end: that attempt is the latest one now. + serveSummary( + summary({ + lines: [line('in_use')], + runs: [ + earlier, + ...[1, 2, 3, 4, 5, 6].map((number) => run(`theirs-${String(number)}`, 100 + number, 'completed')), + ], + }) + ); + await refreshSummary(); + await expect.poll(sectionText).not.toContain('models.workingMemory.failed'); + expect(sectionText()).not.toContain('Failed to fetch'); + expect(buttonNamed('models.workingMemory.recalibrate')?.disabled).toBe(false); + }); + + it('keeps a size the fit leaves out in the table, muted, naming why, in columns the other tables share', async () => { + const counted = line('in_use').sizes; + serveSummary( + summary({ + lines: [ + // Its row headers are narrower than the other table's, as a size note makes them in any language. + line('in_use', [], { sizes: counted.map((size) => ({ ...size, height: 64, width: 64 })) }), + line('in_use', [], { + line: 'decode_fp32', + sizes: [ + ...counted, + { + counted: false, + excluded_reason: 'fragmented', + height: 1216, + measured_bytes: 5_900_000_000, + pixels: 1_478_656, + shipped_bytes: 4_100_000_000, + user_bytes: 3_300_000_000, + width: 1216, + windows: 3, + }, + ], + }), + ], + runs: [run('run-1', 0, 'completed')], + }) + ); + await renderSection(); + await expect.poll(() => host.querySelectorAll('table').length).toBe(2); + + const [plain, mixed] = [...host.querySelectorAll('table')]; + const [first, , excluded] = [...mixed!.querySelectorAll('tbody tr')]; + // The reason is part of the row header, so it is read with the size. + expect(excluded!.querySelector('th')!.textContent).toBe( + 'models.workingMemory.size(height=1216,width=1216)models.workingMemory.notCountedFragmented' + ); + expect(first!.querySelector('th')!.textContent).toBe('models.workingMemory.size(height=512,width=512)'); + const colors = (row: Element) => [...row.children].map((cell) => getComputedStyle(cell).color); + expect(new Set(colors(excluded!)).size).toBe(1); + expect(colors(excluded!)[0]).not.toBe(colors(first!)[0]); + expect([...excluded!.children].map((cell) => getComputedStyle(cell).textAlign)).toEqual([ + 'start', + 'end', + 'end', + 'end', + ]); + // The note under the size does not widen its column: both tables' columns start at the same place. + const columnEdges = (table: Element) => + [...table.querySelectorAll('thead th')].map((cell) => Math.round(cell.getBoundingClientRect().left)); + expect(columnEdges(mixed!)).toEqual(columnEdges(plain!)); + }); + + it('explains each value column from an info button that opens by keyboard', async () => { + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection(); + await expect.poll(() => host.querySelector('table')).not.toBeNull(); + + const headers = [...host.querySelectorAll('thead th')]; + // The size column needs no explanation. + expect(headers[0]!.querySelector('button')).toBeNull(); + + for (const [index, column, hint] of [ + [1, 'columnMeasured', 'hintMeasured'], + [2, 'columnShipped', 'hintShipped'], + [3, 'columnCalibrated', 'hintCalibrated'], + ] as const) { + const button = headers[index]!.querySelector('button')!; + const name = `models.workingMemory.aboutColumn(column=models.workingMemory.${column})`; + expect(button.getAttribute('aria-label')).toBe(name); + + button.focus(); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(openDialog).not.toBeNull(); + expect(openDialog()!.getAttribute('aria-label')).toBe(name); + expect(openDialog()!.textContent).toBe(`models.workingMemory.${hint}`); + + await act(() => userEvent.keyboard('{Escape}')); + await expect.poll(openDialog).toBeNull(); + await expect.poll(() => document.activeElement).toBe(button); + } + }); + + it('starts collapsed, keeps its panel out of the tab order, and asks for the plan only once opened', async () => { + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection({ expanded: false }); + + const toggle = sectionToggle(); + expect(toggle.getAttribute('aria-expanded')).toBe('false'); + const panel = document.getElementById(toggle.getAttribute('aria-controls')!)!; + expect(panel.hidden).toBe(true); + expect(buttonNamed('models.workingMemory.recalibrate')).toBeUndefined(); + // The plan reads the GPU's free memory on the server; a collapsed section never asks for it. + await expect.poll(() => api.getWorkingMemorySummary.mock.calls.length).toBeGreaterThan(0); + expect(api.getWorkingMemoryPlan).not.toHaveBeenCalled(); + + toggle.focus(); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + expect(toggle.getAttribute('aria-expanded')).toBe('true'); + expect(panel.hidden).toBe(false); + expect(api.getWorkingMemoryPlan).toHaveBeenCalledTimes(1); + + await act(() => userEvent.keyboard('[Space]')); + await expect.poll(() => panel.hidden).toBe(true); + expect(toggle.getAttribute('aria-expanded')).toBe('false'); + expect([...panel.querySelectorAll('button')].every((button) => button.getClientRects().length === 0)).toBe(true); + await act(() => userEvent.tab()); + expect(panel.contains(document.activeElement)).toBe(false); + }); + + it('names the GPU and the state in the collapsed header, and the progress while a calibration runs', async () => { + api.getWorkingMemoryStatus.mockImplementation(() => + Promise.resolve( + status({ + device_classes: [ + { + device_class: { + arch: null, + backend: 'cuda', + id: DEVICE.id, + name: DEVICE.name, + total_mib: 24_576, + vendor: 'x', + }, + device_class_id: DEVICE.id, + envs: [], + is_current: true, + measurement_method: 'allocator', + measurement_verified: true, + point_counts: [], + problem: null, + runs: [], + usable: true, + }, + ], + }) + ) + ); + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection({ expanded: false }); + + const header = sectionToggle().closest('h3')!.parentElement!; + await expect + .poll(() => header.textContent) + .toBe(`models.workingMemory.title${DEVICE.name}models.workingMemory.stateInUse`); + + // The attempt id carries its step count, so the header needs no plan to count. + const attemptId = 'attempt.6'; + serveSummary( + summary({ + runs: [ + run('run-1', 1, 'completed', null, attemptId), + run('run-2', 2, 'completed', null, attemptId), + ...[3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'running', null, attemptId)), + ], + }) + ); + await refreshSummary(); + await expect.poll(() => header.textContent).toContain('models.workingMemory.running(done=2,total=6)'); + expect(header.textContent).not.toContain('models.workingMemory.stateInUse'); + // A running calibration does not open the section. + expect(sectionToggle().getAttribute('aria-expanded')).toBe('false'); + expect(api.getWorkingMemoryPlan).not.toHaveBeenCalled(); + }); + + it('says the model was removed when the server no longer knows it', async () => { + serveSummary(new ApiError('{"detail":"The model could not be found"}', 404)); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.modelRemoved'); + + expect(sectionText()).not.toContain('models.workingMemory.loadFailed'); + expect(buttonNamed('common.retry')).toBeUndefined(); + }); + + it('resets after confirmation and restores focus', async () => { + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection(); + await expect.poll(() => buttonNamed('common.reset')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('common.reset')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + // A destructive confirmation opens on Cancel. + expect(document.activeElement?.textContent).toBe('Cancel'); + + api.resetWorkingMemory.mockImplementation(() => { + serveSummary(summary()); + return Promise.resolve(); + }); + await act(() => userEvent.click(buttonNamed('common.reset', openAlertDialog()!)!)); + await expect.poll(openAlertDialog).toBeNull(); + + expect(api.resetWorkingMemory).toHaveBeenCalledWith(MODEL_KEY, expect.anything()); + await expect.poll(sectionText).toContain('models.workingMemory.neverCalibrated'); + await expect.poll(() => document.activeElement?.textContent).toBe('models.workingMemory.calibrate'); + }); + + it('disables Calibrate for an unsupported model, naming the reason', async () => { + api.getWorkingMemoryPlan.mockImplementation(() => + Promise.resolve({ + ...supportedPlan(), + cases: [], + reason: 'The inpaint variant does not generate from text alone.', + supported: false, + }) + ); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + const button = buttonNamed('models.workingMemory.calibrate')!; + expect(button.disabled).toBe(true); + expect(document.getElementById(button.getAttribute('aria-describedby')!)?.textContent).toBe( + 'The inpaint variant does not generate from text alone.' + ); + expect(sectionText()).not.toContain('models.workingMemory.neverCalibrated'); + }); + + it('disables Calibrate without exactly one GPU, naming the status reason', async () => { + api.getWorkingMemoryStatus.mockImplementation(() => + Promise.resolve( + status({ + available: false, + current_device_class_id: null, + unavailable_reason: 'Calibration needs exactly one generation device; this machine has two.', + }) + ) + ); + serveSummary(new ApiError('{"detail":"No device_class_id was given and there is no current one"}', 409)); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + const button = buttonNamed('models.workingMemory.calibrate')!; + expect(button.disabled).toBe(true); + expect(document.getElementById(button.getAttribute('aria-describedby')!)?.textContent).toBe( + 'Calibration needs exactly one generation device; this machine has two.' + ); + // A missing device class is a state, not a load failure. + expect(sectionText()).not.toContain('models.workingMemory.loadFailed'); + }); + + it('names an unusable store and flags hardware the method was not verified on', async () => { + serveSummary( + summary({ + measurement_verified: false, + problem: 'schema_version 9 is newer than this version reads', + usable: false, + }) + ); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.notVerified'); + + expect(buttonNamed('models.workingMemory.calibrate')?.disabled).toBe(true); + expect(sectionText()).toContain( + 'models.workingMemory.blockedStoreUnusable(problem=schema_version 9 is newer than this version reads)' + ); + }); + + it('explains stale measurements and offers to recalibrate', async () => { + serveSummary( + summary({ lines: [line('stale', ['driver', 'setting.attention_type'])], runs: [run('run-1', 0, 'completed')] }) + ); + await renderSection(); + await expect + .poll(sectionText) + .toContain('models.workingMemory.helpStale(facts=models.workingMemory.facts.driver, attention_type)'); + + expect(sectionText()).toContain('models.workingMemory.stateStale'); + expect(buttonNamed('models.workingMemory.recalibrate')?.disabled).toBe(false); + }); + + it('reports a failed attempt with its error and offers Retry', async () => { + serveSummary(summary({ runs: [run('run-1', 0, 'completed'), run('run-2', 1, 'failed', 'CUDA out of memory')] })); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.failed'); + + expect(sectionText()).toContain('CUDA out of memory'); + expect(buttonNamed('common.retry')?.disabled).toBe(false); + + // Retry is a new attempt: a new id, and retry keys that cannot collide with the failed attempt's. + await act(() => userEvent.click(buttonNamed('common.retry')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + const requests = api.calibrateWorkingMemory.mock.calls.map( + ([, request]) => request as { attemptId: string; idempotencyKey: string } + ); + expect(new Set(requests.map((request) => request.attemptId)).size).toBe(1); + expect(requests[0]!.attemptId).not.toBe('earlier-attempt'); + expect(requests.every((request) => request.idempotencyKey.includes(requests[0]!.attemptId))).toBe(true); + }); + + it('shows no notice where the method is verified and leaves runs without an attempt id on their own', async () => { + serveSummary( + summary({ + lines: [line('in_use')], + runs: [run('legacy-1', 0, 'failed', 'old failure', null), run('legacy-2', 1, 'completed', null, null)], + }) + ); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.stateInUse'); + + expect(sectionText()).not.toContain('models.workingMemory.notVerified'); + // Only the newest legacy run is the latest attempt, so the older failure is not reported. + expect(sectionText()).not.toContain('models.workingMemory.failed'); + }); + + it('shows a skeleton while loading and a retryable error when a read fails', async () => { + let resolvePlan: (plan: WorkingMemoryPlan) => void = () => undefined; + api.getWorkingMemoryPlan.mockImplementationOnce( + () => + new Promise((resolve) => { + resolvePlan = resolve; + }) + ); + let failStatus: (error: Error) => void = () => undefined; + api.getWorkingMemoryStatus.mockImplementationOnce( + () => + new Promise((_resolve, reject) => { + failStatus = reject; + }) + ); + await renderSection(); + + // Opened for the first time, the panel waits for the plan. + await expect.poll(() => host.querySelector('[aria-busy="true"]')).not.toBeNull(); + await act(() => { + resolvePlan(supportedPlan()); + failStatus(new ApiError('{"detail":"store locked"}', 503)); + }); + await expect.poll(sectionText).toContain('models.workingMemory.loadFailed'); + expect(sectionText()).toContain('store locked'); + + await act(() => userEvent.click(buttonNamed('common.retry')!)); + await expect.poll(sectionText).toContain('models.workingMemory.neverCalibrated'); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.tsx new file mode 100644 index 00000000000..cef121864d5 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.tsx @@ -0,0 +1,482 @@ +/* eslint-disable react-perf/jsx-no-new-function-as-prop */ +import type { + CalibrationActivity, + CalibrationBlocker, + CalibrationCoverage, + CalibrationOutcome, + WorkingMemoryPlan, + WorkingMemoryStatus, + WorkingMemorySummary, +} from '@features/models/core/workingMemory'; + +import { Alert, Collapsible, HStack, Progress, Skeleton, Stack, Text } from '@chakra-ui/react'; +import { + deriveCalibrationProgress, + describeCoverage, + getCalibrationBlocker, + isMeasurementMethodVerified, + planCalibrationSteps, +} from '@features/models/core/workingMemory'; +import { useModelsSelector } from '@features/models/data/modelsStore'; +import { + attachWorkingMemoryRealtime, + resetCalibration, + startCalibration, + stopCalibration, + useLocalAttempt, +} from '@features/models/data/workingMemoryCalibration'; +import { + workingMemoryPlanQueryOptions, + workingMemoryStatusQueryOptions, + workingMemorySummaryQueryOptions, +} from '@features/models/data/workingMemoryQueries'; +import { useCalibrationGraphCompiler } from '@features/models/ui/ModelsUiContext'; +import { useNotify } from '@features/models/ui/useModelsNotify'; +import { useExitPresence } from '@platform/react/useExitRetainedValue'; +import { useMountEffect } from '@platform/react/useMountEffect'; +import { useConnectionStatusSelector } from '@platform/transport/connectionStore'; +import { ApiError, getApiErrorMessage } from '@platform/transport/http'; +import { Button } from '@platform/ui/Button'; +import { ConfirmDialog } from '@platform/ui/ConfirmDialog'; +import { useQuery, useQueryClient } from '@tanstack/react-query'; +import { ChevronRightIcon } from 'lucide-react'; +import { useId, useRef, useState, type RefObject } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { WorkingMemoryCalibrateDialog } from './WorkingMemoryCalibrateDialog'; +import { LineMeasurements, LineStateBadge, SharedModelsList, useCaseLabel } from './WorkingMemoryParts'; + +const INDICATOR_OPEN_STYLES = { transform: 'rotate(90deg)' } as const; + +/** + * Working-memory calibration of one model on this machine's GPU: what was measured and whether estimates use it, + * and the runs that measure it. Every read is the server's; the section keeps no copy of a run's state, so a reload or + * a reconnect shows the same thing. + * + * Collapsed on every visit; its header still names the GPU and the state, or the progress of a running calibration. + */ +export const WorkingMemorySection = ({ modelKey }: { modelKey: string }) => { + const { t } = useTranslation(); + const queryClient = useQueryClient(); + const headingId = useId(); + const triggerRef = useRef(null); + const [isOpen, setIsOpen] = useState(false); + // The plan measures the GPU's free memory on the server, so it loads only once the panel was opened. + const [wasOpened, setWasOpened] = useState(false); + const local = useLocalAttempt(modelKey); + const plan = useQuery({ ...workingMemoryPlanQueryOptions(modelKey), enabled: wasOpened }); + const isConnected = useConnectionStatusSelector((snapshot) => snapshot.status === 'connected'); + const summary = useQuery( + workingMemorySummaryQueryOptions(modelKey, { isConnected, isSubmitting: local?.submitting ?? false }) + ); + const status = useQuery(workingMemoryStatusQueryOptions()); + const activityCaption = useActivityCaption(); + + useMountEffect(() => attachWorkingMemoryRealtime(queryClient)); + + // Without the plan, an attempt's step count comes from this tab's record or the attempt id. + const progress = deriveCalibrationProgress({ + local, + runs: summary.data?.runs ?? [], + steps: plan.data ? planCalibrationSteps(plan.data) : [], + }); + const coverage = describeCoverage(summary.data ?? null); + const deviceName = plan.data?.device_class?.name ?? (status.data ? currentDeviceName(status.data) : null); + const failed = [plan, summary, status].find((query) => query.isError); + let body; + + // The plan and the summary know only installed models: this one was removed while its details were open. + if (failed?.error instanceof ApiError && failed.error.status === 404) { + body = ( + + {t('models.workingMemory.modelRemoved')} + + ); + } else if (failed) { + body = ( + + + {t('models.workingMemory.loadFailed')} + {': '} + {getApiErrorMessage(failed.error, t('common.unknownError'))} + + + + ); + } else if (!wasOpened) { + body = null; + } else if (plan.data === undefined || summary.data === undefined || status.data === undefined) { + body = ( + + + + + ); + } else { + body = ( + + ); + } + + return ( + { + setIsOpen(open); + if (open) { + setWasOpened(true); + } + }} + > + + + + + + + + + {t('models.workingMemory.title')} + + + {/* Collapsed, the header says what the open panel would; open, the panel's own row does. */} + {!isOpen ? ( + + {deviceName ? ( + + {deviceName} + + ) : null} + {progress.activity.kind === 'running' ? ( + + {activityCaption(progress.activity)} + + ) : summary.data?.usable ? ( + + ) : null} + + ) : null} + + + {t('models.workingMemory.description')} + + + {body} + + + ); +}; + +const currentDeviceName = (status: WorkingMemoryStatus): string | null => + status.device_classes.find((entry) => entry.is_current)?.device_class?.name ?? null; + +const WorkingMemoryBody = ({ + coverage, + deviceName, + focusFallbackRef, + modelKey, + plan, + progress: { activity, outcome }, + status, + summary, +}: { + coverage: CalibrationCoverage; + deviceName: string | null; + /** Takes focus back when a confirmation's opener is gone or disabled. */ + focusFallbackRef: RefObject; + modelKey: string; + plan: WorkingMemoryPlan; + progress: ReturnType; + status: WorkingMemoryStatus; + summary: WorkingMemorySummary | null; +}) => { + const { t } = useTranslation(); + const notify = useNotify(); + const queryClient = useQueryClient(); + const compile = useCalibrationGraphCompiler(); + const reasonId = useId(); + const primaryRef = useRef(null); + const stopRef = useRef(null); + const [isCalibrateOpen, setIsCalibrateOpen] = useState(false); + const [isStopOpen, setIsStopOpen] = useState(false); + const [isResetOpen, setIsResetOpen] = useState(false); + const calibrateDialog = useExitPresence(isCalibrateOpen); + const throughName = useModelsSelector((snapshot) => + plan.calibrated_through ? (snapshot.modelsByKey.get(plan.calibrated_through)?.name ?? null) : null + ); + + const isRunning = activity.kind === 'running'; + const blocker: CalibrationBlocker | { kind: 'noCompiler' } | null = + compile === null ? { kind: 'noCompiler' } : getCalibrationBlocker({ isRunning, plan, status, summary }); + const lines = summary?.lines ?? []; + const verified = isMeasurementMethodVerified(summary, status); + const primaryLabel = + outcome.kind === 'failed' || outcome.kind === 'partial' + ? t('common.retry') + : coverage.hasOwnMeasurements + ? t('models.workingMemory.recalibrate') + : t('models.workingMemory.calibrate'); + const blockerText = useBlockerText(blocker); + + // The opener of a confirmation can disappear or be disabled once it is done; the section's toggle stays. + const getFinalFocus = () => + primaryRef.current && !primaryRef.current.disabled ? primaryRef.current : focusFallbackRef.current; + // Starting records the attempt synchronously, so Stop has rendered by the time the closing dialog restores focus. + const getCalibrateFinalFocus = () => stopRef.current ?? getFinalFocus(); + + const confirmCalibration = () => { + setIsCalibrateOpen(false); + + if (compile) { + void startCalibration({ compile, modelKey, plan, queryClient }); + } + }; + + return ( + + + + {deviceName ? ( + + {deviceName} + + ) : null} + {/* Without a current device class or a readable store nothing can be said about this GPU's measurements. */} + {summary?.usable ? : null} + + + {isRunning ? ( + + ) : null} + {coverage.hasOwnMeasurements ? ( + + ) : null} + + + + + {blockerText ? ( + + {blockerText} + + ) : null} + + + + + {verified === false ? ( + + + + {t('models.workingMemory.notVerified')} + + + ) : null} + + {!coverage.hasCurrentLines && activity.kind === 'idle' && plan.supported && summary?.usable ? ( + + {t('models.workingMemory.neverCalibrated')} + + ) : null} + {coverage.usesOthersOnly ? ( + + {t('models.workingMemory.usesOthers')} + + ) : null} + + {lines.length > 0 ? ( + + {lines.map((line) => ( + + ))} + + ) : null} + + {plan.shared_with.length > 0 ? : null} + + + {plan.not_covered.length > 0 ? {t('models.workingMemory.encodeNotCovered')} : null} + {t('models.workingMemory.floorNote')} + + + {calibrateDialog.isMounted ? ( + setIsCalibrateOpen(false)} + onConfirm={confirmCalibration} + onExitComplete={calibrateDialog.release} + /> + ) : null} + setIsStopOpen(false)} + onConfirm={async () => { + try { + await stopCalibration({ modelKey, queryClient }); + } catch (error) { + notify.error(t('models.workingMemory.stopFailed'), getApiErrorMessage(error, '')); + } + }} + /> + setIsResetOpen(false)} + onConfirm={async () => { + try { + await resetCalibration({ modelKey, queryClient }); + notify.success(t('models.workingMemory.resetDone')); + } catch (error) { + notify.error(t('models.workingMemory.resetFailed'), getApiErrorMessage(error, '')); + } + }} + /> + + ); +}; + +const useBlockerText = (blocker: CalibrationBlocker | { kind: 'noCompiler' } | null): string | null => { + const { t } = useTranslation(); + + switch (blocker?.kind) { + case undefined: + return null; + case 'unsupported': + return blocker.reason ?? t('models.workingMemory.blockedUnsupported'); + case 'unavailable': + return blocker.reason ?? t('models.workingMemory.blockedUnavailable'); + case 'storeUnusable': + return t('models.workingMemory.blockedStoreUnusable', { problem: blocker.reason ?? '' }); + case 'running': + return t('models.workingMemory.blockedRunning'); + case 'noCompiler': + return t('models.workingMemory.blockedUnavailable'); + } +}; + +/** "Queuing: 2/18" while steps are posted, then "Calibrating: 5/18 runs". */ +const useActivityCaption = () => { + const { t } = useTranslation(); + + return ({ done, queuing, total }: Extract): string => + queuing ? t('models.workingMemory.queuing', { done, total }) : t('models.workingMemory.running', { done, total }); +}; + +const ActivityPanel = ({ activity }: { activity: CalibrationActivity }) => { + const { t } = useTranslation(); + const caseLabel = useCaseLabel(); + const activityCaption = useActivityCaption(); + + if (activity.kind !== 'running') { + return null; + } + + const { currentCase, done, total } = activity; + const caption = activityCaption(activity); + + return ( + + + {/* The track is the progressbar; without a name of its own it would be announced by its percentage. */} + + + + + + {caption} + {currentCase ? ` · ${caseLabel(currentCase)}` : null} + + + ); +}; + +const OutcomePanel = ({ outcome }: { outcome: CalibrationOutcome }) => { + const { t } = useTranslation(); + + if (outcome.kind === 'none' || outcome.kind === 'completed') { + return null; + } + + if (outcome.kind === 'canceled') { + return ( + + {t('models.workingMemory.canceled')} + + ); + } + + const isFailed = outcome.kind === 'failed'; + const detail = + outcome.error || + (outcome.kind === 'partial' + ? !outcome.canceled && outcome.queued < outcome.total + ? t('models.workingMemory.partialQueued', { queued: outcome.queued, total: outcome.total }) + : t('models.workingMemory.partialCanceled') + : null); + + return ( + + + + {isFailed ? t('models.workingMemory.failed') : t('models.workingMemory.partial')} + {detail ? {detail} : null} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/queue/contracts.ts b/invokeai/frontend/webv2/src/features/queue/contracts.ts index d9374a3a87c..4a9b43cca76 100644 --- a/invokeai/frontend/webv2/src/features/queue/contracts.ts +++ b/invokeai/frontend/webv2/src/features/queue/contracts.ts @@ -57,6 +57,11 @@ export { shouldSubmitPendingQueueItem, } from './core/submissionRules'; export { extractGenerationMeta, getResultImageName, type QueueGenerationMeta } from './core/generationMeta'; +export { + planGenerateSubmissionBatch, + type GeneratePromptBatchPlan, + type GeneratePromptSubmission, +} from './core/promptBatch'; export { buildProjectQueueItemOriginPrefix, buildQueueItemOrigin, diff --git a/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts b/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts index 1a42d04dfbb..36339aac9bd 100644 --- a/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts +++ b/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts @@ -339,3 +339,37 @@ export const buildGeneratePromptBatchPlan = ({ runs: isHeld ? iterations : 1, }; }; + +/** The prompt and seed fields of a generate submission, from compilation to `enqueue_batch`. */ +export interface GeneratePromptSubmission { + batchCount: number; + negativePrompt: string; + negativePromptNodeId: string; + positivePrompt: string; + positivePromptNodeId: string; + /** + * Prompts to submit as a batch dimension, already expanded by the caller. + * Absent or empty submits `positivePrompt` verbatim. + */ + positivePrompts?: string[]; + seed: number; + seedBehaviour?: QueuePromptSeedBehaviour; + seedNodeId: string; + seedStep: QueueSeedStep; + /** Legacy recovery maps the old random toggle to seedStep and preserves its expansion rules. */ + legacySeedPlan?: true; +} + +/** The batch data and run count `enqueue_batch` receives for a generate submission. */ +export const planGenerateSubmissionBatch = (submission: GeneratePromptSubmission): GeneratePromptBatchPlan => + (submission.legacySeedPlan ? buildLegacyGeneratePromptBatchPlan : buildGeneratePromptBatchPlan)({ + batchCount: sanitizeBatchCount(submission.batchCount), + negativePrompt: submission.negativePrompt, + negativePromptNodeId: submission.negativePromptNodeId, + positivePromptNodeId: submission.positivePromptNodeId, + prompts: submission.positivePrompts?.length ? submission.positivePrompts : [submission.positivePrompt], + seed: submission.seed, + seedBehaviour: submission.seedBehaviour ?? 'per-iteration', + seedNodeId: submission.seedNodeId, + seedStep: submission.seedStep, + }); diff --git a/invokeai/frontend/webv2/src/features/queue/core/types.ts b/invokeai/frontend/webv2/src/features/queue/core/types.ts index e2dd92e53b6..f1f07df5ea8 100644 --- a/invokeai/frontend/webv2/src/features/queue/core/types.ts +++ b/invokeai/frontend/webv2/src/features/queue/core/types.ts @@ -1,4 +1,5 @@ import type { + GeneratePromptSubmission, QueuePromptSeedBehaviour, QueueSeedStep, QueueWorkflowBatchDatum, @@ -58,23 +59,7 @@ export interface QueueEnqueueWorkflowRequest extends QueueEnqueueRequestBase { batchData?: QueueWorkflowBatchDatum[][]; } -export interface QueueEnqueueGenerateRequest extends QueueEnqueueRequestBase { - negativePrompt: string; - negativePromptNodeId: string; - positivePrompt: string; - positivePromptNodeId: string; - /** - * Prompts to submit as a batch dimension, already expanded by the caller. - * Absent or empty submits `positivePrompt` verbatim. - */ - positivePrompts?: string[]; - seed: number; - seedBehaviour?: QueuePromptSeedBehaviour; - seedNodeId: string; - seedStep: QueueSeedStep; - /** Legacy recovery maps the old random toggle to seedStep and preserves its expansion rules. */ - legacySeedPlan?: true; -} +export interface QueueEnqueueGenerateRequest extends QueueEnqueueRequestBase, GeneratePromptSubmission {} export interface QueueEnqueueResult { batchId?: string; diff --git a/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts b/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts index ebbe288f77c..ad61dab8aa0 100644 --- a/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts +++ b/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts @@ -8,12 +8,7 @@ import type { QueueResultVideoOptions, } from '@features/queue/core/types'; -import { - buildGeneratePromptBatchPlan, - buildLegacyGeneratePromptBatchPlan, - buildQueueWorkflowBatchPlan, - sanitizeBatchCount, -} from '@features/queue/core/promptBatch'; +import { buildQueueWorkflowBatchPlan, planGenerateSubmissionBatch } from '@features/queue/core/promptBatch'; import { mapWithConcurrency } from '@platform/core/concurrency'; import { addOutputImageNames } from '@platform/core/outputImages'; import { assertAccountScopeCurrent, captureAccountScope } from '@platform/state/accountLifecycle'; @@ -78,20 +73,7 @@ const mapEnqueueResult = (value: unknown, isReceipt = false): QueueEnqueueResult }; export const enqueueGenerate = async (request: QueueEnqueueGenerateRequest): Promise => { - const planInput = { - batchCount: sanitizeBatchCount(request.batchCount), - negativePrompt: request.negativePrompt, - negativePromptNodeId: request.negativePromptNodeId, - positivePromptNodeId: request.positivePromptNodeId, - prompts: request.positivePrompts?.length ? request.positivePrompts : [request.positivePrompt], - seed: request.seed, - seedBehaviour: request.seedBehaviour ?? 'per-iteration', - seedNodeId: request.seedNodeId, - }; - const plan = (request.legacySeedPlan ? buildLegacyGeneratePromptBatchPlan : buildGeneratePromptBatchPlan)({ - ...planInput, - seedStep: request.seedStep, - }); + const plan = planGenerateSubmissionBatch(request); const result = await apiFetchJson('/api/v1/queue/default/enqueue_batch', { body: JSON.stringify({ batch: { diff --git a/invokeai/frontend/webv2/src/workbench/workbenchState.ts b/invokeai/frontend/webv2/src/workbench/workbenchState.ts index 257a73739ad..46c7c0268cf 100644 --- a/invokeai/frontend/webv2/src/workbench/workbenchState.ts +++ b/invokeai/frontend/webv2/src/workbench/workbenchState.ts @@ -85,7 +85,6 @@ import { type GeneratedVideoContract, generatedVideoToGalleryItem, } from '@features/gallery/contracts'; -import { planSeedSubmission } from '@platform/core/seed'; import { describeError } from '@platform/logging/normalize'; import { WIDGET_REGIONS } from '@workbench/layoutContracts'; import { prependProjectEvent, PROJECT_EVENT_LIMIT } from '@workbench/projectEvents'; @@ -141,14 +140,13 @@ import { applyProjectPromptDraft, cloneGenerateWidgetValues, createDynamicPromptsSampleSeed, - getEffectivePrompts, getGenerationModelAvailabilityReasons, getPromptDraftFromValues, getPromptHistoryItemFromGenerateSettings, - hasDynamicPromptSyntax, migrateProjectPromptDraft, normalizeGenerateSettings, normalizeGenerateWidgetValues, + planGenerateSubmission, removePromptHistoryItem, sanitizeBatchCount, syncGenerateWidgetValuesWithModels, @@ -3578,36 +3576,21 @@ const enqueueCompiledSnapshot = ( : route.sourceId === 'video' ? videoSettings : null; - // Resolve template-merged prompts here, where Generate and Canvas submissions converge; Upscale merges to - // identity. - const effectivePrompts = sourceGenerateSettings ? getEffectivePrompts(sourceGenerateSettings) : null; - // Record expansions for Generate routes even when only one prompt results. Inspect merged prompts because - // templates may introduce dynamic syntax. - const expandedPositivePrompts = - route.sourceId !== 'upscale' && - route.sourceId !== 'video' && - compiled.positivePrompts && - compiled.positivePrompts.length > 0 && - effectivePrompts && - hasDynamicPromptSyntax(effectivePrompts.positivePrompt) - ? compiled.positivePrompts - : undefined; - const expandedSeedBehaviour = expandedPositivePrompts - ? (canvasGenerateSettings ?? generateSettings)?.dynamicPromptsSeedBehaviour - : undefined; + // Resolve template-merged prompts and seeds here, where Generate and Canvas submissions converge; Upscale and Video + // merge to identity and never expand. + const submissionPlan = sourceGenerateSettings + ? planGenerateSubmission( + sourceGenerateSettings, + route.sourceId === 'upscale' || route.sourceId === 'video' ? undefined : compiled.positivePrompts + ) + : null; + const effectivePrompts = submissionPlan?.effectivePrompts ?? null; + const expandedPositivePrompts = submissionPlan?.submission.positivePrompts; // The draw the expansion used. Generate expands before dispatch, so the project's sample seed may have rotated // since; recall and rotation follow the expansion, not the project. const expansionSampleSeed = expandedPositivePrompts ? (compiled.positivePromptsSampleSeed ?? null) : null; // Compilation already reserved the starting seed; failures and cancellations do not roll the sequence back. - const seedPlan = sourceGenerateSettings - ? planSeedSubmission({ - batchCount: sourceGenerateSettings.batchCount, - promptCount: expandedPositivePrompts?.length ?? 1, - seedBehaviour: expandedSeedBehaviour ?? 'per-iteration', - seedMode: sourceGenerateSettings.seedMode, - startSeed: sourceGenerateSettings.seed, - }) - : null; + const seedPlan = submissionPlan?.seedPlan ?? null; const backendSubmission: QueueCompiledSubmission = !backendGraph ? { error: `${route.sourceId} queue item is missing a compiled backend graph.`, kind: 'invalid' } : route.sourceId === 'workflow' @@ -3624,21 +3607,14 @@ const enqueueCompiledSnapshot = ( // active workflow changes. ...(compiled.projectWorkflowId ? { projectWorkflowId: compiled.projectWorkflowId } : {}), } - : sourceGenerateSettings && effectivePrompts + : submissionPlan ? { - batchCount: sourceGenerateSettings.batchCount, + ...submissionPlan.submission, graph: backendGraph, kind: 'generate', - // Disabling the negative prompt also suppresses the template's negative side. - negativePrompt: sourceGenerateSettings.negativePromptEnabled ? effectivePrompts.negativePrompt : '', negativePromptNodeId: generate?.negativePromptNodeId ?? 'negative_prompt', - positivePrompt: effectivePrompts.positivePrompt, positivePromptNodeId: generate?.positivePromptNodeId ?? 'positive_prompt', - ...(expandedPositivePrompts ? { positivePrompts: expandedPositivePrompts } : {}), - seed: sourceGenerateSettings.seed, - ...(expandedSeedBehaviour ? { seedBehaviour: expandedSeedBehaviour } : {}), seedNodeId: generate?.seedNodeId ?? 'seed', - seedStep: seedPlan?.step ?? 0, } : { error: `${route.sourceId} queue item is missing source submission metadata.`, kind: 'invalid' }; // A workflow compiles every node's board in (Auto included), so its board-less results stay Uncategorized. diff --git a/scripts/calibrate_flux2_denoise_working_memory.py b/scripts/calibrate_flux2_denoise_working_memory.py new file mode 100644 index 00000000000..12bdb721769 --- /dev/null +++ b/scripts/calibrate_flux2_denoise_working_memory.py @@ -0,0 +1,635 @@ +"""Measure the working memory of one FLUX.2 transformer forward, as shipped-data points for `denoise.flux2`. + +What a point measures +--------------------- +The demand of one transformer forward, measured as a fresh window (`invokeai/backend/working_memory/measurement.py`, +whose definitions the in-server probe's windows share). With the model and its inputs resident: synchronize, release the +allocator's cached blocks, read the baseline and reset the peaks; run the forward; the demand is then the largest growth +of three counters over the baseline: the driver's figure for this process, torch's reserved peak and its allocated peak. +A forward whose allocations failed, or only succeeded after the allocator released its cache and retried (a squeezed +peak), or after which Windows keeps more than `measurement.SPILL_TOLERANCE_BYTES` of the process in shared memory, is +invalid: it is recorded as such, not as a point. + +Every point runs in a fresh process (`scripts/working_memory/fresh_points.py`, which the calibration scripts share), +which first applies the server's runtime (`apply_server_runtime` in `invokeai/app/run_app.py`): the `MIOPEN_FIND_MODE` +default, the integrated-GPU check, the allocator, the AOTriton decision and the monkeypatches, the ROCm SDPA guard among +them, in the server's order. Its settings are those of a server without a config file, except the allocator +configuration: `--alloc-conf`, or without one the platform default the server uses (expandable segments for ROCm on +Windows, torch's native allocator elsewhere). This script's own process runs the integrated-GPU check once before the +first point: the measuring processes inherit the `HIP_VISIBLE_DEVICES` it sets, which their own check keeps, as the +server keeps an exported one. + +The forward then runs twice, cold and warm, and the point takes the larger demand (`measure_cold_and_warm` and +`measured_bytes_and_method` in `measurement.py`): a `demand_peak` where both forwards read the driver's figure, else a +`reserved_peak` of torch's counters alone. + +The transformer is `Flux2Transformer2DModel` at the geometry the model loader builds for a variant's width (heads of +128, the variant's caption width, guidance embeddings) with random bf16 weights, because memory depends on shapes, not +on weight values. It has 2 double-stream and 2 single-stream blocks: a no-grad forward frees each block's +intermediates, which `--extra-blocks 4` re-checks at 1024px per width. The inputs are what the node hands it: the +image's and the reference images' tokens on one stream, a caption, the timestep and the guidance, a sample of each per +batch entry. A point carries the features of `estimate_flux2_denoise_working_memory`, taken in the measuring process +after the runtime setup, so its `score_matrix_bytes` reflects the attention kernel this build really runs. + +Each point is priced before it runs: the estimate for it plus its weights. It is skipped when that exceeds free device +memory, unless `--force`. An oversized allocation does not always fail: where the driver backs it with system RAM +(NVIDIA's sysmem fallback, AMD on Windows) it pages, which can stall the whole machine. `--dry-run` prints that price +per point as a CPU prices it, with the shipped values of cuDNN under the native allocator. + +Modes +----- +- `fit` (default): the fit grid, per `--widths` (3072, 4096 and 6144: Klein 4B, Klein 9B and [dev]). The `--sizes` + (512, 768, 1024, 1440, 2048 px) run with a `--text-tokens` caption (512). Those among `--short-text-sizes` (1024, + 1440) also run with a `--short-text-tokens` caption (77), for the text slope. At 1024px, `--references` (1 and 3) runs + with that many `--reference-px` (1024) reference images, and `--batches` (2) with that many samples. `--extra-blocks + N` adds one forward with N blocks per stream at 1024px per width. +- `large`: long sequences, per `--widths` (default 3072 and 4096; [dev] fits 24 GB only below 33025 tokens): 2048px + with one 2024px reference, 33025 tokens with a 512-token caption; 1536px with one 2024px reference, 25857 tokens; and + 1328px with three 1328px references, which the node resizes to 1024px each, 19689 tokens. The node caps neither the + image's side nor the number of references; longer sequences price on the line extrapolated. +- `math`: SDPA forced onto the math kernel at 1024px, next to the default kernel, per width: what a materialized score + matrix costs per element, against what the estimate charges for it. + +A reference image's tokens are what `Flux2RefImageExtension` concatenates for it: resized as it resizes them (one +reference to at most 2024x2024 pixels, each of several to at most 1024x1024), encoded at 1/8 of the pixels, padded to +whole 2x2 patches. A 1024px reference is 4096 tokens, a 2024px one 16129. + +The fit grid's and the large grid's points are fit points (`"fit": true`): the plain forwards, reference images and +batches included, which the estimator's line admits (`denoise_working_memory._line`). The checks' points, the +extra-blocks forward included, test analytic terms or assumptions rather than the line: they are marked `"fit": false`, +and their `in_fit` is false. + +Output +------ +Every measured point is appended to `--out` as one JSON object per line, the point file +`scripts/working_memory/build_shipped.py --points` reads. It carries the fields of a `MeasuredPoint` +(`invokeai/backend/working_memory/coefficients.py`): + +- `estimator`, `version` and `features`, the estimate's own; +- `platform`, as `platform_for` gives it, OS included; the builder names the OS only where the data needs it; +- `measured_bytes` (the demand), `method`, `device` and `in_fit`; +- `provenance`, which names the GPU, the torch build, the OS, the allocator configuration, on ROCm `MIOPEN_FIND_MODE`, + the AOTriton switch and the SDPA guard, the block count and the date. + +The other keys are the script's, which the builder ignores: `fit`, `point` (the grid entry) and `raw` (each forward's +counter deltas, the prediction and the runtime facts). An invalid, skipped or failed forward is no point: it shows in +the table on stdout, with its reasons, and stays out of the file. + +Usage +----- +RTX 4090 (CUDA): the native allocator, then `cudaMallocAsync`; the math check once. + + python scripts/calibrate_flux2_denoise_working_memory.py --out 4090-native.jsonl --extra-blocks 4 + python scripts/calibrate_flux2_denoise_working_memory.py --out 4090-async.jsonl --alloc-conf backend:cudaMallocAsync + python scripts/calibrate_flux2_denoise_working_memory.py --out 4090-checks.jsonl --mode math + python scripts/calibrate_flux2_denoise_working_memory.py --out 4090-native-large.jsonl --mode large + python scripts/calibrate_flux2_denoise_working_memory.py --out 4090-async-large.jsonl --mode large \ + --alloc-conf backend:cudaMallocAsync + +RX 9060 XT (ROCm on Windows): the server's default expandable segments, then the native allocator; the math check once. + + python scripts/calibrate_flux2_denoise_working_memory.py --out 9060xt-expandable.jsonl --extra-blocks 4 + python scripts/calibrate_flux2_denoise_working_memory.py --out 9060xt-native.jsonl --alloc-conf expandable_segments:False + python scripts/calibrate_flux2_denoise_working_memory.py --out 9060xt-checks.jsonl --mode math + +`--dry-run` prints the planned points without touching a device, each priced as a CPU prices it, e.g. + + python scripts/calibrate_flux2_denoise_working_memory.py --dry-run --mode large +""" + +import argparse +import dataclasses +import datetime +import sys +from collections.abc import Mapping, Sequence +from contextlib import nullcontext +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any, Literal + +# None of these imports torch: a measuring process has to configure the allocator before torch is imported. +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from invokeai.app.invocations.constants import LATENT_SCALE_FACTOR # noqa: E402 +from scripts.working_memory import fresh_points # noqa: E402 + +if TYPE_CHECKING: + import torch + + from invokeai.backend.working_memory.estimate import WorkingMemoryEstimate + +GIB = 2**30 +MIB = 2**20 + +PATCH_SIZE = 2 +"""FLUX.2 packs 2x2 latent patches into one token.""" +LATENT_CHANNELS = 128 +"""A packed token's channels: 32 latent channels times the 2x2 patch.""" +HEAD_DIM = 128 +CONTEXT_DIMS = {3072: 7680, 4096: 12288, 6144: 15360} +"""The caption width per transformer width: 3x the text encoder's hidden size (Qwen3-4B, Qwen3-8B, Mistral Small 3.1), +as `model_manager/configs/flux2_variant.py` declares them.""" +CHECK_PX = 1024 +"""The size the reference, batch and extra-blocks points and the checks run at.""" +REFERENCE_PX = 1024 +"""The fit grid's reference images' side, `--reference-px`'s default.""" +LARGE_GRID = ((2048, 1, 2024), (1536, 1, 2024), (1328, 3, 1328)) +"""The large grid's forwards: image side, reference count and reference side.""" +LARGE_WIDTHS = (3072, 4096) +DTYPE = "bfloat16" +"""The node runs the transformer in bf16 only.""" +TIMESTEP = 0.5 +GUIDANCE = 4.0 + +LOGGER_NAME = "calibrate_flux2_denoise_working_memory" + +Kind = Literal["fit", "blocks", "baseline", "math"] +Mode = Literal["fit", "large", "math"] + + +@dataclass(frozen=True) +class Point: + """One transformer forward to measure, in a process of its own.""" + + kind: Kind + """`fit` for a point of the fit grid; otherwise the check it serves: another block count (`blocks`), the + default-kernel forward a check compares with (`baseline`) or the forced math kernel (`math`).""" + hidden_size: int + px: int + text_tokens: int + references: int = 0 + """How many reference images of `reference_px` square the forward attends to.""" + reference_px: int = REFERENCE_PX + batch: int = 1 + blocks: int = 2 + """Double-stream and single-stream blocks, each.""" + force_math: bool = False + + @property + def fit(self) -> bool: + return self.kind == "fit" + + +def image_tokens(px: int) -> int: + """A square image's tokens: its latent at 1/8 of the pixels, in 2x2 patches.""" + side = px // LATENT_SCALE_FACTOR // PATCH_SIZE + return side * side + + +def reference_tokens(point: Point) -> int: + """What `Flux2RefImageExtension` concatenates onto the image stream for the point's references: each resized as it + resizes them, its latent at 1/8 of the pixels padded to whole 2x2 patches.""" + if not point.references: + return 0 + from PIL import Image + + from invokeai.backend.flux2.ref_image_extension import ( + MAX_PIXELS_MULTI_REF, + MAX_PIXELS_SINGLE_REF, + resize_image_to_max_pixels, + ) + + cap = MAX_PIXELS_SINGLE_REF if point.references == 1 else MAX_PIXELS_MULTI_REF + width, height = resize_image_to_max_pixels(Image.new("L", (point.reference_px, point.reference_px)), cap).size + rows, columns = (-(-(side // LATENT_SCALE_FACTOR) // PATCH_SIZE) for side in (height, width)) + return point.references * rows * columns + + +def plan( + mode: Mode, + *, + widths: Sequence[int], + sizes: Sequence[int], + text_tokens: int, + short_text_tokens: int, + short_text_sizes: Sequence[int], + references: Sequence[int], + batches: Sequence[int], + blocks: int, + extra_blocks: int, + reference_px: int = REFERENCE_PX, +) -> list[Point]: + """The forwards a run measures, in order: per width, the fit grid, the large grid or the math check's pair of + forwards.""" + points: list[Point] = [] + for width in widths: + if mode == "large": + points += [ + Point("fit", width, px, text_tokens, references=n, reference_px=ref_px, blocks=blocks) + for px, n, ref_px in LARGE_GRID + ] + continue + if mode == "math": + points.append(Point("baseline", width, CHECK_PX, text_tokens, blocks=blocks)) + points.append(Point("math", width, CHECK_PX, text_tokens, blocks=blocks, force_math=True)) + continue + points += [Point("fit", width, px, text_tokens, blocks=blocks) for px in sizes] + points += [Point("fit", width, px, short_text_tokens, blocks=blocks) for px in sizes if px in short_text_sizes] + points += [ + Point("fit", width, CHECK_PX, text_tokens, references=n, reference_px=reference_px, blocks=blocks) + for n in references + if n + ] + points += [Point("fit", width, CHECK_PX, text_tokens, batch=n, blocks=blocks) for n in batches if n != 1] + if extra_blocks: + points.append(Point("blocks", width, CHECK_PX, text_tokens, blocks=extra_blocks)) + return points + + +def provenance(point: Point, facts: Mapping[str, Any], date: str) -> str: + """A measured point's `provenance`: where and how it was measured.""" + regime = ", SDPA forced onto the math kernel" if point.force_math else "" + blocks = f"{point.blocks} double-stream and {point.blocks} single-stream block{'' if point.blocks == 1 else 's'}" + return ( + f"scripts/calibrate_flux2_denoise_working_memory.py ({date}): {fresh_points.runtime_provenance(facts)}. " + "Random-init FLUX.2 " + f"transformer of width {point.hidden_size} with the model loader's geometry, {blocks}{regime}; memory does not " + "depend on weight values. A fresh process with the server's runtime, the larger demand of a cold and a warm " + "forward; no failed allocation, allocator retry or spill." + ) + + +# Everything below imports torch. In a measuring process it runs only after `fresh_points.apply_server_runtime`. + + +def _estimate(point: Point, device: "torch.device") -> "WorkingMemoryEstimate": + """The estimate for the point's forward on `device`, priced in the kernel regime the caller holds.""" + import torch + + from invokeai.backend.flux2.denoise_working_memory import estimate_flux2_denoise_working_memory + + return estimate_flux2_denoise_working_memory( + # The point runs a randomly initialized dense model, which dequantizes and casts nothing per forward. + torch.nn.Module(), + hidden_size=point.hidden_size, + image_seq_len=image_tokens(point.px), + ref_image_seq_len=reference_tokens(point), + text_seq_len=point.text_tokens, + batch_size=point.batch, + lora_count=0, + device=device, + dtype=getattr(torch, DTYPE), + ) + + +def _transformer(hidden_size: int, blocks: int) -> "torch.nn.Module": + """The FLUX.2 transformer of a width, as the model loader builds it, with `blocks` blocks per stream, on the meta + device.""" + import torch + from diffusers import Flux2Transformer2DModel + + with torch.device("meta"): + return Flux2Transformer2DModel( + in_channels=LATENT_CHANNELS, + out_channels=LATENT_CHANNELS, + num_layers=blocks, + num_single_layers=blocks, + attention_head_dim=HEAD_DIM, + num_attention_heads=hidden_size // HEAD_DIM, + joint_attention_dim=CONTEXT_DIMS[hidden_size], + patch_size=1, + ) + + +def _weight_bytes(transformer: "torch.nn.Module") -> int: + """A meta-device transformer's weights in bf16.""" + return sum(param.numel() for param in transformer.parameters()) * 2 + + +def _inputs(point: Point, device: "torch.device") -> dict[str, Any]: + """What the node hands the transformer per forward: the image's and the references' tokens on one stream, their + position ids, a caption, the timestep and the guidance, `point.batch` samples of each.""" + import torch + + from invokeai.backend.flux2.sampling_utils import generate_img_ids_flux2 + + dtype = getattr(torch, DTYPE) + latent = point.px // LATENT_SCALE_FACTOR + references = reference_tokens(point) + img_ids = generate_img_ids_flux2(h=latent, w=latent, batch_size=point.batch, device=device) + if references: + ref_ids = torch.zeros(point.batch, references, 4, device=device, dtype=torch.long) + img_ids = torch.cat([img_ids, ref_ids], dim=1) + txt_ids = torch.zeros(point.batch, point.text_tokens, 4, device=device, dtype=torch.long) + txt_ids[..., 3] = torch.arange(point.text_tokens, device=device, dtype=torch.long) + tokens = image_tokens(point.px) + references + return { + "hidden_states": torch.randn(point.batch, tokens, LATENT_CHANNELS, device=device, dtype=dtype), + "encoder_hidden_states": torch.randn( + point.batch, point.text_tokens, CONTEXT_DIMS[point.hidden_size], device=device, dtype=dtype + ), + "timestep": torch.full((point.batch,), TIMESTEP, device=device, dtype=dtype), + "img_ids": img_ids, + "txt_ids": txt_ids, + "guidance": torch.full((point.batch,), GUIDANCE, device=device, dtype=dtype), + "return_dict": False, + } + + +def _measure(point: Point, force: bool) -> dict[str, Any]: + """The output line of `point`, measured in this process after `fresh_points.apply_server_runtime`.""" + import torch + from torch.nn.attention import SDPBackend, sdpa_kernel + + from invokeai.backend.flux2 import denoise_working_memory + from invokeai.backend.util.devices import TorchDevice + from invokeai.backend.working_memory.measurement import measure_cold_and_warm + from invokeai.backend.working_memory.platform import platform_for + + # The device the node itself takes. + device = TorchDevice.choose_torch_device() + if device.type != "cuda": + raise SystemExit(f"Needs a CUDA or ROCm device; this process would run on {device}.") + + def kernel(): + return sdpa_kernel(SDPBackend.MATH) if point.force_math else nullcontext() + + with torch.inference_mode(): + # In the forward's kernel regime, so the score matrix is priced for the kernel that runs. + with kernel(): + estimate = _estimate(point, device) + platform = platform_for(device) + facts = fresh_points.runtime_facts(device, platform.os, denoise_working_memory.__file__) + common = { + "estimator": estimate.estimator, + "version": estimate.version, + "platform": dataclasses.asdict(platform), + "features": dict(estimate.features), + "device": facts["device"], + } + + transformer = _transformer(point.hidden_size, point.blocks) + weights = _weight_bytes(transformer) + free, _ = TorchDevice.cuda_mem_get_info(device) + raw = { + "reservation_bytes": estimate.reservation_bytes, + "weight_bytes": weights, + "predicted_bytes": estimate.reservation_bytes + weights, + "free_bytes": free, + "runtime": facts, + } + if raw["predicted_bytes"] > free and not force: + return fresh_points.skipped_line(point, common, raw) + + # Allocated directly in bf16 on the device and randomly initialized: [dev]'s blocks would otherwise pass + # through the host in float32 first. + transformer = transformer.to(getattr(torch, DTYPE)).to_empty(device=device) + for param in transformer.parameters(): + torch.nn.init.normal_(param, std=0.02) + transformer.eval() + inputs = _inputs(point, device) + + def forward() -> object: + with kernel(): + return transformer(**inputs) + + windows = measure_cold_and_warm(device, forward) + date = datetime.date.today().isoformat() + return fresh_points.point_line( + point, common=common, provenance=provenance(point, facts, date), windows=windows, raw=raw + ) + + +_COLUMNS = ( + f"{'kind':>8} {'width':>5} {'blocks':>6} {'px':>5} {'text':>5} {'refs':>4} {'ref px':>6} {'batch':>5} {'tokens':>7}" +) + + +def _grid(point: Mapping[str, Any]) -> str: + return ( + f"{point['kind']:>8} {point['hidden_size']:>5} {point['blocks']:>6} {point['px']:>5} {point['text_tokens']:>5} " + f"{point['references']:>4} {point['reference_px'] if point['references'] else '':>6} {point['batch']:>5}" + ) + + +def _attended(line: Mapping[str, Any]) -> int: + """The tokens of every sample of a measured line: its size on the line.""" + return line["features"]["tokens"] * line["features"]["batch_size"] + + +def _head(line: Mapping[str, Any]) -> str: + """A line's grid columns in the table.""" + return f"{_grid(line['point'])} {_attended(line) if 'features' in line else '':>7}" + + +def _px(line: Mapping[str, Any]) -> int: + return line["point"]["px"] + + +def _summarize_fit(lines: Sequence[Mapping[str, Any]], text_tokens: int) -> None: + measured = [line for line in lines if line["status"] == "point"] + fit = [line for line in measured if line["fit"]] + for width in dict.fromkeys(line["point"]["hidden_size"] for line in fit): + of_width = [line for line in fit if line["point"]["hidden_size"] == width] + plain = [line for line in of_width if line["point"]["references"] == 0 and line["point"]["batch"] == 1] + main = sorted((line for line in plain if line["point"]["text_tokens"] == text_tokens), key=_attended) + for line in main: + print(f"width {width}, {_px(line)}px: {line['measured_bytes'] / _attended(line) / MIB:.4f} MiB per token") + if len(main) < 2: + continue + lo, hi = main[0], main[-1] + slope = (hi["measured_bytes"] - lo["measured_bytes"]) / (_attended(hi) - _attended(lo)) + base = lo["measured_bytes"] - slope * _attended(lo) + print(f"width {width}, {_px(lo)}-{_px(hi)}px: {slope / MIB:.4f} MiB per token, intercept {base / GIB:.3f} GiB") + # Reference images and batches sit on the same line, which `build_shipped.py` fits on all of them. + for line in of_width: + if line in main: + continue + on_line = base + slope * _attended(line) + point = line["point"] + print( + f"width {width}, {_px(line)}px, {point['text_tokens']} text, {point['references']} references, batch " + f"{point['batch']}: {line['measured_bytes'] / GIB:.3f} GiB at {_attended(line)} tokens, " + f"{line['measured_bytes'] / on_line:.3f}x the {_px(lo)}-{_px(hi)}px line" + ) + for line in (line for line in measured if line["point"]["kind"] == "blocks"): + same = ("hidden_size", "px", "text_tokens", "references", "batch") + base_line = next((ref for ref in fit if all(ref["point"][key] == line["point"][key] for key in same)), None) + if base_line is not None: + extra = line["measured_bytes"] - base_line["measured_bytes"] + print( + f"{line['point']['blocks']} against {base_line['point']['blocks']} blocks per stream, width " + f"{line['point']['hidden_size']}, {_px(line)}px: {extra / MIB:+.0f} MiB " + f"({line['measured_bytes'] / base_line['measured_bytes']:.3f}x)" + ) + + +def _summarize_large(lines: Sequence[Mapping[str, Any]]) -> None: + for line in (line for line in lines if line["status"] == "point"): + point, reserved = line["point"], line["raw"]["reservation_bytes"] + print( + f"width {point['hidden_size']}, {_px(line)}px + {point['references']}x{point['reference_px']}px: " + f"{line['measured_bytes'] / GIB:.3f} GiB at {_attended(line)} tokens, " + f"{line['measured_bytes'] / _attended(line) / MIB:.4f} MiB per token; the shipped estimate reserves " + f"{reserved / GIB:.3f} GiB ({reserved / line['measured_bytes']:.3f}x)" + ) + + +def _summarize_checks(lines: Sequence[Mapping[str, Any]]) -> None: + from invokeai.backend.util.attention import SDPA_MATH_BYTES_PER_SCORE_ELEMENT + + measured = [line for line in lines if line["status"] == "point"] + for line in (line for line in measured if line["point"]["kind"] == "math"): + point = line["point"] + baseline = next( + ( + ref + for ref in measured + if ref["point"]["kind"] == "baseline" and ref["point"]["hidden_size"] == point["hidden_size"] + ), + None, + ) + if baseline is None: + print(f"math kernel, width {point['hidden_size']}: no default-kernel forward to compare with") + continue + extra = line["measured_bytes"] - baseline["measured_bytes"] + charged = line["raw"]["reservation_bytes"] - baseline["raw"]["reservation_bytes"] + tokens = line["features"]["tokens"] + elements = point["hidden_size"] // HEAD_DIM * line["features"]["batch_size"] * tokens * tokens + print( + f"math kernel, width {point['hidden_size']}, {point['px']}px: {extra / GIB:.3f} GiB over the default " + f"kernel, {extra / elements:.2f} bytes per score element; the estimate adds {charged / GIB:.3f} GiB " + f"({SDPA_MATH_BYTES_PER_SCORE_ELEMENT} bytes per element, one chunk where the ROCm guard chunks)" + ) + + +def _print_plan(points: Sequence[Point]) -> None: + """The planned points, each priced as a CPU prices it: the estimate with the shipped values of cuDNN under the + native allocator, the forced math kernel's score matrix included, plus the weights.""" + import torch + from torch.nn.attention import SDPBackend, sdpa_kernel + + print(f"{_COLUMNS} {'estimate GiB':>12} {'weights GiB':>11} {'total GiB':>9}") + cpu = torch.device("cpu") + weights_of: dict[tuple[int, int], int] = {} + for point in points: + with sdpa_kernel(SDPBackend.MATH) if point.force_math else nullcontext(): + estimate = _estimate(point, cpu) + geometry = (point.hidden_size, point.blocks) + if geometry not in weights_of: + weights_of[geometry] = _weight_bytes(_transformer(*geometry)) + weights = weights_of[geometry] + tokens = estimate.features["tokens"] * point.batch + print( + f"{_grid(dataclasses.asdict(point))} {tokens:>7} {estimate.reservation_bytes / GIB:>12.2f} " + f"{weights / GIB:>11.2f} {(estimate.reservation_bytes + weights) / GIB:>9.2f}" + f"{' forced math' if point.force_math else ''}" + ) + print(f"{len(points)} forwards, one process each") + + +def _ints(text: str) -> list[int]: + try: + return [int(part) for part in text.split(",") if part.strip()] + except ValueError: + raise argparse.ArgumentTypeError(f"not a comma-separated list of integers: {text!r}") + + +def _sizes(text: str) -> list[int]: + sizes = _ints(text) + multiple = LATENT_SCALE_FACTOR * PATCH_SIZE + if any(px <= 0 or px % multiple for px in sizes): + raise argparse.ArgumentTypeError(f"sizes must be positive multiples of {multiple} px, got {text!r}") + return sizes + + +def _widths(text: str) -> list[int]: + widths = _ints(text) + if not widths or not set(widths) <= set(CONTEXT_DIMS): + raise argparse.ArgumentTypeError(f"widths must be among {', '.join(map(str, CONTEXT_DIMS))}, got {text!r}") + return widths + + +def _side(text: str) -> int: + try: + side = int(text) + except ValueError: + raise argparse.ArgumentTypeError(f"not a side length in px: {text!r}") + if side < 2 * LATENT_SCALE_FACTOR: + raise argparse.ArgumentTypeError(f"a reference must be at least {2 * LATENT_SCALE_FACTOR} px, got {side}") + return side + + +def _counts(text: str) -> list[int]: + counts = _ints(text) + if any(count < 0 for count in counts): + raise argparse.ArgumentTypeError(f"counts must not be negative, got {text!r}") + return counts + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--mode", choices=("fit", "large", "math"), default="fit", help="Default: fit.") + parser.add_argument("--out", type=Path, help="The JSON-lines file the lines are appended to. Required to measure.") + parser.add_argument( + "--widths", type=_widths, help="Transformer widths. Default: all, and 3072 and 4096 in large mode." + ) + parser.add_argument("--sizes", type=_sizes, default="512,768,1024,1440,2048", help="The fit grid's sizes, px.") + parser.add_argument("--text-tokens", type=int, default=512, help="The caption at every size. Default 512.") + parser.add_argument("--short-text-tokens", type=int, default=77, help="The text-slope caption. Default 77.") + parser.add_argument( + "--short-text-sizes", type=_sizes, default="1024,1440", help="Fit sizes that also run the text-slope caption." + ) + parser.add_argument( + "--references", type=_counts, default="1,3", help="Reference images at 1024px, each run. Default 1,3." + ) + parser.add_argument( + "--reference-px", type=_side, default=REFERENCE_PX, help="The fit grid's reference side, px. Default 1024." + ) + parser.add_argument("--batches", type=_counts, default="2", help="Batch sizes at 1024px beyond 1. Default 2.") + parser.add_argument("--blocks", type=int, default=2, help="Blocks per stream. Default 2.") + parser.add_argument("--extra-blocks", type=int, default=0, help="Fit mode: this block count at 1024px as well.") + parser.add_argument( + "--alloc-conf", + default="", + help="The allocator configuration, as the server's pytorch_cuda_alloc_conf. Empty: the platform's default.", + ) + parser.add_argument("--force", action="store_true", help="Run points predicted not to fit free device memory.") + parser.add_argument("--dry-run", action="store_true", help="Print the planned points without touching a device.") + parser.add_argument("--single", help=argparse.SUPPRESS) + args = parser.parse_args() + + if args.single is not None: + fresh_points.measure_in_this_process(args.single, point_type=Point, measure=_measure, logger_name=LOGGER_NAME) + widths = args.widths or list(LARGE_WIDTHS if args.mode == "large" else CONTEXT_DIMS) + points = plan( + args.mode, + widths=widths, + sizes=args.sizes, + text_tokens=args.text_tokens, + short_text_tokens=args.short_text_tokens, + short_text_sizes=args.short_text_sizes, + references=args.references, + batches=args.batches, + blocks=args.blocks, + extra_blocks=args.extra_blocks, + reference_px=args.reference_px, + ) + if args.dry_run: + _print_plan(points) + return + if args.out is None: + parser.error("--out is required to measure") + fresh_points.check_alloc_conf(parser, args.alloc_conf) + lines = fresh_points.measure_points( + points, + script=Path(__file__).resolve(), + out=args.out, + alloc_conf=args.alloc_conf, + force=args.force, + logger_name=LOGGER_NAME, + columns=_COLUMNS, + head=_head, + ) + + print() + if args.mode == "fit": + _summarize_fit(lines, args.text_tokens) + elif args.mode == "large": + _summarize_large(lines) + else: + _summarize_checks(lines) + fresh_points.print_counts(lines, args.out) + + +if __name__ == "__main__": + main() diff --git a/scripts/calibrate_flux2_working_memory.py b/scripts/calibrate_flux2_working_memory.py index c429c5b9a49..32368e5c134 100644 --- a/scripts/calibrate_flux2_working_memory.py +++ b/scripts/calibrate_flux2_working_memory.py @@ -1,52 +1,28 @@ -"""Calibrate the FLUX.2 working-memory estimates against measured peak CUDA/HIP memory. - -Background ----------- -Four constants decide how much VRAM the model cache keeps free for a FLUX.2 operation, and every one -of them was fitted on CUDA: - -1. ``SDPA_MATH_BYTES_PER_SCORE_ELEMENT`` (``backend/util/attention.py``) -- bytes per element of the - score matrix, charged only where SDPA has no fused kernel and materializes it. -2. ``estimate_vae_working_memory_flux2``'s ``k_decode`` / ``k_encode`` bytes per pixel per element byte, - shipped per convolution backend in ``backend/working_memory/data/vae.json``. -3. ``Flux2DenoiseInvocation._estimate_working_memory``'s 0.4 MB per token, defined at the Klein 9B - width (4096) and scaled linearly by the loaded variant's width. -4. The dispatch question underneath all of it: does this build's SDPA fuse these shapes, or does it - build the score matrix? - -The estimate is consumed by the model cache via ``free >= estimate`` to decide what to evict, so it -MUST be an upper bound: this measures peak *reserved* (not merely allocated) memory, the conservative -quantity that includes caching-allocator overhead and kernel scratch. - -Why this script exists ----------------------- -ROCm answers (1), (2) and (4) differently from CUDA -- its fused kernels cap the head dim at 128, so -the FLUX.2 VAE's 512-wide mid-block head falls back to ``math`` and the score matrix becomes a real, -dominant term. The constants shipped today are known to be short on ROCm for VAE decode in the middle -of the resolution range. Recalibrating needs the hardware, so this puts the whole measurement in one -runnable file: run it on an AMD card and paste the output. - -Portability ------------ -Backend-agnostic: only ``torch.cuda.*``, which is the same API on NVIDIA/CUDA and AMD/ROCm (HIP) -builds. Run the SAME script on each backend and compare. The curve *shape* is architectural and -should match; the absolute constants can differ (cuDNN vs MIOpen workspaces, fused-kernel -availability, allocator rounding). Ship the max across backends, plus headroom. - -No checkpoints required. Every measurement uses a randomly initialized model at the real geometry -- -memory depends on shapes, not on weight values, and a stock-config ``AutoencoderKLFlux2`` reproduces -the real-weight 1024px decode measurement to within 2%. Pass ``--vae`` to measure a specific VAE (the -small-decoder variant has different ``block_out_channels``). - -Each point is measured in a FRESH SUBPROCESS so the caching allocator's fragmentation history cannot -contaminate the reserved-delta reading. A point that OOMs is recorded as ``oom`` rather than aborting -the run, so the grid can probe up to the card's ceiling safely. +"""Measure what a materialized SDPA score matrix costs, and whether this build materializes one. + +What it measures +---------------- +`sdpa_score_matrix_bytes` (``backend/util/attention.py``) charges a working-memory estimate for the score matrix only +where SDPA has no fused kernel for the shapes and builds it, at ``SDPA_MATH_BYTES_PER_SCORE_ELEMENT`` bytes per +element. Two sections check both halves on the build it runs on: + +1. ``dispatch``: which kernel this build's SDPA picks for a 128-wide head (the transformers') and a 512-wide one (the + FLUX.2 VAE mid-block), with and without an additive mask -- the answer the estimators' probe gets. +2. ``sdpa``: peak reserved bytes per score element with the math kernel forced, over the shapes the estimators + produce. The constant must be at least the largest; the measured table is in ``backend/util/attention.py``. + +The estimators' coefficients are measured elsewhere: the denoise lines by fresh-process calibration scripts such as +``scripts/calibrate_flux2_denoise_working_memory.py``, the VAE lines by calibration runs in a running server whose +exports ``scripts/working_memory/build_shipped.py`` turns into shipped points. + +Each point is measured in a fresh subprocess, so the caching allocator's history cannot contaminate the reserved-delta +reading; a point that runs out of memory is recorded as ``oom`` rather than aborting the run. Only ``torch.cuda.*`` is +used, the same API on CUDA and ROCm (HIP) builds: run it on each build and compare. Usage ----- python scripts/calibrate_flux2_working_memory.py - python scripts/calibrate_flux2_working_memory.py --only vae --csv flux2_rocm.csv - python scripts/calibrate_flux2_working_memory.py --only denoise --max-px 1024 + python scripts/calibrate_flux2_working_memory.py --only sdpa --csv sdpa_rocm.csv """ import argparse @@ -59,42 +35,10 @@ import torch.nn.functional as F from torch.nn.attention import SDPBackend, sdpa_kernel -from invokeai.app.invocations.flux2.flux2_denoise import ( - FLUX2_ATTENTION_HEAD_DIM, - FLUX2_REFERENCE_HIDDEN_SIZE, - Flux2DenoiseInvocation, -) -from invokeai.backend.model_manager.configs.flux2_variant import flux2_hidden_size -from invokeai.backend.model_manager.taxonomy import Flux2VariantType -from invokeai.backend.util.attention import SDPA_MATH_BYTES_PER_SCORE_ELEMENT, sdpa_score_matrix_bytes -from invokeai.backend.util.vae_working_memory import VAE_FLUX2_SPEC, estimate_vae_working_memory_flux2 -from invokeai.backend.working_memory.coefficients import shipped_coefficients -from invokeai.backend.working_memory.platform import platform_for +from invokeai.backend.util.attention import SDPA_MATH_BYTES_PER_SCORE_ELEMENT GIB = 1024**3 -MIB = 1024**2 - - -def _shipped_vae_coefficient(operation: str) -> float: - """The VAE coefficient this build's estimate uses for `operation`: the column its convolution backend selects.""" - return shipped_coefficients(VAE_FLUX2_SPEC, platform_for(torch.device("cuda"))).values[f"k_{operation}"] - - -# The FLUX.2 VAE attends on the 8x-downsampled grid with a single 512-wide head. Mirrors -# `_FLUX2_VAE_*` in vae_working_memory.py. -VAE_SPATIAL_COMPRESSION = 8 -VAE_MID_BLOCK_HEAD_DIM = 512 -LATENT_SCALE_FACTOR = 8 - -# (variant, transformer hidden size, context_in_dim). The source of truth is -# `model_manager/configs/flux2_variant.py`; `_check_variant_table` asserts we have not drifted. -VARIANTS = [ - (Flux2VariantType.Klein4B, 3072, 7680), - (Flux2VariantType.Klein9B, 4096, 12288), - (Flux2VariantType.Dev, 6144, 15360), -] -DEFAULT_VAE_PX = [512, 768, 1024, 1280, 1536] # (heads, seq, head_dim) for the score-matrix constant. Spans the shapes the two estimators actually # produce: the VAE's single 512-wide head, and the transformer's many 128-wide ones. @@ -106,12 +50,6 @@ def _shipped_vae_coefficient(operation: str) -> float: (48, 4608, 128), ] -# Sequence lengths the denoise slope is taken between. 4608 is a plain 1024x1024 generation -# (4096 image + 512 text); 9216 is that doubled, which is also what a batch of 2 costs. -DENOISE_SEQ_SHORT = 4608 -DENOISE_SEQ_LONG = 9216 -DENOISE_TEXT_TOKENS = 512 - DTYPES = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32} # Environment that materially changes a peak-*reserved* reading or which kernel runs. Reported in the @@ -135,14 +73,6 @@ def _shipped_vae_coefficient(operation: str) -> float: } -def _check_variant_table() -> None: - """Fail loudly if the widths hard-coded above ever drift from the model manager's table.""" - for variant, hidden, _ in VARIANTS: - actual = flux2_hidden_size(variant) - if actual != hidden: - raise SystemExit(f"Variant table is stale: {variant.value} is {actual}, not {hidden}.") - - def _peak_reserved(fn) -> int | None: """Run ``fn`` and return the growth in peak reserved bytes, or ``None`` if it OOMed. @@ -215,160 +145,11 @@ def run() -> None: return row -# -------------------------------------------------------------------------------------------- -# 3. The VAE's linear constants -# -------------------------------------------------------------------------------------------- - - -def _load_vae(vae_path: str | None, dtype: torch.dtype): - from diffusers import AutoencoderKLFlux2 - - if vae_path: - return AutoencoderKLFlux2.from_pretrained(vae_path, local_files_only=True, torch_dtype=dtype) - # Weight values do not affect activation memory, only shapes do; the stock config is the real - # FLUX.2 VAE geometry (32 latent channels, block_out_channels ending at 512). - return AutoencoderKLFlux2().to(dtype=dtype) - - -@torch.inference_mode() -def measure_vae(operation: str, px: int, dtype: torch.dtype, force_math: bool, vae_path: str | None) -> dict: - """Peak reserved memory for one untiled decode/encode, against what the estimator predicts. - - ``force_math`` runs the mid-block attention through SDPA's ``math`` fallback even where a fused - kernel exists, so a CUDA box can approximate the regime ROCm is in permanently. It is not a - substitute for measuring on ROCm -- the two are not equivalent, which is itself worth showing. - """ - device = torch.device("cuda") - vae = _load_vae(vae_path, dtype).to(device).eval() - vae.disable_tiling() # the decode/encode invocations do not tile; match them. - - param = next(vae.parameters()) - element_size = param.element_size() - if operation == "decode": - latent_channels = int(vae.config.latent_channels) - x = torch.randn(1, latent_channels, px // LATENT_SCALE_FACTOR, px // LATENT_SCALE_FACTOR, **_td(device, dtype)) - else: - x = torch.randn(1, 3, px, px, **_td(device, dtype)) - - def run() -> None: - ctx = sdpa_kernel([SDPBackend.MATH]) if force_math else _null_context() - with ctx: - if operation == "decode": - vae.decode(x, return_dict=False) - else: - vae.encode(x, return_dict=False)[0].mode() - - peak = _peak_reserved(run) - row = { - "operation": operation, - "px": px, - "force_math": force_math, - "element_size": element_size, - "oom": peak is None, - } - if peak is None: - return row - - estimate = estimate_vae_working_memory_flux2( - operation=operation, image_tensor=x, vae=vae, device=device - ).reservation_bytes - seq_len = (px // VAE_SPATIAL_COMPRESSION) ** 2 - score_bytes = sdpa_score_matrix_bytes( - device=device, dtype=param.dtype, num_heads=1, head_dim=VAE_MID_BLOCK_HEAD_DIM, seq_len=seq_len - ) - if force_math and score_bytes == 0: - # Fused here, so the estimator charged nothing -- but we forced math, so price it anyway. - score_bytes = seq_len * seq_len * SDPA_MATH_BYTES_PER_SCORE_ELEMENT - estimate = max(estimate, score_bytes) - - return row | { - "reserved_delta": peak, - "estimate": estimate, - "score_term": score_bytes, - # The whole measured peak over the pixel area, which is what the `k_decode`/`k_encode` - # coefficients name. The score matrix is deliberately NOT backed out: the two terms peak in different - # phases of the forward, so the estimate takes the larger of them rather than the sum, and - # the convolution phase is what sets the high-water mark until the quadratic term overtakes - # it past ~1280px. Where `score_term` exceeds the measured peak, this column is not the - # quantity being fitted -- the score model is, and it is bounding rather than describing. - "implied_linear_constant": peak / (px * px * element_size), - "covered": peak <= estimate, - } - - -# -------------------------------------------------------------------------------------------- -# 4. The denoise per-token constant, and its width scaling -# -------------------------------------------------------------------------------------------- - - -@torch.inference_mode() -def measure_denoise(hidden: int, context_dim: int, seq_len: int, blocks: int, dtype: torch.dtype) -> dict: - """Peak reserved memory for one transformer forward at a given width and sequence length. - - Uses a reduced block count on purpose: the per-token cost is block-count independent, because a - no-grad forward frees each block's intermediates as it goes. ``--blocks`` measures a second point - so that assumption can be re-checked on this build rather than inherited. - """ - from diffusers import Flux2Transformer2DModel - - device = torch.device("cuda") - model = ( - Flux2Transformer2DModel( - num_layers=blocks, - num_single_layers=blocks, - num_attention_heads=hidden // FLUX2_ATTENTION_HEAD_DIM, - attention_head_dim=FLUX2_ATTENTION_HEAD_DIM, - joint_attention_dim=context_dim, - ) - .to(device=device, dtype=dtype) - .eval() - ) - - img_tokens = seq_len - DENOISE_TEXT_TOKENS - kwargs = { - "hidden_states": torch.randn(1, img_tokens, 128, **_td(device, dtype)), - "encoder_hidden_states": torch.randn(1, DENOISE_TEXT_TOKENS, context_dim, **_td(device, dtype)), - "timestep": torch.full((1,), 0.5, **_td(device, dtype)), - "img_ids": torch.zeros(img_tokens, 4, **_td(device, dtype)), - "txt_ids": torch.zeros(DENOISE_TEXT_TOKENS, 4, **_td(device, dtype)), - "guidance": torch.full((1,), 4.0, **_td(device, dtype)), - "return_dict": False, - } - - # Warm up first, then drop the allocator's cache: the cold call pays one-off weight-cast and - # workspace costs that the per-token slope must not absorb. The slope is a difference between - # two of these, so any constant overhead cancels either way -- warming just reduces the noise. - try: - model(**kwargs) - except (torch.cuda.OutOfMemoryError, RuntimeError) as e: - if "out of memory" not in str(e).lower(): - raise - return {"hidden": hidden, "seq_len": seq_len, "blocks": blocks, "oom": True} - - peak = _peak_reserved(lambda: model(**kwargs)) - row = {"hidden": hidden, "seq_len": seq_len, "blocks": blocks, "oom": peak is None} - if peak is not None: - row |= {"reserved_delta": peak} - return row - - # -------------------------------------------------------------------------------------------- # plumbing # -------------------------------------------------------------------------------------------- -def _td(device: torch.device, dtype: torch.dtype) -> dict: - return {"device": device, "dtype": dtype} - - -class _null_context: - def __enter__(self): - return self - - def __exit__(self, *args): - return False - - def _report_environment() -> None: """Print the env vars that can move these numbers, and warn about the one that invalidates them. @@ -393,26 +174,6 @@ def _report_environment() -> None: ) -def _flag_non_monotonic(rows: list[dict]) -> None: - """Peak cannot fall as the input grows. If it does, the reading is not measuring what it claims.""" - for operation in ("decode", "encode"): - for force_math in (False, True): - series = [ - r for r in rows if r["operation"] == operation and r["force_math"] is force_math and not r.get("oom") - ] - series.sort(key=lambda r: r["px"]) - for earlier, later in zip(series, series[1:], strict=False): - if later["reserved_delta"] < earlier["reserved_delta"]: - print( - f" WARNING: {operation} peak FELL from {earlier['px']}px to {later['px']}px " - f"({earlier['reserved_delta'] / GIB:.3f} -> {later['reserved_delta'] / GIB:.3f} GiB, " - f"force_math={force_math}). Peak cannot decrease as the input grows; the larger " - "points are being clipped (allocator GC threshold, or another process freeing " - "memory). Do not fit a constant to this run." - ) - break - - def _run_point(args: list[str]) -> dict | None: """Run one measurement in a fresh subprocess and return its JSON row.""" proc = subprocess.run([sys.executable, __file__, "--single", *args], capture_output=True, text=True) @@ -463,153 +224,15 @@ def report_sdpa(shapes: list[tuple[int, int, int]], dtype_name: str) -> list[dic return rows -def report_vae(pxs: list[int], dtype_name: str, vae_path: str | None) -> list[dict]: - decode_k = _shipped_vae_coefficient("decode") - encode_k = _shipped_vae_coefficient("encode") - print(f"\n=== 3. VAE linear coefficients (this build selects {decode_k:.0f} decode / {encode_k:.0f} encode) ===") - print("`implied_k` is the measured peak over pixel area, directly comparable to those coefficients;") - print("fit on the rows whose `math` column matches what this build really does (section 1).") - print("`covered` is the question that matters: is the shipped estimate an upper bound here?") - print("Caveat: forcing math on a build that HAS a fused kernel is not equivalent to a build that") - print("has none -- on cuDNN the forced-math decode measures *below* the fused one until 1536px,") - print("because the memory-efficient kernel's workspace is the larger term. Only a real run on the") - print("materializing build calibrates it.\n") - print( - f"{'op':7} {'px':>5} {'math':>5} {'measured(GiB)':>14} {'estimate(GiB)':>14} {'implied_k':>10} {'covered':>8}" - ) - print("-" * 70) - rows = [] - for operation in ("decode", "encode"): - for px in pxs: - for force_math in (False, True): - args = ["vae", operation, str(px), dtype_name, "1" if force_math else "0"] - if vae_path: - args.append(vae_path) - row = _run_point(args) - if row is None: - continue - rows.append(row) - if row.get("oom"): - print(f"{operation:7} {px:>5} {str(force_math):>5} {'OOM':>14}") - continue - k = row["implied_linear_constant"] - print( - f"{operation:7} {px:>5} {str(force_math):>5} {row['reserved_delta'] / GIB:>14.3f} " - f"{row['estimate'] / GIB:>14.3f} {(f'{k:.0f}' if k else 'n/a'):>10} " - f"{('yes' if row['covered'] else 'NO'):>8}" - ) - print("") - _flag_non_monotonic(rows) - for operation in ("decode", "encode"): - # Compare against the column this build actually selects, not against a hard-coded number. - shipped = _shipped_vae_coefficient(operation) - for force_math in (False, True): - ks = [ - r["implied_linear_constant"] - for r in rows - if r["operation"] == operation - and r["force_math"] is force_math - and not r.get("oom") - # Skip rows where the score matrix, not the convolution phase, is what the estimate - # is bounding: there the linear constant is not the thing under test. - and r["score_term"] < r["reserved_delta"] - ] - if not ks: - continue - mode = "math" if force_math else "fused" - verdict = "OK" if shipped >= max(ks) else "SHORT" - print(f" {operation} ({mode}): implied_k max = {max(ks):.0f}, shipped = {shipped:.0f} -> {verdict}") - short = [r for r in rows if not r.get("oom") and not r["covered"]] - if short: - print("\n Points the shipped estimate does NOT cover:") - for r in short: - gap = (r["reserved_delta"] - r["estimate"]) / GIB - # The cache never reserves less than `device_working_mem_gb`, so a point under that floor - # is not actually short in production. - floored = " (absorbed by the 3GB device_working_mem_gb floor)" if r["reserved_delta"] < 3 * GIB else "" - print(f" {r['operation']} {r['px']}px force_math={r['force_math']}: short by {gap:.2f} GiB{floored}") - return rows - - -def report_denoise(dtype_name: str, blocks: int, extra_blocks: int | None) -> list[dict]: - print("\n=== 4. Denoise per-token constant and its width scaling ===") - print(f"Shipped: {0.4:.1f} MB/token at hidden={FLUX2_REFERENCE_HIDDEN_SIZE}, scaled linearly by width.\n") - print( - f"{'variant':10} {'hidden':>7} {'blocks':>7} {'short(GiB)':>11} {'long(GiB)':>10} {'MB/token':>9} {'ratio':>7}" - ) - print("-" * 68) - rows = [] - slopes: dict[int, float] = {} - block_counts = [blocks] + ([extra_blocks] if extra_blocks else []) - for variant, hidden, context_dim in VARIANTS: - for nb in block_counts: - pair = [] - for seq in (DENOISE_SEQ_SHORT, DENOISE_SEQ_LONG): - row = _run_point(["denoise", str(hidden), str(context_dim), str(seq), str(nb), dtype_name]) - if row is None or row.get("oom"): - pair = [] - break - rows.append(row) - pair.append(row["reserved_delta"]) - if not pair: - print(f"{variant.value:10} {hidden:>7} {nb:>7} {'OOM':>11}") - continue - slope = (pair[1] - pair[0]) / (DENOISE_SEQ_LONG - DENOISE_SEQ_SHORT) - if nb == blocks: - slopes[hidden] = slope - print( - f"{variant.value:10} {hidden:>7} {nb:>7} {pair[0] / GIB:>11.3f} {pair[1] / GIB:>10.3f} " - f"{slope / MIB:>9.4f} {slope / slopes.get(FLUX2_REFERENCE_HIDDEN_SIZE, slope):>7.3f}" - ) - - if extra_blocks: - print(f"\n The two block counts should agree per width; if they do not, the '{blocks} blocks stand in") - print(" for the real model' assumption does not hold on this build and the rest is suspect.") - - reference = slopes.get(FLUX2_REFERENCE_HIDDEN_SIZE) - if reference: - shipped = 0.4 * MIB - verdict = "OK" if shipped >= reference else "SHORT" - print( - f"\n hidden={FLUX2_REFERENCE_HIDDEN_SIZE}: measured {reference / MIB:.4f} MB/token, shipped 0.4 -> {verdict}" - ) - for hidden, slope in sorted(slopes.items()): - print( - f" hidden={hidden}: slope ratio {slope / reference:.3f} against width ratio " - f"{hidden / FLUX2_REFERENCE_HIDDEN_SIZE:.3f}" - ) - print("\n Those two columns should match: that is the claim that width scales the constant.") - # What the full estimator would reserve for the case #9500 describes. - for variant, hidden, _ in VARIANTS: - est = Flux2DenoiseInvocation._estimate_working_memory( - None, image_seq_len=4096, ref_image_seq_len=12288, text_seq_len=512, num_loras=0, hidden_size=hidden - ) - need = slopes.get(hidden, reference) * 16896 - print( - f" {variant.value:10} 1024px + 3 refs: estimate {est / GIB:5.2f} GiB, " - f"measured slope implies {need / GIB:5.2f} GiB" - ) - return rows - - def main() -> None: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument( "--only", - choices=["dispatch", "sdpa", "vae", "denoise"], + choices=["dispatch", "sdpa"], action="append", - help="Run only these sections (repeatable). Default: all four.", + help="Run only these sections (repeatable). Default: both.", ) parser.add_argument("--dtype", choices=list(DTYPES), default="bfloat16", help="Compute dtype. Default bfloat16.") - parser.add_argument("--max-px", type=int, default=None, help="Skip VAE resolutions above this.") - parser.add_argument("--vae", type=str, default=None, help="Optional AutoencoderKLFlux2 diffusers dir.") - parser.add_argument("--blocks", type=int, default=2, help="Transformer blocks per stream. Default 2.") - parser.add_argument( - "--extra-blocks", - type=int, - default=None, - help="Measure a second block count too, to re-check block-count independence on this build.", - ) parser.add_argument("--csv", type=str, default=None, help="Write the raw rows to CSV.") # Internal: measure one point in this process and print one JSON line. parser.add_argument("--single", nargs="*", default=None, help=argparse.SUPPRESS) @@ -617,47 +240,28 @@ def main() -> None: if not torch.cuda.is_available(): raise SystemExit("No CUDA/HIP device available.") - _check_variant_table() if args.single: kind, rest = args.single[0], args.single[1:] if kind == "sdpa": heads, seq, head_dim, dtype_name = int(rest[0]), int(rest[1]), int(rest[2]), rest[3] print(json.dumps(measure_sdpa(heads, seq, head_dim, DTYPES[dtype_name]))) - elif kind == "vae": - operation, px, dtype_name, force_math = rest[0], int(rest[1]), rest[2], rest[3] == "1" - vae_path = rest[4] if len(rest) > 4 else None - print(json.dumps(measure_vae(operation, px, DTYPES[dtype_name], force_math, vae_path))) - elif kind == "denoise": - hidden, context_dim, seq, blocks, dtype_name = ( - int(rest[0]), - int(rest[1]), - int(rest[2]), - int(rest[3]), - rest[4], - ) - print(json.dumps(measure_denoise(hidden, context_dim, seq, blocks, DTYPES[dtype_name]))) else: raise SystemExit(f"unknown point kind {kind}") return - sections = args.only or ["dispatch", "sdpa", "vae", "denoise"] + sections = args.only or ["dispatch", "sdpa"] print( f"torch {torch.__version__} | device {torch.cuda.get_device_name(0)} | " f"hip={torch.version.hip} | dtype={args.dtype}" ) _report_environment() - pxs = [p for p in DEFAULT_VAE_PX if args.max_px is None or p <= args.max_px] rows: list[dict] = [] if "dispatch" in sections: rows += [{"section": "dispatch", **r} for r in report_dispatch(args.dtype)] if "sdpa" in sections: rows += [{"section": "sdpa", **r} for r in report_sdpa(DEFAULT_SDPA_SHAPES, args.dtype)] - if "vae" in sections: - rows += [{"section": "vae", **r} for r in report_vae(pxs, args.dtype, args.vae)] - if "denoise" in sections: - rows += [{"section": "denoise", **r} for r in report_denoise(args.dtype, args.blocks, args.extra_blocks)] if args.csv: import csv diff --git a/scripts/calibrate_z_image_working_memory.py b/scripts/calibrate_z_image_working_memory.py index 7ed5c90fb97..8c3e9392f27 100644 --- a/scripts/calibrate_z_image_working_memory.py +++ b/scripts/calibrate_z_image_working_memory.py @@ -1,71 +1,178 @@ -"""Calibrate the Z-Image denoise working-memory estimate against measured peak CUDA/HIP memory. - -Background ----------- -The Z-Image transformer attends over one unified sequence -- image tokens and caption tokens, each padded to a -multiple of 32 -- through diffusers' `dispatch_attention_fn`. Three numbers decide how much VRAM the model cache has -to keep free for one transformer forward (``ZImageDenoiseInvocation._estimate_working_memory``): - -1. the activation cost per attended token on a fused SDPA kernel, which scales linearly with the sequence; -2. the materialized score matrix, only where SDPA has no fused kernel for these shapes -- ROCm/Windows without - the experimental AOTriton kernels, MPS -- which `sdpa_score_matrix_bytes` charges at the shipped - `SDPA_MATH_BYTES_PER_SCORE_ELEMENT`; -3. with a ControlNet attached, the adapter's hints: one per control block, live next to the transformer's own - activations. - -This measures all three on the real geometry (dim 3840, 30 heads of 128, caption features 2560 wide) with randomly -initialized models and a reduced transformer block count: memory depends on shapes, not on weight values, and a -no-grad forward frees each block's intermediates. `--extra-blocks` re-checks that on the build at hand. Every point -runs in a fresh subprocess, so the caching allocator's history cannot contaminate the peak-reserved reading, and a -point that runs out of memory is recorded as such rather than aborting the run. - -Peak *reserved*, not allocated: the estimate is compared against free memory, so it has to bound what the -allocator actually takes off the card. - -Every point is priced before it runs -- the node's own estimate for it, the forced-math score matrix where that is -the point, and the weights -- and skipped when that exceeds free device memory. An oversized allocation does not -always fail: where the driver backs it with system RAM (NVIDIA's sysmem fallback, AMD on Windows) it pages and can -stall the whole machine. On a build whose default kernel materializes the score matrix, that skips every size above -1024px. `--force` runs them anyway. +"""Measure the working memory of one Z-Image transformer forward, as shipped-data points for `denoise.z_image`. + +What a point measures +--------------------- +The demand of one transformer forward, measured as a fresh window (`invokeai/backend/working_memory/measurement.py`, +whose definitions the in-server probe's windows share). With the model and its inputs resident: synchronize, release the +allocator's cached blocks, read the baseline and reset the peaks; run the forward; the demand is then the largest growth +of three counters over the baseline: the driver's figure for this process, torch's reserved peak and its allocated peak. +A forward whose allocations failed, or only succeeded after the allocator released its cache and retried (a squeezed +peak), or after which Windows keeps more than `measurement.SPILL_TOLERANCE_BYTES` of the process in shared memory, is +invalid: it is recorded as such, not as a point. + +Every point runs in a fresh process (`scripts/working_memory/fresh_points.py`, which the calibration scripts share), +which first applies the server's runtime (`apply_server_runtime` in `invokeai/app/run_app.py`): the `MIOPEN_FIND_MODE` +default, the integrated-GPU check, the allocator, the AOTriton decision and the monkeypatches, the ROCm SDPA guard among +them, in the server's order. Its settings are those of a server without a config file, except the allocator +configuration: `--alloc-conf`, or without one the platform default the server uses (expandable segments for ROCm on +Windows, torch's native allocator elsewhere). This script's own process runs the integrated-GPU check once before the +first point: the measuring processes inherit the `HIP_VISIBLE_DEVICES` it sets, which their own check keeps, as the +server keeps an exported one. + +The forward then runs twice, cold and warm, and the point takes the larger demand (`measure_cold_and_warm` and +`measured_bytes_and_method` in `measurement.py`, which every calibration script shares): a `demand_peak` where both +forwards read the driver's figure, else a `reserved_peak` of torch's counters alone. + +The transformer has the real geometry (dim 3840, 30 heads of 128, caption features 2560 wide) and random weights in the +measured dtype, because memory depends on shapes, not on weight values. It has 2 main blocks and one refiner block each: +a no-grad forward frees each block's intermediates, which `--extra-blocks 4` re-checks at 1024px. A point carries the +features of `estimate_z_image_denoise_working_memory`, taken in the measuring process after the runtime setup, so its +`score_matrix_bytes` reflects the attention kernel this build really runs. + +Each point is priced before it runs: the estimate for it plus its weights in its dtype. It is skipped when that exceeds +free device memory, unless `--force`. An oversized allocation does not always fail: where the driver backs it with +system RAM (NVIDIA's sysmem fallback, AMD on Windows) it pages, which can stall the whole machine. + +Modes +----- +- `fit` (default): the fit grid, in each of `--dtypes` (bfloat16, float16, float32). The `--sizes` (512, 768, 1024, + 1440, 2048 px) run with a `--text-tokens` caption (512). Those among `--short-text-sizes` (1024, 1440) also run with + a `--short-text-tokens` caption (77), for the text slope. `--extra-blocks N` adds one forward with N main blocks at + 1024px, in the first dtype. +- `math`: SDPA forced onto the math kernel at 1024px, next to the default kernel, per dtype: what a materialized score + matrix costs per element, against what the estimate charges for it. +- `control`: a `--control-blocks` ControlNet adapter (15) at 1024px, next to no control, per dtype: how many + hint-sized tensors it keeps, against the n + 4 the estimate charges. + +Only the fit grid's points are fit points (`"fit": true`). The checks' points, the extra-blocks forward included, test +analytic terms or assumptions rather than the line: they are marked `"fit": false`, and their `in_fit` is false. + +Output +------ +Every measured point is appended to `--out` as one JSON object per line, the point file +`scripts/working_memory/build_shipped.py --points` reads. It carries the fields of a `MeasuredPoint` +(`invokeai/backend/working_memory/coefficients.py`): + +- `estimator`, `version` and `features`, the estimate's own; +- `platform`, as `platform_for` gives it, OS included; the builder names the OS only where the data needs it; +- `measured_bytes` (the demand), `method`, `device` and `in_fit`; +- `provenance`, which names the GPU, the torch build, the OS, the allocator configuration, on ROCm `MIOPEN_FIND_MODE`, + the AOTriton switch and the SDPA guard, the block count and the date. + +The other keys are the script's, which the builder ignores: `fit`, `point` (the grid entry) and `raw` (each forward's +counter deltas, the prediction and the runtime facts). An invalid, skipped or failed forward is no point: it shows in +the table on stdout, with its reasons, and stays out of the file. Usage ----- - python scripts/calibrate_z_image_working_memory.py - python scripts/calibrate_z_image_working_memory.py --extra-blocks 4 --control-blocks 6 - python scripts/calibrate_z_image_working_memory.py --max-px 1024 --control-blocks 0 +RTX 4090 (CUDA): the native allocator, then `cudaMallocAsync`; the checks once. + + python scripts/calibrate_z_image_working_memory.py --out 4090-native.jsonl --extra-blocks 4 + python scripts/calibrate_z_image_working_memory.py --out 4090-async.jsonl --alloc-conf backend:cudaMallocAsync + python scripts/calibrate_z_image_working_memory.py --out 4090-checks.jsonl --mode math + python scripts/calibrate_z_image_working_memory.py --out 4090-checks.jsonl --mode control + +RX 9060 XT (ROCm on Windows): the server's default expandable segments, then the native allocator; the checks once. + + python scripts/calibrate_z_image_working_memory.py --out 9060xt-expandable.jsonl --extra-blocks 4 + python scripts/calibrate_z_image_working_memory.py --out 9060xt-native.jsonl --alloc-conf expandable_segments:False + python scripts/calibrate_z_image_working_memory.py --out 9060xt-checks.jsonl --mode math + python scripts/calibrate_z_image_working_memory.py --out 9060xt-checks.jsonl --mode control + +`--dry-run` prints the planned points without touching a device. """ import argparse -import json -import subprocess +import dataclasses +import datetime import sys +from collections.abc import Mapping, Sequence from contextlib import nullcontext +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any, Literal + +# None of these imports torch: a measuring process has to configure the allocator before torch is imported. +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from invokeai.app.invocations.constants import LATENT_SCALE_FACTOR # noqa: E402 +from scripts.working_memory import fresh_points # noqa: E402 -import torch -from torch.nn.attention import SDPBackend, sdpa_kernel +if TYPE_CHECKING: + import torch -from invokeai.app.invocations.constants import LATENT_SCALE_FACTOR -from invokeai.app.invocations.z_image.z_image_denoise import ( - Z_IMAGE_ATTENTION_HEADS, - Z_IMAGE_HIDDEN_SIZE, - ZImageDenoiseInvocation, - _padded, -) -from invokeai.backend.util.attention import SDPA_MATH_BYTES_PER_SCORE_ELEMENT -GIB = 1024**3 -MIB = 1024**2 +GIB = 2**30 +MIB = 2**20 CAP_FEAT_DIM = 2560 PATCH_SIZE = 2 CONTROL_IN_DIM = 33 # V2.0 adapters; the channel count only sizes the patch embedder -# One transformer block at full width: attention 4 dim^2, SwiGLU 8 dim^2, adaLN and norms, rounded up. -BLOCK_PARAMS = 13 * Z_IMAGE_HIDDEN_SIZE**2 - -SIZES = (1024, 1440, 2048) -TEXT_TOKENS = 512 - -DTYPES = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32} +CHECK_PX = 1024 +"""The size the checks and the extra-blocks forward run at.""" + +DTYPES = ("bfloat16", "float16", "float32") +LOGGER_NAME = "calibrate_z_image_working_memory" + +Kind = Literal["fit", "blocks", "reference", "math", "control"] +Mode = Literal["fit", "math", "control"] + + +@dataclass(frozen=True) +class Point: + """One transformer forward to measure, in a process of its own.""" + + kind: Kind + """`fit` for a point of the fit grid; otherwise the check it serves: another block count (`blocks`), the + default-kernel forward a check compares with (`reference`), the forced math kernel (`math`) or a ControlNet + (`control`).""" + px: int + text_tokens: int + """The caption's tokens, unpadded.""" + dtype: str + """A torch dtype name, one of `DTYPES`.""" + blocks: int + force_math: bool = False + control_blocks: int = 0 + + @property + def fit(self) -> bool: + return self.kind == "fit" + + +def plan( + mode: Mode, + *, + sizes: Sequence[int], + dtypes: Sequence[str], + text_tokens: int, + short_text_tokens: int, + short_text_sizes: Sequence[int], + blocks: int, + extra_blocks: int, + control_blocks: int, +) -> list[Point]: + """The forwards a run measures, in order: per dtype, the fit grid or a check's pair of forwards.""" + points: list[Point] = [] + for dtype in dtypes: + if mode == "fit": + points += [Point("fit", px, text_tokens, dtype, blocks) for px in sizes] + points += [Point("fit", px, short_text_tokens, dtype, blocks) for px in sizes if px in short_text_sizes] + continue + points.append(Point("reference", CHECK_PX, text_tokens, dtype, blocks)) + points.append( + Point( + mode, + CHECK_PX, + text_tokens, + dtype, + blocks, + force_math=mode == "math", + control_blocks=control_blocks if mode == "control" else 0, + ) + ) + if mode == "fit" and extra_blocks: + points.append(Point("blocks", CHECK_PX, text_tokens, dtypes[0], extra_blocks)) + return points def image_tokens(px: int) -> int: @@ -73,77 +180,30 @@ def image_tokens(px: int) -> int: return side * side -def unified_tokens(px: int, text_tokens: int) -> int: - return _padded(image_tokens(px)) + _padded(text_tokens) - - -def predicted_bytes(px: int, blocks: int, dtype: torch.dtype, force_math: bool, control_blocks: int) -> int: - """What a point is expected to take: working memory as the node estimates it, plus the weights.""" - working = ZImageDenoiseInvocation._estimate_working_memory( - image_seq_len=image_tokens(px), - text_seq_len=TEXT_TOKENS, - num_loras=0, - num_control_blocks=control_blocks, - has_attention_mask=control_blocks > 0, - device=torch.device("cuda"), - dtype=dtype, +def provenance(point: Point, facts: Mapping[str, Any], date: str) -> str: + """A measured point's `provenance`: where and how it was measured.""" + regime = {"math": ", SDPA forced onto the math kernel", "control": f", a {point.control_blocks}-block ControlNet"} + blocks = f"{point.blocks} main block{'' if point.blocks == 1 else 's'}" + return ( + f"scripts/calibrate_z_image_working_memory.py ({date}): {fresh_points.runtime_provenance(facts)}. Random-init " + f"Z-Image transformer with the real geometry, {blocks} and one refiner block each{regime.get(point.kind, '')}; " + "memory does not depend on weight values. A fresh process with the server's runtime, the larger demand of a " + "cold and a warm forward; no failed allocation, allocator retry or spill." ) - if force_math: - tokens = unified_tokens(px, TEXT_TOKENS) - working += Z_IMAGE_ATTENTION_HEADS * tokens * tokens * SDPA_MATH_BYTES_PER_SCORE_ELEMENT - # Main blocks plus refiners and embedders; adapter blocks plus its refiners. - modules = blocks + 4 + (control_blocks + 2 if control_blocks else 0) - return working + modules * BLOCK_PARAMS * dtype.itemsize - - -def _peak_reserved(fn) -> int | None: - """Run ``fn`` and return the growth in peak reserved bytes, or ``None`` if it ran out of memory.""" - torch.cuda.synchronize() - torch.cuda.empty_cache() - torch.cuda.reset_peak_memory_stats() - baseline = torch.cuda.memory_reserved() - try: - fn() - torch.cuda.synchronize() - except (torch.cuda.OutOfMemoryError, RuntimeError) as e: - if "out of memory" not in str(e).lower(): - raise - return None - return torch.cuda.max_memory_reserved() - baseline - - -def _control_adapter(blocks: int, device: torch.device, dtype: torch.dtype) -> torch.nn.Module: - """A randomly initialized adapter, allocated directly in ``dtype`` on the device: 15 blocks are ~3B parameters, - which would otherwise pass through the host in float32 first.""" - from invokeai.backend.z_image.z_image_control_adapter import ZImageControlAdapter - with torch.device("meta"): - adapter = ZImageControlAdapter( - num_control_blocks=blocks, - control_in_dim=CONTROL_IN_DIM, - dim=Z_IMAGE_HIDDEN_SIZE, - n_heads=Z_IMAGE_ATTENTION_HEADS, - n_kv_heads=Z_IMAGE_ATTENTION_HEADS, - ) - adapter = adapter.to(dtype).to_empty(device=device) - for param in adapter.parameters(): - torch.nn.init.normal_(param, std=0.02) - return adapter.eval() + +# The measuring process. Everything below imports torch, so it runs only after `fresh_points.apply_server_runtime`. -@torch.inference_mode() -def measure(px: int, text_tokens: int, blocks: int, dtype: torch.dtype, force_math: bool, control_blocks: int) -> dict: - """Peak reserved memory for one transformer forward at one resolution.""" +def _transformer(blocks: int) -> "torch.nn.Module": + """The Z-Image transformer's geometry with `blocks` main blocks and one refiner block each, on the meta device.""" + import torch from diffusers import ZImageTransformer2DModel - from invokeai.backend.z_image.z_image_controlnet_extension import ( - ZImageControlNetExtension, - z_image_forward_with_control, - ) + from invokeai.backend.z_image.denoise_working_memory import Z_IMAGE_ATTENTION_HEADS, Z_IMAGE_HIDDEN_SIZE - device = torch.device("cuda") - model = ( - ZImageTransformer2DModel( + with torch.device("meta"): + return ZImageTransformer2DModel( all_patch_size=(PATCH_SIZE,), all_f_patch_size=(1,), in_channels=16, @@ -160,143 +220,308 @@ def measure(px: int, text_tokens: int, blocks: int, dtype: torch.dtype, force_ma axes_dims=[32, 48, 48], axes_lens=[1024, 512, 512], ) - .to(device=device, dtype=dtype) - .eval() - ) - latent = px // LATENT_SCALE_FACTOR - x = [torch.randn(16, 1, latent, latent, device=device, dtype=dtype)] - t = torch.full((1,), 0.5, device=device, dtype=dtype) - cap_feats = [torch.randn(text_tokens, CAP_FEAT_DIM, device=device, dtype=dtype)] - - extension = None - if control_blocks: - extension = ZImageControlNetExtension( - control_adapter=_control_adapter(control_blocks, device, dtype), - control_cond=torch.randn(CONTROL_IN_DIM, 1, latent, latent, device=device, dtype=dtype), + + +def _control_adapter(blocks: int) -> "torch.nn.Module": + """A ControlNet adapter with `blocks` control blocks at the transformer's width, on the meta device.""" + import torch + + from invokeai.backend.z_image.denoise_working_memory import Z_IMAGE_ATTENTION_HEADS, Z_IMAGE_HIDDEN_SIZE + from invokeai.backend.z_image.z_image_control_adapter import ZImageControlAdapter + + with torch.device("meta"): + return ZImageControlAdapter( + num_control_blocks=blocks, + control_in_dim=CONTROL_IN_DIM, + dim=Z_IMAGE_HIDDEN_SIZE, + n_heads=Z_IMAGE_ATTENTION_HEADS, + n_kv_heads=Z_IMAGE_ATTENTION_HEADS, ) - def forward() -> None: - with sdpa_kernel(SDPBackend.MATH) if force_math else nullcontext(): - if extension is None: - model(x=x, t=t, cap_feats=cap_feats) - else: - z_image_forward_with_control( - transformer=model, x=x, t=t, cap_feats=cap_feats, control_extension=extension + +def _materialize(module: "torch.nn.Module", device: "torch.device", dtype: "torch.dtype") -> "torch.nn.Module": + """A meta-device module allocated directly in `dtype` on `device`, randomly initialized: a 15-block adapter is ~3B + parameters, which would otherwise pass through the host in float32 first.""" + import torch + + module = module.to(dtype).to_empty(device=device) + for param in module.parameters(): + torch.nn.init.normal_(param, std=0.02) + return module.eval() + + +def _measure(point: Point, force: bool) -> dict[str, Any]: + """The output line of `point`, measured in this process after `fresh_points.apply_server_runtime`.""" + import torch + from torch.nn.attention import SDPBackend, sdpa_kernel + + from invokeai.backend.util.devices import TorchDevice + from invokeai.backend.working_memory.measurement import measure_cold_and_warm + from invokeai.backend.working_memory.platform import platform_for + from invokeai.backend.z_image import denoise_working_memory + from invokeai.backend.z_image.denoise_working_memory import estimate_z_image_denoise_working_memory + from invokeai.backend.z_image.z_image_controlnet_extension import ( + ZImageControlNetExtension, + z_image_forward_with_control, + ) + + # The device the node itself takes. + device = TorchDevice.choose_torch_device() + if device.type != "cuda": + raise SystemExit(f"Needs a CUDA or ROCm device; this process would run on {device}.") + dtype = getattr(torch, point.dtype) + + def kernel(): + return sdpa_kernel(SDPBackend.MATH) if point.force_math else nullcontext() + + with torch.inference_mode(): + # In the forward's kernel regime, so the score matrix is priced for the kernel that runs. + with kernel(): + estimate = estimate_z_image_denoise_working_memory( + # The point runs a randomly initialized dense model, which dequantizes nothing per forward. + torch.nn.Module(), + image_seq_len=image_tokens(point.px), + text_seq_len=point.text_tokens, + lora_count=0, + control_blocks=point.control_blocks, + # The ControlNet forward always builds a padding mask; the plain one passes none at batch size 1. + has_attention_mask=point.control_blocks > 0, + device=device, + dtype=dtype, + ) + platform = platform_for(device) + facts = fresh_points.runtime_facts(device, platform.os, denoise_working_memory.__file__) + common = { + "estimator": estimate.estimator, + "version": estimate.version, + "platform": dataclasses.asdict(platform), + "features": dict(estimate.features), + "device": facts["device"], + } + + transformer = _transformer(point.blocks) + adapter = _control_adapter(point.control_blocks) if point.control_blocks else None + on_meta = [module for module in (transformer, adapter) if module is not None] + weight_bytes = sum(param.numel() for module in on_meta for param in module.parameters()) * dtype.itemsize + free, _ = TorchDevice.cuda_mem_get_info(device) + raw = { + "reservation_bytes": estimate.reservation_bytes, + "weight_bytes": weight_bytes, + "predicted_bytes": estimate.reservation_bytes + weight_bytes, + "free_bytes": free, + "runtime": facts, + } + if raw["predicted_bytes"] > free and not force: + return fresh_points.skipped_line(point, common, raw) + + transformer = _materialize(transformer, device, dtype) + latent = point.px // LATENT_SCALE_FACTOR + x = [torch.randn(16, 1, latent, latent, device=device, dtype=dtype)] + t = torch.full((1,), 0.5, device=device, dtype=dtype) + cap_feats = [torch.randn(point.text_tokens, CAP_FEAT_DIM, device=device, dtype=dtype)] + extension = None + if adapter is not None: + extension = ZImageControlNetExtension( + control_adapter=_materialize(adapter, device, dtype), + control_cond=torch.randn(CONTROL_IN_DIM, 1, latent, latent, device=device, dtype=dtype), + ) + + def forward() -> object: + with kernel(): + if extension is None: + return transformer(x=x, t=t, cap_feats=cap_feats) + return z_image_forward_with_control( + transformer=transformer, x=x, t=t, cap_feats=cap_feats, control_extension=extension ) - row = { - "px": px, - "tokens": unified_tokens(px, text_tokens), - "blocks": blocks, - "force_math": force_math, - "control_blocks": control_blocks, - } - # Warm up first: the cold call pays one-off workspace costs the per-token slope must not absorb. - if _peak_reserved(forward) is None: - return row | {"oom": True} - peak = _peak_reserved(forward) - return row | ({"oom": True} if peak is None else {"oom": False, "reserved_delta": peak}) - - -def _run_point(px: int, blocks: int, dtype_name: str, force_math: bool, control_blocks: int) -> dict | None: - args = [str(px), str(TEXT_TOKENS), str(blocks), dtype_name, "1" if force_math else "0", str(control_blocks)] - proc = subprocess.run([sys.executable, __file__, "--single", *args], capture_output=True, text=True) - line = proc.stdout.strip().splitlines()[-1] if proc.stdout.strip() else "" - try: - return json.loads(line) - except Exception: - tail = proc.stderr.strip().splitlines()[-1:] or ["(no stderr)"] - print(f" FAILED {' '.join(args)}: {tail[0]}") - return None - - -def report( - dtype_name: str, blocks: int, extra_blocks: int | None, control_blocks: int, max_px: int, force: bool -) -> None: - dtype = DTYPES[dtype_name] - sizes = [px for px in SIZES if px <= max_px] - free, _ = torch.cuda.mem_get_info() - print(f"{'kernel':>7} {'blocks':>6} {'control':>7} {'px':>5} {'tokens':>7} {'reserved(GiB)':>14}") - print("-" * 52) - - def run(kernel: str, count: int, px: int, force_math: bool, control: int) -> int | None: - need = predicted_bytes(px, count, dtype, force_math, control) - if need > free and not force: + windows = measure_cold_and_warm(device, forward) + date = datetime.date.today().isoformat() + return fresh_points.point_line( + point, common=common, provenance=provenance(point, facts, date), windows=windows, raw=raw + ) + + +_COLUMNS = f"{'kind':>9} {'dtype':>8} {'blocks':>6} {'control':>7} {'px':>5} {'text':>5} {'tokens':>7}" + + +def _grid(point: Mapping[str, Any]) -> str: + return ( + f"{point['kind']:>9} {point['dtype']:>8} {point['blocks']:>6} {point['control_blocks']:>7} " + f"{point['px']:>5} {point['text_tokens']:>5}" + ) + + +def _head(line: Mapping[str, Any]) -> str: + """A line's grid columns in the table.""" + return f"{_grid(line['point'])} {line.get('features', {}).get('tokens', ''):>7}" + + +def _tokens(line: Mapping[str, Any]) -> int: + return line["features"]["tokens"] + + +def _px(line: Mapping[str, Any]) -> int: + return line["point"]["px"] + + +def _summarize_fit(lines: Sequence[Mapping[str, Any]], text_tokens: int) -> None: + measured = [line for line in lines if line["status"] == "point"] + fit = [line for line in measured if line["fit"]] + for dtype in dict.fromkeys(line["point"]["dtype"] for line in fit): + of_dtype = [line for line in fit if line["point"]["dtype"] == dtype] + main = sorted((line for line in of_dtype if line["point"]["text_tokens"] == text_tokens), key=_tokens) + for line in main: + print(f"{dtype}, {_px(line)}px: {line['measured_bytes'] / _tokens(line) / MIB:.4f} MiB per attended token") + if len(main) >= 2: + lo, hi = main[0], main[-1] + slope = (hi["measured_bytes"] - lo["measured_bytes"]) / (_tokens(hi) - _tokens(lo)) + base = lo["measured_bytes"] - slope * _tokens(lo) + print(f"{dtype}, {_px(lo)}-{_px(hi)}px: {slope / MIB:.4f} MiB per token, intercept {base / GIB:.3f} GiB") + for short in (line for line in of_dtype if line["point"]["text_tokens"] != text_tokens): + long = next((line for line in main if _px(line) == _px(short)), None) + if long is not None and _tokens(long) != _tokens(short): + slope = (long["measured_bytes"] - short["measured_bytes"]) / (_tokens(long) - _tokens(short)) + print( + f"{dtype}, {_px(short)}px, caption of {short['point']['text_tokens']} -> {text_tokens} tokens: " + f"{slope / MIB:.4f} MiB per caption token" + ) + for line in (line for line in measured if line["point"]["kind"] == "blocks"): + same = ("px", "text_tokens", "dtype") + base = next((ref for ref in fit if all(ref["point"][key] == line["point"][key] for key in same)), None) + if base is not None: + extra = line["measured_bytes"] - base["measured_bytes"] print( - f"{kernel:>7} {count:>6} {control:>7} {px:>5} {'':>7} {'skipped':>14}" - f" needs ~{need / GIB:.1f} GiB, {free / GIB:.1f} GiB free" + f"{line['point']['blocks']} against {base['point']['blocks']} main blocks, {line['point']['dtype']}, " + f"{_px(line)}px: {extra / MIB:+.0f} MiB ({line['measured_bytes'] / base['measured_bytes']:.3f}x)" ) - return None - row = _run_point(px, count, dtype_name, force_math, control) - if row is None or row.get("oom"): - print(f"{kernel:>7} {count:>6} {control:>7} {px:>5} {'':>7} {'OOM':>14}") - return None - print(f"{kernel:>7} {count:>6} {control:>7} {px:>5} {row['tokens']:>7} {row['reserved_delta'] / GIB:>14.3f}") - return row["reserved_delta"] - - counts = [blocks] + ([extra_blocks] if extra_blocks else []) - fused: dict[int, dict[int, int]] = {} - for count in counts: - for px in sizes: - if (peak := run("auto", count, px, False, 0)) is not None: - fused.setdefault(count, {})[px] = peak - math_peak = run("math", blocks, SIZES[0], True, 0) - control_peak = run("auto", blocks, SIZES[0], False, control_blocks) if control_blocks else None - print() - for count, peaks in fused.items(): - for px, peak in peaks.items(): - tokens = unified_tokens(px, TEXT_TOKENS) - print(f"auto kernel, {count} blocks, {px}px: {peak / tokens / MIB:.4f} MB per attended token") - if len(peaks) >= 2: - lo, hi = min(peaks), max(peaks) - slope = (peaks[hi] - peaks[lo]) / (unified_tokens(hi, TEXT_TOKENS) - unified_tokens(lo, TEXT_TOKENS)) - base = peaks[lo] - slope * unified_tokens(lo, TEXT_TOKENS) - print(f"auto kernel, {count} blocks: {slope / MIB:.4f} MB/token, intercept {base / GIB:.3f} GiB") - - tokens = unified_tokens(SIZES[0], TEXT_TOKENS) - reference = fused.get(blocks, {}).get(SIZES[0]) - if math_peak is not None and reference is not None: - extra = math_peak - reference - charged = Z_IMAGE_ATTENTION_HEADS * tokens * tokens * SDPA_MATH_BYTES_PER_SCORE_ELEMENT - print( - f"math kernel, {SIZES[0]}px: {extra / GIB:.3f} GiB over the auto kernel, " - f"{extra / (Z_IMAGE_ATTENTION_HEADS * tokens * tokens):.2f} bytes/score element; shipped constant charges " - f"{charged / GIB:.3f} GiB ({SDPA_MATH_BYTES_PER_SCORE_ELEMENT})" - ) - if control_peak is not None and reference is not None: - extra = control_peak - reference - hint_bytes = tokens * Z_IMAGE_HIDDEN_SIZE * dtype.itemsize - print( - f"control, {control_blocks} blocks, {SIZES[0]}px: {extra / GIB:.3f} GiB over no control, " - f"{extra / hint_bytes:.2f} hint-sized tensors; the estimate charges n + 4 = {control_blocks + 4}" + +def _summarize_checks(lines: Sequence[Mapping[str, Any]]) -> None: + from invokeai.backend.util.attention import SDPA_MATH_BYTES_PER_SCORE_ELEMENT + from invokeai.backend.z_image.denoise_working_memory import Z_IMAGE_ATTENTION_HEADS, Z_IMAGE_HIDDEN_SIZE + + measured = [line for line in lines if line["status"] == "point"] + for line in (line for line in measured if line["point"]["kind"] in ("math", "control")): + point = line["point"] + reference = next( + ( + ref + for ref in measured + if ref["point"]["kind"] == "reference" and ref["point"]["dtype"] == point["dtype"] + ), + None, ) + if reference is None: + print(f"{point['kind']}, {point['dtype']}: no default-kernel forward to compare with") + continue + extra = line["measured_bytes"] - reference["measured_bytes"] + charged = line["raw"]["reservation_bytes"] - reference["raw"]["reservation_bytes"] + tokens = _tokens(line) + if point["kind"] == "math": + elements = Z_IMAGE_ATTENTION_HEADS * tokens * tokens + print( + f"math kernel, {point['dtype']}, {point['px']}px: {extra / GIB:.3f} GiB over the default kernel, " + f"{extra / elements:.2f} bytes per score element; the estimate adds {charged / GIB:.3f} GiB " + f"({SDPA_MATH_BYTES_PER_SCORE_ELEMENT} bytes per element, one chunk where the ROCm guard chunks)" + ) + else: + hint_bytes = tokens * Z_IMAGE_HIDDEN_SIZE * (4 if point["dtype"] == "float32" else 2) + blocks = point["control_blocks"] + print( + f"control, {blocks} blocks, {point['dtype']}, {point['px']}px: {extra / GIB:.3f} GiB over no control, " + f"{extra / hint_bytes:.2f} hint-sized tensors; the estimate adds {charged / GIB:.3f} GiB " + f"({blocks} + 4 hints, and any score matrix the mask brings)" + ) + + +def _print_plan(points: Sequence[Point]) -> None: + from invokeai.backend.z_image.denoise_working_memory import padded_tokens + + print(_COLUMNS) + for point in points: + tokens = padded_tokens(image_tokens(point.px)) + padded_tokens(point.text_tokens) + print(f"{_grid(dataclasses.asdict(point))} {tokens:>7}{' forced math' if point.force_math else ''}") + print(f"{len(points)} forwards, one process each") + + +def _sizes(text: str) -> list[int]: + try: + sizes = [int(part) for part in text.split(",") if part.strip()] + except ValueError: + raise argparse.ArgumentTypeError(f"not a comma-separated list of sizes: {text!r}") + multiple = LATENT_SCALE_FACTOR * PATCH_SIZE + if any(px <= 0 or px % multiple for px in sizes): + raise argparse.ArgumentTypeError(f"sizes must be positive multiples of {multiple} px, got {text!r}") + return sizes + + +def _dtypes(text: str) -> list[str]: + names = [part.strip() for part in text.split(",") if part.strip()] + if not names or not set(names) <= set(DTYPES): + raise argparse.ArgumentTypeError(f"dtypes must be among {', '.join(DTYPES)}, got {text!r}") + return names def main() -> None: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("--dtype", choices=list(DTYPES), default="bfloat16") + parser.add_argument("--mode", choices=("fit", "math", "control"), default="fit", help="Default: fit.") + parser.add_argument("--out", type=Path, help="The JSON-lines file the lines are appended to. Required to measure.") + parser.add_argument("--sizes", type=_sizes, default="512,768,1024,1440,2048", help="The fit grid's sizes, px.") + parser.add_argument("--dtypes", type=_dtypes, default=",".join(DTYPES), help="Default: all three.") + parser.add_argument("--text-tokens", type=int, default=512, help="The caption at every size. Default 512.") + parser.add_argument("--short-text-tokens", type=int, default=77, help="The text-slope caption. Default 77.") + parser.add_argument( + "--short-text-sizes", type=_sizes, default="1024,1440", help="Fit sizes that also run the text-slope caption." + ) parser.add_argument("--blocks", type=int, default=2, help="Main transformer blocks. Default 2.") - parser.add_argument("--extra-blocks", type=int, default=None, help="Measure a second block count as well.") - parser.add_argument("--control-blocks", type=int, default=15, help="ControlNet blocks; 0 skips. Default 15.") - parser.add_argument("--max-px", type=int, default=SIZES[-1], help="Largest fused-kernel size to measure.") + parser.add_argument("--extra-blocks", type=int, default=0, help="Fit mode: this block count at 1024px as well.") + parser.add_argument("--control-blocks", type=int, default=15, help="The control check's adapter. Default 15.") + parser.add_argument( + "--alloc-conf", + default="", + help="The allocator configuration, as the server's pytorch_cuda_alloc_conf. Empty: the platform's default.", + ) parser.add_argument("--force", action="store_true", help="Run points predicted not to fit free device memory.") - parser.add_argument("--single", nargs="*", default=None, help=argparse.SUPPRESS) + parser.add_argument("--dry-run", action="store_true", help="Print the planned points without touching a device.") + parser.add_argument("--single", help=argparse.SUPPRESS) args = parser.parse_args() - if not torch.cuda.is_available(): - raise SystemExit("No CUDA/HIP device available.") - if args.single: - px, text_tokens, blocks, dtype_name, force_math, control_blocks = args.single - row = measure( - int(px), int(text_tokens), int(blocks), DTYPES[dtype_name], force_math == "1", int(control_blocks) - ) - print(json.dumps(row)) + if args.single is not None: + fresh_points.measure_in_this_process(args.single, point_type=Point, measure=_measure, logger_name=LOGGER_NAME) + points = plan( + args.mode, + sizes=args.sizes, + dtypes=args.dtypes, + text_tokens=args.text_tokens, + short_text_tokens=args.short_text_tokens, + short_text_sizes=args.short_text_sizes, + blocks=args.blocks, + extra_blocks=args.extra_blocks, + control_blocks=args.control_blocks, + ) + if args.dry_run: + _print_plan(points) return - print( - f"torch {torch.__version__} | device {torch.cuda.get_device_name(0)} | hip={torch.version.hip} | {args.dtype}" + if args.out is None: + parser.error("--out is required to measure") + fresh_points.check_alloc_conf(parser, args.alloc_conf) + lines = fresh_points.measure_points( + points, + script=Path(__file__).resolve(), + out=args.out, + alloc_conf=args.alloc_conf, + force=args.force, + logger_name=LOGGER_NAME, + columns=_COLUMNS, + head=_head, ) - report(args.dtype, args.blocks, args.extra_blocks, args.control_blocks, args.max_px, args.force) + + print() + if args.mode == "fit": + _summarize_fit(lines, args.text_tokens) + else: + _summarize_checks(lines) + fresh_points.print_counts(lines, args.out) if __name__ == "__main__": diff --git a/scripts/family_e2e/__init__.py b/scripts/family_e2e/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/scripts/family_e2e/__main__.py b/scripts/family_e2e/__main__.py new file mode 100644 index 00000000000..db6d3f294e1 --- /dev/null +++ b/scripts/family_e2e/__main__.py @@ -0,0 +1,264 @@ +"""Command line of the family E2E tool: `python -m scripts.family_e2e `. + +See docs/src/content/docs/development/Guides/family-e2e.mdx. +""" + +from __future__ import annotations + +import argparse +import logging +import os +import re +import sys +from datetime import datetime, timezone +from pathlib import Path + +from scripts.family_e2e.calibration import PROBE_REASON, CalibrationError, save_store_files +from scripts.family_e2e.detach import RUNNER_LOG, DetachError, spawn_runner +from scripts.family_e2e.gpu_memory import UnsupportedPlatformError +from scripts.family_e2e.graphs import ExportError +from scripts.family_e2e.manifest import DEFAULT_ARM, Manifest, ManifestError, load_manifest +from scripts.family_e2e.records import RUN_INFO_FILE, load_records, load_run_info, write_run_info +from scripts.family_e2e.references import AcceptError, accept +from scripts.family_e2e.report import write_report +from scripts.family_e2e.runner import ( + RunnerError, + gpu_lock, + is_alive, + kill_identified, + load_run_manifest, + new_run, + now_iso, + open_gpu, + preflight, + read_heartbeat, + remaining_steps, + remove_scratch_roots, + run, + scratch_root, +) +from scripts.family_e2e.server import ServerError + +logger = logging.getLogger("family_e2e") +RUN_ID = re.compile(r"^\d{8}-\d{6}-(?P.+)$") +EXPECTED_ERRORS = ( + ManifestError, + RunnerError, + ServerError, + ExportError, + AcceptError, + DetachError, + UnsupportedPlatformError, + CalibrationError, + # load_records names the malformed line of a damaged records.jsonl. + ValueError, +) + + +class UsageError(ValueError): + pass + + +def _data_dir(run_dir: Path) -> Path: + # /runs/ + return run_dir.parent.parent + + +def _resolve_run(arg: str | None, manifest_path: Path | None) -> Path: + """A run directory, a run id of the manifest's data dir, or (no argument) the manifest profile's latest run.""" + if arg is not None and Path(arg).is_dir(): + return Path(arg).resolve() + if manifest_path is None: + raise UsageError("pass a run directory, or --manifest to look up run ids and the latest run") + manifest = load_manifest(manifest_path) + runs_dir = manifest.data_dir / "runs" + if arg is not None: + if not (runs_dir / arg / RUN_INFO_FILE).is_file(): + raise UsageError(f"no run {arg} in {runs_dir}") + return runs_dir / arg + runs = sorted( + d.name + for d in runs_dir.glob("*") + if (match := RUN_ID.match(d.name)) and match["profile"] == manifest.profile and (d / RUN_INFO_FILE).is_file() + ) + if not runs: + raise UsageError(f"no runs of profile {manifest.profile} in {runs_dir}") + return runs_dir / runs[-1] + + +def _check_startable(manifest: Manifest) -> None: + gpu = open_gpu(manifest.gpu) + with gpu_lock(gpu.adapter): + preflight(manifest, gpu) + + +def _print_started(run_dir: Path, pid: int) -> None: + print(f"run {run_dir.name}") + print(f"dir {run_dir}") + print(f"runner pid {pid}, log {run_dir / RUNNER_LOG}") + print(f"report {run_dir / 'report.html'} (when finished)") + print(f"status python -m scripts.family_e2e status {run_dir}") + + +def cmd_start(args: argparse.Namespace) -> int: + _check_startable(load_manifest(args.manifest)) + run_dir = new_run(args.manifest, args.family, args.case, calibrate=args.calibrate) + _print_started(run_dir, spawn_runner(run_dir)) + return 0 + + +def cmd_run(args: argparse.Namespace) -> int: + try: + return run(args.run_dir.resolve()) + except KeyboardInterrupt: + return 130 + except Exception: + logger.exception("the runner failed") + return 2 + + +def cmd_resume(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + if is_alive(read_heartbeat(run_dir).get("runner")): + raise UsageError(f"the runner of {run_dir.name} is still running") + info = load_run_info(run_dir) + _check_startable(load_run_manifest(info)) + write_run_info(run_dir, info.model_copy(update={"status": "running", "finished_at": None})) + _print_started(run_dir, spawn_runner(run_dir)) + return 0 + + +def cmd_status(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + info = load_run_info(run_dir) + records = load_records(run_dir) + left = remaining_steps(info.plan, records) + heartbeat = read_heartbeat(run_dir) + alive = is_alive(heartbeat.get("runner")) + print(f"run {info.run_id} ({info.status})") + print(f"dir {run_dir}") + print(f"progress {len(info.plan) - len(left)}/{len(info.plan)} steps final, {len(records)} records") + # Every step of a calibration run carries the probe's reason; only the others make a record invalid there. + invalid = sum(1 for r in records if set(r.invalid_reasons) - {PROBE_REASON}) + print(f"invalid {invalid} records with invalid reasons") + print(f"errors {sum(1 for r in records if r.status != 'completed')} records not completed") + if info.calibrate: + print( + f"calibrate time and VRAM not rated; {len(info.calibration_exports)} store exports in {run_dir / 'calibration'}" + ) + if heartbeat: + updated = datetime.fromisoformat(heartbeat["updated_at"]) + age = (datetime.now(timezone.utc) - updated).total_seconds() + state = "alive" if alive else "gone" + print(f"runner {state}, step: {heartbeat.get('step')}, heartbeat {age:.0f}s ago") + print(f"log {run_dir / RUNNER_LOG}") + if (run_dir / "report.html").is_file(): + print(f"report {run_dir / 'report.html'}") + if info.status == "running" and not alive: + print(f"the runner is gone; continue with: python -m scripts.family_e2e resume {run_dir}") + return 0 + + +def cmd_stop(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + heartbeat = read_heartbeat(run_dir) + # The runner first: it would otherwise notice its server dying and record a crash. + for role in ("runner", "server_launcher", "server"): + if kill_identified(heartbeat.get(role)): + print(f"killed {role} (pid {int(heartbeat[role][0])}) and its descendants") + info = load_run_info(run_dir) + if info.calibrate: + # The killed server left its calibration store in the scratch root; the root goes next. + for arm in sorted({DEFAULT_ARM, *info.environments}): + for export in save_store_files(run_dir, info, scratch_root(_data_dir(run_dir), info.run_id, arm)): + print(f"saved the calibration store {run_dir / export.export}") + # Nothing of the run is alive any more, so its scratch roots are unused. + for leftover in remove_scratch_roots(_data_dir(run_dir), info): + print(f"could not remove {leftover}") + if info.status == "running": + write_run_info( + run_dir, + info.model_copy(update={"status": "stopped", "finished_at": now_iso()}), + ) + print(f"{info.run_id} stopped; continue with: python -m scripts.family_e2e resume {run_dir}") + return 0 + + +def cmd_accept(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + for reference in accept(_data_dir(run_dir), run_dir, args.case or None): + print(f"accepted {reference.case_key} from {reference.source_run_id}") + return 0 + + +def cmd_report(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + result = write_report(_data_dir(run_dir), run_dir) + print(f"report {result.report_html}") + print(f"summary {result.summary_md}") + print(f"{result.failures} failures, {result.outliers} outliers, {result.invalid} invalid") + return 1 if result.failures else 0 + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="python -m scripts.family_e2e", description="End-to-end runs per model family." + ) + commands = parser.add_subparsers(dest="command", required=True) + + start = commands.add_parser("start", help="create a run and start its runner in the background") + start.add_argument("--manifest", type=Path, required=True) + start.add_argument("--family", action="append", default=[], help="only this family (repeatable)") + start.add_argument("--case", action="append", default=[], help="only this case id (repeatable)") + start.add_argument( + "--calibrate", + action="store_true", + help="measure working memory: submit every step through the server's calibrate route and save its exports", + ) + start.set_defaults(handler=cmd_start) + + runner = commands.add_parser("run", help="run the remaining steps of a run in the foreground (what start launches)") + runner.add_argument("--run-dir", type=Path, required=True) + runner.add_argument("--log-file", type=Path, help="append stdout and stderr to this file") + runner.set_defaults(handler=cmd_run) + + for name, handler, help_text in ( + ("resume", cmd_resume, "start the runner of a stopped or crashed run again"), + ("status", cmd_status, "show progress and whether the runner is alive"), + ("stop", cmd_stop, "kill the runner and its server"), + ("accept", cmd_accept, "make a run's measured results the references of its profile"), + ("report", cmd_report, "write report.html and summary.md of a run again"), + ): + command = commands.add_parser(name, help=help_text) + command.add_argument("run", nargs="?", help="run directory or run id; default: the profile's latest run") + command.add_argument("--manifest", type=Path, help="resolves run ids and the latest run") + if name == "accept": + command.add_argument("--case", action="append", default=[], help="only this case key (repeatable)") + command.set_defaults(handler=handler) + return parser + + +def _redirect_stdio(path: Path) -> None: + # A background runner has no usable stdio; tracebacks of a crash must land in its log too. + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND) + os.dup2(fd, 1) + os.dup2(fd, 2) + os.close(fd) + sys.stdout = open(1, "w", encoding="utf-8", buffering=1, closefd=False) + sys.stderr = open(2, "w", encoding="utf-8", buffering=1, closefd=False) + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + if getattr(args, "log_file", None): + _redirect_stdio(args.log_file) + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", stream=sys.stderr) + try: + return args.handler(args) + except (UsageError, *EXPECTED_ERRORS) as e: + print(f"error: {e}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/family_e2e/calibration.py b/scripts/family_e2e/calibration.py new file mode 100644 index 00000000000..94f7dcf0dec --- /dev/null +++ b/scripts/family_e2e/calibration.py @@ -0,0 +1,128 @@ +"""Calibration runs: each step goes through the server's working-memory calibrate route instead of enqueue_batch. + +The server measures every model operation of such a queue item with its working-memory probe and keeps the windows in +its calibration store, one file per device class under `/working_memory/` of the scratch root. The scratch root +goes at the end of each server lifetime, so every lifetime saves its store into the run's `calibration/` first: the +export route's bytes and the status while the server still runs, or the store file itself when the server died or was +stopped. `scripts/working_memory/build_shipped.py` turns the exports into shipped working-memory data. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Literal +from urllib.parse import quote + +from scripts.family_e2e.client import ApiError, InvokeClient +from scripts.family_e2e.records import ( + CalibrationExport, + CalibrationSubmission, + RunInfo, + write_bytes_atomic, + write_run_info, +) +from scripts.family_e2e.server import SCRATCH_PATHS + +CALIBRATION_DIR = "calibration" +STATUS_PATH = "/api/v2/models/working_memory/status" +EXPORT_PATH = "/api/v2/models/working_memory/export" +# The store's directory under the root's db_dir (invokeai/app/api/dependencies.py) and its file names +# (working_memory_calibration_store.py). Copied, not imported: the tool must not import invokeai, which imports torch. +STORE_DIR = "working_memory" +STORE_FILE = re.compile(r"^[0-9a-f]{16}\.json$") +PROBE_REASON = "calibration run: the probe synchronizes and empties the allocator cache around every model operation" + + +CASE_REFUSALS = (404, 422) +"""Calibrate refusals that concern one case (an unknown model, a graph the run cannot measure); the step gets an error +record and the run goes on. Any other error response would refuse every later step too.""" + + +class CalibrationError(RuntimeError): + """The server cannot run a calibration run, or refused one.""" + + +class CaseRefused(CalibrationError): + """The server refused one case's calibration run (`CASE_REFUSALS`).""" + + +def check_available(client: InvokeClient) -> None: + status = client.get(STATUS_PATH) + if not status.get("available"): + raise CalibrationError(f"the server cannot calibrate: {status.get('unavailable_reason')}") + + +def submit(client: InvokeClient, model_key: str, batch: dict[str, Any]) -> tuple[int, CalibrationSubmission]: + """Starts a calibration run of the batch's graph and data; returns its one queue item and the run. + + A refusal raises `CaseRefused` for a `CASE_REFUSALS` status and CalibrationError for any other error response. + Each server lifetime has a fresh queue and store and the tool submits each attempt once, so it sends no retry key. + """ + body: dict[str, Any] = {"graph": batch["graph"]} + if batch.get("data"): + body["data"] = batch["data"] + try: + created = client.post(f"/api/v2/models/i/{quote(model_key, safe='')}/working_memory/calibrate", body) + except ApiError as e: + refused = CaseRefused if e.status_code in CASE_REFUSALS else CalibrationError + raise refused(f"the server refused the calibration run: {e}") from e + if len(created["item_ids"]) != 1: + raise CalibrationError(f"expected one queue item from the calibration run, got {len(created['item_ids'])}") + submission = CalibrationSubmission(run_id=created["run_id"], device_class_id=created["device_class_id"]) + return created["item_ids"][0], submission + + +def _save( + run_dir: Path, + info: RunInfo, + stores: Iterable[tuple[str, bytes]], + status: bytes | None, + source: Literal["api", "store_file"], +) -> list[CalibrationExport]: + lifetime = max((e.lifetime for e in info.calibration_exports), default=0) + 1 + (run_dir / CALIBRATION_DIR).mkdir(exist_ok=True) + status_path = None + if status is not None: + status_path = f"{CALIBRATION_DIR}/{info.run_id}-{lifetime}-status.json" + write_bytes_atomic(run_dir / status_path, status) + saved_at = datetime.now(timezone.utc).isoformat(timespec="seconds") + exports = [] + for device_class_id, raw in stores: + path = f"{CALIBRATION_DIR}/{info.run_id}-{lifetime}-{device_class_id}.json" + write_bytes_atomic(run_dir / path, raw) + exports.append( + CalibrationExport( + lifetime=lifetime, + device_class_id=device_class_id, + export=path, + status=status_path, + source=source, + saved_at=saved_at, + ) + ) + info.calibration_exports.extend(exports) + write_run_info(run_dir, info) + return exports + + +def save_exports( + client: InvokeClient, run_dir: Path, info: RunInfo, device_class_ids: Iterable[str] +) -> list[CalibrationExport]: + """Saves the store of every device class the lifetime's runs were created for, and the status, into run.json.""" + stores = [ + (device_class_id, client.get_bytes(EXPORT_PATH, device_class_id=device_class_id)) + for device_class_id in sorted(set(device_class_ids)) + ] + return _save(run_dir, info, stores, client.get_bytes(STATUS_PATH), "api") + + +def save_store_files(run_dir: Path, info: RunInfo, scratch_root: Path) -> list[CalibrationExport]: + """Copies the store files of a scratch root whose server is gone; nothing when it has none.""" + store_dir = scratch_root / SCRATCH_PATHS["db_dir"] / STORE_DIR + files = sorted(p for p in store_dir.glob("*.json") if STORE_FILE.match(p.name)) if store_dir.is_dir() else [] + if not files: + return [] + return _save(run_dir, info, [(path.stem, path.read_bytes()) for path in files], None, "store_file") diff --git a/scripts/family_e2e/client.py b/scripts/family_e2e/client.py new file mode 100644 index 00000000000..441fde05932 --- /dev/null +++ b/scripts/family_e2e/client.py @@ -0,0 +1,309 @@ +"""HTTP and socket.io client for the server under test, and the run timing built from its events. + +This process must never import torch: a CUDA context here would land on the GPU being measured. Socket events are +stamped on arrival with `time.perf_counter()`, the clock of the memory sampler; the server's own event timestamps +have whole-second resolution, and `time.monotonic()` ticks only every 15.6 ms on Windows. +""" + +from __future__ import annotations + +import mimetypes +import threading +import time +from collections import defaultdict, deque +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import requests +import socketio + +from scripts.family_e2e.records import ModelLoad, NodeTiming, Timing + +QUEUE_ID = "default" +SOCKETIO_PATH = "/ws/socket.io" +REQUEST_TIMEOUT_S = 120.0 +# While waiting for a queue item, its status is also polled, in case a socket event was lost. +POLL_S = 2.0 +EVENT_GRACE_S = 1.0 +TERMINAL_STATUSES = ("completed", "failed", "canceled") +EVENT_NAMES = ( + "invocation_started", + "invocation_progress", + "invocation_complete", + "invocation_error", + "queue_item_status_changed", + "model_load_started", + "model_load_complete", +) +# Upper bound for the step count inferred from progress percentages. +MAX_INFERRED_STEPS = 1000 +# The server drops an image-bearing progress frame sent sooner than this after the previous one +# (invokeai/app/services/progress_previews/progress_previews_default.py). +PREVIEW_INTERVAL_S = 0.1 + + +class ApiError(RuntimeError): + def __init__(self, message: str, status_code: int | None = None) -> None: + super().__init__(message) + self.status_code = status_code + """The HTTP status of an error response; None for an accepted response the client cannot use.""" + + +class InvokeClient: + def __init__(self, base_url: str) -> None: + self.base_url = base_url.rstrip("/") + self._http = requests.Session() + + def _request(self, method: str, path: str, **kwargs: Any) -> requests.Response: + response = self._http.request(method, self.base_url + path, timeout=REQUEST_TIMEOUT_S, **kwargs) + if response.status_code >= 400: + raise ApiError(f"{method} {path} -> {response.status_code}: {response.text[:2000]}", response.status_code) + return response + + def get(self, path: str, **params: Any) -> Any: + return self._request("GET", path, params=params or None).json() + + def get_bytes(self, path: str, **params: Any) -> bytes: + return self._request("GET", path, params=params or None).content + + def post(self, path: str, body: Any = None) -> Any: + return self._request("POST", path, json=body).json() + + def image_bytes(self, image_name: str) -> bytes: + return self._request("GET", f"/api/v1/images/i/{image_name}/full").content + + def upload_image(self, path: Path) -> str: + content_type = mimetypes.guess_type(path.name)[0] or "image/png" + with path.open("rb") as f: + dto = self._request( + "POST", + "/api/v1/images/upload", + params={"image_category": "user", "is_intermediate": "true"}, + files={"file": (path.name, f, content_type)}, + ).json() + return dto["image_name"] + + def enqueue(self, batch: dict[str, Any]) -> int: + result = self.post(f"/api/v1/queue/{QUEUE_ID}/enqueue_batch", {"batch": batch, "prepend": False}) + if len(result["item_ids"]) != 1: + raise ApiError(f"expected one queue item from the batch, got {len(result['item_ids'])}") + return result["item_ids"][0] + + def queue_item(self, item_id: int) -> dict[str, Any]: + return self.get(f"/api/v1/queue/{QUEUE_ID}/i/{item_id}") + + def queue_busy(self) -> bool: + queue = self.get(f"/api/v1/queue/{QUEUE_ID}/status")["queue"] + return queue["pending"] + queue["in_progress"] > 0 + + def cancel(self, item_id: int) -> None: + self._request("PUT", f"/api/v1/queue/{QUEUE_ID}/i/{item_id}/cancel") + + +@dataclass(frozen=True) +class Event: + name: str + t: float + """`time.perf_counter()` at arrival.""" + data: dict[str, Any] + + +class EventRecorder: + """Records the queue and model-load events of a server from connect to close.""" + + def __init__(self, base_url: str) -> None: + self._base_url = base_url + # Polling only: websocket-client is not installed, and polling delivers within a few ms on localhost. + self._sio = socketio.Client(reconnection=True) + self._events: list[Event] = [] + self._changed = threading.Condition() + self._sio.on("connect", self._subscribe) + for name in EVENT_NAMES: + self._sio.on(name, self._handler(name)) + + def _subscribe(self) -> None: + # Also runs after a reconnect, which starts a new socket without room memberships. + self._sio.emit("subscribe_queue", {"queue_id": QUEUE_ID}) + + def _handler(self, name: str) -> Callable[[dict[str, Any]], None]: + def handle(data: dict[str, Any]) -> None: + t = time.perf_counter() + if name == "invocation_progress": + # Preview images are large and only their presence matters (it marks denoise steps). + data = {k: v for k, v in data.items() if k != "image"} | {"has_image": data.get("image") is not None} + with self._changed: + self._events.append(Event(name, t, data)) + self._changed.notify_all() + + return handle + + def connect(self) -> None: + self._sio.connect(self._base_url, socketio_path=SOCKETIO_PATH, transports=["polling"], wait_timeout=30) + # The connect handler's emit is not acknowledged; this call returns once the server joined the rooms. + self._sio.call("subscribe_queue", {"queue_id": QUEUE_ID}, timeout=30) + + def close(self) -> None: + self._sio.disconnect() + + def mark(self) -> int: + with self._changed: + return len(self._events) + + def since(self, mark: int) -> list[Event]: + with self._changed: + return self._events[mark:] + + def wait(self, timeout: float) -> None: + with self._changed: + self._changed.wait(timeout) + + +def wait_for_item( + client: InvokeClient, + recorder: EventRecorder, + mark: int, + item_id: int, + timeout: float, + tick: Callable[[], None], +) -> tuple[dict[str, Any], float]: + """Waits for the item to finish. Returns the item and the `perf_counter` time it finished. + + The finish time is the last arrival among the item's events once its terminal status event is in. A poll that + sees the item finished first waits EVENT_GRACE_S for that event, because the poll can overtake the events still + in transit; without it, the poll time stands in. `tick` runs before every check, at least every POLL_S seconds, + and may raise to abort the wait. Raises TimeoutError. + """ + deadline = time.perf_counter() + timeout + next_poll = time.perf_counter() + POLL_S + polled: tuple[dict[str, Any], float] | None = None + while True: + tick() + own = _item_events(recorder.since(mark), item_id) + latest = max((e.t for e in own), default=0.0) + if any(e.name == "queue_item_status_changed" and e.data.get("status") in TERMINAL_STATUSES for e in own): + return client.queue_item(item_id), latest + now = time.perf_counter() + if polled is not None and now - polled[1] >= EVENT_GRACE_S: + return polled[0], max(polled[1], latest) + if polled is None and now >= next_poll: + item = client.queue_item(item_id) + if item["status"] in TERMINAL_STATUSES: + polled = (item, now) + next_poll = now + POLL_S + if now >= deadline: + raise TimeoutError(f"queue item {item_id} did not finish within {timeout:.0f}s") + recorder.wait(min(POLL_S, EVENT_GRACE_S, max(0.0, deadline - now))) + + +def wait_for_idle_queue(client: InvokeClient, timeout: float, tick: Callable[[], None]) -> bool: + """Waits until the queue has nothing pending or in progress; False if it is still busy after `timeout`.""" + deadline = time.perf_counter() + timeout + while True: + tick() + if not client.queue_busy(): + return True + if time.perf_counter() >= deadline: + return False + time.sleep(POLL_S) + + +def _item_events(events: list[Event], item_id: int) -> list[Event]: + return [e for e in events if e.data.get("item_id") == item_id] + + +def per_step_seconds(progress: list[tuple[float, float]], expected_steps: int | None = None) -> float | None: + """Seconds per denoise step from (arrival time, percentage) pairs of one denoise node. + + Image-bearing progress frames are throttled server-side and the throttled steps emit nothing, so each frame's + step comes from its percentage (k / N); the slope over the first and last frame then needs no frame in between. + N is `expected_steps` (a run over the full schedule) when every percentage fits it. Otherwise it is the smallest + count every percentage fits, which may be a divisor of the real one when only every other step left a frame; a + step slower than two throttle intervals rules that out, and a faster one gives None. + """ + if len(progress) < 2: + return None + + def fits(n: int) -> bool: + return all(abs(p * n - round(p * n)) < 1e-6 for _, p in progress) + + trusted = expected_steps is not None and fits(expected_steps) + steps = expected_steps if trusted else next((n for n in range(1, MAX_INFERRED_STEPS + 1) if fits(n)), None) + if steps is None: + return None + (t_first, p_first), (t_last, p_last) = progress[0], progress[-1] + k_first, k_last = round(p_first * steps), round(p_last * steps) + if k_last <= k_first: + return None + seconds = (t_last - t_first) / (k_last - k_first) + if not trusted and seconds < 2 * PREVIEW_INTERVAL_S: + return None + return seconds + + +def build_timing( + events: list[Event], + item_id: int, + t_enqueue: float, + t_done: float, + item: dict[str, Any], + expected_steps: int | None = None, +) -> Timing: + """Run timing from the recorded events; every `t_*` is relative to the enqueue instant. + + `expected_steps` is the denoise step count when the run covers the whole schedule (not img2img). + """ + # The socket client runs every handler on its own thread, so the list order can differ from the arrival order. + events = sorted(events, key=lambda e: e.t) + own = _item_events(events, item_id) + starts: dict[str, tuple[str, str, float]] = {} + ends: dict[str, float] = {} + progress: dict[str, list[tuple[float, float]]] = defaultdict(list) + for event in own: + invocation = event.data.get("invocation") or {} + prepared_id = invocation.get("id") + if prepared_id is None: + continue + if event.name == "invocation_started": + starts[prepared_id] = (event.data["invocation_source_id"], invocation.get("type", ""), event.t) + elif event.name in ("invocation_complete", "invocation_error"): + ends[prepared_id] = event.t + elif event.name == "invocation_progress" and event.data.get("has_image"): + percentage = event.data.get("percentage") + if percentage is not None: + progress[prepared_id].append((event.t, percentage)) + + nodes = [ + NodeTiming( + node_id=source_id, + type=node_type, + t_start=t - t_enqueue, + t_end=ends[prepared_id] - t_enqueue if prepared_id in ends else None, + ) + for prepared_id, (source_id, node_type, t) in sorted(starts.items(), key=lambda kv: kv[1][2]) + ] + + # Model load events carry no queue item; within the run's window they belong to it. + pending: dict[tuple[str, str | None], deque[int]] = defaultdict(deque) + loads: list[dict[str, Any]] = [] + for event in events: + if event.name not in ("model_load_started", "model_load_complete") or not t_enqueue <= event.t <= t_done: + continue + key = ((event.data.get("config") or {}).get("key", ""), event.data.get("submodel_type")) + if event.name == "model_load_started": + pending[key].append(len(loads)) + loads.append({"model": key[0], "submodel": key[1], "t_start": event.t - t_enqueue, "t_end": None}) + elif pending[key]: + loads[pending[key].popleft()]["t_end"] = event.t - t_enqueue + + # The denoise node is the one with the most preview frames. + denoise = max(progress.values(), key=len, default=[]) + return Timing( + total_s=t_done - t_enqueue, + nodes=nodes, + model_loads=[ModelLoad(**load) for load in loads], + per_step_s=per_step_seconds(denoise, expected_steps), + queue_started_at=item.get("started_at"), + queue_completed_at=item.get("completed_at"), + ) diff --git a/scripts/family_e2e/detach.py b/scripts/family_e2e/detach.py new file mode 100644 index 00000000000..8b827a3fcdc --- /dev/null +++ b/scripts/family_e2e/detach.py @@ -0,0 +1,78 @@ +"""Starting the runner as a background process that outlives the caller. + +On Windows the runner breaks away from the caller's job object, so a caller that kills its job's processes (an agent +harness does after two hours) cannot take a long run with it. Where the job forbids breakaway, the runner is created +through WMI instead, which puts it in no job of the caller; such a process starts with the user's default +environment, not the caller's. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + +from scripts.family_e2e.manifest import TOOL_CHECKOUT + +RUNNER_LOG = "runner.log" +WMI_TIMEOUT_S = 120 + +# Hidden window: WMI shows a console program's window otherwise. +_WMI_CREATE = ( + "$startup = New-CimInstance -ClassName Win32_ProcessStartup -ClientOnly -Property @{ShowWindow=[uint16]0}; " + "$r = Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments @{" + "CommandLine=$env:FAMILY_E2E_COMMAND; CurrentDirectory=$env:FAMILY_E2E_CWD; ProcessStartupInformation=$startup}; " + "if ($r.ReturnValue -ne 0) { [Console]::Error.WriteLine('Win32_Process.Create returned ' + $r.ReturnValue); exit 1 }; " + "$r.ProcessId" +) + + +class DetachError(RuntimeError): + pass + + +def runner_command(run_dir: Path) -> list[str]: + return [ + sys.executable, + "-X", + "utf8", + "-m", + "scripts.family_e2e", + "run", + "--run-dir", + str(run_dir), + "--log-file", + str(run_dir / RUNNER_LOG), + ] + + +def spawn_runner(run_dir: Path) -> int: + """Starts the runner for `run_dir` in the background and returns the started process's pid.""" + args = runner_command(run_dir) + stdio = {"stdin": subprocess.DEVNULL, "stdout": subprocess.DEVNULL, "stderr": subprocess.DEVNULL} + if sys.platform != "win32": + return subprocess.Popen(args, cwd=TOOL_CHECKOUT, start_new_session=True, **stdio).pid + # CREATE_NO_WINDOW, not DETACHED_PROCESS: a venv's python.exe is a launcher, and under DETACHED_PROCESS the + # interpreter it starts gets a new visible console, which the server and every tool call would then share. + flags = subprocess.CREATE_NO_WINDOW | subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_BREAKAWAY_FROM_JOB + try: + return subprocess.Popen(args, cwd=TOOL_CHECKOUT, creationflags=flags, close_fds=True, **stdio).pid + except OSError: + return _spawn_via_wmi(args) + + +def _spawn_via_wmi(args: list[str]) -> int: + # Passed through the environment, so neither PowerShell nor WMI has to parse quoting of their own. + env = os.environ | {"FAMILY_E2E_COMMAND": subprocess.list2cmdline(args), "FAMILY_E2E_CWD": str(TOOL_CHECKOUT)} + result = subprocess.run( + ["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", _WMI_CREATE], + env=env, + capture_output=True, + text=True, + timeout=WMI_TIMEOUT_S, + check=False, + ) + if result.returncode != 0: + raise DetachError(f"starting the runner through WMI failed:\n{result.stderr.strip()}") + return int(result.stdout.strip().splitlines()[-1]) diff --git a/scripts/family_e2e/gpu_memory.py b/scripts/family_e2e/gpu_memory.py new file mode 100644 index 00000000000..f4993026140 --- /dev/null +++ b/scripts/family_e2e/gpu_memory.py @@ -0,0 +1,580 @@ +"""Driver-level video memory of another process, sampled without a GPU context of our own. + +The allocator's own figure (``torch.cuda.max_memory_allocated``) misses its pools and the cuDNN/MIOpen workspaces, so +the real peak runs 25-75 % above it. What the driver charges the process is what decides whether a card of a given size +copes, and Windows reports it per process and adapter through the WDDM kernel thunks: ``D3DKMTQueryVideoMemoryInfo`` +on another process's handle answers in about a microsecond, so the dedicated (local segment) usage can be sampled every +few milliseconds. The bytes Windows has paged out to system memory are only visible in the ``GPU Process Memory`` +performance counters (``Shared Usage``), read through PDH at a slower rate. + +The client never imports torch: a CUDA/HIP context of its own would sit on the same adapter and distort what the +server can allocate. Windows (WDDM) only for now; it covers NVIDIA and AMD/ROCm alike. +""" + +import bisect +import ctypes +import functools +import logging +import operator +import re +import sys +import threading +import time +from dataclasses import dataclass +from types import SimpleNamespace + +_STATUS_SUCCESS = 0x00000000 +# The target has no allocation context on the adapter: it never opened it, or has closed it again. +_STATUS_INVALID_PARAMETER = 0xC000000D +_STATUS_ACCESS_DENIED = 0xC0000022 +# D3DKMTQueryVideoMemoryInfo answers STATUS_ACCESS_DENIED for PROCESS_QUERY_LIMITED_INFORMATION; it needs the full +# query right (measured on Windows 10 22H2 with an NVIDIA and an AMD driver). +_PROCESS_QUERY_INFORMATION = 0x0400 +_STILL_ACTIVE = 259 +_ERROR_ACCESS_DENIED = 5 +_ERROR_INVALID_PARAMETER = 87 + +_KMTQAITYPE_ADAPTERREGISTRYINFO = 8 +_KMTQAITYPE_ADAPTERTYPE = 15 +# D3DKMT_KMD_DRIVER_VERSION: the kernel-mode driver's version, the one Device Manager shows. +_KMTQAITYPE_KMD_DRIVER_VERSION = 49 +# D3DKMT_ADAPTERTYPE bits. +_ADAPTER_RENDER_SUPPORTED = 1 << 0 +_ADAPTER_SOFTWARE_DEVICE = 1 << 2 +_ADAPTER_COMPUTE_ONLY = 1 << 11 +_MEMORY_SEGMENT_GROUP_LOCAL = 0 +_MAX_PATH = 260 + +_PDH_FMT_LARGE = 0x00000400 +_PDH_MORE_DATA = 0x800007D2 +_PDH_NO_DATA = 0x800007D5 +# PDH_CSTATUS_VALID_DATA and PDH_CSTATUS_NEW_DATA; any other status leaves `largeValue` undefined. +_PDH_VALID_DATA = (0x00000000, 0x00000001) +# English paths; PdhAddEnglishCounterW resolves them on a localized Windows too. +_PDH_DEDICATED_USAGE = r"\GPU Process Memory(*)\Dedicated Usage" +_PDH_SHARED_USAGE = r"\GPU Process Memory(*)\Shared Usage" +_PDH_INSTANCE = re.compile(r"pid_(\d+)_luid_0x([0-9a-f]{8})_0x([0-9a-f]{8})_phys_(\d+)", re.IGNORECASE) + +_logger = logging.getLogger(__name__) + +# Win32 types spelled as fixed-width ctypes, so this module imports on every platform. +_UINT = ctypes.c_uint32 +_LONG = ctypes.c_int32 +_NTSTATUS = ctypes.c_uint32 + + +class UnsupportedPlatformError(RuntimeError): + """GPU memory sampling is not implemented for this operating system.""" + + +@dataclass(frozen=True) +class Adapter: + luid: int + """``(HighPart << 32) | LowPart``, as D3DKMT and the PDH instance names spell it.""" + description: str + driver_version: str | None = None + """As Device Manager shows it, e.g. ``32.0.16.1047``; ``None`` when the driver does not report one.""" + + +@dataclass(frozen=True) +class ProcessUsage: + dedicated_bytes: int + shared_bytes: int + + +@dataclass(frozen=True) +class MemoryTrace: + """Samples of one process's usage on one adapter, stamped with the client's ``time.perf_counter()``. + + The trace is a sample-and-hold step function: a value stands from its timestamp until the next sample. + """ + + method: str + interval_ms: float + """Mean spacing of the dedicated samples; the configured interval while they span no measurable time.""" + dedicated: tuple[tuple[float, int], ...] + """Local-segment ``CurrentUsage`` in bytes.""" + shared: tuple[tuple[float, int], ...] + """PDH ``Shared Usage`` in bytes: what Windows keeps of this process's allocations in system memory.""" + + def peak(self, t0: float | None = None, t1: float | None = None) -> int | None: + """The highest dedicated usage in ``[t0, t1]``: the value in effect at ``t0`` and every sample up to ``t1``. + + Counting the value in effect at ``t0`` keeps a window shorter than the sampling interval from coming back + empty. ``None`` bounds are open; ``None`` is returned when no sample falls on or before ``t1``. + """ + if t0 is not None and t1 is not None and t1 < t0: + raise ValueError(f"window ends before it starts: [{t0}, {t1}]") + samples = self.dedicated + lo = 0 + if t0 is not None: + lo = bisect.bisect_left(samples, t0, key=_sample_time) + if lo > 0 and (lo == len(samples) or samples[lo][0] > t0): + lo -= 1 + hi = len(samples) if t1 is None else bisect.bisect_right(samples, t1, key=_sample_time) + return max((value for _, value in samples[lo:hi]), default=None) + + def at(self, t: float) -> int | None: + """The dedicated usage in effect at ``t``: the last sample at or before it.""" + index = bisect.bisect_right(self.dedicated, t, key=_sample_time) + return self.dedicated[index - 1][1] if index else None + + +_sample_time = operator.itemgetter(0) + + +class _Luid(ctypes.Structure): + _fields_ = [("LowPart", _UINT), ("HighPart", _LONG)] + + +class _AdapterInfo(ctypes.Structure): + """``D3DKMT_ADAPTERINFO``.""" + + _fields_ = [("hAdapter", _UINT), ("AdapterLuid", _Luid), ("NumOfSources", _UINT), ("bPrecise", _LONG)] + + +class _EnumAdapters2(ctypes.Structure): + """``D3DKMT_ENUMADAPTERS2``.""" + + _fields_ = [("NumAdapters", _UINT), ("pAdapters", ctypes.POINTER(_AdapterInfo))] + + +class _QueryAdapterInfo(ctypes.Structure): + """``D3DKMT_QUERYADAPTERINFO``.""" + + _fields_ = [("hAdapter", _UINT), ("Type", ctypes.c_int), ("pData", ctypes.c_void_p), ("DataSize", _UINT)] + + +class _AdapterRegistryInfo(ctypes.Structure): + """``D3DKMT_ADAPTERREGISTRYINFO``.""" + + _fields_ = [ + ("AdapterString", ctypes.c_wchar * _MAX_PATH), + ("BiosString", ctypes.c_wchar * _MAX_PATH), + ("DacType", ctypes.c_wchar * _MAX_PATH), + ("ChipType", ctypes.c_wchar * _MAX_PATH), + ] + + +class _OpenAdapterFromLuid(ctypes.Structure): + """``D3DKMT_OPENADAPTERFROMLUID``.""" + + _fields_ = [("AdapterLuid", _Luid), ("hAdapter", _UINT)] + + +class _CloseAdapter(ctypes.Structure): + """``D3DKMT_CLOSEADAPTER``.""" + + _fields_ = [("hAdapter", _UINT)] + + +class _QueryVideoMemoryInfo(ctypes.Structure): + """``D3DKMT_QUERYVIDEOMEMORYINFO``. A null ``hProcess`` means the calling process.""" + + _fields_ = [ + ("hProcess", ctypes.c_void_p), + ("hAdapter", _UINT), + ("MemorySegmentGroup", ctypes.c_int), + ("Budget", ctypes.c_uint64), + ("CurrentUsage", ctypes.c_uint64), + ("CurrentReservation", ctypes.c_uint64), + ("AvailableForReservation", ctypes.c_uint64), + ("PhysicalAdapterIndex", _UINT), + ] + + +class _PdhCounterValue(ctypes.Structure): + """``PDH_FMT_COUNTERVALUE`` read as ``PDH_FMT_LARGE``.""" + + _fields_ = [("CStatus", ctypes.c_uint32), ("largeValue", ctypes.c_int64)] + + +class _PdhCounterValueItem(ctypes.Structure): + """``PDH_FMT_COUNTERVALUE_ITEM_W``.""" + + _fields_ = [("szName", ctypes.c_wchar_p), ("FmtValue", _PdhCounterValue)] + + +def _require_windows() -> None: + if sys.platform != "win32": + raise UnsupportedPlatformError( + f"GPU memory sampling is implemented for Windows (WDDM) only, not {sys.platform}" + ) + + +@functools.cache +def _win32() -> SimpleNamespace: + """gdi32, kernel32 and pdh with the prototypes used here; loaded on first use, since only Windows has them.""" + gdi32 = ctypes.WinDLL("gdi32") + for name in ( + "D3DKMTEnumAdapters2", + "D3DKMTQueryAdapterInfo", + "D3DKMTOpenAdapterFromLuid", + "D3DKMTQueryVideoMemoryInfo", + "D3DKMTCloseAdapter", + ): + fn = getattr(gdi32, name) + fn.argtypes = [ctypes.c_void_p] + fn.restype = _NTSTATUS + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.OpenProcess.argtypes = [ctypes.c_uint32, ctypes.c_int32, ctypes.c_uint32] + kernel32.OpenProcess.restype = ctypes.c_void_p + kernel32.GetExitCodeProcess.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint32)] + kernel32.GetExitCodeProcess.restype = ctypes.c_int32 + kernel32.CloseHandle.argtypes = [ctypes.c_void_p] + kernel32.CloseHandle.restype = ctypes.c_int32 + + pdh = ctypes.WinDLL("pdh") + prototypes = { + "PdhOpenQueryW": [ctypes.c_wchar_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)], + "PdhAddEnglishCounterW": [ctypes.c_void_p, ctypes.c_wchar_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)], + "PdhCollectQueryData": [ctypes.c_void_p], + "PdhGetFormattedCounterArrayW": [ + ctypes.c_void_p, + ctypes.c_uint32, + ctypes.POINTER(ctypes.c_uint32), + ctypes.POINTER(ctypes.c_uint32), + ctypes.c_void_p, + ], + "PdhCloseQuery": [ctypes.c_void_p], + } + for name, argtypes in prototypes.items(): + fn = getattr(pdh, name) + fn.argtypes = argtypes + fn.restype = ctypes.c_uint32 + return SimpleNamespace(gdi32=gdi32, kernel32=kernel32, pdh=pdh) + + +def _check(status: int, call: str) -> None: + if status != _STATUS_SUCCESS: + raise OSError(f"{call} failed with status {status:#010x}") + + +def _luid_value(luid: _Luid) -> int: + return ((luid.HighPart & 0xFFFFFFFF) << 32) | luid.LowPart + + +def _luid_struct(value: int) -> _Luid: + return _Luid(value & 0xFFFFFFFF, value >> 32) + + +def _query_adapter_info(handle: int, info_type: int, data: ctypes.Structure | ctypes.c_uint32 | ctypes.c_uint64) -> int: + query = _QueryAdapterInfo(handle, info_type, ctypes.cast(ctypes.byref(data), ctypes.c_void_p), ctypes.sizeof(data)) + return _win32().gdi32.D3DKMTQueryAdapterInfo(ctypes.byref(query)) + + +def _describe(info: _AdapterInfo) -> Adapter | None: + """The adapter, or ``None`` for one that cannot hold a compute process's memory or cannot be named.""" + adapter_type = _UINT() + if _query_adapter_info(info.hAdapter, _KMTQAITYPE_ADAPTERTYPE, adapter_type) != _STATUS_SUCCESS: + return None + # The Microsoft Basic Render Driver is a software device; display-only adapters (indirect displays) run nothing. + if adapter_type.value & _ADAPTER_SOFTWARE_DEVICE: + return None + if not adapter_type.value & (_ADAPTER_RENDER_SUPPORTED | _ADAPTER_COMPUTE_ONLY): + return None + registry = _AdapterRegistryInfo() + if _query_adapter_info(info.hAdapter, _KMTQAITYPE_ADAPTERREGISTRYINFO, registry) != _STATUS_SUCCESS: + return None + version = ctypes.c_uint64() + status = _query_adapter_info(info.hAdapter, _KMTQAITYPE_KMD_DRIVER_VERSION, version) + return Adapter( + luid=_luid_value(info.AdapterLuid), + description=registry.AdapterString, + driver_version=_format_driver_version(version.value) if status == _STATUS_SUCCESS else None, + ) + + +def _format_driver_version(packed: int) -> str | None: + """A driver version packed as four 16-bit fields, most significant first, in dotted form; 0 is unknown.""" + if not packed: + return None + return ".".join(str(packed >> shift & 0xFFFF) for shift in (48, 32, 16, 0)) + + +def list_adapters() -> list[Adapter]: + """The hardware GPU adapters, in the order the kernel enumerates them.""" + _require_windows() + gdi32 = _win32().gdi32 + # A null array asks for the size: the call answers the most adapters it could return. + enum = _EnumAdapters2(0, None) + _check(gdi32.D3DKMTEnumAdapters2(ctypes.byref(enum)), "D3DKMTEnumAdapters2") + infos = (_AdapterInfo * enum.NumAdapters)() + enum.pAdapters = infos + _check(gdi32.D3DKMTEnumAdapters2(ctypes.byref(enum)), "D3DKMTEnumAdapters2") + opened = infos[: enum.NumAdapters] + try: + return [adapter for adapter in map(_describe, opened) if adapter is not None] + finally: + for info in opened: + gdi32.D3DKMTCloseAdapter(ctypes.byref(_CloseAdapter(info.hAdapter))) + + +def _select_adapter(adapters: list[Adapter], description_substring: str) -> Adapter: + needle = description_substring.casefold() + matches = [adapter for adapter in adapters if needle in adapter.description.casefold()] + if len(matches) == 1: + return matches[0] + available = "; ".join(f"{adapter.description} (LUID {adapter.luid:#x})" for adapter in adapters) or "none" + found = "No adapter matches" if not matches else f"{len(matches)} adapters match" + raise LookupError(f"{found} {description_substring!r}. Adapters: {available}") + + +def find_adapter(description_substring: str) -> Adapter: + """The one adapter whose description contains the substring, ignoring case. ``LookupError`` for none or several.""" + return _select_adapter(list_adapters(), description_substring) + + +def _parse_instance(name: str) -> tuple[int, int] | None: + """``(pid, luid)`` of a ``GPU Process Memory`` instance on physical adapter 0, else ``None``.""" + match = _PDH_INSTANCE.fullmatch(name) + if match is None or match[4] != "0": + return None + return int(match[1]), (int(match[2], 16) << 32) | int(match[3], 16) + + +class _GpuProcessMemoryCounters: + """One PDH query over every process's ``GPU Process Memory`` instances. + + Only the first query in a process is slow to open (about 0.35 s while PDH loads the provider); a collection costs + well under a millisecond. + """ + + def __init__(self) -> None: + pdh = _win32().pdh + self._query = ctypes.c_void_p() + _check(pdh.PdhOpenQueryW(None, None, ctypes.byref(self._query)), "PdhOpenQueryW") + try: + self._dedicated = self._add(_PDH_DEDICATED_USAGE) + self._shared = self._add(_PDH_SHARED_USAGE) + except BaseException: + self.close() + raise + + def _add(self, path: str) -> ctypes.c_void_p: + counter = ctypes.c_void_p() + status = _win32().pdh.PdhAddEnglishCounterW(self._query, path, None, ctypes.byref(counter)) + _check(status, f"PdhAddEnglishCounterW({path})") + return counter + + def collect(self) -> dict[tuple[int, int], ProcessUsage]: + """Usage keyed by ``(pid, luid)``, for every process that has opened an adapter.""" + status = _win32().pdh.PdhCollectQueryData(self._query) + if status == _PDH_NO_DATA: + return {} + _check(status, "PdhCollectQueryData") + dedicated = self._values(self._dedicated) + shared = self._values(self._shared) + return { + key: ProcessUsage(dedicated.get(key, 0), shared.get(key, 0)) for key in dedicated.keys() | shared.keys() + } + + def _values(self, counter: ctypes.c_void_p) -> dict[tuple[int, int], int]: + pdh = _win32().pdh + size, count = ctypes.c_uint32(0), ctypes.c_uint32(0) + status = pdh.PdhGetFormattedCounterArrayW( + counter, _PDH_FMT_LARGE, ctypes.byref(size), ctypes.byref(count), None + ) + if status == _STATUS_SUCCESS: + return {} + if status != _PDH_MORE_DATA: + _check(status, "PdhGetFormattedCounterArrayW") + buffer = (ctypes.c_byte * size.value)() + status = pdh.PdhGetFormattedCounterArrayW( + counter, _PDH_FMT_LARGE, ctypes.byref(size), ctypes.byref(count), buffer + ) + _check(status, "PdhGetFormattedCounterArrayW") + items = ctypes.cast(buffer, ctypes.POINTER(_PdhCounterValueItem)) + values: dict[tuple[int, int], int] = {} + for item in items[: count.value]: + key = _parse_instance(item.szName or "") + if key is not None and item.FmtValue.CStatus in _PDH_VALID_DATA: + values[key] = int(item.FmtValue.largeValue) + return values + + def close(self) -> None: + if self._query: + _win32().pdh.PdhCloseQuery(self._query) + self._query = ctypes.c_void_p() + + +def process_usage_on_adapter(adapter: Adapter) -> dict[int, ProcessUsage]: + """Every process that has opened the adapter, by pid, with its dedicated and shared usage.""" + _require_windows() + counters = _GpuProcessMemoryCounters() + try: + usage = counters.collect() + finally: + counters.close() + return {pid: value for (pid, luid), value in usage.items() if luid == adapter.luid} + + +class ProcessMemorySampler: + """Samples one process's usage on one adapter on a background thread, from ``start()`` until ``stop()``. + + Dedicated usage is read every ``interval_s``, shared usage every ``shared_interval_s``. A process that has not + opened the adapter yet reads as zero. Sampling ends early when the process exits. + + ``pid`` must be the process that allocates. On Windows the ``python.exe`` in a venv's ``Scripts`` folder and the + console-script ``.exe`` files there are launchers: they run the interpreter as a child process and read as zero. + """ + + def __init__(self, pid: int, adapter: Adapter, interval_s: float = 0.003, shared_interval_s: float = 0.1) -> None: + _require_windows() + if pid <= 0: + raise ValueError(f"invalid pid {pid}") + if interval_s <= 0 or shared_interval_s <= 0: + raise ValueError(f"sampling intervals must be positive, got {interval_s} and {shared_interval_s}") + self._pid = pid + self._adapter = adapter + self._interval_s = interval_s + self._shared_interval_s = shared_interval_s + self._dedicated: list[tuple[float, int]] = [] + self._shared: list[tuple[float, int]] = [] + self._process: int | None = None + self._adapter_handle: int | None = None + self._counters: _GpuProcessMemoryCounters | None = None + self._stopping = threading.Event() + self._thread: threading.Thread | None = None + self._trace: MemoryTrace | None = None + self._warned: set[str] = set() + + def start(self) -> None: + """Open the process and the adapter, take one sample and start the sampling thread.""" + if self._thread is not None or self._trace is not None: + raise RuntimeError("a sampler starts once") + try: + self._open() + self._sample_dedicated(raise_on_failure=True) + self._sample_shared(raise_on_failure=True) + except BaseException: + self._close() + raise + self._thread = threading.Thread(target=self._run, name=f"gpu-memory-sampler-{self._pid}", daemon=True) + self._thread.start() + + def stop(self) -> MemoryTrace: + """Stop sampling, release the handles and return the trace; later calls return the same trace.""" + if self._trace is None: + self._stopping.set() + if self._thread is not None: + self._thread.join() + self._close() + self._trace = MemoryTrace( + method="d3dkmt", + interval_ms=self._mean_interval_ms(), + dedicated=tuple(self._dedicated), + shared=tuple(self._shared), + ) + return self._trace + + def _open(self) -> None: + win32 = _win32() + process = win32.kernel32.OpenProcess(_PROCESS_QUERY_INFORMATION, 0, self._pid) + if not process: + error = ctypes.get_last_error() + if error == _ERROR_INVALID_PARAMETER: + raise ProcessLookupError(f"no process with pid {self._pid}") + if error == _ERROR_ACCESS_DENIED: + raise PermissionError(f"access denied opening process {self._pid} for memory queries") + raise ctypes.WinError(error) + self._process = process + if self._exited(): + raise ProcessLookupError(f"process {self._pid} has exited") + + opened = _OpenAdapterFromLuid(_luid_struct(self._adapter.luid), 0) + status = win32.gdi32.D3DKMTOpenAdapterFromLuid(ctypes.byref(opened)) + _check(status, f"D3DKMTOpenAdapterFromLuid for {self._adapter.description} (LUID {self._adapter.luid:#x})") + self._adapter_handle = opened.hAdapter + # The target's STATUS_INVALID_PARAMETER only means "no allocation on this adapter" once the same call has + # succeeded for this process, which proves the adapter handle and the struct layout. + status, _ = self._query_usage(None) + _check(status, "D3DKMTQueryVideoMemoryInfo for the sampling process") + self._counters = _GpuProcessMemoryCounters() + + def _close(self) -> None: + win32 = _win32() + if self._counters is not None: + self._counters.close() + self._counters = None + if self._adapter_handle is not None: + win32.gdi32.D3DKMTCloseAdapter(ctypes.byref(_CloseAdapter(self._adapter_handle))) + self._adapter_handle = None + if self._process is not None: + win32.kernel32.CloseHandle(self._process) + self._process = None + + def _query_usage(self, process: int | None) -> tuple[int, int]: + assert self._adapter_handle is not None + info = _QueryVideoMemoryInfo(process, self._adapter_handle, _MEMORY_SEGMENT_GROUP_LOCAL) + status = _win32().gdi32.D3DKMTQueryVideoMemoryInfo(ctypes.byref(info)) + return status, int(info.CurrentUsage) + + def _exited(self) -> bool: + code = ctypes.c_uint32() + if not _win32().kernel32.GetExitCodeProcess(self._process, ctypes.byref(code)): + raise ctypes.WinError(ctypes.get_last_error()) + return code.value != _STILL_ACTIVE + + def _sample_dedicated(self, raise_on_failure: bool = False) -> bool: + """Append one dedicated sample; ``False`` once the process has exited.""" + t = time.perf_counter() + status, usage = self._query_usage(self._process) + if status == _STATUS_SUCCESS: + self._dedicated.append((t, usage)) + return True + if self._exited(): + if raise_on_failure: + raise ProcessLookupError(f"process {self._pid} has exited") + return False + if status == _STATUS_INVALID_PARAMETER: + self._dedicated.append((t, 0)) + return True + if raise_on_failure: + if status == _STATUS_ACCESS_DENIED: + raise PermissionError(f"access denied querying video memory of process {self._pid}") + _check(status, f"D3DKMTQueryVideoMemoryInfo for process {self._pid}") + self._warn_once("dedicated", f"D3DKMTQueryVideoMemoryInfo failed with status {status:#010x}; sample skipped") + return True + + def _sample_shared(self, raise_on_failure: bool = False) -> None: + assert self._counters is not None + t = time.perf_counter() + try: + usage = self._counters.collect().get((self._pid, self._adapter.luid)) + except OSError as error: + if raise_on_failure: + raise + self._warn_once("shared", f"{error}; shared sample skipped") + return + self._shared.append((t, 0 if usage is None else usage.shared_bytes)) + + def _warn_once(self, series: str, message: str) -> None: + if series not in self._warned: + self._warned.add(series) + _logger.warning(f"GPU memory sampler for process {self._pid}: {message} (further failures not logged)") + + def _run(self) -> None: + clock = time.perf_counter + next_dedicated = clock() + self._interval_s + next_shared = clock() + self._shared_interval_s + while not self._stopping.is_set(): + delay = next_dedicated - clock() + if delay > 0: + time.sleep(delay) + if not self._sample_dedicated(): + return + now = clock() + if now >= next_shared: + self._sample_shared() + next_shared = now + self._shared_interval_s + # A fixed rate: a late wake-up shortens the next sleep instead of delaying every later sample, and after a + # stall (GIL, suspend) sampling resumes from now rather than bursting to catch up. + next_dedicated = max(next_dedicated + self._interval_s, now) + + def _mean_interval_ms(self) -> float: + span = self._dedicated[-1][0] - self._dedicated[0][0] if len(self._dedicated) > 1 else 0.0 + if span <= 0: + return self._interval_s * 1000 + return span / (len(self._dedicated) - 1) * 1000 diff --git a/scripts/family_e2e/graphs.py b/scripts/family_e2e/graphs.py new file mode 100644 index 00000000000..8b7f46813ea --- /dev/null +++ b/scripts/family_e2e/graphs.py @@ -0,0 +1,180 @@ +"""Graphs for the cases: the exporter request, the webv2 exporter call, and the runner's batch policy. + +The webv2 exporter builds each case's batch with the real frontend graph builders (contract 1 in +`.scratch/agents/family-e2e/contract.md`). The runner then disables the invocation cache on every node, so repeated +runs really execute, and hashes the result so a frontend graph change shows up against the reference. +""" + +from __future__ import annotations + +import copy +import hashlib +import json +import shutil +import subprocess +from collections.abc import Iterator +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from scripts.family_e2e.manifest import TOOL_CHECKOUT, ResolvedCase +from scripts.family_e2e.server import spawn_in_job, terminate + +EXPORT_VERSION = 1 +WEBV2_DIR = TOOL_CHECKOUT / "invokeai" / "frontend" / "webv2" +EXPORT_TIMEOUT_S = 600 + + +class ExportError(RuntimeError): + pass + + +@dataclass(frozen=True) +class PreparedGraph: + batch: dict[str, Any] + """The `batch` body for enqueue_batch.""" + graph_hash: str + model_keys: frozenset[str] + """Keys of every model identifier in the graph (main model, components, LoRAs, control models).""" + components: dict[str, str] + """The exporter's resolved component choice, for provenance.""" + + +def build_export_request( + cases: list[ResolvedCase], + models: list[dict[str, Any]], + capabilities: Any, + image_names: dict[str, dict[tuple[str, bool], str]], +) -> dict[str, Any]: + """`image_names` maps each case key's (manifest input path, used at the case size) to the uploaded image name.""" + request_cases = [] + for case in cases: + spec = case.spec + names = image_names[case.case_key] + request_cases.append( + { + "id": case.case_key, + "mode": spec.mode, + "model_key": spec.model, + "components": spec.components, + "width": case.width, + "height": case.height, + "steps": spec.steps, + "seed": case.seed, + "positive_prompt": case.prompt, + "negative_prompt": case.negative_prompt, + "loras": [{"key": lora.key, "weight": lora.weight} for lora in spec.loras], + "reference_images": [names[(path, False)] for path in spec.reference_images], + "init_image": ( + None + if spec.init_image is None + else {"image_name": names[(spec.init_image.path, True)], "strength": spec.init_image.strength} + ), + "control": [ + { + "model_key": c.model, + "image_name": names[(c.image, True)], + "weight": c.weight, + "begin": c.begin, + "end": c.end, + } + for c in spec.control + ], + "settings": spec.settings, + } + ) + return {"version": EXPORT_VERSION, "models": models, "capabilities": capabilities, "cases": request_cases} + + +def run_pnpm_exporter(request: dict[str, Any], work_dir: Path) -> dict[str, Any]: + """Runs `pnpm graphs:export` in the tool checkout's webv2 package. Per-case errors stay in the response.""" + pnpm = shutil.which("pnpm") + if pnpm is None: + raise ExportError("pnpm is not on PATH; the graph exporter runs with pnpm from invokeai/frontend/webv2") + work_dir.mkdir(parents=True, exist_ok=True) + # Absolute: pnpm runs the exporter from the webv2 package directory. + request_path = (work_dir / "export-request.json").resolve() + response_path = (work_dir / "export-response.json").resolve() + request_path.write_text(json.dumps(request), encoding="utf-8") + response_path.unlink(missing_ok=True) + # pnpm is a shim (pnpm.cmd on Windows) around node: a timeout must kill the whole tree, not just the shim. + process, job = spawn_in_job( + [pnpm, "graphs:export", str(request_path), str(response_path)], + cwd=WEBV2_DIR, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + stdout, stderr = process.communicate(timeout=EXPORT_TIMEOUT_S) + except subprocess.TimeoutExpired: + raise ExportError(f"pnpm graphs:export did not finish within {EXPORT_TIMEOUT_S}s") from None + finally: + terminate(process, job) + if process.returncode != 0 or not response_path.is_file(): + output = (stderr or stdout).decode("utf-8", errors="replace") + raise ExportError(f"pnpm graphs:export exited with {process.returncode}:\n{output[-4000:]}") + response = json.loads(response_path.read_text(encoding="utf-8")) + if response.get("version") != EXPORT_VERSION: + raise ExportError(f"exporter answered version {response.get('version')!r}, expected {EXPORT_VERSION}") + return response + + +def _model_keys(value: Any) -> Iterator[str]: + # A ModelIdentifierField: {"key", "hash", "name", "base", "type", ...}. + if isinstance(value, dict): + if isinstance(value.get("key"), str) and "base" in value and "type" in value: + yield value["key"] + for item in value.values(): + yield from _model_keys(item) + elif isinstance(value, list): + for item in value: + yield from _model_keys(item) + + +def _replace_strings(value: Any, replacements: dict[str, str]) -> Any: + if isinstance(value, str): + return replacements.get(value, value) + if isinstance(value, dict): + return {k: _replace_strings(v, replacements) for k, v in value.items()} + if isinstance(value, list): + return [_replace_strings(v, replacements) for v in value] + return value + + +def prepare_batch(batch: dict[str, Any], components: dict[str, str], input_ids: dict[str, str]) -> PreparedGraph: + """Applies the runner policy to an exported batch and hashes it. + + `input_ids` maps uploaded image names to a stable identity of the input file. Upload names differ on every + server start, so the hash covers the input's identity instead. + """ + batch = copy.deepcopy(batch) + for node in batch["graph"]["nodes"].values(): + node["use_cache"] = False + batch["runs"] = 1 + hashed = _replace_strings({"graph": batch["graph"], "data": batch.get("data")}, input_ids) + canonical = json.dumps(hashed, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + return PreparedGraph( + batch=batch, + graph_hash=hashlib.sha256(canonical.encode("utf-8")).hexdigest(), + model_keys=frozenset(_model_keys(batch["graph"])), + components=components, + ) + + +def output_image_name(graph: dict[str, Any], session: dict[str, Any]) -> str: + """The image of the graph's one non-intermediate node that produced an image.""" + mapping = session.get("source_prepared_mapping") or {} + results = session.get("results") or {} + names = [] + for node_id, node in graph["nodes"].items(): + # The server's default: a node without the field is not intermediate. + if node.get("is_intermediate", False): + continue + for prepared_id in mapping.get(node_id, []): + image = (results.get(prepared_id) or {}).get("image") + if isinstance(image, dict) and isinstance(image.get("image_name"), str): + names.append(image["image_name"]) + if len(names) != 1: + raise ValueError(f"expected one output image from the graph's non-intermediate nodes, found {len(names)}") + return names[0] diff --git a/scripts/family_e2e/manifest.example.yaml b/scripts/family_e2e/manifest.example.yaml new file mode 100644 index 00000000000..26351a9b6b4 --- /dev/null +++ b/scripts/family_e2e/manifest.example.yaml @@ -0,0 +1,80 @@ +# Family E2E manifest, example for the first wave. Copy it next to your data directory, replace every <...> (paths +# and the model keys of your install; GET /api/v2/models/ lists the keys), then: +# python -m scripts.family_e2e start --manifest +# Relative paths are relative to this file. + +profile: rtx4090 # names runs and references; one profile per machine and GPU +data_dir: # inputs/, references/, runs/, scratch-roots/ +gpu: RTX 4090 # substring of the adapter name; must match one adapter ("Radeon" also matches an AMD iGPU, use "9060") + +server: + # Its invokeai.yaml and a database snapshot are copied; its server must be off. The scratch server shares its + # models directory: it writes models/README.txt and removes install claims of processes that no longer run there. + template_root: