diff --git a/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx b/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx index 408b8ce1b1e..5f8fbdc6ad5 100644 --- a/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx +++ b/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx @@ -172,6 +172,19 @@ device_working_mem_gb: 4 Once decoding completes, the model manager "reclaims" the extra VRAM allocated as working memory for future model loading operations. ::: +#### Calibrating a model's working memory + +Invoke's decode estimates are measured on a few reference GPUs. Your GPU, driver and settings can need more or less. A calibration measures what a model's VAE decode really takes on your GPU, and Invoke uses those measurements from then on. + +- **What it runs**: a few short text-to-image generations of the model at three sizes, up to the largest your GPU can decode in one piece with the VAE fully loaded in the memory free on it when the calibration is planned, in each VAE precision the model can decode in. Before each measured operation, Invoke moves the other models off the GPU, so the measured model is fully loaded. Each size runs once to warm up and twice more. The generations go through the normal queue, so they wait behind queued work and appear there like any other generation. +- **Who and where**: administrators only. It needs exactly one GPU set as the generation device; with several, it is not available. +- **What it changes**: decodes up to the largest size measured use the measured values, plus the same 5% margin the built-in values carry. Measurements that a fragmented GPU memory pool inflated, or taken while the VAE was only partly on the GPU, are left out, and the Model Manager shows those sizes as not counted. Larger decodes use whichever of the measured and the built-in estimate is larger. Other models that decode the same way (for example, SD1.5 and SDXL models when you calibrate an SDXL model) use the measurements too. Because the estimate decides when a large decode is split into tiles up front, a calibration can change which large images are decoded in tiles. +- **Where it applies**: on this kind of GPU only (identical cards share it), and only while the software that affects memory stays the same: the operating system, the PyTorch and GPU runtime versions, the memory-related environment variables and Invoke's memory settings. After such a change Invoke goes back to the built-in values until you calibrate again. A driver update keeps the measurements but marks them as possibly out of date until you calibrate again. +- **What it does not cover**: image-to-image and inpainting encodes keep the built-in values. +- **Reset**: remove a model's calibration from the Model Manager, or every calibration of this GPU; the change applies right away. Resetting one model removes only its own measurements: where other models' measurements cover the same decode, those still apply, and the Model Manager shows which models measured it. Otherwise the built-in values apply again. + +The measurements are stored next to the database in `working_memory/.json`. + ### Keeping a RAM weight copy Invoke has the option of keeping a RAM copy of all model weights, even when they are loaded onto the GPU. This optimization is _on_ by default, and enables faster model switching and LoRA patching. Disabling this feature will reduce the average RAM load while running Invoke (peak RAM likely won't change), at the cost of slower model switching and LoRA patching. If you have limited RAM, you can disable this optimization: diff --git a/docs/src/content/docs/development/Architecture/model-manager.mdx b/docs/src/content/docs/development/Architecture/model-manager.mdx index dd3a5680b95..43681d84e8f 100644 --- a/docs/src/content/docs/development/Architecture/model-manager.mdx +++ b/docs/src/content/docs/development/Architecture/model-manager.mdx @@ -1015,10 +1015,108 @@ An estimator that declares an `EstimatorSpec` returns a `WorkingMemoryEstimate` To write such an estimator (`invokeai/backend/util/vae_working_memory.py` has five): - Register an `EstimatorSpec` at module import, and list the module in `ESTIMATOR_MODULES` (`working_memory/spec.py`); the data test fails on a module that declares a spec without being listed. -- Split the estimator in two. The impure part reads the model and the device and records every runtime decision (tile, element size, score-matrix bytes) as a feature. The pure `terms(features, values)` turns features and coefficient values into terms, building fitted terms with `WorkingMemoryTerm.fitted(...)`. The public function returns `SPEC.estimate(features, device)`. +- Split the estimator in two. The impure part reads the model and the device and records every runtime decision (tile, the dtype the operation runs in, score-matrix bytes) as a feature, and refuses a dtype the spec has no line for with a `ValueError` naming the estimator, dtype and platform. The pure `terms(features, values)` turns features and coefficient values into terms, building fitted terms with `WorkingMemoryTerm.fitted(...)`. The public function returns `SPEC.estimate(features, device)`. +- Record the dtype the operation will run in, not the one the cached model holds. The SD1/SDXL nodes cast the shared VAE with `sd_vae_run_dtype`: fp16 for every GPU run that does not ask for fp32 (bf16 never runs), fp32 when asked and for a CPU decode. Their estimate prices the same function's answer, plus what the in-place cast adds to the resident weights. Uncast VAEs compute in their first floating-point parameter wider than a byte, so fp8 or packed SDNQ storage does not count. - Bump `version` whenever `terms` changes the formula. -- Ship one coefficient entry per measured platform value in `invokeai/backend/working_memory/data/`, either with its own `values` or with `values_from` another estimator's entry. Unmeasured platform values (MPS, XPU, CPU) are priced as `UNMEASURED_PRICED_AS` in `working_memory/platform.py` says. -- Ship the measured points next to them: peak reserved bytes (`reserved_peak`), one fresh process per point. `tests/backend/working_memory/` holds each estimate to at least its point, and where a fitted term decides it, to at most 1.3 times the point, or the point's tighter `max_ratio`. The `device_working_mem_gb` floor counts only where an analytic term decides. Every value an entry carries must be decided by at least one point. +- Ship one coefficient entry per measured platform in `invokeai/backend/working_memory/data/`, either with its own `values` or with `values_from` another estimator's entry, which may carry more values than the borrower reads. + - A platform is the convolution backend (`cudnn`, `miopen`, `mps`, `xpu`, `cpu`), torch's caching allocator and the OS. The allocator is torch's backend first (`cudaMallocAsync` ignores expandable segments), then `expandable` if the allocator environment variables enable expandable segments on the native allocator, else `native`; anything but a CUDA/HIP GPU is `native`. `platform_for(device)` reads them from this process; `platform_for_environment(backend, hard)` reads the same facts from a measurement's recorded environment, through the same function. + - `MEASURED` in `working_memory/platform.py` lists the measured backend, allocator and OS combinations (an OS of `None` for every OS). `priced_as` prices every other pair as one of them, keeping the OS: a backend nothing is measured on as cuDNN with the native allocator, an allocator not measured with its backend as that backend's native allocator. There the entry that carries the values raises them to its `unmeasured_floor`, the pre-dtype slopes. + - An entry applies to the OS only when its platform names one, and then takes precedence there over the one that does not. Only MIOpen names one. With the native allocator, Windows demand is concave and up to 1.6x Linux's, so Windows has its own fit and every other OS keeps the Linux one. Expandable segments are measured on Windows only, where they are Invoke's ROCm default, so `MEASURED` lists them for Windows and every other OS prices them as the native allocator there. cuDNN is priced the same on every OS. +- Ship the measured points next to them. A point's `method` says how it was read: one fresh process per point for `reserved_peak` (the allocator's peak reserved bytes), `driver_peak` (the driver-level device usage), and `demand_peak` (the largest of driver-level, reserved and allocated peak, over two runs, which catches the memory HIP maps beyond the allocator on a process's first operation); or `probe_demand`, the probe's `demand_bytes`, the largest over the valid, fit-eligible windows of calibration runs in a running server. `scripts/working_memory/build_shipped.py` builds `probe_demand` points from calibration store exports and refits the values (see the Family E2E guide). A point names the OS only where it measured an OS-specific entry. `working_memory/bounds.py` holds the rule the data test (`tests/backend/working_memory/`) and the builder apply: each estimate is at least its point, and where a fitted term decides an in-fit point, at most 1.3 times the point, its ratio in `bounds.UPPER_BOUND_EXCEPTIONS` (points no line can hold within 1.3x), or the point's tighter `max_ratio`. A `probe_demand` point is a lower bound only (`MeasuredPoint.holds_upper_bound`) and takes no `max_ratio`: a warm window in a running server reads below the cold first run after a server start, the worst case a reservation must cover. It may raise a line; only the fresh-process points cap one. The `device_working_mem_gb` floor counts only where an analytic term decides. +- Fit the values, never hand-tune them. The AutoencoderKL estimators price their convolutions as a line per operation and dtype: `k__` bytes per output pixel plus a base of `b__` bytes. + - `working_memory/fit.py` fits each line on the untiled single-sample in-fit points of every estimator that resolves to the entry, leaving out points whose VAE an in-place cast grew (`upcast_weight_bytes`, a term of its own). The line is the edge of the points' upper convex hull (with the origin, so the base is never negative) that over-reserves least, rated on the highest reading per size of the points that hold an upper bound, the steeper on a tie and never a falling one. A `probe_demand` point is covered by the line and can raise it, but rates no edge. It needs at least two sizes twice apart in pixels. The shipped values add a 5% margin and round up: the slope to a whole byte per pixel, the base to a whole MiB. + - A point measured in a regime its platform does not run sets `in_fit: false`: it fits nothing and gets only the lower bound (CUDA with the math attention kernel forced). The test lists every such point. + - A line no in-fit point measures is a declared stand-in, listed in the entry's `stand_ins` and never below the pre-dtype slope. MIOpen with the native allocator off Windows prices fp16 and fp32 at the pre-dtype figure, and CogView4's lines there at its expandable-segments ones. + - Every value an entry carries is a stand-in or decided by an in-fit point. The data test fails when a fitted value is not the fit of its points, naming the value to ship, and when an in-fit point measures a stand-in's line. + +#### Measuring estimates: the probe + +`invokeai/backend/working_memory/probe.py` measures what an estimated operation really takes. It is active only inside a `calibration_scope(...)`, which the session runner opens per node of a calibration queue item; without one, `model_on_device()` behaves as above and reads no counter. + +Inside a scope, every `model_on_device(working_memory=...)` on a GPU is a window. The lock reserves the estimate plus calibration headroom (half the estimate, at least 1 GiB), so a low estimate cannot spill. The window runs from the settled lock to the unlock and collects a `MeasuredWindow` in the scope's `windows`; nothing is written while it is open. + +- Counters (`working_memory/counters.py`): torch's reserved and allocated peaks, and this process's usage as the driver counts it. That is D3DKMT on Windows for CUDA and ROCm (the only readers verified on hardware), NVML on Linux CUDA, DRM fdinfo on Linux ROCm, and the MPS driver figure. XPU's Sysman figure covers the whole device, so it only detects another process's load. MPS has no peaks; a sampler thread keeps them while a window is open. +- Demand: per counter, the largest segment peak less the baseline and the bytes co-locks moved onto the device; `demand_bytes` is the largest of those deltas. Each counter misses something: the driver misses HIP's retained pool, and reserved misses cuBLAS and cuDNN workspaces and, under `cudaMallocAsync`, part of the allocated peak. +- Segments: the window opens after a release, so cached blocks and tensors from before the lock form the baseline. Another model locked on the same thread inside the window is a co-lock: its weights go into the offset, and its lock interval is kept out of the peak. An estimated window inside another measures alone while the outer one is frozen. A `TorchDevice.empty_cache()` ends a segment. A direct `torch.cuda.empty_cache()` call (from a library, or the one cuDNN and MIOpen run after an algorithm search) and an offload by the cache's background thread do not: the driver figure can miss what they released, while the reserved and allocated peaks still bound the demand. +- Invalid (`valid=False`, with `reasons`): another thread locked a model on the same device's cache during the window, held one when it opened, or was still locking one then (`foreign_lock`); the allocator failed or retried; Windows still holds more than 256 MiB of the process in shared memory after a short settle; the operation raised or was canceled, or its node failed or was canceled after the window closed; the Krea-2 SDPA benchmark is active; XPU's device-wide usage grew beyond this process's own (`foreign_device_load`); a probe step failed (`probe_error`: a reading, a release, a peak reset or a synchronization, which also voids the scope's later windows when it hit the lock bookkeeping; a window that cannot open is recorded invalid, and its operation runs unmeasured); or no counter could be read. +- Not fit-eligible: a model that was not fully resident when the window opened. The measurement stays valid, but streamed weights are working memory the estimate does not price. + +`working_memory/fingerprint.py` keys measurements. `device_class_for()` gives the same id to identical cards. `environment_for()` splits hard facts, which change an operation's memory and make up its id (the operating system as `windows`, `linux` or `other`, torch, runtime and convolution-library versions, allocator backend and configuration, the ROCm SDPA guard, the memory settings passed in), from soft facts kept for diagnosis (full versions, driver, OS build). + +#### Calibration runs + +A calibration run measures one model's operations on this machine's GPU; estimates read the result as user fits (below). The service is `invokeai/app/services/working_memory_calibration/`, the routes are in `invokeai/app/api/routers/working_memory.py`, and every route is admin-only (403 otherwise). + +1. `POST /api/v2/models/i/{key}/working_memory/calibrate` takes a client-built graph, batch data, an optional `idempotency_key` and an optional `attempt_id`. The server turns off `use_cache` on every node, sets `runs=1`, sets the graph's id to the run id, and sets the origin to `invokeai:wm-calibration:` and the destination to `invokeai:wm-calibration`. It records the run, enqueues the batch as the requesting admin, then records the run's items. + - An `attempt_id` (1–64 of letters, digits, `_ - . :`) groups the runs a client enqueues for one calibration attempt; it is stored on the run (and left out of the file when absent, so older readers keep reading such runs) and returned in the run summaries of the status and the model summary. + - The queue takes all of the batch's items or none (`enqueue_batch(..., all_or_none=True)`, checked under its admission lock): a run with only some of its cases would measure only some of them. Without room for all of them nothing is enqueued, the key stays unused, and the run is removed if this request recorded it. + - With a key, the run id is derived from the owner, the key, the request's fingerprint (the model key, the graph without its id, and the batch data) and the `attempt_id` if given. A client that sends no graph id would otherwise get a fresh one each time and could never repeat a request. A repeated request finds its run and gets the same response; nothing new is enqueued, even after the queue collected the key's receipt. A key therefore names one run. Once a reset removed the run, a repeat is refused (404) whether or not the queue still keeps the receipt: the store keeps the ids of removed runs, so the run is never started again and its queued items keep failing at admission. Another request that reuses the key (another model, graph, batch data or `attempt_id`) gets another run id and so another origin. While the key's receipt exists, the queue refuses that request (409) and the first run stays. Once the queue has collected the acknowledged receipt, the request starts a run of its own. Clients make each key unique per run of an attempt, for example from the attempt id and the step. Acknowledge the key through `POST /api/v1/queue/{queue_id}/enqueue_batch/acknowledge`. + - Once its items are queued, the request answers with them, also when the enqueue raised after its commit (the queue announces a batch after committing it). When the queue cannot tell whether it holds them, the run is kept and the enqueue's own error answers. + - If the store fails for a passing reason (an unwritable root, a lock timeout) to record the queued items, a request without a key still succeeds: its items run, and the status finds them by their origin. With a key that is a 503, and a repeat records them: from the key's receipt while the queue keeps it, otherwise from the queue items with the run's origin and the items the runner saw. It never enqueues a second batch for the run. Recording them from the receipt checks neither the model nor the device, so a model removed meanwhile does not fail the repeat; a request that has to enqueue needs the model. A run removed meanwhile (404) or an unusable store file (409) fails the request with or without a key; its queued items then fail at admission. + - Per run id, one creation runs at a time, and it runs to its end even when its request is canceled: the enqueue runs in a worker thread and can commit after the cancellation. Only the server's shutdown cancels the creation with it; the run then stays as far as it got, and with a key a repeat of the request completes it. The lock belongs to the process, so two processes serving one root and its SQLite database can interleave the creations of one run. A run is removed only when this request recorded it and nothing of it was queued. + - The destination only stops clients from routing the results to a gallery or canvas view. Images the graph's nodes save are stored like those of any other session: on the board and with the intermediate flag each node sets. +2. `enqueue_batch` refuses any origin that starts with that prefix, in any case, for every user (400). Only the calibration route creates calibration items. Retries and workflow-call children inherit origin and owner from their source item. +3. `DefaultSessionRunner.run()` checks each item's origin once. For a calibration item it calls `admit()` before any node runs, also for one left pending before a restart: the invoker starts the session processor after every other service. (The queue's start cancels the items that were in progress.) Retries and children are checked here too. Otherwise the item fails with the reason. The checks: + - the owner is an active admin (single-user mode counts as admin); + - the machine has exactly one GPU generation device; + - the run exists for that GPU's device class. +4. Each admitted node runs in `probe.calibration_scope(...)`, keyed by the node's source id. Its windows are persisted after the node completes, fails or is canceled. Unless the node completed, every one of them is persisted invalid (`exception` when it failed, `canceled` otherwise), including a window whose own operation finished before the node raised: `run_node` swallows the failure or cancellation, so the runner takes the outcome from its node hooks. Windows from a device of another class, or of a run that was removed, are dropped with a warning. So are windows of a model reset after the node started, also by another process sharing the root: the runner reads the device class's reset epochs before the node's scope opens, and the write drops every point attributed to a model whose epoch has moved since, as the reset attributes them (`measured_models`: a submodel's `:vae` window counts for its model). Without that, a reset while the node runs (of a VAE inside a transformer's run, say) would be undone by the node's later write. If they cannot be saved, the item fails and the session stops. Every other item gets no scope and runs as before. +5. After the item, the runner records the item's outcome in the run, even when the session raised. An item the queue no longer has (Clear Queue) is recorded as canceled. An item that was still running when the runner raised is recorded as failed with the error. + +The store has one JSON file per device class at `/working_memory/.json` (`CalibrationStoreFile`, `schema_version` 1). It is machine data, so it lives next to the database rather than inside it. It holds: + +- the device class, with its id; +- the runs: model key, model hash as provenance, owner, environment and its soft facts at creation, queue items, and the last status the runner saw for each item; the soft facts and the `attempt_id` are left out when absent, so runs stored without them still load and older readers still read runs that lack them; +- the points: `MeasuredWindow` plus `env_id`; +- the environments those reference; +- the ids of the runs with a retry key that a reset removed (`removed_run_ids`), so that a repeat of the request that started one cannot start it again. They are kept for good, about 40 bytes per removed run. A run without a key gets a fresh id and is never repeated, so its id is not kept; +- per model key, its reset epoch (`model_reset_epochs`): how often the model was reset. The reset raises it in the same file change, so it is only written when the reset is. + +A file without either of these two fields reads them as empty. + +Writes and reads: + +- Every change takes a process lock and an OS lock on the file's `.lock` sibling, re-reads the file, applies the change, and replaces the file atomically. Two processes sharing a root therefore never lose each other's writes. +- On Windows a process that ignores the lock (an antivirus scanner, a search indexer, an editor) can hold the file open and refuse the replace. The write then retries a few times with backoff, about 1.5 s in all (`REPLACE_ATTEMPTS`, `REPLACE_RETRY_S`). +- Readers take the lock as well, because Windows cannot replace a file another process has open. +- Each change rewrites the whole file, which is fine at calibration volumes. + +A file is unusable when it has another schema version, invalid content, an id that does not match its fields or its file name, or a field this version does not know. Every store model forbids unknown fields: ignoring one would drop it at the next rewrite. An unusable file is reported in the status, makes calibration unavailable for its device class, and is never written, moved or deleted. + +A run's status is derived, never stored: + +- The live queue items with the run's origin come first. For items the queue has pruned, the runner's recorded statuses are used. An item the queue no longer has and that the runner never saw finish (the process died mid-item) counts as canceled. +- A retry replaces the item it retried. The queue's own record of the retry links them while it has the retry, also before the runner started it; afterwards, what the runner saw does. A retry the runner never started and the queue no longer has leaves no trace: the run then reads as the outcome of the item it retried, failed for instance. +- The run is `running` while any item is pending or running, `failed` when any failed, and `completed` when all completed. Otherwise it is `partial` if valid points were recorded, or `canceled` if not: an invalid point measured nothing usable. + +Routes, besides the 403 for non-admins and a 503 when the store cannot be read or written (an unwritable root, a lock timeout): + +| Route | Does | Other errors | +| --- | --- | --- | +| `POST .../i/{key}/working_memory/calibrate` | starts a run (above) | 404 unknown model, the run was removed while it was being created, or a reset removed the run the retry key started; 409 no GPU or several generation devices, unusable store file, no room in the queue for all of the run's items (nothing is enqueued; the run is removed if this request recorded it), or a retry key used for another request; the 404 and 409 for the run's store, and with a key the 503, can come after the items were queued (above); 422 a `call_saved_workflow` node, batch data setting `use_cache`, an invalid batch, or fields the server sets; 429 too many unacknowledged retry keys | +| `GET .../working_memory/status` | every device class with its runs, point counts per estimator, version, validity and fit eligibility, and environments with hard and soft facts compared to the current environment (`soft_matches_current` compares the soft facts first recorded with the environment, not a later run's); `available` says whether a run can start now (one GPU generation device and a usable store file); per device class with a generation device here, `measurement_method` (the driver reader its windows use, `counters.driver_method`) and `measurement_verified` (`counters.VERIFIED_METHODS`), null for other classes | – | +| `DELETE .../working_memory/runs/{run_id}` | removes a run and its points; its items that have not started yet fail at admission, also after a repeat of the request that started it | 404 no usable file holds the run | +| `DELETE .../i/{key}/working_memory` | removes the points the summary attributes to the model (`measured_models`): its runs with all their points, and every point of the model or one of its submodels (`:vae`) taken in other runs (a VAE measured inside a transformer's run, say); other models' points on the same lines stay | 409 no current device class, or an unusable file | +| `DELETE .../working_memory` | removes every run, point and environment of a device class | 409 as above | +| `GET .../working_memory/export` | the file byte for byte: a `CalibrationStoreFile` when this version can read it, otherwise whatever the file holds | 404 no file; 409 no current device class | +| `GET .../i/{key}/working_memory/plan` | the cases that calibrate the model here (`WorkingMemoryCalibrationPlan`, below), or `supported: false` with the reason | 404 unknown model | +| `GET .../i/{key}/working_memory/summary` | per line and environment: the lines the model's windows measured, and in the current environment each line the model's decode prices on (its plan's estimator and dtypes) that other models' windows measured. Per line, every window that feeds it (all models' counted windows of that environment that join the same user line): sizes, largest demand, shipped and user estimate; a size whose valid windows all are excluded is listed too, with `counted: false` and `excluded_reason` (`fragmented` or `partial_residency`), its largest such window and the user estimate there; the user line's measured range and state (`in_use`, `stale`, `unusable`, `insufficient`); `own_windows`, the model's share (0 for a line only others measured); `measured_by`, each contributing model with its name (null once uninstalled) and window count. Also the model's windows that fit no line, and its runs | 404 unknown model; 409 no current device class | + +The reset, export and summary routes act on the current device class unless `device_class_id` is given. + +#### User fits + +`backend/working_memory/user_fit.py` owns the rule; the calibration service owns the store and the cache. + +- A user line is fitted per line of `fit.py`: the shipped entry that carries the values (one hop of `values_from`) and `_`. A window joins where `fit.fits_a_line` admits it (untiled, one sample, no in-place upcast, an AutoencoderKL: the SD estimator's AutoencoderTiny windows join no line), it is valid and fit-eligible, it is not fragmented, its demand is positive, and its estimator version has shipped values on the platform. Every user point is a `probe_demand` window, so all of them rate the hull's edges. `fit.user_pair` adds `USER_MARGIN`, the shipped 5%, to the envelope and rounds up as the shipped values do. A line needs two sizes `MIN_AREA_RATIO` apart. +- Fragmented windows measured the allocator's history, not the operation. One rule, `fragmentation.fragmented`, judges them for the user fit and for `scripts/working_memory/build_shipped.py`, which flags shipped points by it. It judges a measurement: the fit-eligible windows of one attempt (`attempt_id`, else the run; for the builder also one server lifetime) at one environment, estimator, version and features, in the order they ran. A measurement drops out whole when reserved grew by 10% from its first window to its largest while allocated stayed within 2%, or when its largest window's reserved-to-allocated ratio is 1.2 times the median of the same line's smaller measurements that count, from any attempt. Each attempt is judged on its own, because its windows ran back to back on one allocator state: a recalibration can bring back a size an earlier attempt measured fragmented, and a later fragmented attempt drops no clean one. The ratio rule catches a size whose windows all ran fragmented, or that kept only one fit-eligible window: on an RX 9060 XT the fp32 SDXL decode at 1216 px measured 1.47 against 1.10 at 576 and 832 px, with one fully resident window beside two partly resident ones. Windows of a running attempt are judged but set no reference, as they join no user line yet. Partly resident windows (`partial_residency`) never count and enter neither rule. A size counts when any attempt's windows at it count, and its `windows` and `measured_bytes` cover those windows only. +- The windows are those of the current device class's file taken under an environment whose hard facts equal the current ones (same `env_id`). A soft-fact difference keeps the fit. The summary reports a line as `stale`, with the facts that changed, when the soft facts of the newest finished run that fed it differ from the current ones; a run stored without soft facts stands for the environment's, those first recorded with it. So recalibrating after a driver update clears `stale`. Older windows of the same hard environment keep feeding the line: the hard facts are what decide whether a window still measures this machine. A hard mismatch leaves the shipped values. +- `EstimatorSpec.estimate` asks the registered provider (`user_fit.set_user_fit_provider`; the service registers it in `start()` and clears it in `stop()`) for the user fit of the spec, platform and device. Without one, the estimate is the shipped one, bit for bit. With one, `user_fit.resolve` reads the operation's line and output pixels as `fit.line_of` does. At or below the largest measured size the user line replaces the shipped one; above it, the larger of the two estimates wins. A tiled operation is placed by its image, priced on its tile. The `source` starts with `user::` and names the line where the user line decides. A line the user did not measure stays shipped. Because estimates feed `reservation_bytes`, a user line also moves `should_pretile_vae_decode`. +- Only finished runs and attempts feed a fit. The windows of a run whose derived status is `running` (an item pending or running; a run records its items as soon as the calibrate call has queued them, so a run whose items all wait counts, and an item it could not record counts once the runner admitted it) join no user line, whoever refreshes the fits meanwhile. Nor do the windows of any run that shares its `attempt_id`: the client posts each step of an attempt as its own run, so an attempt's windows feed no fit until every run of it recorded so far has finished. A step not yet posted holds back nothing, so an ordinary item or a plan in that gap can see the finished steps' windows; a calibration item refreshes only once its own run exists. They join at the first refresh after that, whether the runs completed, failed, were canceled or ended `partial`. So an attempt's later, larger cases are priced without its own earlier windows. A run without an `attempt_id` is judged on its own; another attempt that finished counts as usual. The summary shows a running run's windows in the sizes, window counts and `measured_by`, while its user line, range, user estimates and state are those of the fit estimates read; a line only a running run measured is `insufficient` and says so. +- The provider answers from memory. `refresh_user_fits()` rebuilds that memory per GPU generation device. `DefaultSessionRunner.run()` calls it once when it starts any queue item, calibration items included; `get_plan` calls it first, so the case sizes price with the store as it is, and every reset calls it after its write, so a reset applies at once. It stats each device class's file and re-reads it only when the mtime, size or inode or the environment changed. To tell which runs are running it asks the queue, by id, only about the items the file does not record as finished; items found finished are remembered, since a finished item never runs again. When every run finished, a refresh reads neither the file nor the queue. A retry's item is not in the file until admission records it, and the runner refreshes before admitting, so a retry of an attempt's last open run is priced with that attempt's finished windows. Nothing on the estimate path reads a file. Several generation devices each use their own class's fit; calibrating them is not supported. +- Shared lines: on cuDNN the SD, SD3, FLUX.1 and FLUX.2 estimators borrow one entry, so an SDXL fp16 decode calibrates every estimator's fp16 decode there. The plan lists the installed models this affects. + +The plan (`working_memory_calibration_plan.py`) covers the decode of SD1, SD2, SDXL, SD3, FLUX.1, FLUX.2, Z-Image and CogView4 main models, and standalone VAEs of those bases through an installed main model that accepts them (`VaeFacet`): through Generate's `vae` slot where the base has one (FLUX.1, FLUX.2, SD3, Z-Image), otherwise with the main model's own VAE, which prices on the same lines. Each case is a two-step `txt2img` graph-export case per size and decode dtype; the SD bases set `vaePrecision` to fp16 and to fp32, the others run the dtype their loader picks (SD3 the `precision` setting). The sizes are about 0.25, 0.5 and 1.0 of the largest square, in steps of 64 and at most 4096 px, whose calibration reservation (the estimate plus `max(1 GiB, 0.5 × estimate)`, as the lock applies it) leaves the VAE fully resident (`decode_budget`): it must fit both 90% of the card (`VAE_PRETILE_VRAM_FRACTION`) and what the model cache can give on the device, each less 1 GiB for the VAE's weights. Nothing else stays beside it: inside a calibration scope, `model_on_device` first moves every other unlocked model on the device to RAM whole (`ModelCache.offload_unlocked_models_from_vram`, under the global load lock, inside the probe's lock interval; locked models, a window's co-locks among them, stay), and only then locks. The reason: a lock offloads only the shortfall it predicts and splits the last model it reaches, and a split can free a little less than predicted. On an RX 9060 XT, 9 of the 12 calibration decodes that had to split the SDXL UNet came up 35 to 55 MB short and ran the VAE partly resident, so their windows fit no line. Outside a calibration scope, locks are unchanged. That capacity is read when the plan is built, as `ModelCache._get_vram_available` counts a lock: the driver's free memory plus what torch has allocated, within `max_cache_vram_gb`, less `reserve_vram_gb`. Memory other processes hold is thereby left out. What this process holds outside torch's allocator counts at the larger of what it holds now and the most a valid stored window of the current environment recorded at its baseline (the driver's figure for the process less torch's reserve, `recorded_outside_torch_bytes`), because it grows with the first generations: on a 16 GB RX 9060 XT under ROCm, HIP's retained pool and MIOpen's workspaces took about 0.3 GB in a fresh server and 2.6 GiB once it had decoded, and sized for the fresh reading its largest cases ran partly resident. A first calibration has no stored window and sizes for the reading alone, so on such a card its largest cases may run partly resident; those windows do not count, and the next plan sizes for what the first one recorded. The recorded figure comes from the store `refresh_user_fits` just read; `estimate_untiled_vae_decode` prices them with the decode nodes' own features. `DECODE_LINES` restates which estimator each base's decode node prices with; `test_working_memory_calibration_plan.py` holds it to the nodes in `invocations/vae/`. Each case runs once to warm up and twice more; `estimated_seconds` is a rough order of magnitude, not a measurement. Encodes are not planned: text-to-image runs none. ### Emitting model loading events diff --git a/docs/src/content/docs/development/Architecture/overview.mdx b/docs/src/content/docs/development/Architecture/overview.mdx index 8cbe18efad5..3d6272db7ff 100644 --- a/docs/src/content/docs/development/Architecture/overview.mdx +++ b/docs/src/content/docs/development/Architecture/overview.mdx @@ -85,6 +85,8 @@ The invoker (`/invokeai/app/services/invoker.py`) is the primary interface throu - **invocation services**, which are used by invocations to interact with core functionality. - **invoker services**, which are used by the invoker to manage sessions and manage the invocation queue. +Creating the invoker starts every service, the session processor last: its workers dequeue at once, and an item left pending before the restart may use any other service. (The queue's start cancels the items that were in progress.) `stop()` stops the session processor first: its workers stop taking items and cancel their sessions, and the other services stop only once the workers have exited. The processor waits up to 30 seconds for all its workers together, and after a second still waiting it logs which worker and queue item it waits for. A worker still in a node at that deadline (one without a cancel point) is given up on with an error in the log, so that shutdown finishes; a later `stop()` does not wait for it again. It keeps its GPU, so until it exits no session processor in the same process starts and none replaces the generation device pool: `ApiDependencies.initialize` refuses a restart before it opens the database. + ### Sessions Invocations and links between them form a graph, which is maintained in a session. Sessions can be queued for invocation, which will execute their graph (either the next ready invocation, or all invocations). Sessions also maintain execution history for the graph (including storage of any outputs). An invocation may be added to a session at any time, and there is capability to add and entire graph at once, as well as to automatically link new invocations to previous invocations. Invocations can not be deleted or modified once added. diff --git a/docs/src/content/docs/development/Guides/family-e2e.mdx b/docs/src/content/docs/development/Guides/family-e2e.mdx new file mode 100644 index 00000000000..f700a17661c --- /dev/null +++ b/docs/src/content/docs/development/Guides/family-e2e.mdx @@ -0,0 +1,149 @@ +--- +title: Family E2E Runs +lastUpdated: 2026-10-17 +--- + +`scripts/family_e2e` runs model families end to end through the HTTP API of a server it launches itself. Per case and size it records the image, the run time, the time per denoise step and the VRAM the server process used, measured at driver level. The report compares them with an explicitly accepted reference and with the previous run of the same profile. + +It is a developer tool. Memory sampling uses WDDM, so it runs on Windows only for now, with NVIDIA and AMD (ROCm) GPUs. + +## How a run works + +1. **Scratch root.** The template root's `invokeai.yaml` and a snapshot of its SQLite database go to `/scratch-roots/-default/`. The template's database files (with a WAL left by an unclean stop) are copied, and the copy is backed up through InvokeAI's database layer, so the WAL's contents are included and the template's files are only read. + - The tool refuses while the database file is open in any process, or while anything listens on the template's configured port. Either means its server is still running. + - It also refuses while a model install is unfinished (see [Shared paths](#shared-paths)). + - Stores whose files the database owns start empty in the scratch root: outputs, uploaded fonts, style preset images, workflow thumbnails and profiles. The scratch server reconciles them with its own database; at the template it would, for example, delete uploaded fonts its database does not list. + - Every other path setting points back at the template root as an absolute path, so models, custom nodes and memory settings are the template's own. + - The tool overrides a few settings: a free port on `127.0.0.1`, a plain-format log file in the run directory, the image index off, and the queue cleared at startup. SSL and `base_url` are dropped. + - `INVOKEAI_*` variables of the calling environment are not passed on. Put settings the server needs into the manifest's `server.env`. +2. **Server.** The tool checks that `server.python` imports `invokeai` from `server.code_dir`. It then starts ` /scripts/invokeai-web.py --root ` and reads the actual port from the server log. Multiuser mode is refused. + - On Windows the server runs in a job object of the runner, so it dies with the runner however the runner ends. +3. **Graphs.** The real webv2 graph builders produce each case's batch through `pnpm graphs:export`, run in the tool checkout. Run `pnpm install` in `invokeai/frontend/webv2` there first. + - The runner disables the invocation cache on every node and saves the batch as `graphs/.json`. + - It hashes the graph and its batch data. Uploaded image names are replaced by the input file's hash, so the graph hash changes only when the graph or an input changes. + - Init and control images enter the canvas compiler as finished layers. One whose size differs from the case size is resized (LANCZOS) into `runs//inputs/` before upload. Reference images are uploaded as they are. +4. **Steps.** Per family the model cache is emptied first. Each case size then gets its warm-ups, then its measured runs. + - Per run, the tool waits up to 10 minutes for the queue to drain, samples the server's GPU memory, enqueues the batch and follows its socket events. + - When the run finishes, it saves the output image and the memory trace, reads the run's slice of the server log and appends a record to `records.jsonl`. + - If the server dies during a step, that step gets an error record and a new server starts on a new scratch root. A case whose steps crashed the server twice is given up; its remaining steps are not run again, not even by `resume`. +5. **Report.** The server is stopped, the scratch root removed, and `report.html` and `summary.md` are written. The exit code is not 0 when a case fails or errors. + +## Data directory + +``` +/inputs/ input images the manifest refers to +/references/// image.png, reference.json (written by `accept` only) +/runs// run.json, records.jsonl, heartbeat.json, runner.log, server-default.log, + graphs/, images/, traces/.csv.gz, report.html, summary.md, + calibration/ (calibration runs only) +/scratch-roots/-default/ +``` + +`run_id` is `YYYYMMDD-HHMMSS-`, and `case_key` is `@x`. A trace holds the raw samples as `series,t,bytes`: series `dedicated` or `shared`, and `t` in seconds from the enqueue. + +One runner per GPU: a lock file per adapter lives in `%LOCALAPPDATA%\invokeai-family-e2e\locks\`, shared by every data directory on the machine. + +## Manifest + +A YAML file per machine and GPU, kept with the data rather than in the repository. Start from `scripts/family_e2e/manifest.example.yaml`. Relative paths are relative to the manifest file. + +| Field | Meaning | +| --- | --- | +| `profile` | Names runs and references. | +| `data_dir` | The data directory above. | +| `gpu` | Substring of the adapter name that matches exactly one adapter, e.g. `RTX 4090` or `9060`. `Radeon` alone also matches an AMD integrated GPU. | +| `server.template_root` | The root whose settings and database are copied. The scratch server shares its other paths; see [Shared paths](#shared-paths). | +| `server.python` | The interpreter that runs the server. | +| `server.code_dir` | The checkout the server runs. Defaults to the tool's checkout. | +| `server.env` | Extra server environment, e.g. `HIP_VISIBLE_DEVICES`. | +| `defaults` | `warmup` (at least 1), `repeats`, `seed`, `prompt`, `negative_prompt`, `psnr_min_db`, `time_tolerance` (relative), `peak_tolerance_mb`, `run_timeout_s`. | +| `families[].cases[]` | `id`, `model` (key), `mode` (`txt2img`, `img2img` or `controlnet`), `sizes` (a list of `[width, height]`), `steps`, optional `seed` and `prompt`, `components` (component slot to model key), `loras`, `reference_images`, `init_image` (`path`, `strength`), `control` (`model`, `image`, `weight`, `begin`, `end`), `settings` (generate settings), and per-case thresholds. | + +Input paths are relative to `/inputs` and must exist. `init_image` belongs to `img2img` cases only and `control` to `controlnet` cases only. The exporter rejects an unknown `settings` key, component slot or model key; that case then gets error records and the run goes on. Several `sizes` per case give a size sweep for working-memory calibration. + +## Commands + +Run the tool from the repository root with the checkout's environment, for example `uv run --no-sync python -m scripts.family_e2e …`. + +| Command | What it does | +| --- | --- | +| `start --manifest M [--family F]… [--case ID]… [--calibrate]` | Checks the GPU lock, the template's server, unfinished model installs and foreign GPU load. It then creates the run and starts its runner in the background, and prints the run id and paths. `--calibrate` makes it a [calibration run](#working-memory-calibration). | +| `status [RUN]` | Shows progress, the current step, the heartbeat age and whether the runner is alive, and for a calibration run the number of saved store exports. | +| `stop [RUN]` | Kills the runner and its server, saves a calibration run's store from its scratch roots, removes them and marks the run stopped. | +| `resume [RUN]` | Starts the runner of a stopped or crashed run again. | +| `report [RUN]` | Writes `report.html` and `summary.md` again. | +| `accept [RUN] [--case KEY]…` | Makes the run's measured results the references of its profile. Nothing else writes references. | +| `run --run-dir D` | The runner in the foreground: what `start` and `resume` launch. | + +`RUN` is a run directory, or a run id together with `--manifest M`. Without it, the profile's latest run is used. + +On Windows the background runner breaks away from the caller's job object, so a terminal or an agent harness that kills its job's processes does not end a long run. Where breakaway is refused, the runner is created through WMI. It then starts with the user's default environment instead of the caller's. + +A resumed run skips every step that already has a final record and keeps the measures of the earlier sessions. Its new server starts with a cold model cache, so each case size with measures left first repeats its last warm-up, once. The manifest, the graphs, and the server's commit and dirty flag must not have changed since the run started; further edits to an already dirty checkout cannot be detected. + +## Working-memory calibration + +`start --calibrate` measures what each model operation of the cases needs on this GPU. Like the rest of the tool it runs on Windows only. Every step compiles its graph as a normal run does, then submits it to `POST /api/v2/models/i/{model}/working_memory/calibrate` with the case's `model` key instead of `enqueue_batch`. The server runs it with its working-memory probe, which records a window per estimated model operation in its calibration store (see "Calibration runs" in the [model manager architecture](/development/architecture/model-manager/#calibration-runs)). Warm-ups and measures alike are submitted, and every window is kept. + +- **Before the steps.** The runner reads `GET /api/v2/models/working_memory/status`. With `available: false` (no GPU, several generation devices, an unusable store file) the run fails with the server's reason. The scratch server runs single-user, so the tool is an administrator. +- **Refusals.** A 404 or 422 answer to a calibrate request (an unknown model, a graph the run cannot measure) gives that step an error record, and the run goes on. Any other error response ends the run with the server's detail, after its store was saved; fix the cause and `resume`. +- **Records.** Each accepted step's record carries `calibration`: the calibration run id and the device class. The probe synchronizes and empties the allocator cache around every model operation, so time and VRAM are invalid with that reason; `status` shows it as a calibration marker rather than counting it, and images are still compared. A step foreign load spoiled is not repeated: the server keeps every window. `accept` refuses calibration runs, and reports of later runs do not compare against them. +- **Exports.** Every server lifetime has its own scratch database and store, so the tool submits each attempt once and sends no retry key. When a lifetime's steps are done, the runner saves `calibration/--.json` (`GET /api/v2/models/working_memory/export`, byte for byte) and `calibration/--status.json`; `n` counts the run's lifetimes. After a server crash, a runner failure or `stop`, the store file is copied from the scratch root instead, without a status. `run.json` lists them all under `calibration_exports`. + +On a machine the tool does not run on, an administrator takes the export of a server's own calibration runs from `GET /api/v2/models/working_memory/export?device_class_id=`; `GET /api/v2/models/working_memory/status` lists the device class ids. + +### Building shipped data + +``` +uv run --no-sync python -m scripts.working_memory.build_shipped /runs//calibration [more exports or directories] [--write] +``` + +`scripts/working_memory/build_shipped.py` reads the exports and prints what it would change in `invokeai/backend/working_memory/data/`; `--write` writes it. A directory stands for the exports in it, in lifetime order. Then run `uv run --no-sync pytest tests/backend/working_memory`. + +- **Points.** Every valid, fit-eligible window with a positive demand becomes a `probe_demand` point. Per card, estimator, version, platform and features it takes the largest `demand_bytes` of all windows given: repetitions, lifetimes, and several runs or machines with the same card. A window given twice (the same store in two files) counts once. The provenance names the card, the torch and runtime versions, the calibration run, item and node of the largest window, and its export file. Each point line shows its windows' demands in the order they ran and the largest over the first (`[3 windows, MiB 1960, 1820, 1820; max/first 1.00]`). +- **Fragmentation.** A point whose reserved bytes grew by 10% or more from its first window to its largest while the allocated bytes stayed within 2% gets a `warning: fragmentation?` line with both series. That is the allocator's cache growing around the same tensors in a long-running server, not the operation needing more. +- **Platforms.** A point's platform is the one `platform_for` gives on the machine that measured it, derived from the export's device class and its environment's hard facts (`platform_for_environment`), whatever machine the builder runs on. It names the OS only where an entry for that platform names it. A window whose environment lacks a fact the platform is read from is left out and counted. +- **Bounds.** `probe_demand` points bound their estimates from below only: the estimate must cover them, and they may raise a line, but they take no part in choosing it (`fit.upper_envelope`), and the 1.3x upper bound and `max_ratio` stay with the fresh-process points, because the cold first run after a server start is the worst case. +- **Merge and fit.** A point joins the data file that declares the estimator's coefficients; where a `probe_demand` point of the same card, platform and features is there, the larger of the two stays. The carried values are then refitted with `fit.py`, and a stand-in line that an in-fit point now measures is fitted from then on. A refitted entry's provenance gets a note. Every moved line gets a `line` row: old and new slope and base, whether it was raised, and its new points. +- **Refusals.** An export of another estimator version than the registered one, a window whose features the estimator cannot evaluate, a platform `MEASURED` does not list, and a line whose points span less than twice the pixels: exit code 2. Exit code 1, with nothing written: + - any point of the result breaks its bounds (`working_memory/bounds.py`, the rule the data test applies); the error lists them. A probe point that raises a line above what the fresh-process points of that line allow is refused this way; + - with `--write`, a line is raised on a point flagged as fragmentation, unless `--accept-fragmentation` is given. + +## Validity + +A run's record can be complete and still unfit for some comparisons. Each invalid flag comes with its reason in `invalid_reasons`. + +- **Residency.** The model cache logs `Loaded model '…' … VRAM: … (P%)` whenever a lock moves weights or leaves a model short of fully resident. A warm run usually logs nothing, because its models stay resident. The tool therefore keeps the latest line per model since the last cache flush. Time and memory are invalid when: + - a line in the run's log slice is below 100 %, + - a `VRAM budget … is short` warning appears, or + - no full load of the graph's models was seen since the flush ("residency unknown"). +- **fp8.** With fp8 weights in play, an incompletely resident run does not reproduce its image, so PSNR is invalid too. fp8 is in play when a graph model sets `fp8_storage` in its default settings, or when the server's `fp8_compute` is on. Model configs do not record which checkpoints are scaled fp8, so then every graph counts. +- **Shared memory.** If the server's shared GPU memory grows by more than 256 MiB during a run, it paged to system memory: time and memory are invalid. +- **Foreign load.** Another process above 1 GiB of dedicated GPU memory at either end of the run, or grown by more than 256 MiB, invalidates time and memory. So does a queue that is still busy after the 10-minute wait. The run is repeated, up to three attempts. `start` refuses outright when another process holds more than 1 GiB. + +## Shared paths + +The scratch server uses the template root's models directory, model caches, legacy model configs, custom fonts and custom nodes. Its startup touches them as starting the template's own server would: + +- `models/README.txt` is written. +- Install claims (`*.invokeai_install_active`) of processes that no longer run are removed. +- Missing model files are reported in the log. +- `configs/` receives the legacy model configs of the server's code when they differ. +- `fonts/README.txt` is written when missing. +- Models a node fetches on first use are cached in `models/.download_cache`. + +The server would also finish an incomplete install, but the model would then be registered in the scratch database only. So `start` and `resume` refuse while the models directory holds a `tmpinstall_*` entry: finish or cancel that install in the dev server first. + +Database migrations of a newer `code_dir` run against the scratch database only. A migration that also moves or rewrites files acts on the shared models directory and caches; keep that in mind before pointing a run at a checkout ahead of the template's own server. + +## Limits + +- Memory is sampled every 3 ms with `D3DKMTQueryVideoMemoryInfo`: shorter peaks are missed. It counts resident memory (`CurrentUsage`), not commit, including the CUDA or HIP context and every resident model. That is why records keep a baseline and a peak per node. +- With PyTorch's native allocator, reserved memory stays allocated between nodes, so a node's peak can stem from an earlier node. Compare peaks only between runs with the same allocator configuration; the run records the runtime config. +- On Windows with ROCm, HIP keeps freed blocks at driver level. Per-node peaks after the first large node are upper bounds there (observed on an RX 9060 XT). +- Time per step comes from the preview frames, which the server throttles to one per 0.1 s; a step's number is read from its progress percentage. + - For txt2img and controlnet the case's `steps` fixes the step count. + - img2img starts inside the schedule at a family-specific step, so the count is inferred. When steps are faster than 0.2 s that inference can be off by a factor, so the time per step is left empty. +- Ideogram 4 ignores `steps`; its step count follows `settings.ideogram4Steps` or `settings.ideogram4SamplerPreset`. `img2img` and `controlnet` are not available for ERNIE-Image and Ideogram 4. Dynamic prompts, canvas rasterization, video and upscale graphs are not covered. +- Linux memory sampling and A/B runs of two checkouts are not available yet. +- A calibration run measures the windows of estimated model operations only, inside a running server: they are not a fresh process's first operation, as the `demand_peak` points are. diff --git a/invokeai/app/api/dependencies.py b/invokeai/app/api/dependencies.py index f11b206427e..ee776633b79 100644 --- a/invokeai/app/api/dependencies.py +++ b/invokeai/app/api/dependencies.py @@ -55,6 +55,7 @@ from invokeai.app.services.session_processor.session_processor_default import ( DefaultSessionProcessor, DefaultSessionRunner, + refuse_while_an_abandoned_worker_runs, ) from invokeai.app.services.session_queue.session_queue_default import SessionQueue from invokeai.app.services.shared.database.database import Database @@ -70,6 +71,10 @@ from invokeai.app.services.wildcard_records.wildcard_records_default import WildcardRecordsStorage from invokeai.app.services.workflow_records.workflow_records_default import WorkflowRecordsStorage from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_disk import WorkflowThumbnailFileStorageDisk +from invokeai.app.services.working_memory_calibration.working_memory_calibration_default import ( + WorkingMemoryCalibrationService, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_store import CalibrationStore from invokeai.backend.architectures import conditioning_safe_globals from invokeai.backend.architectures import validate as validate_architectures from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData @@ -117,6 +122,10 @@ def initialize( # lazily inside this function for the same reason, one step earlier: it is what fills the # registry, and importing this module is all `scripts/generate_openapi_schema.py` does. validate_architectures() + # A worker the previous invoker's stop gave up on still holds its GPU, and the session processor would refuse + # to start beside it. Refused here, before the database opens (and migrates), so a refused restart leaves + # nothing open behind it. + refuse_while_an_abandoned_worker_runs() logger.info(f"InvokeAI version {__version__}") logger.info(f"Root directory = {str(config.root_path)}") @@ -216,6 +225,10 @@ def initialize( image_index_records = ImageIndexRecords(database) image_index = ImageIndexService() intermediates = IntermediatesService(records=IntermediatesRecords(database), logger=logger) + # Machine data next to the database rather than in it: see working_memory_calibration_store. + working_memory_calibration = WorkingMemoryCalibrationService( + CalibrationStore(configuration.db_path.parent / "working_memory") + ) fonts = FontService( database, fonts_dir=configuration.fonts_path, @@ -273,8 +286,13 @@ def initialize( fonts=fonts, intermediates=intermediates, gallery_maintenance=gallery_maintenance, + working_memory_calibration=working_memory_calibration, ) + # Before the Invoker starts any service: work outside the session queue, such as the image index's backfill, + # borrows idle GPUs from the generation device pool as soon as its service starts. + session_processor.register_generation_devices(configuration) + # Constructing the Invoker starts every service, including the session # processor thread (which may immediately resume queue items that were # pending at shutdown). Trigger torch attention's lazy, non-thread-safe diff --git a/invokeai/app/api/routers/session_queue.py b/invokeai/app/api/routers/session_queue.py index 2d3c0221abc..25a25c4d8ef 100644 --- a/invokeai/app/api/routers/session_queue.py +++ b/invokeai/app/api/routers/session_queue.py @@ -41,6 +41,10 @@ from invokeai.app.services.shared.graph import Graph, GraphExecutionState from invokeai.app.services.shared.pagination import SQLiteDirection from invokeai.app.services.video_records.video_records_common import VideoRecordNotFoundException +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CALIBRATION_ORIGIN_PREFIX, + is_reserved_origin, +) session_queue_router = APIRouter(prefix="/v1/queue", tags=["queue"]) @@ -256,6 +260,10 @@ def get_queue_item_for_mutation(queue_id: str, item_id: int, current_user: Curre operation_id="enqueue_batch", responses={ 201: {"model": EnqueueBatchResult}, + 400: { + "description": f"The batch's origin starts with '{CALIBRATION_ORIGIN_PREFIX}' (in any case), which is " + "reserved for working-memory calibration runs" + }, }, ) async def enqueue_batch( @@ -265,6 +273,14 @@ async def enqueue_batch( prepend: bool = Body(default=False, description="Whether or not to prepend this batch in the queue"), ) -> EnqueueBatchResult: """Processes a batch and enqueues the output graphs for execution for the current user.""" + # Calibration items are measured and admitted as such by the session runner; only the calibration route may + # create them, for administrators. + if is_reserved_origin(batch.origin): + raise HTTPException( + status_code=400, + detail=f"Origins starting with '{CALIBRATION_ORIGIN_PREFIX}' are reserved for working-memory calibration " + "runs, which start through POST /api/v2/models/i/{key}/working_memory/calibrate.", + ) await asyncio.to_thread(assert_image_move_maintenance_inactive) try: diff --git a/invokeai/app/api/routers/working_memory.py b/invokeai/app/api/routers/working_memory.py new file mode 100644 index 00000000000..7fd6a1ffa04 --- /dev/null +++ b/invokeai/app/api/routers/working_memory.py @@ -0,0 +1,286 @@ +"""Working-memory calibration runs: administrators measure what a model's operations really take on this machine's GPU. + +Mounted under the model manager's prefix: a run calibrates one model (`/i/{key}/working_memory/...`), and its +measurements belong to the machine's device class (`/working_memory/...`). Every route is admin-only: runs spend GPU +time on the instance's behalf, and the store is machine data that names the accounts that created runs. +""" + +import asyncio +from typing import Final, Optional + +from fastapi import Body, HTTPException, Path, Query, Response +from fastapi.routing import APIRouter + +from invokeai.app.api.auth_dependencies import AdminUserOrDefault +from invokeai.app.api.dependencies import ApiDependencies +from invokeai.app.api.routers.image_move_maintenance import assert_image_move_maintenance_inactive +from invokeai.app.services.model_records import UnknownModelException +from invokeai.app.services.session_queue.session_queue_common import ( + EnqueueIdempotencyConflictError, + EnqueueReceiptLimitError, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_base import ( + WorkingMemoryCalibrationServiceBase, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CalibrationGraphError, + CalibrationQueueFullError, + CalibrationRunNotFoundError, + CalibrationStoreError, + CalibrationStoreUnusableError, + CalibrationUnavailableError, + WorkingMemoryCalibrationModelSummary, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationRequest, + WorkingMemoryCalibrationResetResult, + WorkingMemoryCalibrationRunCreated, + WorkingMemoryCalibrationStatus, +) + +working_memory_router = APIRouter(prefix="/v2/models", tags=["model_manager"]) + +_DEVICE_CLASS_ID = Query( + default=None, + pattern=r"^[0-9a-f]{16}$", + description="The device class to act on; the current generation device's by default", +) + + +# One description per error, shared by the routes that return it. +_STORE_UNREADABLE: Final = { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" +} +_NO_DEVICE_CLASS: Final = { + "description": "No device_class_id was given and there is no current one (no GPU, or several generation " + "devices), or the device class's store file is unusable and kept as it is" +} + + +def _service() -> WorkingMemoryCalibrationServiceBase: + service = ApiDependencies.invoker.services.working_memory_calibration + if service is None: + raise HTTPException(status_code=503, detail="Working-memory calibration is not available") + return service + + +def _http_error(error: Exception) -> HTTPException: + if isinstance(error, (CalibrationRunNotFoundError, UnknownModelException)): + return HTTPException(status_code=404, detail=str(error)) + if isinstance(error, CalibrationGraphError): + return HTTPException(status_code=422, detail=str(error)) + if isinstance( + error, + ( + CalibrationUnavailableError, + CalibrationStoreUnusableError, + CalibrationQueueFullError, + EnqueueIdempotencyConflictError, + ), + ): + return HTTPException(status_code=409, detail=str(error)) + if isinstance(error, EnqueueReceiptLimitError): + return HTTPException(status_code=429, detail=str(error)) + # CalibrationStoreError: the root cannot be read or written (read-only, lock timeout). Also the 503 of `_service`. + return HTTPException(status_code=503, detail=str(error)) + + +@working_memory_router.post( + "/i/{key}/working_memory/calibrate", + operation_id="calibrate_model_working_memory", + status_code=201, + response_model=WorkingMemoryCalibrationRunCreated, + responses={ + 404: { + "description": "The model could not be found; the run was removed while it was being created, whose " + "items are queued then and fail when they start; or a reset removed the run the idempotency key started" + }, + 409: { + "description": "This machine cannot calibrate now (no GPU, or several generation devices), the device " + "class's store file is unusable and kept as it is (possibly after the items were queued, which then fail " + "when they start), the queue has no room for all of the run's items (it then takes none), or the " + "idempotency key was used for another request" + }, + 422: {"description": "The graph cannot be measured as a calibration run"}, + 429: {"description": "Too many unacknowledged idempotency keys; acknowledge or wait, then retry"}, + 503: { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout); with an " + "idempotency key, possibly after the items were queued" + }, + }, +) +async def calibrate_model_working_memory( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model the run calibrates"), + request: WorkingMemoryCalibrationRequest = Body(description="The batch to measure"), +) -> WorkingMemoryCalibrationRunCreated: + """Start a working-memory calibration run: enqueue the batch with every node uncached and measure each node's + model operations on the generation device. Needs exactly one generation device, and it must be a GPU. The queue + takes all of the batch's items or none. + + Once the items are queued, a request without an `idempotency_key` succeeds even when the store fails for a passing + reason (an unwritable root, a lock timeout) to record them: they run all the same, and the status finds them by + their origin. With a key, that failure is a 503 although the items are queued. A run removed meanwhile (404) or an + unusable store file (409) fails the request with or without a key; its queued items then fail when they start. + + With a key, repeating the request (also after a lost response) records the queued items and returns the run; it + enqueues nothing new. The items come from the queue's enqueue receipt for the key while it exists, otherwise from + the queue items that carry the run's origin and those the runner saw. Once a reset removed the run, a repeat is + refused (404). Acknowledge the key through the queue's enqueue_batch/acknowledge route once the response + arrived.""" + await asyncio.to_thread(assert_image_move_maintenance_inactive) + try: + return await _service().create_run(key, request, current_admin.user_id) + except ( + UnknownModelException, + CalibrationRunNotFoundError, + CalibrationGraphError, + CalibrationUnavailableError, + CalibrationStoreError, + CalibrationQueueFullError, + EnqueueIdempotencyConflictError, + EnqueueReceiptLimitError, + ) as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/working_memory/status", + operation_id="get_working_memory_calibration_status", + response_model=WorkingMemoryCalibrationStatus, + responses={503: _STORE_UNREADABLE}, +) +def get_working_memory_calibration_status(current_admin: AdminUserOrDefault) -> WorkingMemoryCalibrationStatus: + """Every device class's calibration runs, with their derived status, point counts and environments.""" + try: + return _service().get_status() + except CalibrationStoreError as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/i/{key}/working_memory/plan", + operation_id="get_model_working_memory_calibration_plan", + response_model=WorkingMemoryCalibrationPlan, + responses={404: {"description": "The model could not be found"}}, +) +def get_model_working_memory_calibration_plan( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model to calibrate"), +) -> WorkingMemoryCalibrationPlan: + """The text-to-image cases that calibrate the model's decode on this machine's GPU, how often each runs, which + other installed models the calibration also moves, or why the model cannot be calibrated here (`supported`).""" + try: + return _service().get_plan(key) + except UnknownModelException as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/i/{key}/working_memory/summary", + operation_id="get_model_working_memory_calibration_summary", + response_model=WorkingMemoryCalibrationModelSummary, + responses={ + 404: {"description": "The model could not be found"}, + 409: {"description": "No device_class_id was given and there is no current one"}, + 503: _STORE_UNREADABLE, + }, +) +def get_model_working_memory_calibration_summary( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model whose measurements to summarise"), + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> WorkingMemoryCalibrationModelSummary: + """What the calibration windows of the model's lines measured on one device class, per estimator line and + environment: the sizes, the largest demand, the shipped and the user estimate, whether the user line is in use, + stale or unusable, and which models measured it, also for a line the model's decode shares with other models' + measurements; and the model's runs.""" + try: + return _service().get_model_summary(key, device_class_id) + except (UnknownModelException, CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.delete( + "/working_memory/runs/{run_id}", + operation_id="delete_working_memory_calibration_run", + response_model=WorkingMemoryCalibrationResetResult, + responses={ + 404: {"description": "No usable store file holds the run"}, + 409: {"description": "The run's store file became unusable after the run was found; it is kept as it is"}, + 503: _STORE_UNREADABLE, + }, +) +def delete_working_memory_calibration_run( + current_admin: AdminUserOrDefault, + run_id: str = Path(description="The calibration run to remove"), +) -> WorkingMemoryCalibrationResetResult: + """Remove a calibration run and its measurements. Its queue items that have not run yet fail when they start.""" + try: + return _service().reset_run(run_id) + except (CalibrationRunNotFoundError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.delete( + "/i/{key}/working_memory", + operation_id="reset_model_working_memory_calibration", + response_model=WorkingMemoryCalibrationResetResult, + responses={409: _NO_DEVICE_CLASS, 503: _STORE_UNREADABLE}, +) +def reset_model_working_memory_calibration( + current_admin: AdminUserOrDefault, + key: str = Path(description="Key of the model whose runs and measurements to remove"), + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> WorkingMemoryCalibrationResetResult: + """Remove a model's calibration runs with all their measurements, and every measurement of the model taken in + other runs (a VAE measured inside a transformer's run, say), from one device class.""" + try: + return _service().reset_model(key, device_class_id) + except (CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.delete( + "/working_memory", + operation_id="reset_working_memory_calibration", + response_model=WorkingMemoryCalibrationResetResult, + responses={409: _NO_DEVICE_CLASS, 503: _STORE_UNREADABLE}, +) +def reset_working_memory_calibration( + current_admin: AdminUserOrDefault, + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> WorkingMemoryCalibrationResetResult: + """Remove every calibration run and measurement of one device class.""" + try: + return _service().reset_all(device_class_id) + except (CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + + +@working_memory_router.get( + "/working_memory/export", + operation_id="export_working_memory_calibration", + response_class=Response, + responses={ + 200: { + "description": "The device class's store file byte for byte: a CalibrationStoreFile (schema_version 1) " + "when this version can read it, otherwise whatever the file holds", + "content": {"application/json": {}}, + }, + 404: {"description": "The device class has no store file"}, + 409: {"description": "No device_class_id was given and there is no current one"}, + 503: _STORE_UNREADABLE, + }, +) +def export_working_memory_calibration( + current_admin: AdminUserOrDefault, + device_class_id: Optional[str] = _DEVICE_CLASS_ID, +) -> Response: + """The device class's calibration store file as it is on disk, readable by this version or not.""" + try: + raw = _service().export(device_class_id) + except (CalibrationUnavailableError, CalibrationStoreError) as error: + raise _http_error(error) from error + if raw is None: + raise HTTPException(status_code=404, detail="This device class has no calibration store") + return Response(content=raw, media_type="application/json") diff --git a/invokeai/app/api_app.py b/invokeai/app/api_app.py index 4802132c2b6..7b359a9e02e 100644 --- a/invokeai/app/api_app.py +++ b/invokeai/app/api_app.py @@ -52,6 +52,7 @@ virtual_boards, wildcards, workflows, + working_memory, ) from invokeai.app.api.sockets import SocketIO from invokeai.app.services.config.config_default import get_config @@ -786,6 +787,7 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: app.include_router(utilities.utilities_router, prefix="/api") app.include_router(fonts.fonts_router, prefix="/api") app.include_router(model_manager.model_manager_router, prefix="/api") +app.include_router(working_memory.working_memory_router, prefix="/api") app.include_router(image_moves.image_moves_router, prefix="/api") app.include_router(images.images_router, prefix="/api") app.include_router(image_map.image_map_router, prefix="/api") diff --git a/invokeai/app/invocations/vae/image_to_latents.py b/invokeai/app/invocations/vae/image_to_latents.py index d599fadd862..1be14f136b2 100644 --- a/invokeai/app/invocations/vae/image_to_latents.py +++ b/invokeai/app/invocations/vae/image_to_latents.py @@ -27,7 +27,7 @@ from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor from invokeai.backend.stable_diffusion.vae_tiling import patch_vae_tiling_params from invokeai.backend.util.devices import TorchDevice -from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl +from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl, sd_vae_run_dtype """ SDXL VAE color compensation values determined experimentally to reduce color drift. @@ -86,9 +86,10 @@ def vae_encode( with vae_info.model_on_device(working_memory=estimated_working_memory) as (_, vae): assert isinstance(vae, (AutoencoderKL, AutoencoderTiny)), "VAE must be of type SD-1.5 or SDXL" orig_dtype = vae.dtype - if upcast: - vae.to(dtype=torch.float32) - + # The dtype the estimate above priced: fp32 when upcasting, fp16 otherwise, on every device. + run_dtype = sd_vae_run_dtype("encode", upcast, vae_info.compute_device) + vae.to(dtype=run_dtype) + if run_dtype == torch.float32: use_torch_2_0_or_xformers = hasattr(vae.decoder, "mid_block") and isinstance( vae.decoder.mid_block.attentions[0].processor, ( @@ -107,10 +108,6 @@ def vae_encode( # else: # latents = latents.float() - else: - vae.to(dtype=torch.float16) - # latents = latents.half() - if tiled: vae.enable_tiling() else: diff --git a/invokeai/app/invocations/vae/latents_to_image.py b/invokeai/app/invocations/vae/latents_to_image.py index 29c6a1b9db8..bb2aafd0bfb 100644 --- a/invokeai/app/invocations/vae/latents_to_image.py +++ b/invokeai/app/invocations/vae/latents_to_image.py @@ -20,7 +20,7 @@ from invokeai.backend.stable_diffusion.extensions.seamless import SeamlessExt from invokeai.backend.stable_diffusion.vae_tiling import patch_vae_tiling_params from invokeai.backend.util.devices import TorchDevice -from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl +from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_sd15_sdxl, sd_vae_run_dtype @invocation( @@ -74,16 +74,12 @@ def invoke(self, context: InvocationContext) -> ImageOutput: # wrongly place the latents (and thus the whole decode) on the CPU (see #9373). device = vae_info.compute_device latents = latents.to(device) - # Force fp32 when running on CPU (e.g. when the VAE is configured cpu_only). fp16 conv does run on CPU with + # fp32 when asked, and on the CPU (e.g. when the VAE is configured cpu_only): fp16 conv does run on CPU with # the pinned torch, but it's much slower than fp32 there, and SD/SDXL VAE has known fp16 overflow issues - # that produce black images — fp32 avoids both. - if self.fp32 or device.type == "cpu": - # FP32 mode: convert everything to float32 for maximum precision - vae.to(dtype=torch.float32) - latents = latents.float() - else: - vae.to(dtype=torch.float16) - latents = latents.half() + # that produce black images — fp32 avoids both. fp16 otherwise. The estimate above priced the same rule. + run_dtype = sd_vae_run_dtype("decode", self.fp32, device) + vae.to(dtype=run_dtype) + latents = latents.to(dtype=run_dtype) if use_tiling: vae.enable_tiling() diff --git a/invokeai/app/services/invocation_services.py b/invokeai/app/services/invocation_services.py index f5416b02bb6..618de627229 100644 --- a/invokeai/app/services/invocation_services.py +++ b/invokeai/app/services/invocation_services.py @@ -56,6 +56,9 @@ from invokeai.app.services.videos.videos_base import VideoServiceABC from invokeai.app.services.workflow_records.workflow_records_base import WorkflowRecordsStorageBase from invokeai.app.services.workflow_thumbnails.workflow_thumbnails_base import WorkflowThumbnailServiceBase + from invokeai.app.services.working_memory_calibration.working_memory_calibration_base import ( + WorkingMemoryCalibrationServiceBase, + ) from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData @@ -117,6 +120,7 @@ def __init__( progress_previews: "ProgressPreviewsBase | None" = None, fonts: "FontService | None" = None, gallery_maintenance: "GalleryMaintenanceService | None" = None, + working_memory_calibration: "WorkingMemoryCalibrationServiceBase | None" = None, board_access: "BoardAccessService | None" = None, ): self.board_images = board_images @@ -170,6 +174,7 @@ def __init__( self.fonts = fonts self.intermediates = intermediates self.gallery_maintenance = gallery_maintenance + self.working_memory_calibration = working_memory_calibration # Stateless over the database, so a real one is the default: no caller should get looser rules by leaving it # out. self.board_access = board_access or BoardAccessService(database) diff --git a/invokeai/app/services/invoker.py b/invokeai/app/services/invoker.py index b504f511ba7..8c01125a05b 100644 --- a/invokeai/app/services/invoker.py +++ b/invokeai/app/services/invoker.py @@ -22,13 +22,25 @@ def __stop_service(self, service) -> None: if callable(stop_op): stop_op(self) + def _services_but_the_session_processor(self) -> list[object]: + return [getattr(self.services, name) for name in vars(self.services) if name != "session_processor"] + def _start(self) -> None: - """Starts the invoker. This is called automatically when the invoker is created.""" - for service in vars(self.services): - self.__start_service(getattr(self.services, service)) + """Starts the invoker. This is called automatically when the invoker is created. + + The session processor starts after every other service: its workers dequeue at once, and an item left pending + before the restart (the queue's start cancels those that were in progress) may reach any service, which must + have been started by then. + """ + for service in self._services_but_the_session_processor(): + self.__start_service(service) + self.__start_service(self.services.session_processor) def stop(self) -> None: """Stops the invoker. A new invoker will have to be created to execute further.""" - # First stop all services - for service in vars(self.services): - self.__stop_service(getattr(self.services, service)) + # The session processor stops first: its workers stop taking queue items and cancel the sessions they run, and + # its stop() waits for them to exit (bounded, see DefaultSessionProcessor.stop) before the services those + # sessions use stop. + self.__stop_service(self.services.session_processor) + for service in self._services_but_the_session_processor(): + self.__stop_service(service) diff --git a/invokeai/app/services/session_processor/session_processor_default.py b/invokeai/app/services/session_processor/session_processor_default.py index 4308e7a4f35..5d41b46eefc 100644 --- a/invokeai/app/services/session_processor/session_processor_default.py +++ b/invokeai/app/services/session_processor/session_processor_default.py @@ -4,13 +4,14 @@ from contextlib import contextmanager, nullcontext, suppress from threading import BoundedSemaphore, Thread from threading import Event as ThreadEvent -from typing import Iterator, Optional +from typing import Iterator, Literal, Optional import torch from starlette.concurrency import run_in_threadpool from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput from invokeai.app.invocations.call_saved_workflow import CallSavedWorkflowInvocation +from invokeai.app.services.config.config_default import InvokeAIAppConfig from invokeai.app.services.events.events_common import ( BatchEnqueuedEvent, FastAPIEvent, @@ -45,6 +46,7 @@ ) from invokeai.app.services.shared.graph import CollectInvocation, IterateInvocation, NodeInputError from invokeai.app.services.shared.invocation_context import InvocationContextData, build_invocation_context +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import calibration_run_id from invokeai.app.util.profiler import Profiler from invokeai.backend.util.device_pool import ( GENERATION_DEVICE_POOL, @@ -52,6 +54,7 @@ ) from invokeai.backend.util.devices import TorchDevice, disable_conv_benchmark_empty_cache from invokeai.backend.util.wddm import paged_bytes +from invokeai.backend.working_memory import probe # Paged-out VRAM worth a warning. Weight uploads page a few hundred MB for an instant (0.25-0.36 GiB measured on an # RX 9060 XT), so the threshold sits above that; a session that ends with more than this paged has lost real VRAM. @@ -64,6 +67,27 @@ OWNER_LOOKUP_ATTEMPTS = 3 OWNER_LOOKUP_RETRY_SECONDS = 0.25 +# How long `DefaultSessionProcessor.stop` waits, in all, for its workers to leave their sessions. A node that checks for +# cancellation stops within one step; the bound also covers most nodes without such a check (a large VAE decode), and +# keeps a node that never returns from holding up shutdown for good. +WORKER_STOP_TIMEOUT_SECONDS = 30.0 +# After this long, `stop` says which worker it is waiting for: the wait runs on the event-loop thread at shutdown. +WORKER_STOP_NOTICE_SECONDS = 1.0 + +# Workers a stop gave up on. Process-wide, like the device pool whose locks they may still hold: no processor of this +# process starts, and none replaces the pool, while one of them runs. +_abandoned_workers: list[Thread] = [] + + +def refuse_while_an_abandoned_worker_runs() -> None: + """Raise RuntimeError while a worker an earlier `DefaultSessionProcessor.stop` gave up on still runs.""" + _abandoned_workers[:] = [thread for thread in _abandoned_workers if thread.is_alive()] + if _abandoned_workers: + raise RuntimeError( + f"Session worker {', '.join(thread.name for thread in _abandoned_workers)} of the previous session " + "processor is still running a node; it would share its GPU with the new workers. Restart the process." + ) + def queue_owner_is_active( services: InvocationServices, queue_item: SessionQueueItem, *, unreadable_is_active: bool = False @@ -165,6 +189,11 @@ def __init__( self.workflow_call_queue_lifecycle = WorkflowCallQueueLifecycle(self) # Per runner, hence per worker and device: each worker remembers the architecture it last ran. self._architecture_switch = ArchitectureSwitchOffload() + # The calibration run of the item this runner is executing, None for every other item. + self._calibration_run_id: Optional[str] = None + # What the outcome hooks last reported: `run_node` swallows a node's failure or cancellation, and a calibration + # node must not keep valid measurements then. + self._node_outcome: Optional[Literal["completed", "failed"]] = None def start(self, services: InvocationServices, cancel_event: ThreadEvent, profiler: Optional[Profiler] = None): self._services = services @@ -203,7 +232,10 @@ def _run_session_loop(self, queue_item: SessionQueueItem) -> None: if self._cancel_if_owner_revoked(queue_item): break - self.run_node(invocation, queue_item) + if self._calibration_run_id is None: + self.run_node(invocation, queue_item) + elif not self._run_calibration_node(invocation, queue_item, self._calibration_run_id): + break # The session is complete if all invocations have been run or there is an error on the session. # At this time, the queue item may be canceled, but the object itself here won't be updated yet. We must @@ -246,10 +278,117 @@ def _cancel_if_owner_revoked(self, queue_item: SessionQueueItem, *, unreadable_i def run(self, queue_item: SessionQueueItem): # Exceptions raised outside `run_node` are handled by the processor. There is no need to catch them here. + self._refresh_user_fits() + # getattr, like the workflow-call lifecycle's reads: runners also execute stand-in items without an origin. + run_id = calibration_run_id(getattr(queue_item, "origin", None)) + if run_id is not None: + self._run_calibration_item(queue_item, run_id) + return self._on_before_run_session(queue_item=queue_item) self._run_session_loop(queue_item) self._on_after_run_session(queue_item=queue_item) + def _refresh_user_fits(self) -> None: + """Let this item's estimates read the calibration store as it is now: once per item, never per estimate.""" + # getattr: stand-in service containers that runners execute with need not carry the calibration service. + service = getattr(self._services, "working_memory_calibration", None) + if service is None: + return + try: + service.refresh_user_fits() + except Exception: + # The fits read so far stay in force; a store that cannot be read must not fail a generation. + self._services.logger.warning("Could not refresh the working-memory user fits", exc_info=True) + + def _run_calibration_item(self, queue_item: SessionQueueItem, run_id: str) -> None: + """Run a working-memory calibration item, measuring each node (see `working_memory.probe`). + + The item is admitted again here, whatever admitted it to the queue -- a retry and a workflow-call child carry + the origin and owner of the item they came from: its owner must still be an administrator, the machine must + still calibrate on one GPU, and its run must still exist. Otherwise it fails before any node runs. + """ + service = self._services.working_memory_calibration + raised: Optional[BaseException] = None + try: + try: + refusal = ( + "Working-memory calibration is not available in this process." + if service is None + else service.admit(queue_item, run_id) + ) + except Exception as error: + self._services.logger.warning(f"Could not admit calibration item {queue_item.item_id}", exc_info=True) + refusal = f"Could not admit the calibration item: {error}" + if refusal is not None: + self._services.logger.warning(f"Failing calibration item {queue_item.item_id}: {refusal}") + with suppress(SessionQueueItemNotFoundError): + self._services.session_queue.fail_queue_item( + queue_item.item_id, "WorkingMemoryCalibrationError", refusal, refusal + ) + return + self._calibration_run_id = run_id + try: + self._on_before_run_session(queue_item=queue_item) + self._run_session_loop(queue_item) + self._on_after_run_session(queue_item=queue_item) + finally: + self._calibration_run_id = None + except BaseException as error: + raised = error + raise + finally: + # Also when the session raised: `admit` recorded the item as running, and only this replaces that. + if service is not None: + try: + service.observe(queue_item, run_id, raised) + except Exception: + self._services.logger.warning( + f"Could not record the outcome of calibration item {queue_item.item_id}", exc_info=True + ) + + def _run_calibration_node(self, invocation: BaseInvocation, queue_item: SessionQueueItem, run_id: str) -> bool: + """Run one node of a calibration item inside a probe scope and persist its windows, whether the node completed, + failed or was canceled; unless it completed, they are persisted invalid. Returns False when they could not be + persisted: the item has failed then.""" + service = self._services.working_memory_calibration + try: + assert service is not None + # Read before the node runs: a model reset while it runs removes points these windows must not bring back. + model_reset_epochs = service.model_reset_epochs(run_id) + except Exception as error: + return self._fail_calibration_node(invocation, queue_item, error) + with probe.calibration_scope( + run_id=run_id, + queue_item_id=queue_item.item_id, + # The source id: what the client that built the graph knows the node by. + node_id=queue_item.session.prepared_source_mapping.get(invocation.id, invocation.id), + node_type=invocation.get_type(), + ) as scope: + self._node_outcome = None + self.run_node(invocation, queue_item) + if self._node_outcome != "completed": + # A node that neither completed nor failed was canceled: `run_node` passes over a CanceledException. + scope.invalidate("exception" if self._node_outcome == "failed" else "canceled") + if not scope.windows: + return True + try: + service.record_windows(run_id, scope.windows, model_reset_epochs) + except Exception as error: + return self._fail_calibration_node(invocation, queue_item, error) + return True + + def _fail_calibration_node( + self, invocation: BaseInvocation, queue_item: SessionQueueItem, error: Exception + ) -> bool: + """Fail the item whose node's measurements cannot be saved. Call from the `except` block (for the traceback).""" + message = f"Could not save the working-memory measurements of node {invocation.id}: {error}" + self._services.logger.error(message, exc_info=True) + with suppress(SessionQueueItemNotFoundError): + self._services.session_queue.fail_queue_item( + queue_item.item_id, "WorkingMemoryCalibrationError", message, traceback.format_exc() + ) + return False + def run_node(self, invocation: BaseInvocation, queue_item: SessionQueueItem): try: # Any unhandled exception in this scope is an invocation error & will fail the graph @@ -573,6 +712,7 @@ def _on_after_run_node( - Emits an invocation complete event. - Run any callbacks registered for this event. + - Records that the node completed (see `_run_calibration_node`). """ self._services.logger.debug( @@ -587,6 +727,7 @@ def _on_after_run_node( for callback in self._on_after_run_node_callbacks: callback(invocation=invocation, queue_item=queue_item, output=output) + self._node_outcome = "completed" def _on_node_error( self, @@ -599,12 +740,14 @@ def _on_node_error( ): """Called when a node errors. Node errors may occur when running or preparing the node.. + - Records that the node failed, ahead of anything that can raise (see `_run_calibration_node`). - Set the node error on the session object. - Log the error. - Fail the queue item. - Emits an invocation error event. - Run any callbacks registered for this event. """ + self._node_outcome = "failed" self._services.progress_previews.clear_node(queue_item.item_id, invocation.id) self._services.logger.debug( @@ -708,20 +851,30 @@ def __init__( self._thread_limit = thread_limit self._polling_interval = polling_interval self._workers: list[_SessionWorker] = [] + # One per worker, once `register_generation_devices` resolved them. + self._devices: Optional[list[Optional[torch.device]]] = None - def _resolve_devices(self) -> list[Optional[torch.device]]: - """Determine the per-worker devices from config. + def register_generation_devices(self, configuration: InvokeAIAppConfig) -> list[Optional[torch.device]]: + """Resolve the workers' devices and register them in `GENERATION_DEVICE_POOL`, so the model loader can find + idle GPUs to borrow. Returns the devices, one per worker. - Resolves `generation_devices` (which defaults to `"auto"` — every available GPU) into one - normalized device per worker. Returns a single `None` (legacy single-worker, device chosen by - the global config) only if the resolution is empty (e.g. `generation_devices` set to an empty - list). + Work outside the session queue borrows from the pool from the moment its service starts (the image index's + backfill, say), so the composition root calls this before the invoker starts any service + (`ApiDependencies.initialize`). `start` calls it only when nothing did: registering again would drop a borrow in + flight. Registering creates no CUDA context: the pool holds device objects, and a worker pins its GPU in CUDA + only once it claims its first queue item (#9413). + + `generation_devices` (by default `"auto"`, every available GPU) resolves to one normalized device per worker, + or to a single `None` (one worker on the device the global config chooses) when the resolution is empty, as + with an empty list. + + Raises RuntimeError while a worker an earlier `stop` gave up on still runs (see `stop`). """ - generation_devices = self._invoker.services.configuration.generation_devices - devices = TorchDevice.get_generation_devices(generation_devices) - if not devices: - return [None] - return list(devices) + refuse_while_an_abandoned_worker_runs() + devices = TorchDevice.get_generation_devices(configuration.generation_devices) + self._devices = list(devices) if devices else [None] + GENERATION_DEVICE_POOL.set_generation_devices([d for d in self._devices if d is not None]) + return self._devices def _clone_session_runner(self, template: SessionRunnerBase) -> SessionRunnerBase: """Create an independent runner for an additional worker. @@ -749,6 +902,9 @@ def _clone_session_runner(self, template: SessionRunnerBase) -> SessionRunnerBas ) def start(self, invoker: Invoker) -> None: + """Start one worker per generation device. Raises RuntimeError while a worker an earlier `stop` gave up on + still runs (see `stop`).""" + refuse_while_an_abandoned_worker_runs() self._invoker: Invoker = invoker self._resume_event = ThreadEvent() @@ -760,11 +916,11 @@ def start(self, invoker: Invoker) -> None: register_events(QueueItemStatusChangedEvent, self._on_queue_item_status_changed) register_events(UserAccessChangedEvent, self._on_user_access_changed) - devices = self._resolve_devices() - - # Register the generation devices so the model loader can discover idle GPUs to host text - # encoders on (see offload_text_encoders_to_idle_gpus). None means legacy single-device mode. - GENERATION_DEVICE_POOL.set_generation_devices([d for d in devices if d is not None]) + devices = ( + self._devices + if self._devices is not None + else self.register_generation_devices(self._invoker.services.configuration) + ) # A worker that stood aside for a lent GPU re-polls as soon as that borrow ends. GENERATION_DEVICE_POOL.set_release_listener(self._poll_now_event.set) @@ -827,6 +983,15 @@ def start(self, invoker: Invoker) -> None: worker.thread.start() def stop(self, *args, **kwargs) -> None: + """Stop the workers and wait for them to exit, up to `WORKER_STOP_TIMEOUT_SECONDS` for all of them together. + + The invoker stops every other service once this returns, so a worker that exits in time never runs beside a + stopped service. A wait longer than `WORKER_STOP_NOTICE_SECONDS` names the worker and its queue item in the + log. A worker still in a node at the deadline is given up on, with an error in the log, so that shutdown + finishes: a node without a cancel point may never return. A later stop does not wait for it again. It keeps + its GPU, so until it exits no session processor of this process starts and none replaces the generation device + pool. The services it uses stop under it; once the database is disposed, its writes are refused. + """ self._stop_event.set() # Cancel any in-progress generation so that long-running nodes (e.g. denoising) stop at # the next step boundary instead of running to completion. Without this, a generation @@ -838,6 +1003,32 @@ def stop(self, *args, **kwargs) -> None: self._poll_now_event.set() self._resume_event.set() + deadline = time.monotonic() + WORKER_STOP_TIMEOUT_SECONDS + for worker in self._workers: + # A worker given up on by an earlier stop gets no second wait. + if worker.thread is None or worker.thread in _abandoned_workers: + continue + worker.thread.join(timeout=max(0.0, min(deadline - time.monotonic(), WORKER_STOP_NOTICE_SECONDS))) + remaining = deadline - time.monotonic() + if worker.thread.is_alive() and remaining > 0: + self._invoker.services.logger.info( + f"Waiting up to {remaining:.0f}s for the {self._describe_running(worker)} to stop" + ) + worker.thread.join(timeout=remaining) + if worker.thread.is_alive(): + _abandoned_workers.append(worker.thread) + self._invoker.services.logger.error( + f"The {self._describe_running(worker)} did not stop within {WORKER_STOP_TIMEOUT_SECONDS:.0f}s; " + "shutting down without it. Restart the process before starting Invoke again." + ) + + @staticmethod + def _describe_running(worker: _SessionWorker) -> str: + assert worker.thread is not None + queue_item = worker.queue_item + item = f" (queue item {queue_item.item_id})" if queue_item is not None else "" + return f"session worker {worker.thread.name} on {worker.label}{item}" + def _poll_now(self) -> None: self._poll_now_event.set() diff --git a/invokeai/app/services/session_queue/session_queue_base.py b/invokeai/app/services/session_queue/session_queue_base.py index 3e7bc4841fb..edf6dc85836 100644 --- a/invokeai/app/services/session_queue/session_queue_base.py +++ b/invokeai/app/services/session_queue/session_queue_base.py @@ -21,6 +21,7 @@ NodeFieldValue, PruneResult, QueueItemDeliveryResult, + QueueItemLineage, QueueItemOwner, RetryItemsResult, SessionQueueCountsByDestination, @@ -54,9 +55,11 @@ def dequeue(self, device: Optional[str] = None) -> Optional[SessionQueueItem]: @abstractmethod def enqueue_batch( - self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system" + self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system", *, all_or_none: bool = False ) -> Coroutine[Any, Any, EnqueueBatchResult]: - """Enqueues all permutations of a batch for execution for a specific user.""" + """Enqueues all permutations of a batch for execution for a specific user. A queue without room for all of + them takes as many as fit, or with `all_or_none` none of them. A repeat with the batch's idempotency key + returns the key's settled receipt as it is: `all_or_none` is not part of the payload the receipt matches.""" pass @abstractmethod @@ -290,6 +293,12 @@ def get_queue_item_ids( matching items only when no limit is given.""" pass + @abstractmethod + def get_queue_item_lineage(self, queue_id: str, origin_prefix: str) -> list[QueueItemLineage]: + """The status, origin and retry source of every item whose origin starts with `origin_prefix` (ignoring + case), in no order.""" + pass + @abstractmethod def get_queue_item_summaries_by_ids(self, queue_id: str, item_ids: list[int]) -> list[SessionQueueItemSummary]: """Gets lightweight queue item summaries in the requested item ID order.""" diff --git a/invokeai/app/services/session_queue/session_queue_common.py b/invokeai/app/services/session_queue/session_queue_common.py index 31fc0ada27c..458f043af4d 100644 --- a/invokeai/app/services/session_queue/session_queue_common.py +++ b/invokeai/app/services/session_queue/session_queue_common.py @@ -216,6 +216,15 @@ def validate_graph(cls, v: Graph): QUEUE_ITEM_STATUS = Literal["pending", "in_progress", "waiting", "completed", "failed", "canceled"] +class QueueItemLineage(NamedTuple): + """An item's status and origin, and the item it retried: a retry of a retry names the first item as well.""" + + item_id: int + status: QUEUE_ITEM_STATUS + origin: Optional[str] + retried_from_item_id: Optional[int] + + class ItemIdsResult(BaseModel): """Response containing ordered item ids with metadata for optimistic updates.""" diff --git a/invokeai/app/services/session_queue/session_queue_default.py b/invokeai/app/services/session_queue/session_queue_default.py index 01df20c0052..33f8968838b 100644 --- a/invokeai/app/services/session_queue/session_queue_default.py +++ b/invokeai/app/services/session_queue/session_queue_default.py @@ -3,7 +3,7 @@ import json import math from datetime import datetime, timedelta, timezone -from typing import Any, Literal, Optional +from typing import Any, Literal, Optional, cast from pydantic_core import to_jsonable_python @@ -32,6 +32,7 @@ NodeFieldValue, PruneResult, QueueItemDeliveryResult, + QueueItemLineage, QueueItemOwner, RetriedQueueItem, RetryItemsResult, @@ -144,11 +145,11 @@ def _prune_terminal_to_limit(self, queue_id: str, keep: int) -> int: return self._queries.session_queue.prune(queue_id, user_id=None, keep=keep) async def enqueue_batch( - self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system" + self, queue_id: str, batch: Batch, prepend: bool, user_id: str = "system", *, all_or_none: bool = False ) -> EnqueueBatchResult: # The route awaits this method, but every operation below is synchronous database/CPU work. Keep the # complete enqueue in one worker operation, off the event loop. - return await asyncio.to_thread(self._enqueue_batch, queue_id, batch, prepend, user_id) + return await asyncio.to_thread(self._enqueue_batch, queue_id, batch, prepend, user_id, all_or_none) def acknowledge_enqueue(self, queue_id: str, idempotency_key: str, user_id: str = "system") -> None: self._queries.session_queue.acknowledge_receipt(queue_id, user_id, idempotency_key) @@ -159,7 +160,9 @@ def get_enqueue_receipt( receipt = self._queries.session_queue.enqueue_receipt(queue_id, user_id, idempotency_key) return _decode_enqueue_receipt(*receipt) if receipt is not None else None - def _enqueue_batch(self, queue_id: str, batch: Batch, prepend: bool, user_id: str) -> EnqueueBatchResult: + def _enqueue_batch( + self, queue_id: str, batch: Batch, prepend: bool, user_id: str, all_or_none: bool = False + ) -> EnqueueBatchResult: requested_count = calc_session_count(batch=batch) payload_hash = ( hashlib.sha256( @@ -208,11 +211,12 @@ def survey(q: Queries) -> SettledReceipt | int: # Preparing the values is CPU work, done outside a transaction; admission below takes what still fits. max_queue_size = self.__invoker.services.configuration.max_queue_size + room = max(0, max_queue_size - surveyed) prepared_values = prepare_values_to_insert( queue_id=queue_id, batch=batch, priority=0, - max_new_queue_items=max(0, max_queue_size - surveyed), + max_new_queue_items=0 if all_or_none and requested_count > room else room, user_id=user_id, ) @@ -225,6 +229,9 @@ def admit(q: Queries) -> SettledReceipt | EnqueueBatchResult: require_project(q) max_new_queue_items = max(0, max_queue_size - q.session_queue.pending_count(queue_id)) + # Checked under the admission lock: the room the survey saw may have been taken meanwhile. + if all_or_none and min(len(prepared_values), max_new_queue_items) < requested_count: + max_new_queue_items = 0 priority = q.session_queue.top_pending_priority(queue_id) + 1 if prepend else 0 values_to_insert = prepared_values[:max_new_queue_items] if priority != 0: @@ -1037,6 +1044,14 @@ def get_queue_item_ids( def has_active_queue_work(self) -> bool: return self._queries.session_queue.has_active_work() + def get_queue_item_lineage(self, queue_id: str, origin_prefix: str) -> list[QueueItemLineage]: + return [ + QueueItemLineage(item_id, cast(QUEUE_ITEM_STATUS, status), origin, retried_from_item_id) + for item_id, status, origin, retried_from_item_id in self._queries.session_queue.lineage( + queue_id, origin_prefix + ) + ] + def get_queue_item_summaries_by_ids(self, queue_id: str, item_ids: list[int]) -> list[SessionQueueItemSummary]: if not item_ids: return [] diff --git a/invokeai/app/services/shared/database/queries/session_queue.py b/invokeai/app/services/shared/database/queries/session_queue.py index 120ee0c38af..0e8056d09aa 100644 --- a/invokeai/app/services/shared/database/queries/session_queue.py +++ b/invokeai/app/services/shared/database/queries/session_queue.py @@ -114,6 +114,10 @@ def _origin_pattern(origin_prefix: Optional[str]) -> Optional[str]: return like_prefix(origin_prefix) if origin_prefix is not None else None +# The origin is tested on the listing index; only the matching items' rows are read. +_LINEAGE = select(_Q.item_id, _Q.status, _Q.origin, _Q.retried_from_item_id).where(_IN_QUEUE, _origin_matches()) + + @functools.cache def _item_by_status(status: str, by_origin: bool, by_owner: bool) -> Select[Any]: """The pending item to run next, or an item in progress (the earliest enqueued of several).""" @@ -249,6 +253,10 @@ def _summaries(rows: Sequence[Sequence[Any]]) -> list[dict[str, Any]]: return [dict(zip(_SUMMARY_NAMES, row, strict=True)) for row in rows] +def _lineage(rows: Sequence[Sequence[Any]]) -> list[tuple[int, str, Optional[str], Optional[int]]]: + return [(row[0], row[1], row[2], row[3]) for row in rows] + + def _counts(rows: Sequence[Sequence[Any]]) -> dict[str, int]: return {str(row[0]): int(row[1]) for row in rows} @@ -841,6 +849,12 @@ def item_ids( } return list(conn.execute(statement, parameters).scalars().all()) + @mapped(_lineage) + @read + def lineage(self, conn: Connection, queue_id: str, origin_prefix: str) -> Sequence[Row[Any]]: + """Id, status, origin and retry source of the queue's items with the origin, in no order.""" + return conn.execute(_LINEAGE, {"queue_id": queue_id, "origin_pattern": like_prefix(origin_prefix)}).all() + @mapped(_summaries) @read def summaries(self, conn: Connection, queue_id: str, item_ids: Sequence[int]) -> Sequence[Row[Any]]: diff --git a/invokeai/app/services/working_memory_calibration/__init__.py b/invokeai/app/services/working_memory_calibration/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_base.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_base.py new file mode 100644 index 00000000000..28f909883b0 --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_base.py @@ -0,0 +1,93 @@ +from abc import ABC, abstractmethod +from collections.abc import Mapping +from typing import Optional + +from invokeai.app.services.session_queue.session_queue_common import SessionQueueItem +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + WorkingMemoryCalibrationModelSummary, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationRequest, + WorkingMemoryCalibrationResetResult, + WorkingMemoryCalibrationRunCreated, + WorkingMemoryCalibrationStatus, +) +from invokeai.backend.working_memory.probe import MeasuredWindow + + +class WorkingMemoryCalibrationServiceBase(ABC): + """Calibration runs: enqueues them, admits their queue items, and keeps their measurements per device class.""" + + @abstractmethod + async def create_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str + ) -> WorkingMemoryCalibrationRunCreated: + """Validate the request, record the run and enqueue its batch as `user_id`, who must be an administrator. + + The queue takes all of the batch's items or none (`CalibrationQueueFullError`). With a retry key, a repeat of + the request returns the run it created and enqueues nothing; once a reset removed the run, it raises + `CalibrationRunNotFoundError`.""" + + @abstractmethod + def admit(self, queue_item: SessionQueueItem, run_id: str) -> Optional[str]: + """Called by the session runner before a calibration item runs: None to run it, or why it must fail.""" + + @abstractmethod + def model_reset_epochs(self, run_id: str) -> dict[str, int]: + """The reset epoch of every model reset on the run's device class; empty when the run does not exist. Read + when a node starts and passed to `record_windows`.""" + + @abstractmethod + def record_windows( + self, run_id: str, windows: list[MeasuredWindow], model_reset_epochs: Optional[Mapping[str, int]] = None + ) -> None: + """Persist one node's measured windows. Windows of a run that no longer exists are dropped, and so are those + of a model reset since `model_reset_epochs` was read (by default, at this call): the reset removed the + model's points, these among them. A reset by another process sharing the root counts as well.""" + + @abstractmethod + def observe(self, queue_item: SessionQueueItem, run_id: str, error: Optional[BaseException] = None) -> None: + """Record the item's queue status in its run, after the session runner is done with it. `error` is what the + runner raised, which fails an item the queue still has as running; a removed item counts as canceled.""" + + @abstractmethod + def get_status(self) -> WorkingMemoryCalibrationStatus: + """Every device class's runs, with derived status, point counts and environments.""" + + @abstractmethod + def reset_run(self, run_id: str) -> WorkingMemoryCalibrationResetResult: + """Remove a run and its points. Every reset keeps the ids of the runs it removed: a repeat of the request that + started one cannot start it again.""" + + @abstractmethod + def reset_model(self, model_key: str, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + """Remove a model's runs and points from one device class, the current one by default.""" + + @abstractmethod + def reset_all(self, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + """Remove every run and point of one device class, the current one by default.""" + + @abstractmethod + def export(self, device_class_id: Optional[str] = None) -> Optional[bytes]: + """The device class's store file as it is on disk, the current one by default; None when it has none.""" + + @abstractmethod + def refresh_user_fits(self) -> None: + """Bring the user fits estimates read (`backend/working_memory/user_fit.py`) up to date with the store: per + GPU generation device, the fit of its device class's windows in the current environment, leaving out the + windows of calibration runs still running. Re-reads a file only when its signature or the environment changed, + so an unchanged store whose runs all finished costs one stat per device; while a run is unfinished, also one + queue read of its unfinished items by id. The session runner calls this when it starts any queue item; + building a plan and a reset call it too. Estimates read the fits from memory.""" + + @abstractmethod + def get_plan(self, model_key: str) -> WorkingMemoryCalibrationPlan: + """The cases that calibrate the model on this machine's GPU, or why it cannot be calibrated here. Raises + UnknownModelException for an unknown key.""" + + @abstractmethod + def get_model_summary( + self, model_key: str, device_class_id: Optional[str] = None + ) -> WorkingMemoryCalibrationModelSummary: + """What the windows that feed the model's lines measured on one device class, the current one by default: per + line and environment, the shipped and the user estimate at each size, whether the user line applies, and which + models measured it; and the model's runs. Raises UnknownModelException for an unknown key.""" diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_common.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_common.py new file mode 100644 index 00000000000..b5792a5c0ae --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_common.py @@ -0,0 +1,495 @@ +"""Working-memory calibration runs: what is stored per device class, and what the API returns about it. + +A calibration run is a batch an administrator enqueues to measure one model's working memory. Its queue items carry the +reserved origin `CALIBRATION_ORIGIN_PREFIX` + run id; the session runner measures every node of such an item with the +working-memory probe and hands the windows to the calibration service, which keeps them in one JSON file per device +class (`CalibrationStoreFile`). The service fits the windows of the current environment into user lines that estimates +read (`backend/working_memory/user_fit.py`), plans the cases a client runs to calibrate a model, and summarises what a +model's runs measured. +""" + +import datetime +from typing import Final, Literal, Optional + +from pydantic import BaseModel, ConfigDict, Field, SerializerFunctionWrapHandler, model_serializer, model_validator + +from invokeai.app.services.session_queue.session_queue_common import QUEUE_ITEM_STATUS, BatchDataCollection +from invokeai.app.services.shared.graph import Graph +from invokeai.backend.working_memory.counters import Backend +from invokeai.backend.working_memory.fingerprint import DeviceClass, Environment +from invokeai.backend.working_memory.probe import MeasuredWindow + +CALIBRATION_ORIGIN_PREFIX: Final = "invokeai:wm-calibration:" +"""Queue items whose origin starts with this are calibration items. Only the calibration service may enqueue them.""" +CALIBRATION_DESTINATION: Final = "invokeai:wm-calibration" +"""The destination of calibration items: not a gallery or canvas destination, so no client routes their results.""" +STORE_SCHEMA_VERSION: Final = 1 + + +def calibration_origin(run_id: str) -> str: + return CALIBRATION_ORIGIN_PREFIX + run_id + + +def calibration_run_id(origin: Optional[str]) -> Optional[str]: + """The run id a queue item's origin names, or None for an item that is not a calibration item.""" + if origin is None or not origin.startswith(CALIBRATION_ORIGIN_PREFIX): + return None + return origin[len(CALIBRATION_ORIGIN_PREFIX) :] + + +def is_reserved_origin(origin: Optional[str]) -> bool: + """Whether a client-supplied origin claims the calibration prefix. Case-insensitive, because the queue's origin + filters are: a look-alike origin would otherwise be counted among a run's items.""" + return origin is not None and origin.casefold().startswith(CALIBRATION_ORIGIN_PREFIX) + + +# region Store +# +# One policy for every store model: unknown fields make the file unusable (`extra="forbid"`). Ignoring them instead +# would drop them on the next rewrite, destroying data a newer writer put there. + + +class _StoreModel(BaseModel): + model_config = ConfigDict(extra="forbid") + + +class CalibrationDeviceClass(_StoreModel): + """A `fingerprint.DeviceClass` with its id as a plain field, as it is stored and returned.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + id: str = Field(description="A short stable hash of the other fields") + vendor: str + name: str + total_mib: int + backend: Backend + arch: Optional[str] + + @classmethod + def of(cls, device_class: DeviceClass) -> "CalibrationDeviceClass": + return cls(id=device_class.id, **device_class.model_dump(exclude={"id"})) + + @model_validator(mode="after") + def _id_matches(self) -> "CalibrationDeviceClass": + if DeviceClass(**self.model_dump(exclude={"id"})).id != self.id: + raise ValueError(f"Device class id {self.id} does not match its fields") + return self + + +class CalibrationEnvironment(_StoreModel): + """A `fingerprint.Environment` with its id as a plain field.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + id: str = Field(description="A short stable hash of the hard facts") + hard: dict[str, str] + soft: dict[str, Optional[str]] + + @classmethod + def of(cls, env: Environment) -> "CalibrationEnvironment": + return cls(id=env.id, hard=env.hard, soft=env.soft) + + @model_validator(mode="after") + def _id_matches(self) -> "CalibrationEnvironment": + if Environment(hard=self.hard, soft=self.soft).id != self.id: + raise ValueError(f"Environment id {self.id} does not match its hard facts") + return self + + +class CalibrationPoint(MeasuredWindow): + """One measured window, keyed to the environment it ran under. Forbids unknown fields, as `MeasuredWindow` does.""" + + env_id: str + + +class ObservedItem(_StoreModel): + """What the session runner saw of one of the run's queue items. Kept so that a run's status survives the queue + pruning its items.""" + + status: QUEUE_ITEM_STATUS + error_message: Optional[str] = None + retried_from_item_id: Optional[int] = None + observed_at: datetime.datetime + + +class CalibrationRun(_StoreModel): + run_id: str + model_key: str + model_hash: Optional[str] = Field(description="The model's hash when the run was created; provenance only") + user_id: str + created_at: datetime.datetime + env_id: str = Field(description="The environment the run was created under") + queue_item_ids: list[int] = Field(default_factory=list, description="The items the run was enqueued with") + observed: dict[int, ObservedItem] = Field( + default_factory=dict, description="The last status the session runner saw per item, retries included" + ) + attempt_id: Optional[str] = Field(default=None, description="The client's attempt the run belongs to, if any") + soft: Optional[dict[str, Optional[str]]] = Field( + default=None, + description="The soft facts of the environment the run was created under; absent in runs stored before they " + "were recorded, which fall back to the environment's", + ) + + @model_serializer(mode="wrap") + def _omit_absent_fields(self, handler: SerializerFunctionWrapHandler) -> dict: + # Written only when set: a run without them stays readable by a version that forbids the unknown fields. + data = handler(self) + for name in ("attempt_id", "soft"): + if data.get(name) is None: + data.pop(name, None) + return data + + +class CalibrationStoreFile(_StoreModel): + """The content of `/working_memory/.json`, which the export returns as it is.""" + + schema_version: Literal[1] + device_class: CalibrationDeviceClass + runs: list[CalibrationRun] = Field(default_factory=list) + points: list[CalibrationPoint] = Field(default_factory=list) + envs: list[CalibrationEnvironment] = Field(default_factory=list) + removed_run_ids: list[str] = Field( + default_factory=list, + description="The runs with a retry key that a reset removed, kept for good. A repeat of the request that " + "started one cannot start it again, so its queue items keep failing at admission", + ) + model_reset_epochs: dict[str, int] = Field( + default_factory=dict, + description="Per model key, how often the model was reset. A node's points of a model reset after the node " + "started are dropped", + ) + + +# endregion + +# region API + +ATTEMPT_ID_DESCRIPTION: Final = ( + "A client-generated id of the calibration attempt the run belongs to (1-64 letters, digits, '_', '-', '.', ':'), " + "so a client that enqueues one run per step can group them; no window of an attempt feeds an estimate until every " + "run of the attempt has finished. The run id derives from the idempotency_key, the request and the attempt, so an " + "idempotency_key names one run: repeating the request under the same attempt returns that run, and reusing the key " + "under another attempt is refused with 409 while the queue keeps the key's receipt. Make each key unique per run " + "of an attempt." +) +MEASUREMENT_METHOD_DESCRIPTION: Final = ( + "Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this " + "platform has no driver reader; null when no generation device of this class is present" +) +MEASUREMENT_VERIFIED_DESCRIPTION: Final = ( + "Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when " + "no generation device of this class is present" +) + + +class WorkingMemoryCalibrationRequest(BaseModel): + """The batch a calibration run enqueues, built by the client. The server sets origin, destination, runs and the + graph's id, and turns the node cache off.""" + + model_config = ConfigDict(extra="forbid") + + graph: Graph = Field(description="The graph to measure") + data: Optional[BatchDataCollection] = Field(default=None, description="The batch data collection") + idempotency_key: Optional[str] = Field( + default=None, + min_length=1, + max_length=255, + description="A caller-owned retry key. Repeating the request with it returns the run it created and enqueues " + "nothing new; once a reset removed that run, the repeat is refused. Reusing the key for another request " + "(another model, graph or batch data) is rejected while the queue keeps the key's enqueue receipt.", + ) + attempt_id: Optional[str] = Field( + default=None, + min_length=1, + max_length=64, + pattern=r"^[A-Za-z0-9_.:-]+$", + description=ATTEMPT_ID_DESCRIPTION, + ) + + +class WorkingMemoryCalibrationRunCreated(BaseModel): + run_id: str + device_class_id: str + item_ids: list[int] = Field( + description="The queue items the run enqueued. A repeat of the request that finds them unrecorded takes over " + "every queue item carrying the run's origin, retries included" + ) + + +CalibrationRunStatus = Literal["running", "completed", "failed", "partial", "canceled"] + + +class WorkingMemoryCalibrationRunSummary(BaseModel): + run_id: str + model_key: str + model_hash: Optional[str] + user_id: str + created_at: datetime.datetime + env_id: str + status: CalibrationRunStatus = Field( + description="Derived from the run's queue items, a retry standing in for the item it retried: running while " + "any is pending or running; failed when any failed; completed when all completed; otherwise partial when " + "valid points were recorded, else canceled" + ) + error: Optional[str] = Field(description="The error of a failed item, when the run failed") + item_ids: list[int] = Field(description="Every queue item of the run, retries included") + point_count: int + attempt_id: Optional[str] = Field(description="The client's attempt the run belongs to; null when none was sent") + + +class WorkingMemoryCalibrationPointCount(BaseModel): + estimator: str + version: int + valid: bool + fit_eligible: bool + count: int + + +class WorkingMemoryCalibrationEnvironment(BaseModel): + id: str + hard: dict[str, str] + soft: dict[str, Optional[str]] + hard_matches_current: Optional[bool] = Field( + description="Whether the current environment has the same hard facts; null off the current device class" + ) + soft_matches_current: Optional[bool] = Field( + description="Whether the current environment has the soft facts first recorded with this environment (`soft`); " + "a later run's own soft facts are not compared here. Null off the current device class" + ) + point_count: int + + +class WorkingMemoryCalibrationDeviceClass(BaseModel): + device_class_id: str + device_class: Optional[CalibrationDeviceClass] = Field(description="Null when the file is unusable") + usable: bool + problem: Optional[str] = Field(description="Why the file is unusable. It is kept as it is and never written") + is_current: bool + measurement_method: Optional[str] = Field(description=MEASUREMENT_METHOD_DESCRIPTION) + measurement_verified: Optional[bool] = Field(description=MEASUREMENT_VERIFIED_DESCRIPTION) + runs: list[WorkingMemoryCalibrationRunSummary] + point_counts: list[WorkingMemoryCalibrationPointCount] + envs: list[WorkingMemoryCalibrationEnvironment] + + +class WorkingMemoryCalibrationStatus(BaseModel): + available: bool = Field( + description="Whether a calibration run can start now: one GPU generation device whose store file is usable" + ) + unavailable_reason: Optional[str] + current_device_class_id: Optional[str] + current_env_id: Optional[str] + device_classes: list[WorkingMemoryCalibrationDeviceClass] + + +class WorkingMemoryCalibrationResetResult(BaseModel): + runs_removed: int + points_removed: int + + +class WorkingMemoryCalibrationDeviceClassRef(BaseModel): + id: str + name: str + + +class WorkingMemoryCalibrationCase(BaseModel): + """One generation a calibration plan runs, in the fields of the client's graph-export case (mode txt2img).""" + + id: str = Field(description="Unique within the plan; becomes the graph id") + mode: Literal["txt2img"] = "txt2img" + model_key: str = Field(description="The main model the case generates with") + components: dict[str, str] = Field( + description="Component picks by Generate slot key; `vae` when a standalone VAE is calibrated or a GPU VAE " + "must be picked over one that runs on the CPU" + ) + width: int + height: int + steps: int = Field(description="The fewest steps that still run the whole pipeline") + seed: int + positive_prompt: str + negative_prompt: str + settings: dict[str, str] = Field( + description="Generate settings overrides: `vaePrecision` (fp16 or fp32) where the decode node casts the VAE" + ) + estimator: str = Field(description="The working-memory estimator of the operation the case measures") + line: str = Field(description="The estimator line the case measures, `_`") + + +class WorkingMemoryCalibrationRepetitions(BaseModel): + warm_up: int = Field(description="Runs of each case before the measured ones; their windows count too") + measured: int + + +class WorkingMemoryCalibrationSharedModel(BaseModel): + key: str + name: str + base: str + type: str + lines: list[str] = Field(description="The lines it shares with the planned model, `_`") + + +class WorkingMemoryCalibrationPlan(BaseModel): + """What calibrating one model on this machine's GPU runs, or why it cannot.""" + + model_key: str + supported: bool + reason: Optional[str] = Field(description="Why the model cannot be calibrated here, when it cannot") + device_class: Optional[WorkingMemoryCalibrationDeviceClassRef] = Field( + description="The class of the GPU the cases run on; null without exactly one GPU generation device" + ) + calibrated_through: Optional[str] = Field( + description="For a standalone VAE, the installed main model the cases generate with" + ) + estimators: list[str] = Field(description="The working-memory estimators of the model's covered operations") + cases: list[WorkingMemoryCalibrationCase] + repetitions: WorkingMemoryCalibrationRepetitions + estimated_seconds: Optional[int] = Field( + description="A rough order of magnitude for all cases and repetitions, model load included; not a measurement" + ) + shared_with: list[WorkingMemoryCalibrationSharedModel] = Field( + description="Other installed models whose operations price with the same lines on this GPU, so a calibration " + "changes their estimates too" + ) + not_covered: list[str] = Field(description="Operations of the model a calibration does not measure, and why") + + +CalibrationLineState = Literal["in_use", "stale", "unusable", "insufficient"] + +CalibrationSizeExclusion = Literal["fragmented", "partial_residency"] + + +class WorkingMemoryCalibrationLineSize(BaseModel): + width: int + height: int + pixels: int + windows: int = Field( + description="Counted: the windows at this size that feed the line, valid, fit-eligible and not fragmented, of " + "every model in the environment; those of a calibration run still running are counted, and join the fit once " + "it finished. Not counted: the valid windows measured at this size" + ) + measured_bytes: int = Field(description="The largest demand of those windows") + counted: bool = Field(description="Whether windows of this size feed the user line") + excluded_reason: Optional[CalibrationSizeExclusion] = Field( + description="Why a size is not counted: fragmented, its windows measured the allocator's fragmentation " + "(`backend/working_memory/fragmentation.py`); partial_residency, the model was not fully on the device. Null " + "for a counted size" + ) + shipped_bytes: Optional[int] = Field( + description="The shipped estimate at this size, priced with the line's estimator where one of its windows " + "measured the size, otherwise with that of the largest window; null when the platform of the environment is " + "unknown" + ) + user_bytes: Optional[int] = Field( + description="The estimate with the user line applied, as estimates read it (without the windows of runs still " + "running); null without a user line" + ) + + +class WorkingMemoryCalibrationPixelRange(BaseModel): + min_pixels: int + max_pixels: int + + +class WorkingMemoryCalibrationLineContributor(BaseModel): + """A model whose windows feed a line: the model a run calibrated, or the model a window measured.""" + + model_key: str + name: Optional[str] = Field(description="The model's name; null when it is no longer installed") + windows: int = Field(description="Its counted windows on the line: valid, fit-eligible and not fragmented") + + +class WorkingMemoryCalibrationModelLine(BaseModel): + """One estimator line in one environment: one the model's windows measured, or, in the current environment, one + the model's decode prices on that other models' windows measured.""" + + estimator: str + version: int + line: str = Field(description="`_`") + carrier: Optional[str] = Field(description="The estimator whose shipped entry carries the line on this platform") + env_id: str + state: CalibrationLineState = Field( + description="in_use: the user line applies; stale: it applies, but the soft facts of the newest finished run " + "that contributed a window differ from the current ones; unusable: hard facts " + "differ from the current environment (or no GPU of this class generates here), so shipped values apply; " + "insufficient: the windows of finished runs cannot fit a line yet" + ) + changed_facts: list[str] = Field( + description="The soft facts that changed (stale) or the hard facts that differ (unusable)" + ) + problem: Optional[str] = Field(description="Why no line could be fitted, when it could not") + measured_range: Optional[WorkingMemoryCalibrationPixelRange] = Field( + description="The sizes the user line was fitted on, over every model's windows of the line from finished runs" + ) + slope_bytes_per_pixel: Optional[int] + base_bytes: Optional[int] + sizes: list[WorkingMemoryCalibrationLineSize] + own_windows: int = Field(description="The model's own windows that feed the line; 0 when only other models' do") + measured_by: list[WorkingMemoryCalibrationLineContributor] = Field( + description="Every model whose windows feed the line, this one included when it has any; by name, then key" + ) + + +class WorkingMemoryCalibrationModelRun(BaseModel): + run_id: str + status: CalibrationRunStatus + error: Optional[str] + created_at: datetime.datetime + point_count: int + attempt_id: Optional[str] = Field(description="The client's attempt the run belongs to; null when none was sent") + + +class WorkingMemoryCalibrationModelSummary(BaseModel): + """What one model's calibration runs measured on one device class, and what estimates make of it.""" + + model_key: str + device_class_id: str + usable: bool + problem: Optional[str] = Field(description="Why the store file is unusable") + current_env_id: Optional[str] = Field(description="Null when no generation device of this class is present") + measurement_method: Optional[str] = Field(description=MEASUREMENT_METHOD_DESCRIPTION) + measurement_verified: Optional[bool] = Field(description=MEASUREMENT_VERIFIED_DESCRIPTION) + lines: list[WorkingMemoryCalibrationModelLine] + excluded_windows: int = Field( + description="The model's windows that feed no line: invalid, not fit-eligible, fragmented, tiled, batched, " + "upcast, a VAE outside the AutoencoderKL family, or no demand. The valid ones among them still show as sizes " + "that are not counted" + ) + runs: list[WorkingMemoryCalibrationModelRun] + + +# endregion + +# region Errors + + +class CalibrationUnavailableError(Exception): + """This machine cannot calibrate now: no GPU, or more than one generation device.""" + + +class CalibrationGraphError(ValueError): + """The graph cannot be measured as a calibration run.""" + + +class CalibrationRunNotFoundError(KeyError): + def __init__(self, run_id: str, reason: str = "does not exist") -> None: + super().__init__(run_id) + self.run_id = run_id + self.reason = reason + + def __str__(self) -> str: + return f"Calibration run {self.run_id} {self.reason}" + + +class CalibrationQueueFullError(Exception): + """The queue had no room for all of the run's items, so it took none.""" + + +class CalibrationStoreError(Exception): + """The store could not be read or written.""" + + +class CalibrationStoreUnusableError(CalibrationStoreError): + """The device class's file has an unknown schema or is invalid; it is kept untouched and not written.""" + + +# endregion diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_default.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_default.py new file mode 100644 index 00000000000..fc5425a77f2 --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_default.py @@ -0,0 +1,1261 @@ +import asyncio +import datetime +import hashlib +import json +import os +import threading +import uuid +import weakref +from collections.abc import Callable, Mapping +from dataclasses import dataclass, replace +from functools import partial +from itertools import chain +from typing import TYPE_CHECKING, Final, Optional + +import torch + +from invokeai.app.invocations.call_saved_workflow import CallSavedWorkflowInvocation +from invokeai.app.services.model_records import UnknownModelException +from invokeai.app.services.session_queue.session_queue_common import ( + DEFAULT_QUEUE_ID, + QUEUE_ITEM_STATUS, + Batch, + QueueItemLineage, + SessionQueueItem, + SessionQueueItemNotFoundError, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_base import ( + WorkingMemoryCalibrationServiceBase, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CALIBRATION_DESTINATION, + CALIBRATION_ORIGIN_PREFIX, + STORE_SCHEMA_VERSION, + CalibrationDeviceClass, + CalibrationEnvironment, + CalibrationGraphError, + CalibrationPoint, + CalibrationQueueFullError, + CalibrationRun, + CalibrationRunNotFoundError, + CalibrationRunStatus, + CalibrationStoreError, + CalibrationStoreFile, + CalibrationStoreUnusableError, + CalibrationUnavailableError, + ObservedItem, + WorkingMemoryCalibrationDeviceClass, + WorkingMemoryCalibrationEnvironment, + WorkingMemoryCalibrationModelRun, + WorkingMemoryCalibrationModelSummary, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationPointCount, + WorkingMemoryCalibrationRequest, + WorkingMemoryCalibrationResetResult, + WorkingMemoryCalibrationRunCreated, + WorkingMemoryCalibrationRunSummary, + WorkingMemoryCalibrationStatus, + calibration_origin, + calibration_run_id, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_fits import ( + measured_models, + model_lines, + recorded_outside_torch_bytes, + run_models, + user_fit_of, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_plan import ( + build_plan, + decode_lines_of, +) +from invokeai.app.services.working_memory_calibration.working_memory_calibration_store import ( + CalibrationStore, + StoreRead, +) +from invokeai.app.util.misc import uuid_string +from invokeai.backend.util.devices import TorchDevice +from invokeai.backend.util.vae_working_memory import estimate_untiled_vae_decode +from invokeai.backend.working_memory import fingerprint, user_fit +from invokeai.backend.working_memory.counters import Backend, backend_for, driver_method, method_verified, read_counters +from invokeai.backend.working_memory.fingerprint import DeviceClass, Environment +from invokeai.backend.working_memory.platform import Platform, platform_for, platform_for_environment +from invokeai.backend.working_memory.probe import MeasuredWindow +from invokeai.backend.working_memory.spec import EstimatorSpec, import_estimator_modules +from invokeai.backend.working_memory.user_fit import UserFit + +if TYPE_CHECKING: + from invokeai.app.services.invocation_services import InvocationServices + from invokeai.app.services.invoker import Invoker + +MEMORY_SETTINGS: Final = ( + "pytorch_cuda_alloc_conf", + "attention_backend", + "attention_type", + "attention_slice_size", + "sequential_guidance", + "pid_memory_optimization", + "wan_memory_optimization", + "rocm_aotriton_experimental", +) +"""The InvokeAI settings that change how much working memory an operation takes; hard facts of a measurement's +environment.""" + +_SUMMARY_CHUNK: Final = 500 +"""Queue items read per summary query when `_live_runs` asks about the unfinished items of the runs.""" + +_RUN_ID_NAMESPACE: Final = uuid.UUID("3c1f6a52-8d0e-4b6f-9a57-2f0d5c8e41b3") +"""Derives a run id from an owner, a retry key and the request's fingerprint.""" + +_RUNNING: Final[frozenset[QUEUE_ITEM_STATUS]] = frozenset({"pending", "in_progress", "waiting"}) + +_RESET: Final = "was reset; a repeat of the request that started it cannot start it again" + +SettingValue = str | int | float | bool | None + + +def _is_keyed(run_id: str) -> bool: + """Whether a retry key named the run: its id is then derived (a UUID 5, see `_identify`), otherwise fresh + (`uuid_string`, a UUID 4) and never repeated.""" + try: + return uuid.UUID(run_id).version == 5 + except ValueError: + return False + + +def _now() -> datetime.datetime: + return datetime.datetime.now(datetime.timezone.utc) + + +def _untiled_decode_reservation(estimator: str, height: int, width: int, dtype: str, device: torch.device) -> int: + return estimate_untiled_vae_decode(estimator, height, width, dtype, device).reservation_bytes + + +@dataclass(frozen=True) +class DeviceMemory: + """A calibration device's memory as the plan sizes for it, read now.""" + + process_bytes: int + """What this process can hold, models included, as `ModelCache._get_vram_available` counts it before its + reserves: what the driver has free plus what torch has allocated, without the cache's credit for the allocator's + reclaimable blocks.""" + outside_torch_bytes: Optional[int] + """What this process holds outside torch's allocator: the driver's figure for it less torch's reserve. None where + the driver has no figure for this process.""" + + +def _device_memory(device: torch.device) -> Optional[DeviceMemory]: + """`DeviceMemory` for `device`; None for a device that shares system memory (MPS).""" + if device.type == "cuda": + free, _ = TorchDevice.cuda_mem_get_info(device) + process_bytes = free + torch.cuda.memory_allocated(device) + elif device.type == "xpu": + free, _ = TorchDevice.xpu_mem_get_info(device) + process_bytes = free + torch.xpu.memory_allocated(device) + else: + return None + reading = read_counters(device) + outside = None if reading.driver is None or reading.reserved is None else reading.driver - reading.reserved + return DeviceMemory(process_bytes=process_bytes, outside_torch_bytes=outside) + + +def _measurement(backend: Optional[Backend]) -> tuple[Optional[str], Optional[bool]]: + """The driver reader a window of a `backend` GPU here uses, and whether it is verified; both None for a device + class no generation device here belongs to.""" + if backend is None: + return None, None + return driver_method(backend), method_verified(backend) + + +def _device_key(device: torch.device) -> str: + """One name per device, whether or not it carries its index: the cache's devices may come without one.""" + if device.index is not None or device.type == "cpu": + return str(device) + index = torch.cuda.current_device() if device.type == "cuda" and torch.cuda.is_available() else 0 + return f"{device.type}:{index}" + + +FileSignature = Optional[tuple[int, int, int]] +"""A store file's mtime, size and inode (file index on Windows), None when it has none. Every write replaces the file, +so a change shows in at least one of them.""" + + +@dataclass(frozen=True) +class _DeviceFit: + """What `refresh_user_fits` found for one generation device, and what it found it from.""" + + signature: FileSignature + env_id: str + platform: Platform + fit: Optional[UserFit] + file: Optional[CalibrationStoreFile] + """The file the fit was made from, kept to re-fit it when only the runs still running changed.""" + live_runs: frozenset[str] + """The runs whose windows the fit left out, because they were still running.""" + settled_items: frozenset[int] + """Calibration queue items found finished in the queue, which `_live_runs` asks about no more.""" + + +def _add_env(file: CalibrationStoreFile, env: Environment) -> None: + if all(known.id != env.id for known in file.envs): + file.envs.append(CalibrationEnvironment.of(env)) + + +def _prune_envs(file: CalibrationStoreFile) -> None: + used = {run.env_id for run in file.runs} | {point.env_id for point in file.points} + file.envs = [env for env in file.envs if env.id in used] + + +def _add_points( + file: CalibrationStoreFile, + *, + run_id: str, + env: Environment, + points: list[CalibrationPoint], + epochs: Mapping[str, int], +) -> tuple[bool, int]: + """Attach `points` to their run, except those attributed (`measured_models`, as a model reset removes them) to a + model whose reset epoch moved past `epochs`: a reset removed the model's points after the points were measured. + How many it attached; none when the file lacks the run.""" + if all(run.run_id != run_id for run in file.runs): + return False, 0 + current = file.model_reset_epochs + models = run_models(file) + kept = [ + point + for point in points + if all(current.get(model, 0) <= epochs.get(model, 0) for model in measured_models(point, models)) + ] + if not kept: + return False, 0 + _add_env(file, env) + file.points.extend(kept) + return True, len(kept) + + +def derive_run_status( + run: CalibrationRun, live: Mapping[int, QueueItemLineage], valid_point_count: int +) -> tuple[CalibrationRunStatus, Optional[str], list[int]]: + """A run's status, error and items, from its items: `live` as the queue has them now, the run's own observations + for items the queue no longer has. An item the queue no longer has counts as canceled unless the runner saw it + finish. A retry stands in for the item it retried, also before the runner started it. A run that ended without + completing is partial only with valid points: an invalid one (a canceled or failed operation, a spill) measured + nothing usable.""" + item_ids = sorted(set(run.queue_item_ids) | set(run.observed) | set(live)) + + def status_of(item_id: int) -> QUEUE_ITEM_STATUS: + if item_id in live: + return live[item_id].status + observed = run.observed.get(item_id) + # Gone from the queue while the runner last saw it running (a crash, a cleared queue): it will not finish. + if observed is None or observed.status in _RUNNING: + return "canceled" + return observed.status + + def retried_from(item_id: int) -> Optional[int]: + if item_id in live: + return live[item_id].retried_from_item_id + observed = run.observed.get(item_id) + return observed.retried_from_item_id if observed is not None else None + + latest: dict[int, int] = {} + for item_id in item_ids: + source = retried_from(item_id) + lineage = item_id if source is None else source + latest[lineage] = max(latest.get(lineage, item_id), item_id) + effective = sorted(latest.values()) + statuses = {item_id: status_of(item_id) for item_id in effective} + + if not effective: + return "canceled", None, item_ids + if any(status in _RUNNING for status in statuses.values()): + return "running", None, item_ids + failed = [item_id for item_id, status in statuses.items() if status == "failed"] + if failed: + observed = run.observed.get(failed[0]) + error = observed.error_message if observed is not None and observed.error_message else None + return "failed", error or f"Queue item {failed[0]} failed", item_ids + if all(status == "completed" for status in statuses.values()): + return "completed", None, item_ids + return ("partial" if valid_point_count > 0 else "canceled"), None, item_ids + + +class WorkingMemoryCalibrationService(WorkingMemoryCalibrationServiceBase): + """Calibration runs and their store. + + `generation_devices`, `device_class_for`, `environment_for`, `untiled_decode_reservation` and `device_memory` + default to the machine's own; tests pass their own because a device class and its memory can only be read from + real hardware. + """ + + def __init__( + self, + store: CalibrationStore, + *, + generation_devices: Optional[Callable[[], list[torch.device]]] = None, + device_class_for: Callable[[torch.device], DeviceClass] = fingerprint.device_class_for, + environment_for: Callable[ + [torch.device, Mapping[str, SettingValue]], Environment + ] = fingerprint.environment_for, + untiled_decode_reservation: Callable[[str, int, int, str, torch.device], int] = _untiled_decode_reservation, + device_memory: Callable[[torch.device], Optional[DeviceMemory]] = _device_memory, + ) -> None: + self._store = store + self._generation_devices = generation_devices + self._device_class_for = device_class_for + self._environment_for = environment_for + self._untiled_decode_reservation = untiled_decode_reservation + self._device_memory = device_memory + # A card's class cannot change while the process runs; reading it touches the driver. + self._device_classes: dict[str, DeviceClass] = {} + # Which file holds a run, so admitting an item and recording its windows do not scan every file. + self._run_files: dict[str, str] = {} + # Creations in flight, referenced here so that one whose request was canceled still runs to its end. + self._creations: set[asyncio.Task[WorkingMemoryCalibrationRunCreated]] = set() + # Per run id while a creation uses it: a repeat of a request waits for the attempt still in flight. + self._run_locks: weakref.WeakValueDictionary[str, asyncio.Lock] = weakref.WeakValueDictionary() + self._invoker: Optional["Invoker"] = None + # The user fit per generation device: replaced whole by `refresh_user_fits`, read by every estimate. + self._fits: Mapping[str, _DeviceFit] = {} + self._fits_lock = threading.Lock() + # One object, so that `stop` unregisters exactly what `start` registered. + self._provider: user_fit.UserFitProvider = self._user_fit_for + + def start(self, invoker: "Invoker") -> None: + self._invoker = invoker + user_fit.set_user_fit_provider(self._provider) + + def stop(self, invoker: "Invoker") -> None: + user_fit.clear_user_fit_provider(self._provider) + + @property + def _services(self) -> "InvocationServices": + assert self._invoker is not None, "The calibration service has not been started" + return self._invoker.services + + # region Devices and environments + + def _devices(self) -> list[torch.device]: + if self._generation_devices is not None: + return list(self._generation_devices()) + configured = TorchDevice.get_generation_devices(self._services.configuration.generation_devices) + return configured or [TorchDevice.choose_torch_device()] + + def _calibration_device(self) -> torch.device: + devices = self._devices() + if len(devices) > 1: + raise CalibrationUnavailableError( + f"Calibration runs on a single generation device, and this machine generates on {len(devices)} " + f"({', '.join(str(device) for device in devices)}). Set generation_devices to one GPU to calibrate it." + ) + if not devices or backend_for(devices[0]) is None: + raise CalibrationUnavailableError( + f"Calibration measures GPU memory, and this machine generates on {devices[0] if devices else 'no device'}." + ) + return devices[0] + + def _device_class(self, device: torch.device) -> DeviceClass: + key = str(device) + if key not in self._device_classes: + self._device_classes[key] = self._device_class_for(device) + return self._device_classes[key] + + def _environment(self, device: torch.device) -> Environment: + config = self._services.configuration + settings: dict[str, SettingValue] = {name: getattr(config, name) for name in MEMORY_SETTINGS} + return self._environment_for(device, settings) + + def _current_device_class_id(self, device_class_id: Optional[str]) -> str: + if device_class_id is not None: + return device_class_id + return self._device_class(self._calibration_device()).id + + # endregion + + # region Runs + + def _find_run(self, run_id: str) -> Optional[tuple[str, CalibrationRun]]: + cached = self._run_files.get(run_id) + candidates = [cached] if cached is not None else [] + candidates += [ + device_class_id for device_class_id in self._store.device_class_ids() if device_class_id != cached + ] + for device_class_id in candidates: + read = self._store.read(device_class_id) + if read is None or read.file is None: + continue + for run in read.file.runs: + if run.run_id == run_id: + self._run_files[run_id] = device_class_id + return device_class_id, run + self._run_files.pop(run_id, None) + return None + + @staticmethod + def _identify(model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str) -> str: + """Check the request's graph and turn its node cache off, then return the run's id, which also becomes the + graph's id. + + Without a retry key the id is fresh. With one it is derived from the owner, the key, the request's + fingerprint (the model and the batch the server builds: graph and batch data) and the client's attempt, if it + names one. A repeat of the request finds its run by that id, and another request reusing the key, another + attempt included, gets another id. The id is part of the batch's origin, so the queue's receipt for the key + refuses that other request. The client's graph id is not part of the request: a graph sent without one gets a + fresh id each time, so a repeat could never match. + """ + graph = request.graph + for node in graph.nodes.values(): + if isinstance(node, CallSavedWorkflowInvocation): + raise CalibrationGraphError( + f"Node {node.id} calls a saved workflow. A calibration graph cannot: the called workflow would " + "run as separate queue items, outside the measured node." + ) + for datum in chain.from_iterable(request.data or []): + if datum.field_name == "use_cache": + raise CalibrationGraphError( + f"Batch data sets use_cache on node {datum.node_path}; calibration runs every node uncached." + ) + # A cached output would skip the very operation the run measures. + for node in graph.nodes.values(): + node.use_cache = False + if request.idempotency_key is None: + run_id = uuid_string() + else: + fingerprint = { + "model_key": model_key, + "graph": graph.model_dump(mode="json", exclude={"id"}), + "data": None + if request.data is None + else [[datum.model_dump(mode="json") for datum in group] for group in request.data], + } + digest = hashlib.sha256( + json.dumps(fingerprint, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8") + ).hexdigest() + # Appended only with an attempt: a request without one keeps the id it had before attempts existed. + attempt = f"\x00{request.attempt_id}" if request.attempt_id is not None else "" + run_id = str(uuid.uuid5(_RUN_ID_NAMESPACE, f"{user_id}\x00{request.idempotency_key}\x00{digest}{attempt}")) + # The queue's receipt hashes the batch, graph id included: a repeat must build the same one. + graph.id = run_id + return run_id + + def _run_lock(self, run_id: str) -> asyncio.Lock: + lock = self._run_locks.get(run_id) + if lock is None: + lock = self._run_locks[run_id] = asyncio.Lock() + return lock + + async def create_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str + ) -> WorkingMemoryCalibrationRunCreated: + # The enqueue runs in a worker thread and can commit after the request was canceled. So the creation runs to + # its end whatever happens to the request: the run keeps the items that were queued, or is removed. Only the + # server's shutdown cancels the creation as well; the run stays as far as it got, for a keyed repeat to finish. + creation = asyncio.create_task(self._create_run(model_key, request, user_id)) + self._creations.add(creation) + creation.add_done_callback(self._creations.discard) + try: + return await asyncio.shield(creation) + except asyncio.CancelledError: + creation.add_done_callback(self._log_unawaited_failure) + raise + + def _log_unawaited_failure(self, creation: "asyncio.Task[WorkingMemoryCalibrationRunCreated]") -> None: + """Nobody awaits the creation of a canceled request: its failure would go unseen.""" + if not creation.cancelled() and (error := creation.exception()) is not None: + self._services.logger.warning( + "The creation of a calibration run whose request was canceled failed", exc_info=error + ) + + async def _create_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str + ) -> WorkingMemoryCalibrationRunCreated: + run_id = await asyncio.to_thread(self._identify, model_key, request, user_id) + # Recording the run, enqueuing its batch and recording its items or removing it: one step for each run id. The + # lock is this process's; two processes serving one root and its SQLite database are not kept apart. + async with self._run_lock(run_id): + prepared = await asyncio.to_thread(self._prepare_run, model_key, request, user_id, run_id) + if isinstance(prepared, WorkingMemoryCalibrationRunCreated): + return prepared + batch, device_class_id, created = prepared + try: + result = await self._services.session_queue.enqueue_batch( + queue_id=DEFAULT_QUEUE_ID, batch=batch, prepend=False, user_id=user_id, all_or_none=True + ) + except Exception: + # The queue announces a batch after committing it, so an enqueue can raise once its items are queued. + # Items of an earlier attempt that a reset left behind carry the run's origin too, and answer a repeat + # of the request as the key's receipt would. + try: + queued: Optional[list[int]] = await asyncio.to_thread(self._queued_item_ids, run_id) + except Exception: + self._services.logger.warning( + f"Could not tell whether calibration run {run_id} has queue items; it is kept", exc_info=True + ) + queued = None + if queued is None: + # The enqueue's own error answers the request. + raise + if not queued: + if created: + await asyncio.to_thread(self._discard, run_id) + raise + self._services.logger.warning( + f"Enqueuing calibration run {run_id} raised, and the queue holds its items {queued}", + exc_info=True, + ) + item_ids = queued + else: + if result.enqueued == 0: + if created: + await asyncio.to_thread(self._discard, run_id) + # A partial run would measure some of the batch's cases, so the queue takes all of them or none. + raise CalibrationQueueFullError( + f"The queue is full: it has no room for all {result.requested} items of the calibration run, " + "so the run was not started." + ) + item_ids = list(result.item_ids) + try: + await asyncio.to_thread(self._record_items, device_class_id, run_id, item_ids) + except CalibrationStoreError as error: + # An unusable file fails the run's items at admission, as a removed run (not caught here) does: an error + # with a key or without. With a key, a repeat records the items, from the key's receipt or, once the + # queue has collected that, from their origin. Without one the request cannot be repeated, only sent + # again, which would queue the batch twice: a passing store failure answers with the items. + if isinstance(error, CalibrationStoreUnusableError) or request.idempotency_key is not None: + raise + self._services.logger.warning( + f"Could not record the queue items {item_ids} of calibration run {run_id}; they run all the same, " + "and the status finds them by their origin", + exc_info=True, + ) + return WorkingMemoryCalibrationRunCreated(run_id=run_id, device_class_id=device_class_id, item_ids=item_ids) + + def _prepare_run( + self, model_key: str, request: WorkingMemoryCalibrationRequest, user_id: str, run_id: str + ) -> WorkingMemoryCalibrationRunCreated | tuple[Batch, str, bool]: + """The run a repeated request already enqueued; or the batch to enqueue, the run's device class and whether + this call recorded the run.""" + key = request.idempotency_key + found: Optional[tuple[str, CalibrationRun]] = None + if key is not None and (found := self._find_run(run_id)) is not None: + device_class_id, run = found + if run.queue_item_ids: + return WorkingMemoryCalibrationRunCreated( + run_id=run_id, device_class_id=device_class_id, item_ids=list(run.queue_item_ids) + ) + # An earlier attempt may have queued the batch without recording its items. While the queue keeps the + # key's receipt, enqueuing the batch again answers with the items it names, all of them. That needs + # neither the model, which may have been removed since, nor the device. Without the receipt, enqueuing + # again would queue the batch a second time: the items carrying the run's origin, and those the runner + # saw, are taken over instead. + if self._services.session_queue.get_enqueue_receipt(DEFAULT_QUEUE_ID, key, user_id) is not None: + return self._batch(request, run_id), device_class_id, False + item_ids = sorted(set(self._queued_item_ids(run_id)) | set(run.observed)) + if item_ids: + self._record_items(device_class_id, run_id, item_ids) + return WorkingMemoryCalibrationRunCreated( + run_id=run_id, device_class_id=device_class_id, item_ids=item_ids + ) + elif key is not None and self._was_reset(run_id): + raise CalibrationRunNotFoundError(run_id, _RESET) + device = self._calibration_device() + # Raises UnknownModelException for an unknown key. + config = self._services.model_manager.store.get_model(model_key) + batch = self._batch(request, run_id) + if found is not None: + # The run is recorded without items, and nothing of it is queued: enqueue its batch. + return batch, found[0], False + device_class = self._device_class(device) + env = self._environment(device) + run = CalibrationRun( + run_id=run_id, + model_key=model_key, + model_hash=getattr(config, "hash", None), + user_id=user_id, + created_at=_now(), + env_id=env.id, + attempt_id=request.attempt_id, + soft=env.soft, + ) + + def add(file: CalibrationStoreFile) -> tuple[bool, bool]: + if run_id in file.removed_run_ids: + raise CalibrationRunNotFoundError(run_id, _RESET) + if any(known.run_id == run_id for known in file.runs): + return False, False + file.runs.append(run) + _add_env(file, env) + return True, True + + created = self._store.change(device_class.id, add, create=CalibrationDeviceClass.of(device_class)) + self._run_files[run_id] = device_class.id + return batch, device_class.id, bool(created) + + @staticmethod + def _batch(request: WorkingMemoryCalibrationRequest, run_id: str) -> Batch: + try: + return Batch( + graph=request.graph, + data=request.data, + runs=1, + origin=calibration_origin(run_id), + destination=CALIBRATION_DESTINATION, + idempotency_key=request.idempotency_key, + ) + except ValueError as error: + raise CalibrationGraphError(str(error)) from error + + def _was_reset(self, run_id: str) -> bool: + for device_class_id in self._store.device_class_ids(): + read = self._store.read(device_class_id) + if read is not None and read.file is not None and run_id in read.file.removed_run_ids: + return True + return False + + def _queued_item_ids(self, run_id: str) -> list[int]: + """The queue items carrying the run's origin: its batch's, with their retries and workflow-call children.""" + queue = self._services.session_queue + return sorted(queue.get_queue_item_ids(DEFAULT_QUEUE_ID, origin_prefix=calibration_origin(run_id)).item_ids) + + def _discard(self, run_id: str) -> None: + """Remove a run this request recorded and queued nothing for. Its key stays unused: a repeat may start it.""" + try: + self._remove_run(run_id, remember=False) + except Exception: + self._services.logger.warning(f"Could not discard calibration run {run_id}", exc_info=True) + + def _record_items(self, device_class_id: str, run_id: str, item_ids: list[int]) -> None: + def remember(file: CalibrationStoreFile) -> tuple[bool, bool]: + for run in file.runs: + if run.run_id == run_id: + changed = run.queue_item_ids != item_ids + run.queue_item_ids = list(item_ids) + return changed, True + return False, False + + if not self._store.change(device_class_id, remember): + # Reset while it was being created: its items fail at admission. + raise CalibrationRunNotFoundError(run_id) + + def admit(self, queue_item: SessionQueueItem, run_id: str) -> Optional[str]: + if self._services.configuration.multiuser: + owner = self._services.users.get(queue_item.user_id) + if owner is None or not owner.is_active or not owner.is_admin: + return ( + f"Calibration runs are for administrators only, and the owner of queue item {queue_item.item_id} " + "is not an active administrator." + ) + try: + device = self._calibration_device() + except CalibrationUnavailableError as error: + return str(error) + found = self._find_run(run_id) + if found is None: + return f"Calibration run {run_id} does not exist; it was reset or never created." + device_class_id, _ = found + if device_class_id != self._device_class(device).id: + return ( + f"Calibration run {run_id} was created for device class {device_class_id}, and this machine's GPU " + f"is of class {self._device_class(device).id}." + ) + self._observe( + run_id, + queue_item.item_id, + ObservedItem( + status="in_progress", retried_from_item_id=queue_item.retried_from_item_id, observed_at=_now() + ), + ) + return None + + def _observe(self, run_id: str, item_id: int, observation: ObservedItem) -> None: + found = self._find_run(run_id) + if found is None: + return + + def record(file: CalibrationStoreFile) -> tuple[bool, None]: + for run in file.runs: + if run.run_id == run_id: + previous = run.observed.get(item_id) + if observation.retried_from_item_id is None and previous is not None: + observation_ = observation.model_copy( + update={"retried_from_item_id": previous.retried_from_item_id} + ) + else: + observation_ = observation + run.observed[item_id] = observation_ + return True, None + return False, None + + self._store.change(found[0], record) + + def observe(self, queue_item: SessionQueueItem, run_id: str, error: Optional[BaseException] = None) -> None: + try: + current = self._services.session_queue.get_queue_item(queue_item.item_id) + except SessionQueueItemNotFoundError: + # Removed while it ran (Clear Queue, a deleted item): it will not finish. + self._observe( + run_id, + queue_item.item_id, + ObservedItem( + status="canceled", + error_message="The queue item was removed from the queue.", + retried_from_item_id=queue_item.retried_from_item_id, + observed_at=_now(), + ), + ) + return + status, error_message = current.status, current.error_message + if error is not None and status in _RUNNING: + # The processor fails the item after the runner raised; recorded here, a later prune cannot lose it. + status, error_message = "failed", f"{type(error).__name__}: {error}" + self._observe( + run_id, + current.item_id, + ObservedItem( + status=status, + error_message=error_message, + retried_from_item_id=current.retried_from_item_id, + observed_at=_now(), + ), + ) + + def model_reset_epochs(self, run_id: str) -> dict[str, int]: + found = self._find_run(run_id) + if found is None: + return {} + read = self._store.read(found[0]) + return dict(read.file.model_reset_epochs) if read is not None and read.file is not None else {} + + def record_windows( + self, run_id: str, windows: list[MeasuredWindow], model_reset_epochs: Optional[Mapping[str, int]] = None + ) -> None: + if model_reset_epochs is None: + model_reset_epochs = self.model_reset_epochs(run_id) + found = self._find_run(run_id) + if found is None: + self._services.logger.warning( + f"Dropped {len(windows)} working-memory measurements of calibration run {run_id}: the run was reset." + ) + return + run_class_id = found[0] + by_device: dict[str, list[MeasuredWindow]] = {} + for window in windows: + by_device.setdefault(window.device, []).append(window) + for device_name, group in by_device.items(): + device = torch.device(device_name) + device_class = self._device_class(device) + if device_class.id != run_class_id: + # Admission made the run's device class the generation device's; a window elsewhere is a model the + # node ran on another device, which this run does not calibrate. + self._services.logger.warning( + f"Dropped {len(group)} working-memory measurements of calibration run {run_id}: they ran on " + f"{device_name} (device class {device_class.id}), not on the run's device class {run_class_id}." + ) + continue + env = self._environment(device) + points = [CalibrationPoint(**window.model_dump(), env_id=env.id) for window in group] + add = partial(_add_points, run_id=run_id, env=env, points=points, epochs=model_reset_epochs) + dropped = len(points) - (self._store.change(run_class_id, add) or 0) + if dropped: + self._services.logger.warning( + f"Dropped {dropped} working-memory measurements of calibration run {run_id}: the run, or the " + "model they measured, was reset." + ) + + # endregion + + # region Status + + def _live_items(self) -> dict[str, dict[int, QueueItemLineage]]: + """The queue's calibration items, per run, as the queue has them now.""" + live: dict[str, dict[int, QueueItemLineage]] = {} + for item in self._services.session_queue.get_queue_item_lineage(DEFAULT_QUEUE_ID, CALIBRATION_ORIGIN_PREFIX): + # The queue's origin filter ignores case; only the exact prefix is the server's. + run_id = calibration_run_id(item.origin) + if run_id is not None: + live.setdefault(run_id, {})[item.item_id] = item + return live + + def get_status(self) -> WorkingMemoryCalibrationStatus: + current_class: Optional[DeviceClass] = None + current_env: Optional[Environment] = None + unavailable: Optional[str] = None + try: + device = self._calibration_device() + current_class = self._device_class(device) + current_env = self._environment(device) + except CalibrationUnavailableError as error: + unavailable = str(error) + + live = self._live_items() + present = self._present_backends() + device_class_ids = set(self._store.device_class_ids()) + if current_class is not None: + device_class_ids.add(current_class.id) + device_classes = [] + for device_class_id in sorted(device_class_ids): + is_current = current_class is not None and device_class_id == current_class.id + read = self._store.read(device_class_id) + if read is None: + if current_class is None or not is_current: + continue + empty = CalibrationStoreFile( + schema_version=STORE_SCHEMA_VERSION, device_class=CalibrationDeviceClass.of(current_class) + ) + read = StoreRead(device_class_id, empty, None) + if is_current and read.file is None: + # Creating a run would have to write this file, which is kept as it is. + unavailable = f"The calibration store of this GPU's device class cannot be written: {read.problem}" + device_classes.append( + self._device_class_status( + read, is_current, current_env if is_current else None, live, present.get(device_class_id) + ) + ) + return WorkingMemoryCalibrationStatus( + available=unavailable is None, + unavailable_reason=unavailable, + current_device_class_id=current_class.id if current_class is not None else None, + current_env_id=current_env.id if current_env is not None else None, + device_classes=device_classes, + ) + + @staticmethod + def _device_class_status( + read: StoreRead, + is_current: bool, + current_env: Optional[Environment], + live: Mapping[str, Mapping[int, QueueItemLineage]], + backend: Optional[Backend], + ) -> WorkingMemoryCalibrationDeviceClass: + method, verified = _measurement(backend) + file = read.file + if file is None: + return WorkingMemoryCalibrationDeviceClass( + device_class_id=read.device_class_id, + device_class=None, + usable=False, + problem=read.problem, + is_current=is_current, + measurement_method=method, + measurement_verified=verified, + runs=[], + point_counts=[], + envs=[], + ) + points_per_run: dict[str, int] = {} + valid_points_per_run: dict[str, int] = {} + points_per_env: dict[str, int] = {} + counts: dict[tuple[str, int, bool, bool], int] = {} + for point in file.points: + points_per_run[point.run_id] = points_per_run.get(point.run_id, 0) + 1 + if point.valid: + valid_points_per_run[point.run_id] = valid_points_per_run.get(point.run_id, 0) + 1 + points_per_env[point.env_id] = points_per_env.get(point.env_id, 0) + 1 + key = (point.estimator, point.version, point.valid, point.fit_eligible) + counts[key] = counts.get(key, 0) + 1 + runs = [] + for run in file.runs: + point_count = points_per_run.get(run.run_id, 0) + status, error, item_ids = derive_run_status( + run, live.get(run.run_id, {}), valid_points_per_run.get(run.run_id, 0) + ) + runs.append( + WorkingMemoryCalibrationRunSummary( + run_id=run.run_id, + model_key=run.model_key, + model_hash=run.model_hash, + user_id=run.user_id, + created_at=run.created_at, + env_id=run.env_id, + status=status, + error=error, + item_ids=item_ids, + point_count=point_count, + attempt_id=run.attempt_id, + ) + ) + envs = [ + WorkingMemoryCalibrationEnvironment( + id=env.id, + hard=env.hard, + soft=env.soft, + hard_matches_current=None if current_env is None else env.id == current_env.id, + soft_matches_current=None if current_env is None else env.soft == current_env.soft, + point_count=points_per_env.get(env.id, 0), + ) + for env in file.envs + ] + return WorkingMemoryCalibrationDeviceClass( + device_class_id=read.device_class_id, + device_class=file.device_class, + usable=True, + problem=None, + is_current=is_current, + measurement_method=method, + measurement_verified=verified, + runs=runs, + point_counts=[ + WorkingMemoryCalibrationPointCount( + estimator=estimator, version=version, valid=valid, fit_eligible=fit_eligible, count=count + ) + for (estimator, version, valid, fit_eligible), count in sorted(counts.items()) + ], + envs=envs, + ) + + # endregion + + # region User fits + + def _user_fit_for(self, spec: EstimatorSpec, platform: Platform, device: torch.device) -> Optional[UserFit]: + """What estimates ask (`user_fit.user_fit_for`): a lookup in memory, no file or device access.""" + found = self._fits.get(_device_key(device)) + if found is None or found.platform != platform: + return None + return found.fit + + @staticmethod + def _signature(path: str) -> FileSignature: + try: + stat = os.stat(path) + except FileNotFoundError: + return None + return stat.st_mtime_ns, stat.st_size, stat.st_ino + + def refresh_user_fits(self) -> None: + with self._fits_lock: + fits: dict[str, _DeviceFit] = {} + for device in self._devices(): + if backend_for(device) is None: + continue + device_class = self._device_class(device) + env = self._environment(device) + platform = platform_for(device) + signature = self._signature(str(self._store.path_for(device_class.id))) + key = _device_key(device) + known = self._fits.get(key) + settled = known.settled_items if known is not None else frozenset() + file: Optional[CalibrationStoreFile] = None + if known is not None and (known.signature, known.env_id, known.platform) == ( + signature, + env.id, + platform, + ): + if known.file is None: + fits[key] = known + continue + # The same file: only a run that finished since (in the queue, unwritten) can change the fit. + live_runs, settled = self._live_runs(known.file, settled) + if live_runs == known.live_runs: + fits[key] = replace(known, settled_items=settled) + continue + file = known.file + elif signature is not None: + read = self._store.read(device_class.id) + if read is not None and read.file is None: + self._services.logger.warning( + f"Working-memory estimates on {device} keep the shipped values: {read.problem}" + ) + file = read.file if read is not None else None + live_runs, settled = self._live_runs(file, settled) if file is not None else (frozenset(), settled) + fit = user_fit_of(file, env.id, platform, exclude_runs=live_runs)[0] if file is not None else None + fits[key] = _DeviceFit( + signature=signature, + env_id=env.id, + platform=platform, + fit=fit, + file=file, + live_runs=live_runs, + settled_items=settled, + ) + self._fits = fits + + def _live_runs( + self, file: CalibrationStoreFile, settled: frozenset[int] = frozenset() + ) -> tuple[frozenset[str], frozenset[int]]: + """The runs whose windows no fit uses yet: those still running (`derive_run_status`; a run records its items as + soon as they are queued, so one whose items all wait counts, and an item it could not record counts once the + runner admitted it), and every run of an attempt (`attempt_id`) one of whose runs still runs. And `settled` + with the items found finished. A finished queue item never runs again (a retry is a new item), so a caller + that keeps `settled` asks the queue about each item until it finished, and no more. + + Reads the queue, by id, only for the items the file does not record as finished and `settled` does not hold: + those pending or running, and those that finished without the runner (canceled while pending, cleared). Once + every run's items finished, it reads nothing. A retry the runner has not started yet is not in the file, so + it holds back nothing until it starts.""" + open_items: dict[str, list[int]] = {} + for run in file.runs: + unfinished = [ + item_id + for item_id in set(run.queue_item_ids) | set(run.observed) + if item_id not in settled + and ((observed := run.observed.get(item_id)) is None or observed.status in _RUNNING) + ] + if unfinished: + open_items[run.run_id] = unfinished + if not open_items: + return frozenset(), settled + item_ids = sorted({item_id for unfinished in open_items.values() for item_id in unfinished}) + statuses: dict[int, QUEUE_ITEM_STATUS] = {} + queue = self._services.session_queue + for start in range(0, len(item_ids), _SUMMARY_CHUNK): + chunk = item_ids[start : start + _SUMMARY_CHUNK] + for summary in queue.get_queue_item_summaries_by_ids(DEFAULT_QUEUE_ID, chunk): + statuses[summary.item_id] = summary.status + live = set() + for run in file.runs: + unfinished = open_items.get(run.run_id) + # Only "running" matters: an item left out reads as finished, and so does a settled one. + if unfinished is not None: + # The summaries carry no retry source; the runner recorded it when it admitted the item. + queued = { + item_id: QueueItemLineage( + item_id=item_id, + status=statuses[item_id], + origin=None, + retried_from_item_id=observed.retried_from_item_id + if (observed := run.observed.get(item_id)) is not None + else None, + ) + for item_id in unfinished + if item_id in statuses + } + if derive_run_status(run, queued, valid_point_count=0)[0] == "running": + live.add(run.run_id) + # A client posts each step of an attempt as its own run, all of them up front: the finished earlier steps + # must not price the later, larger ones either. + attempts = {run.attempt_id for run in file.runs if run.run_id in live and run.attempt_id is not None} + live |= {run.run_id for run in file.runs if run.attempt_id is not None and run.attempt_id in attempts} + # Gone from the queue counts as finished too: it will not run. + finished = {item_id for item_id in item_ids if statuses.get(item_id) not in _RUNNING} + return frozenset(live), settled | finished + + def _try_refresh_user_fits(self, when: str) -> None: + """Refresh the fits `when` (for the log). A store that cannot be read now must not fail what triggered it, a + reset that already happened or a plan; the fits read so far then stay until the next refresh.""" + try: + self.refresh_user_fits() + except Exception: + self._services.logger.warning(f"Could not refresh the working-memory user fits {when}", exc_info=True) + + # endregion + + # region Plan and summary + + def get_plan(self, model_key: str) -> WorkingMemoryCalibrationPlan: + store = self._services.model_manager.store + # Raises UnknownModelException for an unknown key. + target = store.get_model(model_key) + # The case sizes price the decode with the fits estimates read: the store as it is now, not as the last + # queue item found it. + self._try_refresh_user_fits("before building a calibration plan") + device: Optional[torch.device] = None + unavailable: Optional[str] = None + try: + device = self._calibration_device() + except CalibrationUnavailableError as error: + unavailable = str(error) + + def reservation(estimator: str, height: int, width: int, dtype: str) -> int: + assert device is not None + return self._untiled_decode_reservation(estimator, height, width, dtype, device) + + return build_plan( + target, + store.all_models(), + device_class=self._device_class(device) if device is not None else None, + unavailable=unavailable, + platform=platform_for(device) if device is not None else None, + precision=self._services.configuration.precision, + reservation=reservation, + specs=import_estimator_modules(), + capacity_bytes=self._decode_capacity(device) if device is not None else None, + ) + + def _decode_capacity(self, device: torch.device) -> Optional[int]: + """What the model cache can give one decode and its VAE on `device` once it offloaded every other model: this + process's share of the device now, less `reserve_vram_gb`, and within `max_cache_vram_gb` when that is set, + as `ModelCache._get_vram_available` budgets a lock. + + This process's memory outside torch's allocator counts at the larger of what it is now and the most a stored + window of this environment found it at (`recorded_outside_torch_bytes`): it grows with the first generations + (on ROCm under Windows, HIP's retained pool and MIOpen's workspaces, from about 0.3 GB in a fresh server to + 2.6 GB on an RX 9060 XT), so a fresh server's reading alone overstates what a decode will find. Without a stored + window, a first calibration sizes for the reading alone. Where the driver gives no figure now, the recorded one + counts whole. The store is the one `refresh_user_fits` read last; nothing here reads it again.""" + memory = self._device_memory(device) + if memory is None: + return None + held = memory.process_bytes + known = self._fits.get(_device_key(device)) + if known is not None and known.file is not None: + recorded = recorded_outside_torch_bytes(known.file, known.env_id) + if recorded is not None: + held -= max(0, recorded - (memory.outside_torch_bytes or 0)) + config = self._services.configuration + if config.max_cache_vram_gb is not None: + held = min(held, int(config.max_cache_vram_gb * 2**30)) + return held - int(config.reserve_vram_gb * 2**30) + + def _present_backends(self) -> dict[str, Backend]: + """The backend of each device class a GPU generation device here belongs to, by device class id.""" + present: dict[str, Backend] = {} + for device in self._devices(): + backend = backend_for(device) + if backend is not None: + present[self._device_class(device).id] = backend + return present + + def _present(self, device_class_id: str) -> Optional[tuple[Environment, Platform]]: + """The environment and platform of a GPU generation device of this class; None when no such device is one.""" + for device in self._devices(): + if backend_for(device) is not None and self._device_class(device).id == device_class_id: + return self._environment(device), platform_for(device) + return None + + def get_model_summary( + self, model_key: str, device_class_id: Optional[str] = None + ) -> WorkingMemoryCalibrationModelSummary: + store = self._services.model_manager.store + # Raises UnknownModelException for an unknown key. + target = store.get_model(model_key) + class_id = self._current_device_class_id(device_class_id) + read = self._store.read(class_id) + present = self._present(class_id) + method, verified = _measurement(self._present_backends().get(class_id)) + summary = WorkingMemoryCalibrationModelSummary( + model_key=model_key, + device_class_id=class_id, + usable=read is None or read.file is not None, + problem=read.problem if read is not None else None, + current_env_id=present[0].id if present is not None else None, + measurement_method=method, + measurement_verified=verified, + lines=[], + excluded_windows=0, + runs=[], + ) + if read is None or read.file is None: + return summary + file = read.file + runs = [run for run in file.runs if run.model_key == model_key] + run_ids = {run.run_id for run in runs} + + def name_of(key: str) -> Optional[str]: + try: + return getattr(store.get_model(key), "name", None) + except UnknownModelException: + return None + + def platform_of(env: CalibrationEnvironment) -> Optional[Platform]: + if present is not None and env.id == present[0].id: + return present[1] + try: + return platform_for_environment(file.device_class.backend, env.hard) + except ValueError: + return None + + lines, excluded = model_lines( + file, + model_key, + present[0] if present is not None else None, + platform_of, + import_estimator_modules(), + priced_on=decode_lines_of(target, self._services.configuration.precision), + name_of=name_of, + live_runs=self._live_runs(file)[0], + ) + points_per_run: dict[str, int] = {} + valid_points_per_run: dict[str, int] = {} + for point in file.points: + if point.run_id in run_ids: + points_per_run[point.run_id] = points_per_run.get(point.run_id, 0) + 1 + if point.valid: + valid_points_per_run[point.run_id] = valid_points_per_run.get(point.run_id, 0) + 1 + live = self._live_items() if runs else {} + model_runs = [] + for run in sorted(runs, key=lambda run: run.created_at, reverse=True): + point_count = points_per_run.get(run.run_id, 0) + status, error, _ = derive_run_status(run, live.get(run.run_id, {}), valid_points_per_run.get(run.run_id, 0)) + model_runs.append( + WorkingMemoryCalibrationModelRun( + run_id=run.run_id, + status=status, + error=error, + created_at=run.created_at, + point_count=point_count, + attempt_id=run.attempt_id, + ) + ) + return summary.model_copy(update={"lines": lines, "excluded_windows": excluded, "runs": model_runs}) + + # endregion + + # region Reset and export + + @staticmethod + def _remove( + file: CalibrationStoreFile, run_ids: set[str], model_key: Optional[str] = None, *, remember: bool = True + ) -> tuple[bool, WorkingMemoryCalibrationResetResult]: + """Remove the runs `run_ids` with their points, and with `model_key` every point the summary attributes to + that model (`measured_models`). `remember` keeps the removed runs' ids, so that a repeat of the request that + started one cannot start it again.""" + models = run_models(file) + if remember: + file.removed_run_ids.extend( + run.run_id for run in file.runs if run.run_id in run_ids and _is_keyed(run.run_id) + ) + runs = [run for run in file.runs if run.run_id not in run_ids] + points = [ + point + for point in file.points + if point.run_id not in run_ids and (model_key is None or model_key not in measured_models(point, models)) + ] + result = WorkingMemoryCalibrationResetResult( + runs_removed=len(file.runs) - len(runs), points_removed=len(file.points) - len(points) + ) + file.runs, file.points = runs, points + _prune_envs(file) + return result.runs_removed > 0 or result.points_removed > 0, result + + def reset_run(self, run_id: str) -> WorkingMemoryCalibrationResetResult: + return self._remove_run(run_id, remember=True) + + def _remove_run(self, run_id: str, *, remember: bool) -> WorkingMemoryCalibrationResetResult: + found = self._find_run(run_id) + if found is None: + raise CalibrationRunNotFoundError(run_id) + result = self._store.change(found[0], lambda file: self._remove(file, {run_id}, remember=remember)) + self._run_files.pop(run_id, None) + self._try_refresh_user_fits("after a reset") + return result or WorkingMemoryCalibrationResetResult(runs_removed=0, points_removed=0) + + def reset_model(self, model_key: str, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + class_id = self._current_device_class_id(device_class_id) + + def remove(file: CalibrationStoreFile) -> tuple[bool, WorkingMemoryCalibrationResetResult]: + # Written with the removal or not at all: a node that read the older epoch drops its points of the model. + file.model_reset_epochs[model_key] = file.model_reset_epochs.get(model_key, 0) + 1 + run_ids = {run.run_id for run in file.runs if run.model_key == model_key} + return True, self._remove(file, run_ids, model_key)[1] + + result = self._store.change(class_id, remove) + self._try_refresh_user_fits("after a reset") + return result or WorkingMemoryCalibrationResetResult(runs_removed=0, points_removed=0) + + def reset_all(self, device_class_id: Optional[str] = None) -> WorkingMemoryCalibrationResetResult: + def remove(file: CalibrationStoreFile) -> tuple[bool, WorkingMemoryCalibrationResetResult]: + result = WorkingMemoryCalibrationResetResult(runs_removed=len(file.runs), points_removed=len(file.points)) + changed = bool(file.runs or file.points or file.envs) + file.removed_run_ids.extend(run.run_id for run in file.runs if _is_keyed(run.run_id)) + file.runs, file.points, file.envs = [], [], [] + return changed, result + + result = self._store.change(self._current_device_class_id(device_class_id), remove) + self._try_refresh_user_fits("after a reset") + return result or WorkingMemoryCalibrationResetResult(runs_removed=0, points_removed=0) + + def export(self, device_class_id: Optional[str] = None) -> Optional[bytes]: + return self._store.read_raw(self._current_device_class_id(device_class_id)) + + # endregion diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_fits.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_fits.py new file mode 100644 index 00000000000..898b3c4cdcd --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_fits.py @@ -0,0 +1,423 @@ +"""User lines from a calibration store file, which models its windows count for, and what the windows of one model's +lines measured. + +The fitting and resolution rules are the backend's (`backend/working_memory/user_fit.py`); this module picks the +windows a fit uses: those of one environment whose hard facts are the current ones, valid, fit-eligible and not +fragmented (`exclusions`). +""" + +import json +from collections import Counter +from collections.abc import Callable, Iterable, Mapping +from collections.abc import Set as AbstractSet +from dataclasses import dataclass +from typing import Final, Optional + +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + CalibrationEnvironment, + CalibrationLineState, + CalibrationPoint, + CalibrationRun, + CalibrationSizeExclusion, + CalibrationStoreFile, + WorkingMemoryCalibrationLineContributor, + WorkingMemoryCalibrationLineSize, + WorkingMemoryCalibrationModelLine, + WorkingMemoryCalibrationPixelRange, +) +from invokeai.backend.model_manager.taxonomy import SubModelType +from invokeai.backend.working_memory import fragmentation, user_fit +from invokeai.backend.working_memory.coefficients import shipped_coefficients +from invokeai.backend.working_memory.fingerprint import Environment +from invokeai.backend.working_memory.fit import EntryKey, fits_a_line, line_of +from invokeai.backend.working_memory.platform import Platform +from invokeai.backend.working_memory.spec import EstimatorSpec +from invokeai.backend.working_memory.user_fit import UserFit, UserLine + + +def fit_label(device_class_id: str, env_id: str) -> str: + return f"user:{device_class_id}:{env_id}" + + +def _measures_a_line(point: CalibrationPoint) -> bool: + """Whether a window measured a line: valid, a positive demand, and features a line admits.""" + return point.valid and point.demand_bytes is not None and point.demand_bytes > 0 and fits_a_line(point.features) + + +def _suffix(point: CalibrationPoint) -> str: + """The ``_`` of the line a window measured.""" + found = line_of(point.features) + assert found is not None, "a window that measured a line names it" + return found[0] + + +def _attempts(file: CalibrationStoreFile) -> dict[str, str]: + """Per run id, the attempt its windows are judged with: its `attempt_id`, or the run itself without one.""" + return {run.run_id: run.attempt_id or run.run_id for run in file.runs} + + +def exclusions( + file: CalibrationStoreFile, live_runs: AbstractSet[str] = frozenset() +) -> dict[int, CalibrationSizeExclusion]: + """Why each window that measured a line does not count toward its fit, by position in `file.points`; a window + that measured a line and is absent counts. + + - partial_residency: valid but not fit-eligible, its model not fully on the device. + - fragmented: the fit-eligible windows of one measurement -- one attempt (`attempt_id`, else the run) at one + environment, estimator, version and features, in the order they ran -- are what `fragmentation.fragmented` + flags: they grew around a flat allocation, or their largest window is a ratio outlier against the smaller + sizes of the same line in the environment that count, from any attempt. The measurement drops out whole: its + windows ran back to back on one allocator state. Another attempt's windows of the same size ran on another and + are judged on their own, so a recalibration can bring a size back and a later fragmented attempt drops no clean + one. Windows of `live_runs` (runs still running) are judged, but set no reference for larger sizes, as they + join no user line yet. + """ + points = file.points + attempt_of = _attempts(file) + excluded: dict[int, CalibrationSizeExclusion] = {} + groups: dict[tuple[str, str, int, str, str], list[int]] = {} + for index, point in enumerate(points): + if not _measures_a_line(point): + continue + if not point.fit_eligible: + excluded[index] = "partial_residency" + continue + features = json.dumps(point.features, sort_keys=True, separators=(",", ":")) + attempt = attempt_of.get(point.run_id, point.run_id) + groups.setdefault((point.env_id, point.estimator, point.version, features, attempt), []).append(index) + + members = list(groups.values()) + measurements = [] + for (env_id, estimator, version, _, _), indices in groups.items(): + windows = [points[index] for index in indices] + measurements.append( + fragmentation.Measurement( + line=(env_id, estimator, version, _suffix(windows[0])), + pixels=_pixels(windows[0]), + demand=[window.demand_bytes or 0 for window in windows], + reserved=[window.reserved.delta if window.reserved else None for window in windows], + allocated=[window.allocated.delta if window.allocated else None for window in windows], + reference=not any(window.run_id in live_runs for window in windows), + ) + ) + for indices, flagged in zip(members, fragmentation.fragmented(measurements), strict=True): + if flagged: + excluded.update(dict.fromkeys(indices, "fragmented")) + return excluded + + +def _pixels(point: CalibrationPoint) -> int: + found = line_of(point.features) + assert found is not None, "a window that measured a line names it" + return found[1] + + +def counted_windows(file: CalibrationStoreFile, live_runs: AbstractSet[str] = frozenset()) -> list[CalibrationPoint]: + """The windows that count toward a fit: they measured a line, and `exclusions` names no reason against them.""" + excluded = exclusions(file, live_runs) + return [point for index, point in enumerate(file.points) if _measures_a_line(point) and index not in excluded] + + +def recorded_outside_torch_bytes(file: CalibrationStoreFile, env_id: str) -> Optional[int]: + """The most the measuring process held outside torch's allocator when a valid window of environment `env_id` + opened: the driver's figure for the process less torch's reserve, at the window's baseline. None without such a + window, or where the driver had no figure for the process (`torch-only`, and XPU's device-wide reading).""" + found = [ + point.driver.baseline - point.reserved.baseline + for point in file.points + if point.env_id == env_id and point.valid and point.driver is not None and point.reserved is not None + ] + return max(found, default=None) + + +def user_fit_of( + file: CalibrationStoreFile, env_id: str, platform: Platform, *, exclude_runs: AbstractSet[str] = frozenset() +) -> tuple[Optional[UserFit], Mapping[user_fit.LineKey, str]]: + """The user fit of the file's counted windows (`counted_windows`) taken under environment `env_id`, measured on + `platform`, or None when they fit no line; and why the lines they measure but cannot fit cannot be. The windows of + `exclude_runs` (runs still running) take no part: they join once their run finished.""" + points = [ + user_fit.UserPoint(point.estimator, point.version, point.features, point.demand_bytes or 0) + for point in counted_windows(file, exclude_runs) + if point.env_id == env_id and point.run_id not in exclude_runs + ] + lines, problems = user_fit.fit_user_lines(points, platform) + if not lines: + return None, problems + return UserFit(label=fit_label(file.device_class.id, env_id), platform=platform, lines=lines), problems + + +def _changed(old: Mapping[str, Optional[str]], new: Mapping[str, Optional[str]]) -> list[str]: + return sorted(key for key in set(old) | set(new) if old.get(key) != new.get(key)) + + +_SUBMODEL_TYPES: Final = frozenset(submodel.value for submodel in SubModelType) + + +def measured_models(point: CalibrationPoint, run_models: Mapping[str, str]) -> frozenset[str]: + """The models a window is attributed to: the model its run calibrated (`run_models`, by run id), and the model it + measured, where a submodel's window (`:vae`) counts for its model. The summary shows a model's windows by + this, and resetting the model removes exactly these.""" + head, separator, tail = point.model_key.rpartition(":") + measured = head if separator and tail in _SUBMODEL_TYPES else point.model_key + run_model = run_models.get(point.run_id) + return frozenset({measured} if run_model is None else {measured, run_model}) + + +def run_models(file: CalibrationStoreFile) -> dict[str, str]: + return {run.run_id: run.model_key for run in file.runs} + + +@dataclass(frozen=True) +class _EnvFit: + env: Optional[CalibrationEnvironment] + platform: Optional[Platform] + fit: Optional[UserFit] + problems: Mapping[user_fit.LineKey, str] + + +LineId = tuple[str, str, int, str] +"""A summary line: environment id, estimator, version and ``_``.""" + + +def model_lines( + file: CalibrationStoreFile, + model_key: str, + current_env: Optional[Environment], + platform_of: Callable[[CalibrationEnvironment], Optional[Platform]], + specs: Mapping[str, EstimatorSpec], + priced_on: Iterable[tuple[str, str]], + name_of: Callable[[str], Optional[str]], + live_runs: AbstractSet[str] = frozenset(), +) -> tuple[list[WorkingMemoryCalibrationModelLine], int]: + """Per environment and line, what the windows that feed it measured, with the shipped and the user estimate at + each size; and how many of the model's own windows fit no line. + + The lines are those the model's own windows (`measured_models`) measured, and, in `current_env`, each line of + `priced_on` (estimator, ``_``: what the model's operations price on) that other models' windows + measured. A line is fed by every model's counted windows (`exclusions`) of its environment that join the same user + line, as estimates read it: those whose estimator resolves to the same carrying entry on the environment's + platform. Its sizes also show, as not counted, the sizes only excluded windows measured. + + The sizes, window counts and contributors show every such window, those of `live_runs` (runs still running) + included, so a running calibration shows what it measured so far. The user line, its range, the user estimates and + the state are what estimates use: the fit without the windows of `live_runs`. + + The state compares the environment with `current_env` (None: no GPU of the file's class generates here): the same + hard facts use the line, stale when the soft facts of the newest finished run that fed it differ from the current + ones. A run stored without soft facts stands for the environment's, those first recorded with it. Older windows of + the same hard environment keep feeding the line whatever their soft facts: soft facts do not change what a window + measured enough to drop it. + """ + envs = {env.id: env for env in file.envs} + runs = {run.run_id: run for run in file.runs} + models = run_models(file) + reasons = exclusions(file, live_runs) + # By identity: the windows below are the file's own objects. + reason_of: dict[int, CalibrationSizeExclusion] = { + id(file.points[index]): reason for index, reason in reasons.items() + } + excluded = 0 + own: dict[LineId, list[CalibrationPoint]] = {} + measured_by_env: dict[str, list[CalibrationPoint]] = {} + for point in file.points: + measured = _measures_a_line(point) + if measured: + measured_by_env.setdefault(point.env_id, []).append(point) + if model_key not in measured_models(point, models): + continue + if not measured or id(point) in reason_of: + excluded += 1 + if measured: + own.setdefault((point.env_id, point.estimator, point.version, _suffix(point)), []).append(point) + + def counts(point: CalibrationPoint) -> bool: + return id(point) not in reason_of + + env_fits: dict[str, _EnvFit] = {} + + def env_fit(env_id: str) -> _EnvFit: + if env_id not in env_fits: + env = envs.get(env_id) + platform = platform_of(env) if env is not None else None + fit, problems = ( + user_fit_of(file, env_id, platform, exclude_runs=live_runs) if platform is not None else (None, {}) + ) + env_fits[env_id] = _EnvFit(env, platform, fit, problems) + return env_fits[env_id] + + def carrier_of(estimator: str, version: int, platform: Optional[Platform]) -> Optional[EntryKey]: + """The entry whose user line a window of this estimator version joins; None where it is not priced here.""" + spec = specs.get(estimator) + if spec is None or spec.version != version or platform is None: + return None + return user_fit.carrier_key(estimator, version, platform) + + def feeding(line_id: LineId) -> list[CalibrationPoint]: + """The windows of the line's environment that measured its user line, counted or not; without a carrying + entry, those of the same estimator version.""" + env_id, estimator, version, suffix = line_id + platform = env_fit(env_id).platform + carrier = carrier_of(estimator, version, platform) + found = [] + for point in measured_by_env.get(env_id, []): + if _suffix(point) != suffix: + continue + if carrier is None: + same = (point.estimator, point.version) == (estimator, version) + else: + same = carrier_of(point.estimator, point.version, platform) == carrier + if same: + found.append(point) + return found + + line_ids = set(own) + if current_env is not None: + for estimator, suffix in priced_on: + spec = specs.get(estimator) + if spec is not None: + line_ids.add((current_env.id, estimator, spec.version, suffix)) + + names: dict[str, Optional[str]] = {} + lines: list[WorkingMemoryCalibrationModelLine] = [] + for line_id in sorted(line_ids): + env_id, estimator, version, suffix = line_id + measured = feeding(line_id) + if not measured: + # A line the model prices on that no window measured in this environment. + continue + points = [point for point in measured if counts(point)] + own_points = [point for point in own.get(line_id, []) if counts(point)] + found_fit = env_fit(env_id) + env, platform, fit = found_fit.env, found_fit.platform, found_fit.fit + spec = specs.get(estimator) + priced = spec is not None and spec.version == version + carrier = carrier_of(estimator, version, platform) + line: Optional[UserLine] = None + if fit is not None and carrier is not None: + line = fit.lines.get((carrier, suffix)) + + changed: list[str] = [] + problem: Optional[str] = None + state: CalibrationLineState + if env is None or current_env is None: + state, problem = "unusable", "No generation device of this class is present." + elif env.id != current_env.id: + state, changed = "unusable", _changed(env.hard, current_env.hard) + elif not priced: + state = "unusable" + problem = f"Measured with {estimator} version {version}, which this version does not price with." + elif line is None: + state = "insufficient" + known = found_fit.problems.get((carrier, suffix)) if carrier is not None else None + if not points: + problem = "None of this line's windows count: see each size's excluded_reason." + elif any(point.run_id in live_runs for point in points): + problem = "A calibration run is still measuring this line; its windows count once it finished." + if known: + problem += f" So far: {known}" + else: + problem = known or "The shipped data has no such line for this estimator on this platform." + else: + finished = [point for point in points if point.run_id not in live_runs] + changed = _changed(_newest_soft(finished, runs, env), current_env.soft) + state = "stale" if changed else "in_use" + + windows: Counter[str] = Counter() + for point in points: + windows.update(measured_models(point, models)) + for key in windows: + if key not in names: + names[key] = name_of(key) + measured_by = [ + WorkingMemoryCalibrationLineContributor(model_key=key, name=names[key], windows=count) + for key, count in windows.items() + ] + measured_by.sort( + key=lambda contributor: (contributor.name is None, contributor.name or "", contributor.model_key) + ) + + lines.append( + WorkingMemoryCalibrationModelLine( + estimator=estimator, + version=version, + line=suffix, + carrier=carrier[0] if carrier else None, + env_id=env_id, + state=state, + changed_facts=changed, + problem=problem, + measured_range=( + WorkingMemoryCalibrationPixelRange(min_pixels=line.x_min, max_pixels=line.x_max) if line else None + ), + slope_bytes_per_pixel=line.k if line else None, + base_bytes=line.b if line else None, + sizes=_sizes(measured, reason_of, estimator, specs if priced else {}, platform, fit if line else None), + own_windows=len(own_points), + measured_by=measured_by, + ) + ) + return lines, excluded + + +def _newest_soft( + points: Iterable[CalibrationPoint], runs: Mapping[str, CalibrationRun], env: CalibrationEnvironment +) -> Mapping[str, Optional[str]]: + """The soft facts of the newest run among those that measured `points`; the environment's when it recorded none. + `runs` is in the file's order, which is the order they were created in: it decides a tie of the clock.""" + order = {run_id: index for index, run_id in enumerate(runs)} + fed = {point.run_id for point in points if point.run_id in runs} + newest = max((runs[run_id] for run_id in fed), key=lambda run: (run.created_at, order[run.run_id]), default=None) + return newest.soft if newest is not None and newest.soft is not None else env.soft + + +def _sizes( + points: Iterable[CalibrationPoint], + reason_of: Mapping[int, CalibrationSizeExclusion], + estimator: str, + specs: Mapping[str, EstimatorSpec], + platform: Optional[Platform], + fit: Optional[UserFit], +) -> list[WorkingMemoryCalibrationLineSize]: + """Per measured size, the windows and their largest demand, and the shipped and the user estimate there. + + A size with counted windows shows those; a size whose windows all are excluded (`reason_of`, by window identity) + shows them as not counted, fragmented where any of them is. The estimates price the largest window of the line's + own estimator among those shown, or, where only other estimators that share the line measured the size, the largest + window with its own estimator: their features are their own.""" + by_size: dict[tuple[int, int], list[CalibrationPoint]] = {} + for point in points: + by_size.setdefault((int(point.features["width"]), int(point.features["height"])), []).append(point) # type: ignore[arg-type] + sizes = [] + for (width, height), measured in sorted(by_size.items(), key=lambda item: (item[0][0] * item[0][1], item[0])): + group = [point for point in measured if id(point) not in reason_of] + counted = bool(group) + excluded_reason: Optional[CalibrationSizeExclusion] = None + if not counted: + group = measured + found = {reason_of[id(point)] for point in measured} + excluded_reason = "fragmented" if "fragmented" in found else "partial_residency" + largest = max(group, key=lambda point: point.demand_bytes or 0) + mine = [point for point in group if point.estimator == estimator] + priced = max(mine, key=lambda point: point.demand_bytes or 0) if mine else largest + spec = specs.get(priced.estimator) if estimator in specs else None + shipped_bytes = user_bytes = None + if spec is not None and spec.version == priced.version and platform is not None: + shipped = shipped_coefficients(spec, platform) + shipped_bytes = spec.evaluate(priced.features, shipped).reservation_bytes + if fit is not None: + user_bytes = user_fit.resolve(spec, priced.features, shipped, fit).reservation_bytes + sizes.append( + WorkingMemoryCalibrationLineSize( + width=width, + height=height, + pixels=width * height, + windows=len(group), + measured_bytes=largest.demand_bytes or 0, + counted=counted, + excluded_reason=excluded_reason, + shipped_bytes=shipped_bytes, + user_bytes=user_bytes, + ) + ) + return sizes diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_plan.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_plan.py new file mode 100644 index 00000000000..622607420f9 --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_plan.py @@ -0,0 +1,421 @@ +"""The cases a client runs to calibrate one model's working memory, and which other models that calibration moves. + +Covered: the decode of main models whose decode node prices with one of the five AutoencoderKL estimators of +`backend/util/vae_working_memory.py` (`DECODE_LINES`), and standalone VAEs of those bases, calibrated by generating +with a compatible installed main model. Each case is a two-step text-to-image generation, which runs the whole +pipeline and ends in an untiled decode; image-to-image would add an encode, which v1 does not plan. +""" + +import math +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from functools import partial +from typing import Any, Final, Optional + +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + WorkingMemoryCalibrationCase, + WorkingMemoryCalibrationDeviceClassRef, + WorkingMemoryCalibrationPlan, + WorkingMemoryCalibrationRepetitions, + WorkingMemoryCalibrationSharedModel, +) +from invokeai.backend.architectures import VaeFacet +from invokeai.backend.architectures import get as get_facet +from invokeai.backend.model_manager.taxonomy import ( + BaseModelType, + FluxVariantType, + ModelType, + ModelVariantType, +) +from invokeai.backend.util.vae_working_memory import VAE_PRETILE_VRAM_FRACTION +from invokeai.backend.working_memory import user_fit +from invokeai.backend.working_memory.fingerprint import DeviceClass +from invokeai.backend.working_memory.fit import MIN_AREA_RATIO, EntryKey +from invokeai.backend.working_memory.platform import Platform +from invokeai.backend.working_memory.probe import calibration_reservation +from invokeai.backend.working_memory.spec import EstimatorSpec + +ModelConfig = Any +"""An installed model's config. Read by attribute: `key`, `name`, `base`, `type`, and where present `variant` and +`cpu_only`.""" + +ReservationFunction = Callable[[str, int, int, str], int] +"""The reservation an untiled decode of (estimator, height, width, dtype) gets on the calibration device.""" + + +@dataclass(frozen=True) +class DecodeLines: + estimator: str + dtypes: tuple[str, ...] + """The dtypes the decode runs in by default. Where the node casts the VAE (`vaePrecision`), each is a case.""" + precision_setting: bool = False + + +_SD = DecodeLines("vae.sd", ("fp16", "fp32"), precision_setting=True) +_FLUX = DecodeLines("vae.flux", ("bf16",)) +_FLUX2 = DecodeLines("vae.flux2", ("bf16",)) + +DECODE_LINES: Final[Mapping[BaseModelType, DecodeLines]] = { + BaseModelType.StableDiffusion1: _SD, + BaseModelType.StableDiffusion2: _SD, + BaseModelType.StableDiffusionXL: _SD, + BaseModelType.StableDiffusionXLRefiner: _SD, + # The SD3 VAE runs in the configured precision (`_sd3_dtype`). + BaseModelType.StableDiffusion3: DecodeLines("vae.sd3", ()), + # The FLUX.1 loaders avoid fp16, and Z-Image decodes with a FLUX.1 VAE. + BaseModelType.Flux: _FLUX, + BaseModelType.ZImage: _FLUX, + BaseModelType.Flux2: _FLUX2, + # Decodes with the FLUX.2 VAE; shares its lines, but generates from structured prompts the plan does not write. + BaseModelType.Ideogram4: _FLUX2, + # The CogView4 loader forces bf16. + BaseModelType.CogView4: DecodeLines("vae.cogview4", ("bf16",)), +} +"""Per base, the estimator its decode node prices with and the dtypes it runs in.""" + +PLANNABLE_BASES: Final = frozenset( + { + BaseModelType.StableDiffusion1, + BaseModelType.StableDiffusion2, + BaseModelType.StableDiffusionXL, + BaseModelType.StableDiffusion3, + BaseModelType.Flux, + BaseModelType.Flux2, + BaseModelType.ZImage, + BaseModelType.CogView4, + } +) +"""Bases whose main models a text-to-image case can calibrate. The SDXL refiner generates from no text, and Ideogram 4 +from structured prompts only.""" + +VAE_SLOT_BASES: Final = frozenset( + {BaseModelType.Flux, BaseModelType.Flux2, BaseModelType.StableDiffusion3, BaseModelType.ZImage} +) +"""Bases whose Generate widget has a `vae` component slot. The SD bases pick a VAE through a setting a case cannot +override, and CogView4 has none: their standalone VAEs are calibrated through a main model's own VAE, which prices on +the same lines.""" + +NOT_TEXT_TO_IMAGE_VARIANTS: Final = frozenset( + {ModelVariantType.Inpaint.value, ModelVariantType.Depth.value, FluxVariantType.DevFill.value} +) + +CASE_STEPS: Final = 2 +CASE_SEED: Final = 1 +CASE_PROMPT: Final = "a mountain lake at dawn, detailed photograph" +REPETITIONS: Final = WorkingMemoryCalibrationRepetitions(warm_up=1, measured=2) + +AREA_FRACTIONS: Final = (0.25, 0.5, 1.0) +SIDE_MULTIPLE: Final = 64 +"""A multiple of every covered base's dimension grid (8, 16 or 32).""" +MAX_SIDE: Final = 4096 +"""The largest width or height Generate accepts; no covered architecture declares a smaller limit.""" +MIN_LARGEST_SIDE: Final = 512 +VAE_WEIGHTS_ALLOWANCE_BYTES: Final = 2**30 +"""What the VAE's own weights take on the device while it decodes: under 1 GiB for every covered VAE.""" + +SECONDS_MODEL_LOAD: Final = 30 +SECONDS_PER_RUN: Final = 3 +SECONDS_PER_MEGAPIXEL_STEP: Final = 1.0 +SECONDS_PER_MEGAPIXEL_DECODE: Final = 1.0 + +ENCODE_NOT_COVERED: Final = ( + "encode: text-to-image runs no encode, so image-to-image and inpainting encodes keep the shipped values" +) + + +def sd3_dtype(precision: str) -> str: + """The dtype the SD3 VAE runs in under the `precision` setting: `auto` is fp16 on a GPU.""" + return {"bfloat16": "bf16", "float32": "fp32"}.get(precision, "fp16") + + +def decode_dtypes(base: BaseModelType, precision: str) -> tuple[str, ...]: + lines = DECODE_LINES[base] + return lines.dtypes or (sd3_dtype(precision),) + + +def _value(field: object) -> str: + return str(getattr(field, "value", field)) + + +def _variant(config: ModelConfig) -> Optional[str]: + variant = getattr(config, "variant", None) + return None if variant is None else _value(variant) + + +def _runs_on_gpu(vae: ModelConfig) -> bool: + return getattr(vae, "cpu_only", None) is not True + + +def _accepts(main_base: BaseModelType, vae_base: BaseModelType) -> bool: + facet = get_facet(main_base, VaeFacet) + if facet is None: + return main_base == vae_base + return any(compatibility.base == vae_base for compatibility in facet.accepted) + + +def _is_main(config: ModelConfig) -> bool: + return bool(config.type == ModelType.Main) + + +def _is_vae(config: ModelConfig) -> bool: + return bool(config.type == ModelType.VAE) + + +def _plannable_main(config: ModelConfig) -> bool: + return _is_main(config) and config.base in PLANNABLE_BASES and _variant(config) not in NOT_TEXT_TO_IMAGE_VARIANTS + + +def decode_budget(device_class: DeviceClass, capacity_bytes: Optional[int]) -> int: + """The largest calibration reservation a case may take: within the share of the card a decode may claim before the + node tiles it up front (`VAE_PRETILE_VRAM_FRACTION`), and within `capacity_bytes`, what the model cache can give on + the device; each less the VAE's own weights, which must stay fully resident beside it. Nothing else does: a + calibration lock first moves every other unlocked model out (`LoadedModelWithoutConfig.model_on_device`). Past the + capacity the cache loads the VAE partly (`partial_residency`), and such a window fits no line. `capacity_bytes` + None: the device does not say (shared memory), and the share of the card alone bounds it.""" + ceiling = int(VAE_PRETILE_VRAM_FRACTION * device_class.total_mib * 2**20) + if capacity_bytes is not None: + ceiling = min(ceiling, capacity_bytes) + return ceiling - VAE_WEIGHTS_ALLOWANCE_BYTES + + +def case_sides(reservation: Callable[[int], int], budget_bytes: int) -> list[int]: + """The square sides of the three cases: about `AREA_FRACTIONS` of the largest area that decodes untiled here. + + The largest area is the largest square, in steps of `SIDE_MULTIPLE` and at most `MAX_SIDE`, whose calibration + reservation (the estimate plus calibration headroom, `probe.calibration_reservation`) fits `budget_bytes` + (`decode_budget`). A larger decode would be tiled or partly loaded, and neither fits a line. Each fraction of that + area is rounded down to a square side that is a multiple of `SIDE_MULTIPLE`; repeated sides are dropped. Returns + no sides when the largest is below `MIN_LARGEST_SIDE` or the sides do not span two sizes `MIN_AREA_RATIO` apart. + """ + largest = 0 + for side in range(SIDE_MULTIPLE, MAX_SIDE + 1, SIDE_MULTIPLE): + if calibration_reservation(reservation(side)) > budget_bytes: + break + largest = side + if largest < MIN_LARGEST_SIDE: + return [] + sides: list[int] = [] + for fraction in AREA_FRACTIONS: + side = max(SIDE_MULTIPLE, int(math.sqrt(fraction) * largest) // SIDE_MULTIPLE * SIDE_MULTIPLE) + if side not in sides: + sides.append(side) + if sides[-1] ** 2 < MIN_AREA_RATIO * sides[0] ** 2: + return [] + return sides + + +def _square(reservation: ReservationFunction, estimator: str, dtype: str, side: int) -> int: + return reservation(estimator, side, side, dtype) + + +def decode_lines_of(model: ModelConfig, precision: str) -> list[tuple[str, str]]: + """The (estimator, ``decode_``) lines a model's decode prices on: a main model or a GPU VAE of a base in + `DECODE_LINES`; none for any other model.""" + if getattr(model, "base", None) not in DECODE_LINES: + return [] + if not (_is_main(model) or (_is_vae(model) and _runs_on_gpu(model))): + return [] + estimator = DECODE_LINES[model.base].estimator + return [(estimator, f"decode_{dtype}") for dtype in decode_dtypes(model.base, precision)] + + +def estimated_seconds(cases: Sequence[WorkingMemoryCalibrationCase]) -> int: + """A rough order of magnitude: one model load, then per run a fixed cost and a cost per megapixel for each step + and for the decode. Not measured; real times vary several-fold with the GPU and the model.""" + runs = REPETITIONS.warm_up + REPETITIONS.measured + seconds: float = SECONDS_MODEL_LOAD + for case in cases: + megapixels = case.width * case.height / 1e6 + per_run = SECONDS_PER_RUN + megapixels * ( + case.steps * SECONDS_PER_MEGAPIXEL_STEP + SECONDS_PER_MEGAPIXEL_DECODE + ) + seconds += runs * per_run + return math.ceil(seconds) + + +def _lines( + base: BaseModelType, precision: str, platform: Platform, specs: Mapping[str, EstimatorSpec] +) -> set[tuple[EntryKey, str]]: + """The lines a base's decode prices with on `platform`.""" + estimator = DECODE_LINES[base].estimator + spec = specs.get(estimator) + carrier = user_fit.carrier_key(estimator, spec.version, platform) if spec is not None else None + if carrier is None: + return set() + return {(carrier, f"decode_{dtype}") for dtype in decode_dtypes(base, precision)} + + +def shared_with( + target: ModelConfig, + models: Sequence[ModelConfig], + precision: str, + platform: Platform, + specs: Mapping[str, EstimatorSpec], +) -> list[WorkingMemoryCalibrationSharedModel]: + """The other installed main models and GPU VAEs of a covered base whose decode prices with a line the target's + does, on this platform: a line is the shipped entry that carries the values (one estimator may borrow another's) + and the operation and dtype.""" + mine = _lines(target.base, precision, platform, specs) + shared = [] + for model in models: + if model.key == target.key or model.base not in DECODE_LINES: + continue + if not (_is_main(model) or (_is_vae(model) and _runs_on_gpu(model))): + continue + common = sorted(suffix for _, suffix in mine & _lines(model.base, precision, platform, specs)) + if common: + shared.append( + WorkingMemoryCalibrationSharedModel( + key=model.key, name=model.name, base=_value(model.base), type=_value(model.type), lines=common + ) + ) + return sorted(shared, key=lambda model: (model.name, model.key)) + + +def _gpu_vae_pick(main: ModelConfig, models: Sequence[ModelConfig]) -> dict[str, str]: + """A `vae` pick where Generate's own fill could land on a VAE that runs on the CPU: the first GPU VAE the main + model accepts. None needed where every accepted VAE runs on the GPU, or where the base has no VAE slot.""" + if main.base not in VAE_SLOT_BASES: + return {} + accepted = [model for model in models if _is_vae(model) and _accepts(main.base, model.base)] + if all(_runs_on_gpu(vae) for vae in accepted): + return {} + gpu = sorted((vae for vae in accepted if _runs_on_gpu(vae)), key=lambda vae: (vae.name, vae.key)) + return {"vae": gpu[0].key} if gpu else {} + + +def _unsupported( + target: ModelConfig, reason: str, device: Optional[DeviceClass], estimators: list[str] +) -> WorkingMemoryCalibrationPlan: + return WorkingMemoryCalibrationPlan( + model_key=target.key, + supported=False, + reason=reason, + device_class=WorkingMemoryCalibrationDeviceClassRef(id=device.id, name=device.name) if device else None, + calibrated_through=None, + estimators=estimators, + cases=[], + repetitions=REPETITIONS, + estimated_seconds=None, + shared_with=[], + not_covered=[ENCODE_NOT_COVERED] if estimators else [], + ) + + +def build_plan( + target: ModelConfig, + models: Sequence[ModelConfig], + *, + device_class: Optional[DeviceClass], + unavailable: Optional[str], + platform: Optional[Platform], + precision: str, + reservation: ReservationFunction, + specs: Mapping[str, EstimatorSpec], + capacity_bytes: Optional[int] = None, +) -> WorkingMemoryCalibrationPlan: + """The plan for calibrating `target`, among the installed `models`, on the GPU of `device_class`. + + `unavailable` says why this machine cannot calibrate (no GPU, several generation devices); `device_class` and + `platform` are then None. `reservation` prices an untiled decode on the calibration device; `precision` is the + `precision` setting; `capacity_bytes` is what the cache can give on the device (`decode_budget`). + """ + covered = target.base in DECODE_LINES and (_is_main(target) or _is_vae(target)) + estimators = [DECODE_LINES[target.base].estimator] if covered else [] + if not covered: + kind = f"{_value(target.base)} {_value(target.type)}" + return _unsupported( + target, + f"Calibration covers the decode of SD1, SD2, SDXL, SD3, FLUX.1, FLUX.2, Z-Image and CogView4 main models " + f"and their VAEs; this is a {kind} model.", + device_class, + estimators, + ) + + main = target + if _is_vae(target): + if not _runs_on_gpu(target): + return _unsupported( + target, + "This VAE runs on the CPU (cpu_only); calibration measures GPU memory.", + device_class, + estimators, + ) + candidates = sorted( + (model for model in models if _plannable_main(model) and _accepts(model.base, target.base)), + key=lambda model: (model.name, model.key), + ) + if not candidates: + return _unsupported( + target, + "A VAE is calibrated by generating with a main model that decodes with it, and no compatible main " + "model is installed.", + device_class, + estimators, + ) + main = candidates[0] + elif target.base not in PLANNABLE_BASES: + return _unsupported( + target, + "This model shares its decode with others but cannot generate from a plain text prompt; calibrate one of " + "the models listed as sharing it.", + device_class, + estimators, + ) + elif _variant(target) in NOT_TEXT_TO_IMAGE_VARIANTS: + return _unsupported( + target, f"The {_variant(target)} variant does not generate from text alone.", device_class, estimators + ) + + if unavailable is not None or device_class is None or platform is None: + return _unsupported(target, unavailable or "This machine has no GPU to calibrate.", None, estimators) + + if _is_vae(target) and main.base in VAE_SLOT_BASES: + components = {"vae": target.key} + else: + components = _gpu_vae_pick(main, models) + + lines = DECODE_LINES[target.base] + budget = decode_budget(device_class, capacity_bytes) + cases: list[WorkingMemoryCalibrationCase] = [] + for dtype in decode_dtypes(target.base, precision): + sides = case_sides(partial(_square, reservation, lines.estimator, dtype), budget) + for side in sides: + cases.append( + WorkingMemoryCalibrationCase( + id=f"{lines.estimator}-decode-{dtype}-{side}x{side}", + model_key=main.key, + components=components, + width=side, + height=side, + steps=CASE_STEPS, + seed=CASE_SEED, + positive_prompt=CASE_PROMPT, + negative_prompt="", + settings={"vaePrecision": dtype} if lines.precision_setting else {}, + estimator=lines.estimator, + line=f"decode_{dtype}", + ) + ) + if not cases: + return _unsupported( + target, + f"{device_class.name} cannot hold an untiled decode of {MIN_LARGEST_SIDE}x{MIN_LARGEST_SIDE} with " + "calibration headroom and the VAE fully loaded, in the memory free on it now.", + device_class, + estimators, + ) + + return WorkingMemoryCalibrationPlan( + model_key=target.key, + supported=True, + reason=None, + device_class=WorkingMemoryCalibrationDeviceClassRef(id=device_class.id, name=device_class.name), + calibrated_through=main.key if main is not target else None, + estimators=estimators, + cases=cases, + repetitions=REPETITIONS, + estimated_seconds=estimated_seconds(cases), + shared_with=shared_with(target, models, precision, platform, specs), + not_covered=[ENCODE_NOT_COVERED], + ) diff --git a/invokeai/app/services/working_memory_calibration/working_memory_calibration_store.py b/invokeai/app/services/working_memory_calibration/working_memory_calibration_store.py new file mode 100644 index 00000000000..c669385f78f --- /dev/null +++ b/invokeai/app/services/working_memory_calibration/working_memory_calibration_store.py @@ -0,0 +1,249 @@ +"""The calibration store: one JSON file per device class under `/working_memory/`. + +Machine data, not account data, so it lives beside the database rather than in it: a device class's measurements stay +valid whichever database the root points at. Every change re-reads the file and is applied to what is on disk under a +process lock and an OS file lock, so two processes sharing a root never lose each other's writes. Files are replaced +atomically. A file this version cannot read -- another schema version, invalid content, a field it does not know, or a device +class that does not match its name -- is reported and never written, moved or deleted. +""" + +import json +import os +import re +import tempfile +import threading +import time +from collections.abc import Callable, Iterator +from contextlib import contextmanager, suppress +from pathlib import Path +from typing import IO, Final, Optional, TypeVar + +from pydantic import ValidationError + +from invokeai.app.services.working_memory_calibration.working_memory_calibration_common import ( + STORE_SCHEMA_VERSION, + CalibrationDeviceClass, + CalibrationStoreError, + CalibrationStoreFile, + CalibrationStoreUnusableError, +) + +LOCK_TIMEOUT_S: float = 30.0 +"""How long a change waits for another process's lock on the same file before it gives up.""" +LOCK_RETRY_S: float = 0.05 +REPLACE_ATTEMPTS: int = 6 +"""How often a write tries to replace the file. On Windows the replace fails with PermissionError while a process that +does not take the store's lock (an antivirus scanner, a search indexer, an editor) has the file open; such a handle is +usually gone within a fraction of a second.""" +REPLACE_RETRY_S: float = 0.05 +"""The first wait between those tries; it doubles each time (about 1.5 s in all).""" + +_FILE_NAME: Final = re.compile(r"^[0-9a-f]{16}\.json$") + +T = TypeVar("T") + + +class StoreRead: + """A device class's file as read: its content, or why it cannot be used.""" + + def __init__(self, device_class_id: str, file: Optional[CalibrationStoreFile], problem: Optional[str]) -> None: + self.device_class_id = device_class_id + self.file = file + self.problem = problem + + +def _parse(raw: bytes, device_class_id: str) -> tuple[Optional[CalibrationStoreFile], Optional[str]]: + try: + data = json.loads(raw) + except ValueError: + return None, "The file is not valid JSON." + if not isinstance(data, dict): + return None, "The file does not hold a calibration store." + version = data.get("schema_version") + if version != STORE_SCHEMA_VERSION: + return None, f"The file has schema version {version!r}; this version of Invoke reads {STORE_SCHEMA_VERSION}." + try: + file = CalibrationStoreFile.model_validate(data) + except ValidationError as error: + first = error.errors()[0] + location = ".".join(str(part) for part in first["loc"]) + return None, f"The file failed validation ({error.error_count()} errors; first at {location}: {first['msg']})." + if file.device_class.id != device_class_id: + return None, f"The file holds device class {file.device_class.id}, not the {device_class_id} its name says." + return file, None + + +@contextmanager +def _os_lock(handle: IO[bytes]) -> Iterator[None]: + if os.name == "nt": + import msvcrt + + deadline = time.monotonic() + LOCK_TIMEOUT_S + while True: + handle.seek(0) + try: + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + break + except OSError: + if time.monotonic() > deadline: + raise CalibrationStoreError(f"Timed out waiting for the lock on {handle.name}") from None + time.sleep(LOCK_RETRY_S) + try: + yield + finally: + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + deadline = time.monotonic() + LOCK_TIMEOUT_S + while True: + try: + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + break + except BlockingIOError: + if time.monotonic() > deadline: + raise CalibrationStoreError(f"Timed out waiting for the lock on {handle.name}") from None + time.sleep(LOCK_RETRY_S) + try: + yield + finally: + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + + +def _replace(staging: str, path: Path) -> None: + """`os.replace`, retried while another process's handle on the target refuses it (Windows only).""" + delay = REPLACE_RETRY_S + for attempt in range(REPLACE_ATTEMPTS): + try: + os.replace(staging, path) + return + except PermissionError: + if attempt + 1 == REPLACE_ATTEMPTS: + raise + time.sleep(delay) + delay *= 2 + + +class CalibrationStore: + def __init__(self, directory: Path) -> None: + self._dir = directory + self._lock = threading.Lock() + + @property + def directory(self) -> Path: + return self._dir + + def path_for(self, device_class_id: str) -> Path: + return self._dir / f"{device_class_id}.json" + + def device_class_ids(self) -> list[str]: + """The device classes that have a file, readable or not.""" + if not self._dir.is_dir(): + return [] + return sorted(path.stem for path in self._dir.iterdir() if _FILE_NAME.match(path.name) and path.is_file()) + + @contextmanager + def _locked(self, device_class_id: str, *, writing: bool) -> Iterator[None]: + """Hold this process's lock and the OS lock on the file's `.lock` sibling. + + Readers lock too: on Windows a file another process holds open cannot be replaced. A reader that cannot create + the lock file (a read-only root) reads without it, since nobody can write there either. + """ + with self._lock: + lock_path = self._dir / f"{device_class_id}.json.lock" + handle: Optional[IO[bytes]] = None + try: + if writing: + self._dir.mkdir(parents=True, exist_ok=True) + handle = open(lock_path, "a+b") + except OSError as error: + if writing: + raise CalibrationStoreError(f"Cannot open {lock_path}: {error}") from error + if handle is None: + yield + return + with handle, _os_lock(handle): + yield + + def _read_unlocked(self, device_class_id: str) -> Optional[StoreRead]: + path = self.path_for(device_class_id) + try: + raw = path.read_bytes() + except FileNotFoundError: + return None + except OSError as error: + raise CalibrationStoreError(f"Cannot read {path}: {error}") from error + file, problem = _parse(raw, device_class_id) + return StoreRead(device_class_id, file, problem) + + def read(self, device_class_id: str) -> Optional[StoreRead]: + """The device class's file, or None when it has none.""" + if not self.path_for(device_class_id).exists(): + return None + with self._locked(device_class_id, writing=False): + return self._read_unlocked(device_class_id) + + def read_raw(self, device_class_id: str) -> Optional[bytes]: + """The file's bytes as they are, readable or not; None when it has none.""" + path = self.path_for(device_class_id) + if not path.exists(): + return None + with self._locked(device_class_id, writing=False): + try: + return path.read_bytes() + except FileNotFoundError: + return None + except OSError as error: + raise CalibrationStoreError(f"Cannot read {path}: {error}") from error + + def change( + self, + device_class_id: str, + apply: Callable[[CalibrationStoreFile], tuple[bool, T]], + *, + create: Optional[CalibrationDeviceClass] = None, + ) -> Optional[T]: + """Apply `apply` to the file as it is on disk now, and write the file when `apply` says it changed it. + + `apply` returns whether it changed the file, and the result to return. Without a file, `create` names the + device class to start one for; without `create` nothing is written and the result is None. Raises + `CalibrationStoreUnusableError` for a file that cannot be read, which stays as it is. + """ + with self._locked(device_class_id, writing=True): + current = self._read_unlocked(device_class_id) + if current is None: + if create is None: + return None + assert create.id == device_class_id + file = CalibrationStoreFile(schema_version=STORE_SCHEMA_VERSION, device_class=create) + elif current.file is None: + raise CalibrationStoreUnusableError( + f"{self.path_for(device_class_id)} is kept as it is and not written: {current.problem}" + ) + else: + file = current.file + changed, result = apply(file) + if changed: + self._write(self.path_for(device_class_id), file) + return result + + @staticmethod + def _write(path: Path, file: CalibrationStoreFile) -> None: + payload = file.model_dump_json(indent=1).encode("utf-8") + try: + descriptor, staging = tempfile.mkstemp(dir=path.parent, prefix=f"{path.name}.", suffix=".tmp") + except OSError as error: + raise CalibrationStoreError(f"Cannot write {path}: {error}") from error + try: + with os.fdopen(descriptor, "wb") as handle: + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + _replace(staging, path) + except BaseException as error: + with suppress(OSError): + os.unlink(staging) + if isinstance(error, OSError): + raise CalibrationStoreError(f"Cannot write {path}: {error}") from error + raise diff --git a/invokeai/app/util/torch_cuda_allocator.py b/invokeai/app/util/torch_cuda_allocator.py index 7e94be57900..25c23478a77 100644 --- a/invokeai/app/util/torch_cuda_allocator.py +++ b/invokeai/app/util/torch_cuda_allocator.py @@ -4,15 +4,30 @@ import os import re import sys +from collections.abc import Mapping from pathlib import Path -# Every variable torch reads its caching-allocator configuration from. `ModelCache._expandable_segments_enabled` keeps -# its own copy of this list: the model cache imports torch, and this module must not. -_ALLOCATOR_CONF_ENV_VARS = ("PYTORCH_ALLOC_CONF", "PYTORCH_CUDA_ALLOC_CONF", "PYTORCH_HIP_ALLOC_CONF") +ALLOCATOR_CONF_ENV_VARS = ("PYTORCH_ALLOC_CONF", "PYTORCH_CUDA_ALLOC_CONF", "PYTORCH_HIP_ALLOC_CONF") +"""Every variable torch reads its caching-allocator configuration from.""" ROCM_WINDOWS_ALLOC_CONF = "expandable_segments:True" +def expandable_segments_enabled(environ: Mapping[str, str] = os.environ) -> bool: + """Whether the torch caching allocator runs in expandable-segments mode, by default in this process. + + The mode is configured through the allocator env vars before torch import (InvokeAI's own + `pytorch_cuda_alloc_conf` setting is plumbed into PYTORCH_CUDA_ALLOC_CONF the same way), so + parsing them is authoritative for the life of the process. It lives here, beside the variables, because this + module imports neither torch nor its callers: the model cache and the working-memory platform. `environ` is + another process's variables, as a working-memory measurement recorded them. + """ + for var in ALLOCATOR_CONF_ENV_VARS: + if "expandable_segments:true" in environ.get(var, "").replace(" ", "").lower(): + return True + return False + + def _installed_torch_version() -> str | None: """The installed torch distribution's version, read from its metadata so torch itself is not imported.""" try: @@ -61,7 +76,7 @@ def apply_rocm_windows_allocator_default(logger: logging.Logger) -> None: Must run before torch is imported. Any allocator variable already in the environment, like an explicit `pytorch_cuda_alloc_conf` (whose caller skips this default), wins. """ - if sys.platform != "win32" or any(os.environ.get(var) for var in _ALLOCATOR_CONF_ENV_VARS): + if sys.platform != "win32" or any(os.environ.get(var) for var in ALLOCATOR_CONF_ENV_VARS): return if not _installed_torch_is_rocm(): return diff --git a/invokeai/backend/model_manager/load/load_base.py b/invokeai/backend/model_manager/load/load_base.py index fddf9a368dc..cdddcbb3ce6 100644 --- a/invokeai/backend/model_manager/load/load_base.py +++ b/invokeai/backend/model_manager/load/load_base.py @@ -26,6 +26,7 @@ ) from invokeai.backend.model_manager.taxonomy import AnyModel, SubModelType from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.working_memory import probe from invokeai.backend.working_memory.estimate import WorkingMemoryEstimate @@ -181,18 +182,57 @@ def _lock_paced(self, working_mem_bytes: Optional[int]) -> None: # invariant turning this loop into a spin. assert settled, "uncapped continue_lock pass did not settle" - def __enter__(self) -> AnyModel: - self._lock_paced(None) - self._end_first_use_window() + def _lock_on_device(self, working_mem_bytes: Optional[int], *, offload_others: bool = False) -> None: + """Lock the model onto its compute device and repair its required tensors there. Nothing stays locked if this + raises. + + While a calibration scope is open anywhere (`probe.armed`), the working-memory probe hears of the lock from its + start until it has settled with the tensors repaired: inside a measured window on this thread that interval is + a co-lock, which moves weights and is kept out of the window's peak; on another thread it voids the window. + The lock is in `probe.locking` from before `armed` is read until it has settled or failed, so a window that + opens on this cache meanwhile is void even if this read missed the scope arming. + + `offload_others`: first move every other unlocked model on the device to RAM whole, within that interval. A + calibration lock asks for it (`model_on_device`): the lock itself offloads only the shortfall it predicts and + splits the last model it moves, and a split can free a little less than predicted, so the measured model would + run partly resident and its window fit no line. Locked models, a window's co-locks among them, stay. + """ + observed = locked = False + probe.locking.append(self._cache) try: + observed = probe.armed + if observed: + probe.lock_started(self._cache, self._cache_record) + if offload_others: + # The global load lock before the cache's own, as `_lock_paced` takes them: a move to RAM loads state + # dicts, which must not overlap a model construction. + with MODEL_LOAD_LOCK.read_lock(): + self._cache.offload_unlocked_models_from_vram(keep_cache_keys=(self._cache_record.key,)) + self._lock_paced(working_mem_bytes) + locked = True + self._end_first_use_window() self.repair_required_tensors_on_device() - return self.model - except Exception: - self._cache.unlock(self._cache_record) + except BaseException: + if locked: + self._cache.unlock(self._cache_record) + locked = False raise + finally: + probe.locking.remove(self._cache) + if observed: + probe.lock_settled(self._cache, self._cache_record, locked) - def __exit__(self, *args: Any, **kwargs: Any) -> None: + def _unlock(self) -> None: self._cache.unlock(self._cache_record) + if probe.armed: + probe.unlocked(self._cache, self._cache_record) + + def __enter__(self) -> AnyModel: + self._lock_on_device(None) + return self.model + + def __exit__(self, *args: Any, **kwargs: Any) -> None: + self._unlock() @contextmanager def model_on_device( @@ -203,27 +243,46 @@ def model_on_device( :param working_mem_bytes: The amount of working memory to keep available on the compute device when loading the model. :param working_memory: A working-memory estimate whose `reservation_bytes` is kept available instead; see - `invokeai.backend.working_memory`. Pass this or `working_mem_bytes`, not both. + `invokeai.backend.working_memory`. Pass this or `working_mem_bytes`, not both. Inside a calibration scope + (`working_memory.probe`) on a GPU, the lock first moves every other unlocked model on the device to RAM, + reserves calibration headroom on top, and the probe measures the operation from the settled lock until the + unlock. """ if isinstance(working_mem_bytes, WorkingMemoryEstimate): raise TypeError( "model_on_device() got a WorkingMemoryEstimate as working_mem_bytes; pass it as working_memory=." ) + calibration: Optional[probe.ProbeScope] = None if working_memory is not None: if working_mem_bytes is not None: raise ValueError("Pass either working_mem_bytes or working_memory to model_on_device(), not both.") working_mem_bytes = working_memory.reservation_bytes - InvokeAILogger.get_logger(__name__).debug( + message = ( f"Working memory for {self._cache_record.key}: {working_memory.estimator} reserves " f"{working_mem_bytes / 2**20:.0f} MB ({working_memory.source})" ) - self._lock_paced(working_mem_bytes) - self._end_first_use_window() + scope = probe.current_scope() + if scope is not None and self.compute_device.type != "cpu": + calibration = scope + working_mem_bytes = probe.calibration_reservation(working_mem_bytes) + message += f"; the calibration lock reserves {working_mem_bytes / 2**20:.0f} MB" + InvokeAILogger.get_logger(__name__).debug(message) + self._lock_on_device(working_mem_bytes, offload_others=calibration is not None) + window = None try: - self.repair_required_tensors_on_device() + if calibration is not None: + assert working_memory is not None and working_mem_bytes is not None + window = calibration.open_window(self._cache, self._cache_record, working_memory, working_mem_bytes) yield (self._cache_record.cached_model.get_cpu_state_dict(), self._cache_record.cached_model.model) + except BaseException as error: + if window is not None: + window.close(error) + window = None + raise finally: - self._cache.unlock(self._cache_record) + if window is not None: + window.close(None) + self._unlock() @property def model(self) -> AnyModel: @@ -234,11 +293,13 @@ def model(self) -> AnyModel: def model_in_ram(self) -> Generator[AnyModel, None, None]: """Pin the model's cache record in RAM without moving the model to its execution device.""" self._cache.lock_in_ram(self._cache_record) - self._end_first_use_window() + if probe.armed: + probe.locked_in_ram(self._cache, self._cache_record) try: + self._end_first_use_window() yield self.model finally: - self._cache.unlock(self._cache_record) + self._unlock() @property def compute_device(self) -> torch.device: diff --git a/invokeai/backend/model_manager/load/model_cache/cache_record.py b/invokeai/backend/model_manager/load/model_cache/cache_record.py index 14fb4879af4..5a30eb74712 100644 --- a/invokeai/backend/model_manager/load/model_cache/cache_record.py +++ b/invokeai/backend/model_manager/load/model_cache/cache_record.py @@ -114,9 +114,11 @@ def lock(self) -> None: self._locks += 1 def unlock(self) -> None: - """Unlock this record.""" + """Unlock this record. An unlock without a lock to release raises and changes nothing, so the count stays + right for the next lock and for the cache's `locks_held`.""" + if self._locks == 0: + raise RuntimeError(f"Model cache entry {self.key} was unlocked more often than it was locked") self._locks -= 1 - assert self._locks >= 0 @property def is_locked(self) -> bool: diff --git a/invokeai/backend/model_manager/load/model_cache/model_cache.py b/invokeai/backend/model_manager/load/model_cache/model_cache.py index 3691f374425..7932c6fdd64 100644 --- a/invokeai/backend/model_manager/load/model_cache/model_cache.py +++ b/invokeai/backend/model_manager/load/model_cache/model_cache.py @@ -1,6 +1,5 @@ import gc import logging -import os import queue import sys import threading @@ -15,6 +14,7 @@ import psutil import torch +from invokeai.app.util.torch_cuda_allocator import expandable_segments_enabled from invokeai.backend.model_manager.load.memory_snapshot import MemorySnapshot from invokeai.backend.model_manager.load.model_cache.cache_record import CacheRecord from invokeai.backend.model_manager.load.model_cache.cache_stats import CacheStats @@ -39,20 +39,6 @@ from invokeai.backend.util.logging import InvokeAILogger from invokeai.backend.util.prefix_logger_adapter import PrefixedLoggerAdapter - -def _expandable_segments_enabled() -> bool: - """Whether the torch caching allocator runs in expandable-segments mode. - - The mode is configured through the allocator env vars before torch import (InvokeAI's own - `pytorch_cuda_alloc_conf` setting is plumbed into PYTORCH_CUDA_ALLOC_CONF the same way), so - parsing them is authoritative for the life of the process. - """ - for var in ("PYTORCH_ALLOC_CONF", "PYTORCH_CUDA_ALLOC_CONF", "PYTORCH_HIP_ALLOC_CONF"): - if "expandable_segments:true" in os.environ.get(var, "").replace(" ", "").lower(): - return True - return False - - # Size of a GB in bytes. GB = 2**30 @@ -630,6 +616,8 @@ def __init__( self._cached_models: Dict[str, CacheRecord] = {} self._cache_stack: List[str] = [] + # Locks on this cache's records, detached ones included (see `locks_held`); changed only with `_lock` held. + self._locks_held = 0 self._ram_cache_size_bytes = self._calc_ram_available_to_model_cache() @@ -1888,6 +1876,28 @@ def _claim_first_use(self, cache_entry: CacheRecord) -> Optional[FirstUseClaim]: self.release_first_use_hold(cache_entry, hold_epoch) raise + @synchronized + def locks_held(self) -> int: + """How many locks this cache's records hold right now: VRAM and RAM pins of every thread together. + + The working-memory probe compares it with the locks its own thread took, so that a window opening while + another thread holds a model on this device knows it shares the device. Counted as locks are taken and + released, not over the cached records: a record dropped while locked (the VRAM-move error paths) still holds + its model on the device until its holder unlocks it. + """ + return self._locks_held + + def _pin(self, cache_entry: CacheRecord) -> None: + """Lock `cache_entry` and count the lock in `locks_held`. Caller holds the lock.""" + cache_entry.lock() + self._locks_held += 1 + + def _unpin(self, cache_entry: CacheRecord) -> None: + """Release a lock `_pin` took. Caller holds the lock. The record refuses an unlock it has no lock for before + either count changes, so a double unlock leaves `locks_held` right.""" + cache_entry.unlock() + self._locks_held -= 1 + @synchronized @record_activity def lock_in_ram(self, cache_entry: CacheRecord) -> None: @@ -1902,7 +1912,7 @@ def lock_in_ram(self, cache_entry: CacheRecord) -> None: "the RAM cache. This is a sign that the model loading order is non-optimal in the invocation code " "(See https://github.com/invoke-ai/InvokeAI/issues/7513)." ) - cache_entry.lock() + self._pin(cache_entry) cache_entry.awaiting_first_use = False @synchronized @@ -1930,7 +1940,7 @@ def lock( "(See https://github.com/invoke-ai/InvokeAI/issues/7513)." ) # cache_entry = self._cached_models[key] - cache_entry.lock() + self._pin(cache_entry) # End of the post-admission grace: from here the entry is pinned by its lock count, and # after the final unlock it is ordinary evictable cache content. cache_entry.awaiting_first_use = False @@ -1966,10 +1976,10 @@ def lock( ) except torch.OutOfMemoryError: self._logger.warning("Insufficient GPU memory to load model. Aborting") - cache_entry.unlock() + self._unpin(cache_entry) raise except Exception: - cache_entry.unlock() + self._unpin(cache_entry) raise self._log_cache_state() @@ -2014,10 +2024,10 @@ def continue_lock( ) except torch.OutOfMemoryError: self._logger.warning("Insufficient GPU memory to load model. Aborting") - cache_entry.unlock() + self._unpin(cache_entry) raise except Exception: - cache_entry.unlock() + self._unpin(cache_entry) raise if settled: @@ -2036,7 +2046,7 @@ def unlock(self, cache_entry: CacheRecord) -> None: "(See https://github.com/invoke-ai/InvokeAI/issues/7513)." ) # cache_entry = self._cached_models[key] - cache_entry.unlock() + self._unpin(cache_entry) self._logger.debug( f"Unlocked model {cache_entry.key} (Type: {cache_entry.cached_model.model.__class__.__name__})" ) @@ -2231,7 +2241,7 @@ def _release_allocator_blocks_for_reserve(self, working_mem_bytes: Optional[int] driver_free, _ = TorchDevice.cuda_mem_get_info(self._execution_device) if driver_free >= reserve: return - if _expandable_segments_enabled(): + if expandable_segments_enabled(): # Freed pages inside a segment are not counted as reclaimable there, but empty_cache() unmaps them. releasable = torch.cuda.memory_reserved(self._execution_device) - torch.cuda.memory_allocated( self._execution_device @@ -2403,7 +2413,7 @@ def _get_reclaimable_allocator_bytes(self) -> int: `_offload_unlocked_models` runs one after each offload in that mode; what it cannot do is make the credit trustworthy before it runs. """ - if _expandable_segments_enabled(): + if expandable_segments_enabled(): return 0 try: if self._execution_device.type == "cuda": @@ -2618,7 +2628,7 @@ def _offload_unlocked_models( # them -- so it is deliberately loop-local: it stops this loop offloading everything, and # the load that follows re-measures for itself (`lock`, `make_room_in_vram`) and streams # from RAM if the room did not materialize. - empty_cache_per_offload = _expandable_segments_enabled() + empty_cache_per_offload = expandable_segments_enabled() vram_bytes_freed_uncounted = 0 # TODO(ryand): Give more thought to the offloading policy used here. cache_entries_increasing_size = sorted(self._cached_models.values(), key=lambda x: x.cached_model.total_bytes()) @@ -3149,6 +3159,23 @@ def offload_models_from_vram_except(self, keep_model_keys: Collection[str]) -> i keep = set(keep_model_keys) return self._offload_entries_from_vram(lambda key: key.split(":", 1)[0] not in keep) + @synchronized + def offload_unlocked_models_from_vram(self, keep_cache_keys: Collection[str] = ()) -> int: + """Move every unlocked entry but those in `keep_cache_keys` from VRAM to RAM whole, keeping it cached. + + `keep_cache_keys` are exact cache keys: keeping a submodel (`:vae`) keeps none of its siblings, unlike + `offload_models_from_vram_except`. Each entry leaves whole, where `lock()` moves out only the shortfall it + predicts and so splits the last model it reaches. A working-memory calibration calls this before its measured + lock (`LoadedModelWithoutConfig.model_on_device`), so that lock has nothing to split. + + Returns the number of VRAM bytes freed. + """ + if not _has_dedicated_vram(self._execution_device): + # CPU, MPS, integrated GPUs: their "VRAM" is system RAM, so moving models out frees nothing. + return 0 + keep = set(keep_cache_keys) + return self._offload_entries_from_vram(lambda key: key not in keep) + def _offload_entries_from_vram(self, select: Callable[[str], bool]) -> int: bytes_freed = 0 for key, entry in list(self._cached_models.items()): diff --git a/invokeai/backend/util/device_pool.py b/invokeai/backend/util/device_pool.py index e9044160689..fbd68f7af16 100644 --- a/invokeai/backend/util/device_pool.py +++ b/invokeai/backend/util/device_pool.py @@ -75,7 +75,10 @@ def set_release_listener(self, listener: Optional[Callable[[], None]]) -> None: self._on_release = listener def set_generation_devices(self, devices: list[torch.device]) -> None: - """Register the full set of generation devices (called once at processor startup). + """Register the full set of generation devices, replacing those registered before (a borrow in flight is + dropped). The session processor's `register_generation_devices` calls it once: from the composition root + (`ApiDependencies.initialize`) before the invoker starts any service, or from the processor's `start` when + nothing did. Only GPU devices participate in idle-offload; others are ignored. """ diff --git a/invokeai/backend/util/devices.py b/invokeai/backend/util/devices.py index b648adc0c8f..8e1c31479a3 100644 --- a/invokeai/backend/util/devices.py +++ b/invokeai/backend/util/devices.py @@ -2,7 +2,7 @@ import threading from collections import Counter, defaultdict from functools import wraps -from typing import Dict, Literal, Optional, Union +from typing import Dict, Literal, Optional, Protocol, Union import torch from deprecated import deprecated @@ -25,6 +25,19 @@ _XPU_MEM_FALLBACK_WARNED: set[str] = set() +class EmptyCacheObserver(Protocol): + def before_empty_cache(self) -> None: ... + + def after_empty_cache(self) -> None: ... + + +EMPTY_CACHE_OBSERVERS: list[EmptyCacheObserver] = [] +"""Told about every `TorchDevice.empty_cache` that releases memory, just before and just after the release, in +registration order. The working-memory probe registers here while a calibration scope is open, so that a release inside +a measured window ends a segment there (`invokeai.backend.working_memory.probe`); otherwise the list is empty and a +release pays one truth test for it. Observers must not raise.""" + + def _xpu_is_available() -> bool: """Return True if a torch XPU (Intel GPU) device is available.""" return hasattr(torch, "xpu") and torch.xpu.is_available() @@ -447,14 +460,21 @@ def empty_cache(cls, force: bool = False) -> bool: # Clear before running: a skip that races in after this point re-sets the flag, so a # request is never lost, only (harmlessly) repeated. cls._empty_cache_deferred.clear() - if torch.backends.mps.is_available(): - torch.mps.empty_cache() - if torch.cuda.is_available(): - # A forced call must get past `install_peer_aware_empty_cache`'s wrapper, which would defer it again. - cuda_empty_cache = torch.cuda.empty_cache - (getattr(cuda_empty_cache, "__wrapped__", cuda_empty_cache) if force else cuda_empty_cache)() - if _xpu_is_available(): - torch.xpu.empty_cache() + observers = tuple(EMPTY_CACHE_OBSERVERS) if EMPTY_CACHE_OBSERVERS else () + for observer in observers: + observer.before_empty_cache() + try: + if torch.backends.mps.is_available(): + torch.mps.empty_cache() + if torch.cuda.is_available(): + # A forced call must get past `install_peer_aware_empty_cache`'s wrapper, which would defer it again. + cuda_empty_cache = torch.cuda.empty_cache + (getattr(cuda_empty_cache, "__wrapped__", cuda_empty_cache) if force else cuda_empty_cache)() + if _xpu_is_available(): + torch.xpu.empty_cache() + finally: + for observer in observers: + observer.after_empty_cache() return True @classmethod diff --git a/invokeai/backend/util/nvml.py b/invokeai/backend/util/nvml.py new file mode 100644 index 00000000000..cffb36efd67 --- /dev/null +++ b/invokeai/backend/util/nvml.py @@ -0,0 +1,146 @@ +"""Minimal NVML queries through ctypes: this process's memory on an NVIDIA device, and the driver version. + +NVML ships with the NVIDIA driver (``libnvidia-ml.so.1`` on Linux, ``nvml.dll`` in System32 on Windows), so this needs +no package. Under WDDM, Windows reports no per-process memory through NVML; the working-memory probe reads it through +D3DKMT there (`wddm.process_local_usage`), and uses NVML on Windows for the driver version only. + +Everything here is best-effort: any failure yields ``None`` ("unknown") rather than raising. +""" + +import ctypes +import functools +import os +import sys +import threading +from typing import Optional + +import torch + +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.util.pci import pci_address + +_NVML_SUCCESS = 0 +_NVML_ERROR_INSUFFICIENT_SIZE = 7 +# `usedGpuMemory` when the driver cannot attribute memory to the process (always so under WDDM). +_NVML_VALUE_NOT_AVAILABLE = 0xFFFFFFFFFFFFFFFF +# NVML_SYSTEM_DRIVER_VERSION_BUFFER_SIZE. +_DRIVER_VERSION_BUFFER_SIZE = 80 +# Room for processes that start between the count and the fill call. +_SPARE_PROCESS_ENTRIES = 8 + +_LIBRARY_NAMES = ("nvml.dll",) if sys.platform == "win32" else ("libnvidia-ml.so.1", "libnvidia-ml.so") + + +class _ProcessInfo(ctypes.Structure): + """``nvmlProcessInfo_t``, the entry ``nvmlDeviceGetComputeRunningProcesses_v3`` fills.""" + + _fields_ = [ + ("pid", ctypes.c_uint), + ("usedGpuMemory", ctypes.c_ulonglong), + ("gpuInstanceId", ctypes.c_uint), + ("computeInstanceId", ctypes.c_uint), + ] + + +_lock = threading.Lock() +# Per torch device index: the NVML device handle, or None once the lookup has failed. +_handles: dict[int, Optional[ctypes.c_void_p]] = {} + + +def _configure_prototypes(lib: ctypes.CDLL) -> None: + """Declare every entry point used here, so ctypes passes handles at full pointer width.""" + prototypes = { + "nvmlInit_v2": [], + "nvmlDeviceGetHandleByPciBusId_v2": [ctypes.c_char_p, ctypes.POINTER(ctypes.c_void_p)], + "nvmlDeviceGetComputeRunningProcesses_v3": [ + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_uint), + ctypes.c_void_p, + ], + "nvmlSystemGetDriverVersion": [ctypes.c_char_p, ctypes.c_uint], + } + for name, argtypes in prototypes.items(): + fn = getattr(lib, name) + fn.argtypes = argtypes + fn.restype = ctypes.c_int + + +@functools.lru_cache(maxsize=1) +def _load() -> Optional[ctypes.CDLL]: + """Open and initialise NVML once per process; None where there is no NVIDIA driver.""" + for name in _LIBRARY_NAMES: + try: + lib = ctypes.CDLL(name) + except OSError: + continue + try: + _configure_prototypes(lib) + if lib.nvmlInit_v2() != _NVML_SUCCESS: + return None + except Exception as exc: + InvokeAILogger.get_logger(__name__).debug(f"NVML unusable: {exc}") + return None + return lib + return None + + +def _handle_for(lib: ctypes.CDLL, device: torch.device) -> Optional[ctypes.c_void_p]: + index = device.index if device.index is not None else torch.cuda.current_device() + with _lock: + if index not in _handles: + handle = ctypes.c_void_p() + found = lib.nvmlDeviceGetHandleByPciBusId_v2(pci_address(index).encode(), ctypes.byref(handle)) + _handles[index] = handle if found == _NVML_SUCCESS else None + return _handles[index] + + +def process_used_memory(device: torch.device) -> Optional[int]: + """Bytes NVML attributes to this process on `device`: everything the process holds there, the CUDA context + included. ``None`` off CUDA builds, without NVML, and wherever the driver does not attribute memory to processes.""" + if device.type != "cuda" or torch.version.hip is not None: + return None + try: + lib = _load() + if lib is None: + return None + handle = _handle_for(lib, device) + if handle is None: + return None + count = ctypes.c_uint(0) + status = lib.nvmlDeviceGetComputeRunningProcesses_v3(handle, ctypes.byref(count), None) + if status != _NVML_ERROR_INSUFFICIENT_SIZE: + # Success with an empty buffer means no process at all, so not this one either. + return None + count = ctypes.c_uint(count.value + _SPARE_PROCESS_ENTRIES) + infos = (_ProcessInfo * count.value)() + if lib.nvmlDeviceGetComputeRunningProcesses_v3(handle, ctypes.byref(count), infos) != _NVML_SUCCESS: + return None + pid = os.getpid() + for info in infos[: count.value]: + if info.pid == pid: + return None if info.usedGpuMemory == _NVML_VALUE_NOT_AVAILABLE else int(info.usedGpuMemory) + return None + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"NVML process memory lookup for {device} failed", exc_info=True) + return None + + +def driver_version() -> Optional[str]: + """The NVIDIA driver version, e.g. ``"581.57"``, or None without NVML.""" + try: + lib = _load() + if lib is None: + return None + buffer = ctypes.create_string_buffer(_DRIVER_VERSION_BUFFER_SIZE) + if lib.nvmlSystemGetDriverVersion(buffer, _DRIVER_VERSION_BUFFER_SIZE) != _NVML_SUCCESS: + return None + return buffer.value.decode("ascii", errors="replace") or None + except Exception: + return None + + +def reset_cache() -> None: + """Forget the loader and the device handles (tests only). NVML itself stays initialised.""" + with _lock: + _handles.clear() + _load.cache_clear() diff --git a/invokeai/backend/util/pci.py b/invokeai/backend/util/pci.py new file mode 100644 index 00000000000..765f9431d03 --- /dev/null +++ b/invokeai/backend/util/pci.py @@ -0,0 +1,17 @@ +"""Where a CUDA or HIP device sits on the PCI bus: what links a torch device to the driver interfaces that know it by +location (D3DKMT adapters, NVML, DRM fdinfo).""" + +import torch + + +def pci_location(index: int) -> tuple[int, int, int]: + """``(domain, bus, device)`` of torch's CUDA or HIP device `index`.""" + props = torch.cuda.get_device_properties(index) + return int(props.pci_domain_id), int(props.pci_bus_id), int(props.pci_device_id) + + +def pci_address(index: int) -> str: + """The device's address as ``domain:bus:device.function`` in lower-case hex, e.g. ``0000:03:00.0``: how DRM + fdinfo spells it, and one of the forms NVML parses.""" + domain, bus, device = pci_location(index) + return f"{domain:04x}:{bus:02x}:{device:02x}.0" diff --git a/invokeai/backend/util/vae_working_memory.py b/invokeai/backend/util/vae_working_memory.py index bc2f3123d64..35016da6631 100644 --- a/invokeai/backend/util/vae_working_memory.py +++ b/invokeai/backend/util/vae_working_memory.py @@ -16,6 +16,7 @@ from invokeai.backend.util.logging import InvokeAILogger from invokeai.backend.util.vae_tiling_scope import resolve_tile_size from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm +from invokeai.backend.working_memory.platform import platform_for from invokeai.backend.working_memory.spec import EstimatorSpec, register # Share of the device's memory an untiled decode may claim before the decode nodes tile it up front (see @@ -77,7 +78,7 @@ def should_pretile_vae_decode( return True -# The diffusers AutoencoderKL (SD1/SDXL, SD3, CogView4) and the FLUX.1 AutoEncoder run the same +# The diffusers AutoencoderKL (SD1/SDXL, SD3) and the FLUX.1 AutoEncoder run the same # mid-block self-attention as the FLUX.2 VAE: one head over the 512-channel width, on the # 8x-downsampled grid. Where that attention runs on the math kernel (see `sdpa_score_matrix_bytes`) # its score matrix is the dominant term for a large untiled image, exactly as it is for FLUX.2; @@ -114,25 +115,85 @@ def _vae_mid_block_score_matrix_bytes( ) -# The SD1/SDXL, SD3, CogView4, FLUX.1 and FLUX.2 VAEs run the same diffusers AutoencoderKL convolution stack, and pay -# the same for it: peak reserved bytes per output pixel per element byte, one fitted coefficient per operation -# (`k_decode`, `k_encode`) and convolution backend. The values ship in `backend/working_memory/data/vae.json`, with the -# points they were fitted on and the per-card table behind them; `scripts/calibrate_flux2_working_memory.py --only vae` -# reproduces that table on any build. +# The SD1/SDXL, SD3, FLUX.1 and FLUX.2 VAEs run the same diffusers AutoencoderKL convolution stack, and pay the same +# for it: a line in the output's pixel count per operation and the dtype it runs in, `k__` +# bytes per pixel plus `b__` bytes the process pays whatever the size, shipped per convolution +# backend and allocator (`platform_for`). The values ship in `backend/working_memory/data/vae.json` with the points +# they are fitted on (`backend/working_memory/fit.py`); `scripts/calibrate_flux2_working_memory.py --only vae` +# measures a build against them. # -# MIOpen's convolution workspaces are simply larger than cuDNN's -- ~1.6x for a decode, ~2.4x for an encode. This is -# not the attention term: it shows up identically on the fused path, and the implied coefficients are flat across -# resolution on both backends, so the linear model itself holds. Nor is "encoding costs half of decoding" architectural: -# it is a cuDNN property (0.49 there, 0.78 on MIOpen), which is why each operation carries its own coefficient instead -# of a ratio. Shipping the MIOpen values everywhere would add ~60% to every cuDNN decode for nothing, so the -# coefficients follow the backend of the device the VAE runs on (`platform_for`). +# None of the three scales out of the line. The demand does not follow the element size: on MIOpen fp16 needs about a +# third of what bf16 does, and on both backends fp32 needs less per value byte than bf16. The allocator matters as +# much: `cudaMallocAsync` needs 25-35% less than the native allocator on cuDNN. And MIOpen's bf16 workspaces are +# larger than cuDNN's, ~1.5x for a decode and ~2.4x for an encode -- not the attention term, which shows up the same +# on the fused path. The base is why the per-pixel figure rises at small sizes; on HIP it includes the ~300 MiB the +# driver maps beyond the allocator's own figure on the first operation of a process. +# +# Each estimator declares the lines of the dtypes it can run, and the estimate prices the dtype the operation will run +# in, which is not always the dtype the cached VAE holds (see `sd_vae_run_dtype`). CogView4's VAE is an AutoencoderKL +# too, but a wider one (block widths up to 1024, three layers per block, no mid-block attention): it carries its own +# lines, about 1.9x the family's for a decode. # # Caveat for AMD users: `MIOPEN_FIND_MODE=2` (FAST) selects convolution algorithms heuristically instead of by # benchmark, and measured a uniform 1.28x more memory at every resolution when the MIOpen values were first fitted. -# Invoke has since made FAST its default (`run_app.py`); the RX 9060 XT measurement was taken under it and did not show -# the 1.28x. If you see decodes exceed their reservation on another card, raise `device_working_mem_gb`. -_AUTOENCODER_KL_COEFFICIENTS = frozenset({"k_decode", "k_encode"}) -_CONV_BACKEND_KEYS = ("conv_backend",) +# Invoke has since made FAST its default (`run_app.py`); the RX 9060 XT points were taken under it. If you see decodes +# exceed their reservation on another card, raise `device_working_mem_gb`. +_PLATFORM_KEYS = ("conv_backend", "allocator") +_DTYPE_NAMES: dict[torch.dtype, str] = {torch.float16: "fp16", torch.bfloat16: "bf16", torch.float32: "fp32"} +_ELEMENT_SIZES = {"fp16": 2, "bf16": 2, "fp32": 4} + + +def _convolution_coefficients(*dtypes: str) -> frozenset[str]: + """The coefficients of the convolution lines for both operations in these dtypes.""" + return frozenset( + f"{part}_{operation}_{dtype}" for part in ("k", "b") for operation in ("decode", "encode") for dtype in dtypes + ) + + +def _dtype_name(dtype: torch.dtype) -> str: + name = _DTYPE_NAMES.get(dtype) + if name is None: + raise ValueError(f"No VAE working-memory coefficients are measured for {dtype}.") + return name + + +def _compute_dtype(vae: torch.nn.Module) -> torch.dtype: + """The dtype a VAE no node casts computes in: that of its first floating-point parameter wider than a byte. + + Narrower or integer weights (fp8 storage, an SDNQ VAE's packed ones) are what a layer dequantizes from, not what + its activations are. + """ + for param in vae.parameters(): + if param.is_floating_point() and param.element_size() > 1: + return param.dtype + raise ValueError(f"{type(vae).__name__} has no floating-point parameters to compute in.") + + +def _estimate(spec: EstimatorSpec, features: Mapping[str, FeatureValue], device: torch.device) -> WorkingMemoryEstimate: + """`spec.estimate`, refusing a dtype the spec has no line for with the estimator, dtype and platform named, rather + than a bare `KeyError` from inside the terms.""" + line = f"k_{features['operation']}_{features['dtype']}" + if line not in spec.coefficients: + raise ValueError( + f"Working-memory estimator '{spec.id}' has no {features['dtype']} {features['operation']} line on " + f"{platform_for(device)}: the VAE computes in a dtype it is not measured for." + ) + return spec.estimate(features, device) + + +def _convolutions( + features: Mapping[str, FeatureValue], values: Mapping[str, float], pixels: float, samples: int = 1 +) -> tuple[WorkingMemoryTerm, ...]: + """The convolution line of this operation and dtype over `pixels` output pixels, for `samples` samples. + + The base scales with the samples too. That is an upper bound rather than a fit -- the base is paid once per process + -- and it keeps a batch at exactly that many single-sample estimates. + """ + line = f"{features['operation']}_{features['dtype']}" + return ( + WorkingMemoryTerm.fitted("convolutions", f"k_{line}", pixels * samples, values), + WorkingMemoryTerm.fitted("convolution_base", f"b_{line}", samples, values), + ) def _peak_of_two_phases( @@ -170,40 +231,60 @@ def _peak_of_two_phases( def _sd_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: - coefficient = f"k_{features['operation']}" - element_size = cast(int, features["element_size"]) tile = cast(int | None, features["tile"]) if tile is not None: - area_bytes = tile * tile * element_size # We add 25% to the working memory estimate when tiling is enabled to account for factors like tile overlap # and number of tiles. We could make this more precise in the future, but this should be good enough for # most use cases. - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * 1.25, values) + terms = _convolutions(features, values, tile * tile * 1.25) else: - area_bytes = cast(int, features["height"]) * cast(int, features["width"]) * element_size - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes, values) + terms = _convolutions(features, values, cast(int, features["height"]) * cast(int, features["width"])) - terms: tuple[WorkingMemoryTerm, ...] = (convolutions,) # The tiny autoencoder has no mid-block attention, so it is priced without a score matrix. if features["vae_class"] == "AutoencoderKL": terms = _peak_of_two_phases(terms, cast(int, features["score_matrix_bytes"])) - if features["fp32"]: - # If we are running in FP32, then we should account for the likely increase in model size (~250MB). - terms += (WorkingMemoryTerm("fp32_weights", 250 * 2**20, True),) + upcast_weight_bytes = cast(int, features["upcast_weight_bytes"]) + if upcast_weight_bytes: + terms += (WorkingMemoryTerm("upcast_weights", upcast_weight_bytes, True),) return terms VAE_SD_SPEC = register( EstimatorSpec( id="vae.sd", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("fp16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_sd_terms, ) ) +def sd_vae_run_dtype(operation: Literal["encode", "decode"], fp32: bool, device: torch.device) -> torch.dtype: + """The dtype the SD1/SDXL nodes cast the shared VAE to before they run it: `l2i` and `i2l` cast with this, and + `estimate_vae_working_memory_sd15_sdxl` prices it, so the two cannot drift. + + fp32 when the node asks for it, and for a decode on the CPU; fp16 otherwise, on every GPU and for every + `precision`, so bf16 never runs. The cached VAE's own dtype says nothing about it: it is whatever the previous run + cast it to. + """ + if fp32 or (operation == "decode" and device.type == "cpu"): + return torch.float32 + return torch.float16 + + +def _upcast_weight_bytes(vae: torch.nn.Module, run_dtype: torch.dtype) -> int: + """What the node's in-place cast to `run_dtype` adds to the VAE's resident weights. + + The cast runs once the cache has placed the VAE, so the growth comes out of the reservation: 160 MiB for the 83.7M + parameters of the SD1/SDXL VAE going from fp16 to fp32, nothing for a VAE a previous run left in fp32. Measured + points exclude resident weights by construction (they record 0 here), so the fitted line does not cover it. An + upcast encode casts the decoder's mid-block and post-quant conv back, so for it this counts a little more than + grows: conservative. + """ + return sum(p.numel() * max(0, run_dtype.itemsize - p.element_size()) for p in vae.parameters()) + + def estimate_vae_working_memory_sd15_sdxl( operation: Literal["encode", "decode"], image_tensor: torch.Tensor, @@ -214,12 +295,13 @@ def estimate_vae_working_memory_sd15_sdxl( ) -> WorkingMemoryEstimate: """Estimate the working memory required to encode or decode the given tensor. - It was found experimentally that the peak working memory scales linearly with the number of pixels and the element - size (precision). This estimate is accurate for both SD1 and SDXL. `device` is where the VAE will run; it picks the - convolution backend's coefficients and defaults to the session device. + It was found experimentally that the peak working memory is a line in the number of pixels, per dtype. This + estimate is accurate for both SD1 and SDXL. It prices the dtype the node will cast the VAE to (`sd_vae_run_dtype`), + not the one the cached VAE holds. `device` is where the VAE will run; it picks the platform's coefficients and + defaults to the session device. """ - element_size = 4 if fp32 else 2 device = device if device is not None else TorchDevice.choose_torch_device() + run_dtype = sd_vae_run_dtype(operation, fp32, device) latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 height = latent_scale_factor_for_operation * image_tensor.shape[-2] width = latent_scale_factor_for_operation * image_tensor.shape[-1] @@ -233,26 +315,23 @@ def estimate_vae_working_memory_sd15_sdxl( "height": height, "width": width, "tile": tile_size, - "element_size": element_size, - "fp32": fp32, + "dtype": _dtype_name(run_dtype), "vae_class": "AutoencoderKL" if isinstance(vae, AutoencoderKL) else "AutoencoderTiny", + "upcast_weight_bytes": _upcast_weight_bytes(vae, run_dtype), } if isinstance(vae, AutoencoderKL): - score_dtype = torch.float32 if fp32 else next(vae.parameters()).dtype score_h, score_w = (tile_size, tile_size) if tile_size is not None else (height, width) - features["score_matrix_bytes"] = _vae_mid_block_score_matrix_bytes(score_h, score_w, score_dtype, device=device) + features["score_matrix_bytes"] = _vae_mid_block_score_matrix_bytes(score_h, score_w, run_dtype, device=device) - return VAE_SD_SPEC.estimate(features, device) + return _estimate(VAE_SD_SPEC, features, device) def _single_pass_terms( features: Mapping[str, FeatureValue], values: Mapping[str, float] ) -> tuple[WorkingMemoryTerm, ...]: """An untiled AutoencoderKL pass: the convolutions over the whole image, or the mid-block score matrix.""" - coefficient = f"k_{features['operation']}" - area_bytes = cast(int, features["height"]) * cast(int, features["width"]) * cast(int, features["element_size"]) - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes, values) - return _peak_of_two_phases((convolutions,), cast(int, features["score_matrix_bytes"])) + convolutions = _convolutions(features, values, cast(int, features["height"]) * cast(int, features["width"])) + return _peak_of_two_phases(convolutions, cast(int, features["score_matrix_bytes"])) def _single_pass_features( @@ -261,23 +340,30 @@ def _single_pass_features( latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 height = latent_scale_factor_for_operation * image_tensor.shape[-2] width = latent_scale_factor_for_operation * image_tensor.shape[-1] - param = next(vae.parameters()) + dtype = _compute_dtype(vae) return { "operation": operation, "height": height, "width": width, - "element_size": param.element_size(), - "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(height, width, param.dtype, device=device), + "dtype": _dtype_name(dtype), + "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(height, width, dtype, device=device), } +def _cogview4_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: + """An untiled CogView4 VAE pass: its convolutions over the whole image. It has no mid-block attention + (`mid_block_add_attention: false`), so there is no score matrix to price.""" + return _convolutions(features, values, cast(int, features["height"]) * cast(int, features["width"])) + + +# The CogView4 loader forces bf16: the VAE produces black images in fp16. VAE_COGVIEW4_SPEC = register( EstimatorSpec( id="vae.cogview4", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, - terms=_single_pass_terms, + version=2, + coefficients=_convolution_coefficients("bf16"), + platform_keys=_PLATFORM_KEYS, + terms=_cogview4_terms, ) ) @@ -294,7 +380,14 @@ def estimate_vae_working_memory_cogview4( device. """ device = device if device is not None else TorchDevice.choose_torch_device() - return VAE_COGVIEW4_SPEC.estimate(_single_pass_features(operation, image_tensor, vae, device), device) + latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 + features: dict[str, FeatureValue] = { + "operation": operation, + "height": latent_scale_factor_for_operation * image_tensor.shape[-2], + "width": latent_scale_factor_for_operation * image_tensor.shape[-1], + "dtype": _dtype_name(_compute_dtype(vae)), + } + return _estimate(VAE_COGVIEW4_SPEC, features, device) # What a tiled decode does *not* bound: the assembled image, several times over, per output pixel. @@ -330,28 +423,23 @@ def estimate_vae_working_memory_cogview4( def _flux_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: operation = features["operation"] - coefficient = f"k_{operation}" out_h = cast(int, features["height"]) out_w = cast(int, features["width"]) - element_size = cast(int, features["element_size"]) + element_size = _ELEMENT_SIZES[cast(str, features["dtype"])] tile = cast(int | None, features["tile"]) if tile is None: - area_bytes = out_h * out_w * element_size - convolution_phase: tuple[WorkingMemoryTerm, ...] = ( - WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes, values), - ) + convolution_phase = _convolutions(features, values, out_h * out_w) return _peak_of_two_phases(convolution_phase, cast(int, features["score_matrix_bytes"])) # `calc_tiles_min_overlap` clamps the tile to the image *per axis*, so the largest tile the # decode builds is this, not `tile` squared. Without the clamp an 8192px tile on a # 1024x8200 image -- tiled, because one axis exceeds it -- prices 8192x8192. - area_bytes = min(tile, out_h) * min(tile, out_w) * element_size # A 25% margin for tile overlap and the number of tiles, mirroring the SD1/SDXL estimator. - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * 1.25, values) + convolutions = _convolutions(features, values, min(tile, out_h) * min(tile, out_w) * 1.25) if operation == "decode": convolution_phase = ( - convolutions, + *convolutions, WorkingMemoryTerm( "assembled_image", out_h * out_w * _IMAGE_CHANNELS * (_FLUX_VAE_TILED_IMAGE_COPIES * element_size + 1), @@ -377,19 +465,21 @@ def _flux_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float * element_size ) convolution_phase = ( - convolutions, + *convolutions, WorkingMemoryTerm("input_image", out_h * out_w * _IMAGE_CHANNELS * element_size, True), WorkingMemoryTerm("assembled_moments", moments * live_moment_copies, True), ) return _peak_of_two_phases(convolution_phase, cast(int, features["score_matrix_bytes"])) +# The FLUX.1 loaders avoid fp16, which this autoencoder is broken in, except one: an SDNQ VAE folder loads through the +# generic VAE loader and computes in the configured precision, which `auto` makes fp16 on CUDA. VAE_FLUX_SPEC = register( EstimatorSpec( id="vae.flux", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("fp16", "bf16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_flux_terms, ) ) @@ -418,7 +508,7 @@ def estimate_vae_working_memory_flux( backend's coefficients. Defaults to the session device. """ latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 - param = next(vae.parameters()) + dtype = _compute_dtype(vae) device = device if device is not None else TorchDevice.choose_torch_device() out_h = latent_scale_factor_for_operation * image_tensor.shape[-2] @@ -443,9 +533,9 @@ def estimate_vae_working_memory_flux( "operation": operation, "height": out_h, "width": out_w, - "element_size": param.element_size(), + "dtype": _dtype_name(dtype), "tile": tile, - "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(score_h, score_w, param.dtype, device=device), + "score_matrix_bytes": _vae_mid_block_score_matrix_bytes(score_h, score_w, dtype, device=device), } if tile is not None and operation != "decode": # Read from the VAE rather than hard-coded: an overlap factor of 0.5 would make the tiles @@ -454,7 +544,7 @@ def estimate_vae_working_memory_flux( features["latent_channels"] = getattr(getattr(vae, "config", None), "latent_channels", 16) features["tile_overlap_factor"] = getattr(vae, "tile_overlap_factor", 0.25) - return VAE_FLUX_SPEC.estimate(features, device) + return _estimate(VAE_FLUX_SPEC, features, device) # The FLUX.2 VAE runs one attention block at the bottom of the encoder and one at the top of the @@ -468,29 +558,26 @@ def estimate_vae_working_memory_flux( def _flux2_terms(features: Mapping[str, FeatureValue], values: Mapping[str, float]) -> tuple[WorkingMemoryTerm, ...]: - coefficient = f"k_{features['operation']}" - element_size = cast(int, features["element_size"]) batch_size = cast(int, features["batch_size"]) tile = cast(int | None, features["tile"]) height, width = cast(int, features["height"]), cast(int, features["width"]) if tile is not None: - # The largest slice `AutoencoderKLFlux2` takes: the tile, clamped to the image per axis. - area_bytes = min(tile, height) * min(tile, width) * element_size - # Add 25% for tile overlap and the blending buffers, mirroring the SD1/SDXL estimate. - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * 1.25 * batch_size, values) + # The largest slice `AutoencoderKLFlux2` takes: the tile, clamped to the image per axis. Add 25% for tile + # overlap and the blending buffers, mirroring the SD1/SDXL estimate. + convolutions = _convolutions(features, values, min(tile, height) * min(tile, width) * 1.25, samples=batch_size) else: - area_bytes = height * width * element_size - convolutions = WorkingMemoryTerm.fitted("convolutions", coefficient, area_bytes * batch_size, values) + convolutions = _convolutions(features, values, height * width, samples=batch_size) # Both terms are per sample; the larger is reserved (see `_peak_of_two_phases` for why, and the measurements). - return _peak_of_two_phases((convolutions,), cast(int, features["score_matrix_bytes"])) + return _peak_of_two_phases(convolutions, cast(int, features["score_matrix_bytes"])) +# The FLUX.2 loaders avoid fp16 (as FLUX.1's do), and none of them goes through the generic VAE loader. VAE_FLUX2_SPEC = register( EstimatorSpec( id="vae.flux2", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("bf16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_flux2_terms, ) ) @@ -505,16 +592,16 @@ def estimate_vae_working_memory_flux2( ) -> WorkingMemoryEstimate: """Estimate the working memory required to encode or decode with the FLUX.2 (32-channel) VAE. - Peak memory scales linearly with pixel area and element size, as it does for the FLUX.1 VAE, and - the implied coefficient is flat across 512-1536px on every backend measured. What is *not* constant - is the coefficient itself: MIOpen's convolution workspaces cost ~1.6x cuDNN's for a decode and ~2.4x - for an encode, so it is shipped per backend -- see `_AUTOENCODER_KL_COEFFICIENTS` for the caveats - and `backend/working_memory/data/vae.json` for the fitted values and their points. Peak *reserved* - memory is what is measured throughout, the conservative quantity that includes allocator overhead. + Peak memory is a line in pixel area per dtype, as it is for the FLUX.1 VAE, from 512 to 2048px on + every backend measured. The line itself is not one: MIOpen's bf16 convolution workspaces cost ~1.5x + cuDNN's for a decode and ~2.4x for an encode, and the allocator and the dtype move it as much, so it + is shipped per platform and dtype -- see `_convolution_coefficients` for the caveats and + `backend/working_memory/data/vae.json` for the fitted values and their points. The points measure + peak *reserved* memory or more, the conservative quantity that includes allocator overhead. - For reference, decoding 1024x1024 peaks at ~4.3GB on cuDNN and ~6.6GB on MIOpen, and 1536x1536 at - ~9.6GB -- far above the default ``device_working_mem_gb``, which is why this estimate must be - passed to the model cache. + For reference, decoding 1024x1024 in bf16 peaks at ~4.3GB on cuDNN and ~6.6GB on MIOpen, and + 1536x1536 at ~9.6GB -- far above the default ``device_working_mem_gb``, which is why this estimate + must be passed to the model cache. That linear term holds only while ``AutoencoderKLFlux2``'s mid-block attention runs through a fused SDPA kernel, which is what CUDA does (verified: the memory-efficient kernel takes the @@ -546,7 +633,7 @@ def estimate_vae_working_memory_flux2( bound. The score matrix is shaped (batch, heads, S, S), so it scales the same way. The encode call sites all pass batch 1 today; the shared estimator does not assume it. """ - param = next(vae.parameters()) + dtype = _compute_dtype(vae) device = device if device is not None else TorchDevice.choose_torch_device() batch_size = image_tensor.shape[0] if image_tensor.dim() >= 4 else 1 latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 @@ -570,17 +657,17 @@ def estimate_vae_working_memory_flux2( "width": out_w, "tile": tile, "batch_size": batch_size, - "element_size": param.element_size(), + "dtype": _dtype_name(dtype), "score_matrix_bytes": sdpa_score_matrix_bytes( device=device, - dtype=param.dtype, + dtype=dtype, # The score matrix is (batch, heads, S, S); one head per sample prices the whole batch. num_heads=_FLUX2_VAE_MID_BLOCK_HEADS * batch_size, head_dim=_FLUX2_VAE_MID_BLOCK_HEAD_DIM, seq_len=mid_block_seq_len, ), } - return VAE_FLUX2_SPEC.estimate(features, device) + return _estimate(VAE_FLUX2_SPEC, features, device) def estimate_vae_working_memory_anima( @@ -1077,12 +1164,14 @@ def estimate_vae_working_memory_qwen_image( return int(working_memory) +# The SD3 VAE loads in the configured precision and runs uncast: `auto` gives fp16 on CUDA, and bf16 and fp32 are +# honoured. It carries the family's values in `data/vae.json`, as the one member that runs all three. VAE_SD3_SPEC = register( EstimatorSpec( id="vae.sd3", - version=1, - coefficients=_AUTOENCODER_KL_COEFFICIENTS, - platform_keys=_CONV_BACKEND_KEYS, + version=2, + coefficients=_convolution_coefficients("fp16", "bf16", "fp32"), + platform_keys=_PLATFORM_KEYS, terms=_single_pass_terms, ) ) @@ -1100,4 +1189,32 @@ def estimate_vae_working_memory_sd3( device. """ device = device if device is not None else TorchDevice.choose_torch_device() - return VAE_SD3_SPEC.estimate(_single_pass_features(operation, image_tensor, vae, device), device) + return _estimate(VAE_SD3_SPEC, _single_pass_features(operation, image_tensor, vae, device), device) + + +def estimate_untiled_vae_decode( + estimator: str, height: int, width: int, dtype: str, device: torch.device +) -> WorkingMemoryEstimate: + """The estimate an untiled single-image decode of `height` x `width` output pixels in `dtype` gets from the decode + node that prices with `estimator`, on `device`: the features each estimator builds from its VAE, without one. + + For planning sizes before anything is loaded (a calibration plan). The SD1/SDXL VAE is priced as one already in + `dtype`, so without the growth of an in-place cast, which a first fp32 decode after an fp16 one adds on top. + """ + torch_dtype = {name: candidate for candidate, name in _DTYPE_NAMES.items()}.get(dtype) + if torch_dtype is None: + raise ValueError(f"No VAE working-memory coefficients are measured for {dtype}.") + features: dict[str, FeatureValue] = {"operation": "decode", "height": height, "width": width, "dtype": dtype} + if estimator == VAE_COGVIEW4_SPEC.id: + return _estimate(VAE_COGVIEW4_SPEC, features, device) + features["score_matrix_bytes"] = _vae_mid_block_score_matrix_bytes(height, width, torch_dtype, device=device) + if estimator == VAE_SD3_SPEC.id: + return _estimate(VAE_SD3_SPEC, features, device) + if estimator == VAE_FLUX_SPEC.id: + return _estimate(VAE_FLUX_SPEC, features | {"tile": None}, device) + if estimator == VAE_FLUX2_SPEC.id: + return _estimate(VAE_FLUX2_SPEC, features | {"tile": None, "batch_size": 1}, device) + if estimator == VAE_SD_SPEC.id: + sd_features = features | {"tile": None, "vae_class": "AutoencoderKL", "upcast_weight_bytes": 0} + return _estimate(VAE_SD_SPEC, sd_features, device) + raise ValueError(f"No untiled decode is priced by working-memory estimator '{estimator}'.") diff --git a/invokeai/backend/util/wddm.py b/invokeai/backend/util/wddm.py index ec39f7b7389..445cbe1ffd4 100644 --- a/invokeai/backend/util/wddm.py +++ b/invokeai/backend/util/wddm.py @@ -21,6 +21,11 @@ Whether Windows has actually paged this process out is not visible through D3DKMT (``CurrentUsage`` counts paged bytes as local); it is visible in the ``GPU Process Memory`` performance counters Task Manager shows, read here through PDH. +The working-memory probe measures what an operation needs on CUDA and on ROCm alike, so `process_local_usage` (this +process's ``CurrentUsage``) and `process_paged_bytes` answer on either build under Windows; on an RTX 4090 with torch +2.13 the usage follows torch's reserved bytes to three decimals. The ROCm-only readers the model cache decides with keep +their gate. + Everything here is best-effort: any failure yields ``None`` ("unknown") rather than raising, and callers must keep their existing behaviour when the answer is unknown. """ @@ -36,6 +41,7 @@ import torch from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.util.pci import pci_location _KMTQAITYPE_ADAPTERADDRESS = 6 _MEMORY_SEGMENT_GROUP_LOCAL = 0 @@ -149,8 +155,12 @@ class _Adapter: _pdh_queries: dict[str, Optional[tuple[ctypes.c_void_p, ctypes.c_void_p]]] = {} +def _windows_gpu(device: torch.device) -> bool: + return sys.platform == "win32" and device.type == "cuda" + + def _supported(device: torch.device) -> bool: - return sys.platform == "win32" and torch.version.hip is not None and device.type == "cuda" + return _windows_gpu(device) and torch.version.hip is not None @functools.lru_cache(maxsize=1) @@ -211,11 +221,11 @@ def _close(lib: ctypes.CDLL, handle: int) -> None: def _resolve_adapter(lib: ctypes.CDLL, index: int) -> Optional[_Adapter]: """Open the WDDM adapter at the torch device's PCI location, closing every other handle the enumeration opened. - ROCm's device properties carry no LUID, so the PCI bus and device numbers are the link. Anything but exactly one - match is treated as unknown rather than guessed. + torch's device properties carry no LUID, on ROCm or CUDA, so the PCI bus and device numbers are the link. Anything + but exactly one match is treated as unknown rather than guessed. """ props = torch.cuda.get_device_properties(index) - location = (int(props.pci_bus_id), int(props.pci_device_id)) + _domain, bus, device_number = pci_location(index) enum = _EnumAdapters2(0, None) if lib.D3DKMTEnumAdapters2(ctypes.byref(enum)) != 0: @@ -241,9 +251,9 @@ def _resolve_adapter(lib: ctypes.CDLL, index: int) -> Optional[_Adapter]: ctypes.cast(ctypes.byref(address), ctypes.c_void_p), ctypes.sizeof(address), ) - if ( - lib.D3DKMTQueryAdapterInfo(ctypes.byref(query)) == 0 - and (address.BusNumber, address.DeviceNumber) == location + if lib.D3DKMTQueryAdapterInfo(ctypes.byref(query)) == 0 and (address.BusNumber, address.DeviceNumber) == ( + bus, + device_number, ): matches.append(info) else: @@ -293,25 +303,61 @@ def local_video_memory(device: torch.device) -> Optional[tuple[int, Optional[int if not _supported(device): return None try: - resolved = _adapter_for(device) - if resolved is None: - return None - lib, adapter = resolved - info = _QueryVideoMemoryInfo(None, adapter.handle, _MEMORY_SEGMENT_GROUP_LOCAL) - if lib.D3DKMTQueryVideoMemoryInfo(ctypes.byref(info)) != 0: + queried = _query_local_memory(device) + if queried is None: return None + info, adapter = queried if not 0 < info.Budget <= adapter.total_bytes: return None - # `CurrentUsage` counts bytes Windows has already paged out, so it exceeds the adapter's own size exactly when - # the process over-commits. It says nothing about residency then, so it is reported as unknown; the budget - # stands, since the model cache's free-VRAM cap is the caller that needs it most in that state. - usage = int(info.CurrentUsage) - return int(info.Budget), usage if 0 <= usage <= adapter.total_bytes else None + # The budget stands when the usage is unknown (see `_plausible_usage`): the model cache's free-VRAM cap is the + # caller that needs it most in exactly that state. + return int(info.Budget), _plausible_usage(info, adapter) except Exception: InvokeAILogger.get_logger(__name__).debug(f"WDDM memory lookup for {device} failed", exc_info=True) return None +def _query_local_memory(device: torch.device) -> Optional[tuple[_QueryVideoMemoryInfo, _Adapter]]: + """This process's local-segment memory info on the device's adapter, with the adapter; None if the driver cannot + answer.""" + resolved = _adapter_for(device) + if resolved is None: + return None + lib, adapter = resolved + info = _QueryVideoMemoryInfo(None, adapter.handle, _MEMORY_SEGMENT_GROUP_LOCAL) + if lib.D3DKMTQueryVideoMemoryInfo(ctypes.byref(info)) != 0: + return None + return info, adapter + + +def _plausible_usage(info: _QueryVideoMemoryInfo, adapter: _Adapter) -> Optional[int]: + """`CurrentUsage`, or None when it exceeds the adapter's own size. + + It counts bytes Windows has already paged out, so it exceeds the adapter exactly when the process over-commits, + and says nothing about residency then. + """ + usage = int(info.CurrentUsage) + return usage if 0 <= usage <= adapter.total_bytes else None + + +def process_local_usage(device: torch.device) -> Optional[int]: + """This process's ``CurrentUsage`` in the local segment of the device's adapter, in bytes, on a CUDA or a ROCm + build under Windows: everything the process has committed there, the allocator's cached blocks, the libraries' own + allocations and any bytes Windows paged out included. + + ``None`` off Windows, for a device that is not a GPU, when the driver cannot answer, and when the usage exceeds the + adapter. + """ + if not _windows_gpu(device): + return None + try: + queried = _query_local_memory(device) + return None if queried is None else _plausible_usage(*queried) + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"WDDM usage lookup for {device} failed", exc_info=True) + return None + + def video_memory_budget(device: torch.device) -> Optional[int]: """This process's WDDM budget on the device's adapter, in bytes: what Windows keeps resident before paging.""" local = local_video_memory(device) @@ -377,6 +423,13 @@ def paged_bytes(device: torch.device) -> Optional[int]: """ if not _supported(device): return None + return process_paged_bytes(device) + + +def process_paged_bytes(device: torch.device) -> Optional[int]: + """`paged_bytes` on a CUDA or a ROCm build under Windows; ``None`` elsewhere.""" + if not _windows_gpu(device): + return None resolved = _adapter_for(device) if resolved is None: return None diff --git a/invokeai/backend/working_memory/bounds.py b/invokeai/backend/working_memory/bounds.py new file mode 100644 index 00000000000..59116ab7d85 --- /dev/null +++ b/invokeai/backend/working_memory/bounds.py @@ -0,0 +1,93 @@ +"""How a measured point bounds the estimate for its features. + +One rule, read by the data test (`tests/backend/working_memory/test_shipped_coefficients.py`), which holds the shipped +data to it, and by `scripts/working_memory/build_shipped.py`, which refuses to write data that breaks it: + +- Lower bound. The model cache consumes an estimate as ``free >= estimate``, so the estimate must be at least the point. + Where only analytic terms decide it (a materialized score matrix), the cache's `device_working_mem_gb` floor counts. +- Upper bound, for points that hold one (`MeasuredPoint.holds_upper_bound`): where a fitted term decides an in-fit + point, the estimate is at most `MAX_OVER_RESERVATION` times the point, its listed `UPPER_BOUND_EXCEPTIONS` ratio, or + its tighter `max_ratio`, which applies whichever term decides. A reservation the operation never uses is model + weights evicted for nothing. +""" + +from collections.abc import Iterable, Mapping +from types import MappingProxyType +from typing import TYPE_CHECKING, Final, Optional + +from invokeai.app.services.config.config_default import InvokeAIAppConfig +from invokeai.backend.working_memory.coefficients import MeasuredPoint, WorkingMemoryData, resolve_coefficients +from invokeai.backend.working_memory.estimate import WorkingMemoryEstimate +from invokeai.backend.working_memory.platform import Platform + +if TYPE_CHECKING: + from invokeai.backend.working_memory.spec import EstimatorSpec + +MAX_OVER_RESERVATION: Final = 1.3 +"""How far above a measured point a reservation may sit where a fitted coefficient decides it, unless the point is +listed in `UPPER_BOUND_EXCEPTIONS` or carries a tighter `max_ratio`.""" + +WORKING_MEMORY_FLOOR: Final = int(InvokeAIAppConfig.model_fields["device_working_mem_gb"].default * 2**30) +"""What `ModelCache._working_mem_reserve` floors every reservation at. The field default, not a config instance an +environment variable could move.""" + +UPPER_BOUND_EXCEPTIONS: Final[Mapping[str, float]] = MappingProxyType( + { + "vae.sd-miopen,native,windows-decode-fp16-1280x1280-RX9060XT-demand_peak": 1.33, + "vae.sd-miopen,native,windows-encode-fp16-1024x1024-RX9060XT-demand_peak": 1.57, + "vae.sd-miopen,native,windows-encode-fp16-1280x1280-RX9060XT-demand_peak": 1.78, + "vae.flux-miopen,native,windows-decode-bf16-512x512-RX9060XT-demand_peak": 1.33, + "vae.flux2-miopen,native,windows-decode-bf16-512x512-RX9060XT-demand_peak": 1.33, + } +) +"""In-fit points, by `point_id`, whose envelope sits more than 1.3x above them, with the ratio they are held to +instead. All are MIOpen with the native allocator on Windows, where the demand is concave in the size: first-run driver +spikes at small sizes and a second run reserving up to 1.6x the first at large ones, so a line that covers every point +cannot stay within 1.3x of all of them. Windows ROCm defaults to expandable segments; this is the override's price.""" + + +def point_id(point: MeasuredPoint) -> str: + """A readable id of a point: estimator, platform, operation, dtype, size, tile, batch, card and method.""" + features = point.features + shape = f"{features.get('height')}x{features.get('width')}" + tile = f"-tile{features['tile']}" if features.get("tile") is not None else "" + batch = f"-b{features['batch_size']}" if features.get("batch_size", 1) != 1 else "" + out_of_fit = "-outfit" if not point.in_fit else "" + platform = ",".join(point.platform.values()) + card = point.device.split(" (")[0].replace(" ", "") + return ( + f"{point.estimator}-{platform}-{features.get('operation')}-{features.get('dtype')}-{shape}{tile}{batch}" + f"{out_of_fit}-{card}-{point.method}" + ) + + +def platform_of(point: MeasuredPoint) -> Platform: + """The platform a point is evaluated on. One that names no OS measured what holds on every OS.""" + return Platform(**({"os": "other"} | point.platform)) # type: ignore[arg-type] + + +def evaluate_point( + point: MeasuredPoint, spec: "EstimatorSpec", data: Iterable[WorkingMemoryData] +) -> WorkingMemoryEstimate: + """The estimate for the point's features, priced with the coefficients `data` gives its platform.""" + return spec.evaluate(point.features, resolve_coefficients(data, spec, platform_of(point), origin="data")) + + +def fitted_decides(estimate: WorkingMemoryEstimate) -> bool: + return any(term.included and term.coefficient is not None for term in estimate.terms) + + +def bound_violation(point: MeasuredPoint, estimate: WorkingMemoryEstimate) -> Optional[str]: + """Why `estimate` breaks a bound `point` sets for it, or None.""" + fitted = fitted_decides(estimate) + reserved = estimate.reservation_bytes if fitted else max(estimate.reservation_bytes, WORKING_MEMORY_FLOOR) + if reserved < point.measured_bytes: + return f"reserves {reserved / 2**30:.3f} GiB for a measured {point.measured_bytes / 2**30:.3f} GiB" + # An out-of-fit point checks another term, and a lower-bound-only point is no cold first run: neither caps. + if point.holds_upper_bound and (point.max_ratio is not None or (fitted and point.in_fit)): + default = UPPER_BOUND_EXCEPTIONS.get(point_id(point), MAX_OVER_RESERVATION) + max_ratio = point.max_ratio if point.max_ratio is not None else default + ratio = estimate.reservation_bytes / point.measured_bytes + if ratio > max_ratio: + return f"reserves {ratio:.2f}x the measured peak, over {max_ratio}x" + return None diff --git a/invokeai/backend/working_memory/coefficients.py b/invokeai/backend/working_memory/coefficients.py index d8e07b13854..36da2000d27 100644 --- a/invokeai/backend/working_memory/coefficients.py +++ b/invokeai/backend/working_memory/coefficients.py @@ -1,24 +1,25 @@ """The coefficients estimators are evaluated with, and the measured points that bound them. Both ship as JSON under `data/`, read through `importlib.resources` so a wheel install finds them. A coefficient entry -belongs to one estimator version on one measured platform value, and either carries its values or borrows another -entry's (`values_from`); a point records the features of an operation that was measured and its measured peak. -`tests/backend/working_memory/test_shipped_coefficients.py` resolves every entry against its `EstimatorSpec` and holds -every point to the estimate the shipped coefficients give it. +belongs to one estimator version on one measured platform, and either carries its values or borrows another entry's +(`values_from`); a point records the features of an operation that was measured and its measured peak. A carried value +is the fit of the points it is resolved for (`fit.py`) or a declared stand-in (`stand_ins`). +`tests/backend/working_memory/test_shipped_coefficients.py` resolves every entry against its `EstimatorSpec`, holds +every point to the estimate the shipped coefficients give it, and every fitted value to the fit of its points. """ import functools from collections.abc import Iterable, Mapping -from dataclasses import dataclass +from dataclasses import asdict, dataclass from importlib.resources import files from importlib.resources.abc import Traversable from types import MappingProxyType -from typing import TYPE_CHECKING, Literal, Self +from typing import TYPE_CHECKING, Final, Literal, Self from pydantic import BaseModel, ConfigDict, ValidationError, model_validator from invokeai.backend.working_memory.estimate import FeatureValue -from invokeai.backend.working_memory.platform import UNMEASURED_PRICED_AS, Platform +from invokeai.backend.working_memory.platform import Platform, priced_as if TYPE_CHECKING: from invokeai.backend.working_memory.spec import EstimatorSpec @@ -29,7 +30,8 @@ class _DataModel(BaseModel): class ValuesFrom(_DataModel): - """The entry whose values another entry takes: same platform, this estimator id and version.""" + """The entry whose values another entry takes: same platform, this estimator id and version. It may carry more + values than the borrower reads -- a family member that runs fewer dtypes reads fewer of them -- never fewer.""" estimator: str version: int @@ -44,7 +46,15 @@ class CoefficientEntry(_DataModel): values: dict[str, float] | None = None values_from: ValuesFrom | None = None """Borrow the values of another estimator's entry on the same platform, so that its measured points bound this - estimator too. One hop: the entry borrowed from carries its own values.""" + estimator too, and this estimator's points join the fit of those values. One hop: the entry borrowed from carries + its own values.""" + stand_ins: list[str] = [] + """Carried values that are declared conservative stand-ins rather than fits: what this platform prices a line no + in-fit point measures at. `provenance` says what they stand for; every other carried value is a fit.""" + unmeasured_floor: dict[str, float] = {} + """Declared lower bounds, not fits: what carried values are raised to where this entry prices a platform nothing + is measured on (see `priced_as`). A fit describes the platform its points were measured on, not the one it stands + in for. `provenance` says where the floors come from.""" provenance: str """Card, torch build, method and script the values come from, and which points they were fitted on.""" @@ -52,26 +62,62 @@ class CoefficientEntry(_DataModel): def _values_or_values_from(self) -> Self: if (self.values is None) == (self.values_from is None): raise ValueError("a coefficient entry carries exactly one of `values` and `values_from`") + for field, names in (("stand-ins", self.stand_ins), ("unmeasured floors", self.unmeasured_floor)): + unknown = sorted(set(names) - set(self.values or {})) + if unknown: + raise ValueError(f"{field} {unknown} are not values this entry carries") return self +LOWER_BOUND_ONLY_METHODS: Final = frozenset({"probe_demand"}) +"""Point methods that bound an estimate from below only (see `MeasuredPoint.holds_upper_bound`).""" + + class MeasuredPoint(_DataModel): """One measured operation. The estimate for its features must be at least `measured_bytes`, and where a fitted - term decides it, at most 1.3 times that (or `max_ratio`).""" + term decides an in-fit point that `holds_upper_bound`, at most 1.3 times that (or `max_ratio`). Measured with the + model's weights already in place, so `measured_bytes` is the operation's working memory alone.""" estimator: str version: int platform: dict[str, str] features: dict[str, FeatureValue] measured_bytes: int - method: Literal["reserved_peak"] - """``reserved_peak``: the caching allocator's peak reserved bytes over the operation, one fresh process per - point. Reserved rather than allocated, because reserved is what the model cache's reservation has to cover.""" + method: Literal["reserved_peak", "driver_peak", "demand_peak", "probe_demand"] + """How `measured_bytes` was read, each as a peak over its value just before the operation: + + - ``reserved_peak``: the caching allocator's reserved bytes, one fresh process per point. Reserved rather than + allocated, because reserved is what the model cache's reservation has to cover. + - ``driver_peak``: the process's device usage as the driver reports it (Windows ``D3DKMT`` ``CurrentUsage``), the + larger of two runs, one fresh process per point. + - ``demand_peak``: the largest of the driver-level, the reserved and the allocated peak, the larger of two runs, + one fresh process per point. The driver-level figure is what reserved misses: on HIP the first operation of a + process maps ~300 MiB beyond ``max_memory_reserved``. + - ``probe_demand``: the working-memory probe's ``demand_bytes`` (`probe.MeasuredWindow`), the largest over the + valid, fit-eligible windows of calibration runs in a running server, warm-ups included. + `scripts/working_memory/build_shipped.py` builds these points from calibration store exports. A lower bound + only: a warm window in a running server can read well below the cold first run after a server start, which is + the worst case the reservation must cover, so it may raise a line but never caps one.""" device: str provenance: str max_ratio: float | None = None """A tighter upper bound than the default 1.3, as a multiple of `measured_bytes`, for a point the estimate has to stay closer to. It applies whichever term decides the estimate.""" + in_fit: bool = True + """False for a point measured in a regime its platform does not run, kept to check another term: CUDA with the + math attention kernel forced, whose score matrix shares the forward with the convolutions. The fit leaves it out, + and it gets the lower bound only.""" + + @model_validator(mode="after") + def _max_ratio_needs_an_upper_bound(self) -> Self: + if self.max_ratio is not None and not self.holds_upper_bound: + raise ValueError(f"a {self.method} point bounds its estimate from below only; it takes no max_ratio") + return self + + @property + def holds_upper_bound(self) -> bool: + """Whether the point also bounds its estimate from above: a fresh-process peak, not a `probe_demand` window.""" + return self.method not in LOWER_BOUND_ONLY_METHODS class WorkingMemoryData(_DataModel): @@ -123,21 +169,30 @@ def _where(keys: Mapping[str, str]) -> str: return ",".join(f"{key}={value}" for key, value in keys.items()) -def _entry( - entries: list[CoefficientEntry], estimator: str, version: int, keys: Mapping[str, str], where: str +def find_entry( + entries: Iterable[CoefficientEntry], estimator: str, version: int, platform: Mapping[str, str], where: str ) -> CoefficientEntry: - candidates = [entry for entry in entries if entry.estimator == estimator and entry.platform == keys] + """The entry of `estimator` `version` that applies on `platform`: of those whose every platform key it matches, + the most specific one. An entry that also names the OS thereby takes precedence on that OS over the one that does + not (see `Platform`). Raises, naming `where`, if none applies or the most specific is declared twice.""" + candidates = [ + entry + for entry in entries + if entry.estimator == estimator and all(platform.get(key) == value for key, value in entry.platform.items()) + ] matches = [entry for entry in candidates if entry.version == version] - if len(matches) > 1: - raise LookupError( - f"Working-memory coefficients for estimator '{estimator}' version {version} on {where} are declared " - f"{len(matches)} times." - ) if not matches: found = f" (found only version {sorted({entry.version for entry in candidates})})" if candidates else "" raise LookupError( f"No working-memory coefficients for estimator '{estimator}' version {version} on {where}{found}." ) + most_specific = max(len(entry.platform) for entry in matches) + matches = [entry for entry in matches if len(entry.platform) == most_specific] + if len(matches) > 1: + raise LookupError( + f"Working-memory coefficients for estimator '{estimator}' version {version} on {where} are declared " + f"{len(matches)} times." + ) return matches[0] @@ -146,43 +201,51 @@ def resolve_coefficients( ) -> ResolvedCoefficients: """The values `spec` takes on `platform`, and a `source` that says how they were found. - A platform value nothing is measured for is priced as the measured value `UNMEASURED_PRICED_AS` names. The entry - for the spec's id, version and those platform values either carries the values or borrows them, one hop, from - another estimator's entry on the same platform. No other fallback: an estimate priced with coefficients fitted for - something else is the shortfall this data exists to prevent, so a missing, ambiguous, chained or mismatched entry - raises, naming what is wrong. `origin` prefixes the `source`. + A platform nothing is measured on is priced as the measured platform `priced_as` names. The entry for the spec's + id and version that applies there (`find_entry`) either carries the values or borrows them, one hop, from the + entry of another estimator that applies on the same platform. No other fallback: an estimate priced with + coefficients fitted for something else is the shortfall this data exists to prevent, so a missing, ambiguous, + chained or mismatched entry raises, naming what is wrong. `origin` prefixes the `source`. """ + measured = priced_as(platform) + full = {key: str(value) for key, value in asdict(measured).items()} + keys = {key: full[key] for key in spec.platform_keys} requested = {key: getattr(platform, key) for key in spec.platform_keys} - keys = {key: UNMEASURED_PRICED_AS.get(key, {}).get(value, value) for key, value in requested.items()} where = _where(keys) if keys == requested else f"{_where(keys)} (priced for {_where(requested)})" entries = [entry for file in data for entry in file.coefficients] - entry = _entry(entries, spec.id, spec.version, keys, where) - values, lender = entry.values, None + entry = find_entry(entries, spec.id, spec.version, full, where) + carrier, lender = entry, None if entry.values_from is not None: lender = entry.values_from.estimator - lent = _entry(entries, lender, entry.values_from.version, keys, where) - if lent.values is None: + carrier = find_entry(entries, lender, entry.values_from.version, full, where) + if carrier.values is None: raise LookupError( f"Working-memory coefficients for estimator '{spec.id}' on {where} borrow from '{lender}', whose " "entry borrows in turn; borrow from the entry that carries the values." ) - values = lent.values + values = carrier.values assert values is not None, "a validated entry carries values or values_from" - if set(values) != spec.coefficients: + # An entry's own values are exactly what its spec reads; a lender's may be more (see `ValuesFrom`). + matches = set(values) == spec.coefficients if lender is None else spec.coefficients <= set(values) + if not matches: raise LookupError( f"Working-memory coefficients for estimator '{spec.id}' on {where} provide {sorted(values)}" f"{f' (from {lender})' if lender else ''}; the estimator reads {sorted(spec.coefficients)}." ) - priced = ",".join(value for key, value in keys.items() if value != requested[key]) - if priced and lender: - note = f" ({priced} values of {lender})" - elif priced: - note = f" ({priced} values)" - else: - note = f" (values of {lender})" if lender else "" - return ResolvedCoefficients(values=MappingProxyType(dict(values)), source=f"{origin}:{_where(requested)}{note}") + # The source names the spec's keys, and a refining key (the OS) where an entry it read names one. + shown = [*spec.platform_keys, *(key for key in full if key in {*entry.platform, *carrier.platform})] + shown = list(dict.fromkeys(shown)) + read = {name: values[name] for name in sorted(spec.coefficients)} + notes = [f"values of {lender}"] if lender else [] + if measured != platform: + floors = {name: floor for name, floor in carrier.unmeasured_floor.items() if name in read} + read |= {name: max(read[name], floor) for name, floor in floors.items()} + notes.insert(0, f"priced as {_where({key: full[key] for key in shown})}{', floored' if floors else ''}") + note = f" ({', '.join(notes)})" if notes else "" + source = f"{origin}:{_where({key: str(getattr(platform, key)) for key in shown})}{note}" + return ResolvedCoefficients(values=MappingProxyType(read), source=source) @functools.cache diff --git a/invokeai/backend/working_memory/counters.py b/invokeai/backend/working_memory/counters.py new file mode 100644 index 00000000000..2ffeb278722 --- /dev/null +++ b/invokeai/backend/working_memory/counters.py @@ -0,0 +1,352 @@ +"""This process's memory on a compute device, as the working-memory probe reads it at the boundaries of a window. + +Two kinds of counter, and neither alone is an operation's demand: + +- torch's caching allocator: the bytes it reserved from the driver and the bytes live tensors occupy, with exact peaks + since the last reset, on CUDA, ROCm and XPU. MPS keeps current values only; there `PeakSampler` keeps the peaks. +- the driver: what this process holds on the device, including what torch's allocator does not see -- cuBLAS and + cuDNN workspaces, kernel modules, and on ROCm the pool HIP keeps after a free. + +Measured on an RTX 4090 (torch 2.13+cu130, Windows): the driver delta equals torch's reserved-peak delta to three +decimals over VAE decode and encode, matmul, SDPA with and without a mask, and convolutions; first use outside the +allocator adds to the driver alone (+16 MiB at the first matmul, +2 MiB at the first convolution). Under +``backend:cudaMallocAsync`` the reserved peak can sit below the allocated peak (128 vs 136 MiB). On an RX 9060 XT (torch +2.13+rocm10, Windows, expandable segments) the driver keeps HIP's retained pool after ``empty_cache()``, so its delta +reads below reserved (4.3 vs 6.4 GiB for a 1024px decode), except for the first MIOpen decode, which grows it beyond +(2.0 vs 1.72 GiB). So the probe takes the largest delta of the counters it has. + +The driver is read at boundaries only: torch's peaks are exact, so what a boundary read misses is memory outside torch's +allocator that came and went between two boundaries. + +Every reader is best-effort: a failure reads as None. A peak reset or a synchronization that fails raises instead: the +window would go on with peaks from before its boundary or with work still in flight, so the probe, which never lets an +exception reach a generation, marks the window invalid. +""" + +import dataclasses +import sys +import threading +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Final, Literal, Optional + +import torch + +from invokeai.backend.util import nvml, wddm +from invokeai.backend.util.level_zero import xpu_memory_info +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.util.pci import pci_address + +Backend = Literal["cuda", "hip", "mps", "xpu"] + +Method = Literal["windows-d3dkmt", "nvml", "fdinfo", "mps-driver", "xpu-sysman", "torch-only"] +"""Where a reading's driver figures come from. + +- ``windows-d3dkmt``: this process's ``CurrentUsage`` in the adapter's local segment (`wddm.process_local_usage`), on + CUDA and ROCm under Windows, with the paged bytes from PDH. +- ``nvml``: the ``usedGpuMemory`` NVML attributes to this process, on CUDA under Linux. +- ``fdinfo``: ``drm-memory-vram`` summed over this process's DRM clients on the device, on ROCm under Linux. +- ``mps-driver``: ``torch.mps.driver_allocated_memory()``. +- ``xpu-sysman``: the device's used memory from Level Zero Sysman. Device-wide, so it guards against another process + loading the device during a window and is never taken as demand. +- ``torch-only``: no driver figure; the allocator's counters alone. +""" + +VERIFIED_METHODS: Final[frozenset[tuple[Method, Backend]]] = frozenset( + {("windows-d3dkmt", "cuda"), ("windows-d3dkmt", "hip")} +) +"""The (method, backend) pairs whose readings were checked on real hardware: the RTX 4090 and the RX 9060 XT above. The +other readers follow their vendors' documentation, and a window measured with them says so (``verified=False``).""" + +PEAK_SAMPLER_INTERVAL_S: float = 0.003 +"""How often `PeakSampler` samples. A few milliseconds is far below an operation's allocation lifetimes and costs one +getter call per sample.""" + +_FDINFO_DIR = Path("/proc/self/fdinfo") +_FDINFO_UNITS: Final[Mapping[str, int]] = {"": 1, "KiB": 2**10, "MiB": 2**20, "GiB": 2**30} + +# torch.cuda.memory_stats / torch.xpu.memory_stats keys, per CounterReading field. +_STAT_KEYS: Final[Mapping[str, str]] = { + "reserved": "reserved_bytes.all.current", + "allocated": "allocated_bytes.all.current", + "peak_reserved": "reserved_bytes.all.peak", + "peak_allocated": "allocated_bytes.all.peak", + "num_ooms": "num_ooms", + "num_alloc_retries": "num_alloc_retries", +} + + +@dataclass(frozen=True) +class CounterReading: + """One reading of a device's counters for this process, in bytes; None where the platform has no such counter or + its reader failed.""" + + reserved: Optional[int] + allocated: Optional[int] + peak_reserved: Optional[int] + """Since the last `reset_peaks`, like the other peaks.""" + peak_allocated: Optional[int] + num_ooms: Optional[int] + """Allocations that failed, cumulative for the process.""" + num_alloc_retries: Optional[int] + """Allocations that only succeeded after the allocator released its cache and retried, cumulative. A retry + squeezes the peak below what the operation would have taken with room to spare.""" + driver: Optional[int] + """This process's memory on the device as the driver counts it.""" + peak_driver: Optional[int] + """Only where a sampler keeps it (MPS); elsewhere the driver is read at boundaries.""" + paged: Optional[int] + """This process's bytes Windows keeps in shared system memory for the device's adapter.""" + device_used: Optional[int] + """Used memory of the whole device, every process together (XPU).""" + method: Method + + +def backend_for(device: torch.device) -> Optional[Backend]: + """The GPU backend `device` runs on; None for a device that is not a GPU. A HIP build reports ``cuda`` devices.""" + if device.type == "cuda": + return "hip" if torch.version.hip is not None else "cuda" + if device.type in ("mps", "xpu"): + return device.type # type: ignore[return-value] + return None + + +def _allocator_stats(device: torch.device) -> Mapping[str, int]: + try: + if device.type == "cuda": + return torch.cuda.memory_stats(device) + if device.type == "xpu": + return torch.xpu.memory_stats(device) + if device.type == "mps": + return {_STAT_KEYS["allocated"]: torch.mps.current_allocated_memory()} + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"Allocator statistics for {device} unavailable", exc_info=True) + return {} + + +def _fdinfo_size(value: str) -> Optional[int]: + number, _, unit = value.strip().partition(" ") + scale = _FDINFO_UNITS.get(unit.strip()) + return int(number) * scale if number.isdigit() and scale is not None else None + + +def fdinfo_vram_bytes(pdev: str) -> Optional[int]: + """VRAM this process holds on the DRM device at PCI address `pdev`, from ``/proc/self/fdinfo``. + + Each open DRM file reports its client's ``drm-memory-vram``. Duplicated descriptors share a client and report the + same memory, so each ``drm-client-id`` counts once. None when no descriptor of this process is a client of that + device. + """ + try: + entries = list(_FDINFO_DIR.iterdir()) + except OSError: + return None + total: Optional[int] = None + clients: set[str] = set() + for entry in entries: + try: + text = entry.read_text(encoding="utf-8", errors="replace") + except OSError: + # Closed since the listing, or not readable. + continue + fields = {key.strip(): value.strip() for key, _, value in (line.partition(":") for line in text.splitlines())} + if fields.get("drm-pdev", "").lower() != pdev.lower(): + continue + client = fields.get("drm-client-id") + if client is not None: + if client in clients: + continue + clients.add(client) + vram = _fdinfo_size(fields.get("drm-memory-vram", "")) + if vram is not None: + total = vram if total is None else total + vram + return total + + +def driver_method(backend: Optional[Backend], platform_name: Optional[str] = None) -> Method: + """The reader a `backend` device's driver figure comes from on the OS `platform_name` (a ``sys.platform`` value, + this process's by default), ``torch-only`` where there is none. A reading that fails falls back to ``torch-only`` + too.""" + platform_name = sys.platform if platform_name is None else platform_name + if backend in ("cuda", "hip") and platform_name == "win32": + return "windows-d3dkmt" + if backend == "cuda" and platform_name.startswith("linux"): + return "nvml" + if backend == "hip" and platform_name.startswith("linux"): + return "fdinfo" + if backend == "mps": + return "mps-driver" + if backend == "xpu": + return "xpu-sysman" + return "torch-only" + + +def method_verified(backend: Backend, platform_name: Optional[str] = None) -> bool: + """Whether the windows of a `backend` device on this OS read a driver figure verified on real hardware.""" + return (driver_method(backend, platform_name), backend) in VERIFIED_METHODS + + +def _driver_reading(device: torch.device) -> tuple[Optional[int], Optional[int], Optional[int], Method]: + """``(driver, paged, device_used, method)`` for `device` on this platform.""" + backend = backend_for(device) + driver: Optional[int] = None + paged: Optional[int] = None + device_used: Optional[int] = None + method = driver_method(backend) + try: + if method == "windows-d3dkmt": + driver, paged = wddm.process_local_usage(device), wddm.process_paged_bytes(device) + elif method == "nvml": + driver = nvml.process_used_memory(device) + elif method == "fdinfo": + index = device.index if device.index is not None else torch.cuda.current_device() + driver = fdinfo_vram_bytes(pci_address(index)) + elif method == "mps-driver": + driver = int(torch.mps.driver_allocated_memory()) + elif method == "xpu-sysman": + info = xpu_memory_info(device) + device_used = None if info is None else info[1] - info[0] + except Exception: + InvokeAILogger.get_logger(__name__).debug(f"Driver memory reading for {device} failed", exc_info=True) + return None, None, None, "torch-only" + if driver is None and device_used is None: + method = "torch-only" + return driver, paged, device_used, method + + +def read_counters(device: torch.device) -> CounterReading: + """This process's counters on `device` now. Peaks are torch's, since the last `reset_peaks`; MPS has none.""" + stats = _allocator_stats(device) + values = {field: (None if key not in stats else int(stats[key])) for field, key in _STAT_KEYS.items()} + driver, paged, device_used, method = _driver_reading(device) + return CounterReading( + **values, driver=driver, peak_driver=None, paged=paged, device_used=device_used, method=method + ) + + +def reset_peaks(device: torch.device) -> None: + """Start torch's peak counters over at the current values. Cumulative counters (OOMs, retries) are kept. Raises + what torch raises.""" + if device.type == "cuda": + torch.cuda.reset_peak_memory_stats(device) + elif device.type == "xpu": + torch.xpu.reset_peak_memory_stats(device) + + +def synchronize(device: torch.device) -> None: + """Wait for the work queued on `device`, so frees it implies have happened. Raises what torch raises.""" + if device.type == "cuda": + torch.cuda.synchronize(device) + elif device.type == "xpu": + torch.xpu.synchronize(device) + elif device.type == "mps": + torch.mps.synchronize() + + +def _larger(a: Optional[int], b: Optional[int]) -> Optional[int]: + if a is None: + return b + return a if b is None else max(a, b) + + +class PeakSampler: + """The largest values `read` returned since the last `reset`, sampled every `PEAK_SAMPLER_INTERVAL_S` on a daemon + thread between `start` and `stop`. + + For a device whose counters have no peaks: without it, a window would see only the values at its boundaries and + miss everything that came and went in between. `read` returns one value per counter, None where it has none; a + `read` that raises is a sample of Nones. + """ + + def __init__(self, read: Callable[[], tuple[Optional[int], ...]]) -> None: + self._read = read + self._lock = threading.Lock() + self._peaks: tuple[Optional[int], ...] = () + self._stop = threading.Event() + self._thread: Optional[threading.Thread] = None + + def _sample(self) -> tuple[Optional[int], ...]: + try: + return self._read() + except Exception: + return () + + def _fold(self, sample: tuple[Optional[int], ...]) -> tuple[Optional[int], ...]: + with self._lock: + width = max(len(self._peaks), len(sample)) + peaks = self._peaks + (None,) * (width - len(self._peaks)) + sample = sample + (None,) * (width - len(sample)) + self._peaks = tuple(_larger(peak, value) for peak, value in zip(peaks, sample, strict=True)) + return self._peaks + + def _run(self) -> None: + while not self._stop.wait(PEAK_SAMPLER_INTERVAL_S): + self._fold(self._sample()) + + def start(self) -> None: + self.reset() + self._stop.clear() + self._thread = threading.Thread(target=self._run, name="working-memory-peak-sampler", daemon=True) + self._thread.start() + + def stop(self) -> None: + self._stop.set() + if self._thread is not None: + self._thread.join() + self._thread = None + + @property + def running(self) -> bool: + return self._thread is not None and self._thread.is_alive() + + def reset(self) -> None: + """Start the peaks over at the values now.""" + sample = self._sample() + with self._lock: + self._peaks = sample + + def peaks(self) -> tuple[Optional[int], ...]: + """The peaks since the last reset, the values now included.""" + return self._fold(self._sample()) + + +def _mps_values() -> tuple[Optional[int], ...]: + return int(torch.mps.current_allocated_memory()), int(torch.mps.driver_allocated_memory()) + + +class DeviceCounters: + """One device's counters for the length of a window: readings, peak resets and synchronization. + + The peaks are torch's where it keeps them. On MPS, which keeps none, a `PeakSampler` keeps the allocated and the + driver peaks from construction until `close`. + """ + + def __init__(self, device: torch.device) -> None: + self.device = device + self._sampler = PeakSampler(_mps_values) if device.type == "mps" else None + if self._sampler is not None: + self._sampler.start() + + def read(self) -> CounterReading: + reading = read_counters(self.device) + if self._sampler is None: + return reading + peak_allocated, peak_driver = (self._sampler.peaks() + (None, None))[:2] + return dataclasses.replace( + reading, + peak_allocated=_larger(peak_allocated, reading.allocated), + peak_driver=_larger(peak_driver, reading.driver), + ) + + def reset_peaks(self) -> None: + reset_peaks(self.device) + if self._sampler is not None: + self._sampler.reset() + + def synchronize(self) -> None: + synchronize(self.device) + + def close(self) -> None: + if self._sampler is not None: + self._sampler.stop() diff --git a/invokeai/backend/working_memory/data/vae.json b/invokeai/backend/working_memory/data/vae.json index bae9081384f..5321cdbec12 100644 --- a/invokeai/backend/working_memory/data/vae.json +++ b/invokeai/backend/working_memory/data/vae.json @@ -3,131 +3,423 @@ "coefficients": [ { "estimator": "vae.sd", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { - "estimator": "vae.sd", - "version": 1, + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values": { + "k_decode_fp16": 4597, + "b_decode_fp16": 83886080, + "k_decode_bf16": 4500, + "b_decode_bf16": 68157440, + "k_decode_fp32": 7702, + "b_decode_fp32": 0, + "k_encode_fp16": 2226, + "b_encode_fp16": 22020096, + "k_encode_bf16": 2249, + "b_encode_bf16": 18874368, + "k_encode_fp32": 3236, + "b_encode_fp32": 41943040 + }, + "unmeasured_floor": { + "k_decode_fp16": 4400, + "k_decode_bf16": 4400, + "k_decode_fp32": 8800, + "k_encode_fp16": 2200, + "k_encode_bf16": 2200, + "k_encode_fp32": 4400 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on the RTX 4090 points (torch 2.13 and, for the older FLUX.2 rows, 2.7.1) of the SDXL VAE (fp16, fp32), the FLUX.1 VAE (bf16, fp32) and the FLUX.2 VAE (bf16, fp32). Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 2200 bytes per pixel per element byte for a decode, 1100 for an encode." + }, + { + "estimator": "vae.flux", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "cudnn", + "allocator": "native" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { - "estimator": "vae.sd3", - "version": 1, + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values_from": { + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.2 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "values": { + "k_decode_bf16": 8696, + "b_decode_bf16": 66060288, + "k_encode_bf16": 2420, + "b_encode_bf16": 22020096 + }, + "unmeasured_floor": { + "k_decode_bf16": 4400, + "k_encode_bf16": 2200 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on its RTX 4090 points (torch 2.13, native allocator). Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 2200 bytes per pixel per element byte for a decode, 1100 for an encode." + }, + { + "estimator": "vae.sd", + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD3 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { "estimator": "vae.sd3", - "version": 1, + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "values": { + "k_decode_fp16": 3797, + "b_decode_fp16": 0, + "k_decode_bf16": 3614, + "b_decode_bf16": 0, + "k_decode_fp32": 5914, + "b_decode_fp32": 13631488, + "k_encode_fp16": 1882, + "b_encode_fp16": 17825792, + "k_encode_bf16": 1882, + "b_encode_bf16": 17825792, + "k_encode_fp32": 2722, + "b_encode_fp32": 3145728 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on the RTX 4090 points (torch 2.13, backend:cudaMallocAsync) of the SDXL VAE (fp16, fp32), the FLUX.1 VAE (bf16, fp32) and the FLUX.2 VAE (bf16, fp32)." + }, + { + "estimator": "vae.flux", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the SD3 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { - "estimator": "vae.cogview4", - "version": 1, + "estimator": "vae.flux2", + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the CogView4 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.2 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { "estimator": "vae.cogview4", - "version": 1, + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "values": { + "k_decode_bf16": 6765, + "b_decode_bf16": 0, + "k_encode_bf16": 1882, + "b_encode_bf16": 9437184 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on its RTX 4090 points (torch 2.13, backend:cudaMallocAsync)." + }, + { + "estimator": "vae.sd", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the CogView4 VAE runs the same diffusers AutoencoderKL convolution stack. Not measured on this VAE." + "values": { + "k_decode_fp16": 7200, + "b_decode_fp16": 0, + "k_decode_bf16": 7143, + "b_decode_bf16": 68157440, + "k_decode_fp32": 14400, + "b_decode_fp32": 0, + "k_encode_fp16": 5500, + "b_encode_fp16": 0, + "k_encode_bf16": 5645, + "b_encode_bf16": 0, + "k_encode_fp32": 11000, + "b_encode_fp32": 0 + }, + "stand_ins": [ + "k_decode_fp16", + "b_decode_fp16", + "k_decode_fp32", + "b_decode_fp32", + "k_encode_fp16", + "b_encode_fp16", + "k_encode_fp32", + "b_encode_fp32" + ], + "unmeasured_floor": { + "k_decode_fp16": 7200, + "k_decode_bf16": 7200, + "k_decode_fp32": 14400, + "k_encode_fp16": 5500, + "k_encode_bf16": 5500, + "k_encode_fp32": 11000 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Every OS but Windows, which has its own entry. Fitted: bf16 on the FLUX.2 VAE points of Linux with the native allocator (RX 9070 XT gfx1201 and PRO W7900 gfx1100, torch 2.10). Stand-ins, no measurement here: fp16 and fp32 at 3600 decode / 2750 encode bytes per pixel per element byte, no base, the pre-dtype figure. Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." }, { "estimator": "vae.flux", - "version": 1, + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values_from": { + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values_from": { + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.2 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native" + }, + "values": { + "k_decode_bf16": 13720, + "b_decode_bf16": 14680064, + "k_encode_bf16": 5500, + "b_encode_bf16": 161480704 + }, + "stand_ins": [ + "k_decode_bf16", + "b_decode_bf16", + "k_encode_bf16", + "b_encode_bf16" + ], + "unmeasured_floor": { + "k_decode_bf16": 7200, + "k_encode_bf16": 5500 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Every OS but Windows, which has its own entry. Not measured here: both lines are stand-ins at its fitted MIOpen expandable-segments lines, as the family's bf16 lines measure the same under the native allocator on Linux and under expandable segments on Windows (7208 vs 7117 MiB decode at 1024px), with each slope raised to the pre-dtype figure where that is larger (the encode). Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." + }, + { + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "values": { + "k_decode_fp16": 3710, + "b_decode_fp16": 274726912, + "k_decode_bf16": 11009, + "b_decode_bf16": 0, + "k_decode_fp32": 10975, + "b_decode_fp32": 0, + "k_encode_fp16": 3703, + "b_encode_fp16": 77594624, + "k_encode_bf16": 4974, + "b_encode_bf16": 1104150528, + "k_encode_fp32": 4114, + "b_encode_fp32": 531628032 + }, + "unmeasured_floor": { + "k_decode_fp16": 7200, + "k_decode_bf16": 7200, + "k_decode_fp32": 14400, + "k_encode_fp16": 5500, + "k_encode_bf16": 5500, + "k_encode_fp32": 11000 + }, + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows: fitted on the RX 9060 XT (gfx1200, torch 2.13+rocm10) points of the SDXL VAE (fp16, fp32), the FLUX.1 VAE (bf16, fp32) and the FLUX.2 VAE (bf16, fp32) with expandable segments switched off. The demand is concave in the size, so the envelope over-reserves some points by more than 1.3x; the data test lists them. Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "values": { + "k_decode_bf16": 17640, + "b_decode_bf16": 0, + "k_encode_bf16": 5060, + "b_encode_bf16": 1059061760 + }, + "unmeasured_floor": { + "k_decode_bf16": 7200, + "k_encode_bf16": 5500 + }, + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows: fitted on its RX 9060 XT (gfx1200, torch 2.13+rocm10) points with expandable segments switched off. Where this entry prices a platform nothing is measured on (MPS, XPU, CPU, or another allocator on this backend), each slope is floored at the figure these estimators priced every dtype at before lines per dtype: 3600 bytes per pixel per element byte for a decode, 2750 for an encode." + }, + { + "estimator": "vae.sd", + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 + }, + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the SD1/SDXL VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." + }, + { + "estimator": "vae.sd3", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "values": { + "k_decode_fp16": 2016, + "b_decode_fp16": 506462208, + "k_decode_bf16": 6612, + "b_decode_bf16": 529530880, + "k_decode_fp32": 5988, + "b_decode_fp32": 80740352, + "k_encode_fp16": 1960, + "b_encode_fp16": 353370112, + "k_encode_bf16": 3255, + "b_encode_bf16": 483393536, + "k_encode_fp32": 3898, + "b_encode_fp32": 128974848 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. The FLUX.1 point below falls inside them." + "provenance": "The AutoencoderKL family's convolution lines (bytes per output pixel, and a base in bytes) per operation and dtype, carried by vae.sd3 as the one member that runs all three dtypes; vae.sd, vae.flux and vae.flux2 borrow the lines of theirs. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows only, where expandable segments are Invoke's ROCm default: measured on gfx1200 alone. Elsewhere they are a user override nothing is measured for, priced as MIOpen with the native allocator on that OS. Fitted on the RX 9060 XT (gfx1200, torch 2.13+rocm10) points of the SDXL VAE (fp16, fp32) and the FLUX.1 VAE (bf16, fp32). vae.flux2 carries its own lines here: its fp32 decode differs from the family's by more than the margin." }, { "estimator": "vae.flux", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values_from": { - "estimator": "vae.flux2", - "version": 1 + "estimator": "vae.sd3", + "version": 2 }, - "provenance": "Borrows the vae.flux2 values, so its points bound this estimator too: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. Confirmed on the FLUX.1 VAE: RX 9060 XT (gfx1200), torch 2.12, Windows, bf16, 768-1536px, Invoke's ROCm SDPA guard installed, MIOPEN_FIND_MODE=FAST: implied 3451 decode / 2688 encode." + "provenance": "Borrows the lines of its dtypes from vae.sd3, so its points bound them and join their fit: the FLUX.1 VAE runs the same diffusers AutoencoderKL convolution stack. On Windows with MIOpen's native allocator it borrows vae.sd3's Windows entry." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values": { - "k_decode": 2200, - "k_encode": 1100 + "k_decode_bf16": 6612, + "b_decode_bf16": 529530880, + "k_decode_fp32": 5842, + "b_decode_fp32": 272629760, + "k_encode_bf16": 3255, + "b_encode_bf16": 483393536, + "k_encode_fp32": 3898, + "b_encode_fp32": 128974848 }, - "provenance": "Peak reserved bytes per output pixel per element byte of the diffusers AutoencoderKL convolution stack. Fitted on the FLUX.2 VAE (the vae.flux2 cudnn points) with scripts/calibrate_flux2_working_memory.py --only vae: RTX 4090, torch 2.7.1, bf16, untiled, fused SDPA; implied 2180/2185/2165 decode and 1072/1063/1061 encode at 512/768/1024px, flat across resolution. Rounded up to 2200 / 1100, the pair the FLUX.1 estimator already shipped." + "provenance": "The FLUX.2 VAE's own convolution lines under MIOpen with expandable segments. Windows only, where expandable segments are Invoke's ROCm default: measured on gfx1200 alone. Elsewhere they are a user override nothing is measured for, priced as MIOpen with the native allocator on that OS. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Fitted on its RX 9060 XT (gfx1200, torch 2.13+rocm10) points. Not borrowed from vae.sd3: its fp32 decode needs ~140 MiB more at 512px than the SDXL and FLUX.1 VAEs, which would lift the family's fp32 decode line 9% above theirs, past the 5% margin. Its bf16 lines and fp32 encode equal the family's." }, { - "estimator": "vae.flux2", - "version": 1, + "estimator": "vae.cogview4", + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" }, "values": { - "k_decode": 3600, - "k_encode": 2750 + "k_decode_bf16": 13720, + "b_decode_bf16": 14680064, + "k_encode_bf16": 3764, + "b_encode_bf16": 161480704 }, - "provenance": "Peak reserved bytes per output pixel per element byte of the diffusers AutoencoderKL convolution stack on MIOpen. Fitted on the FLUX.2 VAE (the vae.flux2 miopen points) with scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled: RX 9070 XT (gfx1201) implied 3453/3369/3368 decode, PRO W7900 (gfx1100) 3525 at 512px; encode 2687/2688/2688 on both cards, to the byte. Two AMD generations agreeing that closely make this MIOpen rather than a per-card quirk, so the values are the larger card's plus ~2% headroom: 3600 / 2750. MIOpen's convolution workspaces are larger than cuDNN's (~1.6x decode, ~2.4x encode), identically with fused and math attention, and its encode/decode ratio is 0.78 against cuDNN's 0.49." + "provenance": "The CogView4 VAE's own lines: an AutoencoderKL, but not the family's stack (block widths 128/512/1024/1024, three layers per block, no mid-block attention), and a decode costs ~1.9x the family's. bf16 only: its loader forces it. Fitted lines are working_memory/fit.py's: the upper envelope of the untiled single-sample in-fit points of every estimator resolving to this entry, plus 5%; tiled points only bound their estimates. Windows only, where expandable segments are Invoke's ROCm default: measured on gfx1200 alone. Elsewhere they are a user override nothing is measured for, priced as MIOpen with the native allocator on that OS. Fitted on its RX 9060 XT (gfx1200, torch 2.13+rocm10) points." } ], "points": [ { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -135,19 +427,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 1142461301, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -155,19 +448,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 2576980378, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -175,19 +469,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 4540854174, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -195,19 +490,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 561566974, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -215,19 +511,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 1254130450, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -235,19 +532,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 2224793059, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.7.1, bf16, untiled, fused SDPA (score matrix not materialized). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -255,19 +553,20 @@ "width": 1280, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 7075958620, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, bf16, untiled, fused SDPA: the fused column of the max-versus-sum measurement (#9519). Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, bf16, untiled, fused SDPA: the fused column of the max-versus-sum measurement (#9519). Recorded in GiB to three decimals. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -275,19 +574,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 1138753536, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -295,19 +595,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 2531262464, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -315,19 +616,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 4466933760, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -335,19 +637,20 @@ "width": 1328, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 7493124096, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -355,19 +658,20 @@ "width": 1536, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 10043260928, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -375,19 +679,20 @@ "width": 1328, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 3168796672, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement." + "provenance": "First FLUX.2 VAE calibration (#9519): CUDA, bf16, untiled, fused SDPA. Recorded in MiB. The decode rows are a separate run from the 512-1024px GiB rows; the 1536px decode is also the fused 9.353 GiB of the max-versus-sum measurement. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -395,19 +700,21 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 561566974, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -415,19 +722,21 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 1933809025, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -435,19 +744,21 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 4372276707, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -455,19 +766,21 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 3787087413, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -475,19 +788,21 @@ "width": 1280, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 11141120000 + "score_matrix_bytes": 11141120000, + "dtype": "bf16" }, "measured_bytes": 6360846565, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -495,19 +810,21 @@ "width": 1536, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 23102226432 + "score_matrix_bytes": 23102226432, + "dtype": "bf16" }, "measured_bytes": 12847320924, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae with SDPBackend.MATH forced on CUDA -- the regime MPS lands in; bf16, untiled; score_matrix_bytes is the full math-kernel score matrix. Forcing math measures below the fused path until 1536px. The 768px encode sits at the crossover, where the larger term alone is short and the device_working_mem_gb floor covers it; reproducible to three decimals across runs. Recorded in GiB. Standalone script with no allocator configuration: the native allocator.", + "in_fit": false }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -515,19 +832,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 1810328715, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -535,19 +853,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 3973918491, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -555,19 +874,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 7063073718, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -575,20 +895,21 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 1847909679, "method": "reserved_peak", "device": "PRO W7900 (gfx1100)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals.", + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it.", "max_ratio": 1.15 }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -596,20 +917,21 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 4059817837, "method": "reserved_peak", "device": "PRO W7900 (gfx1100)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals.", + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it.", "max_ratio": 1.15 }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "decode", @@ -617,20 +939,21 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 7197291446, "method": "reserved_peak", "device": "PRO W7900 (gfx1100)", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals.", + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it.", "max_ratio": 1.15 }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "encode", @@ -638,19 +961,20 @@ "width": 512, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 285212672 + "score_matrix_bytes": 285212672, + "dtype": "bf16" }, "measured_bytes": 1408749273, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201) and PRO W7900 (gfx1100), identical to the byte", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "encode", @@ -658,19 +982,20 @@ "width": 768, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 1443889152 + "score_matrix_bytes": 1443889152, + "dtype": "bf16" }, "measured_bytes": 3170759606, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201) and PRO W7900 (gfx1100), identical to the byte", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "miopen" + "conv_backend": "miopen", + "allocator": "native" }, "features": { "operation": "encode", @@ -678,19 +1003,20 @@ "width": 1024, "tile": null, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 4563402752 + "score_matrix_bytes": 4563402752, + "dtype": "bf16" }, "measured_bytes": 5637144576, "method": "reserved_peak", "device": "RX 9070 XT (gfx1201) and PRO W7900 (gfx1100), identical to the byte", - "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals." + "provenance": "scripts/calibrate_flux2_working_memory.py --only vae, torch 2.10, bf16, untiled; a clean run (no MIOPEN_FIND_MODE override, no allocator GC threshold). score_matrix_bytes is the math-kernel score matrix the ROCm head-dim guard prices for the 512-wide head, unchunked; on these cards forcing math moved the peak by nothing up to 1024px. Recorded in GiB to three decimals. Linux, no allocator configuration: the native allocator, as the script ran it." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -698,19 +1024,20 @@ "width": 1024, "tile": null, "batch_size": 2, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 8541616210, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above." + "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "decode", @@ -718,19 +1045,20 @@ "width": 1024, "tile": null, "batch_size": 3, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 12765716546, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above." + "provenance": "#9519: 1024px decode by latent batch, CUDA, bf16, fused SDPA, one fresh process per point: 4.23 / 7.96 / 11.89 GiB at batch 1 / 2 / 3. The test table wrote them as GiB x 1000 (4229, 7955, 11889); batch 1 is the 4.229 GiB 1024px decode above. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux2", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", @@ -738,32 +1066,5247 @@ "width": 2024, "tile": 512, "batch_size": 1, - "element_size": 2, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 585105408, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "#9519: reference-image encode at the 2024px reference cap with the forced 512px tile, CUDA, bf16, fused SDPA; flat at ~0.55 GB from 1024px up. Recorded in MiB." + "provenance": "#9519: reference-image encode at the 2024px reference cap with the forced 512px tile, CUDA, bf16, fused SDPA; flat at ~0.55 GB from 1024px up. Recorded in MiB. Standalone script with no allocator configuration: the native allocator." }, { "estimator": "vae.flux", - "version": 1, + "version": 2, "platform": { - "conv_backend": "cudnn" + "conv_backend": "cudnn", + "allocator": "native" }, "features": { "operation": "encode", "height": 3072, "width": 3072, - "element_size": 2, "tile": null, - "score_matrix_bytes": 0 + "score_matrix_bytes": 0, + "dtype": "bf16" }, "measured_bytes": 20229295964, "method": "reserved_peak", "device": "RTX 4090", - "provenance": "Commit 326bdf1bd0: the FLUX.1 VAE as a diffusers AutoencoderKL, bf16, fused SDPA, untiled 3072px encode: 18.84 GiB. Re-measured 18.57 GiB on torch 2.13+cu130, 2026-10-17, with scripts/calibrate_flux_vae_working_memory.py." + "provenance": "Commit 326bdf1bd0: the FLUX.1 VAE as a diffusers AutoencoderKL, bf16, fused SDPA, untiled 3072px encode: 18.84 GiB. Re-measured 18.57 GiB on torch 2.13+cu130, 2026-10-17, with scripts/calibrate_flux_vae_working_memory.py. Standalone script with no allocator configuration: the native allocator." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1488977920, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3334471680, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5817499648, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 9105833984, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1744830464, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4326424576, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7572815872, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 11741954048, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 891289600, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2132803584, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3709861888, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5689573376, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1226833920, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2661285888, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4622123008, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7157579776, + "method": "driver_peak", + "device": "RTX 4090", + "provenance": "Isolated SDXL VAE decode (diffusers AutoencoderKL, force_upcast; random latents as l2i), one fresh process per size, the larger of two decodes: RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, fused SDPA, native allocator. Driver-level usage agreed with max_memory_reserved within 100 MiB. A game held ~3 GB on the card meanwhile; no paging measured, allocated identical. bf16 measured identical to fp16 (not shipped: this node never runs bf16). Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1140850688, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4538236928, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10158604288, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 2048, + "width": 2048, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 18029215744, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 562036736, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2225078272, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4989124608, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 2048, + "width": 2048, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8864661504, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 3072, + "width": 3072, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 19937624064, + "method": "reserved_peak", + "device": "RTX 4090", + "provenance": "scripts/calibrate_flux_vae_working_memory.py (2026-10-17): the FLUX.1 ae.safetensors as a diffusers AutoencoderKL, RTX 4090, torch 2.13.0+cu130, Windows, bf16, fused SDPA, untiled, native allocator (no allocator configuration). The untiled 3072px decode ran into the card's limit and is not shipped." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 985661440, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 1614807040, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2495610880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3271557120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 825229312, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 1433403392, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2293235712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3040870400, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1570766848, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3355443200, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5997854720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 9418309632, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1095761920, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2312110080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3900702720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2154823680, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3984588800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6983516160, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10821304320, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1197473792, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2287992832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3546284032, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5538578432, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1570766848, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3355443200, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5997854720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 9418309632, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1095761920, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2312110080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3900702720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2154823680, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3984588800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6983516160, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10821304320, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1197473792, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2287992832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3548381184, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5538578432, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1717567488, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3376414720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6018826240, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 9374269440, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1095761920, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2312110080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3900702720, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6081740800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1186988032, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2344615936, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3212836864, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4796186624, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 998244352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2153775104, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2526019584, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3462397952, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2686451712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6155141120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10959716352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16733175808, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1533018112, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2495610880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4433379328, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6924795904, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2176843776, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5135925248, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10993270784, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16787701760, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2293235712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3206545408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5639241728, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 8812232704, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2684354560, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6155141120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10959716352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16733175808, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1533018112, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2495610880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4433379328, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6922698752, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2178940928, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5135925248, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10993270784, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16789798912, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2293235712, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 3206545408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 5641338880, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 8810135552, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 2684354560, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6155141120, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 10959716352, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 16689135616, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 285212672 + }, + "measured_bytes": 1533018112, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 2493513728, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 4431282176, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 1073741824 + }, + "measured_bytes": 6922698752, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1157627904, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2592079872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4552916992, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10173284352, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 576716800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1268776960, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2241855488, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5022679040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1675624448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4257218560, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7503609856, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 16770924544, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 847249408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1843396608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3261071360, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7308574720, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1157627904, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2592079872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4555014144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10175381504, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 576716800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1270874112, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2239758336, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5005901824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1675624448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4259315712, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7501512704, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 16770924544, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 845152256, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1843396608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3261071360, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7306477568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1159725056, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2592079872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4555014144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 10057940992, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 578813952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1270874112, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2241855488, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5005901824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1675624448, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4257218560, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7501512704, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 16791896064, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 847249408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1841299456, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3258974208, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 7306477568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 822083584, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2063597568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3640655872, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8069840896, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 486539264, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1056964608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1895825408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp16", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4244635648, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1442840576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3265265664, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5748293632, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 12996050944, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 681574400, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1522532352, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2694840320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.sd", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "vae_class": "AutoencoderKL", + "upcast_weight_bytes": 0, + "score_matrix_bytes": 0 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init SDXL VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 788529152, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2030043136, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3607101440, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8036286464, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 486539264, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1056964608, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1895825408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4244635648, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1442840576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3265265664, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5714739200, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 12962496512, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 681574400, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1520435200, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2694840320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "score_matrix_bytes": 0 + }, + "measured_bytes": 6083837952, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.1 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 788529152, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2030043136, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3607101440, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 8069840896, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 486539264, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1073741824, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1895825408, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 4244635648, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1409286144, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 3265265664, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 5781848064, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 12962496512, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 681574400, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 1520435200, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 2694840320, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.flux2", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "fp32", + "tile": null, + "batch_size": 1, + "score_matrix_bytes": 0 + }, + "measured_bytes": 6117392384, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init FLUX.2 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 2233466880, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 4947181568, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 8690597888, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 19386073088, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 624951296, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 1379926016, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 2436890624, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "native" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 5454692352, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, native allocator, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 1644167168, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 3800039424, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 6685720576, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "decode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 14738784256, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 478150656, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 1048576000, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 1887436800, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "cudnn", + "allocator": "cudaMallocAsync" + }, + "features": { + "operation": "encode", + "height": 1536, + "width": 1536, + "dtype": "bf16" + }, + "measured_bytes": 4236247040, + "method": "demand_peak", + "device": "RTX 4090", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RTX 4090, torch 2.13.0+cu130, Windows, driver 32.0.16.1047, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync, fused SDPA. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 3439329280, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 7717519360, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 13715374080, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 1092616192, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 2267021312, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 3755999232, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "expandable", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16" + }, + "measured_bytes": 5874122752, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch reports the native backend), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (10040 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 4399824896, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 9909043200, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "decode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 17616076800, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 512, + "width": 512, + "dtype": "bf16" + }, + "measured_bytes": 2271215616, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 768, + "width": 768, + "dtype": "bf16" + }, + "measured_bytes": 3206545408, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1024, + "width": 1024, + "dtype": "bf16" + }, + "measured_bytes": 5695864832, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." + }, + { + "estimator": "vae.cogview4", + "version": 2, + "platform": { + "conv_backend": "miopen", + "allocator": "native", + "os": "windows" + }, + "features": { + "operation": "encode", + "height": 1280, + "width": 1280, + "dtype": "bf16" + }, + "measured_bytes": 8902410240, + "method": "demand_peak", + "device": "RX 9060 XT (gfx1200)", + "provenance": "Scratch probe vae_dtype_probe.py (2026-10-17): RX 9060 XT (gfx1200), torch 2.13.0+rocm10.0.0, Windows, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False (the native allocator), MIOPEN_FIND_MODE=FAST, Invoke's ROCm SDPA guard installed, so score_matrix_bytes is its chunked math term. Windows-only: the native allocator's demand there is concave (first-run driver spikes at small sizes, a second run reserving up to 1.6x the first at large ones) and up to 1.6x Linux's. Random-init CogView4 VAE from the released config (memory does not depend on weight values), untiled, one fresh process per point, two runs, no allocator retries. Recorded in MiB. The 1280px decode is not shipped: it needs ~20 GB on this 16 GB card, which Windows pages into system memory instead of failing, so it read low (16650 MiB)." } ] } diff --git a/invokeai/backend/working_memory/fingerprint.py b/invokeai/backend/working_memory/fingerprint.py new file mode 100644 index 00000000000..ccdde0a04c1 --- /dev/null +++ b/invokeai/backend/working_memory/fingerprint.py @@ -0,0 +1,173 @@ +"""What a measurement is keyed by: the class of device it ran on, and the software environment that decides how much +memory an operation takes there. + +A device class names a kind of card, not one card: identical cards share it, whatever their index or PCI slot, so +measurements taken on one apply to the other. An environment splits into hard facts, any change of which can change an +operation's memory (the operating system, torch and runtime versions, the convolution library, the allocator and its +configuration, the attention path, InvokeAI's memory settings), and soft facts recorded for diagnosis only (full +versions, the driver, the OS build). Both are produced deterministically here; which change makes a measurement stale +is the caller's decision. +""" + +import ctypes +import ctypes.util +import hashlib +import json +import os +import platform +import sys +from collections.abc import Mapping +from typing import Final, Literal, Optional + +import psutil +import torch +from pydantic import BaseModel, ConfigDict, computed_field + +from invokeai.backend.util import nvml +from invokeai.backend.util.attention import rocm_sdpa_chunks_math +from invokeai.backend.working_memory.counters import Backend, backend_for +from invokeai.version import __version__ + +_VENDORS: Final[Mapping[Backend, str]] = {"cuda": "nvidia", "hip": "amd", "mps": "apple", "xpu": "intel"} + +HARD_ENVIRONMENT_VARIABLES: Final = ( + "PYTORCH_ALLOC_CONF", + "PYTORCH_CUDA_ALLOC_CONF", + "PYTORCH_HIP_ALLOC_CONF", + "MIOPEN_FIND_MODE", + "PYTORCH_TUNABLEOP_ENABLED", + # The head width above which the ROCm SDPA guard sends attention to the chunked math kernel (`util/attention.py`). + "INVOKE_ROCM_FUSED_SDPA_MAX_HEAD_DIM", + # Whether AOTriton's fused attention runs on GPUs PyTorch marks experimental; without it attention takes the math + # kernel. What `rocm_aotriton_experimental` resolved to (`app/util/rocm_aotriton.py`), or a value set beforehand. + "TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL", + # Native or decomposed causal 3D convolutions on ROCm (`backend/wan/rocm_causal_conv3d.py`). + "INVOKEAI_ROCM_CONV3D", +) +"""Environment variables that change how much memory an operation takes: the allocator's configuration, MIOpen's +algorithm search, TunableOp's GEMM selection, and the ROCm attention and convolution kernel choices. Recorded as +``env.``, empty when unset.""" + + +def _short_hash(fields: Mapping[str, object]) -> str: + canonical = json.dumps(fields, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16] + + +class DeviceClass(BaseModel): + """A kind of compute device. Identical cards share one, whatever their index or PCI location.""" + + model_config = ConfigDict(frozen=True) + + vendor: str + name: str + total_mib: int + backend: Backend + arch: Optional[str] + """``gcnArchName`` on HIP (e.g. ``gfx1200``), ``sm_`` on CUDA, the reported architecture on XPU.""" + + @computed_field # type: ignore[prop-decorator] + @property + def id(self) -> str: + """A short stable hash of the other fields. Derived, never given.""" + return _short_hash(self.model_dump(exclude={"id"})) + + +def _mps_name() -> str: + """The Mac's chip, e.g. ``Apple M2 Max``; MPS names no device.""" + try: + libc = ctypes.CDLL(ctypes.util.find_library("c")) + size = ctypes.c_size_t(0) + if libc.sysctlbyname(b"machdep.cpu.brand_string", None, ctypes.byref(size), None, 0) == 0 and size.value: + buffer = ctypes.create_string_buffer(size.value) + if libc.sysctlbyname(b"machdep.cpu.brand_string", buffer, ctypes.byref(size), None, 0) == 0: + return buffer.value.decode("utf-8", errors="replace") + except Exception: + pass + return "Apple GPU" + + +def device_class_for(device: torch.device) -> DeviceClass: + """The class of `device`. Raises ValueError for a device that is not a GPU.""" + backend = backend_for(device) + if backend is None: + raise ValueError(f"{device} is not a GPU; only GPU devices have a device class.") + if backend in ("cuda", "hip"): + props = torch.cuda.get_device_properties(device) + name, total = torch.cuda.get_device_name(device), int(props.total_memory) + arch = str(props.gcnArchName) if backend == "hip" else f"sm_{props.major}{props.minor}" + elif backend == "xpu": + props = torch.xpu.get_device_properties(device) + name, total = str(props.name), int(props.total_memory) + architecture = getattr(props, "architecture", None) + arch = None if architecture is None else str(architecture) + else: + # Unified memory: the device's share is the machine's RAM. + name, total, arch = _mps_name(), int(psutil.virtual_memory().total), None + return DeviceClass(vendor=_VENDORS[backend], name=name, total_mib=total // 2**20, backend=backend, arch=arch) + + +class Environment(BaseModel): + """The software a measurement ran under: hard facts that change an operation's memory, soft facts that do not.""" + + model_config = ConfigDict(frozen=True) + + hard: dict[str, str] + soft: dict[str, Optional[str]] + + @computed_field # type: ignore[prop-decorator] + @property + def id(self) -> str: + """A short stable hash of the hard facts. Soft facts do not enter it.""" + return _short_hash(self.hard) + + +def _major_minor(version: str) -> str: + return ".".join(version.split("+", 1)[0].split(".")[:2]) + + +def _conv_library_major() -> Optional[str]: + """The major version of cuDNN, or of MIOpen on a HIP build, as loaded at runtime.""" + try: + if torch.backends.cudnn.version() is None: + return None + # (major, minor, patch) on either build; the packed integer `version()` returns is encoded per library. + return str(torch._C._cudnn.getRuntimeVersion()[0]) + except Exception: + return None + + +def os_family(platform_name: str) -> Literal["windows", "linux", "other"]: + """The operating system family of a ``sys.platform`` value, as a hard fact: the driver model differs (WDDM pages + an allocation that does not fit into shared memory instead of failing it), so the same card and torch build can + take different memory on Windows and on Linux.""" + if platform_name == "win32": + return "windows" + return "linux" if platform_name.startswith("linux") else "other" + + +def environment_for(device: torch.device, settings: Mapping[str, str | int | float | bool | None]) -> Environment: + """The environment an operation on `device` runs under. Every key of `settings` is a hard fact, as + ``setting.``: the caller passes the InvokeAI settings that change memory.""" + backend = backend_for(device) + hard: dict[str, str] = {"os": os_family(sys.platform), "torch": _major_minor(torch.__version__)} + if backend in ("cuda", "hip"): + runtime = torch.version.hip if backend == "hip" else torch.version.cuda + hard[backend] = _major_minor(runtime or "") + hard["miopen" if backend == "hip" else "cudnn"] = _conv_library_major() or "" + hard["allocator_backend"] = torch.cuda.get_allocator_backend() + hard["rocm_sdpa_guard"] = str(rocm_sdpa_chunks_math(device.type)).lower() + elif backend == "xpu": + hard["xpu"] = str(getattr(torch.version, "xpu", None) or "") + for name in HARD_ENVIRONMENT_VARIABLES: + hard[f"env.{name}"] = os.environ.get(name, "") + for key, value in settings.items(): + hard[f"setting.{key}"] = str(value) + + soft: dict[str, Optional[str]] = { + "torch_build": torch.__version__, + "driver": nvml.driver_version() if backend == "cuda" else None, + "os_build": platform.platform(), + "invokeai": __version__, + } + return Environment(hard=hard, soft=soft) diff --git a/invokeai/backend/working_memory/fit.py b/invokeai/backend/working_memory/fit.py new file mode 100644 index 00000000000..d6b2758774f --- /dev/null +++ b/invokeai/backend/working_memory/fit.py @@ -0,0 +1,208 @@ +"""The shipped fit: the values under `data/` reproduced from their measured points. + +The AutoencoderKL estimators price their convolutions as a line in the output's pixel count, one per operation and +dtype: `k__` bytes per pixel plus `b__` bytes, the base a process pays whatever the +size. Each carried pair is the upper envelope of the points it is resolved for (see `upper_envelope`), plus +`SHIPPED_MARGIN`. A point joins the fit of the entry its estimator resolves to on its platform -- the lender's, where +the estimator borrows -- when it is a single-sample, untiled operation of that pair (`in_fit` unless measured out of +regime) with no weight growth. Tiled points bound their estimates but fit nothing: the tile term reuses the untiled +pair. Nor does a point whose VAE an in-place cast grew (`upcast_weight_bytes`): its demand includes the growth, which +the estimate prices as a term of its own. A point that bounds its estimate from below only +(`MeasuredPoint.holds_upper_bound`, a `probe_demand` window) is covered by the line and can raise it, but takes no part +in choosing it: the fresh-process points cap a line. + +`tests/backend/working_memory/test_shipped_coefficients.py` holds every carried value that is not a declared stand-in +to `fitted_values` of the shipped points, so a new point that moves a fit fails there with the value to ship. + +A user's own calibration fits the same lines from its windows (`user_pair`, read by `user_fit.py`), with +`USER_MARGIN`, the same margin. +""" + +import math +from collections.abc import Iterable, Mapping, Sequence +from fractions import Fraction +from typing import Final, NamedTuple, Optional + +from invokeai.backend.working_memory.coefficients import CoefficientEntry, MeasuredPoint, WorkingMemoryData, find_entry + +SHIPPED_MARGIN: Final = Fraction(1, 20) +"""What the shipped slope and intercept add over the envelope of their points: 5%, for the runs and cards the points +do not cover.""" + +USER_MARGIN: Final = SHIPPED_MARGIN +"""What a user calibration's slope and intercept add over the envelope of its points: the shipped margin. A user line +measures the card it prices, and the windows that would inflate it (fragmented, partly resident) never join it.""" + +MIN_AREA_RATIO: Final = 2 +"""How far apart the sizes a line is fitted on must lie, as a ratio of pixel counts: closer points pin the intercept +no better than their noise.""" + +_MIB: Final = 2**20 + +EntryKey = tuple[str, int, tuple[tuple[str, str], ...]] +"""An entry's estimator id, version and platform keys: what a point is resolved by.""" + + +def entry_key(entry: CoefficientEntry | MeasuredPoint) -> EntryKey: + return (entry.estimator, entry.version, tuple(sorted(entry.platform.items()))) + + +class FitPoint(NamedTuple): + pixels: int + bytes: int + caps: bool + """Whether the point takes part in choosing the line (`MeasuredPoint.holds_upper_bound`); every point is covered.""" + + +def upper_envelope( + points: Sequence[tuple[int, int]], capping: Optional[Sequence[tuple[int, int]]] = None +) -> tuple[Fraction, Fraction]: + """The line ``y = k * x + b`` on or above every ``(x, y)`` point, with ``b >= 0``, that over-reserves least. + + The candidates are the edges of the upper convex hull of the points and the origin: each such line lies on or + above every point, and the origin keeps its intercept non-negative. Of those, the one whose largest ratio to the + highest reading of a size is smallest wins, the steeper one on a tie, since it is the more conservative past the + largest point. Only the highest reading counts: a lower one at a measured size is covered whatever the line, and + rating against it would tilt the line though no point lies above it. With `capping`, a subset of the points, only + its readings rate the edges: the others are covered but cap nothing. Without any, all points rate them. Raises + unless the points span two sizes at least `MIN_AREA_RATIO` apart. + """ + if not points or any(x <= 0 or y <= 0 for x, y in points): + raise ValueError(f"an envelope needs points with positive sizes and bytes, got {list(points)}") + sizes = sorted({x for x, _ in points}) + if len(sizes) < 2 or sizes[-1] < MIN_AREA_RATIO * sizes[0]: + raise ValueError( + f"an envelope needs two sizes at least {MIN_AREA_RATIO}x apart in pixels, got {sizes}; measure more sizes" + ) + + highest = {x: max(y for px, y in points if px == x) for x in sizes} + hull: list[tuple[int, int]] = [(0, 0)] + for point in highest.items(): + # Drop the last vertex while it is not above the chord from the one before it to this point. + while len(hull) >= 2: + (x0, y0), (x1, y1) = hull[-2], hull[-1] + if (x1 - x0) * (point[1] - y0) - (y1 - y0) * (point[0] - x0) < 0: + break + hull.pop() + hull.append(point) + + rated = {x: max(y for px, y in capping if px == x) for x in sorted({x for x, _ in capping})} if capping else highest + best: tuple[Fraction, Fraction, Fraction] | None = None + for (x0, y0), (x1, y1) in zip(hull, hull[1:], strict=False): + k = Fraction(y1 - y0, x1 - x0) + b = y0 - k * x0 + if k < 0: + continue + worst = max((k * x + b) / y for x, y in rated.items()) + if best is None or (worst, -k) < (best[0], -best[1]): + best = (worst, k, b) + assert best is not None, "the edge from the origin always rises" + return best[1], best[2] + + +def shipped_pair( + points: Sequence[tuple[int, int]], capping: Optional[Sequence[tuple[int, int]]] = None +) -> tuple[int, int]: + """The slope and intercept to ship for these points: the envelope plus `SHIPPED_MARGIN`, the slope rounded up to + a whole byte per pixel and the intercept to a whole MiB.""" + k, b = upper_envelope(points, capping) + margin = 1 + SHIPPED_MARGIN + return math.ceil(k * margin), math.ceil(b * margin / _MIB) * _MIB + + +def user_pair(points: Sequence[tuple[int, int]]) -> tuple[int, int]: + """The slope and intercept a user calibration's points give a line: the envelope plus `USER_MARGIN`, the slope + rounded up to a whole byte per pixel and the intercept to a whole MiB. + + Every user point is a `probe_demand` window, so no point caps the line and all of them rate its edges (the + fallback of `upper_envelope`). Raises like `upper_envelope` unless the points span two sizes `MIN_AREA_RATIO` + apart.""" + k, b = upper_envelope(points) + margin = 1 + USER_MARGIN + return math.ceil(k * margin), math.ceil(b * margin / _MIB) * _MIB + + +def line_of(features: Mapping[str, object]) -> Optional[tuple[str, int]]: + """The convolution line an operation of these features is priced on, ``_``, and its size in + output pixels; None for features that name no such line.""" + operation, dtype = features.get("operation"), features.get("dtype") + height, width = features.get("height"), features.get("width") + if not isinstance(operation, str) or not isinstance(dtype, str): + return None + if not isinstance(height, int) or not isinstance(width, int): + return None + return f"{operation}_{dtype}", height * width + + +def fits_a_line(features: Mapping[str, object]) -> bool: + """Whether a point of these features can join its line's fit: an untiled single-sample operation of an + AutoencoderKL whose VAE no in-place cast grew. Tiled points reuse the untiled line, the cast's growth is a term of + its own, and the lines are the AutoencoderKL convolutions': the SD estimator also prices the tiny autoencoder + (`vae_class` AutoencoderTiny) on them, a different network whose windows say nothing about those lines. + Estimators that name no `vae_class` price an AutoencoderKL only.""" + return ( + features.get("tile") is None + and features.get("batch_size", 1) == 1 + and not features.get("upcast_weight_bytes", 0) + and features.get("vae_class", "AutoencoderKL") == "AutoencoderKL" + and line_of(features) is not None + ) + + +def _pairs(entry: CoefficientEntry) -> set[str]: + """The `_` suffixes an entry carries both halves of a convolution line for.""" + names = set(entry.values or {}) + return {name[2:] for name in names if name.startswith("k_") and f"b_{name[2:]}" in names} + + +def fit_points(data: Iterable[WorkingMemoryData]) -> Mapping[tuple[EntryKey, str], list[FitPoint]]: + """Per carried entry and line, the points that line is fitted on. + + A point belongs to the entry its own estimator's entry resolves to on its platform, one `values_from` hop, and to + the line its operation and dtype name. Lines no point measures are absent. + """ + files = list(data) + entries = [entry for file in files for entry in file.coefficients] + groups: dict[tuple[EntryKey, str], list[FitPoint]] = {} + for point in (point for file in files for point in file.points): + found = line_of(point.features) + if not point.in_fit or found is None or not fits_a_line(point.features): + continue + owner = carrier_of(entries, point) + suffix, pixels = found + if suffix in _pairs(owner): + line = groups.setdefault((entry_key(owner), suffix), []) + line.append(FitPoint(pixels, point.measured_bytes, point.holds_upper_bound)) + return groups + + +def carrier_of(entries: Sequence[CoefficientEntry], point: MeasuredPoint) -> CoefficientEntry: + """The entry that carries the values `point` is evaluated with: its estimator's entry on its platform, or the one + that entry borrows from. A point names the OS only where it measured an OS-specific entry's platform.""" + where = f"{point.estimator} point on {point.platform}" + entry = find_entry(entries, point.estimator, point.version, point.platform, where) + if entry.values_from is None: + return entry + return find_entry(entries, entry.values_from.estimator, entry.values_from.version, point.platform, where) + + +def fitted_values(data: Iterable[WorkingMemoryData]) -> Mapping[EntryKey, Mapping[str, int]]: + """Per entry that carries values, what `shipped_pair` gives each line it does not declare a stand-in. + + Raises, naming the entry and line, where the points cannot fit that line. + """ + files = list(data) + groups = fit_points(files) + fitted: dict[EntryKey, dict[str, int]] = {} + for entry in (entry for file in files for entry in file.coefficients if entry.values is not None): + key = entry_key(entry) + for suffix in sorted(_pairs(entry)): + if {f"k_{suffix}", f"b_{suffix}"} <= set(entry.stand_ins): + continue + line = groups.get((key, suffix), []) + try: + k, b = shipped_pair([(p.pixels, p.bytes) for p in line], [(p.pixels, p.bytes) for p in line if p.caps]) + except ValueError as e: + raise ValueError(f"{entry.estimator} on {entry.platform}, {suffix}: {e}") from e + fitted.setdefault(key, {}).update({f"k_{suffix}": k, f"b_{suffix}": b}) + return fitted diff --git a/invokeai/backend/working_memory/fragmentation.py b/invokeai/backend/working_memory/fragmentation.py new file mode 100644 index 00000000000..c9a4b20dc0e --- /dev/null +++ b/invokeai/backend/working_memory/fragmentation.py @@ -0,0 +1,106 @@ +"""When a measured window's demand is the allocator's fragmentation rather than the operation's. + +A long-running server's caching allocator grows its reserve around the same tensors: reserved bytes climb while the +allocated bytes, what the operation really holds, stay put. A window taken then measures the allocator's history, and a +line fitted on it over-reserves every later operation. One rule, `fragmented`, read by +`scripts/working_memory/build_shipped.py`, which flags such points for review, and by the user fit +(`app/services/working_memory_calibration/`), which leaves them out. It judges measurements: the fit-eligible windows +of one operation at one size from one attempt (a calibration attempt, or one run without one), in the order they ran. +Windows of one attempt at one size ran back to back on one allocator state; another attempt's ran on another, before +or after offloads and other work, so each attempt is judged on its own and a recalibration can bring a size back. + +- `grew_around_flat_allocation`: across a measurement's windows, reserved grew by `FRAGMENTATION_GROWTH` from the + first window to the largest while allocated stayed within `ALLOCATED_FLAT`. +- `ratio_outlier`: a measurement's largest window has a reserved-to-allocated ratio `RATIO_OUTLIER` above the median of + the same line's smaller measurements that are not flagged, from any attempt: the ratio is the operation's on the + card, which no attempt owns. This catches a size whose every window ran fragmented, so that it shows no growth of its + own; a single window, too. Measured on an RX 9060 XT (fp32 SDXL decode): 1216px measured 1.47 against 1.10 at 576px + and 832px, its one fit-eligible window 1.33 times their median, while the fp16 sizes of the same run stayed within + 1.06 of theirs. +""" + +from collections.abc import Hashable, Sequence +from dataclasses import dataclass +from statistics import median +from typing import Final, Optional + +FRAGMENTATION_GROWTH: Final = 1.10 +"""Reserved growth from a measurement's first window to its largest that, with flat allocated bytes, flags +fragmentation.""" + +ALLOCATED_FLAT: Final = 0.02 +"""How far allocated bytes may spread across a measurement's windows and still count as flat.""" + +RATIO_OUTLIER: Final = 1.2 +"""How far a size's reserved-to-allocated ratio may exceed the median of the same line's smaller sizes before it +counts as fragmentation.""" + + +@dataclass(frozen=True) +class Measurement: + """The fit-eligible windows of one operation at one size from one attempt, in the order they ran.""" + + line: Hashable + """What its sizes are compared within: one line (operation and dtype) of one estimator version on one card and + platform or environment.""" + pixels: int + demand: Sequence[int] + reserved: Sequence[Optional[int]] + """Each window's reserved delta.""" + allocated: Sequence[Optional[int]] + reference: bool = True + """Whether larger sizes may be compared with it; a measurement still being taken is judged, but sets no standard.""" + + +def fragmented(measurements: Sequence[Measurement]) -> list[bool]: + """Per measurement, whether its windows measured fragmentation: `grew_around_flat_allocation`, or a + `ratio_outlier` of its largest window against the smaller measurements of its line that are not flagged and may + be a `reference`.""" + flags = [ + grew_around_flat_allocation(measurement.demand, measurement.reserved, measurement.allocated) + for measurement in measurements + ] + lines: dict[Hashable, list[int]] = {} + for index, measurement in enumerate(measurements): + if not flags[index]: + lines.setdefault(measurement.line, []).append(index) + for members in lines.values(): + kept: list[tuple[int, float]] = [] + for index in sorted(members, key=lambda index: measurements[index].pixels): + measurement = measurements[index] + largest = list(measurement.demand).index(max(measurement.demand)) + ratio = reserved_ratio(measurement.reserved[largest], measurement.allocated[largest]) + if ratio_outlier(ratio, [kept_ratio for pixels, kept_ratio in kept if pixels < measurement.pixels]): + flags[index] = True + elif ratio is not None and measurement.reference: + kept.append((measurement.pixels, ratio)) + return flags + + +def grew_around_flat_allocation( + demand: Sequence[int], reserved: Sequence[Optional[int]], allocated: Sequence[Optional[int]] +) -> bool: + """Whether the windows of one measurement, in the order they ran, show reserved growing by `FRAGMENTATION_GROWTH` + from the first window to the one of the largest demand while allocated stayed flat. False when that largest + window is the first, or a counter is missing.""" + largest = list(demand).index(max(demand)) + first, peak = reserved[0], reserved[largest] + if largest == 0 or first is None or peak is None or None in allocated: + return False + flat = [value for value in allocated if value is not None] + return peak >= FRAGMENTATION_GROWTH * first and max(flat) <= (1 + ALLOCATED_FLAT) * min(flat) + + +def reserved_ratio(reserved: Optional[int], allocated: Optional[int]) -> Optional[float]: + """Reserved over allocated bytes of one window; None where either is missing or allocated is not positive.""" + if reserved is None or allocated is None or allocated <= 0: + return None + return reserved / allocated + + +def ratio_outlier(ratio: Optional[float], smaller: Sequence[float]) -> bool: + """Whether a size's `ratio` is `RATIO_OUTLIER` times the median ratio of the same line's `smaller` sizes; False + without a ratio or a smaller size to compare with.""" + if ratio is None or not smaller: + return False + return ratio >= RATIO_OUTLIER * median(smaller) diff --git a/invokeai/backend/working_memory/platform.py b/invokeai/backend/working_memory/platform.py index 5dfc2a00997..6cab1793e02 100644 --- a/invokeai/backend/working_memory/platform.py +++ b/invokeai/backend/working_memory/platform.py @@ -1,45 +1,134 @@ +import os +import sys from collections.abc import Mapping -from dataclasses import dataclass -from types import MappingProxyType -from typing import Final, Literal +from dataclasses import astuple, dataclass, replace +from typing import Final, Literal, Optional, cast, get_args import torch +from invokeai.app.util.torch_cuda_allocator import ALLOCATOR_CONF_ENV_VARS, expandable_segments_enabled +from invokeai.backend.working_memory.counters import Backend, backend_for + ConvBackend = Literal["cudnn", "miopen", "mps", "xpu", "cpu"] +Allocator = Literal["native", "cudaMallocAsync", "expandable"] +"""torch's caching allocator: the native one, the native one in expandable-segments mode, or the driver's +``cudaMallocAsync`` pool (``hipMallocAsync`` on a HIP build, which torch reports under the same name).""" +OperatingSystem = Literal["windows", "linux", "other"] @dataclass(frozen=True) class Platform: - """The traits of the machine an operation runs on, as coefficients are keyed. Each field is a platform key an - `EstimatorSpec` can name, and holds the true value: what was measured, and what was not, is `UNMEASURED_PRICED_AS`'s - business.""" + """The traits of the machine an operation runs on, as coefficients are keyed. Each field holds the true value: + what was measured, and what was not, is `priced_as`'s business. + + `conv_backend` and `allocator` are the keys an `EstimatorSpec` names. `os` refines them only where an entry says + so: an entry whose platform names an `os` applies there and takes precedence over the entry that does not, so the + operating system enters the pricing of exactly the platforms measured on one OS or measured to differ by it, and + of no other. Both are MIOpen: its native allocator demands up to 1.6x more on Windows than on Linux, and expandable + segments, Invoke's Windows default, are measured only on Windows. + """ conv_backend: ConvBackend + allocator: Allocator + os: OperatingSystem -UNMEASURED_PRICED_AS: Final[Mapping[str, Mapping[str, str]]] = MappingProxyType( +MEASURED: Final = frozenset( { - # MPS, XPU and CPU convolutions are unmeasured and take the cuDNN coefficients. On MPS that pairs with a - # score-matrix term the SDPA probe always charges there, so the total is not thin. - "conv_backend": MappingProxyType({"mps": "cudnn", "xpu": "cudnn", "cpu": "cudnn"}), + ("cudnn", "native", None), + ("cudnn", "cudaMallocAsync", None), + ("miopen", "native", None), + ("miopen", "expandable", "windows"), } ) -"""Per platform key, the values no coefficients are measured for, and the measured value each is priced as. The one -place that borrowing is declared: the resolver applies it and names it in an estimate's `source`, and shipped data -carries entries for measured values only.""" +"""The (convolution backend, allocator, OS) combinations coefficients are measured on, None for every OS. Shipped data +carries entries for these only.""" + + +def is_measured(platform: Platform) -> bool: + return {(platform.conv_backend, platform.allocator, None), astuple(platform)} & MEASURED != set() + + +def priced_as(platform: Platform) -> Platform: + """The measured platform an operation on `platform` is priced as: `platform` itself where it is measured. The OS + stays: it only selects among the entries of the measured pair. + + The one place that borrowing is declared; the resolver applies it and names it in an estimate's `source`. + + - A convolution backend nothing is measured on (MPS, XPU, CPU) takes cuDNN under the native allocator. On MPS that + pairs with a score-matrix term the SDPA probe always charges there, so the total is not thin. + - An allocator not measured with its backend on this OS takes that backend's native allocator: cuDNN in + expandable-segments mode, MIOpen under ``hipMallocAsync``, and MIOpen with expandable segments off Windows (a + user override there; Linux defaults to native). On cuDNN, native is the larger of the two measured allocators + (``cudaMallocAsync`` needs 25-35% less); on MIOpen, native is the larger too, on Windows by up to 1.6x. + """ + if is_measured(platform): + return platform + if platform.conv_backend in ("mps", "xpu", "cpu"): + return replace(platform, conv_backend="cudnn", allocator="native") + return replace(platform, allocator="native") + + +_CONV_BACKENDS: Final[Mapping[Optional[Backend], ConvBackend]] = { + "cuda": "cudnn", + "hip": "miopen", + "mps": "mps", + "xpu": "xpu", + None: "cpu", +} +"""The convolution backend of each GPU backend. A HIP build reports its GPUs as ``cuda`` devices, so the torch build -- +not the device string -- separates MIOpen from cuDNN, and a `cpu_only` model on a ROCm build is not on MIOpen. Any +device type other than CUDA, MPS and XPU runs torch's CPU kernels, or none at all, and is priced as the CPU.""" + + +def _allocator(backend: Optional[Backend], allocator_backend: Optional[str], environ: Mapping[str, str]) -> Allocator: + """The caching allocator of a `backend` device under torch's allocator backend and these allocator variables: + `native` for anything but a CUDA/HIP GPU. The driver's pool ignores expandable segments, so torch's backend + decides first; expandable segments are a mode of the native allocator.""" + if backend not in ("cuda", "hip"): + return "native" + if allocator_backend == "cudaMallocAsync": + return "cudaMallocAsync" + return "expandable" if expandable_segments_enabled(environ) else "native" + + +def _allocator_for(device: torch.device) -> Allocator: + """The caching allocator an operation on `device` allocates through. + + The backend is fixed before torch initializes CUDA, so torch's answer holds for the process. The variables are read + from the environment the model cache reads them from. A ``cuda`` device on a build without one (a test pinning the + device) has no backend to ask. + """ + backend = backend_for(device) + allocator_backend = ( + torch.cuda.get_allocator_backend() if backend in ("cuda", "hip") and torch.cuda.is_available() else None + ) + return _allocator(backend, allocator_backend, os.environ) + + +def _host_os() -> OperatingSystem: + """The operating system this process runs on, as platforms name it.""" + if sys.platform == "win32": + return "windows" + return "linux" if sys.platform.startswith("linux") else "other" def platform_for(device: torch.device) -> Platform: - """The platform an operation on `device` runs on. + """The platform an operation on `device` runs on.""" + return Platform(_CONV_BACKENDS[backend_for(device)], _allocator_for(device), _host_os()) + - A HIP build reports ``device.type == "cuda"``, so the torch build -- not the device string -- is what separates - MIOpen from cuDNN, and a `cpu_only` model on a ROCm build is not on MIOpen. Any device type other than CUDA, MPS and - XPU runs torch's CPU kernels, or none at all, and is priced as the CPU. +def platform_for_environment(backend: Backend, hard: Mapping[str, str]) -> Platform: + """The platform `platform_for` gives a `backend` device in the environment whose hard facts are `hard` + (`fingerprint.environment_for`), as a calibration store records them: torch's allocator backend, the allocator + variables and the OS. Raises ValueError when they lack a fact the platform needs. """ - if device.type == "cuda": - return Platform(conv_backend="miopen" if torch.version.hip is not None else "cudnn") - if device.type == "mps": - return Platform(conv_backend="mps") - if device.type == "xpu": - return Platform(conv_backend="xpu") - return Platform(conv_backend="cpu") + os_name = hard.get("os") + if os_name not in get_args(OperatingSystem): + raise ValueError(f"The environment records no operating system the platform knows: os={os_name!r}.") + allocator_backend = hard.get("allocator_backend") + if backend in ("cuda", "hip") and allocator_backend is None: + raise ValueError(f"The environment of a {backend} device records no allocator_backend.") + environ = {var: hard.get(f"env.{var}", "") for var in ALLOCATOR_CONF_ENV_VARS} + allocator = _allocator(backend, allocator_backend, environ) + return Platform(_CONV_BACKENDS[backend], allocator, cast(OperatingSystem, os_name)) diff --git a/invokeai/backend/working_memory/probe.py b/invokeai/backend/working_memory/probe.py new file mode 100644 index 00000000000..839e2d785f2 --- /dev/null +++ b/invokeai/backend/working_memory/probe.py @@ -0,0 +1,768 @@ +"""The working-memory probe: what an operation really took on its device, recorded next to what was estimated for it. + +Active only inside a `calibration_scope`, which the session runner opens around each node of a calibration queue item. +There, every `LoadedModel.model_on_device(working_memory=...)` on a GPU is a *window*: the model is locked with +calibration headroom (`calibration_reservation`, so that a too-low estimate cannot spill and could never be corrected), +and from the moment the lock has settled until just before the unlock, the probe measures the device. The window's +`MeasuredWindow` lands in the scope's `windows`; nothing is written anywhere while a window is open. + +What a window measures, per counter (driver, torch reserved, torch allocated; see `counters`): + +1. Open: synchronize, release the allocator's cached blocks, read the baseline, reset the peaks. Tensors allocated + before the lock are part of the baseline: in use, as they are in the model cache's own budget. +2. A *co-lock* -- another model locked on the same thread and device while the window is open (a ControlNet, an audio + VAE, the vocoder) -- moves weights, not working memory. The segment ends before its lock (R0, read after a release) + and a new one starts after it settled (R2, after a release); R2 - R0 joins the window's offset, negative when the + lock moved more off the device than on. +3. A *nested window* (an estimated `model_on_device` inside an open one, such as a VAE encode inside a transformer's + window): its lock is a co-lock for the parent, the parent is frozen while it is open, and it measures alone. When + it closes, the parent releases the child's cached blocks, resets the peaks and goes on; the child's lasting outputs + stay in the parent's figures. +4. A `TorchDevice.empty_cache()` inside the window ends a segment: the driver is read before the release. A direct + `torch.cuda.empty_cache()` (a library's, or cuDNN's and MIOpen's after an algorithm search) and an offload on the + cache's background thread do not, so the driver figure can miss what they released; torch's peaks still bound it. +5. Close: synchronize and read the final segment. Per counter, the delta is the largest segment peak less the + baseline and that segment's offset; `demand_bytes` is the largest delta of the counters that could be read. + +A window is invalid (`valid=False`, with its reasons) when another thread locked a model on the same device's cache +while it was open, held one when it opened or was still locking one then (Expand Prompt and Image to Prompt run next to +sessions; see `locking`), when an allocation failed or only succeeded after the allocator released its cache (a retry +squeezes the peak: an untiled 3072px FLUX VAE decode on an RTX 4090 read 22.5 GiB at the card's limit, ~1280 bytes per +pixel instead of ~2150), when Windows still keeps more than `SPILL_TOLERANCE_BYTES` of the process in shared memory +`SPILL_SETTLE_S` after close (right after a decode the counter spikes transiently, 589 MiB, and is back at 9 MiB at the +next read), when the operation raised or was canceled, or its node failed or was canceled after the window closed +(`ProbeScope.invalidate`), when the Krea-2 SDPA benchmark (which resets the peaks itself) is active, when XPU's +device-wide usage grew beyond this process's own, when a probe step failed (`probe_error`: a reading, a release, a peak +reset or a synchronization, at the window's open as much as later; a failed lock hook also voids the scope's later +windows, since its lock bookkeeping is suspect), and when no counter could be read. A model less than fully resident at +open keeps the window valid but not `fit_eligible`: streamed weights are working memory the estimate does not price. + +Without a scope, the probe costs a context-variable read per estimated `model_on_device`, an entry in `locking` and a +read of `armed` per lock, a read of `armed` per unlock, and a truth test of `devices.EMPTY_CACHE_OBSERVERS` per +`empty_cache`. +""" + +import contextvars +import os +import threading +import time +from collections.abc import Iterator +from contextlib import contextmanager +from typing import TYPE_CHECKING, Final, Literal, Optional + +from pydantic import BaseModel, ConfigDict + +from invokeai.app.services.session_processor.session_processor_common import CanceledException +from invokeai.backend.util import devices +from invokeai.backend.util.devices import TorchDevice +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.working_memory import counters +from invokeai.backend.working_memory.counters import ( + VERIFIED_METHODS, + CounterReading, + DeviceCounters, + Method, + backend_for, +) +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm + +if TYPE_CHECKING: + from invokeai.backend.model_manager.load.model_cache.cache_record import CacheRecord + +GIB = 2**30 +MIB = 2**20 + +CALIBRATION_HEADROOM_MIN_BYTES: Final = 1 * GIB +CALIBRATION_HEADROOM_FRACTION: Final = 0.5 + +SPILL_TOLERANCE_BYTES: Final = 256 * MIB +"""Shared-memory bytes above the window's baseline that count as spill once they persist.""" +SPILL_SETTLE_S: float = 0.5 +"""How long a close waits before re-reading a raised shared-memory counter. Module-level so tests can shorten it.""" +FOREIGN_DEVICE_TOLERANCE_BYTES: Final = 256 * MIB +"""Growth of a device-wide counter (XPU) beyond this process's own reserved growth that marks another process's load.""" + +Reason = Literal[ + "foreign_lock", + "foreign_device_load", + "oom", + "alloc_retry", + "spill", + "exception", + "canceled", + "krea2_sdpa_benchmark", + "counters_unavailable", + "probe_error", + "partial_residency", +] +"""Why a window is invalid, or -- ``partial_residency`` alone -- valid but not fit-eligible.""" + +_NOT_INVALIDATING: Final[frozenset[Reason]] = frozenset({"partial_residency"}) + + +def _validity(reasons: list[Reason]) -> tuple[bool, bool]: + """``(valid, fit_eligible)`` of a window with `reasons`.""" + valid = all(reason in _NOT_INVALIDATING for reason in reasons) + return valid, valid and "partial_residency" not in reasons + + +CounterName = Literal["driver", "reserved", "allocated"] +_COUNTERS: Final[tuple[CounterName, ...]] = ("driver", "reserved", "allocated") + + +class _Record(BaseModel): + model_config = ConfigDict(frozen=True, extra="forbid") + + +class CounterDelta(_Record): + """One counter over a window, in bytes.""" + + baseline: int + peak: int + """The largest value any segment reached, offsets not applied.""" + offset: int + """What co-locks moved onto the device by the end of the window; negative when they moved more off.""" + delta: int + """The largest of each segment's peak less the baseline and that segment's offset: the operation's own share.""" + + +class MeasuredWindow(_Record): + """One measured `model_on_device` window: who ran it, what was estimated, and what the device saw.""" + + run_id: str + queue_item_id: int + node_id: str + node_type: str + model_key: str + device: str + + estimator: str + version: int + features: dict[str, FeatureValue] + terms: tuple[WorkingMemoryTerm, ...] + reservation_bytes: int + source: str + reservation_applied_bytes: int + """What the lock reserved: the estimate plus calibration headroom.""" + + driver: Optional[CounterDelta] + reserved: Optional[CounterDelta] + allocated: Optional[CounterDelta] + demand_bytes: Optional[int] + """The largest delta of the counters that could be read.""" + method: Method + """Where the driver figures came from; ``torch-only`` when there were none for the whole window.""" + verified: bool + """Whether that method was verified on this backend on real hardware (`counters.VERIFIED_METHODS`).""" + + resident_fraction: float + """The share of the window's model's weights on the device when the window opened.""" + co_locked_models: list[str] + """The other models this thread held on the device while the window was open.""" + segments: int + duration_s: float + + valid: bool + fit_eligible: bool + reasons: list[Reason] + + def invalidated(self, reason: Reason) -> "MeasuredWindow": + """This window with `reason` among its reasons, and valid and fit-eligible as they then are.""" + if reason in self.reasons: + return self + reasons = [*self.reasons, reason] + valid, fit_eligible = _validity(reasons) + return self.model_copy(update={"reasons": reasons, "valid": valid, "fit_eligible": fit_eligible}) + + +def calibration_reservation(reservation_bytes: int) -> int: + """What a calibration lock reserves for an estimate of `reservation_bytes`: half as much again, at least 1 GiB more.""" + return reservation_bytes + max( + CALIBRATION_HEADROOM_MIN_BYTES, int(CALIBRATION_HEADROOM_FRACTION * reservation_bytes) + ) + + +# All window state, and the registry of open windows, under one lock. Re-entrant because a window's own release runs +# `TorchDevice.empty_cache`, which notifies the probe on the same thread. Lock order: the model cache's lock before +# this one, never the other way round -- `TorchDevice.empty_cache` runs inside cache methods, so nothing here calls into +# a cache while holding it. +_state_lock = threading.RLock() +_open_windows: list["_Window"] = [] +_armed_scopes = 0 + +armed: bool = False +"""True while any thread has a calibration scope open. The model loader checks it before telling the probe about a lock +or an unlock, so without a scope that costs one attribute read.""" + +locking: list[object] = [] +"""The cache of every model lock in progress, once per lock, whichever thread takes it. The model loader adds it before +it reads `armed` and removes it once the lock has settled or failed. A lock that read `armed` before a scope armed never +reports to the probe, and one that reported its start before a window was registered reports again only once it has +settled, perhaps after the window closed. Both move weights while the window measures, so a window that opens while +another thread's lock is in progress on its cache is void. A list, so that the loader's side is an append and a remove, +each atomic under the GIL, and takes no lock.""" + +_boundary = threading.local() + + +def _release() -> None: + """Return the allocator's cached blocks to the driver as a boundary of this thread's windows, which handle it + themselves; other threads' windows see it like any other release.""" + _boundary.active = True + try: + TorchDevice.empty_cache(force=True) + finally: + _boundary.active = False + + +def _value(reading: CounterReading, name: CounterName) -> Optional[int]: + return reading.driver if name == "driver" else reading.reserved if name == "reserved" else reading.allocated + + +def _peak(reading: CounterReading, name: CounterName) -> Optional[int]: + if name == "driver": + return reading.peak_driver if reading.peak_driver is not None else reading.driver + return reading.peak_reserved if name == "reserved" else reading.peak_allocated + + +def _rose(before: Optional[int], after: Optional[int]) -> bool: + return before is not None and after is not None and after > before + + +def _krea2_sdpa_benchmark_active() -> bool: + # Imported here: the Krea-2 attention module pulls in diffusers, which a lock must not. + from invokeai.backend.krea2.attention import KREA2_SDPA_BACKEND_ENV_VAR + + # Any non-empty value runs the benchmark (`_Krea2StepBenchmark.create`), or fails the node if it is invalid. + return bool(os.environ.get(KREA2_SDPA_BACKEND_ENV_VAR, "").strip()) + + +class _Window: + """An open window's measurement. Mutated under `_state_lock` only.""" + + def __init__( + self, + scope: "ProbeScope", + cache: object, + record: "CacheRecord", + estimate: WorkingMemoryEstimate, + reservation_applied_bytes: int, + ) -> None: + self.scope = scope + self.cache = cache + self.record = record + self.estimate = estimate + self.reservation_applied_bytes = reservation_applied_bytes + self.device = record.cached_model.compute_device + self.thread = threading.get_ident() + self.parent: Optional[_Window] = None + self.counters: Optional[DeviceCounters] = None + self.state: Literal["measuring", "co_lock", "suspended"] = "measuring" + self.first: Optional[CounterReading] = None + self.baseline: dict[CounterName, Optional[int]] = {} + self.offset: dict[CounterName, int] = dict.fromkeys(_COUNTERS, 0) + self.peak: dict[CounterName, int] = {} + self.delta: dict[CounterName, int] = {} + self.unavailable: set[CounterName] = set() + self.reasons: list[Reason] = [] + self.segments = 0 + self.pending_reset = False + self.co_lock_record: Optional["CacheRecord"] = None + self.co_lock_start: Optional[CounterReading] = None + self.co_locked: list[str] = [] + self.resident_fraction = 1.0 + self.opened_at = 0.0 + self.recorded = False + + def add_reason(self, reason: Reason) -> None: + if reason not in self.reasons: + self.reasons.append(reason) + + def fail(self) -> None: + """A probe step on this window failed: its figures can no longer be trusted. Mark it, and leave it in a state + the rest of the window can run in.""" + self.add_reason("probe_error") + self.pending_reset = False + if self.state == "co_lock": + self.state = "measuring" + self.co_lock_record = self.co_lock_start = None + + def start(self, reading: CounterReading) -> None: + self.first = reading + self.opened_at = time.perf_counter() + for name in _COUNTERS: + self.baseline[name] = _value(reading, name) + if self.baseline[name] is None: + self.unavailable.add(name) + + def fold(self, reading: CounterReading, ends_segment: bool = True) -> None: + """Take `reading` as the end of the current segment, or with `ends_segment=False` as a sample within it.""" + for name in _COUNTERS: + if name in self.unavailable: + continue + value, base = _peak(reading, name), self.baseline[name] + if value is None or base is None: + self.unavailable.add(name) + continue + self.peak[name] = max(self.peak.get(name, value), value) + share = value - base - self.offset[name] + self.delta[name] = max(self.delta.get(name, share), share) + if ends_segment: + self.segments += 1 + + def begin_co_lock(self, record: "CacheRecord") -> None: + assert self.counters is not None + self.counters.synchronize() + self.fold(self.counters.read()) + _release() + self.co_lock_start = self.counters.read() + self.co_lock_record = record + self.state = "co_lock" + + def end_co_lock(self) -> None: + assert self.counters is not None and self.co_lock_start is not None and self.co_lock_record is not None + self.counters.synchronize() + _release() + settled = self.counters.read() + for name in _COUNTERS: + before, after = _value(self.co_lock_start, name), _value(settled, name) + if before is None or after is None: + self.unavailable.add(name) + else: + self.offset[name] += after - before + if self.co_lock_record.key not in self.co_locked and self.co_lock_record.key != self.record.key: + self.co_locked.append(self.co_lock_record.key) + self.co_lock_record = self.co_lock_start = None + self.counters.reset_peaks() + self.state = "measuring" + + def counter(self, name: CounterName) -> Optional[CounterDelta]: + base = self.baseline.get(name) + if name in self.unavailable or base is None or name not in self.delta: + return None + return CounterDelta(baseline=base, peak=self.peak[name], offset=self.offset[name], delta=self.delta[name]) + + def method(self) -> Method: + if self.first is None: + return "torch-only" + # A driver figure that dropped out mid-window leaves the window to torch's counters. + if self.first.driver is not None and "driver" in self.unavailable: + return "torch-only" + return self.first.method + + def measured(self, duration_s: float) -> MeasuredWindow: + counter_deltas = {name: self.counter(name) for name in _COUNTERS} + deltas = [delta.delta for delta in counter_deltas.values() if delta is not None] + if not deltas: + self.add_reason("counters_unavailable") + valid, fit_eligible = _validity(self.reasons) + method = self.method() + backend = backend_for(self.device) + estimate, scope = self.estimate, self.scope + return MeasuredWindow( + run_id=scope.run_id, + queue_item_id=scope.queue_item_id, + node_id=scope.node_id, + node_type=scope.node_type, + model_key=self.record.key, + device=str(self.device), + estimator=estimate.estimator, + version=estimate.version, + features=dict(estimate.features), + terms=estimate.terms, + reservation_bytes=estimate.reservation_bytes, + source=estimate.source, + reservation_applied_bytes=self.reservation_applied_bytes, + driver=counter_deltas["driver"], + reserved=counter_deltas["reserved"], + allocated=counter_deltas["allocated"], + demand_bytes=max(deltas) if deltas else None, + method=method, + verified=backend is not None and (method, backend) in VERIFIED_METHODS, + resident_fraction=self.resident_fraction, + co_locked_models=list(self.co_locked), + segments=self.segments, + duration_s=duration_s, + valid=valid, + fit_eligible=fit_eligible, + reasons=list(self.reasons), + ) + + def close(self, error: Optional[BaseException]) -> None: + """End the window on its own thread, `error` being what the operation raised, and record it in its scope.""" + try: + self._close(error) + except Exception: + InvokeAILogger.get_logger(__name__).warning( + f"Working-memory probe failed to close the window of {self.record.key}", exc_info=True + ) + self.abort() + + def abort(self) -> None: + """End the window after a probe step failed as it opened or closed; record it as invalid. Never raises.""" + try: + with _state_lock: + self.add_reason("probe_error") + self._unregister() + parent = self.parent + if parent is not None and parent.state == "suspended": + # Folded and released for this window, and not reset after it: what it measures from here is + # suspect. + parent.fail() + parent.state = "measuring" + if not self.recorded: + duration_s = time.perf_counter() - self.opened_at if self.first is not None else 0.0 + self.scope.windows.append(self.measured(duration_s)) + self.recorded = True + except Exception: + InvokeAILogger.get_logger(__name__).debug("Working-memory probe could not record", exc_info=True) + + def _unregister(self) -> None: + if self in _open_windows: + _open_windows.remove(self) + if self in self.scope._stack: + self.scope._stack.remove(self) + if self.counters is not None: + self.counters.close() + + def _close(self, error: Optional[BaseException]) -> None: + if error is not None: + self.add_reason("canceled" if isinstance(error, CanceledException) else "exception") + final: Optional[CounterReading] = None + with _state_lock: + if self.counters is not None and self.first is not None: + self.counters.synchronize() + final = self.counters.read() + self.fold(final) + first = self.first + if _rose(first.num_ooms, final.num_ooms): + self.add_reason("oom") + if _rose(first.num_alloc_retries, final.num_alloc_retries): + self.add_reason("alloc_retry") + if ( + first.device_used is not None + and final.device_used is not None + and first.reserved is not None + and final.reserved is not None + and (final.device_used - first.device_used) - (final.reserved - first.reserved) + > FOREIGN_DEVICE_TOLERANCE_BYTES + ): + self.add_reason("foreign_device_load") + if self in _open_windows: + _open_windows.remove(self) + if self in self.scope._stack: + self.scope._stack.remove(self) + duration_s = time.perf_counter() - self.opened_at + + if final is not None and self.first is not None and self.counters is not None: + baseline_paged = self.first.paged + if ( + baseline_paged is not None + and final.paged is not None + and final.paged - baseline_paged > SPILL_TOLERANCE_BYTES + ): + # Shared usage spikes for a moment after large transfers; only what is still there later is spill. + time.sleep(SPILL_SETTLE_S) + settled = self.counters.read().paged + if settled is None or settled - baseline_paged > SPILL_TOLERANCE_BYTES: + self.add_reason("spill") + + with _state_lock: + parent = self.parent + if parent is not None and parent in _open_windows and parent.counters is not None: + _release() + parent.counters.reset_peaks() + parent.state = "measuring" + if self.counters is not None: + self.counters.close() + record = self.measured(duration_s) + self.scope.windows.append(record) + self.recorded = True + InvokeAILogger.get_logger(__name__).debug( + f"Measured {record.estimator} on {record.device} for {record.model_key}: " + f"demand {(record.demand_bytes or 0) / MIB:.0f} MB against {record.reservation_bytes / MIB:.0f} MB " + f"estimated ({record.method}{'' if record.valid else ', invalid: ' + ', '.join(record.reasons)})" + ) + + +class ProbeScope: + """One node of a calibration queue item, on the thread that runs it. `windows` collects that node's measurements + in the order they closed; the caller persists them after the node.""" + + def __init__(self, *, run_id: str, queue_item_id: int, node_id: str, node_type: str) -> None: + self.run_id = run_id + self.queue_item_id = queue_item_id + self.node_id = node_id + self.node_type = node_type + self.windows: list[MeasuredWindow] = [] + # This thread's open windows, innermost last. + self._stack: list[_Window] = [] + # The locks this thread holds, per (cache, model key). + self._held: dict[tuple[int, str], int] = {} + # Set when a probe hook failed on this thread: `_held` may be off, so no later window can trust its check. + self._tainted = False + + def _held_on(self, cache: object) -> int: + return sum(count for (cache_id, _key), count in self._held.items() if cache_id == id(cache)) + + def invalidate(self, reason: Literal["exception", "canceled"]) -> None: + """Void every window recorded so far: the node failed or was canceled. + + The session runner swallows a node's failure or cancellation, and calls this instead. It covers the windows that + closed before the node raised: their own operation finished, but the node did not, and no measurement of a node + that did not complete is trusted. + """ + self.windows = [window.invalidated(reason) for window in self.windows] + + def open_window( + self, cache: object, record: "CacheRecord", estimate: WorkingMemoryEstimate, reservation_applied_bytes: int + ) -> Optional[_Window]: + """Start measuring the operation `record`'s model is about to run, once its lock has settled. + + `cache` is the model cache the record is locked in. Never raises: a probe failure leaves the operation alone + and records the window as invalid. A window that cannot be opened is recorded at once, and the operation runs + unmeasured. + """ + window = _Window(self, cache, record, estimate, reservation_applied_bytes) + # Registered before the cache is asked: from here on, a lock another thread starts or settles on this cache + # voids the window (`lock_started`, `lock_settled`), and one in progress or held shows in `locking` and + # `locks_held()`. Its counters are not open yet, so releases pass it by. + with _state_lock: + _open_windows.append(window) + try: + if self._tainted: + window.add_reason("probe_error") + cached_model = record.cached_model + total = cached_model.total_bytes() + resident = cached_model.cur_vram_bytes() + window.resident_fraction = resident / total if total > 0 else 1.0 + if resident < total: + window.add_reason("partial_residency") + window.co_locked = sorted( + key + for (cache_id, key), count in self._held.items() + if cache_id == id(cache) and count > 0 and key != record.key + ) + # Asked outside the probe's lock (see `_state_lock`). A lock in progress is another thread's: this thread + # opens a window only once its own lock has settled. Every lock this thread took since the scope opened is + # in `_held`, so any lock held beyond those is another thread's too. `locking` is read first: a lock that + # leaves it before `locks_held()` is read still holds its record, or is over before the baseline. + in_progress = any(entry is cache for entry in tuple(locking)) + if in_progress or cache.locks_held() > self._held_on(cache): # type: ignore[attr-defined] + window.add_reason("foreign_lock") + if window.device.type == "cuda" and _krea2_sdpa_benchmark_active(): + window.add_reason("krea2_sdpa_benchmark") + except Exception: + InvokeAILogger.get_logger(__name__).debug("Working-memory probe could not inspect the lock", exc_info=True) + window.add_reason("probe_error") + + try: + window.counters = counters.DeviceCounters(window.device) + with _state_lock: + parent = self._stack[-1] if self._stack else None + window.parent = parent + window.counters.synchronize() + if parent is not None and parent.counters is not None: + if parent.state == "measuring": + parent.fold(parent.counters.read()) + parent.state = "suspended" + _release() + window.start(window.counters.read()) + window.counters.reset_peaks() + self._stack.append(window) + return window + except Exception: + InvokeAILogger.get_logger(__name__).warning( + f"Working-memory probe could not open a window for {record.key}", exc_info=True + ) + window.abort() + return None + + def _abandon(self) -> None: + """Drop the windows still open when the scope ends, so no sampler outlives it.""" + with _state_lock: + for window in list(self._stack): + window._unregister() + + +_current_scope: contextvars.ContextVar[Optional[ProbeScope]] = contextvars.ContextVar( + "working_memory_probe_scope", default=None +) + + +def current_scope() -> Optional[ProbeScope]: + """The calibration scope open on this thread, if any.""" + return _current_scope.get() + + +class _EmptyCacheObserver: + """Turns `TorchDevice.empty_cache` into segment boundaries of the open windows.""" + + def before_empty_cache(self) -> None: + try: + me = threading.get_ident() + own_boundary = getattr(_boundary, "active", False) + with _state_lock: + for window in _open_windows: + if window.counters is None or window.state != "measuring": + continue + if window.thread != me: + # Another thread's release (process-global on CUDA) takes this window's cached blocks too: + # read its driver before they go, but leave its peaks alone mid-operation. + window.fold(window.counters.read(), ends_segment=False) + elif not own_boundary: + window.fold(window.counters.read()) + window.pending_reset = True + except Exception: + _all_failed("follow a release") + + def after_empty_cache(self) -> None: + try: + with _state_lock: + for window in _open_windows: + if window.pending_reset and window.counters is not None: + window.counters.reset_peaks() + window.pending_reset = False + except Exception: + _all_failed("follow a release") + + +_observer = _EmptyCacheObserver() + + +def _arm() -> None: + global armed, _armed_scopes + with _state_lock: + _armed_scopes += 1 + if _armed_scopes == 1: + devices.EMPTY_CACHE_OBSERVERS.append(_observer) + armed = True + + +def _disarm() -> None: + global armed, _armed_scopes + with _state_lock: + _armed_scopes -= 1 + if _armed_scopes == 0: + armed = False + if _observer in devices.EMPTY_CACHE_OBSERVERS: + devices.EMPTY_CACHE_OBSERVERS.remove(_observer) + + +@contextmanager +def calibration_scope(*, run_id: str, queue_item_id: int, node_id: str, node_type: str) -> Iterator[ProbeScope]: + """Measure the windows of one node on this thread. Scopes do not nest.""" + if _current_scope.get() is not None: + raise RuntimeError("A calibration scope is already open on this thread; scopes do not nest.") + scope = ProbeScope(run_id=run_id, queue_item_id=queue_item_id, node_id=node_id, node_type=node_type) + token = _current_scope.set(scope) + _arm() + try: + yield scope + finally: + scope._abandon() + _disarm() + _current_scope.reset(token) + + +def _all_failed(what: str) -> None: + """A release boundary failed part-way: any open window may have been folded without its reset.""" + InvokeAILogger.get_logger(__name__).debug(f"Working-memory probe failed to {what}", exc_info=True) + try: + with _state_lock: + for window in _open_windows: + window.fail() + except Exception: + pass + + +def _hook_failed(scope: Optional[ProbeScope], cache: object, what: str) -> None: + """A lock hook failed part-way. On a scope's own thread every window of the scope is suspect, and so is its lock + bookkeeping; on another thread, the windows on that cache which the hook was there to void.""" + InvokeAILogger.get_logger(__name__).debug(f"Working-memory probe failed to {what}", exc_info=True) + try: + with _state_lock: + if scope is not None: + scope._tainted = True + affected = list(scope._stack) + else: + affected = [window for window in _open_windows if window.cache is cache] + for window in affected: + window.fail() + except Exception: + pass + + +def _invalidate_foreign(cache: object) -> None: + me = threading.get_ident() + for window in _open_windows: + if window.cache is cache and window.thread != me: + window.add_reason("foreign_lock") + + +def lock_started(cache: object, record: "CacheRecord") -> None: + """A lock of `record` on its compute device begins on this thread. Called by the model loader while `armed`.""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is None or not scope._stack: + return + window = scope._stack[-1] + if ( + window.cache is cache + and window.state == "measuring" + and record.cached_model.compute_device.type != "cpu" + ): + window.begin_co_lock(record) + except Exception: + _hook_failed(scope, cache, "follow a lock") + + +def lock_settled(cache: object, record: "CacheRecord", locked: bool) -> None: + """That lock has settled, its required tensors repaired, or it failed (`locked=False`, nothing stays locked).""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is None: + return + if locked: + key = (id(cache), record.key) + scope._held[key] = scope._held.get(key, 0) + 1 + if scope._stack: + window = scope._stack[-1] + if window.state == "co_lock" and window.co_lock_record is record: + window.end_co_lock() + except Exception: + _hook_failed(scope, cache, "follow a lock") + + +def locked_in_ram(cache: object, record: "CacheRecord") -> None: + """`record` was pinned in RAM on this thread: no device memory moved, but the lock counts for `locks_held`.""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is not None: + key = (id(cache), record.key) + scope._held[key] = scope._held.get(key, 0) + 1 + except Exception: + _hook_failed(scope, cache, "follow a lock") + + +def unlocked(cache: object, record: "CacheRecord") -> None: + """`record` was unlocked on this thread.""" + scope = _current_scope.get() + try: + with _state_lock: + _invalidate_foreign(cache) + if scope is not None: + key = (id(cache), record.key) + remaining = scope._held.get(key, 0) - 1 + if remaining > 0: + scope._held[key] = remaining + else: + scope._held.pop(key, None) + except Exception: + _hook_failed(scope, cache, "follow an unlock") diff --git a/invokeai/backend/working_memory/spec.py b/invokeai/backend/working_memory/spec.py index 3f0c116a2eb..ce7635e6702 100644 --- a/invokeai/backend/working_memory/spec.py +++ b/invokeai/backend/working_memory/spec.py @@ -6,6 +6,7 @@ import torch +from invokeai.backend.working_memory import user_fit from invokeai.backend.working_memory.coefficients import ResolvedCoefficients, shipped_coefficients from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate, WorkingMemoryTerm from invokeai.backend.working_memory.platform import platform_for @@ -37,8 +38,14 @@ class EstimatorSpec: terms: TermsFunction = field(compare=False, repr=False) def estimate(self, features: Mapping[str, FeatureValue], device: torch.device) -> WorkingMemoryEstimate: - """The estimate for an operation on `device`, priced with the shipped coefficients of the platform it runs on.""" - return self.evaluate(features, shipped_coefficients(self, platform_for(device))) + """The estimate for an operation on `device`, priced with the shipped coefficients of the platform it runs on, + and with the user's own line where a calibration of this device measured it (`user_fit.resolve`).""" + platform = platform_for(device) + shipped = shipped_coefficients(self, platform) + fit = user_fit.user_fit_for(self, platform, device) + if fit is None: + return self.evaluate(features, shipped) + return user_fit.resolve(self, features, shipped, fit) def evaluate( self, features: Mapping[str, FeatureValue], coefficients: ResolvedCoefficients diff --git a/invokeai/backend/working_memory/user_fit.py b/invokeai/backend/working_memory/user_fit.py new file mode 100644 index 00000000000..83e2107a536 --- /dev/null +++ b/invokeai/backend/working_memory/user_fit.py @@ -0,0 +1,213 @@ +"""A user's own calibration, fitted per line and read by estimates in place of the shipped values where it measured. + +The app keeps a device class's calibration windows (`app/services/working_memory_calibration/`); this module owns the +rule that turns them into lines and the rule by which an estimate uses them. Lines are those of `fit.py`: a window +joins the line its estimator resolves to on the platform it ran on -- the lender's, where the estimator borrows its +values -- so a calibration of one model's decode moves every estimator that prices with the same line. + +An estimate reads a user line through the provider seam: the app registers one (`set_user_fit_provider`) that answers +from memory, and `EstimatorSpec.estimate` asks it per estimate. Without a provider, or when it has no fit for the +device, the estimate is the shipped one, bit for bit. With one, `resolve` applies the user line where it was measured: + +- at or below the largest size the line measured (`UserLine.x_max`), the user line replaces the shipped one; +- above it, the larger of the user and the shipped estimate, since nothing measured says how the line continues. +""" + +import functools +from collections.abc import Callable, Iterable, Mapping +from dataclasses import asdict, dataclass, field, replace +from types import MappingProxyType +from typing import TYPE_CHECKING, Final, Optional + +import torch + +from invokeai.backend.util.logging import InvokeAILogger +from invokeai.backend.working_memory.coefficients import ResolvedCoefficients, find_entry, shipped_data +from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate +from invokeai.backend.working_memory.fit import EntryKey, entry_key, fits_a_line, line_of, user_pair +from invokeai.backend.working_memory.platform import Platform, priced_as + +if TYPE_CHECKING: + from invokeai.backend.working_memory.spec import EstimatorSpec + +LineKey = tuple[EntryKey, str] +"""A line: the entry that carries its values, and its ``_`` suffix.""" + + +@dataclass(frozen=True) +class UserPoint: + """One calibration window as a fit reads it.""" + + estimator: str + version: int + features: Mapping[str, FeatureValue] + demand_bytes: int + + +@dataclass(frozen=True) +class UserLine: + """A line fitted on a user's windows (`fit.user_pair`), and the sizes it was fitted on.""" + + k: int + """Bytes per output pixel.""" + b: int + """Bytes whatever the size.""" + x_min: int + """The smallest size measured, in output pixels.""" + x_max: int + """The largest size measured, in output pixels.""" + points: int + + +@dataclass(frozen=True) +class UserFit: + """A device class's user lines in one environment, on the platform they were measured on.""" + + label: str + """Names the fit in an estimate's `source`, e.g. ``user::``.""" + platform: Platform + lines: Mapping[LineKey, UserLine] + _by_spec: dict[tuple[str, int], Mapping[str, UserLine]] = field( + default_factory=dict, init=False, repr=False, compare=False + ) + + def lines_for(self, spec: "EstimatorSpec") -> Mapping[str, UserLine]: + """The user lines `spec` prices with on this fit's platform, by suffix. Resolved once per spec.""" + key = (spec.id, spec.version) + found = self._by_spec.get(key) + if found is None: + carrier = carrier_key(spec.id, spec.version, self.platform) + found = MappingProxyType({suffix: line for (entry, suffix), line in self.lines.items() if entry == carrier}) + self._by_spec[key] = found + return found + + +@functools.cache +def carrier_key(estimator: str, version: int, platform: Platform) -> Optional[EntryKey]: + """The shipped entry whose values `estimator` `version` prices with on `platform`: its own, or the one it borrows + from. None where the shipped data has no entry for it.""" + full = {key: str(value) for key, value in asdict(priced_as(platform)).items()} + entries = [entry for file in shipped_data().values() for entry in file.coefficients] + where = f"{estimator} on {full}" + try: + entry = find_entry(entries, estimator, version, full, where) + if entry.values_from is not None: + entry = find_entry(entries, entry.values_from.estimator, entry.values_from.version, full, where) + except LookupError: + return None + return entry_key(entry) + + +def _carried_lines(key: EntryKey) -> frozenset[str]: + entries = [entry for file in shipped_data().values() for entry in file.coefficients] + for entry in entries: + if entry_key(entry) == key and entry.values is not None: + names = set(entry.values) + return frozenset(name[2:] for name in names if name.startswith("k_") and f"b_{name[2:]}" in names) + return frozenset() + + +def fit_user_lines( + points: Iterable[UserPoint], platform: Platform +) -> tuple[Mapping[LineKey, UserLine], Mapping[LineKey, str]]: + """The user lines `points`, measured on `platform`, fit; and, per line they measure but cannot fit, why. + + A point joins a line where `fit.fits_a_line` admits its features, its estimator version has shipped values on + the platform, the entry carrying them has the line, and it demanded more than nothing. Each line is + `fit.user_pair` of its points: two sizes at least `fit.MIN_AREA_RATIO` apart, or no line. + """ + groups: dict[LineKey, list[tuple[int, int]]] = {} + for point in points: + found = line_of(point.features) + if found is None or point.demand_bytes <= 0 or not fits_a_line(point.features): + continue + carrier = carrier_key(point.estimator, point.version, platform) + suffix, pixels = found + if carrier is None or suffix not in _carried_lines(carrier): + continue + groups.setdefault((carrier, suffix), []).append((pixels, point.demand_bytes)) + + lines: dict[LineKey, UserLine] = {} + problems: dict[LineKey, str] = {} + for key, measured in groups.items(): + try: + k, b = user_pair(measured) + except ValueError as e: + problems[key] = str(e) + continue + sizes = [pixels for pixels, _ in measured] + lines[key] = UserLine(k=k, b=b, x_min=min(sizes), x_max=max(sizes), points=len(measured)) + return MappingProxyType(lines), MappingProxyType(problems) + + +def resolve( + spec: "EstimatorSpec", + features: Mapping[str, FeatureValue], + shipped: ResolvedCoefficients, + fit: UserFit, +) -> WorkingMemoryEstimate: + """The estimate for `features` with `fit`'s line in place of the shipped one, where `fit` has the line. + + The line and its size are read as `fit.py` reads a point's (`line_of`): the output's pixels, also for a tiled + operation, whose image is at least its tile, so a tiled operation past the measured sizes takes the larger + estimate. At or below `UserLine.x_max` the user line decides; above, the larger estimate wins, the shipped one on + a tie. `source` names the user fit and the line wherever the user line decides. + """ + found = line_of(features) + line = fit.lines_for(spec).get(found[0]) if found is not None else None + if found is None or line is None: + return spec.evaluate(features, shipped) + suffix, pixels = found + values = MappingProxyType({**shipped.values, f"k_{suffix}": line.k, f"b_{suffix}": line.b}) + within = pixels <= line.x_max + where = "within its measured sizes" if within else "above its measured sizes, over shipped" + user = spec.evaluate( + features, ResolvedCoefficients(values=values, source=f"{fit.label} ({suffix} {where}) + {shipped.source}") + ) + if within: + return user + shipped_estimate = spec.evaluate(features, shipped) + if user.reservation_bytes > shipped_estimate.reservation_bytes: + return user + return replace(shipped_estimate, source=f"{shipped.source} (over {fit.label} {suffix}, above its measured sizes)") + + +UserFitProvider = Callable[["EstimatorSpec", Platform, torch.device], Optional[UserFit]] +"""Answers, from memory, the user fit an estimate of `spec` on `device` reads, or None. Called per estimate, so it +reads no file and touches no device.""" + +_provider: Optional[UserFitProvider] = None +_logged_failure: bool = False + +_LOGGER: Final = InvokeAILogger.get_logger(__name__) + + +def set_user_fit_provider(provider: UserFitProvider) -> None: + """Make `provider` the one estimates ask. The app's calibration service registers it when it starts.""" + global _provider + _provider = provider + + +def clear_user_fit_provider(provider: UserFitProvider) -> None: + """Stop asking `provider`, if it is the registered one: a service that stops must not unregister its successor.""" + global _provider + if _provider is provider: + _provider = None + + +def user_fit_for(spec: "EstimatorSpec", platform: Platform, device: torch.device) -> Optional[UserFit]: + """The registered provider's answer, None without one. A provider that raises leaves the estimate shipped: an + estimate must not fail over a calibration.""" + global _logged_failure + provider = _provider + if provider is None: + return None + try: + return provider(spec, platform, device) + except Exception: + if not _logged_failure: + _logged_failure = True + _LOGGER.warning( + "The working-memory user fit could not be read; estimates use the shipped values.", exc_info=True + ) + return None diff --git a/invokeai/frontend/api/openapi.json b/invokeai/frontend/api/openapi.json index 537d0836d51..969b0c477a0 100644 --- a/invokeai/frontend/api/openapi.json +++ b/invokeai/frontend/api/openapi.json @@ -5660,6 +5660,478 @@ ] } }, + "/api/v2/models/i/{key}/working_memory/calibrate": { + "post": { + "tags": ["model_manager"], + "summary": "Calibrate Model Working Memory", + "description": "Start a working-memory calibration run: enqueue the batch with every node uncached and measure each node's\nmodel operations on the generation device. Needs exactly one generation device, and it must be a GPU. The queue\ntakes all of the batch's items or none.\n\nOnce the items are queued, a request without an `idempotency_key` succeeds even when the store fails for a passing\nreason (an unwritable root, a lock timeout) to record them: they run all the same, and the status finds them by\ntheir origin. With a key, that failure is a 503 although the items are queued. A run removed meanwhile (404) or an\nunusable store file (409) fails the request with or without a key; its queued items then fail when they start.\n\nWith a key, repeating the request (also after a lost response) records the queued items and returns the run; it\nenqueues nothing new. The items come from the queue's enqueue receipt for the key while it exists, otherwise from\nthe queue items that carry the run's origin and those the runner saw. Once a reset removed the run, a repeat is\nrefused (404). Acknowledge the key through the queue's enqueue_batch/acknowledge route once the response\narrived.", + "operationId": "calibrate_model_working_memory", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model the run calibrates", + "title": "Key" + }, + "description": "Key of the model the run calibrates" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRequest", + "description": "The batch to measure" + } + } + } + }, + "responses": { + "201": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRunCreated" + } + } + } + }, + "404": { + "description": "The model could not be found; the run was removed while it was being created, whose items are queued then and fail when they start; or a reset removed the run the idempotency key started" + }, + "409": { + "description": "This machine cannot calibrate now (no GPU, or several generation devices), the device class's store file is unusable and kept as it is (possibly after the items were queued, which then fail when they start), the queue has no room for all of the run's items (it then takes none), or the idempotency key was used for another request" + }, + "422": { + "description": "The graph cannot be measured as a calibration run" + }, + "429": { + "description": "Too many unacknowledged idempotency keys; acknowledge or wait, then retry" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout); with an idempotency key, possibly after the items were queued" + } + } + } + }, + "/api/v2/models/working_memory/status": { + "get": { + "tags": ["model_manager"], + "summary": "Get Working Memory Calibration Status", + "description": "Every device class's calibration runs, with their derived status, point counts and environments.", + "operationId": "get_working_memory_calibration_status", + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationStatus" + } + } + } + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + } + }, + "security": [ + { + "HTTPBearer": [] + } + ] + } + }, + "/api/v2/models/i/{key}/working_memory/plan": { + "get": { + "tags": ["model_manager"], + "summary": "Get Model Working Memory Calibration Plan", + "description": "The text-to-image cases that calibrate the model's decode on this machine's GPU, how often each runs, which\nother installed models the calibration also moves, or why the model cannot be calibrated here (`supported`).", + "operationId": "get_model_working_memory_calibration_plan", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model to calibrate", + "title": "Key" + }, + "description": "Key of the model to calibrate" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationPlan" + } + } + } + }, + "404": { + "description": "The model could not be found" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/i/{key}/working_memory/summary": { + "get": { + "tags": ["model_manager"], + "summary": "Get Model Working Memory Calibration Summary", + "description": "What the calibration windows of the model's lines measured on one device class, per estimator line and\nenvironment: the sizes, the largest demand, the shipped and the user estimate, whether the user line is in use,\nstale or unusable, and which models measured it, also for a line the model's decode shares with other models'\nmeasurements; and the model's runs.", + "operationId": "get_model_working_memory_calibration_summary", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model whose measurements to summarise", + "title": "Key" + }, + "description": "Key of the model whose measurements to summarise" + }, + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationModelSummary" + } + } + } + }, + "404": { + "description": "The model could not be found" + }, + "409": { + "description": "No device_class_id was given and there is no current one" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/working_memory/runs/{run_id}": { + "delete": { + "tags": ["model_manager"], + "summary": "Delete Working Memory Calibration Run", + "description": "Remove a calibration run and its measurements. Its queue items that have not run yet fail when they start.", + "operationId": "delete_working_memory_calibration_run", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "run_id", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "The calibration run to remove", + "title": "Run Id" + }, + "description": "The calibration run to remove" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationResetResult" + } + } + } + }, + "404": { + "description": "No usable store file holds the run" + }, + "409": { + "description": "The run's store file became unusable after the run was found; it is kept as it is" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/i/{key}/working_memory": { + "delete": { + "tags": ["model_manager"], + "summary": "Reset Model Working Memory Calibration", + "description": "Remove a model's calibration runs with all their measurements, and every measurement of the model taken in\nother runs (a VAE measured inside a transformer's run, say), from one device class.", + "operationId": "reset_model_working_memory_calibration", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "key", + "in": "path", + "required": true, + "schema": { + "type": "string", + "description": "Key of the model whose runs and measurements to remove", + "title": "Key" + }, + "description": "Key of the model whose runs and measurements to remove" + }, + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationResetResult" + } + } + } + }, + "409": { + "description": "No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/working_memory": { + "delete": { + "tags": ["model_manager"], + "summary": "Reset Working Memory Calibration", + "description": "Remove every calibration run and measurement of one device class.", + "operationId": "reset_working_memory_calibration", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "Successful Response", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationResetResult" + } + } + } + }, + "409": { + "description": "No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, + "/api/v2/models/working_memory/export": { + "get": { + "tags": ["model_manager"], + "summary": "Export Working Memory Calibration", + "description": "The device class's calibration store file as it is on disk, readable by this version or not.", + "operationId": "export_working_memory_calibration", + "security": [ + { + "HTTPBearer": [] + } + ], + "parameters": [ + { + "name": "device_class_id", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + { + "type": "null" + } + ], + "description": "The device class to act on; the current generation device's by default", + "title": "Device Class Id" + }, + "description": "The device class to act on; the current generation device's by default" + } + ], + "responses": { + "200": { + "description": "The device class's store file byte for byte: a CalibrationStoreFile (schema_version 1) when this version can read it, otherwise whatever the file holds", + "content": { + "application/json": {} + } + }, + "404": { + "description": "The device class has no store file" + }, + "409": { + "description": "No device_class_id was given and there is no current one" + }, + "503": { + "description": "The calibration store cannot be read or written (unwritable root, lock timeout)" + }, + "422": { + "description": "Validation Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + } + } + } + } + }, "/api/v1/image_moves/start": { "post": { "tags": ["image_moves"], @@ -12557,6 +13029,9 @@ }, "description": "Created" }, + "400": { + "description": "The batch's origin starts with 'invokeai:wm-calibration:' (in any case), which is reserved for working-memory calibration runs" + }, "422": { "description": "Validation Error", "content": { @@ -24571,6 +25046,48 @@ "title": "CalculateImageTilesOutput", "type": "object" }, + "CalibrationDeviceClass": { + "properties": { + "id": { + "type": "string", + "title": "Id", + "description": "A short stable hash of the other fields" + }, + "vendor": { + "type": "string", + "title": "Vendor" + }, + "name": { + "type": "string", + "title": "Name" + }, + "total_mib": { + "type": "integer", + "title": "Total Mib" + }, + "backend": { + "type": "string", + "enum": ["cuda", "hip", "mps", "xpu"], + "title": "Backend" + }, + "arch": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Arch" + } + }, + "additionalProperties": false, + "type": "object", + "required": ["id", "vendor", "name", "total_mib", "backend", "arch"], + "title": "CalibrationDeviceClass", + "description": "A `fingerprint.DeviceClass` with its id as a plain field, as it is stored and returned." + }, "CallSavedWorkflowInvocation": { "category": "workflow", "class": "invocation", @@ -116446,6 +116963,1095 @@ ], "title": "WorkflowWithoutID" }, + "WorkingMemoryCalibrationCase": { + "properties": { + "id": { + "type": "string", + "title": "Id", + "description": "Unique within the plan; becomes the graph id" + }, + "mode": { + "type": "string", + "const": "txt2img", + "title": "Mode", + "default": "txt2img" + }, + "model_key": { + "type": "string", + "title": "Model Key", + "description": "The main model the case generates with" + }, + "components": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Components", + "description": "Component picks by Generate slot key; `vae` when a standalone VAE is calibrated or a GPU VAE must be picked over one that runs on the CPU" + }, + "width": { + "type": "integer", + "title": "Width" + }, + "height": { + "type": "integer", + "title": "Height" + }, + "steps": { + "type": "integer", + "title": "Steps", + "description": "The fewest steps that still run the whole pipeline" + }, + "seed": { + "type": "integer", + "title": "Seed" + }, + "positive_prompt": { + "type": "string", + "title": "Positive Prompt" + }, + "negative_prompt": { + "type": "string", + "title": "Negative Prompt" + }, + "settings": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Settings", + "description": "Generate settings overrides: `vaePrecision` (fp16 or fp32) where the decode node casts the VAE" + }, + "estimator": { + "type": "string", + "title": "Estimator", + "description": "The working-memory estimator of the operation the case measures" + }, + "line": { + "type": "string", + "title": "Line", + "description": "The estimator line the case measures, `_`" + } + }, + "type": "object", + "required": [ + "id", + "model_key", + "components", + "width", + "height", + "steps", + "seed", + "positive_prompt", + "negative_prompt", + "settings", + "estimator", + "line" + ], + "title": "WorkingMemoryCalibrationCase", + "description": "One generation a calibration plan runs, in the fields of the client's graph-export case (mode txt2img)." + }, + "WorkingMemoryCalibrationDeviceClass": { + "properties": { + "device_class_id": { + "type": "string", + "title": "Device Class Id" + }, + "device_class": { + "anyOf": [ + { + "$ref": "#/components/schemas/CalibrationDeviceClass" + }, + { + "type": "null" + } + ], + "description": "Null when the file is unusable" + }, + "usable": { + "type": "boolean", + "title": "Usable" + }, + "problem": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Problem", + "description": "Why the file is unusable. It is kept as it is and never written" + }, + "is_current": { + "type": "boolean", + "title": "Is Current" + }, + "measurement_method": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Measurement Method", + "description": "Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present" + }, + "measurement_verified": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Measurement Verified", + "description": "Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present" + }, + "runs": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRunSummary" + }, + "type": "array", + "title": "Runs" + }, + "point_counts": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationPointCount" + }, + "type": "array", + "title": "Point Counts" + }, + "envs": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationEnvironment" + }, + "type": "array", + "title": "Envs" + } + }, + "type": "object", + "required": [ + "device_class_id", + "device_class", + "usable", + "problem", + "is_current", + "measurement_method", + "measurement_verified", + "runs", + "point_counts", + "envs" + ], + "title": "WorkingMemoryCalibrationDeviceClass" + }, + "WorkingMemoryCalibrationDeviceClassRef": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "name": { + "type": "string", + "title": "Name" + } + }, + "type": "object", + "required": ["id", "name"], + "title": "WorkingMemoryCalibrationDeviceClassRef" + }, + "WorkingMemoryCalibrationEnvironment": { + "properties": { + "id": { + "type": "string", + "title": "Id" + }, + "hard": { + "additionalProperties": { + "type": "string" + }, + "type": "object", + "title": "Hard" + }, + "soft": { + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "type": "object", + "title": "Soft" + }, + "hard_matches_current": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Hard Matches Current", + "description": "Whether the current environment has the same hard facts; null off the current device class" + }, + "soft_matches_current": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Soft Matches Current", + "description": "Whether the current environment has the soft facts first recorded with this environment (`soft`); a later run's own soft facts are not compared here. Null off the current device class" + }, + "point_count": { + "type": "integer", + "title": "Point Count" + } + }, + "type": "object", + "required": ["id", "hard", "soft", "hard_matches_current", "soft_matches_current", "point_count"], + "title": "WorkingMemoryCalibrationEnvironment" + }, + "WorkingMemoryCalibrationLineContributor": { + "properties": { + "model_key": { + "type": "string", + "title": "Model Key" + }, + "name": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Name", + "description": "The model's name; null when it is no longer installed" + }, + "windows": { + "type": "integer", + "title": "Windows", + "description": "Its counted windows on the line: valid, fit-eligible and not fragmented" + } + }, + "type": "object", + "required": ["model_key", "name", "windows"], + "title": "WorkingMemoryCalibrationLineContributor", + "description": "A model whose windows feed a line: the model a run calibrated, or the model a window measured." + }, + "WorkingMemoryCalibrationLineSize": { + "properties": { + "width": { + "type": "integer", + "title": "Width" + }, + "height": { + "type": "integer", + "title": "Height" + }, + "pixels": { + "type": "integer", + "title": "Pixels" + }, + "windows": { + "type": "integer", + "title": "Windows", + "description": "Counted: the windows at this size that feed the line, valid, fit-eligible and not fragmented, of every model in the environment; those of a calibration run still running are counted, and join the fit once it finished. Not counted: the valid windows measured at this size" + }, + "measured_bytes": { + "type": "integer", + "title": "Measured Bytes", + "description": "The largest demand of those windows" + }, + "counted": { + "type": "boolean", + "title": "Counted", + "description": "Whether windows of this size feed the user line" + }, + "excluded_reason": { + "anyOf": [ + { + "type": "string", + "enum": ["fragmented", "partial_residency"] + }, + { + "type": "null" + } + ], + "title": "Excluded Reason", + "description": "Why a size is not counted: fragmented, its windows measured the allocator's fragmentation (`backend/working_memory/fragmentation.py`); partial_residency, the model was not fully on the device. Null for a counted size" + }, + "shipped_bytes": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Shipped Bytes", + "description": "The shipped estimate at this size, priced with the line's estimator where one of its windows measured the size, otherwise with that of the largest window; null when the platform of the environment is unknown" + }, + "user_bytes": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "User Bytes", + "description": "The estimate with the user line applied, as estimates read it (without the windows of runs still running); null without a user line" + } + }, + "type": "object", + "required": [ + "width", + "height", + "pixels", + "windows", + "measured_bytes", + "counted", + "excluded_reason", + "shipped_bytes", + "user_bytes" + ], + "title": "WorkingMemoryCalibrationLineSize" + }, + "WorkingMemoryCalibrationModelLine": { + "properties": { + "estimator": { + "type": "string", + "title": "Estimator" + }, + "version": { + "type": "integer", + "title": "Version" + }, + "line": { + "type": "string", + "title": "Line", + "description": "`_`" + }, + "carrier": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Carrier", + "description": "The estimator whose shipped entry carries the line on this platform" + }, + "env_id": { + "type": "string", + "title": "Env Id" + }, + "state": { + "type": "string", + "enum": ["in_use", "stale", "unusable", "insufficient"], + "title": "State", + "description": "in_use: the user line applies; stale: it applies, but the soft facts of the newest finished run that contributed a window differ from the current ones; unusable: hard facts differ from the current environment (or no GPU of this class generates here), so shipped values apply; insufficient: the windows of finished runs cannot fit a line yet" + }, + "changed_facts": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Changed Facts", + "description": "The soft facts that changed (stale) or the hard facts that differ (unusable)" + }, + "problem": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Problem", + "description": "Why no line could be fitted, when it could not" + }, + "measured_range": { + "anyOf": [ + { + "$ref": "#/components/schemas/WorkingMemoryCalibrationPixelRange" + }, + { + "type": "null" + } + ], + "description": "The sizes the user line was fitted on, over every model's windows of the line from finished runs" + }, + "slope_bytes_per_pixel": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Slope Bytes Per Pixel" + }, + "base_bytes": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Base Bytes" + }, + "sizes": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationLineSize" + }, + "type": "array", + "title": "Sizes" + }, + "own_windows": { + "type": "integer", + "title": "Own Windows", + "description": "The model's own windows that feed the line; 0 when only other models' do" + }, + "measured_by": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationLineContributor" + }, + "type": "array", + "title": "Measured By", + "description": "Every model whose windows feed the line, this one included when it has any; by name, then key" + } + }, + "type": "object", + "required": [ + "estimator", + "version", + "line", + "carrier", + "env_id", + "state", + "changed_facts", + "problem", + "measured_range", + "slope_bytes_per_pixel", + "base_bytes", + "sizes", + "own_windows", + "measured_by" + ], + "title": "WorkingMemoryCalibrationModelLine", + "description": "One estimator line in one environment: one the model's windows measured, or, in the current environment, one\nthe model's decode prices on that other models' windows measured." + }, + "WorkingMemoryCalibrationModelRun": { + "properties": { + "run_id": { + "type": "string", + "title": "Run Id" + }, + "status": { + "type": "string", + "enum": ["running", "completed", "failed", "partial", "canceled"], + "title": "Status" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error" + }, + "created_at": { + "type": "string", + "format": "date-time", + "title": "Created At" + }, + "point_count": { + "type": "integer", + "title": "Point Count" + }, + "attempt_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Attempt Id", + "description": "The client's attempt the run belongs to; null when none was sent" + } + }, + "type": "object", + "required": ["run_id", "status", "error", "created_at", "point_count", "attempt_id"], + "title": "WorkingMemoryCalibrationModelRun" + }, + "WorkingMemoryCalibrationModelSummary": { + "properties": { + "model_key": { + "type": "string", + "title": "Model Key" + }, + "device_class_id": { + "type": "string", + "title": "Device Class Id" + }, + "usable": { + "type": "boolean", + "title": "Usable" + }, + "problem": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Problem", + "description": "Why the store file is unusable" + }, + "current_env_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Current Env Id", + "description": "Null when no generation device of this class is present" + }, + "measurement_method": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Measurement Method", + "description": "Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present" + }, + "measurement_verified": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Measurement Verified", + "description": "Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present" + }, + "lines": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationModelLine" + }, + "type": "array", + "title": "Lines" + }, + "excluded_windows": { + "type": "integer", + "title": "Excluded Windows", + "description": "The model's windows that feed no line: invalid, not fit-eligible, fragmented, tiled, batched, upcast, a VAE outside the AutoencoderKL family, or no demand. The valid ones among them still show as sizes that are not counted" + }, + "runs": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationModelRun" + }, + "type": "array", + "title": "Runs" + } + }, + "type": "object", + "required": [ + "model_key", + "device_class_id", + "usable", + "problem", + "current_env_id", + "measurement_method", + "measurement_verified", + "lines", + "excluded_windows", + "runs" + ], + "title": "WorkingMemoryCalibrationModelSummary", + "description": "What one model's calibration runs measured on one device class, and what estimates make of it." + }, + "WorkingMemoryCalibrationPixelRange": { + "properties": { + "min_pixels": { + "type": "integer", + "title": "Min Pixels" + }, + "max_pixels": { + "type": "integer", + "title": "Max Pixels" + } + }, + "type": "object", + "required": ["min_pixels", "max_pixels"], + "title": "WorkingMemoryCalibrationPixelRange" + }, + "WorkingMemoryCalibrationPlan": { + "properties": { + "model_key": { + "type": "string", + "title": "Model Key" + }, + "supported": { + "type": "boolean", + "title": "Supported" + }, + "reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Reason", + "description": "Why the model cannot be calibrated here, when it cannot" + }, + "device_class": { + "anyOf": [ + { + "$ref": "#/components/schemas/WorkingMemoryCalibrationDeviceClassRef" + }, + { + "type": "null" + } + ], + "description": "The class of the GPU the cases run on; null without exactly one GPU generation device" + }, + "calibrated_through": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Calibrated Through", + "description": "For a standalone VAE, the installed main model the cases generate with" + }, + "estimators": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Estimators", + "description": "The working-memory estimators of the model's covered operations" + }, + "cases": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationCase" + }, + "type": "array", + "title": "Cases" + }, + "repetitions": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationRepetitions" + }, + "estimated_seconds": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Estimated Seconds", + "description": "A rough order of magnitude for all cases and repetitions, model load included; not a measurement" + }, + "shared_with": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationSharedModel" + }, + "type": "array", + "title": "Shared With", + "description": "Other installed models whose operations price with the same lines on this GPU, so a calibration changes their estimates too" + }, + "not_covered": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Not Covered", + "description": "Operations of the model a calibration does not measure, and why" + } + }, + "type": "object", + "required": [ + "model_key", + "supported", + "reason", + "device_class", + "calibrated_through", + "estimators", + "cases", + "repetitions", + "estimated_seconds", + "shared_with", + "not_covered" + ], + "title": "WorkingMemoryCalibrationPlan", + "description": "What calibrating one model on this machine's GPU runs, or why it cannot." + }, + "WorkingMemoryCalibrationPointCount": { + "properties": { + "estimator": { + "type": "string", + "title": "Estimator" + }, + "version": { + "type": "integer", + "title": "Version" + }, + "valid": { + "type": "boolean", + "title": "Valid" + }, + "fit_eligible": { + "type": "boolean", + "title": "Fit Eligible" + }, + "count": { + "type": "integer", + "title": "Count" + } + }, + "type": "object", + "required": ["estimator", "version", "valid", "fit_eligible", "count"], + "title": "WorkingMemoryCalibrationPointCount" + }, + "WorkingMemoryCalibrationRepetitions": { + "properties": { + "warm_up": { + "type": "integer", + "title": "Warm Up", + "description": "Runs of each case before the measured ones; their windows count too" + }, + "measured": { + "type": "integer", + "title": "Measured" + } + }, + "type": "object", + "required": ["warm_up", "measured"], + "title": "WorkingMemoryCalibrationRepetitions" + }, + "WorkingMemoryCalibrationRequest": { + "properties": { + "graph": { + "$ref": "#/components/schemas/Graph", + "description": "The graph to measure" + }, + "data": { + "anyOf": [ + { + "items": { + "items": { + "$ref": "#/components/schemas/BatchDatum" + }, + "type": "array" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Data", + "description": "The batch data collection" + }, + "idempotency_key": { + "anyOf": [ + { + "type": "string", + "maxLength": 255, + "minLength": 1 + }, + { + "type": "null" + } + ], + "title": "Idempotency Key", + "description": "A caller-owned retry key. Repeating the request with it returns the run it created and enqueues nothing new; once a reset removed that run, the repeat is refused. Reusing the key for another request (another model, graph or batch data) is rejected while the queue keeps the key's enqueue receipt." + }, + "attempt_id": { + "anyOf": [ + { + "type": "string", + "maxLength": 64, + "minLength": 1, + "pattern": "^[A-Za-z0-9_.:-]+$" + }, + { + "type": "null" + } + ], + "title": "Attempt Id", + "description": "A client-generated id of the calibration attempt the run belongs to (1-64 letters, digits, '_', '-', '.', ':'), so a client that enqueues one run per step can group them; no window of an attempt feeds an estimate until every run of the attempt has finished. The run id derives from the idempotency_key, the request and the attempt, so an idempotency_key names one run: repeating the request under the same attempt returns that run, and reusing the key under another attempt is refused with 409 while the queue keeps the key's receipt. Make each key unique per run of an attempt." + } + }, + "additionalProperties": false, + "type": "object", + "required": ["graph"], + "title": "WorkingMemoryCalibrationRequest", + "description": "The batch a calibration run enqueues, built by the client. The server sets origin, destination, runs and the\ngraph's id, and turns the node cache off." + }, + "WorkingMemoryCalibrationResetResult": { + "properties": { + "runs_removed": { + "type": "integer", + "title": "Runs Removed" + }, + "points_removed": { + "type": "integer", + "title": "Points Removed" + } + }, + "type": "object", + "required": ["runs_removed", "points_removed"], + "title": "WorkingMemoryCalibrationResetResult" + }, + "WorkingMemoryCalibrationRunCreated": { + "properties": { + "run_id": { + "type": "string", + "title": "Run Id" + }, + "device_class_id": { + "type": "string", + "title": "Device Class Id" + }, + "item_ids": { + "items": { + "type": "integer" + }, + "type": "array", + "title": "Item Ids", + "description": "The queue items the run enqueued. A repeat of the request that finds them unrecorded takes over every queue item carrying the run's origin, retries included" + } + }, + "type": "object", + "required": ["run_id", "device_class_id", "item_ids"], + "title": "WorkingMemoryCalibrationRunCreated" + }, + "WorkingMemoryCalibrationRunSummary": { + "properties": { + "run_id": { + "type": "string", + "title": "Run Id" + }, + "model_key": { + "type": "string", + "title": "Model Key" + }, + "model_hash": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Model Hash" + }, + "user_id": { + "type": "string", + "title": "User Id" + }, + "created_at": { + "type": "string", + "format": "date-time", + "title": "Created At" + }, + "env_id": { + "type": "string", + "title": "Env Id" + }, + "status": { + "type": "string", + "enum": ["running", "completed", "failed", "partial", "canceled"], + "title": "Status", + "description": "Derived from the run's queue items, a retry standing in for the item it retried: running while any is pending or running; failed when any failed; completed when all completed; otherwise partial when valid points were recorded, else canceled" + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Error", + "description": "The error of a failed item, when the run failed" + }, + "item_ids": { + "items": { + "type": "integer" + }, + "type": "array", + "title": "Item Ids", + "description": "Every queue item of the run, retries included" + }, + "point_count": { + "type": "integer", + "title": "Point Count" + }, + "attempt_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Attempt Id", + "description": "The client's attempt the run belongs to; null when none was sent" + } + }, + "type": "object", + "required": [ + "run_id", + "model_key", + "model_hash", + "user_id", + "created_at", + "env_id", + "status", + "error", + "item_ids", + "point_count", + "attempt_id" + ], + "title": "WorkingMemoryCalibrationRunSummary" + }, + "WorkingMemoryCalibrationSharedModel": { + "properties": { + "key": { + "type": "string", + "title": "Key" + }, + "name": { + "type": "string", + "title": "Name" + }, + "base": { + "type": "string", + "title": "Base" + }, + "type": { + "type": "string", + "title": "Type" + }, + "lines": { + "items": { + "type": "string" + }, + "type": "array", + "title": "Lines", + "description": "The lines it shares with the planned model, `_`" + } + }, + "type": "object", + "required": ["key", "name", "base", "type", "lines"], + "title": "WorkingMemoryCalibrationSharedModel" + }, + "WorkingMemoryCalibrationStatus": { + "properties": { + "available": { + "type": "boolean", + "title": "Available", + "description": "Whether a calibration run can start now: one GPU generation device whose store file is usable" + }, + "unavailable_reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Unavailable Reason" + }, + "current_device_class_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Current Device Class Id" + }, + "current_env_id": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Current Env Id" + }, + "device_classes": { + "items": { + "$ref": "#/components/schemas/WorkingMemoryCalibrationDeviceClass" + }, + "type": "array", + "title": "Device Classes" + } + }, + "type": "object", + "required": ["available", "unavailable_reason", "current_device_class_id", "current_env_id", "device_classes"], + "title": "WorkingMemoryCalibrationStatus" + }, "ZImageConditioningField": { "description": "A Z-Image conditioning tensor primitive value", "properties": { diff --git a/invokeai/frontend/api/schema.ts b/invokeai/frontend/api/schema.ts index a845bdf0a0f..e93dd8b2f77 100644 --- a/invokeai/frontend/api/schema.ts +++ b/invokeai/frontend/api/schema.ts @@ -1195,6 +1195,184 @@ export type paths = { patch?: never; trace?: never; }; + "/api/v2/models/i/{key}/working_memory/calibrate": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Calibrate Model Working Memory + * @description Start a working-memory calibration run: enqueue the batch with every node uncached and measure each node's + * model operations on the generation device. Needs exactly one generation device, and it must be a GPU. The queue + * takes all of the batch's items or none. + * + * Once the items are queued, a request without an `idempotency_key` succeeds even when the store fails for a passing + * reason (an unwritable root, a lock timeout) to record them: they run all the same, and the status finds them by + * their origin. With a key, that failure is a 503 although the items are queued. A run removed meanwhile (404) or an + * unusable store file (409) fails the request with or without a key; its queued items then fail when they start. + * + * With a key, repeating the request (also after a lost response) records the queued items and returns the run; it + * enqueues nothing new. The items come from the queue's enqueue receipt for the key while it exists, otherwise from + * the queue items that carry the run's origin and those the runner saw. Once a reset removed the run, a repeat is + * refused (404). Acknowledge the key through the queue's enqueue_batch/acknowledge route once the response + * arrived. + */ + post: operations["calibrate_model_working_memory"]; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory/status": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get Working Memory Calibration Status + * @description Every device class's calibration runs, with their derived status, point counts and environments. + */ + get: operations["get_working_memory_calibration_status"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/i/{key}/working_memory/plan": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get Model Working Memory Calibration Plan + * @description The text-to-image cases that calibrate the model's decode on this machine's GPU, how often each runs, which + * other installed models the calibration also moves, or why the model cannot be calibrated here (`supported`). + */ + get: operations["get_model_working_memory_calibration_plan"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/i/{key}/working_memory/summary": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Get Model Working Memory Calibration Summary + * @description What the calibration windows of the model's lines measured on one device class, per estimator line and + * environment: the sizes, the largest demand, the shipped and the user estimate, whether the user line is in use, + * stale or unusable, and which models measured it, also for a line the model's decode shares with other models' + * measurements; and the model's runs. + */ + get: operations["get_model_working_memory_calibration_summary"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory/runs/{run_id}": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Delete Working Memory Calibration Run + * @description Remove a calibration run and its measurements. Its queue items that have not run yet fail when they start. + */ + delete: operations["delete_working_memory_calibration_run"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/i/{key}/working_memory": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Reset Model Working Memory Calibration + * @description Remove a model's calibration runs with all their measurements, and every measurement of the model taken in + * other runs (a VAE measured inside a transformer's run, say), from one device class. + */ + delete: operations["reset_model_working_memory_calibration"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Reset Working Memory Calibration + * @description Remove every calibration run and measurement of one device class. + */ + delete: operations["reset_working_memory_calibration"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v2/models/working_memory/export": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Export Working Memory Calibration + * @description The device class's calibration store file as it is on disk, readable by this version or not. + */ + get: operations["export_working_memory_calibration"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/image_moves/start": { parameters: { query?: never; @@ -7975,6 +8153,30 @@ export type components = { */ type: "calculate_image_tiles_output"; }; + /** + * CalibrationDeviceClass + * @description A `fingerprint.DeviceClass` with its id as a plain field, as it is stored and returned. + */ + CalibrationDeviceClass: { + /** + * Id + * @description A short stable hash of the other fields + */ + id: string; + /** Vendor */ + vendor: string; + /** Name */ + name: string; + /** Total Mib */ + total_mib: number; + /** + * Backend + * @enum {string} + */ + backend: "cuda" | "hip" | "mps" | "xpu"; + /** Arch */ + arch: string | null; + }; /** * Call Saved Workflow * @description Displays and later executes against a selected saved workflow. @@ -52635,6 +52837,501 @@ export type components = { [key: string]: components["schemas"]["JsonValue"]; } | null; }; + /** + * WorkingMemoryCalibrationCase + * @description One generation a calibration plan runs, in the fields of the client's graph-export case (mode txt2img). + */ + WorkingMemoryCalibrationCase: { + /** + * Id + * @description Unique within the plan; becomes the graph id + */ + id: string; + /** + * Mode + * @default txt2img + * @constant + */ + mode?: "txt2img"; + /** + * Model Key + * @description The main model the case generates with + */ + model_key: string; + /** + * Components + * @description Component picks by Generate slot key; `vae` when a standalone VAE is calibrated or a GPU VAE must be picked over one that runs on the CPU + */ + components: { + [key: string]: string; + }; + /** Width */ + width: number; + /** Height */ + height: number; + /** + * Steps + * @description The fewest steps that still run the whole pipeline + */ + steps: number; + /** Seed */ + seed: number; + /** Positive Prompt */ + positive_prompt: string; + /** Negative Prompt */ + negative_prompt: string; + /** + * Settings + * @description Generate settings overrides: `vaePrecision` (fp16 or fp32) where the decode node casts the VAE + */ + settings: { + [key: string]: string; + }; + /** + * Estimator + * @description The working-memory estimator of the operation the case measures + */ + estimator: string; + /** + * Line + * @description The estimator line the case measures, `_` + */ + line: string; + }; + /** WorkingMemoryCalibrationDeviceClass */ + WorkingMemoryCalibrationDeviceClass: { + /** Device Class Id */ + device_class_id: string; + /** @description Null when the file is unusable */ + device_class: components["schemas"]["CalibrationDeviceClass"] | null; + /** Usable */ + usable: boolean; + /** + * Problem + * @description Why the file is unusable. It is kept as it is and never written + */ + problem: string | null; + /** Is Current */ + is_current: boolean; + /** + * Measurement Method + * @description Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present + */ + measurement_method: string | null; + /** + * Measurement Verified + * @description Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present + */ + measurement_verified: boolean | null; + /** Runs */ + runs: components["schemas"]["WorkingMemoryCalibrationRunSummary"][]; + /** Point Counts */ + point_counts: components["schemas"]["WorkingMemoryCalibrationPointCount"][]; + /** Envs */ + envs: components["schemas"]["WorkingMemoryCalibrationEnvironment"][]; + }; + /** WorkingMemoryCalibrationDeviceClassRef */ + WorkingMemoryCalibrationDeviceClassRef: { + /** Id */ + id: string; + /** Name */ + name: string; + }; + /** WorkingMemoryCalibrationEnvironment */ + WorkingMemoryCalibrationEnvironment: { + /** Id */ + id: string; + /** Hard */ + hard: { + [key: string]: string; + }; + /** Soft */ + soft: { + [key: string]: string | null; + }; + /** + * Hard Matches Current + * @description Whether the current environment has the same hard facts; null off the current device class + */ + hard_matches_current: boolean | null; + /** + * Soft Matches Current + * @description Whether the current environment has the soft facts first recorded with this environment (`soft`); a later run's own soft facts are not compared here. Null off the current device class + */ + soft_matches_current: boolean | null; + /** Point Count */ + point_count: number; + }; + /** + * WorkingMemoryCalibrationLineContributor + * @description A model whose windows feed a line: the model a run calibrated, or the model a window measured. + */ + WorkingMemoryCalibrationLineContributor: { + /** Model Key */ + model_key: string; + /** + * Name + * @description The model's name; null when it is no longer installed + */ + name: string | null; + /** + * Windows + * @description Its counted windows on the line: valid, fit-eligible and not fragmented + */ + windows: number; + }; + /** WorkingMemoryCalibrationLineSize */ + WorkingMemoryCalibrationLineSize: { + /** Width */ + width: number; + /** Height */ + height: number; + /** Pixels */ + pixels: number; + /** + * Windows + * @description Counted: the windows at this size that feed the line, valid, fit-eligible and not fragmented, of every model in the environment; those of a calibration run still running are counted, and join the fit once it finished. Not counted: the valid windows measured at this size + */ + windows: number; + /** + * Measured Bytes + * @description The largest demand of those windows + */ + measured_bytes: number; + /** + * Counted + * @description Whether windows of this size feed the user line + */ + counted: boolean; + /** + * Excluded Reason + * @description Why a size is not counted: fragmented, its windows measured the allocator's fragmentation (`backend/working_memory/fragmentation.py`); partial_residency, the model was not fully on the device. Null for a counted size + */ + excluded_reason: ("fragmented" | "partial_residency") | null; + /** + * Shipped Bytes + * @description The shipped estimate at this size, priced with the line's estimator where one of its windows measured the size, otherwise with that of the largest window; null when the platform of the environment is unknown + */ + shipped_bytes: number | null; + /** + * User Bytes + * @description The estimate with the user line applied, as estimates read it (without the windows of runs still running); null without a user line + */ + user_bytes: number | null; + }; + /** + * WorkingMemoryCalibrationModelLine + * @description One estimator line in one environment: one the model's windows measured, or, in the current environment, one + * the model's decode prices on that other models' windows measured. + */ + WorkingMemoryCalibrationModelLine: { + /** Estimator */ + estimator: string; + /** Version */ + version: number; + /** + * Line + * @description `_` + */ + line: string; + /** + * Carrier + * @description The estimator whose shipped entry carries the line on this platform + */ + carrier: string | null; + /** Env Id */ + env_id: string; + /** + * State + * @description in_use: the user line applies; stale: it applies, but the soft facts of the newest finished run that contributed a window differ from the current ones; unusable: hard facts differ from the current environment (or no GPU of this class generates here), so shipped values apply; insufficient: the windows of finished runs cannot fit a line yet + * @enum {string} + */ + state: "in_use" | "stale" | "unusable" | "insufficient"; + /** + * Changed Facts + * @description The soft facts that changed (stale) or the hard facts that differ (unusable) + */ + changed_facts: string[]; + /** + * Problem + * @description Why no line could be fitted, when it could not + */ + problem: string | null; + /** @description The sizes the user line was fitted on, over every model's windows of the line from finished runs */ + measured_range: components["schemas"]["WorkingMemoryCalibrationPixelRange"] | null; + /** Slope Bytes Per Pixel */ + slope_bytes_per_pixel: number | null; + /** Base Bytes */ + base_bytes: number | null; + /** Sizes */ + sizes: components["schemas"]["WorkingMemoryCalibrationLineSize"][]; + /** + * Own Windows + * @description The model's own windows that feed the line; 0 when only other models' do + */ + own_windows: number; + /** + * Measured By + * @description Every model whose windows feed the line, this one included when it has any; by name, then key + */ + measured_by: components["schemas"]["WorkingMemoryCalibrationLineContributor"][]; + }; + /** WorkingMemoryCalibrationModelRun */ + WorkingMemoryCalibrationModelRun: { + /** Run Id */ + run_id: string; + /** + * Status + * @enum {string} + */ + status: "running" | "completed" | "failed" | "partial" | "canceled"; + /** Error */ + error: string | null; + /** + * Created At + * Format: date-time + */ + created_at: string; + /** Point Count */ + point_count: number; + /** + * Attempt Id + * @description The client's attempt the run belongs to; null when none was sent + */ + attempt_id: string | null; + }; + /** + * WorkingMemoryCalibrationModelSummary + * @description What one model's calibration runs measured on one device class, and what estimates make of it. + */ + WorkingMemoryCalibrationModelSummary: { + /** Model Key */ + model_key: string; + /** Device Class Id */ + device_class_id: string; + /** Usable */ + usable: boolean; + /** + * Problem + * @description Why the store file is unusable + */ + problem: string | null; + /** + * Current Env Id + * @description Null when no generation device of this class is present + */ + current_env_id: string | null; + /** + * Measurement Method + * @description Where a window on a GPU of this class here reads its driver figure (`counters.Method`): torch-only when this platform has no driver reader; null when no generation device of this class is present + */ + measurement_method: string | null; + /** + * Measurement Verified + * @description Whether that method was checked against real hardware on this backend (`counters.VERIFIED_METHODS`); null when no generation device of this class is present + */ + measurement_verified: boolean | null; + /** Lines */ + lines: components["schemas"]["WorkingMemoryCalibrationModelLine"][]; + /** + * Excluded Windows + * @description The model's windows that feed no line: invalid, not fit-eligible, fragmented, tiled, batched, upcast, a VAE outside the AutoencoderKL family, or no demand. The valid ones among them still show as sizes that are not counted + */ + excluded_windows: number; + /** Runs */ + runs: components["schemas"]["WorkingMemoryCalibrationModelRun"][]; + }; + /** WorkingMemoryCalibrationPixelRange */ + WorkingMemoryCalibrationPixelRange: { + /** Min Pixels */ + min_pixels: number; + /** Max Pixels */ + max_pixels: number; + }; + /** + * WorkingMemoryCalibrationPlan + * @description What calibrating one model on this machine's GPU runs, or why it cannot. + */ + WorkingMemoryCalibrationPlan: { + /** Model Key */ + model_key: string; + /** Supported */ + supported: boolean; + /** + * Reason + * @description Why the model cannot be calibrated here, when it cannot + */ + reason: string | null; + /** @description The class of the GPU the cases run on; null without exactly one GPU generation device */ + device_class: components["schemas"]["WorkingMemoryCalibrationDeviceClassRef"] | null; + /** + * Calibrated Through + * @description For a standalone VAE, the installed main model the cases generate with + */ + calibrated_through: string | null; + /** + * Estimators + * @description The working-memory estimators of the model's covered operations + */ + estimators: string[]; + /** Cases */ + cases: components["schemas"]["WorkingMemoryCalibrationCase"][]; + repetitions: components["schemas"]["WorkingMemoryCalibrationRepetitions"]; + /** + * Estimated Seconds + * @description A rough order of magnitude for all cases and repetitions, model load included; not a measurement + */ + estimated_seconds: number | null; + /** + * Shared With + * @description Other installed models whose operations price with the same lines on this GPU, so a calibration changes their estimates too + */ + shared_with: components["schemas"]["WorkingMemoryCalibrationSharedModel"][]; + /** + * Not Covered + * @description Operations of the model a calibration does not measure, and why + */ + not_covered: string[]; + }; + /** WorkingMemoryCalibrationPointCount */ + WorkingMemoryCalibrationPointCount: { + /** Estimator */ + estimator: string; + /** Version */ + version: number; + /** Valid */ + valid: boolean; + /** Fit Eligible */ + fit_eligible: boolean; + /** Count */ + count: number; + }; + /** WorkingMemoryCalibrationRepetitions */ + WorkingMemoryCalibrationRepetitions: { + /** + * Warm Up + * @description Runs of each case before the measured ones; their windows count too + */ + warm_up: number; + /** Measured */ + measured: number; + }; + /** + * WorkingMemoryCalibrationRequest + * @description The batch a calibration run enqueues, built by the client. The server sets origin, destination, runs and the + * graph's id, and turns the node cache off. + */ + WorkingMemoryCalibrationRequest: { + /** @description The graph to measure */ + graph: components["schemas"]["Graph"]; + /** + * Data + * @description The batch data collection + */ + data?: components["schemas"]["BatchDatum"][][] | null; + /** + * Idempotency Key + * @description A caller-owned retry key. Repeating the request with it returns the run it created and enqueues nothing new; once a reset removed that run, the repeat is refused. Reusing the key for another request (another model, graph or batch data) is rejected while the queue keeps the key's enqueue receipt. + */ + idempotency_key?: string | null; + /** + * Attempt Id + * @description A client-generated id of the calibration attempt the run belongs to (1-64 letters, digits, '_', '-', '.', ':'), so a client that enqueues one run per step can group them; no window of an attempt feeds an estimate until every run of the attempt has finished. The run id derives from the idempotency_key, the request and the attempt, so an idempotency_key names one run: repeating the request under the same attempt returns that run, and reusing the key under another attempt is refused with 409 while the queue keeps the key's receipt. Make each key unique per run of an attempt. + */ + attempt_id?: string | null; + }; + /** WorkingMemoryCalibrationResetResult */ + WorkingMemoryCalibrationResetResult: { + /** Runs Removed */ + runs_removed: number; + /** Points Removed */ + points_removed: number; + }; + /** WorkingMemoryCalibrationRunCreated */ + WorkingMemoryCalibrationRunCreated: { + /** Run Id */ + run_id: string; + /** Device Class Id */ + device_class_id: string; + /** + * Item Ids + * @description The queue items the run enqueued. A repeat of the request that finds them unrecorded takes over every queue item carrying the run's origin, retries included + */ + item_ids: number[]; + }; + /** WorkingMemoryCalibrationRunSummary */ + WorkingMemoryCalibrationRunSummary: { + /** Run Id */ + run_id: string; + /** Model Key */ + model_key: string; + /** Model Hash */ + model_hash: string | null; + /** User Id */ + user_id: string; + /** + * Created At + * Format: date-time + */ + created_at: string; + /** Env Id */ + env_id: string; + /** + * Status + * @description Derived from the run's queue items, a retry standing in for the item it retried: running while any is pending or running; failed when any failed; completed when all completed; otherwise partial when valid points were recorded, else canceled + * @enum {string} + */ + status: "running" | "completed" | "failed" | "partial" | "canceled"; + /** + * Error + * @description The error of a failed item, when the run failed + */ + error: string | null; + /** + * Item Ids + * @description Every queue item of the run, retries included + */ + item_ids: number[]; + /** Point Count */ + point_count: number; + /** + * Attempt Id + * @description The client's attempt the run belongs to; null when none was sent + */ + attempt_id: string | null; + }; + /** WorkingMemoryCalibrationSharedModel */ + WorkingMemoryCalibrationSharedModel: { + /** Key */ + key: string; + /** Name */ + name: string; + /** Base */ + base: string; + /** Type */ + type: string; + /** + * Lines + * @description The lines it shares with the planned model, `_` + */ + lines: string[]; + }; + /** WorkingMemoryCalibrationStatus */ + WorkingMemoryCalibrationStatus: { + /** + * Available + * @description Whether a calibration run can start now: one GPU generation device whose store file is usable + */ + available: boolean; + /** Unavailable Reason */ + unavailable_reason: string | null; + /** Current Device Class Id */ + current_device_class_id: string | null; + /** Current Env Id */ + current_env_id: string | null; + /** Device Classes */ + device_classes: components["schemas"]["WorkingMemoryCalibrationDeviceClass"][]; + }; /** * ZImageConditioningField * @description A Z-Image conditioning tensor primitive value @@ -55795,6 +56492,391 @@ export interface operations { }; }; }; + calibrate_model_working_memory: { + parameters: { + query?: never; + header?: never; + path: { + /** @description Key of the model the run calibrates */ + key: string; + }; + cookie?: never; + }; + requestBody: { + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationRequest"]; + }; + }; + responses: { + /** @description Successful Response */ + 201: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationRunCreated"]; + }; + }; + /** @description The model could not be found; the run was removed while it was being created, whose items are queued then and fail when they start; or a reset removed the run the idempotency key started */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description This machine cannot calibrate now (no GPU, or several generation devices), the device class's store file is unusable and kept as it is (possibly after the items were queued, which then fail when they start), the queue has no room for all of the run's items (it then takes none), or the idempotency key was used for another request */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The graph cannot be measured as a calibration run */ + 422: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Too many unacknowledged idempotency keys; acknowledge or wait, then retry */ + 429: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout); with an idempotency key, possibly after the items were queued */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + get_working_memory_calibration_status: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationStatus"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + get_model_working_memory_calibration_plan: { + parameters: { + query?: never; + header?: never; + path: { + /** @description Key of the model to calibrate */ + key: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationPlan"]; + }; + }; + /** @description The model could not be found */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; + get_model_working_memory_calibration_summary: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path: { + /** @description Key of the model whose measurements to summarise */ + key: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationModelSummary"]; + }; + }; + /** @description The model could not be found */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description No device_class_id was given and there is no current one */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + delete_working_memory_calibration_run: { + parameters: { + query?: never; + header?: never; + path: { + /** @description The calibration run to remove */ + run_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationResetResult"]; + }; + }; + /** @description No usable store file holds the run */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The run's store file became unusable after the run was found; it is kept as it is */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + reset_model_working_memory_calibration: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path: { + /** @description Key of the model whose runs and measurements to remove */ + key: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationResetResult"]; + }; + }; + /** @description No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + reset_working_memory_calibration: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkingMemoryCalibrationResetResult"]; + }; + }; + /** @description No device_class_id was given and there is no current one (no GPU, or several generation devices), or the device class's store file is unusable and kept as it is */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + export_working_memory_calibration: { + parameters: { + query?: { + /** @description The device class to act on; the current generation device's by default */ + device_class_id?: string | null; + }; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description The device class's store file byte for byte: a CalibrationStoreFile (schema_version 1) when this version can read it, otherwise whatever the file holds */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": unknown; + }; + }; + /** @description The device class has no store file */ + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description No device_class_id was given and there is no current one */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + /** @description The calibration store cannot be read or written (unwritable root, lock timeout) */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; start_image_move: { parameters: { query?: never; @@ -59489,6 +60571,13 @@ export interface operations { "application/json": components["schemas"]["EnqueueBatchResult"]; }; }; + /** @description The batch's origin starts with 'invokeai:wm-calibration:' (in any case), which is reserved for working-memory calibration runs */ + 400: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; /** @description Validation Error */ 422: { headers: { diff --git a/invokeai/frontend/webv2/ARCHITECTURE.md b/invokeai/frontend/webv2/ARCHITECTURE.md index d72f3f3a710..6caa7acb868 100644 --- a/invokeai/frontend/webv2/ARCHITECTURE.md +++ b/invokeai/frontend/webv2/ARCHITECTURE.md @@ -104,7 +104,7 @@ Feature interfaces stay ordinary module interfaces unless two adapters add lever | Nodes | Custom-node catalog/templates and execution lifecycle; Workflow editor and node manager | Platform; descriptors, catalog commands, execution snapshots and manager contribution | | Queue | Backend queue state, realtime interpretation, submission/reconciliation and Queue UI; App runtime, Workbench aggregate/shell, Generation/Gallery/Workflow | Platform and public envelopes/capabilities from Identity, Models, Generation, Workflow, Upscale; backend port, queue reads/commands and widget contribution | | Gallery | Image collection, selection, board policy and Gallery UI; App adapter, Generation, Workflow, Upscale, Models, Workbench image actions | Platform and Queue's public progress reads; gallery reads/commands and widget contribution | -| Generation | Generation settings, validation, graph compilation and Generate UI; Workbench invocation, Queue, Gallery, Workflow, Upscale | Platform and public Models/Gallery/Queue interfaces; settings/validation, compiled submission envelope and widget contribution | +| Generation | Generation settings, validation, graph compilation and Generate UI; Workbench invocation, Queue, Gallery, Workflow, Upscale | Platform and public Models/Gallery/Queue interfaces; settings/validation, compiled submission envelope, offline graph export and widget contribution | | Workflow | Workflow document, validation, compilation, preview/editor/library and explicit publication; App adapter, Workbench invocation, Nodes | Platform and public Nodes/Models/Gallery/Queue/Generation interfaces; document reads/commands, compiled submission envelope and widget contribution | | Upscale | Upscale settings, validation, compilation and UI; Workbench invocation and Queue | Platform and public Models/Gallery/Queue interfaces; value/validation reads, submission command and widget contribution | | Intermediates | Per-project intermediates read models, cleanup previews/operations and the Settings section that manages them; Workbench settings, project menus, account administration | Platform plus Identity's public session/capability reads; query keys, read models, the focus intent, the browser hold lease and the `settingsContribution` metadata whose loader owns the section body | diff --git a/invokeai/frontend/webv2/package.json b/invokeai/frontend/webv2/package.json index 8cc729ef6b6..069b1cc3701 100644 --- a/invokeai/frontend/webv2/package.json +++ b/invokeai/frontend/webv2/package.json @@ -17,6 +17,7 @@ "fix": "pnpm run lint:oxc:fix && pnpm run format:write", "format:check": "oxfmt --check .", "format:write": "oxfmt --write .", + "graphs:export": "node scripts/export-graphs.mjs", "i18n:report": "node scripts/i18n-report.mjs", "i18n:report:verbose": "node scripts/i18n-report.mjs --verbose", "lint": "pnpm run format:check && pnpm run lint:oxc && pnpm run lint:tsc && pnpm run architecture:check", @@ -32,7 +33,7 @@ "test:all": "pnpm run test && pnpm run test:browser && pnpm run test:fixtures", "test:browser": "vitest run --config vitest.browser.config.mts", "test:browser:update-goldens": "CANVAS_GOLDEN_UPDATE=1 vitest run --config vitest.browser.config.mts src/workbench/canvas-engine/render/golden", - "test:fixtures": "node --test scripts/mock-backend-fixtures.test.mjs scripts/preview-server.test.mjs scripts/run-project-file-journey.test.mjs scripts/locale-assets-plugin.test.mjs", + "test:fixtures": "node --test scripts/mock-backend-fixtures.test.mjs scripts/preview-server.test.mjs scripts/run-project-file-journey.test.mjs scripts/locale-assets-plugin.test.mjs scripts/export-graphs.test.mjs", "test:performance:architecture": "pnpm run test:performance:build && pnpm run test:performance:browser", "test:performance:browser": "node scripts/measure-architecture-performance.mjs", "test:performance:browser:update-baseline": "node scripts/measure-architecture-performance.mjs --update-baseline", diff --git a/invokeai/frontend/webv2/performance/browser-baseline.json b/invokeai/frontend/webv2/performance/browser-baseline.json index 002ee7bca76..88f75b73e52 100644 --- a/invokeai/frontend/webv2/performance/browser-baseline.json +++ b/invokeai/frontend/webv2/performance/browser-baseline.json @@ -30,7 +30,7 @@ "fontRawBytes": 23664, "imageRawBytes": 0, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 29, "scriptRawBytes": 1868300, "scriptRequestCount": 25, @@ -68,7 +68,7 @@ "fontRawBytes": 23664, "imageRawBytes": 0, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 29, "scriptRawBytes": 1868300, "scriptRequestCount": 25, @@ -106,7 +106,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, @@ -144,7 +144,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, @@ -182,7 +182,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 96, "scriptRawBytes": 4718489, "scriptRequestCount": 91, @@ -220,7 +220,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 96, "scriptRawBytes": 4718489, "scriptRequestCount": 91, @@ -258,7 +258,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 94, "scriptRawBytes": 4644510, "scriptRequestCount": 88, @@ -296,7 +296,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 94, "scriptRawBytes": 4644510, "scriptRequestCount": 88, @@ -334,7 +334,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, @@ -372,7 +372,7 @@ "fontRawBytes": 23664, "imageRawBytes": 44398, "largestAssetRawBytes": 752413, - "otherRawBytes": 220849, + "otherRawBytes": 224640, "requestCount": 87, "scriptRawBytes": 4199867, "scriptRequestCount": 82, diff --git a/invokeai/frontend/webv2/public/locales/en.json b/invokeai/frontend/webv2/public/locales/en.json index d09d2247f9d..154188e6349 100644 --- a/invokeai/frontend/webv2/public/locales/en.json +++ b/invokeai/frontend/webv2/public/locales/en.json @@ -1616,7 +1616,86 @@ "variant": "Variant", "variantHelp": "Architecture variant, e.g. inpaint or dev.", "viewModel": "View model", - "viewQueue": "View queue" + "viewQueue": "View queue", + "workingMemory": { + "title": "Working memory", + "description": "Measures this model's VAE decode on this GPU, so Invoke reserves what it really needs.", + "loadFailed": "Could not load working memory", + "stateInUse": "Calibrated", + "stateStale": "May be outdated", + "stateUnusable": "Not used", + "stateInsufficient": "Too few sizes", + "stateNone": "Not calibrated", + "operationDecode": "Decode", + "operationEncode": "Encode", + "pair": "{{label}} · {{detail}}", + "size": "{{width}} × {{height}}", + "measuredRange": "Fitted on {{min}}–{{max}} MP", + "tableLabel": "{{line}} measurements", + "columnSize": "Size", + "columnMeasured": "Measured", + "columnShipped": "Built-in", + "columnCalibrated": "Calibrated estimate", + "aboutColumn": "About {{column}}", + "hintMeasured": "The most VRAM the decode took at this size on this GPU, beyond the model weights.", + "hintShipped": "What Invoke reserves without calibration: values measured on reference GPUs, plus 5%.", + "hintCalibrated": "What Invoke reserves after calibration: a line fitted over your counted measurements, plus 5%. Beyond the largest measured size, the larger of this and Built-in applies.", + "notCounted": "Not counted", + "notCountedFragmented": "Not counted: allocator fragmentation", + "notCountedPartial": "Not counted: model not fully on GPU", + "helpInUse": "Estimates use these measurements.", + "helpStale": "Still used, but {{facts}} changed since.", + "helpUnusable": "Built-in estimates apply: {{facts}} differs.", + "helpUnusableNoFacts": "Built-in estimates apply: measured on a GPU not present here.", + "helpInsufficient": "Built-in estimates apply until more sizes are measured.", + "sharedWith_one": "Also changes {{count}} other model", + "sharedWith_other": "Also changes {{count}} other models", + "notVerified": "Measurement method not verified on this hardware", + "neverCalibrated": "Not calibrated on this GPU. Built-in estimates apply.", + "usesOthers": "Uses measurements from other models on this GPU.", + "measuredBy": "Measured on: {{models}}", + "removedModel": "Removed model", + "modelRemoved": "This model is no longer installed.", + "encodeNotCovered": "Image-to-image and inpainting encodes keep the built-in estimates.", + "floorNote": "Only matters above the working-memory floor. It can change which large decodes tile, and tiled output is not pixel-identical.", + "calibrate": "Calibrate", + "recalibrate": "Recalibrate", + "stop": "Stop", + "blockedUnsupported": "This model cannot be calibrated.", + "blockedUnavailable": "Not available on this machine.", + "blockedStoreUnusable": "Calibration file unusable: {{problem}}", + "blockedRunning": "A calibration is queued or running.", + "queuing": "Queuing: {{done}}/{{total}}", + "running": "Calibrating: {{done}}/{{total}} runs", + "progressLabel": "Calibration progress", + "canceled": "Calibration stopped. Measurements so far are kept.", + "failed": "Calibration failed", + "partial": "Calibration incomplete", + "partialQueued": "Only {{queued}} of {{total}} runs were queued.", + "partialCanceled": "Some runs were canceled.", + "stopTitle": "Stop calibration?", + "stopBody": "Queued runs are canceled. Measurements so far are kept.", + "stopFailed": "Could not stop calibration", + "resetTitle": "Reset calibration?", + "resetBody": "Removes this model's measurements on this GPU. Other models' measurements on the same lines still apply.", + "resetDone": "Calibration reset", + "resetFailed": "Could not reset calibration", + "dialogTitle": "Calibrate working memory", + "dialogDescription": "Runs short generations on {{device}}.", + "dialogDuration": "About {{minutes}} min.", + "dialogQueueNote": "They wait in the queue behind current work.", + "dialogThrough": "Generates with {{model}}.", + "dialogRuns": "Runs: {{count}}", + "dialogRepetitions": "Per size: {{warmUp}} warm-up, {{measured}} measured.", + "facts": { + "allocator": "memory allocator", + "driver": "GPU driver", + "invokeai": "Invoke version", + "osBuild": "OS build", + "sdpaGuard": "ROCm attention guard", + "torchBuild": "PyTorch build" + } + } }, "nodes": { "addNodes": "Add Nodes", diff --git a/invokeai/frontend/webv2/scripts/export-graphs.mjs b/invokeai/frontend/webv2/scripts/export-graphs.mjs new file mode 100644 index 00000000000..cc44c137198 --- /dev/null +++ b/invokeai/frontend/webv2/scripts/export-graphs.mjs @@ -0,0 +1,80 @@ +/** + * `pnpm graphs:export ` compiles offline export cases into the enqueue batches the UI + * would send (src/app/graphExport.ts). Vite's module runner loads the TypeScript sources with the app's aliases and + * opens no port. Exit 0 once the response is written, with per-case errors inside it; exit 2 for an unreadable or + * malformed request, with no response written. + */ + +import { readFile, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import process from 'node:process'; +import { fileURLToPath } from 'node:url'; +import { loadConfigFromFile, runnerImport } from 'vite'; + +const PACKAGE_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const EXIT_MALFORMED_REQUEST = 2; + +const rejectRequest = (message) => { + process.stderr.write(`graphs:export: ${message}\n`); + process.exitCode = EXIT_MALFORMED_REQUEST; +}; + +const readRequest = async (path) => { + try { + return JSON.parse(await readFile(path, 'utf8')); + } catch (error) { + rejectRequest(`cannot read ${path}: ${error instanceof Error ? error.message : String(error)}`); + return undefined; + } +}; + +/** Only the aliases and defines: plugins transform the browser bundle, not these pure modules. */ +const loadExporter = async () => { + const { config } = await loadConfigFromFile( + { command: 'serve', mode: 'development' }, + resolve(PACKAGE_ROOT, 'vite.config.mts'), + PACKAGE_ROOT, + 'error' + ); + const { module } = await runnerImport(resolve(PACKAGE_ROOT, 'src/app/graphExport.ts'), { + define: config.define, + logLevel: 'error', + resolve: { alias: config.resolve.alias }, + root: PACKAGE_ROOT, + }); + + return module; +}; + +const main = async () => { + const [requestPath, responsePath, ...extra] = process.argv.slice(2); + + if (!requestPath || !responsePath || extra.length > 0) { + rejectRequest('usage: pnpm graphs:export '); + return; + } + + const request = await readRequest(requestPath); + + if (request === undefined) { + return; + } + + const { exportGraphs, GraphExportRequestError } = await loadExporter(); + let response; + + try { + response = exportGraphs(request); + } catch (error) { + if (error instanceof GraphExportRequestError) { + rejectRequest(`malformed request: ${error.message}`); + return; + } + + throw error; + } + + await writeFile(responsePath, `${JSON.stringify(response, null, 2)}\n`); +}; + +await main(); diff --git a/invokeai/frontend/webv2/scripts/export-graphs.test.mjs b/invokeai/frontend/webv2/scripts/export-graphs.test.mjs new file mode 100644 index 00000000000..607daf3856b --- /dev/null +++ b/invokeai/frontend/webv2/scripts/export-graphs.test.mjs @@ -0,0 +1,74 @@ +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { access, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, resolve } from 'node:path'; +import test from 'node:test'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; + +const PACKAGE_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const CLI = resolve(PACKAGE_ROOT, 'scripts/export-graphs.mjs'); +const CAPABILITIES = resolve(PACKAGE_ROOT, 'src/features/generation/core/__fixtures__/architectureCapabilities.json'); + +const runCli = async (context, requestText) => { + const directory = await mkdtemp(resolve(tmpdir(), 'invokeai-graphs-export-')); + context.after(() => rm(directory, { force: true, recursive: true })); + const requestPath = resolve(directory, 'request.json'); + const responsePath = resolve(directory, 'response.json'); + await writeFile(requestPath, requestText); + + const { code, stderr } = await promisify(execFile)(process.execPath, [CLI, requestPath, responsePath], { + cwd: PACKAGE_ROOT, + }).then( + ({ stderr }) => ({ code: 0, stderr }), + (error) => ({ code: error.code, stderr: error.stderr }) + ); + const response = await access(responsePath).then( + async () => JSON.parse(await readFile(responsePath, 'utf8')), + () => null + ); + + return { code, response, stderr }; +}; + +test('writes per-case results through the real module graph and exits 0', async (context) => { + const capabilities = JSON.parse(await readFile(CAPABILITIES, 'utf8')); + const sdxl = { base: 'sdxl', format: 'diffusers', hash: 'blake3:sdxl', key: 'sdxl', name: 'SDXL', type: 'main' }; + const testCase = { height: 1024, mode: 'txt2img', positive_prompt: 'a fox', seed: 7, steps: 4, width: 1024 }; + const request = { + capabilities, + cases: [ + { ...testCase, id: 'ok', model_key: 'sdxl' }, + { ...testCase, id: 'missing', model_key: 'missing' }, + ], + models: [sdxl], + version: 1, + }; + + const { code, response } = await runCli(context, JSON.stringify(request)); + + assert.equal(code, 0); + assert.equal(response.version, 1); + assert.deepEqual( + response.cases.map((result) => [result.id, 'batch' in result]), + [ + ['ok', true], + ['missing', false], + ] + ); + assert.match(response.cases[1].error, /model_key "missing"/); +}); + +for (const [label, requestText] of [ + ['unparsable JSON', '{ "version": 1,'], + ['a malformed request', JSON.stringify({ cases: [], models: [], version: 1 })], +]) { + test(`rejects ${label} with exit 2 and writes no response`, async (context) => { + const { code, response, stderr } = await runCli(context, requestText); + + assert.equal(code, 2); + assert.equal(response, null); + assert.match(stderr, /^graphs:export: /); + }); +} diff --git a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs index 8f4d1fbcb50..561b0bc7c28 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs @@ -224,8 +224,8 @@ test('the HTTP reset contract selects profiles explicitly and restores the start } }); -const withRepresentativeBackend = async (run) => { - const backend = await startMockBackend(0, { profile: 'representative' }); +const withRepresentativeBackend = async (run, options = {}) => { + const backend = await startMockBackend(0, { ...options, profile: 'representative' }); try { await run(backend); @@ -1182,6 +1182,142 @@ test('workflow thumbnails round-trip their bytes and refresh the listed URL on e }); }); +test('working-memory calibration: a run per request, replayed by its key, advanced by summary reads', async () => { + let now = 0; + + await withRepresentativeBackend( + async (backend) => { + const base = '/api/v2/models/i/fixture-model-001/working_memory'; + const plan = await getJson(backend, `${base}/plan`); + const post = (key) => + getJson(backend, `${base}/calibrate`, { + body: JSON.stringify({ + attempt_id: 'attempt-1', + data: [], + graph: { edges: [], id: plan.cases[0].id, nodes: {} }, + idempotency_key: key, + }), + method: 'POST', + }); + + assert.equal(plan.supported, true); + assert.deepEqual((await getJson(backend, `${base}/summary`)).runs, []); + + const first = await post('attempt:0'); + assert.deepEqual(await post('attempt:0'), first); + const second = await post('attempt:1'); + assert.notEqual(second.run_id, first.run_id); + + // A read a second after the last step finishes the running item and starts the next one. + const statuses = async () => { + now += 1_050; + return (await getJson(backend, `${base}/summary`)).runs.map((run) => run.status); + }; + assert.deepEqual(await statuses(), ['running', 'running']); + const summary = await getJson(backend, `${base}/summary`); + assert.deepEqual( + summary.runs.map((run) => run.attempt_id), + ['attempt-1', 'attempt-1'] + ); + assert.equal(summary.measurement_verified, true); + const badAttempt = await fetch(`${backend.origin}${base}/calibrate`, { + body: JSON.stringify({ attempt_id: 'has space', graph: { edges: [], id: 'x', nodes: {} } }), + method: 'POST', + }); + assert.equal(badAttempt.status, 422); + assert.deepEqual(await statuses(), ['completed', 'running']); + + const origin = encodeURIComponent(`invokeai:wm-calibration:${second.run_id}`); + assert.deepEqual( + await getJson(backend, `/api/v1/queue/default/cancel_all?origin_prefix=${origin}`, { method: 'PUT' }), + { + canceled: 1, + } + ); + assert.deepEqual(await statuses(), ['completed', 'canceled']); + + assert.deepEqual(await getJson(backend, base, { method: 'DELETE' }), { points_removed: 1, runs_removed: 2 }); + assert.deepEqual((await getJson(backend, `${base}/summary`)).lines, []); + // Calibration items never join the fixture queue. + assert.equal((await getJson(backend, '/api/v1/queue/default/item_ids')).total_count, 500); + }, + { clock: () => now } + ); +}); + +test('working-memory calibration: seeded model states and device scenarios', async () => { + await withRepresentativeBackend(async (backend) => { + const summaryOf = (key) => getJson(backend, `/api/v2/models/i/${key}/working_memory/summary`); + const states = async (key) => (await summaryOf(key)).lines.map((line) => line.state); + + assert.deepEqual(await states('fixture-model-006'), ['in_use', 'in_use']); + assert.deepEqual(await states('fixture-model-011'), ['stale', 'stale']); + assert.deepEqual(await states('fixture-model-016'), ['unusable', 'unusable']); + assert.deepEqual(await states('fixture-model-036'), ['insufficient']); + assert.ok((await summaryOf('fixture-model-021')).runs.some((run) => run.status === 'running')); + assert.ok((await summaryOf('fixture-model-026')).runs.some((run) => run.status === 'failed')); + assert.equal((await getJson(backend, '/api/v2/models/i/fixture-model-031/working_memory/plan')).supported, false); + + // A model never calibrated itself shows the lines another model measured, which name both sides. + const shared = (await summaryOf('fixture-model-041')).lines; + assert.deepEqual( + shared.map((line) => [line.state, line.own_windows, line.measured_by.map((entry) => entry.model_key)]), + [ + ['in_use', 0, ['fixture-model-006']], + ['in_use', 0, ['fixture-model-006']], + ] + ); + const carrier = (await summaryOf('fixture-model-006')).lines[0]; + assert.equal(carrier.own_windows, 9); + assert.deepEqual(carrier.measured_by, [{ model_key: 'fixture-model-006', name: 'Fixture Model 006', windows: 9 }]); + assert.equal((await fetch(`${backend.origin}/api/v2/models/i/no-such-model/working_memory/summary`)).status, 404); + + // A warm 9060 XT: the largest fp32 size measured fragmentation and is left out of the fit and its range; an fp16 + // size with one clean window among partially resident ones still counts, on that window alone. + const [fp16, fp32] = (await summaryOf('fixture-model-046')).lines; + const sizesOf = (line) => line.sizes.map((size) => [size.width, size.counted, size.excluded_reason, size.windows]); + assert.deepEqual(sizesOf(fp32), [ + [576, true, null, 3], + [832, true, null, 3], + [1216, false, 'fragmented', 3], + ]); + assert.deepEqual(fp32.measured_range, { max_pixels: 832 * 832, min_pixels: 576 * 576 }); + // Above that range the server prices the larger of the user line and the shipped value; here the line is lower. + const [, within, above] = fp32.sizes; + assert.equal(above.user_bytes, above.shipped_bytes); + assert.ok(within.user_bytes < within.shipped_bytes); + assert.equal(fp32.own_windows, 6); + assert.equal(fp32.state, 'in_use'); + assert.deepEqual(sizesOf(fp16), [ + [576, true, null, 3], + [832, true, null, 1], + [1216, true, null, 3], + ]); + assert.equal(fp16.own_windows, 7); + + await getJson(backend, '/__faults', { body: JSON.stringify({ workingMemory: 'notVerified' }), method: 'POST' }); + assert.equal((await summaryOf('fixture-model-006')).measurement_verified, false); + + await getJson(backend, '/__faults', { body: JSON.stringify({ workingMemory: 'multiGpu' }), method: 'POST' }); + const status = await getJson(backend, '/api/v2/models/working_memory/status'); + assert.equal(status.available, false); + assert.match(status.unavailable_reason, /exactly one generation device/); + assert.equal( + (await fetch(`${backend.origin}/api/v2/models/i/fixture-model-001/working_memory/summary`)).status, + 409 + ); + + await getJson(backend, '/__faults', { body: JSON.stringify({ workingMemory: 'error' }), method: 'POST' }); + assert.equal((await fetch(`${backend.origin}/api/v2/models/i/fixture-model-001/working_memory/plan`)).status, 503); + + const unknown = await fetch(`${backend.origin}/__faults`, { + body: JSON.stringify({ workingMemory: 'sideways' }), + method: 'POST', + }); + assert.equal(unknown.status, 400); + }); +}); + test('shared boards answer as the backend role table does', async () => { const backend = await startMockBackend(0, { profile: 'representative' }); const call = (path, init) => diff --git a/invokeai/frontend/webv2/scripts/mock-backend.mjs b/invokeai/frontend/webv2/scripts/mock-backend.mjs index 7d101a6cbda..a23d0f36691 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend.mjs @@ -11,6 +11,7 @@ import { getMockBackendFixtureCounts, MOCK_BACKEND_FIXED_EPOCH, } from './mock-backend-fixtures.mjs'; +import { createWorkingMemoryMock, WORKING_MEMORY_SCENARIOS } from './mock-working-memory.mjs'; /** The canvas schema every mock project declares; the real server keeps the same floor per record. */ const DEFAULT_CANVAS_SCHEMA_VERSION = 3; @@ -77,11 +78,12 @@ const ARCHITECTURE_CAPABILITIES = JSON.parse( * multi-user auth, with a non-admin or an admin session: sign in with any email and password (or reload a page that * already holds the token). `sharedBoards: true` adds boards other accounts own (`SHARED_BOARD_FIXTURES`), which the * caller sees through general access, a membership, or admin rights. + * `workingMemory` selects a calibration device scenario (see mock-working-memory.mjs). */ const CAPABILITY_FAULTS = new Set(['ok', 'error', 'empty']); const INTERMEDIATES_CALLERS = new Set(['admin', 'user', 'multiuser-admin']); -const createFaults = () => ({ capabilities: 'ok', intermediatesCaller: 'admin' }); +const createFaults = () => ({ capabilities: 'ok', intermediatesCaller: 'admin', workingMemory: 'ok' }); const isAdminCaller = (faults) => faults.intermediatesCaller !== 'user'; const isMultiuserCaller = (faults) => faults.intermediatesCaller !== 'admin'; @@ -249,7 +251,8 @@ const STARTER_MODELS_RESPONSE = { const DELIVERY_CLAIM_TTL_MS = 120_000; -const createState = (profile) => { +/** `clock` is the working-memory mock's time source (see mock-working-memory.mjs). */ +const createState = (profile, clock) => { const fixture = assertMockBackendFixture(createMockBackendFixture(profile)); return { @@ -282,6 +285,7 @@ const createState = (profile) => { invocationCache: { enabled: true, hits: 0, max_size: 1000, misses: 0, size: 0 }, videos: new Map(fixture.videos.map((video) => [video.video_name, clone(video)])), workflows: new Map(fixture.workflows.map((workflow) => [workflow.workflow_id, clone(workflow)])), + workingMemory: createWorkingMemoryMock(fixture.models, { clock }), nextWorkflowNumber: fixture.workflows.length + 1, /** Custom workflow thumbnails by workflow id; the version stands in for the server's cache-busting query. */ workflowThumbnails: new Map(), @@ -929,9 +933,9 @@ const parseByteRange = (value, size) => { : null; }; -export const startMockBackend = async (port, { profile = 'empty' } = {}) => { +export const startMockBackend = async (port, { clock = Date.now, profile = 'empty' } = {}) => { const initialProfile = assertMockBackendProfileName(profile); - let state = createState(initialProfile); + let state = createState(initialProfile, clock); let faults = createFaults(); const server = createServer(async (request, response) => { @@ -950,7 +954,7 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { const requestedProfile = url.searchParams.get('profile') ?? body.profile ?? initialProfile; try { - state = createState(assertMockBackendProfileName(requestedProfile)); + state = createState(assertMockBackendProfileName(requestedProfile), clock); } catch (error) { return json(400, { detail: error instanceof Error ? error.message : String(error) }); } @@ -983,6 +987,7 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { const body = await readJsonBody(request); const requested = url.searchParams.get('capabilities') ?? body.capabilities; const caller = url.searchParams.get('intermediatesCaller') ?? body.intermediatesCaller; + const workingMemory = url.searchParams.get('workingMemory') ?? body.workingMemory; if (body.sharedBoards === true && !state.boards.has(SHARED_BOARD_FIXTURES[0].board_id)) { seedSharedBoards(state); @@ -1003,6 +1008,14 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { faults.intermediatesCaller = caller; } + + if (workingMemory !== undefined && workingMemory !== null) { + if (!WORKING_MEMORY_SCENARIOS.has(workingMemory)) { + return json(400, { detail: `Unknown working-memory scenario: ${String(workingMemory)}` }); + } + + faults.workingMemory = workingMemory; + } } return json(200, { ok: true, faults: { ...faults } }); @@ -1649,6 +1662,23 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { return json(200, { item_ids: returned, total_count: returned.length }); } + if (action === 'cancel_all' && method === 'PUT') { + const originPrefix = url.searchParams.get('origin_prefix'); + + // Calibration items live with the working-memory mock; see mock-working-memory.mjs. + if (originPrefix !== null && state.workingMemory.isCalibrationPrefix(originPrefix)) { + return json(200, { canceled: state.workingMemory.cancelByOrigin(originPrefix) }); + } + + let canceled = 0; + for (const item of scopedItems) { + if (item.status === 'pending' || item.status === 'in_progress') { + item.status = 'canceled'; + canceled += 1; + } + } + return json(200, { canceled }); + } if (action === 'clear' && method === 'PUT') { state.queueItems.clear(); return json(200, { deleted: true }); @@ -1732,6 +1762,15 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { return json(200, null); } + if (path.startsWith('/api/v2/models/') && path.includes('working_memory')) { + const body = method === 'POST' ? await readJsonBody(request) : null; + const answer = state.workingMemory.handle({ body, method, path, scenario: faults.workingMemory }); + + if (answer) { + return json(answer.status, answer.body); + } + } + if (method === 'GET' && path === '/api/v2/models/stats') { return json(200, null); } diff --git a/invokeai/frontend/webv2/scripts/mock-working-memory.mjs b/invokeai/frontend/webv2/scripts/mock-working-memory.mjs new file mode 100644 index 00000000000..5845f9ea18e --- /dev/null +++ b/invokeai/frontend/webv2/scripts/mock-working-memory.mjs @@ -0,0 +1,593 @@ +/** + * Working-memory calibration for the mock backend: plans, summaries, the device status, calibration runs and resets + * (`/api/v2/models/.../working_memory`), and cancelling a run's items by origin. The mock has no Socket.IO, so a run + * advances when a client reads the model's summary: a read at least a second after the last step finishes the running + * item and starts the next, which a section polling every two seconds turns into visible progress. + * + * Main models (`fixture-model-001`, `-006`, ...) each show one state, by key (`MODEL_SCENARIOS`); every other main model is never + * calibrated, and VAEs are unsupported. `SHARED_LINES` pairs models whose decode lines pool their measurements, so one + * of them shows lines only the other measured. `fixture-model-046` follows a warm RX 9060 XT: its largest fp32 size + * measured allocator fragmentation and is not counted, and two of three windows at one fp16 size ran partially resident, + * which leaves that size counted on the third. `POST /__faults` with `workingMemory` selects a device scenario instead. + * Calibration items live here rather than in the fixture queue, so profile counts stay fixed. Tests pass their own + * `clock` to advance runs without waiting. + */ + +export const WORKING_MEMORY_SCENARIOS = new Set(['ok', 'multiGpu', 'noGpu', 'storeUnusable', 'error', 'notVerified']); + +const ORIGIN_PREFIX = 'invokeai:wm-calibration:'; +const ATTEMPT_ID_PATTERN = /^[A-Za-z0-9_.:-]{1,64}$/; +const DEVICE_CLASS = { + arch: 'sm_89', + backend: 'cuda', + id: '0123456789abcdef', + name: 'NVIDIA GeForce RTX 4090', + total_mib: 24_564, + vendor: 'nvidia', +}; +const ENV_ID = 'fedcba9876543210'; +const OLD_ENV_ID = '00ff00ff00ff00ff'; +const MIB = 2 ** 20; +const SIDES = [768, 1088, 1536]; +const DTYPES = ['fp16', 'fp32']; +const REPETITIONS = { measured: 2, warm_up: 1 }; +const ENCODE_NOT_COVERED = + 'encode: text-to-image runs no encode, so image-to-image and inpainting encodes keep the shipped values'; +const SEEDED_AT = Date.parse('2025-12-01T10:00:00.000Z'); +/** A run's items advance at most this often, so a section that refetches on every queued step still shows progress. */ +const STEP_MS = 1_000; + +/** What each seeded main model shows, by key. */ +const MODEL_SCENARIOS = { + 'fixture-model-006': 'completed', + 'fixture-model-011': 'stale', + 'fixture-model-016': 'unusable', + 'fixture-model-021': 'running', + 'fixture-model-026': 'failed', + 'fixture-model-031': 'unsupported', + 'fixture-model-036': 'insufficient', + 'fixture-model-046': 'fragmented', +}; + +/** The sizes the `fragmented` scenario measured, as the 9060 XT plan picked them. */ +const FRAGMENTED_SIDES = [576, 832, 1216]; + +/** Why a seeded window of the `fragmented` scenario is excluded from the fit, by its step; undefined when it counts. */ +const excludedIn = (scenario, step, repetition) => { + if (scenario !== 'fragmented') { + return undefined; + } + + if (step.line === 'decode_fp32' && step.width === 1216) { + return 'fragmented'; + } + + return step.line === 'decode_fp16' && step.width === 832 && repetition < 2 ? 'partial_residency' : undefined; +}; + +/** Models whose lines pool both models' measurements; the first is never calibrated itself. */ +const SHARED_LINES = [['fixture-model-041', 'fixture-model-006']]; + +const partnersOf = (modelKey) => + SHARED_LINES.flatMap(([left, right]) => (left === modelKey ? [right] : right === modelKey ? [left] : [])); + +/** A plausible decode demand: the VAE's weights plus a per-pixel activation cost, twice that in fp32. */ +const demandOf = (pixels, dtype) => Math.round((dtype === 'fp32' ? 2 : 1) * (420 * MIB + pixels * 1_100)); + +const casesOf = (model) => + DTYPES.flatMap((dtype) => + (MODEL_SCENARIOS[model.key] === 'fragmented' ? FRAGMENTED_SIDES : SIDES).map((side) => ({ + components: {}, + estimator: 'sd_vae_decode', + height: side, + id: `sd_vae_decode-decode-${dtype}-${side}x${side}`, + line: `decode_${dtype}`, + mode: 'txt2img', + model_key: model.key, + negative_prompt: '', + positive_prompt: 'a mountain lake at dawn, detailed photograph', + seed: 1, + settings: { vaePrecision: dtype }, + steps: 2, + width: side, + })) + ); + +const stepsOf = (cases) => + DTYPES.flatMap((dtype) => + cases + .filter((item) => item.line === `decode_${dtype}`) + .flatMap((item) => Array.from({ length: REPETITIONS.warm_up + REPETITIONS.measured }, () => item)) + ); + +export const createWorkingMemoryMock = (models, { clock = Date.now } = {}) => { + const mainModels = models.filter((model) => model.type === 'main'); + const byKey = new Map(models.map((model) => [model.key, model])); + /** Runs by model key, oldest first: `{ run_id, created_at, items: [{ item_id, case, status, error }] }`. */ + const runs = new Map(); + /** Measured points by model key: `{ line, width, height, bytes, env_id }`. */ + const points = new Map(); + /** Run ids by idempotency key, so a resent request returns the run it created. */ + const receipts = new Map(); + let nextRunNumber = 1; + let nextItemId = 900_000; + let createdCount = 0; + let lastAdvanceAt = 0; + + const now = () => new Date(SEEDED_AT + 86_400_000 * 30 + createdCount++ * 1_000).toISOString(); + + /** `excluded` names why the fit leaves the window out: `fragmented` or `partial_residency`. */ + const addPoint = (modelKey, item, envId = ENV_ID, excluded = undefined) => { + const dtype = item.case.line.split('_').at(-1); + const list = points.get(modelKey) ?? []; + const demand = demandOf(item.case.width * item.case.height, dtype); + + list.push({ + // A fragmented allocator reserves well beyond what the decode needs. + bytes: excluded === 'fragmented' ? Math.round(demand * 1.4) : demand, + env_id: envId, + excluded, + height: item.case.height, + line: item.case.line, + width: item.case.width, + }); + points.set(modelKey, list); + }; + + const createRun = (modelKey, steps, createdAt, attemptId) => { + const run = { + attempt_id: attemptId, + created_at: createdAt, + items: steps.map((step) => ({ case: step, error: null, item_id: nextItemId++, status: 'pending' })), + run_id: `mock-wm-run-${String(nextRunNumber++).padStart(4, '0')}`, + }; + const list = runs.get(modelKey) ?? []; + + list.push(run); + runs.set(modelKey, list); + + return run; + }; + + /** Seeds one model's state: settled runs one per step, two seconds apart, a month before now. */ + const seed = (model, scenario) => { + const cases = casesOf(model); + const steps = scenario === 'insufficient' ? stepsOf(cases).slice(0, 3) : stepsOf(cases); + const settledRuns = steps.map((step, index) => + createRun(model.key, [step], new Date(SEEDED_AT + index * 2_000).toISOString(), `seed-${model.key}`) + ); + + settledRuns.forEach((run, index) => { + const [item] = run.items; + const isRunning = scenario === 'running' && index >= 4; + + if (scenario === 'failed' && index === 7) { + item.status = 'failed'; + item.error = 'OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB'; + } else if (scenario === 'failed' && index > 7) { + item.status = 'canceled'; + } else if (isRunning) { + item.status = index === 4 ? 'in_progress' : 'pending'; + } else { + const repetition = index % (REPETITIONS.warm_up + REPETITIONS.measured); + + item.status = 'completed'; + addPoint( + model.key, + item, + scenario === 'unusable' ? OLD_ENV_ID : ENV_ID, + excludedIn(scenario, item.case, repetition) + ); + } + }); + }; + + const reset = () => { + runs.clear(); + points.clear(); + receipts.clear(); + nextRunNumber = 1; + nextItemId = 900_000; + createdCount = 0; + lastAdvanceAt = 0; + + for (const model of mainModels) { + const scenario = MODEL_SCENARIOS[model.key]; + + if (scenario && scenario !== 'unsupported') { + seed(model, scenario); + } + } + }; + + reset(); + + const runStatus = (run, pointCount) => { + const statuses = run.items.map((item) => item.status); + + if (statuses.some((status) => status === 'pending' || status === 'in_progress')) { + return { error: null, status: 'running' }; + } + + const failed = run.items.find((item) => item.status === 'failed'); + + if (failed) { + return { error: failed.error, status: 'failed' }; + } + + if (statuses.every((status) => status === 'completed')) { + return { error: null, status: 'completed' }; + } + + return { error: null, status: pointCount > 0 ? 'partial' : 'canceled' }; + }; + + const pointCountOf = (run) => run.items.filter((item) => item.status === 'completed').length; + + const modelRuns = (modelKey) => + (runs.get(modelKey) ?? []).map((run) => { + const pointCount = pointCountOf(run); + + return { + attempt_id: run.attempt_id, + created_at: run.created_at, + point_count: pointCount, + run_id: run.run_id, + ...runStatus(run, pointCount), + }; + }); + + /** Finishes the running item of the model's oldest unsettled run and starts the next one. */ + const advance = (modelKey) => { + if (clock() - lastAdvanceAt < STEP_MS) { + return; + } + lastAdvanceAt = clock(); + + const items = (runs.get(modelKey) ?? []).flatMap((run) => run.items); + const running = items.find((item) => item.status === 'in_progress'); + + if (running) { + running.status = 'completed'; + addPoint(modelKey, running); + } + + const next = items.find((item) => item.status === 'pending'); + + if (next) { + next.status = 'in_progress'; + } + }; + + /** + * A model's lines: its own points, plus its partners' points in the current environment, which the server pools + * the same way. A line only partners measured appears with `own_windows` 0. + */ + const linesOf = (modelKey, scenario) => { + const own = (points.get(modelKey) ?? []).map((point) => ({ ...point, model_key: modelKey })); + const shared = partnersOf(modelKey).flatMap((key) => + (points.get(key) ?? []).filter((point) => point.env_id === ENV_ID).map((point) => ({ ...point, model_key: key })) + ); + + return DTYPES.flatMap((dtype) => { + const line = `decode_${dtype}`; + const ownPoints = own.filter((point) => point.line === line); + const envId = ownPoints.at(-1)?.env_id ?? ENV_ID; + const linePoints = [...ownPoints, ...(envId === ENV_ID ? shared.filter((point) => point.line === line) : [])]; + + if (linePoints.length === 0) { + return []; + } + + // Only counted windows feed the fit; a size counts while any of its windows does. + const countedPoints = linePoints.filter((point) => point.excluded === undefined); + const sides = [...new Set(linePoints.map((point) => point.width))].sort((left, right) => left - right); + const countedSides = new Set(countedPoints.map((point) => point.width)); + const fitted = countedSides.size >= 2; + const maxCounted = Math.max(...countedSides); + const sizes = sides.map((side) => { + const counted = countedSides.has(side); + const atSide = (counted ? countedPoints : linePoints).filter((point) => point.width === side); + const largest = atSide.reduce((left, right) => (right.bytes > left.bytes ? right : left)); + const pixels = side * side; + const demand = demandOf(pixels, dtype); + const shipped = Math.round(demand * 1.35); + const line = Math.round(demand * 1.05); + + return { + counted, + excluded_reason: counted ? null : largest.excluded, + height: side, + measured_bytes: largest.bytes, + pixels, + shipped_bytes: shipped, + // Like the server's resolve: above the measured range the larger of the user line and the shipped value. + user_bytes: fitted && envId === ENV_ID ? (side > maxCounted ? Math.max(line, shipped) : line) : null, + width: side, + windows: atSide.length, + }; + }); + const state = + envId !== ENV_ID ? 'unusable' : !fitted ? 'insufficient' : scenario === 'stale' ? 'stale' : 'in_use'; + const measuredBy = [...new Set(countedPoints.map((point) => point.model_key))] + .map((key) => ({ + model_key: key, + name: byKey.get(key)?.name ?? null, + windows: countedPoints.filter((point) => point.model_key === key).length, + })) + .sort((left, right) => left.name.localeCompare(right.name) || left.model_key.localeCompare(right.model_key)); + + return [ + { + base_bytes: fitted ? 420 * MIB : null, + carrier: 'sd_vae_decode', + changed_facts: state === 'stale' ? ['driver'] : state === 'unusable' ? ['torch', 'cuda'] : [], + env_id: envId, + estimator: 'sd_vae_decode', + line, + measured_by: measuredBy, + measured_range: fitted + ? { max_pixels: Math.max(...countedSides) ** 2, min_pixels: Math.min(...countedSides) ** 2 } + : null, + own_windows: countedPoints.filter((point) => point.model_key === modelKey).length, + problem: fitted ? null : 'A line needs two sizes at least 1.5 times apart.', + slope_bytes_per_pixel: fitted ? 1_100 : null, + sizes, + state, + version: 1, + }, + ]; + }); + }; + + const unavailableReason = (scenario) => + scenario === 'multiGpu' + ? 'Calibration needs exactly one generation device; this machine generates on cuda:0 and cuda:1.' + : scenario === 'noGpu' + ? 'This machine has no GPU to calibrate.' + : scenario === 'storeUnusable' + ? `working_memory/${DEVICE_CLASS.id}.json has schema_version 9; this version reads up to 1.` + : null; + + const plan = (model, scenario) => { + const base = { + calibrated_through: null, + cases: [], + device_class: null, + estimated_seconds: null, + estimators: model.type === 'main' ? ['sd_vae_decode'] : [], + model_key: model.key, + not_covered: [], + reason: null, + repetitions: REPETITIONS, + shared_with: [], + supported: false, + }; + + if (model.type !== 'main') { + return { + ...base, + reason: `Calibration covers the decode of SD1, SD2, SDXL, SD3, FLUX.1, FLUX.2, Z-Image and CogView4 main models and their VAEs; no installed main model generates with this ${model.base} ${model.type}.`, + }; + } + + if (MODEL_SCENARIOS[model.key] === 'unsupported') { + return { ...base, reason: 'The inpaint variant does not generate from text alone.' }; + } + + if (scenario === 'multiGpu' || scenario === 'noGpu') { + return { ...base, reason: unavailableReason(scenario) }; + } + + const cases = casesOf(model); + const shared = mainModels.filter((other) => other.key !== model.key).slice(0, 3); + + return { + ...base, + cases, + device_class: { id: DEVICE_CLASS.id, name: DEVICE_CLASS.name }, + estimated_seconds: 30 + stepsOf(cases).length * 6, + not_covered: [ENCODE_NOT_COVERED], + shared_with: shared.map((other) => ({ + base: other.base, + key: other.key, + lines: DTYPES.map((dtype) => `decode_${dtype}`), + name: other.name, + type: other.type, + })), + supported: true, + }; + }; + + const status = (scenario) => { + const reason = unavailableReason(scenario); + const hasClass = scenario !== 'multiGpu' && scenario !== 'noGpu'; + const allRuns = [...runs.entries()].flatMap(([modelKey, list]) => + list.map((run) => { + const pointCount = pointCountOf(run); + + return { + attempt_id: run.attempt_id, + created_at: run.created_at, + env_id: ENV_ID, + item_ids: run.items.map((item) => item.item_id), + model_hash: byKey.get(modelKey)?.hash ?? null, + model_key: modelKey, + point_count: pointCount, + run_id: run.run_id, + user_id: 'system', + ...runStatus(run, pointCount), + }; + }) + ); + + return { + available: reason === null, + current_device_class_id: hasClass ? DEVICE_CLASS.id : null, + current_env_id: hasClass ? ENV_ID : null, + device_classes: hasClass + ? [ + { + device_class: scenario === 'storeUnusable' ? null : DEVICE_CLASS, + device_class_id: DEVICE_CLASS.id, + envs: + scenario === 'storeUnusable' + ? [] + : [ + { + hard: { + cuda: '12.8', + os: scenario === 'notVerified' ? 'linux' : 'windows', + torch: '2.7', + }, + hard_matches_current: true, + id: ENV_ID, + point_count: [...points.values()].flat().filter((point) => point.env_id === ENV_ID).length, + soft: { driver: '576.02' }, + soft_matches_current: true, + }, + ], + is_current: true, + ...measurement(scenario), + point_counts: [], + problem: scenario === 'storeUnusable' ? reason : null, + runs: scenario === 'storeUnusable' ? [] : allRuns, + usable: scenario !== 'storeUnusable', + }, + ] + : [], + unavailable_reason: reason, + }; + }; + + /** The measurement method of the device class; `notVerified` stands for a backend nobody checked it on. */ + const measurement = (scenario) => ({ + measurement_method: 'allocator_peak', + measurement_verified: scenario !== 'notVerified', + }); + + const summary = (model, scenario) => { + advance(model.key); + + return { + current_env_id: ENV_ID, + device_class_id: DEVICE_CLASS.id, + excluded_windows: 0, + lines: scenario === 'storeUnusable' ? [] : linesOf(model.key, MODEL_SCENARIOS[model.key]), + ...measurement(scenario), + model_key: model.key, + problem: scenario === 'storeUnusable' ? unavailableReason(scenario) : null, + runs: scenario === 'storeUnusable' ? [] : modelRuns(model.key), + usable: scenario !== 'storeUnusable', + }; + }; + + const calibrate = (model, body, scenario) => { + if (unavailableReason(scenario) !== null) { + return { body: { detail: unavailableReason(scenario) }, status: 409 }; + } + + if (typeof body?.graph?.nodes !== 'object' || body.graph.nodes === null) { + return { body: { detail: 'The graph cannot be measured as a calibration run.' }, status: 422 }; + } + + const attemptId = body.attempt_id ?? null; + + if (attemptId !== null && !(typeof attemptId === 'string' && ATTEMPT_ID_PATTERN.test(attemptId))) { + return { body: { detail: 'attempt_id must be 1 to 64 of A-Z a-z 0-9 _ . : -' }, status: 422 }; + } + + const key = typeof body.idempotency_key === 'string' ? body.idempotency_key : null; + const known = key ? receipts.get(key) : undefined; + const existing = known && (runs.get(model.key) ?? []).find((run) => run.run_id === known); + const caseId = typeof body.graph.id === 'string' ? body.graph.id : null; + const step = casesOf(model).find((item) => item.id === caseId) ?? casesOf(model)[0]; + const run = existing ?? createRun(model.key, [step], now(), attemptId); + + if (key) { + receipts.set(key, run.run_id); + } + + return { + body: { device_class_id: DEVICE_CLASS.id, item_ids: run.items.map((item) => item.item_id), run_id: run.run_id }, + status: 201, + }; + }; + + const cancelByOrigin = (originPrefix) => { + let canceled = 0; + + for (const list of runs.values()) { + for (const run of list) { + if (!`${ORIGIN_PREFIX}${run.run_id}`.startsWith(originPrefix)) { + continue; + } + + for (const item of run.items) { + if (item.status === 'pending' || item.status === 'in_progress') { + item.status = 'canceled'; + canceled += 1; + } + } + } + } + + return canceled; + }; + + const resetModel = (modelKey) => { + const removed = runs.get(modelKey) ?? []; + const pointsRemoved = (points.get(modelKey) ?? []).length; + + runs.delete(modelKey); + points.delete(modelKey); + + return { points_removed: pointsRemoved, runs_removed: removed.length }; + }; + + /** Answers a working-memory route, or returns null for any other path. */ + const handle = ({ body, method, path, scenario }) => { + if (path === '/api/v2/models/working_memory/status' && method === 'GET') { + return scenario === 'error' + ? { body: { detail: 'The calibration store cannot be read (fixture outage).' }, status: 503 } + : { body: status(scenario), status: 200 }; + } + + const match = /^\/api\/v2\/models\/i\/([^/]+)\/working_memory(?:\/(plan|summary|calibrate))?$/.exec(path); + + if (!match) { + return null; + } + + const model = byKey.get(decodeURIComponent(match[1])); + + if (!model) { + return { body: { detail: 'The model could not be found' }, status: 404 }; + } + + if (scenario === 'error') { + return { body: { detail: 'The calibration store cannot be read (fixture outage).' }, status: 503 }; + } + + const noDeviceClass = scenario === 'multiGpu' || scenario === 'noGpu'; + + switch (`${method} ${match[2] ?? ''}`) { + case 'GET plan': + return { body: plan(model, scenario), status: 200 }; + case 'GET summary': + return noDeviceClass + ? { body: { detail: 'No device_class_id was given and there is no current one' }, status: 409 } + : { body: summary(model, scenario), status: 200 }; + case 'POST calibrate': + return calibrate(model, body, scenario); + case 'DELETE ': + return noDeviceClass + ? { body: { detail: 'No device_class_id was given and there is no current one' }, status: 409 } + : { body: resetModel(model.key), status: 200 }; + default: + return { body: { detail: 'Method Not Allowed' }, status: 405 }; + } + }; + + return { cancelByOrigin, handle, isCalibrationPrefix: (prefix) => prefix.startsWith(ORIGIN_PREFIX), reset }; +}; diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.browser.test.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.browser.test.ts new file mode 100644 index 00000000000..3158852172e --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.browser.test.ts @@ -0,0 +1,118 @@ +import type { ModelConfig, WorkingMemoryCase } from '@features/models'; + +import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; +import { setModelsSnapshotForTests } from '@features/models/data/modelsStore'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +/** Serves the capability table; module mocks cannot be used here, since they would instrument the worker's modules. */ +const fetchCapabilities = () => + vi + .spyOn(globalThis, 'fetch') + .mockImplementation(() => Promise.resolve(Response.json(architectureCapabilitiesFixture))); + +const SDXL = { + base: 'sdxl', + format: 'diffusers', + hash: 'blake3:sdxl', + key: 'sdxl', + name: 'SDXL', + type: 'main', + variant: 'normal', +} as ModelConfig; + +const planCase = (id: string, modelKey = 'sdxl'): WorkingMemoryCase => ({ + components: {}, + estimator: 'sdxl_vae_decode', + height: 768, + id, + line: 'decode_fp32', + mode: 'txt2img', + model_key: modelKey, + negative_prompt: '', + positive_prompt: 'a lake', + seed: 1, + settings: { vaePrecision: 'fp32' }, + steps: 2, + width: 768, +}); + +afterEach(() => { + setModelsSnapshotForTests({ models: [], status: 'idle' }); + vi.restoreAllMocks(); +}); + +/** Runs the real worker: its bundle must start in a browser and compile without the page's modules. */ +describe('compileCalibrationGraphs', () => { + it('compiles the cases in a worker over the tab’s models and the served capability table', async () => { + const fetch = fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { compileCalibrationGraphs } = await import('./calibrationGraphs'); + + const [compiled] = await compileCalibrationGraphs([planCase('fp32-768')], new AbortController().signal); + const nodes = Object.values((compiled!.graph as { nodes: Record> }).nodes); + + expect(String(fetch.mock.calls[0]?.[0])).toMatch(/\/api\/v2\/models\/capabilities$/); + expect(compiled!.caseId).toBe('fp32-768'); + expect(nodes.find((node) => node.type === 'l2i')).toMatchObject({ fp32: true }); + expect(nodes.find((node) => node.type === 'noise')).toMatchObject({ height: 768, width: 768 }); + }); + + it('rejects with the compiler’s message for a case that cannot compile', async () => { + fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { compileCalibrationGraphs } = await import('./calibrationGraphs'); + + await expect( + compileCalibrationGraphs([planCase('lost', 'removed-model')], new AbortController().signal) + ).rejects.toThrow(/^Case lost: model_key "removed-model" is not a selectable main model/); + }); + + /** A worker that never answers: its request is dropped. Resolves once the request would have been sent. */ + const silenceWorker = () => { + const terminate = vi.spyOn(Worker.prototype, 'terminate'); + let sent = () => undefined as void; + const posted = new Promise((resolve) => { + sent = resolve; + }); + vi.spyOn(Worker.prototype, 'postMessage').mockImplementation(() => sent()); + + return { posted, terminate }; + }; + + it('stops the worker and rejects when its owner aborts', async () => { + fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { compileCalibrationGraphs } = await import('./calibrationGraphs'); + const { posted, terminate } = silenceWorker(); + const owner = new AbortController(); + + const compiling = compileCalibrationGraphs([planCase('fp32-768')], owner.signal); + await posted; + owner.abort(new Error('Calibration stopped.')); + + await expect(compiling).rejects.toThrow('Calibration stopped.'); + expect(terminate).toHaveBeenCalledTimes(1); + }); + + it('stops a worker that does not answer in time and rejects', async () => { + fetchCapabilities(); + setModelsSnapshotForTests({ models: [SDXL], status: 'loaded' }); + const { COMPILE_TIMEOUT_MS, compileCalibrationGraphs } = await import('./calibrationGraphs'); + const { posted, terminate } = silenceWorker(); + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + + try { + const compiling = compileCalibrationGraphs([planCase('fp32-768')], new AbortController().signal); + const outcome = expect(compiling).rejects.toThrow(/did not answer within 60 s/); + await posted; + vi.advanceTimersByTime(COMPILE_TIMEOUT_MS - 1); + expect(terminate).not.toHaveBeenCalled(); + vi.advanceTimersByTime(1); + + await outcome; + expect(terminate).toHaveBeenCalledTimes(1); + } finally { + vi.useRealTimers(); + } + }); +}); diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.test.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.test.ts new file mode 100644 index 00000000000..d8bbb86828e --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.test.ts @@ -0,0 +1,86 @@ +import type { WorkingMemoryCase } from '@features/models'; + +import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; +import { describe, expect, it } from 'vitest'; + +import { toCalibrationGraphs, toGraphExportCase } from './calibrationGraphs'; +import { exportGraphs } from './graphExport'; + +const MODELS = [ + { + base: 'sdxl', + format: 'diffusers', + hash: 'blake3:sdxl', + key: 'sdxl', + name: 'SDXL', + type: 'main', + variant: 'normal', + }, +]; + +/** A case as the server's plan sends it for an SDXL decode in fp32. */ +const planCase = (overrides: Partial = {}): WorkingMemoryCase => ({ + components: {}, + estimator: 'sdxl_vae_decode', + height: 1088, + id: 'sdxl_vae_decode-decode-fp32-1088x1088', + line: 'decode_fp32', + mode: 'txt2img', + model_key: 'sdxl', + negative_prompt: '', + positive_prompt: 'a mountain lake at dawn, detailed photograph', + seed: 1, + settings: { vaePrecision: 'fp32' }, + steps: 2, + width: 1088, + ...overrides, +}); + +const compile = (cases: WorkingMemoryCase[]) => + toCalibrationGraphs( + cases, + exportGraphs({ + capabilities: architectureCapabilitiesFixture, + cases: cases.map(toGraphExportCase), + models: MODELS, + version: 1, + }) + ); + +const nodesOf = (graph: unknown) => Object.values((graph as { nodes: Record> }).nodes); + +describe('calibration graphs', () => { + it('compiles a plan case as Generate would, at its size, steps and decode precision', () => { + const [compiled] = compile([planCase()]); + const nodes = nodesOf(compiled!.graph); + + expect(compiled!.caseId).toBe('sdxl_vae_decode-decode-fp32-1088x1088'); + expect((compiled!.graph as { id: string }).id).toBe('sdxl_vae_decode-decode-fp32-1088x1088'); + expect(nodes.find((node) => node.type === 'noise')).toMatchObject({ height: 1088, width: 1088 }); + expect(nodes.find((node) => node.type === 'denoise_latents')).toMatchObject({ steps: 2 }); + expect(nodes.find((node) => node.type === 'l2i')).toMatchObject({ fp32: true }); + // Seed and prompt travel as batch data, as Generate queues them. + expect(compiled!.data).toEqual([ + [ + { field_name: 'value', items: [1], node_path: 'seed' }, + { field_name: 'value', items: ['a mountain lake at dawn, detailed photograph'], node_path: 'positive_prompt' }, + { field_name: 'value', items: [''], node_path: 'negative_prompt' }, + ], + ]); + }); + + it('keeps fp16 cases on the fp16 decode', () => { + const [compiled] = compile([ + planCase({ id: 'fp16-case', line: 'decode_fp16', settings: { vaePrecision: 'fp16' } }), + ]); + const nodes = nodesOf(compiled!.graph); + + expect(nodes.find((node) => node.type === 'l2i')).toMatchObject({ fp32: false }); + }); + + it('rejects naming the case that cannot compile, so nothing is queued', () => { + expect(() => compile([planCase(), planCase({ id: 'missing-model', model_key: 'not-installed' })])).toThrow( + /^Case missing-model: model_key "not-installed" is not a selectable main model/ + ); + }); +}); diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.ts new file mode 100644 index 00000000000..5d619f13693 --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.ts @@ -0,0 +1,109 @@ +/** + * The graphs a working-memory calibration queues, compiled exactly as `pnpm graphs:export` compiles its cases, which + * is how Generate compiles them: over the models this tab holds and the capability table the server serves. The + * Model Manager loads this module only when a calibration starts (`compileCalibrationGraphs` in `router.tsx`); the + * compiler itself runs in `calibrationGraphs.worker.ts`. + */ + +import { ensureModelsLoaded, getModelsSnapshot, type CalibrationGraph, type WorkingMemoryCase } from '@features/models'; +import { apiFetchJson } from '@platform/transport/http'; + +import type { CalibrationWorkerReply } from './calibrationGraphs.worker'; +import type { GraphExportResponse } from './graphExport'; + +const GRAPH_EXPORT_VERSION = 1; + +/** + * A plan case in the export's snake_case case format. The plan adds what the case measures (`estimator`, `line`), + * which the export reader rejects as unknown fields. + */ +export const toGraphExportCase = ({ + estimator: _estimator, + line: _line, + ...record +}: WorkingMemoryCase): Omit => record; + +/** Each case's graph and batch data; rejects naming the first case that cannot compile, before anything is queued. */ +export const toCalibrationGraphs = ( + cases: readonly WorkingMemoryCase[], + response: GraphExportResponse +): CalibrationGraph[] => + cases.map((item) => { + const result = response.cases.find((candidate) => candidate.id === item.id); + + if (!result) { + throw new Error(`Case ${item.id} was not compiled.`); + } + + if ('error' in result) { + throw new Error(`Case ${item.id}: ${result.error}`); + } + + return { caseId: item.id, data: result.batch.data, graph: result.batch.graph }; + }); + +/** Compiling a large plan takes seconds; a worker that has not answered by then is stuck. */ +export const COMPILE_TIMEOUT_MS = 60_000; + +/** Settles exactly once: with the worker's answer, or rejecting on a failure, the timeout or `signal`. */ +const runWorker = (request: unknown, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + signal.throwIfAborted(); + + const worker = new Worker(new URL('./calibrationGraphs.worker.ts', import.meta.url), { + name: 'calibration-graphs', + type: 'module', + }); + const settle = (error: Error | null, response?: GraphExportResponse) => { + worker.terminate(); + clearTimeout(timeout); + signal.removeEventListener('abort', onAbort); + + if (error) { + reject(error); + } else { + resolve(response!); + } + }; + const onAbort = () => settle(signal.reason instanceof Error ? signal.reason : new Error('Compiling was stopped.')); + const timeout = setTimeout( + () => settle(new Error(`The graph compiler did not answer within ${String(COMPILE_TIMEOUT_MS / 1_000)} s.`)), + COMPILE_TIMEOUT_MS + ); + + signal.addEventListener('abort', onAbort, { once: true }); + worker.addEventListener('message', (event: MessageEvent) => { + if (event.data.ok) { + settle(null, event.data.response); + } else { + settle(new Error(event.data.error)); + } + }); + worker.addEventListener('messageerror', () => settle(new Error('The graph compiler’s answer could not be read.'))); + worker.addEventListener('error', (event) => { + settle(new Error(event.message || 'The graph compiler did not start.')); + }); + worker.postMessage(request); + }); + +export const compileCalibrationGraphs = async ( + cases: readonly WorkingMemoryCase[], + signal: AbortSignal +): Promise => { + const [capabilities] = await Promise.all([ + apiFetchJson('/api/v2/models/capabilities', { signal }), + ensureModelsLoaded(), + ]); + const { error, models } = getModelsSnapshot(); + + if (models.length === 0) { + throw new Error(error ?? 'Models are not loaded.'); + } + + const response = await runWorker( + { capabilities, cases: cases.map(toGraphExportCase), models, version: GRAPH_EXPORT_VERSION }, + signal + ); + + return toCalibrationGraphs(cases, response); +}; diff --git a/invokeai/frontend/webv2/src/app/calibrationGraphs.worker.ts b/invokeai/frontend/webv2/src/app/calibrationGraphs.worker.ts new file mode 100644 index 00000000000..f1dd3a477b3 --- /dev/null +++ b/invokeai/frontend/webv2/src/app/calibrationGraphs.worker.ts @@ -0,0 +1,26 @@ +/** + * Compiles working-memory calibration cases with the offline graph export (`graphExport.ts`), in a worker. Its own + * bundle keeps Generate's compiler out of both routes' chunk graphs, and its own capability table cannot replace the + * one an open editor holds. + */ + +import { exportGraphs } from './graphExport'; + +export type CalibrationWorkerReply = + | { ok: true; response: ReturnType } + | { ok: false; error: string }; + +interface WorkerScope { + addEventListener(type: 'message', listener: (event: MessageEvent) => void): void; + postMessage(message: CalibrationWorkerReply): void; +} + +const scope = globalThis as unknown as WorkerScope; + +scope.addEventListener('message', (event) => { + try { + scope.postMessage({ ok: true, response: exportGraphs(event.data) }); + } catch (error) { + scope.postMessage({ error: error instanceof Error ? error.message : String(error), ok: false }); + } +}); diff --git a/invokeai/frontend/webv2/src/app/graphExport.test.ts b/invokeai/frontend/webv2/src/app/graphExport.test.ts new file mode 100644 index 00000000000..5ea045a5dfc --- /dev/null +++ b/invokeai/frontend/webv2/src/app/graphExport.test.ts @@ -0,0 +1,232 @@ +import type { BackendGraphContract, BackendInvocationContract } from '@features/generation/contracts'; + +import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; +import { generateGraphCases, satisfiedSettingsFor } from '@features/generation/core/graphCoverage.testing'; +import { withArchitectureCapabilities } from '@features/generation/graphExport'; +import { describe, expect, it } from 'vitest'; + +import { exportGraphs, GraphExportRequestError, type GraphExportCaseResult } from './graphExport'; + +const model = (key: string, base: string, type: string, extra: Record = {}) => ({ + base, + hash: `blake3:${key}`, + key, + name: `${key} name`, + type, + ...extra, +}); + +const MODELS = [ + model('sdxl', 'sdxl', 'main', { format: 'diffusers', variant: 'normal' }), + model('sdxl-lora', 'sdxl', 'lora', { format: 'lycoris' }), + model('sdxl-canny', 'sdxl', 'controlnet', { format: 'diffusers' }), + model('sdxl-ip-adapter', 'sdxl', 'ip_adapter', { format: 'invokeai' }), + model('z-image-gguf', 'z-image', 'main', { format: 'gguf_quantized' }), + model('qwen3-06b', 'any', 'qwen3_encoder', { format: 'checkpoint', variant: 'qwen3_06b' }), + model('qwen3-4b', 'any', 'qwen3_encoder', { format: 'checkpoint', variant: 'qwen3_4b' }), + model('qwen3-8b', 'any', 'qwen3_encoder', { format: 'checkpoint', variant: 'qwen3_8b' }), + model('qwen3-4b-gguf', 'any', 'qwen3_encoder', { format: 'gguf_quantized', variant: 'qwen3_4b' }), + model('sdxl-vae', 'sdxl', 'vae', { format: 'diffusers' }), + model('flux-vae', 'flux', 'vae', { format: 'checkpoint' }), +]; + +const sdxlCase = (id: string, overrides: Record = {}) => ({ + height: 832, + id, + mode: 'txt2img', + model_key: 'sdxl', + negative_prompt: 'blurry', + positive_prompt: 'a lighthouse at dusk', + seed: 4242, + steps: 12, + width: 1216, + ...overrides, +}); + +const request = (cases: unknown[], models: unknown[] = MODELS) => ({ + capabilities: architectureCapabilitiesFixture, + cases, + models, + version: 1, +}); + +const exportOne = (testCase: Record, models?: unknown[]) => { + const [result] = exportGraphs(request([testCase], models)).cases; + + if (!result || !('batch' in result)) { + throw new Error(`expected a batch, got ${JSON.stringify(result)}`); + } + + return result; +}; + +const nodesOfType = (graph: BackendGraphContract, type: string): BackendInvocationContract[] => + Object.values(graph.nodes).filter((node) => node.type === type); + +const errorOf = (result: GraphExportCaseResult | undefined): string => + result && 'error' in result ? result.error : ''; + +describe('graph export', () => { + it('sends the case through Generate: seed and prompts as batch data, size and steps in the graph', () => { + const { batch, id } = exportOne( + sdxlCase('sdxl-t2i', { loras: [{ key: 'sdxl-lora', weight: 0.65 }], reference_images: ['reference.png'] }) + ); + const { graph } = batch; + const noise = nodesOfType(graph, 'noise')[0]; + const denoise = graph.nodes.denoise_latents; + + expect(id).toBe('sdxl-t2i'); + expect(batch.data).toEqual([ + [ + { field_name: 'value', items: [4242], node_path: 'seed' }, + { field_name: 'value', items: ['a lighthouse at dusk'], node_path: 'positive_prompt' }, + { field_name: 'value', items: ['blurry'], node_path: 'negative_prompt' }, + ], + ]); + expect(batch.data[0]?.map((datum) => graph.nodes[datum.node_path]?.type)).toEqual(['integer', 'string', 'string']); + expect(batch).toMatchObject({ destination: 'gallery', origin: 'graphs-export:sdxl-t2i' }); + expect(graph.id).toBe('sdxl-t2i'); + expect([noise?.width, noise?.height, denoise?.steps]).toEqual([1216, 832, 12]); + expect( + nodesOfType(graph, 'lora_selector').map((node) => [(node.lora as { key: string }).key, node.weight]) + ).toEqual([['sdxl-lora', 0.65]]); + expect(nodesOfType(graph, 'ip_adapter')).toMatchObject([ + { image: { image_name: 'reference.png' }, ip_adapter_model: { key: 'sdxl-ip-adapter' } }, + ]); + }); + + it('sends img2img through Canvas with the uploaded image and strength', () => { + const { graph } = exportOne( + sdxlCase('sdxl-i2i', { init_image: { image_name: 'init.png', strength: 0.3 }, mode: 'img2img' }) + ).batch; + + expect(nodesOfType(graph, 'i2l').map((node) => node.image)).toEqual([{ image_name: 'init.png' }]); + expect(graph.nodes.denoise_latents?.denoising_start).toBeCloseTo(0.7); + expect(nodesOfType(graph, 'core_metadata')[0]).toMatchObject({ generation_mode: 'sdxl_img2img', strength: 0.3 }); + }); + + it('adds each control entry as a Canvas control layer feeding the denoiser', () => { + const { graph } = exportOne( + sdxlCase('sdxl-control', { + control: [{ begin: 0.1, end: 0.6, image_name: 'canny.png', model_key: 'sdxl-canny', weight: 0.9 }], + mode: 'controlnet', + }) + ).batch; + const [controlNet] = nodesOfType(graph, 'controlnet'); + + expect(controlNet).toMatchObject({ + begin_step_percent: 0.1, + control_model: { key: 'sdxl-canny' }, + control_weight: 0.9, + end_step_percent: 0.6, + image: { image_name: 'canny.png' }, + }); + expect(graph.edges).toContainEqual({ + destination: { field: 'control', node_id: 'denoise_latents' }, + source: { field: 'collection', node_id: 'control_net_collector' }, + }); + }); + + it('fills required components from the installed models and keeps explicit picks', () => { + const zImage = { ...sdxlCase('z-image'), model_key: 'z-image-gguf' }; + + // The 0.6B encoder is listed first but Z-Image cannot use it. + expect(exportOne(zImage).components).toEqual({ qwen3EncoderModel: 'qwen3-4b', vae: 'flux-vae' }); + expect(exportOne({ ...zImage, components: { qwen3EncoderModel: 'qwen3-4b-gguf' } }).components).toEqual({ + qwen3EncoderModel: 'qwen3-4b-gguf', + vae: 'flux-vae', + }); + }); + + it('rejects an installed component the slot does not offer', () => { + const [result] = exportGraphs( + request([sdxlCase('z-image', { components: { qwen3EncoderModel: 'qwen3-8b' }, model_key: 'z-image-gguf' })]) + ).cases; + + expect(errorOf(result)).toBe( + 'components.qwen3EncoderModel: "qwen3-8b" cannot be the Qwen3 Encoder of z-image-gguf name. ' + + 'Choices: qwen3-4b, qwen3-4b-gguf.' + ); + }); + + it.each(generateGraphCases)('compiles $label from exactly the components it needs', ({ base, shape }) => { + const { + filled, + model: main, + settings, + } = withArchitectureCapabilities(architectureCapabilitiesFixture, () => satisfiedSettingsFor(base, shape)); + const components = Object.fromEntries(filled.map((key) => [key, (settings[key] as { key: string }).key])); + const testCase = sdxlCase(base, { height: settings.height, model_key: main.key, width: settings.width }); + + expect(exportOne(testCase, [main, ...filled.map((key) => settings[key])]).components).toEqual(components); + }); + + it('applies settings overrides and rejects values Generate would not keep', () => { + const { graph } = exportOne(sdxlCase('cfg', { settings: { cfgScale: 3.5, scheduler: 'dpmpp_2m' } })).batch; + const [badPid, badScheduler] = exportGraphs( + request([ + sdxlCase('pid', { settings: { pidMode: 'sideways' } }), + sdxlCase('scheduler', { settings: { scheduler: 'not-a-scheduler' } }), + ]) + ).cases; + + expect(graph.nodes.denoise_latents).toMatchObject({ cfg_scale: 3.5, scheduler: 'dpmpp_2m' }); + expect(errorOf(badPid)).toMatch(/^settings\.pidMode: Generate does not keep "sideways"/); + expect(errorOf(badScheduler)).toMatch(/^settings\.scheduler: .* Options: .*euler_a/); + }); + + it('reports unknown names per case, with the valid choices, and still exports the other cases', () => { + const response = exportGraphs( + request([ + sdxlCase('unknown-setting', { settings: { cfgscale: 3 } }), + sdxlCase('unknown-slot', { components: { t5EncoderModel: 'qwen3-4b' }, model_key: 'z-image-gguf' }), + sdxlCase('unknown-component', { components: { vae: 'missing-vae' }, model_key: 'z-image-gguf' }), + sdxlCase('unknown-model', { model_key: 'missing' }), + sdxlCase('unknown-lora', { loras: [{ key: 'missing-lora', weight: 1 }] }), + sdxlCase('unknown-control', { + control: [{ begin: 0, end: 1, image_name: 'canny.png', model_key: 'sdxl-lora', weight: 1 }], + mode: 'controlnet', + }), + sdxlCase('ok'), + ]) + ); + const errors = Object.fromEntries(response.cases.map((result) => [result.id, errorOf(result)])); + + expect(errors['unknown-setting']).toMatch(/^Unknown settings key "cfgscale"\. Valid keys: .*\bcfgScale\b/); + expect(errors['unknown-slot']).toMatch( + /^Unknown component slot "t5EncoderModel" .* Valid slots: .*qwen3EncoderModel, vae/ + ); + expect(errors['unknown-component']).toBe( + 'components.vae: model key "missing-vae" is not installed. Choices: flux-vae.' + ); + expect(errors['unknown-model']).toBe( + 'model_key "missing" is not a selectable main model. Choices: sdxl, z-image-gguf.' + ); + expect(errors['unknown-lora']).toMatch(/^loras\[0\]: "missing-lora" .* Choices: sdxl-lora\.$/); + expect(errors['unknown-control']).toMatch(/^control\[0\]: "sdxl-lora" cannot drive .* Choices: sdxl-canny\.$/); + expect(response.cases.find((result) => result.id === 'ok')).toHaveProperty('batch'); + }); + + it('exports identical requests identically, with stable node ids', () => { + const cases = [ + sdxlCase('t2i', { loras: [{ key: 'sdxl-lora', weight: 1 }], reference_images: ['reference.png'] }), + sdxlCase('i2i', { init_image: { image_name: 'init.png', strength: 0.5 }, mode: 'img2img' }), + { ...sdxlCase('z-image'), model_key: 'z-image-gguf' }, + ]; + const first = exportGraphs(request(cases)); + const nodeIds = first.cases.flatMap((result) => ('batch' in result ? Object.keys(result.batch.graph.nodes) : [])); + + expect(exportGraphs(request(cases))).toEqual(first); + expect(nodeIds.filter((id) => /_[a-z0-9]{7,9}_[a-z0-9]{4,6}$/.test(id))).toEqual([]); + }); + + it.each([ + ['a non-object request', []], + ['another version', { ...request([]), version: 2 }], + ['models that are not model configs', request([], [{ key: 'only-a-key' }])], + ['cases without ids', request([{ mode: 'txt2img' }])], + ['duplicate case ids', request([sdxlCase('same'), sdxlCase('same')])], + ])('rejects %s as a malformed request', (_label, malformed) => { + expect(() => exportGraphs(malformed)).toThrow(GraphExportRequestError); + }); +}); diff --git a/invokeai/frontend/webv2/src/app/graphExport.ts b/invokeai/frontend/webv2/src/app/graphExport.ts new file mode 100644 index 00000000000..951736f7525 --- /dev/null +++ b/invokeai/frontend/webv2/src/app/graphExport.ts @@ -0,0 +1,247 @@ +/** + * Composition root of `pnpm graphs:export` (scripts/export-graphs.mjs): installed model configs, the capability table + * and test cases become the enqueue batches the UI would send, offline, so end-to-end runs exercise the graphs users + * submit. Each batch is what Generate or Canvas puts into `enqueue_batch` minus `runs`, `project_id` and + * `idempotency_key`, with the case seed held fixed. Generation and Queue own every rule; this module maps the wire + * format and supplies the Workbench defaults the widgets would. + */ + +import type { GeneratePromptBatchPlan } from '@features/queue/contracts'; + +import { + compileGraphExportCase, + withArchitectureCapabilities, + type ArchitectureCapabilitiesRow, + type CompiledGraphExportCase, + type GraphExportCase, + type GraphExportEnvironment, + type GraphExportMode, +} from '@features/generation/graphExport'; +import { planGenerateSubmissionBatch } from '@features/queue/contracts'; +import { resolveRandDeviceMetadata } from '@features/queue/devices'; +import { SEED_MAX } from '@platform/core/seed'; +import { DEFAULT_CANVAS_COMPOSITING } from '@workbench/widgets/canvas/invoke/canvasCompositing'; +import { DEFAULT_CANVAS_SCALING } from '@workbench/widgets/canvas/invoke/canvasScaling'; +import { DEFAULT_CANVAS_DENOISING_STRENGTH } from '@workbench/widgets/canvas/invoke/canvasStrength'; + +const GRAPH_EXPORT_VERSION = 1; + +/** Durable outputs: Canvas staging would mark them intermediate. */ +const DESTINATION = 'gallery'; +/** Not a webv2 origin, so no open editor adopts the items as its own submissions. */ +const ORIGIN_PREFIX = 'graphs-export:'; +/** `DEFAULT_PROJECT_SETTINGS` (workbench/settings/store.ts); importing it would load the whole settings runtime. */ +const PROJECT_SETTINGS = { useCpuNoise: true, useNsfwChecker: false, useWatermark: false }; +/** ROCm builds of torch report their GPUs as CUDA devices as well. */ +const GPU_DEVICE_OPTIONS = [{ device: 'cuda:0', name: 'GPU' }]; +const MODES: readonly GraphExportMode[] = ['txt2img', 'img2img', 'controlnet']; +const CASE_FIELDS = [ + 'components', + 'control', + 'height', + 'id', + 'init_image', + 'loras', + 'mode', + 'model_key', + 'negative_prompt', + 'positive_prompt', + 'reference_images', + 'seed', + 'settings', + 'steps', + 'width', +]; + +/** The request itself is unusable; no response is written. */ +export class GraphExportRequestError extends Error { + override name = 'GraphExportRequestError'; +} + +export interface GraphExportBatch { + data: GeneratePromptBatchPlan['data']; + destination: typeof DESTINATION; + graph: CompiledGraphExportCase['graph']; + origin: string; +} + +export type GraphExportCaseResult = + | { id: string; batch: GraphExportBatch; components: Record } + | { id: string; error: string }; + +export interface GraphExportResponse { + version: typeof GRAPH_EXPORT_VERSION; + cases: GraphExportCaseResult[]; +} + +type JsonRecord = Record; + +const isRecord = (value: unknown): value is JsonRecord => + typeof value === 'object' && value !== null && !Array.isArray(value); + +const fail = (message: string): never => { + throw new Error(message); +}; + +const readString = (record: JsonRecord, field: string, label = field): string => + typeof record[field] === 'string' ? record[field] : fail(`${label} must be a string.`); + +const readNumber = (record: JsonRecord, field: string, label = field): number => + typeof record[field] === 'number' && Number.isFinite(record[field]) + ? record[field] + : fail(`${label} must be a number.`); + +const readInteger = (record: JsonRecord, field: string, min: number, max = Number.MAX_SAFE_INTEGER): number => { + const value = record[field]; + + return typeof value === 'number' && Number.isInteger(value) && value >= min && value <= max + ? value + : fail(`${field} must be an integer from ${min} to ${max}.`); +}; + +/** Absent and null both read as empty. */ +const readList = (record: JsonRecord, field: string, read: (item: unknown, label: string) => T): T[] => { + const value = record[field] ?? []; + + return Array.isArray(value) + ? value.map((item, index) => read(item, `${field}[${index}]`)) + : fail(`${field} must be a list.`); +}; + +const readObject = (value: unknown, label: string): JsonRecord => + isRecord(value) ? value : fail(`${label} must be an object.`); + +const readCase = (record: JsonRecord): GraphExportCase => { + const unknownField = Object.keys(record).find((field) => !CASE_FIELDS.includes(field)); + + if (unknownField !== undefined) { + fail(`Unknown case field "${unknownField}". Valid fields: ${CASE_FIELDS.join(', ')}.`); + } + + const mode = record.mode; + const components = readObject(record.components ?? {}, 'components'); + const initImage = + record.init_image === undefined || record.init_image === null ? null : readObject(record.init_image, 'init_image'); + + return { + components: Object.fromEntries( + Object.keys(components).map((slot) => [slot, readString(components, slot, `components.${slot}`)]) + ), + control: readList(record, 'control', (item, label) => { + const layer = readObject(item, label); + + return { + begin: readNumber(layer, 'begin', `${label}.begin`), + end: readNumber(layer, 'end', `${label}.end`), + imageName: readString(layer, 'image_name', `${label}.image_name`), + modelKey: readString(layer, 'model_key', `${label}.model_key`), + weight: readNumber(layer, 'weight', `${label}.weight`), + }; + }), + height: readInteger(record, 'height', 1), + id: readString(record, 'id'), + initImage: initImage && { + imageName: readString(initImage, 'image_name', 'init_image.image_name'), + strength: readNumber(initImage, 'strength', 'init_image.strength'), + }, + loras: readList(record, 'loras', (item, label) => { + const lora = readObject(item, label); + + return { key: readString(lora, 'key', `${label}.key`), weight: readNumber(lora, 'weight', `${label}.weight`) }; + }), + mode: MODES.includes(mode as GraphExportMode) + ? (mode as GraphExportMode) + : fail(`mode must be one of ${MODES.join(', ')}.`), + modelKey: readString(record, 'model_key'), + negativePrompt: + record.negative_prompt === null || record.negative_prompt === undefined + ? '' + : readString(record, 'negative_prompt'), + positivePrompt: readString(record, 'positive_prompt'), + referenceImages: readList(record, 'reference_images', (item, label) => + typeof item === 'string' ? item : fail(`${label} must be an image name.`) + ), + seed: readInteger(record, 'seed', 0, SEED_MAX), + settings: readObject(record.settings ?? {}, 'settings'), + steps: readInteger(record, 'steps', 1), + width: readInteger(record, 'width', 1), + }; +}; + +const exportCase = (record: JsonRecord, env: GraphExportEnvironment): GraphExportCaseResult => { + const id = record.id as string; + + try { + const { components, graph, submission } = compileGraphExportCase(readCase(record), env); + const batch: GraphExportBatch = { + data: planGenerateSubmissionBatch(submission).data, + destination: DESTINATION, + graph, + origin: `${ORIGIN_PREFIX}${id}`, + }; + + return { batch, components, id }; + } catch (error) { + return { error: error instanceof Error ? error.message : String(error), id }; + } +}; + +const isModelConfig = (value: unknown): value is GraphExportEnvironment['models'][number] => + isRecord(value) && ['key', 'name', 'base', 'type'].every((field) => typeof value[field] === 'string'); + +const isCapabilitiesRow = (value: unknown): value is ArchitectureCapabilitiesRow => + isRecord(value) && typeof value.base === 'string' && isRecord(value.features); + +const readRequest = (request: unknown) => { + if (!isRecord(request)) { + throw new GraphExportRequestError('The request must be a JSON object.'); + } + + if (request.version !== GRAPH_EXPORT_VERSION) { + throw new GraphExportRequestError(`Unsupported request version ${JSON.stringify(request.version)}; expected 1.`); + } + + const { capabilities, cases, models } = request; + + if (!Array.isArray(models) || !models.every(isModelConfig)) { + throw new GraphExportRequestError('models must be the `models` list of GET /api/v2/models/.'); + } + + if (!Array.isArray(capabilities) || !capabilities.every(isCapabilitiesRow)) { + throw new GraphExportRequestError('capabilities must be the body of GET /api/v2/models/capabilities.'); + } + + if (!Array.isArray(cases) || !cases.every((item) => isRecord(item) && typeof item.id === 'string' && item.id)) { + throw new GraphExportRequestError('cases must be a list of objects with a non-empty string id.'); + } + + const ids = cases.map((item) => item.id as string); + const duplicate = ids.find((id, index) => ids.indexOf(id) !== index); + + if (duplicate !== undefined) { + throw new GraphExportRequestError(`Case id "${duplicate}" is not unique.`); + } + + return { capabilities, cases: cases as JsonRecord[], models }; +}; + +/** Throws GraphExportRequestError for a malformed request; a case that cannot compile carries its error instead. */ +export const exportGraphs = (request: unknown): GraphExportResponse => { + const { capabilities, cases, models } = readRequest(request); + const env: GraphExportEnvironment = { + canvas: { + compositing: DEFAULT_CANVAS_COMPOSITING, + scaling: DEFAULT_CANVAS_SCALING, + strength: DEFAULT_CANVAS_DENOISING_STRENGTH, + }, + destination: DESTINATION, + models, + projectSettings: PROJECT_SETTINGS, + randDevice: resolveRandDeviceMetadata(PROJECT_SETTINGS.useCpuNoise, GPU_DEVICE_OPTIONS), + }; + + return withArchitectureCapabilities(capabilities, () => ({ + cases: cases.map((record) => exportCase(record, env)), + version: GRAPH_EXPORT_VERSION, + })); +}; diff --git a/invokeai/frontend/webv2/src/app/router.tsx b/invokeai/frontend/webv2/src/app/router.tsx index 2d2b4acd503..53860a20ccc 100644 --- a/invokeai/frontend/webv2/src/app/router.tsx +++ b/invokeai/frontend/webv2/src/app/router.tsx @@ -9,7 +9,7 @@ import { useAuthSession, type Capabilities, } from '@features/identity'; -import { ModelInstallRuntime } from '@features/models'; +import { CalibrationGraphCompilerProvider, ModelInstallRuntime, type CalibrationGraphCompiler } from '@features/models'; import { createLogger } from '@platform/logging/logger'; import { createHashHistory, @@ -118,8 +118,18 @@ const requireLaunchpadCapability = async (capability: keyof Capabilities): Promi } }; +/** Loads Generate's compiler only when the Model Manager starts a working-memory calibration. */ +const compileCalibrationGraphs: CalibrationGraphCompiler = (cases, signal) => + import('./calibrationGraphs').then((module) => module.compileCalibrationGraphs(cases, signal)); + +const LaunchpadRoute = () => ( + + + +); + const launchpadRouteOptions = { - component: Launchpad, + component: LaunchpadRoute, getParentRoute: () => authenticatedRoute, }; diff --git a/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts b/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts index 812832d1ce7..60e477c547b 100644 --- a/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts +++ b/invokeai/frontend/webv2/src/architecture/dependencyPolicy.test.ts @@ -235,6 +235,7 @@ describe('feature public-interface registry', () => { ]); expect(checkDependency('workbench/invocationSubmit.ts', '@features/generation/prompts')).toEqual([]); expect(checkDependency('app/GenerateWidgetSyncRuntime.tsx', '@features/generation/runtime')).toEqual([]); + expect(checkDependency('app/graphExport.ts', '@features/generation/graphExport')).toEqual([]); expect(checkDependency('workbench/widgets/canvas/invoke.ts', '@features/generation/canvasGraph')).toEqual([]); expect(checkDependency('workbench/WorkbenchContext.tsx', '@features/intermediates/holdLease')).toEqual([]); expect( diff --git a/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts b/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts index 9405bcfd449..c5fb3a9b00d 100644 --- a/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts +++ b/invokeai/frontend/webv2/src/architecture/featureInterfaces.ts @@ -22,6 +22,7 @@ export const FEATURE_PUBLIC_INTERFACES: Readonly ({ +export const getComponentPolicyContext = ( + model: GenerateModelConfig, + settings: GenerateSettings +): ComponentPolicyContext => ({ model, settings, selectedComponents: { @@ -1235,6 +1238,9 @@ const COMPONENT_SETTING_LABELS: Record = { componentSourceModel: 'Component source', }; +/** Every settings key that holds a component model. */ +export const GENERATE_COMPONENT_VALUE_KEYS = Object.keys(COMPONENT_SETTING_LABELS) as GenerateComponentValueKey[]; + const addClearedLabel = (labels: string[], label: string) => { if (!labels.includes(label)) { labels.push(label); @@ -1509,7 +1515,7 @@ const getSettingsWithCompatibleModelSelections = ( const slotsByKey = new Map(policy.slots.map((slotPolicy) => [slotPolicy.key, slotPolicy])); const uiPolicy = getGenerationUiPolicy(model); - for (const key of Object.keys(COMPONENT_SETTING_LABELS) as GenerateComponentValueKey[]) { + for (const key of GENERATE_COMPONENT_VALUE_KEYS) { if (!nextSettings[key]) { continue; } @@ -1619,7 +1625,7 @@ export const getGenerationModelAvailabilityReasons = ( reasons.push(`Selected model "${model.name}" is no longer installed.`); } - for (const key of Object.keys(COMPONENT_SETTING_LABELS) as GenerateComponentValueKey[]) { + for (const key of GENERATE_COMPONENT_VALUE_KEYS) { const value = settings[key]; if (value && !hasModelKey(models, value.key, value.type)) { diff --git a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts index 38d2ee38047..f1a513880ed 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.test.ts @@ -6,9 +6,11 @@ import { hasDynamicPromptSyntax, matchesKnownWildcard, normalizeWildcardValues, + planGenerateSubmission, scanWildcardReferences, sanitizeDynamicPromptsConfig, sanitizeMaxPrompts, + type GenerateSubmissionSource, } from './dynamicPrompts'; describe('hasDynamicPromptSyntax', () => { @@ -239,3 +241,70 @@ describe('getWildcardValuesError', () => { expect(getWildcardValuesError(['x'.repeat(2_000)])).toBeNull(); }); }); + +const source = (overrides: Partial = {}): GenerateSubmissionSource => ({ + batchCount: 2, + negativePrompt: 'blurry', + negativePromptEnabled: true, + positivePrompt: 'a cat', + promptTemplate: null, + seed: 10, + seedMode: 'increment', + ...overrides, +}); + +describe('planGenerateSubmission', () => { + it('merges the template and suppresses its negative side with the negative prompt', () => { + const promptTemplate = { + id: 't', + name: 'Photo', + negativePrompt: '{prompt}, lowres', + positivePrompt: 'photo of {prompt}', + }; + + expect(planGenerateSubmission(source({ promptTemplate })).submission).toMatchObject({ + negativePrompt: 'blurry, lowres', + positivePrompt: 'photo of a cat', + }); + + const suppressed = planGenerateSubmission(source({ negativePromptEnabled: false, promptTemplate })); + + expect(suppressed.submission.negativePrompt).toBe(''); + expect(suppressed.effectivePrompts.negativePrompt).toBe('blurry, lowres'); + }); + + it('sends an expansion and its seed behaviour only for a prompt with dynamic syntax', () => { + const expansion = ['a red cat', 'a green cat']; + const expanded = planGenerateSubmission( + source({ dynamicPromptsSeedBehaviour: 'per-image', positivePrompt: 'a {red|green} cat' }), + expansion + ); + const plain = planGenerateSubmission(source({ dynamicPromptsSeedBehaviour: 'per-image' }), expansion); + + expect(expanded.submission).toEqual({ + batchCount: 2, + negativePrompt: 'blurry', + positivePrompt: 'a {red|green} cat', + positivePrompts: expansion, + seed: 10, + seedBehaviour: 'per-image', + seedStep: 1, + }); + // Two prompts per image across two runs consume four seeds. + expect(expanded.seedPlan.nextSeed).toBe(14); + expect(plain.submission).toEqual({ + batchCount: 2, + negativePrompt: 'blurry', + positivePrompt: 'a cat', + seed: 10, + seedStep: 1, + }); + expect(plain.seedPlan.nextSeed).toBe(12); + }); + + it('holds a fixed seed', () => { + const { seedPlan, submission } = planGenerateSubmission(source({ seedMode: 'fixed' })); + + expect([submission.seed, submission.seedStep, seedPlan.nextSeed]).toEqual([10, 0, null]); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts index 3512f3c45e7..52967295237 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/dynamicPrompts.ts @@ -1,3 +1,9 @@ +import { planSeedSubmission, type SeedMode, type SeedStep, type SeedSubmissionPlan } from '@platform/core/seed'; + +import type { EffectivePrompts, PromptTemplateSnapshot } from './promptTemplates'; + +import { getEffectivePrompts } from './promptTemplates'; + const SEED_MAX = 4_294_967_295; export const DYNAMIC_PROMPTS_MIN_PROMPTS = 1; @@ -302,3 +308,75 @@ export const sanitizeSampleSeed = (value: unknown): number => typeof value === 'number' && Number.isFinite(value) ? Math.abs(Math.round(value)) % SEED_MAX : 0; export const createDynamicPromptsSampleSeed = (): number => Math.floor(Math.random() * SEED_MAX); + +/** The prompted, seeded values Generate, Canvas, Upscale and Video submit from. */ +export interface GenerateSubmissionSource { + batchCount: number; + negativePrompt: string; + negativePromptEnabled: boolean; + positivePrompt: string; + promptTemplate?: PromptTemplateSnapshot | null; + seed: number; + seedMode: SeedMode; + dynamicPromptsSeedBehaviour?: DynamicPromptsSeedBehaviour; +} + +/** The prompt and seed fields of a queued generate submission; the host adds the graph and its node ids. */ +export interface GenerateSubmissionPrompts { + batchCount: number; + negativePrompt: string; + positivePrompt: string; + /** Present when the positive prompt expanded; the expansion is sent instead of the prompt. */ + positivePrompts?: string[]; + seed: number; + seedBehaviour?: DynamicPromptsSeedBehaviour; + seedStep: SeedStep; +} + +export interface GenerateSubmissionPlan { + /** Template-merged prompts, before negative-prompt suppression. */ + effectivePrompts: EffectivePrompts; + seedPlan: SeedSubmissionPlan; + submission: GenerateSubmissionPrompts; +} + +/** + * The prompts and seeds a submission sends. `expandedPositivePrompts` are what the host expanded the merged positive + * prompt into; they count only when that prompt has dynamic syntax, so templates that introduce syntax count too. + */ +export const planGenerateSubmission = ( + source: GenerateSubmissionSource, + expandedPositivePrompts?: string[] +): GenerateSubmissionPlan => { + const effectivePrompts = getEffectivePrompts(source); + // Record expansions even when only one prompt results. + const positivePrompts = + expandedPositivePrompts && + expandedPositivePrompts.length > 0 && + hasDynamicPromptSyntax(effectivePrompts.positivePrompt) + ? expandedPositivePrompts + : undefined; + const seedBehaviour = positivePrompts ? source.dynamicPromptsSeedBehaviour : undefined; + const seedPlan = planSeedSubmission({ + batchCount: source.batchCount, + promptCount: positivePrompts?.length ?? 1, + seedBehaviour: seedBehaviour ?? 'per-iteration', + seedMode: source.seedMode, + startSeed: source.seed, + }); + + return { + effectivePrompts, + seedPlan, + submission: { + batchCount: source.batchCount, + // Disabling the negative prompt also suppresses the template's negative side. + negativePrompt: source.negativePromptEnabled ? effectivePrompts.negativePrompt : '', + positivePrompt: effectivePrompts.positivePrompt, + ...(positivePrompts ? { positivePrompts } : {}), + seed: source.seed, + ...(seedBehaviour ? { seedBehaviour } : {}), + seedStep: seedPlan.step, + }, + }; +}; diff --git a/invokeai/frontend/webv2/src/features/generation/core/graphExport.ts b/invokeai/frontend/webv2/src/features/generation/core/graphExport.ts new file mode 100644 index 00000000000..72ae821403a --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/core/graphExport.ts @@ -0,0 +1,492 @@ +/** + * Offline export cases compile to the graph Generate or Canvas would submit: the model's defaults, the case's values + * and component picks, required components filled from the installed models the way a user fills them, and the same + * reconciliation and compilers the widget runs. Callers own the wire format and the Workbench canvas defaults. + */ + +import type { GenerationModelCatalogItem as ModelConfig } from '@features/generation/core/contracts'; + +import { areJsonValuesStructurallyEqual } from '@platform/core/json'; + +import type { ArchitectureCapabilitiesRow } from './architectureCapabilities'; +import type { ComponentPolicyContext, ComponentSlotPolicy } from './baseGenerationPolicies'; +import type { ControlLayerGraphInput } from './canvas/addControlLayers'; +import type { CanvasCompositingSettings, CanvasScalingSettings } from './canvas/types'; +import type { BackendGraphContract, GenerationProjectSettings, ResultDestination } from './contracts'; +import type { ComponentModelConfig, GenerateModelConfig, GenerateWidgetValues } from './types'; + +import { resetArchitectureCapabilities, setArchitectureCapabilities } from './architectureCapabilities'; +import { + coerceSchedulerForGraph, + GENERATE_COMPONENT_VALUE_KEYS, + getComponentPolicyContext, + getComponentSectionPolicy, + getDefaultGenerateSettings, + getDefaultReferenceImageConfig, + getGenerateModelSelectionResult, + getSchedulerOptions, + isArchitectureDescribed, + isGenerateModelSelectable, + isLoraSupported, +} from './baseGenerationPolicies'; +import { compileCanvasGraph } from './canvas/compileCanvasGraph'; +import { + CONTROL_ADAPTER_KINDS, + isControlKindSupportedForBase, + isControlModelUsableForKind, +} from './canvas/controlValidation'; +import { hasDynamicPromptSyntax, planGenerateSubmission, type GenerateSubmissionPrompts } from './dynamicPrompts'; +import { compileGenerateGraph } from './graph'; +import { stabilizeBackendGraphNodeIds } from './previewGraph'; +import { resolveGenerateWidgetValues } from './resolveGenerateWidgetValues'; +import { + deriveAspectRatioId, + GENERATE_UI_STATE_KEYS, + getModelDefaultVae, + hasModelDefaultVae, + isLoraCompatibleWithModel, + isLoraModelConfig, + isMainModelConfig, + isModelIdentifierConfig, + isVaeModelConfig, +} from './settings'; + +/** `txt2img` compiles like Generate; `img2img` and `controlnet` compile like a full-frame Canvas invoke. */ +export type GraphExportMode = 'txt2img' | 'img2img' | 'controlnet'; + +export interface GraphExportCase { + /** Becomes the graph id. */ + id: string; + mode: GraphExportMode; + modelKey: string; + /** Picks keyed by component slot; required slots left out are filled from the installed models. */ + components: Readonly>; + width: number; + height: number; + steps: number; + seed: number; + positivePrompt: string; + negativePrompt: string; + loras: readonly { key: string; weight: number }[]; + /** Uploaded image names. */ + referenceImages: readonly string[]; + /** An uploaded image already sized width × height, as the Canvas composite always is. */ + initImage: { imageName: string; strength: number } | null; + /** Each image is an uploaded width × height control composite. */ + control: readonly { modelKey: string; imageName: string; weight: number; begin: number; end: number }[]; + /** Generate settings overrides. Keys the case sets through its own fields are rejected. */ + settings: Readonly>; +} + +export interface GraphExportEnvironment { + models: readonly ModelConfig[]; + destination: ResultDestination; + projectSettings: GenerationProjectSettings; + randDevice: string; + /** What the Canvas widget supplies from its persisted values. */ + canvas: { compositing: CanvasCompositingSettings; scaling: CanvasScalingSettings; strength: number }; +} + +export interface CompiledGraphExportCase { + /** Node ids are stabilized: identical cases compile to identical graphs. */ + graph: BackendGraphContract; + /** The key of every component model the case compiled with, by settings key. */ + components: Record; + /** What Generate queues for the graph, with the prompt and seed node ids in `graph`. */ + submission: GenerateSubmissionPrompts & { + negativePromptNodeId: string; + positivePromptNodeId: string; + seedNodeId: string; + }; +} + +/** Keys a case sets through its own fields, picks as components, or that need backend catalogs or expansion. */ +const NON_OVERRIDABLE_SETTINGS: ReadonlySet = new Set([ + 'aspectRatioId', + 'aspectRatioValue', + 'dynamicPromptsCombinatorial', + 'dynamicPromptsMaxPrompts', + 'dynamicPromptsResample', + 'dynamicPromptsSampleSeed', + 'dynamicPromptsSeedBehaviour', + 'height', + 'loras', + 'model', + 'modelKey', + 'negativePrompt', + 'positivePrompt', + 'promptTemplate', + 'referenceImages', + 'seed', + 'seedMode', + 'steps', + 'width', + ...GENERATE_COMPONENT_VALUE_KEYS, + ...Object.keys(GENERATE_UI_STATE_KEYS), +]); + +const listKeys = (keys: readonly string[]): string => (keys.length > 0 ? keys.join(', ') : 'none'); + +const listModelKeys = (models: readonly ModelConfig[]): string => listKeys(models.map((model) => model.key)); + +/** Hold `rows` as the capability table, which every Generation policy reads, while `run` compiles. */ +export const withArchitectureCapabilities = (rows: readonly ArchitectureCapabilitiesRow[], run: () => T): T => { + setArchitectureCapabilities(rows); + + try { + return run(); + } finally { + resetArchitectureCapabilities(); + } +}; + +const getSelectableModel = (key: string, models: readonly ModelConfig[]): GenerateModelConfig => { + const selectable = models.filter(isGenerateModelSelectable).filter(isArchitectureDescribed); + const model = selectable.find((candidate) => candidate.key === key); + + if (!model) { + throw new Error(`model_key "${key}" is not a selectable main model. Choices: ${listModelKeys(selectable)}.`); + } + + return model; +}; + +/** Selecting the model, then "Reset to model defaults", which also applies the model's default VAE. */ +const getModelDefaultValues = (model: GenerateModelConfig, models: readonly ModelConfig[]): GenerateWidgetValues => { + const { settings } = getGenerateModelSelectionResult({ + currentValues: getDefaultGenerateSettings(model), + model, + models, + }); + const vae = hasModelDefaultVae(model) ? getModelDefaultVae(model, models.filter(isVaeModelConfig)) : settings.vae; + + return { ...settings, model, vae }; +}; + +const withCaseValues = (values: GenerateWidgetValues, input: GraphExportCase): GenerateWidgetValues => ({ + ...values, + aspectRatioId: deriveAspectRatioId(input.width, input.height), + aspectRatioValue: input.width / input.height, + height: input.height, + negativePrompt: input.negativePrompt, + positivePrompt: input.positivePrompt, + seed: input.seed, + seedMode: 'fixed', + steps: input.steps, + width: input.width, +}); + +const withSettingsOverrides = ( + values: GenerateWidgetValues, + overrides: GraphExportCase['settings'] +): GenerateWidgetValues => { + const validKeys = Object.keys(values) + .filter((key) => !NON_OVERRIDABLE_SETTINGS.has(key)) + .sort(); + const unknownKey = Object.keys(overrides).find((key) => !validKeys.includes(key)); + + if (unknownKey !== undefined) { + throw new Error(`Unknown settings key "${unknownKey}". Valid keys: ${listKeys(validKeys)}.`); + } + + return { ...values, ...overrides }; +}; + +const fitsSlot = (slot: ComponentSlotPolicy, candidate: ModelConfig, ctx: ComponentPolicyContext): boolean => + (slot.modelTypes as readonly string[]).includes(candidate.type) && (!slot.filter || slot.filter(candidate, ctx)); + +/** The value the component picker commits for `model`, by the slot's value kind. */ +const toSlotValue = (slot: ComponentSlotPolicy, model: ModelConfig): ComponentModelConfig | null => { + switch (slot.valueKind) { + case 'main': + return isMainModelConfig(model) ? model : null; + case 'vae': + return isVaeModelConfig(model) ? model : null; + case 'component': + return isModelIdentifierConfig(model) ? model : null; + } +}; + +const withComponentPicks = ( + values: GenerateWidgetValues, + picks: GraphExportCase['components'], + models: readonly ModelConfig[] +): GenerateWidgetValues => { + const { slots } = getComponentSectionPolicy(values.model, values); + let next = values; + + for (const [slotKey, modelKey] of Object.entries(picks)) { + const slot = slots.find((candidate) => candidate.key === slotKey); + + if (!slot) { + const validSlots = listKeys(slots.map((candidate) => candidate.key)); + + throw new Error(`Unknown component slot "${slotKey}" for ${values.model.name}. Valid slots: ${validSlots}.`); + } + + const ctx = getComponentPolicyContext(next.model, next); + const fitting = models.filter((candidate) => fitsSlot(slot, candidate, ctx)); + const pick = fitting.find((candidate) => candidate.key === modelKey); + const value = pick ? toSlotValue(slot, pick) : null; + + if (!value) { + const problem = models.some((candidate) => candidate.key === modelKey) + ? `"${modelKey}" cannot be the ${slot.label} of ${values.model.name}` + : `model key "${modelKey}" is not installed`; + + throw new Error(`components.${slotKey}: ${problem}. Choices: ${listModelKeys(fitting)}.`); + } + + next = { ...next, [slot.key]: value }; + } + + return next; +}; + +/** Pick the first installed model for the first required slot that is unfilled; null when none can be filled. */ +const withFirstRequiredComponent = ( + values: GenerateWidgetValues, + models: readonly ModelConfig[] +): GenerateWidgetValues | null => { + const ctx = getComponentPolicyContext(values.model, values); + + for (const slot of getComponentSectionPolicy(values.model, values).slots) { + const current = ctx.selectedComponents[slot.key]; + + if (!slot.required?.(ctx) || (current && (!slot.filter || slot.filter(current as ModelConfig, ctx)))) { + continue; + } + + for (const candidate of models) { + const value = fitsSlot(slot, candidate, ctx) ? toSlotValue(slot, candidate) : null; + + if (value) { + return { ...values, [slot.key]: value }; + } + } + } + + return null; +}; + +const withLoras = ( + values: GenerateWidgetValues, + loras: GraphExportCase['loras'], + models: readonly ModelConfig[] +): GenerateWidgetValues => { + if (loras.length === 0) { + return values; + } + + const { model } = values; + + if (!isLoraSupported(model)) { + throw new Error(`${model.name} does not load LoRAs.`); + } + + const compatible = models.filter(isLoraModelConfig).filter((lora) => isLoraCompatibleWithModel(lora, model)); + + return { + ...values, + loras: loras.map(({ key, weight }, index) => { + const lora = compatible.find((candidate) => candidate.key === key); + + if (!lora) { + const choices = listModelKeys(compatible); + + throw new Error(`loras[${index}]: "${key}" is not an installed LoRA for ${model.name}. Choices: ${choices}.`); + } + + return { isEnabled: true, model: lora, weight }; + }), + }; +}; + +const withReferenceImages = ( + values: GenerateWidgetValues, + input: GraphExportCase, + models: readonly ModelConfig[] +): GenerateWidgetValues => ({ + ...values, + referenceImages: input.referenceImages.map((imageName, index) => ({ + // Cases name images only; the output size stands in for dimensions that reach metadata, not inference. + config: getDefaultReferenceImageConfig(values.model, models, { + original: { image: { height: input.height, image_name: imageName, width: input.width } }, + }), + id: `reference_image_${index + 1}`, + isEnabled: true, + })), +}); + +/** The widget reconciles after every commit; reconciliation can pick a FLUX.2 component source. */ +const reconcile = (values: GenerateWidgetValues, models: readonly ModelConfig[]): GenerateWidgetValues => { + const resolved = resolveGenerateWidgetValues({ models, storedValues: values }); + + if (!resolved) { + throw new Error('Generate cannot resolve settings without the capability table.'); + } + + return resolved.values; +}; + +/** Report a requested value the widget would not keep instead of compiling a graph nobody asked for. */ +const assertRequestedValuesKept = (values: GenerateWidgetValues, input: GraphExportCase): void => { + for (const [key, requested] of Object.entries(input.settings)) { + const kept = values[key as keyof GenerateWidgetValues]; + + if (!areJsonValuesStructurallyEqual(kept, requested)) { + throw new Error( + `settings.${key}: Generate does not keep ${JSON.stringify(requested)}; it holds ${JSON.stringify(kept)}.` + ); + } + } + + if ('scheduler' in input.settings && coerceSchedulerForGraph(values.model, values.scheduler) !== values.scheduler) { + const options = getSchedulerOptions(values.model).map((option) => option.value); + + throw new Error( + `settings.scheduler: ${values.model.name} does not use "${values.scheduler}". Options: ${listKeys(options)}.` + ); + } + + for (const [key, requested] of Object.entries(input.components)) { + const kept = values[key as keyof GenerateWidgetValues] as ComponentModelConfig | null; + + if (kept?.key !== requested) { + throw new Error( + `components.${key}: Generate replaces "${requested}" with ${kept ? `"${kept.key}"` : 'nothing'}.` + ); + } + } +}; + +const getControlLayers = ( + input: GraphExportCase, + model: GenerateModelConfig, + models: readonly ModelConfig[] +): ControlLayerGraphInput[] => { + // A layer's kind is the one its model can drive on this base, the kind a user switches the layer to. + const getKind = (candidate: ModelConfig) => + candidate.base === model.base + ? CONTROL_ADAPTER_KINDS.find( + (kind) => isControlKindSupportedForBase(model.base, kind) && isControlModelUsableForKind(candidate, kind) + ) + : undefined; + + return input.control.map((layer, index) => { + const adapter = models.find((candidate) => candidate.key === layer.modelKey); + const kind = adapter ? getKind(adapter) : undefined; + + if (!adapter || !kind) { + const choices = listModelKeys(models.filter((candidate) => getKind(candidate) !== undefined)); + const problem = `"${layer.modelKey}" cannot drive a control layer of ${model.name}`; + + throw new Error(`control[${index}]: ${problem}. Choices: ${choices}.`); + } + + return { + beginEndStepPct: [layer.begin, layer.end], + // A new ControlNet layer starts "balanced", which the compiler sends for null; other kinds carry no mode. + controlMode: null, + id: `control_${index + 1}`, + imageName: layer.imageName, + kind, + model: { + base: adapter.base, + key: adapter.key, + name: adapter.name, + type: adapter.type, + ...(typeof adapter.hash === 'string' ? { hash: adapter.hash } : {}), + }, + modelCondInChannels: typeof adapter.cond_in_channels === 'number' ? adapter.cond_in_channels : null, + weight: layer.weight, + }; + }); +}; + +const assertModeInputs = (input: GraphExportCase): void => { + if (input.mode === 'txt2img' && (input.initImage || input.control.length > 0)) { + throw new Error('txt2img cases take no init_image or control; use img2img or controlnet.'); + } + + if (input.mode === 'img2img' && !input.initImage) { + throw new Error('img2img cases need init_image.'); + } + + if (input.mode === 'controlnet' && input.control.length === 0) { + throw new Error('controlnet cases need at least one control entry.'); + } +}; + +/** Throws a message naming the offending field and its valid choices when the case cannot compile. */ +export const compileGraphExportCase = ( + input: GraphExportCase, + env: GraphExportEnvironment +): CompiledGraphExportCase => { + assertModeInputs(input); + + const { models } = env; + let values = withCaseValues(getModelDefaultValues(getSelectableModel(input.modelKey, models), models), input); + values = withSettingsOverrides(values, input.settings); + values = withComponentPicks(values, input.components, models); + values = withLoras(values, input.loras, models); + values = reconcile(withReferenceImages(values, input, models), models); + + // Filling one slot can satisfy or require another; the bound stops a reconciliation that keeps undoing a pick. + for (let pass = 0; pass < GENERATE_COMPONENT_VALUE_KEYS.length; pass += 1) { + const filled = withFirstRequiredComponent(values, models); + + if (!filled) { + break; + } + + values = reconcile(filled, models); + } + + assertRequestedValuesKept(values, input); + + // No expansion: the UI expands dynamic prompts through the backend before it queues. + const { effectivePrompts, submission } = planGenerateSubmission(values); + + if (hasDynamicPromptSyntax(effectivePrompts.positivePrompt)) { + throw new Error( + 'positive_prompt has dynamic prompt syntax, which only the backend expands; export each expansion.' + ); + } + + const compiled = + input.mode === 'txt2img' + ? compileGenerateGraph(values, values.model, env.destination, env.projectSettings, env.randDevice) + : compileCanvasGraph({ + bbox: { height: input.height, width: input.width, x: 0, y: 0 }, + compositeImageName: input.initImage?.imageName ?? null, + compositing: env.canvas.compositing, + controlLayers: getControlLayers(input, values.model, models), + destination: env.destination, + mode: input.initImage ? 'img2img' : 'txt2img', + model: values.model, + projectSettings: env.projectSettings, + randDevice: env.randDevice, + scaling: env.canvas.scaling, + settings: values, + strength: input.initImage?.strength ?? env.canvas.strength, + }); + + const stable = stabilizeBackendGraphNodeIds(compiled.backendGraph, { + negativePromptNodeId: compiled.negativePromptNodeId, + positivePromptNodeId: compiled.positivePromptNodeId, + seedNodeId: compiled.seedNodeId, + }); + + return { + components: Object.fromEntries( + GENERATE_COMPONENT_VALUE_KEYS.flatMap((key) => { + const component = values[key]; + + return component ? [[key, component.key]] : []; + }) + ), + graph: { ...stable.graph, id: input.id }, + submission: { ...submission, ...stable.nodeIds }, + }; +}; diff --git a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts index 8858477edb8..026722eab6d 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.test.ts @@ -10,7 +10,7 @@ import { setArchitectureCapabilities, } from './architectureCapabilities'; import { getDefaultGenerateSettings } from './baseGenerationPolicies'; -import { compileGeneratePreviewGraph, stabilizeBackendGraphIds } from './previewGraph'; +import { compileGeneratePreviewGraph, stabilizeBackendGraphIds, stabilizeBackendGraphNodeIds } from './previewGraph'; const sdxlModel: MainModelConfig = { base: 'sdxl', key: 'sdxl-model', name: 'SDXL', type: 'main' }; @@ -191,4 +191,21 @@ describe('stabilizeBackendGraphIds', () => { source: { field: 'value', node_id: 'seed' }, }); }); + + it('follows fixed node ids that lose their name to an earlier created id', () => { + const graph = { + edges: [], + id: 'graph', + nodes: { + seed_lqx1abc_a1b2c3: { id: 'seed_lqx1abc_a1b2c3', type: 'noise' }, + seed: { id: 'seed', type: 'integer' }, + positive_prompt: { id: 'positive_prompt', type: 'string' }, + }, + }; + + const result = stabilizeBackendGraphNodeIds(graph, { positive: 'positive_prompt', seed: 'seed' }); + + expect(result.nodeIds).toEqual({ positive: 'positive_prompt', seed: 'seed_2' }); + expect(result.graph.nodes[result.nodeIds.seed]?.type).toBe('integer'); + }); }); diff --git a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts index 12c3659af8e..9d829efdbf5 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/previewGraph.ts @@ -31,7 +31,11 @@ export type GeneratePreviewResult = const CREATED_ID_PATTERN = /^(.+)_[a-z0-9]{7,9}_[a-z0-9]{4,6}$/; -export const stabilizeBackendGraphIds = (graph: BackendGraphContract): BackendGraphContract => { +/** Stabilize ids and follow `nodeIds` through the renames: a fixed id loses its name to an earlier created one. */ +export const stabilizeBackendGraphNodeIds = >( + graph: BackendGraphContract, + nodeIds: T +): { graph: BackendGraphContract; nodeIds: T } => { const rename = new Map(); const used = new Set(); @@ -50,26 +54,27 @@ export const stabilizeBackendGraphIds = (graph: BackendGraphContract): BackendGr rename.set(id, candidate); } + const renamed = (id: string): string => rename.get(id) ?? id; + return { - edges: graph.edges.map((edge) => ({ - destination: { - field: edge.destination.field, - node_id: rename.get(edge.destination.node_id) ?? edge.destination.node_id, - }, - source: { field: edge.source.field, node_id: rename.get(edge.source.node_id) ?? edge.source.node_id }, - ...(edge.type ? { type: edge.type } : {}), - })), - id: 'generate-preview', - nodes: Object.fromEntries( - Object.entries(graph.nodes).map(([id, node]) => { - const nextId = rename.get(id) ?? id; - - return [nextId, { ...node, id: nextId }]; - }) - ), + graph: { + edges: graph.edges.map((edge) => ({ + destination: { field: edge.destination.field, node_id: renamed(edge.destination.node_id) }, + source: { field: edge.source.field, node_id: renamed(edge.source.node_id) }, + ...(edge.type ? { type: edge.type } : {}), + })), + id: 'generate-preview', + nodes: Object.fromEntries( + Object.entries(graph.nodes).map(([id, node]) => [renamed(id), { ...node, id: renamed(id) }]) + ), + }, + nodeIds: Object.fromEntries(Object.entries(nodeIds).map(([key, id]) => [key, renamed(id)])) as T, }; }; +export const stabilizeBackendGraphIds = (graph: BackendGraphContract): BackendGraphContract => + stabilizeBackendGraphNodeIds(graph, {}).graph; + const toPreviewContract = (backendGraph: BackendGraphContract, label: string): GraphContract => ({ backendGraph, edges: backendGraph.edges.map((edge, index) => ({ diff --git a/invokeai/frontend/webv2/src/features/generation/graphExport.ts b/invokeai/frontend/webv2/src/features/generation/graphExport.ts new file mode 100644 index 00000000000..2d1915f7a74 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/graphExport.ts @@ -0,0 +1,10 @@ +/** Offline graph export for tooling such as `pnpm graphs:export`; the browser bundle never loads it. */ +export { + compileGraphExportCase, + withArchitectureCapabilities, + type CompiledGraphExportCase, + type GraphExportCase, + type GraphExportEnvironment, + type GraphExportMode, +} from './core/graphExport'; +export type { ArchitectureCapabilitiesRow } from './core/architectureCapabilities'; diff --git a/invokeai/frontend/webv2/src/features/generation/settings.ts b/invokeai/frontend/webv2/src/features/generation/settings.ts index d4a29593997..06f3349e407 100644 --- a/invokeai/frontend/webv2/src/features/generation/settings.ts +++ b/invokeai/frontend/webv2/src/features/generation/settings.ts @@ -108,6 +108,7 @@ export { DYNAMIC_PROMPTS_MIN_PROMPTS, hasDynamicPromptSyntax, isDynamicPromptsSeedBehaviour, + planGenerateSubmission, sanitizeDynamicPromptsConfig, sanitizeMaxPrompts, sanitizeSampleSeed, diff --git a/invokeai/frontend/webv2/src/features/models/core/workingMemory.test.ts b/invokeai/frontend/webv2/src/features/models/core/workingMemory.test.ts new file mode 100644 index 00000000000..2ca4f19fe75 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/core/workingMemory.test.ts @@ -0,0 +1,440 @@ +import { describe, expect, it } from 'vitest'; + +import { + calibrationIdempotencyKey, + deriveCalibrationProgress, + describeCoverage, + describeFact, + formatAttemptId, + getCalibrationBlocker, + isMeasurementMethodVerified, + planCalibrationSteps, + selectAttemptRuns, + summarizeLineStates, + type LocalAttempt, + type WorkingMemoryCase, + type WorkingMemoryLine, + type WorkingMemoryPlan, + type WorkingMemoryRun, + type WorkingMemoryStatus, +} from './workingMemory'; + +const planCase = (dtype: string, side: number): WorkingMemoryCase => ({ + components: {}, + estimator: 'sd_vae_decode', + height: side, + id: `sd_vae_decode-decode-${dtype}-${String(side)}x${String(side)}`, + line: `decode_${dtype}`, + mode: 'txt2img', + model_key: 'main', + negative_prompt: '', + positive_prompt: 'a lake', + seed: 1, + settings: { vaePrecision: dtype }, + steps: 2, + width: side, +}); + +const plan = ( + cases: WorkingMemoryCase[], + warmUp = 1, + measured = 2 +): Pick => ({ + cases, + repetitions: { measured, warm_up: warmUp }, +}); + +const T0 = Date.parse('2026-10-01T12:00:00.000Z'); + +const run = ( + id: string, + offsetSeconds: number, + status: WorkingMemoryRun['status'], + error: string | null = null, + attemptId: string | null = 'attempt' +): WorkingMemoryRun => ({ + attempt_id: attemptId, + created_at: new Date(T0 + offsetSeconds * 1_000).toISOString(), + error, + point_count: status === 'completed' ? 1 : 0, + run_id: id, + status, +}); + +const describeSteps = (steps: ReturnType) => + steps.map((step) => `${step.case.line}@${String(step.case.width)}:${step.phase}${String(step.repetition)}`); + +describe('planCalibrationSteps', () => { + it('groups cases by precision and warms each case up right before measuring it', () => { + // The plan lists an fp32 case between two fp16 ones; grouping keeps one precision switch. + const steps = planCalibrationSteps(plan([planCase('fp16', 512), planCase('fp32', 512), planCase('fp16', 1024)])); + + expect(describeSteps(steps)).toEqual([ + 'decode_fp16@512:warm_up1', + 'decode_fp16@512:measured1', + 'decode_fp16@512:measured2', + 'decode_fp16@1024:warm_up1', + 'decode_fp16@1024:measured1', + 'decode_fp16@1024:measured2', + 'decode_fp32@512:warm_up1', + 'decode_fp32@512:measured1', + 'decode_fp32@512:measured2', + ]); + expect(steps.map((step) => step.index)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8]); + }); + + it('starts every precision with a warm-up, whatever the repetition counts', () => { + const steps = planCalibrationSteps( + plan([planCase('fp32', 768), planCase('fp16', 768), planCase('fp32', 1536)], 2, 3) + ); + const switches = steps.filter((step, index) => index === 0 || steps[index - 1]!.case.line !== step.case.line); + + expect(steps).toHaveLength(15); + expect(switches.map((step) => [step.case.line, step.phase])).toEqual([ + ['decode_fp32', 'warm_up'], + ['decode_fp16', 'warm_up'], + ]); + }); +}); + +describe('calibrationIdempotencyKey', () => { + const steps = planCalibrationSteps(plan([planCase('fp16', 512), planCase('fp32', 512)])); + + it('is the same for a resent step and differs across steps and attempts', () => { + const keys = steps.map((step) => calibrationIdempotencyKey('attempt-a', step)); + + expect(steps.map((step) => calibrationIdempotencyKey('attempt-a', step))).toEqual(keys); + expect(new Set(keys).size).toBe(steps.length); + expect(keys.some((key) => steps.some((step) => calibrationIdempotencyKey('attempt-b', step) === key))).toBe(false); + }); + + it('fits the server limit for plan ids of realistic length', () => { + const longStep = planCalibrationSteps( + plan([{ ...planCase('fp16', 4096), id: 'flux2_vae_decode-decode-bf16-4096x4096' }]) + )[2]!; + + expect(calibrationIdempotencyKey('0f8fad5b-d9cb-469f-a165-70867728950e', longStep).length).toBeLessThanOrEqual(255); + }); +}); + +describe('selectAttemptRuns', () => { + it("takes the newest run's attempt, oldest run first, however long its runs took", () => { + const runs = [ + run('new-2', 3_000, 'running', null, 'new'), + run('old-1', 0, 'failed', null, 'old'), + run('new-0', 300, 'completed', null, 'new'), + run('old-2', 1, 'completed', null, 'old'), + ]; + + expect(selectAttemptRuns(runs).map((entry) => entry.run_id)).toEqual(['new-0', 'new-2']); + expect(selectAttemptRuns(runs, 'old').map((entry) => entry.run_id)).toEqual(['old-1', 'old-2']); + expect(selectAttemptRuns([])).toEqual([]); + }); + + it('counts each run without an attempt id as an attempt of its own', () => { + const runs = [run('legacy-1', 0, 'completed', null, null), run('legacy-2', 1, 'failed', 'boom', null)]; + + expect(selectAttemptRuns(runs).map((entry) => entry.run_id)).toEqual(['legacy-2']); + }); +}); + +describe('deriveCalibrationProgress', () => { + const steps = planCalibrationSteps(plan([planCase('fp16', 512), planCase('fp32', 512)])); + const local = (overrides: Partial = {}): LocalAttempt => ({ + attemptId: 'mine', + newestRunAtStart: null, + stopped: false, + submissionError: null, + submitting: false, + total: steps.length, + ...overrides, + }); + + it('reports progress and the case the queue is on while any run is pending or running', () => { + const runs = [ + run('r0', 0, 'completed'), + run('r1', 1, 'completed'), + run('r2', 2, 'running'), + run('r3', 3, 'running'), + ]; + + expect(deriveCalibrationProgress({ local: null, runs, steps })).toEqual({ + activity: { currentCase: steps[2]!.case, done: 2, kind: 'running', queuing: false, total: 6 }, + outcome: { kind: 'none' }, + }); + }); + + it('counts an attempt that is still queuing as running before the server lists any run', () => { + const { activity } = deriveCalibrationProgress({ local: local({ submitting: true }), runs: [], steps }); + + expect(activity).toEqual({ currentCase: steps[0]!.case, done: 0, kind: 'running', queuing: true, total: 6 }); + }); + + it('reports the error of a failed run', () => { + const runs = [run('r0', 0, 'completed'), run('r1', 1, 'failed', 'CUDA out of memory'), run('r2', 2, 'canceled')]; + + expect(deriveCalibrationProgress({ local: null, runs, steps }).outcome).toEqual({ + error: 'CUDA out of memory', + kind: 'failed', + }); + }); + + it('tells completed, canceled and partial attempts apart', () => { + const completed = steps.map((_, index) => run(`r${String(index)}`, index, 'completed')); + const canceled = [run('r0', 0, 'canceled'), run('r1', 1, 'canceled')]; + const someCanceled = [...completed.slice(0, 3), run('r3', 3, 'canceled'), run('r4', 4, 'partial')]; + + expect(deriveCalibrationProgress({ local: null, runs: completed, steps }).outcome).toEqual({ kind: 'completed' }); + expect(deriveCalibrationProgress({ local: null, runs: canceled, steps }).outcome).toEqual({ kind: 'canceled' }); + expect(deriveCalibrationProgress({ local: null, runs: someCanceled, steps }).outcome).toEqual({ + canceled: true, + error: null, + kind: 'partial', + queued: 5, + total: 5, + }); + }); + + it('judges an attempt another tab started by its runs, not by a plan that may have changed since', () => { + const fewerThanPlanned = [run('r0', 0, 'completed'), run('r1', 1, 'completed')]; + + expect(deriveCalibrationProgress({ local: null, runs: fewerThanPlanned, steps }).outcome).toEqual({ + kind: 'completed', + }); + }); + + it('reports an attempt whose queuing stopped early as partial, or failed when nothing was queued', () => { + const runs = [run('r0', 0, 'completed', null, 'mine'), run('r1', 1, 'completed', null, 'mine')]; + + expect(deriveCalibrationProgress({ local: local({ submissionError: 'Queue full' }), runs, steps }).outcome).toEqual( + { + canceled: false, + error: 'Queue full', + kind: 'partial', + queued: 2, + total: 6, + } + ); + // Its graphs did not compile, so nothing of it was queued; an older attempt does not hide the error. + const older = [run('old', 0, 'completed', null, 'old')]; + const failedStart = local({ newestRunAtStart: 'old', submissionError: 'Case x: unknown model' }); + expect(deriveCalibrationProgress({ local: failedStart, runs: older, steps }).outcome).toEqual({ + error: 'Case x: unknown model', + kind: 'failed', + }); + }); + + it('lets an attempt that queued nothing give way to any attempt that ran after it', () => { + // This tab could not reach the server; another tab then calibrated to the end. + const failedStart = local({ newestRunAtStart: 'old', submissionError: 'Failed to fetch' }); + const theirs = steps.map((_, index) => run(`theirs-${String(index)}`, 100 + index, 'completed', null, 'theirs')); + + expect( + deriveCalibrationProgress({ + local: failedStart, + runs: [run('old', 0, 'completed', null, 'old'), ...theirs], + steps, + }).outcome + ).toEqual({ kind: 'completed' }); + expect( + deriveCalibrationProgress({ local: { ...failedStart, newestRunAtStart: null }, runs: theirs, steps }).outcome + ).toEqual({ kind: 'completed' }); + }); + + it('reports an attempt the user stopped as canceled, even where queuing ended before any run was canceled', () => { + const runs = [run('r0', 0, 'completed', null, 'mine'), run('r1', 1, 'completed', null, 'mine')]; + + expect(deriveCalibrationProgress({ local: local({ stopped: true }), runs, steps }).outcome).toEqual({ + canceled: true, + error: null, + kind: 'partial', + queued: 2, + total: 6, + }); + // Stopped while compiling: nothing was queued. + expect(deriveCalibrationProgress({ local: local({ stopped: true }), runs: [], steps }).outcome).toEqual({ + kind: 'canceled', + }); + }); + + it('tells an attempt whose queuing ended early from a complete one by the step count its id carries', () => { + const attemptId = formatAttemptId('0f8fad5b-d9cb-469f-a165-70867728950e', 6); + const truncated = [run('r0', 0, 'completed', null, attemptId), run('r1', 1, 'completed', null, attemptId)]; + const complete = steps.map((_, index) => run(`r${String(index)}`, index, 'completed', null, attemptId)); + + // After a reload this tab has no record of it. + expect(deriveCalibrationProgress({ local: null, runs: truncated, steps }).outcome).toEqual({ + canceled: false, + error: null, + kind: 'partial', + queued: 2, + total: 6, + }); + expect(deriveCalibrationProgress({ local: null, runs: complete, steps }).outcome).toEqual({ kind: 'completed' }); + }); + + it('follows the newer attempt another tab started, not this tab’s older one', () => { + const runs = [ + run('mine', 0, 'failed', 'old failure', 'mine'), + run('theirs-0', 100, 'completed', null, 'theirs'), + run('theirs-1', 101, 'running', null, 'theirs'), + ]; + const { activity } = deriveCalibrationProgress({ local: local(), runs, steps }); + + expect(activity).toMatchObject({ done: 1, kind: 'running' }); + }); +}); + +describe('summarizeLineStates', () => { + it('reports the most severe line, or null without lines', () => { + expect(summarizeLineStates([{ state: 'in_use' }, { state: 'stale' }])).toBe('stale'); + expect(summarizeLineStates([{ state: 'insufficient' }, { state: 'in_use' }])).toBe('insufficient'); + expect(summarizeLineStates([{ state: 'stale' }, { state: 'unusable' }])).toBe('unusable'); + expect(summarizeLineStates([{ state: 'in_use' }])).toBe('in_use'); + expect(summarizeLineStates([])).toBeNull(); + }); +}); + +describe('describeCoverage', () => { + type Line = Pick; + const line = ( + state: WorkingMemoryLine['state'], + { + envId = 'current', + lineName = 'decode_fp16', + ownWindows = 3, + version = 2, + }: { envId?: string; lineName?: string; ownWindows?: number; version?: number } = {} + ): Line => ({ + env_id: envId, + estimator: 'sd_vae_decode', + line: lineName, + own_windows: ownWindows, + state, + version, + }); + const coverage = (lines: Line[], runs: WorkingMemoryRun[] = []) => + describeCoverage({ current_env_id: 'current', lines, runs }); + + it("takes the headline from the current environment's lines, else from all lines", () => { + // An old environment's unusable line does not outrank what applies now. + expect(coverage([line('unusable', { envId: 'old' }), line('in_use')])).toMatchObject({ + hasCurrentLines: true, + state: 'in_use', + }); + expect(coverage([line('unusable', { envId: 'old' })])).toMatchObject({ hasCurrentLines: false, state: 'unusable' }); + expect(coverage([])).toMatchObject({ hasCurrentLines: false, hasOwnMeasurements: false, state: null }); + }); + + it('lets a line recalibrated under a new estimator version supersede the old version’s unusable line', () => { + const oldVersion = line('unusable', { version: 1 }); + + expect(coverage([oldVersion, line('in_use')]).state).toBe('in_use'); + expect(coverage([oldVersion, line('stale')]).state).toBe('stale'); + expect(coverage([oldVersion]).state).toBe('unusable'); + // Only the same line supersedes it. + expect(coverage([oldVersion, line('in_use', { lineName: 'decode_fp32' })]).state).toBe('unusable'); + expect(coverage([oldVersion, line('insufficient')]).state).toBe('unusable'); + }); + + it('tells lines only other models measured from the model’s own', () => { + expect(coverage([line('in_use', { ownWindows: 0 })])).toMatchObject({ + hasOwnMeasurements: false, + usesOthersOnly: true, + }); + expect(coverage([line('in_use', { ownWindows: 0 }), line('stale')])).toMatchObject({ + hasOwnMeasurements: true, + usesOthersOnly: false, + }); + // A shared line that does not apply is not used. + expect(coverage([line('insufficient', { ownWindows: 0 })]).usesOthersOnly).toBe(false); + // Runs that measured nothing still leave something to reset. + expect(coverage([], [run('r0', 0, 'canceled')]).hasOwnMeasurements).toBe(true); + }); +}); + +describe('getCalibrationBlocker', () => { + const supported = { reason: null, supported: true }; + const available = { available: true, unavailable_reason: null }; + const usable = { problem: null, usable: true }; + + it('names the reason a user can least act on first', () => { + expect( + getCalibrationBlocker({ + isRunning: true, + plan: { reason: 'Not a text-to-image model.', supported: false }, + status: { available: false, unavailable_reason: 'Two GPUs.' }, + summary: null, + }) + ).toEqual({ kind: 'unsupported', reason: 'Not a text-to-image model.' }); + expect( + getCalibrationBlocker({ + isRunning: true, + plan: supported, + status: { available: false, unavailable_reason: 'File unusable.' }, + summary: { problem: 'schema_version 9', usable: false }, + }) + ).toEqual({ kind: 'storeUnusable', reason: 'schema_version 9' }); + expect( + getCalibrationBlocker({ + isRunning: false, + plan: supported, + status: { available: false, unavailable_reason: 'Two GPUs.' }, + summary: null, + }) + ).toEqual({ kind: 'unavailable', reason: 'Two GPUs.' }); + expect(getCalibrationBlocker({ isRunning: true, plan: supported, status: available, summary: usable })).toEqual({ + kind: 'running', + }); + expect(getCalibrationBlocker({ isRunning: false, plan: supported, status: available, summary: usable })).toBeNull(); + }); +}); + +describe('isMeasurementMethodVerified', () => { + const status = (verified: boolean | null): WorkingMemoryStatus => ({ + available: true, + current_device_class_id: 'class', + current_env_id: 'env', + device_classes: [ + { + device_class: { arch: null, backend: 'cuda', id: 'class', name: 'GPU', total_mib: 8192, vendor: 'x' }, + device_class_id: 'class', + envs: [], + is_current: true, + measurement_method: 'allocator', + measurement_verified: verified, + point_counts: [], + problem: null, + runs: [], + usable: true, + }, + ], + unavailable_reason: null, + }); + + it("takes the model summary's answer, then the current device class's", () => { + expect(isMeasurementMethodVerified({ measurement_verified: false }, status(true))).toBe(false); + expect(isMeasurementMethodVerified({ measurement_verified: null }, status(false))).toBe(false); + expect(isMeasurementMethodVerified(null, status(true))).toBe(true); + expect(isMeasurementMethodVerified(null, status(null))).toBeNull(); + }); +}); + +describe('describeFact', () => { + it('names runtimes by product, translates the rest, and shows variables and settings by their own name', () => { + expect(['torch', 'hip', 'cudnn', 'os'].map(describeFact)).toEqual([ + { name: 'PyTorch' }, + { name: 'ROCm' }, + { name: 'cuDNN' }, + { name: 'OS' }, + ]); + expect(['driver', 'allocator_backend', 'rocm_sdpa_guard'].map(describeFact)).toEqual([ + { translated: 'driver' }, + { translated: 'allocator' }, + { translated: 'sdpaGuard' }, + ]); + expect(describeFact('setting.attention_type')).toEqual({ name: 'attention_type' }); + expect(describeFact('env.PYTORCH_CUDA_ALLOC_CONF')).toEqual({ name: 'PYTORCH_CUDA_ALLOC_CONF' }); + expect(describeFact('something_new')).toEqual({ name: 'something_new' }); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/models/core/workingMemory.ts b/invokeai/frontend/webv2/src/features/models/core/workingMemory.ts new file mode 100644 index 00000000000..fd85e04b217 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/core/workingMemory.ts @@ -0,0 +1,428 @@ +/** + * Working-memory calibration as the Model Manager sees it: the order a plan's cases run in, the retry keys that make a + * lost response safe to resend, which of a model's runs make up the latest attempt, and what the section shows for + * a plan, a summary and the device status. The server owns every measurement and every state; this module only + * orders, groups and reads them. + */ + +import type { components } from '@api/schema'; + +type Schemas = components['schemas']; + +export type WorkingMemoryPlan = Schemas['WorkingMemoryCalibrationPlan']; +export type WorkingMemoryCase = Schemas['WorkingMemoryCalibrationCase']; +export type WorkingMemorySummary = Schemas['WorkingMemoryCalibrationModelSummary']; +export type WorkingMemoryLine = Schemas['WorkingMemoryCalibrationModelLine']; +export type WorkingMemoryLineSize = Schemas['WorkingMemoryCalibrationLineSize']; +export type WorkingMemoryRun = Schemas['WorkingMemoryCalibrationModelRun']; +export type WorkingMemoryStatus = Schemas['WorkingMemoryCalibrationStatus']; +export type WorkingMemoryRunCreated = Schemas['WorkingMemoryCalibrationRunCreated']; +export type WorkingMemoryLineState = WorkingMemoryLine['state']; +export type WorkingMemoryLineContributor = Schemas['WorkingMemoryCalibrationLineContributor']; + +/** The server reserves this origin prefix for calibration queue items; the run id follows it. */ +export const CALIBRATION_ORIGIN_PREFIX = 'invokeai:wm-calibration:'; + +export const calibrationRunOrigin = (runId: string): string => `${CALIBRATION_ORIGIN_PREFIX}${runId}`; + +export const isCalibrationOrigin = (origin: unknown): boolean => + typeof origin === 'string' && origin.startsWith(CALIBRATION_ORIGIN_PREFIX); + +/** Models whose plan the section asks for: calibration covers main models and standalone VAEs. */ +export const supportsWorkingMemoryCalibration = (model: { type: string }): boolean => + model.type === 'main' || model.type === 'vae'; + +/** A case's enqueue payload as Generate would build it: the graph and its batch data. */ +export interface CalibrationGraph { + caseId: string; + graph: unknown; + data: unknown; +} + +/** + * Builds every case's graph, or rejects naming the case that cannot compile, and rejects once `signal` aborts. + * Generation owns graph compilation and depends on Models, so App supplies this; its implementation loads lazily on + * first call. + */ +export type CalibrationGraphCompiler = ( + cases: readonly WorkingMemoryCase[], + signal: AbortSignal +) => Promise; + +export type CalibrationPhase = 'warm_up' | 'measured'; + +/** One queue item of an attempt: a case at one repetition. Each step is its own POST and its own server run. */ +export interface CalibrationStep { + /** Position in the attempt, which is also the order the server creates the step's run in. */ + index: number; + case: WorkingMemoryCase; + phase: CalibrationPhase; + /** 1-based within the phase. */ + repetition: number; +} + +/** + * The plan's cases grouped by line (one decode dtype each), each case's warm-ups right before its measured runs. + * Grouping keeps one precision switch per dtype: the first decode after a switch recasts the VAE, and the server + * excludes those windows, so a switch must land on a warm-up. + */ +export const planCalibrationSteps = (plan: Pick): CalibrationStep[] => { + const groups = new Map(); + + for (const item of plan.cases) { + const group = groups.get(item.line); + + if (group) { + group.push(item); + } else { + groups.set(item.line, [item]); + } + } + + const steps: CalibrationStep[] = []; + const push = (item: WorkingMemoryCase, phase: CalibrationPhase, count: number) => { + for (let repetition = 1; repetition <= count; repetition += 1) { + steps.push({ case: item, index: steps.length, phase, repetition }); + } + }; + + for (const cases of groups.values()) { + for (const item of cases) { + push(item, 'warm_up', plan.repetitions.warm_up); + push(item, 'measured', plan.repetitions.measured); + } + } + + return steps; +}; + +/** + * The retry key of one step. Resending a step after a lost response with the same key returns the run the first + * request created. The key carries the attempt id: the server keys its retry receipts on the key alone, so a key + * reused by a new attempt would be refused. + */ +export const calibrationIdempotencyKey = (attemptId: string, step: CalibrationStep): string => + `webv2:wm-calibration:${attemptId}:${String(step.index)}:${step.case.id}:${step.phase}-${String(step.repetition)}`; + +const byCreation = (left: WorkingMemoryRun, right: WorkingMemoryRun): number => + Date.parse(left.created_at) - Date.parse(right.created_at); + +/** The id of the most recently created run, or null without runs. */ +export const newestRunId = (runs: readonly WorkingMemoryRun[]): string | null => + [...runs].sort(byCreation).at(-1)?.run_id ?? null; + +/** + * The runs of one attempt, oldest first: those carrying `attemptId`, or by default those of the newest run's attempt. + * A run without an attempt id (created before the server stored one) is an attempt of its own. + */ +export const selectAttemptRuns = (runs: readonly WorkingMemoryRun[], attemptId?: string): WorkingMemoryRun[] => { + const ordered = [...runs].sort(byCreation); + const newest = ordered.at(-1); + const target = attemptId ?? newest?.attempt_id ?? null; + + if (target === null) { + return newest ? [newest] : []; + } + + return ordered.filter((run) => run.attempt_id === target); +}; + +/** + * An attempt's id: a fresh id plus the number of steps it means to queue, so any tab can tell an attempt whose + * queuing ended early (a reload, a closed tab) from a complete one. The server treats the id as opaque. + */ +export const formatAttemptId = (id: string, total: number): string => `${id}.${String(total)}`; + +/** The step count an attempt id carries; null for an id without one. */ +export const plannedStepsOf = (attemptId: string | null | undefined): number | null => { + const match = /\.(\d+)$/.exec(attemptId ?? ''); + + return match ? Number(match[1]) : null; +}; + +/** What this tab knows of an attempt it started. */ +export interface LocalAttempt { + attemptId: string; + /** Steps the attempt meant to queue. */ + total: number; + /** Still compiling graphs or posting steps. */ + submitting: boolean; + /** Why compiling or posting stopped early, when it did. */ + submissionError: string | null; + /** The user stopped it. */ + stopped: boolean; + /** + * The newest of the model's runs this tab knew when the attempt started. While the attempt has no runs of its own + * (it failed before queuing anything), it is the latest attempt only until a newer run appears. + */ + newestRunAtStart: string | null; +} + +export type CalibrationActivity = + | { kind: 'idle' } + | { + kind: 'running'; + /** Steps settled so far. */ + done: number; + total: number; + /** The case the queue is running or about to run, when it can be told. */ + currentCase: WorkingMemoryCase | null; + /** Still compiling graphs or posting steps. */ + queuing: boolean; + }; + +export type CalibrationOutcome = + | { kind: 'none' } + | { kind: 'completed' } + | { kind: 'canceled' } + /** `canceled`: runs were canceled, so the attempt did not stop because queuing ended early. */ + | { kind: 'partial'; canceled: boolean; error: string | null; queued: number; total: number } + | { kind: 'failed'; error: string | null }; + +/** + * Progress and the latest attempt's outcome. A run counts as running while any of its items is pending or running + * (the server derives that from the queue), so a pending item blocks a second start across reloads and tabs. + */ +export const deriveCalibrationProgress = ({ + local, + runs, + steps, +}: { + local: LocalAttempt | null; + runs: readonly WorkingMemoryRun[]; + steps: readonly CalibrationStep[]; +}): { activity: CalibrationActivity; outcome: CalibrationOutcome } => { + // Another tab may have started a newer attempt since; this tab's record then describes an older one. A record whose + // attempt queued nothing (it failed to compile or to post its first step) stays current until a newer run exists. + const newest = [...runs].sort(byCreation).at(-1); + const current = + local !== null && + (local.submitting || + (newest !== undefined && newest.attempt_id === local.attemptId) || + (!runs.some((run) => run.attempt_id === local.attemptId) && + (newest?.run_id ?? null) === local.newestRunAtStart && + !runs.some((run) => run.status === 'running'))) + ? local + : null; + const attempt = selectAttemptRuns(runs, current?.attemptId); + // The attempt's own step count: this tab's record, else the one its id carries. A running attempt with neither is + // assumed to follow the current plan; a settled one is judged by its runs alone, since the plan may have changed. + const planned = current?.total ?? plannedStepsOf(attempt[0]?.attempt_id); + const total = Math.max(planned ?? steps.length, attempt.length); + const plannedTotal = planned ?? attempt.length; + const runningIndex = attempt.findIndex((run) => run.status === 'running'); + const queuing = current?.submitting ?? false; + + if (queuing || runningIndex !== -1) { + // Steps settle in order: the queue runs an attempt's items first in, first out. + const done = attempt.filter((run) => run.status !== 'running').length; + const currentIndex = runningIndex === -1 ? attempt.length : runningIndex; + + return { + activity: { currentCase: steps[currentIndex]?.case ?? null, done, kind: 'running', queuing, total }, + outcome: { kind: 'none' }, + }; + } + + const idle: CalibrationActivity = { kind: 'idle' }; + const canceled = + (current?.stopped ?? false) || attempt.some((run) => run.status === 'canceled' || run.status === 'partial'); + const partial = (error: string | null): CalibrationOutcome => ({ + canceled, + error, + kind: 'partial', + queued: attempt.length, + total: plannedTotal, + }); + + if (current?.submissionError) { + return { + activity: idle, + outcome: + attempt.length === 0 ? { error: current.submissionError, kind: 'failed' } : partial(current.submissionError), + }; + } + + if (attempt.length === 0) { + return { activity: idle, outcome: current?.stopped ? { kind: 'canceled' } : { kind: 'none' } }; + } + + const failed = attempt.find((run) => run.status === 'failed'); + + if (failed) { + return { activity: idle, outcome: { error: failed.error, kind: 'failed' } }; + } + + if (attempt.every((run) => run.status === 'canceled')) { + return { activity: idle, outcome: { kind: 'canceled' } }; + } + + if (attempt.every((run) => run.status === 'completed') && attempt.length >= plannedTotal) { + return { activity: idle, outcome: { kind: 'completed' } }; + } + + return { activity: idle, outcome: partial(null) }; +}; + +const LINE_STATE_SEVERITY: readonly WorkingMemoryLineState[] = ['unusable', 'stale', 'insufficient', 'in_use']; + +/** The most severe state among lines, or null without lines. */ +export const summarizeLineStates = ( + lines: readonly Pick[] +): WorkingMemoryLineState | null => + LINE_STATE_SEVERITY.find((state) => lines.some((line) => line.state === state)) ?? null; + +/** What a summary says about the model as a whole. */ +export interface CalibrationCoverage { + /** The headline state: of the current environment's lines, or of all lines when it has none. */ + state: WorkingMemoryLineState | null; + /** Some line of the current environment exists, measured by this model or another. */ + hasCurrentLines: boolean; + /** This model has windows on some line or has runs, so a reset removes something. */ + hasOwnMeasurements: boolean; + /** Estimates use lines that only other models measured. */ + usesOthersOnly: boolean; +} + +export const describeCoverage = ( + summary: { + current_env_id: string | null; + lines: readonly Pick[]; + runs: readonly unknown[]; + } | null +): CalibrationCoverage => { + const lines = summary?.lines ?? []; + const current = lines.filter((line) => line.env_id === summary?.current_env_id); + const hasOwnWindows = lines.some((line) => line.own_windows > 0); + const applies = (line: (typeof lines)[number]) => line.state === 'in_use' || line.state === 'stale'; + // An estimator update leaves the old version's windows as an unusable line; once the new version's line applies, + // the old one is superseded and does not head the section. + const headline = current.filter( + (line) => + line.state !== 'unusable' || + !current.some( + (other) => + other.estimator === line.estimator && + other.line === line.line && + other.version !== line.version && + applies(other) + ) + ); + + return { + hasCurrentLines: current.length > 0, + hasOwnMeasurements: hasOwnWindows || (summary?.runs.length ?? 0) > 0, + state: summarizeLineStates(current.length > 0 ? headline : lines), + usesOthersOnly: !hasOwnWindows && current.some(applies), + }; +}; + +/** A line's contributors with this model first, the rest in the server's order (by name). */ +export const orderContributors = ( + contributors: readonly WorkingMemoryLineContributor[], + modelKey: string +): WorkingMemoryLineContributor[] => [ + ...contributors.filter((entry) => entry.model_key === modelKey), + ...contributors.filter((entry) => entry.model_key !== modelKey), +]; + +export type CalibrationBlocker = + | { kind: 'unsupported'; reason: string | null } + | { kind: 'unavailable'; reason: string | null } + | { kind: 'storeUnusable'; reason: string | null } + | { kind: 'running' }; + +/** + * Why Calibrate is unavailable, in the order a user can act on: nothing helps an unsupported model; a missing or + * second GPU and an unusable store are machine problems; a running attempt only needs to finish. + */ +export const getCalibrationBlocker = ({ + isRunning, + plan, + status, + summary, +}: { + isRunning: boolean; + plan: Pick; + status: Pick; + /** Null when the server has no current device class to summarise. */ + summary: Pick | null; +}): CalibrationBlocker | null => { + if (!plan.supported) { + return { kind: 'unsupported', reason: plan.reason }; + } + + if (summary && !summary.usable) { + return { kind: 'storeUnusable', reason: summary.problem }; + } + + if (!status.available || summary === null) { + return { kind: 'unavailable', reason: status.unavailable_reason }; + } + + return isRunning ? { kind: 'running' } : null; +}; + +/** + * Whether the server's way of measuring GPU memory was checked against real hardware on this GPU's backend: the + * model summary's answer, else the current device class's; null while neither can tell. + */ +export const isMeasurementMethodVerified = ( + summary: Pick | null, + status: WorkingMemoryStatus +): boolean | null => + summary?.measurement_verified ?? + status.device_classes.find((entry) => entry.is_current)?.measurement_verified ?? + null; + +/** `decode_fp16` → operation `decode`, dtype `fp16`. A line without a dtype suffix keeps its whole name. */ +export const splitLine = (line: string): { dtype: string | null; operation: string } => { + const separator = line.lastIndexOf('_'); + + return separator <= 0 + ? { dtype: null, operation: line } + : { dtype: line.slice(separator + 1), operation: line.slice(0, separator) }; +}; + +/** Changed facts the section names with a translated label. */ +export type TranslatedFact = 'allocator' | 'driver' | 'invokeai' | 'osBuild' | 'sdpaGuard' | 'torchBuild'; + +/** Facts named by a product or runtime name, which needs no translation. */ +const FACT_NAMES: Readonly> = { + cuda: 'CUDA', + cudnn: 'cuDNN', + hip: 'ROCm', + miopen: 'MIOpen', + os: 'OS', + torch: 'PyTorch', + xpu: 'XPU runtime', +}; + +const TRANSLATED_FACTS: Readonly> = { + allocator_backend: 'allocator', + driver: 'driver', + invokeai: 'invokeai', + os_build: 'osBuild', + rocm_sdpa_guard: 'sdpaGuard', + torch_build: 'torchBuild', +}; + +/** + * How the section names a changed environment fact (the server's fingerprint keys): a product name, a translated + * label, or for environment variables and Invoke settings their own name. An unknown key shows as sent. + */ +export const describeFact = (key: string): { name: string } | { translated: TranslatedFact } => { + const translated = TRANSLATED_FACTS[key]; + + if (translated) { + return { translated }; + } + + for (const prefix of ['env.', 'setting.']) { + if (key.startsWith(prefix) && key.length > prefix.length) { + return { name: key.slice(prefix.length) }; + } + } + + return { name: FACT_NAMES[key] ?? key }; +}; diff --git a/invokeai/frontend/webv2/src/features/models/data/workingMemoryApi.ts b/invokeai/frontend/webv2/src/features/models/data/workingMemoryApi.ts new file mode 100644 index 00000000000..61dae8c1fdd --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/data/workingMemoryApi.ts @@ -0,0 +1,71 @@ +import type { + WorkingMemoryPlan, + WorkingMemoryRunCreated, + WorkingMemoryStatus, + WorkingMemorySummary, +} from '@features/models/core/workingMemory'; + +import { calibrationRunOrigin } from '@features/models/core/workingMemory'; + +import { browserModelsDataPort } from './transport'; + +/** REST client for working-memory calibration, admin-only on the server. */ + +const { request, requestJson } = browserModelsDataPort; + +const modelPath = (key: string): string => `/api/v2/models/i/${encodeURIComponent(key)}/working_memory`; + +/** Calibration items are ordinary queue items; the queue routes act on them by origin. */ +const QUEUE_BASE = '/api/v1/queue/default'; + +export const getWorkingMemoryPlan = (key: string, signal?: AbortSignal): Promise => + requestJson(`${modelPath(key)}/plan`, { signal }); + +/** 409 when there is no current device class (no GPU, or several generation devices). */ +export const getWorkingMemorySummary = (key: string, signal?: AbortSignal): Promise => + requestJson(`${modelPath(key)}/summary`, { signal }); + +export const getWorkingMemoryStatus = (signal?: AbortSignal): Promise => + requestJson('/api/v2/models/working_memory/status', { signal }); + +export interface CalibrateRequest { + /** Groups the runs of one Calibrate click; the server returns it on every run. */ + attemptId: string; + graph: unknown; + data: unknown; + idempotencyKey: string; +} + +/** Enqueues one queue item as one server run. Resending with the same key returns that run and enqueues nothing. */ +export const calibrateWorkingMemory = ( + key: string, + { attemptId, data, graph, idempotencyKey }: CalibrateRequest, + signal?: AbortSignal +): Promise => + requestJson(`${modelPath(key)}/calibrate`, { + body: JSON.stringify({ attempt_id: attemptId, data, graph, idempotency_key: idempotencyKey }), + method: 'POST', + signal, + }); + +/** Releases the server's receipt for a key whose response arrived; until then it counts toward a per-owner limit. */ +export const acknowledgeCalibrationEnqueue = async (idempotencyKey: string, signal?: AbortSignal): Promise => { + await request(`${QUEUE_BASE}/enqueue_batch/acknowledge`, { + body: JSON.stringify({ idempotency_key: idempotencyKey }), + headers: { 'Content-Type': 'application/json' }, + method: 'POST', + signal, + }); +}; + +/** Cancels a run's items, the running one included. */ +export const cancelCalibrationRun = async (runId: string, signal?: AbortSignal): Promise => { + const params = new URLSearchParams({ origin_prefix: calibrationRunOrigin(runId) }); + + await request(`${QUEUE_BASE}/cancel_all?${params.toString()}`, { method: 'PUT', signal }); +}; + +/** Removes the model's runs and measurements on the current device class. */ +export const resetWorkingMemory = async (key: string, signal?: AbortSignal): Promise => { + await request(modelPath(key), { method: 'DELETE', signal }); +}; diff --git a/invokeai/frontend/webv2/src/features/models/data/workingMemoryCalibration.ts b/invokeai/frontend/webv2/src/features/models/data/workingMemoryCalibration.ts new file mode 100644 index 00000000000..cd1378d0f3d --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/data/workingMemoryCalibration.ts @@ -0,0 +1,306 @@ +import type { QueryClient } from '@tanstack/react-query'; + +import { + calibrationIdempotencyKey, + formatAttemptId, + isCalibrationOrigin, + newestRunId, + planCalibrationSteps, + type CalibrationGraphCompiler, + type LocalAttempt, + type WorkingMemoryPlan, + type WorkingMemoryRunCreated, + type WorkingMemorySummary, +} from '@features/models/core/workingMemory'; +import { createUuid } from '@platform/browser/randomUuid'; +import { + assertAccountScopeCurrent, + captureAccountScope, + isAccountScopeCurrent, + registerAccountOwnedResource, + type AccountScope, +} from '@platform/state/accountLifecycle'; +import { createExternalStore } from '@platform/state/externalStore'; +import { ApiError, getApiErrorMessage, sleep } from '@platform/transport/http'; +import { socketHub } from '@platform/transport/socketHub'; + +import { + acknowledgeCalibrationEnqueue, + calibrateWorkingMemory, + cancelCalibrationRun, + resetWorkingMemory, + type CalibrateRequest, +} from './workingMemoryApi'; +import { workingMemoryKeys, workingMemorySummaryQueryOptions } from './workingMemoryQueries'; + +/** + * The calibration attempts this tab started, by model key: what only this tab knows (that it is still compiling or + * posting steps, how many it meant to post, why it stopped). Every step's run carries the attempt id, so the server's + * runs say which runs belong together, after a reload too. Kept outside the section so leaving a model's details + * while an attempt queues, and coming back, shows the same attempt. + */ +const attemptsStore = createExternalStore<{ byModel: ReadonlyMap }>({ byModel: new Map() }); + +interface Flight { + promise: Promise; + stopRequested: boolean; + /** Aborts compiling; a step being posted is not aborted, since its run exists once the response arrives. */ + stop: AbortController; +} + +/** The attempt queuing per model; at most one, so a double click cannot post a second attempt. */ +const flights = new Map(); + +registerAccountOwnedResource({ + clear: () => { + for (const flight of flights.values()) { + flight.stopRequested = true; + flight.stop.abort(); + } + flights.clear(); + attemptsStore.setSnapshot({ byModel: new Map() }); + }, + name: 'working-memory-calibration', +}); + +const setAttempt = (modelKey: string, attempt: LocalAttempt | null): void => { + const byModel = new Map(attemptsStore.getSnapshot().byModel); + + if (attempt) { + byModel.set(modelKey, attempt); + } else { + byModel.delete(modelKey); + } + + attemptsStore.setSnapshot({ byModel }); +}; + +const updateAttempt = (modelKey: string, attemptId: string, change: (attempt: LocalAttempt) => LocalAttempt) => { + const attempt = attemptsStore.getSnapshot().byModel.get(modelKey); + + // A reset or a newer attempt replaced it meanwhile. + if (attempt?.attemptId === attemptId) { + setAttempt(modelKey, change(attempt)); + } +}; + +export const useLocalAttempt = (modelKey: string): LocalAttempt | null => + attemptsStore.useSelector((snapshot) => snapshot.byModel.get(modelKey) ?? null, Object.is); + +/** A lost response is resent with the same key, which returns the run the first request created. */ +const POST_ATTEMPTS = 3; +const POST_RETRY_DELAY_MS = 1_000; + +const postStep = async ( + modelKey: string, + request: CalibrateRequest, + signal: AbortSignal +): Promise => { + for (let attempt = 1; ; attempt += 1) { + try { + return await calibrateWorkingMemory(modelKey, request, signal); + } catch (error) { + // The server answered: resending the same request gets the same answer. + if (error instanceof ApiError || signal.aborted || attempt >= POST_ATTEMPTS) { + throw error; + } + await sleep(POST_RETRY_DELAY_MS, signal); + } + } +}; + +/** + * Every model's summary moves: lines are shared across models, and a reset or new points change their fits. The + * changed model's plan prices on those fits too. + */ +const invalidateAfterChange = async (queryClient: QueryClient, owner: AccountScope, modelKey?: string) => { + await Promise.all([ + queryClient.invalidateQueries({ queryKey: workingMemoryKeys.summaries(owner) }), + queryClient.invalidateQueries({ queryKey: workingMemoryKeys.status(owner) }), + modelKey === undefined + ? undefined + : queryClient.invalidateQueries({ queryKey: workingMemoryKeys.plan(owner, modelKey) }), + ]); +}; + +/** + * Compiles the plan's graphs (loading the compiler on first use) and posts every step in order, each after the last + * one's response, so the queue runs them in step order. Stops at the first error the server answers with and keeps + * the runs already queued: they run, and the section reports the attempt as partial. + */ +export const startCalibration = ({ + compile, + modelKey, + plan, + queryClient, +}: { + compile: CalibrationGraphCompiler; + modelKey: string; + plan: WorkingMemoryPlan; + queryClient: QueryClient; +}): Promise => { + const running = flights.get(modelKey); + + if (running) { + return running.promise; + } + + const owner = captureAccountScope(); + const steps = planCalibrationSteps(plan); + const attemptId = formatAttemptId(createUuid(), steps.length); + const flight: Flight = { promise: Promise.resolve(), stop: new AbortController(), stopRequested: false }; + + const known = queryClient.getQueryData(workingMemoryKeys.summary(owner, modelKey)); + + setAttempt(modelKey, { + attemptId, + newestRunAtStart: newestRunId(known?.runs ?? []), + stopped: false, + submissionError: null, + submitting: true, + total: steps.length, + }); + + const queue = async () => { + const compiling = compile(plan.cases, AbortSignal.any([owner.signal, flight.stop.signal])); + const graphs = new Map((await compiling).map((compiled) => [compiled.caseId, compiled])); + + for (const step of steps) { + if (flight.stopRequested || !isAccountScopeCurrent(owner)) { + return; + } + + const compiled = graphs.get(step.case.id); + + if (!compiled) { + throw new Error(`No graph was built for case ${step.case.id}.`); + } + + const idempotencyKey = calibrationIdempotencyKey(attemptId, step); + await postStep(modelKey, { attemptId, data: compiled.data, graph: compiled.graph, idempotencyKey }, owner.signal); + + assertAccountScopeCurrent(owner); + // Best effort: an unacknowledged receipt only expires later. + void acknowledgeCalibrationEnqueue(idempotencyKey, owner.signal).catch(() => undefined); + void queryClient.invalidateQueries({ queryKey: workingMemoryKeys.summary(owner, modelKey) }); + } + }; + + flight.promise = queue() + .catch((error: unknown) => { + // A stop that aborted compiling is not an error. + if (isAccountScopeCurrent(owner) && !flight.stopRequested) { + updateAttempt(modelKey, attemptId, (attempt) => ({ + ...attempt, + submissionError: getApiErrorMessage(error, ''), + })); + } + }) + .finally(() => { + if (flights.get(modelKey) === flight) { + flights.delete(modelKey); + } + + if (isAccountScopeCurrent(owner)) { + updateAttempt(modelKey, attemptId, (attempt) => ({ ...attempt, submitting: false })); + void invalidateAfterChange(queryClient, owner, modelKey); + } + }); + flights.set(modelKey, flight); + + return flight.promise; +}; + +/** + * Stops queuing, then cancels every run of the model the queue still holds, newest first so no later item starts + * while an earlier one is cancelled. Waits for the step in flight: its run exists once the response arrives. + */ +export const stopCalibration = async ({ + modelKey, + queryClient, +}: { + modelKey: string; + queryClient: QueryClient; +}): Promise => { + const owner = captureAccountScope(); + const flight = flights.get(modelKey); + + const local = attemptsStore.getSnapshot().byModel.get(modelKey); + + if (local) { + updateAttempt(modelKey, local.attemptId, (attempt) => ({ ...attempt, stopped: true })); + } + + if (flight) { + flight.stopRequested = true; + flight.stop.abort(); + await flight.promise; + } + + const summary = await queryClient.fetchQuery({ + ...workingMemorySummaryQueryOptions(modelKey, { isConnected: true, isSubmitting: false }, owner), + staleTime: 0, + }); + const running = (summary?.runs ?? []) + .filter((run) => run.status === 'running') + .sort((left, right) => Date.parse(right.created_at) - Date.parse(left.created_at)); + + try { + for (const run of running) { + await cancelCalibrationRun(run.run_id, owner.signal); + } + } finally { + // Awaited, so a confirmation closing after this sees whether Calibrate is available again. + if (isAccountScopeCurrent(owner)) { + await invalidateAfterChange(queryClient, owner, modelKey); + } + } +}; + +/** Removes the model's runs and measurements; the shipped estimates apply again from the next generation on. */ +export const resetCalibration = async ({ + modelKey, + queryClient, +}: { + modelKey: string; + queryClient: QueryClient; +}): Promise => { + const owner = captureAccountScope(); + + await resetWorkingMemory(modelKey, owner.signal); + assertAccountScopeCurrent(owner); + setAttempt(modelKey, null); + await invalidateAfterChange(queryClient, owner, modelKey); +}; + +/** + * Keeps an open section current: a calibration item changing status refreshes the summaries, and a reconnect + * refreshes everything, since events during the gap were lost. Polling covers the time without a socket. + */ +export const attachWorkingMemoryRealtime = (queryClient: QueryClient): (() => void) => { + const owner = captureAccountScope(); + + const detachStatus = socketHub.on('queue_item_status_changed', (payload: never) => { + const origin = (payload as { origin?: unknown } | null)?.origin; + + if (isCalibrationOrigin(origin) && isAccountScopeCurrent(owner)) { + void invalidateAfterChange(queryClient, owner); + } + }); + + // The hub reports the current status on subscribe; only a later transition to connected is a reconnect. + let wasConnected: boolean | null = null; + const detachConnection = socketHub.onConnectionChange((status) => { + const isConnected = status === 'connected'; + + if (isConnected && wasConnected === false && isAccountScopeCurrent(owner)) { + void queryClient.invalidateQueries({ queryKey: workingMemoryKeys.forAccount(owner) }); + } + wasConnected = isConnected; + }); + + return () => { + detachStatus(); + detachConnection(); + }; +}; diff --git a/invokeai/frontend/webv2/src/features/models/data/workingMemoryQueries.ts b/invokeai/frontend/webv2/src/features/models/data/workingMemoryQueries.ts new file mode 100644 index 00000000000..bd9b4e73976 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/data/workingMemoryQueries.ts @@ -0,0 +1,76 @@ +import type { WorkingMemorySummary } from '@features/models/core/workingMemory'; + +import { assertAccountScopeCurrent, captureAccountScope, type AccountScope } from '@platform/state/accountLifecycle'; +import { ApiError } from '@platform/transport/http'; +import { queryOptions } from '@tanstack/react-query'; + +import { getWorkingMemoryPlan, getWorkingMemoryStatus, getWorkingMemorySummary } from './workingMemoryApi'; + +/** Polls while a run is pending or running and the socket is down, which includes a backend without one (the mock). */ +export const WORKING_MEMORY_POLL_MS = 2_000; +/** With a socket, its events drive updates; this only catches a missed one. */ +export const WORKING_MEMORY_FALLBACK_POLL_MS = 30_000; + +const accountKey = (owner: AccountScope) => ({ accountId: owner.accountId, epoch: owner.epoch }); + +/** Keys carry the account so a sign-out never serves one account's calibration to the next. */ +export const workingMemoryKeys = { + forAccount: (owner: AccountScope) => ['models', 'workingMemory', accountKey(owner)] as const, + plan: (owner: AccountScope, modelKey: string) => [...workingMemoryKeys.forAccount(owner), 'plan', modelKey] as const, + status: (owner: AccountScope) => [...workingMemoryKeys.forAccount(owner), 'status'] as const, + summaries: (owner: AccountScope) => [...workingMemoryKeys.forAccount(owner), 'summary'] as const, + summary: (owner: AccountScope, modelKey: string) => [...workingMemoryKeys.summaries(owner), modelKey] as const, +}; + +const fenced = (owner: AccountScope, request: (signal: AbortSignal) => Promise, signal: AbortSignal) => + request(AbortSignal.any([signal, owner.signal])).then((result) => { + assertAccountScopeCurrent(owner); + return result; + }); + +export const hasRunningRun = (summary: WorkingMemorySummary | null | undefined): boolean => + summary?.runs.some((run) => run.status === 'running') ?? false; + +/** The plan depends on installed models and the GPU, both of which change rarely. */ +export const workingMemoryPlanQueryOptions = (modelKey: string, owner = captureAccountScope()) => + queryOptions({ + queryFn: ({ signal }) => fenced(owner, (fencedSignal) => getWorkingMemoryPlan(modelKey, fencedSignal), signal), + queryKey: workingMemoryKeys.plan(owner, modelKey), + staleTime: 60_000, + }); + +/** Null when the server has no current device class to summarise; the status then says why. */ +export const workingMemorySummaryQueryOptions = ( + modelKey: string, + { isConnected, isSubmitting }: { isConnected: boolean; isSubmitting: boolean }, + owner = captureAccountScope() +) => + queryOptions({ + queryFn: ({ signal }) => + fenced( + owner, + (fencedSignal) => + getWorkingMemorySummary(modelKey, fencedSignal).catch((error: unknown) => { + if (error instanceof ApiError && error.status === 409) { + return null; + } + throw error; + }), + signal + ), + queryKey: workingMemoryKeys.summary(owner, modelKey), + refetchInterval: (query) => + isSubmitting || hasRunningRun(query.state.data) + ? isConnected + ? WORKING_MEMORY_FALLBACK_POLL_MS + : WORKING_MEMORY_POLL_MS + : false, + staleTime: 5_000, + }); + +export const workingMemoryStatusQueryOptions = (owner = captureAccountScope()) => + queryOptions({ + queryFn: ({ signal }) => fenced(owner, (fencedSignal) => getWorkingMemoryStatus(fencedSignal), signal), + queryKey: workingMemoryKeys.status(owner), + staleTime: 15_000, + }); diff --git a/invokeai/frontend/webv2/src/features/models/index.ts b/invokeai/frontend/webv2/src/features/models/index.ts index c0f818e38cc..e31ee7ff02d 100644 --- a/invokeai/frontend/webv2/src/features/models/index.ts +++ b/invokeai/frontend/webv2/src/features/models/index.ts @@ -30,7 +30,9 @@ export { useInstallActions } from './ui/add-models/useInstallActions'; export { getStarterModelInstallSources, type StarterInstallSource } from './ui/add-models/starterModelInstallSources'; export { ModelInstallRuntime } from './ui/ModelInstallRuntime'; export { ModelsPage } from './ui/ModelsPage'; +export type { CalibrationGraph, CalibrationGraphCompiler, WorkingMemoryCase } from './core/workingMemory'; export { + CalibrationGraphCompilerProvider, ModelsUiProvider, useOpenAddModelsSearch, useOpenModelInManager, diff --git a/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx b/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx index b6b1ed7cc43..fdfd797510e 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx +++ b/invokeai/frontend/webv2/src/features/models/ui/ModelsUiContext.tsx @@ -1,3 +1,4 @@ +import type { CalibrationGraphCompiler } from '@features/models/core/workingMemory'; import type { ReactNode } from 'react'; import { captureAccountScope, isAccountScopeCurrent } from '@platform/state/accountLifecycle'; @@ -28,6 +29,20 @@ export const ModelsUiProvider = ({ adapter, children }: { adapter: ModelsUiAdapt export const useModelsUi = (): ModelsUiAdapter => use(ModelsUiContext); +const CalibrationGraphCompilerContext = createContext(null); + +/** Supplies working-memory calibration with Generate's graph compiler, which App composes: Models cannot import it. */ +export const CalibrationGraphCompilerProvider = ({ + children, + compile, +}: { + children: ReactNode; + compile: CalibrationGraphCompiler; +}) => {children}; + +/** Null outside a provider; calibration is then unavailable. */ +export const useCalibrationGraphCompiler = (): CalibrationGraphCompiler | null => use(CalibrationGraphCompilerContext); + /** * Opens the Model Manager on a model's details, returning to the manager's project like its other entry points; null * when this session may not manage models. The manager's UI store loads on first use so editor boot does not carry it. diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.browser.test.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.browser.test.tsx new file mode 100644 index 00000000000..7298c8c9313 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.browser.test.tsx @@ -0,0 +1,78 @@ +import type { ModelConfig } from '@features/models/core/types'; + +import { ChakraProvider } from '@chakra-ui/react'; +import { setModelsSnapshotForTests } from '@features/models/data/modelsStore'; +import { accountLifecycle } from '@platform/state/accountLifecycle'; +import { system } from '@theme/system'; +import { act } from 'react'; +import { createRoot, type Root } from 'react-dom/client'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { ModelDetail } from './ModelDetail'; + +// The sections own their data; here they only mark where the page places them. +const { section } = vi.hoisted(() => ({ + section: (name: string) => () =>
, +})); + +vi.mock('@features/identity', async (importOriginal) => ({ + ...(await importOriginal()), + useCapabilities: () => ({ canManageModels: true }), +})); +vi.mock('./DefaultSettingsSection', () => ({ DefaultSettingsSection: section('default-settings') })); +vi.mock('./RelatedModelsSection', () => ({ RelatedModelsSection: section('related-models') })); +vi.mock('./TriggerPhrasesEditor', () => ({ MemoizedTriggerPhrasesEditor: section('trigger-phrases') })); +vi.mock('./WorkingMemorySection', () => ({ WorkingMemorySection: section('working-memory') })); +vi.mock('react-i18next', () => ({ useTranslation: () => ({ t: (key: string) => key }) })); + +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const MAIN = { + base: 'sdxl', + file_size: 1024, + format: 'checkpoint', + hash: 'hash-main', + key: 'main-1', + name: 'Main Model', + path: '/models/main.safetensors', + source: '/models/main.safetensors', + source_type: 'path', + trigger_phrases: [], + type: 'main', + variant: 'normal', +} as unknown as ModelConfig; + +const ignoreConfirm = () => undefined; + +describe('ModelDetail', () => { + let host: HTMLDivElement; + let root: Root; + + beforeEach(() => { + accountLifecycle.activate('model-detail-test', ':user:model-detail-test'); + setModelsSnapshotForTests({ models: [MAIN], status: 'loaded' }); + host = document.createElement('div'); + document.body.append(host); + root = createRoot(host); + }); + + afterEach(async () => { + await act(() => root.unmount()); + host.remove(); + setModelsSnapshotForTests({ models: [], status: 'idle' }); + accountLifecycle.invalidate(); + }); + + it('places working memory last, after the trigger phrases', async () => { + await act(() => + root.render( + + + + ) + ); + + const order = [...host.querySelectorAll('[data-section]')].map((element) => element.getAttribute('data-section')); + expect(order).toEqual(['default-settings', 'related-models', 'trigger-phrases', 'working-memory']); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx index a354aa5f32e..a3e4c9f1d40 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx @@ -3,10 +3,12 @@ import type { ModelConfig } from '@features/models/core/types'; import type { ReactNode } from 'react'; import { chakra, DataList, HStack, Icon, Menu, Portal, Separator, Stack, Text } from '@chakra-ui/react'; +import { useCapabilities } from '@features/identity'; import { isConvertibleToDiffusers } from '@features/models/core/baseIdentity'; import { isLinkableType } from '@features/models/core/relationships'; import { isAbsoluteModelPath, resolveModelAbsolutePath } from '@features/models/core/schemas'; import { getModelSourceHref } from '@features/models/core/taxonomy'; +import { supportsWorkingMemoryCalibration } from '@features/models/core/workingMemory'; import { useModelsSelector, type ModelsSnapshot } from '@features/models/data/modelsStore'; import { discardModelDraft, @@ -35,6 +37,7 @@ import { ModelSettingsMenuItems } from './ModelSettingsMenuItems'; import { RelatedModelsSection } from './RelatedModelsSection'; import { MemoizedTriggerPhrasesEditor } from './TriggerPhrasesEditor'; import { UpdatePathDialog } from './UpdatePathDialog'; +import { WorkingMemorySection } from './WorkingMemorySection'; const TRIGGER_PHRASE_TYPES = new Set(['main', 'lora', 'embedding']); const EMPTY_TRIGGER_PHRASES: readonly string[] = []; @@ -152,6 +155,7 @@ export const ModelDetail = ({ }) => { const { t } = useTranslation(); const model = useModelsSelector((snapshot) => selectModelShell(snapshot, modelKey)); + const { canManageModels } = useCapabilities(); if (!model) { return ( @@ -194,6 +198,14 @@ export const ModelDetail = ({ ) : null} + + {/* Last and collapsed: calibration is a rare, admin-only task. */} + {canManageModels && supportsWorkingMemoryCalibration(model) ? ( + <> + + + + ) : null} ); }; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryCalibrateDialog.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryCalibrateDialog.tsx new file mode 100644 index 00000000000..cc9707f2420 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryCalibrateDialog.tsx @@ -0,0 +1,130 @@ +/* eslint-disable react-perf/jsx-no-new-function-as-prop */ +import type { WorkingMemoryPlan } from '@features/models/core/workingMemory'; + +import { chakra, Portal, Stack, Text } from '@chakra-ui/react'; +import { planCalibrationSteps } from '@features/models/core/workingMemory'; +import { Button, CloseButton } from '@platform/ui/Button'; +import { Dialog } from '@platform/ui/Dialog'; +import { useCallback, useRef } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { SharedModelsList, useCaseLabel } from './WorkingMemoryParts'; + +const SECONDS_PER_MINUTE = 60; + +/** Confirms a calibration: what it runs, where, for how long, and which other models it moves. */ +export const WorkingMemoryCalibrateDialog = ({ + calibratedThroughName, + deviceName, + getFinalFocus, + isOpen, + onClose, + onConfirm, + onExitComplete, + plan, +}: { + /** The main model a standalone VAE's cases generate with. */ + calibratedThroughName: string | null; + deviceName: string | null; + /** Where focus goes once the dialog closes; starting a calibration can disable the opener. */ + getFinalFocus: () => HTMLElement | null; + isOpen: boolean; + onClose: () => void; + onConfirm: () => void; + onExitComplete: () => void; + plan: WorkingMemoryPlan; +}) => { + const { t } = useTranslation(); + const startRef = useRef(null); + const getInitialFocus = useCallback(() => startRef.current, []); + const runCount = planCalibrationSteps(plan).length; + const minutes = + plan.estimated_seconds === null ? null : Math.max(1, Math.round(plan.estimated_seconds / SECONDS_PER_MINUTE)); + const caseLabel = useCaseLabel(); + + return ( + { + if (!event.open) { + onClose(); + } + }} + > + + + + + + + {t('models.workingMemory.dialogTitle')} + {deviceName ? ( + + {t('models.workingMemory.dialogDescription', { device: deviceName })} + + ) : null} + + + + + + {minutes === null ? null : ( + {t('models.workingMemory.dialogDuration', { minutes })} + )} + {t('models.workingMemory.dialogQueueNote')} + {calibratedThroughName ? ( + + {t('models.workingMemory.dialogThrough', { model: calibratedThroughName })} + + ) : null} + + + + {t('models.workingMemory.dialogRuns', { count: runCount })} + + + {plan.cases.map((item) => ( + + {caseLabel(item)} + + ))} + + + {t('models.workingMemory.dialogRepetitions', { + measured: plan.repetitions.measured, + warmUp: plan.repetitions.warm_up, + })} + + + {plan.shared_with.length > 0 ? : null} + + + + + + + + + + + + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryParts.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryParts.tsx new file mode 100644 index 00000000000..ee51e17ef2d --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemoryParts.tsx @@ -0,0 +1,351 @@ +import type { SystemStyleObject } from '@chakra-ui/react'; +import type { + WorkingMemoryCase, + WorkingMemoryLine, + WorkingMemoryLineSize, + WorkingMemoryLineState, + WorkingMemoryPlan, +} from '@features/models/core/workingMemory'; + +import { Badge, chakra, HStack, Icon, Popover, Portal, Stack, Table, Text } from '@chakra-ui/react'; +import { getModelBaseLabel } from '@features/models/core/baseIdentity'; +import { describeFact, orderContributors, splitLine } from '@features/models/core/workingMemory'; +import { formatBytes, formatCount } from '@platform/i18n/languages'; +import { IconButton } from '@platform/ui/Button'; +import { PopoverContent } from '@platform/ui/Popover'; +import { Tooltip, useTooltipTriggerIds } from '@platform/ui/Tooltip'; +import { InfoIcon } from 'lucide-react'; +import { useTranslation } from 'react-i18next'; + +const STATE_PALETTES: Record = { + in_use: 'green', + insufficient: 'gray', + stale: 'orange', + unusable: 'red', +}; + +/** One state chip; the color repeats the text, never replaces it. */ +export const LineStateBadge = ({ state }: { state: WorkingMemoryLineState | null }) => { + const { t } = useTranslation(); + const label = + state === 'in_use' + ? t('models.workingMemory.stateInUse') + : state === 'stale' + ? t('models.workingMemory.stateStale') + : state === 'unusable' + ? t('models.workingMemory.stateUnusable') + : state === 'insufficient' + ? t('models.workingMemory.stateInsufficient') + : t('models.workingMemory.stateNone'); + + return ( + + {label} + + ); +}; + +/** A plan case as "768 × 768 · fp16". */ +export const useCaseLabel = () => { + const { t } = useTranslation(); + + return (item: Pick): string => + t('models.workingMemory.pair', { + detail: splitLine(item.line).dtype ?? item.line, + label: t('models.workingMemory.size', { height: item.height, width: item.width }), + }); +}; + +/** `decode_fp16` → "Decode · fp16". An operation this version has no name for shows as the server sends it. */ +export const useLineLabel = () => { + const { t } = useTranslation(); + + return (line: string): string => { + const { dtype, operation } = splitLine(line); + const operationLabel = + operation === 'decode' + ? t('models.workingMemory.operationDecode') + : operation === 'encode' + ? t('models.workingMemory.operationEncode') + : operation; + + return dtype ? t('models.workingMemory.pair', { detail: dtype, label: operationLabel }) : operationLabel; + }; +}; + +const MEGAPIXEL = 1_000_000; + +const formatMegapixels = (pixels: number): string => formatCount(Math.round((pixels / MEGAPIXEL) * 100) / 100); + +const TABLE_SX: SystemStyleObject = { + '& td, & th': { borderColor: 'border.subtle', fontSize: 'sm', px: '2', py: '1' }, + '& thead th': { + color: 'fg.muted', + fontSize: 'xs', + fontWeight: '700', + letterSpacing: 'wider', + textTransform: 'uppercase', + }, + '& tbody th': { fontWeight: '400', textAlign: 'start' }, + // Every row starts with its size as a `th`, so each `td` and every header after the first holds a number. + '& td, & thead th + th': { fontVariantNumeric: 'tabular-nums', textAlign: 'end' }, +}; + +/** A changed environment fact by a readable name; see `describeFact`. */ +const useFactLabel = () => { + const { t } = useTranslation(); + + return (key: string): string => { + const fact = describeFact(key); + + if ('name' in fact) { + return fact.name; + } + + switch (fact.translated) { + case 'allocator': + return t('models.workingMemory.facts.allocator'); + case 'driver': + return t('models.workingMemory.facts.driver'); + case 'invokeai': + return t('models.workingMemory.facts.invokeai'); + case 'osBuild': + return t('models.workingMemory.facts.osBuild'); + case 'sdpaGuard': + return t('models.workingMemory.facts.sdpaGuard'); + case 'torchBuild': + return t('models.workingMemory.facts.torchBuild'); + } + }; +}; + +/** The state help of a line: why it applies or not, naming the facts that changed or differ. */ +const LineStateHelp = ({ line }: { line: WorkingMemoryLine }) => { + const { t } = useTranslation(); + const factLabel = useFactLabel(); + const facts = line.changed_facts.map(factLabel).join(', '); + const text = + line.state === 'in_use' + ? t('models.workingMemory.helpInUse') + : line.state === 'stale' + ? t('models.workingMemory.helpStale', { facts }) + : line.state === 'unusable' + ? facts + ? t('models.workingMemory.helpUnusable', { facts }) + : t('models.workingMemory.helpUnusableNoFacts') + : (line.problem ?? t('models.workingMemory.helpInsufficient')); + + return ( + + {text} + + ); +}; + +const HINT_POSITIONING = { placement: 'top' } as const; + +/** A column header with an info button after its label. */ +const HintedColumnHeader = ({ hint, label }: { hint: string; label: string }) => { + const { t } = useTranslation(); + const name = t('models.workingMemory.aboutColumn', { column: label }); + // The button stays on the line of the label's last word when a narrow column wraps the label. + const split = label.lastIndexOf(' ') + 1; + + return ( + + {label.slice(0, split)} + + {label.slice(split)} + + + + ); +}; + +/** The explanation opens in a popover, by click or keyboard, so it does not depend on hover. */ +const HintButton = ({ hint, name }: { hint: string; name: string }) => { + const ids = useTooltipTriggerIds(); + + return ( + + + + + + + + + + + + {hint} + + + + + ); +}; + +/** Why a measured size does not feed the line, or null for a size that does. */ +const useNotCountedLabel = () => { + const { t } = useTranslation(); + + return (size: Pick): string | null => { + if (size.counted) { + return null; + } + + switch (size.excluded_reason) { + case 'fragmented': + return t('models.workingMemory.notCountedFragmented'); + case 'partial_residency': + return t('models.workingMemory.notCountedPartial'); + case null: + return t('models.workingMemory.notCounted'); + } + }; +}; + +/** + * One estimator line: its state, the models whose measurements it pools (named when another model is among them, + * this model first), the range the user line was fitted on, and every measured size. + */ +export const LineMeasurements = ({ line, modelKey }: { line: WorkingMemoryLine; modelKey: string }) => { + const { t } = useTranslation(); + const lineLabel = useLineLabel()(line.line); + const notCountedLabel = useNotCountedLabel(); + const isShared = line.measured_by.some((entry) => entry.model_key !== modelKey); + const contributors = orderContributors(line.measured_by, modelKey) + .map((entry) => entry.name ?? t('models.workingMemory.removedModel')) + .join(', '); + + return ( + + + + {lineLabel} + + + + + {isShared ? ( + + {t('models.workingMemory.measuredBy', { models: contributors })} + + ) : null} + {line.measured_range ? ( + + {t('models.workingMemory.measuredRange', { + max: formatMegapixels(line.measured_range.max_pixels), + min: formatMegapixels(line.measured_range.min_pixels), + })} + + ) : null} + {line.sizes.length > 0 ? ( + + + + + + + + + + + {t('models.workingMemory.columnSize')} + + + + + + + {line.sizes.map((size) => { + // A size the fit leaves out stays listed, muted, and names why in its row header. + const notCounted = notCountedLabel(size); + const color = notCounted ? 'fg.muted' : undefined; + + return ( + + + {t('models.workingMemory.size', { height: size.height, width: size.width })} + {notCounted ? ( + + {notCounted} + + ) : null} + + + {formatBytes(size.measured_bytes)} + + + {formatBytes(size.shipped_bytes)} + + + {formatBytes(size.user_bytes)} + + + ); + })} + + + + ) : null} + + ); +}; + +/** Other installed models whose estimates the same lines price, so a calibration moves them too. */ +export const SharedModelsList = ({ models }: { models: WorkingMemoryPlan['shared_with'] }) => { + const { t } = useTranslation(); + const lineLabel = useLineLabel(); + + return ( + + + {t('models.workingMemory.sharedWith', { count: models.length })} + + + {models.map((model) => ( + + {t('models.workingMemory.pair', { + detail: [getModelBaseLabel(model.base), ...model.lines.map(lineLabel)].join(', '), + label: model.name, + })} + + ))} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.browser.test.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.browser.test.tsx new file mode 100644 index 00000000000..c413c6bc595 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.browser.test.tsx @@ -0,0 +1,942 @@ +import type { + CalibrationGraphCompiler, + WorkingMemoryCase, + WorkingMemoryLine, + WorkingMemoryPlan, + WorkingMemoryRun, + WorkingMemoryStatus, + WorkingMemorySummary, +} from '@features/models/core/workingMemory'; + +import { ChakraProvider } from '@chakra-ui/react'; +import { CalibrationGraphCompilerProvider } from '@features/models/ui/ModelsUiContext'; +import { accountLifecycle } from '@platform/state/accountLifecycle'; +import { ApiError } from '@platform/transport/http'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import { system } from '@theme/system'; +import { act } from 'react'; +import { createRoot, type Root } from 'react-dom/client'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { userEvent } from 'vitest/browser'; + +const api = vi.hoisted(() => ({ + acknowledgeCalibrationEnqueue: vi.fn((_idempotencyKey: string) => Promise.resolve()), + calibrateWorkingMemory: vi.fn(), + cancelCalibrationRun: vi.fn((_runId: string) => Promise.resolve()), + getWorkingMemoryPlan: vi.fn(), + getWorkingMemoryStatus: vi.fn(), + getWorkingMemorySummary: vi.fn(), + resetWorkingMemory: vi.fn((_modelKey: string, _signal?: AbortSignal) => Promise.resolve()), +})); + +vi.mock('@features/models/data/workingMemoryApi', () => api); +vi.mock('react-i18next', () => ({ + useTranslation: () => ({ + t: (key: string, options?: Record) => + options && Object.keys(options).length > 0 + ? `${key}(${Object.entries(options) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([name, value]) => `${name}=${String(value)}`) + .join(',')})` + : key, + }), +})); + +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const MODEL_KEY = 'sdxl-main'; +const DEVICE = { id: '0123456789abcdef', name: 'Fixture GPU 24GB' }; +const T0 = Date.parse('2026-10-01T12:00:00.000Z'); + +const planCase = (dtype: string, side: number): WorkingMemoryCase => ({ + components: {}, + estimator: 'sdxl_vae_decode', + height: side, + id: `sdxl_vae_decode-decode-${dtype}-${String(side)}x${String(side)}`, + line: `decode_${dtype}`, + mode: 'txt2img', + model_key: MODEL_KEY, + negative_prompt: '', + positive_prompt: 'a lake', + seed: 1, + settings: { vaePrecision: dtype }, + steps: 2, + width: side, +}); + +const CASES = [planCase('fp16', 512), planCase('fp32', 512)]; + +const supportedPlan = (): WorkingMemoryPlan => ({ + calibrated_through: null, + cases: CASES, + device_class: DEVICE, + estimated_seconds: 150, + estimators: ['sdxl_vae_decode'], + model_key: MODEL_KEY, + not_covered: ['encode: not measured'], + reason: null, + repetitions: { measured: 2, warm_up: 1 }, + shared_with: [{ base: 'sd-1', key: 'sd1', lines: ['decode_fp16'], name: 'Shared SD1', type: 'main' }], + supported: true, +}); + +const status = (overrides: Partial = {}): WorkingMemoryStatus => ({ + available: true, + current_device_class_id: DEVICE.id, + current_env_id: 'env', + device_classes: [], + unavailable_reason: null, + ...overrides, +}); + +const run = ( + id: string, + offset: number, + runStatus: WorkingMemoryRun['status'], + error: string | null = null, + attemptId: string | null = 'earlier-attempt' +): WorkingMemoryRun => ({ + attempt_id: attemptId, + created_at: new Date(T0 + offset * 1_000).toISOString(), + error, + point_count: runStatus === 'completed' ? 1 : 0, + run_id: id, + status: runStatus, +}); + +const OWN = { model_key: MODEL_KEY, name: 'This SDXL', windows: 6 }; + +const line = ( + state: WorkingMemoryLine['state'], + changedFacts: string[] = [], + overrides: Partial = {} +): WorkingMemoryLine => ({ + base_bytes: 400_000_000, + carrier: 'sdxl_vae_decode', + changed_facts: changedFacts, + env_id: 'env', + estimator: 'sdxl_vae_decode', + line: 'decode_fp16', + measured_by: [OWN], + measured_range: { max_pixels: 1_048_576, min_pixels: 262_144 }, + problem: null, + sizes: [ + { + counted: true, + excluded_reason: null, + height: 512, + measured_bytes: 1_234_000_000, + pixels: 262_144, + shipped_bytes: null, + user_bytes: null, + width: 512, + windows: 3, + }, + { + counted: true, + excluded_reason: null, + height: 1024, + measured_bytes: 2_100_000_000, + pixels: 1_048_576, + shipped_bytes: 2_800_000_000, + user_bytes: 2_600_000_000, + width: 1024, + windows: 3, + }, + ], + own_windows: 6, + slope_bytes_per_pixel: 1_100, + state, + version: 1, + ...overrides, +}); + +const summary = (overrides: Partial = {}): WorkingMemorySummary => ({ + current_env_id: 'env', + device_class_id: DEVICE.id, + excluded_windows: 0, + lines: [], + measurement_method: 'allocator', + measurement_verified: true, + model_key: MODEL_KEY, + problem: null, + runs: [], + usable: true, + ...overrides, +}); + +let host: HTMLDivElement; +let root: Root; +let queryClient: QueryClient; +let compile: ReturnType>; +let currentSummary: WorkingMemorySummary | Error; + +/** Renders the section, opened as a user would unless `expanded` is false: it starts collapsed. */ +const renderSection = async ({ expanded = true }: { expanded?: boolean } = {}) => { + host = document.createElement('div'); + host.style.width = '720px'; + document.body.append(host); + root = createRoot(host); + queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + const { WorkingMemorySection } = await import('./WorkingMemorySection'); + + await act(() => + root.render( + + + + + + + + ) + ); + + if (expanded) { + await act(() => userEvent.click(sectionToggle())); + } +}; + +/** The disclosure button inside the section heading. */ +const sectionToggle = () => host.querySelector('h3 button')!; + +const buttonNamed = (name: string, within: ParentNode = document): HTMLButtonElement | undefined => + [...within.querySelectorAll('button')].find((button) => button.textContent.trim() === name); + +const openDialog = () => document.querySelector('[role="dialog"][data-state="open"]'); +const openAlertDialog = () => document.querySelector('[role="alertdialog"][data-state="open"]'); +const sectionText = () => host.textContent; + +/** The summary the section reads next; a poll or an invalidation picks it up. */ +const serveSummary = (next: WorkingMemorySummary | Error) => { + currentSummary = next; +}; + +const refreshSummary = () => act(() => queryClient.invalidateQueries()); + +beforeEach(() => { + document.documentElement.lang = 'en'; + accountLifecycle.activate('wm-section-test', ':user:wm-section-test'); + compile = vi.fn((cases) => + Promise.resolve(cases.map((item) => ({ caseId: item.id, data: [], graph: { edges: [], id: item.id, nodes: {} } }))) + ); + currentSummary = summary(); + api.getWorkingMemoryPlan.mockImplementation(() => Promise.resolve(supportedPlan())); + api.getWorkingMemoryStatus.mockImplementation(() => Promise.resolve(status())); + api.getWorkingMemorySummary.mockImplementation(() => + currentSummary instanceof Error ? Promise.reject(currentSummary) : Promise.resolve(currentSummary) + ); + let runNumber = 0; + api.calibrateWorkingMemory.mockImplementation(() => { + runNumber += 1; + return Promise.resolve({ device_class_id: DEVICE.id, item_ids: [runNumber], run_id: `run-${String(runNumber)}` }); + }); +}); + +afterEach(async () => { + await act(() => root.unmount()); + host.remove(); + accountLifecycle.invalidate(); + vi.clearAllMocks(); +}); + +describe('WorkingMemorySection', () => { + it('calibrates a model that was never calibrated: dialog, queued runs, progress, then the measurements', async () => { + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.neverCalibrated'); + + const section = host.querySelector('section')!; + expect(section.getAttribute('aria-labelledby')).toBe(host.querySelector('h3')!.id); + expect(host.querySelector('h3')!.textContent).toBe('models.workingMemory.title'); + expect(sectionText()).toContain(DEVICE.name); + expect(sectionText()).toContain('models.workingMemory.stateNone'); + // The compiler loads only once a calibration starts. + expect(compile).not.toHaveBeenCalled(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + const dialog = openDialog()!; + expect(dialog.textContent).toContain('models.workingMemory.dialogDuration(minutes=3)'); + expect(dialog.textContent).toContain('models.workingMemory.dialogQueueNote'); + expect(dialog.textContent).toContain( + 'models.workingMemory.pair(detail=fp16,label=models.workingMemory.size(height=512,width=512))' + ); + expect(dialog.textContent).toContain( + 'models.workingMemory.pair(detail=fp32,label=models.workingMemory.size(height=512,width=512))' + ); + expect(dialog.textContent).toContain('models.workingMemory.dialogRuns(count=6)'); + expect(dialog.textContent).toContain('Shared SD1'); + expect(compile).not.toHaveBeenCalled(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + + expect(compile).toHaveBeenCalledTimes(1); + expect(compile).toHaveBeenCalledWith(CASES, expect.any(AbortSignal)); + const requests = api.calibrateWorkingMemory.mock.calls.map( + ([key, request]) => [key, (request as { graph: { id: string } }).graph.id] as const + ); + // Each precision warms up first: fp16 warm-up, two measured, then the same in fp32. + expect(requests).toEqual([ + [MODEL_KEY, CASES[0]!.id], + [MODEL_KEY, CASES[0]!.id], + [MODEL_KEY, CASES[0]!.id], + [MODEL_KEY, CASES[1]!.id], + [MODEL_KEY, CASES[1]!.id], + [MODEL_KEY, CASES[1]!.id], + ]); + const keys = api.calibrateWorkingMemory.mock.calls.map( + ([, request]) => (request as { idempotencyKey: string }).idempotencyKey + ); + expect(new Set(keys).size).toBe(6); + // One attempt id for the click, on every step and inside every retry key. + const attemptIds = new Set( + api.calibrateWorkingMemory.mock.calls.map(([, request]) => (request as { attemptId: string }).attemptId) + ); + expect(attemptIds.size).toBe(1); + const [attemptId] = [...attemptIds]; + expect(keys.every((key) => key.includes(attemptId!))).toBe(true); + expect(api.acknowledgeCalibrationEnqueue.mock.calls.map(([key]) => key)).toEqual(keys); + + serveSummary( + summary({ + runs: [ + run('run-1', 0, 'completed', null, attemptId), + run('run-2', 1, 'completed', null, attemptId), + ...[3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'running', null, attemptId)), + ], + }) + ); + await refreshSummary(); + await expect.poll(sectionText).toContain('models.workingMemory.running(done=2,total=6)'); + expect(host.querySelector('[role="progressbar"]')?.getAttribute('aria-label')).toBe( + 'models.workingMemory.progressLabel' + ); + expect(sectionText()).toContain( + 'models.workingMemory.pair(detail=fp16,label=models.workingMemory.size(height=512,width=512))' + ); + expect(buttonNamed('models.workingMemory.recalibrate')?.disabled).toBe(true); + expect(sectionText()).toContain('models.workingMemory.blockedRunning'); + // Reset names the same reason it is disabled for. + const reset = buttonNamed('common.reset')!; + expect(reset.disabled).toBe(true); + expect(document.getElementById(reset.getAttribute('aria-describedby')!)?.textContent).toBe( + 'models.workingMemory.blockedRunning' + ); + + serveSummary( + summary({ + lines: [line('in_use')], + // An earlier attempt failed; the newest attempt is the one that counts. + runs: [ + run('old', -600, 'failed', 'old failure'), + ...[1, 2, 3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'completed', null, attemptId)), + ], + }) + ); + // Polling picks the end up without any other trigger. + await expect.poll(() => host.querySelector('table'), { timeout: 5_000 }).not.toBeNull(); + + const table = host.querySelector('table')!; + expect(table.getAttribute('aria-label')).toBe( + 'models.workingMemory.tableLabel(line=models.workingMemory.pair(detail=fp16,label=models.workingMemory.operationDecode))' + ); + const rows = [...table.querySelectorAll('tbody tr')].map((row) => + [...row.querySelectorAll('th, td')].map((cell) => cell.textContent) + ); + expect([...table.querySelectorAll('tbody th')].map((cell) => cell.getAttribute('scope'))).toEqual(['row', 'row']); + expect(table.querySelectorAll('thead th')[3]?.textContent).toBe('models.workingMemory.columnCalibrated'); + // The size labels its row on the left; every value sits right-aligned under its header. + const alignments = (selector: string) => + [...table.querySelectorAll(selector)].map((cell) => getComputedStyle(cell).textAlign); + expect(alignments('thead th')).toEqual(['start', 'end', 'end', 'end']); + expect(alignments('tbody tr:first-child > *')).toEqual(['start', 'end', 'end', 'end']); + // Missing estimates show a dash; sizes format like the rest of the app. + expect(rows).toEqual([ + ['models.workingMemory.size(height=512,width=512)', '1.2 GB', '—', '—'], + ['models.workingMemory.size(height=1024,width=1024)', '2.1 GB', '2.8 GB', '2.6 GB'], + ]); + expect(sectionText()).toContain('models.workingMemory.measuredRange(max=1.05,min=0.26)'); + expect(sectionText()).toContain('models.workingMemory.stateInUse'); + expect(host.querySelector('[role="progressbar"]')).toBeNull(); + }); + + it('operates the dialog by keyboard and returns focus to its opener', async () => { + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + const opener = buttonNamed('models.workingMemory.calibrate')!; + opener.focus(); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(openDialog).not.toBeNull(); + await expect + .poll(() => openDialog()?.contains(document.activeElement) && document.activeElement?.textContent) + .toBe('models.workingMemory.calibrate'); + expect(openDialog()!.querySelector('h2, [id$="title"]')?.textContent).toBe('models.workingMemory.dialogTitle'); + + await act(() => userEvent.keyboard('{Escape}')); + await expect.poll(openDialog).toBeNull(); + await expect.poll(() => document.activeElement).toBe(opener); + expect(api.calibrateWorkingMemory).not.toHaveBeenCalled(); + expect(compile).not.toHaveBeenCalled(); + + await act(() => userEvent.keyboard('{Enter}')); + await expect + .poll(() => openDialog()?.contains(document.activeElement) && document.activeElement?.textContent) + .toBe('models.workingMemory.calibrate'); + // Hold the compiler so the attempt stays in its running state while focus settles. + let releaseCompile = () => undefined as void; + const compiled = new Promise((resolve) => { + releaseCompile = resolve; + }); + const compileCases = compile.getMockImplementation()!; + compile.mockImplementationOnce(async (cases) => { + await compiled; + return compileCases(cases, new AbortController().signal); + }); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(openDialog).toBeNull(); + // Starting disables the opener; focus moves to the action that is available now instead of the page. + await expect.poll(() => document.activeElement?.textContent).toBe('models.workingMemory.stop'); + expect(host.contains(document.activeElement)).toBe(true); + + await act(() => releaseCompile()); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + }); + + it('stops a running calibration after confirmation, newest run first', async () => { + serveSummary( + summary({ runs: [run('run-1', 0, 'completed'), run('run-2', 1, 'running'), run('run-3', 2, 'running')] }) + ); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.stop')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + expect(openAlertDialog()!.textContent).toContain('models.workingMemory.stopBody'); + expect(api.cancelCalibrationRun).not.toHaveBeenCalled(); + + api.cancelCalibrationRun.mockImplementation(() => { + serveSummary( + summary({ runs: [run('run-1', 0, 'completed'), run('run-2', 1, 'partial'), run('run-3', 2, 'canceled')] }) + ); + return Promise.resolve(); + }); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop', openAlertDialog()!)!)); + await expect.poll(openAlertDialog).toBeNull(); + + expect(api.cancelCalibrationRun.mock.calls.map(([runId]) => runId)).toEqual(['run-3', 'run-2']); + await expect.poll(sectionText).toContain('models.workingMemory.partial'); + // The Stop button is gone; focus lands on the action that is available now. + await expect.poll(() => document.activeElement?.textContent).toBe('common.retry'); + }); + + it('reports an attempt stopped while queuing as canceled, not as queued short', async () => { + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + // The third step's response arrives only once the stop is under way. + let releasePost = () => undefined as void; + const held = new Promise((resolve) => { + releasePost = resolve; + }); + const post = api.calibrateWorkingMemory.getMockImplementation()!; + api.calibrateWorkingMemory.mockImplementation(async (...args: unknown[]) => { + if (api.calibrateWorkingMemory.mock.calls.length === 3) { + await held; + } + return post(...args); + }); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(3); + + const attemptId = (api.calibrateWorkingMemory.mock.calls[0]![1] as { attemptId: string }).attemptId; + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + serveSummary( + summary({ runs: [1, 2, 3].map((number) => run(`run-${String(number)}`, number, 'completed', null, attemptId)) }) + ); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop', openAlertDialog()!)!)); + await act(() => releasePost()); + await expect.poll(openAlertDialog).toBeNull(); + + expect(api.calibrateWorkingMemory).toHaveBeenCalledTimes(3); + await expect.poll(sectionText).toContain('models.workingMemory.partialCanceled'); + expect(sectionText()).not.toContain('models.workingMemory.partialQueued'); + }); + + it('stops an attempt that is still compiling: the compiler is aborted and nothing is queued', async () => { + compile.mockImplementationOnce( + (_cases, signal) => + new Promise((_resolve, reject) => { + signal.addEventListener('abort', () => reject(new Error('aborted')), { once: true }); + }) + ); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => buttonNamed('models.workingMemory.stop')).toBeDefined(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.stop', openAlertDialog()!)!)); + + // The attempt left queuing: it reads as stopped, not as a failure, and Calibrate is available again. + await expect.poll(sectionText).toContain('models.workingMemory.canceled'); + expect(sectionText()).not.toContain('models.workingMemory.failed'); + expect(buttonNamed('models.workingMemory.stop', host)).toBeUndefined(); + expect(buttonNamed('models.workingMemory.calibrate')?.disabled).toBe(false); + expect(api.calibrateWorkingMemory).not.toHaveBeenCalled(); + }); + + it('heads the section with the current environment’s state, not an old environment’s', async () => { + serveSummary( + summary({ + lines: [line('unusable', ['torch'], { env_id: 'old-env' }), line('in_use')], + runs: [run('run-1', 0, 'completed')], + }) + ); + await renderSection(); + await expect.poll(() => host.querySelectorAll('table').length).toBe(2); + + const header = [...host.querySelectorAll('p')].find( + (element) => element.textContent === DEVICE.name + )!.parentElement!; + expect(header.textContent).toBe(`${DEVICE.name}models.workingMemory.stateInUse`); + expect(sectionText()).not.toContain('models.workingMemory.neverCalibrated'); + + // With only an old environment's lines, they head it, and this GPU counts as not calibrated. + serveSummary(summary({ lines: [line('unusable', ['torch'], { env_id: 'old-env' })] })); + await refreshSummary(); + await expect.poll(() => header.textContent).toBe(`${DEVICE.name}models.workingMemory.stateUnusable`); + expect(sectionText()).toContain('models.workingMemory.neverCalibrated'); + }); + + it('shows lines other models measured as in use, naming them, with nothing of this model to reset', async () => { + const other = { model_key: 'other', name: 'Other SDXL', windows: 6 }; + serveSummary(summary({ lines: [line('in_use', [], { measured_by: [other], own_windows: 0 })] })); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.usesOthers'); + + expect(sectionText()).toContain('models.workingMemory.measuredBy(models=Other SDXL)'); + expect(sectionText()).not.toContain('models.workingMemory.neverCalibrated'); + expect(buttonNamed('common.reset')).toBeUndefined(); + expect(buttonNamed('models.workingMemory.calibrate')?.disabled).toBe(false); + + // Once this model measured too, it is listed first and has measurements to reset. + serveSummary( + summary({ + lines: [line('in_use', [], { measured_by: [other, OWN], own_windows: 6 })], + runs: [run('run-1', 0, 'completed')], + }) + ); + await refreshSummary(); + await expect.poll(sectionText).toContain('models.workingMemory.measuredBy(models=This SDXL, Other SDXL)'); + expect(sectionText()).not.toContain('models.workingMemory.usesOthers'); + expect(buttonNamed('common.reset')).toBeDefined(); + }); + + it('keeps a finished attempt completed through a lost connection to the server', async () => { + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + + const attemptId = (api.calibrateWorkingMemory.mock.calls[0]![1] as { attemptId: string }).attemptId; + const finished = summary({ + lines: [line('in_use')], + runs: [1, 2, 3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'completed', null, attemptId)), + }); + serveSummary(finished); + await refreshSummary(); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + + // The server goes away: every read fails as the browser reports an unreachable host. + const unreachable = () => Promise.reject(new TypeError('Failed to fetch')); + api.getWorkingMemoryPlan.mockImplementation(unreachable); + api.getWorkingMemoryStatus.mockImplementation(unreachable); + serveSummary(new TypeError('Failed to fetch')); + await refreshSummary(); + await expect.poll(sectionText).toContain('models.workingMemory.loadFailed'); + + // It comes back with the same runs. + api.getWorkingMemoryPlan.mockImplementation(() => Promise.resolve(supportedPlan())); + api.getWorkingMemoryStatus.mockImplementation(() => Promise.resolve(status())); + serveSummary(finished); + await refreshSummary(); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + expect(sectionText()).not.toContain('models.workingMemory.failed'); + expect(sectionText()).not.toContain('Failed to fetch'); + }); + + it('lets an attempt that could not reach the server give way to one that ran after it', async () => { + const earlier = run('old', 0, 'completed', null, 'old'); + serveSummary(summary({ lines: [line('in_use')], runs: [earlier] })); + compile.mockImplementationOnce(() => Promise.reject(new TypeError('Failed to fetch'))); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('models.workingMemory.recalibrate')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(sectionText).toContain('models.workingMemory.failed'); + expect(sectionText()).toContain('Failed to fetch'); + + // Another tab, or a script, then calibrated to the end: that attempt is the latest one now. + serveSummary( + summary({ + lines: [line('in_use')], + runs: [ + earlier, + ...[1, 2, 3, 4, 5, 6].map((number) => run(`theirs-${String(number)}`, 100 + number, 'completed')), + ], + }) + ); + await refreshSummary(); + await expect.poll(sectionText).not.toContain('models.workingMemory.failed'); + expect(sectionText()).not.toContain('Failed to fetch'); + expect(buttonNamed('models.workingMemory.recalibrate')?.disabled).toBe(false); + }); + + it('keeps a size the fit leaves out in the table, muted, naming why, in columns the other tables share', async () => { + const counted = line('in_use').sizes; + serveSummary( + summary({ + lines: [ + // Its row headers are narrower than the other table's, as a size note makes them in any language. + line('in_use', [], { sizes: counted.map((size) => ({ ...size, height: 64, width: 64 })) }), + line('in_use', [], { + line: 'decode_fp32', + sizes: [ + ...counted, + { + counted: false, + excluded_reason: 'fragmented', + height: 1216, + measured_bytes: 5_900_000_000, + pixels: 1_478_656, + shipped_bytes: 4_100_000_000, + user_bytes: 3_300_000_000, + width: 1216, + windows: 3, + }, + ], + }), + ], + runs: [run('run-1', 0, 'completed')], + }) + ); + await renderSection(); + await expect.poll(() => host.querySelectorAll('table').length).toBe(2); + + const [plain, mixed] = [...host.querySelectorAll('table')]; + const [first, , excluded] = [...mixed!.querySelectorAll('tbody tr')]; + // The reason is part of the row header, so it is read with the size. + expect(excluded!.querySelector('th')!.textContent).toBe( + 'models.workingMemory.size(height=1216,width=1216)models.workingMemory.notCountedFragmented' + ); + expect(first!.querySelector('th')!.textContent).toBe('models.workingMemory.size(height=512,width=512)'); + const colors = (row: Element) => [...row.children].map((cell) => getComputedStyle(cell).color); + expect(new Set(colors(excluded!)).size).toBe(1); + expect(colors(excluded!)[0]).not.toBe(colors(first!)[0]); + expect([...excluded!.children].map((cell) => getComputedStyle(cell).textAlign)).toEqual([ + 'start', + 'end', + 'end', + 'end', + ]); + // The note under the size does not widen its column: both tables' columns start at the same place. + const columnEdges = (table: Element) => + [...table.querySelectorAll('thead th')].map((cell) => Math.round(cell.getBoundingClientRect().left)); + expect(columnEdges(mixed!)).toEqual(columnEdges(plain!)); + }); + + it('explains each value column from an info button that opens by keyboard', async () => { + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection(); + await expect.poll(() => host.querySelector('table')).not.toBeNull(); + + const headers = [...host.querySelectorAll('thead th')]; + // The size column needs no explanation. + expect(headers[0]!.querySelector('button')).toBeNull(); + + for (const [index, column, hint] of [ + [1, 'columnMeasured', 'hintMeasured'], + [2, 'columnShipped', 'hintShipped'], + [3, 'columnCalibrated', 'hintCalibrated'], + ] as const) { + const button = headers[index]!.querySelector('button')!; + const name = `models.workingMemory.aboutColumn(column=models.workingMemory.${column})`; + expect(button.getAttribute('aria-label')).toBe(name); + + button.focus(); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(openDialog).not.toBeNull(); + expect(openDialog()!.getAttribute('aria-label')).toBe(name); + expect(openDialog()!.textContent).toBe(`models.workingMemory.${hint}`); + + await act(() => userEvent.keyboard('{Escape}')); + await expect.poll(openDialog).toBeNull(); + await expect.poll(() => document.activeElement).toBe(button); + } + }); + + it('starts collapsed, keeps its panel out of the tab order, and asks for the plan only once opened', async () => { + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection({ expanded: false }); + + const toggle = sectionToggle(); + expect(toggle.getAttribute('aria-expanded')).toBe('false'); + const panel = document.getElementById(toggle.getAttribute('aria-controls')!)!; + expect(panel.hidden).toBe(true); + expect(buttonNamed('models.workingMemory.recalibrate')).toBeUndefined(); + // The plan reads the GPU's free memory on the server; a collapsed section never asks for it. + await expect.poll(() => api.getWorkingMemorySummary.mock.calls.length).toBeGreaterThan(0); + expect(api.getWorkingMemoryPlan).not.toHaveBeenCalled(); + + toggle.focus(); + await act(() => userEvent.keyboard('{Enter}')); + await expect.poll(() => buttonNamed('models.workingMemory.recalibrate')).toBeDefined(); + expect(toggle.getAttribute('aria-expanded')).toBe('true'); + expect(panel.hidden).toBe(false); + expect(api.getWorkingMemoryPlan).toHaveBeenCalledTimes(1); + + await act(() => userEvent.keyboard('[Space]')); + await expect.poll(() => panel.hidden).toBe(true); + expect(toggle.getAttribute('aria-expanded')).toBe('false'); + expect([...panel.querySelectorAll('button')].every((button) => button.getClientRects().length === 0)).toBe(true); + await act(() => userEvent.tab()); + expect(panel.contains(document.activeElement)).toBe(false); + }); + + it('names the GPU and the state in the collapsed header, and the progress while a calibration runs', async () => { + api.getWorkingMemoryStatus.mockImplementation(() => + Promise.resolve( + status({ + device_classes: [ + { + device_class: { + arch: null, + backend: 'cuda', + id: DEVICE.id, + name: DEVICE.name, + total_mib: 24_576, + vendor: 'x', + }, + device_class_id: DEVICE.id, + envs: [], + is_current: true, + measurement_method: 'allocator', + measurement_verified: true, + point_counts: [], + problem: null, + runs: [], + usable: true, + }, + ], + }) + ) + ); + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection({ expanded: false }); + + const header = sectionToggle().closest('h3')!.parentElement!; + await expect + .poll(() => header.textContent) + .toBe(`models.workingMemory.title${DEVICE.name}models.workingMemory.stateInUse`); + + // The attempt id carries its step count, so the header needs no plan to count. + const attemptId = 'attempt.6'; + serveSummary( + summary({ + runs: [ + run('run-1', 1, 'completed', null, attemptId), + run('run-2', 2, 'completed', null, attemptId), + ...[3, 4, 5, 6].map((number) => run(`run-${String(number)}`, number, 'running', null, attemptId)), + ], + }) + ); + await refreshSummary(); + await expect.poll(() => header.textContent).toContain('models.workingMemory.running(done=2,total=6)'); + expect(header.textContent).not.toContain('models.workingMemory.stateInUse'); + // A running calibration does not open the section. + expect(sectionToggle().getAttribute('aria-expanded')).toBe('false'); + expect(api.getWorkingMemoryPlan).not.toHaveBeenCalled(); + }); + + it('says the model was removed when the server no longer knows it', async () => { + serveSummary(new ApiError('{"detail":"The model could not be found"}', 404)); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.modelRemoved'); + + expect(sectionText()).not.toContain('models.workingMemory.loadFailed'); + expect(buttonNamed('common.retry')).toBeUndefined(); + }); + + it('resets after confirmation and restores focus', async () => { + serveSummary(summary({ lines: [line('in_use')], runs: [run('run-1', 0, 'completed')] })); + await renderSection(); + await expect.poll(() => buttonNamed('common.reset')).toBeDefined(); + + await act(() => userEvent.click(buttonNamed('common.reset')!)); + await expect.poll(openAlertDialog).not.toBeNull(); + // A destructive confirmation opens on Cancel. + expect(document.activeElement?.textContent).toBe('Cancel'); + + api.resetWorkingMemory.mockImplementation(() => { + serveSummary(summary()); + return Promise.resolve(); + }); + await act(() => userEvent.click(buttonNamed('common.reset', openAlertDialog()!)!)); + await expect.poll(openAlertDialog).toBeNull(); + + expect(api.resetWorkingMemory).toHaveBeenCalledWith(MODEL_KEY, expect.anything()); + await expect.poll(sectionText).toContain('models.workingMemory.neverCalibrated'); + await expect.poll(() => document.activeElement?.textContent).toBe('models.workingMemory.calibrate'); + }); + + it('disables Calibrate for an unsupported model, naming the reason', async () => { + api.getWorkingMemoryPlan.mockImplementation(() => + Promise.resolve({ + ...supportedPlan(), + cases: [], + reason: 'The inpaint variant does not generate from text alone.', + supported: false, + }) + ); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + const button = buttonNamed('models.workingMemory.calibrate')!; + expect(button.disabled).toBe(true); + expect(document.getElementById(button.getAttribute('aria-describedby')!)?.textContent).toBe( + 'The inpaint variant does not generate from text alone.' + ); + expect(sectionText()).not.toContain('models.workingMemory.neverCalibrated'); + }); + + it('disables Calibrate without exactly one GPU, naming the status reason', async () => { + api.getWorkingMemoryStatus.mockImplementation(() => + Promise.resolve( + status({ + available: false, + current_device_class_id: null, + unavailable_reason: 'Calibration needs exactly one generation device; this machine has two.', + }) + ) + ); + serveSummary(new ApiError('{"detail":"No device_class_id was given and there is no current one"}', 409)); + await renderSection(); + await expect.poll(() => buttonNamed('models.workingMemory.calibrate')).toBeDefined(); + + const button = buttonNamed('models.workingMemory.calibrate')!; + expect(button.disabled).toBe(true); + expect(document.getElementById(button.getAttribute('aria-describedby')!)?.textContent).toBe( + 'Calibration needs exactly one generation device; this machine has two.' + ); + // A missing device class is a state, not a load failure. + expect(sectionText()).not.toContain('models.workingMemory.loadFailed'); + }); + + it('names an unusable store and flags hardware the method was not verified on', async () => { + serveSummary( + summary({ + measurement_verified: false, + problem: 'schema_version 9 is newer than this version reads', + usable: false, + }) + ); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.notVerified'); + + expect(buttonNamed('models.workingMemory.calibrate')?.disabled).toBe(true); + expect(sectionText()).toContain( + 'models.workingMemory.blockedStoreUnusable(problem=schema_version 9 is newer than this version reads)' + ); + }); + + it('explains stale measurements and offers to recalibrate', async () => { + serveSummary( + summary({ lines: [line('stale', ['driver', 'setting.attention_type'])], runs: [run('run-1', 0, 'completed')] }) + ); + await renderSection(); + await expect + .poll(sectionText) + .toContain('models.workingMemory.helpStale(facts=models.workingMemory.facts.driver, attention_type)'); + + expect(sectionText()).toContain('models.workingMemory.stateStale'); + expect(buttonNamed('models.workingMemory.recalibrate')?.disabled).toBe(false); + }); + + it('reports a failed attempt with its error and offers Retry', async () => { + serveSummary(summary({ runs: [run('run-1', 0, 'completed'), run('run-2', 1, 'failed', 'CUDA out of memory')] })); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.failed'); + + expect(sectionText()).toContain('CUDA out of memory'); + expect(buttonNamed('common.retry')?.disabled).toBe(false); + + // Retry is a new attempt: a new id, and retry keys that cannot collide with the failed attempt's. + await act(() => userEvent.click(buttonNamed('common.retry')!)); + await expect.poll(openDialog).not.toBeNull(); + await act(() => userEvent.click(buttonNamed('models.workingMemory.calibrate', openDialog()!)!)); + await expect.poll(() => api.calibrateWorkingMemory.mock.calls.length).toBe(6); + const requests = api.calibrateWorkingMemory.mock.calls.map( + ([, request]) => request as { attemptId: string; idempotencyKey: string } + ); + expect(new Set(requests.map((request) => request.attemptId)).size).toBe(1); + expect(requests[0]!.attemptId).not.toBe('earlier-attempt'); + expect(requests.every((request) => request.idempotencyKey.includes(requests[0]!.attemptId))).toBe(true); + }); + + it('shows no notice where the method is verified and leaves runs without an attempt id on their own', async () => { + serveSummary( + summary({ + lines: [line('in_use')], + runs: [run('legacy-1', 0, 'failed', 'old failure', null), run('legacy-2', 1, 'completed', null, null)], + }) + ); + await renderSection(); + await expect.poll(sectionText).toContain('models.workingMemory.stateInUse'); + + expect(sectionText()).not.toContain('models.workingMemory.notVerified'); + // Only the newest legacy run is the latest attempt, so the older failure is not reported. + expect(sectionText()).not.toContain('models.workingMemory.failed'); + }); + + it('shows a skeleton while loading and a retryable error when a read fails', async () => { + let resolvePlan: (plan: WorkingMemoryPlan) => void = () => undefined; + api.getWorkingMemoryPlan.mockImplementationOnce( + () => + new Promise((resolve) => { + resolvePlan = resolve; + }) + ); + let failStatus: (error: Error) => void = () => undefined; + api.getWorkingMemoryStatus.mockImplementationOnce( + () => + new Promise((_resolve, reject) => { + failStatus = reject; + }) + ); + await renderSection(); + + // Opened for the first time, the panel waits for the plan. + await expect.poll(() => host.querySelector('[aria-busy="true"]')).not.toBeNull(); + await act(() => { + resolvePlan(supportedPlan()); + failStatus(new ApiError('{"detail":"store locked"}', 503)); + }); + await expect.poll(sectionText).toContain('models.workingMemory.loadFailed'); + expect(sectionText()).toContain('store locked'); + + await act(() => userEvent.click(buttonNamed('common.retry')!)); + await expect.poll(sectionText).toContain('models.workingMemory.neverCalibrated'); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.tsx new file mode 100644 index 00000000000..cef121864d5 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/WorkingMemorySection.tsx @@ -0,0 +1,482 @@ +/* eslint-disable react-perf/jsx-no-new-function-as-prop */ +import type { + CalibrationActivity, + CalibrationBlocker, + CalibrationCoverage, + CalibrationOutcome, + WorkingMemoryPlan, + WorkingMemoryStatus, + WorkingMemorySummary, +} from '@features/models/core/workingMemory'; + +import { Alert, Collapsible, HStack, Progress, Skeleton, Stack, Text } from '@chakra-ui/react'; +import { + deriveCalibrationProgress, + describeCoverage, + getCalibrationBlocker, + isMeasurementMethodVerified, + planCalibrationSteps, +} from '@features/models/core/workingMemory'; +import { useModelsSelector } from '@features/models/data/modelsStore'; +import { + attachWorkingMemoryRealtime, + resetCalibration, + startCalibration, + stopCalibration, + useLocalAttempt, +} from '@features/models/data/workingMemoryCalibration'; +import { + workingMemoryPlanQueryOptions, + workingMemoryStatusQueryOptions, + workingMemorySummaryQueryOptions, +} from '@features/models/data/workingMemoryQueries'; +import { useCalibrationGraphCompiler } from '@features/models/ui/ModelsUiContext'; +import { useNotify } from '@features/models/ui/useModelsNotify'; +import { useExitPresence } from '@platform/react/useExitRetainedValue'; +import { useMountEffect } from '@platform/react/useMountEffect'; +import { useConnectionStatusSelector } from '@platform/transport/connectionStore'; +import { ApiError, getApiErrorMessage } from '@platform/transport/http'; +import { Button } from '@platform/ui/Button'; +import { ConfirmDialog } from '@platform/ui/ConfirmDialog'; +import { useQuery, useQueryClient } from '@tanstack/react-query'; +import { ChevronRightIcon } from 'lucide-react'; +import { useId, useRef, useState, type RefObject } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { WorkingMemoryCalibrateDialog } from './WorkingMemoryCalibrateDialog'; +import { LineMeasurements, LineStateBadge, SharedModelsList, useCaseLabel } from './WorkingMemoryParts'; + +const INDICATOR_OPEN_STYLES = { transform: 'rotate(90deg)' } as const; + +/** + * Working-memory calibration of one model on this machine's GPU: what was measured and whether estimates use it, + * and the runs that measure it. Every read is the server's; the section keeps no copy of a run's state, so a reload or + * a reconnect shows the same thing. + * + * Collapsed on every visit; its header still names the GPU and the state, or the progress of a running calibration. + */ +export const WorkingMemorySection = ({ modelKey }: { modelKey: string }) => { + const { t } = useTranslation(); + const queryClient = useQueryClient(); + const headingId = useId(); + const triggerRef = useRef(null); + const [isOpen, setIsOpen] = useState(false); + // The plan measures the GPU's free memory on the server, so it loads only once the panel was opened. + const [wasOpened, setWasOpened] = useState(false); + const local = useLocalAttempt(modelKey); + const plan = useQuery({ ...workingMemoryPlanQueryOptions(modelKey), enabled: wasOpened }); + const isConnected = useConnectionStatusSelector((snapshot) => snapshot.status === 'connected'); + const summary = useQuery( + workingMemorySummaryQueryOptions(modelKey, { isConnected, isSubmitting: local?.submitting ?? false }) + ); + const status = useQuery(workingMemoryStatusQueryOptions()); + const activityCaption = useActivityCaption(); + + useMountEffect(() => attachWorkingMemoryRealtime(queryClient)); + + // Without the plan, an attempt's step count comes from this tab's record or the attempt id. + const progress = deriveCalibrationProgress({ + local, + runs: summary.data?.runs ?? [], + steps: plan.data ? planCalibrationSteps(plan.data) : [], + }); + const coverage = describeCoverage(summary.data ?? null); + const deviceName = plan.data?.device_class?.name ?? (status.data ? currentDeviceName(status.data) : null); + const failed = [plan, summary, status].find((query) => query.isError); + let body; + + // The plan and the summary know only installed models: this one was removed while its details were open. + if (failed?.error instanceof ApiError && failed.error.status === 404) { + body = ( + + {t('models.workingMemory.modelRemoved')} + + ); + } else if (failed) { + body = ( + + + {t('models.workingMemory.loadFailed')} + {': '} + {getApiErrorMessage(failed.error, t('common.unknownError'))} + + + + ); + } else if (!wasOpened) { + body = null; + } else if (plan.data === undefined || summary.data === undefined || status.data === undefined) { + body = ( + + + + + ); + } else { + body = ( + + ); + } + + return ( + { + setIsOpen(open); + if (open) { + setWasOpened(true); + } + }} + > + + + + + + + + + {t('models.workingMemory.title')} + + + {/* Collapsed, the header says what the open panel would; open, the panel's own row does. */} + {!isOpen ? ( + + {deviceName ? ( + + {deviceName} + + ) : null} + {progress.activity.kind === 'running' ? ( + + {activityCaption(progress.activity)} + + ) : summary.data?.usable ? ( + + ) : null} + + ) : null} + + + {t('models.workingMemory.description')} + + + {body} + + + ); +}; + +const currentDeviceName = (status: WorkingMemoryStatus): string | null => + status.device_classes.find((entry) => entry.is_current)?.device_class?.name ?? null; + +const WorkingMemoryBody = ({ + coverage, + deviceName, + focusFallbackRef, + modelKey, + plan, + progress: { activity, outcome }, + status, + summary, +}: { + coverage: CalibrationCoverage; + deviceName: string | null; + /** Takes focus back when a confirmation's opener is gone or disabled. */ + focusFallbackRef: RefObject; + modelKey: string; + plan: WorkingMemoryPlan; + progress: ReturnType; + status: WorkingMemoryStatus; + summary: WorkingMemorySummary | null; +}) => { + const { t } = useTranslation(); + const notify = useNotify(); + const queryClient = useQueryClient(); + const compile = useCalibrationGraphCompiler(); + const reasonId = useId(); + const primaryRef = useRef(null); + const stopRef = useRef(null); + const [isCalibrateOpen, setIsCalibrateOpen] = useState(false); + const [isStopOpen, setIsStopOpen] = useState(false); + const [isResetOpen, setIsResetOpen] = useState(false); + const calibrateDialog = useExitPresence(isCalibrateOpen); + const throughName = useModelsSelector((snapshot) => + plan.calibrated_through ? (snapshot.modelsByKey.get(plan.calibrated_through)?.name ?? null) : null + ); + + const isRunning = activity.kind === 'running'; + const blocker: CalibrationBlocker | { kind: 'noCompiler' } | null = + compile === null ? { kind: 'noCompiler' } : getCalibrationBlocker({ isRunning, plan, status, summary }); + const lines = summary?.lines ?? []; + const verified = isMeasurementMethodVerified(summary, status); + const primaryLabel = + outcome.kind === 'failed' || outcome.kind === 'partial' + ? t('common.retry') + : coverage.hasOwnMeasurements + ? t('models.workingMemory.recalibrate') + : t('models.workingMemory.calibrate'); + const blockerText = useBlockerText(blocker); + + // The opener of a confirmation can disappear or be disabled once it is done; the section's toggle stays. + const getFinalFocus = () => + primaryRef.current && !primaryRef.current.disabled ? primaryRef.current : focusFallbackRef.current; + // Starting records the attempt synchronously, so Stop has rendered by the time the closing dialog restores focus. + const getCalibrateFinalFocus = () => stopRef.current ?? getFinalFocus(); + + const confirmCalibration = () => { + setIsCalibrateOpen(false); + + if (compile) { + void startCalibration({ compile, modelKey, plan, queryClient }); + } + }; + + return ( + + + + {deviceName ? ( + + {deviceName} + + ) : null} + {/* Without a current device class or a readable store nothing can be said about this GPU's measurements. */} + {summary?.usable ? : null} + + + {isRunning ? ( + + ) : null} + {coverage.hasOwnMeasurements ? ( + + ) : null} + + + + + {blockerText ? ( + + {blockerText} + + ) : null} + + + + + {verified === false ? ( + + + + {t('models.workingMemory.notVerified')} + + + ) : null} + + {!coverage.hasCurrentLines && activity.kind === 'idle' && plan.supported && summary?.usable ? ( + + {t('models.workingMemory.neverCalibrated')} + + ) : null} + {coverage.usesOthersOnly ? ( + + {t('models.workingMemory.usesOthers')} + + ) : null} + + {lines.length > 0 ? ( + + {lines.map((line) => ( + + ))} + + ) : null} + + {plan.shared_with.length > 0 ? : null} + + + {plan.not_covered.length > 0 ? {t('models.workingMemory.encodeNotCovered')} : null} + {t('models.workingMemory.floorNote')} + + + {calibrateDialog.isMounted ? ( + setIsCalibrateOpen(false)} + onConfirm={confirmCalibration} + onExitComplete={calibrateDialog.release} + /> + ) : null} + setIsStopOpen(false)} + onConfirm={async () => { + try { + await stopCalibration({ modelKey, queryClient }); + } catch (error) { + notify.error(t('models.workingMemory.stopFailed'), getApiErrorMessage(error, '')); + } + }} + /> + setIsResetOpen(false)} + onConfirm={async () => { + try { + await resetCalibration({ modelKey, queryClient }); + notify.success(t('models.workingMemory.resetDone')); + } catch (error) { + notify.error(t('models.workingMemory.resetFailed'), getApiErrorMessage(error, '')); + } + }} + /> + + ); +}; + +const useBlockerText = (blocker: CalibrationBlocker | { kind: 'noCompiler' } | null): string | null => { + const { t } = useTranslation(); + + switch (blocker?.kind) { + case undefined: + return null; + case 'unsupported': + return blocker.reason ?? t('models.workingMemory.blockedUnsupported'); + case 'unavailable': + return blocker.reason ?? t('models.workingMemory.blockedUnavailable'); + case 'storeUnusable': + return t('models.workingMemory.blockedStoreUnusable', { problem: blocker.reason ?? '' }); + case 'running': + return t('models.workingMemory.blockedRunning'); + case 'noCompiler': + return t('models.workingMemory.blockedUnavailable'); + } +}; + +/** "Queuing: 2/18" while steps are posted, then "Calibrating: 5/18 runs". */ +const useActivityCaption = () => { + const { t } = useTranslation(); + + return ({ done, queuing, total }: Extract): string => + queuing ? t('models.workingMemory.queuing', { done, total }) : t('models.workingMemory.running', { done, total }); +}; + +const ActivityPanel = ({ activity }: { activity: CalibrationActivity }) => { + const { t } = useTranslation(); + const caseLabel = useCaseLabel(); + const activityCaption = useActivityCaption(); + + if (activity.kind !== 'running') { + return null; + } + + const { currentCase, done, total } = activity; + const caption = activityCaption(activity); + + return ( + + + {/* The track is the progressbar; without a name of its own it would be announced by its percentage. */} + + + + + + {caption} + {currentCase ? ` · ${caseLabel(currentCase)}` : null} + + + ); +}; + +const OutcomePanel = ({ outcome }: { outcome: CalibrationOutcome }) => { + const { t } = useTranslation(); + + if (outcome.kind === 'none' || outcome.kind === 'completed') { + return null; + } + + if (outcome.kind === 'canceled') { + return ( + + {t('models.workingMemory.canceled')} + + ); + } + + const isFailed = outcome.kind === 'failed'; + const detail = + outcome.error || + (outcome.kind === 'partial' + ? !outcome.canceled && outcome.queued < outcome.total + ? t('models.workingMemory.partialQueued', { queued: outcome.queued, total: outcome.total }) + : t('models.workingMemory.partialCanceled') + : null); + + return ( + + + + {isFailed ? t('models.workingMemory.failed') : t('models.workingMemory.partial')} + {detail ? {detail} : null} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/queue/contracts.ts b/invokeai/frontend/webv2/src/features/queue/contracts.ts index d9374a3a87c..4a9b43cca76 100644 --- a/invokeai/frontend/webv2/src/features/queue/contracts.ts +++ b/invokeai/frontend/webv2/src/features/queue/contracts.ts @@ -57,6 +57,11 @@ export { shouldSubmitPendingQueueItem, } from './core/submissionRules'; export { extractGenerationMeta, getResultImageName, type QueueGenerationMeta } from './core/generationMeta'; +export { + planGenerateSubmissionBatch, + type GeneratePromptBatchPlan, + type GeneratePromptSubmission, +} from './core/promptBatch'; export { buildProjectQueueItemOriginPrefix, buildQueueItemOrigin, diff --git a/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts b/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts index 1a42d04dfbb..36339aac9bd 100644 --- a/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts +++ b/invokeai/frontend/webv2/src/features/queue/core/promptBatch.ts @@ -339,3 +339,37 @@ export const buildGeneratePromptBatchPlan = ({ runs: isHeld ? iterations : 1, }; }; + +/** The prompt and seed fields of a generate submission, from compilation to `enqueue_batch`. */ +export interface GeneratePromptSubmission { + batchCount: number; + negativePrompt: string; + negativePromptNodeId: string; + positivePrompt: string; + positivePromptNodeId: string; + /** + * Prompts to submit as a batch dimension, already expanded by the caller. + * Absent or empty submits `positivePrompt` verbatim. + */ + positivePrompts?: string[]; + seed: number; + seedBehaviour?: QueuePromptSeedBehaviour; + seedNodeId: string; + seedStep: QueueSeedStep; + /** Legacy recovery maps the old random toggle to seedStep and preserves its expansion rules. */ + legacySeedPlan?: true; +} + +/** The batch data and run count `enqueue_batch` receives for a generate submission. */ +export const planGenerateSubmissionBatch = (submission: GeneratePromptSubmission): GeneratePromptBatchPlan => + (submission.legacySeedPlan ? buildLegacyGeneratePromptBatchPlan : buildGeneratePromptBatchPlan)({ + batchCount: sanitizeBatchCount(submission.batchCount), + negativePrompt: submission.negativePrompt, + negativePromptNodeId: submission.negativePromptNodeId, + positivePromptNodeId: submission.positivePromptNodeId, + prompts: submission.positivePrompts?.length ? submission.positivePrompts : [submission.positivePrompt], + seed: submission.seed, + seedBehaviour: submission.seedBehaviour ?? 'per-iteration', + seedNodeId: submission.seedNodeId, + seedStep: submission.seedStep, + }); diff --git a/invokeai/frontend/webv2/src/features/queue/core/types.ts b/invokeai/frontend/webv2/src/features/queue/core/types.ts index e2dd92e53b6..f1f07df5ea8 100644 --- a/invokeai/frontend/webv2/src/features/queue/core/types.ts +++ b/invokeai/frontend/webv2/src/features/queue/core/types.ts @@ -1,4 +1,5 @@ import type { + GeneratePromptSubmission, QueuePromptSeedBehaviour, QueueSeedStep, QueueWorkflowBatchDatum, @@ -58,23 +59,7 @@ export interface QueueEnqueueWorkflowRequest extends QueueEnqueueRequestBase { batchData?: QueueWorkflowBatchDatum[][]; } -export interface QueueEnqueueGenerateRequest extends QueueEnqueueRequestBase { - negativePrompt: string; - negativePromptNodeId: string; - positivePrompt: string; - positivePromptNodeId: string; - /** - * Prompts to submit as a batch dimension, already expanded by the caller. - * Absent or empty submits `positivePrompt` verbatim. - */ - positivePrompts?: string[]; - seed: number; - seedBehaviour?: QueuePromptSeedBehaviour; - seedNodeId: string; - seedStep: QueueSeedStep; - /** Legacy recovery maps the old random toggle to seedStep and preserves its expansion rules. */ - legacySeedPlan?: true; -} +export interface QueueEnqueueGenerateRequest extends QueueEnqueueRequestBase, GeneratePromptSubmission {} export interface QueueEnqueueResult { batchId?: string; diff --git a/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts b/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts index ebbe288f77c..ad61dab8aa0 100644 --- a/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts +++ b/invokeai/frontend/webv2/src/features/queue/data/submissionApi.ts @@ -8,12 +8,7 @@ import type { QueueResultVideoOptions, } from '@features/queue/core/types'; -import { - buildGeneratePromptBatchPlan, - buildLegacyGeneratePromptBatchPlan, - buildQueueWorkflowBatchPlan, - sanitizeBatchCount, -} from '@features/queue/core/promptBatch'; +import { buildQueueWorkflowBatchPlan, planGenerateSubmissionBatch } from '@features/queue/core/promptBatch'; import { mapWithConcurrency } from '@platform/core/concurrency'; import { addOutputImageNames } from '@platform/core/outputImages'; import { assertAccountScopeCurrent, captureAccountScope } from '@platform/state/accountLifecycle'; @@ -78,20 +73,7 @@ const mapEnqueueResult = (value: unknown, isReceipt = false): QueueEnqueueResult }; export const enqueueGenerate = async (request: QueueEnqueueGenerateRequest): Promise => { - const planInput = { - batchCount: sanitizeBatchCount(request.batchCount), - negativePrompt: request.negativePrompt, - negativePromptNodeId: request.negativePromptNodeId, - positivePromptNodeId: request.positivePromptNodeId, - prompts: request.positivePrompts?.length ? request.positivePrompts : [request.positivePrompt], - seed: request.seed, - seedBehaviour: request.seedBehaviour ?? 'per-iteration', - seedNodeId: request.seedNodeId, - }; - const plan = (request.legacySeedPlan ? buildLegacyGeneratePromptBatchPlan : buildGeneratePromptBatchPlan)({ - ...planInput, - seedStep: request.seedStep, - }); + const plan = planGenerateSubmissionBatch(request); const result = await apiFetchJson('/api/v1/queue/default/enqueue_batch', { body: JSON.stringify({ batch: { diff --git a/invokeai/frontend/webv2/src/workbench/workbenchState.ts b/invokeai/frontend/webv2/src/workbench/workbenchState.ts index 257a73739ad..46c7c0268cf 100644 --- a/invokeai/frontend/webv2/src/workbench/workbenchState.ts +++ b/invokeai/frontend/webv2/src/workbench/workbenchState.ts @@ -85,7 +85,6 @@ import { type GeneratedVideoContract, generatedVideoToGalleryItem, } from '@features/gallery/contracts'; -import { planSeedSubmission } from '@platform/core/seed'; import { describeError } from '@platform/logging/normalize'; import { WIDGET_REGIONS } from '@workbench/layoutContracts'; import { prependProjectEvent, PROJECT_EVENT_LIMIT } from '@workbench/projectEvents'; @@ -141,14 +140,13 @@ import { applyProjectPromptDraft, cloneGenerateWidgetValues, createDynamicPromptsSampleSeed, - getEffectivePrompts, getGenerationModelAvailabilityReasons, getPromptDraftFromValues, getPromptHistoryItemFromGenerateSettings, - hasDynamicPromptSyntax, migrateProjectPromptDraft, normalizeGenerateSettings, normalizeGenerateWidgetValues, + planGenerateSubmission, removePromptHistoryItem, sanitizeBatchCount, syncGenerateWidgetValuesWithModels, @@ -3578,36 +3576,21 @@ const enqueueCompiledSnapshot = ( : route.sourceId === 'video' ? videoSettings : null; - // Resolve template-merged prompts here, where Generate and Canvas submissions converge; Upscale merges to - // identity. - const effectivePrompts = sourceGenerateSettings ? getEffectivePrompts(sourceGenerateSettings) : null; - // Record expansions for Generate routes even when only one prompt results. Inspect merged prompts because - // templates may introduce dynamic syntax. - const expandedPositivePrompts = - route.sourceId !== 'upscale' && - route.sourceId !== 'video' && - compiled.positivePrompts && - compiled.positivePrompts.length > 0 && - effectivePrompts && - hasDynamicPromptSyntax(effectivePrompts.positivePrompt) - ? compiled.positivePrompts - : undefined; - const expandedSeedBehaviour = expandedPositivePrompts - ? (canvasGenerateSettings ?? generateSettings)?.dynamicPromptsSeedBehaviour - : undefined; + // Resolve template-merged prompts and seeds here, where Generate and Canvas submissions converge; Upscale and Video + // merge to identity and never expand. + const submissionPlan = sourceGenerateSettings + ? planGenerateSubmission( + sourceGenerateSettings, + route.sourceId === 'upscale' || route.sourceId === 'video' ? undefined : compiled.positivePrompts + ) + : null; + const effectivePrompts = submissionPlan?.effectivePrompts ?? null; + const expandedPositivePrompts = submissionPlan?.submission.positivePrompts; // The draw the expansion used. Generate expands before dispatch, so the project's sample seed may have rotated // since; recall and rotation follow the expansion, not the project. const expansionSampleSeed = expandedPositivePrompts ? (compiled.positivePromptsSampleSeed ?? null) : null; // Compilation already reserved the starting seed; failures and cancellations do not roll the sequence back. - const seedPlan = sourceGenerateSettings - ? planSeedSubmission({ - batchCount: sourceGenerateSettings.batchCount, - promptCount: expandedPositivePrompts?.length ?? 1, - seedBehaviour: expandedSeedBehaviour ?? 'per-iteration', - seedMode: sourceGenerateSettings.seedMode, - startSeed: sourceGenerateSettings.seed, - }) - : null; + const seedPlan = submissionPlan?.seedPlan ?? null; const backendSubmission: QueueCompiledSubmission = !backendGraph ? { error: `${route.sourceId} queue item is missing a compiled backend graph.`, kind: 'invalid' } : route.sourceId === 'workflow' @@ -3624,21 +3607,14 @@ const enqueueCompiledSnapshot = ( // active workflow changes. ...(compiled.projectWorkflowId ? { projectWorkflowId: compiled.projectWorkflowId } : {}), } - : sourceGenerateSettings && effectivePrompts + : submissionPlan ? { - batchCount: sourceGenerateSettings.batchCount, + ...submissionPlan.submission, graph: backendGraph, kind: 'generate', - // Disabling the negative prompt also suppresses the template's negative side. - negativePrompt: sourceGenerateSettings.negativePromptEnabled ? effectivePrompts.negativePrompt : '', negativePromptNodeId: generate?.negativePromptNodeId ?? 'negative_prompt', - positivePrompt: effectivePrompts.positivePrompt, positivePromptNodeId: generate?.positivePromptNodeId ?? 'positive_prompt', - ...(expandedPositivePrompts ? { positivePrompts: expandedPositivePrompts } : {}), - seed: sourceGenerateSettings.seed, - ...(expandedSeedBehaviour ? { seedBehaviour: expandedSeedBehaviour } : {}), seedNodeId: generate?.seedNodeId ?? 'seed', - seedStep: seedPlan?.step ?? 0, } : { error: `${route.sourceId} queue item is missing source submission metadata.`, kind: 'invalid' }; // A workflow compiles every node's board in (Auto included), so its board-less results stay Uncategorized. diff --git a/scripts/calibrate_flux2_working_memory.py b/scripts/calibrate_flux2_working_memory.py index c429c5b9a49..783fe00d04f 100644 --- a/scripts/calibrate_flux2_working_memory.py +++ b/scripts/calibrate_flux2_working_memory.py @@ -7,8 +7,9 @@ 1. ``SDPA_MATH_BYTES_PER_SCORE_ELEMENT`` (``backend/util/attention.py``) -- bytes per element of the score matrix, charged only where SDPA has no fused kernel and materializes it. -2. ``estimate_vae_working_memory_flux2``'s ``k_decode`` / ``k_encode`` bytes per pixel per element byte, - shipped per convolution backend in ``backend/working_memory/data/vae.json``. +2. ``estimate_vae_working_memory_flux2``'s convolution lines, ``k__`` bytes per pixel plus a + ``b__`` base, shipped per convolution backend and allocator in + ``backend/working_memory/data/vae.json``. 3. ``Flux2DenoiseInvocation._estimate_working_memory``'s 0.4 MB per token, defined at the Klein 9B width (4096) and scaled linearly by the loaded variant's width. 4. The dispatch question underneath all of it: does this build's SDPA fuse these shapes, or does it @@ -75,9 +76,12 @@ MIB = 1024**2 -def _shipped_vae_coefficient(operation: str) -> float: - """The VAE coefficient this build's estimate uses for `operation`: the column its convolution backend selects.""" - return shipped_coefficients(VAE_FLUX2_SPEC, platform_for(torch.device("cuda"))).values[f"k_{operation}"] +def _shipped_vae_line(operation: str, dtype_name: str) -> tuple[float, float]: + """The VAE line this build's estimate uses for `operation` in `dtype_name`: bytes per pixel and base bytes, as the + platform (convolution backend, allocator) selects them.""" + dtype = {"float16": "fp16", "bfloat16": "bf16", "float32": "fp32"}[dtype_name] + values = shipped_coefficients(VAE_FLUX2_SPEC, platform_for(torch.device("cuda"))).values + return values[f"k_{operation}_{dtype}"], values[f"b_{operation}_{dtype}"] # The FLUX.2 VAE attends on the 8x-downsampled grid with a single 512-wide head. Mirrors @@ -285,13 +289,13 @@ def run() -> None: "reserved_delta": peak, "estimate": estimate, "score_term": score_bytes, - # The whole measured peak over the pixel area, which is what the `k_decode`/`k_encode` - # coefficients name. The score matrix is deliberately NOT backed out: the two terms peak in different + # The whole measured peak over the pixel area: the line's bytes per pixel plus its base spread over + # that area. The score matrix is deliberately NOT backed out: the two terms peak in different # phases of the forward, so the estimate takes the larger of them rather than the sum, and # the convolution phase is what sets the high-water mark until the quadratic term overtakes # it past ~1280px. Where `score_term` exceeds the measured peak, this column is not the # quantity being fitted -- the score model is, and it is bounding rather than describing. - "implied_linear_constant": peak / (px * px * element_size), + "implied_linear_constant": peak / (px * px), "covered": peak <= estimate, } @@ -464,10 +468,12 @@ def report_sdpa(shapes: list[tuple[int, int, int]], dtype_name: str) -> list[dic def report_vae(pxs: list[int], dtype_name: str, vae_path: str | None) -> list[dict]: - decode_k = _shipped_vae_coefficient("decode") - encode_k = _shipped_vae_coefficient("encode") - print(f"\n=== 3. VAE linear coefficients (this build selects {decode_k:.0f} decode / {encode_k:.0f} encode) ===") - print("`implied_k` is the measured peak over pixel area, directly comparable to those coefficients;") + (decode_k, decode_b), (encode_k, encode_b) = (_shipped_vae_line(op, dtype_name) for op in ("decode", "encode")) + print( + f"\n=== 3. VAE lines (this build selects {decode_k:.0f} B/px + {decode_b / MIB:.0f} MiB decode, " + f"{encode_k:.0f} B/px + {encode_b / MIB:.0f} MiB encode) ===" + ) + print("`implied_k` is the measured peak over pixel area, comparable to those bytes per pixel plus the base;") print("fit on the rows whose `math` column matches what this build really does (section 1).") print("`covered` is the question that matters: is the shipped estimate an upper bound here?") print("Caveat: forcing math on a build that HAS a fused kernel is not equivalent to a build that") @@ -502,10 +508,10 @@ def report_vae(pxs: list[int], dtype_name: str, vae_path: str | None) -> list[di _flag_non_monotonic(rows) for operation in ("decode", "encode"): # Compare against the column this build actually selects, not against a hard-coded number. - shipped = _shipped_vae_coefficient(operation) + shipped_k, shipped_b = _shipped_vae_line(operation, dtype_name) for force_math in (False, True): ks = [ - r["implied_linear_constant"] + r["implied_linear_constant"] - shipped_b / (r["px"] * r["px"]) for r in rows if r["operation"] == operation and r["force_math"] is force_math @@ -517,8 +523,11 @@ def report_vae(pxs: list[int], dtype_name: str, vae_path: str | None) -> list[di if not ks: continue mode = "math" if force_math else "fused" - verdict = "OK" if shipped >= max(ks) else "SHORT" - print(f" {operation} ({mode}): implied_k max = {max(ks):.0f}, shipped = {shipped:.0f} -> {verdict}") + verdict = "OK" if shipped_k >= max(ks) else "SHORT" + print( + f" {operation} ({mode}): implied_k max past the base = {max(ks):.0f}, shipped = {shipped_k:.0f} " + f"-> {verdict}" + ) short = [r for r in rows if not r.get("oom") and not r["covered"]] if short: print("\n Points the shipped estimate does NOT cover:") diff --git a/scripts/family_e2e/__init__.py b/scripts/family_e2e/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/scripts/family_e2e/__main__.py b/scripts/family_e2e/__main__.py new file mode 100644 index 00000000000..db6d3f294e1 --- /dev/null +++ b/scripts/family_e2e/__main__.py @@ -0,0 +1,264 @@ +"""Command line of the family E2E tool: `python -m scripts.family_e2e `. + +See docs/src/content/docs/development/Guides/family-e2e.mdx. +""" + +from __future__ import annotations + +import argparse +import logging +import os +import re +import sys +from datetime import datetime, timezone +from pathlib import Path + +from scripts.family_e2e.calibration import PROBE_REASON, CalibrationError, save_store_files +from scripts.family_e2e.detach import RUNNER_LOG, DetachError, spawn_runner +from scripts.family_e2e.gpu_memory import UnsupportedPlatformError +from scripts.family_e2e.graphs import ExportError +from scripts.family_e2e.manifest import DEFAULT_ARM, Manifest, ManifestError, load_manifest +from scripts.family_e2e.records import RUN_INFO_FILE, load_records, load_run_info, write_run_info +from scripts.family_e2e.references import AcceptError, accept +from scripts.family_e2e.report import write_report +from scripts.family_e2e.runner import ( + RunnerError, + gpu_lock, + is_alive, + kill_identified, + load_run_manifest, + new_run, + now_iso, + open_gpu, + preflight, + read_heartbeat, + remaining_steps, + remove_scratch_roots, + run, + scratch_root, +) +from scripts.family_e2e.server import ServerError + +logger = logging.getLogger("family_e2e") +RUN_ID = re.compile(r"^\d{8}-\d{6}-(?P.+)$") +EXPECTED_ERRORS = ( + ManifestError, + RunnerError, + ServerError, + ExportError, + AcceptError, + DetachError, + UnsupportedPlatformError, + CalibrationError, + # load_records names the malformed line of a damaged records.jsonl. + ValueError, +) + + +class UsageError(ValueError): + pass + + +def _data_dir(run_dir: Path) -> Path: + # /runs/ + return run_dir.parent.parent + + +def _resolve_run(arg: str | None, manifest_path: Path | None) -> Path: + """A run directory, a run id of the manifest's data dir, or (no argument) the manifest profile's latest run.""" + if arg is not None and Path(arg).is_dir(): + return Path(arg).resolve() + if manifest_path is None: + raise UsageError("pass a run directory, or --manifest to look up run ids and the latest run") + manifest = load_manifest(manifest_path) + runs_dir = manifest.data_dir / "runs" + if arg is not None: + if not (runs_dir / arg / RUN_INFO_FILE).is_file(): + raise UsageError(f"no run {arg} in {runs_dir}") + return runs_dir / arg + runs = sorted( + d.name + for d in runs_dir.glob("*") + if (match := RUN_ID.match(d.name)) and match["profile"] == manifest.profile and (d / RUN_INFO_FILE).is_file() + ) + if not runs: + raise UsageError(f"no runs of profile {manifest.profile} in {runs_dir}") + return runs_dir / runs[-1] + + +def _check_startable(manifest: Manifest) -> None: + gpu = open_gpu(manifest.gpu) + with gpu_lock(gpu.adapter): + preflight(manifest, gpu) + + +def _print_started(run_dir: Path, pid: int) -> None: + print(f"run {run_dir.name}") + print(f"dir {run_dir}") + print(f"runner pid {pid}, log {run_dir / RUNNER_LOG}") + print(f"report {run_dir / 'report.html'} (when finished)") + print(f"status python -m scripts.family_e2e status {run_dir}") + + +def cmd_start(args: argparse.Namespace) -> int: + _check_startable(load_manifest(args.manifest)) + run_dir = new_run(args.manifest, args.family, args.case, calibrate=args.calibrate) + _print_started(run_dir, spawn_runner(run_dir)) + return 0 + + +def cmd_run(args: argparse.Namespace) -> int: + try: + return run(args.run_dir.resolve()) + except KeyboardInterrupt: + return 130 + except Exception: + logger.exception("the runner failed") + return 2 + + +def cmd_resume(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + if is_alive(read_heartbeat(run_dir).get("runner")): + raise UsageError(f"the runner of {run_dir.name} is still running") + info = load_run_info(run_dir) + _check_startable(load_run_manifest(info)) + write_run_info(run_dir, info.model_copy(update={"status": "running", "finished_at": None})) + _print_started(run_dir, spawn_runner(run_dir)) + return 0 + + +def cmd_status(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + info = load_run_info(run_dir) + records = load_records(run_dir) + left = remaining_steps(info.plan, records) + heartbeat = read_heartbeat(run_dir) + alive = is_alive(heartbeat.get("runner")) + print(f"run {info.run_id} ({info.status})") + print(f"dir {run_dir}") + print(f"progress {len(info.plan) - len(left)}/{len(info.plan)} steps final, {len(records)} records") + # Every step of a calibration run carries the probe's reason; only the others make a record invalid there. + invalid = sum(1 for r in records if set(r.invalid_reasons) - {PROBE_REASON}) + print(f"invalid {invalid} records with invalid reasons") + print(f"errors {sum(1 for r in records if r.status != 'completed')} records not completed") + if info.calibrate: + print( + f"calibrate time and VRAM not rated; {len(info.calibration_exports)} store exports in {run_dir / 'calibration'}" + ) + if heartbeat: + updated = datetime.fromisoformat(heartbeat["updated_at"]) + age = (datetime.now(timezone.utc) - updated).total_seconds() + state = "alive" if alive else "gone" + print(f"runner {state}, step: {heartbeat.get('step')}, heartbeat {age:.0f}s ago") + print(f"log {run_dir / RUNNER_LOG}") + if (run_dir / "report.html").is_file(): + print(f"report {run_dir / 'report.html'}") + if info.status == "running" and not alive: + print(f"the runner is gone; continue with: python -m scripts.family_e2e resume {run_dir}") + return 0 + + +def cmd_stop(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + heartbeat = read_heartbeat(run_dir) + # The runner first: it would otherwise notice its server dying and record a crash. + for role in ("runner", "server_launcher", "server"): + if kill_identified(heartbeat.get(role)): + print(f"killed {role} (pid {int(heartbeat[role][0])}) and its descendants") + info = load_run_info(run_dir) + if info.calibrate: + # The killed server left its calibration store in the scratch root; the root goes next. + for arm in sorted({DEFAULT_ARM, *info.environments}): + for export in save_store_files(run_dir, info, scratch_root(_data_dir(run_dir), info.run_id, arm)): + print(f"saved the calibration store {run_dir / export.export}") + # Nothing of the run is alive any more, so its scratch roots are unused. + for leftover in remove_scratch_roots(_data_dir(run_dir), info): + print(f"could not remove {leftover}") + if info.status == "running": + write_run_info( + run_dir, + info.model_copy(update={"status": "stopped", "finished_at": now_iso()}), + ) + print(f"{info.run_id} stopped; continue with: python -m scripts.family_e2e resume {run_dir}") + return 0 + + +def cmd_accept(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + for reference in accept(_data_dir(run_dir), run_dir, args.case or None): + print(f"accepted {reference.case_key} from {reference.source_run_id}") + return 0 + + +def cmd_report(args: argparse.Namespace) -> int: + run_dir = _resolve_run(args.run, args.manifest) + result = write_report(_data_dir(run_dir), run_dir) + print(f"report {result.report_html}") + print(f"summary {result.summary_md}") + print(f"{result.failures} failures, {result.outliers} outliers, {result.invalid} invalid") + return 1 if result.failures else 0 + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="python -m scripts.family_e2e", description="End-to-end runs per model family." + ) + commands = parser.add_subparsers(dest="command", required=True) + + start = commands.add_parser("start", help="create a run and start its runner in the background") + start.add_argument("--manifest", type=Path, required=True) + start.add_argument("--family", action="append", default=[], help="only this family (repeatable)") + start.add_argument("--case", action="append", default=[], help="only this case id (repeatable)") + start.add_argument( + "--calibrate", + action="store_true", + help="measure working memory: submit every step through the server's calibrate route and save its exports", + ) + start.set_defaults(handler=cmd_start) + + runner = commands.add_parser("run", help="run the remaining steps of a run in the foreground (what start launches)") + runner.add_argument("--run-dir", type=Path, required=True) + runner.add_argument("--log-file", type=Path, help="append stdout and stderr to this file") + runner.set_defaults(handler=cmd_run) + + for name, handler, help_text in ( + ("resume", cmd_resume, "start the runner of a stopped or crashed run again"), + ("status", cmd_status, "show progress and whether the runner is alive"), + ("stop", cmd_stop, "kill the runner and its server"), + ("accept", cmd_accept, "make a run's measured results the references of its profile"), + ("report", cmd_report, "write report.html and summary.md of a run again"), + ): + command = commands.add_parser(name, help=help_text) + command.add_argument("run", nargs="?", help="run directory or run id; default: the profile's latest run") + command.add_argument("--manifest", type=Path, help="resolves run ids and the latest run") + if name == "accept": + command.add_argument("--case", action="append", default=[], help="only this case key (repeatable)") + command.set_defaults(handler=handler) + return parser + + +def _redirect_stdio(path: Path) -> None: + # A background runner has no usable stdio; tracebacks of a crash must land in its log too. + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND) + os.dup2(fd, 1) + os.dup2(fd, 2) + os.close(fd) + sys.stdout = open(1, "w", encoding="utf-8", buffering=1, closefd=False) + sys.stderr = open(2, "w", encoding="utf-8", buffering=1, closefd=False) + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + if getattr(args, "log_file", None): + _redirect_stdio(args.log_file) + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", stream=sys.stderr) + try: + return args.handler(args) + except (UsageError, *EXPECTED_ERRORS) as e: + print(f"error: {e}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/family_e2e/calibration.py b/scripts/family_e2e/calibration.py new file mode 100644 index 00000000000..94f7dcf0dec --- /dev/null +++ b/scripts/family_e2e/calibration.py @@ -0,0 +1,128 @@ +"""Calibration runs: each step goes through the server's working-memory calibrate route instead of enqueue_batch. + +The server measures every model operation of such a queue item with its working-memory probe and keeps the windows in +its calibration store, one file per device class under `/working_memory/` of the scratch root. The scratch root +goes at the end of each server lifetime, so every lifetime saves its store into the run's `calibration/` first: the +export route's bytes and the status while the server still runs, or the store file itself when the server died or was +stopped. `scripts/working_memory/build_shipped.py` turns the exports into shipped working-memory data. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Literal +from urllib.parse import quote + +from scripts.family_e2e.client import ApiError, InvokeClient +from scripts.family_e2e.records import ( + CalibrationExport, + CalibrationSubmission, + RunInfo, + write_bytes_atomic, + write_run_info, +) +from scripts.family_e2e.server import SCRATCH_PATHS + +CALIBRATION_DIR = "calibration" +STATUS_PATH = "/api/v2/models/working_memory/status" +EXPORT_PATH = "/api/v2/models/working_memory/export" +# The store's directory under the root's db_dir (invokeai/app/api/dependencies.py) and its file names +# (working_memory_calibration_store.py). Copied, not imported: the tool must not import invokeai, which imports torch. +STORE_DIR = "working_memory" +STORE_FILE = re.compile(r"^[0-9a-f]{16}\.json$") +PROBE_REASON = "calibration run: the probe synchronizes and empties the allocator cache around every model operation" + + +CASE_REFUSALS = (404, 422) +"""Calibrate refusals that concern one case (an unknown model, a graph the run cannot measure); the step gets an error +record and the run goes on. Any other error response would refuse every later step too.""" + + +class CalibrationError(RuntimeError): + """The server cannot run a calibration run, or refused one.""" + + +class CaseRefused(CalibrationError): + """The server refused one case's calibration run (`CASE_REFUSALS`).""" + + +def check_available(client: InvokeClient) -> None: + status = client.get(STATUS_PATH) + if not status.get("available"): + raise CalibrationError(f"the server cannot calibrate: {status.get('unavailable_reason')}") + + +def submit(client: InvokeClient, model_key: str, batch: dict[str, Any]) -> tuple[int, CalibrationSubmission]: + """Starts a calibration run of the batch's graph and data; returns its one queue item and the run. + + A refusal raises `CaseRefused` for a `CASE_REFUSALS` status and CalibrationError for any other error response. + Each server lifetime has a fresh queue and store and the tool submits each attempt once, so it sends no retry key. + """ + body: dict[str, Any] = {"graph": batch["graph"]} + if batch.get("data"): + body["data"] = batch["data"] + try: + created = client.post(f"/api/v2/models/i/{quote(model_key, safe='')}/working_memory/calibrate", body) + except ApiError as e: + refused = CaseRefused if e.status_code in CASE_REFUSALS else CalibrationError + raise refused(f"the server refused the calibration run: {e}") from e + if len(created["item_ids"]) != 1: + raise CalibrationError(f"expected one queue item from the calibration run, got {len(created['item_ids'])}") + submission = CalibrationSubmission(run_id=created["run_id"], device_class_id=created["device_class_id"]) + return created["item_ids"][0], submission + + +def _save( + run_dir: Path, + info: RunInfo, + stores: Iterable[tuple[str, bytes]], + status: bytes | None, + source: Literal["api", "store_file"], +) -> list[CalibrationExport]: + lifetime = max((e.lifetime for e in info.calibration_exports), default=0) + 1 + (run_dir / CALIBRATION_DIR).mkdir(exist_ok=True) + status_path = None + if status is not None: + status_path = f"{CALIBRATION_DIR}/{info.run_id}-{lifetime}-status.json" + write_bytes_atomic(run_dir / status_path, status) + saved_at = datetime.now(timezone.utc).isoformat(timespec="seconds") + exports = [] + for device_class_id, raw in stores: + path = f"{CALIBRATION_DIR}/{info.run_id}-{lifetime}-{device_class_id}.json" + write_bytes_atomic(run_dir / path, raw) + exports.append( + CalibrationExport( + lifetime=lifetime, + device_class_id=device_class_id, + export=path, + status=status_path, + source=source, + saved_at=saved_at, + ) + ) + info.calibration_exports.extend(exports) + write_run_info(run_dir, info) + return exports + + +def save_exports( + client: InvokeClient, run_dir: Path, info: RunInfo, device_class_ids: Iterable[str] +) -> list[CalibrationExport]: + """Saves the store of every device class the lifetime's runs were created for, and the status, into run.json.""" + stores = [ + (device_class_id, client.get_bytes(EXPORT_PATH, device_class_id=device_class_id)) + for device_class_id in sorted(set(device_class_ids)) + ] + return _save(run_dir, info, stores, client.get_bytes(STATUS_PATH), "api") + + +def save_store_files(run_dir: Path, info: RunInfo, scratch_root: Path) -> list[CalibrationExport]: + """Copies the store files of a scratch root whose server is gone; nothing when it has none.""" + store_dir = scratch_root / SCRATCH_PATHS["db_dir"] / STORE_DIR + files = sorted(p for p in store_dir.glob("*.json") if STORE_FILE.match(p.name)) if store_dir.is_dir() else [] + if not files: + return [] + return _save(run_dir, info, [(path.stem, path.read_bytes()) for path in files], None, "store_file") diff --git a/scripts/family_e2e/client.py b/scripts/family_e2e/client.py new file mode 100644 index 00000000000..441fde05932 --- /dev/null +++ b/scripts/family_e2e/client.py @@ -0,0 +1,309 @@ +"""HTTP and socket.io client for the server under test, and the run timing built from its events. + +This process must never import torch: a CUDA context here would land on the GPU being measured. Socket events are +stamped on arrival with `time.perf_counter()`, the clock of the memory sampler; the server's own event timestamps +have whole-second resolution, and `time.monotonic()` ticks only every 15.6 ms on Windows. +""" + +from __future__ import annotations + +import mimetypes +import threading +import time +from collections import defaultdict, deque +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import requests +import socketio + +from scripts.family_e2e.records import ModelLoad, NodeTiming, Timing + +QUEUE_ID = "default" +SOCKETIO_PATH = "/ws/socket.io" +REQUEST_TIMEOUT_S = 120.0 +# While waiting for a queue item, its status is also polled, in case a socket event was lost. +POLL_S = 2.0 +EVENT_GRACE_S = 1.0 +TERMINAL_STATUSES = ("completed", "failed", "canceled") +EVENT_NAMES = ( + "invocation_started", + "invocation_progress", + "invocation_complete", + "invocation_error", + "queue_item_status_changed", + "model_load_started", + "model_load_complete", +) +# Upper bound for the step count inferred from progress percentages. +MAX_INFERRED_STEPS = 1000 +# The server drops an image-bearing progress frame sent sooner than this after the previous one +# (invokeai/app/services/progress_previews/progress_previews_default.py). +PREVIEW_INTERVAL_S = 0.1 + + +class ApiError(RuntimeError): + def __init__(self, message: str, status_code: int | None = None) -> None: + super().__init__(message) + self.status_code = status_code + """The HTTP status of an error response; None for an accepted response the client cannot use.""" + + +class InvokeClient: + def __init__(self, base_url: str) -> None: + self.base_url = base_url.rstrip("/") + self._http = requests.Session() + + def _request(self, method: str, path: str, **kwargs: Any) -> requests.Response: + response = self._http.request(method, self.base_url + path, timeout=REQUEST_TIMEOUT_S, **kwargs) + if response.status_code >= 400: + raise ApiError(f"{method} {path} -> {response.status_code}: {response.text[:2000]}", response.status_code) + return response + + def get(self, path: str, **params: Any) -> Any: + return self._request("GET", path, params=params or None).json() + + def get_bytes(self, path: str, **params: Any) -> bytes: + return self._request("GET", path, params=params or None).content + + def post(self, path: str, body: Any = None) -> Any: + return self._request("POST", path, json=body).json() + + def image_bytes(self, image_name: str) -> bytes: + return self._request("GET", f"/api/v1/images/i/{image_name}/full").content + + def upload_image(self, path: Path) -> str: + content_type = mimetypes.guess_type(path.name)[0] or "image/png" + with path.open("rb") as f: + dto = self._request( + "POST", + "/api/v1/images/upload", + params={"image_category": "user", "is_intermediate": "true"}, + files={"file": (path.name, f, content_type)}, + ).json() + return dto["image_name"] + + def enqueue(self, batch: dict[str, Any]) -> int: + result = self.post(f"/api/v1/queue/{QUEUE_ID}/enqueue_batch", {"batch": batch, "prepend": False}) + if len(result["item_ids"]) != 1: + raise ApiError(f"expected one queue item from the batch, got {len(result['item_ids'])}") + return result["item_ids"][0] + + def queue_item(self, item_id: int) -> dict[str, Any]: + return self.get(f"/api/v1/queue/{QUEUE_ID}/i/{item_id}") + + def queue_busy(self) -> bool: + queue = self.get(f"/api/v1/queue/{QUEUE_ID}/status")["queue"] + return queue["pending"] + queue["in_progress"] > 0 + + def cancel(self, item_id: int) -> None: + self._request("PUT", f"/api/v1/queue/{QUEUE_ID}/i/{item_id}/cancel") + + +@dataclass(frozen=True) +class Event: + name: str + t: float + """`time.perf_counter()` at arrival.""" + data: dict[str, Any] + + +class EventRecorder: + """Records the queue and model-load events of a server from connect to close.""" + + def __init__(self, base_url: str) -> None: + self._base_url = base_url + # Polling only: websocket-client is not installed, and polling delivers within a few ms on localhost. + self._sio = socketio.Client(reconnection=True) + self._events: list[Event] = [] + self._changed = threading.Condition() + self._sio.on("connect", self._subscribe) + for name in EVENT_NAMES: + self._sio.on(name, self._handler(name)) + + def _subscribe(self) -> None: + # Also runs after a reconnect, which starts a new socket without room memberships. + self._sio.emit("subscribe_queue", {"queue_id": QUEUE_ID}) + + def _handler(self, name: str) -> Callable[[dict[str, Any]], None]: + def handle(data: dict[str, Any]) -> None: + t = time.perf_counter() + if name == "invocation_progress": + # Preview images are large and only their presence matters (it marks denoise steps). + data = {k: v for k, v in data.items() if k != "image"} | {"has_image": data.get("image") is not None} + with self._changed: + self._events.append(Event(name, t, data)) + self._changed.notify_all() + + return handle + + def connect(self) -> None: + self._sio.connect(self._base_url, socketio_path=SOCKETIO_PATH, transports=["polling"], wait_timeout=30) + # The connect handler's emit is not acknowledged; this call returns once the server joined the rooms. + self._sio.call("subscribe_queue", {"queue_id": QUEUE_ID}, timeout=30) + + def close(self) -> None: + self._sio.disconnect() + + def mark(self) -> int: + with self._changed: + return len(self._events) + + def since(self, mark: int) -> list[Event]: + with self._changed: + return self._events[mark:] + + def wait(self, timeout: float) -> None: + with self._changed: + self._changed.wait(timeout) + + +def wait_for_item( + client: InvokeClient, + recorder: EventRecorder, + mark: int, + item_id: int, + timeout: float, + tick: Callable[[], None], +) -> tuple[dict[str, Any], float]: + """Waits for the item to finish. Returns the item and the `perf_counter` time it finished. + + The finish time is the last arrival among the item's events once its terminal status event is in. A poll that + sees the item finished first waits EVENT_GRACE_S for that event, because the poll can overtake the events still + in transit; without it, the poll time stands in. `tick` runs before every check, at least every POLL_S seconds, + and may raise to abort the wait. Raises TimeoutError. + """ + deadline = time.perf_counter() + timeout + next_poll = time.perf_counter() + POLL_S + polled: tuple[dict[str, Any], float] | None = None + while True: + tick() + own = _item_events(recorder.since(mark), item_id) + latest = max((e.t for e in own), default=0.0) + if any(e.name == "queue_item_status_changed" and e.data.get("status") in TERMINAL_STATUSES for e in own): + return client.queue_item(item_id), latest + now = time.perf_counter() + if polled is not None and now - polled[1] >= EVENT_GRACE_S: + return polled[0], max(polled[1], latest) + if polled is None and now >= next_poll: + item = client.queue_item(item_id) + if item["status"] in TERMINAL_STATUSES: + polled = (item, now) + next_poll = now + POLL_S + if now >= deadline: + raise TimeoutError(f"queue item {item_id} did not finish within {timeout:.0f}s") + recorder.wait(min(POLL_S, EVENT_GRACE_S, max(0.0, deadline - now))) + + +def wait_for_idle_queue(client: InvokeClient, timeout: float, tick: Callable[[], None]) -> bool: + """Waits until the queue has nothing pending or in progress; False if it is still busy after `timeout`.""" + deadline = time.perf_counter() + timeout + while True: + tick() + if not client.queue_busy(): + return True + if time.perf_counter() >= deadline: + return False + time.sleep(POLL_S) + + +def _item_events(events: list[Event], item_id: int) -> list[Event]: + return [e for e in events if e.data.get("item_id") == item_id] + + +def per_step_seconds(progress: list[tuple[float, float]], expected_steps: int | None = None) -> float | None: + """Seconds per denoise step from (arrival time, percentage) pairs of one denoise node. + + Image-bearing progress frames are throttled server-side and the throttled steps emit nothing, so each frame's + step comes from its percentage (k / N); the slope over the first and last frame then needs no frame in between. + N is `expected_steps` (a run over the full schedule) when every percentage fits it. Otherwise it is the smallest + count every percentage fits, which may be a divisor of the real one when only every other step left a frame; a + step slower than two throttle intervals rules that out, and a faster one gives None. + """ + if len(progress) < 2: + return None + + def fits(n: int) -> bool: + return all(abs(p * n - round(p * n)) < 1e-6 for _, p in progress) + + trusted = expected_steps is not None and fits(expected_steps) + steps = expected_steps if trusted else next((n for n in range(1, MAX_INFERRED_STEPS + 1) if fits(n)), None) + if steps is None: + return None + (t_first, p_first), (t_last, p_last) = progress[0], progress[-1] + k_first, k_last = round(p_first * steps), round(p_last * steps) + if k_last <= k_first: + return None + seconds = (t_last - t_first) / (k_last - k_first) + if not trusted and seconds < 2 * PREVIEW_INTERVAL_S: + return None + return seconds + + +def build_timing( + events: list[Event], + item_id: int, + t_enqueue: float, + t_done: float, + item: dict[str, Any], + expected_steps: int | None = None, +) -> Timing: + """Run timing from the recorded events; every `t_*` is relative to the enqueue instant. + + `expected_steps` is the denoise step count when the run covers the whole schedule (not img2img). + """ + # The socket client runs every handler on its own thread, so the list order can differ from the arrival order. + events = sorted(events, key=lambda e: e.t) + own = _item_events(events, item_id) + starts: dict[str, tuple[str, str, float]] = {} + ends: dict[str, float] = {} + progress: dict[str, list[tuple[float, float]]] = defaultdict(list) + for event in own: + invocation = event.data.get("invocation") or {} + prepared_id = invocation.get("id") + if prepared_id is None: + continue + if event.name == "invocation_started": + starts[prepared_id] = (event.data["invocation_source_id"], invocation.get("type", ""), event.t) + elif event.name in ("invocation_complete", "invocation_error"): + ends[prepared_id] = event.t + elif event.name == "invocation_progress" and event.data.get("has_image"): + percentage = event.data.get("percentage") + if percentage is not None: + progress[prepared_id].append((event.t, percentage)) + + nodes = [ + NodeTiming( + node_id=source_id, + type=node_type, + t_start=t - t_enqueue, + t_end=ends[prepared_id] - t_enqueue if prepared_id in ends else None, + ) + for prepared_id, (source_id, node_type, t) in sorted(starts.items(), key=lambda kv: kv[1][2]) + ] + + # Model load events carry no queue item; within the run's window they belong to it. + pending: dict[tuple[str, str | None], deque[int]] = defaultdict(deque) + loads: list[dict[str, Any]] = [] + for event in events: + if event.name not in ("model_load_started", "model_load_complete") or not t_enqueue <= event.t <= t_done: + continue + key = ((event.data.get("config") or {}).get("key", ""), event.data.get("submodel_type")) + if event.name == "model_load_started": + pending[key].append(len(loads)) + loads.append({"model": key[0], "submodel": key[1], "t_start": event.t - t_enqueue, "t_end": None}) + elif pending[key]: + loads[pending[key].popleft()]["t_end"] = event.t - t_enqueue + + # The denoise node is the one with the most preview frames. + denoise = max(progress.values(), key=len, default=[]) + return Timing( + total_s=t_done - t_enqueue, + nodes=nodes, + model_loads=[ModelLoad(**load) for load in loads], + per_step_s=per_step_seconds(denoise, expected_steps), + queue_started_at=item.get("started_at"), + queue_completed_at=item.get("completed_at"), + ) diff --git a/scripts/family_e2e/detach.py b/scripts/family_e2e/detach.py new file mode 100644 index 00000000000..8b827a3fcdc --- /dev/null +++ b/scripts/family_e2e/detach.py @@ -0,0 +1,78 @@ +"""Starting the runner as a background process that outlives the caller. + +On Windows the runner breaks away from the caller's job object, so a caller that kills its job's processes (an agent +harness does after two hours) cannot take a long run with it. Where the job forbids breakaway, the runner is created +through WMI instead, which puts it in no job of the caller; such a process starts with the user's default +environment, not the caller's. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + +from scripts.family_e2e.manifest import TOOL_CHECKOUT + +RUNNER_LOG = "runner.log" +WMI_TIMEOUT_S = 120 + +# Hidden window: WMI shows a console program's window otherwise. +_WMI_CREATE = ( + "$startup = New-CimInstance -ClassName Win32_ProcessStartup -ClientOnly -Property @{ShowWindow=[uint16]0}; " + "$r = Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments @{" + "CommandLine=$env:FAMILY_E2E_COMMAND; CurrentDirectory=$env:FAMILY_E2E_CWD; ProcessStartupInformation=$startup}; " + "if ($r.ReturnValue -ne 0) { [Console]::Error.WriteLine('Win32_Process.Create returned ' + $r.ReturnValue); exit 1 }; " + "$r.ProcessId" +) + + +class DetachError(RuntimeError): + pass + + +def runner_command(run_dir: Path) -> list[str]: + return [ + sys.executable, + "-X", + "utf8", + "-m", + "scripts.family_e2e", + "run", + "--run-dir", + str(run_dir), + "--log-file", + str(run_dir / RUNNER_LOG), + ] + + +def spawn_runner(run_dir: Path) -> int: + """Starts the runner for `run_dir` in the background and returns the started process's pid.""" + args = runner_command(run_dir) + stdio = {"stdin": subprocess.DEVNULL, "stdout": subprocess.DEVNULL, "stderr": subprocess.DEVNULL} + if sys.platform != "win32": + return subprocess.Popen(args, cwd=TOOL_CHECKOUT, start_new_session=True, **stdio).pid + # CREATE_NO_WINDOW, not DETACHED_PROCESS: a venv's python.exe is a launcher, and under DETACHED_PROCESS the + # interpreter it starts gets a new visible console, which the server and every tool call would then share. + flags = subprocess.CREATE_NO_WINDOW | subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_BREAKAWAY_FROM_JOB + try: + return subprocess.Popen(args, cwd=TOOL_CHECKOUT, creationflags=flags, close_fds=True, **stdio).pid + except OSError: + return _spawn_via_wmi(args) + + +def _spawn_via_wmi(args: list[str]) -> int: + # Passed through the environment, so neither PowerShell nor WMI has to parse quoting of their own. + env = os.environ | {"FAMILY_E2E_COMMAND": subprocess.list2cmdline(args), "FAMILY_E2E_CWD": str(TOOL_CHECKOUT)} + result = subprocess.run( + ["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", _WMI_CREATE], + env=env, + capture_output=True, + text=True, + timeout=WMI_TIMEOUT_S, + check=False, + ) + if result.returncode != 0: + raise DetachError(f"starting the runner through WMI failed:\n{result.stderr.strip()}") + return int(result.stdout.strip().splitlines()[-1]) diff --git a/scripts/family_e2e/gpu_memory.py b/scripts/family_e2e/gpu_memory.py new file mode 100644 index 00000000000..f4993026140 --- /dev/null +++ b/scripts/family_e2e/gpu_memory.py @@ -0,0 +1,580 @@ +"""Driver-level video memory of another process, sampled without a GPU context of our own. + +The allocator's own figure (``torch.cuda.max_memory_allocated``) misses its pools and the cuDNN/MIOpen workspaces, so +the real peak runs 25-75 % above it. What the driver charges the process is what decides whether a card of a given size +copes, and Windows reports it per process and adapter through the WDDM kernel thunks: ``D3DKMTQueryVideoMemoryInfo`` +on another process's handle answers in about a microsecond, so the dedicated (local segment) usage can be sampled every +few milliseconds. The bytes Windows has paged out to system memory are only visible in the ``GPU Process Memory`` +performance counters (``Shared Usage``), read through PDH at a slower rate. + +The client never imports torch: a CUDA/HIP context of its own would sit on the same adapter and distort what the +server can allocate. Windows (WDDM) only for now; it covers NVIDIA and AMD/ROCm alike. +""" + +import bisect +import ctypes +import functools +import logging +import operator +import re +import sys +import threading +import time +from dataclasses import dataclass +from types import SimpleNamespace + +_STATUS_SUCCESS = 0x00000000 +# The target has no allocation context on the adapter: it never opened it, or has closed it again. +_STATUS_INVALID_PARAMETER = 0xC000000D +_STATUS_ACCESS_DENIED = 0xC0000022 +# D3DKMTQueryVideoMemoryInfo answers STATUS_ACCESS_DENIED for PROCESS_QUERY_LIMITED_INFORMATION; it needs the full +# query right (measured on Windows 10 22H2 with an NVIDIA and an AMD driver). +_PROCESS_QUERY_INFORMATION = 0x0400 +_STILL_ACTIVE = 259 +_ERROR_ACCESS_DENIED = 5 +_ERROR_INVALID_PARAMETER = 87 + +_KMTQAITYPE_ADAPTERREGISTRYINFO = 8 +_KMTQAITYPE_ADAPTERTYPE = 15 +# D3DKMT_KMD_DRIVER_VERSION: the kernel-mode driver's version, the one Device Manager shows. +_KMTQAITYPE_KMD_DRIVER_VERSION = 49 +# D3DKMT_ADAPTERTYPE bits. +_ADAPTER_RENDER_SUPPORTED = 1 << 0 +_ADAPTER_SOFTWARE_DEVICE = 1 << 2 +_ADAPTER_COMPUTE_ONLY = 1 << 11 +_MEMORY_SEGMENT_GROUP_LOCAL = 0 +_MAX_PATH = 260 + +_PDH_FMT_LARGE = 0x00000400 +_PDH_MORE_DATA = 0x800007D2 +_PDH_NO_DATA = 0x800007D5 +# PDH_CSTATUS_VALID_DATA and PDH_CSTATUS_NEW_DATA; any other status leaves `largeValue` undefined. +_PDH_VALID_DATA = (0x00000000, 0x00000001) +# English paths; PdhAddEnglishCounterW resolves them on a localized Windows too. +_PDH_DEDICATED_USAGE = r"\GPU Process Memory(*)\Dedicated Usage" +_PDH_SHARED_USAGE = r"\GPU Process Memory(*)\Shared Usage" +_PDH_INSTANCE = re.compile(r"pid_(\d+)_luid_0x([0-9a-f]{8})_0x([0-9a-f]{8})_phys_(\d+)", re.IGNORECASE) + +_logger = logging.getLogger(__name__) + +# Win32 types spelled as fixed-width ctypes, so this module imports on every platform. +_UINT = ctypes.c_uint32 +_LONG = ctypes.c_int32 +_NTSTATUS = ctypes.c_uint32 + + +class UnsupportedPlatformError(RuntimeError): + """GPU memory sampling is not implemented for this operating system.""" + + +@dataclass(frozen=True) +class Adapter: + luid: int + """``(HighPart << 32) | LowPart``, as D3DKMT and the PDH instance names spell it.""" + description: str + driver_version: str | None = None + """As Device Manager shows it, e.g. ``32.0.16.1047``; ``None`` when the driver does not report one.""" + + +@dataclass(frozen=True) +class ProcessUsage: + dedicated_bytes: int + shared_bytes: int + + +@dataclass(frozen=True) +class MemoryTrace: + """Samples of one process's usage on one adapter, stamped with the client's ``time.perf_counter()``. + + The trace is a sample-and-hold step function: a value stands from its timestamp until the next sample. + """ + + method: str + interval_ms: float + """Mean spacing of the dedicated samples; the configured interval while they span no measurable time.""" + dedicated: tuple[tuple[float, int], ...] + """Local-segment ``CurrentUsage`` in bytes.""" + shared: tuple[tuple[float, int], ...] + """PDH ``Shared Usage`` in bytes: what Windows keeps of this process's allocations in system memory.""" + + def peak(self, t0: float | None = None, t1: float | None = None) -> int | None: + """The highest dedicated usage in ``[t0, t1]``: the value in effect at ``t0`` and every sample up to ``t1``. + + Counting the value in effect at ``t0`` keeps a window shorter than the sampling interval from coming back + empty. ``None`` bounds are open; ``None`` is returned when no sample falls on or before ``t1``. + """ + if t0 is not None and t1 is not None and t1 < t0: + raise ValueError(f"window ends before it starts: [{t0}, {t1}]") + samples = self.dedicated + lo = 0 + if t0 is not None: + lo = bisect.bisect_left(samples, t0, key=_sample_time) + if lo > 0 and (lo == len(samples) or samples[lo][0] > t0): + lo -= 1 + hi = len(samples) if t1 is None else bisect.bisect_right(samples, t1, key=_sample_time) + return max((value for _, value in samples[lo:hi]), default=None) + + def at(self, t: float) -> int | None: + """The dedicated usage in effect at ``t``: the last sample at or before it.""" + index = bisect.bisect_right(self.dedicated, t, key=_sample_time) + return self.dedicated[index - 1][1] if index else None + + +_sample_time = operator.itemgetter(0) + + +class _Luid(ctypes.Structure): + _fields_ = [("LowPart", _UINT), ("HighPart", _LONG)] + + +class _AdapterInfo(ctypes.Structure): + """``D3DKMT_ADAPTERINFO``.""" + + _fields_ = [("hAdapter", _UINT), ("AdapterLuid", _Luid), ("NumOfSources", _UINT), ("bPrecise", _LONG)] + + +class _EnumAdapters2(ctypes.Structure): + """``D3DKMT_ENUMADAPTERS2``.""" + + _fields_ = [("NumAdapters", _UINT), ("pAdapters", ctypes.POINTER(_AdapterInfo))] + + +class _QueryAdapterInfo(ctypes.Structure): + """``D3DKMT_QUERYADAPTERINFO``.""" + + _fields_ = [("hAdapter", _UINT), ("Type", ctypes.c_int), ("pData", ctypes.c_void_p), ("DataSize", _UINT)] + + +class _AdapterRegistryInfo(ctypes.Structure): + """``D3DKMT_ADAPTERREGISTRYINFO``.""" + + _fields_ = [ + ("AdapterString", ctypes.c_wchar * _MAX_PATH), + ("BiosString", ctypes.c_wchar * _MAX_PATH), + ("DacType", ctypes.c_wchar * _MAX_PATH), + ("ChipType", ctypes.c_wchar * _MAX_PATH), + ] + + +class _OpenAdapterFromLuid(ctypes.Structure): + """``D3DKMT_OPENADAPTERFROMLUID``.""" + + _fields_ = [("AdapterLuid", _Luid), ("hAdapter", _UINT)] + + +class _CloseAdapter(ctypes.Structure): + """``D3DKMT_CLOSEADAPTER``.""" + + _fields_ = [("hAdapter", _UINT)] + + +class _QueryVideoMemoryInfo(ctypes.Structure): + """``D3DKMT_QUERYVIDEOMEMORYINFO``. A null ``hProcess`` means the calling process.""" + + _fields_ = [ + ("hProcess", ctypes.c_void_p), + ("hAdapter", _UINT), + ("MemorySegmentGroup", ctypes.c_int), + ("Budget", ctypes.c_uint64), + ("CurrentUsage", ctypes.c_uint64), + ("CurrentReservation", ctypes.c_uint64), + ("AvailableForReservation", ctypes.c_uint64), + ("PhysicalAdapterIndex", _UINT), + ] + + +class _PdhCounterValue(ctypes.Structure): + """``PDH_FMT_COUNTERVALUE`` read as ``PDH_FMT_LARGE``.""" + + _fields_ = [("CStatus", ctypes.c_uint32), ("largeValue", ctypes.c_int64)] + + +class _PdhCounterValueItem(ctypes.Structure): + """``PDH_FMT_COUNTERVALUE_ITEM_W``.""" + + _fields_ = [("szName", ctypes.c_wchar_p), ("FmtValue", _PdhCounterValue)] + + +def _require_windows() -> None: + if sys.platform != "win32": + raise UnsupportedPlatformError( + f"GPU memory sampling is implemented for Windows (WDDM) only, not {sys.platform}" + ) + + +@functools.cache +def _win32() -> SimpleNamespace: + """gdi32, kernel32 and pdh with the prototypes used here; loaded on first use, since only Windows has them.""" + gdi32 = ctypes.WinDLL("gdi32") + for name in ( + "D3DKMTEnumAdapters2", + "D3DKMTQueryAdapterInfo", + "D3DKMTOpenAdapterFromLuid", + "D3DKMTQueryVideoMemoryInfo", + "D3DKMTCloseAdapter", + ): + fn = getattr(gdi32, name) + fn.argtypes = [ctypes.c_void_p] + fn.restype = _NTSTATUS + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.OpenProcess.argtypes = [ctypes.c_uint32, ctypes.c_int32, ctypes.c_uint32] + kernel32.OpenProcess.restype = ctypes.c_void_p + kernel32.GetExitCodeProcess.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint32)] + kernel32.GetExitCodeProcess.restype = ctypes.c_int32 + kernel32.CloseHandle.argtypes = [ctypes.c_void_p] + kernel32.CloseHandle.restype = ctypes.c_int32 + + pdh = ctypes.WinDLL("pdh") + prototypes = { + "PdhOpenQueryW": [ctypes.c_wchar_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)], + "PdhAddEnglishCounterW": [ctypes.c_void_p, ctypes.c_wchar_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)], + "PdhCollectQueryData": [ctypes.c_void_p], + "PdhGetFormattedCounterArrayW": [ + ctypes.c_void_p, + ctypes.c_uint32, + ctypes.POINTER(ctypes.c_uint32), + ctypes.POINTER(ctypes.c_uint32), + ctypes.c_void_p, + ], + "PdhCloseQuery": [ctypes.c_void_p], + } + for name, argtypes in prototypes.items(): + fn = getattr(pdh, name) + fn.argtypes = argtypes + fn.restype = ctypes.c_uint32 + return SimpleNamespace(gdi32=gdi32, kernel32=kernel32, pdh=pdh) + + +def _check(status: int, call: str) -> None: + if status != _STATUS_SUCCESS: + raise OSError(f"{call} failed with status {status:#010x}") + + +def _luid_value(luid: _Luid) -> int: + return ((luid.HighPart & 0xFFFFFFFF) << 32) | luid.LowPart + + +def _luid_struct(value: int) -> _Luid: + return _Luid(value & 0xFFFFFFFF, value >> 32) + + +def _query_adapter_info(handle: int, info_type: int, data: ctypes.Structure | ctypes.c_uint32 | ctypes.c_uint64) -> int: + query = _QueryAdapterInfo(handle, info_type, ctypes.cast(ctypes.byref(data), ctypes.c_void_p), ctypes.sizeof(data)) + return _win32().gdi32.D3DKMTQueryAdapterInfo(ctypes.byref(query)) + + +def _describe(info: _AdapterInfo) -> Adapter | None: + """The adapter, or ``None`` for one that cannot hold a compute process's memory or cannot be named.""" + adapter_type = _UINT() + if _query_adapter_info(info.hAdapter, _KMTQAITYPE_ADAPTERTYPE, adapter_type) != _STATUS_SUCCESS: + return None + # The Microsoft Basic Render Driver is a software device; display-only adapters (indirect displays) run nothing. + if adapter_type.value & _ADAPTER_SOFTWARE_DEVICE: + return None + if not adapter_type.value & (_ADAPTER_RENDER_SUPPORTED | _ADAPTER_COMPUTE_ONLY): + return None + registry = _AdapterRegistryInfo() + if _query_adapter_info(info.hAdapter, _KMTQAITYPE_ADAPTERREGISTRYINFO, registry) != _STATUS_SUCCESS: + return None + version = ctypes.c_uint64() + status = _query_adapter_info(info.hAdapter, _KMTQAITYPE_KMD_DRIVER_VERSION, version) + return Adapter( + luid=_luid_value(info.AdapterLuid), + description=registry.AdapterString, + driver_version=_format_driver_version(version.value) if status == _STATUS_SUCCESS else None, + ) + + +def _format_driver_version(packed: int) -> str | None: + """A driver version packed as four 16-bit fields, most significant first, in dotted form; 0 is unknown.""" + if not packed: + return None + return ".".join(str(packed >> shift & 0xFFFF) for shift in (48, 32, 16, 0)) + + +def list_adapters() -> list[Adapter]: + """The hardware GPU adapters, in the order the kernel enumerates them.""" + _require_windows() + gdi32 = _win32().gdi32 + # A null array asks for the size: the call answers the most adapters it could return. + enum = _EnumAdapters2(0, None) + _check(gdi32.D3DKMTEnumAdapters2(ctypes.byref(enum)), "D3DKMTEnumAdapters2") + infos = (_AdapterInfo * enum.NumAdapters)() + enum.pAdapters = infos + _check(gdi32.D3DKMTEnumAdapters2(ctypes.byref(enum)), "D3DKMTEnumAdapters2") + opened = infos[: enum.NumAdapters] + try: + return [adapter for adapter in map(_describe, opened) if adapter is not None] + finally: + for info in opened: + gdi32.D3DKMTCloseAdapter(ctypes.byref(_CloseAdapter(info.hAdapter))) + + +def _select_adapter(adapters: list[Adapter], description_substring: str) -> Adapter: + needle = description_substring.casefold() + matches = [adapter for adapter in adapters if needle in adapter.description.casefold()] + if len(matches) == 1: + return matches[0] + available = "; ".join(f"{adapter.description} (LUID {adapter.luid:#x})" for adapter in adapters) or "none" + found = "No adapter matches" if not matches else f"{len(matches)} adapters match" + raise LookupError(f"{found} {description_substring!r}. Adapters: {available}") + + +def find_adapter(description_substring: str) -> Adapter: + """The one adapter whose description contains the substring, ignoring case. ``LookupError`` for none or several.""" + return _select_adapter(list_adapters(), description_substring) + + +def _parse_instance(name: str) -> tuple[int, int] | None: + """``(pid, luid)`` of a ``GPU Process Memory`` instance on physical adapter 0, else ``None``.""" + match = _PDH_INSTANCE.fullmatch(name) + if match is None or match[4] != "0": + return None + return int(match[1]), (int(match[2], 16) << 32) | int(match[3], 16) + + +class _GpuProcessMemoryCounters: + """One PDH query over every process's ``GPU Process Memory`` instances. + + Only the first query in a process is slow to open (about 0.35 s while PDH loads the provider); a collection costs + well under a millisecond. + """ + + def __init__(self) -> None: + pdh = _win32().pdh + self._query = ctypes.c_void_p() + _check(pdh.PdhOpenQueryW(None, None, ctypes.byref(self._query)), "PdhOpenQueryW") + try: + self._dedicated = self._add(_PDH_DEDICATED_USAGE) + self._shared = self._add(_PDH_SHARED_USAGE) + except BaseException: + self.close() + raise + + def _add(self, path: str) -> ctypes.c_void_p: + counter = ctypes.c_void_p() + status = _win32().pdh.PdhAddEnglishCounterW(self._query, path, None, ctypes.byref(counter)) + _check(status, f"PdhAddEnglishCounterW({path})") + return counter + + def collect(self) -> dict[tuple[int, int], ProcessUsage]: + """Usage keyed by ``(pid, luid)``, for every process that has opened an adapter.""" + status = _win32().pdh.PdhCollectQueryData(self._query) + if status == _PDH_NO_DATA: + return {} + _check(status, "PdhCollectQueryData") + dedicated = self._values(self._dedicated) + shared = self._values(self._shared) + return { + key: ProcessUsage(dedicated.get(key, 0), shared.get(key, 0)) for key in dedicated.keys() | shared.keys() + } + + def _values(self, counter: ctypes.c_void_p) -> dict[tuple[int, int], int]: + pdh = _win32().pdh + size, count = ctypes.c_uint32(0), ctypes.c_uint32(0) + status = pdh.PdhGetFormattedCounterArrayW( + counter, _PDH_FMT_LARGE, ctypes.byref(size), ctypes.byref(count), None + ) + if status == _STATUS_SUCCESS: + return {} + if status != _PDH_MORE_DATA: + _check(status, "PdhGetFormattedCounterArrayW") + buffer = (ctypes.c_byte * size.value)() + status = pdh.PdhGetFormattedCounterArrayW( + counter, _PDH_FMT_LARGE, ctypes.byref(size), ctypes.byref(count), buffer + ) + _check(status, "PdhGetFormattedCounterArrayW") + items = ctypes.cast(buffer, ctypes.POINTER(_PdhCounterValueItem)) + values: dict[tuple[int, int], int] = {} + for item in items[: count.value]: + key = _parse_instance(item.szName or "") + if key is not None and item.FmtValue.CStatus in _PDH_VALID_DATA: + values[key] = int(item.FmtValue.largeValue) + return values + + def close(self) -> None: + if self._query: + _win32().pdh.PdhCloseQuery(self._query) + self._query = ctypes.c_void_p() + + +def process_usage_on_adapter(adapter: Adapter) -> dict[int, ProcessUsage]: + """Every process that has opened the adapter, by pid, with its dedicated and shared usage.""" + _require_windows() + counters = _GpuProcessMemoryCounters() + try: + usage = counters.collect() + finally: + counters.close() + return {pid: value for (pid, luid), value in usage.items() if luid == adapter.luid} + + +class ProcessMemorySampler: + """Samples one process's usage on one adapter on a background thread, from ``start()`` until ``stop()``. + + Dedicated usage is read every ``interval_s``, shared usage every ``shared_interval_s``. A process that has not + opened the adapter yet reads as zero. Sampling ends early when the process exits. + + ``pid`` must be the process that allocates. On Windows the ``python.exe`` in a venv's ``Scripts`` folder and the + console-script ``.exe`` files there are launchers: they run the interpreter as a child process and read as zero. + """ + + def __init__(self, pid: int, adapter: Adapter, interval_s: float = 0.003, shared_interval_s: float = 0.1) -> None: + _require_windows() + if pid <= 0: + raise ValueError(f"invalid pid {pid}") + if interval_s <= 0 or shared_interval_s <= 0: + raise ValueError(f"sampling intervals must be positive, got {interval_s} and {shared_interval_s}") + self._pid = pid + self._adapter = adapter + self._interval_s = interval_s + self._shared_interval_s = shared_interval_s + self._dedicated: list[tuple[float, int]] = [] + self._shared: list[tuple[float, int]] = [] + self._process: int | None = None + self._adapter_handle: int | None = None + self._counters: _GpuProcessMemoryCounters | None = None + self._stopping = threading.Event() + self._thread: threading.Thread | None = None + self._trace: MemoryTrace | None = None + self._warned: set[str] = set() + + def start(self) -> None: + """Open the process and the adapter, take one sample and start the sampling thread.""" + if self._thread is not None or self._trace is not None: + raise RuntimeError("a sampler starts once") + try: + self._open() + self._sample_dedicated(raise_on_failure=True) + self._sample_shared(raise_on_failure=True) + except BaseException: + self._close() + raise + self._thread = threading.Thread(target=self._run, name=f"gpu-memory-sampler-{self._pid}", daemon=True) + self._thread.start() + + def stop(self) -> MemoryTrace: + """Stop sampling, release the handles and return the trace; later calls return the same trace.""" + if self._trace is None: + self._stopping.set() + if self._thread is not None: + self._thread.join() + self._close() + self._trace = MemoryTrace( + method="d3dkmt", + interval_ms=self._mean_interval_ms(), + dedicated=tuple(self._dedicated), + shared=tuple(self._shared), + ) + return self._trace + + def _open(self) -> None: + win32 = _win32() + process = win32.kernel32.OpenProcess(_PROCESS_QUERY_INFORMATION, 0, self._pid) + if not process: + error = ctypes.get_last_error() + if error == _ERROR_INVALID_PARAMETER: + raise ProcessLookupError(f"no process with pid {self._pid}") + if error == _ERROR_ACCESS_DENIED: + raise PermissionError(f"access denied opening process {self._pid} for memory queries") + raise ctypes.WinError(error) + self._process = process + if self._exited(): + raise ProcessLookupError(f"process {self._pid} has exited") + + opened = _OpenAdapterFromLuid(_luid_struct(self._adapter.luid), 0) + status = win32.gdi32.D3DKMTOpenAdapterFromLuid(ctypes.byref(opened)) + _check(status, f"D3DKMTOpenAdapterFromLuid for {self._adapter.description} (LUID {self._adapter.luid:#x})") + self._adapter_handle = opened.hAdapter + # The target's STATUS_INVALID_PARAMETER only means "no allocation on this adapter" once the same call has + # succeeded for this process, which proves the adapter handle and the struct layout. + status, _ = self._query_usage(None) + _check(status, "D3DKMTQueryVideoMemoryInfo for the sampling process") + self._counters = _GpuProcessMemoryCounters() + + def _close(self) -> None: + win32 = _win32() + if self._counters is not None: + self._counters.close() + self._counters = None + if self._adapter_handle is not None: + win32.gdi32.D3DKMTCloseAdapter(ctypes.byref(_CloseAdapter(self._adapter_handle))) + self._adapter_handle = None + if self._process is not None: + win32.kernel32.CloseHandle(self._process) + self._process = None + + def _query_usage(self, process: int | None) -> tuple[int, int]: + assert self._adapter_handle is not None + info = _QueryVideoMemoryInfo(process, self._adapter_handle, _MEMORY_SEGMENT_GROUP_LOCAL) + status = _win32().gdi32.D3DKMTQueryVideoMemoryInfo(ctypes.byref(info)) + return status, int(info.CurrentUsage) + + def _exited(self) -> bool: + code = ctypes.c_uint32() + if not _win32().kernel32.GetExitCodeProcess(self._process, ctypes.byref(code)): + raise ctypes.WinError(ctypes.get_last_error()) + return code.value != _STILL_ACTIVE + + def _sample_dedicated(self, raise_on_failure: bool = False) -> bool: + """Append one dedicated sample; ``False`` once the process has exited.""" + t = time.perf_counter() + status, usage = self._query_usage(self._process) + if status == _STATUS_SUCCESS: + self._dedicated.append((t, usage)) + return True + if self._exited(): + if raise_on_failure: + raise ProcessLookupError(f"process {self._pid} has exited") + return False + if status == _STATUS_INVALID_PARAMETER: + self._dedicated.append((t, 0)) + return True + if raise_on_failure: + if status == _STATUS_ACCESS_DENIED: + raise PermissionError(f"access denied querying video memory of process {self._pid}") + _check(status, f"D3DKMTQueryVideoMemoryInfo for process {self._pid}") + self._warn_once("dedicated", f"D3DKMTQueryVideoMemoryInfo failed with status {status:#010x}; sample skipped") + return True + + def _sample_shared(self, raise_on_failure: bool = False) -> None: + assert self._counters is not None + t = time.perf_counter() + try: + usage = self._counters.collect().get((self._pid, self._adapter.luid)) + except OSError as error: + if raise_on_failure: + raise + self._warn_once("shared", f"{error}; shared sample skipped") + return + self._shared.append((t, 0 if usage is None else usage.shared_bytes)) + + def _warn_once(self, series: str, message: str) -> None: + if series not in self._warned: + self._warned.add(series) + _logger.warning(f"GPU memory sampler for process {self._pid}: {message} (further failures not logged)") + + def _run(self) -> None: + clock = time.perf_counter + next_dedicated = clock() + self._interval_s + next_shared = clock() + self._shared_interval_s + while not self._stopping.is_set(): + delay = next_dedicated - clock() + if delay > 0: + time.sleep(delay) + if not self._sample_dedicated(): + return + now = clock() + if now >= next_shared: + self._sample_shared() + next_shared = now + self._shared_interval_s + # A fixed rate: a late wake-up shortens the next sleep instead of delaying every later sample, and after a + # stall (GIL, suspend) sampling resumes from now rather than bursting to catch up. + next_dedicated = max(next_dedicated + self._interval_s, now) + + def _mean_interval_ms(self) -> float: + span = self._dedicated[-1][0] - self._dedicated[0][0] if len(self._dedicated) > 1 else 0.0 + if span <= 0: + return self._interval_s * 1000 + return span / (len(self._dedicated) - 1) * 1000 diff --git a/scripts/family_e2e/graphs.py b/scripts/family_e2e/graphs.py new file mode 100644 index 00000000000..8b7f46813ea --- /dev/null +++ b/scripts/family_e2e/graphs.py @@ -0,0 +1,180 @@ +"""Graphs for the cases: the exporter request, the webv2 exporter call, and the runner's batch policy. + +The webv2 exporter builds each case's batch with the real frontend graph builders (contract 1 in +`.scratch/agents/family-e2e/contract.md`). The runner then disables the invocation cache on every node, so repeated +runs really execute, and hashes the result so a frontend graph change shows up against the reference. +""" + +from __future__ import annotations + +import copy +import hashlib +import json +import shutil +import subprocess +from collections.abc import Iterator +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from scripts.family_e2e.manifest import TOOL_CHECKOUT, ResolvedCase +from scripts.family_e2e.server import spawn_in_job, terminate + +EXPORT_VERSION = 1 +WEBV2_DIR = TOOL_CHECKOUT / "invokeai" / "frontend" / "webv2" +EXPORT_TIMEOUT_S = 600 + + +class ExportError(RuntimeError): + pass + + +@dataclass(frozen=True) +class PreparedGraph: + batch: dict[str, Any] + """The `batch` body for enqueue_batch.""" + graph_hash: str + model_keys: frozenset[str] + """Keys of every model identifier in the graph (main model, components, LoRAs, control models).""" + components: dict[str, str] + """The exporter's resolved component choice, for provenance.""" + + +def build_export_request( + cases: list[ResolvedCase], + models: list[dict[str, Any]], + capabilities: Any, + image_names: dict[str, dict[tuple[str, bool], str]], +) -> dict[str, Any]: + """`image_names` maps each case key's (manifest input path, used at the case size) to the uploaded image name.""" + request_cases = [] + for case in cases: + spec = case.spec + names = image_names[case.case_key] + request_cases.append( + { + "id": case.case_key, + "mode": spec.mode, + "model_key": spec.model, + "components": spec.components, + "width": case.width, + "height": case.height, + "steps": spec.steps, + "seed": case.seed, + "positive_prompt": case.prompt, + "negative_prompt": case.negative_prompt, + "loras": [{"key": lora.key, "weight": lora.weight} for lora in spec.loras], + "reference_images": [names[(path, False)] for path in spec.reference_images], + "init_image": ( + None + if spec.init_image is None + else {"image_name": names[(spec.init_image.path, True)], "strength": spec.init_image.strength} + ), + "control": [ + { + "model_key": c.model, + "image_name": names[(c.image, True)], + "weight": c.weight, + "begin": c.begin, + "end": c.end, + } + for c in spec.control + ], + "settings": spec.settings, + } + ) + return {"version": EXPORT_VERSION, "models": models, "capabilities": capabilities, "cases": request_cases} + + +def run_pnpm_exporter(request: dict[str, Any], work_dir: Path) -> dict[str, Any]: + """Runs `pnpm graphs:export` in the tool checkout's webv2 package. Per-case errors stay in the response.""" + pnpm = shutil.which("pnpm") + if pnpm is None: + raise ExportError("pnpm is not on PATH; the graph exporter runs with pnpm from invokeai/frontend/webv2") + work_dir.mkdir(parents=True, exist_ok=True) + # Absolute: pnpm runs the exporter from the webv2 package directory. + request_path = (work_dir / "export-request.json").resolve() + response_path = (work_dir / "export-response.json").resolve() + request_path.write_text(json.dumps(request), encoding="utf-8") + response_path.unlink(missing_ok=True) + # pnpm is a shim (pnpm.cmd on Windows) around node: a timeout must kill the whole tree, not just the shim. + process, job = spawn_in_job( + [pnpm, "graphs:export", str(request_path), str(response_path)], + cwd=WEBV2_DIR, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + stdout, stderr = process.communicate(timeout=EXPORT_TIMEOUT_S) + except subprocess.TimeoutExpired: + raise ExportError(f"pnpm graphs:export did not finish within {EXPORT_TIMEOUT_S}s") from None + finally: + terminate(process, job) + if process.returncode != 0 or not response_path.is_file(): + output = (stderr or stdout).decode("utf-8", errors="replace") + raise ExportError(f"pnpm graphs:export exited with {process.returncode}:\n{output[-4000:]}") + response = json.loads(response_path.read_text(encoding="utf-8")) + if response.get("version") != EXPORT_VERSION: + raise ExportError(f"exporter answered version {response.get('version')!r}, expected {EXPORT_VERSION}") + return response + + +def _model_keys(value: Any) -> Iterator[str]: + # A ModelIdentifierField: {"key", "hash", "name", "base", "type", ...}. + if isinstance(value, dict): + if isinstance(value.get("key"), str) and "base" in value and "type" in value: + yield value["key"] + for item in value.values(): + yield from _model_keys(item) + elif isinstance(value, list): + for item in value: + yield from _model_keys(item) + + +def _replace_strings(value: Any, replacements: dict[str, str]) -> Any: + if isinstance(value, str): + return replacements.get(value, value) + if isinstance(value, dict): + return {k: _replace_strings(v, replacements) for k, v in value.items()} + if isinstance(value, list): + return [_replace_strings(v, replacements) for v in value] + return value + + +def prepare_batch(batch: dict[str, Any], components: dict[str, str], input_ids: dict[str, str]) -> PreparedGraph: + """Applies the runner policy to an exported batch and hashes it. + + `input_ids` maps uploaded image names to a stable identity of the input file. Upload names differ on every + server start, so the hash covers the input's identity instead. + """ + batch = copy.deepcopy(batch) + for node in batch["graph"]["nodes"].values(): + node["use_cache"] = False + batch["runs"] = 1 + hashed = _replace_strings({"graph": batch["graph"], "data": batch.get("data")}, input_ids) + canonical = json.dumps(hashed, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + return PreparedGraph( + batch=batch, + graph_hash=hashlib.sha256(canonical.encode("utf-8")).hexdigest(), + model_keys=frozenset(_model_keys(batch["graph"])), + components=components, + ) + + +def output_image_name(graph: dict[str, Any], session: dict[str, Any]) -> str: + """The image of the graph's one non-intermediate node that produced an image.""" + mapping = session.get("source_prepared_mapping") or {} + results = session.get("results") or {} + names = [] + for node_id, node in graph["nodes"].items(): + # The server's default: a node without the field is not intermediate. + if node.get("is_intermediate", False): + continue + for prepared_id in mapping.get(node_id, []): + image = (results.get(prepared_id) or {}).get("image") + if isinstance(image, dict) and isinstance(image.get("image_name"), str): + names.append(image["image_name"]) + if len(names) != 1: + raise ValueError(f"expected one output image from the graph's non-intermediate nodes, found {len(names)}") + return names[0] diff --git a/scripts/family_e2e/manifest.example.yaml b/scripts/family_e2e/manifest.example.yaml new file mode 100644 index 00000000000..26351a9b6b4 --- /dev/null +++ b/scripts/family_e2e/manifest.example.yaml @@ -0,0 +1,80 @@ +# Family E2E manifest, example for the first wave. Copy it next to your data directory, replace every <...> (paths +# and the model keys of your install; GET /api/v2/models/ lists the keys), then: +# python -m scripts.family_e2e start --manifest +# Relative paths are relative to this file. + +profile: rtx4090 # names runs and references; one profile per machine and GPU +data_dir: # inputs/, references/, runs/, scratch-roots/ +gpu: RTX 4090 # substring of the adapter name; must match one adapter ("Radeon" also matches an AMD iGPU, use "9060") + +server: + # Its invokeai.yaml and a database snapshot are copied; its server must be off. The scratch server shares its + # models directory: it writes models/README.txt and removes install claims of processes that no longer run there. + template_root: