diff --git a/app/public/paper/policybench.pdf b/app/public/paper/policybench.pdf index 4f891c4d..d577f1f8 100644 Binary files a/app/public/paper/policybench.pdf and b/app/public/paper/policybench.pdf differ diff --git a/app/public/paper/web/index.html b/app/public/paper/web/index.html index 2ca8b8f7..cbf1c81b 100644 --- a/app/public/paper/web/index.html +++ b/app/public/paper/web/index.html @@ -250,7 +250,7 @@

Frozen snapshot an Scoring code SHA-256 prefix -baaf2506800b +e75a972a2f98 US run label @@ -299,7 +299,7 @@

Frozen snapshot an -
+
@@ -1096,15 +1096,15 @@

Uncertainty

Sampling uncertainty is substantial at this sample size. In the frozen snapshot, 6 of the other 49 models have an exact-match 95% interval that overlaps the leader’s. Small differences in the leaderboard ordering should therefore be read as ties rather than as a strict ranking, and the rank-range column makes the set of plausible positions explicit. These intervals are a manuscript artifact: the public leaderboard and app report point estimates only, because the benchmark sample, not run-to-run model variation, is the dominant source of uncertainty here.

Cost and latency

Accuracy is one axis; the price and speed of a no-tools answer are another. Table 7 reports each model’s estimated cost to compute one household’s full output vector and its median per-household latency, next to the headline exact rate. Per-household cost spans $0.001 (Claude Haiku 5.5) to $0.541 (Claude Fable 5), and median per-household latency 3 s (Gemini 3.5 Flash-Lite) to 1023 s (Kimi K3). Neither dimension cleanly tracks accuracy: the top-scoring model, GPT-6 Sol at 95.2% exact, sits mid-pack at $0.027 per household and 23 s.

-
-
+
+
+
-Table 7: Estimated cost and median latency per household for the frozen US snapshot, alongside the headline exact-match rate. Each row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge where no reconstruction was available. List-price overrides apply at request time, not retroactively to recorded costs. Models without per-call costs use the frozen release-metadata cost. Latency is the median across households of each household’s summed request time. A dash marks usage or wall-clock latency that was not measured per call. +Table 7: Estimated cost and median latency per household for the frozen US snapshot, alongside the headline exact-match rate. Each row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge where no reconstruction was available. List-price overrides apply at request time, not retroactively to recorded costs. Claude Fable 5’s cost and latency come from its original run’s per-call rows, which its board rows do not carry. Latency is the median across households of each household’s summed request time. A cost dash marks a provider-reported charge of zero on every call (GLM-5.3-Flash (preview)). A latency dash marks rows that record no request time (GPT-5.5, through OpenAI’s batch API) or a median household answered from the harness’s response cache rather than by the provider (GLM-5.2, Kimi K2.6, MiniMax M3, and Qwen3.7-max; see text).
-
- +
@@ -1417,11 +1417,12 @@

Cost and latency

Model
-
-

Cost preserves each call’s recorded total: a request-time reconstruction from token counts where one was available, and the provider-reported charge otherwise. Configured list-price overrides do not reprice recorded calls. Rows without per-call costs use the frozen release-metadata cost. Latency is wall-clock request time as observed by the harness, summed within each household and taken as the median across the 100 households. The per-call timer spans any provider-side retry or rate-limit backoff, so the median is reported in preference to the mean, which a few throttled calls inflate. Latency is not comparable across serving regimes: Claude-family models issue chunked serial calls that lengthen wall-clock time, several models ran through batch or asynchronous dispatch whose logged times reflect submission rather than generation (shown as a dash), and Claude Fable 5’s usage and latency are recorded at the run level rather than per call. Both cost and latency are properties of this run’s providers, prompts, and serving configurations at evaluation time rather than fixed model constants.

+
+

Cost preserves each call’s recorded total: a request-time reconstruction from token counts where one was available, and the provider-reported charge otherwise. Configured list-price overrides do not reprice recorded calls. Claude Fable 5’s board rows carry no usage or timing fields, because they were rebuilt from release dashboard-data-20260701 without those columns; its cost, tokens and latency come from the per-call rows of its original run, which that release publishes. Latency is wall-clock request time as observed by the harness, summed within each household and taken as the median across the 100 households. The timer spans one request as the provider handles it, including anything that happens inside that call. When the harness retries a failed request, only the attempt that succeeded is timed, without the failed attempts or the wait between them. The median is reported in preference to the mean, which a few slow calls inflate. Latency is not comparable across serving regimes. A model served one or three outputs per request (Table 2) sends a household’s calls one after another, and its household time is their sum. GPT-5.5 ran through OpenAI’s batch API, which records no per-call request time, so it shows a dash.

+

The harness also caches provider responses on disk, keyed on the request and the prompt contract version. A request identical to an earlier one is answered from the cache in hundredths of a second, and the timer records that as if it were a call. A replayed response’s cost is computed from that response when its row is written: its token counts at the configured list price, or the provider’s charge embedded in it where no reconstruction exists. A household’s cost therefore counts each of its responses once, whether or not the board run paid for it. GLM-5.2, Kimi K2.6, MiniMax M3, and Qwen3.7-max were finished over several passes, each of which replayed the earlier passes’ successful responses. Each of them records a replay on 53 to 93 of its 100 households, so its median falls under a second and it shows a dash as well. In 11 other models, 29 households sum to under a second. The audit traces 27 of them to an earlier request on the same household, such as an onboarding smoke or a restarted chunk; the other two come from runs that predate per-call timestamps and left no cache or ledger behind. Dropping them would raise no model’s median by more than 2.5 seconds. A further 33 households replay their first recorded call and then reach the provider, so their times leave out that first call. The benchmark card summarizes the replays, and the repository’s latency replay audit lists each household. Both cost and latency are properties of this run’s providers, prompts, and serving configurations at evaluation time rather than fixed model constants.

Simple baselines

Simple baselines help interpret score levels in a zero-heavy benchmark. Table 8 reports an always-zero response and a median-reference-by-output response on the same frozen household sample, scored with the household-impact weighting. These are not model competitors; they show how much of each metric can be earned without household-specific policy calculation. The always-zero baseline scores 69.7% on the headline exact-match rate — below the 85% of outputs that are exact zeros because the weighting concentrates mass in high-dollar output groups that are rarely zero (Section 3.1). The top model clears the baseline by 25.5 points, and the within-1% companion shows the same separation with near-miss tolerance.

diff --git a/app/src/paperSnapshot.json b/app/src/paperSnapshot.json index 79d714ae..aa918c8a 100644 --- a/app/src/paperSnapshot.json +++ b/app/src/paperSnapshot.json @@ -1,6 +1,6 @@ { - "pdfVersion": "d0b236367fb3", + "pdfVersion": "d16eb03ce07d", "responseWindow": "June 12 and October 13, 2026", "snapshotDate": "2026-10-14", - "webVersion": "533a0cb79b6a" + "webVersion": "6cb2fe1ad2b3" } diff --git a/docs/benchmark_card.md b/docs/benchmark_card.md index 09732563..29b1cc02 100644 --- a/docs/benchmark_card.md +++ b/docs/benchmark_card.md @@ -420,23 +420,64 @@ the site's types and leaderboard already filter on `condition`. Each frozen row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge -where no reconstruction was available. List-price overrides apply at request time, not retroactively to -recorded costs. Models without per-call costs use the frozen release-metadata -cost. Published model costs retain these recorded totals rather than repricing -past calls at today's rates. +where no reconstruction was available. List-price overrides apply at request +time, not retroactively to recorded costs. Published model costs retain these +recorded totals rather than repricing past calls at today's rates. GPT-5.5 ran +through OpenAI's batch API, and its costs are reconstructed at standard +synchronous rates. Claude Fable 5's board rows carry no usage fields: they were +rebuilt from release dashboard-data-20260701 without them. Its published cost, +tokens and latency come from the per-call rows of its original run, which that +release publishes as `claude-fable-5-predictions.csv.gz`. The harness caches provider responses on disk, keyed on the request and the -prompt contract version (`policybench/cache.py`). A board request identical to -one sent earlier, such as an onboarding smoke on the same household, returns -the cached response: the row records that response's answers, tokens and cost, -and a latency of about zero. Each prediction row records its call's latency -divided among the call's outputs, so a household's latency is the sum over its -rows. The Muse Spark 1.3 row replays its four smoke households (scenario_000, -scenario_001, scenario_099 and scenario_119), each with the smoke's provider -response id and a household latency of 0.03 seconds or less. In the frozen -predictions other rows also have households under 0.05 seconds by that sum, -among them scenario_000 and scenario_001 for Muse Wonder and Claude Opus 5.5 -and scenario_000 for Claude Haiku 5.5. +prompt contract version (`policybench/cache.py`). A request identical to an +earlier one returns the cached response, and its row records that response's +answers and tokens. Its cost is computed from the replayed response when the +row is written: its token counts at the configured list price, or the +provider's charge embedded in it where no reconstruction exists. Each +household's cost counts its responses once, whether or not the board run paid +for them, and leaves out spend on attempts that did not reach the board. + +## Latency basis + +Each prediction row records its household's request time divided among the +household's rows, so a household's latency is the sum over its rows +(`analysis.household_latency_seconds`). A household that needs several calls +(one or three outputs per request, a repair, or a re-request with a larger +completion budget) makes them one after another and records their sum. A +request the harness retries after a provider error is timed only on the +attempt that succeeded. Published `latencySeconds` is the median +across households. A household with no recorded time is left out rather than +counted as zero, so GPT-5.5, whose batch rows record none, has no latency. + +A cached response is answered in hundredths of a second, and its row records +that lookup as the call's latency. In the frozen predictions, 330 households +sum to under a second, all under 0.12 s; the fastest live household takes +1.75 s. The audit (`docs/latency_replay_audit.md`, with every household in +`docs/latency_replay_audit/households.csv`) traces 328 of them to an earlier +request on the same household. The other two (Grok 4.3's scenario_000 and +Claude Haiku 4.5's scenario_009) come from runs that predate per-call +timestamps and left no cache or ledger. Another 33 households replay their +first recorded call and then reach the provider, so their times leave out that +call. + +- Qwen3.7-max (93 households), GLM-5.2 (80), Kimi K2.6 (75) and MiniMax M3 + (53) were finished over several passes, each of which replayed the earlier + passes' successes. Their published medians, 0.003 to 0.033 seconds, are + cache lookups, not latency, and the paper shows them as a dash. +- Claude Sonnet 5 (9 households) replays its smoke and two restarted chunks. + Muse Spark 1.3 (4) and Muse Wonder, Claude Opus 5.5 and Claude Haiku 5.5 + (2 each) replay their onboarding smokes. DeepSeek V4 Pro and GPT-5.6 Sol, + Luna and Terra (2 each) replay requests sent minutes to an hour before + their runs. Replays can only lower a median. Dropping these households would + raise no published median by more than 2.5 seconds (DeepSeek V4 Pro, 169.9 + to 172.4 s). +- The 33 partial replays are in GLM-5.3 (18), GLM-5.3-Flash (preview) (4), + Kimi K2.6 (4), Claude Sonnet 5 (2), Kimi K3 (2), and GLM-5.2, MiniMax M3 and + Qwen3.7-max (1 each). + +The leaderboard does not display latency. Whether published medians should +leave out replayed households is an open methodology decision. ## Country data paths diff --git a/docs/gpt61sol/design.md b/docs/gpt61sol/design.md index 72607196..5ffe1740 100644 --- a/docs/gpt61sol/design.md +++ b/docs/gpt61sol/design.md @@ -101,9 +101,10 @@ and `test_every_committed_adjudication_decides_a_case_the_seed_judged` run anywhere, and `prepare`, `bind-seed` and every later step check the seed and the stage's binding against it. -The exporter still needs the Fable 5 usage carry-over. Fable 5 ran through the -Anthropic batch adapter, and its committed rows carry no cost, token or latency -fields, so `export_full_run` reports `costUsd` $0 and omits the rest. The live +The exporter still needs the Fable 5 usage carry-over. Fable 5's committed rows +were rebuilt from release 20260701 without their cost, token and latency +fields (an earlier version of this note said it ran through the Anthropic +batch adapter; it ran synchronously, see `docs/latency_replay_audit.md`), so `export_full_run` reports `costUsd` $0 and omits the rest. The live payload has $54.11, 3,850,174 tokens and 97.68 s. The driver copies those four values from the base, and the drift gate then checks every field and the key order. Ox Alpha's `costUsd` of $0.0 comes out of the exporter unchanged, so it diff --git a/docs/latency_replay_audit.md b/docs/latency_replay_audit.md new file mode 100644 index 00000000..0cf4859d --- /dev/null +++ b/docs/latency_replay_audit.md @@ -0,0 +1,252 @@ +# Latency replay audit (frozen US run, October 2026) + +This audit explains every household in the frozen US run whose recorded latency +sums to about zero, and every model whose latency is missing. It then says how +the published figures treat them. The run is +`paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace` +as of release dashboard-data-20261014, with 50 models and 100 households each. +The household-level evidence is `latency_replay_audit/households.csv`, written by +`scripts/audit_latency_replays.py`. + +## How latency is recorded + +The harness times each provider call. A household that needs several calls +(one or three outputs per request, a repair after an incomplete answer, or a +re-request with a larger completion budget) makes them one after another, as +`run_single_no_tools` does for chunks. Their times are summed, and the sum is +divided equally among the household's rows (`elapsed_seconds / batch_size` +in `policybench/eval_no_tools.py`; all three lines date from 2026-05-01, before +every board run). A household's latency is therefore the sum over its rows. The +timer wraps the provider call, so whatever happens inside that call counts. +When the harness itself retries a failed request, it restarts the timer and +keeps only the attempt that succeeded; the failed attempts and the backoff +between them are not counted (the retry loop in `_request_predictions_once`, the +same in the July code). +`analysis.household_latency_seconds` computes that sum. +`analysis.model_cost_latency` publishes the median across households as +`latencySeconds`. A household whose rows carry no time is left out of the +median rather than counted as zero. + +The harness also caches provider responses on disk (`policybench/cache.py`, +litellm's disk cache namespaced by the prompt contract version). A request +identical to an earlier one is answered from the cache. The timer measures the +lookup, a few hundredths of a second, and the row records it as if it were the +call. + +## Two models have no latency in the board rows + +**GPT-5.5** ran through OpenAI's batch API. All 716 of its board response ids +come from one batch (`batch_6a49d188…`, submitted 2026-07-05 03:37 UTC and +completed 11 minutes later; `results/local/gpt55_default_run_20260706`). Batch +rows leave `elapsed_seconds` and both request timestamps empty by design, +because a batch round trip includes queue time (`policybench/batch_eval.py`). +Its cost is reconstructed from token counts at standard synchronous rates. + +**Claude Fable 5** ran synchronously from 2026-07-01 22:26 to 2026-07-02 01:53 +UTC. The batch adapter (`batch_eval.py`, #88) first landed at 10:03 UTC on +2026-07-02, and the original rows record a per-call request time, which batch +rows never carry. Those rows, with raw responses, tokens, costs and timings, +are the `claude-fable-5-predictions.csv.gz` asset of release +dashboard-data-20260701. The board rows were rebuilt from that release on +2026-07-02 (57d26f1c) without the usage, timing and raw-response columns. Their +predictions and explanations equal the original rows'. The published +`costUsd` ($54.11), `totalTokens` (3,850,174) and `latencySeconds` (97.68 s) +come from the original rows exactly. The release drivers carry them forward as +`CARRIED_USAGE`, and 6cb7b488 restored them after a rebuild had zeroed them. +Three of the original households (scenario_044, scenario_114 and scenario_115) +sum to under a second. No cache or ledger from that run survives to show +whether they were replays. Without them the median is 97.71 s. + +The snapshot manifest, the release drivers' comments and the paper used to say +that Fable 5 was "served through the Anthropic batch adapter" or that its usage +was "recorded at the run level". Neither was true, and both are corrected. + +## Replayed households + +A household counts as replayed when its first recorded response existed +before the household's request began. The evidence is any of the following: + +- **Disk cache.** The response's id sits in a litellm cache entry stored more + than a minute before the request (`.policybench_cache` in the main checkout, + 121,648 entries). +- **OpenRouter id.** OpenRouter ids are `gen--…`, and the time + is more than a minute before the request. On all 962 live households that + carry one, it falls within 23 seconds of `request_started_at`, and within 10 + seconds on 99% of them. +- **OpenAI id.** An OpenAI Responses id embeds its creation time in hex, after + litellm's base64 wrapper is removed. On all 694 live households that carry + one, it falls within 4 seconds. +- **Spend ledger.** The board run's spend ledger records `cache_hit: true` for + the call. +- **Smoke.** The same response id was served live in the model's onboarding + smoke. + +A full replay sums to under a second: every call in the household came from +the cache. Every full replay in the run sums to under 0.12 s, while the fastest +live household takes 1.75 s (Gemini 3.5 Flash-Lite), so the one-second line +(`analysis.REPLAYED_HOUSEHOLD_SECONDS`) separates them cleanly. A partial replay +sums to a second or more. Its first recorded call was a replay, and a later +call (another chunk, a repair, or a re-request with a larger completion budget) +went to the provider. Its time leaves out the replayed call's original +duration. In GLM-5.3's scenario_007, for example, the ledger shows the replayed +16,384-token call, then live calls at 32,768 and 65,536 tokens. + +| Model | Full | Partial | Where the replayed responses came from | Published median (s) | Without sub-second households (s) | +|---|---|---|---|---|---| +| Qwen3.7-max | 93 | 1 | earlier passes of its run | 0.009 | 917.9 (7 households) | +| GLM-5.2 | 80 | 1 | its JSON-contract smoke (scenario_000 and scenario_001); earlier passes of its run | 0.007 | 384.8 (20 households) | +| Kimi K2.6 | 75 | 4 | its JSON-contract smoke (scenario_000 and scenario_001); earlier passes of its run | 0.033 | 1,194.2 (25 households) | +| MiniMax M3 | 53 | 1 | earlier passes of its run | 0.003 | 10.7 (47 households) | +| Claude Sonnet 5 | 9 | 2 | its smoke (scenario_000); two restarted chunks (8 full, 2 partial) | 63.9 | 64.8 | +| Muse Spark 1.3 | 4 | 0 | its onboarding smoke | 40.6 | 41.1 | +| Claude Haiku 5.5 | 2 | 0 | its onboarding smoke | 5.1 | 5.1 | +| Claude Opus 5.5 | 2 | 0 | its onboarding smoke | 24.0 | 24.0 | +| Muse Wonder | 2 | 0 | its onboarding smoke | 26.1 | 26.5 | +| DeepSeek V4 Pro | 2 | 0 | requests about an hour before its run; no file survives | 169.9 | 172.4 | +| GPT-5.6 Luna | 2 | 0 | requests 5 minutes before its run; no file survives | 14.4 | 14.5 | +| GPT-5.6 Sol | 2 | 0 | requests 7 minutes before its run; no file survives | 36.3 | 36.4 | +| GPT-5.6 Terra | 2 | 0 | requests 6 to 7 minutes before its run; no file survives | 18.9 | 19.1 | +| GLM-5.3 | 0 | 18 | earlier rounds of its supervised run | 134.2 | 134.2 | +| GLM-5.3-Flash (preview) | 0 | 4 | earlier rounds of its supervised run | 241.7 | 241.7 | +| Kimi K3 | 0 | 2 | earlier in its run | 1,023.3 | 1,023.3 | + +That is 328 full and 33 partial replays across 16 models. Two more households +sum to under a second with no surviving evidence either way: Grok 4.3's +scenario_000 (0.022 s) and Claude Haiku 4.5's scenario_009 (0.026 s). Both +were already that way in the June 13-model board. Those runs predate per-call +timestamps and spend ledgers, and their cache does not survive. Without them +the two medians move from 22.13 to 22.21 s and from 47.67 to 47.69 s. + +The 0.05-second line used before this audit missed four replays: Kimi K2.6's +scenario_008, scenario_026 and scenario_064 (0.056 to 0.068 s) and Claude Haiku +5.5's scenario_001 (0.118 s). The board ledger records `cache_hit: true` for +the last one. + +### The four open-weight models: a multi-pass run + +Kimi K2.6, GLM-5.2, MiniMax M3 and Qwen3.7-max were added on 2026-07-05 to 07. +The run notes (`results/local/openweight_run_20260705.md`) record that +concurrent long-lived requests exhausted the OpenRouter key's connections, so +whole households errored and wrote no rows. `results/local/rerun_openweight.sh` +and `results/local/grind_model.sh` then re-ran each model's full range several +times. Each pass deletes the part files and re-requests every household, and the +cache answered the households earlier passes had completed. The board rows are +each model's final pass. The notes record MiniMax M3's first pass at 53 of 100 +households, and 53 of its households are full replays. They record Qwen3.7-max +at 93/100 in the round before its last, and 93 of its households are full +replays. Kimi K2.6's +scenario_109 shows a partial replay at work. In earlier rounds its first call +succeeded and was cached, and the household then failed on an explanation +repair that timed out at 300 s (`part_84_100.grind*.log`). In the final pass +its first call was replayed and its later calls reached the provider, so the +household records 777 s next to the replayed first call's id. + +For these four models the households left after removing replays are the ones +earlier passes failed on, many of them after timeouts and retries. Their +medians (10.7 to 1,194 s) are not estimates of the model's latency. For +OpenRouter responses, the cache's store time minus the id's creation time +approximates a call's duration. It agrees with the recorded time to a median of +0.5 s on live households. It times only the first call, though, the one whose +id the rows keep, so it falls far short wherever a household made several +calls. Applied to the replays it gives medians +of about 133 s (Qwen3.7-max), 117 s (GLM-5.2), 294 s (Kimi K2.6) and 10 s +(MiniMax M3). Qwen3.7-max, GLM-5.2 and Kimi K2.6 were served three outputs +per request, so their households make several calls and the first one is only +part of each household's time. It is a different quantity, and nothing +publishes it. + +## What the published figures do with them + +- **`modelStats.latencySeconds` in the dashboard data** (release asset + `dashboard-data.json`, the same values as the frozen `data.json.gz`) is the + median of household sums with replays counted as calls. Every value + reproduces from the frozen predictions except Fable 5's, which is carried + from its original rows. For the four open-weight models it publishes 0.003 to + 0.033 seconds, which is cache-lookup time and not latency. For the eleven + other models with sub-second households it publishes medians lowered by up + to 2.5 s (DeepSeek V4 Pro). Partial replays lower medians further, by an + unknown amount. + GPT-5.5 has no `latencySeconds`. +- **The leaderboard** shows no latency: the column was dropped on 2026-07-05 + (#100), and `latencySeconds` is not rendered anywhere in the app. +- **The paper's cost-and-latency table** treats a sub-second or missing + median as unmeasured and falls back to `modelStats`. The four open-weight + models and GPT-5.5 therefore show a dash, and Fable 5 shows its carried + 98 s. The paragraph under the table used to attribute the dashes to "batch or + asynchronous dispatch whose logged times reflect submission rather than + generation". That fits no model in the board: GPT-5.5's batch rows log no + time at all, and the four sub-second models ran synchronously. The paragraph + and caption now give the real causes, computed from the frozen data + (`PaperResults.latency_dash_models` and its neighbours). The table values + are unchanged. Without sub-second households, Claude Sonnet 5 would read 65 s + rather than 64 s, DeepSeek V4 Pro 172 s rather than 170 s and GPT-5.6 Luna + 15 s rather than 14 s. +- **`costUsd`.** A replayed row's cost is computed when the row is written, + from the replayed response itself (`_extract_usage_metadata`). Where a token + reconstruction exists, its token counts are priced at the list prices + configured then; otherwise it is the provider's charge embedded in the + response. Spend ledgers record each cache hit's spend as $0 and that computed + cost as `cached_response_cost_usd`, and on all ten ledger-era full replays + (Muse Spark 1.3, Muse Wonder, Claude Opus 5.5 and Claude Haiku 5.5) the board + household's cost equals it. That shows the row carries the computed value, + not that it matches what the original call was billed. For OpenRouter models + priced by the provider's charge, the cached response carries the original + charge: on 43 of MiniMax M3's full replays the row cost equals the cost + embedded in the cached response. A household's published cost + therefore counts each of its responses once, whether or not the board run + paid for it, and leaves out spend on attempts that did not make the board, + such as the open-weight models' failed passes. That matches the paper's + description ("each call's recorded total"), so no cost figure misleads + because of replays. Fable 5's `costUsd` is computed from its original rows, + so its NaN board rows do not reach it. `model_cost_latency` itself returns + $0 for a model whose rows carry no tokens when a price override exists, as + Fable 5's do. The release drivers overwrite that with the carried value, so + nothing published shows it. +- **GLM-5.3-Flash (preview)** (artifact key `ox-alpha`) records a + provider-reported charge of $0.00 on all 1,984 rows. The leaderboard prints + $0.000 per household and the paper prints a dash, which the caption now + explains. This is unrelated to replays. + +## Changing the published numbers + +Changing `latencySeconds` changes published numbers, so it is a methodology +choice for Max, queued as a decision. This audit changes only disclosures, +comments, and the tests that tie them to the data. The options: + +- **A. Keep the numbers.** This PR discloses them. The dashboard data keeps + 0.003 to 0.033 s for the four open-weight models. +- **B. Leave replays out of the median (recommended).** `model_cost_latency` + would take the median over households that sum to a second or more. It would + publish no latency when more than half a model's households are replays. + Away from an exact tie, that is the rule the paper's dash already applies + (`test_a_median_under_a_second_means_most_households_replayed`); at exactly + half, the paper's median can fall on either side of a second. The four + open-weight models would lose `latencySeconds`. Eleven other medians would + rise by 0.02 to 2.5 s. In the paper table, Claude Sonnet 5 would read 65 s + instead of 64 s, DeepSeek V4 Pro 172 s instead of 170 s and GPT-5.6 Luna 15 s + instead of 14 s, and nothing else would change after rounding. Partial + replays would stay as disclosed lower bounds, because finding them needs + response-id times or the caches, not just household times. +- **C. Reconstruct the replayed calls' durations** from cache store time minus + OpenRouter creation time. That gives about 133, 117, 294 and 10 s for the four + models, but it times only the call that succeeded, not the retries the + harness timer includes. +- **D. Re-time the four models.** Re-send their replayed households live with + the cache off, for latency only, leaving their predictions alone. The full + replays' recorded cost suggests about $47 (Kimi K2.6 $28.38, Qwen3.7-max + $11.99, GLM-5.2 $6.23, MiniMax M3 $0.36). The timings would come + from a later date than the board's answers. + +## Reproducing + +```bash +uv run python scripts/audit_latency_replays.py \ + --cache-dir .policybench_cache \ + --ledger-root results/local \ + --ledger-root ~/reviews +``` + +The cache, ledgers and smoke files exist only on the machine that ran the +board. The id-time evidence and every sub-second household can be recomputed +from the frozen predictions alone, and `tests/test_latency_replay_audit.py` +does so. diff --git a/docs/latency_replay_audit/households.csv b/docs/latency_replay_audit/households.csv new file mode 100644 index 00000000..f61108ea --- /dev/null +++ b/docs/latency_replay_audit/households.csv @@ -0,0 +1,364 @@ +model,scenario_id,household_latency_seconds,replay,evidence,source,first_response_id,request_started_at,response_existed_at +claude-haiku-4.5,scenario_009,0.026461,untraced,,untraced,chatcmpl-9c7cea35-2052-496a-bf4e-3602ca12515c,, +claude-haiku-5.5,scenario_000,0.017322,full,smoke_file+smoke_ledger+spend_ledger,onboarding smoke,chatcmpl-992b7cba-d5e8-42da-9449-c06899393c28,1791495684.702,1791495644.564 +claude-haiku-5.5,scenario_001,0.117501,full,smoke_file+smoke_ledger+spend_ledger,onboarding smoke,chatcmpl-308e4847-b356-4499-a13a-3ab9f3208450,1791495684.777,1791495651.476 +claude-opus-5.5,scenario_000,0.029634,full,smoke_file+smoke_ledger+spend_ledger,onboarding smoke,chatcmpl-e65b82db-643d-47f9-b87a-d6143e484ca6,1790095878.103,1790095745.462 +claude-opus-5.5,scenario_001,0.029532,full,smoke_file+smoke_ledger+spend_ledger,onboarding smoke,chatcmpl-f5a49889-34ba-4ef9-8e78-3cd1cbd21ef7,1790095878.103,1790095771.107 +claude-sonnet-5,scenario_000,0.031374,full,cache+smoke_file,onboarding smoke,chatcmpl-d78ec31e-c962-43fd-a568-a6f2f8fc0214,1782978630.541,1782978530.271 +claude-sonnet-5,scenario_076,0.034917,full,cache,same run,chatcmpl-a1f0c372-ff8e-441d-9f94-18d544d87f47,1782980398.594,1782979796.859 +claude-sonnet-5,scenario_077,0.014105,full,cache,same run,chatcmpl-1ead1bd1-376b-45fe-8dad-599790a290be,1782980398.635,1782979899.118 +claude-sonnet-5,scenario_078,0.013461,full,cache,same run,chatcmpl-83be4c02-4aed-4c14-ac07-bd271ebb51c4,1782980398.652,1782979955.733 +claude-sonnet-5,scenario_079,0.014319,full,cache,same run,chatcmpl-89a48391-de8f-4e5b-a480-fe521adf292f,1782980398.669,1782980015.737 +claude-sonnet-5,scenario_080,60.086180,partial,cache,same run,chatcmpl-1a8790f3-4377-4791-8019-f8ec29b6b6cc,1782980398.686,1782980097.957 +claude-sonnet-5,scenario_108,0.028484,full,cache,same run,chatcmpl-0d161de2-daa8-474e-86ce-34c9856b4694,1782980402.165,1782979767.387 +claude-sonnet-5,scenario_109,0.040883,full,cache,same run,chatcmpl-d0c0f199-c820-4f87-bb5c-2efa48a36bde,1782980402.197,1782979823.052 +claude-sonnet-5,scenario_110,0.015177,full,cache,same run,chatcmpl-3d121844-95b6-43b5-b3cc-6c2704ffc17a,1782980402.247,1782979963.904 +claude-sonnet-5,scenario_111,0.017898,full,cache,same run,chatcmpl-24573c76-0a32-4835-b898-c519e810f0bd,1782980402.266,1782980022.908 +claude-sonnet-5,scenario_112,43.325586,partial,cache,same run,chatcmpl-f3062f00-4cf9-431b-afb0-dc821d65fb1c,1782980402.287,1782980090.591 +deepseek-v4-pro,scenario_000,0.008594,full,cache,before the board rows,9ef950f9-59da-4f99-9719-3ec6880ca9bd,1783264088.998,1783260231.726 +deepseek-v4-pro,scenario_001,0.001136,full,cache,before the board rows,902543e5-70c5-485d-a8c4-045a6f87e621,1783264089.007,1783260449.021 +glm-5.2,scenario_000,0.016873,full,cache+openrouter_id+smoke_file,onboarding smoke,gen-1783264675-wpwzQ9139F0ixJwnYjU6,1783331940.272,1783264675.000 +glm-5.2,scenario_001,0.007684,full,cache+openrouter_id+smoke_file,onboarding smoke,gen-1783264883-yVzMGCt6HL5zeDEZ1x21,1783331940.290,1783264883.000 +glm-5.2,scenario_002,0.007263,full,cache+openrouter_id,before the board rows,gen-1783266583-dTu6RztBLx3TX7bklky6,1783331940.299,1783266583.000 +glm-5.2,scenario_003,0.005803,full,cache+openrouter_id,before the board rows,gen-1783267235-LNoAViUajkhCjI10rkp8,1783331940.308,1783267235.000 +glm-5.2,scenario_004,0.006318,full,cache+openrouter_id,before the board rows,gen-1783267319-8HesUaIcOkl8n3N2jZQD,1783331940.315,1783267319.000 +glm-5.2,scenario_005,0.008563,full,cache+openrouter_id,before the board rows,gen-1783267850-H6Ou4TYXVgSzFKaVnry8,1783331940.322,1783267850.000 +glm-5.2,scenario_007,0.007492,full,cache+openrouter_id,before the board rows,gen-1783268397-EAIbLTW31jvbpSZcDl2R,1783331940.332,1783268397.000 +glm-5.2,scenario_008,0.030048,full,cache+openrouter_id,before the board rows,gen-1783269453-flMrbpFjayacmwkJJZ2S,1783331940.340,1783269453.000 +glm-5.2,scenario_009,0.005580,full,cache+openrouter_id,before the board rows,gen-1783271962-0K2VHknWlD1llXiMua1Y,1783331940.374,1783271962.000 +glm-5.2,scenario_012,0.007640,full,cache+openrouter_id,before the board rows,gen-1783272224-ppxmCCZVbJx83yRWUeVl,1783331940.380,1783272224.000 +glm-5.2,scenario_013,0.003717,full,cache+openrouter_id,before the board rows,gen-1783273142-UHDA014Q8pNNm0GLS4Pc,1783331940.389,1783273142.000 +glm-5.2,scenario_014,0.006413,full,cache+openrouter_id,before the board rows,gen-1783273348-Xj1UqttlBvGfyFHeDfb5,1783331940.393,1783273348.000 +glm-5.2,scenario_015,0.005958,full,cache+openrouter_id,before the board rows,gen-1783274195-dnJUUoeOYDxKX2w7cxGT,1783331940.401,1783274195.000 +glm-5.2,scenario_016,0.004191,full,cache+openrouter_id,before the board rows,gen-1783274669-dP0cmBPjXNPVCiuoNXtu,1783331940.407,1783274669.000 +glm-5.2,scenario_018,0.004627,full,cache+openrouter_id,before the board rows,gen-1783274711-S631hE9SOEtd3TadO5NN,1783331940.412,1783274711.000 +glm-5.2,scenario_020,0.004869,full,cache+openrouter_id,before the board rows,gen-1783274850-DWxn5YzbSYEEPLef22j1,1783331940.418,1783274850.000 +glm-5.2,scenario_021,0.004261,full,cache+openrouter_id,before the board rows,gen-1783275012-LdzAUp9Z8K3VntB1Tcje,1783331940.424,1783275012.000 +glm-5.2,scenario_022,0.007669,full,cache+openrouter_id,before the board rows,gen-1783275191-F8AF2A7iJv0aw7QxzoGo,1783331940.429,1783275191.000 +glm-5.2,scenario_023,0.007770,full,cache+openrouter_id,before the board rows,gen-1783275779-PfA1SQChuC6s5eqhGhdT,1783331940.437,1783275779.000 +glm-5.2,scenario_025,0.007165,full,cache+openrouter_id,before the board rows,gen-1783284666-lR2mGGVNlZZCEfA1GoHS,1783331940.446,1783284666.000 +glm-5.2,scenario_026,0.011444,full,cache+openrouter_id,before the board rows,gen-1783285120-qHCnsZjM8p00HlPDvcq9,1783331940.454,1783285120.000 +glm-5.2,scenario_027,0.012097,full,cache+openrouter_id,before the board rows,gen-1783285649-q6J5w9m1opbs0yAwdbUd,1783331940.468,1783285649.000 +glm-5.2,scenario_028,0.020727,full,cache+openrouter_id,before the board rows,gen-1783285698-jHyIHr6uFjh0SwpmLEFI,1783331940.482,1783285698.000 +glm-5.2,scenario_029,0.003923,full,cache+openrouter_id,before the board rows,gen-1783285783-m1bwLFWLkrdc3r6iu6Xm,1783331940.506,1783285783.000 +glm-5.2,scenario_030,0.003935,full,cache+openrouter_id,before the board rows,gen-1783285895-O8sipUdlWTV1rnSMK8m6,1783331940.511,1783285895.000 +glm-5.2,scenario_031,0.017882,full,cache+openrouter_id,before the board rows,gen-1783286076-BMAOcIJeOd1MfEPLqv41,1783331940.269,1783286076.000 +glm-5.2,scenario_032,0.023159,full,cache+openrouter_id,before the board rows,gen-1783286232-gAxiAq8YrTbMPsHXqM7U,1783331940.288,1783286232.000 +glm-5.2,scenario_033,0.007156,full,cache+openrouter_id,before the board rows,gen-1783287973-UvbqyCUGudTN6kBzTnmQ,1783331940.313,1783287973.000 +glm-5.2,scenario_036,0.007160,full,cache+openrouter_id,before the board rows,gen-1783288191-2Ko3ftXnBRVOTQdVabuL,1783331940.322,1783288191.000 +glm-5.2,scenario_037,0.004868,full,cache+openrouter_id,before the board rows,gen-1783289078-HSVaQRabeUmKRRnHWzMb,1783331940.330,1783289078.000 +glm-5.2,scenario_038,0.018407,full,cache+openrouter_id,before the board rows,gen-1783289456-0GCSJaw9iMhZShzp6Ucr,1783331940.335,1783289456.000 +glm-5.2,scenario_039,0.006487,full,cache+openrouter_id,before the board rows,gen-1783290940-JehQ5uSX1vbRkVxdX7wu,1783331940.356,1783290940.000 +glm-5.2,scenario_040,0.006169,full,cache+openrouter_id,before the board rows,gen-1783291214-HFIZyXsUnwrbzgNNKf4A,1783331940.363,1783291214.000 +glm-5.2,scenario_042,0.006445,full,cache+openrouter_id,before the board rows,gen-1783291447-f6YPEnMXI1OdCZfRdL49,1783331940.370,1783291447.000 +glm-5.2,scenario_043,0.004685,full,cache+openrouter_id,before the board rows,gen-1783291868-Jab45K33Zp8c5MT8pTVz,1783331940.378,1783291868.000 +glm-5.2,scenario_044,0.004609,full,cache+openrouter_id,before the board rows,gen-1783292488-ZL2duvKdwMKOD22SfKEH,1783331940.383,1783292488.000 +glm-5.2,scenario_045,0.006946,full,cache+openrouter_id,before the board rows,gen-1783292668-DUKUAUgjj8BOu920xIcQ,1783331940.389,1783292668.000 +glm-5.2,scenario_046,0.015504,full,cache+openrouter_id,before the board rows,gen-1783293350-eVBmjZqGx5ilW5av8q24,1783331940.396,1783293350.000 +glm-5.2,scenario_048,0.004639,full,cache+openrouter_id,before the board rows,gen-1783294113-gigBTYbMi87Slhljic2t,1783331940.414,1783294113.000 +glm-5.2,scenario_049,0.013697,full,cache+openrouter_id,before the board rows,gen-1783294772-VIbVIaIun5pCLxBqcI85,1783331940.419,1783294772.000 +glm-5.2,scenario_051,0.007044,full,cache+openrouter_id,before the board rows,gen-1783295024-4z0xWnmtg6rgIV4vMgAK,1783331940.435,1783295024.000 +glm-5.2,scenario_052,0.005729,full,cache+openrouter_id,before the board rows,gen-1783295505-J5mDMuhr39zvV9Pa46rN,1783331940.443,1783295505.000 +glm-5.2,scenario_053,0.003807,full,cache+openrouter_id,before the board rows,gen-1783295787-RkLKvfcewg0shPE6dOCl,1783331940.450,1783295787.000 +glm-5.2,scenario_054,0.011648,full,cache+openrouter_id,before the board rows,gen-1783295869-je6tQQQ8sm4RZaKTZu6O,1783331940.455,1783295869.000 +glm-5.2,scenario_055,0.004366,full,cache+openrouter_id,before the board rows,gen-1783296736-ZiHKeICIFbSE5NHxY9Np,1783331940.468,1783296736.000 +glm-5.2,scenario_056,0.008079,full,cache+openrouter_id,before the board rows,gen-1783296833-mndjxQmm5xxublVHP0RP,1783331940.473,1783296833.000 +glm-5.2,scenario_057,0.010171,full,cache+openrouter_id,before the board rows,gen-1783297498-unnyjX1F09SBP25meHPN,1783331940.482,1783297498.000 +glm-5.2,scenario_059,0.004785,full,cache+openrouter_id,before the board rows,gen-1783298873-ATvwj93EywbbNw0JP3Sr,1783331940.493,1783298873.000 +glm-5.2,scenario_060,0.006202,full,cache+openrouter_id,before the board rows,gen-1783298985-mzDG731e9oEjgBkIjxHw,1783331940.499,1783298985.000 +glm-5.2,scenario_062,0.004458,full,cache+openrouter_id,before the board rows,gen-1783299385-YOc1JzziCuCiyegqFuOO,1783331940.506,1783299385.000 +glm-5.2,scenario_064,0.037649,full,cache+openrouter_id,before the board rows,gen-1783266583-QdBf21Hct0zzY9pXzIZZ,1783331940.269,1783266583.000 +glm-5.2,scenario_066,0.007620,full,cache+openrouter_id,before the board rows,gen-1783268104-mXVnAaUCdHg4IMUH9S10,1783331940.311,1783268104.000 +glm-5.2,scenario_067,0.006880,full,cache+openrouter_id,before the board rows,gen-1783268952-ivYs5IMugKXLxPsj3qU1,1783331940.319,1783268952.000 +glm-5.2,scenario_068,0.006608,full,cache+openrouter_id,before the board rows,gen-1783269620-5G9p4Hu1BhI1ZZcROLMG,1783331940.327,1783269620.000 +glm-5.2,scenario_070,0.005462,full,cache+openrouter_id,before the board rows,gen-1783270125-HFLPRQj4UQC7NrWBcAc6,1783331940.335,1783270125.000 +glm-5.2,scenario_071,0.004727,full,cache+openrouter_id,before the board rows,gen-1783270368-OqxlidC2Sus8r9gXvZ2S,1783331940.341,1783270368.000 +glm-5.2,scenario_072,0.005811,full,cache+openrouter_id,before the board rows,gen-1783284434-C6r0fYsrE1BfqQriqVsP,1783331940.347,1783284434.000 +glm-5.2,scenario_073,0.005650,full,cache+openrouter_id,before the board rows,gen-1783284602-0b0Ked8ikI4CtMZrW1z1,1783331940.353,1783284602.000 +glm-5.2,scenario_074,0.005542,full,cache+openrouter_id,before the board rows,gen-1783284979-Mh64hhfSPmjlh910qTK6,1783331940.360,1783284979.000 +glm-5.2,scenario_075,0.007999,full,cache+openrouter_id,before the board rows,gen-1783285193-B1zfvrUkatvGpxORmdLM,1783331940.367,1783285193.000 +glm-5.2,scenario_076,0.007836,full,cache+openrouter_id,before the board rows,gen-1783285578-gwmPwYJr60p7p0bsbpsp,1783331940.375,1783285578.000 +glm-5.2,scenario_077,0.006071,full,cache+openrouter_id,before the board rows,gen-1783285905-MEbQM65Gj0leNbJPbH0L,1783331940.385,1783285905.000 +glm-5.2,scenario_078,0.004059,full,cache+openrouter_id,before the board rows,gen-1783286197-8bKqbLJsJxOrJKOIUwsk,1783331940.392,1783286197.000 +glm-5.2,scenario_079,0.009051,full,cache+openrouter_id,before the board rows,gen-1783286301-6tjVvXwGUY0OKeMvPTMG,1783331940.396,1783286301.000 +glm-5.2,scenario_080,0.010218,full,cache+openrouter_id,before the board rows,gen-1783286946-NGbwqQc0oX0CVmUQ2wWO,1783331940.406,1783286946.000 +glm-5.2,scenario_081,0.005579,full,cache+openrouter_id,before the board rows,gen-1783287433-kN9fZx22inubux95Wa7M,1783331940.418,1783287433.000 +glm-5.2,scenario_082,0.016309,full,cache+openrouter_id,before the board rows,gen-1783288326-mIJUhf8aXFhHcnwd1rIi,1783331940.425,1783288326.000 +glm-5.2,scenario_083,0.004758,full,cache+openrouter_id,before the board rows,gen-1783289434-n33lp4NDlIqs11pBA3Nk,1783331940.443,1783289434.000 +glm-5.2,scenario_084,0.005038,full,cache+openrouter_id,before the board rows,gen-1783289595-bVOV59e4DFEKfYfFbSlw,1783331940.449,1783289595.000 +glm-5.2,scenario_085,0.005083,full,cache+openrouter_id,before the board rows,gen-1783289830-KsJDuoKjQdmuL61aH94h,1783331940.454,1783289830.000 +glm-5.2,scenario_086,0.007000,full,cache+openrouter_id,before the board rows,gen-1783290022-D6zvgFVuW5HI1HmAGUaY,1783331940.460,1783290022.000 +glm-5.2,scenario_088,0.006379,full,cache+openrouter_id,before the board rows,gen-1783290330-GxRGZ71gWs0d88KAyJFy,1783331940.468,1783290330.000 +glm-5.2,scenario_089,0.006977,full,cache+openrouter_id,before the board rows,gen-1783291192-6Wrf9yE72w2bAD1MUOeK,1783331940.476,1783291192.000 +glm-5.2,scenario_090,0.003895,full,cache+openrouter_id,before the board rows,gen-1783291765-khoqjZLbGGQEp8Ez3B6B,1783331940.484,1783291765.000 +glm-5.2,scenario_091,0.004962,full,cache+openrouter_id,before the board rows,gen-1783292374-czt8mo2NroyfiRmGkZjQ,1783331940.488,1783292374.000 +glm-5.2,scenario_092,0.015390,full,cache+openrouter_id,before the board rows,gen-1783292562-BzyJY9NNHQF47MpCUI9N,1783333189.265,1783292562.000 +glm-5.2,scenario_093,0.007451,full,cache+openrouter_id,before the board rows,gen-1783292717-jSvawisBww1r8gqsMdu6,1783333189.281,1783292717.000 +glm-5.2,scenario_095,0.010188,full,cache+openrouter_id,before the board rows,gen-1783293828-bUEECmItUiDAZiJezOwC,1783333189.290,1783293828.000 +glm-5.2,scenario_098,0.005563,full,cache+openrouter_id,before the board rows,gen-1783294390-KH8cjOLjuqhmfkpVrPZK,1783333189.301,1783294390.000 +glm-5.2,scenario_099,0.013693,full,cache+openrouter_id,before the board rows,gen-1783331940-Aebly3eW1KYrgKWRZtu1,1783333189.308,1783331940.000 +glm-5.2,scenario_100,39.829285,partial,cache+openrouter_id,same run,gen-1783332255-zOaEFi0cUmXvZnyfHh4a,1783333189.323,1783332255.000 +glm-5.3,scenario_007,670.949044,partial,openrouter_id+spend_ledger,same run,gen-1788443437-kHB3di20R9eBcnvuAX6S,1788522468.992,1788443437.000 +glm-5.3,scenario_028,280.114137,partial,openrouter_id+spend_ledger,same run,gen-1788444666-BfwlvmD6jq7BRSDJbm3s,1788531688.961,1788444666.000 +glm-5.3,scenario_037,350.955777,partial,openrouter_id+spend_ledger,same run,gen-1788445259-ZoY2rZUiydfACfOec5d5,1788529640.906,1788445259.000 +glm-5.3,scenario_042,366.712537,partial,openrouter_id+spend_ledger,same run,gen-1788445513-9nhTScDFLYdxyAoTSg0n,1788535656.584,1788445513.000 +glm-5.3,scenario_043,55.408105,partial,openrouter_id+spend_ledger,same run,gen-1788445558-ZUZs55vSz5c1O3kHHDVM,1788523168.543,1788445558.000 +glm-5.3,scenario_046,102.070615,partial,openrouter_id+spend_ledger,same run,gen-1788445754-PFwn6BupdP1KRTp6UFKo,1788523263.040,1788445754.000 +glm-5.3,scenario_057,546.675098,partial,openrouter_id+spend_ledger,same run,gen-1788446232-xmkvi6ebZ6muMtxWdjmh,1788523615.857,1788446232.000 +glm-5.3,scenario_062,8.507851,partial,openrouter_id+spend_ledger,same run,gen-1788523847-iwtKxP7WTlCPcaYP5q2Y,1788556429.722,1788523847.000 +glm-5.3,scenario_064,3671.612833,partial,openrouter_id+spend_ledger,same run,gen-1788529640-XjMEO5vxBk5wuCwEr5NF,1788563482.334,1788529640.000 +glm-5.3,scenario_075,259.822257,partial,openrouter_id+spend_ledger,same run,gen-1788446387-clYogpRgQ6SiBl7JPxZA,1788524652.971,1788446387.000 +glm-5.3,scenario_086,112.633520,partial,openrouter_id+spend_ledger,same run,gen-1788525085-dTSkM98ErRDqCGEMumnx,1788530003.948,1788525085.000 +glm-5.3,scenario_091,45.313927,partial,openrouter_id+spend_ledger,same run,gen-1788525491-pK5oEjPwwzodUcGmRe9T,1788530129.901,1788525491.000 +glm-5.3,scenario_099,124.193102,partial,openrouter_id+spend_ledger,same run,gen-1788526049-zCakBsDZh7x7CoHgXNRV,1788530559.337,1788526049.000 +glm-5.3,scenario_102,158.606853,partial,openrouter_id+spend_ledger,same run,gen-1788526680-wE2KxUb2HCghIoIRzMxD,1788530559.338,1788526680.000 +glm-5.3,scenario_108,71.216645,partial,openrouter_id+spend_ledger,same run,gen-1788526914-2Ipat7Pm9uGb7w7LY3fH,1788531983.981,1788526914.000 +glm-5.3,scenario_117,84.563591,partial,openrouter_id+spend_ledger,same run,gen-1788446575-uo8BgHZi21mleE4CMwSr,1788527751.585,1788446575.000 +glm-5.3,scenario_119,616.636458,partial,openrouter_id+spend_ledger,same run,gen-1788527854-Emxg3msssl6gqhXCtU1w,1788532069.451,1788527854.000 +glm-5.3,scenario_122,389.279603,partial,openrouter_id+spend_ledger,same run,gen-1788446600-cgKyESrmHUNAnL2gAMOd,1788533524.876,1788446600.000 +gpt-5.6-luna,scenario_000,0.004324,full,openai_id,before the board rows,resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDpOb25lO3Jlc3BvbnNlX2lkOnJlc3BfMDQ3OGVhN2ZkMDAyNjVhMDAwNmE0ZmVkMWRkNTY4ODE5ZWE3OGE5M2FhZmRmMWJlN2Q=,1783623249.474,1783622941.000 +gpt-5.6-luna,scenario_001,0.005954,full,openai_id,before the board rows,resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDpOb25lO3Jlc3BvbnNlX2lkOnJlc3BfMDJjMDJiZDY3NWU5MWJjNjAwNmE0ZmVkMmE0YTcwODFhMWJlYmZkYjA4NjUwMzI3NDE=,1783623249.469,1783622954.000 +gpt-5.6-sol,scenario_000,0.006776,full,openai_id,before the board rows,resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDpOb25lO3Jlc3BvbnNlX2lkOnJlc3BfMDhhMzE1ODVmNDJmZWUzMDAwNmE0ZmVjYzM4NTk4ODE5MTgyNWUzM2EwZmY4MDExZDA=,1783623303.475,1783622851.000 +gpt-5.6-sol,scenario_001,0.007025,full,openai_id,before the board rows,resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDpOb25lO3Jlc3BvbnNlX2lkOnJlc3BfMGUxYmY2N2JjNzM4MGY1NzAwNmE0ZmVjZGVjNjU4ODE5ZDg3OWNlYTAzMjIwZjhmMzU=,1783623303.479,1783622878.000 +gpt-5.6-terra,scenario_000,0.006326,full,openai_id,before the board rows,resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDpOb25lO3Jlc3BvbnNlX2lkOnJlc3BfMGE2ODVhMDk0NmQ1NTE1NDAwNmE0ZmVjZjk2MWE4ODE5Mjg5YTBjN2U1NzQxM2RkYzc=,1783623303.487,1783622905.000 +gpt-5.6-terra,scenario_001,0.006229,full,openai_id,before the board rows,resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDpOb25lO3Jlc3BvbnNlX2lkOnJlc3BfMDEyYzY4OWMyMjQ0YTI4ZDAwNmE0ZmVkMGJlZWZjODE5Zjg2NmMwMTBlMWU2Mzg5OTE=,1783623303.477,1783622923.000 +grok-4.3,scenario_000,0.021931,untraced,,untraced,4a87ac1c-9f23-909f-9f8b-d652a150e386,, +kimi-k2.6,scenario_000,0.025560,full,cache+openrouter_id+smoke_file,onboarding smoke,gen-1783264675-MvBkFkbisFcUSVwUKEGM,1783420244.711,1783264675.000 +kimi-k2.6,scenario_001,0.022980,full,cache+openrouter_id+smoke_file,onboarding smoke,gen-1783265511-1BzQDC5BmQGyJzLlgkiw,1783420244.722,1783265511.000 +kimi-k2.6,scenario_002,0.020134,full,cache+openrouter_id,before the board rows,gen-1783266620-jkSxCNXhpwQX7nDLk4BK,1783420244.733,1783266620.000 +kimi-k2.6,scenario_003,0.020995,full,cache+openrouter_id,before the board rows,gen-1783267399-0lvWvDEXPVuWJ16HCkDt,1783420244.720,1783267399.000 +kimi-k2.6,scenario_004,0.019908,full,cache+openrouter_id,before the board rows,gen-1783268478-2A4MEh5EGJN6dACylNNS,1783420257.089,1783268478.000 +kimi-k2.6,scenario_005,0.030770,full,cache+openrouter_id,before the board rows,gen-1783268963-8QkfMfC6O7sMl3Limylo,1783420257.110,1783268963.000 +kimi-k2.6,scenario_007,0.031346,full,cache+openrouter_id,before the board rows,gen-1783271165-hX6BAmHVaFOBEgYBhN2P,1783420257.107,1783271165.000 +kimi-k2.6,scenario_008,0.067598,full,cache+openrouter_id,before the board rows,gen-1783272193-tgrp8KZffPV9vnXUEBnc,1783420257.093,1783272193.000 +kimi-k2.6,scenario_009,0.036505,full,cache+openrouter_id,before the board rows,gen-1783299623-eSJlBQJZNfVWcwKLMz88,1783420269.342,1783299623.000 +kimi-k2.6,scenario_012,0.047160,full,cache+openrouter_id,before the board rows,gen-1783300224-GPUwL5PUemor9fhyMDk3,1783420269.345,1783300224.000 +kimi-k2.6,scenario_013,0.023155,full,cache+openrouter_id,before the board rows,gen-1783301511-C9CR4I2NgE6Avdym8Ka4,1783420269.348,1783301511.000 +kimi-k2.6,scenario_014,0.036954,full,cache+openrouter_id,before the board rows,gen-1783362292-yupk4l3nmTvH2q949EqA,1783420269.342,1783362292.000 +kimi-k2.6,scenario_015,0.032797,full,cache+openrouter_id,before the board rows,gen-1783362896-CvVE77Y0kcpcq7ls9mhg,1783420282.559,1783362896.000 +kimi-k2.6,scenario_016,0.025906,full,cache+openrouter_id,before the board rows,gen-1783364817-0AYQSSQoYEYwj2ZkSuT7,1783420282.557,1783364817.000 +kimi-k2.6,scenario_018,0.030026,full,cache+openrouter_id,before the board rows,gen-1783365746-g0EGwxWi24XYDtwjkkMC,1783420282.559,1783365746.000 +kimi-k2.6,scenario_020,0.026624,full,cache+openrouter_id,before the board rows,gen-1783367534-LyfIwQjJNm2hrc0KYe2Y,1783420282.576,1783367534.000 +kimi-k2.6,scenario_021,0.036108,full,cache+openrouter_id,before the board rows,gen-1783368779-E2S1atFm3KGzYAk03zyT,1783420296.925,1783368779.000 +kimi-k2.6,scenario_022,0.037522,full,cache+openrouter_id,before the board rows,gen-1783355350-MkBJxY74vWFvuKlG7Xhp,1783420296.842,1783355350.000 +kimi-k2.6,scenario_023,0.039451,full,cache+openrouter_id,before the board rows,gen-1783357125-XbMLItNn1lOHOkXFvTlB,1783420296.844,1783357125.000 +kimi-k2.6,scenario_025,0.031495,full,cache+openrouter_id,before the board rows,gen-1783363013-hnaF3lovTZdUb7GYQAjt,1783420296.840,1783363013.000 +kimi-k2.6,scenario_026,0.056106,full,cache+openrouter_id,before the board rows,gen-1783364791-McGn6GLKUem52wjxgGAf,1783420309.768,1783364791.000 +kimi-k2.6,scenario_027,0.023750,full,cache+openrouter_id,before the board rows,gen-1783368433-8t5lHK3JUBnlp0n3kVHB,1783420309.774,1783368433.000 +kimi-k2.6,scenario_028,0.035226,full,cache+openrouter_id,before the board rows,gen-1783369848-4INgAbdDIuSemNHK4vXC,1783420309.768,1783369848.000 +kimi-k2.6,scenario_029,0.021021,full,cache+openrouter_id,before the board rows,gen-1783372098-TqKLVhyHVjQ5SeMVjsjt,1783420309.764,1783372098.000 +kimi-k2.6,scenario_030,0.028253,full,cache+openrouter_id,before the board rows,gen-1783387689-VlK6VB2iHZZgUunthzOL,1783420321.958,1783387689.000 +kimi-k2.6,scenario_031,0.018525,full,cache+openrouter_id,before the board rows,gen-1783266620-zn2a2MqqklxTWmVgi340,1783420321.956,1783266620.000 +kimi-k2.6,scenario_032,0.047279,full,cache+openrouter_id,before the board rows,gen-1783299409-kcjTZm7JVZ4XSH4kZW0R,1783420321.959,1783299409.000 +kimi-k2.6,scenario_033,0.023807,full,cache+openrouter_id,before the board rows,gen-1783388519-XpvVogYeMWers7WPzmqo,1783420321.958,1783388519.000 +kimi-k2.6,scenario_036,0.028435,full,cache+openrouter_id,before the board rows,gen-1783389273-Ylxv6Vaisri1QSSU7HtC,1783420455.259,1783389273.000 +kimi-k2.6,scenario_037,0.027188,full,cache+openrouter_id,before the board rows,gen-1783397356-uKARp2OvQialfu8XLeZj,1783420455.267,1783397356.000 +kimi-k2.6,scenario_038,0.045205,full,cache+openrouter_id,before the board rows,gen-1783403970-BnILa1kDEZNNSJTJEmcl,1783420455.259,1783403970.000 +kimi-k2.6,scenario_039,414.221534,partial,cache+openrouter_id,before the board rows,gen-1783407828-uPMHyKrXftW2GsWL7lMa,1783420455.262,1783407828.000 +kimi-k2.6,scenario_043,0.031529,full,cache+openrouter_id,before the board rows,gen-1783355350-wVYUwd3KlIwr62TNmEK9,1783420467.310,1783355350.000 +kimi-k2.6,scenario_044,0.026887,full,cache+openrouter_id,before the board rows,gen-1783356942-3K8MsVXqTCoxVeq0piNH,1783420476.395,1783356942.000 +kimi-k2.6,scenario_045,0.033757,full,cache+openrouter_id,before the board rows,gen-1783358186-OofNjEsOgja4q1FW2bAw,1783420486.332,1783358186.000 +kimi-k2.6,scenario_046,0.048324,full,cache+openrouter_id,before the board rows,gen-1783362292-1heJXaEJrIcaK3vkMHY6,1783420496.348,1783362292.000 +kimi-k2.6,scenario_048,0.023369,full,cache+openrouter_id,before the board rows,gen-1783363856-XQDv8RBTnfsjAz6Wpzxn,1783420506.712,1783363856.000 +kimi-k2.6,scenario_049,0.027492,full,cache+openrouter_id,before the board rows,gen-1783364291-9MscP3LsthSeEuqHe7s5,1783420516.718,1783364291.000 +kimi-k2.6,scenario_051,0.030375,full,cache+openrouter_id,before the board rows,gen-1783365389-QMJYe1zarXx0s5GDxxIm,1783420526.779,1783365389.000 +kimi-k2.6,scenario_052,0.029431,full,cache+openrouter_id,before the board rows,gen-1783366643-WEohM4ToBwZRv6jk4T74,1783420536.891,1783366643.000 +kimi-k2.6,scenario_053,0.024262,full,cache+openrouter_id,before the board rows,gen-1783367653-BjdCYhXUnAxgRX8z594w,1783420547.236,1783367653.000 +kimi-k2.6,scenario_054,0.037881,full,cache+openrouter_id,before the board rows,gen-1783369166-CzuxJ6WPLTfQysIPvOiw,1783420557.056,1783369166.000 +kimi-k2.6,scenario_055,0.019701,full,cache+openrouter_id,before the board rows,gen-1783371681-4L2RrcRDsL1gpNW55aPS,1783420566.870,1783371681.000 +kimi-k2.6,scenario_056,0.043097,full,cache+openrouter_id,before the board rows,gen-1783387959-cHoq0FsqgRA7uYEuSXfG,1783420576.901,1783387959.000 +kimi-k2.6,scenario_057,0.031445,full,cache+openrouter_id,before the board rows,gen-1783390372-mb7POQAXCoYPVLEHKHev,1783420586.812,1783390372.000 +kimi-k2.6,scenario_059,0.019348,full,cache+openrouter_id,before the board rows,gen-1783392683-WI3bfvIbtD9PfkEj3WNu,1783420596.721,1783392683.000 +kimi-k2.6,scenario_060,0.016183,full,cache+openrouter_id,before the board rows,gen-1783394046-tZ4skudh5inYY0CJwkaE,1783420606.849,1783394046.000 +kimi-k2.6,scenario_062,0.029063,full,cache+openrouter_id,before the board rows,gen-1783395526-hWQ8K7Lwt9zDzlsRTHDv,1783420616.797,1783395526.000 +kimi-k2.6,scenario_064,0.061694,full,cache+openrouter_id,before the board rows,gen-1783266620-06JXdv6urXLyaOi10cc2,1783420626.914,1783266620.000 +kimi-k2.6,scenario_066,0.021828,full,cache+openrouter_id,before the board rows,gen-1783299784-bAYcCcoQBPsKybwhzUWT,1783420636.966,1783299784.000 +kimi-k2.6,scenario_067,0.031418,full,cache+openrouter_id,before the board rows,gen-1783300618-Pt5iY9R6THgh171NHjKv,1783420646.901,1783300618.000 +kimi-k2.6,scenario_068,0.022989,full,cache+openrouter_id,before the board rows,gen-1783387176-i2JnoVlhpl9BKRQvs3tl,1783420656.971,1783387176.000 +kimi-k2.6,scenario_070,0.023721,full,cache+openrouter_id,before the board rows,gen-1783388718-UvkaYcPiOLsiUggnhzzu,1783420666.984,1783388718.000 +kimi-k2.6,scenario_071,0.020934,full,cache+openrouter_id,before the board rows,gen-1783396400-tjo1NPH4Xx49m4IvyV0V,1783420676.919,1783396400.000 +kimi-k2.6,scenario_072,0.024056,full,cache+openrouter_id,before the board rows,gen-1783397131-vmKGu6jqz92c0GHjI1Td,1783420687.177,1783397131.000 +kimi-k2.6,scenario_073,0.021378,full,cache+openrouter_id,before the board rows,gen-1783398699-it8NaXfXvd3vnX9yVdmF,1783420697.243,1783398699.000 +kimi-k2.6,scenario_074,0.029813,full,cache+openrouter_id,before the board rows,gen-1783399020-igoJsO6G8fmVZIbT1qeB,1783420707.170,1783399020.000 +kimi-k2.6,scenario_075,0.023099,full,cache+openrouter_id,before the board rows,gen-1783399775-PpDxK7O9uAbITO5ocuLj,1783420717.134,1783399775.000 +kimi-k2.6,scenario_076,0.036734,full,cache+openrouter_id,before the board rows,gen-1783404273-OcJ55m2J45uRkvjkDMMn,1783420727.193,1783404273.000 +kimi-k2.6,scenario_077,0.037854,full,cache+openrouter_id,before the board rows,gen-1783404763-cSYJjqxkdkuGE3SqIEg4,1783420737.325,1783404763.000 +kimi-k2.6,scenario_078,236.701054,partial,cache+openrouter_id,same run,gen-1783421113-OUloMQcpdEMLMoimeUDm,1783432001.450,1783421113.000 +kimi-k2.6,scenario_083,0.022446,full,cache+openrouter_id,before the board rows,gen-1783355350-P70VIc2gK4rj7XwO3GLE,1783423066.514,1783355350.000 +kimi-k2.6,scenario_084,0.023209,full,cache+openrouter_id,before the board rows,gen-1783356457-7sr7d87oOGyQso7wY1fh,1783423077.727,1783356457.000 +kimi-k2.6,scenario_085,0.024019,full,cache+openrouter_id,before the board rows,gen-1783358252-IveYSn7P4Uvb0GS7JLuK,1783423089.799,1783358252.000 +kimi-k2.6,scenario_086,0.040194,full,cache+openrouter_id,before the board rows,gen-1783359454-YsTX5gIFRb1yJumxlK8D,1783423099.744,1783359454.000 +kimi-k2.6,scenario_088,0.026725,full,cache+openrouter_id,before the board rows,gen-1783387443-K0Pxvh2OWOZLwUQvxcbL,1783423110.443,1783387443.000 +kimi-k2.6,scenario_089,0.034123,full,cache+openrouter_id,before the board rows,gen-1783388646-ChTE56zHNkyy5TDTp8jw,1783423121.315,1783388646.000 +kimi-k2.6,scenario_090,0.019432,full,cache+openrouter_id,before the board rows,gen-1783390934-eHovPWjELurt3tmSkvbY,1783423131.318,1783390934.000 +kimi-k2.6,scenario_091,0.032730,full,cache+openrouter_id,before the board rows,gen-1783396503-Ao1OayHmvfmRc8BjA0x5,1783423141.122,1783396503.000 +kimi-k2.6,scenario_092,0.029124,full,cache+openrouter_id,before the board rows,gen-1783266620-3ziZe5vQVmJXy2b1W9le,1783423151.088,1783266620.000 +kimi-k2.6,scenario_093,0.027950,full,cache+openrouter_id,before the board rows,gen-1783299409-nViiqzgc40F5W7xvIZk3,1783423161.109,1783299409.000 +kimi-k2.6,scenario_095,0.028090,full,cache+openrouter_id,before the board rows,gen-1783300266-4YWBfWhmqrwQbJgrrkF5,1783423171.313,1783300266.000 +kimi-k2.6,scenario_098,0.020831,full,cache+openrouter_id,before the board rows,gen-1783300802-hiHJAcilyIvlnxuz1wP0,1783423175.395,1783300802.000 +kimi-k2.6,scenario_099,0.044692,full,cache+openrouter_id,before the board rows,gen-1783398037-GWXnGXORP9FcoerMFGYg,1783423181.431,1783398037.000 +kimi-k2.6,scenario_100,0.044659,full,cache+openrouter_id,before the board rows,gen-1783404571-dsVPl6Vla5lrjQnaKSL5,1783423185.501,1783404571.000 +kimi-k2.6,scenario_107,0.033450,full,cache+openrouter_id,before the board rows,gen-1783355350-2YwODSzJOJBoA4GjLeyT,1783423908.758,1783355350.000 +kimi-k2.6,scenario_108,0.039933,full,cache+openrouter_id,before the board rows,gen-1783356685-Kn6JyttSXOAJuTnqS3fc,1783423921.083,1783356685.000 +kimi-k2.6,scenario_109,776.957269,partial,cache+openrouter_id,before the board rows,gen-1783358519-0nhzWBSTP5yVevGnCHWn,1783438481.948,1783358519.000 +kimi-k2.6,scenario_123,279.558510,partial,cache+openrouter_id,same run,gen-1783430400-6DcG4kkUriXTUnw43sNo,1783432713.641,1783430400.000 +kimi-k3,scenario_008,1591.501587,partial,cache+openrouter_id,same run,gen-1784382637-oyVCDk8yDTL7Df16gsc4,1784426539.404,1784382637.000 +kimi-k3,scenario_026,693.837113,partial,cache+openrouter_id,same run,gen-1784384047-3E45FwuF1bag4PUOEZNH,1784426539.401,1784384047.000 +minimax-m3,scenario_000,0.014041,full,cache+openrouter_id,before the board rows,gen-1783264298-9iQ10dYE9ySSbhPXFHuq,1783276494.117,1783264298.000 +minimax-m3,scenario_001,0.001252,full,cache+openrouter_id,before the board rows,gen-1783264305-fZUCVHiqrAEKj0tWQvsB,1783276494.132,1783264305.000 +minimax-m3,scenario_002,0.001052,full,cache+openrouter_id,before the board rows,gen-1783264313-4eTNra94ubxqvIfr7BdD,1783276494.133,1783264313.000 +minimax-m3,scenario_003,0.001275,full,cache+openrouter_id,before the board rows,gen-1783264325-Brw7uwILGQLGjSqSzBDU,1783276494.135,1783264325.000 +minimax-m3,scenario_004,0.001078,full,cache+openrouter_id,before the board rows,gen-1783264330-II0NZnN7LDc1zx2qhyUE,1783276494.136,1783264330.000 +minimax-m3,scenario_005,0.004243,full,cache+openrouter_id,before the board rows,gen-1783264341-xwX8ADdMN1vnme4E2f96,1783276494.138,1783264341.000 +minimax-m3,scenario_007,0.000990,full,cache+openrouter_id,before the board rows,gen-1783264355-xs03DPfLvEgYbnU5m34g,1783276494.143,1783264355.000 +minimax-m3,scenario_008,0.002376,full,cache+openrouter_id,before the board rows,gen-1783264362-oPNwqutlfVJHMEiWzoiq,1783276494.145,1783264362.000 +minimax-m3,scenario_009,0.001820,full,cache+openrouter_id,before the board rows,gen-1783264628-diJqtdgjjZJs1Yb8CcLS,1783276494.148,1783264628.000 +minimax-m3,scenario_012,0.001590,full,cache+openrouter_id,before the board rows,gen-1783264781-g9gBXVuoLlBIID4vC1yM,1783276494.150,1783264781.000 +minimax-m3,scenario_013,0.001412,full,cache+openrouter_id,before the board rows,gen-1783264860-nCOXnMppPB5wNbruELOM,1783276494.152,1783264860.000 +minimax-m3,scenario_014,0.000728,full,cache+openrouter_id,before the board rows,gen-1783264870-cBhmqfQNJ1F2XBYRTUd4,1783276494.154,1783264870.000 +minimax-m3,scenario_015,0.000689,full,cache+openrouter_id,before the board rows,gen-1783264883-OQzrFUHtLtid6JgXtPMR,1783276494.155,1783264883.000 +minimax-m3,scenario_016,0.000713,full,cache+openrouter_id,before the board rows,gen-1783264893-oq3eUcaRMtk7W0XxYat0,1783276494.156,1783264893.000 +minimax-m3,scenario_018,0.000704,full,cache+openrouter_id,before the board rows,gen-1783264905-OoKwsbJYdm3aVZGhu1gn,1783276494.157,1783264905.000 +minimax-m3,scenario_020,0.000783,full,cache+openrouter_id,before the board rows,gen-1783264910-yR0yOceot127VZuwneCd,1783276494.158,1783264910.000 +minimax-m3,scenario_021,0.000587,full,cache+openrouter_id,before the board rows,gen-1783264913-YiH6xq5Gra73oYTZleBO,1783276494.159,1783264913.000 +minimax-m3,scenario_022,0.000976,full,cache+openrouter_id,before the board rows,gen-1783264922-hxF931ATMcqMPCp30eo3,1783276494.160,1783264922.000 +minimax-m3,scenario_023,0.002535,full,cache+openrouter_id,before the board rows,gen-1783264929-v3bK3u3swxM6qYMMpgAJ,1783276494.161,1783264929.000 +minimax-m3,scenario_025,0.000733,full,cache+openrouter_id,before the board rows,gen-1783265126-KbnxrJaUFXGaeTQokDpf,1783276494.164,1783265126.000 +minimax-m3,scenario_026,15.547861,partial,cache+openrouter_id,before the board rows,gen-1783265136-hPy76ijxLRfWcnjxyb4P,1783276494.165,1783265136.000 +minimax-m3,scenario_064,0.015252,full,cache+openrouter_id,before the board rows,gen-1783264298-blf0WsiZOgsExfUmXTeW,1783276494.118,1783264298.000 +minimax-m3,scenario_066,0.001026,full,cache+openrouter_id,before the board rows,gen-1783264309-cHoPX5zcLrrDVACVBRTc,1783276494.134,1783264309.000 +minimax-m3,scenario_067,0.002441,full,cache+openrouter_id,before the board rows,gen-1783264315-d2qBotEM2cvL3WIFyb3K,1783276494.135,1783264315.000 +minimax-m3,scenario_068,0.002468,full,cache+openrouter_id,before the board rows,gen-1783264563-KJuskqm5bKLYiWkUjwB0,1783276494.138,1783264563.000 +minimax-m3,scenario_070,0.001029,full,cache+openrouter_id,before the board rows,gen-1783264720-tLcUSD4a86uSiMG5Cs6R,1783276494.141,1783264720.000 +minimax-m3,scenario_071,0.000836,full,cache+openrouter_id,before the board rows,gen-1783264728-OLjqoe1FQsVqUnBrAgcE,1783276494.142,1783264728.000 +minimax-m3,scenario_072,0.000832,full,cache+openrouter_id,before the board rows,gen-1783264734-vvIzLw8SEBCynOblFfVG,1783276494.143,1783264734.000 +minimax-m3,scenario_073,0.001550,full,cache+openrouter_id,before the board rows,gen-1783264746-p4OfKzCRa6VIiEDJh3Bv,1783276494.144,1783264746.000 +minimax-m3,scenario_074,0.000769,full,cache+openrouter_id,before the board rows,gen-1783264757-ulV63c3CWtxwkhylGD28,1783276494.146,1783264757.000 +minimax-m3,scenario_075,0.001613,full,cache+openrouter_id,before the board rows,gen-1783264765-38VvKOCnsROTRGpJ15V8,1783276494.148,1783264765.000 +minimax-m3,scenario_076,0.002518,full,cache+openrouter_id,before the board rows,gen-1783264861-5fYy30PJR8GKSMldc2K3,1783276494.150,1783264861.000 +minimax-m3,scenario_077,0.000735,full,cache+openrouter_id,before the board rows,gen-1783265117-7j33SN52q934PWGeiCX5,1783276494.153,1783265117.000 +minimax-m3,scenario_078,0.000681,full,cache+openrouter_id,before the board rows,gen-1783265125-RiAtULt1i3bEoZsb9Wze,1783276494.154,1783265125.000 +minimax-m3,scenario_079,0.000703,full,cache+openrouter_id,before the board rows,gen-1783265128-8NCDtaC1Jm2NBpUgdOcH,1783276494.154,1783265128.000 +minimax-m3,scenario_080,0.000694,full,cache+openrouter_id,before the board rows,gen-1783265137-kh7lGmbiE471wNFVEPfk,1783276494.155,1783265137.000 +minimax-m3,scenario_081,0.000482,full,cache+openrouter_id,before the board rows,gen-1783265147-SssVfKp0S3JjBZoCmt5S,1783276494.156,1783265147.000 +minimax-m3,scenario_082,0.001966,full,cache+openrouter_id,before the board rows,gen-1783265149-xQok5jaOvQVPkUstFBq8,1783276494.157,1783265149.000 +minimax-m3,scenario_083,0.000991,full,cache+openrouter_id,before the board rows,gen-1783265165-l3tczzxzDxDzjZmucKy9,1783276494.160,1783265165.000 +minimax-m3,scenario_084,0.000547,full,cache+openrouter_id,before the board rows,gen-1783265176-zBTk8o1Gyhhn4VzqHiwP,1783276494.161,1783265176.000 +minimax-m3,scenario_085,0.000940,full,cache+openrouter_id,before the board rows,gen-1783265184-FoJAvNiwLCSbIICf1BXU,1783276494.162,1783265184.000 +minimax-m3,scenario_086,0.001911,full,cache+openrouter_id,before the board rows,gen-1783265193-EtxmU31I1ETRLF7v70zW,1783276494.163,1783265193.000 +minimax-m3,scenario_088,0.000687,full,cache+openrouter_id,before the board rows,gen-1783265428-HHJ1Mk0Vwhhjr1RalLz3,1783276494.165,1783265428.000 +minimax-m3,scenario_089,0.001477,full,cache+openrouter_id,before the board rows,gen-1783265890-OfpppF9i0wJWZ2AsnU3V,1783276494.166,1783265890.000 +minimax-m3,scenario_090,0.000507,full,cache+openrouter_id,before the board rows,gen-1783266101-qzTPneeMSmphEjTxYH43,1783276494.168,1783266101.000 +minimax-m3,scenario_091,0.000852,full,cache+openrouter_id,before the board rows,gen-1783266113-WHcB7JtAn6p3hhzaIdYW,1783276494.169,1783266113.000 +minimax-m3,scenario_092,0.000783,full,cache+openrouter_id,before the board rows,gen-1783266119-oxOQf9soJEHcgQTyPKFy,1783276494.187,1783266119.000 +minimax-m3,scenario_093,0.000669,full,cache+openrouter_id,before the board rows,gen-1783266582-PuPeHRZ1lfIZ0OHCojFt,1783276494.189,1783266582.000 +minimax-m3,scenario_095,0.002148,full,cache+openrouter_id,before the board rows,gen-1783266591-26MwyBg9fxF9qHCS7w50,1783276494.190,1783266591.000 +minimax-m3,scenario_098,0.000572,full,cache+openrouter_id,before the board rows,gen-1783266825-MDSZrcB2ivWsIpGNrNuZ,1783276494.192,1783266825.000 +minimax-m3,scenario_099,0.000622,full,cache+openrouter_id,before the board rows,gen-1783266832-7O292eRUvYPNRaYBAo4n,1783276494.193,1783266832.000 +minimax-m3,scenario_100,0.000629,full,cache+openrouter_id,before the board rows,gen-1783266842-W4TeWKiyf6M0coeXN33V,1783276494.194,1783266842.000 +minimax-m3,scenario_101,0.002176,full,cache+openrouter_id,before the board rows,gen-1783266857-xENsYU7AEs49WC3EZKuj,1783276494.195,1783266857.000 +minimax-m3,scenario_102,0.001282,full,cache+openrouter_id,before the board rows,gen-1783266873-T0P09NrEJkNxDKrQeSeS,1783276494.198,1783266873.000 +muse-spark-1.3,scenario_000,0.021963,full,openrouter_id+smoke_file+smoke_ledger+spend_ledger,onboarding smoke,gen-1791913993-DpxohcSu78LDiLgPpcLx,1791914304.295,1791913993.000 +muse-spark-1.3,scenario_001,0.020273,full,openrouter_id+smoke_file+smoke_ledger+spend_ledger,onboarding smoke,gen-1791913993-5zJ3lqf7ETE5I6JRbs1T,1791914304.295,1791913993.000 +muse-spark-1.3,scenario_099,0.012316,full,openrouter_id+smoke_file+smoke_ledger+spend_ledger,onboarding smoke,gen-1791913993-PQn6iUoqkvByarTtbdyC,1791915179.386,1791913993.000 +muse-spark-1.3,scenario_119,0.010278,full,openrouter_id+smoke_file+smoke_ledger+spend_ledger,onboarding smoke,gen-1791913993-Hwjd8TvOupjGQoEWJMUH,1791915312.922,1791913993.000 +muse-wonder,scenario_000,0.010042,full,openrouter_id+smoke_file+smoke_ledger+spend_ledger,onboarding smoke,gen-1791859722-dbN2jdViiyGxcdRzn0eG,1791859801.894,1791859722.000 +muse-wonder,scenario_001,0.009422,full,openrouter_id+smoke_file+smoke_ledger+spend_ledger,onboarding smoke,gen-1791859722-nZ1hgpzJG024bIN2YP6L,1791859801.912,1791859722.000 +ox-alpha,scenario_008,111.679407,partial,cache+openrouter_id+spend_ledger,same run,gen-1787353297-fclFQjwEeGUWqE54QgTr,1787360471.044,1787353297.000 +ox-alpha,scenario_023,351.659923,partial,cache+openrouter_id+spend_ledger,same run,gen-1787353751-tROrBhYcebBFVfaY5slH,1787360471.043,1787353751.000 +ox-alpha,scenario_038,271.122913,partial,cache+openrouter_id+spend_ledger,same run,gen-1787365665-hVIDbJuHcN35XLjHBTr9,1787395308.191,1787365665.000 +ox-alpha,scenario_039,275.210712,partial,cache+openrouter_id+spend_ledger,same run,gen-1787370980-uTCF2Ylw6h5y2dHyBWuN,1787395308.195,1787370980.000 +qwen-3.7-max,scenario_000,0.020002,full,cache+openrouter_id,before the board rows,gen-1783333641-JiEAUm8237YHFmJUM8hN,1783381171.662,1783333641.000 +qwen-3.7-max,scenario_001,0.011061,full,cache+openrouter_id,before the board rows,gen-1783333839-EWobKn29RdTgB3goyMuK,1783381171.683,1783333839.000 +qwen-3.7-max,scenario_002,0.010231,full,cache+openrouter_id,before the board rows,gen-1783335088-83zwIyqsij6RGWvIsNwp,1783381171.696,1783335088.000 +qwen-3.7-max,scenario_003,0.007404,full,cache+openrouter_id,before the board rows,gen-1783335635-XgbKgfYbuP7Jfv20CdQd,1783381171.708,1783335635.000 +qwen-3.7-max,scenario_004,0.008426,full,cache+openrouter_id,before the board rows,gen-1783335904-6Lat06V7LDjdivbOjQzf,1783381171.716,1783335904.000 +qwen-3.7-max,scenario_005,0.009426,full,cache+openrouter_id,before the board rows,gen-1783336176-aaJrXGJ8Az8bKTEXx1VC,1783381171.726,1783336176.000 +qwen-3.7-max,scenario_007,0.009372,full,cache+openrouter_id,before the board rows,gen-1783336909-SS2xlgkmq5vnjd5j1Xtj,1783381171.737,1783336909.000 +qwen-3.7-max,scenario_008,0.022191,full,cache+openrouter_id,before the board rows,gen-1783348516-629V5MW7GTNItkGi5hrr,1783381171.747,1783348516.000 +qwen-3.7-max,scenario_009,0.007921,full,cache+openrouter_id,before the board rows,gen-1783349087-pCf1n706YpcI5v79tTs7,1783381171.773,1783349087.000 +qwen-3.7-max,scenario_012,0.014097,full,cache+openrouter_id,before the board rows,gen-1783350019-TTHw2FxbeQT8e07kN6dq,1783381171.782,1783350019.000 +qwen-3.7-max,scenario_013,0.007858,full,cache+openrouter_id,before the board rows,gen-1783350966-PnT8Ehb2FIVim6l0NMC6,1783381171.798,1783350966.000 +qwen-3.7-max,scenario_014,0.009577,full,cache+openrouter_id,before the board rows,gen-1783353600-lgoVcdniXAVRpoF7ldPs,1783381171.807,1783353600.000 +qwen-3.7-max,scenario_015,0.008274,full,cache+openrouter_id,before the board rows,gen-1783356907-Y4Dc4EmbwiW3muHRCTn0,1783381171.818,1783356907.000 +qwen-3.7-max,scenario_016,0.006466,full,cache+openrouter_id,before the board rows,gen-1783357376-0Crtw2Ewsess5MC4d8zZ,1783381171.827,1783357376.000 +qwen-3.7-max,scenario_018,0.006626,full,cache+openrouter_id,before the board rows,gen-1783357578-pxOA9NuOmbkVCVnFCGPg,1783381171.835,1783357578.000 +qwen-3.7-max,scenario_020,0.005361,full,cache+openrouter_id,before the board rows,gen-1783359343-sa2sUysnYDtTpa1qx2lN,1783381171.843,1783359343.000 +qwen-3.7-max,scenario_021,0.005762,full,cache+openrouter_id,before the board rows,gen-1783359570-HrdmhDcBIoEAF1g8wrUG,1783381171.849,1783359570.000 +qwen-3.7-max,scenario_022,0.019229,full,cache+openrouter_id,before the board rows,gen-1783353600-FyuOMlAZkhow2KmnxhBU,1783381171.664,1783353600.000 +qwen-3.7-max,scenario_023,0.011890,full,cache+openrouter_id,before the board rows,gen-1783359330-Z3eN0GATWrACWFsdpHtM,1783381171.684,1783359330.000 +qwen-3.7-max,scenario_025,0.009880,full,cache+openrouter_id,before the board rows,gen-1783362291-Xq7WqzynEK2CJGQebRgS,1783381171.697,1783362291.000 +qwen-3.7-max,scenario_026,0.019781,full,cache+openrouter_id,before the board rows,gen-1783363173-PqZdP5x0Lm6m4n3ItLTM,1783381171.708,1783363173.000 +qwen-3.7-max,scenario_027,0.009971,full,cache+openrouter_id,before the board rows,gen-1783364848-Q1vL3H0o3oIWP7NWLWis,1783381171.730,1783364848.000 +qwen-3.7-max,scenario_028,0.017020,full,cache+openrouter_id,before the board rows,gen-1783365278-WgRwtwpxD0bV9nANqjL8,1783381171.741,1783365278.000 +qwen-3.7-max,scenario_029,0.008184,full,cache+openrouter_id,before the board rows,gen-1783366648-Cv4ouGY4pY89XgifVsOj,1783381171.760,1783366648.000 +qwen-3.7-max,scenario_030,0.010005,full,cache+openrouter_id,before the board rows,gen-1783366968-sWroWKZ8tN11tAVPKdgd,1783381171.770,1783366968.000 +qwen-3.7-max,scenario_031,0.007897,full,cache+openrouter_id,before the board rows,gen-1783333189-FWUJCrbL31VTkN2n6cXj,1783381171.781,1783333189.000 +qwen-3.7-max,scenario_032,0.016950,full,cache+openrouter_id,before the board rows,gen-1783368223-5xZMOzbLbmKdNC2cc7xE,1783381171.790,1783368223.000 +qwen-3.7-max,scenario_033,0.014286,full,cache+openrouter_id,before the board rows,gen-1783368908-8Tri7gnqDHUTM8MoIesl,1783381171.809,1783368908.000 +qwen-3.7-max,scenario_036,0.009349,full,cache+openrouter_id,before the board rows,gen-1783370445-I2CtN6zjP8dpRhKXbKfI,1783381171.825,1783370445.000 +qwen-3.7-max,scenario_037,129.838813,partial,cache+openrouter_id,before the board rows,gen-1783371326-XS92eHifygpz54E7dQ3o,1783381171.835,1783371326.000 +qwen-3.7-max,scenario_043,0.020905,full,cache+openrouter_id,before the board rows,gen-1783337764-d0sZATWxmXHLichUcBfm,1783381171.664,1783337764.000 +qwen-3.7-max,scenario_044,0.007368,full,cache+openrouter_id,before the board rows,gen-1783338346-IYrCEavUum1CRhHUwc2V,1783381171.686,1783338346.000 +qwen-3.7-max,scenario_045,0.008963,full,cache+openrouter_id,before the board rows,gen-1783339022-ubhzP6IGxTHxiESMsTBd,1783381171.694,1783339022.000 +qwen-3.7-max,scenario_046,0.011745,full,cache+openrouter_id,before the board rows,gen-1783340108-V23BYBBgCvrL795aUIzB,1783381171.704,1783340108.000 +qwen-3.7-max,scenario_048,0.007667,full,cache+openrouter_id,before the board rows,gen-1783340573-pPo1SKJ2n6x2Gd25nIoW,1783381171.718,1783340573.000 +qwen-3.7-max,scenario_049,0.007503,full,cache+openrouter_id,before the board rows,gen-1783340832-vjWe6xo9WA5nOCfHLqhm,1783381171.727,1783340832.000 +qwen-3.7-max,scenario_051,0.008564,full,cache+openrouter_id,before the board rows,gen-1783341115-B5NXWPqvIEz4AniVO4fB,1783381171.735,1783341115.000 +qwen-3.7-max,scenario_052,0.005778,full,cache+openrouter_id,before the board rows,gen-1783342291-hTV9KgQGxcSR8L1H6wHN,1783381171.745,1783342291.000 +qwen-3.7-max,scenario_053,0.008001,full,cache+openrouter_id,before the board rows,gen-1783342527-0gYfy9mMwR8s468JATYG,1783381171.752,1783342527.000 +qwen-3.7-max,scenario_054,0.014798,full,cache+openrouter_id,before the board rows,gen-1783343589-vjayFl4ZjCRjKWv7AlMn,1783381171.761,1783343589.000 +qwen-3.7-max,scenario_055,0.006931,full,cache+openrouter_id,before the board rows,gen-1783344575-PfI17Qj0CBgBLJ2TRzDJ,1783381171.777,1783344575.000 +qwen-3.7-max,scenario_056,0.009784,full,cache+openrouter_id,before the board rows,gen-1783345020-UvtCyHw6I89pAfMD0ZeQ,1783381171.786,1783345020.000 +qwen-3.7-max,scenario_057,0.012571,full,cache+openrouter_id,before the board rows,gen-1783348735-iZeeLUvBWLmkW8pTq6wX,1783381171.797,1783348735.000 +qwen-3.7-max,scenario_059,0.009834,full,cache+openrouter_id,before the board rows,gen-1783350131-2xH1tofDVu14jas7fW44,1783381171.811,1783350131.000 +qwen-3.7-max,scenario_060,0.009146,full,cache+openrouter_id,before the board rows,gen-1783350638-Rhg5iNCGCLxNjdFiMnXA,1783381171.822,1783350638.000 +qwen-3.7-max,scenario_062,0.007454,full,cache+openrouter_id,before the board rows,gen-1783353148-t5V4OI2mZO1sjS7HLr1x,1783381171.833,1783353148.000 +qwen-3.7-max,scenario_064,0.030334,full,cache+openrouter_id,before the board rows,gen-1783333189-5FTFNAtd8bbVqUJgSoMi,1783381171.662,1783333189.000 +qwen-3.7-max,scenario_066,0.010746,full,cache+openrouter_id,before the board rows,gen-1783335102-k4EXSaZqRnapjmlDEqIE,1783381171.695,1783335102.000 +qwen-3.7-max,scenario_067,0.015826,full,cache+openrouter_id,before the board rows,gen-1783336327-8gMp34PSfwluuQSMhskf,1783381171.707,1783336327.000 +qwen-3.7-max,scenario_068,0.007575,full,cache+openrouter_id,before the board rows,gen-1783341830-3PqmBpy6hlnr9aTcL00V,1783381171.725,1783341830.000 +qwen-3.7-max,scenario_070,0.007975,full,cache+openrouter_id,before the board rows,gen-1783341980-lyMvbC2iW1cvxTtQpdzT,1783381171.734,1783341980.000 +qwen-3.7-max,scenario_071,0.006763,full,cache+openrouter_id,before the board rows,gen-1783342538-avU2Hzb65hk91Ea8dBIv,1783381171.743,1783342538.000 +qwen-3.7-max,scenario_072,0.008540,full,cache+openrouter_id,before the board rows,gen-1783342901-dC6sQh6FSylnz13MkvqR,1783381171.750,1783342901.000 +qwen-3.7-max,scenario_073,0.008586,full,cache+openrouter_id,before the board rows,gen-1783343414-8GwVaI6SLUw27jVgAgny,1783381171.760,1783343414.000 +qwen-3.7-max,scenario_074,0.007795,full,cache+openrouter_id,before the board rows,gen-1783343893-w1TDyAJTpdlSfmx6mapH,1783381171.770,1783343893.000 +qwen-3.7-max,scenario_075,0.005245,full,cache+openrouter_id,before the board rows,gen-1783344229-62E95lH6FfwNnlnIIkl9,1783381171.778,1783344229.000 +qwen-3.7-max,scenario_076,0.012421,full,cache+openrouter_id,before the board rows,gen-1783344928-z4cAmIP9onGpNnttGvto,1783381171.785,1783344928.000 +qwen-3.7-max,scenario_077,0.013032,full,cache+openrouter_id,before the board rows,gen-1783346525-0A1znkWkDDifABJr3Tdi,1783381171.799,1783346525.000 +qwen-3.7-max,scenario_078,0.007941,full,cache+openrouter_id,before the board rows,gen-1783353149-OgJxCuiUEgboC5nKMlxP,1783381171.814,1783353149.000 +qwen-3.7-max,scenario_079,0.008331,full,cache+openrouter_id,before the board rows,gen-1783353479-RWoC9dvfFrDJSv2lBZM2,1783381171.823,1783353479.000 +qwen-3.7-max,scenario_080,0.008497,full,cache+openrouter_id,before the board rows,gen-1783353894-QYaygV8oQt8mzkiYNEOo,1783381171.832,1783353894.000 +qwen-3.7-max,scenario_081,0.006726,full,cache+openrouter_id,before the board rows,gen-1783354533-JVv7KwbPvDC7TxyVrTqi,1783381171.842,1783354533.000 +qwen-3.7-max,scenario_082,0.007264,full,cache+openrouter_id,before the board rows,gen-1783356375-zVeCuaVX93Z8VqqqX7kI,1783381171.850,1783356375.000 +qwen-3.7-max,scenario_083,0.018920,full,cache+openrouter_id,before the board rows,gen-1783337312-dU2wng53YTRLXUsjK5XN,1783381171.664,1783337312.000 +qwen-3.7-max,scenario_084,0.008864,full,cache+openrouter_id,before the board rows,gen-1783337643-e28jNbvjlj6PXyam4bga,1783381171.684,1783337643.000 +qwen-3.7-max,scenario_085,0.007309,full,cache+openrouter_id,before the board rows,gen-1783338166-erSnRqaiFxizZMkQhY00,1783381171.694,1783338166.000 +qwen-3.7-max,scenario_086,0.011688,full,cache+openrouter_id,before the board rows,gen-1783356827-dwoIinxC5irNynlohtOB,1783381171.702,1783356827.000 +qwen-3.7-max,scenario_088,0.006498,full,cache+openrouter_id,before the board rows,gen-1783357253-MuDtGMeR1cLzWvnhZdGH,1783381171.716,1783357253.000 +qwen-3.7-max,scenario_089,0.008486,full,cache+openrouter_id,before the board rows,gen-1783363194-XnAH4cNBO8GyS6QkN4z8,1783381171.723,1783363194.000 +qwen-3.7-max,scenario_090,0.010636,full,cache+openrouter_id,before the board rows,gen-1783363293-gI3BKPC0wyFGSi7qqX7U,1783381171.733,1783363293.000 +qwen-3.7-max,scenario_091,0.008203,full,cache+openrouter_id,before the board rows,gen-1783364169-oSVbVilFU1k8oDTYdImU,1783381171.745,1783364169.000 +qwen-3.7-max,scenario_092,0.006172,full,cache+openrouter_id,before the board rows,gen-1783333189-EmFPfX0kS90tLowl9XYz,1783381171.754,1783333189.000 +qwen-3.7-max,scenario_093,0.009682,full,cache+openrouter_id,before the board rows,gen-1783333946-zWBDA42vUuUzSIBj8FWI,1783381171.761,1783333946.000 +qwen-3.7-max,scenario_095,0.008939,full,cache+openrouter_id,before the board rows,gen-1783334968-LSTgC6PjF1KqcvLSrBbY,1783381171.772,1783334968.000 +qwen-3.7-max,scenario_098,0.007741,full,cache+openrouter_id,before the board rows,gen-1783335618-mybG2kwjoUpYIemZfXtq,1783381171.783,1783335618.000 +qwen-3.7-max,scenario_099,0.014834,full,cache+openrouter_id,before the board rows,gen-1783335936-9toDxItkDiGt9yb54hrn,1783381171.792,1783335936.000 +qwen-3.7-max,scenario_100,0.021292,full,cache+openrouter_id,before the board rows,gen-1783364647-yc9ODjVDuGzB6ll5UgnU,1783381171.809,1783364647.000 +qwen-3.7-max,scenario_101,0.008238,full,cache+openrouter_id,before the board rows,gen-1783366638-5GSOMxW4Ub8odULIBiXM,1783381171.833,1783366638.000 +qwen-3.7-max,scenario_102,0.009545,full,cache+openrouter_id,before the board rows,gen-1783367171-RCWy5E7Z0DlImfDdTc50,1783381171.842,1783367171.000 +qwen-3.7-max,scenario_104,0.005236,full,cache+openrouter_id,before the board rows,gen-1783368135-wEfeEF6gh6oAlcypvsEc,1783381171.853,1783368135.000 +qwen-3.7-max,scenario_107,0.020234,full,cache+openrouter_id,before the board rows,gen-1783337312-LEjY7wjJ1JsMxo6SiJ69,1783381171.664,1783337312.000 +qwen-3.7-max,scenario_108,0.010732,full,cache+openrouter_id,before the board rows,gen-1783338219-6zje5n81g9h6ux9Zhic0,1783381171.685,1783338219.000 +qwen-3.7-max,scenario_109,0.016645,full,cache+openrouter_id,before the board rows,gen-1783339649-GOHfzCLwTlwG4dWMhAvG,1783381171.697,1783339649.000 +qwen-3.7-max,scenario_110,0.008435,full,cache+openrouter_id,before the board rows,gen-1783341830-KfruepW2NIBkhcvSVMeb,1783381171.716,1783341830.000 +qwen-3.7-max,scenario_111,0.008858,full,cache+openrouter_id,before the board rows,gen-1783341929-oPttFtkWILtf11zsWNlR,1783381171.725,1783341929.000 +qwen-3.7-max,scenario_112,0.008089,full,cache+openrouter_id,before the board rows,gen-1783342243-EO3hHHrOuPZfFoLLTxpT,1783381171.736,1783342243.000 +qwen-3.7-max,scenario_114,0.007903,full,cache+openrouter_id,before the board rows,gen-1783353148-s8oYKCb29HEejr38ZK4j,1783381171.745,1783353148.000 +qwen-3.7-max,scenario_115,0.006686,full,cache+openrouter_id,before the board rows,gen-1783353245-JsEybk2B0yx2KqDDrHBk,1783381171.754,1783353245.000 +qwen-3.7-max,scenario_116,0.005974,full,cache+openrouter_id,before the board rows,gen-1783353511-u61td3KKS1wkg6eLxnZM,1783381171.761,1783353511.000 +qwen-3.7-max,scenario_117,0.015305,full,cache+openrouter_id,before the board rows,gen-1783353662-wpmaReU8bZXXldonFU3U,1783381171.769,1783353662.000 +qwen-3.7-max,scenario_118,0.006185,full,cache+openrouter_id,before the board rows,gen-1783354527-n28tpgPPKgfDA1X0zOVs,1783381171.787,1783354527.000 +qwen-3.7-max,scenario_119,0.014331,full,cache+openrouter_id,before the board rows,gen-1783359783-hNpqBKAjtdnjgWB9vBVM,1783381171.794,1783359783.000 +qwen-3.7-max,scenario_120,0.009706,full,cache+openrouter_id,before the board rows,gen-1783362292-rkGAofEpzk8zrObJ2L57,1783381171.811,1783362292.000 +qwen-3.7-max,scenario_121,0.012724,full,cache+openrouter_id,before the board rows,gen-1783363205-tltwvED2zYu9KsWAbLUf,1783381171.822,1783363205.000 diff --git a/paper/index.qmd b/paper/index.qmd index cfd9e3e0..0ecec41e 100644 --- a/paper/index.qmd +++ b/paper/index.qmd @@ -746,12 +746,14 @@ def fmt_cost_per_household(usd) -> str: def load_us_cost_latency() -> dict: # Cost and latency use the frozen run's recorded per-call costs and timings, # through the same model_cost_latency function as the leaderboard pipeline, - # then are reconciled against the frozen - # modelStats: Claude Fable 5's usage is recorded at the run level (its - # per-row usage columns are empty), and batch or asynchronous waves log - # submission times rather than generation time, so a zero or sub-second - # median is treated as unmeasured and falls back to modelStats or "—". - from policybench.analysis import model_cost_latency + # then are reconciled against the frozen modelStats. Claude Fable 5's board + # rows carry no usage or timing fields (they were rebuilt from the 20260701 + # release without them), so its modelStats carries its original run's + # per-call figures. A zero cost is a recorded charge of zero, and a + # sub-second median is a model whose median household was replayed from the + # disk cache (docs/latency_replay_audit.md); both fall back to modelStats + # or "—". + from policybench.analysis import REPLAYED_HOUSEHOLD_SECONDS, model_cost_latency from policybench.config import PRICE_OVERRIDES_PER_1M predictions = pd.read_csv( @@ -768,11 +770,13 @@ def load_us_cost_latency() -> dict: float(fallback) if fallback and fallback > 0 else None ) latency = entry.get("latencySeconds") - if latency is None or pd.isna(latency) or latency < 1: + if latency is None or pd.isna(latency) or latency < REPLAYED_HOUSEHOLD_SECONDS: fallback = None if row is None else row.get("latencySeconds") entry["latencySeconds"] = ( float(fallback) - if fallback is not None and not pd.isna(fallback) and fallback >= 1 + if fallback is not None + and not pd.isna(fallback) + and fallback >= REPLAYED_HOUSEHOLD_SECONDS else None ) return computed @@ -1164,12 +1168,29 @@ Sampling uncertainty is substantial at this sample size. In the frozen snapshot, Accuracy is one axis; the price and speed of a no-tools answer are another. @tbl-us-cost-latency reports each model's estimated cost to compute one household's full output vector and its median per-household latency, next to the headline exact rate. `{python} cost_latency_text` ```{python} -#| label: tbl-us-cost-latency -#| tbl-cap: "Estimated cost and median latency per household for the frozen US snapshot, alongside the headline exact-match rate. Each row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge where no reconstruction was available. List-price overrides apply at request time, not retroactively to recorded costs. Models without per-call costs use the frozen release-metadata cost. Latency is the median across households of each household's summed request time. A dash marks usage or wall-clock latency that was not measured per call." -md_table(cost_latency) +def cost_latency_caption() -> str: + """The table's caption, naming the model behind each dash and why.""" + shown = list(us_model["model"]) + cost_dash = [m for m in shown if us_cost_latency.get(m, {}).get("costPerHousehold") is None] + latency_dash = [m for m in shown if us_cost_latency.get(m, {}).get("latencySeconds") is None] + # Every dash is one the caption explains, or the render fails. + assert sorted(cost_dash) == r.zero_cost_models, cost_dash + assert sorted(latency_dash) == r.latency_dash_models, latency_dash + zero_cost = r.zero_cost_models_fmt + return ( + "Estimated cost and median latency per household for the frozen US snapshot, alongside the headline exact-match rate. Each row uses its recorded per-call cost: reconstructed from token counts at the list price configured at request time, or the provider-reported charge where no reconstruction was available. List-price overrides apply at request time, not retroactively to recorded costs. Claude Fable 5's cost and latency come from its original run's per-call rows, which its board rows do not carry. Latency is the median across households of each household's summed request time. " + f"A cost dash marks a provider-reported charge of zero on every call ({zero_cost}). " + f"A latency dash marks rows that record no request time ({r.batch_latency_models_fmt}, through OpenAI's batch API) " + f"or a median household answered from the harness's response cache rather than by the provider ({r.replayed_latency_models_fmt}; see text)." + ) + + +md_table(cost_latency, caption=cost_latency_caption(), label="tbl-us-cost-latency") ``` -Cost preserves each call's recorded total: a request-time reconstruction from token counts where one was available, and the provider-reported charge otherwise. Configured list-price overrides do not reprice recorded calls. Rows without per-call costs use the frozen release-metadata cost. Latency is wall-clock request time as observed by the harness, summed within each household and taken as the median across the `{python} r.n_households_fmt` households. The per-call timer spans any provider-side retry or rate-limit backoff, so the median is reported in preference to the mean, which a few throttled calls inflate. Latency is not comparable across serving regimes: Claude-family models issue chunked serial calls that lengthen wall-clock time, several models ran through batch or asynchronous dispatch whose logged times reflect submission rather than generation (shown as a dash), and Claude Fable 5's usage and latency are recorded at the run level rather than per call. Both cost and latency are properties of this run's providers, prompts, and serving configurations at evaluation time rather than fixed model constants. +Cost preserves each call's recorded total: a request-time reconstruction from token counts where one was available, and the provider-reported charge otherwise. Configured list-price overrides do not reprice recorded calls. Claude Fable 5's board rows carry no usage or timing fields, because they were rebuilt from release dashboard-data-20260701 without those columns; its cost, tokens and latency come from the per-call rows of its original run, which that release publishes. Latency is wall-clock request time as observed by the harness, summed within each household and taken as the median across the `{python} r.n_households_fmt` households. The timer spans one request as the provider handles it, including anything that happens inside that call. When the harness retries a failed request, only the attempt that succeeded is timed, without the failed attempts or the wait between them. The median is reported in preference to the mean, which a few slow calls inflate. Latency is not comparable across serving regimes. A model served one or three outputs per request (@tbl-model-runs) sends a household's calls one after another, and its household time is their sum. `{python} r.batch_latency_models_fmt` ran through OpenAI's batch API, which records no per-call request time, so it shows a dash. + +The harness also caches provider responses on disk, keyed on the request and the prompt contract version. A request identical to an earlier one is answered from the cache in hundredths of a second, and the timer records that as if it were a call. A replayed response's cost is computed from that response when its row is written: its token counts at the configured list price, or the provider's charge embedded in it where no reconstruction exists. A household's cost therefore counts each of its responses once, whether or not the board run paid for it. `{python} r.replayed_latency_models_fmt` were finished over several passes, each of which replayed the earlier passes' successful responses. Each of them records a replay on `{python} r.replayed_latency_range_fmt` of its `{python} r.n_households_fmt` households, so its median falls under a second and it shows a dash as well. In `{python} r.other_subsecond_model_count_fmt` other models, `{python} r.other_subsecond_household_count` households sum to under a second. The audit traces `{python} r.traced_other_subsecond_count` of them to an earlier request on the same household, such as an onboarding smoke or a restarted chunk; the other `{python} r.untraced_subsecond_count_fmt` come from runs that predate per-call timestamps and left no cache or ledger behind. Dropping them would raise no model's median by more than `{python} r.subsecond_median_shift_fmt` seconds. A further `{python} r.partial_replay_count` households replay their first recorded call and then reach the provider, so their times leave out that first call. The benchmark card summarizes the replays, and the repository's latency replay audit lists each household. Both cost and latency are properties of this run's providers, prompts, and serving configurations at evaluation time rather than fixed model constants. ### Simple baselines diff --git a/paper/snapshot/20260501/manifest.json b/paper/snapshot/20260501/manifest.json index dbdc04e3..20e4975b 100644 --- a/paper/snapshot/20260501/manifest.json +++ b/paper/snapshot/20260501/manifest.json @@ -164,12 +164,12 @@ "rendered_paper_artifacts": { "pdf": { "path": "app/public/paper/policybench.pdf", - "sha256": "d0b236367fb3f30340e4a9ba3d0d25d253b33096a8a789f3d3b00ccd8b4c34ae" + "sha256": "d16eb03ce07dc57c4b8e7c3a3edb70ac8ccbaef8b830ce9cc38d9da892989481" }, "web": { "files": { "figures/positive_zero_scatter.png": "47a28fba15b58e86aa00cb52c991a09c5f393c40b9311c24001c201741610062", - "index.html": "533a0cb79b6a54353d73ace8e3ac2d2bf0ef41eff5001b4422876ad8f9a864a0", + "index.html": "6cb2fe1ad2b32c4c76984c150dbeffec2f623951e8710d038bc282feb15cd238", "pe-tokens.css": "8f24d8da26f583c8ffddffcdcd172b6d52cbecfec20eda55bd39d7aa829f41d8", "policybench-theme.css": "0e12c5fd615558259e5bce0167a38424e54f9ceb280666c4afd660d759cd1cb9", "site_libs/clipboard/clipboard.min.js": "e17a1d816e13c0826e0ed7febfabc3277f45571234bde0bf9120829a7169edc9", @@ -193,7 +193,7 @@ "The top-level scenario, reference-output, and impact-summary CSVs are byte-identical to the corresponding compact source-run artifacts copied under paper/snapshot/20260501/runs/.", "Model responses were collected in waves between June 12 and October 13, 2026, as models were added to the board; each model's full 100-household run is a single consistent wave. PolicyBench computes each scored reference output with policyengine_us.Simulation from policyengine-us 2.41.0, using the household's own listed inputs, the nine publication conventions, the Maryland output-scope adapter and the scenario builder's stated-hours alias, as the reference sidecar's engine_upgrade revision pins them (fix_modules, builder); policyengine.py 6.1.2 is recorded for provenance only. The households were drawn from the certified PolicyEngine US populace dataset (populace-us-2024-5da5a95-20260611, populace_us_2024), as scenarios.csv.meta.json records (household_dataset); the reference_output_refresh dataset fields name the reference runtime's default dataset, which reference computation does not read.", "Canonical prediction files include parser recovery. Later waves ran under the resumable supervised runner, which retries failed or timed-out scenarios in bounded rounds; every model's canonical file covers all 100 households.", - "Raw provider responses are retained in the compressed source-run predictions.csv.gz file wherever the transport exposed them; rows served through the Anthropic batch adapter (Claude Fable 5) and 64 Kimi K3 parse failures carry no raw payload. The separate LiteLLM cache remains local-only because it is a generated request cache, not the canonical snapshot artifact.", + "Raw provider responses are retained in the compressed source-run predictions.csv.gz file wherever the transport exposed them; Claude Fable 5's rows, rebuilt from release dashboard-data-20260701 without their raw payloads (that release's claude-fable-5-predictions.csv.gz asset keeps them), and 64 Kimi K3 parse failures carry none. The separate LiteLLM cache remains local-only because it is a generated request cache, not the canonical snapshot artifact.", "The frozen scenarios.csv source_dataset column carries a stale enhanced_cps_2024 label from the pre-#77 scenario generator; the run metadata (scenarios.csv.meta.json) records the populace_us_2024 build actually loaded.", "Model APIs and upstream model aliases may change after the recorded 2026-06-12 to 2026-10-13 response window, so exact reruns can diverge even with the committed household inputs, reference outputs, parsed dashboard export, and analysis summaries." ], diff --git a/policybench/analysis.py b/policybench/analysis.py index 0ae3b331..6e89f3a6 100644 --- a/policybench/analysis.py +++ b/policybench/analysis.py @@ -1411,6 +1411,32 @@ def _wall_clock_seconds(group: pd.Series) -> float: ) +# A household whose recorded request time sums to under this is a disk-cache +# replay, not a provider call: in the frozen US run every replayed household +# sums to under 0.12 s, and the fastest live one takes 1.75 s +# (docs/latency_replay_audit.md). +REPLAYED_HOUSEHOLD_SECONDS = 1.0 + + +def household_latency_seconds(predictions: pd.DataFrame) -> pd.Series: + """Each household's recorded request time, indexed by (model, scenario_id). + + A row's ``elapsed_seconds`` is its household's request time divided among + the household's rows, so the household's time is the sum over its rows, + not the max. A household whose rows carry no time is NaN rather than zero. + A response the disk cache replays records the lookup's time, a few + hundredths of a second, as if it were a provider call. + """ + if predictions.empty or "elapsed_seconds" not in predictions.columns: + return pd.Series( + dtype=float, + index=pd.MultiIndex.from_tuples([], names=["model", "scenario_id"]), + ) + return predictions.groupby(["model", "scenario_id"])["elapsed_seconds"].sum( + min_count=1 + ) + + def model_cost_latency( predictions: pd.DataFrame, price_overrides: dict[str, dict[str, float]] | None = None, @@ -1431,10 +1457,7 @@ def model_cost_latency( latency_median = pd.Series(dtype=float) if "elapsed_seconds" in predictions.columns: latency_median = ( - predictions.groupby(["model", "scenario_id"])["elapsed_seconds"] - .sum(min_count=1) - .groupby("model") - .median() + household_latency_seconds(predictions).groupby("model").median() ) overrides = price_overrides or {} out: dict[str, dict[str, float]] = {} diff --git a/policybench/paper_results.py b/policybench/paper_results.py index 1373b245..6373337a 100644 --- a/policybench/paper_results.py +++ b/policybench/paper_results.py @@ -54,6 +54,9 @@ # ``paper_results`` lives in ``policybench/``; the repo root is one level up. ROOT = Path(__file__).resolve().parents[1] SNAPSHOT_DIR = ROOT / "paper" / "snapshot" / "20260501" +# The frozen run's replayed households, traced from the disk caches, spend +# ledgers and smoke files (scripts/audit_latency_replays.py). +LATENCY_REPLAY_AUDIT = ROOT / "docs" / "latency_replay_audit" / "households.csv" # The 2026-09-29 move from policyengine-us 1.755.4 to 2.15.17. Its audit, # reference_audit/2026-09-28, holds the publication check, the sweep timing @@ -1256,9 +1259,11 @@ def contract_violation_count_fmt(self) -> str: def blank_raw_response_counts(self) -> Counter: """Frozen prediction rows whose raw_response is empty, by model. - Rows served through the Anthropic batch adapter carry no raw payload, - and some parse failures never captured one; the manuscript states the - preservation rule with these exceptions rather than as absolute. + Claude Fable 5's board rows were rebuilt from the 20260701 release + without their raw payloads (that release's + claude-fable-5-predictions.csv.gz asset keeps them), and some parse + failures never captured one; the manuscript states the preservation + rule with these exceptions rather than as absolute. """ run_dir = self.snapshot_dir / "runs" / self.us_run_label frame = pd.read_csv( @@ -1280,6 +1285,183 @@ def blank_raw_response_note(self) -> str: return "every row carries its raw provider response" return "no raw payload is retained for " + _ordinal_join(items) + # ----- latency: untimed rows and cache replays ------------------------- + @cached_property + def household_latencies(self) -> pd.Series: + """Each frozen household's summed request time, by (model, scenario).""" + from policybench.analysis import household_latency_seconds + + run_dir = self.snapshot_dir / "runs" / self.us_run_label + frame = pd.read_csv( + run_dir / "predictions.csv.gz", + usecols=["model", "scenario_id", "elapsed_seconds"], + ) + return household_latency_seconds(frame) + + def _published_latency(self, model_id: str) -> float | None: + value = self._stats_by_model.get(model_id, {}).get("latencySeconds") + return None if value is None or pd.isna(value) else float(value) + + @cached_property + def untimed_latency_models(self) -> list[str]: + """Models none of whose frozen rows records a request time.""" + timed = self.household_latencies.groupby("model").count() + return sorted(timed[timed == 0].index) + + @cached_property + def replayed_latency_models(self) -> list[str]: + """Models whose median household is a cache replay, not a call.""" + from policybench.analysis import REPLAYED_HOUSEHOLD_SECONDS + + median = self.household_latencies.groupby("model").median() + return sorted(median[median < REPLAYED_HOUSEHOLD_SECONDS].index) + + @cached_property + def latency_dash_models(self) -> list[str]: + """Models the cost-and-latency table shows without a latency. + + A model is dashed when its frozen rows give no latency of a second or + more and its published modelStats carries none either; Claude Fable + 5's rows are untimed but its published latency comes from its + original run's per-call rows, so it is not dashed. + """ + from policybench.analysis import REPLAYED_HOUSEHOLD_SECONDS + + dashed = [] + for model in sorted( + set(self.untimed_latency_models) | set(self.replayed_latency_models) + ): + published = self._published_latency(model) + if published is None or published < REPLAYED_HOUSEHOLD_SECONDS: + dashed.append(model) + return dashed + + @cached_property + def subsecond_household_counts(self) -> Counter: + """Households summing to under a second, by model.""" + from policybench.analysis import REPLAYED_HOUSEHOLD_SECONDS + + latency = self.household_latencies + fast = latency[latency < REPLAYED_HOUSEHOLD_SECONDS] + return Counter(fast.groupby("model").size().to_dict()) + + @cached_property + def zero_cost_models(self) -> list[str]: + """Models every one of whose frozen rows records a charge of zero.""" + run_dir = self.snapshot_dir / "runs" / self.us_run_label + frame = pd.read_csv( + run_dir / "predictions.csv.gz", usecols=["model", "total_cost_usd"] + ) + cost = frame.groupby("model")["total_cost_usd"] + recorded = cost.count() == cost.size() + zero = cost.apply(lambda values: bool((values == 0).all())) + return sorted(recorded[recorded & zero].index) + + @property + def zero_cost_models_fmt(self) -> str: + return _ordinal_join([self.model_name(m) for m in self.zero_cost_models]) + + @property + def batch_latency_models_fmt(self) -> str: + """Dashed models whose rows carry no request time (the batch run).""" + dashed = [ + m for m in self.latency_dash_models if m in self.untimed_latency_models + ] + return _ordinal_join([self.model_name(m) for m in dashed]) + + @property + def replayed_latency_models_fmt(self) -> str: + dashed = [ + m for m in self.latency_dash_models if m in self.replayed_latency_models + ] + return _ordinal_join([self.model_name(m) for m in dashed]) + + @property + def replayed_latency_range_fmt(self) -> str: + """Fewest to most replayed households among the dashed replay models.""" + counts = [ + self.subsecond_household_counts[m] + for m in self.latency_dash_models + if m in self.replayed_latency_models + ] + low, high = min(counts), max(counts) + return f"{low}" if low == high else f"{low} to {high}" + + @cached_property + def _other_subsecond_models(self) -> list[str]: + """Shown models with some households under a second.""" + return sorted( + m + for m, n in self.subsecond_household_counts.items() + if n and m not in self.latency_dash_models + ) + + @property + def other_subsecond_household_count(self) -> int: + return sum( + self.subsecond_household_counts[m] for m in self._other_subsecond_models + ) + + @property + def other_subsecond_model_count(self) -> int: + return len(self._other_subsecond_models) + + @property + def other_subsecond_model_count_fmt(self) -> str: + return count_word(self.other_subsecond_model_count) + + @property + def untraced_subsecond_count(self) -> int: + return self.other_subsecond_household_count - self.traced_other_subsecond_count + + @property + def untraced_subsecond_count_fmt(self) -> str: + return count_word(self.untraced_subsecond_count) + + @cached_property + def subsecond_median_shift_seconds(self) -> float: + """Largest rise in a shown model's median when its sub-second + households are dropped: how far the replays lower a published median.""" + from policybench.analysis import REPLAYED_HOUSEHOLD_SECONDS + + latency = self.household_latencies + shifts = [] + for model in self._other_subsecond_models: + households = latency.loc[model].dropna() + kept = households[households >= REPLAYED_HOUSEHOLD_SECONDS] + shifts.append(float(kept.median() - households.median())) + return max(shifts) if shifts else 0.0 + + @property + def subsecond_median_shift_fmt(self) -> str: + """The shift rounded up to a tenth, so the prose's "no more than" + bound holds for the unrounded value.""" + import math + + return f"{math.ceil(self.subsecond_median_shift_seconds * 10) / 10:.1f}" + + @cached_property + def latency_replay_audit(self) -> pd.DataFrame: + """The audited replayed households (docs/latency_replay_audit).""" + return pd.read_csv(LATENCY_REPLAY_AUDIT) + + @property + def traced_other_subsecond_count(self) -> int: + """Of the shown models' sub-second households, those the audit traced + to an earlier request on the same household.""" + audit = self.latency_replay_audit + traced = audit[ + (audit["replay"] == "full") + & audit["model"].isin(self._other_subsecond_models) + ] + return len(traced) + + @property + def partial_replay_count(self) -> int: + """Households whose first recorded call is a replay and whose later + calls reached the provider.""" + return int((self.latency_replay_audit["replay"] == "partial").sum()) + # ----- populace dataset construction --------------------------------- @property def populace_people_fmt(self) -> str: diff --git a/scripts/audit_latency_replays.py b/scripts/audit_latency_replays.py new file mode 100644 index 00000000..9adb2731 --- /dev/null +++ b/scripts/audit_latency_replays.py @@ -0,0 +1,343 @@ +"""Find the frozen board's households whose recorded latency is a cache replay. + +Writes ``docs/latency_replay_audit/households.csv``: one row for every +household of the frozen US run whose first recorded response was served again +from the harness's disk response cache (``policybench/cache.py``), and for +every household whose rows sum to under a second without such evidence. +``docs/latency_replay_audit.md`` explains the result. + +The evidence comes from artifacts that live outside the repository, so this +script runs only on the machine that holds them: + +- the litellm disk caches (``--cache-dir``, repeatable). Each entry's sqlite + ``store_time`` says when a response was first stored. Values are pickles; the + script searches their bytes for response ids and never unpickles them. +- the supervised runner's spend ledgers (``*.spend.jsonl`` under + ``--ledger-root``, repeatable), which record ``cache_hit`` per call. A ledger + under a path containing ``smoke`` belongs to an onboarding smoke. +- smoke prediction files (CSV under a path containing ``smoke``) below the + ledger roots, which keep each smoke response's id and request times. + +Each kind of evidence is tied to the frozen request it explains, the +household's first call (the one whose id its rows keep): + +- ``cache``: an entry holding the response id was stored before the request + began. Both times are this machine's clock. +- ``spend_ledger``: a ledger outside the smokes records ``cache_hit`` for that + response id on the same model and household, at the frozen request's own + start time. A later run that replayed a response the board fetched live + therefore does not count. +- ``smoke_ledger`` / ``smoke_file``: an onboarding smoke served the same + response id and finished before the request began. +- ``openrouter_id`` / ``openai_id``: the frozen predictions alone suffice. + OpenRouter ids (``gen--...``) and OpenAI Responses ids + (``resp_`` + 16 hex + ``00`` + 8 hex seconds) embed their creation time, on + the provider's clock. On live households that time falls within 23 seconds + of the row's ``request_started_at``, so a creation time more than a minute + before the request began means the response already existed. + +A household counts as replayed when any evidence holds. A full replay sums to +under a second: every call came from the cache. A partial replay sums to a +second or more, so a later call in the household (another chunk, a repair, or +a re-request with a larger completion budget) went to the provider. +""" + +from __future__ import annotations + +import argparse +import base64 +import csv +import json +import os +import re +import sqlite3 +import sys +from pathlib import Path + +import pandas as pd + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from policybench.analysis import household_latency_seconds # noqa: E402 + +FROZEN_PREDICTIONS = ( + ROOT + / "paper" + / "snapshot" + / "20260501" + / "runs" + / "us_full_run_20260612_policyengine_4_16_1_populace" + / "predictions.csv.gz" +) +OUTPUT = ROOT / "docs" / "latency_replay_audit" / "households.csv" +# A response that existed this long before the request began was not generated +# for it. On live households an id's creation time sits within 23 seconds of +# the request start (OpenRouter; OpenAI ids within 4). +MARGIN_SECONDS = 60.0 +# Every replayed household in the frozen run sums to under 0.12 s; the fastest +# live household (Gemini 3.5 Flash-Lite) takes 1.75 s. +FULL_REPLAY_SECONDS = 1.0 +# A ledger record belongs to the frozen request when it starts at the same +# moment; the board's own ledger records it to the microsecond. +SAME_REQUEST_SECONDS = 0.5 + +RESPONSE_ID = re.compile( + rb"(gen-\d{10}-[A-Za-z0-9]{10,40}" + rb"|chatcmpl-[A-Za-z0-9-]{8,80}" + rb"|msg_[A-Za-z0-9]{10,60}" + rb"|resp_[A-Za-z0-9]{20,80}" + rb"|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})" +) +OPENROUTER_ID = re.compile(r"^gen-(\d{10})-") +OPENAI_ID = re.compile(r"^resp_[0-9a-f]{16}00([0-9a-f]{8})[0-9a-f]{24}$") + +FIELDS = [ + "model", + "scenario_id", + "household_latency_seconds", + "replay", + "evidence", + "source", + "first_response_id", + "request_started_at", + "response_existed_at", +] + + +def provider_response_id(recorded: object) -> object: + """The provider's own id; litellm wraps OpenAI Responses ids in base64.""" + if not isinstance(recorded, str) or not recorded.startswith("resp_"): + return recorded + try: + decoded = base64.b64decode(recorded[5:] + "==").decode() + except (ValueError, UnicodeDecodeError): + return recorded + match = re.search(r"response_id:(resp_[0-9a-f]+)", decoded) + return match.group(1) if match else recorded + + +def id_created_at(response_id: object) -> float | None: + """Creation time embedded in an OpenRouter or OpenAI Responses id.""" + if not isinstance(response_id, str): + return None + for pattern, base in ((OPENROUTER_ID, 10), (OPENAI_ID, 16)): + match = pattern.match(response_id) + if match: + return float(int(match.group(1), base)) + return None + + +def cache_store_times(cache_dirs: list[Path]) -> dict[str, float]: + """Earliest store time of a cache entry holding each response id.""" + stored: dict[str, float] = {} + for cache_dir in cache_dirs: + con = sqlite3.connect(f"file:{cache_dir / 'cache.db'}?mode=ro", uri=True) + try: + rows = con.execute("select store_time, filename, value from Cache") + for store_time, filename, value in rows: + if filename: + blob = (cache_dir / filename).read_bytes() + else: + blob = bytes(value or b"") + for match in set(RESPONSE_ID.findall(blob)): + key = match.decode() + if key not in stored or store_time < stored[key]: + stored[key] = float(store_time) + finally: + con.close() + return stored + + +# Trees that hold no run artifacts, skipped so a walk over a large review +# folder stays quick. +SKIP_DIRS = frozenset( + {".git", ".venv", "venv", "node_modules", "site-packages", "__pycache__"} +) + + +def walk_files(root: Path, suffix: str): + """Files under ``root`` ending in ``suffix``, skipping SKIP_DIRS.""" + for directory, subdirs, names in os.walk(root): + subdirs[:] = [name for name in subdirs if name not in SKIP_DIRS] + for name in names: + if name.endswith(suffix): + yield Path(directory) / name + + +def _under_smoke(path: Path, root: Path) -> bool: + """Whether a file sits in a smoke run, judged below its root so a root + whose own path mentions a smoke does not make every file a smoke's.""" + return "smoke" in str(path.relative_to(root)) + + +def ledger_records(roots: list[Path]) -> pd.DataFrame: + records = [] + for root in roots: + for path in walk_files(root, ".spend.jsonl"): + for line in path.read_text().splitlines(): + try: + record = json.loads(line) + except json.JSONDecodeError: + continue + record["_smoke"] = _under_smoke(path, root) + records.append(record) + return pd.DataFrame(records) + + +def smoke_file_times(roots: list[Path]) -> dict[str, float]: + """When each smoke response was first served: the earliest completion time + of a smoke row carrying its id. Rows without a time cannot show order and + are skipped.""" + finished: dict[str, float] = {} + wanted = {"provider_response_id", "request_completed_at"} + for root in roots: + for path in walk_files(root, ".csv"): + if not _under_smoke(path, root): + continue + try: + frame = pd.read_csv( + path, usecols=lambda column: column in wanted, low_memory=False + ) + except (ValueError, pd.errors.ParserError, UnicodeDecodeError): + continue + if not wanted <= set(frame.columns): + continue + completed = pd.to_numeric(frame["request_completed_at"], errors="coerce") + pairs = pd.DataFrame( + {"id": frame["provider_response_id"], "completed": completed} + ).dropna() + for response_id, when in zip(pairs["id"], pairs["completed"]): + key = str(response_id) + if key not in finished or when < finished[key]: + finished[key] = float(when) + return finished + + +def _ledger_hits(ledgers: pd.DataFrame) -> dict[str, list[tuple[str, str, float]]]: + """Cache hits outside the smokes: response id -> (model, household, start).""" + hits: dict[str, list[tuple[str, str, float]]] = {} + if ledgers.empty or "cache_hit" not in ledgers: + return hits + board = ledgers[~ledgers["_smoke"] & (ledgers["cache_hit"] == True)] # noqa: E712 + board = board.dropna(subset=["provider_response_id", "request_started_at"]) + columns = ["provider_response_id", "model", "scenario_id", "request_started_at"] + for response_id, model, scenario, started in board[columns].itertuples(index=False): + hits.setdefault(response_id, []).append((model, scenario, float(started))) + return hits + + +def _smoke_ledger_times(ledgers: pd.DataFrame) -> dict[str, float]: + """When a smoke first served each response id live.""" + if ledgers.empty or "cache_hit" not in ledgers: + return {} + smoke = ledgers[ledgers["_smoke"] & (ledgers["cache_hit"] != True)] # noqa: E712 + return ( + smoke.dropna(subset=["provider_response_id", "request_completed_at"]) + .groupby("provider_response_id")["request_completed_at"] + .min() + .to_dict() + ) + + +def build( + predictions: pd.DataFrame, + stored: dict[str, float], + ledgers: pd.DataFrame, + smoke_times: dict[str, float], +) -> list[dict]: + latency = household_latency_seconds(predictions) + grouped = predictions.groupby(["model", "scenario_id"], sort=True) + first_id = grouped["provider_response_id"].first() + started = grouped["request_started_at"].min() + board_start = predictions.groupby("model")["request_started_at"].min() + ledger_hits = _ledger_hits(ledgers) + smoke_ledger = _smoke_ledger_times(ledgers) + + rows = [] + for (model, scenario), seconds in latency.items(): + if pd.isna(seconds): + continue + recorded = first_id.get((model, scenario)) + inner = provider_response_id(recorded) + start = started.get((model, scenario)) + timed = pd.notna(start) + # When the response is known to have existed, by source and clock. + local: dict[str, float] = {} + if isinstance(inner, str) and inner in stored: + local["cache"] = stored[inner] + if recorded in smoke_ledger: + local["smoke_ledger"] = float(smoke_ledger[recorded]) + if recorded in smoke_times: + local["smoke_file"] = smoke_times[recorded] + existed = {name: when for name, when in local.items() if timed and when < start} + created = id_created_at(inner) + if created is not None and timed and created < start - MARGIN_SECONDS: + name = "openrouter_id" if inner.startswith("gen-") else "openai_id" + existed[name] = created + evidence = set(existed) + if timed and any( + hit_model == model + and hit_scenario == scenario + and abs(hit_start - start) <= SAME_REQUEST_SECONDS + for hit_model, hit_scenario, hit_start in ledger_hits.get(recorded, ()) + ): + evidence.add("spend_ledger") + if not evidence and seconds >= FULL_REPLAY_SECONDS: + continue + if not evidence: + replay, source = "untraced", "untraced" + else: + replay = "full" if seconds < FULL_REPLAY_SECONDS else "partial" + earliest = min(existed.values()) if existed else None + if {"smoke_ledger", "smoke_file"} & evidence: + source = "onboarding smoke" + elif earliest is None: + source = "unknown" + elif earliest >= board_start[model]: + source = "same run" + else: + source = "before the board rows" + rows.append( + { + "model": model, + "scenario_id": scenario, + "household_latency_seconds": f"{seconds:.6f}", + "replay": replay, + "evidence": "+".join(sorted(evidence)), + "source": source, + "first_response_id": recorded, + "request_started_at": f"{start:.3f}" if timed else "", + "response_existed_at": ( + f"{min(existed.values()):.3f}" if existed else "" + ), + } + ) + return rows + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--predictions", type=Path, default=FROZEN_PREDICTIONS) + parser.add_argument("--cache-dir", type=Path, action="append", default=[]) + parser.add_argument("--ledger-root", type=Path, action="append", default=[]) + parser.add_argument("--output", type=Path, default=OUTPUT) + args = parser.parse_args() + + predictions = pd.read_csv(args.predictions, low_memory=False) + rows = build( + predictions, + cache_store_times(args.cache_dir), + ledger_records(args.ledger_root), + smoke_file_times(args.ledger_root), + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + with open(args.output, "w", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=FIELDS, lineterminator="\n") + writer.writeheader() + writer.writerows(rows) + print(f"wrote {len(rows)} households to {os.path.relpath(args.output, ROOT)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/finish_argon.py b/scripts/finish_argon.py index 3c5027df..c426ebbb 100644 --- a/scripts/finish_argon.py +++ b/scripts/finish_argon.py @@ -212,8 +212,10 @@ INPUT_PINS = ROOT / INPUT_PINS_PATH PINNED_INPUTS = ("predictions.csv", "run_state.json") # Incumbent usage the exporter cannot recompute from committed predictions: -# Fable 5 ran through the Anthropic batch adapter, and its rows carry no cost, -# token or latency fields, so export_full_run reports $0 and omits the rest. +# Fable 5's board rows were rebuilt from release 20260701 without their cost, +# token and latency fields (its original per-call rows are that release's +# claude-fable-5-predictions.csv.gz), so export_full_run reports $0 and +# omits the rest. # The released values are carried over; the drift gate then checks every field. CARRIED_USAGE = { "claude-fable-5": ("costUsd", "costPerHousehold", "totalTokens", "latencySeconds") diff --git a/scripts/finish_gpt61sol.py b/scripts/finish_gpt61sol.py index 982ae6f6..1d312c59 100644 --- a/scripts/finish_gpt61sol.py +++ b/scripts/finish_gpt61sol.py @@ -174,8 +174,10 @@ INPUT_PINS = ROOT / INPUT_PINS_PATH PINNED_INPUTS = ("predictions.csv", "run_state.json") # Incumbent usage the exporter cannot recompute from committed predictions: -# Fable 5 ran through the Anthropic batch adapter, and its rows carry no cost, -# token or latency fields, so export_full_run reports $0 and omits the rest. +# Fable 5's board rows were rebuilt from release 20260701 without their cost, +# token and latency fields (its original per-call rows are that release's +# claude-fable-5-predictions.csv.gz), so export_full_run reports $0 and +# omits the rest. # The released values are carried over; the drift gate then checks every field. CARRIED_USAGE = { "claude-fable-5": ("costUsd", "costPerHousehold", "totalTokens", "latencySeconds") diff --git a/scripts/finish_haiku55.py b/scripts/finish_haiku55.py index 233f09ec..b1c081ae 100644 --- a/scripts/finish_haiku55.py +++ b/scripts/finish_haiku55.py @@ -192,8 +192,10 @@ INPUT_PINS = ROOT / INPUT_PINS_PATH PINNED_INPUTS = ("predictions.csv", "run_state.json") # Incumbent usage the exporter cannot recompute from committed predictions: -# Fable 5 ran through the Anthropic batch adapter, and its rows carry no cost, -# token or latency fields, so export_full_run reports $0 and omits the rest. +# Fable 5's board rows were rebuilt from release 20260701 without their cost, +# token and latency fields (its original per-call rows are that release's +# claude-fable-5-predictions.csv.gz), so export_full_run reports $0 and +# omits the rest. # The released values are carried over; the drift gate then checks every field. CARRIED_USAGE = { "claude-fable-5": ("costUsd", "costPerHousehold", "totalTokens", "latencySeconds") diff --git a/scripts/finish_musewonder.py b/scripts/finish_musewonder.py index 16fe4e99..8db8d063 100644 --- a/scripts/finish_musewonder.py +++ b/scripts/finish_musewonder.py @@ -202,8 +202,10 @@ INPUT_PINS = ROOT / INPUT_PINS_PATH PINNED_INPUTS = ("predictions.csv", "run_state.json") # Incumbent usage the exporter cannot recompute from committed predictions: -# Fable 5 ran through the Anthropic batch adapter, and its rows carry no cost, -# token or latency fields, so export_full_run reports $0 and omits the rest. +# Fable 5's board rows were rebuilt from release 20260701 without their cost, +# token and latency fields (its original per-call rows are that release's +# claude-fable-5-predictions.csv.gz), so export_full_run reports $0 and +# omits the rest. # The released values are carried over; the drift gate then checks every field. CARRIED_USAGE = { "claude-fable-5": ("costUsd", "costPerHousehold", "totalTokens", "latencySeconds") diff --git a/scripts/freeze_snapshot.py b/scripts/freeze_snapshot.py index 329caefa..320c9a03 100644 --- a/scripts/freeze_snapshot.py +++ b/scripts/freeze_snapshot.py @@ -1823,9 +1823,11 @@ def build_manifest( "failed or timed-out scenarios in bounded rounds; every model's " "canonical file covers all 100 households.", "Raw provider responses are retained in the compressed source-run " - "predictions.csv.gz file wherever the transport exposed them; rows " - "served through the Anthropic batch adapter (Claude Fable 5) and " - "64 Kimi K3 parse failures carry no raw payload. The separate " + "predictions.csv.gz file wherever the transport exposed them; " + "Claude Fable 5's rows, rebuilt from release dashboard-data-20260701 " + "without their raw payloads (that release's " + "claude-fable-5-predictions.csv.gz asset keeps them), and 64 Kimi " + "K3 parse failures carry none. The separate " "LiteLLM cache remains " "local-only because it is a generated request cache, not the " "canonical snapshot artifact.", diff --git a/scripts/release_20261006.py b/scripts/release_20261006.py index 42b4ab94..a7f8e078 100644 --- a/scripts/release_20261006.py +++ b/scripts/release_20261006.py @@ -86,8 +86,8 @@ STAGE_ROOT = Path("results/local") BOARD_MODELS = 46 TOTAL_OUTPUTS = 1984 -# Fable 5 ran through the Anthropic batch adapter, whose rows carry no cost, -# token or latency fields; release 20260930 carried its released values, and +# Fable 5's board rows were rebuilt from release 20260701 without their cost, +# token and latency fields; release 20260930 carried its released values, and # so does this release (scripts/finish_gpt61sol.py, CARRIED_USAGE). CARRIED_USAGE = { "claude-fable-5": ("costUsd", "costPerHousehold", "totalTokens", "latencySeconds") diff --git a/tests/test_cost_latency.py b/tests/test_cost_latency.py index df3b8dfb..a43555d0 100644 --- a/tests/test_cost_latency.py +++ b/tests/test_cost_latency.py @@ -92,3 +92,140 @@ def test_all_null_latency_is_omitted_not_zero(): def test_empty_predictions_returns_empty(): assert model_cost_latency(pd.DataFrame()) == {} + + +# ----- invariants of household latency (docs/latency_replay_audit.md) ------- + +from hypothesis import given, settings # noqa: E402 +from hypothesis import strategies as st # noqa: E402 + +from policybench.analysis import ( # noqa: E402 + REPLAYED_HOUSEHOLD_SECONDS, + household_latency_seconds, +) + +# A household's calls: each call's latency and how many rows it answered. +CALLS = st.lists( + st.tuples( + st.floats(min_value=0.0, max_value=2_000.0, allow_nan=False), + st.integers(min_value=1, max_value=8), + ), + min_size=1, + max_size=6, +) +HOUSEHOLDS = st.lists(CALLS, min_size=1, max_size=15) + + +def _split_rows(households): + """Rows the way the harness writes them: each call's latency divided + equally among the rows it answered.""" + rows = [] + for index, calls in enumerate(households): + for seconds, n_rows in calls: + rows += [_row("m", f"s{index:03d}", 0.01, seconds / n_rows)] * n_rows + return pd.DataFrame(rows) + + +@given(HOUSEHOLDS) +@settings(max_examples=200, deadline=None) +def test_household_latency_is_the_sum_of_its_calls(households): + """Splitting a call's latency across its rows and summing the rows gives + back the household's total request time.""" + latency = household_latency_seconds(_split_rows(households)) + for index, calls in enumerate(households): + expected = sum(seconds for seconds, _ in calls) + assert latency[("m", f"s{index:03d}")] == pytest.approx(expected, abs=1e-9) + + +@given(HOUSEHOLDS) +@settings(max_examples=200, deadline=None) +def test_published_latency_is_the_median_household(households): + preds = _split_rows(households) + expected = float(np.median([sum(s for s, _ in calls) for calls in households])) + out = model_cost_latency(preds) + assert out["m"]["latencySeconds"] == pytest.approx(expected, abs=1e-6) + # Cost is additive the same way: the run total is every row's cost. + assert out["m"]["costUsd"] == pytest.approx(0.01 * len(preds)) + + +@given( + st.lists( + st.floats(min_value=0.0, max_value=2_000.0, allow_nan=False), + min_size=1, + max_size=20, + ), + st.data(), +) +@settings(max_examples=200, deadline=None) +def test_a_replay_can_only_lower_the_median(seconds, data): + """Replacing a household's time with a cache lookup's (smaller) time + never raises the published median, so replays bias it down, never up.""" + index = data.draw(st.integers(min_value=0, max_value=len(seconds) - 1)) + lookup = data.draw(st.floats(min_value=0.0, max_value=seconds[index])) + before = model_cost_latency( + pd.DataFrame([_row("m", f"s{i}", 0.01, t) for i, t in enumerate(seconds)]) + )["m"]["latencySeconds"] + replayed = list(seconds) + replayed[index] = lookup + after = model_cost_latency( + pd.DataFrame([_row("m", f"s{i}", 0.01, t) for i, t in enumerate(replayed)]) + )["m"]["latencySeconds"] + assert after <= before + 1e-9 + + +@given( + st.integers(min_value=0, max_value=12), + st.integers(min_value=0, max_value=12), +) +def test_a_median_under_a_second_means_most_households_replayed(n_fast, n_slow): + """With replays under the line and calls above it, the median is under the + line when more than half the households replayed and at or above it when + fewer than half did. At exactly half it can fall either side (the tie test + below), so the paper's dash rule is a majority-replay rule only off ties.""" + if n_fast + n_slow == 0: + return + seconds = [0.02] * n_fast + [30.0] * n_slow + median = model_cost_latency( + pd.DataFrame([_row("m", f"s{i}", 0.01, t) for i, t in enumerate(seconds)]) + )["m"]["latencySeconds"] + if 2 * n_fast > n_fast + n_slow: + assert median < REPLAYED_HOUSEHOLD_SECONDS + if 2 * n_fast < n_fast + n_slow: + assert median >= REPLAYED_HOUSEHOLD_SECONDS + + +@given( + st.integers(min_value=1, max_value=12), + st.floats(min_value=0.0, max_value=0.12), + st.floats(min_value=1.0, max_value=3.0), +) +def test_at_exactly_half_replays_the_median_is_the_middle_pairs_mean(n, replay, call): + """Half replays and half calls: the median is the mean of the slowest + replay and the fastest call, which is under a second only when the calls + are fast enough (50 replays at 0.02 s and 50 calls at 1.75 s give 0.885 s).""" + seconds = [replay] * n + [call] * n + median = model_cost_latency( + pd.DataFrame([_row("m", f"s{i}", 0.01, t) for i, t in enumerate(seconds)]) + )["m"]["latencySeconds"] + assert median == pytest.approx((replay + call) / 2) + assert (median < REPLAYED_HOUSEHOLD_SECONDS) == ( + replay + call < 2 * REPLAYED_HOUSEHOLD_SECONDS + ) + + +@given( + st.lists(st.floats(min_value=1.0, max_value=500.0), min_size=1, max_size=10), + st.integers(min_value=1, max_value=10), +) +def test_untimed_households_are_left_out_not_counted_as_zero(seconds, n_untimed): + timed = [_row("m", f"t{i}", 0.01, t) for i, t in enumerate(seconds)] + untimed = [_row("m", f"u{i}", 0.01, np.nan) for i in range(n_untimed)] + out = model_cost_latency(pd.DataFrame(timed + untimed)) + assert out["m"]["latencySeconds"] == pytest.approx(float(np.median(seconds))) + latency = household_latency_seconds(pd.DataFrame(untimed)) + assert latency.isna().all() + + +def test_household_latency_of_nothing_is_empty(): + assert household_latency_seconds(pd.DataFrame()).empty + assert household_latency_seconds(pd.DataFrame([{"model": "m"}])).empty diff --git a/tests/test_disclosures.py b/tests/test_disclosures.py index 37fe51e9..6cd217d4 100644 --- a/tests/test_disclosures.py +++ b/tests/test_disclosures.py @@ -66,6 +66,19 @@ ) PAPER_FALSE_CLAIM_PATTERNS = ( + # No board model logged submission times: GPT-5.5's batch rows log none, + # and the sub-second medians are cache replays of synchronous calls. + # Claude Fable 5 ran synchronously, and its usage was recorded per call + # (docs/latency_replay_audit.md). + r"batch\s+or\s+asynchronous\s+dispatch", + r"reflect\s+submission\s+rather\s+than\s+generation", + r"recorded\s+at\s+the\s+run\s+level", + r"Anthropic\s+batch\s+adapter", + r"not\s+measured\s+per\s+call", + # The harness restarts its timer on each retry and times only the attempt + # that succeeded; a replayed row's cost is recomputed when it is written. + r"spans\s+any\s+provider-side\s+retry", + r"keeps\s+the\s+cost\s+recorded\s+when\s+it\s+was\s+generated", # Exclusion covers what the audits found, not every defect or unstated # input: an output a fix or reading not run on the reference engine would # move can still be scored (the paper's limitations say so). @@ -1165,65 +1178,54 @@ def names(model: str, sentence: str) -> bool: assert sorted(m for m in serving if names(m, note_choice)) == sorted(card_choice) -# The benchmark card's cache-replay examples ("Cost basis"): household -# latency is the sum of its rows' elapsed_seconds, since each row records its -# call's latency divided among the call's outputs. -REPLAYED_SMOKE_HOUSEHOLDS = { - ("muse-spark-1.3", "scenario_000"), - ("muse-spark-1.3", "scenario_001"), - ("muse-spark-1.3", "scenario_099"), - ("muse-spark-1.3", "scenario_119"), -} -NEAR_ZERO_EXAMPLES = { - ("muse-wonder", "scenario_000"), - ("muse-wonder", "scenario_001"), - ("claude-opus-5.5", "scenario_000"), - ("claude-opus-5.5", "scenario_001"), - ("claude-haiku-5.5", "scenario_000"), -} - +# The benchmark card's cache-replay accounting ("Latency basis"), checked +# against the frozen predictions; tests/test_latency_replay_audit.py checks the +# audit's household list itself. +def test_card_latency_basis_matches_the_frozen_households(): + """Every count the card states is the frozen data's: the sub-second + households, the traced and untraced ones, the per-model replays, and the + largest median shift.""" + from policybench.analysis import REPLAYED_HOUSEHOLD_SECONDS -def _household_latency() -> dict[tuple[str, str], tuple[float, int]]: - """(model, scenario_id) -> (summed row latency in seconds, row count), - from the frozen predictions.""" - import pandas as pd - - manifest = json.loads(SNAPSHOT_MANIFEST.read_text()) - run_dir = ( - ROOT - / manifest["source_run_artifacts"][manifest["source_run_labels"]["us"]]["path"] - ) - predictions = pd.read_csv( - run_dir / "predictions.csv.gz", - usecols=["model", "scenario_id", "elapsed_seconds"], - ) - grouped = predictions.groupby(["model", "scenario_id"])["elapsed_seconds"] - sums, counts = grouped.sum(min_count=1), grouped.size() - return {key: (float(sums[key]), int(counts[key])) for key in sums.index} - - -def test_cache_replay_examples_match_the_frozen_household_latencies(): - """The card's replayed smoke households take 0.03 seconds or less, and - each other example it names sums to under 0.05 seconds; every example is - a household the card names, with recorded (not missing) latency.""" text = re.sub(r"\s+", " ", BENCHMARK_CARD.read_text()) + latency = r.household_latencies + subsecond = latency[latency < REPLAYED_HOUSEHOLD_SECONDS] + audit = r.latency_replay_audit + full = audit[audit["replay"] == "full"].groupby("model").size() + partial = audit[audit["replay"] == "partial"].groupby("model").size() + assert "a household's latency is the sum over its rows" in text + assert f"{len(subsecond)} households sum to under a second" in text + assert f"traces {int(full.sum())} of them" in text + assert f"Another {int(partial.sum())} households replay their first" in text + untraced = audit[audit["replay"] == "untraced"] + assert sorted(zip(untraced["model"], untraced["scenario_id"])) == [ + ("claude-haiku-4.5", "scenario_009"), + ("grok-4.3", "scenario_000"), + ] + assert "(Grok 4.3's scenario_000 and Claude Haiku 4.5's scenario_009)" in text + for model, label in ( + ("qwen-3.7-max", "Qwen3.7-max"), + ("glm-5.2", "GLM-5.2"), + ("kimi-k2.6", "Kimi K2.6"), + ("minimax-m3", "MiniMax M3"), + ("claude-sonnet-5", "Claude Sonnet 5"), + ("muse-spark-1.3", "Muse Spark 1.3"), + ): + assert f"{label} ({int(full[model])}" in text, model + for model in ("muse-wonder", "claude-opus-5.5", "claude-haiku-5.5"): + assert int(full[model]) == 2, model + for model in ("deepseek-v4-pro", "gpt-5.6-sol", "gpt-5.6-luna", "gpt-5.6-terra"): + assert int(full[model]) == 2, model + for model, label in ( + ("glm-5.3", "GLM-5.3"), + ("ox-alpha", "GLM-5.3-Flash (preview)"), + ): + assert f"{label} ({int(partial[model])})" in text, model assert ( - "The Muse Spark 1.3 row replays its four smoke households (scenario_000, " - "scenario_001, scenario_099 and scenario_119)" - ) in text - assert "a household latency of 0.03 seconds or less" in text - assert ( - "among them scenario_000 and scenario_001 for Muse Wonder and Claude " - "Opus 5.5 and scenario_000 for Claude Haiku 5.5." - ) in text - latency = _household_latency() - for key in REPLAYED_SMOKE_HOUSEHOLDS: - seconds, rows = latency[key] - assert rows > 1 and 0 < seconds <= 0.03, (key, seconds) - for key in NEAR_ZERO_EXAMPLES: - seconds, rows = latency[key] - assert rows > 1 and 0 < seconds < 0.05, (key, seconds) - # Round 1's wrong example: the household is slower once its rows are - # summed, so the card must not name it. - assert latency[("claude-haiku-5.5", "scenario_001")][0] >= 0.05 + f"no published median by more than {r.subsecond_median_shift_fmt} seconds" + in text + ) + published = {row["model"]: row.get("latencySeconds") for row in r.model_stats} + replayed = [published[m] for m in r.replayed_latency_models] + assert f"{min(replayed):.3f} to {max(replayed):.3f} seconds" in text diff --git a/tests/test_latency_replay_audit.py b/tests/test_latency_replay_audit.py new file mode 100644 index 00000000..0367891f --- /dev/null +++ b/tests/test_latency_replay_audit.py @@ -0,0 +1,439 @@ +"""The latency replay audit agrees with the frozen predictions. + +``docs/latency_replay_audit/households.csv`` was built from artifacts outside +the repository (disk caches, spend ledgers, smoke files). These tests check +everything about it that the frozen predictions alone can check: that each +row is the household it names, that the sub-second households are exactly the +full replays plus the untraced ones, and that the id-time evidence recomputes. +They also pin the facts the paper and card state about the untimed models. +""" + +from __future__ import annotations + +import gzip +import json +import re +from pathlib import Path + +import pandas as pd +import pytest + +from policybench.analysis import ( + REPLAYED_HOUSEHOLD_SECONDS, + household_latency_seconds, + model_cost_latency, +) +from policybench.config import PRICE_OVERRIDES_PER_1M +from policybench.paper_results import LATENCY_REPLAY_AUDIT, r +from scripts.audit_latency_replays import ( + MARGIN_SECONDS, + id_created_at, + provider_response_id, +) + +ROOT = Path(__file__).resolve().parents[1] +RUN_DIR = ( + ROOT + / "paper" + / "snapshot" + / "20260501" + / "runs" + / "us_full_run_20260612_policyengine_4_16_1_populace" +) +AUDIT_DOC = ROOT / "docs" / "latency_replay_audit.md" +MANIFEST = ROOT / "paper" / "snapshot" / "20260501" / "manifest.json" +# Fable 5's usage, carried from its original run's per-call rows (release +# dashboard-data-20260701, claude-fable-5-predictions.csv.gz). +FABLE_CARRIED = { + "costUsd": 54.109099999999955, + "costPerHousehold": 0.5410909999999995, + "totalTokens": 3850174, + "latencySeconds": 97.6780539804895, +} + + +@pytest.fixture(scope="module") +def predictions() -> pd.DataFrame: + return pd.read_csv(RUN_DIR / "predictions.csv.gz", low_memory=False) + + +@pytest.fixture(scope="module") +def households(predictions) -> pd.DataFrame: + grouped = predictions.groupby(["model", "scenario_id"], sort=True) + return pd.DataFrame( + { + "latency": household_latency_seconds(predictions), + "first_response_id": grouped["provider_response_id"].first(), + "started": grouped["request_started_at"].min(), + } + ) + + +@pytest.fixture(scope="module") +def audit() -> pd.DataFrame: + # An untraced household cites no evidence: an empty field, not NaN. + return pd.read_csv(LATENCY_REPLAY_AUDIT).fillna({"evidence": ""}) + + +@pytest.fixture(scope="module") +def model_stats() -> dict[str, dict]: + payload = json.loads(gzip.open(RUN_DIR / "data.json.gz").read()) + return {row["model"]: row for row in payload["modelStats"]} + + +def test_each_audit_row_is_the_household_it_names(audit, households): + assert not audit.duplicated(["model", "scenario_id"]).any() + for row in audit.itertuples(): + household = households.loc[(row.model, row.scenario_id)] + assert row.household_latency_seconds == pytest.approx( + household["latency"], abs=1e-6 + ), row + assert row.first_response_id == household["first_response_id"], row + + +def test_subsecond_households_are_the_full_replays_and_the_untraced(audit, households): + """A full replay sums to under a second and a partial one does not, and no + sub-second household is missing from the audit.""" + subsecond = set( + households.index[households["latency"] < REPLAYED_HOUSEHOLD_SECONDS] + ) + audited = { + (m, s): kind for m, s, kind in audit[["model", "scenario_id", "replay"]].values + } + assert subsecond == {key for key, kind in audited.items() if kind != "partial"} + for (model, scenario), kind in audited.items(): + seconds = households.loc[(model, scenario), "latency"] + if kind == "partial": + assert seconds >= REPLAYED_HOUSEHOLD_SECONDS, (model, scenario) + else: + assert seconds < REPLAYED_HOUSEHOLD_SECONDS, (model, scenario) + + +def test_replays_and_live_households_are_far_apart(households, audit): + """Every full replay sums to under 0.12 s and the fastest household the + audit does not list takes over 1.5 s, so the one-second line is not close + to any household.""" + full = audit[audit["replay"] == "full"] + assert full["household_latency_seconds"].max() < 0.12 + listed = set(zip(audit["model"], audit["scenario_id"])) + unlisted = households[[key not in listed for key in households.index]] + assert unlisted["latency"].dropna().min() > 1.5 + + +def test_id_time_evidence_recomputes_and_is_complete(audit, households): + """Where an id embeds its creation time, the audit cites it exactly when + the response predates its request by more than the margin; on every other + household that time sits within the margin of the request.""" + cited = { + (m, s) + for m, s, evidence in audit[["model", "scenario_id", "evidence"]].values + if re.search(r"openrouter_id|openai_id", evidence) + } + checked = 0 + for (model, scenario), household in households.iterrows(): + created = id_created_at(provider_response_id(household["first_response_id"])) + if created is None or pd.isna(household["started"]): + continue + checked += 1 + early = created < household["started"] - MARGIN_SECONDS + assert early == ((model, scenario) in cited), (model, scenario) + if not early: + assert abs(household["started"] - created) <= MARGIN_SECONDS + assert checked > 1000 + + +def test_the_untraced_households_predate_per_call_timestamps(audit, predictions): + untraced = audit[audit["replay"] == "untraced"] + assert sorted(untraced["model"]) == ["claude-haiku-4.5", "grok-4.3"] + for model in untraced["model"]: + rows = predictions[predictions["model"] == model] + assert rows["request_started_at"].isna().all(), model + + +def test_the_audit_doc_counts_are_the_csvs(audit): + text = re.sub(r"\s+", " ", AUDIT_DOC.read_text()) + full = int((audit["replay"] == "full").sum()) + partial = int((audit["replay"] == "partial").sum()) + replayed_models = audit.loc[audit["replay"] != "untraced", "model"].nunique() + assert ( + f"That is {full} full and {partial} partial replays across " + f"{replayed_models} models." + ) in text + for line in AUDIT_DOC.read_text().splitlines(): + cells = [cell.strip() for cell in line.strip("|").split("|")] + if len(cells) != 6 or not cells[1].isdigit(): + continue + name, n_full, n_partial = cells[0], int(cells[1]), int(cells[2]) + model = next(m for m in audit["model"].unique() if r.model_name(m) == name) + rows = audit[audit["model"] == model] + assert n_full == int((rows["replay"] == "full").sum()), name + assert n_partial == int((rows["replay"] == "partial").sum()), name + + +def test_the_audit_doc_medians_are_the_frozen_ones(households, model_stats): + """The table's published median is modelStats', and its other median + drops the model's sub-second households.""" + for line in AUDIT_DOC.read_text().splitlines(): + cells = [cell.strip() for cell in line.strip("|").split("|")] + if len(cells) != 6 or not cells[1].isdigit(): + continue + model = next(m for m in model_stats if r.model_name(m) == cells[0]) + published = float(cells[4].replace(",", "")) + assert published == pytest.approx( + model_stats[model]["latencySeconds"], abs=0.051 + ), model + latency = households.loc[model, "latency"].dropna() + kept = latency[latency >= REPLAYED_HOUSEHOLD_SECONDS] + without = float(cells[5].split(" ")[0].replace(",", "")) + assert without == pytest.approx(kept.median(), abs=0.051), model + if "households" in cells[5]: + assert f"({len(kept)} households)" in cells[5], model + + +def test_published_cost_and_latency_reproduce_except_fables_carried_usage( + predictions, model_stats +): + """modelStats' cost and latency are model_cost_latency's over the frozen + rows; the only exceptions are Claude Fable 5's four carried fields.""" + computed = model_cost_latency(predictions, PRICE_OVERRIDES_PER_1M) + for model, row in model_stats.items(): + for field in ("costUsd", "costPerHousehold", "latencySeconds", "totalTokens"): + if model == "claude-fable-5" and field in FABLE_CARRIED: + assert row[field] == FABLE_CARRIED[field] + continue + expected = computed.get(model, {}).get(field) + if expected is None: + assert row.get(field) is None, (model, field) + else: + assert row[field] == pytest.approx(expected, rel=1e-12), (model, field) + + +def test_the_untimed_models_are_the_batch_run_and_fables_rebuilt_rows(predictions): + """GPT-5.5's batch rows have costs but no times; Fable 5's rebuilt rows + have neither, nor raw payloads, but keep their explanations.""" + assert r.untimed_latency_models == ["claude-fable-5", "gpt-5.5"] + gpt = predictions[predictions["model"] == "gpt-5.5"] + assert gpt["request_started_at"].isna().all() + assert gpt["total_cost_usd"].notna().all() + fable = predictions[predictions["model"] == "claude-fable-5"] + for column in ("elapsed_seconds", "total_cost_usd", "total_tokens", "raw_response"): + assert fable[column].isna().all(), column + assert fable["explanation"].notna().all() + + +def test_the_paper_dashes_exactly_the_untimed_and_replayed_medians(model_stats): + assert r.latency_dash_models == [ + "glm-5.2", + "gpt-5.5", + "kimi-k2.6", + "minimax-m3", + "qwen-3.7-max", + ] + assert r.replayed_latency_models == [ + "glm-5.2", + "kimi-k2.6", + "minimax-m3", + "qwen-3.7-max", + ] + for model in r.replayed_latency_models: + assert model_stats[model]["latencySeconds"] < 0.05, model + assert r.zero_cost_models == ["ox-alpha"] + + +def test_the_manifest_no_longer_says_fable_ran_through_the_batch_adapter(): + notes = " ".join(json.loads(MANIFEST.read_text())["reproducibility_notes"]) + assert "Anthropic batch adapter" not in notes + assert "rebuilt from release dashboard-data-20260701" in notes + + +PAPER_HTML = ROOT / "app" / "public" / "paper" / "web" / "index.html" + + +def _cost_latency_rows() -> list[list[str]]: + """The rendered cost-and-latency table's rows, as visible cell text.""" + from html import unescape + + page = PAPER_HTML.read_text() + header = page.index("Latency (median)") + table = page[page.rfind("", header)] + rows = [] + for row in re.findall(r"]*>(.*?)", table, re.S): + cells = re.findall(r"]*>(.*?)", row, re.S) + rows.append( + [ + unescape(re.sub(r"<[^>]+>", "", cell)).replace("​", "").strip() + for cell in cells + ] + ) + return rows + + +def _visible_text() -> str: + from html import unescape + + text = re.sub(r"<[^>]+>", " ", PAPER_HTML.read_text()) + text = re.sub(r"\s+", " ", unescape(text)).replace("​", "") + # Quarto renders apostrophes as typographic quotes. + return text.replace("’", "'").replace("‘", "'") + + +def test_every_dash_in_the_rendered_table_is_one_the_text_explains(): + header, *rows = _cost_latency_rows() + assert header == ["Model", "Exact (%)", "Cost / household", "Latency (median)"] + assert len(rows) == len(r.model_stats) + latency_dash = sorted(row[0] for row in rows if row[3] == "—") + cost_dash = sorted(row[0] for row in rows if row[2] == "—") + assert latency_dash == sorted(r.model_name(m) for m in r.latency_dash_models) + assert cost_dash == sorted(r.model_name(m) for m in r.zero_cost_models) + # Fable 5's untimed rows are not dashed: its carried per-call median shows. + fable = next(row for row in rows if row[0] == r.model_name("claude-fable-5")) + assert fable[3] == f"{round(FABLE_CARRIED['latencySeconds'])} s" + + +def test_the_rendered_paper_states_why_each_latency_is_missing(): + text = _visible_text() + assert ( + f"{r.batch_latency_models_fmt} ran through OpenAI's batch API, which " + "records no per-call request time, so it shows a dash." in text + ) + assert ( + f"{r.replayed_latency_models_fmt} were finished over several passes, " + "each of which replayed the earlier passes' successful responses." in text + ) + assert ( + f"records a replay on {r.replayed_latency_range_fmt} of its " + f"{r.n_households_fmt} households" in text + ) + assert ( + f"In {r.other_subsecond_model_count_fmt} other models, " + f"{r.other_subsecond_household_count} households sum to under a second." in text + ) + assert f"The audit traces {r.traced_other_subsecond_count} of them" in text + assert f"more than {r.subsecond_median_shift_fmt} seconds" in text + assert f"A further {r.partial_replay_count} households replay" in text + assert "Claude Fable 5's board rows carry no usage or timing fields" in text + + +# ----- the builder's evidence rules (scripts/audit_latency_replays.py) ------- + + +def _frozen(rows): + """Frozen-style rows: (model, scenario, response id, start, household s).""" + records = [] + for model, scenario, response_id, start, seconds in rows: + for variable in ("a", "b"): + records.append( + { + "model": model, + "scenario_id": scenario, + "variable": variable, + "provider_response_id": response_id, + "request_started_at": start, + "elapsed_seconds": seconds / 2, + } + ) + return pd.DataFrame(records) + + +def _ledger(rows): + """Ledger records: (response id, model, scenario, start, cache hit, smoke).""" + return pd.DataFrame( + [ + { + "provider_response_id": response_id, + "model": model, + "scenario_id": scenario, + "request_started_at": start, + "request_completed_at": start + 30.0, + "cache_hit": hit, + "_smoke": smoke, + } + for response_id, model, scenario, start, hit, smoke in rows + ] + ) + + +def _built(predictions, stored=None, ledgers=None, smoke_times=None): + from scripts.audit_latency_replays import build + + rows = build( + predictions, + stored or {}, + ledgers if ledgers is not None else pd.DataFrame(), + smoke_times or {}, + ) + return {(row["model"], row["scenario_id"]): row for row in rows} + + +def test_a_later_replay_of_a_live_board_response_is_not_evidence(): + """A run after the board that replays a response the board fetched live + records a cache hit for the same id, but not at the board request.""" + board = _frozen([("m", "s1", "chatcmpl-live", 1_000.0, 30.0)]) + later = _ledger([("chatcmpl-live", "m", "s1", 5_000.0, True, False)]) + assert _built(board, ledgers=later) == {} + same = _ledger([("chatcmpl-live", "m", "s1", 1_000.0, True, False)]) + row = _built(board, ledgers=same)[("m", "s1")] + assert (row["replay"], row["evidence"]) == ("partial", "spend_ledger") + other = _ledger([("chatcmpl-live", "m", "s2", 1_000.0, True, False)]) + assert _built(board, ledgers=other) == {} + + +def test_a_smoke_counts_only_if_it_finished_before_the_request(): + board = _frozen([("m", "s1", "chatcmpl-x", 1_000.0, 0.02)]) + after = _built(board, smoke_times={"chatcmpl-x": 1_500.0})[("m", "s1")] + assert (after["replay"], after["evidence"]) == ("untraced", "") + before = _built(board, smoke_times={"chatcmpl-x": 900.0})[("m", "s1")] + assert (before["replay"], before["evidence"], before["source"]) == ( + "full", + "smoke_file", + "onboarding smoke", + ) + ledger = _ledger([("chatcmpl-x", "m", "s1", 1_500.0, False, True)]) + late = _built(board, ledgers=ledger)[("m", "s1")] + assert late["evidence"] == "" + + +def test_a_cached_response_counts_only_if_stored_before_the_request(): + board = _frozen([("m", "s1", "chatcmpl-x", 1_000.0, 45.0)]) + assert _built(board, stored={"chatcmpl-x": 1_010.0}) == {} + row = _built(board, stored={"chatcmpl-x": 990.0})[("m", "s1")] + assert (row["replay"], row["evidence"]) == ("partial", "cache") + + +def test_id_times_need_the_margin_and_live_households_are_not_listed(): + created = 1_783_264_675 + early = f"gen-{created}-abcdefghijABCDEFGHIJ" + board = _frozen( + [ + ("m", "s1", early, created + 120.0, 0.03), + ("m", "s2", early.replace("abc", "xyz"), created + 10.0, 40.0), + ("m", "s3", "chatcmpl-fast", created + 5.0, 0.04), + ] + ) + built = _built(board) + assert built[("m", "s1")]["evidence"] == "openrouter_id" + assert built[("m", "s1")]["replay"] == "full" + assert ("m", "s2") not in built + assert built[("m", "s3")]["replay"] == "untraced" + + +def test_smoke_file_times_read_columns_by_name(tmp_path): + """Smoke CSVs order their columns differently; a row without a completion + time cannot show order and is skipped, and files outside a smoke path are + not read.""" + from scripts.audit_latency_replays import smoke_file_times + + smoke = tmp_path / "x_smoke" / "run" + smoke.mkdir(parents=True) + pd.DataFrame( + { + "request_completed_at": [20.0, 10.0, None], + "provider_response_id": ["gen-a", "gen-a", "gen-b"], + } + ).to_csv(smoke / "predictions.csv", index=False) + board = tmp_path / "board" + board.mkdir() + pd.DataFrame( + {"provider_response_id": ["gen-c"], "request_completed_at": [5.0]} + ).to_csv(board / "predictions.csv", index=False) + assert smoke_file_times([tmp_path]) == {"gen-a": 10.0}