diff --git a/.env.example b/.env.example index 4a8a40f..abf1c78 100644 --- a/.env.example +++ b/.env.example @@ -1,11 +1,6 @@ -# Benchmarking against the hosted Jev API. TYPESAFE_API_KEY= - -# The LLM baseline `levbench compare` measures against. ANTHROPIC_API_KEY= -# Only if you front a local /v1/systemone server with auth. Your real -# TYPESAFE_API_KEY is deliberately never sent to a --base-url host. -# Left commented on purpose: `VAR=` sets the variable to the empty string, -# which is not the same as leaving it unset. +# Optional local-server auth; TYPESAFE_API_KEY is never sent to --base-url hosts. +# Leave commented to keep unset; VAR= assigns an empty string. # LEVBENCH_LOCAL_API_KEY= diff --git a/CLAUDE.md b/CLAUDE.md index 208444c..dcc546b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,34 +1,57 @@ -## Comments +# lev + +lev is an open System One decision model: `f(state, questions) → typed, calibrated distributions`, served over the same `/v1/systemone` wire format as TypeSafe's Jev, with no token generation. It is `Qwen/Qwen3.5-4B` with a rank-32 LoRA adapter, a set-attention candidate head, and per-bucket temperature calibration, released on the Hub as [`interfaze-ai/lev`](https://huggingface.co/interfaze-ai/lev). `levbench` is the harness that measures it, and Jev, on S1Bench. + +@CONTRIBUTING.md + +## Where the facts live -Keep comments rare and information-dense. +Read the relevant doc before changing anything it covers; most "obvious" improvements here were already tried and are recorded with their evidence. -- Do not add comments by default. -- Do not explain code that is already obvious from the implementation. -- Never narrate the implementation line-by-line. -- Do not add comments for simple assignments, conditionals, loops, function calls, or standard library usage. -- Prefer better naming and structure over explanatory comments. -- Comments should explain information that cannot be expressed clearly in the code. +| Doc | Holds | +| --- | --- | +| [`docs/STATUS.md`](docs/STATUS.md) | What is built, how far each part is exercised, the current headline numbers | +| [`docs/DECISIONS.md`](docs/DECISIONS.md) | ADR-001 to ADR-030; the next decision is ADR-031 | +| [`docs/FINDINGS.md`](docs/FINDINGS.md) | Every measurement and head-to-head, in order, with provenance | +| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | The design, and the trade-offs refused | +| [`docs/TRAINING.md`](docs/TRAINING.md) | The training order, presets, budgets, resume | +| [`docs/SETUP.md`](docs/SETUP.md) | Local and Modal setup, volumes, deploy knobs | +| [`hf/README.md`](hf/README.md) | The Hub model card; keep it consistent with `README.md` | -Add a comment only when it explains one of: +## The model -- non-obvious intent or rationale -- an important business rule -- an external API/protocol assumption -- a performance or security constraint -- an edge case that is easy to miss -- a workaround for a known bug or limitation +- **Readout.** Mode A reads label-token logits at the answer boundary whenever every label code is a single token — 68 options for this tokenizer, since the 69th code, `BQ`, is two. Above that, Mode B scores candidate text with the trained head. Training and serving route the same way (ADR-025, ADR-026); serving also skips split codes (ADR-028). Routing is in `router.py`. +- **Calibration** is per bucket, keyed `{type}:{mode}[:{band}]`, selected by leave-one-family-out transfer ECE. `calibrate.fit` refuses `test`, `eval` and `holdout` splits — fit only on the dedicated `calibration` split. +- **The release** (`lev-4b`) is preset `4b-instruct`, step 18750, chat prompt style. Its `lev_release.json` manifest sets the base model, prompt style and optional `adapter_scale`, and `lev.load` honours all three. +- **The released weights were trained on 2026-09-23 code, before this repository's first commit.** Changes to data, sources, prompts or the training loop since are not in the weights until a retrain; do not attribute a benchmark move to them. -Keep implementation comments to one or two concise lines whenever possible. +## Running things -Do not write multi-paragraph comments or verbose explanatory blocks. +- **Anything that needs a GPU runs on Modal**, through `modal/app.py` and the `make` targets. Do not download checkpoints or run models locally to check something — probe the deployed endpoint or add a `modal run` function. +- **Pass `--preset 4b-instruct` to every Modal function you call directly.** Their defaults are preset `4b`, the plain base-model run, not the release; the `make` targets pass `PRESET=4b-instruct` for you. `modal deploy` likewise serves `4b` unless `LEV_SERVE_PRESET` is set, which `make deploy` does. +- `make deploy` replaces the live endpoint. `modal/app.py` must import inside the container, where `python-dotenv` is not installed — keep its import guarded, or every container crash-loops. +- Modal volumes: `lev-models` (base weights), `lev-checkpoints` (runs and `releases/`), `lev-data` (mixture and splits). -For public APIs, use docstrings/documentation when the API contract, parameters, return behavior, exceptions, or usage constraints need to be documented. +## Benchmarks -When modifying existing code: +- S1Bench is 13 subsets pinned by Bespoke Nimble manifests at commit `62076b4`, in `packages/lev/src/lev/data/s1bench_subsets/`. All 13 are banned from training (ADR-009). +- **Never select anything on S1Bench** — a hyperparameter, a scale, a prompt. Select with leave-one-family-out over the subsets, as FINDINGS §19 does, or on data outside both training and S1Bench. The `calibration` split cannot select out-of-distribution knobs: it is drawn from the training mixture. +- Compare against Jev through identical task files, one complete run each. + +## Safety + +- **Ask before any paid call**: `levbench` against Jev, the Anthropic backend, or a sweep. They bill per token. +- `levbench` never sends `TYPESAFE_API_KEY` to a `--base-url` host; a custom endpoint gets `LEVBENCH_LOCAL_API_KEY` or `local`. Keep it that way. +- Do not read `.env`. +- This is the public repository. Development happens in [`Abhinavexists/lev`](https://github.com/Abhinavexists/lev) and is ported here by hand; the two diverge, so a change made here must be ported back, not assumed. + +## Comments -- Preserve useful existing comments. -- Remove comments that are made obsolete by the change. -- Do not rewrite comments merely for stylistic reasons. -- Do not perform unrelated comment cleanup. +The code should explain itself; a comment exists only for what it cannot say. -Default assumption: if the code is understandable without the comment, do not write the comment. +- **One line is the unit.** A comment is one line, a docstring is one line. Multi-line is earned — a module's two or three lines of orientation, or tensor shapes and wire formats. A second paragraph on a function belongs in `docs/` with a pointer from the code. +- **Do not restate the code.** If a reader would have to check the prose against the implementation to confirm they agree, the comment costs more than it gives. +- **Compress before deleting.** A real fact buried in a block above a statement usually becomes a short trailing comment on the statement. +- **Point, do not restate.** `(ADR-028)` beats a paragraph reproducing the decision. A contract the code states plainly — a guard that raises, a named check like `_refuse_held_out` — needs no prose. +- **Public surface is the exception.** `lev.load`, the server, and the `/v1/systemone` types are consumed without reading the body; document their contracts. +- Do not rewrite comments for style. When changing code, remove comments the change made obsolete, and do no unrelated cleanup. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d598c1c..996f384 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,7 @@ make check # ruff + the full test suite ``` -Both must be clean. Tests run without a GPU, network or API keys. A basic `uv sync` skips tests that need optional model dependencies; use `uv sync --extra train` to include the CPU tensor, cache and training-loop tests. +Both must be clean. Tests run without a GPU, network or API keys. use `uv sync --extra train` to include the CPU tensor, cache and training-loop tests. ## Finding the code diff --git a/Makefile b/Makefile index e3d49b6..5ba5612 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,15 @@ -# `make help` lists every target. Variables are overridable: `make train PRESET=9b`. +# setup: Install workspace | setup-train: + training deps +# setup-serve: + serving deps | setup-modal: + Modal client + +# test: Run tests | lint: Check code | fmt: Format code +# check: Lint + test | clean: Remove caches + +# smoke-local: CPU smoke test | smoke: Modal H100 smoke test +# train: Full training run | calibrate: Fit temperatures + +# serve: Dev server | deploy: Stable endpoint +# release: Package checkpoint | weights: Pull weights | publish: Push to HF + .DEFAULT_GOAL := help PRESET ?= 4b-instruct @@ -15,111 +26,101 @@ URL ?= http://localhost:8000 FRESH ?= RESUME ?= -help: ## Show this help - @awk 'BEGIN{FS=":.*?## "} \ - /^##@/ {printf "\n\033[1m%s\033[0m\n", substr($$0, 5)} \ - /^[a-zA-Z0-9_-]+:.*?## / {printf " \033[36m%-12s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) - -##@ Setup +help: + @awk '/^#/ {sub(/^# ?/, ""); print; next} /^$$/ {print; next} {exit}' $(MAKEFILE_LIST) .PHONY: setup setup-train setup-serve setup-modal -setup: ## Install the workspace +setup: uv sync -setup-train: ## Install with the training extras (torch, transformers, peft) +setup-train: uv sync --extra train -setup-serve: ## Install with the serving extras (adds fastapi, uvicorn) +setup-serve: uv sync --extra serve -setup-modal: ## Install with the Modal client +setup-modal: uv sync --extra modal -##@ Develop - .PHONY: test lint fmt check clean -test: ## Run every test +test: uv run pytest -lint: ## Check formatting and lint rules +lint: uv run ruff check . uv run ruff format --check . -fmt: ## Apply formatting and autofixes +fmt: uv run ruff check --fix . uv run ruff format . -check: lint test ## Lint then test — run before pushing +check: lint test -clean: ## Remove caches and build artefacts +clean: find . -type d -name __pycache__ -prune -exec rm -rf {} + rm -rf .pytest_cache .ruff_cache dist build -##@ Data .PHONY: plan check-data data eval-set s1bench -plan: ## Print the H100 training budget without spending it +plan: uv run lev plan --preset $(PRESET) -check-data: ## Contamination guard over a source list (SOURCES=path) +check-data: uv run lev check-data $(SOURCES) -data: ## Build the training mixture locally (LIMIT=rows per source) +data: uv run lev data build --out $(OUT) --limit-per-source $(LIMIT) -eval-set: ## Export the held-out split as levbench task files +eval-set: uv run lev data eval --data $(OUT) --out $(EVAL) -s1bench: ## Export the S1Bench eval subsets as levbench task files +s1bench: uv run lev s1bench export --out $(S1) -##@ Train .PHONY: smoke-local smoke train calibrate -smoke-local: ## Train 0.8B on CPU for a few steps — proves the path without Modal +smoke-local: uv run lev data build --out /tmp/lev-mix --limit-per-source 2000 --n-examples 4000 uv run lev train --preset smoke --data /tmp/lev-mix \ --output-dir /tmp/lev-smoke --max-steps $(STEPS) -smoke: ## Modal: exercise the whole training path on 0.8B (~5 min of H100) +smoke: modal run modal/app.py::smoke -train: ## Modal: the real run; resumes the newest checkpoint (FRESH=1 to start over, RESUME=path) +train: modal run modal/app.py::train --preset $(PRESET) $(if $(FRESH),--fresh,) $(if $(RESUME),--resume $(RESUME),) -calibrate: ## Modal: fit per-bucket temperatures after training +calibrate: modal run modal/app.py::calibrate --preset $(PRESET) -##@ Serve and release .PHONY: serve deploy release weights publish -serve: ## Modal: dev server on an H100 (ephemeral; any `modal run` on this app steals its URL) - LEV_SERVE_PRESET=$(PRESET) modal serve modal/app.py +serve: + LEV_SERVE_PRESET=$(PRESET) $(if $(NAME),LEV_SERVE_RELEASE=$(NAME)) modal serve modal/app.py -deploy: ## Modal: deploy /v1/systemone with a stable URL (PRESET selects the checkpoint) - LEV_SERVE_PRESET=$(PRESET) modal deploy modal/app.py +deploy: + LEV_SERVE_PRESET=$(PRESET) $(if $(NAME),LEV_SERVE_RELEASE=$(NAME)) modal deploy modal/app.py -release: ## Modal: package the newest PRESET checkpoint into a release dir on the volume +release: modal run modal/app.py::export_checkpoint --preset $(PRESET) $(if $(NAME),--name $(NAME),) -weights: ## Pull a packaged release to weights/RELEASE (RELEASE=name, default PRESET) +weights: mkdir -p weights modal volume get --force lev-checkpoints releases/$(RELEASE) weights/ -publish: ## Upload weights/RELEASE to the Hub (REPO=org/name; needs HF_TOKEN) +publish: uv run lev release publish weights/$(RELEASE) --repo $(REPO) -##@ Benchmark .PHONY: bench bench-local sweep snake -bench: s1bench ## Benchmark Jev on the S1Bench subsets (needs TYPESAFE_API_KEY) +bench: s1bench uv run levbench eval --backend jev --tasks $(S1) -bench-local: s1bench ## Benchmark a /v1/systemone server on the S1Bench subsets (URL=server) +bench-local: s1bench uv run levbench eval --backend lev --base-url $(URL) --tasks $(S1) -sweep: ## Measure shared-state batching economics +sweep: uv run levbench sweep --backend jev -snake: ## The decision model plays Snake over /v1/systemone (URL=server; --record in artifacts/) +snake: uv run levbench snake --backend lev --base-url $(URL) --record artifacts/snake/run.jsonl diff --git a/README.md b/README.md index b7d1488..31ae9c2 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,9 @@ lev is an open System One decision model and the harness that measures it. Give Read full blog here: https://interfaze.ai/blog/jev-now-open-source-lev

-

68.9% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.

+

74.3% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.

-

Qwen3.5-4B + LoRA on one H100. On the six subsets the public S1Bench board completed, level with reflex-4b and behind only Jev and three open models of 26B–35B.
Accuracy receipt · Speed receipt

+

Qwen3.5-4B + LoRA on one H100. On the six subsets the public S1Bench board completed, behind only Jev and two open models of 26B and up, ahead of reflex-4b and a 35B model.
Accuracy receipt · Speed receipt

Quickstart · Why it works · What mattered · Benchmark harness · Snake demo · Train it · Decision log · Model card

@@ -22,27 +22,27 @@ lev is an open System One decision model and the harness that measures it. Give | subset | task | **lev** | Jev | always the most common label | | -------------------- | ------------------------------ | --------: | --------: | ---------------------------: | -| vitaminc-dev | claim verification | 0.668 | **0.801** | 0.503 | -| massive-en-US | intent routing, 18 scenarios | 0.857 | **0.874** | 0.163 | -| massive-de-DE | intent routing, German | 0.823 | **0.871** | 0.163 | -| boolq | yes/no reading comprehension | 0.827 | **0.893** | 0.580 | -| squad2 | answerability | 0.813 | **0.836** | 0.502 | -| paws | adversarial paraphrase | 0.776 | **0.900** | 0.516 | -| multinli | natural language inference | **0.890** | 0.836 | 0.361 | -| civil_comments | toxicity | 0.760 | **0.803** | 0.893 | -| aegis2 | safety moderation | 0.800 | **0.804** | 0.568 | -| helpsteer2 | helpfulness, 5 levels | **0.386** | 0.341 | 0.422 | -| summeval-relevance | summary relevance, 5 levels | 0.358 | 0.358 | 0.458 | -| summeval-consistency | summary faithfulness, 5 levels | 0.271 | **0.812** | 0.840 | -| pubmedqa | biomedical yes/no/maybe | 0.732 | **0.764** | 0.532 | -| **macro** | | 0.689 | **0.761** | | - -lev and Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap. - -- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on multinli and helpsteer2 are inside that. -- **Where Jev is clearly ahead:** the minimal-edit pairs (paws −12.4, vitaminc −13.3) and summeval-consistency (−54.1), where lev rates most fully faithful summaries one level low. Fine-tuning introduced that: the untuned backbone scores 0.826. +| vitaminc-dev | claim verification | 0.726 | **0.801** | 0.503 | +| massive-en-US | intent routing, 18 scenarios | 0.869 | **0.874** | 0.163 | +| massive-de-DE | intent routing, German | 0.829 | **0.871** | 0.163 | +| boolq | yes/no reading comprehension | 0.853 | **0.893** | 0.580 | +| squad2 | answerability | 0.819 | **0.836** | 0.502 | +| paws | adversarial paraphrase | 0.844 | **0.900** | 0.516 | +| multinli | natural language inference | **0.849** | 0.836 | 0.361 | +| civil_comments | toxicity | 0.783 | **0.803** | 0.893 | +| aegis2 | safety moderation | **0.836** | 0.804 | 0.568 | +| helpsteer2 | helpfulness, 5 levels | **0.361** | 0.341 | 0.422 | +| summeval-relevance | summary relevance, 5 levels | **0.371** | 0.358 | 0.458 | +| summeval-consistency | summary faithfulness, 5 levels | 0.806 | **0.812** | 0.840 | +| pubmedqa | biomedical yes/no/maybe | 0.716 | **0.764** | 0.532 | +| **macro** | | 0.743 | **0.761** | | + +lev and Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap. The adapter scale was chosen by leave-one-family-out over these subsets; scored only on families held out of that choice, it estimates 0.728 ([FINDINGS §19](docs/FINDINGS.md#19-the-adapter-scale-is-an-out-of-distribution-knob-and-it-is-worth-39-macro-points)). + +- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on aegis2, helpsteer2, multinli and summeval-relevance are all inside that. +- **Where Jev is clearly ahead:** the minimal-edit pairs (vitaminc −7.5, paws −5.6). The adapter runs at a quarter of its trained strength, which narrowed those gaps from −13.3 and −12.4 and closed summeval-consistency's from −54.1 to −0.6: at full strength it rated most fully faithful summaries one level low. - **Constant baselines:** on civil_comments (89% not toxic) and the three 5-level rating subsets, always answering the most common label beats both models. -- **Calibration:** mean ECE 0.115 for lev, 0.091 for Jev; lev is better calibrated on 5 of 13. +- **Calibration:** mean ECE 0.097 for lev, 0.091 for Jev; lev is better calibrated on 6 of 13. summeval-consistency is the outlier, under-confident at 0.303. - **Speed:** 69 ms is lev's engine compute for a short request inside the container. End to end from a laptop, Jev answered in 335–346 ms median and lev on one Modal H100 in 414–654 ms across two runs. [Every run, per-subset breakdowns and what each run changed →](docs/FINDINGS.md) @@ -166,7 +166,7 @@ flowchart LR style O fill:#2a78d6,stroke:#2a78d6,color:#ffffff ``` -Each option gets a short code, and the answer is read from the next-token logits over those codes. Choices are read in two option orders and averaged, which cancels position bias. Codes that would split into two tokens are skipped, so this reaches several hundred options. +Each option gets a short code, and the answer is read from the next-token logits over those codes. Choices are read in two option orders and averaged, which cancels position bias. Score levels stay in their trained order unless the server runs with `--score-order-average` (off by default, [ADR-029](docs/DECISIONS.md#adr-029--score-is-read-in-one-order-measure-the-position-effect-and-offer-averaging-off-by-default)). Codes that would split into two tokens are skipped, so this reaches several hundred options. **The router is the part nobody else had.** Every other implementation picks one readout family and hits a wall when options stop fitting in single tokens: it caps the option count or rejects the request. lev routes those questions to a small learned head that matches the state against each option's text, so there is no fixed option ceiling. [ADR-005 →](docs/DECISIONS.md#adr-005--dual-mode-readout-the-differentiator) @@ -265,7 +265,7 @@ The data build refuses any source that resolves to one of the 13 S1Bench subsets ```text packages/ - lev/ the model — schema, prompt layouts, router, readouts, calibration, loader, server, S1Bench export + lev/ the model — schema, prompt layouts, router, readouts, calibration, loader, server, presentation checks, S1Bench export levbench/ the harness — accuracy, ECE, cost, latency, batching economics, Snake modal/ app.py — train / calibrate / serve / check a release on one H100 hf/ README.md — the Hub model card, and the charts it shows @@ -276,7 +276,7 @@ docs/ ARCHITECTURE — the design, and every model it was derived fro STATUS — what is built and how far it has been run SETUP, TRAINING — local and Modal, the pipeline step by step charts/ — the lev-vs-Jev comparison page, built from the logs -data/ sample state + the S1Bench snapshot the analysis rests on +data/ sample state, the S1Bench snapshot the analysis rests on, the presentation report ``` ## Prior work diff --git a/data/adr-029-summary.json b/data/adr-029-summary.json new file mode 100644 index 0000000..da41b22 --- /dev/null +++ b/data/adr-029-summary.json @@ -0,0 +1,930 @@ +{ + "about": "Summary of the ADR-029 runs after review (2026-10-04/05): the numbers behind its tables. interfaze-ai/lev (HF 7bdc748; weights and calibration identical to f8ef711, only README.md changed), bf16, one RTX 5090, lev branch rebased on main 8ca94b7 (#4).", + "benches": { + "bench_en": "sokudan data/bench_en.jsonl, 290 states (English)", + "bench_ja": "sokudan data/bench_ja.jsonl, 300 states (Japanese, outside Lev's target language; reference only)" + }, + "shipped_temperatures": { + "choice:B:large": 0.6333, + "choice:B": 0.6333, + "noul:A": 2.3333, + "score:A": 2.8033, + "choice:A": 1.7667, + "choice:A:small": 1.7898, + "choice:A:mid": 1.6086, + "choice:A:large": 1.6641 + }, + "identical_one_read": { + "bench_en": { + "score": { + "calibrated": -0.3323, + "leave_one_out": [ + -0.3364, + -0.3298 + ], + "raw": -0.9317, + "by_config_calibrated": { + "moderate|3": -0.1191, + "moderate|4": -0.2373, + "moderate|5": -0.3198, + "a request|3": -0.3249, + "a request|4": -0.4614, + "a request|5": -0.5315 + }, + "by_config_raw": { + "moderate|3": -0.3338, + "moderate|4": -0.6652, + "moderate|5": -0.8966, + "a request|3": -0.9108, + "a request|4": -1.2935, + "a request|5": -1.49 + }, + "max_abs_diff_across_modes": 0.0 + }, + "choice": { + "calibrated": -0.454, + "leave_one_out": [ + -0.4556, + -0.4515 + ], + "raw": -0.8125, + "by_config_calibrated": { + "a team|3": -0.5365, + "a team|4": -0.534, + "a team|5": -0.5335, + "a request|3": -0.323, + "a request|4": -0.3803, + "a request|5": -0.4165 + }, + "by_config_raw": { + "a team|3": -0.9601, + "a team|4": -0.9558, + "a team|5": -0.9548, + "a request|3": -0.5781, + "a request|4": -0.6807, + "a request|5": -0.7455 + }, + "max_abs_diff_across_modes": 0.0 + } + }, + "bench_ja": { + "score": { + "calibrated": -0.5181, + "leave_one_out": [ + -0.5222, + -0.5151 + ], + "raw": -1.4524, + "by_config_calibrated": { + "中程度|3": -0.1889, + "中程度|4": -0.2989, + "中程度|5": -0.3897, + "依頼|3": -0.5875, + "依頼|4": -0.7656, + "依頼|5": -0.8779 + }, + "by_config_raw": { + "中程度|3": -0.5294, + "中程度|4": -0.838, + "中程度|5": -1.0926, + "依頼|3": -1.647, + "依頼|4": -2.1462, + "依頼|5": -2.4609 + }, + "max_abs_diff_across_modes": 0.0 + }, + "choice": { + "calibrated": -0.6602, + "leave_one_out": [ + -0.6624, + -0.6577 + ], + "raw": -1.1817, + "by_config_calibrated": { + "部署|3": -0.611, + "部署|4": -0.7262, + "部署|5": -0.7265, + "依頼|3": -0.5278, + "依頼|4": -0.6493, + "依頼|5": -0.7206 + }, + "by_config_raw": { + "部署|3": -1.0936, + "部署|4": -1.2998, + "部署|5": -1.3002, + "依頼|3": -0.9447, + "依頼|4": -1.162, + "依頼|5": -1.2897 + }, + "max_abs_diff_across_modes": 0.0 + } + } + }, + "first_slot": { + "bench_en": { + "off": { + "score": { + "metric": 0.2646, + "leave_one_out": [ + 0.2641, + 0.2655 + ], + "argmax_by_slot": [ + 460.3333, + 595.8333, + 683.8333 + ], + "consistent": 0.7138, + "decisions": 1740 + }, + "choice": { + "metric": 0.331, + "leave_one_out": [ + 0.331, + 0.3319 + ], + "argmax_by_slot": [ + 576.0, + 588.0, + 576.0 + ], + "consistent": 0.9862, + "decisions": 1740 + } + }, + "reversed": { + "score": { + "metric": 0.3191, + "leave_one_out": [ + 0.3184, + 0.3202 + ], + "argmax_by_slot": [ + 555.1667, + 575.6667, + 609.1667 + ], + "consistent": 0.8724, + "decisions": 1740 + }, + "choice": { + "metric": 0.3317, + "leave_one_out": [ + 0.3317, + 0.3323 + ], + "argmax_by_slot": [ + 577.1667, + 585.6667, + 577.1667 + ], + "consistent": 0.9793, + "decisions": 1740 + } + }, + "cyclic": { + "score": { + "metric": 0.3101, + "leave_one_out": [ + 0.3091, + 0.3111 + ], + "argmax_by_slot": [ + 539.5, + 590.5, + 610.0 + ], + "consistent": 0.8931, + "decisions": 1740 + }, + "choice": { + "metric": 0.332, + "leave_one_out": [ + 0.3314, + 0.3328 + ], + "argmax_by_slot": [ + 577.6667, + 584.6667, + 577.6667 + ], + "consistent": 0.9828, + "decisions": 1740 + } + } + }, + "bench_ja": { + "off": { + "score": { + "metric": 0.2422, + "leave_one_out": [ + 0.2414, + 0.243 + ], + "argmax_by_slot": [ + 436.0, + 631.5, + 732.5 + ], + "consistent": 0.6267, + "decisions": 1800 + }, + "choice": { + "metric": 0.3311, + "leave_one_out": [ + 0.3305, + 0.3322 + ], + "argmax_by_slot": [ + 596.0, + 608.0, + 596.0 + ], + "consistent": 0.9733, + "decisions": 1800 + } + }, + "reversed": { + "score": { + "metric": 0.3142, + "leave_one_out": [ + 0.3135, + 0.3152 + ], + "argmax_by_slot": [ + 565.5, + 606.0, + 628.5 + ], + "consistent": 0.87, + "decisions": 1800 + }, + "choice": { + "metric": 0.3306, + "leave_one_out": [ + 0.3305, + 0.3317 + ], + "argmax_by_slot": [ + 595.0, + 610.0, + 595.0 + ], + "consistent": 0.9767, + "decisions": 1800 + } + }, + "cyclic": { + "score": { + "metric": 0.2994, + "leave_one_out": [ + 0.2993, + 0.3004 + ], + "argmax_by_slot": [ + 539.0, + 619.0, + 642.0 + ], + "consistent": 0.8533, + "decisions": 1800 + }, + "choice": { + "metric": 0.33, + "leave_one_out": [ + 0.3294, + 0.3311 + ], + "argmax_by_slot": [ + 594.0, + 612.0, + 594.0 + ], + "consistent": 0.9733, + "decisions": 1800 + } + } + } + }, + "packed_max_abs_diff": { + "bench_en": { + "off": 0.0375, + "reversed": 0.0375, + "cyclic": 0.0375 + }, + "bench_ja": { + "off": 0.0408, + "reversed": 0.0408, + "cyclic": 0.0408 + } + }, + "bench_accuracy": { + "bench_en": { + "off": { + "score_acc": 0.5828, + "score_rps": 0.1504, + "score_ece15": 0.1238, + "score_ece15_equal_mass": 0.1545, + "score_mean_confidence": 0.6814, + "choice_acc": 0.9172, + "noul_acc": 0.9379, + "noul_auroc": 0.9778, + "ms_per_request": 156.5 + }, + "reversed": { + "score_acc": 0.6517, + "score_rps": 0.1275, + "score_ece15": 0.1013, + "score_ece15_equal_mass": 0.1066, + "score_mean_confidence": 0.6594, + "choice_acc": 0.9172, + "noul_acc": 0.9379, + "noul_auroc": 0.978, + "ms_per_request": 179.3 + }, + "cyclic": { + "score_acc": 0.6414, + "score_rps": 0.1249, + "score_ece15": 0.0601, + "score_ece15_equal_mass": 0.0912, + "score_mean_confidence": 0.6453, + "choice_acc": 0.9172, + "noul_acc": 0.9379, + "noul_auroc": 0.9784, + "ms_per_request": 210.2 + } + }, + "bench_ja": { + "off": { + "score_acc": 0.6433, + "score_rps": 0.1384, + "score_ece15": 0.0934, + "score_ece15_equal_mass": 0.113, + "score_mean_confidence": 0.7184, + "choice_acc": 0.8867, + "noul_acc": 0.8167, + "noul_auroc": 0.889, + "ms_per_request": 149.2 + }, + "reversed": { + "score_acc": 0.73, + "score_rps": 0.1079, + "score_ece15": 0.0588, + "score_ece15_equal_mass": 0.1072, + "score_mean_confidence": 0.6829, + "choice_acc": 0.8867, + "noul_acc": 0.8133, + "noul_auroc": 0.8877, + "ms_per_request": 169.9 + }, + "cyclic": { + "score_acc": 0.7167, + "score_rps": 0.108, + "score_ece15": 0.0765, + "score_ece15_equal_mass": 0.0879, + "score_mean_confidence": 0.6813, + "choice_acc": 0.8867, + "noul_acc": 0.8133, + "noul_auroc": 0.8876, + "ms_per_request": 197.3 + } + } + }, + "score_temperature_refit": { + "bench_en": { + "split": "seed 0, stratified by gold level, alternate halves", + "off": { + "n_fit": 146, + "n_eval": 144, + "T_shipped": 2.8033, + "T_refit": 3.9628, + "held_out_shipped": { + "acc": 0.5903, + "rps": 0.152, + "ece15": 0.1296, + "ece15_equal_mass": 0.1683, + "negative_log_likelihood": 0.9348 + }, + "held_out_refit": { + "acc": 0.5903, + "rps": 0.1466, + "ece15": 0.108, + "ece15_equal_mass": 0.134, + "negative_log_likelihood": 0.9087 + } + }, + "reversed": { + "n_fit": 146, + "n_eval": 144, + "T_shipped": 2.8033, + "T_refit": 3.1595, + "held_out_shipped": { + "acc": 0.6736, + "rps": 0.1264, + "ece15": 0.1108, + "ece15_equal_mass": 0.1453, + "negative_log_likelihood": 0.8388 + }, + "held_out_refit": { + "acc": 0.6736, + "rps": 0.1263, + "ece15": 0.1193, + "ece15_equal_mass": 0.1668, + "negative_log_likelihood": 0.8381 + } + }, + "cyclic": { + "n_fit": 146, + "n_eval": 144, + "T_shipped": 2.8033, + "T_refit": 2.9175, + "held_out_shipped": { + "acc": 0.6597, + "rps": 0.1238, + "ece15": 0.0775, + "ece15_equal_mass": 0.1387, + "negative_log_likelihood": 0.8279 + }, + "held_out_refit": { + "acc": 0.6597, + "rps": 0.1238, + "ece15": 0.0765, + "ece15_equal_mass": 0.1382, + "negative_log_likelihood": 0.8284 + } + } + }, + "bench_ja": { + "split": "seed 0, stratified by gold level, alternate halves", + "off": { + "n_fit": 151, + "n_eval": 149, + "T_shipped": 2.8033, + "T_refit": 3.5967, + "held_out_shipped": { + "acc": 0.6577, + "rps": 0.1321, + "ece15": 0.0896, + "ece15_equal_mass": 0.1275, + "negative_log_likelihood": 0.7807 + }, + "held_out_refit": { + "acc": 0.6577, + "rps": 0.13, + "ece15": 0.0772, + "ece15_equal_mass": 0.0914, + "negative_log_likelihood": 0.7792 + } + }, + "reversed": { + "n_fit": 151, + "n_eval": 149, + "T_shipped": 2.8033, + "T_refit": 2.3927, + "held_out_shipped": { + "acc": 0.7584, + "rps": 0.1015, + "ece15": 0.0888, + "ece15_equal_mass": 0.1219, + "negative_log_likelihood": 0.641 + }, + "held_out_refit": { + "acc": 0.7584, + "rps": 0.1008, + "ece15": 0.0611, + "ece15_equal_mass": 0.1142, + "negative_log_likelihood": 0.6267 + } + }, + "cyclic": { + "n_fit": 151, + "n_eval": 149, + "T_shipped": 2.8033, + "T_refit": 2.3974, + "held_out_shipped": { + "acc": 0.7517, + "rps": 0.1013, + "ece15": 0.0681, + "ece15_equal_mass": 0.0959, + "negative_log_likelihood": 0.6436 + }, + "held_out_refit": { + "acc": 0.7517, + "rps": 0.1006, + "ece15": 0.0865, + "ece15_equal_mass": 0.0909, + "negative_log_likelihood": 0.6286 + } + } + } + }, + "off_vs_main_8ca94b7": { + "bench_en": { + "responses": 40, + "identical": true + }, + "bench_ja": { + "responses": 40, + "identical": true + } + }, + "large_request_without_averaging": { + "what": "20 Choice + 40 Score questions per request, default settings, through lev.batcher (max_batch_tokens 16384), first 3 bench_en states: every forward's rows and width", + "requests": [ + { + "item_id": "bench_en-0000", + "rows": 80, + "padded_tokens": 26800, + "forwards_main": [ + { + "rows": 80, + "width": 335 + } + ], + "forwards_branch": [ + { + "rows": 80, + "width": 335 + } + ], + "answers_identical_to_main": true + }, + { + "item_id": "bench_en-0001", + "rows": 80, + "padded_tokens": 15760, + "forwards_main": [ + { + "rows": 80, + "width": 197 + } + ], + "forwards_branch": [ + { + "rows": 80, + "width": 197 + } + ], + "answers_identical_to_main": true + }, + { + "item_id": "bench_en-0002", + "rows": 80, + "padded_tokens": 35680, + "forwards_main": [ + { + "rows": 80, + "width": 446 + } + ], + "forwards_branch": [ + { + "rows": 80, + "width": 446 + } + ], + "answers_identical_to_main": true + } + ] + }, + "averaged_token_budget_rtx5090": { + "what": "the largest Score-averaged request max_batch_tokens (16384) admits, by row length: averaged Score questions topped up with one-row Noul; one question more is refused in every case. The value is the batcher's budget; this only checks it.", + "results": [ + { + "state_tokens": 350, + "mode": "reversed", + "score_questions": 17, + "noul_questions": 0, + "rows": 34, + "width": 474, + "padded_tokens": 16116, + "one_more_refused": true, + "seconds": 1.686, + "peak_allocated_gib": 12.1, + "peak_reserved_gib": 14.62 + }, + { + "state_tokens": 350, + "mode": "cyclic", + "score_questions": 11, + "noul_questions": 1, + "rows": 34, + "width": 474, + "padded_tokens": 16116, + "one_more_refused": true, + "seconds": 1.653, + "peak_allocated_gib": 12.18, + "peak_reserved_gib": 14.63 + }, + { + "state_tokens": 2000, + "mode": "reversed", + "score_questions": 3, + "noul_questions": 1, + "rows": 7, + "width": 2124, + "padded_tokens": 14868, + "one_more_refused": true, + "seconds": 1.539, + "peak_allocated_gib": 11.7, + "peak_reserved_gib": 14.03 + }, + { + "state_tokens": 2000, + "mode": "cyclic", + "score_questions": 2, + "noul_questions": 1, + "rows": 7, + "width": 2124, + "padded_tokens": 14868, + "one_more_refused": true, + "seconds": 1.527, + "peak_allocated_gib": 11.7, + "peak_reserved_gib": 14.03 + }, + { + "state_tokens": 4000, + "mode": "reversed", + "score_questions": 1, + "noul_questions": 1, + "rows": 3, + "width": 4124, + "padded_tokens": 12372, + "one_more_refused": true, + "seconds": 1.387, + "peak_allocated_gib": 11.09, + "peak_reserved_gib": 13.06 + }, + { + "state_tokens": 4000, + "mode": "cyclic", + "score_questions": 1, + "noul_questions": 0, + "rows": 3, + "width": 4124, + "padded_tokens": 12372, + "one_more_refused": true, + "seconds": 1.998, + "peak_allocated_gib": 16.29, + "peak_reserved_gib": 19.84 + }, + { + "state_tokens": 5300, + "mode": "cyclic", + "score_questions": 1, + "noul_questions": 0, + "rows": 3, + "width": 5423, + "padded_tokens": 16269, + "one_more_refused": true, + "seconds": 2.776, + "peak_allocated_gib": 21.72, + "peak_reserved_gib": 25.83 + }, + { + "state_tokens": 8000, + "mode": "reversed", + "score_questions": 1, + "noul_questions": 0, + "rows": 2, + "width": 8124, + "padded_tokens": 16248, + "one_more_refused": true, + "seconds": 18.927, + "peak_allocated_gib": 27.72, + "peak_reserved_gib": 31.68 + } + ], + "averaging_off_one_choice_8000_tokens": { + "what": "averaging off (default), one Choice question read in two orders", + "state_tokens": 8000, + "rows": 2, + "width": 8109, + "padded_tokens": 16218, + "seconds": 30.932, + "peak_allocated_gib": 27.66, + "peak_reserved_gib": 31.63 + } + }, + "rows_without_a_limit_rtx5090": { + "what": "one system_one request of R single-order Score rows; peak CUDA memory in GiB (bf16 weights alone are ~8.1). Past ~31 GiB reserved, Windows spills into shared system memory instead of raising out-of-memory, and the request slows 5-10x.", + "bench_row_tokens": { + "bench_en": { + "n": 290, + "median": 259, + "p95": 449, + "max": 509 + }, + "bench_ja": { + "n": 300, + "median": 249, + "p95": 419, + "max": 526 + } + }, + "results": [ + { + "state_tokens": 350, + "rows": 8, + "width": 474, + "padded_tokens": 3792, + "seconds": 0.376, + "peak_allocated_gib": 9.05, + "above_weights_gib": 0.94, + "peak_reserved_gib": 9.66 + }, + { + "state_tokens": 350, + "rows": 16, + "width": 474, + "padded_tokens": 7584, + "seconds": 0.782, + "peak_allocated_gib": 9.99, + "above_weights_gib": 1.88, + "peak_reserved_gib": 11.21 + }, + { + "state_tokens": 350, + "rows": 24, + "width": 474, + "padded_tokens": 11376, + "seconds": 1.194, + "peak_allocated_gib": 10.93, + "above_weights_gib": 2.81, + "peak_reserved_gib": 12.71 + }, + { + "state_tokens": 350, + "rows": 32, + "width": 474, + "padded_tokens": 15168, + "seconds": 1.598, + "peak_allocated_gib": 11.86, + "above_weights_gib": 3.75, + "peak_reserved_gib": 14.24 + }, + { + "state_tokens": 350, + "rows": 48, + "width": 474, + "padded_tokens": 22752, + "seconds": 2.425, + "peak_allocated_gib": 13.74, + "above_weights_gib": 5.63, + "peak_reserved_gib": 17.29 + }, + { + "state_tokens": 350, + "rows": 64, + "width": 474, + "padded_tokens": 30336, + "seconds": 3.277, + "peak_allocated_gib": 15.61, + "above_weights_gib": 7.5, + "peak_reserved_gib": 20.34 + }, + { + "state_tokens": 350, + "rows": 96, + "width": 474, + "padded_tokens": 45504, + "seconds": 5.254, + "peak_allocated_gib": 19.36, + "above_weights_gib": 11.25, + "peak_reserved_gib": 26.44 + }, + { + "state_tokens": 700, + "rows": 8, + "width": 824, + "padded_tokens": 6592, + "seconds": 0.696, + "peak_allocated_gib": 9.65, + "above_weights_gib": 1.54, + "peak_reserved_gib": 11.18 + }, + { + "state_tokens": 700, + "rows": 16, + "width": 824, + "padded_tokens": 13184, + "seconds": 1.389, + "peak_allocated_gib": 11.19, + "above_weights_gib": 3.07, + "peak_reserved_gib": 14.49 + }, + { + "state_tokens": 700, + "rows": 24, + "width": 824, + "padded_tokens": 19776, + "seconds": 2.098, + "peak_allocated_gib": 12.73, + "above_weights_gib": 4.61, + "peak_reserved_gib": 17.68 + }, + { + "state_tokens": 700, + "rows": 32, + "width": 824, + "padded_tokens": 26368, + "seconds": 2.813, + "peak_allocated_gib": 14.26, + "above_weights_gib": 6.15, + "peak_reserved_gib": 21.05 + }, + { + "state_tokens": 700, + "rows": 48, + "width": 824, + "padded_tokens": 39552, + "seconds": 4.317, + "peak_allocated_gib": 17.33, + "above_weights_gib": 9.22, + "peak_reserved_gib": 27.51 + }, + { + "state_tokens": 1000, + "rows": 8, + "width": 1124, + "padded_tokens": 8992, + "seconds": 0.947, + "peak_allocated_gib": 10.39, + "above_weights_gib": 2.28, + "peak_reserved_gib": 12.9 + }, + { + "state_tokens": 1000, + "rows": 16, + "width": 1124, + "padded_tokens": 17984, + "seconds": 1.946, + "peak_allocated_gib": 12.67, + "above_weights_gib": 4.56, + "peak_reserved_gib": 17.62 + }, + { + "state_tokens": 1000, + "rows": 24, + "width": 1124, + "padded_tokens": 26976, + "seconds": 2.925, + "peak_allocated_gib": 14.95, + "above_weights_gib": 6.84, + "peak_reserved_gib": 22.38 + }, + { + "state_tokens": 1000, + "rows": 32, + "width": 1124, + "padded_tokens": 35968, + "seconds": 3.916, + "peak_allocated_gib": 17.22, + "above_weights_gib": 9.11, + "peak_reserved_gib": 27.12 + }, + { + "state_tokens": 2000, + "rows": 8, + "width": 2124, + "padded_tokens": 16992, + "seconds": 1.956, + "peak_allocated_gib": 14.71, + "above_weights_gib": 6.6, + "peak_reserved_gib": 19.12 + }, + { + "state_tokens": 2000, + "rows": 16, + "width": 2124, + "padded_tokens": 33984, + "seconds": 12.348, + "peak_allocated_gib": 21.29, + "above_weights_gib": 13.18, + "peak_reserved_gib": 30.12 + }, + { + "state_tokens": 4000, + "rows": 8, + "width": 4124, + "padded_tokens": 32992, + "seconds": 42.186, + "peak_allocated_gib": 29.8, + "above_weights_gib": 21.69, + "peak_reserved_gib": 39.3 + } + ], + "first_run_past_the_card": [ + { + "state_tokens": 350, + "rows": 128, + "width": 474, + "padded_tokens": 60672, + "seconds": 59.979, + "peak_allocated_gib": 23.11, + "above_weights_gib": 15.0, + "peak_reserved_gib": 32.53 + } + ] + } +} diff --git a/data/presentation-checks.json b/data/presentation-checks.json new file mode 100644 index 0000000..748a9f5 --- /dev/null +++ b/data/presentation-checks.json @@ -0,0 +1,91 @@ +{ + "about": "Committed report for `lev presentation-checks --compare` (ADR-029). One run: interfaze-ai/lev (HF 7bdc748), bf16, one RTX 5090, the shipped calibration, score_order_average off, kinds score and choice, lang en, 2026-10-05. The identical check reads one order.", + "checkpoint": "interfaze-ai/lev", + "states": "sokudan bench_en.jsonl (github.com/hiroki-abe-58/sokudan, data/bench_en.jsonl, CC BY 4.0), 290 states", + "n_states": 290, + "lang": "en", + "raw": false, + "kinds": [ + "score", + "choice" + ], + "score_order_average": "off", + "max_rows": 32, + "checkpoint_revision": "7bdc748dffebd85b57ee0dbea8f994c6354fed31", + "dtype": "torch.bfloat16", + "packed_max_abs_diff": 0.0375, + "checks": { + "score": { + "identical": { + "metric": -0.3323, + "leave_one_out": [ + -0.3364, + -0.3298 + ], + "gate": -0.2, + "passed": false, + "read": "single order", + "by_config": { + "moderate|3": -0.1191, + "moderate|4": -0.2373, + "moderate|5": -0.3198, + "a request|3": -0.3249, + "a request|4": -0.4614, + "a request|5": -0.5315 + } + }, + "first_slot": { + "metric": 0.2646, + "leave_one_out": [ + 0.2641, + 0.2655 + ], + "gate": 0.15, + "passed": true, + "decisions": 1740, + "argmax_by_slot": [ + 460.3333, + 595.8333, + 683.8333 + ], + "consistent": 0.7138 + } + }, + "choice": { + "identical": { + "metric": -0.454, + "leave_one_out": [ + -0.4556, + -0.4515 + ], + "gate": -0.2, + "passed": false, + "read": "single order", + "by_config": { + "a team|3": -0.5365, + "a team|4": -0.534, + "a team|5": -0.5335, + "a request|3": -0.323, + "a request|4": -0.3803, + "a request|5": -0.4165 + } + }, + "first_slot": { + "metric": 0.331, + "leave_one_out": [ + 0.331, + 0.3319 + ], + "gate": 0.15, + "passed": true, + "decisions": 1740, + "argmax_by_slot": [ + 576.0, + 588.0, + 576.0 + ], + "consistent": 0.9862 + } + } + } +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 580044c..9772bdc 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -107,7 +107,7 @@ This is our plan's §14 ("encode the question, don't build per-question heads") ### 3.6 Training objectives — from `NanoJev` and `decider` - NanoJev: CE / Brier / paired-proper-reward variants, with exact gradient checks. -- decider: cross-entropy fine-tune with **random layout per example** and **abstain augmentation**; evaluation reports accuracy / NLL / Brier / ECE / AURC / selective accuracy per task; v10 adds **RL with a proper-score belief reward** — the open analogue of RLCD. +- decider: cross-entropy fine-tune with **random layout per example** and **abstain augmentation**; evaluation reports accuracy / negative_log_likelihood / Brier / ECE / AURC / selective accuracy per task; v10 adds **RL with a proper-score belief reward** — the open analogue of RLCD. Supervised proper-scoring first, RL only after. That vindicates the plan's §17 ordering. @@ -201,6 +201,8 @@ LoRA r32 on `q,k,v,o,gate,up,down` ≈ 25–40 M trainable parameters. **The Fam - they get **separate fitted temperatures** — one global scalar would be wrong for at least one of them; - on option sets where **both** are valid we measure agreement. Material disagreement means the router is a correctness hazard, not merely a capacity switch. This is an explicit eval, not an assumption. +**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `order_average=False` reads every question once. A request that Score averaging grew is refused in `prepare`, before any GPU work, when its padded tokens pass `max_batch_tokens` (16384, the batcher's budget; the server passes its own value to the engine); one without averaged Score rows never is, since the batcher runs an oversized request alone (ADR-030). `lev presentation-checks` measures the position bias of one read, the bias averaging has to remove (ADR-029). + ### 5.4 Output types | Type | Mode A | Mode B | @@ -235,7 +237,7 @@ Three splits, not two. Temperature is fitted on a split disjoint from **both** t 1. Train with a proper scoring rule (cross-entropy + Brier). 2. **Abstain augmentation** (decider): examples where the answer is not determinable from the state, teaching the model to spread mass rather than guess confidently. 3. Post-hoc temperature fit on the calibration split. -4. Report accuracy, NLL, Brier and **ECE** per source, with and without the temperature. +4. Report accuracy, negative_log_likelihood, Brier and **ECE** per source, with and without the temperature. Evidence this is worth the effort: the untuned Qwen3-8B (`qwen3-8b-full`) sits at **ECE 0.4252**; reflex, Qwen3.5-4B plus one fitted scalar, reaches **0.0849**. Jev is **0.0764**. **A single scalar is most of the gap.** diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index dff4ea3..008fd2d 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -482,7 +482,7 @@ Both are flat in the number of questions, which says the forward is bound by ker With `causal_conv1d` present (the image now builds it against `torch 2.14.0+cu130`): fork 140-156 ms, single 65-75 ms -- about 10% each. That the kernel bought so little is the diagnosis confirmed from a second angle: the time is in launching ~200 kernels twice, not in any one of them. What removes launches is CUDA-graph replay, so the engine gains `EngineConfig.compile` (`torch.compile(mode="reduce-overhead", dynamic=True)`) with inputs padded to shape buckets -- width to 32 tokens, rows to a power of two -- so the set of recorded graphs stays small, and a `warmup()` the server runs at startup so no request pays a capture. Padding changes no real row's logits: right padding is masked and rows are independent. The GPU forward is taken under a lock, because graph replay is not thread-safe and the server now accepts concurrent requests; parsing, tokenising and the network overlap around it. -**Compile: measured, rejected.** On the same container, eager single-forward ran 86–94 ms and the compiled path 103–137 ms, after a 104 s warmup; dynamo hit its recompile limit (8) on the hybrid's linear-attention layers, so much of the forward stayed eager with extra guard overhead, and the Mode B shape then failed with a CUDA-graphs output-buffer overwrite when hidden states from one replay were read after the next. The flag stays (`LEV_SERVE_COMPILE=1`) so the measurement can be repeated on a future transformers/FLA release, and defaults off. Getting decider's 8 ms would mean a static-shape engine built for this model, not `torch.compile` over the stock one. +**Compile: measured, rejected.** On the same container, eager single-forward ran 86–94 ms and the compiled path 103–137 ms, after a 104 s warmup; `torch.compile` cannot reach this model's dominant cost at all: FLA decorates `chunk_gated_delta_rule` and `fused_gdn_gate` with `@torch.compiler.disable` (`fla/ops/gated_delta_rule/chunk.py`, `gate.py`), so each of the 24 GDN layers is a permanent graph break and stays eager at full launch cost whatever the mode. What is left to capture is the 8 attention layers, the MLPs and the norms, while every partition boundary copies inputs to static buffers -- overhead without the win. Dynamo's `recompile_limit` (8, per code object, so all 24 GDN layers share one budget) compounded it, and the Mode B shape then failed with a CUDA-graphs output-buffer overwrite when hidden states from one replay were read after the next. `dynamic=True` was not the cause: it does not veto graph capture. The flag stays (`LEV_SERVE_COMPILE=1`) so the measurement can be repeated once FLA lifts the decorator ([fla-org RFC #1155](https://github.com/fla-org/flash-linear-attention/issues/1155) tracks graph capture for the GDN paths), and defaults off. Getting decider's 8 ms means manual `torch.cuda.CUDAGraph` capture over the eager forward, one graph per padded length bucket, warmed eagerly first so the Triton autotune choice is fixed before capture -- not `torch.compile` over the stock model. Also decided here, from the same session's measurements (FINDINGS.md §13): the image builds `causal_conv1d` from a CUDA devel base, since no wheel exists for `torch 2.14.0+cu130` and every log said the reference path was in use; and the serving path was verified correct end to end -- in-distribution accuracy over HTTP matches the collator-path `evaluate` on every source -- so no inference-side error is hiding behind the S1Bench gap. @@ -581,6 +581,95 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode --- +## ADR-029 — Score is read in one order; measure the position effect, and offer averaging off by default + +**Open.** Proposed in [#1](https://github.com/Abhinavexists/lev/issues/1); the default is the maintainer's call. + +**Problem.** Choice and binary Noul are read in two orders and averaged, which cancels the preference for the first letter (ADR-020). Score is not: its levels keep the low-to-high order training uses, because reversing them measured -2.4 points on helpsteer2, inside sampling noise, on a prompt training never produces (FINDINGS §12). What that leaves was not measured. On `interfaze-ai/lev` with Score questions over the 290 states of sokudan's `bench_en`, an identical-option control (every level the same text, so levels differ only by position and code) put slot 0 at -0.332 below the mean in calibrated log-probability and -0.931 in raw scores, and the first slot won 0.270 of the decisions over all six orders of three levels, where an order-free readout is at 1/3 (#1). The check is the one NandhaKishorM/laya#259 added to Laya, whose gate is -0.20 on raw scores. + +**Options.** + +- (a) Average Score over orders by default: `reversed` (2 rows, as Choice) or `cyclic` (every rotation, K rows). Both show the model level orders training never produces; in the `chat` style each line still names its level (`(level i of K)`). +- (b) Keep Score in one order by default, ship averaging as `EngineConfig.score_order_average` (off), and put the control in CI: `lev presentation-checks` against a committed report, failing on drift. + +**What is measured, fixed before the runs.** One checkpoint (`interfaze-ai/lev`, bf16, one RTX 5090), sokudan's `bench_en` (290) and `bench_ja` (300), `score_order_average` in {off, reversed, cyclic}: + +1. The identical-option control and the first-slot rate, for Score and Choice, with the shipped calibration and raw (an empty `CalibrationProfile`). +2. Score RPS and accuracy on the benches' 3-level urgency question, with Choice and Noul on the same requests expected unchanged (Score rows are the only rows added). +3. Milliseconds per three-question request, as the cost of each mode. +4. With averaging off, every answer identical to the current code on the same items. + +Every result is reported whichever way it goes, including accuracy that does not improve. + +**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error. Score's first-slot rate on real options is kept as a reference only: in the chat style the release uses, the probe's permuted listings renumber the levels as well as move them. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine refuses a request Score averaging grew past the batcher's token budget. + +**Evidence.** Run 2026-10-04/05 on `interfaze-ai/lev` (HF 7bdc748: the weights and calibration of f8ef711, only README.md changed), bf16, one RTX 5090, this branch rebased on main after #4. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json); the committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision. + +The bias of one read: identical-option control, slot 0 minus the mean (gate -0.20 raw). Raw is calibrated x T, exact within one temperature bucket. The three runs per bench, one per mode, agree exactly, as they must. + +```text + bench_en (290) bench_ja (300, reference) + calibrated raw calibrated raw + Score -0.332 -0.932 -0.518 -1.452 + Choice -0.454 -0.813 -0.660 -1.182 +``` + +Choice, read once, avoids slot 0 more than Score does; it is already read in two orders when served, and the first-slot rate below is what that leaves. + +What averaging changes. The benches' three questions per request (4-way Choice, 3-level Score, Noul), shipped calibration; Score ECE over 15 equal-width bins. For reference, the first-slot rate over all six orders of three real options, ties split (1/3 is order-free), and `consistent`, the share of states whose six orders give one answer: + +```text + Score Choice Noul + acc RPS ECE first slot consistent acc first slot consistent acc AUROC ms / request + en off 0.583 0.150 0.124 0.265 0.714 0.917 0.331 0.986 0.938 0.9778 156.5 + reversed 0.652 0.127 0.101 0.319 0.872 0.917 0.332 0.979 0.938 0.9780 179.3 (1.15x) + cyclic 0.641 0.125 0.060 0.310 0.893 0.917 0.332 0.983 0.938 0.9784 210.2 (1.34x) + ja off 0.643 0.138 0.093 0.242 0.627 0.887 0.331 0.973 0.817 0.8890 149.2 + reversed 0.730 0.108 0.059 0.314 0.870 0.887 0.331 0.977 0.813 0.8877 169.9 (1.14x) + cyclic 0.717 0.108 0.077 0.299 0.853 0.887 0.330 0.973 0.813 0.8876 197.3 (1.32x) +``` + +Accuracy, RPS, Choice and Noul are identical, to the last digit, to the runs before the rebase and the review fixes. In the chat style the release uses, a Score line names its level by its place in the request (`(level i of K)`), so the probe's permuted listings renumber the levels as well as move them, and Score's first-slot rate measures both; the engine's own averaging moves each level with its number. + +Calibration of the averaged modes. The shipped Score temperature (2.80) was fitted on single-order rows. Refit per mode on half of each bench (stratified by the gold level, seed 0), judged on the other half: + +```text + T refit NLL held-out ECE held-out RPS + shipped refit shipped refit shipped refit + en (144) off 3.96 0.935 0.909 0.130 0.108 0.152 0.147 + reversed 3.16 0.839 0.838 0.111 0.119 0.126 0.126 + cyclic 2.92 0.828 0.828 0.077 0.077 0.124 0.124 + ja (149) off 3.60 0.781 0.779 0.090 0.077 0.132 0.130 + reversed 2.39 0.641 0.627 0.089 0.061 0.102 0.101 + cyclic 2.40 0.644 0.629 0.068 0.086 0.101 0.101 +``` + +On bench_en a refit gains nothing for either averaged mode (held-out negative_log_likelihood within 0.001), and the shipped temperature is if anything a little sharp for `off`. On the Japanese reference the refit goes the other way for the averaged modes. A per-mode temperature is not proposed. + +The batch budget for averaged requests (`max_batch_tokens`, 16384). The batcher does not split a request: on main, 20 Choice and 40 Score questions with averaging off ran as one forward of 80 rows, up to 35,680 padded tokens. So `prepare` refuses only a request that Score averaging grew, when its padded tokens (rows x widest row) pass the batcher's own budget; the server passes its `max_batch_tokens` to the engine, so the two always agree, and a request without averaged Score rows is never refused (the 80-row request above answers exactly as on main). The value is the batcher's; this run only checks it on the RTX 5090, with the largest averaged request the budget admits at each row length, averaged Score questions topped up with one-row Noul (one question more is refused in every case): + +```text + row tokens mode rows padded tokens reserved GiB seconds + 474 reversed 34 16,116 14.6 1.7 + 474 cyclic 34 16,116 14.6 1.7 + 2,124 reversed 7 14,868 14.0 1.5 + 2,124 cyclic 7 14,868 14.0 1.5 + 4,124 reversed 3 12,372 13.1 1.4 + 4,124 cyclic 3 12,372 19.8 2.0 + 5,423 cyclic 3 16,269 25.8 2.8 + 8,124 reversed 2 16,248 31.7 18.9 at the card's edge, slow +``` + +Rows of ~5,400 tokens fit with room to spare. At ~8,100-token rows attention outgrows the token count: the admitted request reached the edge of the 32 GB card and took 18.9 s. Averaging is not the cause there: one Choice question on the same state with averaging off, read in two orders as always, needs the same (31.6 GiB reserved, 30.9 s). A row count could not have bounded this either: in a first measurement without any limit, 16 single-order rows of 2,124 tokens already spilled (30.1 GiB reserved, 12.3 s). + +The probe's own calls hold at most 32 rows (`--max-rows`, a measurement setting): a full presentation run with 64-row calls filled the card on bench_en (32 GB in use, 100% busy, no result after 22 minutes) and was stopped; with 32-row calls the six runs peaked at 22.5 GiB reserved, about 50 minutes each. + +With averaging off, every answer on 20 items of each bench (40 responses each, the probe included) is identical to main at 8ca94b7. Asking the probe's questions together or alone differs by up to |Δp| 0.038 (bench_en) and 0.041 (bench_ja), so the committed report records `max_rows`. + +**Reading, for the decision.** `reversed` lifts Score accuracy by 6.9 points on bench_en and lowers its RPS and ECE at 1.15x the time, with the shipped temperature. `cyclic` calibrates best on bench_en (ECE 0.060) but is no more accurate than `reversed` and costs 1.34x. Neither was tested on helpsteer2, the case that kept Score out of averaging in ADR-020, and both benches are one family of synthetic support messages. The proposal keeps the default off and leaves the choice between (a) and (b) to the maintainer. + +--- + ## ADR-030 — The server batches across requests and refuses work it cannot do in time **Accepted.** diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index c53ed27..7ff0b05 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -677,3 +677,40 @@ A user on a 3090 reported that lev "suffers in boundary conditions. When the ans **What should, in order:** graded or smoothed Noul targets through the existing `soft_targets` path, instead of one-hot 0/8; a paired consistency loss across whitespace and paraphrase variants of the same row; distillation toward the frozen model's Noul distribution, which these measurements show is well behaved at the boundary. Stripping trailing whitespace from the state before rendering is a near-free partial patch. **Limits.** The sample is 30 items per subset with one seed, so per-subset rates are noisy (civil_comments, the worst, is 8 of 30 counting the layout flips). bf16 on MPS is not the reporter's CUDA. On the same weights, batching alone moves p by up to 0.07 in bf16 (ADR-030), so small differences between perturbations are within that floor; spreads of 0.5 or more are not. + +## 19. The adapter scale is an out-of-distribution knob, and it is worth +3.9 macro points + +The released adapter is applied at full strength. Scaling it down at serving time interpolates back toward the frozen backbone — WiSE-FT ([arXiv 2109.01903](https://arxiv.org/abs/2109.01903)), except applied to a LoRA scale rather than a weight average. Swept 2026-10-07/08 on one H100, over all 13 pinned S1Bench subsets, by setting `LoraLayer.set_scale` on each of the 128 adapter layers. Scale 1.0 scores 0.6892 against the 0.6893 measured in §17, which is the control for the whole table. + +```text + 0 0.25 0.5 0.75 1 LOFO scale + aegis2 0.4320 0.8360 0.8200 0.8040 0.8000 <- 0.25 + boolq 0.4200 0.8533 0.8467 0.8367 0.8267 <- 0.25 + civil_comments 0.8900 0.7867 0.7767 0.7800 0.7600 <- 0.25 + helpsteer2 0.3133 0.3574 0.3855 0.3896 0.3855 <- 0.25 + massive-de-DE 0.8000 0.8257 0.8486 0.8429 0.8229 <- 0.25 + massive-en-US 0.8257 0.8686 0.8743 0.8686 0.8571 <- 0.25 + multinli 0.7893 0.8495 0.8662 0.8729 0.8896 <- 0.25 + paws 0.5160 0.8520 0.8360 0.8040 0.7760 <- 0.25 + pubmedqa 0.7040 0.7160 0.7160 0.7160 0.7320 <- 0.25 + squad2 0.5017 0.8227 0.8161 0.8294 0.8127 <- 0.25 + summeval-consistency 0.8264 0.8056 0.6250 0.3958 0.2708 <- 0.5 + summeval-relevance 0.4750 0.3792 0.3458 0.3667 0.3583 <- 0.5 + vitaminc-dev 0.6912 0.7279 0.7179 0.6828 0.6678 <- 0.25 + ------------------------------------------------------------------- + macro 0.6296 0.7447 0.7288 0.7069 0.6892 +``` + +**Scale 0.25 beats full strength by 5.5 macro points — but that number is test-selected.** +Choosing the scale on the same 13 subsets we report is fitting the benchmark, which is exactly what the contamination guard exists to forbid. Two honest selections: + +**The calibration split cannot do it.** Swept 2026-10-08 on 6,000 of its 48,691 rows at seven scales: accuracy is monotone in scale — 0.6233, 0.6508, 0.6810, 0.7112, 0.7393, 0.7942, 0.8133 at 0.125 through 1.0 — and picks 1.0, the scale already shipped. The split is drawn from the training mixture, so it is in-distribution, and shrinking the adapter *costs* in-distribution accuracy while buying out-of-distribution robustness. An in-distribution split is structurally blind to the trade, whatever its size. (Uncalibrated ECE moves the other way, 0.0619 at 0.125 to 0.1400 at 1.0.) + +**Leave-one-family-out over S1Bench can.** The 13 subsets share task families with training but no datasets. Grouping them into 7 families — safety, qa-bool, nli, paraphrase, intent, summeval, rating — and scoring each family at the scale chosen on the *other* six gives a macro of **0.7282**, against 0.6892 shipped and 0.7610 for Jev. Six of seven families pick 0.25 independently; only summeval picks 0.5, because summeval-consistency is the one subset that prefers a weak adapter (0.8264 frozen, 0.2708 at full strength) and the remaining subsets cannot see that when it is held +out. The estimate is robust to the grouping: merging helpsteer2 into summeval, splitting vitaminc off NLI, and leave-one-*subset*-out give 0.7447, 0.7282 and 0.7447. + +**Measured end to end, 2026-10-15.** With the scale and refit temperatures shipped in `releases/lev` and served by `make deploy NAME=lev`, `levbench` over HTTP scores macro **0.743**, from 0.689, and mean ECE 0.097, from 0.115, against Jev's 0.761 and 0.091 (`docs/charts/logs/lev-s1bench-2026-10-15.txt`). That is the selected scale scored on the subsets that selected it, so it sits above the 0.728 held-out estimate, as it should. summeval-consistency carries most of the gain, 0.271 to 0.806, and the worst calibration: ECE 0.303, under-confident, with accuracy 0.89–0.94 in the 0.4–0.7 confidence bins. The temperatures were refit on the calibration split, which contains no summeval rows. + +**This needs no retraining.** The scale is a serving-time multiplier on weights we already ship. + +**Limits.** The grid is coarse — 0, 0.25, 0.5, 0.75, 1.0 — and the optimum sits at its low end next to 0.25, so a finer sweep may do better; it would have to be selected under the same leave-one-out discipline to stay quotable. Accuracy is an argmax and so is invariant to temperature, which is why these numbers hold even though the shipped temperatures were fitted at scale 1.0; every *calibration* figure at any other scale requires a refit, which is what `fit_at_adapter_scale` in `modal/app.py` does. Per-subset n runs from 144 (summeval-consistency) to 599 (vitaminc-dev), so a single-subset move under about 6 points is noise at the small end and about 3 at the large end; the macro, pooled over all 13, is tighter than either. Closing the gap to Jev takes 3.3 points more than this buys. diff --git a/docs/SETUP.md b/docs/SETUP.md index cf0aaab..e51fe8d 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -78,6 +78,7 @@ Set on the machine where `modal deploy` runs, from the environment or `.env`; th | --- | --- | --- | | `LEV_SERVE_PRESET` | `4b` | Which preset's newest checkpoint to serve. `make deploy PRESET=...` sets it | | `LEV_SERVE_MODEL` | unset | Serve this model frozen instead — no adapter, binary Noul, the zero-shot baseline | +| `LEV_SERVE_RELEASE` | unset | Serve `releases/` instead of the training output, with its manifest's adapter scale and calibration. `make deploy NAME=...` sets it | | `LEV_SERVE_COMPILE` | `0` | `torch.compile`. Off unless asked: measured slower than eager here (ADR-023) | | `LEV_SERVE_MAX_LABEL_OPTIONS` | unset | Cap on label options. Unset means Mode A up to the tokenizer limit (ADR-025) | | `LEV_SERVE_PROMPT` | the preset's | Override the prompt style, mainly for a frozen model | diff --git a/docs/STATUS.md b/docs/STATUS.md index d565a44..ffbb300 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -23,4 +23,4 @@ What is built, and how far each part has been exercised. | Mode B head | **trains and serves**; weak unseen-taxonomy transfer measured on massive-en-US ([FINDINGS §12](FINDINGS.md#12-first-head-to-head-on-s1bench-two-named-failure-modes)) | | `lev.load`, release packaging, Hub publishing | **done**; the release loads through both `lev.load` and the server on an H100 (`check_release`) | -Three checkpoints have been trained, calibrated and scored on S1Bench against Jev through identical task files: 0.489, then 0.697, then **0.725** macro against Jev's 0.754 on the earlier six-subset definitions. The released checkpoint scores **0.689** against Jev's 0.761 on all 13 subsets as S1Bench pins them. What each run changed and why is in [FINDINGS §12–17](FINDINGS.md); the reader bug that corrupted the labels of one intermediate run is [ADR-024](DECISIONS.md#adr-024--the-mixture-reader-merged-shuffled-questions-and-the-instruct-run-trained-on-it). +Three checkpoints have been trained, calibrated and scored on S1Bench against Jev through identical task files: 0.489, then 0.697, then **0.725** macro against Jev's 0.754 on the earlier six-subset definitions. The released checkpoint scores **0.743** against Jev's 0.761 on all 13 subsets as S1Bench pins them, with its adapter at a quarter of its trained strength ([FINDINGS §19](FINDINGS.md#19-the-adapter-scale-is-an-out-of-distribution-knob-and-it-is-worth-39-macro-points)); at full strength it scored 0.689. What each run changed and why is in [FINDINGS §12–17](FINDINGS.md); the reader bug that corrupted the labels of one intermediate run is [ADR-024](DECISIONS.md#adr-024--the-mixture-reader-merged-shuffled-questions-and-the-instruct-run-trained-on-it). diff --git a/docs/TRAINING.md b/docs/TRAINING.md index d4fd4d2..4070cbf 100644 --- a/docs/TRAINING.md +++ b/docs/TRAINING.md @@ -260,7 +260,7 @@ make publish RELEASE=4b-instruct REPO=org/name # -> huggingface.co/org/name (HF ### On Modal: which checkpoint, and `serve` vs `deploy` -`make deploy PRESET=4b-instruct` serves that preset's newest checkpoint; the preset travels to the container as `LEV_SERVE_PRESET`. `LEV_SERVE_MODEL=Qwen/Qwen3.5-4B make deploy` serves that model frozen — no adapter, binary Noul — the zero-shot baseline. +`make deploy PRESET=4b-instruct` serves that preset's newest checkpoint; the preset travels to the container as `LEV_SERVE_PRESET`. `make deploy NAME=lev` serves the packaged release instead — what the Hub ships, including an `adapter_scale` or calibration refit after export. `LEV_SERVE_MODEL=Qwen/Qwen3.5-4B make deploy` serves that model frozen — no adapter, binary Noul — the zero-shot baseline. `modal serve` is an ephemeral dev server. So is every `modal run` of this app file, and each one registers the `serve` web function under the same `-dev` label — so running an export or a diagnostic while the dev server is up steals its URL, and the URL returns 404 when the run exits. `make deploy` gives the server a stable URL that runs cannot take. diff --git a/docs/charts/build.py b/docs/charts/build.py index 53cdc1a..78d68a2 100644 --- a/docs/charts/build.py +++ b/docs/charts/build.py @@ -1,16 +1,5 @@ # ruff: noqa: E501 -- SVG and HTML markup, one element per line -"""Render docs/charts/lev-vs-jev.html from measured logs and recorded history. - - uv run python docs/charts/build.py - -Inputs, all checked in: - logs/jev-s1bench-*.txt, logs/lev-s1bench-*.txt both on the 13 S1Bench task files, same laptop - logs/pre-nimble/ earlier six-subset definitions, the scale of history.json - history.json lev iterations and speed measurements, with FINDINGS sections - ../../data/s1bench-snapshot.json the public S1Bench board: every other model - -lev blue and Jev orange (CVD-checked in both modes); every other model is context gray. -""" +"""Render docs/charts/lev-vs-jev.html from checked-in logs, history, and the S1Bench snapshot.""" from __future__ import annotations @@ -25,7 +14,6 @@ HERE = Path(__file__).parent SNAPSHOT = HERE.parent.parent / "data" / "s1bench-snapshot.json" -# The six subsets every model on the board completed; the leaderboard is over these. SUBSETS = ["aegis2", "boolq", "helpsteer2", "massive-en-US", "paws", "vitaminc-dev"] ALL_SUBSETS = list(EVAL_SUBSETS) W = 760 @@ -42,7 +30,6 @@ def pct(v: float) -> str: def bar_path(x0: float, x1: float, y: float, h: float, r: float = 4) -> str: - """Square at the baseline, 4px rounded at the data end.""" if x1 - x0 <= 0.5: return "" r = min(r, (x1 - x0) / 2, h / 2) @@ -74,7 +61,6 @@ def table(headers, rows) -> str: def leaderboard(rows, vmin=0.2, vmax=0.8) -> str: - """Ranked horizontal bars. Rows: dicts with name, value, var, meta, bold.""" label_w, meta_w, row_h, bar_h, top = 196, 150, 23, 13, 6 plot_w = W - label_w - meta_w - 40 x = lambda v: label_w + (v - vmin) / (vmax - vmin) * plot_w # noqa: E731 @@ -142,7 +128,6 @@ def grouped_bars(cats, series, vmax, fmt, ticks, label_w=118) -> str: def scatter( points, xr, yr, xfmt, yfmt, xticks, yticks, xlabel, ylabel, xlog=False, hline=None ) -> str: - """Points: dicts with x, y, name, var, and label (direct-label it).""" h, left, right, top, bottom = 380, 58, 24, 14, 44 pw, ph = W - left - right, h - top - bottom tx = math.log10 if xlog else (lambda v: v) @@ -169,7 +154,7 @@ def scatter( f'' f'{esc(hline["name"])}' ) - for p in sorted(points, key=lambda p: p["var"] != CTX): # context underneath + for p in sorted(points, key=lambda p: p["var"] != CTX): cx, cy = px(p["x"]), py(p["y"]) r = 6 if p["var"] != CTX else 4.5 tip = f"{p['name']}|{xfmt(p['x'])} · {yfmt(p['y'])}" @@ -288,14 +273,11 @@ def mean(scores, key, subsets=SUBSETS): def majority_rate(subset: str) -> float: - """Accuracy of always answering the most common label.""" labels = definition(subset)["labels"] return max(labels.values()) / sum(labels.values()) def completed_targets(board: dict) -> list[dict]: - """Board models that completed all six subsets; a model stopped after one - subset reports a one-subset macro and is not comparable.""" return [ t for t in board["targets"] @@ -316,7 +298,6 @@ def leaderboard_row(name: str, value: float, var: str, size: str, ece: float, bo def leaderboard_rows(done: list[dict], data: Inputs) -> list[dict]: - """Board models plus lev and Jev as measured here, best first.""" rows = [] for t in done: is_jev = t["target"] == "jev" @@ -526,7 +507,6 @@ def size_section(points: list[dict], jev_board: dict) -> str: def per_subset_series(data: Inputs, value) -> list[dict]: - """lev and Jev as grouped-bar series, `value(scores)` per subset.""" return [ {"name": "lev", "var": LEV, "values": [value(data.lev[s]) for s in ALL_SUBSETS]}, {"name": "Jev", "var": JEV, "values": [value(data.jev[s]) for s in ALL_SUBSETS]}, diff --git a/docs/charts/lev-vs-jev.html b/docs/charts/lev-vs-jev.html index 3eea664..751d8e2 100644 --- a/docs/charts/lev-vs-jev.html +++ b/docs/charts/lev-vs-jev.html @@ -51,9 +51,9 @@
S1Bench · performance and speed

lev: a 4B open System One model, against Jev and the open field

Qwen3.5-4B with a LoRA adapter, a candidate-path head and fitted temperatures. Scored on the same S1Bench task files as TypeSafe's Jev, and placed against every model on the public S1Bench board.

-
Accuracy
68.9%lev
76.1%Jev
macro, all 13 subsets · higher is better
Accuracy, board subsets
71.9%lev
76.9%Jev
macro, the 6 the board completed
Calibration error
0.115lev
0.091Jev
mean ECE, 13 subsets · lower is better
-
Where lev stands

Behind only Jev and 3 open models of 26B and up

Macro accuracy over the six S1Bench subsets that every listed model completed.

levJevopen models on the S1Bench board
20%30%40%50%60%70%80%Jev (board)77.5%TypeSafe API · ECE 0.076Jev (measured here)76.9%TypeSafe API · ECE 0.090simplejev-qwen38-27b75.8%27B · ECE 0.121djev-full74.9%26B · ECE 0.166simplejev-qwen36-35b-a3b74.4%35B · ECE 0.136lev (measured here)71.9%4B · ECE 0.123reflex-4b71.9%4B · ECE 0.085decider-2b †70.3%2B · ECE 0.114laya-gpu62.5%421M · ECE 0.130laya62.5%421M · ECE 0.132jeff56.1%400M · ECE 0.075jeff-gpu-full55.9%400M · ECE 0.074qwen3-8b-full53.5%8B · ECE 0.425open-jev-deberta †52.4%435M · ECE 0.067reflex-08b51.6%800M · ECE 0.243kev-05b †49.3%500M · ECE 0.149gliner-base43.2%194M · ECE 0.316gliner-multi42.6%287M · ECE 0.181gliner-small40.0%74M · ECE 0.224simplejev-rwkv-mid38.0%size n/a · ECE 0.298simplejev-rwkv-std32.7%size n/a · ECE 0.282simplejev-rwkv-small31.3%size n/a · ECE 0.279

Board models were scored by S1Bench's own harness, lev by ours, on the same pinned items. On the same model our harness scores Jev 0.6 points lower than the board, all of it on aegis2. † self-declared training contamination on an evaluation subset. Models stopped after one subset are omitted. Gemini, Claude and GPT have not been run on these subsets, so they are not shown.

Data
modelmacro accuracysizeECE
Jev (board)77.5%TypeSafe API0.076
Jev (measured here)76.9%TypeSafe API0.090
simplejev-qwen38-27b75.8%27B0.121
djev-full74.9%26B0.166
simplejev-qwen36-35b-a3b74.4%35B0.136
lev (measured here)71.9%4B0.123
reflex-4b71.9%4B0.085
decider-2b †70.3%2B0.114
laya-gpu62.5%421M0.130
laya62.5%421M0.132
jeff56.1%400M0.075
jeff-gpu-full55.9%400M0.074
qwen3-8b-full53.5%8B0.425
open-jev-deberta †52.4%435M0.067
reflex-08b51.6%800M0.243
kev-05b †49.3%500M0.149
gliner-base43.2%194M0.316
gliner-multi42.6%287M0.181
gliner-small40.0%74M0.224
simplejev-rwkv-mid38.0%size n/a0.298
simplejev-rwkv-std32.7%size n/a0.282
simplejev-rwkv-small31.3%size n/a0.279
Accuracy and calibration

Further right is more accurate; lower is better calibrated

Macro accuracy against calibration error (ECE). The ideal model sits in the bottom-right corner.

30.0%40.0%50.0%60.0%70.0%80.0%0.000.100.200.300.40macro accuracyECEsimplejev-qwen38-27bdjev-fullreflex-4bdecider-2blaya-gpujeff-gpu-fullqwen3-8b-fullopen-jev-debertaJevlev

lev's point uses our harness for both axes; the others use the board's.

Data
modelmacro accuracyECE
Jev77.5%0.076
simplejev-qwen38-27b75.8%0.121
djev-full74.9%0.166
simplejev-qwen36-35b-a3b74.4%0.136
lev71.9%0.123
reflex-4b71.9%0.085
decider-2b70.3%0.114
laya-gpu62.5%0.130
laya62.5%0.132
jeff56.1%0.075
jeff-gpu-full55.9%0.074
qwen3-8b-full53.5%0.425
open-jev-deberta52.4%0.067
reflex-08b51.6%0.243
kev-05b49.3%0.149
gliner-base43.2%0.316
gliner-multi42.6%0.181
gliner-small40.0%0.224
simplejev-rwkv-mid38.0%0.298
simplejev-rwkv-std32.7%0.282
simplejev-rwkv-small31.3%0.279
Accuracy per parameter

lev matches open models six times its size

Macro accuracy against parameter count, log scale. Jev's size is not published, so it is drawn as a line.

100M300M1B3B10B30B30.0%40.0%50.0%60.0%70.0%80.0%parametersmacro accuracyJev (board) 77.5%simplejev-qwen38-27breflex-4bdecider-2blaya-gpujeff-gpu-fullqwen3-8b-fullopen-jev-debertalev
Data
modelparametersmacro accuracy
gliner-small0.074B40.0%
gliner-base0.194B43.2%
gliner-multi0.287B42.6%
jeff0.4B56.1%
jeff-gpu-full0.4B55.9%
laya-gpu0.421B62.5%
laya0.421B62.5%
open-jev-deberta0.435B52.4%
kev-05b0.5B49.3%
reflex-08b0.8B51.6%
decider-2b2B70.3%
reflex-4b4B71.9%
lev4B71.9%
qwen3-8b-full8B53.5%
djev-full26B74.9%
simplejev-qwen38-27b27B75.8%
simplejev-qwen36-35b-a3b35B74.4%
Head to head

Accuracy by subset

lev and Jev on all 13 S1Bench subsets (3,880 task items), same harness, same laptop.

levJev
0.0%25.0%50.0%75.0%100.0%vitaminc-dev66.8%80.1%massive-en-US85.7%87.4%massive-de-DE82.3%87.1%boolq82.7%89.3%squad281.3%83.6%paws77.6%90.0%multinli89.0%83.6%civil_comments76.0%80.3%aegis280.0%80.4%helpsteer238.6%34.1%summeval-relevance35.8%35.8%summeval-consistency27.1%81.2%pubmedqa73.2%76.4%

At these sizes a per-subset difference needs roughly 5–9 points to clear sampling noise. On civil_comments, helpsteer2 and both summeval subsets, always answering the most common label beats both models.

Data
subsetnlevJevlev − Jevmajority label
vitaminc-dev59966.8%80.1%-13.3 pts50.3%
massive-en-US35085.7%87.4%-1.7 pts16.3%
massive-de-DE35082.3%87.1%-4.8 pts16.3%
boolq30082.7%89.3%-6.6 pts58.0%
squad229981.3%83.6%-2.3 pts50.2%
paws25077.6%90.0%-12.4 pts51.6%
multinli29989.0%83.6%+5.4 pts36.1%
civil_comments30076.0%80.3%-4.3 pts89.3%
aegis225080.0%80.4%-0.4 pts56.8%
helpsteer224938.6%34.1%+4.5 pts42.2%
summeval-relevance24035.8%35.8%+0.0 pts45.8%
summeval-consistency14427.1%81.2%-54.1 pts84.0%
pubmedqa25073.2%76.4%-3.2 pts53.2%
Head to head

Calibration error by subset

The gap between stated confidence and observed accuracy. Lower is better; 0 is perfect.

levJev
0.0000.1000.2000.300vitaminc-dev0.1410.099massive-en-US0.0560.071massive-de-DE0.0670.059boolq0.1260.035squad20.1030.036paws0.1780.028multinli0.0370.057civil_comments0.1180.044aegis20.1340.050helpsteer20.1040.258summeval-relevance0.1360.230summeval-consistency0.1740.086pubmedqa0.1210.136
Data
subsetlev ECEJev ECElev log-lossJev log-loss
vitaminc-dev0.1410.0990.8831.571
massive-en-US0.0560.0710.6231.377
massive-de-DE0.0670.0590.6671.218
boolq0.1260.0350.4930.281
squad20.1030.0360.4850.347
paws0.1780.0280.7360.293
multinli0.0370.0570.3160.462
civil_comments0.1180.0440.6060.444
aegis20.1340.0500.5800.414
helpsteer20.1040.2581.3864.672
summeval-relevance0.1360.2301.4451.841
summeval-consistency0.1740.0861.5031.446
pubmedqa0.1210.1360.6951.600
Progress

From 48.9% to 72.5% across four iterations

lev's macro accuracy per model iteration, against Jev and the frozen backbone it was trained from. Measured on the six subsets as this repo defined them before adopting S1Bench's pinned items, so the scale differs from the charts above.

40%50%60%70%80%run 1Base, 9 sourcesrun 2instruct, 23 srcrun 2+ Mode A to limitrun 3chat, 29 sourcesfinalserving fixesJev 75.4%frozen 4B 71.0%48.9%61.2%69.7%72.5%72.5%
Data
iterationmacro accuracyrecorded in
run 1 Base, 9 sources48.9%FINDINGS §12
run 2 instruct, 23 src61.2%FINDINGS §15
run 2 + Mode A to limit69.7%FINDINGS §15, ADR-025
run 3 chat, 29 sources72.5%FINDINGS §16
final, serving fixes72.5%logs/pre-nimble/lev-s1bench-final.txt
Speed

lev compute per call: 169 ms → 69 ms

Inside the container on one H100, before any network. lev only; Jev's internals are not observable.

0 ms50 ms100 ms150 ms200 msprefill + fork (original)169 msprefill + fork, conv kernel140 mssingle batched forward81 mssingle + conv kernel (served)69 ms

Benchmark throughput rose from 1.02 to 6.36 items/s (new path, concurrency 4); a question costs the same whether a request asks one or eight.

Data
engine pathms per call
prefill + fork (original)169
prefill + fork, conv kernel140
single batched forward81
single + conv kernel (served)69
-
Sources: lev: logs/lev-s1bench-2026-09-24.txt (Qwen/Qwen3.5-4B) · Jev: logs/jev-s1bench-2026-09-24.txt · board: data/s1bench-snapshot.json (build 468a014bc5) · iterations and speed: history.json
+
Accuracy
74.3%lev
76.1%Jev
macro, all 13 subsets · higher is better
Accuracy, board subsets
74.8%lev
76.9%Jev
macro, the 6 the board completed
Calibration error
0.097lev
0.091Jev
mean ECE, 13 subsets · lower is better
+
Where lev stands

Behind only Jev and 2 open models of 26B and up

Macro accuracy over the six S1Bench subsets that every listed model completed.

levJevopen models on the S1Bench board
20%30%40%50%60%70%80%Jev (board)77.5%TypeSafe API · ECE 0.076Jev (measured here)76.9%TypeSafe API · ECE 0.090simplejev-qwen38-27b75.8%27B · ECE 0.121djev-full74.9%26B · ECE 0.166lev (measured here)74.8%4B · ECE 0.091simplejev-qwen36-35b-a3b74.4%35B · ECE 0.136reflex-4b71.9%4B · ECE 0.085decider-2b †70.3%2B · ECE 0.114laya-gpu62.5%421M · ECE 0.130laya62.5%421M · ECE 0.132jeff56.1%400M · ECE 0.075jeff-gpu-full55.9%400M · ECE 0.074qwen3-8b-full53.5%8B · ECE 0.425open-jev-deberta †52.4%435M · ECE 0.067reflex-08b51.6%800M · ECE 0.243kev-05b †49.3%500M · ECE 0.149gliner-base43.2%194M · ECE 0.316gliner-multi42.6%287M · ECE 0.181gliner-small40.0%74M · ECE 0.224simplejev-rwkv-mid38.0%size n/a · ECE 0.298simplejev-rwkv-std32.7%size n/a · ECE 0.282simplejev-rwkv-small31.3%size n/a · ECE 0.279

Board models were scored by S1Bench's own harness, lev by ours, on the same pinned items. On the same model our harness scores Jev 0.6 points lower than the board, all of it on aegis2. † self-declared training contamination on an evaluation subset. Models stopped after one subset are omitted. Gemini, Claude and GPT have not been run on these subsets, so they are not shown.

Data
modelmacro accuracysizeECE
Jev (board)77.5%TypeSafe API0.076
Jev (measured here)76.9%TypeSafe API0.090
simplejev-qwen38-27b75.8%27B0.121
djev-full74.9%26B0.166
lev (measured here)74.8%4B0.091
simplejev-qwen36-35b-a3b74.4%35B0.136
reflex-4b71.9%4B0.085
decider-2b †70.3%2B0.114
laya-gpu62.5%421M0.130
laya62.5%421M0.132
jeff56.1%400M0.075
jeff-gpu-full55.9%400M0.074
qwen3-8b-full53.5%8B0.425
open-jev-deberta †52.4%435M0.067
reflex-08b51.6%800M0.243
kev-05b †49.3%500M0.149
gliner-base43.2%194M0.316
gliner-multi42.6%287M0.181
gliner-small40.0%74M0.224
simplejev-rwkv-mid38.0%size n/a0.298
simplejev-rwkv-std32.7%size n/a0.282
simplejev-rwkv-small31.3%size n/a0.279
Accuracy and calibration

Further right is more accurate; lower is better calibrated

Macro accuracy against calibration error (ECE). The ideal model sits in the bottom-right corner.

30.0%40.0%50.0%60.0%70.0%80.0%0.000.100.200.300.40macro accuracyECEsimplejev-qwen38-27bdjev-fullreflex-4bdecider-2blaya-gpujeff-gpu-fullqwen3-8b-fullopen-jev-debertaJevlev

lev's point uses our harness for both axes; the others use the board's.

Data
modelmacro accuracyECE
Jev77.5%0.076
simplejev-qwen38-27b75.8%0.121
djev-full74.9%0.166
lev74.8%0.091
simplejev-qwen36-35b-a3b74.4%0.136
reflex-4b71.9%0.085
decider-2b70.3%0.114
laya-gpu62.5%0.130
laya62.5%0.132
jeff56.1%0.075
jeff-gpu-full55.9%0.074
qwen3-8b-full53.5%0.425
open-jev-deberta52.4%0.067
reflex-08b51.6%0.243
kev-05b49.3%0.149
gliner-base43.2%0.316
gliner-multi42.6%0.181
gliner-small40.0%0.224
simplejev-rwkv-mid38.0%0.298
simplejev-rwkv-std32.7%0.282
simplejev-rwkv-small31.3%0.279
Accuracy per parameter

lev matches open models six times its size

Macro accuracy against parameter count, log scale. Jev's size is not published, so it is drawn as a line.

100M300M1B3B10B30B30.0%40.0%50.0%60.0%70.0%80.0%parametersmacro accuracyJev (board) 77.5%simplejev-qwen38-27breflex-4bdecider-2blaya-gpujeff-gpu-fullqwen3-8b-fullopen-jev-debertalev
Data
modelparametersmacro accuracy
gliner-small0.074B40.0%
gliner-base0.194B43.2%
gliner-multi0.287B42.6%
jeff0.4B56.1%
jeff-gpu-full0.4B55.9%
laya-gpu0.421B62.5%
laya0.421B62.5%
open-jev-deberta0.435B52.4%
kev-05b0.5B49.3%
reflex-08b0.8B51.6%
decider-2b2B70.3%
reflex-4b4B71.9%
lev4B74.8%
qwen3-8b-full8B53.5%
djev-full26B74.9%
simplejev-qwen38-27b27B75.8%
simplejev-qwen36-35b-a3b35B74.4%
Head to head

Accuracy by subset

lev and Jev on all 13 S1Bench subsets (3,880 task items), same harness, same laptop.

levJev
0.0%25.0%50.0%75.0%100.0%vitaminc-dev72.6%80.1%massive-en-US86.9%87.4%massive-de-DE82.9%87.1%boolq85.3%89.3%squad281.9%83.6%paws84.4%90.0%multinli84.9%83.6%civil_comments78.3%80.3%aegis283.6%80.4%helpsteer236.1%34.1%summeval-relevance37.1%35.8%summeval-consistency80.6%81.2%pubmedqa71.6%76.4%

At these sizes a per-subset difference needs roughly 5–9 points to clear sampling noise. On civil_comments, helpsteer2 and both summeval subsets, always answering the most common label beats both models.

Data
subsetnlevJevlev − Jevmajority label
vitaminc-dev59972.6%80.1%-7.5 pts50.3%
massive-en-US35086.9%87.4%-0.5 pts16.3%
massive-de-DE35082.9%87.1%-4.2 pts16.3%
boolq30085.3%89.3%-4.0 pts58.0%
squad229981.9%83.6%-1.7 pts50.2%
paws25084.4%90.0%-5.6 pts51.6%
multinli29984.9%83.6%+1.3 pts36.1%
civil_comments30078.3%80.3%-2.0 pts89.3%
aegis225083.6%80.4%+3.2 pts56.8%
helpsteer224936.1%34.1%+2.0 pts42.2%
summeval-relevance24037.1%35.8%+1.3 pts45.8%
summeval-consistency14480.6%81.2%-0.6 pts84.0%
pubmedqa25071.6%76.4%-4.8 pts53.2%
Head to head

Calibration error by subset

The gap between stated confidence and observed accuracy. Lower is better; 0 is perfect.

levJev
0.0000.1000.2000.300vitaminc-dev0.1070.099massive-en-US0.0440.071massive-de-DE0.0520.059boolq0.0600.035squad20.0660.036paws0.1120.028multinli0.0470.057civil_comments0.0790.044aegis20.1110.050helpsteer20.1140.258summeval-relevance0.0550.230summeval-consistency0.3030.086pubmedqa0.1050.136
Data
subsetlev ECEJev ECElev log-lossJev log-loss
vitaminc-dev0.1070.0990.7321.571
massive-en-US0.0440.0710.5531.377
massive-de-DE0.0520.0590.6351.218
boolq0.0600.0350.3800.281
squad20.0660.0360.4340.347
paws0.1120.0280.5140.293
multinli0.0470.0570.3820.462
civil_comments0.0790.0440.4820.444
aegis20.1110.0500.5030.414
helpsteer20.1140.2581.4334.672
summeval-relevance0.0550.2301.3431.841
summeval-consistency0.3030.0860.8471.446
pubmedqa0.1050.1360.6681.600
Progress

From 48.9% to 72.5% across four iterations

lev's macro accuracy per model iteration, against Jev and the frozen backbone it was trained from. Measured on the six subsets as this repo defined them before adopting S1Bench's pinned items, so the scale differs from the charts above.

40%50%60%70%80%run 1Base, 9 sourcesrun 2instruct, 23 srcrun 2+ Mode A to limitrun 3chat, 29 sourcesfinalserving fixesJev 75.4%frozen 4B 71.0%48.9%61.2%69.7%72.5%72.5%
Data
iterationmacro accuracyrecorded in
run 1 Base, 9 sources48.9%FINDINGS §12
run 2 instruct, 23 src61.2%FINDINGS §15
run 2 + Mode A to limit69.7%FINDINGS §15, ADR-025
run 3 chat, 29 sources72.5%FINDINGS §16
final, serving fixes72.5%logs/pre-nimble/lev-s1bench-final.txt
Speed

lev compute per call: 169 ms → 69 ms

Inside the container on one H100, before any network. lev only; Jev's internals are not observable.

0 ms50 ms100 ms150 ms200 msprefill + fork (original)169 msprefill + fork, conv kernel140 mssingle batched forward81 mssingle + conv kernel (served)69 ms

Benchmark throughput rose from 1.02 to 6.36 items/s (new path, concurrency 4); a question costs the same whether a request asks one or eight.

Data
engine pathms per call
prefill + fork (original)169
prefill + fork, conv kernel140
single batched forward81
single + conv kernel (served)69
+
Sources: lev: logs/lev-s1bench-2026-10-15.txt (Qwen/Qwen3.5-4B) · Jev: logs/jev-s1bench-2026-09-24.txt · board: data/s1bench-snapshot.json (build 468a014bc5) · iterations and speed: history.json