+ System One models don't generate text. They answer a typed question about some state with a
+ probability for every answer, in milliseconds. These experiments test where that's enough to stand in
+ for a parser, a grader, a release manager, or the choice of which model to call.
+
Which description fits the work this task asks for?
+
+
+
+
+
+
+
+ Replaying real answers from TypeSafe Jev (jev-1.13.0), model-router run of 2026-10-08.
+
+
+
+
+
+
+
+
+
01
+
+
Three kinds of question, no prose
+
+ Jev is TypeSafe's System
+ One model, and Open Jev serves the same
+ /v1/systemone API from open weights on your own machine. You send state and questions; you
+ get back distributions. The code around it decides what to do with them.
+
+
+
+
+
+
Noul yes / no
+
The probability that a statement about the state is true.
+
+ attempt: "a genuine attempt?" → 0.97
+ example from misconception-gates' design
+
+
+
+
Choice one of ≤ 255
+
A probability for each option. It can only pick something that's on the list.
+
+ "grab the torch" → take__torch
+ godot-jev: options are the moves possible right now
+
+
+
+
Score 2–10 levels
+
Where the input sits on an ordered scale, with a probability per level.
+ Each is a standalone directory with its own stack, a hypothesis, the checks that would confirm it, and
+ tests that run without an API key. Skeleton means the design is in code and tested against
+ scripted answers. Live means runs against real models are recorded, with dates.
+
+ An agent harness asks one Score question about a sub-agent's task, then takes the cheapest model
+ whose chance of being too weak is acceptable. The orchestrator spends no tokens deciding.
+
+
+ One Jev call over the text of a sub-agent task chooses a model tier well enough to move a large
+ share of work off the most capable model, while rarely sending a task to a model too weak for it.
+
+
+
+ First live run · 2026-10-08 · 24 tasks
+
+
+
0
+
tasks routed too low, by any of the three decision models
+
+
+
67%
+
kept off the most capable model, by all three
+
+
+
93 ms
+
median decision, TypeSafe Jev (Flash 9B 181, 27B 567)
+ A dungeon crawl whose parser understands whatever you type but can only choose moves the author wrote.
+ Each turn, the possible moves become one Choice, plus "none of these".
+
+
+ Players type naturally with few dead ends, the game stays fully authored, and it's fast enough to feel
+ like a parser, not a chat.
+
+
+
2026-09-30
+
TypeSafe Jev and Open Jev both landed 48 of 48 turns and ranked the same move first on all 48.
+
Speed
+
p95 502 ms hosted, 683 ms local. Neither met the 500 ms bar.
+ Read a learner's free-text answer for which misconception it shows, and send them to the part
+ of the lesson written for it.
+
+
+ With four to six authored misconceptions per gate, one Jev call diagnoses free-text answers accurately
+ enough to act on most of them automatically.
+
+
+
Next
+
Accuracy at a useful coverage on ≥ 50 labelled answers per gate.
+ Decide the bump from the commits since the last tag. Code reads what messages declare, Jev judges what
+ they don't, and a human sees only commits that are uncertain and could change the answer.
+
+
+ A single Jev call per commit agrees with the bump a maintainer would choose on commits without
+ Conventional prefixes.
+
+
+
Next
+
Replay the tagged history of public repos and compare with the bumps maintainers made.
+ Every experiment is built the same way. It keeps the model replaceable and the tests honest.
+
+
+
+
+
+
A decider port
+
Code asks decide(state, questions). One adapter calls Jev; a scripted one drives the tests. No unit test calls the network.
+
+
+
Policy in code
+
Thresholds, arithmetic, dates and counting stay in ordinary code. A policy change is a code change, not a prompt rewrite.
+
+
+
Ask only what's open
+
Deterministic rules go first. The model answers only what they leave undecided.
+
+
+
Confidence decides the path
+
Act, ask or escalate by how sure the model is, with the bar set by the cost of being wrong.
+
+
+
Fail safe
+
If the model is unreachable, fall back to what the system did before it had one.
+
+
+
+
+
+
+
+
+
04
+
+
Notebook
+
Dated entries, newest first. Each experiment's README records its evidence the same way.
+
+
+
+
+
+
+ model-router's first live run
+ TypeSafe Jev, Open Jev 27B and Open Jev Flash 9B route 24 sub-agent tasks. None goes too low; the three differ in how sure they are and how fast.
+
+
+
+
+
+ The Sunken Crypt on TypeSafe Jev and Open Jev
+ Same 16 lines, three rounds each. Both 48 of 48. p50 213 ms hosted, 600 ms local.
+
+
+
+
+
+ A scripted 16-turn session through the crypt
+ Every turn 197–300 ms on jev-latest, once the HTTP request moved off the main thread.
+
+
+
+
+
+ godot-jev becomes a dungeon crawl
+ Six rooms, five items, pixel-art rooms painted by code, and a status panel that shows what Jev heard.
+
+
+
+
+
+ Three experiment skeletons
+ Picked from areas awesome-jev had no entries for.
+
+
+
+
+
+ Proof of concept: pattern matching on meaning
+ The fuzzymatch library, now on the archive/fuzzymatch branch.
+
+
+
+
+
+
+
+
+
+
+
05
+
+
Read before trusting
+
Limits that apply to every experiment here.
+
+
+
+
+
Jev takes text only and reads instructions literally.
+
It can't count or compare dates. That's why arithmetic and thresholds live in code.
+
Text inside the state can argue for an answer: a learner, a commit author, a player or an orchestrating model can write "this is trivial".
+
Where the state comes from someone with a reason to game it, keep the stakes of any automatic action low, or keep a human in the loop.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/model-router/app.js b/docs/model-router/app.js
new file mode 100644
index 0000000..285bb9c
--- /dev/null
+++ b/docs/model-router/app.js
@@ -0,0 +1,742 @@
+/* model-router results page. Draws window.MR_DATA (data.js), which
+ docs/tools/build-model-router-data.mjs generates from the recorded runs
+ with model-router's own policy code. Nothing here re-decides a route:
+ routes at any bar come from rows[].routeAt. */
+
+(function () {
+ "use strict";
+
+ const D = window.MR_DATA;
+ const runs = D.runs;
+ const routes = D.routes;
+ const ids = routes.map((r) => r.id);
+ const N = D.tasks.length;
+ const TOP = routes.length - 1;
+ const DEFAULT_IDX = Math.round(D.defaultBar * 100);
+ const byId = Object.fromEntries(runs.map((r) => [r.id, r]));
+ const ref = runs[0]; // TypeSafe Jev: the reference for deltas
+ const reduceMotion = matchMedia("(prefers-reduced-motion: reduce)").matches;
+
+ const $ = (sel) => document.querySelector(sel);
+ const esc = (s) => String(s).replace(/[&<>"]/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """ })[c]);
+ const f2 = (x) => x.toFixed(2);
+ const ms = (x) => `${Math.round(x)} ms`;
+ const mColor = (run) => `var(--m-${run.id})`;
+ const rColor = (l) => `var(--r-${l})`;
+ const rInk = (l) => `var(--r-ink-${l})`;
+ const pct = (n) => `${Math.round((n / N) * 100)}%`;
+ const swatch = (run) => ``;
+ const summaryAt = (run, bar) => run.summaries.find((s) => Math.abs(s.bar - bar) < 1e-9);
+ const atDefault = (run) => summaryAt(run, D.defaultBar);
+
+ function outcome(route, label) {
+ if (route === label) return "agree";
+ return route < label ? "low" : "high";
+ }
+ const GLYPH = { agree: "✓", high: "▲", low: "▼" };
+ const OUT_WORD = { agree: "as labelled", high: "too high", low: "too low" };
+
+ /* ---------- tooltip ---------- */
+ const tip = $("#tip");
+ const tipAttr = (html) => `data-tip="${esc(html)}" tabindex="0"`;
+ function placeTip(x, y) {
+ const pad = 14;
+ const w = tip.offsetWidth;
+ const h = tip.offsetHeight;
+ let left = x + pad;
+ let top = y + pad;
+ if (left + w > innerWidth - 8) left = Math.max(8, x - w - pad);
+ if (top + h > innerHeight - 8) top = Math.max(8, y - h - pad);
+ tip.style.left = `${left}px`;
+ tip.style.top = `${top}px`;
+ }
+ function showTip(el, x, y) {
+ tip.innerHTML = el.dataset.tip;
+ tip.hidden = false;
+ if (x == null) {
+ const r = el.getBoundingClientRect();
+ x = r.left + r.width / 2;
+ y = r.bottom;
+ }
+ placeTip(x, y);
+ }
+ document.addEventListener("pointerover", (e) => {
+ const el = e.target.closest("[data-tip]");
+ if (el) showTip(el, e.clientX, e.clientY);
+ });
+ document.addEventListener("pointermove", (e) => {
+ if (!tip.hidden && e.target.closest("[data-tip]")) placeTip(e.clientX, e.clientY);
+ });
+ document.addEventListener("pointerout", (e) => {
+ if (e.target.closest("[data-tip]") && !e.relatedTarget?.closest?.("[data-tip]")) tip.hidden = true;
+ });
+ document.addEventListener("focusin", (e) => {
+ const el = e.target.closest("[data-tip]");
+ if (el) showTip(el);
+ });
+ document.addEventListener("focusout", () => (tip.hidden = true));
+ addEventListener("scroll", () => (tip.hidden = true), { passive: true });
+
+ function taskTip(n, run) {
+ const row = run.rows[n - 1];
+ const o = outcome(row.route, row.label);
+ return (
+ `Task ${n} · ${esc(run.name)} ${esc(D.tasks[n - 1])}` +
+ `${ids.map((id, l) => `${id} ${f2(row.p[l])}`).join(" · ")}` +
+ `label ${ids[row.label]} · likeliest ${ids[row.likeliest]} · at ${D.defaultBar}: ${ids[row.route]} (${OUT_WORD[o]})${row.escalated ? ", moved up by the rule" : ""}`
+ );
+ }
+
+ /* ---------- small SVG helpers ---------- */
+ const lin = (d0, d1, r0, r1) => (v) => r0 + ((v - d0) / (d1 - d0)) * (r1 - r0);
+ const widthOf = (el) => Math.max(280, el.clientWidth);
+
+ /* ---------- hero ---------- */
+ function hero() {
+ const under = runs.reduce((a, r) => a + atDefault(r).under, 0);
+ const misses = runs.reduce((a, r) => a + atDefault(r).under + atDefault(r).over, 0);
+ const offTop = runs.map((r) => N - atDefault(r).routes[ids[TOP]]);
+ const sameOff = offTop.every((v) => v === offTop[0]);
+ $("#hero-stats").innerHTML = `
+
${N} tasks
hand-written sub-agent tasks, each asked once of ${runs.length} decision models
+
${under}
tasks sent to a model too weak for them, by any of the three
Send to ${r.id}? Risk it needed more = ${why} = ${f2(risk)}. ${pass ? "At or under" : "Over"} the bar of ${f2(bar)}.`)}>
+ ${r.id}
+
+ ${f2(risk)} ${pass ? "✓ pass" : "✗"}
+
`;
+ })
+ .join("");
+ const o = outcome(chosen, row.label);
+ let note;
+ if (chosen > row.likeliest) {
+ const why = `The rule moved it up from ${ids[row.likeliest]}: there was a ${f2(row.risk[row.likeliest])} chance that ${ids[row.likeliest]} would be too weak, over the bar.`;
+ note =
+ outcome(row.likeliest, row.label) === "low"
+ ? `${why} Taking the biggest number would have sent it too low.`
+ : `${why} At a bar this strict it goes past its label; here the biggest number would have been right.`;
+ } else if (chosen < row.likeliest) {
+ note = `The bar is loose enough to settle for a cheaper route than the likeliest one.`;
+ } else if (maxP >= D.sureAt) {
+ note = `Jev was sure, so the bar makes little difference: the policy and the biggest number agree.`;
+ } else {
+ note = `The cheapest route that passes is also the likeliest.`;
+ }
+ $("#ex-verdict").innerHTML =
+ `→ Sent to ${ids[chosen]} (${esc(routes[chosen].model)}): ${GLYPH[o]} ${OUT_WORD[o]}` +
+ `${note}`;
+ }
+ $("#presets").addEventListener("click", (e) => {
+ const b = e.target.closest(".btn");
+ if (!b) return;
+ const p = PRESETS[+b.dataset.i];
+ Object.assign(state, { m: p.m, t: p.t, bar: p.bar });
+ sync();
+ });
+ selM.addEventListener("change", () => ((state.m = selM.value), sync()));
+ selT.addEventListener("change", () => ((state.t = +selT.value), sync()));
+ range.addEventListener("input", () => ((state.bar = +range.value), sync()));
+ sync();
+ }
+
+ /* ---------- §6 scoreboard ---------- */
+ function scoreboard() {
+ const metrics = [
+ { k: "Routed as labelled", v: (r) => atDefault(r).agree, f: (v) => `${v} of ${N}`, better: 1, key: true },
+ { k: "Routed too low", v: (r) => atDefault(r).under, f: String, better: -1, key: true },
+ { k: "Routed too high", v: (r) => atDefault(r).over, f: String, better: -1 },
+ { k: "Kept off the top model", v: (r) => N - atDefault(r).routes[ids[TOP]], f: (v) => `${v} (${pct(v)})`, better: 1 },
+ { k: `Sure (≥ ${D.sureAt} on one route)`, v: (r) => r.sure, f: (v) => `${v} of ${N}`, better: 1 },
+ { k: "Mean probability on the label", v: (r) => r.meanOnLabel, f: f2, better: 1, frac: true },
+ { k: "Moved up by the rule", v: (r) => r.escalatedCount, f: String, better: 0 },
+ { k: "Too low if it took the biggest number", v: (r) => r.likeliestWrongLow, f: String, better: -1 },
+ { k: "Median per decision", v: (r) => r.latency.p50, f: ms, better: -1, ratio: true },
+ { k: "p95 per decision", v: (r) => r.latency.p95, f: ms, better: -1, ratio: true },
+ ];
+ $("#scoreboard").innerHTML = runs
+ .map((r) => {
+ const rows = metrics
+ .map((m) => {
+ const v = m.v(r);
+ let d = "";
+ let worse = false;
+ if (r !== ref) {
+ const rv = m.v(ref);
+ if (m.ratio) {
+ d = `×${(v / rv).toFixed(1)}`;
+ worse = v > rv;
+ } else {
+ const diff = v - rv;
+ const near = Math.abs(diff) < (m.frac ? 0.005 : 0.5);
+ d = near ? "=" : `${diff > 0 ? "+" : "−"}${m.frac ? f2(Math.abs(diff)) : Math.abs(diff)}`;
+ worse = !near && m.better !== 0 && Math.sign(diff) === -m.better;
+ }
+ }
+ return `
${m.k}${m.f(v)}${d}
`;
+ })
+ .join("");
+ return `
${esc(r.name)}${r === ref ? 'reference' : ""}
${rows}
`;
+ })
+ .join("");
+
+ const s = runs.map(atDefault);
+ const missRuns = runs.filter((r, i) => s[i].over + s[i].under > 0);
+ const sureText = runs.map((r) => `${r.short} ${r.sure}`).join(", ");
+ const flash = byId.flash;
+ const fastest = [...runs].sort((a, b) => a.latency.p50 - b.latency.p50)[0];
+ const local = runs.filter((r) => r.where === "Local").sort((a, b) => a.latency.p50 - b.latency.p50);
+ $("#findings").innerHTML = [
+ `No model sent a task too low. ${missRuns
+ .map((r) => {
+ const misses = r.rows.map((row, i) => [row, i + 1]).filter(([row]) => row.route !== row.label);
+ return `The only miss, by the ${r.short}, was task ${misses.map(([, n]) => n).join(", ")}, one tier too high.`;
+ })
+ .join(" ")}`,
+ `They differ in certainty. Tasks where a model put ≥ ${D.sureAt} on one route: ${sureText}, out of ${N}.`,
+ `Flash 9B's ${s[runs.indexOf(flash)].agree} of ${N} depends on the routing rule. Its biggest number would have sent ${flash.likeliestWrongLow} tasks to too weak a model; the rule moved them up. For the other two the rule changed nothing.`,
+ `${fastest.name} was fastest, network round trip included. Of the local models, ${local[0].short} answered ×${(local[1].latency.p50 / local[0].latency.p50).toFixed(1)} faster than the ${local[1].short}, at about a third of the download (${local[0].size} against ${local[1].size}).`,
+ ]
+ .map((t) => `
${t}
`)
+ .join("");
+ }
+
+ /* ---------- §7a certainty curves ---------- */
+ let sureKey = "top";
+ function sureChart() {
+ const el = $("#sure-chart");
+ const W = widthOf(el);
+ const narrow = W < 560;
+ const H = narrow ? 300 : 340;
+ const m = { l: 40, r: narrow ? 16 : 150, t: 30, b: 40 };
+ const x = lin(1, N, m.l, W - m.r);
+ const y0 = sureKey === "top" ? 0.3 : 0; // a likeliest route has at least 1/3 of 3 routes
+ const y = lin(y0, 1, H - m.b, m.t);
+ const val = (run, row) => (sureKey === "top" ? Math.max(...row.p) : row.p[row.label]);
+ let g = "";
+ for (const t of sureKey === "top" ? [0.3, 0.5, 0.7, 0.9, 1] : [0, 0.25, 0.5, 0.75, 1]) {
+ g += `${t.toFixed(2)}`;
+ }
+ const refLine = sureKey === "top" ? D.sureAt : 0.5;
+ g += ``;
+ g += ``;
+ g += `${sureKey === "top" ? `sure: ≥ ${D.sureAt}` : "0.50: below it, the label wasn't the biggest number"}`;
+ g += `tasks, each model's sorted from most to least sure →`;
+ let dots = "";
+ let lines = "";
+ let labels = "";
+ const ends = [];
+ runs.forEach((run) => {
+ const pts = run.rows.map((row, i) => ({ n: i + 1, v: val(run, row), run })).sort((a, b) => b.v - a.v || a.n - b.n);
+ const d = pts.map((p, i) => `${i ? "L" : "M"}${x(i + 1).toFixed(1)},${y(p.v).toFixed(1)}`).join("");
+ lines += ``;
+ pts.forEach((p, i) => {
+ dots += `${sureKey === "top" ? "on its likeliest route" : "on the labelled route"}: ${f2(p.v)}`)}/>`;
+ });
+ const count = sureKey === "top" ? pts.filter((p) => p.v >= D.sureAt).length : null;
+ const mean = sureKey === "top" ? null : run.meanOnLabel;
+ ends.push({ run, y: y(pts[pts.length - 1].v), text: sureKey === "top" ? `${run.short}: ${count} sure` : `${run.short}: mean ${f2(mean)}` });
+ });
+ if (!narrow) {
+ ends.sort((a, b) => a.y - b.y);
+ for (let i = 1; i < ends.length; i++) ends[i].y = Math.max(ends[i].y, ends[i - 1].y + 16);
+ ends.forEach((e) => {
+ labels += `${esc(e.text)}`;
+ });
+ }
+ el.innerHTML = `${
+ narrow ? `
` : ""
+ }`;
+ $("#sure-cap").innerHTML =
+ sureKey === "top"
+ ? `Fig. 3The probability each model put on its likeliest route, for each task, sorted. TypeSafe Jev and the 27B sit at the top for most tasks and drop off at the end; Flash 9B declines from the start. Hover or tap a dot for the task and the full answer.`
+ : `Fig. 3The probability each model put on the route the task was labelled with, sorted. Below 0.50 the label wasn't the biggest number, but the rule can still route the task correctly, as it did for Flash 9B's tasks 14–16. Hover or tap a dot for the task.`;
+ }
+
+ /* ---------- §7b task grid ---------- */
+ const differSet = new Set();
+ const unsureSet = new Set();
+ for (let i = 0; i < N; i++) {
+ const rs = new Set(runs.map((r) => r.rows[i].route));
+ const ls = new Set(runs.map((r) => r.rows[i].likeliest));
+ if (rs.size > 1 || ls.size > 1) differSet.add(i + 1);
+ if (runs.some((r) => Math.max(...r.rows[i].p) < 0.7)) unsureSet.add(i + 1);
+ }
+ let gridFilter = "differ";
+ function taskGrid() {
+ const keep = (n) => gridFilter === "all" || (gridFilter === "differ" ? differSet.has(n) : unsureSet.has(n));
+ const head = `
`,
+ )
+ .join("");
+ const miss = byId["27b"].rows[6];
+ const flash = byId.flash;
+ const ts16 = byId.typesafe.rows[15];
+ $("#delta-notes").innerHTML = `
+
+ Task 7 · the only miss
+
"Rename MAX_RETRY … defined and used twice"
+
Labelled ${ids[miss.label]}. The 27B sent it to ${ids[miss.route]} (${miss.p.map(f2).join(" / ")}). The lookup description says "one small change whose exact content is given", and a rename across a definition and two uses is three edits, which may be why. TypeSafe Jev put ${f2(byId.typesafe.rows[6].p[0])} on lookup and Flash 9B ${f2(flash.rows[6].p[0])}.
+
${mini(7, runs)}
+
+
+ Tasks 14, 15, 16 · Flash 9B split, the rule caught it
+
Small, well-specified changes labelled standard
+
Flash 9B split all three between lookup and standard, with lookup slightly ahead (${[13, 14, 15].map((i) => f2(flash.rows[i].p[0])).join(", ")}). The rule moved each up to standard. The 27B was sure of all three; TypeSafe Jev was sure of 14 and 15 and less sure of 16, the README task (${f2(ts16.p[1])} on standard).
${pass ? "pass" : `misses by ${Math.round(L.p95 - D.speedCheckMs)} ms`}
`;
+ })
+ .join("");
+ }
+
+ /* ---------- misc ---------- */
+ function sources() {
+ $("#sources").innerHTML = D.generatedFrom
+ .map((p, i) => `${esc(runs[i].short)}`)
+ .join(", ");
+ }
+ function routeKey() {
+ $("#route-key").innerHTML = routes.map((r, l) => `${r.id}`).join("");
+ }
+ function toggles() {
+ $("#sure-toggle").addEventListener("click", (e) => {
+ const b = e.target.closest(".btn");
+ if (!b) return;
+ sureKey = b.dataset.k;
+ document.querySelectorAll("#sure-toggle .btn").forEach((x) => x.setAttribute("aria-pressed", String(x === b)));
+ sureChart();
+ });
+ $("#grid-filter").addEventListener("click", (e) => {
+ const b = e.target.closest(".btn");
+ if (!b) return;
+ gridFilter = b.dataset.f;
+ document.querySelectorAll("#grid-filter .btn").forEach((x) => x.setAttribute("aria-pressed", String(x === b)));
+ taskGrid();
+ });
+ }
+ function tocAndProgress() {
+ const links = [...document.querySelectorAll("#toc a")];
+ const secs = links.map((a) => document.querySelector(a.getAttribute("href")));
+ const bar = $("#progress");
+ const onScroll = () => {
+ const h = document.documentElement;
+ bar.style.width = `${(h.scrollTop / Math.max(1, h.scrollHeight - h.clientHeight)) * 100}%`;
+ let active = -1;
+ secs.forEach((s, i) => {
+ if (s.getBoundingClientRect().top < innerHeight * 0.35) active = i;
+ });
+ links.forEach((a, i) => a.classList.toggle("active", i === active));
+ };
+ addEventListener("scroll", onScroll, { passive: true });
+ onScroll();
+ }
+
+ /* Charts sized to their container; redrawn when the width changes. */
+ function drawCharts() {
+ sureChart();
+ barPanels();
+ latChart();
+ }
+
+ hero();
+ checks();
+ modelCards();
+ levels();
+ explorer();
+ scoreboard();
+ routeKey();
+ taskGrid();
+ deltaNotes();
+ pairs();
+ sources();
+ latTable();
+ barNotes();
+ toggles();
+ barControls();
+ drawCharts();
+ tocAndProgress();
+
+ let rt;
+ let lastW = innerWidth;
+ addEventListener("resize", () => {
+ if (innerWidth === lastW) return;
+ lastW = innerWidth;
+ clearTimeout(rt);
+ rt = setTimeout(drawCharts, 150);
+ });
+})();
diff --git a/docs/model-router/data.js b/docs/model-router/data.js
new file mode 100644
index 0000000..56ce92e
--- /dev/null
+++ b/docs/model-router/data.js
@@ -0,0 +1,2 @@
+/* Generated by docs/tools/build-model-router-data.mjs from model-router's eval runs. Don't edit. */
+window.MR_DATA = {"generatedFrom":["model-router/eval/results-2026-10-08T18-51-11-416Z.json","model-router/eval/results-2026-10-08T18-41-32-335Z.json","model-router/eval/results-2026-10-08T16-41-27-867Z.json"],"date":"2026-10-08","defaultBar":0.25,"bars":[0.05,0.1,0.15,0.25,0.35,0.5],"slider":[0,0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.11,0.12,0.13,0.14,0.15,0.16,0.17,0.18,0.19,0.2,0.21,0.22,0.23,0.24,0.25,0.26,0.27,0.28,0.29,0.3,0.31,0.32,0.33,0.34,0.35,0.36,0.37,0.38,0.39,0.4,0.41,0.42,0.43,0.44,0.45,0.46,0.47,0.48,0.49,0.5,0.51,0.52,0.53,0.54,0.55,0.56,0.57,0.58,0.59,0.6],"sureAt":0.95,"speedCheckMs":500,"routes":[{"id":"lookup","model":"claude-haiku-5-5","suits":"Finding, reading, listing or summarising what already exists, or making one small change whose exact content is given. Every step is known before starting."},{"id":"standard","model":"claude-sonnet-5-5","suits":"Writing or changing code or text to a clear specification within a few files: a described feature, a bug with a known cause, tests for existing behaviour. The goal is settled, and how to reach it takes some judgment."},{"id":"open_ended","model":"claude-opus-5-5","suits":"Work where the approach itself has to be worked out: a failure with an unknown cause, a design or plan with trade-offs, a change that reaches across many parts of a system, or a review that has to find subtle faults."}],"tasks":["Find every file that imports parseConfig and list the paths with line numbers.","Read CHANGELOG.md and report the date and headline changes of the three most recent releases.","List the environment variables this service reads, with the file each is read in.","In package.json, change the version from 2.3.0 to 2.3.1. Change nothing else.","Run the test suite and report which tests fail, with the first line of each error. Do not fix anything.","Summarise what src/billing/invoice.ts does in five bullet points for someone new to the codebase.","Rename the constant MAX_RETRY to MAX_RETRIES in src/http/client.py, where it is defined and used twice.","Check whether the repo has a LICENSE file and say which licence it is.","Add a --dry-run flag to the sync command. With it set, print each file that would be uploaded and upload nothing. Add a test.","The date parser in utils/dates.py rejects ISO strings with a trailing Z. Accept them as UTC and add tests for both forms.","Write unit tests for the existing slugify function covering empty input, accents, repeated dashes and very long titles.","Add pagination to GET /api/orders: page and per_page query parameters, defaults 1 and 50, and a total count in the response.","Convert the callbacks in src/storage/upload.js to async/await without changing behaviour. The existing tests must still pass.","The settings form submits when Enter is pressed in the search box. Stop that, keeping Enter-to-submit in the other fields.","Add a retry with exponential backoff, at most three attempts, to the webhook sender when the response is a 5xx.","Write the README section for the new export command from its --help output and the two examples in docs/notes.md.","Checkout intermittently double-charges customers, a few times a week, with nothing in the logs. Find the cause.","We need to move the job queue off Redis. Compare the realistic options for our workload and recommend one, with a migration plan.","Memory use of the worker grows by about 50 MB an hour in production but not locally. Work out why.","Review the new authentication middleware for security problems. It handles session tokens, refresh and tenant isolation.","Design how offline editing should work in the mobile app, including conflict resolution when two devices edit the same note.","Split the monolith's billing module into its own service. Work out the boundary, the data it owns and the order to do it in.","Two tests pass alone and fail when run together, in either order. Find what state they share and fix it properly.","The search results page got three times slower after last week's release. Find which change did it and what to do about it."],"runs":[{"id":"typesafe","file":"results-2026-10-08T18-51-11-416Z.json","name":"TypeSafe Jev","short":"TypeSafe","where":"Hosted","size":"Hosted API","note":"Pinned to jev-1.13.0, what jev-latest served on the day. Latency includes the network round trip.","servedModel":"jev-1.13.0","temperature":null,"rows":[{"p":[1,0,0],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0,0,0],"latencyMs":132,"routeAt":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[1,0,0],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0,0,0],"latencyMs":80.8,"routeAt":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9,0.02,0.08],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.1,0.08,0],"latencyMs":115.7,"routeAt":[2,2,2,2,2,2,2,2,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[1,0,0],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0,0,0],"latencyMs":75.2,"routeAt":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9,0.01,0.09],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.09999999999999999,0.09,0],"latencyMs":81.8,"routeAt":[2,2,2,2,2,2,2,2,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.99,0.01,0],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.01,0,0],"latencyMs":101,"routeAt":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.98,0.02,0],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.02,0,0],"latencyMs":111.6,"routeAt":[1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[1,0,0],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0,0,0],"latencyMs":107.3,"routeAt":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0,1,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[1,0,0],"latencyMs":87.5,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.01,0.99,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.99,0,0],"latencyMs":85.2,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0,1,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[1,0,0],"latencyMs":110.8,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.04,0.96,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.96,0,0],"latencyMs":96.5,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0,1,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[1,0,0],"latencyMs":76,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.06,0.9400000000000001,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9400000000000001,0,0],"latencyMs":91.9,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.01,0.99,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.99,0,0],"latencyMs":89.4,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.37,0.63,0],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.63,0,0],"latencyMs":134.6,"routeAt":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":127.9,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":108.1,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":121.4,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":93.4,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":84.7,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":96.2,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0.04,0.96],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,0.96,0],"latencyMs":76,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0,0,1],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[1,1,0],"latencyMs":90.6,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]}],"summaries":[{"bar":0.05,"n":24,"agree":22,"under":0,"over":2,"routes":{"lookup":6,"standard":8,"open_ended":10}},{"bar":0.1,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}},{"bar":0.15,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}},{"bar":0.25,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}},{"bar":0.35,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}},{"bar":0.5,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}}],"latency":{"min":75.2,"p50":93.4,"mean":98.98333333333333,"p95":132,"max":134.6},"sure":20,"meanOnLabel":0.9683333333333334,"escalatedCount":0,"likeliestWrongLow":0},{"id":"flash","file":"results-2026-10-08T18-41-32-335Z.json","name":"Open Jev Flash 9B","short":"Flash 9B","where":"Local","size":"5 GB","note":"Released the day before the run. MLX 4-bit on an Apple M5 Pro.","servedModel":"OpenJev-Flash-9B-MLX-4bit","temperature":"1.07","rows":[{"p":[0.9312,0.0186,0.0502],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0688,0.0502,0],"latencyMs":183.5,"routeAt":[2,2,2,2,2,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9774,0.0097,0.013],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.022699999999999998,0.013,0],"latencyMs":184.6,"routeAt":[2,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9489,0.0226,0.0285],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0511,0.0285,0],"latencyMs":185.7,"routeAt":[2,2,2,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9807,0.0155,0.0038],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0193,0.0038,0],"latencyMs":181,"routeAt":[2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.7743,0.0748,0.1509],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.2257,0.1509,0],"latencyMs":178.8,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9674,0.0144,0.0182],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.032600000000000004,0.0182,0],"latencyMs":179.3,"routeAt":[2,2,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.7898,0.1944,0.0158],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.2102,0.0158,0],"latencyMs":179.2,"routeAt":[2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9619,0.0152,0.0229],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0381,0.0229,0],"latencyMs":184.4,"routeAt":[2,2,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.1571,0.8064,0.0365],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.8429,0.0365,0],"latencyMs":185.1,"routeAt":[2,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.1599,0.8206,0.0195],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.8401,0.0195,0],"latencyMs":182,"routeAt":[2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0626,0.92,0.0173],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9373,0.0173,0],"latencyMs":180.6,"routeAt":[2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.2916,0.6607,0.0477],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.7083999999999999,0.0477,0],"latencyMs":182.3,"routeAt":[2,2,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.4227,0.534,0.0433],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.5773,0.0433,0],"latencyMs":179.1,"routeAt":[2,2,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0]},{"p":[0.543,0.3825,0.0745],"label":1,"likeliest":0,"route":1,"escalated":true,"risk":[0.457,0.0745,0],"latencyMs":180.9,"routeAt":[2,2,2,2,2,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.5377,0.4257,0.0366],"label":1,"likeliest":0,"route":1,"escalated":true,"risk":[0.46230000000000004,0.0366,0],"latencyMs":180.3,"routeAt":[2,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.5377,0.4257,0.0366],"label":1,"likeliest":0,"route":1,"escalated":true,"risk":[0.46230000000000004,0.0366,0],"latencyMs":179.9,"routeAt":[2,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.0191,0.0228,0.9581],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9809,0.9581,0],"latencyMs":177.9,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0553,0.0833,0.8614],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9447000000000001,0.8614,0],"latencyMs":179,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0414,0.0439,0.9147],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9586,0.9147,0],"latencyMs":183.1,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0652,0.0824,0.8524],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9348000000000001,0.8524,0],"latencyMs":188.2,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0377,0.1286,0.8337],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9622999999999999,0.8337,0],"latencyMs":179.8,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0727,0.0817,0.8455],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9272,0.8455,0],"latencyMs":181.4,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.1074,0.2734,0.6193],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.8926999999999999,0.6193,0],"latencyMs":180.2,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.074,0.0658,0.8602],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9259999999999999,0.8602,0],"latencyMs":179.8,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]}],"summaries":[{"bar":0.05,"n":24,"agree":19,"under":0,"over":5,"routes":{"lookup":4,"standard":9,"open_ended":11}},{"bar":0.1,"n":24,"agree":22,"under":0,"over":2,"routes":{"lookup":6,"standard":9,"open_ended":9}},{"bar":0.15,"n":24,"agree":22,"under":0,"over":2,"routes":{"lookup":6,"standard":9,"open_ended":9}},{"bar":0.25,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}},{"bar":0.35,"n":24,"agree":24,"under":0,"over":0,"routes":{"lookup":8,"standard":8,"open_ended":8}},{"bar":0.5,"n":24,"agree":21,"under":3,"over":0,"routes":{"lookup":11,"standard":5,"open_ended":8}}],"latency":{"min":177.9,"p50":180.6,"mean":181.5041666666667,"p95":185.7,"max":188.2},"sure":5,"meanOnLabel":0.7938541666666667,"escalatedCount":3,"likeliestWrongLow":3},{"id":"27b","file":"results-2026-10-08T16-41-27-867Z.json","name":"Open Jev 27B","short":"27B","where":"Local","size":"14 GB","note":"MLX 4-bit on an Apple M5 Pro.","servedModel":"openjev-MLX-4bit","temperature":"0.85","rows":[{"p":[0.9956,0.0006,0.0037],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0043,0.0037,0],"latencyMs":607.4,"routeAt":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9991,0.0004,0.0006],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.001,0.0006,0],"latencyMs":607.6,"routeAt":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9409,0.0016,0.0576],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.059199999999999996,0.0576,0],"latencyMs":605.4,"routeAt":[2,2,2,2,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9962,0.0037,0.0001],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0038,0.0001,0],"latencyMs":595.1,"routeAt":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.8402,0.0799,0.0799],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.1598,0.0799,0],"latencyMs":594.5,"routeAt":[2,2,2,2,2,2,2,2,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.9939,0.0037,0.0024],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0060999999999999995,0.0024,0],"latencyMs":578.2,"routeAt":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.1654,0.8337,0.001],"label":0,"likeliest":1,"route":1,"escalated":false,"risk":[0.8347,0.001,0],"latencyMs":563.6,"routeAt":[2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.9979,0.0006,0.0015],"label":0,"likeliest":0,"route":0,"escalated":false,"risk":[0.0021,0.0015,0],"latencyMs":572.5,"routeAt":[2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"p":[0.0018,0.9905,0.0077],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9982000000000001,0.0077,0],"latencyMs":575.5,"routeAt":[2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0037,0.9949,0.0013],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9962,0.0013,0],"latencyMs":567.9,"routeAt":[2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0028,0.9959,0.0013],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9972,0.0013,0],"latencyMs":565.9,"routeAt":[2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0032,0.9865,0.0103],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9968,0.0103,0],"latencyMs":573.6,"routeAt":[2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0049,0.9767,0.0184],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9951,0.0184,0],"latencyMs":564.4,"routeAt":[2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0434,0.9511,0.0055],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9565999999999999,0.0055,0],"latencyMs":566.7,"routeAt":[2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0024,0.9857,0.012],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9977,0.012,0],"latencyMs":563.5,"routeAt":[2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.0499,0.946,0.0041],"label":1,"likeliest":1,"route":1,"escalated":false,"risk":[0.9501,0.0041,0],"latencyMs":564.8,"routeAt":[2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]},{"p":[0.005,0.0008,0.9942],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.995,0.9942,0],"latencyMs":563.4,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0028,0.0043,0.9929],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9972,0.9929,0],"latencyMs":564.9,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0037,0.0021,0.9942],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9963,0.9942,0],"latencyMs":562.9,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0018,0.0032,0.995],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9982,0.995,0],"latencyMs":579.2,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.001,0.0067,0.9923],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.999,0.9923,0],"latencyMs":565.7,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0089,0.005,0.9861],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9911,0.9861,0],"latencyMs":566.7,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0016,0.0103,0.988],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9983,0.988,0],"latencyMs":564,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]},{"p":[0.0184,0.0049,0.9767],"label":2,"likeliest":2,"route":2,"escalated":false,"risk":[0.9816,0.9767,0],"latencyMs":564.9,"routeAt":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]}],"summaries":[{"bar":0.05,"n":24,"agree":21,"under":0,"over":3,"routes":{"lookup":5,"standard":9,"open_ended":10}},{"bar":0.1,"n":24,"agree":22,"under":0,"over":2,"routes":{"lookup":6,"standard":10,"open_ended":8}},{"bar":0.15,"n":24,"agree":22,"under":0,"over":2,"routes":{"lookup":6,"standard":10,"open_ended":8}},{"bar":0.25,"n":24,"agree":23,"under":0,"over":1,"routes":{"lookup":7,"standard":9,"open_ended":8}},{"bar":0.35,"n":24,"agree":23,"under":0,"over":1,"routes":{"lookup":7,"standard":9,"open_ended":8}},{"bar":0.5,"n":24,"agree":23,"under":0,"over":1,"routes":{"lookup":7,"standard":9,"open_ended":8}}],"latency":{"min":562.9,"p50":566.7,"mean":574.9291666666667,"p95":607.4,"max":607.6},"sure":20,"meanOnLabel":0.9448291666666666,"escalatedCount":0,"likeliestWrongLow":0}],"pairs":[{"a":"typesafe","b":"flash","sameRoute":24,"sameLikeliest":21},{"a":"typesafe","b":"27b","sameRoute":23,"sameLikeliest":23},{"a":"flash","b":"27b","sameRoute":23,"sameLikeliest":20}]};
diff --git a/docs/model-router/index.html b/docs/model-router/index.html
new file mode 100644
index 0000000..14efec9
--- /dev/null
+++ b/docs/model-router/index.html
@@ -0,0 +1,531 @@
+
+
+
+
+
+ Routing With Jev
+
+
+
+
+
+
+
+
+
+
+
+
EXP-04 · model-router · first live run · 2026-10-08
+
Three Jevs pick the model for 24 sub-agent tasks.
+
+ One request to a System One model decides whether a sub-agent runs on Haiku, Sonnet or Opus. We asked
+ three different Jevs, a hosted one and two running on a laptop, to make that call for the same tasks. On
+ the scoreboard they nearly tie. Underneath, they differ a lot in how sure they are and how fast they
+ answer.
+
+
+
+
+
+
+
+
+
+
+
01
+
+
The problem: every sub-agent gets the same model
+
+ An agent harness spawns sub-agents all the time: find the callers of a function, add a flag, chase down
+ a memory leak. Either the orchestrator spends its own tokens deciding which model each one gets, or
+ everything runs on the most capable model. model-router makes that choice with one call, just before
+ the spawn.
+
+
+
+
+
+
+ in
+ Orchestrator spawns a task
+ "Checkout intermittently double-charges customers. Find the cause."
+
+
+
+ 1 · code
+ Pinned?
+ The task names a model → use it, ask nothing.
+
+
+
+ 2 · code
+ A rule?
+ e.g. agent type Explore → lookup.
+
+
+
+ 3 · Jev
+ One Score question
+ A probability for each kind of work.
+
+
+
+ 4 · code
+ Policy
+ The cheapest route whose risk ≤ the bar.
+
+
+
+ out
+ A model from the config
+ haiku sonnet opus
+
+
+
+ Fig. 1This run skipped steps 1 and 2 so that every task was judged by the model. If the decision
+ model fails or times out, the router returns the config's fallback route and never throws.
+
+
+
+
+
+
+
+
+
02
+
+
Hypothesis, and what would confirm it
+
+
+
+ Hypothesis
+ One Jev call over the text of a sub-agent task chooses a model tier well enough that a harness can move
+ a large share of sub-agent work off its most capable model, while
+ rarely sending a task to a model too weak for it, at a latency that is
+ small next to starting the sub-agent.
+
+
+
+
+
+
Check
+
Done when
+
TypeSafe
+
Flash 9B
+
27B
+
Status
+
+
+
+
+
+
+ Table 1Checks from the model-router README, with this run's numbers at the default bar of 0.25.
+ Check 1 needs ≥ 100 labelled tasks from real harness sessions. This run had 24 starter tasks, so its
+ routing numbers are not evidence for it yet.
+
+
+
+
+
+
+
+
+
03
+
+
The three decision models
+
+ Same API, same question, same 24 tasks, each asked once. One model is hosted by TypeSafe; two are
+ Open Jev models running 4-bit on an Apple M5 Pro laptop.
+ Flash 9B was released the day before this run.
+
+
+
+
+
+
+
+
+
+
+
+
04
+
+
What Jev was asked
+
+ Each request carries the task text and one Score question. The levels describe kinds of work, cheapest
+ first. Each maps to a model in the router's config, and Jev never sees a model name:
+ it can judge what a task asks for, but knows nothing about what claude-haiku-5-5 is good
+ at.
+
+
+
+
+
Which description fits the work that task asks a sub-agent to do?
+
+
+
+
+
+
+
+
+
+
05
+
+
How an answer becomes a route
+
+ Jev returns three probabilities that add up to 1. The obvious policy takes the biggest. The router asks
+ something different of each route: if the task is sent here, how likely is it that it needed a more
+ capable model? That's the sum of everything above it. It takes the cheapest route where that risk
+ is at or under the bar. Sending a task too low costs a failed run and a retry; too high
+ only costs money.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Jev's answer
+
+
+
+
+
+ The bar
+
+
+
+
+
+
+
+
+
+ Fig. 2Pick an example, or any task and model. Risk for each route is the probability that the task
+ belonged higher. The most capable route always passes, since nothing is above it. Routes come from
+ model-router's own policy code, replayed at each bar; no new requests are made.
+
+
+
+
+
+
+
+
+
06
+
+
Result: a near tie on the scoreboard
+
At the default bar of 0.25, with differences shown against TypeSafe Jev.
+
+
+
+
+
+
+
+
+
+
+
+
07
+
+
The deltas between the Jevs
+
+ Same routes, different paths to them. The three differ in how much probability they put on an answer,
+ which tasks they hesitated over, and how much they depend on the routing rule to land.
+
+
+
+
+
7aHow sure each model was
+
+
+
+
+
+
+
+
+
+
+ Fig. 3Each dot is one task. Hover or tap a dot for the task and the full answer.
+
+
+
7bEvery task, side by side
+
+
+
+
+
+
+
+
+
+
+
+
+ Fig. 4Each bar is one model's answer, split lookup / standard / open_ended. A mark at the right
+ gives the outcome at the bar of 0.25: ✓ as labelled,
+ ▲ too high, ▼ too low, and
+ ↑ when the rule moved a task above its likeliest route. "Unsure" means some model
+ put less than 0.70 on its likeliest route.
+
+
+
+
+
7cHow often each pair agreed
+
+
+ Fig. 5Out of 24 tasks. "Same route" is after the policy at 0.25; "same likeliest" is what each
+ model ranked first before the policy. The rule closes Flash 9B's gap to the others.
+
+
+
+
+
+
+
+
+
08
+
+
Move the bar
+
+ The bar is arithmetic on answers already returned, so it can be tuned after the fact. Drag it and watch
+ where each model's tasks go. For a sure model it barely matters; for an unsure one it decides the
+ result.
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Tasks routed too low
+
+
+
+ Tasks routed too high
+
+
+
+
+
+ Fig. 6Panels: the 24 tasks in rows by their label, each cell shaded by the route it was sent to at
+ this bar. Charts: how many tasks go too low or too high as the bar moves from 0 to 0.60, one line per
+ model; the vertical rule follows the slider.
+
+
+
+
+
+
+
+
+
+
09
+
+
Speed: one model per lane
+
+ Check 2 asks for p95 ≤ 500 ms per routing decision. Measured around each request, one at a time, after
+ one untimed warm-up.
+
+
+
+
+
+
+
+ Fig. 7Each dot is one request. Ticks mark p50 and p95. The hosted figure includes the network round
+ trip from the laptop; the local models ran on that laptop in ordinary use, and the three runs were spread
+ over two hours. A rough indication of what a routing decision costs, not a benchmark.
+
+
+
+
+
+
Decision model
+
min
+
p50
+
mean
+
p95
+
max
+
Check 2
+
+
+
+
+
+
+
+
+
+
+
+
+
10
+
+
What this does and doesn't show
+
Read it as "the mechanism works", not as a measured accuracy.
+
+
+
+
+ It shows
+
+
The request works end to end on all three decision models.
+
A Score over route descriptions separates clear cases.
+
A model a third of the size can do the job in a third of the time, when the routing rule allows for its doubt.
+
The policy can be tuned after the fact without asking again.
+
+
+
+ It doesn't show
+
+
24 tasks, asked once each. One task is 4 percentage points.
+
One author wrote everything. The tasks, their labels and the route descriptions came from one sitting, so tasks echo the wording they're judged against ("find the cause" against "a failure with an unknown cause").
+
Labels are opinions. Nobody ran these tasks on the three Claude models to see which was enough. That needs a harness (check 4).
+
Flash 9B's 24 of 24 has little margin. On tasks 13 to 16 it was close to an even split between two routes.
+
The tasks don't separate the models. All three are at or near 24 of 24, so this says nothing about which decides best.
+
+
+
+
+ Next, from the README's Open list
+
+
Replace the starter tasks with ≥ 100 sub-agent prompts from real harness sessions, labelled by the model that turned out to be enough.
+
Wire it into a harness's spawn path and log each decision with the sub-agent's outcome.
+
Try a second question for work the tiers don't order, such as "the task needs to read images".
+
Compare with the orchestrator choosing the model itself: agreement, and the tokens it spends deciding.
+
+
+
+
+
+
+
+
+
+
11
+
+
Run details and reproducing it
+
+
+
+
+
+
+
Date
2026-10-08
+
TypeSafe Jev
jev-1.13.0, pinned (what jev-latest served on the day), hosted API
+ Every number on this page is generated from the three recorded runs by
+ docs/tools/build-model-router-data.mjs, using model-router's own policy and summary code,
+ and checked against the summaries each run saved. Sources:
+ . The written report is
+ results-2026-10-08.md.
+