diff --git a/README.md b/README.md index 8ab397d..187e121 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ confirm it, and tests that run without an API key. The first three were picked f | Experiment | The idea | Stack | Status | | --- | --- | --- | --- | | [misconception-gates](misconception-gates/) | Grade a learner's free-text answer by *which misconception* it shows, and jump to the lesson section that fixes it | Node, no dependencies (drops into js-animation-sandbox) | Skeleton | -| [godot-jev](godot-jev/) | A Godot 4 add-on for Jev, and a text adventure whose parser understands anything but can only choose moves the author wrote | Godot 4.7, GDScript | Live: TypeSafe Jev and Open Jev | +| [godot-jev](godot-jev/) | A Godot 4 add-on for Jev, and a text adventure whose parser understands anything but can only choose moves the author wrote | Godot 4.7, GDScript | Live: TypeSafe Jev, Open Jev 27B and Flash 9B, on 16 scripted lines | | [semver-judge](semver-judge/) | Decide the next version from the commits since the last tag, flagging only the uncertain commits that could change the answer | Python, GitHub Action | Skeleton | | [model-router](model-router/) | Pick the model for a sub-agent task in an agent harness: rules first, then a Score over the configured routes, moving up when a weaker model is too likely to fall short | Node, no dependencies | Live: TypeSafe Jev, Open Jev 27B and Flash 9B, on starter tasks only | @@ -53,8 +53,12 @@ cd model-router && node --test && node eval/run.mjs ``` godot-jev and model-router can use a local Open Jev server instead (`JEV_BACKEND=openjev`). -godot-jev's `eval/compare_backends.gd` asks both the same questions, and model-router's -`eval/compare.mjs` puts runs side by side; see their READMEs. +godot-jev's `eval/compare_backends.gd` asks every backend the same questions, and model-router's +`eval/compare.mjs` puts runs side by side; see their READMEs. Both recorded runs are walked +through on the [docs site](https://alteredcraft.github.io/system-one-experiments/). + +Latencies recorded for the local models describe one laptop (4-bit MLX builds on an Apple M5 Pro, +macOS, other applications open), not how fast those models are. ## Limits that apply to all of them diff --git a/docs/README.md b/docs/README.md index 230832d..6e6e927 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,9 +10,11 @@ folder `/docs`. | Path | What it is | | --- | --- | | `index.html` | Landing page: what System One models are, the four experiments, the protocol, the notebook log | -| `assets/` | Shared styles (`site.css`) and the theme toggle (`site.js`) | -| `model-router/` | The 2026-10-08 model-router run: `index.html`, `results.css`, `app.js`, and generated `data.js` | +| `assets/` | Shared by every page: styles (`site.css`) and the theme toggle (`site.js`). Shared by the results pages: `results.css`, and `results.js` (tooltips, chart helpers, the side contents) | +| `model-router/` | The 2026-10-08 model-router run: `index.html`, `app.js`, and generated `data.js` | +| `godot-jev/` | The 2026-10-08 three-model godot-jev run: `index.html`, `page.css`, `app.js`, a screenshot of the game, and generated `data.js` | | `tools/build-model-router-data.mjs` | Builds `model-router/data.js` | +| `tools/build-godot-jev-data.gd` | Builds `godot-jev/data.js` | `model-router/data.js` is generated, not written by hand. The script reads the three recorded runs in `model-router/eval/results-2026-10-08T*.json`, routes every answer with model-router's own policy @@ -23,4 +25,15 @@ ones each run saved. After a change to those runs or to the policy, regenerate i node docs/tools/build-model-router-data.mjs ``` +`godot-jev/data.js` is generated the same way, by a Godot script, because the policy it replays is +the game's parser. It reads `godot-jev/eval/results-2026-10-08T134205.json`, puts every recorded +answer back through `demo/parser.gd` in the game state where it was asked, at the run's thresholds +and at every setting the page's sliders reach, and refuses to write if an outcome or a summary +differs from what the run saved: + +```bash +godot --headless --path godot-jev --import +godot --headless --path godot-jev --script ../docs/tools/build-godot-jev-data.gd +``` + To preview locally: `python3 -m http.server --directory docs`. diff --git a/docs/model-router/results.css b/docs/assets/results.css similarity index 96% rename from docs/model-router/results.css rename to docs/assets/results.css index fa6aba2..ee3834b 100644 --- a/docs/model-router/results.css +++ b/docs/assets/results.css @@ -1,4 +1,4 @@ -/* model-router results page. Shared tokens are in ../assets/site.css. */ +/* Shared by the results pages in docs/. Tokens are in site.css. */ @media (max-width: 520px) { .hide-xs { @@ -1210,6 +1210,37 @@ input[type="range"] { } } +/* A note the reader needs before the figures beside it. */ +.callout { + background: color-mix(in srgb, var(--warning) 8%, var(--surface)); + border: 1px solid color-mix(in srgb, var(--warning) 45%, var(--rule)); + border-left-width: 3px; + border-radius: var(--radius); + padding: 16px 20px; + margin: 0 0 20px; + max-width: 900px; + font-size: 0.95rem; + color: var(--ink-2); +} +.callout .label { + display: block; + margin-bottom: 6px; + color: var(--warning-text); +} +.callout p { + margin: 0 0 0.6em; +} +.callout p:last-child { + margin-bottom: 0; +} +.callout b { + color: var(--ink); + font-weight: 600; +} +.hero-stats .k a { + white-space: nowrap; +} + /* Caveats and details */ .caveats { display: grid; diff --git a/docs/assets/results.js b/docs/assets/results.js new file mode 100644 index 0000000..4acd246 --- /dev/null +++ b/docs/assets/results.js @@ -0,0 +1,158 @@ +/* Shared by the results pages: tooltips, the scoreboard, the latency lanes, + chart scales, the side contents with the reading-progress bar, and + redrawing charts on resize. + Load it after the page's #tip element and before the page's own script. */ + +window.S1X = (function () { + "use strict"; + + const $ = (sel) => document.querySelector(sel); + const esc = (s) => String(s).replace(/[&<>"]/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """ })[c]); + const f2 = (x) => x.toFixed(2); + const mColor = (run) => `var(--m-${run.id})`; + const swatch = (run) => ``; + + /* ---------- tooltip: any element with data-tip ---------- */ + const tip = $("#tip"); + const tipAttr = (html) => `data-tip="${esc(html)}" tabindex="0"`; + function placeTip(x, y) { + const pad = 14; + const w = tip.offsetWidth; + const h = tip.offsetHeight; + let left = x + pad; + let top = y + pad; + if (left + w > innerWidth - 8) left = Math.max(8, x - w - pad); + if (top + h > innerHeight - 8) top = Math.max(8, y - h - pad); + tip.style.left = `${left}px`; + tip.style.top = `${top}px`; + } + function showTip(el, x, y) { + tip.innerHTML = el.dataset.tip; + tip.hidden = false; + if (x == null) { + const r = el.getBoundingClientRect(); + x = r.left + r.width / 2; + y = r.bottom; + } + placeTip(x, y); + } + document.addEventListener("pointerover", (e) => { + const el = e.target.closest("[data-tip]"); + if (el) showTip(el, e.clientX, e.clientY); + }); + document.addEventListener("pointermove", (e) => { + if (!tip.hidden && e.target.closest("[data-tip]")) placeTip(e.clientX, e.clientY); + }); + document.addEventListener("pointerout", (e) => { + if (e.target.closest("[data-tip]") && !e.relatedTarget?.closest?.("[data-tip]")) tip.hidden = true; + }); + document.addEventListener("focusin", (e) => { + const el = e.target.closest("[data-tip]"); + if (el) showTip(el); + }); + document.addEventListener("focusout", () => (tip.hidden = true)); + addEventListener("scroll", () => (tip.hidden = true), { passive: true }); + + /* ---------- small SVG helpers ---------- */ + const lin = (d0, d1, r0, r1) => (v) => r0 + ((v - d0) / (d1 - d0)) * (r1 - r0); + const widthOf = (el) => Math.max(280, el.clientWidth); + + /* One card per run, each metric beside its difference from `ref`. A metric is + { k, v(run), f(value), better: 1 | -1 | 0 } with optional key, frac + (a fraction, shown to two places) and ratio (shown as a multiple of ref). */ + function scoreCards(runs, ref, metrics) { + return runs + .map((r) => { + const rows = metrics + .map((m) => { + const v = m.v(r); + let d = ""; + let worse = false; + if (r !== ref) { + const rv = m.v(ref); + if (m.ratio) { + d = `×${(v / rv).toFixed(1)}`; + worse = v > rv; + } else { + const diff = v - rv; + const near = Math.abs(diff) < (m.frac ? 0.005 : 0.5); + d = near ? "=" : `${diff > 0 ? "+" : "−"}${m.frac ? f2(Math.abs(diff)) : Math.abs(diff)}`; + worse = !near && m.better !== 0 && Math.sign(diff) === -m.better; + } + } + return `
${m.k}${m.f(v)}${d}
`; + }) + .join(""); + return `

${esc(r.name)}${r === ref ? 'reference' : ""}

${rows}
`; + }) + .join(""); + } + + /* One lane per run with a dot per row, ticks at the run's latency.p50 and + latency.p95, and the speed check as a dashed line. `time(row)` is a row's + milliseconds, `tip(run, row, i)` its tooltip, `per` what was timed. */ + function laneChart(el, runs, { time, tip, per, checkMs }) { + const W = widthOf(el); + const narrow = W < 560; + const rowH = 84; + const m = { l: narrow ? 70 : 130, r: 16, t: 26, b: 36 }; + const H = m.t + rowH * runs.length + m.b; + const maxMs = Math.ceil(Math.max(...runs.map((r) => r.latency.max), checkMs) / 100) * 100; + const x = lin(0, maxMs, m.l, W - m.r); + let g = ""; + for (let t = 0; t <= maxMs; t += 100) { + g += `${t}`; + } + g += `ms per ${per} →`; + g += `check 2: p95 ≤ ${checkMs} ms`; + let body = ""; + runs.forEach((run, k) => { + const cy = m.t + rowH * k + rowH / 2; + body += `${esc(narrow ? run.short : run.name)}`; + body += ``; + run.rows.forEach((row, i) => { + const jitter = ((i * 7) % 5) - 2; + body += ``; + }); + // p50 labelled above its tick, p95 below, so close ticks don't collide. + for (const [k2, v, ty] of [ + ["p50", run.latency.p50, cy - 24], + ["p95", run.latency.p95, cy + 32], + ]) { + body += `${k2}`; + } + }); + el.innerHTML = `${g}${body}`; + } + + function tocAndProgress() { + const links = [...document.querySelectorAll("#toc a")]; + const secs = links.map((a) => document.querySelector(a.getAttribute("href"))); + const bar = $("#progress"); + const onScroll = () => { + const h = document.documentElement; + bar.style.width = `${(h.scrollTop / Math.max(1, h.scrollHeight - h.clientHeight)) * 100}%`; + let active = -1; + secs.forEach((s, i) => { + if (s.getBoundingClientRect().top < innerHeight * 0.35) active = i; + }); + links.forEach((a, i) => a.classList.toggle("active", i === active)); + }; + addEventListener("scroll", onScroll, { passive: true }); + onScroll(); + } + + /* Charts are sized to their container, so they are redrawn when the width changes. */ + function redrawOnResize(draw) { + let timer; + let lastW = innerWidth; + addEventListener("resize", () => { + if (innerWidth === lastW) return; + lastW = innerWidth; + clearTimeout(timer); + timer = setTimeout(draw, 150); + }); + } + + return { $, esc, f2, mColor, swatch, tipAttr, lin, widthOf, scoreCards, laneChart, tocAndProgress, redrawOnResize }; +})(); diff --git a/docs/godot-jev/app.js b/docs/godot-jev/app.js new file mode 100644 index 0000000..5463b3c --- /dev/null +++ b/docs/godot-jev/app.js @@ -0,0 +1,808 @@ +/* godot-jev results page. Draws window.GJ_DATA (data.js), which + docs/tools/build-godot-jev-data.gd generates by replaying the recorded run + through the game's own parser. Nothing here re-decides a turn: the outcome + at any pair of thresholds comes from rows[].actMax and rows[].clarifyMax, + the highest slider settings at which the parser still acted or still asked. + Tooltips and chart helpers are in ../assets/results.js. */ + +(function () { + "use strict"; + + const D = window.GJ_DATA; + const runs = D.runs; + const lines = D.lines; + const L = lines.length; + const R = D.rounds; + const N = L * R; + const ACT = D.defaultAct; + const CLARIFY = D.defaultClarify; + const LAST = D.slider.length - 1; + const byId = Object.fromEntries(runs.map((r) => [r.id, r])); + const ref = runs[0]; // TypeSafe Jev: the reference for deltas + const reduceMotion = matchMedia("(prefers-reduced-motion: reduce)").matches; + + const { $, esc, f2, mColor, swatch, tipAttr, lin, widthOf, scoreCards, laneChart, tocAndProgress, redrawOnResize } = window.S1X; + const ms = (x) => `${Math.round(x).toLocaleString("en-US")} ms`; + const mean = (xs) => xs.reduce((a, b) => a + b, 0) / xs.length; + + const rowAt = (run, line, round) => run.rows[line * R + round]; + const rowsOf = (run, line) => run.rows.slice(line * R, line * R + R); + const meant = (line) => lines[line].means || D.none; + const textOf = (line, id) => (id === D.none ? "none of these" : (lines[line].options.find((o) => o.id === id)?.text ?? id)); + + /* The parser's outcome for a recorded answer with the sliders at a (act) and c (clarify). */ + const KINDS = ["unknown", "clarify", "act"]; // also the order of the shade ramp, --r-0 to --r-2 + const kindAt = (row, a, c) => (a <= row.actMax ? "act" : c <= row.clarifyMax ? "clarify" : "unknown"); + const kColor = (kind) => `var(--r-${KINDS.indexOf(kind)})`; + const kInk = (kind) => `var(--r-ink-${KINDS.indexOf(kind)})`; + + /* How the turn went for the player: ok, ask, dead (a dead end) or wrong. */ + function turnAt(row, a, c) { + const kind = kindAt(row, a, c); + if (row.lands[kind]) return kind === "clarify" ? "ask" : "ok"; + return kind === "unknown" ? "dead" : "wrong"; + } + const GLYPH = { ok: "✓", ask: "?", dead: "▼", wrong: "✗" }; + const O_CLASS = { ok: "o-agree", ask: "o-high", dead: "o-low", wrong: "o-low" }; + function turnWord(row, a, c) { + const kind = kindAt(row, a, c); + const turn = turnAt(row, a, c); + if (turn === "ok") return kind === "act" ? "acted on the move the line means" : "unknown, as it should be"; + if (turn === "ask") return "asked, offering the move the line means"; + if (turn === "dead") return "unknown: a dead end"; + return kind === "act" ? "acted on a move the line doesn't mean" : "asked without offering the move"; + } + const countsAt = (run, a, c) => { + const n = { ok: 0, ask: 0, dead: 0, wrong: 0 }; + for (const row of run.rows) n[turnAt(row, a, c)] += 1; + return n; + }; + const mark = (row, a, c) => { + const t = turnAt(row, a, c); + return `${GLYPH[t]}`; + }; + + function lineTip(line, run) { + const rows = rowsOf(run, line); + const first = rows[0]; + return ( + `Line ${line + 1} · ${esc(run.name)}
"${esc(lines[line].say)}"` + + `means: ${esc(lines[line].meansText || "nothing the author wrote")}` + + `${first.ranked.map(([id, p]) => `${esc(textOf(line, id))} ${f2(p)}`).join(" · ")}` + + `rounds: ${rows.map((r) => `${kindAt(r, ACT, CLARIFY)} ${f2(r.confidence)}`).join(" · ")}` + ); + } + + /* ---------- hero ---------- */ + function hero() { + const at = runs.map((r) => countsAt(r, ACT, CLARIFY)); + const wrong = at.reduce((a, c) => a + c.wrong + c.dead, 0); + const allTopRight = runs.every((r) => r.rows.every((row) => row.top === meant(row.line))); + const asks = at.reduce((a, c) => a + c.ask, 0); + const askers = runs.filter((r, i) => at[i].ask > 0); + const askLines = new Set(askers.flatMap((r) => r.rows.filter((row) => turnAt(row, ACT, CLARIFY) === "ask").map((row) => row.line))); + $("#hero-stats").innerHTML = ` +
${L} lines × ${R}
lines of slang that take the dungeon from the stairs to the crown, each asked ${R} times of ${runs.length} decision models
+
${wrong}
wrong moves or dead ends in ${N * runs.length} turns.${allTopRight ? " Every model ranked the right option first, every time." : ""}
+
${asks}
turns where the game had to ask "did you mean?": all ${askers.map((r) => esc(r.short)).join(" and ")}, on ${askLines.size} of the ${L} lines
+
${runs + .map((r) => `${swatch(r)}${r.latency.p50}`) + .join("")}ms
median time per turn in this run: TypeSafe hosted, then Flash 9B and 27B on one laptop. Not the models' speed.
`; + $("#legend-models").innerHTML = runs + .map((r) => `${swatch(r)}${esc(r.name)} · ${r.where === "Local" ? `local, ${esc(r.size)}` : "hosted"}`) + .join(""); + } + + /* ---------- checks ---------- */ + function checks() { + const speedPass = runs.map((r) => r.latency.p95 <= D.speedCheckMs); + const speedStatus = runs.filter((r, i) => !speedPass[i]).map((r) => `${r.short} misses by ${r.latency.p95 - D.speedCheckMs} ms`); + const rowsHtml = [ + [ + "1 · Understanding", + "On ≥ 100 player inputs, ≥ 90% land on the intended move or a clarify that offers it, and ≥ 90% of impossible ones come back unknown", + runs.map((r) => `${r.summary.landed} of ${N} landed`), + `Not yet evidence
${L} lines in one voice, one of them impossible`, + ], + [ + "2 · Speed", + `p95 ≤ ${D.speedCheckMs} ms per turn`, + runs.map((r, i) => `${ms(r.latency.p95)}`), + `${speedStatus.length ? `${runs.length - speedStatus.length} of ${runs.length} pass. ${speedStatus.join("; ")}.` : "All pass."}
on this setup: see the note under Speed`, + ], + [ + "3 · Nothing invented", + "The options offered are exactly the possible moves, plus none", + runs.map(() => `by construction`), + `Pass
a unit test over the demo's states`, + ], + [ + "4 · Shippable", + "An exported build works with no key in the client, through a small proxy", + runs.map(() => `—`), + `Not started: needs the proxy`, + ], + ]; + $("#checks-table tbody").innerHTML = rowsHtml + .map(([c, done, cells, status]) => `${c}${done}${cells.map((x) => `${x}`).join("")}${status}`) + .join(""); + } + + /* ---------- model cards ---------- */ + function modelCards() { + $("#model-cards").innerHTML = runs + .map( + (r) => ` +
+ ${esc(r.where)} +

${esc(r.name)}

+
+
Served model
${esc(r.servedModel)}
+
Size
${esc(r.size)}
+ ${r.temperature ? `
Calibration
T=${esc(r.temperature)}
` : ""} +
Median
${ms(r.latency.p50)} per turn${r.where === "Local" ? ", on this laptop" : ", network included"}
+
Rounds
${r.sameEveryRound ? "the same answer to a line every round" : "confidence moved a little between rounds"}
+
+

${esc(r.note)}

+
`, + ) + .join(""); + } + + /* ---------- §4 the question ---------- */ + function question() { + const steps = $("#q-steps"); + steps.innerHTML = lines.map((_, i) => ``).join(""); + let at = 8; + function render() { + const line = lines[at]; + steps.querySelectorAll(".btn").forEach((b) => b.setAttribute("aria-pressed", String(+b.dataset.i === at))); + $("#q-say").innerHTML = `${at + 1}. ${esc(line.say)}· in the ${esc(line.room)}`; + // Godot writes the request with its keys sorted, so this is the order on the wire. + const state = { ...line.state, player_typed: line.say }; + const sorted = Object.fromEntries(Object.keys(state).sort().map((k) => [k, state[k]])); + $("#q-state").textContent = JSON.stringify(sorted, null, 2); + $("#q-text").textContent = D.question; + $("#q-options").innerHTML = + line.options + .map( + (o) => + `
  • ${esc(o.id)}${esc(o.text)}${o.id === line.means ? 'what this line means' : ""}
    • `, + ) + .join("") + + `
  • ${esc(D.none)}${esc(D.noneText)}${line.means ? "" : 'what this line means: nothing the author wrote'}
    • `; + } + steps.addEventListener("click", (e) => { + const b = e.target.closest(".btn"); + if (!b) return; + at = +b.dataset.i; + render(); + }); + render(); + const offered = lines.map((l) => l.options.length + 1); + $("#q-cap").innerHTML = + `Fig. 3The request for each of the ${L} lines, as the game built it. The list runs from ${Math.min(...offered)} to ${Math.max(...offered)} options along the route, ` + + `counting "none of these". The model's answer can only be one of them.`; + } + + /* ---------- an answer as a stacked bar: the top three, shaded by what they are ---------- */ + function role(line, id) { + if (id === meant(line)) return 2; + return id === D.none ? 0 : 1; + } + function segments(row, { big = false } = {}) { + const rest = Math.max(0, 1 - row.ranked.reduce((a, [, p]) => a + p, 0)); + const segs = row.ranked + .map(([id, p]) => { + if (p < 0.005) return ""; + const l = role(row.line, id); + const label = big && p >= 0.14 ? `${p >= 0.3 ? esc(textOf(row.line, id)) + " " : ""}${f2(p)}` : ""; + return `
    ${label}
    `; + }) + .join(""); + return segs + (rest >= 0.005 ? `
    ` : ""); + } + const stack = (row) => `
    ${segments(row)}
    `; + + /* ---------- §5 explorer ---------- */ + function explorer() { + const PRESETS = [ + { m: "flash", l: 5, act: ACT, clarify: CLARIFY, txt: "Line 6 · Flash 9B: asks" }, + { m: "flash", l: 6, act: ACT, clarify: CLARIFY, txt: "Line 7 · Flash 9B: 0.01 short" }, + { m: "27b", l: 8, act: ACT, clarify: CLARIFY, txt: "Line 9 · 27B: sure" }, + { m: "flash", l: 0, act: ACT, clarify: CLARIFY, txt: "Line 1 · Flash 9B: the gin line" }, + { m: "typesafe", l: 7, act: 90, clarify: CLARIFY, txt: "Line 8 · TypeSafe: a stricter game" }, + ]; + const selM = $("#ex-model"); + const selL = $("#ex-line"); + const selR = $("#ex-round"); + const actR = $("#ex-act"); + const clarR = $("#ex-clarify"); + selM.innerHTML = runs.map((r) => ``).join(""); + selL.innerHTML = lines.map((l, i) => ``).join(""); + selR.innerHTML = Array.from({ length: R }, (_, i) => ``).join(""); + $("#presets").innerHTML = PRESETS.map((p, i) => ``).join(""); + + const state = { m: "flash", l: 5, round: 0, act: ACT, clarify: CLARIFY }; + function sync() { + selM.value = state.m; + selL.value = String(state.l); + selR.value = String(state.round); + actR.value = String(state.act); + clarR.value = String(state.clarify); + document.querySelectorAll("#presets .btn").forEach((b) => { + const p = PRESETS[+b.dataset.i]; + b.setAttribute("aria-pressed", String(p.m === state.m && p.l === state.l && p.act === state.act && p.clarify === state.clarify && state.round === 0)); + }); + render(); + } + function render() { + const run = byId[state.m]; + const row = rowAt(run, state.l, state.round); + const line = lines[state.l]; + const act = D.slider[state.act]; + const clarify = D.slider[state.clarify]; + const kind = kindAt(row, state.act, state.clarify); + const none = row.top === D.none; + $("#ex-act-out").textContent = f2(act); + $("#ex-clarify-out").textContent = f2(clarify); + $("#ex-tasktext").innerHTML = `${state.l + 1}. ${esc(line.say)}· means: ${esc(line.meansText || "nothing the author wrote")}`; + $("#ex-stack").innerHTML = segments(row, { big: true }); + $("#ex-top3").innerHTML = row.ranked + .map(([id, p]) => `
  • ${esc(textOf(state.l, id))}${f2(p)}
    • `) + .join(""); + $("#ex-likeliest").innerHTML = + `Chose ${esc(textOf(state.l, row.top))} with a confidence of ${f2(row.confidence)}. ` + + `${esc(run.name)} ${row.confidence >= D.sureAt ? "was sure." : row.confidence < act ? "was not sure enough to act on it." : "leaned clearly one way."}`; + + const rungs = [ + { + kind: "act", + threshold: act, + pass: state.act <= row.actMax, + why: `Act if the confidence, ${f2(row.confidence)}, is at least ${f2(act)}.`, + }, + { + kind: "clarify", + threshold: clarify, + pass: state.clarify <= row.clarifyMax, + why: `Otherwise ask if it is at least ${f2(clarify)}.`, + }, + ]; + $("#ex-ladder").innerHTML = + rungs + .map((g) => { + const res = none ? `chose "none" ✗` : `${f2(row.confidence)} ${g.pass ? "✓ pass" : "✗"}`; + return ` +
    ${g.kind === "act" ? "Act" : "Ask"}?
    ${g.why}${none ? '
    The choice is "none of these", which is unknown at any confidence.' : ""}`)}> + ${g.kind} + + ${res} +
    `; + }) + .join("") + + `
    Unknown
    What is left when neither threshold is met, or when the choice is "none of these".`)}> + unknown + + ${kind === "unknown" ? "what's left" : "not needed"} +
    `; + + const turn = turnAt(row, state.act, state.clarify); + const verdictMark = `${GLYPH[turn]} ${turnWord(row, state.act, state.clarify)}`; + let head; + let note; + if (kind === "act") { + head = `→ Acts: ${esc(textOf(state.l, row.top))}`; + note = "The game applies the move and prints the response the author wrote for it."; + } else if (kind === "clarify") { + head = `→ Asks: Did you mean: ${row.asks.map((id, i) => `${i + 1}. ${esc(textOf(state.l, id))}`).join("  ")}`; + note = + state.act - 1 <= row.actMax + ? `The right move is option 1, and the confidence is ${f2(act - row.confidence)} under the act threshold: one step lower and the game acts.` + : "The player answers with one keypress. Nothing in the game has changed yet."; + } else if (none) { + head = `→ Unknown: the game prints an authored line such as "Nothing here answers to that."`; + note = `The model chose "none of these" at ${f2(row.ranked[0][1])}. That is unknown whatever the thresholds are.`; + } else { + head = `→ Unknown: the game prints an authored line such as "Nothing here answers to that."`; + note = `The model's top choice was right, at ${f2(row.confidence)}, but the ask threshold is above it, so the player gets no question to answer.`; + } + $("#ex-verdict").innerHTML = `${head} ${verdictMark}${note}`; + } + $("#presets").addEventListener("click", (e) => { + const b = e.target.closest(".btn"); + if (!b) return; + const p = PRESETS[+b.dataset.i]; + Object.assign(state, { m: p.m, l: p.l, round: 0, act: p.act, clarify: p.clarify }); + sync(); + }); + selM.addEventListener("change", () => ((state.m = selM.value), sync())); + selL.addEventListener("change", () => ((state.l = +selL.value), sync())); + selR.addEventListener("change", () => ((state.round = +selR.value), sync())); + actR.addEventListener("input", () => { + state.act = +actR.value; + state.clarify = Math.min(state.clarify, state.act); + sync(); + }); + clarR.addEventListener("input", () => { + state.clarify = +clarR.value; + state.act = Math.max(state.act, state.clarify); + sync(); + }); + sync(); + } + + /* ---------- §6 scoreboard ---------- */ + function scoreboard() { + const at = (r) => countsAt(r, ACT, CLARIFY); + const metrics = [ + { k: "Lines landed", v: (r) => r.summary.landed, f: (v) => `${v} of ${N}`, better: 1, key: true }, + { k: "Wrong moves and dead ends", v: (r) => at(r).wrong + at(r).dead, f: String, better: -1, key: true }, + { k: "Acted at once", v: (r) => r.summary.act, f: String, better: 1 }, + { k: 'Asked "did you mean?"', v: (r) => r.summary.clarify, f: String, better: -1 }, + { k: "Unknown (the gin line)", v: (r) => r.summary.unknown, f: String, better: 0 }, + { k: `Sure (confidence ≥ ${D.sureAt})`, v: (r) => r.sure, f: (v) => `${v} of ${N}`, better: 1 }, + { k: "Mean confidence", v: (r) => r.meanConfidence, f: f2, better: 1, frac: true }, + { k: "Lowest confidence in a move", v: (r) => r.lowestOnAMove, f: f2, better: 1, frac: true }, + { k: "Median per turn, this setup", v: (r) => r.latency.p50, f: ms, better: -1, ratio: true }, + { k: "p95 per turn, this setup", v: (r) => r.latency.p95, f: ms, better: -1, ratio: true }, + ]; + $("#scoreboard").innerHTML = scoreCards(runs, ref, metrics); + + const moves = lines.filter((l) => l.means).length; + const askers = runs.filter((r) => r.summary.clarify > 0); + const askText = askers + .map((r) => { + const asked = [...new Set(r.rows.filter((row) => kindAt(row, ACT, CLARIFY) === "clarify").map((row) => row.line))]; + return `${r.name} asked on ${asked.length} of the ${moves} lines that mean a move (${asked + .map((l) => `line ${l + 1} at ${f2(rowAt(r, l, 0).confidence)}`) + .join(", ")}), in each of the ${R} rounds, and offered the right move first both times.`; + }) + .join(" "); + const local = runs.filter((r) => r.where === "Local").sort((a, b) => a.latency.p50 - b.latency.p50); + $("#findings").innerHTML = [ + `No wrong move in ${N * runs.length} turns. Every model ranked the move the line means first on every turn, and chose "none of these" for the gin line every time.`, + `They differ in certainty. Turns with a confidence of ${D.sureAt} or more: ${runs.map((r) => `${r.short} ${r.sure}`).join(", ")}, out of ${N}.`, + `Less certainty shows up as questions. ${askText} The other two acted on every one.`, + `On this setup the hosted model and ${local[0].short} took about the same time, a median of ${ms(ref.latency.p50)} and ${ms(local[0].latency.p50)}, and the ${local[1].short} took about ${(local[1].latency.p50 / local[0].latency.p50).toFixed(1)} times as long as ${local[0].short}. These are timings from one laptop with other things running, and not a measure of how fast the models are.`, + ] + .map((t) => `
  • ${t}
    • `) + .join(""); + } + + /* ---------- §7a confidence along the route ---------- */ + let sureKey = "route"; + function sureChart() { + const el = $("#sure-chart"); + const W = widthOf(el); + const narrow = W < 560; + const H = narrow ? 300 : 340; + const m = { l: 40, r: narrow ? 16 : 150, t: 30, b: 40 }; + const x = lin(1, L, m.l + 8, W - m.r - 8); + const y = lin(0.3, 1, H - m.b, m.t); + let g = ""; + for (const t of [0.3, 0.5, 0.7, 0.9, 1]) { + g += `${t.toFixed(2)}`; + } + for (const [v, word] of [ + [D.thresholds.act, "act"], + [D.thresholds.clarify, "ask"], + ]) { + g += ``; + g += `${word} at ≥ ${f2(v)}`; + } + if (sureKey === "route") { + for (let i = 1; i <= L; i++) g += `${i}`; + g += `line, in the order the route types them →`; + } else { + g += `lines, each model's sorted from most to least sure →`; + } + let dots = ""; + let paths = ""; + let labels = ""; + const ends = []; + runs.forEach((run) => { + let pts = lines.map((_, i) => ({ line: i, v: mean(rowsOf(run, i).map((r) => r.confidence)) })); + if (sureKey === "sorted") pts = pts.sort((a, b) => b.v - a.v || a.line - b.line); + const d = pts.map((p, i) => `${i ? "L" : "M"}${x(i + 1).toFixed(1)},${y(p.v).toFixed(1)}`).join(""); + paths += ``; + pts.forEach((p, i) => { + dots += ``; + }); + ends.push({ run, y: y(pts[pts.length - 1].v), text: `${run.short}: mean ${f2(run.meanConfidence)}` }); + }); + if (!narrow) { + ends.sort((a, b) => a.y - b.y); + for (let i = 1; i < ends.length; i++) ends[i].y = Math.max(ends[i].y, ends[i - 1].y + 16); + ends.forEach((e) => { + labels += `${esc(e.text)}`; + }); + } + el.innerHTML = `${g}${paths}${dots}${labels}${ + narrow ? `
    ${ends.map((e) => `${swatch(e.run)}${esc(e.text)}`).join("")}
    ` : "" + }`; + const flash = byId.flash; + $("#sure-cap").innerHTML = + sureKey === "route" + ? `Fig. 5The confidence each model reported for its choice on each line, averaged over the ${R} rounds. Line 1 is the gin line, where the choice is "none of these". Flash 9B dips under the act threshold on lines 6 and 7 (${f2(rowAt(flash, 5, 0).confidence)} and ${f2(rowAt(flash, 6, 0).confidence)}); the other two stay above it throughout. Hover or tap a dot for the line and the full answer.` + : `Fig. 5The same confidences, each model's sorted. The 27B sits near 1.00 for almost every line, TypeSafe Jev drops to ${f2(ref.lowestOnAMove)} at its least sure, and Flash 9B declines from the start. Hover or tap a dot for the line.`; + } + + /* ---------- §7b line grid ---------- */ + const differSet = new Set(); + const unsureSet = new Set(); + for (let i = 0; i < L; i++) { + const kinds = new Set(runs.flatMap((r) => rowsOf(r, i).map((row) => kindAt(row, ACT, CLARIFY)))); + if (kinds.size > 1) differSet.add(i); + if (runs.some((r) => rowsOf(r, i).some((row) => row.confidence < D.thresholds.act))) unsureSet.add(i); + } + let gridFilter = "all"; + function lineGrid() { + const keep = (i) => gridFilter === "all" || (gridFilter === "differ" ? differSet.has(i) : unsureSet.has(i)); + const head = `
    #The player typesMeans${runs + .map((r) => `${swatch(r)}${esc(r.short)}`) + .join("")}
    `; + const rows = lines + .map((line, i) => { + if (!keep(i)) return ""; + const cells = runs + .map((r) => { + const row = rowAt(r, i, 0); + return `
    ${esc(r.short)}${stack(row)}${mark(row, ACT, CLARIFY)}
    `; + }) + .join(""); + return `
    ${i + 1}${esc(line.say)}${esc(line.room)} · ${line.options.length + 1} options${esc(line.meansText || "nothing the author wrote")}${cells}
    `; + }) + .join(""); + $("#taskgrid").innerHTML = head + rows; + } + + function deltaNotes() { + const mini = (i) => + runs + .map((r) => `
    ${swatch(r)}${esc(r.short)}${stack(rowAt(r, i, 0))}${f2(rowAt(r, i, 0).confidence)}
    `) + .join(""); + const c = (id, i) => f2(rowAt(byId[id], i, 0).confidence); + const p = (id, i) => f2(rowAt(byId[id], i, 0).ranked[0][1]); + const gin = rowAt(byId.flash, 0, 0); + $("#delta-notes").innerHTML = ` +
    + Lines 6 and 7 · Flash 9B asked +

    "${esc(lines[5].say)}" and "${esc(lines[6].say)}"

    +

    Flash 9B ranked the right move first on both, with a confidence of ${c("flash", 5)} and ${c("flash", 6)}, under the act threshold of ${f2(D.thresholds.act)}. The game asked "Did you mean:" with the right move as option 1. TypeSafe Jev (${c("typesafe", 5)}, ${c("typesafe", 6)}) and the 27B (${c("27b", 5)}, ${c("27b", 6)}) acted. Line 7 was ${f2(D.thresholds.act - rowAt(byId.flash, 6, 0).confidence)} short: at an act threshold of ${f2(D.slider[rowAt(byId.flash, 6, 0).actMax])} it acts.

    +
    ${[5, 6].map((i) => `
    Line ${i + 1}
    ${mini(i)}`).join("")}
    +
    +
    + Line 1 · the request the author never wrote +

    "${esc(lines[0].say)}"

    +

    All three chose "none of these", so the game printed an authored refusal. TypeSafe Jev put ${p("typesafe", 0)} on it and the 27B ${p("27b", 0)}. Flash 9B put ${p("flash", 0)}, with ${f2(gin.ranked[1][1])} on "${esc(textOf(0, gin.ranked[1][0]))}" and ${f2(gin.ranked[2][1])} on "${esc(textOf(0, gin.ranked[2][0]))}". The parser treats a choice of "none" as unknown at any confidence, so the thresholds played no part. One line is too few to say how often Flash 9B would pick a move here.

    +
    ${mini(0)}
    +
    `; + } + + /* ---------- §7c pairs ---------- */ + function pairs() { + $("#pairs").innerHTML = D.pairs + .map((p) => { + const a = byId[p.a]; + const b = byId[p.b]; + const track = (same, say) => + a.rows + .map((r, i) => ``) + .join(""); + return `
    +
    ${swatch(a)}${esc(a.short)} vs ${swatch(b)}${esc(b.short)}
    +
    Same top choice${track( + (r, s) => r.top === s.top, + (r) => esc(textOf(r.line, r.top)), + )}${p.sameTop}/${p.turns}
    +
    Same outcome${track( + (r, s) => r.kind === s.kind, + (r) => r.kind, + )}${p.sameKind}/${p.turns}
    +
    `; + }) + .join(""); + const allTop = D.pairs.every((p) => p.sameTop === p.turns); + $("#pairs-cap").innerHTML = + `Fig. 7Out of ${N} turns. "Same top choice" is the option each model ranked first; "same outcome" is whether the game acted, asked or said unknown at its thresholds. ` + + (allTop ? "Every pair ranked the same option first on every turn, so the only differences come from confidence." : ""); + } + + /* ---------- §8 move the thresholds ---------- */ + let actIdx = ACT; + let clarifyIdx = CLARIFY; + function barPanels() { + const OC = { ask: "var(--warning)", dead: "var(--critical)", wrong: "var(--critical)" }; + const CELL = { ok: "", ask: "high", dead: "low", wrong: "low" }; + $("#bar-panels").innerHTML = runs + .map((run) => { + const lanes = Array.from({ length: R }, (_, round) => { + const cells = lines + .map((_, i) => { + const row = rowAt(run, i, round); + const kind = kindAt(row, actIdx, clarifyIdx); + const turn = turnAt(row, actIdx, clarifyIdx); + return `Line ${i + 1}, round ${round + 1} · ${esc(run.name)}
    "${esc(lines[i].say)}"confidence ${f2(row.confidence)} → ${kind}: ${turnWord(row, actIdx, clarifyIdx)}`, + )}>${turn === "ok" ? "" : GLYPH[turn]}    `; + }) + .join(""); + return `
    round ${round + 1}${cells}
    `; + }).join(""); + const c = countsAt(run, actIdx, clarifyIdx); + return `

    ${esc(run.name)}

    ${lanes} +
    +
    ${c.ok + c.ask}landed
    +
    ${c.ask}asked ?
    +
    ${c.dead}dead ends ▼
    +
    ${c.wrong}wrong ✗
    +
    `; + }) + .join(""); + $("#act-out").textContent = f2(D.slider[actIdx]); + $("#clarify-out").textContent = f2(D.slider[clarifyIdx]); + $("#act-range").value = String(actIdx); + $("#clarify-range").value = String(clarifyIdx); + // Each chart moves one threshold and carries the other along, as the sliders do. + drawCountChart($("#ask-chart"), "ask", actIdx, (i) => [i, Math.min(clarifyIdx, i)]); + drawCountChart($("#dead-chart"), "dead", clarifyIdx, (i) => [Math.max(actIdx, i), i]); + } + function drawCountChart(el, key, cursor, settingAt) { + const W = widthOf(el); + const H = 210; + const m = { l: 34, r: 12, t: 12, b: 34 }; + const series = runs.map((run) => D.slider.map((_, i) => countsAt(run, ...settingAt(i))[key])); + const yMax = Math.ceil(Math.max(4, ...series.flat()) / 4) * 4; + const x = lin(0, 1, m.l, W - m.r); + const y = lin(0, yMax, H - m.b, m.t); + const word = key === "ask" ? "answered with a question" : "dead ends"; + let g = ""; + for (let t = 0; t <= yMax; t += yMax / 4) { + g += `${t}`; + } + for (const b of [0, 0.2, 0.4, 0.6, 0.8, 1]) { + g += `${b.toFixed(1)}`; + } + g += `${key === "ask" ? "act" : "ask"} threshold →`; + const def = D.slider[key === "ask" ? ACT : CLARIFY]; + g += ``; + // Step lines, offset by a pixel or two so overlapping series stay visible. + const off = [-1.5, 0, 1.5]; + const paths = series + .map((s, k) => { + let d = `M${x(D.slider[0])},${y(s[0]) + off[k]}`; + for (let i = 1; i < s.length; i++) d += `H${x(D.slider[i])}V${y(s[i]) + off[k]}`; + return ``; + }) + .join(""); + const cx = x(D.slider[cursor]); + const markers = series.map((s, k) => ``).join(""); + const hits = D.slider + .map((b, i) => { + const x0 = i === 0 ? m.l : (x(D.slider[i - 1]) + x(b)) / 2; + const x1 = i === LAST ? W - m.r : (x(b) + x(D.slider[i + 1])) / 2; + return `${key === "ask" ? "Act" : "Ask"} threshold ${f2(b)} · ${word}
    ${runs.map((r, k) => `${esc(r.short)}: ${series[k][i]} of ${N}`).join("
    ")}`, + )}/>`; + }) + .join(""); + el.innerHTML = `${g}${paths}${markers}${hits}`; + } + function barNotes() { + const moves = (run) => run.rows.filter((r) => r.top !== D.none); + const notes = runs.map((run) => { + const actsUpTo = Math.min(...moves(run).map((r) => r.actMax)); + const asksUpTo = Math.min(...moves(run).map((r) => r.clarifyMax)); + return `${esc(run.name)}: acts at once on all ${moves(run).length} turns that mean a move while the act threshold is ${f2(D.slider[actsUpTo])} or lower, and starts to ask above it. No dead end until the ask threshold passes ${f2(D.slider[asksUpTo])}.`; + }); + const canGoWrong = runs.some((run) => run.rows.some((r) => (r.actMax >= 0 && !r.lands.act) || (r.clarifyMax >= 0 && !r.lands.clarify))); + if (!canGoWrong) { + notes.push( + `No setting produces a wrong move in this run, because every model ranked the right option first. Here the thresholds only trade acting against asking, and asking against a dead end. They protect against a confident wrong choice, and these lines never produced one.`, + ); + } + notes.push(`The gin line is unknown at every setting. All three chose "none of these", and the parser doesn't weigh that choice against a threshold.`); + $("#bar-notes").innerHTML = notes.map((t) => `
  • ${t}
    • `).join(""); + } + function barControls() { + const actR = $("#act-range"); + const clarR = $("#clarify-range"); + const play = $("#bar-play"); + let timer = null; + const stop = () => { + clearInterval(timer); + timer = null; + play.textContent = "▶ Sweep the act threshold"; + play.setAttribute("aria-pressed", "false"); + }; + const setAct = (i) => { + actIdx = i; + clarifyIdx = Math.min(clarifyIdx, i); + }; + const setClarify = (i) => { + clarifyIdx = i; + actIdx = Math.max(actIdx, i); + }; + actR.addEventListener("input", () => { + stop(); + setAct(+actR.value); + barPanels(); + }); + clarR.addEventListener("input", () => { + stop(); + setClarify(+clarR.value); + barPanels(); + }); + $("#bar-reset").addEventListener("click", () => { + stop(); + actIdx = ACT; + clarifyIdx = CLARIFY; + barPanels(); + }); + play.addEventListener("click", () => { + if (timer) return stop(); + if (actIdx >= LAST) actIdx = clarifyIdx; + play.textContent = "❚❚ Pause"; + play.setAttribute("aria-pressed", "true"); + timer = setInterval( + () => { + actIdx += 1; + barPanels(); + if (actIdx >= LAST) stop(); + }, + reduceMotion ? 250 : 110, + ); + }); + for (const [id, set] of [ + ["#ask-chart", setAct], + ["#dead-chart", setClarify], + ]) { + $(id).addEventListener("click", (e) => { + const hit = e.target.closest(".hit"); + if (!hit) return; + stop(); + set(+hit.dataset.idx); + barPanels(); + }); + } + } + + /* ---------- §9 latency ---------- */ + function latChart() { + laneChart($("#lat-chart"), runs, { + time: (row) => row.ms, + tip: (run, row) => + `Line ${row.line + 1}, round ${row.round + 1} · ${esc(run.name)}
    ${ms(row.ms)}"${esc(lines[row.line].say)}" · ${row.offered} options`, + per: "turn", + checkMs: D.speedCheckMs, + }); + $("#lat-cap").innerHTML = + `Fig. 9Each dot is one turn: ${N} per model. Ticks mark p50 and p95. The three models were asked each line one after another, so they shared whatever else the laptop was doing at that moment. ` + + `Times are taken in the game loop, which ticks about every 7 ms in a headless run, so dots line up in steps of that size.`; + } + function routeChart() { + const el = $("#route-chart"); + const W = widthOf(el); + const narrow = W < 560; + const H = 320; + const m = { l: 44, r: narrow ? 12 : 120, t: 16, b: 54 }; + const maxMs = Math.ceil(Math.max(...runs.map((r) => r.latency.max)) / 100) * 100; + const x = lin(1, L, m.l + 10, W - m.r - 10); + const y = lin(0, maxMs, H - m.b, m.t); + let g = ""; + for (let t = 0; t <= maxMs; t += 100) { + g += `${t}`; + } + lines.forEach((line, i) => { + g += `${i + 1}`; + g += `${line.options.length + 1}`; + }); + g += `lineoptions`; + g += `ms per turn`; + let body = ""; + const ends = []; + runs.forEach((run) => { + const means = lines.map((_, i) => mean(rowsOf(run, i).map((r) => r.ms))); + body += ``; + run.rows.forEach((row) => { + body += `Line ${row.line + 1}, round ${row.round + 1} · ${esc(run.name)}
    ${ms(row.ms)}${row.offered} options · in the ${esc(lines[row.line].room)}`, + )}/>`; + }); + ends.push({ run, y: y(means[L - 1]) + 4 }); + }); + if (!narrow) { + ends.sort((a, b) => a.y - b.y); + for (let i = 1; i < ends.length; i++) ends[i].y = Math.max(ends[i].y, ends[i - 1].y + 16); + ends.forEach((e) => (body += `${esc(e.run.short)}`)); + } + el.innerHTML = `${g}${body}${ + narrow ? `
    ${runs.map((r) => `${swatch(r)}${esc(r.short)}`).join("")}
    ` : "" + }`; + const span = (run) => { + const first = mean(rowsOf(run, 0).map((r) => r.ms)); + const last = mean(rowsOf(run, L - 1).map((r) => r.ms)); + return `from about ${Math.round(first)} ms on line 1 to about ${Math.round(last)} ms on line ${L}`; + }; + $("#route-cap").innerHTML = + `Fig. 10The same turns in route order, with the number of options each question offered. The local models take longer as the request grows, with more options and more items in the state: ` + + `the 27B ${span(byId["27b"])}, Flash 9B ${span(byId.flash)}. The hosted model shows no such trend at this size.`; + } + function latTable() { + const check = (p95) => { + const pass = p95 <= D.speedCheckMs; + return `${pass ? "pass" : `misses by ${p95 - D.speedCheckMs} ms`}`; + }; + const day = (when) => when.slice(0, 10); + const now = runs.map((r) => { + const T = r.latency; + return `${swatch(r)}${esc(r.name)}${day(D.when)}${[T.min, T.p50, T.mean, T.p95, T.max].map((v) => `${ms(v)}`).join("")}${check(T.p95)}`; + }); + const before = runs + .filter((r) => D.earlier[r.id]) + .map((r) => { + const s = D.earlier[r.id]; + return `${swatch(r)}${esc(r.name)}${day(D.earlier.when)}—${ms(s.p50_ms)}—${ms(s.p95_ms)}${ms(s.max_ms)}${check(s.p95_ms)}`; + }); + $("#lat-table tbody").innerHTML = [...now, ...before].join(""); + const e = D.earlier; + $("#lat-table-cap").innerHTML = + `Table 2This run, and the same ${L} lines and ${e.rounds} rounds on ${day(e.when)} with the same pinned TypeSafe model and the same 27B build. ` + + `The hosted p95 went from ${ms(e.typesafe.p95_ms)} to ${ms(ref.latency.p95)}: that run had two slow round trips (the slowest ${ms(e.typesafe.max_ms)}), this one had none. ` + + `The 27B's median went from ${ms(e["27b"].p50_ms)} to ${ms(byId["27b"].latency.p50)}. The model versions were the same on both days and the conditions were not, which is why these read as measurements of a setup.`; + } + + /* ---------- misc ---------- */ + function sources() { + $("#sources").innerHTML = `${esc(D.generatedFrom.split("/").pop())}`; + } + function routeKey() { + $("#route-key").innerHTML = + `what the line meansanother move` + + `none of thesethe other options`; + } + function toggles() { + $("#sure-toggle").addEventListener("click", (e) => { + const b = e.target.closest(".btn"); + if (!b) return; + sureKey = b.dataset.k; + document.querySelectorAll("#sure-toggle .btn").forEach((x) => x.setAttribute("aria-pressed", String(x === b))); + sureChart(); + }); + $("#grid-filter").addEventListener("click", (e) => { + const b = e.target.closest(".btn"); + if (!b) return; + gridFilter = b.dataset.f; + document.querySelectorAll("#grid-filter .btn").forEach((x) => x.setAttribute("aria-pressed", String(x === b))); + lineGrid(); + }); + } + + /* Charts sized to their container; redrawn when the width changes. */ + function drawCharts() { + sureChart(); + barPanels(); + latChart(); + routeChart(); + } + + hero(); + checks(); + modelCards(); + question(); + explorer(); + scoreboard(); + routeKey(); + lineGrid(); + deltaNotes(); + pairs(); + sources(); + latTable(); + barNotes(); + toggles(); + barControls(); + drawCharts(); + tocAndProgress(); + redrawOnResize(drawCharts); +})(); diff --git a/docs/godot-jev/data.js b/docs/godot-jev/data.js new file mode 100644 index 0000000..483e819 --- /dev/null +++ b/docs/godot-jev/data.js @@ -0,0 +1,2 @@ +/* Generated by docs/tools/build-godot-jev-data.gd from godot-jev's eval run. Don't edit. */ +window.GJ_DATA = {"defaultAct":75,"defaultClarify":40,"earlier":{"27b":{"act":45,"clarify":0,"error":0,"landed":48,"max_ms":716,"p50_ms":600,"p95_ms":683,"turns":48,"unknown":3},"rounds":3,"typesafe":{"act":45,"clarify":0,"error":0,"landed":48,"max_ms":1144,"p50_ms":213,"p95_ms":502,"turns":48,"unknown":3},"when":"2026-09-30T17:13:13"},"generatedFrom":"godot-jev/eval/results-2026-10-08T134205.json","lines":[{"means":"","meansText":"","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__north","text":"go north"},{"id":"take__torch","text":"take the torch"},{"id":"examine__torch","text":"examine the torch"}],"room":"Crypt stairs","say":"ayo where the gin and juice at, nephew?","state":{"carrying":[],"dark":false,"location":"Crypt stairs","visible":["torch"]}},{"means":"take__torch","meansText":"take the torch","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__north","text":"go north"},{"id":"take__torch","text":"take the torch"},{"id":"examine__torch","text":"examine the torch"}],"room":"Crypt stairs","say":"fo shizzle, scoop up that torch-izzle off the flo","state":{"carrying":[],"dark":false,"location":"Crypt stairs","visible":["torch"]}},{"means":"light__torch","meansText":"light the torch in the brazier","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"drop__torch","text":"drop the torch"},{"id":"light__torch","text":"light the torch in the brazier"}],"room":"Crypt stairs","say":"light that thang up in the brazier, ya dig","state":{"carrying":["torch"],"dark":false,"location":"Crypt stairs","visible":[]}},{"means":"go__north","meansText":"go north","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"drop__torch","text":"drop the torch"}],"room":"Crypt stairs","say":"roll on up north, cuz","state":{"carrying":["torch"],"dark":false,"location":"Crypt stairs","visible":[]}},{"means":"go__east","meansText":"go east","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"go__east","text":"go east"},{"id":"go__west","text":"go west"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"drop__torch","text":"drop the torch"}],"room":"Hall of pillars","say":"slide east to where the steel be stashed","state":{"carrying":["torch"],"dark":false,"location":"Hall of pillars","visible":[]}},{"means":"take__sword","meansText":"take the notched shortsword","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__west","text":"go west"},{"id":"take__sword","text":"take the notched shortsword"},{"id":"take__vellum","text":"take the scrap of vellum"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"examine__vellum","text":"examine the scrap of vellum"},{"id":"examine__torch","text":"examine the torch"},{"id":"drop__torch","text":"drop the torch"}],"room":"Armory","say":"cop that shortsword, fo shizzle","state":{"carrying":["torch"],"dark":false,"location":"Armory","visible":["notched shortsword","scrap of vellum"]}},{"means":"examine__vellum","meansText":"examine the scrap of vellum","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__west","text":"go west"},{"id":"take__vellum","text":"take the scrap of vellum"},{"id":"examine__vellum","text":"examine the scrap of vellum"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"}],"room":"Armory","say":"what that lil paper say, homie?","state":{"carrying":["torch","notched shortsword"],"dark":false,"location":"Armory","visible":["scrap of vellum"]}},{"means":"go__west","meansText":"go west","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__west","text":"go west"},{"id":"take__vellum","text":"take the scrap of vellum"},{"id":"examine__vellum","text":"examine the scrap of vellum"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"}],"room":"Armory","say":"bounce back west, ya heard","state":{"carrying":["torch","notched shortsword"],"dark":false,"location":"Armory","visible":["scrap of vellum"]}},{"means":"attack__skeleton","meansText":"attack the skeleton with the shortsword","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"go__east","text":"go east"},{"id":"go__west","text":"go west"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"},{"id":"attack__skeleton","text":"attack the skeleton with the shortsword"}],"room":"Hall of pillars","say":"lay the smack down on Mr. Bones wit the blade","state":{"carrying":["torch","notched shortsword"],"dark":false,"location":"Hall of pillars","visible":[]}},{"means":"go__west","meansText":"go west","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"go__east","text":"go east"},{"id":"go__west","text":"go west"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"}],"room":"Hall of pillars","say":"creep west into the dark, torch-izzle blazin","state":{"carrying":["torch","notched shortsword"],"dark":false,"location":"Hall of pillars","visible":[]}},{"means":"take__key","meansText":"take the silver key","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__east","text":"go east"},{"id":"take__key","text":"take the silver key"},{"id":"examine__key","text":"examine the silver key"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"}],"room":"Ossuary","say":"snatch that silver key off the altar, baby","state":{"carrying":["torch","notched shortsword"],"dark":false,"location":"Ossuary","visible":["silver key"]}},{"means":"go__east","meansText":"go east","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__east","text":"go east"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"examine__key","text":"examine the silver key"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"},{"id":"drop__key","text":"drop the silver key"}],"room":"Ossuary","say":"back east, back east","state":{"carrying":["torch","notched shortsword","silver key"],"dark":false,"location":"Ossuary","visible":[]}},{"means":"go__north","meansText":"go north","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"go__east","text":"go east"},{"id":"go__west","text":"go west"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"examine__key","text":"examine the silver key"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"},{"id":"drop__key","text":"drop the silver key"}],"room":"Hall of pillars","say":"head north, the Doggfather got business","state":{"carrying":["torch","notched shortsword","silver key"],"dark":false,"location":"Hall of pillars","visible":[]}},{"means":"unlock__door","meansText":"unlock the black door with the silver key","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"examine__key","text":"examine the silver key"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"},{"id":"drop__key","text":"drop the silver key"},{"id":"unlock__door","text":"unlock the black door with the silver key"}],"room":"Antechamber","say":"unlock that black door wit the key-izzle","state":{"carrying":["torch","notched shortsword","silver key"],"dark":false,"location":"Antechamber","visible":[]}},{"means":"go__north","meansText":"go north","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"go__north","text":"go north"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"examine__key","text":"examine the silver key"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"},{"id":"drop__key","text":"drop the silver key"}],"room":"Antechamber","say":"roll north thru that door, it's time to get paid","state":{"carrying":["torch","notched shortsword","silver key"],"dark":false,"location":"Antechamber","visible":[]}},{"means":"take__crown","meansText":"take the jeweled crown","options":[{"id":"look__around","text":"look around the room"},{"id":"inventory__self","text":"check what I'm carrying"},{"id":"go__south","text":"go south"},{"id":"take__crown","text":"take the jeweled crown"},{"id":"examine__crown","text":"examine the jeweled crown"},{"id":"examine__torch","text":"examine the torch"},{"id":"examine__sword","text":"examine the notched shortsword"},{"id":"examine__key","text":"examine the silver key"},{"id":"drop__torch","text":"drop the torch"},{"id":"drop__sword","text":"drop the notched shortsword"},{"id":"drop__key","text":"drop the silver key"}],"room":"Vault","say":"cop that crown, it's the Doggfather's now","state":{"carrying":["torch","notched shortsword","silver key"],"dark":false,"location":"Vault","visible":["jeweled crown"]}}],"none":"none_of_these","noneText":"The player isn't asking for any of these: a question, small talk, or something not possible here.","pairs":[{"a":"typesafe","b":"flash","sameKind":42,"sameTop":48,"turns":48},{"a":"typesafe","b":"27b","sameKind":48,"sameTop":48,"turns":48},{"a":"flash","b":"27b","sameKind":42,"sameTop":48,"turns":48}],"question":"Which action is the player asking to take?","rounds":3,"runs":[{"backend":"typesafe","id":"typesafe","latency":{"max":213,"mean":167.791666666667,"min":138,"p50":165,"p95":200},"lowestOnAMove":0.81,"meanConfidence":0.954375,"name":"TypeSafe Jev","note":"Pinned to jev-1.13.0. A turn's time includes the network round trip from the laptop.","rows":[{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.96,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":179,"offered":6,"ranked":[["none_of_these",0.97],["look__around",0.02],["take__torch",0.01]],"round":0,"top":"none_of_these"},{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.95,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":166,"offered":6,"ranked":[["none_of_these",0.96],["look__around",0.03],["take__torch",0.01]],"round":1,"top":"none_of_these"},{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.96,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":165,"offered":6,"ranked":[["none_of_these",0.97],["look__around",0.02],["take__torch",0.01]],"round":2,"top":"none_of_these"},{"actMax":100,"asks":["take__torch","inventory__self"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":151,"offered":6,"ranked":[["take__torch",1.0],["inventory__self",0.0],["examine__torch",0.0]],"round":0,"top":"take__torch"},{"actMax":100,"asks":["take__torch","examine__torch"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":213,"offered":6,"ranked":[["take__torch",1.0],["examine__torch",0.0],["go__north",0.0]],"round":1,"top":"take__torch"},{"actMax":100,"asks":["take__torch","examine__torch"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":158,"offered":6,"ranked":[["take__torch",1.0],["examine__torch",0.0],["go__north",0.0]],"round":2,"top":"take__torch"},{"actMax":99,"asks":["light__torch","look__around"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":165,"offered":7,"ranked":[["light__torch",0.99],["none_of_these",0.01],["look__around",0.0]],"round":0,"top":"light__torch"},{"actMax":99,"asks":["light__torch","examine__torch"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":172,"offered":7,"ranked":[["light__torch",0.99],["none_of_these",0.01],["examine__torch",0.0]],"round":1,"top":"light__torch"},{"actMax":99,"asks":["light__torch","drop__torch"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":178,"offered":7,"ranked":[["light__torch",0.99],["none_of_these",0.01],["drop__torch",0.0]],"round":2,"top":"light__torch"},{"actMax":89,"asks":["go__north","look__around"],"clarifyMax":89,"confidence":0.89,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":165,"offered":6,"ranked":[["go__north",0.91],["none_of_these",0.08],["look__around",0.01]],"round":0,"top":"go__north"},{"actMax":91,"asks":["go__north","look__around"],"clarifyMax":91,"confidence":0.91,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":160,"offered":6,"ranked":[["go__north",0.93],["none_of_these",0.06],["look__around",0.01]],"round":1,"top":"go__north"},{"actMax":91,"asks":["go__north","look__around"],"clarifyMax":91,"confidence":0.91,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":166,"offered":6,"ranked":[["go__north",0.92],["none_of_these",0.07],["look__around",0.01]],"round":2,"top":"go__north"},{"actMax":87,"asks":["go__east","go__west"],"clarifyMax":87,"confidence":0.87,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":158,"offered":9,"ranked":[["go__east",0.88],["none_of_these",0.12],["go__west",0.0]],"round":0,"top":"go__east"},{"actMax":85,"asks":["go__east","drop__torch"],"clarifyMax":85,"confidence":0.85,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":144,"offered":9,"ranked":[["go__east",0.87],["none_of_these",0.13],["drop__torch",0.0]],"round":1,"top":"go__east"},{"actMax":89,"asks":["go__east","drop__torch"],"clarifyMax":89,"confidence":0.89,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":151,"offered":9,"ranked":[["go__east",0.91],["none_of_these",0.09],["drop__torch",0.0]],"round":2,"top":"go__east"},{"actMax":87,"asks":["take__sword","examine__sword"],"clarifyMax":87,"confidence":0.87,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":165,"offered":10,"ranked":[["take__sword",0.88],["none_of_these",0.11],["examine__sword",0.01]],"round":0,"top":"take__sword"},{"actMax":89,"asks":["take__sword","examine__sword"],"clarifyMax":89,"confidence":0.89,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":166,"offered":10,"ranked":[["take__sword",0.91],["none_of_these",0.08],["examine__sword",0.01]],"round":1,"top":"take__sword"},{"actMax":88,"asks":["take__sword","examine__sword"],"clarifyMax":88,"confidence":0.88,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":144,"offered":10,"ranked":[["take__sword",0.89],["none_of_these",0.1],["examine__sword",0.01]],"round":2,"top":"take__sword"},{"actMax":94,"asks":["examine__vellum","take__vellum"],"clarifyMax":94,"confidence":0.94,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":200,"offered":10,"ranked":[["examine__vellum",0.95],["none_of_these",0.04],["take__vellum",0.01]],"round":0,"top":"examine__vellum"},{"actMax":96,"asks":["examine__vellum","take__vellum"],"clarifyMax":96,"confidence":0.96,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":193,"offered":10,"ranked":[["examine__vellum",0.96],["none_of_these",0.03],["take__vellum",0.01]],"round":1,"top":"examine__vellum"},{"actMax":95,"asks":["examine__vellum","take__vellum"],"clarifyMax":95,"confidence":0.95,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":158,"offered":10,"ranked":[["examine__vellum",0.96],["none_of_these",0.03],["take__vellum",0.01]],"round":2,"top":"examine__vellum"},{"actMax":81,"asks":["go__west","drop__sword"],"clarifyMax":81,"confidence":0.81,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":193,"offered":10,"ranked":[["go__west",0.84],["none_of_these",0.16],["drop__sword",0.0]],"round":0,"top":"go__west"},{"actMax":87,"asks":["go__west","take__vellum"],"clarifyMax":87,"confidence":0.87,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":193,"offered":10,"ranked":[["go__west",0.88],["none_of_these",0.12],["take__vellum",0.0]],"round":1,"top":"go__west"},{"actMax":83,"asks":["go__west","examine__sword"],"clarifyMax":83,"confidence":0.83,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":172,"offered":10,"ranked":[["go__west",0.85],["none_of_these",0.15],["examine__sword",0.0]],"round":2,"top":"go__west"},{"actMax":100,"asks":["attack__skeleton","drop__sword"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":151,"offered":12,"ranked":[["attack__skeleton",1.0],["drop__sword",0.0],["drop__torch",0.0]],"round":0,"top":"attack__skeleton"},{"actMax":100,"asks":["attack__skeleton","go__north"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":152,"offered":12,"ranked":[["attack__skeleton",1.0],["none_of_these",0.0],["go__north",0.0]],"round":1,"top":"attack__skeleton"},{"actMax":100,"asks":["attack__skeleton","drop__sword"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":179,"offered":12,"ranked":[["attack__skeleton",1.0],["drop__sword",0.0],["drop__torch",0.0]],"round":2,"top":"attack__skeleton"},{"actMax":99,"asks":["go__west","drop__sword"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":158,"offered":11,"ranked":[["go__west",0.99],["none_of_these",0.01],["drop__sword",0.0]],"round":0,"top":"go__west"},{"actMax":99,"asks":["go__west","drop__sword"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":180,"offered":11,"ranked":[["go__west",0.99],["none_of_these",0.01],["drop__sword",0.0]],"round":1,"top":"go__west"},{"actMax":99,"asks":["go__west","go__east"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":186,"offered":11,"ranked":[["go__west",0.99],["none_of_these",0.01],["go__east",0.0]],"round":2,"top":"go__west"},{"actMax":100,"asks":["take__key","drop__sword"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":186,"offered":10,"ranked":[["take__key",1.0],["drop__sword",0.0],["drop__torch",0.0]],"round":0,"top":"take__key"},{"actMax":100,"asks":["take__key","drop__sword"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":138,"offered":10,"ranked":[["take__key",1.0],["drop__sword",0.0],["drop__torch",0.0]],"round":1,"top":"take__key"},{"actMax":100,"asks":["take__key","drop__sword"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":158,"offered":10,"ranked":[["take__key",1.0],["drop__sword",0.0],["drop__torch",0.0]],"round":2,"top":"take__key"},{"actMax":97,"asks":["go__east","drop__key"],"clarifyMax":97,"confidence":0.97,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":151,"offered":10,"ranked":[["go__east",0.98],["none_of_these",0.02],["drop__key",0.0]],"round":0,"top":"go__east"},{"actMax":97,"asks":["go__east","drop__key"],"clarifyMax":97,"confidence":0.97,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":152,"offered":10,"ranked":[["go__east",0.98],["none_of_these",0.02],["drop__key",0.0]],"round":1,"top":"go__east"},{"actMax":98,"asks":["go__east","examine__key"],"clarifyMax":98,"confidence":0.98,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":213,"offered":10,"ranked":[["go__east",0.98],["none_of_these",0.02],["examine__key",0.0]],"round":2,"top":"go__east"},{"actMax":99,"asks":["go__north","examine__key"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":172,"offered":13,"ranked":[["go__north",0.99],["none_of_these",0.01],["examine__key",0.0]],"round":0,"top":"go__north"},{"actMax":98,"asks":["go__north","drop__key"],"clarifyMax":98,"confidence":0.98,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":158,"offered":13,"ranked":[["go__north",0.99],["none_of_these",0.01],["drop__key",0.0]],"round":1,"top":"go__north"},{"actMax":99,"asks":["go__north","drop__key"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":179,"offered":13,"ranked":[["go__north",0.99],["none_of_these",0.01],["drop__key",0.0]],"round":2,"top":"go__north"},{"actMax":100,"asks":["unlock__door","drop__sword"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":179,"offered":12,"ranked":[["unlock__door",1.0],["drop__sword",0.0],["none_of_these",0.0]],"round":0,"top":"unlock__door"},{"actMax":100,"asks":["unlock__door","drop__key"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":186,"offered":12,"ranked":[["unlock__door",1.0],["drop__key",0.0],["drop__sword",0.0]],"round":1,"top":"unlock__door"},{"actMax":100,"asks":["unlock__door","drop__key"],"clarifyMax":100,"confidence":1.0,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":145,"offered":12,"ranked":[["unlock__door",1.0],["drop__key",0.0],["drop__sword",0.0]],"round":2,"top":"unlock__door"},{"actMax":98,"asks":["go__north","drop__key"],"clarifyMax":98,"confidence":0.98,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":151,"offered":11,"ranked":[["go__north",0.98],["none_of_these",0.02],["drop__key",0.0]],"round":0,"top":"go__north"},{"actMax":98,"asks":["go__north","drop__key"],"clarifyMax":98,"confidence":0.98,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":151,"offered":11,"ranked":[["go__north",0.98],["none_of_these",0.02],["drop__key",0.0]],"round":1,"top":"go__north"},{"actMax":99,"asks":["go__north","drop__sword"],"clarifyMax":99,"confidence":0.99,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":158,"offered":11,"ranked":[["go__north",0.99],["none_of_these",0.01],["drop__sword",0.0]],"round":2,"top":"go__north"},{"actMax":94,"asks":["take__crown","drop__key"],"clarifyMax":94,"confidence":0.94,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":180,"offered":12,"ranked":[["take__crown",0.95],["none_of_these",0.05],["drop__key",0.0]],"round":0,"top":"take__crown"},{"actMax":96,"asks":["take__crown","drop__key"],"clarifyMax":96,"confidence":0.96,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":165,"offered":12,"ranked":[["take__crown",0.96],["none_of_these",0.04],["drop__key",0.0]],"round":1,"top":"take__crown"},{"actMax":95,"asks":["take__crown","drop__sword"],"clarifyMax":95,"confidence":0.95,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":151,"offered":12,"ranked":[["take__crown",0.95],["none_of_these",0.05],["drop__sword",0.0]],"round":2,"top":"take__crown"}],"sameEveryRound":false,"servedModel":"jev-1.13.0","short":"TypeSafe","size":"Hosted API","summary":{"act":45,"clarify":0,"error":0,"landed":48,"max_ms":213,"p50_ms":165,"p95_ms":200,"turns":48,"unknown":3},"sure":34,"temperature":"","where":"Hosted"},{"backend":"flash9b","id":"flash","latency":{"max":228,"mean":175.5,"min":143,"p50":185,"p95":200},"lowestOnAMove":0.4833,"meanConfidence":0.833537499999999,"name":"Open Jev Flash 9B","note":"MLX 4-bit on an Apple M5 Pro laptop, with the 27B loaded beside it.","rows":[{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.5173,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":158,"offered":6,"ranked":[["none_of_these",0.5977],["inventory__self",0.1036],["examine__torch",0.0922]],"round":0,"top":"none_of_these"},{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.5173,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":144,"offered":6,"ranked":[["none_of_these",0.5977],["inventory__self",0.1036],["examine__torch",0.0922]],"round":1,"top":"none_of_these"},{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.5173,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":145,"offered":6,"ranked":[["none_of_these",0.5977],["inventory__self",0.1036],["examine__torch",0.0922]],"round":2,"top":"none_of_these"},{"actMax":96,"asks":["take__torch","examine__torch"],"clarifyMax":96,"confidence":0.9619,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":151,"offered":6,"ranked":[["take__torch",0.9682],["none_of_these",0.0205],["examine__torch",0.0072]],"round":0,"top":"take__torch"},{"actMax":96,"asks":["take__torch","examine__torch"],"clarifyMax":96,"confidence":0.9619,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":151,"offered":6,"ranked":[["take__torch",0.9682],["none_of_these",0.0205],["examine__torch",0.0072]],"round":1,"top":"take__torch"},{"actMax":96,"asks":["take__torch","examine__torch"],"clarifyMax":96,"confidence":0.9619,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":144,"offered":6,"ranked":[["take__torch",0.9682],["none_of_these",0.0205],["examine__torch",0.0072]],"round":2,"top":"take__torch"},{"actMax":94,"asks":["light__torch","drop__torch"],"clarifyMax":94,"confidence":0.9462,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":158,"offered":7,"ranked":[["light__torch",0.9539],["none_of_these",0.016],["drop__torch",0.0113]],"round":0,"top":"light__torch"},{"actMax":94,"asks":["light__torch","drop__torch"],"clarifyMax":94,"confidence":0.9462,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":144,"offered":7,"ranked":[["light__torch",0.9539],["none_of_these",0.016],["drop__torch",0.0113]],"round":1,"top":"light__torch"},{"actMax":94,"asks":["light__torch","drop__torch"],"clarifyMax":94,"confidence":0.9462,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":152,"offered":7,"ranked":[["light__torch",0.9539],["none_of_these",0.016],["drop__torch",0.0113]],"round":2,"top":"light__torch"},{"actMax":89,"asks":["go__north","look__around"],"clarifyMax":89,"confidence":0.897,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":143,"offered":6,"ranked":[["go__north",0.9142],["none_of_these",0.0309],["look__around",0.0244]],"round":0,"top":"go__north"},{"actMax":89,"asks":["go__north","look__around"],"clarifyMax":89,"confidence":0.897,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":144,"offered":6,"ranked":[["go__north",0.9142],["none_of_these",0.0309],["look__around",0.0244]],"round":1,"top":"go__north"},{"actMax":89,"asks":["go__north","look__around"],"clarifyMax":89,"confidence":0.897,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":144,"offered":6,"ranked":[["go__north",0.9142],["none_of_these",0.0309],["look__around",0.0244]],"round":2,"top":"go__north"},{"actMax":79,"asks":["go__east","go__west"],"clarifyMax":79,"confidence":0.7915,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":151,"offered":9,"ranked":[["go__east",0.8147],["none_of_these",0.0623],["go__west",0.0275]],"round":0,"top":"go__east"},{"actMax":79,"asks":["go__east","go__west"],"clarifyMax":79,"confidence":0.7915,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":151,"offered":9,"ranked":[["go__east",0.8147],["none_of_these",0.0623],["go__west",0.0275]],"round":1,"top":"go__east"},{"actMax":79,"asks":["go__east","go__west"],"clarifyMax":79,"confidence":0.7915,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":151,"offered":9,"ranked":[["go__east",0.8147],["none_of_these",0.0623],["go__west",0.0275]],"round":2,"top":"go__east"},{"actMax":48,"asks":["take__sword","examine__sword"],"clarifyMax":48,"confidence":0.4833,"kind":"clarify","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":185,"offered":10,"ranked":[["take__sword",0.535],["examine__sword",0.1317],["take__vellum",0.1172]],"round":0,"top":"take__sword"},{"actMax":48,"asks":["take__sword","examine__sword"],"clarifyMax":48,"confidence":0.4833,"kind":"clarify","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":179,"offered":10,"ranked":[["take__sword",0.535],["examine__sword",0.1317],["take__vellum",0.1172]],"round":1,"top":"take__sword"},{"actMax":48,"asks":["take__sword","examine__sword"],"clarifyMax":48,"confidence":0.4833,"kind":"clarify","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":178,"offered":10,"ranked":[["take__sword",0.535],["examine__sword",0.1317],["take__vellum",0.1172]],"round":2,"top":"take__sword"},{"actMax":74,"asks":["examine__vellum","take__vellum"],"clarifyMax":74,"confidence":0.7447,"kind":"clarify","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":186,"offered":10,"ranked":[["examine__vellum",0.7702],["take__vellum",0.1501],["none_of_these",0.0292]],"round":0,"top":"examine__vellum"},{"actMax":74,"asks":["examine__vellum","take__vellum"],"clarifyMax":74,"confidence":0.7447,"kind":"clarify","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":179,"offered":10,"ranked":[["examine__vellum",0.7702],["take__vellum",0.1501],["none_of_these",0.0292]],"round":1,"top":"examine__vellum"},{"actMax":74,"asks":["examine__vellum","take__vellum"],"clarifyMax":74,"confidence":0.7447,"kind":"clarify","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":180,"offered":10,"ranked":[["examine__vellum",0.7702],["take__vellum",0.1501],["none_of_these",0.0292]],"round":2,"top":"examine__vellum"},{"actMax":82,"asks":["go__west","take__vellum"],"clarifyMax":82,"confidence":0.8215,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":179,"offered":10,"ranked":[["go__west",0.8393],["take__vellum",0.0508],["examine__vellum",0.0358]],"round":0,"top":"go__west"},{"actMax":82,"asks":["go__west","take__vellum"],"clarifyMax":82,"confidence":0.8215,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":179,"offered":10,"ranked":[["go__west",0.8393],["take__vellum",0.0508],["examine__vellum",0.0358]],"round":1,"top":"go__west"},{"actMax":82,"asks":["go__west","take__vellum"],"clarifyMax":82,"confidence":0.8215,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":179,"offered":10,"ranked":[["go__west",0.8393],["take__vellum",0.0508],["examine__vellum",0.0358]],"round":2,"top":"go__west"},{"actMax":84,"asks":["attack__skeleton","drop__sword"],"clarifyMax":84,"confidence":0.8434,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":185,"offered":12,"ranked":[["attack__skeleton",0.8565],["drop__sword",0.0655],["none_of_these",0.0273]],"round":0,"top":"attack__skeleton"},{"actMax":84,"asks":["attack__skeleton","drop__sword"],"clarifyMax":84,"confidence":0.8434,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":185,"offered":12,"ranked":[["attack__skeleton",0.8565],["drop__sword",0.0655],["none_of_these",0.0273]],"round":1,"top":"attack__skeleton"},{"actMax":84,"asks":["attack__skeleton","drop__sword"],"clarifyMax":84,"confidence":0.8434,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":186,"offered":12,"ranked":[["attack__skeleton",0.8565],["drop__sword",0.0655],["none_of_these",0.0273]],"round":2,"top":"attack__skeleton"},{"actMax":86,"asks":["go__west","go__east"],"clarifyMax":86,"confidence":0.8649,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":185,"offered":11,"ranked":[["go__west",0.8772],["none_of_these",0.0473],["go__east",0.0249]],"round":0,"top":"go__west"},{"actMax":86,"asks":["go__west","go__east"],"clarifyMax":86,"confidence":0.8649,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":178,"offered":11,"ranked":[["go__west",0.8772],["none_of_these",0.0473],["go__east",0.0249]],"round":1,"top":"go__west"},{"actMax":86,"asks":["go__west","go__east"],"clarifyMax":86,"confidence":0.8649,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":186,"offered":11,"ranked":[["go__west",0.8772],["none_of_these",0.0473],["go__east",0.0249]],"round":2,"top":"go__west"},{"actMax":97,"asks":["take__key","examine__key"],"clarifyMax":97,"confidence":0.9755,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":186,"offered":10,"ranked":[["take__key",0.9779],["examine__key",0.0103],["none_of_these",0.0061]],"round":0,"top":"take__key"},{"actMax":97,"asks":["take__key","examine__key"],"clarifyMax":97,"confidence":0.9755,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":186,"offered":10,"ranked":[["take__key",0.9779],["examine__key",0.0103],["none_of_these",0.0061]],"round":1,"top":"take__key"},{"actMax":97,"asks":["take__key","examine__key"],"clarifyMax":97,"confidence":0.9755,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":186,"offered":10,"ranked":[["take__key",0.9779],["examine__key",0.0103],["none_of_these",0.0061]],"round":2,"top":"take__key"},{"actMax":91,"asks":["go__east","look__around"],"clarifyMax":91,"confidence":0.9122,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":186,"offered":10,"ranked":[["go__east",0.921],["look__around",0.0184],["none_of_these",0.0173]],"round":0,"top":"go__east"},{"actMax":91,"asks":["go__east","look__around"],"clarifyMax":91,"confidence":0.9122,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":186,"offered":10,"ranked":[["go__east",0.921],["look__around",0.0184],["none_of_these",0.0173]],"round":1,"top":"go__east"},{"actMax":91,"asks":["go__east","look__around"],"clarifyMax":91,"confidence":0.9122,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":186,"offered":10,"ranked":[["go__east",0.921],["look__around",0.0184],["none_of_these",0.0173]],"round":2,"top":"go__east"},{"actMax":87,"asks":["go__north","go__east"],"clarifyMax":87,"confidence":0.8754,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":228,"offered":13,"ranked":[["go__north",0.885],["none_of_these",0.0299],["go__east",0.0177]],"round":0,"top":"go__north"},{"actMax":87,"asks":["go__north","go__east"],"clarifyMax":87,"confidence":0.8754,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":186,"offered":13,"ranked":[["go__north",0.885],["none_of_these",0.0299],["go__east",0.0177]],"round":1,"top":"go__north"},{"actMax":87,"asks":["go__north","go__east"],"clarifyMax":87,"confidence":0.8754,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":193,"offered":13,"ranked":[["go__north",0.885],["none_of_these",0.0299],["go__east",0.0177]],"round":2,"top":"go__north"},{"actMax":96,"asks":["unlock__door","drop__key"],"clarifyMax":96,"confidence":0.9629,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":193,"offered":12,"ranked":[["unlock__door",0.966],["none_of_these",0.0162],["drop__key",0.0057]],"round":0,"top":"unlock__door"},{"actMax":96,"asks":["unlock__door","drop__key"],"clarifyMax":96,"confidence":0.9629,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":185,"offered":12,"ranked":[["unlock__door",0.966],["none_of_these",0.0162],["drop__key",0.0057]],"round":1,"top":"unlock__door"},{"actMax":96,"asks":["unlock__door","drop__key"],"clarifyMax":96,"confidence":0.9629,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":186,"offered":12,"ranked":[["unlock__door",0.966],["none_of_these",0.0162],["drop__key",0.0057]],"round":2,"top":"unlock__door"},{"actMax":92,"asks":["go__north","go__south"],"clarifyMax":92,"confidence":0.9295,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":192,"offered":11,"ranked":[["go__north",0.9359],["none_of_these",0.021],["go__south",0.0083]],"round":0,"top":"go__north"},{"actMax":92,"asks":["go__north","go__south"],"clarifyMax":92,"confidence":0.9295,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":193,"offered":11,"ranked":[["go__north",0.9359],["none_of_these",0.021],["go__south",0.0083]],"round":1,"top":"go__north"},{"actMax":92,"asks":["go__north","go__south"],"clarifyMax":92,"confidence":0.9295,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":192,"offered":11,"ranked":[["go__north",0.9359],["none_of_these",0.021],["go__south",0.0083]],"round":2,"top":"go__north"},{"actMax":80,"asks":["take__crown","examine__crown"],"clarifyMax":80,"confidence":0.8094,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":200,"offered":12,"ranked":[["take__crown",0.8253],["examine__crown",0.0897],["none_of_these",0.0296]],"round":0,"top":"take__crown"},{"actMax":80,"asks":["take__crown","examine__crown"],"clarifyMax":80,"confidence":0.8094,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":200,"offered":12,"ranked":[["take__crown",0.8253],["examine__crown",0.0897],["none_of_these",0.0296]],"round":1,"top":"take__crown"},{"actMax":80,"asks":["take__crown","examine__crown"],"clarifyMax":80,"confidence":0.8094,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":200,"offered":12,"ranked":[["take__crown",0.8253],["examine__crown",0.0897],["none_of_these",0.0296]],"round":2,"top":"take__crown"}],"sameEveryRound":true,"servedModel":"OpenJev-Flash-9B-MLX-4bit","short":"Flash 9B","size":"5 GB","summary":{"act":39,"clarify":6,"error":0,"landed":48,"max_ms":228,"p50_ms":185,"p95_ms":200,"turns":48,"unknown":3},"sure":9,"temperature":"1.07","where":"Local"},{"backend":"openjev","id":"27b","latency":{"max":634,"mean":545.604166666667,"min":434,"p50":579,"p95":620},"lowestOnAMove":0.9328,"meanConfidence":0.9847375,"name":"Open Jev 27B","note":"MLX 4-bit on an Apple M5 Pro laptop, with Flash 9B loaded beside it.","rows":[{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.974,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":462,"offered":6,"ranked":[["none_of_these",0.9783],["take__torch",0.0102],["look__around",0.0057]],"round":0,"top":"none_of_these"},{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.974,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":441,"offered":6,"ranked":[["none_of_these",0.9783],["take__torch",0.0102],["look__around",0.0057]],"round":1,"top":"none_of_these"},{"actMax":-1,"asks":[],"clarifyMax":-1,"confidence":0.974,"kind":"unknown","landed":true,"lands":{"act":false,"clarify":false,"unknown":true},"line":0,"ms":441,"offered":6,"ranked":[["none_of_these",0.9783],["take__torch",0.0102],["look__around",0.0057]],"round":2,"top":"none_of_these"},{"actMax":99,"asks":["take__torch","examine__torch"],"clarifyMax":99,"confidence":0.9999,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":441,"offered":6,"ranked":[["take__torch",0.9999],["examine__torch",0.0001],["go__north",0.0]],"round":0,"top":"take__torch"},{"actMax":99,"asks":["take__torch","examine__torch"],"clarifyMax":99,"confidence":0.9999,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":441,"offered":6,"ranked":[["take__torch",0.9999],["examine__torch",0.0001],["go__north",0.0]],"round":1,"top":"take__torch"},{"actMax":99,"asks":["take__torch","examine__torch"],"clarifyMax":99,"confidence":0.9999,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":1,"ms":434,"offered":6,"ranked":[["take__torch",0.9999],["examine__torch",0.0001],["go__north",0.0]],"round":2,"top":"take__torch"},{"actMax":93,"asks":["light__torch","drop__torch"],"clarifyMax":93,"confidence":0.9328,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":455,"offered":7,"ranked":[["light__torch",0.9424],["none_of_these",0.043],["drop__torch",0.0132]],"round":0,"top":"light__torch"},{"actMax":93,"asks":["light__torch","drop__torch"],"clarifyMax":93,"confidence":0.9328,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":462,"offered":7,"ranked":[["light__torch",0.9424],["none_of_these",0.043],["drop__torch",0.0132]],"round":1,"top":"light__torch"},{"actMax":93,"asks":["light__torch","drop__torch"],"clarifyMax":93,"confidence":0.9328,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":2,"ms":456,"offered":7,"ranked":[["light__torch",0.9424],["none_of_these",0.043],["drop__torch",0.0132]],"round":2,"top":"light__torch"},{"actMax":98,"asks":["go__north","look__around"],"clarifyMax":98,"confidence":0.9898,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":441,"offered":6,"ranked":[["go__north",0.9915],["none_of_these",0.0077],["look__around",0.0005]],"round":0,"top":"go__north"},{"actMax":98,"asks":["go__north","look__around"],"clarifyMax":98,"confidence":0.9898,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":434,"offered":6,"ranked":[["go__north",0.9915],["none_of_these",0.0077],["look__around",0.0005]],"round":1,"top":"go__north"},{"actMax":98,"asks":["go__north","look__around"],"clarifyMax":98,"confidence":0.9898,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":3,"ms":441,"offered":6,"ranked":[["go__north",0.9915],["none_of_these",0.0077],["look__around",0.0005]],"round":2,"top":"go__north"},{"actMax":96,"asks":["go__east","go__west"],"clarifyMax":96,"confidence":0.9645,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":468,"offered":9,"ranked":[["go__east",0.9685],["none_of_these",0.0284],["go__west",0.0015]],"round":0,"top":"go__east"},{"actMax":96,"asks":["go__east","go__west"],"clarifyMax":96,"confidence":0.9645,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":462,"offered":9,"ranked":[["go__east",0.9685],["none_of_these",0.0284],["go__west",0.0015]],"round":1,"top":"go__east"},{"actMax":96,"asks":["go__east","go__west"],"clarifyMax":96,"confidence":0.9645,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":4,"ms":462,"offered":9,"ranked":[["go__east",0.9685],["none_of_these",0.0284],["go__west",0.0015]],"round":2,"top":"go__east"},{"actMax":99,"asks":["take__sword","take__vellum"],"clarifyMax":99,"confidence":0.993,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":634,"offered":10,"ranked":[["take__sword",0.9937],["none_of_these",0.0028],["take__vellum",0.0024]],"round":0,"top":"take__sword"},{"actMax":99,"asks":["take__sword","take__vellum"],"clarifyMax":99,"confidence":0.993,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":572,"offered":10,"ranked":[["take__sword",0.9937],["none_of_these",0.0028],["take__vellum",0.0024]],"round":1,"top":"take__sword"},{"actMax":99,"asks":["take__sword","take__vellum"],"clarifyMax":99,"confidence":0.993,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":5,"ms":572,"offered":10,"ranked":[["take__sword",0.9937],["none_of_these",0.0028],["take__vellum",0.0024]],"round":2,"top":"take__sword"},{"actMax":99,"asks":["examine__vellum","take__vellum"],"clarifyMax":99,"confidence":0.9948,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":572,"offered":10,"ranked":[["examine__vellum",0.9954],["take__vellum",0.0032],["none_of_these",0.001]],"round":0,"top":"examine__vellum"},{"actMax":99,"asks":["examine__vellum","take__vellum"],"clarifyMax":99,"confidence":0.9948,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":572,"offered":10,"ranked":[["examine__vellum",0.9954],["take__vellum",0.0032],["none_of_these",0.001]],"round":1,"top":"examine__vellum"},{"actMax":99,"asks":["examine__vellum","take__vellum"],"clarifyMax":99,"confidence":0.9948,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":6,"ms":571,"offered":10,"ranked":[["examine__vellum",0.9954],["take__vellum",0.0032],["none_of_these",0.001]],"round":2,"top":"examine__vellum"},{"actMax":97,"asks":["go__west","take__vellum"],"clarifyMax":97,"confidence":0.9783,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":586,"offered":10,"ranked":[["go__west",0.9805],["none_of_these",0.016],["take__vellum",0.0013]],"round":0,"top":"go__west"},{"actMax":97,"asks":["go__west","take__vellum"],"clarifyMax":97,"confidence":0.9783,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":572,"offered":10,"ranked":[["go__west",0.9805],["none_of_these",0.016],["take__vellum",0.0013]],"round":1,"top":"go__west"},{"actMax":97,"asks":["go__west","take__vellum"],"clarifyMax":97,"confidence":0.9783,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":7,"ms":572,"offered":10,"ranked":[["go__west",0.9805],["none_of_these",0.016],["take__vellum",0.0013]],"round":2,"top":"go__west"},{"actMax":97,"asks":["attack__skeleton","drop__sword"],"clarifyMax":97,"confidence":0.9726,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":586,"offered":12,"ranked":[["attack__skeleton",0.9749],["none_of_these",0.0247],["drop__sword",0.0002]],"round":0,"top":"attack__skeleton"},{"actMax":97,"asks":["attack__skeleton","drop__sword"],"clarifyMax":97,"confidence":0.9726,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":586,"offered":12,"ranked":[["attack__skeleton",0.9749],["none_of_these",0.0247],["drop__sword",0.0002]],"round":1,"top":"attack__skeleton"},{"actMax":97,"asks":["attack__skeleton","drop__sword"],"clarifyMax":97,"confidence":0.9726,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":8,"ms":586,"offered":12,"ranked":[["attack__skeleton",0.9749],["none_of_these",0.0247],["drop__sword",0.0002]],"round":2,"top":"attack__skeleton"},{"actMax":99,"asks":["go__west","go__north"],"clarifyMax":99,"confidence":0.9956,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":579,"offered":11,"ranked":[["go__west",0.996],["none_of_these",0.0037],["go__north",0.0001]],"round":0,"top":"go__west"},{"actMax":99,"asks":["go__west","go__north"],"clarifyMax":99,"confidence":0.9956,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":572,"offered":11,"ranked":[["go__west",0.996],["none_of_these",0.0037],["go__north",0.0001]],"round":1,"top":"go__west"},{"actMax":99,"asks":["go__west","go__north"],"clarifyMax":99,"confidence":0.9956,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":9,"ms":579,"offered":11,"ranked":[["go__west",0.996],["none_of_these",0.0037],["go__north",0.0001]],"round":2,"top":"go__west"},{"actMax":99,"asks":["take__key","examine__key"],"clarifyMax":99,"confidence":0.9997,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":586,"offered":10,"ranked":[["take__key",0.9997],["examine__key",0.0001],["none_of_these",0.0001]],"round":0,"top":"take__key"},{"actMax":99,"asks":["take__key","examine__key"],"clarifyMax":99,"confidence":0.9997,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":586,"offered":10,"ranked":[["take__key",0.9997],["examine__key",0.0001],["none_of_these",0.0001]],"round":1,"top":"take__key"},{"actMax":99,"asks":["take__key","examine__key"],"clarifyMax":99,"confidence":0.9997,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":10,"ms":579,"offered":10,"ranked":[["take__key",0.9997],["examine__key",0.0001],["none_of_these",0.0001]],"round":2,"top":"take__key"},{"actMax":99,"asks":["go__east","look__around"],"clarifyMax":99,"confidence":0.9975,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":579,"offered":10,"ranked":[["go__east",0.9977],["none_of_these",0.0021],["look__around",0.0001]],"round":0,"top":"go__east"},{"actMax":99,"asks":["go__east","look__around"],"clarifyMax":99,"confidence":0.9975,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":579,"offered":10,"ranked":[["go__east",0.9977],["none_of_these",0.0021],["look__around",0.0001]],"round":1,"top":"go__east"},{"actMax":99,"asks":["go__east","look__around"],"clarifyMax":99,"confidence":0.9975,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":11,"ms":586,"offered":10,"ranked":[["go__east",0.9977],["none_of_these",0.0021],["look__around",0.0001]],"round":2,"top":"go__east"},{"actMax":98,"asks":["go__north","drop__key"],"clarifyMax":98,"confidence":0.9881,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":612,"offered":13,"ranked":[["go__north",0.989],["none_of_these",0.0104],["drop__key",0.0001]],"round":0,"top":"go__north"},{"actMax":98,"asks":["go__north","drop__key"],"clarifyMax":98,"confidence":0.9881,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":599,"offered":13,"ranked":[["go__north",0.989],["none_of_these",0.0104],["drop__key",0.0001]],"round":1,"top":"go__north"},{"actMax":98,"asks":["go__north","drop__key"],"clarifyMax":98,"confidence":0.9881,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":12,"ms":600,"offered":13,"ranked":[["go__north",0.989],["none_of_these",0.0104],["drop__key",0.0001]],"round":2,"top":"go__north"},{"actMax":99,"asks":["unlock__door","drop__key"],"clarifyMax":99,"confidence":0.9988,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":599,"offered":12,"ranked":[["unlock__door",0.9989],["none_of_these",0.001],["drop__key",0.0]],"round":0,"top":"unlock__door"},{"actMax":99,"asks":["unlock__door","drop__key"],"clarifyMax":99,"confidence":0.9988,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":600,"offered":12,"ranked":[["unlock__door",0.9989],["none_of_these",0.001],["drop__key",0.0]],"round":1,"top":"unlock__door"},{"actMax":99,"asks":["unlock__door","drop__key"],"clarifyMax":99,"confidence":0.9988,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":13,"ms":600,"offered":12,"ranked":[["unlock__door",0.9989],["none_of_these",0.001],["drop__key",0.0]],"round":2,"top":"unlock__door"},{"actMax":97,"asks":["go__north","go__south"],"clarifyMax":97,"confidence":0.9771,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":593,"offered":11,"ranked":[["go__north",0.9792],["none_of_these",0.0185],["go__south",0.001]],"round":0,"top":"go__north"},{"actMax":97,"asks":["go__north","go__south"],"clarifyMax":97,"confidence":0.9771,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":593,"offered":11,"ranked":[["go__north",0.9792],["none_of_these",0.0185],["go__south",0.001]],"round":1,"top":"go__north"},{"actMax":97,"asks":["go__north","go__south"],"clarifyMax":97,"confidence":0.9771,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":14,"ms":593,"offered":11,"ranked":[["go__north",0.9792],["none_of_these",0.0185],["go__south",0.001]],"round":2,"top":"go__north"},{"actMax":99,"asks":["take__crown","examine__crown"],"clarifyMax":99,"confidence":0.9993,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":613,"offered":12,"ranked":[["take__crown",0.9994],["none_of_these",0.0004],["examine__crown",0.0001]],"round":0,"top":"take__crown"},{"actMax":99,"asks":["take__crown","examine__crown"],"clarifyMax":99,"confidence":0.9993,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":620,"offered":12,"ranked":[["take__crown",0.9994],["none_of_these",0.0004],["examine__crown",0.0001]],"round":1,"top":"take__crown"},{"actMax":99,"asks":["take__crown","examine__crown"],"clarifyMax":99,"confidence":0.9993,"kind":"act","landed":true,"lands":{"act":true,"clarify":true,"unknown":false},"line":15,"ms":620,"offered":12,"ranked":[["take__crown",0.9994],["none_of_these",0.0004],["examine__crown",0.0001]],"round":2,"top":"take__crown"}],"sameEveryRound":true,"servedModel":"openjev-MLX-4bit","short":"27B","size":"14 GB","summary":{"act":45,"clarify":0,"error":0,"landed":48,"max_ms":634,"p50_ms":579,"p95_ms":620,"turns":48,"unknown":3},"sure":45,"temperature":"0.85","where":"Local"}],"slider":[0.0,0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.11,0.12,0.13,0.14,0.15,0.16,0.17,0.18,0.19,0.2,0.21,0.22,0.23,0.24,0.25,0.26,0.27,0.28,0.29,0.3,0.31,0.32,0.33,0.34,0.35,0.36,0.37,0.38,0.39,0.4,0.41,0.42,0.43,0.44,0.45,0.46,0.47,0.48,0.49,0.5,0.51,0.52,0.53,0.54,0.55,0.56,0.57,0.58,0.59,0.6,0.61,0.62,0.63,0.64,0.65,0.66,0.67,0.68,0.69,0.7,0.71,0.72,0.73,0.74,0.75,0.76,0.77,0.78,0.79,0.8,0.81,0.82,0.83,0.84,0.85,0.86,0.87,0.88,0.89,0.9,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99,1.0],"speedCheckMs":500,"sureAt":0.95,"thresholds":{"act":0.75,"clarify":0.4},"when":"2026-10-08T20:42:05"}; diff --git a/docs/godot-jev/index.html b/docs/godot-jev/index.html new file mode 100644 index 0000000..100b5a8 --- /dev/null +++ b/docs/godot-jev/index.html @@ -0,0 +1,593 @@ + + + + + + Parsing With Jev + + + + + + + + + + + +
    +
    + + + System One Experiments / EXP-02 godot-jev + + +
    + +
    + + + +
    +
    +
    +
    EXP-02 · godot-jev · three-model run · 2026-10-08
    +

    Three Jevs parse 16 lines of player slang.

    +

    + The Sunken Crypt is a dungeon crawl whose parser sends whatever the player typed to a System One + model, with the moves possible right now as the options of one Choice. We gave three Jevs, a hosted one + and two running on a laptop, the same 16 lines in the same game states, three times each. All three got + the player to the crown without a wrong move. They differ in how sure they were, how often the game had + to ask what the player meant, and how long a turn took on this setup. +

    +
    +
    +
    +
    + + +
    +
    +
    +
    01
    +
    +

    The problem: a parser that understands, and can't invent

    +

    + Classic parser games failed on vocabulary: "I don't know the word 'grab'." Games driven by a + text-generating model fail the other way, inventing doors and items that don't exist. Here the author + writes every room, item and line. Each turn, the moves possible right now become the options of one + Choice, plus "none of these". The model can't pick a door that isn't there, because it isn't an + option. +

    +
    +
    + +
    + The game's hall of pillars: a skeleton guards the north arch, and the transcript shows the parser asking which torch action the player meant. +
    +

    + Fig. 1The game. Room art on the left, the transcript on the right, and under the art a panel that + shows what Jev heard: the outcome, the move, and a confidence meter with ticks at the two thresholds. +

    + + +

    + Fig. 2One turn. Acting applies the move and prints its authored response. Clarifying prints "Did you + mean:" with the two likeliest moves. Unknown prints one of three authored lines such as "Nothing here + answers to that." If the model can't be reached, the game says the parser is offline and nothing happens. +

    +
    +
    + + +
    +
    +
    +
    02
    +
    +

    Hypothesis, and what would confirm it

    +
    +
    +
    + Hypothesis + A Choice over the currently possible moves lets players + type naturally with few dead ends, while + the game stays fully authored, and it's + fast enough to feel like a parser, not a chat. +
    +
    + + + + + + + + + + + + +
    CheckDone whenTypeSafeFlash 9B27BStatus
    +
    +

    + Table 1Checks from the godot-jev README, with this run's numbers at the game's thresholds. Check 1 + needs a corpus of at least 100 player inputs with typos, indirect requests and impossible ones. This run + had 16 lines written to solve the dungeon, so its count of lines landed is not evidence for it yet. +

    +
    +
    + + +
    +
    +
    +
    03
    +
    +

    The three decision models

    +

    + Same API, same game states, same 16 lines. One model is hosted by TypeSafe; two are + Open Jev models running 4-bit on an Apple M5 Pro laptop. + The first run of these lines, on 2026-09-30, had two of the three. Flash 9B is new since then. +

    +
    +
    +
    +
    +
    + + +
    +
    +
    +
    04
    +
    +

    What Jev was asked

    +

    + Each request carries a small state, the player's words, and one Choice whose options are rebuilt every + turn from the world's rules. The option text is the author's ("attack the skeleton with the + shortsword"), and it is all the model has to match the player's words against. Step through the 16 + lines to see the question change as the player moves and picks things up. +

    +
    +
    +
    +
    +

    +
    +
    + State sent with it +
    +
    +
    + The question +

    +
      +
      +
      +
      +

      +
      +
      + + +
      +
      +
      +
      05
      +
      +

      How an answer becomes a turn

      +

      + Jev returns a probability for every option and a confidence in the one it chose. The parser doesn't + just take the choice. At 0.75 or more it acts. From 0.40 it asks + "Did you mean:" and offers the two likeliest moves. Below that, or when the choice is "none of these", + the game says the request isn't possible. A wrong move changes the game's state; a question costs the + player one keypress. +

      +
      +
      + +
      +
      +
      +
      + + + +
      +
      +

      +
      +
      + Jev's answer: its three likeliest options +
      +
        +
        +
        +
        +
        + Act at a confidence of at least + 0.75 +
        + +
        + Ask at a confidence of at least + 0.40 +
        + +
        +
        +
        +
        +
        +

        + Fig. 4Pick an example, or any line, model and round. The bar shows the three likeliest options as + the run recorded them; the pale remainder is every other option together. The outcome at each setting + comes from the game's own parser, replayed over the recorded answer; no new requests are made. The ask + threshold can't be above the act threshold, so moving one past the other carries it along. +

        +
        +
        + + +
        +
        +
        +
        06
        +
        +

        Result: nobody made a wrong move

        +

        At the game's thresholds of 0.75 and 0.40, with differences shown against TypeSafe Jev.

        +
        +
        +
        +
          +
          +
          + + +
          +
          +
          +
          07
          +
          +

          The deltas between the Jevs

          +

          + Same top choice on every turn, different confidence in it. That difference is what the player would + notice: a model that is less sure makes the game ask more often. +

          +
          +
          + +

          7aHow sure each model was

          +
          +
          +
          + + +
          +
          +
          +
          +

          + +

          7bEvery line, side by side

          +
          +
          +
          + + + +
          +
          +
          +
          +
          +

          + Fig. 6Each bar is one model's first-round answer: its three likeliest options, shaded by what they + are. The mark gives the turn at the game's thresholds: ✓ acted on the move the + line means, or came back unknown for the line that means nothing; ? asked, with + the move among the two offered; ▼ a dead end; ✗ a + wrong move. Hover or tap a bar for the options and all three rounds. +

          + +
          + +

          7cHow often each pair agreed

          +
          +

          +
          +
          + + +
          +
          +
          +
          08
          +
          +

          Move the thresholds

          +

          + The thresholds are two numbers in the parser, applied to answers already returned, so they can be + tried after the fact. Raise the act threshold and a less sure model asks about more lines. Raise the + ask threshold and its questions turn into dead ends. +

          +
          +
          +
          +
          + Act at ≥ + + 0.75 + Ask at ≥ + + 0.40 +
          +
          + + +
          +
          +
          +
          + Turns answered with a question, by act threshold +
          +
          +
          + Dead ends, by ask threshold +
          +
          +
          +
          +

          + Fig. 8Panels: one row per round, one cell per line in route order, shaded by what the game did at + these thresholds. Charts: how many of each model's 48 turns ask, or dead-end, as one threshold moves and + the other stays where its slider is; the vertical rule follows the slider. A dead end is a line that + means a move and comes back unknown. +

          +
            +
            +
            + + +
            +
            +
            +
            09
            +
            +

            Speed: one model per lane

            +

            + Check 2 asks for p95 ≤ 500 ms per turn, from the player's Enter to the game's response. Measured in + the game loop around each parse, one request at a time, after one untimed warm-up per model. +

            +
            +
            +
            + Before reading these numbers +

            + They are not a statement of how fast these models are. The two Open Jev figures are what one + laptop measured: 4-bit MLX builds on an Apple M5 Pro running macOS, a desktop operating system and not a + server. Both local models were loaded at the same time, and an editor, a browser and other applications + were running. The same weights on server hardware, at another quantisation or under another runtime + would time differently. +

            +

            + The TypeSafe figure is a round trip over the network from that laptop to a hosted API, so it mixes + their serving time with our connection. Read all three as what a turn cost in this setup, on this day. +

            +
            +
            +
            +
            +

            + +

            9aTime per turn along the route

            +
            +
            +
            +

            + +
            + + + + + + + + + + + + + + +
            Decision modelRunminp50meanp95maxCheck 2
            +
            +

            +
            +
            + + +
            +
            +
            +
            10
            +
            +

            What this does and doesn't show

            +

            Read it as "the mechanism works on three models", not as a measured accuracy.

            +
            +
            +
            +
            + It shows +
              +
            • The same game, parser and thresholds work unchanged on all three decision models.
              • +
            • A Choice over the possible moves maps heavy slang to the right authored move.
              • +
            • When a model is less sure, the thresholds turn that into a question, not a wrong move.
              • +
            • The thresholds can be tried after the fact without asking again.
              • +
            +
            +
            + It doesn't show +
              +
            • 16 lines, in one voice. They were written to solve the dungeon in order for a recording, not sampled from what players type. One line is 6 percentage points.
              • +
            • Three rounds are not three samples. The two local models gave the same answer to a line every round, so Flash 9B's 6 questions are 2 lines asked 3 times.
              • +
            • One impossible request. Check 1 needs many: wrong-room requests, items that aren't there, small talk. Flash 9B put only 0.60 on "none of these" for the one it got.
              • +
            • No typos, no indirect requests, and nothing ambiguous between two moves in the same room.
              • +
            • Short option lists. Between 6 and 13 options a turn, against the 255 a Choice can take.
              • +
            • The lines don't separate the models. All three landed 48 of 48, so this says nothing about which understands best.
              • +
            • How fast the models are. The local timings come from 4-bit builds on one laptop running a desktop OS and other applications; the hosted one includes the network. See the note under Speed.
              • +
            +
            +
            + +
            +
            + + +
            +
            +
            +
            11
            +
            +

            Run details and reproducing it

            +
            +
            +
            +
            + + + + + + + + + + + + + +
            Date2026-10-08, 20:42 UTC
            TypeSafe Jevjev-1.13.0, pinned, hosted API
            Open Jev 27Bopenjev/openjev-MLX-4bit (MLX, 4-bit, 14 GB), T=0.85
            Open Jev Flash 9Bopenjev/OpenJev-Flash-9B-MLX-4bit (MLX, 4-bit, 5 GB), T=1.07
            Open Jev helpershim.py 81a22f1b for both, with each model card's calibration settings
            HardwareApple M5 Pro, 64 GB, macOS 27.2. Both local models loaded at once (ports 3002 and 3003), other applications running.
            Linestools/session.gd: 16, of which one asks for something the author never wrote
            Requests3 rounds. Each line goes to each model in turn, in the same game state, before the route moves on. First answer only, no retries. One untimed warm-up per model.
            ThresholdsAct at 0.75, clarify at 0.40, as in the game
            TimingGodot 4.7.2, headless, around each parse. The loop ticks about every 7 ms, so times fall on steps of that size.
            +
            +
            +
            cd godot-jev
+export TYPESAFE_API_KEY=sk-... TYPESAFE_DEFAULT_MODEL=jev-1.13.0
+# Open Jev 27B on :3002 and Flash 9B on :3003
+godot --headless --path . --import
+godot --headless --path . \
+  --script res://eval/compare_backends.gd -- \
+  --rounds 3 --openjev flash9b=http://127.0.0.1:3003
            +

            + Every number on this page is generated from the recorded run by + docs/tools/build-godot-jev-data.gd, which replays each answer through the game's own + parser and refuses to write if an outcome or a summary differs from what the run saved. Source: + . The written report is + results-2026-10-08.md. +

            +
            +
            +
            +
            +
            + + + + + + + + + + diff --git a/docs/godot-jev/page.css b/docs/godot-jev/page.css new file mode 100644 index 0000000..4f85b38 --- /dev/null +++ b/docs/godot-jev/page.css @@ -0,0 +1,192 @@ +/* godot-jev results page. Shared styles are in ../assets/results.css. */ + +/* Outcomes use the ordinal ramp from site.css: unknown (--r-0), clarify (--r-1), act (--r-2). + In an answer's bar the same three shades mean: "none of these", another move, the move the line means. */ + +.shot { + margin: 0 0 12px; +} +.shot img { + display: block; + width: 100%; + max-width: 900px; + height: auto; + border: 1px solid var(--rule); + border-radius: var(--radius); + box-shadow: var(--shadow-lg); +} + +/* The question: one line's state and options */ +.line-steps { + display: flex; + flex-wrap: wrap; + gap: 6px; + margin-bottom: 14px; +} +.line-steps .btn { + min-width: 34px; + padding: 6px 0; + text-align: center; + font-variant-numeric: tabular-nums; +} +.q-grid { + display: grid; + grid-template-columns: minmax(0, 0.9fr) minmax(0, 1.4fr); + gap: 28px; + align-items: start; +} +.q-grid pre { + margin: 8px 0 0; + white-space: pre-wrap; + word-break: break-word; +} +.q-grid .qtext { + margin: 6px 0 12px; +} +@media (max-width: 760px) { + .q-grid { + grid-template-columns: 1fr; + } +} +.options-list { + list-style: none; + margin: 0; + padding: 0; + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 6px 16px; + font-size: 0.85rem; +} +.options-list li { + display: flex; + flex-direction: column; + gap: 2px; + padding: 8px 10px; + border: 1px solid var(--rule); + border-radius: 6px; + color: var(--ink-2); +} +.options-list li code { + align-self: flex-start; + font-size: 0.72rem; +} +.options-list li.meant { + border-color: color-mix(in srgb, var(--stamp) 55%, transparent); + background: color-mix(in srgb, var(--stamp) 6%, var(--surface)); + color: var(--ink); +} +.options-list li.none { + grid-column: 1 / -1; + border-style: dashed; + border-color: var(--rule-strong); +} +.options-list .tag { + font-size: 0.7rem; + font-weight: 500; + color: var(--accent-ink); +} +@media (max-width: 520px) { + .options-list { + grid-template-columns: 1fr; + } +} + +/* Explorer: two thresholds */ +.ex-top3 { + list-style: none; + margin: 0 0 10px; + padding: 0; + display: grid; + gap: 4px; + font-size: 0.85rem; + color: var(--ink-2); +} +.ex-top3 li { + display: grid; + grid-template-columns: 12px minmax(0, 1fr) 40px; + gap: 8px; + align-items: center; +} +.ex-top3 i { + width: 10px; + height: 10px; + border-radius: 2px; + background: var(--c); +} +.ex-top3 b { + text-align: right; + font-weight: 600; + color: var(--ink); + font-variant-numeric: tabular-nums; +} +.stackbar .seg.rest { + background: var(--surface-2); + box-shadow: inset 0 0 0 1px var(--rule); +} +.rung .rres { + min-width: 96px; +} +.ladder .rung { + grid-template-columns: 78px minmax(0, 1fr) auto; +} + +/* Line grid: the "means" column needs room for a sentence. */ +@media (min-width: 761px) { + .tg-head, + .tg-row { + grid-template-columns: 26px minmax(0, 1.25fr) minmax(0, 1fr) repeat(3, minmax(0, 0.95fr)); + } +} +.tg-row .t small { + display: block; + color: var(--muted); + font-size: 0.72rem; +} +.tg-row .means { + font-size: 0.78rem; + color: var(--ink-2); + line-height: 1.35; +} +.tg-row .means.nothing { + color: var(--muted); + font-style: italic; +} + +/* Threshold panels: one lane per round, one cell per line. */ +.thresholds { + display: grid; + grid-template-columns: auto minmax(0, 1fr) 54px; + gap: 8px 14px; + align-items: center; + margin-bottom: 14px; +} +.thresholds input { + margin: 0; +} +.thresholds .label { + white-space: nowrap; +} +.threshold-buttons { + display: flex; + flex-wrap: wrap; + gap: 8px; + margin-bottom: 18px; +} +@media (max-width: 560px) { + .thresholds { + grid-template-columns: minmax(0, 1fr) 54px; + } + .thresholds .label { + grid-column: 1 / -1; + } +} +.bp .cells { + grid-template-columns: repeat(16, minmax(0, 1fr)); + gap: 2px; +} +.bp .cell { + font-size: 0.6rem; +} +.bp .lane { + grid-template-columns: 52px minmax(0, 1fr); +} diff --git a/docs/godot-jev/screenshot.png b/docs/godot-jev/screenshot.png new file mode 100644 index 0000000..8cd4b14 Binary files /dev/null and b/docs/godot-jev/screenshot.png differ diff --git a/docs/index.html b/docs/index.html index 014df74..6164082 100644 --- a/docs/index.html +++ b/docs/index.html @@ -254,7 +254,7 @@ /* Specimen cards */ .specimens { display: grid; - grid-template-columns: repeat(3, minmax(0, 1fr)); + grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 20px; } .specimen { @@ -346,22 +346,16 @@ margin-top: 6px; line-height: 1.4; } - @media (max-width: 1000px) { - .specimens { - grid-template-columns: repeat(2, minmax(0, 1fr)); - } - .specimens > .specimen:last-child { - grid-column: 1 / -1; - } + .timing-note { + margin: 0; + font-size: 0.78rem; + color: var(--muted); } @media (max-width: 860px) { .specimens, .specimen.featured .grid { grid-template-columns: 1fr; } - .specimens > .specimen:last-child { - grid-column: auto; - } } @media (max-width: 480px) { .finding-row { @@ -532,7 +526,7 @@ Experiments Protocol Notebook - Latest results + Latest results GitHub