From 7ae829318d1dbb516327a2ace449c146d70d9acd Mon Sep 17 00:00:00 2001
From: Etienne Filhol
Date: Wed, 18 Nov 2026 00:51:58 +0100
Subject: [PATCH 1/7] refactor(judge): call the System One judgement Decision
Every page, message, document, setting and name says Decision: the
DECISION_PROVIDER, DECISION_PROTOCOL and DECISION_MODEL settings, the
decision lane, /settings/decision/models and docs/settings/decision.md.
Settings saved as SYSTEM_ONE_* are still read, and saving the new name
from the page takes the old line out of the file. Judgements stay in the
same folder. TypeSafe's protocol keeps its id and path, systemone.
Without a provider, the judge says it needs a model, not the TypeSafe
key: OpenRouter and OpenAI serve decision models too.
---
README.md | 2 +-
app/.env.example | 18 ++---
app/nullius/judge/__init__.py | 2 +-
app/nullius/judge/batch.py | 2 +-
app/nullius/judge/client.py | 80 ++++++++++----------
app/nullius/judge/store.py | 12 +--
app/nullius/onboarding.py | 4 +-
app/nullius/providers.py | 4 +-
app/nullius/settings.py | 39 ++++++----
app/site/mkdocs.yml | 2 +-
app/static/settings/autosave.js | 2 +-
app/static/settings/judges.js | 4 +-
app/templates/compare/compare.html | 2 +-
app/templates/runs/feasibility.html | 2 +-
app/templates/runs/job.html | 2 +-
app/templates/settings/settings.html | 18 ++---
app/templates/traces/_judgement.html | 4 +-
app/templates/traces/trial.html | 2 +-
app/tests/test_judge.py | 4 +-
app/tests/test_judge_decisions.py | 65 +++++++++-------
app/tests/test_judge_openrouter.py | 22 +++---
app/tests/test_provider_registry.py | 4 +-
app/tests/test_settings.py | 10 ++-
app/ui_tests/test_every_route.py | 4 +-
app/ui_tests/test_judge_cards.py | 6 +-
app/ui_tests/test_routes.py | 2 +-
app/ui_tests/test_settings_page.py | 14 ++--
app/web/judge.py | 18 ++---
app/web/settings.py | 14 ++--
app/web/traces.py | 2 +-
docs/judgement.md | 6 +-
docs/settings/README.md | 4 +-
docs/settings/{system-one.md => decision.md} | 27 +++----
docs/settings/llm-judge.md | 6 +-
docs/settings/model-providers.md | 8 +-
35 files changed, 227 insertions(+), 190 deletions(-)
rename docs/settings/{system-one.md => decision.md} (63%)
diff --git a/README.md b/README.md
index 0c587b99..77440b41 100644
--- a/README.md
+++ b/README.md
@@ -143,7 +143,7 @@ site](app/site/README.md).
| [Judgement](docs/judgement.md) | What a judge model is asked |
| [The Nullius standard](docs/standard.md) | Ports, kits, setups, controls, proofs; importing |
| [Architecture](docs/how-it-works.md) | Execution paths, a run on disk, the pages |
-| [Settings](docs/settings/README.md) | Providers, System One, LLM judge, hosts, keys |
+| [Settings](docs/settings/README.md) | Providers, Decision, LLM judge, hosts, keys |
| [Benchmarks](docs/benchmarks/supported/README.md), [Agents](docs/agents/supported/README.md), [Capabilities](docs/capabilities/supported/README.md) | What is supported; adding one |
| [CLI](docs/cli/README.md) | Every command, by topic |
| [Metric field guide](docs/field-guide.md) | Every metric and outcome |
diff --git a/app/.env.example b/app/.env.example
index 4fb3b953..feda3fbc 100644
--- a/app/.env.example
+++ b/app/.env.example
@@ -33,20 +33,20 @@ OPENAI_API_KEY=
OPENROUTER_API_KEY=
# An Anthropic key, for a run of the Nullius standard whose agent speaks Anthropic's Messages, and for the LLM judge: the calls go to Anthropic's own base with this key. Only the relay and the judge hold it, never an agent, a recipe or a log.
ANTHROPIC_API_KEY=
-# A TypeSafe key, for the System One judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.
+# A TypeSafe key, for the Decision judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.
# e.g. ts-…
TYPESAFE_API_KEY=
-# The provider the System One judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.
-SYSTEM_ONE_PROVIDER=
-# The decision protocol System One is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.
-SYSTEM_ONE_PROTOCOL=
-# The System One model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.
-SYSTEM_ONE_MODEL=
-# The provider of the LLM judgement, the second judgement beside System One: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.
+# The provider the Decision judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.
+DECISION_PROVIDER=
+# The decision protocol Decision is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.
+DECISION_PROTOCOL=
+# The Decision model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.
+DECISION_MODEL=
+# The provider of the LLM judgement, the second judgement beside Decision: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.
LLM_JUDGE_PROVIDER=
# The protocol the LLM judge is asked in, on a provider that serves both: chat (OpenAI Chat Completions) or messages (Anthropic Messages). Unset, or one the provider does not serve: the first the provider serves, chat before messages.
LLM_JUDGE_PROTOCOL=
-# The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as System One and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.
+# The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as Decision and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.
LLM_JUDGE_MODEL=
# Additions to the OpenRouter provider, as a JSON object without the key: endpoints of protocols the preset does not list, {"endpoints": {"": "/path" or "https://full/url"}}. In the environment, a partial object merged over this one.
OPENROUTER=
diff --git a/app/nullius/judge/__init__.py b/app/nullius/judge/__init__.py
index 8299ed5f..0ec9c22d 100644
--- a/app/nullius/judge/__init__.py
+++ b/app/nullius/judge/__init__.py
@@ -1,5 +1,5 @@
"""Judgement: structured, confidence-scored answers about a recorded trial
-from a judge model (TypeSafe.ai System One), shown beside the deterministic
+from a judge model (TypeSafe.ai Decision), shown beside the deterministic
evidence and never in its place.
The contract: the question
diff --git a/app/nullius/judge/batch.py b/app/nullius/judge/batch.py
index 4b207638..6410e3be 100644
--- a/app/nullius/judge/batch.py
+++ b/app/nullius/judge/batch.py
@@ -116,7 +116,7 @@ def ask_and_store(ws_dir: str, job_id: str, trial_name: str, states: dict, key,
rec = ask_states(states, key, post=post, calibration=calibration)
if rec is not None:
# a call over some of the kinds keeps the earlier readings for the others, as the answers are kept
- lane = getattr(key, "lane", "system-one") # the LLM judge's records apart from System One's
+ lane = getattr(key, "lane", "decision") # the LLM judge's records apart from Decision's
earlier = store.load(ws_dir, job_id, trial_name, version=rec["questions"], lane=lane, started=started) or {}
rec["hygiene"] = {**(earlier.get("hygiene") or {}), **rec.get("hygiene", {})}
store.save(ws_dir, job_id, trial_name, rec, lane=lane, started=started)
diff --git a/app/nullius/judge/client.py b/app/nullius/judge/client.py
index 7208038e..496b4d6e 100644
--- a/app/nullius/judge/client.py
+++ b/app/nullius/judge/client.py
@@ -3,7 +3,7 @@
one Settings chose); each asks a provider of the registry (``nullius.providers``)
that serves its protocol:
-- **System One**, a decision model on a decision protocol: ``systemone``
+- **Decision**, a decision model on a decision protocol: ``systemone``
(TypeSafe's Jev, on TypeSafe or OpenRouter), ``decisions`` (OpenRouter's,
for many makers' models) or ``openai-decisions``. Its answers are the
model's own: a Noul's probability, a Choice's distribution over its
@@ -43,37 +43,42 @@
CALIBRATION_MIN_CHARS = 4_000 # a stored call this small says nothing about a large request's ratio: it is kept out of the band
-LANES = ("system-one", "llm") # the two judgements a trial can have, each with its own records
+LANES = ("decision", "llm") # the two judgements a trial can have, each with its own records
# The protocols each judge speaks, in its order of preference; the providers come from the registry.
LLM_PROTOCOLS = ("chat", "messages")
-SYSTEM_ONE_PROTOCOLS = ("systemone", "decisions", "openai-decisions")
-# System One's models as each provider names them: the first is its default. Another is typed in Settings.
-SYSTEM_ONE_MODELS = {"typesafe": [MODEL],
+DECISION_PROTOCOLS = ("systemone", "decisions", "openai-decisions")
+# Decision's models as each provider names them: the first is its default. Another is typed in Settings.
+DECISION_MODELS = {"typesafe": [MODEL],
"openrouter": ["~typesafe/jev-latest", "typesafe/jev-1.14", "typesafe/jev-1.13", "typesafe/jev-router", "jaredpalmer/kev-4b"],
"openai": ["gpt-6-luna"]}
-def system_one_models(choice: str, fetch: bool = False) -> list:
- """System One's models on a provider, its default first: on OpenRouter, its decision models
+def decision_judge_models(choice: str, fetch: bool = False) -> list:
+ """Decision's models on a provider, its default first: on OpenRouter, its decision models
(``providers.decision_models``, cached; asked only when ``fetch``), TypeSafe's next; else, or with
- none, ``SYSTEM_ONE_MODELS``."""
- listed = SYSTEM_ONE_MODELS.get(choice, [])
+ none, ``DECISION_MODELS``."""
+ listed = DECISION_MODELS.get(choice, [])
live = providers.decision_models(fetch=fetch) if choice == "openrouter" else []
if not live:
return list(listed)
return [listed[0]] + sorted(set(live) - {listed[0]}, key=lambda m: ("typesafe/" not in m, m))
-def system_one_protocol(choice: str, model: str, env=os.environ) -> str:
- """The decision protocol System One speaks on a provider: ``SYSTEM_ONE_PROTOCOL`` when the provider
+def _setting(env, suffix: str) -> str:
+ """``DECISION_``, else its name before the rename (``settings.RENAMED``), stripped."""
+ return (env.get(f"DECISION_{suffix}") or env.get(f"SYSTEM_ONE_{suffix}") or "").strip()
+
+
+def decision_judge_protocol(choice: str, model: str, env=os.environ) -> str:
+ """The decision protocol Decision speaks on a provider: ``DECISION_PROTOCOL`` when the provider
lists it; else, on one that serves both ``systemone`` and ``decisions`` (OpenRouter), ``systemone``
for TypeSafe's models and the ones listed for it, and ``decisions`` for another maker's; else the
first the provider lists."""
rec = providers.record(choice, env)
- listed = [p for p in SYSTEM_ONE_PROTOCOLS if providers.serves(rec, p)]
- if (env.get("SYSTEM_ONE_PROTOCOL") or "").strip() in listed:
- return env["SYSTEM_ONE_PROTOCOL"].strip()
- typesafe = "typesafe/" in model or model.startswith("jev") or model in SYSTEM_ONE_MODELS.get(choice, [])
+ listed = [p for p in DECISION_PROTOCOLS if providers.serves(rec, p)]
+ if _setting(env, "PROTOCOL") in listed:
+ return _setting(env, "PROTOCOL")
+ typesafe = "typesafe/" in model or model.startswith("jev") or model in DECISION_MODELS.get(choice, [])
if "systemone" in listed and "decisions" in listed:
return "systemone" if typesafe else "decisions"
return listed[0] if listed else "systemone"
@@ -107,11 +112,11 @@ class Judge:
def __init__(self, provider: str, key: str, model: str, url: str = "", lane: str = "", name: str = "", protocol: str = ""):
self.provider, self.key, self.model, self.url, self.name = provider, key, model, url, name
- self.lane = lane or ("system-one" if provider == "typesafe" else "llm")
- self.protocol = protocol or ("systemone" if self.lane == "system-one" else "chat")
+ self.lane = lane or ("decision" if provider == "typesafe" else "llm")
+ self.protocol = protocol or ("systemone" if self.lane == "decision" else "chat")
def __bool__(self) -> bool:
- if self.lane == "system-one":
+ if self.lane == "decision":
return bool(self.key) and bool(self.model)
return bool(self.model) and bool(self.url) and (bool(self.key) or self.provider.startswith("custom"))
@@ -129,9 +134,8 @@ def label(self) -> str:
@property
def missing(self) -> str:
"""What Settings still needs for this judge, in a phrase; '' when it can be asked."""
- if self.lane == "system-one":
- return " and ".join(([f"the {self.who} key"] if not self.key else [])
- + (["a model"] if not self.model else []))
+ if self.lane == "decision":
+ return "" if self.key and self.model else "a model"
need = []
if not self.url:
need.append("an endpoint (MODEL_API_URL)")
@@ -142,31 +146,31 @@ def missing(self) -> str:
return " and ".join(need)
-def system_one_providers(env=os.environ) -> dict:
- """System One's providers: every one that lists one of its protocols, each with its models
- (``system_one_models``, never asked here; the first, ``model``, its default; ``live`` when the
+def decision_judge_providers(env=os.environ) -> dict:
+ """Decision's providers: every one that lists one of its protocols, each with its models
+ (``decision_judge_models``, never asked here; the first, ``model``, its default; ``live`` when the
provider has a list to ask) and the protocol it would be asked in."""
- out = {k: dict(d, models=system_one_models(k), model=(SYSTEM_ONE_MODELS.get(k) or [""])[0], live=k == "openrouter")
- for k, d in _table(SYSTEM_ONE_PROTOCOLS, env, known=True).items()}
+ out = {k: dict(d, models=decision_judge_models(k), model=(DECISION_MODELS.get(k) or [""])[0], live=k == "openrouter")
+ for k, d in _table(DECISION_PROTOCOLS, env, known=True).items()}
for k, d in out.items():
- d["protocol"] = system_one_protocol(k, (env.get("SYSTEM_ONE_MODEL") or "").strip() or d["model"], env)
+ d["protocol"] = decision_judge_protocol(k, _setting(env, "MODEL") or d["model"], env)
return out
-def system_one(env=os.environ) -> Judge:
- """The System One judgement: on ``SYSTEM_ONE_PROVIDER`` (unset, or one
- that serves no System One protocol: TypeSafe when its key is set, else
+def decision_judge(env=os.environ) -> Judge:
+ """The Decision judgement: on ``DECISION_PROVIDER`` (unset, or one
+ that serves no Decision protocol: TypeSafe when its key is set, else
OpenRouter when its is), with that provider's key, and the model
- ``SYSTEM_ONE_MODEL`` names or the provider's default."""
- table = system_one_providers(env)
- chosen = _choice(env.get("SYSTEM_ONE_PROVIDER"))
+ ``DECISION_MODEL`` names or the provider's default."""
+ table = decision_judge_providers(env)
+ chosen = _choice(_setting(env, "PROVIDER"))
if chosen not in table:
chosen = "openrouter" if not env.get("TYPESAFE_API_KEY") and env.get("OPENROUTER_API_KEY") else "typesafe"
d = table[chosen]
- model = (env.get("SYSTEM_ONE_MODEL") or "").strip() or d["model"]
- protocol = system_one_protocol(chosen, model, env)
+ model = _setting(env, "MODEL") or d["model"]
+ protocol = decision_judge_protocol(chosen, model, env)
return Judge(chosen, env.get(d["key"], ""), model, providers.url(providers.record(chosen, env), protocol),
- lane="system-one", name=d["label"], protocol=protocol)
+ lane="decision", name=d["label"], protocol=protocol)
def llm_providers(env=os.environ) -> dict:
@@ -203,7 +207,7 @@ def settings_placeholder() -> str:
def judge_for(lane: str, env=os.environ) -> Judge:
- return llm_judge(env) if lane == "llm" else system_one(env)
+ return llm_judge(env) if lane == "llm" else decision_judge(env)
class JudgeError(Exception):
@@ -426,7 +430,7 @@ def ask(state: str, ids, key: str, truncated: bool = False, post: Optional[Calla
key = key.key
if not key:
raise JudgeError(401, "no TypeSafe.ai key: set TYPESAFE_API_KEY in Settings, Judge" if provider == PROVIDER
- else f"no {who} key for System One: set it in Settings, Model providers")
+ else f"no {who} key for Decision: set it in Settings, Model providers")
asked = questions if questions is not None else Q.request_questions(ids)
openai = protocol == "openai-decisions"
body = {"model": model, "input": state, "questions": openai_questions(asked)} if openai else {"state": state, "model": model, "questions": asked}
diff --git a/app/nullius/judge/store.py b/app/nullius/judge/store.py
index 93e012b5..fe35cb87 100644
--- a/app/nullius/judge/store.py
+++ b/app/nullius/judge/store.py
@@ -21,14 +21,14 @@
_SAFE = re.compile(r"[^A-Za-z0-9._@+-]")
-ROOTS = {"system-one": "judgements", "llm": "judgements-llm"} # one directory per judgement
+ROOTS = {"decision": "judgements", "llm": "judgements-llm"} # one directory per judgement
def _root(ws_dir: str, lane: str) -> str:
return os.path.join(ws_dir, ROOTS.get(lane, "judgements"))
-def _path(ws_dir: str, job_id: str, trial_name: str, lane: str = "system-one") -> str:
+def _path(ws_dir: str, job_id: str, trial_name: str, lane: str = "decision") -> str:
return os.path.join(_root(ws_dir, lane), _SAFE.sub("_", job_id), _SAFE.sub("_", trial_name) + ".json")
@@ -52,7 +52,7 @@ def _pick(data: dict, version: Optional[str], started: Optional[str] = None) ->
return None
-def load(ws_dir: str, job_id: str, trial_name: str, version: Optional[str] = None, lane: str = "system-one",
+def load(ws_dir: str, job_id: str, trial_name: str, version: Optional[str] = None, lane: str = "decision",
started: Optional[str] = None) -> Optional[dict]:
"""The record under `version`; with none named, the current version or
the newest earlier one that has a record. `started`: the trial's own
@@ -69,7 +69,7 @@ def is_current(record: Optional[dict]) -> bool:
return bool(record) and record.get("questions") == Q.VERSION
-def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "system-one",
+def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "decision",
started: Optional[str] = None) -> None:
"""The record under its question version; other versions kept. A
record's answers merge over the version's earlier answers, so judging
@@ -94,7 +94,7 @@ def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "s
os.replace(tmp, path)
-def for_job(ws_dir: str, job_id: str, version: Optional[str] = None, lane: str = "system-one",
+def for_job(ws_dir: str, job_id: str, version: Optional[str] = None, lane: str = "decision",
started: Optional[dict] = None) -> dict:
"""Every judged trial of a job: {trial_name: record}, each the current
version's record or the newest earlier one, as `load` picks it.
@@ -129,7 +129,7 @@ def count(ws_dir: str) -> int:
return sum(n.endswith(".json") for lane in ROOTS for _, _, names in os.walk(_root(ws_dir, lane)) for n in names)
-def all_records(ws_dir: str, job_id: Optional[str] = None, lane: str = "system-one"):
+def all_records(ws_dir: str, job_id: Optional[str] = None, lane: str = "decision"):
"""Every stored record, every version: yields (job_id, trial_name,
version, record). One job's when `job_id` is given."""
root = _root(ws_dir, lane)
diff --git a/app/nullius/onboarding.py b/app/nullius/onboarding.py
index 92510219..1b4047c1 100644
--- a/app/nullius/onboarding.py
+++ b/app/nullius/onboarding.py
@@ -30,7 +30,7 @@
{"id": "where", "text": "Choose where runs happen", "fact": "where",
"target": [["#hosts", "Add a host, or run on this machine with Docker or Podman"]]},
{"id": "judge", "text": "Pick a judge", "fact": "judge", "optional": True,
- "target": [["#judge", "Choose System One or an LLM judge"]]},
+ "target": [["#judge", "Choose Decision or an LLM judge"]]},
]},
{"id": "bench", "name": "Pick a benchmark", "page": "benchmarks_page", "where": "Benchmarks", "subs": [
{"id": "browse", "text": "Browse the catalogue", "visit": r"^/benchmarks/?$",
@@ -86,7 +86,7 @@ def facts(env, jobs: Iterable[dict] = (), judged: bool = False, docker: Optional
return {
"provider": bool(providers.available(env)), # a run provider: a TypeSafe key serves the judge only
"where": bool((env.get("HOSTS") or "").strip()) or found,
- "judge": bool(client.system_one(env)) or bool(client.llm_judge(env)),
+ "judge": bool(client.decision_judge(env)) or bool(client.llm_judge(env)),
"control_run": any(a in CONTROLS for a in agents),
"agent_run": any(a and a not in CONTROLS for a in agents),
"judged": bool(judged),
diff --git a/app/nullius/providers.py b/app/nullius/providers.py
index f2f80427..4e05b26b 100644
--- a/app/nullius/providers.py
+++ b/app/nullius/providers.py
@@ -1,7 +1,7 @@
"""Where model calls go: the one definition of each model provider, and the builder's steps.
Every feature reads the registry below (``record``, ``records``, ``serves``) and asks for the
-protocol it speaks: a run what its agent speaks, the LLM judge ``chat`` or ``messages``, System One
+protocol it speaks: a run what its agent speaks, the LLM judge ``chat`` or ``messages``, Decision
a decision protocol. A provider is a preset (``PRESETS``) or a custom provider, ``custom``: the
OpenAI-compatible endpoint at MODEL_API_URL, and one per id in Settings (``settings.endpoint_ids``).
A recipe names one by ``provider`` (``PROVIDERS``, the ones a run is sent to) and, for a custom one,
@@ -38,7 +38,7 @@
"chat": {"label": "OpenAI Chat Completions", "path": "/chat/completions", "auth": "authorization"},
"responses": {"label": "OpenAI Responses", "path": "/responses", "auth": "authorization"},
"messages": {"label": "Anthropic Messages", "path": "/messages", "auth": "x-api-key"},
- "systemone": {"label": "TypeSafe System One", "path": "/systemone", "auth": "authorization"},
+ "systemone": {"label": "TypeSafe Decision", "path": "/systemone", "auth": "authorization"},
"decisions": {"label": "OpenRouter Decisions", "path": "/decisions", "auth": "authorization"},
"openai-decisions": {"label": "OpenAI Decisions", "path": "/decisions", "auth": "authorization"},
}
diff --git a/app/nullius/settings.py b/app/nullius/settings.py
index f241a09a..08663b21 100644
--- a/app/nullius/settings.py
+++ b/app/nullius/settings.py
@@ -41,13 +41,13 @@
_TEXT_OK = re.compile(r"^[^\x00-\x1f]{1,400}$")
_MODEL_OK = re.compile(r"^~?[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,159}$") # an OpenRouter model id: [~]vendor/model[:variant]
# a provider choice each judge can use: a preset that serves its protocols, or custom (local, its old name);
-# System One's custom provider is a named one, the only kind whose record lists a decision protocol
+# Decision's custom provider is a named one, the only kind whose record lists a decision protocol
_ID = r"[A-Z0-9]+(_[A-Z0-9]+)*"
-_S1_OK = re.compile(rf"^(typesafe|openrouter|openai|(custom|local):{_ID})$")
+_DECISION_PROVIDER_OK = re.compile(rf"^(typesafe|openrouter|openai|(custom|local):{_ID})$")
_LLM_JUDGE_OK = re.compile(rf"^(openrouter|openai|anthropic|(custom|local)(:{_ID})?)$")
-# the protocols each judge speaks (``judge.client``: LLM_PROTOCOLS, SYSTEM_ONE_PROTOCOLS)
+# the protocols each judge speaks (``judge.client``: LLM_PROTOCOLS, DECISION_PROTOCOLS)
_LLM_PROTOCOL_OK = re.compile(r"^(chat|messages)$")
-_S1_PROTOCOL_OK = re.compile(r"^(systemone|decisions|openai-decisions)$")
+_DECISION_PROTOCOL_OK = re.compile(r"^(systemone|decisions|openai-decisions)$")
_ENVNAME_OK = re.compile(r"^[A-Z_][A-Z0-9_]*$")
_BOOL = {"0", "1", "true", "false", "yes", "no"}
@@ -149,7 +149,7 @@ class Variable:
name: str
group: str
purpose: str
- kind: str # secret | localkey | endpoint | mirror | provider | path | paths | url | repo | name | bool | text | envname | hosts | constraints | model | llmprovider | s1provider | llmprotocol | s1protocol | envnames
+ kind: str # secret | localkey | endpoint | mirror | provider | path | paths | url | repo | name | bool | text | envname | hosts | constraints | model | llmprovider | decisionprovider | llmprotocol | decisionprotocol | envnames
example: str = "" # shown as a placeholder, written to .env.example as a comment
@property
@@ -182,9 +182,9 @@ def validate(self, value: str) -> Optional[str]:
"constraints": _constraints,
"model": _check(_MODEL_OK, "a model id as OpenRouter lists it, vendor/model"),
"llmprovider": _check(_LLM_JUDGE_OK, "a provider: openrouter, openai, anthropic, custom or custom:"),
- "s1provider": _check(_S1_OK, "a provider: typesafe, openrouter, openai or custom:"),
+ "decisionprovider": _check(_DECISION_PROVIDER_OK, "a provider: typesafe, openrouter, openai or custom:"),
"llmprotocol": _check(_LLM_PROTOCOL_OK, "a protocol: chat or messages"),
- "s1protocol": _check(_S1_PROTOCOL_OK, "a protocol: systemone, decisions or openai-decisions"),
+ "decisionprotocol": _check(_DECISION_PROTOCOL_OK, "a protocol: systemone, decisions or openai-decisions"),
"envnames": _agent_key_names,
}[self.kind](value)
@@ -213,13 +213,13 @@ def validate(self, value: str) -> Optional[str]:
Variable("OPENAI_API_KEY", GROUPS[0], "An OpenAI key, for a recipe whose provider is OpenAI: that run is sent to OpenAI's own base with this key, never with the model API key or its endpoint. It reaches the run as the model key does, and never a recipe or a log.", "secret"),
Variable("OPENROUTER_API_KEY", GROUPS[0], "An OpenRouter key, for a recipe whose provider is OpenRouter: that run is sent to OpenRouter's own base with this key, never with the model API key or its endpoint. It reaches the run as the model key does, and never a recipe or a log.", "secret"),
Variable("ANTHROPIC_API_KEY", GROUPS[0], "An Anthropic key, for a run of the Nullius standard whose agent speaks Anthropic's Messages, and for the LLM judge: the calls go to Anthropic's own base with this key. Only the relay and the judge hold it, never an agent, a recipe or a log.", "secret"),
- Variable("TYPESAFE_API_KEY", GROUPS[0], "A TypeSafe key, for the System One judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.", "secret", "ts-…"),
- Variable("SYSTEM_ONE_PROVIDER", GROUPS[0], "The provider the System One judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.", "s1provider"),
- Variable("SYSTEM_ONE_PROTOCOL", GROUPS[0], "The decision protocol System One is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.", "s1protocol"),
- Variable("SYSTEM_ONE_MODEL", GROUPS[0], "The System One model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.", "model"),
- Variable("LLM_JUDGE_PROVIDER", GROUPS[0], "The provider of the LLM judgement, the second judgement beside System One: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.", "llmprovider"),
+ Variable("TYPESAFE_API_KEY", GROUPS[0], "A TypeSafe key, for the Decision judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.", "secret", "ts-…"),
+ Variable("DECISION_PROVIDER", GROUPS[0], "The provider the Decision judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.", "decisionprovider"),
+ Variable("DECISION_PROTOCOL", GROUPS[0], "The decision protocol Decision is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.", "decisionprotocol"),
+ Variable("DECISION_MODEL", GROUPS[0], "The Decision model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.", "model"),
+ Variable("LLM_JUDGE_PROVIDER", GROUPS[0], "The provider of the LLM judgement, the second judgement beside Decision: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.", "llmprovider"),
Variable("LLM_JUDGE_PROTOCOL", GROUPS[0], "The protocol the LLM judge is asked in, on a provider that serves both: chat (OpenAI Chat Completions) or messages (Anthropic Messages). Unset, or one the provider does not serve: the first the provider serves, chat before messages.", "llmprotocol"),
- Variable("LLM_JUDGE_MODEL", GROUPS[0], "The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as System One and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.", "model"),
+ Variable("LLM_JUDGE_MODEL", GROUPS[0], "The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as Decision and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.", "model"),
*(Variable(name, GROUPS[0], f"Additions to the {title} provider, as a JSON object without the key: endpoints of protocols the preset does not list, "
'{"endpoints": {"": "/path" or "https://full/url"}}. In the environment, a partial object merged over this one.', "provider")
for name, title in (("OPENROUTER", "OpenRouter"), ("OPENAI", "OpenAI"), ("ANTHROPIC", "Anthropic"), ("TYPESAFE", "TypeSafe"))),
@@ -260,7 +260,11 @@ def validate(self, value: str) -> Optional[str]:
# says which of its own names was meant - for HOST and PORT, a start-up
# option, read at start. OPENAI_API_KEY and OPENROUTER_API_KEY are its own:
# the keys a recipe on OpenAI or on OpenRouter is sent with.
-INSTEAD: Dict[str, str] = {"API_TOKEN": "MODEL_API_KEY", "API_URL": "MODEL_API_URL", "OPENAI_BASE_URL": "MODEL_API_URL",
+# The Decision judge's settings before it had that name: still read (``judge.client``), and taken out of
+# the file when the page saves the new one, so a saved choice survives the rename.
+RENAMED: Dict[str, str] = {"SYSTEM_ONE_PROVIDER": "DECISION_PROVIDER", "SYSTEM_ONE_PROTOCOL": "DECISION_PROTOCOL",
+ "SYSTEM_ONE_MODEL": "DECISION_MODEL"}
+INSTEAD: Dict[str, str] = {**RENAMED, "API_TOKEN": "MODEL_API_KEY", "API_URL": "MODEL_API_URL", "OPENAI_BASE_URL": "MODEL_API_URL",
"API_KEY_TYPESAFE": "TYPESAFE_API_KEY", "JUDGE_MODEL": "LLM_JUDGE_MODEL", "HOST": "APP_HOST", "PORT": "APP_PORT"}
@@ -641,8 +645,15 @@ def save_many(path: str, changes: Dict[str, Optional[str]]) -> None:
problems.append(f"{name} must be {why}.")
if problems:
raise ValueError(" ".join(problems))
+ # a setting saved under its new name takes its old one out of the file
+ old = {o for o, n in RENAMED.items() if n in changes and o not in changes}
lines, done = [], set()
for line, key, _ in parse(path):
+ if key in old:
+ if key in LOADED_KEYS:
+ os.environ.pop(key, None)
+ LOADED_KEYS.discard(key)
+ continue
if key in changes:
value = changes[key]
if value is not None and key not in done:
diff --git a/app/site/mkdocs.yml b/app/site/mkdocs.yml
index 82ac2df5..ce75b8cd 100644
--- a/app/site/mkdocs.yml
+++ b/app/site/mkdocs.yml
@@ -109,7 +109,7 @@ nav:
- Settings:
- Overview: settings/README.md
- Model providers: settings/model-providers.md
- - System One provider: settings/system-one.md
+ - Decision provider: settings/decision.md
- LLM judge provider: settings/llm-judge.md
- Custom keys: settings/agent-keys.md
- Model prices: settings/model-prices.md
diff --git a/app/static/settings/autosave.js b/app/static/settings/autosave.js
index 72c2809c..7cd43b8f 100644
--- a/app/static/settings/autosave.js
+++ b/app/static/settings/autosave.js
@@ -33,7 +33,7 @@
} else if (key.startsWith('value:')) names.add(key.slice(6));
}
// A model edit always carries the displayed provider, including a sole auto-selection.
- for (const prefix of ['SYSTEM_ONE', 'LLM_JUDGE']) {
+ for (const prefix of ['DECISION', 'LLM_JUDGE']) {
if (body.has('value:' + prefix + '_MODEL') || body.has('clear:' + prefix + '_MODEL')) {
const provider = current.get('value:' + prefix + '_PROVIDER');
if (provider) body.set('value:' + prefix + '_PROVIDER', provider);
diff --git a/app/static/settings/judges.js b/app/static/settings/judges.js
index 7f0864a6..c01f5b58 100644
--- a/app/static/settings/judges.js
+++ b/app/static/settings/judges.js
@@ -47,13 +47,13 @@
if (model) model.placeholder = on?.dataset.default || '';
}
// A provider with no list of its own here: its catalogue, asked once when the field is first used; a live list
- // (System One on OpenRouter) replaces the one drawn with the page. Typing a model still works.
+ // (Decision on OpenRouter) replaces the one drawn with the page. Typing a model still works.
const catalogues = new Map();
async function offerCatalogue(card, model) {
const on = chosen(card);
if (!on || (on.dataset.models && !('live' in on.dataset)) || !model.list) return;
const provider = on.value.replace(/^local(?=$|:)/, 'custom'); // local:, a named endpoint, is the builder's custom:
- const url = 'live' in on.dataset ? '/settings/system-one/models' : '/new-run/models';
+ const url = 'live' in on.dataset ? '/settings/decision/models' : '/new-run/models';
if (!catalogues.has(url + provider)) {
catalogues.set(url + provider, nulliusFetch(url, {method: 'POST', headers: {'Content-Type': 'application/json',
'X-CSRF-Token': document.querySelector('#settings-form [name=csrf]').value}, body: JSON.stringify({provider})})
diff --git a/app/templates/compare/compare.html b/app/templates/compare/compare.html
index ee3fe3fb..bdaaa627 100644
--- a/app/templates/compare/compare.html
+++ b/app/templates/compare/compare.html
@@ -144,7 +144,7 @@
Task by task
- {% elif judgement.unjudged %}System One not configured — set its key in Settings{% endif %}
+ {% elif judgement.unjudged %}Decision not configured: needs a model — choose one in Settings{% endif %}
{% endif %}
diff --git a/app/templates/runs/feasibility.html b/app/templates/runs/feasibility.html
index 1ba7cd28..3508c1c2 100644
--- a/app/templates/runs/feasibility.html
+++ b/app/templates/runs/feasibility.html
@@ -68,7 +68,7 @@
- {% elif triage.unjudged %}System One not configured — set its key in Settings{% endif %}
+ {% elif triage.unjudged %}Decision not configured: needs a model — choose one in Settings{% endif %}
{% endif %}
diff --git a/app/templates/runs/job.html b/app/templates/runs/job.html
index 97a571b7..c240c199 100644
--- a/app/templates/runs/job.html
+++ b/app/templates/runs/job.html
@@ -298,7 +298,7 @@
{% if (judgement.unjudged) or (judgement.llm.unjudged) or (tq and tq.unjudged) %}
{# one button per judge: each judges apart, so a person asks for the one they want #}
- {% for lane, name, n, keyed in [('system-one', 'System One', judgement.unjudged, judgement.keyed), ('llm', 'LLM judge', judgement.llm.unjudged, judgement.llm.keyed)] if n %}
+ {% for lane, name, n, keyed in [('decision', 'Decision', judgement.unjudged, judgement.keyed), ('llm', 'LLM judge', judgement.llm.unjudged, judgement.llm.keyed)] if n %}
{% if keyed %}
@@ -80,7 +80,7 @@
Settings
{% for b in benchmark_access %}{% set access_search.words = access_search.words ~ ' ' ~ b.title %}{% for s in b.needs %}{% set access_search.words = access_search.words ~ ' ' ~ s.name ~ ' ' ~ s.summary %}{% endfor %}{% endfor %}
{{ section_link('data-providers', 'Data providers', 'cloud-down', 'hugging face huggingface hf_token docker hub login registry token dataset pull limit') }}
{{ section_link('custom-keys', 'Custom keys', 'key', access_search.words) }}
- {{ section_link('judge', 'Judges', 'scale', 'typesafe system one llm model judgement') }}
+ {{ section_link('judge', 'Judges', 'scale', 'typesafe decision llm model judgement') }}
{{ section_link('prices', 'Model prices', 'dollar', 'cost price estimate usd tokens list update') }}
@@ -186,7 +186,7 @@
Add a provider
Judges
Choose the judges that evaluate trial evidence.
- {% set s1, llm = judges %}
+ {% set decision, llm = judges %}
{#- Provider, then Protocol when the provider lists two this judge speaks (else a quiet line), then Model. -#}
{% macro judge_rows(prefix, choices, judge, label) %}
@@ -218,12 +218,12 @@
Judges
{% endmacro %}
- {% set srow = by.get('SYSTEM_ONE_PROVIDER') %}
- {% if srow and by.get('SYSTEM_ONE_MODEL') %}
-
', s1)
+ s1, llm = (html[html.index(f'id="{p}_PROVIDER"'):html.index(f'id="{p}_MODEL"')] for p in ("DECISION", "LLM_JUDGE"))
+ self.assertIn('
via TypeSafe Decision
', s1)
self.assertIn('
via Anthropic Messages
', llm)
self.assertIn('id="LLM_JUDGE_PROTOCOL" data-source="unset" data-secret="no" hidden>', llm)
self.assertNotIn("messages<", llm); self.assertNotIn(">chat<", llm) # names, never keys
diff --git a/app/web/judge.py b/app/web/judge.py
index 06abdce8..ac455336 100644
--- a/app/web/judge.py
+++ b/app/web/judge.py
@@ -16,9 +16,9 @@
from . import settings as settings_web
from . import traces
-def key(lane: str = "system-one") -> client.Judge:
+def key(lane: str = "decision") -> client.Judge:
"""A judgement's judge, with its key and model: true when it can be
- asked. System One unless the LLM judgement is named; passed down to
+ asked. Decision unless the LLM judgement is named; passed down to
every call as the key is."""
return client.judge_for(lane, os.environ)
@@ -73,12 +73,12 @@ def started_by_name(job) -> dict:
return {store._SAFE.sub("_", t.trial_name): started_of(t) for t in job.trials}
-def judgement_of(job, trial, lane: str = "system-one") -> dict | None:
+def judgement_of(job, trial, lane: str = "decision") -> dict | None:
d = ws_dir()
return store.load(d, job.job_id, trial.trial_name, lane=lane, started=started_of(trial)) if d else None
-def panel(job, trial, lane: str = "system-one") -> dict:
+def panel(job, trial, lane: str = "decision") -> dict:
"""What the trial page renders: each question with its answer or 'not
judged', the record's provenance, whether a Judge action is possible.
A record asked under an earlier vocabulary is shown under its own
@@ -271,7 +271,7 @@ def judge_trial_route(job_id, trial_name):
"""One click, one trial: the page's questions asked and stored."""
_guard()
job, trial = traces._trial_evidence(job_id, trial_name)
- lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "system-one"
+ lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "decision"
try:
rec = judge_trial(job, trial, key(lane))
except client.JudgeError as e:
@@ -289,11 +289,11 @@ def judge_trial_route(job_id, trial_name):
@app.route("/judge/job/", methods=["POST"])
def judge_job_route(job_id):
"""Judge the job's unjudged trials with one judge, the form's ``lane``
- (System One unless it names the LLM judge): the click is the request."""
+ (Decision unless it names the LLM judge): the click is the request."""
job = shared._find_job(shared._jobs_for_request(), job_id)
if job is None:
abort(404)
- lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "system-one"
+ lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "decision"
d = ws_dir()
judged = store.for_job(d, job_id, lane=lane, started=started_by_name(job)) if d else {}
todo = [t for t in job.trials if t.has_trajectory and store._SAFE.sub("_", t.trial_name) not in judged]
@@ -301,7 +301,7 @@ def judge_job_route(job_id):
k = key(lane)
done, failed = run_batch(todo, lambda t: judge_trial(job, t, k), lambda t: t.trial_name)
from flask import flash
- flash(f"{'LLM judge' if lane == 'llm' else 'System One'}: judged {done} of {len(todo)} trial(s)." + (" Failed: " + "; ".join(failed[:3]) if failed else ""), "error" if failed else "")
+ flash(f"{'LLM judge' if lane == 'llm' else 'Decision'}: judged {done} of {len(todo)} trial(s)." + (" Failed: " + "; ".join(failed[:3]) if failed else ""), "error" if failed else "")
return redirect(url_for("job", job_id=job_id) + "#judgement")
@@ -347,7 +347,7 @@ def job_summary(job) -> dict:
def _llm_count(d, job) -> dict:
"""The LLM judge's own count for the job's button: it judges apart from
- System One, so a trial judged by one is still unjudged by the other."""
+ Decision, so a trial judged by one is still unjudged by the other."""
judged = store.for_job(d, job.job_id, lane="llm", started=started_by_name(job)) if d else {}
return {"unjudged": len([t for t in job.trials if store._SAFE.sub("_", t.trial_name) not in judged]), "keyed": bool(key("llm"))}
diff --git a/app/web/settings.py b/app/web/settings.py
index f7601ae6..b74eb2ba 100644
--- a/app/web/settings.py
+++ b/app/web/settings.py
@@ -54,8 +54,8 @@ def settings_page():
prices=prices.state(),
machines=_machines(), constraint_table=constraints.TABLE,
provider_cards=provider_cards({r["name"]: r for r in rows}),
- ssh_hosts=hosts.scan_ssh_config(), judges=(client.system_one(os.environ), client.llm_judge(os.environ)), llm_providers=client.llm_providers(os.environ), s1_providers=client.system_one_providers(os.environ),
- judge_protocols={"SYSTEM_ONE": client.SYSTEM_ONE_PROTOCOLS, "LLM_JUDGE": client.LLM_PROTOCOLS}, protocols=providers.PROTOCOLS, agent_keys=settings.agent_key_names(), reserved_agent_keys=settings.reserved_agent_keys(),
+ ssh_hosts=hosts.scan_ssh_config(), judges=(client.decision_judge(os.environ), client.llm_judge(os.environ)), llm_providers=client.llm_providers(os.environ), decision_providers=client.decision_judge_providers(os.environ),
+ judge_protocols={"DECISION": client.DECISION_PROTOCOLS, "LLM_JUDGE": client.LLM_PROTOCOLS}, protocols=providers.PROTOCOLS, agent_keys=settings.agent_key_names(), reserved_agent_keys=settings.reserved_agent_keys(),
tested={k: _shown(v) for k, v in providers.tests().items() if isinstance(v, dict)})
@@ -78,14 +78,14 @@ def settings_provider_test():
return jsonify(_shown(got))
-@app.route("/settings/system-one/models", methods=["POST"])
-def settings_system_one_models():
- """System One's models on OpenRouter for its card, asked after the page has loaded: OpenRouter's
- public list of decision models, kept in the cache (``client.system_one_models``)."""
+@app.route("/settings/decision/models", methods=["POST"])
+def settings_decision_judge_models():
+ """Decision's models on OpenRouter for its card, asked after the page has loaded: OpenRouter's
+ public list of decision models, kept in the cache (``client.decision_judge_models``)."""
shared._require_csrf()
if not _local_client():
abort(403)
- return jsonify(ok=True, models=[{"id": m, "name": m} for m in client.system_one_models("openrouter", fetch=True)])
+ return jsonify(ok=True, models=[{"id": m, "name": m} for m in client.decision_judge_models("openrouter", fetch=True)])
def _size(mb: int) -> str:
diff --git a/app/web/traces.py b/app/web/traces.py
index 03df52dc..e41a4fa9 100644
--- a/app/web/traces.py
+++ b/app/web/traces.py
@@ -310,7 +310,7 @@ def trial_attempts(job_id, trial_name):
imported_from=_imported_from(j))
-def _judgement_panel(job, trial, lane="system-one"):
+def _judgement_panel(job, trial, lane="decision"):
from . import judge
return judge.panel(job, trial, lane)
diff --git a/docs/judgement.md b/docs/judgement.md
index 8796dc21..7db3a660 100644
--- a/docs/judgement.md
+++ b/docs/judgement.md
@@ -6,10 +6,10 @@ Two judgements per trial, a tab each, records in `judgements/` and `judgements-l
| Judgement | Answers |
| --- | --- |
-| [System One](settings/system-one.md) | A decision model's answers: a yes/no's probability, a choice's distribution, a score, each with the model's confidence when it gives one; one call per state |
+| [Decision](settings/decision.md) | A decision model's answers: a yes/no's probability, a choice's distribution, a score, each with the model's confidence when it gives one; one call per state |
| [LLM judge](settings/llm-judge.md) | What a chat model states, one JSON object per state, no confidence |
-The job, compare, feasibility and task-instruction pages ask System One. A record names the provider and the model that answered, and the questions the model refused. The cost estimate for TypeSafe is calibrated on its own stored calls.
+The job, compare, feasibility and task-instruction pages ask Decision. A record names the provider and the model that answered, and the questions the model refused. The cost estimate for TypeSafe is calibrated on its own stored calls.
!!! note "A judge never changes an outcome, a count or a verdict"
@@ -76,7 +76,7 @@ Follow the evidence: a recorded rate limit is a provider's failure whatever the
-- [System One provider](settings/system-one.md)
+- [Decision provider](settings/decision.md)
- [LLM judge provider](settings/llm-judge.md)
- [Read results](results.md)
diff --git a/docs/settings/README.md b/docs/settings/README.md
index ecfefed8..4b5f4e4f 100644
--- a/docs/settings/README.md
+++ b/docs/settings/README.md
@@ -7,7 +7,7 @@ changes save automatically; data providers and benchmark credentials use **Save*
| Page | Sets |
| --- | --- |
| [Model providers](model-providers.md) | A run's key or endpoint |
-| [System One provider](system-one.md) | The System One judgement |
+| [Decision provider](decision.md) | The Decision judgement |
| [LLM judge provider](llm-judge.md) | The second judge |
| [Custom keys](agent-keys.md) | Agent credentials, dataset keys and benchmark tool licences |
| [Model prices](model-prices.md) | What a run's cost is estimated from |
@@ -26,7 +26,7 @@ changes save automatically; data providers and benchmark credentials use **Save*
| `OPENAI_API_KEY` | Runs on OpenAI |
| `OPENROUTER_API_KEY` | Runs on OpenRouter |
| `ANTHROPIC_API_KEY` | Standard runs whose agent speaks `messages`, and the LLM judge on Anthropic |
-| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL`, `TYPESAFE_API_KEY` | [System One](system-one.md) |
+| `DECISION_PROVIDER`, `DECISION_PROTOCOL`, `DECISION_MODEL`, `TYPESAFE_API_KEY` | [Decision](decision.md) |
| `LLM_JUDGE_PROVIDER`, `LLM_JUDGE_PROTOCOL`, `LLM_JUDGE_MODEL` | [LLM judge](llm-judge.md) |
| `AGENT_KEYS` | Comma-separated agent key names; Harbor runs only |
| `HOSTS` | The hosts |
diff --git a/docs/settings/system-one.md b/docs/settings/decision.md
similarity index 63%
rename from docs/settings/system-one.md
rename to docs/settings/decision.md
index f41623d5..094d3ac9 100644
--- a/docs/settings/system-one.md
+++ b/docs/settings/decision.md
@@ -1,11 +1,11 @@
-# System One provider
+# Decision provider
-The provider and model that answer the System One judgement: questions about recorded evidence.
+The provider and model that answer the Decision judgement: questions about recorded evidence.
-System One asks a provider that lists a decision protocol
+Decision asks a provider that lists a decision protocol
([protocols](model-providers.md#protocols)).
-| `SYSTEM_ONE_PROVIDER` | Credential | Protocol | Default model |
+| `DECISION_PROVIDER` | Credential | Protocol | Default model |
| --- | --- | --- | --- |
| `typesafe` | `TYPESAFE_API_KEY` | `systemone` | `jev-latest` |
| `openrouter` | `OPENROUTER_API_KEY` | `systemone` for TypeSafe's models and `jaredpalmer/kev-4b`, `decisions` for other makers' | `~typesafe/jev-latest` |
@@ -14,9 +14,10 @@ System One asks a provider that lists a decision protocol
| Setting | Value |
| --- | --- |
-| `SYSTEM_ONE_PROVIDER` | Unset: TypeSafe when its key is set, else OpenRouter |
-| `SYSTEM_ONE_PROTOCOL` | `systemone`, `decisions` or `openai-decisions`, on a provider that serves more than one; unset, or not served: the protocol in the table above |
-| `SYSTEM_ONE_MODEL` | A model the provider accepts, such as `typesafe/jev-1.14` |
+| `DECISION_PROVIDER` | Unset: TypeSafe when its key is set, else OpenRouter |
+| `DECISION_PROTOCOL` | `systemone`, `decisions` or `openai-decisions`, on a provider that serves more than one; unset, or not served: the protocol in the table above |
+| `DECISION_MODEL` | A model the provider accepts, such as `typesafe/jev-1.14` |
+| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL` | The names before; still read. Saving in Settings moves each to its new name |
| Reply | Recorded as |
| --- | --- |
@@ -26,7 +27,7 @@ System One asks a provider that lists a decision protocol
## Configure
-Add the connection in **Settings → Model providers** first. **Judges** offers only configured, compatible providers. A sole provider is selected automatically; with several, choose one and set the model. Under the provider, the card names the protocol System One uses. When the provider serves more than one, choose one under **Protocol**. Keys and endpoints stay in Model providers.
+Add the connection in **Settings → Model providers** first. **Judges** offers only configured, compatible providers. A sole provider is selected automatically; with several, choose one and set the model. Under the provider, the card names the protocol Decision uses. When the provider serves more than one, choose one under **Protocol**. Keys and endpoints stay in Model providers.
On OpenRouter, the **Model** suggestions are OpenRouter's public list of decision models. Nullius gets the list without a key when you first use the field, and keeps it in the cache for 6 hours. With no network and no cache, the suggestions are a short built-in list.
@@ -38,8 +39,8 @@ your existing `settings.json` ([location and precedence](README.md#save)).
```json title="settings.json"
{
"TYPESAFE_API_KEY": "YOUR_TYPESAFE_KEY",
- "SYSTEM_ONE_PROVIDER": "typesafe",
- "SYSTEM_ONE_MODEL": "jev-latest"
+ "DECISION_PROVIDER": "typesafe",
+ "DECISION_MODEL": "jev-latest"
}
```
@@ -48,14 +49,14 @@ your existing `settings.json` ([location and precedence](README.md#save)).
```json title="settings.json"
{
"OPENROUTER_API_KEY": "YOUR_OPENROUTER_KEY",
- "SYSTEM_ONE_PROVIDER": "openrouter",
- "SYSTEM_ONE_MODEL": "~typesafe/jev-latest"
+ "DECISION_PROVIDER": "openrouter",
+ "DECISION_MODEL": "~typesafe/jev-latest"
}
```
## Request a judgement
-A trial's or a job's System One tab.
+A trial's or a job's Decision tab.
!!! warning "A judgement is a paid call"
Configuration alone calls nothing; a batch button says how many calls it will make.
diff --git a/docs/settings/llm-judge.md b/docs/settings/llm-judge.md
index fd4719ce..45cd1d76 100644
--- a/docs/settings/llm-judge.md
+++ b/docs/settings/llm-judge.md
@@ -1,6 +1,6 @@
# LLM judge provider
-The same questions as System One, answered as JSON by a chat model.
+The same questions as Decision, answered as JSON by a chat model.
| Setting | Value |
| --- | --- |
@@ -9,7 +9,7 @@ The same questions as System One, answered as JSON by a chat model.
| Credential | [Model providers](model-providers.md); a custom provider needs its base |
| `LLM_JUDGE_PROTOCOL` | `chat` or `messages`, on a provider that serves both; unset, or not served: `chat` where the provider serves it, else `messages` |
-| Differs from System One | |
+| Differs from Decision | |
| --- | --- |
| Answers | What the model states; no confidence value |
| Out of range | An answer outside its question's options or range is left out |
@@ -39,7 +39,7 @@ your existing `settings.json` ([location and precedence](README.md#save)).
Evidence, storage, cost.
-- [System One](system-one.md)
+- [Decision](decision.md)
The first judgement.
diff --git a/docs/settings/model-providers.md b/docs/settings/model-providers.md
index f26a5bcc..826dddc9 100644
--- a/docs/settings/model-providers.md
+++ b/docs/settings/model-providers.md
@@ -1,7 +1,7 @@
# Model providers
Where model calls go, and with which key. Each provider is defined once, and
-every feature reads that definition: runs, the provider test, System One and
+every feature reads that definition: runs, the provider test, Decision and
the LLM judge. A feature asks for the protocol it speaks and gets the providers
that can serve it.
@@ -28,7 +28,7 @@ and how the key is sent. The feature that calls the provider picks the protocol.
| `chat` | OpenAI Chat Completions | `/chat/completions` | Bearer |
| `responses` | OpenAI Responses | `/responses` | Bearer |
| `messages` | Anthropic Messages | `/messages` | `x-api-key` and `anthropic-version` |
-| `systemone` | TypeSafe System One | `/systemone` | Bearer |
+| `systemone` | TypeSafe Decision | `/systemone` | Bearer |
| `decisions` | OpenRouter Decisions (alpha) | `/decisions` | Bearer |
| `openai-decisions` | OpenAI Decisions (beta) | `/decisions` | Bearer |
@@ -36,7 +36,7 @@ and how the key is sent. The feature that calls the provider picks the protocol.
| --- | --- |
| A run | What its agent speaks (`model.apis` in its definition); a Harbor run, `chat` |
| LLM judge | `chat`, or `messages` on Anthropic |
-| System One | `systemone`, `decisions` or `openai-decisions` |
+| Decision | `systemone`, `decisions` or `openai-decisions` |
Anthropic is offered to agents that speak `messages`. It is not offered to an
agent that speaks only `chat`, to a Harbor recipe, or to a judged benchmark.
@@ -178,7 +178,7 @@ Keys keep their names. `LLM_JUDGE_PROVIDER` still reads `local` and
Precedence, reloads.
-- [System One](system-one.md), [LLM judge](llm-judge.md)
+- [Decision](decision.md), [LLM judge](llm-judge.md)
The judges.
From 95b1ba46846eea6d166cf764a5a11babf300798d Mon Sep 17 00:00:00 2001
From: Etienne Filhol
Date: Tue, 17 Nov 2026 00:06:04 +0100
Subject: [PATCH 2/7] fix(settings): read a Decision setting saved under its
old name as the new one
The page showed DECISION_* unset and listed SYSTEM_ONE_* as unread while
the judge still used them. The file's old names are now read as the new
ones, so the page shows them set and the next save writes the new names.
---
app/nullius/settings.py | 16 +++++++++++++++-
app/tests/test_settings.py | 23 ++++++++++++++++++-----
docs/settings/decision.md | 2 +-
3 files changed, 34 insertions(+), 7 deletions(-)
diff --git a/app/nullius/settings.py b/app/nullius/settings.py
index 08663b21..90bea904 100644
--- a/app/nullius/settings.py
+++ b/app/nullius/settings.py
@@ -287,7 +287,7 @@ def parse(path: str) -> List[Tuple[str, Optional[str], Optional[str]]]:
for k, v in values.items()
):
raise ValueError("Settings must be an object of variable names and single-line strings.")
- return [(f"{key}={value}", key, value) for key, value in values.items()]
+ return _renamed([(f"{key}={value}", key, value) for key, value in values.items()])
for raw in fh:
line = raw.rstrip("\n")
s = line.strip()
@@ -296,6 +296,20 @@ def parse(path: str) -> List[Tuple[str, Optional[str], Optional[str]]]:
continue
key, _, value = s.partition("=")
out.append((line, key.strip(), value.strip().strip('"').strip("'")))
+ return _renamed(out)
+
+
+def _renamed(entries: List[Tuple[str, Optional[str], Optional[str]]]) -> List[Tuple[str, Optional[str], Optional[str]]]:
+ """A setting written under its old name (``RENAMED``) read as its new one, its line too, so the page
+ shows it set and the next save writes the new name. A file that holds both keeps the new name's
+ value; the old line stays unread until a save of the new one takes it out."""
+ present = {k for _, k, _ in entries if k}
+ out = []
+ for line, key, value in entries:
+ new = RENAMED.get(key or "")
+ if new and new not in present:
+ line, key = line.replace(key, new, 1), new
+ out.append((line, key, value))
return out
diff --git a/app/tests/test_settings.py b/app/tests/test_settings.py
index 1c400410..c2a80c52 100644
--- a/app/tests/test_settings.py
+++ b/app/tests/test_settings.py
@@ -255,13 +255,26 @@ def test_named_endpoints_are_settings_of_their_own_and_the_environment_wins(self
with self.assertRaises(ValueError):
settings.save_many(self.path, {"LLM_JUDGE_PROVIDER": "local:gpu"})
- def test_saving_a_decision_setting_takes_its_old_name_out_of_the_file(self):
+ def test_a_decision_setting_saved_under_its_old_name_reads_as_the_new_one(self):
+ """Saved before the rename: the page shows it set, nothing calls it unread, and a save of
+ another setting keeps it and writes it under the new name."""
self.write("SYSTEM_ONE_MODEL=typesafe/jev-1.14\nSYSTEM_ONE_PROVIDER=openrouter\nHOSTS=bench\n")
settings.load(self.path)
- self.assertEqual([u["instead"] for u in settings.unread(self.path)], ["DECISION_MODEL", "DECISION_PROVIDER"])
- settings.save_many(self.path, {"DECISION_MODEL": "typesafe/jev-2"})
- self.assertEqual(settings.in_file(self.path), {"SYSTEM_ONE_PROVIDER": "openrouter", "HOSTS": "bench", "DECISION_MODEL": "typesafe/jev-2"})
- self.assertNotIn("SYSTEM_ONE_MODEL", os.environ); self.assertEqual(os.environ["DECISION_MODEL"], "typesafe/jev-2")
+ by = {st["name"]: st for st in settings.states(self.path)}
+ self.assertEqual((by["DECISION_MODEL"]["source"], by["DECISION_MODEL"]["shown"]), ("file", "typesafe/jev-1.14"))
+ self.assertEqual(settings.unread(self.path), [])
+ settings.save_many(self.path, {"DECISION_PROVIDER": "typesafe"}) # the provider alone: the model stays
+ with open(self.path) as fh:
+ self.assertEqual(fh.read(), "DECISION_MODEL=typesafe/jev-1.14\nDECISION_PROVIDER=typesafe\nHOSTS=bench\n")
+ self.assertEqual(os.environ["DECISION_MODEL"], "typesafe/jev-1.14")
+
+ def test_with_both_names_in_the_file_the_new_one_is_read_and_a_save_removes_the_old(self):
+ self.write("SYSTEM_ONE_MODEL=old\nDECISION_MODEL=new\n")
+ settings.load(self.path)
+ self.assertEqual(os.environ["DECISION_MODEL"], "new")
+ self.assertEqual([u["instead"] for u in settings.unread(self.path)], ["DECISION_MODEL"])
+ settings.save_many(self.path, {"DECISION_MODEL": "newer"})
+ self.assertEqual(settings.in_file(self.path), {"DECISION_MODEL": "newer"})
def test_an_endpoint_url_is_shown_so_it_holds_no_credential(self):
"""An endpoint's URL is shown on the page: an http(s) URL with a host,
diff --git a/docs/settings/decision.md b/docs/settings/decision.md
index 094d3ac9..1807ce92 100644
--- a/docs/settings/decision.md
+++ b/docs/settings/decision.md
@@ -17,7 +17,7 @@ Decision asks a provider that lists a decision protocol
| `DECISION_PROVIDER` | Unset: TypeSafe when its key is set, else OpenRouter |
| `DECISION_PROTOCOL` | `systemone`, `decisions` or `openai-decisions`, on a provider that serves more than one; unset, or not served: the protocol in the table above |
| `DECISION_MODEL` | A model the provider accepts, such as `typesafe/jev-1.14` |
-| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL` | The names before; still read. Saving in Settings moves each to its new name |
+| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL` | The names before: read as the new ones; the next save in Settings writes the new names |
| Reply | Recorded as |
| --- | --- |
From 49ff0a8f8ffc86c34152da02ea8907755ba981cc Mon Sep 17 00:00:00 2001
From: Etienne Filhol
Date: Tue, 17 Nov 2026 00:22:38 +0100
Subject: [PATCH 3/7] fix(settings): name the judge cards Decision model and
LLM
The Judges section already says judge.
---
app/templates/settings/settings.html | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/app/templates/settings/settings.html b/app/templates/settings/settings.html
index 9ad2d793..d37f2b77 100644
--- a/app/templates/settings/settings.html
+++ b/app/templates/settings/settings.html
@@ -221,7 +221,7 @@
Judges
{% set srow = by.get('DECISION_PROVIDER') %}
{% if srow and by.get('DECISION_MODEL') %}
From adc1c06e81230ead0b91751182abcc5a1764ff44 Mon Sep 17 00:00:00 2001
From: Etienne Filhol
Date: Wed, 18 Nov 2026 00:54:07 +0100
Subject: [PATCH 4/7] docs(settings): say the old Decision names in three words
---
docs/settings/decision.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/settings/decision.md b/docs/settings/decision.md
index 1807ce92..f8ce9fe0 100644
--- a/docs/settings/decision.md
+++ b/docs/settings/decision.md
@@ -17,7 +17,7 @@ Decision asks a provider that lists a decision protocol
| `DECISION_PROVIDER` | Unset: TypeSafe when its key is set, else OpenRouter |
| `DECISION_PROTOCOL` | `systemone`, `decisions` or `openai-decisions`, on a provider that serves more than one; unset, or not served: the protocol in the table above |
| `DECISION_MODEL` | A model the provider accepts, such as `typesafe/jev-1.14` |
-| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL` | The names before: read as the new ones; the next save in Settings writes the new names |
+| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL` | Old names, still read |
| Reply | Recorded as |
| --- | --- |
From 5ec15bec90c8f8e59c5d12d3c49a93169714f2d7 Mon Sep 17 00:00:00 2001
From: Etienne Filhol
Date: Tue, 17 Nov 2026 00:54:15 +0100
Subject: [PATCH 5/7] feat(judge): put the judge's button under its badge, and
say what its last call took
On a trial's judge tabs the button sits on the left under the experimental badge. Once judged, a line beside it says how long the calls took, the model that answered, the tokens and the cost, each when the record has it. The client times its calls and keeps the seconds with the usage; a judgement of several calls adds them up.
---
app/nullius/judge/client.py | 6 +++++-
app/static/style.css | 4 ++++
app/templates/traces/_judgement.html | 3 +++
app/tests/test_judge.py | 2 +-
app/tests/test_judge_openrouter.py | 1 +
app/ui_tests/test_trial_head.py | 19 +++++++++++++++++++
6 files changed, 33 insertions(+), 2 deletions(-)
diff --git a/app/nullius/judge/client.py b/app/nullius/judge/client.py
index 7208038e..7082e956 100644
--- a/app/nullius/judge/client.py
+++ b/app/nullius/judge/client.py
@@ -432,6 +432,7 @@ def ask(state: str, ids, key: str, truncated: bool = False, post: Optional[Calla
body = {"model": model, "input": state, "questions": openai_questions(asked)} if openai else {"state": state, "model": model, "questions": asked}
post = post or http_post
status, data = 0, {}
+ started = time.monotonic()
for attempt in range(retries + 1):
status, data = post(url, body, key)
if status in (429, 503, 529) and attempt < retries:
@@ -455,6 +456,7 @@ def ask(state: str, ids, key: str, truncated: bool = False, post: Optional[Calla
rec = {"questions": Q.VERSION, "judge": {"provider": provider, "model": str(data.get("model") or model)},
"asked_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), "state_chars": len(state), "truncated": bool(truncated),
"answers": answers, "usage": {"input_tokens": usage.get("input_tokens"), "output_tokens": usage.get("output_tokens"),
+ "seconds": round(time.monotonic() - started, 2),
**({"cost_usd": float(usage["cost"])} if _number(usage.get("cost")) is not None else {})}}
except (AttributeError, KeyError, TypeError, ValueError) as e: # a reply of another shape: said, never a crash of the batch
raise JudgeError(502, f"the judge's reply could not be read: {e.__class__.__name__}") from e
@@ -521,6 +523,7 @@ def ask_chat(state: str, ids, judge: Judge, truncated: bool = False, post: Optio
"response_format": {"type": "json_object"}, **({"usage": {"include": True}} if judge.provider == "openrouter" else {})}
post = post or http_post
status, data = 0, {}
+ started = time.monotonic()
for attempt in range(retries + 1):
status, data = post(judge.url, body, judge.key, auth=providers.auth("messages", judge.key)) if messages else post(judge.url, body, judge.key)
if status in (429, 502, 503, 529) and attempt < retries:
@@ -563,6 +566,7 @@ def ask_chat(state: str, ids, judge: Judge, truncated: bool = False, post: Optio
rec = {"questions": Q.VERSION, "judge": {"provider": judge.provider, "model": str(data.get("model") or judge.model)},
"asked_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), "state_chars": len(state), "truncated": bool(truncated),
"answers": answers, "usage": {"input_tokens": usage.get(tokens_in), "output_tokens": usage.get(tokens_out),
+ "seconds": round(time.monotonic() - started, 2),
**({"cost_usd": usage["cost"]} if isinstance(usage.get("cost"), (int, float)) else {})}}
if kind is not None:
rec["states"] = {kind: {"chars": len(state), "truncated": bool(truncated), "input_tokens": usage.get(tokens_in),
@@ -589,7 +593,7 @@ def merge(records) -> Optional[dict]:
if rec.get("refusals"):
merged["refusals"] = {**(merged.get("refusals") or {}), **rec["refusals"]}
merged["asked_at"] = rec["asked_at"]; merged["judge"] = rec["judge"]
- for k in ("input_tokens", "output_tokens", "cost_usd"):
+ for k in ("input_tokens", "output_tokens", "cost_usd", "seconds"): # the calls run one after another: their times add up
if rec["usage"].get(k) is not None:
merged["usage"][k] = (merged["usage"].get(k) or 0) + rec["usage"][k]
return merged
diff --git a/app/static/style.css b/app/static/style.css
index 75de2ae6..cc25d710 100644
--- a/app/static/style.css
+++ b/app/static/style.css
@@ -1818,3 +1818,7 @@ select.ui-select-native { display: none !important; }
.tree-kids { box-sizing: border-box; }
@media (prefers-reduced-motion: no-preference) { .shell { transition: grid-template-columns .24s ease; } .tree-caret .ic { transition: transform .24s ease; } }
@media (prefers-reduced-motion: reduce) { :root .slide-disclosure > summary, :root .slide-disclosure > summary::before, .la-fold-btn::before, :root .disclosure-chevron::before, .tree-caret .ic, .side { transition: none; } }
+
+/* the judge's button under its badge, what its last call took beside it */
+.judge-actions { display: flex; align-items: center; gap: .75rem; flex-wrap: wrap; margin: .6rem 0 1rem; }
+.judge-actions .header-form { margin: 0; }
diff --git a/app/templates/traces/_judgement.html b/app/templates/traces/_judgement.html
index 9f06c5e3..417fc43f 100644
--- a/app/templates/traces/_judgement.html
+++ b/app/templates/traces/_judgement.html
@@ -13,9 +13,12 @@
experimental
+
+
{% if j.keyed %}
{% else %}{{ 'LLM judge' if llm else 'System One' }} not configured{% if j.judge is not none and j.judge.missing %}: needs {{ j.judge.missing }}{% endif %} — set it in Settings{% endif %}
+ {% if j.record %}{% set u = j.record.usage or {} %}{% if u.get('seconds') is not none %}Judged in {{ u.seconds|secs }} · {% endif %}{{ j.record.judge.model }}{% if u.get('input_tokens') is not none %} · {{ ((u.input_tokens or 0) + (u.output_tokens or 0))|tokens }} tokens{% endif %}{% if u.get('cost_usd') is not none %} · {{ u.cost_usd|usd }}{% endif %}{% endif %}
{% if j.record and (j.record.refusals or j.earlier or j.record.truncated) %}{% set note = namespace(sep='') %}
{%- if j.record.refusals %}{{ j.record.refusals|length }} refused{% set note.sep = ' · ' %}{% endif -%}
diff --git a/app/tests/test_judge.py b/app/tests/test_judge.py
index 8ca868ad..190aeed8 100644
--- a/app/tests/test_judge.py
+++ b/app/tests/test_judge.py
@@ -132,7 +132,7 @@ def test_the_record_shape_from_the_api_answer(self):
self.assertEqual(rec["answers"]["task_completed"], {"noul": 0.93})
self.assertEqual(rec["answers"]["failure_cause"]["choice"], "rate_limit_or_timeout"); self.assertEqual(rec["answers"]["failure_cause"]["probabilities"]["rate_limit_or_timeout"], 0.81)
self.assertEqual(rec["answers"]["transcript_quality"]["score"], 1.7); self.assertEqual(rec["answers"]["transcript_quality"]["legend"][2], "direct, correct, well explained")
- self.assertNotIn("not_a_question", rec["answers"]); self.assertEqual(rec["usage"], {"input_tokens": 4821, "output_tokens": 96})
+ self.assertNotIn("not_a_question", rec["answers"]); self.assertGreaterEqual(rec["usage"].pop("seconds"), 0); self.assertEqual(rec["usage"], {"input_tokens": 4821, "output_tokens": 96}) # the call timed, its wait included
url, body, key = seen[0]
self.assertEqual(url, client.URL); self.assertEqual(key, KEY); self.assertEqual(body["model"], "jev-latest"); self.assertEqual(body["state"], "the state")
self.assertEqual(sorted(body["questions"]), ["failure_cause", "task_completed", "transcript_quality"])
diff --git a/app/tests/test_judge_openrouter.py b/app/tests/test_judge_openrouter.py
index 11a21031..5c2212c9 100644
--- a/app/tests/test_judge_openrouter.py
+++ b/app/tests/test_judge_openrouter.py
@@ -95,6 +95,7 @@ def test_one_call_asks_every_question_and_reads_each_answer_by_its_type(self):
self.assertEqual(rec["answers"]["failure_cause"]["probabilities"], {"agent_error": 0.75, "environment_error": 0.25}) # normalised
self.assertIsNone(rec["answers"]["failure_cause"]["confidence"]) # stated, not calibrated
self.assertEqual(rec["answers"]["transcript_quality"]["legend"], Q.QUESTIONS["transcript_quality"]["criteria"])
+ self.assertGreaterEqual(rec["usage"].pop("seconds"), 0)
self.assertEqual(rec["usage"], {"input_tokens": 1200, "output_tokens": 80, "cost_usd": 0.0012})
self.assertEqual(rec["states"]["trial"]["input_tokens"], 1200)
self.assertNotIn("sk-or-1", json.dumps(rec))
diff --git a/app/ui_tests/test_trial_head.py b/app/ui_tests/test_trial_head.py
index f522471c..d1bdc793 100644
--- a/app/ui_tests/test_trial_head.py
+++ b/app/ui_tests/test_trial_head.py
@@ -161,6 +161,25 @@ def test_a_refusal_and_a_truncated_state_are_two_notes_and_a_clean_judgement_has
self.assertEqual(self.notes("failed__x"), "1 refused · state truncated (trial)")
self.assertIsNone(self.notes("plain__x"))
+ def test_the_last_call_says_its_time_model_tokens_and_cost_beside_the_button(self):
+ """After a judgement, beside the button: how long the calls took, the model, the tokens and
+ the cost, each only when the record has it."""
+ rec = record("environment_error", 0.9)
+ store.save(self.ws.dir, "r-agent", "failed__x", dict(rec, usage={"input_tokens": 1200, "output_tokens": 34,
+ "seconds": 12.4, "cost_usd": 0.0031}))
+ store.save(self.ws.dir, "r-agent", "plain__x", rec) # an older record: no time, no cost
+
+ def said(trial):
+ response = self.client.get(f"/trial/r-agent/{trial}")
+ self.addCleanup(response.close)
+ panel = response.get_data(as_text=True).split('id="judgement"', 1)[1]
+ row = re.search(r'
(.*?)
', panel, re.S).group(1)
+ self.assertIn("Judge again", row) if "
- {% elif judgement.unjudged %}Decision not configured: needs a model — choose one in Settings{% endif %}
+ {% elif judgement.unjudged %}Decision model not configured: needs a model — choose one in Settings{% endif %}
{% endif %}
diff --git a/app/templates/runs/feasibility.html b/app/templates/runs/feasibility.html
index 3508c1c2..af04743d 100644
--- a/app/templates/runs/feasibility.html
+++ b/app/templates/runs/feasibility.html
@@ -68,7 +68,7 @@
- {% elif triage.unjudged %}Decision not configured: needs a model — choose one in Settings{% endif %}
+ {% elif triage.unjudged %}Decision model not configured: needs a model — choose one in Settings{% endif %}
{% endif %}
diff --git a/app/templates/runs/job.html b/app/templates/runs/job.html
index c240c199..4a6128a0 100644
--- a/app/templates/runs/job.html
+++ b/app/templates/runs/job.html
@@ -298,7 +298,7 @@
{% if (judgement.unjudged) or (judgement.llm.unjudged) or (tq and tq.unjudged) %}
{# one button per judge: each judges apart, so a person asks for the one they want #}
- {% for lane, name, n, keyed in [('decision', 'Decision', judgement.unjudged, judgement.keyed), ('llm', 'LLM judge', judgement.llm.unjudged, judgement.llm.keyed)] if n %}
+ {% for lane, name, n, keyed in [('decision', 'Decision model', judgement.unjudged, judgement.keyed), ('llm', 'LLM judge', judgement.llm.unjudged, judgement.llm.keyed)] if n %}
{% if keyed %}
@@ -222,7 +222,7 @@
Judges
{% if srow and by.get('DECISION_MODEL') %}
- {{ judge_rows('DECISION', decision_providers, decision, 'Where Decision is asked') }}
+ {{ judge_rows('DECISION', decision_providers, decision, 'Where the Decision model is asked') }}
{{ field(by['DECISION_MODEL'], 'Model', writable, decision_providers[decision.provider].models) }}
{% endif %}
diff --git a/app/templates/traces/_judgement.html b/app/templates/traces/_judgement.html
index e3780b0f..0c85e0b4 100644
--- a/app/templates/traces/_judgement.html
+++ b/app/templates/traces/_judgement.html
@@ -15,7 +15,7 @@
{% if j.keyed %}
- {% else %}{{ 'LLM judge' if llm else 'Decision' }} not configured{% if j.judge is not none and j.judge.missing %}: needs {{ j.judge.missing }}{% endif %} — set it in Settings{% endif %}
+ {% else %}{{ 'LLM judge' if llm else 'Decision model' }} not configured{% if j.judge is not none and j.judge.missing %}: needs {{ j.judge.missing }}{% endif %} — set it in Settings{% endif %}
{% if j.record and (j.record.refusals or j.earlier or j.record.truncated) %}{% set note = namespace(sep='') %}
{% endif %}
diff --git a/app/ui_tests/test_routes.py b/app/ui_tests/test_routes.py
index fa0e15f2..3df718f0 100644
--- a/app/ui_tests/test_routes.py
+++ b/app/ui_tests/test_routes.py
@@ -292,7 +292,7 @@ def test_conversation_returns_a_bounded_page_with_full_text_search_reachable(sel
def test_a_trial_has_two_judgements_in_two_tabs(self):
with patch.dict(os.environ, {"TYPESAFE_API_KEY": "", "OPENROUTER_API_KEY": "sk-or-route-test", "LLM_JUDGE_MODEL": ""}):
html = self.get_ok("/trial/active-run/task-001").get_data(as_text=True)
- self.assertIn('id="tab-judgement" role="tab" type="button" data-tab="judgement"', html); self.assertIn(">Decision judge", html)
+ self.assertIn('id="tab-judgement" role="tab" type="button" data-tab="judgement"', html); self.assertIn(">Decision model", html)
self.assertIn('id="tab-llm-judge"', html); self.assertIn(">LLM judge", html)
self.assertIn('id="judgement"', html); self.assertIn('id="llm-judgement"', html)
self.assertIn("LLM judge not configured: needs a model", html) # the OpenRouter key is set, the model is not
diff --git a/app/web/judge.py b/app/web/judge.py
index ac455336..95ab677d 100644
--- a/app/web/judge.py
+++ b/app/web/judge.py
@@ -301,7 +301,7 @@ def judge_job_route(job_id):
k = key(lane)
done, failed = run_batch(todo, lambda t: judge_trial(job, t, k), lambda t: t.trial_name)
from flask import flash
- flash(f"{'LLM judge' if lane == 'llm' else 'Decision'}: judged {done} of {len(todo)} trial(s)." + (" Failed: " + "; ".join(failed[:3]) if failed else ""), "error" if failed else "")
+ flash(f"{'LLM judge' if lane == 'llm' else 'Decision model'}: judged {done} of {len(todo)} trial(s)." + (" Failed: " + "; ".join(failed[:3]) if failed else ""), "error" if failed else "")
return redirect(url_for("job", job_id=job_id) + "#judgement")
diff --git a/docs/judgement.md b/docs/judgement.md
index 7db3a660..b7ef3368 100644
--- a/docs/judgement.md
+++ b/docs/judgement.md
@@ -9,7 +9,7 @@ Two judgements per trial, a tab each, records in `judgements/` and `judgements-l
| [Decision](settings/decision.md) | A decision model's answers: a yes/no's probability, a choice's distribution, a score, each with the model's confidence when it gives one; one call per state |
| [LLM judge](settings/llm-judge.md) | What a chat model states, one JSON object per state, no confidence |
-The job, compare, feasibility and task-instruction pages ask Decision. A record names the provider and the model that answered, and the questions the model refused. The cost estimate for TypeSafe is calibrated on its own stored calls.
+The job, compare, feasibility and task-instruction pages ask the Decision model. A record names the provider and the model that answered, and the questions the model refused. The cost estimate for TypeSafe is calibrated on its own stored calls.
!!! note "A judge never changes an outcome, a count or a verdict"
diff --git a/docs/settings/decision.md b/docs/settings/decision.md
index f8ce9fe0..9c0098d2 100644
--- a/docs/settings/decision.md
+++ b/docs/settings/decision.md
@@ -56,7 +56,7 @@ your existing `settings.json` ([location and precedence](README.md#save)).
## Request a judgement
-A trial's or a job's Decision tab.
+A trial's or a job's Decision model tab.
!!! warning "A judgement is a paid call"
Configuration alone calls nothing; a batch button says how many calls it will make.
From fd4df81252bd1bb8b6719900641c9e715bdedc3c Mon Sep 17 00:00:00 2001
From: Etienne Filhol
Date: Wed, 18 Nov 2026 01:13:10 +0100
Subject: [PATCH 7/7] docs(judge): say a judgement records its calls' time
---
docs/settings/decision.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/docs/settings/decision.md b/docs/settings/decision.md
index 9c0098d2..6948d0b7 100644
--- a/docs/settings/decision.md
+++ b/docs/settings/decision.md
@@ -23,6 +23,7 @@ Decision asks a provider that lists a decision protocol
| --- | --- |
| A question the model refuses | `refusals` in the record, never an answer; the trial page counts them |
| The provider's own cost (`usage.cost`) | `usage.cost_usd` |
+| The calls' time | `usage.seconds`; the judge tab shows it beside the button |
| HTTP 402 | Said as such: the key's credit is spent, or the model is not free |
## Configure