diff --git a/app/nullius/catalogue/benchmarks/osworld1/benchmark.json b/app/nullius/catalogue/benchmarks/osworld1/benchmark.json index c79eacef..c334a8d8 100644 --- a/app/nullius/catalogue/benchmarks/osworld1/benchmark.json +++ b/app/nullius/catalogue/benchmarks/osworld1/benchmark.json @@ -12,7 +12,7 @@ "An agent runs it through the computer port: the agent's tool calls drive OSWorld's own loop, one step for each acting call. A step's observation is the screenshot that loop hands its agent for the next decision, with no capture added; the files a step names are read once its actions and their pauses have run. Steps (100) and Pause (3.0 seconds) set that loop.", "Original runs one of OSWorld's own agents, chosen by the Agent option (M3 by default), with a model it is published with.", "Providers: OpenAI for OpenAI CUA; OpenAI-compatible for Muse Spark (Meta's API) and for EvoCUA, AutoGLM-OS and Mano (your model server); OpenRouter for the others.", - "Only M3, PromptAgent and Qwen-CUA have completed tasks through Nullius. The other agents were checked up to their first model call.", + "At this version only the null control has run through Nullius. OSWorld's agents are the same code as in main (044bc33), where M3, PromptAgent and Qwen-CUA completed tasks.", "Each task's VM takes 4 cores and 4 GB of memory; OS-Symphony can start a second. Run no more tasks at once than the machine holds.", "Prepare the machine before the first run (Docker, KVM, five system libraries, a VM image of about 25 GB): see OSWorld1 machine setup under Benchmarks in the documentation.", "Scores depend on the machine's speed, because OSWorld waits a fixed time after each action. Compare runs made on similar machines.", @@ -37,21 +37,56 @@ "The agent's python and upload calls do not mark OSWorld's desktop as used; each unit's desktop is reset once, before the agent's first call.", "Only 12 of OSWorld's agents are offered; the others fail at the pinned commit or need services the relay cannot reach.", "The VMs run only on Docker's default socket, not on rootless Docker or a Docker context. Podman is untested.", - "A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score." + "A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score.", + "Its evaluator fixes, key repeat turned off in the VM and one retired task make its scores comparable only with this version's, not with the paper's or main (044bc33)'s." ], "datasets": [ { "key": "osworld1", "title": "OSWorld, Ubuntu tasks", - "tasks": 369, + "tasks": 368, "official": "https://github.com/xlang-ai/OSWorld", "source": "GitHub", "registry": "", "hub": "", "verifier": "OSWorld's own evaluator, DesktopEnv.evaluate: the task's evaluator functions on the VM's final state.", - "notes": "369 tasks in 10 domains; 361 without Google Drive. Apache-2.0." + "notes": "368 tasks in 10 domains; 360 without Google Drive. Apache-2.0." } ], "citation": "@misc{OSWorld,\n title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments}, \n author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},\n year={2024},\n eprint={2404.07972},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}", - "citation_source": "https://github.com/xlang-ai/OSWorld" + "citation_source": "https://github.com/xlang-ai/OSWorld", + "versions": { + "044bc33": { + "good_to_know": [ + "An agent runs it through the computer port: the agent's tool calls drive OSWorld's own loop, one step for each acting call. A step's observation is the screenshot that loop hands its agent for the next decision, with no capture added; the files a step names are read once its actions and their pauses have run. Steps (100) and Pause (3.0 seconds) set that loop.", + "Original runs one of OSWorld's own agents, chosen by the Agent option (M3 by default), with a model it is published with.", + "Providers: OpenAI for OpenAI CUA; OpenAI-compatible for Muse Spark (Meta's API) and for EvoCUA, AutoGLM-OS and Mano (your model server); OpenRouter for the others.", + "Only M3, PromptAgent and Qwen-CUA have completed tasks through Nullius. The other agents were checked up to their first model call.", + "Each task's VM takes 4 cores and 4 GB of memory; OS-Symphony can start a second. Run no more tasks at once than the machine holds.", + "Prepare the machine before the first run (Docker, KVM, five system libraries, a VM image of about 25 GB): see OSWorld1 machine setup under Benchmarks in the documentation.", + "Scores depend on the machine's speed, because OSWorld waits a fixed time after each action. Compare runs made on similar machines.", + "OSWorld's infeasible tasks are listed publicly, and its scorer checks only the last action. An agent that looks a task up online and answers FAIL scores it." + ], + "limitations": [ + "An agent needs the computer port, proven by its conformance kit. The agent side has been checked on a fake desktop alone.", + "Preparing the VM image (its lock and its download) is not bounded by a unit's setup time; the core's own limit on setup ends it.", + "A VM start that hangs holds the start gate until its unit's setup time ends; units waiting for the gate fail at the end of theirs.", + "On the local backend an agent runs as you and can write where you can, the runner's folder included.", + "A runner let go at the end of scoring's time that then ends by itself is scored from its complete result.", + "A stop during the runner's last clean-up keeps the result already written.", + "A driver killed outright leaves the runner to stop on its own; a kill during a VM's start can leave that VM.", + "A reaper checks a runner's start time before it ends the runner's group; a process number reused in between is not caught.", + "The agent's python and upload calls do not mark OSWorld's desktop as used; each unit's desktop is reset once, before the agent's first call.", + "Only 12 of OSWorld's agents are offered; the others fail at the pinned commit or need services the relay cannot reach.", + "The VMs run only on Docker's default socket, not on rootless Docker or a Docker context. Podman is untested.", + "A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score." + ], + "datasets": { + "osworld1": { + "tasks": 369, + "notes": "369 tasks in 10 domains; 361 without Google Drive. Apache-2.0." + } + } + } + } } diff --git a/app/nullius/catalogue/benchmarks/osworld1/standard.json b/app/nullius/catalogue/benchmarks/osworld1/standard.json index d2845333..739a0a01 100644 --- a/app/nullius/catalogue/benchmarks/osworld1/standard.json +++ b/app/nullius/catalogue/benchmarks/osworld1/standard.json @@ -245,57 +245,219 @@ } } }, - "proven": { - "test_nogdrive": { - "units": { - "units": 361, - "passed": "361 ids, the same twice, unique", - "on": "2026-11-13" + "default": "desktop-env-v1.0.3", + "versions": { + "desktop-env-v1.0.3": { + "upstream": { + "tag": "desktop-env-v1.0.3", + "tested": "9d6f711d234b84cbff1b5199211610baa03b000c", + "tested_date": "2026-11-19" }, - "nop": { - "passed": "0/8", - "value": 0.0, - "expect": { - "at_most": 0.0 + "datasets": [ + { + "id": "test_nogdrive", + "label": "Without Google Drive", + "summary": "The upstream's test_nogdrive.json: every task but the 8 that need a Google account's credentials. The live MBTA booking task is retired at this release.", + "path": "evaluation_examples/test_nogdrive.json", + "sha256": "9270c11fe809eb4c4e35a78a47cdb50717ea76557c1b9164c183e8e8b8d9ce91", + "units": 360 }, - "units": 8, - "job": "2026-10-29__18-38-14__osworld1__test_nogdrive__prove-nop", - "source": "nullius", - "on": "2026-10-29", - "limit": 8 - } - }, - "test_all": { - "units": { - "units": 369, - "passed": "369 ids, the same twice, unique", - "on": "2026-11-13" - } - }, - "test_infeasible": { - "units": { - "units": 25, - "passed": "25 ids, the same twice, unique", - "on": "2026-11-13" + { + "id": "test_all", + "label": "All tasks", + "summary": "The upstream's test_all.json, its runners' default: the 8 Google Drive tasks need credentials under evaluation_examples/settings, which the upstream does not ship. The live MBTA booking task is retired at this release.", + "path": "evaluation_examples/test_all.json", + "sha256": "47c54203369bd438abf279f6b9836bffc5aa43de6cbaccdfed546993f3fe9149", + "units": 368 + }, + { + "id": "test_small", + "label": "Small", + "summary": "The upstream's test_small.json: a few tasks of each domain.", + "path": "evaluation_examples/test_small.json", + "sha256": "bb29275171b78728d71825b125af311d74e66bb2a6d693c45c497d380b1fc640", + "units": 39 + }, + { + "id": "test_infeasible", + "label": "Infeasible", + "summary": "The upstream's test_infeasible.json: tasks that cannot be done, passed by answering FAIL.", + "path": "evaluation_examples/test_infeasible.json", + "sha256": "1b447d0462b1bdac4b660e0dd6e6add6d01c17a9e684702c701abcffda409abb", + "units": 25 + } + ], + "driver": { + "command": [ + "{install}/.venv/bin/python", + "{here}/driver.py", + "{here}/agents.json", + "{here}/nop.py", + "{here}/runner.py" + ], + "install": { + "python": "3.12", + "project": ".", + "locked": true, + "virtual": true + }, + "participant": true, + "unit_end": true, + "task_given": true + }, + "setups": { + "original": { + "summary": "OSWorld's own agents as published: the agent option's script (scripts/python/run_*.py) on the task, with its own prompts, parser, history and step limit, asking one of the models it is published with, on the docker provider's VM, scored by DesktopEnv.evaluate. At OSWorld's desktop-env 1.0.3 release: its evaluator fixes change the score of some tasks and one task is retired, so its runs compare with each other only, not with the paper's or the earlier pin's.", + "models": [ + "MiniMax-M3", + "gpt-4o", + "claude-sonnet-4-5-20250929", + "claude-opus-4-5", + "claude-opus-4-6", + "claude-opus-4-7", + "gpt-5.4", + "computer-use-preview", + "qwen3.7-plus", + "gpt-5", + "EvoCUA-S2", + "EvoCUA-S1", + "autoglm-os", + "mano", + "muse-spark-1.1" + ], + "default": "MiniMax-M3", + "served_as": { + "MiniMax-M3": "minimax/minimax-m3", + "gpt-4o": "openai/gpt-4o", + "claude-sonnet-4-5-20250929": "anthropic/claude-sonnet-4.5", + "claude-opus-4-5": "anthropic/claude-opus-4.5", + "claude-opus-4-6": "anthropic/claude-opus-4.6", + "claude-opus-4-7": "anthropic/claude-opus-4.7", + "gpt-5.4": "openai/gpt-5.4", + "qwen3.7-plus": "qwen/qwen3.7-plus", + "gpt-5": "openai/gpt-5", + "UI-TARS-1.5-7B": "bytedance/ui-tars-1.5-7b" + }, + "providers": { + "openrouter": "", + "openai": "", + "custom": "" + }, + "turn_seconds": 7200, + "side_models": [ + "UI-TARS-1.5-7B" + ], + "comparable": false + }, + "model": { + "summary": "The same agent with any model in its own model's place: the upstream's code asks for the agent's first published model, with its prompt and action format, and the relay sends the model chosen, every side call included. Its scores compare with other runs of this setup.", + "runs_as": "MiniMax-M3" + } + }, + "proven": { + "test_nogdrive": { + "units": { + "units": 360, + "passed": "360 ids, the same twice, unique", + "on": "2026-11-19" + } + }, + "test_all": { + "units": { + "units": 368, + "passed": "368 ids, the same twice, unique", + "on": "2026-11-19" + } + }, + "test_small": { + "units": { + "units": 39, + "passed": "39 ids, the same twice, unique", + "on": "2026-11-19" + }, + "nop": { + "passed": "0/8", + "value": 0.0, + "expect": { + "at_most": 0.0 + }, + "units": 8, + "job": "2026-11-19__13-12-06__osworld1-desktop-env-v1.0.3__test_small__prove-nop", + "source": "nullius", + "options": { + "max_steps": "default", + "vm_provider": "docker" + }, + "on": "2026-11-19", + "limit": 8 + } + }, + "test_infeasible": { + "units": { + "units": 25, + "passed": "25 ids, the same twice, unique", + "on": "2026-11-19" + } + } } }, - "test_small": { - "units": { - "units": 39, - "passed": "39 ids, the same twice, unique", - "on": "2026-11-13" + "044bc33": { + "upstream": { + "branch": "main" }, - "nop": { - "passed": "0/8", - "value": 0.0, - "expect": { - "at_most": 0.0 + "proven": { + "test_nogdrive": { + "units": { + "units": 361, + "passed": "361 ids, the same twice, unique", + "on": "2026-11-13" + }, + "nop": { + "passed": "0/8", + "value": 0.0, + "expect": { + "at_most": 0.0 + }, + "units": 8, + "job": "2026-10-29__18-38-14__osworld1__test_nogdrive__prove-nop", + "source": "nullius", + "on": "2026-10-29", + "limit": 8 + } + }, + "test_all": { + "units": { + "units": 369, + "passed": "369 ids, the same twice, unique", + "on": "2026-11-13" + } }, - "units": 8, - "job": "2026-11-13__14-08-41__osworld1__test_small__prove-nop", - "source": "nullius", - "on": "2026-11-13", - "limit": 8 + "test_infeasible": { + "units": { + "units": 25, + "passed": "25 ids, the same twice, unique", + "on": "2026-11-13" + } + }, + "test_small": { + "units": { + "units": 39, + "passed": "39 ids, the same twice, unique", + "on": "2026-11-13" + }, + "nop": { + "passed": "0/8", + "value": 0.0, + "expect": { + "at_most": 0.0 + }, + "units": 8, + "job": "2026-11-13__14-08-41__osworld1__test_small__prove-nop", + "source": "nullius", + "on": "2026-11-13", + "limit": 8 + } + } } } } diff --git a/app/nullius/catalogue/catalog.py b/app/nullius/catalogue/catalog.py index 401ee5cc..b3b15343 100644 --- a/app/nullius/catalogue/catalog.py +++ b/app/nullius/catalogue/catalog.py @@ -47,7 +47,7 @@ import json import os -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from nullius import paths from . import storage @@ -121,6 +121,10 @@ class Benchmark: citation_source: str = "" #: Distinct tasks across dataset views, when known; otherwise show their count summary. task_count: int | None = None + #: what reads differently in another version of it on the standard (its ``standard.json`` + #: ``versions``), by version: whole lists (``VERSIONED``) and its datasets' fields + #: (``VERSIONED_DATASET``, by dataset key); the entry itself is the default version + versions: dict = field(default_factory=dict) @property def label(self) -> str: @@ -174,6 +178,11 @@ def route_case(self) -> str: return "multi-route" if "harbor" in usable else "non-harbor" +# What a version of a benchmark may say differently: these lists whole, and these fields of a dataset. +VERSIONED = ("good_to_know", "upstream_limitations", "limitations") +VERSIONED_DATASET = ("tasks", "notes", "verifier") + + @dataclass class Dataset: key: str @@ -215,20 +224,33 @@ def datasets_of(self, b: "Benchmark") -> list: """A benchmark's datasets, as `Dataset`, in the order its entry writes them.""" return [d for d in (self.dataset(k) for k in b.datasets) if d] - def tasks_of(self, b: "Benchmark") -> tuple: + def tasks_of(self, b: "Benchmark", datasets: list | None = None) -> tuple: """Its tasks as a number and a label, for a card: ``("500", "tasks")``, ``("500", "tasks each")`` when its datasets agree, ``("10–89", "tasks")`` when they do not, ``("—", "tasks")`` when none is recorded. - An explicit distinct task count takes precedence over those summaries.""" + An explicit distinct task count takes precedence over those summaries. + ``datasets``: one version's (``in_version``), its own counts.""" if type(b.task_count) is int and b.task_count > 0: return f"{b.task_count:,}", "tasks" - counts = [d.tasks for d in self.datasets_of(b) if d.tasks] + counts = [d.tasks for d in (self.datasets_of(b) if datasets is None else datasets) if d.tasks] if not counts: return "—", "tasks" if len(set(counts)) == 1: return f"{counts[0]:,}", "tasks each" if len(counts) > 1 else "tasks" return f"{min(counts):,}–{max(counts):,}", "tasks" + def in_version(self, b: "Benchmark", version: str) -> tuple: + """(Benchmark, [Dataset]) as one version of it reads: the entry with that + version's lists and dataset fields over it; the entry itself for the default + or a version that says nothing different.""" + own = (b.versions or {}).get(version) or {} + ds = self.datasets_of(b) + if not own: + return b, ds + changed = own.get("datasets") or {} + return (replace(b, **{k: list(own[k]) for k in VERSIONED if k in own}), + [replace(d, **{k: v for k, v in (changed.get(d.key) or {}).items() if k in VERSIONED_DATASET}) for d in ds]) + def files(self, key: str) -> list: """The files in a benchmark's folder, by name: what its page shows it is made of.""" base = self.entry_dir(key) @@ -411,6 +433,7 @@ def validate(cat: Catalog) -> list: for d in b.datasets: if d not in dkeys: bad.append(f"{b.key}: dataset {d!r} is not in the catalogue") + bad.extend(_versions(b)) for d in cat.datasets: # A dataset is shown on its benchmark's page and # nowhere else, so one its benchmark does not list would never be seen. @@ -424,3 +447,30 @@ def validate(cat: Catalog) -> list: if not d.source: bad.append(f"dataset {d.key}: no source") return bad + + +def _versions(b: "Benchmark") -> list: + """A ``versions`` block: by version, only what may differ, each list of strings, + each dataset one of its own with fields that may differ.""" + if not b.versions: + return [] + if not isinstance(b.versions, dict): + return [f"{b.key}: versions is {{version: what reads differently in it}}"] + bad = [] + for v, own in b.versions.items(): + if not isinstance(own, dict) or not own: + bad.append(f"{b.key}: versions.{v} says what reads differently in it") + continue + for k, x in own.items(): + if k in VERSIONED: + if not isinstance(x, list) or not all(isinstance(i, str) and i for i in x): + bad.append(f"{b.key}: versions.{v}.{k} is a list of lines") + elif k == "datasets": + for dk, fields in (x.items() if isinstance(x, dict) else [(None, None)]): + if dk not in b.datasets or not isinstance(fields, dict) or not fields or any(f not in VERSIONED_DATASET for f in fields): + bad.append(f"{b.key}: versions.{v}.datasets is {{one of its datasets: {', '.join(VERSIONED_DATASET)}}}") + elif "tasks" in fields and (type(fields["tasks"]) is not int or fields["tasks"] < 0): + bad.append(f"{b.key}: versions.{v}.datasets.{dk}.tasks is a count") + else: + bad.append(f"{b.key}: versions.{v}.{k} cannot differ between versions (only {', '.join(VERSIONED + ('datasets',))})") + return bad diff --git a/app/nullius/standard/definitions.py b/app/nullius/standard/definitions.py index 96f443c9..ba1d9b65 100644 --- a/app/nullius/standard/definitions.py +++ b/app/nullius/standard/definitions.py @@ -220,15 +220,32 @@ def resolve(d: dict, version: Optional[str] = None) -> Optional[dict]: out[k] = block[k] if isinstance(d.get("options"), dict) and isinstance(block.get("options"), dict): out["options"] = dict(d["options"], **block["options"]) # a version adds an option or restates one - up = dict(d.get("upstream") or {}) - own = block.get("upstream") or {} - if ("repository" in own or "tested" in own) and "tag" not in own: - up.pop("tag", None) # another repository or commit: the shared tag is not its - out["upstream"] = dict(up, **own, version=v) + out["upstream"] = _version_upstream(d, v) out["proven"] = dict(block.get("proven") or {}) if not bench: # a benchmark offers nothing: it needs out["offers"] = [p for p in ports.PORTS if p in out["proven"]] - return dict(out, _version=v, _versions=list(vs), _default=d.get("default"), _ref=f"{d.get('key')}@{v}") + labels = {x: version_label({"upstream": _version_upstream(d, x), "_version": x}) for x in vs if isinstance(vs[x], dict)} + return dict(out, _version=v, _versions=list(vs), _default=d.get("default"), _ref=f"{d.get('key')}@{v}", _labels=labels) + + +def _version_upstream(d: dict, v: str) -> dict: + """One version's upstream: the shared one, its own block over it.""" + up = dict(d.get("upstream") or {}) + own = (d["versions"][v] or {}).get("upstream") or {} + if ("repository" in own or "tested" in own) and "tag" not in own: + up.pop("tag", None) # another repository or commit: the shared tag is not its + return dict(up, **own, version=v) + + +def version_label(d: dict) -> str: + """A version as a reader names it: its release tag, else its branch and short commit + (``main (044bc33)``), else its key.""" + up = d.get("upstream") or {} + if up.get("tag"): + return str(up["tag"]) + if up.get("branch") and up.get("tested"): + return f"{up['branch']} ({str(up['tested'])[:7]})" + return str(d.get("_version") or "") def title(d: dict) -> str: @@ -322,7 +339,9 @@ def _install(d: dict, where: str = "install") -> List[str]: project's resolution keeps them; each a name, or a name pinned), and ``prefetch``, models fetched from the Hugging Face hub at install time so a run needs no network for them; ``locked``, the project's own lock file (``uv.lock``) installed as its - pins, ``pip`` resolved beside them, rather than its loose requirements. + pins, ``pip`` resolved beside them, rather than its loose requirements; + ``virtual``, with ``locked``, those pins without the project itself, as + ``uv sync`` installs a project that declares no build system. A benchmark's driver has the same block, ``driver.install`` (:func:`driver_install`): ``where`` is the name its problems are said under.""" inst = d.get("install") or {} @@ -364,6 +383,10 @@ def _install_block(d: dict, inst: dict) -> List[str]: bad.append("install.locked: true or false") elif inst.get("locked") and "project" not in inst: bad.append("install.locked: the lock file of install.project, which is not set") + if "virtual" in inst and not isinstance(inst["virtual"], bool): + bad.append("install.virtual: true or false") + elif inst.get("virtual") and not inst.get("locked"): + bad.append("install.virtual: the lock's packages without the project, which needs install.locked") for i, x in enumerate(inst.get("indexes") or []): if not isinstance(x, dict) or not _HTTPS.match(str(x.get("url", ""))) or not x.get("packages") \ or any(not _PKG.match(str(p)) for p in x.get("packages") or []): @@ -748,6 +771,8 @@ def _upstream(u) -> List[str]: bad.append("upstream.tag: the release tag the tested commit is, a plain tag name (v1.0.1)") elif "tag" in u and "repository" not in u: bad.append("upstream.tag: a tag of upstream.repository, which is not set") + if "branch" in u and (not tag_ok(u["branch"]) or "tag" in u or "repository" not in u): + bad.append("upstream.branch: the branch an untagged tested commit is on (main), with upstream.repository and no tag") if not u.get("license"): bad.append("upstream.license: say it, or 'unknown'") return bad @@ -1318,7 +1343,7 @@ def setup(bench: dict, which: str, model: str = "", provider: str = "", agent: s AGENTS_FILE = "agents.json" -FROM = {"bundled": "Bundled", "yours": "Yours", "benchmark": "A benchmark's own"} +FROM = {"bundled": "Nullius", "yours": "Yours", "benchmark": "A benchmark's own"} def _driver_agent(d: dict) -> List[str]: diff --git a/app/nullius/standard/install.py b/app/nullius/standard/install.py index c4108856..3bb91c3b 100644 --- a/app/nullius/standard/install.py +++ b/app/nullius/standard/install.py @@ -258,7 +258,8 @@ def _steps(inst: dict, where: str, up: Optional[dict], src: Optional[str], log: _submodules(proj, lock, log) step(["uv", "--directory", proj, "pip", "install", "--quiet", "--python", py, "-r", lock] + list(inst.get("pip") or []) + wheels) - step(["uv", "pip", "install", "--quiet", "--python", py, "--no-deps", "-e", proj + extras]) + if not inst.get("virtual"): # virtual: the lock's packages only, as uv sync leaves a project with no build system + step(["uv", "pip", "install", "--quiet", "--python", py, "--no-deps", "-e", proj + extras]) else: step(["uv", "pip", "install", "--quiet", "--python", py, "-e", proj + extras]) if inst.get("prefetch"): diff --git a/app/nullius/standard/recipe.py b/app/nullius/standard/recipe.py index 56d34b4f..ba289628 100644 --- a/app/nullius/standard/recipe.py +++ b/app/nullius/standard/recipe.py @@ -123,7 +123,8 @@ def _bench_page(b: dict) -> dict: "ref": definitions.ref_of(b) if "_version" in b else b["key"], "version": b["_version"] if versioned else "", # its versions, each by the ref a recipe names it with (the bare key: the default) - "versions": [{"ref": b["key"] if v == b.get("_default") else f"{b['key']}@{v}", "version": v, "default": v == b.get("_default")} + "versions": [{"ref": b["key"] if v == b.get("_default") else f"{b['key']}@{v}", "version": v, "default": v == b.get("_default"), + "label": (b.get("_labels") or {}).get(v) or v} for v in b.get("_versions") or []] if versioned else [], "environments": {n: {"summary": e.get("summary", ""), "local_ok": bool(e.get("local_ok")), "network": e.get("network"), "needs_secrets": [{"name": s["name"], "to": s["to"], "kind": s.get("kind") or "value", @@ -133,7 +134,7 @@ def _bench_page(b: dict) -> dict: "needs": list(b.get("needs") or []), "units": b.get("units") or {"noun": "unit", "nouns": "units"}, "score": (b.get("score") or {}).get("says", ""), "judged": bool((b.get("score") or {}).get("judge")), # the upstream it runs: its repository, the release tag when it has one, the commit that is the authority - "upstream": {k: str((b.get("upstream") or {}).get(k) or "") for k in ("repository", "tag", "tested")}, + "upstream": {k: str((b.get("upstream") or {}).get(k) or "") for k in ("repository", "tag", "branch", "tested")}, "controls": controls, # One Original procedure, with separate reference-participant and paper-model metadata. "setups": [{"key": k, "label": definitions.setup_label(b, k), "summary": x["summary"], @@ -213,6 +214,18 @@ def _other_page(o: dict) -> dict: if isinstance((o.get("proven") or {}).get("fidelity"), dict) else {})} +def _with_versions(b: dict) -> dict: + """A benchmark's page for the builder: its default version's, each other version + carrying its own whole ``page`` (upstream, datasets, proofs, setups, options), so + choosing a version changes everything the steps show, not its name alone.""" + page = _bench_page(b) + for v in page["versions"]: + if not v["default"]: + other = definitions.find(_all()["benchmark"], v["ref"]) + v["page"] = dict(_bench_page(dict(other, label=b.get("label") or other["label"])), versions=[]) if other else None + return page + + def _install_words(o: dict) -> list: """What a definition installs, in a few words each.""" inst, up = o.get("install") or {}, o.get("upstream") or {} @@ -274,7 +287,7 @@ def choices() -> dict: d = _all() # one entry per benchmark, its default version, which lists the others (``versions``); the fits cover every version defaults = [b for b in d["benchmark"] if b.get("_version") == b.get("_default")] - return {"benchmarks": [_bench_page(b) for b in defaults], + return {"benchmarks": [_with_versions(b) for b in defaults], # an example (the reference agent, the example memory) is there to read or copy, never to choose "agents": [_other_page(o) for o in d["agent"] + definitions.benchmark_agents(d["benchmark"]) if not o.get("example")], "capabilities": [_other_page(o) for o in d["capability"] if not o.get("example")], @@ -406,8 +419,9 @@ def add(step, message, shown=False): if b is None: add("benchmark", "Choose a benchmark of the standard.") return out - b = dict(b, label=definitions.title(b)) - recipe = with_own_agents(recipe, b) + # its own agents are the benchmark's, whatever its version: they keep its plain name + base, b = b, dict(b, label=definitions.title(b)) + recipe = with_own_agents(recipe, base) # one dataset or several (``datasets``; a recipe from before names one, ``dataset``): each its own jobs wanted = recipe.get("datasets") if isinstance(recipe.get("datasets"), list) else ([recipe["dataset"]] if recipe.get("dataset") else []) wanted = list(dict.fromkeys(str(x) for x in wanted if isinstance(x, str))) @@ -437,11 +451,11 @@ def add(step, message, shown=False): if ref == "model": add("agent", "Choose Original, then choose the model on the Model step.") continue - own = definitions.own_agent(b, ref) if ":" in ref else None # one of the benchmark's own agents + own = definitions.own_agent(base, ref) if ":" in ref else None # one of the benchmark's own agents if ":" in ref and own is None: other = definitions.find(definitions.benchmark_agents(d["benchmark"]), ref) - add("agent", f"{other['label']} cannot run {b['label']}: " + "; ".join(match.fit(b, other)) + "." if other - else f"Choose an agent for {b['label']}: {ref} is not one of its own.") + add("agent", f"{other['label']} cannot run {base['label']}: " + "; ".join(match.fit(b, other)) + "." if other + else f"Choose an agent for {base['label']}: {ref} is not one of its own.") continue if ref == "original" or own: orig = (b.get("setups") or {}).get("original") or {} @@ -855,7 +869,7 @@ def row(p: dict) -> list: if p["control"]: return [STEP["agent"], "Check", f"{o['label']} ({o['key']}) · {o['source']} · {o['from']}"] if p.get("agent"): - return [STEP["agent"], "Participant", f"{p['agent']['label']} · {b['label']}'s own"] + return [STEP["agent"], "Participant", f"{p['agent']['label']} · {p['agent'].get('benchmark_label') or b['label']}'s own"] if p.get("setup"): original = (b.get("setups") or {}).get("original") or {} label = original.get("reference_agent") or ("No agent · Original" if side == "agent" else "Original " + (_bench_page(b)["kind"] or "capability")) diff --git a/app/static/builder/command-builder-core.js b/app/static/builder/command-builder-core.js index 25fe1b83..7002fa04 100644 --- a/app/static/builder/command-builder-core.js +++ b/app/static/builder/command-builder-core.js @@ -30,8 +30,15 @@ const isStandard = r => !!(r && r.harness === 'standard'); const standard = () => schema.standard || {benchmarks: [], agents: [], capabilities: [], fits: {}, ports: {}}; // A recipe names a benchmark by its key (its default version) or key@version, one of the page's versions. - const standardBench = r => standard().benchmarks.find(b => b.key === r.benchmark || b.ref === r.benchmark + // The entry is the default version's; another version's own page (its upstream, datasets, proofs, setups) is what a + // recipe naming it reads. standardBase is the entry itself, with the list of versions. + const standardBase = r => standard().benchmarks.find(b => b.key === r.benchmark || b.ref === r.benchmark || (b.versions || []).some(v => v.ref === r.benchmark)) || null; + const standardBench = r => { + const b = standardBase(r); + const v = b && (b.versions || []).find(x => x.ref === r.benchmark && !x.default && x.page); + return v ? v.page : b; + }; // Every agent (or capability) a benchmark of the standard could take, each with the reasons it does not fit: none, it fits. // Then the controls it declares (nop, oracle): no agent, no model, always runnable - each with its source // (official, adapter, nullius) and, per dataset, its proof. @@ -148,11 +155,13 @@ } const removeServed = (r, role, at) => setServed(r, role, ((r.served_by || {})[role] || []).filter((_, i) => i !== Number(at))); function createStandard(benchmark = '', other = '') { - const st = standard(), b = st.benchmarks.find(x => x.key === benchmark) || st.benchmarks[0]; + const st = standard(), b = (benchmark && standardBase({benchmark})) || st.benchmarks[0]; const r = {version: 1, harness: 'standard', name: 'standard-experiment', title: 'Standard experiment', benchmark: '', datasets: [], others: [], model_choice: 'any', provider: firstAvailable(providersOf({}), 'openai'), model: bareModel(schema.defaultModel || ''), limit: '', concurrency: '', unsandboxed: false, where: 'local', backend: 'local', options: {}}; - return b ? setStandardBenchmark(r, b.key, other) : r; + // a version's ref (key@version) opens that version + const ref = b && (b.versions || []).some(v => v.ref === benchmark) ? benchmark : b && b.key; + return b ? setStandardBenchmark(r, ref, other) : r; } // Keep only options and values the benchmark accepts. const ownOptions = (b, opts) => Object.fromEntries(Object.entries(opts || {}).filter(([k, v]) => { @@ -161,10 +170,10 @@ function setStandardBenchmark(r, key, other = '') { const b = standardBench({benchmark: key}); if (!b) return r; - const was = standardBench(r); - if (was !== b) { r.model_choice = 'any'; delete r.served_by; } + const was = standardBench(r), same = !!was && was.key === b.key; // another version of it keeps the choices + if (!same) { r.model_choice = 'any'; delete r.served_by; } r.benchmark = key; - const kept = was === b ? datasetsOf(r).filter(id => b.datasets.some(d => d.id === id)) : []; + const kept = same ? datasetsOf(r).filter(id => b.datasets.some(d => d.id === id)) : []; r.datasets = kept.length ? kept : [b.datasets[0].id]; delete r.dataset; const fitting = standardOthers(r).filter(o => !o.why.length); @@ -868,5 +877,5 @@ at: String(d.at || ''), saved: typeof d.saved === 'string' ? d.saved : ''}; } catch (e) { return e && e.message === RUNNER_GONE ? {recipe: null, gone: RUNNER_GONE} : null; } } - return {STANDARD_PHASES,STEPS,STEP,runners,RUNNER_GONE,isStandard,standard,standardBench,standardOthers,ownAgentsOf,ownChosen,runsOf,standardKeyless,paperModels,paperListed,paperOffered,leaderboard,setModelChoice,servedRoles,servedList,setServed,addServed,removeServed,setModelOf,servedModelOf,servedKey,servedCompiled,servedRow,stepShown,chosenOf,datasetsOf,whereList,machineLabel,createStandard,setStandardBenchmark,openingStep,runnersOf,runnerFor,benchKey,whereLabel,create,benchmarks,routeBlocker,dataset,sources,setBenchmark,parameters,scopes,validate,compile,runnable,fromJSON,fromJob,prefill,review,copy,packDraft,unpackDraft,adapterPath,taskCount,PROVIDERS,PROVIDER_LABEL,PROVIDER_NEEDS,available,labelOf,needsOf,choiceOf,setChoice,providersOf,recipeProvider,providerBlocker,providerOf,bareModel,settleModel,agentModel,quote:q,path,defaults,needsCredentials,credentialBindings,declaredTimeouts,effectiveTimeouts,setPhaseMinutes,PHASES,harborExecutable,harborPin,harborProblem,harborKey,harborRepoUrl,harborLabel,repoLabel,agents,mergeAgents,setScanned,COMMON,agentById}; + return {STANDARD_PHASES,STEPS,STEP,runners,RUNNER_GONE,isStandard,standard,standardBase,standardBench,standardOthers,ownAgentsOf,ownChosen,runsOf,standardKeyless,paperModels,paperListed,paperOffered,leaderboard,setModelChoice,servedRoles,servedList,setServed,addServed,removeServed,setModelOf,servedModelOf,servedKey,servedCompiled,servedRow,stepShown,chosenOf,datasetsOf,whereList,machineLabel,createStandard,setStandardBenchmark,openingStep,runnersOf,runnerFor,benchKey,whereLabel,create,benchmarks,routeBlocker,dataset,sources,setBenchmark,parameters,scopes,validate,compile,runnable,fromJSON,fromJob,prefill,review,copy,packDraft,unpackDraft,adapterPath,taskCount,PROVIDERS,PROVIDER_LABEL,PROVIDER_NEEDS,available,labelOf,needsOf,choiceOf,setChoice,providersOf,recipeProvider,providerBlocker,providerOf,bareModel,settleModel,agentModel,quote:q,path,defaults,needsCredentials,credentialBindings,declaredTimeouts,effectiveTimeouts,setPhaseMinutes,PHASES,harborExecutable,harborPin,harborProblem,harborKey,harborRepoUrl,harborLabel,repoLabel,agents,mergeAgents,setScanned,COMMON,agentById}; }); diff --git a/app/static/builder/command-builder.css b/app/static/builder/command-builder.css index 7ab09aba..b54dcbc7 100644 --- a/app/static/builder/command-builder.css +++ b/app/static/builder/command-builder.css @@ -297,7 +297,7 @@ /* the Agent step's grid: the card toolkit's (catalogue.css), fitted to the form */ #nullius-builder .nb-step { border-radius: 0; margin: 0; } /* catalogue.css has a .nb-step of its own */ #nullius-builder #nb-agent-grid { margin-bottom: 20px; } -#nullius-builder #nb-agent-grid .bench-grid { grid-template-columns: repeat(auto-fill, minmax(min(100%, 230px), 1fr)); gap: 12px; } +#nullius-builder #nb-agent-grid .bench-grid { grid-template-columns: repeat(auto-fill, minmax(min(100%, 230px), 1fr)); gap: 12px; grid-auto-rows: 1fr; } #nullius-builder #nb-agent-grid .bench-card { padding: 16px; gap: 10px; } #nullius-builder #nb-agent-grid .bench-card-title { font-size: 15px; } #nullius-builder #nb-agent-grid .std-tag { white-space: nowrap; } diff --git a/app/static/builder/command-builder.js b/app/static/builder/command-builder.js index 12cc62af..d174a40c 100644 --- a/app/static/builder/command-builder.js +++ b/app/static/builder/command-builder.js @@ -205,27 +205,29 @@ // Nullius's own mark for the standard; Harbor's for Harbor const runnerMark = x => x.standard ? '' : ''; - // Two cards: Harbor, and Nullius (the standard) - a benchmark's own scripts run there, as its Original setup. + // A card per runner that runs the benchmark, alone when only one does: Harbor, and Nullius (the standard) - a benchmark's own scripts run there, as its Original setup. const runnerCard = (x, on, label, summary, mark) => tile({attr: 'runner', key: x.key, on, mark, title: label, tip: summary || ''}); const runnerStep = rec => { const has = C.runnersOf(C.benchKey(rec)), list = C.runners().filter(x => x.builtin && has.includes(x.key)); - const cards = list.length > 1 ? `
` + list.map(x => runnerCard(x, x.key === rec.harness, x.label, x.summary, runnerMark(x))).join('') + '
' + const cards = list.length ? `
` + list.map(x => runnerCard(x, x.key === rec.harness, x.label, x.summary, runnerMark(x))).join('') + '
' : ``; return `

What runs it?

` + cards + (rec.harness === 'harbor' ? harborPanel(rec) : ''); }; + // the version a standard recipe names (``key@version``), '' for the default + const versionOf = rec => C.isStandard(rec) && String(rec.benchmark || '').includes('@') ? String(rec.benchmark).split('@')[1] : ''; // Every benchmark of both runners in one list; picking one sets the runner that runs it (Nullius when both do). const benchmarkStep = rec => { - const st = C.standard(), sb = C.standardBench(rec), on = C.benchKey(rec), seen = new Set(), tiles = []; + const st = C.standard(), sb = C.standardBase(rec), on = C.benchKey(rec), seen = new Set(), tiles = []; const add = (key, title, fam, kind, summary) => { if (seen.has(key)) return; seen.add(key); const c = catOf(key); tiles.push({attr: 'bench', key, on: key === on, title: c.title || title, mark: ``, cat: c.category, origin: c.origin, - fam, famLabel: fam === 'agent' ? 'Agents' : 'Capabilities · ' + kind, tip: c.domain || summary || '', meta: tasksOf(key), + fam, famLabel: fam === 'agent' ? 'Agents' : 'Capabilities · ' + kind, tip: c.domain || summary || '', meta: tasksOf(key, key === on ? versionOf(rec) : ''), find: [c.title || title, key, c.category, c.domain, kind].join(' ')}); }; st.benchmarks.forEach(x => add(x.key, x.label, x.family, x.kind, x.summary)); C.benchmarks().forEach(x => add(x.key, x.title, 'agent', '', x.domain)); tiles.sort((a, b) => a.title.localeCompare(b.title, 'en', {sensitivity: 'base'})); const versions = C.isStandard(rec) && sb && (sb.versions || []).length > 1 - ? `` : ''; + ? `` : ''; return `

Choose a benchmark.

` + picker(tiles, [['agent', 'Agents'], ['capability', 'Capabilities']]) + versions; }; // Put the recipe on a runner, keeping the one each runner had: the benchmark follows. @@ -630,9 +632,10 @@ // of tiles, as the Benchmarks page draws them. The filter hides tiles in place (applyPick), so typing keeps focus. const picks = {}, pickOf = () => (picks[step] ||= {q: '', cat: '', fam: ''}); // each step its own search and filters const catOf = key => (schema.benchmarks || []).find(b => b.key === key) || {}; - const tasksOf = key => { const total = catOf(key).task_count; + // its tasks; `version`: a version of the standard's that counts its datasets apart (the catalogue entry's ``versions``) + const tasksOf = (key, version) => { const total = catOf(key).task_count, own = ((catOf(key).versions || {})[version] || {}).datasets || {}; if (Number.isInteger(total) && total > 0) return `${total.toLocaleString('en-US')} tasks`; - const t = schema.datasets.filter(d => d.benchmark === key && d.tasks).map(d => d.tasks); + const t = schema.datasets.filter(d => d.benchmark === key && d.tasks).map(d => (own[d.key] || {}).tasks || d.tasks); return !t.length ? '' : t.every(n => n === t[0]) && t.length > 1 ? `${t[0]} tasks each` : `${t.reduce((a, n) => a + n, 0)} tasks`; }; const pickChip = (attr, v, label, on) => ``; // one tile: {attr, key, on, title, mark, cat, origin, line, meta, foot, find, fam, data (more attributes), tip (its title)} @@ -990,14 +993,20 @@ return i===S.agent&&b?(b.family==='agent'?'Agent':(b.kind||'capability').replace(/^./,c=>c.toUpperCase())):t;} // Next and Back skip a step no recipe shows (Setup, its number kept for saved drafts) const nextShown=(i,dir)=>{let j=i+dir;while(j>S.benchmark&&j{e.scrollTop=t;});if(page)page.scrollTop=y; + } + function render(focus=false,form=true) { const nav=$('.nb-steps'); if(!nav.querySelector('.nb-stepbar'))nav.innerHTML=''; nav.querySelectorAll('.nb-step').forEach(b=>b.remove()); if(!C.stepShown(r(),step))step=nextShown(step,step>S.setup?-1:1); const shown=steps.map((t,i)=>i).filter(i=>C.stepShown(r(),i)); nav.insertAdjacentHTML('afterbegin',shown.map((i,n)=>button(`0${n+1}${stepName(steps[i],i)}`,`data-step="${i}" aria-pressed="${step===i}"`,'nb-step')).join('')); - renderForm();refreshPlan();moveUnderline();saveDraft();harborFollow(); + if(form)renderForm();refreshPlan();moveUnderline();saveDraft();harborFollow(); if(focus){const heading=$('#nb-form h2');if(heading){heading.tabIndex=-1;heading.focus({preventScroll:true});heading.scrollIntoView({block:'nearest'});}} } async function loadSaved(){try{saved=await store.list();}catch(e){status('Saved recipes cannot be read: '+e.message);}if($('#nb-saved').open)renderSaved();} @@ -1169,7 +1178,7 @@ render(d.step!==undefined||['nb-next','nb-back'].includes(b.id)); }); // an agent, a capability or the setup picked or let go on the Agent step's grid; the focus stays on its card - root.addEventListener('nullius-pick',event=>{const k=event.detail.value,rec=r(),c=C.chosenOf(rec);rec.others=c.includes(k)?c.filter(x=>x!==k):[...c,k];delete rec.other;if(rec.options)delete rec.options[k];render();event.target.focus();}); + root.addEventListener('nullius-pick',event=>{const k=event.detail.value,rec=r(),c=C.chosenOf(rec);rec.others=c.includes(k)?c.filter(x=>x!==k):[...c,k];delete rec.other;if(rec.options)delete rec.options[k];inPlace(()=>render());event.target.focus({preventScroll:true});}); root.addEventListener('input',event=>{ const el=event.target,d=el.dataset,rec=r(); if(el.id==='nb-recipe-title'){updateSavePreview();return;} @@ -1209,7 +1218,9 @@ const el=event.target,d=el.dataset; if(d.custom?.endsWith(':type')){$('#nb-parameters').innerHTML=customRows(!['common','extra'].includes(tab)?tab:'');} else if(d.stdUnsandboxed!==undefined){r().unsandboxed=el.checked;refreshPlan();} - else if(d.stdVersion!==undefined){C.setStandardBenchmark(r(),el.value);render();} + else if(d.stdVersion!==undefined){ // in place: only the card's count changes on this step, so nothing else is drawn again + C.setStandardBenchmark(r(),el.value);const key=C.benchKey(r()),meta=root.querySelector(`[data-bench="${CSS.escape(key)}"] .nb-tile-foot > span`); + if(meta)meta.textContent=tasksOf(key,versionOf(r()));render(false,false);} // an option at its default is left out: the recipe names only what was changed. A select's value can show or hide // the options that apply under it (`when`): the step is drawn again and the focus goes back to the select, or to the // menu button controls.js puts after it once this change is handled; a typed value is left as it is diff --git a/app/static/catalogue/catalogue.js b/app/static/catalogue/catalogue.js index b240a60a..4f46121c 100644 --- a/app/static/catalogue/catalogue.js +++ b/app/static/catalogue/catalogue.js @@ -57,3 +57,22 @@ document.querySelectorAll('.bench-page .std-copy').forEach(b => b.addEventListen try { await navigator.clipboard.writeText(text); b.textContent = 'Copied'; } catch { b.textContent = 'Select it'; } setTimeout(() => { b.textContent = 'Copy'; }, 1500); })); + +// A benchmark's versions: every version is on the page, one shown; a version link swaps them in place and keeps ?v= +// in the address, so the link still opens that version on its own. +document.querySelectorAll('.bench-page .std-versions').forEach(nav => nav.addEventListener('click', event => { + const link = event.target.closest('a'); + if (!link || event.defaultPrevented || event.button !== 0 || + event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) return; + const v = new URL(link.href).searchParams.get('v'); + event.preventDefault(); + document.querySelectorAll('.bench-page .std-version[data-version]').forEach(el => { el.hidden = el.dataset.version !== v; }); + nav.querySelectorAll('a').forEach(a => { + if (a === link) a.setAttribute('aria-current', 'page'); else a.removeAttribute('aria-current'); + }); + const build = document.querySelector('.bench-page [data-build-run]'); // Build a run opens the version shown + if (build) { const to = new URL(build.href); to.searchParams.set('standard', link.dataset.ref); build.href = to.pathname + to.search; } + const url = new URL(location.href); + url.searchParams.set('v', v); + history.replaceState(null, '', url.pathname + url.search + location.hash); +})); diff --git a/app/templates/builder/agents.html b/app/templates/builder/agents.html index 524d5033..6ef60638 100644 --- a/app/templates/builder/agents.html +++ b/app/templates/builder/agents.html @@ -7,7 +7,7 @@ off until asked; Fit a menu of its own, on Can run it at first -#} {% set cols = [dict(key='icon', label='Logo', card='icon', table=false, class='std-mark small'), dict(key='title', label='Name', card='title', width='expand'), dict(key='alert', label='Why not', card='alert'), dict(key='protocol', label='Protocol', card='tag', class='std-tag'), - dict(key='published', label='Setup', card='tag', class='std-tag'), dict(key='needs', label='Proven', card='badge', class='std-tag'), + dict(key='published', label='Setup', card='tag', class='std-tag'), dict(key='ticks', label='Checks', card='badge', class='nb-tick'), dict(key='fit', label='Fit', filter='dropdown', filter_icon='puzzle', off=true), dict(key='from', label='From', card='meta'), dict(key='proves', label='Proves', class='std-tag', off=true)] %} {{ cards('nb-agents', cols, items, search='Search ' ~ noun, choices=choices, empty='No ' ~ kind ~ ' matches.', noun=noun, layout=layout, default='cards', only={'fit': 'fit'}) }} diff --git a/app/templates/catalogue/benchmark.html b/app/templates/catalogue/benchmark.html index 4a13f240..ffbf2877 100644 --- a/app/templates/catalogue/benchmark.html +++ b/app/templates/catalogue/benchmark.html @@ -5,13 +5,13 @@ {% block crumb %}/ benchmarks / {{ b.key }}{% endblock %} {% block head %}{% endblock %} {% block body %} -{% set ds = cat.datasets_of(b) %}{% set tasks = cat.tasks_of(b) %}{% set files = cat.files(b.key) %} +{% set ds = cat.datasets_of(b) %}{% set files = cat.files(b.key) %} {% set ways = (['the Nullius standard'] if 'standard' in b.harness else []) + (['Harbor'] if 'harbor' in b.harness else []) %}{% set runs_with = ways|join(' or ') if ways else 'no harness here' %}
{% set mark %}{% endset %} {% set tags %}{% if b.category %}{{ b.category }}{% endif %}{{ b.label }}{% endset %} {% set actions %} - {% if b.usable_routes %}Build a run {% endif %} + {% if b.usable_routes %}Build a run {% endif %} {% if b.citation %}Cite{% endif %} {% if editable %}Edit{% endif %} {#- deleting asks first, in place: the button opens the question, never a dialog #} @@ -32,6 +32,14 @@
+ {% if versions %} + {#- Each version is the whole page once: its pin, proofs, setup, options, tasks and limitations. A click swaps them in place. -#} + + {% endif %} + {%- macro vattr(y, cls='') %} class="{{ (cls ~ ' std-version')|trim }}"{% if y.version %} data-version="{{ y.version }}"{% endif %}{% if not y.current %} hidden{% endif %}{% endmacro %} {% if std %} {#- The standard's proofs of it, per dataset (standard/prove.py): its controls as chips, their source in the title - the @@ -48,35 +56,19 @@ {% macro check(name, says, rec, failed) -%} {{ name }}{% if rec %}✓{% else %}{{ proof(rec, failed) }}{% endif %} {%- endmacro %} - {%- if std.setups %} -
-

Setup

- {% set x = std.setups[0] %} -
-

Original

-
Model Choose in the run builder
-
-
How it runs -
-

{{ x.summary }}

- {{ std.ref }}:original -
-
-
- {%- endif %} -
-

Proofs

- {% if std.upstream.repository %}

{{ std.upstream.repository|replace('https://github.com/', '') }} @ {% if std.upstream.tag %}{{ std.upstream.tag }} · {% endif %}{{ std.upstream.tested[:7] }}

{% endif %} + {#- One version's proofs: its pin, each dataset's checks, and the form that runs them on it. -#} + {% macro proofs(s) -%} + {% if s.upstream.repository %}

{{ s.upstream.repository|replace('https://github.com/', '') }} @ {% if s.upstream.tag %}{{ s.upstream.tag }} · {% elif s.upstream.branch %}{{ s.upstream.branch }} · {% endif %}{{ s.upstream.tested[:7] }}

{% endif %}

Null and reference controls check the scorer; stability and leak checks validate the tasks.

- {% for d in std.datasets %} + {% for d in s.datasets %} - + @@ -89,33 +81,61 @@

Proofs

Run control checks + {% if versions %}{% endif %}

Check dataset stability, answer leaks and control scores. Results update the recorded proofs. - {% if std.judged %}The judge needs a model key.{% else %}No model calls or key required.{% endif %}

+ {% if s.judged %}The judge needs a model key.{% else %}No model calls or key required.{% endif %}

- {% if std.datasets|length == 1 %} - -
Dataset{{ std.datasets[0].label }}
+ {% if s.datasets|length == 1 %} + +
Dataset{{ s.datasets[0].label }}
{% else %} - + {% endif %} - - {% if std.judged %} + + {% if s.judged %} {% if providers %}{% endif %} {% endif %}
- {% if std.judged and not providers %}

The judge needs a model key first: set one in Settings.

+ {% if s.judged and not providers %}

The judge needs a model key first: set one in Settings.

{% else %}{% endif %} {% endif %} + {%- endmacro %} + {%- if std.setups %} +
+

Setup

+
+

Original

+
Model Choose in the run builder
+
+
How it runs + {% for y in views %} + +

{{ y.std.setups[0].summary }}

+ {{ y.std.ref }}:original + + {% endfor %} +
+
+ {%- endif %} +
+

Proofs

+ {% for x in views %} + + {{ proofs(x.std) }} + + {% endfor %}
{% if std.options %} {#- Its own options (a search, a mode): each value, a line on it, what it needs; the machine setup some need. -#}

Options

- {% for n, o in std.options.items() %} -

{{ o.label }}{% if o.datasets %} {{ std.datasets|selectattr('id', 'in', o.datasets)|map(attribute='label')|join(', ') }}{% endif %}

+ {% for y in views %}{% set sfx = '' if y.default else '-' ~ y.version %} + + {% for n, o in y.std.options.items() %} +

{{ o.label }}{% if o.datasets %} {{ y.std.datasets|selectattr('id', 'in', o.datasets)|map(attribute='label')|join(', ') }}{% endif %}

{{ o.summary }}

{% if o.pattern %}
{{ 'Provider default' if o.default == 'default' and not o.limit else o.default }} @@ -126,19 +146,21 @@

{{ o.label }}{% if o.datasets %} {{ std.datasets| {% for v in o['values'] %}{% set need = o.needs.get(v) or {} %}
  • {% if o.names.get(v) %}{{ o.names[v] }}{% else %}{{ v }}{% endif %}{% if v == o.default and v != 'default' %} default{% endif %} {% if need.embeddings %}OpenAI key · {{ need.embeddings }}{% endif %} - {% for h in need.host or [] %}needs {{ std.host[h].label }}{% endfor %} + {% for h in need.host or [] %}needs {{ y.std.host[h].label }}{% endfor %} {% if o.about.get(v) %}{{ o.about[v] }}{% endif %}
  • {% endfor %} {% endif %} {% endfor %} - {% for h, r in std.host.items() %} -
    + {% for h, r in y.std.host.items() %} +

    Setup: {{ r.label }}

    {{ r.summary }} Checked by {{ r.check|join(', ') }} on PATH, on each machine a run uses; the builder's Where step checks it{% if r.install %} and can install it{% endif %}.

    {% if r.install %}
    install, on each machine
    {{ r.install|join('\n') }}
    {% endif %}
    {% endfor %} +
    + {% endfor %}

    {% endif %} @@ -154,18 +176,20 @@

    {{ 'Datasets' if ds|length > 1 else 'Dataset' }}{% if ds|lengt
    1 %} role="tabpanel" aria-labelledby="tab-{{ d.key }}"{% endif %}>

    {{ d.title }}

    -
    Tasks{{ '{:,}'.format(d.tasks) if d.tasks else '—' }}
    +
    Tasks{% for y in views %}{% set e = y.ds|selectattr('key', 'equalto', d.key)|first %}{{ '{:,}'.format(e.tasks) if e.tasks else '—' }}{% endfor %}
    {% if not data[d.key] %}
    Source{{ d.source }}
    {% endif %} {% if d.official %}{% endif %}
    - {% if d.notes %}

    {{ d.notes }}

    {% endif %} + {% for y in views %}{% set e = y.ds|selectattr('key', 'equalto', d.key)|first %} + + {% if e.notes %}

    {{ e.notes }}

    {% endif %}
    - {% if d.verifier %}
    Verifier
    {{ d.verifier }}
    {% endif %} + {% if e.verifier %}
    Verifier
    {{ e.verifier }}
    {% endif %} {#- One row per way it runs: where that harness reads the data, at which pin. The harness fetches it when a run starts; nobody downloads anything by hand. #} - {% if data[d.key] %}
    Data
    + {% if y.data[d.key] %}
    Data
      - {% for r in data[d.key] %} + {% for r in y.data[d.key] %}
    • {{ r.label }} {{ r.from }} @@ -177,33 +201,39 @@

      {{ d.title }}

    {% endif %}
    + + {% endfor %}
    {% endfor %} {% endif %} - {% if b.good_to_know %} -
    -

    Good to know

    -
      {% for note in b.good_to_know %}
    • {{ note }}
    • {% endfor %}
    + {% for y in views %}{% set sfx = '' if y.default else '-' ~ y.version %} + + {% if y.b.good_to_know %} +
    +

    Good to know

    +
      {% for note in y.b.good_to_know %}
    • {{ note }}
    • {% endfor %}
    {% endif %} - {% if b.upstream_limitations or b.limitations %} + {% if y.b.upstream_limitations or y.b.limitations %}
    - {% if b.upstream_limitations %} -
    -

    Upstream limitations

    -
      {% for l in b.upstream_limitations %}
    • {{ l }}
    • {% endfor %}
    + {% if y.b.upstream_limitations %} +
    +

    Upstream limitations

    +
      {% for l in y.b.upstream_limitations %}
    • {{ l }}
    • {% endfor %}
    {% endif %} - {% if b.limitations %} -
    -

    Nullius limitations

    -
      {% for l in b.limitations %}
    • {{ l }}
    • {% endfor %}
    + {% if y.b.limitations %} +
    +

    Nullius limitations

    +
      {% for l in y.b.limitations %}
    • {{ l }}
    • {% endfor %}
    {% endif %}
    {% endif %} + + {% endfor %} {% if b.citation %}
    @@ -250,7 +280,7 @@

    As published

    - + diff --git a/docs/benchmarks/supported/osworld1.md b/docs/benchmarks/supported/osworld1.md index 68a3f870..71b7f35f 100644 --- a/docs/benchmarks/supported/osworld1.md +++ b/docs/benchmarks/supported/osworld1.md @@ -10,7 +10,7 @@ Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of - +
    {{ d.label }}{% if d.units %}{{ d.units }} {{ std.units.nouns }}{% endif %}{{ d.label }}{% if d.units %}{{ d.units }} {{ s.units.nouns }}{% endif %}
    {{ check('stable', 'the same ids twice, each unique', d.proofs.units, (d.failed or {}).units) }} - {% if std.controls|selectattr('key', 'equalto', 'oracle')|list %}{{ check('no leak', 'nothing of the reference shows before the run', d.proofs.leak, (d.failed or {}).leak) }}{% endif %} + {% if s.controls|selectattr('key', 'equalto', 'oracle')|list %}{{ check('no leak', 'nothing of the reference shows before the run', d.proofs.leak, (d.failed or {}).leak) }}{% endif %} {% for c in d.controls %}{{ chip(c) }}{% endfor %}
    {%- for c in d.controls if c.proven and c.proven.job in std_jobs %}{{ c.key }}'s job{% endfor %}
    LoCoMo
    locomo
    Very long-term conversational memory over conversations of 19 to 32 sessions.Memory✓–10Official
    LongMemEval
    longmemeval
    Long-term chat memory: questions over timestamped chat histories.Memory✓–1500Official
    Open Telco
    open-telco
    GSMA's eight telecom tests, from 3GPP standards and maths to 5G root-cause analysis and 6G reasoning.Telecom✓–20588Official
    OSWorld1
    osworld1
    Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of its own.Computer use✓–369Official
    OSWorld1
    osworld1
    Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of its own.Computer use✓–368Official
    OSWorld2
    osworld2
    Long-horizon, real-world tasks on an Ubuntu desktop, across its apps, each in a VM of its own.Computer use✓–108Official
    SWE-bench Verified
    swe-bench-verified
    Software engineering: fix a real GitHub issue so the failing tests pass.Coding✓✓500Official
    τ³-bench
    tau3-bench
    Customer service with domain tools and a simulated user: airline, retail, telecom, banking.Customer service✓✓375Official
    CategoryComputer use
    OriginOfficial
    Runs onNullius
    Tasks369
    Tasks368
    PaperOSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, arXiv · 2024
    Needs, ports provenagent.computer/v1
    @@ -19,7 +19,7 @@ Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of | Dataset | Key | Tasks | From | Verifier | |---|---|---|---|---| -| OSWorld, Ubuntu tasks | `osworld1` | 369 | GitHub [xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld/tree/044bc33901cb893808e3f91ebc9a17ed8e54b44c) | OSWorld's own evaluator, DesktopEnv.evaluate: the task's evaluator functions on the VM's final state. | +| OSWorld, Ubuntu tasks | `osworld1` | 368 | GitHub [xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld/tree/9d6f711d234b84cbff1b5199211610baa03b000c) | OSWorld's own evaluator, DesktopEnv.evaluate: the task's evaluator functions on the VM's final state. | ## On the standard @@ -31,7 +31,10 @@ Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of | Version | Setups as published, whose model | Options | |---|---|---| -| `osworld1` | Original · its own agents · Like the paper or Any of your choice | `agent`, `max_steps`, `pause`, `thinking`, `vm_provider` | +| `osworld1@desktop-env-v1.0.3` (default) | Original · its own agents · Like the paper or Any of your choice | `agent`, `max_steps`, `pause`, `thinking`, `vm_provider` | +| `osworld1@044bc33` | Original · its own agents · Like the paper or Any of your choice | `agent`, `max_steps`, `pause`, `thinking`, `vm_provider` | + +- `key@version`: a revision of the definition and driver; a recipe pins one, the builder offers the default. ## Options @@ -52,7 +55,7 @@ Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of - An agent runs it through the computer port: the agent's tool calls drive OSWorld's own loop, one step for each acting call. A step's observation is the screenshot that loop hands its agent for the next decision, with no capture added; the files a step names are read once its actions and their pauses have run. Steps (100) and Pause (3.0 seconds) set that loop. - Original runs one of OSWorld's own agents, chosen by the Agent option (M3 by default), with a model it is published with. - Providers: OpenAI for OpenAI CUA; OpenAI-compatible for Muse Spark (Meta's API) and for EvoCUA, AutoGLM-OS and Mano (your model server); OpenRouter for the others. -- Only M3, PromptAgent and Qwen-CUA have completed tasks through Nullius. The other agents were checked up to their first model call. +- At this version only the null control has run through Nullius. OSWorld's agents are the same code as in main (044bc33), where M3, PromptAgent and Qwen-CUA completed tasks. - Each task's VM takes 4 cores and 4 GB of memory; OS-Symphony can start a second. Run no more tasks at once than the machine holds. - Prepare the machine before the first run (Docker, KVM, five system libraries, a VM image of about 25 GB): see OSWorld1 machine setup under Benchmarks in the documentation. - Scores depend on the machine's speed, because OSWorld waits a fixed time after each action. Compare runs made on similar machines. @@ -80,6 +83,7 @@ Open-ended tasks on a real Ubuntu desktop and its applications, each in a VM of - Only 12 of OSWorld's agents are offered; the others fail at the pinned commit or need services the relay cannot reach. - The VMs run only on Docker's default socket, not on rootless Docker or a Docker context. Podman is untested. - A task run by one of OSWorld's own agents and still running after 2 hours is stopped and gets no score. For an agent, the task's setup may take 1 hour, the agent's turn 2 hours and the scoring 10 minutes; past any of them the task gets no score. +- Its evaluator fixes, key repeat turned off in the VM and one retired task make its scores comparable only with this version's, not with the paper's or main (044bc33)'s. ## Cite diff --git a/docs/standard.md b/docs/standard.md index e552f315..a2723b47 100644 --- a/docs/standard.md +++ b/docs/standard.md @@ -484,6 +484,7 @@ The upstream's dependencies go in `driver.install` (a capability's `install` is | `indexes` | Packages from another index | | `wheels` | A wheel served by URL only (a model package such as spaCy's `en_core_web_sm`), `{url, sha256}`: refused unless the file matches, resolved with `pip` | | `project` | The upstream itself, from its clone; `locked: true` installs it at its own lock file's pins (`uv export --frozen`), `pip` resolved beside them | +| `virtual` | With `locked`: the lock's pins without the project itself, as `uv sync` leaves a project with no build system; the driver imports the upstream from its clone | | `extras`, `prefetch` | The project's extras; what to download ahead | #### Benchmark options @@ -539,6 +540,8 @@ An agent or a capability may keep several versions in its one file: an older rel | Naming | `KEY@VERSION` in `cli run`, `cli prove`, `cli fit` and the recipe's `benchmark` | | In jobs | `…__KEY-VERSION__…` in the name; `standard.benchmark_version` and `standard.benchmark_ref` in `config.json` | | A fork is a version | `versions..upstream = {repository, tag, tested}`, never a patch of ours; cloned at its commit (`upstreams///`), its datasets checked by their own `sha256`; a private repository uses the machine's git credentials. It proves its own controls (`cli prove KEY@NAME`) | +| A release is the default | An earlier untagged pin stays as a version with `upstream.branch` (`main`): shown as `main (044bc33)` | +| Its page | `benchmark.json` `versions.`: whole `good_to_know`, `upstream_limitations`, `limitations`, and `datasets.` `tasks`, `notes`, `verifier`; the page and builder switch all of it with the version | ### Options