diff --git a/CHANGELOG.md b/CHANGELOG.md index c1c36ee..2d2de59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ semantics have been stable since 1.0.0. ## [Unreleased] ### Changed — documentation +- **README opens with the result.** The opening now states the problem, the recorded httpx + catch (a retrospective reproduction), links to the engineering evidence with the benchmark + labelled historical (last rerun at v1.2.0), and the checker boundary, before the runnable + "Try it" recipe, which is unchanged. The raster portrait (`docs/assets/dorian-hero*.png`) is + retired for a generated evidence-card hero, `docs/assets/hero-{light,dark}.svg`, produced by the + standard-library script `docs/assets/src/make_figures.py`; `tests/test_figures.py` runs its + `--check`, which also confirms the card's strings still appear in the recipe and its test. The + Evidence section adds the benchmark's misses next to its false-alarm count and describes the + Loop Guard dogfood test; "Commands at a glance" is condensed around the command reference; the + repeated tagline is gone. The demo-link guard in `tests/test_readme_example.py` now accepts the + opening's Markdown link as well as a badge. - **README onboarding and claim wording.** "Try it in 30 seconds" is now "Try it": the recipe starts in a fresh virtual environment (`python -m pip install dorian-vwp==1.4.0`) and commits with a command-local Git identity, so it needs no prior install and no global Git configuration; the diff --git a/README.md b/README.md index 9fc5f86..520b8c7 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,33 @@ -
- - - - - dorian — hold AI agents to what they said they did - - # dorian -**Hold AI agents to what they said they did.** +**Keep code claims checkable after the next refactor.** -

- Quickstart - Demo - GitHub Action -

+Turn explicit claims into checks sealed beside your code. +Re-run affected checks when their sources change; revoke the warrant when a load-bearing check fails. -

- CI - Python 3.11+ - Apache-2.0 - Latest release -

+**Recorded example:** httpx raised its Python requirement from `>=3.8` to `>=3.9`. +A previously sealed claim became **REVOKED (exit 4)**. +[Captured output + pinned commits](docs/REAL_CATCH_LOG.md) — retrospective reproduction, not deployment evidence. -
+**Engineering:** [Executable README demo](tests/test_readme_example.py) · [Own-repo warrants](docs/changes/dorian-loop-guard.md.warrant) +[240-pair synthetic benchmark](docs/BENCHMARK_v0.7.0.md) · [Release history](CHANGELOG.md) +Benchmark results are historical; the last recorded rerun was at v1.2.0. -`dorian` turns explicit claims about a code change into executable checks and stores their results -in a `.warrant` beside the code. Later revalidation reruns affected checks and revokes warrants when -those checks fail. Verification makes no model calls (**zero model tokens at check time**); its -strength depends on the claims, checkers and trusted execution environment. Its question is: -**"Does the code still satisfy this recorded claim?"** + + + dorian evidence card from the Try it recipe: the claim 'handler() lives in app.py'; dorian verify reports verified 1/1 claim(s), WARRANTED (exit 0); a refactor renames handler() and the note never changes; dorian revalidate reports handler-exists BROKEN, REVOKED (exit 4). + -The agent (or you) writes a `claims.json` of checkable claims about a change — "`handler()` lives in -`app.py`", "the login timeout is 30 seconds", "`test_login_ratelimit` passes". `dorian verify` runs -each claim's checker against the code and seals the results in a git-committed `.warrant` sidecar. On -each later commit, `dorian revalidate` re-runs only the checks whose watched files changed and flips -the warrant to **REVOKED** when a load-bearing claim's check fails, naming the claim. It ships as a -CLI, a GitHub Action, and Claude Code hooks, with **zero runtime dependencies**. +**Boundary:** a claim is only as strong as its checker. +A symbol-existence check can miss changed behavior. Test and shell checkers execute code. +[Trust and execution limits](docs/SECURITY_BOUNDARY.md) +[Try the runnable demo](#try-it) · [Install](#install) · [GitHub Action](action/README.md) ## Try it -A self-contained run in a throwaway directory, with its own virtual environment and a command-local -Git identity, so it needs no prior install and no global Git configuration. The demo repository and virtual environment stay in the temporary directory. (A black-box test pins the commands and their exit codes.) +The example below seals a claim, renames its function and reports REVOKED. +It uses a throwaway repository and an isolated environment; Python 3.11+ is required. ```bash tmp=$(mktemp -d) @@ -75,10 +60,15 @@ dorian revalidate --since HEAD # -> handler-exists BROKEN; WARRA ``` `note.md` never changed, but the warrant flips to REVOKED and names the claim whose check failed. -`python3` must be 3.11 or newer. For other ways to install, see [Install](#install). +[`tests/test_readme_example.py`](tests/test_readme_example.py) runs the same steps, with the recipe's +command-local Git identity, against this checkout and pins the commands, exit codes and install +version; it does not install from PyPI. For other ways to install, see [Install](#install). ## What it checks and what it does not +Verification makes no model calls (**zero model tokens at check time**); its strength depends on +the claims, checkers and trusted execution environment. + A claim is bound to one of four checker families: **C1** (a quoted span of the artifact itself), **C3** (a path, symbol, string, or regex in a file, plus the structural `py-signature:` / `py-const:` / `code:` / `config-value:` forms), **C4** (a `pytest:` node id), or **C5** (typed data @@ -116,6 +106,9 @@ a sandbox. See [SECURITY.md](SECURITY.md) and [`docs/SECURITY_BOUNDARY.md`](docs ## Install +[![CI](https://github.com/ajaysurya1221/dorian/actions/workflows/ci.yml/badge.svg)](https://github.com/ajaysurya1221/dorian/actions/workflows/ci.yml) [![Latest release](https://img.shields.io/github/v/release/ajaysurya1221/dorian?label=release&color=blue)](https://github.com/ajaysurya1221/dorian/releases/latest) +![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue) ![Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-green) + The PyPI distribution is `dorian-vwp`; the import and CLI are `dorian`. Python 3.11+, zero runtime dependencies, trusted publishing (latest: **`v1.4.0`**). @@ -184,28 +177,22 @@ install` adds a `SubagentStop` gate and a fail-closed `PreToolUse` veto for unat ## Commands at a glance -- `dorian verify --claims claims.json` — run every checker and seal the `.warrant` - (born-verifiable). `--supersede ` re-seals over an earlier warrant, `--no-quotes` writes a - content-free sidecar, `--allow-restricted` overrides the `[tool.dorian.scopes]` seal-time lint. -- `dorian revalidate --since --format md` — re-check only the claims whose watched files - changed; `md` is the PR-comment body the Action posts. -- `dorian status ` · `dorian blast ` — trust state; downstream warrants, which - are flagged `recalled` when a claim they build on breaks. -- `dorian bindings ` · `dorian bind-suggest --claims claims.json` · `dorian rebind` — - binding diagnostics, a preview of the files `verify` would auto-bind, re-derived watches. -- `dorian suggest-claims ` · `dorian suggest-data-checks ` — born-verifiable - claim and C5 checker suggestions to paste into `claims.json`. -- `dorian report --audit` — the event log as byte-identical JSONL. -- `dorian bench mutation` · `bench large-mutation` · `bench binding-lifecycle` · `bench public-repos` - — the benchmark suites; they need a source checkout with development dependencies (the wheel does - not ship `bench/`). +`dorian verify --claims claims.json` seals a warrant (`--supersede ` re-seals over +an earlier one, `--no-quotes` writes a content-free sidecar, `--allow-restricted` overrides the +`[tool.dorian.scopes]` seal-time lint). `dorian revalidate --since --format md` re-checks the +affected claims; `md` is the PR-comment body the Action posts. Also: `dorian status`, +`dorian blast` (downstream warrants are flagged `recalled`), `dorian bindings`, +`dorian suggest-claims` / `suggest-data-checks`, `dorian report --audit`, and the +`dorian bench mutation` / `bench large-mutation` suites (source checkout only). Full reference: +[`docs/COMMANDS.md`](docs/COMMANDS.md). Exit codes: `0` ok/TRUSTED · `2` usage/infra · `3` DEGRADED · `4` REVOKED/integrity · `5` ERRORED-only (checkers could not run; never conflated with broken) · `6` scope violation. -Full reference: [`docs/COMMANDS.md`](docs/COMMANDS.md). ## Evidence +On the historical 240-pair synthetic suite, Dorian produced 5 false alarms versus the path-scope watcher's 58, while missing 5 stale pairs that the watcher detected. These results are specific to the authored fixtures. + - **Synthetic benchmark.** Over 240 (artifact, mutation) pairs across six invented fixture domains with known-truth labels, claim-level revalidation flagged broken claims at precision **0.93** / recall **0.93**, versus file-change watchers at recall 1.00 but precision **0.34** (naive), @@ -223,6 +210,12 @@ Full reference: [`docs/COMMANDS.md`](docs/COMMANDS.md). warrant WARRANTED → REVOKED (exit 4). Full output and a from-scratch reproduction: [`docs/REAL_CATCH_LOG.md`](docs/REAL_CATCH_LOG.md). One documented case, not universal validation. +**Keep a change note checkable across later edits** + +Dorian commits warrants for its own change notes. [The Loop Guard warrant](docs/changes/dorian-loop-guard.md.warrant) includes checks against implementation symbols and selected README text. [Its dogfood test](tests/test_loop_guard_dogfood.py) copies those real files into a temporary repository, seals the claims, then renames a referenced function and checks that the loop decision changes from continue to repair. + +This exercises specific authored claims. It does not certify the entire README or establish a current TRUSTED result for every warrant. + ## Docs - [`docs/START_HERE.md`](docs/START_HERE.md) — the docs map, by what you are trying to do. @@ -246,6 +239,9 @@ make install && make lint && make test Small, focused PRs with tests are welcome. Benchmark contributions carry aggregate numbers only. Development setup, checks, and conventions: [`CONTRIBUTING.md`](CONTRIBUTING.md). +Maintained by Ajay Surya Senthilrajan, with AI pair-programming recorded in commit trailers. +See the tests, design records and release evidence linked here. + ## License Apache-2.0. Protocol: VWP (Validity Warrant Protocol), spec in [`spec/`](spec/). diff --git a/docs/assets/dorian-hero-light.png b/docs/assets/dorian-hero-light.png deleted file mode 100644 index a8aaabe..0000000 Binary files a/docs/assets/dorian-hero-light.png and /dev/null differ diff --git a/docs/assets/dorian-hero.png b/docs/assets/dorian-hero.png deleted file mode 100644 index 580fbc2..0000000 Binary files a/docs/assets/dorian-hero.png and /dev/null differ diff --git a/docs/assets/hero-dark.svg b/docs/assets/hero-dark.svg new file mode 100644 index 0000000..0da08d0 --- /dev/null +++ b/docs/assets/hero-dark.svg @@ -0,0 +1,21 @@ + +Does the code still satisfy what the agent claimed it did? +dorian: claim warrants for code changes. Does the code still satisfy what the agent claimed it did? Claims become executable checks sealed beside the code and re-run on every commit. No model is called at check time. Evidence card, from the README Try it recipe: claim handler-exists, 'handler() lives in app.py'. dorian verify: verified 1/1 claim(s), WARRANTED (exit 0). A refactor renames handler(); the note never changes. dorian revalidate: handler-exists BROKEN, REVOKED (exit 4). A black-box test pins this recipe. + +dorian·claim warrants for code changes +Does the code still +satisfy what the agent +claimed it did? +Claims become executable checks sealed beside +the code and re-run on every commit. +No model is called at check time. + +CLAIM · "handler() lives in app.py" +dorian verify → verified 1/1 claim(s) +→ WARRANTED (exit 0) +a refactor renames handler(); +the note never changes +dorian revalidate → handler-exists BROKEN +→ REVOKED (exit 4) +README "Try it" · a black-box test pins this recipe + diff --git a/docs/assets/hero-light.svg b/docs/assets/hero-light.svg new file mode 100644 index 0000000..4662354 --- /dev/null +++ b/docs/assets/hero-light.svg @@ -0,0 +1,21 @@ + +Does the code still satisfy what the agent claimed it did? +dorian: claim warrants for code changes. Does the code still satisfy what the agent claimed it did? Claims become executable checks sealed beside the code and re-run on every commit. No model is called at check time. Evidence card, from the README Try it recipe: claim handler-exists, 'handler() lives in app.py'. dorian verify: verified 1/1 claim(s), WARRANTED (exit 0). A refactor renames handler(); the note never changes. dorian revalidate: handler-exists BROKEN, REVOKED (exit 4). A black-box test pins this recipe. + +dorian·claim warrants for code changes +Does the code still +satisfy what the agent +claimed it did? +Claims become executable checks sealed beside +the code and re-run on every commit. +No model is called at check time. + +CLAIM · "handler() lives in app.py" +dorian verify → verified 1/1 claim(s) +→ WARRANTED (exit 0) +a refactor renames handler(); +the note never changes +dorian revalidate → handler-exists BROKEN +→ REVOKED (exit 4) +README "Try it" · a black-box test pins this recipe + diff --git a/docs/assets/src/make_figures.py b/docs/assets/src/make_figures.py new file mode 100644 index 0000000..d5ca57e --- /dev/null +++ b/docs/assets/src/make_figures.py @@ -0,0 +1,456 @@ +"""Generate the README hero: ``docs/assets/hero-{light,dark}.svg``. + +Standard library only, deterministic, no network. ``--write`` regenerates both files; +``--check`` regenerates them in memory and exits 1 if a committed file is missing or differs. + +The SVGs are plain ```` and ```` elements: no ``