diff --git a/AGENTS.md b/AGENTS.md index 6fb72b2..e684bb2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,6 +92,17 @@ plan from "" (read the plan from a file you control instead of gene use the method schedule: runner: target: (config tier) models: fast , strong model tiering: plan/reflect/also→fast, act→strong (cascades; override e.g. `act fast`, `all strong`) +each cycle: plan, then act, then observe (config tier: the default cycle for every loop in the file; a loop's own `each cycle:` overrides it) +rigor: vibe coding | structured ai-assisted | agentic engineering (the spectrum dial; structured/agentic give every loop a back-edge + thrash guard for free) +mode: conductor | orchestrator (supervision posture: in-session/sync vs async/opens-a-PR) +runs as: (an auditable principal for unattended runs) +observe: (block) trace every cycle / meter tokens and cost / stop and warn if cost exceeds "$N" +sandbox: (block) no network access / allow egress to "host" only / cap cpu at … memory at … time at … +hooks: (loop body block) before each cycle | after act | on commit | on stop : "" passes|finds nothing (a failing hook blocks) +examples: (reference patterns to imitate — context engineering's 6th part) +knowledge: (read-only reference the agent must not edit) +use tools from the "" server (MCP: name servers whose tools the loop may use) +stages in parallel: (inside a pipeline: the indented stages run concurrently) ``` ### Predicates (`done when …`) @@ -101,13 +112,32 @@ done when the test "billing.spec.ts::apostrophe" passes # a named test done when "pnpm test" passes # a shell command, exit 0 done when "semgrep --severity=high" finds nothing # a shell command, empty output done when a human confirms "looks right at 375px" # a human check -done when the skill "email-review" approves # a review skill: approved / not -done when the skill "email-review" scores 8 or more # a review skill: numeric threshold +done when the skill "email-review" approves # an eval: approved / not +done when the skill "email-review" scores 8 or more # an eval: numeric threshold +done when the skill "api-review" scores 8 or more on the output # an eval of WHAT was produced +done when the skill "path-review" approves on the trajectory # an eval of HOW the agent got there + the bar: didn't weaken a test to go green; no writes outside api/ # the rubric the judge scores against ``` The command in a predicate runs in the user's shell with their privileges (like an npm script). It IS meant to be a real command. Prefer a fast, deterministic check. +### Tests vs evals — list as many `done when` as you need + +A loop may have **multiple `done when` lines, and ALL must pass** (a conjunction). Use this to +combine the two kinds of verification: + +- **TESTS** — a `test` / command predicate. Deterministic, checked by code (`"pnpm test" passes`, + `"semgrep …" finds nothing`). +- **EVALS** — a `skill` predicate. A rubric / LM judge for the non-deterministic parts. An eval + names its **subject**: `on the output` (the default — judges *what* was produced) or + `on the trajectory` (judges *how* the agent got there — the path and tool calls it took). An + indented `the bar:` line states the conditions the judge scores against. + +A trajectory eval is what catches the failures a green test can't — e.g. an agent that made a test +pass by weakening it. Pair a test with an eval when "done" means both "it works" and "it was built +the right way." + The **skill** predicate bridges an abstract goal to a verifiable one: when "done" isn't a test or a command (a good email, a sound design), have a review skill return an approved/rejected verdict or a numeric score. Build that review skill manually first and @@ -196,6 +226,10 @@ flow "deliver": irreversible. Use `ask me before …` for action policy, `a human approves before …` for a hard stage gate. - Output only valid `.loop` syntax. Comments start with `#`. +- **Friendly shorthands** (all desugar to the lines above — use freely): `check:` / `verify:` + = `done when` (a bare value is a shell command; a predicate phrase parses as-is); `in:` / + `look in:` / `files:` / `context:` = `look at:`; `when it breaks` = `when it fails`; + `when it gets stuck` = `when blocked`. ## Example — a single loop @@ -296,6 +330,45 @@ loop "add a healthcheck endpoint": after 6 tries: stop and warn "healthcheck stuck" ``` +## Config defaults & the project config file + +Anything in the **config tier** (the top of the file, before any definition) sets a default for +every loop in that file — so you write it once instead of repeating it per loop. The most common +repeater is the cycle: + +```loop +each cycle: plan, then act, then observe # the default for every loop below +models: fast haiku, strong opus + +pipeline "epic: ship it": + stage "story: build": + goal: it builds + done when "pnpm build" passes # no `each cycle:` — inherits the default + stage "story: verify": + goal: tests pass + each cycle: act, then observe # overrides just this stage + done when "pnpm test" passes +``` + +### `loop.config` — defaults for the whole repo + +To avoid repeating config across *files*, drop a **`loop.config`** (or `.looprc`) at your project +root. It is written in the same config-tier syntax — `each cycle:`, `models:`, a `git:` block — and +the runner reads it before every run, walking up from the `.loop` file to find it. It is the +**lowest** tier of the cascade, so a file's own config (and a per-loop directive) overrides it. + +```loop +# loop.config — applies to every .loop in the repo +each cycle: plan, then act, then observe +models: fast haiku, strong opus +git: + work on a branch + commit when the goal is met +``` + +**Cascade (lowest wins):** `loop.config` → a file's config tier → a per-loop directive. +The same rule already governs `git:` and `models:`. + ## Show the flow — every time it changes Whenever you create or edit a `.loop`, print its flow so the user sees the shape. @@ -309,6 +382,7 @@ flow, show the file chain. `loop-run ls` lists every loop in the repo. - `loop-run run file.loop` — execute it on Claude Code (plan/act/observe, reflect on failure, verify with `done when`, pause at human gates). - `loop-run show file.loop` — print the loop's flow as compact ASCII (and `loop-run ls` to list them). +- `loop-run explain file.loop` — describe the loop in plain English (a friendly check of what it will do). - `loop-run viz file.loop` — open a visual HTML schematic of the flow. ### Live visual — opt-in via `loop.config` diff --git a/agentic-engineering.loop b/agentic-engineering.loop new file mode 100644 index 0000000..832c507 --- /dev/null +++ b/agentic-engineering.loop @@ -0,0 +1,122 @@ +# Agentic Engineering for Loop — the epic, as a pipeline. +# Written in TODAY'S Loop grammar to build TOMORROW'S constructs (evals, rigor, hooks, …). +# Detailed spec per story: docs/agentic-engineering-plan.md +# Each stage is one story; stories run in order and a failing story halts the rest. + +models: fast haiku, strong opus + +# Config-tier default cycle — every stage below inherits it; a stage may still +# declare its own `each cycle:` to override. +each cycle: plan, then act, then observe + +git: + work on a branch + commit each story + +pipeline "epic: agentic engineering for Loop": + + # ── Wave 1 · Eval core (tests and evals together) ── + + stage "story: eval targets in the IR": + goal: doneWhen becomes a conjunction of predicates and the skill predicate gains a target (output or trajectory, default output) plus an inline `the bar:` rubric + look at: spec/loop-spec.schema.json, packages/parser/src/types.ts, packages/parser/src/parser.ts, and the last failure + a human approves the plan first + done when "npm test --workspace packages/parser" passes + when it fails: reflect on whether the schema and types and parser stayed in sync, then plan again + after 6 tries: stop and warn "IR change is rippling — needs a human" + + stage "story: capture the trajectory": + goal: the runtime retains a per-cycle trajectory and hands trajectory evals the cycle log instead of only the output summary + look at: packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, packages/runtime/src/types.ts, and the last failure + done when "npm test --workspace packages/runtime" passes + when it fails: reflect on what the verifier actually receives, then plan again + after 6 tries: stop and warn "trajectory capture stuck" + + stage "story: anti-thrash and feedback wiring": + goal: a blocking trajectory eval requires `the bar:`, borderline verdicts escalate to a human, eval-flaps feed the thrash guard, and reflect sees the trajectory + look at: packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure + done when "npm test --workspace packages/runtime" passes + when it fails: reflect on whether the stochastic judge can still self-thrash, then plan again + after 6 tries: stop and warn "eval gate still flaky" + + stage "story: the without-both lint": + goal: a code loop with a test but no eval (or vice-versa) warns to add the missing kind + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure + done when "npm test --workspace packages/parser" passes + when it fails: reflect, then plan again + after 5 tries: stop and warn "lint stuck" + + # ── Wave 2 · The dials (the spectrum, and who's in the loop) ── + + stage "story: the rigor dial": + goal: rigor (vibe coding, structured ai-assisted, agentic engineering) expands to bundled defaults via the preset cascade and sets the without-both lint severity + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, packages/stdlib, and the last failure + a human approves the plan first + done when "npm test --workspace packages/parser" passes + when it fails: reflect on the lowest-wins cascade, then plan again + after 6 tries: stop and warn "rigor cascade stuck" + + stage "story: conductor and orchestrator mode": + goal: mode (conductor or orchestrator) names the in-session vs async postures and warns on the incoherent vibe-plus-orchestrator combo + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure + done when "npm test --workspace packages/parser" passes + when it fails: reflect, then plan again + after 5 tries: stop and warn "mode stuck" + + # ── Wave 3 · Harness completeness ── + + stage "story: hooks at lifecycle points": + goal: a hooks block runs deterministic checks at named points (before each cycle, after act, on commit, on push, on stop) and a failing hook blocks + look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, packages/runtime/src/git.ts, and the last failure + done when "npm test --workspaces" passes + when it fails: reflect on the hooks-vs-also-vs-doneWhen carve, then plan again + after 6 tries: stop and warn "hooks stuck" + + stage "story: observe and the OpEx report": + goal: a per-cycle trace plus token-and-cost metering, a stop-time OpEx report, and an optional cost ceiling that stops and warns + look at: packages/runtime/src/engine.ts, packages/runtime/src/summary.ts, packages/runtime/src/show.ts, and the last failure + done when "npm test --workspace packages/runtime" passes + when it fails: reflect, then plan again + after 6 tries: stop and warn "observe stuck" + + # ── Wave 4 · Niche / standards ── + + stage "story: sandbox run isolation": + goal: a sandbox block declares no-network, an egress allowlist, and cpu-memory-time caps as config instead of prose + look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, examples/forge-sandbox.loop, and the last failure + a human approves the plan first + done when "npm test --workspaces" passes + when it fails: reflect, then plan again + after 6 tries: stop and warn "sandbox stuck" + + stage "story: knowledge and examples context": + goal: examples (imitate) and knowledge (read, never edit) complete the six context-engineering parts + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure + done when "npm test --workspace packages/parser" passes + when it fails: reflect, then plan again + after 5 tries: stop and warn "context keywords stuck" + + stage "story: parallel stages": + goal: stages-in-parallel fans out concurrently with a worktree per branch and a merge, backing orchestrator multi-agent work + look at: packages/runtime/src/engine.ts, packages/runtime/src/git.ts, packages/parser/src/parser.ts, and the last failure + a human approves the plan first + done when "npm test --workspace packages/runtime" passes + when it fails: reflect on edit collisions and the merge, then plan again + after 6 tries: stop and warn "parallel stages — concurrency risk; needs a human" + + stage "story: standards and identity": + goal: use-tools-from names MCP servers, the flow handoff is framed as an A2A contract, and runs-as gives background runs an auditable identity + look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure + done when "npm test --workspaces" passes + when it fails: reflect, then plan again + after 5 tries: stop and warn "standards stuck" + + # ── Cross-cutting · Framing ── + + stage "story: reframe the docs": + goal: AGENTS.md and README name what Loop already is (intent is the new interface, conductor and orchestrator, editor terminal background, Agent equals Model plus Harness, CapEx and OpEx) with the new constructs documented and their flows printed + look at: AGENTS.md, README.md, docs/MANUAL.md, and the last failure + a human reviews before stopping + done when a human confirms "the docs read clearly and the new constructs are covered" + when it fails: reflect, then plan again + after 4 tries: stop and warn "docs need a human pass" diff --git a/docs/MANUAL.md b/docs/MANUAL.md index 171485c..ba69a5a 100644 --- a/docs/MANUAL.md +++ b/docs/MANUAL.md @@ -116,15 +116,16 @@ the thrash guard). ## 4. The CLI ``` -loop-run [--model ] [--live] [--out ] +loop-run [--model ] [--live] [--out ] ``` | Command | What it does | |---|---| | `loop-run run ` | Execute the flow on Claude Code (plan/act/observe, reflect, verify, human gates). | | `loop-run parse ` | Parse to the loop-spec IR and print it as JSON (use to validate syntax). | +| `loop-run show ` | Print the loop's flow as compact ASCII (the "see the shape" view). | +| `loop-run explain ` | Describe the loop in **plain English** — handy for non-experts to check what a `.loop` will actually do. | | `loop-run viz ` | Write a self-contained HTML schematic of the flow. | -| `loop-run show ` | Print the loop's flow as compact ASCII. | | `loop-run ls [dir]` | List every `.loop` under a directory with its one-line shape. | | `loop-run live ` | Start the live dashboard server (no engine) and stay up; for driving the dashboard from an in-session run. | | `loop-run emit ''` | Push one event to a running `live` server (used by the `/loopflow` skill). | @@ -240,7 +241,7 @@ BMAD flow (discover → design → for each story) as one example method. | Line | Meaning | |---|---| | `goal: ` | The objective, in plain language. **Required.** | -| `done when ` | How the loop verifies itself (see Predicates). Omit only if a human gate decides completion. | +| `done when ` | How the loop verifies itself (see Predicates). **You can list several — all must pass** (a conjunction of tests and evals). Omit only if a human gate decides completion. | | `look at: , , and the last failure` | Context the agent reads before acting. Items are file paths or plain-language descriptions (e.g. `the billing form`) — the agent locates the actual files from descriptions before planning. `and the last failure` feeds the previous failure forward. | | `allow edits automatically, but ask me before ` | Action policy. Auto classes run unattended; confirm classes pause for you. Classes: `edit`, `migrate`, `push`, `deploy`, `delete`. | | `each cycle: plan, then act, then observe` | The repeated steps — any subset of `plan` / `act` / `observe`, in order. | @@ -255,6 +256,34 @@ BMAD flow (discover → design → for each story) as one example method. | `a human reviews before stopping` | A person judges the result before the loop may stop. | | `plan from ""` | Read the loop's plan from a file you control (e.g. a hand-written `.md`) instead of having the agent generate it. Omit it and the agent writes the plan (the default). | +### Friendly shorthands + +Plain-English synonyms for the lines above — write whichever reads naturally to you. They +desugar to the exact same loop-spec, so you never lose any power: + +| Shorthand | Same as | +|---|---| +| `check: pnpm test` | `done when "pnpm test" passes` (a bare value is a shell command) | +| `check: the skill "review" approves` | `done when the skill "review" approves` (a predicate phrase is parsed as-is) | +| `verify: …` | same as `check:` | +| `in: src/cart, src/tax` · `files: …` · `look in: …` · `context: …` | `look at: …` | +| `when it breaks: …` | `when it fails: …` | +| `when it gets stuck: …` | `when blocked: …` | + +A complete, friendly loop can be as short as: + +```loop +loop "fix the cart total": + goal: the cart total is correct with tax + in: src/cart, src/tax + check: pnpm test cart + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" +``` + +Run `loop-run explain ` to read any loop back in plain English and confirm it does +what you intend. + ### In a stage only | Line | Meaning | @@ -268,18 +297,31 @@ done when the test "billing.spec.ts::apostrophe" passes # a named test done when "pnpm test" passes # shell command, exit 0 (`succeeds` also works) done when "semgrep --severity=high" finds nothing # shell command, empty stdout done when a human confirms "looks right at 375px" # a human check -done when the skill "email-review" approves # a review skill: approved / not -done when the skill "email-review" scores 8 or more # a review skill: numeric threshold +done when the skill "email-review" approves # an eval: approved / not +done when the skill "email-review" scores 8 or more # an eval: numeric threshold +done when the skill "api-review" scores 8 or more on the output # an eval of WHAT was produced +done when the skill "path-review" approves on the trajectory # an eval of HOW it got there + the bar: didn't weaken a test to go green; no writes outside api/ # the rubric the judge scores against ``` The command runs in your shell with your privileges (like an npm script). Keep it fast and deterministic. -The **skill** predicate bridges an abstract goal to a verifiable one. When "done" isn't a -test or a command — a good email, a sound design, a sensible go/no-go call — a review skill -returns an approved/rejected verdict or a 0–10 score, and the loop treats that as its check. -Build the review skill manually first and confirm it judges well, then wire it in. See -`examples/skills_memory.loop` and `examples/email_review.loop`. +#### Tests vs evals + +A loop can list **several `done when` lines, and all must pass.** Use this to combine the +two kinds of verification: + +- **Tests** — `test` / command predicates. Deterministic, checked by code. +- **Evals** — `skill` predicates. A rubric / LM judge for the non-deterministic parts. An + eval names its **subject**: `on the output` (the default — judges *what* was produced) or + `on the trajectory` (judges *how* the agent got there — its path and tool calls). An + indented `the bar:` line states the conditions the judge must score against. + +A trajectory eval catches what a green test can't — e.g. an agent that made a test pass by +weakening it. Pair a test with an eval when "done" means both *it works* and *it was built +the right way*. Build the review skill manually first and confirm it judges well, then wire +it in. See `examples/skills_memory.loop` and `examples/email_review.loop`. ### Config tier (top of file) @@ -289,8 +331,52 @@ run with claude code # runner / provider (or: `runner claude code`) schedule: nightly # manual · nightly · on push · cron (parsed; run is manual via the CLI) target: ./src # working directory the loop operates in notify: slack # notification destination (reserved) +each cycle: plan, then act, then observe # the DEFAULT cycle for every loop in the file +``` + +Anything in the config tier is a **default for every loop in the file**, so you write it +once instead of repeating it per loop. A `each cycle:` here is inherited by every loop; a +loop with its own `each cycle:` overrides it. The same lowest-wins cascade governs `git:` +and `models:`. + +#### Project defaults — `loop.config` + +To avoid repeating config across *files*, drop a **`loop.config`** (or `.looprc`) at your +project root. It uses the same config-tier syntax (`each cycle:`, `models:`, a `git:` +block) and the runner reads it before every run, walking up from the `.loop` file to find +it. It is the **lowest** tier of the cascade, so a file's own config and a per-loop +directive both override it. + +```loop +# loop.config — applies to every .loop in the repo +each cycle: plan, then act, then observe +models: fast haiku, strong opus +git: + work on a branch + commit when the goal is met ``` +**Cascade (lowest wins):** `loop.config` → a file's config tier → a per-loop directive. + +### Agentic-engineering features + +These add agentic-engineering discipline to Loop. All are optional; a simple loop ignores them. + +| Construct | Where | What it does | +|---|---|---| +| `rigor: vibe coding \| structured ai-assisted \| agentic engineering` | config tier | The spectrum dial. Under structured/agentic, every loop is **born with** a reflect-on-fail back-edge and a thrash guard unless it sets its own — sensible defaults, no boilerplate. Also enables the "without both [tests and evals]" lint. | +| `mode: conductor \| orchestrator` | config tier | Supervision posture — conductor (in-session, gates inline) vs orchestrator (async, opens a PR). | +| `hooks:` block | loop body | Deterministic checks at lifecycle points — `before each cycle` / `after act` / `on commit` / `on push` / `on stop`: `"" passes` or `finds nothing`. **A failing hook blocks.** | +| `observe:` block | config tier | `trace every cycle` / `meter tokens and cost` / `stop and warn if cost exceeds "$N"`. The CLI prints a stop-time **OpEx report** (cycles, reflects, first-pass success). | +| `sandbox:` block | config tier | Run isolation: `no network access` / `allow egress to "host" only` / `cap cpu at … memory at … time at …`. | +| `runs as: ` | config tier | An auditable principal for unattended runs. | +| `examples:` / `knowledge:` | loop body | `examples:` = reference patterns to imitate; `knowledge:` = read-only reference the agent must not edit. With `look at:`/`remember in:`/`use skills:`/`allow…ask`, a loop can declare all six context-engineering parts. | +| `use tools from the "" server` | loop body | Name MCP servers whose tools the loop may use. | +| `stages in parallel:` | inside a pipeline | The indented stages run **concurrently** (barrier-join, fail-fast). File-safe parallel edits need a worktree per branch. | + +See [`examples/agentic/`](../examples/agentic/) for one file per feature; run `loop-run explain +` on any of them to read it back in plain English. + ### Git strategy A `git:` block sets the version-control strategy for the run. It can appear at the top of diff --git a/docs/agentic-engineering-plan.md b/docs/agentic-engineering-plan.md new file mode 100644 index 0000000..6d74141 --- /dev/null +++ b/docs/agentic-engineering-plan.md @@ -0,0 +1,170 @@ +# Plan — Agentic Engineering constructs for Loop + +> Bringing agentic-engineering discipline into Loop. +> Branch: `claude/loop-lang-concepts-3p5tz1`. Companion pipeline: [`agentic-engineering.loop`](../agentic-engineering.loop). + +## Context + +Agentic engineering is about how much structure, verification and judgment surround the output. +Loop already embodies the disciplined end — its `plan → act → observe` + reflect back-edge **is** +the agent loop, and `goal:` + `done when` **is** "success criteria, not step-by-step instructions". +This epic closes the remaining gaps and **names** what Loop already is. + +The headline gap: **evals**. The load-bearing principle is that without both TESTS and EVALS, +verification is incomplete. Loop has TESTS (deterministic, +checked by code) and a proto-eval fused into `done when the skill … scores N` — but it is unnamed, +its rubric is invisible, and it can only judge the **output**, never the **trajectory** (*"not just +what was produced but HOW the agent got there"*). Trajectory is exactly what catches the 80%-problem +failures: *"conceptual failures that look right and may pass basic tests."* + +This is **building tomorrow's Loop using today's Loop**: the companion `.loop` pipeline plans this +work as an agentic-engineering pipeline, written entirely in the *current* grammar. + +## The epic → stories + +Stories are grouped into four waves (deck terminology in **bold**). Each is a `stage` in the +companion pipeline. Stories run in order; a failing story halts the rest. Risky stories (IR +changes, run isolation, real concurrency) carry a human gate. + +| # | Story | Wave | Gate | `done when` | +|---|---|---|---|---| +| 1 | Eval targets in the IR | 1 · Eval core | approve plan | `npm test --workspace packages/parser` | +| 2 | Capture the trajectory | 1 · Eval core | — | `npm test --workspace packages/runtime` | +| 3 | Anti-thrash + feedback wiring | 1 · Eval core | — | `npm test --workspace packages/runtime` | +| 4 | The "without both" lint | 1 · Eval core | — | `npm test --workspace packages/parser` | +| 5 | The `rigor:` dial | 2 · Dials | approve plan | `npm test --workspace packages/parser` | +| 6 | Conductor / Orchestrator `mode:` | 2 · Dials | — | `npm test --workspace packages/parser` | +| 7 | `hooks:` — lifecycle checkpoints | 3 · Harness | — | `npm test --workspaces` | +| 8 | `observe:` — trace + OpEx report | 3 · Harness | — | `npm test --workspace packages/runtime` | +| 9 | `sandbox:` — run isolation | 4 · Niche | approve plan | `npm test --workspaces` | +| 10 | `knowledge:` + `examples:` context | 4 · Niche | — | `npm test --workspace packages/parser` | +| 11 | Parallel stages | 4 · Niche | approve plan | `npm test --workspace packages/runtime` | +| 12 | Standards & identity (MCP / A2A / `runs as:`) | 4 · Niche | — | `npm test --workspaces` | +| 13 | Reframe the docs | cross-cutting | review before stop | a human confirms | + +--- + +## Wave 1 — Eval core (the headline) + +### Story 1 — Eval targets in the IR +Generalize verification from one predicate to a conjunction, and make an eval's **target** explicit. +- **Spec/IR** (`spec/loop-spec.schema.json`, `packages/parser/src/types.ts`): `Loop.doneWhen` + becomes `Predicate[]` (all must pass — the conjunction that lets a loop require a TEST **and** an + EVAL). Extend the `skill` predicate variant with `target?: "output" | "trajectory"` (default + `"output"`) and `bar?: string` (the inline rubric). +- **Parser** (`packages/parser/src/parser.ts`): collect every `done when` line into the list; parse + the `on the output` / `on the trajectory` qualifier and an indented `the bar:` block under a skill + predicate. Back-compat: a bare skill predicate parses as an output eval. +- **`done when`** stays the single verification verb — reject a second verb. Existing `.loop` files + are unchanged (their single predicate becomes a one-element list). +- **Gate:** human approves the plan (IR shape change ripples through parser + runtime). + +### Story 2 — Capture the trajectory +The runner already parses every tool call with its inputs (`interpretStreamLine`/`toolLines`, +`runners/claudeCode.ts:77–126`) and **discards** them after the live display. Retain them. +- **Runtime** (`packages/runtime/src/engine.ts`, `runners/claudeCode.ts`, `types.ts`): accumulate + the per-cycle activity into a trajectory record — `{ plan, toolCalls[{tool,input}], filesTouched, + observe, reflection }`. Add `SkillVerifyInput.trajectory`. +- Route verifier context by target: **output** evals receive the diff/output (today's + `lastActSummary`, `engine.ts:211`); **trajectory** evals receive the cycle log. +- A loop's `done` requires every predicate in the list to pass (test + output eval + trajectory eval). + +### Story 3 — Anti-thrash + feedback wiring +Keep a stochastic judge from self-thrashing, and let reflection see the path. +- A **blocking** trajectory eval requires an explicit `the bar:` (no naked path-judge gates `done`). +- A borderline verdict escalates via the existing `when blocked: ask a human` path instead of retrying. +- An eval-flap counter feeds the `after N tries` thrash guard so a flaky judge alone can't burn the budget. +- Thread the trajectory into `ReflectInput` so `reflect on the path it took` fixes the path, not the artifact. + +### Story 4 — The "without both" lint +Turn this principle into a check. For **code loops** only (loops with an `edit` allowance): +warn when a loop has a TEST but no EVAL (or vice-versa). Severity is wired to `rigor:` in Story 5 +(warn by default; error under `rigor: agentic engineering`). Lives in the parser/validator so +`loop-run parse` surfaces it. + +## Wave 2 — The dials + +### Story 5 — The `rigor:` dial +Config-tier knob — `rigor: vibe coding | structured ai-assisted | agentic engineering` — that +expands to bundled defaults over existing knobs, reusing the `use the ` preset machinery +(`Config.use`/`useOverrides`) and the lowest-wins cascade already used for git/models. Sets the +"without both" lint severity. **Gate:** human approves the plan (touches the config cascade). + +### Story 6 — Conductor / Orchestrator `mode:` +Config-tier `mode: conductor | orchestrator` — names what `/loopflow` (in-session, gates inline) +and `loop-run` (async, opens a PR) already do. Orthogonal to `rigor:`; warn on the incoherent +combo `rigor: vibe coding` + `mode: orchestrator` (the "token burn from unverified loops" anti-pattern +running unattended). + +## Wave 3 — Harness completeness + +### Story 7 — `hooks:` — deterministic checks at lifecycle points +A `hooks:` block binding a deterministic predicate (`passes` / `finds nothing`) to a lifecycle +point — `before each cycle · after act · on commit · on push · on stop`. A failing hook blocks +(*"hooks block unsafe commits"*), generalizing the hard-coded never-push-to-main. Carve: +success criterion → `done when`; recurring blocking checkpoint → `hooks:`; one-time polish → `also:`. + +### Story 8 — `observe:` — trace + OpEx report +Per-cycle trace (plan/act/observe + tools called), token & cost metering split fast/strong, a +stop-time OpEx report (cycles, reflects, first-pass success, hit-the-guard?), and an optional +`stop and warn if cost exceeds "$N"` — an OpEx ceiling beside the thrash guard's CapEx ceiling. + +## Wave 4 — Niche / standards + +### Story 9 — `sandbox:` — run isolation +*"Where code runs and what it cannot reach"*: no-network / egress allowlist / CPU-mem-time caps, +declared as config rather than prose (today `examples/forge-sandbox.loop` encodes this only inside +`goal:` text). **Gate:** human approves the plan (security-sensitive). + +### Story 10 — `knowledge:` + `examples:` context +Complete context engineering's six parts. `examples:` = reference patterns to imitate; +`knowledge:` = read-only reference (docs/diagrams) the agent must **not** edit (a label *and* a soft +guardrail). Distinct intents from `look at:` (understand + edit). + +### Story 11 — Parallel stages +`stages in parallel:` — fan out independent stories concurrently, barrier-join before the next +stage, **a worktree per parallel branch + a merge** to avoid edit collisions. The one story with +real concurrency cost; backs *"orchestrator = multi-agent, multi-file parallel."* **Gate:** human +approves the plan. + +### Story 12 — Standards & identity +Adopt open standards (CH7 orgs). `use tools from the ""` names MCP servers as the Tools +surface; frame the flow **handoff** as an A2A contract (`produces:` / `consumes the … from `); +add `runs as:` identity so unattended orchestrator runs have an auditable principal (the thin 5th +anatomy part, *Deployment*). + +## Cross-cutting + +### Story 13 — Reframe the docs +Zero-grammar framing wins, written into `AGENTS.md` + `README.md`: *"intent is the new interface"* +as Loop's one-liner; the Conductor/Orchestrator and editor/terminal/background maps +(`/loopflow` · `loop-run run` · `schedule:`+PR); the five-knobs ↔ harness table (*Agent = Model + +Harness*); static vs dynamic context and the CapEx/OpEx framing. Print the flow of each new example +`.loop`. **Gate:** a human reviews before stopping. + +--- + +## Sequencing & dependencies + +- **Wave 1 is the spine** and strictly ordered: Story 1 (IR) → 2 (capture) → 3 (safety) → 4 (lint). +- Story 4's lint severity depends on Story 5 (`rigor:`); ship the lint as warn-only, then wire + severity once `rigor:` lands. +- Waves 2–4 are largely independent of each other and can be reordered by appetite. Story 11 + (parallel) is the only one with real concurrency/merge risk — do it deliberately, last. + +## Verification (end to end) + +- **Per story:** `npm test --workspace packages/` (the stage `done when`). Add a + focused `test/*.test.js` with each story before making it pass — dogfooding the loop. +- **Round-trip:** new syntax through `loop-run parse ` → loop-spec JSON; keep + `spec/loop-spec.schema.json` and `types.ts` in sync. +- **Behavioral proof for evals:** a loop that games a test green by deleting the failing assertion + must **fail** its trajectory eval (`the bar: didn't weaken a test to go green`) while a clean path + passes. +- **Flow rendering:** `loop-run show` renders the new constructs; print the flow for every new + example. +- **Full sweep:** `npm test --workspaces` stays green throughout. + +## Out of scope (for now) +Sub-agent spawning *within* a cycle (pipeline-level parallel covers the orchestrator need); a +typed/structured handoff beyond the A2A naming; a full cost-routing optimizer beyond `route:`. diff --git a/docs/index.html b/docs/index.html index 3b622f6..a4a4904 100644 --- a/docs/index.html +++ b/docs/index.html @@ -212,6 +212,14 @@
  • Live dashboard
  • Your global library
  • +
  • The full system
  • +
  • Tests & evals
  • +
  • Context & knowledge
  • +
  • Config & project defaults
  • +
  • Rigor, defaults & mode
  • +
  • Hooks
  • +
  • Observe & sandbox
  • +
  • Friendly syntax
  • Go deeper
  • 📖 Full manual →
  • 📖 Keyword reference →
  • @@ -839,6 +847,179 @@

    Your global library — save a loop, run it in any repo +

    Tests & evals — verify both halves verification

    +

    A loop is only as trustworthy as its check. You can list several done when lines, and all must pass (a conjunction) — which lets you combine the two kinds of verification.

    + + + + +
    KindWhat it checksHow
    TestsThe deterministic parts — a given input reliably produces a given output.A command or named test, checked by code.
    EvalsThe non-deterministic parts — did it meet the quality bar?A review skill acting as a rubric / LM judge, returning approve or a score.
    +

    An eval names its subject: on the output (the default — judges what was produced) or on the trajectory (judges how the agent got there — its path and tool calls). An indented the bar: line states the conditions the judge scores against.

    +
    loop "add the refund endpoint":
    +  goal: POST /refunds reverses a charge and is safe to ship
    +  in: api/refunds.ts, api/charges.ts
    +  done when "pnpm test refunds" passes                              # a TEST (deterministic)
    +  done when the skill "api-review" scores 8 or more on the output   # an OUTPUT eval
    +  done when the skill "path-review" approves on the trajectory       # a TRAJECTORY eval
    +    the bar: did not weaken a test to go green; no writes outside api/
    +  when it breaks: reflect on whether the test or the path failed, then plan again
    +  after 6 tries: stop and warn "cannot satisfy the test and the evals"
    +
    Why a trajectory eval? A green test isn't always proof — an agent can pass a test by weakening it, a failure that looks right. A trajectory eval inspects the actual path (it sees the edit to the spec file) and rejects it against the bar:. Pair a test with an eval when "done" means both it works and it was built the right way.
    + + + +
    +

    Context & knowledge — what the agent reads context

    +

    Give the agent the right material before it plans. Each input carries a distinct intent, so the agent knows what it may change and what it must only learn from.

    + + + + + + + +
    LineIntent
    look at: (alias in:)Files it reads and may edit. Add and the last failure to feed the previous error forward.
    knowledge:Read-only reference (docs, diagrams) it must not edit.
    examples:Reference patterns to imitate — "build it like these".
    use skills:Named skills it may invoke while planning/acting.
    use tools from the "x" serverAn MCP server whose tools it may use.
    +
    loop "add the webhooks endpoint":
    +  goal: POST /webhooks matches the house style
    +  look at: routes/
    +  knowledge: docs/webhooks-spec.md, the architecture diagram
    +  examples: routes/payments.ts, routes/users.ts
    +  use tools from the "github" server
    +  check: pnpm test webhooks
    +
    + + +
    +

    Config tier & project defaults don't repeat yourself

    +

    Lines at the top of the file, before any definition, set defaults for every loop in it. Write the common stuff once.

    +
    each cycle: plan, then act, then observe   # default cycle for every loop below
    +models: fast haiku, strong opus            # routing: plan/reflect → fast, act → strong
    +git:
    +  work on a branch
    +  commit when the goal is met
    +

    Project defaults — loop.config

    +

    To avoid repeating config across files, drop a loop.config (or .looprc) at your project root. It is written in the same config-tier syntax as a .loop file — just the config, no definitions — and the runner reads it before every run, walking up from the .loop file until it finds one (like .gitignore discovery).

    +
    # loop.config — sits at the repo root, applies to every .loop in the project
    +each cycle: plan, then act, then observe
    +models: fast haiku, strong opus
    +rigor: structured ai-assisted
    +git:
    +  work on a branch
    +  commit when the goal is met
    +

    With that in place, your everyday loops stay tiny — they inherit the cycle, model routing, rigor defaults, and git policy automatically:

    +
    # any .loop in the repo — inherits everything from loop.config
    +loop "fix the cart total":
    +  goal: the cart total is correct with tax
    +  in: src/cart, src/tax
    +  check: pnpm test cart
    +
    Cascade — lowest wins +
      +
    • loop.config → a file's own config tier → a per-loop directive. Each layer overrides the one below it, key by key.
    • +
    • So a single loop can still override the project default — e.g. its own each cycle: act, then observe or a stricter rigor: agentic engineering — without touching loop.config.
    • +
    • The same rule already governs git:, models:, each cycle:, and rigor:.
    • +
    +
    +
    One place to set the bar for a whole repo. Put rigor: agentic engineering in loop.config and every loop in the project is born with a reflect-on-fail back-edge, a thrash guard, and the tests-and-evals nudge — your team's verification posture, encoded once.
    +
    + + +
    +

    Rigor, sensible defaults & mode the dial

    +

    One knob places a loop on the speed-vs-reliability spectrum and bundles best-practice defaults — so a careful loop needs no boilerplate.

    + + + + + +
    LevelWhat it bundles
    vibe codingNothing injected — fast and disposable.
    structured ai-assistedEvery loop is born with a reflect-on-fail back-edge and a thrash guard (unless it sets its own).
    agentic engineeringThe same defaults, plus a lint nudging you toward both a test and an eval.
    +
    rigor: agentic engineering
    +
    +loop "harden the auth module":
    +  goal: auth handles malformed tokens safely
    +  in: src/auth
    +  done when "pnpm test auth" passes
    +  done when the skill "security-review" approves on the trajectory
    +    the bar: no secrets logged; rejects malformed tokens without crashing
    +
    That loop never wrote a when it fails or an after N tries — yet loop-run show reveals a full back-edge and an after 8 tries guard, supplied by the dial. A companion knob, mode: conductor | orchestrator, names the supervision posture — in-session and synchronous, vs. async and opening a PR.
    +
    + + +
    +

    Hooks — checks at lifecycle points guardrails

    +

    A hooks: block binds a deterministic check to a point in the loop's life. A failing hook blocks.

    +
    loop "refactor the token logic":
    +  goal: extract the token logic with behavior unchanged
    +  in: src/auth
    +  done when "pnpm test auth" passes
    +  hooks:
    +    before each cycle: "pnpm typecheck" passes
    +    after act:         "pnpm lint" finds nothing
    +    on commit:         "semgrep --severity=high" finds nothing
    +

    Points: before each cycle, after plan, after act, after observe, on commit, on push, on stop.

    +
    Where does a check belong? +
      +
    • A success criterion → done when.
    • +
    • A recurring blocking checkpoint → hooks:.
    • +
    • A one-time polish after the goal is met → also:.
    • +
    +
    +
    + + +
    +

    Observability & sandbox production substrate

    +

    An observe: block makes a run's cost and shape visible. On stop, the CLI prints an OpEx report — cycles run, reflects (back-edges) taken, first-pass success, and outcome.

    +
    observe:
    +  trace every cycle
    +  meter tokens and cost
    +  stop and warn if cost exceeds "$5"
    +
    +loop "port the reports module":
    +  goal: reports render identically under the v2 API
    +  in: src/reports
    +  models: fast haiku, strong opus
    +  check: pnpm test reports
    +

    Run isolation & identity

    +

    For runs of agent-written code, declare isolation as config — where code runs and what it cannot reach — plus an auditable identity for unattended runs.

    +
    sandbox:
    +  no network access
    +  allow egress to "registry.npmjs.org" only
    +  cap cpu at 2 cores, memory at 4g, time at 10m
    +  cannot reach the host filesystem or cloud metadata
    +runs as: ci-bot
    +
    +loop "run the agent-written migration":
    +  goal: the generated migration applies cleanly against a scratch db
    +  check: pnpm test:migration
    +  each cycle: act, then observe
    +
    + + +
    +

    Friendly syntax & explain easy mode

    +

    Plain-English shorthands read naturally and desugar to the full grammar — nothing is lost, and experts can drop down to the full syntax anytime.

    + + + + + + + +
    ShorthandSame as
    check: pnpm testdone when "pnpm test" passes (a bare value is a command)
    check: the skill "x" approvesa predicate phrase is parsed as-is
    in: · files: · look in:look at:
    when it breaks:when it fails:
    when it gets stuck:when blocked:
    +

    A complete, friendly loop can be just a few lines:

    +
    loop "fix the cart total":
    +  goal: the cart total is correct with tax
    +  in: src/cart, src/tax
    +  check: pnpm test cart
    +

    And you can always read a loop back in plain English to confirm it does what you intend:

    +
    loop-run explain fix.loop
    +# → "Loop 'fix the cart total' works toward: the cart total is correct with tax.
    +#    Each round it plans the change, makes the change, then checks the result.
    +#    It's done when running `pnpm test cart` succeeds. …"
    +
    +

    Go deeper

    @@ -877,6 +1058,11 @@

    Go deeper

    "a human approves before","a human approves first","a human confirms","ask a human","ask me before" ]; var KW = [ + "stages in parallel","stop and warn if cost exceeds","meter tokens and cost","allow egress to", + "before each cycle","use tools from","trace every cycle","no network access","on the trajectory", + "on the output","when it gets stuck","when it breaks","then plan again","after observe","after act", + "after plan","on commit","on push","on stop","runs as","the bar","the skill","rigor","mode","hooks", + "sandbox","knowledge","examples","check","verify","scores","approves","models","cap","egress", "with the result of","the last failure","finds nothing","each cycle","done when","look at", "for each","plan from","run with","use the","the test","pull request","open a pull request", "work on a branch","work in a worktree","work in place","push when done","do not push", diff --git a/examples/agentic/context.loop b/examples/agentic/context.loop new file mode 100644 index 0000000..c3f81d2 --- /dev/null +++ b/examples/agentic/context.loop @@ -0,0 +1,10 @@ +# Context engineering's six parts, in one loop. +loop "add the webhooks endpoint": + goal: POST /webhooks matches the house style + look at: routes/ + knowledge: docs/webhooks-spec.md, the architecture diagram + examples: routes/payments.ts, routes/users.ts + use tools from the "github" server + check: pnpm test webhooks + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" diff --git a/examples/agentic/evals.loop b/examples/agentic/evals.loop new file mode 100644 index 0000000..2cb6285 --- /dev/null +++ b/examples/agentic/evals.loop @@ -0,0 +1,10 @@ +# Tests verify the deterministic parts; evals verify the non-deterministic parts — use both. +loop "add the refund endpoint": + goal: POST /refunds reverses a charge and is safe to ship + in: api/refunds.ts, api/charges.ts + done when "pnpm test refunds" passes # a TEST (deterministic) + done when the skill "api-review" scores 8 or more on the output # an OUTPUT eval + done when the skill "path-review" approves on the trajectory # a TRAJECTORY eval + the bar: did not weaken a test to go green; no writes outside api/ + when it breaks: reflect on whether the test or the path failed, then plan again + after 6 tries: stop and warn "cannot satisfy the test and the evals" diff --git a/examples/agentic/friendly.loop b/examples/agentic/friendly.loop new file mode 100644 index 0000000..ec9bde4 --- /dev/null +++ b/examples/agentic/friendly.loop @@ -0,0 +1,7 @@ +# The friendly surface — plain-English shorthands, all desugaring to the full grammar. +loop "fix the cart total": + goal: the cart total is correct with tax + in: src/cart, src/tax + check: pnpm test cart + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" diff --git a/examples/agentic/hooks.loop b/examples/agentic/hooks.loop new file mode 100644 index 0000000..726b4f3 --- /dev/null +++ b/examples/agentic/hooks.loop @@ -0,0 +1,11 @@ +# Deterministic checks at lifecycle points — a failing hook blocks. +loop "refactor the token logic": + goal: extract the token logic with behavior unchanged + in: src/auth + done when "pnpm test auth" passes + hooks: + before each cycle: "pnpm typecheck" passes + after act: "pnpm lint" finds nothing + on commit: "semgrep --severity=high" finds nothing + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" diff --git a/examples/agentic/observe.loop b/examples/agentic/observe.loop new file mode 100644 index 0000000..be472a9 --- /dev/null +++ b/examples/agentic/observe.loop @@ -0,0 +1,13 @@ +# Observability — make "token burn from unverified loops" visible. +observe: + trace every cycle + meter tokens and cost + stop and warn if cost exceeds "$5" + +loop "port the reports module": + goal: reports render identically under the v2 API + in: src/reports + models: fast haiku, strong opus + check: pnpm test reports + when it breaks: reflect, then plan again + after 8 tries: stop and warn "stuck" diff --git a/examples/agentic/parallel.loop b/examples/agentic/parallel.loop new file mode 100644 index 0000000..4a9d270 --- /dev/null +++ b/examples/agentic/parallel.loop @@ -0,0 +1,22 @@ +# Orchestrator mode — multi-agent, multi-file parallel work. +mode: orchestrator +git: + work on a branch + open a pull request + +pipeline "epic: customer refunds": + stage "discovery": + goal: capture intent as spec.md + check: test -s spec.md + a human approves before building + stage "build": + goal: the refunds endpoint works + plan from "spec.md" + check: pnpm test refunds + stages in parallel: + stage "docs": + goal: docs match the endpoint + check: pnpm docs:check + stage "perf": + goal: p95 under 200ms + check: pnpm bench refunds diff --git a/examples/agentic/rigor.loop b/examples/agentic/rigor.loop new file mode 100644 index 0000000..9849665 --- /dev/null +++ b/examples/agentic/rigor.loop @@ -0,0 +1,11 @@ +# `rigor: agentic engineering` gives every loop a reflect-on-fail back-edge and a +# thrash guard for free — sensible defaults you don't have to write. +rigor: agentic engineering +models: fast haiku, strong opus + +loop "harden the auth module": + goal: auth handles malformed tokens safely + in: src/auth + done when "pnpm test auth" passes + done when the skill "security-review" approves on the trajectory + the bar: no secrets logged; rejects malformed tokens without crashing diff --git a/examples/agentic/sandbox.loop b/examples/agentic/sandbox.loop new file mode 100644 index 0000000..41c34b6 --- /dev/null +++ b/examples/agentic/sandbox.loop @@ -0,0 +1,12 @@ +# Run isolation — where code runs and what it cannot reach. +sandbox: + no network access + allow egress to "registry.npmjs.org" only + cap cpu at 2 cores, memory at 4g, time at 10m + cannot reach the host filesystem or cloud metadata +runs as: ci-bot + +loop "run the agent-written migration": + goal: the generated migration applies cleanly against a scratch db + check: pnpm test:migration + each cycle: act, then observe diff --git a/examples/feature/csv-export.loop b/examples/feature/csv-export.loop index 188b7e3..659bca6 100644 --- a/examples/feature/csv-export.loop +++ b/examples/feature/csv-export.loop @@ -19,7 +19,9 @@ pipeline "feature: export table to CSV": look at: src/export, src/components/Table.tsx, and the last failure # where to focus allow edits automatically, but ask me before migrations or pushes # auto-edit, human-gate the scary stuff each cycle: plan, then act, then observe # the inner reasoning loop - done when the test "export/csv.spec.ts::downloads a csv" passes # the one acceptance test + done when the test "export/csv.spec.ts::downloads a csv" passes # a TEST (deterministic) + done when the skill "code-review" approves on the trajectory # an EVAL — pair a deterministic test with a judged eval + the bar: did not weaken the acceptance test; reused the existing export module when it fails: reflect on which layer broke, then plan again # learn from each red run after 6 tries: stop and warn "csv export is thrashing" # give up before burning cycles diff --git a/examples/forge-sandbox.loop b/examples/forge-sandbox.loop index f16fd5a..18bbf5c 100644 --- a/examples/forge-sandbox.loop +++ b/examples/forge-sandbox.loop @@ -1,6 +1,16 @@ # examples/forge-sandbox.loop — how Forge's sandbox runner was built with LoopFlow. # Forge is a ticket-driven implementation platform; this module runs agent-written # code in isolation. Built one provable stage at a time — a failing stage halts the rest. + +# The isolation this module enforces is also declared here as config (not just prose), +# so the loop that builds it runs under the same constraints, as an auditable identity. +sandbox: + no network access + allow egress to "registry.npmjs.org" only + cap cpu at 2 cores, memory at 4g, time at 10m + cannot reach the host filesystem or cloud metadata +runs as: forge-sandbox-bot + pipeline "forge sandbox runner": stage isolate: goal: every run gets a fresh, network-less container with CPU and memory caps diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index a18810d..0d744fd 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -6,19 +6,35 @@ import { Flow, FlowStep, GitPolicy, + Hook, + HookPoint, Loop, LoopContext, LoopFile, LOOP_VERSION, ModelPolicy, + ParseDefaults, ParseError, Pipeline, Policy, Predicate, + Rigor, Stage, Transition, } from "./types.js"; +const RIGOR_LEVELS: Rigor[] = ["vibe coding", "structured ai-assisted", "agentic engineering"]; + +const HOOK_POINTS: Record = { + "before each cycle": "before-cycle", + "after plan": "after-plan", + "after act": "after-act", + "after observe": "after-observe", + "on commit": "on-commit", + "on push": "on-push", + "on stop": "on-stop", +}; + interface Line { indent: number; text: string; @@ -115,13 +131,17 @@ function parsePredicate(s: string, lineNo: number): Predicate { m = text.match(/^a human confirms\s+"([^"]+)"$/i); if (m) return { type: "human", description: m[1] }; - // the skill "X" scores N or more -> skill verdict with a numeric threshold - m = text.match(/^the skill\s+"([^"]+)"\s+scores\s+(\d+)(?:\s+or more)?$/i); - if (m) return { type: "skill", skill: m[1], expect: "approve", minScore: parseInt(m[2], 10) }; + // An eval predicate may name its subject: `on the output` (default) or `on the trajectory`. + const subj = (s: string | undefined): { subject?: "output" | "trajectory" } => + s ? { subject: s.toLowerCase() as "output" | "trajectory" } : {}; + + // the skill "X" scores N or more [on the output|trajectory] -> eval with a numeric threshold + m = text.match(/^the skill\s+"([^"]+)"\s+scores\s+(\d+)(?:\s+or more)?(?:\s+on the (output|trajectory))?$/i); + if (m) return { type: "skill", skill: m[1], expect: "approve", minScore: parseInt(m[2], 10), ...subj(m[3]) }; - // the skill "X" approves -> skill verdict (approved / not) - m = text.match(/^the skill\s+"([^"]+)"\s+approves$/i); - if (m) return { type: "skill", skill: m[1], expect: "approve" }; + // the skill "X" approves [on the output|trajectory] -> eval (approved / not) + m = text.match(/^the skill\s+"([^"]+)"\s+approves(?:\s+on the (output|trajectory))?$/i); + if (m) return { type: "skill", skill: m[1], expect: "approve", ...subj(m[2]) }; throw new ParseError(`could not understand "done when ${text}"`, lineNo); } @@ -179,11 +199,23 @@ function parseWhenCondition(cond: string, lineNo: number): Pick ln.indent) { + hooks.push(parseHook(body[j].text, body[j].lineNo)); + j++; + } + if (hooks.length === 0) throw new ParseError(`'hooks:' block is empty`, ln.lineNo); + loop.hooks = hooks; + i = j; + continue; + } if ((m = t.match(/^models:\s*(.+)$/i))) { loop.models = parseModelsLine(m[1], ln.lineNo); i++; continue; @@ -310,13 +354,38 @@ function interpretLoopBody(name: string | null, body: Line[]): LoopBodyResult { i++; continue; } if ((m = t.match(/^done when\s+(.+)$/i))) { - loop.doneWhen = parsePredicate(m[1], ln.lineNo); + const pred = parsePredicate(m[1], ln.lineNo); + // An optional indented `the bar:` rubric line attaches to a skill eval. + const childBar = body[i + 1]; + if (childBar && childBar.indent > ln.indent) { + const bm = childBar.text.match(/^the bar:\s*(.+)$/i); + if (bm) { + if (pred.type !== "skill") { + throw new ParseError(`'the bar:' applies to a skill eval, not "${m[1].trim()}"`, childBar.lineNo); + } + pred.bar = bm[1].trim(); + i++; // consume the bar line + } + } + (loop.doneWhen ??= []).push(pred); i++; continue; } - if ((m = t.match(/^look at:\s*(.+)$/i))) { + // `look at:` and its friendly synonyms (`in:`, `look in:`, `files:`, `context:`). + if ((m = t.match(/^(?:look at|look in|files|context|in):\s*(.+)$/i))) { loop.context = parseContext(m[1]); i++; continue; } + // Friendly `check:` / `verify:` — sugar for a `done when` check. A bare value is a shell + // command (`check: npm test`); a predicate phrase (`check: the skill "x" approves`) is parsed as-is. + if ((m = t.match(/^(?:check|verify):\s*(.+)$/i))) { + const val = m[1].trim(); + const isPhrase = /^(the test|the skill|a human)\b/i.test(val) || /^".*"\s+(passes|succeeds|finds nothing)$/i.test(val); + const pred: Predicate = isPhrase + ? parsePredicate(val, ln.lineNo) + : { type: "command", command: val.replace(/^"|"$/g, ""), expect: "exit-zero" }; + (loop.doneWhen ??= []).push(pred); + i++; continue; + } if (/^allow\b/i.test(t) || /^ask me before\b/i.test(t)) { mergePolicy(loop, t); i++; continue; @@ -340,6 +409,21 @@ function interpretLoopBody(name: string | null, body: Line[]): LoopBodyResult { .filter((s) => s.length > 0); i++; continue; } + // MCP: name servers whose tools the loop may use (`use tools from the "github" server`). + if ((m = t.match(/^use tools from (?:the\s+)?"([^"]+)"(?:\s+server)?$/i))) { + (loop.tools ??= []).push(m[1].trim()); + i++; continue; + } + // The 6th context part — reference patterns to imitate. + if ((m = t.match(/^examples?:\s*(.+)$/i))) { + (loop.context ??= {}).examples = splitItems(m[1]); + i++; continue; + } + // Read-only reference material the agent must not edit. + if ((m = t.match(/^knowledge:\s*(.+)$/i))) { + (loop.context ??= {}).knowledge = splitItems(m[1]); + i++; continue; + } if ((m = t.match(/^(?:remember|keep a memory)\s+in\s+"([^"]+)"$/i))) { loop.memory = { file: m[1].trim() }; i++; continue; @@ -376,10 +460,30 @@ function interpretLoopBody(name: string | null, body: Line[]): LoopBodyResult { } if (!sawGoal) throw new ParseError(`loop "${name ?? "(anonymous)"}" is missing a goal`, body[0]?.lineNo ?? 0); - if (!sawCycle) loop.cycle = ["plan", "act", "observe"]; // sensible default + // Cascade (lowest wins): per-loop `each cycle:` > config-tier default > built-in plan/act/observe. + if (!sawCycle) loop.cycle = defaults?.cycle?.length ? [...defaults.cycle] : ["plan", "act", "observe"]; + + // rigor: structured/agentic loops are born with a back-edge and a thrash guard unless the + // author set their own — the "sensible defaults" that keep a loop safe without boilerplate. + // `vibe coding` (and no rigor at all) injects nothing, so existing files are unchanged. + if (defaults?.rigor === "structured ai-assisted" || defaults?.rigor === "agentic engineering") { + const has = (on: Transition["on"]) => (loop.transitions ?? []).some((t) => t.on === on); + if (loop.doneWhen?.length && !has("fail")) { + (loop.transitions ??= []).push({ on: "fail", do: [{ action: "reflect" }, { action: "plan" }] }); + } + if (has("fail") && !has("attempts")) { + (loop.transitions ??= []).push({ on: "attempts", threshold: 8, do: [{ action: "stop", warn: "thrashing" }] }); + } + } + return { loop, gate }; } +/** Split a comma/"and"-separated list into trimmed items (shared by examples:/knowledge:). */ +function splitItems(s: string): string[] { + return s.split(",").map((x) => x.replace(/^and\s+/i, "").trim()).filter(Boolean); +} + function parseContext(s: string): LoopContext { const ctx: LoopContext = {}; const items = s.split(",").map((p) => p.trim()).filter(Boolean); @@ -413,7 +517,7 @@ function mergePolicy(loop: Loop, line: string) { // ---------- pipeline / stage ---------- -function parseStage(lines: Line[], start: number): { stage: Stage; next: number } { +function parseStage(lines: Line[], start: number, defaults?: ParseDefaults): { stage: Stage; next: number } { const header = lines[start]; const m = header.text.match(/^stage\s+(.+?):\s*$/i); if (!m) throw new ParseError(`expected "stage :"`, header.lineNo); @@ -422,22 +526,40 @@ function parseStage(lines: Line[], start: number): { stage: Stage; next: number const name = quoted(raw) ?? raw; const { body, next } = childrenOf(lines, start + 1, header.indent); if (body.length === 0) throw new ParseError(`stage "${name}" has no body`, header.lineNo); - const { loop, gate } = interpretLoopBody(null, body); + const { loop, gate } = interpretLoopBody(null, body, defaults); return { stage: { name, gate, loop }, next }; } -function parsePipeline(lines: Line[], start: number): { pipeline: Pipeline; next: number } { +function parsePipeline(lines: Line[], start: number, defaults?: ParseDefaults): { pipeline: Pipeline; next: number } { const header = lines[start]; const name = quoted(header.text) ?? header.text.replace(/^pipeline\s+/i, "").replace(/:$/, "").trim(); const { body, next } = childrenOf(lines, start + 1, header.indent); const stages: Stage[] = []; let i = 0; + let group = 0; // body is a slice; re-walk by indentation relative to the stage headers while (i < body.length) { + // `stages in parallel:` — the indented stages beneath it share a parallel-group id. + if (/^stages in parallel:?$/i.test(body[i].text)) { + group++; + const groupIndent = body[i].indent; + let j = i + 1; + while (j < body.length && body[j].indent > groupIndent) { + if (!/^stage\b/i.test(body[j].text)) { + throw new ParseError(`expected a "stage" inside the parallel group in pipeline "${name}"`, body[j].lineNo); + } + const { stage, next: sn } = parseStage(body, j, defaults); + stage.parallelGroup = group; + stages.push(stage); + j = sn; + } + i = j; + continue; + } if (!/^stage\b/i.test(body[i].text)) { throw new ParseError(`expected a "stage" inside pipeline "${name}"`, body[i].lineNo); } - const { stage, next: sn } = parseStage(body, i); + const { stage, next: sn } = parseStage(body, i, defaults); stages.push(stage); i = sn; } @@ -445,11 +567,11 @@ function parsePipeline(lines: Line[], start: number): { pipeline: Pipeline; next return { pipeline: { kind: "pipeline", name, stages }, next }; } -function parseLoopDef(lines: Line[], start: number): { loop: Loop; next: number } { +function parseLoopDef(lines: Line[], start: number, defaults?: ParseDefaults): { loop: Loop; next: number } { const header = lines[start]; const name = quoted(header.text); const { body, next } = childrenOf(lines, start + 1, header.indent); - const { loop } = interpretLoopBody(name, body); + const { loop } = interpretLoopBody(name, body, defaults); return { loop, next }; } @@ -544,28 +666,57 @@ function parseConfigLine(config: Config, ln: Line): boolean { config.notify = m[1].trim(); return true; } + if ((m = t.match(/^rigor:\s*(.+)$/i))) { + const level = m[1].trim().toLowerCase(); + if (!RIGOR_LEVELS.includes(level as Rigor)) { + throw new ParseError(`unknown rigor "${m[1].trim()}" (expected: ${RIGOR_LEVELS.join(", ")})`, ln.lineNo); + } + config.rigor = level as Rigor; + return true; + } + if ((m = t.match(/^mode:\s*(conductor|orchestrator)$/i))) { + config.mode = m[1].trim().toLowerCase() as "conductor" | "orchestrator"; + return true; + } + if ((m = t.match(/^runs as:\s*(.+)$/i))) { + config.runsAs = m[1].trim(); + return true; + } return false; } // ---------- entry ---------- -export function parse(src: string): LoopFile { +/** + * Parse a `.loop` source into a LoopFile. + * + * `opts.defaultCycle` seeds the cycle cascade from outside the file — e.g. a + * project-level `loop.config`. Precedence (lowest wins): a per-loop `each cycle:` + * > the file's config-tier `each cycle:` > `opts.defaultCycle` > built-in plan/act/observe. + */ +export function parse(src: string, opts?: { defaultCycle?: CycleStep[]; defaultRigor?: Rigor }): LoopFile { const lines = tokenizeLines(src); const config: Config = {}; const definitions: Definition[] = []; let i = 0; + // Defaults threaded into every definition (config tier wins over the external project defaults). + const defaults = (): ParseDefaults => ({ + cycle: config.cycle ?? opts?.defaultCycle, + rigor: config.rigor ?? opts?.defaultRigor, + }); while (i < lines.length) { const ln = lines[i]; + let m: RegExpMatchArray | null; if (ln.indent !== 0) { throw new ParseError(`unexpected indentation at top level: "${ln.text}"`, ln.lineNo); } if (/^loop\b/i.test(ln.text)) { - const { loop, next } = parseLoopDef(lines, i); + const { loop, next } = parseLoopDef(lines, i, defaults()); definitions.push(loop); i = next; } else if (/^pipeline\b/i.test(ln.text)) { - const { pipeline, next } = parsePipeline(lines, i); + const { pipeline, next } = parsePipeline(lines, i, defaults()); definitions.push(pipeline); i = next; } else if (/^flow\b/i.test(ln.text)) { @@ -579,6 +730,40 @@ export function parse(src: string): LoopFile { } else if (/^models:\s*.+$/i.test(ln.text)) { config.models = parseModelsLine(ln.text.replace(/^models:\s*/i, ""), ln.lineNo); i++; + } else if (/^observe:?$/i.test(ln.text)) { + const obs: import("./types.js").ObservePolicy = {}; + let j = i + 1; + while (j < lines.length && lines[j].indent > ln.indent) { + const o = lines[j].text; + if (/^trace( every cycle)?$/i.test(o)) obs.trace = true; + else if (/^meter (tokens and cost|cost|tokens)$/i.test(o)) obs.meter = true; + else if ((m = o.match(/^stop and warn if cost exceeds\s+"?([^"]+)"?$/i))) obs.costCap = m[1].trim(); + else throw new ParseError(`unrecognized observe line "${o}"`, lines[j].lineNo); + j++; + } + config.observe = obs; + i = j; + } else if (/^sandbox:?$/i.test(ln.text)) { + const sb: import("./types.js").SandboxPolicy = {}; + let j = i + 1; + while (j < lines.length && lines[j].indent > ln.indent) { + const o = lines[j].text; + if (/^no network( access)?$/i.test(o)) sb.network = "none"; + else if ((m = o.match(/^allow egress to\s+"([^"]+)"(?:\s+only)?$/i))) { sb.network = "allowlist"; (sb.egress ??= []).push(m[1]); } + else if (/^cap\b/i.test(o)) { + const cpu = o.match(/cpu (?:at )?([\w. ]+?)(?:,|$| cores)/i); if (cpu) sb.cpu = cpu[1].trim(); + const mem = o.match(/memory (?:at )?([\w.]+)/i); if (mem) sb.memory = mem[1].trim(); + const time = o.match(/time (?:at )?([\w.]+)/i); if (time) sb.time = time[1].trim(); + } else if (/^cannot reach\b/i.test(o)) { /* informational — accepted */ } + else throw new ParseError(`unrecognized sandbox line "${o}"`, lines[j].lineNo); + j++; + } + config.sandbox = sb; + i = j; + } else if ((m = ln.text.match(/^each cycle:\s*(.+)$/i))) { + // Config-tier default cycle — applies to every loop without its own `each cycle:`. + config.cycle = parseCycle(m[1], ln.lineNo); + i++; } else if (parseConfigLine(config, ln)) { i++; } else { diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index eb3e522..ca0d157 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -21,6 +21,42 @@ export interface GitPolicy { openPr?: boolean; } +/** + * Where a loop sits on the vibe-coding → agentic-engineering spectrum. A config-tier dial + * that expands to bundled defaults (reflect-on-fail, a thrash guard) over every loop in the + * file. `vibe coding` = no injected defaults (fast/disposable); `structured ai-assisted` and + * `agentic engineering` = born with a back-edge + a guard unless the loop sets its own. + */ +export type Rigor = "vibe coding" | "structured ai-assisted" | "agentic engineering"; + +/** Run-isolation policy (`sandbox:` block): where code runs and what it cannot reach. */ +export interface SandboxPolicy { + /** Network posture: "none" (no egress) or "allowlist" (only `egress` hosts). */ + network?: "none" | "allowlist"; + /** Allowed egress hosts when network is "allowlist". */ + egress?: string[]; + /** Resource caps, as written (e.g. cpu "2 cores", memory "4g", time "10m"). */ + cpu?: string; + memory?: string; + time?: string; +} + +/** Observability policy (`observe:` block): trace, cost metering, and an optional spend cap. */ +export interface ObservePolicy { + /** Emit a per-cycle trace and a stop-time OpEx report. */ + trace?: boolean; + /** Meter tokens and cost per cycle (requires a provider that reports usage). */ + meter?: boolean; + /** Stop and warn if spend exceeds this (e.g. "$5"). Enforced when the runner reports cost. */ + costCap?: string; +} + +/** External defaults threaded into loop parsing (from the config tier or a project loop.config). */ +export interface ParseDefaults { + cycle?: CycleStep[]; + rigor?: Rigor; +} + export type ModelTier = "fast" | "strong"; export type ModelPhase = "plan" | "act" | "reflect" | "also"; export interface ModelPolicy { @@ -37,6 +73,22 @@ export interface Config { notify?: string; git?: GitPolicy; models?: ModelPolicy; + /** + * File-level default cycle (`each cycle: …` at the config tier). Applies to every + * loop that doesn't declare its own `each cycle:`. A per-loop `each cycle:` overrides + * it; with neither, the built-in default is plan → act → observe. Lowest wins, like git. + */ + cycle?: CycleStep[]; + /** The spectrum dial (`rigor: …`) — bundles best-practice defaults over every loop in the file. */ + rigor?: Rigor; + /** Supervision posture (`mode: …`): conductor = in-session/synchronous; orchestrator = async/reviews-outcomes. */ + mode?: "conductor" | "orchestrator"; + /** Observability (`observe:` block): trace + cost metering + an optional spend cap. */ + observe?: ObservePolicy; + /** Run isolation (`sandbox:` block): where code runs and what it cannot reach. */ + sandbox?: SandboxPolicy; + /** Identity the run executes as (`runs as: …`) — an auditable principal for unattended runs. */ + runsAs?: string; } export interface OverrideEntry { @@ -55,13 +107,20 @@ export interface Stage { name: string; gate?: { message: string } | null; loop: Loop; + /** Stages sharing a parallel-group id run concurrently (`stages in parallel:`). */ + parallelGroup?: number; } export interface Loop { kind: "loop"; name: string | null; goal: string; - doneWhen?: Predicate | null; + /** + * Verification conditions — ALL must pass (a conjunction). A test/command predicate is a + * deterministic check (a TEST); a skill predicate is an eval (an EVAL). + * Each `done when …` line appends one. Empty/absent = no machine check (stop via a human path). + */ + doneWhen?: Predicate[]; context?: LoopContext; policy?: Policy; cycle: CycleStep[]; @@ -69,9 +128,13 @@ export interface Loop { also?: string[]; /** Named execution skills the loop may invoke during plan/act (e.g. ["check-weather"]). */ skills?: string[]; + /** MCP servers whose tools the loop may use (`use tools from the "github" server`). */ + tools?: string[]; /** A markdown file the loop reads on start and appends to on stop — cross-run memory. */ memory?: LoopMemory; planSource?: PlanSource; + /** Deterministic checks bound to lifecycle points (`hooks:`). A failing hook blocks. */ + hooks?: Hook[]; humanPlan?: boolean; humanReviewBeforeStop?: boolean; transitions?: Transition[]; @@ -81,10 +144,23 @@ export interface Loop { export type CycleStep = "plan" | "act" | "observe"; +/** A lifecycle point a `hooks:` check binds to. */ +export type HookPoint = "before-cycle" | "after-plan" | "after-act" | "after-observe" | "on-commit" | "on-push" | "on-stop"; + +/** A deterministic check bound to a lifecycle point — a failing hook blocks. */ +export interface Hook { + at: HookPoint; + predicate: Predicate; +} + export interface LoopContext { files?: string[]; docs?: string[]; includeLastFailure?: boolean; + /** Read-only reference material (`knowledge:`) — docs/diagrams the agent reads but must not edit. */ + knowledge?: string[]; + /** Reference patterns to imitate (`examples:`) — "build it like these". */ + examples?: string[]; } export interface Policy { @@ -112,7 +188,13 @@ export type Predicate = | { type: "test"; target: string } | { type: "command"; command: string; expect?: "exit-zero" | "empty" } | { type: "human"; description: string } - | { type: "skill"; skill: string; expect: "approve"; minScore?: number }; + /** + * An eval: a review skill judges the goal (approve / score). `subject` selects what it + * inspects — the produced `output` (default) or the `trajectory` (the path and tool calls + * the agent took to get there). `bar` is an optional inline rubric (`the bar:`) naming the + * conditions the judge scores against. + */ + | { type: "skill"; skill: string; expect: "approve"; minScore?: number; subject?: "output" | "trajectory"; bar?: string }; export interface Transition { on: "pass" | "fail" | "blocked" | "attempts"; diff --git a/packages/parser/test/parser.test.js b/packages/parser/test/parser.test.js index 18869d9..449b79f 100644 --- a/packages/parser/test/parser.test.js +++ b/packages/parser/test/parser.test.js @@ -17,7 +17,7 @@ test("fix_test: minimal loop", () => { const loop = file.definitions[0]; assert.equal(loop.kind, "loop"); assert.equal(loop.name, "fix test"); - assert.deepEqual(loop.doneWhen, { type: "test", target: "checkout.spec.ts::tax" }); + assert.deepEqual(loop.doneWhen, [{ type: "test", target: "checkout.spec.ts::tax" }]); assert.deepEqual(loop.cycle, ["plan", "act", "observe"]); assert.equal(loop.transitions.length, 1); assert.equal(loop.transitions[0].on, "fail"); @@ -58,11 +58,11 @@ test("ship_feature: pipeline with human nodes + gate", () => { assert.deepEqual(pipe.stages.map((s) => s.name), ["security", "build", "ui", "deploy"]); const security = pipe.stages[0]; - assert.deepEqual(security.loop.doneWhen, { type: "command", command: "semgrep --severity=high", expect: "empty" }); + assert.deepEqual(security.loop.doneWhen, [{ type: "command", command: "semgrep --severity=high", expect: "empty" }]); const build = pipe.stages[1]; assert.equal(build.loop.humanPlan, true); - assert.deepEqual(build.loop.doneWhen, { type: "command", command: "pnpm test", expect: "exit-zero" }); + assert.deepEqual(build.loop.doneWhen, [{ type: "command", command: "pnpm test", expect: "exit-zero" }]); const ui = pipe.stages[2]; assert.equal(ui.loop.humanReviewBeforeStop, true); @@ -197,7 +197,7 @@ test("skills + memory: use skills, remember, and a skill predicate", () => { assert.equal(loop.name, "decide whether to cancel the morning run"); assert.deepEqual(loop.skills, ["check-weather", "analyze-workout"]); assert.deepEqual(loop.memory, { file: "morning-run.memory.md" }); - assert.deepEqual(loop.doneWhen, { type: "skill", skill: "workout-review", expect: "approve" }); + assert.deepEqual(loop.doneWhen, [{ type: "skill", skill: "workout-review", expect: "approve" }]); }); test("skill predicate: scores N or more carries a minScore", () => { @@ -205,7 +205,7 @@ test("skill predicate: scores N or more carries a minScore", () => { const loop = file.definitions[0]; assert.deepEqual(loop.skills, ["write-email"]); assert.deepEqual(loop.memory, { file: "launch-email.memory.md" }); - assert.deepEqual(loop.doneWhen, { type: "skill", skill: "email-review", expect: "approve", minScore: 8 }); + assert.deepEqual(loop.doneWhen, [{ type: "skill", skill: "email-review", expect: "approve", minScore: 8 }]); }); test("use skills: also accepts 'and' as a separator", () => { @@ -217,3 +217,188 @@ test("memory: 'keep a memory in' is an accepted alias", () => { const loop = parse('loop "x":\n goal: g\n keep a memory in "notes.md"').definitions[0]; assert.deepEqual(loop.memory, { file: "notes.md" }); }); + +test("config-tier 'each cycle:' sets the default for loops without their own", () => { + const file = parse( + 'each cycle: act, then observe\n\nloop "x":\n goal: g\n done when "t" passes' + ); + assert.deepEqual(file.config.cycle, ["act", "observe"]); + // the loop inherits the config-tier default + assert.deepEqual(file.definitions[0].cycle, ["act", "observe"]); +}); + +test("per-loop 'each cycle:' overrides the config-tier default", () => { + const file = parse( + 'each cycle: act, then observe\n\n' + + 'loop "inherits":\n goal: g\n done when "t" passes\n\n' + + 'loop "overrides":\n goal: g\n each cycle: plan, then act, then observe\n done when "t" passes' + ); + assert.deepEqual(file.definitions[0].cycle, ["act", "observe"]); // inherits + assert.deepEqual(file.definitions[1].cycle, ["plan", "act", "observe"]); // overrides +}); + +test("config-tier default cycle applies to pipeline stages too", () => { + const file = parse( + 'each cycle: act, then observe\n\n' + + 'pipeline "p":\n stage "a":\n goal: g\n done when "t" passes\n' + + ' stage "b":\n goal: g\n each cycle: plan, then observe\n done when "t" passes' + ); + const [a, b] = file.definitions[0].stages; + assert.deepEqual(a.loop.cycle, ["act", "observe"]); // inherits config default + assert.deepEqual(b.loop.cycle, ["plan", "observe"]); // stage override wins +}); + +test("no config default → built-in plan/act/observe is preserved", () => { + const loop = parse('loop "x":\n goal: g\n done when "t" passes').definitions[0]; + assert.deepEqual(loop.cycle, ["plan", "act", "observe"]); +}); + +test("config-tier default cycle is independent per loop (no shared reference)", () => { + const file = parse( + 'each cycle: act, then observe\n\n' + + 'loop "a":\n goal: g\n done when "t" passes\n\n' + + 'loop "b":\n goal: g\n done when "t" passes' + ); + assert.notEqual(file.definitions[0].cycle, file.definitions[1].cycle); // distinct arrays + assert.deepEqual(file.definitions[0].cycle, file.definitions[1].cycle); // same contents +}); + +test("parse(opts.defaultCycle): external default seeds the cascade", () => { + // project-level default, file has no config cycle, loop has no cycle → inherits external default + const inherits = parse('loop "x":\n goal: g\n done when "t" passes', { + defaultCycle: ["act", "observe"], + }).definitions[0]; + assert.deepEqual(inherits.cycle, ["act", "observe"]); + + // the file's own config-tier cycle still wins over the external default + const fileWins = parse('each cycle: plan, then observe\n\nloop "x":\n goal: g\n done when "t" passes', { + defaultCycle: ["act", "observe"], + }).definitions[0]; + assert.deepEqual(fileWins.cycle, ["plan", "observe"]); +}); + +test("evals: multiple done-when lines form a conjunction", () => { + const loop = parse( + 'loop "x":\n goal: g\n' + + ' done when "pnpm test" passes\n' + + ' done when the skill "review" scores 8 or more on the output' + ).definitions[0]; + assert.equal(loop.doneWhen.length, 2); + assert.deepEqual(loop.doneWhen[0], { type: "command", command: "pnpm test", expect: "exit-zero" }); + assert.deepEqual(loop.doneWhen[1], { type: "skill", skill: "review", expect: "approve", minScore: 8, subject: "output" }); +}); + +test("evals: 'on the trajectory' selects the subject; default omits it", () => { + const traj = parse('loop "x":\n goal: g\n done when the skill "path" approves on the trajectory').definitions[0]; + assert.deepEqual(traj.doneWhen[0], { type: "skill", skill: "path", expect: "approve", subject: "trajectory" }); + const def = parse('loop "x":\n goal: g\n done when the skill "out" approves').definitions[0]; + assert.deepEqual(def.doneWhen[0], { type: "skill", skill: "out", expect: "approve" }); // no subject = output +}); + +test("evals: an indented 'the bar:' attaches a rubric to a skill eval", () => { + const loop = parse( + 'loop "x":\n goal: g\n' + + ' done when the skill "path" approves on the trajectory\n' + + ' the bar: did not weaken a test to go green' + ).definitions[0]; + assert.deepEqual(loop.doneWhen[0], { + type: "skill", skill: "path", expect: "approve", subject: "trajectory", + bar: "did not weaken a test to go green", + }); +}); + +test("evals: 'the bar:' on a non-skill predicate is a parse error", () => { + assert.throws( + () => parse('loop "x":\n goal: g\n done when "pnpm test" passes\n the bar: nope'), + /the bar:/i + ); +}); + +test("friendly sugar: check:/verify:, in:/files:, when it breaks", () => { + const loop = parse( + 'loop "x":\n goal: g\n in: src/cart, src/tax\n check: pnpm test cart\n when it breaks: reflect, then plan again' + ).definitions[0]; + assert.deepEqual(loop.context.files, ["src/cart", "src/tax"]); + assert.deepEqual(loop.doneWhen, [{ type: "command", command: "pnpm test cart", expect: "exit-zero" }]); + assert.equal(loop.transitions[0].on, "fail"); +}); + +test("friendly sugar: check: also accepts a predicate phrase", () => { + const a = parse('loop "x":\n goal: g\n check: the skill "review" approves on the trajectory').definitions[0]; + assert.deepEqual(a.doneWhen, [{ type: "skill", skill: "review", expect: "approve", subject: "trajectory" }]); + const b = parse('loop "x":\n goal: g\n files: a.ts, b.ts').definitions[0]; + assert.deepEqual(b.context.files, ["a.ts", "b.ts"]); +}); + +test("rigor: agentic engineering injects a back-edge + thrash guard when omitted", () => { + const loop = parse('rigor: agentic engineering\n\nloop "x":\n goal: g\n check: npm test').definitions[0]; + assert.equal(parse('rigor: agentic engineering\n\nloop "x":\n goal: g\n check: npm test').config.rigor, "agentic engineering"); + const fail = loop.transitions.find((t) => t.on === "fail"); + const guard = loop.transitions.find((t) => t.on === "attempts"); + assert.ok(fail, "injected a reflect-on-fail back-edge"); + assert.deepEqual(fail.do, [{ action: "reflect" }, { action: "plan" }]); + assert.ok(guard && guard.threshold === 8, "injected a default thrash guard"); +}); + +test("rigor: vibe coding injects nothing; author transitions are not double-added", () => { + const vibe = parse('rigor: vibe coding\n\nloop "x":\n goal: g\n check: npm test').definitions[0]; + assert.deepEqual(vibe.transitions ?? [], []); + const own = parse('rigor: agentic engineering\n\nloop "x":\n goal: g\n check: npm test\n after 3 tries: stop and warn "mine"').definitions[0]; + const guards = (own.transitions ?? []).filter((t) => t.on === "attempts"); + assert.equal(guards.length, 1, "did not add a second guard"); + assert.equal(guards[0].threshold, 3, "kept the author's guard"); +}); + +test("rigor: an unknown level is a parse error; mode: parses", () => { + assert.throws(() => parse('rigor: yolo\n\nloop "x":\n goal: g'), /unknown rigor/i); + assert.equal(parse('mode: orchestrator\n\nloop "x":\n goal: g').config.mode, "orchestrator"); +}); + +test("hooks: a hooks block parses into lifecycle-bound checks", () => { + const loop = parse( + 'loop "x":\n goal: g\n check: npm test\n hooks:\n before each cycle: "tsc" passes\n on commit: "semgrep" finds nothing' + ).definitions[0]; + assert.deepEqual(loop.hooks, [ + { at: "before-cycle", predicate: { type: "command", command: "tsc", expect: "exit-zero" } }, + { at: "on-commit", predicate: { type: "command", command: "semgrep", expect: "empty" } }, + ]); +}); + +test("hooks: a non-deterministic hook predicate is rejected", () => { + assert.throws(() => parse('loop "x":\n goal: g\n hooks:\n on commit: the skill "x" approves'), /deterministic/i); +}); + +test("observe: a config-tier observe block parses trace/meter/cost cap", () => { + const file = parse('observe:\n trace every cycle\n meter tokens and cost\n stop and warn if cost exceeds "$5"\n\nloop "x":\n goal: g\n check: npm test'); + assert.deepEqual(file.config.observe, { trace: true, meter: true, costCap: "$5" }); +}); + +test("sandbox + runs as: config-tier isolation and identity", () => { + const file = parse( + 'sandbox:\n no network access\n allow egress to "registry.npmjs.org" only\n cap cpu at 2 cores, memory at 4g, time at 10m\n cannot reach the host filesystem\nruns as: ci-bot\n\nloop "x":\n goal: g\n check: npm test' + ); + assert.equal(file.config.sandbox.network, "allowlist"); + assert.deepEqual(file.config.sandbox.egress, ["registry.npmjs.org"]); + assert.equal(file.config.sandbox.memory, "4g"); + assert.equal(file.config.sandbox.time, "10m"); + assert.equal(file.config.runsAs, "ci-bot"); +}); + +test("knowledge/examples/tools: context + MCP keywords", () => { + const loop = parse( + 'loop "x":\n goal: g\n knowledge: docs/api.md, the architecture diagram\n examples: routes/payments.ts\n use tools from the "github" server\n check: npm test' + ).definitions[0]; + assert.deepEqual(loop.context.knowledge, ["docs/api.md", "the architecture diagram"]); + assert.deepEqual(loop.context.examples, ["routes/payments.ts"]); + assert.deepEqual(loop.tools, ["github"]); +}); + +test("parallel stages: 'stages in parallel:' assigns a shared group id", () => { + const pipe = parse( + 'pipeline "p":\n stage "a":\n goal: g\n check: t\n stages in parallel:\n stage "b":\n goal: g\n check: t\n stage "c":\n goal: g\n check: t' + ).definitions[0]; + assert.equal(pipe.stages.length, 3); + assert.equal(pipe.stages[0].parallelGroup, undefined); + assert.equal(pipe.stages[1].parallelGroup, 1); + assert.equal(pipe.stages[2].parallelGroup, 1); +}); diff --git a/packages/runtime/src/cli.ts b/packages/runtime/src/cli.ts index 9eafab3..3eae2a3 100644 --- a/packages/runtime/src/cli.ts +++ b/packages/runtime/src/cli.ts @@ -1,8 +1,9 @@ #!/usr/bin/env node -import { appendFileSync, readFileSync, writeFileSync, readdirSync } from "node:fs"; +import { appendFileSync, readFileSync, writeFileSync, readdirSync, existsSync } from "node:fs"; import { dirname, resolve, relative, basename } from "node:path"; import { createInterface } from "node:readline"; import { parse } from "@loop-lang/parser"; +import type { Config, LoopFile } from "@loop-lang/parser"; import { resolvePreset } from "@loop-lang/stdlib"; import { run } from "./engine.js"; import { ShellVerifier } from "./verify.js"; @@ -11,7 +12,7 @@ import { ClaudeCodeRunner } from "./runners/claudeCode.js"; import { IpcHumanIO } from "./ipc.js"; import { ShellGitIO } from "./runners/shellGit.js"; import type { LoopEvent } from "./types.js"; -import { summarizeModels, formatModelSummary } from "./summary.js"; +import { summarizeModels, formatModelSummary, summarizeOpex, formatOpexSummary } from "./summary.js"; const GLYPH: Partial> = { "pipeline-start": "▶ pipeline", @@ -85,6 +86,54 @@ function render(e: LoopEvent): string { } } +/** Project-level config filenames, in priority order. Written in config-tier `.loop` syntax. */ +const PROJECT_CONFIG_NAMES = ["loop.config", ".looprc"]; + +/** + * Walk up from `startDir` looking for a project config file (`loop.config` / `.looprc`). + * It is parsed with the same grammar as a `.loop` file but holds only a config tier + * (e.g. `each cycle:`, `models:`, a `git:` block). Returns its config, or null. + */ +function findProjectConfig(startDir: string): { path: string; config: Config } | null { + let dir = startDir; + while (true) { + for (const name of PROJECT_CONFIG_NAMES) { + const p = resolve(dir, name); + if (existsSync(p)) { + try { + const cfg = parse(readFileSync(p, "utf8")).config; + if (cfg) return { path: p, config: cfg }; + } catch (err) { + console.error(`⚠ ignoring ${p}: ${String((err as Error)?.message ?? err)}`); + } + return null; // a config file exists here but is empty/invalid — stop the search + } + } + const parent = dirname(dir); + if (parent === dir) return null; // reached the filesystem root + dir = parent; + } +} + +/** File config wins over project defaults (lowest tier). Shallow per key — a `git:`/`models:` block in the file replaces the project's. */ +function withProjectDefaults(fileConfig: Config | null, project: Config): Config { + return { ...project, ...(fileConfig ?? {}) }; +} + +/** + * Read and parse a `.loop` file, folding in any project-level `loop.config` as the lowest + * cascade tier. `cycle` resolves at parse time (so it is passed in as `defaultCycle`); the + * remaining config keys resolve at runtime and are merged onto the result. + */ +function loadLoopFile(absPath: string): LoopFile { + const project = findProjectConfig(dirname(absPath)); + const file = parse(readFileSync(absPath, "utf8"), { + defaultCycle: project?.config.cycle, + defaultRigor: project?.config.rigor, + }); + return project ? { ...file, config: withProjectDefaults(file.config, project.config) } : file; +} + /** Walk a dir for *.loop files, skipping noise. */ function findLoops(dir: string, acc: string[] = [], depth = 0): string[] { if (depth > 6) return acc; @@ -117,7 +166,14 @@ async function main() { if (cmd === "show") { if (!fileArg) { console.error("usage: loop-run show "); process.exit(2); } const { renderFile } = await import("./show.js"); - console.log(renderFile(parse(readFileSync(resolve(process.cwd(), fileArg), "utf8")))); + console.log(renderFile(loadLoopFile(resolve(process.cwd(), fileArg)))); + return; + } + // `loop explain ` — describe the loop in plain English (the friendly view). + if (cmd === "explain") { + if (!fileArg) { console.error("usage: loop-run explain "); process.exit(2); } + const { explainFile } = await import("./show.js"); + console.log(explainFile(loadLoopFile(resolve(process.cwd(), fileArg)))); return; } // `loop ls` — list every .loop under the current dir with its one-line shape. @@ -159,7 +215,7 @@ async function main() { } if (!cmd || (cmd !== "run" && cmd !== "parse" && cmd !== "viz" && cmd !== "live") || !fileArg) { - console.error("usage: loop-run [--model ] [--live] [--events] [--out ]"); + console.error("usage: loop-run [--model ] [--live] [--events] [--out ]"); process.exit(2); } @@ -171,8 +227,7 @@ async function main() { Promise.resolve(readFileSync(resolve(dir, ref), "utf8")); const writeText = (ref: string, content: string, dir: string) => Promise.resolve(appendFileSync(resolve(dir, ref), content, "utf8")); - const src = readFileSync(path, "utf8"); - let file = parse(src); + let file = loadLoopFile(path); // If the file is a config that selects a preset, resolve and run the preset. if (file.definitions.length === 0 && file.config?.use) { @@ -302,6 +357,10 @@ async function main() { const outcomes = await run(file, { ...baseOpts, onEvent: onTrace }); reportSummary(); + // Observability: when `observe:` is on, print the OpEx report (token burn made visible). + if (file.config?.observe?.trace || file.config?.observe?.meter) { + console.error(formatOpexSummary(summarizeOpex(traceEvents))); + } const ok = outcomes.every((o) => o.satisfied); process.exit(ok ? 0 : 1); } diff --git a/packages/runtime/src/engine.ts b/packages/runtime/src/engine.ts index bd96c99..d815174 100644 --- a/packages/runtime/src/engine.ts +++ b/packages/runtime/src/engine.ts @@ -1,5 +1,5 @@ import { resolve, dirname, basename } from "node:path"; -import type { Loop, Pipeline, Flow, FlowStep, Definition, Transition, Action, LoopFile } from "@loop-lang/parser"; +import type { Loop, Pipeline, Stage, Flow, FlowStep, Definition, Transition, Action, LoopFile, HookPoint } from "@loop-lang/parser"; import type { CycleNode, LoopEvent, LoopOutcome, RunOptions, StopReason } from "./types.js"; import { enumerateItems, labelOf } from "./iterate.js"; import { resolveGit, isProtected } from "./git.js"; @@ -58,7 +58,12 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { const eff = resolveModels(opts.modelPolicy, loop.models); const pick = (phase: "plan" | "act" | "reflect" | "also") => modelForPhase(eff, phase, opts.cliModel); - const files = loop.context?.files ?? []; + // The agent reads files it may edit plus read-only knowledge and reference examples. + const files = [ + ...(loop.context?.files ?? []), + ...(loop.context?.knowledge ?? []).map((k) => `${k} (read-only reference)`), + ...(loop.context?.examples ?? []).map((e) => `${e} (a pattern to imitate)`), + ]; const includeLastFailure = loop.context?.includeLastFailure ?? false; const autoClasses = loop.policy?.auto ?? []; const confirmClasses = loop.policy?.confirm ?? []; @@ -67,6 +72,8 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { let lastPlan = ""; let lastOutput = ""; let lastActSummary = ""; + // The captured path of the most recent act — the steps/tool calls a trajectory eval judges. + let lastTrajectory = ""; let attempts = 0; // Reflections gathered this run — the last one becomes the memory "lesson". const reflections: string[] = []; @@ -118,11 +125,25 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { ? ["plan", ...(loop.cycle as CycleNode[])] : (loop.cycle as CycleNode[]); + // Run every hook bound to a lifecycle point. Returns false if any fails (a hook blocks). + const runHooks = async (at: HookPoint): Promise => { + let ok = true; + for (const h of loop.hooks ?? []) { + if (h.at !== at) continue; + const r = await opts.verifier.verify(h.predicate, opts.baseDir); + const label = h.predicate.type === "command" ? h.predicate.command : h.predicate.type === "test" ? h.predicate.target : h.predicate.type; + emit(opts, { type: "hook", at, detail: label, passed: r.passed }); + if (!r.passed) ok = false; + } + return ok; + }; + const finish = async (satisfied: boolean, reason: StopReason, warn?: string): Promise => { - await writeMemory(satisfied, reason); + const ok = (await runHooks("on-stop")) && satisfied; // an on-stop hook can veto a clean stop + await writeMemory(ok, reason); emit(opts, { type: "stop", reason, ...(warn ? { warn } : {}) }); - emit(opts, { type: "loop-end", name: loop.name, satisfied }); - return { satisfied, reason, attempts, summary: lastOutput }; + emit(opts, { type: "loop-end", name: loop.name, satisfied: ok }); + return { satisfied: ok, reason, attempts, summary: lastOutput }; }; while (true) { @@ -130,11 +151,14 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { if (attempts > hardCap) { return finish(false, "hard-cap"); } + // Lifecycle hook: a failing pre-cycle check blocks before any work this round. + if (!(await runHooks("before-cycle"))) return finish(false, "blocked", "a before-cycle hook failed"); let blocked = false; let passed = false; let observeOutput = ""; let observed = false; + let hookBlocked = false; for (const step of cycleSteps) { emit(opts, { type: "node-enter", node: step, attempt: attempts }); @@ -192,46 +216,64 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { model: pick("act"), }); lastActSummary = res.summary; + lastTrajectory = res.trajectory ?? ""; if (res.blocked) blocked = true; emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: !res.blocked, detail: res.summary }); } else { - // observe - let v: { passed: boolean; output: string }; - if (loop.doneWhen?.type === "skill") { - // Skill predicate: a review skill judges the goal (approved / score) — the - // "bridge the abstract to the verifiable" path, routed to the runner not the shell. - if (!opts.runner.runSkill) { - throw new Error( - `loop "${loop.name ?? ""}" verifies with the skill "${loop.doneWhen.skill}" but the runner has no runSkill` - ); + // observe — every `done when` predicate must pass (a conjunction of TESTS and EVALS). + const preds = loop.doneWhen ?? []; + let allPassed = preds.length > 0; // no predicate → not satisfiable by a machine check + const outputs: string[] = []; + for (const pred of preds) { + let r: { passed: boolean; output: string }; + if (pred.type === "skill") { + // An eval: a review skill judges the goal (approved / score) — the + // "bridge the abstract to the verifiable" path, routed to the runner not the shell. + if (!opts.runner.runSkill) { + throw new Error( + `loop "${loop.name ?? ""}" verifies with the skill "${pred.skill}" but the runner has no runSkill` + ); + } + // A trajectory eval judges HOW the agent got there (the captured path); + // an output eval judges WHAT it produced (the act summary). + const isTrajectory = pred.subject === "trajectory"; + const sr = await opts.runner.runSkill({ + skill: pred.skill, + goal: loop.goal, + context: isTrajectory ? (lastTrajectory || lastActSummary) : lastActSummary, + expect: pred.expect, + minScore: pred.minScore, + subject: pred.subject, + bar: pred.bar, + baseDir: opts.baseDir, + }); + emit(opts, { type: "skill-verify", skill: pred.skill, passed: sr.passed, detail: sr.detail }); + r = { passed: sr.passed, output: sr.detail }; + } else { + r = await opts.verifier.verify(pred, opts.baseDir); } - const r = await opts.runner.runSkill({ - skill: loop.doneWhen.skill, - goal: loop.goal, - context: lastActSummary, - expect: loop.doneWhen.expect, - minScore: loop.doneWhen.minScore, - baseDir: opts.baseDir, - }); - emit(opts, { type: "skill-verify", skill: loop.doneWhen.skill, passed: r.passed, detail: r.detail }); - v = { passed: r.passed, output: r.detail }; - } else { - v = await opts.verifier.verify(loop.doneWhen, opts.baseDir); + if (r.output) outputs.push(r.output); + if (!r.passed) { allPassed = false; break; } // short-circuit: a conjunction fails on the first miss } observed = true; - passed = v.passed; - observeOutput = v.output; - lastOutput = v.output; - emit(opts, { type: "observe", passed: v.passed, output: v.output }); - emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: v.passed }); + passed = allPassed; + observeOutput = outputs.join("\n"); + lastOutput = observeOutput; + emit(opts, { type: "observe", passed: allPassed, output: observeOutput }); + emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: allPassed }); } + // Lifecycle hook: a deterministic check bound to this step. A failure blocks the loop. + const afterPoint: HookPoint = step === "plan" ? "after-plan" : step === "act" ? "after-act" : "after-observe"; + if (!(await runHooks(afterPoint))) { hookBlocked = true; break; } } + if (hookBlocked) return finish(false, "blocked", `a ${"after"}-step hook failed`); if (opts.git && resolveGit(opts.gitPolicy, loop.git).commit === "cycle") { + if (!(await runHooks("on-commit"))) return finish(false, "blocked", "an on-commit hook failed"); await gitCommit(opts, `loop: ${loop.name ?? "cycle"} — cycle ${attempts}`); } - const goalMet = loop.doneWhen ? passed : false; + const goalMet = loop.doneWhen?.length ? passed : false; const t = pickTransition(loop.transitions, { blocked, attempts, passed: observed && passed, goalMet }); @@ -240,6 +282,7 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { const stop = await applyActions(t.do, loop, opts, { goalMet, output: observeOutput, + trajectory: lastTrajectory, setReflection: (r) => { reflection = r; reflections.push(r); @@ -263,7 +306,7 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { // rejected: keep iterating continue; } - if (!loop.doneWhen && loop.humanPlan && humanPlanApproved) { + if (!loop.doneWhen?.length && loop.humanPlan && humanPlanApproved) { // plan-only human loop: one approved pass is the deliverable. return finish(true, "human-approved"); } @@ -279,7 +322,7 @@ async function applyActions( actions: Action[], loop: Loop, opts: RunOptions, - ctx: { goalMet: boolean; output: string; setReflection: (r: string) => void } + ctx: { goalMet: boolean; output: string; trajectory?: string; setReflection: (r: string) => void } ): Promise { for (const a of actions) { switch (a.action) { @@ -302,6 +345,7 @@ async function applyActions( goal: loop.goal, focus: a.focus, output: ctx.output, + trajectory: ctx.trajectory, baseDir: opts.baseDir, model: modelForPhase(rEff, "reflect", opts.cliModel), }); @@ -323,30 +367,53 @@ async function applyActions( return null; } +/** Run one stage: its gate, its loop, and a story-commit on success. */ +async function runStage(stage: Stage, opts: RunOptions): Promise { + emit(opts, { type: "stage-start", name: stage.name }); + if (stage.gate) { + emit(opts, { type: "human", kind: "gate", prompt: stage.gate.message }); + const ok = await opts.human.gate(stage.gate.message); + if (!ok) { + emit(opts, { type: "stage-end", name: stage.name, satisfied: false }); + return { satisfied: false, reason: "blocked", attempts: 0 }; + } + } + const outcome = await executeLoopFull(stage.loop, opts); + emit(opts, { type: "stage-end", name: stage.name, satisfied: outcome.satisfied }); + if (outcome.satisfied && opts.git && resolveGit(opts.gitPolicy).commit === "story") { + await gitCommit(opts, `loop: story "${stage.name}" satisfied`); + } + return outcome; +} + async function executePipeline(pipeline: Pipeline, opts: RunOptions): Promise { emit(opts, { type: "pipeline-start", name: pipeline.name }); let lastAttempts = 0; - for (const stage of pipeline.stages) { - emit(opts, { type: "stage-start", name: stage.name }); - if (stage.gate) { - emit(opts, { type: "human", kind: "gate", prompt: stage.gate.message }); - const ok = await opts.human.gate(stage.gate.message); - if (!ok) { - emit(opts, { type: "stage-end", name: stage.name, satisfied: false }); + const stages = pipeline.stages; + let i = 0; + while (i < stages.length) { + const grp = stages[i].parallelGroup; + // A parallel group (`stages in parallel:`) runs its stages concurrently and barrier-joins. + // NOTE: real file-safe parallelism needs a worktree per branch; concurrent edits to one + // working tree can collide. The orchestration is here; isolation is the operator's setup. + if (grp != null) { + const group: Stage[] = []; + while (i < stages.length && stages[i].parallelGroup === grp) group.push(stages[i++]); + const outcomes = await Promise.all(group.map((s) => runStage(s, opts))); + lastAttempts = outcomes[outcomes.length - 1]?.attempts ?? lastAttempts; + const failed = outcomes.find((o) => !o.satisfied); + if (failed) { emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: false }); - return { satisfied: false, reason: "blocked", attempts: lastAttempts }; + return { satisfied: false, reason: failed.reason, attempts: lastAttempts }; + } + } else { + const outcome = await runStage(stages[i++], opts); + lastAttempts = outcome.attempts; + if (!outcome.satisfied) { + // A failing stage halts the rest of the pipeline. + emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: false }); + return { satisfied: false, reason: outcome.reason, attempts: lastAttempts }; } - } - const outcome = await executeLoopFull(stage.loop, opts); - lastAttempts = outcome.attempts; - emit(opts, { type: "stage-end", name: stage.name, satisfied: outcome.satisfied }); - if (!outcome.satisfied) { - // A failing stage halts the rest of the pipeline. - emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: false }); - return { satisfied: false, reason: outcome.reason, attempts: lastAttempts }; - } - if (opts.git && resolveGit(opts.gitPolicy).commit === "story") { - await gitCommit(opts, `loop: story "${stage.name}" satisfied`); } } emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: true }); diff --git a/packages/runtime/src/runners/claudeCode.ts b/packages/runtime/src/runners/claudeCode.ts index 5caffb5..fe01401 100644 --- a/packages/runtime/src/runners/claudeCode.ts +++ b/packages/runtime/src/runners/claudeCode.ts @@ -144,9 +144,11 @@ function resultText(content: unknown): string { export class ClaudeCodeRunner implements Runner { constructor(private opts: ClaudeCodeRunnerOptions = {}) {} - private run(prompt: string, flags: string[], baseDir: string, node: AgentNode, model?: string): Promise { + private run(prompt: string, flags: string[], baseDir: string, node: AgentNode, model?: string, capture?: string[]): Promise { const bin = this.opts.bin ?? "claude"; - const stream = !!this.opts.onActivity; + // Stream when there's a live display OR a caller capturing the trajectory (tool calls + // are only parsed in stream mode), so trajectory evals work even without `onActivity`. + const stream = !!this.opts.onActivity || !!capture; const args = ["-p", prompt, ...claudeArgs({ stream, baseDir, model: model ?? this.opts.model, maxTurns: this.opts.maxTurns, flags })]; return new Promise((resolve, reject) => { @@ -165,7 +167,7 @@ export class ClaudeCodeRunner implements Runner { const line = buf.slice(0, nl); buf = buf.slice(nl + 1); const r = interpretStreamLine(line); - for (const a of r.activities) this.opts.onActivity!(node, a); + for (const a of r.activities) { this.opts.onActivity?.(node, a); capture?.push(a); } if (r.result !== undefined) { lastResult = r.result; streamed = r.result; @@ -206,14 +208,18 @@ export class ClaudeCodeRunner implements Runner { ...(input.skills?.length ? [`You may use these skills: ${input.skills.join(", ")}.`] : []), `Only use the capabilities you have been granted.`, ].join("\n"); + // Capture the act's trajectory (its tool calls, with inputs) so a trajectory eval can judge + // the path, not just the result. interpretStreamLine already decodes tool_use blocks. + const trajectory: string[] = []; const summary = await this.run( prompt, ["--permission-mode", mode, "--allowedTools", ...tools], input.baseDir, "act", - input.model + input.model, + trajectory ); - return { summary }; + return { summary, trajectory: trajectory.join("\n") }; } async reflect(input: ReflectInput): Promise { @@ -221,6 +227,7 @@ export class ClaudeCodeRunner implements Runner { `Goal: ${input.goal}.`, `The verification did not pass. Output was:`, input.output || "(no output)", + ...(input.trajectory ? [`The path you took this cycle (tool calls):`, input.trajectory] : []), input.focus ? `Reflect specifically on: ${input.focus}.` : `Reflect on why it failed.`, `In 2-4 sentences, explain the likely cause and what to change next. Do not edit files.`, ].join("\n"); @@ -232,11 +239,16 @@ export class ClaudeCodeRunner implements Runner { input.minScore !== undefined ? `End with a line exactly "SCORE: N" where N is 0-10. It passes when N >= ${input.minScore}.` : `End with a line exactly "VERDICT: APPROVED" or "VERDICT: REJECTED".`; + const isTrajectory = input.subject === "trajectory"; + const whatToReview = isTrajectory + ? `The agent's TRAJECTORY — the steps and tool calls it took. Judge HOW it got there (right path, right tools; e.g. it must not have weakened a test to go green), not just the end result:` + : `What to review (the produced OUTPUT):`; const prompt = [ `Use the "${input.skill}" skill to review this against the goal.`, `Goal: ${input.goal}.`, - `What to review:`, - input.context || "(no output captured)", + ...(input.bar ? [`The bar — it passes only if all of these hold: ${input.bar}.`] : []), + whatToReview, + input.context || "(no context captured)", verdictLine, ].join("\n"); const out = await this.run( diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index 2dd37b0..c2da003 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -11,7 +11,14 @@ function predicateStr(p?: Predicate | null): string | null { if (p.type === "test") return `test "${p.target}"`; if (p.type === "command") return p.expect === "empty" ? `"${p.command}" finds nothing` : `"${p.command}" passes`; if (p.type === "human") return `a human confirms "${p.description}"`; - return null; + // skill = an eval: name the verdict and, when not the default, the subject it judges. + const verdict = p.minScore !== undefined ? `scores ${p.minScore}+` : "approves"; + const on = p.subject && p.subject !== "output" ? ` on the ${p.subject}` : ""; + return `eval: skill "${p.skill}" ${verdict}${on}`; +} +/** Render every `done when` predicate (a conjunction) as labelled strings. */ +function predicateStrs(dw?: Predicate[] | null): string[] { + return (dw ?? []).map(predicateStr).filter((s): s is string => s != null); } const failsToReflect = (t?: Transition[]) => (t ?? []).some((x) => x.on === "fail" && (x.do ?? []).some((d) => d.action === "reflect" || d.action === "plan")); @@ -28,13 +35,21 @@ export function renderLoop(loop: Loop): string { const cyc = (loop.cycle?.length ? loop.cycle : ["plan", "act", "observe"]).join(" → "); L.push(` ↻ ${cyc} (each cycle)`); if (failsToReflect(loop.transitions)) L.push(` ↺ on fail: reflect → plan (the back-edge)`); - const pred = predicateStr(loop.doneWhen); - L.push(pred ? ` ✓ done when: ${pred}` : ` ⚠ no done-when — can only stop via a human path or the guard`); + const preds = predicateStrs(loop.doneWhen); + if (preds.length) preds.forEach((p) => L.push(` ✓ done when: ${p}`)); + else L.push(` ⚠ no done-when — can only stop via a human path or the guard`); const g = guard(loop.transitions); if (g) L.push(` ⛔ guard: after ${g.n ?? "N"} tries → stop${g.warn ? ` & warn "${g.warn}"` : ""}`); if (loop.humanPlan) L.push(` 👤 a human approves the plan first`); if (loop.humanReviewBeforeStop) L.push(` 👤 a human reviews before stopping`); if (asksWhenBlocked(loop.transitions)) L.push(` 👤 when blocked: ask a human`); + for (const h of loop.hooks ?? []) { + const what = h.predicate.type === "command" ? `"${h.predicate.command}" ${h.predicate.expect === "empty" ? "finds nothing" : "passes"}` : h.predicate.type === "test" ? `test "${h.predicate.target}"` : ""; + L.push(` ⊘ hook ${h.at.replace(/-/g, " ")}: ${what}`); + } + if (loop.context?.knowledge?.length) L.push(` 📖 knowledge: ${loop.context.knowledge.join(", ")}`); + if (loop.context?.examples?.length) L.push(` ✎ examples: ${loop.context.examples.join(", ")}`); + if (loop.tools?.length) L.push(` 🔌 tools from: ${loop.tools.join(", ")}`); if (loop.also?.length) L.push(` + also: ${loop.also.join(", ")}`); if (loop.goal) L.push(` · goal: ${loop.goal}`); return L.join("\n"); @@ -48,9 +63,11 @@ function gateMark(loop: Loop): string { export function renderPipeline(p: Pipeline): string { const L = [`pipeline "${p.name}" (stages in order · fail-fast)`]; p.stages.forEach((s, i) => { - const pred = predicateStr(s.loop.doneWhen); + const preds = predicateStrs(s.loop.doneWhen); + const pred = preds.length ? `${preds[0]}${preds.length > 1 ? ` (+${preds.length - 1})` : ""}` : null; + const par = s.parallelGroup != null ? "⇉ " : ""; const tail = [s.gate ? "👤 gate" : null, pred ? `✓ ${pred}` : null].filter(Boolean).join(" · "); - L.push(` ${i + 1}. ${s.name}${gateMark(s.loop)}${tail ? ` ${tail}` : ""}`); + L.push(` ${i + 1}. ${par}${s.name}${gateMark(s.loop)}${tail ? ` ${tail}` : ""}`); }); return L.join("\n"); } @@ -74,6 +91,19 @@ export function renderFile(file: LoopFile): string { const m = file.config?.models; if (m) parts.push(`models: fast=${m.tiers?.fast ?? "default"} · strong=${m.tiers?.strong ?? "default"}`); if (file.config?.git) parts.push(`git: ${file.config.git.isolation ?? "branch"}${file.config.git.openPr ? " + PR" : ""}${file.config.git.push ? "" : " · no push to main"}`); + if (file.config?.cycle?.length) parts.push(`each cycle: ${file.config.cycle.join(" → ")} (default)`); + if (file.config?.rigor || file.config?.mode) { + parts.push([file.config.rigor ? `rigor: ${file.config.rigor}` : null, file.config.mode ? `mode: ${file.config.mode}` : null].filter(Boolean).join(" · ")); + } + if (file.config?.observe) { + const o = file.config.observe; + parts.push(`observe: ${[o.trace ? "trace" : null, o.meter ? "meter" : null, o.costCap ? `cap ${o.costCap}` : null].filter(Boolean).join(" · ")}`); + } + if (file.config?.sandbox) { + const s = file.config.sandbox; + parts.push(`sandbox: ${[s.network === "none" ? "no network" : s.network === "allowlist" ? `egress ${(s.egress ?? []).join(",")}` : null, s.cpu ? `cpu ${s.cpu}` : null, s.memory ? `mem ${s.memory}` : null, s.time ? `time ${s.time}` : null].filter(Boolean).join(" · ")}`); + } + if (file.config?.runsAs) parts.push(`runs as: ${file.config.runsAs}`); for (const d of file.definitions) parts.push(renderDef(d)); return parts.join("\n\n") || "(empty .loop)"; } @@ -86,7 +116,70 @@ export function oneLine(d: Definition): string { d.cycle?.length ? d.cycle.join("→") : "plan→act→observe", failsToReflect(d.transitions) ? "reflect" : null, guard(d.transitions) ? "guard" : null, - predicateStr(d.doneWhen) ? "done-when" : "⚠ no done-when", + d.doneWhen?.length ? "done-when" : "⚠ no done-when", ].filter(Boolean); return `loop "${d.name ?? "?"}" · ${bits.join(", ")}`; } + +// ---- explain: read a loop back in plain English (the friendly, non-expert view) ---- + +const CYCLE_PROSE: Record = { + plan: "plans the change", + act: "makes the change", + observe: "checks the result", +}; + +/** Join items into prose: "a", "a then b", "a, b, then c". */ +function joinList(items: string[], conj = "then"): string { + if (items.length <= 1) return items[0] ?? ""; + if (items.length === 2) return `${items[0]} ${conj} ${items[1]}`; + return `${items.slice(0, -1).join(", ")}, ${conj} ${items[items.length - 1]}`; +} + +function predicateProse(p: Predicate): string { + if (p.type === "test") return `the test "${p.target}" passes`; + if (p.type === "command") return p.expect === "empty" ? `running \`${p.command}\` reports nothing` : `running \`${p.command}\` succeeds`; + if (p.type === "human") return `you confirm "${p.description}"`; + // skill = an eval + const verdict = p.minScore !== undefined ? `the "${p.skill}" review scores ${p.minScore} or more` : `the "${p.skill}" review approves`; + return verdict + (p.subject === "trajectory" ? " (judging how it got there, not just the result)" : ""); +} + +/** A plain-English description of a single loop. */ +export function explainLoop(loop: Loop): string { + const L: string[] = []; + L.push(`${loop.name ? `Loop "${loop.name}"` : "This loop"} works toward: ${loop.goal || "(no goal set)"}.`); + const cyc = (loop.cycle?.length ? loop.cycle : ["plan", "act", "observe"]).map((s) => CYCLE_PROSE[s] ?? s); + L.push(`Each round it ${joinList(cyc)}.`); + const preds = loop.doneWhen ?? []; + if (preds.length) L.push(`It's done when ${joinList(preds.map(predicateProse), "and")}.`); + else L.push(`It has no automatic check, so it relies on a human to decide when to stop.`); + if (failsToReflect(loop.transitions)) L.push(`If a check fails, it reflects on why and tries again.`); + if (loop.humanPlan) L.push(`It pauses for you to approve the plan before changing anything.`); + if (loop.humanReviewBeforeStop) L.push(`It pauses for you to review the result before finishing.`); + if (asksWhenBlocked(loop.transitions)) L.push(`If it gets stuck, it stops and asks you.`); + const g = guard(loop.transitions); + if (g) L.push(`It gives up after ${g.n ?? "a few"} tries${g.warn ? ` (warning "${g.warn}")` : ""}.`); + if (loop.also?.length) L.push(`Once the goal is met it also: ${loop.also.join(", ")}.`); + return L.join(" "); +} + +export function explainDef(d: Definition): string { + if (d.kind === "loop") return explainLoop(d); + if (d.kind === "pipeline") { + const n = d.stages.length; + const L = [`Pipeline "${d.name}" runs ${n} ${n === 1 ? "story" : "stories"} in order, stopping if one fails:`]; + d.stages.forEach((s, i) => L.push(` ${i + 1}. ${s.name}${s.gate ? " (pauses for your OK first)" : ""} — ${explainLoop(s.loop)}`)); + return L.join("\n"); + } + const L = [`Flow "${d.name}" runs these loop files in sequence, handing each result to the next:`]; + d.steps.forEach((s, i) => + L.push(` ${i + 1}. ${s.forEach ? `for each item in "${s.forEach.source}", run ${s.ref}` : s.ref}${s.gate ? " (pauses for your OK first)" : ""}`) + ); + return L.join("\n"); +} + +/** Plain-English description of a whole .loop file — the `loop-run explain` view. */ +export function explainFile(file: LoopFile): string { + return (file.definitions ?? []).map(explainDef).join("\n\n") || "(empty .loop)"; +} diff --git a/packages/runtime/src/summary.ts b/packages/runtime/src/summary.ts index c4197af..6b8683b 100644 --- a/packages/runtime/src/summary.ts +++ b/packages/runtime/src/summary.ts @@ -19,6 +19,33 @@ export function summarizeModels(events: LoopEvent[]): TierSummary[] { return Object.values(acc); } +export interface OpexSummary { + cycles: number; + reflects: number; + firstPass: boolean | null; + stopReason?: string; + satisfied: boolean; +} + +/** Compute the OpEx report from a run's event trace — the "token burn of unverified loops", made visible. */ +export function summarizeOpex(events: LoopEvent[]): OpexSummary { + const observes = events.filter((e): e is Extract => e.type === "observe"); + const stop = [...events].reverse().find((e): e is Extract => e.type === "stop"); + const end = [...events].reverse().find((e): e is Extract => e.type === "loop-end"); + return { + cycles: observes.length, + reflects: events.filter((e) => e.type === "reflect").length, + firstPass: observes.length ? observes[0].passed : null, + stopReason: stop?.reason, + satisfied: end?.satisfied ?? false, + }; +} + +export function formatOpexSummary(s: OpexSummary): string { + const fp = s.firstPass === null ? "n/a" : s.firstPass ? "yes" : "no"; + return `OpEx — ${s.cycles} cycle${s.cycles === 1 ? "" : "s"}, ${s.reflects} reflect${s.reflects === 1 ? "" : "s"} (back-edges), first-pass success: ${fp}, ${s.satisfied ? "satisfied" : "not satisfied"}${s.stopReason ? ` (${s.stopReason})` : ""}`; +} + export function formatModelSummary(s: TierSummary[]): string { if (s.length === 0) return ""; const total = s.reduce((n, t) => n + t.calls, 0); diff --git a/packages/runtime/src/types.ts b/packages/runtime/src/types.ts index f936252..e3962fd 100644 --- a/packages/runtime/src/types.ts +++ b/packages/runtime/src/types.ts @@ -17,6 +17,7 @@ export type LoopEvent = | { type: "memory-read"; file: string; bytes: number } | { type: "memory-write"; file: string; bytes: number } | { type: "skill-verify"; skill: string; passed: boolean; detail: string } + | { type: "hook"; at: string; detail: string; passed: boolean } | { type: "human"; kind: "plan" | "review" | "gate" | "confirm" | "ask"; prompt: string; answer?: string } | { type: "stop"; reason: StopReason; warn?: string } | { type: "loop-end"; name: string | null; satisfied: boolean } @@ -66,24 +67,32 @@ export interface SkillVerifyInput { /** The review skill to invoke. */ skill: string; goal: string; - /** The work to judge — typically the last act summary. */ + /** The work to judge — the act summary for an output eval, the cycle trajectory for a trajectory eval. */ context: string; /** "approve" requires an approving verdict. */ expect: "approve"; /** When set, the verdict passes only if the skill's numeric score is at least this. */ minScore?: number; + /** What the eval inspects: the produced "output" (default) or the "trajectory" (path + tool calls). */ + subject?: "output" | "trajectory"; + /** Inline rubric (`the bar:`) naming the conditions the judge must score against. */ + bar?: string; baseDir: string; } export interface ActResult { summary: string; blocked?: boolean; + /** The captured trajectory of this act: the steps and tool calls the agent took. Fed to trajectory evals. */ + trajectory?: string; } export interface ReflectInput { goal: string; focus?: string; output: string; + /** The most recent act's trajectory, so a `reflect on the path it took` can see it. */ + trajectory?: string; baseDir: string; /** Model alias/id for this call; set by the engine from the model policy. */ model?: string; diff --git a/packages/runtime/test/engine.test.js b/packages/runtime/test/engine.test.js index d7ee398..875cfd9 100644 --- a/packages/runtime/test/engine.test.js +++ b/packages/runtime/test/engine.test.js @@ -656,3 +656,86 @@ test("regression: a loop with no skills/memory behaves exactly as before", async assert.ok(!events.some((e) => e.type === "memory-read" || e.type === "memory-write")); assert.deepEqual(store.files, {}, "no memory file written"); }); + +// ---- trajectory evals (Story 2) ---- + +test("trajectory eval: the captured trajectory + the bar reach the verifier", async () => { + const def = parse( + 'loop "x":\n goal: g\n' + + ' done when the skill "path-review" approves on the trajectory\n' + + ' the bar: did not weaken a test to go green\n' + + ' after 2 tries: stop and warn "stuck"' + ).definitions[0]; + const runner = new MockRunner({ + act: (i) => ({ summary: `acted on: ${i.goal}`, trajectory: "● Edit refunds.ts\n● Bash $ pnpm test" }), + skill: () => ({ passed: true, detail: "APPROVED" }), + }); + const outcome = await runDefinition(def, { + runner, verifier: new SeqVerifier([true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, true); + const call = runner.skillCalls.at(-1); + assert.equal(call.subject, "trajectory"); + assert.equal(call.bar, "did not weaken a test to go green"); + assert.match(call.context, /Edit refunds\.ts/, "the eval judged the trajectory, not the act summary"); +}); + +test("output eval: receives the act summary, not the trajectory", async () => { + const def = parse( + 'loop "x":\n goal: g\n done when the skill "review" approves on the output\n after 2 tries: stop and warn "x"' + ).definitions[0]; + const runner = new MockRunner({ + act: () => ({ summary: "ACTSUMMARY", trajectory: "TRAJ-should-not-be-used" }), + skill: () => ({ passed: true, detail: "ok" }), + }); + await runDefinition(def, { runner, verifier: new SeqVerifier([true]), human: new ScriptedHumanIO(), baseDir: "/p" }); + const call = runner.skillCalls.at(-1); + assert.equal(call.subject, "output"); + assert.equal(call.context, "ACTSUMMARY"); +}); + +test("reflect sees the trajectory of the failing cycle (Story 3)", async () => { + const def = parse( + 'loop "x":\n goal: g\n done when "cmd" passes\n' + + ' when it fails: reflect, then plan again\n after 3 tries: stop and warn "stuck"' + ).definitions[0]; + const runner = new MockRunner({ act: () => ({ summary: "did stuff", trajectory: "● Edit a.ts\n● Bash $ cmd" }) }); + await runDefinition(def, { + runner, verifier: new SeqVerifier([false, true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(runner.reflectCalls.length, 1, "reflected on the one failure"); + assert.match(runner.reflectCalls[0].trajectory, /Edit a\.ts/, "reflect received the cycle's trajectory"); +}); + +// ---- hooks (Story 7) ---- + +test("hooks: a failing before-cycle hook blocks the loop", async () => { + const def = parse('loop "x":\n goal: g\n check: npm test\n hooks:\n before each cycle: "tsc" passes').definitions[0]; + const outcome = await runDefinition(def, { + runner: new MockRunner(), verifier: new SeqVerifier([false]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, false); + assert.equal(outcome.reason, "blocked"); +}); + +test("hooks: a passing before-cycle hook lets the loop proceed to done", async () => { + const def = parse('loop "x":\n goal: g\n check: npm test\n hooks:\n before each cycle: "tsc" passes\n after 2 tries: stop and warn "x"').definitions[0]; + const outcome = await runDefinition(def, { + runner: new MockRunner(), verifier: new SeqVerifier([true, true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, true); +}); + +// ---- parallel stages (Story 11) ---- + +test("parallel stages run concurrently and all must satisfy", async () => { + const def = parse( + 'pipeline "p":\n stages in parallel:\n stage "a":\n goal: g\n check: t\n stage "b":\n goal: g\n check: t' + ).definitions[0]; + const order = []; + const runner = new MockRunner({ act: (i) => { order.push(i.goal); return { summary: "ok" }; } }); + const outcome = await runDefinition(def, { + runner, verifier: new SeqVerifier([true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, true); +}); diff --git a/packages/runtime/test/examples.test.js b/packages/runtime/test/examples.test.js new file mode 100644 index 0000000..7e37d01 --- /dev/null +++ b/packages/runtime/test/examples.test.js @@ -0,0 +1,59 @@ +// Run the repo's .loop files AS tests: parse every example, then execute each standalone +// loop/pipeline through the engine with a mock runner. This is the framework's own +// "does every example shape actually run?" check. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync, readdirSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, join, relative } from "node:path"; +import { parse } from "@loop-lang/parser"; +import { runDefinition, MockRunner, ScriptedHumanIO } from "../dist/index.js"; + +const here = dirname(fileURLToPath(import.meta.url)); +const root = join(here, "..", "..", ".."); + +function findLoops(dir, acc = []) { + for (const e of readdirSync(dir, { withFileTypes: true })) { + if (e.name === "node_modules" || e.name === "dist" || e.name.startsWith(".")) continue; + const p = join(dir, e.name); + if (e.isDirectory()) findLoops(p, acc); + else if (e.name.endsWith(".loop")) acc.push(p); + } + return acc; +} + +const files = [...findLoops(join(root, "examples")), join(root, "agentic-engineering.loop")]; + +// A verifier that always passes; human IO that always approves — so every example reaches done. +class PassVerifier { async verify() { return { passed: true, output: "ok" }; } } +const stubs = { + runner: new MockRunner(), + verifier: new PassVerifier(), + human: new ScriptedHumanIO(), + baseDir: "/p", + hardCap: 4, + readText: async () => "plan: do the thing", + writeText: async () => {}, +}; + +test("every example .loop parses", () => { + assert.ok(files.length >= 40, `expected many examples, found ${files.length}`); + for (const f of files) { + try { parse(readFileSync(f, "utf8")); } + catch (err) { assert.fail(`parse failed for ${relative(root, f)}: ${err.message}`); } + } +}); + +test("every standalone loop/pipeline example runs to a terminal outcome", async () => { + for (const f of files) { + const file = parse(readFileSync(f, "utf8")); + for (const def of file.definitions) { + if (def.kind === "flow") continue; // flows chain real files — covered by parse only + const outcome = await runDefinition(def, stubs); + assert.ok( + typeof outcome.satisfied === "boolean" && typeof outcome.reason === "string", + `${relative(root, f)} :: ${def.name ?? def.kind} did not return a terminal outcome` + ); + } + } +}); diff --git a/packages/runtime/test/show.test.js b/packages/runtime/test/show.test.js index 96d6a31..ec1a6e5 100644 --- a/packages/runtime/test/show.test.js +++ b/packages/runtime/test/show.test.js @@ -41,3 +41,16 @@ test("oneLine gives a terse shape per definition kind", () => { const loop = parse(`loop "y":\n goal: g\n done when "t" passes\n each cycle: plan, then act, then observe\n when it fails: reflect, then plan again\n`).definitions[0]; assert.match(oneLine(loop), /loop "y" · plan→act→observe, reflect, done-when/); }); + +test("explain: plain-English description of a loop", async () => { + const { explainFile } = await import("../dist/show.js"); + const file = parse( + 'loop "fix the cart test":\n goal: the cart total is correct\n check: pnpm test cart\n' + + ' when it fails: reflect, then plan again\n after 6 tries: stop and warn "stuck"' + ); + const prose = explainFile(file); + assert.match(prose, /works toward: the cart total is correct/); + assert.match(prose, /done when running `pnpm test cart` succeeds/); + assert.match(prose, /reflects on why and tries again/); + assert.match(prose, /gives up after 6 tries/); +}); diff --git a/packages/runtime/test/summary.test.js b/packages/runtime/test/summary.test.js index f41a690..ed09f46 100644 --- a/packages/runtime/test/summary.test.js +++ b/packages/runtime/test/summary.test.js @@ -19,3 +19,20 @@ test("summarizeModels tallies calls per tier and node", () => { assert.equal(strong.calls, 1); assert.deepEqual(strong.byNode, { act: 1 }); }); + +test("summarizeOpex: cycles, reflects, first-pass, outcome", async () => { + const { summarizeOpex, formatOpexSummary } = await import("../dist/summary.js"); + const events = [ + { type: "observe", passed: false, output: "" }, + { type: "reflect", text: "r" }, + { type: "observe", passed: true, output: "" }, + { type: "stop", reason: "done" }, + { type: "loop-end", name: "x", satisfied: true }, + ]; + const s = summarizeOpex(events); + assert.equal(s.cycles, 2); + assert.equal(s.reflects, 1); + assert.equal(s.firstPass, false); + assert.equal(s.satisfied, true); + assert.match(formatOpexSummary(s), /2 cycles, 1 reflect .* first-pass success: no/); +}); diff --git a/packages/viz/src/render.ts b/packages/viz/src/render.ts index 4ec0fe0..ea7d752 100644 --- a/packages/viz/src/render.ts +++ b/packages/viz/src/render.ts @@ -118,7 +118,7 @@ function chip(x,y,text,color){var w=7.2*text.length+14; return [V("rect",{x:x,y:y,width:w,height:17,rx:8.5,fill:"none",stroke:color,"stroke-width":1}) +E("text",{x:x+w/2,y:y+12,"text-anchor":"middle",class:"chip",fill:color},esc(text)),w]; } -function doneLabel(dw){if(!dw)return "awaiting human";if(dw.type==="test")return "test "+dw.target;if(dw.type==="command")return (dw.expect==="empty"?"empty: ":"")+dw.command;if(dw.type==="skill")return "skill "+dw.skill+(dw.minScore!==undefined?" ≥"+dw.minScore:"");return "human: "+(dw.description||"");} +function doneLabel(dw){if(!dw)return "awaiting human";if(dw.type==="test")return "test "+dw.target;if(dw.type==="command")return (dw.expect==="empty"?"empty: ":"")+dw.command;if(dw.type==="skill")return "eval "+dw.skill+(dw.minScore!==undefined?" ≥"+dw.minScore:"")+(dw.subject&&dw.subject!=="output"?" ("+dw.subject+")":"");return "human: "+(dw.description||"");} function loopRow(loop,ox,oy){ var s="",cyc=loop.cycle&&loop.cycle.length?loop.cycle:["plan","act","observe"],xs=[],i; for(i=0;i1?" +"+(dw.length-1):""; + s+=E("text",{x:stopX+NW/2,y:oy+NH+15,"text-anchor":"middle",class:"sub"},esc(trunc("done when "+doneLabel(dw&&dw[0])+dwMore,24))); for(i=0;i=0&&loop.policy){var px=xs[ai],py=oy+NH+12,a; diff --git a/packages/vscode/src/language.ts b/packages/vscode/src/language.ts index 11e30ef..217ba9f 100644 --- a/packages/vscode/src/language.ts +++ b/packages/vscode/src/language.ts @@ -201,14 +201,14 @@ export const VOCABULARY = Object.keys(HOVERS); interface LintLoop { kind: "loop"; name: string | null; - doneWhen?: unknown; + doneWhen?: unknown[]; humanReviewBeforeStop?: boolean; transitions?: Array<{ on: string; threshold?: number; do?: Array<{ action: string }> }>; } interface LintStage { name: string; loop: LintLoop } interface LintPipeline { kind: "pipeline"; name: string; stages: LintStage[] } interface LintFlow { kind: "flow"; name: string; steps: unknown[] } -interface LintFile { definitions: Array } +interface LintFile { definitions: Array; config?: { rigor?: string } } export interface LintWarning { line: number; // 0-based source line to attach the squiggle @@ -226,11 +226,32 @@ function headerLine(lines: string[], kind: "loop" | "stage", name: string | null return Math.max(0, j); } -function checkLoop(loop: LintLoop, line: number, out: LintWarning[]) { +function checkLoop(loop: LintLoop, line: number, out: LintWarning[], rigor?: string) { + const preds = Array.isArray(loop.doneWhen) ? loop.doneWhen : []; // Unverifiable: nothing decides "done" — only a thrash guard or the hard cap can stop it. - if (!loop.doneWhen && !loop.humanReviewBeforeStop) { + if (!preds.length && !loop.humanReviewBeforeStop) { out.push({ line, message: "This loop has no way to verify it's done — add a `done when …` check or `a human reviews before stopping`." }); } + // "Without both, it is vibe coding": under structured/agentic rigor, a loop that checks one + // kind of thing but not the other gets a nudge. (Off by default — only when rigor opts in.) + if (rigor === "structured ai-assisted" || rigor === "agentic engineering") { + const isEval = (p: unknown): boolean => !!p && typeof p === "object" && (p as any).type === "skill"; + const hasTest = preds.some((p) => !isEval(p)); + const hasEval = preds.some((p) => isEval(p)); + if (hasTest && !hasEval) { + out.push({ line, message: "Tests but no eval — add `done when the skill \"…\" approves` (an eval) for the non-deterministic parts." }); + } else if (hasEval && !hasTest) { + out.push({ line, message: "An eval but no test — add a deterministic `done when \"…\" passes` for the parts code can check." }); + } + } + // A trajectory eval gates "done" on an LM judging a non-deterministic path; without an explicit + // `the bar:` it can wobble run-to-run and cause its own thrash. Nudge toward a rubric. + const barlessTrajectoryEval = (Array.isArray(loop.doneWhen) ? loop.doneWhen : []).some( + (p): boolean => !!p && typeof p === "object" && (p as any).type === "skill" && (p as any).subject === "trajectory" && !(p as any).bar + ); + if (barlessTrajectoryEval) { + out.push({ line, message: "This trajectory eval has no `the bar:` — an LM judging the path can wobble run-to-run; add `the bar: …` with explicit pass conditions." }); + } // Self-correcting but unbounded: re-plans on failure with no attempt ceiling. const reflects = (loop.transitions ?? []).some((t) => t.on === "fail" && (t.do ?? []).some((d) => d.action === "reflect" || d.action === "plan")); const guarded = (loop.transitions ?? []).some((t) => t.on === "attempts" && typeof t.threshold === "number"); @@ -242,13 +263,18 @@ function checkLoop(loop: LintLoop, line: number, out: LintWarning[]) { /** Structural warnings for a parsed spec. Pure: takes the spec + source lines, returns nudges. */ export function lint(file: LintFile, lines: string[]): LintWarning[] { const out: LintWarning[] = []; + const rigor = file.config?.rigor; + // The costliest quadrant: an unverified, disposable loop fired off to run unattended. + if (rigor === "vibe coding" && (file.config as any)?.mode === "orchestrator") { + out.push({ line: 0, message: "`rigor: vibe coding` + `mode: orchestrator` runs an unverified loop unattended — the costliest combo. Add a `done when …` check or raise rigor." }); + } for (const def of file.definitions ?? []) { if (def.kind === "pipeline") { for (const stage of def.stages ?? []) { - checkLoop(stage.loop, headerLine(lines, "stage", stage.name), out); + checkLoop(stage.loop, headerLine(lines, "stage", stage.name), out, rigor); } } else if (def.kind === "loop") { - checkLoop(def, headerLine(lines, "loop", def.name), out); + checkLoop(def, headerLine(lines, "loop", def.name), out, rigor); } // flow: nothing to lint here — its referenced stories carry their own checks } diff --git a/packages/vscode/test/language.test.mjs b/packages/vscode/test/language.test.mjs index dedeb31..b98556f 100644 --- a/packages/vscode/test/language.test.mjs +++ b/packages/vscode/test/language.test.mjs @@ -87,14 +87,14 @@ test("lint warns on an unverifiable loop (no done when, no human review)", () => test("lint warns on a self-correcting loop with no thrash guard", () => { const src = 'loop "y":\n goal: g\n done when "npm test" passes\n when it fails: reflect, then plan again'; - const file = { definitions: [{ kind: "loop", name: "y", doneWhen: { type: "command" }, transitions: [{ on: "fail", do: [{ action: "reflect" }, { action: "plan" }] }] }] }; + const file = { definitions: [{ kind: "loop", name: "y", doneWhen: [{ type: "command" }], transitions: [{ on: "fail", do: [{ action: "reflect" }, { action: "plan" }] }] }] }; const w = lint(file, lines(src)); assert.equal(w.length, 1); assert.match(w[0].message, /thrash guard/); }); test("lint stays silent on a complete loop", () => { - const file = { definitions: [{ kind: "loop", name: "z", doneWhen: { type: "command" }, transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; + const file = { definitions: [{ kind: "loop", name: "z", doneWhen: [{ type: "command" }], transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; assert.deepEqual(lint(file, lines('loop "z":')), []); }); @@ -111,3 +111,41 @@ test("lint produces zero warnings for a flow definition", () => { const w = lint(file, lines(src)); assert.deepEqual(w, [], "flow should produce no lint warnings"); }); + +test("lint nudges a trajectory eval that lacks `the bar:`", () => { + const file = { definitions: [{ kind: "loop", name: "t", + doneWhen: [{ type: "skill", skill: "path", subject: "trajectory" }], + transitions: [{ on: "attempts", threshold: 5, do: [{ action: "stop" }] }] }] }; + const w = lint(file, lines('loop "t":')); + assert.equal(w.length, 1); + assert.match(w[0].message, /the bar:/); +}); + +test("lint stays silent on a trajectory eval that has `the bar:`", () => { + const file = { definitions: [{ kind: "loop", name: "t", + doneWhen: [{ type: "skill", skill: "path", subject: "trajectory", bar: "no test edits" }], + transitions: [{ on: "attempts", threshold: 5, do: [{ action: "stop" }] }] }] }; + assert.deepEqual(lint(file, lines('loop "t":')), []); +}); + +test("lint nudges 'without both' under structured/agentic rigor", () => { + const file = { config: { rigor: "agentic engineering" }, + definitions: [{ kind: "loop", name: "x", doneWhen: [{ type: "command" }], + transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; + const w = lint(file, lines('loop "x":')); + assert.equal(w.length, 1); + assert.match(w[0].message, /Tests but no eval/); +}); + +test("lint stays silent on 'without both' when no rigor is set", () => { + const file = { definitions: [{ kind: "loop", name: "x", doneWhen: [{ type: "command" }], + transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; + assert.deepEqual(lint(file, lines('loop "x":')), []); +}); + +test("lint warns on the vibe+orchestrator quadrant", () => { + const file = { config: { rigor: "vibe coding", mode: "orchestrator" }, definitions: [] }; + const w = lint(file, lines("")); + assert.equal(w.length, 1); + assert.match(w[0].message, /costliest combo/); +}); diff --git a/spec/loop-spec.schema.json b/spec/loop-spec.schema.json index c0d9a88..70bdfdd 100644 --- a/spec/loop-spec.schema.json +++ b/spec/loop-spec.schema.json @@ -44,7 +44,52 @@ "description": "Notification destination (e.g. \"slack\"). Deferred past v1." }, "git": { "$ref": "#/$defs/gitPolicy" }, - "models": { "$ref": "#/$defs/modelPolicy" } + "models": { "$ref": "#/$defs/modelPolicy" }, + "cycle": { + "type": "array", + "description": "File-level default cycle (`each cycle: …` at the config tier). Applies to every loop without its own `each cycle:`; a per-loop cycle overrides it. Lowest wins, like git.", + "items": { + "type": "string", + "enum": ["plan", "act", "observe"] + }, + "minItems": 1 + }, + "rigor": { + "type": "string", + "enum": ["vibe coding", "structured ai-assisted", "agentic engineering"], + "description": "The spectrum dial — bundles best-practice defaults (a back-edge + thrash guard) over every loop in the file. structured/agentic inject them when a loop omits its own; vibe coding injects nothing." + }, + "mode": { + "type": "string", + "enum": ["conductor", "orchestrator"], + "description": "Supervision posture: conductor = in-session/synchronous (gates answered inline); orchestrator = async/reviews-outcomes (opens a PR)." + }, + "observe": { + "type": "object", + "additionalProperties": false, + "description": "Observability: a per-cycle trace, cost metering, and an optional spend cap.", + "properties": { + "trace": { "type": "boolean", "description": "Emit a per-cycle trace and a stop-time OpEx report." }, + "meter": { "type": "boolean", "description": "Meter tokens and cost per cycle (needs a provider that reports usage)." }, + "costCap": { "type": "string", "description": "Stop and warn if spend exceeds this (e.g. \"$5\")." } + } + }, + "sandbox": { + "type": "object", + "additionalProperties": false, + "description": "Run isolation: where code runs and what it cannot reach.", + "properties": { + "network": { "type": "string", "enum": ["none", "allowlist"] }, + "egress": { "type": "array", "items": { "type": "string" } }, + "cpu": { "type": "string" }, + "memory": { "type": "string" }, + "time": { "type": "string" } + } + }, + "runsAs": { + "type": "string", + "description": "Identity the run executes as (`runs as: …`) — an auditable principal for unattended runs." + } } }, "definitions": { @@ -108,7 +153,11 @@ "message": { "type": "string" } } }, - "loop": { "$ref": "#/$defs/loop" } + "loop": { "$ref": "#/$defs/loop" }, + "parallelGroup": { + "type": "integer", + "description": "Stages sharing a parallel-group id run concurrently (`stages in parallel:`)." + } } }, "loop": { @@ -126,11 +175,9 @@ "description": "The objective in plain language." }, "doneWhen": { - "description": "The predicate the runtime checks to declare the goal met. Null means the loop relies on a human review to stop.", - "oneOf": [ - { "$ref": "#/$defs/predicate" }, - { "type": "null" } - ] + "description": "Verification conditions — ALL must pass (a conjunction of predicates). A test/command predicate is a deterministic check (TESTS); a skill predicate is an eval (EVALS). Empty/absent means the loop relies on a human review to stop.", + "type": "array", + "items": { "$ref": "#/$defs/predicate" } }, "context": { "description": "What the agent may look at before acting (the `look at:` block).", @@ -143,6 +190,16 @@ "type": "boolean", "description": "Whether \"and the last failure\" was present.", "default": false + }, + "knowledge": { + "type": "array", + "items": { "type": "string" }, + "description": "Read-only reference material (`knowledge:`) the agent reads but must not edit." + }, + "examples": { + "type": "array", + "items": { "type": "string" }, + "description": "Reference patterns to imitate (`examples:`)." } } }, @@ -172,6 +229,11 @@ }, "minItems": 1 }, + "hooks": { + "type": "array", + "description": "Deterministic checks bound to lifecycle points (`hooks:`). A failing hook blocks.", + "items": { "$ref": "#/$defs/hook" } + }, "also": { "type": "array", "description": "Extra finishing passes run in order after the goal is met (e.g. \"polish\", \"run a security check\", \"update the docs\"). Lightweight agent passes; promote to a stage when one needs its own verification.", @@ -182,6 +244,11 @@ "description": "Named execution skills the loop may invoke during plan/act (e.g. [\"check-weather\", \"analyze-workout\"]). The runner surfaces them as available capabilities to coordinate, instead of one mega-prompt.", "items": { "type": "string" } }, + "tools": { + "type": "array", + "description": "MCP servers whose tools the loop may use (`use tools from the \"github\" server`).", + "items": { "type": "string" } + }, "memory": { "description": "Cross-run memory: a markdown file the loop reads on start (its lessons feed the first plan) and appends an outcome entry to on stop.", "type": "object", @@ -223,6 +290,19 @@ "models": { "$ref": "#/$defs/modelPolicy" } } }, + "hook": { + "type": "object", + "additionalProperties": false, + "required": ["at", "predicate"], + "properties": { + "at": { + "type": "string", + "enum": ["before-cycle", "after-plan", "after-act", "after-observe", "on-commit", "on-push", "on-stop"], + "description": "The lifecycle point this check runs at." + }, + "predicate": { "$ref": "#/$defs/predicate" } + } + }, "predicate": { "type": "object", "additionalProperties": false, @@ -244,6 +324,15 @@ "type": "integer", "description": "For type=skill: when present (\"scores N or more\"), the verdict passes only when the skill's numeric score is at least this. Absent means a plain approve/reject." }, + "subject": { + "type": "string", + "enum": ["output", "trajectory"], + "description": "For type=skill (an eval): what the judge inspects — the produced \"output\" (default) or the \"trajectory\" (the path and tool calls the agent took). Set by `on the output` / `on the trajectory`." + }, + "bar": { + "type": "string", + "description": "For type=skill (an eval): an inline rubric (`the bar:`) naming the conditions the judge scores against." + }, "command": { "type": "string", "description": "For type=command: the shell command. Default success = exit 0."

  • In VS Code