diff --git a/.archive/README.md b/.archive/README.md new file mode 100644 index 00000000..4467f7a1 --- /dev/null +++ b/.archive/README.md @@ -0,0 +1,23 @@ +# Archive + +Kept in the working tree on purpose. Git history is not a substitute for +opening these files while reading the demos and backend-migration docs. + +**Retention:** do not delete this directory. `.gitignore` lists `.archive/` so +*new* debris (`compare-debug/`, screenshot dumps) stays untracked; the paths +below stay in git. Local-only files (phase plans, drafts) may also sit here — +they are not deleted by this spec. + +## Tracked survivors + +| Path | Why it stays | +| --- | --- | +| `demos-removed-2026-06-28/01_hello_world/` | Retired demo 01; cited from `docs/DEMOS.md` | +| `demos-removed-2026-06-28/03_data_pipeline/` | Retired demo 03 | +| `demos-removed-2026-06-28/04_ml_api/` | Retired demo 04 | +| `docs/DATA_INTEGRITY_FIXES.md` | Historical integrity notes | +| `docs/claude-agent-sdk/CLAUDE_AGENT_SDK_PLAN.md` | SDK adoption plan (PROMPT 7 / AgentCore thread) | +| `docs/langgraph/LANGGRAPH_MIGRATION_PLAN.md` | LangGraph migration plan | +| `docs/ui-ux/UI_UX_IMPROVEMENT_PLAN*.md` | Pre-dashboard UI plans | + +Do not add new product docs here — use `docs/` or `.kiro/specs/`. diff --git a/.cursor/rules/agent-roles-and-routing.mdc b/.cursor/rules/agent-roles-and-routing.mdc index 05e39c73..73a0f123 100644 --- a/.cursor/rules/agent-roles-and-routing.mdc +++ b/.cursor/rules/agent-roles-and-routing.mdc @@ -1,6 +1,6 @@ --- description: Agent roles and flow routing reference -globs: ["src/ai_team/crews/**/*.py", "src/ai_team/flows/**/*.py"] +globs: ["src/ai_team/backends/crewai_backend/crews/**/*.py", "src/ai_team/backends/crewai_backend/flows/**/*.py"] alwaysApply: false --- diff --git a/.cursor/rules/ai-team-core.mdc b/.cursor/rules/ai-team-core.mdc index 64130fb6..96030ca2 100644 --- a/.cursor/rules/ai-team-core.mdc +++ b/.cursor/rules/ai-team-core.mdc @@ -23,7 +23,9 @@ This is a portfolio-grade capstone project. All code must be production-quality, ## Project Structure (abbreviated) -- `src/ai_team/` — config (settings, agents.yaml, tasks.yaml), agents/, crews/, flows/, tools/, guardrails/, memory/, models/, utils/ +- `src/ai_team/` — config, harness, tools, guardrails, memory, models, backends (`crewai_backend` holds agents/crews/tasks/flows) - `tests/` — unit/, integration/, e2e/, performance/, fixtures/ - `src/ai_team/ui/` — FastAPI + Vite web dashboard (`ai-team-web`), Textual TUI (`ai-team-tui`), shared API client - `docs/`, `scripts/`, `demos/`, `docker/`, `.github/` +- `.kiro/specs//` — Kiro-style specs (`README` → `requirements` → `design` → + `tasks`). `tasks.md` is a living checklist; see the `kiro-specs` rule. diff --git a/.cursor/rules/crewai-agents-crews-flows.mdc b/.cursor/rules/crewai-agents-crews-flows.mdc index 567c031d..6028ac17 100644 --- a/.cursor/rules/crewai-agents-crews-flows.mdc +++ b/.cursor/rules/crewai-agents-crews-flows.mdc @@ -1,6 +1,6 @@ --- description: CrewAI agent, crew, and flow patterns -globs: ["src/ai_team/agents/**/*.py", "src/ai_team/crews/**/*.py", "src/ai_team/flows/**/*.py"] +globs: ["src/ai_team/backends/crewai_backend/agents/**/*.py", "src/ai_team/backends/crewai_backend/crews/**/*.py", "src/ai_team/backends/crewai_backend/flows/**/*.py"] alwaysApply: false --- @@ -9,7 +9,7 @@ alwaysApply: false Agents: YAML config (agents.yaml) + Python class extending BaseAgent. Use `role_name` (maps to YAML key), `create()` → `load_yaml_config()`, `get_ollama_llm()`, return `Agent(...)`. ```python -from ai_team.agents.base import BaseAgent +from ai_team.backends.crewai_backend.agents.base import BaseAgent class ArchitectAgent(BaseAgent): """Solutions Architect agent for system design.""" diff --git a/.cursor/rules/crewai-tools.mdc b/.cursor/rules/crewai-tools.mdc index 1b56c1ca..23037539 100644 --- a/.cursor/rules/crewai-tools.mdc +++ b/.cursor/rules/crewai-tools.mdc @@ -4,6 +4,9 @@ globs: "src/ai_team/tools/**/*.py" alwaysApply: false --- +CrewAI agents, crews, tasks, and flows live under `src/ai_team/backends/crewai_backend/`. +Tools stay in `src/ai_team/tools/`. + # Tool Definition Pattern - Use `@tool("Display Name")`; docstring with Args and Returns diff --git a/.cursor/rules/github-ci.mdc b/.cursor/rules/github-ci.mdc new file mode 100644 index 00000000..7303ca03 --- /dev/null +++ b/.cursor/rules/github-ci.mdc @@ -0,0 +1,28 @@ +--- +description: How to start and watch GitHub Actions without hanging on gh or MCP auth +alwaysApply: true +--- + +# GitHub CI + +`.github/workflows/ci.yml` runs on **push to `main`/`develop`** and **pull_request into those**. A feature-branch push is a no-op until a PR exists. + +## Watch CI on the public site + +Do **not** block on `gh auth login` or GitHub MCP `mcp_auth` (that prompt has hung this repo). Git push already works over HTTPS. + +Monitor with unauthenticated HTTP: + +- Actions: https://github.com/RickZee/ai-team/actions +- Branch runs: https://github.com/RickZee/ai-team/actions?query=branch%3Afeature%2Fphase-harness-alignment +- API: `https://api.github.com/repos/RickZee/ai-team/actions/runs?branch=` +- Releases: https://github.com/RickZee/ai-team/releases +- Compare / open PR: https://github.com/RickZee/ai-team/compare/main... + +`gh run view` is optional when `gh` is already authenticated. + +## After a push that should run CI + +1. Confirm a PR into `main` or `develop` exists (or open one via the compare URL / git credentials). +2. Poll the public Actions API or page until jobs complete. +3. If red, follow `.cursor/skills/fix-ci/SKILL.md` from the failed job name. diff --git a/.cursor/rules/how-to-add-features.mdc b/.cursor/rules/how-to-add-features.mdc index fd6bd03d..19882637 100644 --- a/.cursor/rules/how-to-add-features.mdc +++ b/.cursor/rules/how-to-add-features.mdc @@ -6,7 +6,7 @@ alwaysApply: true # When creating a new agent 1. Add YAML config to `config/agents.yaml` (role, goal, backstory, verbose, memory, allow_delegation, max_iter) -2. Create Python class in `agents/` extending `BaseAgent` +2. Create Python class in `src/ai_team/backends/crewai_backend/agents/` extending `BaseAgent` 3. Define tools as `@tool` in `tools/` 4. Add model in `config/settings.py` `OllamaModelConfig` 5. Register in the appropriate crew's agent list @@ -32,8 +32,8 @@ alwaysApply: true # When creating a new crew -1. Create in `src/ai_team/crews/` with `kickoff()` method -2. Define tasks in tasks/ (YAML + Python) +1. Create in `src/ai_team/backends/crewai_backend/crews/` with `kickoff()` method +2. Define tasks in `src/ai_team/config/tasks.yaml` and `src/ai_team/backends/crewai_backend/tasks/` 3. Set process (hierarchical or sequential) 4. Configure memory, planning, verbose 5. Add context dependencies between tasks @@ -42,7 +42,7 @@ alwaysApply: true # When adding to the flow -1. Add `@listen` to `AITeamFlow` in `flows/main_flow.py` +1. Add `@listen` to `AITeamFlow` in `src/ai_team/backends/crewai_backend/flows/main_flow.py` 2. Add `@router` for branching if needed 3. Update `ProjectState` if needed 4. Add phase to `ProjectPhase` enum if new diff --git a/.cursor/rules/kiro-specs.mdc b/.cursor/rules/kiro-specs.mdc new file mode 100644 index 00000000..db161ae5 --- /dev/null +++ b/.cursor/rules/kiro-specs.mdc @@ -0,0 +1,45 @@ +--- +description: How to execute and update Kiro-style specs under .kiro/specs/ +alwaysApply: true +--- + +# Kiro-style specs + +Specs live at `.kiro/specs//` as four files. Read **in order**: `README.md` → +`requirements.md` → `design.md` → `tasks.md`. Do not implement from memory of an +older spec. + +## `tasks.md` is a living checklist + +It is not a read-only brief. After each task whose **Definition of done** is +literally true: + +1. Flip `- [ ]` to `- [x]` in that file in the same sitting (same commit if you + are already committing). +2. Leave **human-triggered / live-spend / budget-capped** tasks unchecked. +3. If you batched work, still update every completed box before you stop. An + all-empty checklist after a landed implementation is a process bug. + +Do **not** check a box because the code exists. Check it because each DoD +bullet is true (tests, fixtures, journal note, flag default, etc.). + +## How to execute + +- Prefer **one task at a time**, in order. The spec's Cursor prompt shape is the + default: implement N.M, run the listed quality commands, stop. +- When the user says "implement this spec", do every in-repo task; skip only + tasks marked human-triggered spend unless they explicitly approve spend. +- New behavior on a normal run ships **off by default** until the spec says it + earned its place. +- Record suite/CI failures in a journal entry (what failed, why, the fix). Do + not bury that trail only in chat. +- Watch GitHub Actions on the **public** repo pages/API (see + `.cursor/rules/github-ci.mdc`). Do not hang on `gh auth` or GitHub MCP + `mcp_auth`. Feature-branch pushes do not run CI until a PR targets + `main`/`develop`. + +## Do not + +- Invent a fifth spec document or rewrite requirements/design unless asked. +- Treat "I built it" as "the checklist can stay empty". +- Enable a billed eval, live arm, or `--execute` sweep from an agent session. diff --git a/.cursor/rules/models-and-state.mdc b/.cursor/rules/models-and-state.mdc index 1bc04168..581ff2aa 100644 --- a/.cursor/rules/models-and-state.mdc +++ b/.cursor/rules/models-and-state.mdc @@ -1,6 +1,6 @@ --- description: Pydantic output and flow state models reference -globs: ["src/ai_team/models/**/*.py", "src/ai_team/flows/state.py"] +globs: ["src/ai_team/models/**/*.py", "src/ai_team/backends/crewai_backend/flows/state.py"] alwaysApply: false --- @@ -13,7 +13,7 @@ alwaysApply: false - **DeploymentConfig**: dockerfile, docker_compose, ci_pipeline, environment_variables, infrastructure - **ProjectReport**: project_id, project_name, status, files, test_results, summary, duration -# Flow State (src/ai_team/flows/state.py) +# Flow State (src/ai_team/backends/crewai_backend/flows/state.py) **ProjectState**: project_id, project_description, current_phase, requirements, architecture, generated_files, test_results, deployment_config, phase_history, errors, retry_counts, max_retries, started_at, completed_at, metadata diff --git a/.cursor/rules/pydantic-and-errors.mdc b/.cursor/rules/pydantic-and-errors.mdc index f1cd9ed8..2ce6588d 100644 --- a/.cursor/rules/pydantic-and-errors.mdc +++ b/.cursor/rules/pydantic-and-errors.mdc @@ -9,11 +9,11 @@ alwaysApply: false - Use Pydantic v2 `BaseModel`; `Field(description=...)` for all fields - Validators: `@field_validator` or `@model_validator`; `model_config` with `json_schema_extra` where helpful - Settings: `pydantic_settings.BaseSettings` with `SettingsConfigDict` -- Output models: `src/ai_team/models/outputs.py`; state models: `src/ai_team/flows/state.py` +- Output models: `src/ai_team/models/outputs.py`; state models: `src/ai_team/backends/crewai_backend/flows/state.py` # Error Handling -- Custom exceptions in `src/ai_team/utils/exceptions.py`: `AITeamError` → `AgentError`, `ToolError`, `GuardrailError`, `FlowError` +- Custom exceptions live next to the code that raises them; never use a catch-all ``utils.exceptions`` module. - Never use bare `except:`; catch specific exceptions - Use `tenacity` for retry with exponential backoff - Log errors with structlog including context (agent, tool, phase) diff --git a/.cursor/rules/quality-pitfalls-and-deps.mdc b/.cursor/rules/quality-pitfalls-and-deps.mdc index 38cd1bc0..759a48d4 100644 --- a/.cursor/rules/quality-pitfalls-and-deps.mdc +++ b/.cursor/rules/quality-pitfalls-and-deps.mdc @@ -36,3 +36,6 @@ Guardrail <100ms; guardrail overhead <10% of task time; memory op <50ms - **Commits**: Conventional Commits — `feat(scope):`, `fix(scope):`, `test(scope):`, `docs(scope):` - **PR**: one phase or one feature per PR - **Checks**: ruff, mypy, pytest (unit), pytest (integration) +- **CI**: watch https://github.com/RickZee/ai-team/actions (public). Do not hang + on `gh auth` or GitHub MCP `mcp_auth`. Feature-branch push is a no-op until a + PR targets `main`/`develop`. diff --git a/.cursor/skills/fix-ci/SKILL.md b/.cursor/skills/fix-ci/SKILL.md index b1e43eb5..2986bd88 100644 --- a/.cursor/skills/fix-ci/SKILL.md +++ b/.cursor/skills/fix-ci/SKILL.md @@ -17,7 +17,11 @@ For **pip-audit / bandit** only, same skill. ## 0. Systematic triage (start here) -1. Open the [failed run](https://github.com/RickZee/ai-team/actions) → note **which job** is red (ignore **cancelled** matrix legs). +1. Open the public [Actions](https://github.com/RickZee/ai-team/actions) page (or + `https://api.github.com/repos/RickZee/ai-team/actions/runs?branch=`). + Do **not** hang on `gh auth login` or GitHub MCP `mcp_auth`. Note **which job** + is red (ignore **cancelled** matrix legs). Feature-branch pushes do not run CI + until a PR targets `main`/`develop`. 2. Map the job to a local command (table in §1). 3. Reproduce locally. Prefer `./scripts/ci_check.sh --matrix` before push to main (lint + 3.12 cov + 3.11 + security). 4. Fix root cause — do not lower coverage threshold or skip tests without evidence. diff --git a/.dockerignore b/.dockerignore index db38d610..61674c17 100644 --- a/.dockerignore +++ b/.dockerignore @@ -22,6 +22,13 @@ htmlcov venv .venv env +.coverage-data + +# Frontend (built in the Node stage) +**/node_modules +**/dist +src/ai_team/ui/web/frontend/node_modules +src/ai_team/ui/web/frontend/dist # IDE and editor .idea @@ -30,18 +37,14 @@ env *.swp *.swo -# Docs and dev (not needed in image) +# Docs and dev docs *.md !README.md -# Tests (not needed in runtime image) +# Tests tests -# Demos and scripts (optional; include if app needs them) -# demos -# scripts - # Local env and secrets .env .env.* @@ -52,8 +55,10 @@ Dockerfile docker-compose*.yml .dockerignore -# Output (mounted at runtime) +# Runtime mounts / generated output +workspace +logs # OS .DS_Store diff --git a/.env.example b/.env.example index 623df360..1292499e 100644 --- a/.env.example +++ b/.env.example @@ -25,3 +25,8 @@ AI_TEAM_ENV=dev # AI_TEAM_USE_REAL_LLM=1 # Live OpenRouter in integration/evals # AI_TEAM_TEST_MEMORY=1 # Live memory/embedder integration tests # AI_TEAM_EVAL_BUDGET_USD=5.00 # Live eval suite hard ceiling (Tier B/C); Tier A is $0 +# AI_TEAM_WEB_TOKEN= # Shared control-plane token. Required to bind 0.0.0.0. +# AI_TEAM_CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173 +# AI_TEAM_SESSION_LOOP=1 # Dormant multi-session continuation (off by default) +# AI_TEAM_MAX_SESSIONS=3 # Used only when AI_TEAM_SESSION_LOOP=1 + diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 00000000..bdc2f6e0 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1 @@ +* @RickZee diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..a539ed2f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,13 @@ +--- +name: Bug report +about: Something is wrong +labels: bug +--- + +**What happened** + +**What you expected** + +**How to reproduce** (offline if possible) + +**Backend** (crewai / langgraph / claude-agent-sdk / n/a) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..d4e05ecc --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,5 @@ +*What changed, and why it matters.* + +* [ ] Unit tests (`uv run pytest tests/unit`) +* [ ] Frontend (`npm test` / `npm run lint`) if UI changed +* [ ] No live model calls unless the PR says so diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7a95d22e..591642e6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,16 +32,16 @@ jobs: matrix: python-version: ["3.11", "3.12"] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: ${{ matrix.python-version }} @@ -60,6 +60,12 @@ jobs: - name: Mypy evals run: uv run mypy evals/ + - name: Hatchling package build + run: uv build + + - name: Repository invariant guards + run: uv run pytest tests/unit/repo -q --tb=line + test: name: Test runs-on: ubuntu-latest @@ -68,16 +74,16 @@ jobs: matrix: python-version: ["3.11", "3.12"] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: ${{ matrix.python-version }} @@ -92,7 +98,7 @@ jobs: mkdir -p test-results .coverage-data if [ "${{ matrix.python-version }}" = "3.12" ]; then uv run pytest tests/unit -v --tb=short \ - --cov=src/ai_team --cov-report=xml --cov-report=term \ + --cov=src/ai_team --cov=evals --cov-report=xml --cov-report=term \ --junitxml=test-results/unit.xml else uv run pytest tests/unit -v --tb=short \ @@ -101,13 +107,13 @@ jobs: - name: Upload coverage artifact if: matrix.python-version == '3.12' - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: coverage-${{ matrix.python-version }} path: coverage.xml - name: Upload test results - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: always() with: name: test-results-unit-${{ matrix.python-version }} @@ -117,21 +123,21 @@ jobs: name: Web UI E2E runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" - name: Set up Node - uses: actions/setup-node@v4 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version: "22" cache: npm @@ -143,9 +149,21 @@ jobs: - name: Install Playwright Chromium run: uv run playwright install chromium + - name: Install frontend dependencies + working-directory: src/ai_team/ui/web/frontend + run: npm ci + + - name: Frontend lint + working-directory: src/ai_team/ui/web/frontend + run: npm run lint + + - name: Frontend unit tests + working-directory: src/ai_team/ui/web/frontend + run: npm test + - name: Build frontend working-directory: src/ai_team/ui/web/frontend - run: npm ci && npm run build + run: npm run build - name: Run web E2E (demo/mock — no LLM) run: uv run pytest tests/e2e/web -m web_e2e -v --tb=short --timeout=120 @@ -155,16 +173,16 @@ jobs: runs-on: ubuntu-latest if: github.ref == 'refs/heads/main' && github.event_name == 'push' steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" @@ -177,7 +195,7 @@ jobs: uv run pytest tests/integration -v --tb=short --junitxml=test-results/integration.xml - name: Upload test results - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: always() with: name: test-results-integration @@ -187,16 +205,16 @@ jobs: name: Security runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" @@ -220,10 +238,17 @@ jobs: # starlette 1.0.1+ requires fastapi to lift starlette<1.0.0 (see fastapi pin). # chromadb CVE-2026-45830/31/33: Python server IDOR; no PyPI fix (see pip_audit.sh). - name: Pip-audit dependency vulnerabilities - run: ./scripts/pip_audit.sh + run: ./scripts/pip_audit.sh | tee pip-audit-report.txt + + - name: Upload pip-audit report + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + if: always() + with: + name: pip-audit-report + path: pip-audit-report.txt - name: Upload bandit report - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: always() with: name: bandit-report @@ -237,16 +262,16 @@ jobs: name: Eval Tier A runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Install uv - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" @@ -266,7 +291,7 @@ jobs: - name: Upload Tier A report artifacts if: always() - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: eval-tier-a-report path: | @@ -275,3 +300,44 @@ jobs: evals/results/ci-tier-a/report.html evals/results/ci-tier-a/summary.txt evals/results/ci-tier-a/gate.md + + image: + name: Container image + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + + - name: Build image + run: docker build -f docker/Dockerfile -t ai-team:ci . + + - name: Assert non-root, no gcc, size, health + run: | + set -euo pipefail + uid="$(docker run --rm --entrypoint "" ai-team:ci id -u)" + test "$uid" != "0" + if docker run --rm --entrypoint "" ai-team:ci sh -c 'command -v gcc'; then + echo "gcc must not be on PATH in the runtime image" >&2 + exit 1 + fi + size="$(docker image inspect ai-team:ci --format '{{.Size}}')" + echo "image_bytes=$size" + mb="$(awk -F' *= *' '/^docker_image_mb/{print $2; exit}' tests/unit/repo/ratchets.toml)" + echo "ratchet_mib=$mb" + test -n "$mb" + test "$size" -lt $((mb * 1024 * 1024)) + cid="$(docker run -d --name aiteam-ci ai-team:ci)" + trap 'docker rm -f "$cid" >/dev/null 2>&1 || true' EXIT + for i in $(seq 1 30); do + if docker exec "$cid" curl -fsS http://127.0.0.1:8421/api/health >/dev/null; then + echo "health ok" + exit 0 + fi + sleep 2 + done + docker logs "$cid" || true + echo "healthcheck failed" >&2 + exit 1 + + - name: Compose build (no GPU) + run: docker compose -f docker/docker-compose.yml build diff --git a/.github/workflows/eval-nightly.yml b/.github/workflows/eval-nightly.yml index d0b634f9..2956acd4 100644 --- a/.github/workflows/eval-nightly.yml +++ b/.github/workflows/eval-nightly.yml @@ -5,13 +5,18 @@ # mirrors without keys stay green. # # On gate failure, opens or updates a tracking issue with the summary. +# +# Schedule disabled 2026-09-13: Actions has neither ANTHROPIC_API_KEY nor +# OPENROUTER_API_KEY, so the cron only produced empty green skips. Restore the +# cron below when a key is configured: +# on: +# schedule: +# - cron: "15 6 * * *" # 06:15 UTC daily +# workflow_dispatch: name: Eval Nightly on: - schedule: - # 06:15 UTC daily - - cron: "15 6 * * *" workflow_dispatch: concurrency: @@ -30,7 +35,7 @@ jobs: contents: write issues: write steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - name: Check secrets id: secrets @@ -56,14 +61,14 @@ jobs: - name: Install uv if: steps.secrets.outputs.present == 'true' - uses: astral-sh/setup-uv@v5 + uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2 with: enable-cache: true cache-dependency-glob: "uv.lock" - name: Set up Python if: steps.secrets.outputs.present == 'true' - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" @@ -105,7 +110,7 @@ jobs: - name: Open or update failure issue if: steps.secrets.outputs.present == 'true' && failure() - uses: actions/github-script@v7 + uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 with: script: | const title = 'Eval nightly Tier B gate failure'; diff --git a/.gitignore b/.gitignore index 4df50aa4..c5b5516d 100644 --- a/.gitignore +++ b/.gitignore @@ -237,7 +237,11 @@ evals/results/ # Eval harness live corpus / working dirs (fixtures, golden, cache, baselines stay tracked) evals/traces/ evals/annotations/ +evals/golden/.validation_log.jsonl evals/samples/ # Generated performance artifacts (regenerate with AI_TEAM_BENCHMARK_FULL=1) docs/benchmark_results.json + +# Files delivered into the chat by Claude land here; canonical copies live in docs/ +Claude outputs/ diff --git a/.kiro/specs/eval-claim-surfaces/README.md b/.kiro/specs/eval-claim-surfaces/README.md new file mode 100644 index 00000000..c5258cc8 --- /dev/null +++ b/.kiro/specs/eval-claim-surfaces/README.md @@ -0,0 +1,137 @@ +# Spec: `eval-claim-surfaces` + +Kiro-style three-document spec for the three free fixes that stop this repo teaching its own +defects. Read in order: + +1. **[`requirements.md`](./requirements.md)** — 11 requirements with EARS acceptance criteria, + constraints, and non-goals. +2. **[`design.md`](./design.md)** — where each field lands, the one-stamp rule, the + `na_reason` vocabulary, schema/baseline compatibility, and four open decisions (§9). +3. **[`tasks.md`](./tasks.md)** — 4 phases, 26 tasks, each with a definition of done and + requirement traceability. + +## What this spec owns, and what it only sequences + +This is a **slice spec**. Almost every fix in it already has a requirement home: + +| Fix | Owning requirement | Owning task | +| --- | --- | --- | +| Corpus source is the run-record tree | `eval-methodology-alignment` R5 | alignment 2.4 | +| Corpus-kind on every rate | `eval-methodology-alignment` R15.1–R15.2 | alignment 8.3 | +| Generic metrics demoted | `eval-methodology-alignment` R15.3 | alignment 8.4 | +| Dated tables + 30-day CI warning | `eval-methodology-alignment` R15.4 | alignment 8.5 | +| Staleness in `index stats` | `eval-methodology-alignment` R14.3–R14.4 | alignment 8.1 | +| Cadence written down | `eval-methodology-alignment` R14.1–R14.2 | alignment 8.2 | +| `not_applicable` first-class | `eval-coverage` R3 | coverage 4.1–4.3b | +| `EVIDENCE-STARVED`, verdict suppression | `eval-coverage` R11, R2.4–R2.6 | coverage 4.4–4.6 | + +**None of those are restated.** This spec references them and puts them in one executable +order. It owns exactly two things nothing else owns: + +- **R1** — that no tracked document may present a command whose result the repo already knows + to be wrong. `README.md:337` and `evals/README.md:12` both still tell a new reader to run + `trace backfill --workspace-root ./workspace`, which is the exact invocation the + 2026-09-13 audit identified as the root cause. The quickstart reproduces the bug. +- **R2** — that `FIXTURE-ONLY` / `CORPUS` / `LIVE` and `EVIDENCE-STARVED` get **one** + implementation, in `evals/coverage.py`, consumed by `evals/report.py`. Both already exist + there as of `f883d75`. Alignment 8.3 and coverage R11.2 would each satisfy independently by + writing their own, and two stamp implementations that disagree is worse than none. + +## The one-line version + +`eval-methodology-alignment` fixes the inputs. `eval-coverage` fixes the instruments. This +fixes the **surfaces** — the three places a human meets the eval system and currently learns +something false from it: the quickstart, the report, and the absence of a clock. + +## Why these three, and why now + +All three are **$0.00**, none is blocked on `harness/telemetry.py`, and each one removes a +specific false lesson that survives every future improvement: + +| Surface | What it teaches today | Evidence | +| --- | --- | --- | +| Quickstart | "point backfill at `./workspace`" | `README.md:337`, `evals/README.md:12` — the audited root cause, still the documented path | +| Report | "FM-006 occurs at 0.976 (n=84)" | `evals/results/tierA_4671d49ae035_eb36475d7c/report.md` §4 — fixture arithmetic rendered with a Wilson CI and no corpus stamp | +| Cadence | nothing; there is no clock | `index stats` prints `backend/scenario/status/count` and nothing else; `EVAL_METHODOLOGY.md` describes a loop with no rhythm | + +The report is the sharpest. A reader — including the author six months out — sees seventeen +failure-mode rates with confidence intervals and reasonably concludes the system has been +measured. What the table actually reports is that 20 checks behaved as written against 94 +synthetic fixture documents carrying 72 distinct ids, with 76.3% of results abstaining. Every +number is correct. The composition is a claim nobody made on purpose. + +## The asymmetry that makes this worth doing before the corpus lands + +Fixing the corpus makes the numbers *true*. It does not make them *readable*, and a true +number rendered the current way is more dangerous than a false one, because the corpus stamp +is the only thing that would have told a reader which they were looking at. + +``` + today FIXTURE-ONLY data → unstamped renderer → reader infers a measurement + after 2.4 CORPUS data → unstamped renderer → reader infers a measurement + and is right for the wrong reason +``` + +The renderer has to learn to say which corpus it read before the corpus changes underneath it, +or the change is invisible at exactly the moment it matters most. + +## Scope decision (2026-09-14) + +Asked and answered: **full honest renderer**, not the minimal stamp. That means all of +`eval-coverage` Phase 4 — `na_count`, `n_decided`, all 17 FM rows carrying `liveness`, +distinct-`trace_id` counting, `EVIDENCE-STARVED` at the 50% threshold, headline +blind/unreachable counts, verdict suppression, and the liveness table printed in CI — plus +alignment 8.3 and 8.4. Rationale: the stamp alone tells a reader the corpus is fixtures while +still showing them `0.976 (n=84)` over a corpus of 72 distinct traces. The denominator defect +and the stamp defect are the same defect seen twice; splitting them ships half a fix and +leaves the more misleading half in place. + +## Constraints baked in + +| | | +| --- | --- | +| Location | extends `evals/aggregate.py`, `evals/report.py`, `evals/coverage.py`, `evals/cli.py`, `evals/store.py`, `docs/`; no new package | +| Cost | **$0.00.** Every task is offline, deterministic, and runs on the corpus already on disk. | +| One implementation per stamp | corpus kind and `EVIDENCE-STARVED` live in `evals/coverage.py` and are imported, never re-derived (R2) | +| No vendor | no Braintrust / LangSmith / Arize / DeepEval, consistent with the other four specs | +| Renderer-enforced, never prose | every stamp is a computed field with a test, not a sentence in a markdown file (R2.5) | +| Determinism preserved | `tests/integration/evals/test_tier_a_integration.py` compares two runs to each other; every field added here SHALL be deterministic (R10.2) | +| Staleness never gates | R8.4, following alignment R14.4. The one CI gate is R1.5's doc-command audit, and it fails on a known-wrong command, not on a data state. | +| Evals never reach agents | unchanged from `eval-coverage` R14 — nothing here puts a rate, stamp or FM id into an agent-facing path | +| Out of scope | the telemetry writers, the re-index implementation, the human annotation pass — see Non-goals | + +## Dependencies + +**Nothing here is blocked.** Two soft couplings: + +- **R1's end state** wants alignment task 2.4 (make `output/runs/` the primary source). Until + that lands, R1 is satisfied by the caveat path (R1.3): the quickstart carries a one-line + warning naming the defect and the task, rather than a command the repo knows returns + shells. Task 1.1 delivers the caveat; task 1.4 removes it and closes R1 when 2.4 lands. +- **R6's `unreachable` counts** consume `evals/coverage.py`, which is committed (`f883d75`). + No dependency on the uncommitted workbench. + +`evals/ui/` is still untracked as of this writing, which is a separate carry-forward +(2026-09-14 journal §8 item 2) and not this spec's business. + +## Minimum defensible slice + +**Phase 1, plus tasks 2.1, 2.2 and 3.1.** The quickstart stops teaching the bug, every rate +carries its corpus kind and its decided denominator, and `index stats` reports how stale the +corpus is. Three files, no schema migration, about half a day. + +If even that is too long: **task 1.1 alone.** Two lines of markdown in two files. It is the +only item in any of the five eval specs where the cost is measured in minutes and the thing +being prevented is a new reader reproducing a defect that took two months to find. + +## Executing this with Cursor + +``` +Read .kiro/specs/eval-claim-surfaces/requirements.md and design.md for context. +Implement task 2.1 from .kiro/specs/eval-claim-surfaces/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +All four phases are free and safe to run unattended. **No task in this spec spends money, and +no task in this spec writes to `evals/annotations/`, `evals/golden/` or `evals/traces/`.** diff --git a/.kiro/specs/eval-claim-surfaces/design.md b/.kiro/specs/eval-claim-surfaces/design.md new file mode 100644 index 00000000..72b5f5d6 --- /dev/null +++ b/.kiro/specs/eval-claim-surfaces/design.md @@ -0,0 +1,269 @@ +# Design — Eval Claim Surfaces + +**Spec ID:** `eval-claim-surfaces` +**Reads with:** [`requirements.md`](./requirements.md). Requirement ids below are this spec's +unless prefixed — `alignment R15.1`, `coverage R3.2`. + +--- + +## 1. The shape of the change + +Three surfaces, three mechanisms, no new module. + +``` + surface mechanism files + ───────────────────────────────────────────────────────────────────────────────────── + quickstart caveat now, corrected command after 2.4, README.md, evals/README.md, + CI audit forever scripts/audit_doc_commands.py + + report SuiteReport gains coverage + stamps; evals/aggregate.py, + four renderers learn to show a denominator evals/report.py, + and three orthogonal stamps evals/checks/base.py + + clock index stats gains staleness + floor evals/store.py, evals/cli.py, + shortfall; docs gain a dated rhythm docs/EVAL_METHODOLOGY.md +``` + +The report is where nearly all the code is. The other two are a markdown edit with a test and +a SQL query with a formatter. + +## 2. The one-stamp rule (R2) + +`evals/coverage.py` already owns both vocabularies as of `f883d75`: + +```python +CorpusKind = Literal["FIXTURE-ONLY", "CORPUS", "LIVE"] # coverage.py:82 +EVIDENCE_STARVED_THRESHOLD = 0.50 # coverage.py:77 +class CoverageReport(BaseModel): + corpus_kind: CorpusKind # coverage.py:346 + def stamps(self) -> list[str]: ... # coverage.py:366 +``` + +`SuiteReport` imports them. It does not define a parallel enum, and `evals/report.py` never +writes the string `"FIXTURE-ONLY"` itself — it renders `report.stamps`. + +**Why this is worth a requirement.** `alignment R15.1` says every rate renders its corpus +kind. `coverage R11.2` says the `EVIDENCE-STARVED` stamp is renderer-enforced. Executed +independently, each is satisfied by a local implementation in the file its author was editing, +and the failure mode is not that one is wrong — it is that they drift and a reader has no way +to know which one they are looking at. One import is cheaper than one reconciliation. + +The enforcement is a test, not a review convention: + +```python +def test_no_second_corpus_kind_literal() -> None: + """R2.2 — the vocabulary lives in coverage.py and is imported.""" + offenders = [ + p for p in Path("evals").rglob("*.py") + if p.name != "coverage.py" + and not p.match("*/tests/*") + and "FIXTURE-ONLY" in p.read_text() + ] + assert offenders == [] +``` + +### 2.1 Three stamps, composed + +| Stamp | Owner | Answers | Computed from | +| --- | --- | --- | --- | +| `FIXTURE-ONLY` / `CORPUS` / `LIVE` | alignment R15 | what are the traces? | which root was scored | +| `NON-REPRESENTATIVE` | alignment R4 | is the corpus wide enough? | `CorpusProfile` vs R4 floors | +| `EVIDENCE-STARVED` | coverage R11 | could the instruments see? | `na` share excluding `not_in_scope` | + +`SuiteReport.stamps` is the sorted union of whichever apply. A Tier A run today earns all +three; the renderer shows all three. The order is fixed (corpus kind, then representativeness, +then evidence) so the field is deterministic per R10.2. + +## 3. `SuiteReport` v2 (R2.3, R10.1) + +Additive only. Five new fields, one bumped version: + +```python +class SuiteReport(BaseModel): + schema_version: int = 2 # was 1 (R10.1) + ... + coverage: CoverageReport | None = None # folded liveness for the scored corpus + stamps: list[str] = Field(default_factory=list) + n_files: int = 0 # R4.2 + n_distinct_traces: int = 0 # R4.2 + abstention: AbstentionSummary | None = None +``` + +```python +class AbstentionSummary(BaseModel): + """R3.6 / R6.1 — why the suite declined to answer.""" + n_results: int + n_na: int + n_na_in_scope: int # the EVIDENCE-STARVED denominator + share_in_scope: float + top_abstainers: list[TopAbstainer] # exactly three, or fewer if fewer checks abstained + +class TopAbstainer(BaseModel): + check_id: str + na_count: int + dominant_reason: NaReason +``` + +`ScorecardCell` and `FailureModeIncidence` each gain `na_count: int` and `n_decided: int` +(coverage R3.2). `n_decided` is `pass + fail` and is the only denominator any renderer is +permitted to divide by. + +A version-1 report loads because every added field has a default. `schema_version` is bumped +so a consumer can branch, not so a loader can refuse. + +## 4. `na_reason` as a closed vocabulary (R5) + +Today `CheckResult` carries `evidence_text: str` and nothing structured, and +`evals/coverage.py:465` clusters those strings with `Counter.most_common(3)`. That works until +someone rewords a message, at which point one cause silently becomes two. + +```python +NaReason = Literal[ + "missing_span_type", # the trace has no spans of a type the check requires + "missing_artifact", # a required file is absent from the workspace + "missing_raw_key", # raw_result lacks a key the check reads + "missing_scalar", # a cost/duration/count field is None + "not_in_scope", # this check was never meant to score this trace +] +``` + +`not_in_scope` carries the design weight. It is the difference between *blind* and *not +asked*, and conflating them is how `CHK-required-artifacts` contributed 65 of 182 headline +failures by firing on fixtures that were never going to have artifacts. Per R5.3 and +coverage R11.1 it is excluded from the `EVIDENCE-STARVED` denominator — hence +`n_na` and `n_na_in_scope` as separate fields rather than one number. + +**Migration.** `evals/checks/base.py:82` already centralizes the `na(...)` constructor +alongside `passed(...)`. The parameter becomes required there, and the twenty registered checks +are updated call site by call site. R5.5's guard is the safety net: an outcome-for-outcome diff against +`evals/results/tierA_4671d49ae035_eb36475d7c/report.json` must be empty. This change is +allowed to add a field; it is not allowed to change a verdict. + +## 5. Renderer rules (R3, R6) + +One function, four formats. The rules are properties of the data, not of the output: + +1. **No bare rate.** A rate renders as `value (n_decided=, na=)`. `evals/report.py` + gets a `_rate_cell(rate, n_decided, na_count)` helper and no format string bypasses it. +2. **Suppress under ten.** `n_decided < 10` renders `n=` with no percentage and no CI + (R3.3). This is the rule that removes **eleven of the current seventeen** incidence + percentages — more once task 2.3's distinct-id collapse halves the duplicated denominators — + and that removal is the point: eleven of twenty checks are decided by one fixture pair. +3. **Every active FM gets a row.** Built from the taxonomy loader, not from observed results, + so a mode nothing could see is a row reading `liveness: unreachable` rather than an absence + (R3.4). `origin: hypothesis` modes are filtered out before this step and rendered, if at + all, in a separate unpromoted section (R3.5). +4. **Headline names the blind.** `FAIL — 182 failed, 1435 abstained (76.3%), 2 checks + unreachable` replaces `FAIL — 182 failed check(s) across suite`. The verdict downgrades to + `pass (instruments incomplete)` when any check is `unreachable` (R6.3). + +### 5.1 The HTML renderer + +It is the format a human opens and the only one with no test (R11.4). Three changes beyond +the shared rules: a stamp row directly under the headline; `not_applicable` as a column in +both tables; and the existing light-only palette gains a `prefers-color-scheme` block, since +the report is read at 11pm more often than at 11am. No new dependency — the template stays +inline Jinja2 with inline SVG, per `eval-harness` R13. + +## 6. The doc-command audit (R1.5) + +A twenty-line script, not a linter plugin: + +```python +# scripts/audit_doc_commands.py +EXEMPT = ("docs/journal/", "docs/eval-runs/", "docs/posts/", + "docs/showcase/", "docs/campaign/", ".archive/") +PATTERN = re.compile(r"trace\s+backfill[^\n]*--workspace-root\s+\./workspace") +``` + +Tracked `*.md` files outside `EXEMPT` are scanned; a match is an error naming the file, the +line, and the owning task. Runs in the existing lint job — no new workflow. + +**Why a gate here and nowhere else.** Every other signal in this spec describes a state of the +data, and gating on data states is how a team learns to ignore a gate. This one fires on a +string in a tracked file that the repo has already documented as wrong in three places. It can +only be tripped by adding the defect back. + +The exemption list is the interesting half. `docs/showcase/starved-harness.html:657` quotes +the wrong command *as the finding*. Rewriting it would destroy the artifact. The rule is by +path rather than by content because content-based exemption drifts into argument. + +## 7. The clock (R8, R9) + +`index stats` currently renders `store.stats()` — a four-column `GROUP BY`. It gains a header +block above the existing table: + +``` +corpus 50 traces (72 distinct ids in 94 fixture files) +span 2026-08-16 → 2026-08-16 (1.97 seconds) +floors UNMET: traces 50/100 · backends 1/3 · scenarios 0/4 · statuses 1/2 · span 0/14d +annots NONE — evals/annotations/ is empty; 50 traces created since +next open coding overdue (no anchor date recorded) +``` + +Two new store methods — `corpus_profile()` and `annotation_staleness()`. The first already has +a home in alignment design §4.2 as `CorpusProfile`; this spec consumes it if it exists and +computes the four floor counts inline if it does not, which keeps the task unblocked either way. + +R8.3 is a small thing that matters: empty annotations render as `NONE`, never `0 days`. A zero +next to a date format reads as recent, and the state this command exists to expose is exactly +the one where nothing has happened. + +### 7.1 The cadence, anchored + +R9.5 rejects "every 2–4 weeks" without an anchor. `EVAL_METHODOLOGY.md` gains: + +```markdown +## Cadence + +| Pass | Period | Trigger | +| --- | --- | --- | +| Open coding | ≥100 fresh traces every 2–4 weeks | calendar | +| Outlier review | 10–20 traces weekly between passes | calendar | +| Forced review | regardless of calendar | default model changes · backend added · harness component toggled | + +**Anchor:** no pass has occurred. First pass due on the first date the corpus clears its R4 +floors. `index stats` reports the shortfall. +``` + +The honest anchor is the absence of one. Writing "first pass due 2026-10-01" would be a +schedule invented to fill a table; writing what the anchor depends on is checkable and puts +the dependency where the reader can see it. + +## 8. Order of operations + +Phases 2 and 3 are independent. Phase 1 is independent of both. The only internal ordering +constraint is within Phase 2: `na_reason` (R5) before the abstention summary (R3.6, R6.1), +because the summary groups by the code. + +``` + Phase 1 quickstart + audit ── independent, ~1 hour + Phase 2 na_reason → fields → renderers → stamps ── sequential internally + Phase 3 index stats + cadence docs ── independent + Phase 4 baseline, determinism, coverage-of-coverage ── after 2 +``` + +## 9. Open decisions + +**9.1 Does `n_decided < 10` suppress the CI summary line too, or only the report?** Leaning +report-and-summary both, because `summary.txt` is what gets pasted into a PR comment and a +suppressed number that reappears one surface over is not suppressed. Decide before task 2.4. + +**9.2 Should `EVIDENCE-STARVED` suppress the verdict the way `unreachable` does?** Leaning no, +matching the 2026-09-14 journal's reasoning on `thin`: a starved corpus is already stamped and +the stamp is the signal. An unreachable check is a code defect and gets the stronger treatment. +Revisit once the corpus clears its floors — a suite still `EVIDENCE-STARVED` on a +representative corpus means something different. + +**9.3 Where does `CHK-listener-self-trigger` land?** It inspects `ai_team.core.flow_wiring` +rather than a Trace, so its 83 fixture passes are 83 copies of one fact and R4's distinct-id +collapse will not fix that. Carried from the 2026-09-14 journal §9 and `eval-coverage` design +§9.6 — leaning `static: true`, which would exclude it from denominators here. **This spec does +not decide it**, but R3's denominators change meaning depending on the answer, so the flag must +exist before task 2.2 even if its value is provisional. + +**9.4 Does the HTML dark-mode block belong in this spec at all?** It is a real improvement to +the surface a human reads, and it is also scope creep into `ui-refinement`'s token system, +which the eval report does not use. Leaning yes because the report is standalone by design +(`eval-harness` R13: no CDN, no build) and borrowing the app's tokens would couple them. +Cheap to drop from task 2.5 if it slows the phase. diff --git a/.kiro/specs/eval-claim-surfaces/requirements.md b/.kiro/specs/eval-claim-surfaces/requirements.md new file mode 100644 index 00000000..d1918764 --- /dev/null +++ b/.kiro/specs/eval-claim-surfaces/requirements.md @@ -0,0 +1,339 @@ +# Requirements — Eval Claim Surfaces + +**Spec ID:** `eval-claim-surfaces` +**Status:** Draft for implementation +**Owner:** Rick Zakharov +**Target repo:** `ai-team` (extends `evals/aggregate.py`, `evals/report.py`, `evals/coverage.py`, +`evals/cli.py`, `evals/store.py`, `docs/`) +**Created:** 2026-09-14 +**Cost:** $0.00 — every requirement is satisfied offline against files already on disk. + +**Depends on:** [`../eval-harness/`](../eval-harness/) — Trace boundary (R1), check registry +(R5), `SuiteReport` (R13), tiers (R11). [`../eval-methodology-alignment/`](../eval-methodology-alignment/) +— corpus source (R5), review cadence (R14), claim discipline (R15). +[`../eval-coverage/`](../eval-coverage/) — evidence declarations (R1), liveness (R2), +`not_applicable` as first-class (R3), coverage claim discipline (R11). +**None of those are restated here.** Where a criterion below implements one of them it cites +it and adds only what the citation does not already fix. + +--- + +## Introduction + +Three specs already describe a correct eval system. This one is about the three places a +human touches it and learns something false. + +`eval-methodology-alignment` fixes the **inputs** — the corpus is built from the wrong tree +and phase telemetry is asked of the model. `eval-coverage` fixes the **instruments** — checks +read spans no code writes, and 76.3% of results abstain unreported. Both are correct and +neither covers the following: + +1. **`README.md:337` and `evals/README.md:12` tell a new reader to run the audited defect.** + `uv run python -m evals.cli trace backfill --workspace-root ./workspace` is, verbatim, the + invocation that produced 50 traces with 0 spans. It is still the documented quickstart. The + audit is written up in three places; the command a reader actually copies is unchanged. + +2. **The report renders fixture arithmetic in the visual grammar of a measurement.** + `evals/results/tierA_4671d49ae035_eb36475d7c/report.md` §4 publishes + `FM-006 | harness | 0.976 (n=84, 95% CI [0.917, 0.993]) | — | 82`. Every figure is + arithmetically correct. There is no corpus stamp, no abstention count, and the `n=84` counts + files rather than distinct `trace_id`s. Seven of seventeen active failure modes are absent + from the table with no indication whether they did not occur or could not be seen. + +3. **There is no clock.** `index stats` prints four columns of counts. Nothing reports days + since the last annotation, traces added since, or which R4 diversity floor is unmet. + `EVAL_METHODOLOGY.md` describes the loop and never says how often it turns. The + methodology's continuous half exists only as unimplemented spec text — alignment Phase 8, + 0 of 47 tasks done. + +### The composition problem + +No single number in the current report is wrong. The report as a whole asserts something +nobody decided to assert, because the renderer has no vocabulary for the three facts that +would qualify it: + +``` + what the reader sees what is true + ──────────────────────────────────────────────────────────────────────────── + 17 FM rates with Wilson CIs 20 checks behaved as written + n=84 72 distinct traces in 94 files + 0.976 1435 of 1880 results abstained + FAIL — 182 failed checks 65 of the 182 are one check out of scope +``` + +This spec's entire content is: make the renderer able to say the right-hand column, make the +quickstart stop teaching the defect, and start a clock. Nothing here improves the eval +system's accuracy. It makes the system's own output legible, which is the precondition for +anyone — including its author — noticing the next defect faster than two months. + +--- + +## R1 — No tracked document presents a command the repo knows to be wrong + +**User story.** As a new reader, I want the quickstart to work, because the first command I run +is the one that shapes what I believe the system does. + +### Acceptance criteria + +1. `README.md` and `evals/README.md` SHALL NOT present `trace backfill --workspace-root + ./workspace` as the corpus-building path. +2. WHEN `eval-methodology-alignment` task 2.4 has landed, every runnable corpus-building + snippet in a tracked document SHALL invoke the run-record tree as its primary source. +3. UNTIL task 2.4 lands, each such snippet SHALL carry a one-line caveat naming the defect and + linking the owning task, so that a reader who runs it knows what they will get. A correct + caveat satisfies this requirement; a silent wrong command does not. +4. Historical documents SHALL be exempt by path: `docs/journal/`, `docs/eval-runs/`, + `docs/posts/`, `docs/showcase/`, `docs/campaign/` and `.archive/` record what was run at + the time and SHALL NOT be rewritten. A defect narrative quoting the wrong command is + evidence, not an error. +5. CI SHALL fail WHEN a tracked document outside those paths contains a corpus-building + command whose source argument resolves to `./workspace`. This is the one gate this spec + adds: it fires on a known-wrong command, never on a data state. +6. `docs/GETTING_STARTED.md` SHALL link the eval entry point at the same prominence as the + harness and guardrail docs, and that link SHALL reach a page whose first runnable command + satisfies R1.2 or R1.3. + +## R2 — Corpus kind and evidence stamps have exactly one implementation + +**User story.** As a maintainer, I want one definition of `FIXTURE-ONLY` in the codebase, +because two renderers that disagree about what corpus they read is a worse defect than neither +of them saying. + +### Acceptance criteria + +1. `CorpusKind` and the `EVIDENCE-STARVED` threshold SHALL remain defined in + `evals/coverage.py` — where they already exist as of `f883d75` — and SHALL be imported by + every other consumer. +2. `evals/report.py` and `evals/aggregate.py` SHALL NOT define, duplicate, or re-derive either. + A second string literal `"FIXTURE-ONLY"` outside `evals/coverage.py` and its tests SHALL + fail a unit test. +3. `SuiteReport` SHALL carry a `coverage` field holding the folded `CoverageReport` for the + corpus it scored, and a `stamps: list[str]` field holding the computed stamps in a stable + order. +4. The `NON-REPRESENTATIVE` stamp (`eval-methodology-alignment` R4.4) SHALL compose with the + two stamps here rather than replace them. All three are orthogonal — corpus kind is *what + the traces are*, `NON-REPRESENTATIVE` is *the corpus is too narrow*, `EVIDENCE-STARVED` is + *the instruments could not see* — and a report SHALL show every stamp that applies. +5. Every stamp SHALL be a computed field with a test asserting the boundary condition. No stamp + SHALL be satisfied by a sentence in a markdown file. + +## R3 — Abstention is rendered wherever a rate is rendered + +**User story.** As a reviewer, I want to see how many results a rate was computed from, so that +a denominator cannot quietly shrink to two. + +Implements `eval-coverage` R3.1–R3.4 and R3.6 in the `SuiteReport` renderer. The criteria there +define the behavior; these define where it lands and what proves it. + +### Acceptance criteria + +1. `ScorecardCell` and `FailureModeIncidence` SHALL carry `na_count` and `n_decided`, and all + four renderers — `report.json`, `report.md`, `report.html`, `summary.txt` — SHALL show both. +2. No renderer SHALL emit a percentage or a rate without displaying the denominator it was + computed over, in the same cell or the adjacent one. +3. WHERE `n_decided < 10`, the renderer SHALL emit `n=` and SHALL NOT emit a percentage or + a confidence interval. A Wilson interval over four decided results is arithmetic performing + a confidence it does not have. +4. `failure_mode_incidence` SHALL contain one row per **active** failure mode, including modes + with zero incidence, each carrying a `liveness` field from `evals/coverage.py`. The 1.2.0 + report emits 10 rows for 17 active modes; a reader cannot currently distinguish "did not + occur" from "nothing could see it". +5. Failure modes with `origin: hypothesis` SHALL be excluded from the active set and, WHERE + shown at all, SHALL be rendered in a separate section labeled as unpromoted — consistent + with `eval-coverage` R9. FM-019…FM-024 SHALL NOT appear in an incidence table. +6. The per-check abstention summary SHALL name the three checks contributing the most + `not_applicable` results and their dominant reason code, so the single largest cause of + abstention is visible without reading 1435 strings. + +## R4 — A corpus is counted by distinct `trace_id` + +**User story.** As a reader, I want `n` to mean the number of traces, because it is the +denominator under every rate in the report. + +Implements `eval-coverage` R3.7–R3.8. + +### Acceptance criteria + +1. Tier A SHALL count its corpus by distinct `trace_id`. `evals/fixtures/traces/` holds 94 + files carrying **72 distinct ids**; `tier_a.load_fixture_traces` globs by filename and + scores both copies of each of the 22 duplicates. +2. WHERE `n_files` and `n_distinct_traces` differ, the report SHALL name both. +3. Tier A's denominators and `TraceStore.rebuild_index` SHALL NOT disagree about the size of a + corpus. `TraceStore.write` already refuses a duplicate id; the fixture loader SHALL adopt + the same rule. +4. CI SHALL warn WHEN a fixture directory contains two files sharing a `trace_id`. +5. The 22 duplicated ids SHALL be named in the task record, and every `2 pass / 2 fail` row in + the current report SHALL resolve to `1 pass / 1 fail` after this requirement lands. That + collapse is the expected outcome, not a regression, and the baseline note SHALL say so. + +## R5 — `na_reason` is a closed vocabulary, not a string + +**User story.** As a maintainer, I want to aggregate abstentions by cause, because clustering +free text is how the current top-reasons table works and it is one rephrasing away from +splitting a cause in two. + +Implements `eval-coverage` R3.5–R3.6. + +### Acceptance criteria + +1. `CheckResult` SHALL carry `na_reason: NaReason | None` drawn from the closed vocabulary + `missing_span_type`, `missing_artifact`, `missing_raw_key`, `missing_scalar`, + `not_in_scope`, alongside its existing human-readable `evidence_text`. +2. Every `not_applicable` result produced by a registered check SHALL set it. A unit test SHALL + assert that no registered check can return `not_applicable` with `na_reason: None`. +3. `not_in_scope` SHALL be excluded from the `EVIDENCE-STARVED` denominator, per + `eval-coverage` R11.1. A check declining a trace it was never meant to score is not + blindness, and conflating the two is how `CHK-required-artifacts` produced 65 of 182 + headline failures. +4. `evals/coverage.py` SHALL aggregate by code and MAY keep the text sample as a secondary + display. The code SHALL be the grouping key. +5. Adding the field SHALL NOT change any existing check's `outcome`. A task SHALL assert + outcome-for-outcome equality against the committed report before and after. + +## R6 — The headline and the verdict cannot outrun the instruments + +**User story.** As a reader of CI, I want a green suite to be unable to claim more than it +measured, because a badge outlives every caveat in the prose around it. + +Implements `eval-coverage` R2.4–R2.6 and R11.1–R11.3. + +### Acceptance criteria + +1. A report WHERE `not_applicable` exceeds 50% of all check results, excluding `not_in_scope`, + SHALL carry `EVIDENCE-STARVED` naming the share and the three largest abstainers. +2. The headline SHALL name the count of `blind` and `unreachable` checks whenever either is + non-zero, in the same sentence as the pass/fail counts. +3. WHERE any check is `unreachable`, the verdict SHALL render as `pass (instruments + incomplete)` and SHALL NOT render as a bare `pass`. A blind check is a data state; an + unreachable one is a code defect, and a green verdict over a code defect is the failure this + criterion exists to prevent. +4. The corpus kind SHALL be shown adjacent to the liveness summary, so a fixture result cannot + be read as a coverage claim. +5. CI SHALL print the liveness table on every Tier A run and SHALL NOT gate on it. +6. Threshold behavior SHALL be tested at 49.9% and 50.1%. + +## R7 — Generic metrics are labeled as a sampling signal + +**User story.** As a reader, I want an off-the-shelf metric to look like a way of choosing +traces to read, not like a quality result. + +Implements `eval-methodology-alignment` R15.3. + +### Acceptance criteria + +1. `evals/metrics.py` output SHALL be rendered under a heading naming it a sampling signal. +2. No generic metric SHALL appear as a headline figure, in `summary.txt`, or in any CI summary + line. +3. WHERE a generic metric is used to select traces for annotation, the sample manifest SHALL + record which metric and threshold selected each trace, so a later reader can see the + selection was not random and correct for it. + +## R8 — Staleness is reported, never gated + +**User story.** As the owner, I want to see how old the corpus and the annotations are without +asking, because the thing that decays silently is the thing that decays. + +Implements `eval-methodology-alignment` R14.3–R14.4. + +### Acceptance criteria + +1. `index stats` SHALL report days since the most recent `AnnotationRecord`, the count of + traces created since that record, and the newest and oldest trace timestamps. +2. `index stats` SHALL report, per R4 diversity floor, what the corpus is currently short by, + naming the floor and the shortfall. +3. WHERE `evals/annotations/` is empty, the output SHALL say so explicitly rather than + rendering a null or a zero. Zero annotations and a recent annotation are different states + and SHALL be distinguishable at a glance. +4. Nothing in this requirement SHALL gate. Staleness is reported; the exit code is unchanged. +5. The output SHALL remain a single screen of plain text. This is the command the owner runs + most and its value is that it costs nothing to read. + +## R9 — The cadence is written down + +**User story.** As the owner, I want the review rhythm recorded where the methodology is +recorded, because a loop with no period is a diagram, not a practice. + +Implements `eval-methodology-alignment` R14.1–R14.2 and R15.4. + +### Acceptance criteria + +1. `docs/EVAL_METHODOLOGY.md` SHALL state the recurring review: ≥100 fresh traces open-coded + every 2–4 weeks, and 10–20 outlier traces reviewed weekly between passes. +2. It SHALL state the event triggers that force a review regardless of calendar: the default + model changes, a backend is added, or a harness component is enabled or disabled. +3. It SHALL carry a dated corpus-state table matching current `index stats` output, and a + dated liveness table matching current `coverage liveness` output. +4. CI SHALL warn WHEN either table is more than 30 days older than the newest trace or the + newest report. Warn, not fail. +5. The cadence SHALL be stated as a rhythm with a next date, not as an aspiration. "Every 2–4 + weeks" with no anchor date is unfalsifiable and SHALL NOT satisfy this requirement. + +## R10 — Schema, baseline and determinism compatibility + +**User story.** As a maintainer, I want this spec's fields to be additive, because the $0 gate +and its baseline are the only working part of the eval system today. + +### Acceptance criteria + +1. `SuiteReport.schema_version` SHALL be bumped to `2`, and a reader SHALL be able to load a + version-1 report without error. +2. Every field added SHALL be deterministic. `tests/integration/evals/test_tier_a_integration.py` + compares two runs of the same corpus to each other; that equality SHALL hold. +3. `evals/baselines/tier_a.json` records `check_outcomes` per check id and nothing derived from + denominators, so the gate SHALL be unaffected by R3 and R4. A task SHALL verify this rather + than assume it. +4. WHERE R4's distinct-id collapse changes a check's fixture outcome, the baseline SHALL be + regenerated with a `reason` naming R4 as the cause. +5. `report_json_without_generated_at` SHALL remain the determinism comparator; no field added + here SHALL carry a timestamp, a path outside the repo, a hostname, or an iteration-ordered + collection. + +## R11 — Test hygiene + +**User story.** As a maintainer, I want this spec tested to the standard of the specs it +completes. + +### Acceptance criteria + +1. No test SHALL write to `evals/annotations/`, `evals/golden/`, or `evals/traces/`. Tests + SHALL use `tmp_path` roots. +2. Every threshold and suppression path added here SHALL reach 100% branch coverage: the + 50% `EVIDENCE-STARVED` boundary, the `n_decided < 10` suppression, the `unreachable` + verdict downgrade, the empty-annotations branch, and the doc-command audit. +3. The doc-command audit (R1.5) SHALL have a fixture proving it fires on the current + `README.md:337` text and does not fire on a `docs/journal/` path. +4. Renderer changes SHALL be asserted against all four output formats, not only markdown. The + HTML renderer is the one a reader opens and the one with no test today. + +--- + +## Constraints + +| | | +| --- | --- | +| Location | `evals/aggregate.py`, `evals/report.py`, `evals/coverage.py`, `evals/cli.py`, `evals/store.py`, `evals/checks/base.py`, `docs/`; no new package | +| Cost | **$0.00.** No task spends, and no task requires a key. | +| One implementation per stamp | R2. Corpus kind and `EVIDENCE-STARVED` are imported from `evals/coverage.py`, never re-derived | +| No vendor in the critical path | no Braintrust / LangSmith / Arize / DeepEval, consistent with the other four specs | +| Additive only | no existing check's `outcome` changes (R5.5); no existing requirement is restated | +| One gate | R1.5's doc-command audit. Liveness, staleness and stamps never gate | +| Evals never reach agents | unchanged from `eval-coverage` R14 | +| Renderer-enforced | every stamp is a computed field with a boundary test (R2.5) | + +## Non-goals + +- **Writing the telemetry.** `harness/telemetry.py` is `eval-methodology-alignment` Phase 1. + This spec renders what the instruments report, including that they report nothing. +- **The re-index itself.** Making `output/runs/` primary is alignment task 2.4. R1 documents + the correct path and gates the wrong one; it does not implement the parser. +- **The human annotation pass.** Alignment Phase 4, not delegable, and R8's staleness output + exists to make its absence visible rather than to substitute for it. +- **Promoting FM-019…FM-024.** R3.5 keeps them out of incidence tables. Promotion requires an + annotation record, per `eval-coverage` R9. +- **New checks.** Nothing here adds a detector. The claim is that the existing twenty become + legible, not that there should be more of them. +- **A coverage dashboard.** Considered and rejected on 2026-09-14 for the same reason the + workbench was built instead: an artifact about how starved the corpus is, in place of the + reading that would unstarve it. The report and `index stats` are surfaces that already + exist and are already read. +- **Multi-annotator κ.** A standing deviation, recorded in `eval-coverage`, not coverage. diff --git a/.kiro/specs/eval-claim-surfaces/tasks.md b/.kiro/specs/eval-claim-surfaces/tasks.md new file mode 100644 index 00000000..546b9c45 --- /dev/null +++ b/.kiro/specs/eval-claim-surfaces/tasks.md @@ -0,0 +1,249 @@ +# Tasks — Eval Claim Surfaces + +**Spec ID:** `eval-claim-surfaces` +**Read first:** [`requirements.md`](./requirements.md), then [`design.md`](./design.md). + +This file is a **living checklist**. Check a box only when its Definition of done is literally +true. Nothing in this spec spends money, so there are no human-triggered spend tasks to leave +open — but tasks 1.4 and 4.3 are blocked on work in another spec and stay open until it lands. + +--- + +## How to execute this plan + +One task per session. Do not start a second task before the first satisfies its Definition of +done and: + +```bash +uv run ruff check . && uv run ruff format --check . +uv run mypy src/ evals/ +uv run pytest tests/unit tests/integration/evals -q +``` + +Phases 1 and 3 are independent of everything. Phase 2 is internally sequential — `na_reason` +before the abstention summary, fields before renderers. Phase 4 runs after Phase 2. + +**Cost for every task in this file: $0.00.** + +--- + +## Phase 1 — The quickstart stops teaching the defect (free, no dependencies) + +- [ ] **1.1 Caveat the two live quickstarts** + - `README.md:337` and `evals/README.md:12` gain a one-line caveat naming the defect and + linking `eval-methodology-alignment` task 2.4. Wording states what the command returns + today (run workspaces, not run records) rather than describing it as broken. + - **Definition of done:** neither snippet presents `--workspace-root ./workspace` without + the caveat; a reader who runs it knows what they will get. No code changes. + - _Requirements: R1.1, R1.3_ + +- [ ] **1.2 `scripts/audit_doc_commands.py`** + - Per design §6. Scans tracked `*.md` outside the exemption paths for a corpus-building + command resolving to `./workspace`; errors name file, line and owning task. + - **Definition of done:** exits non-zero on the pre-1.1 `README.md` text, zero on the + post-1.1 text, and zero on every path in `EXEMPT`. + - _Requirements: R1.5, R1.4_ + +- [ ] **1.3 Wire the audit into the existing lint job** + - `.github/workflows/ci.yml`, lint job. No new workflow, no new job. + - **Definition of done:** the audit runs on every PR and fails the lint job on a match. + - _Requirements: R1.5_ + +- [ ] **1.4 Correct the commands — BLOCKED on `eval-methodology-alignment` task 2.4** + - Replace the caveated snippets with the corrected invocation against the run-record tree; + remove the caveats added in 1.1. + - **Definition of done:** every runnable corpus-building snippet in a non-exempt tracked doc + invokes the run-record tree as primary source, and `index stats` after following the + quickstart reports ≥3 backends. **Do not start before 2.4 lands** — writing a command for + a parser that does not exist is the defect this spec is about. + - _Requirements: R1.2_ + +- [ ] **1.5 Eval entry point in `GETTING_STARTED.md`** + - Raise the eval link from a single line at `:180` to the same prominence as the harness and + guardrail sections. + - **Definition of done:** the link is reachable without scrolling past the harness section + and lands on a page satisfying R1.2 or R1.3. + - _Requirements: R1.6_ + +--- + +## Phase 2 — The report tells the truth (free, internally sequential) + +- [ ] **2.1 `NaReason` vocabulary on `CheckResult`** + - Add the closed `Literal` per design §4; make the parameter required in + `evals/checks/base.py:82`'s `na(...)` constructor; update all twenty registered checks call site by + call site. + - **Definition of done:** no registered check can return `not_applicable` with + `na_reason: None` (asserted by a test that walks the registry); an outcome-for-outcome diff + against `evals/results/tierA_4671d49ae035_eb36475d7c/report.json` is **empty**. + - _Requirements: R5.1, R5.2, R5.5_ + +- [ ] **2.2 `na_count` and `n_decided` on the aggregate models** + - `ScorecardCell` and `FailureModeIncidence` per design §3. `n_decided = pass + fail` and is + the only permitted denominator. + - **Definition of done:** every rate in `report.json` carries both; no code path divides by + anything else. Requires design §9.3's `static` flag to exist, provisional value acceptable. + - _Requirements: R3.1, R3.2, coverage R3.1–R3.2_ + +- [ ] **2.3 Count by distinct `trace_id`** + - `tier_a.load_fixture_traces` adopts `TraceStore.write`'s duplicate rule; report `n_files` + and `n_distinct_traces` separately when they differ; CI warns on a fixture directory + holding two files with one id. + - **Definition of done:** the 22 duplicated ids are named in the commit message; Tier A + denominators and `rebuild_index` agree on corpus size; every `2 pass / 2 fail` row resolves + to `1 / 1`. That collapse is expected — see 4.1. + - _Requirements: R4.1, R4.2, R4.3, R4.4, R4.5_ + +- [ ] **2.4 Renderer rules across all four formats** + - `_rate_cell(rate, n_decided, na_count)` helper; no format string bypasses it; suppress + percentage and CI under `n_decided < 10`. Resolve design §9.1 (does suppression extend to + `summary.txt`) in this task. + - **Definition of done:** no rendered percentage anywhere without its displayed denominator; + the eleven incidence rows currently below `n=10` render `n=` with no percentage and no CI; + §9.1 is answered in the task record. + - _Requirements: R3.1, R3.2, R3.3_ + +- [ ] **2.5 One row per active failure mode, carrying liveness** + - Build the incidence table from the taxonomy loader rather than from observed results. + Filter `origin: hypothesis` out of the active set; render unpromoted modes, if at all, in a + separate labeled section. + - **Definition of done:** the 1.2.0 corpus produces 17 rows, not 10; an FM nothing could see + reads `liveness: unreachable` rather than being absent; FM-019…FM-024 appear in no + incidence table. + - _Requirements: R3.4, R3.5_ + +- [ ] **2.6 `AbstentionSummary` and the top-three abstainers** + - Per design §3. Group by `na_reason` code, not by text. `not_in_scope` excluded from + `n_na_in_scope`. + - **Definition of done:** the largest cause of abstention is visible without reading 1435 + strings; `CHK-required-artifacts`'s 65 out-of-scope results no longer inflate the starvation + share. + - _Requirements: R3.6, R5.3, R5.4_ + +- [ ] **2.7 `SuiteReport.coverage` + `stamps`, imported not re-derived** + - Fold the `CoverageReport` for the scored corpus onto the report; compose the three stamps + in fixed order per design §2.1. + - **Definition of done:** a Tier A fixture run renders `FIXTURE-ONLY`, + `NON-REPRESENTATIVE` and `EVIDENCE-STARVED` together; the R2.2 test finds no second + `"FIXTURE-ONLY"` literal outside `evals/coverage.py`; boundary tested at 49.9% / 50.1%. + - _Requirements: R2.1, R2.2, R2.3, R2.4, R2.5, R6.1, R6.4, R6.6_ + +- [ ] **2.8 Headline and verdict cannot outrun the instruments** + - Headline names abstention share and blind/unreachable counts; verdict renders + `pass (instruments incomplete)` when any check is `unreachable`. Resolve design §9.2 in the + task record. + - **Definition of done:** a suite with one unreachable check cannot render a bare `pass`; the + current run's headline names 1435 abstentions and 2 unreachable checks. + - _Requirements: R6.2, R6.3_ + +- [ ] **2.9 CI prints the liveness table; never gates on it** + - `coverage liveness` runs in the `eval-tier-a` job and writes to the step summary. + - **Definition of done:** the table appears on every Tier A run; the job's exit code is + unchanged by it. + - _Requirements: R6.5_ + +- [ ] **2.10 Demote generic metrics** + - `evals/metrics.py` output rendered under a heading naming it a sampling signal; absent from + every headline, `summary.txt` line and CI summary. + - **Definition of done:** no generic metric appears as a headline figure; a sample manifest + that used a metric to select traces records which metric and threshold. + - _Requirements: R7.1, R7.2, R7.3_ + +- [ ] **2.11 HTML renderer: stamps, abstention column, dark mode** + - Per design §5.1. Stamp row under the headline; `not_applicable` as a column in both tables; + `prefers-color-scheme` block. No new dependency, no CDN. Drop the dark-mode half if it + slows the phase (design §9.4). + - **Definition of done:** the HTML report shows every stamp and both denominators, renders + legibly in both themes, and has assertions — it has none today. + - _Requirements: R3.1, R3.2, R11.4_ + +--- + +## Phase 3 — The clock (free, no dependencies) + +- [ ] **3.1 `index stats` staleness and floor shortfall** + - Header block per design §7: corpus size with distinct-id count, span, per-floor shortfall, + annotation staleness, next-pass state. Two new store methods; consume alignment's + `CorpusProfile` if present, compute inline if not. + - **Definition of done:** output fits one screen; empty annotations render `NONE` and never + `0 days`; every R4 floor names its shortfall; exit code unchanged. + - _Requirements: R8.1, R8.2, R8.3, R8.4, R8.5_ + +- [ ] **3.2 Cadence written into `EVAL_METHODOLOGY.md`** + - The table from design §7.1: open coding every 2–4 weeks, weekly outlier review, the three + event triggers, and the honest anchor. + - **Definition of done:** the cadence is a rhythm with a stated anchor dependency, not an + aspiration; the three event triggers are named. + - _Requirements: R9.1, R9.2, R9.5_ + +- [ ] **3.3 Dated corpus-state and liveness tables** + - `EVAL_METHODOLOGY.md` carries both, each matching current command output, each dated. + - **Definition of done:** both tables reproduce the current `index stats` and + `coverage liveness` output; both carry a date. + - _Requirements: R9.3_ + +- [ ] **3.4 CI staleness warning on the dated tables** + - Warn when either table is more than 30 days older than the newest trace or newest report. + - **Definition of done:** the warning fires on a deliberately stale table and does not fail + the build. + - _Requirements: R9.4, R8.4_ + +--- + +## Phase 4 — Compatibility and closing the loop (free, after Phase 2) + +- [ ] **4.1 Baseline regeneration with a reason naming R4** + - Re-run Tier A, accept the baseline, and write a `reason` stating that the distinct-id + collapse changed fixture denominators and why that is expected. + - **Definition of done:** `evals/baselines/tier_a.json` is current; its `reason` names R4 as + the cause; the gate is green against it. + - _Requirements: R10.3, R10.4, R4.5_ + +- [ ] **4.2 Determinism and schema compatibility verified, not assumed** + - Confirm `test_tier_a_two_runs_deterministic` still holds; confirm a version-1 report loads; + confirm no added field carries a timestamp, external path, hostname, or iteration-ordered + collection. + - **Definition of done:** the integration test passes; a stored version-1 fixture report + loads without error; the field audit is recorded in the task note. + - _Requirements: R10.1, R10.2, R10.5_ + +- [ ] **4.3 Branch coverage on every threshold added here** + - The 50% boundary, the `n_decided < 10` suppression, the `unreachable` downgrade, the + empty-annotations branch, and the doc audit's fire/no-fire pair. + - **Definition of done:** 100% branch coverage on those paths; no test writes to + `evals/annotations/`, `evals/golden/` or `evals/traces/`. + - _Requirements: R11.1, R11.2, R11.3_ + +- [ ] **4.4 Correct the claims this spec's output invalidates** + - `evals/taxonomy/COVERAGE.md`'s *"every check-detected FM has a named implementation"* + gains how many of those implementations have ever decided anything. Same pass over + `docs/HARNESS.md`'s `enforced` column and `README.md`'s taxonomy paragraph. + - **Definition of done:** no tracked document claims coverage on the strength of check + registration; every such claim cites liveness instead. + - _Requirements: R6.4, coverage R11.5, coverage R11.6_ + +- [ ] **4.5 Journal entry** + - `docs/journal/2026-09-
-claim-surfaces.md` in house style: what was found, what was + built, what did not change, what the next session starts with. + - **Definition of done:** entry exists, is linked from `docs/journal/README.md`, and states + plainly that the corpus and the annotation count are unchanged by this spec. + - _Requirements: R9.3_ + +- [ ] **4.6 Update `EVALS_ROADMAP.md` and the four sibling spec READMEs** + - One line each: this spec exists, what it owns, and that the tasks it sequences in alignment + Phase 8 and coverage Phase 4 are executed here rather than there. + - **Definition of done:** no reader of alignment 8.1–8.4 or coverage 4.1–4.6 can be unsure + which spec owns execution; no requirement text is duplicated. + - _Requirements: R2.1_ + +--- + +## Minimum defensible slice + +**Phase 1, plus tasks 2.1, 2.2 and 3.1.** The quickstart stops teaching the defect, every rate +carries its decided denominator, and the corpus has a staleness reading. Half a day, no schema +migration, no baseline regeneration. + +If even that is too long: **task 1.1.** Two lines of markdown. It is the only item across the +five eval specs whose cost is minutes and whose effect is that the next person to clone this +repo does not reproduce, on their first command, the defect that took two months to find. diff --git a/.kiro/specs/eval-coverage/README.md b/.kiro/specs/eval-coverage/README.md new file mode 100644 index 00000000..baa1ef03 --- /dev/null +++ b/.kiro/specs/eval-coverage/README.md @@ -0,0 +1,223 @@ +# Spec: `eval-coverage` + +Kiro-style three-document spec for making the eval harness report what it **cannot see**. Read in +order: + +1. **[`requirements.md`](./requirements.md)** — 12 requirements with EARS acceptance criteria, six + new failure modes (FM-019…FM-024), constraints, and non-goals. +2. **[`design.md`](./design.md)** — the evidence-chain model, `EvidenceRequirement`, the liveness + classifier, signal-chain inventory, orphan-signal resolutions (§4), claim discipline (§8), and + five open decisions (§9). +3. **[`tasks.md`](./tasks.md)** — 8 phases, 38 tasks, each with a definition of done and + requirement traceability. + +Builds on [`../eval-harness/`](../eval-harness/) (Trace boundary, check registry, taxonomy, +tiers), [`../harness-alignment/`](../harness-alignment/) (arms, FM-014…FM-017), and +[`../eval-methodology-alignment/`](../eval-methodology-alignment/) (harness-owned telemetry, +FM-018, corpus source, human open coding, taxonomy provenance, claim discipline). None are +restated. + +## The one-line version + +`eval-methodology-alignment` fixes the **inputs**. This fixes the **instruments** — because +pointing the corpus at the right tree does not help a check reading a signal no code ever writes, +and today nothing in the repo reports the difference. + +## What the audit found + +The suite's own most recent report — `evals/results/tierA_4671d49ae035_eb36475d7c/report.json`, +taxonomy 1.2.0, 20 checks over 94 fixtures: + +| | | +| --- | ---: | +| check results | 1880 | +| `not_applicable` | **1435 (76.3%)** | +| `pass` | 263 | +| `fail` | 182 | + +The headline reads `FAIL — 182 failed check(s) across suite`. It does not mention that three +quarters of the suite declined to answer, because **`not_applicable` appears in no headline, no +verdict, no scorecard cell, and no incidence row.** The renderer has no field for it. + +| Finding | Detail | +| --- | --- | +| 11 of 20 checks are decided by one fixture pair | `1 pass / 1 fail` or `2 / 2`, `not_applicable` on 96–98% of the corpus. A unit test wearing a measurement's clothes. | +| 65 of the 182 headline failures are one check | `CHK-required-artifacts`, firing on fixtures that were never going to have artifacts. | +| On the real corpus, coverage is ~zero and unreported | The 50 traces in `evals/traces/` carry **0 spans, 0 artifacts, `cost.usd: null`**. Every span-, artifact- or cost-reading check abstains on all 50. No command prints this. | +| `CHK-interrupt-latency` (FM-003) can never fire | It needs `human_interrupt` spans. **No parser in `evals/trace/` emits that span type** — the literal appears only in the `SpanType` union. Unreachable by construction, on any corpus. | +| `CHK-guardrail-fp-budget` (FM-005) cannot see the LangGraph path | `guardrail_check` spans come only from Claude SDK hook events. LangGraph guardrail decisions reach **structlog and stop** — `langgraph_guardrail_nodes.py:273` logs `route_after_behavioral`; no `.jsonl` writer exists. | +| The fixture corpus double-counts 22 of its own fixtures | `evals/fixtures/traces/` holds **94 files carrying 72 distinct `trace_id`s**. `tier_a` globs by filename and scores both copies; `TraceStore.write` would refuse the duplicate. Every `2 pass / 2 fail` row is one pass fixture and one fail fixture, counted twice. | +| 5 span types are parsed and read by nobody | `qa_verdict`, `regression_check`, `session_start`, `subagent_start`, `subagent_stop` — no file in `evals/` outside `evals/trace/` mentions any of them. | + +### The sharpest one + +`CHK-evaluator-capitulation` (FM-017) needs QA verdicts. `parse_qa_verdicts_jsonl` already turns +them into typed `qa_verdict` spans. The check ignores those spans, reads +`trace.raw_result["qa_verdicts"]`, and when that is empty **re-parses the JSONL by hand** +(`verification.py:242–261`) — and when the artifact is absent too, abstains, while the spans it +needed sit in the trace unread. + +### The run that proves it + +The 2026-09-13 LangGraph smoke +([`docs/eval-runs/2026-09-13-langgraph-smoke/`](../../../docs/eval-runs/2026-09-13-langgraph-smoke/README.md)) +ended in `interrupt()` → `human_review` after burning four behavioral-relevance retries at +**5%, 0%, 9%, 0%** against a 15% floor. Textbook FM-003 and FM-005. **Both checks would abstain +on that trace.** The run README already concedes it — *"do not claim CHK-guardrail-fp-budget +'caught' this run"* — which is honest, and is honesty doing work that instrumentation should do. + +## The three layers + +``` + layer question measured today? + ────────────────────────────────────────────────────────────────────── + 1. the system does ai-team work? no — corpus starved + 2. the checks does the check code work? yes — 94 fixtures + 3. the instruments can the checks see anything? NOTHING MEASURES THIS +``` + +Layer 1 is `eval-methodology-alignment`. Layer 2 is what a green Tier A proves, and the repo says +so with `FIXTURE-ONLY`. Layer 3 is this spec, and the reason it is separate is that **fixing +layer 1 does not fix it**. + +## The evidence chain + +``` + harness writer → log file → parser → span → requires → check → liveness + telemetry.py *.jsonl parsers typed declared runs reported + │ │ + └─ no writer → no_producer ├─ no parser → unreachable_signal + └─ no check → orphan_signal +``` + +Every arrow can break. Today none of the breaks are visible and the suite reports a verdict +across all of them. + +## Liveness verdicts + +| Verdict | Means | Fix is | +| --- | --- | --- | +| `unreachable` | blind, and the evidence has no producer — no corpus could change it | **code** | +| `blind` | zero `pass` and zero `fail` over this corpus | data, usually | +| `thin` | decided on <10 traces, or >90% abstention | data, usually | +| `live` | decided on ≥10 traces with ≤90% abstention | — | + +`unreachable` ranks above `blind` on purpose: a starved corpus is a data problem, a broken chain +is a code problem, and only one of them gets better by waiting. + +## New failure modes + +All six are `origin: hypothesis` — read from run logs by a human, **not** from an unaided +open-coding annotation. They are excluded from active coverage counts until +`eval-methodology-alignment` R7's human pass promotes them, and promotion requires an annotation +record, never an argument and never a check firing. + +| ID | Slug | Check | Receipt from 2026-09-13 | +| --- | --- | --- | --- | +| FM-019 | `watchdog_not_enforced` | `CHK-watchdog-enforced` | 1600 s wall against a 900 s timeout; `DemoTimeoutError` is a catchable `Exception` and every subgraph swallows it | +| FM-020 | `vacuous_verification` | `CHK-test-collection-nonempty` | quality gate recorded ruff ok + pytest **exit 5**, `collected 0 items`, with `calc.py` and `test_calc.py` present | +| FM-021 | `run_record_incomplete` | `CHK-run-record-complete` | `completed_at: null`, no `costs.jsonl` — CLI never calls `finalize()` | +| FM-022 | `workspace_path_escape` | `CHK-write-path-scope` | drafts landed three levels deep in `workspace//workspace//workspace//`, poisoning pytest collection | +| FM-023 | `retry_rewind_after_success` | `CHK-no-rewind-after-artifact` | a testing-phase `GuardrailError` mapped to `retry_development` **after** the files were committed; two nested loops, 3×3 | +| FM-024 | `harness_overhead_unbounded` | `CHK-overhead-ratio` | 1600 s against a 1.528 s / $0.0000716 bare provider control on the same brief | + +FM-023 is the interesting one, and it is the smoke run's real finding. The files landed in ~90 s. +What burned the next 25 minutes was a relevance scorer reading conversation prose instead of disk, +and a router that rewound development because testing raised a guardrail error. FM-002 bounds how +many times a phase may repeat; it says nothing about whether the work was already done. + +## Three stamps, orthogonal + +| Stamp | Owned by | Means | +| --- | --- | --- | +| `FIXTURE-ONLY` / `CORPUS` / `LIVE` | alignment R15 | what the traces are | +| `NON-REPRESENTATIVE` | alignment R4 | the corpus is too narrow | +| `EVIDENCE-STARVED` | **this spec, R11** | the instruments could not see | + +A Tier A run today earns all three. The renderer shows one. + +## Constraints baked in + +| | | +| --- | --- | +| Location | extends `evals/checks/`, `evals/trace/`, `evals/aggregate.py`, `src/ai_team/harness/`; no new package | +| No vendor in the critical path | no Braintrust / LangSmith / Arize / DeepEval, consistent with the other three specs | +| Tier A stays $0.00 | every check added here is deterministic and offline | +| Budget | ≤ **$1.00**, control runs only (R10.5). Additive to the $5 suite, $25 ladder, $15 alignment. | +| No semantic changes | the `requires` migration adds declarations and changes **no** existing check's verdict — task 2.5 is the guard | +| Two gates only | a check whose mandatory span type has no parser fails CI (R4.6); eval vocabulary in an agent prompt fails CI (R14.2). Liveness itself never gates. | +| Evals never reach agents | check results, liveness verdicts and FM ids stay out of every agent-facing path (R14.1). Product gate results — pytest, ruff, smoke — are unaffected. | +| Telemetry is harness-owned | R5/R6 write through `harness/telemetry.py` per alignment R1 — **not a second writer** | +| Hypothesis ≠ observed | FM-019…FM-024 stay out of active coverage counts until a human annotation promotes them | +| Out of scope | changing existing check semantics, re-opening the arm/ladder design, multi-annotator κ, online monitoring | + +## What a run may learn from an eval + +Per-run check evidence (R13) makes an automated feedback loop tempting, and R14 draws the line: + +``` + PRODUCT LOOP run → pytest / ruff / smoke → agent context → next attempt (unchanged) + EVAL LOOP run → reports/check_results.json → human → harness fix → replay + CORPUS LOOP coverage → cells "never measured" → which scenarios run next (the one automation) +``` + +An eval verdict in an agent's context turns the measurement into a target, and this taxonomy +already has three modes for that: FM-014 (mutates the definition of done), FM-016 (writer is also +the passer), FM-018 (reports its own work). The corpus loop is safe because it changes the +sampling, not the subject — the contract handed to an agent is byte-identical either way. + +## On fidelity to the source material + +The methodology in `eval-methodology-alignment` R7–R15 is Husain and Shankar's: ~100-trace pool, +≥30 unaided, saturation, benevolent dictator, binary verdicts, 100–200 labels per judged mode, +TPR/TNR never accuracy, generic metrics as a sampling signal only, 2–4 week cadence. + +**Instrument liveness is not theirs.** They write about LLM application traces, where the trace +*is* the model input and output and exists by construction. In an agentic harness the telemetry is +something the harness must write, so "the check cannot see anything" is a failure mode their +setting does not produce. This spec is an extension for this domain, not an implementation of +something in the article, and it should not borrow the authority. + +Two acknowledged gaps: **multi-annotator agreement** (they discuss κ between humans; this repo +declines it for the single dictator — a deliberate deviation, not coverage), and **"evals as living +requirements"** (nothing here re-derives a requirement from a confirmed failure mode; design +decision §9.8). + +## Cross-spec dependency + +**Phase 5 is blocked** on `eval-methodology-alignment` Phase 1, which creates +`src/ai_team/harness/telemetry.py`. The guardrail and interrupt writers belong in that module. If +Phase 1 has not landed, the correct action is to stop — adding a second telemetry writer here +recreates the scattered-writer defect both specs exist to remove. + +Everything else is independent and free. + +## Minimum defensible slice + +**Phase 1, plus tasks 2.1–2.4, plus 6.3.** Liveness measured and published, every check declaring +what it reads, and one new check (`CHK-run-record-complete`) that fires on the corpus that already +exists. That turns "20 checks registered, 17 failure modes covered" into a number that means +something. + +If even that is too long: **tasks 1.3 and 1.4** — publish the liveness table and correct the +claims it invalidates. No code, and it removes the most misleading sentence the repo currently +ships: + +> `evals/taxonomy/COVERAGE.md`: *"None — every check-detected FM has a named implementation."* + +True. Seventeen implementations, two of which cannot execute on any corpus, eleven of which have +decided two fixtures each. + +## Executing this with Cursor + +``` +Read .kiro/specs/eval-coverage/requirements.md and design.md for context. +Implement task 2.1 from .kiro/specs/eval-coverage/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +Phases 1–4, 6 and 7 are free and safe to run unattended. **Phase 5 is blocked** on the alignment +spec. **Phase 8 spends** (≤$1.00) and is human-triggered. **Task 6.9 wires the promotion path but +must not exercise it** — promotion needs the human annotation pass, which is alignment Phase 4 and +is not delegable. diff --git a/.kiro/specs/eval-coverage/design.md b/.kiro/specs/eval-coverage/design.md new file mode 100644 index 00000000..b02be6ec --- /dev/null +++ b/.kiro/specs/eval-coverage/design.md @@ -0,0 +1,507 @@ +# Design — Eval Coverage and Check Liveness + +**Spec ID:** `eval-coverage` +**Requirements:** [`requirements.md`](./requirements.md) +**Created:** 2026-09-13 + +--- + +## 1. The idea + +An eval system has three layers, and this repo has been measuring only the middle one. + +``` + layer question it answers measured today? + ───────────────────────────────────────────────────────────────────────── + 1. the system does ai-team work? no (corpus starved) + 2. the checks does the check code work? yes (94 fixtures) + 3. the instruments can the checks see anything? NOTHING MEASURES THIS +``` + +Layer 2 is what a green Tier A proves. The repo knows this and says so — +`FIXTURE-ONLY`, `NON-REPRESENTATIVE`, the honest tables in `EVAL_METHODOLOGY.md`. Layer 1 is +what `eval-methodology-alignment` fixes by pointing the corpus at `output/runs/`. + +Layer 3 is the gap. **Fixing layer 1 does not fix layer 3**, and this is the load-bearing claim +of this spec. Re-index `output/runs/` tomorrow and `CHK-interrupt-latency` still cannot fire, +because no code anywhere produces a `human_interrupt` span. `CHK-guardrail-fp-budget` still +cannot see a LangGraph guardrail failure, because those decisions reach structlog and stop. The +richest possible corpus does not help a check reading a signal that is never written. + +So the deliverable is an **evidence chain** made explicit, end to end, and a report that names +every break in it: + +``` + harness writer → log file → parser → span → requires → check → liveness + telemetry.py *.jsonl parsers typed declared runs reported + │ │ + └── no writer → no_producer └── no parser → unreachable_signal + no check → orphan_signal +``` + +Every arrow can break. Today none of the breaks are visible, and the suite reports a verdict +across all of them. + +### Why liveness is the first thing to build + +It is free, it is offline, it needs no corpus, and it is the only artifact that makes the rest of +the work legible. Without it, "we added six checks" and "we fixed the guardrail telemetry" are +assertions. With it, each one moves a row from `unreachable` to `live` and the table is the +receipt. It is also the cheapest thing in any of the four specs: one module, no spend, no new +dependency, and it re-reads reports that already exist on disk. + +--- + +## 2. Data model + +### 2.1 `EvidenceRequirement` (R1) + +Declared per check, at registration, as data. + +```python +class EvidenceRequirement(BaseModel): + """What a check reads. Mandatory evidence absent ⇒ not_applicable.""" + + span_types: tuple[SpanType, ...] = () + span_types_optional: tuple[SpanType, ...] = () + artifacts: tuple[str, ...] = () # glob/suffix predicates + raw_result_keys: tuple[str, ...] = () + scalars: tuple[str, ...] = () # "cost.usd", "duration_s", "status" + any_of: bool = False # True ⇒ one mandatory item suffices + + def is_empty(self) -> bool: ... +``` + +`any_of` matters for real checks. `CHK-provider-error-rate` reads `error`, `llm_call` **or** +`retry` and decides on whichever is present; `CHK-constraint-survival` needs `phase_start` +specifically. Collapsing both into "requires these span types" would misclassify one of them, so +the flag is part of the declaration rather than a convention. + +The decorator gains one keyword and stays backwards compatible in shape: + +```python +@check( + id="CHK-guardrail-fp-budget", + failure_mode_id="FM-005", + tier="A", + requires=EvidenceRequirement(span_types=("guardrail_check",)), +) +def guardrail_fp_budget(trace: Trace) -> CheckResult: ... +``` + +R1.3 rejects an empty `requires` at registration. That is deliberate: registration-time failure +is loud, and a check that declares nothing is the case this whole spec is about. + +### 2.2 `na_reason` vocabulary (R3.5) + +`CheckResult` gains `na_reason: NaReason | None` beside its existing prose `evidence_text`. The +prose stays — it is good prose — but it is not aggregable. 1435 abstentions across 20 distinct +sentence templates cannot be counted; five codes can. + +```python +NaReason = Literal[ + "missing_span_type", # a mandatory span type is absent from the trace + "missing_artifact", # a required artifact path is absent + "missing_raw_key", # raw_result lacks a key the check reads + "missing_scalar", # cost/duration/status is null + "not_in_scope", # evidence present; this trace is genuinely outside the check +] +``` + +`not_in_scope` is the one honest `not_applicable`. A run with no UI does not need a UI smoke +probe, and that is not an instrument failure. Separating it from the other four is what keeps the +`EVIDENCE-STARVED` stamp (R11.1) from crying wolf on legitimate abstentions. + +### 2.2b Corpus identity (R3.7) + +`CoverageReport` counts distinct `trace_id`, and carries `n_files` beside it so a disagreement is +visible rather than silently resolved. The fixture directory is the case that motivates it: 94 +files, 72 ids. `tier_a` scores files, the store keys on ids, and the report has been showing the +file count. Counting identities is the correct choice — a duplicated fixture is one piece of +evidence, not two — and showing both is what makes the drift reportable. + +### 2.3 `CheckLiveness` and `CoverageReport` (R2) + +```python +Liveness = Literal["live", "thin", "blind", "unreachable"] + +class CheckLiveness(BaseModel): + check_id: str + failure_mode_id: str | None + tier: CheckTier + n_pass: int = 0 + n_fail: int = 0 + n_na: int = 0 + n_error: int = 0 + na_reasons: dict[str, int] = {} # NaReason → count + na_reason_text_top: list[tuple[str, int]] = [] + liveness: Liveness + unreachable_signals: tuple[str, ...] = () # span types with no parser + + @property + def n_decided(self) -> int: # R3.1 — always shown beside a rate + return self.n_pass + self.n_fail + +class CoverageReport(BaseModel): + corpus_label: str # "fixtures" | "corpus" | a path + corpus_kind: Literal["FIXTURE-ONLY", "CORPUS", "LIVE"] + n_traces: int + checks: list[CheckLiveness] + na_share: float + evidence_starved: bool # R11.1 + top_abstainers: list[tuple[str, int]] + provenance: Provenance +``` + +### 2.4 Classification rule (R2.2) + +Order matters. `unreachable` is tested first, because a check that is both starved and broken is +a code defect and should be reported as one. + +``` +if mandatory span type has no parser in the signal chain: unreachable +elif n_decided == 0: blind +elif n_decided < 10 or n_na / n_total > 0.90: thin +else: live +``` + +`thin` is the verdict that catches the eleven checks decided by one fixture pair. They are not +blind — they returned a `pass` and a `fail` — and calling them `live` is how a 98%-abstention +check ends up in a coverage table looking identical to `CHK-listener-self-trigger` at 10%. + +The 10-trace floor is a deliberately low bar chosen to be embarrassing rather than rigorous. It +is not a statistical threshold; `eval-harness` R6 already owns `min_cases` for anything that +gates. It exists so that `1 pass / 1 fail` cannot read as coverage. + +### 2.5 `SignalChain` (R4) + +Generated statically — from the `requires` declarations plus a hand-maintained producer table — +so it is correct on a fresh clone with an empty trace store. + +```python +class SignalRow(BaseModel): + span_type: SpanType + harness_writers: tuple[str, ...] # module paths that write the record + log_files: tuple[str, ...] + parsers: tuple[str, ...] # functions in evals/trace/parsers.py + consumers: tuple[str, ...] # check ids whose requires name it + flags: tuple[SignalFlag, ...] + +SignalFlag = Literal["orphan_signal", "unreachable_signal", "no_producer"] +``` + +The producer table is hand-maintained and that is a real cost — it can drift from the code. The +mitigation is R4.6: the one CI gate in this spec fires when a check's mandatory span type has no +parser. Drift in the *writer* column degrades a report; drift in the *parser* column breaks a +check, and only the second one is worth a gate. + +Known initial state (R4.7): + +| Span type | Writers | Parser | Consumers | Flags | +| --- | --- | --- | --- | --- | +| `human_interrupt` | — | **none** | `CHK-interrupt-latency` | `unreachable_signal`, `no_producer` | +| `guardrail_check` | `guardrails/*` (structlog only) | `parse_audit_jsonl` (SDK hooks only) | `CHK-guardrail-fp-budget` | `no_producer` (LangGraph path) | +| `qa_verdict` | `harness/qa_verdicts.py` | `parse_qa_verdicts_jsonl` | — | `orphan_signal` | +| `regression_check` | `harness/session_loop.py` | `parse_sessions_jsonl` | — | `orphan_signal` | +| `session_start` | `harness/session_loop.py` | `parse_sessions_jsonl` | — | `orphan_signal` | +| `subagent_start` | SDK hooks | `parse_audit_jsonl` | — | `orphan_signal` | +| `subagent_stop` | SDK hooks | `parse_audit_jsonl` | — | `orphan_signal` | +| `phase_start` / `phase_end` | **prompt only** → `harness/telemetry.py` | `parse_phases_jsonl` | 5 checks | `no_producer` until alignment Phase 1 | + +That last row is the alignment spec's R1 and is listed here only to show the chain is complete. +This spec does not implement it. + +--- + +## 3. Components + +### 3.1 `evals/coverage.py` — new + +Pure functions, no I/O beyond reading traces and reports. + +| Function | Purpose | +| --- | --- | +| `evidence_for(check_id)` | the declared `EvidenceRequirement` | +| `classify(counts, unreachable)` | the R2.2 rule, as one testable function | +| `liveness_from_results(results, ...)` | fold `list[CheckResult]` → `list[CheckLiveness]` | +| `liveness_over_corpus(traces)` | run all checks, fold | +| `liveness_from_report(path)` | fold an existing `report.json` — no re-run needed | +| `signal_chain()` | static `SignalChain`, no corpus | +| `render_markdown(report, ...)` | the table, with stamps | +| `render_side_by_side(a, b)` | R2.8 — fixtures vs corpus in one table | + +`liveness_from_report` is what makes this immediately useful: the two reports already in +`evals/results/` can be folded into a liveness table before a single new check exists, which is +how the numbers in `requirements.md` were produced. + +### 3.2 `evals/cli.py` — extended + +```bash +uv run python -m evals.cli coverage liveness # corpus + fixtures, side by side +uv run python -m evals.cli coverage liveness --from-report evals/results//report.json +uv run python -m evals.cli coverage signals # static chain, works on empty repo +``` + +Both are $0.00 and offline. `liveness` refuses to touch the network under the same +socket-blocking discipline as `tier_a.py`. + +### 3.3 `evals/aggregate.py` — extended + +- `ScorecardCell`, `FailureModeIncidence` gain `na_count` (R3.2). +- `FailureModeIncidence` is emitted for **every** active FM with a `liveness` field (R3.4), + replacing today's behavior where 7 of 17 FMs are simply absent from the table. +- `SuiteReport` gains `coverage: CoverageReport`. +- The renderer gains the `EVIDENCE-STARVED` stamp and the R2.5 verdict suppression. + +R12.5 is the constraint that keeps this safe: the migration adds fields and changes no verdict. +The regression test asserts all 20 checks' outcomes fixture-by-fixture before and after. + +### 3.4 `src/ai_team/harness/telemetry.py` — extended + +`eval-methodology-alignment` R1.3 establishes this module as the single telemetry writer. This +spec adds two record families to it rather than creating new writers: + +```python +def emit_guardrail_decision(...) -> None # R5 → logs/guardrails.jsonl +def emit_interrupt(...) -> None # R6 → logs/interrupts.jsonl +def emit_terminal_state(...) -> None # R6.4 +``` + +All carry `writer: "harness"` and land under `/logs/` per alignment R1.7. **This +spec must not create a second telemetry module**; if alignment Phase 1 has not landed, the tasks +here block on it. + +The `scored_text_source` field (R5.3) deserves a note. The smoke run's finding was not "a +guardrail failed" but "the guardrail scored the last 12 AI messages instead of the workspace." +A record saying `decision: fail, score: 0.05, threshold: 0.15` reproduces the symptom. A record +adding `scored_text_source: "last_12_ai_messages"` reproduces the *diagnosis*, and that is the +difference between a detector and an explanation. + +### 3.5 New checks (R8) + +Six checks, one module each where the layer fits, all Tier A, all offline. + +| Check | Reads | Fails when | +| --- | --- | --- | +| `CHK-watchdog-enforced` | `duration_s`, timeout from contract, terminal state | duration > timeout × (1 + tol) **and** terminal state ≠ `watchdog_expired` | +| `CHK-test-collection-nonempty` | test invocation records, artifacts | tests collected 0 while source + test artifacts exist | +| `CHK-run-record-complete` | `run.json` fields via `raw_result`, `cost` | `completed_at` null, or no cost record on a run that made LLM calls | +| `CHK-write-path-scope` | `tool_result` spans, artifact paths | a committed path repeats the workspace-root segment | +| `CHK-no-rewind-after-artifact` | `phase_start`, `phase_end`, artifacts | a phase re-entered after its output contract was already satisfied | +| `CHK-overhead-ratio` | `duration_s`, control-arm baseline | ratio > contract ceiling; `not_applicable` with no control | + +`CHK-no-rewind-after-artifact` is the one with real design risk. "Output contract already +satisfied" is a judgment, and encoding it wrongly produces false positives on legitimate +iteration — a developer phase re-entered because the tests genuinely failed is correct behavior, +not FM-023. The v1 rule is deliberately narrow: fail only when the re-entry was triggered by a +**guardrail** error whose phase differs from the phase being re-entered, which is exactly the +`route_after_testing` → `retry_development` mapping the smoke run hit. Broader definitions wait +for open-coding evidence. + +### 3.6 Taxonomy (R9) + +`failure_modes.yaml` gains, per new mode: + +```yaml + - id: FM-019 + slug: watchdog_not_enforced + origin: hypothesis + hypothesis_evidence: + runs: ["2026-09-13_182650_write-a-single-python-module_01"] + documents: ["docs/eval-runs/2026-09-13-langgraph-smoke/README.md"] + detection: check + implemented_by: [CHK-watchdog-enforced] + status: active +``` + +`loader.py` gains the `hypothesis` origin and excludes those modes from active coverage counts +and origin shares (R9.3). `COVERAGE.md` grows an origin column and a liveness column, which +together answer the question the current file cannot: not "does this FM have a check" but "has +that check ever decided anything, and where did the FM come from." + +--- + +## 4. Resolving the orphan signals (R7) + +A decision per span type, recorded here as R7.5 requires. + +| Span type | Decision | Reason | +| --- | --- | --- | +| `qa_verdict` | **consume** — re-point `CHK-evaluator-capitulation` | The check needs precisely this signal and currently hand-re-parses the file the parser already typed. Deleting `verification.py:242–261` removes a second JSON parser and a second failure path. | +| `subagent_start` / `subagent_stop` | **consume** — new `CHK-subagent-fanout-bounded` | Unbounded or orphaned subagents are a real harness failure on the SDK backend, and the spans already exist. Cheap. | +| `session_start` | **consume** — fold into `CHK-run-record-complete` | Needs no check of its own; it is corroborating evidence for a complete run record. | +| `regression_check` | **remove parser** | Nothing in the taxonomy is about regression checks, the session loop that writes it is behind a flag, and inventing an FM to justify an existing parser is the tail wagging the dog. Re-add with a consumer if a mode ever needs it. | + +Four orphans consumed, one parser deleted. The general rule this encodes: a parser earns its +place by having a reader, and the fix for "no reader" is sometimes deletion. + +--- + +## 4b. The two loops (R13, R14) + +Per-run check evidence and feed-forward are one design, because the first is what makes the second +tempting and the second is what keeps the first honest. + +``` + PRODUCT LOOP — agent-facing, already correct, unchanged by this spec + run → pytest / ruff / smoke → agent context → next attempt + + EVAL LOOP — maintainer-facing + run → reports/check_results.json → human reads → harness or prompt fix → replay + │ + └→ corpus fold → liveness + coverage report + + CORPUS LOOP — the one safe automation + coverage report → dimension cells "never measured" → selects which scenarios run next + (never what an agent is told) +``` + +The boundary is not stylistic. An eval verdict in an agent's context turns the measurement into a +target, and this taxonomy already has three modes for that outcome — FM-014, FM-016, FM-018. The +per-run file is therefore written *beside* the run, read by people and by the fold, and excluded +from every prompt path by a CI gate (R14.2). + +The corpus loop is safe because it changes the *sampling*, not the *subject*. Choosing to run an +unmeasured scenario cell is the same decision an analyst makes by hand; the contract handed to the +agent is byte-identical either way (R14.6). That is a self-improvement loop on what the project +knows about itself, which is the useful half, without the half that corrupts the data. + +## 5. Error handling + +| Condition | Behavior | +| --- | --- | +| Check raises during liveness | record `outcome: error`, continue; `n_error` is reported and never folded into `n_decided` — per `EVAL_METHODOLOGY.md` §6, errors are not failures | +| Trace fails to load | skip, count in `n_unloadable`, name the ids | +| Empty trace store | valid: `n_traces: 0`, every check `blind`, `EVIDENCE-STARVED` set | +| Producer table names a module that no longer exists | warn in the report; do not fail | +| `--from-report` given a schema the code predates | refuse with the schema versions; never guess | +| Telemetry write fails (R5/R6) | warn on the run, continue — alignment R1.4, unchanged | + +--- + +## 6. Testing strategy (R12) + +| Target | Test | +| --- | --- | +| Classification | table-driven over all four verdicts and both boundaries (`n_decided == 10`, `na == 0.90`) | +| Mandatory-evidence contract | for each of 20 checks: strip mandatory evidence ⇒ `not_applicable` + correct `na_reason` (R1.5) | +| Empty `requires` | registration raises | +| No-verdict-change | all 20 checks, all 94 fixtures, outcome-identical before/after migration (R12.5) | +| Purity | socket-blocked, per `test_check_purity.py` | +| Signal chain on empty repo | no trace store, full table (R12.6) | +| Stamp threshold | 49.9% / 50.1% `na` share | +| New checks | `pass` / `fail` / `not_applicable` fixture each (R12.3) | +| Hygiene | `tmp_path` only; a test asserts no writes under `evals/` tracked dirs (R12.1) | + +--- + +## 7. Phasing and dependency + +``` + Phase 1 liveness over existing reports ← free, no deps, do this first + Phase 2 requires declarations + na_reason ← no deps + Phase 3 signal chain + the one CI gate ← needs Phase 2 + Phase 4 report integration + stamps ← needs Phase 2 + Phase 5 guardrail + interrupt telemetry ← BLOCKED on alignment Phase 1 + Phase 6 six hypothesis FMs + checks ← needs Phase 2; Phase 5 for FM-005/003 to go live + Phase 7 orphan resolution ← needs Phase 2 + Phase 8 efficiency / control arm ← needs harness-alignment arms; spends ≤$1 +``` + +Phase 1 is worth isolating because it delivers the spec's core finding with no code changes to +anything that runs in CI, and because its output is what justifies Phases 2–8 to a reader who +does not already believe the problem exists. + +Phase 5 is the only hard cross-spec block. If alignment Phase 1 has not landed, +`emit_guardrail_decision` has no module to live in, and adding one anyway recreates the +scattered-writer problem both specs exist to remove. + +--- + +## 8. Claim discipline (R11) + +Three stamps, orthogonal, all renderer-enforced: + +| Stamp | Owned by | Means | +| --- | --- | --- | +| `FIXTURE-ONLY` / `CORPUS` / `LIVE` | alignment R15 | what the traces are | +| `NON-REPRESENTATIVE` | alignment R4 | the corpus is too narrow | +| `EVIDENCE-STARVED` | **this spec, R11** | the instruments could not see | + +A Tier A run today earns all three, and the current renderer shows one. The sentence this spec +makes available — and the only coverage sentence it authorizes — has the shape: + +> Tier A: 20 checks registered, `k` live, `m` thin, `j` blind, `i` unreachable, over `n` traces +> (`FIXTURE-ONLY`, `NON-REPRESENTATIVE`, `EVIDENCE-STARVED`, `na` 76%). + +And the sentence it forbids, which `evals/taxonomy/COVERAGE.md` currently implies: + +> ~~Every check-detected failure mode has a named implementation.~~ + +True, and not a coverage claim. Seventeen implementations, two of which cannot execute on any +corpus and eleven of which have decided two fixtures each. + +--- + +## 9. Open decisions + +**9.1 — Should `thin` suppress a green verdict, like `unreachable` does (R2.5)?** +Leaning no. `unreachable` is a code defect and deserves to break the shine; `thin` is usually a +corpus problem and alignment already stamps for that. Revisit once the corpus clears its floors — +if checks stay `thin` on a representative corpus, that is an instrument problem after all. + +**9.2 — Should the 10-trace `thin` floor be per-check?** +`CHK-workspace-isolation` needs concurrent runs, which are rare by nature, and will look `thin` +forever on an honest corpus. A per-check `expected_applicability` field would fix it and is more +bookkeeping than the problem currently justifies. Deferred. + +**9.3 — Does `CHK-overhead-ratio` belong in Tier A at all?** +It needs a control-arm baseline, so on most corpora it returns `not_applicable` — a permanent +`thin` row. Arguably it belongs in the ladder report (`harness-alignment`) rather than the check +suite. Leaning: keep the check, but report overhead in the ladder. + +**9.4 — Should `regression_check`'s parser really be deleted (§4)?** +It is the only orphan with no natural consumer, and deletion is cheap to reverse. The risk is +deleting a parser for a signal the session loop starts writing next month. Mitigated by R4.6, +which would flag the reverse case loudly. + +**9.5 — Should FM-024 (`harness_overhead_unbounded`) be a failure mode?** +Overhead is a cost, not a failure, and the taxonomy is about failures. Counter-argument: 1000× +the control on a smoke test is a failure of the harness's purpose. Keeping it as +`origin: hypothesis` defers the question honestly — if open coding never produces it as a code, +it becomes `unobserved` and the question answers itself. + +**9.8 — "Evals as living requirements": how far should this go?** +The source material frames evals as requirements that evolve with the system. This spec supports +that and stops at the human boundary: a confirmed failure mode motivates a requirement, and a +person writes it. Automating spec-authoring from taxonomy changes is attractive and is where the +self-grading risk reappears at the document layer — a system that writes its own acceptance +criteria from its own measurements has the FM-014 problem one level up. Left open deliberately. +Minimum useful step, if taken: `taxonomy coverage` emitting a list of confirmed FMs with no +corresponding requirement, as a prompt for a human to write one. + +**9.9 — Is instrument liveness actually part of this methodology?** +No, and the spec should not borrow authority it has not earned. Husain and Shankar write about LLM +application traces, where the trace *is* the model input and output and is present by +construction. In an agentic harness the telemetry is something the harness must write, so +"the check cannot see anything" is a failure mode their setting does not produce. Everything in +`eval-methodology-alignment` R7–R15 is theirs. The liveness layer is an extension for this domain, +and `README.md` says so. + +**9.6 — `CHK-listener-self-trigger` reads source code, not traces. Does R1.3 reject it?** +As written, yes — its `requires` would be empty, because it inspects +`ai_team.core.flow_wiring` rather than the Trace, and R1.3 rejects an empty declaration at +registration. That is a real tension and the requirement is not wrong to surface it: a check whose +verdict is identical for every trace in the corpus is a static lint, and its per-trace `pass` +count is 94 copies of one fact. Three options, in preference order: (a) add a +`static: true` flag to `EvidenceRequirement` and report such checks in a separate section with +n=1, which is honest about what they measure; (b) move it out of the check registry into a lint; +(c) widen R1.3. Leaning (a) — it keeps the check where it is useful while stopping it from +inflating a coverage table. Resolve before task 2.3. + +**9.7 — Should a check that raises on every trace be its own verdict?** +Today it classifies as `blind`, with the error count shown separately. An `erroring` verdict +would be more precise, since the fix is different from both a starved corpus and a missing +parser. Deferred: the current corpus produces no genuine check errors, so the distinction has no +data behind it yet, and inventing a verdict for a case that has never occurred is how the +taxonomy got into trouble in the first place. diff --git a/.kiro/specs/eval-coverage/requirements.md b/.kiro/specs/eval-coverage/requirements.md new file mode 100644 index 00000000..86e213bb --- /dev/null +++ b/.kiro/specs/eval-coverage/requirements.md @@ -0,0 +1,561 @@ +# Requirements — Eval Coverage and Check Liveness + +**Spec ID:** `eval-coverage` +**Status:** Draft for implementation +**Owner:** Rick Zakharov +**Target repo:** `ai-team` (extends `evals/checks/`, `evals/trace/parsers.py`, `evals/aggregate.py`, +`src/ai_team/harness/telemetry.py`) +**Created:** 2026-09-13 +**Depends on:** [`../eval-harness/`](../eval-harness/) — Trace boundary (R1), check registry (R5), +taxonomy (R5), tiers (R11), provenance (R14). [`../harness-alignment/`](../harness-alignment/) — +FM-014…FM-017. [`../eval-methodology-alignment/`](../eval-methodology-alignment/) — harness-owned +telemetry (R1), FM-018 (R2), corpus source (R5), human open coding (R7), taxonomy provenance (R9), +claim discipline (R15). **None of those are restated here.** + +--- + +## Introduction + +`eval-methodology-alignment` fixes the **inputs**: the corpus is built from the wrong tree and +phase telemetry is asked for in a prompt. This spec fixes the **instruments**. They are separate +defects and fixing the first does not fix the second. + +An eval suite has two ways to be wrong. It can score the wrong thing — that is the alignment +spec. Or it can score nothing and report a verdict anyway. This repo does the second, and no +artifact it produces says so. + +### What the check suite actually decided + +Tier A's own most recent report, `evals/results/tierA_4671d49ae035_eb36475d7c/report.json`, +taxonomy `1.2.0`, 20 checks over 94 synthetic fixtures — **1880 check results**: + +| Outcome | Count | Share | +| --- | ---: | ---: | +| `not_applicable` | **1435** | **76.3%** | +| `pass` | 263 | 14.0% | +| `fail` | 182 | 9.7% | + +The headline reads `FAIL — 182 failed check(s) across suite`. It does not mention that three +quarters of the suite declined to answer. `not_applicable` appears in no headline, no verdict, no +scorecard cell, and no `failure_mode_incidence` row. The renderer has no field for it. + +### Eleven of twenty checks are decided by one fixture pair + +Per-check outcomes from that same report: + +| Check | pass | fail | n/a | n/a share | +| --- | ---: | ---: | ---: | ---: | +| `CHK-acceptance-monotonic` | 1 | 1 | 92 | 98% | +| `CHK-constraint-survival` | 1 | 1 | 92 | 98% | +| `CHK-evaluator-capitulation` | 1 | 1 | 92 | 98% | +| `CHK-hallucination-density` | 1 | 1 | 92 | 98% | +| `CHK-lesson-effectiveness` | 1 | 1 | 92 | 98% | +| `CHK-premature-termination` | 1 | 1 | 92 | 98% | +| `CHK-verifier-independence` | 1 | 1 | 92 | 98% | +| `CHK-gate-env-fidelity` | 2 | 2 | 90 | 96% | +| `CHK-guardrail-fp-budget` | 2 | 2 | 90 | 96% | +| `CHK-interrupt-latency` | 2 | 2 | 90 | 96% | +| `CHK-metric-source-agreement` | 2 | 2 | 90 | 96% | +| `CHK-workspace-isolation` | 2 | 2 | 90 | 96% | +| `CHK-provider-error-rate` | 8 | 2 | 84 | 89% | +| `CHK-tool-call-emitted` | 14 | 5 | 75 | 80% | +| `CHK-draft-commit` | 20 | 1 | 73 | 78% | +| `CHK-phase-repeat-bounded` | 28 | 2 | 64 | 68% | +| `CHK-required-artifacts` | 15 | 65 | 14 | 15% | +| `CHK-spend-ceiling` | 76 | 6 | 12 | 13% | +| `CHK-runtime-smoke-present` | 2 | 82 | 10 | 11% | +| `CHK-listener-self-trigger` | 83 | 2 | 9 | 10% | + +A check answering `1 pass / 1 fail` over a 94-trace corpus has been exercised by exactly the two +fixtures written to exercise it — and for the `2 / 2` rows, by **one** pass fixture and **one** +fail fixture counted twice, because 22 of those 94 files share a `trace_id` with another file +(R3.7). It is a unit test wearing a measurement's clothes. And 65 of the +182 headline failures are one check, `CHK-required-artifacts`, firing on fixtures that were never +going to have artifacts. + +### On real traces the number is worse and is not reported at all + +The 50 traces in `evals/traces/` carry **0 spans, 0 artifacts, and `cost.usd: null`** between +them (`status: failed`, 50 of 50 — see `eval-methodology-alignment` R3). Every check that reads +a span, an artifact, or a cost returns `not_applicable` on every one of them. That is the actual +coverage of this eval system over this project's own history, it is approximately zero, and no +command in the repo prints it. + +### Two checks cannot fire on any corpus, ever + +This is distinct from a starved corpus. These are breaks in the signal chain: + +| Check | FM | Needs | Chain status | +| --- | --- | --- | --- | +| `CHK-interrupt-latency` | FM-003 | `human_interrupt` spans | **No parser in `evals/trace/` emits this span type.** The literal appears only in `models.py`'s `SpanType` union. Unreachable by construction. | +| `CHK-guardrail-fp-budget` | FM-005 | `guardrail_check` spans | Emitted only by `parse_audit_jsonl` from Claude SDK hook events (`parsers.py:296`). LangGraph guardrail decisions go to **structlog only** — `langgraph_guardrail_nodes.py:273` logs `route_after_behavioral`; no `.jsonl` writer exists. | + +The 2026-09-13 LangGraph smoke run +([`docs/eval-runs/2026-09-13-langgraph-smoke/`](../../../docs/eval-runs/2026-09-13-langgraph-smoke/README.md)) +demonstrates both at once. It ended in a LangGraph `interrupt()` → `human_review`, and it burned +four behavioral-relevance retries at 5%, 0%, 9%, 0% against a 15% floor. Those are textbook +FM-003 and FM-005 events. Both checks would return `not_applicable` on that trace. The run README +already concedes this — *"do not claim CHK-guardrail-fp-budget 'caught' this run"* — which is +honest, and is the honesty doing work that instrumentation should do. + +### Five span types are parsed and read by nobody + +`evals/trace/parsers.py` builds these spans. No file in `evals/` outside `evals/trace/` mentions +any of them: + +| Span type | Producer | Consumer | +| --- | --- | --- | +| `qa_verdict` | `parse_qa_verdicts_jsonl` | **none** | +| `regression_check` | `parse_sessions_jsonl` | **none** | +| `session_start` | `parse_sessions_jsonl` | **none** | +| `subagent_start` | `parse_audit_jsonl` | **none** | +| `subagent_stop` | `parse_audit_jsonl` | **none** | + +`qa_verdict` is the sharpest case. `CHK-evaluator-capitulation` (FM-017) needs exactly this +signal, and reads `trace.raw_result["qa_verdicts"]` and then a raw artifact re-parse +(`verification.py:242–261`) — never the spans. So the harness parses the file into typed spans, +the check ignores them and re-parses the file by hand, and when `raw_result` is empty the check +returns `not_applicable` while the spans it needed sit in the trace unread. + +### Summary of the gap this spec closes + +| Area | Gap today | Requirement | +| --- | --- | --- | +| Check capability | no check declares what evidence it needs | R1 | +| Liveness | nothing reports that a check never fired | R2 | +| `not_applicable` | absent from headline, verdict, scorecard, incidence | R3 | +| Signal chain | no inventory of producer → parser → consumer | R4 | +| FM-005 blindness | guardrail decisions are structlog-only | R5 | +| FM-003 blindness | `human_interrupt` has no producer at all | R6 | +| Orphan signals | 5 span types parsed, 0 consumers | R7 | +| Unmodeled failures | 6 modes visible in the smoke run, no FM | R8 | +| Hypothesis discipline | `origin` has no value for "read from logs, not annotated" | R9 | +| Efficiency | 1600 s vs a 1.528 s control, measured by nothing | R10 | +| Claim discipline | a rate over 76% n/a renders like a rate | R11 | +| Duplicate identities | 94 fixture files, 72 distinct `trace_id`s | R3.7, R3.8 | +| Test hygiene | — | R12 | +| Per-run evidence | run record has a quality-gate scorecard, no eval check results | R13 | +| Feed-forward | no rule on what a run may learn from evals | R14 | + +--- + +## R1 — Checks declare the evidence they require + +**User story.** As the owner of the check suite, I want every check to state what evidence it +needs in machine-readable form, so that "this check cannot run here" is computable before the +check runs rather than inferred from a prose `not_applicable` string. + +### Acceptance criteria + +1. The `@check` decorator SHALL accept a `requires` argument holding an `EvidenceRequirement`: + span types, artifact predicates, `raw_result` keys, and trace scalar fields the check reads. +2. Every one of the 20 registered checks SHALL declare `requires` in a single migration, and the + declaration SHALL be derived from the check's code, not from its docstring. +3. A check whose `requires` is empty SHALL be rejected at registration time. A check that reads + nothing cannot decide anything. +4. `EvidenceRequirement` SHALL distinguish **mandatory** evidence (absent ⇒ the check must + return `not_applicable`) from **optional** evidence (absent ⇒ the check still decides). +5. A test SHALL assert, for every check, that constructing a Trace with all mandatory evidence + removed yields `not_applicable`, and that the reason string names the missing evidence. +6. `requires` SHALL be data, readable without importing the check body, so that the signal-chain + report (R4) does not execute check code. + +## R2 — Check liveness is computed and reported + +**User story.** As a reader of a Tier A report, I want to know which checks actually decided +something, so that a green suite cannot mean an empty one. + +### Acceptance criteria + +1. The harness SHALL compute, for a given corpus, a `CheckLiveness` record per check holding + `pass`, `fail`, `not_applicable`, and `error` counts, the `na_reason` histogram, and a + `liveness` verdict. +2. `liveness` SHALL be one of: + - `blind` — zero `pass` and zero `fail` over the corpus; + - `unreachable` — `blind`, and the mandatory evidence has **no producer** in the signal + chain (R4), so no corpus could change the result; + - `thin` — decided on fewer than 10 traces, or `not_applicable` on more than 90% of them; + - `live` — decided on ≥10 traces with ≤90% `not_applicable`. +3. `unreachable` SHALL rank above `blind` in severity, because a starved corpus is a data problem + and a broken chain is a code problem. +4. `SuiteReport` SHALL carry the liveness table, and the rendered headline SHALL state the count + of `blind` and `unreachable` checks whenever either is non-zero. +5. A report with any `unreachable` check SHALL NOT render a suite-level verdict of `pass`. It + SHALL render `pass (instruments incomplete)` with the unreachable check ids named. +6. Liveness SHALL NOT gate CI. It is a state to display, consistent with + `eval-methodology-alignment` R4.6. The CI job SHALL print the liveness table on every run. +7. `evals.cli` SHALL expose `coverage liveness` computing the table over any trace root at + **$0.00** with no network access, and SHALL support `--fixtures` to render the fixture corpus + and the real corpus side by side. +8. The side-by-side view SHALL be the default when both roots are present, because a check that + is `live` on fixtures and `blind` on the corpus is the exact defect this spec exists to + surface, and one column cannot show it. + +## R3 — `not_applicable` is first-class in every report + +**User story.** As a reviewer, I want an abstention counted and shown wherever a pass rate is +shown, so that a denominator cannot quietly shrink to two. + +### Acceptance criteria + +1. Every rendered rate SHALL show `n_decided` (pass + fail) beside it, and SHALL NOT show a + percentage computed over a denominator it does not display. +2. `ScorecardCell` and `FailureModeIncidence` SHALL carry `na_count`, and the renderer SHALL + show it. +3. WHEN a check's `n_decided` is below 10, its rate SHALL render as `n=` with no percentage. +4. `FailureModeIncidence` SHALL be emitted for every active FM, including those with zero + incidence, with a `liveness` field — so that an FM absent from the table because nothing could + see it is distinguishable from an FM absent because it did not occur. The 1.2.0 report emits + incidence rows for 10 of 17 FMs and the other 7 are simply missing. +5. A `not_applicable` result SHALL carry a structured `na_reason` code from a closed vocabulary + (`missing_span_type`, `missing_artifact`, `missing_raw_key`, `missing_scalar`, + `not_in_scope`), in addition to its human-readable text. +6. `na_reason` codes SHALL be aggregated per check so that the single most common cause of + abstention is visible without reading 1435 strings. +7. A corpus SHALL be counted by **distinct `trace_id`**, not by file. `evals/fixtures/traces/` + holds 94 files carrying **72 distinct ids** — 22 ids appear in two files each, and + `tier_a.load_fixture_traces` globs by filename, so both copies are scored while + `TraceStore.write` would refuse the duplicate and `rebuild_index` would collapse them. Every + `2 pass / 2 fail` row in the current report is therefore **one pass fixture and one fail + fixture, counted twice**. +8. The report SHALL name `n_files` and `n_distinct_traces` separately whenever they differ, and + CI SHALL warn WHEN a fixture directory contains two files with the same `trace_id`. Tier A's + denominators and the trace store SHALL NOT disagree about the size of a corpus. + +## R4 — The signal chain is inventoried and gaps are named + +**User story.** As a maintainer, I want one table from harness writer to check consumer, so that +a signal nobody reads and a signal nobody writes are both visible. + +### Acceptance criteria + +1. The harness SHALL emit a `SignalChain` report with one row per span type: the harness module + that writes the underlying record, the log file, the parser function, and the checks that + consume it. +2. A row with a parser and no consumer SHALL be flagged `orphan_signal`. +3. A row consumed by a check with no parser SHALL be flagged `unreachable_signal`. +4. A row with a parser but no harness writer SHALL be flagged `no_producer`. +5. The report SHALL be generated by static inspection of `requires` declarations (R1.6) and a + declared producer table — never by running a corpus, so that it is correct on an empty repo. +6. `coverage signals` SHALL render it, and CI SHALL fail WHEN a new check is registered whose + mandatory span type has no parser. That is a code defect, not a data state, and it is the one + thing in this spec that gates. +7. The initial run SHALL record the known state: `human_interrupt` as `unreachable_signal`; five + span types as `orphan_signal`; `guardrail_check` as `no_producer` for the LangGraph path. + +## R5 — Guardrail decisions become harness-owned telemetry + +**User story.** As an analyst, I want every guardrail decision in a durable record, so that +FM-005 is detectable on the path where it actually happens. + +### Acceptance criteria + +1. The harness SHALL write `/logs/guardrails.jsonl` from harness code on the + execution path, with `writer: "harness"` per `eval-methodology-alignment` R1. +2. Each record SHALL carry `run_id`, `backend`, `phase`, `guardrail_id`, `decision` + (`pass`|`fail`), `score`, `threshold`, `retry_index`, `scored_text_source`, and `timestamp`. +3. `scored_text_source` SHALL name what the guardrail actually read — for the behavioral scope + guardrail, that it scored the last N AI messages and not the workspace. The smoke run's + central finding is that the scorer never looks at disk; a record that omits this is not + evidence of it. +4. All guardrail call sites SHALL write through one module. A guardrail that logs only to + structlog SHALL be treated as a defect, and `langgraph_guardrail_nodes.py:273` + (`route_after_behavioral`) SHALL be the first site converted. +5. `parse_guardrails_jsonl` SHALL build `guardrail_check` spans, and + `CHK-guardrail-fp-budget` SHALL read them without change to its semantics. +6. WHEN the LangGraph smoke run of 2026-09-13 is replayed after this requirement lands, the check + SHALL return `fail` with the four retry scores as evidence. Until then it SHALL return + `not_applicable`, and no document SHALL claim otherwise. +7. A guardrail record SHALL be written whether the decision passes or fails. A corpus of failures + only cannot produce a false-positive rate. + +## R6 — Human-interrupt and terminal-state telemetry + +**User story.** As an analyst, I want HITL and terminal states recorded, so that a run ending in +`human_review` is visible to the checks rather than only to a human reading `state.json`. + +### Acceptance criteria + +1. The harness SHALL write `human_interrupt` records — at minimum LangGraph `interrupt()`, CLI + HITL prompts, and `hitl_source` transitions — to `/logs/interrupts.jsonl` with + `writer: "harness"`. +2. Each record SHALL carry `requested_at`, `resumed_at` or `abandoned_at`, `source`, and the + graph node or phase that raised it, so that `CHK-interrupt-latency` has a latency to measure. +3. `parse_interrupts_jsonl` SHALL build `human_interrupt` spans, closing the + `unreachable_signal` flagged in R4.7. +4. The harness SHALL write a terminal-state record naming the run's exit condition + (`completed`, `failed`, `awaiting_human`, `killed`, `budget_abort`, `watchdog_expired`) and + whether `finalize()` ran. +5. `CHK-interrupt-latency` SHALL return `not_applicable` — never `pass` — for a run with no + interrupt records, and the `na_reason` SHALL be `missing_span_type`. + +## R7 — Orphan signals earn a consumer or are retired + +**User story.** As a maintainer, I want every parsed signal to be read by something, so that +parser code is not maintained for nobody. + +### Acceptance criteria + +1. Each of the five orphan span types SHALL be resolved in one of two ways: a check reads it, or + its parser is removed. No span type SHALL remain parsed and unread. +2. `CHK-evaluator-capitulation` SHALL be re-pointed at `qa_verdict` spans as its primary source, + with the `raw_result` path retained as a fallback and the hand-rolled artifact re-parse in + `verification.py:242–261` deleted. A check that re-parses a file the parser already typed is + the bug this requirement removes. +3. `session_start` and `regression_check` SHALL gain a consumer or be removed, and the decision + SHALL be recorded in the design document with a reason. +4. `subagent_start` / `subagent_stop` SHALL gain a consumer bounding subagent fan-out and + orphaned subagents, or be removed. +5. Removal SHALL be a deliberate, recorded decision — not the default outcome of nobody choosing. + +## R8 — Failure modes observed in the smoke run are modeled + +**User story.** As the owner, I want the defects a real run demonstrated to be first-class +failure modes with deterministic checks, so that the next occurrence is caught by the harness +rather than by me reading 1600 seconds of structlog. + +### Acceptance criteria + +1. The taxonomy SHALL gain six modes, all `origin: hypothesis` (R9), all `detection: check`, + all Tier A at $0.00: + +| ID | Slug | Layer | Check | Evidence from 2026-09-13 | +| --- | --- | --- | --- | --- | +| FM-019 | `watchdog_not_enforced` | harness | `CHK-watchdog-enforced` | `DemoTimeoutError` is a catchable `Exception`; every subgraph wraps `sub.invoke` in `except Exception`; SIGALRM swallowed; wall 1600 s against a 900 s timeout | +| FM-020 | `vacuous_verification` | harness | `CHK-test-collection-nonempty` | quality gate recorded ruff ok + pytest **exit 5**, `collected 0 items`, while `calc.py` and `test_calc.py` existed at the run root | +| FM-021 | `run_record_incomplete` | harness | `CHK-run-record-complete` | `completed_at: null`; no `costs.jsonl`; CLI never calls `ResultsBundle.finalize()` | +| FM-022 | `workspace_path_escape` | harness | `CHK-write-path-scope` | testing subgraph resolved the **parent** `./workspace`; drafts landed in `workspace//workspace//workspace//`, three levels deep, poisoning pytest collection | +| FM-023 | `retry_rewind_after_success` | harness | `CHK-no-rewind-after-artifact` | a testing-phase `GuardrailError` mapped to `retry_development` after `calc.py` and `test_calc.py` were already committed; two nested retry loops, 3×3 | +| FM-024 | `harness_overhead_unbounded` | harness | `CHK-overhead-ratio` | 1600 s harness wall against a 1.528 s / $0.0000716 bare OpenRouter control on the same brief | + +2. FM-019's check SHALL fail WHEN a trace's wall duration exceeds its configured timeout by more + than a tolerance AND no `watchdog_expired` terminal state was recorded — the conjunction is the + point, since a watchdog that fires correctly is not this failure. +3. FM-020's check SHALL fail on a recorded test invocation that collected zero tests while source + and test artifacts exist, and SHALL be distinct from FM-006 (`runtime_verification_gap`), + which is about a smoke probe being absent rather than vacuous. +4. FM-022's check SHALL fail on any committed write path containing a repeated workspace-root + segment, and SHALL report nesting depth. +5. FM-023's check SHALL fail WHEN a phase is re-entered after a later phase recorded artifacts + satisfying that phase's output contract. It SHALL be distinct from FM-002 + (`self_triggering_retry_loop`), which bounds repeat count without regard to whether the work + was already done. +6. FM-024's check SHALL require a recorded control baseline and SHALL return `not_applicable` + without one. An overhead ratio with no denominator is not a measurement. +7. Every one of the six SHALL ship `pass`, `fail`, and `not_applicable` fixtures, and SHALL be + scoreable retroactively over the existing corpus. +8. None of these six SHALL be described in any document as observed, confirmed, or prevalent + until R9's promotion path has run. They are hypotheses with receipts, which is not the same + as labels. + +## R9 — `origin: hypothesis` and the promotion path + +**User story.** As a reader of the taxonomy, I want a mode derived from reading logs to be +distinguishable from one derived from open coding, so that axial-coding guesses cannot +accumulate into apparent evidence. + +### Acceptance criteria + +1. `origin: hypothesis` SHALL be a first-class value alongside `essay`, `reference`, and + `open_coding` (`eval-methodology-alignment` R9.1), and SHALL mean: derived by a human reading + run logs, without an unaided open-coding annotation record. +2. A `hypothesis` mode SHALL carry `hypothesis_evidence`: the run ids and document paths it was + read from — for FM-019…FM-024, the 2026-09-13 smoke run id and its README. +3. `hypothesis` modes SHALL be **excluded from active coverage counts and from taxonomy origin + shares** in `taxonomy coverage`, and SHALL be reported in their own row. +4. Promotion from `hypothesis` to `open_coding` SHALL require ≥1 annotation record with a + `trace_id` present in the index, per `eval-methodology-alignment` R9.2. Promotion SHALL NOT be + available by argument, by a check firing, or by the author's confidence. +5. A `hypothesis` mode whose check fires SHALL NOT be auto-promoted. A check firing is evidence + the check works; only a human annotation is evidence the mode is real. +6. `taxonomy propose --from-annotations` SHALL surface, after the first open-coding pass, which + `hypothesis` modes the annotations independently reached — the useful signal being whether + unaided human coding found the same six. +7. A `hypothesis` mode unconfirmed after two open-coding passes SHALL be marked + `status: unobserved` per `eval-methodology-alignment` R10.2, not deleted. + +## R10 — Harness efficiency is measured against a control + +**User story.** As the owner, I want the cost of the harness measured against doing nothing, so +that overhead is a tracked number rather than an anecdote in a handoff note. + +### Acceptance criteria + +1. The repo SHALL define a **bare-provider control**: the same scenario brief sent as a single + provider call, recorded with wall time, tokens, and cost. +2. Control runs SHALL be recorded as traces with `arm: "control"`, reusing the arm axis from + `harness-alignment` R1 rather than adding a parallel concept. +3. `SuiteReport` SHALL render `overhead_ratio` = harness wall ÷ control wall, and + `cost_ratio`, each with `n` and the control trace id. +4. A single pair SHALL render as `n=1, ANECDOTE` and SHALL NOT be published as a ratio. The + 1600 s / 1.528 s observation is an anecdote and the renderer SHALL say so. +5. Control runs SHALL be capped at **$1.00** total for this spec and SHALL be human-triggered. +6. `CHK-overhead-ratio` SHALL fail only above a configured ceiling, and the ceiling SHALL be + recorded in the scenario contract rather than hardcoded, because acceptable overhead differs + per scenario size. +7. No document SHALL state a harness-overhead figure without `n`, the control trace id, and the + models both sides used. + +## R11 — Claim discipline for coverage + +**User story.** As a reader of this repo's claims, I want a rate over a mostly-abstaining suite +to look different from a rate over a deciding one, so that 76% `not_applicable` cannot render as +a measurement. + +### Acceptance criteria + +1. A report WHERE `not_applicable` exceeds 50% of all check results SHALL carry an + **`EVIDENCE-STARVED`** stamp naming the share and the three checks contributing most + abstentions. +2. The stamp SHALL be rendered by the report renderer, never asserted in prose — same mechanism + as `NON-REPRESENTATIVE` (`eval-methodology-alignment` R4.4). The two stamps are orthogonal and + both SHALL be shown when both apply: one is about the corpus, the other about the instruments. +3. `FIXTURE-ONLY` (`eval-methodology-alignment` R15.1) SHALL be shown together with the + liveness summary, so that a fixture-corpus result cannot be read as a coverage claim. +4. `docs/EVAL_METHODOLOGY.md` SHALL gain a dated liveness table alongside its corpus-state + table, and CI SHALL warn WHEN it is more than 30 days older than the newest report. +5. No document, post, badge, or deck SHALL state that a failure mode is *covered* on the strength + of a registered check. Coverage claims SHALL cite liveness, not registration. The current + `evals/taxonomy/COVERAGE.md` line *"every check-detected FM has a named implementation"* is + true and SHALL be qualified with how many of those implementations have ever decided anything. +6. WHERE a check is `unreachable`, every artifact naming its FM as covered SHALL be corrected in + the same change that discovers it. + +## R12 — Test hygiene + +**User story.** As a maintainer, I want this spec's additions tested to the standard of the +harness they instrument. + +### Acceptance criteria + +1. No test SHALL write to `evals/annotations/`, `evals/golden/`, `evals/traces/`, or + `evals/results/`. Tests SHALL use `tmp_path` roots. +2. Every refusal and classification path added here SHALL reach 100% branch coverage: empty + `requires` rejection, each of the four `liveness` verdicts, each `na_reason` code, each of the + three signal-chain flags, the R2.5 verdict suppression, and the `EVIDENCE-STARVED` threshold. +3. Each of the six new checks SHALL have `pass`, `fail`, and `not_applicable` fixtures, matching + existing convention. +4. A test SHALL assert the liveness computation is pure and offline, under the socket-blocking + pattern already used by `tests/unit/evals/test_check_purity.py`. +5. A regression test SHALL assert that the 20 pre-existing checks' outcomes are unchanged by the + `requires` migration, fixture by fixture. This spec adds declarations; it changes no verdicts. +6. A test SHALL assert `coverage signals` is correct with an empty trace store, since R4.5 + requires static generation. + +--- + +## R13 — Per-run check evidence lives in the run record + +**User story.** As someone reading a single run, I want to see which checks decided, which +abstained, and why, without running a suite or reading structlog — because the run record is +where I look when a run goes wrong. + +### Background + +`output/runs//reports/scorecard.json` exists and is the **quality gate** — `lint_ok`, +`test_passed`, `phases`, `guardrails`. It is not eval checks. On the 2026-09-13 smoke run it +reads `"guardrails": []` and `"phases": {}` despite four behavioral-guardrail failures, so the +scorecard is starved by the same telemetry defect the rest of this spec addresses. There is no +per-run record of eval check outcomes anywhere. + +### Acceptance criteria + +1. The harness SHALL write `/reports/check_results.json` after a run, holding one + entry per registered Tier A check: `check_id`, `failure_mode_id`, `outcome`, `na_reason`, + `evidence_text`, and `evidence_span_ids`. +2. It SHALL be written by harness code with `writer: "harness"`, co-located with the run record + per `eval-methodology-alignment` R1.7. +3. It SHALL be rendered as **evidence, never as a score**. A per-run summary SHALL NOT display a + pass rate, a percentage, a ratio, or a count of passes without the abstention count beside it. + `n` is 1. "18 of 20 checks passed" on a single run, most of them `not_applicable`, is the + claim R11 exists to prevent. +4. WHERE a check abstained, the per-run view SHALL show the `na_reason` prominently. The value of + this artifact is that the smoke run would have shown + `CHK-guardrail-fp-budget: na — no guardrail_check spans` in the run view, instead of that + requiring a 1600-second structlog read. +5. `coverage liveness` SHALL be able to fold these per-run files directly, so corpus liveness is + an aggregation of run records rather than a separate scoring pass. +6. Writing this file SHALL NOT fail a run, and SHALL NOT be attempted when no checks are + registered. +7. The existing run UI SHALL gain a panel reading this file, following the conventions of + `TestResultsPanel` and `GuardrailsPanel`. It SHALL show abstentions by default rather than + hiding them behind a toggle — a hidden abstention is the defect this spec started from. + +## R14 — Feed-forward discipline: what a run may learn from evals + +**User story.** As the owner, I want evals to improve the system without becoming the thing the +system optimizes for, so that a measurement stays a measurement. + +### Background + +An eval result that reaches an agent's context becomes an optimization target. The taxonomy +already names that failure three times — FM-014 (`acceptance_criteria_mutation`), FM-016 +(`self_graded_verification`), FM-018 (`self_reported_telemetry`). An agent that can read +`CHK-guardrail-fp-budget: fail` and act on it is precisely the self-grading loop those modes +describe, and it also contaminates the corpus: traces recorded after such feedback sample +system-behaviour-under-eval-pressure, not system behaviour. Husain is explicit that generic +metrics are a signal for choosing traces to inspect, never a target +(`eval-methodology-alignment` R15.3). + +The distinction this requirement draws: **product-gate results feed the agent; eval-check results +feed the human and the corpus.** + +### Acceptance criteria + +1. Eval check results, liveness verdicts, failure-mode ids, and taxonomy content SHALL NOT appear + in any agent prompt, system message, tool result, or memory or lesson record consumed by an + agent at runtime. +2. A test SHALL assert (1) by scanning prompt-construction paths for check ids, FM ids, and + liveness vocabulary. This is the second gate in this spec, alongside R4.6, and for the same + reason: it is a code property, not a data state. +3. Product quality-gate results — pytest, ruff, type checks, runtime smoke — SHALL continue to + reach the agent unchanged. That is a build loop and is correct; this requirement does not + restrict it. +4. The maintainer-facing loop SHALL be: check fires → human reads → harness or prompt fix → + replay the same scenario. This is the loop `docs/eval-runs/2026-09-13-langgraph-smoke/README.md` + §"Debug loop" already describes, and no requirement here replaces it with an automated one. +5. Exactly one automated feed-forward SHALL be permitted: **corpus coverage selects the next + runs.** Dimension cells reported `never measured` (`eval-methodology-alignment` R6.5) SHALL be + readable as a run-selection input, so that what gets run next is driven by what has not been + measured. +6. That selection SHALL affect **which scenarios run**, never what any agent is told. A scenario + contract chosen because its cell is unmeasured SHALL be byte-identical to the same contract + chosen for any other reason. +7. The existing lessons loop (`memory/lessons.py`, `harness/lessons_loop.py`) SHALL remain the + only agent-facing feedback path, and FM-013 (`lesson_ineffective`) SHALL remain its detector. + No eval verdict SHALL be written into a lesson. +8. WHERE a human decides an eval finding justifies a configuration change — a timeout, a + guardrail threshold, a retry ceiling — the change SHALL be recorded as a human decision with + the finding that motivated it, and SHALL NOT be applied automatically. + +### On "evals as living requirements" + +The source material treats evals as requirements that evolve. This spec supports the direction and +deliberately stops short of automating it: a confirmed failure mode SHOULD motivate a requirement, +and that authoring step is human. Nothing here writes to a spec. Recorded so the gap is visible +rather than implied — see `design.md` §9.8. + +## Constraints + +| | | +| --- | --- | +| No new package | extends `evals/checks/`, `evals/trace/`, `evals/aggregate.py`, `src/ai_team/harness/` | +| No vendor in the critical path | no Braintrust / LangSmith / Arize / DeepEval; consistent with the other three specs | +| Tier A stays $0.00 | every check added here is deterministic and offline | +| Budget | ≤ **$1.00** for control runs (R10.5). Nothing else in this spec spends. Additive to the $5 suite, $25 ladder, $15 alignment. | +| No semantic changes | the `requires` migration adds declarations and changes no existing check's verdict (R12.5) | +| Two gates only | R4.6 — a check whose mandatory span type has no parser fails CI. R14.2 — eval vocabulary in an agent prompt fails CI. Liveness itself never gates. | +| Evals never reach agents | R14.1 — check results, liveness and FM ids stay out of every agent-facing path; product gate results are unaffected | +| Telemetry is harness-owned | R5 and R6 write through harness code on the call path, per `eval-methodology-alignment` R1; no new agent instructions | +| Hypothesis ≠ observed | R8's six modes stay out of active coverage counts until a human annotation promotes them (R9) | +| Out of scope | changing any existing check's semantics, re-opening the arm/ladder design, multi-annotator κ, online monitoring, vendor platforms | + +## Non-goals + +- Fixing the corpus. That is `eval-methodology-alignment` R5 and this spec depends on it. +- Performing the open coding. That is `eval-methodology-alignment` R7 and is human-only. +- Adding judges. R11 of the alignment spec governs when an FM earns one; nothing here is + judge-detected. +- Raising the number of checks as a goal in itself. A `blind` check is worse than no check, + because it produces a row in a table that reads like coverage. +- Proving a framework ranking. Still out of reach, and `harness-alignment` says why. diff --git a/.kiro/specs/eval-coverage/tasks.md b/.kiro/specs/eval-coverage/tasks.md new file mode 100644 index 00000000..908cb3c4 --- /dev/null +++ b/.kiro/specs/eval-coverage/tasks.md @@ -0,0 +1,425 @@ +# Implementation Plan — Eval Coverage and Check Liveness + +**Spec ID:** `eval-coverage` +**Requirements:** [`requirements.md`](./requirements.md) · **Design:** [`design.md`](./design.md) + +11 phases, 54 tasks. Phase 1 tasks 1.1–1.2 and all of Phase 1b are **done** (shipped 2026-09-13/14: +`evals/coverage.py`, `evals/ui/workbench.html`, `annotate bundle`); everything else is +unexecuted. + +--- + +## How to execute this plan + +- Work **one task at a time, in order**. Do not batch phases. +- Every task lists a **Definition of done**. Do not mark it complete until each bullet is + literally true. +- Run `uv run ruff check . && uv run ruff format --check . && uv run mypy src/ && uv run mypy evals/ && uv run pytest tests/unit` + before marking any task complete. +- **Nothing in Phases 1–7 spends money.** Phase 8 spends ≤ $1.00 and is human-triggered. +- **Phase 5 is blocked** on `eval-methodology-alignment` Phase 1 (`harness/telemetry.py`). Do not + work around the block by creating a second telemetry writer — that is the defect both specs + remove. If Phase 1 has not landed, stop and say so. +- No task in this plan may change an existing check's verdict. Task 2.5 is the guard. + +Cursor prompt shape: + +``` +Read .kiro/specs/eval-coverage/requirements.md and design.md for context. +Implement task 2.1 from .kiro/specs/eval-coverage/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +--- + +## Phase 1 — Liveness over what is already on disk (free, no dependencies) + +- [x] **1.1 `evals/coverage.py` — liveness folding and renderers** + - `Liveness`, `CheckLiveness`, `CoverageReport`, `classify()`, + `liveness_from_results()`, `liveness_from_report()`, `liveness_over_corpus()`, + `signal_chain()`, `render_markdown()`, `render_side_by_side()`. + - Offline, pure, no new dependency. + - **Definition of done:** module imports with no network; `classify` covers all four verdicts; + `liveness_from_report` folds `evals/results/*/report.json` without re-running checks. + - _Requirements: R2.1, R2.2, R2.7, R4.5, R11.1_ + +- [x] **1.2 `coverage liveness` / `coverage signals` CLI** + - Subcommands with `--traces-root`, `--from-report`, `--fixtures`, `--json`, `--out`. + - **Definition of done:** both run at $0.00 with sockets blocked; `signals` works with an empty + trace store. + - _Requirements: R2.7, R2.8, R4.6, R12.6_ + +- [x] **1.3 Record the baseline liveness artifact** + - `docs/eval-runs/2026-09-13-check-liveness/README.md` — the fixture-vs-corpus table, the + 1435/1880 abstention split, the two unreachable checks, the five orphan signals. Corpus kind + stamped; no rate without `n`. + - **Definition of done:** table reproduces from `coverage liveness --from-report`; every figure + traceable to a committed report.json; no claim of a pass rate. + - _Requirements: R11.1, R11.3, R11.5_ + +- [ ] **1.4 Correct the coverage claims this finding invalidates** + - `evals/taxonomy/COVERAGE.md` — add `origin` and `liveness` columns; qualify + *"every check-detected FM has a named implementation."* + - `docs/EVAL_METHODOLOGY.md` — dated liveness table beside the corpus-state table. + - **Definition of done:** no document states an FM is covered on the strength of registration + alone; both files name the two unreachable checks. + - _Requirements: R11.4, R11.5, R11.6_ + +- [x] **1.5 Journal entry** + - [`docs/journal/2026-09-14-check-liveness.md`](../../../docs/journal/2026-09-14-check-liveness.md) + — the findings, what was built, what did not change, the live next-steps list, and the + open questions. `docs/journal/README.md` index row and a `journey.md` chapter added. + - Dated the 14th, not the 13th: the sitting ran past midnight and the handoff chain is + dated by when it was written, never by when the work started. + - _Requirements: R11.4_ + +## Phase 1b — The annotation workbench (R13 of eval-methodology-alignment) + +Shipped 2026-09-13/14. This phase exists because `eval-methodology-alignment` R13 names +annotation friction as the reason open coding never happened, and R8.3 requires a human +accept/merge/reject surface between the clusterer and the taxonomy that did not exist anywhere. + +- [x] **1b.1 `evals/ui/workbench.html` — four stages, gated in the loop's order** + - Stage 1 open coding (keyboard-driven, no `$EDITOR`), stage 2 axial coding + (accept/rename/reject + Wilson intervals), stage 3 taxonomy YAML draft, stage 4 judge gate. + - Self-contained: no dependency, no network, no build step, opens from `file://`. + - **Definition of done:** stage 1 surfaces no model output and no taxonomy vocabulary; gates + lock at 30 unaided records, ≥1 accepted category, and 100 labels respectively; a locked + stage states what it is waiting for. + - _Requirements: alignment R7.3, R7.4, R7.5, R8.1, R8.3, R8.4, R9.2, R11.1, R12.3, R13.1–R13.5_ + +- [x] **1b.2 `annotate bundle` — export a sample for the page** + - `--skip-annotated` resumes a sitting; missing traces are recorded, never fatal. + - **Definition of done:** bundling the existing real sample yields 50 traces, 0 missing. + - _Requirements: alignment R13.3_ + +- [x] **1b.3 Export ingests through the command that already exists** + - The page's JSONL matches `run_batch_annotate`'s accepted shape, so + `annotate --sample --batch-file ` ingests it with no new code. + - **Definition of done:** round trip verified end to end into a temp annotations root, + producing valid `AnnotationRecord` lines with `saturation_streak` computed. + - _Requirements: alignment R7.1, R13.1_ + +- [x] **1b.4 Mirror parity with the Python originals** + - `normalizeTag` / `wilson` / `saturationStreak` verified against + `propose.normalize_tag`, `reliability.wilson_ci`, `annotate.saturation_streak`. + - **Definition of done:** identical on 20 normalisation cases, Wilson to 6 decimal places, + saturation across 7 sequences; pinned in `test_annotate_bundle.py` so drift fails a test. + - _Requirements: R12.5_ + +- [x] **1b.5 Headless render smoke test** + - `tests/unit/evals/workbench/smoke.js` + `test_workbench_smoke.py`; 23 assertions over a + fixture bundle mixing traces with and without spans. Skips without node. + - **Definition of done:** asserts the refusal paths, both card shapes, all four gates, and + that no FM id or taxonomy slug is embedded in the page. + - _Requirements: R12.2, R12.4_ + +- [ ] **1b.6 Decide whether the node smoke test joins CI** + - The repo already has node for the frontend. Not wired in — a deliberate open question, not + an oversight. + - _Requirements: R12.4_ + +- [ ] **1b.7 Carry `unaided` into `AnnotationRecord`** + - The page emits it and the ingest path currently drops it, because the field does not exist + yet. It is alignment R7.4, and until it lands the unaided count is inferred rather than + recorded. + - **Definition of done:** `AnnotationRecord.unaided` exists; `apply_batch_item` preserves it; + `taxonomy propose` counts it rather than assuming. + - _Requirements: alignment R7.4_ + +## Phase 2 — Evidence declarations + +- [ ] **2.1 `EvidenceRequirement` model** + - In `evals/checks/base.py`; `span_types`, `span_types_optional`, `artifacts`, + `raw_result_keys`, `scalars`, `any_of`, `is_empty()`. + - **Definition of done:** typed, mypy-clean, importable without importing check bodies. + - _Requirements: R1.1, R1.4, R1.6_ + +- [ ] **2.2 `@check(requires=...)` with empty-rejection** + - Decorator accepts and stores `requires`; raises at registration when `is_empty()`. + - **Definition of done:** a test asserts registration of a `requires`-less check raises. + - _Requirements: R1.1, R1.3_ + +- [ ] **2.3 Declare `requires` for all 20 existing checks** + - Derived from each check's code, not its docstring. Cross-check against the observed + `not_applicable` reasons in the 1.2.0 report. + - **Definition of done:** all 20 declare; `any_of` set on `CHK-provider-error-rate`; no + declaration contradicts the check body. + - _Requirements: R1.2, R1.4_ + +- [ ] **2.4 `NaReason` vocabulary on `CheckResult`** + - Five codes; `na()` helper takes the code; all 20 checks pass one. + - **Definition of done:** every `na()` call site supplies a code; `not_in_scope` used only where + evidence is present and the trace is genuinely out of scope. + - _Requirements: R3.5, R3.6_ + +- [ ] **2.5 No-verdict-change regression test** + - Snapshot all 20 checks × 94 fixtures before the migration; assert outcome-identical after. + - **Definition of done:** test fails if any outcome changes; snapshot committed under + `tests/unit/evals/`, not `evals/`. + - _Requirements: R12.1, R12.5_ + +- [ ] **2.6 Mandatory-evidence contract test** + - For each check: strip mandatory evidence ⇒ `not_applicable` with the matching `na_reason`. + - **Definition of done:** 20 parametrized cases; each asserts code and that the text names the + missing evidence. + - _Requirements: R1.5, R12.2_ + +## Phase 3 — Signal chain and the parser gate + +- [ ] **3.1 Producer table** + - `evals/trace/producers.py` — span type → harness writer modules → log files. Hand-maintained, + documented as such. + - **Definition of done:** every `SpanType` has a row; missing writers recorded as `—`, not + guessed. + - _Requirements: R4.1, R4.4_ + +- [ ] **3.2 `signal_chain()` static generation** + - Joins producer table, parser inventory, and `requires` consumers. Flags `orphan_signal`, + `unreachable_signal`, `no_producer`. + - **Definition of done:** correct with an empty trace store; reproduces the R4.7 known state + exactly. + - _Requirements: R4.1–R4.5, R4.7_ + +- [ ] **3.3 CI gate: mandatory span type with no parser** + - Fails the build when a registered check's mandatory span type has no parser. + - **Definition of done:** gate fails on a deliberately broken check in a test; passes on the + current registry **after** Phase 6 lands `human_interrupt`'s parser, and until then is + allow-listed with `CHK-interrupt-latency` named and dated in the allow-list. + - _Requirements: R4.6_ + +- [ ] **3.4 `coverage signals` renders the chain** + - Markdown + JSON. + - _Requirements: R4.6, R12.6_ + +## Phase 4 — Reports tell the truth + +- [ ] **4.1 `na_count` on `ScorecardCell` and `FailureModeIncidence`** + - _Requirements: R3.2_ + +- [ ] **4.2 Emit incidence for every active FM with a `liveness` field** + - Replaces today's behavior where 7 of 17 FMs are absent from the table. + - **Definition of done:** the 1.2.0 corpus produces 17 rows, not 10; an FM with no incidence is + distinguishable from an FM nothing could see. + - _Requirements: R3.4_ + +- [ ] **4.3 `n_decided` beside every rate; suppress percentages under 10** + - **Definition of done:** no rendered percentage without its displayed denominator. + - _Requirements: R3.1, R3.3_ + +- [ ] **4.3b Count corpora by distinct `trace_id`; report `n_files` beside it** + - Warn when a fixture directory holds two files sharing a `trace_id`. + - **Definition of done:** the 22 duplicated fixture ids are named; Tier A denominators and + `TraceStore.rebuild_index` agree on corpus size; every `2 pass / 2 fail` row resolves to + `1 / 1`. + - _Requirements: R3.7, R3.8_ + +- [ ] **4.4 `SuiteReport.coverage` + `EVIDENCE-STARVED` stamp** + - Threshold 50% of all results, excluding `not_in_scope`. Names the top three abstainers. + - **Definition of done:** renderer-enforced; boundary tested at 49.9% / 50.1%. + - _Requirements: R11.1, R11.2, R12.2_ + +- [ ] **4.5 Headline names blind/unreachable counts; verdict suppression** + - `pass (instruments incomplete)` when any check is `unreachable`. + - **Definition of done:** a suite with one unreachable check cannot render a bare `pass`. + - _Requirements: R2.4, R2.5_ + +- [ ] **4.6 CI prints the liveness table every run; never gates on it** + - _Requirements: R2.6_ + +## Phase 4c — Per-run check evidence (R13) + +- [ ] **4c.1 Write `/reports/check_results.json`** + - Harness-owned, `writer: "harness"`, one entry per Tier A check with `na_reason`. + - **Definition of done:** a failed write warns and the run completes; no file when no checks are + registered. + - _Requirements: R13.1, R13.2, R13.6_ + +- [ ] **4c.2 Render it as evidence, not a score** + - **Definition of done:** no pass rate, percentage or ratio anywhere in the per-run view; pass + counts always shown beside abstention counts; a test asserts a per-run percentage never renders. + - _Requirements: R13.3_ + +- [ ] **4c.3 `coverage liveness --from-run-records`** + - Fold per-run files into corpus liveness. + - **Definition of done:** folding the run-record tree reproduces the same table as scoring the + traces directly, for runs where both exist. + - _Requirements: R13.5_ + +- [ ] **4c.4 Run-UI eval panel** + - Follows `TestResultsPanel` / `GuardrailsPanel` conventions; abstentions visible by default. + - **Definition of done:** the 2026-09-13 smoke run displays + `CHK-guardrail-fp-budget: na — no guardrail_check spans` without opening a log. + - _Requirements: R13.4, R13.7_ + +## Phase 4d — Feed-forward discipline (R14) + +- [ ] **4d.1 CI gate: eval vocabulary must not reach an agent prompt** + - Scans prompt-construction paths for check ids, FM ids, liveness vocabulary. + - **Definition of done:** gate fails on a deliberately planted `FM-005` in a prompt template; + passes on the current tree; product gate results untouched. + - _Requirements: R14.1, R14.2, R14.3_ + +- [ ] **4d.2 Assert no eval verdict is written into a lesson** + - **Definition of done:** a test asserts `memory/lessons.py` records carry no check or FM ids; + FM-013 remains the lessons detector. + - _Requirements: R14.7_ + +- [ ] **4d.3 Corpus coverage as run selection** + - `never measured` dimension cells readable as a run-selection input. + - **Definition of done:** selection changes which scenario contracts run; a test asserts the + emitted contract is byte-identical to the same contract selected any other way. + - _Requirements: R14.5, R14.6_ + +- [ ] **4d.4 Record configuration changes as human decisions** + - A timeout, threshold or retry ceiling changed on eval evidence is recorded with the finding + that motivated it, and never applied automatically. + - _Requirements: R14.8_ + +## Phase 5 — Guardrail and interrupt telemetry — BLOCKED on alignment Phase 1 + +- [ ] **5.1 `emit_guardrail_decision` in `harness/telemetry.py`** + - `logs/guardrails.jsonl`; `writer: "harness"`; fields per R5.2 including `scored_text_source`. + - **Definition of done:** written for pass **and** fail decisions; a read-only logs dir warns + and the run completes. + - _Requirements: R5.1, R5.2, R5.3, R5.7_ + +- [ ] **5.2 Convert `route_after_behavioral` first** + - `langgraph_guardrail_nodes.py:273` writes through the module; structlog line retained for + humans, no longer the only record. + - **Definition of done:** a LangGraph run produces `guardrails.jsonl` rows with + `scored_text_source: "last_12_ai_messages"`. + - _Requirements: R5.4_ + +- [ ] **5.3 Convert remaining guardrail call sites** + - **Definition of done:** no guardrail decision reaches only structlog; a test asserts every + guardrail module imports the writer. + - _Requirements: R5.4_ + +- [ ] **5.4 `parse_guardrails_jsonl` → `guardrail_check` spans** + - **Definition of done:** `CHK-guardrail-fp-budget` semantics unchanged; `no_producer` flag + clears for the LangGraph path. + - _Requirements: R5.5_ + +- [ ] **5.5 `emit_interrupt` + `parse_interrupts_jsonl`** + - LangGraph `interrupt()`, CLI HITL, `hitl_source` transitions → `logs/interrupts.jsonl` → + `human_interrupt` spans. + - **Definition of done:** `CHK-interrupt-latency` leaves `unreachable`; R4.6's allow-list entry + is removed in the same change. + - _Requirements: R6.1, R6.2, R6.3, R6.5_ + +- [ ] **5.6 `emit_terminal_state`** + - Exit condition + whether `finalize()` ran. + - _Requirements: R6.4_ + +- [ ] **5.7 Replay the 2026-09-13 smoke brief** + - **Definition of done:** `CHK-guardrail-fp-budget` returns `fail` with the four retry scores + (5%, 0%, 9%, 0%) as evidence, and `CHK-interrupt-latency` decides rather than abstaining. + Until this task passes, no document claims either check caught anything. + - _Requirements: R5.6, R6.5, R11.6_ + +## Phase 6 — The six hypothesis failure modes + +- [ ] **6.1 `origin: hypothesis` in the taxonomy loader** + - Plus `hypothesis_evidence`; excluded from active coverage counts and origin shares. + - **Definition of done:** `taxonomy coverage` reports hypothesis modes in their own row. + - _Requirements: R9.1, R9.2, R9.3_ + +- [ ] **6.2 Seed FM-019…FM-024** + - All `origin: hypothesis`, evidence = the smoke run id + its README. Taxonomy version bumped. + - **Definition of done:** no FM claims an annotation it does not have. + - _Requirements: R8.1, R9.2_ + +- [ ] **6.3 `CHK-run-record-complete` (FM-021)** + - Start here: it fires on the existing corpus immediately and needs no new telemetry. + - _Requirements: R8.1, R8.7_ + +- [ ] **6.4 `CHK-write-path-scope` (FM-022)** + - Reports nesting depth. + - _Requirements: R8.4, R8.7_ + +- [ ] **6.5 `CHK-test-collection-nonempty` (FM-020)** + - Distinct from FM-006: vacuous, not absent. + - _Requirements: R8.3, R8.7_ + +- [ ] **6.6 `CHK-watchdog-enforced` (FM-019)** + - Conjunction rule: over timeout **and** no `watchdog_expired` terminal state. Needs 5.6. + - _Requirements: R8.2, R8.7_ + +- [ ] **6.7 `CHK-no-rewind-after-artifact` (FM-023)** + - v1 narrow rule only: re-entry triggered by a guardrail error whose phase differs from the + phase re-entered. + - **Definition of done:** a legitimate test-failure-driven re-entry fixture returns `pass`; the + narrow rule is documented in the check docstring with its reason. + - _Requirements: R8.5, R8.7_ + +- [ ] **6.8 Retroactive scoring over the existing corpus** + - **Definition of done:** results recorded with corpus kind and `n`; nothing described as + observed or prevalent. + - _Requirements: R8.7, R8.8_ + +- [ ] **6.9 Promotion path wired, not exercised** + - `taxonomy propose --from-annotations` surfaces which hypothesis modes unaided annotation + independently reached. No auto-promotion on a check firing. + - **Definition of done:** a test asserts a firing check does not promote; promotion requires an + annotation record with an indexed `trace_id`. + - _Requirements: R9.4, R9.5, R9.6_ + +## Phase 7 — Orphan signals + +- [ ] **7.1 Re-point `CHK-evaluator-capitulation` at `qa_verdict` spans** + - Delete the hand-rolled artifact re-parse at `verification.py:242–261`; keep `raw_result` as + fallback. + - **Definition of done:** outcomes unchanged on existing fixtures (task 2.5 guard); one JSON + parser fewer in the check layer. + - _Requirements: R7.1, R7.2_ + +- [ ] **7.2 `CHK-subagent-fanout-bounded`** + - Consumes `subagent_start` / `subagent_stop`; bounds fan-out and catches orphaned subagents. + - _Requirements: R7.1, R7.4_ + +- [ ] **7.3 Fold `session_start` into `CHK-run-record-complete`** + - _Requirements: R7.1, R7.3_ + +- [ ] **7.4 Delete the `regression_check` parser** + - **Definition of done:** parser and span type removed; decision and reason recorded in + `design.md` §4; R4.6's gate would catch a future re-introduction without a consumer. + - _Requirements: R7.1, R7.3, R7.5_ + +- [ ] **7.5 Assert zero orphans** + - **Definition of done:** a test fails when any span type has a parser and no consumer. + - _Requirements: R7.1_ + +## Phase 8 — Harness efficiency (spends ≤ $1.00, human-triggered) + +- [ ] **8.1 Bare-provider control runner** + - Same brief, one provider call; wall, tokens, cost recorded as a trace with `arm: "control"` + reusing `harness-alignment` R1. + - **Definition of done:** `--dry-run` resolution printed and approved before any call. + - _Requirements: R10.1, R10.2, R10.5_ + +- [ ] **8.2 `overhead_ratio` / `cost_ratio` in the report** + - With `n` and the control trace id; `n=1` renders `ANECDOTE`. + - **Definition of done:** the 1600 s / 1.528 s observation renders as an anecdote, not a ratio. + - _Requirements: R10.3, R10.4, R10.7_ + +- [ ] **8.3 `CHK-overhead-ratio` (FM-024)** + - Ceiling from the scenario contract, not hardcoded; `not_applicable` with no control. + - _Requirements: R8.6, R10.6_ + +- [ ] **8.4 Resolve open decision 9.3** + - Decide whether overhead reports in Tier A or only in the ladder; record the decision. + - _Requirements: R10.3_ + +--- + +## Minimum defensible slice + +**Phase 1, plus tasks 2.1–2.4, plus 6.3.** Liveness measured and published, checks declaring +what they read, and one new check that fires on the corpus that already exists. That converts +"20 checks registered" into a number that means something. + +If even that is too long: **tasks 1.3 and 1.4**. Publish the liveness table and correct the +coverage claims it invalidates. No code, and it removes the most misleading sentence the repo +currently ships. diff --git a/.kiro/specs/eval-methodology-alignment/README.md b/.kiro/specs/eval-methodology-alignment/README.md new file mode 100644 index 00000000..c1a71ab6 --- /dev/null +++ b/.kiro/specs/eval-methodology-alignment/README.md @@ -0,0 +1,170 @@ +# Spec: `eval-methodology-alignment` + +Kiro-style three-document spec for closing the gap between the eval machinery this repo +has built and the error-analysis methodology it claims to follow. Read in order: + +1. **[`requirements.md`](./requirements.md)** — 16 requirements with EARS acceptance + criteria, one new failure mode (FM-018), constraints, and non-goals. +2. **[`design.md`](./design.md)** — harness-owned telemetry, honest backfill states, + corpus diversity floors, the error-analysis session protocol, taxonomy provenance, + and claim discipline (§8). +3. **[`tasks.md`](./tasks.md)** — 9 phases, 47 tasks, each with a definition of done and + requirement traceability. + +Builds on [`../eval-harness/`](../eval-harness/) (Trace boundary, check registry, +taxonomy, tiers, provenance) and [`../harness-alignment/`](../harness-alignment/) (arms, +ablation, FM-014…FM-017). Neither is restated here. + +## The one-line version + +Every stage of the error-analysis loop downstream of "look at your data" is implemented, +and **not one of them has ever seen real data**. Fix the source: make runs emit telemetry +the harness owns, build a corpus that is actually diverse, and do the human open coding +the taxonomy was supposed to come from. + +## What the audit found + +| Claim in the docs | What the repo actually holds | +| --- | --- | +| "Backfilled live corpus — 50 traces indexed" | 50 traces, **0 spans between them**, all `crewai`, all `scenario_id: unknown`, all `status: failed`, all created inside a **1.97-second window** on 2026-08-16 | +| "Error-analysis-first methodology" | `evals/annotations/` is **empty**. Zero traces open-coded. | +| Taxonomy FM-001…FM-017 | Derived from [`docs/posts/failure-taxonomy.md`](../../../docs/posts/failure-taxonomy.md), written from memory. **No FM traces to an annotation.** | +| "Judges … validated against human labels" | 1 judge prompt for 17 failure modes; golden set empty; alignment report `n: 0`, `eligible_to_gate: false` | +| Tier A "$0.00 per PR, checks green" | Scores **94 synthetic `__pass`/`__fail`/`__na` fixtures**. A green Tier A proves the check code works, not that `ai-team` works. | + +## Root cause: two parts, neither of them in `evals/` + +### 1. The corpus is built from the wrong tree + +`trace backfill` defaults to `--workspace-root ./workspace`. That tree holds what the +**agents wrote** — 440 directories, 221 of them empty, the other 219 containing +generated `src/` and `tests/` and no telemetry at all. + +What the **harness** wrote went somewhere else entirely: + +| `./output/runs/` | count | +| --- | --- | +| dated run directories | **211** | +| `run.json` (backend, workspace_dir, completed_at, final_status) | **210** | +| `state.json` (actual cost, total tokens, monitor snapshot) | **141** | +| `logs/costs.jsonl` | **60** | +| `receipt.json` + `receipt.md` | **8** | + +Backends recorded there: `langgraph` 148, `crewai` 1, `claude-agent-sdk` 1, unset 60. +Date range **2026-07-06 → 2026-09-13 — a 69-day span.** + +So the corpus that was indexed as *50 traces, one backend, one scenario, one status, +1.97 seconds* could have been ~210 runs across three backends spanning 69 days, for +**$0.00**, on day one. Four of the five diversity floors in R4 were reachable from +disk the entire time. The eval harness has been reading the directory where the +generated code lands instead of the directory where the run records live. + +### 2. Phase telemetry is requested from the model, not written by the harness + +`phases.jsonl` is the file that yields the phase and retry spans the checks reason +over. Its only code writer is `src/ai_team/harness/context_pressure.py`, added +2026-09-12, which appends a single `phase_end` row when it records context pressure. +Otherwise it is asked for: + +``` +src/ai_team/backends/claude_agent_sdk_backend/agents/prompts.py:23 + 7. Write phase transition entries to workspace/logs/phases.jsonl +``` + +**Zero `phases.jsonl` files exist under `workspace/` or `output/runs/`.** So even +pointed at the right tree, the corpus would be run-level only — statuses, costs and +durations, no trajectory. Note the contrast: `audit.jsonl` and `costs.jsonl` *do* have +real code writers (`tools/bus.py`, `core/results/writer.py`, +`claude_agent_sdk_backend/costs.py`). Phase telemetry is the one signal left to a +prompt, and it is the one every trajectory check depends on. + +The honest statement of the gap is therefore not "we haven't done error analysis yet," +and not "we logged nothing." It is: **the harness logged plenty, into a tree the eval +corpus never looks at, and left the one signal the checks need to an instruction the +model was free to ignore.** + +## Sources it aligns with + +| Source | What it supplies | +| --- | --- | +| [Why AI evals are the hottest new skill for product builders](https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill) — Hamel Husain & Shreya Shankar, Lenny's Podcast | The loop: traces → open coding → axial coding → count → judge. The benevolent dictator. Theoretical saturation. Binary verdicts. "LLMs can't replace humans in the initial error analysis." | +| [Hamel Husain, *AI Evals FAQ*](https://hamel.dev/blog/posts/evals-faq/) | ~100 diverse traces as the working pool; ≥30 annotated unaided before agent assist; 100–200 labeled examples per judged failure mode; TPR **and** TNR, never accuracy; generic metrics as a sampling signal and never a quality claim; build the annotation tool yourself; re-run analysis every 2–4 weeks; "60–80% of development time on error analysis and evaluation." | + +The methodology is not new to this repo — `docs/EVAL_METHODOLOGY.md` already cites it and +`evals/annotate.py` already refuses to show LLM suggestions (R3.7). This spec is not a +change of direction. It is the part that was deferred, and the instrumentation work that +turns out to gate it. + +## Why instrumentation is phase zero + +``` + harness owns the logs → corpus reads the right tree → human open-codes 100 → axial + (Phase 1) (Phase 2) (Phase 4) (Phase 5) + ↓ + judges earn their FMs ← labels reach 100–200/FM ← taxonomy re-derived + (Phase 7) (Phase 6) (Phase 5) +``` + +Everything to the right of Phase 1 is already built. `evals/sampling.py`, +`evals/annotate.py`, `evals/taxonomy/propose.py`, `evals/golden.py`, `evals/alignment.py`, +`judge align`, `judge validate`, split discipline, bias correction, Wilson intervals — all +of it exists, is typed, and is tested. It has been starved, not missing. + +Which makes this the cheapest spec of the three: **most of the work is deleting the +excuse, not building the tool.** Task 2.4 alone — re-index `output/runs/` instead of +`workspace/` — is a one-argument change that takes the corpus from 1 backend and a +1.97-second span to 3 backends and 69 days, for nothing. + +## New failure mode + +| ID | Slug | Layer | Check | +| --- | --- | --- | --- | +| FM-018 | `self_reported_telemetry` | harness | `CHK-telemetry-provenance` | + +A run whose phase, cost, or tool records were written by the agent rather than by the +harness. Deterministic, Tier A, $0.00, and scoreable retroactively over every trace the +corpus already holds — which will mark all 50 of them. + +FM-018 is the reason FM-016 (`self_graded_verification`) does not go far enough. FM-016 +catches an agent grading its own work. FM-018 catches an agent *reporting* its own work — +the weaker and more corrosive version, because it silently degrades every measurement +built on top of it rather than producing a visibly wrong verdict. + +## Constraints baked in + +| | | +| --- | --- | +| Location | extends `src/ai_team/harness/`, `evals/trace/`, `evals/annotate.py`; no new package | +| Telemetry | **harness-owned**: written by harness code on the call path, never by agent instruction; the prompt line is deleted, not supplemented | +| Corpus source | run records under `output/runs/`, joined to `workspace//` for artifacts — never the workspace tree alone | +| Backfill | a trace with zero spans is `unindexable`, never `failed` — absence of evidence is not evidence of failure | +| Corpus floor | ≥100 traces, ≥3 backends, ≥4 scenario ids, ≥2 statuses, spanning ≥14 days; below any floor the corpus is stamped `NON-REPRESENTATIVE` | +| Open coding | human, single annotator, ≥30 unaided before any agent assist; `taxonomy propose` may cluster, never label | +| Judges | an FM earns a judge only after it survives a prompt fix and reaches ≥100 labels; below that it stays `detection: check` or `manual` | +| Claim discipline | fixture pass rates render as `FIXTURE-ONLY`; corpus rates carry `n`, Wilson CI, and the diversity stamp | +| Spend | ≤ **$15.00** total: ≤$10 to re-run scenarios for a diverse corpus, ≤$5 for judge align/validate. Additive to the eval-harness $5 and the ladder $25. | +| Out of scope | multi-annotator κ, online/production monitoring, vendor eval platforms, RAG evals, changing any existing check's semantics | + +## Minimum defensible slice + +**Phases 1, 2, and 4.** Harness-owned telemetry, a corpus that clears the diversity floor, +and 100 traces open-coded by a human. That alone converts the project's central claim — +"we measure our failures" — from scaffolding into a number. + +If even that is too long: **Phase 1 plus task 4.2**. Instrument the runs, then annotate 30 +of them yourself. Thirty traces read by hand will tell you more about what to build next +than the other eight phases combined, and that is the finding the source material is +actually about. + +## Executing this with Cursor + +``` +Read .kiro/specs/eval-methodology-alignment/requirements.md and design.md for context. +Implement task 1.2 from .kiro/specs/eval-methodology-alignment/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +Phases 0–3 and 5–8 are free and safe to run unattended. **Phase 2.5 (corpus re-runs) and +Phase 7 (judge align/validate) spend money** and are human-triggered with ceilings. +**Phase 4 cannot be delegated to an agent at all** — that is the entire point. diff --git a/.kiro/specs/eval-methodology-alignment/design.md b/.kiro/specs/eval-methodology-alignment/design.md new file mode 100644 index 00000000..4d49405e --- /dev/null +++ b/.kiro/specs/eval-methodology-alignment/design.md @@ -0,0 +1,345 @@ +# Design — Eval Methodology Alignment + +**Spec ID:** `eval-methodology-alignment` +**Requirements:** [`requirements.md`](./requirements.md) · **Plan:** [`tasks.md`](./tasks.md) + +--- + +## 1. Design intent + +One sentence: **move the boundary of what the harness owns upstream, from scoring to +observation.** + +`eval-harness` drew a boundary at the Trace — left of it billed and flaky, right of it free +and deterministic — and then built everything to the right of that line. What it assumed +was that the left side produces traces where the right side looks for them. It does not. +The harness writes run records into `output/runs/` — 210 of them, three backends, 69 days — +and the corpus builder reads `workspace/`, which holds only the code the agents generated. +The one signal that is genuinely missing, phase telemetry, is the one still requested from +the model in a prompt; in 440 runs it was never produced. + +So this spec extends the same idea one step left: + +``` + execution observation scoring + ┌──────────┐ ┌───────────────┐ ┌──────────────────┐ + │ backend │ → │ harness-owned │ → │ Trace → checks │ + │ runs │ │ telemetry │ │ judges → report │ + └──────────┘ └───────────────┘ └──────────────────┘ + billed free, ours free, ours + flaky deterministic deterministic + ↑ NEW +``` + +Observation belongs with scoring, not with execution. A backend may be swapped, ablated, +vendored, or written by someone else; the record of what it did must not be. + +Everything downstream of this — sampling, annotation, clustering, golden sets, alignment — +is already built and needs no design work. §4 through §7 are therefore short: they describe +how existing components behave once they are fed, and the few places where a refusal path +has to be added so the loop cannot be skipped. + +## 2. Telemetry + +### 2.1 Module + +`src/ai_team/harness/telemetry.py` — the only writer of `/logs/*`, co-located +with `run.json` under `output/runs//` so the corpus and the telemetry share a tree. + +```python +class TelemetryWriter: + """Harness-owned run telemetry. The agent never writes here.""" + + def __init__(self, workspace: Path, run_id: str, backend: str, arm: str | None) -> None: ... + def session_start(self, scenario_id: str, model_ids: Mapping[str, str], git_sha: str) -> None: ... + def phase_start(self, phase: str) -> None: ... + def phase_end(self, phase: str, end_status: str, used_tokens: int | None) -> None: ... + def cost(self, model_id: str, input_tokens: int, output_tokens: int, source: CostSource) -> None: ... + def tool_call(self, name: str, args_digest: str, outcome: str) -> None: ... +``` + +Every record carries `writer: "harness"`. The field is written by the module, not passed in, +so there is no call site that can forge it. + +### 2.2 Placement + +The writer is constructed by the `Backend` protocol wrapper, not by individual backends. +`run_backend(...)` opens the writer, hands it to the backend as an opaque callback object, +and closes it. A backend that ignores the callback still gets `session.json`, `phase_start` +for the run as a whole, and `phase_end` with a terminal status — degraded telemetry rather +than none. A new backend inherits the floor for free (R1.3). + +### 2.3 Failure is soft + +`TelemetryWriter` catches `OSError` on every write, records the first failure on the run's +warning list, and sets `degraded: true` in `session.json`. A run never fails because its +logging failed (R1.4) — but the trace built from it carries the degradation, so the number +is discounted rather than silently trusted. + +### 2.4 The prompt line goes away + +`src/ai_team/backends/claude_agent_sdk_backend/agents/prompts.py:23` is deleted (R1.5). +Keeping it alongside harness telemetry would produce two writers for one file and make +`writer` unresolvable at exactly the moment it matters. The prompt line is not a fallback; +it is the defect. + +## 3. Provenance and FM-018 + +### 3.1 The check + +``` +CHK-telemetry-provenance + fail ← any span derives from a record with writer != "harness" + pass ← every contributing record has writer == "harness" + na ← the trace has no telemetry records at all +``` + +`na` rather than `pass` on the empty case (R2.6) is the whole point. The current corpus +would otherwise score 50/50 clean on a check about record-keeping, having kept no records. + +### 3.2 Why FM-018 is not FM-016 + +| | FM-016 `self_graded_verification` | FM-018 `self_reported_telemetry` | +| --- | --- | --- | +| What the agent does | grades its own output | records its own behavior | +| Failure is | visible — a wrong verdict | invisible — a missing or flattering row | +| Blast radius | one run's gate decision | every statistic built on the corpus | +| Detection | compare verifier identity to producer | inspect record `writer` | + +FM-016 produces a bad answer you can argue with. FM-018 produces a clean-looking dataset, +which is worse, because nothing downstream has any way to know. + +### 3.3 Retroactive scoring + +FM-018 runs over the corpus as it stands and marks all 50 traces. That is recorded as the +**baseline**, stamped with the date and taxonomy version (R2.5). A later drop in the FM-018 +rate is then attributable to Phase 1 landing, which is the cleanest before/after this repo +will ever get for a harness change — and worth capturing as a result rather than quietly +fixing. + +## 4. Corpus + +### 4.1 States + +| State | Meaning | Counts toward rates | Eligible for open coding | Eligible for golden labels | +| --- | --- | --- | --- | --- | +| `complete` | full telemetry present | yes | yes | yes | +| `partial` | run record present, phase telemetry absent | yes, flagged | yes | **no** (R5.5) | +| `unindexable` | nothing recoverable | **no** (R3.2) | no | no | +| *(skipped)* | empty workspace, no trace created | — | — | — | + +The migration (R3.4) reclassifies today's 50 traces. They have no spans and no logs, so they +land as `unindexable` and are then superseded by the re-index of `output/runs/`, where most +of the same runs reappear as `partial` with a real backend, status, cost and duration. + +### 4.2 Profile and stamp + +```python +class CorpusProfile(BaseModel): + n_total: int + n_indexable: int + n_partial: int + n_unindexable: int + backends: dict[str, int] + scenarios: dict[str, int] # "unknown" excluded from the distinct count (R4.5) + statuses: dict[str, int] + arms: dict[str, int] + day_span: int + representative: bool + unmet_floors: list[str] +``` + +Floors: `n_indexable ≥ 100`, backends ≥ 3, scenarios ≥ 4, statuses ≥ 2, day span ≥ 14. + +Today's profile — built from `workspace/` — fails all five and renders: + +``` +NON-REPRESENTATIVE — n_indexable=0 (<100), backends=1 (<3), +scenarios=0 (<4), statuses=1 (<2), day_span=0 (<14) +``` + +The stamp is produced by the renderer (R4.4). Prose cannot opt out of it, and prose that +states a rate without it is a defect — which is how `docs/EVAL_METHODOLOGY.md` came to +describe a "backfilled live corpus" of 50 traces that contain nothing, while 210 usable run +records sat one directory over. + +### 4.3 Filling the floors + +Order of preference, cheapest first: + +1. **Re-index `output/runs/`** under the R5 rules — not `workspace/`. Free. Expected to + yield ~210 traces across three backends over a 69-day span, which clears the count, + backend and day-span floors immediately and likely the status floor. This single change + is the highest-value task in the spec and costs nothing. +2. **Re-run** only the cells still empty after (1), under the $10 ceiling (R5.4), chosen to + fill backend and scenario cells rather than to add volume. +3. **Generate** scenarios from dimension tuples (§5) for cells no existing scenario covers. + +The report in R5.3 prints the shortfall per floor *before* any spend is authorized, so the +$10 buys named cells rather than another batch of the same run. + +## 5. Scenario dimensions + +`evals/scenarios/dimensions.yaml` enumerates axes; generation is two-step (R6.2) — tuples +first, prose second — because a model asked for "20 varied scenarios" produces 20 rewordings +of one scenario, and a model asked to render a specified tuple produces the tuple. + +| Axis | Values | +| --- | --- | +| `task_size` | `single_file`, `small_service`, `full_stack` | +| `spec_completeness` | `thin` (1–4 sentences), `normal`, `over_specified` | +| `stack_familiarity` | `mainstream`, `niche` | +| `existing_codebase` | `greenfield`, `brownfield` | +| `expected_surface` | `tooling`, `runtime`, `integration`, `spec_ambiguity` | + +`spec_completeness: thin` closes the limitation `harness-alignment` §9.4 left open: every +arm today receives a full contract, so planner over-specification risk has never been +exercised. It arrives here as a corpus dimension rather than as a new arm, which costs +nothing extra. + +Coverage is reported per cell, and an unvisited cell renders `never measured` — distinct +from a cell measured at zero (R6.5), the same distinction `harness-alignment` R15 draws for +stale ablations. + +## 6. The error-analysis session + +### 6.1 Protocol + +``` +sample (stratified, n=100, seed recorded) + ↓ +annotate ─ 30 unaided, no tooling assist ─┐ + ↓ │ unaided: true +annotate ─ continue to saturation ─────────┘ + ↓ +saturation test: 20 consecutive traces, no new tag → stop + ↓ +taxonomy propose --from-annotations (clusters only; never writes) + ↓ +human accept / merge / reject → failure_modes.yaml v2.0.0 + ↓ +frequency table with Wilson CIs, stamped with sample_id and corpus profile +``` + +### 6.2 The refusals that make it real + +The loop has been skippable because nothing refused. Three refusals are added: + +| Refusal | Trigger | Requirement | +| --- | --- | --- | +| `taxonomy propose` exits non-zero | < 30 records with `unaided: true` | R7.4 | +| `judge validate` exits non-zero | < 100 human labels for the FM | R12.3 | +| re-run resolver exits non-zero | resolved cost > ceiling | R5.4 | + +Each refusal names what is missing and what would satisfy it. None of them is overridable by +a flag, because a flag is how a refusal becomes a formality. + +### 6.3 What the tooling may and may not do + +Permitted (per the source material): first-pass axial clustering, mapping annotations to +existing FM ids *after* they exist, summarizing annotation patterns, drafting judge prompts +from confirmed categories. + +Refused: initial open coding, validating the taxonomy, producing golden labels, deciding +root cause. `evals/annotate.py`'s module docstring already states this and instructs future +agents not to "helpfully" add AI assistance. That docstring is load-bearing and is extended, +not softened (R7.3). + +### 6.4 Saturation is operationalized + +"Stop when you stop learning" is unfalsifiable as written, so: **20 consecutive annotated +traces producing no new open-coding tag**, or 100 traces, whichever comes first, with the +terminating condition recorded in the session manifest (R7.5). The running distinct-tag +count is shown live in the TUI (R13.2) so the annotator can watch the curve flatten rather +than guess. + +## 7. Taxonomy provenance + +```yaml +- id: FM-001 + slug: tool_call_omission + origin: essay # essay | open_coding | reference | hypothesis + status: active # active | unobserved | retired + detection: check + evidence_annotations: [] +``` + +The migration (R9.3) stamps FM-001…FM-013 as `essay` and FM-014…FM-017 as `reference`. No +entry is promoted to `open_coding` without annotation ids behind it (R9.5). The share by +origin appears in `SuiteReport`, so a reader can see at a glance that a 17-mode taxonomy is +currently 100% un-observed — which is true today and should be uncomfortable to look at +until it changes. + +`status: unobserved` (R10.2) exists so that the first pass can be honest without being +destructive. A mode the data did not show is not thereby wrong; it is unconfirmed, and +deleting it would discard a real hypothesis while pretending to be rigorous. + +## 8. Claim discipline + +Three corpus kinds, rendered on every rate (R15.1): + +| Kind | Source | What a pass rate means | +| --- | --- | --- | +| `FIXTURE-ONLY` | `evals/fixtures/traces/` | the check code behaves as written | +| `CORPUS` | indexed traces from real runs | the system behaved this way on this sample | +| `LIVE` | a budgeted tier run just executed | the system behaves this way now | + +Today every green number in this repo is `FIXTURE-ONLY`. Nothing about that is dishonest — +`evals/golden/README.md` says so plainly — but the label has lived in a README while the +numbers travelled without it. Attaching the kind at render time is what stops a fixture +pass rate from arriving in a post as a quality claim (R15.5). + +Generic metrics stay demoted to a sampling signal (R15.3): useful for deciding which trace +to read next, never reported as quality. This is the one place the source material is +bluntest, and the repo's existing `metrics.py` scorecard is close enough to a generic +dashboard that the rule needs writing down. + +R15.6 closes the concrete instance already in the tree: `docs/campaign/EVAL_GATE_STATUS.md` +is referenced twice and does not exist. + +## 9. Error handling + +| Condition | Behavior | +| --- | --- | +| Telemetry write fails | warning, `degraded: true`, run continues (R1.4) | +| Workspace empty | skipped, no trace (R3.3) | +| Workspace has no logs but has artifacts | `partial` trace with recovered sources (R5.1) | +| Annotation session interrupted | resumable from the sample manifest (R13.3) | +| `taxonomy propose` under 30 unaided | exit 2, names the shortfall (R7.4) | +| `judge validate` under 100 labels | exit 2, names the count (R12.3) | +| Re-run resolves above ceiling | exit 2, prints the resolved plan and cost (R5.4) | +| Corpus below a floor | `NON-REPRESENTATIVE` stamp, never a CI failure (R4.6) | + +## 10. Testing strategy + +- **Telemetry:** unit tests per record type; a read-only logs dir test for the degraded path + (R16.3); a test asserting no backend module writes to `logs/` (grep-style guard, so the + defect cannot return). +- **FM-018 check:** `pass` / `fail` / `na` fixtures (R16.4), plus a retroactive run over a + frozen copy of the current corpus asserting 50 marks. +- **Backfill states:** table-driven over synthetic workspace shapes — empty, artifacts-only, + logs-only, complete. +- **Refusal paths:** 100% branch coverage on all three refusals (R16.2). +- **Isolation:** no test writes to `evals/annotations/`, `evals/golden/`, `evals/traces/` + (R16.1). This repo has already been bitten by golden-file writes in tests; the guard is a + conftest fixture that fails on any write under those roots. + +## 11. Open decisions + +1. **Does `partial` earn a judge label after Phase 1 lands?** A trace re-created with full + telemetry supersedes its `partial` predecessor; whether the old annotation carries forward + is unresolved. Proposal: annotations carry forward, golden labels do not. +2. **Annotation UI: TUI or local web?** R13 is satisfiable either way. The source material + favors building it yourself in hours over adopting a vendor; the repo already has a web UI + under `src/ai_team/ui/web/` that could host it, at the cost of coupling the eval loop to a + frontend build. +3. **How much of the 219 `partial` set is worth annotating?** Reading artifacts without a + phase timeline is slow and low-yield. Proposal: annotate 30 `partial` traces to calibrate, + then decide whether to wait for post-Phase-1 runs before the full 100. +4. **FM-018 on `reference` and `solo` arms.** The vendored reference harness writes its own + progress files. Whether those count as `writer: agent` is a judgement call with a real + consequence for the ladder comparison. Proposal: vendored-harness records are + `writer: vendor`, reported separately, and excluded from FM-018 rather than failed by it. +5. **Whether the frequency table is published.** The first honest failure-rate table this + repo produces will be worse-looking than the fixture numbers it replaces. That is the + point, and it is still a decision. diff --git a/.kiro/specs/eval-methodology-alignment/requirements.md b/.kiro/specs/eval-methodology-alignment/requirements.md new file mode 100644 index 00000000..906b2e72 --- /dev/null +++ b/.kiro/specs/eval-methodology-alignment/requirements.md @@ -0,0 +1,425 @@ +# Requirements — Eval Methodology Alignment + +**Spec ID:** `eval-methodology-alignment` +**Status:** Draft for implementation +**Owner:** Rick Zakharov +**Target repo:** `ai-team` (extends `src/ai_team/harness/`, `evals/trace/`, `evals/annotate.py`) +**Created:** 2026-09-13 +**Depends on:** [`../eval-harness/`](../eval-harness/) — Trace boundary (R1), sampling (R2), +annotation (R3), taxonomy (R5), golden sets and alignment (R8), tiers and budget (R11), +provenance (R14). [`../harness-alignment/`](../harness-alignment/) — arms (R1), FM-014…FM-017. +Neither is restated. + +--- + +## Introduction + +This repo has built the error-analysis loop twice over and never run it once. + +`evals/` contains a trace store, five sampling strategies, an open-coding TUI that +deliberately refuses to show model suggestions, an axial-coding clusterer, a golden-label +store with deterministic dev/test splits, judge alignment with bootstrap CIs, Cohen's κ, +prevalence bias correction, a verdict cache, a budget ledger, and a gate. `docs/EVAL_METHODOLOGY.md` +cites Husain and Shankar by name and states the loop correctly. The machinery is real. + +The inputs are not. + +| Stage | Implemented | Has ever run on real data | +| --- | --- | --- | +| Trace capture | yes | **no** — 50 traces, 0 spans | +| Sampling | yes | once; asked for 100, got 50, all one stratum | +| Open coding | yes | **no** — `evals/annotations/` is empty | +| Axial coding | yes | **no** — nothing to cluster | +| Taxonomy | 17 FMs | derived from an essay, not from traces | +| Golden labels | yes | **no** — 0 labels | +| Judge alignment | yes | `n: 0`, `eligible_to_gate: false` | +| Tier A gate | yes | scores 94 synthetic fixtures | + +The cause is upstream of all of it, and it is two things. + +**The corpus is built from the wrong tree.** `trace backfill` defaults to +`--workspace-root ./workspace`, which holds what the *agents* wrote: 440 directories, 221 +empty, 219 containing generated `src/` and `tests/` and no telemetry. What the *harness* +wrote went to `./output/runs/` — **211 dated run directories** (plus 18 stray entries: eight GUID dirs and loose `h1`–`h4`, `p`, `ws`, `artifacts/`, `logs/`, `reports/`, `state.json` — the same run-id litter the July journal recorded), of which **210** carry a +`run.json` (backend, `workspace_dir`, `completed_at`, `final_status`), **141** a +`state.json` (actual cost, total tokens), **60** a `logs/costs.jsonl`, and **8** a signed +receipt. Backends recorded there: `langgraph` 148, `crewai` 1, `claude-agent-sdk` 1, unset +60, across **2026-07-06 → 2026-09-13 — a 69-day span.** The corpus indexed as one backend +over 1.97 seconds could have been three backends over 69 days, for $0.00, on day one. + +**Phase telemetry is requested from the model rather than written by the harness.** +`phases.jsonl` yields the phase and retry spans every trajectory check reasons over. Its +only code writer is `src/ai_team/harness/context_pressure.py` (added 2026-09-12, one +`phase_end` row); otherwise it is asked for at `prompts.py:23`: *"Write phase transition +entries to workspace/logs/phases.jsonl."* **Zero such files exist under `workspace/` or +`output/runs/`.** `audit.jsonl` and `costs.jsonl`, by contrast, have real code writers +(`tools/bus.py`, `core/results/writer.py`, `claude_agent_sdk_backend/costs.py`). Phase +telemetry is the single signal left to a prompt, and it is the one the checks need most. + +So the harness is not measuring `ai-team`. It is measuring its own fixtures, honestly +labelled as such in `evals/golden/README.md` and `docs/EVAL_METHODOLOGY.md`, and the +honesty has been doing the work that data should be doing. + +### What this spec adds + +| Area | Gap today | Requirement | +| --- | --- | --- | +| Corpus source | backfill reads `workspace/`; 210 run records sit unread in `output/runs/` | R5 | +| Telemetry ownership | phase records asked for in a prompt; 0 `phases.jsonl` anywhere | R1, R2 | +| Backfill honesty | empty trace recorded as `failed` | R3 | +| Corpus diversity | 50 traces, 1 backend, 1 scenario, 1 status, 2-second span | R4, R5 | +| Scenario variation | one contract shape; no dimension sampling | R6 | +| Error analysis | never performed | R7, R8 | +| Taxonomy provenance | no FM records where it came from | R9, R10 | +| Judge scope | 1 prompt, 17 FMs, 0 labels | R11, R12 | +| Annotation ergonomics | `$EDITOR` round-trip per trace | R13 | +| Review cadence | none | R14 | +| Claim discipline | fixture pass rate reads like a system pass rate | R15 | +| Test hygiene | — | R16 | + +--- + +## R1 — Harness-owned phase telemetry + +**User story.** As the owner of the eval harness, I want run telemetry written by harness +code on the execution path, so that a measurement does not depend on an agent choosing to +report itself. + +### Acceptance criteria + +1. WHEN any backend begins a phase, the harness SHALL append a `phase_start` record to + `/logs/phases.jsonl` containing `phase`, `timestamp`, `run_id`, `backend`, and + `writer: "harness"`. +2. WHEN any backend completes or aborts a phase, the harness SHALL append a `phase_end` + record carrying `end_status`, `duration_s`, and `context_pressure` where a token count is + available, and `context_pressure: null` where it is not. +3. The harness SHALL write these records from a single module + (`src/ai_team/harness/telemetry.py`) invoked by the `Backend` protocol wrapper, so that a + new backend inherits telemetry without implementing it. +4. IF a telemetry write fails, THEN the harness SHALL record a warning on the run and + continue; telemetry failure SHALL NOT fail a run. +5. The instruction *"Write phase transition entries to workspace/logs/phases.jsonl"* SHALL be + **removed** from `agents/prompts.py`, not supplemented. Agent-written telemetry is the + defect, and leaving the instruction in place makes provenance unresolvable. +6. WHERE a backend can report per-call token and dollar usage, the harness SHALL append it to + `/logs/costs.jsonl` with `writer: "harness"` and a `source` field distinguishing + `provider_response` from `estimated`. +7. Harness-written telemetry SHALL be co-located with the run record under + `output/runs//logs/`, which is the tree the corpus reads (R5.1). Writing run + telemetry only into `workspace//logs/` is the defect this requirement removes. +8. The harness SHALL write `/logs/session.json` at run start with `run_id`, + `scenario_id`, `backend`, `arm`, `model_ids`, `git_sha`, and `started_at`, so that a trace + built later never has to infer `scenario_id` from a directory name. + +## R2 — Telemetry provenance is checkable (FM-018) + +**User story.** As a reviewer, I want to see at a glance whether a trace's telemetry was +written by the harness or by the model, so that I know what a number is worth. + +### Acceptance criteria + +1. Every telemetry record SHALL carry a `writer` field with value `harness` or `agent`. +2. The taxonomy SHALL gain **FM-018 `self_reported_telemetry`**, layer `harness`, detection + `check`, bound to `CHK-telemetry-provenance`. +3. `CHK-telemetry-provenance` SHALL fail a trace WHEN any span derives from a record whose + `writer` is `agent` or absent, and SHALL pass WHEN every record is `writer: harness`. +4. The check SHALL run in Tier A at $0.00 and SHALL be scoreable retroactively over the + existing corpus. +5. WHEN the check is run over the corpus as it stands on 2026-09-13, the report SHALL show + FM-018 present in all 50 traces. That result is the baseline, not a regression. +6. `CHK-telemetry-provenance` SHALL return `na` — not `pass` — for a trace with no telemetry + records at all, so that an empty run is never counted as clean. + +## R3 — Honest backfill states + +**User story.** As an analyst, I want an unreadable run to be distinguishable from a failed +run, so that corpus statistics are not poisoned by absence. + +### Acceptance criteria + +1. WHEN the trace builder finds no log files and no spans for a workspace, it SHALL record + `status: "unindexable"` with `unindexable_reason`, and SHALL NOT record `status: "failed"`. +2. `unindexable` traces SHALL be excluded from every pass-rate numerator and denominator, and + SHALL be counted and displayed separately in `index stats` and in `SuiteReport`. +3. WHEN a workspace directory is empty, the builder SHALL skip it and SHALL NOT create a trace. +4. `evals/cli.py index rebuild` SHALL reclassify the 50 existing traces under these rules in + a single migration, and the migration SHALL be idempotent. +5. Sampling strategies SHALL exclude `unindexable` traces by default, with `--include-unindexable` + available for corpus-health work. + +## R4 — Corpus diversity floor + +**User story.** As an analyst, I want the harness to refuse to call a monoculture a corpus, +so that I do not draw conclusions from 50 copies of one run. + +### Acceptance criteria + +1. The harness SHALL compute a `CorpusProfile` over the indexed corpus with counts by + `backend`, `scenario_id`, `status`, `arm`, and calendar day. +2. The corpus SHALL be considered **representative** only WHERE all of: `n_indexable ≥ 100`, + distinct backends ≥ 3, distinct `scenario_id` ≥ 4, distinct statuses ≥ 2, and span of + `started_at` ≥ 14 days. +3. IF any floor is unmet, THEN every report and sample manifest derived from that corpus + SHALL carry a `NON-REPRESENTATIVE` stamp naming the specific unmet floors. +4. The stamp SHALL be rendered by the report renderer, never asserted in prose. A document + that states a corpus rate without the stamp is a defect. +5. `scenario_id: "unknown"` SHALL count as zero distinct scenarios, not one. +6. Corpus floors SHALL NOT gate CI. An undersized corpus is a state to display, not a build + to break. + +## R5 — Corpus construction from the run-record tree + +**User story.** As the owner, I want the corpus built from where the harness actually +writes, so that new spend is only used for what is genuinely not on disk. + +### Acceptance criteria + +1. The trace builder SHALL take **`./output/runs/`** as its primary source, reading + `run.json`, `state.json`, `logs/*.jsonl` and `receipt.json` where present, and SHALL join + each run to its artifacts via the `workspace_dir` field that `run.json` already carries. +2. `--workspace-root` SHALL be retained as a secondary source for runs with no record in + `output/runs/`, and SHALL no longer be the default. +3. WHEN a run record supplies `backend`, `completed_at`, or `final_status`, the builder + SHALL use it rather than inferring from a directory name, and `scenario_id` SHALL fall + back to `unknown` only when no record supplies it. +4. The builder SHALL extract, from a run with records but no phase telemetry, whatever is + recoverable — backend, status, cost, tokens, duration, generated file inventory, test + results — and SHALL mark the resulting trace `partial: true` with a list of recovered + sources. +5. `partial` traces SHALL be eligible for open coding but SHALL NOT be eligible for judge + golden labels, since their evidence is incomplete. +6. The harness SHALL report, before any re-run spend is approved, how many traces each floor + in R4 is short by **after** the re-index, and which scenario/backend cells remain empty. +7. WHERE re-runs are still required to clear a floor, they SHALL be executed under an + explicit ceiling of **$10.00**, sequentially, with `--dry-run` resolution printed and + approved first. +8. The re-index SHALL be free. No requirement in this spec authorizes spend before R5.6 has + been run and read. + +## R6 — Scenario dimension sampling + +**User story.** As an analyst, I want the corpus to vary along dimensions I chose +deliberately, so that diversity is designed rather than hoped for. + +### Acceptance criteria + +1. The repo SHALL define `evals/scenarios/dimensions.yaml` enumerating the axes along which a + scenario can vary — at minimum: task size, stack familiarity, specification completeness + (including the `thin` brief left open by `harness-alignment` §9.4), presence of an existing + codebase, and expected failure surface. +2. Scenario generation SHALL proceed in two steps: emit structured dimension tuples, then + render each tuple to a scenario contract — never generate prose contracts directly. +3. Each generated scenario SHALL record its dimension tuple in its contract, so that corpus + coverage can be reported per dimension rather than only per `scenario_id`. +4. Generated scenarios SHALL be marked `synthetic: true` and SHALL be reported separately from + scenarios drawn from real use. +5. WHERE a dimension cell is unrepresented in the corpus, the coverage report SHALL name it as + `never measured`, distinct from a cell with a zero pass rate. + +## R7 — Error analysis is human and unaided + +**User story.** As the single annotator, I want the first pass of open coding to be mine +alone, so that the taxonomy reflects what the system does rather than what a model guessed. + +### Acceptance criteria + +1. Open coding SHALL be performed by one named human annotator — the benevolent dictator — + recorded in every `AnnotationRecord`. +2. The annotator SHALL complete **≥30 traces with no model assistance of any kind** before any + clustering, proposal, or suggestion tooling is run over the annotations. +3. `evals/annotate.py` SHALL continue to display no model output of any kind (R3.7 of + `eval-harness`), and this spec SHALL NOT relax that constraint under any justification. +4. An annotation session SHALL record `unaided: true|false` per record, and + `taxonomy propose` SHALL refuse to run WHEN fewer than 30 `unaided: true` records exist. +5. The session SHALL continue until **theoretical saturation** — defined operationally as 20 + consecutive traces producing no new open-coding tag — or until 100 traces are annotated, + whichever comes first, and the terminating condition SHALL be recorded. +6. Annotation SHALL NOT be delegated to a subagent, contractor, or model. A model-labeled + golden set makes every downstream number meaningless and the harness SHALL provide no + affordance for producing one. + +## R8 — Open codes are counted before they are acted on + +**User story.** As the owner, I want failure categories ranked by frequency, so that I fix +what is common rather than what is memorable. + +### Acceptance criteria + +1. After open coding, the harness SHALL produce a frequency table of axial categories with + counts, percentages, and Wilson 95% intervals over the annotated sample. +2. The table SHALL name the sample it was computed over (`sample_id`, `n`, corpus profile + stamp) and SHALL NOT be extrapolated to the corpus without that context. +3. `taxonomy propose` SHALL cluster open-coding tags into candidate categories and SHALL + present them for human accept/merge/reject — it SHALL NOT write to + `failure_modes.yaml` directly. +4. Axial categories SHALL be specific enough that a second reader can apply them; the spec's + working target is 5–8 categories over the first pass. + +## R9 — Taxonomy provenance + +**User story.** As a reader of the taxonomy, I want to know which failure modes were observed +and which were imagined, so that I can weight them accordingly. + +### Acceptance criteria + +1. Every entry in `failure_modes.yaml` SHALL carry `origin` with one of: `essay` (derived from + `docs/posts/failure-taxonomy.md`), `open_coding` (derived from annotations), `reference` + (derived from an external source, as FM-014…FM-017 are), or `hypothesis`. +2. Every `origin: open_coding` entry SHALL carry `evidence_annotations`: the annotation record + ids it was derived from, minimum 1. +3. All 17 existing entries SHALL be stamped at their true origin in a single migration — + FM-001…FM-013 as `essay`, FM-014…FM-017 as `reference` — with no re-labelling of essay + modes as observed. +4. `taxonomy coverage` SHALL report the count and share of active FMs by origin, and this + share SHALL appear in `SuiteReport`. +5. WHERE an `origin: essay` mode is later confirmed by open coding, its origin SHALL be + promoted to `open_coding` and its evidence recorded — promotion requires evidence, never + argument. + +## R10 — Taxonomy re-derivation and retirement + +**User story.** As the owner, I want the taxonomy to be answerable to the data, so that it +shrinks when the data does not support it. + +### Acceptance criteria + +1. After the first open-coding pass, each existing FM SHALL be classified as `confirmed`, + `unobserved`, or `refined` against the annotations. +2. An `unobserved` mode SHALL NOT be deleted. It SHALL be marked `status: unobserved` with the + sample it was not observed in, and excluded from active coverage counts. +3. Candidate categories from open coding with no existing FM SHALL be added as new FMs with + `origin: open_coding`, and SHALL NOT reuse a retired id. +4. `taxonomy_version` SHALL be bumped to `2.0.0` on first re-derivation, and every trace scored + under a prior version SHALL remain readable. + +## R11 — Judges are earned, not assumed + +**User story.** As the owner, I want a judge to exist only where a cheap check cannot do the +job and the failure has survived a prompt fix, so that I am not maintaining evaluators for +problems I already solved. + +### Acceptance criteria + +1. An FM SHALL be eligible for `detection: judge` only WHEN all of: it is `origin: open_coding` + or `confirmed`; it cannot be decided deterministically from a Trace; and it persisted across + at least one recorded prompt or harness fix. +2. The FM's `judge_rationale` field SHALL state why a deterministic check cannot decide it. +3. WHERE an FM is not judge-eligible, it SHALL remain `detection: check` or `detection: manual`. + `manual` SHALL be a first-class detection value reported as such, not a gap. +4. The existing `fm-001-tool-call-omission.v1.md` judge prompt SHALL be retired or re-scoped, + since FM-001 is `detection: check` and the prompt therefore evaluates nothing that gates. + +## R12 — Judge validation discipline + +**User story.** As a reviewer, I want every judge's true-positive and true-negative rates on a +held-out split, so that a judge that always says "pass" cannot look good. + +### Acceptance criteria + +1. A judge SHALL emit a **binary** verdict from a versioned prompt file. Continuous scores + SHALL NOT gate. +2. Each judge prompt SHALL contain explicit **exclusion rules** — named conditions that are + *not* the failure — and at least one labeled example of each side. +3. A judge SHALL require **≥100 human labels** for its failure mode before `judge validate` + may be run, with 100–200 as the working target. +4. Gating eligibility SHALL require TPR ≥ 0.90, TNR ≥ 0.90, κ ≥ 0.70, n ≥ 100 on the test split + (unchanged from `eval-harness` R8). +5. Overall accuracy SHALL NOT appear as a headline figure anywhere a judge is reported. WHERE + accuracy is shown, TPR and TNR SHALL be shown beside it. +6. Judge align and validate spend SHALL be capped at **$5.00** for this spec and SHALL be + human-triggered. + +## R13 — Annotation ergonomics + +**User story.** As the annotator, I want to move through 100 traces without friction, because +the cost of the loop is the reason it does not happen. + +### Acceptance criteria + +1. The annotation interface SHALL support single-keystroke advance, back, tag, and flag without + an `$EDITOR` round-trip for the common path. +2. It SHALL display session progress (`k of n`), elapsed time, and running distinct-tag count so + the annotator can see saturation approaching. +3. It SHALL support filtering and ordering the sample by backend, scenario, status, and + duration, and SHALL support resuming an interrupted session. +4. It SHALL render a trace in a form suited to this domain — phase timeline, first divergence, + generated file list, test output — not a raw JSON dump. +5. It SHALL remain local, dependency-light, and in-repo. No annotation vendor SHALL enter the + critical path. +6. Every ergonomic addition SHALL be evaluated against R7.3: if it would surface a model's + opinion, it SHALL NOT be built. + +## R14 — Review cadence + +**User story.** As the owner, I want error analysis to recur, so that the taxonomy tracks the +system rather than a moment in its history. + +### Acceptance criteria + +1. The repo SHALL define a recurring review: ≥100 fresh traces open-coded every 2–4 weeks, and + 10–20 outlier traces reviewed weekly between passes. +2. A review SHALL be triggered, regardless of calendar, WHEN the default model changes, a + backend is added, or a harness component is enabled or disabled. +3. `evals/cli.py index stats` SHALL report days since the last annotation record and the count + of traces created since, so staleness is visible without ceremony. +4. Staleness SHALL be reported, never gated. + +## R15 — Claim discipline + +**User story.** As a reader of this repo's public claims, I want to know what corpus a number +came from, so that a fixture pass rate cannot be mistaken for a quality measurement. + +### Acceptance criteria + +1. Every pass rate SHALL be rendered with its corpus kind: `FIXTURE-ONLY`, `CORPUS`, or `LIVE`. +2. A Tier A result over `evals/fixtures/traces/` SHALL render as `FIXTURE-ONLY` and SHALL NOT be + described as a measurement of `ai-team` in any document, README badge, or post. +3. Generic or off-the-shelf metrics SHALL be used only as a signal for selecting traces to + inspect, and SHALL NOT be reported as a quality claim. +4. `docs/EVAL_METHODOLOGY.md` SHALL carry a dated corpus-state table matching the current + `index stats` output, and CI SHALL warn WHEN the table is more than 30 days older than the + newest trace. +5. Every external-facing artifact derived from this work — posts, showcase pages, decks — SHALL + state the corpus kind and `n` for any rate it reports. A claim without `n` is a defect in the + claim, not a rounding of it. +6. The broken link `docs/campaign/EVAL_GATE_STATUS.md`, referenced from + `docs/EVAL_METHODOLOGY.md` and `docs/posts/harness-map.md`, SHALL be created or the + references removed. A gate-status claim pointing at a missing file is the failure mode this + requirement exists to prevent. + +## R16 — Test hygiene + +**User story.** As a maintainer, I want this spec's additions to be tested to the same standard +as the harness they instrument. + +### Acceptance criteria + +1. No test SHALL write to `evals/annotations/`, `evals/golden/`, or `evals/traces/`. Tests + SHALL use `tmp_path` roots. +2. Every refusal path added by this spec — `taxonomy propose` under 30 unaided records, judge + validate under 100 labels, re-run above ceiling, `unindexable` exclusion — SHALL reach 100% + branch coverage. +3. Telemetry writing SHALL be tested for the failure case: a read-only logs directory SHALL + produce a warning and a completed run. +4. The FM-018 check SHALL have `pass`, `fail`, and `na` fixtures like every other check. + +--- + +## Constraints + +| | | +| --- | --- | +| No new package | extends `src/ai_team/harness/`, `evals/trace/`, `evals/taxonomy/`, `evals/annotate.py` | +| No vendor in the critical path | no Braintrust / LangSmith / Arize / DeepEval; consistent with `eval-harness` | +| Budget | ≤ $10 corpus re-runs + ≤ $5 judge work = **≤ $15.00**, additive to the $5 suite and $25 ladder | +| Tier A stays $0.00 | FM-018 and every check added here is deterministic | +| Backwards compatible | traces scored under `taxonomy_version` 1.x remain readable | +| Human step is human | R7 is not delegable, and no task in `tasks.md` may be marked complete by an agent that performed it | + +## Non-goals + +- Multi-annotator agreement (κ between humans). Single benevolent dictator is the design. +- Online/production monitoring of a deployed system. This is a development-time loop. +- RAG or retrieval evaluation. +- Replacing any existing check's semantics, or re-opening the arm/ladder design. +- Proving a framework ranking. That remains out of reach and `harness-alignment` says why. diff --git a/.kiro/specs/eval-methodology-alignment/tasks.md b/.kiro/specs/eval-methodology-alignment/tasks.md new file mode 100644 index 00000000..8488f6d4 --- /dev/null +++ b/.kiro/specs/eval-methodology-alignment/tasks.md @@ -0,0 +1,335 @@ +# Implementation Plan — Eval Methodology Alignment + +**Spec ID:** `eval-methodology-alignment` +**Requirements:** [`requirements.md`](./requirements.md) · **Design:** [`design.md`](./design.md) + +9 phases, 47 tasks. Nothing is checked off — this plan has not been executed. + +--- + +## How to execute this plan + +- Work **one task at a time, in order**. Do not batch phases. +- Every task lists a **Definition of done**. Do not mark it complete until each bullet is + literally true. +- Run `uv run ruff check . && uv run ruff format --check . && uv run mypy src/ && uv run mypy evals/` + before marking any task complete. +- **Do not spend money** outside tasks 2.5 and 7.4, both human-triggered and capped. +- **Phase 4 is human work.** An agent may prepare the sample, run the tool, and process the + output. It may not annotate, and it may not mark 4.2–4.4 complete. If you are an agent + reading this: stop at 4.1 and say so. + +Cursor prompt shape: + +``` +Read .kiro/specs/eval-methodology-alignment/requirements.md and design.md for context. +Implement task 1.2 from .kiro/specs/eval-methodology-alignment/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +--- + +## Phase 0 — Ground truth about the present + +- [ ] **0.1 Freeze the current corpus as a baseline artifact** + - Copy `evals/traces/index.db` and the 50 trace JSONs to + `evals/baselines/corpus-2026-09-13/` with a `README.md` recording: 50 traces, 0 spans, + 1 backend, 1 scenario, 1 status, 2-second creation span. + - **Definition of done:** the frozen copy exists; a test asserts it is never written to. + - _Requirements: R2.5, R15.4_ + +- [ ] **0.2 Run census script — both trees** + - `scripts/run_census.py` — counts `workspace/` (empty vs artifacts-only vs logs) **and** + `output/runs/` (`run.json`, `state.json`, `logs/*.jsonl`, receipts), with backend, + status and date-span breakdowns. Emits JSON and a markdown table. + - **Definition of done:** reproduces `workspace/` 440 / 221 empty / 219 artifacts-only / + 0 with logs, and `output/runs/` 211 dated dirs (+18 stray) / 210 run.json / 141 state.json / 60 costs.jsonl + / 8 receipts, backends langgraph 148 · crewai 1 · claude-agent-sdk 1 · unset 60, span + 2026-07-06 → 2026-09-13. + - _Requirements: R5.6_ + +- [ ] **0.3 Record the audit in the journal** + - `docs/journal/2026-09-13-eval-methodology-audit.md` — what the census found, verbatim + numbers, the `prompts.py:23` root cause. + - **Definition of done:** entry follows house conventions; `journey.md` updated. + - _Requirements: R15.4_ + +## Phase 1 — Harness-owned telemetry + +- [ ] **1.1 `TelemetryWriter`** + - New `src/ai_team/harness/telemetry.py` per design §2.1, writing under + `output/runs//logs/` beside `run.json`. All records stamped `writer: "harness"` + by the module, not by callers. + - **Definition of done:** unit tests for every record type; `writer` cannot be overridden + from a call site (test asserts this). + - _Requirements: R1.1, R1.2, R1.6, R1.7, R2.1_ + +- [ ] **1.2 Wire it into the `Backend` wrapper** + - Construct in `run_backend(...)`; hand to the backend as a callback object; close on exit. + - A backend that ignores the callback still produces `session.json`, one `phase_start`, and + a terminal `phase_end`. + - **Definition of done:** a stub backend that calls nothing still yields a trace with + ≥2 spans and a resolved `scenario_id`. + - _Requirements: R1.3, R1.7_ + +- [ ] **1.3 Migrate `context_pressure.py`** + - Its direct `phases.jsonl` write becomes a `TelemetryWriter.phase_end(...)` call. No second + writer remains. + - **Definition of done:** grep for `phases.jsonl` in `src/` returns only `telemetry.py` + and the trace parsers. + - _Requirements: R1.3_ + +- [ ] **1.4 Delete the agent logging instruction** + - Remove line 23 of `agents/prompts.py`. Do not replace it with a weaker instruction. + - **Definition of done:** grep for `phases.jsonl` in any prompt template returns nothing; + a test guards it. + - _Requirements: R1.5_ + +- [ ] **1.5 Soft-failure path** + - `OSError` on write → warning + `degraded: true` in `session.json`; run completes. + - **Definition of done:** read-only logs dir test passes and the run succeeds. + - _Requirements: R1.4, R16.3_ + +- [ ] **1.6 Backend write guard** + - Test that no module under `src/ai_team/backends/` writes to `logs/`. + - **Definition of done:** guard fails when a write is reintroduced. + - _Requirements: R1.3, R16.1_ + +## Phase 2 — Corpus + +- [ ] **2.1 `unindexable` and `partial` states** + - Add to the trace model and builder per design §4.1; empty workspaces skipped. + - **Definition of done:** table-driven tests over empty / artifacts-only / logs-only / + complete workspace shapes. + - _Requirements: R3.1, R3.3, R5.1_ + +- [ ] **2.2 Exclude `unindexable` from rates** + - Numerators, denominators, sampling defaults; `--include-unindexable` for corpus health. + - **Definition of done:** a corpus of 10 traces, 4 unindexable, reports `n=6` with the + excluded count shown separately. + - _Requirements: R3.2, R3.5_ + +- [ ] **2.3 Idempotent reclassification migration** + - `index rebuild` reclassifies the existing 50 under the new rules. + - **Definition of done:** running twice produces identical output; the frozen baseline from + 0.1 is untouched. + - _Requirements: R3.4_ + +- [ ] **2.4 Re-index from `output/runs/` — the single highest-value task here** + - Make `output/runs/` the primary source; join to artifacts via `run.json:workspace_dir`; + keep `--workspace-root` as a secondary source, no longer the default. Recovery + extraction per R5.4; record recovered sources per trace. + - **Definition of done:** `index stats` reports ≥200 traces, ≥3 distinct backends and a + ≥60-day span, with `backend` and `final_status` taken from run records rather than + inferred. Costs **$0.00**. + - _Requirements: R5.1, R5.2, R5.3, R5.4, R5.5_ + +- [ ] **2.4b Shortfall report before any spend** + - Print, per R4 floor, what the re-indexed corpus is still short by and which + backend/scenario cells are empty. + - **Definition of done:** report runs free and is read before task 2.5 is started; task + 2.5 refuses to start without it. + - _Requirements: R5.6, R5.8_ + +- [ ] **2.5 Fill remaining cells — SPENDS ≤ $10.00, human-triggered** + - Print the R5.3 shortfall report; choose cells; `--dry-run` first; run sequentially. + - **Definition of done:** every dollar spent is attributable to a named empty cell; the + resolver refuses above ceiling. + - _Requirements: R5.3, R5.4_ + +- [ ] **2.6 `CorpusProfile` and the `NON-REPRESENTATIVE` stamp** + - Per design §4.2; rendered by the renderer, never by prose. + - **Definition of done:** the stamp appears on every report and sample manifest derived from + a sub-floor corpus and names the specific floors. + - _Requirements: R4.1, R4.2, R4.3, R4.4, R4.5, R4.6_ + +## Phase 3 — FM-018 and provenance + +- [ ] **3.1 Add FM-018 to the taxonomy** + - `self_reported_telemetry`, layer `harness`, detection `check`, origin `open_coding` + pending evidence — stamp `hypothesis` until Phase 5 confirms it. + - **Definition of done:** `taxonomy coverage` includes it; `taxonomy_version` → `1.3.0`. + - _Requirements: R2.2_ + +- [ ] **3.2 `CHK-telemetry-provenance`** + - `pass` / `fail` / `na` semantics per design §3.1, including `na` on no records. + - **Definition of done:** three fixtures committed; Tier A cost unchanged at $0.00. + - _Requirements: R2.3, R2.4, R2.6, R16.4_ + +- [ ] **3.3 Score the frozen baseline** + - Run the check over `evals/baselines/corpus-2026-09-13/`; record the result. + - **Definition of done:** result recorded as a baseline, explicitly not a regression. + - _Requirements: R2.5_ + +- [ ] **3.4 `origin` and `status` fields on every FM** + - Schema + loader + migration stamping FM-001…FM-013 `essay`, FM-014…FM-017 `reference`. + - **Definition of done:** no entry is `open_coding` without `evidence_annotations`; loader + rejects one that is. + - _Requirements: R9.1, R9.2, R9.3_ + +- [ ] **3.5 Report origin share** + - `taxonomy coverage` and `SuiteReport` show counts and share by origin. + - **Definition of done:** today's output reads 13 essay / 4 reference / 1 hypothesis / + **0 open_coding**. + - _Requirements: R9.4_ + +## Phase 4 — Error analysis (HUMAN — not delegable) + +- [ ] **4.1 Prepare the sample** *(agent may do this)* + - `sample --strategy stratified -n 100 --seed 1` over the post-Phase-2 corpus; print the + imbalance report and the corpus stamp. + - **Definition of done:** manifest written; imbalances named; stamp shown. + - _Requirements: R4.3, R8.2_ + +- [ ] **4.2 Annotate 30 traces unaided** *(human only)* + - No clustering, no proposals, no model output of any kind. Records carry `unaided: true`. + - **Definition of done:** ≥30 records with `unaided: true` in `evals/annotations/`. + - _Requirements: R7.1, R7.2, R7.6_ + +- [ ] **4.3 Continue to saturation** *(human only)* + - Stop at 20 consecutive traces with no new tag, or at 100. Record which. + - **Definition of done:** terminating condition recorded in the session manifest. + - _Requirements: R7.5_ + +- [ ] **4.4 Axial coding** *(human decides; tooling clusters)* + - `taxonomy propose --from-annotations`; accept / merge / reject by hand into 5–8 categories. + - **Definition of done:** categories are specific enough for a second reader; none was + written by the clusterer directly into `failure_modes.yaml`. + - _Requirements: R8.3, R8.4_ + +- [ ] **4.5 Frequency table** + - Counts, percentages, Wilson 95% CIs, stamped with `sample_id`, `n`, corpus profile. + - **Definition of done:** table renders; no figure appears without its stamp. + - _Requirements: R8.1, R8.2, R15.1_ + +## Phase 5 — Taxonomy re-derivation + +- [ ] **5.1 Classify every existing FM against the annotations** + - `confirmed` / `unobserved` / `refined`, with evidence for each confirmation. + - **Definition of done:** all 18 classified; promotions carry annotation ids. + - _Requirements: R10.1, R9.5_ + +- [ ] **5.2 Mark unobserved modes** + - `status: unobserved` with the sample they were not observed in. Nothing deleted. + - **Definition of done:** unobserved modes excluded from active coverage counts and still + present in the file. + - _Requirements: R10.2_ + +- [ ] **5.3 Add newly observed modes** + - New FMs with `origin: open_coding` and evidence; no id reuse. + - **Definition of done:** loader accepts; `taxonomy_version` → `2.0.0`; old traces still + readable. + - _Requirements: R10.3, R10.4_ + +## Phase 6 — Annotation ergonomics + +- [ ] **6.1 Resolve open decision 2 (TUI vs local web)** + - Write the decision and its rationale into design §11 before building. + - **Definition of done:** decision recorded with the trade-off stated. + - _Requirements: R13.5_ + +- [ ] **6.2 Single-keystroke navigation and tagging** + - No `$EDITOR` round-trip on the common path. + - **Definition of done:** annotating a trace with one tag takes one keystroke plus the tag. + - _Requirements: R13.1_ + +- [ ] **6.3 Progress, elapsed time, distinct-tag counter** + - The saturation curve is visible while annotating. + - **Definition of done:** `k of n`, elapsed, and distinct-tag count update per trace. + - _Requirements: R13.2, R7.5_ + +- [ ] **6.4 Filter, order, resume** + - By backend, scenario, status, duration; interrupted sessions resume from the manifest. + - **Definition of done:** a killed session resumes at the next unannotated trace. + - _Requirements: R13.3_ + +- [ ] **6.5 Domain-shaped trace rendering** + - Phase timeline, first divergence, generated file list, test output — not raw JSON. + - **Definition of done:** a reviewer unfamiliar with the trace schema can read a card. + - _Requirements: R13.4_ + +- [ ] **6.6 Re-assert the no-model-output constraint** + - Extend the `annotate.py` docstring to cover every new surface added in 6.2–6.5. + - **Definition of done:** no code path in the annotation UI can display model output. + - _Requirements: R7.3, R13.6_ + +## Phase 7 — Judges + +- [ ] **7.1 Judge eligibility gate** + - An FM may be `detection: judge` only under R11.1; `judge_rationale` required. + - **Definition of done:** loader rejects a judge-detection FM that fails eligibility. + - _Requirements: R11.1, R11.2_ + +- [ ] **7.2 `manual` as a first-class detection value** + - Reported as `manual`, not as a coverage gap. + - **Definition of done:** coverage output distinguishes `check` / `judge` / `manual`. + - _Requirements: R11.3_ + +- [ ] **7.3 Retire or re-scope the FM-001 judge prompt** + - FM-001 is `detection: check`; its judge prompt gates nothing. + - **Definition of done:** the prompt is removed or bound to an eligible FM; the stale + `n: 0` alignment report is removed with it. + - _Requirements: R11.4_ + +- [ ] **7.4 Align and validate — SPENDS ≤ $5.00, human-triggered** + - Only for FMs with ≥100 labels. Binary verdicts, exclusion rules, labeled examples on both + sides. Align on dev, validate once per `prompt_hash` on test. + - **Definition of done:** TPR, TNR, κ and n reported; accuracy never reported alone. + - _Requirements: R12.1, R12.2, R12.3, R12.4, R12.5, R12.6_ + +- [ ] **7.5 Refusal coverage** + - 100% branch coverage on the three refusals in design §6.2. + - **Definition of done:** coverage report confirms; no refusal has an override flag. + - _Requirements: R16.2_ + +## Phase 8 — Cadence and claims + +- [ ] **8.1 Staleness reporting** + - `index stats` reports days since the last annotation and traces added since. + - **Definition of done:** reported, never gated. + - _Requirements: R14.3, R14.4_ + +- [ ] **8.2 Review triggers** + - Document the 2–4 week cycle, the weekly 10–20 outlier review, and the event triggers + (model change, new backend, component toggle). + - **Definition of done:** written into `docs/EVAL_METHODOLOGY.md`. + - _Requirements: R14.1, R14.2_ + +- [ ] **8.3 Corpus-kind rendering** + - Every rate renders `FIXTURE-ONLY` / `CORPUS` / `LIVE`. + - **Definition of done:** a Tier A fixture run cannot render a rate without + `FIXTURE-ONLY`. + - _Requirements: R15.1, R15.2_ + +- [ ] **8.4 Demote generic metrics** + - `metrics.py` scorecard output labeled as a sampling signal, not a quality claim. + - **Definition of done:** no generic metric appears as a headline figure. + - _Requirements: R15.3_ + +- [ ] **8.5 Dated corpus-state table + CI staleness warning** + - `docs/EVAL_METHODOLOGY.md` carries a table matching `index stats`; CI warns past 30 days. + - **Definition of done:** table matches current output; warning fires on a stale table. + - _Requirements: R15.4_ + +- [ ] **8.6 Fix the broken gate-status link** + - Create `docs/campaign/EVAL_GATE_STATUS.md` or remove both references to it. + - **Definition of done:** no dangling link remains in `docs/`. + - _Requirements: R15.6_ + +- [ ] **8.7 Claim audit of external artifacts** + - Every post, showcase page, and deck derived from this work states corpus kind and `n`. + - **Definition of done:** an artifact reporting a rate without `n` is treated as a defect + and corrected. + - _Requirements: R15.5_ + +--- + +## Minimum defensible slice + +**Phases 1, 2, 4.** Harness-owned telemetry, a corpus that clears its floors, and 100 traces +open-coded by a human. + +If that is too long: **Phase 1 plus task 4.2.** Instrument the runs, annotate thirty of them +yourself. Thirty traces read by hand will change what you build next more than the remaining +eight phases combined. diff --git a/.kiro/specs/harness-alignment/README.md b/.kiro/specs/harness-alignment/README.md new file mode 100644 index 00000000..10e23bb6 --- /dev/null +++ b/.kiro/specs/harness-alignment/README.md @@ -0,0 +1,109 @@ +# Spec: `harness-alignment` + +Kiro-style three-document spec for aligning `ai-team` with two published Anthropic +references and turning the comparison into a measurement. Read in order: + +1. **[`requirements.md`](./requirements.md)** — 17 requirements with EARS acceptance + criteria, four new failure modes (FM-014…FM-017), constraints, and non-goals. +2. **[`design.md`](./design.md)** — the arm abstraction, data models, component design, + the $25 sweep envelope, error handling, testing strategy (§7), claim discipline (§8), + and five open decisions (§9). +3. **[`tasks.md`](./tasks.md)** — 12 phases, ~47 tasks, each with a definition of done and + requirement traceability. + +Builds on [`../eval-harness/`](../eval-harness/) — the Trace boundary, check registry, +taxonomy, tiers, and provenance all come from there and are not restated here. + +## The one-line version + +Add an **arm** axis orthogonal to the existing `--backend` axis, so the project can measure +what its harness buys — against a solo agent, against Anthropic's published minimal +harness, and against itself with one component removed at a time — and let every new +detector run free in the $0 Tier A gate. + +## Sources + +| Source | What it supplies | +| --- | --- | +| [`anthropics/claude-quickstarts/autonomous-coding`](https://github.com/anthropics/claude-quickstarts/tree/main/autonomous-coding) @3313e97, MIT | The reference arm; the monotonic checklist invariant; the session loop; default-deny bash | +| [Harness Design for Long-Running Application Development](https://www.anthropic.com/engineering/harness-design-long-running-apps) (Rajasekaran, Anthropic Engineering) | The solo control arm; contract negotiation before implementation; context resets over compaction; "context anxiety"; self-evaluation bias | + +## Why a ladder rather than a leaderboard + +``` + solo → harnessed_solo → reference → ai_team (ablated) → ai_team (full) + nothing harness, 1 agent minimal one component off all components +``` + +The README already concedes the three-backend matrix "cannot yet support framework X beats +Y" at n=5 with mixed models. More samples do not fix that — a missing control arm does. +Every rung above runs the same scenario contract, emits the same `Trace`, and is scored by +the same checks, which turns "the harness is the real deliverable" from an assertion into +a number with a confidence interval. + +`harnessed_solo` is the rung that makes the ladder answer two questions instead of one: + +| Comparison | Isolates | +| --- | --- | +| `solo` → `harnessed_solo` | what the harness buys, holding agent count at one | +| `harnessed_solo` → `ai_team` | what the nine-role decomposition buys, harness fixed | + +Without it, `solo → ai_team` confounds harness with decomposition. The nine-agent team is +the most expensive assumption the project makes and currently the only one it never tests. + +## Constraints baked in + +| | | +| --- | --- | +| Location | extends `evals/` and `src/ai_team/harness/`; no new repo, no fourth backend | +| New checks | all deterministic, **Tier A at $0.00** | +| Sweep budget | ≤ **$25.00** per full ladder sweep, separate from the eval-harness $5 suite; `--dry-run` by default | +| Reference harness | vendored, pinned, **never edited**; adaptations recorded as divergences | +| Claim discipline | `UNDERPOWERED` / `MIXED-MODEL` stamps enforced in the renderer, not the README | +| Default state | Phases 8–10 ship **off by default** until their ablation shows they earn their place | +| Test hygiene | No test may write to tracked ground truth; every refusal path this spec adds reaches 100% branch coverage before its component is enabled (R17) | +| Component staleness | Every ablation result is stamped with the model it was measured on and reported `STALE` when that model is no longer the default — never enforced, only shown | +| Out of scope | framework rankings, gating design-quality judges, multi-annotator labeling, online evals | + +## New failure modes + +| ID | Slug | Layer | Check | +| --- | --- | --- | --- | +| FM-014 | `acceptance_criteria_mutation` | model | `CHK-acceptance-monotonic` | +| FM-015 | `premature_termination_under_context_pressure` | model | `CHK-premature-termination` | +| FM-016 | `self_graded_verification` | harness | `CHK-verifier-independence` | +| FM-017 | `evaluator_capitulation` | model | `CHK-evaluator-capitulation` | + +All four are checkable from a Trace with no model call, and all four can be run +retroactively over the existing corpus (task 2.7) before a single new dollar is spent. + +FM-017 is the evaluator that finds a real blocker and then approves the work anyway — +distinct from FM-016, where the evaluator is not independent in the first place, and +distinct from `eval-harness`'s offline judge alignment, which scores traces after the fact +rather than gating a build. Its false-negative rate needs no human labels: every item QA +accepted that a deterministic verifier later failed is a counted miss (R16.4). + +## Executing this with Cursor + +Point the agent at one task at a time: + +``` +Read .kiro/specs/harness-alignment/requirements.md and design.md for context. +Implement task 1.2 from .kiro/specs/harness-alignment/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +Phases 0–3 and 6 are free and safe to run unattended. **Phases 4, 5, 7.4 and 11 spend +money** and are human-triggered with explicit ceilings. Phases 8–10 change how a normal run +behaves and must stay behind their flags until measured. + +## Minimum defensible slice + +**Phases 0–2, plus 4, plus 6.** A protected acceptance list, three new deterministic failure +modes scored retroactively, the `solo` and `harnessed_solo` control arms, and a report that +refuses to overclaim. Phase 8 without Phase 1 is not worth building — resumption without a +durable, protected checklist is just a longer run. + +If even that is too long: **task 4.5**. Two live runs plus an existing `ai_team` trace +produce the only number here that could change what you build next. diff --git a/.kiro/specs/harness-alignment/design.md b/.kiro/specs/harness-alignment/design.md new file mode 100644 index 00000000..5849c592 --- /dev/null +++ b/.kiro/specs/harness-alignment/design.md @@ -0,0 +1,561 @@ +# Design — Harness Alignment + +**Spec ID:** `harness-alignment` +**Requirements:** [`requirements.md`](./requirements.md) +**Tasks:** [`tasks.md`](./tasks.md) + +--- + +## 1. Overview + +### 1.1 The central architectural move + +`eval-harness` separated **execution** from **scoring** at the `Trace` boundary. This +spec adds a second axis to the left of that boundary: **what produced the Trace.** + +``` + ARMS (billed, Tier C) TRACE SCORING (free, Tier A) + ──────────────────────── ───── ────────────────────── + solo ──┐ + reference ──┤ + ai_team ──┼──► evals/traces/*.json ──► checks FM-001…FM-016 + ai_team−smoke ──┤ ladder aggregation + ai_team−lessons ──┘ gate / report +``` + +The existing `--backend` axis swaps the *framework* while holding the harness constant. +The new `--arm` axis swaps the *harness* while holding framework and model constant. +They are orthogonal, and the second is the one the project's thesis actually rests on. + +A single rule keeps this cheap: **an arm is an adapter, never a `Backend`.** Arms produce +workspaces; `evals.trace.from_workspace()` does the rest. No check, judge, gate, or report +component learns that arms exist beyond one new grouping key. + +### 1.2 Design principles + +1. **Adapters, not integrations.** The reference harness is vendored and pinned, never + edited, never registered in `src/ai_team/backends/registry.py`. If it drifts upstream, + the pin is the answer. +2. **The harness owns the definition of done.** `ACCEPTANCE.json` is written by harness + code and denied to agents at the hook layer. An agent that can edit the bar is an agent + that will. +3. **Reset, don't compact.** Sessions start clean and re-derive from four files. This + matches what `harness/context.py` was already built for; R8 is the loop that was + missing around it, not a new state mechanism. +4. **Every new detector is deterministic.** All three new failure modes are checkable from + a Trace with no model call, which is what keeps them in the $0 Tier A gate. +5. **Components must earn their place.** Anything R4 can ablate is a hypothesis. That + includes the components this spec adds — contract negotiation and the bash allowlist + are ablatable from day one (R11.6, R14.6). +6. **The tooling enforces the caveats.** R5 puts `UNDERPOWERED` and `MIXED-MODEL` stamps in + code because a caveat that lives in prose is a caveat that gets dropped from the slide. +7. **Every component encodes an assumption about what the model cannot do alone, and + assumptions expire.** So an ablation result is scoped to the model it was measured on + (R15), and a component whose evidence predates the current default model is reported + `STALE` rather than quietly carried forward. The failure this guards against is not a + bug — it is a harness that keeps charging for a crutch the model outgrew. + +### 1.3 What this deliberately does not change + +`src/ai_team/backends/*` gains nothing except the hook changes in R14 and the acceptance +write-deny in R6.3. `evals/checks/*` gains three modules and one extension. The Trace +schema gains two optional fields (`context_pressure`, `arm_id`). Everything else is new +surface under `evals/arms/` and `src/ai_team/harness/`. + +--- + +## 2. The ladder + +### 2.1 Arm families + +| Arm id | Family | What it holds constant | What it removes | +| --- | --- | --- | --- | +| `solo` | `solo` | model, scenario, workspace layout | everything: roles, guardrails, smoke, lessons, constraints, sessions | +| `reference` | `reference` | model, scenario | `ai-team`'s harness entirely; substitutes the quickstart's | +| `harnessed_solo` | `ai_team` | model, scenario, seed, **the whole harness** | only the nine-role decomposition | +| `ai_team` | `ai_team` | — | nothing (control for ablations) | +| `ai_team_ablated:` | `ai_team` | model, scenario, seed, all other components | exactly one named component | + +`solo` and `reference` are *external* reference points; the ablation arms are *internal* +attribution. Both are needed: external tells you the harness is worth having, internal +tells you which parts. + +`harnessed_solo` is the rung that makes the ladder answer two questions instead of one. +Without it, `solo → ai_team` confounds **harness** with **decomposition**: nine roles and +every guardrail arrive together, so a win cannot be attributed to either. With it: + +| Comparison | Isolates | +| --- | --- | +| `solo` → `harnessed_solo` | what the harness buys, holding agent count at one | +| `harnessed_solo` → `ai_team` | what role decomposition buys, holding the harness fixed | + +This is not a hypothetical concern. The reference article removed its sprint-decomposition +construct outright at the next model version, on the finding that the generator "ran +coherently for over two hours without the sprint decomposition." A nine-agent team is the +single most expensive assumption `ai-team` makes, and it is currently the only one the +project never tests. `harnessed_solo` is implemented as `ai_team_ablated:role_decomposition` +and given its own id because it is a headline rung, not a footnote. + +### 2.2 Arm interface + +```python +# evals/arms/base.py +class ArmSpec(BaseModel): + arm_id: str + family: Literal["solo", "reference", "ai_team"] + model_ids: dict[str, str] # role -> model id; {"*": id} for single-agent + harness_components: frozenset[str] # active component names (R4.1 vocabulary) + source_ref: str # git sha, or "vendor:" for third-party + cost_controls: CostControls # spend_ceiling_usd, wall_clock_ceiling_s, max_sessions + divergences: list[Divergence] # forced differences, each with a reason + +class ArmRun(BaseModel): + arm_id: str + sweep_id: str + workspace: Path + status: Literal["ok", "failed", "budget_exhausted", "unavailable"] + started_at: datetime + ended_at: datetime + cost_usd: float + trace_id: str | None + +class Arm(Protocol): + arm_id: str + def describe(self) -> ArmSpec: ... + def run(self, scenario: ScenarioContract, workspace: Path, budget_usd: float) -> ArmRun: ... +``` + +`Divergence` is `{field, expected, actual, reason}`. It is the honesty mechanism: every +place an arm could not be held equal is written down and surfaced in the report (R5.4). + +### 2.3 Sweep execution + +`python -m evals.cli ladder run --scenario todo-api-beginner --arms solo,reference,ai_team --n 3` + +1. Resolve arms from the registry; fail fast on unknown ids. +2. Allocate a `sweep_id`; compute the shared workspace seed. +3. Run arms **sequentially** (not in parallel) — concurrent launches are FM-004 + `run_id_collision` and the point is not to reproduce it accidentally. +4. After each arm, assemble its Trace immediately and free the workspace to the corpus. +5. Write `evals/results/ladder//manifest.json` and per-arm `ArmRun` records. +6. Score offline: the ladder report is produced from Traces, so it can be re-rendered for + free after any check change. + +Default is `--dry-run`, which resolves and prints the plan and the projected cost without +spending. + +### 2.4 Cost envelope + +| Arm | Ceiling | Rationale | +| --- | --- | --- | +| `solo` | $3.00 / 30 min | Article's reference point was $9 at 20 min on a larger brief; beginner scenarios are smaller | +| `reference` | $8.00 / 90 min | Feature cap 25 (R3.3); the session loop is the expensive part | +| `ai_team` | $6.00 / 60 min | Current full-run cost plus headroom | +| each ablation | $6.00 / 60 min | Same as control | + +A full three-arm sweep at n=1 is ≈ $17; the R1 sweep ceiling is **$25.00** and is checked +before any arm starts, not after. This is entirely separate from the eval-harness $5 suite +budget: scoring stays $0 and Tier A is untouched. + +--- + +## 3. Data models + +### 3.1 Acceptance list + +```python +# src/ai_team/harness/acceptance.py +class AcceptanceItem(BaseModel): + id: str # sha256(description + steps)[:12], stable + category: Literal["functional", "style", "security", "performance"] + description: str + steps: list[str] + priority: int # lower is higher priority; assigned at plan time + passes: bool = False + verified_by: Literal["smoke", "ui_smoke", "test", "qa_agent"] | None = None + verified_at: datetime | None = None + verifier_identity: VerifierIdentity | None = None # R12.2 + evidence: list[str] = [] # workspace-relative artifact paths + demotions: list[Demotion] = [] + +class AcceptanceList(BaseModel): + schema_version: int = 1 + run_id: str + created_at: datetime + source: Literal["planning"] = "planning" + items: list[AcceptanceItem] +``` + +`id` is content-hashed so that a mutated description produces a *different* id — which is +precisely what makes `CHK-acceptance-monotonic` a set-difference rather than a diff +heuristic. + +### 3.2 Session record + +```python +class SessionRecord(BaseModel): + session_id: str + index: int + started_at: datetime + ended_at: datetime + status: Literal["ok", "dirty_exit", "error", "budget_exhausted"] + items_attempted: list[str] + items_passed: list[str] + items_demoted: list[str] + cost_usd: float + context_pressure_at_end: float | None + termination_reason: str +``` + +Written to `logs/sessions.jsonl`, which becomes a new span source in Trace assembly +(`session_start`, `session_end`, `regression_check`). + +### 3.3 Build contract + +```python +class BuildContract(BaseModel): + item_id: str + round: int + approach: str + testable_behaviors: list[str] + files_to_touch: list[str] + out_of_scope: list[str] + review: ContractReview | None = None # {verdict, reasons[], reviewed_at, reviewer} +``` + +At `docs/contracts/.json`. File-based handoff, per `CLAUDE.md`. + +### 3.4 Trace additions + +Two optional fields, both nullable so every existing fixture stays valid: + +- `Trace.arm_id: str | None` — grouping key for the ladder report. +- span `payload.context_pressure: float | None` on `phase_end` and `session_end`. + +Schema version bumps; the loader tolerates absence (eval-harness R1.6 already requires +graceful degradation). + +--- + +## 4. Components + +### 4.1 `evals/arms/` + +``` +evals/arms/ +├── base.py Arm protocol, ArmSpec, ArmRun, Divergence, CostControls +├── registry.py id -> factory, duplicate-registration guard +├── solo.py SoloArm (R2) +├── reference.py ReferenceArm wrapping the vendored harness (R3) +├── ai_team.py AiTeamArm + AblatedArm factory (R4) +├── ladder.py sweep orchestration, manifest, budget pre-check +└── vendor/ + └── autonomous_coding/ pinned MIT copy + PROVENANCE.md +``` + +**SoloArm** constructs a single `ClaudeSDKClient` with one system prompt, the scenario +brief, and the standard workspace scaffold pre-created so Trace assembly is uniform. It +reuses the existing cost and audit hooks — the arm is unharnessed with respect to +*behavioral* components, not with respect to *observability*. Measuring it requires +logging it. + +**ReferenceArm** shells the vendored `autonomous_agent_demo.py` with +`--project-dir --max-iterations N --model `, captures stdout, and parses +the demo's `[Tool: …]` / `[Done]` / `[BLOCKED]` markers into `tool_use` / `tool_result` +spans. Coverage is partial by construction: the reference harness emits no cost stream, so +cost comes from the SDK's own usage accounting at the adapter layer, and `trace.warnings[]` +records every stream it cannot supply (R3.5). Ceilings are enforced by the adapter +(wall-clock kill + iteration cap), and the *absence* of internal ceilings is reported as an +expected FM-007 positive rather than patched (R3.4). + +**AblatedArm** is `AiTeamArm` with a component set subtracted. Ablation is config, so the +implementation is a settings override object threaded into run construction — never a +code path fork (R4.2). + +### 4.2 `src/ai_team/harness/acceptance.py` + +Sole writer of `ACCEPTANCE.json`. + +```python +def write_initial(workspace: Path, requirements: RequirementsDoc, run_id: str) -> AcceptanceList +def load(workspace: Path) -> AcceptanceList +def mark_passing(workspace: Path, item_id: str, *, evidence: list[str], + verified_by: str, identity: VerifierIdentity) -> AcceptanceList +def demote(workspace: Path, item_id: str, *, session_id: str, reason: str) -> AcceptanceList +def status(workspace: Path) -> AcceptanceStatus # counts + next unsatisfied item +``` + +`mark_passing` validates that every evidence path exists in the workspace before writing +(R6.4). Writes are atomic (temp file + `os.replace`) and append a snapshot hash to +`logs/acceptance.jsonl`, which is what gives the monotonic check a first-and-last state +to compare without needing git. + +Enforcement is two-layer: the PreToolUse hook denies `Write`/`Edit` on the path, and +`mark_passing` is exposed to agents only through the MCP tool surface. + +### 4.3 `src/ai_team/harness/session_loop.py` + +```python +def run_sessions(workspace, *, backend, max_sessions, total_budget_usd, + no_progress_sessions=2, regression_k=2) -> list[SessionRecord] +``` + +Per session: build fresh context from the four files → run regression check (R9) → select +target item → negotiate contract (R11, if enabled) → implement → verify → `mark_passing` +→ commit → write `SessionRecord`. + +Termination is checked *before* constructing the next session, and every exit path writes a +record. A session that ends with uncommitted changes is `dirty_exit` and feeds FM-012. + +The loop does not summarize. The next session's context is `CONSTRAINTS.md` + `STATE.md` +(last-N phase facts) + `LESSONS.md` + `ACCEPTANCE.json` + `git log` — all already produced +by `harness/context.py` and the receipt writer. + +### 4.4 `src/ai_team/harness/contracts.py` + +Reads and writes `docs/contracts/.json`, enforces the round ceiling (R11.3), and +exposes `require_accepted(item_id) -> bool` to the PreToolUse hook so writes to `src/` can +be denied for un-contracted items (R11.4). When the component is ablated, the hook check +is a no-op and the negotiation step is skipped entirely. + +### 4.5 `src/ai_team/tools/ui_smoke_tools.py` + +Mirrors `smoke_tools.py`'s shape deliberately — same result-file contract, same +"never probe a foreign service" rule, same `load_or_run` caching by age. + +```python +class UiStepResult(BaseModel): + action: str; selector: str | None; outcome: Literal["pass","fail","skipped"] + console_errors: list[str]; page_errors: list[str] + screenshot_path: str | None; duration_ms: int + +class UiSmokeResult(BaseModel): + status: Literal["pass","fail","skipped"]; steps: list[UiStepResult] + started_at: datetime; duration_s: float; base_url: str | None; skip_reason: str | None +``` + +Playwright, headless, Chromium, fixed viewport for screenshot comparability. Steps come +from the acceptance item's `steps[]` where they are machine-interpretable, and from the +scenario's `ui` block otherwise. Console and page errors fail the step deterministically — +no judge involved (R13.3). + +Scenario contracts gain an optional `ui` block (`base_url`, `boot_cmd`, `ready_path`, +`viewport`); its absence means `skipped`, never `pass` (R13.6). + +### 4.5a QA verdict store + +`docs/qa_verdicts.jsonl`, one record per item per QA pass: + +```python +class QaVerdict(BaseModel): + item_id: str + verdict: Literal["accept", "reject"] + issues: list[QaIssue] # {description, severity, evidence[]} + evidence: list[str] + qa_prompt_hash: str # R16.6 — attribute rate changes to the prompt + identity: VerifierIdentity + emitted_at: datetime +``` + +Structured output rather than prose is what makes FM-017 checkable at all: the check needs +to see that the evaluator *recorded* a blocker and then *accepted anyway*. Parsing that +intent out of free text would be a judge; reading it out of a field is a comparison. + +The false-negative rate (R16.4) needs no labels: every item the QA agent accepted and a +deterministic verifier later failed is a counted miss. The deterministic verifier is the +ground truth, which is the cheapest ground truth this project has access to. + +### 4.6 Checks + +| Check | Module | FM | Evidence | +| --- | --- | --- | --- | +| `CHK-acceptance-monotonic` | `evals/checks/acceptance.py` | FM-014 | `logs/acceptance.jsonl` first/last snapshots + artifact inventory | +| `CHK-premature-termination` | `evals/checks/context.py` (extend) | FM-015 | `phase_end` / `session_end` spans, trailing span window, acceptance state | +| `CHK-verifier-independence` | `evals/checks/verification.py` (extend) | FM-016 | `verifier_identity` on passes transitions, write spans per item | +| `CHK-evaluator-capitulation` | `evals/checks/verification.py` (extend) | FM-017 | `docs/qa_verdicts.jsonl` verdicts vs their own `issues[]`, remediation spans between | +| `CHK-runtime-smoke-present` | `evals/checks/verification.py` | FM-006 | extended to require UI smoke for UI-bearing scenarios | + +All four are pure functions over a Trace, registered through the existing +`evals/checks/registry.py`, and carry fail/pass fixtures under `evals/fixtures/traces/`. + +### 4.7 Reporting + +`evals/report.py` gains `render_ladder(traces, *, allow_mixed_model, allow_underpowered)`: + +- Groups by `arm_id`; computes per-arm `n`, mean and CI for cost and wall-clock, FM + incidence per mode, smoke pass rate, accepted-change rate, demotion rate. +- Emits JSON first, renders Markdown *from that JSON* — never computes twice (FM-008). +- Applies the stamps of R5.2 / R5.3 before any table is rendered, and refuses outright + without the corresponding flag. +- Ablation deltas are rendered against the `ai_team` control with CIs, and components with + no measured effect get their own named section (R4.5). +- **Per-phase cost and wall-clock** per arm, from `logs/costs.jsonl` (R5.1a), so a + component's price sits beside what it buys. The optional `cost_per_fm_avoided` is a ratio + of two noisy estimates and is stamped and suppressed accordingly (R5.1b). +- **Staleness** (R15): every ablation row carries the model it was measured on, and is + marked `STALE` when that differs from the configured default. `never measured` renders as + its own state, never as a zero delta. +- **`qa_false_negative_rate`** per arm (R16.4), computed for free against the deterministic + verifiers, beside `smoke_pass_rate` — because an arm can buy a pass rate by lowering its + evaluator, and the table should make that visible rather than reward it. + +--- + +## 5. Sequencing + +The phases are ordered so that the cheapest, highest-signal work lands first and nothing +expensive runs before the free scoring path can consume it. + +``` +Phase 1 acceptance list + FM-014 free, no arms, immediately useful +Phase 2 FM-015/016/017 checks free, retroactive over existing corpus +Phase 3 arms scaffold + ai_team arm no new spend (wraps existing runs) +Phase 4 solo + harnessed_solo arms first spend; the two-question ladder +Phase 5 reference arm vendoring + adapter +Phase 6 ladder report + claim discipline free; makes phases 4–5 publishable +Phase 7 UI smoke + FM-006 extension medium cost, unlocks style criteria +Phase 8 session loop + regression check largest change; consumes phases 1–2 +Phase 9 contract negotiation hypothesis; ablatable from the start +Phase 10 bash allowlist last, behind the FP budget +Phase 11 ablation sweep + writeup the payoff +``` + +Phases 1–2 are strictly additive and can land before any decision about arms is made. +Phases 8–9 are the only ones that change agent behavior in a normal run, and both are off +by default. + +--- + +## 6. Error handling + +| Failure | Behavior | +| --- | --- | +| Arm exceeds ceiling | Terminate, `status = "budget_exhausted"`, Trace still emitted (R1.5) | +| Vendored harness unavailable | `status = "unavailable"`, sweep continues, report shows the arm as absent — never substituted (R3.6) | +| `ACCEPTANCE.json` corrupt or missing mid-run | Session aborts with `error`; the loop does not regenerate it (R8.4), because regeneration is indistinguishable from goalpost-moving | +| Evidence path missing on `mark_passing` | Reject the write, return an error to the agent, log a `guardrail_check` span | +| Contract rounds exhausted | Escalate through the existing `flows/error_handling.py` path; never loop (R11.3, FM-002) | +| Playwright unavailable or browser launch fails | `UiSmokeResult.status = "skipped"` with `skip_reason`; FM-006 check treats skip as unsatisfied for UI scenarios, so it fails loudly rather than passing quietly | +| Token usage unavailable for `context_pressure` | Field is `None`; `CHK-premature-termination` returns `inconclusive`, not `pass` | +| Bash parse failure | Block (R14.3) | + +The recurring rule: **absence of evidence is never recorded as evidence of success.** Skips, +inconclusives, and unavailables are distinct outcomes from passes, and the aggregation +keeps them out of pass-rate numerators *and* denominators. + +--- + +## 7. Testing strategy + +**Unit** (`tests/unit/harness/`, `tests/unit/evals/arms/`) + +- `acceptance.py`: monotonic enforcement, atomic write, evidence validation, demotion + records, content-hash stability across whitespace changes. +- Bash allowlist parser: the full adversarial list of R14.4, one test per construct, each + asserting *block* for anything undecomposable. +- `ArmSpec` / `Divergence` round-trips; registry duplicate guard. +- Ladder report: stamp application, refusal without flags, Markdown-rendered-from-JSON + identity. + +**Check tests** (`tests/unit/evals/checks/`) + +- Each new check: the fail fixture fails, the pass fixture passes, a trace missing the + required span type returns `inconclusive`. +- `CHK-premature-termination` negative controls: spend event present, watchdog kill, + `max_turns` exhaustion — each must *not* fire (R10.4). + +**Integration** (`tests/integration/`) + +- Session loop over a stubbed backend: three sessions, one injected regression, assert + demotion → prioritization → re-pass, and assert `sessions.jsonl` shape. +- Contract negotiation with a stubbed QA returning `rejected` twice, asserting escalation + at round 3 rather than a fourth round. +- UI smoke against a fixture static app with a deliberate console error, asserting step + failure without any model call. + +**Suite-level guards** (`tests/conftest.py`, `tests/unit/evals/test_suite_drift_guards.py`) + +- Tracked ground truth (`evals/golden/`, `evals/fixtures/traces/`, `evals/taxonomy/`) is + hashed at session start and asserted unchanged at teardown (R17.1). This is not + belt-and-braces: it caught a live defect the day it was written. +- `ALL_CHECK_IDS` ≡ registry, no duplicates, fixtures present for every check, no check + pointing at an unknown failure mode, no implemented mode left `reserved` (R17.2, R17.3). +- The four checks this spec adds are covered by those guards automatically — which is the + point of writing them before Phase 2 rather than after. + +**Cover before extend.** Tasks 1.4, 1.5, 7.3, 9.3 and 10.2 attach to modules whose deny +paths are largely untested (`mcp_server.py` at 0% branch being the worst). Each of those +tasks carries the existing module's tests as part of its definition of done (R17.5), and +every new refusal path reaches 100% branch coverage before its component is enabled by +default (R17.6). `hooks/security.py` — 100%/100% as of 2026-09-12 — is the reference. + +**Never in pytest:** live arms, live model calls, live judge quality assertions. Arm runs +are Tier C, human-triggered, and budgeted. + +--- + +## 8. Claim discipline + +What this spec licenses the project to say, once the ladder has n ≥ 3 per arm at a fixed +model: + +- "Removing `` changed FM-`` incidence by X ± CI at n=N on scenario S." +- "A solo agent on the same brief and model produced W at cost C; the full harness + produced W' at cost C'." +- "The published reference harness exhibits FM-007 and FM-006 on this scenario; here are + the traces." + +What it still does not license: + +- "Framework X beats framework Y." Unchanged, and out of scope. +- "This harness generalizes." One scenario family, one model, one author. +- Any design-quality claim from the R13.7 rubric until it clears the alignment bar. + +The `UNDERPOWERED` / `MIXED-MODEL` stamps exist so that a table lifted out of context +carries its own caveat. That is the whole point of putting them in the renderer rather +than the README. + +### 8.1 Effect on claims already published + +Three of this spec's four new modes are `layer: model`. The taxonomy essay's central +statistic — "only one of the ten was caused by an LLM being dumb" — becomes four of +seventeen, and that sentence is load-bearing for the project's whole "it was systems +engineering, not the model" framing. Two honest readings, and the second is the right one: + +1. The thesis weakened. More failures turned out to be the model's after all. +2. The instruments improved. FM-014, FM-015 and FM-017 describe failures that were always + occurring and were previously undetectable — goalposts moved without a diff to catch it, + sessions wrapped up early with no measure of context pressure, evaluators waived their + own findings in prose nobody parsed. The denominator grew because detection grew. + +Reading 2 is supportable and reading 1 is not, because the new modes were found by building +detectors, not by observing a regression. But the essay has to say so explicitly, and the +ratio has to be computed from the YAML rather than typed, or the next three modes will +silently falsify it again (task 11.5). + +--- + +## 9. Open decisions + +1. **Scenario for the first ladder sweep.** `todo-api-beginner` is cheapest and has a UI + story; `architecture-planning-intermediate` better exercises the contract mechanism but + has no runnable artifact. Recommendation: `todo-api-beginner` first. +2. **Whether `reference` counts toward FM incidence rates in the headline table** or is + reported in a separate external-reference block. Recommendation: separate block — it is + a different harness, not a variant of yours. +3. **Whether `context_pressure` can be sourced reliably per-phase** from the SDK across all + three backends, or only from the Claude Agent SDK backend. If only the latter, FM-015 + is scoped to that backend in v1 and the check returns `inconclusive` elsewhere. +4. **Brief fidelity.** The reference article's planner expands a 1–4 sentence brief, and + deliberately withholds granular technical detail so that a wrong guess cannot cascade + downstream. `ai-team`'s scenario contracts are structured and detailed, so the thin-brief + path — the one most real users take — is never exercised, and the planner's + over-specification risk is never measured. Options: (a) leave it, recording detailed + contracts as a known limitation in every arm's divergence list; (b) add a `thin` variant + per scenario carrying only the brief, as a second sweep dimension. Recommendation: (a) + for v1, (b) once the ladder has produced its first result — it doubles sweep cost and the + ladder should prove itself first. +5. **Whether `harnessed_solo` or the `smoke_gate` ablation is the better second live arm** + if budget allows only one beyond `solo`. `harnessed_solo` answers the bigger question + (is the nine-role team earning its cost?) and is the more uncomfortable one to ask, which + is usually the sign it should be asked first. diff --git a/.kiro/specs/harness-alignment/requirements.md b/.kiro/specs/harness-alignment/requirements.md new file mode 100644 index 00000000..a8633e49 --- /dev/null +++ b/.kiro/specs/harness-alignment/requirements.md @@ -0,0 +1,740 @@ +# Requirements — Harness Alignment + +**Spec ID:** `harness-alignment` +**Status:** Draft for implementation +**Owner:** Rick Zakharov +**Target repo:** `ai-team` (extends `evals/` and `src/ai_team/harness/`) +**Created:** 2026-09-12 +**Depends on:** [`../eval-harness/`](../eval-harness/) — Trace boundary (R1), check registry +(R4), taxonomy (R5), tiers and budget (R11), provenance (R14). This spec adds arms, +ablation, three failure modes, and four harness components. It does not restate those. + +--- + +## Introduction + +`ai-team` compares three orchestration backends behind one `Backend` protocol and +scores them through a Trace boundary against a thirteen-mode failure taxonomy +(FM-001…FM-013). The README already concedes the central weakness: with n=5 and mixed +models, the published tables "cannot yet support framework X beats Y." + +That weakness is not fixable by running more samples. It is a **missing-arm** problem. +The project measures three points that differ in framework *and* in nothing else it +controls, and has no point of reference outside itself. Two external sources published +by Anthropic supply exactly the reference points that are missing: + +1. **`anthropics/claude-quickstarts/autonomous-coding`** (MIT, @3313e97) — a 2,300-line + minimal harness: two-agent loop (initializer + coding agent), one fresh context window + per session, all state on disk (`feature_list.json`, `claude-progress.txt`, git), a + default-deny bash allowlist, and browser-based verification. It has no spend guard, no + subprocess isolation, and no runtime gate. It is the *unhardened control* for the + harness layer. +2. **"Harness Design for Long-Running Application Development"** (Rajasekaran, Anthropic + Engineering) — a generator/evaluator architecture reporting a solo-agent control arm + (20 min, $9, core feature broken) against a full harness (6 h, $200, working), plus + three mechanisms `ai-team` does not have: **contract negotiation before + implementation**, **context resets with structured handoff instead of compaction**, and + the observation that models exhibit **"context anxiety"** — premature wrap-up as the + window fills. + +Aligning with these yields a **ladder** rather than a leaderboard: + +``` + solo agent → reference harness → ai-team (ablated) → ai-team (full) + no harness minimal harness component removed all components +``` + +Every rung runs the same scenario contract, emits the same `Trace`, and is scored by the +same deterministic checks. That is a claim the current n=5 matrix cannot make and this +one can: **not "which framework is better" but "what does each harness component buy, +and at what cost."** It answers the question the README actually asks — which failures +are the framework's, which the model's, and which are mine. + +### What this spec adds + +| Area | Gap today | Requirement | +| --- | --- | --- | +| Comparison arms | Only three same-shape backends | R1, R2, R3 | +| Component attribution | No way to isolate a component's contribution | R4 | +| Claim discipline | Prose caveats, not enforced | R5 | +| Durable acceptance criteria | Run-scoped objects in `models/requirements.py`; nothing survives a session | R6 | +| Goalpost-moving detection | FM-011 covers constraint drop, not criteria mutation | R7 (FM-014) | +| Multi-session builds | One pipeline pass per run; no resume | R8, R9 | +| Premature termination | Not in the taxonomy; `token_tracker` counts cost, not pressure | R10 (FM-015) | +| Contract before code | Criteria flow one way from planning; QA never validates the plan | R11 | +| Self-graded verification | Agent identities are separate by construction but nothing asserts it | R12 (FM-016) | +| UI-level verification | `run_app_smoke` proves HTTP 200; FM-006 is `runtime_verification_gap` by name | R13 | +| Bash confinement | Denylist of patterns; parse failure is not fail-safe | R14 | +| Component staleness | Nothing records which model a component's value was measured against | R15 | +| Role decomposition | Nine roles vs one agent is never measured — `solo` also removes the harness | R2, R4.1 | +| In-run QA quality | Judge alignment is offline; the agent that gates a build is unmeasured | R16 (FM-017) | +| Test-suite trust | Two isolation defects and an unguarded parametrize list, all green | R17 | + +### Design constraints (decided) + +| Constraint | Decision | +| --- | --- | +| Location | Extends `evals/` and `src/ai_team/harness/` in-repo. No new repo, no fourth first-class backend. | +| Reference harness | Adapter over a **pinned, vendored** copy. Never edited to make it look worse; never promoted to a `Backend` implementation. | +| Tier discipline | Every new check is deterministic and runs in **Tier A at $0.00**. Arms and ablation runs are Tier C only. | +| Arm budget | ≤ **$25.00** per full ladder sweep (one scenario × four rungs), separate from and additive to the eval-harness $5 suite budget. Default `--dry-run`. | +| Backwards compatibility | `run_evals.py --compare`, the three `evals/backends/test_*_eval.py` files, and existing FM-001…013 checks must keep working unchanged. | +| Claims | No arm comparison may be published without n, a cost column, and the model id per arm. Enforced in code (R5), not in prose. | + +### Non-goals + +- Ranking CrewAI vs LangGraph vs Claude Agent SDK. Out of scope here as it is there. +- Vendoring the quickstart's 200-feature initializer prompt, its unbounded iteration + loop (`max_turns=1000`, no ceiling), or its `.claude_settings.json` generation as the + primary permission mechanism. The first is pure spend; the second is a live specimen of + FM-007 and is *kept* as such, not adopted. +- A design-quality LLM judge that gates a build. R13 defines the rubric and the artifacts; + gating stays behind the eval-harness alignment bar (TPR ≥ 0.90 / TNR ≥ 0.90) and is not + claimed by this spec. +- Replacing `harness/context.py`. `CONSTRAINTS.md` / `STATE.md` / `LESSONS.md` is already + the structured-handoff mechanism the article argues for; R8 consumes it, it does not + replace it. +- Multi-annotator labeling, online evaluation, fine-tuning. As in `eval-harness`. + +### Glossary + +| Term | Definition | +| --- | --- | +| **Arm** | One configuration under comparison that is *not* a backend swap: `solo`, `reference`, `ai_team`, or `ai_team_ablated:`. | +| **Ladder** | The ordered set of arms for one scenario, run under one sweep id. | +| **Ablation** | An `ai_team` run with one named harness component disabled by config, holding model, scenario, and seed fixed. | +| **Acceptance list** | `ACCEPTANCE.json` in the run workspace: the durable, monotonic checklist of acceptance criteria. | +| **Monotonic** | Items may be added by the harness at plan time and thereafter only the `passes` field may change, and only `false → true`, except for harness-issued demotions (R9). | +| **Session** | One fresh-context execution over a workspace. A run may span many sessions. | +| **Build contract** | A file-based agreement between developer and QA on what "done" means for one acceptance item, written before implementation begins. | +| **Context pressure** | Fraction of the usable context window consumed at the moment a phase or session ends. | +| **Harnessed solo** | The full `ai-team` harness driving a single generalist agent instead of nine roles — the rung that separates *harness* from *decomposition*. Equivalent to `ai_team_ablated:role_decomposition`. | +| **Stale ablation** | A component whose most recent measured delta was taken against a model that is no longer the configured default. | + +--- + +## Requirements + +### R1 — Arm abstraction and registry + +**User story:** As an eval engineer, I want every rung of the ladder to be addressable +through one interface, so that a solo agent, a third-party harness, and `ai-team` can be +scored by exactly the same checks. + +**Acceptance criteria** + +1.1 THE SYSTEM SHALL define `evals/arms/base.py::Arm`, a protocol with +`arm_id: str`, `describe() -> ArmSpec`, and +`run(scenario: ScenarioContract, workspace: Path, budget_usd: float) -> ArmRun`. + +1.2 THE SYSTEM SHALL implement arms as adapters that terminate at the existing Trace +boundary: every `Arm.run` SHALL produce a workspace from which +`evals.trace.from_workspace()` yields a valid `Trace`, and SHALL NOT require changes to +`evals/checks/*`. + +1.3 An `ArmSpec` SHALL record `arm_id`, `family` (`solo` | `reference` | `ai_team`), +`model_ids` (per role, or one entry for single-agent arms), `harness_components` (the +set of components active), `source_ref` (git sha or vendored pin for third-party code), +and `cost_controls` (`spend_ceiling_usd`, `wall_clock_ceiling_s`, `max_sessions`). + +1.4 THE SYSTEM SHALL register arms in `evals/arms/registry.py` by string id and SHALL +raise on duplicate registration, mirroring `evals/checks/registry.py`. + +1.5 WHERE an arm exceeds `spend_ceiling_usd` or `wall_clock_ceiling_s`, THE SYSTEM SHALL +terminate it, mark `ArmRun.status = "budget_exhausted"`, and still emit a Trace. + +1.6 THE SYSTEM SHALL hold model id, scenario contract, and workspace seed fixed across +all arms in one sweep, and SHALL record any forced divergence in +`ArmRun.divergences[]` with a reason string. + +--- + +### R2 — Solo-agent control arm + +**User story:** As an engineer defending the claim that the harness is the deliverable, I +want a no-harness control, so that "the harness bought us X" is a measurement rather than +an assertion. + +**Acceptance criteria** + +2.1 THE SYSTEM SHALL provide `evals/arms/solo.py::SoloArm`: one Claude Agent SDK session, +one general-purpose system prompt, the scenario brief as its only input, no role +decomposition, no guardrails, no smoke gate, no lessons, no constraint pinning. + +2.2 THE SOLO ARM SHALL write the same workspace layout the rest of the repo assumes +(`docs/`, `src/`, `tests/`, `logs/`) so that Trace assembly needs no special case. + +2.3 THE SOLO ARM SHALL be subject to `spend_ceiling_usd` and `wall_clock_ceiling_s` from +its `ArmSpec` and SHALL default to `$3.00` and `1800` seconds. + +2.4 THE SYSTEM SHALL run the identical FM-001…FM-016 check set over the solo arm's Trace +and SHALL NOT suppress or special-case any check for it. + +2.5 THE SYSTEM SHALL record, for the solo arm, the same receipt fields as an `ai_team` +run (`cost_usd`, `wall_clock_s`, `accepted_change`, `workspace_tree_hash`) via +`src/ai_team/harness/receipt.py`. + +--- + +### R3 — Reference-harness arm + +**User story:** As an engineer, I want Anthropic's own minimal autonomous-coding harness +scored by my taxonomy, so that my claims about the harness layer are tested against a +published reference rather than only against my own earlier code. + +**Acceptance criteria** + +3.1 THE SYSTEM SHALL vendor the reference harness under +`evals/arms/vendor/autonomous_coding/` with `PROVENANCE.md` recording upstream URL, +commit sha, licence (MIT), and retrieval date. + +3.2 THE SYSTEM SHALL NOT modify vendored source. WHERE adaptation is required (spend +ceiling, iteration ceiling, scenario injection, log capture), THE SYSTEM SHALL implement +it in `evals/arms/reference.py` as wrapping, monkeypatch-free configuration, and SHALL +list every adaptation in `ArmSpec.divergences[]`. + +3.3 THE REFERENCE ARM SHALL translate a `ScenarioContract` into the harness's +`app_spec.txt` input and SHALL cap generated features at a configurable +`max_features` (default `25`, upstream default `200`) recorded as a divergence. + +3.4 THE REFERENCE ARM SHALL impose the ceilings of R1.5 externally, since the vendored +harness has none, and SHALL record that absence as an expected FM-007 positive rather +than a defect of the adapter. + +3.5 THE SYSTEM SHALL assemble a Trace from the reference arm's artifacts +(`feature_list.json`, `claude-progress.txt`, git history, captured stdout tool events) +and SHALL record coverage gaps in `trace.warnings[]` where the reference harness emits no +equivalent of an `ai-team` log stream, rather than fabricating spans. + +3.6 IF the vendored harness cannot run in the current environment (missing +`@anthropic-ai/claude-code`, missing node, blocked network), THEN THE SYSTEM SHALL fail +the arm with `status = "unavailable"` and SHALL NOT silently substitute another arm. + +--- + +### R4 — Ablation runner + +**User story:** As the author of the harness, I want to disable one component at a time +and measure what changes, so that "which failures are mine?" has a numeric answer per +component. + +**Acceptance criteria** + +4.1 THE SYSTEM SHALL define an `AblationSpec` naming one or more ablatable components +from a closed set: `guardrails.behavioral`, `guardrails.security`, `guardrails.quality`, +`smoke_gate`, `constraints_pinning`, `lessons_loop`, `spend_guard`, +`subprocess_isolation`, `contract_negotiation`, `session_regression_check`, +`role_decomposition`. + +4.2 THE SYSTEM SHALL implement ablation as configuration read at run construction — no +code deletion, no branch — and SHALL fail loudly if a named component cannot be disabled +by config. + +4.3 THE SYSTEM SHALL record the active component set in `ArmSpec.harness_components` and +in the Trace's provenance, so a Trace is self-describing without its sweep manifest. + +4.4 FOR each ablation arm, THE SYSTEM SHALL report the delta against the full `ai_team` +arm on: FM incidence per mode, `cost_usd`, `wall_clock_s`, accepted-change rate, and +smoke outcome. + +4.5 WHERE an ablation produces no measurable delta across a sweep, THE SYSTEM SHALL +surface that component in the report under a heading that names it plainly +(`components with no measured effect at this n`), because a component that earns nothing +is the finding. + +4.6 THE SYSTEM SHALL NOT treat a single-sweep ablation delta as significant; the report +SHALL carry `n` and a confidence interval for every delta, per eval-harness design +principle 2. + +--- + +### R5 — Arm-aware reporting and enforced claim discipline + +**User story:** As someone who publishes these numbers, I want the tooling to refuse to +emit an unfalsifiable comparison, so that the honesty of the results does not depend on +my remembering to add a caveat. + +**Acceptance criteria** + +5.1 THE SYSTEM SHALL extend the existing report with a **ladder table**: one row per arm, +columns `arm_id`, `n`, `cost_usd` (mean, CI), `wall_clock_s` (mean, CI), +`fm_incidence` per mode, `smoke_pass_rate`, `accepted_change_rate`, `demotion_rate`. + +5.1a THE REPORT SHALL additionally break cost and wall-clock down **by phase** per arm +(planning, build, verification, QA), sourced from `logs/costs.jsonl`, so that a +component's price is legible next to what it buys rather than only in the run total. + +5.1b WHERE both an ablation delta and a phase cost share are available for a component, +THE REPORT MAY render a derived `cost_per_fm_avoided`, and SHALL mark it +`DERIVED — RATIO OF TWO NOISY ESTIMATES` and suppress it entirely at `n < 5` per arm. + +5.2 THE SYSTEM SHALL refuse to render a cross-arm comparison WHEN model ids differ across +arms without an explicit `--allow-mixed-model` flag, and SHALL stamp any such table +`MIXED-MODEL — NOT A HARNESS COMPARISON`. + +5.3 THE SYSTEM SHALL refuse to render a cross-arm comparison at `n < 3` per arm without +`--allow-underpowered`, and SHALL stamp such tables `UNDERPOWERED`. + +5.4 THE REPORT SHALL state, for every arm, the `source_ref` and the divergence list, so a +reader can see exactly how the reference harness was configured. + +5.5 THE SYSTEM SHALL emit the ladder table as both Markdown (for `docs/`) and JSON (for +regression diffing), and the JSON SHALL be the source the Markdown is rendered from, per +FM-008 `metric_source_drift`. + +--- + +### R6 — Durable, monotonic acceptance list + +**User story:** As a harness author, I want acceptance criteria to survive a context +reset as a file the harness owns, so that progress is resumable and goalpost-moving is +detectable from a diff. + +**Acceptance criteria** + +6.1 THE SYSTEM SHALL write `ACCEPTANCE.json` to the run workspace root at the end of the +planning phase, derived from the existing acceptance criteria in +`src/ai_team/models/requirements.py`, so no new authoring step is introduced. + +6.2 Each item SHALL contain `id` (stable, content-hashed), `category` +(`functional` | `style` | `security` | `performance`), `description`, `steps[]`, +`passes: bool`, `verified_by` (`null` | `smoke` | `ui_smoke` | `test` | `qa_agent`), +`verified_at`, `evidence[]` (artifact paths), and `demotions[]`. + +6.3 THE SYSTEM SHALL treat `ACCEPTANCE.json` as **harness-owned**: it SHALL be written +through `src/ai_team/harness/acceptance.py` only, and the SDK backend's PreToolUse hook +SHALL deny direct `Write`/`Edit` to that path by any agent. + +6.4 THE SYSTEM SHALL permit exactly one agent-driven mutation: `passes: false → true`, +and only when accompanied by at least one `evidence[]` entry that resolves to a file in +the workspace. + +6.5 THE SYSTEM SHALL permit harness-issued demotion (`true → false`) only from the R9 +regression check, appending a `demotions[]` record with `session_id`, `reason`, and +`detected_at`. + +6.6 THE SYSTEM SHALL expose acceptance state to agents read-only through an MCP tool +(`acceptance_status`) returning counts and the highest-priority unsatisfied item, so +agents never need to parse the file to orient. + +6.7 THE ACCEPTANCE LIST SHALL be included in the change receipt and in the Trace's +artifact inventory. + +--- + +### R7 — FM-014 `acceptance_criteria_mutation` + +**User story:** As an eval engineer, I want an agent that edits the definition of done to +be caught by a free check, so that a run cannot pass by lowering the bar. + +**Acceptance criteria** + +7.1 THE SYSTEM SHALL add `FM-014 acceptance_criteria_mutation` to +`evals/taxonomy/failure_modes.yaml` with `layer: model`, `harness_layer: tools`, +`severity: blocker`, `detection: check`, `introduced_in: "1.2.0"`. + +7.2 THE SYSTEM SHALL implement `CHK-acceptance-monotonic` over a Trace: comparing the +first and last observed states of `ACCEPTANCE.json`, THE CHECK SHALL fail on any removed +item, any changed `id`/`description`/`steps`, any reordering that changes item identity, +or any `true → false` transition lacking a `demotions[]` record. + +7.3 THE CHECK SHALL fail on any `false → true` transition whose `evidence[]` is empty or +names a path absent from the workspace inventory. + +7.4 THE SYSTEM SHALL ship one failing and one passing trace fixture under +`evals/fixtures/traces/`, named per the existing convention +(`CHK-acceptance-monotonic__fail__fixture`, `…__pass__fixture`). + +7.5 THE CHECK SHALL run in Tier A at $0.00 and SHALL require no model call. + +--- + +### R8 — Multi-session continuation + +**User story:** As an operator, I want a run to continue across many fresh-context +sessions against the durable checklist, so that a build larger than one context window is +possible and cross-session regression becomes observable. + +**Acceptance criteria** + +8.1 THE SYSTEM SHALL provide `src/ai_team/harness/session_loop.py` driving N sessions over +one workspace, where each session begins with a **fresh context** — not a compaction — +seeded only by `CONSTRAINTS.md`, `STATE.md`, `LESSONS.md`, `ACCEPTANCE.json`, and git +history. + +8.2 THE LOOP SHALL terminate on any of: all acceptance items `passes: true`; +`max_sessions` reached; cumulative spend ceiling reached; wall-clock ceiling reached; or +`no_progress_sessions` consecutive sessions with zero net `passes` transitions +(default `2`). + +8.3 THE LOOP SHALL write one `SessionRecord` per session to `logs/sessions.jsonl`: +`session_id`, `index`, `started_at`, `ended_at`, `status`, `items_attempted[]`, +`items_passed[]`, `items_demoted[]`, `cost_usd`, `context_pressure_at_end`, +`termination_reason`. + +8.4 THE LOOP SHALL be resumable: invoking it on an existing workspace SHALL continue from +the persisted state without re-running planning, and SHALL NOT regenerate +`ACCEPTANCE.json`. + +8.5 THE LOOP SHALL require a git commit at session end and SHALL mark a session +`status = "dirty_exit"` when the workspace has uncommitted changes, feeding FM-012 +`uncommitted_write`. + +8.6 THE LOOP SHALL be off by default; enabling it SHALL require an explicit +`--max-sessions N` and an explicit total budget. + +--- + +### R9 — Session-start regression verification + +**User story:** As an operator of a long build, I want each session to prove that +previously passing work still passes before it writes anything new, so that silent +cross-session regression is caught at the session that caused it. + +**Acceptance criteria** + +9.1 AT the start of every session after the first, THE SYSTEM SHALL re-verify `k` +previously passing acceptance items (default `k = 2`) selected by a deterministic policy: +the highest-priority passing item plus one sampled by a seeded RNG over items not +verified in the last `m` sessions. + +9.2 WHERE re-verification fails, THE SYSTEM SHALL demote the item per R6.5 and SHALL +require the session to address demoted items **before** attempting any unsatisfied item. + +9.3 THE SYSTEM SHALL emit a `regression_check` span per verified item into the Trace, with +`item_id`, `outcome`, `evidence[]`, and `duration_s`. + +9.4 THE SYSTEM SHALL surface `demotion_rate` (demotions per session) in the ladder report +as a first-class reliability metric, since it measures exactly what single-pass runs +cannot see. + +--- + +### R10 — FM-015 `premature_termination_under_context_pressure` + +**User story:** As an eval engineer, I want "context anxiety" — an agent wrapping up early +as its window fills — recorded as a named, detected failure mode rather than folklore. + +**Acceptance criteria** + +10.1 THE SYSTEM SHALL add `FM-015 premature_termination_under_context_pressure` to the +taxonomy with `layer: model`, `harness_layer: context`, `severity: major`, +`detection: check`, `introduced_in: "1.2.0"`. + +10.2 THE SYSTEM SHALL record `context_pressure` — consumed tokens over usable window — on +every `phase_end` and `session_end` span, sourced from the SDK's usage reporting and +falling back to `src/ai_team/config/token_tracker.py`. + +10.3 THE SYSTEM SHALL implement `CHK-premature-termination`, failing WHEN a phase or +session ends with `status = "ok"`, at least one unsatisfied acceptance item remains, no +blocking error or budget event is present in the trailing span window, AND +`context_pressure ≥ 0.75` (configurable). + +10.4 THE CHECK SHALL NOT fire where termination is explained by a `spend_event`, a +watchdog kill, an `error` span, or `max_turns` exhaustion — those are FM-007 and +FM-003 territory and SHALL remain attributed there. + +10.5 THE SYSTEM SHALL ship fail/pass trace fixtures and SHALL run in Tier A at $0.00. + +10.6 THE SYSTEM SHALL report FM-015 incidence **by arm**, since the hypothesis under test +is that fresh-context sessions (R8) reduce it relative to a single long context. + +--- + +### R11 — Build contract negotiation + +**User story:** As a harness author, I want QA to agree on what "done" means for an item +*before* the developer implements it, so that specification drift is caught at plan time +rather than at review time. + +**Acceptance criteria** + +11.1 BEFORE implementation of an acceptance item, THE SYSTEM SHALL have the developer +agent write `docs/contracts/.json` containing `item_id`, `approach` (prose), +`testable_behaviors[]` (each a concrete, observable assertion), `files_to_touch[]`, and +`out_of_scope[]`. + +11.2 THE QA AGENT SHALL validate the contract against the acceptance item and SHALL +return `accepted` or `rejected` with `reasons[]`, written to the same file under +`review`, using file-based handoff per `CLAUDE.md` (agents coordinate through paths, not +shared memory). + +11.3 WHERE a contract is rejected, THE SYSTEM SHALL permit at most `max_contract_rounds` +(default `2`) renegotiations before escalating to the existing error-handling path, so +negotiation cannot become an unbounded loop (FM-002). + +11.4 IMPLEMENTATION SHALL NOT begin on an item whose contract is not `accepted`; the +PreToolUse hook SHALL deny writes to `src/` for an item lacking an accepted contract when +contract negotiation is enabled. + +11.5 VERIFICATION SHALL be scored against `testable_behaviors[]`, not against the +developer's own summary of what it built. + +11.6 CONTRACT NEGOTIATION SHALL be an ablatable component (R4.1), because its value is a +hypothesis this spec exists to test, not an assumption. + +--- + +### R12 — FM-016 `self_graded_verification` + +**User story:** As an eval engineer, I want a check that the thing which wrote the code is +not the thing which passed it, so that the generator/evaluator separation is enforced +rather than assumed. + +**Acceptance criteria** + +12.1 THE SYSTEM SHALL add `FM-016 self_graded_verification` to the taxonomy with +`layer: harness`, `harness_layer: verification`, `severity: major`, `detection: check`. + +12.2 THE SYSTEM SHALL record, on every acceptance `passes` transition, the +`agent_role`, `session_id`, and `subagent_id` responsible for the verifying evidence. + +12.3 THE SYSTEM SHALL implement `CHK-verifier-independence`, failing WHEN an item's +`passes: true` transition was produced by the same `(agent_role, subagent_id)` that +produced the writes for that item, and no independent verifier +(`smoke`, `ui_smoke`, `test`) contributed evidence. + +12.4 THE CHECK SHALL pass where a deterministic verifier supplied the evidence, since a +passing smoke probe is independent of who wrote the code. + +12.5 THE SYSTEM SHALL ship fail/pass fixtures and run in Tier A at $0.00. + +--- + +### R13 — UI-level verification + +**User story:** As a harness author, I want verification that drives the real UI and +reports console errors, so that "HTTP 200" stops counting as proof that a feature works. + +**Acceptance criteria** + +13.1 THE SYSTEM SHALL add `src/ai_team/tools/ui_smoke_tools.py::run_ui_smoke` using +**Playwright**, mirroring the contract of `run_app_smoke`: boot, act, probe, write +`docs/ui_smoke_results.json`, never probe a foreign service. + +13.2 `run_ui_smoke` SHALL emit, per step: `action`, `selector`, `outcome`, +`console_errors[]`, `page_errors[]`, `screenshot_path`, `duration_ms`. + +13.3 CONSOLE ERRORS SHALL be a **deterministic** signal: a step with any +`console_errors[]` or `page_errors[]` entry SHALL fail that step without a model call. + +13.4 SCREENSHOTS SHALL be written under `docs/verification//` and SHALL be listed +in the Trace artifact inventory so they survive as evidence (R6.4). + +13.5 THE SYSTEM SHALL expose `run_ui_smoke` as an MCP tool on the `ai_team_tools` server +and SHALL add it to the QA role's allow-list only. + +13.6 WHERE the scenario declares no UI (`scenario.ui == null`), `run_ui_smoke` SHALL skip +cleanly and SHALL record `skipped` rather than `pass`, so skips cannot inflate pass rates +(FM-008). + +13.7 THE SYSTEM SHALL define, in `docs/`, a four-criterion design rubric — design quality, +originality, craft, functionality — with calibration examples, for use by a QA judge. +THE RUBRIC SHALL carry an explicit warning that criterion wording steers output toward +convergence, and SHALL NOT gate a build until it clears the eval-harness alignment bar. + +13.8 `CHK-runtime-smoke-present` SHALL be extended to recognise a UI smoke result as +satisfying FM-006 for UI-bearing scenarios, and a backend-only probe SHALL NOT satisfy it +for those scenarios. + +--- + +### R14 — Default-deny bash confinement + +**User story:** As a harness author, I want bash to be default-deny with fail-safe +parsing, so that the security layer does not depend on having enumerated every dangerous +pattern in advance. + +**Acceptance criteria** + +14.1 THE SYSTEM SHALL add an allowlist layer to +`src/ai_team/backends/claude_agent_sdk_backend/hooks/security.py`, evaluated **after** the +existing deny patterns, such that a command is permitted only when every extracted command +name is in the role's allowlist. + +14.2 ALLOWLISTS SHALL be per-role, defined in `tools/permissions.py` beside the existing +tool allow-lists. + +14.3 WHERE the command string cannot be parsed, THE SYSTEM SHALL **block** and record the +reason, rather than falling through to permit. + +14.4 THE PARSER SHALL handle, and SHALL have adversarial tests for: `&&`, `||`, `;`, +pipes, subshells `$(…)` and backticks, `env VAR=x cmd`, `xargs`, `nohup`, `timeout`, +absolute paths, `git -c core.hooksPath=…` and `git -c core.pager=…`, and unclosed quotes. +Anything the parser cannot decompose SHALL be blocked (14.3). + +14.5 THE SYSTEM SHALL account for allowlist blocks in the guardrail false-positive corpus +and SHALL hold `CHK-guardrail-fp-budget` (FM-005) at or below its existing budget; a rise +in false positives SHALL fail the gate exactly as any other guardrail regression. + +14.6 THE ALLOWLIST SHALL be an ablatable component, and SHALL be enabled by default only +after 14.5 is demonstrated on the fixture corpus. + +14.7 THE SYSTEM SHALL NOT adopt the reference harness's practice of writing a settings +file into the generated workspace; filesystem confinement SHALL remain hook-enforced and +workspace-bound as it is today. + +--- + +### R15 — Ablation results expire with the model + +**User story:** As a harness author, I want every component's measured value stamped with +the model it was measured against, so that a harness does not silently accumulate +components that stopped earning their place two model versions ago. + +**Rationale:** every component in a harness encodes an assumption about what the model +cannot do on its own, and those assumptions go stale as models improve. The article this +spec aligns with removed its sprint-decomposition construct outright when a newer model +handled the job natively. A harness that cannot detect that about itself will keep paying +for it. + +**Acceptance criteria** + +15.1 THE SYSTEM SHALL record on every ablation result: `model_id`, `model_snapshot_date`, +`measured_at`, `scenario_id`, `n`, and the resulting delta with its confidence interval, +persisted to `evals/results/ablations/.json`. + +15.2 WHEN the repo's configured default model for a role differs from the `model_id` of +that component's most recent ablation result, THE SYSTEM SHALL mark the result `STALE` in +the report and SHALL NOT present its delta as current evidence. + +15.3 THE SYSTEM SHALL provide `python -m evals.cli ablation status`, listing every +ablatable component with its last measured delta, the model it was measured on, and +`current` | `STALE` | `never measured`. + +15.4 THE SYSTEM SHALL treat `never measured` as a first-class state and SHALL NOT render +it as a zero delta, because "we have never checked" and "it makes no difference" are +different claims. + +15.5 THE GATE SHALL NOT fail on staleness. Staleness is reporting, not enforcement — a +harness component is not a regression. + +--- + +### R16 — In-run evaluator calibration and FM-017 `evaluator_capitulation` + +**User story:** As a harness author, I want to know how often my QA agent finds a real +defect and then waives it, so that "QA passed" means something. + +**Rationale:** an out-of-the-box model used as QA identifies legitimate issues and then +talks itself into approving the work anyway, and tests superficially rather than at the +edges. This is distinct from FM-016: the evaluator here *is* independent, and still +returns the wrong verdict. It is also distinct from the offline judge alignment in +`eval-harness`, which scores traces after the fact; this is the in-run agent whose verdict +gates a build. + +**Acceptance criteria** + +16.1 THE SYSTEM SHALL add `FM-017 evaluator_capitulation` to the taxonomy with +`layer: model`, `harness_layer: verification`, `severity: major`, `detection: check`, +`introduced_in: "1.2.0"`. + +16.2 THE QA AGENT SHALL emit a structured verdict per acceptance item — +`{item_id, verdict, issues[], severity_per_issue, evidence[]}` — to +`docs/qa_verdicts.jsonl`, rather than prose the harness must parse. + +16.3 THE SYSTEM SHALL implement `CHK-evaluator-capitulation`, failing WHEN a verdict is +`accept` while `issues[]` contains at least one entry of severity `major` or `blocker`, +and no remediation write span for that item appears between the issue's detection and the +verdict. + +16.4 THE SYSTEM SHALL compute the QA agent's **false-negative rate against deterministic +verifiers**: for every item the QA agent accepted where `smoke`, `ui_smoke`, or the test +suite subsequently failed, count a QA false negative. THIS SHALL require no human labels +and no judge, since the deterministic verifier is the ground truth. + +16.5 THE REPORT SHALL surface `qa_false_negative_rate` per arm, and the ladder table SHALL +show it beside `smoke_pass_rate`, because an arm can buy its pass rate by lowering its +evaluator. + +16.6 WHERE the QA prompt is revised, THE SYSTEM SHALL record the prompt hash on every +verdict, so a change in false-negative rate can be attributed to the prompt rather than to +the model or the scenario. + +16.7 THE SYSTEM SHALL ship fail/pass fixtures for `CHK-evaluator-capitulation` and SHALL +run it in Tier A at $0.00. + +--- + +### R17 — Test hygiene and cover-before-extend + +**User story:** As the author of a harness whose whole claim is that it measures things +honestly, I want the test suite itself to be measurably trustworthy, so that a silent gap +in the instruments cannot masquerade as a green build. + +**Rationale:** a 2026-09-12 audit found two defects that **no amount of line coverage +would have caught**, both in the eval suite this spec extends: a unit test rewrote a +committed golden record's `validated_at` on every run +(`test_r16_unit_gaps.py` monkeypatched `VALIDATION_LOG` but not `ALIGNMENT_DIR`), and a +second test's assertion silently depended on the repo's `workspace/` directory being +empty. Both suites stayed green throughout. The same audit found that `ALL_CHECK_IDS` — +the hand-maintained list driving the network-purity, sensitivity and outcome suites — was +never asserted to match the check registry, so a registered-but-unlisted check would +escape all three. This spec adds **four** checks. + +**Acceptance criteria** + +17.1 THE TEST SUITE SHALL fail WHEN any test modifies, creates or removes a tracked file +under `evals/golden/`, `evals/fixtures/traces/` or `evals/taxonomy/`, enforced by a +session-scoped guard in `tests/conftest.py`. A test needing to exercise a writer SHALL +redirect it to `tmp_path` by monkeypatching the module's directory constant. + +17.2 THE TEST SUITE SHALL assert that `ALL_CHECK_IDS` and the check registry are equal in +both directions, that the list is duplicate-free, that every registered check has +committed `__fail__` and `__pass__` fixtures, and that no check references a failure mode +absent from the taxonomy. + +17.3 THE TEST SUITE SHALL fail WHEN a failure mode with a registered check is still +`status: reserved` in the taxonomy — the half-landed state task 0.2 creates deliberately. + +17.4 WORKSPACE DISCOVERY SHALL NOT depend on hardcoded filesystem roots. +`evals/run_evals.py::_WORKSPACE_PATH_RE` matches only paths under +`/Users|/home|/tmp|/var|/private`; on a host whose temp root differs, no candidate is +found and `_find_workspace` falls through to "newest directory in `./workspace/`" — which +returns a **wrong workspace rather than an error**, so a trace can be assembled from +someone else's run. `test_find_workspace_from_log` fails for exactly this reason on such a +host. THE FUNCTION SHALL either derive candidate roots from the log text without a root +allowlist, or return `None` when the log yields no match, rather than guessing. Related to +FM-008 `metric_source_drift`: a silently-wrong source is worse than a missing one. + +17.5 WHERE a task in this plan extends an existing module, THAT TASK SHALL bring the +module's existing behaviour under test **before** adding to it. The modules this plan +lands on, with their branch coverage as of 2026-09-12: +`tools/mcp_server.py` **0%** (tasks 1.5, 7.3), `tools/permissions.py` 35% (task 10.2), +`hooks/quality.py` 31% and `hooks/audit.py` 33% (tasks 1.4, 9.3). +`hooks/security.py` was brought to 100% line / 100% branch on 2026-09-12 and is the bar. + +17.6 EVERY deny, refusal or guard path introduced by this spec — the `ACCEPTANCE.json` +write denial (R6.3), the contract gate (R11.4), the bash allowlist (R14) — SHALL reach +**100% branch coverage**, with adversarial tests, before the component is enabled by +default. Feature paths carry no such bar; refusal paths do, because a guard's untested +branch is a guard that is not there. + +17.7 WHERE a test documents current behaviour that the plan intends to change (the +allowlist bypasses of R14.4, the case-sensitive `AI_TEAM_DENY_NATIVE_TOOLS` switch), THE +TEST SHALL say so in its docstring and name the requirement that will invert it, so the +change arrives with a deliberately flipped assertion rather than a silent edit. + +17.8 THE COVERAGE FLOOR (`fail_under`) SHALL be re-ratcheted at the end of each phase that +adds modules, and SHALL never be lowered to accommodate new untested code. + +--- + +## Traceability summary + +| Requirement | Source | New FM | New check | +| --- | --- | --- | --- | +| R1–R3 | Article (solo control arm) + quickstart (reference harness) | — | — | +| R4 | Article ("iteratively strip complexity") | — | — | +| R5 | `ai-team` README's own caveat | — | — | +| R6 | Quickstart `feature_list.json` | — | — | +| R7 | Quickstart monotonic invariant | FM-014 | `CHK-acceptance-monotonic` | +| R8–R9 | Quickstart session loop + article context resets | — | — | +| R10 | Article ("context anxiety") | FM-015 | `CHK-premature-termination` | +| R11 | Article (sprint contracts) | — | — | +| R12 | Article (self-evaluation bias) | FM-016 | `CHK-verifier-independence` | +| R13 | Quickstart Puppeteer verification + article Playwright evaluator | — | extends `CHK-runtime-smoke-present` | +| R14 | Quickstart bash allowlist | — | extends `CHK-guardrail-fp-budget` | +| R15 | Article (components encode assumptions that go stale; sprint construct removed in v2) | — | — | +| R16 | Article (evaluator waives its own findings; QA is weak out of the box) | FM-017 | `CHK-evaluator-capitulation` | +| R17 | 2026-09-12 coverage audit (golden-file writes, order-dependent test, unguarded `ALL_CHECK_IDS`) | — | — | diff --git a/.kiro/specs/harness-alignment/tasks.md b/.kiro/specs/harness-alignment/tasks.md new file mode 100644 index 00000000..fff6cbd4 --- /dev/null +++ b/.kiro/specs/harness-alignment/tasks.md @@ -0,0 +1,553 @@ +# Implementation Plan — Harness Alignment + +**Spec ID:** `harness-alignment` +**Requirements:** [`requirements.md`](./requirements.md) · **Design:** [`design.md`](./design.md) + +In-repo work is checked off. Live-spend tasks **4.3, 4.5, 5.4, 11.4** stay open +(human-triggered; not run). + +--- + +## How to execute this plan + +- Work **one task at a time, in order**. Do not batch phases. Each task is sized to be + reviewable on its own. +- Every task lists a **Definition of done**. Do not mark it complete until each bullet is + literally true. +- Run `uv run ruff check . && uv run ruff format --check . && uv run mypy src/ && uv run mypy evals/` + before marking any task complete. +- **Do not spend money.** Only Phases 4, 5, 7.4 and 11 make live model calls, and every one + of them is human-triggered and budget-capped. Everything else runs against fixtures. +- Phases 1–2 are strictly additive: no existing check, backend, or report path changes. +- New behavior that touches a normal run (Phases 8, 9, 10) ships **off by default** behind + an explicit flag until its ablation shows it earns its place. + +Cursor prompt shape: + +``` +Read .kiro/specs/harness-alignment/requirements.md and design.md for context. +Implement task 1.2 from .kiro/specs/harness-alignment/tasks.md. +Do not start any other task. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +--- + +## Phase 0 — Scaffolding + +- [x] **0.1 Create package skeleton** + - Create `evals/arms/` (with `__init__.py`, `py.typed` not needed — inherits from `evals`), + `evals/arms/vendor/.gitkeep`, `evals/results/ladder/.gitkeep`, + `src/ai_team/harness/` additions are in-place (package exists). + - Add `docs/contracts/.gitkeep` to the workspace scaffold template, not to the repo root. + - **Definition of done:** `uv run python -c "import evals.arms"` succeeds; `uv run mypy evals/` passes. + - _Requirements: R1.4_ + +- [x] **0.2 Taxonomy version bump and reservations** + - Bump `evals/taxonomy/failure_modes.yaml` `version` to `1.2.0`. + - Reserve FM-014, FM-015, FM-016, FM-017 ids with `status: reserved` and no + `implemented_by`, so nothing else claims them mid-implementation. + - **Definition of done:** `python -m evals.cli taxonomy coverage` runs clean and reports the + four reserved modes as unimplemented. + - _Requirements: R7.1, R10.1, R12.1, R16.1_ + +- [x] **0.3 Trace schema additions** + - Add `Trace.arm_id: str | None = None` and allow `context_pressure: float | None` inside + `phase_end` / `session_end` span payloads. Bump `SCHEMA_VERSION`. + - Confirm every existing fixture under `evals/fixtures/traces/` still loads. + - **Definition of done:** round-trip test passes for a trace with and without the new fields; + `python -m evals.cli index rebuild` succeeds on the existing corpus. + - _Requirements: R1.2, R5.1, R10.2_ + +- [x] **0.4 Suite guards (landed 2026-09-12 — verify, do not rewrite)** + - `tests/conftest.py` hashes `evals/golden/`, `evals/fixtures/traces/` and + `evals/taxonomy/` at session start and asserts them unchanged at teardown. + - `tests/unit/evals/test_suite_drift_guards.py` asserts `ALL_CHECK_IDS` ≡ registry, no + duplicates, fixtures present per check, no check pointing at an unknown failure mode, + and no implemented mode still `reserved`. + - **Definition of done:** both files present and green; confirm the reserved-mode guard + (R17.3) fails while FM-014…017 are reserved *and* a check for one of them is + registered — this is the half-landed state Phase 2 passes through, and it should be + loud. + - Fix `_find_workspace` (R17.4): drop the hardcoded root allowlist in + `_WORKSPACE_PATH_RE`, or return `None` when the log yields no candidate instead of + falling back to the newest directory in `./workspace/`. The fallback returns a wrong + workspace rather than an error, so a trace can be built from an unrelated run — + `test_find_workspace_from_log` fails on any host whose temp root is outside + `/Users|/home|/tmp|/var|/private`. + - _Requirements: R17.1, R17.2, R17.3, R17.4_ + +--- + +## Phase 1 — Durable acceptance list (free) + +- [x] **1.1 Acceptance data model** + - Implement `src/ai_team/harness/acceptance.py` models per design §3.1: `AcceptanceItem`, + `AcceptanceList`, `Demotion`, `VerifierIdentity`, `AcceptanceStatus`. + - `id` = `sha256(normalized_description + "\x00".join(steps))[:12]`; normalization collapses + whitespace so cosmetic reformatting does not change identity. + - **Definition of done:** unit test asserts id stability across whitespace/casing-of-punctuation + changes and instability across any semantic edit. + - _Requirements: R6.1, R6.2_ + +- [x] **1.2 Writer, loader, and the snapshot log** + - Implement `write_initial`, `load`, `mark_passing`, `demote`, `status`. + - Atomic writes (temp + `os.replace`). Every write appends + `{ts, op, item_id, sha256_of_file}` to `logs/acceptance.jsonl`. + - `mark_passing` rejects when `evidence` is empty or any path is absent from the workspace. + - **Definition of done:** unit tests cover accept, reject-on-missing-evidence, demote, + concurrent-write safety (two writers, no torn file), and that `demote` refuses without a reason. + - _Requirements: R6.3, R6.4, R6.5_ + +- [x] **1.3 Wire into planning** + - Call `write_initial` at the end of the planning phase from the existing requirements + output (`src/ai_team/models/requirements.py`), for all three backends. + - Priority is assigned from existing MoSCoW ordering where present, else source order. + - **Definition of done:** a stubbed planning run produces `ACCEPTANCE.json` with ≥ 1 item; + the file appears in the change receipt and the Trace artifact inventory. + - _Requirements: R6.1, R6.7_ + +- [x] **1.4 Deny agent writes to the acceptance file** + - Extend the SDK backend PreToolUse hook to deny `Write`/`Edit`/`MultiEdit` whose + `file_path` resolves to `ACCEPTANCE.json`, with a reason that names the MCP tool to use instead. + - **Definition of done:** adversarial unit tests cover the direct path, a relative path, a + path with `./`, and a symlink pointing at it — all denied; the new denial reaches + **100% branch coverage** (R17.6). `hooks/security.py` is already at 100%/100% — keep it + there, and bring `hooks/quality.py` (31% branch) and `hooks/audit.py` (33%) up as part + of this task, since the same hook chain now carries the acceptance rule (R17.5). + - _Requirements: R6.3, R17.5, R17.6_ + +- [x] **1.5 MCP tools `acceptance_status` and `acceptance_mark_passing`** + - Add both to `tools/mcp_server.py`; `acceptance_status` on all role allow-lists, + `acceptance_mark_passing` on QA only. + - `acceptance_mark_passing` records `VerifierIdentity` (`agent_role`, `session_id`, `subagent_id`). + - **Definition of done:** integration test drives both tools through the MCP surface and + asserts the identity is persisted. **Cover first (R17.5):** `tools/mcp_server.py` is at + **0% branch coverage** — bring its existing tool registration, argument validation and + error paths under test before adding to it, or these two tools will be the only part of + that module anyone can see break. + - _Requirements: R6.6, R12.2, R17.5_ + +--- + +## Phase 2 — Three new checks (free, retroactive) + +- [x] **2.1 `CHK-acceptance-monotonic`** + - Implement `evals/checks/acceptance.py`. Compare first and last acceptance snapshots: + fail on removed ids, changed ids, `true → false` without a `demotions[]` record, or a + `false → true` with empty/dangling evidence. + - Return `inconclusive` when the trace has no acceptance artifacts at all. + - **Definition of done:** registered in the check registry; fail and pass fixtures added under + `evals/fixtures/traces/` following the existing naming convention; FM-014 flipped from + `reserved` to `active` with `implemented_by: [CHK-acceptance-monotonic]`. + - _Requirements: R7.2, R7.3, R7.4, R7.5_ + +- [x] **2.2 Record `context_pressure`** + - Emit `context_pressure` on `phase_end` and `session_end` spans from SDK usage reporting, + falling back to `src/ai_team/config/token_tracker.py`. Emit `None` rather than guessing + when the window size is unknown. + - **Definition of done:** a stubbed run produces the field on every phase end; a stub with + no usage data produces `None` and does not raise. + - _Requirements: R10.2_ + +- [x] **2.3 `CHK-premature-termination`** + - Implement in `evals/checks/context.py`. Fire only when: status ok, ≥ 1 unsatisfied + acceptance item, no `spend_event` / `error` / watchdog / `max_turns` evidence in the + trailing window, and `context_pressure ≥ 0.75` (configurable via check params). + - Return `inconclusive` when `context_pressure` is `None`. + - **Definition of done:** fail/pass fixtures plus **four negative-control fixtures** + (spend event, watchdog kill, error span, max_turns) each asserted not to fire; + FM-015 set `active`. + - _Requirements: R10.3, R10.4, R10.5_ + +- [x] **2.4 `CHK-verifier-independence`** + - Implement in `evals/checks/verification.py`. Fail when a `passes` transition's + `VerifierIdentity` matches the identity that produced the write spans for that item and no + deterministic verifier contributed evidence. + - **Definition of done:** fail/pass fixtures; a fixture where `smoke` supplied the evidence + passes even with a matching identity; FM-016 set `active`. + - _Requirements: R12.3, R12.4, R12.5_ + +- [x] **2.5 Structured QA verdicts** + - Have the QA agent emit `docs/qa_verdicts.jsonl` per design §4.5a: + `{item_id, verdict, issues[], evidence[], qa_prompt_hash, identity, emitted_at}`. + Structured output, not prose — the check needs to read the evaluator's own findings, not + infer them. + - Add the verdict log as a Trace span source (`qa_verdict`). + - **Definition of done:** a stubbed QA pass writes parseable verdicts including the prompt + hash; Trace assembly tolerates the file's absence. + - _Requirements: R16.2, R16.6_ + +- [x] **2.6 `CHK-evaluator-capitulation` and the QA false-negative rate** + - Implement in `evals/checks/verification.py`: fail when `verdict == "accept"` while + `issues[]` holds a `major`/`blocker` entry and no remediation write span for that item + appears between detection and verdict. + - Compute `qa_false_negative_rate`: items the QA agent accepted that a deterministic + verifier (`smoke`, `ui_smoke`, tests) later failed. No labels, no judge. + - **Definition of done:** fail/pass fixtures; a fixture where the issue *was* fixed before + acceptance passes; FM-017 set `active`; the rate appears in the metrics output. + - _Requirements: R16.1, R16.3, R16.4, R16.5, R16.7_ + +- [x] **2.7 Backfill the existing corpus** + - Run all four new checks over every trace already in `evals/traces/` and record outcomes. + - **Definition of done:** `python -m evals.cli run --tier A` passes; a short note in + `docs/journal/` records what the new checks found retroactively, with trace ids. + - _Requirements: R7.5, R10.5, R12.5, R16.7_ + +--- + +## Phase 3 — Arm scaffold (no new spend) + +- [x] **3.1 Arm protocol and registry** + - Implement `evals/arms/base.py` and `evals/arms/registry.py` per design §2.2. + - **Definition of done:** round-trip tests for `ArmSpec`/`ArmRun`/`Divergence`; duplicate + registration raises. + - _Requirements: R1.1, R1.3, R1.4_ + +- [x] **3.2 `ai_team` arm over the existing runner** + - Implement `evals/arms/ai_team.py` wrapping the current run path. No behavior change; it + only produces an `ArmSpec` and stamps `arm_id` onto the Trace. + - **Definition of done:** an existing scenario run through the arm produces a byte-identical + workspace to running it directly, plus `arm_id` on the trace. + - _Requirements: R1.2, R1.6_ + +- [x] **3.3 Budget and ceiling enforcement** + - Implement `CostControls`, the pre-flight sweep budget check, and the wall-clock kill. + - **Definition of done:** a stub arm that sleeps past its ceiling is terminated and still + yields a Trace with `status = "budget_exhausted"`. + - _Requirements: R1.5_ + +- [x] **3.4 `ladder` CLI with `--dry-run` default** + - `python -m evals.cli ladder run --scenario S --arms a,b --n N [--execute]`. + - Dry run prints the resolved plan, per-arm ceilings, and projected total; refuses when the + projection exceeds the $25 sweep ceiling. + - **Definition of done:** dry run works end-to-end with stub arms and spends nothing; + `--execute` is required for any live call. + - _Requirements: R1.5, R4.6_ + +--- + +## Phase 4 — Control arms (first spend) + +- [x] **4.1 Implement `SoloArm`** + - Single SDK session, one general system prompt, scenario brief only. Pre-create the standard + workspace scaffold. Keep cost/audit hooks; remove all behavioral components. + - **Definition of done:** unit test with a mocked client asserts exactly one session, no + subagents, no guardrail invocation, and the standard workspace layout. + - _Requirements: R2.1, R2.2, R2.3_ + +- [x] **4.2 Trace assembly and receipt parity** + - Ensure `from_workspace` yields a valid Trace for a solo workspace and that + `receipt.write_from_run` produces the same fields as an `ai_team` run. + - **Definition of done:** a recorded solo workspace fixture scores through all sixteen + checks with no special-casing and no crash. + - _Requirements: R2.4, R2.5_ + +- [ ] **4.3 First live solo run** *(human-triggered, ≤ $3)* + - Run `todo-api-beginner` once. Commit the trace as a fixture (redacted via + `python -m evals.cli fixtures redact`). + - **Definition of done:** trace committed; FM incidence recorded; cost and wall-clock logged + in the journal entry. + - _Requirements: R2.3, R2.4_ + +- [x] **4.4 `harnessed_solo` arm — the missing middle rung** + - Implement `role_decomposition` as an ablatable component: the full harness (guardrails, + smoke gate, constraints pinning, lessons, acceptance list) driving **one generalist + agent** instead of nine roles. Register it under its own arm id `harnessed_solo` as an + alias of `ai_team_ablated:role_decomposition`. + - **Definition of done:** unit test asserts one agent, zero subagent spans, and every + harness component still active in `ArmSpec.harness_components`. + - _Requirements: R4.1, R4.2, R4.3_ + +- [ ] **4.5 First live `harnessed_solo` run** *(human-triggered, ≤ $6)* + - Same scenario as 4.3. Commit the redacted trace. + - **Definition of done:** with 4.3 and the existing `ai_team` traces in hand, both + comparisons are computable — `solo → harnessed_solo` (what the harness buys) and + `harnessed_solo → ai_team` (what decomposition buys). Record both in the journal, and + record them even if the second delta is zero or negative. Especially then. + - _Requirements: R4.4, R4.5_ + +--- + +## Phase 5 — Reference-harness arm + +- [x] **5.1 Vendor the quickstart** + - Copy `anthropics/claude-quickstarts/autonomous-coding` to + `evals/arms/vendor/autonomous_coding/` at a pinned sha. Write `PROVENANCE.md` with URL, + sha, MIT licence text reference, retrieval date, and an explicit "do not edit" note. + - **Definition of done:** a test asserts the vendored tree's hash matches a recorded manifest, + so accidental edits fail CI. + - _Requirements: R3.1, R3.2_ + +- [x] **5.2 Implement `ReferenceArm`** + - Scenario → `app_spec.txt`; `max_features` default 25; external wall-clock and iteration + ceilings; stdout capture. Every adaptation recorded as a `Divergence`. + - Handle unavailability (`claude` CLI, node, network) as `status = "unavailable"`. + - **Definition of done:** unit tests with a stubbed subprocess cover the happy path, the + ceiling kill, and each unavailability cause. + - _Requirements: R3.2, R3.3, R3.4, R3.6_ + +- [x] **5.3 Trace assembly from reference artifacts** + - Parse `[Tool: …]` / `[Done]` / `[BLOCKED]` markers, `feature_list.json`, + `claude-progress.txt`, and git history into spans. Record every missing stream in + `trace.warnings[]` rather than fabricating it. + - **Definition of done:** a captured stdout fixture yields a Trace whose warnings explicitly + name the absent cost and guardrail streams. + - _Requirements: R3.5_ + +- [ ] **5.4 First live reference run** *(human-triggered, ≤ $8)* + - Same scenario as 4.3. Commit the redacted trace as a fixture. + - **Definition of done:** trace committed; the FM-007 positive (no internal spend ceiling) is + recorded as expected rather than filed as an adapter bug. + - _Requirements: R3.4, R3.5_ + +--- + +## Phase 6 — Ladder report and claim discipline (free) + +- [x] **6.1 `render_ladder`** + - Implement per design §4.7: JSON first, Markdown rendered from that JSON, per-arm `n`, + means with CIs, FM incidence, smoke pass rate, accepted-change rate, demotion rate. + - **Definition of done:** golden-file test on the three committed fixture traces; a test + asserts the Markdown numbers are read from the JSON, not recomputed. + - _Requirements: R5.1, R5.5_ + +- [x] **6.2 Stamps and refusals** + - Implement `MIXED-MODEL` and `UNDERPOWERED` stamping and the flag-gated refusals. + - **Definition of done:** tests assert refusal without flags, stamping with them, and that the + stamp text appears in both JSON and Markdown outputs. + - _Requirements: R5.2, R5.3_ + +- [x] **6.3 Per-phase cost, QA rate, and staleness columns** + - Add per-phase cost and wall-clock breakdown per arm from `logs/costs.jsonl`; + `qa_false_negative_rate` beside `smoke_pass_rate`; and the `STALE` / + `never measured` markers on ablation rows. + - Implement `cost_per_fm_avoided` as optional, stamped + `DERIVED — RATIO OF TWO NOISY ESTIMATES`, suppressed below n=5. + - **Definition of done:** golden-file test covers a stale row, a never-measured row, and a + suppressed ratio; no ablation row can render a zero delta for a component that was never + measured. + - _Requirements: R5.1a, R5.1b, R15.2, R15.4, R16.5_ + +- [x] **6.4 Divergence surfacing** + - Render each arm's `source_ref` and full divergence list beneath the table. + - **Definition of done:** the reference arm's row shows `max_features=25` and every other + adaptation; a divergence added in code appears in the report without a report change. + Every arm's row notes that it received a full scenario contract rather than a thin brief + (design §9.4). + - _Requirements: R5.4_ + +--- + +## Phase 7 — UI verification + +- [x] **7.1 Scenario `ui` block** + - Extend the scenario contract schema with an optional `ui` block + (`base_url`, `boot_cmd`, `ready_path`, `viewport`). Populate it for `todo-api-beginner`. + - **Definition of done:** existing scenarios without the block still validate. + - _Requirements: R13.6_ + +- [x] **7.2 `run_ui_smoke`** + - Implement `src/ai_team/tools/ui_smoke_tools.py` per design §4.5 with Playwright, headless + Chromium, fixed viewport. Write `docs/ui_smoke_results.json` and screenshots to + `docs/verification//`. Never probe a foreign service. + - Console and page errors fail a step deterministically. + - **Definition of done:** integration test against a fixture static page with a deliberate + console error fails the step with no model call; a UI-less scenario returns `skipped`. + - _Requirements: R13.1, R13.2, R13.3, R13.4, R13.6_ + +- [x] **7.3 Expose as an MCP tool and extend FM-006** + - Register `run_ui_smoke` on `ai_team_tools`, QA allow-list only. Extend + `CHK-runtime-smoke-present` so UI-bearing scenarios require a UI smoke result and a + backend-only probe does not satisfy them. + - **Definition of done:** a fixture trace with only an HTTP probe on a UI scenario now fails + the check; the same trace on a non-UI scenario still passes. + - _Requirements: R13.5, R13.8_ + +- [x] **7.4 Design rubric document** + - Write `docs/UI_QUALITY_RUBRIC.md`: four criteria (design quality, originality, craft, + functionality), calibration examples at each level, and an explicit note that criterion + wording steers output toward convergence. State plainly that it is advisory until it clears + the eval-harness alignment bar. + - **Definition of done:** the document exists, is linked from `docs/EVAL_METHODOLOGY.md`, and + no gate reads it. + - _Requirements: R13.7_ + +--- + +## Phase 8 — Multi-session continuation + +- [x] **8.1 `SessionRecord` and the session log** + - Implement the model and `logs/sessions.jsonl` writing; add `session_start` / `session_end` / + `regression_check` span types to Trace assembly. + - **Definition of done:** round-trip test; Trace assembly reads the new log without breaking + on its absence. + - _Requirements: R8.3_ + +- [x] **8.2 Session loop** + - Implement `run_sessions` per design §4.3. Fresh context per session from the four files. + Terminate on all-pass, `max_sessions`, spend, wall-clock, or `no_progress_sessions`. + - Off by default; requires explicit `--max-sessions` and a total budget. + - **Definition of done:** integration test over a stubbed backend runs three sessions and + exercises each termination condition in its own test. + - _Requirements: R8.1, R8.2, R8.4, R8.6_ + +- [x] **8.3 Commit discipline** + - Require a commit at session end; mark `dirty_exit` otherwise and feed FM-012. + - **Definition of done:** a stubbed session leaving uncommitted changes produces + `status = "dirty_exit"` and the existing FM-012 check fires on its trace. + - _Requirements: R8.5_ + +- [x] **8.4 Regression verification** + - Implement the R9 policy: highest-priority passing item plus one seeded-random item not + verified in the last `m` sessions. Demote on failure and prioritize demoted items. + - **Definition of done:** integration test injects a regression between sessions and asserts + demotion → prioritization → re-pass, with `demotion_rate` surfacing in the report. + - _Requirements: R9.1, R9.2, R9.3, R9.4_ + +--- + +## Phase 9 — Contract negotiation + +- [x] **9.1 Contract model and store** + - Implement `src/ai_team/harness/contracts.py` and the `docs/contracts/.json` format. + - **Definition of done:** round-trip tests; round ceiling enforced at the store layer. + - _Requirements: R11.1, R11.3_ + +- [x] **9.2 Negotiation step** + - Developer writes the contract; QA validates against the acceptance item and returns + `accepted` / `rejected` with reasons. File-based handoff only. + - **Definition of done:** integration test with a stubbed QA rejecting twice asserts + escalation at round 3 through `flows/error_handling.py`, not a fourth round. + - _Requirements: R11.2, R11.3_ + +- [x] **9.3 Gate implementation on an accepted contract** + - PreToolUse hook denies writes to `src/` for an item without an accepted contract, when the + component is enabled. No-op when ablated. + - **Definition of done:** adversarial tests for enabled-deny and ablated-allow; verification + scores against `testable_behaviors[]`. + - _Requirements: R11.4, R11.5, R11.6_ + +--- + +## Phase 10 — Default-deny bash + +- [x] **10.1 Command extraction with fail-safe semantics** + - Implement the parser. Block on anything it cannot decompose. + - **Definition of done:** one unit test per construct in R14.4 — `&&`, `||`, `;`, pipes, + `$(…)`, backticks, `env VAR=x cmd`, `xargs`, `nohup`, `timeout`, absolute paths, + `git -c core.hooksPath=…`, `git -c core.pager=…`, unclosed quotes — each asserting the + expected allow or block, with undecomposable input always blocked. + - `tests/unit/backends/test_claude_sdk_security_hook_adversarial.py::test_known_bypasses_are_not_caught_today` + already documents the denylist's gaps and names this requirement. **Invert it here** + (R17.7) rather than deleting it — the diff should show the bypasses closing. + - Also decide on `AI_TEAM_DENY_NATIVE_TOOLS` case-sensitivity: the switch matches + lowercase only, so `TRUE` fails **open**. One-line fix (`.strip().lower()`), with the + documenting test inverted in the same commit. + - _Requirements: R14.3, R14.4, R17.7_ + +- [x] **10.2 Per-role allowlists** + - Define them in `tools/permissions.py` beside the tool allow-lists. Evaluate the allowlist + **after** the existing deny patterns. + - **Definition of done:** a developer-role command permitted for developers is blocked for the + architect role; existing deny patterns still fire first. **Cover first (R17.5):** + `tools/permissions.py` sits at 35% branch — add a guard asserting every role's allow-list + references only registered tool names, which is how a typo'd MCP tool name (silently + granting nothing) would otherwise ship. + - _Requirements: R14.1, R14.2, R17.5_ + +- [x] **10.3 False-positive budget** + - Run the guardrail corpus with the allowlist active; measure the FP delta against the current + budget. Keep the allowlist **disabled by default** unless `CHK-guardrail-fp-budget` holds. + - **Definition of done:** corpus numbers recorded in the journal; the default-on decision is + made from that measurement, not from preference. + - _Requirements: R14.5, R14.6_ + +- [x] **10.4 Confirm no settings-file confinement** + - Assert in a test that no `.claude_settings.json` is written into a generated workspace; + confinement stays hook-enforced. + - **Definition of done:** test present and passing. + - _Requirements: R14.7_ + +--- + +## Phase 11 — Ablation sweep and writeup + +- [x] **11.1 `AblatedArm` and the component registry** + - Implement config-driven ablation for the closed set in R4.1. Fail loudly on a component + that cannot be disabled by config. + - **Definition of done:** each named component can be disabled and the active set is recorded + in `ArmSpec.harness_components` and in trace provenance. + - _Requirements: R4.1, R4.2, R4.3_ + +- [x] **11.2 Ablation result store and staleness** + - Persist each result to `evals/results/ablations/.json` with `model_id`, + `model_snapshot_date`, `measured_at`, `scenario_id`, `n`, delta and CI. + - Implement `python -m evals.cli ablation status` listing every component as + `current` | `STALE` | `never measured`. + - Staleness never fails the gate (R15.5). + - **Definition of done:** changing the configured default model flips every prior result to + `STALE` without re-running anything; a component with no result reads `never measured`. + - _Requirements: R15.1, R15.2, R15.3, R15.4, R15.5_ + +- [x] **11.3 Ablation deltas in the report** + - Render deltas against the `ai_team` control with CIs, plus the + `components with no measured effect at this n` section. + - **Definition of done:** golden-file test on synthetic traces; a zero-delta component lands + in the named section rather than being silently omitted. + - _Requirements: R4.4, R4.5, R4.6_ + +- [ ] **11.4 Full ladder sweep** *(human-triggered, ≤ $25)* + - `todo-api-beginner`, n ≥ 3 per arm, fixed model: `solo`, `harnessed_solo`, `reference`, + `ai_team`, plus the ablation with the strongest prior (`smoke_gate`). + - **Definition of done:** all traces committed as fixtures; ladder report rendered; every + table carries its stamps honestly. + - _Requirements: R5.1–R5.5, R4.4_ + +- [x] **11.5 Journal entry and taxonomy update** + - Write `docs/journal/.md` with commit references per house style. Record what each + arm cost, what each ablation changed, and — explicitly — any component that bought nothing. + - Update `docs/posts/failure-taxonomy.md` with narrative sections for **FM-011 through + FM-017** — not just this spec's four. The essay documents ten modes; the YAML has + thirteen as of taxonomy 1.1.0, so FM-011/012/013 are already undocumented drift that this + task inherits. + - **Headline-stat consequence — decide deliberately, do not discover at publish time.** The + essay's punchline is "only one of the ten was caused by an LLM being dumb," and it is also + the flagship post's alternate title. FM-014, FM-015 and FM-017 are all `layer: model`, so + the ratio becomes **four of seventeen**. The honest framing is that the model-layer count + rose because the instruments improved — these are failures that were always happening and + were previously invisible — not because the model got worse. Rewrite the punchline; do not + quietly keep the old number beside a seventeen-row table. + - **Definition of done:** journal entry follows the existing entry shape; the taxonomy essay + and `failure_modes.yaml` agree on all seventeen modes; the layer ratio stated in the essay + matches a count computed from the YAML rather than typed by hand. + - _Requirements: R4.5, R7.1, R10.1, R12.1, R16.1_ + +--- + +- [x] **11.6 Re-ratchet the coverage floor** + - Recompute the combined `src/ai_team` + `evals` figure on macOS and in CI, then raise + `fail_under` in `pyproject.toml` to the measured value less the observed + macOS/ubuntu branch variance (~6pt). Record the new value and its basis in the comment + block that already tracks this history. + - **Definition of done:** floor raised; the gate fails on a deliberately reverted test + file; no module added by this spec is below the bar of R17.6. + - _Requirements: R17.8_ + +--- + +## Minimum defensible slice + +If the full plan is too long: **Phases 0–2, plus Phase 4, plus Phase 6.** + +The acceptance list with a monotonic check, three new deterministic failure modes scored +retroactively over the existing corpus, the `solo` and `harnessed_solo` control arms, and a +report that refuses to overclaim. That is a coherent result on its own — "here is what the +harness buys, what the nine-role team buys on top of it, and here are the caveats, enforced +in code." Everything past it is refinement. + +If even that is too long, task 4.5 is the one to keep. Two runs and an existing `ai_team` +trace give you the only number in this spec that could change what you build next. + +The session loop (Phase 8) without Phase 1 is not worth building: resumption with no durable, +protected checklist is just a longer run. diff --git a/.kiro/specs/production-hardening/BASELINE.md b/.kiro/specs/production-hardening/BASELINE.md new file mode 100644 index 00000000..2b2bae19 --- /dev/null +++ b/.kiro/specs/production-hardening/BASELINE.md @@ -0,0 +1,57 @@ +# Production-hardening baseline + +Recorded 2026-09-13 before Track A. Every number is reproduced by the command +in the same row. Docker image size was not built at baseline (no local daemon +requirement for this record); the ratchet is set when task 5.3 first builds. + +## Commands and values + +| Claim | Command | Value | +| --- | --- | --- | +| LOC per top-level package | `find src/ai_team/$pkg -name '*.py' ! -path '*/node_modules/*' \| xargs wc -l` | agents 1327; backends 6551; config 1582; core 1516; crews 1388; flows 2986; guardrails 2335; harness 2159; knowledge (md, not py); memory 781; models 939; reports 389; tasks 1071; tools 5887; ui 2442; utils 1100 | +| Orphan modules | `wc -l` on the five files | lessons_loop 161; session_loop 315; verifiers 25; qa_verdicts 75; ladder_report 228; **804 total** | +| Uncollected tests | `wc -l evals/backends/test_*.py evals/test_backend_comparison.py` | 175+169+191+273 = **808** | +| `ignore_errors` module patterns | parse `pyproject.toml` `[[tool.mypy.overrides]]` with `ignore_errors` | **18** patterns (spec intro said 16; vendor + fixtures globs included) | +| Complexity (src/*.py excl. frontend/node_modules) | AST walk | functions >80 lines: **42**; >150: **5**; >8 params: **14** (spec intro 63/8/17 counted evals + more) | +| `docs/**/*.md` | `find docs -name '*.md' \| wc -l` | **48** (spec intro 42; journal/campaign/showcase grew) | +| Tracked images | `git ls-files docs/images/` | **33** files, `du -sh docs/images` = **9.0M**; 20 unreferenced | +| Tracked `.archive/` | `git ls-files .archive/` | **21** files | +| `evals/golden/.validation_log.jsonl` | `wc -c` | **0** bytes, tracked | +| Unit tests collected | `uv run pytest tests/unit --collect-only -q` | **1483** | +| Docker image size | not measured (no build at baseline) | set at task 5.3 | + +## After-state + +Recorded 2026-09-13 after Track A (Phases 0–5, 8–9 except live 9.1). +Refreshed the same day after Track B (Phases 6–7) and the Phase 3 hygiene sweep. + +Commands vs the pre-Track-A parent `946f6d9`: + +| Claim | Command | Value | +| --- | --- | --- | +| Python LOC `src/ai_team` + `evals` (excl. `node_modules`, `vendor`) | `git ls-tree -r --name-only ` filtered `*.py`, `wc -l` | **46,704 → 46,049** (**−655**) | +| Tracked `docs/images/` blob bytes | `git cat-file -s` per path | **9,329,198 → 5,289,340** (**−4.0 MB**) | +| Tracked image files | `git ls-files docs/images/` | **34** (working-tree `du` includes untracked campaign files; ignore that) | +| Unit tests collected | `uv run pytest tests/unit --collect-only -q` | **1534** | +| Complexity (≤2-branch exempt) | AST walk / `ratchets.toml` | **55 / 4 / 14** | +| `ignore_errors` LOC | `test_type_budget.py` | **12,726** / 70 files (ratchet 13,000) | +| Docker image size | `docker image inspect --format '{{.Size}}'` after `docker build -f docker/Dockerfile` | Track A **790 MB** was Docker Desktop compressed Size (ratchet 900). GHA run 34773066923: **2.55 GB** uncompressed with `RUN chown -R` doubling. Post-`--chown` predicted **~1.45 GB**; ratchet **1800 MiB** until the next green `image_bytes=` | +| Web E2E | `pytest tests/e2e/web -m web_e2e` | **27 passed, 1 skipped** | +| `.archive/` tracked files | `git ls-files .archive/` | **21** (kept; see `.archive/README.md`) | + +The Phase 1 guess was roughly −1,500 to −2,000 LOC and −7 MB of images. Net Python +is smaller because Track B added shims and Phase 7 extracted modules rather than +deleting them. Image bytes dropped ~4 MB, not 7: publication allowlisted assets +were kept. `.archive/` stayed. + +Package LOC after the CrewAI move (shims at top level; logic under `backends/`): +agents 129; backends 13923; config 1701; core 1653; crews 56; flows 54; +guardrails 2336; harness 2242; memory 781; models 479; reports 389; tasks 49; +tools 5926; ui 2754; root py 1098; **no `utils/`**. + +## Guard suite CI time + +Local `pytest tests/unit/repo` is a few seconds inside the unit run. Target < 30 s +added (R20.5) holds locally; GitHub Actions wall-clock is recorded on the first +green `lint` job after a PR into `main`/`develop`. Feature-branch pushes do not +trigger `.github/workflows/ci.yml`. diff --git a/.kiro/specs/production-hardening/README.md b/.kiro/specs/production-hardening/README.md new file mode 100644 index 00000000..e550c731 --- /dev/null +++ b/.kiro/specs/production-hardening/README.md @@ -0,0 +1,150 @@ +# Spec: `production-hardening` + +Kiro-style three-document spec closing the gap between what this repository **claims** and +what it **wires** — so the codebase reads the way a reviewer expects a production-grade +enterprise system to read. Read in order: + +1. **[`requirements.md`](./requirements.md)** — 24 requirements with EARS acceptance + criteria, constraints, non-goals, and a file-level traceability table. +2. **[`design.md`](./design.md)** — the wire-or-delete decision table, machine-checkable + status definitions, the package contract, the documentation model, security design, + container fixes, the enforcement suite, migration risk, and eight open decisions. +3. **[`tasks.md`](./tasks.md)** — 10 phases, 62 tasks, each with a definition of done and + requirement traceability. + +Companion to [`../ui-refinement/`](../ui-refinement/) (frontend design system and +accessibility). Builds on, and does not restate, [`../eval-harness/`](../eval-harness/) +and [`../harness-alignment/`](../harness-alignment/). + +## The one-line version + +The engineering here is strong — a real failure taxonomy bound to deterministic checks, a +$0 replay gate, subprocess isolation, spend guards, adversarial tests, an honest threat +model, 1,456 tests, and a journal that records corrections. What has drifted is the wiring +*behind the claims*: ~800 LOC that nothing imports, 808 LOC of tests that never run, a +type gate that opts out of half the code, a control plane with no control, and a container +that does not contain the app it advertises. + +## What the audit found + +| | Evidence | +| --- | --- | +| **Unreachable code** | `harness/lessons_loop.py` (161), `session_loop.py` (315), `verifiers.py` (25), `qa_verdicts.py` (75), `evals/ladder_report.py` (228) — imported by **nothing but their own tests**, while `HARNESS.md` marks Feedback *closed-loop* and Verification *enforced* citing two of them | +| **Tests that never run** | `testpaths = ["tests"]`, so `evals/backends/test_*_eval.py` + `evals/test_backend_comparison.py` — **808 LOC** — are never collected (and are `ignore_errors` under mypy) | +| **Frontend gates idle** | CI runs `npm ci && npm run build` — never `npm test` or `npm run lint`, so **110 test cases and ESLint never execute in CI** | +| **Type gate opt-out** | `ignore_errors = true` covers two of three backends, `agents.*`, `crews.*`, `flows.main_flow`, `tools.test_tools`, `tools.smoke_tools`, `evals.run_evals`, and more — roughly half the Python | +| **Misleading layout** | `agents/` + `crews/` + `tasks/` + `flows/` = **6,772 LOC** at top level, reachable only through `crewai_backend/backend.py`; the README's structure tree omits those four **and** `harness/`, `models/`, `reports/`, `utils/` | +| **Control plane** | `server.py` (1,791 LOC, 24 routes): **no authentication on any route**, `--host` defaults to `0.0.0.0`. Those routes start paid runs, resume HITL, delete workspaces, and stream workspace files | +| **Path containment** | Artifact reads validate strings *before* the join but never assert containment after `resolve()` — a symlink written by an agent escapes the workspace | +| **Container** | Titled "FastAPI + React" with **no Node stage** (serves no UI); `build-essential` in the runtime stage; `.dockerignore` misses `node_modules` so `COPY src ./src` ships ~230 MB; healthcheck probes `/api/backends`; compose builds against Ollama + NVIDIA GPU with a context that cannot see `pyproject.toml` | +| **Complexity** | 63 functions > 80 lines, 8 > 150, 17 with > 8 params. Worst: `main.py:_cmd_run` — 259 lines, 71 branches, **16 parameters** | +| **Performance & scale** | Two of the four properties the repo claims have **no evidence and no stated limits**: eight benchmark tests publish nothing, the only perf doc is being deleted as meaningless, and run state lives in a process-local dict wiped on restart | +| **Repo surface** | All 27 GitHub Action `uses:` lines pin mutable tags (`@v4`), not SHAs; no PR/issue template, no `CODEOWNERS`, no `CHANGELOG`, `version` still `0.1.0` | +| **Data artifacts** | The accepted Tier A baseline (`evals/baselines/tier_a.json`) is a month stale, pinned to a pre-harness-alignment sha, covers **13 of 20 registered checks** and marks every one `fail` with `suite_pass_pow_k: 0.0` — with no sentence explaining why. **25 of 38 tracked images (~7 MB)** are referenced by nothing. `evals/golden/.validation_log.jsonl` is a tracked, hidden, zero-length append log | +| **Doc rot** | 42 markdown files; ~70 dead relative links (≈65 in `PROMPT_TRACKING.md`); a 68 KB self-declared "aspirational" roadmap; a 22-line performance report whose content explains that it isn't real; `pyproject.toml` author reads `"Your Name "` | + +**Clean, and worth protecting:** the eval fixture corpus — 94 traces, 20 check ids, **zero orphan fixtures and zero checks without one** — is the best-maintained data in the repository. R21.8 adds a guard to keep it that way. + +## The principle + +> **Nothing ships that is not reachable, and nothing is claimed that is not checked.** + +Every requirement makes a claim true, makes a gate real, or deletes something that is +neither. + +## What is explicitly *not* wrong + +The harness architecture, the Trace boundary, the check registry, the taxonomy, the tier +model, subprocess isolation, the spend guard, and the guardrail layering are sound and are +not touched. `SECURITY.md`, the "Honest gaps" paragraph in `HARNESS.md`, and the dated +`fail_under` history comment in `pyproject.toml` are the most senior-looking artifacts in +the repository — this spec extends that voice rather than trimming it. + +## Two tracks + +Phase numbers are a dependency order, **not a priority order**. Payoff per hour differs by an +order of magnitude: + +| Track | Phases | Effort | Payoff | +| --- | --- | --- | --- | +| **A — what a reviewer sees** | 1 delete · 2 truth · 3 gates · 4 security · 5 container · 9 evidence · 8 guards | ~1 week | Everything findable in a 30-minute read, plus real numbers for "performant / scalable" | +| **B — what the maintainer feels** | 6 boundaries · 7 complexity | ~1 week | The two largest diffs in the repo (6,772 LOC moved, 24 routes re-homed) for changes near-invisible to a reader | + +Recommended order: **1 → 2 → 3 → 4 → 5 → 9 → 8**, then Track B only if the repository keeps +being developed. + +Deletion first, so nothing later is spent on code that leaves. Truth second — nearly free, and +it is what a reviewer reads first. Gates third, so every later phase is protected. Security +before the big moves: highest consequence, smallest diff. + +**Sequencing with [`ui-refinement`](../ui-refinement/):** task 3.1 turns on `npm test` in CI, +and that spec's drift test is red until its Phases 1–6 land. Land 3.1 first *or* finish +`ui-refinement` first — not both at once, or `main` stays broken for days. + +**The highest-value hour is task 2.1** — making the harness status table true. It is the +table a reviewer reads to decide whether the project is serious, and it currently contains +two claims the code does not support. + +## Tests this spec adds + +Twelve new test modules. Eight are repository invariants that make the spec self-enforcing; +four cover behaviour it changes. + +| Test | Locks | Requirement | +| --- | --- | --- | +| `tests/unit/ui/test_auth.py` | 401 without a token, 200 with it; **route-coverage**: every route is authed or explicitly public; bind-default rules | R15 | +| `tests/unit/ui/test_artifacts_adversarial.py` | Symlink to `/etc/passwd`, symlink to a sibling run, nested symlink dir, absolute `project_id`, sensitive-name bypass | R16 | +| `tests/unit/repo/test_reachability.py` | No orphan modules; the dormant allowlist names each module's activation flag | R1, R9 | +| `tests/unit/repo/test_references.py` | Relative links and source paths resolve (`/` convention excepted) | R4 | +| `tests/unit/repo/test_readme_structure.py` | README's structure tree ≡ real packages | R5 | +| `tests/unit/repo/test_collection.py` | No `test_*.py` outside `testpaths` | R6 | +| `tests/unit/repo/test_type_budget.py` | `ignore_errors` **LOC** budget, ratcheting down only | R8 | +| `tests/unit/repo/test_complexity.py` | Branch/length/param ratchets | R12 | +| `tests/unit/repo/test_import_direction.py` | `core`/`harness`/`evals` never import a backend | R10, R11 | +| `tests/unit/repo/test_data_artifacts.py` | Fixture↔check mapping both ways; baseline covers every check; no unreferenced image | R21 | +| Frontend vitest case | Token header attached when configured, absent when not | R15.6 | +| Retention unit test | Prune command removes/keeps correctly and is idempotent | R22.6 | + +Plus a CI image job (non-root, no `gcc`, size ratchet, `/api/health` → 200) and a compose +build check, and **R24**, which sets the bar for all of them: offline, deterministic, +order-independent, no writes to tracked files, a recorded way to watch each guard fail, and a +guard that raises rather than passes when its match set is empty. + +The 808 LOC of currently-uncollected eval tests are either moved under `tests/` and made to +pass, or deleted (task 1.2) — not left where pytest cannot see them. + +## Constraints baked in + +| | | +| --- | --- | +| Behaviour | No change to run semantics, the taxonomy, check IDs, or the trace schema | +| Compatibility | CLI flags, console scripts, REST/WS paths and `data-testid` values stay stable; Python import paths change only behind one-release shims | +| Spend | **Every task runs offline.** No task may add a live model call | +| Deletion bias | Unreachable code is deleted, not archived — git history is the archive | +| Wiring bias | Deliberately dormant code must be activatable in one documented step, with a test for both paths | +| Enforcement | Eleven guards in `tests/unit/repo/`, running in the existing `lint` job, adding < 30 s | +| Scale | R22 requires **measuring and stating** the envelope — implementing a persistent run store, queue, or worker pool is explicitly out of scope | +| Out of scope | Multi-tenancy, RBAC/SSO, Kubernetes, coverage-floor raises, rewriting the journal | + +## Executing this with Cursor + +``` +Read .kiro/specs/production-hardening/requirements.md and design.md for context. +Implement task 1.3 from .kiro/specs/production-hardening/tasks.md. +Do not start any other task. Do not change run semantics, check IDs, route paths, +or data-testid values. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +Two tasks need a **human decision before an agent starts**: task 0.2 (wire / dormant / +delete for the five orphan modules) and the eight open decisions in `design.md` §12. + +## Minimum defensible slice + +**Track A** (Phases 1, 2, 3, 4, 5, 9, 8). Delete what is unreachable, make the documents true, +make the gates real, put a control on the control plane, ship a container that contains the +app, and publish one honest benchmark with a stated envelope. Roughly a week, no architectural +change, and it fixes everything a reviewer finds in their first thirty minutes. + +If even that is too long: **task 2.1 and Phase 4** — a status table that is true and an API +that cannot be driven by anyone on the network. diff --git a/.kiro/specs/production-hardening/design.md b/.kiro/specs/production-hardening/design.md new file mode 100644 index 00000000..ef7fbd0c --- /dev/null +++ b/.kiro/specs/production-hardening/design.md @@ -0,0 +1,414 @@ +# Design — Production Hardening + +**Spec ID:** `production-hardening` +**Requirements:** [`requirements.md`](./requirements.md) +**Tasks:** [`tasks.md`](./tasks.md) + +--- + +## 1. Overview + +### 1.1 What is actually wrong + +Nothing here is a rewrite. The audit found no architectural mistake in the harness, the +taxonomy, the trace boundary, or the check registry — those are the parts worth showing. +What it found is **drift between what the repository asserts and what it wires**, in four +places, each with the same shape: something was built, something was written about it, and +then the wiring moved without the writing moving. + +``` + CLAIMED ACTUAL + ─────── ────── + HARNESS.md: Feedback closed-loop → lessons_loop.py imported by its own test only + README: "the harness is the → harness/ absent from the README's own + real deliverable" project-structure tree + pyproject: mypy gate → ignore_errors over ~half the Python + CI: frontend job → builds, never lints or tests (110 cases idle) + Dockerfile: "FastAPI + React" → no Node stage; container serves no UI + SECURITY.md: full threat model → no row for the unauthenticated control plane +``` + +Every one of those is a one-line fix to the claim or a one-day fix to the wiring. What +makes them worth a spec is that they are exactly the checks a senior reviewer runs, in the +order they run them. + +### 1.2 Design principles + +1. **Delete before you refactor.** ~1,600 LOC (orphans + uncollected tests) can leave in + Phase 1. Every later phase is cheaper for it. +2. **A claim without a check is a liability.** The repository's credibility comes from its + claims being verifiable. Each claim this spec keeps gets a test (R20); each claim it + cannot check gets downgraded (R2.3). +3. **Move, don't redesign.** R10 relocates 6,772 LOC without changing a line of logic. A + relocation with an opportunistic refactor inside it is unreviewable. +4. **The boundary is the default, not the exception.** Loopback binding and a required + token are what the server does unless told otherwise (R15.4–15.5). +5. **Enforcement lives in the cheap job.** All guards run in `lint`, add < 30 s, and print + the fix in the failure message (R20.5, R9.5). +6. **Honesty is the differentiator.** `SECURITY.md`, the "Honest gaps" paragraph in + `HARNESS.md`, the `fail_under` history comment, and `performance_report.md`'s admission + that its numbers were meaningless are the most senior-looking artifacts in the repo. + This spec extends that voice; it never trims it. + +### 1.3 What this deliberately does not change + +The failure taxonomy and its IDs. The `Trace` boundary and schema. Check IDs and tier +definitions. Backend orchestration behaviour. Agent prompts and personas. The journal. +Route paths, response shapes, and `data-testid` values. + +--- + +## 2. The wire-or-delete decision table (R1) + +Five modules, ~800 LOC, imported only by their own tests. Each needs a recorded decision +before any of them is touched. The recommendation column is the audit's reading; the +decision is the owner's. + +| Module | LOC | Evidence | Recommended | **Decision (2026-09-13)** | Why | +| --- | --- | --- | --- | --- | --- | +| `harness/lessons_loop.py` | 161 | Only `tests/unit/harness/test_lessons.py`. Named `closed-loop` in `HARNESS.md`; `SELF_IMPROVEMENT.md` says lessons write through it. Its name appears as an ablation *string* in `evals/arms/base.py`. | **wire** | **wire** | Closed-loop is the most valuable harness-table claim; `ResultsBundle` is the default-path hook. | +| `harness/session_loop.py` | 315 | Only its own test. Phase 8 of `harness-alignment`, explicitly "off by default until its ablation shows it earns its place". | **dormant** | **dormant** | Behind `AI_TEAM_SESSION_LOOP=1` until an ablation earns a default-on place. | +| `harness/qa_verdicts.py` | 75 | Only a test. The *reader* side exists and is wired (`evals/trace/parsers.py` → `parse_qa_verdicts_jsonl`, `evals/checks/verification.py`), but nothing writes `docs/qa_verdicts.jsonl`. | **wire** | **wire** | Without a producer, `CHK-evaluator-capitulation` can only return `inconclusive`. | +| `harness/verifiers.py` | 25 | Only `tests/unit/harness/test_router.py`. Cited in the `enforced` Verification row. | **wire or downgrade** | **wire** | Cheap-path routing is called from `run_app_smoke` on the default path; smoke itself remains the enforcement. | +| `evals/ladder_report.py` | 228 | Only its own test. | **dormant** | **dormant** | Activated by `python -m evals.cli ladder report`; not on a default run. | + +The general rule that comes out of this table, and that R9.2 encodes: **a dormant module +is one you can turn on in one documented step.** Anything else is an orphan wearing a +plan. + +--- + +## 3. Machine-checkable status definitions (R2.2) + +`HARNESS.md`'s three status words become testable predicates: + +| Status | Predicate | Test shape | +| --- | --- | --- | +| `instrumented` | The named module is imported on a default run path. | Reachability graph (R9.1) contains it from an entry point. | +| `enforced` | Bypassing the layer makes the default path fail or block. | A test disables/stubs the layer and asserts the run raises, blocks, or records a guardrail failure. | +| `closed-loop` | Output of run *n* changes input of run *n+1*. | A test writes an artifact in run 1 and asserts run 2 loads it and behaves differently. | + +Applying these to the current table predicts three downgrades and two fixes. That is the +expected, healthy outcome — the table has never been checked, so its first check should +move rows. R2.3's rule matters more than the specific rows: **a downgrade is not a +regression; a false claim is.** + +--- + +## 4. Package contract (R11) + +Target layout after R10. The only structural change is that CrewAI's implementation moves +next to the other two backends. + +``` +src/ai_team/ +├── core/ protocols, results, run identity, spend guard, team profile +├── config/ settings, models, routes, cost estimation, token tracking +├── harness/ the product: acceptance, context, receipts, routing, journal, lessons +├── tools/ ToolBus + tool implementations (file, code, git, test, smoke) +├── guardrails/ behavioral, security, quality (facade __init__ only — R11.3) +├── memory/ long-term store +├── monitor.py thread-safe event collector +├── backends/ +│ ├── registry.py +│ ├── crewai_backend/ ← + agents/ crews/ tasks/ flows/ (R10.1) +│ ├── langgraph_backend/ +│ └── claude_agent_sdk_backend/ +└── ui/ web server (routers — R13) + artifacts service + frontend +``` + +**Import rules** (enforced by R10.5): + +| Package | May import | May never import | +| --- | --- | --- | +| `core/` | stdlib, pydantic | anything in `ai_team` except `config` | +| `config/` | `core` | `backends`, `harness`, `ui`, `tools` | +| `harness/` | `core`, `config`, `tools`, `guardrails` | `backends`, `ui` (already its docstring rule) | +| `tools/`, `guardrails/`, `memory/` | `core`, `config`, `harness` | `backends`, `ui` | +| `backends/*` | everything above, and only its own subtree | another backend's subtree | +| `ui/` | everything above | another backend's internals | +| `evals/` | `core`, `harness`, `config` (contracts only) | `backends` (R10.2) | + +**After Track B (R11.4–11.5):** `utils/` is dissolved. Owners: `tools/coverage_paths.py`, +`config/demo_input.py`, `backends/comparison.py`, `backends/crewai_backend/callbacks.py`, +`backends/crewai_backend/llm_wrapper.py`. `reasoning.py` was unreachable and deleted. +`models/` remains — it owns shared domain types (`requirements`, `architecture`, +`development`, `qa_models`, `comparison_report`). `models/outputs.py` was a single-consumer +LLM schema dump used only by the artifacts service; that consumer now normalizes dicts +and the file is gone. + +--- + +## 5. Documentation model (R3, R4) + +### 5.1 Four kinds of document, one rule each + +| Kind | Files | Rule | +| --- | --- | --- | +| **Entry** | `README.md` | Accurate index, verified by test (R5.2). One screen to the interesting part. | +| **Reference** | `ARCHITECTURE`, `HARNESS`, `GUARDRAILS`, `SECURITY`, `EVAL_METHODOLOGY`, `MODELS`, `TEAM_PROFILES`, `GETTING_STARTED`, `AGENTS` | Describes the system as it is **now**. Every path in it resolves (R4.2). One subject, one file. | +| **Record** | `docs/journal/**`, `docs/posts/**`, `docs/troubleshooting/**`, `docs/eval-runs/**` | Append-only, dated, never edited except for broken links. This is the repo's strongest asset and its voice. | +| **Plan** | `.kiro/specs/**` | The only place aspirations live. Roadmaps in `docs/` are plans in the wrong folder. | + +`EVALS_ROADMAP.md` (68 KB, self-labelled aspirational) is a Plan sitting in Reference +space; `COMPARISON_RESULTS.md` (512 lines, "historical run diary") is a Record duplicating +the journal; `performance_report.md` is a Reference whose content is an apology for not +existing. Each resolves by moving to its correct kind or leaving. + +### 5.2 Workspace-artifact notation (R4.2) + +Docs legitimately name files that exist only inside a generated run workspace +(`run.json`, `ACCEPTANCE.json`, `logs/costs.jsonl`, `docs/qa_verdicts.jsonl`, `calc.py`). +The reference checker cannot distinguish those from stale source paths — in the audit they +were ~200 of ~300 raw hits, which is why the number looked alarming and mostly was not. + +Convention: **workspace artifacts are written with a `/` prefix** +(`` `/logs/costs.jsonl` ``). The checker skips any path starting with +`/`, and flags every other unresolvable path. One convention, applied once, +turns a noisy check into a precise one. + +### 5.3 What "outdated" means here + +Not age. `docs/journal/2026-06-24.md` is three months old and perfectly current as a +record of 24 June. A document is outdated when it (a) describes behaviour the code no +longer has, (b) points at files that no longer exist, or (c) states a plan that a spec has +since superseded. Age is only evidence. + +### 5.4 Data artifacts (R21) + +Three categories, one rule each. The rule is stated in the README of the directory that holds +the files, so the classification travels with the data. + +| Category | Meaning | Examples | Rule | +| --- | --- | --- | --- | +| **Authored** | A human wrote it as an input | `evals/scenarios/*.json`, `config/agents.yaml`, `evals/corpora/guardrails/thresholds.yaml` | Reviewed like code | +| **Generated** | A named command produces it | reports, `docs/benchmark_results.json` (absent), coverage output | Committed only with its regeneration command stated; otherwise gitignored | +| **Accepted** | A human blessed a measurement | `evals/baselines/tier_a.json` | Carries date, git sha, and a `reason` that explains the numbers | + +The audit's result by category: + +- **Authored: clean.** 94 fixture traces, 20 distinct check ids, **zero orphans in either + direction**, plus 12 consistently named backend coverage fixtures. `tests/fixtures/` is + likewise fully referenced. This is the best-maintained data in the repo and §9's guard exists + to keep it that way. +- **Accepted: stale.** `tier_a.json` is a month old, pinned to a pre-harness-alignment sha, + covers 13 of 20 registered checks, and records all thirteen as `fail` with + `suite_pass_pow_k: 0.0`. For a corpus deliberately built from fail/pass pairs that may be + correct — but the file says nothing about it, and it is the file that defines what the + flagship $0 gate accepts. +- **Unclassified: 7 MB of images.** 25 of 38 are referenced by nothing. Roughly half are + external publication assets (`substack-*`, `hero-linkedin.*`) that have a purpose the + repository never states; the rest are run screenshots from July whose documents have moved on. + +`evals/golden/.validation_log.jsonl` fits none of the three: a hidden, tracked, zero-length +append log. It is also the shape of the recorded golden-file test-isolation defect — a tracked +file that tests append to. + +--- + +## 6. Complexity ratchets (R12) + +Three counters, stored in one file, enforced by one test: + +```toml +# tests/unit/repo/ratchets.toml — see R20.3; values dated and sourced +functions_over_80_lines = 63 # 2026-09-13 baseline +functions_over_150_lines = 8 +functions_over_8_params = 17 +mypy_ignore_errors_modules = 16 +docker_image_mb = 0 # set by task 8.4 +``` + +**Exemption (R12.6):** a function is exempt when its branch count is ≤ 2 and it is a flat +declarative builder — `evals/cli.py:build_parser` (215 lines, 0 branches) is the canonical +case. Long ≠ complex; the ratchet targets `_cmd_run` (259 lines, **71 branches**, 16 +params), not a parser definition. + +`_cmd_run`'s 16 parameters are the deeper smell: the CLI signature and the run signature +are the same function. R12.4 splits them with a `RunOptions` dataclass built from +`argparse.Namespace`, which also makes the run path callable from a test without +constructing a CLI invocation. + +--- + +## 7. Control-plane security (R15, R16) + +### 7.1 Auth + +```python +# ui/web/auth.py +def require_token(x_ai_team_token: str = Header(default="")) -> None: + expected = get_settings().web.token # AI_TEAM_WEB_TOKEN + if not expected: # loopback-only mode (R15.4) + return + if not secrets.compare_digest(x_ai_team_token, expected): + raise HTTPException(401, "Invalid or missing token") +``` + +One dependency, applied to every router in R13's split except `/api/health`. Three rules +make it safe by default rather than by configuration: + +1. `--host` defaults to `127.0.0.1` (R15.5). +2. A non-loopback bind without `AI_TEAM_WEB_TOKEN` is **refused at startup**, not warned + about (R15.4 sets the warning for loopback; the refusal is what makes 15.5 real). +3. WebSockets validate the token at handshake **and** check `Origin` — CORS middleware + never sees a WebSocket upgrade, which is the mistake this rule exists to prevent + (R15.7). + +Local development keeps its single command: loopback, no token, warning in the log. + +### 7.2 Path containment + +```python +def _abs_path_for_rel(project_id, root, rel_path) -> Path: + if ".." in rel_path or rel_path.startswith("/"): # existing, keep + raise ValueError("Invalid path") + if _is_sensitive(rel_path): # existing, keep as defence-in-depth + raise ValueError("Sensitive path not allowed") + base = _tree_root(project_id, root).resolve() + resolved = (base / rel_path).resolve() + if not resolved.is_relative_to(base): # NEW — the actual boundary + raise ValueError("Path escapes project root") + return resolved +``` + +The existing checks are string-shaped and run *before* the join, so they cannot see a +symlink. Agents write into this workspace; a symlink is a file an agent can create. The +post-resolve assertion is the boundary, and the string checks become what they always +were — defence in depth (R16.3). + +`project_id` is already validated (`resolve_run_workspace_dir` rejects `..`, `/`, `\`), +which is why this is a containment gap rather than an open traversal. Worth stating +precisely in the commit message: over-claiming a finding is its own credibility problem. + +--- + +## 8. Container and compose (R18, R19) + +| Defect | Fix | +| --- | --- | +| No Node stage; image serves no UI despite its title | Add `FROM node:22-slim AS frontend` → `npm ci && npm run build` → copy `dist/` into runtime; or retitle to API-only | +| `register_frontend()` returns silently when `dist/` is absent | `logger.warning` naming the expected path | +| `build-essential` in the **runtime** stage | Builder only; `git` justified by agent tooling or removed | +| `.dockerignore` misses `node_modules`, `dist`, `workspace`, `logs`, `.coverage-data` | Add them — `COPY src ./src` currently carries ~230 MB of frontend deps | +| Healthcheck probes `/api/backends` | Probe `/api/health` | +| `build: .` with compose in `docker/` | `context: ..`, `dockerfile: docker/Dockerfile` | +| Ollama service + NVIDIA GPU reservation in the default file | Remove, or move behind `profiles: [local-llm]` | +| No resource limits, no restart policy | Add both | + +The CI image job (R18.6) asserts four properties: non-root user, no `gcc` on `PATH`, +image size under the ratchet, and `/api/health` → 200 in a started container. Those four +are what a reviewer would check by hand, which is the point of automating them. + +--- + +## 9. Enforcement suite (R9, R20) + +New package `tests/unit/repo/` — eleven assertions across eight test modules: + +| Test | Asserts | Fails with | +| --- | --- | --- | +| `test_reachability.py` | No orphan modules; dormant allowlist accurate | `harness/foo.py is imported only by tests. Wire it, add it to DORMANT_MODULES with its flag, or delete it.` | +| `test_references.py` | Relative links and source paths resolve (§5.2 convention) | `docs/X.md:42 → ../campaign/EVAL_GATE_STATUS.md does not exist` | +| `test_readme_structure.py` | README tree ≡ real packages | `README structure omits: harness, models, utils` | +| `test_collection.py` | No `test_*.py` outside `testpaths` | `evals/backends/test_crewai_eval.py is never collected` | +| `test_type_budget.py` | mypy `ignore_errors` module count ≤ ratchet | `ignore_errors covers 17 modules; budget is 16` | +| `test_complexity.py` | Three complexity ratchets | `main.py:_cmd_run is 259 lines (budget: functions >150 lines = 8, now 9)` | +| `test_import_direction.py` | §4 import rules | `evals/checks/trajectory.py imports ai_team.flows; evals may not import backends` | +| `test_data_artifacts.py` | Fixture↔check mapping both ways; baseline covers every registered check; no unreferenced image outside the publication allowlist | `CHK-foo has no fixture`; `baseline covers 13 of 20 checks`; `docs/images/dashboard.png is referenced by nothing` | + +Plus, outside `tests/unit/repo/`, the behaviour tests for what this spec changes: +`tests/unit/ui/test_auth.py` (route coverage + bind rules, R15) and +`tests/unit/ui/test_artifacts_adversarial.py` (symlink containment, R16). + +Total budget: < 30 s (R20.5), measured once at task 8.6. All the repo guards read files and +parse ASTs; none import the application. + +**The bar for all of them is R24**, and two of its clauses carry most of the weight: + +- *Fail on an empty match set (R24.4).* A glob that stops matching after a refactor reports + success. This is the default failure mode of repo-hygiene tests and the reason most such + suites quietly stop working. +- *Ship a recorded way to watch it fail (R24.3).* Either a negative fixture or a one-line + mutation in the docstring. A guard nobody has seen fail is a guard nobody knows works. + +--- + +## 10. Migration order and risk + +Track A is phases 1–5, 9 and 8; Track B is 6 and 7 (see `tasks.md`, "Two tracks"). The risk +table below is ordered by phase number, which is the dependency order — not the order to do +them in. + +| Phase | Content | Risk | Reversible | +| --- | --- | --- | --- | +| 1 Delete | orphans, uncollected tests, empty dirs, dead docs, ~7 MB of unreferenced images | **Low** | trivially | +| 2 Truth | harness table, README, links, docs consolidation, Tier A baseline refresh | **Low** | trivially | +| 3 Gates | frontend CI, type budget, collection check | **Low** — may go red immediately, which is the point | yes | +| 4 Security | token, bind default, path containment, SECURITY.md | **Medium** — changes the dev command's default host | yes | +| 5 Container | Node stage, dockerignore, compose context | **Medium** — image build changes | yes | +| 6 Boundaries | move 6,772 LOC under `crewai_backend/` | **High** — largest diff in the repo | via shims | +| 7 Complexity | three worst functions, web router split | **Medium-High** — touches the run path and every route | per-commit | +| 8 Guards | enforcement suite + ratchets | **Low** | yes | +| 9 Evidence | one real benchmark, operating envelope, retention policy | **Low** — additive; the only risk is publishing a number that invites scrutiny, which is the point | yes | + +**Rationale.** Deletion first so nothing later is spent on code that leaves. Truth second +because it is nearly free and it is what a reviewer reads first. Gates third so every +later phase is protected by them. Security before the big moves because it is the highest +consequence and the smallest diff. The two large refactors (6, 7) come last, behind +compatibility shims and a green suite, and are independently revertable. + +**On Track B.** Phases 6 and 7 are the two largest diffs in the repository — 6,772 LOC moved +and 24 routes re-homed — and a reviewer reading for thirty minutes will not notice either. They +are in the spec because the package tree currently tells a false story and a 259-line, +71-branch function is a real maintenance cost. They are *not* in the spec because they improve +the showcase. Schedule them on the maintainer's budget, not the portfolio's, and note that +R10.3 was deliberately relaxed: this repository has no external consumers, so deprecation shims +for the moved packages are optional and the default is a hard move. + +**The single highest-value hour in this spec is Phase 2, task 2.1** — making the harness +status table true. It is the table a reviewer reads to decide whether the project is +serious, and it currently contains two claims the code does not support. + +--- + +## 11. Testing strategy + +| Layer | Covers | Command | +| --- | --- | --- | +| Lint / format | style | `uv run ruff check . && uv run ruff format --check .` | +| Types | `src/`, `evals/` minus the shrinking ignore list | `uv run mypy src/ && uv run mypy evals/` | +| Repo guards (new) | R9/R20 | `uv run pytest tests/unit/repo` | +| Unit (1,456 functions) | behaviour | `uv run pytest tests/unit` | +| Frontend (110 cases, **new to CI**) | components, utils | `npm run lint && npm test` | +| Web E2E (Playwright) | the app still works — the control for phases 6 and 7 | `uv run pytest tests/e2e/web -m web_e2e` | +| Image (new) | R18.6 | `docker build` + assertions | +| Tier A evals | $0 replay gate | `uv run python -m evals.cli run --tier A` | + +The Playwright suite is what makes the two large refactors safe: it selects by +`data-testid` and asserts behaviour, so if it stays green while 6,772 LOC move and 24 +routes are re-homed, the changes were structural only. + +--- + +## 12. Open decisions + +Recorded 2026-09-13. No unanswered row remains. + +| # | Decision | Default if unanswered | Status | +| --- | --- | --- | --- | +| 12.1 | `lessons_loop` — wire it, or downgrade the `closed-loop` claim? | **Wire** (§2). The claim is worth more than the week. | **Done** — wired through `ResultsBundle` | +| 12.2 | `session_loop` — dormant behind a flag, or delete until its ablation runs? | **Dormant behind `AI_TEAM_SESSION_LOOP`** | **Done** — dormant | +| 12.3 | Container serves the UI, or is honestly API-only? | **Serves the UI** — add the Node stage; the dashboard is a large part of what the project demonstrates | **Done** — Node stage in `docker/Dockerfile` | +| 12.4 | `.archive/` — delete from the tree, or keep with a stated retention rule? | **Delete**; git history is the archive | **Keep** — owner override 2026-09-13; `.archive/README.md` states retention and why each tracked survivor stays | +| 12.5 | `COMPARISON_RESULTS.md` — merge into the journal, or keep as a distinct diary? | **Keep, with a purpose line** (R3.2); merging 512 lines of dated records loses their order | **Done** — kept | +| 12.6 | Build backend after dropping Poetry tables — `hatchling` or `uv_build`? | **hatchling** (boring, widely understood, reads PEP 621 directly) | **Done** — hatchling | +| 12.7 | Move `agents/crews/tasks/flows` with shims, or hard-move and update imports? | **Shims for one release** with a removal date in the docstring | **Done** — shims warn until 2026-12-31 (v0.3.0) | +| 12.8 | Web auth — shared token, or skip auth and document loopback-only? | **Shared token**; "it's local-only" is the sentence every incident report starts with | **Done** — `AI_TEAM_WEB_TOKEN` | +| 12.9 | R22 benchmark — spend real money on one live multi-backend run, or publish mock-labelled numbers only? | **One live run, budget-capped.** A labelled mock benchmark is honest but answers nothing; this is the only place in the spec where spending is worth arguing for | **Pending human spend** (task 9.1, $25). Envelope documented; mock numbers not published | +| 12.10 | Track B (phases 6–7) — do it, or record the layout problem in `ARCHITECTURE.md` and move on? | **Record and defer** unless the repo keeps being developed | **Done** — Phases 6 and 7 landed on `feature/phase-harness-alignment` | + +Decisions already closed by the requirements, recorded so they are not reopened: no +behaviour change to the harness or taxonomy (constraints); deletion beats archiving for +unreachable code (constraints); route paths and `data-testid`s stay stable (R13.2); no +task in this spec spends money (constraints). diff --git a/.kiro/specs/production-hardening/requirements.md b/.kiro/specs/production-hardening/requirements.md new file mode 100644 index 00000000..d4271d96 --- /dev/null +++ b/.kiro/specs/production-hardening/requirements.md @@ -0,0 +1,1014 @@ +# Requirements — Production Hardening + +**Spec ID:** `production-hardening` +**Status:** Draft for implementation +**Owner:** Rick Zakharov +**Target repo:** `ai-team` (whole repo: `src/`, `evals/`, `tests/`, `docs/`, `docker/`, CI) +**Created:** 2026-09-13 +**Related:** [`../ui-refinement/`](../ui-refinement/) covers the web frontend's design system +and accessibility; this spec does not restate it. [`../eval-harness/`](../eval-harness/) and +[`../harness-alignment/`](../harness-alignment/) define the eval and arm machinery whose +*wiring* this spec audits. + +--- + +## Introduction + +This repository is the portfolio artifact. Its claim is not "a multi-agent demo" — it is +*"I build production-ready, enterprise-grade, performant, secure, scalable systems."* A +reviewer with thirty minutes will test that claim in a specific order: skim the README, +open the Dockerfile, look for auth on the API, check whether CI actually gates anything, +then grep for the module a doc just named. This spec fixes what that reviewer finds. + +The codebase is genuinely strong in the places most projects are weak: a real failure +taxonomy bound to deterministic checks, a $0 replay gate, subprocess isolation with hard +kills, per-run spend guards, adversarial guardrail tests, an honest `SECURITY.md` threat +model, 1,456 test functions, and an engineering journal that records corrections rather +than hiding them. None of that is in question here. + +What the audit found is a consistent, fixable pattern: **the repository's claims have +drifted ahead of its wiring.** Five specific gaps, each of which a reviewer can find in +minutes: + +**1. Code that ships but is not reachable (~800 LOC).** `harness/lessons_loop.py` (161), +`harness/session_loop.py` (315), `harness/verifiers.py` (25), `harness/qa_verdicts.py` +(75), and `evals/ladder_report.py` (228) are imported by **nothing but their own unit +tests**. `docs/HARNESS.md`'s status table nevertheless marks Feedback **closed-loop** +citing `lessons_loop.py`, and Verification **enforced** citing `verifiers.py`; +`docs/SELF_IMPROVEMENT.md` states lessons "now write through +`src/ai_team/harness/lessons_loop.py`". The harness-alignment spec deliberately ships +Phases 8–10 off by default — that is sound — but "off by default" currently means +*unreferenced*, with no flag that could turn it on, while two documents say it is live. + +**2. Gates that do not gate.** `testpaths = ["tests"]`, so `evals/backends/test_*_eval.py` +and `evals/test_backend_comparison.py` — **808 LOC of tests** — are never collected, and +are additionally `ignore_errors = true` under mypy. CI runs `npm ci && npm run build` for +the frontend but never `npm test` or `npm run lint`, so **110 frontend test cases and the +ESLint config never execute in CI**. `mypy` "passes" while `ignore_errors = true` covers +two of three backends, `agents.*`, `crews.*`, `flows.main_flow`, `tools.test_tools`, +`tools.smoke_tools`, `evals.run_evals`, and more — roughly half the Python by volume. + +**3. A layout that tells the wrong story.** `agents/` (1,327), `crews/` (1,388), +`tasks/` (1,071), and `flows/` (2,986) — **6,772 LOC** — sit at the top level of +`src/ai_team/` as though they were the shared domain model. They are reachable only +through `backends/crewai_backend/backend.py`. The other two backends are self-contained +subtrees. Meanwhile the README's "Project structure" section omits `agents/`, `crews/`, +`tasks/`, `flows/`, `harness/`, `models/`, `reports/`, and `utils/` entirely — including +`harness/`, which the same README calls "the real deliverable". + +**4. A control plane with no controls.** `src/ai_team/ui/web/server.py` (1,791 LOC, 24 +routes) has **no authentication on any route** and defaults to `--host 0.0.0.0`. Those +routes start paid LLM runs (`/ws/run`, `POST /api/demo`), resume human-gated runs, delete +workspaces (`DELETE /api/runs/{run_id}`), and stream arbitrary workspace files +(`GET /api/projects/{id}/file`). `SECURITY.md` models agent-layer threats carefully and +does not mention the control plane at all. The container story has the same shape: the +Dockerfile is titled "web UI (FastAPI + React)" but has **no Node build stage**, so +`register_frontend()` silently returns and the container serves no UI; the runtime stage +installs `build-essential`; `.dockerignore` does not exclude `node_modules`, so +`COPY src ./src` drags ~230 MB of frontend dependencies into both images; the healthcheck +probes `/api/backends` rather than `/api/health`; and `docker-compose.yml` still builds +against an Ollama service with NVIDIA GPU reservations from an era the docs have moved on +from — with a `build: .` context that cannot see `pyproject.toml`. + +**5. Committed data nobody can classify.** 195 tracked JSON/JSONL/YAML files, and the one +that matters most is stale: `evals/baselines/tier_a.json` — the accepted baseline that defines +what the $0 gate passes — was accepted `2026-08-16` against a pre-harness-alignment sha, covers +**13 of 20 registered checks**, and marks every one `"fail"` with `suite_pass_pow_k: 0.0`, +explaining none of it. Alongside it, **25 of 38 tracked images (~7 MB of the 9 MB in +`docs/images/`)** are referenced by nothing, and `evals/golden/.validation_log.jsonl` is a +tracked, hidden, zero-length append log. The counterweight is the fixture corpus: 94 traces, 20 +check ids, **zero orphans in either direction** — the best-maintained data here, and the reason +R21 guards it rather than touching it. + +### The principle this spec enforces + +> **Nothing ships that is not reachable, and nothing is claimed that is not checked.** + +Every requirement below either makes a claim true, makes a gate real, or deletes something +that is neither. + +### What this spec adds + +| Area | Gap today | Requirement | +| --- | --- | --- | +| Unreachable modules | ~800 LOC imported only by their own tests | R1 | +| Harness status claims | `HARNESS.md` marks unwired layers *enforced* / *closed-loop* | R2 | +| Doc sprawl | 42 markdown files, three overlapping narrative streams, 68 KB aspirational roadmap | R3 | +| Reference rot | ~65 dead links in `PROMPT_TRACKING.md`; ~20 doc→source paths that no longer exist | R4 | +| README accuracy | Structure omits 8 packages incl. `harness/`; taxonomy cited as FM-001…010 (now …017) | R5 | +| Uncollected tests | 808 LOC never run by pytest | R6 | +| Frontend gates | 110 vitest cases + ESLint never run in CI | R7 | +| Type coverage | `ignore_errors` over ~half the Python | R8 | +| Drift | Nothing prevents the next orphan module or dead doc link | R9 | +| Package boundaries | 6,772 LOC of CrewAI-private code at top level | R10 | +| Core contract | No stated rule for what `core/` and `harness/` may own | R11 | +| Complexity | 63 functions > 80 lines; worst is 259 lines / 71 branches / 16 params | R12 | +| Web module size | One 1,791-line module holds all 24 routes | R13 | +| Dependency source | PEP 621 **and** Poetry tables, hand-synced; author is "Your Name" | R14 | +| Control-plane auth | No authentication; binds `0.0.0.0` by default | R15 | +| Path containment | No post-resolve boundary assertion on artifact reads | R16 | +| Threat model | `SECURITY.md` omits the API surface entirely | R17 | +| Container image | No frontend build; `build-essential` in runtime; `node_modules` shipped | R18 | +| Compose topology | Ollama + GPU leftovers; broken build context | R19 | +| Enforcement | Conventions live in prose | R20 | +| Data artifacts | Stale/partial Tier A baseline; 25 unreferenced images (~7 MB); a tracked empty log | R21 | +| Performance & scale | Two of the four claimed properties have no evidence and no stated limits | R22 | +| Repo surface | Actions pinned by mutable tag; no PR/issue template, CODEOWNERS, or CHANGELOG | R23 | +| Test quality | ~12 new test modules, and no stated bar for the tests themselves | R24 | + +### Design constraints (decided) + +| Constraint | Decision | +| --- | --- | +| Behaviour | No change to run semantics, the failure taxonomy, check IDs, or trace schema. This spec moves, deletes, wires, and gates — it does not redesign the harness. | +| Backward compatibility | `ai-team` and `ai-team-web` console entry points, CLI flags, REST/WS paths, and `data-testid` values stay stable. Python import paths **may** change (R10) — that is the point — but only behind a deprecation shim for one release. | +| Deletion bias | When code is unreachable, the default is **delete**, not comment out or archive. Git history is the archive. | +| Wiring bias | When a module is deliberately dormant (R1's "off by default" set), it must be reachable behind an explicit, documented, tested flag — not left unreferenced. | +| Spend | Every task runs offline **except task 9.1**, the published benchmark, which needs one real multi-backend run. That run SHALL be human-triggered, budget-capped at **$25**, and recorded with its receipt. No other task may add a live model call. | +| Scope boundary | Frontend design-system and accessibility work belongs to [`../ui-refinement/`](../ui-refinement/) and is not duplicated here. R7 (running its tests in CI) is the only overlap. | +| Verification | `uv run ruff check . && uv run ruff format --check . && uv run mypy src/ && uv run mypy evals/ && uv run pytest tests/unit` green after every task; full CI green at each phase boundary. | + +### Non-goals + +- Rewriting any backend, changing orchestration behaviour, or altering agent prompts. +- New features, new pages, new metrics, new failure modes. +- Multi-tenancy, RBAC, SSO, or a hosted deployment. R15 adds a single-operator auth + boundary, not an identity system. +- Kubernetes manifests, Helm charts, or a cloud deployment story. +- Raising the coverage floor as a goal in itself. R8 raises *type* coverage; the + `fail_under = 60` line-coverage ratchet is left to its existing process. +- **Building scale.** R22 requires measuring and stating the envelope; it explicitly forbids + implementing a persistent run store, a queue, or a worker pool for a single-operator tool. +- Rewriting the engineering journal. It is an append-only record and stays as written + (R3.5). + +### Glossary + +| Term | Definition | +| --- | --- | +| **Reachable** | Imported, directly or transitively, from a console entry point, a CLI, the web app, or a check registry — not merely from a test. | +| **Dormant** | Deliberately not on the default path, but reachable behind a documented flag and exercised by at least one test that sets that flag. | +| **Orphan** | A module that is neither reachable nor dormant. R1 deletes orphans. | +| **Control plane** | The FastAPI app in `src/ai_team/ui/web/server.py` and everything it can start, stop, read, or delete. | +| **Claim** | A statement in a doc about what the system does. R2/R4/R5 require every claim to be checkable. | +| **Ratchet** | A numeric floor stored in config that a test enforces and that may only move in the improving direction. | + +--- + +## Requirements + +### R1 — Every shipped module is reachable or dormant-by-flag + +**User story:** As a reviewer grepping for the module a doc just named, I want to find it +wired into something. + +**Acceptance criteria** + +1.1 NO MODULE under `src/ai_team/` or `evals/` (excluding `evals/arms/vendor/`, which is a +pinned third-party copy) SHALL be imported exclusively by its own tests. + +1.2 FOR EACH of `harness/lessons_loop.py`, `harness/session_loop.py`, +`harness/verifiers.py`, `harness/qa_verdicts.py`, and `evals/ladder_report.py`, THE TEAM +SHALL make an explicit, recorded decision: **wire**, **dormant**, or **delete**. + +1.3 WHERE the decision is **wire**, the module SHALL be reachable from a default run, +CLI command, or registry, and at least one test SHALL exercise it through that path rather +than by direct import. + +1.4 WHERE the decision is **dormant**, THE SYSTEM SHALL expose a single documented +environment flag or CLI option that activates it, the flag SHALL appear in +`.env.example` and the configuration reference, and a test SHALL exercise the module +**through the flag**, asserting both the on and off paths. + +1.5 WHERE the decision is **delete**, the module and its tests SHALL be removed in one +commit whose message names the superseding mechanism, if any. + +1.6 THE `ablation component` STRING LIST in `evals/arms/base.py` SHALL name only +components that are reachable or dormant per 1.3/1.4; a component that names an orphan +SHALL be removed from the list. + +1.7 `src/ai_team/rag/`, `src/ai_team/optimizers/`, `tests/unit/rag/`, and any other +directory containing no tracked files SHALL be removed. + +--- + +### R2 — The harness status table is checked, not asserted + +**User story:** As a reader of `HARNESS.md`, I want the word "enforced" to mean something +a test verified this morning. + +**Acceptance criteria** + +2.1 EACH ROW of the `docs/HARNESS.md` status table SHALL name a module path, and a test +SHALL assert that every named path exists and is reachable per R1.1. + +2.2 THE STATUS VALUES `instrumented`, `enforced`, and `closed-loop` SHALL each have a +machine-checkable definition recorded in `design.md` §3, and a test SHALL assert the +claimed status against it: +- `instrumented` — the module is imported on a default run path; +- `enforced` — a test demonstrates the default path *fails or blocks* when the layer is + bypassed; +- `closed-loop` — a test demonstrates output from one run changing input to the next. + +2.3 WHERE a row cannot meet its claimed status, the row SHALL be downgraded in the same +commit that discovers it. Downgrading is not a regression; a false claim is. + +2.4 `docs/SELF_IMPROVEMENT.md`'s statement that lessons write through +`harness/lessons_loop.py` SHALL be made true (R1.3) or corrected (R1.4/R1.5). + +2.5 THE TABLE'S "Honest gaps" PARAGRAPH SHALL be retained. It is the most credible +paragraph in the document and its style SHALL be extended to the rest of the docs, not +removed. + +--- + +### R3 — One home per subject + +**User story:** As a reader, I want one document per topic and a clear reason for every +file that survives. + +**Acceptance criteria** + +3.1 THE DOCUMENTATION SET SHALL be reduced to documents that are current, distinct, and +linked from `README.md` or from another surviving document. Today there are 42 markdown +files under `docs/`, of which the README indexes 16. + +3.2 THE FOLLOWING SHALL be resolved explicitly: + +| Document | Issue | Required outcome | +| --- | --- | --- | +| `docs/EVALS_ROADMAP.md` (68 KB) | Self-declared "aspirational", scoped 2026-07, superseded by two Kiro specs | Reduce to a short backlog that links the specs, or move to `.archive/` | +| `docs/prompts/PROMPT_TRACKING.md` | ~65 links to `phase-*.md` files that do not exist | Rewrite as a short provenance note, or archive | +| `docs/performance_report.md` | 22 lines whose entire content explains that the document is not real | Delete; the explanation belongs in the journal | +| `docs/COMPARISON_RESULTS.md` (512 lines) | "Historical run diary" overlapping `docs/journal/` | Merge into the journal or state its distinct purpose in its first line | +| `docs/RESULTS.md` | Named "Results" but documents bundle layout | Rename to reflect content (e.g. `RUN_ARTIFACTS.md`) | +| `docs/EVALS.md` vs `docs/EVAL_METHODOLOGY.md` vs `evals/README.md` | Three entry points to one subject | Keep one entry point; the others link to it | +| `.archive/` (21 tracked files) | Removed demos and superseded plans kept in-tree | Delete from the working tree — git history is the archive — or state the retention rule in `.archive/README.md` | + +3.3 EVERY SURVIVING DOCUMENT SHALL open with one sentence stating what it is for and, for +any document containing measurements, the date and conditions of those measurements. + +3.4 NO DOCUMENT SHALL present numbers from a mock or demo run without labelling them as +such in the same visual block as the numbers. *(Review item — judged by a human at the phase +gate, not by a test. It is listed as a criterion because it is verifiable by reading, not +because it is automatable.)* + +3.5 `docs/journal/` IS APPEND-ONLY and SHALL NOT be edited by this spec, except to fix +broken relative links (R4.2). Its value is that it was written at the time. + +--- + +### R4 — Reference integrity + +**User story:** As a reader following a path in a doc, I want the file to be there. + +**Acceptance criteria** + +4.1 EVERY RELATIVE MARKDOWN LINK in a tracked `.md` file SHALL resolve to a tracked file +or directory. The audit found ~70 that do not, concentrated in +`docs/prompts/PROMPT_TRACKING.md` (~65), plus +`docs/EVAL_METHODOLOGY.md` and `docs/posts/harness-map.md` +(→ `campaign/EVAL_GATE_STATUS.md`), `docs/prompts/PROMPTS.md` +(→ `.archive/phase-7-agentcore-deployment.md`), and three journal `handoff-*.md` links. + +4.2 EVERY INLINE SOURCE PATH in a tracked `.md` file SHALL resolve to a tracked file, +**except** paths that name artifacts produced inside a run workspace +(`run.json`, `state.json`, `ACCEPTANCE.json`, `logs/*.jsonl`, `docs/qa_verdicts.jsonl`, +generated `calc.py`, and similar). Workspace artifacts SHALL be written in a visually +distinct form so a checker can tell them apart — `design.md` §5 specifies the convention. + +4.3 THE FOLLOWING referenced-but-absent paths SHALL be fixed or removed: +`evals/evidence.py`, `evals/trajectory.py`, `evals/fixtures.py` (now a package), +`docs/security_report.md`, `docs/WEB_DASHBOARD.md`, `docs/SELF_IMPROVEMENT_AUDIT.md`, +`docs/SELF_IMPROVEMENT_DESIGN.md`, `memory/knowledge_base.py`, +`scripts/setup_openrouter.sh`, `scripts/setup_ollama.sh`, `scripts/test_models.py`, +`scripts/extract_lessons.py`, `scripts/show_metrics.py`. + +4.4 A LINK CHECK SHALL run in CI over tracked markdown and SHALL fail the build on an +unresolvable relative link (R20). + +4.5 EXTERNAL (http) LINKS ARE OUT OF SCOPE for the CI gate — they fail for reasons the +repo does not control. + +--- + +### R5 — The README is an accurate index + +**User story:** As a first-time reader, I want the first page to match the repository. + +**Acceptance criteria** + +5.1 THE "Project structure" TREE SHALL list every top-level package under +`src/ai_team/`. It currently omits `agents/`, `crews/`, `tasks/`, `flows/`, `harness/`, +`models/`, `reports/`, and `utils/` — including `harness/`, which the same README +identifies as the real deliverable. + +5.2 THE TREE SHALL be generated or verified by a test, so it cannot drift again (R20). + +5.3 THE README SHALL cite the taxonomy by its current range. It says "FM-001…010"; the +taxonomy is at 1.2.0 with FM-001…017. + +5.4 THE README'S DESCRIPTION of `evals/backends/` as "Live backend eval clients (pytest)" +SHALL be corrected or removed in line with R6. + +5.5 THE DOCUMENTATION TABLE SHALL list every surviving document (R3.1), or state that it +is a curated subset and where the full index lives. + +5.6 THE README SHALL state, in one line near the top, what a reader should look at first +to judge the engineering — the harness, the taxonomy, and the $0 gate — because a +portfolio artifact that does not direct attention wastes the reviewer's first two minutes. + +--- + +### R6 — Tests that exist, run + +**User story:** As a maintainer, I want the test count on the badge to mean something. + +**Acceptance criteria** + +6.1 `evals/backends/test_claude_sdk_eval.py` (175), `evals/backends/test_crewai_eval.py` +(169), `evals/backends/test_langgraph_eval.py` (191), and +`evals/test_backend_comparison.py` (273) — **808 LOC never collected** because +`testpaths = ["tests"]` — SHALL be either moved under `tests/` and made to pass, or +deleted. + +6.2 WHERE they are moved, they SHALL carry an existing marker (`integration`, `real_llm`, +or `slow`) so the default suite stays offline and free, and they SHALL be removed from the +mypy `ignore_errors` list. + +6.3 NO TEST FILE SHALL live outside `testpaths` after this spec, and a check SHALL assert +it (R20). + +6.4 THE TWO TEST-ISOLATION DEFECTS already recorded in project memory (golden-file writes; +the order-dependent workspace test) SHALL be confirmed fixed or ticketed here with a task. + +6.5 THE 35 `pytest.skip` CALL SITES SHALL be reviewed; each SHALL skip on a *missing +precondition* (no API key, service unreachable), never to route around a failing +assertion. + +--- + +### R7 — Frontend quality gates run in CI + +**User story:** As a maintainer, I want the frontend's 110 tests to be able to fail the +build. + +**Acceptance criteria** + +7.1 THE CI WEB JOB SHALL run `npm run lint` and `npm test` in addition to +`npm run build`. It currently runs only `npm ci && npm run build`. + +7.2 THE FRONTEND TEST STEP SHALL fail the job on a failing test; it SHALL NOT be +`continue-on-error`. + +7.3 FRONTEND COVERAGE MAY be reported but SHALL NOT gate in this spec. + +7.4 THE `ui-refinement` SPEC'S DRIFT TEST, once it exists, runs inside `npm test` and +needs no separate CI step. + +--- + +### R8 — Type checking covers the code it claims to + +**User story:** As a reviewer, I want "mypy passes" to be a statement about the codebase, +not about a shrinking subset of it. + +**Acceptance criteria** + +8.1 THE `ignore_errors = true` OVERRIDES in `pyproject.toml` — currently covering +`ai_team.backends.langgraph_backend.*`, `ai_team.backends.claude_agent_sdk_backend.*`, +`ai_team.agents.*`, `ai_team.crews.*`, `ai_team.flows.main_flow`, +`ai_team.memory.memory_config`, `ai_team.tools.git_tools`, `ai_team.tools.test_tools`, +`ai_team.tools.smoke_tools`, `ai_team.utils.callbacks`, `evals.fixtures*`, +`evals.metrics`, `evals.run_evals`, `evals.backends.*`, and +`evals.test_backend_comparison` — SHALL be recorded as a **countable debt** with a floor +that only moves down. + +8.2 THE SYSTEM SHALL enforce the debt as a **line count, not a module count**: a test that +sums the LOC of every module matched by an `ignore_errors` override and fails when the sum +exceeds the recorded floor. Counting modules is gameable by splitting a file and treats a +30-line module as equal to a 1,440-line one — `flows/main_flow.py` alone is 1,440 of the +untyped lines. + +8.3 THIS SPEC SHALL retire at least the overrides whose modules it already touches: +`evals.backends.*` and `evals.test_backend_comparison` (removed or moved by R6), +`ai_team.tools.smoke_tools` and `ai_team.tools.test_tools` (named *enforced* in the +harness table, R2). + +8.4 EACH REMAINING OVERRIDE SHALL carry an inline comment naming the reason and the +condition under which it is removed. An override with no stated exit is not permitted. + +8.5 NO NEW MODULE SHALL be added to the `ignore_errors` list without an accompanying +entry in the same commit's message explaining why the floor moved the wrong way. + +--- + +### R9 — Drift guards for reachability and references + +**User story:** As the maintainer, I want the next orphan module to fail CI, not to be +discovered in a review six months later. + +**Acceptance criteria** + +9.1 THE SYSTEM SHALL add a reachability test that builds the import graph over +`src/ai_team/` and `evals/` (vendored code excluded) and fails on any module that is +neither reachable from an entry point nor listed in an explicit, commented +`DORMANT_MODULES` allowlist. + +9.2 THE ALLOWLIST IS THE DOCUMENTATION of every deliberately dormant module and SHALL +name the flag that activates each (R1.4). + +9.3 THE SYSTEM SHALL add a link/reference checker per R4.4. + +9.4 THE SYSTEM SHALL add a test asserting the README structure tree matches the real +package layout (R5.2). + +9.5 EVERY GUARD SHALL fail with a message naming the file and the fix. A guard whose +output is `expected 3 to be 0` will be skipped by the next person in a hurry. + +9.6 THE GUARDS SHALL run in the existing `lint` CI job, not a new one. + +--- + +### R10 — CrewAI-private code lives under its backend + +**User story:** As a reader of the package tree, I want the layout to tell me the truth +about what is shared and what belongs to one engine. + +**Acceptance criteria** + +10.1 `src/ai_team/agents/` (1,327 LOC), `crews/` (1,388), `tasks/` (1,071), and `flows/` +(2,986) — **6,772 LOC reachable only through `backends/crewai_backend/backend.py`** — +SHALL move under `src/ai_team/backends/crewai_backend/`, matching the self-contained shape +of the other two backends. + +10.2 THE ONE EXTERNAL CONSUMER, `evals/checks/trajectory.py`, imports `ai_team.flows` for +phase names. That coupling SHALL be replaced by a backend-neutral source (a constant in +`core/`, or the taxonomy), so a check never imports a backend. + +10.3 IMPORT-PATH COMPATIBILITY: this repository has **no external consumers** — nothing +outside it imports `ai_team.crews`. Shim modules are therefore **optional**, and the default +is a hard move with imports updated in the same commit. Add shims only if an external +consumer is known to exist; a deprecation cycle for an audience of zero is ceremony, and a +reviewer reads it as cargo-culting rather than as care. + +10.4 AFTER THE MOVE, the three backend subtrees SHALL be symmetric: each owns its agents, +its orchestration, and its adapters, and imports shared contracts only from `core/`, +`harness/`, `config/`, `tools/`, and `guardrails/`. + +10.5 A TEST SHALL assert that no module under `core/`, `harness/`, `config/`, or `evals/` +imports from `ai_team.backends.*` (the dependency direction rule). + +10.6 THE MOVE SHALL be a pure relocation: no behaviour change, no signature change, no +opportunistic refactor in the same commit. + +--- + +### R11 — Stated contract for the shared core + +**User story:** As a contributor adding a feature, I want to know which package it goes in +without reading all fifteen. + +**Acceptance criteria** + +11.1 `design.md` §4 SHALL state, in one table, what each surviving top-level package owns +and what it may import. The table SHALL fit on one screen — if it does not, the package +structure is the problem, not the table. *(Review item.)* + +11.2 `harness/__init__.py`'s EXISTING DOCSTRING RULE — "This module MUST NOT import +backends or crews" — SHALL be generalized into that table and enforced by the test in +R10.5. + +11.3 `src/ai_team/guardrails/__init__.py` (572 LOC of logic in a package `__init__`) SHALL +move its implementation into named modules, leaving the `__init__` as a facade of imports +and `__all__`. + +11.4 `src/ai_team/utils/` SHALL be dissolved or justified: a package named "utils" states +that its contents had no home. Each member (`reasoning.py` 348, `backend_comparison.py`, +`callbacks.py`, `llm_wrapper.py`, `demo_input.py`, `coverage_paths.py`) SHALL move to the +package that owns its subject or be deleted if unreachable (R1). + +11.5 THE `models/` PACKAGE (939 LOC, of which `outputs.py` at 459 is imported only by +`ui/artifacts/service.py`) SHALL be reviewed against the same rule. + +--- + +### R12 — Complexity budget + +**User story:** As a reviewer opening a random file, I want to be able to read a function +without scrolling twice. + +**Acceptance criteria** + +12.1 THE AUDIT BASELINE IS: **63 functions over 80 lines**, 8 over 150, 17 with more than +8 parameters. The worst are `src/ai_team/main.py:_cmd_run` (259 lines, 71 branches, 16 +parameters), `backends/claude_agent_sdk_backend/tools/mcp_server.py:build_ai_team_mcp_tools` +(264 lines), `evals/gate.py:evaluate_gate` (230 lines, 38 branches), +`ui/web/server.py:_execute_run` (183 lines), and `flows/main_flow.py:_kickoff_impl` +(154 lines). + +12.2 THE SYSTEM SHALL record these counts as ratchets in a config file and SHALL add a +test that fails when any count rises. **Branch count is the primary axis**; line count is +secondary. A ratchet on length alone rewards splitting a function in half at an arbitrary +point, which moves the number without improving anything — `_cmd_run`'s 71 branches are the +defect, its 259 lines are the symptom. + +12.3 THIS SPEC SHALL reduce the three worst offenders below 120 lines each by extraction, +not by reformatting: `_cmd_run`, `build_ai_team_mcp_tools`, `evaluate_gate`. + +12.4 `_cmd_run`'s 16 PARAMETERS SHALL be replaced by a single typed options object built +from the parsed arguments, so the CLI surface and the run surface stop being the same +signature. + +12.5 EXTRACTED FUNCTIONS SHALL be covered by the existing tests for their parent; where +the parent had no test, one SHALL be added before the extraction. + +12.6 THE RATCHET SHALL NOT be applied to `evals/cli.py:build_parser` (215 lines, 0 +branches) or other flat declarative builders; `design.md` §6 SHALL define the exemption +narrowly. + +--- + +### R13 — The web API is decomposed + +**User story:** As a contributor fixing one endpoint, I do not want to open a +1,791-line module. + +**Acceptance criteria** + +13.1 `src/ai_team/ui/web/server.py` (1,791 LOC, 24 routes) SHALL be split into FastAPI +routers by resource: health/catalog, runs, comparisons, projects/artifacts, websockets — +plus an app factory. + +13.2 EVERY ROUTE PATH, METHOD, RESPONSE SHAPE, AND STATUS CODE SHALL be unchanged. The +Playwright suite (`tests/e2e/web`) and the frontend API client are the proof. + +13.3 THE SPLIT SHALL NOT introduce a new abstraction layer — no service/repository +indirection, no DI container. Routers plus the existing modules are enough. + +13.4 THE MODULE-LEVEL SIDE EFFECTS (the module-scope `app`, the CORS middleware, the +`_FRONTEND_REGISTERED` global) SHALL move into an app factory so tests can build an app +without import-time state. + +13.5 THE COST-RESOLUTION FALLBACK CHAIN (`_resolve_token_estimate` tries the monitor, then +the receipt, then live spend) SHALL be consolidated into one documented resolver with a +stated precedence order, since it currently reads as three competing sources of truth for +the number the dashboard shows. + +--- + +### R14 — One dependency source, real project metadata + +**User story:** As someone installing this, I want one answer to "what does it depend on". + +**Acceptance criteria** + +14.1 `pyproject.toml` SHALL declare dependencies **once**. It currently carries a PEP 621 +`[project.dependencies]` list **and** a parallel `[tool.poetry.dependencies]` list that +must be hand-synced, while the build backend is `poetry-core` and the documented workflow +is `uv sync`. + +14.2 THE SURVIVING DECLARATION SHALL be PEP 621 `[project]`, with a build backend that +reads it. The Poetry tables SHALL be removed. + +14.3 `[project]` SHALL carry real `authors` metadata. The Poetry table currently reads +`"Your Name "` in a repository whose purpose is attribution. + +14.4 THE PINNING COMMENTS that explain non-obvious constraints (`litellm == 1.74.9`, +the `aiohttp` CVE floor, the `pip >= 26.2` constraint) SHALL be preserved verbatim in the +surviving declaration. They are evidence of real dependency work and are worth more than +the lines they occupy. + +14.5 `uv sync --frozen` AND `uv build` SHALL both succeed after the change, and CI SHALL +prove it. + +--- + +### R15 — The control plane has a control + +**User story:** As anyone who runs this on a laptop on a shared network, I want the API +that spends money to require a credential. + +**Acceptance criteria** + +15.1 EVERY MUTATING ROUTE SHALL require authentication: `POST /api/estimate`, +`POST /api/demo`, `POST /api/runs/{id}/resume`, `POST /api/runs/{id}/cancel`, +`DELETE /api/runs/{id}`, and the `/ws/run` WebSocket. + +15.2 READ ROUTES that expose workspace contents — `/api/projects/{id}/tree`, +`/api/projects/{id}/file`, `/api/projects/{id}/download.zip`, `/api/runs*` — SHALL require +the same credential. + +15.3 THE MECHANISM SHALL be a single shared token supplied by environment variable +(`AI_TEAM_WEB_TOKEN`), checked by one FastAPI dependency, compared with +`secrets.compare_digest`. This is a single-operator boundary, not an identity system +(non-goal). + +15.3a THE SYSTEM SHALL add a **route-coverage test** that enumerates `app.routes` at import +time and asserts every route is either listed in an explicit public allowlist (`/api/health`, +the SPA catch-all, static assets) or carries the auth dependency. Asserting that today's 24 +routes return 401 checks this spec's work; the route-coverage test is what catches the +twenty-fifth route somebody adds next year. The allowlist is the documentation of every +deliberately public endpoint. + +15.4 WHERE `AI_TEAM_WEB_TOKEN` IS UNSET, THE SERVER SHALL bind loopback only and SHALL log +a prominent warning at startup stating that the control plane is unauthenticated. It SHALL +NOT bind a non-loopback interface without a token. + +15.5 THE DEFAULT `--host` SHALL become `127.0.0.1`. Binding `0.0.0.0` SHALL require both +an explicit flag and a configured token. + +15.6 THE FRONTEND SHALL send the token when present; the development flow SHALL remain +one command with no token required, because it binds loopback (15.4). + +15.7 THE WEBSOCKET ENDPOINTS SHALL validate the token at handshake and SHALL validate +`Origin` against the configured CORS allowlist, since CORS does not apply to WebSockets. + +15.8 RATE LIMITING IS OUT OF SCOPE, but `design.md` §7 SHALL record it as the next +control. + +--- + +### R16 — Artifact reads cannot escape the workspace + +**User story:** As an operator, I want a symlink written by an agent to be unable to read +my home directory through the dashboard. + +**Acceptance criteria** + +16.1 `_abs_path_for_rel` in `src/ai_team/ui/artifacts/service.py` SHALL assert containment +**after** resolution: `resolved.is_relative_to(base)` (or `os.path.commonpath`), raising +otherwise. Today it checks `".." in rel_path` and a leading `/` before joining, then +returns `(base / rel_path).resolve()` with no post-resolve assertion — so a symlink inside +the workspace, which agents can create, resolves outside the boundary. + +16.2 DIRECTORY WALKS (`_list_relative_files`, `build_tree`, `workspace_zip_bytes`) SHALL +not follow symlinks out of the base, and SHALL skip symlinked entries whose target is +outside it. + +16.3 THE SENSITIVE-PATH FILTER (`_is_sensitive`, a substring denylist) SHALL be documented +as defence-in-depth rather than as the boundary, and the boundary SHALL be 16.1. + +16.4 ADVERSARIAL TESTS SHALL cover: a symlink to `/etc/passwd`, a symlink to a sibling +run's workspace, a nested symlinked directory, and an absolute-path `project_id`. This +matches the project's existing rule that every tool ships with adversarial tests. + +16.5 THE ZIP DOWNLOAD SHALL apply the same containment rules as the file read, so the two +paths cannot diverge. + +--- + +### R17 — The threat model covers the control plane + +**User story:** As a security reviewer, I want the document to model the system I can +reach over the network. + +**Acceptance criteria** + +17.1 `SECURITY.md` SHALL add control-plane rows to its threat table: unauthenticated run +start (cost), unauthenticated deletion (destructive), workspace read exposure +(confidentiality), and WebSocket origin spoofing. + +17.2 EACH NEW ROW SHALL name its control per R15/R16, in the same +threat/vector/control format already used. + +17.3 `SECURITY.md` SHALL state the intended deployment posture in one line — a +single-operator tool bound to loopback unless a token is configured — so that a reader +knows what is and is not claimed. + +17.4 THE EXISTING AGENT-LAYER CONTENT SHALL be preserved. It is the strongest document in +the repository. + +17.5 THE DOCUMENT SHALL state the vulnerability-reporting channel and the supported +version window, since a repository that claims enterprise readiness is expected to have +one. + +--- + +### R18 — The container is what it says it is + +**User story:** As a reviewer running `docker build`, I want the image to contain the +application it advertises and nothing else. + +**Acceptance criteria** + +18.1 THE DOCKERFILE IS TITLED "web UI (FastAPI + React)" BUT HAS NO NODE BUILD STAGE, so +`frontend/dist/index.html` never exists in the image and `register_frontend()` silently +returns — the container serves the API with no UI. THE SYSTEM SHALL either add a Node +build stage that produces `dist/` and copies it into the runtime image, or retitle the +image and document it as API-only. + +18.2 `register_frontend()` SHALL log a warning when `dist/index.html` is missing instead +of returning silently, so this failure is visible in container logs. + +18.3 THE RUNTIME STAGE SHALL NOT install `build-essential`. A compiler toolchain in a +production image is attack surface and image weight; it belongs in the builder stage only. +`git` SHALL be justified (the agents use it) or removed. + +18.4 `.dockerignore` SHALL exclude `node_modules`, `dist`, `workspace`, `logs`, and +`.coverage-data`. Today `COPY src ./src` carries the frontend's `node_modules` +(~230 MB locally) into both stages. + +18.5 THE HEALTHCHECK SHALL probe `/api/health`, which exists for that purpose, rather than +`/api/backends`. + +18.6 THE IMAGE SHALL declare a non-root `USER` (it does) and SHALL be verified by a CI +step that builds the image and asserts: non-root user, no `gcc` on `PATH`, image size +below a recorded ratchet, and `/api/health` returning 200 in a started container. + +18.7 THE IMAGE SHALL NOT bake secrets; `.env` is already excluded and SHALL remain so. + +--- + +### R19 — Compose describes the current system + +**User story:** As someone running `docker compose up`, I want it to work and to reflect +how the project actually runs. + +**Acceptance criteria** + +19.1 `docker/docker-compose.yml` USES `build: .` WITH THE COMPOSE FILE IN `docker/`, so +the build context cannot see `pyproject.toml`, `uv.lock`, or `src/`. THE CONTEXT SHALL be +corrected (`context: ..`, `dockerfile: docker/Dockerfile`) and a CI step SHALL prove the +compose build succeeds. + +19.2 THE `ollama` SERVICE with NVIDIA GPU reservations SHALL be removed or moved behind an +optional profile. The documented provider path is OpenRouter/Anthropic, the Ollama setup +scripts the docs reference no longer exist (R4.3), and a GPU reservation in the default +compose file will fail on most reviewers' machines. + +19.3 THE APP SERVICE SHALL set `AI_TEAM_WEB_TOKEN` (R15) and SHALL NOT publish a port +without it. + +19.4 THE APP SERVICE SHALL declare resource limits and `restart` policy, since the claim +under test is operational maturity. + +19.5 VOLUME MOUNTS SHALL cover `workspace/` as well as `output/`, or the compose run SHALL +document that run workspaces are ephemeral. + +--- + +### R20 — Everything here is enforced by a check + +**User story:** As the maintainer, I want this spec to be the last time these things are +found by a human. + +**Acceptance criteria** + +20.1 EACH OF THE FOLLOWING SHALL have an automated check that fails the build: +reachability (R9.1), dormant-module allowlist (R9.2), reference integrity (R9.3), README +structure (R9.4), tests-outside-testpaths (R6.3), mypy-ignore budget (R8.2), complexity +ratchets (R12.2), dependency-direction rule (R10.5), data artifacts and image references +(R21.2, R21.4, R21.8), image properties (R18.6), compose build (R19.1). + +20.2 THE CHECKS SHALL live beside the code they check (`tests/unit/repo/`) and SHALL run +in the existing `lint` and `test` CI jobs. + +20.3 EVERY RATCHET VALUE SHALL be stored in one file with a comment recording the date and +the measurement that set it, in the style of the existing `fail_under` history block in +`pyproject.toml` — which is a model of how to record a ratchet and SHALL be cited as the +precedent. + +20.4 A RATCHET MAY ONLY MOVE in the improving direction; a commit that loosens one SHALL +state why in its message. + +20.5 THE FULL CHECK SUITE SHALL add no more than 30 seconds to CI, measured once at task +8.6 and recorded in `BASELINE.md`. A guard that makes the build slow is a guard that gets +disabled. *(Measured once, not asserted per-run — a wall-clock assertion in CI is itself a +flaky test.)* + +--- + +### R21 — Data and generated artifacts + +**User story:** As a reviewer, I want every committed data file to be either an input the +system reads, a baseline someone accepted, or evidence — and to be able to tell which. + +**Context.** The audit's first pass covered markdown, Python, and the frontend; it did not +cover the 195 tracked JSON/JSONL/YAML files, 38 images, or the accepted baselines. A second +pass found the eval fixture corpus in excellent condition and three problems elsewhere. + +**Acceptance criteria** + +21.1 **The Tier A baseline is stale and partial.** `evals/baselines/tier_a.json` was accepted +`2026-08-16` against git sha `f534cee` — before the harness-alignment work landed — records +outcomes for **13 checks while 20 are registered**, marks every one of those 13 `"fail"`, and +carries `suite_pass_pow_k: 0.0`. + +THE TEAM SHALL **first diagnose, then decide**: run `python -m evals.cli run --tier A` against +the current corpus and determine whether an aggregate per-check `"fail"` is the correct encoding +for a corpus deliberately built from fail/pass fixture pairs, or a defect. Only then SHALL the +baseline be regenerated (`baseline accept --reason …`) or left as-is with an explanatory +`reason`. + +THIS ORDER IS MANDATORY. This audit did not run the suite; it read the file. Regenerating a +baseline whose semantics nobody has confirmed replaces one unexplained reference state with +another, in the single file that defines what the $0 gate accepts. + +21.2 A TEST SHALL assert that the baseline covers every registered check id, so a new check +cannot be added without either a baseline entry or an explicit exemption. + +21.3 **25 of 38 tracked images (~7 MB of the repo's 9 MB `docs/images/`) are referenced by no +tracked file**, including `dashboard.png` (808 KB), `ai-team-work-sample-{1,2}.png` (1.4 MB), +twelve `compare-2026-07-03-*` screenshots, `compare-verified-2026-07-06.gif` (676 KB), and the +`substack-*` / `hero-linkedin.*` publication assets. EACH SHALL be either referenced from a +document, moved under a clearly named directory with a `README.md` stating it is an external +publication asset, or deleted. + +21.4 A DRIFT GUARD SHALL fail on an image under `docs/images/` that no tracked file references +and that is not listed in the publication-assets allowlist (R9, R20). + +21.5 `evals/golden/.validation_log.jsonl` IS TRACKED, HIDDEN, AND EMPTY (0 lines) — an +append-only log committed at zero length. It SHALL be untracked and gitignored, or its purpose +and write path SHALL be documented in `evals/golden/README.md`. A tracked append log that tests +write to is also how the recorded golden-file test-isolation defect (R6.4) reproduces. + +21.6 EVERY TRACKED DATA FILE SHALL be classifiable as **authored** (an input a human wrote: +scenarios, agent/task YAML, thresholds), **generated** (produced by a named command), or +**accepted** (a baseline a human blessed, with date and sha). THE CLASSIFICATION SHALL be +stated in the README of the directory that holds it. Directories needing one today: +`evals/fixtures/traces/`, `evals/baselines/`, `evals/golden/`, `tests/fixtures/`, +`evals/corpora/`, `docs/images/`. + +21.7 NO GENERATED FILE SHALL be committed without a stated regeneration command. +`docs/benchmark_results.json` — referenced by `docs/performance_report.md` and absent from the +tree — is the existing example of the confusion this rule prevents (R3.2 deletes that document; +this rule stops the next one). + +21.8 **The eval fixture corpus SHALL be protected as-is.** The audit found 94 fixture traces +covering 20 distinct check ids with **zero orphan fixtures and zero checks lacking a fixture**, +plus 12 consistently named backend coverage fixtures. This is the best-maintained data in the +repository. A guard SHALL assert both directions of that mapping so it stays true. + +21.9 THE VERSION FILES SHALL state what they version. `evals/VERSION` reads `0.1.0` while +`evals/taxonomy/failure_modes.yaml` is at `1.2.0` with 17 failure modes; a reader cannot tell +whether these track the same thing. + +21.10 `.cursor/rules/crewai-agents-crews-flows.mdc` AND `.cursor/rules/crewai-tools.mdc` +describe paths that R10 moves. THEY SHALL be updated in the same phase as the move, since a +stale agent rule silently teaches the wrong layout to every future contributor — human or not. + +--- + +### R22 — Performance and scale envelope + +**User story:** As a reviewer testing the claim "performant and scalable", I want a number and +a stated limit, not an adjective. + +**Context.** The repository's purpose sentence claims four properties. Production-readiness and +security have requirements above. **Performance and scale had none** — this spec's own gap, and +the one a technically strong reviewer probes hardest, because it is where most portfolio +projects are silent. Today: `tests/performance/test_benchmarks.py` holds eight benchmark tests +that publish nothing; the only performance document is being deleted (R3.2) because its numbers +came from a mock-LLM run; and `ui/web/server.py` holds run state in a **process-local dict wiped +on every restart** (`RunState.runs`, whose own docstring says so — the Compare tab's +reattachment machinery exists to work around it). + +The fix is not to build scale. It is to **measure once and state the envelope honestly** — which +is the same move that makes `SECURITY.md` and the "Honest gaps" paragraph credible. + +**Acceptance criteria** + +22.1 THE SYSTEM SHALL publish one real benchmark with real numbers, generated by a recorded +command, stating hardware, model, date, and sample size. THE RUN SHALL be human-triggered and +capped at **$25** (design §12.9), and SHALL reuse the existing spend guard rather than a new +one. A benchmark that cannot be reproduced +from the repository SHALL NOT be published. + +22.2 THE BENCHMARK SHALL measure what the harness controls and what a reader would actually ask: +wall-clock and cost per phase, tokens per run, and the overhead the harness adds over a bare +backend call — not a synthetic microbenchmark. + +22.3 WHERE the benchmark is produced by a mock or demo run, it SHALL say so in the same block as +the numbers, and SHALL state what it does and does not measure (R3.4). The deleted +`performance_report.md` is the precedent: its admission that mock timings "read like performance +data but measured nothing" is the standard to meet. + +22.4 THE SYSTEM SHALL document the **operating envelope** in one place: single process; run +registry in memory and lost on restart; concurrent-run ceiling and where it comes from; +per-run workspace disk growth; the spend ceiling; and the wall-clock kill. + +22.5 THE ENVELOPE DOCUMENT SHALL state what would have to change to lift each limit — a +persistent run store, a queue, a worker pool — **without implementing any of it**. Naming the +next architectural step is evidence of judgement; building it for a single-operator tool is +evidence of the opposite. + +22.6 THE SYSTEM SHALL define a **retention policy** for `workspace/` and `output/`, which grow +without bound (437 run directories, 28 MB, on the audit machine), and SHALL provide the command +that enforces it. + +22.7 `tests/performance/` SHALL either produce the published artifact of 22.1 or be removed. +Eight benchmark tests that publish nothing are the same defect as a module nothing imports +(R1). + +22.8 NO SCALABILITY CLAIM SHALL appear in the README that 22.4 does not support. If the envelope +is "one operator, one machine, runs are ephemeral", the README says that — a precise small claim +reads as senior; a vague large one reads as marketing. + +--- + +### R23 — Repository surface and supply chain + +**User story:** As a reviewer, I want the repository's own metadata to show the same care as its +code. + +**Context.** These are 30-minute items with disproportionate payoff, because they are the +conventions a reviewer checks reflexively before reading any code. + +**Acceptance criteria** + +23.1 EVERY GITHUB ACTION SHALL be pinned to a commit SHA, not a floating tag. All 27 `uses:` +lines across both workflows currently pin `@v4` / `@v5`, which is a mutable reference — the +standard supply-chain finding for a repository that ships a `security` CI job and a threat +model. + +23.2 THE REPOSITORY SHALL provide a pull-request template and at least one issue template. +Neither exists. + +23.3 THE REPOSITORY SHALL provide `CODEOWNERS`, even with a single owner. It states who is +accountable, which is the point. + +23.4 THE PROJECT SHALL carry a `CHANGELOG.md` and at least one tagged release. `version` has +been `0.1.0` since the first commit while the project has passed through three orchestration +backends and two spec cycles. + +23.5 EVERY BADGE in the README SHALL reflect a check that runs. The CI badge is accurate; any +badge added later SHALL meet the same bar. + +23.6 THE SECURITY CI JOB SHALL publish its `pip-audit` output as an artifact, so "the security +job passed" is inspectable rather than asserted. + +23.7 AN SBOM IS OPTIONAL. WHERE one is produced, it SHALL be generated in CI from the lockfile, +never committed by hand. + +--- + +### R24 — Test design bar for everything this spec adds + +**User story:** As the maintainer, I do not want ten new guard modules to become the next +source of flaky, order-dependent tests. + +**Context.** This spec adds roughly a dozen test modules — eight repo guards, adversarial +artifact tests, auth tests, container assertions. They parse files, walk the tree, and read +config, which is precisely the shape that produced this repository's two existing +test-isolation defects (golden-file writes; an order-dependent workspace test). A guard suite +that is itself unreliable is worse than no guard suite, because it trains the team to re-run +CI until it goes green. + +**Acceptance criteria** + +24.1 EVERY TEST this spec adds SHALL be **offline, deterministic, and order-independent**: no +network, no model call, no dependence on the working directory, no dependence on another +test having run. The suite SHALL pass under a shuffled run. + +24.2 NO TEST this spec adds SHALL write to a tracked file. The existing `tests/conftest.py` +hash guard over `evals/golden/`, `evals/fixtures/traces/`, and `evals/taxonomy/` already +asserts this for those paths; the new tests SHALL respect it rather than extend the ignore +list. + +24.3 EVERY GUARD SHALL ship with a demonstration that it can fail. Either a negative fixture +committed alongside it, or a one-line mutation recorded in the test's docstring +(`to see this fail: add padding: 0.35rem to App.css`). A guard nobody has watched fail is a +guard nobody knows works. + +24.4 EVERY GUARD THAT SCANS A SET SHALL fail when the set is empty. A glob that matches +nothing, a registry that imports nothing, or an AST walk over zero files SHALL raise rather +than report success — the most common silent failure in repo-hygiene tests is a path that +stopped matching after a refactor. + +24.5 NO NEW PYTEST MARKER SHALL be introduced without an entry in `pyproject.toml`'s `markers` +list. The repo registers twelve markers today and that discipline SHALL hold. + +24.6 EVERY GUARD'S FAILURE MESSAGE SHALL name the offending file, the line where possible, and +the action that fixes it (R9.5). This is restated here because it is the single property that +decides whether a guard survives its first angry morning. + +24.7 THE NEW TESTS SHALL live in `tests/unit/repo/` (repository invariants) or beside the code +they cover (`tests/unit/ui/`, `tests/unit/harness/`), never in a new top-level directory. + +24.8 THE ADDED RUNTIME SHALL be measured once at task 8.6 and recorded, not asserted per-run +(R20.5). + +--- + +## Traceability summary + +| Requirement | Primary files | +| --- | --- | +| R1, R2 | `src/ai_team/harness/*`, `evals/ladder_report.py`, `evals/arms/base.py`, `docs/HARNESS.md`, `docs/SELF_IMPROVEMENT.md` | +| R3, R4, R5 | `docs/**`, `README.md`, `.archive/` | +| R6 | `evals/backends/test_*.py`, `evals/test_backend_comparison.py`, `pyproject.toml` | +| R7 | `.github/workflows/ci.yml` | +| R8 | `pyproject.toml`, `tests/unit/repo/test_type_budget.py` (new) | +| R9, R20 | `tests/unit/repo/*` (new), `.github/workflows/ci.yml` | +| R10, R11 | `src/ai_team/{agents,crews,tasks,flows,utils,models,guardrails}/`, `evals/checks/trajectory.py` | +| R12 | `src/ai_team/main.py`, `backends/claude_agent_sdk_backend/tools/mcp_server.py`, `evals/gate.py` | +| R13 | `src/ai_team/ui/web/server.py` → `ui/web/routers/*` | +| R14 | `pyproject.toml` | +| R15, R17 | `src/ai_team/ui/web/server.py`, `config/settings.py`, `SECURITY.md`, `.env.example` | +| R16 | `src/ai_team/ui/artifacts/service.py`, `tests/unit/ui/test_artifacts_adversarial.py` (new) | +| R18, R19 | `docker/Dockerfile`, `docker/docker-compose.yml`, `.dockerignore` | +| R21 | `evals/baselines/tier_a.json`, `evals/golden/`, `docs/images/`, `evals/VERSION`, `.cursor/rules/*.mdc` | +| R22 | `tests/performance/`, `docs/PERFORMANCE.md` (new), `ui/web/server.py` (`RunState`), `scripts/` (retention) | +| R23 | `.github/workflows/*.yml`, `.github/` templates, `CODEOWNERS`, `CHANGELOG.md` | +| R24 | every test module this spec adds; `tests/conftest.py`; `pyproject.toml` markers | diff --git a/.kiro/specs/production-hardening/screenshots/9.1/README.md b/.kiro/specs/production-hardening/screenshots/9.1/README.md new file mode 100644 index 00000000..22b63838 --- /dev/null +++ b/.kiro/specs/production-hardening/screenshots/9.1/README.md @@ -0,0 +1,14 @@ +# 9.1 in-flight screenshots (2026-09-13) + +Taken while LangGraph smoke `2026-09-13_182650_write-a-single-python-module_01` +was still running. They are **not** a 9.1 baseline. Full analysis: +[`docs/eval-runs/2026-09-13-langgraph-smoke/README.md`](../../../../../docs/eval-runs/2026-09-13-langgraph-smoke/README.md). + +| File | What it shows | +| --- | --- | +| `live-workspace-files.png` | Workspace tree + `calc.py` as written | +| `live-workspace-retry.png` | Same view, labeled behavioral guardrail retry 3/3 | +| `live-tests-scrolled.png` | `test_calc.py` (add/subtract cases) | +| `home-empty-session.png` | Dashboard Home empty — CLI run is not in `GET /api/runs` | + +PNGs may be local-only until committed. The handoff does not depend on them. diff --git a/.kiro/specs/production-hardening/screenshots/9.1/home-empty-session.png b/.kiro/specs/production-hardening/screenshots/9.1/home-empty-session.png new file mode 100644 index 00000000..23b3bfa0 Binary files /dev/null and b/.kiro/specs/production-hardening/screenshots/9.1/home-empty-session.png differ diff --git a/.kiro/specs/production-hardening/screenshots/9.1/live-tests-scrolled.png b/.kiro/specs/production-hardening/screenshots/9.1/live-tests-scrolled.png new file mode 100644 index 00000000..26ea1467 Binary files /dev/null and b/.kiro/specs/production-hardening/screenshots/9.1/live-tests-scrolled.png differ diff --git a/.kiro/specs/production-hardening/screenshots/9.1/live-workspace-files.png b/.kiro/specs/production-hardening/screenshots/9.1/live-workspace-files.png new file mode 100644 index 00000000..54b504cb Binary files /dev/null and b/.kiro/specs/production-hardening/screenshots/9.1/live-workspace-files.png differ diff --git a/.kiro/specs/production-hardening/screenshots/9.1/live-workspace-retry.png b/.kiro/specs/production-hardening/screenshots/9.1/live-workspace-retry.png new file mode 100644 index 00000000..e3e6eb1a Binary files /dev/null and b/.kiro/specs/production-hardening/screenshots/9.1/live-workspace-retry.png differ diff --git a/.kiro/specs/production-hardening/tasks.md b/.kiro/specs/production-hardening/tasks.md new file mode 100644 index 00000000..161aa578 --- /dev/null +++ b/.kiro/specs/production-hardening/tasks.md @@ -0,0 +1,613 @@ +# Implementation Plan — Production Hardening + +**Spec ID:** `production-hardening` +**Requirements:** [`requirements.md`](./requirements.md) · **Design:** [`design.md`](./design.md) + +--- + +## How to execute this plan + +### Two tracks, and they are not the same priority + +The phase numbers are a dependency order, **not a priority order**. For a repository whose +purpose is to be read, the payoff per hour differs by an order of magnitude between phases: + +| Phase | Track | Est. effort | What a reviewer sees | +| --- | --- | --- | --- | +| 1 Delete | **A** | 0.5 day | Nothing dead; ~1,600 LOC and ~7 MB lighter | +| 2 Truth | **A** | 1 day | Docs that match code — the highest-value hour in the spec is 2.1 | +| 3 Gates | **A** | 0.5 day | CI that can actually fail; pinned actions; a release | +| 4 Security | **A** | 1–1.5 days | An API that requires a credential; a threat model that covers it | +| 5 Container | **A** | 0.5 day | `docker build` produces the app it advertises | +| 9 Evidence | **A** | 1 day | Real numbers and a stated envelope for "performant / scalable" | +| 8 Guards | **A−** | 0.5 day | Nothing regresses — invisible to a reviewer, decisive for the maintainer | +| 6 Boundaries | **B** | 2–3 days | A tidier package tree. Near-invisible in a 30-minute read | +| 7 Complexity | **B** | 2–3 days | Shorter functions. Near-invisible in a 30-minute read | + +**Track A is roughly a week and addresses everything a reviewer finds.** Track B is another +week of the largest, riskiest diffs in the repository (6,772 LOC moved; 24 routes re-homed) +for changes that mostly serve the maintainer. Do Track A first, in phase order; treat Track B +as optional and schedule it only if the repository is going to keep being developed. + +Recommended order: **1 → 2 → 3 → 4 → 5 → 9 → 8**, then 6 and 7 if warranted. + +### Coordination with `ui-refinement` + +Task 3.1 turns on `npm test` in CI. The [`ui-refinement`](../ui-refinement/) spec adds a +stylesheet drift test that **fails until its Phases 1–6 land**. Sequence one of: + +- land this spec's 3.1 first, then run `ui-refinement` Phases 1–6 before its Phase 9; or +- land `ui-refinement` Phases 1–9 entirely, then 3.1. + +Turning on the CI step while a red-by-design test exists leaves `main` broken for days, which +is worse than either spec's problem. + +### Rules + +- Work **one task at a time, in order within a phase**. Do not batch phases. +- Every task lists a **Definition of done**. Do not mark it complete until each bullet is + literally true. +- **Gate after every task:** + `uv run ruff check . && uv run ruff format --check . && uv run mypy src/ && uv run mypy evals/ && uv run pytest tests/unit` +- **Gate after every phase** that touches the web layer or the run path: + `uv run pytest tests/e2e/web -m web_e2e` +- **No task spends money except 9.1.** Nothing else here makes a live model call; if a task + seems to need one, it is the wrong task. Task 9.1 (the published benchmark) is the single + exception: human-triggered, capped at $25, receipt recorded. +- One commit per task, one PR per phase, each green on its own so any phase can be + reverted independently. +- Phase 8's guards will fail until Phases 1–7 land. That is expected — do not write them + early and do not weaken them to make them pass. + +Cursor prompt shape: + +``` +Read .kiro/specs/production-hardening/requirements.md and design.md for context. +Implement task 1.3 from .kiro/specs/production-hardening/tasks.md. +Do not start any other task. Do not change run semantics, check IDs, route paths, +or data-testid values. Stop when its Definition of done is satisfied and +`uv run ruff check . && uv run mypy src/ evals/ && uv run pytest tests/unit` passes. +``` + +--- + +## Phase 0 — Baseline + +- [x] **0.1 Record the before state** + - Write `.kiro/specs/production-hardening/BASELINE.md` with: LOC per top-level package; + the five orphan modules and their LOC; the 808 LOC of uncollected tests; the + `ignore_errors` module count (16); complexity counts (63 / 8 / 17); `docs/**.md` file + count (42); count of unresolvable doc references; `docker build` image size; and the + current `pytest tests/unit` pass count. + - **Definition of done:** every number in `requirements.md`'s introduction is reproduced + by a command recorded in `BASELINE.md`, so the spec's own claims are checkable. + - _Requirements: R20.3_ + +- [x] **0.2 Record the wire-or-delete decisions** + - Fill the decision column of `design.md` §2 for all five modules. This is a **human + decision**, not an agent one — the agent records it. + - **Definition of done:** each of the five has `wire`, `dormant`, or `delete` written in + the table with one sentence of rationale. + - _Requirements: R1.2_ + +--- + +## Phase 1 — Deletion + +Nothing in this phase changes behaviour. Every deletion is preceded by a repo-wide search. + +- [x] **1.1 Remove orphan modules marked `delete`** + - For each module marked `delete` in 0.2, remove it and its tests in one commit naming + the superseding mechanism. + - **Definition of done:** `uv run pytest tests/unit` green; no reference to the module + remains in `src/`, `evals/`, `tests/`, or `docs/`. + - _Requirements: R1.5_ + +- [x] **1.2 Resolve the uncollected test files** + - `evals/backends/test_claude_sdk_eval.py` (175), `test_crewai_eval.py` (169), + `test_langgraph_eval.py` (191), `evals/test_backend_comparison.py` (273): move under + `tests/integration/` with an appropriate marker and make them pass, or delete. + - Remove the corresponding `ignore_errors` overrides from `pyproject.toml`. + - **Definition of done:** no `test_*.py` outside `testpaths`; `uv run pytest tests/unit` + and `uv run pytest tests/integration -m "not real_llm"` green; two `ignore_errors` + entries gone. + - _Requirements: R6.1, R6.2, R8.3_ + +- [x] **1.3 Remove empty and orphan directories** + - `src/ai_team/rag/`, `src/ai_team/optimizers/`, `tests/unit/rag/`, `evals/annotations/` + (if it has no `.gitkeep` purpose), and the untracked + `src/ai_team/ui/web/frontend/workspace/` debris. + - **Definition of done:** `find src evals tests -type d -empty` returns only directories + with a documented `.gitkeep` purpose. + - _Requirements: R1.7_ + +- [x] **1.4 Resolve `.archive/`** + - Per decision 12.4: **keep** the 21 tracked files and add + `.archive/README.md` stating the retention rule and why each survivor is kept. + - Fix the two docs linking into it (`docs/prompts/PROMPTS.md`, + `docs/prompts/PROMPT_TRACKING.md`). + - **Definition of done:** no doc links into a non-existent archive path. + - _Requirements: R3.2, R4.1_ + +- [x] **1.5 Resolve unreferenced images and the tracked empty log** + - Triage the 25 unreferenced images (~7 MB): reference, move to a publication-assets + directory with a `README.md` stating their purpose, or delete. + - Untrack and gitignore `evals/golden/.validation_log.jsonl`, or document its purpose and + write path in `evals/golden/README.md`. + - **Definition of done:** every file under `docs/images/` is referenced or listed in the + publication-assets allowlist; no tracked zero-length append log remains. + - _Requirements: R21.3, R21.5_ + +- [x] **1.6 Consolidate the dependency declaration and fix project metadata** + - Remove the `[tool.poetry]` / `[tool.poetry.dependencies]` / `[tool.poetry.group.dev]` + tables; keep PEP 621 `[project]` as the single source; switch the build backend to one + that reads it (design §12.6 defaults to `hatchling`). + - Preserve the pinning comments verbatim — `litellm == 1.74.9`, the `aiohttp` CVE floor, + the `pip >= 26.2` constraint. They are evidence of real dependency work. + - Set real `authors` metadata in `[project]`; the Poetry table currently reads + `"Your Name "` in a repository whose purpose is attribution. + - **Definition of done:** dependencies declared once; `uv sync --frozen` and `uv build` + both succeed; CI proves both; no placeholder metadata remains anywhere in the file. + - _Requirements: R14.1, R14.2, R14.3, R14.4, R14.5_ + +- [x] **1.7 Phase gate** + - **Definition of done:** full CI green; `BASELINE.md` updated with the LOC delta + (expect roughly −1,500 to −2,000) and the repository size delta (expect roughly −7 MB + from images). + - _2026-09-13: CI run 34774054947 success. LOC −655 (not −1,500; Track B added shims). + Tracked `docs/images/` −4.0 MB (not −7; publication assets kept). See `BASELINE.md`._ + - _Requirements: R20.3_ + +--- + +## Phase 2 — Truth + +The cheapest phase and the one a reviewer reads first. + +- [x] **2.1 Make the harness status table true** *(highest-value task in this spec)* + - Apply the §3 predicates to all seven rows of `docs/HARNESS.md`. For each row: verify + the module path exists and is reachable; verify the claimed status against its + predicate; downgrade the row if it cannot be demonstrated. + - Expect downgrades on Feedback (`closed-loop` → until 2.2 wires it) and Verification. + - **Definition of done:** every row's module path resolves; every status word is backed + by a named test; the "Honest gaps" paragraph is retained and extended if the audit + found new ones. + - _Requirements: R2.1, R2.2, R2.3, R2.5_ + +- [x] **2.2 Wire the modules marked `wire`** + - For each: put it on a default path, and add a test that exercises it *through* that + path rather than by direct import. + - `qa_verdicts`: wire the writer so `docs/qa_verdicts.jsonl` is produced — without it + `CHK-evaluator-capitulation` can only return `inconclusive`, which is what the + 2026-09-12 journal records. + - **Definition of done:** each wired module appears in the reachability graph from an + entry point; `HARNESS.md` rows restored to their claimed status with the test named. + - _Requirements: R1.3, R2.4_ + +- [x] **2.3 Make the modules marked `dormant` activatable** + - Add one documented flag per module (`AI_TEAM_SESSION_LOOP`, a `ladder` CLI + subcommand); add each to `.env.example` and the configuration reference; add a test + asserting both the on and off paths. + - **Definition of done:** each dormant module can be activated in one documented step; + `evals/arms/base.py`'s component list names no orphan. + - _Requirements: R1.4, R1.6_ + +- [x] **2.4 Fix the README** + - Add the eight missing packages to the structure tree (`agents`, `crews`, `tasks`, + `flows`, `harness`, `models`, `reports`, `utils` — noting that Phase 6 will move four + of them). Correct `FM-001…010` to the current taxonomy range. Correct or remove the + `evals/backends/` description per 1.2. Add the one-line "read this first" pointer. + - **Definition of done:** the tree matches `ls src/ai_team/`; no claim in the README + contradicts the code. + - _Requirements: R5.1, R5.3, R5.4, R5.6_ + +- [x] **2.5 Adopt the workspace-artifact notation and fix references** + - Apply the `/` prefix convention (design §5.2) across docs; fix the ~70 + unresolvable relative links and the ~20 stale source paths listed in R4.3. + - **Definition of done:** a manual run of the (not yet CI-wired) reference checker + reports zero unresolvable references. + - _Requirements: R4.1, R4.2, R4.3_ + +- [x] **2.6 Consolidate the document set** + - Apply the R3.2 table: `EVALS_ROADMAP.md`, `PROMPT_TRACKING.md`, + `performance_report.md`, `COMPARISON_RESULTS.md`, `RESULTS.md`, and the + EVALS/EVAL_METHODOLOGY/evals-README overlap. + - Add the one-line purpose statement to every surviving document (R3.3); label every + mock/demo number (R3.4). + - Do not edit `docs/journal/` except for broken links. + - **Definition of done:** every surviving doc is linked from the README or another + surviving doc; each opens with its purpose; no doc presents unlabelled mock numbers. + - _Requirements: R3.1, R3.2, R3.3, R3.4, R3.5, R5.5_ + +- [x] **2.7 Refresh and explain the Tier A baseline** + - Regenerate `evals/baselines/tier_a.json` against the current corpus + (`python -m evals.cli baseline accept --reason …`) so it covers all 20 registered checks + rather than 13, and pins a current sha. + - Write a `reason` that explains what a per-check outcome means for a corpus built from + deliberate fail/pass fixture pairs — the current file reads as "every check fails" with no + explanation, and it is the file that defines what the $0 gate accepts. + - Add `README.md` files classifying the contents of `evals/fixtures/traces/`, + `evals/baselines/`, `evals/golden/`, `evals/corpora/`, `tests/fixtures/`, and + `docs/images/` as authored / generated / accepted (design §5.4). + - Reconcile `evals/VERSION` (`0.1.0`) with the taxonomy version (`1.2.0`) or state what each + one versions. + - **Definition of done:** the baseline covers every registered check id; its `reason` explains + the outcome semantics; each named directory has a README stating its category; no reader has + to guess whether a committed file is an input or an output. + - _Requirements: R21.1, R21.6, R21.7, R21.9_ + +- [x] **2.8 Phase gate** + - _Requirements: R20.3_ + +--- + +## Phase 3 — Gates + +- [x] **3.1 Frontend quality gates in CI** + - Add `npm run lint` and `npm test` to the web CI job, before `npm run build`. Not + `continue-on-error`. + - **Definition of done:** CI runs 110 frontend cases; a deliberately broken test fails + the job. + - _Requirements: R7.1, R7.2_ + +- [x] **3.2 Type budget** + - Add `tests/unit/repo/test_type_budget.py` counting modules matched by `ignore_errors`; + record the floor (16 minus whatever 1.2 removed) in `ratchets.toml`; give every + surviving override an inline comment naming its exit condition. + - **Definition of done:** the test passes at the current count and fails when a module is + added; no override lacks a stated exit. + - _Requirements: R8.1, R8.2, R8.4, R8.5_ + +- [x] **3.3 Retire the overrides this spec has earned** + - Remove `ignore_errors` for `ai_team.tools.smoke_tools` and `ai_team.tools.test_tools` + (both cited as *enforced* in the harness table) and fix the resulting type errors. + - **Definition of done:** `uv run mypy src/` green with two fewer overrides; ratchet + lowered in the same commit. + - _Requirements: R8.3_ + +- [x] **3.4 Collection guard** + - `tests/unit/repo/test_collection.py`: no `test_*.py` outside `testpaths`. + - **Definition of done:** passes now; fails if a test file is added under `evals/`. + - _Requirements: R6.3_ + +- [x] **3.5 Test hygiene sweep** + - Confirm the two recorded isolation defects (golden-file writes; order-dependent + workspace test) are fixed; review all 35 `pytest.skip` sites for the + missing-precondition rule. + - **Definition of done:** `uv run pytest tests/unit -p no:randomly` and a shuffled run + both green; every skip names a precondition, not a failure. + - _Requirements: R6.4, R6.5_ + +- [x] **3.6 Repository surface and supply chain** + - Pin all 27 `uses:` lines in both workflows to commit SHAs (keep the version as a trailing + comment); add a PR template and one issue template; add `CODEOWNERS`; add `CHANGELOG.md` + and cut a tagged release; publish the `pip-audit` output as a CI artifact. + - **Definition of done:** no floating action tag remains; `gh release list` shows a tag; + the security job's audit output is downloadable from the run. + - _2026-09-13: Release https://github.com/RickZee/ai-team/releases/tag/v0.2.0 ; + `pip-audit-report` on green Security job of run 34774054947._ + - _Requirements: R23.1, R23.2, R23.3, R23.4, R23.5, R23.6_ + +- [x] **3.7 Phase gate** + - _2026-09-13: Phase 3 complete. Full CI green on run 34774054947 + (https://github.com/RickZee/ai-team/actions/runs/34774054947)._ + +--- + +## Phase 4 — Control plane + +- [x] **4.1 Token authentication** + - Add `AI_TEAM_WEB_TOKEN` to settings and `.env.example`; implement + `ui/web/auth.py::require_token` per design §7.1 using `secrets.compare_digest`; apply + to every mutating route and every workspace-reading route. `/api/health` stays open. + - Add `tests/unit/ui/test_auth.py` covering, per route: 401 without the header, 200 with it, + and 200 without it when no token is configured. + - Add the **route-coverage test** in the same module: enumerate `app.routes` and assert each + is either in the public allowlist (`/api/health`, SPA catch-all, static assets) or carries + the auth dependency. This is the test that catches the twenty-fifth route, not the + twenty-four this task secures. + - **Definition of done:** both tests green; deleting the dependency from any one router makes + the route-coverage test fail by name; `tests/e2e/web` green (it runs loopback with no + token). + - _Requirements: R15.1, R15.2, R15.3, R15.3a_ + +- [x] **4.2 Safe bind defaults** + - Default `--host` to `127.0.0.1`; refuse a non-loopback bind unless a token is + configured; log a startup warning when running unauthenticated on loopback. + - **Definition of done:** `ai-team-web` binds loopback by default; `--host 0.0.0.0` without a + token exits non-zero with a message naming the variable; `tests/unit/ui/test_auth.py` + covers all three cases (default bind, non-loopback without token, non-loopback with token) + by calling the argument parser and the bind guard directly — no socket is opened in a test. + - _Requirements: R15.4, R15.5_ + +- [x] **4.3 WebSocket handshake checks** + - Validate the token and the `Origin` header at `/ws/run` and `/ws/monitor/{run_id}`. + - **Definition of done:** a WS connection from a disallowed origin is rejected at + handshake; a test covers it. (CORS middleware never sees an upgrade — this is the gap + the task exists to close.) + - _Requirements: R15.7_ + +- [x] **4.4 Frontend token support** + - Send the token when configured; no change to the zero-config local flow. + - **Definition of done:** dev server works with no token; a configured token is sent on + every API and WS call; a vitest case asserts the header is attached when configured and + absent when not; frontend tests green. + - _Requirements: R15.6_ + +- [x] **4.5 Artifact path containment** + - Add the post-resolve `is_relative_to(base)` assertion; stop directory walks following + symlinks out of the base; apply the same rules to the ZIP download. + - **Definition of done:** the new adversarial tests (4.6) pass; no existing artifact test + regresses. + - _Requirements: R16.1, R16.2, R16.5_ + +- [x] **4.6 Adversarial artifact tests** + - `tests/unit/ui/test_artifacts_adversarial.py`: symlink to `/etc/passwd`; symlink to a + sibling run's workspace; nested symlinked directory; absolute-path `project_id`; + sensitive-name bypass attempt. + - **Definition of done:** all five fail closed; reverting 4.5 makes at least three fail. + - _Requirements: R16.4_ + +- [x] **4.7 Update the threat model** + - Add the four control-plane rows to `SECURITY.md` in its existing format; state the + deployment posture in one line; add the reporting channel and supported-version window. + - **Definition of done:** every new row names a control that exists in code after 4.1–4.5. + - _Requirements: R17.1, R17.2, R17.3, R17.4, R17.5_ + +- [x] **4.8 Phase gate** + - **Definition of done:** full CI green including `tests/e2e/web`; a manual check that + the documented local quickstart still works end to end with no token. + +--- + +## Phase 5 — Container + +- [x] **5.1 `.dockerignore` and build context** + - Add `node_modules`, `dist`, `workspace`, `logs`, `.coverage-data`. + - **Definition of done:** build context size drops by roughly the frontend's + `node_modules` (~230 MB locally); `docker build` succeeds. + - _Requirements: R18.4_ + +- [x] **5.2 Frontend build stage (decision 12.3)** + - Add a Node stage producing `dist/`; copy it into the runtime image; make + `register_frontend()` log a warning when `dist/index.html` is absent. + - **Definition of done:** a started container serves the dashboard at `/`; the warning + appears when the stage is skipped. + - _Requirements: R18.1, R18.2_ + +- [x] **5.3 Runtime image minimality** + - Remove `build-essential` from the runtime stage; justify or remove `git`; switch the + healthcheck to `/api/health`. + - **Definition of done:** `gcc` absent from the runtime image; healthcheck passes; + image size recorded as the ratchet. + - _Requirements: R18.3, R18.5_ + +- [x] **5.4 Compose correctness** + - Fix the build context (`context: ..`, `dockerfile: docker/Dockerfile`); remove or + profile-gate the Ollama + GPU service; set `AI_TEAM_WEB_TOKEN`; add resource limits and + a restart policy; decide the `workspace/` mount. + - **Definition of done:** `docker compose -f docker/docker-compose.yml build` succeeds + from a clean checkout on a machine with no GPU. + - _Requirements: R19.1, R19.2, R19.3, R19.4, R19.5_ + +- [x] **5.5 Image CI job** + - Build the image in CI and assert: non-root user, no `gcc` on `PATH`, size under the + ratchet, `/api/health` → 200 in a started container. + - **Definition of done:** the job fails if any assertion is violated; runtime under two + minutes. + - _Requirements: R18.6, R18.7_ + +- [x] **5.6 Phase gate** + +--- + +## Phase 6 — Package boundaries *(largest diff)* + +- [x] **6.1 Break the eval→backend coupling** + - Replace `evals/checks/trajectory.py`'s import of `ai_team.flows` with a + backend-neutral phase-name source in `core/` or the taxonomy. + - **Definition of done:** no module under `evals/` imports `ai_team.backends` or + `ai_team.flows`; check behaviour unchanged (Tier A replay identical). + - _Requirements: R10.2_ + +- [x] **6.2 Move the CrewAI-private packages** + - `git mv` `agents/`, `crews/`, `tasks/`, `flows/` under + `src/ai_team/backends/crewai_backend/`. Update imports. **No other change in this + commit.** + - Update `.cursor/rules/crewai-agents-crews-flows.mdc` and `.cursor/rules/crewai-tools.mdc` + to the new paths in the same commit — a stale agent rule silently teaches the wrong + layout to every future contributor, human or not. + - **Definition of done:** `uv run pytest tests/unit` green; the diff contains only path, + import, and agent-rule changes; `tests/e2e/web` green. + - _Requirements: R10.1, R10.6, R21.10_ + +- [x] **6.3 Deprecation shims** + - Add `ai_team.agents`, `ai_team.crews`, `ai_team.tasks`, `ai_team.flows` shim modules + that re-export and emit `DeprecationWarning` with a removal date. + - **Definition of done:** old import paths work and warn; a test asserts the warning. + - _Requirements: R10.3_ + +- [x] **6.4 Import-direction guard** + - `tests/unit/repo/test_import_direction.py` enforcing the design §4 table. + - **Definition of done:** passes; adding `from ai_team.backends...` to a `core/` module + fails it with a message naming the rule. + - _Requirements: R10.4, R10.5, R11.2_ + +- [x] **6.5 Guardrails facade** + - Move the 572 lines of logic out of `guardrails/__init__.py` into named modules, + leaving imports and `__all__`. + - **Definition of done:** no behaviour change; every existing guardrail test green + without import changes. + - _Requirements: R11.3_ + +- [x] **6.6 Dissolve `utils/` and review `models/`** + - Move each member of `utils/` to the package that owns its subject, or delete it if + unreachable. Move `models/outputs.py` to `ui/artifacts/` if it remains single-consumer. + - **Definition of done:** no package named `utils` remains, or `design.md` §4 records why + it does; every moved module has a stated owner. + - _Requirements: R11.4, R11.5_ + +- [x] **6.7 Phase gate** + - **Definition of done:** full CI green; README structure tree updated to the post-move + layout; `tests/e2e/web` green — the proof the move was structural only. + +--- + +## Phase 7 — Complexity + +- [x] **7.1 `main.py:_cmd_run` (259 lines, 71 branches, 16 params)** + - Introduce a `RunOptions` dataclass built from the parsed namespace; extract the + argument-resolution, backend-selection, and result-reporting blocks. + - **Definition of done:** `_cmd_run` under 120 lines and 8 parameters; the run path is + callable from a test without constructing a CLI invocation; CLI behaviour identical + (every flag still works). + - _Requirements: R12.3, R12.4, R12.5_ + +- [x] **7.2 `mcp_server.py:build_ai_team_mcp_tools` (264 lines)** + - Extract one builder per tool group. + - **Definition of done:** under 120 lines; the MCP tool list is byte-identical before and + after (assert it in a test). + - _Requirements: R12.3, R12.5_ + +- [x] **7.3 `evals/gate.py:evaluate_gate` (230 lines, 38 branches)** + - Extract per-criterion evaluation; keep the decision table in one readable place. + - **Definition of done:** under 120 lines; Tier A gate output identical on the committed + corpus. + - _Requirements: R12.3, R12.5_ + +- [x] **7.4 Web router split** + - Split `server.py` (1,791 LOC, 24 routes) into routers by resource plus an app factory; + move module-level side effects into the factory. + - **Definition of done:** every route path, method, response shape and status code + unchanged; `tests/e2e/web` green; no new abstraction layer introduced. + - _Requirements: R13.1, R13.2, R13.3, R13.4_ + +- [x] **7.5 One cost resolver** + - Consolidate `_resolve_token_estimate`'s three-source fallback into one resolver with a + documented precedence order. + - **Definition of done:** one function owns the number the dashboard shows; its + precedence is stated in a docstring and covered by a test per source. + - _Requirements: R13.5_ + +- [x] **7.6 Phase gate** + +--- + +## Phase 8 — Guards + +Every task in this phase is subject to **R24**: offline, deterministic, order-independent, no +writes to tracked files, a recorded way to watch it fail, and a failure on an empty match set. +A guard that silently matches nothing is the default failure mode of repo-hygiene tests, and +it is worse than no guard because it reports success. + +- [x] **8.1 Reachability guard** + - `tests/unit/repo/test_reachability.py` with the commented `DORMANT_MODULES` allowlist + naming each module's activation flag. + - **Definition of done:** passes; adding an unreferenced module fails it with the + three-option message from design §9. + - _Requirements: R9.1, R9.2, R9.5_ + +- [x] **8.2 Reference guard** + - `tests/unit/repo/test_references.py` implementing the `/` convention; + relative links and source paths only, no external HTTP. + - **Definition of done:** passes; a deliberately broken link fails it with file and line. + - _Requirements: R4.4, R9.3_ + +- [x] **8.3 README structure guard** + - `tests/unit/repo/test_readme_structure.py`. + - **Definition of done:** passes; adding a package without updating the README fails it. + - _Requirements: R5.2, R9.4_ + +- [x] **8.4 Data-artifact guard** + - `tests/unit/repo/test_data_artifacts.py`: fixture↔check mapping in both directions (94 + fixtures / 20 check ids / zero orphans today — keep it that way); baseline covers every + registered check; no image outside the publication allowlist is unreferenced. + - **Definition of done:** passes; adding a check without a fixture, or an image nothing + references, fails it by name; the test raises rather than passes if the fixture glob or the + check registry comes back empty (R24.4); the docstring records the one-line mutation that + makes it fail. + - _Requirements: R21.2, R21.4, R21.8_ + +- [x] **8.5 Complexity ratchets** + - `tests/unit/repo/test_complexity.py` + `ratchets.toml` with dated comments in the style + of the existing `fail_under` history block. + - **Definition of done:** passes at the post-Phase-7 counts; a new 150-line function fails + it; the AST walk raises if it visits zero files (R24.4); the docstring records the mutation + that makes it fail. + - _Requirements: R12.1, R12.2, R12.6, R20.3, R20.4_ + +- [x] **8.6 Wire the guards into CI and close out** + - Add `tests/unit/repo` to the `lint` job; measure and record the total added CI time. + - Run the full suite shuffled and confirm order-independence for every test this spec added + (R24.1); confirm the `tests/conftest.py` hash guard still passes, i.e. no new test writes to + a tracked golden/fixture/taxonomy file (R24.2). + - Update `BASELINE.md` with the after-state; answer every open decision in `design.md` + §12. + - **Definition of done:** all guards green in CI; §12 has no unanswered row; the numbers + in `requirements.md`'s introduction are either fixed or restated as history. + - _Requirements: R9.6, R20.1, R20.2, R20.5_ + +--- + +## Phase 9 — Evidence (performance and envelope) + +Numerically last, **high priority** — see Track A in the execution order above. This phase is +what turns two of the four claimed properties from adjectives into artifacts. + +- [ ] **9.1 Publish one real benchmark** + - **Spends money — human-triggered only, $25 cap, reuse the existing spend guard.** + - Run the existing `tests/performance/` benchmarks (or a replacement) against a real backend + on a recorded machine, and publish `docs/PERFORMANCE.md` with: hardware, model ids, date, + sample size, wall-clock and cost per phase, tokens per run, and the harness overhead over a + bare backend call. + - State the regeneration command in the document (R21.7). + - **Definition of done:** every number traces to a command in the document; nothing is from a + mock run unless labelled in the same block; a reader can reproduce it. + - _Requirements: R22.1, R22.2, R22.3_ + +- [x] **9.2 Document the operating envelope** + - One section of `docs/PERFORMANCE.md` (or `docs/OPERATIONS.md`): single process; run registry + in memory and lost on restart (`RunState.runs` — its docstring already admits this); + concurrent-run ceiling and its source; per-run disk growth; spend ceiling; wall-clock kill. + - For each limit, name what would lift it — persistent run store, queue, worker pool — and + state explicitly that none is implemented, and why that is correct for a single-operator + tool. + - **Definition of done:** every limit has a number or a mechanism; no limit is described + without its next step; the README's scalability language matches this section exactly. + - _Requirements: R22.4, R22.5, R22.8_ + +- [x] **9.3 Retention policy** + - Define and implement retention for `workspace/` and `output/` (437 run directories on the + audit machine, unbounded), with a command that enforces it and a note in the envelope doc. + - **Definition of done:** a documented command prunes runs older than the stated window; + a unit test over a temporary tree asserts it removes what it should, keeps what it should, + and is idempotent on a second run; the demo/quickstart path is unaffected. + - _Requirements: R22.6_ + +- [x] **9.4 Resolve `tests/performance/`** + - The eight benchmark tests either produce 9.1's artifact or are removed — publishing nothing + is the same defect as a module nothing imports. + - **Definition of done:** the directory is wired to the published benchmark, or gone, with the + reason in the commit message. + - _Requirements: R22.7_ + +- [x] **9.5 Phase gate** + - **Definition of done:** full CI green; a reader can answer "how fast, how much, and what + breaks first?" from one document. + +--- + +## Minimum defensible slice + +If the whole plan is too long: **Track A** — Phases 1, 2, 3, 4, 5, 9, 8. Roughly a week, no +architectural change, and it fixes everything a reviewer would find in their first thirty +minutes while turning "performant and scalable" from adjectives into a document. + +Track B (Phases 6 and 7) is a second week of the two largest diffs in the repository for +changes a reviewer will not notice. Schedule it for the maintainer's benefit, not the +portfolio's. + +If even that is too long: **task 2.1 and Phase 4.** A harness status table that is true, +and an API that cannot be driven by anyone on the network. Those two are the difference +between a very good field-study repository and one that also demonstrates production +judgement. diff --git a/.kiro/specs/ui-refinement/BASELINE.md b/.kiro/specs/ui-refinement/BASELINE.md new file mode 100644 index 00000000..1d112a5d --- /dev/null +++ b/.kiro/specs/ui-refinement/BASELINE.md @@ -0,0 +1,76 @@ +# UI refinement baseline (task 0.1 / 0.2) + +Captured 2026-09-13 from `src/ai_team/ui/web/frontend`. + +## Counts + +| Metric | Before | +| --- | ---: | +| `src/App.css` lines | 2160 | +| `src/index.css` lines | 16 | +| Raw spacing/radius declarations (`padding`/`margin`/`gap`/`inset`/`inset-*`/`border-radius` with a numeric literal) | 249 | +| Tokenized spacing declarations | 49 | +| Distinct `font-size` values | 14 | +| Distinct `border-radius` values | 7 | +| `data-testid` literals (`TESTIDS.txt`, sorted unique including template patterns) | 99 | +| `npm test` | 112 passed, 1 failed, 3 errors (28 files). Failure: `Dashboard.cancel.test.tsx` — `getRunReceipt` is imported by `RunDetail` but missing from the `useApi` mock (pre-existing). | + +## Screenshots + +**Before-state (task 0.1):** not captured before Phases 1–9 landed; cannot be reconstructed. + +**After-state (2026-09-13):** `.kiro/specs/ui-refinement/screenshots/` + +| File | What it shows | +| --- | --- | +| `after-home.png` | Home with run cards; cyan `New run` | +| `after-run.png` | `/run` pipeline form | +| `after-compare-idle.png` | Compare after clicking Play sample runs | +| `after-compare-1280.png` | Three live demo columns side by side | +| `after-compare-800.png` | Narrow layout: Comparison Summary first | +| `after-rundetail-overview.png` | RunDetail Overview | +| `after-rundetail-activity.png` | RunDetail Activity | +| `after-rundetail-artifacts.png` | RunDetail Artifacts (demo empty) | +| `after-rundetail-tests.png` | RunDetail Tests (demo empty) | +| `after-stop-confirm.png` | Stop run? + red `.btn-danger` | +| `keyboard-command-palette.png` | Command palette | + +`/artifacts` redirects to `/runs/:id#artifacts` (IA-1); the Artifacts tab shot covers that surface. + +## Orphan classes (task 1.4) + +Recorded after a static scan of `src/**/*.tsx` vs `App.css` (dynamic prefixes excluded). + +### CSS-defined, unused in TSX (delete in Phase 1 unless noted) + +Dashboard-era / dead: `dashboard`, `dashboard-header`, `dashboard-empty`, `dashboard-page`, `dashboard-alerts`, `dashboard-layout`, `sidebar-collapsed`, `sidebar-drawer-open`, `sidebar-toggle`, `run-sidebar`, `run-monitor`, `run-monitor-grid`, `grid-2`, `grid-3`, `panel-toolbar`, `panel-collapsed-hint`, `empty-actions`, `raw-events`, `demo-helper`, `btn-warning`, `run-list-assignment`, `run-list-date`, `run-list-meta`, `run-list-delete`, `run-meta-row`, `run-meta-date`, `run-complete-actions`. + +Hue utilities (retire in 3.2): `cyan`, `green`, `red`, `yellow`, `blue`, `dim`. + +Kept (dynamic or later collapse): `crewai-title`, `langgraph-title`, `claude-title`, `code-search-hit`, `empty-state-block`, `agent-timeline-phase`, `log-error`/`log-success`/`log-warn`, `gr-pass`/`gr-fail`/`gr-warn`, `status-*` chips, `summary-failed-reason`. + +### TSX-used, undefined in CSS (resolve in 1.4) + +| Class | Resolution | +| --- | --- | +| `muted` | → `.text-muted` (R3.7) | +| `panel-nested` | define as `.panel-section` (R4.4) | +| `empty-state-icon` | style | +| `compact` | style (`ActivityLog` density) | +| `how-it-works-cost` | style (measure-capped prose) | + +Structural hooks (no visual rule; allowlisted in `design.md` §9): `run-config-form`, `run-detail-header`, `tests-panel`, `activity-log-wrap`, `home-empty`, `home-run-list`, `panel-inner`, `run-list-panel`, `run-list-day-group`, `coverage-chart`, `test-failures`, `command-palette-overlay`, `arch-panel`, `form-grid-backend`, `download-hint`, `pytest-raw`, `adr-block`, `code-view-modes`. + +## After-state (task 9.3) + +Captured 2026-09-13 after Phases 1–9. Gates: `npm run lint` (0 errors), `npm run build`, +`npm test` (125 passed / 30 files), `uv run pytest tests/e2e/web -m web_e2e` +(27 passed, 1 skipped). + +| Metric | After | +| --- | ---: | +| `src/App.css` lines | 602 (budget 1600; remainder split to `styles/{run,compare,artifacts,responsive}.css`) | +| Raw spacing/radius outside `tokens.css` | 0 | +| Distinct `font-size` outside `tokens.css` | 0 | +| Orphan classes | 0 (enforced by `styles.drift.test.ts`; structural allowlist in design §9) | +| `data-testid` diff vs `TESTIDS.txt` | **Removed:** `nav-open-run` (task 8.1). **Added:** EmptyState/LoadingState ids (`agent-table-empty`, `artifacts-no-files`, `artifacts-select-run`, `file-tree-empty`, `file-tree-loading`, `guardrails-empty`, `hitl-banner`, `run-list-empty`, `run-list-empty-filtered`, `${testIdPrefix}-empty`, `${testIdPrefix}-starting`). | diff --git a/.kiro/specs/ui-refinement/README.md b/.kiro/specs/ui-refinement/README.md new file mode 100644 index 00000000..55212e79 --- /dev/null +++ b/.kiro/specs/ui-refinement/README.md @@ -0,0 +1,110 @@ +# Spec: `ui-refinement` + +Kiro-style three-document spec for making the web UI clean, simple, and tasteful — +by making its existing design system the only way to express a value, deleting the +stylesheet that styles nothing, and fixing six accessibility defects. Read in order: + +1. **[`requirements.md`](./requirements.md)** — 22 requirements with EARS acceptance + criteria, constraints, non-goals, and a file-level traceability table. +2. **[`design.md`](./design.md)** — token architecture with measured contrast figures, + component primitives, accessibility patterns, the Compare redesign, enforcement + strategy (§9), migration risk (§10), and six open decisions (§12). +3. **[`tasks.md`](./tasks.md)** — 10 phases, 38 tasks, each with a definition of done and + requirement traceability. + +Scope is `src/ai_team/ui/web/frontend` only. No Python, no API contract, no run semantics. + +## The one-line version + +The UI already has a design system; it is applied to about an eighth of the stylesheet. +This spec makes the tokens load-bearing, deletes the ~15% of `App.css` that styles a page +that no longer exists, fixes the accessibility defects a keyboard or screen-reader user +actually hits, and adds a test that fails the build on the next raw `0.35rem`. + +## What the audit found + +| | Today | +| --- | --- | +| `App.css` | 2,160 lines | +| Spacing | **29** tokenized declarations vs **208** raw, across 15 distinct values | +| Type | **14** distinct `font-size` values, no scale; `body { font-size: 14px }` in px | +| Radius | **7** distinct values | +| Duplicate selectors | **6** (`.btn-sm` resolves to the second of two disagreeing blocks) | +| Dead rules | an entire Dashboard-era block — `.dashboard-layout`, `.run-sidebar`, `.sidebar-drawer-open`, `.grid-2/3`, … — plus the media query that exists only for it | +| `--content-max-width` | declared, referenced **zero** times; gutter applied twice | +| `

` | **none in the application** | +| `htmlFor` | **zero occurrences**; 6 unassociated `