diff --git a/domains/perps/skills/recipe-cook/references/advisor.md b/domains/perps/skills/recipe-cook/references/advisor.md new file mode 100644 index 00000000..a6a784b5 --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/advisor.md @@ -0,0 +1,18 @@ +# Optional recipe advice + +Use the task's recorded harness throughout. The harness owns settings, precedence, provider selection and credential detection. Do not read configuration or secrets yourself, call a provider directly, or install a newer harness for this optional feature. + +1. Run `mm-harness recipe-quality advisor-status --json`, forwarding the caller's explicit advisor options. Save its output as `/artifacts/advisor-policy.json`. Recheck policy on resume so a current opt-out takes effect, but reuse existing assessments instead of making another request. Preserve a prior policy record when it changes. An unsupported command, disabled policy, missing key or configuration error leaves ordinary authoring available; record the reason and do not retry. +2. When status is `ready`, use only its listed `stages`. The default is `plan`. Copy the recorded acceptance criteria into `/artifacts/advisor-task.json` using the installed command's task schema. Preserve supplied wording and any labels for assumptions or derived criteria. +3. Before drafting, if `plan` is enabled, call `recipe-quality plan --task --adapter --output /artifacts/advisor-plan.json --json`. Pass the resolved policy back as `--advisor --advisor-model --advisor-stages `. Redirect command stdout to a task-local file instead of loading the duplicate full artifact into context. Inspect a compact summary first, then read only the selected candidate contracts needed for authoring. Record accepted or rejected suggestions briefly in the recipe coverage notes; a suggested match does not establish a postcondition. +4. If explicitly enabled in the policy, `draft` permits one `recipe-quality advise` call on the statically valid recipe; `evidence` permits one such call with focused execution observations through `--evidence`. Keep outputs at `recipe-advice.json` and `evidence-advice.json` under task artifacts. Forward the same frozen policy and actual root parameters. Supply observed results for every claimed case; image filenames are not pixel proof. + +Limit each enabled stage to one request per materialized task. Reuse existing results on resume. A materially revised task needs an explicit new assessment decision, not an automatic retry. Retain provider status, model, input/question hashes, usage and duration from the output, including skips or unavailability. Treat all suggestions as advice; validate and execute revised recipes normally. Advice cannot waive a gate, authorize app actions, supply missing evidence or replace canonical quality. + +The execution environment can supply overrides. The same status command resolves them; the skill does not implement separate policy. `--advisor off` disables advice for this task even when the saved profile is enabled. + +A compact planning summary preserves the outcome and accounting without repeating the catalog: + +```bash +jq '{status, reason, provider, model, usage, durationMs, assessments: [.assessments[]? | {id, classification, ref: .candidate.ref, confidence}]}' /artifacts/advisor-plan.json +``` diff --git a/domains/perps/skills/recipe-cook/references/dev-fix-bug.md b/domains/perps/skills/recipe-cook/references/dev-fix-bug.md new file mode 100644 index 00000000..a70c5f5b --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/dev-fix-bug.md @@ -0,0 +1,143 @@ +# Dev and fix-bug rules + +Apply the sections for this flow and platform at each step. + +## Task and sandbox + +- Work top to bottom; apply Cook's blocked rule. Mark each completed gate with artifact paths. +- `` means TASK.md `TASK_DIR`; bare `mm-harness` means `/mm-harness`. +- Keep proof under `/artifacts/` and out of product commits. +- Put clones, installs, tooling and loop workspaces in `/../scratch-` or `temp/`, never `artifacts/`. +- Confirm the handoff's checklist path; never estimate or rewrite it. +- Capture the summary and ACs, then verify retained tooling with `.agents/skills/mms-recipe-cook/scripts/init-template --resume-task-dir ` and read its recorded `mm-harness help --json`. +- Preserve TASK.md, CHECKLIST.md, markers and receipts. If tooling verification fails, stop with that cause; never resolve a replacement for this task. +- Follow `/mms-recipe-harness`: doctor/status/launch-or-reuse/verify for this checkout. Require `artifacts/sandbox.json` with `ready: true`. +- Mobile requires the running simulator/device app; Extension requires reachable CDP. First recover with harness launch/doctor; block only on an external farm/device failure. +- Create or switch to the local task branch before product edits. Do not open a PR. + +## Plan and baseline + +- Discover the harness catalog before drafting: `mm-harness run --list --json`; use Recipe v1 `workflow.{entry,nodes}` and live runtime evidence. Tests corroborate only. +- Core targets packages/controllers and fixtures, using headless `state`, `command` or `log` proof without PNGs. A `command` proof drives the controller through its public API as a client does (real messenger, mocked only at the network boundary); it never runs Jest or another unit-test probe. +- Mobile targets iOS, Android or both; Extension targets popup, sidepanel, fullscreen or dapp. Record the rationale. +- UI ACs use state/visual/mixed proof; visual proof requires a human-read PNG. Extension copy, labels, warnings, modals and toasts are visual or mixed. +- Development UI proof plans bind a fixture, semantic intent and verified live postcondition. Mobile uses adapter-owned current route mappings; Extension discovers runtime controls and keeps task recipes in `artifacts/recipe-library/recipes/`. +- Name UI AC nodes `ac-...`, plumbing nodes `setup-`/`teardown-`/`gate-`, and baseline captures `before-...`. +- For persisted UI changes, assert the expected start state in setup and restore it in teardown. + +### Development + +- Read affected code end to end; write `artifacts/approach.md` with affected files, current behavior/insertion point, chosen approach and trade-offs. +- Mobile/Extension: choose the most defensible approach without pausing. Before editing existing UI, capture comparable `before-*` evidence on the same route, state, target and viewport. For purely additive work, write `Baseline: N/A. additive` in TASK.md. +- Draft the recipe file and require `mm-harness run --plan` PASS. +- Run it before implementation with `--artifacts-dir /artifacts/recipe-baseline-run --json`; require failure at the first new-behavior AC assertion, not setup, before marking the recipe gate. +- Place regression guards before the first new-behavior AC; guards must pass before and after implementation. If every AC is a guard, require the baseline run to pass end to end. +- Record the recipe path and command, including for visual/mixed proof. Implement only after the recipe gate is marked PASS. + +### Bug fixing + +- Keep the sandbox green before baseline and fix. No product behavior edits before baseline; locator testIDs are allowed, as is the Mobile reproduction marker. +- Mobile/Extension: until the minimal-fix step, write only TASK.md, checklist marks and `/artifacts/`, apart from those locator/marker exceptions. +- A premature UI behavior change requires `artifacts/ordering-violation.md`; restore baseline order before continuing. +- Plan the expected buggy evidence and each AC's state. Missing state that setup actions can build is not UNTESTABLE; Extension reserves that verdict for infrastructure limits. +- Probe reused recipe entry/setup boundaries with a short bound before baseline. For recipe, recipe-library or action-contract mismatches, copy the closest recipe into the task, adapt to current actions and continue; never block. +- Core: list `mm-harness actions --json` as well as recipes and write `artifacts/recipe-decision.json` with `version: 1`, boolean `required`, a non-empty `reason` string and non-empty `actionsChecked`/`claims` arrays. +- When proof is required, the decision also sets `recipePath: "artifacts/recipe.json"`. +- Core: `required: false` requires no listed action that can exercise the change. Missing setup state is not an exemption; diagnose and repair missing runtime prerequisites first. +- Core: when required, draft `artifacts/recipe.json` asserting fixed behavior, prefer reusable recipes and require plan PASS. Run it into `recipe-baseline-run/` and require failure at the reported bug's AC assertion. Focused tests are never the baseline proof for a runtime claim. +- Mobile/Extension: draft `artifacts/recipe-baseline.json` asserting buggy real-app behavior through live actions or run-scoped markers/logs; require plan PASS, then run it into `recipe-baseline-run/` with `--record-video=full-run --json` and read every visual PNG. +- For already-fixed/not-reproduced cases, write `artifacts/no-change-report.md` and `artifacts/learnings.md`, then follow Cook's Finish before marking no-change. UI flows add `--already-fixed` to the terminal mark when this branch already fixes it; setup/environment failures follow Cook's blocked rule. +- After the minimal fix, UI `artifacts/recipe.json` keeps the baseline nodes, asserts fixed values, passes plan and fails with the fix reverted. + +## Implementation and checks + +- Keep implementation AC-mapped; bug fixes are minimal, without unrelated refactoring. +- Development reuses domain helpers for external identifiers. +- Reuse or extract a helper beside the existing hook/util before repeating a derivation or parser; leave unrelated duplication alone. +- Core changes add each required package CHANGELOG entry. +- New development UI interactive elements use Mobile `testID` or Extension `data-testid`; bug fixes allow those locator additions. +- Mobile development literal-colour requirements use `theme.brandColors`; semantic tokens change in dark mode. +- Run `mm-harness check diff [--fix] --artifacts-dir /artifacts` as the fast lint/format gate. Never discover or run separate repository lint/format commands. +- The default fast profile skips typecheck. Require PASS before marking the gate; UI flows retain `validation-summary.json`. +- Extension bug fixes also run `yarn verify-locales --quiet` for changed locale files and `yarn circular-deps:check` for changed imports. + +## Runtime proof + +- Keep the authored final recipe at `artifacts/recipe.json`; run `/mm-harness run /artifacts/recipe.json --artifacts-dir /artifacts/recipe-run --json`. +- Mobile/Extension add `--record-video=full-run`; Mobile video is under `recipe-run/videos/`. Core bug-fix recipes run only when required and must pass. +- Core setup failures: inspect retained state/logs, diagnose and repair drift with `--heal auto`; never weaken assertions. Apply Cook's blocked rule. +- After self-review, rerun proof when proven behavior changed; otherwise record an explicit no-rerun decision. + +### UI evidence + +- Read every visual/mixed PNG. The claimed target must be visible, without clipping, covers or a generic home/empty screen, and follow `ui.wait_for` on that target. +- Bug-fix after captures must differ from `before-...` as the bug-to-fix claim describes. Fix insufficient evidence and rerun instead of shipping a caveat. +- If video cannot start, retain the unsupported reason and mark visual ACs UNTESTABLE/BLOCKED; never substitute a recorder. +- Stop missing-capability probing after 10 minutes; record the gap in `learnings.md` and use the simplest supported proof. +- Prove disabled controls with their visible reason, a read PNG and a real press attempt. Retain press refusals in a separate expected-fail probe. +- If the press succeeds, assert unchanged state and no action/submission log. Never claim a press had no effect when it did not run. +- Write recipe `intent` as the user-visible goal, without action narration or UI text targets. + +### Mobile runtime + +- After source edits and before each recorded run, run `mm-harness call app.lifecycle command=restart`; HMR is off. +- Fetch the bundle URL and find an introduced string. On a miss, run `mm-harness stop`, then `launch --clear-metro --verify`, and recheck. +- Label stale runs `*-stale*` and never cite them. Use testIDs when hidden mounted sheets can absorb text presses. +- The HUD renders intents on screen; Mobile text matching uses case-insensitive contains. + +### Extension runtime + +- Reload before recorded runs with `mm-harness doctor --expect-live`; run `mm-harness launch --verify` if the extension stopped responding. HMR is off. +- Accept capture-helper screenshots only; reject screencapture and `Page.captureScreenshot` fallbacks. +- File/JSON assertions over command output must depend on that command's success and read a file written inside this run's artifacts directory; old files are not evidence. +- The HUD renders intents in-page; `ui.wait_for` matches any rendered text. + +## Tests + +- Core: focused tests and AC-required domain smoke only. Do not substitute repo build/typecheck for the fast-profile skip. +- Build Core only for changed exports, types or build wiring, with root `yarn build`; per-package builds need sibling outputs. Development records why a build is unnecessary. +- Check new/changed Core exports in emitted files named by `exports`, including `types` for type-only exports. +- Run Core build, tests and lint sequentially. +- Mobile: use `/mms-mobile-testing` for the layer policy and [component view guidance](mobile-testing/component-view.md) for UI behavior. If absent, follow `docs/testing/unit-testing.md` and record the gap. +- Run Mobile `yarn test:view:one --runInBand`, `yarn test:integration:one --runInBand` or `yarn jest --runInBand --no-coverage`, including affected consumers. Confirm every suite ran tests. +- Require >=80% Mobile changed-file coverage using the same command with `--coverage --collectCoverageFrom=`; `coverage:analyze` needs an open PR. +- Extension: use `/mms-extension-testing`; if absent, follow `.cursor/rules/unit-testing-guidelines.mdc` and record the gap. +- Run Extension `yarn jest --no-coverage`, then require >=80% changed-file coverage with `--coverage --coverageReporters=text-summary --collectCoverageFrom=`. +- Never use `--findRelatedTests`. Bug-fix tests cover the reported bug condition. + +## Self-review + +Apply [review verdicts](team-libraries.md#review-verdicts). + +- Mobile/Extension: run `/mms-review-pr non-interactive` as self-review of this checkout; apply Cook's blocked rule if the skill is missing. +- Core has no `review-pr` overlay; review correctness, error/boundary paths, performance, named constants, separation of concerns, dead/debug code, scope, simpler alternatives, downstream compatibility and exports. +- Include committed, staged, unstaged and untracked changes; record base, head, working-tree state, findings and skipped/unchecked areas in `artifacts/review.md`. +- Fix findings here; rerun affected tests, `check diff` and proof when proven behavior changed. +- Core tests must fail when their guarded behavior is removed. +- Block on duplicated derivation, parsing or formulas introduced by this diff, including copies of existing logic. +- A test that waits on a real timer (sleep, real poll, backoff or timeout) is a blocking finding; use fake timers. +- At the domain step, follow `references/team-libraries.md#dev-and-fix-bug-domain-review` with the checklist's stated parent step number. + +## Mainnet + +Apply Cook's Mainnet rule in `.agents/skills/mms-recipe-cook/SKILL.md`. + +## Coverage and finish + +- Write `artifacts/recipe-coverage.md` with one row per AC/claim: proof mode, evidence path under `recipe-run/`, recipe nodes, verdict and rationale. +- UI verdicts are PROVEN/WEAK/UNTESTABLE/MISSING. Check bug-fix rows against `recipe-run/trace.json`; failed nodes are never PROVEN. Fix WEAK/MISSING proof and rerun. +- Include `What this recipe does NOT prove`, ending with `Overall recipe coverage: / ACs PROVEN (untestable: , weak: 0, missing: 0)`. +- Build `artifacts/recipe-quality.json` through Cook's [Finish instructions](../skill.md#finish). Never pass UI WEAK/MISSING rows, development assertions that also pass before implementation, or bug-fix recipes that pass with the fix reverted. +- Core bug fixes write recipe coverage/quality when `artifacts/recipe.json` exists; non-recipe AC verdicts remain required. +- List all ACs with `.agents/skills/mms-recipe-cook/scripts/acceptance-cli list`. +- Render the engine-owned coverage with `.agents/skills/mms-recipe-cook/scripts/acceptance-cli render`; reconcile it with the retained recipe assertions and coverage rows. +- Missing verdicts require eligible recipe proof, including non-UI claims. Never write verdicts; only a verified person handles exceptions. +- Write `artifacts/pr-description.md` with only repository-template prose, `TICKET_URL` when set and proof summary; bug fixes include root cause, and Mobile includes marker SHA/log excerpt. Never paste recipe JSON or run output into renderer-owned sections. +- Run `mm-harness pr-body render --out /artifacts/pr-body.md --json` with mm-harness 0.92.21 or newer, then read that file. Preserve `--command ` when supplied. Require exit 0, `status: ok` and the generated body; renderer failure must not fall back to authored prose. Keep task markers and the locked harness receipt unchanged. +- Write `artifacts/evidence-manifest.json`: UI includes PROVEN visual evidence only, paths relative to `artifacts/`. Read video names from each run's `artifact-manifest.json` and prefix its run folder; manifests use run-relative paths. +- Put visual comparisons under `before_after_pairs`, with `label`, `covers`, `before` and `after`; use `standalone` for evidence that is not a comparison. Keep paths relative to `artifacts/`, including their run folders; never match captures by basename or invent a missing side. +- The manifest accepts only `version`, `preferred_mode`, `summary`, `before_after_pairs`, `standalone`, `omit` and `videos`. Keep draft, review and runtime-status notes in the report, not additional manifest keys. +- Development `before` images require screenshot nodes before the failing AC assertion. Core uses a media-free manifest, `{"version":1,"summary":"..."}`. +- Core also writes `artifacts/report.md` with summary/root cause, changed files, validation commands/results, recipe verdict/evidence, risks and client follow-ups. +- Write `artifacts/learnings.md` and satisfy host handoff prerequisites. Commit only when requested: one Conventional Commit of product paths, scope from the task domain/changed area, descriptive subject; never amend. Mobile marker commits remain separate. +- Finish with `/mark complete --mark-last`, subject to the skill's required-proof completion rules. diff --git a/domains/perps/skills/recipe-cook/references/development.md b/domains/perps/skills/recipe-cook/references/development.md new file mode 100644 index 00000000..c2d81047 --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/development.md @@ -0,0 +1,21 @@ +# Recipe skill development + +The recipe skills are maintained in `MetaMask/skills` under `domains/perps/skills/`. Install this branch into a clean target with the repository installer and `--domain perps --maturity experimental`. + +Run the domain checks from the repository root: + +```sh +node --test domains/perps/skills/recipe-cook/scripts/tests/*.test.mjs domains/perps/skills/recipe-qa/scripts/*.test.mjs domains/perps/skills/recipe-pretty-pr/scripts/tests/*.test.cjs +``` + +Set `MM_HARNESS_BIN` to the real path of a published `mm-harness` executable to include the published-runtime checks. Without it, only those checks skip. Root CI currently discovers `test/*.test.mjs`, so run this domain command separately. + +Run the existing shell integration suites separately: + +```sh +bash domains/perps/skills/recipe-cook/scripts/tests/qa-install.test.sh +bash domains/perps/skills/recipe-cook/scripts/tests/resolve-harness.test.sh +bash domains/perps/skills/recipe-cook/scripts/tests/prepare-sandbox.test.sh +``` + +The source move preserves the catalog ID `consensys-recipe-cook` so recorded template references remain valid. The two thin dev/fix-bug entry points move with Cook. Recipe Harness uses `mm-harness`; its former Farmslot CLI fallback is omitted. diff --git a/domains/perps/skills/recipe-cook/references/mobile-testing/component-view.md b/domains/perps/skills/recipe-cook/references/mobile-testing/component-view.md new file mode 100644 index 00000000..3d87059d --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/mobile-testing/component-view.md @@ -0,0 +1,147 @@ +# Component View Testing Guidelines + +Create, update, and fix component view tests (`*.view.test.tsx`) using the `tests/component-view/` framework. + +## When to use + +**Default Mobile layer** for screen/view UI behavior through real Redux/app state. + +Use this skill whenever you need to: + +- Cover new or changed screen/view behavior (prefer CV over broad unit tests) +- Write a new component view test file +- Update tests after a component or preset has changed +- Diagnose and fix a failing component view test + +If the case is pure logic / helpers, or CV cannot cover it yet, open [`unit.md`](unit.md) for the smallest focused unit test and document why. + +## Testing layers (read first) + +Follow installed `.agents/skills/mms-mobile-testing/knowledge/testing-layers.md` for the full decision tree. Choose by layer inside **mobile-testing**. Prefer **CV → integration → unit fallback → E2E**; multi-screen journeys are CV-first when routes can be registered — do not jump to Appium for “journey” alone. + +Your job is to figure out whether the user needs to **write a new test**, **fix a failing test**, or **update tests after a component/preset change**, then follow the corresponding path and open the relevant reference when that path indicates. + +**Decision tree — which reference to use:** + +``` +Task → What do you need? +├─ Write new test or update after change +│ → Read component + existing tests +│ → Open component-view/writing-tests.md (use cases, coverage, renderer/preset, file structure) +│ → If test needs navigation: also open component-view/navigation-mocking.md +│ → After writing: run tests, then open component-view/reference.md for self-review +│ +├─ Fix failing test +│ → Run: yarn jest -c jest.config.view.js --runInBand --silent --coverage=false +│ → Identify error type → Open component-view/reference.md (Diagnosing Failures) +│ +└─ Run tests or self-review after tests pass + → Open component-view/reference.md (Run the Tests, Self-Review Checklist) +``` + +Do not read the full reference files until the decision tree or workflow sends you there. + + +## What Are Component View Tests? + +Component view tests are **integration-level** tests that test views through real Redux state — no mocked hooks or selectors. They live alongside the component as `ComponentName.view.test.tsx` and use a dedicated framework in `tests/component-view/`. + +Key constraint: **only Engine and allowed native modules may be mocked** (enforced at runtime by `app/util/test/testSetupView.js` and by ESLint override in `.eslintrc.js` for `**/*.view.test.*`). + + +## The Framework at a Glance + +``` +tests/component-view/ +├── mocks.ts ← Engine + native mocks (import this first, always) +├── render.tsx ← renderComponentViewScreen, renderScreenWithRoutes +├── stateFixture.ts ← StateFixtureBuilder (createStateFixture) +├── platform.ts ← describeForPlatforms, itForPlatforms (run per iOS/Android) +├── api-mocking/ ← HTTP API mocks (nock) — extensible, one file per feature +├── presets/ ← initialState() builders — one file per feature area +└── renderers/ ← renderView() functions — one file per feature area +``` + + +## Workflow (summary) + +- **Write new test**: Read component and existing tests → list use cases and map to test patterns → check coverage and deduplicate → use or create renderer/preset → write test (use `renderScreenWithRoutes` if asserting navigation). Every test must have at least one of: `fireEvent`, `waitFor`/`findBy`, `store.dispatch`/`act`, or Engine spy (no render-only scenarios). Run tests, then run the self-review checklist in `component-view/reference.md`. +- **Fix failing test**: Run with `jest.config.view.js` → identify error type from the table in `component-view/reference.md` (Diagnosing Failures) → apply the fix (remove disallowed mock, add state override, add preset, wrap in `waitFor`, add `deterministicFiat`, etc.) → re-run. +- **Update after change**: Same as write — review existing tests, extend preset/renderer if needed, update tests, run and self-review. + +For full detail (use cases, coverage, presets, route probes, self-review checklist, failure table), use the reference files when the decision tree sends you there. + + +## Run the tests + +Always use `jest.config.view.js` — the default Jest config does not apply component view test rules. + +**Run tests (no coverage):** + +```bash +yarn jest -c jest.config.view.js --runInBand --silent --coverage=false +``` + +Example: `yarn jest -c jest.config.view.js app/components/UI/Bridge/Views/BridgeView/BridgeView.view.test.tsx --runInBand --silent --coverage=false` + +**Coverage for a feature folder** (use this instead of `--coverage` to avoid OOM): + +```bash +yarn test:view:coverage:folder app/components/UI/MyFeature +``` + +For run-by-name, watch mode, or other options, see `component-view/reference.md` (Run the Tests). + + +## Golden Rules (Enforced) + +1. **Only mock Engine and allowed native modules** — no arbitrary `jest.mock()` in `*.view.test.*` files. Allowed: + - `../../app/core/Engine` + - `../../app/core/Engine/Engine` + - `react-native-device-info` + - (these are already handled by `tests/component-view/mocks.ts`) + +2. **Drive all behavior through Redux state** — no mocking of hooks or selectors. Provide data via state overrides. + +3. **Reuse presets and renderers** — never rebuild the full state manually from scratch. + +4. **No fake timers** — never use `jest.useFakeTimers()`, `jest.advanceTimersByTime()`, or `jest.useRealTimers()`. + +5. **Test behavior, not snapshots** — use `toBeOnTheScreen()`, `not.toBeOnTheScreen()`, interaction assertions. + +6. **Follow AAA** — Arrange → Act → Assert, blank lines between each section. One test = one user journey or business outcome; multiple chained actions in a single test are fine. + +7. **No render scenarios** — every test must have at least one of: `fireEvent`, `waitFor`/`findBy`, `store.dispatch`/`act`, or an Engine spy. Static visibility checks are not tests. See [`component-view/writing-tests.md`](component-view/writing-tests.md) for examples. + +8. **Use constants and i18n helpers, never raw strings** — every `getByTestId` / `findByTestId` / `queryByTestId` must reference a constant from `ComponentName.testIds.ts`. Every `getByText` / `findByText` / `getAllByText` that targets a localised label must use `strings('key')` from `locales/i18n`, not a hardcoded string literal. Dynamic testIds (e.g. `trader-row-${trader.id}`) must be derived from fixture data, not embedded as `'trader-row-trader-1'`. Create the testIds file if it does not exist. + +9. **Every view with async data needs one data-completeness test** — wait for the load and validate all significant fields of all items in the base mock using `within()` per row. One per independent async data flow. + +10. **Filter / segmentation tests must assert both sides** — after selecting a filter, assert both what appears (positive `findByTestId`) and what disappears (negative `queryByTestId(...).not.toBeOnTheScreen()`). Spy-only checks (refetch count / analytics) are not enough — seed distinct before/after rows. + +11. **Match loading asserts to real UX** — pending-phase tests assert skeleton / not-yet-visible content, not optimistic titles the production screen does not show while loading. + +12. **Pull-to-refresh via `refreshControl.props.onRefresh`** — prefer the prop handler inside `act`; `fireEvent(scrollView, 'refresh')` often never hits the handler. + +13. **Await the content, not just its container** — a container arriving on screen says nothing about values inside it that have their own async source (a child query, a debounce, a skeleton). Await the gated value with `findBy*`, then re-query the container and scope the remaining synchronous assertions with `within()`. + +14. **Do not nest `find*` inside `waitFor`** — `findBy*` already *is* `waitFor` + `getBy` (default timeout 1000 ms). Nesting them shares that budget, so CI can fail with a bare `Timed out in waitFor.` and no assertion detail. Use `await findBy*` **or** `waitFor(() => { getBy*; expect(...) })` with an explicit `{ timeout }` when you need extra time. Never `waitFor(async () => { await findBy*(...) })`. + +15. **Reset mock call history between tests** — add `beforeEach(() => jest.clearAllMocks())` to every `describe` block that asserts on `Engine.controllerMessenger.call` call counts or arguments. The CV mocks define `Engine.controllerMessenger.call` as a single shared `jest.fn()`; without a reset, accumulated calls from prior tests corrupt `toHaveBeenCalledTimes(N)` and `toHaveBeenCalledWith(...)` assertions. + +16. **Run the tests before declaring done** — run `yarn jest -c jest.config.view.js --runInBand --silent --coverage=false` and confirm all tests are green. A test file that has never been run is not done. + +17. **Run format:check before declaring done** — run `npx prettier --check ` (or `yarn format:check`) on the new test file and every new supporting file (renderer, preset, api-mock). Auto-fix with `npx prettier --write ` if needed, then re-check. + + +## Reference files (when to use) + +Documentation is split by **action**. Open only the reference that matches what you are doing. + +| Action | File | When to open it | +| ----------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| **Writing or updating view tests** | [`component-view/writing-tests.md`](component-view/writing-tests.md) | New test file, new or updated preset/renderer. Read before writing, use cases and coverage, file structure, renderers, presets, route params. | +| **Testing navigation** | [`component-view/navigation-mocking.md`](component-view/navigation-mocking.md) | Route probes, single nav push, multi-screen renderer, cross-screen journey, external/API mocking. | +| **Running tests, self-review, fixing failures** | [`component-view/reference.md`](component-view/reference.md) | Run the Tests, Self-Review Checklist, Diagnosing Failures, assertion patterns, Deterministic Fiat Assertions, What NOT to Do, Quick Reference. | + +**Where self-review and What NOT to Do live:** Both are in `component-view/reference.md`. Self-review is the checklist you run after tests pass. What NOT to Do is the antipatterns section in the same file. Keeping them there means when you run tests or fix failures you have run commands, the checklist, the failure table, and the antipatterns in one place — open that reference for any run/fix/review task. diff --git a/domains/perps/skills/recipe-cook/references/mobile-testing/component-view/reference.md b/domains/perps/skills/recipe-cook/references/mobile-testing/component-view/reference.md new file mode 100644 index 00000000..0d474211 --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/mobile-testing/component-view/reference.md @@ -0,0 +1,352 @@ +# Running Tests, Self-Review, and Diagnosing Failures + +Use this reference when you need to **run** component view tests, **self-review** after tests pass, or **diagnose and fix** failures. It also covers assertion patterns, deterministic fiat, and What NOT to Do. + +Reference: [SKILL.md](../SKILL.md) · [Writing Tests](writing-tests.md) · [Navigation & Mocking](navigation-mocking.md) + +--- + +## Table of contents + +- [Deterministic Fiat Assertions](#deterministic-fiat-assertions) +- [Run the Tests](#run-the-tests) +- [Self-Review Checklist](#self-review-checklist) +- [Diagnosing Failures](#diagnosing-failures) +- [Assertion Patterns](#assertion-patterns) +- [What NOT to Do](#what-not-to-do) +- [Quick Reference](#quick-reference) + +--- + +## Deterministic Fiat Assertions + +Pass `deterministicFiat: true` whenever a test asserts exact currency values. This injects stable exchange rates: + +```typescript +const { getByText } = renderBridgeView({ + deterministicFiat: true, + overrides: { bridge: { sourceAmount: '1' } }, +}); +expect(getByText('$2,000.00')).toBeOnTheScreen(); +``` + +--- + +## Run the Tests + +**Always use `jest.config.view.js`** — the default Jest config does not apply the component view test rules. + +```bash +# Run a single file +yarn jest -c jest.config.view.js app/components/UI/Bridge/Views/BridgeView/BridgeView.view.test.tsx --runInBand --silent --coverage=false + +# Run a specific test by name +yarn jest -c jest.config.view.js -t "renders the source token" --runInBand --silent --coverage=false + +# Watch mode +yarn jest -c jest.config.view.js --watch + +# Coverage for a feature folder (use this, not --coverage directly — avoids OOM) +yarn test:view:coverage:folder app/components/UI/MyFeature +``` + +--- + +## Self-Review Checklist + +Before declaring the task done, go through this checklist for every test written or modified. If any item fails, fix it and re-run. + +| # | Check | What to do if it fails | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- | +| 1 | **No render scenarios** — every test has at least one `fireEvent`, `waitFor`/`findBy`, `store.dispatch`, or Engine spy | Rewrite the test to add a user interaction or system reaction | +| 2 | **No selector mocking** — no `(useSelector as jest.Mock).mockImplementation(...)` anywhere in the file | Remove; drive behavior through state overrides instead | +| 3 | **No fake timers** — no `jest.useFakeTimers()`, `jest.advanceTimersByTime()`, or `jest.useRealTimers()` | Remove fake timers; use `waitFor` / `findBy` for async flows | +| 4 | **Data-completeness test exists** — if the view loads data asynchronously (API, Engine polling), there is one test that waits for the load and validates all fields of all items in the full base mock using `within()` per row | Add the data-completeness test | +| 5 | **Filter/segmentation tests have paired assertions** — every test that selects a filter or changes a network asserts both what appears (`findByTestId`) AND what disappears (`queryByTestId(...).not.toBeOnTheScreen()`) for each item from the previous set. Spy-only checks (refetch count / analytics) are **not** enough | Seed distinct before/after rows; add the missing negative assertions | +| 6 | **No raw strings in `getByTestId` / `findByTestId` / `queryByTestId`** — all test IDs reference constants from the component's `ComponentName.testIds.ts` | Create or update the testIds file; replace raw strings with constants | +| 7 | **Any `jest.mock` for non-Engine modules is flagged** — if a service module is mocked directly, the `eslint-disable` comment is present and a tracking issue is linked | Add the comment and issue link | +| 8 | **AAA formatting** — blank lines between the Arrange, Act, and Assert blocks in every test | Add the blank line separators | +| 9 | **Import order** — `mocks.ts` is first; remaining order follows project ESLint rules | Ensure `mocks.ts` is the very first import; reorder the rest as needed | +| 10 | **No stale press targets** — do not `fireEvent.press` a node held across `await`s when the UI re-renders (live countdown, polling). Re-query with `getByTestId` / `findByTestId` immediately before press | Re-query right before press; see What NOT to Do | +| 11 | **Loading asserts match real UX** — pending-phase tests assert skeleton / “not yet visible”, not optimistic titles the production screen does not show while `isLoading` | Rename and assert the real pending UI; resolve then assert loaded state | +| 12 | **Pull-to-refresh uses `refreshControl.props.onRefresh`** — not `fireEvent(scrollView, 'refresh')` | Call the prop handler inside `act` | +| 13 | **Unit→CV migrations keep assert specificity** — deleted unit payload fields (`tabId`, formatted dates, full analytics) still appear in the CV replacement | Restore dropped fields in CV or KEEP a focused unit; see unit-cv-overlap | +| 14 | **Awaits cover the asserted content, not just its container** — after `await findByTestId(CONTAINER)`, no synchronous `getBy*` asserts a value that has its own async source (child query, debounce, skeleton) | Await the gated value with `findBy*` first, then re-query the container and scope the sync asserts | +| 15 | **No nested `find*` inside `waitFor`** — no `waitFor(async () => { await findBy*(...) })`. `findBy*` already polls with the same 1s default timeout | Use `await findBy*` **or** `waitFor(() => { getBy*; expect(...) })` with `{ timeout }` | +| 16 | **Mock state is reset between tests** — any `describe` block that asserts on `Engine.controllerMessenger.call` call counts or arguments includes `beforeEach(() => jest.clearAllMocks())`. Without this, a shared `jest.fn()` accumulates calls across tests and makes `toHaveBeenCalledTimes(N)` assertions silently wrong. | Add `beforeEach(() => jest.clearAllMocks())` at the top of the `describe` block | +| 17 | **Tests pass** — run `yarn jest -c jest.config.view.js --runInBand --silent --coverage=false` and all tests are green. | Fix the failing test before marking the task done | +| 18 | **Format check passes** — run `yarn format:check` (or `npx prettier --check `) on the new test file and every new supporting file (renderer, preset, api-mock). | Run `npx prettier --write ` to auto-fix, then re-run the check | + +--- + +## Diagnosing Failures + +### Identify the error type first + +| Error pattern | Likely cause | Fix | +| ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `jest.mock is not allowed in *.view.test.*` | Arbitrary `jest.mock` added to test | Remove it; drive via state instead | +| `Unable to find an element with testID: xxx` | State not providing needed data, or element hidden | Add the relevant state via overrides or check rendering condition | +| `Unable to find … route-X` after press; dump still on the source screen | Held element went stale (live clock / polling re-render); `fireEvent.press` was a no-op | Re-query immediately before press: `fireEvent.press(getByTestId(...))` — never press a node captured across `await`s | +| `Unable to find an element with text: X`, but the dump shows the container and empty `pointerEvents="none"` views where the value belongs | The value is still behind a `Skeleton` — a child query / debounce has not settled, while the container rendered on the first tick | `await findByText('X')` before the sync `within(...)` asserts; the container appearing does not mean its values have loaded | +| `Cannot read property 'X' of undefined` | Preset missing a required state slice | Add `.withMinimalXController()` or override in preset | +| `Warning: An update was not wrapped in act(...)` | Async state update not awaited | Use `await waitFor(...)` | +| `No QueryClient set` | Missing provider — not in Engine mock | Add to mocks.ts or wrap with QueryClientProvider in renderer | +| Flakey number assertions | Non-deterministic exchange rates | Add `deterministicFiat: true` | +| Test passes locally, fails in CI | Time-sensitive assertions, stale press under CI load, or a sync assert on content that is still loading | Use `waitFor` / `findBy`; re-query before press when the UI re-renders on a timer; await each async-gated value | +| `Timed out in waitFor.` with no assertion detail | Nested `find*` inside `waitFor` — both share the 1s default; the outer waiter expires first | Use `await findBy*` **or** `waitFor(() => getBy*)` with `{ timeout }`; never `waitFor(async () => findBy*)` | +| Pull-to-refresh never refetches | `fireEvent(scrollView, 'refresh')` did not hit the handler | `await act(async () => { await scrollView.props.refreshControl.props.onRefresh(); })` | +| `toHaveBeenCalledTimes(N)` passes locally but is wrong in a full suite run | `Engine.controllerMessenger.call` is a shared `jest.fn()` — prior tests left calls on it | Add `beforeEach(() => jest.clearAllMocks())` to the `describe` block | +| Call-count assertion is flaky: passes without the action under test | Background prefetches (idle tab queries, cache refreshes) inflate the count in the same window | `spy.mockClear()` immediately before the action; assert the exact call signature with `toHaveBeenCalledWith(...)` instead of comparing before/after counts | +| Sheet / branch `testID` missing | Remote feature flag off in Redux; UI routes elsewhere | Override `RemoteFeatureFlagController` / preset so the gated UI mounts | + +### Inspect what's rendered + +```typescript +// Add temporarily inside the test +const { debug } = renderBridgeView(); +debug(); // prints full component tree +``` + +### Check that state data reaches the component + +Add a `console.log` in the component temporarily, or use `debug()` to confirm the Redux state is wired correctly before writing assertions. + +### Check stale presets + +When a controller's state shape changes (e.g. a new required field added to `BridgeController`), the preset becomes stale. Compare the component's actual selector usage against what the preset provides. + +--- + +## Assertion Patterns + +```typescript +// Presence / absence +expect(getByText('Label')).toBeOnTheScreen(); +expect(queryByText('Label')).not.toBeOnTheScreen(); + +// Enabled / disabled state +expect(getByTestId('cta-button')).toBeEnabled(); +expect(getByTestId('cta-button')).toBeDisabled(); + +// After interaction +fireEvent.press(getByTestId('some-button')); +await waitFor(() => expect(getByText('Result')).toBeOnTheScreen()); + +// findBy* already waits — do not wrap it in waitFor +expect(await findByText('Token A')).toBeOnTheScreen(); + +// waitFor + getBy* when you need extra time or a length check +await waitFor( + () => { + expect(getAllByTestId(MyViewSelectorsIDs.ROW)).toHaveLength(3); + }, + { timeout: 5000 }, +); + +// Navigation assertion +await findByTestId(`route-${Routes.SOME_SCREEN}`); + +// findByTestId 3rd-arg timeout (NOT 2nd arg) +await findByTestId('my-element', {}, { timeout: 3000 }); + +// Re-query before press when the target can re-render (live countdown, polling). +await findByTestId(MyViewSelectorsIDs.CARD); +await findByTestId(MyViewSelectorsIDs.LIVE_BADGE); +fireEvent.press(getByTestId(MyViewSelectorsIDs.CARD)); + +// Pull-to-refresh — call the RefreshControl handler (fireEvent 'refresh' often no-ops) +await act(async () => { + await scrollView.props.refreshControl.props.onRefresh(); +}); + +// Within a subtree — scope queries to avoid false positives when the same text or +// testID appears in multiple list items (e.g., every row shows a "price" label). +// Use within(rowElement) to constrain the query to a single row. +import { within } from '@testing-library/react-native'; +const card = getByTestId(MyViewSelectorsIDs.TOKEN_CARD_ETH); +expect(within(card).getByText('ETH')).toBeOnTheScreen(); +expect(within(card).getByText('$2,000.00')).toBeOnTheScreen(); + +// Async-gated value inside a container — await the value itself, then re-query the +// container so the scoped sync asserts run against the settled tree. +await findByTestId(MyViewSelectorsIDs.ROW_CONTAINER); +await findByText('$60'); // behind a skeleton until a child query resolves +const row = getByTestId(MyViewSelectorsIDs.ROW_CONTAINER); +expect(within(row).getByText('$60')).toBeOnTheScreen(); +expect(within(row).getByTestId(MyViewSelectorsIDs.CASH_OUT_BUTTON)).toBeOnTheScreen(); +``` + +--- + +## What NOT to Do + +```typescript +// ❌ Render scenario — no interaction, no system reaction, just static visibility +it('renders input areas and hides confirm button without tokens or amount', () => { + const { getByTestId, queryByTestId } = renderBridgeView({ overrides: { ... } }); + expect(getByTestId(SOURCE_AREA)).toBeOnTheScreen(); // render check + expect(getByTestId(DEST_AREA)).toBeOnTheScreen(); // render check + expect(queryByTestId(CONFIRM_BUTTON)).toBeNull(); // render check +}); +// More assertions does NOT make it a better test if they're all static. +// ✅ Instead: drive the test through a user interaction, Redux action, or Engine spy + +// ❌ Feature-flag gate test — render-only, even though it checks Redux-driven visibility: +// it hides the section when the master flag is disabled +it('hides the section when the master flag is disabled', () => { + renderMySection({ presetOptions: { featureEnabled: false } }); + expect(screen.queryByTestId(MySelectorsIDs.SECTION)).not.toBeOnTheScreen(); +}); +// Gate tests belong in unit tests. In CV, every test must exercise the interaction→Engine pipeline. +// ✅ If you need to verify the gate, pair it with an interaction in an existing test: +it('calls Engine when the toggle is pressed (only renders when flag is on)', async () => { + const { store } = renderMySection({ presetOptions: { featureEnabled: true } }); + await act(async () => { fireEvent(screen.getByTestId(TOGGLE_ID), 'valueChange', false); }); + expect(Engine.controllerMessenger.call).toHaveBeenCalledWith('MyController:optOut'); +}); + +// ❌ Call-count assertion that is contaminated by background prefetches +const before = spy.mock.calls.filter(([a]) => a === 'SocialService:fetchLeaderboard').length; +await triggerPullToRefresh(); +const after = spy.mock.calls.filter(([a]) => a === 'SocialService:fetchLeaderboard').length; +expect(after).toBeGreaterThan(before); // passes even if onRefresh is a no-op +// ✅ Clear the spy immediately before the action, then assert the exact call: +spy.mockClear(); +await triggerPullToRefresh(); +expect(spy).toHaveBeenCalledWith('SocialService:fetchLeaderboard', + expect.objectContaining({ chains: SPOT_CHAINS })); + +// ❌ Hardcoded i18n text in getByText / getAllByText +fireEvent.press(screen.getAllByText('Follow')[0]); +fireEvent.press(screen.getByText('Following')); +// ✅ Use strings() so tests survive locale-key renames: +fireEvent.press(screen.getAllByText(strings('social_leaderboard.follow'))[0]); +fireEvent.press(screen.getByText(strings('social_leaderboard.following'))); + +// ❌ Arbitrary mock — blocked by ESLint and runtime guard +jest.mock('../../some/hook', () => ({ useMyHook: jest.fn() })); + +// ❌ Mocking a selector +(useSelector as jest.Mock).mockImplementation(...); + +// ❌ Fake timers +jest.useFakeTimers(); + +// ❌ Snapshot assertion +expect(wrapper).toMatchSnapshot(); + +// ❌ Rebuilding the whole state from scratch +renderComponentViewScreen(MyView, { name: 'X' }, { + state: { engine: { backgroundState: { /* 200 lines */ } } }, +}); +// ✅ Instead: use a preset + minimal overrides + +// ❌ Hold a node across awaits when the UI re-renders (live countdown, polling) +const card = await findByTestId(MyViewSelectorsIDs.CARD); +await findByTestId(MyViewSelectorsIDs.LIVE_BADGE); +fireEvent.press(card); // stale under CI — press may be a no-op +// ✅ Re-query immediately before press +await findByTestId(MyViewSelectorsIDs.CARD); +await findByTestId(MyViewSelectorsIDs.LIVE_BADGE); +fireEvent.press(getByTestId(MyViewSelectorsIDs.CARD)); + +// ❌ Treat the container's arrival as proof its values have loaded +const row = await findByTestId(MyViewSelectorsIDs.ROW_CONTAINER); +expect(within(row).getByText('$50 on Yes to win $50')).toBeOnTheScreen(); // static prop — always there +expect(within(row).getByText('$60')).toBeOnTheScreen(); // still a skeleton under CI timing +// ✅ Await the gated value, then re-query the container for the scoped sync asserts +await findByTestId(MyViewSelectorsIDs.ROW_CONTAINER); +await findByText('$60'); +const settledRow = getByTestId(MyViewSelectorsIDs.ROW_CONTAINER); +expect(within(settledRow).getByText('$60')).toBeOnTheScreen(); +// Static props in the same row render immediately, so a neighbouring assert +// passing proves nothing about the gated one. + +// ❌ Nest find* inside waitFor — double polling, both default to 1s +await waitFor(async () => { + expect(await findByText('Token A')).toBeOnTheScreen(); +}); +// ✅ One waiter: findBy* already polls +expect(await findByText('Token A')).toBeOnTheScreen(); +// ✅ Or waitFor + synchronous getBy* when you need extra time or a count +await waitFor( + () => { + expect(getAllByTestId(MyViewSelectorsIDs.ROW)).toHaveLength(3); + }, + { timeout: 5000 }, +); + +// ❌ Custom nested navigator only to assert navigation occurred +extraRoutes: [{ name: Routes.FEATURE.ROOT, Component: NestedStackProbe }]; +// ✅ Default route probe when you only need to prove navigation +extraRoutes: [{ name: Routes.FEATURE.ROOT }]; + +// ❌ Filter/segmentation: analytics or refetch count only +await waitFor(() => expect(listSpy.mock.calls.length).toBeGreaterThan(n)); +expect(trackFilterSpy).toHaveBeenCalled(); +// ✅ Assert both list membership sides (Golden Rule 10) +expect(await findByText('Games market')).toBeOnTheScreen(); +fireEvent.press(getByText('Props')); +expect(await findByText('Props market')).toBeOnTheScreen(); +expect(queryByText('Games market')).not.toBeOnTheScreen(); + +// ❌ Claim optimistic title while loading when production shows a skeleton +expect(getByText(routeTitle)).toBeOnTheScreen(); // while getMarket is pending +// ✅ Assert the real pending UI, then resolve +expect(await findByTestId(MyDetailSelectorsIDs.SKELETON)).toBeOnTheScreen(); + +// ❌ fireEvent(scrollView, 'refresh') — often never calls onRefresh in RNTL +// ✅ await act(async () => { await scrollView.props.refreshControl.props.onRefresh(); }); + +// ❌ Weaken unit→CV analytics: drop tabId/filterId after deleting the full unit assert +expect(trackSpy).toHaveBeenCalledWith(expect.objectContaining({ feedId })); +// ✅ Keep the same payload specificity the unit had +expect(trackSpy).toHaveBeenCalledWith( + expect.objectContaining({ feedId, tabId, filterId, entryPoint }), +); + +// ❌ Raw string literal in getByTestId / findByTestId / queryByTestId +getByTestId('my-view-scroll-view'); +queryByTestId('confirm-button'); + +// ✅ Use the constant from the component's testIds file +import { MyViewSelectorsIDs } from './MyView.testIds'; +getByTestId(MyViewSelectorsIDs.SCROLL_VIEW); +queryByTestId(MyViewSelectorsIDs.CONFIRM_BUTTON); + +// If the testIds file does not exist yet, create it first: +// export const MyViewSelectorsIDs = { +// SCROLL_VIEW: 'my-view-scroll-view', +// CONFIRM_BUTTON: 'my-view-confirm-button', +// } as const; +``` + +--- + +## Quick Reference + +```bash +# Run component view tests +yarn jest -c jest.config.view.js --runInBand --silent --coverage=false + +# Coverage for a feature folder +yarn test:view:coverage:folder app/components/UI/MyFeature + +# Lint check +yarn eslint +``` + +**Key locations:** + +| What | Where | +| ------------------------------ | -------------------------------------------------------------- | +| Engine + native mocks | `tests/component-view/mocks.ts` | +| render, renderScreenWithRoutes | `tests/component-view/render.tsx` | +| StateFixtureBuilder | `tests/component-view/stateFixture.ts` | +| HTTP API mocks (nock) | `tests/component-view/api-mocking/` (per-feature) | +| Feature renderers (per view) | `tests/component-view/renderers/` (e.g. bridge, wallet) | +| Feature presets (per view) | `tests/component-view/presets/` (e.g. bridge, wallet) | +| DeepPartial type | `app/util/test/renderWithProvider` | +| Routes | `app/constants/navigation/Routes.ts` | +| Skill + rules | `.agents/skills/mms-mobile-testing/SKILL.md` and `.agents/skills/mms-recipe-cook/references/mobile-testing/component-view/` | diff --git a/domains/perps/skills/recipe-cook/references/qa-example.md b/domains/perps/skills/recipe-cook/references/qa-example.md new file mode 100644 index 00000000..467907b7 --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/qa-example.md @@ -0,0 +1,141 @@ +# Review PR worked example + +This example is illustrative. Rediscover every action, field, selector, and recipe ref in the active checkout. + +## PR claims and frozen review contract + +PR body: + +> When the user changes order type to Limit, the limit-price field is visible and Market remains available. + +Changed UI code adds `data-testid="order-type-toggle"` and `data-testid="limit-price-input"`. + +Freeze this contract: + +| # | Acceptance criterion | Source | Proof mode | Planned proof | Preconditions | +|---|---|---|---|---|---| +| 1 | When the user changes order type to Limit, the limit-price field is visible and Market remains available. | authored: PR body | visual | Select Limit; exact wait/assert for limit-price input; screenshot | Review account already has no matching BTC position or order on the selected testnet | + +```text +Adapter: extension +Proof profile: extension-cdp-live +Recipe decision: generate-ui +Target environment: fullscreen +Known limitations: none + +Continue from this frozen basis. +``` + +## Recipe skeleton after contract freeze + +Assume current discovery confirms `metamask.perps.start_state`, `metamask.perps.teardown_state`, `ui.press`, `ui.wait_for`, and `ui.screenshot` with the fields below. The selector values came from this PR's diff. + +```json +{ + "$schema": "https://farmslot.io/schemas/recipe-v1.schema.json", + "description": "Prove the Limit order form acceptance criterion.", + "paramsSchema": { + "type": "object", + "additionalProperties": false + }, + "workflow": { + "entry": "setup-clean-market", + "teardown": "teardown-clean-market", + "nodes": { + "setup-clean-market": { + "action": "metamask.perps.start_state", + "network": "testnet", + "market": "BTC", + "page": "market_details", + "positions": { + "state": "none" + }, + "orders": { + "state": "none" + }, + "intent": "Open BTC Perps and assert that the required testnet state already holds.", + "next": "ac1-select-limit" + }, + "ac1-select-limit": { + "action": "ui.press", + "test_id": "order-type-toggle", + "intent": "Select the Limit order form.", + "next": "ac1-assert-limit-price" + }, + "ac1-assert-limit-price": { + "action": "ui.wait_for", + "test_id": "limit-price-input", + "expected": "visible", + "intent": "Confirm the Limit price field is visibly available.", + "next": "ac1-assert-market-option" + }, + "ac1-assert-market-option": { + "action": "ui.wait_for", + "test_id": "market-order-option", + "expected": "visible", + "intent": "Confirm the Market order option remains visibly available.", + "next": "ac1-screenshot-limit-form" + }, + "ac1-screenshot-limit-form": { + "action": "ui.screenshot", + "path": "evidence-ac1-limit-form.png", + "label": "AC1 — Limit price field visible", + "intent": "Preserve the visible Limit form as reviewer evidence.", + "next": "done" + }, + "done": { + "action": "end", + "status": "pass" + }, + "teardown-clean-market": { + "action": "metamask.perps.teardown_state", + "market": "BTC", + "positions": { + "state": "none" + }, + "orders": { + "state": "none" + }, + "intent": "Assert that the final BTC Perps state still holds after proof.", + "next": "teardown-done" + }, + "teardown-done": { + "action": "end", + "status": "pass" + } + } + } +} +``` + +Extension `start_state` and `teardown_state` only navigate, observe, and assert; +they do not create, close, or cancel financial state. Establish mutable financial +preconditions through an explicitly authorized funded leaf or outside the proof, +then state that dependency in the frozen contract. The selector values must come +from the current PR and action discovery. The revert test is: reverting the PR +must make at least one AC assertion fail. + +## Example final report + +```markdown +## QA review + +**PR:** https://github.com/MetaMask/metamask-extension/pull/12345 +**Validated head:** 0123456789abcdef0123456789abcdef01234567 +**Platform/proof profile:** extension / extension-cdp-live +**Overall verdict:** PASS +**Reviewer recommendation:** READY FOR HUMAN REVIEW +**Run evidence:** temp/tasks/recipe-cook//artifacts/recipe-run + +| # | Acceptance criterion | Source | Proof mode | Recipe nodes | Evidence | Verdict | Rationale | +|---|---|---|---|---|---|---|---| +| 1 | When the user changes order type to Limit, the limit-price field is visible and Market remains available. | authored: PR body | visual | `ac1-select-limit`, `ac1-assert-limit-price`, `ac1-assert-market-option`, `ac1-screenshot-limit-form` | `evidence-ac1-limit-form.png`; `trace.json` | PASS | Both exact visibility assertions executed successfully, and the screenshot shows both controls in the Limit form. | + +**Trace:** 8/8 nodes passed; 0 failed. +**Untestable:** none +**Notes:** The recipe established and restored a clean BTC testnet market state. No PR-specific runtime warnings were recorded. + +### Validated findings + +- None. +``` diff --git a/domains/perps/skills/recipe-cook/references/qa.md b/domains/perps/skills/recipe-cook/references/qa.md new file mode 100644 index 00000000..102a139e --- /dev/null +++ b/domains/perps/skills/recipe-cook/references/qa.md @@ -0,0 +1,357 @@ +# Runtime QA flow + +This is Recipe Cook's `qa` execution contract. Keep product code unchanged. Use `gh` for PR facts and `mm-harness` as the only runtime front door. Runtime setup and task-owned test state may be created within the caller's existing authority; QA is not inherently limited to read-only app interactions. Never post to the PR or invent a runner action. + +Run phases in order. A failed shared prerequisite blocks dependent cases; preserve its evidence and report the exact recovery command or operator decision. A case failure does not block independent cases with verified prerequisites and safe teardown. Keep the failed criterion in the report, classify it through the locked harness's drift guidance, and continue independent coverage without routing around its assertion. + +Serialize harness runtime commands: launch, fixtures, call, run and doctor. On `SANDBOX_BUSY`, wait for this task's lock-owning PID to exit before retrying; report a foreign or hung owner as a blocker with its PID. List the frozen scope's commits with `git log` once to map changed behavior to criteria. Mark steps after all their subprocesses exit. Inspect only secret-free fixture metadata: labels, addresses and networks. + +## Artifact contract + +State this contract before runtime work: + +```text +/artifacts/ + qa-scope.json frozen change scope from recipe-qa + qa-suite/ standard Recipe v1 suite scope/result and recipe packages + qa-proof.json portable scope/suite digests, package paths and smoke target + qa-result.json the same proof bound to the task (written only when inputs/qa.json exists); never hand-authored + recipe.json review AC proof graph, when generated + recipe-plan.json mm-harness plan output + recipe-run/ report, summary, trace, diagnostics, executed + recipe, artifact manifest, and evidence + recipe-coverage.md AC -> mode -> planned proof -> nodes -> evidence -> verdict + evidence-manifest.json strict reviewer-facing evidence selection + recipe-quality.json Recipe Cook quality verdict + report.html single human QA report + report-data.json renderer input from the same retained evidence + pr-verdicts.json per-PR rollup from the shared HTML renderer + review-reply.md checklist-formatted proposed PR reply + review.md portable copy of review-reply.md + line-comments.json validated changed-line findings; never posted + learnings.md reusable, secret-free workflow learnings +``` + +For visual ACs, require AC-named PNGs and request the full recipe video. For state/log ACs, require trace entries plus the exact state, command output, or watched log evidence. Do not claim an artifact that the run package does not contain. + +For Extension, record the driven view and viewport: `home.html`, side panel or popup. Establish the requested display mode during setup through the locked harness, and check for unintended duplicate views before proof. Use the declared `ui.capture_surface` for report evidence of the driven document; a native `ui.screenshot` may include both the tab and an unrelated side panel. Inspect the actual PNG and label its target and observation limits. Full-document capture does not prove viewport clipping, and resizing `home.html` does not prove popup or side-panel behavior. Preserve ambiguous or stale captures as gaps; obtain fresh evidence in a new retained run rather than cropping or replacing the original to imply a clean run. + +Every relative review artifact named below lives under `/artifacts/`. + +Cook locks and records the harness during materialization. Use its task-local front door for every harness command so PATH cannot bypass that provenance: + +```bash +HARNESS="/mm-harness" +test -x "$HARNESS" +``` + +## Observable case execution + +For `qa/default`, use the task runtime's existing child checklist for selected feature cases. Other explicit templates retain their own checklist unless they declare a child binding. After freezing the plan, write `inputs/qa-cases.md`: one flat checkbox per selected case, in execution order, with a readable case/PR name and criterion references. Register one `qa-cases` child on the parent feature-execution step (step 5 in `qa/default`) with `/mark sub start qa-cases --step --from /inputs/qa-cases.md`. Keep the parent numbering unchanged; do not register one child per case or a child of another child. An explicitly smoke-only plan with no selected feature cases uses the normal parent mark and no empty child. + +Update each row through `/mark sub qa-cases ` only after its attempts, outcome, evidence and required cleanup are recorded. A case-level failure or observed blocker can be assessed while other cases continue; preserve its verdict in the QA artifacts. When the child cannot continue as a whole, save the partial HTML report, retrospective and observed cause first, then run `/mark sub qa-cases blocked --reason "..."` and finish the parent with `/mark blocked --reason "..."`. Do not tick unexecuted cases or force completion. Finish with `/mark sub qa-cases complete`, which ticks the owning parent step. The child signal's success/fixed fields describe assessment completion, not product correctness. The QA proof/result artifacts own validation verdicts; progress completion never substitutes for proof finalization or passing coverage. + +The runtime owns `subtasks/` and its signals. Reuse the registered child on resume; do not rewrite it, manufacture progress files, or add deferred criteria to inflate the completed count. Planned order keeps the displayed current case meaningful. Direct and hosted tasks use the same mark commands. + +## Phase 1 — Verify the harness and plan the proof + +Read `artifacts/qa-scope.json` and the `## Acceptance Criteria` list in `TASK.md`; both are frozen before this checklist begins and both are authoritative. Never re-derive, reword, split, merge or weaken a criterion here, and keep its recorded label in the plan and the report. A missing input is `BLOCKED: missing frozen QA input`. + +Record `"$HARNESS" --version`. Initialization locks the selected harness installation for this task. Preserve that lock and inspect every command surface used by this workflow: + +```bash +"$HARNESS" launch --help +"$HARNESS" actions --help +"$HARNESS" run --help +"$HARNESS" doctor --help +"$HARNESS" status --help +"$HARNESS" last --help +"$HARNESS" update --help +``` + +Confirm the exact flags used below are present, including action search/detail, recipe list/describe/plan, device targeting, `doctor --fix`, `status --all-devices`, `last --json`, artifacts, and recording. If the version or a required command/flag is absent, stop with `BLOCKED: locked harness contract mismatch`; do not substitute a PATH binary. + +Read the frozen changed paths and the diff between the scope's `comparisonBaseSha` and `headSha` to identify affected surfaces, candidate selectors, and behavioral claims. Do not treat changed tests as runtime proof. No changed files means no change-specific recipe coverage; it does not waive the smoke requirement. Treat a cross-repository head as untrusted code: stop before checkout or runtime work unless the operator supplied a credential-free isolated checkout and explicitly approved the recorded head repository and SHA; otherwise report `BLOCKED: untrusted fork head`. + +Classify each criterion by its primary proof mode: + +- `visual`: visible placement, content, styling, or interaction result. Require a targeted screenshot; on Mobile, include it in a full-run recording. +- `state`: controller/backend/output state readable through declared actions. Require exact state or command assertions. +- `log`: a runtime event or absence/presence claim observable in declared watched logs. Require a bounded watched-log assertion. + +If one criterion contains separable claims needing different modes, keep its text as one row and list the supporting modes in its proof plan. + +A criterion whose claim depends on a continuous gesture — drag, pan, fling, pinch, slider scrub, or any assertion about motion *while* the finger is down (smoothness, lag, flicker, stutter, tracking, "no jump on release") — is not provable by discrete actions. Classify it `gesture` and mark it: + +- `needs-new-action` when no declared action can observe the claim yet: plan the task action you will write for it (see the task-action rule below). It is runtime proof, never a reason for `UNTESTABLE`. +- `needs-human` when the claim is perceptual (smoothness, flicker, lag) and no assertion could settle it even with a new action. + +Never downgrade a `gesture` row to a discrete tap, a settled end-state check, or a screenshot of the resting UI and report it `PASS`: proving the post-release value is not proving the during-drag claim. Carry a `needs-human` row into the plan and the final report as `UNTESTABLE` with its exact reason; a `needs-new-action` row gets the verdict its task action's assertion earns. Either way, state which weaker discrete claim (if any) was actually proved. Separable discrete sub-claims from the same criterion may be proved on their own row. + +Choose one proof profile: + +- `mobile-device-recorded`: one selected iOS/Android target, AC screenshots, and `--record-video=full-run`. +- `extension-cdp-live`: a harness-owned browser/CDP runtime and AC screenshots for visual claims. +- `core-controller-live`: the configured Core controller/command runtime, observed through declared recipe actions and assertions. + +Choose one recipe decision: + +- `generate-ui`: at least one criterion is primarily visual. +- `generate-internal`: the criteria are state/log proof with no necessary visual claim. +- `blocked-`: a verified unavailable runtime, capability or authorized target prevents the smoke path. A docs-only, test-only or empty change set does not waive explicitly requested QA smoke. + +Backend-only is never by itself a skip reason. Use `generate-internal` when `command`, `assert_exit_code`, `assert_output`, `assert_json`, `watch_logs`, or a product action can prove it. + +This decision is provisional until post-checkout action discovery. Do not claim a missing action before querying the active manifest. + +Write the plan as the first version of `recipe-coverage.md`, one row per criterion: + +| # | Acceptance criterion | Source | Proof mode | Planned proof | Preconditions | +|---|---|---|---|---|---| + +Also record adapter, proof profile, recipe decision, changed surfaces, target device/environment, and every known limitation. The audit phase completes the same table with nodes, evidence and verdicts. + +Full-run recording is not available for every platform. `"$HARNESS" run --help` proves only that the flag exists; support is proven by the actual target run producing its video artifact. For Android visual criteria, request `--record-video=full-run` on the recipe run and disclose the result. Continue when the artifact is produced. If the adapter returns a structured unsupported or provider-unavailable result, mark the visual criterion `UNTESTABLE`/`BLOCKED` with that exact reason; do not substitute `adb`, another recorder, or unrecorded visual proof. Never pre-block an Android-specific visual criterion solely because it targets Android. Android-only state/log criteria may run without video when their declared evidence fully observes the claim. + +Write the plan before checkout or runtime work, then continue in the same run. If the decision is `blocked-`, do not check out or launch: write one `UNTESTABLE` coverage row per criterion with that exact limitation, emit a `BLOCKED` report, and recommend the named static/check workflow instead. + +## Phase 2 — Prepare the exact frozen head + +For range, window or ref scope, the entry point has already put the clean, owned checkout at `qa-scope.json.headSha` with the operator's consent; verify HEAD equals that SHA and stop if it does not. Do not resolve the original moving ref again. Continue with steps 4 onward below. For Core, use the harness's declared controller/command runtime and omit mobile/browser launch commands. The following PR refresh/checkout steps apply only to PR scope. + +1. Require a clean tree in this checkout. If `git status --short` is non-empty, stop and ask the operator to commit or stash; never stash or discard for them, and never move to another checkout, worktree or clone. +2. For PR scope, re-fetch the head metadata immediately before checkout. If `headRefOid` differs from the frozen SHA, retain the contract and report `BLOCKED: PR head changed during validation`. A new task may validate the new head; never retarget this task or overwrite its frozen scope. +3. Verify `git rev-parse HEAD` equals the frozen SHA from `qa-scope.json`; the entry point performed the checkout with the operator's consent. If it differs, stop and report; never substitute a moving branch, base branch or merge commit for the frozen head. +4. Read changed production files at the frozen head before deciding how to exercise them. For deletions, inspect the file at its recorded before revision and validate the intended removal. +5. For Mobile, run `"$HARNESS" status --all-devices --json`. Use the selected target. If it is unavailable or ambiguous, stop and ask for one iOS/Android target. +6. Launch the checked-out PR through the published harness: + + ```bash + # Run exactly one Mobile command for the selected target + "$HARNESS" launch ios --device "" --verify --json + "$HARNESS" launch android --device "" --verify --json + + # Extension; add --sidepanel only when the review contract requires it + "$HARNESS" launch --verify --json + ``` + + When `qa-scope.json` carries `remoteFlagSpec`, append `--remote-flag ""` to every launch above, including any `--build` rerun. The harness pins the variant and fails the launch if the runtime does not hold it (`REMOTE_FLAG_NOT_APPLIED`), so the launch's own `remoteFeatureFlags` output is the proof; record it in the report. + + Let launch decide whether the current app build can be reused. If its next action says the build is missing or stale, run those exact launch arguments through `"$HARNESS"` with `--build`, and no product build command of your own devising. + + Dependency preparation is the exception, because the harness cannot do it: launch refuses a checkout whose `node_modules` tree is incomplete and names the checkout's own setup command (for Mobile, `yarn setup:expo --no-build-ios --no-build-android`). Run the command launch names, verbatim, then re-run the same launch arguments. Do not substitute a different install or build command, and do not treat this as licence to build the product yourself. + + If launch returns any other recovery action, follow it once through `"$HARNESS"` or stop with the exact blocker. Never request or expose a recovery phrase, private key, or password. + +7. Establish the test plan's starting state before proof. For domain QA that requires an existing wallet, apply the approved fixture through `"$HARNESS" fixtures set --json` on this owned isolated profile, or verify an already matching wallet with declared wallet/account, feature-flag and domain-network reads. Extension `fixtures set` resets the profile and relaunches from prelaunch E2E state, so the earlier launch receipt is no longer current. When the frozen scope has `remoteFlagSpec`, append `--remote-flag ""` to `fixtures set` or `reset` too. Retain the fixture result and post-setup runtime/account/network/flag observations before smoke; use `metamask.feature_flags.read` for flag readback. UI `wallet.import` tests a different onboarding journey and may leave consent/tutorial gates; do not substitute it for ordinary fixture setup. Onboarding, migration, account creation and first-time feature criteria instead keep their declared unfinished state. Fixture setup does not authorize financial mutations. After proof starts, do not reseed the active attempt: preserve its packages, record any necessary reseed as a new explicit setup/lifecycle phase, re-establish runtime identity and rerun smoke before dependent cases. + +## Phase 3 — Discover the capability boundary + +Run discovery from the PR checkout: + +```bash +"$HARNESS" actions --json +"$HARNESS" run --list --json +``` + +For each proposed action or reusable recipe, narrow to the exact current contract: + +```bash +"$HARNESS" actions "" --json +"$HARNESS" actions --action "" --json +"$HARNESS" run "" --describe --json +``` + +The discovered list is law, including team/custom actions. Never recall names, fields, selectors, recipe refs, assertion operators, or platform support from memory. When no declared action can observe an AC, write a task action under a new name, modeled on a library action (`"$HARNESS" actions --action --json` shows its `sourceManifest`) but self-contained (node built-ins, plus any helper copied next to it): keep its manifest in `/artifacts/task-actions/manifests/` and its script in the sibling `/artifacts/task-actions/actions/`, pass `--action-manifest ` to every `"$HARNESS"` `actions`, `run --plan`, `run` and `call`, and flag it in the report as a library candidate. A missing declared action is never a reason for `UNTESTABLE`. + +Launch and the supported fixture setup above are the runtime bring-up steps. Do not call Cook's sandbox preparation path. The recipe itself must use discovered actions to establish and assert every required start-state prerequisite before a mutation or user-flow node. If the active manifest cannot deterministically establish a safe wallet, account, network, permission, or page state required by the review, mark the affected AC `UNTESTABLE` or stop `BLOCKED: non-deterministic runtime prerequisite`; never mutate arbitrary pre-existing state. Feature-flag state is not such a prerequisite: a variant the review needs is pinned at launch from the scope's `remoteFlagSpec`, read back with `metamask.feature_flags.read`, and changed on Mobile mid-run with `metamask.feature_flags.set` / `.clear` (the Extension needs a relaunch). A flag the scope does not name is a report finding naming the flag and the arm, never a hunt for undeclared overrides. + +Re-evaluate the recipe decision against discovered capabilities. If it changes, update `recipe-coverage.md`, name the capability-driven change, and continue. + +## Phase 4 — Author and plan the recipe + +Use the frozen changed paths and criteria to select a dynamic set of discovered library recipes. Record each selected recipe's reason, covered criteria and source digest from the harness plan. Prefer composition or parameterization; author a missing path only when no existing recipe covers it. Preserve coverage gaps explicitly. + +Keep the full frozen change/criterion inventory, then choose a risk-based execution plan for the requested assurance level. PR bug verification may require each acceptance criterion; daily/release integration validation normally combines essential domain smoke with focused checks for risky changes and cross-feature interactions. Do not recreate every prior PR recipe by default. For each selected proof target record its user situation, reusable setup/action recipes, assertion and task-owned cleanup. Record deliberate exclusions with risk/prior-evidence rationale before execution, without deleting or weakening authored criteria. Group compatible cases to reuse setup, but isolate conflicting or destructive states. An authorized prerequisite belongs in setup, not the blocker list. Compose existing primitives when no complete recipe exists. + +Preserve the criterion's actual difficulty too. Do not add a stronger timing, platform or fixture requirement and then cite that invented requirement as a blocker. “Double-tap” requires two rapid user inputs and evidence of their effect; it does not automatically require two handler calls in the same JavaScript tick. Record the implemented input timing and check it against the authored claim. Likewise, a UI warning or slider interaction should not require submitting an order unless the claim depends on submission. + +After smoke, execute the selected independent cases whose prerequisites can be established. If a case is not executable, attempt the smallest supported setup or capability check that determines why. Retain the result, not just “needs a fixture.” Use bounded retries for observed failures and continue other cases. A genuine unavailable capability, incompatible build, missing authority or failed prerequisite may block proof. An unfinished recipe, unattempted setup or exhausted run budget is nonexecution, not an inherent app limitation. Preserve the existing machine verdict vocabulary and add this precise coverage reason to the report; never improve a coverage number by weakening a criterion. + +Every successful QA execution includes a real smoke path on the selected app, or live controller/command assertions for Core. Launch/health checks alone are insufficient. Plan, execute and validate the reusable smoke package before spending time on change-specific helpers or exploratory probes. Then compose independent cases around the recorded criteria so an early failure cannot hide unrelated coverage. Include smoke even when the change set is empty; change-specific checks cannot replace it. + +Declare root `proofTargets` and bind each to portable `assert_output`, retained-state `assert_json`, or live `ui.wait_for` proof. Domain assertions return structured results. Use `assert_output` with its `source` and `assert` object to check them, for example `{"source":"assert-positions","assert":{"path":"$.matchingCount","operator":"eq","value":0}}`. Its `contains` and `match` options inspect stdout/stderr strings; do not use them for a structured action result. Keep composed actions within the same recipe run rather than nesting another harness invocation for the same checkout. + +Compose the selected recipes into the task's Recipe v1 proof graph using the installed harness's documented composition mechanism. Preserve its resolved recipe/source digests and verify the execution used those same inputs. Keep the selected recipe list in the frozen task evidence. + +Write `recipe.json` as Recipe Protocol v1 with `$schema`, `description`, root `paramsSchema`, and `workflow.{entry,nodes}`. Enforce: + +- Every AC node ID starts `ac-`; plumbing starts `setup-`, `gate-`, or `teardown-`. +- Every non-terminal node has a concise human-facing `intent`. The validator rejects an intent that restates the action, the node ID, a selector, or a generic verb (`workflow.invalid_intent`). Write the user-visible goal — "The trade screen is ready for the trader to choose an amount", not "Wait for the slider to mount". +- Every selected proof target has a planned assertion and start-state composition, or an evidence-backed unavailable prerequisite. Full-scope ACs not selected for a smart smoke run retain their explicit exclusion rationale; they are not labelled runtime blockers or certified by related smoke. Planned but unattempted cases remain execution gaps, not deliberate exclusions added after the fact. +- Visual evidence paths use `evidence-ac-.png`. +- Use only exact discovered action names and fields. +- Use `eq`/exact equality rather than `contains`; use meaningful thresholds rather than existence checks. If a current watched-log action exposes only substring matching, require the full stable log line and disclose that weaker operator in the report. +- Use `workflow.teardown` for cleanup that must run after success or failure. +- Identify every node that mutates wallet, network, order, transaction, or persistent app state. Record its reset/teardown path before execution; absence of a proven reset makes any post-mutation replay ineligible. +- Never use a default branch to route around an AC assertion, a manual step, or a delay as a substitute for a declared wait action. +- For visual ordering/placement claims, combine a state/geometry assertion with a screenshot. +- Ask: “Would this assertion fail if the PR change were reverted?” If not, strengthen it. + +Plan without touching the runtime. Run exactly one command matching the frozen adapter and selected Mobile target: + +```bash +# Mobile +"$HARNESS" run /artifacts/recipe.json --plan --adapter mobile --platform ios --device "" --json +"$HARNESS" run /artifacts/recipe.json --plan --adapter mobile --platform android --device "" --json + +# Extension +"$HARNESS" run /artifacts/recipe.json --plan --adapter extension --json + +# Core +"$HARNESS" run /artifacts/recipe.json --plan --adapter core --json +``` + +Save machine stdout as `recipe-plan.json`, diagnostic stderr separately, and the harness process's own exit code; do not pipe the command through `tail` or mix logs into JSON. Use the installed harness's proof-preflight option when its help advertises one. This checks structural proof bindings only; it does not verify executed branches, runtime outcomes, or artifact indexing. Exit 5 or any schema/action error means the recipe is invalid: fix only against current discovery output, then plan again. If the CLI lacks a needed capability, stop and report the gap; never fake a plan. + +## Phase 5 — Execute on the prepared runtime + +Validate the first executed smoke package with the installed QA finalizer's `--check-only` mode and a smoke-only index before expanding the suite. This performs the same package, containment and runtime-proof checks without writing the final suite or QA result. Run this only after the smoke package exists; a runtime-proof failure is distinct from a recipe-plan schema error. + +Phase 2 already launched and verified the checked-out PR app. Do not relaunch a healthy matching runtime. If liveness fails, run `"$HARNESS" last --json`, preserve its evidence paths, and stop with the returned next action. Do not loop or borrow another checkout's runtime. + +Execute: + +```bash +# Mobile +"$HARNESS" run /artifacts/recipe.json --adapter mobile --platform ios --device "" --artifacts-dir /artifacts/recipe-run --record-video=full-run --json +"$HARNESS" run /artifacts/recipe.json --adapter mobile --platform android --device "" --artifacts-dir /artifacts/recipe-run --record-video=full-run --json + +# Extension +"$HARNESS" run /artifacts/recipe.json --adapter extension --artifacts-dir /artifacts/recipe-run --json + +# Core +"$HARNESS" run /artifacts/recipe.json --adapter core --artifacts-dir /artifacts/recipe-run --json +``` + +Run exactly one command matching the selected platform. + +Do not retry a deterministic state mutation with different values. Before any rerun after a partial execution, read `trace.json`. If a mutation node succeeded, first execute and prove its declared reset/teardown. If no reset exists or reset evidence fails, preserve the first run and stop with `BLOCKED: unsafe mutation replay`; never replay the whole recipe merely with the same values. + +Bound exploratory diagnosis to a concrete causal question. Once a focused reproduction cannot distinguish product behavior from observation or environment, retain the uncertain finding and move to independent cases. Do not retry until green or claim a live-price discrepancy from unsynchronized captures as a reproduced regression. + +Preserve the first failure and classify it: + +- Product assertion failure -> continue to the evidence audit and report `FAIL`. +- Recipe/schema failure -> correct only from `actions --action` output, rerun `--plan`, then execute once. +- Fixture/runtime failure -> run `"$HARNESS" doctor --fix --json` and follow its exact next action. +- Harness failure -> run `"$HARNESS" last --json`, report the gap, and stop. + +Every reproduction, retry or narrowing of a failure is a recipe run: extend `recipe.json` (or a variant beside it) and execute it with `"$HARNESS" run` into its own `--artifacts-dir` under `/artifacts/`, so it is retained. Bare `"$HARNESS" call` invocations are for discovering and validating a node while authoring; they are never evidence. The one exception is a capture the report will cite (`ui.screenshot`, a state read): run it with `--artifacts-dir /artifacts/calls/