diff --git a/BUGS.md b/BUGS.md index dc494a8..c6a4cdf 100644 --- a/BUGS.md +++ b/BUGS.md @@ -286,3 +286,29 @@ entries' IDs; manual edits to fix typos are fine. - **Severity**: low - **Proposed fix**: Re-run the smoke corpus a few times to measure the rate; if it persists, have the coordinator check its dispatch count against policy.json's runnable lenses before Step 5, or record per-lens agent ids in run so a packed dispatch is visible in the report. +## BUG-35: --launch-env check ignores the default LIKEUSER credentials +- **Observed**: 2026-10-13 +- **File**: skills/using-apps-like-a-user/scripts/likeuser.py:981 +- **Description**: cmd_begin loads secret_values only when --credentials-env or --credentials-file is given. A begin with neither still redacts the default LIKEUSER_* values, but _refuse_credential_launch_env and _launch_env never see them, so begin --launch-env PUBLIC_COPY passes a variable whose value equals the default password through to the launched Electron app. Reproduced by the targeted review of f41f461..c337402. +- **Introduced by**: 52dc460 +- **Severity**: high +- **Proposed fix**: Load the default credential source before the --launch-env check and add both values to secret_values and run.secrets when no explicit source is given. +- **Blocker for**: Electron runs that pass --launch-env while LIKEUSER_* is exported + +## BUG-36: Electron observe swallows a failed native screenshot observation while a sheet is open +- **Observed**: 2026-10-13 +- **File**: skills/using-apps-like-a-user/scripts/likeuser.py:478 +- **Description**: _observe_surface now raises when the first native observation of an Electron run fails, but the second native observation that captures the sheet screenshot still catches DriverError and returns a successful snapshot with native_error set and no screenshot. The 'a native failure never reads as no sheet' fix is incomplete on this path. +- **Introduced by**: 52dc460 (partial fix) +- **Severity**: high +- **Proposed fix**: Let DriverError from the screenshot-bearing native observation propagate so observe exits 4. +- **Blocker for**: Electron runs with native sheets open + +## BUG-37: A failed delete of a password-bearing screenshot is reported as withheld +- **Observed**: 2026-10-13 +- **File**: skills/using-apps-like-a-user/scripts/likeuser.py:451 +- **Description**: _observe deletes a screenshot that shows the password under contextlib.suppress(OSError) and then marks the snapshot screenshot_withheld unconditionally, so a failed unlink leaves the file in the run folder while the output says it was withheld. +- **Introduced by**: pre-dates 52dc460 (text-match withholding) +- **Severity**: medium +- **Proposed fix**: Raise when unlink fails and mark screenshot_withheld only after confirming the path is gone. +- **Blocker for**: credential handling guarantee in references/safety.md diff --git a/DEFERRED.md b/DEFERRED.md index 16e0374..c163919 100644 --- a/DEFERRED.md +++ b/DEFERRED.md @@ -157,3 +157,50 @@ Reviewed every sprint planning. Use `/quirk:artifacts:defer` to append. - **Estimated effort**: S - **Priority**: P3 +## DEFER-19: Per-app memory for using-apps-like-a-user +- **Deferred**: 2026-10-11 +- **Session context**: brainstorming quirk:using-apps-like-a-user (docs/quirk/specs/2026-10-11-using-apps-like-a-user/logic.md) +- **Why deferred**: v1 starts every run fresh. Saved routes or "remember this" notes go stale silently because the drivers exit 0 on stale targets, and first-time-user eyes are the point of dogfooding. Revisit once the effect-status guard can re-verify a saved route before reuse. +- **Estimated effort**: M +- **Priority**: P3 +- **Proposed owner**: unassigned + +## DEFER-20: Replayable test scripts from using-apps-like-a-user runs +- **Deferred**: 2026-10-11 +- **Session context**: brainstorming quirk:using-apps-like-a-user +- **Why deferred**: Turning a successful run into a Playwright or native replay script makes the skill a test framework, and a replay built from refs or coordinates would need per-step assertions to avoid false passes. Out of v1 scope. +- **Estimated effort**: L +- **Priority**: P4 +- **Proposed owner**: unassigned + +## DEFER-21: Checkpointed resume for half-finished using-apps-like-a-user runs +- **Deferred**: 2026-10-11 +- **Session context**: brainstorming quirk:using-apps-like-a-user +- **Why deferred**: Resuming from a checkpoint needs persisted state, a format, and invalidation rules. v1 covers the cheap case by inspecting current state and doing only what is missing. +- **Estimated effort**: M +- **Priority**: P4 +- **Proposed owner**: unassigned + +## DEFER-22: iOS and Android simulator support for using-apps-like-a-user +- **Deferred**: 2026-10-11 +- **Session context**: brainstorming quirk:using-apps-like-a-user +- **Why deferred**: v1 covers web, Electron, and native macOS. Orca ships separate emulator guides (orca-emulator, orca-emulator-android) that a later version could route to. +- **Estimated effort**: M +- **Priority**: P3 +- **Proposed owner**: unassigned + +## DEFER-23: Stop hotkey for foreground using-apps-like-a-user runs +- **Deferred**: 2026-10-11 +- **Session context**: brainstorming quirk:using-apps-like-a-user +- **Why deferred**: macOS cannot reliably tell the user's input from synthetic events, so v1 detects takeover with a cursor and frontmost-app drift check. A dedicated stop hotkey needs Input Monitoring and an event tap, which can silently disable itself after re-signing. +- **Estimated effort**: S +- **Priority**: P3 +- **Proposed owner**: unassigned + +## DEFER-24: OpenAI Decisions API as a cheap screenshot verifier +- **Deferred**: 2026-10-11 +- **Session context**: brainstorming quirk:using-apps-like-a-user +- **Why deferred**: The Decisions API accepts images and returns bounded predicate/choice answers, which could answer "is an error banner visible?" cheaply. Its accuracy on UI screenshots is unmeasured, and v1 needs no second model provider. +- **Estimated effort**: M +- **Priority**: P4 +- **Proposed owner**: unassigned diff --git a/README.md b/README.md index 42a6218..2a5c22a 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Version: **2026.7.9** ## What ships -- **27 skills** under `skills/` covering TDD, systematic debugging, brainstorming (which produces a logic spec), deep-research/brainstorming exploration (with a `--involve` user-inclusion dial: checkpoints, mid-run steering, and idea-gate co-creation), divergent ideation, the two-document spec rubric (logic spec + tech spec) and plan writing, scannable-prose revision, code review, a full multi-lens pre-merge review for GitHub and GitLab, adversarial review (typed profiles, a promote/refute two-stage, and an evidence gate), parallel agent dispatch, guided bug/feature/code-change filing with provenance-marked artifacts, gating simplification behind a correctness check, and more. +- **28 skills** under `skills/` covering TDD, systematic debugging, brainstorming (which produces a logic spec), deep-research/brainstorming exploration (with a `--involve` user-inclusion dial: checkpoints, mid-run steering, and idea-gate co-creation), divergent ideation, the two-document spec rubric (logic spec + tech spec) and plan writing, scannable-prose revision, code review, a full multi-lens pre-merge review for GitHub and GitLab, adversarial review (typed profiles, a promote/refute two-stage, and an evidence gate), parallel agent dispatch, guided bug/feature/code-change filing with provenance-marked artifacts, gating simplification behind a correctness check, driving web, Electron, and native macOS apps like a user with a no-false-pass guard script, and more. - **Typed artifacts** (added in 5.1.0) — surface-routing for observations Claude cannot act on. See below. ## Installation diff --git a/docs/quirk/specs/2026-10-11-using-apps-like-a-user/bakeoff.md b/docs/quirk/specs/2026-10-11-using-apps-like-a-user/bakeoff.md new file mode 100644 index 0000000..88e8488 --- /dev/null +++ b/docs/quirk/specs/2026-10-11-using-apps-like-a-user/bakeoff.md @@ -0,0 +1,106 @@ +# Native-driver bake-off + +Date: 2026-10-12 +Machine: macOS 26.6.2 (build 25G83) on MacBookPro18,2, Stage Manager on +Versions: Peekaboo 4.9.1, Cua Driver 0.34.1 (`cua-driver`, app `/Applications/CuaDriver.app`), `orca computer` 1.4.216-rc.0.zy03 + +**Winner:** cua + +background_capable: true +Version pin: `cua-driver 0.34.1` (Homebrew cask `cuadriver`) +`NATIVE_DRIVER_PREFERENCE` for `_native.py`: `["cua", "orca"]` +Runs: 2 full passes, both winners cua, so no third pass. Rule applied: zero focus steals and every input landed; only Cua Driver met it. + +Harness: `python3 skills/using-apps-like-a-user/scripts/bakeoff.py --runs 2`, 10 clicks on "Working button" and 10 distinct strings into "Notes", a sentinel app frontmost throughout, frontmost app and cursor polled about every 20 ms. Landing is read from the fixture's state file, never from the driver's own report. + +## What decided it + +- **Cua Driver** delivered 10/10 clicks and 10/10 strings with zero steals and zero cursor moves. Element clicks use `AXPress`, typing uses `type_text` against an element token. Every call exited 0. +- **Peekaboo** clicked 10/10 in the background with no steal and no cursor move, but 0/10 strings landed. Its background `type` refused every call with `INVALID_INPUT: The focused element frame is outside the exact target window bounds`. The cause is Stage Manager: with the sentinel in front, the fixture window is a 126x123 pt thumbnail in the strip while its AX frames stay at the real 900x450 geometry, so Peekaboo's frame check fails. With the fixture frontmost, the same `type` call landed the text but reported `INTERACTION_FAILED` (unverifiable). Peekaboo also marks a snapshot stale after any mutation (`SNAPSHOT_STALE`), so every action needs a fresh `see`. +- **orca** landed everything but its element click raises the target app, so it stole focus once and never gave it back (the duration is open-ended: it runs to the end of the measurement). Its element click on a text field only performs `AXConfirm`; focusing the field needed a coordinate click at the field center, taken from System Events because orca's tree has no frames. + +**The winner depends on Stage Manager being on.** Peekaboo ships menu and dialog commands where Cua has only a menu command, so a re-run with Stage Manager off could let it qualify and win the tie-break. The harness is re-runnable for that: `python3 skills/using-apps-like-a-user/scripts/bakeoff.py --runs 2 --candidates peekaboo,cua --report `. An earlier pass of this harness (orca focused the notes field with an element click) scored orca 0/10 strings; the harness was fixed and both recorded runs below use the final version. + +## Environment findings that affect adapters + +- **Stage Manager scales background windows.** `list_windows` and screenshots of a non-frontmost window report the thumbnail frame (126x123 pt, a 252x246 px capture) instead of 900x450. A Cua coordinate click against that capture posts without error and does not land; `scroll` with an element token or x,y is refused in the background (`lies outside window ... 126x123 pt frame`); `drag` maps into the thumbnail. Coordinate actions are reliable only while the target window is frontmost at its true size. +- **Cua Tahoe direct-capture consent.** The first screenshot call made macOS show "CuaDriver is requesting to bypass the system private window picker" while `cua-driver permissions status --json` still reported both permissions granted (`direct_capture_status: not_checked`). Screenshots succeeded anyway. The prompt was left unanswered for the user to decide. + +## Cua Driver command forms that worked + +Every call is `cua-driver call ''` against the fixture's `$P` (pid) and `$W` (window id). Every call carries the same `"session":"